ump-intake

Data onboarding import ump_intake as ui

Client files arrive messy: a title row, a blank line, a subtotal, spend written as "1,200" or "(5)". Ump-intake turns them into one canonical weekly table and checks it still adds up.

It implements the parsing side of spec §6 (datasets, extraction and the G-DATA-02 tie-out). The onboarding wizard and the data catalogue both use it, so a file parses the same way in each.

Quick path

Install: pip install -e 'packages/ump-intake' from a checkout.
Parse, map, build.

import ump_intake as ui

raw = (
    "Weekly media report,,,\n"
    "Week,Revenue,TV Spend ($),Paid Search\n"
    "2025-01-06,1000,100,50\n"
    '2025-01-13,1100,"1,200",60\n'
    "Total,2100,1300,110\n"
).encode()

table = ui.parse_csv_bytes(raw, "weekly.csv")
table.issues
# ['1 title row(s) above the header skipped', 'printed total row separated from the data']

mapping = {s["column"]: s["target"] for s in ui.suggest_mapping(table)}
# {'Week': 'date', 'Revenue': 'kpi:revenue', 'TV Spend ($)': 'media:tv', 'Paid Search': 'media:search'}

panel = ui.build_panel(table, mapping)
panel["columns"], panel["tie_out"]["status"]
# (['date', 'revenue', 'search', 'tv'], 'ties')

Main entry points

Call	What it does
`parse_upload(body)`	Routes an upload (`csv_text`, `content_b64` or inline `data`) to the right parser.
`parse_csv_bytes` / `parse_xlsx_bytes`	Read a file, separating title, blank, total and subtotal rows. XLSX hidden rows, merged cells and extra sheets are reported as issues.
`parse_rows(rows)`	Rows read through a data connection, as lists or dicts.
`suggest_mapping(table)`	A target per column (`date`, `kpi:<id>`, `media:<id>`, `exposure:<id>`, `control:<name>`, `ignore`), each with a confidence, a reason and alternatives.
`build_panel(table, mapping)`	The canonical CSV with channels rolled up and controls as `ctl_<name>`, plus the tie-out.
`tie_out(...)`	Compares extracted totals with printed ones (gate `G-DATA-02`) and names the worst column.

When the totals do not tie

gap = ui.build_panel(table, mapping, {"Paid Search": 200.0})
gap["tie_out"]["status"]      # 'gap'
gap["tie_out"]["headline"]
# {'extracted': 110.0, 'printed': 200.0, 'column': 'Paid Search', 'gap_rel': 0.45}

gap_rel is the relative gap between the extracted and printed column totals, here (200 minus 110) over 200. The default tolerance, TOTALS_TOLERANCE_REL, is 0.01 and is proposed. Pass tolerance_rel= to use another.

Helpers worth knowing

to_number("(1,234.5)") gives -1234.5: brackets read as negatives, and currency signs and thousands separators are dropped.
to_date returns None for text that is not a date, instead of guessing.
media_tree() and channel_of("social.meta") (giving "social") expose the channel hierarchy that mapping and roll-ups use.
put_blob and read_blob store bytes through core’s blob API, given any client with upload_blob and get_blob.

Spec §6 Datasets, extraction and lineage Extraction ratings and the tie-out rule. Reference ump-intake API Every class and function, with signatures.

ump-intake

Quick path

Main entry points

When the totals do not tie

Next