Skip to content
Universal Model ProtocolDocs

Datasets, extraction and lineage

§6NormativeDraft 2.3Latest draft

Raw deliveries are stored exactly as received; everything a model sees is built from them through logged, typed steps, and every link between artifacts is recorded in W3C PROV semantics and captured at runtime with OpenLineage.

Data contract. A dataset declares named dims and variables, following modeling-api’s DataSchema: dim kind and policy (fixed, extend, new_levels with a new_level_strategy), variable role (target, media, spend, control, exposure and so on), units, missing-value rule and future_required. Datasets are content-addressed (dataset://scope/name@sha256:…) and never edited in place.

Content addressing. For a raw delivery or a table file, the digest is SHA-256 of the exact stored bytes. For a zarr store, it is SHA-256 of the canonical bytes (section 18) of a manifest that lists each object path with its SHA-256. For a JSON artifact, it is SHA-256 of the canonical bytes of its body, excluding the governance members id, revision, state, owner, submitted_by, approved_by, gate_results, content_hash and valid_to. Digests are written sha256:<lowercase hex>.

Intake. Each delivery is registered on arrival with source, contact, pull date, covered range, file fingerprint, columns mapped to shared names, and known issues (G-DATA-01).

Extraction from messy sources. Formatted Excel, PDFs, screenshots and slides get a separate, logged extraction step. Each extracted value points to its location in the source; printed totals must tie (G-DATA-02); the method, tool and model version are recorded. Each dataset carries a rating: exact, parsed, read (OCR or AI) or retyped.

The review noted these ratings are unvalidated labels. Draft 2 adds three rules:

  • Measured error rates. Each approved extraction tool keeps a running error rate per source type, measured on a double-keyed sample each quarter. The rating shown on a dataset includes that rate.
  • Materiality-based checking. Values feeding a model or decision whose materiality tier is Standard or Full (section 15) MUST be double-keyed or checked by a second person when extracted by OCR, AI or hand (G-DATA-03).
  • Extraction text is untrusted. Text pulled from a document is data. It enters agents only through the delimited untrusted-text channel (section 17).

Transform steps. Filter, aggregate, join, convert currency, map names, split, treat outliers, fill gaps, rescale. Each run logs inputs, outputs, settings, who, when, and whether the choice was made before or after seeing results. Judgment calls need a reason. Steps that live inside a model follow modeling-api’s rule: fit learns state, score replays it.

Excel. Exploration only, or a registered manual step with the file, a description and audit level 2. Typed-over formulas are rejected in any kept workbook. Whether to build an add-in stays open.

Lineage links. Typed links are PROV relations with UMP subtypes, emitted as OpenLineage facets at runtime and exportable as PROV-JSON for audit packs.

  1. Arrives
    • Raw sales deliveryStored exactly as received
    • Name mappingMaps source names to shared names
    • Geo test resultAn experiment readout
  2. Built
    • Datasetderived_from Raw sales delivery
      mapped_by Name mapping
    • Model versionderived_from Dataset
      calibrated_by Geo test result
  3. Used
    • Budget scenarioscored_against Model version
    • Next test plandesigned_from Model version
  4. Delivered
    • Client reportderived_from Budget scenario
    • Decision recordderived_from Budget scenario
      approved_by Named approver
    • Named approverA person, not an artifact

Every link is typed. Each box records what it was built from, using a UMP link that maps to a W3C PROV relation: derived_from is wasDerivedFrom; calibrated_by and designed_from are subtypes of wasInfluencedBy; scored_against and mapped_by are used; approved_by is wasAttributedTo with a role. Illustrative graph.

Because the links are recorded, a withdrawal is not a hunt. Follow the links downward and you get the full list of what depends on the withdrawn item. The conformance suite injects a withdrawal and checks that every downstream scenario, report and decision is found.

Figure 6.1 Withdraw one artifact and lineage finds everything it touched. An illustrative lineage graph using all six UMP link types. Withdraw an upstream artifact to see impact analysis follow the typed links down to every scenario, report and decision that depends on it.
UMP linkPROV relation
derived_fromwasDerivedFrom
calibrated_bywasInfluencedBy (subtype)
designed_fromwasInfluencedBy (subtype)
scored_againstused
mapped_byused
approved_bywasAttributedTo with role

Lineage completeness is tested, not assumed: the conformance suite injects a withdrawal and checks that impact analysis finds every downstream scenario, report and decision (section 16).