ump_spec.models.dataset

module ump-spec

provisional (not yet in the spec) G-DATA-06 is in the gate registry but not yet in the specification’s gate catalogue, so it is not normative.

Datasets, deliveries and transform steps (spec §6).

Classes

Delivery

class

class Delivery(StrictModel)

G-DATA-01 intake record.

Fields

Name	Type	Default	Description
`source`	`str \| None`	`None`
`contact`	`str \| None`	`None`
`pull_date`	`date \| None`	`None`
`covered_range`	`dict[str, date] \| None`	`None`
`file_fingerprint`	`str \| None`	`None`	`sha256:<hex>` of the raw file (spec §6); a bare hex digest is read and rewritten with the prefix.
`column_mappings`	`dict[str, str]`	`dict()`	column -> registry id
`known_issues`	`list[str]`	`list()`
`registered`	`bool`	`True`

Source: packages/ump-spec/src/ump_spec/models/dataset.py, line 59

DimSpec

class

class DimSpec(StrictModel)

Fields

Name	Type	Default
`kind`	`Literal['categorical', 'time', 'integer', 'float', 'index']`	`'categorical'`
`policy`	`Literal['fixed', 'extend', 'new_levels']`	`'fixed'`
`freq`	`str \| None`	`None`

Source: packages/ump-spec/src/ump_spec/models/dataset.py, line 78

VarSpec

class

class VarSpec(StrictModel)

Fields

Name	Type	Default
`dims`	`list[str]`	`list()`
`dtype`	`str`	`'float64'`
`role`	`Literal['target', 'covariate', 'media', 'spend', 'control', 'index', 'mask', 'constant', 'weight', 'exposure']`	`'covariate'`
`units`	`str \| None`	`None`
`missing`	`Literal['forbid', 'allow', 'mask']`	`'forbid'`

Source: packages/ump-spec/src/ump_spec/models/dataset.py, line 84

LicenceLimits

class

class LicenceLimits(StrictModel)

Vendor or client licence limits checked before export (spec §20).

Fields

Name	Type	Default	Description
`text`	`str \| None`	`None`
`ids`	`list[str]`	`list()`

Source: packages/ump-spec/src/ump_spec/models/dataset.py, line 117

AccessPolicy

class

class AccessPolicy(StrictModel)

Access rules on a data connection, copied onto each connected dataset (POC DATA-CATALOGUE §2).

Fields

Name	Type	Default	Description
`copy_policy`	`CopyPolicy`	`'transient_only'`
`retention_days`	`int \| None`	`None`	For cached copies; defaults to the client contract’s retention_days (rules.effective_access).
`min_segment_size`	`int \| None`	`None`	For aggregates_only; defaults to the contract’s G-PRIV-01 floor.
`allowed_uses`	`list[DataUse]`	`lambda: list(DEFAULT_USES)()`
`licence_limits`	`LicenceLimits \| None`	`None`

Source: packages/ump-spec/src/ump_spec/models/dataset.py, line 124

FetchRecord

class

class FetchRecord(StrictModel)

What a run saw when it read a dataset (POST /v1/datasets/{id}:materialize). Run records and model lineage keep this record, never the rows of a transient_only dataset.

Fields

Name	Type	Default	Description
`dataset_id`	`str`
`dataset_version`	`int \| None`	`None`
`storage`	`DatasetStorage`	`'stored'`
`connection_id`	`str \| None`	`None`
`fingerprint_seen`	`str \| None`	`None`	sha256 hex of what came back this time.
`fingerprint_registered`	`str \| None`	`None`	The dataset’s sha256 at registration.
`drift`	`bool`	`False`
`rows`	`int \| None`	`None`
`fetched_at`	`datetime \| None`	`None`
`copy_policy`	`CopyPolicy`	`'allowed'`
`reachable`	`bool`	`True`
`source`	`Literal['live', 'stored', 'cached_copy', 'none']`	`'live'`	Where the rows came from: a live connector fetch, the stored file, a cached copy used because the connection was unreachable (CACHED_COPY_USED), or nothing (CONNECTION_UNAVAILABLE, step not available).
`connector_stub`	`bool`	`False`
`use`	`DataUse \| None`	`None`
`purpose_artifact_id`	`str \| None`	`None`

Source: packages/ump-spec/src/ump_spec/models/dataset.py, line 136

Dataset

class

class Dataset(ArtifactDoc)

Fields

Name	Type	Default	Description
`ref`	`str`		dataset://{scope}/{name}@sha256:{hash}
`scope`	`str`
`name`	`str`
`sha256`	`str`
`client_id`	`str \| None`	`None`
`layout`	`Literal['dense', 'long']`	`'long'`
`dims`	`dict[str, DimSpec]`	`dict()`
`vars`	`dict[str, VarSpec]`	`dict()`
`extraction_rating`	`ExtractionRating`	`'exact'`
`tool_error_rate`	`float \| None`	`None`
`audit_level`	`Literal[1, 2, 3, 4]`	`2`
`delivery`	`Delivery`	`Delivery()`
`n_rows`	`int \| None`	`None`
`columns`	`list[str]`	`list()`
`blob_uri`	`str \| None`	`None`	Content-addressed blob location (POC: var/blobs).
`totals_check`	`dict[str, Any] \| None`	`None`	G-DATA-02 extracted vs printed.
`storage`	`DatasetStorage`	`'stored'`	stored: a file in the blob store; connected: read through a data_connection when used, rows not kept.
`connection_id`	`str \| None`	`None`	data_connection artifact id (connected only).
`request_spec`	`dict[str, Any] \| None`	`None`	Query or request spec for the connector (connected only). Never holds secrets.
`materialization`	`Materialization`	`'none'`	cached: a copy is kept in the blob store, only when access.copy_policy is allowed.
`access`	`AccessPolicy \| None`	`None`	Copied from the connection; null on a stored dataset means the client contract’s terms apply.
`fingerprint_basis`	`FingerprintBasis \| None`	`None`	Filled from storage when absent: file_bytes (stored) or canonical_rows (connected).
`recipe`	`RecipeRef \| None`	`None`	How this dataset was made: a pointer to the transform recipe uploaded with it (G-DATA-06). Null for raw deliveries and datasets uploaded without one.
`role`	`str \| None`	`None`	Role of the dataset, e.g. raw_delivery or derived. Written by data (`service.py`), onboarding and experiments.
`source_format`	`str \| None`	`None`	Source file format, e.g. csv. Written by data (`service.py`), onboarding and experiments.
`description`	`str \| None`	`None`	Plain description. Written by data (`service.py`), onboarding and experiments.
`connector`	`str \| None`	`None`	Connector kind of a connected dataset. Written by data (`service.py`), onboarding and experiments.
`mapping`	`dict[str, Any] \| None`	`None`	Column mapping onto canonical names. Written by data (`service.py`), onboarding and experiments.
`mapping_stale`	`str \| None`	`None`	Why the stored mapping no longer fits the source. Written by data (`service.py`), onboarding and experiments.
`canonical_columns`	`list[str] \| None`	`None`	Canonical column names after mapping. Written by data (`service.py`), onboarding and experiments.
`fit_hints`	`dict[str, Any] \| None`	`None`	Hints for a first fit: date and KPI columns, channels, controls. Written by data (`service.py`), onboarding and experiments.
`annual_spend`	`dict[str, Any] \| None`	`None`	Annual spend per channel. Written by data (`service.py`), onboarding and experiments.
`extraction`	`dict[str, Any] \| None`	`None`	How the data was extracted: method, tool, checks. Written by data (`service.py`), onboarding and experiments.
`tie_out`	`dict[str, Any] \| None`	`None`	Tie-out against printed totals (tolerance proposed). Written by data (`service.py`), onboarding and experiments.
`recipe_check`	`dict[str, Any] \| None`	`None`	G-DATA-06 recipe check result. Written by data (`service.py`), onboarding and experiments.
`previous_fingerprint`	`str \| None`	`None`	Fingerprint before the latest refresh. Written by data (`service.py`), onboarding and experiments.
`refreshed_at`	`str \| None`	`None`	Last refresh, RFC 3339. Written by data (`service.py`), onboarding and experiments.
`truth`	`dict[str, Any] \| None`	`None`	Known truth of a synthetic panel (synthetic data only). Written by data (`service.py`), onboarding and experiments.
`notes`	`list[str] \| None`	`None`	Note codes. Written by data (`service.py`), onboarding and experiments.

Source: packages/ump-spec/src/ump_spec/models/dataset.py, line 159

TransformStep

class

class TransformStep(ArtifactDoc)

Fields

Name	Type	Default
`type`	`TransformType`
`inputs`	`list[str]`	`list()`
`outputs`	`list[str]`	`list()`
`settings`	`dict[str, Any]`	`dict()`
`learned_state`	`dict[str, Any] \| None`	`None`
`who`	`str \| None`	`None`
`when`	`datetime \| None`	`None`
`timing`	`Literal['before_results', 'after_results']`	`'before_results'`
`reason`	`str \| None`	`None`
`split_assumption`	`str \| None`	`None`

Source: packages/ump-spec/src/ump_spec/models/dataset.py, line 294

Functions

make_dataset_ref

function

def make_dataset_ref(scope: str, name: str, sha256_hex: str) -> str

dataset://{scope}/{name}@sha256:{hash}.

Source: packages/ump-spec/src/ump_spec/models/dataset.py, line 21

parse_dataset_ref

function

def parse_dataset_ref(ref: str) -> dict[str, str]

Source: packages/ump-spec/src/ump_spec/models/dataset.py, line 29

worst_extraction

function

def worst_extraction(ratings: list[str]) -> str | None

Source: packages/ump-spec/src/ump_spec/models/dataset.py, line 36

digest_hex

function

def digest_hex(value: str | None) -> str | None

The bare lowercase hex of a sha256:<hex> or bare-hex digest (compare digests with this on both sides); anything else is returned unchanged.

Source: packages/ump-spec/src/ump_spec/models/dataset.py, line 44

sha256_digest

function

def sha256_digest(value: str | None) -> str | None

Spec §6: a digest is written sha256:<lowercase hex>; a bare hex digest gains the prefix.

Source: packages/ump-spec/src/ump_spec/models/dataset.py, line 53

Constants

Name	Type	Value	Description
`ExtractionRating`		`Literal['exact', 'parsed', 'read', 'retyped']`
`EXTRACTION_ORDER`	`tuple[str, ...]`	`('exact', 'parsed', 'read', 'retyped')`	Best to worst; a model shows the lowest rating among its inputs.
`CopyPolicy`		`Literal['allowed', 'transient_only', 'aggregates_only']`	`allowed`: may be cached to the blob store. `transient_only`: rows live in memory for one run (the default for connections). `aggregates_only`: clean room, only aggregates above the minimum segment size come out (G-PRIV-01..03 run on every fetch).
`COPY_POLICIES`	`tuple[str, ...]`	`('allowed', 'transient_only', 'aggregates_only')`
`DataUse`		`Literal['fit', 'score', 'side_modelling', 'benchmark', 'transform']`	What a client lets UMP do with the data. Benchmark pooling is off unless listed and the licence allows it.
`DATA_USES`	`tuple[str, ...]`	`('fit', 'score', 'side_modelling', 'benchmark', 'transform')`
`DEFAULT_USES`	`tuple[DataUse, ...]`	`('fit', 'score', 'side_modelling')`	Granted when a policy does not list uses; benchmark pooling and transform must be listed explicitly.
`DatasetStorage`		`Literal['stored', 'connected']`
`Materialization`		`Literal['none', 'cached']`
`FingerprintBasis`		`Literal['file_bytes', 'canonical_rows']`	What `sha256` hashes: the raw file’s bytes (stored) or the canonical JSON of the query result’s rows (connected; `rules.canonical_rows_fingerprint`).
`TransformType`		`Literal['filter', 'aggregate', 'join', 'convert_currency', 'map_names', 'spli...`