ump_spec.models.evidence

module ump-spec

Evidence cards (spec §17, §19 /evidence-card) and acceptance reports (§14).

Classes

EvidenceCard

class

class EvidenceCard(ArtifactDoc)

Fields

Name	Type	Default	Description
`subject_id`	`str`		Artifact the card summarises.
`subject_kind`	`str \| None`	`None`
`client_id`	`str \| None`	`None`
`title`	`str`
`summary`	`str`		Plain-words summary; every number in it appears in headline.
`headline`	`list[ShownNumber]`	`list()`
`trust_level`	`int`	`2`
`audit_level`	`Literal[1, 2, 3, 4]`	`2`
`lineage`	`list[str]`	`list()`	Upstream artifact ids.
`gate_results`	`list[GateResult]`	`list()`
`fallbacks`	`list[FallbackEntry]`	`list()`
`notes`	`list[Note]`	`list()`
`session_id`	`str \| None`	`None`	Onboarding session that produced the card. Written by onboarding (`wizard.py`, `plan.py`), governance, planner and experiments (`mei_anchor.py`).
`dataset_ref`	`str \| None`	`None`	Dataset the card is about. Written by onboarding (`wizard.py`, `plan.py`), governance, planner and experiments (`mei_anchor.py`).
`checklist`	`list[Any] \| None`	`None`	Onboarding checklist items. Written by onboarding (`wizard.py`, `plan.py`), governance, planner and experiments (`mei_anchor.py`).
`contract_summary`	`dict[str, Any] \| None`	`None`	Contract terms in brief. Written by onboarding (`wizard.py`, `plan.py`), governance, planner and experiments (`mei_anchor.py`).
`first_model`	`dict[str, Any] \| None`	`None`	Recommended first model and why. Written by onboarding (`wizard.py`, `plan.py`), governance, planner and experiments (`mei_anchor.py`).
`first_tests`	`list[Any] \| None`	`None`	Ranked first tests. Written by onboarding (`wizard.py`, `plan.py`), governance, planner and experiments (`mei_anchor.py`).
`materiality`	`dict[str, Any] \| None`	`None`	Materiality tier with reasons, inputs and proposed thresholds (G-MAT-01). Written by onboarding (`wizard.py`, `plan.py`), governance, planner and experiments (`mei_anchor.py`).
`materiality_tier`	`str \| None`	`None`	Materiality tier. Written by onboarding (`wizard.py`, `plan.py`), governance, planner and experiments (`mei_anchor.py`).
`maturity`	`dict[str, Any] \| None`	`None`	Spec §25.5 maturity at the time of the card. Written by onboarding (`wizard.py`, `plan.py`), governance, planner and experiments (`mei_anchor.py`).
`side`	`dict[str, Any] \| None`	`None`	Engine side-run summary. Written by onboarding (`wizard.py`, `plan.py`), governance, planner and experiments (`mei_anchor.py`).
`result`	`dict[str, Any] \| None`	`None`	Engine result summary. Written by onboarding (`wizard.py`, `plan.py`), governance, planner and experiments (`mei_anchor.py`).
`model_id`	`str \| None`	`None`	Model the card is about. Written by onboarding (`wizard.py`, `plan.py`), governance, planner and experiments (`mei_anchor.py`).
`anchor`	`dict[str, Any] \| None`	`None`	MEI anchor summary (experiments `mei_anchor.py`). Written by onboarding (`wizard.py`, `plan.py`), governance, planner and experiments (`mei_anchor.py`).
`assumptions`	`list[Any] \| None`	`None`	Assumed values, each labelled. Written by onboarding (`wizard.py`, `plan.py`), governance, planner and experiments (`mei_anchor.py`).
`attach_only`	`bool \| None`	`None`	True when the evidence is attach-only (R6, Tier C). Written by onboarding (`wizard.py`, `plan.py`), governance, planner and experiments (`mei_anchor.py`).
`sent`	`list[Any] \| None`	`None`	Test results sent to the anchor. Written by onboarding (`wizard.py`, `plan.py`), governance, planner and experiments (`mei_anchor.py`).
`skipped`	`list[Any] \| None`	`None`	Test results skipped, with reasons. Written by onboarding (`wizard.py`, `plan.py`), governance, planner and experiments (`mei_anchor.py`).

Source: packages/ump-spec/src/ump_spec/models/evidence.py, line 16

AcceptanceMetric

class

class AcceptanceMetric(StrictModel)

Fields

Name	Type	Default
`name`	`str`
`value`	`float \| None`
`threshold`	`Proposed`
`comparator`	`Literal['lt', 'le', 'gt', 'ge', 'abs_le']`	`'le'`
`passed`	`bool \| None`	`None`

Methods and properties (1)

evaluate method

def evaluate(self) -> AcceptanceMetric

Source: packages/ump-spec/src/ump_spec/models/evidence.py, line 100

AcceptanceLayer

class

class AcceptanceLayer(StrictModel)

Fields

Name	Type	Default
`name`	`Literal['computation', 'identification', 'decisions']`
`status`	`Literal['untested', 'passed', 'failed']`	`'untested'`
`metrics`	`list[AcceptanceMetric]`	`list()`
`detail`	`str`	`''`

Methods and properties (1)

evaluate method

def evaluate(self) -> AcceptanceLayer

Source: packages/ump-spec/src/ump_spec/models/evidence.py, line 117

ControlOutcome

class

class ControlOutcome(StrictModel)

Fields

Name	Type	Default
`engine`	`str`
`model_id`	`str \| None`	`None`
`expected`	`Literal['pass', 'fail']`
`observed`	`Literal['pass', 'fail', 'untested']`	`'untested'`

Methods and properties (1)

as_expected property

as_expected: bool

Source: packages/ump-spec/src/ump_spec/models/evidence.py, line 132

AcceptanceReport

class

class AcceptanceReport(ArtifactDoc)

Fields

Name	Type	Default	Description
`model_id`	`str \| None`	`None`
`capability_id`	`str \| None`	`None`
`engine`	`str`
`family`	`str`	`'media_mix'`
`preregistration`	`dict[str, Any]`	`dict()`
`layers`	`list[AcceptanceLayer]`	`list()`
`null_battery`	`dict[str, Any]`	`dict()`
`negative_control`	`ControlOutcome \| None`	`None`
`positive_control`	`ControlOutcome \| None`	`None`
`role`	`Literal['candidate', 'negative_control', 'positive_control']`	`'candidate'`
`status`	`Literal['untested', 'passed', 'failed']`	`'untested'`
`bar_broken`	`bool`	`False`	True if the bar passes MTM or fails the positive control.
`notes`	`list[Note]`	`list()`
`threshold_set_version`	`str \| None`	`None`	§14: the threshold-set version the bar was tested against, e.g. thresholds/0.1.0.
`regrade_by`	`date \| None`	`None`	§14: re-run due by this date after a mark binds or changes (90 days, proposed).
`details`	`dict[str, Any] \| None`	`None`	Per-layer detail of the acceptance run. Written by models (`acceptance.py`) and Newton’s seed.
`engine_adapter`	`str \| None`	`None`	Adapter that ran the battery. Written by models (`acceptance.py`) and Newton’s seed.
`stub`	`bool \| None`	`None`	True when a stub engine produced the report. Written by models (`acceptance.py`) and Newton’s seed.
`seed_tag`	`str \| None`	`None`	Seed label (Newton seed data). Written by models (`acceptance.py`) and Newton’s seed.

Methods and properties (1)

evaluate method

def evaluate(self) -> AcceptanceReport

Status: failed if any layer failed; passed if all three passed; else untested. Untested is usable with reduced trust; failed is excluded (G-ACC-01).

A candidate passes only against a working bar (spec §14: the suite MUST include the MTM negative control). If the control was not run, did not fail, or the bar is broken, a three-layer pass is held at untested with a note; the layer results are kept.

Source: packages/ump-spec/src/ump_spec/models/evidence.py, line 143

AgentEvalReport

class

class AgentEvalReport(ArtifactDoc)

§17 agent evaluation harness run (G-NEW-03 release gate) for one agent version.

evaluate() sets each measure’s passed and the overall status: failed if any measured value misses its (proposed) mark, passed only when all six measures were run and pass, untested otherwise. G-NEW-03 reads this report: anything but passed holds the version (AGENT_VERSION_HELD).

Fields

Name	Type	Default	Description
`agent`	`str`		Agent principal, e.g. agent:newton-actor.
`agent_version`	`str`
`playbook_version`	`str \| None`	`None`
`tool_server_version`	`str \| None`	`None`
`golden_set`	`dict[str, Any]`	`dict()`	{id, n_questions, held_out_share}; held-out third never shared.
`measures`	`list[AcceptanceMetric]`	`_harness_measures()`
`status`	`Literal['untested', 'passed', 'failed']`	`'untested'`
`previous_passing_version`	`str \| None`	`None`	Version that stays live if this one is held.
`run_ids`	`list[str]`	`list()`
`notes`	`list[Note]`	`list()`

Methods and properties (1)

evaluate method

def evaluate(self) -> AgentEvalReport

Source: packages/ump-spec/src/ump_spec/models/evidence.py, line 235

Constants

Name	Type	Value	Description
`HARNESS_MEASURES`	`tuple[str, ...]`	`('exact_answer_accuracy', 'citation_validity', 'citation_precision', 'honest_...`	The six §17 harness measures (injection resistance split into write/run successes and exfiltrations).