Storage, privacy and clean rooms
Large immutable files live in GCS. The catalogue, lineage and anything else transactional live in Postgres, which is the source of truth, and BigQuery mirrors it in near real time for anything people query. Data that may not leave a clean room stays there, with privacy limits enforced in UMP’s own queries and agent tools as well as in the clean room.¶
| Store | Holds | Why |
|---|---|---|
| GCS, content-addressed, write-once for bundles¶ | Posterior draws, spend tables, code, raw deliveries | Large files that never change |
| Postgres¶ | Catalogue, lineage edges, shared names, scenario results | Transactional source of truth |
| BigQuery¶ | Near-real-time CDC mirror of Postgres, plus a queryable mirror of the log | Analysts, dashboards and agents query it |
| Merkle log plus external anchors¶ | Sealed records (section 18) | Source of truth for integrity; the BigQuery mirror is checked against it |
| Clean rooms (InfoSum first)¶ | Member-level data and models that may not move | Privacy and contract limits |
Detail levels. A model may keep all draws, a thinned set and a summary. A storage rule decides which to keep and for how long. Thinning preserves the shared thinning index so joint dependence survives.¶
Clean rooms. Catalogue entries for clean-room items hold ID, fingerprint, schema and access rules only. Computation runs inside where possible; only permitted aggregates come out, as new items, and every crossing is logged.¶
Privacy beyond the clean room. Aggregation thresholds inside a clean room do not stop differencing attacks, where two allowed aggregates are subtracted to reveal a small group. UMP therefore enforces, in its query layer and in every agent tool:¶
- the minimum segment size from the client contract (
G-PRIV-01). When the client contract states no minimum segment size, the registered default applies (proposed 50 members);¶ - an overlap check that refuses a new aggregate if, combined with any set of earlier released aggregates over the same population, it isolates a group below the floor. Pairwise checks are necessary and not sufficient (
G-PRIV-03);¶ - a per-scope query budget, counted in released aggregates per scope per year. Any release that crosses a client boundary outside a clean room MUST use the privacy-budget ledger of section 25.8: epsilon per client per year, basic sequential composition, client-level replace-one neighbours.¶
Access and retention. Client data is separated by scope with per-item access rules, encryption and retention set by contract. Vendor licence limits are stored on the item and checked before export.¶