Data Connections

Two ways to bring data in

Augur draws a clean line between a file you hand it once and a source it can re-read whenever you ask:

Upload a file Upload

Oracle workspace · /workspace

Drop a CSV or Parquet into the agent workspace and it becomes the active dataset for the session — the simplest path, and the one that works regardless of where your data lives. Use this for a one-off extract or a hand-built Master Flat File.

Save a live connection Connection

Sanctum · /settings → Data connections

Point Augur at a warehouse table, a query, or a bucket object once. It can then pull fresh data on demand or on a schedule — so a weekly refit reads this week’s numbers without anyone re-exporting a file.

Not the same as the knowledge base

The Codex knowledge base (/knowledge) is for documents — briefs, decks, reports the copilot can cite. It is not a data connection and does not feed the model. Modeling data comes in through an upload or a saved connection, covered here.

Available connections

Two live connectors ship today, both on Google Cloud and both authenticated with Application Default Credentials. They require the optional GCP extra:

uv sync --extra gcp   # google-cloud-bigquery, google-cloud-storage, pyarrow

If a connector isn’t installed, Sanctum → Data connections shows the exact uv sync --extra … command on its card under Data warehouses & storage.

BigQuery Live

kind: bigquery

Run a SQL query, or read a table, into a DataFrame.

query — an arbitrary SQL statement, or
dataset + table — read a table directly
project — GCP project (optional; defaults to the ADC project)
location — e.g. US, EU (optional)
A per-query byte ceiling guards against runaway cost (default 2 GiB billed).

Google Cloud Storage Live

kind: gcs

Read a CSV, Parquet, or TSV object from a bucket.

bucket — bucket name (no gs:// prefix) (required)
object path — e.g. exports/mmm.csv (required)
prefix — scope listings to a folder (optional)
project — GCP project (optional; ADC default)

Ad platforms Via warehouse

Google Ads · Meta Ads · TikTok Ads

There is no direct ad-platform connector yet. The supported path is to land spend in BigQuery with a managed transfer, then connect BigQuery — see Ad-platform spend.

Anything else Upload

CSV / Parquet

Snowflake, Redshift, Postgres, S3, a spreadsheet export — bring it in as a file upload, or route it through BigQuery / GCS. Direct connectors for other warehouses are not in the current release.

How authentication works

The guiding rule is no secrets in the connection. A saved connection stores only non-secret configuration (project, dataset, bucket, query); the credential stays ambient on the server. Both connectors authenticate, in the words of the in-app help, with

“Application Default Credentials (ADC) — the same identity used for Vertex AI.”

Augur resolves credentials in this order:

Application Default Credentials (ADC) — the GCP VM’s service account, a local gcloud auth application-default login, or GOOGLE_APPLICATION_CREDENTIALS. This is the default and needs no key in the app.
An explicit service-account JSON — set MMM_GCP_CREDENTIALS_PATH (or a per-connector path) to a key file on the server.

Because the credential is the server’s ADC identity, the service account it runs as must be granted read access to the BigQuery datasets and GCS buckets you connect — typically roles/bigquery.dataViewer + roles/bigquery.jobUser and roles/storage.objectViewer. The same identity already powers the Vertex AI agent, so on a properly configured GCP deployment there is usually nothing extra to set up.

Save, test, preview

Saved connections live in Sanctum → Data connections (/settings). To add one, give it a Name (e.g. weekly-spend), pick a Type (BigQuery or Google Cloud Storage), fill the fields above, and click Save connection. Each saved connection then offers three actions:

Test — a cheap reachability + auth probe. It returns whether the source was reached and what it reached, with no data read.
Preview — reads the first ~20 rows (capped at 200) and shows them inline as a small table, so you can confirm the columns before committing to a full sync.
Delete — removes the connection and its cached snapshot.

The list shows each connection’s type, a summary of its config (the query, dataset.table, or gs://bucket/object), its auto-sync schedule, and its last-sync status. Preview executes a billable read, so it is gated to the analyst role and stamps the connection’s last synced time.

Syncing data

A saved connection turns into a usable dataset two ways.

On demand

Run a Preview from Sanctum, or simply ask the copilot in the Oracle workspace — the empty-state hint says it best: “in chat say sync my <name> connection.” The agent reads the source and makes the result the active dataset for the session, ready to fit. Under the hood the copilot uses sync_data_connection, or reads a source directly with load_from_bigquery / load_from_gcs.

On a schedule

Each connection has an Auto-sync control with presets Manual, Hourly, Daily, and Weekly (any custom interval from 1 minute to 30 days is allowed via the API). A background scheduler periodically reads every connection that is due, writes a per-project CSV snapshot, and records the outcome on the connection: a green check and row count on success, a red error with a scrubbed message on failure. The status line reads, for example, “synced 2h ago · 1,234 rows.”

What scheduled sync is — and isn’t

Scheduled sync keeps a fresh snapshot on hand; it does not automatically refit your model. You still kick off a refit (the T₃ calibrate / T₀ fit step) from the workspace — the snapshot is simply there waiting, so the refit reads current numbers. Failed syncs back off automatically (up to about a day) and are clearly marked as errors rather than silently reported as “ok.” Each sync is row-capped (5M rows by default) and time-limited so one slow source can’t stall the batch.

Ad-platform spend

Google Ads, Meta Ads, and TikTok Ads are listed in Sanctum (ranked easiest-first) but ship as stubs — there is no live, direct pull from their APIs yet. That is deliberate: the robust, supported pattern is to let a purpose-built transfer service land platform spend in your warehouse, then point Augur at the warehouse.

Platform	Recommended ingestion	Then connect
Google Ads	BigQuery Data Transfer Service (first-party, native)	BigQuery connection (`bigquery`)
Meta Ads	Fivetran / Supermetrics → BigQuery
TikTok Ads	Fivetran / Supermetrics → BigQuery

Once spend is in BigQuery, a single bigquery connection — a query that joins your platforms into one weekly table — feeds the whole model. The rationale and connector contract are spelled out in technical-docs/ad-platform-integrations.md.

Security & governance

No stored secrets. Connections persist only non-secret config; authentication is the server’s ambient ADC identity (or a service-account file path), never a key pasted into the app.
Errors are scrubbed. Cloud errors surfaced in previews and sync results have project IDs, service-account emails, and credential paths redacted before they reach the UI.
Role-gated. Viewers can list and test connections; creating, previewing, syncing, scheduling, and deleting require the analyst role — because preview and sync run billable reads.
Tenant-isolated. Every connection is scoped to its project; a request for a connection that belongs to another project returns a plain 404, so IDs can’t be probed across tenants.

Hosted-deployment caveat

In the current build, scheduled sync runs under the server’s ADC identity for every tenant. On a single-tenant or trusted-team deployment that is fine; true per-tenant credential isolation (workload identity per organization) is a tracked hosted-posture item, not yet shipped. Plan connected sources accordingly.

Config & tool reference

For analysts and engineers wiring this up directly.

Connection config fields

Kind	Fields	Example
`bigquery`	`query` or `dataset`+`table`; `project`, `location`, `maximum_bytes_billed`	`{"project": "acme", "query": "SELECT * FROM mmm.weekly_spend"}`
`gcs`	`bucket`, object path; `prefix`, `project`	`{"bucket": "acme-mmm", "object": "exports/mmm.csv"}`

Server environment variables

# GCP identity / project (shared with the Vertex AI agent)
MMM_GCP_PROJECT=acme-gcp
MMM_GCP_CREDENTIALS_PATH=/path/to/service-account.json   # empty = ADC

# BigQuery defaults
MMM_BIGQUERY_DATASET=marketing
MMM_BIGQUERY_LOCATION=US

# GCS defaults
MMM_GCS_BUCKET=acme-mmm
MMM_GCS_PREFIX=exports/

# Scheduled-sync tuning
MMM_CONNECTION_SYNC_INTERVAL=300        # scheduler tick, seconds (default 300)
MMM_CONNECTION_SYNC_MAX_ROWS=5000000    # row cap per sync (default 5M)
MMM_CONNECTION_SYNC_READ_TIMEOUT=600    # per-connection read timeout, seconds

Agent tools & HTTP endpoints

From chat, the copilot uses load_from_bigquery, load_from_gcs, and sync_data_connection. The REST surface (all under a project) is:

GET    /projects/{id}/data-connections                      # list (viewer)
POST   /projects/{id}/data-connections                      # create (analyst)
PATCH  /projects/{id}/data-connections/{conn}               # set auto-sync interval (analyst)
DELETE /projects/{id}/data-connections/{conn}               # delete (analyst)
POST   /projects/{id}/data-connections/{conn}/test          # reachability probe (viewer)
POST   /projects/{id}/data-connections/{conn}/preview?rows=20 # read sample rows (analyst)
GET    /integrations/catalog                                # available connectors (auth)

For what the data itself must look like once it’s connected, see Data Requirements & Runtime and the Real-Data Guide. The platform tour covers where this lives in the app: Sanctum.