Observability v0.3

The API and the worker send three signals:

• Metrics: Prometheus text at GET /metrics on each API process. The worker records the series of the orchestrator (dispatch, reconciliation, finalization, pipe event ingestion). When MAPI_WORKER_METRICS_PORT is set, the worker serves them on that port.
• Logs: one JSON object on each line on stderr. Each line has trace_id and span_id, and the request, run and tenant that it is related to.
• Traces: OpenTelemetry spans for each request, each outgoing HTTP call, and the dispatch and finalization of each run. The server exports spans over OTLP/HTTP only when you configure a collector.

Run tracking is a different thing

These signals tell about the platform. The metrics, params, tags, images and tables that a model logs in a run (v0.11 and v0.14) are run data. They go through the run protocol into the database and the bundles bucket, not to /metrics. Refer to Track metrics, params and tags and Compare runs and use experiments. The budgets of the run tracking give the warnings TRACKING_BUDGET and MEDIA_BUDGET (Troubleshooting).

Nothing goes out from inside a sandbox. Each execution gets the traceparent of the dispatching span in MAPI_TRACEPARENT. Thus you can join the stderr diagnostics of the supervisor to the trace of the run, and no telemetry goes out of the exec project.

Settings

Variable	Default	Effect
MAPI_LOG_JSON	true	JSON logs. false gives a console format that is easy to read for local work
MAPI_METRICS_ENABLED	true	Serve GET /metrics
MAPI_METRICS_TOKEN	unset	The bearer that a scraper must send. Necessary in the multi-tenant profile
MAPI_WORKER_METRICS_PORT, MAPI_WORKER_METRICS_HOST	unset, 127.0.0.1	The address where the worker serves its own /metrics (the same bearer rule)
MAPI_TRACING_ENABLED	true	Make spans (their ids go into the logs also without a collector)
MAPI_OTLP_ENDPOINT	unset	OTLP/HTTP collector (OTEL_EXPORTER_OTLP_ENDPOINT also works). Needs the otlp extra of mapi-server

Scraping /metrics

The series contain tenant ids and spend. Thus /metrics is not a public route. It authenticates its own bearer, as the Pub/Sub push endpoint does:

# prometheus.yml
scrape_configs:
  - job_name: mapi-api
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/mapi-metrics-token
    static_configs:
      - targets: ["api.internal:8000", "worker.internal:9464"]

In the multi-tenant profile, a server without MAPI_METRICS_TOKEN refuses all scrapes (401). The single-tenant and dev profiles serve /metrics without authentication on their loopback API, if no token is set.

The metric set

This table comes from mapi_server.observability.metrics.Metrics. This is the one registry that defines all series. Thus the table always agrees with what /metrics serves. The number of label values is limited on purpose. tenant_id is the only unlimited label, on the two counters that have no meaning without it. HTTP series have the route template as label, never the raw path.

Metric	Type	Labels	Meaning
mapi_runs_total	counter	kind, state	Runs that reached a terminal state, by kind and terminal state.
mapi_runs_active	gauge	state	Runs currently in a non-terminal state (sampled each reconcile pass).
mapi_run_transitions_total	counter	state	Run state transitions, by target state (every edge, terminal or not).
mapi_dispatch_latency_seconds	histogram	—	Seconds from a run entering QUEUED to the executor accepting its submission.
mapi_time_to_running_seconds	histogram	—	Seconds from a run entering QUEUED to its first event (RUNNING).
mapi_event_ingestion_lag_seconds	histogram	—	Seconds between a run event's timestamp and its ingestion.
mapi_events_ingested_total	counter	result	Run events seen by the ingestor, by result (accepted, duplicate, bad_hmac, ...).
mapi_finalize_outcomes_total	counter	kind, outcome	Finalize attempts, by outcome (succeeded, partial, integrity, infra, retry).
mapi_integrity_failures_total	counter	tenant_id	Output-integrity verification failures at FINALIZING (a security signal).
mapi_sandbox_violations_total	counter	reason	Sandbox violations observed by the control plane, by reason.
mapi_quota_rejections_total	counter	scope, code	Requests denied by a quota or rate limit, by scope and error code.
mapi_cost_micros_total	counter	tenant_id	Billed compute cost in currency micros, by tenant (cost ledger).
mapi_global_spend_micros	gauge	—	Current spend against the global cap, in currency micros (0 until set).
mapi_global_spend_cap_micros	gauge	—	The configured global spend cap, in currency micros (0 = no cap set).
mapi_sse_clients	gauge	—	Live Server-Sent-Events subscribers connected to this process.
mapi_executor_submit_throttled_total	counter	executor	429 answers from an executor backend to a submission (Cloud Run jobs.run), each one, retried or not. Not the API's own 429s to tenants.
mapi_executor_running_executions	gauge	executor	Runs holding a live execution on this executor (fleet-wide, from the run table; each orchestrator reports the same value, so aggregate with max).
mapi_http_requests_total	counter	method, route, status	HTTP requests handled by the API, by method, route template and status.
mapi_http_request_duration_seconds	histogram	method, route	HTTP request handling time, by method and route template.
mapi_endpoint_promotions_total	counter	outcome	Endpoint promotion and rollback decisions, by outcome (allowed, overridden, needs_override, blocked, forbidden, rollback).
mapi_endpoint_score_requests_total	counter	result, inline	POST /v1/endpoints/{alias}/score calls, by result (accepted, replayed, invalid) and whether the payload was inline XrJSON.
mapi_webhook_events_total	counter	type	Webhook events written to the outbox, by event type.
mapi_webhook_deliveries_total	counter	status	Webhook delivery tries, by resulting status (succeeded, retrying, failed, blocked).
mapi_webhook_delivery_duration_seconds	histogram	—	Time of one webhook delivery try (resolution, connection, answer).
mapi_webhook_ssrf_blocked_total	counter	stage	Webhook URLs refused by the SSRF policy, by stage (create, update, replay, delivery).
mapi_webhooks_disabled_total	counter	reason	Webhooks the platform disabled, by reason (failures, gone).

Monitor these series first:

• mapi_time_to_running_seconds: the M2 target is a p95 below 90 s.
• mapi_integrity_failures_total and mapi_sandbox_violations_total: each increase is a security signal, not a reliability signal.
• mapi_global_spend_micros against mapi_global_spend_cap_micros: the window of the kill switch.
• mapi_runs_active{state="stalled"}: runs that sent no events for some time. The status of the executor decides what occurs to them, not the absence of events.

The security suite adds one more series from outside the server, mapi_security_probe_unexpected{executor}, through a textfile collector or a Pushgateway (Security suite).

Each response has X-Request-Id. This is the header of the caller if the caller sent one. If not, it is the trace id. When you report a problem, give this id. It finds the log lines and the trace.

Dashboards, alerts and runbooks

All the files that operators need are in ops/ at the root of the repository. ops/README.md tells how to import each one. The tables below come from those files.

Dashboards

A Grafana dashboard and a Cloud Monitoring dashboard show the same series:

File	Dashboard	Sections
ops/dashboards/cloud-monitoring-mapi-overview.json	Cloud Monitoring: modeling-api — control plane	15 tiles: Running, Stalled, SSE clients, Integrity failures (1h), Active runs by state, Terminal outcomes (rate), Dispatch latency p95, Time to RUNNING p95 (SLO 90s), Event ingestion lag p95, Finalize outcomes (rate), Sandbox violations (rate), Quota rejections (rate), Cost per tenant (micros/s), HTTP requests by status, jobs.run 429 (Cloud Run)
ops/dashboards/grafana-mapi-overview.json	Grafana: modeling-api — control plane	18 panels in 6 rows: Fleet overview, Runs, Dispatch & time-to-RUNNING, Events & finalize, Security & abuse, HTTP API

Alerts

ops/alerts/prometheus-rules.yml has the Prometheus rules, with recording rules such as mapi:global_spend_ratio. ops/alerts/cloud-monitoring-alerts.yaml has the same policies for Cloud Monitoring. A page alert wakes a person. A ticket alert is for the next working hour. The thresholds are initial values for the private beta.

Alert	Severity	Fires when	Says	Runbook
MapiIntegrityFailure	page	increase(mapi_integrity_failures_total[10m]) > 0	Integrity failure finalizing a run (N in 10m)	ops/runbooks/sandbox-escape.md
MapiSecurityProbeFailure	page	max(mapi_security_probe_unexpected) > 0	Security probe had an unexpected outcome on an enforcing executor	ops/runbooks/sandbox-escape.md
MapiSandboxViolationSpike	ticket	mapi:sandbox_violations:rate5m > 0.2 for 10m	Sandbox violations () elevated	ops/runbooks/sandbox-escape.md
MapiGlobalSpendCap70	ticket	mapi:global_spend_ratio >= 0.70 and mapi:global_spend_ratio < 0.90 for 5m	Global spend at N % of the cap	ops/runbooks/incident-response.md
MapiGlobalSpendCap90	page	mapi:global_spend_ratio >= 0.90 for 1m	Global spend at N % of the cap	ops/runbooks/incident-response.md
MapiJobsRun429	ticket	mapi:executor_submit_throttled:rate5m > 0 for 5m	429s on jobs.run from (N/s)	ops/runbooks/stuck-runs.md
MapiRunningExecutions70	ticket	max by (executor) (mapi_executor_running_executions{executor=~"cloudrun.*"}) > 700 for 10m	: N running executions (70 % of the 1,000 quota)	ops/runbooks/stuck-runs.md
MapiQuotaRejections	ticket	sum by (scope) (rate(mapi_quota_rejections_total{code="QUOTA_EXCEEDED"}[15m])) > 0.05 for 15m	Quota rejections sustained at the level	ops/runbooks/incident-response.md
MapiStalledRateHigh	page	mapi:runs_stalled / clamp_min(mapi:runs_running + mapi:runs_stalled, 1) > 0.2 for 10m	N % of live runs are STALLED	ops/runbooks/stuck-runs.md
MapiTimeToRunningSlow	ticket	histogram_quantile(0.95, sum(rate(mapi_time_to_running_seconds_bucket[30m])) by (le)) > 90 for 15m	p95 time-to-RUNNING Ns (> 90s SLO)	ops/runbooks/stuck-runs.md
MapiNoProgressIngested	page	(sum(rate(mapi_events_ingested_total{result="accepted"}[10m])) or vector(0)) == 0 and sum(mapi_runs_active{state=~"running\|provisioning"}) > 0 for 10m	Runs are live but no events are being ingested	ops/runbooks/db-failover.md

Runbooks

Each alert names the runbook that solves it:

Runbook	For	Starts with
ops/runbooks/db-failover.md	database (Postgres) failover / outage	Triggered by MapiNoProgressIngested, /readyz reporting postgres: fail, a Cloud SQL failover event, or widespread 5xx.
ops/runbooks/incident-response.md	incident response (control plane)	The general flow for any control-plane incident.
ops/runbooks/rotating-keys.md	rotating the HMAC / signing keys	Two long-lived secrets protect run integrity (plan §8.6, §8.7, §9).
ops/runbooks/sandbox-escape.md	suspected or confirmed sandbox escape	Triggered by MapiIntegrityFailure, MapiSecurityProbeFailure, a MapiSandboxViolationSpike, or any external report of untrusted code reaching something it should not.
ops/runbooks/stuck-runs.md	stuck runs (STALLED, stuck in a state, jobs.run 429)	Triggered by MapiStalledRateHigh, MapiTimeToRunningSlow, MapiJobsRun429, MapiRunningExecutions70, MapiNoProgressIngested, or tenant reports that a run "isn't moving".

The sandbox-escape runbook is the incident response of the plan for a confirmed escape. It has these steps:

1. Turn off uploaded code with the kill switch.
2. Rotate the signing and HMAC keys. This makes all signed URLs that are open now invalid.
3. Put the affected bundles in quarantine.
4. Tell the tenants.

Tests that guard the signals

Command	What it checks
make chaos	A worker that stops during PROVISIONING and RUNNING, Pub/Sub redelivery and a changed sequence, redelivery with a bad HMAC, and a PostgreSQL restart during a run. Each run comes to one outcome, with one attempt and no duplicate execution
make load-test	100 concurrent fits through the real dispatcher, reconciler and finalizer on the fake Cloud Run backend, with injected 429s, lost responses and duplicate pushes. No run is lost, and the p95 time-to-RUNNING stays below 90 s
make security-suite	The sandbox-escape probes (refer to Security suite)

All three run offline, against the embedded PostgreSQL and the in-memory or shim object store.