Server settings¶
Every setting of mapi-server (api, worker, migrate, dev), generated from mapi_server.settings.Settings by scripts/gen_docs_reference.py. Each field reads the environment variable MAPI_<FIELD>; .env files are not read. The configuration reference explains them by task and lists the variables of the other components.
| Variable | Type | Default | Description |
|---|---|---|---|
MAPI_ENV |
str |
local |
Selects bucket names mapi-{env}-* |
MAPI_PROFILE |
single-tenant | multi-tenant | dev |
single-tenant |
|
MAPI_TENANT_ID |
str |
default |
|
MAPI_PROJECT_ID |
str |
default |
|
MAPI_DEBUG |
bool |
False |
Include tracebacks in problem.untrusted_text |
MAPI_API_HOST |
str |
127.0.0.1 |
|
MAPI_API_PORT |
int |
8000 |
|
MAPI_API_URL |
str |
http://127.0.0.1:8000 |
|
MAPI_WEB_ORIGIN |
str |
http://127.0.0.1:5173 |
|
MAPI_PLOTS_ORIGIN |
str |
http://127.0.0.1:5174 |
|
MAPI_DATA_DIR |
Path |
.local |
|
MAPI_ALLOWED_HOSTS |
list |
(computed) | Host header values the API answers (comma-separated in the environment; '*' turns the check off). The host of api_url is always allowed. The API is unauthenticated in M1, so this is what stops DNS-rebinding pages from using it. |
MAPI_DATABASE_URL or DATABASE_URL |
str \| None |
unset | postgresql:// URL; unset = embedded pgserver under data_dir/pg |
MAPI_DB_POOL_SIZE |
int |
10 |
|
MAPI_DB_SYSTEM_POOL_SIZE |
int |
5 |
|
MAPI_DB_GATEWAY_POOL_SIZE |
int |
5 |
|
MAPI_DB_ECHO |
bool |
False |
|
MAPI_DATABASE_APP_URL |
str \| None |
unset | Login of the request-path role (row-level security applies). Unset: connect with DATABASE_URL and SET ROLE db_app_role on every connection |
MAPI_DATABASE_SYSTEM_URL |
str \| None |
unset | Login of the platform role: the orchestrator's dispatcher, reconciler, finalizer and housekeeper (bypasses row-level security; the API never needs it). Unset: DATABASE_URL with SET ROLE db_system_role |
MAPI_DATABASE_GATEWAY_URL |
str \| None |
unset | Login of the gateway role: what the API and events service look up before a tenant is known (sign-in, memberships, operator metadata, pushed events; row-level security applies). Unset: DATABASE_URL with SET ROLE db_gateway_role |
MAPI_DB_APP_ROLE |
str |
mapi_app |
|
MAPI_DB_SYSTEM_ROLE |
str |
mapi_system |
|
MAPI_DB_GATEWAY_ROLE |
str |
mapi_gateway |
|
MAPI_OBJECT_STORE |
shim | memory | gcs |
shim |
|
MAPI_GCS_ENDPOINT |
str |
http://127.0.0.1:4443 |
|
MAPI_GCS_PUBLIC_ENDPOINT |
str \| None |
unset | Base URL put into signed URLs when executions/browsers reach the shim under another host name (default: gcs_endpoint) |
MAPI_GCP_PROJECT |
str \| None |
unset | Google Cloud project of the Cloud Storage client (MAPI_OBJECT_STORE=gcs) |
MAPI_URL_SIGNER |
str \| None |
unset | Service account that signs Cloud Storage URLs through IAM signBlob (keyless; required with MAPI_OBJECT_STORE=gcs) |
MAPI_URL_UPLOADER |
str \| None |
unset | Service account that signs create-only upload policies (objects.create, never delete; required with MAPI_OBJECT_STORE=gcs and distinct from url_signer) |
MAPI_GCS_ROOT |
Path \| None |
unset | The GCS shim's state directory (default: data_dir/gcs) |
MAPI_GCS_KEY_FILE |
Path \| None |
unset | Key file of the shim's control principal (default: gcs_root/_shim/keys/control.json) |
MAPI_EXEC_TOKEN |
str \| None |
unset | Bearer token of the shim's exec principal (local executors only; default: read from gcs_root/_shim/keys/exec.json) |
MAPI_EXECUTOR |
local-process | docker | docker-gvisor | cloudrun-jobs |
local-process |
|
MAPI_RUNNER_COMMAND |
list |
(computed) | argv of the supervisor (shell-quoted string in the environment) |
MAPI_DOCKER_IMAGE |
str |
mapi/base-bayes:0.15.1 |
Runner image for runs without a mapped plugin (ingest, describe, ...) |
MAPI_DOCKER_IMAGES |
dict |
(computed) | Plugin namespace → runner image (JSON in the environment) |
MAPI_DOCKER_NETWORK |
str |
none |
Network of run containers: 'container: |
MAPI_DOCKER_NETWORK_ISOLATED |
bool |
False |
Declare MAPI_DOCKER_NETWORK an internal network that holds only the GCS shim (no route out, no DNS beyond it), so docker-gvisor reports enforced egress and DNS. Shared tiers still need a passing security-suite run recorded for these caps |
MAPI_DOCKER_RUNTIME_GVISOR |
str |
runsc |
|
MAPI_DOCKER_GPUS |
bool |
False |
Give GPU-class run containers one NVIDIA GPU (a docker device request; needs the NVIDIA container toolkit on the host). Off: GPU-class runs use the CPU |
MAPI_EXECUTOR_WORKDIR |
Path \| None |
unset | |
MAPI_CANCEL_GRACE_S |
float |
10.0 |
|
MAPI_CLOUDRUN_PROJECT |
str \| None |
unset | Exec project that holds the Cloud Run Job inventory (cloudrun-jobs) |
MAPI_CLOUDRUN_REGION |
str |
europe-west1 |
Region of the Job inventory |
MAPI_DEFAULT_RESIDENCY |
str \| None |
unset | Data residency of the platform tenant, of tenants adopted by the migrations and of tenants created without one (default: the multi-region of MAPI_CLOUDRUN_REGION for cloudrun-jobs, e.g. eu, else us) |
MAPI_CLOUDRUN_JOB_TEMPLATE |
str |
mapi-{env}-{base}-{class} |
Job id convention; {env}, {base} (base-bayes|base-ml|base-forecast|base-dl) and {class} (s|m|l|gpu) are substituted. Must match the Jobs Terraform creates |
MAPI_CLOUDRUN_PLUGIN_BASES |
dict |
(computed) | Plugin namespace → base image family (JSON in the environment) |
MAPI_CLOUDRUN_DEFAULT_BASE |
str |
base-bayes |
Base for runs without a mapped plugin (ingest, describe) |
MAPI_CLOUDRUN_GPU_JOBS |
bool |
False |
The inventory has the GPU Job of base-dl (Terraform exec gpu_jobs), so GPU-class runs of dl models are placed on it. Off: the GPU class is not offered |
MAPI_CLOUDRUN_SUBMITS_PER_MINUTE |
float |
150.0 |
Token-bucket rate of jobs.run calls per orchestrator; the regional quota is 180/min [verify M0], so split it between orchestrator replicas |
MAPI_CLOUDRUN_SUBMIT_BURST |
int |
10 |
|
MAPI_CLOUDRUN_MAX_429_RETRIES |
int |
6 |
429 answers before giving up |
MAPI_CLOUDRUN_BACKOFF_BASE_S |
float |
1.0 |
|
MAPI_CLOUDRUN_BACKOFF_CAP_S |
float |
64.0 |
|
MAPI_CLOUDRUN_TIMEOUT_MARGIN_S |
int |
120 |
Task timeout = run timeout + margin (the supervisor flushes) |
MAPI_CLOUDRUN_EGRESS_ENFORCED |
bool |
True |
The exec network denies all egress except the restricted VIP |
MAPI_CLOUDRUN_DNS_ENFORCED |
bool |
False |
The DNS response policy is verified to apply to Jobs [verify M0]; verified and public tenants are only placed when both egress and DNS are enforced (and suite evidence is recorded) |
MAPI_CLOUDRUN_ADOPT_LOOKBACK_S |
float |
93600 |
How far back orphan adoption lists executions |
MAPI_PUBSUB_PUSH_AUDIENCE |
str \| None |
unset | Expected audience of the Pub/Sub push OIDC token (usually the push endpoint URL). When set, POST /internal/events requires a Google-signed token |
MAPI_PUBSUB_PUSH_SERVICE_ACCOUNT |
str \| None |
unset | Service account the push subscription signs its tokens as |
MAPI_PUBSUB_PUSH_SUBSCRIPTIONS |
list |
(computed) | Accepted subscription names (comma-separated; empty = any) |
MAPI_GOOGLE_JWKS_URL |
str |
https://www.googleapis.com/oauth2/v3/certs |
JWKS of Google's OIDC signing keys (push token verification) |
MAPI_DISPATCH_RATE_PER_MINUTE |
float |
180.0 |
|
MAPI_DISPATCH_BURST |
int |
10 |
|
MAPI_DISPATCH_BATCH |
int |
10 |
|
MAPI_DISPATCH_POLL_S |
float |
1.0 |
|
MAPI_DISPATCH_LEASE_S |
float |
120.0 |
|
MAPI_DISPATCH_TIERS |
list |
(computed) | Trust tiers whose runs this orchestrator leases (comma-separated); empty: every tier. Give each orchestrator a disjoint list when several serve different exec pools, so tenant tiers never share a pool |
MAPI_RECONCILE_INTERVAL_S |
float |
10.0 |
|
MAPI_STALL_INTERVALS |
int |
15 |
K: silence > K x progress interval |
MAPI_COMPLETE_EVENT_GRACE_S |
float |
60.0 |
|
MAPI_LOST_EXECUTION_GRACE_S |
float |
60.0 |
|
MAPI_MAX_INFRA_RETRIES |
int |
2 |
|
MAPI_AUTO_RETRY_WINDOW_S |
float |
1800.0 |
No auto-retry beyond 30 min |
MAPI_AUTO_RETRY_COST_FACTOR |
float |
2.0 |
No auto-retry once a run's metered cost reaches this multiple of its estimate (plan §8.3: retries capped by cost) |
MAPI_FINALIZE_MAX_ATTEMPTS |
int |
5 |
|
MAPI_DEADLINE_GRACE_S |
float |
60.0 |
|
MAPI_HOUSEKEEPING_INTERVAL_S |
float |
300.0 |
Sweep of expired upload intents and other expiring rows |
MAPI_TRACKING_RETENTION_DAYS |
int \| None |
unset | Days after a run finishes that its metric points and user log lines are kept (unset: kept with the run). Summaries, params, tags and notes are always kept |
MAPI_CATALOG_PATH |
Path \| None |
unset | Catalog JSON exported at image build (runner describe) |
MAPI_IMAGE_DIGEST |
str \| None |
unset | |
MAPI_UV_LOCK_PATH |
Path \| None |
unset | |
MAPI_SIGNING_KEY_PATH |
Path \| None |
unset | PEM of the bundle-signing key (default: data_dir/keys/bundle-signing-p256.pem); every process that signs or verifies bundles must share it |
MAPI_SIGNING_TRUSTED_KEY_PATHS |
list |
(computed) | PEM files (public or private) of retired bundle-signing keys whose signatures still verify (comma-separated): key rotation without invalidating the bundles the old key signed |
MAPI_SIGNING_KEY_CREATE |
bool |
True |
Create the signing key when it is missing; off where mapi-server migrate creates it on shared storage (compose api/worker) |
MAPI_SIGNER |
local | kms |
local |
local = PEM key file; kms = Cloud KMS asymmetric key |
MAPI_KMS_KEY_VERSION |
str \| None |
unset | projects/…/cryptoKeys/…/cryptoKeyVersions/N that signs bundle manifests |
MAPI_KMS_TRUSTED_KEY_VERSIONS |
list |
(computed) | Older key versions whose signatures still verify (comma-separated) |
MAPI_INTERNAL_TOKEN |
str \| None |
unset | If set, POST /internal/events requires this bearer |
MAPI_AUTH_REQUIRED |
bool \| None |
unset | Require a session or API token on every non-public route (default: required in every profile except single-tenant) |
MAPI_SERVER_SECRET |
str \| None |
unset | Root secret of the token pepper and audit fingerprints (default: read or create server-secret next to the bundle-signing key). Changing it invalidates every API token. |
MAPI_SERVER_SECRET_PATH |
Path \| None |
unset | |
MAPI_COOKIE_SECURE |
bool \| None |
unset | Secure, __Host- prefixed cookies (default: on when api_url is https) |
MAPI_OIDC_ISSUER |
str \| None |
unset | OpenID Connect issuer (e.g. https://accounts.google.com) |
MAPI_OIDC_DISCOVERY_URL |
str \| None |
unset | Default: |
MAPI_OIDC_CLIENT_ID |
str \| None |
unset | |
MAPI_OIDC_CLIENT_SECRET |
str \| None |
unset | |
MAPI_OIDC_REDIRECT_URI |
str \| None |
unset | Registered redirect URI (default: |
MAPI_OIDC_ALGORITHMS |
list |
(computed) | |
MAPI_OIDC_CLI_CLIENT_IDS |
list |
(computed) | Client ids of public OIDC clients (the mapi CLI) whose ID tokens of the same issuer are accepted as bearers (comma-separated; empty: API tokens only) |
MAPI_OIDC_EXTRA_PARAMS |
dict |
(computed) | Extra authorization parameters, e.g. {"hd": "acme.com"} (JSON) |
MAPI_OPERATOR_EMAILS |
list |
(computed) | Verified e-mails that sign in as platform operators (comma-separated) |
MAPI_DEFAULT_TENANT_ADMINS |
list |
(computed) | E-mails invited as admins of the default tenant at start-up (comma-separated; the invitation is claimed at first sign-in) |
MAPI_DEVIDP_USERS |
list |
(computed) | Accounts of the development identity provider (MAPI_PROFILE=dev only) |
MAPI_PRICE_TABLE_PATH |
Path \| None |
unset | JSON PriceTable (default: mapi_spec.cost.DEFAULT_PRICE_TABLE) |
MAPI_ESTIMATE_WALL_S |
dict |
(computed) | |
MAPI_RATE_LIMITS_ENABLED |
bool |
True |
|
MAPI_LEASE_CANDIDATES |
int |
500 |
Queued runs the fair-share planner considers |
MAPI_LEASE_CANDIDATES_PER_TENANT |
int |
50 |
At most this many of them per tenant (a tenant whose runs are held cannot fill the window) |
MAPI_EXECUTOR_REPORT_INTERVAL_S |
float |
10.0 |
|
MAPI_EXECUTOR_EVIDENCE_MAX_AGE_S |
float |
259200.0 |
How long a recorded conformance or security-suite report lets an executor take verified and public tenants (the nightly suites renew it) |
MAPI_SSE_HEARTBEAT_S |
float |
15.0 |
|
MAPI_SSE_POLL_S |
float |
5.0 |
Fallback poll when no NOTIFY arrives |
MAPI_SSE_REAUTH_S |
float |
60.0 |
How often an open event stream re-checks its caller's credential (a revoked token, an expired session or a lost membership closes the stream) |
MAPI_MAX_SIDECAR_BYTES |
int |
8388608 |
|
MAPI_LOG_LEVEL |
debug | info | warning | error |
info |
|
MAPI_RENDER_TIMEOUT_S |
int |
300 |
Wall-clock limit of a POST /v1/plots:render run (the sandbox kills it) |
MAPI_INSPECT_TIMEOUT_S |
int |
300 |
Wall-clock limit of an upload inspect run (M4; it reads only metadata) |
MAPI_RENDER_PLOTLY_ENABLED |
bool |
False |
Accept Plotly plots in POST /v1/plots:render. Plotly static export needs kaleido and a Chrome/Chromium in the runner image, which the base images do not ship; while off, Plotly renders are refused with MODE_NOT_SUPPORTED |
MAPI_EMBED_RATE_PER_MINUTE |
int |
300 |
Public embed views per minute and embed link (429 beyond) |
MAPI_EMBED_IP_RATE_PER_MINUTE |
int |
120 |
Public embed requests per minute and client address, valid or not |
MAPI_EMBED_CACHE_S |
float |
60.0 |
How long an API instance reuses an embed's resolved plot (revocation and expiry are still checked on every request; responses are never cached by browsers or shared caches, so a revoked link stops rendering on the next load) |
MAPI_EMBED_IMAGE_URL_TTL_S |
int |
900 |
Lifetime of the PNG fallback's signed URL |
MAPI_LOG_JSON |
bool |
True |
Emit JSON logs (structlog); off gives a human console renderer |
MAPI_TRUSTED_PROXIES |
list[str] \| None |
unset | Addresses or CIDR ranges of the proxies in front of the API (load balancer, Cloud Run front end, the web container's /v1/ proxy), comma-separated. A request from one of them takes the right-most untrusted X-Forwarded-For hop as the client address (per-address limits, Sybil checks, audit). 'none': clients reach the API directly and X-Forwarded-For is ignored. Required in the multi-tenant profile |
MAPI_METRICS_ENABLED |
bool |
True |
Mount GET /metrics (Prometheus) |
MAPI_WORKER_METRICS_PORT |
int \| None |
unset | Serve GET /metrics of the worker (dispatcher, reconciler, finalizer, events) on this port; unset: the worker exposes no metrics |
MAPI_WORKER_METRICS_HOST |
str |
127.0.0.1 |
Bind address of the worker's metrics endpoint |
MAPI_METRICS_TOKEN |
str \| None |
unset | Bearer a scraper must present on GET /metrics; required in the multi-tenant profile (the series carry tenant ids and spend) |
MAPI_TRACING_ENABLED |
bool |
True |
Install the OpenTelemetry tracer provider |
MAPI_OTLP_ENDPOINT |
str \| None |
unset | OTLP/HTTP collector base URL for span export (also read from OTEL_EXPORTER_OTLP_ENDPOINT); unset means spans are created but not exported |
MAPI_WEBHOOK_ALLOW_HTTP |
bool |
False |
Development only: allow http:// webhook URLs, for the hosts in MAPI_WEBHOOK_DEV_ALLOWLIST only (refused in the multi-tenant profile) |
MAPI_WEBHOOK_DEV_ALLOWLIST |
list |
(computed) | Development only: webhook host names (comma-separated) exempt from the SSRF address and port checks, e.g. a receiver on localhost (refused in the multi-tenant profile) |
MAPI_WEBHOOK_CA_FILE |
Path \| None |
unset | Extra PEM CA bundle trusted for webhook receivers (a private CA); the system trust store is always used |
MAPI_WEBHOOK_POLL_INTERVAL_S |
float |
2.0 |
How often the worker looks for due webhook deliveries |
MAPI_WEBHOOK_MAX_ATTEMPTS |
int |
8 |
Tries per delivery (the retry schedule allows up to 8) |
MAPI_WEBHOOK_AUTO_DISABLE_AFTER |
int |
20 |
Disable a webhook after this many consecutive events failed |
MAPI_WEBHOOK_MAX_PER_PROJECT |
int |
25 |
|
MAPI_WEBHOOK_RETENTION_DAYS |
int |
30 |
Days webhook events and their delivery logs are kept |
MAPI_WORKFLOW_TICK_S |
float |
1.0 |
How often the orchestrator advances active workflow runs |
MAPI_WORKFLOW_MAX_ACTIVE_RUNS |
int |
20 |
Workflow runs a project may have going at once, sub-workflow runs included |
MAPI_WORKFLOW_MAX_SCHEDULES |
int |
20 |
Active cron schedules per project |
MAPI_WORKFLOW_MAX_TRIGGERS |
int |
20 |
Active dataset triggers per project |
MAPI_WORKFLOW_NODE_TIMEOUT_S |
int |
1800 |
Timeout of a node without its own timeout |
MAPI_WORKFLOW_MAX_JSON_OUTPUT_BYTES |
int |
1048576 |
Largest JSON output of a node |
MAPI_SIGNUP_ENABLED |
bool \| None |
unset | Self-service sign-up at POST /auth/signup (default: on in the dev and multi-tenant profiles when a mailer is configured, off single-tenant) |
MAPI_MAILER |
MailerKind \| None |
unset | How e-mail is sent: console, outbox, smtp or none (default: outbox in dev, console single-tenant, smtp multi-tenant) |
MAPI_MAIL_FROM |
str |
modeling-api <no-reply@mapi.localhost> |
From: header of the e-mails the server sends |
MAPI_OUTBOX_DIR |
Path \| None |
unset | Where the outbox mailer writes messages (default: data_dir/outbox) |
MAPI_SMTP_HOST |
str \| None |
unset | SMTP server (MAPI_MAILER=smtp) |
MAPI_SMTP_PORT |
int |
587 |
|
MAPI_SMTP_TLS |
starttls | ssl |
starttls |
starttls (port 587) or ssl (implicit TLS, port 465) |
MAPI_SMTP_USERNAME |
str \| None |
unset | |
MAPI_SMTP_PASSWORD |
str \| None |
unset | |
MAPI_SMTP_TIMEOUT_S |
float |
10.0 |
|
MAPI_SIGNUP_VERIFY_URL |
str \| None |
unset | Verification link template with {token} (default: |
MAPI_SIGNUP_BLOCKED_DOMAINS |
list |
(computed) | E-mail domains refused at sign-up besides the built-in disposable list (comma-separated; subdomains included) |
MAPI_SIGNUP_BLOCKLIST_PATH |
Path \| None |
unset | A file of disposable e-mail domains (one per line, # comments), e.g. a maintained public list, added to the built-in one |
MAPI_SIGNUP_PER_IP_PER_HOUR |
int |
5 |
Sign-ups accepted per client address and hour |
MAPI_SIGNUP_PER_DOMAIN_PER_DAY |
int |
25 |
Verified sign-ups per e-mail domain and day (shared free-mail domains are exempt) |
MAPI_SIGNUP_GLOBAL_PER_HOUR |
int |
500 |
Verified sign-ups per hour on the whole platform |
MAPI_SIGNUP_SHARED_EMAIL_DOMAINS |
list |
(computed) | Free-mail domains shared by unrelated people (comma-separated; replaces the built-in list): no per-domain sign-up cap and no domain-burst Sybil flag |
MAPI_VERIFY_PER_IP_PER_MINUTE |
int |
20 |
E-mail verification attempts per client address and minute |
MAPI_SIGNUP_SYBIL_WINDOW_DAYS |
int |
30 |
Look-back of the Sybil heuristics (same address, browser, e-mail) |
MAPI_SIGNUP_SYBIL_THRESHOLD |
int |
3 |
Workspaces created from one address, browser or e-mail within the window before a new one is flagged for review |
MAPI_SIGNUP_FLAGGED_SUSPEND |
bool |
False |
Suspend a flagged sign-up's tenant until an operator approves it (default: it works and waits in the review queue) |
MAPI_SIGNUP_RETENTION_DAYS |
int |
30 |
Unverified sign-ups (with their e-mail) are deleted after this |
MAPI_PUBLIC_CODE_UPLOAD_ENABLED |
bool |
False |
Cut line B: let identity-verified public tenants run uploaded code (off through M4: public tenants run built-ins and declarative specs only) |
MAPI_TENANT_DELETION_GRACE_S |
float |
604800.0 |
Grace period of a tenant deletion: the tenant is suspended and may cancel |
MAPI_DELETION_REAUTH_MAX_AGE_S |
float \| None |
unset | Also require a sign-in at most this old to request a deletion (re-auth); unset: the confirmation phrase alone |
MAPI_DELETION_RUN_STOP_TIMEOUT_S |
float |
3600.0 |
How long a purge waits for the tenant's cancelled runs to stop |
MAPI_DELETION_MAX_ATTEMPTS |
int |
8 |
Automatic retries of a failed purge before operators must retry |
MAPI_BUNDLE_RETENTION_DAYS |
int |
30 |
The bundles bucket's retention period (Terraform bundle_retention_days): when a purge may retry objects a retention lock refused |
MAPI_LIFECYCLE_INTERVAL_S |
float |
15.0 |
Cadence of the deletion and export scheduler (orchestrator) |
MAPI_EXPORT_MAX_INLINE_BYTES |
int |
268435456 |
Dataset bytes a GDPR export archive includes; larger exports list the remaining objects (with sizes and hashes) instead |
MAPI_EXPORT_RETENTION_S |
float |
604800.0 |
How long a GDPR export archive is kept |
MAPI_EXPORT_LINK_TTL_S |
float |
900.0 |
Lifetime of an export's signed download link |
MAPI_EXPORTS_PER_DAY |
int |
3 |
GDPR exports a tenant may request per day |
MAPI_EXPORT_MAX_METADATA_BYTES |
int |
268435456 |
Bytes of metadata rows (runs, logs, events, audit, ...) a GDPR export archive includes; the tables cut short are listed in its manifest |
MAPI_EXPORT_BUILD_TIMEOUT_S |
float |
3600.0 |
How long one GDPR export build may take; a build still marked running 5 minutes after that (its process died) is built again once, then failed |