Governance checks

Every check a governance policy can require, generated from mapi_spec.governance.GOVERNANCE_CHECKS (the catalog behind GET /v1/catalog/governance-checks, mapi governance checks and the web app's Check catalog). Each entry starts with its plain-language About card. Can you trust this model? explains the null tests in plain words and in statistics.

A policy may set a check's severity (blocking, warning, info), its own warn and fail limits (never looser than the order below allows) and its parameters. A check whose requirement does not hold for a model is reported as skipped with the reason; one that cannot be computed is error, which fails a blocking check.

Check	Title	Measures	Default limits	Needs
baseline.beats_naive	Better than simple benchmarks	relative_skill	pass at ≥ 0.05, warn down to 0, fail below	holdout, time_series
convergence	The fit is complete and stable	r_hat_max	pass at ≤ 1.01, warn up to 1.05, fail above	–
data.quality	The data is good enough	data_issue_count	pass at ≤ 0, warn up to 2, fail above	–
holdout.accuracy	Accurate on test data	holdout_error	pass at ≤ 0.15, warn up to 0.3, fail above	holdout
null.noise_regressors	No effect from false drivers	noise_effect_share	pass at ≤ 0.02, warn up to 0.05, fail above	–
null.permutation_target	Better than shuffled data	permutation_p_value	pass at ≤ 0.05, warn up to 0.1, fail above	holdout
null.placebo_time_shift	No effect after a time shift	placebo_effect_ratio	pass at ≤ 0.25, warn up to 0.5, fail above	time_series
plausibility.signs	Effects have the correct direction	wrong_sign_share	pass at ≤ 0.1, warn up to 0.25, fail above	–
recovery.simulated	Finds known values	recovery_coverage	pass within 0.1 of 0.9, warn within 0.25, fail further away	–
residuals.whiteness	No pattern in the errors	ljung_box_p_value	pass at ≥ 0.05, warn down to 0.01, fail below	residuals, time_series
sensitivity.prior	The results come from the data	prior_sensitivity	pass at ≤ 0.2, warn up to 0.5, fail above	bayesian
stability.rolling_params	Estimates are stable over time	param_variation	pass at ≤ 0.25, warn up to 0.5, fail above	time_series

baseline.beats_naive - Better than simple benchmarks

Compares the error of the model on the test periods with very simple forecasts, for example 'same as last week' or 'same as last year'. The model must be better.

Difficulty: beginner. Typical runtime: seconds.

Use it when: Each forecast model.

Avoid it when: Models that do not forecast over time.

It needs: The test periods of the model and the history before them.

It assumes: The check scores the benchmarks on the same periods.

You get: The skill against the best benchmark (1 - model error / benchmark error).

Example: The error of the model is 30% lower than 'same as last year': skill 0.30, pass.

Related: evaluate.baseline, holdout.accuracy.

Expert detail

Relative skill 1 - L(model)/min_b L(b) on the holdout or backtest folds; MASE-style scaling by the in-sample seasonal-naive error is reported as well. References: Hyndman & Koehler (2006), Another look at measures of forecast accuracy.

Category	baseline
Metric	relative_skill
Default limits	pass at ≥ 0.05, warn down to 0, fail below
Default severity	blocking
Applies to	every model family
Skipped when	the model has no test period (add a data split and fit the model again); the data of the model has no time dimension
Version	1.0.0

Parameters (params in a policy)

Field	Type	Default	Description
methods	list of naive or seasonal_naive or mean or drift	["naive", "seasonal_naive"]
metric	string	mae

convergence - The fit is complete and stable

Makes sure that the fit procedure found a stable answer. If you fit the model again, you get the same result.

Difficulty: beginner. Typical runtime: seconds.

Use it when: Every model.

Avoid it when: Models that you fit outside the platform, without diagnostics.

It needs: The diagnostics of the fit.

It assumes: The fit recorded its diagnostics.

You get: R-hat, effective draws and divergences (Bayesian) or the optimiser status.

Example: All chains agree (R-hat 1.003): pass.

Related: recovery.simulated.

Expert detail

Bayesian: max split R-hat (rank-normalised), with ESS and divergences as evidence; frequentist: optimiser convergence flag (value 1.0 when converged, inf otherwise).

Category	convergence
Metric	r_hat_max
Default limits	pass at ≤ 1.01, warn up to 1.05, fail above
Default severity	blocking
Applies to	every model family
Skipped when	never
Version	1.0.0

data.quality - The data is good enough

Makes sure that the data is good before you trust the model. The data must have few gaps, enough history, no constant columns and no impossible values.

Difficulty: beginner. Typical runtime: seconds.

Use it when: Every model.

Avoid it when: Synthetic data that you made yourself.

It needs: The dataset that the model fit used.

It assumes: The limits are correct for your data.

You get: The data problems that the check found, each in plain words.

Example: No gaps, 3 years of history, each driver changes: pass.

Related: convergence.

Category	data
Metric	data_issue_count
Default limits	pass at ≤ 0, warn up to 2, fail above
Default severity	blocking
Applies to	every model family
Skipped when	never
Version	1.0.0

Parameters (params in a policy)

Field	Type	Default	Description
max_missing	number	0.05	≥ 0, ≤ 1
min_history_periods	integer	52	≥ 1

holdout.accuracy - Accurate on test data

Makes sure that the error of the model on periods that the fit did not use stays below the limit of your organisation.

Difficulty: beginner. Typical runtime: seconds.

Use it when: Each model that you use for forecasts or what-ifs.

Avoid it when: Models without a test period.

It needs: A test period or a backtest.

It assumes: The test periods are similar to future use.

You get: The selected error metric on the test data.

Example: MAPE 9% against a limit of 15%: pass.

Related: baseline.beats_naive, evaluate.metrics.

Category	holdout
Metric	holdout_error
Default limits	pass at ≤ 0.15, warn up to 0.3, fail above
Default severity	blocking
Applies to	every model family
Skipped when	the model has no test period (add a data split and fit the model again)
Version	1.0.0

Parameters (params in a policy)

Field	Type	Default	Description
metric	mape or smape or mase or rmse or mae	mape

null.noise_regressors - No effect from false drivers

Adds some false drivers that are only noise and fits the model again. A model that you can trust gives them almost no effect.

Difficulty: intermediate. Typical runtime: long.

Use it when: Marketing mix models and other models that divide a result between drivers.

Avoid it when: Forecast models that have no drivers.

It needs: The data and the settings of the model.

It assumes: The false drivers look like real drivers, but they have no signal.

You get: The largest share of the effect of the real drivers that goes to a false driver; If a false driver is clearly above the limit.

Example: Three false channels get 0.4%, 0.1% and 0.9% of the media effect: pass.

Related: null.permutation_target, plausibility.signs.

Expert detail

Placebo regressors drawn as Gaussian noise or as a driver shuffled in blocks, with the model's default prior (never a real channel's prior or calibration); the statistic is the largest placebo share of the real drivers' total effect (by ablation through the score path). Bayesian: flagged when P(share > warn limit) >= 0.9 over the posterior (practical equivalence, valid for sign-constrained effects); frequentist: placebo p-values with Holm's correction.

Category	null_test
Metric	noise_effect_share
Default limits	pass at ≤ 0.02, warn up to 0.05, fail above
Default severity	blocking
Applies to	every model family
Skipped when	never
Version	1.0.0

Parameters (params in a policy)

Field	Type	Default	Description
n_noise	integer	3	≥ 1, ≤ 10
kind	gaussian or shuffled_driver	shuffled_driver
seed	integer	0	≥ 0

null.permutation_target - Better than shuffled data

Fits the model again on copies of the data where the drivers have a random time order. A real model must be clearly better on the true data than on the shuffled copies.

Difficulty: intermediate. Typical runtime: long.

Use it when: Each model whose accuracy you use to make decisions.

Avoid it when: Models without a test period (there is nothing to compare).

It needs: The data and the settings of the model, and a test period.

It assumes: A random order of the drivers removes each real link between the drivers and the result; The trend, the season and the result stay as they are; For a model without drivers, the check shuffles the ups and downs of the result.

You get: The fraction of shuffled fits that were as good as the real fit (a p-value); The distribution of the shuffled scores.

Example: The real model forecasts better than all 19 shuffled fits: p = 0.05, pass.

Related: null.noise_regressors, baseline.beats_naive.

Expert detail

Permutation (randomisation) test on a scale-free statistic: holdout skill 1 - L(model) / L(null forecast) on each dataset (the null forecast is the last training value of an integrated series, else its mean). Null: models with drivers keep the result and get their drivers block-permuted jointly (so trend, season and controls cannot pass on their own; a model that ignores its drivers is tested as one without); models without drivers get the d-th differences of the result permuted i.i.d. and re-integrated from the original start (d from the ARIMA order or ADF/KPSS), which keeps the integration order and the increments' distribution but destroys predictable dynamics. Without a time order the result is permuted across rows and the statistic is the holdout error. p = (1 + #{perm >= real}) / (K + 1). References: Ojala & Garriga (2010), Permutation tests for studying classifier performance.

Category	null_test
Metric	permutation_p_value
Default limits	pass at ≤ 0.05, warn up to 0.1, fail above
Default severity	blocking
Applies to	every model family
Skipped when	the model has no test period (add a data split and fit the model again)
Version	1.0.0

Parameters (params in a policy)

Field	Type	Default	Description
n_permutations	integer	20	≥ 5, ≤ 200
metric	rmse or mae or mape or smape or crps	rmse
block	integer	–	Periods of the drivers shuffled together (default: a twentieth of the series) (≥ 1, ≤ 520)

null.placebo_time_shift - No effect after a time shift

Moves the drivers some months away from their real dates and fits the model again. The effects must become much smaller, or else the model finds time patterns, not real effects.

Difficulty: expert. Typical runtime: long.

Use it when: Models that divide results between drivers over time (marketing mix, regressions).

Avoid it when: Very short series (after the shift, the overlap is too small).

It needs: The data and the settings of the model over time.

It assumes: A real effect has a connection to the date of the driver.

You get: The placebo effect as a share of the real effect, for each shift.

Example: A shift of the spend by half a year keeps only 8% of the effect: pass.

Related: null.noise_regressors.

Expert detail

Time-shift placebo (leads/lags beyond any plausible carry-over); ratio of the placebo total effect to the real one, maximised over shifts.

Category	null_test
Metric	placebo_effect_ratio
Default limits	pass at ≤ 0.25, warn up to 0.5, fail above
Default severity	blocking
Applies to	every model family
Skipped when	the data of the model has no time dimension
Version	1.0.0

Parameters (params in a policy)

Field	Type	Default	Description
shifts	list of integer	[-26, 26]	min items 1, max items 6

plausibility.signs - Effects have the correct direction

Makes sure that each effect has the direction that business sense expects. For example, advertising must not decrease sales, and higher prices must not increase demand.

Difficulty: beginner. Typical runtime: seconds.

Use it when: Each model with effects that you can explain.

Avoid it when: Models that only forecast.

It needs: The estimates of the model and the expected directions.

It assumes: The expected directions are correct.

You get: The share of the estimate (or of the uncertainty) on the wrong side, for each effect.

Example: The effect of radio is negative with a probability of 40%: fail.

Related: null.noise_regressors, test.vif.

Category	plausibility
Metric	wrong_sign_share
Default limits	pass at ≤ 0.1, warn up to 0.25, fail above
Default severity	blocking
Applies to	every model family
Skipped when	never
Version	1.0.0

Parameters (params in a policy)

Field	Type	Default	Description
expected	map of positive or negative	–	Parameter -> expected sign (default: media positive)

recovery.simulated - Finds known values

Simulates data from the model with known true values and fits the model again. Then it makes sure that the 90% ranges of the model contain the true values about 90% of the time.

Difficulty: expert. Typical runtime: long.

Use it when: New models, or models with many changes; Before you trust a complex Bayesian model.

Avoid it when: Models that cannot simulate data.

It needs: The settings of the model and the shape of the data.

It assumes: The simulated data is similar to the real data.

You get: How often the 90% ranges contained the true values.

Example: The true values were in the 90% ranges 88% of the time: pass.

Related: sensitivity.prior, convergence.

Expert detail

Simulation-based calibration lite: N simulate-fit cycles; coverage of central 90% intervals and rank uniformity per parameter. References: Talts et al. (2018), Validating Bayesian inference algorithms with simulation-based calibration.

Category	recovery
Metric	recovery_coverage
Default limits	pass within 0.1 of 0.9, warn within 0.25, fail further away
Default severity	blocking
Applies to	every model family
Skipped when	never
Version	1.0.0

Parameters (params in a policy)

Field	Type	Default	Description
n_sims	integer	20	≥ 5, ≤ 100
params	list of string	–

residuals.whiteness - No pattern in the errors

Makes sure that the errors of the model look like random noise over time. If a pattern stays in the errors, the model does not use all of the information in the data.

Difficulty: intermediate. Typical runtime: seconds.

Use it when: Time-series and regression models.

Avoid it when: Models without residuals.

It needs: The in-sample errors of the model.

It assumes: The time steps are regular.

You get: The Ljung-Box p-value and the autocorrelations of the errors.

Example: Ljung-Box p = 0.38 at 10 lags: pass.

Related: test.ljung_box, evaluate.residuals.

Expert detail

Ljung-Box Q on residuals (model_df adjusted for ARMA terms); the minimum p-value over series for panels.

Category	residuals
Metric	ljung_box_p_value
Default limits	pass at ≥ 0.05, warn down to 0.01, fail below
Default severity	blocking
Applies to	every model family
Skipped when	the model has no residuals; the data of the model has no time dimension
Version	1.0.0

Parameters (params in a policy)

Field	Type	Default	Description
lags	integer	10	≥ 1, ≤ 200

sensitivity.prior - The results come from the data

Fits a Bayesian model again with wider and narrower starting assumptions. The key answers must change very little, so that you know that they come from the data.

Difficulty: expert. Typical runtime: long.

Use it when: Bayesian models, mainly models with business priors.

Avoid it when: Models that are not Bayesian.

It needs: The data, the settings and the priors of the model.

It assumes: The scales that the check tries include all reasonable prior changes.

You get: The largest relative change of a key estimate.

Example: Prior widths that are two times larger move the return of TV by 6%: pass.

Related: plausibility.signs, recovery.simulated.

Expert detail

Power-scaling or refit-based prior sensitivity: relative change of posterior means of key quantities under scaled prior widths. References: Kallioinen et al. (2023), Detecting and diagnosing prior and likelihood sensitivity with power-scaling.

Category	sensitivity
Metric	prior_sensitivity
Default limits	pass at ≤ 0.2, warn up to 0.5, fail above
Default severity	blocking
Applies to	every model family
Skipped when	only Bayesian models have priors that the check can change
Version	1.0.0

Parameters (params in a policy)

Field	Type	Default	Description
scales	list of number	[0.5, 2.0]	Prior widths tried

stability.rolling_params - Estimates are stable over time

Fits the model again on rolling windows of history. Then it makes sure that the key estimates do not change much from one window to the next.

Difficulty: expert. Typical runtime: long.

Use it when: Models with estimates that you use for decisions (effects, returns).

Avoid it when: Short histories (the windows are then too small).

It needs: The data over time and the settings of the model.

It assumes: The true effects change slowly.

You get: How much each key estimate changes between windows.

Example: The return of TV changes by 12% between windows: pass.

Related: test.cusum.

Expert detail

Coefficient of variation (or max relative deviation from the full-sample estimate) of each watched parameter across rolling-window refits; maximum over parameters.

Category	stability
Metric	param_variation
Default limits	pass at ≤ 0.25, warn up to 0.5, fail above
Default severity	blocking
Applies to	every model family
Skipped when	the data of the model has no time dimension
Version	1.0.0

Parameters (params in a policy)

Field	Type	Default	Description
window	integer	–	Periods per window (≥ 8)
step	integer	4	≥ 1
params	list of string	–	Parameters to watch (default: the key effects)

The default policies

Seeded as tenant-wide policies (version 1) the first time a tenant uses governance. They decide what a validation checks per model family; they do not require approval and do not gate promotion until an admin turns that on.

Policy	Families	Checks (severity, parameters)
Marketing mix models (platform default)	mmm	data.quality (blocking) convergence (blocking) holdout.accuracy (blocking, metric=mape) null.permutation_target (blocking, n_permutations=20) null.noise_regressors (blocking, n_noise=3) plausibility.signs (blocking) baseline.beats_naive (warning) null.placebo_time_shift (warning) sensitivity.prior (warning) residuals.whiteness (info)
Forecasting models (platform default)	ts, forecast, ts_multivariate, regime	data.quality (blocking) convergence (blocking) baseline.beats_naive (blocking) holdout.accuracy (blocking, metric=mape) residuals.whiteness (warning) null.permutation_target (warning, n_permutations=20) stability.rolling_params (info)
Regression and machine-learning models (platform default)	tabular, choice, sklearn	data.quality (blocking) holdout.accuracy (blocking, metric=mape) null.permutation_target (blocking, n_permutations=20) null.noise_regressors (warning, n_noise=3) convergence (info)