Workflow node reference¶
Every node a workflow can use, grouped by kind. Each entry starts with the plain description the web app shows on the node's About card. See the workflows guide for how to wire nodes together.
Statistical test (test)¶
Tests a property of the data or of the errors of a model. It gives a yes/no verdict with its evidence.
test.acf_pacf — How far back does the past matter? (ACF/PACF)¶
Measures how strongly each past period is related to the present. It also suggests the number of past values for a forecast model.
Runs in a sandboxed workflow_node run. Difficulty: intermediate. Typical runtime: seconds.
Use it when: You must select the orders of an ARIMA model; You want a quick look at the seasonality.
Avoid it when: The series still wanders (apply a difference first).
Example: The partial correlations stop after lag 2 and the plain correlations decrease slowly. Try an AR(2).
Inputs: data (dataset).
Outputs: result (test_result).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
alpha |
number | 0.05 |
The significance level |
var |
string | The variable to test (default: the dataset's target) | |
nlags |
integer | 24 |
The number of lags to calculate |
max_order |
integer | 5 |
The largest order to suggest |
Result fields (nodes.<id>.<field> in conditions): statistic (number), p_value (number), alpha (number), reject (boolean), decision (string), verdict_plain (string), critical_values (object), n_obs (integer), lags (integer), details (object), per_series (array), reject_share (number), significant_acf_lags (array), significant_pacf_lags (array), suggested_p (integer), suggested_q (integer), acf (array), pacf (array).
Expert detail
Sample ACF/PACF (statsmodels acf, pacf with Yule-Walker-MLE) with Bartlett / 1.96/sqrt(n) bands; suggested p = last significant PACF lag before the cut-off, q from the ACF.
test.adf — Does the series wander? (ADF)¶
Tests whether a series returns to a stable level or wanders away. The result tells you whether to apply a difference before you forecast.
Runs in a sandboxed workflow_node run. Difficulty: intermediate. Typical runtime: seconds.
Use it when: Before you fit a forecast model that needs a stable series; With the KPSS test, which examines the same question from the opposite side.
Avoid it when: The series is very short (less than about 30 periods); The series has one clear break (use Zivot-Andrews).
Example: Weekly sales give p = 0.57. The series wanders, so apply one difference.
Inputs: data (dataset).
Outputs: result (test_result).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
alpha |
number | 0.05 |
The significance level |
var |
string | The variable to test (default: the dataset's target) | |
regression |
c \ |
ct \ |
ctt \ |
autolag |
AIC \ |
BIC \ |
t-stat \ |
maxlag |
integer | The largest lag to try (default: Schwert's rule) | |
per_series |
boolean | true |
For panels, test every series (geo, product) separately |
Result fields (nodes.<id>.<field> in conditions): statistic (number), p_value (number), alpha (number), reject (boolean), decision (string), verdict_plain (string), critical_values (object), n_obs (integer), lags (integer), details (object), per_series (array), reject_share (number).
Expert detail
Augmented Dickey-Fuller (statsmodels adfuller); H0: unit root. Lag length by AIC/BIC/t-stat; regression 'c', 'ct', 'ctt' or 'n'; MacKinnon p-values.
test.arch_lm — Do volatile periods cluster? (ARCH-LM)¶
Tests whether calm periods and turbulent periods occur in groups. If they do, the ranges must become wider after a shock.
Runs in a sandboxed workflow_node run. Difficulty: expert. Typical runtime: instant.
Use it when: Financial or demand series with bursts of volatility.
Avoid it when: Series without visible bursts.
Example: The errors after promotions are larger for several weeks (p = 0.02).
Inputs: data (dataset, optional), model (model_version, optional).
Outputs: result (test_result).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
alpha |
number | 0.05 |
The significance level |
var |
string | The series to test (default: the residuals of the model input) | |
nlags |
integer | 4 |
Lags |
Result fields (nodes.<id>.<field> in conditions): statistic (number), p_value (number), alpha (number), reject (boolean), decision (string), verdict_plain (string), critical_values (object), n_obs (integer), lags (integer), details (object), per_series (array), reject_share (number).
Expert detail
Engle's ARCH LM test (statsmodels het_arch); H0: no ARCH effects.
test.breusch_pagan — Even-sized errors? (Breusch-Pagan)¶
Tests whether the errors of a model are larger at some times or for some inputs. Then the ranges of the model are too narrow in some places.
Runs in a sandboxed workflow_node run. Difficulty: expert. Typical runtime: instant.
Use it when: You want to test the errors of a regression model.
Avoid it when: Models without inputs.
Example: The errors increase with spend (p = 0.003). The ranges are too narrow for weeks with a large spend.
Inputs: model (model_version).
Outputs: result (test_result).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
alpha |
number | 0.05 |
The significance level |
robust |
boolean | true |
Use the studentised version of Koenker |
Result fields (nodes.<id>.<field> in conditions): statistic (number), p_value (number), alpha (number), reject (boolean), decision (string), verdict_plain (string), critical_values (object), n_obs (integer), lags (integer), details (object), per_series (array), reject_share (number).
Expert detail
Breusch-Pagan LM test (statsmodels het_breuschpagan); H0: homoskedasticity.
test.chow — Did it change on this date? (Chow)¶
Tests whether the relationship between a result and its drivers is different before and after a date that you select.
Runs in a sandboxed workflow_node run. Difficulty: intermediate. Typical runtime: instant.
Use it when: You know a date when something changed (a relaunch, a new competitor).
Avoid it when: You do not know the date (use CUSUM).
Example: The price effect is different after the 2023 relaunch (p = 0.01).
Inputs: data (dataset).
Outputs: result (test_result).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
alpha |
number | 0.05 |
The significance level |
y |
string | required | The result |
x |
array | required | The drivers |
break_at |
string or integer | required | The date or position of the break |
Result fields (nodes.<id>.<field> in conditions): statistic (number), p_value (number), alpha (number), reject (boolean), decision (string), verdict_plain (string), critical_values (object), n_obs (integer), lags (integer), details (object), per_series (array), reject_share (number).
Expert detail
Chow F test comparing pooled and split OLS fits at the break point.
test.cusum — Did the relationship change? (CUSUM)¶
Tests whether the relationship between the result and the drivers of a model stayed the same over time. You do not have to guess when it changed.
Runs in a sandboxed workflow_node run. Difficulty: expert. Typical runtime: instant.
Use it when: You think that something changed, but you do not know when.
Avoid it when: You know the date (use the Chow test).
Example: The cumulative errors go across the bound in 2022. The relationship changed.
Inputs: model (model_version).
Outputs: result (test_result).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
alpha |
number | 0.05 |
The significance level |
Result fields (nodes.<id>.<field> in conditions): statistic (number), p_value (number), alpha (number), reject (boolean), decision (string), verdict_plain (string), critical_values (object), n_obs (integer), lags (integer), details (object), per_series (array), reject_share (number), break_detected (boolean), break_at (string).
Expert detail
OLS-residual CUSUM test (statsmodels breaks_cusumolsresid); H0: parameter stability.
test.engle_granger — Are two series tied together? (Engle-Granger)¶
Tests whether two wandering series stay tied together in the long run, so that the gap between them closes again.
Runs in a sandboxed workflow_node run. Difficulty: expert. Typical runtime: seconds.
Use it when: Before you make an error-correction model of one result and one driver.
Avoid it when: More than two series (use Johansen).
Example: Your price and the price of a competitor are tied (p = 0.02).
Inputs: data (dataset).
Outputs: result (test_result).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
alpha |
number | 0.05 |
The significance level |
y |
string | required | The first series |
x |
string | required | The second series |
trend |
c \ |
ct \ |
ctt \ |
Result fields (nodes.<id>.<field> in conditions): statistic (number), p_value (number), alpha (number), reject (boolean), decision (string), verdict_plain (string), critical_values (object), n_obs (integer), lags (integer), details (object), per_series (array), reject_share (number).
Expert detail
Engle-Granger two-step test (statsmodels coint); H0: no cointegration.
test.granger — Does one series help predict another? (Granger)¶
Tests whether the past values of a driver help to predict a result better than the past values of the result alone.
Runs in a sandboxed workflow_node run. Difficulty: intermediate. Typical runtime: seconds.
Use it when: You must decide which series to include in a VAR; You want to make sure that a leading indicator really leads.
Avoid it when: You want to prove cause and effect (the test shows only help with the prediction).
Example: Search interest helps to predict sales two weeks later (p = 0.004).
Inputs: data (dataset).
Outputs: result (test_result).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
alpha |
number | 0.05 |
The significance level |
y |
string | required | The result |
x |
string | required | The candidate driver |
maxlag |
integer | 4 |
The largest lag to test |
lag |
integer | Test only this lag (no correction is necessary) |
Result fields (nodes.<id>.<field> in conditions): statistic (number), p_value (number), alpha (number), reject (boolean), decision (string), verdict_plain (string), critical_values (object), n_obs (integer), lags (integer), details (object), per_series (array), reject_share (number), min_p_value (number), best_lag (integer), p_values (object), correction (string).
Expert detail
Granger causality F-tests for lags 1..maxlag (statsmodels grangercausalitytests, ssr F test); H0: the driver's lags have zero coefficients. The verdict uses the smallest p-value Holm-corrected for the maxlag lags tried (an uncorrected minimum about doubles the false-positive rate), or the F test at one prespecified lag; the per-lag p-values are descriptive.
test.jarque_bera — Bell-shaped errors? (Jarque-Bera)¶
Tests whether values, usually the errors of a model, follow a bell curve. If not, the ranges that use this assumption can be wrong.
Runs in a sandboxed workflow_node run. Difficulty: intermediate. Typical runtime: instant.
Use it when: You want to test the errors of a model before you trust its ranges.
Avoid it when: Small samples (the test is not reliable with less than about 50 values).
Example: The errors are skewed (p < 0.001). A small number of misses are very large.
Inputs: data (dataset, optional), model (model_version, optional).
Outputs: result (test_result).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
alpha |
number | 0.05 |
The significance level |
var |
string | The series to test (default: the residuals of the model input) |
Result fields (nodes.<id>.<field> in conditions): statistic (number), p_value (number), alpha (number), reject (boolean), decision (string), verdict_plain (string), critical_values (object), n_obs (integer), lags (integer), details (object), per_series (array), reject_share (number), skew (number), kurtosis (number).
Expert detail
Jarque-Bera on skewness and kurtosis (statsmodels jarque_bera); H0: normality.
test.johansen — Are the series tied together? (Johansen)¶
Tests whether several wandering series stay tied together in the long run. It also finds the number of these ties.
Runs in a sandboxed workflow_node run. Difficulty: expert. Typical runtime: seconds.
Use it when: Before you make one model of several series that have a trend.
Avoid it when: The series do not wander (they need no difference); Only one series.
Example: The prices of two close competitors share one long-run tie (rank 1).
Inputs: data (dataset).
Outputs: result (test_result).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
alpha |
0.1 \ |
0.05 \ |
0.01 |
vars |
array | The series: variable names, or labels along the series dimension of the data (if empty, all the series of the target) | |
det_order |
integer | 0 |
Deterministic terms: -1 none, 0 constant, 1 trend |
k_ar_diff |
integer | 1 |
Lagged differences |
method |
trace \ |
maxeig |
"trace" |
Result fields (nodes.<id>.<field> in conditions): statistic (number), p_value (number), alpha (number), reject (boolean), decision (string), verdict_plain (string), critical_values (object), n_obs (integer), lags (integer), details (object), per_series (array), reject_share (number), rank (integer), trace_stats (array), max_eig_stats (array).
Expert detail
Johansen trace and maximum-eigenvalue tests (statsmodels coint_johansen); rank from sequential tests at the critical values for the chosen level.
test.kpss — Is the series stable? (KPSS)¶
Tests whether a series stays near a stable level or trend. It is the opposite of the ADF test, so the two tests together give a better answer.
Runs in a sandboxed workflow_node run. Difficulty: intermediate. Typical runtime: seconds.
Use it when: With ADF, to confirm whether the series needs a difference.
Avoid it when: The series is very short.
Example: KPSS rejects stability (p = 0.01) and ADF cannot reject a wandering series. Apply a difference.
Inputs: data (dataset).
Outputs: result (test_result).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
alpha |
number | 0.05 |
The significance level |
var |
string | The variable to test (default: the dataset's target) | |
regression |
c \ |
ct |
"c" |
nlags |
auto \ |
legacy |
"auto" |
per_series |
boolean | true |
For panels, test every series (geo, product) separately |
Result fields (nodes.<id>.<field> in conditions): statistic (number), p_value (number), alpha (number), reject (boolean), decision (string), verdict_plain (string), critical_values (object), n_obs (integer), lags (integer), details (object), per_series (array), reject_share (number).
Expert detail
KPSS (statsmodels kpss); H0: level (or trend) stationarity; p-values are interpolated within [0.01, 0.1] and clipped outside.
test.ljung_box — Is there a pattern left? (Ljung-Box)¶
Tests whether the errors of a model are still related to their own past. If they are, the model does not find all of the structure.
Runs in a sandboxed workflow_node run. Difficulty: intermediate. Typical runtime: seconds.
Use it when: On the errors of a forecast model, after you fit it.
Avoid it when: On the raw series (use ACF/PACF to look for structure).
Example: The errors of an ARIMA model show no pattern up to lag 10 (p = 0.42). This is good.
Inputs: data (dataset, optional), model (model_version, optional).
Outputs: result (test_result).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
alpha |
number | 0.05 |
The significance level |
var |
string | The series to test (default: the residuals of the model input) | |
lags |
integer | 10 |
The number of lags to test |
model_df |
integer | The number of fitted ARMA terms to subtract (default: from the model) |
Result fields (nodes.<id>.<field> in conditions): statistic (number), p_value (number), alpha (number), reject (boolean), decision (string), verdict_plain (string), critical_values (object), n_obs (integer), lags (integer), details (object), per_series (array), reject_share (number).
Expert detail
Ljung-Box Q (statsmodels acorr_ljungbox); H0: no autocorrelation up to the lag; model_df adjusts the degrees of freedom for fitted ARMA terms.
test.pp — Does the series wander? (Phillips-Perron)¶
Tests whether a series wanders, as ADF does. It gives better results when the errors are uneven or related over time.
Runs in a sandboxed workflow_node run. Difficulty: expert. Typical runtime: seconds.
Use it when: You want a second test after ADF on a noisy series.
Avoid it when: The series is very short.
Example: PP and ADF agree that monthly demand wanders.
Inputs: data (dataset).
Outputs: result (test_result).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
alpha |
number | 0.05 |
The significance level |
var |
string | The variable to test (default: the dataset's target) | |
trend |
n \ |
c \ |
ct |
lags |
integer | Newey-West lags (default: automatic) | |
per_series |
boolean | true |
For panels, test every series (geo, product) separately |
Result fields (nodes.<id>.<field> in conditions): statistic (number), p_value (number), alpha (number), reject (boolean), decision (string), verdict_plain (string), critical_values (object), n_obs (integer), lags (integer), details (object), per_series (array), reject_share (number).
Expert detail
Phillips-Perron Z-tau test (arch.unitroot.PhillipsPerron or an equivalent Newey-West corrected regression); H0: unit root.
test.seasonal_strength — How strong is the season?¶
Measures how much of the changes in a series come from a seasonal pattern that repeats. The scale is from 0 (none) to 1 (all).
Runs in a sandboxed workflow_node run. Difficulty: beginner. Typical runtime: seconds.
Use it when: You must decide whether to model the seasonality or to apply a seasonal difference.
Avoid it when: Less than two full seasons of data.
Example: Weekly ice-cream sales have a seasonal strength of 0.85. Model the season.
Inputs: data (dataset).
Outputs: result (test_result).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
alpha |
number | 0.05 |
The significance level |
var |
string | The variable to test (default: the dataset's target) | |
period |
integer | The season length in periods (default: from the frequency) | |
threshold |
number | 0.64 |
The strength that the test counts as seasonal |
Result fields (nodes.<id>.<field> in conditions): statistic (number), p_value (number), alpha (number), reject (boolean), decision (string), verdict_plain (string), critical_values (object), n_obs (integer), lags (integer), details (object), per_series (array), reject_share (number), strength (number), trend_strength (number), period (integer), seasonal (boolean).
Expert detail
STL decomposition; F_s = max(0, 1 - Var(R) / Var(S + R)) (Wang, Smith & Hyndman); 'seasonal' when F_s >= threshold.
test.vif — Are drivers too alike? (VIF)¶
Tests whether some drivers change almost together. Then it is difficult to find their separate effects.
Runs in a sandboxed workflow_node run. Difficulty: intermediate. Typical runtime: instant.
Use it when: Before a regression or marketing mix model with many drivers.
Avoid it when: You only predict and the separate effects are not important.
Example: TV spend and radio spend have a VIF of 14. You cannot reliably separate their effects.
Inputs: data (dataset).
Outputs: result (test_result).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
alpha |
number | 0.05 |
The significance level |
vars |
array | required | The drivers |
limit |
number | 10 |
The VIF limit for a driver |
Result fields (nodes.<id>.<field> in conditions): statistic (number), p_value (number), alpha (number), reject (boolean), decision (string), verdict_plain (string), critical_values (object), n_obs (integer), lags (integer), details (object), per_series (array), reject_share (number), max_vif (number), table (object), high_vif_vars (array).
Expert detail
Variance inflation factor 1/(1 - R_j^2) per regressor (statsmodels variance_inflation_factor) with a constant added.
test.white — Even-sized errors? (White)¶
Tests whether the errors of a model are larger in some places. It is a more general test than Breusch-Pagan.
Runs in a sandboxed workflow_node run. Difficulty: expert. Typical runtime: instant.
Use it when: You think that the size of the errors has a curved relation to the inputs.
Avoid it when: Models with many inputs and little data (the test uses all the combinations of the inputs).
Example: The White test also finds uneven errors (p = 0.01).
Inputs: model (model_version).
Outputs: result (test_result).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
alpha |
number | 0.05 |
The significance level |
Result fields (nodes.<id>.<field> in conditions): statistic (number), p_value (number), alpha (number), reject (boolean), decision (string), verdict_plain (string), critical_values (object), n_obs (integer), lags (integer), details (object), per_series (array), reject_share (number).
Expert detail
White's test (statsmodels het_white); H0: homoskedasticity.
test.zivot_andrews — Wandering or one break? (Zivot-Andrews)¶
Tests whether a series that seems to wander is stable with one sudden jump. For example, a price change or a new product can cause a jump.
Runs in a sandboxed workflow_node run. Difficulty: expert. Typical runtime: seconds.
Use it when: The series has one clear shift.
Avoid it when: There are several breaks; The series is short.
Example: Sales jump after a relaunch in March 2023. If the test permits that break, the series is stable.
Inputs: data (dataset).
Outputs: result (test_result).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
alpha |
number | 0.05 |
The significance level |
var |
string | The variable to test (default: the dataset's target) | |
regression |
c \ |
t \ |
ct |
trim |
number | 0.15 |
The share at each end that cannot be a break date |
per_series |
boolean | true |
For panels, test every series (geo, product) separately |
Result fields (nodes.<id>.<field> in conditions): statistic (number), p_value (number), alpha (number), reject (boolean), decision (string), verdict_plain (string), critical_values (object), n_obs (integer), lags (integer), details (object), per_series (array), reject_share (number), break_index (integer), break_at (string).
Expert detail
Zivot-Andrews (statsmodels zivot_andrews); H0: unit root without a break; break in intercept, trend or both.
Data processing (transform)¶
Changes the data before later steps use it. For example, it calculates differences or logs.
transform.box_cox — Stabilise the spread¶
Bends a positive series so that its changes have about the same size at high and low levels. It finds the best bend from the data.
Runs in a sandboxed workflow_node run. Difficulty: intermediate. Typical runtime: instant.
Use it when: The spread increases with the level and a log is too strong or too weak.
Avoid it when: The values can be zero or negative.
Example: Lambda 0.3 makes the spread of monthly sales approximately constant.
Inputs: data (dataset).
Outputs: data (dataset).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
vars |
array | The variables (default: the target) | |
lmbda |
number | A fixed lambda (default: an estimate from the data) |
Result fields (nodes.<id>.<field> in conditions): vars (array), lmbda (number).
Expert detail
Box-Cox with lambda by maximum likelihood (scipy.stats.boxcox) or fixed.
transform.difference — Change from the previous period¶
Replaces each value with its change from the previous period. This changes a wandering series into a stable series.
Runs in a sandboxed workflow_node run. Difficulty: intermediate. Typical runtime: instant.
Use it when: A unit-root test shows that the series wanders.
Avoid it when: The series is already stable (a difference adds noise).
Example: Sales of 100, 104, 103 become changes of +4 and -1.
Inputs: data (dataset).
Outputs: data (dataset).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
vars |
array | The variables for the difference (default: the target) | |
order |
integer | 1 |
The number of differences |
dim |
string | "time" |
Result fields (nodes.<id>.<field> in conditions): vars (array).
Expert detail
First differences of order d along the time dim, per series; the last d levels are kept as state to integrate forecasts.
transform.log — Logarithm¶
Calculates the logarithm of positive values. This changes growth into steady steps, and you can compare large series with small series.
Runs in a sandboxed workflow_node run. Difficulty: beginner. Typical runtime: instant.
Use it when: The values increase by percentages; The spread increases with the level.
Avoid it when: The values can be zero or negative.
Example: Sales of 10, 100 and 1000 become 2.3, 4.6 and 6.9.
Inputs: data (dataset).
Outputs: data (dataset).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
vars |
array | The variables (default: the target) |
Result fields (nodes.<id>.<field> in conditions): vars (array).
transform.seasonal_difference — Change from the same period last season¶
Replaces each value with its change from the same period one season earlier. This removes a strong pattern that repeats.
Runs in a sandboxed workflow_node run. Difficulty: intermediate. Typical runtime: instant.
Use it when: Strong seasonality that you do not want a seasonal model to learn.
Avoid it when: Less than two seasons of history.
Example: The sales of week 53 minus the sales of week 1 one year earlier.
Inputs: data (dataset).
Outputs: data (dataset).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
vars |
array | The variables for the difference (default: the target) | |
period |
integer | required | The season length in periods |
order |
integer | 1 |
The number of differences |
dim |
string | "time" |
Result fields (nodes.<id>.<field> in conditions): vars (array).
transform.select — Keep some variables or dates¶
Keeps only the variables, markets or dates that you select. For example, you can test one series or a recent period.
Runs in a sandboxed workflow_node run. Difficulty: beginner. Typical runtime: instant.
Use it when: Later nodes must see only part of the data.
Avoid it when: The workflow needs all of the data.
Example: Keep only y and price for the market DE.
Inputs: data (dataset).
Outputs: data (dataset).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
vars |
array | The variables to keep | |
sel |
object | dim -> labels to keep | |
start |
string or integer | The first time label to keep | |
end |
string or integer | The last time label to keep |
Result fields (nodes.<id>.<field> in conditions): vars (array).
transform.steps — Preprocessing steps¶
Runs catalog preprocessing steps on the data. For example, the steps can make weekly totals, scale values or add calendar features.
Runs in a sandboxed workflow_node run. Difficulty: intermediate. Typical runtime: seconds.
Use it when: The data needs the same preparation as a model spec gives.
Avoid it when: A model node runs the same steps (put them in its pipeline).
Example: Weekly totals, then a log of spend, before the tests.
Inputs: data (dataset).
Outputs: data (dataset).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
steps |
array | required | Pipeline steps, as in a ModelSpec ({id, use, with}) |
Result fields (nodes.<id>.<field> in conditions): vars (array).
Decision (branch)¶
Uses earlier results to select the path of the workflow.
branch.if — Choose a path¶
Uses earlier results to decide the path of the workflow. For example: 'apply a difference only if the tests show that the series wanders'.
Runs in the orchestrator (no data; computed from earlier results). Difficulty: intermediate. Typical runtime: instant.
Use it when: A step must run only under a condition.
Avoid it when: All steps must always run.
Example: nodes.adf.p_value >= params.alpha or nodes.kpss.reject decides whether to apply a difference.
| Parameter | Type | Default | Meaning |
|---|---|---|---|
condition |
string | required | The condition in the workflow expression language |
explain_then |
string | What it means when the condition is true, in plain words for the run report (for example, 'The tests found that the series wanders, so the workflow applied a difference.') | |
explain_else |
string | What it means when the condition is false, in plain words |
Result fields (nodes.<id>.<field> in conditions): value (boolean), arm (string).
Expert detail
Conditions use the safe expression language (mapi_spec.expr): comparisons, and/or/not, arithmetic, a few functions; no code is executed.
Train/test plan (split)¶
Decides which periods or groups the model learns from, and which periods or groups test the model. It can do this many times.
split.blocked_kfold — Blocked cross-validation¶
Cuts the time line into some blocks and uses each block one time as the test. A buffer around each block stops data leaks from adjacent periods.
Runs in a sandboxed workflow_node run. Difficulty: expert. Typical runtime: instant.
Use it when: Models that explain, where the future is not the only important thing; Short series, where rolling backtests leave too little data.
Avoid it when: Forecasts only (a model that trains on the future is not realistic).
Example: Five blocks with a 4-week buffer on each side.
Inputs: data (dataset).
Outputs: plan (split_plan).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
after_step |
see schema | null |
Stage the folds are cut at (a step id or 'raw'), as data.split |
dim |
string | "time" |
|
k |
integer | required | |
gap |
integer | 0 |
Result fields (nodes.<id>.<field> in conditions): n_folds (integer), folds (array).
split.expanding_window — Rolling backtest (growing history)¶
Repeats the test at several points in time. Each time, the model trains on all the history up to that point, as in a real forecast.
Runs in a sandboxed workflow_node run. Difficulty: intermediate. Typical runtime: instant.
Use it when: You forecast regularly and want a realistic estimate of the error; You need a gap between the training data and the test data because the data comes late.
Avoid it when: Very short series.
Example: Six origins 4 weeks apart, a 1-week gap, horizons 1 to 8.
Inputs: data (dataset).
Outputs: plan (split_plan).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
after_step |
see schema | null |
Stage the folds are cut at (a step id or 'raw'), as data.split |
dim |
string | "time" |
|
horizons |
array | required | Forecast horizons scored per fold |
step |
see schema | null |
Periods between origins (default: max) |
gap |
integer | 0 |
Periods between training and test (not scored) |
n_folds |
see schema | null |
|
min_train |
integer | 1 |
Fewest training periods a fold may have |
anchor |
end \ |
start |
"end" |
Result fields (nodes.<id>.<field> in conditions): n_folds (integer), folds (array).
split.geo_holdout — Hold out markets¶
Keeps whole markets or stores out of the training data and tests the model on them. You can also test only their latest periods.
Runs in a sandboxed workflow_node run. Difficulty: intermediate. Typical runtime: instant.
Use it when: You will apply the model to new markets; You want to read the result of a geo experiment.
Avoid it when: You only predict for markets that the model knows.
Example: Hold out 2 random markets, 5 times with different draws.
Inputs: data (dataset).
Outputs: plan (split_plan).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
after_step |
see schema | null |
Stage the folds are cut at (a step id or 'raw'), as data.split |
dim |
string | "geo" |
|
groups |
see schema | null |
|
n |
see schema | null |
|
n_folds |
integer | 1 |
|
seed |
integer | 0 |
|
time_dim |
see schema | null |
|
test_last |
see schema | null |
|
include_pre_period |
boolean | false |
Result fields (nodes.<id>.<field> in conditions): n_folds (integer), folds (array).
split.holdout — One test period¶
Keeps one part of the data, usually the latest periods, to test the model.
Runs in a sandboxed workflow_node run. Difficulty: beginner. Typical runtime: instant.
Use it when: You want one quick, honest test.
Avoid it when: You want to know how stable the error is (use a rolling backtest).
Example: Hold out the last 8 weeks.
Inputs: data (dataset).
Outputs: plan (split_plan).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
after_step |
see schema | null |
Stage the folds are cut at (a step id or 'raw'), as data.split |
split |
object | required |
Result fields (nodes.<id>.<field> in conditions): n_folds (integer), folds (array).
split.nested — Tuning inside a backtest¶
Runs a second, inner backtest in each training window, so that you can tune settings without data from the outer test periods. The search.grid node tunes on the inner folds and scores on the outer folds.
Runs in a sandboxed workflow_node run. Difficulty: expert. Typical runtime: instant.
Use it when: You tune settings (orders, penalties) and want an honest final error.
Avoid it when: You do not tune settings.
Example: Outer: expanding window, 4 origins. Inner: expanding window, 3 origins.
Inputs: data (dataset).
Outputs: plan (split_plan).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
after_step |
see schema | null |
Stage the folds are cut at (a step id or 'raw'), as data.split |
outer |
object | required | |
inner |
object | required |
Result fields (nodes.<id>.<field> in conditions): n_folds (integer), folds (array).
split.purged_kfold — Purged cross-validation¶
Works like blocked cross-validation, but removes the periods immediately before and after each test block. Use it for results that cover several periods.
Runs in a sandboxed workflow_node run. Difficulty: expert. Typical runtime: instant.
Use it when: The results come from windows that overlap the test block.
Avoid it when: Plain forecasts.
Example: Five blocks, purge 2 periods before and embargo 4 after each test block.
Inputs: data (dataset).
Outputs: plan (split_plan).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
after_step |
see schema | null |
Stage the folds are cut at (a step id or 'raw'), as data.split |
dim |
string | "time" |
|
k |
integer | required | |
purge |
integer | 0 |
|
embargo |
integer | 0 |
Result fields (nodes.<id>.<field> in conditions): n_folds (integer), folds (array).
split.rolling_origin — Rolling backtest (fixed window)¶
Repeats the test at several points in time. Each time, the model trains on the same length of recent history, so you can see if the error stays the same.
Runs in a sandboxed workflow_node run. Difficulty: intermediate. Typical runtime: instant.
Use it when: Old history is possibly not relevant now; You want errors at several forecast horizons.
Avoid it when: Little history (each fold needs a full window).
Example: Train on 104 weeks and test 1, 4 and 8 weeks ahead. Move 4 weeks and do this 10 times.
Inputs: data (dataset).
Outputs: plan (split_plan).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
after_step |
see schema | null |
Stage the folds are cut at (a step id or 'raw'), as data.split |
dim |
string | "time" |
|
horizons |
array | required | Forecast horizons scored per fold |
step |
see schema | null |
Periods between origins (default: max) |
gap |
integer | 0 |
Periods between training and test (not scored) |
n_folds |
see schema | null |
|
min_train |
integer | 1 |
Fewest training periods a fold may have |
anchor |
end \ |
start |
"end" |
window |
integer | required | Training periods per fold |
Result fields (nodes.<id>.<field> in conditions): n_folds (integer), folds (array).
Search (search)¶
Tries many settings and selects the best setting by a score.
search.arma_order — Find the AR and MA orders¶
Tries a grid of past-value (AR) and past-error (MA) orders for a forecast model. It selects the order with the best balance of fit and simplicity.
Runs in a sandboxed workflow_node run. Difficulty: intermediate. Typical runtime: seconds.
Use it when: Before you fit an ARIMA or ARIMAX model.
Avoid it when: The series still wanders (apply a difference first); Very short series.
Example: Out of p = 0..4 and q = 0..4, ARMA(2,1) has the lowest AIC.
Inputs: data (dataset), split (split_plan, optional).
Outputs: table (table).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
var |
string | The series (default: the target) | |
max_p |
integer | 4 |
Largest AR order |
max_q |
integer | 4 |
Largest MA order |
d |
integer | 0 |
The differences in the model |
D |
integer | 0 |
The seasonal differences in the model |
period |
integer | Season length (with D = 1) | |
exog |
array | The drivers to include in each fit | |
criterion |
aic \ |
bic \ |
hqic \ |
trend |
n \ |
c \ |
t \ |
Result fields (nodes.<id>.<field> in conditions): best_p (integer), best_q (integer), best_order (array), best_score (number), n_failed (integer).
Expert detail
Grid search of SARIMAX(p,d,q)(0,D,0)s fits by AIC/BIC/HQIC/AICc (statsmodels arma_order_select_ic style), with the differencing of the model that will be fitted; failed fits are skipped and reported. With a split plan, only the periods before its first test window are searched, so a backtest on the plan stays out of sample.
search.coint_rank — How many long-run ties¶
Finds the number of long-run relationships that tie a set of wandering series together. The error-correction model needs this number.
Runs in a sandboxed workflow_node run. Difficulty: expert. Typical runtime: seconds.
Use it when: Before you fit a VECM.
Avoid it when: The series do not wander.
Example: Rank 1 for three related prices.
Inputs: data (dataset).
Outputs: table (table).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
vars |
array | The series: variable names, or labels along the series dimension of the data (if empty, all the series of the target) | |
det_order |
integer | 0 |
-1 none, 0 constant, 1 trend |
k_ar_diff |
integer | 1 |
Lagged differences |
method |
trace \ |
maxeig |
"trace" |
signif |
0.1 \ |
0.05 \ |
0.01 |
Result fields (nodes.<id>.<field> in conditions): rank (integer).
search.grid — Try settings on a backtest¶
Fits a model with each combination of the settings that you list. It scores each combination on a backtest and selects the best.
Runs in its own child run (a fit run for model.fit, else a workflow_node run). Difficulty: expert. Typical runtime: long.
Use it when: You want to tune some settings of a catalog model.
Avoid it when: Large grids on slow models (each point is a full fit for each fold).
Example: Try 3 saturation shapes x 2 carry-over lengths. Keep the lowest MAPE.
Inputs: data (dataset), split (split_plan).
Outputs: table (table), best (model_version).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
model |
string | required | Catalog model ref (id@range) |
settings |
object | The base settings | |
grid |
object | required | JSON pointer into settings -> values |
metric |
string | "rmse" |
|
direction |
min \ |
max |
"min" |
max_points |
integer | 50 |
The maximum number of grid points |
Result fields (nodes.<id>.<field> in conditions): best_settings (object), best_score (number), n_points (integer), nested (boolean), outer_scores (array), selected_per_fold (array).
Expert detail
Every grid point is backtested on the plan and the best score wins. With a split.nested plan this is nested cross-validation: each outer fold picks its point on its own inner folds (inside its training window), refits it on the outer training window and scores it on the outer test window; best_score pools those outer forecasts (the honest estimate of the tuned model, free of the selection bias of scoring the winner on the folds that chose it) and best_settings is the point the outer folds chose most often. From v0.11, the search run of the node logs each grid point. The search run is the first of the node's child runs; the node's run is the refit of the winner. A param such as trial.3.adstock/l_max gives a setting of point 3. The scores are metrics with the point number as the step. The search run also gets a metric such as best_rmse and the tag grid.best_point. The Metrics and Params tabs of its run page show them.
search.sarima_order — Find seasonal orders¶
Adds seasonal past values and errors to the order search, for series with a pattern that repeats.
Runs in a sandboxed workflow_node run. Difficulty: expert. Typical runtime: minutes.
Use it when: The series has a clear season that a difference alone does not remove.
Avoid it when: No seasonality (use the plain order search).
Example: SARIMA(1,0,1)(1,1,0)[52] wins on weekly sales.
Inputs: data (dataset).
Outputs: table (table).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
var |
string | The series (default: the target) | |
max_p |
integer | 2 |
Largest AR order |
max_q |
integer | 2 |
Largest MA order |
max_P |
integer | 1 |
Largest seasonal AR order |
max_Q |
integer | 1 |
Largest seasonal MA order |
d |
integer | 0 |
Differences |
D |
integer | 0 |
Seasonal differences |
period |
integer | required | Season length |
exog |
array | Drivers | |
criterion |
aic \ |
bic \ |
hqic \ |
Result fields (nodes.<id>.<field> in conditions): best_order (array), best_seasonal_order (array), best_score (number).
search.var_lag_order — How many lags for series together¶
Selects the number of past periods for a model of several related series.
Runs in a sandboxed workflow_node run. Difficulty: intermediate. Typical runtime: seconds.
Use it when: Before you fit a VAR or VECM.
Avoid it when: One series (use the AR/MA order search).
Example: AIC selects 3 lags and BIC selects 1.
Inputs: data (dataset).
Outputs: table (table).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
vars |
array | The series: variable names, or labels along the series dimension of the data (if empty, all the series of the target) | |
maxlags |
integer | 8 |
Largest lag |
criterion |
aic \ |
bic \ |
hqic \ |
Result fields (nodes.<id>.<field> in conditions): best_lag (integer), by_criterion (object).
Model (model)¶
Fits, backtests or forecasts with any model from the catalog.
model.backtest — Backtest a model¶
Fits a model again on each fold of a split plan and measures its forecast error at each horizon. This shows how reliable the model is.
Runs in its own child run (a fit run for model.fit, else a workflow_node run). Difficulty: intermediate. Typical runtime: long.
Use it when: Before you trust a forecast model; You want to compare models fairly.
Avoid it when: Slow models with many folds (each fold is a fit).
Example: Ten folds of ARIMAX: MAPE 6% at 1 week, 11% at 8 weeks.
Inputs: data (dataset), split (split_plan).
Outputs: metrics (metrics), table (table).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
model |
string | required | Catalog model ref (id@range), for example ts.sarimax@^1 |
settings |
object | ||
fit |
object | ||
pipeline |
array | ||
bind |
object | ||
metrics |
array | ["mae", "rmse", "mape", "mase"] |
|
refit |
boolean | true |
Fit again on each fold (if false, fit one time and roll forward) |
nodes |
object | Tree datasets: the nodes merged into the model input (data.nodes: {name: {path, vars?, prefix?}}). If not given, the node reads the root node | |
join |
exact \ |
inner \ |
outer |
Result fields (nodes.<id>.<field> in conditions): metrics (object), by_horizon (object), n_folds (integer).
Expert detail
From v0.11, the node run logs the metrics of each fold as scalar metrics with the label fold. The Metrics tab of the run page shows them.
model.fit — Fit a model¶
Fits a model from the catalog to the data and saves it as a model version. It can also score the model on a test period.
Runs in its own child run (a fit run for model.fit, else a workflow_node run). Difficulty: beginner. Typical runtime: minutes.
Use it when: Each workflow that ends with a model.
Avoid it when: You only need checks of the data.
Example: Fit ts.sarimax@^1 with the order from the search node.
Inputs: data (dataset), split (split_plan, optional).
Outputs: model (model_version), metrics (metrics).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
model |
string | required | Catalog model ref (id@range), for example ts.sarimax@^1 |
settings |
object | Model settings | |
fit |
object | Fit configuration (FitConfig) | |
pipeline |
array | Preprocessing steps | |
bind |
object | Binding to canonical names | |
evaluate |
object | Evaluation block (metrics) | |
calibration |
object | The calibration block of the model (business priors and lift tests); lift tests can come from model.lift_test | |
nodes |
object | Tree datasets: the nodes merged into the model input (data.nodes: {name: {path, vars?, prefix?}}). If not given, the node reads the root node | |
join |
exact \ |
inner \ |
outer |
Result fields (nodes.<id>.<field> in conditions): model_version (string), metrics (object), scorecard (string).
model.forecast — Forecast with a model¶
Uses a fitted model version to forecast the future periods. It uses future drivers if the model needs them.
Runs in its own child run (a fit run for model.fit, else a workflow_node run). Difficulty: beginner. Typical runtime: seconds.
Use it when: You want the forecast after you select a model.
Avoid it when: The model cannot forecast (look at its score modes).
Example: Forecast 12 weeks with planned prices.
Inputs: model (model_version), data (dataset, optional).
Outputs: forecast (dataset), plot (plot).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
horizon |
integer | 12 |
The number of periods ahead |
future_covariates |
carry_last \ |
seasonal_naive \ |
zero \ |
summary |
array | ["mean", "hdi_90"] |
Result fields (nodes.<id>.<field> in conditions): total (number).
model.lift_test — Make a lift test from a causal model¶
Adds the effect that a causal model measured in the test period and gives it as a lift test. A marketing mix model can use this lift test as calibration.
Runs in a sandboxed workflow_node run. Difficulty: intermediate. Typical runtime: seconds.
Use it when: You measured a campaign or a geo test with causal.synthetic_control, causal.impact or causal.did and you want the marketing mix model to agree with the result; You want the total effect of a test with its uncertainty in one table.
Avoid it when: The model is not a causal model (the node needs the impact group of the model); The periods of the data are not dates (a lift test needs the start and end dates).
Example: TV in market geo_03 for six weeks with 50 000 extra spend: KPI change 4 100, 90% range 2 300 to 5 900.
Inputs: model (model_version).
Outputs: table (table).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
channel |
string | required | The marketing channel of the test (a channel of the marketing mix model) |
spend_delta |
number | required | The extra spend during the test, in spend units (a negative number for a holdout) |
id |
string | The id of the lift test (optional) | |
hdi_prob |
number | 0.9 |
The probability inside the range of kpi_delta |
Result fields (nodes.<id>.<field> in conditions): lift_test (object), kpi_delta (number), se (number), hdi_low (number), hdi_high (number), p_positive (number), excludes_zero (boolean), relative_effect (number), start (string), end (string), geo (string), n_treated (integer), n_cells (integer), n_missing (integer), hdi_prob (number), model (string).
Expert detail
The node reads only the impact group of the bundle. kpi_delta is the posterior mean of the sum of effect over the post cells and the treated units, se its standard deviation, and ci the HDI at hdi_prob (ci_level = hdi_prob). The test dates are the first and the last post period. geo is the treated unit when exactly one unit has post cells; with more units the lift test is national (geo null) and the node gives a warning. The effect includes the observation noise of the counterfactual (predictive draws), so the range is wide on purpose.
model.score — Score a model¶
Uses a fitted model version to predict, simulate, forecast or test a scenario. It gives the predictions as a dataset for the next steps.
Runs in a sandboxed workflow_node run. Difficulty: intermediate. Typical runtime: seconds.
Use it when: You want predictions or simulations of a fitted model in the workflow; A plot or a second model must use the predictions.
Avoid it when: You only want a forecast with a fan chart (use model.forecast).
Example: Simulate 200 graphs from the fitted network model to check its fit.
Inputs: model (model_version), data (dataset, optional).
Outputs: scores (dataset).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
mode |
predict \ |
simulate \ |
forecast \ |
request |
object | More fields of the score request (horizon, output, select, interventions, seed, future_covariates) | |
group |
string | The group of the score result to output (default: predictions, else summary) |
Result fields (nodes.<id>.<field> in conditions): mode (string), n_vars (integer), group (string), representation (string).
Expert detail
The node runs the score path of a score run (mapi_sdk.harness.score_bundle) with the full representation. 'request' takes the fields of a ScoreRequest (horizon, output, select, interventions, seed, future_covariates).
model.summary — Summarise the parameters¶
Gives the estimate, the spread and the credible range of each parameter of a fitted Bayesian model, with checks that the sampler converged.
Runs in a sandboxed workflow_node run. Difficulty: intermediate. Typical runtime: seconds.
Use it when: You want to branch on a parameter, for example when its range excludes zero; You must stop when the sampler did not converge (R-hat, effective sample size).
Avoid it when: Models that give only point estimates (the node needs draws).
Example: slope: mean 0.42, 94% range 0.18 to 0.66, R-hat 1.00. The range excludes zero.
Inputs: model (model_version).
Outputs: table (table).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
vars |
array | The posterior variables (default: all of them) | |
hdi_prob |
number | 0.94 |
The probability inside the HDI |
focus |
string | One value to give in the result field focus: its name in the table (beta[north]) or only its label (north) |
Result fields (nodes.<id>.<field> in conditions): params (object), max_rhat (number), min_ess (number), n_divergent (integer), n_params (integer), truncated (boolean), focus (object).
Expert detail
Rank-normalised split R-hat (the larger of bulk and tail) and bulk ESS (Vehtari et al. 2021); the HDI is the narrowest interval with hdi_prob of the draws (0.94 gives the 3% and 97% bounds). Vector parameters are flattened to name[label] (labels of several dims joined by ', '), at most 200 values; max_rhat and min_ess check all the values (up to 20 000). Read the values with nodes.
Evaluation (evaluate)¶
Measures how good a model or a simple benchmark is on test data.
evaluate.baseline — Simple benchmark forecasts¶
Scores very simple forecasts, for example 'same as last week' or 'same as last year', on the same test periods. You can then make sure that a model is better than them.
Runs in a sandboxed workflow_node run. Difficulty: beginner. Typical runtime: seconds.
Use it when: All forecast workflows (a model must be better than these benchmarks to be useful).
Avoid it when: There is no time order.
Example: Seasonal naive has a MAPE of 14%. The ARIMAX model has 9%.
Inputs: data (dataset), split (split_plan).
Outputs: metrics (metrics).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
methods |
array | ["naive", "seasonal_naive"] |
|
season |
integer | The season length (default: from the frequency) | |
var |
string | The series (default: the target) |
Result fields (nodes.<id>.<field> in conditions): by_method (object), best_method (string).
evaluate.metrics — Measure forecast error¶
Measures the difference between the predictions of a model and the actual values on the test data. It gives several measures that are easy to read.
Runs in a sandboxed workflow_node run. Difficulty: beginner. Typical runtime: seconds.
Use it when: After you fit or backtest a model.
Avoid it when: No data was kept for the test.
Example: MAPE 8%, and 88% of the actual values are in the 90% range.
Inputs: model (model_version, optional), backtest (metrics, optional).
Outputs: metrics (metrics).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
metrics |
array | ["mae", "rmse", "mape", "mase", "coverage_90"] |
|
season |
integer | The season length for MASE (default: from the frequency) |
Result fields (nodes.<id>.<field> in conditions): mae (number), rmse (number), mape (number), smape (number), mase (number), coverage_90 (number), crps (number), by_horizon (object).
Expert detail
MASE scales by the in-sample seasonal-naive MAE (Hyndman & Koehler 2006); CRPS from draws or quantiles.
evaluate.ppc — Check the predictions against the data¶
Simulates new data from a fitted Bayesian model and compares it with the observed data. It shows if the model can make data that looks like the real data.
Runs in a sandboxed workflow_node run. Difficulty: intermediate. Typical runtime: seconds.
Use it when: After each Bayesian fit, before you trust its ranges; You must stop a workflow when the model does not agree with the data.
Avoid it when: Models that give only point estimates or summaries (the node needs draws).
Example: 88% of the values are in the 90% range and p = 0.41 for the mean: the check passes.
Inputs: model (model_version).
Outputs: metrics (metrics).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
statistic |
mean \ |
sd \ |
max \ |
n_draws |
integer | 400 |
The number of predictive draws to use |
p_min |
number | 0.025 |
The smallest p-value (and 1 - p_min the largest) that passes |
coverage_tolerance |
number | 0.1 |
The largest gap between the coverage and its level that passes |
pit_ks_max |
number | The largest PIT distance that passes (default: 1.36 / sqrt(n)) |
Result fields (nodes.<id>.<field> in conditions): coverage_50 (number), coverage_90 (number), pit_ks (number), p_values (object), ok (boolean), verdict_plain (string).
Expert detail
Posterior predictive draws come from the score path (predict, output kind predictive) on the fit data. coverage_p is the share of observations inside the central p interval of their predictive draws; pit_ks is the Kolmogorov-Smirnov distance of the PIT values P(y_rep < y) + 0.5 P(y_rep = y) from uniform, with the limit 1.36 / sqrt(n) unless pit_ks_max is set; the p-value is P(T(y_rep) >= T(y)) for T = statistic. ok needs both coverages within coverage_tolerance of nominal, the p-value in [p_min, 1 - p_min] and pit_ks under its limit.
evaluate.residuals — Test the model's errors¶
Tests the errors of a model for patterns that remain, uneven size and shapes that are not a bell curve. It does all these tests at the same time.
Runs in a sandboxed workflow_node run. Difficulty: intermediate. Typical runtime: seconds.
Use it when: After you fit a time-series model or a regression model.
Avoid it when: Models without residuals (classifiers).
Example: No pattern remains, the tails are slightly heavy and the spread is even.
Inputs: model (model_version).
Outputs: table (table), plot (plot).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
lags |
integer | 10 |
Ljung-Box lags |
alpha |
number | 0.05 |
The significance level |
Result fields (nodes.<id>.<field> in conditions): white_noise (boolean), normal (boolean), equal_spread (boolean).
Comparison (compare)¶
Puts the scores of several candidates side by side, or tests whether one candidate is really better.
compare.diebold_mariano — Is one forecast really better?¶
Tests whether the forecasts of one model are really more accurate than the forecasts of another model, or whether the difference is possibly luck.
Runs in a sandboxed workflow_node run. Difficulty: expert. Typical runtime: instant.
Use it when: Two models are close and you need to know if the gap is real.
Avoid it when: Very few test periods.
Example: ARIMAX is better than ETS, but p = 0.31. The difference is possibly luck.
Inputs: a (metrics), b (metrics).
Outputs: result (test_result).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
loss |
squared \ |
absolute |
"squared" |
horizon |
integer | 1 |
h |
step |
integer | The periods between fold origins (default: from the folds) | |
alpha |
number | 0.05 |
The significance level |
Result fields (nodes.<id>.<field> in conditions): statistic (number), p_value (number), alpha (number), reject (boolean), decision (string), verdict_plain (string), critical_values (object), n_obs (integer), lags (integer), details (object), per_series (array), reject_share (number).
Expert detail
Diebold-Mariano test with the Harvey-Leybourne-Newbold small-sample correction on squared or absolute loss, on one h-step error per backtest fold. Folds step periods apart share test periods only up to ceil(h / step) - 1 folds, which is the number of autocovariances in the long-run variance (h - 1 for consecutive origins); the step is read from the folds.
compare.models — Compare models side by side¶
Puts the scores of several models and benchmarks in one table. You can see which one is best on each measure.
Runs in the orchestrator (no data; computed from earlier results). Difficulty: beginner. Typical runtime: instant.
Use it when: You fitted or backtested several candidates.
Avoid it when: The models have tests on different periods (then the comparison is not fair).
Example: ARIMAX 9% MAPE, ETS 11%, seasonal naive 14%.
Inputs: candidates (metrics, several).
Outputs: table (table).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
metric |
string | "rmse" |
|
direction |
min \ |
max |
"min" |
Result fields (nodes.<id>.<field> in conditions): ranking (array), best (string).
compare.stacking — Weigh models by their predictions¶
Gives each candidate model a weight from how well it predicts each data point that it did not see. A mix of the models with these weights predicts best.
Runs in a sandboxed workflow_node run. Difficulty: expert. Typical runtime: seconds.
Use it when: You fitted several Bayesian models of the same data and none is clearly best; You want to average the predictions of the models, not select one of them.
Avoid it when: The models used different data (the scores are not comparable).
Example: Weights: bym2 0.71, icar 0.29, iid 0.00.
Inputs: candidates (metrics, several).
Outputs: table (table).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
metric |
elpd_loo \ |
elpd_waic |
"elpd_loo" |
method |
stacking \ |
pseudo_bma_plus |
"stacking" |
n_bootstrap |
integer | 1000 |
Bootstrap draws for pseudo-BMA+ |
Result fields (nodes.<id>.<field> in conditions): weights (object), pseudo_bma (object), best (string), method (string), note_plain (string).
Expert detail
Stacking maximises sum_i log sum_k w_k exp(elpd_loo_ik) over the simplex (Yao, Vehtari, Simpson & Gelman 2018), from the pointwise PSIS-LOO values that model.fit stores for data of at most 10 000 points. pseudo_bma (always reported) is pseudo-BMA+: softmax weights averaged over a Bayesian bootstrap of the pointwise elpd. Without pointwise values the weights are pseudo-BMA from elpd ± se (method pseudo_bma_se) or from elpd alone (pseudo_bma), and note_plain says so.
Selection (select)¶
Selects the winner, or all candidates that pass a limit.
select.best — Select the best model¶
Selects the model with the best score from a comparison. It can also make sure that the model is better than the simple benchmarks.
Runs in the orchestrator (no data; computed from earlier results). Difficulty: beginner. Typical runtime: instant.
Use it when: The end of a bake-off between candidates.
Avoid it when: Only one candidate.
Example: Select the lowest RMSE, but only if it is 5% better than seasonal naive.
Inputs: table (table), models (model_version, several, optional).
Outputs: model (model_version).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
metric |
string | "rmse" |
|
direction |
min \ |
max |
"min" |
must_beat |
array | The benchmark ids that the winner must be better than | |
margin |
number | 0.0 |
The necessary relative improvement over must_beat |
Result fields (nodes.<id>.<field> in conditions): selected (string), selected_model (string), score (number), beats_baselines (boolean), reason_plain (string).
select.threshold — Keep models that pass a limit¶
Keeps only the models with a score that passes a limit that you set, for example an error below 10%.
Runs in the orchestrator (no data; computed from earlier results). Difficulty: beginner. Typical runtime: instant.
Use it when: You need all models that are good enough, not only the best.
Avoid it when: You need one winner only.
Example: Keep all models with a MAPE below 10%.
Inputs: table (table).
Outputs: table (table).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
metric |
string | required | |
max |
number | Upper limit | |
min |
number | Lower limit |
Result fields (nodes.<id>.<field> in conditions): passed (array), n_passed (integer).
Governance (governance)¶
Runs the checks that a model must pass before you can validate it. It stores the evidence.
governance.check — Run one governance check¶
Runs one check from the governance catalog on a model, for example the shuffled-result test.
Runs in its own child run (a fit run for model.fit, else a workflow_node run). Difficulty: intermediate. Typical runtime: minutes.
Use it when: You want to try a check before you add it to a policy.
Avoid it when: Formal validation (run the full policy).
Example: The noise-channel check gives the fake channel a 2% share. The check passes.
Inputs: model (model_version), data (dataset, optional).
Outputs: evidence (evidence).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
check |
string | required | Check id (governance catalog) |
params |
object | Check parameters | |
warn |
number | Warn threshold | |
fail |
number | Fail threshold |
Result fields (nodes.<id>.<field> in conditions): status (string), value (number).
governance.policy — Run the governance checks¶
Runs on a model all the checks that the policy of your organisation needs, and records the evidence. For example, the model must be better than random noise and a simple benchmark.
Runs in its own child run (a fit run for model.fit, else a workflow_node run). Difficulty: intermediate. Typical runtime: long. Never cached (its result depends on more than its inputs).
Use it when: Before you validate and promote a model.
Avoid it when: Exploratory work that nobody will use.
Example: Seven checks pass and the placebo check fails, so the model is not validated.
Inputs: model (model_version), data (dataset, optional).
Outputs: evidence (evidence).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
policy |
string | The policy id (default: the policy that is in force) | |
submit |
boolean | false |
Submit the result for approval when the checks pass |
Result fields (nodes.<id>.<field> in conditions): passed (boolean), failed_checks (array), status (string).
Report (report)¶
Changes the results of the workflow into a report that is easy to read, or into an evidence pack.
report.evidence_pack — Evidence pack¶
Puts the governance evidence, decisions and approvals of a model into one document for auditors, as HTML and JSON.
Runs in its own child run (a fit run for model.fit, else a workflow_node run). Difficulty: beginner. Typical runtime: seconds.
Use it when: An audit, a model risk review or an approval.
Avoid it when: Before the governance checks run.
Example: A pack with 9 checks, 2 approvals and the lineage of model v7.
Inputs: model (model_version), evidence (evidence, optional).
Outputs: report (report).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
format |
html \ |
json |
"html" |
report.summary — Plain-language report¶
Writes a short report of the workflow results: the test verdicts, the decisions, the selected model and how good it is.
Runs in the orchestrator (no data; computed from earlier results). Difficulty: beginner. Typical runtime: instant.
Use it when: At the end of a workflow that you share with other persons.
Avoid it when: You only need the raw numbers.
Example: Sales wander, so the workflow applied a difference. ARIMAX(2,1,1) was 30% better than seasonal naive.
Inputs: items (table, several, optional), tests (test_result, several, optional), metrics (metrics, several, optional), model (model_version, optional), plots (plot, several, optional).
Outputs: report (report).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
title |
string | ||
sections |
array | Node ids to include (default: all) |
Plot (plot)¶
Draws a plot from the plot catalog in the run, for the run view and the report.
plot.render — Draw a plot¶
Draws a plot from the plot catalog with a model or a dataset of earlier steps. The run shows the plot, and a report can include it.
Runs in a sandboxed workflow_node run. Difficulty: beginner. Typical runtime: seconds.
Use it when: You want a chart of a model or of predictions in the run and in the report.
Avoid it when: The Model Explorer already shows the chart that you need.
Example: Draw sci.field_map of the fitted heat plate model.
Inputs: model (model_version, optional), data (dataset, optional).
Outputs: plot (plot).
| Parameter | Type | Default | Meaning |
|---|---|---|---|
recipe |
string | required | Plot recipe ref (id or id@range), for example core.ppc_fan |
settings |
object | The settings of the recipe | |
source |
auto \ |
data \ |
model \ |
title |
string | The title of the plot (default: the recipe title) |
Result fields (nodes.<id>.<field> in conditions): recipe (string), kind (string), source (string), bytes (integer).
Expert detail
The node renders the recipe with mapi_plots.recipes.render_plots and the sanitizer. With 'source: auto' it tries the dataset, the model bundle and then the in-sample predictions of the model with the observed values.
Sub-workflow (subworkflow)¶
Runs another saved workflow as one step.
subworkflow.run — Run another workflow¶
Runs a saved workflow as one step of this workflow. You write common sequences, for example data readiness checks, one time and use them again.
Runs in its own child run (a fit run for model.fit, else a workflow_node run). Difficulty: intermediate. Typical runtime: minutes.
Use it when: Several workflows use the same sequence of steps.
Avoid it when: Only this workflow uses the steps.
Example: Use 'time-series readiness' again before three different model workflows.
| Parameter | Type | Default | Meaning |
|---|---|---|---|
workflow |
string | required | workflow:// |
params |
object | The parameters to give to the other workflow |