Built-in steps catalog¶

Generated from the step catalog export (python -m mapi_steps.catalog and the steps of the installed plugins): the same static JSON the control plane uses to validate specs without importing step code. For how the steps behave together, read Built-in steps.

Step	Kind	Learns state	Invertible	Looks back	Summary
`core.calendar`	feature	–	–	–	Calendar features (day of week, month, ISO week, ...)
`core.encode_levels`	feature	yes	–	–	Integer codes for categorical levels
`core.formula`	feature	–	–	–	Derived variable from an arithmetic expression
`core.fourier`	feature	yes	–	–	Fourier seasonality features
`core.holidays`	feature	–	–	–	Holiday indicators per period from a side table
`core.lag`	feature	–	–	yes	Lagged copies of variables within groups
`core.lag_tensor`	feature	–	–	yes	Lag tensor (lags 0 .. l_max-1) for in-model carry-over
`core.missing_indicator`	feature	–	–	–	Indicators of missing values
`core.one_hot`	feature	yes	–	–	One-hot indicators of categorical variables
`core.trend`	feature	yes	–	–	Polynomial time trend
`dl.embed`	feature	yes	–	–	Image features from a frozen pretrained network
`core.count_events`	reshape	–	–	–	Count events per period on a dense grid
`core.filter_coords`	reshape	–	–	–	Select labels of a dim
`core.resample`	reshape	–	–	–	Aggregate a time dim to a coarser frequency
`core.stack`	reshape	–	–	–	Stack dims into one
`core.to_long`	reshape	–	–	–	Padded ragged data to long form
`core.to_padded`	reshape	–	–	–	Long ragged data to padded form with a mask
`core.unstack`	reshape	–	–	–	Unstack a dim into several
`core.adstock`	value	–	–	yes	Adstock (carry-over) with fixed parameters
`core.clip`	value	–	–	–	Clip values to bounds
`core.fx_convert`	value	–	yes	–	Currency conversion with per-period rates from a side table
`core.impute`	value	yes	–	–	Fill missing values (mean, median, constant or forward fill)
`core.log1p`	value	–	yes	–	log(1 + x) of the named variables
`core.logit`	value	–	yes	–	Logit of proportions
`core.max_abs_scale`	value	yes	yes	–	Divide by the maximum absolute value over the named dims
`core.mean_scale`	value	yes	yes	–	Divide by the mean absolute value over the named dims
`core.replace_sentinel`	value	–	–	–	Turn sentinel codes into missing values
`core.saturation`	value	–	–	–	Saturation curve with fixed parameters
`core.standardize`	value	yes	yes	–	Standardise to zero mean and unit variance over the named dims

`core.calendar`¶

Adds calendar facts such as the month or whether it is a weekend

version 1.0.0 · kind feature

- {id: my_calendar, use: core.calendar@^1, with: {...}}

Field	Type	Default	Description
`along`	`string`	`time`
`features`	list of `day_of_week` or `day_of_month` or `day_of_year` or `week_of_year` or `month` or `quarter` or `year` or `is_weekend` or `is_month_start` or `is_month_end`	required	min items 1, max items 10
`out`	`string`	`calendar`

Schema effect template

{
  "template_params": [
    "out",
    "along"
  ],
  "transforms_dims": {
    "{out}_feature": {
      "kind": "categorical",
      "policy": "fixed"
    }
  },
  "adds": {
    "{out}": {
      "dims": [
        "{along}",
        "{out}_feature"
      ],
      "dtype": "float64"
    }
  }
}

`core.encode_levels`¶

Numbers each category

version 1.0.0 · kind feature · learns state on the training data

- {id: my_encode_levels, use: core.encode_levels@^1, with: {...}}

Field	Type	Default	Description
`vars`	list of `string`	`[]`	Categorical variables to encode (max items 256)
`dims`	list of `string`	`[]`	Dims whose labels to encode (max items 256)
`role`	`covariate` or `index`	`covariate`
`max_levels`	`integer`	`100000`	≥ 1, ≤ 1000000
`unseen_levels`	`error` or `pooled` or `fill`	`error`	error, fill (fill_code) or pooled (one shared code n_levels)
`fill_code`	`integer`	`-1`	Code for missing values and 'fill' unseen levels

Schema effect template

{
  "template_params": [
    "vars",
    "dims",
    "role"
  ],
  "derives": {
    "{*vars as v}_code": {
      "from": "{v}",
      "dtype": "int64",
      "role": "{role}",
      "missing": "forbid",
      "units": "1"
    }
  },
  "adds": {
    "{*dims as d}_code": {
      "dims": [
        "{d}"
      ],
      "dtype": "int64",
      "role": "{role}"
    }
  }
}

`core.formula`¶

Creates a new column from a formula such as price * units

version 1.0.0 · kind feature

- {id: my_formula, use: core.formula@^1, with: {...}}

Field	Type	Default	Description
`out`	`string`	required	Name of the new variable
`expr`	`string`	required
`dims`	list of `string`	`[]`	Dims of the result (broadcast if it has fewer) (max items 256)
`role`	`covariate` or `media` or `spend` or `control` or `constant` or `weight` or `exposure` or `mask`	`covariate`
`units`	`string` (optional)	`null`

Schema effect template

{
  "template_params": [
    "out",
    "dims",
    "role",
    "units"
  ],
  "adds": {
    "{out}": {
      "dims": "{dims}",
      "dtype": "float64",
      "role": "{role}",
      "units": "{units}",
      "missing": "allow"
    }
  }
}

`core.fourier`¶

Adds smooth repeating patterns (for example yearly seasonality)

version 1.0.0 · kind feature · learns state on the training data

- {id: my_fourier, use: core.fourier@^1, with: {...}}

Field	Type	Default	Description
`along`	`string`	`time`
`period`	`number`	required	Season length in units of `unit` (> 0)
`order`	`integer`	`2`	≥ 1, ≤ 50
`out`	`string`	`fourier`
`unit`	`step` or `days`	`step`	'step': periods of the dim's frequency; 'days': calendar days

Schema effect template

{
  "template_params": [
    "out",
    "along"
  ],
  "transforms_dims": {
    "{out}_mode": {
      "kind": "categorical",
      "policy": "fixed"
    }
  },
  "adds": {
    "{out}": {
      "dims": [
        "{along}",
        "{out}_mode"
      ],
      "dtype": "float64"
    }
  }
}

`core.holidays`¶

Marks periods that contain public holidays

version 1.0.0 · kind feature · reads side tables via table

- {id: my_holidays, use: core.holidays@^1, with: {...}}

Field	Type	Default	Description
`table`	`string`	required	Side table with a daily (or finer) holiday variable
`along`	`string`	`time`
`table_along`	`string` (optional)	`null`	Time dim of the table, if different
`table_var`	`string`	`holiday`	Indicator/weight variable in the table
`out`	`string`	`holiday`
`by`	list of `string`	`[]`	Dims shared by table and data (e.g. country/geo) (max items 256)
`how`	`sum` or `max`	`sum`	Aggregate table rows inside each data period (count or any)
`freq`	`string` (optional)	`null`	Frequency of `along` (default: from the data)
`missing_levels`	`error` or `zero`	`error`	Levels of `by` dims absent from the table: error or zero

Schema effect template

{
  "template_params": [
    "out",
    "along",
    "by"
  ],
  "adds": {
    "{out}": {
      "dims": [
        "{along}",
        "{*by}"
      ],
      "dtype": "float64",
      "role": "control"
    }
  }
}

`core.lag`¶

Adds last period's (or earlier) values as extra columns

version 1.0.0 · kind feature · depends on the target (recursive at forecast time)

- {id: my_lag, use: core.lag@^1, with: {...}}

Field	Type	Default	Description
`vars`	list of `string`	required	min items 1, max items 256
`along`	`string`	`time`
`within`	list of `string`	`[]`	Group dims (dense) or row group variables (long) (max items 256)
`mask`	`string` (optional)	`null`	Boolean variable; cells where it is False count as missing
`lags`	list of `integer`	required	min items 1, max items 64
`lag_dim`	`string`	`lag`
`fill`	`number` (optional)	`null`	Value for lags with no history (default NaN)

Context need (per group, stored at fit): {"along": "{along}", "n": "{max:lags}", "within": "{within}", "min_lag": "{min:lags}"}

Schema effect template

{
  "template_params": [
    "vars",
    "lag_dim"
  ],
  "transforms_dims": {
    "{lag_dim}": {
      "kind": "integer",
      "policy": "fixed"
    }
  },
  "derives": {
    "{*vars as v}_lagged": {
      "from": "{v}",
      "append_dims": [
        "{lag_dim}"
      ],
      "dtype": "float64",
      "role": "covariate",
      "missing": "allow"
    }
  }
}

`core.lag_tensor`¶

Keeps recent history of each series for carry-over effects

version 1.0.0 · kind feature · depends on the target (recursive at forecast time)

- {id: my_lag_tensor, use: core.lag_tensor@^1, with: {...}}

Field	Type	Default	Description
`vars`	list of `string`	required	min items 1, max items 256
`along`	`string`	`time`
`within`	list of `string`	`[]`	Group dims (dense) or row group variables (long) (max items 256)
`mask`	`string` (optional)	`null`	Boolean variable; cells where it is False count as missing
`l_max`	`integer`	required	Number of lags, 0 .. l_max - 1 (≥ 1, ≤ 520)
`pad`	`first` or `zero` or `nan`	`zero`	History before a series starts: its first value, 0 or NaN
`dim`	`string`	`lag`	Name of the new lag dim

Context need (per group, stored at fit): {"along": "{along}", "n": "{l_max}", "within": "{within}", "min_lag": 0}

Schema effect template

{
  "template_params": [
    "vars",
    "dim"
  ],
  "transforms_dims": {
    "{dim}": {
      "kind": "integer",
      "policy": "fixed"
    }
  },
  "updates": {
    "{*vars as v}": {
      "append_dims": [
        "{dim}"
      ],
      "dtype": "float64"
    }
  }
}

`core.missing_indicator`¶

Flags where values were missing

version 1.0.0 · kind feature

- {id: my_missing_indicator, use: core.missing_indicator@^1, with: {...}}

Field	Type	Default	Description
`vars`	list of `string`	required	min items 1, max items 256

Schema effect template

{
  "template_params": [
    "vars"
  ],
  "derives": {
    "{*vars as v}_missing": {
      "from": "{v}",
      "dtype": "float64",
      "role": "covariate",
      "missing": "forbid",
      "units": "1"
    }
  }
}

`core.one_hot`¶

Turns categories into yes/no columns

version 1.0.0 · kind feature · learns state on the training data

- {id: my_one_hot, use: core.one_hot@^1, with: {...}}

Field	Type	Default	Description
`vars`	list of `string`	required	Categorical variables to encode (min items 1, max items 256)
`drop_first`	`boolean`	`false`
`max_levels`	`integer`	`256`	≥ 1, ≤ 4096
`unseen_levels`	`error` or `pooled` or `fill`	`error`	error, fill (all zeros) or pooled (training level frequencies)

Schema effect template

{
  "template_params": [
    "vars"
  ],
  "transforms_dims": {
    "{*vars as v}_level": {
      "kind": "categorical",
      "policy": "fixed"
    }
  },
  "derives": {
    "{*vars as v}_onehot": {
      "from": "{v}",
      "append_dims": [
        "{v}_level"
      ],
      "dtype": "float64",
      "role": "covariate",
      "missing": "forbid",
      "units": "1"
    }
  }
}

`core.trend`¶

Adds a steady upward or downward drift over time

version 1.0.0 · kind feature · learns state on the training data

- {id: my_trend, use: core.trend@^1, with: {...}}

Field	Type	Default	Description
`along`	`string`	`time`
`out`	`string`	`trend`
`unit`	`step` or `days`	`step`
`power`	`integer`	`1`	Polynomial power of the scaled trend (≥ 1, ≤ 3)
`scale_to_fit`	`boolean`	`true`	Scale so the fitted range maps to [0, 1] (forecasts continue past 1)

Schema effect template

{
  "template_params": [
    "out",
    "along"
  ],
  "adds": {
    "{out}": {
      "dims": [
        "{along}"
      ],
      "dtype": "float64",
      "role": "control"
    }
  }
}

`dl.embed`¶

Turns each image into a list of numbers with a pretrained network

version 1.0.0 · kind feature · learns state on the training data · reads side tables via weights

- {id: my_embed, use: dl.embed@^1, with: {...}}

Field	Type	Default	Description
`pixels`	`string`	`pixels`	The image variable (image, y, x, channel)
`weights`	`string`	`weights`	The data.side_tables entry with the weights
`out`	`string`	`embedding`	The feature variable the step writes
`feature_dim`	`string`	`feature`	The dim of the features
`image_dim`	`string`	`image`	The dim that lists the images
`size`	list of any (optional)	`null`	Height and width the images are resized to (default: the weights')
`batch_size`	`integer`	`64`	Images per forward pass (≥ 1, ≤ 4096)
`keep_pixels`	`boolean`	`false`	Keep the pixel variable after the step

Schema effect template

{
  "template_params": [
    "out",
    "feature_dim",
    "image_dim"
  ],
  "transforms_dims": {
    "{feature_dim}": {
      "kind": "integer",
      "policy": "fixed"
    }
  },
  "adds": {
    "{out}": {
      "dims": [
        "{image_dim}",
        "{feature_dim}"
      ],
      "dtype": "float32",
      "role": "covariate"
    }
  }
}

`core.count_events`¶

Turns a list of events into counts per day, week or month

version 1.0.0 · kind reshape

- {id: my_count_events, use: core.count_events@^1, with: {...}}

Field	Type	Default	Description
`along`	`string`	`event`	Row dim of the event table
`time_var`	`string`	`timestamp`	Datetime variable along the rows
`freq`	`string`	required	Period length of the counts, e.g. 'D' or 'W-SUN'
`dim`	`string`	`time`	Name of the new time dim
`by`	list of `string`	`[]`	Row variables that become dims (max items 256)
`sum_vars`	list of `string`	`[]`	Numeric row variables summed per period (max items 256)
`out`	`string`	`count`
`levels`	map of list of `string`	`{}`	Pinned levels per `by` variable (default: observed)
`start`	`string` (optional)	`null`	Earliest event time kept (ISO); the grid starts at its period
`end`	`string` (optional)	`null`	Latest event time kept (ISO); the grid ends at its period

Schema effect template

{
  "template_params": [
    "along",
    "time_var",
    "by",
    "sum_vars",
    "dim",
    "freq",
    "out"
  ],
  "drops": [
    "{time_var}",
    "{*by}",
    "{*sum_vars}",
    "{along}"
  ],
  "transforms_dims": {
    "{dim}": {
      "kind": "time",
      "policy": "extend",
      "freq": "{freq}"
    },
    "{*by as b}": {
      "kind": "categorical",
      "policy": "fixed"
    }
  },
  "adds": {
    "{out}": {
      "dims": [
        "{*by}",
        "{dim}"
      ],
      "dtype": "float64",
      "units": "1"
    },
    "{*sum_vars as v}": {
      "dims": [
        "{*by}",
        "{dim}"
      ],
      "dtype": "float64"
    }
  }
}

`core.filter_coords`¶

Keeps only some markets, products or dates

version 1.0.0 · kind reshape

- {id: my_filter_coords, use: core.filter_coords@^1, with: {...}}

Field	Type	Default	Description
`dim`	`string`	required
`keep`	list of `string` or `integer` or `number` or `boolean` (optional)	`null`
`drop`	list of `string` or `integer` or `number` or `boolean` (optional)	`null`
`start`	`string` or `integer` or `number` or `boolean` (optional)	`null`	Inclusive lower bound (ordered dims)
`end`	`string` or `integer` or `number` or `boolean` (optional)	`null`	Inclusive upper bound; also applies at score

`core.resample`¶

Turns daily data into weekly or monthly totals

version 1.0.0 · kind reshape

- {id: my_resample, use: core.resample@^1, with: {...}}

Field	Type	Default	Description
`dim`	`string`	`time`
`freq`	`string`	required	Target frequency, e.g. 'W-SUN' or 'MS'
`how`	`sum` or `mean` or `max` or `min` or `first` or `last`	`sum`
`how_vars`	map of `sum` or `mean` or `max` or `min` or `first` or `last`	`{}`	Per-variable aggregation overriding `how`
`full_periods_only`	`boolean`	`true`	Drop periods that are not fully covered by the source data
`skipna`	`boolean`	`false`	Ignore missing values (a period with only missing values stays NaN)

Schema effect template

{
  "template_params": [
    "dim",
    "freq"
  ],
  "transforms_dims": {
    "{dim}": {
      "kind": "time",
      "policy": "extend",
      "freq": "{freq}"
    }
  }
}

`core.stack`¶

Combines several dimensions into a single list of cells

version 1.0.0 · kind reshape

- {id: my_stack, use: core.stack@^1, with: {...}}

Field	Type	Default	Description
`dims`	list of `string`	required	min items 2, max items 256
`new_dim`	`string`	required

Schema effect template

{
  "template_params": [
    "dims",
    "new_dim"
  ],
  "stacks": {
    "{new_dim}": "{dims}"
  }
}

`core.to_long`¶

Turns a padded table into one row per observation

version 1.0.0 · kind reshape

- {id: my_to_long, use: core.to_long@^1, with: {...}}

Field	Type	Default	Description
`group_dim`	`string`	required	Outer ragged dim (e.g. household)
`within_dim`	`string`	required	Inner padded dim (e.g. trip position)
`row_dim`	`string`	`obs`
`vars`	list of `string`	required	Every variable on the padded grid (min items 1, max items 256)
`mask`	`string` (optional)	`null`	Boolean padding mask (one of vars); rows where it is False go
`drop_missing`	`all` or `none`	`all`	Drop rows where every listed variable is missing

Schema effect template

{
  "template_params": [
    "group_dim",
    "within_dim",
    "row_dim",
    "vars"
  ],
  "renames": {
    "{within_dim}": "{row_dim}"
  },
  "transforms_dims": {
    "{row_dim}": {
      "kind": "integer",
      "policy": "extend"
    },
    "{within_dim}": {
      "kind": "integer",
      "policy": "fixed"
    }
  },
  "updates": {
    "{*vars as v}": {
      "remove_dims": [
        "{group_dim}"
      ],
      "missing": "allow"
    }
  },
  "adds": {
    "{group_dim}_idx": {
      "dims": [
        "{row_dim}"
      ],
      "dtype": "int64",
      "role": "index",
      "index_of": "{group_dim}"
    },
    "{within_dim}_idx": {
      "dims": [
        "{row_dim}"
      ],
      "dtype": "int64",
      "role": "index",
      "index_of": "{within_dim}"
    }
  }
}

`core.to_padded`¶

Lines up each group's rows side by side, padding short groups

version 1.0.0 · kind reshape

- {id: my_to_padded, use: core.to_padded@^1, with: {...}}

Field	Type	Default	Description
`row_dim`	`string`	`obs`
`group`	`string`	required	Variable along the rows naming each row's group
`group_dim`	`string`	required	Name of the outer dim (existing or new)
`within_dim`	`string`	required	Name of the new position dim
`vars`	list of `string`	required	Every other variable on the rows (min items 1, max items 256)
`mask_var`	`string`	`mask`	Name of the boolean validity mask to add
`group_policy`	`DimPolicy`	`fixed`

Schema effect template

{
  "template_params": [
    "row_dim",
    "group",
    "group_dim",
    "within_dim",
    "vars",
    "mask_var",
    "group_policy"
  ],
  "renames": {
    "{row_dim}": "{within_dim}"
  },
  "drops": [
    "{group}"
  ],
  "transforms_dims": {
    "{within_dim}": {
      "kind": "integer",
      "policy": "extend"
    },
    "{group_dim}": {
      "kind": "categorical",
      "policy": "{group_policy}"
    }
  },
  "updates": {
    "{*vars as v}": {
      "append_dims": [
        "{group_dim}"
      ],
      "missing": "allow",
      "dtype": "float64"
    }
  },
  "adds": {
    "{mask_var}": {
      "dims": [
        "{group_dim}",
        "{within_dim}"
      ],
      "dtype": "bool",
      "role": "mask"
    }
  }
}

`core.unstack`¶

Splits a combined dimension back into its parts

version 1.0.0 · kind reshape

- {id: my_unstack, use: core.unstack@^1, with: {...}}

Field	Type	Default	Description
`dim`	`string`	required
`into`	list of `string`	required	The stacked levels (min items 2, max items 256)

Schema effect template

{
  "template_params": [
    "dim",
    "into"
  ],
  "unstacks": {
    "{dim}": "{into}"
  }
}

`core.adstock`¶

Spreads each period's media effect over the following periods

version 1.0.0 · kind value

- {id: my_adstock, use: core.adstock@^1, with: {...}}

Field	Type	Default	Description
`vars`	list of `string`	required	min items 1, max items 256
`along`	`string`	`time`
`within`	list of `string`	`[]`	max items 256
`kind`	`geometric` or `delayed` or `weibull_pdf` or `weibull_cdf`	`geometric`
`l_max`	`integer`	`12`	≥ 1, ≤ 520
`alpha`	`number` or map of `number` (optional)	`null`	geometric/delayed decay in [0, 1)
`theta`	`number` or map of `number` (optional)	`null`	delayed: lag of the peak effect (>= 0)
`lam`	`number` or map of `number` (optional)	`null`	weibull: scale (> 0)
`k`	`number` or map of `number` (optional)	`null`	weibull: shape (> 0)
`by`	`string` (optional)	`null`	Dim whose levels key per-level parameters
`normalize`	`boolean`	`false`	Scale the weights to sum to 1
`mask`	`string` (optional)	`null`

Context need (per group, stored at fit): {"along": "{along}", "n": "{l_max}", "within": "{within}"}

Schema effect template

{
  "template_params": [
    "vars"
  ],
  "updates": {
    "{*vars as v}": {
      "dtype": "float64"
    }
  }
}

`core.clip`¶

Caps values that are too small or too large

version 1.0.0 · kind value

- {id: my_clip, use: core.clip@^1, with: {...}}

Field	Type	Default	Description
`vars`	list of `string`	required	min items 1, max items 256
`lower`	`number` (optional)	`null`
`upper`	`number` (optional)	`null`

Schema effect template

{
  "template_params": [
    "vars"
  ],
  "updates": {
    "{*vars as v}": {
      "dtype": "float64"
    }
  }
}

`core.fx_convert`¶

Converts money columns into one currency using daily exchange rates

version 1.0.0 · kind value · invertible · reads side tables via table

- {id: my_fx_convert, use: core.fx_convert@^1, with: {...}}

Field	Type	Default	Description
`vars`	list of `string`	required	min items 1, max items 256
`table`	`string`	required	Name of the side table in data.side_tables
`to`	`string`	`EUR`	Target currency (ISO 4217)
`along`	`string`	`time`	Time dim shared by the data and the table
`table_along`	`string` (optional)	`null`	Time dim name in the table, if different
`rate_var`	`string`	`rate`	Rate variable: units of `to` per local unit
`table_currency_dim`	`string`	`currency`	Currency dim of the rate table
`currency_dim`	`string` (optional)	`null`	Data dim whose levels have different local currencies (e.g. geo)
`currencies`	map of `string`	`{}`	Local currency per level of currency_dim
`default_currency`	`string` (optional)	`null`	Currency of levels missing from currencies

Schema effect template

{
  "template_params": [
    "vars",
    "to"
  ],
  "updates": {
    "{*vars as v}": {
      "units": "{to}",
      "dtype": "float64"
    }
  }
}

`core.impute`¶

Fills gaps in the data

version 1.0.0 · kind value · learns state on the training data

- {id: my_impute, use: core.impute@^1, with: {...}}

Field	Type	Default	Description
`vars`	list of `string`	required	min items 1, max items 256
`method`	`mean` or `median` or `constant` or `ffill`	`mean`
`over`	list of `string`	`[]`	mean/median: dims the statistic reduces over (max items 256)
`value`	`number`	`0.0`	constant: the fill value
`along`	`string`	`time`	ffill: ordered dim to carry values along
`within`	list of `string`	`[]`	ffill: group dims (dense) or row group vars (long) (max items 256)
`limit`	`integer` (optional)	`null`	ffill: max consecutive missing values filled
`unseen_levels`	`error` or `pooled` or `fill`	`error`
`fill_value`	`number`	`0.0`	mean/median: statistic for unseen levels

Schema effect template

{
  "template_params": [
    "vars"
  ],
  "updates": {
    "{*vars as v}": {
      "dtype": "float64"
    }
  }
}

`core.log1p`¶

Compresses large values with a logarithm

version 1.0.0 · kind value · invertible

- {id: my_log1p, use: core.log1p@^1, with: {...}}

Field	Type	Default	Description
`vars`	list of `string`	required	min items 1, max items 256

Schema effect template

{
  "template_params": [
    "vars"
  ],
  "updates": {
    "{*vars as v}": {
      "dtype": "float64"
    }
  }
}

`core.logit`¶

Maps shares between 0 and 1 onto an unbounded scale

version 1.0.0 · kind value · invertible

- {id: my_logit, use: core.logit@^1, with: {...}}

Field	Type	Default	Description
`vars`	list of `string`	required	min items 1, max items 256
`eps`	`number`	`1e-06`	Values are clipped to [eps, 1 - eps] (> 0, < 0.5)

Schema effect template

{
  "template_params": [
    "vars"
  ],
  "updates": {
    "{*vars as v}": {
      "dtype": "float64"
    }
  }
}

`core.max_abs_scale`¶

Puts each series between -1 and 1

version 1.0.0 · kind value · learns state on the training data · invertible

- {id: my_max_abs_scale, use: core.max_abs_scale@^1, with: {...}}

Field	Type	Default	Description
`vars`	list of `string`	required	min items 1, max items 256
`over`	list of `string`	required	Dims the statistic reduces over (min items 1, max items 256)
`unseen_levels`	`error` or `pooled` or `fill`	`error`	Unseen levels at score time: error, pooled (mean) or fill
`fill_value`	`number`	`1.0`	Scale used for unseen levels with 'fill' (> 0)

Schema effect template

{
  "template_params": [
    "vars"
  ],
  "updates": {
    "{*vars as v}": {
      "units": "1",
      "dtype": "float64"
    }
  }
}

`core.mean_scale`¶

Puts each series on a comparable scale (average becomes 1)

version 1.0.0 · kind value · learns state on the training data · invertible

- {id: my_mean_scale, use: core.mean_scale@^1, with: {...}}

Field	Type	Default	Description
`vars`	list of `string`	required	min items 1, max items 256
`over`	list of `string`	required	Dims the statistic reduces over (min items 1, max items 256)
`unseen_levels`	`error` or `pooled` or `fill`	`error`	Unseen levels at score time: error, pooled (mean) or fill
`fill_value`	`number`	`1.0`	Scale used for unseen levels with 'fill' (> 0)

Schema effect template

{
  "template_params": [
    "vars"
  ],
  "updates": {
    "{*vars as v}": {
      "units": "1",
      "dtype": "float64"
    }
  }
}

`core.replace_sentinel`¶

Marks placeholder numbers such as -99 as missing

version 1.0.0 · kind value

- {id: my_replace_sentinel, use: core.replace_sentinel@^1, with: {...}}

Field	Type	Default	Description
`vars`	list of `string`	required	min items 1, max items 256
`sentinels`	list of `number`	`[]`	Exact codes that mean 'missing' (max items 64)
`below`	`number` (optional)	`null`	Values strictly below this are missing
`above`	`number` (optional)	`null`	Values strictly above this are missing

Schema effect template

{
  "template_params": [
    "vars"
  ],
  "updates": {
    "{*vars as v}": {
      "dtype": "float64",
      "missing": "allow"
    }
  }
}

`core.saturation`¶

Models diminishing returns of extra spend

version 1.0.0 · kind value

- {id: my_saturation, use: core.saturation@^1, with: {...}}

Field	Type	Default	Description
`vars`	list of `string`	required	min items 1, max items 256
`kind`	`logistic` or `hill` or `michaelis_menten` or `tanh`	`logistic`
`lam`	`number` or map of `number` (optional)	`null`	logistic: rate (> 0); michaelis_menten: K
`slope`	`number` or map of `number` (optional)	`null`	hill: slope (> 0)
`half`	`number` or map of `number` (optional)	`null`	hill: half-saturation point (> 0)
`alpha`	`number` or map of `number` (optional)	`null`	michaelis_menten / tanh: maximum (> 0)
`c`	`number` or map of `number` (optional)	`null`	tanh: initial cost per unit (> 0)
`by`	`string` (optional)	`null`	Dim whose levels key per-level parameters

Schema effect template

{
  "template_params": [
    "vars"
  ],
  "updates": {
    "{*vars as v}": {
      "dtype": "float64",
      "units": "1"
    }
  }
}

`core.standardize`¶

Centres each series and scales it to unit spread

version 1.0.0 · kind value · learns state on the training data · invertible

- {id: my_standardize, use: core.standardize@^1, with: {...}}

Field	Type	Default	Description
`vars`	list of `string`	required	min items 1, max items 256
`over`	list of `string`	required	Dims the moments reduce over (min items 1, max items 256)
`with_mean`	`boolean`	`true`
`with_sd`	`boolean`	`true`
`ddof`	`integer`	`0`	≥ 0, ≤ 1
`unseen_levels`	`error` or `pooled` or `fill`	`error`
`fill_mean`	`number`	`0.0`	Mean used for unseen levels with 'fill'
`fill_sd`	`number`	`1.0`	SD used for unseen levels with 'fill' (> 0)

Schema effect template

{
  "template_params": [
    "vars"
  ],
  "updates": {
    "{*vars as v}": {
      "units": "1",
      "dtype": "float64"
    }
  }
}

Splits¶

The data.split.kind values. Each has its own settings.

`time_holdout`¶

Hold out the end of an ordered dim (last N, a fraction, or after a label)

Tests the model on the most recent periods it has not seen

Field	Type	Default	Description
`after_step`	`string` (optional)	`null`	Stage the split happens at (a step id or 'raw'); default: end of the stateless prefix
`allow_leakage`	`boolean`	`false`	Downgrade LEAKAGE_RISK to a warning (acknowledged)
`dim`	`string`	required
`last`	`integer` (optional)	`null`
`fraction`	`number` (optional)	`null`
`after`	`string` or `integer` or `number` or `boolean` (optional)	`null`

`as_of`¶

Train up to an as-of label, hold out the rest (optional gap)

Pretends it is a past date and checks what the model would have said

Field	Type	Default	Description
`after_step`	`string` (optional)	`null`	Stage the split happens at (a step id or 'raw'); default: end of the stateless prefix
`allow_leakage`	`boolean`	`false`	Downgrade LEAKAGE_RISK to a warning (acknowledged)
`dim`	`string`	required
`as_of`	`string` or `integer` or `number` or `boolean`	required
`gap`	`integer`	`0`	≥ 0

`leave_group_out`¶

Hold out whole groups (explicit or random)

Checks how well the model handles markets or stores it has never seen

Field	Type	Default	Description
`after_step`	`string` (optional)	`null`	Stage the split happens at (a step id or 'raw'); default: end of the stateless prefix
`allow_leakage`	`boolean`	`false`	Downgrade LEAKAGE_RISK to a warning (acknowledged)
`dim`	`string`	required
`groups`	list of `string` or `integer` or `number` or `boolean` (optional)	`null`
`n`	`integer` (optional)	`null`
`seed`	`integer` (optional)	`null`

`last_per_group`¶

Hold out the last rows of each group with at least min_len rows

Tests each customer or store on its latest observations

Field	Type	Default	Description
`after_step`	`string` (optional)	`null`	Stage the split happens at (a step id or 'raw'); default: end of the stateless prefix
`allow_leakage`	`boolean`	`false`	Downgrade LEAKAGE_RISK to a warning (acknowledged)
`dim`	`string`	required
`within`	list of `string`	required	min items 1, max items 8
`last`	`integer`	`1`	≥ 1
`min_len`	`integer`	`1`	≥ 1

`mask_block`¶

Hold out random blocks of cells over several dims

Hides random patches of the data and checks the model fills them in

Field	Type	Default	Description
`after_step`	`string` (optional)	`null`	Stage the split happens at (a step id or 'raw'); default: end of the stateless prefix
`allow_leakage`	`boolean`	`false`	Downgrade LEAKAGE_RISK to a warning (acknowledged)
`dims`	list of `string`	required	min items 1, max items 8
`fraction`	`number`	required	> 0, < 1
`block`	map of `integer`	–
`seed`	`integer` (optional)	`null`

`column`¶

Hold out the rows whose column value is in holdout (a fixed split)

Uses the train and test assignment that the data already has

Field	Type	Default	Description
`after_step`	`string` (optional)	`null`	Stage the split happens at (a step id or 'raw'); default: end of the stateless prefix
`allow_leakage`	`boolean`	`false`	Downgrade LEAKAGE_RISK to a warning (acknowledged)
`dim`	`string`	required
`column`	`string`	required
`holdout`	list of `string` or `integer` or `number` or `boolean`	–	min items 1, max items 64
`train`	list of `string` or `integer` or `number` or `boolean` (optional)	`null`

`stratified`¶

Hold out a random fraction of the rows of each class

Tests the model on the same share of each class, selected at random

Field	Type	Default	Description
`after_step`	`string` (optional)	`null`	Stage the split happens at (a step id or 'raw'); default: end of the stateless prefix
`allow_leakage`	`boolean`	`false`	Downgrade LEAKAGE_RISK to a warning (acknowledged)
`dim`	`string`	required
`by`	`string`	required
`fraction`	`number`	required	> 0, < 1
`seed`	`integer` (optional)	`null`

Built-in steps catalog¶

core.calendar¶

core.encode_levels¶

core.formula¶

core.fourier¶

core.holidays¶

core.lag¶

core.lag_tensor¶

core.missing_indicator¶

core.one_hot¶

core.trend¶

dl.embed¶

core.count_events¶

core.filter_coords¶

core.resample¶

core.stack¶

core.to_long¶

core.to_padded¶

core.unstack¶

core.adstock¶

core.clip¶

core.fx_convert¶

core.impute¶

core.log1p¶

core.logit¶

core.max_abs_scale¶

core.mean_scale¶

core.replace_sentinel¶

core.saturation¶

core.standardize¶

Splits¶

time_holdout¶

as_of¶

leave_group_out¶

last_per_group¶

mask_block¶

column¶

stratified¶

`core.calendar`¶

`core.encode_levels`¶

`core.formula`¶

`core.fourier`¶

`core.holidays`¶

`core.lag`¶

`core.lag_tensor`¶

`core.missing_indicator`¶

`core.one_hot`¶

`core.trend`¶

`dl.embed`¶

`core.count_events`¶

`core.filter_coords`¶

`core.resample`¶

`core.stack`¶

`core.to_long`¶

`core.to_padded`¶

`core.unstack`¶

`core.adstock`¶

`core.clip`¶

`core.fx_convert`¶

`core.impute`¶

`core.log1p`¶

`core.logit`¶

`core.max_abs_scale`¶

`core.mean_scale`¶

`core.replace_sentinel`¶

`core.saturation`¶

`core.standardize`¶

`time_holdout`¶

`as_of`¶

`leave_group_out`¶

`last_per_group`¶

`mask_block`¶

`column`¶

`stratified`¶