Forecasting | MMM Framework

What We're Actually Asking

Forecasting isn't about predicting a single number—it's about understanding the range of likely outcomes given what we know. A forecast that says "We'll sell $10-12M next quarter" is more useful than "$11M" because it helps you plan for both optimistic and pessimistic scenarios.

The key questions forecasting can answer:

What's our expected outcome if we maintain current marketing levels?
How would outcomes change if we increase or decrease budget?
What's the probability of hitting a specific sales target?
What's the downside risk we should plan for?

The Weather Forecast Analogy

When a meteorologist says "70% chance of rain tomorrow," they're not saying it will definitely rain—they're telling you their confidence level. Good weather forecasts communicate uncertainty honestly, so you can decide whether to bring an umbrella.

Marketing forecasts work the same way. "We have a 75% probability of hitting our target" is more useful than a point estimate because it helps you decide how much risk buffer to plan.

The Core Concepts

2.1 Point Estimates vs. Prediction Intervals

A point estimate is a single number: "We predict $11M in sales." A prediction interval gives a range: "We're 94% confident sales will be between $9.5M and $12.5M."

Why intervals matter more than points

Point estimates give false confidence. If someone tells you "sales will be $11M," you might plan your operations, hiring, and inventory around exactly that number. When reality comes in at $9.8M or $12.2M, you're caught off guard.

Intervals let you plan for scenarios: "If we hit the low end, we can handle it. If we hit the high end, we're ready to scale." That's operational resilience.

2.2 The Fan Chart

A fan chart shows how uncertainty grows over time. Near-term forecasts are more certain (narrow band); far-term forecasts are less certain (wide band).

Interactive: Forecast Fan Chart

Adjust the budget multiplier to see how different spending levels affect the forecast.

Budget scenario: 100%

Forecast horizon:

Reading this chart: The solid line is actual sales (historical) and expected forecast (future). The shaded bands show uncertainty: darker = more likely, lighter = less likely. Notice how the band widens further into the future—that's growing uncertainty.

2.3 Scenario-Based Forecasting

Rather than a single forecast, you can generate multiple scenarios based on different budget assumptions. This helps stakeholders understand the trade-offs.

🛡 Conservative

Budget: -20% from baseline

Expected Outcome

$9.2M

[$8.4M - $10.1M]

P(Hit $10M Target)

32%

⚖ Baseline

Budget: Current levels

Expected Outcome

$11.2M

[$10.1M - $12.4M]

P(Hit $10M Target)

89%

🚀 Aggressive

Budget: +20% from baseline

Expected Outcome

$12.8M

[$11.5M - $14.2M]

P(Hit $10M Target)

98%

Target Achievement Calculator

Often the most useful question isn't "What's the expected value?" but rather "What's the probability we hit our target?" This calculator answers that directly.

What's the probability of hitting your target?

Sales Target: $ M

89%

Based on your current marketing plan, there's an 89% probability of exceeding $10M in sales next quarter.

Using target probabilities for decisions

Different targets warrant different confidence thresholds:

Break-even target: You need high confidence (>90%) to ensure survival
Budget target: Moderate confidence (70-80%) is often acceptable for planning
Stretch goal: Lower confidence (40-60%) is fine—it's aspirational

Downside Risk Assessment

Good planning considers not just the expected outcome but the downside risk. What happens if things go worse than expected?

📉

5th Percentile

$8.8M

"Worst case" (5% chance of being lower)

📊

25th Percentile

$10.2M

Conservative planning floor

🎯

50th Percentile

$11.2M

Median expectation

📈

95th Percentile

$13.6M

"Best case" (5% chance of being higher)

Interactive: Budget Sensitivity Analysis

How does the expected outcome and risk profile change as you adjust total marketing spend?

Reading this chart: The solid line shows expected outcome at different budget levels. The shaded band shows the 94% prediction interval. The vertical line marks your current budget. Drag the slider above to explore.

Practical Guidance

Common Pitfalls

Point forecast tunnel vision: Don't plan around a single number. The range matters more than the midpoint.
Extrapolating too far: Forecasts become unreliable with longer horizons. Be cautious beyond 2 quarters.
Ignoring external factors: The model doesn't know about competitor launches, economic shifts, or regulatory changes.
False precision: "$10,247,382" is not more accurate than "about $10M." Don't let spurious precision mislead stakeholders.

Best Practices

Present forecasts as ranges, not point estimates
Update forecasts regularly (monthly or quarterly) as new data arrives
Use scenario planning to explore sensitivities to key assumptions
Build buffer into plans based on the prediction interval, not just the mean
Track forecast accuracy over time to calibrate confidence

Implementation in MMM Framework

A scoping note before the code: predict() re-predicts within the fitted window — given the fitted model and a (possibly modified) media plan, it returns the posterior predictive for the modeled periods. That is the right tool for the what-if scenarios below. Genuine future-period forecasting — predicting weeks the model never saw — goes through the backtesting machinery in mmm_framework.validation.backtest (PosteriorForecaster extrapolates trend and seasonality past the training window and carries adstock across the cutoff), and its accuracy is measured in the section below.

Basic Forecasting

import numpy as np 

# Predict with current media plan
baseline_pred = fitted_model.predict()

print(f"Expected sales: ${baseline_pred.y_pred_mean.sum():,.0f}")
print(f"94% CI: [${baseline_pred.y_pred_hdi_low.sum():,.0f}, "
      f"${baseline_pred.y_pred_hdi_high.sum():,.0f}]")

# Output:
# Expected sales: $11,200,000
# 94% CI: [$10,100,000, $12,400,000]

Scenario-Based Forecasting

# Predict with modified media plan (20% budget increase)
X_media_increased = fitted_model.X_media_raw * 1.2
scenario_pred = fitted_model.predict(X_media=X_media_increased)

print(f"With 20% budget increase:")
print(f"Expected sales: ${scenario_pred.y_pred_mean.sum():,.0f}")
print(f"94% CI: [${scenario_pred.y_pred_hdi_low.sum():,.0f}, "
      f"${scenario_pred.y_pred_hdi_high.sum():,.0f}]")

# Output:
# With 20% budget increase:
# Expected sales: $12,800,000
# 94% CI: [$11,500,000, $14,200,000]

Target Probability Calculation

# Calculate probability of exceeding a target
target = 10_000_000

# Sum predictions across time to get total for the period
total_samples = scenario_pred.y_pred_samples.sum(axis=1)  # Sum across time


# Calculate probability
prob_exceed = (total_samples > target).mean()
print(f"Probability of exceeding ${target:,.0f}: {prob_exceed:.1%}")

# Output:
# Probability of exceeding $10,000,000: 98.2%

# Get various percentiles for risk assessment
percentiles = np.percentile(total_samples, [5, 25, 50, 75, 95])
print(f"5th percentile: ${percentiles[0]:,.0f}")
print(f"Median: ${percentiles[2]:,.0f}")
print(f"95th percentile: ${percentiles[4]:,.0f}")

Multi-Scenario Comparison

from mmm_framework.analysis import MMMAnalyzer


analyzer = MMMAnalyzer(fitted_model)


# Define scenarios
scenarios = {
     'Conservative': {'TV': 0.8, 'Digital': 0.8},
     'Baseline': {'TV': 1.0, 'Digital': 1.0},
     'Aggressive': {'TV': 1.2, 'Digital': 1.2}
}

# Run and compare
for name, spend_changes in scenarios.items():
     result = analyzer.what_if_scenario(spend_changes=spend_changes)
     print(f"{name}: ${result.scenario_outcome:,.0f}")

# Output:
# Conservative: $9,200,000
# Baseline: $11,200,000
# Aggressive: $12,800,000

Forecast Accuracy — Measured, Not Asserted

"How accurate were the model's forecasts on data it had not seen?" now has a measured answer. The framework ships a rolling-origin backtest harness (mmm_framework.validation.backtest): refit the full model on an expanding training window, forecast a fixed horizon past each cutoff, and grade against held-out actuals — no information from the forecast window reaches the fit, and adstock carryover is convolved across the cutoff correctly.

Measured (from the baked notebook nbs/validation/backtest_validation.ipynb) — realistic synthetic world (156 weeks × 7 channels, confounded spend, low signal-to-noise), 4 refits at cutoffs {104, 117, 130, 143}, 13-week horizons, 52 graded out-of-time weekly forecasts (NumPyro, 4 chains × 500 draws; each refit 15–19 s):

Forecaster	MAPE	MASE	80% PI coverage	95% PI coverage
MMM (this framework)	3.0%	0.24	90%	94%
Seasonal naive (copy last year)	13.8%	1.19	—	—
Naive (carry last value)	13.2%	1.07	—	—

Three readings of that table, in order of importance:

Skill is relative. MASE 0.24 means the model's out-of-time error is roughly a quarter of "copy the same week last year." Bare MAPE on a seasonal series flatters everyone; quote the baseline comparison.
Intervals survive contact with unseen data. The 80% prediction interval covered 90% of held-out actuals and the 95% interval covered 94%, at mean widths of 11% and 17% of the weekly KPI — honest, slightly conservative, not decoratively wide.
Error is roughly flat across the 13-week horizon (h 1–4 ≈ 3.3% vs h 10–13 ≈ 2.3% MAPE). That is structural: the model carries no autoregressive term, so week 13 is forecast from the same ingredients as week 1 (planned media, seasonality, trend). Good for quarter-ahead budget planning; not a substitute for short-range operational forecasting.

The harness also fails honestly: on the trend-break stress world (a level shock a linear-trend model cannot represent), the identical protocol reports 11.4% MAPE and 80%-interval coverage collapsing to 58% — a backtest that cannot fail cannot validate.

from mmm_framework.validation import BacktestConfig, run_backtest

config = BacktestConfig(
    min_train_size=104,   # two seasonal cycles before the first cutoff
    horizon=13,           # forecast a quarter past each cutoff
    step=13,              # non-overlapping forecast windows
    coverage_levels=(0.5, 0.8, 0.95),
)
result = run_backtest(mmm, config)   # refits per origin; ~15s each here

result.fits          # per-refit convergence — read this first
result.summary()     # MAPE/sMAPE/RMSE/MASE vs naive + seasonal-naive
result.by_horizon()  # accuracy and coverage by lead time
result.coverage()    # nominal vs empirical interval coverage + sharpness

Forecast skill is not causal validity

A model can forecast well while attributing wrongly — the confounded worlds in the pressure-testing series forecast fine while misattributing by >100%. Backtests validate the predictive model; attribution is validated by pressure testing and experiment calibration. Use both.

Run it yourself

This section mirrors nbs/validation/backtest_validation.ipynb (authored by nbs/builders/build_backtest_validation.py). Every computational cell ends in a seeded assert encoding the claim it demonstrates; the measured headlines above are recorded in nbs/artifacts/backtest_metrics.json at bake time. If a re-bake moves a number materially, this page is stale.