When a marketer asks "Which channels work?", they're really asking a causal question:
If I spend money on TV advertising, will it cause more sales? And if so, how much?
This is harder than it sounds. Sales might go up after a TV campaign for many reasons that have
nothing to do with the ads: maybe it was the holiday season, maybe competitors raised prices,
or maybe the economy improved. The challenge is separating the true effect of marketing
from everything else that happened at the same time.
The Restaurant Analogy
Imagine you manage a restaurant with five servers. Some bring in more tips than others.
But does that mean they're better servers? Maybe one server always works Saturday nights
(busy, generous diners), while another works Tuesday lunches (slow, frugal customers).
To know who's actually good, you'd need to compare: "How would tips change if I swapped
their shifts?" That's counterfactual thinking—comparing what happened
to what would have happened under different circumstances.
Marketing Mix Models (MMMs) try to answer this counterfactual question:
"What would sales have been without this marketing channel?"
The difference is the channel's true contribution.
The Core Concepts
2.1 Contribution vs. Correlation
Just because two things move together doesn't mean one causes the other. This is perhaps the
most important concept in marketing measurement.
Interactive: Spot the Spurious Correlation
Ice cream sales and drowning deaths are strongly correlated. Does ice cream cause drowning?
Of course not—summer causes both. This is what happens when you
ignore confounders.
With seasonality controlled: The apparent relationship between ice cream
and drowning disappears. The same principle applies to marketing—without proper
controls, media effects absorb the effects of everything that happened at the same time.
2.2 Counterfactual Analysis
A counterfactual is a "what if" scenario. To measure TV's effect, we ask:
"What would sales have been if we hadn't spent anything on TV, but everything else stayed the same?"
The difference between actual sales and this hypothetical "no TV" world is TV's
contribution.
Interactive: Counterfactual Contribution
Toggle channels on/off to see their counterfactual contribution to sales.
Reading this chart: The gray area shows baseline sales (what you'd get with
zero marketing). Each colored layer shows a channel's contribution. Total height = total sales.
Toggle channels off to see the counterfactual world without them.
2.3 Understanding Uncertainty
No measurement is perfect. When we say "TV has an ROI of 2.1", what we really mean is
"Our best estimate is 2.1, but it could plausibly be anywhere from 1.8 to 2.5."
This uncertainty range (called a credible interval in Bayesian statistics)
is not a weakness—it's a feature. It tells you how confident you can be in the estimate.
Narrow Interval = High Confidence
If ROI is estimated as 2.1 [2.0–2.2], the range is tight. You can make
decisions confidently because the true value is likely close to 2.1.
Wide Interval = Low Confidence
If ROI is estimated as 1.3 [0.6–2.0], the range is wide. The channel
might be great or might be unprofitable—you need more data or an experiment.
Why uncertainty is your friend
A wide credible interval doesn't mean the channel is bad—it means you don't have
enough information yet. This honesty prevents overconfident decisions. It's far better
to know you're uncertain than to falsely believe a precise but wrong number.
Reading Your Results
3.1 The Contribution Waterfall
A waterfall chart shows how different components build up to your total outcome.
Starting from a baseline, each bar adds (or subtracts) until you reach actual sales.
Reading this chart: Start from the left. Baseline is what you'd sell with no marketing
or seasonal effects. Trend adds long-term growth. Seasonality captures calendar patterns. Then each
channel adds its contribution. The final bar shows total sales.
Explore Component Breakdown
3.2 The ROI Forest Plot
A forest plot shows ROI estimates for all channels on one chart, making it
easy to compare effectiveness and spot which channels are confidently profitable.
Reading this chart: Each bar is a channel. The center point is the best estimate;
the error bar shows the 94% credible interval. The red dashed line at 1.0 is break-even
(ROI = 1 means you get back exactly what you spent). Channels entirely to the right of this line
are confidently profitable.
3.3 Probability Metrics
Beyond point estimates, you can ask probability questions:
P(ROI > 0): What's the probability this channel has any positive effect?
P(ROI > 1): What's the probability this channel is profitable (returns more than it costs)?
Select a Channel to Explore
P(ROI > 0)
99.8%
Probability of positive effect
P(ROI > 1)
97.2%
Probability of profitability
P(ROI > 1.5)
84.5%
Probability of strong ROI
3.4 The Posterior Distribution
The full posterior distribution shows all plausible values for a channel's ROI,
weighted by how likely each value is. This is the complete picture of your uncertainty.
Reading this chart: The curve shows the probability density—higher means more
likely. The shaded regions show different interpretations. The peak is the most likely value,
but all values under the curve are plausible.
Practical Guidance
Interpreting Your Results
Scenario
What It Means
Recommended Action
High ROI, Narrow Interval
Confidently effective channel
Safe to maintain or increase investment
High ROI, Wide Interval
Potentially effective but uncertain
Consider a controlled experiment (geo-test)
Low ROI, Narrow Interval
Confidently ineffective
Consider reducing or reallocating budget
Low ROI, Wide Interval
Possibly ineffective but uncertain
Gather more data before major changes
Common Pitfalls
Ignoring uncertainty: Don't make major budget decisions based on point estimates alone. Always consider the credible interval.
Forgetting the baseline: Much of your sales would happen without marketing. Contribution is the incremental effect, not total sales during campaign.
Confusing correlation with causation: Ensure your model controls for confounders (seasonality, promotions, economic factors).
Over-reacting to noise: Short-term fluctuations in ROI estimates are normal. Look for consistent patterns over multiple model runs.
Best Practices
Run sensitivity analysis to see if conclusions hold under different model specifications
For uncertain channels, consider geo-experiments to validate MMM findings
Update your model regularly (quarterly) as new data arrives
Present uncertainty honestly to stakeholders—it builds trust and leads to better decisions
Implementation in MMM Framework
Here's how to compute channel effectiveness metrics using the mmm-framework:
Computing Counterfactual Contributions
# After fitting your model...from mmm_framework.analysis import MMMAnalyzer
analyzer = MMMAnalyzer(fitted_model)
# Compute counterfactual contributions per channel
contributions = analyzer.compute_counterfactual_contributions(
compute_uncertainty=True,
hdi_prob=0.94# 94% credible interval
)
# View the summaryprint(contributions.summary())
# Output:# Channel Contribution Pct HDI_Low HDI_High# TV $2.45M 18.2% $1.89M $3.12M# Search $1.82M 13.5% $1.45M $2.24M# Social $0.95M 7.1% $0.42M $1.58M# Display $0.34M 2.5% -$0.21M $0.92M# Radio $0.78M 5.8% $0.41M $1.19M
Computing ROI with Uncertainty
from mmm_framework.reporting.helpers import compute_roi_with_uncertainty
# Compute ROI with full uncertainty quantification
roi_df = compute_roi_with_uncertainty(
model=fitted_model,
hdi_prob=0.94
)
# Key columns in roi_df:# - channel: Channel name# - roi_mean: Point estimate of ROI# - roi_hdi_low, roi_hdi_high: 94% credible interval# - prob_positive: P(ROI > 0)# - prob_profitable: P(ROI > 1)# Example output:print(roi_df[['channel', 'roi_mean', 'roi_hdi_low', 'roi_hdi_high', 'prob_profitable']])
# Channel roi_mean roi_hdi_low roi_hdi_high prob_profitable# TV 2.12 1.78 2.52 0.998# Search 2.41 2.08 2.81 0.999# Social 1.28 0.76 1.89 0.872# Display 0.68 0.28 1.12 0.284# Radio 1.62 1.28 2.01 0.991