Question Type 1

Understanding Channel Effectiveness

Separating real marketing impact from coincidence

What We're Actually Asking

When a marketer asks "Which channels work?", they're really asking a causal question: If I spend money on TV advertising, will it cause more sales? And if so, how much?

This is harder than it sounds. Sales might go up after a TV campaign for many reasons that have nothing to do with the ads: maybe it was the holiday season, maybe competitors raised prices, or maybe the economy improved. The challenge is separating the true effect of marketing from everything else that happened at the same time.

The Restaurant Analogy

Imagine you manage a restaurant with five servers. Some bring in more tips than others. But does that mean they're better servers? Maybe one server always works Saturday nights (busy, generous diners), while another works Tuesday lunches (slow, frugal customers).

To know who's actually good, you'd need to compare: "How would tips change if I swapped their shifts?" That's counterfactual thinking—comparing what happened to what would have happened under different circumstances.

Marketing Mix Models (MMMs) try to answer this counterfactual question: "What would sales have been without this marketing channel?" The difference is the channel's true contribution.

The Core Concepts

2.1 Contribution vs. Correlation

Just because two things move together doesn't mean one causes the other. This is perhaps the most important concept in marketing measurement.

Interactive: Spot the Spurious Correlation

Ice cream sales and drowning deaths are strongly correlated. Does ice cream cause drowning? Of course not—summer causes both. This is what happens when you ignore confounders.

With seasonality controlled: The apparent relationship between ice cream and drowning disappears. The same principle applies to marketing—without proper controls, media effects absorb the effects of everything that happened at the same time.

2.2 Counterfactual Analysis

A counterfactual is a "what if" scenario. To measure TV's effect, we ask: "What would sales have been if we hadn't spent anything on TV, but everything else stayed the same?"

The difference between actual sales and this hypothetical "no TV" world is TV's contribution.

Interactive: Counterfactual Contribution

Toggle channels on/off to see their counterfactual contribution to sales.

Reading this chart: The gray area shows baseline sales (what you'd get with zero marketing). Each colored layer shows a channel's contribution. Total height = total sales. Toggle channels off to see the counterfactual world without them.

2.3 Understanding Uncertainty

No measurement is perfect. When we say "TV has an ROI of 2.0", what we really mean is "Our best estimate is 2.0, but it could plausibly be anywhere from 1.6 to 2.6."

This uncertainty range (called a credible interval in Bayesian statistics) is not a weakness—it's a feature. It tells you how confident you can be in the estimate.

Narrow Interval = High Confidence

If ROI is estimated as 2.1 [2.0–2.2], the range is tight. You can make decisions confidently because the true value is likely close to 2.1.

Wide Interval = Low Confidence

If ROI is estimated as 1.3 [0.6–2.0], the range is wide. The channel might be great or might be unprofitable—you need more data or an experiment.

Why uncertainty is your friend

A wide credible interval doesn't mean the channel is bad—it means you don't have enough information yet. This honesty prevents overconfident decisions. It's far better to know you're uncertain than to falsely believe a precise but wrong number.

Reading Your Results

Before You Trust These Numbers

Contribution and ROI estimates inherit every flaw in the data and the model specification. Before acting on anything below, run the EDA pre-flight in the walkthrough (Part 1), confirm every variable plays the right causal role via variable selection, and know the silent failure modes catalogued in pressure testing. An unchecked model can put narrow intervals around the wrong answer.

3.1 The Contribution Waterfall

A waterfall chart shows how different components build up to your total outcome. Starting from a baseline, each bar adds (or subtracts) until you reach actual sales.

Reading this chart: Start from the left. Baseline is what you'd sell with no marketing or seasonal effects. Trend adds long-term growth. Seasonality captures calendar patterns. Then each channel adds its contribution. The final bar shows total sales.

Explore Component Breakdown

3.2 The ROI Forest Plot

A forest plot shows ROI estimates for all channels on one chart, so you can compare effectiveness at a glance and spot which channels are confidently profitable.

Reading this chart: Each bar is a channel. The center point is the best estimate; the error bar shows the 94% credible interval. The red dashed line at 1.0 is break-even (ROI = 1 means you get back exactly what you spent). Channels entirely to the right of this line are confidently profitable.

How Much Weight Can Each Estimate Bear?

Not every row of a forest plot deserves equal trust. The framework's confidence tiers, in descending order: validated (anchored by an experiment) > narrow-unvalidated > wide-unvalidated. Here is the same fictional dataset, tiered (Illustrative):

Channel ROI (94% interval) Evidence Tier
Paid Search 2.43 (1.93 – 2.99) Validated — calibrated against a geo-lift test
TV 2.04 (1.58 – 2.60) Narrow-unvalidated — precise, but never experimentally tested
Radio 1.95 (1.03 – 2.98) Wide-unvalidated — direction clear, magnitude not
Paid Social 1.27 (0.56 – 2.11) Wide-unvalidated — profitability uncertain; strong test candidate
Display 0.68 (−0.42 – 1.84) Wide-unvalidated — even the sign is uncertain; test before cutting or growing

3.3 Probability Metrics

Beyond point estimates, you can ask probability questions:

Select a Channel to Explore

P(ROI > 0)
99.9%
Probability of positive effect
P(ROI > 1)
99.9%
Probability of profitability
P(ROI > 1.5)
97.7%
Probability of strong ROI

3.4 The Posterior Distribution

The full posterior distribution shows all plausible values for a channel's ROI, weighted by how likely each value is. This is the complete picture of your uncertainty.

Reading this chart: The curve shows the probability density—higher means more likely. The shaded regions show different interpretations. The peak is the most likely value, but all values under the curve are plausible.

Practical Guidance

Interpreting Your Results

Scenario What It Means Recommended Action
High ROI, Narrow Interval Confidently effective channel Safe to maintain or increase investment
High ROI, Wide Interval Potentially effective but uncertain Consider a controlled experiment (geo-test)
Low ROI, Narrow Interval Confidently ineffective Consider reducing or reallocating budget
Low ROI, Wide Interval Possibly ineffective but uncertain Gather more data before major changes

Common Pitfalls

  • Ignoring uncertainty: Don't make major budget decisions based on point estimates alone. Always consider the credible interval.
  • Forgetting the baseline: Much of your sales would happen without marketing. Contribution is the incremental effect, not total sales during campaign.
  • Confusing correlation with causation: Ensure your model controls for confounders (seasonality, promotions, economic factors).
  • Over-reacting to noise: Short-term fluctuations in ROI estimates are normal. Look for consistent patterns over multiple model runs.

Best Practices

  • Run sensitivity analysis to see if conclusions hold under different model specifications
  • For uncertain channels, consider geo-experiments to validate MMM findings
  • Update your model regularly (quarterly) as new data arrives
  • Present uncertainty honestly to stakeholders—it builds trust and leads to better decisions

For practical guidance on communicating these results to stakeholders—turning credible intervals and contributions into language a CMO or media planner can act on—see Interpreting Model Results.

Implementation in MMM Framework

Here's how to compute channel effectiveness metrics using the mmm-framework:

Computing Counterfactual Contributions

# After fitting your model... from mmm_framework.analysis import MMMAnalyzer analyzer = MMMAnalyzer(fitted_model) # Compute counterfactual contributions per channel contributions = analyzer.compute_counterfactual_contributions( compute_uncertainty=True, hdi_prob=0.94 # 94% credible interval ) # View the summary ("Contribution %" is each channel's share of the # total media effect, so the column sums to 100%) print(contributions.summary()) # Output (Illustrative): # Channel Total Contribution Contribution % HDI 3% HDI 97% # TV $2.45M 38.6% $1.89M $3.12M # Search $1.82M 28.7% $1.45M $2.24M # Social $0.95M 15.0% $0.42M $1.58M # Display $0.34M 5.4% -$0.21M $0.92M # Radio $0.78M 12.3% $0.41M $1.19M

Computing ROI with Uncertainty

from mmm_framework.reporting.helpers import compute_roi_with_uncertainty # Compute ROI with full uncertainty quantification roi_df = compute_roi_with_uncertainty( model=fitted_model, hdi_prob=0.94 ) # Key columns in roi_df: # - channel: Channel name # - spend: Total channel spend (the ROI denominator) # - contribution_mean, contribution_hdi_low/high: Revenue contribution # - roi_mean: Point estimate of ROI (= contribution / spend) # - roi_hdi_low, roi_hdi_high: 94% credible interval # - prob_positive: P(ROI > 0) # - prob_profitable: P(ROI > 1) # Example output -- same fictional dataset as the contribution table # above, so each row is exactly that contribution divided by spend: print(roi_df[['channel', 'spend', 'roi_mean', 'roi_hdi_low', 'roi_hdi_high', 'prob_profitable']]) # Output (Illustrative): # channel spend roi_mean roi_hdi_low roi_hdi_high prob_profitable # TV $1.20M 2.04 1.58 2.60 0.999 # Search $0.75M 2.43 1.93 2.99 0.999 # Social $0.75M 1.27 0.56 2.11 0.745 # Display $0.50M 0.68 -0.42 1.84 0.298 # Radio $0.40M 1.95 1.03 2.98 0.966

Getting Component Decomposition

# Decompose total outcome into components. # This one is a method on the fitted model itself (not the analyzer). decomposition = fitted_model.compute_component_decomposition() # summary() returns one row per component: # - Base (Intercept): baseline demand # - Trend: long-term trajectory # - Seasonality: calendar patterns # - Media (Total): combined channel contributions # - Controls (Total): other factors print(decomposition.summary())