Frequently Asked Questions

Bayesian methods provide genuine uncertainty quantification. Traditional regression gives you confidence intervals that answer: "If I repeated this sampling process infinitely, 95% of intervals would contain the true value." That's a statement about a hypothetical procedure, not about your actual estimate.

Bayesian credible intervals answer a more useful question: "Given our data and prior knowledge, there's a 94% probability the true value lies in this range." This is a direct probability statement about the parameter—exactly what decision-makers need.

Additional advantages:

• Prior incorporation: External evidence (meta-analyses, experiments, domain expertise) can be formally included
• Natural regularization: Priors prevent overfitting without ad-hoc penalty terms
• Hierarchical modeling: Partial pooling across geographies improves sparse estimates
• Full posterior: You get distributions, not just point estimates—enabling decision analysis under uncertainty

Specification shopping (also called specification searching, data dredging, or p-hacking) occurs when analysts test multiple model specifications and selectively report the one that produces desired results—typically significant coefficients, expected signs, or "reasonable" ROIs.

The fundamental problem is that this invalidates all statistical inference. When you test 20 specifications and report the best one, your reported confidence interval dramatically understates true uncertainty. The "95% confidence" claim becomes meaningless.

This conflates two different things: not having an answer, and being honest about confidence in the answer. Clients won't accept "we don't know" as a final answer—agreed. But they should accept (and increasingly expect) honest uncertainty bounds.

This is not "we don't know"—it's a precise, actionable statement that acknowledges uncertainty while providing a clear recommendation. Compare this to the alternative:

The honest version is more valuable because it enables appropriate decision-making. It also protects the relationship: if the recommendation fails, you acknowledged the uncertainty upfront.

No. A negative coefficient is informative, not a mistake. It might indicate:

• The channel genuinely has no effect (or negative effect) in this context
• The effect is too small to detect with available data
• Measurement error is attenuating estimates toward zero
• Confounding is biasing estimates (e.g., media increases during low-demand periods)
• The model is misspecified in ways that obscure the true effect

The correct response is to investigate why the estimate is unexpected, not to iterate until it matches expectations. If you require the model to show advertising works, and adjust until it does, you cannot then cite the model as evidence that advertising works. The reasoning is circular.

Year-over-year instability usually reflects one of two things:

1. Genuine changes in media effectiveness (new creative, changed competitive environment, audience saturation)
2. Sensitivity to specification choices that differ between modeling cycles

The second is more common and more problematic. If different analysts, or the same analyst with slightly different judgment calls, produce substantially different results, then the "result" is an artifact of the process, not a property of reality.

This framework addresses instability by:

• Pre-specifying models before seeing results, reducing researcher degrees of freedom
• Reporting sensitivity analyses that show how results vary across reasonable specifications
• Using hierarchical priors that can pool information across time periods
• Quantifying uncertainty so that changes within credible intervals aren't over-interpreted

Causation—that is the whole point. A marketing mix model answers a causal question: "what would sales have been if we hadn't run this media?" Not just "what moved together?" Lots of things move together without one causing the other: ice cream sales and sunburns rise in the same weeks, and holiday demand lifts both ad spend and sales at the same time. A model that only tracks co-movement would hand your media credit for the holiday.

This framework is built to separate coincidence from contribution, in three ways:

• It accounts for confounders—outside forces like underlying demand, seasonality, and pricing that push spend and sales up together—so media only gets credit for what those forces don't explain.
• The model design is locked in before results are seen (pre-specification), so the answer can't be quietly reshaped until it looks the way someone hoped.
• Its answers are checked against real-world experiments, such as regional holdout tests, and those results are folded back into the model (experiment calibration)—a taste test for the recipe.

The result is an estimate of incremental impact—what your media actually caused. For the full story of how the framework reasons about cause and effect, see the causal inference guide.

This is perhaps the most important misconception to address. Statistics does not remove uncertainty—it quantifies uncertainty.

Before analysis, you don't know what TV's ROI is. After analysis, you still don't know for certain—but you have a principled range of plausible values, given your data and assumptions. That range might be narrow (high confidence) or wide (low confidence), but it's always a range, never a single "true" value.

This matters for decisions. If TV ROI is "1.4 ± 0.2" (narrow range), you can confidently increase TV spend. If it's "1.4 ± 1.2" (wide range), the same point estimate suggests a very different action—probably running an experiment before making major changes.

A credible interval gives you a direct probability statement about the parameter. When we report "TV ROI: 1.4 (1.1–1.8, 80% CI)", we're saying:

"Given our data and analysis, there is an 80% probability that TV's true ROI falls between 1.1 and 1.8."

This is useful because:

• If the interval is entirely above 1.0, we're confident the channel is profitable
• If the interval spans 1.0, we can't confidently say whether it's profitable
• The width of the interval tells you how precise the estimate is
• You can make probability statements: "There's a 90% chance ROI exceeds 1.2"

Uncertainty is valuable because it enables appropriate decision-making. Without it, you can't distinguish between situations that require different actions.

Consider two scenarios, both with the same point estimate:

If you only see "TV ROI: 1.4" without the interval, you'd take the same action in both cases—but that would be a mistake. Scenario B has a 25% probability that TV is actually unprofitable. Increasing spend based on that estimate is gambling, not optimization.

Uncertainty also creates opportunities:

• High-uncertainty parameters are opportunities for experiments—resolving the uncertainty through a geo-test may have high ROI itself
• Low-uncertainty parameters need less validation—you can act more quickly on robust estimates
• Portfolio effects matter—you can diversify across channels where you're uncertain

Experiments (geo-holdouts, incrementality tests) should be considered when:

• Uncertainty is high for a channel that matters. If credible intervals are wide and the channel represents significant spend, an experiment may have high ROI.
• You're considering a major budget change. Moving 20% of budget based on an estimate with wide uncertainty is risky; an experiment de-risks the decision.
• The model result is surprising. If the model says a historically strong channel is underperforming, validation provides confidence before acting.
• Stakeholders need conviction. Sometimes the issue isn't statistical uncertainty but organizational buy-in; experiments create shared evidence.

Experiments are typically not needed when:

• Credible intervals are narrow and entirely above/below decision thresholds
• The decision is easily reversible
• The channel is small relative to total spend
• An experiment is logistically infeasible or too expensive relative to the decision value

Different approaches make different tradeoffs. Here's how to think about the major distinctions—including the tools that earn checkmarks of their own:

To be clear: Meridian earns its checkmarks—it is a serious Bayesian geo-MMM, and on these three columns it is closer to this framework than to OLS. The remaining differences are in the operating loop around the model, covered honestly in the Robyn / Meridian / PyMC-Marketing comparison below.

Key questions to ask about any MMM methodology:

1. Do they report credible/confidence intervals? Point estimates alone are incomplete.
2. How are specifications chosen? If it's "we tried several and picked the best," that's specification shopping.
3. Is there validation? Do model predictions ever get tested against experiments?
4. How is domain knowledge incorporated? Formal priors vs. ad-hoc constraints make a big difference.
5. What happens with negative coefficients? If they're always "fixed," ask how and why.

No. Augur—the framework's React web app—covers the full measurement loop without code. Orrery (the home page) tracks where you are in the measurement cycle, Auspices prioritizes and manages real-world experiments, and Chronicle shows how estimates sharpen cycle over cycle. At the center is Oracle, the chat-aided modeling workspace—an analyst-assistant that validates your data, configures and fits models, analyzes results, plans experiments, and drafts client-ready reports through conversation. See the page-name guide below, or take the platform tour to see each surface.

The Python library underneath is there for analysts and data scientists who want direct control—custom priors, extended and bespoke model types, scripted workflows. And if you want a new model family without waiting on a redeploy, the Model Garden & Atelier lets you author, test, and publish one from the browser. The getting started guide walks through both paths: launching the web app and fitting a first model in code.

Augur is the application layer of the MMM Framework—the web app that runs the whole measurement loop in one place, from the first data check to calibrated budget decisions. (The framework is the open-source Python library; Augur is the platform built on top of it, so anything the app does, you can also script directly.) Each page is named for a piece of the augur's craft—reading the signs to decide before committing—with the plain-language job underneath:

One color language runs through every page to encode evidence quality: calibrated (experiment-backed), running (experiment in flight), model-only (observational), and stale (evidence has aged out). The full walkthrough—with a screenshot of each page—is the platform overview.

The Model Garden is a versioned, shareable library of model families that live beside the built-in MMM rather than inside a redeploy. An author writes a model once; every project in the organization can then load it by reference, fit it, read its estimands, and report on it through the same pipeline as the core model—no engineering ticket, no new release.

The Atelier is the in-browser IDE where those models are built and proven: a code editor, a docs tab, and a Jupyter-like Notebook for running cells against a live dataset, with an AI copilot that diagnoses errors and proposes fixes. Before a model can ship it must pass a nine-tier compatibility contract (static checks, build, fit, instance and trace structure, scaling, and read-op smoke tests, among others), and a governance surface tracks it from draft → tested → published. Untrusted model code only ever executes inside the managed kernel—sandboxed and isolated in hosted deployments.

The full authoring recipe, the compatibility tiers, and worked examples to fork are in the Model Garden & Atelier guide.

Yes. The same garden → fit → estimand → serialize → report pipeline now carries genuinely non-MMM Bayesian families—models with no channels, spend, or single KPI. Shipped worked examples include a confirmatory factor analysis (CFA), a latent class analysis (LCA), and a joint latent-factor MMM that estimates a latent construct (for example, "economic health") from many indicators in the same graph as the media model—so the factor's uncertainty propagates into the channel coefficients and closes the back-door where a common cause drives both spend and sales.

Each family declares its own config, likelihood, and named estimands, and the report adapts to match—a factor-analysis section for the latent models, the usual ROI and decomposition sections for the MMM, and both for the joint latent-factor model. They are authored and shared through the Model Garden; the authoring guide covers how.

All three are serious tools, and the honest answer differs by tool. Robyn (Meta) searches thousands of model candidates with an evolutionary optimizer and asks the analyst to pick from a Pareto front of finalists—a workflow where the selection step itself can quietly become specification shopping. This framework fits one pre-registered Bayesian model and reports its full posterior, so uncertainty reflects the data rather than which finalist was chosen.

Meridian (Google) is a capable Bayesian geo-MMM with genuine lift-test calibration. The difference is the operating loop around the model: this framework also prices which experiment to run next (expected information gain and expected value of information, in dollars), pre-registers the design, tracks the experiment lifecycle, and folds readouts back into the next fit automatically. It additionally supports mediation structures (TV driving search driving sales) and jointly modeled KPIs.

PyMC-Marketing is not a competitor—it targets the same Bayesian-MMM space, and this framework can interoperate with a PyMC-Marketing model (through a reporting extractor) while its own core is a standalone PyMC 6 engine. On top of that engine it adds causal guardrails (declared variable roles, refutation checks), the experiment planning loop, the Augur web platform with its Oracle analyst workspace, and a Model Garden of versioned, shareable model families (including non-MMM ones).

Two claims you can verify rather than take on faith: model designs are locked before results are seen, and the methodology is pressure-tested in public against synthetic markets with known ground truth—including the scenarios where it struggles.

This framework is designed from the ground up to enable honest measurement:

• Full Bayesian inference via PyMC provides genuine posterior distributions, not just point estimates. Every parameter has a complete uncertainty distribution.
• Structured prior specification through configuration objects means priors are declared before fitting, not adjusted after seeing results.
• Comprehensive diagnostics (trace plots, R-hat, ESS, posterior predictive checks) ensure the inference is reliable before results are interpreted.
• Contribution uncertainty propagates coefficient uncertainty through to business metrics (ROI, contributions) rather than reporting only point estimates.
• Variable selection methods (horseshoe, spike-and-slab) handle uncertainty about which controls to include, rather than requiring manual specification shopping.

# The framework computes full posterior distributions for business metrics
contrib = model.compute_counterfactual_contributions(
    compute_uncertainty=True, hdi_prob=0.94
)

# Per-channel contribution with uncertainty bounds
print(contrib.summary())  # contribution, share, HDI low/high per channel

# ROAS with its interval: contribution / spend
tv_roas = contrib.total_contributions["TV"] / tv_spend_total
print(f"TV ROAS: {tv_roas:.2f} "
      f"[{contrib.contribution_hdi_low['TV'] / tv_spend_total:.2f}, "
      f"{contrib.contribution_hdi_high['TV'] / tv_spend_total:.2f}]")

Priors encode what we believe before seeing the data. In marketing, we typically have genuine prior knowledge:

• Media elasticities are usually small (0.01–0.3 based on meta-analyses)
• Effects are generally positive (advertising shouldn't decrease sales)
• Adstock decay rates are bounded (effects don't last forever)

This information is valuable and should be used. The alternative—pretending we know nothing—is just as much a choice, and often a worse one.

The key difference from specification shopping is that priors are declared before seeing results. You can't "prior shop" in the same way because you commit to priors before fitting. The framework's configuration-based approach enforces this discipline.

The framework supports multiple forms of validation:

• Prior predictive checks: Before fitting, simulate data from your priors to verify they imply reasonable data distributions.
• Posterior predictive checks: After fitting, simulate data from the posterior and compare to actual observations. Systematic discrepancies indicate model misspecification.
• Out-of-sample prediction: Hold out recent periods and evaluate predictive accuracy on unseen data.
• Experimental calibration: Compare model-predicted effects to geo-experiment results when available.

# Generate posterior predictive samples
pred = model.predict()  # PredictionResults with full posterior predictive

# Compare to observed data
az.plot_ppc(pred.posterior_predictive)

Validation doesn't prevent specification shopping, but it creates accountability: if a specification-shopped model makes predictions that fail validation, you'll know.

Effective uncertainty visualization is critical for stakeholder communication. The framework provides:

• Response curves with uncertainty bands: Show saturation curves with 80% and 94% credible intervals, so stakeholders see where we're confident vs. uncertain.
• Contribution waterfall charts: Decompose sales into components with error bars showing contribution uncertainty.
• ROI forest plots: Display channel ROIs with credible intervals, making it easy to see which channels are confidently above/below profitability thresholds.
• Prior vs. posterior comparisons: Show how much the data moved beliefs from the prior, indicating data informativeness.

Apache-2.0. The LICENSE file sits at the repository root. It's a permissive license: commercial use, modification, redistribution, and forking are all allowed, with an explicit patent grant. There is no dual-license scheme and no proprietary tier holding features back.

From PyPI. Pin an exact version for production use:

pip install mmm-framework==0.2.0

Source installs (git clone + uv sync) remain the development path.

Python 3.12+ is required. The getting started guide walks through setup, and the changelog page explains the pre-1.0 versioning policy and which modules are safe to depend on.

Measured, not estimated: a production-size national fit (156 weeks × 7 channels, NumPyro, 4 chains × 500 draws) completes in ~15 seconds on an Apple M3 laptop (PyMC 6), including compilation. Geo panels scale sub-linearly per row, and the extension models stay in the same runtime ballpark as the core model. The full benchmark table—with the caveat that absolute seconds are machine-specific and the ratios are the transferable part—is in the technical guide's Computational Scaling section.

Weekly data in the MFF (Master Flat File) long format, loaded and validated by MFFLoader. 104+ weeks of weekly history is the comfortable starting point—enough to separate seasonality from media effects. The data requirements page covers granularity, history length, channel coverage, and what to do when your data falls short.

Honestly: GitHub issues, with no SLA. There is no commercial support contract currently offered, and the project has a single maintainer. If your procurement process requires a vendor organization, this project does not currently meet that bar; if it requires software your team can own and audit, the Apache-2.0 license and the 2,000+ test suite are designed for exactly that. The Evaluator's Guide lays out the full risk picture, including the bus-factor mitigations.

Don't switch—run in parallel first. The recommended path is to fit the Bayesian MMM alongside your incumbent OLS model and only decommission the legacy model after two full modeling cycles of comparable answers under audit. That gives stakeholders a side-by-side record of where the two approaches agree, where they diverge, and why—rather than asking them to trust a discontinuous jump in methodology. See the OLS entry in the glossary for the framework's stance.