A vision for rigorous, actionable marketing measurement—built on honest uncertainty quantification, experimental validation, and the belief that better methodology leads to better decisions.
The marketing measurement industry is at an inflection point. Clients are asking harder questions: not just "what is the ROI of this channel" but "how confident should we be in that number?" They notice when last year's model says television was the top performer and this year's model says it's digital—with no change in strategy. They're beginning to ask about validation.
This framework represents a fundamental rethinking of how marketing measurement should work. It's built on the premise that honest uncertainty is more valuable than false precision, and that validated predictions matter more than impressive-looking outputs.
The industry is moving toward greater rigor. Organizations that lead this transition—rather than resist it—will build differentiated capabilities and client relationships grounded in demonstrated rather than asserted credibility.
Traditional marketing mix modeling often involves a practice known as specification shopping: iteratively adjusting model parameters—lags, decay rates, control variables—until results achieve desired statistical properties or match prior expectations. While this can incorporate genuine domain knowledge, it introduces systematic risks.
When you test multiple specifications and select based on results, you invalidate standard statistical inference. The reported confidence intervals don't reflect actual uncertainty. Worse, the process systematically selects for confirming rather than disconfirming evidence, creating models that look good but may be dangerously miscalibrated.
Common post-hoc adjustments—like zeroing out negative media effects—don't just violate statistical principles. They systematically bias results upward and make downstream optimization recommendations unreliable. When everyone uses the same biased methods, an entire industry can be confidently wrong.
A marketing mix model answers a causal question—"what would sales have been if we hadn't run this media?"—not just "what moved together?" Plenty of things move together without one causing the other: holiday demand lifts both ad spend and sales in the same weeks. Getting from correlation to contribution requires three disciplines, and the framework enforces all of them.
First, every variable has a declared role. Confounders—forces like underlying demand that drive both spend and sales—must be in the model, no exceptions, or media gets credit for sales it never caused. Precision controls—variables that explain noise without distorting media effects—are optional sharpeners. The framework keeps these roles explicit so the distinction can never be quietly blurred to make results look better.
Second, the model design is locked in before results are seen. Pre-specification removes the temptation to adjust the recipe after tasting the outcome, so findings reflect the data rather than the analyst's preferences.
Third, the model's answers are checked against real-world experiments. Think of it as a taste test for the recipe: the model proposes an effect, and a regional holdout test disposes. Lift-test results feed directly back into the model—calibration that observational data alone can never provide.
The product of these three disciplines is an estimate of incremental impact: what your media actually caused, stated with honest uncertainty.
Where feasible, design holdout experiments that test model predictions against reality. This creates a feedback loop distinguishing working models from non-working ones.
Instead of point estimates implying false precision, we quantify and communicate uncertainty. When confident, we say so. When not, we recommend experiments rather than papering over it with specification choices.
Different business questions require different tools. Attribution, incrementality, and optimization questions aren't all best answered by the same model. We match methodology to question.
Define modeling decisions before seeing results. This reduces researcher degrees of freedom and ensures that findings reflect data patterns rather than analyst choices.
"We estimate TV ROI at 1.4 (1.2–1.6, 80% CI). This estimate is validated against geo experiments and robust to specification choices."
"Display ROI estimates are highly uncertain, ranging from 0.5 to 2.5 across specifications. We cannot confidently recommend budget changes without additional data."
This kind of transparency builds trust. Clients can distinguish confident recommendations from uncertain ones. They can make informed decisions about where to act immediately versus where to invest in additional validation.
The framework is built directly on PyMC (6.x) for Bayesian modeling — a standalone inference engine that can optionally interoperate with PyMC-Marketing models for reporting, rather than subclassing or depending on them — with a complete technical stack: a Python library at the core, a FastAPI backend, and a modern React web application—Program, Experiments, and Performance views plus an AI agent workspace that acts as a chat-based analyst assistant (a legacy Streamlit interface remains as a secondary surface). See the platform overview for a tour. It supports sophisticated modeling scenarios including nested models with mediated causal pathways, multivariate outcomes with cross-correlations, and principled variable selection that maintains causal validity.
Key technical innovations include proper handling of geo-level random effects (which can't identify national media effects), Bayesian variable selection that distinguishes confounders from precision controls, and extensive diagnostics following the Bayesian workflow framework from Gelman et al. (2020).
The framework is open source under the Apache-2.0 license and designed for both individual use and organizational adoption. Comprehensive documentation, mathematical foundations, and educational content help teams understand not just how to use these methods but why they matter.
The framework is built and maintained by Matthew Reda—and honestly, just him. A solo-maintained project carries real bus-factor risk, and the same honesty this site asks of measurement applies to itself: the mitigations are the Apache-2.0 license (the codebase is fully fork-able), a test suite of 2,000+ passing tests, seeded notebooks behind every measured claim (so the evidence can be re-derived by anyone), a pressure-testing harness with a recorded public scorecard, and methodology stated in documentation rather than hidden in code. For the full procurement view—license, versioning, support model, and known limitations on one page—see the Evaluator's Guide.
Citing the framework: please cite it as MMM Framework, version 0.2.0, Matthew Reda, https://github.com/redam94/mmm-framework.
If you're working on similar problems—fighting specification shopping in your organization, building rigorous measurement practices, or exploring Bayesian approaches to marketing analytics—I'd love to hear from you.
Whether you have questions about the framework, want to discuss implementation challenges, or are interested in collaboration, don't hesitate to reach out.
Get in TouchOr open an issue on GitHub for technical questions and feature requests.
Explore the documentation, dive into the technical guide, or browse the code on GitHub.