Folder: Questions-and-Answers

Oct 13, 2026

Q: Does peeking (optional stopping) matter for a Bayesian? Reconcile the frequentist peeking problem, always-valid p-values / mSPRT and confidence sequences with the likelihood principle, Thompson sampling and sequential Bayesian experimental design.

Oct 13, 2026

Q: Exchangeability underlies permutation/randomization tests, conformal prediction and hierarchical priors. Which vault methods break when it fails (time series, covariate shift, interference, clustering), and what replaces it in each case?

Oct 13, 2026

Q: How should I handle multiple comparisons when selecting from hundreds of models?

Oct 13, 2026

Q: Are pre-registration, multiplicity control, hierarchical partial pooling, cross-fitting, honest trees and train/calibration splits all the same cure for the garden of forking paths?

Oct 13, 2026

Q: How should a geo-lift or A/B experiment result become a prior in a Bayesian media mix model, and what should happen when the experiment and the MMM disagree?

Sep 23, 2026

Q: Multi-armed and contextual bandits, Bayesian optimisation, sequential Bayesian experimental design / deep adaptive design, dynamic treatment regimes with Q- and A-learning, switchback experiments and RLHF all choose actions from accumulating data. Laid out on one map, what is each optimizing, what is the state, and what feedback does it assume?

Sep 23, 2026

Q: For calibrating an agent-based model, how do I choose among SMM, indirect inference, EMM, synthetic likelihood, ABC, history matching, genetic-algorithm calibration and neural posterior / likelihood / ratio estimation?

Sep 23, 2026

Q: What are some common pitfalls in statistical modeling a data scientist should be aware of?

Sep 23, 2026

Q: What would continuous learning look like in media measurement, given that media has interaction effects and learning all interactions is costly or needs more cells than available techniques support?

Sep 23, 2026

Q: What are some differences between frequentist and Bayesian statistics?

Sep 23, 2026

Q: How does advertising carryover (adstock) violate the assumptions of switchback experiments, always-valid sequential tests and geo tests, and what design changes fix it?

Sep 23, 2026

Q: In-context learning has been read as amortized (implicit) Bayesian inference. How does that reading compare with neural posterior estimation, variational autoencoders' amortized encoders, deep adaptive design and hierarchical models — and where does the analogy break?

Sep 23, 2026

Q: What changes in the ROAS / mROAS optimization if the outcome is customer lifetime value rather than sales, given that CLV is a model-based forecast observed with delay and censoring?

Sep 23, 2026

Q: Partial pooling shows up as hierarchical models, James–Stein / empirical Bayes shrinkage, global-local shrinkage priors, the Gamma-Gamma and NBD customer models, global forecasting models and LLM pretraining. What is the shared mechanism, and when does pooling hurt?

Sep 23, 2026

Q: Doubly-robust estimation appears in the vault as AIPW, the DML interactive-model score, the Callaway–Sant'Anna doubly-robust ATT(g,t), SDID's double robustness, the X-learner and weighted conformal prediction. What is the common structure, and in what sense is each one 'doubly' robust?

Sep 23, 2026

Q: The Kalman filter appears in the vault in three places — BSTS / CausalImpact, the linear-Gaussian state-space notes, and probabilistic ODE solvers (plus Gauss–Markov priors in probabilistic numerics). What is shared, what differs, and what does seeing them together buy?

Sep 23, 2026

Q: What are some ways to uncover causal estimates from non-experimental data?

Sep 23, 2026

Q: When can LLM 'silicon samples' stand in for consumer data or hand-written decision rules in an agent-based model, and what validation protocol is needed?

Sep 23, 2026

Q: S-, T-, X- and R-learners, causal forests / GRF, BART and Bayesian causal forests, moderation analysis, hierarchical models and conformal ITE intervals all address heterogeneous treatment effects. Which question does each actually answer, and what does its uncertainty interval mean?

Sep 18, 2026

Q: Honest DiD, Rosenbaum-style sensitivity analysis for unobserved confounding, Plausible GMM's prior over moment misspecification, synthetic-control placebo and backdating checks, prior/likelihood power-scaling sensitivity and Sobol global sensitivity indices all ask how wrong the assumptions can be before the conclusion changes. What would a unified view look like?

Sep 18, 2026

Q: Chinchilla compute-optimal training, optimal media mix under saturating response (ROAS / mROAS, Dorfman–Steiner), and power analysis / experimental design are all budget-allocation problems under diminishing returns. What transfers between them, and what does not?

Sep 18, 2026

Q: For one geo-test dataset, when would Time-Based Regression, CausalImpact (BSTS), synthetic control, generalized synthetic control, synthetic difference-in-differences and Bayesian DiD give different answers, and which assumption drives each difference?

Sep 18, 2026

Q: CUPED, regression adjustment, double/debiased ML residualization and CausalImpact covariates all 'adjust for other variables'. When is adjustment for precision, and when is it for identification — and what goes wrong when the two are confused?

Sep 18, 2026

Q: The vault uses 'calibration' in at least four senses — simulation-based calibration of a Bayesian computation, coverage calibration in conformal prediction, probabilistic calibration of forecasts under proper scoring rules, and parameter calibration of agent-based models (plus uncertainty calibration of probabilistic numerical solvers and LLM-population calibration). What does each one guarantee, and what does it not?

Sep 18, 2026

Q: How do the ELBO, the Barber–Agakov posterior bound and the marginal / VNMC bounds on expected information gain, the contrastive PCE and ACE bounds, neural ratio estimation and the forward-KL objective of neural posterior estimation relate? Which direction of KL does each use, is each an upper or lower bound, and what failure does that choice cause?

Sep 18, 2026

Index: Questions and Answers

type/index

Sep 18, 2026

Q: How can SMM be used to calibrate agent based models?

Jul 04, 2026

Q: When should continuous media learning use Bayesian experimental design vs Bayesian optimization vs a bandit?

Jul 04, 2026

Q: How would a geo-holdout experiment be encoded as a design ξ and its EIG computed against an MMM posterior?

Jul 01, 2026

Second Brain

Explorer

Folder: Questions-and-Answers

Q: Does peeking (optional stopping) matter for a Bayesian? Reconcile the frequentist peeking problem, always-valid p-values / mSPRT and confidence sequences with the likelihood principle, Thompson sampling and sequential Bayesian experimental design.

Q: Exchangeability underlies permutation/randomization tests, conformal prediction and hierarchical priors. Which vault methods break when it fails (time series, covariate shift, interference, clustering), and what replaces it in each case?

Q: How should I handle multiple comparisons when selecting from hundreds of models?

Q: Are pre-registration, multiplicity control, hierarchical partial pooling, cross-fitting, honest trees and train/calibration splits all the same cure for the garden of forking paths?

Q: How should a geo-lift or A/B experiment result become a prior in a Bayesian media mix model, and what should happen when the experiment and the MMM disagree?

Q: For calibrating an agent-based model, how do I choose among SMM, indirect inference, EMM, synthetic likelihood, ABC, history matching, genetic-algorithm calibration and neural posterior / likelihood / ratio estimation?

Q: What are some common pitfalls in statistical modeling a data scientist should be aware of?

Q: What would continuous learning look like in media measurement, given that media has interaction effects and learning all interactions is costly or needs more cells than available techniques support?

Q: What are some differences between frequentist and Bayesian statistics?

Q: How does advertising carryover (adstock) violate the assumptions of switchback experiments, always-valid sequential tests and geo tests, and what design changes fix it?

Q: In-context learning has been read as amortized (implicit) Bayesian inference. How does that reading compare with neural posterior estimation, variational autoencoders' amortized encoders, deep adaptive design and hierarchical models — and where does the analogy break?

Q: What changes in the ROAS / mROAS optimization if the outcome is customer lifetime value rather than sales, given that CLV is a model-based forecast observed with delay and censoring?

Q: Partial pooling shows up as hierarchical models, James–Stein / empirical Bayes shrinkage, global-local shrinkage priors, the Gamma-Gamma and NBD customer models, global forecasting models and LLM pretraining. What is the shared mechanism, and when does pooling hurt?

Q: Doubly-robust estimation appears in the vault as AIPW, the DML interactive-model score, the Callaway–Sant'Anna doubly-robust ATT(g,t), SDID's double robustness, the X-learner and weighted conformal prediction. What is the common structure, and in what sense is each one 'doubly' robust?

Q: The Kalman filter appears in the vault in three places — BSTS / CausalImpact, the linear-Gaussian state-space notes, and probabilistic ODE solvers (plus Gauss–Markov priors in probabilistic numerics). What is shared, what differs, and what does seeing them together buy?

Q: What are some ways to uncover causal estimates from non-experimental data?

Q: When can LLM 'silicon samples' stand in for consumer data or hand-written decision rules in an agent-based model, and what validation protocol is needed?

Q: S-, T-, X- and R-learners, causal forests / GRF, BART and Bayesian causal forests, moderation analysis, hierarchical models and conformal ITE intervals all address heterogeneous treatment effects. Which question does each actually answer, and what does its uncertainty interval mean?

Q: Chinchilla compute-optimal training, optimal media mix under saturating response (ROAS / mROAS, Dorfman–Steiner), and power analysis / experimental design are all budget-allocation problems under diminishing returns. What transfers between them, and what does not?

Q: For one geo-test dataset, when would Time-Based Regression, CausalImpact (BSTS), synthetic control, generalized synthetic control, synthetic difference-in-differences and Bayesian DiD give different answers, and which assumption drives each difference?

Q: CUPED, regression adjustment, double/debiased ML residualization and CausalImpact covariates all 'adjust for other variables'. When is adjustment for precision, and when is it for identification — and what goes wrong when the two are confused?

Index: Questions and Answers

Q: How can SMM be used to calibrate agent based models?

Q: When should continuous media learning use Bayesian experimental design vs Bayesian optimization vs a bandit?

Q: How would a geo-holdout experiment be encoded as a design ξ and its EIG computed against an MMM posterior?

Q: How do adstock/carryover dynamics interact with the timing of sequential media experiments (delayed outcomes)?