Two prediction models and two residuals can pin down an effect very precisely. Which effect you get depends on a causal graph the data can't show you. This page explains what those methods buy, what they assume, and why experiments and independent lines of evidence have to carry the rest.
Each of these six graphs links every pair of variables. So none of them predicts a pattern the data could rule out. Statisticians call a group like this a Markov equivalence class. Any data on demand, spend and revenue fits all six equally well, so picking one is an assumption you make. The plot on the right shows the same customers the way the selected graph says to estimate the effect. For graph 1, that means removing demand from both spend and revenue first. For the other graphs, it means using spend and revenue as recorded.
Every estimator on this page answers a question about the data. Assumptions decide whether that answer is the causal effect. The data can't check those assumptions.
Dana runs measurement for an online retailer. This year Finance wants two answers from her. She can only get either one by making assumptions she has no way to test.
Built on outcomes nobody sees. Y(0) is missing for every exposed customer.
Something you can measure from (X, D, Y). More data makes it more precise.
Parts II and III show how to compute this well. This one is for the ad campaign.
Identification is the first arrow. Everything after Part I is about the second. In this page's simulations we built the world ourselves and left nothing out of X, so the first arrow holds. With real data, it holds only if the assumptions do.
Assumes two things. Once you account for demand X, what's left of each week's budget has nothing to do with revenue. And the effect is the same every week.
Gives the effect as a ratio of averages Dana can measure. Nothing in it depends on an outcome she can't see.
Assumes three things. The platform targeted only on data Dana also has. Every kind of exposed customer has similar customers who weren't exposed. And seeing the ads means the same thing for everyone, with no spillover between customers.
Gives the missing piece, E[Y(0) | D = 1], as the conversion rate of unexposed customers, reweighted to match the exposed group.
Spend: once you account for demand, the rest of each week's budget is as good as random. Ads: the platform used nothing to target that Dana can't see. You can run a placebo check, such as conversions in the weeks before launch, which should show no lift. That check carries an assumption too. It only works if the hidden signal would also have moved those earlier conversions.
The exposed group mixes people who saw one ad with people who saw thirty. People in the same household also share screens. Both make it less clear what the effect measures and who it applies to.
The propensity is the chance a customer was exposed. You can spot estimated propensities near 0 or 1. You only see them for the features you have, though, and only through a model. A customer type the platform never skips looks the same in the data as bad luck.
The spend model assumes one θ for every week. Double ML lets the shapes of g and m bend freely. The effect itself stays a single number. If the effect really varies from week to week, θ becomes a weighted average, and you should know which weeks count most.
Both results come down to one instruction. Compare customers, or weeks, that look alike on X. Neither result says how to do that. When X has 140 columns and curved relationships, simple models get the comparison wrong, and Parts II and III cover how. Before that, look at how much the assumptions are carrying. Nothing in the data can confirm them.
Start with what this campaign's data say about the effect when you assume nothing. Then add one assumption at a time and watch the range shrink.
The exposed group's rate without ads could be anything from 0% to 100%, so the effect could be anything from −66 to +34 points.
Called monotone response, this rules out ads that hurt sales. That's reasonable for most ads. It's wrong for the ones people find annoying.
Called monotone selection, this says hidden targeting leans the same way as the targeting Dana can see. If so, her estimate becomes an upper limit.
Nothing outside X drives both who saw the ads and who bought. Every estimator in Parts II and III needs this to give a single number.
Effect on conversion among exposed customers, in percentage points. The rows follow Manski (1990) and Manski & Pepper (2000) and use only this campaign's data. Each range is estimated from the sample. The last row also shows its 95% interval.
Read the ladder from the top. On their own, the data can't even tell Dana whether the campaign helped. The first assumption rules out harm. The second caps the effect at her estimate. Only the last gives a single number. Each step down is a claim about customers she can't observe.
The CRM has dozens of columns, and someone always suggests giving all of them to a flexible learner. The learner can't tell good columns from bad. Whether a column belongs in X depends on where it sits in the causal graph. How well it predicts tells you nothing about that.
The opening animation showed two of these cases. Graph 2 treated demand as a mediator, and graph 5 treated it as a collider.
Intent drives both exposure and conversion, which makes it a confounder. It belongs in the model. Leave it out, and part of the platform's targeting gets counted as lift.
The platform targets both recent and lapsed buyers, so recency shapes who sees the ads. Leave it out, and the platform's choices get counted as lift.
Ads drive visits, and visits drive orders. That makes visits a mediator. Adjust for them, and you remove the part of the effect that runs through visits. Here that's most of it.
Both exposure and conversion drive loyalty activity, which makes it a collider. Adjusting for it links the two through a path that isn't causal. The ads end up looking harmful.
Inventory changes who gets exposed and nothing else, which makes it an instrument. It adds no bias here. It does push propensities toward 0 and 1, so the estimates spread out. If a hidden confounder exists, it also makes that bias bigger.
These fields connect to nothing, and they do little harm in this model. A flexible learner would pay for them with some extra noise and more room to overfit.
Sessions and loyalty activity predict conversion better than any other column in the file. A feature-importance plot would rank them first. Any variable selection that scores prediction would keep them. Yet the ads cause both, so including them wrecks the estimate. You only know to drop them if you know the order of events and the causal graph.
Suppose you have the right columns and the other assumptions hold. You still have to estimate the effect from a finite, messy dataset. That sounds like the easy part. The next two parts show how simple models get it wrong, and why.
Part I turned the spend question into a formula the data can answer. Now we have to compute it. When demand X is curved and has many columns, the obvious methods fail, and each fails for a reason you can name.
Start with the first question. Every week the media team sets a paid social budget, and every week revenue moves. Finance is about to lock next year's plan. They want one figure from Dana: how much revenue does one more dollar of spend bring in?
If she overstates it, the company spends money on a channel that won't pay it back. If she understates it, they cut something that works. Nobody gets to rerun the year.
The problem is how the budget gets set. The team spends more in weeks they expect to be busy, like holidays, paydays and warm weekends. Those weeks bring in more revenue anyway. We call these conditions demand, written X.
Each extra dollar of spend brings in fifty cents of revenue. We simulate the data, so we can grade every method against this number.
Plus noise v ~ N(0, 0.8²). This is the part of each week's budget that demand doesn't explain.
It's curved, and it peaks where spend peaks. That overlap causes all the trouble below.
Dana starts where most people start. She fits a straight line of revenue against spend. The slope comes back steep, with a standard error of 0.04. It looks precise, and it's wrong.
Color each week by demand and you can see why. Slow weeks sit at the bottom left and busy weeks at the top right. Most of the slope comes from demand, and the line credits it to spend.
The fix seems clear: control for demand X. But its effect is curved, and in the real data it has 140 columns.
A straight-line control would miss the shape of g. So Dana uses a flexible machine learning model, a penalized regression on many demand features, tuned to predict well. She fits one model of revenue on spend plus demand. Then she reads θ off the spend coefficient.
The answer drops to 0.64. That's better. To check, she runs the same steps on 300 new datasets. The estimates cluster around 0.64, not 0.50. Her 95% intervals contain the truth only 4% of the time.
The penalty causes the bias. The penalty, called regularization, helps the model predict well by shrinking its estimates. So ĝ flattens the peaks of g. Those peaks fall in the weeks when spend is high. The spend coefficient picks up the revenue that ĝ missed.
Frisch and Waugh used this idea in 1933, and Robinson extended it to flexible models in 1988. You give machine learning two prediction tasks it's good at. You never ask it for θ directly.
Spend runs across and revenue runs up. The fitted slope is 1.42, the same wrong answer as before.
Indigo marks a slow week and yellow a busy one. Demand sorts the weeks along both axes at once, which creates most of the diagonal.
The cloud flattens vertically, and the slope now falls too far, to 0.17. Spend still moves with demand, but revenue no longer does. So the two stop lining up.
Now the cloud narrows sideways. What's left of spend is the part demand can't predict, the budget surprises. The colors mix because demand no longer sorts the weeks.
The slope settles at 0.49. This uses the same model and penalty as Attempt 2. The difference is how it's used. It makes two predictions and never estimates the spend effect itself.
Across 300 datasets, the estimates center on the truth. The 95% intervals contain it about 95% of the time, as they should.
The model still uses the same penalty. So ℓ and m are shrunk and wrong in the same way ĝ was. This time the errors don't reach the estimate.
Statisticians call this property Neyman orthogonality. An error in one model only matters where it lines up with an error in the other, so the bias depends on the two errors multiplied together. Two small errors make a much smaller product. Part III draws this as a picture.
The method works in simulation, so Dana tries it on real data. There the ridge model looks too stiff. She swaps the revenue model for a much more flexible one.
The new model fits very well. Its R² on the data it trained on climbs from 0.84 to 0.92. Then the estimate drops to 0.25, half the truth. Finance would conclude that paid social barely works and cut it.
The new model averages each week with its nearest neighbour. On its training data, each week counts as one of its own two neighbours. Deep random forests and long boosting runs do something similar. Any model that nearly memorizes its training data has this problem to some degree.
Take one week. Its revenue has three parts: what demand predicts, the effect of that week's unplanned spend θv, and noise. A model trained on that same week has already seen its revenue. So its prediction absorbs part of the effect. When you subtract the prediction, you subtract part of the effect you wanted to measure.
Split the weeks into five groups, called folds. Train both helper models on four folds and compute residuals on the fifth. Rotate until every week has been scored by a model that never saw it.
Each residual now comes from a model that never saw that week's noise. So the problem term averages to zero again. No data goes to waste. Every week helps train the models four times and gets scored once. All 500 residuals then go into one residual-on-residual regression.
This is the same memorizing model as in Attempt 3, on the same data. The only change is which weeks each prediction could learn from.
Each card shows where one method landed across 300 datasets, compared with the truth of 0.50. It also shows how often its 95% interval actually contained the truth.
95% interval 0.37 to 0.65. Each extra dollar of paid social brings back about fifty cents in revenue.
The first regression would have led Finance to nearly triple the budget for a channel that loses money on each extra dollar. The flexible model without cross-fitting would have led them to cut it. The number they get now is the one the data support, as long as the assumption from Part I holds.
The scorecard also shows something less flattering. With the stiff ridge model, partialling out worked fine without cross-fitting. Cross-fitting is what lets you use any model you like on 140 messy columns. That freedom is why you would reach for machine learning in the first place.
The campaign is over. Some customers saw the ads, most didn't, and the platform picked who. Matching, weighting and outcome models each get one try at guessing what would have happened without the ads. A doubly robust estimate gets two.
A few months after the paid social work, Dana gets a harder question. The lifecycle team ran a six-week win-back campaign on connected TV and paid social, aimed at existing customers. Nobody held back a control group, because the media plan had no room for one. The platform decided on its own which customers saw the ads.
The platform's dashboard says customers who saw the ads converted at 34.3%, against 11.0% for those who didn't. That's a lift of +213%. The brand team wants to triple the next flight because of it. Before the money moves, Dana has to tell the CMO what the campaign really did.
For each of 5,000 customers, she has two fields from the CRM. One is a purchase-intent score from browsing. The other is the number of weeks since their last order. She doesn't have the platform's targeting rule, and nobody outside the platform does.
Higher intent means more ads. Recency has a U-shape, because the platform retargets recent buyers and chases lapsed ones.
Customers at both ends of the recency range buy more on their own. That's the same U the platform targets.
The ads add 0.35 + 0.15·intent to the log-odds of buying. For the customers who saw them, that's a lift of +24%.
The dashboard compares everyone who saw an ad with everyone who didn't. But the platform didn't pick at random. It went after high-intent customers and customers at both ends of the recency range. Those people buy a lot whether or not they see an ad.
This is a simulation, so we know what these customers would have done without the campaign. They would have converted at 28.5%. The campaign added 5.9 points to that. The dashboard gives it credit for 23.4 points.
Dana knows the dashboard number is too high. She needs to know by how much, and she needs an answer she can defend.
Matching on the propensity score is the standard fix, and Dana does it by the book.
She fits a logistic model for the chance of seeing the ads, using intent and recency. Then she pairs each of the 1,578 exposed customers with the unexposed customer whose score is closest. Some unexposed customers get used more than once. The balance table looks perfect. The gap in intent drops from 0.74 to 0.00, and recency was already balanced.
The estimate falls to 17.9 points, a lift of +109%. That's half the dashboard's number, backed by a clean balance table. She's tempted to send it.
Then she plots the matched pairs. They're the red lines in the animation above. Many pairs join someone who ordered last week with someone who hasn't ordered in a year. They have the same intent and the same score, but opposite recency. Her model treats recency as a straight line, so customers at either extreme look average to it. The balance table only checks average recency, and the two extremes cancel out.
Matching is a form of weighting. Write out the matched estimate and the pairs drop away.
Every look-back analysis like this needs a model. Weighting models who saw the ads. An outcome model predicts who buys. If you use only one and it's wrong, so is your answer.
Each unexposed customer is scaled by ê/(1 − ê), the odds that a customer like them saw the ads. As the weights grow, the unexposed group takes on the exposed group's U-shape in recency (side panel). Their conversion rate climbs toward what exposed customers would have done without ads.
Even correct weights can cause trouble. Some exposed customers had a better than 95% chance of seeing the campaign. The platform almost never skipped people like them. So their unexposed look-alikes are rare, and each one gets a weight of 19 or more.
In this file, 81 exposed customers have a propensity above ê = 0.9. Only 4 unexposed customers do.
Show this overlap plot with every lift estimate. You can drop customers above a propensity cutoff, or only accept close matches. Both steady the estimate, but they change the question. You're now measuring lift only among exposed customers who had real look-alikes, so say that when you report it. The doubly robust estimator softens extreme weights a little, because the weights only multiply model errors. No method can create comparisons the data never had.
This gives unexposed customers more or less weight until they look like the exposed ones. It works when the propensity ê is right. It ignores everything about who buys. Matching is a version of this.
This learns who buys from unexposed customers, then predicts the same for exposed ones. It works when μ̂₀ is right. Where there are no unexposed look-alikes, it guesses, and it won't warn you.
A model is right here if it includes the recency² term. Each dot is one campaign's estimate. Green rows center on the truth. Red rows miss it.
In three of the four worlds, the doubly robust estimate lands on the truth. The fourth is Dana's world. Both her models treat recency as a straight line, and both errors push the same way. Her estimate comes back at 17.1 points. Doubly robust gives you two chances to get a model right. If both are wrong, it can't save you.
Work out how far the doubly robust estimate can be off, and you get the propensity error multiplied by the outcome error.
The weighting map has vertical stripes because its error depends only on the propensity model. The outcome model map has horizontal stripes for the same reason. The doubly robust map is clear along both axes. Near the corner where both models are right, it stays small in every direction, because it grows with the product of the two errors.
This is the same orthogonality as in Part II, now with a yes-or-no treatment. The doubly robust formula for the effect on exposed customers (the ATT) is Neyman orthogonal. So the two models' errors only matter through their product. Two flexible models that each improve at a rate of n−1/4 are enough. Cross-fit them, and you get a √n-accurate estimate with an interval you can trust.
Dana can't see the platform's rule, so she stops trying to write it down. She swaps both logistic models for flexible ones and cross-fits them, just as in Part II.
Each model is now a flexible logistic regression that can bend with each feature, using nine smooth curves per feature. A gradient-boosted classifier would work just as well. Nobody tells either model about the U-shape in recency. Both find it in the data. Each customer's ê and μ̂₀ come from models trained on the other four folds. That way, no customer's own conversion leaks into their own residual.
She reruns the check on simulated campaigns. This time the estimates land on the truth. The 95% intervals contain the true effect 95% of the time, just as they should.
Her matching work isn't wasted. She can keep the pairs and subtract each pair's predicted difference, μ̂₀(Xᵢ) − μ̂₀(Xⱼ). This is Abadie and Imbens's bias-corrected matching. It's also doubly robust, with the match counts acting as weights. With the flexible models, it gives 6.8 points on this campaign.
95% interval +9% to +42%. That's 7.0 extra conversions for every 100 exposed customers.
The interval is wide. A ratio needs a lot of data, and 1,578 exposed customers isn't much. Even so, the whole interval sits far below the dashboard's +213%. It also sits well below the +109% that the matching analysis almost reported.
Dana takes this number to the CMO, with the overlap plot attached. The campaign works, at a fraction of what the dashboard claimed. That holds as long as the platform didn't target on something Dana can't see. That condition is Part I's assumption, and the last section tests how much it matters.
import numpy as np from sklearn.model_selection import KFold from sklearn.ensemble import HistGradientBoostingClassifier as HGBC # X: (n, p) pre-exposure customer features d: 1 if exposed y: 1 if converted n = len(y); e_hat = np.zeros(n); m0_hat = np.zeros(n) for train, test in KFold(5, shuffle=True, random_state=0).split(X): e_hat[test] = HGBC().fit(X[train], d[train]).predict_proba(X[test])[:, 1] ctl = train[d[train] == 0] # outcome model: unexposed only m0_hat[test] = HGBC().fit(X[ctl], y[ctl]).predict_proba(X[test])[:, 1] e_hat = np.clip(e_hat, 0.005, 0.995) w, r, t, c = e_hat / (1 - e_hat), y - m0_hat, d == 1, d == 0 t1 = r[t].mean() t2 = np.sum(w[c] * r[c]) / np.sum(w[c]) att = t1 - t2 # doubly robust ATT p, q = t.mean(), np.mean(w * c) phi = np.where(t, (r - t1) / p, -w * (r - t2) / q) # influence function base = y[t].mean() - att # exposed, had they not seen ads phi_c = np.where(t, (y - y[t].mean()) / p, 0) - phi phi_l = phi / base - att * phi_c / base**2 # delta method for the ratio se, lift_se = phi.std() / np.sqrt(n), phi_l.std() / np.sqrt(n) print(f"ATT {att:.3f} ± {1.96*se:.3f} lift {att/base:+.0%} ± {1.96*lift_se:.0%}")
DoubleML's DoubleMLIRM(score="ATTE") runs this estimator with cross-fitting and repeated splits. EconML's DRLearner fits the lift-by-segment curve.
The ATT asks what the campaign did for the people who saw it. It needs only μ₀ and comparable unexposed customers for each exposed one. The ATE asks what would happen if everyone saw the ads. It needs both outcome models and comparable customers on both sides.
Choose how each model handles recency and how sharply the platform targets. Then see which estimates hold up.
Flexible models use five-fold cross-fitting, so they take a few seconds longer.
Changes how strongly the platform targets. Sharper targeting leaves fewer comparable customers and makes the weights swing more.
Take the ad question. The CFO asks how Dana knows the unexposed customers are a fair comparison. She shows balance tables and overlap plots. She can estimate the number as precisely as she likes. He asks again. What if the platform knew something she doesn't, like which customers had been browsing its app that week?
She can't rule it out. No amount of data from this campaign would let her. She needs to know how exposed customers would have converted without ads, and that is missing for every one of them. The table shows ten customers. Drag the slider to give exposed customers a hidden head start. Their hidden column changes. Every column Dana can see stays exactly the same.
This is what untestable means. Two worlds can produce the same (X, D, Y) and still have different effects. Any test you run on (X, D, Y) gives the same result in both. More customers won't help, and neither will a better model.
| Intent | Recency | Saw ads | Converted | Would have converted without ads |
|---|
No one can prove the key assumption. So once you have the estimate, show how far that assumption would have to break before the decision changes.
Call δ the hidden head start. It measures how much more likely exposed customers were to buy anyway, compared with unexposed customers who share their intent and recency. If nothing is hidden, δ = 0. For any other value, subtract δ from the estimate to get the effect.
The interval touches zero at a head start of 3.3 points. The estimate itself hits zero at 7.0. To judge whether those numbers are big, compare them with signals Dana did measure. Dropping intent from both models moves the estimate by 8.0 points. Dropping recency moves it by 11.3.
That gives Dana one sentence for the deck. The campaign lifted conversions by about +26%. It stays clear of zero unless the platform's hidden targeting created more than about 40% of the head start that intent did. The CFO may doubt that premise, but now he can see it and argue with it.
We kept this dial simple on purpose. It assumes the same gap in conversion rate for every customer. More formal tools work the same way. Examples are Rosenbaum bounds for matched designs, Cinelli and Hazlett's partial-R² benchmarks, and the omitted-variable bounds for debiased ML by Chernozhukov and coauthors. Each one names the assumption, turns it into a dial, and compares the dial with what you measured. That comparison is still a judgment call. Nothing guarantees a hidden signal behaves like a measured one.
These tools make the most of one dataset. They cannot tell you whether the dataset holds the causal answer. For that you need evidence from outside the sample, which is the subject of Part IV.
Everything so far drew a causal answer out of one dataset, and that always needs an assumption the dataset can't check. An experiment removes the need for it. Hold out a random slice of the customers the platform picked, and the comparison is fair by construction.
In Part I, six causal graphs fit the same data. A coin flip leaves one. Nothing about a customer can cause a coin to land heads, so nothing can confound the comparison.
The held-out customers are the missing half from Part I, made visible. The platform picked them for the same reasons it picked everyone else, hidden reasons included. Then chance kept the ads away.
So their conversion rate is a direct measurement of what exposed customers would have done without ads. You need no propensity model, no outcome model and no list of confounders. Because the holdout comes from the customers the platform picked, it measures the same thing as Part III: the effect on exposed customers.
An experiment answers one question, about one campaign, for one audience, at one point in time. And it costs something: every held-out customer is a sale you may have given up.
Small holdouts give wide ranges. Experiments can also break in practice. Held-out customers see the ads on another device, or the platform delivers differently when a test is running. An experiment replaces an untestable assumption with ones you can inspect. It does not remove the need to think.
No single method earns full trust. What earns it is agreement between methods that would fail for different reasons. Turn up the platform's hidden signal and watch which lines move.
Something the platform uses to pick customers that also predicts who buys. It is not in the CRM, so no model can adjust for it.
Share of the picked customers who are randomly kept from seeing the ads.
The placebo test only works if whatever is hidden also moved sales before the ads ran.
Uses every customer, so it is precise.
Runs the same analysis on sales from before the campaign. Ads can't change the past, so the answer should be zero.
Subtracts the placebo result from the estimate, on the logic that the same bias shows up in both.
Compares picked customers who saw the ads with picked customers who were randomly held back.
Read the chart by asking which lines agree. When the model-based estimate and the experiment agree, you have two independent reasons to believe the number, and the tighter one tells you how big it is. When they disagree, the pattern of disagreement points at the assumption that broke.
The lines also help each other. A small holdout is honest but vague. A doubly robust estimate is sharp but may be off. A holdout that confirms the sharp estimate lets you keep its precision. A holdout that contradicts it tells you how far off the model is, and that correction carries over to the next campaign's model.
Most audience work looks for customers who are likely to buy. Those customers are easy to find and they make every dashboard glow. Many of them were going to buy anyway.
Dana's team builds a model of who is likely to buy and hands the top fifth to the platform. The launch looks like a triumph. The audience converts at 57%, far above the file average of 22%, and the cost per order is the lowest the team has seen.
Now look at what the ads did. Without any ads, this audience would have converted at 53%. The campaign added 4 points. Each sale the ads caused cost $73, more than twice what it would cost to advertise to customers picked at random.
No dashboard metric after launch can catch this. Conversion rate, cost per order and return on ad spend all count sales that sit next to the ads. A likely-buyer audience is built to put sales next to the ads. The better the audience model predicts purchases, the better these numbers look, whether or not the ads change anything.
This is Part III's problem moved one step earlier. There, the platform's targeting made exposed customers look good. Here, your own targeting does it, and it is harder to see because the selection was the goal.
The audience worth paying for is the one the ads change. Those customers sit at the top of the chart: a modest chance of buying without ads, and a much higher chance with them. Finding them takes a model of the effect, like the DR-learner below, trained on data where exposure varied for reasons you understand.
And only a holdout can confirm it. Hold back a random slice of any audience before launch. The gap between the slice that saw ads and the slice that didn't is the one number on this chart that a dashboard can't produce. Judge audiences by that gap.
Back to the win-back campaign from Part III. The same two models that measured its lift can also show which customers it moved.
Give every customer a score that estimates their own lift. Then average those scores within each segment you care about. Kennedy's DR-learner builds that score from the same two models:
Any one customer's score is noisy. Averaged over similar customers, it gives the true effect, as long as either the propensity model or the outcome models are right. This still relies on the platform not targeting on anything hidden. Smoothing the scores across intent gives the curve on the left.
The dashboard compares exposed and unexposed customers within each intent band. By that view, the campaign adds 14 to 23 points for everyone. The DR-learner tells a different story. Lift is close to zero for low-intent customers and climbs to about 14 points for the highest. That curve tells the team who to target in the next flight.
A causal claim from one sample always rests on something the sample can't show. So treat each analysis as one piece of evidence, and plan for more than one.
Dana's final deck has three numbers on one page. The doubly robust estimate says the campaign lifted conversions by about a quarter. The sensitivity analysis says how much hidden targeting it would take to erase that. And the plan for the next flight includes a ten percent holdout, so that next quarter she can replace the assumption with a measurement.