Second Brain

Blake Nosko Tadelis 2015 - Paid Search Effectiveness - Overview

13 min read

Blake Nosko Tadelis 2015 - Paid Search Effectiveness - Overview

Summary

Blake, Nosko & Tadelis report a series of eBay field experiments that switched paid search (SEM) off. Brand-keyword ads (“ebay shoes”) had no measurable short-term value: when they were paused on MSN, 99.5% of the lost click traffic came back through the free organic link. For non-brand keywords, eBay stopped bidding in 68 of 210 US DMAs for about 60 days. The difference-in-differences effect of the whole paid search programme on sales was +0.66% (s.e. 0.55%, not significant). Converted into return on investment this is −63% (95% CI −124% to −3%), against +4,173% from a naive OLS regression of log revenue on log spend and +1,632% with DMA and day fixed effects. The average hides heterogeneity: ads moved new registrants and customers who bought only 1-2 times in the prior year, but most paid search spend went on frequent buyers who would have bought anyway. The authors read this as support for the informative view of advertising.

Overview

Bibliographic and context details (title and abstract pages).

  • Thomas (Tom) Blake, Chris Nosko and Steven Tadelis, “Consumer Heterogeneity and Paid Search Effectiveness: A Large Scale Field Experiment”. NBER Working Paper No. 20171, May 2014. Published in Econometrica 83(1):155-174, January 2015 (DOI 10.3982/ECTA12423), with an online supplement holding most regression tables.
  • JEL codes: C93 (field experiments), D22 (firm behaviour, empirical), L10, L20, L81 (retail and wholesale trade; e-commerce), M37 (advertising).
  • Affiliations (WP): Blake, eBay Research Labs. Nosko, Booth School of Business, University of Chicago, and eBay Research Labs. Tadelis, Haas School of Business, UC Berkeley, and NBER.
  • Data access and interests. The WP title page states: “This work was done while Tadelis and Nosko were employed by ebay Research Labs.” All three authors worked inside the advertiser whose campaigns were tested. That gave them access to eBay’s internal sales and click data and to the experiments themselves. Note the direction of the potential interest: the finding that the advertiser’s own spend was wasted runs against the ad platform, not against the employer, and the experiments led eBay to change strategy (as Lewis & Rao 2015 report, see Lewis Rao 2015 - Robustness, Scale, Targeting and Measurement Technology). The same access is why DMA-level revenue and cost are withheld, and why public figures are used for the ROI (§5).
  • Differences between the WP (used for these notes) and the journal version: Blake 2015 - Working Paper vs Econometrica Version Differences. All estimates are identical, but tables and equations are renumbered and some Discussion passages appear only in the WP.

Research question. What is the causal short-term return on paid search for a large, well-known advertiser, and why do standard industry methods get it so wrong?

Contribution.

  1. A clean statement of a form of endogeneity specific to pay-per-click media. Spend is generated by consumer intent, since the advertiser pays only when someone clicks, so spend and sales are correlated even with zero causal effect (Blake 2015 - SEM Mechanics and the Endogeneity of Click-Priced Spend).
  2. Large-scale experimental evidence. Platform-level shutdowns of brand keywords and a geo-randomized shutdown of all non-brand keywords together produce ROI estimates two orders of magnitude below observational ones.
  3. A heterogeneity analysis by purchase frequency and recency showing who responds, plus a null result across 378 product categories.
  4. A simple formula that turns a log-sales DiD coefficient into ROI (), and a way to put OLS and IV elasticities on the same scale.

Key findings (numbers).

QuantityValueWhere
Share of brand-keyword click traffic retained when MSN/Yahoo! brand ads paused99.5% (DiD on log clicks −0.00529, s.e. 0.0177)§3, Table 1 col 2
Google brand pause, pre-post drop in total Google-referred clicks−3.2% (no control available)§3, Table 1 col 3
Germany brand geo-test (8 of 16 states) DiD on log sales−0.00143 (full window), −0.00422 (from 1 Jan 2013), −0.000937 (from 1 Jan + state trends), all insignificant (s.e. 0.010-0.014)Appendix, Table 5
Non-brand geo-test DiD on log sales (daily, DMA FE)0.00659 (s.e. 0.00555); 95% CI [−0.42%, 1.74%] (CI printed only in the Econometrica version)§4.1, Table 2
Naive OLS elasticity of log revenue on log spend0.885§4.1, Table 3 col 1
OLS with DMA + date FE0.126Table 3 col 2
IV elasticity (DiD indicators as instruments)0.00401 / 0.00188 (with FE)Table 3 cols 3-4
ROI: OLS / OLS+FE / IV / IV+FE / DiD4173% / 1632% / −22% / −63% / −63%§5, Table 4
Paid clicks lost vs total clicks lost (non-brand)−41% paid, −2% total, so 42% of paid clicks are “newly acquired”§4.4
Elasticity of organic / direct visits w.r.t. paid visits (IV)−0.5 / −0.23§4.4
Level-regression ROI−80% (coefficient 0.199, s.e. 0.161)Appendix, Table 8

One-line takeaway. For an advertiser that consumers already know, click-priced ads mostly buy the clicks of people who were coming anyway. Attribution and spend-on-sales regressions credit those clicks to the ads, and only a randomized holdout separates the two.

Main Content

Argument flow

  1. §1-2 Motivation. Internet ads promised targeting and measurement. Pay-per-click pricing ties spend to intent, which creates a new kind of endogeneity. → Blake 2015 - SEM Mechanics and the Endogeneity of Click-Priced Spend
  2. §3 + Appendix (Germany). Brand keywords are the extreme case, where organic search is a near-perfect substitute. → Blake 2015 - Brand Keyword Experiments and Paid-Organic Substitution
  3. §4.1 + Appendix (randomization, candidacy checks). Non-brand geo experiment: design and DiD estimates. → Blake 2015 - Non-Brand DMA Geo Experiment and DiD Estimates
  4. §4.1 eq. (3)-(4), Table 3, Appendix Table 8. OLS vs IV: how much of the observational elasticity is endogeneity. → Blake 2015 - OLS vs IV Spend Elasticities and the Endogeneity Gap
  5. §4.2-4.3. Heterogeneity by user frequency, recency, zip demographics and product category. → Blake 2015 - Consumer and Product Heterogeneity in Paid Search Response
  6. §4.4. Where the traffic went: clicks vs sessions, organic and direct substitution. → Blake 2015 - Non-Brand Traffic Substitution (Clicks vs Visits)
  7. §5 + Appendix ROI derivation. ROI formula and Table 4. → Blake 2015 - ROI Derivation from Experimental Log-Sales Effects
  8. §6. Why firms overspend, the brand-bidding Prisoner’s Dilemma, incentives of the measurement industry, limitations. → Blake 2015 - Discussion, Limitations and Lessons for Marketing Measurement
  9. Versions. How the NBER working paper maps onto the Econometrica article and supplement. → Blake 2015 - Working Paper vs Econometrica Version Differences

Source Map

Every section, equation, table and figure of the source (NBER WP 20171 printed page numbers; add 2 for PDF page) and the note that holds it.

Source elementContentNote
Title page, abstract page (front matter)NBER WP 20171, May 2014; JEL C93, D22, L10, L20, L81, M37; affiliations (eBay Research Labs, Chicago Booth, Berkeley Haas/NBER); “work was done while Tadelis and Nosko were employed by ebay Research Labs”This note, Overview § “Bibliographic and context details”
Abstract, §1 Introduction (pp. 1-6)IAB/Google revenue context, two main findings, Wanamaker quote, targeting and measurement promise, intent-driven endogeneity, preview of brand/non-brand tests, informative vs persuasive views, Ackerberg (2001), activity bias, ROI preview (−63% vs >4,100%), top-10 spendersBlake 2015 - SEM Mechanics and the Endogeneity of Click-Priced Spend
Footnotes 1-13Data sources, Berndt survey, brand-bidding rationale (fn. 4), sales vs revenue, 100M keywords managed by daily-updated algorithms feeding Google/MSN/Yahoo! (fn. 6), Vaver & Koehler, Bagwell’s three views, display-ad literature, demographic heterogeneity by age, gender and location (fn. 10), SEM literature, Edelman and Lewis & Rao (fn. 12), Kantar top-10 listSEM Mechanics note (Wanamaker quote, infra-marginal consumers, 16 Feb 2013 brand-ad examples, keyword examples, fn. 6); Brand note (fn. 4); Discussion note § “What the literature footnotes actually say” (fn. 10-12)
§2 Overview of SEM (pp. 6-7), Fig. 1a-b, fn. 14-15Paid vs organic results, PageRank vs bid × quality score, pay-per-click, position auctions (Edelman et al.; Varian; Athey & Ellison), brand vs non-brand query, search vs display ads (fn. 14), eBay’s two paid and two organic slots in Fig. 1a (named in §4.4 p. 19)Blake 2015 - SEM Mechanics and the Endogeneity of Click-Priced Spend
§3 Brand Search Experiments (pp. 7-10), Fig. 2a-b, Table 1, fn. 16-17MSN/Yahoo! halt (March 2012), 99.5% retention, pre-post and cross-platform DiD, Table 1 nuisance rows (platform dummies, constant) and N = 118 / 3×60 / 120 read, Google halt and resume (July 2012)Blake 2015 - Brand Keyword Experiments and Paid-Organic Substitution
Appendix: Controlled Brand-Keyword Experiments (pp. 27-28), Fig. 6, Table 5Germany 8/16 states, Jan 2013, DiD on log sales (all three columns), timing reconciled (Fig. 6 window 1 Dec-26 Jan; N 912 = 16×57, 416 = 16×26), the “narrowing reduces magnitude” claim checked against the tableBlake 2015 - Brand Keyword Experiments and Paid-Organic Substitution
§4 intro (p. 10)Why non-brand terms are informative about informative advertisingBlake 2015 - Non-Brand DMA Geo Experiment and DiD Estimates
§4.1 Experimental design (pp. 11-12), Fig. 3a-b, fn. 18-21210 DMAs (SF-Oakland-San Jose vs Butte-Bozeman size examples), 68 test / 65 matched control / 77 rest, attributed sales fall 72%, location misclassification, ratio and log difference plotsBlake 2015 - Non-Brand DMA Geo Experiment and DiD Estimates
Eq. (1)-(2), Table 2, fn. 22DiD and two-way FE specifications, daily vs totaled, fixed-price only, N decompositions, Experiment Period sign flip, col (4) FE structure, Search Group ≈ 0 vs “30 percent larger” reconciledBlake 2015 - Non-Brand DMA Geo Experiment and DiD Estimates
Eq. (3)-(4), Table 3OLS log-log elasticity vs 2SLS with DiD instruments, “Log Revenue” vs ln(Sales) label, IV s.e. 0.0410 → 0.00157 and col 3’s unexplained specificationBlake 2015 - OLS vs IV Spend Elasticities and the Endogeneity Gap
Appendix: Randomization Procedure Detail (pp. 28-29)133 candidates, re-drawn groups by serial correlation, coin flipBlake 2015 - Non-Brand DMA Geo Experiment and DiD Estimates
Appendix: Candidacy IV, eq. (7)-(8), Table 6Candidate status as instrument for treatment; reconstructed intensive-margin equation for col 2Blake 2015 - Non-Brand DMA Geo Experiment and DiD Estimates
Appendix: Candidacy DiD, Table 7Diluted effect and placebo among never-treated DMAs; unidentified Candidate/Period coefficients flaggedBlake 2015 - Non-Brand DMA Geo Experiment and DiD Estimates
Appendix: ROI Results in Levels, Table 8Level regression, ROI −80%Blake 2015 - OLS vs IV Spend Elasticities and the Endogeneity Gap and Blake 2015 - ROI Derivation from Experimental Log-Sales Effects
§4.2 Consumer Response Heterogeneity (pp. 15-18), eq. (5), Fig. 4a-b, Fig. 5a-b, fn. 23-26Frequency segments with plotted values read off, the authors’ “Bluntly…” interpretive paragraph (p. 16), recency bins and spline (+0.02 pp per month), Fig. 5 axes, buckets and line values, attribution concentration on heavy buyers, zip-code demographicsBlake 2015 - Consumer and Product Heterogeneity in Paid Search Response
§4.3 Product Response Heterogeneity (p. 18)Framing (competitiveness, market thickness, desirability; “to our surprise”), 38 top-level categories, 378 granular categories, under 5% significantBlake 2015 - Consumer and Product Heterogeneity in Paid Search Response
§4.4 Where did the Traffic Go? (pp. 18-20), fn. 27Paid clicks −41%, total −2%, sessions vs referring clicks, IV substitution elasticities 0.5 and 0.23, unreported IV details, clicks-vs-visits reconciliationBlake 2015 - Non-Brand Traffic Substitution (Clicks vs Visits)
§5 Deriving ROI (pp. 20-21), eq. (6), Table 4, fn. 28-29ROI definition, approximation, Spend adjustment 3.51, public revenue/spend inputs, “total” vs “Marketplaces” revenue wording, 2010 spend vs 2012 revenue timing mismatch, the three conflicting OLS + FE ROI figures (“over 1,400%” p. 5, “greater than 1500 percent” p. 21, 1632% Table 4), and the “IV significantly below zero” claim checked against col 3Blake 2015 - ROI Derivation from Experimental Log-Sales Effects
Appendix: ROI Calculations, eq. (9)-(11)Full derivation, the appendix’s own (ads-off) regression and its sign/identification slips, and the conversionBlake 2015 - ROI Derivation from Experimental Log-Sales Effects
§6 Discussion (pp. 21-24), fn. 30-35Generalization to big brands, long-term effects, brand-bidding Prisoner’s Dilemma, Lewis et al. (2011) comparison, attribution vs incrementality, Google’s “incremental clicks”, Google’s AdWords ROI advice (quoted in full), misaligned industry incentives, user-level randomization as the idealBlake 2015 - Discussion, Limitations and Lessons for Marketing Measurement
References (pp. 25-26)27 references (Ackerberg 2001 to Yao & Mela 2011; counted against the raw text)Cited inline where used
Published version (Econometrica 83(1), accepted ms. and online supplement)Renumbered tables and equations, added DiD CI and frequency slope, dropped Discussion passages, all estimates identicalBlake 2015 - Working Paper vs Econometrica Version Differences

Examples

Reading the paper as a marketing analyst (worked example)

A retailer’s attribution dashboard credits paid search with $10M of sales on $1M of spend, a “ROAS of 10”. The paper’s logic gives three checks:

  1. Is the term navigational? For brand terms, assume near-complete organic substitution (§3) until a holdout shows otherwise.
  2. Who is clicking? If most attributed buyers are frequent customers (the Fig. 5 pattern), the incremental share is small.
  3. Run the geo holdout. With a DiD log-sales effect and the ratio of total revenue to search spend, . Example: , revenue $500M, spend $5M gives . The channel roughly breaks even, whatever the dashboard says. See Blake 2015 - ROI Derivation from Experimental Log-Sales Effects.

Connections

See Also

Graph View

Backlinks