On February 12th, I attended UC Berkeley School of Information’s Dean Lecture with David Reiley where he presented his study, “Online Ads and Offline Sales: Measuring the Effects of Retail Advertising via a Controlled Experiment on Yahoo!”
David Reiley is a research scientist at Google and an innovative pioneer in the field of experimental economics. Although economics has historically been observational, Reiley advocates using economic experiments to gain new insights. In graduate school, Reiley discovered online auctions and began designing field experiments to test theories of auction bidding and charitable fundraising.
Before moving to Google in 2012, Reiley spent five years at Yahoo! where he studied the effects of online advertising on consumer behavior. At the Dean Lecture, Reiley presented the results of Yahoo’s! randomized experiment with 1.6 million customers measuring positive effects of online advertising and consumer behavior for a major retailer.
“Half the money I spend on advertising is wasted; the trouble is I don’t know which half.” –John Wanamaker
Hundreds of billions of dollars are spent each year on advertising with advertising as a share of GDP at more than 1 percent. While online advertising represents 21 percent of all advertising, online retail represents only 5.5 percent of all retail purchases. This divergence insinuates that online advertising’s effects extend beyond e-commerce to offline purchases.
Despite the economic importance of the advertising industry, the causal effects of advertising have been extremely difficult to quantify. Many economists believe that a very large sample size would yield any effect of advertising as highly statistically significant. However, the opposite is true—the variance of individual purchases makes for a large haystack to find the needle of advertising’s effects.
Observational study can be skewed by effect of selection bias meaning that the sample only includes users whom the advertiser intended to target with the advertising campaign. A 2008 Harvard Business Review highlighted one of the main problems with individual cross-sectional data is that the people who see a certain ad are not from the same population as those who do not see the same ad. For instance, someone who sees an ad for eTrade on Google is someone who typed in “online brokerage” or similar keywords. Since the identified people were already looking for eTrade brokers, they should expect higher baseline purchases from those exposed to the ad than unexposed users.
Experiment is the best way to establish a causal relationship
Yahoo!’s study combined a large-scale experiment with individual panel data. It matched nationwide retailer’s customer databases against Yahoo!’s user database, yielding a sample of 1,577,256 individuals.
Eighty percent of matched customers were assigned to the treatment group. In the fall of 2007, these 1.3 million Yahoo! users were shown two banner ad campaigns on Yahoo! from the retailer for two weeks. Several weeks later, Yahoo! exposed the same group to a follow-up advertising campaign. The remaining twenty percent of match customers saw a variety of other advertisements but did not see any ads from the retailer. The control treatment difference is the right comparison to answer the question, “What is the total difference in sales caused by this retailer’s ad campaign?”
The study found positive, sizable, and persistent effects of online retail advertising on sales. Yahoo! researchers estimate that the retailer’s total revenues were more than four times the cost of the ads. While many believe that online advertising mostly impacts online retails sales, the study finds that online advertising has a large effect on offline sales.
Yahoo!’s study found that although click-through-rate (CTR), the ratio of how often people who see your ad end up clicking it, is a standard measure of performance, focusing only on clickers leads to a serious underestimate of the campaign’s effects. CTR is a good predictor of online sales but not of offline sales. In fact, the study found that 78 percent of a lift in sales comes from those who view ads but do not click them, while only 22 percent comes from those who actually click.
One of the greatest realizations from the study is how poorly one can measure the causal effects of advertising using cross-sectional variation. Without the experiment, one would have observed a positive correlation between ad exposure and baseline exposure; the opposite of the study results. Moreover, the selection bias would have been three times the size of the true measured effect of advertising.
Ultimately, Yahoo!’s study was revolutionary in that it created an empirical strategy to identify causal effects of advertising. The experiment takes a major step in understanding the effects of brand advertising on consumer purchases and the effects of online ads and offline sales beyond the click. The field of experimental economics brings new insights into advertising and consumer behavior and should be further exploited to revolutionize what we know about the advertising industry.