Conducting research in marketing with quasi-experiments

Researchers from University of Toronto, MIT, and University of British Columbia published a new paper in the Journal of Marketing that aims to broaden the use and usefulness of quasi-experimental methods in marketing by describing the underlying logic and actions that make this work convincing.

The study is authored by Avi Goldfarb, Catherine Tucker, and Yanwen Wang.

Quasi-experimental methods have been widely applied in marketing to explain changes in consumer behavior, firm behavior, and market-level outcomes. As Goldfarb explains, “The purpose of quasi-experimental methods is to determine the presence of a causal relationship in the absence of experimental variation.”

As a vivid example, the authors describe a quasi-experiment that occurred when eBay shut down all the paid search advertising on Bing during a dispute with Microsoft, but lost little traffic. These quasi-experimental results inspired a follow-up field experiment where eBay randomized suspension of its branded paid search advertising and found results consistent with the quasi-experiment. They begin by establishing various type of quasi-experimental variation at the individual, organizational, and market-levels. In each type, given the lack of an experiment, some individuals, companies, or markets receive an action or policy (i.e., treatment group) and some do not (i.e., control group). For example, some markets are affected by a new policy and some are not. The question is how the markets receiving the treatment would act if they had not received it (i.e., the counterfactual). Tucker explains that “The unobservability of the counterfactual means assumptions are required to ensure that differences, both observed and unobserved, are as untroubling as possible, thereby mimicking random assignment as closely as possible.” 

The article discusses how to structure an empirical strategy to identify a causal relationship between using methods such as difference-in-differences, regression discontinuity, instrumental variables, propensity score matching, synthetic control, and selection bias correction. The authors emphasize the importance of clearly communicating identifying assumptions underlying the assertion of causality and establishing the generalizability of the findings.

The following topics are examined:

• Topic 1—Research question: Do we care whether x causes y? “The first and hardest step is identifying a question about which marketing scholars, managers, or policy makers actually care whether x causes y,” says Wang.
• Topic 2—Data question: How to find data with quasi-experimental variation in x? Much of the work using quasi-experimental variations in marketing harnesses easily understood events such as contract changes, ecological issues such as the weather, geography, and macroeconomic, individual, organizational, and regulatory changes. The key is to consider why each of these sources of variation can approximate random assignment.
• Topic 3—Identification strategy: Does x cause y to change? The researcher must first explain where the variation claimed to be exogenous comes from. Second, the researcher needs to demonstrate that the relationship between the variation and the outcome of interest is very likely driven by the relationship between x and y and not by some other factor.
• Topic 4—Empirical analysis: How to estimate the effect of x on y? Three different regression analysis frameworks using quasi-experiments are described: difference-in-differences, regression discontinuity, and instrumental variables.
• Topic 5—Challenges to research question: What if variation in x is not exogenous? Three methods are examined—propensity score matching, synthetic control methods, and selection bias correction—with steps to take when comparability between the control and treatment groups is violated.
• Topic 6—Robustness: How robust is the effect of x on y? The idea here is to show that the sign, significance, and magnitude of the estimate remain broadly consistent across a vast range of possible models. A few examples of robustness checks include different controls, functional forms, choices of time periods, dependent variables, the size of the control group, and a placebo test.
• Topic 7—Mechanism: Why does x cause y to change? Understanding how the process of the change unfolds adds insight that can drive new knowledge and stronger actions.
• Topic 8—External validity: How generalizable is the effect of x on y? The external validity discussion in a paper should recognize the assumptions required for the analysis to capture the average treatment effect across the population of interest rather than a more local effect that is an artifact of the data sample or the source of quasi-experimental variation.
• Topic 9—Apologies: What remains unproven and what are the caveats? Any identification strategy relies on assumptions that need to be explicit throughout the paper. While apologies do not mean all is forgiven, the objective should be to clarify the boundaries of claims made in the paper.