Testing Interactions in Linear Processes: The Benefits of New Binomial Tests Over Logistic Regression
Oleg Urminsky (University of Chicago)
Presented at the Marketing Science Conference, 2010
Brief Abstract:
This paper questions the use of logit-based models for statistical testing of interactions in experiments, when the underlying process is assumed to be additive. Using both simulations and exact tests, I show evidence that logistic regression suffers from bias as well as low power in these contexts. Instead, I present four tests that are much better at testing interactions in choice experiments, including two new tests based on the binomial properties of choice data.

Extended Abstract:
Behavioral researchers often use a model-based approach to test for statistical significance in between-subjects experiments where choice is the dependent variable. In the past, such data was often analyzed using ANOVA models, but in recent years logit-based models (i.e. logistic regression) have become the standard approach.
In this research, I contrast the additive assumptions implied by the ANOVA test with the multiplicative model assumed by using logistic regression to test an interaction in a 2x2 between subjects design. Then, using both an extensive series of Monte Carlo simulations as well as comparisons with exact tests, I demonstrate how different the findings from the two approaches can be, and under what conditions these differences are most prevalent.
When the theory presumes an additive process, an interaction means that the size of the difference in choice proportions caused by one manipulation depends on the other manipulation. I show that logistic regression performs poorly as a test of the null hypothesis in this specific case. First, it has low power when there really is an interaction. Second, it has excessively high probability of failing to reject the null hypothesis, even when it is true and there is no interaction. I demonstrate that it is even possible to get a significant interaction for a logistic regression when no difference of differences in choice proportions is observed. In contrast, four tests are described that