Differential Registration Bias in Voter File Data: A Sensitivity Analysis Approach

In the following blog post, the authors summarize the forthcoming American Journal of Political Science article titled “Differential Registration Bias in Voter File Data: A Sensitivity Analysis Approach”:

In recent years, voter files have become an important data source for research in American politics. These lists of registered voters, which are maintained by the states, usually include contextual information such as date of birth, address, and sometimes race. The files also include records of whether each registrant voted in each election, making it possible to conduct studies of voter turnout with extremely large sample sizes.

However, these new data sources also present new challenges. Our new article in AJPS identifies an important concern for studies that use voter files to analyze turnout. Because the voter file contains a comprehensive list of registered voters, researchers often treat registrants as the population of interest. However, the fact that people select into the voter file by registering to vote has the potential to cause inferential problems in many common research designs.

For example, imagine a researcher is studying whether people turn out to vote more frequently when their elected official shares their racial or ethnic background. The researcher might wish to compare a treatment group of people who were redistricted to have a coethnic representative with a control group of those who were not – a design that might seem to make it possible to estimate the causal effect of having a coethnic incumbent on voter turnout.

The potential problem is that the treatment of interest may affect registration as well as turnout. In this example, people who are redistricted to have a coethnic incumbent might be induced to vote more, but they might also register at higher rates. If a treatment causes people to participate more, it may affect registration in that group as well as turnout, leading to what we call differential registration bias.

For instance, consider a case in which 10,000 people are in the treatment group and 10,000 are in the control group. 2,500 people vote in each group, but 5,000 register in the treatment group and only 4,000 register in the control group. If we analyze voter file data and use registered voters as the denominator, it will seem like people in the control group were more likely to turn out to vote (62.5% compared to 50%). We must instead use the voting-eligible population as the denominator (which is often not possible) to correctly recover the true population-level effect of zero (25% versus 25%).

To demonstrate the implications of this problem, we return to an issue that several other scholars have studied previously – whether eligibility to vote in the year someone turns 18 affects future turnout. We first conduct an original analysis of voter file data from Catalist, a leading vendor in this area. We compare people who were born within a one-week window around a voting eligibility cutoff. Some of these people were narrowly eligible to vote in the general election in the year they turned 18, while others were narrowly ineligible.

When we compare turnout rates among registered voters in the voter file, our results are at odds with many other studies in the literature – people who were narrowly ineligible when they were 18 appear to turn out at higher rates in subsequent elections. As we show, however, this finding seems to be the result of higher registration rates in the just-eligible group than in the just-ineligible group. Once we adjust for this difference by using birth counts as the denominator (approximating the voting-eligible population), our results reverse. Consistent with other research, we instead find that people who were narrowly eligible to vote in the year they turned 18 turn out at higher rates in future elections. This finding shows that differential registration bias can affect results in meaningful ways.

Researchers can sometimes circumvent this problem by using a design that avoids conditioning on registration or by approximating the voting-eligible population using U.S. Census data. In many cases, however, these approaches are not feasible. We therefore develop a simple sensitivity analysis that allows researchers to assess how vulnerable their results are to differential registration bias. Using our approach, researchers can calculate the extent of differential registration under which the observed difference in turnout among registrants would result in no difference in true turnout (i.e., among the voting-eligible population).

We hope this research will encourage scholars using voter file data to avoid differential registration bias in their research designs whenever possible and to report the sensitivity of their results to potential bias otherwise.

About the Authors:  Brendan J. Nyhan is a Professor of Government at Dartmouth College, Christopher Skovron is a doctoral student in the Department of Political Science at the University of Michigan, and Rocío Titiunik is an Associate Professor in the Department of Political Science at the University of Michigan. Their article “Differential Registration Bias in Voter File Data: A Sensitivity Analysis Approach” will be published in a forthcoming issue of the American Journal of Political Science and is currently available for Early View.