United States Elections Project

A Critique of the Berkeley Voting Study
Dr.  Michael P. McDonald 
Assistant Professor, George Mason University

A group of sociologists (Hout, et. al 2004) at University of California, Berkeley have produced a report that claims President Bush received approximately 130,000 votes more than he should have in Florida as a consequence of an unspecified error with touch screen voting technology in the 15 counties that use touch screens.  A copy of the report and data are available at: 

http://ucdata.berkeley.edu/new_web/VOTE2004/

Since the claim has important implications to the legitimacy of the 2004 election, it is important to understand the substance of the claim.  There are a number of issues, some statistical in nature and others that are simple common sense, which seriously raise into question the validity of the study.  Some of my, and other's, initial reactions to it are presented below.

The Claim: Bush received more votes than he should have in touch screen counties.

The Berkeley report claims that the change in Bush's share of the vote was greater than it should have been in counties that used electronic voting machines.  The main problem with the claim is that it does not conform to the actual election results at face value.  There are three serious points:

• The average change in Bush’s % vote from the 2000 election in counties without touch screens was +4.1 points, while in touch screen counties, the average change in Bush’s % vote was +2.4 points.

• In order for the claim to be true, Bush must have received fewer total votes than he received in 2000 in the counties with touch screens (as reported in the Boston Globe). 

• Running the same model, except explaining the vote for Kerry, shows that he, too, received more votes than he should have (see a report by Dr. Bruce McCullough, at Drexel University, and Dr. Florenz Plassmann at SUNY Binghamton: http://election04.ssrc.org/research/critique-of-hmcb.pdf).

We should be suspicious of statistical models that aren’t confirmed by simple common sense, such as in this case.  In order to uncover their finding, the scholars create a complicated statistical model that they offer without theoretical grounding.  Here are the essential steps behind the construction of the model:

(1) Create an indicator variable [0,1] for whether or not one of Florida’s 67 counties uses touch screen technology.   There are 15 counties that used touch screens.  A statistical model run with this variable alone to uncover the effect of touch screens does not support the report’s claim that Bush received more votes than he should have.

Dr. Alan Abramowitz at Emory concurs that a simple regression using just these variables does not support the Berkeley claims (the negative coefficient on EVOTE means that touch screen counties were estimated to have a smaller change towards Bush than other counties, and the t-stat of -1.724 indicates that this change was not statistically significant, so it is not a finding we can be confident of) 

 

(2) Create an “interaction term,” which is one variable in the model multiplied by another.  In this case, the variable for touch screen technology is multiplied by the variable for % vote for Bush in 2000.  This interaction term will equal zero for the 52 counties without touch screens and the % vote for Bush in 2000 for those with.  Already, this should raise statistical warning flags.  We now have two variables in the model explaining the election outcome in 15 counties.  Generally, statistics should be applied to a greater number of observations.  Furthermore, there is no theoretical justification for this interaction term.  Even so, with (1) and (2) in the model, the model still does not support the report's claim that Bush received more votes than he was expected to.

(3) Create another interaction term, which is the variable for touch screen technology multiplied by the variable for % vote for Bush in 2000, squared.  No theoretical justification is provided for this variable, and we now have three variables explaining the election outcome in 15 counties.  Both of these facts are of concern, but it works for the authors.  This interaction term plus (1) and (2) is the model presented in the report, and the model that is used to establish the report's findings.

What is Really Happening with the Berkeley Statistical Model 

The statistical model is essentially picking up vote gains for Bush in two Florida counties that are not the same as vote swings in other counties: Broward and Palm Beach.  The interaction terms, discussed above, fit the data in just the right way to produce the statistically significant result reported in the Berkeley paper.  But, it is not enough that models fit the data, they must also have what is known as construct validity: they must fit the way in which we believe the world works.  To offer variables which happen to fit the data in a particular way, without providing a compelling theory, is not proper science.

In a discussion I had with Dr. Hout on Monday, Nov 22 on KPFA radio, he acknowledged that the results are driven by these two counties, though he adds another to the list: Miami-Dade. Below, I report the 2000 and 2004 election returns, for the latter I use the most recent report for the Associated Press (as of Nov 23). I also report the change in the vote for Bush, both from the (dated) Berkeley study and the Nov. 23 report.

As the table shows, Miami-Dade County saw a slight shift towards Kerry, so I don’t include it in the list of outlier counties.  A study at Columbia comes to the same conclusion, that the Berkeley study results are driven by Broward and Palm Beach:

http://www.stat.columbia.edu/~cook/movabletype/archives/2004/11/vote_swings_in.html

The Nov. 23 AP election report from Florida also shows that the initial Berkeley study is over-estimating the effect of touch screens, since the vote share for Bush has dropped in the key counties of Broward and Palm Beach as more votes are counted. 

Where these counties fit in with the rest of Florida can be seen in this scatterplot of the % Bush 2000 vote versus the % Bush 2004 vote (Nov. 23 AP report):

Seen in this context, Broward and Palm Beach were not even two of the most variable counties in 2004 compared to the 2000 election.  What makes these counties stand out is that they are among the most Democratic, and there are so few highly Democratic counties.  Gadsen and Leon, the other two counties in the lower lefthand corner of the scatterplot did not use touch screens.  This means that the results of the study are not only driven by outlier counties, as acknowledged by Dr. Hout, they are driven by outlier counties that occur in the tail end of the distribution of counties that voted electronically.  Dr. Ben Highton at UC Davis points that this further violates the statistical assumptions in regression analysis that are underlying the Berkeley model (see Beck and Jackman’s article in the American Journal of Political Science, 1998).

Bottom line 

When seen in the context of all Florida counties, Broward and Palm Beach don’t look out of line.  The only way to achieve the results found in the Berkeley study is to create a statistical model, not grounded in theory, that is fitted to the election results in these two counties.  I don't believe this was done intentionally, but it is what happened.  We should be highly skeptical of statistical results that depend on two observations.  That these two observations violate other statistical assumptions only adds icing to this terrible tasting cake.

A Reality Check: What Really Happened in Broward and Palm Beach

If we apply some common sense, we can further dispense with the Berkeley study.

Let’s suppose that the Berkeley study is correct.  The findings imply that electronic touch screen voting machines somehow provided 130,000 more votes for Bush than he should have received.  We are talking about tens of thousands of votes in Broward and Palm Beach counties.  For the election result to be shifted that much, the vote gain for Bush could not have come from a handful of precincts that, say, recorded all their votes for Bush.  Local election officials and Democratic activists would have noted the aberrant results.  Instead, the implication borders on intentional vote fraud: that somehow, countywide, tens of votes were added to Bush’s column in every precinct.

Vote fraud on this scale seems highly unlikely, but we cannot know for sure because there is no method of verifying the results.  This is the last, and weakest, defense of the Berkeley study made during our discussion on KPFA.  The claim that the vote is not what the voters intended could be made in every county that uses electronic voting machines in the country, except Nevada, which uses paper verified vote trails.  Personally, I am sympathetic to the cause of paper verified vote trails, having written a book on when statistical software can go awry, and I suggest that interested people check out the lead advocate organization: http://verifiedvoting.org/index.php.  There are many much better (and verified) examples of electronic touch screen technological screw-ups that affect candidates of both political parties.

Other Explanations of What Happened in Broward and Palm Beach

Because the Berkeley findings are driven by election results in two counties, it is easy to provide other explanations of the vote change in these two counties.  In statistical terms, we call excluding important variables from a model "omitted variable bias."  We should particularly favor explanations (or statistical models) that are grounded in what we know about voting behavior, rather than a model which is provided without theoretical grounding for the important variables in the analysis.  We should also favor simple models over complicated ones.  The Berkeley study fails on both of these criteria since the complicated interaction terms are included in the model without theoretical justification.  Other proposed, and simpler, models explain the election results better than the Berkeley study, are grounded in what we known about politics, and do not find that touch screens provided Bush with extra votes.  

Dr. McCullough and Dr. Plassman's Study

This report, mentioned above, discusses many alternative models that fit the data better, but render the Berkeley results statistically insignificant.  They rightfully shows that the Berkeley model is not even the vote model the authors think that they are estimating since they are regressing change in Bush vote (% Bush vote 2004 - % Bush vote 2000) on % Bush vote in 2000.  In simple algebra, the Berkeley model is:

(% Bush vote 2004 - % Bush vote 2000) = B1* (% Bush vote in 2000) + (other variables in the analysis)

Which, by adding % Bush vote in 2000 to both sides of the equation, can be rewritten as:

% Bush vote 2004 = (B1+1) * (% Bush vote in 2000) + (other variables in the analysis)   

http://election04.ssrc.org/research/critique-of-hmcb.pdf  

Increased Voter Turnout

Jason Lenderman, graduate student in statistics at UCLA, shows that including an interaction term between the increase in voter turnout from 2000 to 2004 (v_change) and the % Bush vote in 2000 renders the three electronic vote variables in the Berkeley model statistically insignificant (i.e., the Berkeley findings are not supported).  This model uses fewer variables, explains more, and is consistent with observations, such as those by the Los Angeles Times, which shows that Bush gains were concentrated in the growing counties across the U.S.  His report is available at: 

http://www.stat.ucla.edu/~jasonl/vote_note.html

Cuban vs. non-Cuban Hispanics

Dr. Ben Bishin at the University of Miami divides the Hispanic variable into sub-categories of Cuban, Puerto Rican and other Hispanics.  The results of Dr. Bishin's statistical analysis are given below.  These models outperform the Berkley model in terms of predictive power, and an examination of the three electronic touch screen variables shows that these variables are now all not statistically significant (i.e., the Berkeley findings are not supported).

Table of results from E Voting analysis.

#p<.051, p<.05, **p<.01

DV: Change in Bush Vote from 2000.

Model 1:  Hout’s DV: Change in Bush Vote from 2000.

Model 1:  Hout’s model from Table 2 of their paper.
Model 2:  Bishin’s initial model breaking Hispanic into Cuban vs. Non-Cuban.
Model 3:  Bishin’s model substituting Cuban for Hispanic.
Model 4:  Bishin’s model accounting for conflicting preferences of two major groups of Hispanic Floridians.

For a rational of dividing up the Hispanic vote into different categories, see: “Hispanic Vote in Florida: neither a Bloc nor a Lock”  10/17/2004.  New York Times.  Abby Goodnough.  That different Hispanic sub-groups have different voting patterns, especially among Cubans, Puerto Ricans, and other Hispanics, is well-documented in academic polling for previous elections.  Unfortunately, the national exit poll did not break out Latino voters into sub-categories, so we cannot know at this time if the supposition is externally validated by other data.

A Word on the Academic Integrity of the Report

At the end of the report, the Berkeley authors discuss "reviewers," as if the report has undergone the gold standard in academic publications: peer-review.  It is impossible that the report underwent rigorous double-blind peer-review in the short time between the election and when the report was posted.  What is more likely is that the authors got feedback as they circulated the report.  Peer-review will occur when the manuscript is sent to an academic journal.  As the report stands now, I would be surprised if it is accepted for publication in a reputable journal that specializes in elections.  It lacks theoretical grounding and there are many troublesome methodological issues that reviewers will raise with the study.

Office: 703-993-4191
Fax: 703-993-1399
Email: mmcdon@gmu.edu