« It is now harder for Hillary to Win. The math is the real story. | M Miller's Blog | Indiana: Obama 48%, Clinton 47% - Research 2000: »
SurveyUSA track record
SurveyUSA
California
Clinton 52%, Obama 42%
Actual vote
Clinton 52%, Obama 42%
C 0, O 0
South Carolina
Obama 43%, Clinton 30%, Edwards 24%
Actual vote
Obama 55%, Clinton 27%, Edwards 18%
O +12, C -3, E -6
Oklahoma
Clinton 54%, Obama 27%
Actual vote
Clinton 55%, Obama 31%
C +1, O +4
Missouri
Clinton 54%, Obama 43%
Actual vote
Obama 49%, Clinton 48%
O +6, C -6
Illinois
Obama 66%, Clinton 30%
Actual vote
Obama 65%, Clinton 33%
O -1, C +3
Alabama
Obama 49%, Clinton 47%
Actual vote
Obama 56%, Clinton 42%
O +7, C -5
New York
Clinton 56%, Obama 38%
Actual vote
Clinton 57%, Obama 40%
O +2, C +1
Virginia
Obama 60%, Clinton 38%
Actual vote
Obama 64%, Clinton 35%
O +4, C -3
Maryland
Obama 55%, Clinton 32%
Actual vote
Obama 60%, Clinton 37%
O +5, C +5
Massachusetts
Clinton 56%, Obama 39%
Actual vote
Clinton 56%, Obama 41%
O +2, C 0
Connecticut
Obama 48%, Clinton 46%
Actual vote
Obama 51%, Clinton 47%
O +3, C +1
New Jersey
Clinton 52%, Obama 41%
Actual vote
Clinton 54%, Obama 44%
O +3, C +2
Ohio
Clinton 54%, Obama 44%
Actual vote
Clinton 54%, Obama 44%
O 0, C 0
Texas
Obama 49%, Clinton 48%
Actual vote
Clinton 51%, Obama 47%
O -2, C +3
I looked around and tabulated all the polls for
primaries (only) that SurveyUSA did and also the final results and compared
them.
In the big states of Ohio and California, SurveyUSA
was dead accurate with the percentages (and winners of course). In the big state of Texas, they
actually were wrong about the winner as well as in Missouri.
Of all the states listed, SurveyUSA had
underestimated Obama's election showing 10 times, overestimated 2 times, and
were right 2 times. Underestimation: 2-12%, average of 4.8%.
Overestimation: 1-2%, average of 1.5%.
Of all the states listed, SurveyUSA had
underestimated Clinton's election showing7 times, overestimated4
times, and were right3 times. Underestimation: 1-5%, average of
2.29%. Overestimation: 3-6%, average of 4.25%.
California
Clinton 52%, Obama 42%
Actual vote
Clinton 52%, Obama 42%
C 0, O 0
South Carolina
Obama 43%, Clinton 30%, Edwards 24%
Actual vote
Obama 55%, Clinton 27%, Edwards 18%
O +12, C -3, E -6
Oklahoma
Clinton 54%, Obama 27%
Actual vote
Clinton 55%, Obama 31%
C +1, O +4
Missouri
Clinton 54%, Obama 43%
Actual vote
Obama 49%, Clinton 48%
O +6, C -6
Illinois
Obama 66%, Clinton 30%
Actual vote
Obama 65%, Clinton 33%
O -1, C +3
Alabama
Obama 49%, Clinton 47%
Actual vote
Obama 56%, Clinton 42%
O +7, C -5
New York
Clinton 56%, Obama 38%
Actual vote
Clinton 57%, Obama 40%
O +2, C +1
Virginia
Obama 60%, Clinton 38%
Actual vote
Obama 64%, Clinton 35%
O +4, C -3
Maryland
Obama 55%, Clinton 32%
Actual vote
Obama 60%, Clinton 37%
O +5, C +5
Massachusetts
Clinton 56%, Obama 39%
Actual vote
Clinton 56%, Obama 41%
O +2, C 0
Connecticut
Obama 48%, Clinton 46%
Actual vote
Obama 51%, Clinton 47%
O +3, C +1
New Jersey
Clinton 52%, Obama 41%
Actual vote
Clinton 54%, Obama 44%
O +3, C +2
Ohio
Clinton 54%, Obama 44%
Actual vote
Clinton 54%, Obama 44%
O 0, C 0
Texas
Obama 49%, Clinton 48%
Actual vote
Clinton 51%, Obama 47%
O -2, C +3
I looked around and tabulated all the polls for
primaries (only) that SurveyUSA did and also the final results and compared
them.
In the big states of Ohio and California, SurveyUSA
was dead accurate with the percentages (and winners of course). In the big state of Texas, they
actually were wrong about the winner as well as in Missouri.
Of all the states listed, SurveyUSA had
underestimated Obama's election showing 10 times, overestimated 2 times, and
were right 2 times. Underestimation: 2-12%, average of 4.8%.
Overestimation: 1-2%, average of 1.5%.
Of all the states listed, SurveyUSA had
underestimated Clinton's election showing7 times, overestimated4
times, and were right3 times. Underestimation: 1-5%, average of
2.29%. Overestimation: 3-6%, average of 4.25%.
Advertisement
This is great info...good job. I was impressed with the fact that when Clinton or Obama are projected to be large winners, as in Massachusetts or Maryland, the actual votes held true. This is legitimate cause for concern for Obama supporters.
But look at Missouri - what a turnaround! And, as I recall, the final votes in St. Louis and St. Louis County were where Obama squeaked it out.
Lesson for PA: if you want Obama to win, get thee to Philadelphia...
April 9, 2008 8:35 AM | Reply | Permalink
I remember that too - Demographic was 80% black or something and they turned out massively for him.
April 9, 2008 1:45 PM | Reply | Permalink
I agree that overall this was a great analysis.
I think it would be more accurate and even more "on target" if you were to divide the "undecided" polling numbers among the same percentages as the rest of the poll. THAT accounts for the majority of the of the underestimates.
(Most of the time, the undecided poll numbers are reported in the complete report. Also, be careful not to include Edwards poll numbers in the undecided)
April 9, 2008 2:09 PM | Reply | Permalink
Wait just a second here. You're not dealing with the proper data set. Here you are, comparing SUSA's last poll before a contest to the final election result. And then you're attempting to extrapolate their poll from a week out to the election day results.
It's probably a good guess that inasmuch as SUSA HAS been accurate in their immediate pre-election polling, it's because they've managed to tweak their model in the days leading up to the election, not because their models are always perfect from the beginning of an election cycle.
April 9, 2008 8:52 AM | Reply | Permalink
This critique is close to the mark, but needs to be tweaked just a little.
The problem isn't that SurveyUSA adjusts its "model." In fact, the key strength of the SurveyUSA methodology is that it doesn't have a demographic model. It calls at random, and reports its results. Unlike many other pollsters, it doesn't adjust the data to match its demographic model. So if more women than men respond that they're going to vote, they assume they're documenting a bona fide gender gap, and report it, instead of massaging the data to match past turnout. In this year of record turnout, that's given them a crucial edge. The past data are remarkably unhelpful in forecasting who's going to show up at the polls in this precedent-breaking cycle, and pollsters who have relied on past turnout to predict results have found themselves in a lot of trouble.
But there's another crucial difference between SurveyUSA and the other polls. Take a look at their data - there are almost no undecideds. That's because they don't volunteer it as an option. That has the effect of pushing wavering voters to choose a candidate. The conventional wisdom had always been that such forced decisions skew results, but that hasn't been the case this year - it appears that most voters are pretty sure how they're going to vote long before the election actually takes place. Demography, this cycle, is destiny.
But the main point Anarchofascist makes is correct. SurveyUSA, in its own "Report Card," and M Miller, both make the same mistake. SurveyUSA continues polling right up to the edge of the election - a habit that accounts for much of their fabled accuracy. Compare a mainstream poll taken two weeks before to the robopoll on election eve, and most of the time, the later poll will be more accurate. These last minute polls are also less susceptible to the quirks of SurveyUSA's methodology - respondents are more likely to accurately report their likelihood of voting the day before the election, and their also more likely to have made up their minds, so pushing them to voice an opinion creates less distortion.
None of which, of course, applies to polls taken several weeks in advance of the election. In fact, there's virtually no way to assess the accuracy of such polls. Even if a poll three weeks out precisely predicts the result, that's not enough - perhaps the poll was way off, but voters changed their minds in the interim.
The most accurate predictions this cycle - more accurate even that SurveyUSA - have been the synthesized "polls of polls," which aggregate data to predict outcomes. They tend to iron out the individual biases and quirks, and produce fairly accurate results. And those polls show the race tightening. That's where I'd put my money.
April 9, 2008 9:49 AM | Reply | Permalink
That's what SurveyUSA CLAIMS their technique is. Unfortunately, it doesn't hold up to much scrutiny. The claim, that they simply call "at random" is true, of course.
BUT FROM WHAT LIST OF PHONE NUMBERS? Do they randomly dial from the population of all seven digit phone numbers of all area codes associated with a state? Do they randomly dial from the entire residential directory of a given state? Both of these may be possible, but they also both represent a "model."
April 9, 2008 4:41 PM | Reply | Permalink
I think I recall reading somewhere that SUSA subcontracts it number selection to some company in Connecticut, and they use RDD.
April 9, 2008 6:08 PM | Reply | Permalink
SUSA's MO poll was through Feb 3--the day before the election--and they were spectacularly wrong.
April 9, 2008 9:21 AM | Reply | Permalink
Good information. However, I would caution your analysis presented in the last two paragraphs. The mean is not the best choice as a measure of central tendency your data.
Even more problematic is using a mean of means.
There is also the variable of different SDs in each of the individual poll results which you have not corrected for in your the final analysis, nor did you provide the SD for your final analysis. Much of the variance you are seeing between paired polls and the in the aggregate numbers is statistically insignificant.
Another problem is that you have combined 2-person and 3-person races in your data, as well as did not include the undecided counts.
Sorry, I am a real egghead when it comes to stats, something to do with growing up with a statistician and a public opinion poller in the family. Is that child abuse? :)
Having said all of the above, your analysis still merits an A-minus-plus... so keep up the good work. If I have time today, I take your data, try to fine the missing undecided numbers, SDs, and do my egg-headed analysis. But, in short, we will both say the same thing: SurveyUSA is pretty good.
April 9, 2008 9:42 AM | Reply | Permalink
Hey People lets get some perspective here. SUSA has been pretty good this cycle but polling is not an exact science. That is why we still have votes.
April 9, 2008 9:56 AM | Reply | Permalink
One other thought. Scanning down your list of polls, I noticed something interesting about the underestimations of Obama's support. Let's look at the five states where the difference between the predicted and actual results for Obama were outside of the polls' margins of error:
South Carolina: -12% 42%
Alabama: -7% 42%
Missouri: -6% 16%
Maryland: -5% 39%
Virginia: -4% 29%
The first number is the percentage difference. The second is the percentage of the sample that was black. I'd suggest that the biggest problem facing SurveyUSA in its election eve polling this cycle has been the Reverse Bradley/Wilder effect - the reluctance of black voters to honestly report their support for Obama. These polls have consistently under-predicted the extent of his black support. It's a particular problem for SurveyUSA because, as I noted above, it doesn't massage its data - even though they're probably aware of the issue, they're not going to correct for it.
Of course, black voters are just 14% of their Pennsylvania sample, so even though the 74% support reported in the latest poll is absurdly low, that's not what's going on this time. But it's worth remembering that however elegant the statistical model, we still need to take into account the varied demographics of these different states. Averaging the data actually masks this effect, because it combines states with small black voting populations (where SurveyUSA has mostly been spot-on) with those with large black turnout (where it has not).
And just to reiterate, none of this tells us a thing about the latest SurveyUSA poll or its predictive accuracy. We just don't know how to read polls this far out for anything other than momentum.
April 9, 2008 10:22 AM | Reply | Permalink
Fly - two questions about your analysis:
1) First, you seem to suggest on this thread that SUSA's latest PA poll may be inaccurate in that it is not showing a tightening race. On another thread, you analyzed the difference in undecideds and seemed to indicate that Hillary's lead in the SUSA poll was attributable to a 'break' toward her when forced to choose. So, do you consider the 'break' to be accurate and the polls showing the race tightening to be skewed b/c the large numbers of undecideds?
2) If SUSA is underestimating the black vote as a percentage of the voting population (14% - I think in their last poll it was 18%) in PA and also in how this vote will split in PA (74%), wouldn't both of these factors underestimate Obama's performance?
April 9, 2008 1:42 PM | Reply | Permalink
They're both good questions. I'll take a stab at them.
On the first one, I'll split the difference. It's almost certainly the case (barring a cataclysmic event in the race) that Hillary's margin in Pennsylvania will be higher than the current consensus of polling. In northern states, the consensus has generally pegged Obama's percentage of the vote with a high degree of accuracy, but not the margin by which the race is decided. So it's a mistake to see this race "tied" at the moment. At the same time, simply forcing undecided voters to make up their minds can't account for the entire difference between SurveyUSA and these other polls. Quinnipiac, for example, puts the race at 50-44%, with the same 6% undecided/other. You could tweak that a few points either way, but my guess is that if the election were held today, we'd see something on the order of a 10-12 point Clinton margin.
Which brings us to your second question. Yes, it's entirely possible that SurveyUSA is again underestimating black turnout. And that may well be the other factor at play here - the difference between 74% of 14% (10% of the vote) and 85% of 18% (14%) might well explain the rest of the gap. Even adjusting the percentage voting for Obama, while holding the 14% share constant, gives him a couple extra points (and I'm not sure the lower number is wrong). Either way, it'd be enough to pull Obama's SurveyUSA numbers to 40-42%, more in line with the other estimates. It'd also set their margin at 12-14%, which, IMHO, is roughly where it should be.
Or, at least, that's what seems to make the most sense to me at the moment.
April 9, 2008 2:16 PM | Reply | Permalink
This information is appreciated, but WHEN were the pre-election polls taken?
Without that information, I don't know how to judge what I'm looking at.
April 9, 2008 10:46 AM | Reply | Permalink
see pollster.com for all the appropirate links...
April 9, 2008 10:54 AM | Reply | Permalink
OK, I am back and have just completed a more egg-headed analysis. First, as suggested by the initial post SurveyUSA is pretty damned good. Second, keep in mind the polling this election cycle is very difficult for the reason (1) many younger voters do not have land lines (SurveyUSA seems to have handled that problem well), and (2) the measurement of African-American voters and Hispanic voters remains a major challenge for pollsters. Hence, we have six statistically signficant outliers: CA, SC, MO, AL, VA, and MD.
Here, I need to alert everyone to the difficulty of doing state-by-state comparisions. Polling is as much an art as a science. However, SurveyUSA results do track trends well. In other words, polls are best understood in the context of other polls.
After crunching all the numbers, I can say: (1) SurveyUSA is pretty damned good at what they do, and (2) there is a statistically significant chance that SurveyUSA is more likely to underestimate Obama's performance. This is probably attributable to Obama's attraction to young voters who have only cell phones and do not have land lines and poorer African-Americans who do not have land lines.
Because I'm an egghead, I keep a spreadsheet of polls... what is clear from the data is the SurveyUSA tracks well with other polls in terms of trends. For example, SurveyUSA's trend lines, vis-a-vis HRC's decline in PA vs Obama mirrors that of 7 out of 8 polls.
April 9, 2008 11:13 AM | Reply | Permalink
He's back! Where've you been Fly? The lull between primaries got ya down?
April 9, 2008 12:05 PM | Reply | Permalink
Oh, I've been posting in the threads. I just haven't thought of anything that seems to merit its own post - I try not to post just for the sake of typing the words.
I'm developing a new post this afternoon on polling data, looking at the difference between the predicted levels of support for Obama (very good all cycle long) and the predicted margins (absolutely terrible). Feel free to chime in on this thread
April 9, 2008 12:42 PM | Reply | Permalink
Could this be one of the WORST presidential candidate pools ever? Check out what America's Youth is saying about what each of the candidates have done for their campaign and for their future.
www.YourThreeCents.com
April 9, 2008 3:43 PM | Reply | Permalink
Nice spam.
In answer to your question: No. One of the BEST presidential pools ever.
I can't really even hate McCain, even though I disagree with his policies.
April 9, 2008 5:37 PM | Reply | Permalink
These were the last polls listed here on TPM done by SurveyUSA before or even on the day of election.
I went in hoping that I could find a trend with SurveyUSA polls that would give a strong suggestion as to what to expect in Pennsylvania, regardless of whether Obama did better or Clinton did better in the state.
I just wanted to say that I don't feel comfortable making any solid opinions at this point about the information I provided.
Provided for information purposes only.
SurveyUSA was totally correct on both candidates percentages twice and only in pretty big states. They also were wrong about the winner in a big state and very wrong in a medium-sized state.
In the case of PA, hopefully a newer SUSA poll will help to determine what the trend is and give a better idea of how the race is going.
April 9, 2008 6:39 PM | Reply | Permalink
In fact, ironically enough, it was Philly and its suburbs where Ed Rendell won the votes needed to beat Bob Casey for Governor. Obama MUST do well in the Philly area to make a strong showing against Clinton.
April 10, 2008 6:55 AM | Reply | Permalink