The Real Problem With Polls

We need more of them, not fewer.

It has not been a great month for the polling industry.

First, Abdul El-Sayed won his Senate primary in Michigan by only a single percentage point, after the polls said he was on track to clean up by double digits. Then, Francesca Hong narrowly lost her gubernatorial primary in Wisconsin, after polling averages had her ahead by 20 points.

And now, pollsters have to contend with concerns not just about whether they’re wrong — but about whether they’re even real. The Los Angeles Times reported last week that “Median Strategies,” a supposed polling firm that had announced survey results in the Los Angeles mayoral race and Wisconsin gubernatorial primary, had been a complete sham.

The perpetrator was later revealed to be a 21-year-old recent college graduate named Rahil Prakash, who said he was conducting a “short-term social experiment.”

“I wanted to see if fake polls could really penetrate the ecosystem that easily,” Prakash told The Guardian. “And as it turned out, it could.” Before their origin was revealed, Prakash’s “polls” were shared by the mayor of Los Angeles and news outlets including the California Post, the West Coast offshoot of the New York Post tabloid.

These results and revelations have combined to create yet another round of hand-wringing about the role of polls in American politics and journalism. Some commentators, including the former New York Times and Washington Post media critic Margaret Sullivan, are now once again calling on reporters to stop covering polls entirely.

But this is a misdiagnosis. Our problem right now isn’t having too many polls; it’s having too few of them.

I know this might sound surprising. After all, we seem to be living through a time of polling abundance. The number of active national pollsters in the U.S. more than doubled between 2000 and 2022, according to the Pew Research Center. In the 2024 election cycle, more than 2,600 polls were fielded across the country. That’s not enough?

Well, here’s the problem: it’s easier than it’s ever been to field a poll — but harder than it’s ever been to field a good one.

This chart from Pew shows that the recent explosion in pollsters mostly comes from an increase in outfits fielding online opt-in surveys, at the expense of live phone polls, the “gold standard” method that once had a near-monopoly on the industry, and other forms of probability-based surveys.

Online opt-in polls are what are known as “non-probability” polls, where participants voluntarily select to be polled, by responding to ads on a website or social media platform. (Pollsters still generally screen respondents before letting them into the poll.) In a “probability” poll, meanwhile, pollsters use a database of phone numbers or addresses to proactively contact a random sample of the area they’re polling. Live phone polls generally fall into this category.

In both cases, after collecting responses, pollsters will “weight” their pool of respondents to match the demographics of the area being polled: as a very crude example, if an area is 10% Black, but their respondents were only 5% Black, then the Black respondents’ answers would be weighted to count for double, to try to represent the true proportion of Black voters in the area. (In reality, pollsters are trying to capture an accurate representation of a much broader range of demographics at once — race, gender, education, income, etc. — so the calculations are more complicated than this.)

Non-probability polls generally require more weighting than probability polls, since they don’t start out with a random sample: rather than choosing who to contact, they’re at the mercy of who decided to opt in to the poll at the beginning, which could have massively overrepresented some groups. The more weighting a poll has to do to mimic a representative sample, the more potential for error is introduced.

Now, none of this is to say that non-probability polls are all bogus and probability-based polls are all perfect. In fact, some evidence exists to suggest that the two types of polls performed similarly in the 2024 election. But it is worth noting that non-probability polls are much, much cheaper to produce — putting out ads and setting up an online panel for whoever answers is relatively easy; for a phone poll trying to reach a specific sample, 50 phone calls will often yield just one respondent — which means they dramatically lower the barrier for entry for polling.

Not all non-probability polls are bad. But their rise has allowed the zone to be flooded with all sorts of low-quality polls, which end up mucking up poll averages and skewing perceptions of a race.

I often hear from people who say, “How can you still trust the polls? No one answers their landline anymore, especially not young people.” First of all, phone polls reach people via landline and cell phones, just to clear that up. (And, as can be seen by the Pew chart above, few pollsters only use phone polling now. Most use it in conjunction with other methods.) The age critique, meanwhile, mostly gets things backwards.

While phone polling might oversample older voters, online polling often oversamples young voters — and online polling is much more common nowadays, so more frequently, pollsters are dealing with samples that are disproportionately younger than the electorate, in addition to their samples being disproportionately politically engaged (a problem for pollsters using any method, since politically engaged people are more likely to opt in to a poll or pick up the phone for one).

The Michigan and Wisconsin polling misses were likely due to a plethora of reasons. Both states host open primaries, which make it harder for pollsters to predict who the universe of voters will be, since they’re not restricted to just Democrats or Republicans voting. The Wisconsin gubernatorial primary, in particular, contained a lot of upheaval: three top Democratic candidates ended their bids in the final month of the contest, including the eventual winner David Crowley (who only re-entered the race a few weeks before Election Day). If a poll was fielded while Crowley was out of the race, it can’t exactly be blamed for not correctly projecting the result.

But there are also sampling issues at play. In general elections in the Trump era, polls have often overestimated Democrats, since the most politically engaged slice of the country has leaned to the left, skewing polling samples. In these recent Democratic primary elections, it appears that polls suffered from a similar issue: the most politically engaged slice of the electorate has leaned really to the left, skewing polls towards candidates like El-Sayed and Hong. Since so many of the polls were online surveys, they appear to have similarly overrepresented young voters, who also disproportionately vote for more progressive candidates in Democratic primaries.

These ubiquitous online polls are also vulnerable other pitfalls, including respondents merely clicking things and not telling the truth, which can lead to eye-popping (but probably false) results, or being duped by AI bots that pretend to be human respondents.

So you have the existing background problems with polling any primaries, and the problems that popped up in these primaries specifically. Then you have the dominance of online polls, which bring a host of potential problems, including overrepresenting a specific type of voter. Plus, add in the sheer number of new actors in the polling space: some entirely fake, others merely irresponsible (one new polling outfit recently admitted to changing the raw responses they received, an enormous polling no-no), and still more who have an interest in changing perceptions of the race (the Washington Monthly’s Bill Scher has documented just how many polls now are internal polls, conducted and released by campaigns themselves).

Suddenly, you have a toxic stew of polls of varying quality, all of which are flattened and equalized once they’re blasted out on social media and injected into the discourse all looking the same, without much discrimination based on quality. A poll from Median Strategies, the firm that was pumping out fake numbers, is showered with retweets and treated the same as a poll from The Public Sentiment Institute, the firm that was fiddling with its own results, is treated the same as an internal poll, which campaigns only ever release for a reason (either to scare donors into thinking a favored campaign is losing or an underdog campaign has a shot, or to manufacture false momentum and convince voters that their team is winning and they should hop onto the bandwagon now) is treated the same as a poll by the New York Times or the Washington Post.

Any of these polls can then impact what prediction markets like Kalshi or Polymarket have to say about a race. Since actual news outlets now report these numbers as data — CNN has a partnership with Kalshi, for example — these prediction market figures can influence broad perceptions about a race, even though they generally just move in response to polls, and many of the polls are shoddy. Garbage in, garbage out. By the end of this not-so-virtuous cycle, questionable numbers from subpar (and maybe even malicious) pollsters have hardened into conventional wisdom about a race.

After all that, how can I still be saying we need more polls?

Well, we need more good polls. Because of the high expenses involved (and reputational costs when they get things wrong), a number of blue-chip pollsters have exited the game. Gallup stopped polling elections in 2016, and stopped polling presidential approval earlier this year. Monmouth University closed up its well-respected polling shop last year. Iowa’s polling “queen” Ann Selzer stopped producing political surveys after getting the 2024 election very wrong.

This is the void that’s been filled by shoddy online pollsters and internal surveys that are trying to skew perceptions of a race, rather than dispassionately analyzing the electorate.

According to the New York Times, there were 11 polls conducted in the final month of the Michigan Democratic Senate primary. But only one was conducted by a firm deemed a “select pollster” by the Times. (“Select pollsters” must be conducting their survey for a non-partisan sponsor and meet two of the following three criteria: having a “track record of accuracy in recent elections,” being a member of a professional polling organization, and conducting probability-based surveys.) Most of the rest were conducted on behalf of El-Sayed’s campaign, or by other less reliable sources.

In the Wisconsin Democratic gubernatorial primary, there was only two polls in the final month by a “select” pollster (and they were both conducted by the same firm). The Minnesota Democratic Senate primary, another closely watched contest, didn’t see a single “select” poll.

We are in striking need of more high-quality polls of the 2026 midterms; otherwise, we are flying blind, guided by more surveys than ever, but surveys of low or uncertain quality. Think of how many people were absolutely sure about the results of the aforementioned primaries, based mainly on subpar polls and the prediction markets that moved in response to them, without much (or any) mainstream polling being conducted.

This problem isn’t brand-new. In 2022, the last midterm cycle, we also saw a surge of internal polls flood the zone, primarily from Republican sponsors. Many of them incorrectly overestimated the emergence of a “red wave,” skewing the broader narrative before the election — and leading to a narrative afterwards that the polls had gotten the election wrong. But, in fact, most traditional non-partisan pollsters had a good year. It wasn’t a question of listening to the polls. It was a question of which polls you listened to.

This will only get worse as time goes on, as more and more people realize they can make a quick buck by putting out a fake poll to juice prediction markets, and more and more candidates realize they can skew perceptions by blasting out internal polls. “Every candidate in 2028 will have their own cooked poll factory,” the Democratic strategist Lis Smith recently told the New York Times. (And, yes, studies show that seeing that a candidate polls well can increase the likelihood of people voting for them.)

That means that we all need to become more discerning poll consumers. When you see a poll, CTRL+F for “methodology” and see how the survey was conducted. (If the poll offers no information on its methodology, dismiss it out of hand.) Look into whether the poll was conducted on behalf of a campaign, or on behalf of a non-partisan organization or well-known news outlet. Consult the Silver Bulletin pollster ratings to see if the pollster in question has a track record of accuracy (or even any track record at all).

If all that sounds like a lot of work, well, you probably shouldn’t put too much stock in any individual poll anyway. It’s better to look at polling averages, which take an array of surveys into account, especially ones like the Times’, which separates the wheat from the chaff. Many mainstream news outlets didn’t include the surveys by “Median Strategies” in their polling averages — even before the firm was revealed to be fake — because they have a policy of not including pollsters whose methodologies they can’t discern.

Many polling aggregators on social media (including accounts with tens of thousands of followers) failed to put that same work in, and unwittingly promoted the sham numbers. Prediction markets also failed the test, proving liable to being swayed by misinformation: both Polymarket and Kalshi saw Los Angeles Mayor Karen Bass’ odds of being re-elected rise after her campaign shared the fake Median Strategies survey.

Why are polls — good polls, that is — still worth paying attention to, despite these pitfalls (and despite the calls from some corners for media outlets to ignore them entirely)?

Because polls serve an important democratic purpose. We hold elections every two or four or six years; polls are the best tool, in between those contests, for Americans to understand how their fellow countrymen are thinking and for politicians to understand the mood of their constituents.

We all have mental models of the world. But our mental models tend to be very, very wrong. This is one of my all-time favorite charts, from a YouGov poll in 2022 that shows just how bad Americans are at accurately perceiving America, by comparing the proportion of the country that people estimate that certain groups make up, versus the actual proportion of those groups.

We’re also really bad at accurately estimating metrics like crime and unemployment. And the same dynamics hold true in public opinion. We tend to believe our policy positions are more common than they really are, an effect that is even more common among political elites, who are particularly vulnerable to this “egocentrism bias.”

This is why good data, including polling data, serves as such an important corrective in American life.

I often hear from people who express skepticism about a certain poll result, because the people they know in their life don’t reflect that outcome. But think about our vast, diverse country. What are the odds that the people you know make up a representative sample of America? Pretty darn low (especially in an era of geographic and social polarization). Polls, on the other hand, are purposefully designed to capture a representative sample of the country. Between a well-designed poll and your own experience, I’ll take the poll any day.

Good polling gives us the accurate picture of reality that we can’t just seem to construct on our own, which makes voters savvier observers of the country and what’s going on around us and makes politicians more attuned to the opinions of their voters.

The era of polling-driven journalism isn’t perfect, but it is much better than the sort of reporting that came before it — and that some are trying to usher in now — which relies merely on gut instinct, and “vibes,” and interviews with a handful of voters who might not be a representative sample of anything.

That doesn’t mean all polls are created equal, or that you shouldn’t be careful when consuming them. But, when done right, polls help us understand the country, beyond our narrow slice of it, and gird us against falling for fake “vibe shifts” that might show up in the conventional wisdom but don’t show up in the data.

That’s why we need more polls, not fewer. More good polls, at least.