Podcast: John Geraci talks about the problem with polls

John Geraci, President at Crux Research and author of the recently published book “Poll-arized – Why Americans don’t trust the polls” joined us on our podcast. With 30 years in the industry, John spent half of his career at Harris Poll, pioneering techniques for online research and panels. He founded Crux in 2006, a boutique agency. The episode covers the serious data quality issues across all types of research, including election polls. Leaving politics aside, Michael and John discuss some of the reasons for the drop in quality across the insights ecosystem, as well as what prompted John to write his new book. 

John says some of the biggest impacts on data quality have come from low response rates, something which has been largely ignored. If we don’t improve this metric, He believes that survey research and polling will cease to exist as we know it. We have tended to focus on how to retain the percentage of people who enter the survey, rather than focusing on those who never showed up in the first place. An underlying issue here is that most investments in improving technology and methodology have not gone toward improving respondent trust and response. As an industry, we focus almost exclusively on sampling errors, when there are many more consequential errors to consider. 

Trust is falling quickly in election polls, with most Americans not believing that the polls will actually predict the election. During elections, pollsters were being berated in the media. He decided to write his new book because of the uncomfortable feeling after the last two U.S. presidential elections. John was wondering when his clients were going to ask him “if you can’t predict the election, why should I trust you with my complicated research project?” he wanted to have a good answer for this question. 

When he set out to write his book, he ignored what others had written and got a hold of all the polling data himself. He had found that most things written about the polls had not been written by market research professionals. The book provides “a provocative examination of what has gone wrong with US pre-election polls written from the unique perspective of a market research industry insider. Blending actual data from polls, interviews with leading pollsters, and a proprietary survey conducted specifically for this book, it positively reframes the narrative on what’s wrong with our polling system and how pollsters should move forward.” You can find it here. 

John believes that public opinion is essential to any democracy (and capitalism itself), and that we often understate the importance of the insights profession. He says we are the link between politicians and their constituencies, but also what holds together the economic system and makes it efficient. So, it really doesn’t benefit anyone to bash the polls, because like it or not, they remain the best way to measure public opinion, which is critical for us to communicate our needs to our leaders.

The podcast goes on to examine the common errors in polling, what different groups think is wrong with the polls, and some of what goes on behind the scenes. We also dove into the history of where polling “went wrong” (Truman!) and what’s happened since then. Today, demand for insights is at an all-time high, and more and more are making investments in the industry. This means more and more surveys among fewer and fewer people. We are getting faster and cheaper, but not better at what we do, because the low response rates are hurting us across the board.  

Keeping all of this in mind, John says the first step toward building trust in polls is very simple: “getting better at what we do.” We can apply all the methodology in the world, but when we have miniscule response rates it won’t matter. Pollsters are set to lower expectations for the polls – which isn’t a solution to the problem. Right now, with such low levels of trust in the polls, we are in danger of listening to the loudest voices and not the representative ones. Media is set to rely on polls like never before, and lack of trust in polls reflects on a lack of trust in the media. 

He gives ten suggestions in the book on how to turn things around, including doing things like establishing a trusted brand, getting pollsters more in control of the media/pollster partnership, mandating a polling archive, investing more in polls, and more. The biggest thing we can do to improve trust is to capture the large percentage of the population that isn’t currently willing to engage in polls and surveys, and start increasing response rates.