Hey, Elon Musk? Twitter polls are not the ‘voice of the people.’

The Washington Post / Philip Bump

12:32 JST, November 29, 2022

For those seeking input from a large number of people, it is the best of times. It is also very much the worst of times.

The advent of the internet and social media has both made it trivial to cobble together a group of opinions from an audience and created an audience for whatever putative data analysis you might offer. Perhaps you’ve noticed a glut of inexpensive polling from outlets you’ve never heard of. Maybe you’ve noticed the avalanche of news articles informing you that, to make up an example, the favorite ice cream flavor of residents of Kansas is rum raisin, research conducted by the Baskin-Robbins Polling Institute. You can whip up a survey and find an audience for it with all of the ease of filling out an online form.

For those only vaguely interested in the accuracy of what they’re measuring, this is all great. An unprecedented moment for making numbers appear and for putting those numbers in a news release. If, on the other hand, you are concerned about measuring how people actually feel about things? This is pretty grim.

Yes, online panels have expanded the ability of legitimate pollsters to reach people – pollsters who do things like attempt to weight responses to match the composition of the population. But it also means that your precise measurements compete for attention or influence against things that are hopelessly inaccurate, tainted by bias or simply invented.

Which brings us to Elon Musk and his predilection for Twitter polls.

In recent weeks, Musk has outsourced the making of decisions about people banned from Twitter to users of the platform. First, he asked whether former president Donald Trump should be reinstated on the platform, opting to unlock Trump’s account after 52 percent of respondents said he should. Then he asked whether there should be a “general amnesty” for suspended accounts to be reinstated. Again, responses supported the idea.

“Vox Populi, Vox Dei,” Musk replied to the results of each poll – a snippet of Latin that equates the voice of the people with the voice of God.

But of course, Twitter polls are not the “voice of the people” to any significant extent. They are at best the voice of a subset of the people who are on Twitter. Even that is overstating things, though: They are a minor subset of Twitter users, a self-selected group that is aware of the poll being conducted. They are also very possibly automated accounts of the sort that one prominent critic spent much of the year railing against as he leveled criticisms of Twitter as a platform.

That was Elon Musk, naturally – before being forced into upholding his promise to buy the service.

It’s actually not clear whether Musk understands that Twitter polls are not particularly meaningful. In discussing his polls with users on the platform, for example, Musk embraced ideas that would plaster a veneer of accuracy on top of the fundamentally unscientific process.

One user asked whether polls couldn’t be limited to a geographic area by checking a user’s internet address, which Musk said could be done. And it can – though that wouldn’t prevent anyone from manipulating the poll by routing through a different address and it wouldn’t fix the self-selection problem that taints these polls from the outset.

When another user suggested that there should be an “all users” poll pushed to everyone on the platform – an idea that in practice would be indistinguishable from spam – Musk was receptive.

That user also gushed to Musk that with “116.6 million followers, your polls are starting to become statistically significant.” Musk seemed to agree: “When polls are about a significant question, even those who don’t follow me tend to hear about it.”

Neither of these comments uses the word “significant” in a way that pollsters would recognize as useful. Just because a poll goes to a lot of people does not mean its results are significantly representative. If you polled 50 million people about who they thought should have won the 2020 election, that 80 percent of them might choose Trump – because you only polled Trump voters – doesn’t actually tell us anything useful. Nor is a “significant” question that drives engagement meaningful: Who’s engaged and why?

Twitter polls are easily gamed in the way that polls in which participation is controlled are not. I can make a poll on Twitter and send it out to whoever I want, encouraging them to weigh in. Trump, for example, encouraged people to vote in Musk’s poll in a post on his own social media network. If anyone can participate in a poll at will – or make new accounts to increase the number of votes they can cast – that’s not “significant” as a result and therefore not significant as a question to ask. Democratic elections both police and encourage participation in a way that makes them legitimate. Twitter polls do neither, and aren’t.

One would think Musk would know this. One would expect that he would, as an adult who has run businesses involving a lot of numbers. If a Tesla executive collected signatures on a petition from a large group of employees who disliked Musk calling for him to sell the company, would he shrug and say, “Well, the people have spoken”? I propose that he would not.

But the polls give him cover. He can bring back Trump and bring back abusive accounts and point at Twitter users and then blame the users for it. Not his fault! He asked people what they wanted and they said they wanted this, so please keep buying ads from Musk’s Twitter anyway.

Twitter polls are a toy. A lark. They are at best fun in the way the Baskin-Robbins ice-cream survey is fun: you ask a question and people weigh in and you talk about it, like arguing about sports at a bar.

It is not a good method for deciding whether to reinstate accounts that, for example, sent Nazi imagery to Jewish users. After all, even if you could somehow correct for the likely overrepresentation of Nazi supporters in the results, you still have a more fundamental issue: Sometimes the populi say fraught things with their vox.