How Statistics Is Getting Better at Linking Cause and Effect
A transformative tool is being used to answer an ever-increasing number of questions.
How Statistics Is Getting Better at Linking Cause and Effect
Martin León Barreto
The chess world is no stranger to scandals, from a 1960s fistfight between grand masters Bobby Fischer and Pal Benko to allegations of another grand master, Vladimir Kramnik, cheating in the 2006 world championship by accessing a phone during bathroom breaks. Just this fall, Kramnik expressed “concerns” on X about grand master Daniel Naroditsky and unspecified other players.
A prominent recent controversy erupted in November 2023, when Kramnik insinuated on his chess.com blog and in a YouTube video that Hikaru Nakamura cheated. Nakamura is a grand master and five-time US chess champion known for his aggressive style of play. Kramnik’s comments were a reaction to Nakamura’s almost flawless 45.5 out of 46 game winning streak in the high-speed chess.com online blitz tournament, where each player only has a total of three minutes of playing time per game. Kramnik pointed out the statistical improbability of Nakamura’s streak and stated that such a winning run would require the chess prodigy to play at a level higher than his current Elo rating (an estimate of a player’s skill level based on their historical play).
Was this just one of those rare streaks that occurs from time to time in sports, or was Nakamura relying on more than just his talent, as Kramnik believed?
The statistical elements of the criticism drew the attention of researchers including Shiva Maharaj of the chess school CHESS-ED, Chicago Booth’s Nicholas Polson, and George Mason University’s Vadim Sokolov. Using statistical analysis to investigate the cheating allegation, they find a 99.6 percent probability that Nakamura did not cheat. Moreover, their research highlights how the improper use of statistical evidence can distort or bias interpretations and lead to flawed conclusions.
For data, the researchers used Nakamura’s performance in more than 3,500 games he played on chess.com, including the 46 games in question. They also compared Nakamura’s Elo rating with those of his opponents, and find that Nakamura was a much stronger player than those whom he played.
This skill imbalance may have been an underestimated factor in the resulting winning streak, the researchers write. After calculating that there was a less than 3 percent chance of Nakamura’s demonstrated winning streak given his opponents’ ratings, the researchers used Bayesian analysis to reexamine the likelihood of it having occurred without cheating. This type of analysis refines an initial hypothesis by continuously incorporating new information to produce a more accurate assessment.
As a starting point, the researchers needed an estimate of the level of cheating that occurs in online chess games. Viswanathan Anand, deputy president of the World Chess Federation, stated in a 2022 discussion with the Hindustan Times that the number of online chess games in which cheating occurs “must be 1 in 10,000.” Using this estimated probability as an initial measure, they were able to calculate the high likelihood of Nakamura’s innocence.
But what if Anand’s estimate was off and cheating was more prevalent? After all, online games have less oversight than in-person tournaments. Maharaj, Polson, and Sokolov recalculated the probability of Nakamura’s innocence using a number of harsher estimations of online cheating. While the probability of his innocence was lower when it was assumed that cheating occurs in 1 out of every 500 games, the probability was high thereafter and sat in the 98 percent range once that estimate rose to 1 out of 1,500 games. This highlights how crucial the initial assumptions are in any analysis, especially when dealing with probabilities.
A real winning streak
Although the likelihood of any chess player winning 45.5 out of 46 online chess games is statistically low, the researchers estimate that the probability that grand master Nakamura won that many games without cheating—given his skill and the rarity of cheating among top players—is high.
Not only are initial assumptions important; improper use of statistical evidence can lead to misinterpretations. The researchers state that Kramnik’s claim—that the low probability of the streak was evidence of Nakamura’s guilt—falls into what statisticians call the prosecutor’s fallacy, a common misunderstanding in statistical reasoning that confuses the probability of evidence given innocence with the probability of innocence given evidence. Just because an event is unlikely does not mean the opposite of the situation must be true. For example, just because a low probability of a winning streak implies a high probability of cheating, that doesn’t mean Nakamura cheated, despite the streak’s unlikeliness.
Nakamura responded to Kramnik’s allegations by arguing that focusing on a particular streak while ignoring other games was cherry-picking. The researchers note that there’s a problem with this argument, too, as it violates the likelihood principle. This principle tells us the interpretation should only rely on the actual data observed, not the context in which it was collected.
The researchers also highlighted another statistical concept, Cromwell’s rule. This cautions against assigning 0 percent or 100 percent probability to an event. Even when things seem either impossible or certain, there may be factors that have not yet been considered or nuances in the data that could change our understanding of the probability.
Overall, the researchers issue a reminder to be critical consumers of information. No matter how rare or unusual an event may seem, before interpreting the data, consider the assumptions you’re making. Otherwise, the data’s framing can affect the conclusions you draw, even if you had no intention of manipulation. And that can damage reputations.
Shiva Maharaj, Nicholas Polson, and Vadim Sokolov, “Kramnik vs. Nakamura: A Chess Scandal,” preprint, arXiv, September 2024.
More from Chicago Booth Review
How Companies Can Run More-Informative Experiments
Researchers have developed a method to explore the relationships between experimental units in randomized control trials.
How Companies Can Run More-Informative Experiments
Can Automated Art Forgers Become Economists?
Competing algorithms can help find answers to complicated policy questions.
Can Automated Art Forgers Become Economists?More from Chicago Booth
Updating results
Your Privacy
We want to demonstrate our commitment to your privacy. Please review Chicago Booth's privacy notice, which provides information explaining how and why we collect particular information when you visit our website.