Randomized experiments have long been a cornerstone of scientific research. And many tech companies run randomized tests to learn from the huge amounts of data their customers generate. In 2019, Google conducted nearly half a million experiments on web search alone.
In randomized testing, researchers randomly assign different treatments to different people (or other experimental units) and analyze the outcomes, which are all theoretically independent of each other.
But in practice, one participant’s assignment can affect how another participant behaves, a phenomenon known as interference. If Facebook assigns two users to different experimental conditions, and those users happen to be friends, the behavior of one could affect the other’s, undermining the social media platform’s ability to draw classical causal conclusions.
While this can be problematic, identifying how experimental units interact, and testing how these relationships affect outcomes, can also make experiments more informative and valuable. Research by Chicago Booth principal researcher David Puelz, Booth’s Panos Toulis, Stanford’s Guillaume Basse, and University of California at Berkeley’s Avi Feller demonstrates a graph-based method that can help experimenters incorporate interference into their causal analyses.
In their method, the researchers construct a graph showing each member of the experimental group on one axis and every possible combination of treatment assignments on the other. If a social media company wanted to test how promotional content affects user engagement, for example, all the users in the experiment would be arrayed on one axis and all the content options, or combinations of options, would be on the other.
The researchers then use an algorithm to identify a subset of experimental units and treatment assignments, which they dub a clique, that are relevant to the hypothesis being tested. (For the social media example above, a clique might include females aged 18–29 with friends or followers who are exposed to a particular type of advertisement.) This clique is then used to conduct a randomization test—a statistical procedure used to determine how much variation should be expected between experimental units in each treatment group if the treatment has no effect. Any more variation than that suggests there’s a causal relationship between receiving the treatment and changes in the outcome.