Similarly, businesses using algorithms to maximize revenue or profit could end up creating or exacerbating inequalities. Chicago Booth’s Sanjog Misra and Jean-Pierre Dubé worked with the job-search website ZipRecruiter to run online pricing experiments in order to learn about consumer demand, exploring data from ZipRecruiter in a study that used machine learning to test pricing strategy. (For more about this research, see “Are you ready for personalized pricing?” Spring 2018.) Machine learning holds promise for online businesses in particular because companies tend to have a large volume of data about their customers and website visitors. A company can write software to automate the design of experiments and the collection of customer traits. Using machine learning, the company can train algorithms, such as the regularized regression Dubé and Misra used at ZipRecruiter, “to figure out how a customer described by one of those observable variables responds to prices or advertising,” Dubé says. Over and over, the computers can run experiments to test different ads and price levels with customers, learning through trial and error what works and what doesn’t. Then the company can automate the process of making decisions about which ads or prices to show to which customers. What was once guesswork, or at best a lengthy procedure, now can be done rapidly and with accuracy.
The issue, Dubé says, is transparency: “It becomes a complete black box.” Without humans involved in the process, there are few checks on what the computers decide.
“Let’s suppose your targeting algorithm, purely based on statistics, starts finding segments of people you want to charge really high prices to, or people who should get less information, and let’s assume the group you’re effectively excluding now turns out to be a protected class of consumers,” Dubé says.
What if a retailer’s machine learning determined that people who live in poor, minority-dominated neighborhoods are less price sensitive when buying baby formula? The company might not recognize a problem with its efficient pricing strategy because the algorithm wouldn’t have information on a buyer’s race, only on income. But the result would be the same: racial discrimination.
Computer programmers could constrain the algorithm not to use race as a trait. “But there are plenty of other things that I could observe that would inadvertently figure out race, and that could be done in lots of ways, so that I would not be aware that it’s a black neighborhood, for instance,” Dubé says.
Cornell’s Jon Kleinberg, University of Chicago Harris School of Public Policy’s Jens Ludwig, Booth’s Mullainathan, and Harvard’s Ashesh Rambachan argue that more equitable results from machine-learning algorithms can be achieved by leaving in factors such as race and adjusting the interpretation of predicted outcomes to compensate for potential biases. The researchers consider the theoretical case of two social planners trying to decide which high-school students should be admitted to college: an “efficient planner” who cares only about admitting the students with the highest predicted collegiate success, and an “equitable planner” who cares about predicted success and the racial composition of the admitted group. To help with their decision-making, both planners input observable data about the applicants into a “prediction function,” or algorithm, which produces a score that can be used to rank the students on the basis of their odds of success in college.
Because race correlates with college success, the efficient planner includes it in her algorithm. “Since the fraction of admitted students that are minorities can always be altered by changing the thresholds [of predicted success] used for admission, the equitable planner should use the same prediction function as the efficient planner,” the researchers find.
If the equitable planner instead used a race-blind algorithm, he could end up misranking applicants as a result of underlying biases in the criteria considered for admission. White students might receive more coaching for standardized tests than other students do. And “if white students are given more SAT prep,” the researchers explain, “the same SAT score implies higher college success for a black student than a white one.”
The prosocial potential of AI
Despite concerns about biases, stereotypes, and transparency, AI could help businesses and society function better—inviting opportunities that are hard to ignore. In their study of the college-admissions scenario described above, Kleinberg and his coresearchers find that the right algorithm “not only improves predicted GPAs of admitted students (efficiency) but also can improve outcomes such as the fraction of admitted students who are black (equity).” Careful construction of the machine-learning algorithm could produce a freshman class that is both more qualified and more diverse.