As AI systems become increasingly powerful, policymakers around the world are grappling with a difficult question: How can society encourage innovation—and the now-plausible race to truly intelligent AI—while managing potentially catastrophic risks?
A new working paper by Ethan Bueno de Mesquita and Wioletta Dziuda of the University of Chicago’s Harris School of Public Policy suggests the answer may depend as much on economics as on technology.
With co-author Mattias Polborn of Vanderbilt University, the researchers developed a model of competition among firms racing to develop artificial general intelligence (AGI), systems that could perform a wide range of cognitive tasks at or beyond human levels.
Their analysis focuses on a central trade-off facing industry and policymakers: resources devoted to moving faster are resources that cannot be devoted to making advanced AI systems safer.
“We started by thinking about market structure,” said Bueno de Mesquita, who is dean and Sydney Stein Professor at Harris. “Should there be one firm? Should there be many firms? Eventually, the paper became more broadly about understanding the incentives driving safety and speed, and what policy levers governments might use to shape those incentives.”
The model begins with two assumptions. First, firms believe there is enormous value in being first to achieve AGI. Second, while AGI could generate tremendous benefits, many researchers and industry leaders also acknowledge the possibility, however uncertain, of catastrophic, existential outcomes if advanced systems are developed or deployed unsafely.
One of the paper's central findings is that a high degree of competition can make the race to AGI riskier. As more firms enter the market, each firm devotes a larger share of its resources toward speed and a smaller share toward safety. The result is faster development and a higher probability of harmful outcomes.
“The more firms you have, the riskier the race becomes as the firms try to outpace one another,” said Dziuda, an associate professor at Harris who also serves as deputy dean for faculty and research.
“Our model challenges the assumption that more competition necessarily produces better outcomes. While competition often benefits consumers and spurs innovation, we show that in a race where being first carries enormous rewards, competition can also create incentives to cut corners on safety.”
In effect, the paper describes a classic collective-action problem. Individual firms may prefer a slower and safer race, but competitive pressure makes it difficult for any one company to slow down on its own.
The researchers find that firms can sometimes benefit from credible commitments to slower, safer development. If one company can convincingly commit to investing more in safety and moving more cautiously, competitors should respond by slowing down as well. The result is a safer race overall—and one that can leave all participants better off.
In other words, being the firm that brings about catastrophe is bad for business, but the negative consequences affect everyone, whether your firm did it or not.
Perhaps more surprisingly, the model suggests there may be circumstances in which firms continue racing even when the expected value of achieving AGI has become negative.
“Why would they still race?” Dziuda asked. “Because other firms are racing. If a catastrophic outcome occurs, whether they're in the market or not, they're affected anyway. So, they may as well participate and hope to be the one that wins."
That dynamic helps explain a conundrum that has emerged in recent years in which some leaders of AI companies have publicly warned about the risks of advanced AI—including with calls for stronger regulation—while also investing billions of dollars to develop such technology.
“These findings offer a new way to think about public calls for AI regulation,” Bueno de Mesquita said. “Calls for industry-wide rules need not be interpreted solely as acts of public-minded restraint. In some circumstances, firms may support regulation because common constraints reduce pressure to sacrifice safety in order to keep pace with rivals.”
The paper examines several policy interventions that have become central to debates about AI governance, including restrictions on computing resources, industry consolidation, public investment and government participation in AI development.
The authors find the effects of these policies are often complex and interconnected.
For example, many proposals focus on limiting access to computing power or other key inputs. But the researchers’ model suggests resource restrictions are not always helpful. In some market situations, especially one where there are fewer players, providing firms with additional resources may actually improve safety outcomes by allowing developers to devote more resources to safety alongside capability development.
“We discovered that the answer isn't always to tax or restrict resources,” Dziuda explained, “it can be a combination of policies that seek to limit the number of competitors and then offer support so that they have room to pursue both safety and new capability.”
The working paper also finds merit in publicly supported AI development. A government-sponsored AI project designed to prioritize safety rather than speed could improve overall outcomes by providing a safer pathway to innovation while encouraging private competitors to behave more cautiously. Switzerland is now advancing a similar model.
—This article originally appeared on the Harris website.
