A Forecasting Model Used by the CIA Predicts a Surprising Turn in U.S.-China Relations

The idea for the exercise sprang from the Biden administration’s halting efforts to craft a new Asia economic policy to keep nations out of China’s orbit. (Biden finally launched what he calls the “Indo-Pacific Economic Framework” on Monday.) Jon Grady, a graduate student at New York University’s Stern Business School, proposed running a game theory exercise to forecast the next steps the U.S. and China would take in their economic rivalry. Grady, a preternaturally patient 32-year-old with a shaved head, licensed game theory software from Bruce Bueno de Mesquita, an NYU game theorist so prominent in the field that he’s known simply by his initials BDM.

BDM’s game theories are different from war games where experts play the roles of political leaders during a crisis. It’s also very different from simply asking experts in the field what actions either government might take. Bueno de Mesquita and other game theorists contend that experts do a crappy job of making predictions and war games lack rigor.

“An expert may be very, very skillful at telling you a very good story about Chinese history and telling good stories about possible future actions,” says Philip Tetlock, a University of Pennsylvania political scientist who has won a Pentagon forecasting competition. “That doesn’t translate all that well” into making predictions. Political experts are only slightly more likely to make accurate calls than a chimpanzee tossing darts, he says.

Forecasting strategies vary. The BDM model looks at power relations, figuring that decision-making is a battle of people following their self-interest and self-aggrandizement. Although the president has the most power in Washington, for instance, he is often thwarted by other power players who are more focused on an issue (i.e., the National Rifle Association on gun control) or inflexible (Joe Machin on Build Back Better). The model doesn’t try to assess the best possible resolution of a problem or the decision that would best reflect the national interest. The BDM model is more akin to assessing scorpions in a bottle.

For our exercise, we recruited experts to serve on the U.S. or China teams. The American team consisted of several people close to the Biden administration, including Matthew Goodman, an Obama National Security Council economic official; Daniel Russel, an Obama State Department Asia official; and Barbara Weisel, a former senior U.S. trade negotiator. Jeremie Waterman, who runs the U.S. Chamber of Commerce’s China Center, and I were also part of that team.

On the China side were Lucy Hornby, a former Financial Times China reporter who is now a visiting scholar at Harvard; David Dollar, Obama’s Treasury representative in Beijing; James McGregor, who has spent decades in China representing American business; and Ryan Hass, an Obama NSC China expert.

Matthew Goodman, Center for Strategic and International Studies

Daniel Russel, Asia Society Policy Institute

Barbara Weisel, Rock Creek Global Advisors

Jeremie Waterman, U.S. Chamber of Commerce China Center

Bob Davis, former Wall Street Journal senior editor

Lucy Hornby, former Financial Times China reporter

James McGregor, APCO Worldwide, greater China region

The game was designed in February to forecast whether China would ever meet its obligations to boost purchases of U.S. goods by more than $200 billion, as required under the Phase One trade deal negotiated under former President Donald Trump. And if Beijing didn’t — as seemed certain even before the game started — what the U.S. would do in response.

Ultimately, the game stretched on through the end of April. By that time, the trade deal had become somewhat irrelevant as Washington’s focus turned to Russia’s invasion of Ukraine and the impending midterm elections. Biden’s recent comments on whether the U.S. would defend Taiwan have also unsettled some China watchers. Fortunately, the game also examined broader questions of what economic moves the U.S. would take next to influence China and how Beijing would respond.

Developed by New York University political scientist Bruce Bueno de Mesquita, the forecasting model make predictions by assessing power relations among decision makers. The model looks at four characteristics: decision makers’ influence, their positions on an issue, their negotiating flexibility, and their focus on the issue. The model’s algorithms calculate millions of different possible interactions, and using game theory, spits out the most likely outcome.

“The focus on China right now is much more on how they’re going to respond to Russia” than whether they will carry out a trade deal, said Goodman, the ex-NSC Asia hand now at the Center for Strategic and International Studies, during an April 8 U.S. team meeting.

Under Grady’s prodding, the two teams were asked to list those in Washington and Beijing involved in deciding what to do next on U.S.-China trade relations. Then they would rank each player on a scale of zero-to-100 on four attributes: their influence, the importance of the issue to them, their position on the issue and how flexible they would be during negotiations.

In Washington, the team came up with two dozen power players, including White House officials, cabinet secretaries, business and labor lobbyists and four shades of lawmakers — progressive Democrats, labor Democrats, free-trade Republicans and agriculture state Republicans. For China, whose political decision-making is wrapped in mystery, the exercise was a struggle. It was tough to know who to pick beyond members of the ruling Politburo Standing Committee and some prominent economic and foreign policy advisers.

“I look at this more as a Tony Soprano setup where if [the top leaders] want something, they’ll figure out the corruption in each of the families and threaten them” to get what they want, said McGregor, chair of APCO Worldwide’s China region, who started in China as a Wall Street Journal reporter.

Next, each team was asked to rank the influence of the players — essentially how much clout they could bring to bear on the decision should they choose to battle for it. Joe Biden and Xi Jinping were ranked as 100. What combination of other players could equal Biden or Xi’s power, Grady asked the teams.

“It forces people to think logically and numerically about an issue,” Grady says.

The question initially stumped the team members. That’s not how decisions are made, they argued. White House chief of staff Ron Klain isn’t going to line up three cabinet members and march into the Oval Office and demand Biden make one decision or another.

Think of it differently, Grady said. Imagine that a group of officials argued to Biden in favor of a decision. Would they be able to convince him to accept their recommendation? Ditto with Xi and Chinese officials? What combination of officials would then equal the leader’s score of 100?

Still, the discussion had an otherworldly feel to it. It was like the conversations I had as a kid: If Batman and Wonder Woman fought with Superman, could they defeat him?

Eventually, the teams came up with rankings. The U.S. team decided that Klain had roughly half the influence of Biden, given his long relationship with the president and the broad mandate of a chief of staff. Then they ranked others compared to Klain. Janet Yellen had just about one-third of Klain’s influence on the China trade issue, they decided, and U.S. Trade Representative Katherine Tai had about half.

On the Chinese side, the decisions were tougher, given how thoroughly Xi dominates politics there. By some accounts, he is the most powerful Communist Party leader since Mao Zedong. Eventually, the Chinese team decided that Vice Premier Liu He, Xi’s economic envoy to the U.S., had about one-third of Xi’s influence largely because of his mastery of economic issues and his deep roots in the Chinese bureaucracy. Premier Li, who has been overshadowed by Xi and seen much of his economic portfolio stripped away, had just about 10 percent.

On and on the assessments went. How would the teams rank each of the players, from zero-to-100 on their passion for the issue — what Grady called the issue’s “salience.” How flexible were they? What was their real position on U.S.-China relations, not just what they said in public?

At various points, the choices seemed arbitrary. How could anyone know what Wang Yang, a member of the Politburo Standing Committee who is viewed as having a shot at succeeding Li as premier, really thought about trade relations with the U.S.? Would he be more flexible in dealing with the U.S. because he might think that could improve China’s economic outlook? Or might he be less flexible for fear of being seen as weak?

McGregor broke the tension in the fourth and last meeting of the China team: “You guys are getting lost in the real world here,” he said. “This is a game.”

Still, the game players’ reactions reflected one of the biggest problems of forecasting. No matter how sophisticated the model, a model’s accuracy depends on the quality of the information it processes. Garbage-in, garbage-out.

To convince the players of the reliability of the BDM model, Grady would point to a Central Intelligence Agency assessment. The CIA compared the predictions by the BDM model in the 1980s to one it used and found BDM’s “got the bull’s eye twice as often.”

“The CIA study found [the BDM model] was accurate 89 percent of the time,” said Stanley Feder, a former CIA analyst who ran the review. “We used it pretty extensively at the CIA.” The study was declassified in 1995 and has been a marketing calling card for Bueno de Mesquita ever since.

Bueno de Mesquita has had some big hits. He correctly forecast the successors to the Ayatollah Khomeini five years before the Iranian leader died. But he admits to some misses too, including getting nearly every prediction wrong about how Bill Clinton’s health care bill would turn out. House Ways and Means Chair Dan Rostenkowski’s indictment, he says, messed up the model’s assumptions about his influence in pushing through legislation.

By the end of the game, Grady had U.S. team members put together a list of possible U.S. actions toward China with the lightest being removing all tariffs immediately and the harshest hitting Beijing immediately with a fresh round of sanctions without even bothering to seek industry input. In between was starting a new trade action that could eventually lead to sanctions.

As the U.S. players assessed how American authorities would look at the options, China’s behavior toward Russia became more important. Since President Richard Nixon unfroze relations with Beijing in 1972, trade has played a crucial role in bringing the U.S. and China closer together. Whatever the vast differences between the two political systems, both countries recognized that trade could benefit them by expanding markets and places for production. Even during the Trump administration, the two countries eventually signed a deal to end trade hostilities.

Russel, the former Obama State official who is now a vice president at the Asia Society Policy Institute, a think tank, laid out the dilemma facing Biden officials. “Every week we get closer to the midterms, there will be pressure to do something that looks strong on trade,” he said in the third U.S. session on April 8. But on the other hand, “they don’t want to rock the boat” because they want to convince China to “exercise some degree of restraint in material support for the Russian military operation.”

As the game drew to a close at the end of April, the conundrum had the effect of elevating the power of Janet Yellen. That’s because the U.S. team saw her as laser-focused on removing tariffs, which Yellen complained were having an inflationary impact — although she has been talking tougher on China recently. Easing inflation could help Biden politically as well as act as an opening to the Chinese, which scored well in the forecasting model.

On the Chinese end, a similar dynamic was elevating the importance of Liu, the economic adviser, and Li, the premier. Both are responsible for China’s economy, which was starting to sputter due to unending Covid lockdowns. An easing of tariffs, especially if it indicated a possible rapprochement with the U.S., could help.

The BDM model uses the following equation: Influence X Salience = Power. While the influence of Yellen and Li started out as minimal, their salience — their focus on the trade issue — grew immensely as the game played out. Ergo, their power grew.

“Salience inflates the clout of some players that look like they don’t have much more clout because they care intensely,” says Bueno de Mesquita.

Once the game players ranked officials from zero-to-100 in the various categories, the computer model took over, with algorithms that used those rankings to play out millions of possible interactions between officials in both countries. The formulas look to assess the clout each country’s players have, how focused they are on the issue and other criteria. It also simulates interactions between U.S. and Chinese players.

The bottom line: In the U.S., Yellen’s power grew on the trade issue over time while Klain, who the team viewed as mainly interested in politics, saw his shrink. Biden went along with a position of eliminating most — but not all — tariffs on Chinese consumer goods. The computer model didn’t explain itself. It spit out spreadsheets filled with numbers. But a reasonable explanation for the finding would be that cutting some tariffs, but not all of them, could be presented as an overture to China but not caving in.

On the Chinese side, economic officials, including Premier Li and Vice Premier Liu, turned out to be much more flexible in negotiations than expected and had enough clout to convince ideologically minded officials to go along with them. Although all Chinese officials started off by demanding that the U.S. immediately lift all tariffs, the model forecast that the partial rollback would be welcome, and China would be open to negotiations where it would be willing to increase its U.S. purchases.

None of the experts recruited to play the game predicted the outcome forecast by the model, although a few said they hoped the model would turn out to be accurate. Most figured that the Biden team would pair any tariff relief with some aggressive action toward China too in order not to look weak.

1) Expect a thaw in U.S.-China relations.

2) Russia’s war on Ukraine changes the equation on trade.

3) Treasury Secretary Janet Yellen and Premier Li Keqiang both pack a punch.

Grady said it wasn’t unusual for the players to disagree with the outcome of their work. As the U.S. Chamber’s Waterman noted, “We weren’t playing a war game, we were helping build an economic model” through assessments of power and influence.

Biden and Xi, who initially showed no interest in a deal, would go along, the model predicted, in part, because so much of their attention was now focused on other issues and because a deal met their needs. In fact, Biden said earlier this month that he was weighing whether to ease the tariffs. In the game, as in real life, the leaders make compromises they didn’t expect to make.