Why no one really knows how many jobs automation will replace

Tech CEOs and politicians alike have issued grave warnings about the capability of automation, including AI, to replace large swaths of our current workforce. But the people who actually study this for a living — economists — have very different ideas about just how large the scale of that automation will be.

For example, researchers at Citibank and the University of Oxford estimated that 57 percent of jobs in OECD countries — an international group of 36 nations including the U.S. — were at high risk of automation within the next few decades. In another well-cited study, researchers at the OECD calculated only 14 percent of jobs to be at high risk of automation within the same timeline. That’s a big range when you consider this means a difference of hundreds of millions of potential lost jobs in the next few decades.

Of course, technology also has the capability to create new jobs — or just change the nature of the work people are doing — rather than eliminate jobs altogether. But sizing the scope of sheer job loss is an important metric, because for every job lost, a member of the workforce will have to find a new one, oftentimes in an entirely different profession.

Even within the scope of the U.S., the estimates for how many jobs could be lost in a single year vary widely. Earlier this year, MIT Technology Review analyzed and plotted dozens of across-the-board predictions from researchers at places like McKinsey Global Institute, Gartner and the International Federation of Robotics. Here, we’ve charted some of the data they compiled, with some of our own analysis from additional reports:

So why do these predictions cover so much range? Recode asked leading academics and economists in the field and found some of the challenges in sizing how automation and similar technology will change the workforce:

Just because a technology exists doesn’t mean it’s going to be used

Even as new groundbreaking tech becomes available, there’s no guarantee that it will be implemented right away. For example, while autonomous-vehicle technology could one day eliminate or change the jobs of the estimated five million workers in the U.S. who drive professionally, there’s a long road ahead to getting legal clearance to do that.

“The fact that a job can be automated doesn’t mean it will be,” Glenda Quintini, a senior economist at the OECD, told Recode. “There’s a question of implementing, the cost of labor versus technology, and social desirability.”

Take the job of a waiter. A robot may be able to take over some aspects of that job, like taking orders, serving the food or handling payments. But other parts, like dealing with an angry customer, maybe less so. Some studies, such as the OECD report, assess the likelihood of each task within an occupation, while the Oxford studies make an overall assessment of each job.

There’s a debate among academics about which methodology makes more sense. The authors of the OECD report say that the granularity in their approach is more accurate, while the Oxford report authors argue that for most occupations, the detailed tasks don’t matter: As long as technology like AI can do the critical portion of the work, it ultimately has a binary “yes” or “no” capability to be automated.

The data isn’t good enough because it only measures what we know

To model the future, researchers have to start with data from the present — which is not always perfect. Economists do their best to take inventory of all the jobs out there and what tasks they involve, but this list admittedly isn’t exhaustive.

“There’s no assurance in the end that that we’ve captured every aspect of those jobs, so inevitably we might be overlooking some things,” said Carl Benedikt Frey, an economist at the University of Oxford.

It helps to know just how these experts make the predictions to fully understand the room for human error. In the case of the Oxford study, researchers gathered a list of hundreds of occupations and asked a panel of machine learning experts to make their best judgment as to whether or not some of those jobs were likely to be computerized. The researchers weighed in on only 70 out of the about 702 total jobs that they were most confident they could assess.

For the rest of the occupations, the researchers used an algorithm that attributed a numerical value to how much each job included tasks that are technology bottlenecks — things like “the ability to come up with unusual or clever ideas” or “persuading others to change their minds or behavior.” But ultimately, even that algorithmic modeling isn’t perfect, because not everybody agrees on just how socially complex any given job is. So while quantitative models can help reduce bias, they don’t eliminate it completely, and that can trickle down into differences in the final results.

For all these reasons, some academics prefer not to forecast an exact number of jobs lost in a specific timeframe, but instead focus on the relative percentage of jobs in an economy at risk.

“All of these studies that have tried to put a number on how many jobs are going to be lost in a decade or two decades or five years — they’re trying to do something that is just impossible,” Frey said.

Economist John Maynard Keynes famously said that by 2030, due to rapid advancements in technology, we’d see widespread “technological unemployment” and be working an average of only 15 hours a week. It was a positive vision for a world where mankind would finally have “freedom from pressing economic cares” and live a life of leisure. Those estimates seem widely overblown now. While Keynes was right that technology has helped increase productivity in entirely new industries, the average workweek in the U.S. hasn’t declined since the 1970s.

Thanks in large part to persistent wage stagnation and rising income inequality in the last few decades, most people still have to work just as many hours as they did before in order to make ends meet.

Keynes’s comments remind us that there’s a bad track record of punditry in this field, and that even the greats can be wrong when it comes to predicting just how much, or how fast, technology will impact the workforce.