Facebook’s answer to privacy concerns? A dating site

What a gift Charlie Brooker’s dystopian series Black Mirror has been to journalists: any time a reporter covers a story about even vaguely dystopian technology, the main thing they have to do is determine which episode of Black Mirror it’s like, saving everyone the bother of finding out what’s really going on.

For instance: ‘Black Mirror is coming true in China’, ‘Black Mirror's chilling social credit score is a reality in China’, ‘A 'Black Mirror' Episode Is Coming to Life in China’. The story here, apparently, is that the Chinese government is implementing a system in some way like the Black Mirror episode “Nosedive”. This portrayed a world in which everyone has a personal score based on ratings given to them by the rest of the population – and in which changes to this score can have severe socioeconomic consequences.

There are some similarities. In both China and the People’s Republic of Black Mirror, there is an app with a personal score that can go up and down depending on how you behave. In both cases, this leads to people suddenly discovering they have been unable to buy airline tickets because of a judgement about their behaviour. But fitting China into a Black Mirror-shaped mould requires you to squint so hard you risk missing the real, arguably more disturbing, story.

One of the most prominent kinds of “social credit score” in China is that issued by Zhima Credit (sometimes translated as Sesame Credit), a subsidiary of Chinese retail giant Alibaba. Through the company’s app, you can (voluntarily) choose to be scored based on a range of things, including not only your credit history, but your behaviour.

If someone is tracked playing video games, they’re probably lazy. Decrease their score! If they buy nappies, they’re probably a parent, and so probably responsible. Level up! (Real examples given by the firm’s technology director.) Similarly, interacting with people deemed trustworthy by the system raises your score, while presumably spending too much time with ne’er-do-well gamers will crash it.

Why do this? Because at the moment an enormous number of Chinese citizens don’t have a traditional credit history, such as the one homeowners build up by paying their mortgage on time. By finding other ways to measure how much of a credit risk someone is, you can maximise the number of people you can lend money to.

The Zhima Credit system has been highly successful in terms of uptake – participants even show off their high scores as a badge of honour on social media. As well as access to better financial products, high scorers are also entitled to perks such as reserving a hotel room without a deposit, or even free umbrella rental. But what if you’re not one of the elite high scorers? Will you really be bumped off a flight?

In 2016, there was a Black Mirror-esque incident: a lawyer named Li Xiaolin found himself unable to buy a plane ticket because of an insincere apology. The apology was court ordered and had officially been judged hollow, reportedly in a part because he’d sent it on April Fool’s Day. But this had nothing to do with Zhima Credit. Instead, the “failed” apology officially meant he’d failed to comply with a court order, which landed him on a government blacklist. These lists contain millions of names, and you can end up on them for anything from spreading false information on terrorism to being caught smoking on a train. The results can be severe: beyond losing access to certain forms of transport it might stop you from buying a house, or sending your children to certain schools.

So what’s the connection between these two things? In 2014, the Chinese government issued Planning Outline for the Construction of a Social Credit System (2014-2020), a document aimed at increasing the amount of “trust” in Chinese society. Part of this plan is about regulating businesses – fraud and financial scams are a massive problem for the country – and the government has set out to establish blacklists of businesses that break contracts, or fail to pay taxes (there are also “redlists” of especially compliant firms). But the same principle also applies to individuals.

The Chinese government has a long history of this kind of data collection; the dang'an (personal record) and the hukou (household record) have been used to control migration and employment for decades. The difference now is that technology makes it so much easier to compile and share the blacklists, which are a collaborative effort between many different departments of government.

Just to make things even more complicated, there are also a variety local “social credit score” initiatives. The city of Rongcheng assigns each resident a starting score of 1,000 – points are deducted things like traffic violations, while they can be earned by volunteering or donating to charity. Especially “good” citizens are celebrated and their pictures displayed outside the town hall, while “bad” citizens might, for example, find it difficult to apply for a government job. The system is designed to be “objective” – you can’t deduct points from your neighbours for being rude. But, ultimately, it does rely on a government-determined list of what counts as good and bad behaviour.

Zhima Credit isn’t completely unconnected to the government’s Social Credit System – it was one of eight firms given a licence to pilot alternative credit score measures – and the blacklists are publicly available, so Zhima is able to factor the lists into its scores. But it isn’t part of an integrated system. To put it another way, a poor Zhima score can’t get on you the blacklist. In fact, the government ended up pulling the plan to let these companies run any kind of official credit scoring system, as it turned out there a bit of a conflict of interest in letting the companies who want to sell financial services determine their own customer’s credit scores.

Is China the real Black Mirror? Alternative credit scoring, the blacklists and the local schemes clearly have things in common, and they each have alarming implications, but they aren’t part of a single, central, system. And the key difference from the Nosedive scenario is that the implied threat isn’t the tyranny of the crowd. Your social credit score isn’t the result of other people judging you, but a representation of either state or corporate power. And this is certainly not a problem unique to China.

Financial start-ups worldwide are developing methods like Zhima’s scoring system to help sell loans to people without a credit score. Facebook holds a patent on a credit score that takes into account the scores of your friends (albeit that it hasn’t done anything with it, yet). Health insurance companies are starting to offer discounts to people who wear fitness trackers. It’s easier than ever to generate, collect and process vast amounts of data about us: to make money, certainly, but as we’ve seen with Cambridge Analytica, there are also political implications.

There’s a lot to be concerned by about the direction that China is heading in, but it is worth unpacking what’s happening rather than treating it as one unified force for ill. Because, whichever episode of Black Mirror it turns out we may or may not be walking into, it’s important that we understand who is collecting what data about us, what they can do with it, and why they might want to.