when life-changing decisions are made by machines

Rafael Behr in The Guardian:

Is it possible to have mild tyranny? It sounds like an oxymoron, and certainly not the kind of thing citizens in a democracy might choose. But when you consider the relationship many of us have with technology there is something gently tyrannical involved. In theory we are free to abandon our computer screens, at liberty not to check our phones. In practice we are ensnared in digital networks for most of our waking hours (and longer, for those with smart watches that monitor sleep patterns). In submission to devices, we surrender vast quantities of personal data. Somewhere in the information harvested by powerful tech companies – Amazon, Apple, Facebook and Google – there is a reliable account of where you are, where you are going and who you will see. With a bit of algorithmic extrapolation, it is possible also to predict how you feel.

This knowledge goes beyond any monitoring apparatus established by an authoritarian state. Jamie Susskind writes about a new era of mass “scrutability”, a degree of penetration into our private realms more profound than old-fashioned surveillance. And we sign up for it. We tick the box confirming we have read the terms and conditions, although, of course, we haven’t, because the immediate utility outweighs any abstract cost. Susskind defines this as the “data deal” – a new form of social contract, insufficiently understood by most who enter into it, that entrenches an imbalance of power between the givers of information (you and me) and those who benefit from it (companies and states). He quotes the legal scholar Tim Wu: “Consumers on the whole seem content to bear a little totalitarianism for convenience.”

More here.