Prophecy and power: why we should be wary of AI’s crystal ball

Humans have always searched for ways to glimpse the future. In Ancient Greece, people turned to the Oracle of Delphi for guidance on everything from war to their personal fate. Today, says philosopher Carissa Véliz, we increasingly look to artificial intelligence for similar answers – whether about crime, creditworthiness, hiring or political risk.

In her new book, Prophecy, Véliz, Associate Professor at Oxford University’s Institute for Ethics in AI, argues that forecasts can function like magnets – influencing events so that they inevitably reach their anticipated outcomes. This tendency becomes more dangerous in an AI-dominated world – AI is ‘prediction on steroids’, Véliz warns, and ‘algorithms are often the hit men sent by companies to carry out their will to power’.

When algorithms influence who gets a loan, a job or parole, predictions become self-fulfilling – reinforcing inequality while appearing objective. For Véliz, this is not just a technological issue but a democratic one. AI-driven forecasting, she argues, concentrates power in the hands of governments and technology companies, while giving an illusion of certainty in an unpredictable world.

So what can be done to help us to live our lives on our own terms? Véliz’s book is a call for both individuals and institutions to act differently – to prepare rather than predict. It’s a wide-ranging book, encompassing philosophy, history and even comedy (Seinfeld, Ted Lasso and Fawlty Towers all get a mention). Ahead of an RSA event, Who controls the future?, Véliz spoke to RSAJournal+ about algorithmic power, self-fulfilling prophecies, and why a true act of rebellion in the digital age starts by reading a physical book.

Mike Thatcher: You describe AI as the ‘new Oracle of Delphi’. Can you explain the link between ancient prophets and modern tech companies?

Carissa Véliz: We are in a historical moment in which we are relying on prediction to make important decisions in many spheres of life, from policy making to sentencing and bailing in the justice system, to decisions about whether we give someone a job or a loan or an opportunity to rent an apartment. And many of these decisions are being made with AI.

If you go back to the past and look at how people made decisions, prediction was also used, even though the methods were very different. Human beings have always asked questions about our health, our jobs, the political situation around us. We’re very anxious about the future. The questions that used to be asked to the Oracle of Delphi, we ask to AI today.

Any kind of prediction is never a fact. At best it is an educated guess, but, more often than not, it's a power play in disguise. It's wishful thinking.

MT: You argue that the main impact of prediction is power over others rather than knowledge of the future. But can you see some circumstances where there is a place for prediction?

CV: Yes, absolutely. I use my weather app every day and I’m not going to stop using it. And, to a certain extent, we need prediction to make decisions in government planning, allocation of resources and in all kinds of situations. But I hope to invite people to think more deeply about what predictions are, what they’re not, what they can do, what they can’t do, when it’s appropriate to use them, when it might not be appropriate to use them.

Just bear in mind that any kind of prediction is never a fact. At best it is an educated guess, but, more often than not, it’s a power play in disguise. It’s wishful thinking. It’s marketing. We should use prediction in a more enlightened way and in a way that accords better with democracy.

MT: Prediction is never a fact, but sometimes could it be the least worst option? Pandemic preparation, for instance, is never going to be perfect, but maybe it’s better to have the prediction than not have the prediction?

CV: Well, if you have no idea of what’s going to happen, it’s not better to have a prediction than not to have a prediction, because a prediction can give you a false sense of security. It gives you a number or an anchor. And if that’s completely wrong and you have no way of knowing what’s happening, that is not helpful.

I make a distinction between preparing and predicting. For example, we know there will be a next pandemic. We have no idea when and we have no idea what it will look like. But we can prepare for it. We can make buildings a lot safer by having clean air and having the air circulate better. So I encourage the mentality of preparation versus prediction precisely because we know that we cannot predict what’s coming.

If I make a prediction about the weather, the clouds couldn't care less about it, and it's going to rain whether I predict it's going to rain or not. But predictions about human beings are different because they change people's expectations.

MT: You say that AI increases risk and creates a false sense of security. Where do you think the risk is most acute right now?

CV: I think there are two areas in which the risk is highest. One is predictions about people, because predictions about things don’t influence the thing. If I make a prediction about the weather, the clouds couldn’t care less about it, and it’s going to rain whether I predict it’s going to rain or not. But predictions about human beings are different because they change people’s expectations.

The second area is where ‘justice’ is the most important value, because using prediction in those areas very often leads to Kafkaesque processes that are not compatible with due process. For instance, if I deny you a loan on the basis that you don’t have £10,000 in the bank, I’m either right or I’m wrong. And if I’m wrong, you can prove me wrong. But if I deny you a loan on the basis that you won’t be able to pay it back, and that’s a prediction, there’s no way you can question me, there’s no way you can challenge it.

When we don’t have clear, transparent and contestable criteria, we don’t have due process. You can’t argue with our prediction, and you can’t argue with an algorithm that is making a decision about you on the basis of our prediction.

MT: How do predictions become self-fulfilling prophesies?

CV: Because social predictions change our expectations, they tend to act like magnets. They bend reality towards themselves. The classic example is Oedipus getting the prophecy that he will murder his father and marry his mother. He freaks out about it and tries to flee his destiny. And, in fleeing, he ends up making the prophecy come true.

But we see this phenomenon everywhere – in the lab, in business, in education, and everywhere in between. We don’t realise how much our expectations shape the future And we see self-fulfilling prophesies very clearly with algorithms, because they partly create the reality they’re purporting to predict. When an algorithm says that somebody can’t get a job, of course they won’t get a job because we’re using algorithms to determine who gets a job.

Self-fulfilling prophecies don’t create error signals. We never get to collect the data of the people who don’t get the job or don’t get the loan.

MT: Your book is described as a call to action, but what can I do as an individual to resist the march of Big Tech predictions?

CV: It partly depends on who you are. We all play a role in society. Maybe you’re a teacher, maybe you run a business, maybe you direct some kind of institution. But there is a lot that we can do.

Predictions can only become self-fulfilling prophecies if people believe them. If we don’t believe the prediction, then it doesn’t change our expectations. And it’s partly a matter of culture, of just realising how dangerous predictions are for democracy. Next time you hear a prediction from a tech executive or someone else with a financial interest, just stop for a second and realise, oh, that’s a prediction. It will lead you to be a lot more critical.

It’s also about minimising your exposure to these kinds of predictions. One excellent example is the importance of reading. When you’re reading a paper book, Big Tech can’t touch you. You’re not getting notifications, you’re not being influenced or nudged, you’re just having a conversation with the author and you’re building your critical thinking skills and your attention. It’s become a true act of rebellion in the digital age to read a paper book.

When you're reading a paper book, Big Tech can’t touch you. It's become a true act of rebellion in the digital age to read a paper book.

MT: So is this a call to embrace a more analogue world, to resist the digital world?

CV: Yes and no, because I think embracing the analogue world doesn’t necessarily mean ditching the digital. So keep your laptop and keep your phone. But one sure way of losing some of the most precious things that we have is neglecting them. And most of the things that matter most to your wellbeing are analogue – from the water you drink and the food you eat to the place you live in and the people you love and the natural world on which you depend.

I fear that sometimes we focus so much on the doom and gloom of the digital, and we forget the brightness and the hope that lies in the analogue.

I fear that sometimes we focus so much on the doom and gloom of the digital, and we forget the brightness and the hope that lies in the analogue.

MT: Geopolitical predictions seem particularly problematic when you have someone as mercurial as President Trump in charge of the world’s most powerful country. How do you make predictions in that sort of world?

CV: You realise that you can’t and you don’t. Geopolitics has always been completely unpredictable. In the book, I mention the start of the First World War and how serendipitous it was [involving a failed assassination attempt on Archduke Franz Ferdinand followed by a successful attempt]. A series of flukes led to that moment. And had that moment not happened, maybe the First World War wouldn’t have started.

It’s not new that you cannot predict your politics, and the illusion of prediction is just making us blinder to risks rather than more enlightened.

MT: So you don’t think the current world is any more unpredictable than the previous century?

CV: I don’t know if it makes sense to even ask the question, because another element for why geopolitics is very unpredictable is because it interacts with phenomena in the natural world that are unpredictable, like a volcano erupting or a meteorite hitting the Earth.

I think what you’re asking me is about the nature of the world. But the response has more to do with our perception of risk, and the current world is perhaps more unpredictable because there’s so much anxiety about uncertainty and because, by having the illusion of predictability, we are creating greater risks than in the past. But there is, at the same time, the fact that the most earth-shattering and life-changing events have always been unpredictable.