What Dune predicted about AI

The part of Dune I want in our AI conversation is the possibility of people using machines to control other people. That danger does not require a machine to develop ambitions of its own. It requires someone to control a system that others have come to depend on.

Read the line about who holds the machine

In the authorised excerpt of Chapter One, the Reverend Mother explains an earlier surrender of human thinking to machines. She says: But that only permitted other men with machines to enslave them.

This is my reading of a warning about power in a novel, rather than a technical forecast or a complete statement of Herbert's beliefs.

There is an uncomfortable detail in the scene. The person explaining freedom has just subjected Paul to a coercive test. For me, that complicates any easy reading in which removing a machine automatically removes domination. We should examine the authority of the people offering protection too.

Convenience can hide a transfer of authority

Consider a hypothetical organisation that introduces an AI assistant to help employees find procedures. At first, the assistant points to documents. Later, its answer becomes the only explanation staff receive. Eventually, an employee cannot find the underlying rule or reach anyone empowered to reconsider it.

The system may be convenient and often accurate. The governance problem is that a recommendation has become difficult to challenge. That change can happen through decisions about access and procedure without anyone announcing that authority has moved.

A similar question applies when a system ranks applications or recommends which customers receive attention. Who selected the objective? Who can inspect the relevant evidence? Who can correct an error before it changes someone's opportunities? These are proposed checks for such uses, not claims about a particular company or an inevitable outcome.

A person in the loop needs power to act

It is easy to say that a human will review an AI recommendation. The harder question is what the reviewer can do. Can they see the source material, ask for more information, disagree and obtain a correction? Do they have enough time to make that review meaningful?

NIST's AI Risk Management Framework discusses how organisational structure and control of decision processes can influence bias across an AI system's life. It also calls for human responsibilities to be defined. My reading is that assigning a reviewer should come with practical authority and access to evidence.

If an employee is expected to approve every recommendation while lacking either, the person's presence offers little reassurance. Responsibility has been assigned without the means to exercise it.

Keep a route to challenge the answer

I would start with one consequential decision in an organisation. Trace who sets the goal, what information reaches the model, who approves the action and how an affected person can challenge it. Where that route ends at an unexplained output, identify who can review the decision before expanding the system's authority.

The question I would carry from Dune into an AI procurement meeting is specific: whose power increases when we rely on this system, and what can the people subject to it do when it is wrong?