An uncontrolled acceptance of AI.

Before you expand an AI system's authority, estimate the financial exposure before someone can stop it. Then test the controls against the people and decisions they are meant to protect.

My work on AI adoption, staff training and return on investment keeps bringing me back to a question: what exactly are we accepting when we accept an AI answer? A useful draft, a recommendation and permission to act are different things.

Take one workflow you are considering automating. Write down its current permissions and the worst action it could take within them.

The objective can be wrong even when the answer looks right

For organisational decisions, I use alignment to mean whether an AI system's behaviour serves the intended purpose while respecting the people affected and the constraints agreed for its use. Safety asks what harm the system could cause, including when it follows its instructions.

Consider an illustrative customer-service assistant rewarded for closing cases quickly. It can improve the closure count by discouraging difficult complaints. The metric improves while the service deteriorates. Changing the wording of its replies will not fix an incentive that rewards the wrong outcome. Amodei and colleagues' Concrete Problems in AI Safety distinguishes problems involving objectives, supervision and behaviour outside familiar conditions.

Acceptance becomes uncontrolled when a satisfactory demonstration is treated as evidence for a broader permission. A model that drafts a sound response has not thereby demonstrated that it should send messages, alter records or settle disputes. Each added permission changes the possible consequences and needs its own evidence.

Borrow a failure from automated trading

The Knight Capital incident is a useful warning about automation, although it was not a generative-AI incident. The US Securities and Exchange Commission reported that erroneous automated trading in August 2012 left the firm with losses exceeding US$460 million. Its findings included inadequate deployment controls and safeguards against excessive exposure.

If an agent can issue instructions faster than your team can inspect them, the inspection process needs an enforceable limit on what happens while people investigate. An alert with no effective response is a record of danger, not containment.

SEC enforcement official Daniel M. Hawke asked what safety nets are in place to limit the harm. For an AI workflow, ask: how much can the system commit before the organisation regains control?

Use S4 to decide what AI may do

My S4 AI Safety Framework brings four checks into the permission decision: Specify, Stress-test, Safeguard and Scrutinise. It draws on established risk management and safety engineering.

NIST's AI Risk Management Framework 1.0 organises its core around Govern, Map, Measure and Manage. Governance and consultation apply throughout S4.

The four Ds of AI fluency were developed by Rick Dakan and Joseph Feller, who partnered with Anthropic on the course. Fluency concerns how people work with AI. S4 concentrates on the evidence and authority surrounding a particular deployment.

Measure exposure before arguing about a safety score

Start with an intentionally simple stress scenario. Let a be consequential actions per minute, t the minutes until effective containment and c the maximum direct commitment per action. An illustrative exposure envelope is E = a × t × c. The units are currency. It assumes a constant action rate, maximum commitment on every action and no recovery during that interval.

If an assistant can authorise 12 payments a minute, each capped at US$500, and containment takes 10 minutes, the envelope is US$60,000. Reducing effective containment to two minutes makes it US$12,000. These are hypothetical commitments at risk, not observed losses, expected losses or probabilities. Recovery, fraud, knock-on effects and legal consequences could change the eventual loss substantially.

A separately enforced aggregate cap can limit the direct commitment in this scenario. If that cap is US$5,000 across all relevant accounts and workers, the model takes the smaller of the calculated envelope and the cap. A cap applied independently to several agents does not automatically create a shared limit. Test its scope, concurrent actions and behaviour when the cap is reached.

The accompanying calculator lets you vary these assumptions and export the result. It also calculates an annual expected direct loss for a different, explicitly stated model: actions per year × assumed harmful-action probability × average direct loss per harmful action. Keep the annual estimate separate from the short-interval envelope. An assumed probability is not evidence about your deployment.

I would use a range of assumptions to decide what deserves further testing. I would also set non-negotiable constraints on unacceptable harm. A small estimated average financial loss cannot settle questions about discrimination, privacy or an irreversible decision affecting a person.

Ask who can challenge the decision

Ask someone who does the work to describe a failure the dashboard would miss. Ask an affected person what a fair appeal would require. Then record disagreement in the decision log, including the evidence that would change each person's view.

A useful implementation might keep an assistant drafting routine replies while staff retain sending authority. A second workflow might permit tightly capped actions after independent approval. Treat both as design options to evaluate, rather than assuming that greater autonomy is the natural destination.

A restriction can help adoption when it lets people attempt a useful task without accepting the full consequences of an unrestricted system. Test that assumption: compare review burden, task completion, missed harms and staff willingness to report problems. A control that merely drives work into an unapproved tool can make observation worse.

Choose one workflow today. Complete the canvas with its owner and a colleague who can challenge the objective. Test the stop mechanism using non-sensitive examples. Leave consequential actions disabled if the owner, permission boundary or recovery route remains unresolved.

This issue builds on assigning an owner to AI exceptions and testing questions a demonstration would avoid. The next issue, Lessons from Colonization: AI Misalignment., will examine who sets the objective, who receives the benefit and who retains the ability to refuse.