Language and display
Independent thinking about artificial intelligenceBy Adrian Dunkley
AI BOSS

The daily AI fluency newspaper

AI Says It’s 95% Sure. Should You Believe It?

A confidence score is useful only if it helps you judge the answer. Compare the score with checked results before showing it to staff or customers.

A folded ivory arrow points upward from a navy block. A longer copper arrow lies across the dark surface.
The Long Shadow of a Guess · Editorial artwork · AI Boss Ltd

In a 2024 study, Miao Xiong and colleagues tested ways of asking language models to report uncertainty. On a set of grade-school maths problems, GPT-4 answered 93.6% correctly when prompted to explain its reasoning step by step. It reported 100% confidence on every answer. Those figures came from one model and prompt on that test.

The 100% score was 6.4 percentage points above the observed success rate. Since the scores were identical, they gave a reader no way to tell which calculations to double-check.

Continue reading →
A copper wedge holds a curved cream sheet above a stack of folded navy paper.
A Fresh Page for the Test · Editorial artwork · AI Boss Ltd

Has Your AI Passed the Test, or Learnt the Answers?

Repeatedly fixing the same examples can make a tool look better prepared than it is. Keep a separate set of unseen cases to check what happens next.

Use one set of examples to improve instructions and another, kept private, to assess the result.

Read the essay →

How well do you know your AI?

Practise AI vocabulary and judgement through workplace clues. Choose the crossword, or fill the grid by answering questions in Quiz Mode.

Preview of the AI vocabulary crossword; open the puzzle to see its clues.

The 4Ds framework was developed by Rick Dakan and Joseph Feller in partnership with Anthropic. Framework source.