Model evaluation
AI Says It’s 95% Sure. Should You Believe It?
A confidence score is useful only if it helps you judge the answer. Compare the score with checked results before showing it to staff or customers.

In a 2024 study, Miao Xiong and colleagues tested ways of asking language models to report uncertainty. On a set of grade-school maths problems, GPT-4 answered 93.6% correctly when prompted to explain its reasoning step by step. It reported 100% confidence on every answer. Those figures came from one model and prompt on that test.
The 100% score was 6.4 percentage points above the observed success rate. Since the scores were identical, they gave a reader no way to tell which calculations to double-check.


