Model evaluation
AI Says It’s 95% Sure. Should You Believe It?
A confidence score is useful only if it helps you judge the answer. Compare the score with checked results before showing it to staff or customers.

Suppose an AI assistant tells a finance clerk it is 95% confident about which account to charge for a purchase. Before your team relies on that number, compare it with decisions an experienced clerk has checked. Does “95% confident” usually mean the answer is correct, and do wrong answers receive lower scores?
What 100% confidence missed
In a 2024 study, Miao Xiong and colleagues tested ways of asking language models to report uncertainty. On a set of grade-school maths problems, GPT-4 answered 93.6% correctly when prompted to explain its reasoning step by step. It reported 100% confidence on every answer. Those figures came from one model and prompt on that test.
The 100% score was 6.4 percentage points above the observed success rate. Since the scores were identical, they gave a reader no way to tell which calculations to double-check.
Check the confidence score against answers you know
For a finance assistant, have a clerk assign the correct account to each test purchase using the accounts policy that applies to the purchase. Give the assistant the same instructions for each case and record its answer and the percentage it states. Group answers with the same or similar percentages. For each group, record the average stated confidence and the share of answers that were correct. If 14 of 20 answers marked 90% confident are right, the observed rate for that group is 70%. Show how many cases sit behind each rate. A handful of correct answers gives little evidence about the next purchase.
Flag mistakes that could charge the wrong project or grant. See whether they get lower scores than correct answers. If they routinely get the same scores, leave the percentage off the screen and show the relevant policy entry when you can verify it.
Sources and further reading
Sources checked 8 November 2026. Edited and reviewed by AI Boss.