Evidence & judgement
Measure what your AI reviewers disagree on
Two models can agree often and still miss the same mistake. Compare their verdicts with decisions made by someone who knows the work.
In a 2023 study, GPT-4 and Claude agreed on 66% of 3,062 paired verdicts when they judged responses in Chatbot Arena and ties were included. Agreement was 95% among the 1,712 paired verdicts where both chose a winner. The higher figure describes a narrower set. It leaves out cases that a team using AI reviewers would still have to resolve.
In the study by Lianmin Zheng and colleagues, the judges compared two answers to the same prompt. If a judge reversed its choice when the researchers swapped the order of the answers, the study treated that as a tie. These figures compare the verdicts of particular models under a particular setup. They do not show how often the answers were correct or what to expect from a model in use today.
