Measure what your AI reviewers disagree on

In a 2023 study, GPT-4 and Claude agreed on 66% of 3,062 paired verdicts when they judged responses in Chatbot Arena and ties were included. Agreement was 95% among the 1,712 paired verdicts where both chose a winner. The higher figure describes a narrower set. It leaves out cases that a team using AI reviewers would still have to resolve.

In the study by Lianmin Zheng and colleagues, the judges compared two answers to the same prompt. If a judge reversed its choice when the researchers swapped the order of the answers, the study treated that as a tie. These figures compare the verdicts of particular models under a particular setup. They do not show how often the answers were correct or what to expect from a model in use today.

Give each reviewer the same test

Say your team wants AI to flag customer notices that misstate an eligibility rule. Ask a colleague who knows the policy to review a small set of fictional or properly de-identified notices first. Include a correct notice, one that omits an exception and one where the rule cannot be applied from the information given. For each, record what the policy supports and what would justify holding the notice.

Give two AI reviewers the same materials in separate sessions, without showing either the other’s judgement. Ask each to mark the notice as supported, needs correction or needs more information, with the relevant policy passage. Keep the model versions, instructions and policy version with the results. If you use the same model twice, describe the result as a repeatability check.

Count differences and shared misses separately

If the models give different verdicts on 6 of 20 cases, record 6 out of 20, or 30%, as the disagreement in that sample. In a separate column, record any mistake both reviewers miss. Two models approving a notice that omits a valid exception have agreed, but a customer could still be wrongly turned away.

Keep uncertain verdicts visible, even if both models say they need more information. Send those cases and disagreements to the policy owner. If the test finds a shared miss on a rule that affects eligibility, keep human approval for that rule and add the missed case to future testing.

Sources and further reading

• Zheng et al.: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023)

Sources checked 4 November 2026. Prepared with AI assistance and editorial review.