Benchmarks
Has Your AI Passed the Test, or Learnt the Answers?
Repeatedly fixing the same examples can make a tool look better prepared than it is. Keep a separate set of unseen cases to check what happens next.

The familiar test comes back with high marks. The next case may expose a gap. If the questions and answers reached a model’s training data, the score alone cannot tell you whether the model worked out the answers or recognised them. A team can weaken its own test in a similar way by repeatedly changing its instructions to fix the same examples.
How LiveBench planned to refresh its questions
In their 2025 conference paper on LiveBench, Colin White and colleagues described building questions from recent sources and grading them against known answers. One task added misspellings to recent research abstracts and checked whether the model fixed them. The maintenance plan called for replacing about a sixth of the questions in an update and waiting a month before releasing the new batch.
The authors still identified older coding questions and lightly edited maths problems as likely points of overlap with training data. The approach reduces opportunities for exposure, but a date alone cannot establish what a model was trained on.
Save the documents and correct answer before testing
For example, a seed distributor could test stock calculations using invented records. Give the assistant a November stock sheet with five cartons and a supplier bulletin that changes the pack size from 24 to 18 on 1 November. The stock sheet should say the cartons arrived under the new terms. A staff member can save both files and record the expected answer of 90 packets before running the case. If the assistant uses the old size, it could promise a buyer 120 packets.
Keep the cases and answer key in a restricted folder. At test time, give the assistant the stock sheet and supplier bulletin. Use separate examples to improve its instructions. If someone shares a case during a fix, move it into the practice set and replace it with a fresh one that tests the same skill.
If you compare two tools, give both the same batch. Record which set of cases you used with the result, so a lower score after a refresh does not get mistaken for a decline in the tool.
Sources and further reading
Sources checked 8 November 2026. Edited and reviewed by AI Boss.