Testing & operations
Test a new AI model against the cases you already handle
An automated judge accepted fewer replies from a newer GPT-4 version on 50 Python problems. Removing extra text reversed the comparison.
In a study of two 2023 GPT-4 versions, the share of answers that passed a programming judge as delivered fell from 52% in March to 10% in June. After researchers removed surrounding noncode text, 52% of the March answers and 70% of the June answers passed. A developer could see a usable solution on screen while a system receiving the raw reply rejected it.
What the programming judge accepted
Lingjiao Chen, Matei Zaharia and James Zou gave both versions 50 Python problems from LeetCode’s easy category. Each prompt included the problem and its Python template. The researchers submitted each reply as received. The judge accepted an answer only if it was valid Python and passed its tests. One June answer wrapped the code in formatting marks that stopped the judge from running it. The rates apply to the two API versions and the problems in this test.
Give both versions the same inputs
Suppose a venue uses AI to read booking confirmations and return a room code and start time for a calendar import. Build a small set of approved or fictional examples covering routine bookings and mistakes you know the importer must catch. Include a missing room code and a changed start time. Write down the expected fields and when a person should review a case.
Test both versions in a separate environment using the same inputs, instructions and importer. Check whether the fields are correct, then check what the calendar actually accepts. A reply can contain the right room and time but still fail if it adds text where the importer expects only fields. Repeat consequential cases to see whether the failure recurs.
Set the release conditions before you compare, then review each case the current version passes and the replacement fails. Withhold the switch if the new version imports a booking with the wrong time or leaves a case you marked for review unflagged. If the previous version is unavailable, compare against saved outputs from the same inputs and record any differences in setup before drawing conclusions.
Sources and further reading
Sources checked 6 November 2026. Prepared with AI assistance and editorial review.