Testing & work
How AI handles a brief with missing information
In OpenAI’s GDPval benchmark, GPT-5 did worse when researchers stripped context from professional tasks. Test what happens when your own inputs are incomplete.
OpenAI’s researchers tested GPT-5 on professional assignments with some of the instructions removed. On an earlier version of their GDPval benchmark, expert graders preferred GPT-5’s output to a professional’s less often when the researchers removed details about finding data, approaching the task or formatting the finished work. For a team choosing a tool, that is a reason to test the briefs people actually receive, including the ones they have to send back for more information.
What the judges could compare
GDPval asked models to produce work drawn from 44 occupations in nine large sectors of the US economy. On the public subset of 220 tasks, occupational experts assessed pairs of unlabelled outputs against the request and reference files. That public subset describes the main benchmark, not the earlier GPT-5 experiment. The researchers shortened instructions; the trial below instead tests a missing input.
The benchmark tested finished files against a defined brief. It did not follow an employee through a week of settling conflicting requests, seeking approval or taking responsibility for a decision. Treating a file comparison as a staffing forecast would skip those duties, the distinction explored in our earlier piece on job replacement.
Leave out an input on purpose
For a low-risk trial, use authorised past cases or make synthetic ones that preserve the difficulty. Give the system and a colleague the same brief and access to the same materials. Ask an experienced reviewer to judge outputs without knowing who produced them where that is practical. Agree in advance what counts as a correct result, a useful request for clarification or a serious error.
In a hypothetical purchasing trial, give both participants a request to compare delivery dates across supplier quotes. In some cases, withhold the date for one supplier. A report that identifies the gap and asks for the missing delivery date can be more useful than one that ranks every supplier by silently filling in a date. Have the reviewer check the dates against the supplied records before scoring presentation.
Record the ordinary and incomplete cases separately. Track review and correction time alongside the result; a draft that looks finished can still leave the reviewer with the hardest part of the work. If the tool handles complete files but guesses when they are incomplete, restrict the pilot to cases where a person has already confirmed the required inputs.
Sources and further reading
- Patwardhan and colleagues: GDPval, arXiv:2510.04374v1, 5 October 2025
- ICLR 2026: GDPval conference record
Sources checked 4 November 2026. Prepared with AI assistance and editorial review.