Reading & evidence
Check the pages an AI summary can miss
Language models in a reading test found answers less reliably when the relevant passage sat in the middle of their input.
A community centre’s booking guide might say the building closes at 8 pm, while an appendix gives a 6 pm deadline for returning a hired projector. A summary that gives only the building’s hours could lead an evening group to plan to keep the projector past its return deadline.
The same evidence moved to a different position
Nelson F. Liu and colleagues drew on 2,655 questions from a search benchmark. For each question, they gave models a Wikipedia passage containing the answer and other passages selected to lack the annotated answer. They moved the answer passage to different positions and varied the length of the pack in separate comparisons. They counted whether the reply included an accepted answer.
Models often did best with the relevant passage near the start or end. For GPT-3.5-Turbo, the lowest scores with 20 passages and with 30 passages were both below 56.1%, the score it achieved without any supplied passages. The test sent the passages directly to the models. A file upload feature may select or extract text first, so test the whole workflow you actually use.
Find the passage before you ask for the summary
Check the contents page and appendices for terms such as “equipment hire”, “return” and “collection”. Give the assistant the relevant passage with its heading and page number, plus any section it refers to. Ask it to state the return deadline and identify the words that support it. Compare those words with the original, then use the checked finding in the summary.
If your team routinely reviews long packs, try a controlled check with a synthetic packet and a question whose answer you know. Move the answer passage from the start to the middle, keeping the question and other passages fixed. Record whether the answer is correct and whether the cited passage supports it in each position. If the tool misses evidence when it moves, keep a manual check for those sections in your reading procedure.
Sources and further reading
Sources checked 6 November 2026. Prepared with AI assistance and editorial review.