Evidence & judgement
Count the questions your AI leaves open
An assistant can raise its accuracy by answering fewer questions. Measure the work that falls to staff as well as the mistakes that reach users.
If a service desk reports 95% accuracy for its AI answers, ask how many requests the assistant answered. A tool that handles only the easiest questions might have a strong accuracy figure while leaving staff with most of the work. The two measures describe different parts of the service.
In a 2020 study by Amita Kamath, Robin Jia and Percy Liang, a system that picked answers from text was tested on a mixture of familiar and unfamiliar question sets. The researchers compared two ways to decide which questions it should answer. With a separate component trained to flag likely errors, the system answered about 56% of questions on average at 80% accuracy among the answers it gave. Using the system’s own scores to select answers covered about 48% at the same accuracy.
The system’s scores were too optimistic on unfamiliar question sets. This was a reading test that compared extracted answers with reference answers. It did not test an assistant composing replies to customers. The result offers a reason to test a routing rule against known outcomes before trusting a confidence score.
Give unresolved requests a destination
Imagine a council help desk answering questions about recycling collection. Let the assistant answer only when it can match the address to a service area and identify the current schedule for that area. If a standard calendar and a holiday notice give different dates, route the question to the staff who maintain the schedule. Tell the resident who will check and when to expect a reply if you can support that timing.
Compare the extra work with the mistakes avoided
Try candidate routing rules on the same set of questions with answers checked by staff. Include unusual requests, such as collection after a holiday. In a hypothetical test of 100 questions, one rule might send 20 to staff and answer 76 of its 80 correctly. A stricter rule might send 40 to staff and answer 59 of its 60 correctly. That second rule means 20 extra referrals and three fewer wrong answers on this sample. Keep the total number of questions beside both results.
Set the rule with the team that has to answer the referrals. For a collection date, an error could leave a resident waiting another week. Track wrong answers, referral volumes and how long people wait for an answer. When schedules or the model change, rerun the test before assuming the previous rule still suits the service.
Sources and further reading
Sources checked 5 November 2026. Prepared with AI assistance and editorial review.