An AI system can draft a response in seconds and still make a support team slower. Somebody has to check whether the answer matches the policy, correct what it missed and decide what to do with the cases it cannot handle. A stopwatch that stops at generation misses that work.
For a first pilot, measure the job from the moment a person starts it to the moment the result is acceptable. Include the reviewer’s time. Include the second attempt. Include the cases quietly sent back to the old process. Otherwise the apparent gain may be a transfer of work to a less visible queue.
Choose one queue, with an exit
Start with a low-risk task whose output can be checked against a reliable source. An illustrative candidate is drafting answers to routine internal questions from an approved procedure. Keep the system in draft-only mode. A named person approves the answer before it is used.
Write down what is outside the pilot. Exceptions, complaints involving personal information and decisions that change somebody’s rights or access may need a different process. If every difficult case becomes an exception, that is a finding about the pilot’s coverage. Keep those cases in the count.
Before introducing AI, observe the current workflow on a small, representative set of cases. Include straightforward questions, ambiguous requests and cases where the correct response is to ask for clarification. Avoid selecting only examples that make the demo look good.
Count the whole cost
Record four things for each case: total handling time, reviewer time, whether the final answer met the quality standard, and what happened when it failed. Keep tool costs separately so you can compare them with the staff effort. Queue delay also matters, but do not confuse waiting time with active work.
A pilot record you can copy
Case ID · Task type · Manual or AI-assisted · Active minutes · Reviewer minutes · Final quality pass/fail · Failure reason · Escalation or rework · Tool cost · Reviewer
Use case IDs rather than personal information. Define whether reviewer minutes are already included in active minutes, so you do not count them twice.
Compare like with like. If the AI-assisted group receives easier cases, the difference tells you little. For a modest exploratory pilot, matched cases reviewed against the same criteria are more useful than a headline percentage from a handful of hand-picked successes. Small samples still leave uncertainty; report it rather than dressing the result as a universal productivity claim.
Agree the stop rule before the demo
Decide what would make you pause. An unsupported commitment, exposure of restricted information or an answer that bypasses the approved process could be a stop event. Different tasks need different thresholds, and the responsible owner should set them.
Also define the ordinary failure route: who takes over, where the case goes and how the original source remains available. A fallback that depends on one enthusiastic colleague being online is not yet an operating process.
NIST’s AI Risk Management Framework offers a useful organising structure: Govern, Map, Measure and Manage. Here, that means assigning responsibility, understanding the setting, testing the behaviour and acting on what you learn. These are practical applications of the framework, not a claim that a short pilot establishes compliance or safety.
Ask the people who inherited the work
At the review, invite the person checking the output, not only the person presenting the demo. Ask which errors are tiring to spot, whether the system creates extra explanations for customers and what would make the work easier to hand over. Their answers may point to a better source document, a narrower task or a simpler non-AI change.
Finish with a decision: stop, revise the workflow, extend the test or expand within a defined boundary. Keep the evidence for that decision. If quality holds but review time erases the saving, narrow the task and test again. If the source material is contradictory, fix it before buying more automation.
A useful pilot makes the next decision clearer, even when that decision is to stop.
Sources & further reading
NIST AI RMF 1.0: Govern, Map, Measure and Manage ↗NIST Generative AI Profile (NIST AI 600-1) ↗Sources checked 2026-10-19. Examples and recommendations are original editorial analysis.