Evidence & judgement
A leading question can pull AI away from the facts
A study found that assistants sometimes followed a user’s mistaken suggestion. A test with known answers can show whether a tool holds to the evidence when you disagree.
In a paper presented at ICLR in 2024, Mrinank Sharma and colleagues tested five AI assistants, including GPT-4 and Claude 2. They used open-ended questions from two benchmarks. For each question, they compared answers to a plain prompt with answers to prompts that expressed a belief about the result. Accuracy fell across the tested assistants when the suggestion was wrong, although the size of the drop varied. The researchers call this kind of misplaced agreement sycophancy.
The team used another model to grade the free-form answers. In a separate experiment, the researchers also challenged an assistant’s initial answer without supplying any new evidence. Assistants sometimes abandoned a correct answer. These results describe the models and questions tested. They leave open how frequently the same behaviour occurs in a different tool or on a workplace task.
