RESEARCH · RESEARCH · #2098
Answer configuration determines whether LLMs follow their reasoning trace or current evidence
The arXiv preprint (2610.20805v1) studies how keeping a model's reasoning trace at answer time interacts with verified evidence by comparing two answer configurations: continuation of the model's own thinking versus a separate answer call. Across gpt-oss, Gemma4 and Qwen3 on HotpotQA and MuSiQue, using a separate answer call raises evidence-following by 29.9–60.6 points versus continuation; removing the trace lowers accuracy similarly under both configurations, but continuation disproportionately repeats stale conclusions while a separate call follows current evidence far more often. The paper also finds that the final line of a trace carries most of its value, gold evidence helps most before the trace is written, distractors weaken correction under a separate call, and renewed reasoning (as opposed to a verification instruction) typically follows current evidence in conflicts, leading the authors to recommend reporting both trace benefit and evidence-following alongside answer configuration when evaluating traces.
KEY POINTS
- The arXiv preprint (2610.20805v1) studies how keeping a model's reasoning trace at answer time interacts with verified evidence by comparing two answer configurations: continuation of the model's own thinking versus a separate answer call.
- Across gpt-oss, Gemma4 and Qwen3 on HotpotQA and MuSiQue, using a separate answer call raises evidence-following by 29.9–60.6 points versus continuation; removing the trace lowers accuracy similarly under both configurations, but continuation disproportionately repeats stale conclusions while a separate call follows current evidence far more often.
- The paper also finds that the final line of a trace carries most of its value, gold evidence helps most before the trace is written, distractors weaken correction under a separate call, and renewed reasoning (as opposed to a verification instruction) typically follows current evidence in conflicts, leading the authors to recommend reporting both trace benefit and evidence-following alongside answer configuration when evaluating traces.
WHY IT MATTERS
The result shows that whether a model defers to its earlier trace or to current evidence depends on answer configuration, so evaluations and prompting practices must report and consider configuration to avoid misleading conclusions about model reliability.