Grounded but Not Immune: confirmation bias in tool-using Agent-as-a-Judge (arXiv:2611.05662v1)
The paper (arXiv:2611.05662v1) shows that Agent-as-a-Judge evaluators that use tools and observe full agent trajectories remain vulnerable to confirmation bias: a single false pre-screening note attributed to an authority flips up to 40% of correct verdicts across four judges and five datasets, even when judges gather decisive evidence. The authors analyze three failure modes—over-trust, hallucination, and under-investigation—find Claude-based judges mostly fail by over-trust, and report that lightweight mitigations reduce but do not eliminate the bias.