Does Every Instance Count Equally? Order effects in LLM multi-instance processing (arXiv:2610.29057v1)
This arXiv preprint evaluates 15 LLMs on a task that requires aggregating binary labels across instance sets and finds substantial sensitivity to the input order: accuracy can vary by about 90 percentage points across seven tested order patterns (five datasets). The authors report that models tend to perform better when majority-label instances appear early, and that available reasoning traces and gradient attribution often emphasize early (and sometimes end) positions more than middle ones.