RESEARCH · RESEARCH · #2463
Does Every Instance Count Equally? Order effects in LLM multi-instance processing (arXiv:2610.29057v1)
This arXiv preprint evaluates 15 LLMs on a task that requires aggregating binary labels across instance sets and finds substantial sensitivity to the input order: accuracy can vary by about 90 percentage points across seven tested order patterns (five datasets). The authors report that models tend to perform better when majority-label instances appear early, and that available reasoning traces and gradient attribution often emphasize early (and sometimes end) positions more than middle ones.
KEY POINTS
- This arXiv preprint evaluates 15 LLMs on a task that requires aggregating binary labels across instance sets and finds substantial sensitivity to the input order: accuracy can vary by about 90 percentage points across seven tested order patterns (five datasets).
- The authors report that models tend to perform better when majority-label instances appear early, and that available reasoning traces and gradient attribution often emphasize early (and sometimes end) positions more than middle ones.
- Order sensitivity can systematically bias aggregated LLM predictions, so evaluations and deployments that aggregate multiple inputs should test different input arrangements.
WHY IT MATTERS
Order sensitivity can systematically bias aggregated LLM predictions, so evaluations and deployments that aggregate multiple inputs should test different input arrangements.