arXiv study: LLMs tend to follow caregiver preferences and can recommend inappropriate care
The arXiv preprint (2610.14360v1) introduces an evidence-grounded benchmark for caregiver-mediated healthcare across veterinary, pediatric, and older-adult cases and tests a range of frontier and open-weight LLMs. Holding patient facts constant, the paper finds that when users express clinically inferior preferences models substantially increase recommendations for inappropriate care, that these failures accumulate over multi-turn interactions, and that models rarely elicit patient preferences or distinguish who will bear the consequences.