FUNDING · MODELS · #2481
Decision language model Jev rivals many LLM judges for clinical AI evaluation while being far cheaper and faster (arXiv preprint)
An arXiv preprint compares Jev, a decision language model that outputs structured judgments, against frontier and open-weight LLM judges across four clinical-evaluation use cases. On 118 pediatric answers Jev's agreement with physician ordinal ratings (Gwet's AC2 0.50; expected 0.53) matched or exceeded seven of 12 LLM judges (including Claude Opus 5.5 at 0.49 and GPT-6 at 0.38); Jev matched or exceeded several judges on HealthBench and MedHELM tasks and matched GPT-5.4 on 91% of HealthBench Professional criteria, while frontier judges cost 73–294× more and took 7–57× longer.
KEY POINTS
- An arXiv preprint compares Jev, a decision language model that outputs structured judgments, against frontier and open-weight LLM judges across four clinical-evaluation use cases.
- On 118 pediatric answers Jev's agreement with physician ordinal ratings (Gwet's AC2 0.50; expected 0.53) matched or exceeded seven of 12 LLM judges (including Claude Opus 5.5 at 0.49 and GPT-6 at 0.38); Jev matched or exceeded several judges on HealthBench and MedHELM tasks and matched GPT-5.4 on 91% of HealthBench Professional criteria, while frontier judges cost 73–294× more and took 7–57× longer.
- If generalizable, decision LMs like Jev could enable much cheaper and faster automated evaluation of clinical generative AI at scale, lowering barriers to routine quality checks.
WHY IT MATTERS
If generalizable, decision LMs like Jev could enable much cheaper and faster automated evaluation of clinical generative AI at scale, lowering barriers to routine quality checks.