Tech Meridian ← LIVE FEED
PROMY MERIDIAN RU

RESEARCH · RESEARCH · #3954

System 1 model on frozen LMs reaches 46% on MedMCQA (arXiv:2611.23832v1)

The paper rebuilds a System 1 multiple-choice decision model using frozen language models (Qwen3.5-0.8B, Granite-4.2-3B, Qwen3-8B) and trains only two small projection heads plus a temperature with a contrastive loss; with Qwen3-8B it reaches 46.0% on 3,842 held-out MedMCQA questions (chance 25%), while simpler settings scored ~33%. The authors present ablations (including shuffling and layer-reading changes), note validation gains exceeded test gains, and emphasize this is a cheap, fast baseline rather than a competitive medical QA system compared to Flan-PaLM and Med-PaLM 2 under different evaluation protocols.

KEY POINTS

  1. The paper rebuilds a System 1 multiple-choice decision model using frozen language models (Qwen3.5-0.8B, Granite-4.2-3B, Qwen3-8B) and trains only two small projection heads plus a temperature with a contrastive loss; with Qwen3-8B it reaches 46.0% on 3,842 held-out MedMCQA questions (chance 25%), while simpler settings scored ~33%.
  2. The authors present ablations (including shuffling and layer-reading changes), note validation gains exceeded test gains, and emphasize this is a cheap, fast baseline rather than a competitive medical QA system compared to Flan-PaLM and Med-PaLM 2 under different evaluation protocols.
  3. Provides a simple, reproducible frozen-LM baseline for medical multiple-choice QA that helps contextualize gains from larger or fine-tuned systems.

WHY IT MATTERS

Provides a simple, reproducible frozen-LM baseline for medical multiple-choice QA that helps contextualize gains from larger or fine-tuned systems.

SOURCES & TIMELINE

1