Tech Meridian ← LIVE FEED
PROMY MERIDIAN RU

RESEARCH · RESEARCH · #3581

Paper evaluates certify-or-decline proof-step verification for open-weight mathematical reasoning

This arXiv v1 paper studies 'certify-or-decline' proof-step verification applied to open-weight LLMs on GPQA, AIME-2025, and a MATH-500 subset. Certification substantially raises answer-key-match precision (e.g., +31 points on GPQA strong config, +46 on AIME) while answering far fewer attempts (35% and 21% respectively); on near-ceiling MATH-500 gains are ≤5 points with 13–27 point coverage loss. An output-token-matched self-consistency selector attains 82.9% precision at 35% coverage versus the verifier's pooled 87.6%; the authors identify a frequent failure mode where many problems end in invalid proof construction before reaching a judge, and external-LLM acceptance of rendered certified chains (61.0–67.6%) is much lower than the verifier's answer-key match (87.6%). The audits lack human or formal-kernel ground truth and open-weight auditor pairs show near-zero chance-corrected agreement.

KEY POINTS

  1. This arXiv v1 paper studies 'certify-or-decline' proof-step verification applied to open-weight LLMs on GPQA, AIME-2025, and a MATH-500 subset.
  2. Certification substantially raises answer-key-match precision (e.g., +31 points on GPQA strong config, +46 on AIME) while answering far fewer attempts (35% and 21% respectively); on near-ceiling MATH-500 gains are ≤5 points with 13–27 point coverage loss.
  3. An output-token-matched self-consistency selector attains 82.9% precision at 35% coverage versus the verifier's pooled 87.6%; the authors identify a frequent failure mode where many problems end in invalid proof construction before reaching a judge, and external-LLM acceptance of rendered certified chains (61.0–67.6%) is much lower than the verifier's answer-key match (87.6%).

WHY IT MATTERS

The results quantify the precision–coverage trade-offs and operational failure modes of LLM proof-step verifiers, informing how such systems should be reported and deployed and motivating independent validity checks.

SOURCES & TIMELINE

1