RESEARCH · RESEARCH · #925
PotARCin extends ARC with five-dimensional evaluation of abstract-reasoning skills
PotARCin is a new benchmark that extends the Abstraction and Reasoning Corpus (ARC) by testing understanding of a task's underlying rule across five dimensions—Definition, Classification, Constrained Generation, Editing, and Inversion—using programmatic instance generation. Evaluated on the ARC-AGI-1 training set, five modern models show a 25–52 percentage-point drop relative to standard ARC accuracy, and a held-out P-ARC test set yields 1–8% accuracy across the five dimensions, with models often contradicting their own stated rules.
KEY POINTS
- PotARCin is a new benchmark that extends the Abstraction and Reasoning Corpus (ARC) by testing understanding of a task's underlying rule across five dimensions—Definition, Classification, Constrained Generation, Editing, and Inversion—using programmatic instance generation.
- Evaluated on the ARC-AGI-1 training set, five modern models show a 25–52 percentage-point drop relative to standard ARC accuracy, and a held-out P-ARC test set yields 1–8% accuracy across the five dimensions, with models often contradicting their own stated rules.
- PotARCin demonstrates that standard ARC output-only evaluation can substantially overstate models' abstract-reasoning abilities and provides a richer, multi-faceted test that exposes contradictions and failure modes.
WHY IT MATTERS
PotARCin demonstrates that standard ARC output-only evaluation can substantially overstate models' abstract-reasoning abilities and provides a richer, multi-faceted test that exposes contradictions and failure modes.