RESEARCH · RESEARCH · #1756
Harness Compilation (arXiv:2610.11231v1): allocate runtime decisions between frozen small VLMs and an external harness
This arXiv preprint introduces Harness Compilation (HC), an offline procedure where a large teacher rewrites reusable content and control from student execution traces to produce an external harness for a frozen small vision-language model; deployment requires no weight updates or teacher calls. Evaluated on seven VQA settings with students up to 9B parameters, HC reportedly improves scores over bare students by 9.9–23.9 points (averaged across three builds per setting), analyzes five runtime decision types, finds that requesting evidence and open query generation are costly while bounded choices and reading supplied text remain useful, and shows fact cards help all students and that HC with 100 practice items can outperform answer-only LoRA on three tasks, with mixed transfer of decision policies and benefits from recompilation or combining HC with training.
KEY POINTS
- This arXiv preprint introduces Harness Compilation (HC), an offline procedure where a large teacher rewrites reusable content and control from student execution traces to produce an external harness for a frozen small vision-language model; deployment requires no weight updates or teacher calls.
- Evaluated on seven VQA settings with students up to 9B parameters, HC reportedly improves scores over bare students by 9.9–23.9 points (averaged across three builds per setting), analyzes five runtime decision types, finds that requesting evidence and open query generation are costly while bounded choices and reading supplied text remain useful, and shows fact cards help all students and that HC with 100 practice items can outperform answer-only LoRA on three tasks, with mixed transfer of decision policies and benefits from recompilation or combining HC with training.
- HC offers a practical, deployment-friendly way to improve small VLMs' use of external evidence without fine-tuning, by measuring which runtime decisions should remain in-model versus offloaded.
WHY IT MATTERS
HC offers a practical, deployment-friendly way to improve small VLMs' use of external evidence without fine-tuning, by measuring which runtime decisions should remain in-model versus offloaded.