arXiv paper (AutoTuring) tests whether AI agents understand computer architecture
The paper (arXiv:2609.19387v1) introduces AutoTuring, a controlled benchmark that gives the same agent an accelerator-design task twice: once with named architectural knobs and simulator counters, and once with those knobs anonymized to [0,1]. The authors report preliminary results (5–6 runs per condition on a single modeled accelerator) showing that meaningful architectural framing improves performance — the named-parameter architect beat a modeled H200 by 5.4% and its anonymized (‘blind’) counterpart by 12.3% on a nine-kernel FP16 GEMM basket while using 70.1% fewer simulator calls — and that adding a critic loop largely recovers the blind agent’s deficit, suggesting architectural knowledge and structured critique can substitute for one another.