Frozen Scenes, Shifting Winners — audit finds text-to-3D evaluation highly sensitive to render and caption settings

KEY POINTS

1. The arXiv preprint (arXiv:2610.00447v1) audits rendered-image evaluation for text-to-3D by fixing 300 generated scenes from six generators and varying eight render and caption factors across 19 alignment evaluators plus one perceptual-quality control.
2. The authors find that configuration-induced score variance often exceeds between-generator variance, many evaluators change their point-estimate winner under some settings, ranking directions are more stable than scores, and no reversal survives simultaneous inference across the full search; they recommend reporting (generator, score, card ID) and using protocol-dependent comparisons with selection-aware uncertainty.
3. Because protocol choices (camera, render, caption) can dominate measured differences between text-to-3D generators, undermining claims of superiority and requiring reporting and uncertainty-aware comparisons.