RESEARCH · RESEARCH · #2002
SRCoD: structure-response co-calibration for item-level diagnosis of multimodal benchmarks
This paper (arXiv:2610.14542v1) introduces SRCoD, a structure–response co-calibrated framework that estimates sample-wise multimodal information structure and uses it as a soft structural anchor in modality-conditioned item-response models to produce calibrated item-level diagnostics and attributions. Experiments using benchmark-refinement and human-labeled validations show SRCoD outperforms prior benchmark-diagnosis methods in detecting modality dependence and shortcut items.
KEY POINTS
- This paper (arXiv:2610.14542v1) introduces SRCoD, a structure–response co-calibrated framework that estimates sample-wise multimodal information structure and uses it as a soft structural anchor in modality-conditioned item-response models to produce calibrated item-level diagnostics and attributions.
- Experiments using benchmark-refinement and human-labeled validations show SRCoD outperforms prior benchmark-diagnosis methods in detecting modality dependence and shortcut items.
- Item-level, interpretable diagnosis of modality dependence helps reveal shortcut or single-modality items that can mislead evaluation of multimodal models and guide benchmark improvement.
WHY IT MATTERS
Item-level, interpretable diagnosis of modality dependence helps reveal shortcut or single-modality items that can mislead evaluation of multimodal models and guide benchmark improvement.