RESEARCH · RESEARCH · #3188
CST-Bench: benchmark and annotations for counterfactual spatial reasoning, plus CT-OPSD method
The authors introduce CST-Bench, a dataset of 7,082 examples for counterfactual spatial reasoning covering 6 intervention types and 12 transformation operators with dense causal annotations (intervention sources, affected/invariant relations, expected effects, metric changes). They benchmark general-purpose multimodal LLMs on CST-Bench, compare post-training strategies, and propose CT-OPSD (Causal Transition On-Policy Self-Distillation) to teach counterfactual state transitions via annotated teacher chains distilled into a student model, reporting CST-Bench as a challenging evaluation and reusable supervision resource.
KEY POINTS
- The authors introduce CST-Bench, a dataset of 7,082 examples for counterfactual spatial reasoning covering 6 intervention types and 12 transformation operators with dense causal annotations (intervention sources, affected/invariant relations, expected effects, metric changes).
- They benchmark general-purpose multimodal LLMs on CST-Bench, compare post-training strategies, and propose CT-OPSD (Causal Transition On-Policy Self-Distillation) to teach counterfactual state transitions via annotated teacher chains distilled into a student model, reporting CST-Bench as a challenging evaluation and reusable supervision resource.
- CST-Bench supplies dense causal annotations and focused tasks that reveal failures of standard spatial reasoning methods under interventions and provides a benchmarking and supervision resource to improve MLLM causal spatial reasoning.
WHY IT MATTERS
CST-Bench supplies dense causal annotations and focused tasks that reveal failures of standard spatial reasoning methods under interventions and provides a benchmarking and supervision resource to improve MLLM causal spatial reasoning.