RESEARCH · RESEARCH · #2173
Brain-aligned evaluation of multimodal LLMs for social and emotional understanding of cartoons
This arXiv paper compares diffusion-based and autoregressive multimodal LLMs to continuous human ratings of Theory of Mind and affective valence across three short cartoon movies, assessing temporal Pearson correlations, distributional differences, and the effect of preceding scene context. The authors also used model and human ratings in subject-specific regressions to predict regional fMRI responses, finding that MLLMs capture aspects of human temporal variation (stronger for valence than ToM), that context helps in some cases, and that model-to-brain alignment is model- and stimulus-dependent with limited overlap between model and human encodings.
KEY POINTS
- This arXiv paper compares diffusion-based and autoregressive multimodal LLMs to continuous human ratings of Theory of Mind and affective valence across three short cartoon movies, assessing temporal Pearson correlations, distributional differences, and the effect of preceding scene context.
- The authors also used model and human ratings in subject-specific regressions to predict regional fMRI responses, finding that MLLMs capture aspects of human temporal variation (stronger for valence than ToM), that context helps in some cases, and that model-to-brain alignment is model- and stimulus-dependent with limited overlap between model and human encodings.
- Shows MLLMs can be used as scalable, brain-predictive annotators of social/emotional content but also highlights systematic rating differences and model- and stimulus-specific limits to substituting human judgments.
WHY IT MATTERS
Shows MLLMs can be used as scalable, brain-predictive annotators of social/emotional content but also highlights systematic rating differences and model- and stimulus-specific limits to substituting human judgments.