Brain-aligned evaluation of multimodal LLMs for social and emotional understanding of cartoons
This arXiv paper compares diffusion-based and autoregressive multimodal LLMs to continuous human ratings of Theory of Mind and affective valence across three short cartoon movies, assessing temporal Pearson correlations, distributional differences, and the effect of preceding scene context. The authors also used model and human ratings in subject-specific regressions to predict regional fMRI responses, finding that MLLMs capture aspects of human temporal variation (stronger for valence than ToM), that context helps in some cases, and that model-to-brain alignment is model- and stimulus-dependent with limited overlap between model and human encodings.