RESEARCH · RESEARCH · #2995
Incongruence-aware evaluation finds audio–text mismatch harms speech emotion recognition in audio LLMs
The arXiv preprint introduces a continuous, threshold-free incongruence score (based on valence–arousal–dominance and a two-component Gaussian mixture, validated against human judgment) to quantify audio–text disagreement and evaluates three audio-capable LLMs on eNTERFACE'05, MELD and CMU-MOSEI. The authors report that greater incongruence increases error odds whenever audio is present across all corpora, that standard supervised classifiers behave differently, and that code, data tables and prompts are released.
KEY POINTS
- The arXiv preprint introduces a continuous, threshold-free incongruence score (based on valence–arousal–dominance and a two-component Gaussian mixture, validated against human judgment) to quantify audio–text disagreement and evaluates three audio-capable LLMs on eNTERFACE'05, MELD and CMU-MOSEI.
- The authors report that greater incongruence increases error odds whenever audio is present across all corpora, that standard supervised classifiers behave differently, and that code, data tables and prompts are released.
- Highlights a concrete failure mode of audio-capable LLMs—audio–text incongruence—that affects emotion recognition and provides an empirically validated metric and released assets for further study.
WHY IT MATTERS
Highlights a concrete failure mode of audio-capable LLMs—audio–text incongruence—that affects emotion recognition and provides an empirically validated metric and released assets for further study.