RESEARCH · RESEARCH · #2640
IRON-Meme benchmark released for structured understanding of identities and roles in political memes
The authors introduce IRON-Meme, a human-annotated benchmark of 1,110 political memes built on a social-science grounded framework that labels social identities, viewpoints (ingroup/outgroup), and narrative roles (moral character, intelligence, status, power). They evaluate nine vision-language models, find frontier-scale and reasoning models score highest, note poorer accuracy on infrequent labels, and report that adding template background knowledge does not improve narrative role classification; the dataset and code are publicly released.
KEY POINTS
- The authors introduce IRON-Meme, a human-annotated benchmark of 1,110 political memes built on a social-science grounded framework that labels social identities, viewpoints (ingroup/outgroup), and narrative roles (moral character, intelligence, status, power).
- They evaluate nine vision-language models, find frontier-scale and reasoning models score highest, note poorer accuracy on infrequent labels, and report that adding template background knowledge does not improve narrative role classification; the dataset and code are publicly released.
- Provides a theory-grounded, human-annotated benchmark that reveals vision-language models' strengths and weaknesses on politically charged meme interpretation, useful for research on automated narrative and misinformation analysis.
WHY IT MATTERS
Provides a theory-grounded, human-annotated benchmark that reveals vision-language models' strengths and weaknesses on politically charged meme interpretation, useful for research on automated narrative and misinformation analysis.