MIT–IBM team on benchmarks and pseudonym-substitution for multimodal models
Researchers from the MIT-IBM Computing Research Lab, co-led by Jim Glass (MIT CSAIL), describe a multi-year effort to expose and address failures of state-of-the-art multimodal foundation models by releasing public benchmarks and developing data-efficient techniques. Their IPLoc work uses a 'pseudonym substitution' training trick to force models to rely on visual/audio cues rather than memorized labels and highlights gaps such as reported difficulties in object localization by models like GPT-40.