RESEARCH · RESEARCH · #1977
Code-switching detection from speech using language-selective dimensions in Whisper-small
The authors identify language-selective hidden-state dimensions in pretrained speech encoders and aggregate their frame-level activations to detect code-switching (CS) directly from audio. Using a small tuning set, the Whisper-small encoder achieved a mean F1 of 0.798 across 14 language pairs on CS-FLEURS, outperforming a CNN (0.776) and a HuBERT linear baseline (0.751).
KEY POINTS
- The authors identify language-selective hidden-state dimensions in pretrained speech encoders and aggregate their frame-level activations to detect code-switching (CS) directly from audio.
- Using a small tuning set, the Whisper-small encoder achieved a mean F1 of 0.798 across 14 language pairs on CS-FLEURS, outperforming a CNN (0.776) and a HuBERT linear baseline (0.751).
- Finding CS directly from audio using pretrained encoders is a lightweight way to discover utterances for more representative multilingual pretraining without requiring accurate multilingual transcription.
WHY IT MATTERS
Finding CS directly from audio using pretrained encoders is a lightweight way to discover utterances for more representative multilingual pretraining without requiring accurate multilingual transcription.