RESEARCH · RESEARCH · #3655
Airbnb introduces CASTLE framework to synthesize queries and relevance labels for natural-language search
Airbnb researchers introduced CASTLE, a framework that uses de-identified booking-session listing pairs and large language models to jointly generate synthetic natural-language search queries and built-in relevance labels, plus a Virtual Judge for cases lacking booking history and a production variant called CASTLE-R. In evaluations CASTLE reduced KL divergence to real-user query distributions (1.01) versus 9.33 for the best baseline and showed 91–93% agreement with human reviewers on sampled relevance assignments.
KEY POINTS
- Airbnb researchers introduced CASTLE, a framework that uses de-identified booking-session listing pairs and large language models to jointly generate synthetic natural-language search queries and built-in relevance labels, plus a Virtual Judge for cases lacking booking history and a production variant called CASTLE-R.
- In evaluations CASTLE reduced KL divergence to real-user query distributions (1.01) versus 9.33 for the best baseline and showed 91–93% agreement with human reviewers on sampled relevance assignments.
- CASTLE offers a practical method to generate realistic synthetic queries and labels before real search traffic exists, addressing cold-start problems for natural-language search systems and improving alignment with real user behavior.
WHY IT MATTERS
CASTLE offers a practical method to generate realistic synthetic queries and labels before real search traffic exists, addressing cold-start problems for natural-language search systems and improving alignment with real user behavior.