RESEARCH · RESEARCH · #2080
NVIDIA and AI4Bharat release Nemotron-pretraining-Indic-v1: 2.8T-token Indic corpus and open GPU data pipeline
NVIDIA, IIT Madras (AI4Bharat) and collaborators are releasing Nemotron-pretraining-Indic-v1, a synthetic pretraining corpus of about 2.8 trillion tokens spanning 22 Indian languages, together with an open, GPU-accelerated data-generation pipeline (including prompts, LLM workflows, and quality checks). The corpus includes native-script and romanized variants, preserves technical content (math, code, tables), traces outputs back to curated source documents, and was partially used to add 10 billion Indic tokens to continued pretraining of Nemotron 3.5 Super VL; the data and pipeline are being shared so others can reproduce and extend the workflow.
KEY POINTS
- NVIDIA, IIT Madras (AI4Bharat) and collaborators are releasing Nemotron-pretraining-Indic-v1, a synthetic pretraining corpus of about 2.8 trillion tokens spanning 22 Indian languages, together with an open, GPU-accelerated data-generation pipeline (including prompts, LLM workflows, and quality checks).
- The corpus includes native-script and romanized variants, preserves technical content (math, code, tables), traces outputs back to curated source documents, and was partially used to add 10 billion Indic tokens to continued pretraining of Nemotron 3.5 Super VL; the data and pipeline are being shared so others can reproduce and extend the workflow.
- An open 2.8T-token Indic corpus plus a reproducible GPU pipeline addresses a major data bottleneck for Indian-language models and lowers the barrier for building and evaluating large-scale Indic LLMs.
WHY IT MATTERS
An open 2.8T-token Indic corpus plus a reproducible GPU pipeline addresses a major data bottleneck for Indian-language models and lowers the barrier for building and evaluating large-scale Indic LLMs.