Indian language AI dataset reaches 2.8 trillion tokens
Hugging Face: Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline
The effort addresses a gap in AI training data for Indian languages, which have historically received far less computational investment than English or Chinese.
Get smart on it
A large dataset of training text has been created for Indian languages, containing approximately 2.8 trillion tokens across 22 languages listed in India's Constitution. The dataset addresses a documented gap in available training data for Indian languages compared to English, which has limited the development of AI models for these languages. The release includes both the data itself and an open pipeline showing how the data was generated through translation, quality assessment, and repair processes, allowing developers to create additional training data for their own needs. The corpus includes text in native scripts for all 22 languages and in romanized form for twelve of them, with content covering mathematics, code, and question-answer material translated from English sources.
Relevant courses
Want to go deeper than the news? Explore live, cohort-based AI courses taught by practitioners.
Browse AI courses on MavenRelated news
Gemini 4 Argon: Built to solve your hardest problems
The new model aims to handle complex reasoning tasks where current systems often struggle or fail completely.
Also reported by The Verge, Ars Technica, The Decoder and SiliconANGLE
Welcome Muse Spark Open: open-weight frontier multimodal mixture-of-experts model from Meta
Open-weight models let researchers and companies build AI systems without relying on proprietary providers, potentially accelerating innovation across industries.
Also reported by The Verge, The Decoder, TechCrunch and SiliconANGLE
Open From Data Center to Desktop: NVIDIA Expands Nemotron Family With Highest Efficiency Super VL and Bolt Models
NVIDIA today released Nemotron 3.5 Super VL, an open 120-billion-parameter model that delivers the fastest task completion and highest accuracy for specialized always-on agents. Joining it are two new edge models: Nemotron 3.5 Bolt 74B-A9B 24GB, optimized to balance intelligence and speed per GB for local agentic workloads, and Nemotron 3.5 Bolt E4B 4GB, optimized for intelligence per GB for fast conversational agents. Super VL and Bolt 74B-A9B are available now, with Bolt E4B coming Oct. 23. T
Also reported by The Decoder, SiliconANGLE and MarkTechPost
OpenAI will reportedly ship at least one model this year but its flagship still needs alignment work
OpenAI is racing to finish its next flagship model, which uses the same looped transformer technique as Astra to boost math and coding performance. The tradeoff is that looping makes safety monitoring harder. Microsoft accidentally revealed details about how the technique works on a public page. Whether or not the flagship is ready in time, employees say one or two new models will likely ship by year's end. The article OpenAI will reportedly ship at least one model this year but its fla
Welcome Beam: a 501B open model you can own, customize, and scale
Open-source alternatives to proprietary AI systems could shift control over large language models from commercial labs to individual organizations and researchers.
Welcome FLUX.3 Image Klein in Diffusers
The new model joins a competitive landscape of open-source image generators, offering developers another option for customizing visual AI tools.
Also reported by The Decoder