Meet Gigatoken: A Rust BPE Tokenizer that Encodes Text at 24.53 GB/s, up to 989x Faster than HuggingFace Tokenizers
Tokenization is the one part of the language modeling stack that almost nobody profiles. Gigatoken, released by Marcel Rød (a PhD student from Stanford) under an MIT license, argues that this was a mistake. The library encodes text at gigabytes per second on a single machine, against baselines that are already multithreaded Rust. The GPT-2 tokenizer benchmarking yields remarkable results: evaluated on the 11.9 GB owt_train.txt corpus using a 144-core AMD EPYC 9565 dual-socket setup, Gi
Gigatoken is a tokenizer written in Rust that converts text into numerical tokens for language models, operating at significantly higher speeds than existing tools. Tokenization is a foundational step in language modeling that has traditionally received little performance optimization. The tool achieves its speed through two main techniques: hand-written pretokenization logic instead of regex engines, and caching of previously encoded words to avoid recomputation. Performance gains vary by setup, ranging from roughly 20x faster on consumer hardware to nearly 1,000x faster on high-core-count systems compared to existing tokenizers, though gains are substantially lower for certain vocabulary types.
Explainer: How to Actually Read an AI Benchmark (Without Being Fooled)
One short email a day with the stories that matter. Free. Unsubscribe with one click.
Want to go deeper than the news? Explore live, cohort-based AI courses taught by practitioners.
Browse AI courses on Maven
Anthropic releases new Mythos and Fable models amid RSI concerns
Anthropic says the new models "show improvements in areas like coding, reasoning, and vision," as well as in communicating more clearly with users.
Also reported by The Verge, SiliconANGLE and MarkTechPost
Google Launches Gemini Decisions API With Gemini Decision 1 at $0.10/M Tokens
Google has launched the Gemini Decisions API, powered by a new model, Gemini Decision 1 (gemini-decision-1). The Gemini Decisions API does not generate text. It scores up to 1,000 typed questions against one shared input and returns calibrated probabilities. For developers, this targets the small branching calls inside agents: routing, policy checks, tool selection and grading. Google prices it at $0.10 per 1M input tokens, with zero output tokens billed. TL;DR Size: Context is 262,144 t
Also reported by SiliconANGLE
Google releases Gemini 4 Carbon with agent ‘swarm’ feature
Google is rolling out its latest AI model, Gemini 4 Carbon, to the general public after releasing it to a select group of "cybersecurity defenders" two weeks ago. Google claims Carbon is its "strongest model" yet, and capable of competing with the very best systems from Anthropic and OpenAI. Google says Gemini 4 Carbon "pushes the frontier even further" than Argon, referring to the company's other frontier model that launched only last month. The company said Argon will likely
Gemini 4 Carbon is now available
The latest version aims to deliver improved performance while reducing the computational resources and energy required for AI operations.
Introducing AlphaProof Code: a breakthrough in verifying complex software at depth
AlphaProof Code agents wrote three million lines of proofs in Lean to verify Signal’s core encryption.
NVIDIA launches GR00T 2 and Isaac Lab 3.0 at CoRL
A range of developers, including RLWRLD, Skild AI, and Flexion, are already using NVIDIA’s latest technology. | Source: NVIDIA NVIDIA Corp. today introduced GR00T 2, its updated open frontier world action model, and Isaac Lab 3.0, the latest version of its open-source robot learning framework. The company made these announcements at the Conference on Robot Learning, or CoRL, in Austin, Texas. NVIDIA also gave a look at how its partners, like Ambi Robotics, Bosch Research, Dexmate, Dyna Rob