DeepSeek open-sources DeepInfer inference framework
MarkTechPost: DeepSeek Open-Sources DeepInfer, the Inference Framework Behind DeepSeek-V4.1-Flash
DeepSeek has open-sourced DeepInfer, the inference framework that runs its production services. The DeepSeek DeepInfer inference framework is the engine behind DeepSeek-V4.1-Flash. It ships under the Apache 2.0 license. DeepSeek also released deepseek-recipe, its MIT-licensed token encoding and decoding layer. For developers, this is a rare look at a frontier lab’s real serving stack. TL;DR Size / scope: 1 supported model (DeepSeek-V4.1-Flash), 1M-token context window per the model
What happened
DeepSeek has open-sourced DeepInfer, the inference framework that powers its production AI services and runs the DeepSeek-V4.1-Flash model. The framework is released under the Apache 2.0 license, along with deepseek-recipe, an MIT-licensed token encoding and decoding layer. The release provides developers rare access to a frontier lab's production serving stack.
What each outlet adds
- MarkTechPost ↗: According to the company's self-reported benchmarks, the framework achieves above 25,000 output tokens per second per GPU, though it currently supports only one model and handles offline inference without certain production components like KV cache systems and load balancing.
- The Decoder ↗: The vLLM-based system cuts compute costs and energy use, and the release adds pricing pressure on Western providers like OpenAI and Anthropic.
- SiliconANGLE ↗: The framework includes technical optimizations like speculative decoding and prompt token management designed to improve throughput and reduce computational demands, and the announcement comes one day after the U.S. Department of Homeland Security ordered federal civilian agencies to remove DeepSeek products from their systems.
Why it matters
DeepSeek identifies reducing the cost of serving large models to many users as the core problem its inference framework solves. The release increases pricing pressure on Western AI providers and may have geopolitical implications for the US economy.
Get smart on it
DeepSeek has released DeepInfer, an inference framework under Apache 2.0 license that powers its production AI services. The framework enables high-performance processing of requests, achieving above 25,000 output tokens per second per GPU according to the company's self-reported benchmarks. DeepInfer currently supports only one model and handles offline inference, though it lacks certain production components like KV cache systems and load balancing across replicas that would be needed for a full service. The release provides rare public access to a frontier lab's production serving stack, originally built primarily on vLLM with ideas from other inference engines.
Relevant courses
Want to go deeper than the news? Explore live, cohort-based AI courses taught by practitioners.
Browse AI courses on MavenRelated news
Mistral AI launches Voxtral 3, a full voice agent stack that listens, thinks, and talks in real time
Mistral AI launched Voxtral 3, a new family of language models built for real-time voice agents. The system combines speech-to-text, text-to-speech, and a voice agent SDK, supports more than 50 languages, and lets agents tap into external tools during live conversations. It can run on your own infrastructure or through EU-hosted APIs to keep data under your control. The article Mistral AI launches Voxtral 3, a full voice agent stack that listens, thinks, and talks in real time appeared
Voxtral 3
Voxtral 3: Mistral's voice agent SDK for real-time speech in 50+ languages — accurate transcription, emotional TTS, deep tool use. EU-hosted or self-deployed.
Nvidia launches new open models to run agents in the enterprise and on PCs
Nvidia is leaning hard into the trend of enterprise companies wanting to run their own open models in their own data centers and also trying to make on-device, private AI agents a thing.
Meta and Apple team up to make Muse AI models run better on Macs
The new Muse Spark Open AI model can work on M5 Ultra Mac Studios. | Photo: Amelia Holowaty Krales / The Verge Meta announced the Muse Spark Open AI model today, claiming that it's the "most capable US open-weight model to date." But the company also says that it's the "the first model in its class" that can "run on your desktop using two Apple M5 Ultra Mac Studios with the latest MLX framework." Meta and Apple, typically rivals, worked together on the launch and are publicly su
Also reported by The Decoder, TechCrunch, SiliconANGLE and MarkTechPost
Elevenlabs releases two new speech-to-speech models designed for real-time AI conversations
Elevenlabs' new Eleven Realtime speech-to-speech models combine speed and accuracy. The v1 Max variant scores 98.8 percent on the Big Bench Audio benchmark with a 0.57-second response time. Elevenlabs says no model on the Artificial Analysis leaderboard is both faster and more accurate. The article Elevenlabs releases two new speech-to-speech models designed for real-time AI conversations appeared first on The Decoder.
Cloudflare Releases Clairic, a System One Extraction Model on Workers AI
Cloudflare has released a preview of Clairic, a new System One extraction model on Workers AI. The Cloudflare Clairic extraction model reads text or images and returns typed values, not generated text. It targets a job developers usually give to LLMs: pulling totals, dates and IDs into JSON. Clairic answers every requested field in 1 forward pass, with no reasoning and no output tokens. TL;DR Size: Parameter count not disclosed. 65,536-token context window, up to 4 images per request.
Also reported by The Decoder