IBM-backed llm-d framework adds controls for in-house AI
SiliconANGLE: IBM-backed llm-d framework adds controls for running AI models in-house
IBM Corp. and several other contributors to the open-source llm-d project are releasing version 1.0 of their distributed artificial intelligence inference framework, adding operational controls intended to help enterprises run large models reliably on their own infrastructure. The release, announced ahead of the PyTorch Conference North America in San Jose today and tomorrow, focuses on […] The post IBM-backed llm-d framework adds controls for running AI models in-house appeared first on S
Get smart on it
A distributed artificial intelligence inference framework backed by IBM has released version 1.0, adding operational controls designed to help organizations run large language models on their own infrastructure rather than relying on external providers. The framework manages competing workloads, enables service updates without downtime, and improves hardware efficiency by optimizing how requests are routed and cached information is preserved across computing systems. This matters because running AI models in-house can significantly reduce inference costs and give organizations more control over their infrastructure choices, as demonstrated by tests showing the framework enabled concurrent operations at five to ten times lower cost than hosted services. The framework integrates techniques for request routing, context reuse, and workload optimization with operational capabilities that enterprises need to run AI services reliably at scale.
Relevant courses
Want to go deeper than the news? Explore live, cohort-based AI courses taught by practitioners.
Browse AI courses on MavenRelated news
Mistral AI launches Voxtral 3, a full voice agent stack that listens, thinks, and talks in real time
Mistral AI launched Voxtral 3, a new family of language models built for real-time voice agents. The system combines speech-to-text, text-to-speech, and a voice agent SDK, supports more than 50 languages, and lets agents tap into external tools during live conversations. It can run on your own infrastructure or through EU-hosted APIs to keep data under your control. The article Mistral AI launches Voxtral 3, a full voice agent stack that listens, thinks, and talks in real time appeared
Voxtral 3
Voxtral 3: Mistral's voice agent SDK for real-time speech in 50+ languages — accurate transcription, emotional TTS, deep tool use. EU-hosted or self-deployed.
DeepSeek Open-Sources DeepInfer, the Inference Framework Behind DeepSeek-V4.1-Flash
DeepSeek has open-sourced DeepInfer, the inference framework that runs its production services. The DeepSeek DeepInfer inference framework is the engine behind DeepSeek-V4.1-Flash. It ships under the Apache 2.0 license. DeepSeek also released deepseek-recipe, its MIT-licensed token encoding and decoding layer. For developers, this is a rare look at a frontier lab’s real serving stack. TL;DR Size / scope: 1 supported model (DeepSeek-V4.1-Flash), 1M-token context window per the model
Also reported by The Decoder and SiliconANGLE
Nvidia launches new open models to run agents in the enterprise and on PCs
Nvidia is leaning hard into the trend of enterprise companies wanting to run their own open models in their own data centers and also trying to make on-device, private AI agents a thing.
Meta and Apple team up to make Muse AI models run better on Macs
The new Muse Spark Open AI model can work on M5 Ultra Mac Studios. | Photo: Amelia Holowaty Krales / The Verge Meta announced the Muse Spark Open AI model today, claiming that it's the "most capable US open-weight model to date." But the company also says that it's the "the first model in its class" that can "run on your desktop using two Apple M5 Ultra Mac Studios with the latest MLX framework." Meta and Apple, typically rivals, worked together on the launch and are publicly su
Also reported by The Decoder, TechCrunch, SiliconANGLE and MarkTechPost
Elevenlabs releases two new speech-to-speech models designed for real-time AI conversations
Elevenlabs' new Eleven Realtime speech-to-speech models combine speed and accuracy. The v1 Max variant scores 98.8 percent on the Big Bench Audio benchmark with a 0.57-second response time. Elevenlabs says no model on the Artificial Analysis leaderboard is both faster and more accurate. The article Elevenlabs releases two new speech-to-speech models designed for real-time AI conversations appeared first on The Decoder.