MLCommons launches Candela benchmarking service
MLCommons: Introducing Candela: AI risk benchmarking-as-a-service for developers and deployers
Anyone putting AI to work needs to know where it might give responses it shouldn’t, and how often. That’s hard to know, even for the people who built the model: behavior shifts with every configuration, and there’s been no common way to test for it. Today MLCommons is launching Candela, benchmarking-as-a-service for AI risk and reliability. As an industry organization of over 125 companies (including Microsoft, Google, Qualcomm, NVIDIA, OpenAI, Anthropic), academic institut
Get smart on it
MLCommons has launched Candela, a benchmarking service that tests AI systems for safety and reliability risks across various hazard categories. The service addresses a gap in AI deployment: while companies need to measure how often their AI systems give dangerous or inappropriate responses, there has been no common way to test for this across different configurations and deployments. Candela provides standardized tests through consensus-built AILuminate benchmarks, allowing developers and deployers to run practice tests frequently and publish official versioned scores graded on separate prompt sets that the tested system cannot have learned from. The scoring uses a custom machine learning classifier rather than a language model as a judge, which provides more stable and comparable results over time, particularly for detecting jailbreak techniques designed to circumvent AI guardrails.
Explainer: How to Actually Read an AI Benchmark (Without Being Fooled)
Relevant courses
Want to go deeper than the news? Explore live, cohort-based AI courses taught by practitioners.
Browse AI courses on MavenRelated news
Mistral Says Its AI Model ‘Le Chonk’ Has Leapfrogged China in the Open-Weight Race
The French AI lab claims its latest release matches or outperforms even the best models from DeepSeek and Z.ai in coding and cybersecurity.
OpenAI announces three big science discoveries and a new way to work with academics
Less than a month after its mass release of hundreds of manuscripts that it said contained solutions to long-standing mathematics questions, OpenAI is back with three more "major" discoveries in math, computer science, and physics. What's different this time is the choice to have external experts that OpenAI describes as "shepherds" guiding the results in computer science and physics, while for the math problem it says that they advised it to release the AI output for researche
Also reported by The Decoder
WorldOracle Tests Whether World Models Can Reliably Predict Real-World Executions
Key Takeaways WorldOracle is a benchmark for action-conditioned world models (ACWMs) that scores each model’s predicted rollout against the paired real-world execution, with both starting from the same initial observations and following the same action sequence. Seven representative models were evaluated; arm motion was predicted relatively well, but object positions and movements often deviated from reality, and performance dropped over complete operation sequences. The platform opened t
Frontier AI labs say their models refuted the Exponential Time Hypothesis. What does that mean?
TechCrunch spoke to the MIT computer scientist charged with explaining OpenAI's latest mind-bending discovery
Working with academic communities to build scientific understanding
OpenAI shares model results on the Kakeya conjecture, Exponential Time Hypothesis, and quantum chromodynamics—and how researchers can verify and build on them.
Nace AI Releases Drex for Search for AI Agents: 92.05% on BrowseComp-Plus With 12.6 Search Calls
Nace AI has released Drex for Search, which brings its Drex decision models to document search. Drex for Search is a retrieval layer that helps people and agents find evidence. Coding agents such as Claude Code, Codex, Cursor and OpenCode call it through CLI, SDKs, MCP or REST. In Nace’s evaluation, it scores 92.05% on BrowseComp-Plus with GPT-5, using 12.6 search calls per question. TL;DR Size: a hosted search system built to search over 1M+ files. It pairs with an answering model