WorldOracle tests world models' real-world predictions
The Robot Report: WorldOracle Tests Whether World Models Can Reliably Predict Real-World Executions
Key Takeaways WorldOracle is a benchmark for action-conditioned world models (ACWMs) that scores each model’s predicted rollout against the paired real-world execution, with both starting from the same initial observations and following the same action sequence. Seven representative models were evaluated; arm motion was predicted relatively well, but object positions and movements often deviated from reality, and performance dropped over complete operation sequences. The platform opened t
Get smart on it
WorldOracle is a benchmark platform that tests whether action-conditioned world models, which are AI systems trained to predict what will happen next in a video given a sequence of actions, can accurately forecast real-world robot manipulations. Researchers evaluated seven representative models and found two major limitations: while robot arm motions are predicted relatively well, the positions and movements of objects being manipulated often deviate from reality, and prediction accuracy declines significantly when evaluated over complete task sequences rather than short time windows. The benchmark matters because it reveals that world models can generate plausible-looking but incorrect predictions, such as predicting a successful grasp when the robot actually missed, potentially limiting their reliability for real-world robotic applications. The evaluation platform accepts both open-weight and closed-source models and uses pre-recorded real manipulation episodes from different robot embodiments performing tasks like fruit picking and water pouring.
Explainer: How to Actually Read an AI Benchmark (Without Being Fooled)
Relevant courses
Want to go deeper than the news? Explore live, cohort-based AI courses taught by practitioners.
Browse AI courses on MavenRelated news
OpenAI announces three big science discoveries and a new way to work with academics
Less than a month after its mass release of hundreds of manuscripts that it said contained solutions to long-standing mathematics questions, OpenAI is back with three more "major" discoveries in math, computer science, and physics. What's different this time is the choice to have external experts that OpenAI describes as "shepherds" guiding the results in computer science and physics, while for the math problem it says that they advised it to release the AI output for researche
Also reported by The Decoder
Frontier AI labs say their models refuted the Exponential Time Hypothesis. What does that mean?
TechCrunch spoke to the MIT computer scientist charged with explaining OpenAI's latest mind-bending discovery
Working with academic communities to build scientific understanding
OpenAI shares model results on the Kakeya conjecture, Exponential Time Hypothesis, and quantum chromodynamics—and how researchers can verify and build on them.
Nace AI Releases Drex for Search for AI Agents: 92.05% on BrowseComp-Plus With 12.6 Search Calls
Nace AI has released Drex for Search, which brings its Drex decision models to document search. Drex for Search is a retrieval layer that helps people and agents find evidence. Coding agents such as Claude Code, Codex, Cursor and OpenCode call it through CLI, SDKs, MCP or REST. In Nace’s evaluation, it scores 92.05% on BrowseComp-Plus with GPT-5, using 12.6 search calls per question. TL;DR Size: a hosted search system built to search over 1M+ files. It pairs with an answering model
Anthropic co-founder Jack Clark moves his prediction for self-improving AI forward to 2027
A new benchmark by P-Zero Research measures how well AI models design and evaluate experiments on their own. The results are improving so fast that Anthropic co-founder Jack Clark now thinks self-improving AI could arrive by 2027. The article Anthropic co-founder Jack Clark moves his prediction for self-improving AI forward to 2027 appeared first on The Decoder.
Fancy harnesses make coding agents more expensive but not better, study finds
A controlled study shows that switching between coding-agent harnesses like Claude Code, OpenCode, or mini-SWE-agent barely moves pass rates more than rerunning the same setup. Cost per task, however, can triple, driven mainly by the system prompt that gets resent with every step. The article Fancy harnesses make coding agents more expensive but not better, study finds appeared first on The Decoder.