Keenable AI Open-Sources NEEDLE: A Live Search Benchmark That Rebuilds Its Query Set Every Hour
How do you benchmark a web search API when the thing being tested can read the answer key? A search agent has a fetch tool. If the gold labels sit in a public dataset, the agent can download them mid-evaluation and skip retrieval entirely. A similar problem arises when the answers are already encoded in the model’s parametric memory: a correct response no longer demonstrates that web search worked. Keenable’s answer is NEEDLE, a live open-source benchmark that rebuilds its query set from
Make your prediction
Will NEEDLE appear as a dataset on Hugging Face by September 15, 2026?
Resolves by Sep 15, 2026
Your prediction
50% · 50/50 coin flip
NOYES
Get smart on it
NEEDLE is an open-source benchmark that tests web search APIs by generating fresh query sets hourly from live sources like RSS feeds and Google Trends, rather than using a static dataset that could be memorized or downloaded during evaluation. It matters because search agents with access to training data could cheat on fixed benchmarks, so NEEDLE solves this by constantly rebuilding its queries and measuring performance across five categories: news, finance, academic research, rare entities, and legal documents. The benchmark runs the same queries against fifteen different search APIs under identical conditions and compares each result against an "ultimate" ceiling created by pooling what all engines combined could retrieve, which reveals whether poor results stem from ranking failures or retrieval problems shared across the entire field.
Related news
Google says its AI weather model is getting better
Google is rolling out an updated AI weather model that's supposed to be more accurate, especially when it comes to predicting rain and snowfall. In the announcement today, the company says it's now able to make forecasts with "unprecedented resolution" using its new WeatherNext 3 AI model. It can produce a global picture that's five times sharper than Google's previous model by learning from real-time weather observations, according to the company. "One of the main developments
BenchMIRT: What are LLM benchmarks actually measuring?
Researchers question whether popular AI tests truly capture real-world language abilities or merely reflect memorization and training data overlap.
