CWE-bench v2 tests AI at finding code vulnerabilities
Hugging Face: CWE-bench v2: More coverage, quality, and transparency
Better evaluation of software vulnerabilities helps developers identify and fix security flaws before they reach production.
Get smart on it
CWE-bench v2 is a public benchmark containing 100 tasks designed to test how well AI models can find and fix security vulnerabilities in real open-source code. Each task presents an agent with a repository checkout and instructions to locate and repair a security weakness while maintaining existing functionality, with scoring done through automated, reproducible checks. The benchmark covers 91 different types of security vulnerabilities known as CWEs, requiring models to recognize multiple attack surfaces rather than simply pattern-match familiar problems. Results show that while the leading models achieve around 68% success rates, even the best performers fail on approximately 19% of tasks on every attempt, indicating substantial room for improvement in automated security vulnerability detection and repair.
Relevant courses
Want to go deeper than the news? Explore live, cohort-based AI courses taught by practitioners.
Browse AI courses on MavenRelated news
LifeVal
Introducing LifeVal, a benchmark measuring how well AI helps people complete everyday tasks, from finding childcare to managing finances and government paperwork.
GDPx: Measuring Frontier Model Capabilities Across the Most Valuable Domains
GDPx is a benchmark that covers the most valuable domains for frontier intelligence. It aims to be the world’s best benchmark for measuring general agentic capabilities.We believe the first trillion dollars of AI revenue will be concentrated in sectors that can deploy frontier intelligence at high ROI, not broadly diffused throughout the whole economy. Building a general-purpose benchmark based on share of GDP or the SOC/O*NET occupational taxonomy is therefore a mistake. If your goal is t
AI Security Agents Compared: Claude, Codex, Mantis, Cloudflare
Anthropic, OpenAI, Cloudflare and Google now each ship an AI agent that hunts security bugs in source code. We checked the repos, docs and technical details behind each one. The key finding: all 4 tools converge on one design. One agent finds a bug, a separate agent tries to disprove it, and only survivors get a patch. The differences are in where they run, who can use them and how much proof they demand. Key Takeaways 4 labs, 1 pattern: find, independently disprove, then patch. Cl
Samaya AI Releases Sterling: A 3B-Active Finance Research Agent Model that Beats Claude Fable 5.1 on FrontierFinance at 11x Lower Cost
Samaya AI has released Samaya Sterling, a post-trained model for agentic financial research. Sterling is built on NVIDIA’s Nemotron 3.5 Lightning, a 30B mixture-of-experts model with 3B active parameters. On Samaya’s FrontierFinance benchmark, it scores 58.9%. Claude Fable 5.1 scores 55.9%. Each Sterling question costs $0.61, against $6.89 for Fable 5.1. For developers, the takeaway is a concrete recipe: co-train a small model with its harness, and it can win in a narrow domain.
StateBench: Testing how language models track state changes over long documents
Tracking state across extended text is crucial for real-world applications like document analysis and multi-turn reasoning.
LLM use is acknowledged almost as often as specialized AI tools in biology preprints
17% of biology preprints acknowledge LLM use, nearly matching specialized tools like AlphaFold. Epoch AI analyzes trends in AI acknowledgment in research.