FairStress reveals hidden biases in language model decisions
Hugging Face: FairStress: A deeper evaluation of fairness in LLM decision-making
Testing how language models handle bias in high-stakes decisions like lending or hiring reveals gaps that standard fairness checks miss.
Get smart on it
FairStress is a benchmark that measures fairness in large language models beyond their final decisions in high-stakes scenarios like hiring, lending, and healthcare. While standard fairness tests check whether models choose the same candidate when demographic identities are swapped, FairStress goes deeper by examining whether models change their decisions equally when presented with persuasive arguments that favor different demographic groups. The benchmark tests across more than 13 million scenarios involving demographic identity variations and adversarial challenges to reveal hidden biases, finding that models often show asymmetrical behavior by more readily abandoning correct choices when doing so favors minority-coded candidates. This matters because models are increasingly used in consequential decisions affecting people's opportunities and access to resources, making it critical to understand not just what they decide but how resistant those decisions are to pressure depending on which group benefits.
Relevant courses
Want to go deeper than the news? Explore live, cohort-based AI courses taught by practitioners.
Browse AI courses on MavenRelated news
Inception says Mercury 3 beats GPT-6 Luna and Claude Haiku 5.5 at up to 8x the speed
Diffusion language model startup Inception AI Inc. today released Mercury 3, its new flagship model. In the company’s own benchmarks, Mercury 3 beat OpenAI Group PBC’s GPT-6 Luna and Anthropic PBC’s Claude Haiku 5.5 on eight of 12 tests at four to eight times their speed. The release takes on the main doubt about Inception’s […] The post Inception says Mercury 3 beats GPT-6 Luna and Claude Haiku 5.5 at up to 8x the speed appeared first on SiliconANGLE.
Deepseek V4.1 Flash 1017 beats larger rivals in agentic tasks while once again massively cutting per-task costs
Deepseek has released V4.1-Flash-1017 as an open-weight model that posts better benchmarks, finishes tasks faster, and produces shorter outputs than its predecessor. The company is also opening up its internal search engine through new Search and Fetch APIs at $0.75 per 1,000 calls. API users in tools like Claude Code and Codex get free access for now. The article Deepseek V4.1 Flash 1017 beats larger rivals in agentic tasks while once again massively cutting per-task costs appeared fir
Also reported by SiliconANGLE
Four Gemini 4 Argon agents beat Minecraft without any human help
Four AI agents running on Google's Gemini 4 Argon have beaten Minecraft for the first time without human help, slaying the Ender Dragon in a fully autonomous run. With no predefined roles or strategies, they coordinated on their own, wrote 2,783 programs, exchanged 229 chat messages, and died 605 times. The session cost $1,620. The article Four Gemini 4 Argon agents beat Minecraft without any human help appeared first on The Decoder.
ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model
A system using adaptive memory mechanisms outperformed larger proprietary models on customer service benchmarks, suggesting scale isn't the only path to performance.
Anthropics' Claude cracks a 25-year-old NSA cryptography mystery in under 30 minutes
Anthropic's AI model Claude has solved a 25-year-old cryptography puzzle. The mysterious seeds behind NIST's standard curves for digital signatures were provided by the NSA without explanation, raising long-standing suspicion about built-in backdoors. Claude figured out in under 30 minutes that the seeds came from "lrand48," a simple C random number generator, making deliberate backdoors far less likely. The article Anthropics' Claude cracks a 25-year-old NSA cryptography mystery i
Exa 3.0 Search API Tops Perplexity, Brave and Parallel on 8 Benchmarks
Exa has released Exa 3.0, an upgrade to its web search API for AI agents. The Exa 3.0 search API ships a larger embedding model and a cheaper search type, Exa Lite. Dynamic Highlights is now generally available. Exa claims that it leads Perplexity, Brave and Parallel on 8 benchmarks. Those benchmarks were run by Exa, so you can treat them as self-reported. For developers, the practical change is cost: Lite costs $4 per 1,000 requests. TL;DR Size: Exa tracks over 2 trillion URLs and serve