
Auditing Preference Biases and Fine-Tuning Language Models with Direct Preference Optimization on Anthropic HH-RLHF Using TRL and LoRA
In this tutorial, we design an end-to-end preference-learning workflow using the Anthropic HH-RLHF dataset and Direct Preference Optimization (DPO). We begin by preparing a robust Colab environment, loading and parsing chosen–rejected response pairs, and auditing the dataset for structural and length-based preference biases. We then run lexical shortcut diagnostics to determine whether surface-level linguistic patterns can separate preferred from rejected responses, prepare conversational data
This tutorial describes how to fine-tune language models using Direct Preference Optimization on a dataset of human feedback, specifically examining and correcting for biases in what responses are labeled as preferred versus rejected. The process involves loading conversation data, analyzing it for structural patterns that might unfairly favor certain responses, and then training a smaller language model while optionally using parameter-efficient adaptation techniques. The workflow includes evaluating how well the trained model learns to distinguish preferred responses and testing its performance across different data subsets to identify potential issues like length bias.
- Introducing GPT-6.2 Astra and Sol · OpenAI
- Introducing Gemini 4 Neon · Google DeepMind
- Alibaba Releases Qwen4-Max: 2.4T MoE, 71B Active, 1M Context · MarkTechPost
- All of today’s AI news, biggest first →
One short email a day with the stories that matter. Free. Unsubscribe with one click.
Want to go deeper than the news? Explore live, cohort-based AI courses taught by practitioners.
Browse AI courses on Maven
Introducing GPT-6.2 Astra and Sol
Introducing GPT‑6.2 Astra and Sol, with better professional outputs, coding, computer use, and efficiency at lower cost across ChatGPT Work, Codex, and the API.
Also reported by MarkTechPost and SiliconANGLE

Introducing Gemini 4 Neon
Google's latest model aims to compete with rival systems in reasoning and multimodal capabilities as the AI arms race intensifies.
Also reported by SiliconANGLE and MarkTechPost

Alibaba Releases Qwen4-Max: 2.4T MoE, 71B Active, 1M Context
Alibaba’s Qwen team has released Qwen4-Max, the first model of its Qwen4 generation. Qwen4-Max is a proprietary sparse MoE model with a 2.4T-parameter backbone and 71B parameters active per token. It accepts text, images and video, and supports a 1M-token context window. Alibaba also cut API prices and reports a 62% drop in average cost per task. TL;DR Size: 2.4T backbone parameters, 71B active per token, plus 328B N-gram Embedding parameters. 1M context, 393K max output tokens.
Also reported by SiliconANGLE
Rumble-2-Flash: an open 0.5B speech model for 40+ languages
Smaller open-source speech models could democratize voice AI for developers working with languages beyond English and major European tongues.
Also reported by SiliconANGLE
Genie 4 is now available
The latest version expands capabilities for developers building generative applications across different domains.
Microsoft rolls out new first-party models and updates to agentic AI continuous improvement
Microsoft Corp. today rolled out new generative artificial intelligence models and updates to Foundry, the company’s unified enterprise platform that helps developers and organizations ground, test and deploy AI applications and autonomous agents. Updates to Foundry IQ, part of Microsoft IQ, now provide an enterprise knowledge graph out of the box, delivering a full estate […] The post Microsoft rolls out new first-party models and updates to agentic AI continuous improvement appeared firs