Models & Releases

Auditing Preference Biases and Fine-Tuning Language Models with Direct Preference Optimization on Anthropic HH-RLHF Using TRL and LoRA

In this tutorial, we design an end-to-end preference-learning workflow using the Anthropic HH-RLHF dataset and Direct Preference Optimization (DPO). We begin by preparing a robust Colab environment, loading and parsing chosen–rejected response pairs, and auditing the dataset for structural and length-based preference biases. We then run lexical shortcut diagnostics to determine whether surface-level linguistic patterns can separate preferred from rejected responses, prepare conversational data

Get smart on it

This tutorial describes how to fine-tune language models using Direct Preference Optimization on a dataset of human feedback, specifically examining and correcting for biases in what responses are labeled as preferred versus rejected. The process involves loading conversation data, analyzing it for structural patterns that might unfairly favor certain responses, and then training a smaller language model while optionally using parameter-efficient adaptation techniques. The workflow includes evaluating how well the trained model learns to distinguish preferred responses and testing its performance across different data subsets to identify potential issues like length bias.

Read next
The biggest AI stories, once a day

One short email a day with the stories that matter. Free. Unsubscribe with one click.

Relevant courses

Want to go deeper than the news? Explore live, cohort-based AI courses taught by practitioners.

Browse AI courses on Maven
Related news
Models & ReleasesOpen story →

Introducing GPT-6.2 Astra and Sol

Introducing GPT‑6.2 Astra and Sol, with better professional outputs, coding, computer use, and efficiency at lower cost across ChatGPT Work, Codex, and the API.

Also reported by MarkTechPost and SiliconANGLE

Models & ReleasesOpen story →

Introducing Gemini 4 Neon

Google's latest model aims to compete with rival systems in reasoning and multimodal capabilities as the AI arms race intensifies.

Also reported by SiliconANGLE and MarkTechPost

Models & ReleasesOpen story →

Alibaba Releases Qwen4-Max: 2.4T MoE, 71B Active, 1M Context

Alibaba’s Qwen team has released Qwen4-Max, the first model of its Qwen4 generation. Qwen4-Max is a proprietary sparse MoE model with a 2.4T-parameter backbone and 71B parameters active per token. It accepts text, images and video, and supports a 1M-token context window. Alibaba also cut API prices and reports a 62% drop in average cost per task. TL;DR Size: 2.4T backbone parameters, 71B active per token, plus 328B N-gram Embedding parameters. 1M context, 393K max output tokens.

Also reported by SiliconANGLE

Models & ReleasesOpen story →

Rumble-2-Flash: an open 0.5B speech model for 40+ languages

Smaller open-source speech models could democratize voice AI for developers working with languages beyond English and major European tongues.

Also reported by SiliconANGLE

Models & ReleasesOpen story →

Genie 4 is now available

The latest version expands capabilities for developers building generative applications across different domains.

Models & ReleasesOpen story →

Microsoft rolls out new first-party models and updates to agentic AI continuous improvement

Microsoft Corp. today rolled out new generative artificial intelligence models and updates to Foundry, the company’s unified enterprise platform that helps developers and organizations ground, test and deploy AI applications and autonomous agents. Updates to Foundry IQ, part of Microsoft IQ, now provide an enterprise knowledge graph out of the box, delivering a full estate […] The post Microsoft rolls out new first-party models and updates to agentic AI continuous improvement appeared firs