NVIDIA Releases Nemotron-Labs-3-Puzzle-75B-A9B: A Compressed Hybrid MoE LLM Delivering 2.03x Server Throughput at Matched User Throughput

A compressed hybrid MoE language model has been released that reduces the total parameters of its parent model from 120.7B to 75.3B while maintaining the same architectural layout across its 88 blocks. This matters because the smaller model delivers significantly higher server throughput on the same hardware, achieving up to 2.03x throughput gains in decode-heavy scenarios while using substantially less memory, allowing a single processor to handle 8 concurrent 1-million-token requests instead of just 1. The model was created using an iterative architecture search process that selectively pruned different components like expert intermediate sizes and routed expert counts at different depths, then recovered quality through knowledge distillation. Trade-offs exist, with the compressed version showing notable performance drops on some benchmarks like Arena-Hard-V2 and SWE-Bench, though minimal impact on others.