Making Knowledge Distillation Cheap Enough to Run at Scale

Knowledge distillation is a machine learning technique where a smaller model is trained to match the performance of a much larger model, which matters because deploying very large language models requires enormous computing resources and memory. The distillation process itself has been prohibitively expensive, typically requiring hundreds of GPUs and careful memory management strategies because it must keep both the large teacher model and smaller student model loaded simultaneously while computing probability distributions across the entire vocabulary. Researchers have developed two system changes to reduce this cost: caching the teacher model's output once so it never needs to be loaded during training, and using a memory-efficient loss calculation method that processes data in chunks rather than building massive grids in memory all at once. These changes make it possible to run knowledge distillation on a single GPU and allow large-scale experimentation that was previously impractical.