Pruning LLMs Like a Physicist: Block Removal as an Ising Optimization Problem

Block removal is a method to make large language models faster by deleting entire transformer blocks, which saves memory and speeds up inference while stacking with other compression techniques. The challenge is that removing blocks involves interacting choices: whether to remove one block depends on which other blocks are also removed, making it a combinatorial problem rather than a simple ranking task. Researchers reformulated this as an Ising optimization problem from physics, where each block is represented as a binary variable and the goal becomes minimizing an energy function derived from the model's loss, with the key finding that low-energy states correspond to high-performing pruned models. This approach is practical because the energy can be computed cheaply from a small calibration dataset and reused for different compression targets, and the resulting optimization problem can be solved with classical and quantum-inspired solvers when brute-force enumeration becomes infeasible.