Heaps do lie: debugging a memory leak in vLLM.

A memory leak was discovered in vLLM during testing of a disaggregated serving setup, where system memory increased by 400 MB per minute until running out of memory after several hours. The investigation revealed the leak occurred outside the heap memory that standard profiling tools typically monitor, making it invisible to conventional debugging approaches. The leak appeared specifically when using Prefill/Decode disaggregated processing with KV Cache transfer through NIXL, a communication library that relies on UCX for optimized data exchange in distributed systems. Understanding this required moving beyond heap analysis to examine resident memory, which includes the stack, anonymous memory mappings, and other allocations beyond just the traditional heap.