Accelerating Gemini Nano models on Pixel with frozen Multi-Token Prediction

Large language models on phones typically generate text one word at a time, which wastes processing power and drains battery due to memory bandwidth limitations. Google has developed a method that retrofits Multi-Token Prediction onto existing frozen models by attaching a lightweight component to predict multiple words at once, then verifying them in parallel. This approach uses a zero-copy architecture that reuses the main model's memory cache instead of duplicating it, reducing memory usage and eliminating redundant processing steps. The result achieves speedups of fifty percent or more on mobile devices for tasks like notification summaries and text proofreading, with lower energy consumption and no changes to the base model's safety or capabilities.