Model Routing Is Simple. Until It Isn’t.

Model routing refers to systems that automatically direct different requests to different AI models based on factors like cost and task complexity. The article explains that while routing seems straightforward in theory, it actually involves complex tradeoffs across multiple dimensions including cost, performance, latency, and compliance requirements. The authors found that real-world costs depend not just on pricing but on factors like caching behavior and infrastructure conditions, that task difficulty is often unknowable until execution begins, and that latency depends on serving infrastructure rather than just model speed. Rather than treating routing as a simple classification problem, they developed an optimization approach that balances multiple factors simultaneously to find the best operating point for the entire system rather than just selecting the best model for a given task.