Predictable Performance in Model Routing and Cascades

Recent research explores performance prediction and model routing in large language models by studying when and why an AI's errors are predictable. The authors demonstrate that any performance predictor surpassing a model's self-confidence can directly improve the model's answer loss through recalibration. Furthermore, a predictor's advantage is strictly bounded by the model's suboptimality combined with the additional parameter capacity of the predictor. Empirical experiments across multiple benchmarks reveal that less optimal models yield greater prediction and routing advantages, which naturally diminish after task-specific finetuning. Finally, these theoretical limits extend to two-model cascades, showing that stronger routing advantages emerge in suboptimal models before they absorb the benefits of further training.