From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoning

This paper studies how post-training pipelines transform large language models into effective reasoners through compositional generalization. The authors propose a hierarchical latent selection model that separates reasoning into atomic skills, such as local operations, and routing mechanisms that dictate how information is composed. Their theory suggests that supervised fine-tuning (SFT) provides the necessary raw materials, while reinforcement learning (RL) identifies and decomposes these elements into reusable modules. Controlled experiments validate that RL enables models to solve novel tasks by recombining learned atoms in ways not seen during training. Ultimately, the study concludes that SFT should focus on broad module coverage while RL should target genuinely new compositions to maximize out-of-distribution performance.