Meta’s new paper exposes why reinforcement learning struggles with code optimization, and how to fix it

Teaching an AI to write code that works is one thing. Teaching it to write code that works fast is, apparently, a completely different beast.

A new paper from Meta AI’s FAIR team, published July 29, reveals that extending reinforcement learning from code correctness to code speed optimization is riddled with problems that standard approaches simply can’t handle. The culprits: noisy timing measurements, sparse rewards, and algorithms that fall apart when you ask them to care about performance, not just accuracy.

The problem with speed

When you measure whether code produces the right answer, you get a clean binary signal. But measuring how fast code runs is messy. Run the same code twice on the same machine and you’ll get slightly different execution times. Background processes, CPU scheduling, memory allocation, all introduce noise into timing measurements.

The Meta team found that Generalized Reinforcement Policy Optimization, or GRPO, a standard algorithm in the reinforcement learning toolkit, becomes unstable when you feed it these kinds of noisy speed measurements.

Reward sparsity compounds the issue. Most code optimizations yield marginal speed improvements, meaning the model rarely gets a strong positive signal telling it “yes, that change made things faster.”

DMC-Optim: a new benchmark for a new problem

To tackle these challenges, the researchers built DMC-Optim, a benchmark featuring a calibrated sandbox environment and a specialized training pipeline. The system combines rewards for both correctness and execution speed within an offline simulator, essentially creating a controlled environment where timing noise can be managed rather than ignored.

The results are hard to argue with. The pass rate for Qwen 2.5 7B jumped from 18.0% to 31.3%, a roughly 74% relative improvement. CWM 32B saw its pass rate climb from 30.7% to 50.4%, a 64% relative gain. On the LCB benchmark, CWM 32B trained with this approach beat median speed comparisons 83% of the time, compared to models trained with standard reinforcement learning from verifiable rewards.

Under degraded timing conditions, where measurement noise is deliberately amplified, the DMC-Optim benchmark demonstrated performance improvements between 100% and 200% over standard methods.

The paper was authored by Pierre Chambon, Kunhao Zheng, Juliette Decugis, Benoît Sagot, and Gabriel Synnaeve, all from Meta AI.

Disclosure: This article was edited by Editorial Team. For more information on how we create and review content, see our Editorial Policy.