LEEPS boosts prompt sampling reinforcement learning accuracy by up to 3.7%

6 hours ago 29

Researchers have introduced a new method called LEEPS that could make one of the most stubborn inefficiencies in AI model training a lot less costly. The approach targets a specific bottleneck in prompt sampling reinforcement learning pipelines used to sharpen the reasoning skills of large language models, and early benchmark results suggest it delivers measurable gains without slowing training down.

Key takeaways

LEEPS is a Latent-Guided Explore-Exploit Prompt Sampler built to balance reuse of known informative prompts with continued exploration of uncertain ones during reinforcement learning with verifiable rewards.
Across six mathematical reasoning benchmarks, LEEPS posted the highest average scores at both the Qwen2.5-Math-1.5B and Qwen2.5-Math-7B scales.
Relative gains over the strongest baseline reached 2.6% for the 1.5B model and 3.7% for the 7B model.
LEEPS also topped three out-of-distribution general-reasoning benchmarks at both model scales.
The method adds only about 2 seconds of online sampling overhead per training step, a small cost for the performance it delivers.

Introduction to LEEPS and its role in reinforcement learning

LEEPS is a prompt sampling method designed to balance exploration and exploitation during reinforcement learning training for large language models. That balancing act sits at the center of a problem researchers have flagged with reinforcement learning with verifiable rewards, or RLVR, a training technique that has become one of the main levers for improving how well language models reason through math and logic problems.

The issue is straightforward once you see it: when a model generates multiple rollouts for the same prompt and every single one returns an identical reward, the training process learns nothing from that round. It burns through generation budget for zero signal. Pre-rollout prompt selection, screening prompts before generating rollouts, is one way to cut that waste. But according to the researchers behind LEEPS, existing pre-rollout methods run into a tricky trade-off: leaning too hard on prompts that have historically been informative narrows the range of training coverage, while casting too wide a net for exploration lowers the share of prompts that actually produce useful learning signals.

Balancing exploration and exploitation in prompt sampling

LEEPS is built specifically to resolve that tension rather than pick a side. Instead of treating exploitation and exploration as competing strategies, the method adaptively shifts between them based on what the training process is actually seeing in real time. That adaptive quality is what separates it from static sampling approaches, which tend to lock in a fixed ratio of familiar versus uncertain prompts regardless of how training is unfolding.

How LEEPS operates: prompt partitioning and prioritization

In practice, LEEPS splits candidate prompts into two groups: an exploit portfolio built from prompts that have proven informative in the past, and an explore portfolio made up of prompts whose value remains uncertain. It then adaptively allocates rollout budget between the two based on their recent non-trivial ratios, meaning it tracks how often each portfolio has recently produced rollouts with meaningful reward variance and adjusts the budget split accordingly.

Latent space neighbors and historical rollout outcomes

The exploration side is where LEEPS gets more sophisticated. Rather than exploring blindly, the method uses representation-space neighbors, essentially prompts that sit close together in latent space, along with historical rollout outcomes, to prioritize uncertain prompts that are statistically more likely to produce non-zero reward variance. In effect, this makes exploration targeted instead of random, without requiring any additional rollouts to figure out where to look. This latent-guided exploration is arguably the core technical contribution: it lets the sampler make an educated guess about which uncertain prompts are worth the generation budget, rather than spending that budget uniformly across everything unknown.

This is also where the training efficiency argument comes together. Prompt groups that return identical rollout rewards consume generation budget without offering effective learning signals, and by screening prompts before rollout generation, LEEPS is designed to reduce that specific waste. In a training regime where compute is the scarce resource, cutting down on rollouts that teach the model nothing has direct downstream value.

Performance gains and efficiency improvements with LEEPS

The headline result is that LEEPS outperformed the strongest baseline it was tested against, across two model sizes and nine total benchmarks. Across six mathematical reasoning benchmarks, LEEPS achieved the highest average score at both model scales tested: Qwen2.5-Math-1.5B and Qwen2.5-Math-7B. That translated into relative gains of 2.6% for the 1.5B model and 3.7% for the 7B model over the strongest baseline, according to the research. The larger model saw the bigger relative jump, which hints that the benefits of smarter prompt targeting may scale with model capacity, though the paper only reports results at these two sizes.

Benchmark results across mathematical and OOD reasoning tasks

The gains were not confined to math-specific tasks. LEEPS also achieved the highest average score across three evaluated out-of-distribution, or OOD, general-reasoning benchmarks at both model scales. That out-of-distribution performance matters because it suggests the method’s advantage is not narrowly tied to the mathematical reasoning benchmarks it was primarily tested on. A prompt sampling method that only helps on the exact task it was tuned for would be a far less useful tool; one that generalizes to reasoning benchmarks outside that training distribution is a stronger signal that the underlying mechanism, better-targeted exploration, is doing real work rather than overfitting to a specific benchmark suite.

Training speed and computational overhead

None of this came at a heavy computational price. LEEPS adds only about 2 seconds of online sampling overhead per training step, a modest cost relative to the performance improvements reported. The method also generally improved faster during the training process itself, meaning models trained with LEEPS tended to reach stronger performance earlier in the training run, not just at the finish line.

Taken together, the results point to a fairly practical conclusion for teams running large-scale RLVR training: the cost of adding latent-guided prompt selection is small, while the payoff, both in final benchmark scores and in how quickly those scores are reached, is not. For a field where every training run competes for limited compute, a 2-second-per-step tax in exchange for faster convergence and higher final accuracy is the kind of trade researchers tend to take.

FAQ

What problem does the LEEPS method address in reinforcement learning?

LEEPS addresses wasteful generation budget usage caused by prompt groups with identical rollout rewards by pre-rollout prompt selection to improve learning efficiency.

How does LEEPS balance exploration and exploitation in prompt sampling?

LEEPS partitions prompts into exploit and explore portfolios, adaptively allocating rollout budgets based on recent results to balance reuse of known informative prompts and exploration of uncertain ones.

What are the benchmark performance improvements achieved by LEEPS?

LEEPS achieves the highest average scores on six mathematical reasoning benchmarks at two model scales, with relative gains of 2.6% and 3.7% over the strongest baselines, and leads on three out-of-distribution general-reasoning benchmarks.