Harvard paper reveals new scaling axis for generative models

For years, the playbook for making AI models better has been straightforward: more parameters, more data. A new paper from researchers at Harvard and the University of Illinois at Urbana-Champaign argues there’s a third lever that’s been hiding in plain sight: exploration.

The paper, titled “Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation,” introduces what the authors call Explorative Modeling, or XM. The core idea is that increasing the number of candidate generations a model explores during training produces outsized improvements in efficiency and output quality.

The numbers behind the claim

Expanding the exploration mode, referred to as K, yielded a 4.1x increase in FLOP efficiency, a 6.2x boost in sample efficiency, and a 47% improvement in parameter efficiency. In practical terms, that means models can achieve the same quality outputs while burning through far fewer computational resources.

Those gains also appear to compound as you scale. The paper reports that efficiency improvements grew from 7% to 36% as the amount of training data increased, and from 13% to 23% as model sizes expanded.

On the benchmark front, the XM approach achieved an FID score of 1.43 on ImageNet at 256×256 resolution without any guidance. FID, or Frechet Inception Distance, measures how closely generated images resemble real ones, with lower scores indicating better quality. The paper claims the model converged roughly 300 times faster than standard training recipes.

Beyond image generation

The XM framework isn’t limited to producing pretty pictures. The researchers demonstrated that their approach extends across multiple domains, including video generation and natural language processing. They tested Explorative Models on robotics and control tasks, where the models matched or exceeded diffusion model baselines while requiring 16 to 256 times fewer inference steps.

The paper also positions XM as enabling true end-to-end training and inference without relying on the factorization methods that current generative models typically depend on.

What this means for the AI landscape

The paper was authored by Alexi Gladstone from UIUC, along with Heng Ji and Yilun Du. Project resources including code are available publicly at explorative-modeling.github.io and on GitHub.

Disclosure: This article was edited by Editorial Team. For more information on how we create and review content, see our Editorial Policy.