Dust: Pretraining Transformers Without Backpropagation

TL;DR

Prime Big Deal Days · Oct 6–7Offer from Amazon

Get school and study supplies delivered free — and shop member deals

• Fast, free delivery on millions of items
• Access to Prime Big Deal Days deals on October 6–7
• Prime Video, Amazon Music and more included

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Q Labs Research reports that Dust, a zeroth-order training method, pretrained transformer language models without a backward pass. The authors say its activation-based approach can approach or exceed backpropagation in some tested settings, while requiring substantially larger populations; the report does not establish that it is cheaper or better for production-scale training.

Q Labs Research has reported that its new method, Dust, can pretrain transformer language models without a backpropagation pass, using perturbations to model activations to estimate how training changes affect loss. The report says Dust’s estimates approach backpropagation’s at larger population sizes and outperform it in some tested settings, but the gains involve substantially more computation and do not establish that the method is more efficient overall.

Dust is described as a zeroth-order optimization method: instead of calculating analytic gradients through the network, it perturbs activations, measures the resulting loss, and combines the perturbations according to their effect. The report says the method applies perturbations independently at each token, treating tokens as members of a virtual population that can be evaluated in parallel during a forward pass.

Q Labs says its experiments include transformer pretraining and comparisons with backpropagation. According to the report, Dust’s gradient estimates align more closely with backpropagation as population size grows, and remain well aligned across the scales tested, up to 1 billion tokens. The researchers also report that a 243-million-parameter model outperformed a model 120 times smaller at most population sizes. These are findings reported by the authors, not independently verified results in the supplied material.

The report contrasts Dust with evolution strategies that perturb model weights. It estimates that from 1 million tokens onward Dust is roughly 1,000 to 10,000 times more efficient than a transformer implementation of EGGROLL, a separate evolution-strategy method. That figure is based on the authors’ extrapolations; it is not a measured comparison of full production training runs. The report also acknowledges that Dust approximates backprop closely at large populations, which entails substantially more compute.

A Different Route to Transformer Training

Most modern neural-network training depends on backpropagation, which calculates how changes to model parameters relate to changes in the loss. A method that could train large models without that backward computation would widen the set of training approaches researchers can test, including approaches that do not rely on the same differentiability assumptions.

Dust’s immediate importance is as an experimental result, not a replacement for standard training. The report suggests that activation-level search can be made more practical than weight-space evolution strategies by evaluating a virtual population in a forward pass. But its own account makes the trade-off clear: closer agreement with backpropagation comes with larger populations and more compute. Whether this strategy offers useful benefits in cost, training time, or final model quality remains unproven by the information provided.

From Weight Search to Activations

Evolution strategies train models by testing changes and using their effects on an objective to guide subsequent updates. In weight-space versions, candidate perturbations must be represented and evaluated, making large populations expensive. Dust instead perturbs activations inside the network, which the authors say lets one forward pass evaluate many token-level perturbations in parallel.

Q Labs frames the work against the prevailing role of backpropagation in deep learning. Its report argues that analytic gradients are highly useful when compute is limited, while more search-heavy methods could become relevant if substantially more computation is available. That is the authors’ rationale for exploring Dust; the report does not show that this broader possibility has been realized in deployed language-model training.

“We present the first zeroth-order method that is competitive with backprop at pretraining transformer language models.”

— Q Labs Research, in the report’s summary

Limits of the Reported Evidence

The available source is Q Labs Research’s own report, and the supplied material does not include independent replication, peer review status, or a full account of the experimental setup and results. It is therefore unclear how Dust performs across different model architectures, datasets, hardware configurations, or training budgets, and whether the reported comparisons would hold in larger or longer runs.

The report’s efficiency comparison with EGGROLL is explicitly an extrapolation, not a direct end-to-end benchmark. The material also does not establish whether Dust can match backpropagation on final model quality at comparable compute, what practical costs arise from its population settings, or whether the method can be integrated into routine training workflows. The authors’ claim that Dust is the first method of its kind to be competitive also remains their characterization in the supplied source.

Replication and Larger-Scale Tests

The next useful evidence would be detailed, reproducible benchmarks comparing Dust with backpropagation under matched compute and training conditions, including final model quality, wall-clock time, and hardware use. Independent tests would help establish whether the results extend beyond the settings reported by Q Labs.

The report includes a code link, but the supplied material does not specify when or how outside researchers will reproduce the experiments, nor does it announce a follow-up milestone. For now, Dust is best understood as a research result that challenges the assumption that zeroth-order methods cannot compete in transformer pretraining, while leaving its practical cost and scalability open.

Key Questions

What is Dust?

Dust is a zeroth-order optimization method from Q Labs Research that perturbs a model’s activations and uses the effects on loss to estimate updates, rather than using a backpropagation pass.

Does Dust eliminate all computation used in training?

No. The report says Dust avoids the backward pass, but its estimates become closer to backpropagation at larger population sizes, which the authors describe as requiring substantially more compute.

What results does Q Labs report?

The authors report that Dust’s estimates align more closely with backpropagation as population size grows, that a 243-million-parameter model outperformed one 120 times smaller at most tested population sizes, and that Dust exceeded backpropagation in some settings. These are claims from the report, not independent confirmation.

Is Dust proven more efficient than backpropagation?

No. The report claims an estimated efficiency advantage over EGGROLL from 1 million tokens onward, based on extrapolations. It does not establish that Dust is more efficient than backpropagation in full-scale training.

What remains unknown?

Independent replication, performance across other training settings, and comparisons of final model quality and total compute remain unclear from the supplied material. The practical value of Dust will depend on those tests.

Source: hn