Dust
AssessTechniques
A zeroth-order optimization method for training transformer language models without backpropagation.
Why it's here
Placed in Assess: 1 article(s) of evidence from 1 source(s), led by research-stage coverage, with 1 in the last 30 days. Confidence 24%. Low accumulated evidence, so it defaults conservatively pending more signal.
Evidence (1)
- 8Hacker News·10/5/2026researchDust proposes transformer pretraining without backpropagation
Dust is a zeroth-order optimization method that trains transformer language models by perturbing activations and using reward-weighted averages to estimate gradients, rather than relying on backpropagation. The authors report that it can be competitive with backprop at scale, is far more efficient than weight-space evolutionary strategies, and may even outperform backprop in some settings with enough compute.