Dust: Pretraining Transformers Without Backpropagation(qlabs.sh)
269 points by E-Reverance | 79 comments
Research page from QLabs presenting Dust, a method for pretraining transformer models without backpropagation, with authors, code link, and architecture diagram.