3 stories tagged pretraining, newest first.
269 points by E-Reverance | 79 comments
Research page from QLabs presenting Dust, a method for pretraining transformer models without backpropagation, with authors, code link, and architecture diagram.
453 points by tamnd | 278 comments
Project page for a research paper from Texas A&M, UT Austin, and Purdue showing LLMs degrade ("brain rot") when trained on low-quality junk web data.
392 points by nee1r | 267 comments
Standard Intelligence blog post describing how they built a 30-petabyte hard drive storage cluster in a San Francisco colocation center for under $500k to store video pretraining data.