Bf16 might be freezing your weight updates(lacuna.tiptreesystems.com)
9 points by martincsweiss | 3 comments
An article showing that training bf16 weights directly with AdamW silently drops small updates, acting as a sparse mask, demonstrated by diffing the Quiet-STaR checkpoint.