Scale transformers, skip side quests
Scaling is still paying off and resources like GPUs are scarce, so companies are right to drop exploratory research. Several commenters argue Meta was right to move on from LeCun, whose predictions about LLMs they say proved wrong.
Killing exploration is short-sighted
Industry side projects gave us the transistor, the laser and the transformer itself. Big firms can afford a skunkworks team, and whoever finds a cheaper architecture could upend the market.
Scaling is already hitting limits
Data shortages, Chinchilla-style limits, power-law costs and political resistance to data centers point to diminishing returns. Recent gains have come from RL and agent harnesses rather than bigger models.