Beyond self-attention: How a small language model predicts the next token (shyam.blog) | HN Screenshots

YHacker Newsscreenshots

Screenshot of shyam.blog: Beyond self-attention: How a small language model predicts the next token

First screen, captured 2 years ago.

Looks similar

Screenshot of ishamf.dev: Show HN: LLM Attention Visualization

Show HN: LLM Attention Visualization (ishamf.dev)

145 points by ifz 1 month ago | 23 comments

Screenshot of seangoedecke.com: Diffusion models explained simply

Diffusion models explained simply (seangoedecke.com)

123 points by onnnon 1 year ago | 25 comments

Screenshot of brettgfitzgerald.com: I built a large language model "from scratch"

I built a large language model "from scratch"(brettgfitzgerald.com)

66 points by controversy187 1 year ago | 4 comments

Screenshot of jalammar.github.io: The Illustrated Transformer (2018)

The Illustrated Transformer (2018)(jalammar.github.io)

138 points by debdut 2 years ago | 7 comments

Screenshot of ndingwall.github.io: Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)

Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)(ndingwall.github.io)

65 points by phewlink 1 year ago | 19 comments

Screenshot of martinalderson.com: Which programming languages are most token-efficient?

Which programming languages are most token-efficient?(martinalderson.com)

87 points by tehnub 9 months ago | 61 comments

Related stories

Screenshot of techcommunity.microsoft.com: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning (techcommunity.microsoft.com)

399 points by lappa 1 year ago | 136 comments

Microsoft AI Platform Blog post announcing Phi-4, a small language model specializing in complex reasoning.

phi-4 microsoft small-language-models llm

Screenshot of github.com: SmolGPT: A minimal PyTorch implementation for training a small LLM from scratch

SmolGPT: A minimal PyTorch implementation for training a small LLM from scratch (github.com)

415 points by amrrs 1 year ago | 52 comments

GitHub repository for smolGPT, a minimal PyTorch implementation for training a small GPT-style language model from scratch.

pytorch llm gpt transformers

Screenshot of anthropic.com: AI models may be capable of self-replicating without access to a datacenter

AI models may be capable of self-replicating without access to a datacenter (anthropic.com)

36 points by artninja1988 13 days ago | 26 comments

Anthropic Frontier Red Team post reporting experiments suggesting small AI models may soon self-replicate on commercial IoT devices without datacenter access.

ai-safety self-replication anthropic iot

Screenshot of rachel.fast.ai: Deep learning gets the glory, deep fact checking gets ignored

Deep learning gets the glory, deep fact checking gets ignored (rachel.fast.ai)

549 points by chmaynard 1 year ago | 134 comments

Rachel Thomas blog post on how a high-profile Nature paper using a Transformer to predict enzyme functions contained many errors that received far less attention than the original result.

deep-learning enzyme-function-prediction transformers scientific-errors

Screenshot of yitay.net: Training LLMs from ground zero as a startup

Training LLMs from ground zero as a startup (yitay.net)

360 points by swyx 2 years ago | 92 comments

Yi Tay's blog post on the experience of training large language models from scratch at a startup (Reka), covering compute, infrastructure, and lessons learned.

llm-training reka startups gpu-compute

Screenshot of github.com: Train Your Own LLM from Scratch

Train Your Own LLM from Scratch (github.com)

376 points by kristianpaul 5 months ago | 43 comments

GitHub repository by angelos-p providing code and docs for training your own large language model from scratch.

llm language-model-training python tutorial