Scaling Transformer to 1M tokens and beyond with RMT (arxiv.org) | HN Screenshots

YHacker Newsscreenshots

Screenshot of arxiv.org: Scaling Transformer to 1M tokens and beyond with RMT

First screen, captured 3 years ago.

Looks similar

Screenshot of arxiv.org: Scaling Transformers to 1B Tokens

Scaling Transformers to 1B Tokens (arxiv.org)

208 points by mottiden 3 years ago | 64 comments

Screenshot of arxiv.org: Scaling Data-Constrained Language Models

Scaling Data-Constrained Language Models (arxiv.org)

44 points by ijk 3 years ago | 5 comments

Screenshot of arxiv.org: TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters

TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters (arxiv.org)

170 points by og_kalu 2 years ago | 28 comments

Screenshot of arxiv.org: HMT: Hierarchical Memory Transformer for Long Context Language Processing

HMT: Hierarchical Memory Transformer for Long Context Language Processing (arxiv.org)

83 points by jasondavies 2 years ago | 5 comments

Screenshot of arxiv.org: LLM.int8(): 8-Bit Matrix Multiplication for Transformers at Scale (2022)

LLM.int8(): 8-Bit Matrix Multiplication for Transformers at Scale (2022)(arxiv.org)

125 points by tosh 3 years ago | 15 comments

Screenshot of arxiv.org: RWKV: Reinventing RNNs for the Transformer Era

RWKV: Reinventing RNNs for the Transformer Era (arxiv.org)

318 points by ianbutler 3 years ago | 164 comments

arXiv paper introducing RWKV, a model architecture combining Transformer-style parallel training with efficient RNN inference via linear attention.

rwkv rnn transformers linear-attention