Transformer Layers as Painters (arxiv.org)

Screenshot of arxiv.org: Transformer Layers as Painters

First screen, captured 2 years ago.

Looks similar

Screenshot of arxiv.org: Transformers as Support Vector Machines

Transformers as Support Vector Machines (arxiv.org)

209 points by fofoz 3 years ago | 129 comments

Screenshot of arxiv.org: RWKV: Reinventing RNNs for the Transformer Era

RWKV: Reinventing RNNs for the Transformer Era (arxiv.org)

318 points by ianbutler 3 years ago | 164 comments

arXiv paper introducing RWKV, a model architecture combining Transformer-style parallel training with efficient RNN inference via linear attention.

rwkv rnn transformers linear-attention

Screenshot of arxiv.org: Jamba: A Hybrid Transformer-Mamba Language Model

Jamba: A Hybrid Transformer-Mamba Language Model (arxiv.org)

58 points by eitanturok 2 years ago | 6 comments

Screenshot of arxiv.org: Differential Transformer

Differential Transformer (arxiv.org)

525 points by weirdcat 2 years ago | 166 comments

arXiv paper introducing Diff Transformer, which computes attention as the difference of two softmax maps to cancel noise and improve long-context modeling and reduce hallucination.

differential-transformer attention-mechanism transformers large-language-models

Screenshot of arxiv.org: Unlimiformer: Long-Range Transformers with Unlimited Length Input

Unlimiformer: Long-Range Transformers with Unlimited Length Input (arxiv.org)

315 points by shishy 3 years ago | 97 comments

Screenshot of arxiv.org: Transformers Are Bayesian Networks

Transformers Are Bayesian Networks (arxiv.org)

27 points by Anon84 7 months ago | 16 comments

Looks similar

Transformers as Support Vector Machines(arxiv.org)

RWKV: Reinventing RNNs for the Transformer Era(arxiv.org)

Jamba: A Hybrid Transformer-Mamba Language Model(arxiv.org)

Differential Transformer(arxiv.org)

Unlimiformer: Long-Range Transformers with Unlimited Length Input(arxiv.org)

Transformers Are Bayesian Networks(arxiv.org)