Sorbet: A neuromorphic hardware-compatible transformer-based spiking model (arxiv.org)

Screenshot of arxiv.org: Sorbet: A neuromorphic hardware-compatible transformer-based spiking model

First screen, captured 2 years ago.

Looks similar

Screenshot of arxiv.org: Scalable MatMul-Free Language Modeling

Scalable MatMul-Free Language Modeling (arxiv.org)

177 points by lykahb 2 years ago | 27 comments

Screenshot of arxiv.org: Jamba: A Hybrid Transformer-Mamba Language Model

Jamba: A Hybrid Transformer-Mamba Language Model (arxiv.org)

58 points by eitanturok 2 years ago | 6 comments

Screenshot of arxiv.org: RWKV: Reinventing RNNs for the Transformer Era

RWKV: Reinventing RNNs for the Transformer Era (arxiv.org)

318 points by ianbutler 3 years ago | 164 comments

arXiv paper introducing RWKV, a model architecture combining Transformer-style parallel training with efficient RNN inference via linear attention.

rwkv rnn transformers linear-attention

Screenshot of arxiv.org: HMT: Hierarchical Memory Transformer for Long Context Language Processing

HMT: Hierarchical Memory Transformer for Long Context Language Processing (arxiv.org)

83 points by jasondavies 2 years ago | 5 comments

Screenshot of arxiv.org: Differential Transformer

Differential Transformer (arxiv.org)

525 points by weirdcat 2 years ago | 166 comments

arXiv paper introducing Diff Transformer, which computes attention as the difference of two softmax maps to cancel noise and improve long-context modeling and reduce hallucination.

differential-transformer attention-mechanism transformers large-language-models

Screenshot of arxiv.org: S-LoRA: Serving Concurrent LoRA Adapters

S-LoRA: Serving Concurrent LoRA Adapters (arxiv.org)

28 points by Palmik 3 years ago | 4 comments

Looks similar

Scalable MatMul-Free Language Modeling(arxiv.org)

Jamba: A Hybrid Transformer-Mamba Language Model(arxiv.org)

RWKV: Reinventing RNNs for the Transformer Era(arxiv.org)

HMT: Hierarchical Memory Transformer for Long Context Language Processing(arxiv.org)

Differential Transformer(arxiv.org)

S-LoRA: Serving Concurrent LoRA Adapters(arxiv.org)