Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation (arxiv.org)

Screenshot of arxiv.org: Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation

First screen, captured 8 months ago.

Looks similar

Screenshot of arxiv.org: Multi-Token Attention

Multi-Token Attention (arxiv.org)

131 points by fzliu 1 year ago | 35 comments

Screenshot of arxiv.org: Log-Linear Attention

Log-Linear Attention (arxiv.org)

19 points by sva_ 1 year ago | 3 comments

Screenshot of arxiv.org: TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters

TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters (arxiv.org)

170 points by og_kalu 2 years ago | 28 comments

Screenshot of arxiv.org: The FFT Strikes Back: An Efficient Alternative to Self-Attention

The FFT Strikes Back: An Efficient Alternative to Self-Attention (arxiv.org)

438 points by iNic 1 year ago | 161 comments

arXiv paper introducing FFTNet, an FFT-based adaptive spectral filtering framework offering O(n log n) token mixing as an alternative to self-attention.

fft self-attention transformers fftnet

Screenshot of arxiv.org: HyperAttention: Long-Context Attention in Near-Linear Time

HyperAttention: Long-Context Attention in Near-Linear Time (arxiv.org)

61 points by kelseyfrog 3 years ago | 10 comments

Screenshot of arxiv.org: Self-Adapting Language Models

Self-Adapting Language Models (arxiv.org)

210 points by archon1410 1 year ago | 61 comments

Looks similar

Multi-Token Attention(arxiv.org)

Log-Linear Attention(arxiv.org)

TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters(arxiv.org)

The FFT Strikes Back: An Efficient Alternative to Self-Attention(arxiv.org)

HyperAttention: Long-Context Attention in Near-Linear Time(arxiv.org)

Self-Adapting Language Models(arxiv.org)