New exponent functions that make SiLU and SoftMax 2x faster, at full accuracy(github.com)

368 points by weinzierl 2 years ago | 72 comments

GitHub pull request to llama.cpp by jart that upstreams llamafile's vectorized expf() functions to compute SiLU and softmax faster and more accurately on CPU.

SDL3 new GPU API merged(github.com)

281 points by caspar 2 years ago | 104 comments

AI has a deep understanding of how this code works(github.com)

212 points by theresistor 11 months ago | 93 comments

Apple's MLX adding CUDA support(github.com)

522 points by nsagent 1 year ago | 181 comments

A GitHub pull request (closed, WIP) in Apple's ml-explore/mlx repository adding an experimental CUDA backend to the MLX framework.

ONNX Runtime merges WebGPU backend(github.com)

165 points by b_mc2 3 years ago | 30 comments

Java Panama Vector API Integrated with Apache Lucene(github.com)

67 points by kurhan 3 years ago | 12 comments

Llama: Add grammar-based sampling(github.com)

379 points by davepeck 3 years ago | 95 comments

A GitHub pull request to llama.cpp adding an API for grammar-based sampling that constrains LLM output with a BNF-like context-free grammar.

Llama.cpp 30B runs with only 6GB of RAM now(github.com)

1182 points by msoad 3 years ago | 356 comments

GitHub pull request #613 in llama.cpp by jart, merged, which uses mmap to make loading model weights 10-100x faster and reduce RAM usage.

Llama2.c: Inference llama 2 in one file of pure C(github.com)

659 points by anjneymidha 3 years ago | 158 comments

GitHub repository by Andrej Karpathy for llama2.c, a minimal implementation for running Llama 2 model inference in a single file of pure C.

Llamafile lets you distribute and run LLMs with a single file(github.com)

972 points by tfinch 2 years ago | 253 comments

GitHub repository for Mozilla-Ocho's llamafile, a tool to distribute and run large language models as a single executable file.

Apple's MLX adding CUDA support(github.com)

522 points by nsagent 1 year ago | 181 comments

A GitHub pull request (closed, WIP) in Apple's ml-explore/mlx repository adding an experimental CUDA backend to the MLX framework.

Run Llama 13B with a 6GB graphics card(gist.github.com)

564 points by rain1 3 years ago | 233 comments

A GitHub Gist with instructions for running Meta's Llama 13B model on a 6GB GPU using llama.cpp's CUDA/cuBLAS layer offloading.