Llama.cpp on a jailbroken PS5: run any GGUF model natively on the console(github.com)
356 points by Anon84 | 83 comments
GitHub repository for PS5LM, a port of llama.cpp that runs GGUF language models natively on a jailbroken PlayStation 5.
#llama-cpp on Hacker News
29 stories tagged llama-cpp, newest first.
Janas: Run LLMs larger than RAM by streaming experts from SSD(github.com)
207 points by khalidkhair | 77 comments
GitHub repository for Janas, a pure C inference engine that runs LLMs larger than RAM by streaming MoE experts from SSD, on CPU or GPU.
Launch HN: Magnitude (YC S25) – Self-optimizing inference engine for agents(github.com)
187 points by anerli | 88 comments
GitHub repository for Magnitude, an open source Rust inference engine for local AI agents that autotunes GPU kernels to your hardware, claiming up to 2x faster than llama.cpp.
Faster prompt lookup drafting in llama.cpp(jadidbourbaki.github.io)
72 points by pptadversary | 11 comments
Blog post by Hayder Tirmazi describing optimizations that make prompt lookup drafting in llama.cpp up to 42x faster (140x with Daniel Lemire's follow-up PR) with less memory.
How to setup a local coding agent on macOS(ikyle.me)
464 points by kkm | 114 comments
A blog tutorial on running Gemma 4 and Qwen3.6 locally on macOS with llama.cpp, MTP speculative decoding, and multimodal support as a coding agent.
Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks(github.com)
627 points by zambelli | 225 comments
GitHub repository for Forge, a Python framework adding guardrails to improve reliability of self-hosted LLM tool-calling and multi-step agentic workflows.
Stop Using Ollama(sleepingrobots.com)
471 points by Zetaphor | 122 comments
Opinion blog post arguing against using Ollama, recounting its history as a llama.cpp wrapper, attribution disputes, cloud pivot, and recommending alternatives.
$500 GPU outperforms Claude Sonnet on coding benchmarks(github.com)
447 points by yogthos | 247 comments
GitHub repository for ATLAS (Adaptive Test-time Learning and Autonomous Specialization), a system for running local LLMs on consumer GPUs for coding tasks.
My Journey to a reliable and enjoyable locally hosted voice assistant (2025)(community.home-assistant.io)
404 points by Vaslo | 128 comments
Home Assistant community forum post describing a user's setup of a fully local voice assistant using Home Assistant Assist and llama.cpp, replacing Google Home.
BitNet: Inference framework for 1-bit LLMs(github.com)
357 points by redm | 163 comments
GitHub repository for Microsoft's BitNet, the official inference framework for 1-bit large language models.
Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI(github.com)
807 points by lairv | 211 comments
GitHub Discussions announcement by Georgi Gerganov that ggml.ai, the founding team of llama.cpp, is joining Hugging Face to support local AI.
Vision Now Available in Llama.cpp(github.com)
475 points by redman25 | 104 comments
Documentation page in the llama.cpp GitHub repository describing multimodal (vision) input support via libmtmd and the tools that use it.
Show HN: Clippy – 90s UI for local LLMs(felixrieseberg.github.io)
1061 points by felixrieseberg | 257 comments
Download page for Clippy, a desktop app with a 1990s Windows-style interface for chatting with large language models running locally via llama.cpp.
Promising results from DeepSeek R1 for code(simonwillison.net)
926 points by k__ | 689 comments
Simon Willison blog post on a llama.cpp PR doubling WASM speed via SIMD, mostly written by DeepSeek R1, plus his own results using R1 for code.
Llama.vim – Local LLM-assisted text completion(github.com)
496 points by kgwgk | 97 comments
GitHub repository for llama.vim, a Vim plugin that provides local LLM-assisted code and text completion using llama.cpp.
Llama.cpp guide – Running LLMs locally on any hardware, from scratch(steelph0enix.github.io)
318 points by zarekr | 60 comments
A long blog guide on building and running llama.cpp to run large language models locally on various hardware, from scratch.
New exponent functions that make SiLU and SoftMax 2x faster, at full accuracy(github.com)
368 points by weinzierl | 72 comments
GitHub pull request to llama.cpp by jart that upstreams llamafile's vectorized expf() functions to compute SiLU and softmax faster and more accurately on CPU.
LLaMA now goes faster on CPUs(justine.lol)
1286 points by lawrencechen | 413 comments
Justine Tunney's blog post describing 84 new matrix multiplication kernels that make llamafile prompt evaluation 30-500% faster on CPUs than llama.cpp.
Show HN: I made an app to use local AI as daily driver(recurse.chat)
496 points by xyc | 210 comments
Landing page for RecurseChat, a macOS app for chatting with local LLMs offline, with full-text search and ChatGPT history import.
Show HN: Reor – An AI note-taking app that runs models locally(github.com)
341 points by samlhuillier | 83 comments
GitHub repository for Reor, an open-source desktop note-taking app that runs local LLMs and vector search for semantic linking and RAG Q&A over markdown notes.
Llamafile lets you distribute and run LLMs with a single file(github.com)
972 points by tfinch | 253 comments
GitHub repository for Mozilla-Ocho's llamafile, a tool to distribute and run large language models as a single executable file.
Talk-Llama(github.com)
428 points by plurby | 126 comments
GitHub directory for talk-llama, a whisper.cpp example that lets you talk to a LLaMA model locally using speech recognition and text-to-speech.
How Is LLaMa.cpp Possible?(finbarr.ca)
648 points by birriel | 210 comments
Blog post by Finbarr Timbers explaining the math of why llama.cpp can run LLaMA models locally on consumer hardware, focusing on memory bandwidth and compute limits.
Guide to running Llama 2 locally(replicate.com)
573 points by bfirsh | 153 comments
Replicate blog post explaining how to run Meta's Llama 2 model locally on M1/M2 Macs, Windows, Linux, and phones without an internet connection.
Llama: Add grammar-based sampling(github.com)
379 points by davepeck | 95 comments
A GitHub pull request to llama.cpp adding an API for grammar-based sampling that constrains LLM output with a BNF-like context-free grammar.
Llama.cpp: Full CUDA GPU Acceleration(github.com)
640 points by gzer0 | 265 comments
A draft GitHub pull request to llama.cpp adding full CUDA GPU acceleration for all ggml tensors, including storing the KV cache in VRAM.
GGML – AI at the Edge(ggml.ai)
817 points by georgehill | 206 comments
Homepage for ggml, a C tensor library for running machine learning models on commodity hardware, used by llama.cpp and whisper.cpp.
Run Llama 13B with a 6GB graphics card(gist.github.com)
564 points by rain1 | 233 comments
A GitHub Gist with instructions for running Meta's Llama 13B model on a 6GB GPU using llama.cpp's CUDA/cuBLAS layer offloading.
Llama.cpp 30B runs with only 6GB of RAM now(github.com)
1182 points by msoad | 356 comments
GitHub pull request #613 in llama.cpp by jart, merged, which uses mmap to make loading model weights 10-100x faster and reduce RAM usage.