24 stories tagged quantization, newest first.
440 points by gumby | 208 comments
Substack essay from the Musical Form Institute comparing tempo variation in 1975 vs 2025 Billboard hits to argue modern pop production is grid-quantized to machine perfection.
18 points by anemll | 3 comments
A tweet from ANEMLL reporting a 27B-parameter Qwen model running on the iPhone 18 Neural Engine at 15–70 tokens per second using 1.5-bit quantization.
308 points by albinstman | 85 comments
Llmspeed is a web tool that simulates the token generation speed of local LLMs on chosen hardware, model, and quantization settings before buying hardware.
271 points by logicallee | 99 comments
MicroLLM Lab is a browser-based tool for running, chatting with, and benchmarking small quantized language models locally via WebGPU with no server.
551 points by JonSchneider | 176 comments
PrismML blog post announcing Bonsai 2 27B, a compressed language model claiming near-lossless quality at a 9x smaller footprint.
417 points by felineflock | 171 comments
A Level1Techs forum post presenting technical experiments on how quantization and implementation differences make locally run LLMs perform worse than reference versions.
882 points by vforno | 223 comments
GitHub repo for Colibrì, a dependency-free C engine that runs the 744B GLM-5.2 MoE model on a ~25GB-RAM machine by streaming experts from disk.
559 points by TechTechTech | 266 comments
Unsloth documentation guide explaining how to run Z.ai's GLM-5.2 open model locally using Unsloth Dynamic quantized GGUFs.
438 points by modinfo | 186 comments
PrismML blog post announcing Bonsai Image 4B, a 1-bit and ternary quantized image generation model designed to run on local devices.
411 points by caust1c | 173 comments
Blog post by antirez reflecting on the rapid popularity of DwarfStar 4 (DS4), his local AI inference project, and its future plans beyond DeepSeek v4 Flash.
495 points by ray__ | 134 comments
Google Research blog post introducing TurboQuant, a method for extreme compression to improve AI model efficiency.
379 points by mft_ | 119 comments
GitHub repository for Flash-MoE, a project for running a 397B-parameter mixture-of-experts model on a laptop using Metal inference code.
357 points by redm | 163 comments
GitHub repository for Microsoft's BitNet, the official inference framework for 1-bit large language models.
424 points by Curiositry | 138 comments
Unsloth documentation guide explaining how to run Alibaba's Qwen3.5 model family locally using GGUF quantized versions, with updates on improved quantization.
338 points by dataminer | 121 comments
ByteShape blog post describing how they quantized Qwen3-30B-A3B-Instruct with their Shapelearn bitlength method to run in real time on a Raspberry Pi.
457 points by quesomaster9000 | 101 comments
GitHub repository for Z80-μLM, a 2-bit quantized character-level language model that runs as a 40KB CP/M .COM binary on a Z80 processor.
544 points by simonw | 362 comments
Simon Willison blog post showing the GLM-4.5 Air model, 3-bit quantized for MLX, generating a working Space Invaders game in JavaScript on a 64GB laptop.
701 points by noch | 308 comments
Unsloth blog post describing dynamic 1.58-bit quantization of DeepSeek-R1 671B, shrinking it to 131GB so users can run it locally.
471 points by egnehots | 112 comments
Meta AI blog post announcing quantized versions of its Llama models that run faster and use less memory.
962 points by fgfm | 419 comments
arXiv paper introducing BitNet b1.58, a ternary-weight (-1, 0, 1) LLM matching full-precision Transformers while being far cheaper in latency, memory, and energy.
349 points by Gedxx | 84 comments
An Ikkaro blog article explaining TinyML, covering model compression techniques like pruning, quantization and distillation, tools, and uses on low-power devices.
648 points by birriel | 210 comments
Blog post by Finbarr Timbers explaining the math of why llama.cpp can run LLaMA models locally on consumer hardware, focusing on memory bandwidth and compute limits.
959 points by elbasti | 307 comments
Evan Miller's blog post arguing that the softmax in the transformer attention formula has an off-by-one flaw and proposing a modified softmax.
817 points by georgehill | 206 comments
Homepage for ggml, a C tensor library for running machine learning models on commodity hardware, used by llama.cpp and whisper.cpp.