60 stories tagged distributed-systems, newest first.
76 points by bootlegbilly | 28 comments
Blog post arguing that observability should be built on wide events, as in Meta's Scuba, rather than OpenTelemetry's split into metrics, logs and traces.
365 points by otoolep | 62 comments
Philip O'Toole's blog post recounting the history of distributed SQLite since he launched rqlite in 2014, covering dqlite, Litestream, LiteFS, libSQL/Turso, cr-sqlite and Cloudflare D1.
55 points by KraftyOne | 3 comments
Marc Brooker blog post showing LLM-built animations of system failures: Raft lost writes from lying fsync, and metastable failures in Postgres and etcd.
150 points by ibobev | 121 comments
Blog post arguing that UUIDv7 does not reliably provide monotonic ordering, accurate creation timestamps, or unguessability, examining RFC 9562 and implementations like Python's.
53 points by Klaster_1 | 20 comments
GitHub engineering blog post on migrating Git repositories to Azure using per-reference quorums and object-storage-backed virtual replicas to scale reads and writes.
3 points by gravity13 | 0 comments
An interactive visual explanation of the Byzantine Generals Problem, a classic distributed systems consensus problem involving faulty or malicious participants.
49 points by aphyr | 1 comments
Jepsen analysis by Kyle Kingsbury of Orkes Conductor 5.3.2–5.5.0-rc2, finding concurrent task execution, lost or abandoned workflows, and startWorkflow errors.
32 points by mpweiher | 0 comments
Blog post by Dave Pacheco on designing Oxide's control plane DNS propagation services so problems are easier to debug.
121 points by mpweiher | 30 comments
Homepage of Lingua Franca, a coordination language for building deterministic, concurrent, time-sensitive, embedded, and distributed systems.
18 points by porridgeraisin | 2 comments
Marc Brooker's blog post using Lorenz curves and Little's Law to quantify how latency percentiles contribute to mean latency and the cost and capacity of tail latency.
47 points by o8vm | 35 comments
GitHub repository for hibana, a Rust library providing choreography-derived runtime enforcement for distributed protocols.
26 points by handfuloflight | 2 comments
Engineering blog post explaining how celld, a self-hosted runtime for Cloudflare Workers and Durable Objects, uses deterministic simulation testing to explore execution orders and reproduce bugs.
45 points by eatonphil | 3 comments
Jane Street blog post on scaling and benchmarking a critical internal message bus using a new indexing strategy.
4 points by cs1996 | 3 comments
Mainbrella engineering blog post on designing its backend so retries are safe when a Linux machine is created but the reply to the client is lost.
7 points by hypervs | 0 comments
GitHub repository for Trigora, a durable execution engine for long-running AI agents and programs that checkpoints state instead of replaying history.
322 points by alentred | 107 comments
GitHub repository for git-bug, a distributed, offline-first bug tracker that stores issues inside a Git repository.
24 points by tonychang430 | 5 comments
A blog post from Instacloud arguing that running a million AI agents at scale is fundamentally a distributed systems problem.
28 points by badrequest | 7 comments
Blog post by a developer at a gig-economy delivery app about discovering Redis Cluster hash slots while building a courier-matching service that relies on distance data.
44 points by matt_d | 3 comments
Blog post on the Metadata blog exploring compositional theories of self-stabilization, using a rely-guarantee TLA+ model of a retry storm and metastable failure.
461 points by f311a | 109 comments
Cloudflare engineering blog post explaining how they saved about 100TB of RAM across their fleet using consistent hashing math and Rust.
329 points by grep_it | 68 comments
A curated reading list of classic, influential distributed systems research papers, such as Lamport's work on clocks and Byzantine generals, with DOI links.
329 points by KraftyOne | 137 comments
DBOS blog post by Peter Kraft arguing that Postgres alone is sufficient to build durable workflow execution, describing DBOS's architecture.
323 points by sanity | 215 comments
Homepage of the redesigned Freenet, a peer-to-peer platform for decentralized apps built on a global key-value store of WebAssembly contracts.
443 points by jfantl | 140 comments
A blog post by Jason Fantl exploring how to generate IDs that would remain unique across the entire universe, from UUIDs to cosmological scales.
446 points by eieio | 160 comments
Marc Brooker's blog post arguing that Nagle's algorithm is outdated and TCP_NODELAY should be enabled to avoid latency issues in distributed systems.
681 points by ibobev | 155 comments
Homepage of Garage, an open-source S3-compatible distributed object store designed for self-hosting outside datacenters.
412 points by aphyr | 151 comments
Jepsen analysis by Kyle Kingsbury of NATS JetStream 2.12.1, finding lost writes from file corruption, power failures, and lazy fsync behavior.
338 points by todsacerdoti | 147 comments
A Big Data Stream newsletter article explaining how AWS S3 achieves 1 PB/s throughput and 150M QPS using commodity hard drives.
318 points by yodon | 165 comments
Medium essay by Julien Simon arguing that the Model Context Protocol ignores decades of RPC and distributed systems best practices, creating risks for enterprises.
795 points by gasull | 269 comments
Ink & Switch essay from April 2019 proposing local-first software principles for collaboration and data ownership, surveying storage approaches and CRDTs.
345 points by tanelpoder | 127 comments
Communications of the ACM practice article by Marc Brooker and Ankush Desai on how AWS uses formal and semi-formal methods to ensure systems correctness.
581 points by aphyr | 138 comments
Jepsen analysis by Kyle Kingsbury showing Amazon RDS for PostgreSQL multi-AZ clusters violate Snapshot Isolation, exhibiting Long Fork anomalies across versions 13.15 to 17.4.
751 points by cyndunlop | 293 comments
Blog post explaining how Bluesky uses intentionally lossy timeline fanout to trade perfect consistency for lower latency and better scalability.
326 points by antics | 105 comments
Moment devlog post by Alex Clemmer arguing that CRDT and OT collaborative editing algorithms handle offline editing and merge conflicts poorly.
379 points by jakey_bakey | 209 comments
Discord engineering blog post describing how the company migrated its trillions of stored messages from Cassandra to ScyllaDB, including Rust data services.
331 points by doesntmeananyth | 53 comments
A 2013 blog post by Jeff Hodges listing practical lessons for new distributed systems engineers, beyond the CAP theorem and the fallacies of distributed computing.
350 points by riv991 | 79 comments
A guest post on Werner Vogels' All Things Distributed blog by Marc Olson recounting the evolution of AWS Elastic Block Store and lessons learned.
457 points by n3t | 116 comments
Homepage of Garage, an open-source distributed object storage service tailored for self-hosting, with download and getting-started links.
441 points by thunderbong | 101 comments
GitHub repository for Tau, an open-source distributed Platform as a Service positioned as a self-hosted alternative to Vercel, Netlify, and Cloudflare.
362 points by paulgb | 142 comments
Blog post by Phil Eaton on the 2024 resurgence of high-quality systems programming meetups, with history and guidance for running your own events.
538 points by WhyNotHugo | 227 comments
Nikita Prokopov (tonsky) blog post on building local-first apps with CRDTs synced via ordinary file-sync services like Dropbox.
371 points by aphyr | 84 comments
Jepsen analysis by Kyle Kingsbury of Datomic Pro 1.0.7075, evaluating its transaction safety and serializability properties.
744 points by todsacerdoti | 232 comments
Marc Brooker's blog post arguing that Nagle's algorithm causes latency issues in distributed systems and TCP_NODELAY should be enabled by default.
526 points by abelanger | 156 comments
GitHub repository for Hatchet, an open-source, Postgres-backed distributed, fault-tolerant task queue with SDKs for Python, TypeScript and Go.
441 points by gempir | 170 comments
Engineer's Codex newsletter post explaining how Apple uses Cassandra and FoundationDB to power CloudKit and store billions of iCloud databases.
395 points by todsacerdoti | 98 comments
Homepage of Willow, a protocol for peer-to-peer data stores with fine-grained permissions, privacy, destructive edits, and low bandwidth and memory use.
321 points by aphyr | 119 comments
Jepsen analysis by Peter Alvaro and Kyle Kingsbury showing MySQL 8.0.34 Repeatable Read violates internal consistency and AWS RDS MySQL clusters violate Serializability.
645 points by tontinton | 118 comments
Blog post summarizing core database engineering concepts drawn from Database Internals and Designing Data-Intensive Applications, written by the author of the dbeel database.
415 points by manicennui | 210 comments
Kyle Kingsbury (Aphyr) blog post explaining why the Jepsen distributed-systems testing tool is written in Clojure, citing concurrency support and JVM interop.
381 points by kiyanwang | 347 comments
Blog post by Roberto Vitillo examining the trade-offs and costs of splitting a monolithic backend into microservices.
843 points by jakelazaroff | 121 comments
Jake Lazaroff's blog post giving an interactive introduction to CRDTs, the first of a two-part series building toward a collaborative pixel art editor.
405 points by thunderbong | 277 comments
A Renegade Otter blog post by Andrei Taranchenko arguing that the software industry is relearning that microservice complexity kills.
348 points by serial_dev | 213 comments
Slack Engineering blog post describing the company's migration of its infrastructure to a cellular architecture to improve resilience against availability-zone failures.
856 points by tekacs | 317 comments
Red Planet Labs blog post by Nathan Marz describing how they built a Twitter-scale Mastodon implementation at 100x lower cost using their Rama platform.
328 points by richieartoul | 228 comments
WarpStream blog post by Richard Artoul introducing WarpStream, a Kafka-compatible streaming platform built directly on object storage.
631 points by carride | 222 comments
Documentation homepage for BlazingMQ, Bloomberg's open source high-performance distributed message queuing system.
700 points by werner | 135 comments
Werner Vogels' All Things Distributed blog post on how Amazon S3 is built and operated at massive scale, from hard drives to organizational practices.
564 points by cendyne | 226 comments
Satirical blog post by Xe Iaso about misusing an unconventional service as a message queue, explicitly not meant for production workloads.
1011 points by kiyanwang | 202 comments
A blog post by Sam Rose explaining how a load balancer distributes HTTP requests across servers, building up from basic approaches to modern load balancing algorithms.
358 points by alexzeitler | 89 comments
A 2022 blog article explaining logical clock designs and how they help order events and track causality in distributed systems.