Amazonbot is finally respecting robots.txt (xeiaso.net) | HN Screenshots

YHacker Newsscreenshots

Screenshot of xeiaso.net: Amazonbot is finally respecting robots.txt

First screen, captured 5 months ago.

Looks similar

Screenshot of xeiaso.net: Amazon's AI crawler is making my Git server unstable

Amazon's AI crawler is making my Git server unstable (xeiaso.net)

532 points by OptionOfT 1 year ago | 207 comments

Xe Iaso's blog note describing how Amazon's AI crawler is overloading their self-hosted git server and an nginx ingress snippet used to block it.

ai-crawlers amazonbot git-server web-scraping

Screenshot of bvisness.me: Coroutines make robot code easy

Coroutines make robot code easy (bvisness.me)

181 points by bvisness 3 years ago | 100 comments

Screenshot of blog.habets.se: Io_uring, kTLS and Rust for zero syscall HTTPS server

Io_uring, kTLS and Rust for zero syscall HTTPS server (blog.habets.se)

444 points by guntars 1 year ago | 138 comments

Blog post describing building an HTTPS web server in Rust using io_uring and kernel TLS to avoid system calls on the request path.

io-uring ktls rust linux

Screenshot of v8.dev: JavaScript's New Superpower: Explicit Resource Management

JavaScript's New Superpower: Explicit Resource Management (v8.dev)

307 points by olalonde 1 year ago | 202 comments

Screenshot of evgeniipendragon.com: I was wrong about robots.txt

I was wrong about robots.txt (evgeniipendragon.com)

160 points by EPendragon 1 year ago | 182 comments

Screenshot of calebjohn.xyz: Claude Code is your customer

Claude Code is your customer (calebjohn.xyz)

61 points by mfbx9da4 9 months ago | 43 comments