Harden package pages against scraper-driven heap exhaustion#490
Merged
Conversation
v0.9.11 serialised decodes of package docs JSON files of 5MB or more, but concurrent decodes of files just below that cutoff can still exhaust the heap: scrapers crawling the ~1,900 module pages of next-purs-rsc (a 4.4MB file) repeatedly took the server down by stacking four or five decodes of it at once (RTS heap-overflow aborts, systemd status=251). Decodes of files of 1MB or more now share a 16MB in-flight byte budget instead of a single >=5MB lock: a decode is admitted once the total size of large files currently being decoded fits within the budget, or immediately when it is the only large decode running, so files bigger than the budget itself are still served. The bounded queue and its 503 fast-fail behaviour are unchanged, as is the search index regeneration's exemption from the bound.
The OOM incidents of early July 2026 were driven by distributed scraper fleets - on 2026-07-06, ~4,800 distinct IPs presenting spoofed Chrome/Edge user agents crawled the ~1,900 module pages of next-purs-rsc - which robots.txt cannot address. Deploys now install Anubis (https://github.com/TecharoHQ/anubis), a challenge proxy, pinned by version and checksum in remote.sh with its config generated on first deploy. nginx routes cache misses under /packages/ (the decode-triggering routes) through Anubis on the way to the backend. Clients with browser-like user agents must solve a JavaScript proof-of-work challenge once and then carry a cookie; non-browser clients (curl, package uploads, editor tooling, badge fetchers) and known-good search crawlers pass through untouched. Cache hits, /search (including the DigitalOcean uptime check, which must observe the real backend), POST /packages uploads, and every other route bypass Anubis entirely. The bot policy is a verbatim copy of the upstream v1.25.0 default, kept in the repo so deploys are reproducible and changes reviewable.
Measured with the exact decode path (strict read, aeson decode, deepseq) against real package files: decoding transiently holds ~11x the file size in live heap and ~24x in heap blocks, consistently across a 1.4MB (halogen 7.0.0), 4.4MB (next-purs-rsc 0.1.0) and 25MB (react-icons 1.0.2) file. A full 16MB budget therefore costs ~400MB of heap and the worst single admission (react-icons, alone) ~660MB, both well within the ~1.4GB of headroom under the 3.4GB RTS cap.
Adversarial review findings against the Anubis routing: The Accept-header map had no default, so the composite Accept strings real browsers send never matched an entry and try_files probed for a file literally named "index." - meaning browsers never hit the nginx page cache and every browser page view was re-rendered by the backend (verified against the live site: an exact "Accept: text/html" returns the cached file with an etag, a real Chrome Accept string returns a fresh Yesod render with a session cookie). That also made the /packages/ Anubis detour apply to all browser traffic rather than genuine cache misses, and made an Anubis outage a 502 for every browser package view. Cache lookups now default to html. HEAD requests took the non-GET escape hatch straight to the backend, where Yesod runs the full GET handler for them - a full-cost decode that skipped the challenge entirely, handing scrapers a free bypass. HEAD now takes the same cache/challenge path as GET in both locations. Also fail the deploy loudly if anubis is not running shortly after restart (a broken policy file makes it exit after startup while systemctl restart reports success), rather than leaving a redeploy serving 502s.
The decode budget admits a file bigger than the whole budget only once no other large decode is in flight, and STM retries impose no arrival order, so under a steady stream of smaller decodes such a file - and the search index regeneration queued behind it - can be overtaken indefinitely. Load-testing a crawler burst against the startup regeneration starved the index build for 71 minutes; a single react-icons page request against sustained sub-budget traffic reliably never completes. Admission now happens while holding an MVar, whose wakeup order is first-come-first-served: later arrivals queue behind a waiting oversize decode, the in-flight total drains within seconds, and the same probe is served in about two seconds.
Member
Author
|
Added one more commit after load-testing this branch against the full pursuit-backups dataset. Found a starvation problem where a file bigger than the whole budget (react-icons is 24MB) is only admitted when nothing else is in flight, and STM retries impose no arrival order — so under a steady stream of smaller large-file decodes, its waiters are overtaken indefinitely. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v0.9.10 (#486) and v0.9.11 (#487) handled the kernel OOM kills, but the server is still going down (albeit much less frequently): since the v0.9.11 deploy the process has died a few times with RTS heap-overflow aborts (systemd
status=251) and frozen in GC thrash, tripping the/searchuptime check each time.The trigger is bursts of scrapers (~5000 distinct IPs in the last day, with spoofed Chrome/Edge user agents, crawling the full site). Crawl bursts produce the crashes. So this change institutes yet another set of checks.
Two changes, one per commit:
Give large package decodes an aggregate memory budget. Decodes of files of 1MB or more share a 16MB in-flight byte budget. A decode is admitted once the total size of large files being decoded fits within the budget, or immediately when it is the only large decode running (so files bigger than the budget, e.g. react-icons at 25MB, are still served, by themselves). The search index regeneration is exempted.
Challenge browser-like scraper traffic to package pages with Anubis. Deploys now install Anubis, and nginx routes cache misses under
/packages/through it. The basic idea is that clients with browser-like user agents have to solve a JavaScript proof of work challenge once and then carry a cookie for a week; non-browser clients like curl, package uploads, etc. are not checked. Cache hits, search, and post requests to packages do not go through anubis.