Serialise decoding of large package files#487
Merged
Conversation
Rendering any documentation page decodes the entire docs JSON for that package version, which transiently needs tens of times the file's size in heap. A few generated packages (react-icons, elmish-html, deku, ...) have files of 10MB or more with enormous numbers of rarely-cached documentation pages, so a handful of concurrent crawler requests for those pages could exhaust the heap and restart the server (observed 15 times in the 24 hours after v0.9.10 fixed the search index build's memory use). Decodes of files over 5MB now run one at a time; smaller packages (nearly all of them) are unaffected. The file is only read once the lock is held, so queued requests do not pin file contents, and the queue is bounded: excess requests fail fast with a 503 rather than accumulating while clients time out. The hourly index regeneration is exempt from the bound so the index never silently omits a package. Package pages also no longer decode the same file twice when the requested version is the latest. Verified against the full production data set: a 30-way concurrent burst of requests for a 15MB package's version page kills the previous code (heap exhausted at -M3400m, 3.7GB RSS) and is served by this code as 16 200s and 14 fast 503s with RSS flat at 2.6GB. Also: - robots.txt: disallow SEO/bulk crawlers (Semrush, Ahrefs, DotBot, MJ12, Amazonbot, Bytespider, PetalBot), which generate most of the long-tail page traffic - Evict page-cache files not accessed in 90 days via a weekly cron job installed by the deploy, so the cache cannot fill the disk - Raise the heap limit to -M3400m now that the server has swap - Add deploy/SERVER.md documenting server-only and DO account state
This was referenced Jul 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
v0.9.10 fixed the search index build's memory use, which surfaced the next constraint: rendering any documentation page decodes the entire docs JSON for that package version (transiently ~tens of times the file size in heap). A few generated packages (react-icons 24MB, elmish-html 18MB, deku 13MB × hundreds of versions, ...) have enormous numbers of rarely-cached documentation pages. Crawlers walk them in parallel; concurrent giant decodes stack and exhaust the heap (
-Mcap), restarting the server — observed 15× in the 24h after the v0.9.10 deploy, withpurescript-dekupages alone accounting for 94 backend 502s in one day.Fix
Handler.Database.lookupPackage). Packages under the threshold — nearly all of them — are unaffected.stat), so queued requests don't each pin a copy of the file bytes.findPackageWithLatestdid, and held both copies across a synchronous GitHub readme fetch).Also
-M3G→-M3400mnow that the server has 4GB swap to absorb spikesdeploy/SERVER.mddocumenting server-only and DO-account stateVerification (full production data set, empty cache, no nginx shield)
react-icons/1.1.6's version page: previous code dies (heap exhausted at-M3400m, 3.7GB RSS); this code serves 16×200 + 14×503 with RSS flat at 2.6GB, search responsive throughout.--pedanticclean; test suite passes.