/history and /since load an object's entire version chain into memory, unpaged #252

Re-scoped 2026-09-24. This issue originally covered queryHeadRequest, getAllVersions, and the version endpoints together. The /query half is now resolved — see What this no longer covers at the bottom. What remains is the version chain.

Summary

/history/{id} and /since/{id} load every version of an object's tree into memory and serialize all of it into one response. Neither accepts limit or skip, and neither emits the Pagination-* headers or the rel="next" Link that /query and /search have had since #310 and #314.

The shared root cause is getAllVersions() (controllers/utils.js:472):

ls_versions = await db.find({ "__rerum.history.prime": { $eq: rootObj['@id'] } }).toArray()

No limit, no skip. It reads the entire tree regardless of how much of it the caller will use. getAllAncestors() and getAllDescendants() then walk that array in memory and the whole result is handed to res.json().

Evidence

Measured on production, 2026-09-24, read-only

The tree rooted at 5c9534ecd5de4de2a7acde11 is a linear chain of 410 versions — 409 distinct previous values, every next of length 1, exactly one leaf.

An 11 MB unpaged response per request, and there is no way for a client to ask for less:

GET /v1/history/{id}?limit=1&skip=0  -> 200, no Pagination-* headers, no rel="next"
GET /v1/since/{id}?limit=1&skip=0    -> 200, no Pagination-* headers, no rel="next"

Cost is quadratic, and recursion depth tracks chain length

getAllAncestors() (controllers/utils.js:490) and getAllDescendants() (:525) each scan the full ls_versions array to find one neighbour, then recurse. On the chain above that is 409 recursive frames and roughly 168,000 comparisons to answer one request. The walk is not tail-recursive, so a sufficiently long chain reaches the stack limit rather than returning a slow answer.

From the original load testing (Run 4, March 2026)

• History Tree Attack: 50 VUs hitting /v1/history/{id} on an object with 100+ versions for 3 minutes. Each request loaded 100+ documents into memory simultaneously.

That contributed to the memory growth documented in #251, which has since been closed.

The constraint any fix has to respect

getAllVersions() has five call sites, and only two of them are the paged read path:

So the original recommendation — "add a limit to getAllVersions" — cannot be applied to the function as it stands. Capping it would silently truncate the release tree and the delete walk, turning a memory problem into a data-integrity one. The read path and the integrity walks have to be separated first.

Proposed change

1. Give /history and /since the same pagination contract the other read endpoints now have: getPagination() for limit/skip, the Pagination-* headers, and setNextPageLink() for rel="next". Both already import from controllers/utils.js, so the pieces are in place.
2. Split the read path off from the integrity walks. The ancestor chain and the descendant set can each be resolved by a query against __rerum.history.previous / __rerum.history.next rather than by pulling the whole tree and filtering it in application memory — the same move /search paginates in application memory, so every page costs the server everything up to that page #309 made for /search.
3. Leave delete and the two release-tree functions reading the full tree until they are addressed on their own terms. They are correctness-critical and are not what this issue is about.
4. Document the new contract for both endpoints in public/API.html.

Cursor-based continuation for deep paging is tracked separately in #303 and is not required here.

What this no longer covers

• queryHeadRequest — resolved. The handler no longer exists anywhere in the codebase; it was dropped via HEAD history - since - query #96 / HEAD /query answers an empty page with 404 where POST /query answers 200 [] #304 and shipped in /query optimization #312. Verified on production: HEAD /v1/api/query returns 405 with Allow: POST.
• /query default pagination — resolved by Behavioral Fixes for limit and skip #310 and /query optimization #312. Default limit 100, maximum 500 (configurable via MAX_QUERY_LIMIT), skip maximum 100000, with Pagination-* headers and rel="next". Note this landed at a maximum of 500 rather than the 1000 originally proposed here.
• Query Flood and Query Amplification load scenarios — both were POST /v1/api/query with broad filters, which the above now bounds.