v1.5.0: codegraph.db-wal grew to 64 GiB and filled disk on a Linux-kernel-sized repo #1539

Activity

1. TheOrdinaryWow commented on Aug 12, 2026

   

   TheOrdinaryWow

   on Aug 12, 2026

   More actions

   same here
2. maxmilian commented on Aug 23, 2026

   

   maxmilian

   on Aug 23, 2026

   Contributor

   More actions

   Your Q1 — "is there a path that bypasses the file cap?" — has a yes, and I think it's the one you hit.

   Both caps are only ever checked inside backpressure(), and sync() never calls it.

   src/db/wal-valve.ts:185 is the single place the hard cap and the file cap are read:

   if (this.growthBytes() <= this.hardBytes && this.db.getWalSizeBytes() <= this.fileCapBytes) return null;

   That line lives inside backpressure() (:174). The timer path — check() / fire() — only issues PASSIVE checkpoints; it never looks at fileCapBytes and never parks the writer.

   And backpressure() has exactly two call sites in the entire repo, both inside indexAll():

   src/index.ts:544   walValve ? () => walValve!.backpressure() : undefined
   src/index.ts:642   walValve ? () => walValve!.backpressure() : undefined
   

   sync() (:777–:1040) arms the valve at :801 and does one foldNow() at :823, but its two resolveReferencesBatched(...) calls at :902 and :968 pass only the two progress callbacks — the third parameter is left undefined. IndexOrchestrator.sync() (src/extraction/index.ts:2651) doesn't take a backpressure parameter at all, while indexAll() in the same file does.

   So during a sync there is no writer-side backstop and no file-size ceiling — only timer-driven passive checkpoints.

   The verdict is already written in the parameter's own doc comment (src/index.ts:1231-1236):

   "Without it the valve's only lever during resolution is timer-driven passive checkpoints, which the pool's continuous reads keep perpetually partial — the WAL then accretes the whole phase's write volume (22GB on a 4.6GB DB at kernel scale)."

   That is this bug, described in advance, for the path it wasn't wired into.

   How it got missed — the two commits are 16 hours apart:

   commit	date	what
   18f0745	2026-07-16 15:21	perf(sync): defer WAL autocheckpoint for the whole incremental run (#1312) — sync starts arming the valve
   6e52295	2026-07-17 07:48	fix(resolution): WAL containment for the pooled superphase — writer backpressure at pool-idle boundaries (#1332) — backpressure invented, wired into indexAll only

   That makes this the same shape as #1248, where the WAL-defer itself had never been wired into sync. Same mechanism, second time it landed on indexAll and not on sync.

   It also lines up with the reporter's setup: the daemon's catchUpSync() (src/mcp/engine.ts:296) calls cg.sync(), and the Query pool: up to 7 worker thread(s) line in their log is exactly the "continuous reads keep passive checkpoints perpetually partial" population. On a 3.5 GiB DB the file cap works out to roughly 3.4 GiB; 64 GiB is about 19× that, which is consistent with a ceiling that was never consulted.

   What I have not shown. I can't prove that this specific 64 GiB incident took that path — that needs kernel-scale indexing on a slow disk with a pinning reader, and the WAL debug log wasn't on. The Catch-up sync failed: cannot rollback - no transaction is active in the same log may be tangled up in it too. The static claims above (sync has no backstop, the caps are only read inside backpressure()) are the part I've verified line by line on main at 44e1812.

   One idea for a deterministic repro that wouldn't need a kernel-sized repo, which I have not run yet: CODEGRAPH_WAL_VALVE_MB=1 puts the file cap at ~4 MB, so a sync writing more than that with a reader holding a read transaction open should diverge from the equivalent indexAll run. If that reproduces, it also gives a regression test for the fix.

   On Q2 (should the valve fail closed, or check free disk) — that changes what users experience when it triggers, so I'd rather you set the direction than guess. The narrow version is a parity fix: pass the same walValve ? () => walValve!.backpressure() : undefined at :902 and :968 that :642 already passes, and thread the parameter through IndexOrchestrator.sync()'s store loop so sync and indexAll actually match. __tests__/wal-deferral.test.ts:303-366 already has the spy shape for asserting the hook gets called and the writer parks.

   Happy to send that if you want it — it's your instrumentation and your measurements, so I'd rather not put a PR into it uninvited. Whatever it ends up as, I'd scope the claim to "wires up the hook #1332 missed on the sync path" rather than "fixes #1539", since the 64 GiB end-to-end isn't something I can demonstrate.
3. yk4464 commented on Aug 25, 2026

   

   yk4464

   on Aug 25, 2026

   Author

   More actions

   Thanks for your reply, and thank you so much for putting together such a detailed analysis.

   Unfortunately, I'm unable to reproduce the original environment right now—I'm no longer using Linux, so I can't verify firsthand whether the 64 GiB growth indeed followed the path you pointed out. That said, your analysis aligns closely with the behavior and logs I observed at the time, and the explanation is very convincing.

   If I run into a similar issue in the future, I'll make sure to capture more complete diagnostic information and update this issue. As for the specific fix and whether to submit a PR, I'll leave that up to you and the project maintainers to decide.
4. maxmilian commented on Aug 27, 2026

   

   maxmilian

   on Aug 27, 2026

   Contributor

   More actions

   @yk4464 — no reproduction needed on your side, and thanks for saying so plainly rather than guessing.

   The gap I described is visible in the source regardless of environment: both caps are read only inside backpressure() (src/db/wal-valve.ts:185), and backpressure() has exactly two call sites, both in indexAll() — sync() arms the valve but never passes it, so its resolveReferencesBatched(...) calls run with no writer-side backstop. That doesn't depend on your machine, so the 64 GiB number is the only part your absence costs us, and it isn't load-bearing.

   Status update that matters for whether anyone should open a PR: #1583 (fix: bound large-repo memory and traversal work) already contains this wiring — its description states it applies the WAL pressure valve during incremental changed-file storage and both batched resolution paths, cites this issue, and reports a focused regression proving sync() invokes backpressure. So the containment hook sync() missed is covered there, pending review.

   That's why I'm not opening a separate PR for it: a second change to the same call sites would just have to be closed. If #1583 stalls or that hunk gets split out, the fix is small enough to land on its own and I'm happy to carry it then.
5. colbymchenry commented on Sep 8, 2026

   

   colbymchenry

   on Sep 8, 2026

   Owner

   More actions

   Findings + fix

   Confirmed on current main (no kernel-scale repro needed — the unbounded path is visible in source):

   1. sync() never called backpressure(). Both hard and file caps are only checked inside WalCheckpointValve.backpressure() (src/db/wal-valve.ts). Call sites were only in indexAll(). Daemon catch-up arms the valve + timer PASSIVE checkpoints, but with the query pool holding read marks those stay partial — matching the issue log (Query pool: up to 7 worker… / catch-up failure).
   2. Futility latch allowed unbounded growth. After consecutive parked-backfill give-ups, parking was disabled for 60s so the writer could continue past the documented caps (the path hypothesized in the report).

   Fix (PR #1751)

   • Wire backpressure into sync store (per changed file) and both batched resolution paths.
   • Fail closed: throw WalValveAbortError when parked backfills cannot progress while still past hard/file caps (no more parking-disable cooldown).
   • Unit tests prove abort over cap + sync invokes backpressure; soft fold under caps still does not abort.

   Leaving this issue open until #1751 merges. #1583 also mentioned sync backpressure wiring but is conflicting and mixes unrelated memory/traversal work — not a substitute for this focused containment.

   Diagnostics if anyone hits this again before the fix ships: CODEGRAPH_WAL_VALVE_DEBUG=1 / CODEGRAPH_SYNTH_TIMINGS=1, and watch for [wal-valve] backfill gave up / WalValveAbortError once on the fixed build.
6. colbymchenry

   added a commit that references this issue on Sep 8, 2026

   fix(db): fail closed when WAL valve cannot checkpoint past caps (#1539)…

   ...

   9b8bb4a
7. colbymchenry commented on Sep 8, 2026

   

   colbymchenry

   on Sep 8, 2026

   Owner

   More actions

   Merged via #1751 onto main.

   sync() now calls WAL backpressure and fail-closes with WalValveAbortError instead of the futility latch that allowed unbounded growth under query-pool readers. Verified with wal-deferral tests (24/24) on Linux — no full 64 GiB repro (disk-safe).
8. colbymchenry

   closed this as completedon Sep 8, 2026