Repository navigation
Daemon OOMs (~4 GB heap) on small project during startup catch-up sync; stale daemon.pid + OS PID reuse then blocks all daemon restarts (unlock does not help) #1553
Description
Summary
Two related problems observed with v1.5.0 (latest) on macOS (Apple Silicon):
- The background daemon crashes with
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory— 178 times in this project'sdaemon.log— on a small project (524 indexed files, ~22 MBcodegraph.db). Memory balloons to the default ~4 GB heap cap during what looks like the startup catch-up sync, usually ~20–30 s after the daemon starts. - After the daemon dies it leaves stale
daemon.pid/daemon.sock/ lock files behind. If the OS later reuses the dead daemon's PID for an unrelated process, the daemon liveness check (PID exists ⇒ daemon alive) is permanently wrong: every new daemon exits withAnother daemon (pid …) already holds the lock; exiting, andcodegraph unlockdoes not fix it.
Environment
- codegraph 1.5.0 (latest on npm), installed globally via pnpm (bundled node from
@colbymchenry/codegraph-darwin-arm64) - macOS, Apple Silicon
- Project: React Native 0.84 app (iOS/Android/HarmonyOS), 524 indexed files,
codegraph.db≈ 22 MB;status --jsonreportspendingChanges: 0, index statecomplete— the index itself is healthy
Bug 1 — daemon OOM during startup catch-up sync
Typical crash pattern (most entries look like this — heap pinned at the ~4 GB default cap ~20–30 s after daemon start):
[44485:0xa4380c000] 21323 ms: Scavenge (interleaved) 4041.6 (4058.8) -> 4041.6 (4105.8) MB … allocation failure
[44485:0xa4380c000] 23695 ms: Mark-Compact (reduce) 4041.6 (4105.8) -> 4041.6 (4044.0) MB …
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
The first daemon in the log survived ~21 minutes before its first OOM (1260216 ms into the run) — consistent with the OOM being triggered by a large catch-up sync rather than steady-state serving. My hypothesis: a mass file-change event (e.g. npm install / patch-package touching many files under node_modules) makes the watcher enqueue a huge change set that is then processed in memory all at once; the watcher appears to follow the whole project tree including node_modules even though the index excludes it.
Workaround I verified — raising the heap cap makes the same startup sync survive:
NODE_OPTIONS=--max-old-space-size=12288 codegraph serve --mcp …
With 12 GB: the startup sync balloons to ~3.6 GB RSS, then completes and exits cleanly; the daemon settles at ~230 MB RSS, queries work, and no new FATALs occur. So this looks like a legitimately large transient allocation (or a leak confined to the sync pass) rather than a steady-state leak. Either way, ~4 GB for a 524-file project's sync seems well worth a look (streaming/chunking the change set, or excluding node_modules from the watcher).
Bug 2 — stale daemon.pid + OS PID reuse permanently blocks daemon startup; unlock doesn't help
After an OOM crash the daemon left daemon.pid (containing 54444), daemon.sock, and a lock file. Later, macOS assigned pid 54444 to a completely unrelated system process (cloudd). From then on:
- Every daemon start logs
[CodeGraph daemon] Another daemon (pid 54444) already holds the lock; exiting.(28 occurrences) — and this kept happening even aftercodegraph unlockremoved the lock file, because the liveness check appears to be "does the pid indaemon.pidexist" (a barekill(pid, 0)/process-exists check succeeds againstcloudd). codegraph daemonsreports the phantom daemon as real:pid 54444 v1.5.0 up 5358m …codegraph serve --mcp(spawned by editor/agent integrations at session start) can never become ready, so MCP clients just time out.
Only manual cleanup recovers:
rm .codegraph/daemon.pid .codegraph/daemon.sock
codegraph unlockSuggestions
- Verify daemon liveness by process identity (match the process command line / binary path against the codegraph install, or probe
daemon.sockwith a handshake) instead of relying on PID existence — PIDs get reused by the OS all the time. - Have
codegraph unlock(or daemon startup) also invalidatedaemon.pid/daemon.sockwhen the pid fails identity verification. - Consider a crash/exit handler that cleans up
daemon.pid/daemon.sock/ lock on exit.
Happy to provide the full 2.1 MB daemon.log if useful — it contains all 178 OOM stack traces.
Activity
colbymchenry commentedon Aug 20, 2026
The second half of this report — the stale
daemon.pid+ OS PID reuse permanently blocking daemon restarts — is fixed in #1562 (thanks @danusha2345): daemon management now proves a recorded process is really a CodeGraph daemon via its socket handshake before trusting or signaling it, a reused PID can never wedge startup or be signaled by mistake, andcodegraph unlockclears stale daemon artifacts too. That ships in the next release.The first half — the daemon OOMing at the ~4 GB heap cap during startup catch-up sync on a 524-file project — is still unexplained and stays open here. The 178 crash logs you captured are exactly the right evidence; if you can share one more thing, it would help a lot: the tail of
.codegraph/daemon.logfrom daemon start to the first OOM (the phase labels would tell us whether it dies in scan, parse, store, or resolution), and whetherCODEGRAPH_PARSE_WORKERS=1changes the behavior. The upcoming release also fixes a related class of recovery-sync failures (#1541, #1558) that could have been compounding the churn on that machine.hcg1023 commentedon Aug 21, 2026
Thanks for the #1562 fix — that matches what we captured exactly (the recycled PID had been handed to
cloudd,codegraph daemonskept reporting the phantom as healthy, and every newserve --mcpdied on the lock). For what it's worth, both of our daemons are dead again right now with stale registry entries — still on 1.5.0 here, so expected until the release lands.Phase where it dies: resolution
The daemon log itself never prints phase labels before the crash — crash-loop cycles go straight from
Listening onto the GC dump with nothing in between (the one long-lived daemon that survived 21 minutes only ever printed per-file auto-sync lines). Butcodegraph syncruns the same pipeline in the foreground and does print phase labels, and it reproduces the crash deterministically.Default env (reproduces; crash signature matches the daemon's — GC offsets 19–23 s after start, heap pinned at ~4.03 GB, same
Ineffective mark-compactsFATAL):CODEGRAPH_PARSE_WORKERS=1: no change. Same phase (Resolving refs...), same ~4 GB, same ~21 s to FATAL.The working set just barely exceeds the default heap cap
With
NODE_OPTIONS=--max-old-space-size=12288, the identical sync completes:Peak RSS of the sync child process: 4.34 GB — resolution on this project builds a >4 GB structure, so the default ~4 GB heap cap makes it a guaranteed OOM rather than an unbounded leak.
Two details that may help narrow it down:
pendingChangeseither, yet the full scan→resolve pipeline still ran and still needed the full 4.34 GB. So the resolution working set appears to be rebuilt from scratch on every sync regardless of the change count — it never gets a chance to be incremental.RSS trajectory of the OOMing child (default env): 5 s → 1.8 GB, 10 s → 3.7 GB, 16 s → 4.1 GB, dead at ~26 s.
Requested daemon.log tail (start → first OOM)
Last complete cycle in the current log, sanitized (paths masked). Note the zero phase lines between
Listening onand the GC dump:.codegraph/daemon.log — last start→OOM cycle (pid 35455, native stack truncated at 42 frames)
Environment
@colbymchenry/codegraph-darwin-arm641.5.0, system Node v24.8.0)node_modules, 613 indexed filesWorkaround for anyone hitting this in the meantime: raise the heap for the daemon, e.g.
NODE_OPTIONS=--max-old-space-size=8192in the environment that spawns it — measured peak is 4.34 GB here, so 6–8 GB is comfortable at this project size. That's obviously just papering over the per-sync 4 GB resolution cost.hcg1023 commentedon Aug 21, 2026
Correction to our previous comment, plus new measurements — two of our earlier claims were wrong, and the experiments below replace them with a cleaner picture.
What we got wrong
Also a small correction to the workaround sizing: the catch-up path peaked at 4.34 GB, but a full re-index peaked at 6.49 GB —
--max-old-space-size=8192is fine for catch-ups, but go higher (we used 12288) if you re-index.New measurements (1.5.0, M1 Pro / 32 GB, heap raised so nothing dies mid-run)
codegraph index(default)codegraph index+CODEGRAPH_NO_PARALLEL_RESOLVE=1Side note:
pendingChangesis content-hash based — touching ~60k files by mtime alone registers as zero pending.So the OOM is precisely: full resolution pass working set (4–6.5 GB on this project) vs the default ~4 GB heap cap, triggered by full re-index or daemon restart with pending churn. Steady-state incremental syncs never approach it.
Dependency directories are not involved
This is a React Native HarmonyOS (RNOH) project, so besides
node_modules(6.3 GB) it also hasharmony/oh_modules(76,964 files / 1.1 GB),ios/Pods(9,862 files), and android build outputs (3.2 GB). None of them are indexed (0 rows infiles/nodes; the harmony/ entries in the index are 30 hand-written.etsfiles), and none are read during indexing —lsofsampled across scan/parse/resolve/link of a full index shows 0 fds into node_modules / oh_modules / Pods at every resolution-phase sample. (The only 2node_moduleshits were CodeGraph's own pnpm-store runtime files during linking.)What the full pass is actually resolving
The 613 indexed files total 3 MB on disk (largest 50 KB — no giant-file effect) and are multi-language: 231
.tsx/ 224.ts/ 49.py/ 35.ets/ 31.js/ 14.kt/ 10.h/ 5.cpp/ 4.mm/ 1.swift. Unresolved refs: 22,399, allstatus='failed', regenerated by every full extract — tsx 10,720 / ts 6,337 / js 2,263 / py 1,430 / arkts 778 / kotlin 506 / objc 237 / cpp 76 / swift 52 (~36 refs per indexed file).For comparison, another project on this machine (3,090 files, 35,616 nodes, 56,966 unresolved refs — plain Vue/TS, ~18 refs/file) indexes and catches up fine under the default heap and has never OOM'd once. So it's not ref count, not file count, not node_modules — it's the per-ref cost of this project's language mix (the ArkTS/Kotlin/ObjC/Swift resolution machinery plus a heavy tsx share) that turns the pass into multi-GB.
What we couldn't pin down
We tried
NODE_OPTIONS='--heap-prof'; profiles were only written for bootstrap-sized isolates (largest 25 MB total self-size, top framesreadFileSync/rowToNode @ queries.js/ sqlite-adapterall), so the multi-GB allocations landed in an isolate that never wrote a profile (or the bundled binary doesn't honor NODE_OPTIONS there). Naming the exact structure will need instrumentation on your side — happy to run any flags/build you want to point us at.danusha2345 commentedon Aug 21, 2026
I pushed an opt-in diagnostic build that samples the main-thread resolver every 250 refs without logging paths or symbol names:
eb11399It reports RSS/heap plus the live sizes of every resolver LRU, method-owner/kind index, deferred-ref list, receiver-inference/regex memo, and import/export memo. This should distinguish a tracked cache retaining the 4–6.5 GB from allocation outside those structures.
I validated the probe locally on:
Could you run this against the affected project?
Please attach
resolve-memory.log. The probe logs only language, reference kind, file extension, reference-name length, aggregate cache counts, and memory totals — no file paths or reference names.Local evidence has already ruled out raw ref count, resolver workers, parse-worker lifetime, and the global compiled-regex memo as sufficient causes. The first few samples where heap jumps should identify the retaining structure or show that it sits below/outside the tracked JS caches.
diagnostic(resolution): sample warmup and first batch for colbymchenr…
hcg1023 commentedon Aug 22, 2026
Ran the diagnostic build (
10f164f, current HEAD of the branch — one commit pasteb11399) against the affected project with your exact command. The fullresolve-memory.logis inlined below (137 probe lines, no file paths or symbol names).Probe verdict: the 4–6.5 GB sits entirely outside every tracked structure. All LRU / owner-index / memo counts stay tiny for the whole run;
extMemnever exceeds 62 MiB, yetheapjumps from 45 MiB to 5.42 GB in a single sampling interval (betweenrefs=3250andrefs=3500, ~4.5 s in — the first ArkTS batch).CODEGRAPH_NO_PARALLEL_RESOLVE=1unchanged, matching your earlier local exclusions.We then chased the jump independently with an fs spy (preloaded
--requirepatch logging everyreadFileSync) plus a heap snapshot taken at the moment of the blow-up:\x1f\x8b); everything else totals ~65 MB — i.e. ~95% of the 5.7 GBheapUsedis churn around that string, not a retained cache.readFileSyncofnode_modules/@react-native-oh/react-native-harmony/react_native_openharmony.har→ 289,606,387-byte string (152 MB gzip decoded as utf-8). One earlier failed probe of<same path>/oh-package.json5(~2 s before) shows the resolver first treating the.haras a package directory.harmony/entry/oh-package.json5declares"@rnoh/react-native-openharmony": "file:../../node_modules/@react-native-oh/react-native-harmony/react_native_openharmony.har"— standard for RNOH projects, the npm package ships the HarmonyOS archive as afile:dependency target. The ohpm workspace machinery maps the ref to the.harand the resolver ends up reading the archive itself as text.node_modulespresent → reproduces (3.7 GB peak, same first-ArkTS-batch jump);oh_modules(1.1 GB unpacked) orios/Podsalone → no jump (350 MiB); neither → 438 MiB. So it's precisely this one file insidenode_modules, not ref count, not dependency-dir size.Suggested guards (any one would fix us): in the resolver's content-cache read path, skip files by magic bytes (gzip/zip) or a size cap (e.g. >10 MB), and/or treat an ohpm
file:target that is a file rather than a directory as external — never read it as entry text.Happy to run any follow-up probe builds.
resolve-memory.log (full, 137 lines)
danusha2345 commentedon Aug 22, 2026
Thanks — the
.harreadback evidence pinned the remaining OOM to a precise pre-cache allocation path.I added a pre-read size guard in #1583 (
f3fd47e): resolution nowstats a package target and rejects anything above the same 1 MiB source limit used by extraction beforereadFileSync(..., 'utf-8')can decode it. The failed result is cached asnull, so the target is not retried during the pass.Regression coverage uses a sparse gzip-shaped
react_native_openharmony.har; the test fails on main and passes on this commit. A separate 290 MiB sparse-archive probe against the built branch returnednullin 0.05 s with 58 MiB peak RSS. Full suite: 3,131 passed, 9 skipped.Could you run the affected project against #1583 when convenient? That is the only remaining validation needed for the exact RNOH/ohpm path on macOS.
hcg1023 commentedon Aug 22, 2026
Validated #1583 (
f3fd47e) against the affected project on macOS (Apple Silicon). Built the PR head, ran the same fullindexpipeline that used to OOM:node_modulessymlinkAn fs spy on the run confirms the guard fires as designed: at ~2.7 s the resolver
statSyncsreact_native_openharmony.har, reads nothing, and moves on (the pre-fix run read 289,606,387 bytes at that exact spot). Index completeness is identical to pre-fix: 627 files / 7,885 nodes / 18,973 edges / 22,727 failed unresolved refs — the rejected archive contributed nothing, so no resolution coverage is lost.One minor observation, not a blocker:
.harfiles under the 1 MiB cap (e.g.screens.har183 KB,worklets.har98 KB from@react-native-ohos/*packages) still get decoded as utf-8 into the content cache. Memory-wise they're harmless (~1 MB total), but they're gzip binaries, so their "contents" are mojibake — a magic-byte check (gzip\x1f\x8b/ zipPK) alongside the size guard would close that gap if you ever want it.From our side this fixes the RNOH/ohpm OOM path completely. Thanks for the fast turnaround on this!
bompus commentedon Sep 5, 2026
For anyone still on 1.5.0: #1562 (81e1f4a) is in the v1.6.0 release, so the PID-reuse half of this report is fixed there. The OOM half stays with #1583.
fix(resolution): reject oversized targets before decoding (#1553) (#2006
colbymchenry commentedon Sep 27, 2026
Thanks for the detailed report, @hcg1023, and thanks for using CodeGraph! This is fixed on
mainby #2006: files over the 1 MB source limit (such as afile:*.harpackage archive an import points at) are now rejected before they are read and decoded during resolution, which removes the multi-gigabyte heap spike behind the startup out-of-memory. The fix will be in the next release.fix(daemon): 身份证明之后再拒绝向自身/父进程发信号
fix(cli): index no longer blocks on a daemon lock naming a process th…
fix(cli): index no longer blocks on a daemon lock naming a process th…