Skip to content

Walk get-in and Pointer.get by reference instead of copying the document - #25

Merged
hellerve merged 1 commit into
mainfrom
claude/lookup-by-reference
Oct 29, 2026
Merged

hellerve merged 1 commit into
mainfrom
claude/lookup-by-reference

Conversation

@carpentry-agent

@carpentry-agent carpentry-agent Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

JSON.get-in and JSON.Pointer.get folded over the path starting from (Maybe.Just @doc), and every step copied the subtree it entered. So a lookup of one scalar copied the whole document plus every subtree on the path. Each RFC 6902 test/copy/move op (they go through Pointer.get) copied the document again, and update-in paid the same cost through get-in.

Both now call one recursive walker, JSON.get-in-at. It descends by reference and copies only the addressed value. Object steps go through Map.value-ref!, the same public API diff-members uses; array steps check bounds, then use unsafe-nth. Each entry point passes its own array-token rule, so neither accepts anything new. get-in keeps Int.from-string, so "01", "+1", " 1" and "" index as before. Pointer.get keeps array-index. Signatures and doc strings are unchanged.

The object step lives in a forward-declared get-in-member, because a recursive call reached through a closure segfaults (the same workaround as serialize-obj-into!). get-in-at is hidden but not private, because JSON.Pointer calls it and a private binding is only visible inside its own module.

Rebased onto 0.7.2 (core's open-addressing Map): suite 439/0 on carp-lang master cda0b32, angler and carp-fmt --check clean. A --optimize rerun of 1,000 lookups on the n = 10,000 document went from 394 ms on main to under 1 ms, with the same memory balance on both.

Timing, master → branch, on the Pi. The document is {"meta":{"id":1},"data":[0..n-1]} and the figures are totals over all reps (measured while other builds ran, so they are rough):

n reps get-in ["meta","id"] Pointer.get "/meta/id" get-in ["data", last] Patch.apply, 100 test ops
1,000 200 27.4 → 0.26 ms 24.0 → 0.50 ms 47.1 → 3.2 ms 11.9 → 0.67 ms
10,000 200 239 → 0.21 ms 224 → 0.50 ms 585 → 0.19 ms 219 → 1.7 ms
100,000 50 642 → 0.07 ms 576 → 0.14 ms 1318 → 0.05 ms 1229 → 13.0 ms

At n = 100,000, Patch.apply with an empty patch costs 9.4 / 9.2 ms; that is the one document copy it always makes. So most of what remains for the 100 test ops is that copy.

Verification

  • test/json.carp: 439 passed, 0 failed. Eight new assertions pin the token rules:

    • get-in: empty path, "01" on an array, index equal to the length, "-1", "-".
    • Both entry points: a numeric or leading-zero token on an object is used as a key.
  • Differential: 3,000 generated documents, with paths drawn from awkward tokens ("", "01", "+1", " 1", "-", "-1", ~0/~1, overflowing integers, numeric object keys, malformed escapes). Each case ran through get-in, Pointer.get, update-in and a test/copy/move patch. Master and branch output is byte-identical: 18,001 lines, same md5.

  • Debug.memory-balance returns to its starting value around 13 lookups and patches, on both master and branch. They cover hit, miss, out of range, into a scalar, empty path/pointer, invalid pointer, test/copy/move and a failing test.

  • 7/7 walker mutants are killed:

    • object miss check < → <=
    • no >= 0 check
    • array bound < → <=
    • Pointer.get using Int.from-string
    • empty path returns Nothing
    • index-like tokens refused on objects
    • a scalar step returns the scalar

    The >= 0 and index-like-token mutants pass master's suite (431/0); only the new assertions catch them.

  • angler and carp-fmt --check are clean. carp -x gendocs.carp builds, and none of the hidden helpers show up in the docs.

apply-test still copies the value it tests before comparing. Comparing in place would mean passing the leaf action into the walker, which didn't seem worth it to save one copy of the tested value.


Opened by the carpentry-org heartbeat agent (Claude). Veit has not reviewed this yet.

@hellerve

Copy link
Copy Markdown
Member

port to the new interface.

JSON.get-in and JSON.Pointer.get seeded their fold with a copy of the
whole document, and every step copied the subtree it entered. One lookup
of a scalar therefore cost O(document size), and every RFC 6902 test,
copy and move operation copied the document once more.

Both now use one recursive walker that descends by reference and copies
only the addressed value. get-in still indexes arrays with
Int.from-string and Pointer.get with array-index, so their token rules
are unchanged.

bucket-for and member-at move from JSON.Patch to JSON so the walker can
use them. They and the walker are hidden but not private, because
JSON.Pointer and JSON.Patch call them and a private binding is only
visible inside its own module.
@hellerve
hellerve force-pushed the claude/lookup-by-reference branch from 8a1268b to b7ba0ad Compare October 28, 2026 23:31
@hellerve
hellerve marked this pull request as ready for review October 29, 2026 00:17
@hellerve
hellerve merged commit 4360f62 into main Oct 29, 2026
2 checks passed
@hellerve
hellerve deleted the claude/lookup-by-reference branch October 29, 2026 00:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant