Skip to content

test(ts): check the differential harnesses' step lists, named cases, and kept references - #253

Merged
kixelated merged 2 commits into
mainfrom
quest/m3/ts-harness
Oct 30, 2026
Merged

kixelated merged 2 commits into
mainfrom
quest/m3/ts-harness

Conversation

@kixelated

@kixelated kixelated commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The differential harnesses could silently test less than they claim:

  • The task fixture matches harness.rs's and tests/task.ts's step lists by order only. A TS step whose name no longer matches the Rust step at its index still passes, because both sides run the same function, but a scripted case written by name then runs another step. For example, swapping only the names helpers and boxed in task.ts passes main's task test.
  • stock's caller.ts picks buf::reserves as all().length - 1. With a case appended after it, main's stock test still passes while its reserves inputs run the new case.
  • fix(ts): read a tuple's parts as they were when it was made #148's review asked for differential coverage of reborrows through a reassigned reference, and of indexed borrows.

Approach

  • Task steps: each step logs step <name> on both sides, so the logs differ wherever the lists do. The exhaustive single-op cases run every index, so any drift fails, and the failure names both steps.
  • Stock cases: harness::cases() lists (name, fn) pairs and gains the root harness::op(name), as lang's does. caller.ts targets op("buf::reserves"), which panics on a name that no longer exists.
  • refs audit: I instrumented the fix(ts): read a tuple's parts as they were when it was made #148 predicate (rereads / rvalue_rereads, ts/src/lower/mir/mod.rs:1699) and translated lang (instrumentation not committed). The table below shows which branches refs reaches. refs::kept adds the two shapes the review asked for: a tuple of references read after the array index they borrowed at moves on, and after the reference they reborrowed through is reassigned, directly, through a struct field, and via a copy.
Branch refs before refs after Notes
held value yes yes
alias def true only true only
plain place yes yes
constant no no
copy of a place no yes
borrow: index fixed no no see below
borrow of a variable yes yes
borrow of a non-variable root false only false only
borrow, deref at root false only both
borrow, deref past root no no unreachable, see below

What the audit found, checked by forcing each branch to true and diffing lang's translation and differential run:

  • Deref past the root (Some(_) => false) is unreachable. The translator reads optimized_mir, and the MIR validator only allows Deref as a place's first projection. A reborrow through a reassigned reference arrives as a CopyForDeref temp plus a deref at the root.
  • The index guard never decides output. Index expressions fail Expr::is_path, so an indexed part is always materialized. Even with is_path admitting them, the borrow indexes MIR's single-assignment copy of the index (_42), which is fixed.
  • No output change for these shapes. Forcing any or all of the rereads and rvalue_rereads branches to true leaves kept's output unchanged: a held value that reads a later-written variable is flushed before the write. So kept pins the semantics against future changes to either mechanism, not today's predicate arms.
  • Untested false answers. Across all of lang, forcing the plain-place, copy, deref-at-root, index, or other-rvalue branches to true passes the differential run, with no or behavior-identical output changes. Only the held, alias, and borrow-of-non-variable branches' false answers are load-bearing in lang.

Each check fails on a broken harness first

  • Task: swapping only the names of helpers and boxed in task.ts:
    • main: test result: ok
    • this branch: 35366 cases, 4988 panics, 8122 differences, first input 0e: native step helpers, ts step boxed
  • Stock:
    • Appending slices after buf::reserves with main's caller: test result: ok. A probe asserting the picked case is still reserves throws reserves is no longer the last case.
    • Renaming the case to buf::reserve on this branch: Panic: no case buf::reserves, test fails.
  • refs: see the audit above. No predicate mutation changes kept's output, so the case guards semantics rather than one arm.

Cost

  • Task test: 16.6 / 16.2 / 16.7 CPU-s before and 14.3 / 15.8 / 18.7 after, over three warm runs each (cargo test --test fixtures task, user + sys of the process tree). That is within noise.
  • kept: runs only on random inputs that pick refs (about 1/90 of lang's cases).
  • Stock: op runs once at import.

Impact

None: test fixtures and the task test runner only. stock's ts.toml gains the root harness::op, so its snapshot gains op and named cases. lang's snapshot gains refs::kept, Nest, and Held.

Alternatives

  • Compare the two step-name lists directly in a test. The native list lives only in the wasm build, which exports run alone, so this needs a second export or parsing source text. Logging names reuses the differential run and also labels every failure.
  • Generate one list from the other: the quest ruled this out.
  • Name lang's sub-op scripts (cells, locks) too. Their literal bytes pick sub-ops modulo a fixed match, so an append moves none of them. Not done.

Choices made without asking

  • The step marker is step <name>, logged before the step runs.
  • stock's case names follow lang's (the called path; local cases use their bare name).
  • The audit scope is the fix(ts): read a tuple's parts as they were when it was made #148 predicate, rereads and rvalue_rereads, which decides how a reference part of an aggregate is lowered.

Follow-ups

  • None. The deref-past-root arm and the index guard are conservative and cost nothing, so I left them in place rather than delete them on an invariant MIR keeps today.

Review

  • Reviewer: quest review, Codex gpt-6.1-sol, medium effort
  • Head: c247b8e
  • Round 1: 0 findings.

Local checks: just ci passed on this head.

🤖 Generated with Claude Code

(written by Claude Opus 5.5)

kixelated and others added 2 commits October 29, 2026 21:10
Quest-Claim: 41d91de2d6de9d75
…and kept references

The task harness logs each step's name on both sides, so a step list that
drifts, or a name that no longer picks its step, fails the differential run.
stock names its cases and its caller targets `buf::reserves` by name, as
lang's does. lang's `refs` gains references kept in a tuple and read after
the index or reference they were borrowed through changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Checks on the differential harnesses test(ts): check the differential harnesses' step lists, named cases, and kept references Oct 30, 2026
@kixelated
kixelated marked this pull request as ready for review October 30, 2026 04:52
@coderabbitai

coderabbitai Bot commented Oct 30, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 15 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 693ff099-35ca-4e27-91b6-8816d15c01b1
Commits

Reviewing files that changed from the base of the PR and between e51ecad and c247b8e.

Files selected for processing: 12
  • quest/m3/README.md
  • quest/m3/ts-harness.md
  • ts/fixtures/lang/expected/refs.Held.ts
  • ts/fixtures/lang/expected/refs.Nest.ts
  • ts/fixtures/lang/expected/refs.ts
  • ts/fixtures/lang/src/refs.rs
  • ts/fixtures/stock/caller.ts
  • ts/fixtures/stock/expected/harness.ts
  • ts/fixtures/stock/src/harness.rs
  • ts/fixtures/stock/ts.toml
  • ts/fixtures/task/src/harness.rs
  • ts/tests/task.ts
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed c247b8e7895e11feb58fcdd718863f24e7e22c86 against base e51ecadaf9ff14fb779915e75dd492e7be61a5ec.

No actionable bugs found. The differential runner's exhaustive single-op inputs make the new native/TS step-name logs detect drift in the current step lists, including steps that otherwise do nothing without a channel. Stock's named lookup preserves its case order and removes the fragile last-index assumption. The new reference fixture and generated snapshot preserve the indexed value and original reborrow targets across subsequent index/reference reassignment.

Direction: reusing differential output and a named case lookup is simpler than adding another native export or generating one step list from the other. The PR correctly distinguishes semantic regression coverage from proof that every lowering-predicate branch is independently necessary.

Verification limits: GitHub-only static review with an independent check. No compilation, differential runs, branch instrumentation, mutation experiments or timing measurements were performed; the reported local results and exhaustive branch-audit claims were not independently reproduced.

(Written by OpenAI)

@kixelated
kixelated merged commit b29d80f into main Oct 30, 2026
2 checks passed
@kixelated
kixelated deleted the quest/m3/ts-harness branch October 30, 2026 05:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant