Skip to content

fix(land): a known flaky test's verdict names it by its quest - #597

Draft
kixelated wants to merge 3 commits into
mainfrom
quest/m0/queue-watcher/flaky-signature
Draft

kixelated wants to merge 3 commits into
mainfrom
quest/m0/queue-watcher/flaky-signature

Conversation

@kixelated

@kixelated kixelated commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Finishes quest/m0/queue-watcher/flaky-signature.md.

Summary

The watcher matched a flaky verdict to its test's record by signature(failure), so a fixer that reworded a known test (with or without the command that ran it) started a new record, and no test reached FLAKY_VERDICTS.

  • A flaky requeue of a known test now carries that known entry's quest (quest/m0/flaky/<name>.md), copied from the input instead of retyping the name. recordFlaky finds the record by quest; only a requeue with no quest falls back to the signature.
  • parseFixResult takes the known list and rejects a quest that is not a known one. disable now requires quest, since only a listed test may be disabled. A flaky requeue must name its test (failure or quest), so it never falls back to the job's step name and merges unrelated tests.
  • assets/land-fix.md: name a new failure by its package and test alone, without the command, and add quest for a known test.
  • Test: the disable test's four verdicts use the wordings seen on moq-dev/moq (cargo nextest run --locked --tests: render_timeout, moq-relay render_timeout, render_timeout) and land in one record that disables at the fourth.

Public API and wire impact

The fixer's result format gains an optional quest field on a flaky requeue; disable without it is now rejected. Internal to quest land, nothing in production.

Choices made without asking

  • Dropped nothing: the first flaky verdict guard in recordFlaky stays, though a known test always has a verdict already, because a re-collected job replaces its own verdict.
  • Kept the signature fallback for a new test rather than stripping a command prefix heuristically: the prompt change asks for the test alone, and a parser for every runner's prefix would be guesswork.

Checks

bun run check (tsc, prettier), quest check, and vitest run test/land.test.ts test/land-watch.test.ts (247 passed) ran locally. The full just check / just test sat in quest gate behind host load (52) for over an hour and did not run; CI covers them.

Decisions

  • Cause of the split records: ✅ accept the code reading (rejected: confirm from flaky.json on the watcher's host first). A record is created only on a signature miss, so rewording a known test is the only way records split. Nobody checked flaky.json on the watcher's host.

Models

  • Implementer: Claude Code, claude-opus-5-5 (iterate, S)

(Written by Claude Opus 5.5)

@pullfrog

pullfrog Bot commented Nov 10, 2026

Copy link
Copy Markdown

Pullfrog  | Review this ➔

kixelated and others added 2 commits November 10, 2026 00:10
The fixer retyped a known flaky test's failure, sometimes with the command
that ran it, so each verdict started a new record and none reached
FLAKY_VERDICTS. A flaky requeue of a known test now names that entry's quest,
which only a known test may disable, and a new one names its test without
the command.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…aky tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title A flaky test's second verdict finds its first fix(land): a known flaky test's verdict names it by its quest Nov 10, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant