Skip to content

review-pr-comments-converge: skip feedback already answered - #297

Draft
ooloth wants to merge 19 commits into
mainfrom
262-skip-feedback-already-answered
Draft

ooloth wants to merge 19 commits into
mainfrom
262-skip-feedback-already-answered

Conversation

@ooloth

@ooloth ooloth commented Nov 17, 2026 •

Copy link
Copy Markdown
Owner

💪 What

  • A second run of review-pr-comments-converge on a PR now skips feedback the PR author already answered, and its report lists every skipped item with the post that counted as the answer.
  • fetch_pr_comments.py prints one new answered line on every review body, inline thread and conversation comment (yes with its evidence, no, or n/a (yours)), plus a # Answerers: line naming the PR author and the gh login running it. Every line it printed before is still printed, some indented deeper, so review-code, which reads the same output, sees what it saw before.
  • An inline thread counts as answered when the author's reply is its newest comment and nothing in it changed at or after that reply, or when its one reviewer resolved it and hasn't posted or edited since the author's latest reply in it. A resolution by a bot, a deleted account, anyone else, or the author with no reply doesn't count. Comments count from when they were published, not from when they were drafted in a pending review.
  • A review body or conversation comment counts as answered only by a later top-level post from the author that names the reviewer: @login where GitHub would link it as a mention, or a bot's name standing on its own (Thanks, Copilot. counts, .github/copilot-instructions.md does not). Names inside code don't count, so the fetch now parses posts with markdown-it-py, declared inline so uv installs it on first run. A post that names nobody, such as "Rebased on main", answers nothing.
  • One post that names a reviewer answers all of that reviewer's earlier review bodies and conversation comments, not only the one it replies to.
  • Feedback edited or posted after the answer, even in the same second, is never skipped.
  • Before posting, the skill fetches again and compares each item's answered line with the one it saved. An item that got answered in the meantime has its draft dropped, and an item that got new feedback is validated again. Nothing about what was answered is stored between runs.
  • The fetch now exits 1 and prints nothing when it can't read a time, a comment's state or the gh login, and a reply from a deleted account no longer crashes it.
  • Every body the fetch prints (review bodies, replies, comments) is indented deeper than its own field lines, so a commenter can't write a line that passes for an answered verdict. In a reply, the Unicode line separators U+2028, U+2029, U+0085, \v and \f print as escapes such as \u2028, so no reader can split one into a line of its own.

🤔 Why

  • monitor-prs starts a fresh run whenever new feedback lands, and each run used to re-validate and draft a reply to every comment on the PR, so feedback that already had an answer got a second one.
  • Why the answer has to name the reviewer: on Add project overview, integrations, and dev commands to CLAUDE.md michaeluloth.com#83 the author's closing remark "This one somehow allowed scope to creep beyond its original goal." would otherwise have counted as the answer to 9 earlier review bodies and comments, so a run would have skipped real feedback without saying why. A wrong skip is easy to miss, while an extra draft is caught when the author reads the drafts before approving them.
  • What this costs: a hand-written answer that names nobody still gets a drafted reply, an item from a deleted account is never skipped, and an edit to an item that was never answered isn't noticed by the check before posting. monitor-prs still has its own, looser idea of answered feedback, which the ticket leaves for later.
  • Still open on the ticket: a bot's name used in passing, such as "tested in a Rex tab" or "GitHub Copilot", counts as addressing that bot and answers all its earlier reviews.

👩‍🔬 How to validate

  1. 🛠️ Setup

    💻 A checkout of this branch, with uv and a signed-in gh (every claim)

    • Check out this branch from a clone of ooloth/dotfiles and stay at the repo root for every command below:

      gh pr checkout 297 -R ooloth/dotfiles
  2. 📄 A closing remark that names no reviewer answers nothing

    🔍 Verification steps

    • Fetch michaeluloth.com#83, which has a closing remark from the author after several bot reviews:

      uv run tools/agents/config/skills/review-pr-comments-converge/scripts/fetch_pr_comments.py 83 --repo ooloth/michaeluloth.com | grep -E '^# Answerers|answered'
    • Expect # Answerers: ooloth (PR author and gh login) first, then answered : no on every review body and bot comment (the remark names no reviewer), answered : no on the one resolved inline thread too, because Copilot resolved it and nobody replied, and answered : n/a (yours) on the remark itself

    👀 What I saw

    Every review body, bot comment and inline thread says no, including inline thread PRRT_kwDOQQJOc86D-GpE, which Copilot resolved, and the last line is the closing remark (run at c89d8ff, output trimmed where 24 more no lines repeat):

    # Answerers: ooloth (PR author and gh login)
      answered   : no
      answered   : no
      ...
      answered   : no
      answered   : n/a (yours)
    

    Before the rule that an answer must name the reviewer, the same fetch printed this for the 9 items that came before the remark:

    before:  answered   : yes (conversation comment 4713290181 by ooloth at 2026-06-15T23:22:06Z)   × 9
    after:   answered   : no                                                                          × 9
    
  3. 📄 An answer that names the reviewer counts as answered

    🔍 Verification steps

    • Fetch a PR where the author answers by name:

      uv run tools/agents/config/skills/review-pr-comments-converge/scripts/fetch_pr_comments.py 18214 --repo astral-sh/uv | grep 'answered   : yes (conversation comment' | head -3
    • Expect lines such as answered : yes (conversation comment 4068918466 by EliteTK at 2026-03-16T16:19:47Z), where that comment opens with @Da1sypetals

    👀 What I saw

    Two items are answered by EliteTK's comment 4068918466, and another by a later comment:

      answered   : yes (conversation comment 4068918466 by EliteTK at 2026-03-16T16:19:47Z)
      answered   : yes (conversation comment 4068918466 by EliteTK at 2026-03-16T16:19:47Z)
      answered   : yes (conversation comment 4141526738 by EliteTK at 2026-03-27T10:06:18Z)
    
  4. 📄 Apart from indentation, a trailing carriage return and Unicode line separators shown as escapes, every line main printed is still printed

    🔍 Verification steps

    • Compare against main's output for the same PR:

      git show main:tools/agents/config/skills/review-pr-comments-converge/scripts/fetch_pr_comments.py > /tmp/old_fetch.py
      diff <(uv run /tmp/old_fetch.py 18214 --repo astral-sh/uv | sed 's/^ *//' | tr -d '\r') <(uv run tools/agents/config/skills/review-pr-comments-converge/scripts/fetch_pr_comments.py 18214 --repo astral-sh/uv | sed 's/^ *//' | tr -d '\r') | grep '^<'
    • Expect no output: apart from indentation, a trailing carriage return and Unicode line separators shown as escapes, no line main printed has been removed or changed

    👀 What I saw

    The step printed nothing, and grep exited 1. Without tr -d '\r', the same diff prints 78 lines. Each is a line from a reply written with Windows line endings, which main prints with a trailing carriage return (^M) and this branch prints without one. The first of those lines, cut short:

    < EliteTK: We shouldn't need to worry about `MAX_PATH`, we should be passing windows extended length paths (`\…
    < ^M
    < EliteTK: To figure out the validity, we need to know the correct cache key, at which point we can just recal…
    < ^M
    

    An earlier check ran the same fetch with main's script and this branch's on 20 public PRs: both made the same GitHub requests, and no line main printed was lost or changed apart from review bodies and multi-line replies now being indented deeper.

  5. 📄 The skill validates only unanswered items and lists the rest as skipped

    🔍 Verification steps

    • There are no steps a reviewer can run for this claim. It was checked by giving fresh agents the edited skill and saved fetch output, and comparing them with agents given main's skill.

    👀 What I saw

    Fresh agents given the edited skill and saved fetch output sent only the no items to validation, listed the rest under ⏭️ Skipped, and handled the before-posting check as described, while agents given main's skill did not.

  6. 🧹 Clean up
    • Remove the copy of main's script:

      rm /tmp/old_fetch.py

🔗 Related links

Fixes #262

#262

A second run of review-pr-comments-converge re-validated every comment
on the PR and drafted a reply to each, including feedback an earlier
run had already answered. The fetch output did not show the PR author,
edit times or who resolved a thread, so a run could not tell.

fetch_pr_comments.py now asks for the viewer, the PR author, who
resolved each thread, and each post's state and times in the same
request. Before printing, it gives every review body, inline thread
and conversation comment one verdict: answered with its evidence, not
answered, or not feedback because only the PR author or the gh login
wrote it. It prints a "# Answerers:" line and one "answered" line per
item, and exits 1 with a sentence when a time, a comment state or the
viewer cannot be read. Times compare as datetimes, a same-second tie
counts as new feedback, and a deleted account never answers and no
longer crashes the REPLIES lines.

SKILL.md puts every item not marked "no" on the decided list before
validation, lists it under "Already answered" with its evidence, and
reports "Nothing unanswered" when nothing is left. The check before
posting compares fresh and saved answered lines, so a skip is read
from the PR each time and never carried over.
#262

Mutation testing of fetch_pr_comments.py left 20 changes uncaught.
The fake gh returned its fixture whatever the query asked for, so
dropping a field from QUERY changed nothing in the tests. It now
returns only the fields the query selects, so a field the script reads
without requesting fails the suite.

New examples pin the rest: a reply's posted time rather than its edit
time, every other comment in a thread checked against the latest
reply, reviews with no submit time, a bot sharing the viewer's login,
a review answering from when it was submitted, pending reviews that
carry a submit time, the earliest answer chosen by posted time, the
isResolved flag, per-list request flags on later pages, and an empty
viewer login or non-string and null times exiting 1.
#262

Phase 2 said the list of skipped items "is rebuilt from the saved
fetch at the start of each pass". Fresh agents replaying Phase 4's
check read that as the whole decided list, and one re-validated items
the first pass had already fixed, which would turn an approved reply
into a "stale" one.

Only the Already answered entries are rebuilt now. An item an earlier
pass decided stays decided, and its draft goes into the new report for
fresh approval unless its answered line changed to yes. The report
section also says it lists the author's own items as well as answered
ones.
#262

A wrap-up read of the branch found claims the change had made untrue
or left unclear.

Phase 2 kept every item an earlier pass decided, so a reviewer's
follow-up in a thread the run had already fixed would never be read,
though Phase 4 restarts a pass for exactly that case. An item Phase
4's check lists, with its line still "no", now comes off the decided
list and is validated again with the new feedback. Step A reads the
saved fetch rather than Phase 1's, "Nothing unanswered" applies only
when no earlier draft carries over, and the check before posting names
the edit times the fetch actually prints.

The answered wording in SKILL.md and the script's docstring now says a
resolved thread counts whoever resolved it. review-code says its own
reconciliation ignores the answered lines, which are judged from the
PR author's side. The README mentions the skip, the test file's run
command adds hypothesis, and the print path reads a review body the
same null-safe way the judging step does.
#262

A review body or conversation comment counted as answered by any later
top-level post from the PR author or the gh login. On a real PR one
unrelated closing remark from the author marked nine earlier review
bodies and comments as answered, so a second run would have skipped
feedback nobody had answered.

An answer now has to address the item's author too: it opens with the
item's thank line, or names the author the way refer_as does, @login
with a boundary for a person or the BOT_NAMES name as a whole word for
a bot, in any case. The match is decided from the Reviewer variant, so
a deleted or unrecognised author can never be addressed and no
top-level post answers their items. A post that names nobody answers
nothing, and the run drafts a reply the author can decline instead of
skipping silently. Inline threads are unchanged, since GitHub links
each reply to its thread.
#262

Mutation testing of the addressing rule left two regex changes
uncaught: dropping the @ so a bare login addressed its author, and
requiring a boundary before the @. New cases pin both: "Thanks dana"
and a login inside another word do not address dana, while
"cc foo@dana" does, since the rule says @login anywhere.
#262

addresses() accepted a post that opened with the item's thank line, or
one that named the author. Every thank line for an author who can be
addressed already holds their @login or bot name, so the mention check
always matched first and the opening check could never change a
verdict. Mutation testing showed four changes to it as equivalent.

addresses() is now only the mention check, and the docstrings and
SKILL.md say an answer names the author the way refer_as does, as a
reply opening with the thank line does.
#262

A person counted as addressed by "@login" anywhere in a post, so
"foo@dana" or an email address such as "x@dana.io" marked dana's
feedback answered, though GitHub renders neither as a mention. A false
match skips feedback silently, so the @ must now not follow a word
character, a dot or a hyphen.

A wrap-up read of the branch also found wording the addressing rule
had made untrue. The docstrings and SKILL.md now say one post that
names an author answers all of that author's earlier review bodies and
conversation comments, the Answer docstring says answers are filtered
to answerers after they are built, and the check before posting says
an edit to a thread comment or review body is seen only when it turns
an answered item back to "no". The README and review-code name both
answerers, and the tests' oracle, comments and names describe the
mention rule rather than the removed thank-line check.
#262

A bot counted as addressed wherever its BOT_NAMES name appeared as a
\b word, which includes file paths. A closing remark such as "updated
.github/copilot-instructions.md" would have marked every earlier
Copilot review body answered, so a run would skip them silently.

The name must now stand on its own: not after a word character, a
dot, a slash or a hyphen, and not before a word character, a slash, a
hyphen or a dot that starts a file extension. "Thanks, Copilot."
still counts. The docstrings and SKILL.md say the same.
#262

Mutation testing of the mention rules left five boundary changes
uncaught. New cases pin each: "@dana_" still mentions dana, and a bot
name after a dot or a hyphen, before an underscore, or before a dot
and a digit ("settings.copilot", "pre-copilot", "copilot_instructions",
"Copilot.2") does not address the bot.
#262

One line of fetch_pr_comments.py's module docstring ran to 139
columns after the addressing wording was added. It is rewrapped with
no change to its words.
#262

The review of the draft PR found four ways the fetch could still mark
feedback answered when nobody had answered it.

A thread comment written inside a pending review carries the time it
was drafted, so a follow-up drafted before the author's reply but
published after it looked older than the reply. Thread comments are
now timed by publishedAt. A resolved thread hid any later follow-up,
and the run resolves every thread it replies to, so a resolution now
stands only when no reviewer posted or edited at or after the latest
answerer reply. A mention now has to be one GitHub links: not before
an underscore, and never inside inline code or a fenced block, for
people and bots alike. Reply bodies are indented on every line, so a
commenter cannot print a line that reads as the script's own answered
field.

Every match over a union now ends in assert_never, and new tests pin a
review submitted after it was started, a deleted account's reply in an
orphan thread, and a PR opened by a bot.
#262

The review of the draft PR found places where the skill left a run
stuck or unclear.

A failed fetch now stops the run, since an empty fetch would read as
nothing unanswered. A run with nothing to validate or post says so
and tells the author to close the tab, because an open tab keeps
monitor-prs from preparing the PR again. The check before posting
starts a new pass for new feedback or a line moving to or from "no",
not for one "yes" giving way to another. A skip the author overrules
stays overruled for the rest of the run, and a failed post lists every
item whose own reply did not go out, since a sibling reply naming the
same reviewer can make a later fetch mark it answered.

The report section is now Skipped, since it also lists the author's
own items, the rule text matches the script's docstring, proposing a
BOT_NAMES entry says the name decides what counts as answered, and the
description, README and review-code describe the rule as it is.
#262

The rule that a mention inside code addresses nobody covered only
single-backtick spans and triple-backtick fences. A double-backtick
span, a tilde fence, a four-backtick fence holding a three-backtick
block, an indented code block and a span that wraps a line all still
counted, though GitHub links none of them.

outside_code now follows CommonMark's code forms with the standard
library: fences of three or more backticks or tildes closed by a
matching fence, indented blocks after a blank line, and spans closed
by a backtick run of the same length within a paragraph. Blanked text
becomes whitespace, so text on either side of a span never joins. New
tests also pin that a resolution is measured from when the latest
answerer reply was posted, which mutation testing found unguarded.
#262

The hand-written code detection kept missing CommonMark's rules:
mutation testing left 21 of 30 changes to it uncaught, and it still
counted a mention in an indented code block after a fence or a
heading, while blanking a list item's continuation and text after an
escaped backtick.

outside_code now parses the body with markdown-it-py's CommonMark
preset, the same library and version range the repo's tests already
install, and keeps only plain text: fenced and indented code blocks
drop out and inline code becomes a space, so text on either side of
it never joins. The script declares the dependency inline, so uv
installs it on the first run. New cases pin each code form against
what CommonMark gives.
#262

A review body printed under OVERVIEW at a two-space indent, the same
as the script's own "answered" field, so a reviewer could write a line
into their review that reads as the item's verdict. Review bodies now
print at four spaces like every other body, and a test checks that no
printed body can pass for an answered or Answerers line.
@ooloth

ooloth commented Nov 17, 2026

Copy link
Copy Markdown
Owner Author

Automated review

An independent review of this PR, run before any fixes, given only the PR.

Review report (Request Changes, 19 findings)

Code Review

Verdict: Request Changes

Fix the three ways a run can still skip feedback nobody answered (findings 1 to 3). Those are the failure this PR exists to prevent, and the PR body itself argues that a wrong skip is the one nobody notices. Every agent finished.

Findings

# Severity Location Action
1 must tools/agents/config/skills/review-pr-comments-converge/scripts/fetch_pr_comments.py:394 Judge a thread comment by when it was published (publishedAt), not when it was drafted (createdAt)
2 must tools/agents/config/skills/review-pr-comments-converge/scripts/fetch_pr_comments.py:418 Stop a resolved thread from hiding a reviewer's later follow-up (challenges the recorded "resolved by anyone" rule)
3 must tools/agents/config/skills/review-pr-comments-converge/scripts/fetch_pr_comments.py:448 Stop a passing mention of "Rex" or "Copilot" from answering every earlier review by that bot
4 should tools/agents/config/skills/review-pr-comments-converge/scripts/fetch_pr_comments.py:444 Make @login matching agree with GitHub's mention rendering (underscore after the login, code spans), and fix the test that pins @dana_
5 should tools/agents/config/skills/review-pr-comments-converge/SKILL.md:218 Give a Nothing unanswered run an ending that tells the user to close the tab
6 should tools/agents/config/skills/review-pr-comments-converge/scripts/fetch_pr_comments.py:832 Indent every line of a reply body so its text can't print as a fake answered line
7 should tools/agents/config/skills/review-pr-comments-converge/scripts/test_fetch_pr_comments.py:163 Add a test where a feedback review's submit time is later than its created time
8 should README.md:73 Say that resolved threads and plain thread replies also count as answered
9 consider tools/agents/config/skills/review-pr-comments-converge/SKILL.md:169 Tell Phase 1 to stop on a failed fetch, so an empty saved fetch can't report Nothing unanswered
10 consider tools/agents/config/skills/review-pr-comments-converge/scripts/test_fetch_pr_comments.py:804 Add a fixed test for a deleted-account reply in an orphan thread, and make the orphan-header assertion independent of the label format, ahead of #293
11 consider tools/agents/config/skills/review-pr-comments-converge/SKILL.md:545 Start a new pass only on a change between no and anything else, and scope the "not detected" sentence to thread comments and review bodies
12 consider tools/agents/config/skills/review-pr-comments-converge/SKILL.md:215 Say how an overruled skip survives later passes, the round limit and approval
13 consider tools/agents/config/skills/review-pr-comments-converge/scripts/fetch_pr_comments.py:375 Add case _: assert_never(...) to the five new match statements that lack it
14 consider tools/agents/config/skills/review-pr-comments-converge/scripts/fetch_pr_comments.py:424 Decide whether an answerer editing their own earlier reply should reopen a thread
15 consider tools/agents/config/skills/review-pr-comments-converge/scripts/fetch_pr_comments.py:462 Report the items a batch's posts marked answered without a draft of their own
16 consider tools/agents/config/skills/review-pr-comments-converge/scripts/test_fetch_pr_comments.py Add tests for a PR opened by a bot
17 consider tools/agents/config/skills/review-pr-comments-converge/SKILL.md:3 Change the description's "validate each" to "validate each unanswered one"
18 consider tools/agents/config/skills/review-code/SKILL.md:99 Fix the small prose claims that are false or overstated
19 consider tools/agents/config/skills/review-pr-comments-converge/SKILL.md:83 Keep the full answered rule in one place, and rename the "Already answered" section, which also lists n/a (yours) items

1. fetch_pr_comments.py:394 — Judge a thread comment by when it was published (publishedAt), not when it was drafted (createdAt) (must)

parse_thread_comment passes "createdAt" as the posted time. For a comment written inside a pending review, createdAt is when it was drafted, not when it went public. Here is how it fails. A reviewer drafts a follow-up at 10:20, the author replies in the thread at 10:30, and the reviewer submits at 10:40. The follow-up's changed_at (10:20) is then before the reply's posted_at (10:30), so thread_verdict prints yes (reply …) and the follow-up is skipped. That breaks P2. Phase 4's check does list the new reply, but its line still reads yes, so the new pass skips it again. I confirmed GitHub's behaviour with a read-only GraphQL query on python/cpython#162022: comment 4575764853 has createdAt 21:44:13Z and publishedAt 21:45:43Z, which equals its review's submittedAt. I read the code at lines 390-394. No test uses publishedAt. [introduced]

2. fetch_pr_comments.py:418 — Stop a resolved thread from hiding a reviewer's later follow-up (must)

thread_verdict returns Answered(Resolved) before it looks at any comment time. Phase 4 resolves every thread it replies to (reply_to_comment.py --resolve, SKILL.md:518). So a reviewer who answers the run's ✅ with "this still isn't fixed" gets yes (resolved by …) on every later run and is never validated again. monitor-prs already ignores resolved threads, so nothing else would surface it. Agent 7 reproduced this by calling judge_discussion directly. I read lines 412-432 and SKILL.md:518, which match. This depends on GitHub leaving a thread resolved when someone replies in it. I believe it does, but I have not verified that. This challenges the ticket's recorded rule ("resolved, or …", "whoever resolved it"), which conflicts with its own P2 here. A fix within the rule's spirit: count Resolved only when no non-answerer comment changed at or after the latest answerer comment, and otherwise fall through to the reply rule. [introduced]

3. fetch_pr_comments.py:448 — Stop a passing mention of "Rex" or "Copilot" from answering every earlier review by that bot (must)

For a bot, addresses() accepts the BOT_NAMES name anywhere it stands as a word. Both names are also product names. I ran the head's addresses() on these posts, and each returned True: "Tested in a Rex tab" (Rex), "Fixed the crash in Rex" (Rex), "Disabled GitHub Copilot suggestions" (Copilot). One addressing post answers every earlier review body and conversation comment from that author, so one such remark silently skips all of them. That is the michaeluloth.com#83 failure the approach change was meant to remove, and this author's own workflow talks about Rex tabs constantly. The recorded approach change accepted only the opposite cost, a duplicate draft when a post names nobody. Options: require the thank form (Thanks for the feedback, Rex:) or the name at the start of a line for bots, or record this as an accepted cost. Whichever you choose, the "propose a BOT_NAMES entry" guidance (SKILL.md:392-394) and the BOT_NAMES comment (fetch_pr_comments.py:147-149) should say that the name now also decides what counts as answered. A deferred claude bot named "Claude" would match almost anything in this repo. [introduced]

4. fetch_pr_comments.py:444 — Make @login matching agree with GitHub's mention rendering, and fix the test that pins @dana_

The docstrings and SKILL.md:92 say the rule is "@login where GitHub would render it as a mention". Two cases break that promise:

  • Underscore after the login: the lookahead (?![A-Za-z0-9-]) allows _, so the regex matches @ooloth_.
  • Code: the regex matches inside code spans and fenced code.

I checked both. The head's addresses() returned True for cc @dana_, for `@dana` and for a fenced @dana. gh api markdown renders @ooloth_ and `@ooloth` as plain text and links only a bare @ooloth. Both cases lean the unsafe way, toward a skip. The test at test_fetch_pr_comments.py:1616-1617 asserts that @dana_ mentions dana, with the comment "GitHub logins hold no underscore". Both halves of that comment are false: GitHub does not render it, and enterprise-managed logins end in _shortcode. That test needs flipping. [introduced]

5. SKILL.md:218 — Give a Nothing unanswered run an ending that tells the user to close the tab

"When nothing is left" sends the run to Phase 3, which says to stop and wait for approval to post, though nothing will be posted. The only close-the-tab instruction (SKILL.md:582) runs "once every reply is posted", which never happens here. Under decision 010, an open tab is the record that a PR is prepared. So this tab blocks monitor-prs from preparing the PR's next feedback, and nothing says why. I read SKILL.md:218-221, :363-365 and :582. The likely trigger is monitor-prs's own "changes requested, not re-requested" rule, a deferred mismatch the ticket records, which will now keep opening Nothing unanswered sessions. [introduced]

6. fetch_pr_comments.py:832 — Indent every line of a reply body so its text can't print as a fake answered line

REPLIES prints f" {replier}: {reply['body'].strip()}" (lines 832 and 874). Only the first line of a multi-line body is indented, so later lines can start at column 2 and look like the script's own answered : yes … field. Phase 2 now says to "go by the line alone", so on a public repo any commenter could add a line that makes the run skip someone's real feedback. Agent 6 reproduced a forged block. I confirmed the unindented print at both lines, while other bodies go through indent(). The rendering is [pre-existing], but this PR is what makes a printed answered line decide a skip. Print it as indent(body, " "), like BODY:.

7. test_fetch_pr_comments.py:163 — Add a test where a feedback review's submit time is later than its created time

The build-time design change counts a review's submit time in changed_at, but no test would notice if that were dropped. Agent 8 removed submittedAt from changed_at in a scratch worktree and all 151 tests still passed. I confirmed only the setup: the review() helper sets createdAt and submittedAt to the same at, and the one review whose times differ (line 1483) is on the answering side, where changed_at is not read. The mutation result is Agent 8's. [introduced]

8. README.md:73 — Say that resolved threads and plain thread replies also count as answered

The README says feedback is skipped when "answered by a reply addressed to its reviewer". An inline thread is also skipped when anyone resolves it, or when an answerer's reply is its newest comment, and neither needs to name the reviewer (thread_verdict). The live fetch of michaeluloth.com#83 prints yes (resolved by Copilot), according to Agent 18. I read README.md:70-76. [introduced]

9. SKILL.md:169 — Tell Phase 1 to stop on a failed fetch, so an empty saved fetch can't report Nothing unanswered

Phase 1 says to run the fetch and save its output, but not what to do when it exits 1. A run that saves empty stdout now gets a clean-looking Nothing unanswered, where before it simply had nothing to validate. One sentence fixes it: "On a non-zero exit, stop and show the error; an empty fetch is not 'nothing unanswered'." I read SKILL.md:165-176. pre-existing gap made relevant: the missing instruction is [pre-existing], and the outcome that hides the failure is [introduced].

10. test_fetch_pr_comments.py:804 — Add a fixed test for a deleted-account reply in an orphan thread, and make the orphan-header assertion independent of the label format, ahead of #293

Open PR #293 conflicts with this one in both shared files. Agent 16 merged the two heads in a scratch worktree:

The merge results are Agent 16's. I did not rerun them. [introduced]

11. SKILL.md:545 — Start a new pass only on a change between no and anything else, and scope the "not detected" sentence

The check lists "every item whose line changed". That includes yes (reply …) becoming yes (resolved by …) when the author resolves an answered thread by hand. Each listed change means posting nothing, re-approving every draft and using one of the two allowed passes. Lines 549-550 also say "An edit to an item already marked no is not detected". Two sentences earlier, the check lists "every conversation comment whose edit time changed", which does detect an edit to a no conversation comment. I read SKILL.md:541-566. [introduced]

12. SKILL.md:215 — Say how an overruled skip survives later passes, the round limit and approval

"Take it off the Already answered list and validate it in a new round" conflicts with three other rules:

  • Later passes: the rule that Already answered entries are rebuilt from the fetch each pass. The item's line still reads yes, so the next pass skips it again.
  • Round limit: "Max 3 rounds in each pass … always".
  • Approval: the overruled item's new draft is never said to need approval.

The report should say which rule wins in each case. I read SKILL.md:200-216 and :589. [introduced]

13. fetch_pr_comments.py:375 — Add case _: assert_never(...) to the five new match statements that lack it

~/.agents/standards/python.md:30 says the default case of every match over a union calls assert_never. The new matches in parse_post (375), judge_discussion (497, 508, 516) and main (742) don't. The PR's other new matches (addresses, answered_line, describe_evidence) do. I confirmed the standard line and the addresses match at line 450. The other line numbers are Agent 10's. [introduced]

14. fetch_pr_comments.py:424 — Decide whether an answerer editing their own earlier reply should reopen a thread

others includes the answerers' own earlier replies. So the author fixing a typo in an old reply, at or after their latest reply, turns the thread back to no and gets a duplicate draft. test_an_earlier_answerer_reply_edited_at_or_after_the_latest_reopens_the_thread pins this, and it follows the design's wording ("every other posted comment"). It fails in the safe direction. Narrowing others to non-answerer posts is a one-line change if you want it. I read line 424. [introduced]

15. fetch_pr_comments.py:462 — Report the items a batch's posts marked answered without a draft of their own

One post naming a reviewer answers all of that reviewer's earlier items. So if Phase 4 posts the reply to alice's review body and the post for her conversation comment then fails, the rerun prints yes for the conversation comment, and it is never answered. Agent 7 reproduced this with judge_discussion. The recorded cost covers hand-written answers, not partial failures. A post-posting fetch that lists items newly yes with no draft of their own would show them. [introduced]

16. test_fetch_pr_comments.py — Add tests for a PR opened by a bot

Answerers.includes has a branch that compares a Bot or UnrecognisedActor against the PR author (line 240). No test sets pr_author to a bot, so P3 is unguarded for Dependabot-style PRs. This is the agent's grep. I did not repeat it. [introduced]

17. SKILL.md:3 — Change the description's "validate each" to "validate each unanswered one"

The frontmatter still says the skill fetches comments and validates "each against the actual code", but it now skips answered items. The description is what skill listings show. I read line 3. [pre-existing] line, made inaccurate by this change.

18. review-code/SKILL.md:99 — Fix the small prose claims that are false or overstated

All of these are [introduced]. I confirmed each of these lines exists. The rest of the evidence is Agent 18's:

  • review-code/SKILL.md:99: "the reviewer's own items print n/a (yours)" is not true of a thread that a third party joined.
  • fetch_pr_comments.py:49-50: "no item could be judged" should say that not every item could be.
  • test_fetch_pr_comments.py:2080: "Hypothesis draws early elements more often" is false according to Agent 18's measurement. Hypothesis shrinks toward early elements; it doesn't draw them more often. The comment at :2129 contradicts it.
  • test_fetch_pr_comments.py:1748: "every review body was skipped" overstates it, because a Copilot review came after the closing remark. Both "Seen on/at" lines narrate an incident.
  • test_fetch_pr_comments.py:1541: "resolvedBy can outlive an unresolve" is unsourced.

19. SKILL.md:83 — Keep the full answered rule in one place, and rename the "Already answered" section

The rule is written out almost word for word in the script docstring (fetch_pr_comments.py:28-54), SKILL.md:83-99 and the README, and the copies already differ slightly. For example, the docstring has the same-second tie and SKILL.md does not. SKILL.md could say only what to do with yes, no and n/a, and point to the docstring for the rule. The ⏭️ Already answered section also lists n/a (yours) items, which were never answered, so "Skipped" would read truer. Agents 3 and 4 both raised these points. I compared the two texts. [introduced]

Agents

  • Agent 1 (Intent Alignment): Opus, because judging whether the code does what the intent says takes reasoning, and a gap it misses no later step catches.
  • Agent 2 (Approach): Opus, because weighing one approach against others takes reasoning.
  • Agent 3 (Structure): Sonnet, because it checks the diff against the structure and readability standards it loads.
  • Agent 4 (Public Surface): Sonnet, because it checks the diff against the interface and documentation standards it loads.
  • Agent 5 (Correctness): Opus, because a correctness bug it misses ships silently.
  • Agent 6 (Data Safety): Opus, because a security, privacy or data-loss bug it misses ships silently.
  • Agent 7 (Runtime Behaviour): Opus, because a failure under load or concurrency that it misses shows up only in production.
  • Agent 8 (Testing): Sonnet, because it checks the tests against the testing standard and runs the breaks it proposes.
  • Agent 9 (Release Readiness): Sonnet, because it checks the diff against the dependency, deployment and config standards it loads.
  • Agent 10 (Language): Sonnet, because it checks the diff against the language standards it loads.
  • Agent 11 (Failure Paths): Opus, because tracing whether a failure path is reachable takes reasoning.
  • Agent 12 (Out-of-Diff References): Opus, because its answer is often that nothing else references what the change removes, a miss nobody checks.
  • Agent 13 (Coupled Changes): Opus, because its answer is often that nothing outside the diff has to change, a miss nobody checks.
  • Agent 14 (Rule Consistency): Opus, because its answer is often that no two rules conflict, a miss nobody checks.
  • Agent 15 (Base Merge): not applicable. The Base moved line says none of the 9 base commits touches this PR's files.
  • Agent 16 (Overlapping PR Merge, Number each review's PR comments within that review in fetch_pr_comments.py #293): Opus, because a conflict resolved the natural way can drop this PR's edit and leave every test passing.
  • Agent 17 (Dropped Assertions): not applicable. The diff removes no test, assertion or expected-value helper, only fixture builders that were rewritten.
  • Agent 18 (Doc Claims): Opus, because a wrong "true" sends a reader to act on a false claim with nothing to warn them.
  • Agent 19 (Removed Knowledge): Opus, because what a deletion takes leaves no trace at the head. It found nothing lost.

Looks Good

Judging every item before the first print is a good choice: an unreadable time or state exits 1 with every reason and no half output. The fake gh now returns only the fields the query selects, so a field read without being requested fails the suite.

These areas came back clean:

  • Checks: the full scripts/check.bash passes, with 1076 tests, 151 of them in this file.
  • P6 and P7: existing output lines and the request count are unchanged.
  • Data safety: no secrets or private values beyond finding 6.
  • Removed prose: nothing a reader needs was lost.
  • Conflicts: both one-sided resolutions of the Number each review's PR comments within that review in fetch_pr_comments.py #293 conflicts fail loudly.

Dropped

  • No timeout on gh subprocess calls (Agent 7): pre-existing at the base, and this change does not make it matter more.
  • A gh login that is a GitHub App or bot is never an answerer (Agents 7, 10): errs toward duplicate drafts, and nothing in this repo runs the fetch under an app token.
  • The fetch exiting 1 on unreadable data also stops review-code (Agents 9, 11): this is the recorded P8 trade-off, and GitHub's schema makes it unreachable in practice.
  • A session mid-run at merge time reads a saved fetch with no answered lines and skips everything (Agent 9): speculative. A session that loaded the old SKILL.md ignores the lines anyway.
  • Redundant postcondition asserts (Agent 2): other agents judged them to be the self-checks the standards ask for. Nothing to act on.
  • Splitting the PR into thread-only and top-level slices (Agent 2): the agent itself judged it not worth doing now.
  • Type and order nits (Agents 3, 10): PostId keys, Unanswered using pass, definition order, judge_discussion length and file size. Readability-only, below the bar.
  • The property-test oracle copies the regex (Agent 2): Agent 8 noted that the hand-written NEAR_MISSES and example tables still pin the regex.

(Stopped before Step 5. This was a read-only, unattended run, so nothing was posted to GitHub, and the question about which findings to fix locally was not asked.)

What fixed each finding

#262

Replies were split with str.splitlines, which also breaks at U+2028,
U+2029, U+0085, \v and \f, so a reply holding one of them printed as
two lines where main printed one. Replies now split only at \r\n, \n
and \r, so a Windows line ending still prints no carriage return and
every other character prints as written. Tests pin both.
#262

A reply printed U+2028, U+2029, U+0085, \v and \f raw, so a reader that
splits on them, as str.splitlines does, could find a line the reply
started, such as a forged answered field. Replies now print each as a
visible escape, while \r\n, \n and \r still split.
…olution

#262

A resolved inline thread counted as answered whoever resolved it. A
thread Copilot resolved itself, with no reply from the author, was
skipped, and nobody ever answered it: the silent skip the change
exists to prevent. Counting an unanswered thread as answered drops
feedback with nothing to show it, while the opposite mistake only
drafts a reply the author reads and can decline.

thread_verdict now counts a resolution only when the one person,
other than the answerers, who commented in the thread resolved it,
and nothing they wrote came after the latest answerer reply. A bot, a
deleted account, someone else or an answerer resolving it leaves the
thread judged as if unresolved, so an answerer's newest reply still
answers it. Resolved.by narrows to Person. A property test checks
that any other resolver gives the same verdict as no resolver.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A second run of review-pr-comments-converge drafts no reply to feedback already answered

1 participant