You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
A second run of review-pr-comments-converge drafts no reply to feedback already answered #262
Each run of review-pr-comments-converge re-validates every comment on the PR and drafts a reply to each one, including feedback an earlier run already answered, so a PR that gets new feedback after a first round can receive duplicate replies. The list of decided comments lasts only one run. Nothing in Phase 1 or Step A skips a resolved thread or one whose last reply is the author's, and monitor-prs starts a fresh run whenever new feedback arrives.
Ideal state
A run drafts no reply to an inline thread, review body or conversation comment that the PR author already answered
The report lists any feedback it skipped as already answered
When monitor-prs starts a second run of review-pr-comments-converge on a PR, the run drafts replies only to feedback nobody has answered yet. Its report lists everything it skipped, with the comment or resolution that counted as the answer, so a wrong skip is easy to spot and overrule. fetch_pr_comments.py works out the verdict from the PR's current state, in the same single request it already makes, and prints one answered line on every review body, inline thread and conversation comment: yes with its evidence, no, or n/a for the author's own. SKILL.md puts every item not marked no straight onto the decided list, so it never reaches validation, and reads the lines again in the extra pass that starts when the PR moves before posting. A comment from a deleted account, which crashes the fetch today, prints as a deleted account and never counts as an answer. "Answered" means answered by the PR author or the account the run posts as:
Inline thread: resolved, or its newest posted comment is an answerer's and no other comment in it was edited at or after that time.
Review body or conversation comment: an answerer posted a top-level comment, or a review with a non-empty body, after the feedback was submitted or last edited. Empty-body reviews, which GitHub creates for inline replies, and unsubmitted comments are not answers.
What a good fix must achieve 🎯
Property
Category
P1
A run drafts no reply to an inline thread, review body or conversation comment that is already answered, as defined above
🛡️ Safety
P2
Feedback that arrived after the answer is never skipped: a reviewer's follow-up, a newer review, or feedback edited after the answer, with a same-second tie counting as new
🛡️ Safety
P3
The PR author's own comments and reviews are never treated as feedback
🛡️ Safety
P4
The report lists every skipped item with the evidence that counted as its answer
🛡️ Safety
P5
Whether an item is answered is read from the PR's state on every fetch, including the one before posting, and never from a stored record
🛡️ Safety
P6
review-code, which reads the same fetch output, still sees every item it sees today
🙂 Experience
P7
The fetch makes no more GitHub requests than it does today
⚡ Performance
Assumptions 🤔
A later top-level comment or non-empty review from the author answers every earlier review body and conversation comment, because GitHub records no link from an answer to the review body or comment it answers. The run itself answers each review body with one conversation comment, and monitor-prs decides "has feedback" by the same time order. If authors often post unrelated comments after feedback (such as "rebased"), an option that matches the answer's text would beat time order.
Constraints 🧱
An inline reply makes GitHub create an empty-body review by the replier, so "the author's newest review" alone would mark review bodies answered that were never answered.
review-code calls the same fetch script to reconcile earlier reviews, so the default output keeps every item.
🏆 The fetch prints an answered verdict per item; SKILL.md skips and lists those items (chosen)
✅
✅
✅
✅
✅
✅
✅
❌ one function, one output line per item, one report section
The fetch prints the raw facts (PR author, viewer, edit times); SKILL.md states the rule
✅
✅
✅
✅
✅
✅
✅
❌ five new output fields and a rule in prose that each run applies by hand, with no test
SKILL.md alone states the rule; the fetch is unchanged
❌
❌
✅
✅
✅
✅
✅
✅
The fetch hides answered items
✅
✅
✅
✅
✅
❌
✅
❌ a second output mode
Count only replies that link to what they answer
❌
✅
✅
✅
✅
✅
✅
❌ a link format every reply must carry
Keep a list of answered items across runs
✅
❌
✅
✅
❌
✅
✅
❌ a store
Decided by 🧭
Instructions alone cannot work: today's output does not show the PR author, edit times, or which lines of a multi-line reply belong to whom, so a run cannot tell who answered last or whether feedback changed afterwards. Hiding answered items breaks review-code, a reply-link format misses every answer written by hand, and a stored list goes stale, which decision 010 rules out. That leaves two options that change the fetch the same way, and they differ in where the rule lives. Printing a verdict keeps the rule in one function that tests cover, and the report still shows its evidence for each skip. Printing raw facts leaves the same rule in prose, applied by hand across fetch output that runs to 1,200 lines on a busy PR, where a missed comparison would go unnoticed. Both are easy to reverse, since nothing is stored.
Deferred ⏳
monitor-prs keeps its own prose definition of unanswered feedback. One definition both skills use would stop them drifting apart.
An escalation the run posts (options left for the author) ends the thread with the author's comment, so the next run skips it and only the skipped list shows it. Nothing marks an escalation as still open.
A bot that posts on PRs but is missing from BOT_NAMES (such as claude) prints as UNKNOWN.
fetch_discussion already gets every page of threads, reviews and conversation comments. The same request now also returns the viewer's login, the PR author, who resolved each thread, and each item's state and created and edited times. Before anything prints, a judging step parses only what the verdict needs into typed values: who wrote each post, as an answerer or not, and when it was posted and last edited, as real datetimes. It drops unsubmitted posts, gives every review body, inline thread and conversation comment exactly one Verdict, and stops the run with a sentence if any input cannot be read. The printer is unchanged apart from a # Answerers: line under # Source: and one answered line per item, so review-code reads the same output it reads today. In SKILL.md, every item not marked no goes onto the decided list before validation and into a new report section, ⏭️ Already answered, with its evidence. The author can name any of those items to have it validated anyway. When nothing is left, no validation runs and the Outcome is Nothing unanswered. The check before posting compares the fresh fetch's answered lines with the saved fetch's, so a skip is never carried over. Types guarantee that a deleted account never answers, that an unsubmitted post never answers, and that every item gets one of three verdicts. Assertions check that each yes points at an answerer's post made after the feedback. Tests pin the time rules, ties included, and fresh-agent replays check the skill's handling.
The approach's properties P1 to P7 still apply, plus one this design adds:
Property
Category
P8
A fetch that cannot judge every item prints nothing and exits with a sentence naming the item and field
🛡️ Safety
Types 🧬
Progressions considered
Option
P1
P2
P3
P4
P5
P6
P7
P8
Leaves behind
🏆 Parse only what the verdict reads, judge every item before printing, and print from today's dicts (chosen)
✅
✅
✅
✅
✅
✅
✅
✅
❌ Answerers, Post, Answer, the Verdict and Evidence unions, one judging function and two verdict functions
Parse the whole discussion into typed threads, reviews and comments, and rewrite the printer over them
✅
✅
✅
✅
✅
✅
✅
✅
❌ about six more types and a rewrite of every print line review-code reads
Verdict functions over the raw dicts, comparing ISO strings while printing
✅
❌
✅
✅
✅
✅
✅
❌
❌ comparisons on author['login'] that crash on a deleted account
Decided by: comparing timestamps as strings orders 06:46:39.5Z before 06:46:39Z, so one format change would skip feedback edited after the answer, with nothing reported. Judging while printing would leave half an output when a value fails to parse. The full re-parse meets every property, but it adds types no property needs and rewrites the output review-code depends on.
Progression
GraphQL page(s): data.viewer, pullRequest { author, reviewThreads, reviews, comments }
↓ fetch_discussion() viewer and PR author read from the first page
Discussion { threads, reviews, conversation (raw dicts, unchanged), answerers: Answerers }
Answerers { pr_author: Reviewer, viewer: str } .includes(Reviewer) -> bool, Ghost never
↓ judge_discussion() runs before the first print
parse_post(node) -> Post { id, author: Reviewer, posted_at: datetime, changed_at: datetime }
only posted nodes: thread comments in state SUBMITTED, reviews not PENDING, every conversation comment
answers: tuple[Answer, ...] sorted by time
Answer { kind: "conversation comment" | "review", post: Post }
every answerer's conversation comment, and every answerer's review with a non-blank body
thread_verdict(posts, resolved_by, answerers) -> Verdict
item_verdict(feedback: Post, answers, answerers) -> Verdict
Verdicts { threads, reviews, conversation: dict[id, Verdict] }
Verdict = Answered(evidence: Evidence) | Unanswered() | NotFeedback()
Evidence = Resolved(by: Reviewer) | ThreadReply(post: Post) | TopLevelReply(answer: Answer)
↓ print_discussion(target, discussion, verdicts)
stdout: "# Answerers: …" under "# Source:", and " answered : …" on each review body, thread and conversation comment
Answerers.includes matches on Reviewer variants, never on the printed login.Ghost is never an answerer, even when the PR author's account was deleted. A real account named ghost cannot be confused with a deleted one. (P1, P3)
Post carries times as timezone-aware datetimes. A string or naive comparison cannot reach the rules. changed_at is the later of created and last edited. (P2)
Only posted nodes become Posts, and only non-blank reviews become Answers. A pending reply or review, and the empty review GitHub creates for an inline reply, can never answer anything. (P1)
The thread rule.
NotFeedback: every posted comment is an answerer's.
Answered(Resolved(by)): the thread is resolved.
Answered(ThreadReply): some posted comment is an answerer's, the latest such comment is A, and every other posted comment's changed_at is strictly before A's posted_at.
Unanswered: anything else.
A reviewer's reply in a thread the author started is feedback, and an edit in the same second as the answer reopens the thread. (P1, P2, P3)
The review body and conversation comment rule.
NotFeedback: the author is an answerer.
Unanswered: a submitted review has no submit time, so the run drafts a reply rather than skipping.
Answered(TopLevelReply): the earliest Answer posted strictly after the feedback's changed_at, which ignores answers in the same second.
Unanswered: no such answer exists. (P1, P2)
Verdict is a closed union printed by an exhaustive match. Each item prints exactly one line, and Answered cannot exist without evidence. (P1, P4)
Judging reads only this fetch and writes nothing. (P5)
The query gains fields only. The request count is unchanged, and every existing output line is still printed in its place. (P6, P7)
What a failed parse keeps. It exits naming the item's id and the field, before printing. Re-running the fetch replays it. (P8)
The crash sites. The REPLIES lines and the orphan thread header print a deleted account as ghost, the same as the review and conversation headers do today. Thread-comment authors gain __typename so they parse as Reviewer. (P6)
Output lines
# Answerers: ooloth (PR author and gh login), or alice (PR author), ooloth (gh login), or ooloth (gh login); the PR author's account was deleted.
answered : yes (resolved by ooloth)
answered : yes (reply <id> by ooloth at <time>)
answered : yes (conversation comment <id> by ooloth at <time>)
answered : yes (review <id> by ooloth at <time>)
answered : no
answered : n/a (yours)
A thread's line follows its outdated line. A review's line follows refer_as, and only a review with a body gets one. A conversation comment's line follows edited.
Assertions 🛡️
The types cannot enforce time order or that the evidence came from an answerer, because both depend on values.
Boundary validation (exit 1 with a sentence, before printing):
Missing viewer login: the response has no viewer login, so the run cannot tell its own replies apart.
Unreadable timestamp: a timestamp does not parse, or has no timezone.
Unknown thread-comment state: the state is neither PENDING nor SUBMITTED, so whether the comment was posted is unknown. Treating it either way could hide a reviewer's comment.
Why boundary validation: all three come from GitHub, not from this code.
Assertions (only a bug here can break them), one condition each:
ThreadReply evidence: the reply's author is an answerer, and its posted_at is after every other posted comment's changed_at.
TopLevelReply evidence: the answer's author is an answerer, and its posted_at is after the feedback's changed_at.
NotFeedback: the item's author, or every posted comment's author for a thread, is an answerer.
Printing: every printed thread, review with a body and conversation comment has a verdict.
No recorded-signal check. Nothing spans runs, because nothing is stored.
Telemetry 📡
Which accounts counted as answerers? A wrong viewer or PR author would change every verdict with no visible cause. Record: the # Answerers: line.
Why was an item skipped? Record: the evidence in its answered line, which the report copies.
Tests 🧪
Neither the types nor the assertions say which items count as answered, which is the meaning of the rule, and nothing but a replay shows that the skill acts on the lines. So both need tests.
Expectation
Property
Technique
A reply from a deleted account prints as ghost instead of crashing, and never answers
P1, P6
Example
An orphan thread opened by a deleted account prints
P6
Example
The # Answerers: line names the PR author and the gh login [same account, different accounts, deleted PR author]
P4
Parametrized example
A resolved thread is yes (resolved by …)
P1, P4
Example
A thread whose newest comment is the PR author's is yes (reply …)
P1, P4
Example
A thread whose newest comment is the viewer's, when the viewer is not the PR author, is answered
P1
Example
An orphan thread gets a verdict by the same rule
P1
Example
A reviewer's follow-up after the author's reply makes the thread no
P2
Example
A thread comment posted or edited in the same second as the reply, or later, makes the thread no
P2
Parametrized example
An edit made before the reply leaves the thread answered
P1
Example
A pending reply from the author does not answer the thread
P1
Example
A thread only answerers wrote in is n/a (yours)
P3
Example
A reviewer's reply in a thread the author started is no
P2, P3
Example
A deleted account never answers, even when the PR author's account was deleted
P1
Example
A review body followed by an answerer's conversation comment is yes, naming that comment
P1, P4
Example
A review body followed by an answerer's review with a body is yes, naming that review
P1, P4
Example
A conversation comment followed by an answerer's conversation comment is yes
P1
Example
The empty review an inline reply creates does not answer a review body
P1
Example
A pending review from the author does not answer
P1
Example
A later comment from someone other than an answerer does not answer
P1
Example
Feedback posted or edited in the same second as the answer, or edited after it, is no [review body, conversation comment]
P2
Parametrized example
A bot comment edited into a full review after the answer is no
P2
Example
A newer review after the answer is no
P2
Example
The PR author's own review and conversation comment are n/a (yours)
P3
Example
Every review body, thread and conversation comment prints exactly one answered line
P4
Example
An answered item is still printed in full
P6
Example
The same PR fetched before and after a reviewer's follow-up prints yes and then no
P5
Example
An answer on a later page is seen, with one request per page
P1, P7
Example
The existing line-order and one-call tests still pass
P6, P7
Example
A missing viewer, an unreadable timestamp or an unknown comment state exits 1 with a sentence and prints nothing
P8
Parametrized example
Any thread with a non-answerer's post or edit at or after the answerer's latest reply is no
P2
Property
Any thread whose answerer reply is strictly after all other activity is yes (reply …)
P1, P4
Property
Any review body or conversation comment is yes exactly when an answerer's answer was posted strictly after its last change
P1, P2
Property
With the edited SKILL.md and a fetch holding no, yes and n/a items, validation receives only the no items, and the report lists the rest with their evidence
P1, P3, P4
Replay
When every item is yes or n/a, no validation runs and the Outcome is Nothing unanswered
P1, P4
Replay
In the check before posting, an item whose line changed from no to yes has its draft dropped and is listed as skipped
P1, P5
Replay
In the check before posting, an item whose line changed from yes to no is validated in the new pass
P2, P5
Replay
Example: each rule has a named edge, such as the tie, the empty review, the pending post or the deleted account, and a failing test should say which edge broke.
Parametrized example: the same claim holds across a few values of one dimension, such as same second against later.
Property: a wrongly skipped item is the one failure nobody would notice, so generated timelines, shaped toward same-second times and edits, check the rule against the approach's wording rather than against the code.
Replay: what changes in SKILL.md is what an agent does, which only a fresh agent given the edited skill and a saved fetch can show.
Deferred ⏳
fetch_pr_comments.py's test fake builds only pullRequest's three lists, so a new field elsewhere in the response reaches the script as missing unless the fake is extended. This work extends it for the viewer and the PR author. Making the fake serve a whole response from the fixture would stop the next field being missed.
Three small changes to the design comment's Types and Assertions, made while implementing. None changes the approach.
changed_at also counts a review's submit time. The design defined it as the later of created and last edited. A review is created when it is started and submitted later, so without the submit time a review could be judged as changed before it was posted. Counting it can only turn a yes into a no, never skip feedback.
A submitted review with no submit time is treated like an unsubmitted post. It gets the verdict the design already gives it: n/a when an answerer wrote it, otherwise no. It never answers anything.
The NotFeedback assertions run once over the finished verdicts in the judging step, instead of inside each verdict function, where they would only repeat the condition that chose the verdict.
Design change during the build: items decided in an earlier pass
The design said the check before posting compares the fresh fetch's answered lines with the saved fetch's, so a skip is never carried over. It did not say what happens, in the new pass that check starts, to items an earlier pass already fixed, dismissed or escalated. Fresh agents replaying the skill disagreed about it, so SKILL.md now says:
Not listed by the check: an item an earlier pass decided stays decided and is not validated again. Its draft goes into the new report and needs fresh approval. Validating it again would find the pushed fix and turn an approved ✅ reply into a ❌ "already handled" one.
Listed by the check, line still no: an item with a new reply or a changed edit time comes off the decided list, its old draft is dropped, and the new pass validates it with the new feedback. This is the case the extra pass exists for.
Listed by the check, line now yes or n/a: unchanged from the design. The draft is dropped and the item is skipped.
Nothing unanswered is now the Outcome only when no draft carries over from an earlier pass.
Escalation: an unrelated comment from the author skips every earlier review body
The build is finished up to the proof and pushed to the branch 262-skip-feedback-already-answered (4 commits, all checks green, every mutant caught, every replay passing). No PR is open yet, because the proof found the case the approach's Assumptions section said would reopen the time-order rule.
What the proof found
The approach assumed that a later top-level comment or non-empty review from the author answers every earlier review body and conversation comment, and said that if authors often post unrelated comments after feedback, matching the answer to the item would beat time order.
A read-only fetch of ooloth/michaeluloth.com#83 with the new script shows that case on a real PR. The author's closing remark, "This one somehow allowed scope to creep beyond its original goal.", is the evidence for 9 answered : yes lines on review bodies and conversation comments posted before it. One of them is a claude review whose numbered "Minor observations" nobody answered. A second run would skip all 9 without validating them, and the report would list them as already answered.
Every fresh agent that replayed the edited skill raised the same thing on the test fixture: a reply addressed to one reviewer ("Thanks, Rex: …") marked a different reviewer's review body as answered.
The rest of the proof held on 18 real PRs: old output is unchanged apart from the added lines, the request count is identical, newer feedback and same-second ties print no, and broken input exits 1 with nothing printed.
Decision needed
Which posts count as an answer to a review body or conversation comment? Inline threads are not affected: GitHub links each reply to its thread.
Keep time order, as planned. Ship as it is. A wrong skip shows in the ⏭️ Already answered list with its evidence, and the author can name the item to have it validated. On a PR like michaeluloth.com#83, one remark skips every earlier review body, and the author has to spot that in the list.
Count a top-level post only when it addresses that item's author. The post must open with the item's thank line or name its author the way refer_as does. The run's own replies always open with the thank line, so the answers it posts still count. A hand-written answer that names nobody gives no, and the run drafts a reply the author can decline before anything is posted. A remark like the one on perf: Add command caching to UV feature utilities #83 answers nothing. This adds one rule to the verdict function, with tests and replays, and amends the approach, which rejected linking an answer to its item because hand-written answers would be missed.
Skip only inline threads. Never skip a review body or conversation comment, so every one is validated each run. This removes the false skip completely, but a second run drafts a reply again to every review body it has already answered, which is the duplicate this ticket exists to stop for those two kinds.
Recommendation: 2. A wrongly skipped item is the failure nobody notices, while an extra draft is caught when the author reads the drafts before approving them. Option 2 removes the first and turns the hand-written case into the second.
Approach change: an answer must address the item's author
Option 2 from the escalation is chosen. This changes the approach's definition of an answered review body or conversation comment, and the Assumptions entry it rested on.
Before: an answerer posted a top-level comment, or a review with a non-empty body, after the feedback was submitted or last edited.
Now: the same, and the post must also address that item's author: it opens with the item's thank line, or names its author the way refer_as does (@login for a person, the BOT_NAMES name for a bot). The evidence is still the earliest such post after the feedback's last change.
Why: on Add project overview, integrations, and dev commands to CLAUDE.md michaeluloth.com#83 an unrelated closing remark from the author marked 9 earlier review bodies and comments as answered. A post that names nobody now answers nothing, so the run drafts a reply the author can decline before it is posted, instead of skipping feedback silently.
What stays the same: inline threads, where GitHub links each reply to its thread; the tie rule; pending posts and empty reviews never answering; and properties P1 to P8, with P1's "answered" now meaning this narrower rule.
Cost: a hand-written answer that names nobody gets a duplicate draft. One post that addresses a reviewer answers every earlier review body and conversation comment from that reviewer.
The build resumes from the red step for this rule.
The independent review of #297 found ways a run could still skip feedback nobody answered. These change the approach's thread rule and parts of the skill's handling.
A resolved thread no longer hides a later follow-up. The approach counted a thread as answered when it was resolved, whoever resolved it. The run resolves every thread it replies to, and GitHub leaves a thread resolved when someone replies in it, so a reviewer's "this still isn't fixed" would have been skipped on every later run. That breaks P2. A resolved thread now counts as answered only when no reviewer's comment in it changed at or after an answerer's latest reply in it. Otherwise it is judged by the reply rule.
Thread comments are timed by when they were published, not drafted. A comment written inside a pending review has an earlier createdAt than the moment it went public, so a follow-up drafted before the author's reply but submitted after it would have been skipped.
@login counts only where GitHub links it, so not before an underscore and not inside code.
Reply bodies are fully indented, so a commenter cannot write a line that looks like the script's own answered line.
Skill handling:
A new pass starts only when an item moves between no and something else.
An item the author asks to have validated stays off the skipped list for the rest of the run.
Posting reports any item a sibling post marked answered while its own post failed.
A failed fetch stops the run, and a Nothing unanswered run tells the author to close the tab.
The report section is now called ⏭️ Skipped, since it also lists the author's own items.
One finding needs a decision and is raised separately: a bot's name used in passing ("tested in a Rex tab", "GitHub Copilot") counts as addressing that bot.
Escalation: a bot's name used in passing answers all its earlier reviews
The build is done apart from this one point. Draft PR #297 has every other review finding fixed and pushed, the checks pass, and the proof holds on 20 public PRs.
What the review found
Under option 2, a top-level post answers a bot's review bodies and conversation comments when it names the bot the way refer_as does, which for a bot is its BOT_NAMES name standing on its own. "Rex" and "Copilot" are also product names, so a post that mentions the product, not the reviewer, counts as an answer. On the current code each of these addresses the bot:
"Tested in a Rex tab" (Rex)
"Fixed the crash in Rex" (Rex)
"Disabled GitHub Copilot suggestions" (Copilot)
One post that names an author answers all of that author's earlier review bodies and conversation comments, so one such remark skips every earlier review by that bot. That is the same silent skip as the michaeluloth.com#83 remark, and Rex tabs come up often in this repo's own PRs. A future claude entry named "Claude" would match even more.
People are unaffected: an @login is not an ordinary word.
Decision needed
How should a top-level post address a bot?
Keep the name rule, as chosen. A passing mention of the product skips that bot's earlier reviews, and the skip shows in the ⏭️ Skipped list with its evidence.
Only a post that opens with the bot's thank line addresses it (Thanks for the feedback, Rex:). The run's own replies always open this way, so its answers still count. A hand-written "Thanks copilot" no longer counts, so the run drafts a reply the author can decline. This narrows option 2 for bots only.
The bot's name must open the post or a line ("Copilot: fixed", "Rex — done"). Hand-written replies that lead with the name still count. A remark that mentions the product mid-sentence does not, but one that starts a line with it ("Rex tabs now restore") still would.
Recommendation: 2. It removes the silent skip for bots with one rule the run already follows, and the cost is a draft the author sees before anything is posted, the same trade option 2 made for people.
Approach and design change: a resolution counts only when the feedback's author resolved it
This changes the approach's definition of an answered inline thread, and the design's thread_verdict.
What changes
An inline thread counts as answered by its resolution only when the one account, other than the answerers, that commented in it resolved it, and that account is a person. Any other resolution no longer counts: one by a bot such as Copilot, by a third person, by a deleted account, or by an answerer who never replied. A thread answered by a reply is unchanged: an answerer's reply that is the thread's newest comment still counts, whoever resolved it.
Why
A thread counted as answered whenever it was resolved, by anyone. So a thread Copilot resolved itself, with no reply from the author, was skipped, and nobody ever answered it. That is the silent skip P1 and P2 exist to prevent. The two mistakes cost different amounts: counting an unanswered thread as answered drops feedback with nothing to show it, while counting an answered one as unanswered only drafts a reply the author reads and can decline. So the rule counts only the resolution that settles the feedback: its author's own. When two reviewers commented in a thread, one resolving it can leave the other waiting, so that resolution doesn't count either.
Effect on the proof
On the public PR the PR description's first check uses, the inline thread Copilot resolved now prints answered : no instead of yes (resolved by Copilot).
Activity
ooloth commentedon Nov 17, 2026
Approach
When
monitor-prsstarts a second run ofreview-pr-comments-convergeon a PR, the run drafts replies only to feedback nobody has answered yet. Its report lists everything it skipped, with the comment or resolution that counted as the answer, so a wrong skip is easy to spot and overrule.fetch_pr_comments.pyworks out the verdict from the PR's current state, in the same single request it already makes, and prints oneansweredline on every review body, inline thread and conversation comment:yeswith its evidence,no, orn/afor the author's own.SKILL.mdputs every item not markednostraight onto the decided list, so it never reaches validation, and reads the lines again in the extra pass that starts when the PR moves before posting. A comment from a deleted account, which crashes the fetch today, prints as a deleted account and never counts as an answer. "Answered" means answered by the PR author or the account the run posts as:What a good fix must achieve 🎯
review-code, which reads the same fetch output, still sees every item it sees todayAssumptions 🤔
monitor-prsdecides "has feedback" by the same time order. If authors often post unrelated comments after feedback (such as "rebased"), an option that matches the answer's text would beat time order.Constraints 🧱
review-codecalls the same fetch script to reconcile earlier reviews, so the default output keeps every item.fetch_pr_comments.pyand its tests).Options ⚖️
answeredverdict per item;SKILL.mdskips and lists those items (chosen)SKILL.mdstates the ruleSKILL.mdalone states the rule; the fetch is unchangedDecided by 🧭
Instructions alone cannot work: today's output does not show the PR author, edit times, or which lines of a multi-line reply belong to whom, so a run cannot tell who answered last or whether feedback changed afterwards. Hiding answered items breaks
review-code, a reply-link format misses every answer written by hand, and a stored list goes stale, which decision 010 rules out. That leaves two options that change the fetch the same way, and they differ in where the rule lives. Printing a verdict keeps the rule in one function that tests cover, and the report still shows its evidence for each skip. Printing raw facts leaves the same rule in prose, applied by hand across fetch output that runs to 1,200 lines on a busy PR, where a missed comparison would go unnoticed. Both are easy to reverse, since nothing is stored.Deferred ⏳
monitor-prskeeps its own prose definition of unanswered feedback. One definition both skills use would stop them drifting apart.BOT_NAMES(such asclaude) prints asUNKNOWN.ooloth commentedon Nov 17, 2026
Design
fetch_discussionalready gets every page of threads, reviews and conversation comments. The same request now also returns the viewer's login, the PR author, who resolved each thread, and each item's state and created and edited times. Before anything prints, a judging step parses only what the verdict needs into typed values: who wrote each post, as an answerer or not, and when it was posted and last edited, as real datetimes. It drops unsubmitted posts, gives every review body, inline thread and conversation comment exactly oneVerdict, and stops the run with a sentence if any input cannot be read. The printer is unchanged apart from a# Answerers:line under# Source:and oneansweredline per item, soreview-codereads the same output it reads today. InSKILL.md, every item not markednogoes onto the decided list before validation and into a new report section,⏭️ Already answered, with its evidence. The author can name any of those items to have it validated anyway. When nothing is left, no validation runs and the Outcome isNothing unanswered. The check before posting compares the fresh fetch'sansweredlines with the saved fetch's, so a skip is never carried over. Types guarantee that a deleted account never answers, that an unsubmitted post never answers, and that every item gets one of three verdicts. Assertions check that eachyespoints at an answerer's post made after the feedback. Tests pin the time rules, ties included, and fresh-agent replays check the skill's handling.The approach's properties P1 to P7 still apply, plus one this design adds:
Types 🧬
Progressions considered
Answerers,Post,Answer, theVerdictandEvidenceunions, one judging function and two verdict functionsreview-codereadsauthor['login']that crash on a deleted accountDecided by: comparing timestamps as strings orders
06:46:39.5Zbefore06:46:39Z, so one format change would skip feedback edited after the answer, with nothing reported. Judging while printing would leave half an output when a value fails to parse. The full re-parse meets every property, but it adds types no property needs and rewrites the outputreview-codedepends on.Progression
Answerers.includesmatches onReviewervariants, never on the printed login.Ghostis never an answerer, even when the PR author's account was deleted. A real account namedghostcannot be confused with a deleted one. (P1, P3)Postcarries times as timezone-aware datetimes. A string or naive comparison cannot reach the rules.changed_atis the later of created and last edited. (P2)Posts, and only non-blank reviews becomeAnswers. A pending reply or review, and the empty review GitHub creates for an inline reply, can never answer anything. (P1)NotFeedback: every posted comment is an answerer's.Answered(Resolved(by)): the thread is resolved.Answered(ThreadReply): some posted comment is an answerer's, the latest such comment is A, and every other posted comment'schanged_atis strictly before A'sposted_at.Unanswered: anything else.NotFeedback: the author is an answerer.Unanswered: a submitted review has no submit time, so the run drafts a reply rather than skipping.Answered(TopLevelReply): the earliestAnswerposted strictly after the feedback'schanged_at, which ignores answers in the same second.Unanswered: no such answer exists. (P1, P2)Verdictis a closed union printed by an exhaustive match. Each item prints exactly one line, andAnsweredcannot exist without evidence. (P1, P4)ghost, the same as the review and conversation headers do today. Thread-comment authors gain__typenameso they parse asReviewer. (P6)Output lines
# Answerers: ooloth (PR author and gh login), oralice (PR author), ooloth (gh login), orooloth (gh login); the PR author's account was deleted.answered : yes (resolved by ooloth)answered : yes (reply <id> by ooloth at <time>)answered : yes (conversation comment <id> by ooloth at <time>)answered : yes (review <id> by ooloth at <time>)answered : noanswered : n/a (yours)A thread's line follows its
outdatedline. A review's line followsrefer_as, and only a review with a body gets one. A conversation comment's line followsedited.Assertions 🛡️
The types cannot enforce time order or that the evidence came from an answerer, because both depend on values.
PENDINGnorSUBMITTED, so whether the comment was posted is unknown. Treating it either way could hide a reviewer's comment.ThreadReplyevidence: the reply's author is an answerer, and itsposted_atis after every other posted comment'schanged_at.TopLevelReplyevidence: the answer's author is an answerer, and itsposted_atis after the feedback'schanged_at.NotFeedback: the item's author, or every posted comment's author for a thread, is an answerer.Telemetry 📡
# Answerers:line.answeredline, which the report copies.Tests 🧪
Neither the types nor the assertions say which items count as answered, which is the meaning of the rule, and nothing but a replay shows that the skill acts on the lines. So both need tests.
ghostinstead of crashing, and never answers# Answerers:line names the PR author and the gh login [same account, different accounts, deleted PR author]yes (resolved by …)yes (reply …)nonon/a (yours)noyes, naming that commentyes, naming that reviewyesno[review body, conversation comment]nonon/a (yours)answeredlineyesand thennonoyes (reply …)yesexactly when an answerer's answer was posted strictly after its last changeSKILL.mdand a fetch holdingno,yesandn/aitems, validation receives only thenoitems, and the report lists the rest with their evidenceyesorn/a, no validation runs and the Outcome isNothing unanswerednotoyeshas its draft dropped and is listed as skippedyestonois validated in the new passSKILL.mdis what an agent does, which only a fresh agent given the edited skill and a saved fetch can show.Deferred ⏳
fetch_pr_comments.py's test fake builds onlypullRequest's three lists, so a new field elsewhere in the response reaches the script as missing unless the fake is extended. This work extends it for the viewer and the PR author. Making the fake serve a whole response from the fixture would stop the next field being missed.ooloth commentedon Nov 17, 2026
Design change during the build
Three small changes to the design comment's Types and Assertions, made while implementing. None changes the approach.
changed_atalso counts a review's submit time. The design defined it as the later of created and last edited. A review is created when it is started and submitted later, so without the submit time a review could be judged as changed before it was posted. Counting it can only turn ayesinto ano, never skip feedback.n/awhen an answerer wrote it, otherwiseno. It never answers anything.NotFeedbackassertions run once over the finished verdicts in the judging step, instead of inside each verdict function, where they would only repeat the condition that chose the verdict.ooloth commentedon Nov 17, 2026
Design change during the build: items decided in an earlier pass
The design said the check before posting compares the fresh fetch's
answeredlines with the saved fetch's, so a skip is never carried over. It did not say what happens, in the new pass that check starts, to items an earlier pass already fixed, dismissed or escalated. Fresh agents replaying the skill disagreed about it, soSKILL.mdnow says:no: an item with a new reply or a changed edit time comes off the decided list, its old draft is dropped, and the new pass validates it with the new feedback. This is the case the extra pass exists for.yesorn/a: unchanged from the design. The draft is dropped and the item is skipped.Nothing unansweredis now the Outcome only when no draft carries over from an earlier pass.skills/factory: attack the design's correctness plan and check red
ooloth commentedon Nov 17, 2026
Escalation: an unrelated comment from the author skips every earlier review body
The build is finished up to the proof and pushed to the branch
262-skip-feedback-already-answered(4 commits, all checks green, every mutant caught, every replay passing). No PR is open yet, because the proof found the case the approach's Assumptions section said would reopen the time-order rule.What the proof found
The approach assumed that a later top-level comment or non-empty review from the author answers every earlier review body and conversation comment, and said that if authors often post unrelated comments after feedback, matching the answer to the item would beat time order.
A read-only fetch of ooloth/michaeluloth.com#83 with the new script shows that case on a real PR. The author's closing remark, "This one somehow allowed scope to creep beyond its original goal.", is the evidence for 9
answered : yeslines on review bodies and conversation comments posted before it. One of them is aclaudereview whose numbered "Minor observations" nobody answered. A second run would skip all 9 without validating them, and the report would list them as already answered.Every fresh agent that replayed the edited skill raised the same thing on the test fixture: a reply addressed to one reviewer ("Thanks, Rex: …") marked a different reviewer's review body as answered.
The rest of the proof held on 18 real PRs: old output is unchanged apart from the added lines, the request count is identical, newer feedback and same-second ties print
no, and broken input exits 1 with nothing printed.Decision needed
Which posts count as an answer to a review body or conversation comment? Inline threads are not affected: GitHub links each reply to its thread.
thankline or name its author the wayrefer_asdoes. The run's own replies always open with thethankline, so the answers it posts still count. A hand-written answer that names nobody givesno, and the run drafts a reply the author can decline before anything is posted. A remark like the one on perf: Add command caching to UV feature utilities #83 answers nothing. This adds one rule to the verdict function, with tests and replays, and amends the approach, which rejected linking an answer to its item because hand-written answers would be missed.Recommendation: 2. A wrongly skipped item is the failure nobody notices, while an extra draft is caught when the author reads the drafts before approving them. Option 2 removes the first and turns the hand-written case into the second.
ooloth commentedon Nov 17, 2026
Approach change: an answer must address the item's author
Option 2 from the escalation is chosen. This changes the approach's definition of an answered review body or conversation comment, and the Assumptions entry it rested on.
Before: an answerer posted a top-level comment, or a review with a non-empty body, after the feedback was submitted or last edited.
Now: the same, and the post must also address that item's author: it opens with the item's
thankline, or names its author the wayrefer_asdoes (@loginfor a person, theBOT_NAMESname for a bot). The evidence is still the earliest such post after the feedback's last change.The build resumes from the red step for this rule.
skills/factory: decide everything and stop only before harm
skills/review-pr-comments-converge: count only answers to the author
skills/review-pr-comments-converge: pin @login matching
skills/review-pr-comments-converge: drop the redundant thank-line check
skills/review-pr-comments-converge: match @login only as a mention
skills/review-pr-comments-converge: match bot names only as names
skills/review-pr-comments-converge: pin each name boundary
skills/review-pr-comments-converge: rewrap a long docstring line
ooloth commentedon Nov 17, 2026
Approach and design changes from the PR review
The independent review of #297 found ways a run could still skip feedback nobody answered. These change the approach's thread rule and parts of the skill's handling.
createdAtthan the moment it went public, so a follow-up drafted before the author's reply but submitted after it would have been skipped.@logincounts only where GitHub links it, so not before an underscore and not inside code.answeredline.noand something else.Nothing unansweredrun tells the author to close the tab.One finding needs a decision and is raised separately: a bot's name used in passing ("tested in a Rex tab", "GitHub Copilot") counts as addressing that bot.
skills/review-pr-comments-converge: close skips the review found
skills/review-pr-comments-converge: settle the skill's edges
skills/review-pr-comments-converge: ignore mentions in every code form
skills/review-pr-comments-converge: find code with a CommonMark parser
skills/review-pr-comments-converge: indent review bodies deeper
ooloth commentedon Nov 17, 2026
Escalation: a bot's name used in passing answers all its earlier reviews
The build is done apart from this one point. Draft PR #297 has every other review finding fixed and pushed, the checks pass, and the proof holds on 20 public PRs.
What the review found
Under option 2, a top-level post answers a bot's review bodies and conversation comments when it names the bot the way
refer_asdoes, which for a bot is itsBOT_NAMESname standing on its own. "Rex" and "Copilot" are also product names, so a post that mentions the product, not the reviewer, counts as an answer. On the current code each of these addresses the bot:One post that names an author answers all of that author's earlier review bodies and conversation comments, so one such remark skips every earlier review by that bot. That is the same silent skip as the michaeluloth.com#83 remark, and Rex tabs come up often in this repo's own PRs. A future
claudeentry named "Claude" would match even more.People are unaffected: an
@loginis not an ordinary word.Decision needed
How should a top-level post address a bot?
thankline addresses it (Thanks for the feedback, Rex:). The run's own replies always open this way, so its answers still count. A hand-written "Thanks copilot" no longer counts, so the run drafts a reply the author can decline. This narrows option 2 for bots only.Recommendation: 2. It removes the silent skip for bots with one rule the run already follows, and the cost is a draft the author sees before anything is posted, the same trade option 2 made for people.
skills/review-pr-comments-converge: split replies only at line breaks
skills/review-pr-comments-converge: escape other separators in replies
ooloth commentedon Nov 19, 2026
Approach and design change: a resolution counts only when the feedback's author resolved it
This changes the approach's definition of an answered inline thread, and the design's
thread_verdict.What changes
An inline thread counts as answered by its resolution only when the one account, other than the answerers, that commented in it resolved it, and that account is a person. Any other resolution no longer counts: one by a bot such as Copilot, by a third person, by a deleted account, or by an answerer who never replied. A thread answered by a reply is unchanged: an answerer's reply that is the thread's newest comment still counts, whoever resolved it.
Why
A thread counted as answered whenever it was resolved, by anyone. So a thread Copilot resolved itself, with no reply from the author, was skipped, and nobody ever answered it. That is the silent skip P1 and P2 exist to prevent. The two mistakes cost different amounts: counting an unanswered thread as answered drops feedback with nothing to show it, while counting an answered one as unanswered only drafts a reply the author reads and can decline. So the rule counts only the resolution that settles the feedback: its author's own. When two reviewers commented in a thread, one resolving it can leave the other waiting, so that resolution doesn't count either.
Effect on the proof
On the public PR the PR description's first check uses, the inline thread Copilot resolved now prints
answered : noinstead ofyes (resolved by Copilot).skills/review-pr-comments-converge: count only the reviewer's own res…