Skip to content

A second run of review-pr-comments-converge drafts no reply to feedback already answered #262

Open

Description

@ooloth

Current state

Each run of review-pr-comments-converge re-validates every comment on the PR and drafts a reply to each one, including feedback an earlier run already answered, so a PR that gets new feedback after a first round can receive duplicate replies. The list of decided comments lasts only one run. Nothing in Phase 1 or Step A skips a resolved thread or one whose last reply is the author's, and monitor-prs starts a fresh run whenever new feedback arrives.

Ideal state

  • A run drafts no reply to an inline thread, review body or conversation comment that the PR author already answered
  • The report lists any feedback it skipped as already answered

Decisions

Constraints

Open

No open decisions.

Starting points

  • tools/agents/config/skills/review-pr-comments-converge/SKILL.md: the decided list and Step A
  • tools/agents/config/skills/review-pr-comments-converge/scripts/fetch_pr_comments.py: the resolved flag and REPLIES it prints

Activity

  1. ooloth commented on Nov 17, 2026

    @ooloth
    OwnerAuthor

    Approach

    When monitor-prs starts a second run of review-pr-comments-converge on a PR, the run drafts replies only to feedback nobody has answered yet. Its report lists everything it skipped, with the comment or resolution that counted as the answer, so a wrong skip is easy to spot and overrule. fetch_pr_comments.py works out the verdict from the PR's current state, in the same single request it already makes, and prints one answered line on every review body, inline thread and conversation comment: yes with its evidence, no, or n/a for the author's own. SKILL.md puts every item not marked no straight onto the decided list, so it never reaches validation, and reads the lines again in the extra pass that starts when the PR moves before posting. A comment from a deleted account, which crashes the fetch today, prints as a deleted account and never counts as an answer. "Answered" means answered by the PR author or the account the run posts as:

    • Inline thread: resolved, or its newest posted comment is an answerer's and no other comment in it was edited at or after that time.
    • Review body or conversation comment: an answerer posted a top-level comment, or a review with a non-empty body, after the feedback was submitted or last edited. Empty-body reviews, which GitHub creates for inline replies, and unsubmitted comments are not answers.

    What a good fix must achieve 🎯

    Property Category
    P1 A run drafts no reply to an inline thread, review body or conversation comment that is already answered, as defined above 🛡️ Safety
    P2 Feedback that arrived after the answer is never skipped: a reviewer's follow-up, a newer review, or feedback edited after the answer, with a same-second tie counting as new 🛡️ Safety
    P3 The PR author's own comments and reviews are never treated as feedback 🛡️ Safety
    P4 The report lists every skipped item with the evidence that counted as its answer 🛡️ Safety
    P5 Whether an item is answered is read from the PR's state on every fetch, including the one before posting, and never from a stored record 🛡️ Safety
    P6 review-code, which reads the same fetch output, still sees every item it sees today 🙂 Experience
    P7 The fetch makes no more GitHub requests than it does today ⚡ Performance

    Assumptions 🤔

    • A later top-level comment or non-empty review from the author answers every earlier review body and conversation comment, because GitHub records no link from an answer to the review body or comment it answers. The run itself answers each review body with one conversation comment, and monitor-prs decides "has feedback" by the same time order. If authors often post unrelated comments after feedback (such as "rebased"), an option that matches the answer's text would beat time order.

    Constraints 🧱

    • An inline reply makes GitHub create an empty-body review by the replier, so "the author's newest review" alone would mark review bodies answered that were never answered.
    • review-code calls the same fetch script to reconcile earlier reviews, so the default output keeps every item.
    • Open PR Number each review's PR comments within that review in fetch_pr_comments.py #293 changes the same two files (fetch_pr_comments.py and its tests).

    Options ⚖️

    Option P1 P2 P3 P4 P5 P6 P7 Leaves behind
    🏆 The fetch prints an answered verdict per item; SKILL.md skips and lists those items (chosen) ✅ ✅ ✅ ✅ ✅ ✅ ✅ ❌ one function, one output line per item, one report section
    The fetch prints the raw facts (PR author, viewer, edit times); SKILL.md states the rule ✅ ✅ ✅ ✅ ✅ ✅ ✅ ❌ five new output fields and a rule in prose that each run applies by hand, with no test
    SKILL.md alone states the rule; the fetch is unchanged ❌ ❌ ✅ ✅ ✅ ✅ ✅ ✅
    The fetch hides answered items ✅ ✅ ✅ ✅ ✅ ❌ ✅ ❌ a second output mode
    Count only replies that link to what they answer ❌ ✅ ✅ ✅ ✅ ✅ ✅ ❌ a link format every reply must carry
    Keep a list of answered items across runs ✅ ❌ ✅ ✅ ❌ ✅ ✅ ❌ a store

    Decided by 🧭

    Instructions alone cannot work: today's output does not show the PR author, edit times, or which lines of a multi-line reply belong to whom, so a run cannot tell who answered last or whether feedback changed afterwards. Hiding answered items breaks review-code, a reply-link format misses every answer written by hand, and a stored list goes stale, which decision 010 rules out. That leaves two options that change the fetch the same way, and they differ in where the rule lives. Printing a verdict keeps the rule in one function that tests cover, and the report still shows its evidence for each skip. Printing raw facts leaves the same rule in prose, applied by hand across fetch output that runs to 1,200 lines on a busy PR, where a missed comparison would go unnoticed. Both are easy to reverse, since nothing is stored.

    Deferred ⏳

    • monitor-prs keeps its own prose definition of unanswered feedback. One definition both skills use would stop them drifting apart.
    • An escalation the run posts (options left for the author) ends the thread with the author's comment, so the next run skips it and only the skipped list shows it. Nothing marks an escalation as still open.
    • A bot that posts on PRs but is missing from BOT_NAMES (such as claude) prints as UNKNOWN.
  2. ooloth commented on Nov 17, 2026

    @ooloth
    OwnerAuthor

    Design

    fetch_discussion already gets every page of threads, reviews and conversation comments. The same request now also returns the viewer's login, the PR author, who resolved each thread, and each item's state and created and edited times. Before anything prints, a judging step parses only what the verdict needs into typed values: who wrote each post, as an answerer or not, and when it was posted and last edited, as real datetimes. It drops unsubmitted posts, gives every review body, inline thread and conversation comment exactly one Verdict, and stops the run with a sentence if any input cannot be read. The printer is unchanged apart from a # Answerers: line under # Source: and one answered line per item, so review-code reads the same output it reads today. In SKILL.md, every item not marked no goes onto the decided list before validation and into a new report section, ⏭️ Already answered, with its evidence. The author can name any of those items to have it validated anyway. When nothing is left, no validation runs and the Outcome is Nothing unanswered. The check before posting compares the fresh fetch's answered lines with the saved fetch's, so a skip is never carried over. Types guarantee that a deleted account never answers, that an unsubmitted post never answers, and that every item gets one of three verdicts. Assertions check that each yes points at an answerer's post made after the feedback. Tests pin the time rules, ties included, and fresh-agent replays check the skill's handling.

    The approach's properties P1 to P7 still apply, plus one this design adds:

    Property Category
    P8 A fetch that cannot judge every item prints nothing and exits with a sentence naming the item and field 🛡️ Safety

    Types 🧬

    Progressions considered

    Option P1 P2 P3 P4 P5 P6 P7 P8 Leaves behind
    🏆 Parse only what the verdict reads, judge every item before printing, and print from today's dicts (chosen) ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ❌ Answerers, Post, Answer, the Verdict and Evidence unions, one judging function and two verdict functions
    Parse the whole discussion into typed threads, reviews and comments, and rewrite the printer over them ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ❌ about six more types and a rewrite of every print line review-code reads
    Verdict functions over the raw dicts, comparing ISO strings while printing ✅ ❌ ✅ ✅ ✅ ✅ ✅ ❌ ❌ comparisons on author['login'] that crash on a deleted account

    Decided by: comparing timestamps as strings orders 06:46:39.5Z before 06:46:39Z, so one format change would skip feedback edited after the answer, with nothing reported. Judging while printing would leave half an output when a value fails to parse. The full re-parse meets every property, but it adds types no property needs and rewrites the output review-code depends on.

    Progression

    GraphQL page(s): data.viewer, pullRequest { author, reviewThreads, reviews, comments }
      ↓ fetch_discussion()                       viewer and PR author read from the first page
    Discussion { threads, reviews, conversation (raw dicts, unchanged), answerers: Answerers }
      Answerers { pr_author: Reviewer, viewer: str }   .includes(Reviewer) -> bool, Ghost never
      ↓ judge_discussion()                       runs before the first print
        parse_post(node) -> Post { id, author: Reviewer, posted_at: datetime, changed_at: datetime }
          only posted nodes: thread comments in state SUBMITTED, reviews not PENDING, every conversation comment
        answers: tuple[Answer, ...] sorted by time
          Answer { kind: "conversation comment" | "review", post: Post }
          every answerer's conversation comment, and every answerer's review with a non-blank body
        thread_verdict(posts, resolved_by, answerers) -> Verdict
        item_verdict(feedback: Post, answers, answerers) -> Verdict
      Verdicts { threads, reviews, conversation: dict[id, Verdict] }
        Verdict  = Answered(evidence: Evidence) | Unanswered() | NotFeedback()
        Evidence = Resolved(by: Reviewer) | ThreadReply(post: Post) | TopLevelReply(answer: Answer)
      ↓ print_discussion(target, discussion, verdicts)
    stdout: "# Answerers: …" under "# Source:", and "  answered   : …" on each review body, thread and conversation comment
    
    • Answerers.includes matches on Reviewer variants, never on the printed login. Ghost is never an answerer, even when the PR author's account was deleted. A real account named ghost cannot be confused with a deleted one. (P1, P3)
    • Post carries times as timezone-aware datetimes. A string or naive comparison cannot reach the rules. changed_at is the later of created and last edited. (P2)
    • Only posted nodes become Posts, and only non-blank reviews become Answers. A pending reply or review, and the empty review GitHub creates for an inline reply, can never answer anything. (P1)
    • The thread rule.
      • NotFeedback: every posted comment is an answerer's.
      • Answered(Resolved(by)): the thread is resolved.
      • Answered(ThreadReply): some posted comment is an answerer's, the latest such comment is A, and every other posted comment's changed_at is strictly before A's posted_at.
      • Unanswered: anything else.
      • A reviewer's reply in a thread the author started is feedback, and an edit in the same second as the answer reopens the thread. (P1, P2, P3)
    • The review body and conversation comment rule.
      • NotFeedback: the author is an answerer.
      • Unanswered: a submitted review has no submit time, so the run drafts a reply rather than skipping.
      • Answered(TopLevelReply): the earliest Answer posted strictly after the feedback's changed_at, which ignores answers in the same second.
      • Unanswered: no such answer exists. (P1, P2)
    • Verdict is a closed union printed by an exhaustive match. Each item prints exactly one line, and Answered cannot exist without evidence. (P1, P4)
    • Judging reads only this fetch and writes nothing. (P5)
    • The query gains fields only. The request count is unchanged, and every existing output line is still printed in its place. (P6, P7)
    • What a failed parse keeps. It exits naming the item's id and the field, before printing. Re-running the fetch replays it. (P8)
    • The crash sites. The REPLIES lines and the orphan thread header print a deleted account as ghost, the same as the review and conversation headers do today. Thread-comment authors gain __typename so they parse as Reviewer. (P6)

    Output lines

    • # Answerers: ooloth (PR author and gh login), or alice (PR author), ooloth (gh login), or ooloth (gh login); the PR author's account was deleted.
    • answered : yes (resolved by ooloth)
    • answered : yes (reply <id> by ooloth at <time>)
    • answered : yes (conversation comment <id> by ooloth at <time>)
    • answered : yes (review <id> by ooloth at <time>)
    • answered : no
    • answered : n/a (yours)

    A thread's line follows its outdated line. A review's line follows refer_as, and only a review with a body gets one. A conversation comment's line follows edited.

    Assertions 🛡️

    The types cannot enforce time order or that the evidence came from an answerer, because both depend on values.

    • Boundary validation (exit 1 with a sentence, before printing):
      • Missing viewer login: the response has no viewer login, so the run cannot tell its own replies apart.
      • Unreadable timestamp: a timestamp does not parse, or has no timezone.
      • Unknown thread-comment state: the state is neither PENDING nor SUBMITTED, so whether the comment was posted is unknown. Treating it either way could hide a reviewer's comment.
      • Why boundary validation: all three come from GitHub, not from this code.
    • Assertions (only a bug here can break them), one condition each:
      • ThreadReply evidence: the reply's author is an answerer, and its posted_at is after every other posted comment's changed_at.
      • TopLevelReply evidence: the answer's author is an answerer, and its posted_at is after the feedback's changed_at.
      • NotFeedback: the item's author, or every posted comment's author for a thread, is an answerer.
      • Printing: every printed thread, review with a body and conversation comment has a verdict.
    • No recorded-signal check. Nothing spans runs, because nothing is stored.

    Telemetry 📡

    • Which accounts counted as answerers? A wrong viewer or PR author would change every verdict with no visible cause. Record: the # Answerers: line.
    • Why was an item skipped? Record: the evidence in its answered line, which the report copies.

    Tests 🧪

    Neither the types nor the assertions say which items count as answered, which is the meaning of the rule, and nothing but a replay shows that the skill acts on the lines. So both need tests.

    Expectation Property Technique
    A reply from a deleted account prints as ghost instead of crashing, and never answers P1, P6 Example
    An orphan thread opened by a deleted account prints P6 Example
    The # Answerers: line names the PR author and the gh login [same account, different accounts, deleted PR author] P4 Parametrized example
    A resolved thread is yes (resolved by …) P1, P4 Example
    A thread whose newest comment is the PR author's is yes (reply …) P1, P4 Example
    A thread whose newest comment is the viewer's, when the viewer is not the PR author, is answered P1 Example
    An orphan thread gets a verdict by the same rule P1 Example
    A reviewer's follow-up after the author's reply makes the thread no P2 Example
    A thread comment posted or edited in the same second as the reply, or later, makes the thread no P2 Parametrized example
    An edit made before the reply leaves the thread answered P1 Example
    A pending reply from the author does not answer the thread P1 Example
    A thread only answerers wrote in is n/a (yours) P3 Example
    A reviewer's reply in a thread the author started is no P2, P3 Example
    A deleted account never answers, even when the PR author's account was deleted P1 Example
    A review body followed by an answerer's conversation comment is yes, naming that comment P1, P4 Example
    A review body followed by an answerer's review with a body is yes, naming that review P1, P4 Example
    A conversation comment followed by an answerer's conversation comment is yes P1 Example
    The empty review an inline reply creates does not answer a review body P1 Example
    A pending review from the author does not answer P1 Example
    A later comment from someone other than an answerer does not answer P1 Example
    Feedback posted or edited in the same second as the answer, or edited after it, is no [review body, conversation comment] P2 Parametrized example
    A bot comment edited into a full review after the answer is no P2 Example
    A newer review after the answer is no P2 Example
    The PR author's own review and conversation comment are n/a (yours) P3 Example
    Every review body, thread and conversation comment prints exactly one answered line P4 Example
    An answered item is still printed in full P6 Example
    The same PR fetched before and after a reviewer's follow-up prints yes and then no P5 Example
    An answer on a later page is seen, with one request per page P1, P7 Example
    The existing line-order and one-call tests still pass P6, P7 Example
    A missing viewer, an unreadable timestamp or an unknown comment state exits 1 with a sentence and prints nothing P8 Parametrized example
    Any thread with a non-answerer's post or edit at or after the answerer's latest reply is no P2 Property
    Any thread whose answerer reply is strictly after all other activity is yes (reply …) P1, P4 Property
    Any review body or conversation comment is yes exactly when an answerer's answer was posted strictly after its last change P1, P2 Property
    With the edited SKILL.md and a fetch holding no, yes and n/a items, validation receives only the no items, and the report lists the rest with their evidence P1, P3, P4 Replay
    When every item is yes or n/a, no validation runs and the Outcome is Nothing unanswered P1, P4 Replay
    In the check before posting, an item whose line changed from no to yes has its draft dropped and is listed as skipped P1, P5 Replay
    In the check before posting, an item whose line changed from yes to no is validated in the new pass P2, P5 Replay
    • Example: each rule has a named edge, such as the tie, the empty review, the pending post or the deleted account, and a failing test should say which edge broke.
    • Parametrized example: the same claim holds across a few values of one dimension, such as same second against later.
    • Property: a wrongly skipped item is the one failure nobody would notice, so generated timelines, shaped toward same-second times and edits, check the rule against the approach's wording rather than against the code.
    • Replay: what changes in SKILL.md is what an agent does, which only a fresh agent given the edited skill and a saved fetch can show.

    Deferred ⏳

    • fetch_pr_comments.py's test fake builds only pullRequest's three lists, so a new field elsewhere in the response reaches the script as missing unless the fake is extended. This work extends it for the viewer and the PR author. Making the fake serve a whole response from the fixture would stop the next field being missed.
  3. ooloth commented on Nov 17, 2026

    @ooloth
    OwnerAuthor

    Design change during the build

    Three small changes to the design comment's Types and Assertions, made while implementing. None changes the approach.

    • changed_at also counts a review's submit time. The design defined it as the later of created and last edited. A review is created when it is started and submitted later, so without the submit time a review could be judged as changed before it was posted. Counting it can only turn a yes into a no, never skip feedback.
    • A submitted review with no submit time is treated like an unsubmitted post. It gets the verdict the design already gives it: n/a when an answerer wrote it, otherwise no. It never answers anything.
    • The NotFeedback assertions run once over the finished verdicts in the judging step, instead of inside each verdict function, where they would only repeat the condition that chose the verdict.
  4. ooloth commented on Nov 17, 2026

    @ooloth
    OwnerAuthor

    Design change during the build: items decided in an earlier pass

    The design said the check before posting compares the fresh fetch's answered lines with the saved fetch's, so a skip is never carried over. It did not say what happens, in the new pass that check starts, to items an earlier pass already fixed, dismissed or escalated. Fresh agents replaying the skill disagreed about it, so SKILL.md now says:

    • Not listed by the check: an item an earlier pass decided stays decided and is not validated again. Its draft goes into the new report and needs fresh approval. Validating it again would find the pushed fix and turn an approved ✅ reply into a ❌ "already handled" one.
    • Listed by the check, line still no: an item with a new reply or a changed edit time comes off the decided list, its old draft is dropped, and the new pass validates it with the new feedback. This is the case the extra pass exists for.
    • Listed by the check, line now yes or n/a: unchanged from the design. The draft is dropped and the item is skipped.

    Nothing unanswered is now the Outcome only when no draft carries over from an earlier pass.

  5. ooloth commented on Nov 17, 2026

    @ooloth
    OwnerAuthor

    Escalation: an unrelated comment from the author skips every earlier review body

    The build is finished up to the proof and pushed to the branch 262-skip-feedback-already-answered (4 commits, all checks green, every mutant caught, every replay passing). No PR is open yet, because the proof found the case the approach's Assumptions section said would reopen the time-order rule.

    What the proof found

    The approach assumed that a later top-level comment or non-empty review from the author answers every earlier review body and conversation comment, and said that if authors often post unrelated comments after feedback, matching the answer to the item would beat time order.

    A read-only fetch of ooloth/michaeluloth.com#83 with the new script shows that case on a real PR. The author's closing remark, "This one somehow allowed scope to creep beyond its original goal.", is the evidence for 9 answered : yes lines on review bodies and conversation comments posted before it. One of them is a claude review whose numbered "Minor observations" nobody answered. A second run would skip all 9 without validating them, and the report would list them as already answered.

    Every fresh agent that replayed the edited skill raised the same thing on the test fixture: a reply addressed to one reviewer ("Thanks, Rex: …") marked a different reviewer's review body as answered.

    The rest of the proof held on 18 real PRs: old output is unchanged apart from the added lines, the request count is identical, newer feedback and same-second ties print no, and broken input exits 1 with nothing printed.

    Decision needed

    Which posts count as an answer to a review body or conversation comment? Inline threads are not affected: GitHub links each reply to its thread.

    1. Keep time order, as planned. Ship as it is. A wrong skip shows in the ⏭️ Already answered list with its evidence, and the author can name the item to have it validated. On a PR like michaeluloth.com#83, one remark skips every earlier review body, and the author has to spot that in the list.
    2. Count a top-level post only when it addresses that item's author. The post must open with the item's thank line or name its author the way refer_as does. The run's own replies always open with the thank line, so the answers it posts still count. A hand-written answer that names nobody gives no, and the run drafts a reply the author can decline before anything is posted. A remark like the one on perf: Add command caching to UV feature utilities #83 answers nothing. This adds one rule to the verdict function, with tests and replays, and amends the approach, which rejected linking an answer to its item because hand-written answers would be missed.
    3. Skip only inline threads. Never skip a review body or conversation comment, so every one is validated each run. This removes the false skip completely, but a second run drafts a reply again to every review body it has already answered, which is the duplicate this ticket exists to stop for those two kinds.

    Recommendation: 2. A wrongly skipped item is the failure nobody notices, while an extra draft is caught when the author reads the drafts before approving them. Option 2 removes the first and turns the hand-written case into the second.

  6. ooloth commented on Nov 17, 2026

    @ooloth
    OwnerAuthor

    Approach change: an answer must address the item's author

    Option 2 from the escalation is chosen. This changes the approach's definition of an answered review body or conversation comment, and the Assumptions entry it rested on.

    Before: an answerer posted a top-level comment, or a review with a non-empty body, after the feedback was submitted or last edited.

    Now: the same, and the post must also address that item's author: it opens with the item's thank line, or names its author the way refer_as does (@login for a person, the BOT_NAMES name for a bot). The evidence is still the earliest such post after the feedback's last change.

    • Why: on Add project overview, integrations, and dev commands to CLAUDE.md michaeluloth.com#83 an unrelated closing remark from the author marked 9 earlier review bodies and comments as answered. A post that names nobody now answers nothing, so the run drafts a reply the author can decline before it is posted, instead of skipping feedback silently.
    • What stays the same: inline threads, where GitHub links each reply to its thread; the tie rule; pending posts and empty reviews never answering; and properties P1 to P8, with P1's "answered" now meaning this narrower rule.
    • Cost: a hand-written answer that names nobody gets a duplicate draft. One post that addresses a reviewer answers every earlier review body and conversation comment from that reviewer.

    The build resumes from the red step for this rule.

  7. ooloth commented on Nov 17, 2026

    @ooloth
    OwnerAuthor

    Approach and design changes from the PR review

    The independent review of #297 found ways a run could still skip feedback nobody answered. These change the approach's thread rule and parts of the skill's handling.

    • A resolved thread no longer hides a later follow-up. The approach counted a thread as answered when it was resolved, whoever resolved it. The run resolves every thread it replies to, and GitHub leaves a thread resolved when someone replies in it, so a reviewer's "this still isn't fixed" would have been skipped on every later run. That breaks P2. A resolved thread now counts as answered only when no reviewer's comment in it changed at or after an answerer's latest reply in it. Otherwise it is judged by the reply rule.
    • Thread comments are timed by when they were published, not drafted. A comment written inside a pending review has an earlier createdAt than the moment it went public, so a follow-up drafted before the author's reply but submitted after it would have been skipped.
    • @login counts only where GitHub links it, so not before an underscore and not inside code.
    • Reply bodies are fully indented, so a commenter cannot write a line that looks like the script's own answered line.
    • Skill handling:
      • A new pass starts only when an item moves between no and something else.
      • An item the author asks to have validated stays off the skipped list for the rest of the run.
      • Posting reports any item a sibling post marked answered while its own post failed.
      • A failed fetch stops the run, and a Nothing unanswered run tells the author to close the tab.
      • The report section is now called ⏭️ Skipped, since it also lists the author's own items.

    One finding needs a decision and is raised separately: a bot's name used in passing ("tested in a Rex tab", "GitHub Copilot") counts as addressing that bot.

  8. ooloth commented on Nov 17, 2026

    @ooloth
    OwnerAuthor

    Escalation: a bot's name used in passing answers all its earlier reviews

    The build is done apart from this one point. Draft PR #297 has every other review finding fixed and pushed, the checks pass, and the proof holds on 20 public PRs.

    What the review found

    Under option 2, a top-level post answers a bot's review bodies and conversation comments when it names the bot the way refer_as does, which for a bot is its BOT_NAMES name standing on its own. "Rex" and "Copilot" are also product names, so a post that mentions the product, not the reviewer, counts as an answer. On the current code each of these addresses the bot:

    • "Tested in a Rex tab" (Rex)
    • "Fixed the crash in Rex" (Rex)
    • "Disabled GitHub Copilot suggestions" (Copilot)

    One post that names an author answers all of that author's earlier review bodies and conversation comments, so one such remark skips every earlier review by that bot. That is the same silent skip as the michaeluloth.com#83 remark, and Rex tabs come up often in this repo's own PRs. A future claude entry named "Claude" would match even more.

    People are unaffected: an @login is not an ordinary word.

    Decision needed

    How should a top-level post address a bot?

    1. Keep the name rule, as chosen. A passing mention of the product skips that bot's earlier reviews, and the skip shows in the ⏭️ Skipped list with its evidence.
    2. Only a post that opens with the bot's thank line addresses it (Thanks for the feedback, Rex:). The run's own replies always open this way, so its answers still count. A hand-written "Thanks copilot" no longer counts, so the run drafts a reply the author can decline. This narrows option 2 for bots only.
    3. The bot's name must open the post or a line ("Copilot: fixed", "Rex — done"). Hand-written replies that lead with the name still count. A remark that mentions the product mid-sentence does not, but one that starts a line with it ("Rex tabs now restore") still would.

    Recommendation: 2. It removes the silent skip for bots with one rule the run already follows, and the cost is a draft the author sees before anything is posted, the same trade option 2 made for people.

  9. ooloth commented on Nov 19, 2026

    @ooloth
    OwnerAuthor

    Approach and design change: a resolution counts only when the feedback's author resolved it

    This changes the approach's definition of an answered inline thread, and the design's thread_verdict.

    What changes

    An inline thread counts as answered by its resolution only when the one account, other than the answerers, that commented in it resolved it, and that account is a person. Any other resolution no longer counts: one by a bot such as Copilot, by a third person, by a deleted account, or by an answerer who never replied. A thread answered by a reply is unchanged: an answerer's reply that is the thread's newest comment still counts, whoever resolved it.

    Why

    A thread counted as answered whenever it was resolved, by anyone. So a thread Copilot resolved itself, with no reply from the author, was skipped, and nobody ever answered it. That is the silent skip P1 and P2 exist to prevent. The two mistakes cost different amounts: counting an unanswered thread as answered drops feedback with nothing to show it, while counting an answered one as unanswered only drafts a reply the author reads and can decline. So the rule counts only the resolution that settles the feedback: its author's own. When two reviewers commented in a thread, one resolving it can leave the other waiting, so that resolution doesn't count either.

    Effect on the proof

    On the public PR the PR description's first check uses, the inline thread Copilot resolved now prints answered : no instead of yes (resolved by Copilot).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions