Repository navigation
LLM-assisted PR triage #5281
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+1,263
−288
Merged
LLM-assisted PR triage #5281
Changes from all commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
3597068
LLM-assisted PR triage; drop the no-changelog label
dsmiley 6bcaa9e
Harden advisory triage and automate contribution reminders
nick-boss-tech a8e01ab
Trim scripts README; document pr-triage-llm.py in its header
dsmiley 42eed39
PR triage: address review feedback
dsmiley 8346ead
PR triage: run without a PR (--from-git) and from supplied verdicts (…
dsmiley 886eb2b
PR triage: --from-git detects the branch it stemmed from
dsmiley 197f377
no license header needed
dsmiley 91e15a0
PR triage: skip the Jira judgement when the title has a reference; ti…
dsmiley 53101ee
PR triage: one Changelog judgement that picks the type and compares i…
dsmiley f28f775
link to skill
dsmiley 237f45b
PR triage: accept true/false/yes/no text verdicts; questions say the …
dsmiley 84d2d24
PR triage: default model qwen3.8-27b
dsmiley d05b602
no license needed
dsmiley File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,80 @@ | ||
| { | ||
| "_doc": "Each judgement is a question about what the change warrants on its merits, whatever the author included; the comment compares the verdict with what the PR includes.\nJudgement keys: id, title, question, help_url (the title links there).\nWhat the PR includes: evidence_files (globs; a changed file counts unless removed) minus evidence_exclude, or evidence_title (regex on the PR title). Matching files are hidden from the model. skip_when_included: the model is not asked when the evidence is present.\nChoice questions: choices (the verdict is one of these strings); evidence_value {files, pattern} gives the values the PR provides (group 1 of pattern, matched on added lines); absent_choice is the choice meaning nothing is warranted.\nTop level: model_defaults; max_diff_chars; exclude_patches (patches listed by name only); min_confidence (below it a mismatch is a weak signal); summary_from {files, pattern}: text given to every judgement as the author's own summary (the changelog entry title).", | ||
| "model_defaults": { | ||
| "temperature": 0, | ||
| "max_tokens": 2000, | ||
| "extra_body": {"chat_template_kwargs": {"enable_thinking": false}} | ||
| }, | ||
| "max_diff_chars": 60000, | ||
| "exclude_patches": [ | ||
| "**/*.lockfile", | ||
| "solr/licenses/**", | ||
| "**/*.sha1", | ||
| "gradle/libs.versions.toml" | ||
| ], | ||
| "min_confidence": "medium", | ||
| "summary_from": { | ||
| "files": ["changelog/unreleased/**"], | ||
| "pattern": "^\\+title:[ \\t]*(.*(?:\\n\\+[ \\t]+\\S.*)*)" | ||
| }, | ||
| "judgements": [ | ||
| { | ||
| "id": "needs_jira", | ||
| "title": "Jira issue in title", | ||
| "help_url": "https://github.com/apache/solr/blob/main/CONTRIBUTING.md", | ||
| "evidence_title": "\\bSOLR-[1-9][0-9]*\\b", | ||
| "skip_when_included": true, | ||
| "question": "Is this change substantial enough to warrant a Solr Jira issue under the contribution guidelines? A Jira issue is warranted only for a change that also warrants a changelog entry, i.e. one that Solr users or operators would notice. If it needs no release note (pure refactoring, test-only, build/CI, internal cleanup, typo fixes, documentation-only), the verdict is false. Judge the change itself. This is advisory, not verification that an issue exists or that the referenced issue matches the change." | ||
| }, | ||
| { | ||
| "id": "needs_changelog", | ||
| "title": "Changelog", | ||
| "help_url": "https://github.com/apache/solr/blob/main/dev-docs/changelog.adoc#_changelog_types_reference", | ||
| "choices": [ | ||
| "added", | ||
| "changed", | ||
| "fixed", | ||
| "deprecated", | ||
| "removed", | ||
| "dependency_update", | ||
| "other", | ||
| "(not-worthy)" | ||
| ], | ||
| "absent_choice": "(not-worthy)", | ||
| "evidence_value": { | ||
| "files": [ | ||
| "changelog/unreleased/**" | ||
| ], | ||
| "pattern": "^\\+type:\\s*([a-z_]+)" | ||
| }, | ||
| "question": "Which changelog type fits this change? Choose a type only if the change is one that Solr users or operators would notice and so warrants a release-note entry. added: a change requiring a user to take action to use it (opt-in), such as a new feature or new configuration value, typically documented in the Reference Guide. changed: an improvement that is not opt-in, modifying behavior or performance of existing requests or configuration. fixed: fixes buggy behavior with a user-visible symptom, such as a NullPointerException, data corruption, or a UI glitch. A fix to the build, tests, or developer tooling is not fixed; use other. deprecated: marks a configuration option, class, or API endpoint deprecated ahead of removal. removed: removes something previously deprecated, such as a parameter, class, or legacy module. dependency_update: a non-trivial dependency update, such as Lucene, Jetty, or Jackson. other: anything else worth a note, e.g. a large refactoring, or a build change or build fix worth a note. Answer (not-worthy) when no entry is warranted: pure refactoring, test-only changes, build/CI changes, internal cleanup, typo fixes, and documentation-only changes." | ||
| }, | ||
| { | ||
| "id": "needs_ref_guide", | ||
| "title": "Reference Guide update", | ||
| "help_url": "https://github.com/apache/solr/tree/main/solr/solr-ref-guide", | ||
| "evidence_files": ["solr/solr-ref-guide/**"], | ||
| "evidence_exclude": ["solr/solr-ref-guide/modules/upgrade-notes/**"], | ||
| "question": "Does this PR add, change, or remove something documented (or that ought to be documented) in the Solr Reference Guide, such as a request parameter, a solrconfig.xml or schema setting, a system property or env var, a V1/V2 API, a CLI option, a metric, or a user-facing behavior, such that the Reference Guide (solr/solr-ref-guide) should be updated? Internal changes do not count." | ||
| }, | ||
| { | ||
| "id": "needs_upgrade_notes", | ||
| "title": "Upgrade notes", | ||
| "help_url": "https://github.com/apache/solr/tree/main/solr/solr-ref-guide/modules/upgrade-notes/pages", | ||
| "evidence_files": ["solr/solr-ref-guide/modules/upgrade-notes/**"], | ||
| "question": "Is this PR a backwards-incompatible change that users must act on when upgrading: a removed or renamed parameter, API, config element, class used by plugins, or a changed default that alters existing behavior? The verdict is false for additive changes and for changes to internal classes that plugins would not reasonably use." | ||
| }, | ||
| { | ||
| "id": "needs_tests", | ||
| "title": "Tests", | ||
| "evidence_files": [ | ||
| "**/src/test*/**", | ||
| "**/src/commonTest/**", | ||
| "**/src/jvmTest/**", | ||
| "**/src/wasmJsTest/**", | ||
| "dev-tools/scripts/tests/test_*.py" | ||
| ], | ||
| "question": "Does this PR change non-trivial production logic (beyond refactoring, renames, logging, or comments) such that it should come with new or changed tests, whether or not the diff already includes some? Build, CI and developer tooling (workflows, Gradle build logic, scripts under dev-tools) does not need tests: the verdict is false." | ||
| } | ||
| ] | ||
| } |
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rather than give users a checklist for these things, we let an LLM make some judgements to help the user (and reviewers!).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Inovative!