Skip to content

devops(bots): add the triage, revision, roll and release bots #491

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Pavel Feldman (pavelfeldman) merged 1 commit into from
Oct 30, 2026
+1,101 −0
Merged

devops(bots): add the triage, revision, roll and release bots #491

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
102 changes: 102 additions & 0 deletions .claude/skills/playwright-triage/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
---
name: playwright-triage
description: Triage and fix one microsoft/playwright-cli issue, or do a maintainer's task, unattended inside the Issue triage GitHub Actions workflow (.github/workflows/triage.yml), which a needs-triage, needs-triage-mac or needs-triage-win label starts, or a maintainer starts by hand with an issue, a task or both. Reproduces the issue on the runner, then leaves a verified fix proposal for the workflow to open as a draft PR, in this repo or in microsoft/playwright where the source of the CLI is, or an investigation note for it to post on the issue. Only for that workflow.
---

# Playwright CLI Issue Triage

You run unattended in the [Issue triage](../../../.github/workflows/triage.yml) workflow of
`microsoft/playwright-cli`. First read the shared guide, `$AGENT_GUIDES_DIR/triage.md`, a copy of
[microsoft/playwright/.github/actions/run-agent/guides/triage.md](https://github.com/microsoft/playwright/blob/main/.github/actions/run-agent/guides/triage.md),
and then [CLAUDE.md](../../../CLAUDE.md), both in full. The guide says how to triage an issue in any
Playwright repo. CLAUDE.md says how to build, test and lint this one. This file covers what neither
has: where the code is, the checkouts, how to run a published version, and the outcome that this
repo adds, a fix in `microsoft/playwright`.

## Where the code is

This repo is the npm package `@playwright/cli`: `playwright-cli.js` with its update check,
`skillCheck.js`, the README, the skill in `skills/playwright-cli/` and some tests of the packaged
CLI. `package.json` pins the alpha of `playwright` and `playwright-core` that the package runs. The
CLI is in `microsoft/playwright`, in `packages/playwright-core/src/tools/`: `cli-client/` has the
commands and their options, `cli-daemon/` has the daemon that keeps the browser open between
commands, and `backend/` has the tools that the commands run. Most bugs and most feature requests
are fixed there.

Fix a bug here only when the cause is in a file that this repo owns. `skills/playwright-cli/` is a
copy of `packages/playwright-core/src/tools/skills/playwright-cli/`, which `scripts/update.js`
refreshes on a roll, so its fix is upstream too.

## The checkouts

`$REPO_DIR` is the checkout of `microsoft/playwright-cli` that the guide describes. `npm ci` ran in
it, and the browsers of the pinned version are installed. The CLI uses Google Chrome from the
runner image by default. When it is missing, install it with `npx playwright install chrome`. The
clone is shallow: run `git fetch --unshallow --tags origin` in it
before you need its history.

`$PLAYWRIGHT_DIR` is a clone of `microsoft/playwright` at the tip of `main`, next to `$REPO_DIR`.
`npm ci` and `npm run build` ran in it, and Chromium, Firefox and WebKit are installed. It is
tip-of-tree for this repo: reproduce there, as a test in its `tests/mcp/cli-*.spec.ts`, with the
`cli` fixture of `tests/mcp/cli-fixtures.ts`. `npm run test-playwright-cli` runs those tests. Read
its `AGENTS.md` in full before you write the test. It says how to build, test and lint it. The
clone has the full history, without file contents until git needs them, so `git log` and
`git tag --contains` work there.

The pinned version is older than tip-of-tree. `npm view playwright-core@<alpha> gitHead` prints the
upstream commit that an alpha was built from.

## Fix in microsoft/playwright

When the fix is in `$PLAYWRIGHT_DIR`, follow the Fix section of the guide there, in place of
`$REPO_DIR`: create the branch `fix-<n>` from `main` in `$PLAYWRIGHT_DIR`, put the regression test
in its `tests/mcp/cli-*.spec.ts`, build and lint as its `AGENTS.md` says, and commit in its
convention, for example `fix(cli): ...`. Then leave the proposal in `$OUTPUT_DIR/upstream-fix/` in
place of `$OUTPUT_DIR/fix/`, with the same files:

- `fix.patch`: the output of `git diff --binary main HEAD` in `$PLAYWRIGHT_DIR`.
- `pr-body.md`: as the guide says. The `Fixes` line has the full URL of the issue,
`Fixes https://github.com/microsoft/playwright-cli/issues/<n>`, so that the PR closes the issue in
this repo.
- `proposal.json`: as the guide says. Write it last.

The publish job opens the draft PR in `microsoft/playwright`. The revision workflow of that repo
addresses the review. Leave one proposal: `fix/` or `upstream-fix/`, not both. Do not change
`package.json` here to pick up the fix. It reaches `@playwright/cli` with the next roll.

**Check the linked PRs in both repos.** A PR in `microsoft/playwright` links this issue with its
full URL or as `microsoft/playwright-cli#<n>`. Search there too:
`repo:microsoft/playwright is:pr "playwright-cli/issues/<n>"` and
`repo:microsoft/playwright is:pr "playwright-cli#<n>"`. A PR there from the team or from
`microsoft-playwright-automation` is a team PR. Verify a PR there in `$PLAYWRIGHT_DIR`.

**Already fixed** means that tip-of-tree passes and the pinned version fails. The fix reaches users
with the next roll. Find the upstream PR with
`git -C $PLAYWRIGHT_DIR log <gitHead of the pinned alpha>..main -- packages/playwright-core/src/tools`,
and name it in the note.

## Published versions

To run a published version, for example the version of the reporter or to bisect, create
`$SCRATCH_DIR/playwright-cli-<n>/<version>/` and run there:

```bash
npm init -y
npm i @playwright/cli@<version>
npx playwright-cli --help
```

The fixtures of the repo are not available there. Run the condensed repro as `npx playwright-cli`
commands, or as a script that spawns them, in that directory. Serve the page from a `data:` URL or
from `http.createServer()` in the script. Run `npx playwright-cli close-all` when you are done, so
that no daemon stays open.

`npm view @playwright/cli@<version> dependencies.playwright-core` prints the alpha of a release.
Match a regression to upstream commits with
`git -C $PLAYWRIGHT_DIR log <good gitHead>..<bad gitHead> -- packages/playwright-core/src/tools`.

## PRs from third parties

Do not run a PR in this repo that changes `package.json`, `package-lock.json`, `scripts/`,
`.github/`, `.azure-pipelines/` or `.claude/`. For a PR in `microsoft/playwright`, the list in
`$PLAYWRIGHT_DIR/.claude/skills/playwright-triage/SKILL.md` applies.
34 changes: 34 additions & 0 deletions .github/actions/setup-triage-runner/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
name: 'Set up the triage runner'
description: >
Prepares the environment that the triage and revision agents work in: the checkout with its
dependencies and the browsers of the pinned version, the Copilot CLI, and next to the checkout a
clone of microsoft/playwright at the tip of main, built, where the source of the CLI is. Exports
PLAYWRIGHT_DIR, the path of that clone, and CI_ENV_EXEC, the wrapper from microsoft/playwright
that runs a command under xvfb on Linux and at normal I/O priority on Windows.
runs:
using: composite
steps:
- uses: microsoft/playwright/.github/actions/ci-env@main
- uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
with:
node-version: '24'
- name: Install the dependencies and the browsers
shell: bash
run: |
npm ci
npx playwright install --with-deps
# Next to the checkout, where the skills expect it.
- name: Clone and build microsoft/playwright
shell: bash
run: |
dir="$(dirname "${GITHUB_WORKSPACE//\\//}")/playwright"
rm -rf "$dir"
git clone --filter=blob:none https://github.com/microsoft/playwright.git "$dir"
cd "$dir"
npm ci
npm run build
npx playwright install --with-deps chromium firefox webkit
echo "PLAYWRIGHT_DIR=$dir" >> "$GITHUB_ENV"
- name: Install the Copilot CLI
shell: bash
run: npm install -g @github/copilot
215 changes: 215 additions & 0 deletions .github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,215 @@
# Prepares a release. Copilot CLI writes the release notes, as .claude/skills/dev/release.md says,
# without write credentials. Fresh jobs bump the version, open the "chore: mark v<version>" pull
# request with the notes as its description using the app, so CI runs on it, and comment on the
# issue. A maintainer copies the notes into the release draft.
name: Prepare release

on:
# Apply needs-release to an issue; re-apply it to write the notes again.
issues:
types: [labeled]
workflow_dispatch:
inputs:
issue:
description: "Issue to report on; leave empty for none"
type: string
required: false
default: ""
version:
description: "Version to release; leave empty for the next patch version"
type: string
required: false
default: ""

permissions: {}

jobs:
start:
name: Find the version
if: >-
github.repository == 'microsoft/playwright-cli' &&
(github.event_name == 'workflow_dispatch' || github.event.label.name == 'needs-release')
runs-on: ubuntu-latest
permissions:
contents: read
issues: write
outputs:
version: ${{ steps.version.outputs.version }}
comment: ${{ steps.post.outputs.id }}
env:
ISSUE: ${{ github.event.issue.number || inputs.issue }}
VERSION: ${{ inputs.version }}
GH_TOKEN: ${{ github.token }}
GH_REPO: ${{ github.repository }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Find the next version
id: version
run: |
current=$(node -p "require('./package.json').version")
version=${VERSION:-$(node -p 'const [major, minor, patch] = process.argv[1].split("."); `${major}.${minor}.${+patch + 1}`' "$current")}
[[ "$version" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] || { echo "invalid version: $version" >&2; exit 1; }
echo "Releasing v$version; package.json has $current" >> "$GITHUB_STEP_SUMMARY"
echo "version=$version" >> "$GITHUB_OUTPUT"
- name: Post the run link
id: post
if: env.ISSUE != ''
env:
NEXT: ${{ steps.version.outputs.version }}
run: |
[[ "$ISSUE" =~ ^[1-9][0-9]*$ ]] || { echo "issue must be a positive integer" >&2; exit 1; }
body=$(printf "Hi, I'm the Playwright bot. I started to write the release notes of v%s: [agent run](%s)\n<!-- playwright-release-started -->" "$NEXT" "$RUN_URL")
id=$(gh api "repos/$GH_REPO/issues/$ISSUE/comments" -f body="$body" --jq .node_id)
echo "id=$id" >> "$GITHUB_OUTPUT"

notes:
name: Write the release notes of v${{ needs.start.outputs.version }}
needs: start
runs-on: ubuntu-latest
timeout-minutes: 90
concurrency:
group: release
cancel-in-progress: false
defaults:
run:
shell: bash
permissions:
contents: read
copilot-requests: write
env:
VERSION: ${{ needs.start.outputs.version }}
outputs:
has_notes: ${{ steps.agent.outputs.has-result }}
steps:
# The full history, for the previous release. No credentials in the checkout.
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0
persist-credentials: false
- name: Record the directories
run: |
echo "OUTPUT_DIR=$RUNNER_TEMP/release" >> "$GITHUB_ENV"
echo "SCRATCH_DIR=$RUNNER_TEMP/scratch" >> "$GITHUB_ENV"
echo "REPO_DIR=$GITHUB_WORKSPACE" >> "$GITHUB_ENV"
echo "PLAYWRIGHT_DIR=$(dirname "$GITHUB_WORKSPACE")/playwright" >> "$GITHUB_ENV"
mkdir -p "$RUNNER_TEMP/release" "$RUNNER_TEMP/scratch"
- uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
with:
node-version: '24'
- name: Clone microsoft/playwright and install the Copilot CLI
run: |
git clone --filter=blob:none https://github.com/microsoft/playwright.git "$PLAYWRIGHT_DIR"
npm install -g @github/copilot
- name: Write the notes with Copilot CLI
id: agent
uses: microsoft/playwright/.github/actions/run-agent@main
with:
prompt: |
Write the release notes of @playwright/cli v${{ env.VERSION }}. Follow steps 2 to 7 of .claude/skills/dev/release.md. Read the skill in full before you start.

The workflow does the other steps. A later job does step 1, the version bump, and step 8: it commits, pushes and opens the pull request with your notes as its description. The notes cover the Playwright version that package.json on main pins. Do not change, commit or push files in the checkout, and do not run gh.

`~/code/playwright` in the skill is $PLAYWRIGHT_DIR here: a clone of microsoft/playwright at the tip of main, with the full history. The checkout of this repo has the full history too. `gh` is not authenticated: read pull requests and issues with the GitHub MCP tools.

Nobody watches the run: do not ask questions.

Keep $OUTPUT_DIR/summary.md up to date as you work: the commit window, what you kept, what you dropped and why. Draft the notes in $SCRATCH_DIR. When they are done, write them to $OUTPUT_DIR/release-notes.md, in place of RELEASE_NOTES_v<version>.md, last, in one go.
output-dir: ${{ env.OUTPUT_DIR }}
result-file: release-notes.md
artifact-name: release
github-tools: true

publish:
name: Open the pull request
needs: [start, notes]
# release-notes.md is written last, so finished notes are published even when the job failed later.
if: ${{ !cancelled() && needs.notes.outputs.has_notes == 'true' }}
runs-on: ubuntu-latest
outputs:
pr: ${{ steps.pr.outputs.url }}
env:
VERSION: ${{ needs.start.outputs.version }}
TITLE: "chore: mark v${{ needs.start.outputs.version }}"
BRANCH: release/v${{ needs.start.outputs.version }}
steps:
- uses: actions/create-github-app-token@64ab729a6294000f5aa4a4bc0b73d1f32bc45510 # v3.3.0
id: app-token
with:
app-id: ${{ vars.PLAYWRIGHT_APP_ID }}
private-key: ${{ secrets.PLAYWRIGHT_PRIVATE_KEY }}
# Pushes as the app, so CI runs on the pull request.
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
token: ${{ steps.app-token.outputs.token }}
- uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
with:
node-version: '24'
- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
name: release
path: ${{ runner.temp }}/release
# Step 1 of the skill. Only the notes come from the agent, as the description.
- name: Bump the version and push the branch
run: |
npm version "$VERSION" --no-git-tag-version --ignore-scripts --allow-same-version
git config user.name "microsoft-playwright-automation[bot]"
git config user.email "203992400+microsoft-playwright-automation[bot]@users.noreply.github.com"
git checkout -b "$BRANCH"
git commit -am "$TITLE"
git push --force origin "$BRANCH"
# The description is the notes verbatim, for a maintainer to copy into the release draft.
- name: Open the pull request
id: pr
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
env:
BASE: ${{ github.ref_name }}
with:
github-token: ${{ steps.app-token.outputs.token }}
script: |
const fs = require('fs');
const { BASE, BRANCH, TITLE, RUNNER_TEMP } = process.env;
const body = fs.readFileSync(`${RUNNER_TEMP}/release/release-notes.md`, 'utf8').trim();
if (!body)
throw new Error('release-notes.md is empty');
const { owner, repo } = context.repo;
const { data: open } = await github.rest.pulls.list({ owner, repo, head: `${owner}:${BRANCH}`, state: 'open' });
let pr = open[0];
if (pr)
await github.rest.pulls.update({ owner, repo, pull_number: pr.number, body });
else
({ data: pr } = await github.rest.pulls.create({ owner, repo, head: BRANCH, base: BASE, title: TITLE, body }));
core.setOutput('url', pr.html_url);
await core.summary.addRaw(`Pull request: ${pr.html_url}\n`).write();

comment:
name: Comment on the issue
needs: [start, notes, publish]
if: ${{ !cancelled() && needs.start.result == 'success' && (github.event_name == 'issues' || inputs.issue != '') }}
runs-on: ubuntu-latest
permissions:
issues: write
steps:
- name: Post the result
env:
GH_TOKEN: ${{ github.token }}
GH_REPO: ${{ github.repository }}
ISSUE: ${{ github.event.issue.number || inputs.issue }}
VERSION: ${{ needs.start.outputs.version }}
PR_URL: ${{ needs.publish.outputs.pr }}
START_COMMENT: ${{ needs.start.outputs.comment }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
run: |
if [[ -n "$PR_URL" ]]; then
text="I opened the release PR of v$VERSION. Its description has the release notes, ready to copy into the release draft: $PR_URL"
else
text="I couldn't prepare the release of v$VERSION. The agent run has the details."
fi
printf "Hi, I'm the Playwright bot. %s\n\n<sub>Prepared by the Playwright bot - [agent run](%s)</sub>\n" "$text" "$RUN_URL" > "$RUNNER_TEMP/comment.md"
gh issue comment "$ISSUE" --body-file "$RUNNER_TEMP/comment.md"
if [[ -n "$START_COMMENT" ]]; then
gh api graphql -f query='mutation($id: ID!) { minimizeComment(input: { subjectId: $id, classifier: OUTDATED }) { clientMutationId } }' -f id="$START_COMMENT" \
|| echo "::warning::Could not hide the comment with the run link"
fi
Loading
Loading