ci: add a dispatchable sfw CI simulation

[Julian Gruber (juliangruber)]

Why

Verify stability by simulating load in GitHub Actions context. The shape is common to any user running the action plus an explicit sfw npm install in GitHub Actions.

What

One dispatchable workflow, test-sfw-ci-simulation.yml. A measurement rather than a pass/fail test. Inputs:

• baseline_action_ref (default: latest tag v1.3.2; empty skips the baseline)
• fixture_repo (default expressjs/express)
• iterations (default 10)

Install — matrix of windows-2025, windows-11-arm, ubuntu-26.04 × candidate/baseline × N iterations. Each job installs sfw with uses: ./ (candidate) or a second fetch of the action at the baseline ref, then runs sfw npm install --ignore-scripts in the fixture. Failures are classified by symptom text. Report aggregates a flake table into the run summary and splits binary download failures out of action-setup-failed from the install jobs' failure annotations.

Fleet form throughout: inline bootstrap checkout (no actions/checkout), fleet upload-artifact/download-artifact composites, no third-party actions, pinned runner images, every step named. pnpm run lint --all passes; the remaining pnpm run check failures are machine-local (pricing data, telemetry env, Homebrew, roster).

Running it

gh workflow run test-sfw-ci-simulation.yml --repo SocketDev/action --ref <branch> -f iterations=20

Simulation cost: iterations=20 with baseline is ~100 install jobs, most on Windows.

🤖 Generated with Claude Code

---

Note

Low Risk
Adds optional CI measurement only; no changes to the action runtime, customer workflows, or release behavior.

Overview
Adds a manually dispatched workflow (test-sfw-ci-simulation.yml) to measure how reliably the Socket Firewall action plus an explicit sfw npm install behaves in GitHub Actions—before shipping action or binary changes.

Each run repeats installs across windows-2025, windows-11-arm, and ubuntu-26.04, comparing the checked-out ref (candidate) to an optional baseline action ref (default v1.3.2). Jobs install firewall mode from ./ or a fetched baseline copy, run sfw npm install in a configurable public fixture (default expressjs/express), classify outcomes (setup failure, PATH, timeouts, libuv, telemetry, etc.), and upload JSON results. A Report job always runs, merges artifacts, writes a flake table to the step summary, and breaks out binary download failures from action-setup annotations via the GitHub API.

Uses the repo’s fleet pattern: inline bootstrap checkout (no actions/checkout), local upload-artifact / download-artifact composites, and continue-on-error so the run is measurement—not a gate.

^{Reviewed by Cursor Bugbot for commit d087b21. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Fleet artifact actions never hydrated
  • The install and report jobs now run the tracked fleet checkout composite after bootstrap so upload-artifact and download-artifact exist before those steps resolve.

Or push these changes by commenting:

@cursor push 08fd781737

Preview (08fd781737)

diff --git a/.github/workflows/test-sfw-ci-simulation.yml b/.github/workflows/test-sfw-ci-simulation.yml
--- a/.github/workflows/test-sfw-ci-simulation.yml
+++ b/.github/workflows/test-sfw-ci-simulation.yml
@@ -96,6 +96,14 @@
           export GIT_CONFIG_VALUE_0="AUTHORIZATION: basic ${AUTH_B64}"
           git fetch --no-tags --prune --depth 1 origin "${TRIGGER_REF}"
           git checkout -q --detach FETCH_HEAD
+      # Artifact composites are untracked fleet payload. Hydrate them here
+      # so upload-artifact resolves; checkout:false keeps this revision.
+      - name: 'Hydrate fleet Actions'
+        uses: ./.github/actions/fleet/checkout
+        with:
+          checkout: 'false'
+          payload-token-client-id: ${{ secrets.SOCKET_RELEASE_CLIENT_ID || vars.SOCKET_RELEASE_CLIENT_ID }}
+          payload-token-private-key: ${{ secrets.SOCKET_RELEASE_APP_PRIVATE_KEY }}
       # The baseline is a second copy of the action at the requested ref, in
       # its own directory so `./` stays the candidate.
       - name: 'Fetch the baseline action'
@@ -211,6 +219,14 @@
           export GIT_CONFIG_VALUE_0="AUTHORIZATION: basic ${AUTH_B64}"
           git fetch --no-tags --prune --depth 1 origin "${TRIGGER_REF}"
           git checkout -q --detach FETCH_HEAD
+      # Artifact composites are untracked fleet payload. Hydrate them here
+      # so download-artifact resolves; checkout:false keeps this revision.
+      - name: 'Hydrate fleet Actions'
+        uses: ./.github/actions/fleet/checkout
+        with:
+          checkout: 'false'
+          payload-token-client-id: ${{ secrets.SOCKET_RELEASE_CLIENT_ID || vars.SOCKET_RELEASE_CLIENT_ID }}
+          payload-token-private-key: ${{ secrets.SOCKET_RELEASE_APP_PRIVATE_KEY }}
       - name: 'Download the results'
         uses: ./.github/actions/fleet/download-artifact
         with:

_{You can send follow-ups to the cloud agent here.}

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit d087b21. Configure here.}

[cursor]

Fleet artifact actions never hydrated

High Severity

The install and report jobs call the fleet upload-artifact and download-artifact composites after an inline bootstrap only. Those action.yml files are gitignored payload and never land in a plain checkout, so every cell fails at upload and the report job cannot read results.

Additional Locations (1)

• .github/workflows/test-sfw-ci-simulation.yml#L213-L217

 

^{Reviewed by Cursor Bugbot for commit d087b21. Configure here.}