NAS-144436 / 28.0.0-BETA.1 / Add truenas-appliance-runner: boot the TrueNAS appliance a GARM runner's scale set attached, inside the runner by william-gr · Pull Request #21 · iXsystems/ux-github-workflows

william-gr · 2026-10-12T17:27:49Z

Adds .github/actions/truenas-appliance-runner, the second way to give an e2e job a TrueNAS appliance: the appliance is inside the runner.

The job runs on an ephemeral GARM runner from iXsystems/github-truenas-runner's TrueNAS provider. Its scale set attaches a copy-on-write clone of a prebuilt appliance image as a second disk. The action then:

boots that disk under qemu (UEFI, a qcow2 overlay, ten sparse data disks);
rotates the admin password to one made for this job, and masks it;
snapshots the running VM, memory included, as fresh-install;
exports the same TN_* variables as the lab claim action.

There is no claim and no release: picking the runner label is the claim, and GARM deleting the runner is the release. truenas-appliance goes on PATH with these commands: revert, snapshot, call, console, probe.

Proven end to end: webui's whole suite, 75 passed and 6 skipped, ran against it in truenas/webui run 38281104721. Numbers, the comparison with the lab, and what the first runs hit are in github-truenas-runner's docs/e2e-appliances.md.

Why here: webui is public and in another org, and a public repository's workflow cannot use an action from github-truenas-runner, which is private. These scripts are the one copy: github-truenas-runner's image build fetches them from this directory, at the last commit that changed it, to install and test-boot the appliance image.

Also in this PR:

The existing truenas-appliance lab claim action is untouched. webui moves off it in its own PR.
CI gains a job that runs tnctl's tests and shellcheck on the scripts.
README section.

Merge order: this first, then iXsystems/github-truenas-runner e2e-appliances and truenas/webui e2e-appliances, which both point at @master.

…r's pool attached, under qemu inside the runner

…nce TrueNAS replaces the serial; truenas-appliance on PATH before start

bugclerk · 2026-10-12T17:28:08Z

Jira URL: https://ixsystems.atlassian.net/browse/NAS-144436

github-actions · 2026-10-12T17:30:13Z

+        run: python3 test_tnctl.py
+
+      - name: shellcheck
+        run: shellcheck -x appliance.sh lib.sh


BLOCKER: The new job this PR adds, Check the appliance runner scripts, fails on this commit (run 38284369261, job 114929083739). Every push will stay red until it passes. I wasn't able to read the job log from this run, so the cause below comes from reading the scripts. The test fakes in test_tnctl.py look consistent with tnctl.py, so I'd look at shellcheck first. It exits non-zero on findings at every level, including info. From reading the code, it would report:

appliance.sh:48: . "$(dirname "$(realpath "$0")")/lib.sh" is a non-constant source, so -x can't follow it (SC1090). Then every name that comes from lib.sh (APPLIANCE_USER, TNCTL, OVMF_CODE, …) is reported as unassigned (SC2154). A # shellcheck source=lib.sh directive on the line above fixes both, because -x resolves the path relative to the working directory.

appliance.sh:240: running && … || true is SC2015 (info).

Please check this against the job log.

Fixed in 2c671e5. The job log showed shellcheck's SC2015 on the stop) line (appliance.sh:240) as the only finding: the runner's shellcheck is newer than the one I ran locally. It's now an if. Clean under shellcheck:latest.

github-actions · 2026-10-12T17:30:15Z

+# APPLIANCE_BOOT_SECONDS, APPLIANCE_SNAPSHOT_SECONDS.
+
+set -euo pipefail
+. "$(dirname "$(realpath "$0")")/lib.sh"


BLOCKER (same finding as the one on ci.yml:174): this is probably why the new shellcheck step fails. The source path isn't a constant, so shellcheck -x can't follow it (SC1090), and then every variable that lib.sh defines is reported as unassigned (SC2154).

Suggested change

. "$(dirname "$(realpath "$0")")/lib.sh"

# shellcheck source=lib.sh

. "$(dirname "$(realpath "$0")")/lib.sh"

Added the # shellcheck source=lib.sh directive in 2c671e5. The job log didn't show SC1090 or SC2154 (the failure was SC2015 alone, see the ci.yml thread), but the directive makes the check not depend on how -x resolves the path.

github-actions · 2026-10-12T17:30:17Z

+| `memory-mb`, `vcpus` | `6144`, `4` | The appliance's; the runner's flavor needs room for it and the job |
+| `data-disk-count`, `data-disk-gb` | `10`, `10` | Sparse qcow2 files |
+| `ssh` | `'false'` | `true` enables ssh for the admin, on `TN_SSH` |
+| `disk` | the attached one | `/dev/disk/by-id/virtio-truenas_appliance` by default |


LOW: This row is out of date after 1ba3df9. attached_disk only uses /dev/disk/by-id/virtio-truenas_appliance when that serial survived. Otherwise it picks the one disk that holds a boot-pool and isn't the root disk, which is the case the commit says TrueNAS normally hits. action.yml already says "found by its boot-pool". This row should say the same.

Fixed in 2c671e5: the row says the serial is used where the box keeps it, and otherwise the one non-root disk with a boot-pool.

github-actions · 2026-10-12T17:30:20Z

+          echo "snapshot-seconds=$(get APPLIANCE_SNAPSHOT_SECONDS)"
+        } >> "$GITHUB_OUTPUT"
+        if [ "$EXPORT_ENV" = true ]; then
+          grep '^TN_' <<<"$out" >> "$GITHUB_ENV"


LOW: grep '^TN_' exports TN_SSH=127.0.0.1:2222 to the job environment even when ssh is false. The ssh output is empty in that case (line 119), so the output and the env disagree. A suite that uses TN_SSH being set to decide whether ssh is available would try a port nobody is listening on. Also, the README says the action "exports the same TN_* variables as the claim above", but TN_GUEST_ISO, which the lab action exports, isn't here.

Fixed in 2c671e5: appliance.sh prints TN_SSH only when APPLIANCE_SSH is true, so the env and the ssh output agree. The README now says the variables are the claim's less TN_GUEST_ISO (an image, not an ISO), plus TN_SSH when ssh is on.

github-actions · 2026-10-12T17:30:28Z

Two findings: both LOW. No human review needed.

I reviewed the whole PR range (gh pr diff). It adds the new truenas-appliance-runner composite action and its scripts, a CI job for them, and a README section. No existing workflow or action changes, so no current caller is affected until it opts into the new action.

Checked and correct:

Errors exit non-zero. start runs under set -euo pipefail and is called as out="$(appliance.sh start)", so a failed boot, password rotation or savevm fails the step. revert turns an HMP error into a non-zero exit, and the new test covers it.
Password handling. The password is masked before anything can print it, and it is saved in the state file before the baseline snapshot. That means revert brings back an appliance with the rotated password.
Empty baseline input. It really does skip the snapshot, because the script reads ${APPLIANCE_BASELINE-fresh-install} without the colon.
README matches the action. Its inputs, defaults and outputs table agrees with action.yml.
New CI job. It inherits the workflow's top-level permissions: and pins actions/checkout@v4 like the other jobs.

Findings (inline):

LOW — tnctl.py:362: the check "rror" in out makes truenas-appliance snapshots exit 1 when any snapshot name contains "rror".
LOW — tnctl.py:275: cmd_wait retries system.ready on a single connection. If that connection drops, it keeps retrying a dead socket until the boot timeout instead of logging in again.

github-actions

Changes requested. 4 finding(s): 2 BLOCKER, 2 LOW.

Findings at MEDIUM and above are on the lines they are about. Fix them, or say on the PR why one was mis-rated.

…ly when ssh is on, README on how the disk is found

github-actions · 2026-10-12T17:43:50Z

+                f.write(b'{"event": "STOP"}\n')
+                out = {"return": {}}
+                if msg["execute"] == "human-monitor-command":
+                    out = {"return": "Error: no snapshot" if "nope" in msg["arguments"]["command-line"] else ""}


LOW — The fake QMP server can answer "Error: no snapshot" for a command line containing nope, but no test sends one. The test calls tnctl.qmp directly, never cmd_hmp. So nothing covers the "rror" in out → exit 1 path, and revert relies on that path to report a missing snapshot. A cmd_hmp(SimpleNamespace(socket=path, line="loadvm nope")) wrapped in assertRaises(SystemExit) would cover it. Note that the server thread exits after its first connection, so it would need to accept in a loop.

github-actions

Approved. 1 finding(s): 1 LOW.

Nothing blocks and no human review is needed.

…h revert relies on

github-actions · 2026-10-12T17:47:28Z

+    # HMP reports its own errors as text, savevm and loadvm included.
+    if out and out.strip():
+        print(out.strip())
+        if "rror" in out:


LOW — "rror" in out also matches ordinary output, not only errors. savevm and loadvm print nothing when they succeed, so they're fine. info snapshots is different: it lists the snapshot names. Take truenas-appliance snapshot pre-error-dialog followed by truenas-appliance snapshots. The listing prints and the command still exits 1. Matching only lines that start with Error: would keep the revert guard and drop that false failure.

github-actions · 2026-10-12T17:47:31Z

+            raise ValueError("not ready yet")
+
+    try:
+        retry("system.ready", max(10, a.timeout - (time.monotonic() - start)), ready, every=2)


LOW — ready() reuses the one connection login opened, and retry treats Closed/OSError as worth retrying. If the middleware drops that websocket after the login and before system.ready turns true, every later attempt fails on the dead socket. start then waits out the whole APPLIANCE_BOOT_TIMEOUT (900 s) and reports that the API never came up. Logging in again inside ready when the socket has closed would recover from this.

github-actions

Approved. 2 finding(s): 2 LOW.

Nothing blocks and no human review is needed.

bugclerk · 2026-10-12T17:53:31Z

This PR has been merged and conversations have been locked.
If you would like to discuss more about this issue please use our forums or raise a Jira ticket.

william-gr added 3 commits October 12, 2026 10:13

Add truenas-appliance-runner: boot the TrueNAS appliance a GARM runne…

7bba3ed

…r's pool attached, under qemu inside the runner

Ignore __pycache__ from running the appliance scripts' tests

91c6ebe

truenas-appliance-runner: find the attached disk by its boot-pool, si…

1ba3df9

…nce TrueNAS replaces the serial; truenas-appliance on PATH before start

william-gr added the jira label Oct 12, 2026

bugclerk changed the title ~~Add truenas-appliance-runner: boot the TrueNAS appliance a GARM runner's scale set attached, inside the runner~~ NAS-144436 / 28.0.0-BETA.1 / Add truenas-appliance-runner: boot the TrueNAS appliance a GARM runner's scale set attached, inside the runner Oct 12, 2026

william-gr mentioned this pull request Oct 12, 2026

NAS-144437 / 28.0.0-BETA.1 / Run E2E on runner-attached appliances: each shard a GARM runner booting its own TrueNAS truenas/webui#14306

Merged

github-actions Bot reviewed Oct 12, 2026

View reviewed changes

github-actions Bot requested changes Oct 12, 2026

View reviewed changes

truenas-appliance-runner: shellcheck-clean stop and source, TN_SSH on…

2c671e5

…ly when ssh is on, README on how the disk is found

github-actions Bot reviewed Oct 12, 2026

View reviewed changes

github-actions Bot approved these changes Oct 12, 2026

View reviewed changes

truenas-appliance-runner: test that an HMP error exits non-zero, whic…

10f9262

…h revert relies on

github-actions Bot reviewed Oct 12, 2026

View reviewed changes

github-actions Bot approved these changes Oct 12, 2026

View reviewed changes

william-gr merged commit 441426f into master Oct 12, 2026
8 checks passed

william-gr deleted the e2e-appliances branch October 12, 2026 17:53

bugclerk added the no-time-tracked label Oct 12, 2026

iXsystems locked as resolved and limited conversation to collaborators Oct 12, 2026

	. "$(dirname "$(realpath "$0")")/lib.sh"
	# shellcheck source=lib.sh
	. "$(dirname "$(realpath "$0")")/lib.sh"

Conversation

william-gr commented Oct 12, 2026

Uh oh!

bugclerk commented Oct 12, 2026

Uh oh!

github-actions Bot Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

william-gr Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

william-gr Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

william-gr Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

william-gr Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bugclerk commented Oct 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Oct 12, 2026 •

edited

Loading