Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
126 changes: 126 additions & 0 deletions .github/actions/truenas-appliance-runner/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
name: TrueNAS appliance
description: >-
Boot the TrueNAS appliance attached to this runner under qemu, and hand
the job its address and a credential of its own. The appliance lives
and dies with the runner, so there is nothing to release.

# The runner comes from a GARM pool on a TrueNAS box whose extra specs
# attach the appliance's disk:
#
# {"disks": ["truenas_appliance:nightly"]}
#
# and whose flavor has room for it beside the job (the appliance's memory,
# 6 GB by default, plus the browser and the build). The pool, the provider
# and the appliance image are iXsystems/github-truenas-runner's (its
# docs/e2e-appliances.md).
#
# These scripts are the one copy: that repository's image build fetches
# them from here, at the last commit that changed this directory, to
# install the appliance image and test-boot it. A change here is a new
# appliance image there. What the two sides share is in lib.sh.
#
# Usage:
# - uses: iXsystems/ux-github-workflows/.github/actions/truenas-appliance-runner@master
# id: appliance
# - run: curl -sk "https://$TN_HOST/api/v2.0/system/info" -u "$TN_USERNAME:$TN_PASSWORD"
# # between tests, back to the baseline in seconds:
# - run: truenas-appliance revert fresh-install
#
# The TN_* variables are the e2e suite's contract, as the lab's claim
# action sets them; truenas-appliance (appliance.sh) is on PATH for
# snapshot, revert, probe, call and console.

inputs:
baseline:
description: Snapshot taken once the appliance is up, to revert to between tests. Empty takes none.
default: fresh-install
memory-mb:
description: Appliance memory.
default: '6144'
vcpus:
description: Appliance vCPUs.
default: '4'
data-disk-count:
description: Data disks, sparse.
default: '10'
data-disk-gb:
description: Size of each data disk.
default: '10'
ssh:
description: '`true` enables ssh for the admin, on TN_SSH.'
default: 'false'
disk:
description: The appliance disk, if not the one the pool attached (found by its boot-pool).
default: ''
export-env:
description: '`true` writes TN_* to the job environment, for suites that read them from there.'
default: 'true'

outputs:
host:
description: host:port of the appliance over https.
value: ${{ steps.start.outputs.host }}
host-http:
description: host:port of the appliance over http.
value: ${{ steps.start.outputs.host-http }}
username:
description: Admin user.
value: ${{ steps.start.outputs.username }}
password:
description: Admin password, made for this job and masked.
value: ${{ steps.start.outputs.password }}
ssh:
description: host:port of its ssh, when asked for.
value: ${{ steps.start.outputs.ssh }}
boot-seconds:
description: From qemu starting to the API taking a login.
value: ${{ steps.start.outputs.boot-seconds }}
snapshot-seconds:
description: What taking the baseline took.
value: ${{ steps.start.outputs.snapshot-seconds }}

runs:
using: composite
steps:
- id: start
shell: bash
env:
APPLIANCE_BASELINE: ${{ inputs.baseline }}
APPLIANCE_MEMORY_MB: ${{ inputs.memory-mb }}
APPLIANCE_VCPUS: ${{ inputs.vcpus }}
APPLIANCE_DATA_DISKS: ${{ inputs.data-disk-count }}
APPLIANCE_DATA_DISK_GB: ${{ inputs.data-disk-gb }}
APPLIANCE_SSH: ${{ inputs.ssh }}
APPLIANCE_DISK_INPUT: ${{ inputs.disk }}
EXPORT_ENV: ${{ inputs.export-env }}
run: |
set -euo pipefail
lib="$GITHUB_ACTION_PATH"
[ -n "$APPLIANCE_DISK_INPUT" ] && export APPLIANCE_DISK="$APPLIANCE_DISK_INPUT"
export APPLIANCE_DIR="$RUNNER_TEMP/truenas-appliance"
# The password is made here, so it is masked before anything can
# print it.
APPLIANCE_PASSWORD="$(head -c 15 /dev/urandom | base64 | tr -d '/+=')"
echo "::add-mask::$APPLIANCE_PASSWORD"
export APPLIANCE_PASSWORD
# On PATH before it starts, so that a failed start's console is
# there for a later step to show.
echo "APPLIANCE_DIR=$APPLIANCE_DIR" >> "$GITHUB_ENV"
mkdir -p "$RUNNER_TEMP/truenas-appliance-bin"
ln -sf "$lib/appliance.sh" "$RUNNER_TEMP/truenas-appliance-bin/truenas-appliance"
echo "$RUNNER_TEMP/truenas-appliance-bin" >> "$GITHUB_PATH"
out="$("$lib/appliance.sh" start)"
get() { sed -n "s/^$1=//p" <<<"$out"; }
{
echo "host=$(get TN_HOST)"
echo "host-http=$(get TN_HOST_HTTP)"
echo "username=$(get TN_USERNAME)"
echo "password=$(get TN_PASSWORD)"
echo "ssh=$(get TN_SSH)"
echo "boot-seconds=$(get APPLIANCE_BOOT_SECONDS)"
echo "snapshot-seconds=$(get APPLIANCE_SNAPSHOT_SECONDS)"
} >> "$GITHUB_OUTPUT"
if [ "$EXPORT_ENV" = true ]; then
grep '^TN_' <<<"$out" >> "$GITHUB_ENV"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LOW: grep '^TN_' exports TN_SSH=127.0.0.1:2222 to the job environment even when ssh is false. The ssh output is empty in that case (line 119), so the output and the env disagree. A suite that uses TN_SSH being set to decide whether ssh is available would try a port nobody is listening on. Also, the README says the action "exports the same TN_* variables as the claim above", but TN_GUEST_ISO, which the lab action exports, isn't here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2c671e5: appliance.sh prints TN_SSH only when APPLIANCE_SSH is true, so the env and the ssh output agree. The README now says the variables are the claim's less TN_GUEST_ISO (an image, not an ISO), plus TN_SSH when ssh is on.

fi
echo "Appliance $(get TN_DOMAIN) at $(get TN_HOST): API up in $(get APPLIANCE_BOOT_SECONDS)s, baseline '${APPLIANCE_BASELINE:-none}' in $(get APPLIANCE_SNAPSHOT_SECONDS)s"
248 changes: 248 additions & 0 deletions .github/actions/truenas-appliance-runner/appliance.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,248 @@
#!/usr/bin/env bash
# A TrueNAS appliance for this job, booted under qemu inside the runner.
#
# The runner's pool attaches the appliance's disk ("disks" in the pool's
# extra specs: a clone of truenas_appliance:<channel> on the box), and the
# appliance lives and dies with the runner: nothing to claim from anyone,
# nothing to release, nothing to leak. See iXsystems/github-truenas-runner's
# docs/e2e-appliances.md.
#
# appliance.sh start boot it, wait for its API, give it this
# claim's admin password, and snapshot it
# as the baseline; KEY=value on stdout
# appliance.sh snapshot <name> save the running appliance, memory and
# disks, as <name>
# appliance.sh revert <name> back to <name>, in seconds, no boot
# appliance.sh probe [baseline] change it, revert, check the change went
# appliance.sh call <method> [params...]
# a middleware API call, as the admin, its
# result as JSON on stdout
# appliance.sh snapshots the names there are
# appliance.sh console [lines] the end of its serial console
# appliance.sh stop
#
# Settings, environment variables, all optional:
#
# APPLIANCE_DISK the appliance's disk: a device or a qcow2/raw file
# (default: the disk the pool attached, found by
# attached_disk)
# APPLIANCE_DIR its state: overlay, data disks, sockets, console
# (default $RUNNER_TEMP/truenas-appliance)
# APPLIANCE_MEMORY_MB 6144 APPLIANCE_VCPUS 4
# APPLIANCE_DATA_DISKS 10 APPLIANCE_DATA_DISK_GB 10 (sparse)
# APPLIANCE_HTTP_PORT 8080 APPLIANCE_HTTPS_PORT 8443
# APPLIANCE_SSH_PORT 2222 APPLIANCE_SSH false
# APPLIANCE_BASELINE fresh-install: the snapshot start takes; empty
# takes none
# APPLIANCE_PASSWORD the claim's admin password (default: generated)
# APPLIANCE_BOOT_TIMEOUT 900 seconds
# APPLIANCE_ACCEL kvm; tcg runs without KVM, too slowly for
# anything but testing this script
#
# start prints TN_PROFILE, TN_HOST (host:https port), TN_HOST_HTTP,
# TN_USERNAME, TN_PASSWORD, TN_DOMAIN, TN_BASELINE (the suite's contract,
# as the lab's claim action gives it, less its TN_GUEST_ISO), TN_SSH when
# APPLIANCE_SSH is true, and what it measured:
# APPLIANCE_BOOT_SECONDS, APPLIANCE_SNAPSHOT_SECONDS.

set -euo pipefail
# shellcheck source=lib.sh
. "$(dirname "$(realpath "$0")")/lib.sh"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BLOCKER (same finding as the one on ci.yml:174): this is probably why the new shellcheck step fails. The source path isn't a constant, so shellcheck -x can't follow it (SC1090), and then every variable that lib.sh defines is reported as unassigned (SC2154).

Suggested change
. "$(dirname "$(realpath "$0")")/lib.sh"
# shellcheck source=lib.sh
. "$(dirname "$(realpath "$0")")/lib.sh"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the # shellcheck source=lib.sh directive in 2c671e5. The job log didn't show SC1090 or SC2154 (the failure was SC2015 alone, see the ci.yml thread), but the directive makes the check not depend on how -x resolves the path.


DIR="${APPLIANCE_DIR:-${RUNNER_TEMP:-/tmp}/truenas-appliance}"
QMP="$DIR/qmp.sock"
STATE="$DIR/state.env"

hmp() { "${TNCTL[@]}" hmp "$QMP" "$*"; }
# resume: a failed loadvm leaves the VM paused, and later loadvms leave it
# so, however they go; cont is a no-op on one that runs.
resume() { "${TNCTL[@]}" qmp "$QMP" cont >/dev/null; }
url() { echo "ws://127.0.0.1:${APPLIANCE_HTTP_PORT:-8080}"; }
running() { [ -f "$DIR/qemu.pid" ] && kill -0 "$(cat "$DIR/qemu.pid")" 2>/dev/null; }
# shellcheck source=/dev/null
state() { [ -r "$STATE" ] || die "no appliance started here ($STATE)"; . "$STATE"; }

# attached_disk: the appliance disk the runner's pool attached. By its
# serial when the box kept the one the provider gave it; TrueNAS gives
# every VM disk a random serial of its own instead, so otherwise by what is
# on it: the one disk, not the one / is on, that holds a ZFS pool named
# boot-pool (a TrueNAS install).
attached_disk() {
local serial=/dev/disk/by-id/virtio-truenas_appliance
[ -e "$serial" ] && { realpath "$serial"; return; }
lsblk -J -p -o NAME,PKNAME,FSTYPE,LABEL,MOUNTPOINT | python3 -c '
import json, sys
devs = []
def walk(ds):
for d in ds:
devs.append(d)
walk(d.get("children") or [])
walk(json.load(sys.stdin)["blockdevices"])
root = {d["pkname"] for d in devs if d.get("mountpoint") == "/"}
found = sorted({d["pkname"] for d in devs
if d.get("fstype") == "zfs_member" and d.get("label") == "boot-pool"
and d.get("pkname") and d["pkname"] not in root})
if len(found) != 1:
sys.exit("found %d disks with a TrueNAS boot-pool: %s" % (len(found), " ".join(found) or "none"))
print(found[0])
'
}

start() {
local disk="${APPLIANCE_DISK:-}"
local memory="${APPLIANCE_MEMORY_MB:-6144}" vcpus="${APPLIANCE_VCPUS:-4}"
local ndata="${APPLIANCE_DATA_DISKS:-10}" datagb="${APPLIANCE_DATA_DISK_GB:-10}"
local http="${APPLIANCE_HTTP_PORT:-8080}" https="${APPLIANCE_HTTPS_PORT:-8443}" sshp="${APPLIANCE_SSH_PORT:-2222}"
local baseline="${APPLIANCE_BASELINE-fresh-install}"
local password="${APPLIANCE_PASSWORD:-$(head -c 15 /dev/urandom | base64 | tr -d '/+=')}"
local name="appliance-${GITHUB_RUN_ID:-local}-${GITHUB_RUN_ATTEMPT:-1}-${GITHUB_JOB:-job}"

running && die "an appliance is running already ($DIR)"
needs qemu-system-x86_64 qemu-system-x86 qemu-img qemu-utils python3 python3
ovmf
local accel="${APPLIANCE_ACCEL:-kvm}" cpu=host
if [ "$accel" = kvm ]; then
[ -w /dev/kvm ] || die "no /dev/kvm this user can use: the runner needs nested virtualization (cpu_mode HOST-PASSTHROUGH) and to be in the kvm group"
else
cpu=max
fi
if [ -z "$disk" ] && ! disk="$(attached_disk)"; then
echo "appliance: no appliance disk attached to this runner. Its disks:" >&2
lsblk -p -o NAME,SIZE,SERIAL,FSTYPE,LABEL,MOUNTPOINT >&2 || true
die "give the runner's pool {\"disks\": [\"truenas_appliance:nightly\"]} in its extra specs, or set APPLIANCE_DISK"
fi
[ -e "$disk" ] || die "no appliance disk at $disk"
disk="$(realpath "$disk")"
# Read only: the appliance writes to an overlay of it.
if [ ! -r "$disk" ]; then
sudo -n chown "$(id -u)" "$disk" || die "$disk is not readable here, and sudo cannot change that"
fi
local format=raw
[ -f "$disk" ] && format="$(qemu-img info --output=json "$disk" | python3 -c 'import json,sys; print(json.load(sys.stdin)["format"])')"

rm -rf "$DIR"
mkdir -p "$DIR"
qemu-img create -q -f qcow2 -F "$format" -b "$disk" "$DIR/os.qcow2"
vars_qcow2 "$DIR/vars.qcow2"
local drives=() i
for i in $(seq 1 "$ndata"); do
qemu-img create -q -f qcow2 "$DIR/data$i.qcow2" "${datagb}G"
drives+=(-drive "file=$DIR/data$i.qcow2,if=none,id=data$i,format=qcow2,discard=unmap"
-device "virtio-blk-pci,drive=data$i,serial=data$i")
done

local booted
say "booting $disk as $name: ${memory} MB, $vcpus vCPUs, $ndata data disks of ${datagb}G"
qemu-system-x86_64 -name "$name" -machine "q35,accel=$accel" -cpu "$cpu" -smp "$vcpus" -m "$memory" \
-smbios "type=1,serial=$name" \
-drive "if=pflash,format=raw,readonly=on,file=$OVMF_CODE" \
-drive "if=pflash,format=qcow2,file=$DIR/vars.qcow2" \
-drive "file=$DIR/os.qcow2,if=none,id=os,format=qcow2,discard=unmap" \
-device "virtio-blk-pci,drive=os,serial=os,bootindex=0" \
"${drives[@]}" \
-nic "user,model=virtio-net-pci$(hostfwd 80 "$http")$(hostfwd 443 "$https")$(hostfwd 22 "$sshp")" \
-device virtio-rng-pci \
-qmp "unix:$QMP,server=on,wait=off" \
-display none -serial "file:$DIR/console.log" \
-daemonize -pidfile "$DIR/qemu.pid"

if ! booted=$("${TNCTL[@]}" wait "$(url)" "$APPLIANCE_USER" "$APPLIANCE_TEMPLATE_PASSWORD" \
--timeout "${APPLIANCE_BOOT_TIMEOUT:-900}"); then
say "--- its console, last 60 lines"
tail -n 60 "$DIR/console.log" >&2 || true
die "the appliance's API never came up"
fi
say "API up ${booted}s after qemu started"
"${TNCTL[@]}" rotate "$(url)" "$APPLIANCE_USER" "$APPLIANCE_TEMPLATE_PASSWORD" "$password"
if [ "${APPLIANCE_SSH:-false}" = true ]; then
"${TNCTL[@]}" enable-ssh "$(url)" "$APPLIANCE_USER" "$password"
say "ssh on 127.0.0.1:$sshp"
fi
{
echo "APPLIANCE_PASSWORD_NOW=$(printf %q "$password")"
echo "APPLIANCE_HTTP_PORT=$http"
} > "$STATE"

local snapped=""
if [ -n "$baseline" ]; then
local s0
s0=$(date +%s.%N)
hmp savevm "$baseline" >&2
snapped=$(awk -v a="$s0" -v b="$(date +%s.%N)" 'BEGIN {printf "%.1f", b - a}')
say "snapshot $baseline taken in ${snapped}s"
fi

cat <<EOF
TN_PROFILE=shipped
TN_HOST=127.0.0.1:$https
TN_HOST_HTTP=127.0.0.1:$http
TN_USERNAME=$APPLIANCE_USER
TN_PASSWORD=$password
TN_DOMAIN=$name
TN_BASELINE=${baseline:-none}
APPLIANCE_BOOT_SECONDS=$booted
APPLIANCE_SNAPSHOT_SECONDS=$snapped
EOF
# Only when it is there to use: a suite may take TN_SSH being set to
# mean it can ssh in.
if [ "${APPLIANCE_SSH:-false}" = true ]; then
echo "TN_SSH=127.0.0.1:$sshp"
fi
}

snapshot() {
local name="${1:?snapshot <name>}"
running || die "no appliance running"
local rc=0
hmp savevm "$name" >&2 || rc=$?
resume
return "$rc"
}

# revert: loadvm restores memory and disks as they were, so the API is up
# the moment it returns; its connections from before are gone, and its
# clock is the snapshot's until it catches up.
revert() {
local name="${1:?revert <name>}"
running || die "no appliance running"
state
local t0
t0=$(date +%s.%N)
if ! hmp loadvm "$name" >&2; then
resume
die "no snapshot $name to revert to; the appliance runs on as it was"
fi
resume
"${TNCTL[@]}" wait "$(url)" "$APPLIANCE_USER" "$APPLIANCE_PASSWORD_NOW" --timeout 120 >/dev/null
awk -v a="$t0" -v b="$(date +%s.%N)" 'BEGIN {printf "%.1f\n", b - a}'
}

# probe <baseline>: that revert works and what it costs. A group is made
# over the API, the appliance reverted to <baseline>, and the group must
# be gone; KEY=value on stdout.
probe() {
local baseline="${1:-fresh-install}" took left
state
call() { "${TNCTL[@]}" call "$(url)" "$APPLIANCE_USER" "$APPLIANCE_PASSWORD_NOW" "$@"; }
call group.create '{"name": "e2e_probe"}' >/dev/null
[ "$(call group.query '[["name", "=", "e2e_probe"]]' | python3 -c 'import json,sys; print(len(json.load(sys.stdin)))')" = 1 ] \
|| die "probe: the group made over the API is not there"
took="$(revert "$baseline")"
left="$(call group.query '[["name", "=", "e2e_probe"]]' | python3 -c 'import json,sys; print(len(json.load(sys.stdin)))')"
[ "$left" = 0 ] || die "probe: after reverting to $baseline the group is still there"
say "revert to $baseline took ${took}s, and undid the change"
echo "APPLIANCE_REVERT_SECONDS=$took"
}

case "${1:-}" in
start) start ;;
call) shift; [ $# -gt 0 ] || die "call <method> [params...]"; state
"${TNCTL[@]}" call "$(url)" "$APPLIANCE_USER" "$APPLIANCE_PASSWORD_NOW" "$@" ;;
probe) shift; probe "$@" ;;
snapshot) shift; snapshot "$@" ;;
revert) shift; revert "$@" ;;
snapshots) running || die "no appliance running"; hmp info snapshots ;;
console) tail -n "${2:-100}" "$DIR/console.log" ;;
stop) if running; then "${TNCTL[@]}" qmp "$QMP" quit >/dev/null || true; fi ;;
*) die "usage: appliance.sh start | snapshot <name> | revert <name> | probe [baseline] | call <method> [params...] | snapshots | console [lines] | stop" ;;
esac
Loading
Loading