feat(stats): report testrun stats to the tcache

[OlufemiAdeOlusile]

What it does

A test run now reports its test counts to the tcache.

The tcache gained a /stats endpoint. Nothing called it. This adds the client
half.

File	Purpose
scripts/stats_json.py	Write the statistics document.
runner/report_stats.sh	Upload it.

It is wired into both test scripts that produce Allure results:

Script	Rows per run
runner/regression.sh	one, step=main
runner/node_upgrade_pytest.sh	three, step1, step2, step3

How it works

scripts/stats_json.py does not count the tests itself.
count_test_results.py already groups Allure result files by historyId, so
the --skipall registration pass of run_tests.sh is not counted twice. This
script reuses that logic. It adds only the fields a JUnit report cannot carry:
the run identity, the timing, the software versions, and the CLI invocation and
submitted-transaction counts from cli_coverage.json.

runner/report_stats.sh runs after create_results.sh, which is what creates
the Allure directory. Every call is || :, so a failed upload cannot mask the
pytest exit code. curl has a 60 second limit, so a tcache that does not answer
cannot hold up the testrun.

How each run is named

Each workflow's runner/env_* file now sets TCACHE_TESTRUN_NAME: nightly,
nightly-cli, nightly-dbsync, nightly-upgrade, upgrade, regression and
regression-dbsync. A name typed for a manual run (CI_TESTRUN_NAME) still
wins, and that run is marked "release": true. Without this, every CI run
except a typed one arrived as local.

The run id is GITHUB_RUN_ID. GITHUB_RUN_NUMBER counts per workflow, so the
same typed name in regression.yaml and regression-dbsync.yaml could share
one.

The nightly, nightly-cli and nightly-dbsync workflows do not pass the tcache
secrets yet, so their names take effect once they do (#3624).

Turning it on

The upload is off by default. It runs only when ENABLE_TCACHE_STATS is
truthy, checked with is_truthy like the other ENABLE_* flags. In CI, set
the ENABLE_TCACHE_STATS repository variable to true. Locally, export it.
The tcache secrets alone do not turn it on, because the pypassed and
/import steps use the same secrets.

The document is about 660 bytes:

{
  "schema": 1,
  "project": "cardano-node-tests",
  "testrun_name": "node-1050",
  "run_id": "18562938417",
  "step": "main",
  "origin": "ci",
  "timestamp": "2026-05-30T23:39:29.232000+00:00",
  "duration": 4527.316,
  "exit_code": 0,
  "filtered": false,
  "release": true,
  "counts": {"total": 2145, "passed": 1892, "failed": 0, "broken": 0, "skipped": 253},
  "quality": {"never_run": 0, "no_history_id": 0, "read_errors": 0},
  "versions": {"cardano_node": "10.5.0", "cardano_cli": "10.11.1.0", "db_sync": null},
  "commands": {"count": 213130, "submits": 9990, "coverage_pct": 31.0}
}

Auth

It reuses TCACHE_BASIC_AUTH. No new secret is needed.

TCACHE_URL points at the /results prefix, so the script replaces that
suffix with /stats. The nightly /history upload does the same.

The secrets were not in scope for the steps that run the test scripts.
They were set only on the steps that call the tcache with curl directly.
Without this the upload would have skipped silently and recorded nothing.
upgrade_reusable.yaml did not even declare them, so no caller could pass
them. Fixed in regression_reusable.yaml, upgrade_reusable.yaml,
nightly_upgrade.yaml and upgrade.yaml.

Why the upgrade path is included

It was going to be left out, because you scoped this to regression.sh and
the non-upgrade scripts. That reason was that node_upgrade_pytest.sh emits
no JUnit XML. Still true, but the counts come from Allure now, and that script
calls create_results.sh three times. The step column already exists for
this. Say the word and I will drop it.

No CLI coverage is sent for the upgrade steps. cli_coverage.sh runs in the
finish step and covers the whole run, so attributing it to one step would
misreport it.

Details that are easy to get wrong

• Duration is the wall clock span of the real results. Tests run in
  parallel under xdist, so the sum of the per-test durations is many times the
  elapsed time.
• never_run means the run was interrupted. It counts tests the
  registration pass registered that never got a real result, so every count is
  a floor. It is a subset of skipped.
• The testrun name keeps only ASCII letters, digits, _ and -, the
  same set the workflows keep, but without depending on the locale. It drops
  dots, so node-10.5.0 becomes node-1050. That is lossy, and the tcache
  itself accepts dots. Matching the existing calls matters more, because a
  different name here could not be joined to the /import rows. Happy to
  change this if you prefer the dots kept everywhere.
• A partial run is marked filtered when MARKEXPR or
  DESELECT_FROM_FILE is set. A skip-passed rerun sets the second one.

Testing

No unit tests are added, the same as for count_test_results.py. All
framework tests pass. No new ruff, shellcheck, actionlint or zizmor
findings. scripts/stats_json.py is added to the mypy hook, which did not
check it, and it passes.

Verified against a real Allure directory of 4269 result files. The counts match
count_test_results.py exactly: 2145 total, 1892 passed, 253 skipped. The
document was uploaded through report_stats.sh to a real tcache under
gunicorn and read back unchanged. The three upgrade steps were checked too:
one run id holding step1, step2 and step3 with exit codes 0, 1 and 2.

The failure cases were run through the same calling pattern as
regression.sh: flag off, secrets missing, tcache down, a wrong password,
tcache returning 404 or 500, and a tcache that accepts the connection but
never answers. In every case the caller kept its exit code, and curl now
prints why the upload failed.

Each env file was loaded the way GitHub loads it and uploaded to a local
tcache. Every workflow stored its own name, the typed release name won and was
marked as a release, and DESELECT_FROM_FILE marked the run as filtered.

Depends on

The server side, mkoura/testing-results-cache#25. Because the upload is off by
default, this can merge first. Turn the flag on after /stats is deployed.

🤖 Generated with Claude Code