Repository navigation
Dataset cache-key step silently succeeds when gin is unreachable #1896
Description
Summary
The Get ephy_testing_data current head hash step in IO / cache workflows stays green when git ls-remote against gin fails (e.g. HTTP 403), writing an empty dataset_hash and degrading the cache key to ${{ runner.os }}-datasets-.
Where
Same pattern in:
.github/workflows/io-test.yml(lines 41 to 45).github/workflows/caches_cron_job.yml(lines 70 to 73).github/workflows/plexon2-testing.yml(lines 32 to 36)
- name: Get ephy_testing_data current head hash
id: ephy_testing_data
run: |
echo "dataset_hash=$(git ls-remote https://gin.g-node.org/NeuralEnsemble/ephy_testing_data.git HEAD | cut -f1)" >> $GITHUB_OUTPUTMechanics (verified)
The step's exit status comes from echo, not from the command substitution. set -e alone does not catch it either, because git's failure is masked by cut succeeding. pipefail is required.
| Form | Exit code |
|---|---|
echo "x=$(git … | cut -f1)" >> $GITHUB_OUTPUT |
0 (silent) |
set -eu + variable assignment |
0 (still silent) |
set -euo pipefail + variable assignment |
128 (the step fails) |
Observed in run 30934750446: gin returned 403 on git ls-remote, the step still concluded success, the cache key became Linux-datasets-, and 95 IO tests then errored on datalad fetch with the same 403.
Proposed fix
- id: ephy_testing_data
run: |
set -o pipefail
if ! hash="$(git ls-remote https://gin.g-node.org/NeuralEnsemble/ephy_testing_data.git HEAD | cut -f1)" || [ -z "$hash" ]; then
echo "::error::could not resolve ephy_testing_data HEAD on gin; refusing to build a degenerate cache key"
exit 1
fi
echo "dataset_hash=$hash" >> "$GITHUB_OUTPUT"Keep id: ephy_testing_data and each workflow's existing step name: the cache steps read steps.ephy_testing_data.outputs.dataset_hash. The assignment sits in the if condition so that the ::error:: line is printed even when the step runs under bash -e (as in caches_cron_job.yml); with pipefail on, a bare assignment under set -e exits with git's code before reaching it.
Apply consistently in the three workflows listed above (or factor into a composite action later).
Activity
zm711 commentedon Aug 11, 2026
@h-mayorquin could you check this out. You're way better at GitHub actions than me.
AxelNoun commentedon Oct 7, 2026
I have opened #1912, which applies the corrected step to the three workflows.
I have also corrected the proposed fix in the issue body. The snippet I first posted here used the wrong step id (
dataset_hashinstead ofephy_testing_data), and, whengit ls-remotefailed, its ownset -euo pipefailmade the step exit with git's code before the::error::line was reached.The pattern recurs in the logs: counting the latest attempt of each run, 9 io-test runs created between 2026-07-13 and 2026-10-06 had a job with an empty key after a 403 on
git ls-remote(most recently on master on 2026-10-05), and 12 of the 80 cache cron runs created between 2026-07-26 and 2026-10-06 concluded green with an empty key. Details are in the PR. The 403 responses themselves are tracked in #1897; the PR only makes the failure explicit.h-mayorquin commentedon Oct 8, 2026
@zm711, #1912 looks good to me and I approved it. Once it merges I will delete the
Linux-datasets-cache entry that was saved by accident.I think a better plan is to resolve the hash once in a separate job and pass it to the test jobs with
needs. That makes onegit ls-remoterequest per run instead of one per job. If the hash can't be resolved, the test jobs would only restore the latest dataset cache throughrestore-keysand run on it, and onlycaches_cron_job.ymlwould fail so it never saves under an empty key. I can open that as a follow-up after this one merges. What do you think?CI: fail when the ephy_testing_data HEAD hash cannot be resolved
alejoe91 commentedon Oct 8, 2026
I removed the
Linux-datasets-cache entry! Thanks @AxelNoun