run-tests.sh reports STATUS: PASS with exit 0 when no tests ran (missing test/bats submodule) #95

Repo: pgxntool-test

Observed

In a freshly created git worktree of pgxntool-test, the test/bats git submodule is not initialized. make test-setup fails, but .claude/skills/test/scripts/run-tests.sh test-all printed STATUS: PASS and exited 0 without executing any tests. This was found while running the suite for an unrelated change, and was caught only because the real test counts were checked.

Why it matters

This is a false pass. Any automated or agent-driven run in a fresh worktree can silently report success with zero coverage.

Where the status is computed (run-tests.sh)

• L47-58: CMD is make <target> for a non-.bats target.
• L79-81: the make command runs inside a pipeline element subshell ({ $CMD 2>&1 || EXIT_CODE=$?; } | tee "$FULL_LOG" | while ...). The EXIT_CODE=$? assignment at L80 happens in that subshell, so the parent's EXIT_CODE stays 0.
• L146-149: the only correction is if [ "$FAILED" -gt 0 ] && [ "$EXIT_CODE" -eq 0 ]. When make fails before producing any TAP output, FAILED is 0, so the failure is never restored.
• L172-178: STATUS is FAIL only when FAILED > 0, PASS_WITH_SKIPS when SKIPPED > 0, otherwise PASS. With zero parsed tests this yields PASS.
• L197-218 and L220: the summary is written with STATUS: PASS and EXIT_CODE: 0, and the script ends with exit "$EXIT_CODE".

Nothing checks that TOTAL > 0 or that the make target succeeded.

Expected

• The runner exits non-zero, and reports a STATUS other than PASS, whenever the make/test command exits non-zero, regardless of parsed TAP counts.
• It also fails when the parsed output contains zero tests.
• The make exit code should be captured in the parent shell (for example via PIPESTATUS or a temp file), not inside the pipeline subshell.

Possible additional fix

Have make test-setup, or the worktree setup docs/command, initialize submodules (git submodule update --init --recursive), so a fresh worktree has test/bats available.

Related

• Test runner reports PASS with exit 0 when make fails before any TAP output pgxntool#139 reports the same runner bug from the cause side: the make exit status is lost in the pipeline subshell, so a make failure without a not ok line is reported as a pass. PR run-tests.sh (Claude /test skill): fail on non-zero exit, zero tests, or plan mismatch #106 fixes both.
• make pgtle exits 0 when pgtle.sh fails for any extension except the last pgxntool#140 is a separate bug of the same kind: make pgtle joins the per-extension pgtle.sh calls with ;, so only the last call's exit status counts. PR run-tests.sh (Claude /test skill): fail on non-zero exit, zero tests, or plan mismatch #106 does not address it.

Line references come from reading run-tests.sh. The failing run is taken from the observed report and was not re-executed for this issue.

🤖 Generated with Claude Code