Add option for bundle script to exit non-zero on failure

[andrewmogan]

Description

With this change, providing --raise-failure to the bundle script will cause the script to exit 1. This will be useful for the nightly integration test workflows, where we want the GitHub Actions output to show an obvious red X for failed tests. If the script exits zero, the GitHub Actions output shows a counterintuitive green checkmark. See, for example, this test run where two tests failed but there are no obvious indications of this beyond the summary table.

To test, I manually sabotaged minimal_system_quick_test.py by setting "max_size_bytes": 10472, (instead of 14472) and ran

dunedaq_integtest_bundle.sh -k min --raise-failure
echo $?

For comparison, you can not provide --raise-failure and see what echo $? returns 0.

Type of change

• Documentation (non-breaking change that adds or improves the documentation)
• New feature or enhancement (non-breaking change which adds functionality)
• Optimization (non-breaking change that improves code/performance)
• Bug fix (non-breaking change which fixes an issue)
• Breaking change (whatever its nature)

[andrewmogan]

Pushed a fix that actually implements the functionality that I apparently failed to push on Friday.

[andrewmogan]

After discussing with Kurt, we decided to make the behavior of exiting non-zero on test failure the default. I removed the --raise-failure argument but kept the bit that makes the script exit non-zero when one or more tests fail.

[bieryAtFnal]

This looks good to me. I ran tests of the following commands:

dunedaq_integtest_bundle.sh -k min --verb 2; echo $?
dunedaq_integtest_bundle.sh -k 1df --verb 2 --pytest-options "-k TPG"; echo $?
dunedaq_integtest_bundle.sh -k 'min|1df|small' --verb 2; echo $?
dunedaq_integtest_bundle.sh -r hsilibs -k 'iceberg' --verb 2; echo $?
dunedaq_integtest_bundle.sh -r drunc -k 'failure' --verb 2; echo $?

The tests with the 3ru_1df integtest had a non-zero exit code, as expected, because that integtest is currently showing problems. The other tests had exit codes of zero.

Just to confirm, we expect an exit code of zero when all tests are skipped, correct?

[andrewmogan]

Yes, I think exiting 0 when all tests are skipped is what we want, since it doesn't indicate a true test failure.

[bieryAtFnal]

Sounds good to me. Thanks.