Repository navigation
fix(circleci): upload every shard's JUnit file, and failed attempts on opt-in - #1596
Open
jeffpignataro wants to merge 1 commit into
Open
jeffpignataro wants to merge 1 commit into
jeffpignataro wants to merge 1 commit into
Conversation
…n opt-in CircleCITestResults keyed each test.xml by label, and BEP names every run's, shard's and attempt's file test.xml, so each overwrote the last and only one survived per target. Key by run, shard and attempt with the existing testlog_dest_dir layout. By default only each shard's final attempt is uploaded, so a flaky target Bazel reports as passed still shows as passed. upload_failed_attempts uploads each attempt as its own file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014D5dBJA2eDKDosVdJUepMb
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Aspect Workflows TasksTask Results
Install |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CircleCITestResultsuploaded at most one JUnit file per test target. It keyed eachtest.xmlby the target's label, and the Build Event Protocol names every run's, shard's and attempt's filetest.xml, so each one overwrote the one before it. A target with 12 shards reported about a twelfth of its test cases, and a test that failed and then passed under--flaky_test_attemptsshowed only the pass.Each file is now placed with the
testlog_dest_dirlayoutArtifactUploadalready uses for test logs: arun_N,shard_Norattempt_Nsegment is added only for a number above 1, so a test with one run and one shard keeps its current path.By default only each shard's final attempt is uploaded. A later attempt replaces the earlier one even when its own status is not collected, so a flaky target that Bazel reports as passed shows as passed in CircleCI, including under
upload_test_results = "failed". The newupload_failed_attemptsoption uploads every attempt as its own file, so an attempt that failed before a passing retry shows as a failure. It is opt-in because it shows a failure for a target Bazel reported as passed.Linear: CS-296
Changes are visible to end-users: yes
Suggested release notes
CircleCITestResultsnow uploads the JUnit results of every shard and every--runs_per_testrun of a test target. Before, only one file per target reached CircleCI's Tests tab.CircleCITestResultsoptionupload_failed_attempts(defaultFalse): uploads each--flaky_test_attemptsattempt as its own file, so a test that failed before passing on retry shows as a failure in CircleCI.Test plan
circleci_test_results_test.axl(aspect-cli dev test-circleci-test-results): a target with two shards, where shard 2 fails on attempt 1 and passes on attempt 2, plus a target with one shard. It checks the uploaded paths withupload_failed_attemptsoff and on, underupload_test_resultsallandfailed.aspect-cli tests axl(1017 tests),aspect-cli dev test-artifactsandaspect-cli dev test-circlecipass.🤖 Generated with Claude Code
https://claude.ai/code/session_014D5dBJA2eDKDosVdJUepMb