Skip to content

fix(e2e): wait for RunCommand deletion before returning - #10034

Merged
Abigail Liang (abigailliang-aks-sig-node) merged 2 commits into
mainfrom
abigailliang/e2e-await-runcommand-cleanup
Nov 10, 2026
Merged

Abigail Liang (abigailliang-aks-sig-node) merged 2 commits into
mainfrom
abigailliang/e2e-await-runcommand-cleanup

Conversation

@abigailliang-aks-sig-node

Copy link
Copy Markdown
Contributor

Summary

Prerequisite for #10021. Merge this fix first; then update #10021 from main and rerun its SSH-disabled reboot scenarios before merging that PR.

  • Wait for managed RunCommand deletion to finish before runCommand returns, preventing subsequent VM operations from racing an in-flight deletion.
  • Keep the existing independent two-minute cleanup context, including when the caller's context is canceled.
  • Return deletion initiation, polling, and timeout errors, joining them with any existing command error instead of allowing callers to proceed as if cleanup succeeded.
  • Add six regression cases and include them in the existing E2E-helper unit-test step.

Background

The three SSH-disabled reboot scenarios in #10021 submitted a VM restart immediately after an asynchronous RunCommand deletion was accepted. ARM activity records show both operations remained outstanding until VMSS teardown; the RunCommand deletion was then preempted and the restart recorded completion after the test deadline.

The shared cleanup code discarded the BeginDelete poller. This PR fixes that sequencing independently of the SSH policy changes. It does not change SSH configuration, images, or the VMSS test timeout.

Validation

  • All six new regression cases fail on the original code and pass with this fix.
  • Race-enabled RunCommand, sysprep, and Windows guest-validation unit-test selection passed.
  • The exact updated GitHub Actions E2E-helper test selection passed, retaining the existing NVIDIA driver-validation tests.
  • go vet ./scenario, formatting/whitespace checks, and repository-pinned actionlint passed.

Live ARM before/after confirmation remains pending. The SSH reboot E2E scenarios in #10021 must be rerun after this prerequisite merges.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c815829b-19d2-4af1-8269-5d704d14325c
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Windows Unit Test Results

  3 files   21 suites   1m 4s ⏱️
789 tests 789 ✅ 0 💤 0 ❌
815 runs  815 ✅ 0 💤 0 ❌

Results for commit 22f7bd3.

♻️ This comment has been updated with latest results.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

When script execution and cleanup both fail, the wrapper returns only the cleanup error, losing the script’s exit code, stdout, and stderr instead of preserving both errors.

1 open issue
What changed in this PR

This PR makes the E2E RunCommand helper wait for resource deletion before returning, preventing cleanup from racing subsequent VM operations. It is a prerequisite for the SSH-disabled reboot scenarios in #10021; live ARM validation remains pending.

Changes:

  • Poll deletion to completion using the independent two-minute cleanup context.
  • Propagate cleanup failures, joining them with existing command errors.
  • Add six regression cases and expand the existing helper-test CI selection.
File Description
e2e/​scenario/​provision.go Waits for RunCommand deletion and propagates cleanup errors.
e2e/​scenario/​exec_runcommand_test.go Tests asynchronous cleanup, cancellation, failures, and timeout.
.github/​workflows/​go-test.yml Runs cleanup and Windows guest-validation tests alongside existing NVIDIA checks.

Review effort: Balanced

Comment thread e2e/scenario/provision.go Resolved
Comment thread .github/workflows/go-test.yml Outdated
run: |
cd e2e
go test -short ./scenario -run '^TestNvidiaDriverVersionValidationScript'
go test -short ./scenario -run '^(TestNvidiaDriverVersionValidationScript|TestRunCommandCleanup|TestWindowsGuestValidationWithoutSSH)'

@r2k1 Artur Khantimirov (r2k1) Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't feel right. We should use test names only in exceptional situtations

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

restore it. curious the test step Test GRID driver version validation doesnt Illustrate the reason it pickup the specific test name. so i ll just keep it as original.

Copilot AI balanced review requested due to automatic review settings November 10, 2026 21:08
@github-actions

Copy link
Copy Markdown
Contributor

Code Coverage

Package Line Rate Health
test 0% ❌
windows 27% ❌
staging_cse_windows Package 1 71% ➖
debug 0% ❌
provisioningscripts 2% ❌
parts_windows Package 1 73% ➖
Summary 39% (1983 / 7220) ❌

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The CI workflow’s test filter excludes TestRunCommandCleanup, leaving the new cleanup regressions unexecuted; expand the filter while retaining the NVIDIA tests.

1 open issue
1 resolved since last review

Review effort: Balanced

require.Equal(t, commands, deletes)
}

func TestRunCommandCleanup(t *testing.T) {
@github-actions

Copy link
Copy Markdown
Contributor

Code Coverage

Package Line Rate Health
shellspec spec 30% ❌
Summary 30% (4912 / 16591) ❌

@github-actions

Copy link
Copy Markdown
Contributor

Code Coverage

Package Line Rate Health
github.com/Azure/agentbaker/aks-node-controller 73% ➖
github.com/Azure/agentbaker/aks-node-controller/common 100% ✔
github.com/Azure/agentbaker/aks-node-controller/helpers 68% ➖
github.com/Azure/agentbaker/aks-node-controller/parser 88% ✔
github.com/Azure/agentbaker/aks-node-controller/pkg/gpu 100% ✔
github.com/Azure/agentbaker/aks-node-controller/pkg/nodeconfigutils 92% ✔
github.com/Azure/agentbaker/aks-node-controller/utils 0% ❌
github.com/Azure/agentbaker/apiserver 24% ❌
github.com/Azure/agentbaker/cmd 0% ❌
github.com/Azure/agentbaker/cmd/starter 0% ❌
github.com/Azure/agentbaker/fuzz/api 0% ❌
github.com/Azure/agentbaker/hotfix/render-nodecustomdata 35% ❌
github.com/Azure/agentbaker/pkg/agent 79% ➖
github.com/Azure/agentbaker/pkg/agent/datamodel 77% ➖
github.com/Azure/agentbaker/pkg/agent/toggles 0% ❌
github.com/Azure/agentbaker/pkg/vhdbuilder/datamodel 87% ✔
Summary 76% (6094 / 8040) ➖

@abigailliang-aks-sig-node
Abigail Liang (abigailliang-aks-sig-node) merged commit 2728e70 into main Nov 10, 2026
26 of 29 checks passed
@abigailliang-aks-sig-node
Abigail Liang (abigailliang-aks-sig-node) deleted the abigailliang/e2e-await-runcommand-cleanup branch November 10, 2026 21:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants