[Ganeshkumar Ashokavardhanan (ganeshkumarashok)]

Summary

Give AgentBaker GPU E2E VM creation priority over non-AgentBaker standalone GPU VMs and uniform scale sets in subscription 8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8.

On a confirmed Compute core-quota error, use the final VMSS request model and Azure's GPU capability catalogue to find resources with the exact same VM size, region, and Spot/regular priority. Delete one eligible resource, wait for deletion to finish, pause 15 seconds for quota propagation, and retry creation. Recovery is bounded to three retries per scenario attempt, including outer provisioning retries and VHD-caching stages.

Destructive-operation boundary

• No reclamation in any other subscription, for another GPU size (including sibling sizes in the same family), or for allocation/capacity and non-core-quota errors.
• Protect AgentBaker names/resource groups, buildID and KEEP_VMSS tags, AKS-managed resources, managed resource groups, flexible scale sets, and VMSS-member VMs.
• Recheck resource properties and ownership immediately before deletion. Delete only the selected VM or uniform VMSS, never its resource group. A selected uniform scale set is deleted in full.
• Complete all inventory pages before deleting anything. Surface inventory/deletion failures together with the original quota error; these failures cannot become PR capacity skips.
• Serialize recovery within a runner and retry before evicting after waiting for another recovery. This is not a distributed lock across pipeline runs.
• Close a preceding partially provisioned attempt's SSH tunnel before replacing its VM.

Verification follow-up

Adversarial tests reproduced and fixed four issues:

• Parse the structured ARM error message, not ResponseError.Error(): request URLs containing cores must not turn a VM-count quota into destructive core-quota recovery.
• A 404 from an asynchronous deletion-status endpoint is an unconfirmed deletion, not proof that the VM is gone. Stop recovery and retain the error.
• When a candidate disappears concurrently, retry creation before considering another resource.
• Share the three-retry budget across the complete scenario attempt instead of resetting it whenever existing provisioning retries re-enter VMSS creation.

Validation

• cd e2e && go test ./... -race -count=1 -timeout 5m
• Focused quota-recovery and existing VMSS provisioning/classification regression tests, with race detection.
• go vet ./..., gofmt, and git diff --check.

The full E2E module race run passed 967 test/subtest records across eight tested packages, including 34 GPU-quota cases. Mocked Azure tests cover VM/VMSS reclamation, final-model SKU selection, async quota/deletion responses, protected resources, exact SKU/region/priority, pagination, changed resource metadata, shared retry limits, cleanup errors, cancellation, recovery contention, and SSH cleanup.

On the initial PR revision (1e7090dc), hosted GPU E2E build 190297396 passed and its task log confirms one exact-SKU resource deletion completed after a core-quota failure. No manual Azure cleanup was executed. Linux E2E 190297378 failed containerd package validation, and Windows E2E 190297388 failed exporter/capacity checks; those are not quota-reclamation failures and remain outside this change. Hosted checks for this follow-up revision must finish before claiming that revision is green.

[github-actions]

Windows Unit Test Results

  3 files   21 suites   1m 5s ⏱️
789 tests 789 ✅ 0 💤 0 ❌
815 runs  815 ✅ 0 💤 0 ❌

Results for commit 82321cc.

♻️ This comment has been updated with latest results.

[github-actions]

Package	Line Rate	Health
test	0%	❌
windows	27%	❌
staging_cse_windows Package 1	71%	➖
debug	0%	❌
provisioningscripts	2%	❌
parts_windows Package 1	73%	➖
Summary	39% (1983 / 7220)	❌

[github-actions]

Package	Line Rate	Health
shellspec spec	30%	❌
Summary	30% (4912 / 16591)	❌

[github-actions]

Package	Line Rate	Health
github.com/Azure/agentbaker/aks-node-controller	73%	➖
github.com/Azure/agentbaker/aks-node-controller/common	100%	✔
github.com/Azure/agentbaker/aks-node-controller/helpers	68%	➖
github.com/Azure/agentbaker/aks-node-controller/parser	88%	✔
github.com/Azure/agentbaker/aks-node-controller/pkg/gpu	100%	✔
github.com/Azure/agentbaker/aks-node-controller/pkg/nodeconfigutils	92%	✔
github.com/Azure/agentbaker/aks-node-controller/utils	0%	❌
github.com/Azure/agentbaker/apiserver	24%	❌
github.com/Azure/agentbaker/cmd	0%	❌
github.com/Azure/agentbaker/cmd/starter	0%	❌
github.com/Azure/agentbaker/fuzz/api	0%	❌
github.com/Azure/agentbaker/hotfix/render-nodecustomdata	35%	❌
github.com/Azure/agentbaker/pkg/agent	79%	➖
github.com/Azure/agentbaker/pkg/agent/datamodel	77%	➖
github.com/Azure/agentbaker/pkg/agent/toggles	0%	❌
github.com/Azure/agentbaker/pkg/vhdbuilder/datamodel	87%	✔
Summary	76% (6093 / 8041)	➖