AKS Windows Server 2022: RSTs to a deleted pod's IP loop between the synthetic NIC and the accelerated networking VF, pinning cores at high DPC

Please fill out all the sections below for bug issues, otherwise it'll be closed as it won't be actionable for us to address.

Describe the bug

On AKS Windows Server 2022 nodes, after a pod is torn down while it holds outbound TCP connections to an in-cluster LoadBalancer VIP, the host gets stuck forwarding TCP RST packets addressed to the dead pod's IP. The packets enter on the synthetic Hyper-V adapter and are sent straight back out on the accelerated networking VF, indefinitely: 18,000–183,000 packets/sec of ~60-byte packets. They never reach any container.

The kernel DPC work this generates pins specific cores at 25–96%, and every workload on the node slows 1.5x–13x. In our fleet 18 of 138 Windows nodes were in this state at once after a weekend of redeployments. We have seen this class of Windows-node degradation for several years and have worked around it by rotating to fresh nodes on every release.

The dead pod's IP is still present as an ipConfiguration on the node's Azure NIC (Azure CNI node-subnet preallocation), but HNS has no endpoint for it. Only replacing the node has cleared it so far.

To Reproduce

We cannot reproduce it on demand: it is not every teardown, and the same node has had pods torn down with no loop afterwards. The sequence we observed each time:

An AKS cluster with Windows Server 2022 nodes, Azure CNI (node subnet, IPs preallocated on the node NIC), accelerated networking enabled.
An ingress controller (ingress-nginx, on Linux nodes) exposed as an internal type: LoadBalancer Service, externalTrafficPolicy: Cluster, ipMode: VIP.
Windows pods (IIS / .NET) calling other in-cluster services through hostnames that CoreDNS resolves to that LB VIP, so each Windows pod holds keep-alive HTTPS client connections to <LB_VIP>:443.
One of those pods goes away while its connections are open. We saw it after either of:
- a rollout restart / redeploy that moves the pod to another node;
- a node reboot with the pod still scheduled, where the pod came back with a new IP and the old IP began looping.
On the original node, RST,ACK packets from <LB_VIP>:443 to <DEAD_POD_IP>:<ephemeral ports> start circulating, with the receive rate on the synthetic adapter equal to the send rate on the VF. They continue until the node is replaced.

Evidence from an affected node.

Packet counters, 3 × 2 s samples:

\network interface(microsoft hyper-v network adapter _2)\packets received/sec        = 177,859
\network interface(mellanox connectx-4 lx virtual ethernet adapter)\packets sent/sec = 177,990
Hyper-V Virtual Switch ext pkts/s                                                     = ~336,000

An unaffected node in the same pool reads 27–28 packets/sec.

Per-core DPC on an 8-core node:

cpu 0  dpc% 91.4   cpu 2  dpc% 72.0   cpu 4  dpc% 85.5   cpu 6  dpc% 67.0
cpu 1,3,5,7  dpc% 0.0          _total dpc% 39.5   dpcq/s 97,282

No process accounts for this CPU.

pktmon capture, 8 s, headers only:

<LB_VIP>:443 > <DEAD_POD_IP>:51198  Flags [R.]  n=25,739
<LB_VIP>:443 > <DEAD_POD_IP>:51222  Flags [R.]  n=25,388
<LB_VIP>:443 > <DEAD_POD_IP>:51197  Flags [R.]  n=25,388
... 7 flows, 145,294 packets in 8 s

We confirmed the same pattern on a second cluster: 61,224 packets in 5 s to one dead IP.

<DEAD_POD_IP> has no pod and no HNS endpoint (Get-HnsEndpoint).
It is still listed in the node NIC's ipConfigurations.
Pod inventory shows its last owner was a pod on that same node, torn down at the times above.

Expected behavior

A packet arriving for a NIC-owned secondary IP that has no HNS endpoint behind it should be dropped, or answered once, by the host. Instead it is forwarded back out through the VF, apparently without decrementing TTL, delivered back to the same node by Azure, and looped indefinitely.

Configuration:

Edition: Windows Server 2022 (AKS node image AKSWindows-2022-containerd-20348.5139.260513, OS build 10.0.20348.5139)
Base Image being used: Windows Server Core, mcr.microsoft.com/dotnet/framework/aspnet:4.8.1-windowsservercore-ltsc2022 (container OS 10.0.20348.5622)
Container engine: containerd (AKS), kubelet v1.35.4
Container Engine version: containerd 1.7.20+azure

Also relevant:

Network: Azure CNI node subnet; maxPods 250, so 251 ipConfigurations per NIC
Accelerated networking: enabled; Mellanox ConnectX-4 Lx Virtual Ethernet Adapter, driver 23.4.26054.1
VM sizes affected: Standard_D8d_v4, Standard_E8as_v4, Standard_F16s_v2

Additional context

Impact: IIS request latency for pods on affected nodes went from p50 ~250 ms to ~2,700 ms. Two replicas of one deployment, on an affected and an unaffected node, measured 2,769 ms and 253 ms p50 over the same 6 hours. Slowdown correlated with node DPC (r ≈ 0.78 across 12 workloads).
Normal on affected nodes: disk latency (~0.2 ms), Defender and Windows Update activity, the System event log, HNS network and endpoint counts, kubelet, containerd and kube-proxy state. The node image is identical to healthy nodes.
Not reproduced elsewhere: a smaller, low-traffic cluster with the same pod churn and node reuse (node image 20348.5020) shows no loop.
Questions:
1. Is this a known HNS / VFP / accelerated networking defect, and is there a fix in a newer Windows Server 2022 or 2025 node image?
2. Is there a supported way to clear an affected node without replacing it, e.g. an hnsdiag / vfpctrl action or releasing the stale IP?
3. Is there a supported configuration that prevents it, such as a CNI setting, VFP policy, or kube-proxy flag (DSR)?
4. Does Azure CNI Overlay on Windows avoid it, given pod IPs would no longer be NIC ipConfigurations?
Possibly related: HNS endpoints are staying as stale even after pods are terminated #631 (stale HNS endpoints after pod deletion); [BUG] Frequent intermittent network issues on AKS windows nodes Azure/AKS#5483 (intermittent Windows pod networking failures, closed without root cause).
Our current workaround: replace affected nodes, and alert on node % DPC Time > 2% (any core > 15%) or synthetic-NIC receive > 5,000 pps with VF send within 10% of it.
Full pktmon --comp all captures, collectlogs.ps1 output and the NIC ipConfiguration listing are available on request.

Thank you for creating an Issue. Please note that GitHub is not an official channel for Microsoft support requests. To create an official support request, please open a ticket here. Microsoft and the GitHub Community strive to provide a best effort in answering questions and supporting Issues on GitHub.

Update: full component capture on an affected node

We took a pktmon start --capture --comp all capture (6 s, filtered to the stale pod IP) on a node that was still looping, plus collectlogs.ps1 output.

The same packets circulate indefinitely; nothing generates new ones.

465,230 packet events for the single dead pod IP in 6 seconds.
Every one had TTL 128, and nearly all had IP ID 1. Nothing on the path decrements TTL, so one reset loops forever.

The reset appears to be generated on the Windows side.

TTL 128 is the Windows default.
The ingress controller whose VIP is the source address runs on Linux nodes, and Linux would send TTL 64.
So the circulating RST looks like it came from the Windows host stack or VFP answering for a connection it still tracked, not from the remote server.

Per-component counters: no component drops anything, and all drop counters read 0.

vSwitch ext
  Microsoft NetVsc Nic #2 (upper)   Rx   841,559 pkts   | Tx 1,018,969 pkts
  VMS Protocol (lower)              Rx 1,018,939 pkts   | Tx 1,018,939 pkts

Mellanox ConnectX-4 Lx Virtual Ethernet Adapter
  Mellanox ConnectX-4 Lx (upper)    Rx         0 pkts   | Tx 1,018,969 pkts
  NETVSCVFPP (lower)                Rx         0 pkts   | Tx 1,018,939 pkts

Packets enter on the synthetic NetVsc path and leave through the VF, and none come back in through the VF.

HNS has no endpoint for the IP, yet it is still one of the node NIC's ipConfigurations.

The capture ETL, HNS endpoint/network/policy dumps, VFP port list and collectlogs.ps1 output are available on request.

Update: restarting HNS clears the loop; restarting the VF does not

We tested mitigations on two looping nodes in a non-customer cluster. The nodes run the same image and versions as the original report: AKSWindows-2022-containerd-20348.5139.260513, containerd 1.7.20, kubelet v1.35.4, accelerated networking with a Mellanox ConnectX-4 Lx VF. Each node was its own baseline, and the other looping node served as the control.

Action on the node	Before (synthetic NIC rx / VF tx, DPC)	After	Pods
`Restart-NetAdapter` on the Mellanox VF	15,797 / 15,817 pps, 3.2%	16,961 / 16,984 pps, 2.4% at +3 min. No effect	not restarted
`Restart-Service hns -Force`, node A	16,270 / 16,292 pps, 2.2%	24 / 34 pps, 0% at +20 s; still clear at +3 min	not restarted
`Restart-Service hns -Force`, node B	17,222 / 17,240 pps, 3.2%	16 / 15 pps, 0% at +20 s; still clear at +3 min	not restarted

The control node kept looping at full rate throughout node A's test, so the drop is not coincidental.
HNS endpoint counts were identical before and after the restart (227), and every pod kept its IP.
Cost of the restart: about one second of connectivity. We restarted HNS again on node B while probing a pod on it over HTTP from another node, every ~270 ms with a 1 s timeout. 436 of 437 probes succeeded in 12-38 ms. One probe timed out, starting 0.7 s after the restart was issued, and the next probe succeeded 1.3 s later. We did not measure in-flight connections across that window.

A VF reset does not clear the loop, but an HNS restart clears it within seconds. That suggests the forwarding state lives in the rules HNS programs into the virtual switch (VFP), not in the NIC. Our inference is that on restart HNS reprograms VFP from its live endpoint list, and the torn-down pod's IP is absent from that list, so the stale state is not rebuilt. We would appreciate confirmation, or a correction, from the HNS/VFP side.

Fresh nodes are not immune

Our original assumption was that only nodes reused across deployments were affected. That is not the case.

Six nodes created the same day they started looping. They hosted pods for a few hours during a deployment and the pods were then moved off. All six were looping the next day, with little or nothing running on them. Every torn-down pod's IP was still among the node NIC's secondary IP configurations: 33-76 per node, all still present.
Two nodes created as replacements during our remediation showed a low-rate symmetric loop (about 1,400-3,000 pps, DPC under 1%) within hours of creation.
Teardown is necessary but not sufficient. A node with pods torn down on two earlier dates did not loop afterwards. Loops also sometimes decay on their own.

So rotating onto fresh nodes each release lowers the odds, but does not prevent the loop.

Additional capture detail

A full pktmon --comp all capture filtered to one dead pod IP recorded 465,230 events. Every packet carried TTL 128 and IP ID 1, with zero drops reported by any component. The packets are TCP RST,ACK from the in-cluster ingress load balancer VIP on 443 to the dead pod's ephemeral ports. Because the TTL is never decremented, the packets are being switched rather than routed, and TTL 128 suggests the resets are generated on the Windows host rather than by the Linux ingress pods. Both points are inference. The pktmon ETL with counters, HNS endpoint/network/policy JSON, the VFP port list and collectlogs.ps1 output are available on request.

Mitigation we are deploying

A HostProcess DaemonSet on every Windows node that:

samples the synthetic NIC's Packets Received/sec and the VF's Packets Sent/sec every 30 s;
treats the node as looping when both are at least 5,000 pps and within 10% of each other. Across 138 production nodes, healthy nodes measured 10-500 pps, and looping nodes measured 15,800-183,000 pps with the two rates within 0.2%;
after 10 minutes sustained, restarts HNS with Restart-Service hns -Force, waits 60 s and re-measures;
backs off: at most one restart per node per hour and three per day. A loop that returns inside the cooldown, or survives a restart, is reported as an error and the node is replaced;
posts Warning events on the Node and logs JSON, so the condition is alertable.

This remediates the symptom, not the cause.

Questions

Why does the host forward, rather than drop, packets addressed to a secondary IP the NIC still owns when HNS has no endpoint for it?
Is restarting HNS a supported mitigation on a production AKS Windows node? Is there a less disruptive way to make HNS reprogram VFP?
Is there a fix, or a known-good node image or Windows build, that addresses this?

Thanks for the detailed report and all the additional investigation. There’s a lot of useful diagnostic information here, particularly around the packet path and the difference in behavior between restarting the VF and restarting HNS.

We’ll need to investigate further to determine the underlying cause. If possible, please share the diagnostic artifacts you mentioned, including the full pktmon capture, HNS/VFP state, and collectlogs.ps1 output. These should help with narrowing down the behavior.

I will post it this evening once i have the logs scrubbed and zipped.

Regarding the open ended questions on my last post, 2 is the most pertinent as we have a daemonset deployed right now monitoring for this and restarting hns when it passes a threshold of events (currently only monitoring on prod but restarting on our non prod clusters - will switch it when we get confirmation on question 2 above that it is a safe approach).

Thanks for picking this up. Two attachments, plus a finding from today that changes the picture.

Attachments (all host names, workload names and addresses are pseudonymised, consistently across both files and the text below):

wincontainers-655-raw-diagnostics.zip - captured today from a node that was looping at the time: a pktmon start --capture --comp all capture filtered to the looped IP (pcapng sample of 73,842 packets; IPv4 headers rewritten and checksums recomputed), Get-HnsNetwork / Get-HnsEndpoint / Get-HnsPolicyList JSON, vfpctrl port list and spaces, collectlogs.ps1 text output, and HNS / VmSwitch / Hyper-V Compute / System event logs as text. The raw ETL could not be anonymised; we can share it privately if you need it.
wincontainers-655-evidence.zip - the processed evidence behind the earlier comments (per-core DPC, NIC rates, pktmon summaries, the HNS-restart and VF-restart tests).

Both contain a README with the pseudonym scheme and a timeline.

New: what keeps feeding the loop, and why restarting HNS sometimes does not hold

Edited: an earlier version of this comment attributed the seed packets to the Azure load balancer's idle-timeout reset. That was wrong; see the third bullet.

On today's node an HNS restart cleared the loop within seconds, and it re-formed within a minute, twice. A component capture filtered to the looped IP showed:

Every looping flow is a single packet. Each is one RST,ACK from the ingress VIP (10.102.191.0:443) to an ephemeral port on a dead pod IP (10.102.160.223), with exactly one distinct TCP sequence number per flow, circulating at roughly 5,000 times a second each.
New flows kept appearing after the pod left: 4 at 02:50 UTC, 11 at 02:53, on new ports each time. The pod owning that IP was last seen at 02:27:30, so seeds were still arriving at least 26 minutes later.
The seeds are generated on the Windows node itself. The ingress Service uses ipMode: VIP, so a pod's connection to the VIP is DNAT'd by the node's HNS load-balancer policy straight to an nginx pod and never passes through the Azure load balancer. Every RST carries TTL 128, where the Linux ingress nodes would send 64, and the nginx pods held no sockets to the dead IP by then. Our inference is that VFP / the HNS load-balancer policy expires the departed pod's NAT flows some minutes later and answers each with an RST addressed to the pod, sourced from the VIP.
Each of those RSTs becomes a permanent loop packet on the node, because the IP is still an ipConfiguration on the node's NIC with no HNS endpoint.

So restarting HNS does work, but only once no further RSTs can arrive. Inside that window the loop re-forms. We confirmed it on the same node: an HNS restart at 04:19 UTC, 75 minutes after its last application pod left, cleared the loop within 15 seconds (synthetic NIC receive fell from ~264k to ~1k packets/s, DPC from 26.5% to 0.1%) and it stayed clear. Our earlier HNS-restart tests held because the dead pods had been gone for days.

This sharpens the question we asked before. The host itself generates a reset for a connection of a deleted endpoint, addresses it to that endpoint's IP, and then forwards that packet out of the VF and straight back in on the synthetic path, indefinitely, with no TTL decrement and no drop.

Questions

Is VFP (or the HNS load-balancer policy) expected to emit RSTs for NAT flows belonging to an endpoint that has been deleted? What sets that timeout?
Is forwarding, rather than dropping, traffic to a NIC-owned IP that has no HNS endpoint expected VFP behaviour? Is there a supported way to make the host drop it?
When HNS reprograms VFP on restart, is there an equivalent that runs when an endpoint is deleted?
Is restarting HNS a supported mitigation on a production AKS Windows node? Is there a better-scoped action, for example reprogramming a single port?
Is a fix planned, or is there a known-good node image or Windows build?

For anyone else hitting this: we are changing our remediation DaemonSet to defer the HNS restart until about 38 minutes after the last pod left the node (a window measured empirically, not derived from a known timeout), rather than a fixed delay.

wincontainers-655-evidence.zip
wincontainers-655-raw-diagnostics.zip

One more data point: the loop is not specific to the Mellanox VF.

The same cluster has a node pool on Standard_D8d_v5, whose accelerated-networking VF is the Microsoft Azure Network Adapter (MANA) rather than a Mellanox ConnectX-4 Lx. Two of its nine nodes were looping tonight, at about 148k and 101k packets/s against 26-56 packets/s on the other seven.

A capture on one of them shows the same signature as the Mellanox nodes: RST,ACK from the ingress VIP (10.102.191.0:443) to ports on two dead pod IPs, neither of which has an HNS endpoint (60k and 26k packets in 5 s). The pods owning those IPs had left the node 5 days and 5.4 hours earlier, so one of these loops had been running for about 5 days. Restarting HNS cleared it (152,855 to 45 packets/s within 23 s).

Two differences from the Mellanox nodes:

DPC stays at 0%. On Mellanox nodes the loop pins individual cores at 25-96% DPC and slows every pod on the node. On MANA it costs the host no measurable DPC time, so it is invisible to a DPC check. We only found it by its packet rate.
The counters read differently. On MANA the synthetic adapter's receive and send rates and the MANA adapter's receive and send rates are all equal. On Mellanox, packets enter on the synthetic path and leave through the VF, with VF receive near zero.

So the forwarding behaviour is common to both VF types and is not a Mellanox driver issue. MANA hides the cost from the host CPU but still carries 100k+ packets/s of looping traffic.

Everything in it is measured. The IP is in its pseudonymised form, consistent with the earlier comment.

smiezah-msft

smiezah-msft - the most pressing question we have is this one currently:

Is restarting HNS a supported mitigation on a production AKS Windows node? Is there a better-scoped action, for example reprogramming a single port?

Knowing this will allow us to move from just monitoring to resolving the issue automatically in our production clusters via the daemonset we have installed.

Navigation Menu

Repository navigation

AKS Windows Server 2022: RSTs to a deleted pod's IP loop between the synthetic NIC and the accelerated networking VF, pinning cores at high DPC #655

Update: restarting HNS clears the loop; restarting the VF does not

Fresh nodes are not immune

Additional capture detail

Mitigation we are deploying

Questions

New: what keeps feeding the loop, and why restarting HNS sometimes does not hold

Questions

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

AKS Windows Server 2022: RSTs to a deleted pod's IP loop between the synthetic NIC and the accelerated networking VF, pinning cores at high DPC #655

Description

Activity

github-actions commented on Oct 5, 2026

richshadman commented on Oct 5, 2026

richshadman commented on Oct 5, 2026

Update: restarting HNS clears the loop; restarting the VF does not

Fresh nodes are not immune

Additional capture detail

Mitigation we are deploying

Questions

smiezah-msft commented on Oct 14, 2026

richshadman commented on Oct 15, 2026

richshadman commented on Oct 15, 2026

New: what keeps feeding the loop, and why restarting HNS sometimes does not hold

Questions

richshadman commented on Oct 15, 2026

richshadman commented on Oct 16, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions