You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(localdns): keep node and pod DNS working via NAT when localdns fails - #9486
Pods get 169.254.10.11 or 169.254.10.10 written into /etc/resolv.conf by kubelet at sandbox creation, and it is never revisited. LocalDNS does not DNAT — build_localdns_iptable_rules() installs raw-table NOTRACK rules only — so when nothing is listening, queries are not rerouted anywhere. They fail.
That makes a dead localdns.service a DNS blackhole nothing recovers from. AKS#5930 was ~16 minutes of it on a customer node, ended by manual intervention.
This PR keeps both listeners working while localdns is in terminal failed, using kernel NAT: one nat chain that rewrites the destination of DNS packets so they never arrive at the dead addresses in the first place.
listener
redirected to
169.254.10.10:53 udp/tcp
the VNet DNS from the restored systemd-resolved upstream file
169.254.10.11:53 udp/tcp
kube-dns — kube-proxy's own service chain, or DNAT across the Cilium agent's backends
Do not attribute the #5930 fix to this PR — #9361 is the fix; this is the floor underneath failure causes nobody has enumerated.
Why NAT and not a second CoreDNS
An earlier revision ran a second CoreDNS against a corefile derived from localdns's own. That shares a failure mode with the thing it is backing up. If localdns died because the baked binary cannot parse a directive the RP sent, the fallback inherits the same binary and the same directive and dies with it.
This is reachable, not theoretical. failfast_all_unhealthy_upstreams requires CoreDNS ≥ 1.12.1; measured across the five builds cached in components.json:
VERSION baseline health_ck no_rec domain failfast
v1.11.3-32 ok ok ok ok FAIL(1) plugin/forward: unknown property
v1.12.1-25 ok ok ok ok ok
v1.13.1-20 … v1.14.7-2 all ok
A hard startup failure, so the fallback would have crash-looped every 5s forever on such a node. A fallback must not fail for the same reason its principal did. iptables rules cannot crash-loop or misparse a config.
Why both listeners, not just .11
Which address a pod gets depends on its dnsPolicy: ClusterFirst → .11, Default and hostNetwork → .10. The second group includes CoreDNS itself, konnectivity-agent, the CSI node drivers, kube-proxy and azure-cns. Leaving .10 dark black-holes all of them — and also breaks external resolution for .11 clients, because that query goes .11 → ClusterIP → a CoreDNS replica on this same node → its own dead .10.
Why .11 is not DNAT'd at the kube-dns ClusterIP
The nat table is traversed once per connection and DNAT is a terminal verdict for it. A rule rewriting .11 to the ClusterIP consumes the traversal KUBE-SERVICES needed to translate that ClusterIP into a pod IP, so the packet leaves addressed to a virtual IP nothing owns. On Cilium the service load balancer runs before netfilter, so a ClusterIP target is never translated at all.
Going one hop further — straight into KUBE-SVC-<kube-dns>, or at the backends the Cilium agent reports — lands on a real pod IP. It also means the fallback never needs the ClusterIP, which is why the COREDNS_SERVICE_IP plumbing comes back out of parser.go, baker.go, cse_cmd.sh and cse_config_localdns.sh, along with its repro test.
The gate
apply refuses — exit 1, logged — unless localdns is ActiveState=failed, an upstream other than .10 exists in the restored resolv.conf, and either kube-proxy's kube-dns chains or a Cilium agent reporting active backends andEnableHostLegacyRouting=true is found. With eBPF host routing, pod egress skips netfilter entirely and a nat rule would never see the packet.
OnFailure= fires on every failed start attempt, not only the terminal one — measured on systemd 255.4, six firings across a five-restart budget. Five of those are starts where localdns is mid-restart and the only correct action is to do nothing. StartLimitIntervalSec=0 stops that burst rate-limiting the one start that matters.
Those no-ops are skipped, not failed. ExecStart= has no exit code meaning "nothing to do" — exit 0 latches the unit active under RemainAfterExit= and turns the one start that matters into a no-op — so the unit gates on ExecCondition=/usr/bin/systemctl is-failed localdns.service, whose non-zero exit skips the unit without failing it. Without it, a node whose localdns flaked once and recovered was left with localdns-fallback.service in failed permanently, because ExecStartPre's stop is not reset-failed: a false alarm in the field, and a red scenario on the repo-wide failed-unit validator. apply re-checks the same state, so running the script by hand stays guarded.
Recovery is unchanged: localdns.service's ExecStartPre stops this unit, whose ExecStop flushes the chain. The rules are runtime-only, so a reboot leaves nothing behind.
The systemd version split
ActiveState=failed means terminally failed on every version: through the whole restart budget the unit sits in activating/auto-restart, measured on 249 and 255. What differs is how often OnFailure= fires.
VHD
systemd
firings per crash storm
Ubuntu2204
249
1 — terminal only
Ubuntu2404, AzureLinuxV3, ACL
255
6 — one per attempt
Ubuntu2604
259
untested
The fallback behaves identically on both; only how often it is asked to do nothing changes. That is why the gated no-ops are rare on 249 and routine on 255.
One deviation from the prototype, flagged for review
applypolls for the resolv.conf upstream rather than reading once, bounded at 10s. ExecStopPost removes the drop-in and runs networkctl reload, which is asynchronous, and OnFailure= fires as soon as ExecStopPost returns — so the file can still hold the drop-in state (.10 and nothing else) at that instant. The oneshot has no Restart=, and OnFailure= will not fire again once localdns is terminally failed, so a single early read would leave no fallback at all. Same race wait_for_localdns_removed_from_resolv_conf exists for on the start path.
Two things deliberately not done
No cache, serve_stale, hosts plugin or overrides, and .10 clients lose cluster.local. That is the same position they are in on a node with localdns disabled. This is a bridge that keeps DNS moving until the node is remediated, not a replacement for localdns.
No attempt to recover localdns. The failed unit is left alone and a node on the fallback keeps reporting LocalDNSError, which is correct: once .10 is served again, systemctl is-active is the only remaining signal that anything is wrong. Note the division of labour — NPD only increments problem_counter{reason="LocalDNSError"} (type: temporary, no NodeCondition); aks-operator's local-dns.rules turn 5 hits in 10m into Reboot after 5m and Redeploy after 15m; the remediator does the cordon, drain and reboot. localdns is dead and needs remediation; the fallback only keeps DNS working until that happens.
Verification
Live clusters, against this PR's own VHD build (VHD_BUILD_ID=184891026, so the lane resolves branch-built images rather than main's), kubenet / kube-proxy, scriptless and CSE paths.
scenario
result
LocalDNS/Ubuntu2204
29 assertions, 0 failures
LocalDNS/Ubuntu2404
38 assertions, 0 failures
LocalDNSNoHostsPlugin/Ubuntu2204
passed
Ubuntu2404 is the systemd 255 case, and it is the one that proves the gate works:
Ten of eleven fires were skipped, and the unit never landed in failed on a healthy node. On 2204 (systemd 249) OnFailure= fires once, so both halves of the version split are now covered by a real run rather than by argument.
LocalDNSNoHostsPlugin is the only scenario that drives a real failing startup of localdns.sh — a malformed LOCALDNS_COREFILE_BASE with Restart=no. The node's journal confirms it reached the path under test:
localdns.sh[13478]: Timed out waiting for CoreDNS to create PID file at /run/localdns.pid.
systemd[1]: localdns.service: Main process exited, code=exited, status=216/GROUP
Exactly one occurrence, which is what Restart=no should produce.
Also: ShellSpec 32 examples / 0 failures, make validate-shell exit 0, go build + go vet + gofmt clean, 9 packer JSONs valid.
How the e2e is organised
The fallback no longer has scenarios of its own. LocalDNS/{Ubuntu2204,Ubuntu2404,AzureLinuxV3} already drove localdns to terminal failed and recovered it, which is exactly the state the fallback needs — so the assertions run against that fault instead of manufacturing a second one on three more VMs. Net −2 VMs per lane and one fault per node.
That required splitting the restart-budget validation: localdnsTerminalFailedScript carried a trap teardown EXIT that restarted localdns the moment the limiter assertion passed, so the terminal state never survived. It is now driveLocalDNSToTerminalFailed, which deliberately exits with the unit failed, plus validateLocalDNSRestartBudgetDirectives for the static half. Restore is the caller's job and runs on every path.
Ordering is deliberate: budget directives before the fault, so a budget regression reads as a budget regression rather than as a fallback that never activated; probe pods before it too, since their resolv.conf is written at sandbox creation and never revisited.
LocalDNSHostsPlugin was renamed LocalDNS. The old name described the config, not the test — the hosts-plugin assertions live in ValidateCommonLocalDNS, while the scenario's own validator ran lifecycle and budget. It now covers four things. Note this resets ADO test-result history for those scenarios.
Static checks live in ShellSpec, not e2e: unit directives (OnFailure=, ExecCondition=, Type=oneshot, RemainAfterExit=, ExecStop=, StartLimitIntervalSec=0, the absence of Conflicts= and of [Install], the ExecStartPre handoff) and rule generation (udp/tcp DNAT, the KUBE-SVC jump, Cilium spreading, never DNAT'ing at a ClusterIP, insertion order). Each was mutation-checked — every one of ten mutations reddens exactly one example. What stays on a node is what needs one: the chain installed in the kernel, both hooks, NOTRACK removal, real DNS from pods, the systemd handoff, recovery, and that ExecCondition= actually gates rather than merely being present.
The artifacts are no longer staged onto the node. Staging let the scenario run before they shipped, but once they do it is a mask: it repaired the exact gap the scenario exists to detect and then reported a pass — which is why the OSGuard mode split was invisible here. Absence now fails and names the VHD.
Still outstanding
kube-proxy iptables mode only.kube_dns_chain() reads iptables -S KUBE-SERVICES. In ipvs mode the chain shape differs; in nftables mode the chains are not in iptables at all. In both, apply finds neither a kube-dns chain nor a Cilium agent and refuses — the node is left as it was. Neither mode is tested.
No Cilium scenario in e2e, by agreement: e2e covers the kube-proxy cases and Cilium is verified by hand (matrix posted in-thread, both dataplanes green on all 8 rows). ValidateIPTablesCompatibleWithCiliumEBPF additionally fails on a pre-existing, unrelated allowlist gap — a mangle-table -A FORWARD -d 168.63.129.16/32 -p tcp -j DROP with no --dport, which no AgentBaker script writes. Being raised with acndp separately.
AzureLinuxV3 and the ANC lane were not in the hand runs above — both are covered by the PR check-in gate.
The outage window is not measured. The scenario bounds recovery but never times the gap.
Conntrack: cleanup_iptables_and_dns deletes the NOTRACK rules on exit and DNAT requires conntrack, so they cannot return while the fallback is active. Every pod DNS query creates a conntrack entry for the length of the outage — bounded by the remediation window.
Cilium backends are snapshotted at apply. The kube-proxy path follows endpoint changes because it jumps into a live chain; the Cilium path does not.
localdns-fallback.service mode differs by build path.packer_source.sh installs it 644, azlosguard.yml600, and checkLocaldnsScriptsAndConfigs asserts 644. This is the same split localdns.service already has on main, not something this PR introduces — flagged rather than resolved here, since fixing it means either an OS-SKU-aware assertion or moving azlosguard's units off 600.
Companion aks-rp fix (separate repo) — populate CoreDnsServiceIp on the scriptless/ANC path
The ANC bug documented here (docs/bugs/scriptless-coredns-serviceip-not-populated.md, pinned by the regression tests in aks-node-controller/parser/coredns_service_ip_repro_test.go) is in aks-rp, not AgentBaker: aks-rp never populates ClusterNetworkConfig.core_dns_service_ip, so getCoreDnsServiceIp falls back to the hardcoded 10.0.0.10 regardless of a custom service CIDR. The .11 fallback (and the LocalDNS corefile) on the scriptless path therefore forward to the wrong ClusterIP for custom-CIDR clusters.
The value already flows correctly on the CSE path (typeconversion.go → KubernetesConfig.DNSServiceIP); only the two scriptless builders need wiring. There's even a dead helper (getCoreDNSServiceIP in scriptless_variables.go, zero callers) that produces the value.
getCoreDNSServiceIP(config) already returns config.AgentPoolProfile.KubernetesConfig.DNSServiceIP (empty string when unset, which AgentBaker safely defaults). No new code needed beyond the field assignment; this also removes the "declared and not used" dead-code smell.
2. scriptless_direct.go (proto-sourced path) — pull from the network profile, mirroring the sibling mc.Get… fields:
GetKubeDnsServiceIP() is the proto accessor on NetworkProfile (field 5). It is empty when the customer used the default CIDR, which AgentBaker defaults to 10.0.0.10.
Verification after the aks-rp change
Unit-test both builders assert CoreDnsServiceIp == the configured DNSServiceIP / KubeDnsServiceIP (not empty / not silently defaulted).
The AgentBaker-side regression tests here already prove the ANC template honors the value once supplied (TestReproCustomCoreDnsServiceIpHonoredWhenSet) and defaults when unset (TestReproDefaultUsedWhenCoreDnsServiceIpUnset), so once aks-rp populates the field, custom-CIDR scriptless nodes get the correct ClusterIP end to end.
Ordering / impact
Independent, pre-existing bug — not introduced by this PR; the fallback work merely surfaced it.
Default-CIDR clusters are unaffected either way. The aks-rp fix is required for custom-CIDR correctness on the scriptless path (both for the existing LocalDNS corefile and this PR's .11 fallback).
Finding (root cause + blast radius): CoreDnsServiceIp never populated on the scriptless / ANC bootstrap path
This is the standalone write-up of the bug that the .11 fallback work surfaced (companion to the fix I posted above, and mirrored in docs/bugs/scriptless-coredns-serviceip-not-populated.md in this PR).
Status: Latent correctness bug (pre-existing; surfaced while designing the LocalDNS .11 fallback) Affected path: Scriptless / aks-node-controller (ANC) node bootstrapping Not affected: CSE (script-based) bootstrapping Blast radius: Clusters that are (scriptless/ANC) × (LocalDNS enabled) × (custom, non-default DNSServiceIP)
Summary
On the scriptless (ANC) bootstrap path, aks-rp never sets the ClusterNetworkConfig.core_dns_service_ip proto field it sends to AgentBaker. AgentBaker therefore falls back to a hardcoded default of 10.0.0.10. For any cluster whose real kube-dns Service ClusterIP is not10.0.0.10 (i.e. a custom service CIDR / custom DNSServiceIP), the generated LocalDNS corefile forwards cluster.local queries to the wrong address.
The CSE path is unaffected because it carries DNSServiceIP correctly through a different mechanism.
Root cause (verified in code)
Ultimate origin of the value — NetworkProfile.DNSServiceIP (customer ARM input), defaulted to 10.0.0.10 when unset:
Both set VnetName, Subnet, SecurityGroupName, RouteTable, ... but neverCoreDnsServiceIp. There is even a helper that would produce the value, but it has zero callers (dead code — evidence the wiring was intended but never completed): resourceprovider/sharedlib/agentbaker/scriptless_variables.go:142. A repo-wide search for CoreDnsServiceIp (excluding generated *.pb.go) returns no assignments, setters, or proto merges anywhere in aks-rp.
AgentBaker silently defaults — because the field arrives empty, AgentBaker substitutes the hardcoded default: aks-node-controller/parser/helper.go:924-931 returns defaultCoreDnsServiceIp ("10.0.0.10", consts.go:54) when the proto field is empty. The scriptless path does otherwise support LocalDNS (so the two features coexist and the bug is reachable): scriptless.go:322 and scriptless_direct.go:247 both wire LocalDnsProfile.
Impact
Condition
Result
Default service CIDR (kube-dns at 10.0.0.10)
No visible impact — the wrong-by-omission default coincides with reality
Custom DNSServiceIP (non-10.0.0.10) on scriptless + LocalDNS
LocalDNS corefile forwards cluster.local to 10.0.0.10 instead of the real ClusterIP → in-cluster DNS resolution via LocalDNS is misdirected
Likely gone unnoticed because the overwhelming majority of clusters use the default service CIDR, where the hardcoded default is accidentally correct.
Why it matters here
This PR's .11 fallback persists the real kube-dns ClusterIP to /etc/localdns/environment so the fallback can forward 169.254.10.11 → real ClusterIP. On the scriptless path that persisted value is currently the wrong default for custom-CIDR clusters, so the fallback foundation depends on fixing this gap. Pre-existing and independent of this PR / #9439 — the fallback design merely surfaced it.
The two regression tests in this PR pin the behavior: TestReproCustomCoreDnsServiceIpHonoredWhenSet (ANC honors a custom value when supplied) and TestReproDefaultUsedWhenCoreDnsServiceIpUnset (silently defaults when unset). The concrete aks-rp fix is in the companion comment above.
Open questions to confirm before filing in aks-rp
Is scriptless/ANC + LocalDNS GA to customers, or still pre-GA? (Determines whether this is currently reachable in production vs. latent-only.)
Is there any RP-side defaulting webhook or proto-default that fills core_dns_service_ip outside the two builders inspected? (Static search found none.)
the design write-up is good, but it doesn't describe what the PR actually does.
1. this PR restarts kubelet, and the description never says so.localdns-kubelet-dns.sh:69-77 runs systemctl restart kubelet.service on repoint and again on restore. so one localdns failure episode = two kubelet restarts on a production node. that is a much bigger hammer than the DNS gap it's covering, and it isn't in the TL;DR, the design section, or the behaviour matrix — a reviewer reading the description would approve this without knowing kubelet gets bounced.
put it in the description, and then justify it: what's the node's NotReady window during a kubelet restart, and how does that compare to node-monitor-grace-period? on a node where localdns flaps slowly, do we get repeated kubelet restarts? (:25-26 says "only restart when they actually changed the file", which handles fast flapping — but a localdns that fails, recovers, fails again an hour later restarts kubelet 4 times.)
honestly I'd land the fallback without the kubelet repoint first. .11 answering is the fix for the reported problem; repointing new pods is a separate, riskier optimisation that deserves its own PR and its own e2e.
2. the ANC "known issue" is not a known issue, it's a live regression.resolve_coredns_service_ip:43-50 falls back to the hardcoded 10.0.0.10 when COREDNS_SERVICE_IP is empty. you've established that aks-rp never populates it on the scriptless path. so on a custom-service-CIDR ANC cluster this PR points kubelet's --cluster-dns at an IP that does not exist, and every pod created from then on has no DNS at all. that's worse than the outage.
fail closed instead of guessing:
resolve_coredns_service_ip() {
local ip="${COREDNS_SERVICE_IP:-}"
[ -n"${ip}" ] ||return 1 # caller no-ops; never guess the ClusterIPprintf'%s'"${ip}"
}
and land the aks-rp change first. this is a release-ordering problem, not a follow-up.
related: restore:117-129 needs COREDNS_SERVICE_IP for its fallback branch. empty value + lost ORIGINAL_STATE_FILE = kubelet pinned at 10.0.0.10 permanently with nothing to undo it.
3. set_cluster_dns:58-66 is a non-atomic write to a file kubelet reads at start.mktemp lands in /tmp, then cat tmp > /etc/default/kubelet. a reboot or power loss mid-cat leaves KUBELET_FLAGS= truncated and kubelet comes up with no flags. stage on the same filesystem and rename:
also chmod 0600 at :66 — is /etc/default/kubelet 0600 today? if it's 0644 you're silently tightening it as a side effect of a DNS repoint. preserve the original mode rather than asserting one.
4. ExecStartPre=-/usr/bin/systemctl stop localdns-fallback.service is unbounded. you argue in cse_config_localdns.sh:148-150 that provisioning systemctl calls need timeout because an unresponsive PID 1 / D-Bus must not eat the deadline — same argument applies here, more so, because this one runs systemctl stop from inside another unit's Exec*, which is the documented way to deadlock systemd. the fallback also declares Before=localdns.service, so the stop job can be ordered against the start job you're inside. at minimum:
you say "this synchronous stop is the ONLY handoff" — so what happens to .11 when it times out or deadlocks? that path needs an answer, it's the one that yanks the socket mid-flight.
5. probe timer is every 5s forever, with AccuracySec=1s. that's a process spawn every 5s on every node for the life of the node, and AccuracySec=1s opts out of systemd's timer coalescing so it wakes the CPU precisely each time. why 5s? the debounce is 3 failures, so 10s/3 gets you ~30s switch-over instead of ~15s at half the wakeups. and drop AccuracySec unless you can show the coalescing jitter actually breaks the debounce.
6. 9 packer JSONs bake 4 new artifacts and there's no linux-vhd-content-test.sh change. same ask as #6082 — assert the new units and scripts are present and executable in the VHD, otherwise a packer JSON typo ships a VHD where the fallback silently doesn't exist and we find out during an incident. cse_config_localdns.sh:147's systemctl cat ... || skip means a missing unit is silent by design.
7. the description's "Follow-ups (not in this PR)" is stale. it lists the E2E scenario and the localdns-fallback.sh ShellSpec as not-in-this-PR, but the diff adds e2e/scenario/scenario_localdns_fallback.go, validators_localdns_fallback.go (+1040), and localdns-fallback_spec.sh (+370).
8. 2855 lines / 28 files. split it: (a) env-plumbing of COREDNS_SERVICE_IP, (b) the .11 fallback unit + OnFailure=, (c) the kubelet repoint, (d) the probe. (a) and (b) are the change that fixes the reported problem and could land this week; (c) is the one that needs the aks-rp fix ahead of it.
… with a spec
The TERM trap added to reap the watchdog sleep also made 'exit 0' fire the
EXIT trap, so cleanup_localdns_configs now runs in-process on every systemd
stop where previously bash died first and only ExecStopPost ran. That was a
side effect, not a decision, and nothing recorded it.
Keeping the behaviour. Measured on a node rather than argued (journal from
gate build 181777373, AzureLinuxV3, n=11 stops / n=18 starts):
stop 5-6s (essentially all of it LOCALDNS_SHUTDOWN_DELAY draining)
start 1-2s
restart 6-8s against the 'timeout 30 systemctl restart localdns' in
enableLocalDNS() and its e2e mirror -- ~22-24s of headroom
So the cost lands at about a quarter of the only bound that constrains it,
and the behaviour it buys is better than the alternative: without the drain
CoreDNS is SIGKILLed by the cgroup on every stop, dropping in-flight queries.
Trimming it would spend real shutdown quality to reclaim time nothing needs.
It does not move the restart-budget math. The worst cycle is a hung start,
where this trap cannot run at all -- bash defers a trapped signal until the
foreground command returns, and that fault never returns -- so systemd spends
the full TimeoutStopSec there regardless.
Second consequence, previously unrecorded: cleanup_localdns_configs now runs
to completion, so the dummy interface carrying .10/.11 is torn down on a stop.
localdns_cleanup_mode (ExecStopPost) deliberately leaves the link alone in case
an orphaned CoreDNS is still answering on .11, so the two paths used to differ;
for a clean stop they now agree. The teardown/recreate cycle was clean in the
same run: 12 teardowns, 18 setups, zero address-in-use or RTNETLINK errors.
Worth knowing for #9486 -- its "create the dummy interface idempotently"
constraint is now the common case on a clean stop rather than the exception.
The spec covers the trap wiring, which nothing did before. The traps sit below
"${__SOURCED__:+return}", so Include cannot reach them; the spec greps the real
trap lines out of the shipped script and executes them against stubs instead of
restating them, so deleting or altering a trap fails the test. Mutation-checked
both ways: removing the TERM trap fails all three examples, and changing its
'exit 0' to 'exit 1' fails all three.
Comment and test only; no behaviour change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… with a spec
The TERM trap added to reap the watchdog sleep also made 'exit 0' fire the
EXIT trap, so cleanup_localdns_configs now runs in-process on every systemd
stop where previously bash died first and only ExecStopPost ran. That was a
side effect, not a decision, and nothing recorded it.
Keeping the behaviour. Measured on a node rather than argued (journal from
gate build 181777373, AzureLinuxV3, n=11 stops / n=18 starts):
stop 5-6s (essentially all of it LOCALDNS_SHUTDOWN_DELAY draining)
start 1-2s
restart 6-8s against the 'timeout 30 systemctl restart localdns' in
enableLocalDNS() and its e2e mirror -- ~22-24s of headroom
So the cost lands at about a quarter of the only bound that constrains it,
and the behaviour it buys is better than the alternative: without the drain
CoreDNS is SIGKILLed by the cgroup on every stop, dropping in-flight queries.
Trimming it would spend real shutdown quality to reclaim time nothing needs.
It does not move the restart-budget math. The worst cycle is a hung start,
where this trap cannot run at all -- bash defers a trapped signal until the
foreground command returns, and that fault never returns -- so systemd spends
the full TimeoutStopSec there regardless.
Second consequence, previously unrecorded: cleanup_localdns_configs now runs
to completion, so the dummy interface carrying .10/.11 is torn down on a stop.
localdns_cleanup_mode (ExecStopPost) deliberately leaves the link alone in case
an orphaned CoreDNS is still answering on .11, so the two paths used to differ;
for a clean stop they now agree. The teardown/recreate cycle was clean in the
same run: 12 teardowns, 18 setups, zero address-in-use or RTNETLINK errors.
Worth knowing for #9486 -- its "create the dummy interface idempotently"
constraint is now the common case on a clean stop rather than the exception.
The spec covers the trap wiring, which nothing did before. The traps sit below
"${__SOURCED__:+return}", so Include cannot reach them; the spec greps the real
trap lines out of the shipped script and executes them against stubs instead of
restating them, so deleting or altering a trap fails the test. Mutation-checked
both ways: removing the TERM trap fails all three examples, and changing its
'exit 0' to 'exit 1' fails all three.
Comment and test only; no behaviour change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Base automatically changed from
fix/localdns-pod-service-recovery to
mainSeptember 23, 2026 23:31
The reason will be displayed to describe this comment to others. Learn more.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Adds a systemd-driven LocalDNS pod-DNS fallback that keeps 169.254.10.11 answering (forwarding to the kube-dns ClusterIP) when localdns.service is down or not serving .11, using both OnFailure= and a debounce probe timer.
Changes:
Bake new fallback/probe scripts and systemd units into VHDs, and install them via packer.
Persist COREDNS_SERVICE_IP into /etc/localdns/environment (CSE + ANC plumbing) and wire OnFailure= + ExecStartPre stop-handoff.
Add ShellSpec coverage and a new e2e scenario/validator for crash-storm + clean-stop fallback behavior.
Run:184262078 Failed job/stage/task: e2e / Run AgentBaker E2E / Run AgentBaker E2E
Detective summary: LocalDNSFallback failed phase0 wiring in AzureLinuxV3, Ubuntu2204, and Ubuntu2404: pods using 169.254.10.11 resolved kube-dns to the VNet DNS forwarder IP (10.0.0.10) instead of this cluster's kube-dns ClusterIP (172.16.0.10). The E2E summary shows 117 passed, 0 flaky, 76 skipped, and 4 failed; the additional LocalDNSHostsPlugin failure matches the existing restart-budget repair item.
Likely cause / signature:e2e-localdns-fallback-clusterip-mismatch - PR-specific LocalDNS fallback wiring/cluster DNS resolution regression.
Confidence: High for PR-change-caused LocalDNS fallback validation failure because the failing scenarios and changed files are all LocalDNS fallback-specific.
Recommended owner/action: PR owner should fix fallback DNS wiring so .11 fallback resolves cluster names through the cluster kube-dns ClusterIP; keep the separate restart-budget failure under repair #39909321.
Strongest alternative: existing LocalDNS restart-budget flake also occurred in this run, but three failed LocalDNSFallback scenarios share a new feature-specific phase0 kube-dns IP mismatch.
Evidence:build run, failed timeline record Run AgentBaker E2E, focused E2E log, scenario summary, PR LocalDNS fallback changed files, and wiki tracker.
Posting this before running it, because the first step is a blocker that the CI e2e already hit and it changes what the rest of the run can prove.
Blocker: COREDNS_SERVICE_IP resolves to 10.0.0.10 on a custom-CIDR cluster
The e2e run on df230a17 failed phase0-wiring on all three distros with:
... [10.0.0.10], not this cluster's kube-dns ClusterIP 172.16.0.10
-- pods using 169.254.10.11 cannot resolve cluster names
This is not an aks-rp problem. AgentBaker's e2e never calls aks-rp — it builds the bootstrap config itself — so ADO PR 17204945 (merged 2026-09-21, which populates ClusterNetworkConfig.CoreDnsServiceIp on the ANC builders) cannot affect this lane.
Traced it properly. Two separate defects:
1. e2e harness.e2e/scenario/aks_model.go:134 creates the cluster with DNSServiceIP: "172.16.0.10", but baseTemplateLinux() sets DNSServiceIP: "" at node_config.go:450 and :526. baseTemplateWindows() at :923 sets it correctly. So the Linux lanes hand AgentBaker an empty value.
Worth noting the scenario's own header says it runs "on a default service CIDR. A custom-CIDR variant is a follow-up" — but the e2e cluster is custom-CIDR. The scenario has been running the case it thought it was deferring, which is why it caught this.
2. Product gap on the CSE path — the same one 17204945 fixed for ANC.
It reads only the agent-pool-level KubernetesConfig and then hardcodes. There is no cluster-level fallback — and aks-rp's own getCoreDNSServiceIP has one, with a comment explaining exactly why it is load-bearing:
typeconversion.go only populates the pool-level KubernetesConfig when the HCP agent pool proto carries one, and the frontend inheritance that fills it (managedcluster/helper.go) runs on PUT only. The async provisioning path reads persisted goal state and never re-runs it, so the pool value can be unset.
So an empty pool-level DNSServiceIP is a production-reachable state, not a harness artifact — the e2e is reproducing a real condition. GetCoreDNSServiceIP is wired in getContainerServiceFuncMap(config *NodeBootstrappingConfiguration) (baker.go:1657), so the cluster-level value is in scope at that point; the method just can't see it because it hangs off *AgentPoolProfile.
Fixes needed before this run proves anything:
harness: set DNSServiceIP in baseTemplateLinux (mirror the Windows template) and CoreDnsServiceIp on the ANC config — CoreDnsServiceIp appears nowhere in e2e/ today except a comment
product: add the cluster-level fallback, and fail closed rather than guessing, per Ye Wang (@yewmsft)'s review item 2. A fallback that binds .11 and forwards to an address that does not exist is worse than not binding at all.
These interact: ship fail-closed without the harness fix and the e2e goes red differently (fallback never starts, phase 1 fails). Harness first, then product.
The plan, once the above is fixed
Instrument first — a canary pod created before the fault. This is what the current e2e does not do, and it is what measures the headline claim. The PR says the outage drops from ~16 min to ~43 s; nothing measures it, and the e2e only asserts recovery inside a wait_for 30.
On the node, probe continuously from inside the canary's netns (same nsenter technique as validators_localdns_fallback.go, so it sees exactly what the pod sees):
The crash path matters separately because the dummy interface survives with .11 still assigned, which is why ensure_cluster_listener_interface() has to be idempotent. A clean stop takes the other cleanup path (cleanup_localdns_configs deletes the link), so testing only that would miss the one that actually occurs in production.
Assertions at each step:
sudo ss -lunpH | awk '$4=="169.254.10.11:53"'# owner flips to the fallback
systemctl is-failed localdns.service # failed
systemctl is-active localdns-fallback.service # active
dig +short @169.254.10.11 kubernetes.default.svc.cluster.local # answers
dig +short @169.254.10.10 kubernetes.default.svc.cluster.local # SILENT, by design
getent hosts mcr.microsoft.com # node DNS fine via DHCP
Negative controls, without which the result means nothing:
a pod created during the outage must also be born on .11 and served by the fallback (the kubelet repoint is removed, so anything repointing kubelet is a bug)
the identical fault on a node without the fallback artifacts must show an unbounded gap — otherwise "the canary kept resolving" could just mean the fault never bit
restart localdns normally a few times afterwards and confirm the fallback does not engage; the 3x5s debounce should absorb a RestartSec=2 cycle
Recovery: restore the corefile, reset-failed + start, then assert ExecStartPre stopped the fallback, ss shows localdns.service again, .10 answers again, and the canary log shows at most a brief blip across the handoff. That window is the one thing the missing timeout on ExecStartPre could stretch (review item 4, still unfixed).
I will run this once the ClusterIP issue is resolved and report the measured numbers.
Ran it. Reproduced the failure locally, and traced it to the end.
Ran LocalDNSFallback/Ubuntu2204 against this PR's own VHD (buildId=184262078, SIG image 1.1791234323.13735) on subscription 8ecadfc9…. Identical failure to CI, at phase0-wiring, before any fault injection.
Full assertion output (CI truncated this)
PASS: localdns declares OnFailure=localdns-fallback.service
PASS: fallback declares no Conflicts= on localdns.service
PASS: fallback StartLimitIntervalUSec=0 (key is in [Unit])
PASS: dig is present, so the fallback probe can actually evaluate 169.254.10.11
FAIL: COREDNS_SERVICE_IP is 10.0.0.10 but this cluster's kube-dns ClusterIP is 172.16.0.10
FAIL: localdns's corefile forwards cluster DNS to [10.0.0.10], not this cluster's
kube-dns ClusterIP 172.16.0.10 -- pods using 169.254.10.11 cannot resolve cluster names
The wiring is all correct. Every structural assertion about the fallback passes. The failure is entirely about the upstream address.
This is bigger than the fallback
Note the second FAIL. It is not about COREDNS_SERVICE_IP or the fallback — it is about localdns's own corefile. On this cluster LocalDNS itself forwards cluster.local to an address that does not exist. Pod DNS is broken before any fallback is involved. The fallback inherits the wrong upstream because it derives from that corefile, which is the derivation working as designed on bad input.
Root cause
The scenario ran the scriptless/ANC path (DisableScriptless=false; aks-node-controller.output confirms it). On that path:
and the e2e never sets that field. grep -rn "CoreDnsServiceIp" e2e/ --include=*.go returns exactly one hit: the comment in scenario_localdns_fallback.go:18 saying a custom-CIDR variant is deferred. The cluster is created at 172.16.0.10 (aks_model.go:134), so the node silently gets the hardcoded default.
The CSE path has the same hole for a different reason: baseTemplateLinux() sets DNSServiceIP: "" at node_config.go:450 and :526, while baseTemplateWindows() at :923 sets it correctly. And GetCoreDNSServiceIP() (pkg/agent/datamodel/types.go:2748) reads only the agent-pool-level KubernetesConfig before hardcoding 10.0.0.10 — no cluster-level fallback, unlike aks-rp's helper, whose comment explains why that fallback is load-bearing:
typeconversion.go only populates the pool-level KubernetesConfig when the HCP agent pool proto carries one, and the frontend inheritance that fills it runs on PUT only. The async provisioning path reads persisted goal state and never re-runs it, so the pool value can be unset.
So an empty pool-level DNSServiceIP is production-reachable, not merely a harness artifact. Both lanes of this scenario are affected; ANC is simply the one that ran.
What this means for the PR
The scenario's header says it runs "on a default service CIDR. A custom-CIDR variant is a follow-up gated on the aks-rp CoreDnsServiceIp fix." The e2e cluster is not default-CIDR. The scenario has been exercising the custom-CIDR case it believed it was deferring — which is the only reason this surfaced.
Three things, in order:
Harness — set CoreDnsServiceIp on the ANC config and DNSServiceIP in baseTemplateLinux (mirror the Windows template). Needed before this scenario can test anything.
Product — GetCoreDNSServiceIP() should fall back to the cluster-level DNSServiceIP before hardcoding. The value is in scope: the template func is wired in getContainerServiceFuncMap(config *NodeBootstrappingConfiguration) (baker.go:1657); the method just cannot see it because it hangs off *AgentPoolProfile.
Fail closed, per Ye Wang (@yewmsft)'s review item 2. A fallback that binds .11 and forwards to an address that does not exist is worse than not binding. Given FAIL Add vhdbuilder and publisher #2, the same argument arguably applies to LocalDNS itself — silently serving a corefile that points at a non-existent ClusterIP is a worse failure than refusing to start.
Note these interact: shipping fail-closed without the harness fix turns this red a different way (the fallback never starts, phase 1 fails). Harness first.
The canary measurement in the plan above is still unrun — it is blocked behind step 1. I will post the measured outage window once the scenario can get past phase0.
Fixes the LocalDNS/Ubuntu2404 failure in the required VHD gate (build 185408461).
The fallback worked -- the node journal shows "applied (kube-proxy)" -- and
lookup-0 (cluster name via .11) succeeded. Only lookup-1 failed:
nslookup mcr.microsoft.com 169.254.10.11, which is the longest path in the
suite: pod -> .11 -> KUBE-SVC -> a CoreDNS replica -> its own .10 -> VNet DNS.
The cause was a regression af39081 introduced. The shell that version replaced
polled this exact check with 'expect_resolves 5'; the one-shot pod design has no
retry at all, so a single timeout anywhere along that path fails the scenario.
That is why it passed three local runs before failing in CI.
The container now retries ten times at 3s. The final attempt runs with output
attached, so nslookup's own message reaches the container log, which is what
TerminationMessageFallbackToLogsOnError and logPodDebugInfo surface on failure.
Also splits the probes into one name per pod. The build log shows why:
"Logs": "{\"status\":\"Failure\",\"message\":\"a container name must be
specified for pod ..., choose one of: [lookup-0 lookup-1]\"}"
logPodDebugInfo calls GetLogs without a container name and discards the error
(kube.go:259), so on a multi-container pod the 400 was stored as if it were the
logs -- leaving the failure with no nslookup output at all. Three
single-container pods (node-ext, cluster-svc, cluster-ext) fix that without
touching the shared helper, and the failing probe now names itself.
The unit test asserts both properties, so a regression to one-shot or to
multi-container fails without needing a cluster.
Verified live against this PR's VHD build, all four scenarios:
LocalDNS/Ubuntu2204 9 DNS pods, VM validation succeeded
LocalDNS/Ubuntu2404 9 DNS pods (the scenario CI failed on)
LocalDNS/AzureLinuxV3 9 DNS pods (first local run)
LocalDNSNoHostsPlugin/Ubuntu2204 4 assertions
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
azure-iptables-monitor runs on Cilium nodes, scans every iptables table against
an allowlist, and labels the CiliumNode kubernetes.azure.com/user-iptables-rules
for anything it cannot match -- which makes the RP reject enabling ACNS
Performance on that cluster. Today all nine of the fallback's rules are
unmatched, so a localdns outage on a Cilium node with host legacy routing would
trip it for the duration.
The allowlist already carries a global pattern for AgentBaker-managed rules,
and has since last year:
^.*--comment.*AKS managed: added by AgentBaker
so tagging these the way ensure_imds_restriction.sh tags its own costs nothing
and needs no aks-rp change. That matters: the cilium synth rolls out more slowly
than the VHD, so a new allowlist pattern would have had to land well ahead of
this to avoid a window where nodes carry rules the monitor does not know.
Verified by driving the real apply_rules in an isolated netns -- not a
reproduction of it -- and matching the output with Go's regexp, the same package
the monitor uses:
kube-proxy and cilium branches, 12 rules -> 0 unexpected
without the comment -> 9 unexpected
The hook jumps need the comment too, since the monitor checks PREROUTING and
OUTPUT, and that is the part with a trap: iptables -D matches on the full rule
spec, so clear_rules has to delete with the same comment it inserts with or the
jumps survive and keep redirecting past a recovered localdns. The netns run
confirms the chain and both jumps are gone after clear_rules, and phase 4
asserts the same thing on a real node.
Three shellspec examples pin the literal, the insert/delete symmetry and the
count, because getting the comment wrong fails silently: the rules install,
work, and are simply reported as user iptables rules. Eight existing
expectations needed updating for the new rule shape -- they caught the change,
which is what they are for.
Shellspec 35 examples, 0 failures. Verified live against this PR's VHD build on
LocalDNS/{Ubuntu2204,Ubuntu2404,AzureLinuxV3} and LocalDNSNoHostsPlugin: 9 DNS
pods each, fallback unit never entered 'failed'.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Detective summary: LocalDNS fallback scenarios on AzureLinuxV3, Ubuntu2204, and Ubuntu2404 failed because the validator could not find the expected top-level -A PREROUTING -j LOCALDNS-FALLBACK jump, while AgentBaker-managed fallback DNAT rules to 169.254.10.11 were present. A fourth scenario also repeated the LocalDNS NOTRACK exact-rule regex miss.
Likely cause / signature: e2e-localdns-fallback-prerouting-chain-missing plus e2e-localdns-notrack-rule-regex-missing - LocalDNS fallback NAT chain validation and recurring NOTRACK validator sensitivity.
Confidence: High for surface signatures: failed test records, E2E log summary, timeline, and PR LocalDNS fallback changes all point to fallback rule-generation/validation; medium-high on PR causality because PR 9486 directly changes LocalDNS fallback behavior.
Recommended owner/action: LocalDNS fallback owner should verify the LOCALDNS-FALLBACK chain/jump installation and adjust validator expectations for the AgentBaker-managed fallback rules.
Strongest alternative: Shared cluster networking flake, but multiple OS variants fail with the identical missing fallback-chain assertion in LocalDNS-specific validation.
Evidence:build run, failed test run 687633464, failed E2E timeline, PR LocalDNS fallback changes, prior LocalDNS rows, and wiki tracker.
Fixes LocalDNS/{Ubuntu2204,Ubuntu2404,AzureLinuxV3} in the required VHD gate
(build 185460351), broken by 11ae357.
validateLocalDNSNATRules asserted the hook jumps as exact lines:
-A PREROUTING -j LOCALDNS-FALLBACK
Once the rules carry -m comment --comment "AKS managed: added by AgentBaker ...",
that text sits between the hook and the -j, so the exact match fails. The chain's
own rules were fine -- only the two jumps are asserted this way.
Matched as patterns now, and both forms have to pass. A lane resolving a
main-built image runs a localdns-fallback.sh that predates the comment; the VHD
gate builds from the branch and gets one that writes it. The validator has to
work against either.
Why my local runs missed it: they pin VHD_BUILD_ID=184891026, an image built
from c12de4a, so the node ran the fallback script as it was before the comment
and the old assertion passed. With staging removed the node always uses the
VHD's copy, which means a local run can verify changes under e2e/ but CANNOT
verify changes under parts/ -- only a gate that builds the VHD from the branch
can. That is working as intended, and is worth knowing before trusting a local
pass on a shipped-code change.
The unit test now covers both forms, so this cannot regress without a cluster.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Enable errexit before mutating LocalDNS configuration
e2e/scenario/scenario_localdns_fallback.go:75
🟡 Medium Risk — 🔧 Script Logic: This fault test mutates /etc/localdns/environment without errexit. If the backup command or another setup command fails, execution can continue into the mutation and teardown may have no valid configuration to restore, leaving the scenario node broken for later validators. Enable -e; the intentionally tolerated commands already use conditions or || true.
🔴 High Risk — 🔧 Script Logic: This accepts an IPv6 nameserver, but the result is later passed to IPv4 iptables ... DNAT --to-destination. If an IPv6 resolver appears before a usable IPv4 resolver, activation fails even though fallback could use the later IPv4 entry. Filter this selection to IPv4 addresses and cover an IPv6-first resolver list in ShellSpec.
Rename scenarios to the documented OS-first format
e2e/scenario/scenario_localdns_hosts.go:34
The renamed scenarios still put the configuration before the OS (LocalDNS/Ubuntu2204), contrary to the required <OS/image>_<distinguishing configuration> format in e2e/README.md:33-47. Since exact names are selectors and have no aliases, use the documented OS-first form while performing this rename.
Run:185535615 Failed job/stage/task: e2e / Run AgentBaker E2E / AzureLinuxV3_PreinstalledBinaries and Ubuntu2204Gen2_ImagePullIdentityBinding_Disabled
Detective summary: This run had two known signatures: AzureLinuxV3_PreinstalledBinaries breached the AKS.CSE.ensureContainerd 2s threshold at 6.954s, and Ubuntu2204Gen2_ImagePullIdentityBinding_Disabled repeated the LocalDNS NOTRACK exact-rule validation miss. The first is generic CSE timing sensitivity; the second is LocalDNS validation/rule matching and is plausible PR-related for PR 9486.
Likely cause / signature: cse-ensurecontainerd-duration-threshold and e2e-localdns-notrack-rule-regex-missing.
Confidence: High for both surface signatures from failed test records, build report warning, E2E task/timeline data, and prior matching wiki rows; medium for PR causality only on the LocalDNS signature.
Recommended owner/action: LocalDNS owner should inspect NOTRACK validator/rule expectations in the fallback PR; CSE timing threshold is likely recurring gate noise unless it persists.
Strongest alternative: Shared infra latency caused both failures, but the NOTRACK failure is deterministic validator output while ensureContainerd is a known timing threshold flake.
Evidence:build run, failed test run 688783423, E2E task log 624, failed timeline, PR LocalDNS fallback changes, and wiki tracker.
…ssing
The check ran `echo "$rules" | grep -Eq "$rule"` under `set -o pipefail`.
grep -q exits on its first match, which closes the pipe and kills the
still-writing echo with SIGPIPE; pipefail then reports the whole pipeline
as failed. Once `iptables -t raw -S` exceeds the 64KB pipe buffer the
writer is forced to block mid-write, so a rule that is present -- and
found early -- gets reported missing.
Measured false failures on a healthy table: 0/100 at 50KB, 2/100 at 81KB,
100/100 at 204KB. Build 185535615 hit it on
Ubuntu2204Gen2_ImagePullIdentityBinding_Disabled with all 12 rules present
and the counts exactly right.
Replace the three counts and the 12 per-rule greps with one sorted set
comparison. awk and sort read to EOF, so no reader can orphan the writer.
Comparing the full set also rejects duplicate or unexpected extra rules,
which per-rule matching could not. On failure it now prints a diff and the
raw table size, since an oversized raw table is the one thing the old
output never revealed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
🟡 Medium Risk — 🔧 Script Logic: This accepts an IPv6 nameserver even though the fallback programs the IPv4 iptables table. If the restored resolver file lists IPv6 first (mixed resolver files are already supported by localdns.sh), --to-destination "${vnet_dns}:53" fails and the fallback never installs despite a usable IPv4 resolver later in the file. Select an IPv4 upstream here (or implement an explicit cross-family path) and add an IPv6-first fixture.
Run:185588062 Failed job/stage/task: e2e / Run AgentBaker E2E / Ubuntu2604Minimal_DisableKubeletServingCertificateRotationWithTags_AlreadyDisabled_CustomKubeletConfig
Detective summary: One E2E scenario failed during LocalDNS conntrack validation, repeating the repair-linked NOTRACK exact-rule regex miss in a LocalDNS fallback PR run. Likely cause / signature: e2e-localdns-notrack-rule-regex-missing - recurring LocalDNS NOTRACK exact-rule validator sensitivity.
Confidence: High: failed test record, E2E task log, timeline, and prior repair-linked wiki row all isolate the same LocalDNS validation family.
Recommended owner/action: Repair #40133693 tracks LocalDNS/E2E owner follow-up on raw-table NOTRACK rule generation and exact regex validator assumptions.
Strongest alternative: Generic infra flake is weaker because the failed assertion is deterministic LocalDNS raw-table validation and PR 9486 changes LocalDNS behavior. Evidence:build run, failed test run 689547422, failed E2E timeline, PR LocalDNS fallback metadata, repair #40133693, and wiki tracker.
The host-side probe was the only un-retried check in the fallback validator:
one `dig +timeout=3 +tries=1` piped into grep, with +tries=1 disabling dig's
own retry. The gate runs 60 scenarios in parallel with 72 of them sharing one
cluster and its two CoreDNS replicas, so a single attempt on that load
occasionally loses -- which failed LocalDNS/AzureLinuxV3 in build 185588062
while the pod probes, which retry ten times, passed seconds earlier.
Piping into grep also discarded dig's output, so the failure reported a bare
exit 1: no timeout, no SERVFAIL, nothing to diagnose from.
Retry in Go with the PollUntilContextTimeout already used twice in this file,
and send the node a bare dig with no shell logic at all. Using the helper that
does not assert the exit code keeps a timed-out attempt a retry rather than a
fatal error, so there is no errexit guard to get wrong, and the last answer is
captured in Go and reported in the error.
dnsAnswerHasAddress replaces the regex and is unit-tested off-node: 13 cases
covering CNAME-only answers, dig error text, IPv6, and CRLF. Mutation-tested --
dropping .To4(), dropping TrimSpace, or returning true each redden a case.
Measured on a live AzureLinuxV3 node with the fallback active: the path this
probes succeeded 60/60, so the structural concern that host traffic entering
KUBE-SVC skips KUBE-MARK-MASQ does not break resolution here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Following up on the iptables allowlist thread now that we have the full picture:
Please revert the allowlist change: 11ae357 (tag the fallback NAT rules as AKS managed: added by AgentBaker), plus the e2e hook-pattern loosening in 0d47b43 that exists only to accept both forms. No test exercises it. Nothing runs azure-iptables-monitor while the fallback is active, and the shellspec only pins the comment literal. So it adds surface without buying anything we can verify.
Adding the fallback rules to the Cilium allowlist (aks-rp allowed-iptables-patterns configmap) can come later, as explicit patterns. Nice to have, not required for this PR. The monitor's kubernetes.azure.com/user-iptables-rules label only matters for one thing: the RP rejects a PUT that turns on ACNS Performance for a cluster while any node carries it. The fallback rules exist only while localdns is down, and they're removed once it recovers. So a missing allowlist entry can only fail that one PUT during an outage, and the customer can retry once the node recovers. DNS itself is never affected.
The real bug is on ACNS Performance (eBPF host routing) nodes. The azure-block-iptables LSM that Cilium attaches rejects the raw-table NOTRACK adds in localdns.sh. The first boot works only because localdns adds its rules before the blocker attaches. Any later restart (manual, watchdog, re-apply) gets iptables: Permission denied and localdns ends up terminally failed, which kills pod DNS on the node. I reproduced this on 1.35 / AzureLinux / Cilium overlay / ACNS BpfVeth. It needs its own fix in a separate PR, not this one.
Reverts 11ae357 and 0d47b43, per review.
The --comment tag was added so azure-iptables-monitor's existing
"AKS managed: added by AgentBaker" allowlist pattern would match the
fallback's nat rules. Nothing in the suite exercises that: the monitor never
runs while the fallback is active, and the shellspec only pinned the comment
literal back to itself. It was surface we could not verify.
0d47b43 only existed to let the e2e accept hooks both with and without that
comment, so it goes with it. validateLocalDNSNATRules returns to exact-line
matching, which is the stricter check.
The allowlist entry can be added later as explicit patterns. A missing one
cannot affect DNS: the monitor's user-iptables-rules label only gates turning
on ACNS Performance, the fallback rules exist only while localdns is down, and
they are removed when it recovers -- so the worst case is one PUT failing
during an outage, retryable once the node is back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run:185670984 Failed job/stage/task: e2e / Run AgentBaker E2E / LocalDNS/AzureLinuxV3
Detective summary: LocalDNS/AzureLinuxV3 failed lifecycle validation after localdns was killed with Restart=no; the check reached DNS recovery and reported that the node could not resolve after localdns died.
Confidence: High: the failed test record, E2E log, timeline, adjacent LocalDNS scenario output, and PR 9486 LocalDNS fallback change scope all point at LocalDNS lifecycle behavior rather than generic infrastructure.
Recommended owner/action: LocalDNS fallback/lifecycle owner should inspect terminal-death DNS restoration and ExecStopPost cleanup behavior on AzureLinuxV3.
Strongest alternative: Shared E2E environment flake, but the failure is a deterministic LocalDNS lifecycle assertion while node provisioning and adjacent scenarios reached ready.
Evidence:build run, failed test run 690532741, E2E task log 624, failed timeline, PR metadata, and wiki tracker.
Build 185670984 failed LocalDNS/AzureLinuxV3 on a bare `systemctl restart`
at scenario_localdns_hosts.go:218 with one line of output:
Job for localdns.service failed because a timeout was exceeded.
Stdout was empty -- the script runs under `set -eu` and aborted mid-call
before printing anything -- so a start-limit refusal, a bind conflict and a
genuinely slow start are indistinguishable. The two journalctl calls already
in the script sit after line 218 and never ran.
Capture Result, the unit state and the last 50 journal lines on any failure
of this validator. Result alone separates the candidates: `timeout` means the
start budget was breached, `start-limit-hit` means StartLimitBurst was spent
and retries would not help, `exit-code` means localdns ran and failed.
TimeoutStartUSec is included because it is not set in the unit and the OS
default differs by distro -- 90s on Ubuntu, 45s on Azure Linux, measured on a
live node -- while the script's own readiness wait is 60s. On Azure Linux
systemd therefore wins that race and the script can never report its own
error, which fits the empty stdout and the distro-specific failure. That
remains a hypothesis; this is what would confirm it.
Evidence only: no shipped artifact changes, and nothing here alters node
behaviour.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Rechecked since my last approval: the allowlist-tagging revert brings localdns-fallback.sh and its spec back to their pre-11ae357d5b state with exact-line NAT matching, and the node dig retry and lifecycle diagnostics are test-only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ye Wang (yewmsft)
left a comment
What this PR does / why we need it:
Pods get
169.254.10.11or169.254.10.10written into/etc/resolv.confby kubelet at sandbox creation, and it is never revisited. LocalDNS does not DNAT —build_localdns_iptable_rules()installsraw-tableNOTRACKrules only — so when nothing is listening, queries are not rerouted anywhere. They fail.That makes a dead
localdns.servicea DNS blackhole nothing recovers from. AKS#5930 was ~16 minutes of it on a customer node, ended by manual intervention.This PR keeps both listeners working while localdns is in terminal
failed, using kernel NAT: one nat chain that rewrites the destination of DNS packets so they never arrive at the dead addresses in the first place.169.254.10.10:53udp/tcp169.254.10.11:53udp/tcpWhat landed before this
wait_for_localdns_removed_from_resolv_conf()no longer accepts an empty nameserver list as "refreshed".ExecStopPostrestores node DNS to the DHCP resolvers after an unexpected exit.RestartSec=2and a720/5budget, so unrecoverable failures terminate infailed. That terminal state is whatOnFailure=and this unit's gate key on.Do not attribute the #5930 fix to this PR — #9361 is the fix; this is the floor underneath failure causes nobody has enumerated.
Why NAT and not a second CoreDNS
An earlier revision ran a second CoreDNS against a corefile derived from localdns's own. That shares a failure mode with the thing it is backing up. If localdns died because the baked binary cannot parse a directive the RP sent, the fallback inherits the same binary and the same directive and dies with it.
This is reachable, not theoretical.
failfast_all_unhealthy_upstreamsrequires CoreDNS ≥ 1.12.1; measured across the five builds cached incomponents.json:A hard startup failure, so the fallback would have crash-looped every 5s forever on such a node. A fallback must not fail for the same reason its principal did. iptables rules cannot crash-loop or misparse a config.
Why both listeners, not just
.11Which address a pod gets depends on its
dnsPolicy:ClusterFirst→.11,Defaultand hostNetwork →.10. The second group includes CoreDNS itself, konnectivity-agent, the CSI node drivers, kube-proxy and azure-cns. Leaving.10dark black-holes all of them — and also breaks external resolution for.11clients, because that query goes.11→ ClusterIP → a CoreDNS replica on this same node → its own dead.10.Why
.11is not DNAT'd at the kube-dns ClusterIPThe
nattable is traversed once per connection andDNATis a terminal verdict for it. A rule rewriting.11to the ClusterIP consumes the traversalKUBE-SERVICESneeded to translate that ClusterIP into a pod IP, so the packet leaves addressed to a virtual IP nothing owns. On Cilium the service load balancer runs before netfilter, so a ClusterIP target is never translated at all.Going one hop further — straight into
KUBE-SVC-<kube-dns>, or at the backends the Cilium agent reports — lands on a real pod IP. It also means the fallback never needs the ClusterIP, which is why theCOREDNS_SERVICE_IPplumbing comes back out ofparser.go,baker.go,cse_cmd.shandcse_config_localdns.sh, along with its repro test.The gate
applyrefuses — exit 1, logged — unless localdns isActiveState=failed, an upstream other than.10exists in the restored resolv.conf, and either kube-proxy's kube-dns chains or a Cilium agent reporting active backends andEnableHostLegacyRouting=trueis found. With eBPF host routing, pod egress skips netfilter entirely and a nat rule would never see the packet.OnFailure=fires on every failed start attempt, not only the terminal one — measured on systemd 255.4, six firings across a five-restart budget. Five of those are starts where localdns is mid-restart and the only correct action is to do nothing.StartLimitIntervalSec=0stops that burst rate-limiting the one start that matters.Those no-ops are skipped, not failed.
ExecStart=has no exit code meaning "nothing to do" — exit 0 latches the unitactiveunderRemainAfterExit=and turns the one start that matters into a no-op — so the unit gates onExecCondition=/usr/bin/systemctl is-failed localdns.service, whose non-zero exit skips the unit without failing it. Without it, a node whose localdns flaked once and recovered was left withlocaldns-fallback.serviceinfailedpermanently, becauseExecStartPre'sstopis notreset-failed: a false alarm in the field, and a red scenario on the repo-wide failed-unit validator.applyre-checks the same state, so running the script by hand stays guarded.Recovery is unchanged:
localdns.service'sExecStartPrestops this unit, whoseExecStopflushes the chain. The rules are runtime-only, so a reboot leaves nothing behind.The systemd version split
ActiveState=failedmeans terminally failed on every version: through the whole restart budget the unit sits inactivating/auto-restart, measured on 249 and 255. What differs is how oftenOnFailure=fires.The fallback behaves identically on both; only how often it is asked to do nothing changes. That is why the gated no-ops are rare on 249 and routine on 255.
One deviation from the prototype, flagged for review
applypolls for the resolv.conf upstream rather than reading once, bounded at 10s.ExecStopPostremoves the drop-in and runsnetworkctl reload, which is asynchronous, andOnFailure=fires as soon asExecStopPostreturns — so the file can still hold the drop-in state (.10and nothing else) at that instant. The oneshot has noRestart=, andOnFailure=will not fire again once localdns is terminally failed, so a single early read would leave no fallback at all. Same racewait_for_localdns_removed_from_resolv_confexists for on the start path.Two things deliberately not done
.10clients losecluster.local. That is the same position they are in on a node with localdns disabled. This is a bridge that keeps DNS moving until the node is remediated, not a replacement for localdns.LocalDNSError, which is correct: once.10is served again,systemctl is-activeis the only remaining signal that anything is wrong. Note the division of labour — NPD only incrementsproblem_counter{reason="LocalDNSError"}(type: temporary, no NodeCondition); aks-operator'slocal-dns.rulesturn 5 hits in 10m intoRebootafter 5m andRedeployafter 15m; the remediator does the cordon, drain and reboot. localdns is dead and needs remediation; the fallback only keeps DNS working until that happens.Verification
Live clusters, against this PR's own VHD build (
VHD_BUILD_ID=184891026, so the lane resolves branch-built images rather than main's), kubenet / kube-proxy, scriptless and CSE paths.LocalDNS/Ubuntu2204LocalDNS/Ubuntu2404LocalDNSNoHostsPlugin/Ubuntu2204Ubuntu2404is the systemd 255 case, and it is the one that proves the gate works:Ten of eleven fires were skipped, and the unit never landed in
failedon a healthy node. On 2204 (systemd 249)OnFailure=fires once, so both halves of the version split are now covered by a real run rather than by argument.LocalDNSNoHostsPluginis the only scenario that drives a real failing startup oflocaldns.sh— a malformedLOCALDNS_COREFILE_BASEwithRestart=no. The node's journal confirms it reached the path under test:Exactly one occurrence, which is what
Restart=noshould produce.From a node's journal during the outage:
Also: ShellSpec 32 examples / 0 failures,
make validate-shellexit 0,go build+go vet+gofmtclean, 9 packer JSONs valid.How the e2e is organised
The fallback no longer has scenarios of its own.
LocalDNS/{Ubuntu2204,Ubuntu2404,AzureLinuxV3}already drove localdns to terminalfailedand recovered it, which is exactly the state the fallback needs — so the assertions run against that fault instead of manufacturing a second one on three more VMs. Net −2 VMs per lane and one fault per node.That required splitting the restart-budget validation:
localdnsTerminalFailedScriptcarried atrap teardown EXITthat restarted localdns the moment the limiter assertion passed, so the terminal state never survived. It is nowdriveLocalDNSToTerminalFailed, which deliberately exits with the unit failed, plusvalidateLocalDNSRestartBudgetDirectivesfor the static half. Restore is the caller's job and runs on every path.Ordering is deliberate: budget directives before the fault, so a budget regression reads as a budget regression rather than as a fallback that never activated; probe pods before it too, since their
resolv.confis written at sandbox creation and never revisited.LocalDNSHostsPluginwas renamedLocalDNS. The old name described the config, not the test — the hosts-plugin assertions live inValidateCommonLocalDNS, while the scenario's own validator ran lifecycle and budget. It now covers four things. Note this resets ADO test-result history for those scenarios.Static checks live in ShellSpec, not e2e: unit directives (
OnFailure=,ExecCondition=,Type=oneshot,RemainAfterExit=,ExecStop=,StartLimitIntervalSec=0, the absence ofConflicts=and of[Install], theExecStartPrehandoff) and rule generation (udp/tcp DNAT, the KUBE-SVC jump, Cilium spreading, never DNAT'ing at a ClusterIP, insertion order). Each was mutation-checked — every one of ten mutations reddens exactly one example. What stays on a node is what needs one: the chain installed in the kernel, both hooks,NOTRACKremoval, real DNS from pods, the systemd handoff, recovery, and thatExecCondition=actually gates rather than merely being present.The artifacts are no longer staged onto the node. Staging let the scenario run before they shipped, but once they do it is a mask: it repaired the exact gap the scenario exists to detect and then reported a pass — which is why the OSGuard mode split was invisible here. Absence now fails and names the VHD.
Still outstanding
kube_dns_chain()readsiptables -S KUBE-SERVICES. In ipvs mode the chain shape differs; in nftables mode the chains are not in iptables at all. In both,applyfinds neither a kube-dns chain nor a Cilium agent and refuses — the node is left as it was. Neither mode is tested.ValidateIPTablesCompatibleWithCiliumEBPFadditionally fails on a pre-existing, unrelated allowlist gap — a mangle-table-A FORWARD -d 168.63.129.16/32 -p tcp -j DROPwith no--dport, which no AgentBaker script writes. Being raised with acndp separately.cleanup_iptables_and_dnsdeletes theNOTRACKrules on exit and DNAT requires conntrack, so they cannot return while the fallback is active. Every pod DNS query creates a conntrack entry for the length of the outage — bounded by the remediation window.apply. The kube-proxy path follows endpoint changes because it jumps into a live chain; the Cilium path does not.localdns-fallback.servicemode differs by build path.packer_source.shinstalls it644,azlosguard.yml600, andcheckLocaldnsScriptsAndConfigsasserts644. This is the same splitlocaldns.servicealready has on main, not something this PR introduces — flagged rather than resolved here, since fixing it means either an OS-SKU-aware assertion or moving azlosguard's units off600.