Repository navigation
fix(agent): capture ScaleToZero wake frames in the runtime host netns - #106
Merged
Merged
Conversation
The wake-on-traffic hook opened its AF_PACKET socket inside the agent pod. The impbr-* bridges and VM TAPs live in the host network namespace (owned by the runtime DaemonSet, hostNetwork: true), so the pod-side socket only ever saw pod traffic (loopback, pod veths, Service IPs) — never the overlay frames a suspended ScaleToZero VM must be woken by. The nightly datapath E2E failure streak since 2026-08-29 shows it directly: framesSeen increments while lastDstIP is always 10.244.x/10.96.x/127.0.0.1, never the target VM's 10.45.0.x. Mirror the PR #99 pattern used for TAP stats: capture on the runtime (host netns) and expose it over the existing runtime RPC. - runtimeapi: WakeHits backend method + RPC (drains observed destination IPs) - noderuntime: host-netns AF_PACKET capture (ETH_P_ALL — preserved the ptype_all/br_handle_frame analysis; do not simplify to ETH_P_IP) - agent: NewLinuxScaleToZeroWithRuntime now wires a runtimeWakeSource that polls WakeHits at interval/2 instead of opening a pod-side socket
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The nightly datapath E2E has failed every night since 2026-08-29. The wake-on-traffic hook opened its AF_PACKET socket inside the agent pod, but the
impbr-*bridges and VM TAPs live in the host network namespace (owned by the runtime DaemonSet,hostNetwork: true). A pod-side socket can never see bridged overlay traffic — CI logs prove it:framesSeenincrements whilelastDstIPis always10.244.x/10.96.x/127.0.0.1, never the target VM's10.45.0.x. The wake frame is sent, never observed, the VM never wakes, the test times out.This is the same class of bug as #99 (TAP stats read from the wrong namespace), which moved stats to the runtime RPC. The wake capture was left behind.
Fix
Mirror the #99 pattern — capture on the runtime, consume via RPC:
WakeHitsbackend method + RPC — drains destination IPs observed on the host since the previous call.ETH_P_ALL— the ptype_all/br_handle_frameanalysis is preserved; IPv4 filtered in userspace). Capture starts lazily on first poll and lives for the process lifetime.NewLinuxScaleToZeroWithRuntimenow wires aruntimeWakeSourcethat pollsWakeHitsatinterval/2(wake registers well inside the 2-minute E2E resume window) instead of opening a pod-side socket.Verification
linux): pod-traffic-only poll must NOT signal the wake registry; the poll carrying10.45.0.3must signal it. Passes.go buildon linux and darwin;golangci-lintclean on the touched packages (no new issues vs main baseline).go test ./internal/agent/hits a pre-existing envtest/etcd sandbox failure identically on clean main; CI is the gate there.