[scotwells]

Requested by Scot Schuchert-Wells

Related: datum-cloud/enhancements#921, datum-cloud/compute#630.

Problem

When an instance is deleted and recreated under the same name, its new attachment can be selected by a service route policy before the new host interface exists, or before the CNI has registered its attachment identity. The router treated that as a programming error. It also stopped at the first error, so other new consumers of the same policy on that node weren't programmed until a retry succeeded. The router logged "resolve consumer interface ...: Link not found" or "lookup attachment identity ...: key does not exist" until the interface appeared, which can trip the service route programming alert.

Measured in a test cell: every same-name redeploy produced 10 to 14 such errors in the 0.3 to 0.7 s before the new interface came up. First deploys of a name produced none.

Change

• Hold, don't fail: EBPFRouteProgrammer.Apply holds a consumer whose interface is missing, or has no attachment identity, with nothing installed. This is what repair already does for an interface that disappears. The next repair installs it once the interface is ready. serviceroutemap.IdentityToken now wraps ErrNoAttachmentIdentity so that case can be told apart.
• Event-based install: galactic-router watches the node's interfaces with the existing linkWatch. On a change, it requeues the policies whose applied intents use that interface.
• Fallback for a missing identity: the CNI registers an interface's attachment identity after the interface appears, and that write sends no event. So a policy that holds a consumer is also rechecked after 1 s, doubling to the 30 s resync while the hold lasts. The router logs when a policy starts holding. Policies that hold nothing keep their normal requeue.
• Don't stop at one failure: replacePolicyLocked tries every intent of a policy and returns the failures together.

A held consumer counts as applied. For a frontend policy, the node's programming acknowledgment can then report ready while that one consumer's interface isn't there yet. Until the interface exists, that consumer has nothing to serve.

Testing

• New tests cover:
  • holding a consumer for a missing interface and for a missing identity, then installing it on repair
  • applying past a failing consumer
  • the in-place LocalSet replacement when every producer is gone
  • the per-policy recheck backing off from 1 s to 30 s, and not affecting policies that hold nothing
  • interface events requeueing only the policies that use the interface
• go build ./..., golangci-lint (0 issues) and go test -race pass for ./internal/serviceroute/, ./internal/controller/, ./internal/plumbing/ebpf/serviceroutemap/ and ./cmd/galactic-router/ in a Linux build container.
• Not yet run in a lab. After rollout, the check is that same-name redeploys log no "Link not found" programming errors, and that their DNS lookups still answer on the first try.

[ecv]

Approve. A failed Apply rolls back its own entries and only successes enter current, so going past a failing intent leaves nothing half-installed, and real failures still reach the programming alert. ErrNoAttachmentIdentity classifies through recoverableEntries, which apply, rebuild and repair all share.

Follow-ups:

• A held consumer counts as a successful apply, sets Ready=true and adds to the programmed gauge, while GalacticServiceRouteProgrammingFailing fires only on result="error". A consumer whose identity never arrives is held at the 30 s recheck forever with only the one "holding" log. Add a held-intents gauge with an alert for a hold longer than a few minutes, and leave held intents out of the programmed gauge.
• When Cleanup fails, replacePolicyLocked still returns early and skips the remaining intents, contrary to its new comment.