fix(router): requeue only the service route policies an event affects

docs/router/architecture.md

@@ -122,6 +122,7 @@ there instead.
 - **A VRF's routes follow its kernel table.** `applyVRFs` re-reads the table of every VRF it has applied on each reconcile, from one VRF listing. When the kernel VRF was recreated on another table, the VRF is applied again, the route-target index moves to the new table, and `withdrawMovedVRFRoutes` removes the routes of the VRF's best paths from the old table, deleting only entries that still point at the SID each path names (#1270). A VRF that is absent, or a listing that fails, keeps the table it has. A VPC's table normally survives recreation through its node reservation; see [VRF table IDs](../cni/architecture.md#vrf-table-ids).
 - **`ServiceVIPBinding`s are generated, one per node per backend.** `NetworkServiceBindingReconciler` (`internal/controller/networkservice_binding_controller.go`) runs on every node and writes a binding for each VIP of each accepted `NetworkService`, at most one per family (`ruleTranslatedVIPs`) and, for the IPv4 VIP, only while some gateway node's `<node>/IPv4Served` condition reports it loaded (`ipv4LoadedOnSomeNode`; a status change to that signal passes `ruleBindingInputsChanged`, and the node deletes its IPv4 bindings once no gateway does), and each backend the rule's `backendSelector` places on this node, through the same `selectRuleBackends` the gateway uses, so the gateway never sends a backend's flows to a node with no binding planned for it. Workloads are IPv6 only, so both VIPs' backends are the selected attachments' IPv6 addresses: an IPv4 VIP's binding pairs the IPv4 `vipAddress` with an IPv6 `backendAddress`, which the backend node serves by translating the flows (see [ARCHITECTURE-CNI.md](../cni/architecture.md#key-design-decisions)). `ServiceVIPBindingReconciler` refuses an IPv4 `backendAddress` with `BindFailed`. Each node writes and deletes only its own bindings (labelled `galactic.datum.net/binding-node` and `app.kubernetes.io/managed-by: galactic-router`), so there is one writer per object and no leader election. A binding is owned by its rule, records the rule's `vpcRef` (which names the VRF its rows go in) and takes `egressKind` from the attachment's interface mode. `ServiceVIPBindingReconciler` programs a binding only when a `BGPAdvertisement` on this node's `BGPRouter` with that VRF's ID has a prefix containing `backendAddress`, and otherwise reports `BindFailed`; it watches advertisements on its own router so a binding written before its advertisement converges, and teardown skips the check. A backend serves only the oldest rule (by creation time, then namespace/name) selecting it on the same backend port and protocol, since its node translates the backend's replies back to only one VIP; a claim (`backendClaim`) carries the VIP's family, since a reply to an IPv4 client takes the IPv4 VIP's egress-translated row and any other reply the IPv6 VIP's egress row, so one backend socket answers for one VIP of each family, an IPv4 rule never competes with an IPv6 one, and two rules with IPv4 VIPs on one socket do; both sides compute that owner with `ruleBackendOwners` over every rule in the namespace, so a newer rule gets no binding for the backend and no flows for it from the gateway, and every rule in a namespace is re-queued when one of them changes. Rules compete only within a VPC, so the per-VPC `VPCAttachment` fan-out still reaches every rule an attachment change can affect. The node summarizes its bindings on the rule as `<node>/BackendsBound` (`BackendsClaimed` names a backend an older rule serves); a rule whose spec cannot be built into bindings (a second IPv6 VIP, or a selector that does not parse) keeps the node's existing bindings, the same as a rule that loses `Accepted`, so an invalid edit does not drop the backend nodes' `vip_xlat_table` rows; a second IPv4 VIP does the same for the IPv4 family alone, writing no IPv4 binding and keeping the IPv4 ones it had while the IPv6 VIP's bindings stay current, and both report `InvalidRule`. Only the nodes that hold bindings for such a rule report `InvalidRule`, since the gateways already report the error and a condition from every node would be a status write per node per edit. Each rule reconcile lists only its own VPC's attachments, through the VPCAttachment→`status.vpc` field index (`listVPCAttachmentsInVPC`), and skips the cache's deep copy because nothing it does writes to them. Several backends of one VIP can share a node: the gateway writes each backend's slot into the uSID and `usid.c` keys the ingress row on it (#799).
 - **Each node probes its own backends and publishes their health.** For a `NetworkService` with `spec.healthCheck`, `BackendHealthReconciler` (`internal/controller/backendhealth_controller.go`) runs one TCP probe loop per backend on its node (`internal/healthprobe`), over the backend's host-side interface (the attachment's `status.hostInterface`) from that interface's link-local address, on `healthCheck.port` or the rule's `backendPort`. The link-local source is what lets the SYN-ACK back: `usid_egress` hands a reply to a link-local destination to the kernel before its VIP rewrite (see [ARCHITECTURE-CNI.md](../cni/architecture.md#key-design-decisions)). A decision moves after `unhealthyThreshold` consecutive failures or `healthyThreshold` consecutive successes; a probe that cannot be sent (no interface, no link-local address) counts toward `Unknown`/`ProbeUnavailable`, never toward `Unhealthy`. The decision goes into `status.health` of each of the backend's bindings, with the attachment UID and probed port, by a merge patch of that field alone and only when it changes, so it never races the `Bound` condition. Every binding of one backend shares one loop. A restarted router seeds each loop from the published decision when its attachment UID and port still match, so a restart changes no gateway's table. A rule losing its `healthCheck` clears `status.health` and stops the loops. The design is [docs/gateway/backend-health.md](../gateway/backend-health.md).
+- **A service route event requeues only the policies it can change.** `ServiceRoutePolicyReconciler` has one worker and each reconcile reads every attachment, so requeueing every policy for every event in the cell keeps its queue full and a new consumer waits a whole pass before its route is installed. `internal/controller/serviceroute_requeue.go` maps a `VPCAttachment` event, old and new object, to the policies that select it as a consumer (selector and `consumerVPCRef`) or whose endpoint selects it as a producer; a `ServiceEndpoint` or `ServiceEndpointProducerReport` event to the policies routing to that endpoint; and a `VPC` event to the policies whose `consumerVPCRef` names it. A policy whose selector does not parse, or whose endpoint is not cached, is requeued for every attachment event. Policy changes, and `BGPRouter` or `BGPVRFInstance` spec changes, still requeue every policy, since conflicts and SIDs are decided across all of them.
 - **A `Spread` service endpoint's producers are probed by their own node, and a consumer's route reads a producer set.** `ServiceProducerHealthReconciler` (`internal/controller/serviceproducerhealth_controller.go`) runs one `internal/healthprobe` loop per producer on its node of each `ServiceEndpoint` with `spec.distribution: Spread`, to the producer attachment's first IPv6 address on `healthCheck.port`, and server-side applies that node's `ServiceEndpointProducerReport` for the endpoint (at most 16 entries, owned by the endpoint) under its own field manager (`galactic-router/<node>`), so no object grows with the number of nodes. It deletes its report when it has no producers left, deletes reports of other nodes not renewed for 10 minutes, and removes its own entries from the endpoint's `status.producers`, where routers that predate reports write. Each node renews its entries every 15 seconds (`renewTime`); an entry not renewed for 60 seconds is expired, drains nothing, and ranks a remote producer after every currently reported one (`serviceplan.ProducerReportCurrent`). `ServiceRoutePolicyReconciler` requeues for the next expiry and ignores report and endpoint updates that only renew current entries. `ServiceRoutePolicyReconciler` reads them, through `serviceplan.WithProducerReports`, which keeps, for each producer and node, the more recently renewed of its report entry and its `status.producers` entry: for a Spread endpoint, `serviceplan.Compile` emits one `LocalSet` intent per consumer holding every producer on the consumer's node, `Unhealthy` ones marked draining unless all are, and, while one is currently `Healthy`, those with no current result (`Unknown` or unreported) marked draining too. With `PreferNodeLocal`, a consumer whose node's producers include one currently `Unhealthy` and none currently `Healthy` or with an expired report gets a remote producer instead, as one on a node with no producer does, when another node has a producer currently `Healthy`, `Unknown` or unreported (`selectProducers`, shared by `serviceplan.ProgrammingPlan`). `EBPFRouteProgrammer` writes it as a mode-3 `service_route_table` row plus a `service_producer_set_table` row under the same key; `usid_egress` pins each TCP flow in `service_flow_table` and picks a producer per UDP datagram (see [docs/router/private-service-routes.md](../private-services/README.md#several-producers-on-one-node)). A changed `LocalSet` intent is applied before the old one is removed, and the programmer keeps the newest of a key's producer sets in the map, so a health or membership change never drops the consumer's route. A datapath without the producer set map gets `serviceroute.ErrProducerSetsUnavailable` for `LocalSet` intents only.
 - **Surviving routers clean up after a departed node.** Only a node's own `NetworkServiceBindingReconciler` clears its `<node>/BackendsBound` conditions and deletes its generated bindings, and only its own `ServiceVIPBindingReconciler` removes the teardown finalizer from bindings targeting it. So once a node's `Node` object is deleted, `NodeDepartureReconciler` (`internal/controller/node_departure_controller.go`) on every other router removes that node's `BackendsBound` condition from every `NetworkService`, deletes its generated bindings and removes the finalizer from every binding whose `targetRef.name` names it, hand-written ones included; the kernel rows went with the node. Without it the conditions stayed forever and a deleted rule's bindings stayed `Terminating`. Every Node create and delete maps to one sweep key, so the sweep runs on the initial Node list (startup), on every Node delete and every `--gc-interval`, and a router down during the delete still converges. A node missing from the cached Node list counts as departed only once an uncached `Get` returns NotFound. Every router sweeps, with no leader election: status writes and finalizer patches retry on conflict (the patch carries the resourceVersion) and NotFound counts as done.
 - **Deleting a `ServiceVIPBinding` does not need its VRF on the node.** A binding's two `vip_xlat_table` rows are keyed on its VRF's uSID Argument. `ServiceVIPBindingReconciler` reads it from the `BGPVRFInstance` its `vpcRef` names on this node, and that is gone once the VPC leaves the node. Before #750, deletion re-derived it, failed, held the finalizer forever and left both rows in the map. Teardown now finds the rows by value with `vipxlatmap.UnregisterBinding`: since #764 each key carries the VIP or backend address, and since #799 the ingress key also carries the backend's slot, so the binding's ingress row (its VIP, port and slot, rewriting to its backend) and egress row (its backend and port, rewriting to its VIP) are found exactly, under any Argument. Each row is removed only while it still holds the binding's values, so a row another binding has since written stays. When the VRF context still resolves, `UnregisterBindingAt` applies the same checks at that location too. Neither pass deletes a row without first reading it, and a process-wide lock keeps a bind from landing between the read and the delete. A value check cannot tell two bindings with identical values apart, so while such a twin is live on the node, deleting one removes nothing. Rows exist only for an IPv6 backend, so an IPv4 or unparseable `backendAddress` skips both passes instead of blocking deletion. Only a failed kernel delete keeps the finalizer. Rows left behind some other way, by a finalizer cleared by hand, a crash or a manual write, are removed by the periodic sweep below; to remove one sooner, use `galactic-router vip xlat list` and `vip xlat remove --protocol --vip --port --backend --backend-port` (`--slot` defaults to the backend's own), run in that node's `galactic-router` pod (`kubectl exec <pod> -- /galactic-router vip xlat list`).

internal/controller/serviceroute_controller.go

@@ -30,6 +30,7 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/log"
 	"sigs.k8s.io/controller-runtime/pkg/manager"
 	"sigs.k8s.io/controller-runtime/pkg/predicate"
+	"sigs.k8s.io/controller-runtime/pkg/source"
 
 	cloudv1alpha1 "go.datum.net/galactic/api/cloud/v1alpha1"
 	networkv1alpha1 "go.datum.net/galactic/api/network/v1alpha1"
@@ -64,6 +65,12 @@ type ServiceRoutePolicyReconciler struct {
 
 	mu      sync.Mutex
 	Applied map[types.NamespacedName]map[serviceroute.IntentSlot]serviceroute.RouteIntent
+
+	// conflicted is the conflict set the last reconcile computed, and
+	// requeue carries the policies that entered or left it since
+	// (noteConflicts).
+	conflicted map[types.NamespacedName]bool
+	requeue    chan event.GenericEvent
 }
 
 // compile compiles policy for node with this router's options.
@@ -228,6 +235,7 @@ func (r *ServiceRoutePolicyReconciler) reconcilePolicy(
 		*errorReason = serviceroute.ReconcileConflictLookup
 		return ctrl.Result{}, errors.Join(conflictErr, r.removePolicyLocked(req.NamespacedName))
 	}
+	r.noteConflicts(req.NamespacedName, conflicts)
 	if conflicts[req.NamespacedName] {
 		var errs []error
 		for key := range conflicts {
@@ -346,9 +354,12 @@ func (r *ServiceRoutePolicyReconciler) SetupWithManager(mgr ctrl.Manager) error
 	})); err != nil {
 		return fmt.Errorf("add initial service route policy map sync: %w", err)
 	}
+	r.requeue = make(chan event.GenericEvent, 1024)
 	return ctrl.NewControllerManagedBy(mgr).
 		For(&networkv1alpha1.ServiceRoutePolicy{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})).
-		Watches(&networkv1alpha1.ServiceRoutePolicy{}, handler.EnqueueRequestsFromMapFunc(r.attachmentPolicies),
+		// A policy another policy's reconcile moved into or out of conflict.
+		WatchesRawSource(source.Channel(r.requeue, &handler.EnqueueRequestForObject{})).
+		Watches(&networkv1alpha1.ServiceRoutePolicy{}, handler.EnqueueRequestsFromMapFunc(r.policyEventPolicies),
 			builder.WithPredicates(predicate.GenerationChangedPredicate{})).
 		// A renewal of producer entries that are still current changes no
 		// route; expiry is handled by requeueing.
@@ -360,8 +371,10 @@ func (r *ServiceRoutePolicyReconciler) SetupWithManager(mgr ctrl.Manager) error
 			builder.WithPredicates(predicate.Funcs{UpdateFunc: func(e event.UpdateEvent) bool {
 				return !producerReportsOnlyRenewed(e.ObjectOld, e.ObjectNew, time.Now())
 			}})).
-		Watches(&cloudv1alpha1.VPCAttachment{}, handler.EnqueueRequestsFromMapFunc(r.attachmentPolicies)).
-		Watches(&cloudv1alpha1.VPC{}, handler.EnqueueRequestsFromMapFunc(r.attachmentPolicies)).
+		// An attachment or VPC event requeues only the policies it can change
+		// (serviceroute_requeue.go).
+		Watches(&cloudv1alpha1.VPCAttachment{}, r.attachmentEventPolicies()).
+		Watches(&cloudv1alpha1.VPC{}, handler.EnqueueRequestsFromMapFunc(r.vpcPolicies)).
 		// Only the spec of a BGPRouter or BGPVRFInstance feeds an attachment's
 		// SID. Their status, and the router's config-hash annotation, change
 		// on nearly every BGPRouter reconcile, and each change would requeue
@@ -635,14 +648,6 @@ func (r *ServiceRoutePolicyReconciler) withProducerReports(ctx context.Context,
 	return serviceplan.WithProducerReports(endpoint, reports.Items), nil
 }
 
-func (r *ServiceRoutePolicyReconciler) endpointPolicies(ctx context.Context, obj client.Object) []ctrl.Request {
-	return r.allPolicies(ctx)
-}
-
-func (r *ServiceRoutePolicyReconciler) attachmentPolicies(ctx context.Context, _ client.Object) []ctrl.Request {
-	return r.allPolicies(ctx)
-}
-
 // routingDependencyChanged passes the BGPRouter and BGPVRFInstance events that
 // can change an attachment's SID: creation, deletion, and a spec change.
 var routingDependencyChanged = predicate.GenerationChangedPredicate{}