[scotwells]

Requested by Scot Schuchert-Wells · project thread

Several compute controllers report reconcile errors for outcomes that a retry resolves, which inflates the reconcile error rate and puts healthy controllers on the error backoff. The outcomes are a write that lost to a newer version of the object, a wait for a projection or write-back to finish deleting, and a finalizer release on an object that is already gone. This change retries conflicts after a second and polls deletion waits every five seconds without reporting either as a failure, and treats an already-deleted object as done. A real error combined with one of these outcomes is still reported, and every absorbed retry is counted, so genuine failures still surface and stuck waits stay visible.

Details

• A shared helper applies to the workload, workload deployment (control plane and cell), federator, instance (cell), instance projector and internal DNS publisher reconcilers. It splits aggregated and joined errors at the first combining error on the wrap chain and absorbs them only when every member is routine. A mix of conflicts and removal waits polls on the removal interval.
• compute_reconcile_routine_retries_total{controller,reason} counts each absorbed retry, so a wait that never ends or a conflict loop shows as a steady rate. An alert on wait age is left as a follow-up.
• The federator's existing deletion-pending sentinel and 5s poll become the shared ones. The projector's projection wait, its DNS lifetime replacement wait and the cell instance's write-back wait now use them instead of plain errors.
• The finalizer-release update ignores NotFound, which a stale cache produces after the object has been released.

Test plan

• New unit tests cover the helper (conflicts, removal waits, mixed and nested aggregates, NotFound passthrough) and the federator's finalizer release racing a stale copy or a deleted object
• Existing projector tests updated to expect a timed requeue instead of an error
• Full unit and envtest suite passes locally, and golangci-lint reports no issues
• Once deployed, the reconcile error rate for these controllers falls while the routine retry counter picks up the same outcomes

🤖 Generated with Claude Code

[cla-assistant]

All committers have signed the CLA.

[cla-assistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[ecv]

Approve.

settleRoutineErrors only absorbs an error when every member of an aggregate or join is a conflict or the removal-pending sentinel, and a real error alongside either still surfaces; the tests cover mixed aggregate and join cases.

• Classification is sound for the shapes callers produce today: a wrapped real error, join(conflict, real), agg[conflict, real] and join(pending, real) all surface. Two holes: everyError uses errors.As(err, &agg), which finds the first Aggregate anywhere in the tree, so errors.Join(kerrors.NewAggregate([conflict]), real) or fmt.Errorf("%w; %w", agg, real) classifies as routine and drops real; type-switch on the root's Errors() / Unwrap() []error instead. And a real error stringified with %v next to a %w sentinel is lost by construction. No current controller nests these, so neither is live.
• A mix of one conflict and one removal-pending in a single aggregate is reported as an error (neither predicate holds for all members). Harmless, but it is the common shape if a Finalize collects both; consider a combined predicate.
• client.IgnoreNotFound on the post-finalization Update is safe: Updated only means the in-memory finalizer was dropped, and NotFound means the object is already gone, so no cleanup on another object is skipped (hub and downstream cleanup run inside Finalize before this write).
• A wait that never finishes now polls every 5s forever with no error signal: the pending branch logs at V(1) only, and the backoff that used to grow to minutes is gone. A stuck hub copy or instance projection (including the DNS projection replacement, which previously backed off) shows nothing on the dashboards. Same for a 1s conflict loop. Add a counter or an info log with the elapsed wait, or return the error after a bounded wait.
• In InternalDNSPublisherReconciler.Reconcile the deferred settle reads ctx after the timeout cancel has fired; harmless for logging, but pass the original context to be safe.

Follow-ups:

• Expose a reconcile_waits_total{reason} counter and an alert on a wait older than some threshold, so routine waits stay out of the error rate without becoming invisible.

[scotwells]

Addressed the review in a9eb498 and e7e276b:

• Combined errors are now split at the first aggregate or join on the wrap chain, rather than at the first aggregate found anywhere in the tree. Both of your examples, a joined nested aggregate beside a real error and %w; %w, now report the real error, and tests cover them. A real error stringified with %v is still lost, as you said, and nothing produces that today.
• An error that mixes conflicts and removal waits is now routine and polls on the 5s removal interval.
• Added compute_reconcile_routine_retries_total{controller,reason}, which counts every retry this absorbs, so a stuck wait or a conflict loop shows as a steady rate per controller. I left the wait-age alert as a follow-up.
• The DNS publisher's deferred settle now captures the context before the timeout wraps it.

---

Generated by Claude Code

[ecv]

Approve.

Both gaps from my last review are closed at e7e276b: routine retries are now counted, and a real error beside a nested aggregate is reported.

• everyError now walks the single wrap chain and splits at the first Aggregate or multi-%w/Join it meets, recursing into each member, so errors.Join(agg[conflict], real) and fmt.Errorf("%w; %w", agg[pending], real) both surface real; both cases are in the table test. A chain with no combining error falls back to matching the original error, as before.
• reconcile_routine_retries_total{controller,reason} is incremented on every absorbed retry, so a wait or conflict loop that never ends shows on dashboards without entering the error rate.
• A conflict mixed with a removal wait now polls on the 5 s removal interval rather than erroring, and the internal DNS publisher's deferred settle captures the context before the timeout wraps it.

Follow-ups:

• Add an alert on a sustained reconcile_routine_retries_total rate per controller (for example, a removal wait retrying for over 30 minutes) so a stuck wait pages, not only charts.