[scotwells]

Requested by Scot Schuchert-Wells

Part of datum-cloud/enhancements#921

Problem

When an update withdraws a context, zone or record, every member clears its approved configuration before installing the update. Readiness then fails with no approved configuration is active, and every member closes its health port at the same moment, because they all receive the same update. The router has no healthy member until the members verify the withdrawal, usually about a second later. If a resolver probe in the same pass is slow, that stretches to several seconds and lookups time out. Follow-up to #516, which fixed the false withdrawals but left real ones behaving this way.

Fix

The daemons keep answering from the old files until the new ones are live, so closing the health port during the install gave no safety.

• When a withdrawal starts, the replaced approval moves to a new checkpoint field, withdrawing, and active is still cleared as before.
• Readiness uses withdrawing only while a live agent holds the activation lock. A stopped agent, an abandoned activation or a fail-closed gate ends that.
• The agent takes the activation lock right before it saves a withdrawal it is about to install, and readiness checks once more if it sees a withdrawal without a live installer, so there is no moment where the member reads as unapproved. A withdrawal that changes no files drops the approval without the lock, as before, so the lock is never held for an install that doesn't happen.
• Failing closed now writes its own activation phase (gating), which readiness rejects. A member running its agent's gate is never ready, even while the agent holds the lock. The restart inside a structural transaction still runs under the installing phase; there the live canary decides, as before.
• Approval clears withdrawing.

Unchanged: restart (restoreActive), retention (canRetainActive) and older agents never read the new field, so a crash or rollback mid-withdrawal still fails closed rather than restoring withdrawn data.

Testing

• TestWithdrawalKeepsReadinessWhileInstalling (record and zone-detach cases): at the validate step, before the transaction takes its own lock, readiness serves the replaced approval. Afterwards the withdrawal is approved and nothing is left pending.
• TestFailedWithdrawalIsNotReadyWhileGating: a failed withdrawal is not ready while its gate runs or after.
• TestReadinessUsesReplacedApprovalOnlyDuringLiveWithdrawal: not ready without an installer, ready with one, not ready once it is abandoned.
• Mutation-checked: dropping the lock-first ordering or the gating check fails these tests. go test ./internal/internaldns/... passes (two macOS-only TestReloadDNSDist* helper tests skipped, as on main). Lint is clean.

[ecv]

Approve.

Readiness honours withdrawing only while an flock on activation.pending.lock is held, so a crash or hang cannot keep a member ready beyond the one-minute install grace. Restart and retention never read the new field, and every failing path that I traced either writes gating or leaves a marker that readiness rejects.

• Lock liveness is kernel flock, not a PID or lease. There is no PID reuse, a crashed agent drops it on exit, and a hung agent is bounded by activationInstallGrace (60s from the first marker write, preserved across phase writes in writeActivation). withdrawing is therefore honoured for at most about 60s.
• TOCTOU in servedApproval (readiness.go): installationReadiness is read before the checkpoint, and the one re-check when Active==nil && Withdrawing!=nil && !installing closes the save-then-lock gap. The ordering in reconcileDesired (lock, then save) is what makes that sound.
• restoreActive and canRetainActive only read Active, so a crash mid-withdrawal restarts through failClosed and writes gating. failClosed clears Withdrawing in memory only. The disk copy survives until the next Save, and reconcileDesired persists the cleared state before any Apply when Active==nil, so no install runs with a stale withdrawing on disk. Every other Apply site I found sits after that save or requires Active != nil.
• Rollback: Load does not use DisallowUnknownFields, so an older agent ignores the field and its next save drops it, and an older readiness fails on Active==nil as before.
• Mutation spot-check: removing the gating check fails TestFailedWithdrawalIsNotReadyWhileGating, as claimed. Removing the !installing re-check in servedApproval leaves the whole package green, so the claimed lock-ordering mutation guard does not cover that re-check.
• The gating tests use the validate and gate hooks, so they test the install window rather than a true concurrent save-versus-probe interleave.

Follow-ups:

• A withdrawal for a tenant or security reason now keeps every member in rotation, serving the withdrawn data, until its install finishes (seconds, at most about 60s with a stuck installer). Before, members left rotation first. This is the intended trade, but say so in the docs, and consider a withdrawal flag that opts out of it for tenant removal.
• The probe-failed gate path (verifyResolverBindings then tx.Apply(gated)) does not call failClosed, so it writes no gating phase. A member removing a failing binding now stays ready on the replaced approval for the grace window, whereas before it was unready. The body says a member running its agent's gate is never ready, so either gate that path too or narrow the claim.
• Add a test that interleaves a checkpoint save with a readiness read, to cover the re-check in servedApproval.
• If beginGate fails (disk error) while the install marker still says installing, the lock is held through the up-to-5s FailClosed and readiness stays ready for that time. This needs a disk fault, so it is low priority.