Repository navigation
fix: wait for CN SQL handshake before publishing Working in CNClaim - #627
Merged
loveRhythm1990 merged 6 commits intoNov 2, 2026
Merged
loveRhythm1990 merged 6 commits into
loveRhythm1990 merged 6 commits into
Conversation
syncBindPod published WorkState_Working as soon as the pod lifecycle was Normal. MO listens on the SQL port before bootstrap but only starts accepting after it, so proxy could route new connections to a CN that cannot complete the MySQL handshake, surfacing as MySQL 2013 / handshake timeouts for clients. Probe the target pod IP directly for a MySQL v10 greeting before the first bind of a pod, and requeue quickly while it is not ready. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
Kruise moves a pod back to lifecycle Normal after an in-place update as soon as the container image changed, without waiting for the app. Gating the probe on pod name alone let a restarted, still-bootstrapping CN be republished as Working. Probe whenever the main container started at or after the last bind, and refresh the store bound time once verified. Also requeue after the short probe interval on the Bind path instead of surfacing the not-ready state as a reconcile error with backoff. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ariznawlll
approved these changes
Nov 2, 2026
The CNClaim controller runs a single reconcile worker. A CN that accepts TCP but has not started accepting SQL makes the handshake read wait the full 3s timeout, so probing inline stalled binding, sync, migration and deletion of every other claim during batch upgrades or scale-out. Run probes in the background with bounded concurrency and let reconcile only consume cached results keyed by pod UID and main container start time. Successful results are single-use and expire, and a failed result immediately triggers the next attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI keeps no operator logs, so a failed spec such as a CNClaim that never binds cannot be diagnosed. Print the operator logs and the remaining CN claims, pools and pods right after the suite fails, before the webhook outage scenario redeploys the operator. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
This pull request does not currently match the merge queue conditions, so it cannot be queued from here. The box comes back if it matches again. |
loveRhythm1990
enabled auto-merge (squash)
November 2, 2026 09:06
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CNClaim.syncBindPodpublishedWorkState_Working(and the tenant labels) to HAKeeper as soon as the pod's Kruise lifecycle wasNormal. That does not mean the CN can serve SQL:cloneset_update.go) movesPreparingNormal → Normalimmediately when noPreNormalhook is configured (the operator configures onlyPreDelete/InPlaceUpdate), andUpdating → Normalonce the container image changed. Neither waits for the application.net.Listen()before bootstrap but only starts accepting (and sending the MySQL greeting) after bootstrap, so the port is connectable while handshakes just hang.Workingwas published).Working, probe<podIP>:6001directly (not through any LB) and require a MySQL protocol v10 initial handshake packet. A bare TCP connect is not sufficient, and the 15spropagationDelayinmigrateruns afterWorkingis published so it does not help.Status.Store.BoundTime(in-place update / restart of a bound pod). After a successful probeStore.BoundTimeis refreshed, so steady-state re-syncs do not dial the pod.SyncandBindpaths requeue after 3s instead of publishingWorking; the claim stays un-Bound.MaxConcurrentReconciles, controller-runtime default 1), and a bootstrapping CN accepts TCP but makes the greeting read wait the full 3s timeout, so an inline probe would stall every other claim. Reconcile now only consumes a cached result. When none exists it starts a background probe and requeues.Scope / limitations
Workingis only published by CNClaim (cnstorepooling only patches readiness forBound/Idle). The non-pooling CNSet path (cnstore.defaultCNNormalReconcile) publishesWorkingwithout an SQL check too; that is left for a separate PR.Workingin HAKeeper is still routable during bootstrap; the operator does not touch the store in that case.CNSQLPort(6001). Reachability from the operator to CN pod IPs on 6001 (e.g. NetworkPolicy) was not verified on production clusters.Test plan
CGO_ENABLED=0 go test ./pkg/controllers/cnclaim/: handshake accepted / non-greeting rejected / listening-but-not-accepting times out / connection refused; probe decision table (never bound, other pod, restarted after bind, same-second start, verified);Syncrequeues after 3s on first bind and after an in-place restart;Syncdoes not probe an already verified CN;Bindrequeues after 3s and leaves the claim un-Boundcheckdoes not block on a hanging probe and dedupes in-flight probes; concurrency bound enforced; success consumed once; failure keeps the cause and retries immediately; stale success discarded; a new container start time does not reuse the old result;Syncreturns within 1s (3 consecutive calls) while the probe hangsgo test -race(3 runs)Test_Sync_doesNotBlockOnHangingProbe; removing the semaphore failsTest_asyncSQLProber_boundsConcurrencyBindmapping each make the corresponding new test failgofmt/go vetclean🤖 Generated with Claude Code