Skip to content

fix: wait for CN SQL handshake before publishing Working in CNClaim - #627

Merged
loveRhythm1990 merged 6 commits into
matrixorigin:mainfrom
loveRhythm1990:lr90/cnclaim-wait-sql-ready
Nov 2, 2026
Merged

loveRhythm1990 merged 6 commits into
matrixorigin:mainfrom
loveRhythm1990:lr90/cnclaim-wait-sql-ready

Conversation

@loveRhythm1990

@loveRhythm1990 loveRhythm1990 commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • CNClaim.syncBindPod published WorkState_Working (and the tenant labels) to HAKeeper as soon as the pod's Kruise lifecycle was Normal. That does not mean the CN can serve SQL:
    • Kruise (checked v1.4.0, v1.5.0, v1.8.3, identical logic in cloneset_update.go) moves PreparingNormal → Normal immediately when no PreNormal hook is configured (the operator configures only PreDelete / InPlaceUpdate), and Updating → Normal once the container image changed. Neither waits for the application.
    • MO calls net.Listen() before bootstrap but only starts accepting (and sending the MySQL greeting) after bootstrap, so the port is connectable while handshakes just hang.
  • Proxy therefore could route a new connection to a CN that was still bootstrapping, and the backend handshake timed out (observed in prod as MySQL 2013 on a scheduled sync job, see matrixorigin/matrixflow#20241; the CN became ready ~24s after Working was published).
  • Before publishing Working, probe <podIP>:6001 directly (not through any LB) and require a MySQL protocol v10 initial handshake packet. A bare TCP connect is not sufficient, and the 15s propagationDelay in migrate runs after Working is published so it does not help.
  • The probe runs when the CN process may not have been verified yet: the pod was never bound by this claim, or its main container started at or after Status.Store.BoundTime (in-place update / restart of a bound pod). After a successful probe Store.BoundTime is refreshed, so steady-state re-syncs do not dial the pod.
  • While not ready, both the Sync and Bind paths requeue after 3s instead of publishing Working; the claim stays un-Bound.
  • Probes never run on the reconcile worker. The CNClaim controller has a single worker (no MaxConcurrentReconciles, controller-runtime default 1), and a bootstrapping CN accepts TCP but makes the greeting read wait the full 3s timeout, so an inline probe would stall every other claim. Reconcile now only consumes a cached result. When none exists it starts a background probe and requeues.
    • At most 8 probes run concurrently.
    • Results are keyed by pod UID + main container start time + address, so a restarted CN never reuses a previous result.
    • A success is single-use and expires after 10s.
    • A failed result immediately starts the next attempt.

Scope / limitations

  • For pooled pods, Working is only published by CNClaim (cnstore pooling only patches readiness for Bound/Idle). The non-pooling CNSet path (cnstore.defaultCNNormalReconcile) publishes Working without an SQL check too; that is left for a separate PR.
  • A crash-restart of a bound CN that never left Working in HAKeeper is still routable during bootstrap; the operator does not touch the store in that case.
  • The SQL port is the operator's fixed CNSQLPort (6001). Reachability from the operator to CN pod IPs on 6001 (e.g. NetworkPolicy) was not verified on production clusters.
  • The Kruise version actually running in production was not checked; this repo's chart pins kruise-manager v1.8.3.

Test plan

  • CGO_ENABLED=0 go test ./pkg/controllers/cnclaim/: handshake accepted / non-greeting rejected / listening-but-not-accepting times out / connection refused; probe decision table (never bound, other pod, restarted after bind, same-second start, verified); Sync requeues after 3s on first bind and after an in-place restart; Sync does not probe an already verified CN; Bind requeues after 3s and leaves the claim un-Bound
  • Async prober: check does not block on a hanging probe and dedupes in-flight probes; concurrency bound enforced; success consumed once; failure keeps the cause and retries immediately; stale success discarded; a new container start time does not reuse the old result; Sync returns within 1s (3 consecutive calls) while the probe hangs
  • go test -race (3 runs)
  • Mutation checks: making the probe synchronous hangs Test_Sync_doesNotBlockOnHangingProbe; removing the semaphore fails Test_asyncSQLProber_boundsConcurrency
  • Mutation checks: name-only gate, always-probe, and removing the Bind mapping each make the corresponding new test fail
  • gofmt / go vet clean
  • Not verified: end-to-end on a real cluster (no e2e run, no production-equivalent reproduction)

🤖 Generated with Claude Code

syncBindPod published WorkState_Working as soon as the pod lifecycle was
Normal. MO listens on the SQL port before bootstrap but only starts
accepting after it, so proxy could route new connections to a CN that
cannot complete the MySQL handshake, surfacing as MySQL 2013 / handshake
timeouts for clients.

Probe the target pod IP directly for a MySQL v10 greeting before the first
bind of a pod, and requeue quickly while it is not ready.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@qodo-code-review

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

Kruise moves a pod back to lifecycle Normal after an in-place update as
soon as the container image changed, without waiting for the app. Gating
the probe on pod name alone let a restarted, still-bootstrapping CN be
republished as Working. Probe whenever the main container started at or
after the last bind, and refresh the store bound time once verified.

Also requeue after the short probe interval on the Bind path instead of
surfacing the not-ready state as a reconcile error with backoff.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
loveRhythm1990 and others added 3 commits November 2, 2026 15:31
The CNClaim controller runs a single reconcile worker. A CN that accepts
TCP but has not started accepting SQL makes the handshake read wait the
full 3s timeout, so probing inline stalled binding, sync, migration and
deletion of every other claim during batch upgrades or scale-out.

Run probes in the background with bounded concurrency and let reconcile
only consume cached results keyed by pod UID and main container start
time. Successful results are single-use and expire, and a failed result
immediately triggers the next attempt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI keeps no operator logs, so a failed spec such as a CNClaim that never
binds cannot be diagnosed. Print the operator logs and the remaining CN
claims, pools and pods right after the suite fails, before the webhook
outage scenario redeploys the operator.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mergify

mergify Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

This pull request does not currently match the merge queue conditions, so it cannot be queued from here. The box comes back if it matches again.

@loveRhythm1990
loveRhythm1990 enabled auto-merge (squash) November 2, 2026 09:06
@loveRhythm1990
loveRhythm1990 merged commit 12dcef4 into matrixorigin:main Nov 2, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants