Repository navigation
kolla: always use the active backup path - #314
Merged
Merged
Conversation
mariadb_backup_target selects which script kolla-ansible runs for a
database backup. Neither of its two values is sound as shipped, so stop
deriving it from the load balancer and pin it to "active", which the
companion patch described below makes correct everywhere.
Why "replica" is broken. kolla-ansible runs mariabackup in a container
on mariadb_backup_host and copies the data directory of that same host.
The replica script then points mariabackup's SQL control connection
somewhere else on purpose: it asks through the VIP which node is
serving, and picks the first node that is not that one. Because
mariadb_backup_host defaults to the first host of the MariaDB shard
group, and both load balancers prefer that same first host, the node the
script excludes is normally the very node being copied -- so it does not
merely risk selecting a different node, it systematically does. That
matters because mariabackup is a physical backup tool: it takes its
target LSN over the SQL connection while its redo-copy thread reads the
local redo log. Galera replicates at writeset level rather than redo
level, so the two LSN sequences are unrelated, and the copy thread waits
until the local counter happens to reach a foreign value. Upstream's
2026-08-25 fix pins that control connection to the local node, which
makes the exclusion deterministic rather than incidental: the script
now always resolves the node being copied, and always picks another. Measured on a
3-node cluster: 30 s pointed at the local node, 17 m 30 s pointed at a
sibling 1.5 MB ahead, and unbounded as that offset grows.
Why "active" was only accidentally right. It sends mariabackup to the
VIP, which works only while the load balancer's chosen node happens to
be mariadb_backup_host. That coincidence held with HAProxy. It cannot
hold from 2025.2 on, because kolla-ansible removed the internal HAProxy
MariaDB frontend there: the VIP can then only be ProxySQL, and ProxySQL
announces its own version in the connection handshake (measured:
"8.0.11 (ProxySQL)"), which mariabackup rejects outright with
"Unsupported server version". Note this is a separate failure from the
stall above -- a fast, loud error rather than a hang.
Making ProxySQL announce the backend version would clear that handshake,
and on a default deployment it would even let "active" work through the
VIP: ProxySQL routes to the writer, the writer is the first shard host,
and mariadb_backup_host defaults to that same host. That is not an
adequate fix. It buys correctness only for as long as those three
coincide. A failover moves the writer, and an operator who sets
mariadb_backup_host to a non-writer -- the supported way to keep backup
load off the node serving traffic -- would have mariabackup copy one
node while taking its target LSN from another, reproducing exactly the
stall described above. Fixing the announcement is worthwhile in its own
right, but as a route to a working backup it trades a loud failure for a
latent, configuration-dependent one.
So the conditional has nothing worth deciding. Upstream now points
mariabackup at the node it backs up, which takes the load balancer out
of the control path altogether, and the commits this tag pins for
2025.1, 2025.2 and 2026.1 all carry it -- 2025.2 and 2026.1 are pinned
at that very commit.
Wherever this value changes behaviour, that fix is present. OSISM
forces enable_proxysql off for 2024.1, 2024.2 and 2025.1, so those
deploy HAProxy and the conditional already yields "active" there --
pinning it is a no-op. 2024.1 is inert regardless, defining neither
mariadb_backup_target nor the replica script. Only 2025.2 and 2026.1
move from "replica" to "active", and both connect to the local node.
No image patch is required.
The one combination this does not help is ProxySQL enabled by hand on
2024.2, where the template still renders host={{ database_address }}
and "active" would meet the handshake described above. That branch is
retired upstream and OSISM ships it security fixes only, so it will not
gain the local-connect change. Such a deployment has no working backup
path either way: "replica" stalls and "active" cannot connect.
Upstream reports for the two failure modes:
https://bugs.launchpad.net/kolla-ansible/+bug/2091559 (the stall)
https://bugs.launchpad.net/kolla-ansible/+bug/2080818 (the handshake)
Assisted-by: Claude:claude-opus-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stops deriving
mariadb_backup_targetfrom the load balancer and pins it toactive.Why
mariadb_backup_targetselects which script kolla-ansible runs for a backup.kolla-ansible runs mariabackup on
mariadb_backup_hostand copies that host'sdata directory, but the
replicascript then points mariabackup's SQL controlconnection somewhere else on purpose: it asks which node is serving and picks the
first node that is not that one.
Because
mariadb_backup_hostdefaults to the first host of the MariaDB shardgroup, and the load balancer prefers that same host, the node the script excludes
is normally the node being copied — so it does not merely risk selecting a
different node, it systematically does.
That matters because mariabackup is a physical backup tool: it takes its target
LSN over the SQL connection while its redo-copy thread reads the local redo log.
Galera replicates at writeset level rather than redo level, so the two LSN
sequences are unrelated, and the copy thread waits until the local counter happens
to reach a foreign value. Measured on a three-node cluster: 30 s pointed at the
local node, 17 m 30 s pointed at a sibling 1.5 MB ahead, and unbounded as that
offset grows.
Upstream's 2026-08-25 fix pins the control connection to the local node, which
makes the exclusion deterministic rather than incidental — the script now always
resolves the node being copied, and always picks another.
Why it is safe on every release
Wherever this value changes behaviour, the upstream local-connect fix is present:
mariadb_backup_target, no replica scriptactiveactivereplicaactive, pinned at the fix commit6112487bareplicaactive, pinned at the fix commit6e12d2250OSISM forces
enable_proxysqloff for 2024.1, 2024.2 and 2025.1, so those deployHAProxy and the conditional already resolves to
activethere. No image patch isrequired.
Upstream reports for the two failure modes:
2091559 (the stall) and
2080818 (the handshake).
🤖 Generated with Claude Code