Repository navigation
kolla: Derive cinder_cluster_name for RBD-only HA - #329
Merged
Merged
Conversation
kolla-ansible leaves cinder_cluster_name empty. Its cinder precheck
fails when cinder-volume has more than one host and no cluster name is
set, but OSISM does not run prechecks, so every deployment with several
control nodes runs one independent cinder-volume service per host. Each
volume is bound to the service that created it (host@backend), and
every operation on it is sent to that host's queue. While that host is
down, the volume cannot be extended, snapshotted, attached or deleted;
requests wait in the queue until it comes back.
The RBD driver supports active/active (SUPPORTS_ACTIVE_ACTIVE), so
these services can form one cluster, which kolla-ansible configures
through cinder_cluster_name. With a cluster, any member serves the
volumes of all members.
Derive it: use ceph_cluster as the name when cinder-volume has more
than one host, cinder_backend_ceph is on, and no other kolla backend in
cinder_backends (LVM, NFS, Pure, ...) is enabled, since those are not
known to support active/active. In every other case the value stays ""
as upstream, which also keeps kolla's single-host precheck satisfied.
Existing deployments are adopted on the next cinder deploy or upgrade:
when cinder-volume starts with a cluster name for the first time,
init_host includes all volumes, groups and image-cache entries of its
host in the cluster (_include_resources_in_cluster). Each host adopts
only its own resources, so volumes of a host whose cinder-volume does
not start stay outside the cluster until it does.
Tested on an OSISM 11.0.1 cluster with three control nodes and
Ceph-backed cinder:
- Before: the cinder precheck fails ("Multiple cinder-volume instances
detected but cinder_cluster_name is not set"). With the owning
cinder-volume stopped, extending its volume stays in "extending" for
minutes and completes only after the service returns.
- After setting this expression and running osism apply cinder: the
precheck passes, all three services report cluster ceph@rbd-volumes,
cinder-volume logs that the existing volume was included in the
cluster, and with the owning cinder-volume stopped, extending the
same volume completes in seconds; snapshotting it and creating a new
volume work as well.
- The same on a second cluster with osism apply -a upgrade cinder
instead of a deploy: the volume is adopted, and with its owner
stopped, extending and snapshotting it complete in seconds.
The expression was also evaluated for one host, no cinder-volume
group, Ceph plus LVM, no Ceph, and a site override.
Limitation: a backend configured only through an overlay, outside
kolla's backend switches, is invisible to the condition. A site with
such a backend that does not support active/active sets
cinder_cluster_name: "".
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
ideaship
marked this pull request as ready for review
October 16, 2026 15:44
This was referenced Oct 20, 2026
ideaship
added a commit
to osism/container-images
that referenced
this pull request
Oct 21, 2026
VolumeRetypeWithoutMigrationTest.test_available_volume_retype retypes a volume with migration_policy='never' and asserts that os-vol-host-attr:host is unchanged. That only holds while cinder-volume is not clustered. Since osism/defaults derives cinder_cluster_name for RBD-only HA deployments (osism/defaults#329), kolla renders `cluster =` for cinder-volume, and the test fails at random on such deployments: testtools.matchers._impl.MismatchError: !=: reference = 'testbed-node-0@rbd-volumes#volumes' actual = 'testbed-node-2@rbd-volumes#volumes' Cinder behaves as intended here. find_retype_backend() matches the volume's cluster, and on a successful in-place retype VolumeManager.retype() writes the selected backend's host and cluster_name back to the volume. All members of a cluster share one backend state in the scheduler, whose host is whichever member that scheduler instance saw first, so the recorded host can be any cluster member while the volume stays on the same backend. The code dates from 2016/2017 and is unchanged on stable/2026.1. Carry a patch that compares cluster_name and only compares the host when the volume is not clustered. cluster_name is always compared, so a volume that gains or loses a cluster still fails the test. Cinder returns cluster_name in the admin volume details only from microversion 3.61, and returns it as null when the volume is not clustered, so the test class now requests 3.61. Tempest's 3.61 response schema types cluster_name as uuid_or_null, but the uuid format is not enforced, so real cluster names such as 'ceph@rbd-volumes#volumes' validate. Limitation: the test is now skipped where [volume] max_microversion is unset or below 3.61. The validations role's tempest.conf sets min_microversion 3.0 and max_microversion latest, so it keeps running there. An upstream fix should keep a host-only path for such clouds. The upstream bug is https://bugs.launchpad.net/tempest/+bug/1863729. This follows the approach of the abandoned upstream change https://review.opendev.org/c/openstack/tempest/+/708332, which cannot be used as is. The alternative upstream change https://review.opendev.org/c/openstack/tempest/+/712077 adds a config option that drops the host assertion without replacing it. Drop this patch once requirements.txt moves to a tempest release that fixes the bug; the Containerfile's dry-run will then fail the build and force its deletion. Verified offline against tempest 46.3.0: the patch applies with the Containerfile's dry-run/apply sequence, and a second dry-run is rejected as already applied. Driving _verify_migration with fabricated volume responses, the unpatched test fails on the data above. The patched test passes when the host moves within a cluster. It fails when the cluster changes or is lost, when an unclustered volume's host changes, and when an unclustered volume gains a cluster. The class requests 3.61 with the validations role's microversion range. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Roger Luethi <luethi@osism.tech>
berendt
pushed a commit
to osism/container-images
that referenced
this pull request
Oct 22, 2026
VolumeRetypeWithoutMigrationTest.test_available_volume_retype retypes a volume with migration_policy='never' and asserts that os-vol-host-attr:host is unchanged. That only holds while cinder-volume is not clustered. Since osism/defaults derives cinder_cluster_name for RBD-only HA deployments (osism/defaults#329), kolla renders `cluster =` for cinder-volume, and the test fails at random on such deployments: testtools.matchers._impl.MismatchError: !=: reference = 'testbed-node-0@rbd-volumes#volumes' actual = 'testbed-node-2@rbd-volumes#volumes' Cinder behaves as intended here. find_retype_backend() matches the volume's cluster, and on a successful in-place retype VolumeManager.retype() writes the selected backend's host and cluster_name back to the volume. All members of a cluster share one backend state in the scheduler, whose host is whichever member that scheduler instance saw first, so the recorded host can be any cluster member while the volume stays on the same backend. The code dates from 2016/2017 and is unchanged on stable/2026.1. Carry a patch that compares cluster_name and only compares the host when the volume is not clustered. cluster_name is always compared, so a volume that gains or loses a cluster still fails the test. Cinder returns cluster_name in the admin volume details only from microversion 3.61, and returns it as null when the volume is not clustered, so the test class now requests 3.61. Tempest's 3.61 response schema types cluster_name as uuid_or_null, but the uuid format is not enforced, so real cluster names such as 'ceph@rbd-volumes#volumes' validate. Limitation: the test is now skipped where [volume] max_microversion is unset or below 3.61. The validations role's tempest.conf sets min_microversion 3.0 and max_microversion latest, so it keeps running there. An upstream fix should keep a host-only path for such clouds. The upstream bug is https://bugs.launchpad.net/tempest/+bug/1863729. This follows the approach of the abandoned upstream change https://review.opendev.org/c/openstack/tempest/+/708332, which cannot be used as is. The alternative upstream change https://review.opendev.org/c/openstack/tempest/+/712077 adds a config option that drops the host assertion without replacing it. Drop this patch once requirements.txt moves to a tempest release that fixes the bug; the Containerfile's dry-run will then fail the build and force its deletion. Verified offline against tempest 46.3.0: the patch applies with the Containerfile's dry-run/apply sequence, and a second dry-run is rejected as already applied. Driving _verify_migration with fabricated volume responses, the unpatched test fails on the data above. The patched test passes when the host moves within a cluster. It fails when the cluster changes or is lost, when an unclustered volume's host changes, and when an unclustered volume gains a cluster. The class requests 3.61 with the validations role's microversion range. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Roger Luethi <luethi@osism.tech>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
kolla-ansible leaves
cinder_cluster_nameempty. Its cinder precheck fails when cinder-volume has more than one host and no cluster name is set, but OSISM does not run prechecks. So every deployment with several control nodes runs one independent cinder-volume service per host. Each volume belongs to the host that created it, and while that host's cinder-volume is down the volume cannot be extended, snapshotted, attached or deleted.The RBD driver supports active/active, so this derives the cluster name in
all/099-kolla.yml:ceph_clusterwhen cinder-volume has more than one host,cinder_backend_cephis on and no other kolla backend incinder_backendsis enabled. Otherwise the value stays""as upstream. A site can override it either way.Existing deployments are adopted on the next
osism apply cinderorosism apply -a upgrade cinder: when cinder-volume first starts with a cluster name, it moves its host's volumes into the cluster. Each host adopts only its own volumes.Tested on three-controller clusters with Ceph-backed cinder, through both the deploy and the upgrade action:
ceph@rbd-volumes, the existing volume is adopted, and with its owner stopped extend, snapshot and volume creation complete in seconds.Limitation: a backend configured only through an overlay is invisible to the condition. A site with such a backend that does not support active/active sets
cinder_cluster_name: "".🤖 Generated with Claude Code