Skip to content

Configure the Octavia amphora provider driver and related networking in ci-multinode on OVS - #2685

Draft
saadtariq38 wants to merge 11 commits into
stackhpc/2026.1from
INFRA-830-Configure-Multinode-OVS-Octavia-Amphora
Draft

saadtariq38 wants to merge 11 commits into
stackhpc/2026.1from
INFRA-830-Configure-Multinode-OVS-Octavia-Amphora

Conversation

@saadtariq38

@saadtariq38 saadtariq38 commented Oct 14, 2026 •

Copy link
Copy Markdown

The current ci-multinode environment does not have the networking support
needed to run Octavia's Amphora provider driver

Octavia now picks its driver from a new octavia_amphora_driver_enabled
variable, which defaults to false. When it is false the environment keeps the
OVN provider driver and QEMU, exactly as before.

The amphora driver needs a few extra things, which this adds:

  • a management network for Octavia to reach the amphorae. It is a VLAN on the
    physnet we already have
  • KVM on compute hosts, since amphorae are far too slow under emulation and
    leads to timeout failures in tempest tests
  • a check that fails early if the host is not x86_64 or has no /dev/kvm -> This is specific to our current SMSLab env and support for nested virtual. Could be changed as needed in the future.
  • Tempest override files for the octavia and octavia-smoke load lists

Fix

The kernel was giving a frame to the VLAN subinterface first since it had the health manager, which would
accept and then drop it because the health manager lives on the controller
host, so Open vSwitch would never see it. Amphorae started up but Octavia could not
reach them, and they could not even reach DHCP

The fix was to put the VXLAN mesh device inside a Linux bridge and connect Open
vSwitch to that same bridge with a veth pair. A Linux bridge sends a copy of
the traffic to everything attached to it (broadcast), so the host and Open
vSwitch both get one and things work. All the VLAN subinterfaces were also
moved onto the bridge.

This also prevents a lot of similar problems that might have come up in future,
since the previous implementation seemed like a workaround. It fixes one such
problem already: stackhpc.vxlan and MichaelRigart.interfaces both wrote
vxlan<vni>.nmconnection, and Kayobe wrote it second as an ethernet profile,
which cannot create a software device. The mesh device would never reboot since ethernet profile could not recreate the device.

Applies to

By default, no CI job uses the amphora driver. octavia_amphora_driver_enabled
is off because the leaf cloud that runs the multinode jobs has no nested
virt. Every job therefore behaves as it does today.

Deployments that do have nested virtualisation can set the variable to true
and get the driver, the certificates, the amphora image and the Octavia Tempest
run with no further configuration.

Moving the multinode jobs to SMS and turning
this on was decided to be a separate ticket: https://stackhpc.atlassian.net/browse/INFRA-1972

Testing

Leafcloud is not testing Octavia at all currently. This work can not test it there either and so was
verified by hand on SMSLab with KVM enabled
octavia load list: 51 passed, 16 skipped, 0 failed

Regression test with OVN on the CI cloud, to confirm the networking change does
not affect existing jobs:

  • host configure completed on all 9 hosts
  • the flag correctly kept the OVN driver and QEMU
  • 257 of 258 non-skipped Tempest tests passed. The one failure,
    test_create_delete_image, timed out with the snapshot stuck in SAVING.
    seems to be multimodes usual failures on OVN/OVS periodics

Depends on/Linked to

(stackhpc/terraform-kayobe-multinode#131)

Future Work: https://stackhpc.atlassian.net/browse/INFRA-1972

@coderabbitai

coderabbitai Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: Repository: stackhpc/stackhpc-kayobe-config/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 6393520e-90d7-464b-a002-65c74e5f416d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: stackhpc/stackhpc-kayobe-config/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 5e731c81-7d36-4e21-bd49-ce22bee0a62f
📥 Commits

Reviewing files that changed from the base of the PR and between 5bfc457 and 7b02b50.

📒 Files selected for processing: 16
  • .automation.conf/tempest/tempest-ci-multinode-octavia-smoke.overrides.conf
  • .automation.conf/tempest/tempest-ci-multinode-octavia.overrides.conf
  • .gitignore
  • doc/source/operations/octavia.rst
  • etc/kayobe/ansible/tools/configure-vxlan.yml
  • etc/kayobe/environments/ci-multinode/controllers.yml
  • etc/kayobe/environments/ci-multinode/globals.yml
  • etc/kayobe/environments/ci-multinode/hooks/overcloud-host-configure/post.d/10-check-kvm.yml
  • etc/kayobe/environments/ci-multinode/inventory/group_vars/all/vxlan.yml
  • etc/kayobe/environments/ci-multinode/inventory/group_vars/compute/network-interfaces
  • etc/kayobe/environments/ci-multinode/inventory/group_vars/controllers/network-interfaces
  • etc/kayobe/environments/ci-multinode/kolla/config/octavia.conf
  • etc/kayobe/environments/ci-multinode/kolla/globals.yml
  • etc/kayobe/environments/ci-multinode/networks.yml
  • etc/kayobe/environments/ci-multinode/stackhpc-ci.yml
  • releasenotes/notes/ci-multinode-octavia-amphora-bc2f989937fc8a59.yaml

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

📝 Walkthrough

The ci-multinode environment adds Amphora support when OVN is disabled. It configures a load-balancer management network, connects the VXLAN mesh and VLAN interfaces through a Linux bridge, and adds deployment checks and Tempest overrides.

Changes: CI multinode Amphora support

Layer / File(s) Summary
Management network and VXLAN topology
etc/kayobe/environments/ci-multinode/networks.yml, etc/kayobe/environments/ci-multinode/inventory/group_vars/..., etc/kayobe/environments/ci-multinode/controllers.yml, etc/kayobe/ansible/tools/configure-vxlan.yml, doc/source/operations/octavia.rst, releasenotes/notes/ci-multinode-octavia-amphora-bc2f989937fc8a59.yaml
Adds the lb_mgmt network and configures VLAN interfaces to use br-vxlan. The VXLAN configuration task creates a bridge connection when it is not already present. The operations documentation describes the management-network requirements.
Amphora deployment configuration
etc/kayobe/environments/ci-multinode/kolla/globals.yml, etc/kayobe/environments/ci-multinode/kolla/config/octavia.conf, etc/kayobe/environments/ci-multinode/{globals.yml,stackhpc-ci.yml}, etc/kayobe/environments/ci-multinode/hooks/.../10-check-kvm.yml, releasenotes/notes/ci-multinode-octavia-amphora-bc2f989937fc8a59.yaml
Selects the Octavia provider and health-manager interface according to OVN use, configures the Amphora management network and worker retries, and adds image credentials and CA trust. When OVN is disabled, host checks require x86_64 architecture and /dev/kvm.
Tempest CI configuration
.automation.conf/tempest/tempest-ci-multinode-octavia*.overrides.conf, .gitignore, releasenotes/notes/ci-multinode-octavia-amphora-bc2f989937fc8a59.yaml
Adds shared Tempest settings and Amphora-specific overrides. The smoke-test override links to the main override file, and generated Octavia PEM files are ignored.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to 7b02b

The management-network mapping follows the intended bridge path. No identified issue currently prevents merging after normal checks.

Pre-merge checks: Passed 3
Passed Passed checks: 3 passed
Check name Status Explanation
Docstring Coverage Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added the waiting-author-response PR is waiting for the author to respond label Oct 14, 2026
@saadtariq38
saadtariq38 force-pushed the INFRA-830-Configure-Multinode-OVS-Octavia-Amphora branch from 3bfa53d to 7b02b50 Compare October 14, 2026 15:06
LeafCloud does not support kvm and is the default for CI multinode runs at the moment. Amphora driver can be manually enabled for any SMSLab runs

This branch had an error being deployed

1 failed and 1 active deployments
Leafcloud — 87f21ecf Deployed Oct 16, 2026 by saadtariq38 via Multinode / Multinode #387
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting-author-response PR is waiting for the author to respond

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant