Skip to content

VF-LAG host configure #2682

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
holliefae wants to merge 3 commits into
base: stackhpc/2026.1
Choose a base branch
from
Open

VF-LAG host configure #2682

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6101e18
VF-LAG host configure
holliefae Oct 14, 2026
271336f
Add VFLAG services
holliefae Oct 14, 2026
20ebffe
Disable SRIOV role on an empty list
holliefae Oct 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 18 additions & 2 deletions doc/source/contributor/ofed.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,6 @@
OFED
====

Warning: Experimental workflow subject to change

The Nvidia DOCA framework is distributed as part of StackHPC Release Train for OFED driver support,
this repository is synced into Ark as part of the Release Train workflows, however to ensure
compatibility with Release Train packages, we are required to build OFED modules with support for
Expand Down Expand Up @@ -104,3 +102,21 @@ playbook:
.. code-block:: console

kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/tools/install-doca.yml

Host configuration
------------------

Ensure DOCA hosts are a member of the ``vflag`` group, this will be needed to ensure VF-LAG
and DOCA configuration is applied on DOCA hosts.

.. code-block:: ini

[vflag:children]
mlnx

A playbook is provided to ensure VF-LAG and OFED setup is applied on ``vflag`` hosts. To run
the configuration playbook:

.. code-block:: console

kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/deployment/vf-lag-host-configure.yml
138 changes: 138 additions & 0 deletions etc/kayobe/ansible/deployment/vf-lag-host-configure.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
---
# Setup of VF-LAG systemd configuration.
# This playbook should execute after kayobe overcloud host configure has completed.

- name: Enable SRIOV
hosts: vflag
tags:
- sriov
handlers:
- name: Reboot
ansible.builtin.set_fact:
kayobe_needs_reboot: true
tasks:
# NOTE: define sriov_devices in group_vars
- name: Include SRIOV role
ansible.builtin.include_role:
name: stackhpc.linux.sriov
vars:
sriov_install_enabled: true
when:
- sriov_devices is defined
- sriov_devices | length > 0

- name: Prepare hosts for VF-LAG support
hosts: vflag
tags:
- vflag
vars:
# This playbook will execute after nodes are deployed
# and before overcloud host configure - we can't assume
# users and venvs exist.
ansible_user: "{{ ansible_user }}"
ansible_python_interpreter: "/usr/bin/python3"
tasks:
# TC offload requires kernel-modules-extra (act_ct module)
- name: Ensure kernel-modules-extra is installed
ansible.builtin.package:
name: kernel-modules-extra
state: present
become: true
when: ansible_facts.os_family == "RedHat"

# /etc/sysconfig is a RedHat convention - create equivalent here
- name: Create /etc/sysconfig for Ubuntu
become: true
ansible.builtin.file:
path: /etc/sysconfig
owner: "root"
group: "root"
mode: 0755
state: directory
when: ansible_facts.os_family == "Debian"

# Write /etc/sysconfig/sriov configuration to match this host
- name: Define parameters in /etc/sysconfig/sriov
ansible.builtin.template:
src: "sysconfig-sriov.j2"
dest: "/etc/sysconfig/sriov"
owner: "root"
group: "root"
mode: 0755
become: true
notify: Reboot and wait

- name: Transfer VF-LAG helper scripts to /usr/local/bin
ansible.builtin.copy:
src: "{{ item }}"
dest: "/usr/local/bin/{{ item }}"
owner: "root"
group: "root"
mode: 0755
become: true
with_items:
- "mlnx-vflag-early"
- "mlnx-vflag-final"
Comment thread
holliefae marked this conversation as resolved.
notify: Reboot and wait

- name: Transfer VF-LAG systemd units to /etc/systemd/system
ansible.builtin.template:
src: "{{ item }}.j2"
dest: "/etc/systemd/system/{{ item }}"
owner: "root"
group: "root"
mode: 0644
become: true
with_items:
- "mlnx-vflag-early.service"
- "mlnx-vflag-final.service"
notify: Reboot and wait

- name: Enable VF-LAG systemd units
ansible.builtin.systemd:
daemon_reload: true
name: "{{ item }}"
enabled: true
become: true
with_items:
- "mlnx-vflag-early.service"
- "mlnx-vflag-final.service"
notify: Reboot and wait

# Purge some stuff that OFED typically installs
# which may race with the setup we perform.

- name: Disable OFED openibd service
ansible.builtin.systemd:
name: "openibd.service"
enabled: false
state: "stopped"
become: true
Comment on lines +105 to +110

@coderabbitai coderabbitai Bot Oct 14, 2026 •
edited
Loading

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Check that openibd.service exists before disabling it.

If a VF-LAG host has no OFED service installed, this task fails rather than treating the service as already stopped. Ansible’s systemd module checks that the unit exists before it applies either enabled: false or state: stopped. The failure also prevents the later host tasks from completing. Check for the unit and run this task only when it exists. (raw.githubusercontent.com)

@holliefae holliefae Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure whether this is important to catch, should we be running this script if OFED isn't installed?

@coderabbitai coderabbitai Bot Oct 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
rg -n -i -C3 'ofed|openibd|vf-lag|vf_lag|vflag' --glob '!*.lock' --glob '!*.svg' . | head -n 280
fd -i 'mlnx-vflag|sysconfig-sriov'
ast-grep outline etc/kayobe/ansible/deployment/vf-lag-host-configure.yml

Length of output: 22668

@holliefae The documented workflow targets DOCA hosts. In doc/source/contributor/ofed.rst, the host configuration section places DOCA hosts in vflag and directs operators to run this playbook.

If OFED must already be installed, you do not need to support hosts without OFED. In that case, failing is reasonable. An explicit prerequisite check could give a clearer error, but it is not essential to this change.

The remaining question is whether every supported OFED/DOCA installation provides openibd.service. If that is guaranteed, my suggested guard is unnecessary. If a supported installation can omit the service, guard only this cleanup task.

My original Major rating was too strong without establishing that a supported host can lack the service.

You are interacting with an AI system.

@coderabbitai coderabbitai Bot Oct 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like the humans are having a chat. I'll hop back into my burrow for now. If you need me again, just tag @coderabbitai in a new comment, and I'll come hopping out!

@Alex-Welsh Alex-Welsh Oct 16, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

explicit prerequisite check could give a clearer error

Let's go with this option @holliefae


- name: Move the openib config file
ansible.builtin.command: "mv /etc/infiniband/openib.cf /etc/infiniband/openib.cf-"
args:
removes: "/etc/infiniband/openib.cf"
creates: "/etc/infiniband/openib.cf-"
become: true

# There appears to be an issue where the reciprocal symlink from a
# WantedBy= to a Wants= doesn't get created. Explicitly ensure it exists.
- name: Ensure the .wants symlink exists (Ubuntu)
when: ansible_facts.os_family == "Debian"
become: true
block:
- name: Ensure the .wants directory exists
ansible.builtin.file:
path: "/etc/systemd/system/systemd-networkd.service.wants"
state: "directory"
- name: Ensure the .wants symlink exists
ansible.builtin.file:
path: "/etc/systemd/system/systemd-networkd.service.wants/mlnx-vflag-early.service"
src: "/etc/systemd/system/mlnx-vflag-early.service"
state: "link"

handlers:
- name: Reboot and wait
ansible.builtin.set_fact:
kayobe_needs_reboot: true
23 changes: 23 additions & 0 deletions etc/kayobe/ansible/templates/mlnx-vflag-early.service.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
[Unit]
Description=Early configuration for Mellanox SR-IOV and VF-LAG
{% for nic in vflag_pfs %}
Requires=sys-subsystem-net-devices-{{ nic }}.device
After=sys-subsystem-net-devices-{{ nic }}.device
{% endfor %}
Requires=network-pre.target
Before=network-pre.target
{% if ansible_facts.os_family == "Debian" %}
Requires=systemd-networkd.service
Before=systemd-networkd.service
{% endif %}

[Service]
Type=oneshot
ExecStart=/usr/local/bin/mlnx-vflag-early
RemainAfterExit=yes

[Install]
WantedBy=network-pre.target
{% if ansible_facts.os_family == "Debian" %}
WantedBy=systemd-networkd.service
{% endif %}
13 changes: 13 additions & 0 deletions etc/kayobe/ansible/templates/mlnx-vflag-final.service.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
[Unit]
Description=Final configuration for Mellanox SR-IOV and VF-LAG
Wants=network-online.target
After=network-online.target
Before=docker.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/mlnx-vflag-final
RemainAfterExit=yes

[Install]
WantedBy=docker.service
4 changes: 4 additions & 0 deletions releasenotes/notes/vflag-host-configure-4ba351fa23327e16.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
---
features:
- |
Move VF-LAG host configure script from SMSLab Ansible to SKC.
Loading