Add OpenBao Monitoring

[technowhizz]

To be merged after stackhpc/ansible-collection-hashicorp#93 merges and then bumping the requirements.yml for the collection

[gemini-code-assist]

Code Review

This pull request introduces monitoring support for OpenBao, including Prometheus scrape configurations, alerting rules for sealed or unhealthy nodes, and corresponding documentation. A critical issue was identified in the Prometheus scrape configuration template, where the Kayobe-specific inventory group 'controllers' and filter 'net_ip' were used instead of Kolla-native equivalents ('control' group and 'kolla_address' filter), which would cause template rendering failures.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[technowhizz]

Hmm CI is failing because the prometheus scrape job is enabled by default (assuming most clients have openbao deployed) but because we manually have to deploy openbao, it's not set up in the CI and so there is a prometheus alert for exporter down.

Should we have it disabled by default and get the deploy openbao playbook to change some var to enable it?

@jovial @dougszumski What do you think?

[jovial]

Hmm CI is failing because the prometheus scrape job is enabled by default (assuming most clients have openbao deployed) but because we manually have to deploy openbao, it's not set up in the CI and so there is a prometheus alert for exporter down.

Should we have it disabled by default and get the deploy openbao playbook to change some var to enable it?

@jovial @dougszumski What do you think?

Not a bad idea. I think a lot of people bring up the cloud without TLS and then enable it. I agree its better to start with it being set to false and then we can work out the best way to "enable it out the box". Potentially, you could use a custom fact:

https://docs.ansible.com/projects/ansible/latest/playbook_guide/playbooks_vars_facts.html#adding-custom-facts

That you can then use in templating.

[technowhizz]

@jovial Thanks for taking a look.

Is there native custom fact support in kayobe?

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: Repository: stackhpc/stackhpc-kayobe-config/.coderabbit.yaml
• Review profile: CHILL
• Plan: Advanced
• Run ID: ea0fac2a-53a3-4997-a1ba-c4a31a019380

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

OpenBao monitoring adds conditional Prometheus scraping and alert rules. The secret-store deployment playbook records OpenBao deployment state. Documentation and release notes describe enablement, disabling monitoring, and steps for existing deployments.

Changes: OpenBao monitoring

Layer / File(s)	Summary
Persist OpenBao monitoring state etc/kayobe/stackhpc-monitoring.yml, etc/kayobe/ansible/secret-store/secret-store-deploy-overcloud.yml	The configuration defines OpenBao deployment state and enables monitoring when OpenBao is the configured CA secret store and is deployed. The deployment playbook sets the state to true for OpenBao deployments and displays a reminder when the file changes.
Configure OpenBao metrics and alerts etc/kayobe/kolla/config/prometheus/prometheus.yml.d/50-openbao.yml, etc/kayobe/kolla/config/prometheus/openbao.rules	When monitoring is enabled, Prometheus scrapes each controller’s internal address over HTTPS on port 8200. Critical alerts fire after five minutes when OpenBao is sealed or an autopilot node is unhealthy.
Document monitoring operations doc/source/configuration/openbao.rst, releasenotes/notes/add-openbao-monitoring-alerts-1c0837df41c80459.yaml	The documentation and release note describe monitoring enablement and disablement. They instruct existing deployments to rerun the deployment playbook, commit the configuration change, and reconfigure Prometheus and Grafana.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant SecretStoreDeploymentPlaybook
  participant KayobeMonitoringConfig
  participant Prometheus
  participant OpenBao
  SecretStoreDeploymentPlaybook->>KayobeMonitoringConfig: set stackhpc_openbao_enabled=true
  KayobeMonitoringConfig->>Prometheus: configure conditional scrape job and alert rules
  Prometheus->>OpenBao: GET /v1/sys/metrics
  OpenBao-->>Prometheus: return Prometheus metrics
  Prometheus->>Prometheus: evaluate sealed and unhealthy-node alerts

Loading

Merge Risk: 🟡 Moderate · up to f138f

Monitoring is disabled until OpenBao deployment is recorded, limiting immediate impact. Once enabled, certificate validation can prevent metric collection and sealed nodes can go unalerted. Address both monitoring gaps before merging, or explicitly accept those limitations.

Security Architecture Review

Security architecture risk: 🟡 Moderate · up to f138f

The documented monitoring upgrade reruns the CA-secret-store deployment without first checking current cluster leadership. After leader failover, this can conflict with the documented recovery procedure and threaten cluster consistency. Metrics access also depends on an upstream change whose authentication scope remains unverified.

Retained concerns

• Medium · reliability · inferred: The new monitoring upgrade instructs existing deployments to rerun the deployment playbook directly. That playbook defaults Raft leadership to controller index zero and passes that address into OpenBao deployment and initialization. If leadership has moved after controller failure, the prescribed upgrade omits the leader-discovery step that existing recovery guidance requires to avoid separate clusters. This creates an inferred risk to CA-secret-store consistency and certificate-service recovery; the deployment behavior predates this PR, but the new monitoring upgrade introduces another operational path into it.

Security review details

Security Blast Radius

• inferred — The monitoring integration addresses all OpenBao controller instances in the configured overcloud. Its rollout operates on the CA secret store using the existing privileged unseal and PKI workflow, so a leadership-related deployment error can affect CA-state consistency and certificate operations, not merely monitoring availability.

Trust Boundaries and Controls

• inferred — The intended boundary allows a metrics reader to access operational data without OpenBao credentials through an internal HTTPS endpoint. Internal addressing and HTTPS constrain the intended path, but do not demonstrate endpoint-only authorization or network isolation. The upstream role must establish that ordinary secret-store APIs retain their authentication controls.

Resilience and Maintainability Implications

• inferred — The added rules detect reported sealed and unhealthy states after five minutes, but do not themselves detect a completely failed scrape or absent metrics. Consequently, these rules alone cannot validate successful monitoring rollout or continued CA-store reachability. Global scrape-failure coverage remains unverified.

Hardening Proposals

• proposed — Make current-leader discovery a prerequisite of the monitoring deployment rerun, using the existing recovery workflow rather than relying on controller index zero.
• proposed — Validate the required collection version, metrics-only unauthenticated access, permitted network readers and Prometheus certificate trust before activation. Treat disabling the scrape and reversing endpoint exposure as separate rollback actions.

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Commit to this branch
• Create a new PR

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository: stackhpc/stackhpc-kayobe-config/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 93112030-123b-4771-90ed-3929668b6a12

📥 Commits

Reviewing files that changed from the base of the PR and between 80b2731 and f138f18.

📒 Files selected for processing (7)

• doc/source/configuration/openbao.rst
• etc/kayobe/ansible/secret-store/secret-store-deploy-overcloud.yml
• etc/kayobe/kolla/config/grafana/dashboards/openstack/openbao.json
• etc/kayobe/kolla/config/prometheus/openbao.rules
• etc/kayobe/kolla/config/prometheus/prometheus.yml.d/50-openbao.yml
• etc/kayobe/stackhpc-monitoring.yml
• releasenotes/notes/add-openbao-monitoring-alerts-1c0837df41c80459.yaml

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

📜 Review details

⚠️ CI failures not shown inline (1)

GitHub Check: stackhpc/check: Summary

Conclusion: failure

View job details

Merge Failed.
This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
  Error merging github/stackhpc/stackhpc-kayobe-config for 2419,f138f186ac16ba4e934a92ac41e49173972ad1f4

🔇 Additional comments (4)

etc/kayobe/stackhpc-monitoring.yml (1)

80-85: LGTM!

etc/kayobe/ansible/secret-store/secret-store-deploy-overcloud.yml (1)

183-201: LGTM!

doc/source/configuration/openbao.rst (1)

224-252: LGTM!

releasenotes/notes/add-openbao-monitoring-alerts-1c0837df41c80459.yaml (1)

1-18: LGTM!

[Alex-Welsh]

Needs some linter fixes