Skip to content

Replace Trivy with Syft & Grype - #2692

Merged
MoteHue merged 13 commits into
stackhpc/2026.1from
replace-trivy-v2
Oct 22, 2026
Merged

MoteHue merged 13 commits into
stackhpc/2026.1from
replace-trivy-v2

Conversation

@Alex-Welsh

Copy link
Copy Markdown
Member

This PR replaces Trivy with Syft & Grype in our image build pipeline.

It also contains a set of images that built cleanly without any critical CVEs. That included bumping all repos, and actually fixing some templating issues in our pulp config (see also #2691 for another change that is included in here)

I made a lot of tweaks to try and improve runtime, it's not too bad, about 1h20s-1h40s, but it's still slower than Trivy was.

SBOMs are also larger. Again, I've tried to trim them a lot. They use a custom format which helps the golang images in particular.

@Alex-Welsh
Alex-Welsh requested a review from a team as a code owner October 19, 2026 08:21
@github-actions github-actions Bot added the waiting-review PR is waiting for a review label Oct 19, 2026
@coderabbitai

coderabbitai Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Walkthrough

The change replaces Trivy with Grype and Syft for container image scans, adds configurable parallel scans, and updates vulnerability allowlists. It also updates Pulp repository version references and changes Kolla image tags and the Prometheus libvirt exporter source pin.

Changes: Image vulnerability scanning

Layer / File(s) Summary
Scanner setup and documented inputs
.github/workflows/stackhpc-container-image-build.yml, tools/scan-images.sh, doc/source/contributor/testing-ci-automation.rst
The workflow installs pinned Grype and Syft versions. The script configures Grype and Syft, checks for the tools, and sets scan parallelism to four by default. The documentation describes direct image scans and the optional Syft SBOM path.
Image scanning and result handling
tools/scan-images.sh, etc/kayobe/grype/allowed-vulnerabilities.yml
The script sorts images by size, generates Grype ignore configuration, optionally generates SBOMs, and processes scan results. The allowlist adds global and image-specific vulnerability entries.
Database preparation and parallel execution
tools/scan-images.sh, .github/workflows/stackhpc-container-image-build.yml
The script updates the Grype database and runs scans in parallel. It records failures and stops starting scans after a failure. The workflow waits for its background database update before scanning.

Pulp repository version references

Layer / File(s) Summary
Repository version pins
etc/kayobe/pulp-repo-versions.yml
Repository timestamps are updated. Grafana and OpenSearch use unified version keys, and the Fluentd and ProxySQL keys are updated.
Repository URL and suite configuration
etc/kayobe/pulp.yml, etc/kayobe/kolla/repos.yaml
Pulp URLs use the updated version variables. The Grafana repository uses suite stable and component main.
Environment repository mappings
etc/kayobe/environments/aio/release-train.yml, etc/kayobe/environments/ci-builder/stackhpc-ci.yml, etc/kayobe/environments/ci-multinode/stackhpc-ci.yml
The AIO, CI builder, and CI multinode settings reference updated Pulp version variables.

Kolla image inputs

Layer / File(s) Summary
Image tags and exporter source pin
etc/kayobe/kolla-image-tags.yml, etc/kayobe/kolla.yml, releasenotes/notes/full-rebuild-3c545359275d8fdc.yaml
The OpenStack image tags change for Rocky 10 and Ubuntu Noble. Kolla pins prometheus-libvirt-exporter to version 2.6.1 with architecture-specific checksums. The release note records the rebuild and exporter upgrade.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Workflow as Build workflow
  participant Script as scan-images.sh
  participant Syft
  participant Grype
  Workflow->>Script: Run image scans after database update
  Script->>Grype: Update vulnerability database
  alt SBOM input is true
    Script->>Syft: Generate JSON SBOM for image
    Syft-->>Script: Return SBOM
    Script->>Grype: Scan SBOM
  else SBOM input is false
    Script->>Grype: Scan image
  end
  Grype-->>Script: Return scan results
Loading
Merge Risk: 🟡 Moderate · up to 013c3

AIO and CI-multinode builds can use moving Fluentd and Grafana repositories instead of the intended snapshots. Add the missing mappings before merging.

Pre-merge checks: Passed 3
Check name Status Explanation
Docstring Coverage Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 12 functions across 1 files. (12 skipped: …
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 12 functions across 1 files. (12 skipped: 12 unsupported.)

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@Alex-Welsh

Copy link
Copy Markdown
Member Author

Here is the build for this set of images https://github.com/stackhpc/stackhpc-kayobe-config/actions/runs/38931400535

@Alex-Welsh
Alex-Welsh force-pushed the replace-trivy-v2 branch 2 times, most recently from 9145ca2 to 013c3e3 Compare October 19, 2026 08:27

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


ℹ️ Review info
⚙️ Run configuration
  • Configuration used: Repository: stackhpc/stackhpc-kayobe-config/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 764edf06-a6de-42fb-8f6c-62b5f1c402e2
📥 Commits

Reviewing files that changed from the base of the PR and between fe4e839 and 013c3e3.

📒 Files selected for processing: 13
  • .github/workflows/stackhpc-container-image-build.yml
  • doc/source/contributor/testing-ci-automation.rst
  • etc/kayobe/environments/aio/release-train.yml
  • etc/kayobe/environments/ci-builder/stackhpc-ci.yml
  • etc/kayobe/environments/ci-multinode/stackhpc-ci.yml
  • etc/kayobe/grype/allowed-vulnerabilities.yml
  • etc/kayobe/kolla-image-tags.yml
  • etc/kayobe/kolla.yml
  • etc/kayobe/kolla/repos.yaml
  • etc/kayobe/pulp-repo-versions.yml
  • etc/kayobe/pulp.yml
  • releasenotes/notes/full-rebuild-3c545359275d8fdc.yaml
  • tools/scan-images.sh

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

Comment thread etc/kayobe/pulp-repo-versions.yml
Alex-Welsh and others added 8 commits October 22, 2026 13:24
Without function symbols, Grype matches Go vulnerabilities in all
modules regardless of whether they are used.

Enabling symbol capture for the standard library and golang.org/x/
modules lets us filter based on what is actually used.

Switched the SBOM to the Syft JSON format, since SPDX does
not preserve the symbols.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Allow critical vulnerabilities reported by Grype for Ubuntu images that
are not exploitable in our deployments:

* CVE-2026-56857: Only affects
  Windows.
* CVE-2026-78663: Can only
  cause memory exhaustion. No upstream binaries include the fix yet.
* CVE-2024-24790 and CVE-2025-22871 in Bifrost: the Ubuntu podman and
  buildah packages do not serve HTTP.
* CVE-2025-22871 in exporters: requires a proxy in front of the server
  that accepts bare LF in chunk extensions. Prometheus scrapes exporters
  directly.
* CVE-2026-106445 and CVE-2026-106446 in OpenSearch Dashboards: require
  compiling untrusted handlebars templates with
  allowProtoMethodsByDefault, or untrusted ASTs. User templates are
  compiled from strings with kbn-handlebars.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Kernel packages are already excluded from the summary, since containers
use the host kernel. Extend this to kernel images and modules found by
Syft.
The Grafana Ubuntu repository in the release train is now synced and
published in verbatim mode, so now uses (stable/main).
Kolla uses prometheus-libvirt-exporter 2.3.1, which contains
CVE-2026-39834, which is reported as a critical vulnerability. Version
2.6.1 is built with golang.org/x/crypto v0.57.0.

Upstream fix TODO
Lego includes CVE-2026-39834, an infinite loop on large SSH channel
writes. Lego does not make SSH connections when completing ACME HTTP-01
challenges and it is only fixed in lego v5.
@Alex-Welsh

Copy link
Copy Markdown
Member Author

Rebased to include recently rebuilt containers

@MoteHue
MoteHue enabled auto-merge October 22, 2026 14:36
@MoteHue
MoteHue merged commit 71b5197 into stackhpc/2026.1 Oct 22, 2026
29 checks passed
@MoteHue
MoteHue deleted the replace-trivy-v2 branch October 22, 2026 15:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting-review PR is waiting for a review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants