Skip to content

perftest: add a ROCm build to CI - #418

Merged
sshaulnv merged 1 commit into
linux-rdma:masterfrom
sbates130272:feat/rocm-ci-build
Nov 8, 2026
Merged

sshaulnv merged 1 commit into
linux-rdma:masterfrom
sbates130272:feat/rocm-ci-build

Conversation

@sbates130272

Copy link
Copy Markdown
Contributor

What this does

Adds a CI job that builds the ROCm configuration, alongside the two existing CUDA jobs.

One new file, .github/workflows/ubuntu24_04_rocm7_2.yaml. No source changes.

Why

The two existing workflows pass no --enable-rocm, so src/rocm_memory.c and everything behind HAVE_ROCM and HAVE_ROCM_DMABUF is never compiled on a pull request. A change that breaks the ROCm backend goes green today.

Details worth reviewing

It's a matrix over gfx950 and gfx1250, with fail-fast: false. Those two sit on opposite sides of the differences HIP code has to get right — CDNA against RDNA, and a 64-lane wavefront against a 32-lane one — so building both says more than building either twice. fail-fast is off because a failure is far more likely to be specific to one target's code generation than common to both, and cancelling the other leg hides exactly the comparison that would tell you which.

It runs in AMD's slim ROCm image rather than installing ROCm on the runner. The -full and -complete tags, and an apt install of the amdrocm-core metapackages, all pull the whole ML stack: Composable Kernel alone is over 2 GB, before rocSHMEM, MIOpen, BLAS and FFT. A compile check touches none of it.

approach size
rocm/dev-ubuntu-24.04:7.2.4 (used here) 1.1 GB
rocm/dev-ubuntu-26.04:10.1.0-full 8.3 GB
apt install of amdrocm-core-sdk10.1 9.3 GB installed

ROCm 7.2.4 rather than 10.1 is deliberate, not an oversight. AMD publishes slim tags only through 7.2.x; every 10.x tag is -full. What this job needs is a toolchain that compiles the sources for the targets in the matrix, and that image provides it — llc -mcpu=help lists both gfx950 and gfx1250, and both compile. Moving to a newer ROCm is a one-line change to the container tag if a slim 10.x image appears, or if something here starts depending on a newer HIP API.

Using the image also keeps repository plumbing out of the workflow. ROCm 10.1 is no longer served from the repo.radeon.com layout older releases use — it moved to stable.repo.amd.com with deb822 sources and amdrocm-* package names — and a workflow carrying that recipe would need revisiting the next time it changes.

Two small things that look defensive but aren't.

The ROCm prefix is resolved at runtime rather than assumed. The image ships /opt/rocm as a symlink to a versioned prefix, and configure needs the include path, so the job resolves whichever spelling is present.

--with-rocm-arch is probed for rather than passed. It selects the GPU that HIP kernels are compiled for, so it only exists once there are HIP sources to compile. configure treats an unknown --with-* as a warning rather than an error, so passing it blind would silently do nothing while appearing to work. Probing means this workflow is correct both before and after such sources exist, without needing to be amended when they do.

Testing

Replayed these exact steps in the container, both matrix architectures, against this tree: configure accepts every option, make and make install succeed, zero warnings.

Also replayed against a branch that does carry HIP sources, to confirm the probe does what it claims: configure reports the matrix architecture as its target, and llvm-objdump --offloading shows the resulting object contains a bundle for it. So the matrix selects what it says it does rather than merely accepting a flag.

Scope

Builds and installs only, like the CUDA jobs. There is no RDMA device in a GitHub runner and the project has no test suite, so this catches compile and link breakage and nothing more.

sbates130272 added a commit to sbates130272/perftest that referenced this pull request Nov 4, 2026
CI builds only the CUDA configuration. The two existing workflows pass no
--enable-rocm, so src/rocm_memory.c and everything behind HAVE_ROCM and
HAVE_ROCM_DMABUF is never compiled on a pull request, and a change that
breaks the ROCm backend goes green.

Add an Ubuntu 24.04 job building --enable-rocm --enable-rocm-dmabuf,
alongside the two CUDA jobs and triggered the same way, as a matrix over
gfx950 (Instinct MI355X, CDNA4) and gfx1250 (Instinct MI455X, CDNA5). Those
two were picked to cover consecutive Instinct generations: HIP sources that
compile correctly for one generation may have microarchitecture-specific issues
on the next, and testing both catches that. fail-fast is off: a failure is far
more likely to be specific to one generation than common to both, and cancelling
the other leg hides exactly the comparison that would say so.

It runs in AMD's slim ROCm image rather than installing ROCm on the runner.
The -full and -complete tags, and an apt install of the amdrocm-core
metapackages, all pull the whole ML stack: Composable Kernel alone is over
2 GB, before rocSHMEM, MIOpen and the BLAS and FFT libraries. A compile check
touches none of it. rocm/dev-ubuntu-24.04:7.2.4 is about 1.1 GB against 8.3 GB
for the smallest 10.1 image, and still ships hipcc, the HIP and HSA headers,
and device support for both matrix targets -- llc -mcpu=help lists gfx950 and
gfx1250, and both compile.

7.2.4 rather than 10.1 because AMD publishes slim tags only through 7.2.x;
every 10.x tag is -full. What this job needs is a toolchain that compiles the
sources for the targets in the matrix, which that image provides. Moving to a
newer ROCm is a one-line change to the container tag if a slim 10.x image
appears, or if something here starts depending on a newer HIP API.

Like the CUDA jobs this builds and installs only. There is no RDMA device in
a runner, and the project has no test suite.

Verified by replaying these steps in the container across both matrix
architectures: configure accepts every option, and make and make install succeed
warning-free.

Link: linux-rdma#418
Signed-off-by: Stephen Bates <sbates@raithlin.com>
sbates130272 added a commit to sbates130272/perftest that referenced this pull request Nov 4, 2026
CI builds only the CUDA configuration. The two existing workflows pass no
--enable-rocm, so src/rocm_memory.c and everything behind HAVE_ROCM and
HAVE_ROCM_DMABUF is never compiled on a pull request, and a change that
breaks the ROCm backend goes green.

Add an Ubuntu 24.04 job building --enable-rocm --enable-rocm-dmabuf,
alongside the two CUDA jobs and triggered the same way, as a matrix over
gfx950 (Instinct MI355X, CDNA4) and gfx1250 (Instinct MI455X, CDNA5). Those
two were picked to cover consecutive Instinct generations: HIP sources that
compile correctly for one generation may have microarchitecture-specific issues
on the next, and testing both catches that. fail-fast is off: a failure is far
more likely to be specific to one generation than common to both, and cancelling
the other leg hides exactly the comparison that would say so.

It runs in AMD's slim ROCm image rather than installing ROCm on the runner.
The -full and -complete tags, and an apt install of the amdrocm-core
metapackages, all pull the whole ML stack: Composable Kernel alone is over
2 GB, before rocSHMEM, MIOpen and the BLAS and FFT libraries. A compile check
touches none of it. rocm/dev-ubuntu-24.04:7.2.4 is about 1.1 GB against 8.3 GB
for the smallest 10.1 image, and still ships hipcc, the HIP and HSA headers,
and device support for both matrix targets -- llc -mcpu=help lists gfx950 and
gfx1250, and both compile.

7.2.4 rather than 10.1 because AMD publishes slim tags only through 7.2.x;
every 10.x tag is -full. What this job needs is a toolchain that compiles the
sources for the targets in the matrix, which that image provides. Moving to a
newer ROCm is a one-line change to the container tag if a slim 10.x image
appears, or if something here starts depending on a newer HIP API.

Like the CUDA jobs this builds and installs only. There is no RDMA device in
a runner, and the project has no test suite.

Verified by running this workflow in a fork of the repository across both
matrix architectures. Both legs passed: configure accepts every option, and
make and make install succeed warning-free.

Link: linux-rdma#418
Signed-off-by: Stephen Bates <sbates@raithlin.com>
sbates130272 added a commit to sbates130272/perftest that referenced this pull request Nov 4, 2026
CI builds only the CUDA configuration. The two existing workflows pass no
--enable-rocm, so src/rocm_memory.c and everything behind HAVE_ROCM and
HAVE_ROCM_DMABUF is never compiled on a pull request, and a change that
breaks the ROCm backend goes green.

Add an Ubuntu 24.04 job building --enable-rocm --enable-rocm-dmabuf,
alongside the two CUDA jobs and triggered the same way, as a matrix over
gfx950 (Instinct MI355X, CDNA4) and gfx1250 (Instinct MI455X, CDNA5). Those
two were picked to cover consecutive Instinct generations: HIP sources that
compile correctly for one generation may have microarchitecture-specific issues
on the next, and testing both catches that. fail-fast is off: a failure is far
more likely to be specific to one generation than common to both, and cancelling
the other leg hides exactly the comparison that would say so.

It runs in AMD's slim ROCm image rather than installing ROCm on the runner.
The -full and -complete tags, and an apt install of the amdrocm-core
metapackages, all pull the whole ML stack: Composable Kernel alone is over
2 GB, before rocSHMEM, MIOpen and the BLAS and FFT libraries. A compile check
touches none of it. rocm/dev-ubuntu-24.04:7.2.4 is about 1.1 GB against 8.3 GB
for the smallest 10.1 image, and still ships hipcc, the HIP and HSA headers,
and device support for both matrix targets -- llc -mcpu=help lists gfx950 and
gfx1250, and both compile.

7.2.4 rather than 10.1 because AMD publishes slim tags only through 7.2.x;
every 10.x tag is -full. What this job needs is a toolchain that compiles the
sources for the targets in the matrix, which that image provides. Moving to a
newer ROCm is a one-line change to the container tag if a slim 10.x image
appears, or if something here starts depending on a newer HIP API.

Like the CUDA jobs this builds and installs only. There is no RDMA device in
a runner, and the project has no test suite.

Verified by running this workflow in a fork of the repository across both
matrix architectures. Both legs passed: configure accepts every option, and
make and make install succeed warning-free.

Link: linux-rdma#418
Signed-off-by: Stephen Bates <sbates@raithlin.com>
sbates130272 added a commit to sbates130272/perftest that referenced this pull request Nov 4, 2026
CI builds only the CUDA configuration. The two existing workflows pass no
--enable-rocm, so src/rocm_memory.c and everything behind HAVE_ROCM and
HAVE_ROCM_DMABUF is never compiled on a pull request, and a change that
breaks the ROCm backend goes green.

Add an Ubuntu 24.04 job building --enable-rocm --enable-rocm-dmabuf,
alongside the two CUDA jobs and triggered the same way, as a matrix over
gfx950 (Instinct MI355X, CDNA4) and gfx1250 (Instinct MI455X, CDNA5). Those
two were picked to cover consecutive Instinct generations: HIP sources that
compile correctly for one generation may have microarchitecture-specific issues
on the next, and testing both catches that. fail-fast is off: a failure is far
more likely to be specific to one generation than common to both, and cancelling
the other leg hides exactly the comparison that would say so.

It runs in AMD's slim ROCm image rather than installing ROCm on the runner.
The -full and -complete tags, and an apt install of the amdrocm-core
metapackages, all pull the whole ML stack: Composable Kernel alone is over
2 GB, before rocSHMEM, MIOpen and the BLAS and FFT libraries. A compile check
touches none of it. rocm/dev-ubuntu-24.04:7.2.4 is about 1.1 GB against 8.3 GB
for the smallest 10.1 image, and still ships hipcc, the HIP and HSA headers,
and device support for both matrix targets -- llc -mcpu=help lists gfx950 and
gfx1250, and both compile.

7.2.4 rather than 10.1 because AMD publishes slim tags only through 7.2.x;
every 10.x tag is -full. What this job needs is a toolchain that compiles the
sources for the targets in the matrix, which that image provides. Moving to a
newer ROCm is a one-line change to the container tag if a slim 10.x image
appears, or if something here starts depending on a newer HIP API.

Like the CUDA jobs this builds and installs only. There is no RDMA device in
a runner, and the project has no test suite.

Verified by running this workflow in a fork of the repository across both
matrix architectures. Both legs passed: configure accepts every option, and
make and make install succeed warning-free.

Link: linux-rdma#418
Signed-off-by: Stephen Bates <sbates@raithlin.com>
CI builds only the CUDA configuration. The two existing workflows pass no
--enable-rocm, so src/rocm_memory.c and everything behind HAVE_ROCM and
HAVE_ROCM_DMABUF is never compiled on a pull request, and a change that
breaks the ROCm backend goes green.

Add an Ubuntu 24.04 job building --enable-rocm --enable-rocm-dmabuf,
alongside the two CUDA jobs and triggered the same way. It runs in AMD's
slim ROCm image rather than installing ROCm on the runner. The -full and
-complete tags, and an apt install of the amdrocm-core metapackages, all
pull the whole ML stack: Composable Kernel alone is over 2 GB, before
rocSHMEM, MIOpen and the BLAS and FFT libraries. A compile check touches
none of it. rocm/dev-ubuntu-24.04:7.2.4 is about 1.1 GB against 8.3 GB
for the smallest 10.1 image, and still ships hipcc and the HIP and HSA
headers needed to compile the ROCm backend.

7.2.4 rather than 10.1 because AMD publishes slim tags only through 7.2.x;
every 10.x tag is -full. What this job needs is a toolchain that compiles
the sources, which that image provides. Moving to a newer ROCm is a
one-line change to the container tag if a slim 10.x image appears, or if
something here starts depending on a newer HIP API.

This is a single host-only build; there are no HIP kernel sources in the
project yet, so there is nothing to compile for a specific device target.
When device code and --with-rocm-arch support are introduced this job can
gain an architecture matrix at that point.

Like the CUDA jobs this builds and installs only. There is no RDMA device
in a runner, and the project has no test suite.

Verified by running this workflow in a fork of the repository. The job
passed: configure accepts every option, and make and make install succeed
warning-free.

Link: linux-rdma#418

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Stephen Bates <sbates@raithlin.com>
@sshaulnv
sshaulnv merged commit 2180c6e into linux-rdma:master Nov 8, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants