Repository navigation
CI: make sure GNU is the system coreutils in the jobs that compare against it - #15024
abendrothj wants to merge 1 commit into
Conversation
|
GNU testsuite comparison: |
|
Would you change runner of |
bf81051 to
23333fd
Compare
|
Done in 23333fd. |
xtqqczze
left a comment
There was a problem hiding this comment.
Please don't change runner image in same PR, instead detect whether coreutils-from-gnu is available.
| needs: detect | ||
| if: needs.detect.outputs.all == 'true' | ||
| runs-on: ubuntu-latest | ||
| runs-on: ubuntu-26.04 |
There was a problem hiding this comment.
| runs-on: ubuntu-26.04 | |
| runs-on: ubuntu-latest |
There was a problem hiding this comment.
Done in 47face8: back on ubuntu-latest, and the PR doesn't change any runner now.
| ## The examples compare against the system coreutils, which must be GNU. | ||
| ## ubuntu-26.04 ships uutils there. | ||
| sudo apt-get update | ||
| sudo apt-get install -y --allow-remove-essential coreutils-from-gnu coreutils-from-uutils- |
There was a problem hiding this comment.
| sudo apt-get install -y --allow-remove-essential coreutils-from-gnu coreutils-from-uutils- | |
| - name: Use GNU as the system coreutils | |
| shell: bash | |
| run: | | |
| ## The examples compare against the system coreutils, which must be GNU. | |
| ## ubuntu-26.04 ships uutils there; older images have no such switch. | |
| sudo apt-get update | |
| if apt-cache show coreutils-from-gnu >/dev/null 2>&1; then | |
| sudo apt-get install -y --allow-remove-essential coreutils-from-gnu coreutils-from-uutils- | |
| fi |
There was a problem hiding this comment.
Why do we need to introduce such complex detection instead of using host that we are 100% sure that the package is available?
There was a problem hiding this comment.
The package being available on a specific host doesn’t necessarily justify coupling the CI job to that host.
We’re interested in whether coreutils-from-gnu is available, not specifically whether we’re on Ubuntu 26.04. The check expresses that requirement directly and keeps the workflow resilient to runner changes.
There was a problem hiding this comment.
We can just change the job when we lost it. No need to detect dynamically.
There was a problem hiding this comment.
Actually, we should catch failure instead of silently fallbacking to uutils (when package name was changed by some reason).
There was a problem hiding this comment.
It's outside of the scope of the linked issue #15023 to change the runner image to ubuntu-26.04.
There was a problem hiding this comment.
fair, a renamed package is the only case where they differ, and it's unlikely. two smaller differences: where GNU is already the system coreutils (ubuntu-latest today) the step skips apt-get update, and the check after the install fails the job if the switch didn't happen, which oech3 asked for. if you'd still rather have apt-cache show, I can switch.
There was a problem hiding this comment.
I think there's a much higher likelihood that utilities are added to or removed from the coreutils-from-uutils package than that the package itself is renamed. For example, ls could be removed from the package while coreutils-from-uutils remains installed. In that case, relying on ls --version to detect the switch would be unreliable.
There was a problem hiding this comment.
It is hard to imagine, but we can
for _b in $(target/release/coreutils --list);do
sudo cp -sf /usr/bin/gnu${_b} /usr/bin/${_b}
done
cp -sf fails at missing source binary.
There was a problem hiding this comment.
Agreed, switched to the apt-cache show step. After it, the step fails if coreutils-from-uutils is still installed, which covers every utility, not just ls: on 26.04 cp, mv, rm, df and true already point at GNU while ls is uutils. I didn't use the cp -sf loop: on 26.04 it stops at the first utility without a GNU twin (cp: cannot stat '/usr/bin/gnuchroot'), and the fuzz jobs don't build coreutils before this step.
| needs: detect | ||
| if: needs.detect.outputs.targets != '[]' | ||
| runs-on: ubuntu-latest | ||
| runs-on: ubuntu-26.04 |
There was a problem hiding this comment.
| runs-on: ubuntu-26.04 | |
| runs-on: ubuntu-latest |
There was a problem hiding this comment.
Same, ubuntu-latest again.
| ## ubuntu-26.04 ships uutils as the system coreutils; give configure and the | ||
| ## test harness GNU's tools instead of whichever rust-coreutils the image has. | ||
| sudo apt-get install -y --allow-remove-essential coreutils-from-gnu coreutils-from-uutils- |
There was a problem hiding this comment.
| ## ubuntu-26.04 ships uutils as the system coreutils; give configure and the | |
| ## test harness GNU's tools instead of whichever rust-coreutils the image has. | |
| sudo apt-get install -y --allow-remove-essential coreutils-from-gnu coreutils-from-uutils- | |
| - name: Use GNU as the system coreutils | |
| shell: bash | |
| run: | | |
| ## The examples compare against the system coreutils, which must be GNU. | |
| ## ubuntu-26.04 ships uutils there; older images have no such switch. | |
| sudo apt-get update | |
| if apt-cache show coreutils-from-gnu >/dev/null 2>&1; then | |
| sudo apt-get install -y --allow-remove-essential coreutils-from-gnu coreutils-from-uutils- | |
| fi |
There was a problem hiding this comment.
I'd keep this one unconditional. This job is already pinned to ubuntu-26.04 on main (# some strace? test needs newer glibc), so the package is always there, and if it ever isn't, a failed install is the right outcome for the GNU job.
There was a problem hiding this comment.
it would be great if we weren't as tightly coupled to the runner image in case we want to change it in the future
There was a problem hiding this comment.
Check is useful if and only if we know every possible error. But it is very hard to do that. We should just do what we want to do and fail.
There was a problem hiding this comment.
f1b4415 does that. There's no availability check anymore, and all three jobs run the same step: if ls --version already says GNU it does nothing, otherwise it installs coreutils-from-gnu and then fails unless ls --version reports GNU. A missing or renamed package fails the apt install, and an install that doesn't switch fails the check, so this job isn't tied to 26.04 either. I ran the step in Ubuntu containers: 24.04 stays on GNU 9.4 without apt, 26.04 switches from uutils 0.8.0 to GNU 9.7, a renamed package fails with 100, and an install that doesn't switch fails the check.
|
This PR has 2 commit to separate things. No worry about confusion. |
23333fd to
47face8
Compare
|
The runner changes are gone, and all three jobs run on the image they had on main. Rather than probing for |
|
Actually, fuzzer itself is designed to reject non-GNU coreutils on |
|
the native GNU job already runs on ubuntu-26.04, and its runner has uutils as the system coreutils: this PR's run removes |
f1b4415 to
44e0404
Compare
| if apt-cache show coreutils-from-gnu >/dev/null 2>&1; then | ||
| sudo apt-get install -y --allow-remove-essential coreutils-from-gnu coreutils-from-uutils- | ||
| fi | ||
| if dpkg -s coreutils-from-uutils >/dev/null 2>&1; then |
There was a problem hiding this comment.
why do check whether coreutils-from-uutil is installed when we've just uninstalled it? if the previous operation succeeded than this state is impossible
There was a problem hiding this comment.
If coreutils-from-gnu isn't in the index (renamed, say), the install is skipped and uutils stays the system coreutils. This check is what fails the job then.
There was a problem hiding this comment.
If we can check whether coreutils-from-uutils is installed, why not:
if dpkg -s coreutils-from-uutils >/dev/null 2>&1; then
sudo apt-get update
sudo apt-get install -y --allow-remove-essential coreutils-from-gnu coreutils-from-uutils-
fiThere was a problem hiding this comment.
Good point, changed.
44e0404 to
b0003f1
Compare
|
@abendrothj please rewrite the commit description, it reads like AI and doesn't really add anything |
| - name: Use GNU as the system coreutils | ||
| shell: bash | ||
| run: | | ||
| ## Images such as ubuntu-26.04 ship uutils as the system coreutils; switch to GNU, or fail. |
There was a problem hiding this comment.
I'm not sure these comments are adding much value.
|
Fair, dropped the commit body and the comments. |
b0003f1 to
ed8c931
Compare
Newer Ubuntu images ship uutils as the system coreutils, so the GNU test job and the two fuzz jobs that compare against the system coreutils could end up comparing uutils with itself.
Each of those jobs now has the same step: if
coreutils-from-uutilsis installed, it installscoreutils-from-gnuin its place, and the job fails if that install does. Runner images are unchanged.Should land after #15005: once the system coreutils are GNU, the check added there is what stops the root tests from passing on GNU's binaries.
Closes #15023