Skip to content

Fix CI on ubuntu-26.04 runners - #204

Merged
wu-sheng merged 4 commits into
masterfrom
bump-infra-e2e-docker29
Oct 31, 2026
Merged

wu-sheng merged 4 commits into
masterfrom
bump-infra-e2e-docker29

Conversation

@wu-sheng

@wu-sheng wu-sheng commented Oct 31, 2026 •

Copy link
Copy Markdown
Member

Four CI fixes. Helm master's last CI run was on 2026-10-13, before ubuntu-latest moved to ubuntu-26.04; all four problems showed up after that.

1. docker/login-action is no longer on the ASF allow-list.

  • The Test workflow now fails at startup: "docker/login-action@af1e73f… is not allowed in apache/skywalking-helm".
  • Fix: pin v4.7.0 (81c6f93d4bc991e510b342a00b419b3bb4efc5c6), which approved_patterns.yml lists with no expiry date.
  • This applies to e2e.ci.yaml and also publish-helm.yaml, which would hit the same error on the next master push.

2. Move off the infra-e2e containerd snapshotter workaround.

  • ubuntu-latest now resolves to ubuntu-26.04, which ships Docker 29 with the containerd image store.
  • The pinned apache/skywalking-infra-e2e@8c21e43 disables the containerd snapshotter and restarts dockerd before running a case. Upstream removed that workaround in feat: migrate to testcontainers-go v0.42.0 and upgrade dependencies skywalking-infra-e2e#146.
  • On the new runners, that restart hid skywalking-java's locally built agent image, so its E2E failed with "pull access denied" until Fix NPE and lost trace context of reactive Lettuce commands when tracing is enabled skywalking-java#841 moved off it.
  • The chart E2E pulls every image from a registry, so it may not be affected. It has not run on ubuntu-26.04 yet; this PR's CI is that run.
  • Fix:
    • Pin infra-e2e to the current main, 56ac8befeb5f910f2dc9339e65d07cd18953f90e, the same pin skywalking-java uses.
    • Drop the workflow's actions/setup-go (Go 1.24) step. The action installs Go 1.26 itself, and that Go is also on the PATH for the go install fallback in install-yq.sh.

3. Helm 4 rejects the duplicate SW_QUERY_ZIPKIN the E2E installs created.

  • The ubuntu-26.04 runner image ships Helm 4.3.1, whose server-side apply rejects duplicate env entries.
  • The three OAP 11 cases set oap.ports.zipkin-query, which already makes the chart emit SW_QUERY_ZIPKIN. They also set oap.env.SW_QUERY_ZIPKIN=default.
  • So helm install failed with duplicate entries for key [name="SW_QUERY_ZIPKIN"]. Helm 3's client-side apply had accepted the duplicate.
  • Fix: drop the redundant oap.env setting; the chart still sets the same value. docs/contributing/e2e-tests.md now says why.
  • I rendered every E2E install with helm template (Helm 4.1.1). master has exactly these three duplicates; this branch has no duplicate env name in any container.

4. BanyanDB standalone ran out of inotify instances.

  • On ubuntu-26.04 the BanyanDB standalone pod exits at startup: failed to create fsnotify watcher: couldn't initialize inotify: too many open files.
  • OAP then never became available, and the case timed out after 24 minutes.
  • The four kind nodes and all their pods share the host's fs.inotify.* limits. This is kind's documented "too many open files" issue.
  • Fix: a step before the E2E action raises max_user_watches and max_user_instances to at least kind's recommended 524288 / 512, never lowering them. It also prints the runner's own values.

Every third-party action in .github is now on the allow-list.

A chart user on Helm 4 who sets oap.ports.zipkin-query and also oap.env.SW_QUERY_ZIPKIN gets the same clear error. The chart docs already say the port sets it.

ubuntu-latest now resolves to ubuntu-26.04, which ships Docker 29. The pinned
infra-e2e action (8c21e43) disables the containerd snapshotter and restarts
dockerd before running a case, a workaround upstream removed in
apache/skywalking-infra-e2e#146. It broke the skywalking-java E2E on the new
runners, and the chart E2E has not run on them yet.

Pin the current main (56ac8befeb5f910f2dc9339e65d07cd18953f90e), as
apache/skywalking-java#841 did. The action installs Go 1.26 itself, so drop
the workflow's own Go 1.24 setup.
The ASF GitHub Actions allow-list no longer has docker/login-action
af1e73f (v4.4.0), so the Test workflow fails at startup with "is not
allowed in apache/skywalking-helm". Use v4.7.0
(81c6f93d4bc991e510b342a00b419b3bb4efc5c6), which the allow-list
approves without an expiry date. publish-helm.yaml uses the same pin.
@wu-sheng wu-sheng changed the title Bump skywalking-infra-e2e off the containerd snapshotter workaround Fix CI: approved docker/login-action pin, and infra-e2e off the containerd snapshotter workaround Oct 31, 2026
The ubuntu-26.04 runner image ships Helm 4.3.1, whose server-side apply
rejects duplicate env entries. The three OAP 11 cases set
oap.ports.zipkin-query, which already makes the chart emit SW_QUERY_ZIPKIN,
and set oap.env.SW_QUERY_ZIPKIN as well, so `helm install` failed with
"duplicate entries for key [name="SW_QUERY_ZIPKIN"]". Helm 3's client-side
apply accepted the duplicate.

Drop the redundant oap.env setting; the chart still sets the same value.
@wu-sheng wu-sheng changed the title Fix CI: approved docker/login-action pin, and infra-e2e off the containerd snapshotter workaround Fix CI on ubuntu-26.04: approved docker/login-action pin, infra-e2e off the snapshotter workaround, no duplicate env for Helm 4 Oct 31, 2026
On the ubuntu-26.04 runners BanyanDB standalone exits at startup with
"failed to create fsnotify watcher: couldn't initialize inotify: too many
open files". The four kind nodes and all their pods share the host's
inotify limits, which Ubuntu sets low, so OAP never became available and
the case timed out. Raise fs.inotify.max_user_watches and
max_user_instances to at least the values the kind known-issues page
recommends, and print the runner's own values first.
@wu-sheng wu-sheng changed the title Fix CI on ubuntu-26.04: approved docker/login-action pin, infra-e2e off the snapshotter workaround, no duplicate env for Helm 4 Fix CI on ubuntu-26.04 runners Oct 31, 2026
@wu-sheng
wu-sheng merged commit 1b422e0 into master Oct 31, 2026
6 checks passed
@wu-sheng
wu-sheng deleted the bump-infra-e2e-docker29 branch October 31, 2026 13:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants