Skip to content

Merge conflicts on node-test-commit-* jobs #4445

Open
Open
Merge conflicts on node-test-commit-* jobs#4445

Description

@sxa
sxa
opened on Aug 28, 2026

Raising this as a bit of an 'umbrella' issue to track occurrences of this problem. There have been various issues on it in the past and this one regarding debian12-x64 seems to have had most of the discussion on it as well as a potential PR that might fix it. I'm splitting this out as a seprate "umbrella" issue independent of the live incident ones to see what we can do about it. As can be seen from the list of issues in the search link above it's not specific to debian12, but anecdotally it feels like it's happened more often on there (technically I can't see why that might be, so I may be imagining it - I hit it again on Debian with the tests on nodejs/node#62251.

There as also a bit of analysis if it occurring on three separate test runs of a PR at nodejs/node#65564 (comment)

Activity

  1. sxa commented on Aug 28, 2026

    @sxa
    sxa
    on Aug 28, 2026
    MemberAuthor

    I created a duplicate of the node-test-commit job to try and replicate this but annoyingly despite using the same parameters as the failing job and executing in the common workspace directory on the same node (test-ibm-debian12-x64-1 it did not hit the same failure 😞

  2. sxa commented on Aug 28, 2026

    @sxa
    sxa
    on Aug 28, 2026
    MemberAuthor

    I've looked at making modifications to the main node-test-commit-linux job:

    • I've implemented the changes from PR4346 and tried it. Before, after, after, undone - I've locked the last two of those so they can be re-run to reproduce this if required.
    • "Before" and "undone" (i.e. with the same code) is consistent and has the conflict:
    14:09:01 + git rebase --abort
14:09:01 fatal: No rebase in progress?
14:09:01 + true
14:09:01 + git checkout -f refs/remotes/origin/_jenkins_local_branch
14:09:01 HEAD is now at c15711fb600 build: use ARCHTYPE instead of DESTCPU
14:09:01 + git config user.name
14:09:01 Dummy
14:09:01 + git config user.email
14:09:01 dummy@dummy.com
14:09:01 + echo
14:09:01 
14:09:01 + echo
14:09:01 
14:09:01 + git status
14:09:01 HEAD detached at c15711fb600
14:09:01 nothing to commit, working tree clean
14:09:01 ++ git rev-parse HEAD
14:09:01 ++ git rev-parse c15711fb6000f373e5dab2ee406fab09352950ce
14:09:01 + '[' c15711fb6000f373e5dab2ee406fab09352950ce '!=' c15711fb6000f373e5dab2ee406fab09352950ce ']'
14:09:29 + '[' -n 9f04fcd70d28ac691f592d3c0c1e00a28998eb51 ']'
14:09:29 + git rebase --committer-date-is-author-date 9f04fcd70d28ac691f592d3c0c1e00a28998eb51
14:09:29 Rebasing (1/45903)
14:09:29 Rebasing (2/45903)
14:09:29 Auto-merging Makefile
14:09:29 CONFLICT (add/add): Merge conflict in Makefile
    • The two "after" (with the fix from the PR) gives this:
    14:07:25 + git rebase --abort
14:07:25 fatal: No rebase in progress?
14:07:25 + true
14:07:25 + '[' -n refs/pull/62251/head ']'
14:07:25 + fetch_from_origin +refs/pull/62251/head:refs/remotes/origin/_jenkins_local_branch
14:07:25 + fetch_args=--no-tags
14:07:25 ++ git rev-parse --git-dir
14:07:25 + '[' -f .git/shallow ']'
14:07:25 + fetch_args='--no-tags --unshallow'
14:07:25 + git fetch --no-tags --unshallow origin +refs/pull/62251/head:refs/remotes/origin/_jenkins_local_branch
14:07:25 git@github.com: Permission denied (publickey).
14:07:25 fatal: Could not read from remote repository.
14:07:25 
14:07:26 Please make sure you have the correct access rights
14:07:26 and the repository exists.
14:07:26 Build step 'Execute shell' marked build as failure

    On the basis of the above not being implementable I'm going to just clear up the workspace on the machine for now and re-enable the node.

  3. sxa
    pinned this issue on Sep 2, 2026

  4. richardlau commented on Sep 17, 2026

    @richardlau
    richardlau
    on Sep 17, 2026
    Member

    Looks like we saw again on these (from https://ci.nodejs.org/job/node-test-commit-linux/73223/)

    I've cleared the node-test-commit-linux workspace on those.

  5. richardlau commented on Sep 21, 2026

    @richardlau
    richardlau
    on Sep 21, 2026
    Member

    Occurring on test-ibm-rhel8-x64-3.
    e.g. https://ci.nodejs.org/job/node-test-commit-linux/nodes=rhel8-x64/73345/console

    Wiping out the node workspace on the machine.

  6. panva commented on Sep 26, 2026

    @panva
    panva
    on Sep 26, 2026
    Member

    Looking into these with @aduh95 and Codex related to this non-conflicting PR and repeated failures.

    Found a concrete cause on test-digitalocean-debian12-x64-1: stress CI makes the shared checkout shallow.

    Both jobs use /home/iojs/build/workspace/node, but their checkout settings differ:

    • node-stress-single-test: shallow checkout enabled, depth 1.
    • node-test-commit-linux: shallow checkout disabled.

    On September 16, stress build #893 ran git fetch --depth=1 in that workspace. Reading the workspace's .git/shallow on September 26 showed these boundaries still present:

    6f723ec6e724dba723f03b5c5754267038a664fe
b2e7b650891c70a393e02dfdbc9544541d7302c9

    These correspond to main on September 16 and the stress run's PR head.

    The next recorded Linux build on that worker, #73226, attempted to replay 48,028 commits and conflicted on the initial Makefile commit. #73521 has the same failure for nodejs/node#65559.

    Fetching without --depth does not remove existing shallow boundaries. main's earlier ancestry remains hidden, so PRs branching before the boundary can lose their merge base and have their entire history selected for rebasing. This also explains why newer PRs can pass on the same worker.

    A local reproduction confirmed the sequence: full clone → depth-1 fetch → ordinary fetch → historical Makefile add/add conflict. An explicit --unshallow restored the expected one-commit rebase.

    Can we give stress CI its own workspace, or disable shallow fetching for jobs sharing this workspace, then unshallow/recreate the affected checkouts? Disabling shallow fetching alone will leave existing boundaries in place, while cleaning the checkouts alone lets a subsequent stress run recreate the problem.

  7. richardlau commented on Sep 26, 2026

    @richardlau
    richardlau
    on Sep 26, 2026
    Member

    I've disabled shallow clone for node-stress-single-test. I won't be able to delete the existing workspaces until on my work computer on Monday, although if someone can compile a list of the affected checkouts (the specific machines) that would help.

    We only started using shallow clones for that job on 14 May this year (https://github.com/nodejs/jenkins-config-test/commit/68a5d98b6e24b062f66fa19ed24142f2adb1ba34 (restricted to build wg)) (cc @sxa it looks like it was done in the middle of you adding AIX machines to the job), which was just prior to the first reports of the merge conflicts occurring in #4345.

    Reusing workspaces between jobs was done under #3897 to conserve disk space on the machines.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions