Skip to content

STATUS_HEAP_CORRUPTION in rust-lld with a custom target on x86_64-pc-windows-gnu #115985

New issue
New issue
Open
Open
STATUS_HEAP_CORRUPTION in rust-lld with a custom target on x86_64-pc-windows-gnu#115985
Labels
A-LLVMArea: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues.A-linkageArea: linking into static, shared libraries and binariesC-bugCategory: This is a bug.CI-spurious-fail-rust-lld-crashCI spurious failure: `rust-lld` crashing / SIGSEGV / 0xc0000374 heap corruptionO-windows-gnuToolchain: GNU, Operating system: Windows

Description

@ehuss
ehuss
opened on Sep 19, 2023

The following example has recently started causing rust-lld to crash on x86_64-pc-windows-gnu:

#![no_main]
#![allow(internal_features)]
#![feature(no_core)]
#![feature(lang_items)]
#![no_core]

#[lang = "sized"]
pub trait Sized {
    // Empty.
}
#[lang = "copy"]
pub trait Copy {
    // Empty.
}

with a custom JSON target:

{
    "llvm-target": "x86_64-unknown-none-gnu",
    "data-layout": "e-m:e-i64:64-f80:128-n8:16:32:64-S128",
    "arch": "x86_64",
    "target-endian": "little",
    "target-pointer-width": "64",
    "target-c-int-width": "32",
    "os": "none",
    "linker-flavor": "ld.lld",
    "linker": "rust-lld",
    "executables": true
}

Building with:

rustc --target custom-target.json main.rs

This causes rust-lld to fail about 50% of the time, with errors like:

error: linking with `rust-lld` failed: exit code: 0xc0000374
  |
  = note: "rust-lld" "-flavor" "gnu" "main.main.4fa2520d328b6ba3-cgu.0.rcgu.o" "--as-needed" "-L" "C:\\Users\\eric\\.rustup\\toolchains\\nightly-2023-09-01-x86_64-pc-windows-gnu\\lib\\rustlib\\custom-target\\lib" "-Bdynamic" "--eh-frame-hdr" "-z" "noexecstack" "-L" "C:\\Users\\eric\\.rustup\\toolchains\\nightly-2023-09-01-x86_64-pc-windows-gnu\\lib\\rustlib\\custom-target\\lib" "-o" "main" "--gc-sections"
  = note: rust-lld: warning: cannot find entry symbol _start; not setting start address

error: aborting due to previous error

or

error: linking with `rust-lld` failed: exit code: 0xc0000005
  |
  = note: "rust-lld" "-flavor" "gnu" "main.main.4fa2520d328b6ba3-cgu.0.rcgu.o" "--as-needed" "-L" "C:\\Users\\eric\\.rustup\\toolchains\\nightly-2023-09-01-x86_64-pc-windows-gnu\\lib\\rustlib\\custom-target\\lib" "-Bdynamic" "--eh-frame-hdr" "-z" "noexecstack" "-L" "C:\\Users\\eric\\.rustup\\toolchains\\nightly-2023-09-01-x86_64-pc-windows-gnu\\lib\\rustlib\\custom-target\\lib" "-o" "main" "--gc-sections"
  = note: rust-lld: warning: cannot find entry symbol _start; not setting start address
          PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
          Stack dump:
          0.    Program arguments: rust-lld -flavor gnu main.main.4fa2520d328b6ba3-cgu.0.rcgu.o --as-needed -L C:\\Users\\eric\\.rustup\\toolchains\\nightly-2023-09-01-x86_64-pc-windows-gnu\\lib\\rustlib\\custom-target\\lib -Bdynamic --eh-frame-hdr -z noexecstack -L C:\\Users\\eric\\.rustup\\toolchains\\nightly-2023-09-01-x86_64-pc-windows-gnu\\lib\\rustlib\\custom-target\\lib -o main --gc-sections
          Exception Code: 0xC0000005
          0x00007FFDD755633C, C:\WINDOWS\SYSTEM32\ntdll.dll(0x00007FFDD7530000) + 0x2633C byte(s), RtlGetCurrentServiceSessionId() + 0x1AEC byte(s)
          0x00007FFDD7555B74, C:\WINDOWS\SYSTEM32\ntdll.dll(0x00007FFDD7530000) + 0x25B74 byte(s), RtlGetCurrentServiceSessionId() + 0x1324 byte(s)
          0x00007FFDD75547B1, C:\WINDOWS\SYSTEM32\ntdll.dll(0x00007FFDD7530000) + 0x247B1 byte(s), RtlFreeHeap() + 0x51 byte(s)
          0x00007FFDD5609C9C, C:\WINDOWS\System32\msvcrt.dll(0x00007FFDD55F0000) + 0x19C9C byte(s), free() + 0x1C byte(s)
          0x00007FFDD5609366, C:\WINDOWS\System32\msvcrt.dll(0x00007FFDD55F0000) + 0x19366 byte(s), _aligned_free() + 0x16 byte(s)
          0x00007FF74451890A, C:\Users\eric\.rustup\toolchains\nightly-2023-09-01-x86_64-pc-windows-gnu\lib\rustlib\x86_64-pc-windows-gnu\bin\rust-lld.exe(0x00007FF741000000) + 0x351890A byte(s)
          0x00007FF742807BF9, C:\Users\eric\.rustup\toolchains\nightly-2023-09-01-x86_64-pc-windows-gnu\lib\rustlib\x86_64-pc-windows-gnu\bin\rust-lld.exe(0x00007FF741000000) + 0x1807BF9 byte(s)
          0x00007FF7426D9634, C:\Users\eric\.rustup\toolchains\nightly-2023-09-01-x86_64-pc-windows-gnu\lib\rustlib\x86_64-pc-windows-gnu\bin\rust-lld.exe(0x00007FF741000000) + 0x16D9634 byte(s)
          0x00007FF742256947, C:\Users\eric\.rustup\toolchains\nightly-2023-09-01-x86_64-pc-windows-gnu\lib\rustlib\x86_64-pc-windows-gnu\bin\rust-lld.exe(0x00007FF741000000) + 0x1256947 byte(s)
          0x00007FF74208C943, C:\Users\eric\.rustup\toolchains\nightly-2023-09-01-x86_64-pc-windows-gnu\lib\rustlib\x86_64-pc-windows-gnu\bin\rust-lld.exe(0x00007FF741000000) + 0x108C943 byte(s)
          0x00007FF74106032E, C:\Users\eric\.rustup\toolchains\nightly-2023-09-01-x86_64-pc-windows-gnu\lib\rustlib\x86_64-pc-windows-gnu\bin\rust-lld.exe(0x00007FF741000000) + 0x6032E byte(s)
          0x00007FF74457EB45, C:\Users\eric\.rustup\toolchains\nightly-2023-09-01-x86_64-pc-windows-gnu\lib\rustlib\x86_64-pc-windows-gnu\bin\rust-lld.exe(0x00007FF741000000) + 0x357EB45 byte(s)
          0x00007FF741001395, C:\Users\eric\.rustup\toolchains\nightly-2023-09-01-x86_64-pc-windows-gnu\lib\rustlib\x86_64-pc-windows-gnu\bin\rust-lld.exe(0x00007FF741000000) + 0x1395 byte(s)
          0x00007FF7410014E6, C:\Users\eric\.rustup\toolchains\nightly-2023-09-01-x86_64-pc-windows-gnu\lib\rustlib\x86_64-pc-windows-gnu\bin\rust-lld.exe(0x00007FF741000000) + 0x14E6 byte(s)
          0x00007FFDD6CF7614, C:\WINDOWS\System32\KERNEL32.DLL(0x00007FFDD6CE0000) + 0x17614 byte(s), BaseThreadInitThunk() + 0x14 byte(s)
          0x00007FFDD75826B1, C:\WINDOWS\SYSTEM32\ntdll.dll(0x00007FFDD7530000) + 0x526B1 byte(s), RtlUserThreadStart() + 0x21 byte(s)

This is part of cargo's testsuite, which is causing issues in our CI.

I have bisected the regression to #114048 (starting nightly-2023-08-09). cc @nikic

It doesn't seem to be happening on any other platforms.

Meta

rustc --version --verbose:

rustc 1.74.0-nightly (203c57dbe 2023-09-17)
binary: rustc
commit-hash: 203c57dbe20aee67eaa8f7be45d1e4ef0b274109
commit-date: 2023-09-17
host: x86_64-pc-windows-gnu
release: 1.74.0-nightly
LLVM version: 17.0.0

Activity

  1. ehuss
    added
    O-windows-gnuToolchain: GNU, Operating system: Windows
    C-bugCategory: This is a bug.
    on Sep 19, 2023
  2. rustbot
    added
    needs-triageThis issue may need triage. Remove when done. See docs forge.rust-lang.org/release/issue-triaging
    on Sep 19, 2023
  3. saethlin
    added
    A-linkageArea: linking into static, shared libraries and binaries
    and removed
    needs-triageThis issue may need triage. Remove when done. See docs forge.rust-lang.org/release/issue-triaging
    on Sep 19, 2023
  4. ehuss
    added
    A-LLVMArea: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues.
    on Oct 2, 2023
  5. bors
    added a commit that references this issue on Oct 2, 2023

    Auto merge of #12763 - ehuss:custom_bin_target-windows-gnu, r=weihanglo

    1f94bf2

  6. cr1901 commented on Oct 10, 2023

    @cr1901
    cr1901
    on Oct 10, 2023
    Contributor

    Maybe this is another useful data point, but I can duplicate with the following from riscv-rt:

    RUSTFLAGS="-C link-arg=-Texamples/device.x" cargo build --example empty --target=riscv32i-unknown-none-elf

    Sometimes it succeeds, sometimes it crashes, sometimes it hangs.

    I found this line in my global cargo config from a while ago:

    [target.riscv32i-unknown-none-elf]
rustflags = [ "-C", "link-arg=--threads=1" ]

    Unfortunately, I don't remember where I learned this- there's probably a Zulip thread- but multithreaded lld is (was?) known to be buggy and hang a lot on Windows, and there's not been much interest in fixing it on the LLD side. So I was a bit surprised to see this issue appear after a while not needing lld.

    Apparently, setting RUSTFLAGS on the command line for quick testing overrides my config. When I add the link-arg flag back in, the crashes go away for me:

    RUSTFLAGS="-C link-arg=--threads=1 -C link-arg=-Texamples/device.x" cargo build --example empty --target=riscv32i-unknown-none-elf

    William@DESKTOP-3H1DSBV MINGW64 ~/Projects/rust-embedded/riscv-rt
$ rustc -Vv
rustc 1.73.0 (cc66ad468 2023-10-03)
binary: rustc
commit-hash: cc66ad468955717ab92600c770da8c1601a4ff33
commit-date: 2023-10-03
host: x86_64-pc-windows-gnu
release: 1.73.0
LLVM version: 17.0.2
  7. bors
    added a commit that references this issue on Nov 14, 2023

    Auto merge of #12972 - ehuss:windows-gnu-ignore-json-target, r=epage

    360059e
  8. yupferris
    added a commit that references this issue on Jan 2, 2024

    sw: Link rv32 code with only one thread

    8788155

  9. mati865 commented on Nov 7, 2024

    @mati865
    mati865
    on Nov 7, 2024
    Member

    This shouldn't happen since GCC update in #119229
    Is this still an issue?

  10. mati865 commented on Apr 25, 2025

    @mati865
    mati865
    on Apr 25, 2025
    Member

    Answering myself: yes, it still fails on CI.

  11. jieyouxu
    added
    CI-spurious-fail-rust-lld-crashCI spurious failure: `rust-lld` crashing / SIGSEGV / 0xc0000374 heap corruption
    on Apr 25, 2025

  12. jieyouxu commented on Apr 25, 2025

    @jieyouxu
    jieyouxu
    on Apr 25, 2025
    Member

    Has also been crashing on rust-lang/rust windows-gnu CI quite frequently, cf. #137938

  13. jieyouxu
    marked rust-lld: on x86_64-pc-windows-gnu it spuriously fails with 0xc0000374 #137938 as a duplicate of this issue on Apr 25, 2025

  14. mati865 commented on Apr 25, 2025

    @mati865
    mati865
    on Apr 25, 2025
    Member

    Yeah, I meant Rust's CI.

  15. ChrisDenton commented on Apr 28, 2025

    @ChrisDenton
    ChrisDenton
    on Apr 28, 2025
    Member

    I think the frequency may have been increasing lately. In the last 24hrs #140356 and #123239 (twice).

  16. ChrisDenton commented on Apr 28, 2025

    @ChrisDenton
    ChrisDenton
    on Apr 28, 2025
    Member

    Looking back through the logs it does seem we are indeed hitting this more often in recent days:

    February (1):

    March (3):

    April (10):

  17. bors
    added a commit that references this issue on Apr 28, 2025

    Auto merge of rust-lang#140396 - ChrisDenton:gnu-threads, r=<try>

    46b91f6
  18. ChrisDenton
    added 3 commits that reference this issue on Apr 28, 2025

    Rollup merge of rust-lang#140396 - ChrisDenton:gnu-threads, r=jieyouxu

    fe2835c

    Rollup merge of rust-lang#140396 - ChrisDenton:gnu-threads, r=jieyouxu

    7aa661f

    Rollup merge of rust-lang#140396 - ChrisDenton:gnu-threads, r=jieyouxu

    469f03d
  19. rust-timer
    added a commit that references this issue on Apr 29, 2025

    Unrolled build for rust-lang#140396

    60bb7d5
  20. github-actions
    added a commit that references this issue on Apr 29, 2025

    Rollup merge of #140396 - ChrisDenton:gnu-threads, r=jieyouxu

    566169d

  21. mati865 commented on Sep 7, 2026

    @mati865
    mati865
    on Sep 7, 2026
    Member

    We have disabled memory freeing in LLD shipped with windows-gnu because this issue is much more frequent with LLVM 23 https://rust-lang.zulipchat.com/#narrow/channel/131828-t-compiler/topic/LLD.20crash.20on.20mingw.20related.20to.20TLS.20destruction/with/611316681

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-LLVMArea: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues.A-linkageArea: linking into static, shared libraries and binariesC-bugCategory: This is a bug.CI-spurious-fail-rust-lld-crashCI spurious failure: `rust-lld` crashing / SIGSEGV / 0xc0000374 heap corruptionO-windows-gnuToolchain: GNU, Operating system: Windows

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions