[guitargeek]

All source archives of the builtin_* options were downloaded from a single location, the CERN mirror (lcgpackages, or root.cern for libjpeg-turbo and libpng). This lists a second, external mirror for each of them, so that the download keeps working when the CERN network is unreachable: CMake tries the URLs in order.

Every external URL was verified to serve the byte-identical file (same SHA256 as the pinned one): 18 from GitHub (clad, ftgl, gtest, jsroot, lz4, lzma, mathjax, nlohmann, openssl, openui5, pcre, tbb, vdt, xrootd, xxhash, zlib, zstd, and libjpeg-turbo's tag archive), the rest from the upstream project sites (fftw.org, savannah for freetype, osgeo for libtiff, HEASARC for cfitsio, statmath.wu.ac.at for unuran, ftp.gnu.org for gsl, geuz.org for gl2ps, SourceForge for giflib and libpng). The git clones (civetweb, AdaptiveCpp, clad with CLAD_GIT_TAG) are not archives and stay as they are.

Along the way:

• Since the version now appears in two URLs, every builtin keeps it in a single variable (added for clad, jsroot, mathjax and tbb; tbb's ROOT_TBB_VERSION was referenced for the parent scope but never set), and the SHA256 is kept in a *_HASH variable on the line right after the version, so that a version bump cannot forget the hash (added where it was missing: cfitsio, clad, fftw3, gsl, jsroot, mathjax, openssl, openui5, tbb, vdt, xrootd). The single-use *_URL variables are dropped and the URLs are written inline in the ExternalProject_Add call.

• The unuran tarball on LCG turned out to be the pristine upstream release, so its stale "repackaged" comment is removed.

• giflib uses the LCG copy instead of root.cern, like the other builtins.

[guitargeek]

Thanks for the review! @ferdymercury, does it look okay for you now?

[ferdymercury]

The git clones (civetweb) are not archives and stay as they are.

Might change with #23948 and #23354

One unrelated question: are there plans to make CppInterOp also behave like an external? Rather than copy-pasting the whole code at every sha commit upgrade?

If the main reason against this is to make sure that ROOT compiles without internet, there could be an easy workaround that could be given clear 'instructions' to those subset of users:

• For users with internet --> git clone XXX and then cmake, CppInterOp will be pulled via AddExternalProject
• For users without internet (with connection to local network git mirror) --> git config --global submodule.recurse true && git pull , so that it gets pulled via git submodules.
• For users without internet (initially with, downloading tarball .tar.gz) ---> the tar.gz source tarball could be generated after recursing submodules.

Or just always using git submodules and forgetting about AddExternalProject. After https://stackoverflow.com/a/49427199/7471760 life is much easier with submodules.

[ferdymercury]

Cool, thanks.

Nitpick: OpenSSL, xrootd and cfitsio varibles start with PACKAGE_HASH rather than ROOT_PACKAGE_HASH like the majority.

[guitargeek]

One unrelated question: are there plans to make CppInterOp also behave like an external? Rather than copy-pasting the whole code at every sha commit upgrade?

Maybe one day when we have reduced the required patches (hopefully during next year). Until then, we treat CppInterOp and cppjit like LLVM/Clang: tracked as a copy of the source code with well-specified patches, including a CI check that the the differences don't drift beyond the current patches.

If the main reason against this is to make sure that ROOT compiles without internet, there could be an easy workaround that could be given clear 'instructions' to those subset of users:

* For users with internet --> git clone XXX and then cmake, CppInterOp will be pulled via AddExternalProject

* For users without internet (with connection to local network git mirror) --> git config --global submodule.recurse true && git pull , so that it gets pulled via git submodules.

* For users without internet (initially with, downloading tarball .tar.gz)  ---> the tar.gz source tarball could be generated after recursing submodules.

Or just always using git submodules and forgetting about AddExternalProject. After https://stackoverflow.com/a/49427199/7471760 life is much easier with submodules.

You're asking the right question here! Building without internet is indeed a requirement, and we discussed this also in #23877 (comment) with @linev.

Fortunately, there is a native mechanism to allow this, with ExternalProject_Add' s DOWNLOAD_DIR option: I plan to define a new ROOT build flag called ROOT_DOWNLOAD_DIR, and if set, it will forward to DOWNLOAD_DIR for all ExternalProject_Adds in builtins/ and Clad. That's quite a minimal change and can even be backported to 6.42, so we don't regress on ROOT's capability to be build without internet connection with the right CMake configuration flags.

Git submodules are maybe a nice solution if you want to use random tags and commits for the builtins, but then it's more difficult again to use mirrors. GitHub is often having hick-ups these days. And given there are clear release artifacts for all of our builtins (except for civetweb, until its next release), the tarball solution works quite well and I don't see a compelling reason to move away from them. Would you prefer submodules for some specific reason? We changed already a lot for the builtins in 6.42, which will already create quite some churn for packagers... We should be careful that we don't have this in every release.

[guitargeek]

Cool, thanks.

Nitpick: OpenSSL, xrootd and cfitsio varibles start with PACKAGE_HASH rather than ROOT_PACKAGE_HASH like the majority.

Thanks! No reason not to address the nitpick. Will update.

[ferdymercury]

new ROOT build flag called ROOT_DOWNLOAD_DIR

sounds great.

Would you prefer submodules for some specific reason?

No, no problem, as long as there is an alternative solution that does not imply keeping mirroring all file changes at each upgrade.

tracked as a copy of the source code with well-specified patches, including a CI check that the the differences don't drift beyond the current patches.

This is to me a more involved solution (and bloating the git repo size) than just using addExternalProject (or the future DOWNLOAD_DIR) plus "git apply MYPATCHES.diff".

For example, the clang patches are just 100KB size: https://github.com/llvm/llvm-project/compare/release/22.x...root-project:llvm-project:ROOT-llvm22.diff

so you could remove the interpreter/llvm-project directory and only copy the source into your build directory via ExternalProject or via download_dir, and then PATCH_COMMAND that hard-coded diff.
Or https://github.com/llvm/llvm-project/compare/release/22.x...root-project:llvm-project:ROOT-llvm22.patch if you want the separate commits rather than a single diff.

That way, at every upgrade, you only need to update that diff file and HASH.

The benefit would be that you would save 300Mb in disk space in you root_src source folder. Git repository size is already 2GB, and LLVM receives many file changes every release, far less than the modified patches, so to keep it from growing indefinitely (until patches are no longer needed or clang-repl is used).

[guitargeek]

This is a little bit of a tangent from this PR, but the point with the repo size is an important one I think. However, the interpreter packages are different because they are pretty much co-developed with ROOT itself by the same people. So having all code together in one repo is a major simplification for day-to-day developer work and onboarding. But I hope that one day as the interpreter code stabilizes, we can do what you suggest 🙂

[guitargeek]

/backport to 6.42

[root-project-bot]

Preparing to backport PR #23963 to branch 6.42 requested by guitargeek

[root-project-bot]

This PR has been backported to branch 6.42: #23973

[ferdymercury]

unuran seems to complain in https://github.com/root-project/root/actions/runs/41820421662/job/126541889380?pr=23979

do we need a clean-build label temporarily?

[guitargeek]

unuran seems to complain in https://github.com/root-project/root/actions/runs/41820421662/job/126541889380?pr=23979

do we need a clean-build label temporarily?

Unfortunately yes, at least until the next inremental build is done (I guess in half an hour). I would just wait for that, then rebase the PR on master and push. Like this, we avoid the full rebuild.