[Stardust0831]

What's changed?

Enable optional GPU acceleration of LCAO exact-exchange RI tensor contractions through LibRI's DDLA CUDA backend. ENABLE_LIBRI_GPU defaults to OFF, and INPUT device_exx defaults to cpu. Setting device gpu alone keeps RI contractions on the CPU; set device_exx gpu to select GPU RI as well.

This PR remains a draft pending a compatible LibRI release. The required accumulation, OpenMP, atom-list and symmetry fixes have merged upstream in LibRI #15 and #16, but are not included in the latest published release, v0.2.1.1. Development tests use an exact upstream commit; the supported dependency will be a validated official release.

Before marking this PR ready:

• Confirm that the published LibRI release contains the required fixes.
• Set the minimum supported GPU LibRI version and its configure-time check, and update installation instructions to the release tag.
• Repeat CPU/GPU integration checks with that exact release and record the result.

Build configuration, once compatible dependencies are installed:

cmake -B build -DENABLE_LIBRI=ON -DUSE_CUDA=ON -DENABLE_LIBRI_GPU=ON \
  -DLIBRI_DIR=/path/to/LibRI -DDDLA_ROOT=/path/to/DDLA

device      gpu
device_exx  gpu

Real-space EXX symmetry is supported with symmetry 1 and exx_symmetry_realspace 1. The obsolete ABACUS rejection has been removed, and its input test now verifies acceptance without changing the user's settings. Installation instructions no longer require a Gitee branch or extra OpenMP patch.

Dependencies and supported scope

• GPU RI requires CUDA and an explicit LIBRI_DIR. DDLA v0.0.4 (9c404ac) is the tested backend. CMake checks GPU API compilation and linking; the minimum release check remains on the draft checklist above. The default CPU dependency download is unchanged.
• The selector applies to LCAO Kohn-Sham exchange. Independent LR/RDMFT exchange engines retain CPU contractions; LR inherits the selection when reusing the Kohn-Sham exchange object.
• All MPI ranks on a node must see the same allocated GPU list, with at least as many visible GPUs as local ranks. Per-rank single-GPU masking is unsupported.
• Details are in docs/advanced/install.md and abacus -h device_exx.

Unit Tests and/or Case Tests for my changes

Development validation uses ABACUS 1bd79c473 and upstream LibRI a0661cb, with no additional patches to either repository. Hardware: NVIDIA V100-SXM2 16 GB GPUs, GCC 13.3/OpenMP and DDLA v0.0.4/CUDA. These results do not replace the final release validation above.

Check	Result
Full GPU build, including BSE	Passed
INPUT unit tests with GPU RI enabled / disabled	16/16 in each build
CLI version/help/metadata and INPUT acceptance/rejection	Passed; all 9 INPUT cases matched expectations
CPU/GPU dependency consumers and signed accumulation	3/3 CTest checks passed
HSE with real/complex RI tensors, one/two MPI ranks and symmetry on/off	20/20 repository checks and 16/16 backend comparisons passed
LibRI symmetry, accumulation, Hartree and atom-list regressions	All passed, including both CPU implementations and GPU execution

Complete build logs, inputs, commands and comparisons include both source revisions and checksums. OMP_NUM_THREADS=1 is used unless a test explicitly checks four threads. Repository reference files and tolerances are unchanged.

Verification commands and limits

The linked record contains the complete configure commands and raw output. The principal ABACUS checks are:

cmake --build build-gpu --target abacus_std_gpu MODULE_IO_read_item_serial --parallel 12
ctest --test-dir build-gpu -V -R '^MODULE_IO_read_item_serial$'
cmake --build build-cpu --target MODULE_IO_read_item_serial --parallel 12
ctest --test-dir build-cpu -V -R '^MODULE_IO_read_item_serial$'
abacus --version
abacus -h device_exx
abacus --check-input

HSE tests use the existing tests/integrate/Autotest.sh and scf_hse_symm_multik fixture. Five solver/RI/symmetry combinations run with real and complex RI tensors on one and two MPI ranks. The fixture's forces vanish by symmetry, so this matrix does not establish nonzero-force behavior with symmetry enabled.

BSE is compiled but its numerical paths are not tested. MAGMA, full MKL application builds, NCCL runtime behavior, multi-node execution, SOC and long MD runs remain unverified. GPU regression builds disable NCCL because the input-test target previously had unresolved NCCL symbols. Earlier Si32 testing needed scf_ene_thr=1e-8 for consistent outer-loop stopping; default-threshold convergence needs further validation.

Earlier Si64 force benchmark

These timings were measured at ABACUS 63ca5ef with the earlier dependency configuration and have not been rerun with the current LibRI code. A displaced Si64 HSE/DZP Gamma-point case uses 832 orbitals. Hardware: V100-SXM2 16 GB GPUs and an AMD Ryzen Threadripper PRO 5995WX CPU, with four physical CPU cores per GPU.

CPU-only and single-GPU runs have the same four physical cores; two-GPU runs have eight. Seven CPU MPI/OpenMP ratios and two single-GPU configurations were tested. Selected configurations were run twice; the table reports their mean times. The CPU baseline uses the fastest tested configuration for each stage.

Stage	Tuned CPU only (s)	1 GPU + 4 CPU cores (s)	2 GPUs + 8 CPU cores (s)	CPU / 1 GPU
EXX force	72.75	21.45	19.42	3.39x
All force work	75.44	22.91	20.37	3.29x
SCF + forces	352.58	238.50	192.07	1.48x

CPU-only uses ScaLAPACK and CPU RI; GPU runs use cuSOLVER and GPU RI. Holding cuSOLVER, MPI/OpenMP settings and CPU affinity fixed, changing CPU RI to GPU RI gives 3.87x for all force work and 1.49x overall. This comparison uses one CPU-RI run and the mean of two GPU-RI runs.

All 15 runs converged in 7 outer and 96 inner iterations. Across all 192 force components, the largest difference from the CPU reference was 1.5e-9 eV/Angstrom. These measurements cover one case on one node; EXX and all-force times are nested timers, while complete-calculation time includes launch, initialization and SCF.

Linked Issue

No linked ABACUS issue; this adds optional support for an existing LibRI backend.

Governance Notes

• INPUT/docs: device_exx includes validation, MPI broadcast, help metadata and focused tests. GPU symmetry acceptance and installation guidance are updated together; generated parameter documentation is not committed.
• Core impact: select the backend before LCAO EXX initialization and encapsulate dependencies in LibRI::LibRI and DDLA::DDLA. No new global dependencies. GPU RI requires C++17; CPU RI retains C++14.
• The standalone dependency test initializes MPI directly because it does not link the ABACUS runtime. Communicator queries use the LibRI wrapper.
• Follow-up: add DDLA installation to the existing toolchain workflow.

[userchba]

the LibDDLA version which is compatible with LibRI GPU is https://github.com/userchba/LibDDLA/releases/tag/v0.0.4, current LibRI GPU can't correctly compute symmetry situation which LibRI CPU can compute correctly

[ABACUS-PR-BOT]

Static review of f07d8d14f39d26782490cc4bc15d0a58c5700e62 with independent source review and adjudication. No build/runtime/MPI/GPU tests were run; CI diagnostics were unavailable.

[ABACUS-PR-BOT]

[P2] State the GPU visibility requirement for MPI runs

For two ranks on one node, exposing one different GPU per rank through CUDA_VISIBLE_DEVICES passes ABACUS device setup but makes the pinned LibRI validate_local_gpu_count() throw at the first GPU contraction (local_size=2 > device_count=1). This backend therefore requires each process to see at least as many GPUs as there are node-local ranks, a constraint missing here. Please document it with a compatible launch layout, or extend the compatibility patch to respect the existing per-rank binding.

[ABACUS-PR-BOT]

Non-blocking: Please clarify the selector's scope for independent LR/RDMFT exchange engines. Standalone LR, fresh LR-kernel setup after KS, and RDMFT construct their own Exx_LRI objects, which retain CPU mode despite accepted exx_device=gpu; matching-kernel KS pointer reuse retains GPU mode. If those engines are in scope, pass the selection to them; otherwise document or reject the unsupported combinations. A follow-up after merge is fine for these additional workflows.

[ABACUS-PR-BOT]

Non-blocking: Consider an opt-in numerical regression for the patch's signed accumulation in a follow-up after merge. This consumer switches cal_mode but then executes only CPU nrm2, and the configure probe only compiles/links. Alongside the reported force/stress runs, a tiny real/complex CPU-versus-GPU case with nonempty output and repeated +1/-1 contractions would preserve the specific accumulation fix across dependency updates.

[mohanchen]

To be consistent, I think device_exx maybe better than exx_device

[Stardust0831]

To be consistent, I think device_exx maybe better than exx_device

Thank you for the review and suggestions; I’ve renamed exx_device to device_exx.

[QuantumMisaka]

@PeizeLin @maki49 May you have a look? Thanks !

[mohanchen]

@Growl1234 Could you check the cmake changes for this PR?

[Growl1234]

Does it mean that LibRI master branch hasn't yet accepted the LibRI changes? If this is true then this PR should not be merged on abacus side first.

[Growl1234]

LibDDLA has CMake config export, why is FindDDLA module needed here?

[maki49]

I have no further comment within this PR's scope.
LR and symmetry with LibRI on GPU are already supported in my local branch but they need to update LibRI. I will make a new PR in the future when everything is ready.

[Stardust0831]

An update on the LibRI dependency fixes:

• Fix DDLA GPU compilation abacusmodeling/LibRI#14 has been merged, resolving the DDLA GPU compilation issues.
• Fix accumulator aliasing and Hartree atom-list indexing abacusmodeling/LibRI#15 fixes signed and scaled accumulation in GPU contractions and removes the unnecessary Ds_result_thread from the GPU path. It has been submitted but is not yet merged.

The ABACUS test results reported in this PR still use the temporary compatibility patch. The new direct-accumulation implementation has passed LibRI-level GPU regression tests with DDLA v0.0.4, but this PR has not yet been updated to use it or retested with full ABACUS HSE calculations.

I plan to move this PR back to draft while the remaining LibRI fix is pending upstream. Once merged, I will update the dependency, remove the compatibility patch, address the remaining review comments, and sync with develop. I will then rerun the HSE single-point energy, force, and stress regressions before marking this PR ready for review again.