Skip to content

Score diatomics on PBE bond length, wall, and force flips - #456

Closed
anyangml wants to merge 10 commits into
deepmodeling:mainfrom
anyangml2nd:feat/diatomic-pbe-metrics
Closed

anyangml wants to merge 10 commits into
deepmodeling:mainfrom
anyangml2nd:feat/diatomic-pbe-metrics

Conversation

@anyangml

Copy link
Copy Markdown
Collaborator

Summary

  • Replace the 14-dimer MLIP Arena curves with the Matbench Discovery PBE homonuclear set (H–U, 87 scored elements; Po, At, Rn, Fr, and Ra omitted). r2SCAN is not used.
  • Score each curve on equilibrium bond length, repulsive-wall radius (1–100 eV above the well), and whether Fx changes sign exactly once. Jumpy or non-finite model curves are dropped from the averages and reduce coverage.
  • Combine the three terms into one leaderboard score: dummy-normalize the two geometry MAEs (mean PBE predictor, capped at 1), average them equally with the force-flip rate, and divide by coverage. This replaces Applicability-Roughness.

Test plan

  • pytest tests/tasks/calculator/test_diatomics.py tests/metrics/test_metrics_calculator.py
  • Re-run homonuclear_diatomics for leaderboard models; stored roughness records will aggregate to a null score until then

Made with Cursor

…rce flips.

The previous roughness score was scale-invariant and only covered 14 main-group dimers, so it missed equilibrium geometry and the short-range wall.

Co-authored-by: Cursor <cursoragent@cursor.com>
@codecov

codecov Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.65217% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.45%. Comparing base (e137fd9) to head (2034026).
⚠️ Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
lambench/tasks/calculator/diatomics/diatomics.py 86.44% 12 Missing and 4 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #456      +/-   ##
==========================================
+ Coverage   65.19%   69.45%   +4.26%     
==========================================
  Files          37       37              
  Lines        1879     2092     +213     
  Branches      230      251      +21     
==========================================
+ Hits         1225     1453     +228     
+ Misses        602      587      -15     
  Partials       52       52              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The scored reference was already committed, but as one 91 KB line, so the pull request diff looked like the old curves had been deleted.

Co-authored-by: Cursor <cursoragent@cursor.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

anyangml and others added 8 commits September 29, 2026 15:40
Each of the 87 stored curves runs from 0.8 covalent radii to 6 Å with a short-range wall, and bound curves that pass the smoothness gate have an interior minimum.

Co-authored-by: Cursor <cursoragent@cursor.com>
The published file deleted 45 distances across 11 curves, and the raw spin candidates are not available to fill those holes. Scoring a gappy curve would treat the deletion as if that geometry had never been computed.

Co-authored-by: Cursor <cursoragent@cursor.com>
The 76 complete PBE curves are already the contents of diatomics.json, so the loader no longer drops elements or rebuilds the design grid on each read.

Co-authored-by: Cursor <cursoragent@cursor.com>
Replace wall-radius and boolean flip metrics with interpretable well-depth error and raw force-flip counts, using explicit blind baselines for aggregation.

Co-authored-by: Cursor <cursoragent@cursor.com>
Preserve the simplified PBE diatomic metrics while adopting the latest upstream project refactor.

Co-authored-by: Cursor <cursoragent@cursor.com>
Keep the repository's established E/F rule scope explicit and resolve the remaining import-order findings without changing runtime behavior.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants