Skip to content

fix: resolve native sources for pip installations - #3970

Merged
paul-nechifor merged 3 commits into
mainfrom
cc/fix/rust-recorder-chore
Oct 7, 2026
Merged

paul-nechifor merged 3 commits into
mainfrom
cc/fix/rust-recorder-chore

Conversation

@TomCC7

@TomCC7 TomCC7 commented Sep 8, 2026 •

Copy link
Copy Markdown
Member

Problem

  • Pip installations lack native build sources. Inconsistent source and executable paths break installed use, and early source resolution can fetch a checkout during replay.

Solution

  • Require source builds to declare a repository-relative source_dir and relative executable. Build and run from that source directory.
  • Remove public cwd; old arguments raise a migration error. External binaries use source_dir=None and build_command=None, anchor relative executable paths at initialization, and run from the executable directory without fetching sources.
  • Defer source preparation until build or start. Start reuses completed preparation; recorder replay skips native preparation and launch. Preserve existing auto-build rules.
  • Reuse get_project_root(): use the development checkout, or clone main once under a file lock and reuse it without automatic updates. The cached revision can differ from the installed package.
  • Migrate built-in modules, examples, and the existing build scanner. Keep Habitat paths consistent and preserve IsolatedPython behavior. Add no CI jobs.
  • Target main directly. MCAP reader and CLI enhancements from fix: decode native MCAP recordings in memory tools #4434 are excluded and remain separate.
  • Validation: 123 focused tests, mypy for 23 production files, applicable hooks, and an offline lockfile check passed. Native compilation and hardware compatibility were not revalidated.

@codecov

codecov Bot commented Sep 8, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.06630% with 7 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
dimos/experimental/memory/rust_cli_recorder.py 91.30% 1 Missing and 1 partial ⚠️
dimos/experimental/memory/rust_recorder.py 80.00% 1 Missing and 1 partial ⚠️
dimos/utils/data.py 81.81% 2 Missing ⚠️
dimos/utils/test_data.py 95.00% 1 Missing ⚠️
@@            Coverage Diff             @@
##             main    #3970      +/-   ##
==========================================
+ Coverage   80.08%   80.12%   +0.03%     
==========================================
  Files        1611     1612       +1     
  Lines      151456   151694     +238     
  Branches    12841    12855      +14     
==========================================
+ Hits       121294   121541     +247     
+ Misses      26601    26594       -7     
+ Partials     3561     3559       -2     
Components Coverage Δ
Tests 95.07% <99.55%> (+0.01%) ⬆️
Flag Coverage Δ
OS-macos-latest 74.61% <97.79%> (+0.04%) ⬆️
OS-ubuntu-24.04-arm 74.26% <97.79%> (+0.04%) ⬆️
OS-ubuntu-latest 74.67% <97.79%> (+0.04%) ⬆️
Py-3.10 74.66% <97.79%> (+0.04%) ⬆️
Py-3.11 74.66% <97.79%> (+0.04%) ⬆️
Py-3.12 74.66% <97.79%> (+0.04%) ⬆️
SelfHosted-Large 31.97% <27.01%> (-0.01%) ⬇️
SelfHosted-Linux 39.56% <30.36%> (-0.02%) ⬇️
SelfHosted-macOS 39.90% <30.36%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
dimos/core/baked_host.py 97.22% <100.00%> (ø)
dimos/core/native_module.py 84.06% <100.00%> (+3.27%) ⬆️
dimos/core/test_build_native_modules.py 94.24% <100.00%> (-0.03%) ⬇️
dimos/core/test_native_module.py 98.02% <100.00%> (+0.65%) ⬆️
dimos/evals/environments/habitat.py 89.41% <100.00%> (+0.25%) ⬆️
dimos/evals/environments/test_habitat.py 100.00% <100.00%> (ø)
...imos/experimental/memory/test_rust_cli_recorder.py 100.00% <100.00%> (ø)
dimos/experimental/memory/test_rust_recorder.py 100.00% <100.00%> (ø)
...imos/experimental/memory/test_rust_recorder_e2e.py 90.64% <100.00%> (-0.05%) ⬇️
dimos/experimental/robot/bosdyn/spot/recorder.py 100.00% <100.00%> (ø)
... and 20 more

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs.

@github-actions github-actions Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

go2 replay realtime (arm64)

Details
Benchmark suite Current: 9d93ef7 Previous: faf1bed Ratio
peak memory 1571.949 MB 1616.949 MB 0.97
peak threads 395 threads 402 threads 0.98
network (transport) 2732.688 MB 2738.458 MB 1.00
disk write 1.617 MB 1.598 MB 1.01
instructions 124.645 G 124.651 G 1.00

This comment was automatically generated by workflow using github-action-benchmark.

@github-actions github-actions Bot removed the stale label Oct 3, 2026
@TomCC7
TomCC7 changed the base branch from main to cc/fix/native-mcap-tools October 6, 2026 01:14
@TomCC7
TomCC7 added this pull request to stack #4442 October 6, 2026 03:19
@TomCC7
TomCC7 force-pushed the cc/fix/native-mcap-tools branch from 0982249 to c9f7428 Compare October 6, 2026 03:41
Comment thread bin/build-native-modules Resolved
Comment thread dimos/native_packages.json Resolved
@TomCC7 TomCC7 changed the title fix: unify native package preparation for pip installs fix: resolve native sources for pip installations Oct 6, 2026
@TomCC7
TomCC7 marked this pull request as ready for review October 6, 2026 07:04
@github-actions github-actions Bot added the ready-to-merge Required CI checks have passed on this PR label Oct 7, 2026
paul-nechifor
paul-nechifor previously approved these changes Oct 7, 2026
@TomCC7
TomCC7 force-pushed the cc/fix/rust-recorder-chore branch from 5fa504e to c0118ac Compare October 7, 2026 02:33
@github-actions github-actions Bot removed the ready-to-merge Required CI checks have passed on this PR label Oct 7, 2026
@TomCC7
TomCC7 removed this pull request from stack #4442 October 7, 2026 02:39
@TomCC7
TomCC7 changed the base branch from cc/fix/native-mcap-tools to main October 7, 2026 02:40
@TomCC7
TomCC7 enabled auto-merge (squash) October 7, 2026 02:58
paul-nechifor
paul-nechifor previously approved these changes Oct 7, 2026
@TomCC7 TomCC7 closed this Oct 7, 2026
auto-merge was automatically disabled October 7, 2026 03:44

Pull request was closed

@TomCC7 TomCC7 reopened this Oct 7, 2026
@TomCC7
TomCC7 enabled auto-merge (squash) October 7, 2026 03:49
Comment thread dimos/core/native_module.py Resolved
Comment thread docs/usage/native_modules.md Resolved
@github-actions github-actions Bot added the ready-to-merge Required CI checks have passed on this PR label Oct 7, 2026
@github-actions github-actions Bot added ready-to-merge Required CI checks have passed on this PR and removed ready-to-merge Required CI checks have passed on this PR labels Oct 7, 2026
@paul-nechifor
paul-nechifor disabled auto-merge October 7, 2026 06:18
@paul-nechifor
paul-nechifor merged commit f22803d into main Oct 7, 2026
72 of 74 checks passed
@paul-nechifor
paul-nechifor deleted the cc/fix/rust-recorder-chore branch October 7, 2026 06:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-to-merge Required CI checks have passed on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants