Skip to content

fix: stop DO changes from being lost when a peer misses a sync - #9563

Open
pjwerneck wants to merge 2 commits into
devfrom
pjwerneck/fix-lost-sync
Open

pjwerneck wants to merge 2 commits into
devfrom
pjwerneck/fix-lost-sync

Conversation

@pjwerneck

Copy link
Copy Markdown
Collaborator

Summary

DO clients that share a SyftBox folder share one on-disk hash cache, and a sync recorded a change as seen before it sent the change. A peer that the version gate skipped, or whose upload failed, never got the change. This PR keeps each change pending per peer until the upload to that peer succeeds.

Changes

  • Seen with syft-bg on Colab: apis and auto-approved job results never reached the submitter, and a later do.sync() sent nothing
  • The DO records each queued path as pending per peer in .cache/owner_pending_deliveries.json, and clears it after the upload to that peer succeeds
  • Approved peers that fail the version gate are deferred: their changes stay pending and go out in the first sync where the peer passes
  • Each sync resends the current state of pending paths (a deletion if the file is gone) and drops paths the peer can no longer read
  • A pending file that exists but cannot be read raises instead of going out as a deletion
  • PersistedDict.update_entry: read-modify-write under one exclusive lock

Testing

  • tests/unit/test_persisted_cache.py: a second DO process (syft-bg) syncs first and skips the peer, fails the upload, defers a deletion, or hits a read error; the peer still ends up with the right state. All four fail without the fix

Asana task

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant