Skip to content

Make FFN subvolume checkpointing and preemption resume deterministically: - #155

Merged
copybara-service[bot] merged 1 commit into
masterfrom
test_1025452684
Nov 20, 2026
Merged

copybara-service[bot] merged 1 commit into
masterfrom
test_1025452684

Conversation

@copybara-service

Copy link
Copy Markdown

Make FFN subvolume checkpointing and preemption resume deterministically:

  • Track the popped in-flight FoV candidate in FaceMaxMovementPolicy and re-queue it at the head of scored_coords if checkpointed before update() completes.
  • Rewind seed_policy (in_progress_seed=True) when checkpointing on TerminationException at num_iters == 0 or at pre-segment periodic checkpoints, and check is_shutting_down() between seeds in Canvas.segment_all.
  • Serialize nested movement_policy and seed_policy_state objects as 0-d object arrays in Canvas.save_checkpoint, serialize saves with a reentrant Canvas.io_lock, and preserve _max_id and integer partial_segment_iters on restore.
  • Evaluate SequentialPolicies sub-policies lazily so later stages see earlier stages' segmentation, and rewind only the active sub-policy on in-progress checkpoints.
  • Make ThreadingExecutorClient.finish and ThreadingBatchExecutor.stop_server idempotent when exit_request is already set, and clear Runner.canvases entries in a finally block.

…lly:

- Track the popped in-flight FoV candidate in `FaceMaxMovementPolicy` and re-queue it at the head of `scored_coords` if checkpointed before `update()` completes.
- Rewind `seed_policy` (`in_progress_seed=True`) when checkpointing on `TerminationException` at `num_iters == 0` or at pre-segment periodic checkpoints, and check `is_shutting_down()` between seeds in `Canvas.segment_all`.
- Serialize nested `movement_policy` and `seed_policy_state` objects as 0-d object arrays in `Canvas.save_checkpoint`, serialize saves with a reentrant `Canvas.io_lock`, and preserve `_max_id` and integer `partial_segment_iters` on restore.
- Evaluate `SequentialPolicies` sub-policies lazily so later stages see earlier stages' segmentation, and rewind only the active sub-policy on in-progress checkpoints.
- Make `ThreadingExecutorClient.finish` and `ThreadingBatchExecutor.stop_server` idempotent when `exit_request` is already set, and clear `Runner.canvases` entries in a `finally` block.

PiperOrigin-RevId: 1026329651
@copybara-service
copybara-service Bot merged commit 29fbe28 into master Nov 20, 2026
3 of 4 checks passed
@copybara-service
copybara-service Bot deleted the test_1025452684 branch November 20, 2026 10:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant