Skip to content

fix: report sub-minute job timeouts in seconds - #9560

Open
choksi2212 wants to merge 1 commit into
OpenMined:devfrom
choksi2212:fix/timeout-duration-formatting-9527
Open

choksi2212 wants to merge 1 commit into
OpenMined:devfrom
choksi2212:fix/timeout-duration-formatting-9527

Conversation

@choksi2212

Copy link
Copy Markdown

Summary

All three job-runner timeout messages formatted the configured duration with integer division (timeout // 60), so any timeout below 60 seconds printed as timed out after 0 minutes (#9527). This PR reports sub-minute timeouts in seconds and keeps the existing minutes phrasing for durations of a minute or longer.

Changes

  • Add format_timeout_duration(seconds) to syft_job/job_runner.py: < 60 renders as N second(s), >= 60 renders as N minute(s) with correct singular/plural; the truncation for minute-plus values is unchanged (61s was already 1 minute before, just with the wrong plural).
  • Replace the inline timeout // 60 expression at all three timeout message sites: the streaming loop (_execute_job_streaming), the captured runner (_execute_job_captured), and the runner-level subprocess.TimeoutExpired handler.

Testing

  • New packages/syft-job/tests/test_timeout_duration.py (8 tests):
    • Unit coverage of format_timeout_duration across the sub-minute regime (1, 30, 59), the minute boundary (60, 61, 120, 600), and singular/plural forms.
    • End-to-end: run a job that sleeps past a 1-second timeout through _execute_job_captured and assert the printed message reads timed out after 1 second and never 0 minutes (the captured path runs the same print site the streaming path uses; the streaming loop's selectors-on-pipes behavior is not exercised on Windows, matching the existing suite which runs it on Linux CI).
  • Negative control, run both directions on Linux (Python 3.14):
    • With this change reverted (pristine dev), a real 1-second timeout prints Job <name> timed out after 0 minutes — the exact symptom from the issue.
    • With this change, the same run prints Job <name> timed out after 1 second.
  • Full packages/syft-job test suite: 134 passed, 11 failed with this change; 126 passed, the identical 11 failed on pristine dev (the test_staging.py returncode-127 cluster, pre-existing and environment-dependent, untouched by this diff).
  • ruff check and ruff format (v0.12.9, the pinned pre-commit rev) clean on both files.

Asana task


All three job-runner timeout paths formatted the duration with integer
division (timeout // 60), so a timeout configured below 60 seconds
printed as "timed out after 0 minutes". Add a shared
format_timeout_duration helper that reports sub-minute timeouts in
seconds and keeps the minutes phrasing for a minute or longer, and use
it at every timeout message site.

Closes OpenMined#9527

Signed-off-by: choksi2212 <choksi2212@users.noreply.github.com>
@choksi2212

Copy link
Copy Markdown
Author

Thanks for the review bandwidth in advance. The workflow runs on this PR are waiting on a maintainer approval since this is the first contribution from this fork — they can be approved from the PR's checks section or the Actions tab whenever someone has a moment.

@choksi2212

Copy link
Copy Markdown
Author

Both red checks look unrelated to this change and infra-side rather than code-side:

  • enclave-tests failed during job setup before any test executed (curl: (22) The requested URL returned error: 403 while fetching a dependency).
  • integration-tests failed on test_peer_request_blocks_sync_until_approved with a JSONDecodeError during test setup (peer-sync fixture), not on any job-timeout path. The same suite is currently red on dev nightly too, on three different gdrive/sync tests (e.g. test_do_cache_aware_sync_gdrive).

The suites that cover the change all pass: unit-tests, job-tests, integration-mock-mode-tests, pre-commit, rds-tests, migration-tests, perm-tests. Happy to rebase/re-run if it's cleaner for review.

@choksi2212

Copy link
Copy Markdown
Author

Both red checks on the latest head look like runner-side infra flakes rather than this change:

  • enclave-tests fails during environment setup on a curl ... 403 while downloading a tool — no test code involved (log).
  • integration-tests errors at setup of test_peer_request_blocks_sync_until_approved (json.decoder.JSONDecodeError before the test body runs), with 2 passed / 7 skipped. The touched files here are only packages/syft-job/src/syft_job/job_runner.py + its test — a sync-manager fixture setup failure is unrelated. The same job also failed on other open PRs the same week (e.g. the Oct 14 14:12Z failure on auto-encryption-settings) and passed on rerun without any code change.

Everything in the unit-test matrix is green (unit-tests, job-tests, perm-tests, permission-tests, integration-mock-mode-tests, dataset-tests, rds-tests, migration-tests, restrict-tests, enclave-model-api-tests, pre-commit, snyk).

If someone with re-run permissions could retry the two failed jobs (gh run rerun 38190718424 and gh run rerun 38190718453 --failed), I'd expect them to pass like the rest of the matrix.

@choksi2212

Copy link
Copy Markdown
Author

Hi — gentle nudge on this one when you have a moment. Sub-minute job timeouts printed as timed out after 0 minutes at all three runner sites because of integer division; this reports them in seconds while keeping the minutes phrasing for durations of a minute or more (issue #9527). The two red checks are the known runner-side infra flakes (enclave-tests fails during env setup, integration-tests during image pull — triaged in the comments above); everything that touches this change is green. Happy to adjust anything.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant