Skip to content

fix(backtests): print the deadline-hold notice on a late submit too - #650

Merged
FlyM1ss merged 2 commits into
Open-Finance-Lab:mainfrom
subidhkhanal:fix/submit-deadline-hold-notice
Oct 22, 2026
Merged

FlyM1ss merged 2 commits into
Open-Finance-Lab:mainfrom
subidhkhanal:fix/submit-deadline-hold-notice

Conversation

@subidhkhanal

@subidhkhanal subidhkhanal commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #375

Problem. A decision step auto-holds (no trade, the agent's decision skipped) at one of four places when its deadline has passed. #374 added a printed warning at three of them — get_current_step, drain_expired, get_status — all of which go through the shared _maybe_apply_timeout. submit_decisions had its own, independent deadline check that bypassed that helper entirely, so a hold created by a late POST printed nothing. Two real routes reach this: the legacy /api/v1/backtest/* surface (Discord bot, built-in agents) and the canonical /api/v2 contract. Both are ordinary GET-then-POST flows where the deadline can elapse in the gap, which is exactly the case this feature exists to surface. The periodic reaper doesn't catch it either, since submit_decisions has already advanced the step by the time it runs.

Fix. submit_decisions's late-submit branch now goes through the same shared _maybe_apply_timeout the other three sites use, instead of its own inline check, and prints the warning via the existing _emit_deadline_hold_notice helper on a hold. Routing through the shared function means there is only one place in the code that decides "this step is late" — a future change to that rule can't miss this spot the way #374 did.

Tests. Two new cases in test_deadline_hold_visibility.py:

  • a late submit prints exactly one warning line, naming the held step;
  • an on-time submit prints nothing.

I verified the new late-submit test actually catches the bug: run against the pre-fix code it fails (0 warnings printed instead of 1).

Verification run locally:

pytest dashboard/backend/tests/test_deadline_and_holds.py dashboard/backend/tests/test_deadline_hold_visibility.py dashboard/backend/tests/test_external_backtest_api.py dashboard/backend/tests/test_external_minute_source.py dashboard/backend/tests/test_external_session_compat.py dashboard/backend/tests/test_protocol_api.py dashboard/backend/tests/test_v2_runs.py -q

84 passed. Two pre-existing failures in that run also fail on unmodified main on Windows and are unrelated (Windows clock-resolution flakiness in deadline-timing tests — real-clock comparisons at a 0-second deadline can land exactly on the boundary; both pass reliably on Linux/CI).

Review follow-up (dd54d9f). Both items from the scope note are fixed here, plus three related ones from review:

  • One helper, _apply_expired_holds, now handles hold-and-print for all four sites. It is the only caller of _maybe_apply_timeout.
  • A late hold of the final step returns run_id/metrics/compare_url. On v2 it also marks the protocol_runs row completed right away.
  • A decision built for a step that already closed is refused (409 step_already_closed) instead of being applied to the next step. That covers the reaper or a status poll holding the step first, and a hold landing between v2's step read and apply. Mechanism: an expected_step pin plus served-step inference.
  • v2 audits idempotency and trace records under the step that actually executed or was refused.
  • The accepted ack reports decision_source again, instead of None.

🤖 Generated with Claude Code

@vercel

vercel Bot commented Oct 18, 2026

Copy link
Copy Markdown
Contributor

@subidhkhanal is attempting to deploy a commit to the allan-feng's projects Team on Vercel.

A member of the Team first needs to authorize it.

Review follow-up on the late-submit hold notice.

- submit_decisions refuses a decision built for a step that has already
  closed instead of applying it to the next step. It catches this in two
  ways: an explicit expected_step (v2 _submit_for, v1 protocol submit),
  and an inference from the last step served whose step was auto-held
  (the legacy route, and a v2 POST arriving after the reaper's hold).
- All four hold sites go through _apply_expired_holds, the only caller of
  _maybe_apply_timeout, so no hold can be silent and the count is no
  longer hard-coded.
- A late hold of the final step returns run_id/metrics/compare_url, and v2
  marks protocol_runs completed and closes the trace right away.
- v2 audits the idempotency row and trace under the step that actually
  executed or was refused.
- The accepted ack reports decision_source again: _open_current_step no
  longer resets it.
- Tests: submit and route-level coverage for each case. The load-test
  README and test docstring no longer describe Open-Finance-Lab#375 as open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@FlyM1ss
FlyM1ss merged commit b84bf9e into Open-Finance-Lab:main Oct 22, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deadline auto-holds in submit_decisions() are still silent

2 participants