[Testing]: Acknowledge invocation input without losing pending updates #861

Split from PR #795 — match replay state to the service, reference commit f8ac9b4; baseline main is d7ae95571c44b18c2a2efdef9a308b1c916c5049. This issue owns one independently implementable slice of that PR. Start a focused branch from current main and port only this slice, rather than stacking on the full PR.

Problem and reproduction

The baseline clears updated operation IDs when invocation input is built, before delivery has succeeded. Conversely, a valid input-only resume that returns PENDING without a checkpoint does not advance the handler's seen sequence. Failures can lose updates, and later successful invocations/checkpoints can redeliver updates already consumed. Clearing all IDs also risks dropping changes made after the input snapshot.

Prepare input at sequence S with pending update A, then record newer update B while invocation processing is underway. Exercise (a) a valid PENDING response with no checkpoint, (b) invocation/transport failure, and (c) a checkpoint that advances the watermark beyond S before the invocation finishes. A successful acknowledgment consumes only versions covered by S; failure retains A; B remains pending in all applicable cases.

Scope and implementation

• Stop acknowledging in Executor._begin_invocation. Keep the captured input sequence and acknowledge it after a valid response in _finish_invocation.
• Reuse OperationPaginatorState.advance_handler_seen for versioned removal. Keep the watermark monotonic while removing IDs only when their last-touched version is covered by that acknowledgment.
• Handle legacy stored executions with missing per-operation versions, including equal-sequence acknowledgment, without discarding newer changes.
• Preserve checkpoint acknowledgment behavior and rejected-checkpoint state. Do not introduce another delivery ledger or infer success merely from building input.

Primary code: executor.py (_begin_invocation, _finish_invocation) and execution.py (advance_handler_seen, mark_state_delivered documentation).

Acceptance criteria

• Valid checkpoint-free PENDING and terminal responses acknowledge their input snapshot; failed/invalid delivery does not.
• Newer updates, including another version of the same operation, survive an older input acknowledgment.
• A checkpoint that already advanced the watermark is not undone by invocation completion.
• Equal/older acknowledgments clean only covered IDs and preserve monotonic sequence state, including legacy persisted formats.
• Rejected/stale/token-invalid checkpoints do not consume undelivered updates.
• Public-handler integration covers successful delivery, failure before input reaches the SDK, multiple resumes, both stores, and saved steps executing once.
• Document the acknowledgment point and the distinction between delivery of a snapshot and later concurrent changes.

Reference tests and evidence

• invocation_input_delivery_test.py
• execution_test.py
• executor_checkpoint_test.py
• invocation_input_delivery_test.py

Two test_input_ack_preserves_newer_completion_and_checkpoint_watermark cases failed against the baseline source during #795 validation and pass with the full PR. Other relevant cases are test_checkpoint_consumes_only_updates_covered_by_its_watermark and test_acknowledged_updates_are_pruned_without_watermark_progress.

Independence

Implement directly on main using persisted fixtures with pending IDs or an existing update producer such as wait completion. This establishes the consumer/acknowledgment contract without needing the callback-producer fix first. Adapt the reference callback-heavy E2E test to an already-supported producer for this PR, then add the combined callback scenario when both fixes are present. No replay filtering, flag serialization, OTel export ledger, or empty-error normalization is required. An acknowledgment here concerns service input/checkpoint delivery, not acknowledgment by a telemetry exporter.

Validation and evidence

Add a regression that fails on main without this fix and passes with this slice alone. Run from the repository root:

hatch run dev-testing:test
hatch run dev-testing:typecheck
hatch run dev-testing:js-examples

Use the repository's CI-pinned Hatch version (1.16.5 at the reference) and Node.js 22+ for the JS harness; run hatch fmt --check from packages/aws-durable-execution-sdk-python-testing. Use public-handler integration tests for cross-component behavior and both memory/filesystem stores where relevant. List changed test expectations and their reasons in the implementing PR.

The service-parity expectations are carried forward from #795. This split does not independently verify them against deployed AWS Lambda; cite service evidence or mark the external assumption unverified before claiming deployed-service qualification. Shared conformance work belongs to #796 / conformance PR #131; coordinate new requirements there.

Related issues in this split

• #858 — Replay-state descendant filtering
• #859 — ReplayChildren lifecycle and wire representation
• #860 — Callback updated-operation production
• #862 — Empty-error history and runtime-state consistency
• #863 — Built testing-wheel validation