[Epic]: Remaining Python OpenTelemetry work #873

Parent tracker for the remaining Python OpenTelemetry fixes, core/testing prerequisites, and conformance qualification. Deliver the work through small, independently reviewable PRs; each issue owns its reproduction, design choices, acceptance criteria, and tests.

Status reviewed on 2026-11-02. Of the 13 native sub-issues, 5 are closed and 8 remain open. The original split from closed PRs #758 and #795 remains below; newer OTel follow-ups are listed separately by reference.

Latest progress

• Merged fixes: terminal status/event timing (fix: correct terminal telemetry status and timing #865 → [OTel]: Report terminal failures and keep exception events within span bounds #853), isolated terminal error snapshots (fix: isolate terminal plugin error snapshots #866 → [Bug]: Isolate terminal plugin error snapshots from checkpointed state #852), empty-error preservation (fix: preserve empty errors in state and history #870 → [Testing]: Preserve empty callback error history without changing runtime errors #862), and callback update/input acknowledgment semantics (fix(testing): derive updated operation IDs #898 → [Testing]: Record callback terminal completions as updated operations #860 and [Testing]: Acknowledge invocation input without losing pending updates #861).
• test(otel): cover replay and chained execution #796 merged on November 2, delivering conformance cases 22–27, including the previously deferred cases 25–26. Its latest four-job cloud run passed 26 cases per job, with case 21 explicitly NOT_IMPLEMENTED. The additional case 17 callback-error regression is still outstanding.
• [Bug]: OTel adds missing type/message to partial callback errors #894's partial callback-error type/message fix also merged in fix(otel): omit absent fields from error exception events #901. Stacktrace emission remains separate work in [Bug]: Align OTel exception stacktrace emission with OpenTelemetry JS #886.
• Replay-completion discovery ([OTel]: Export unvisited replay completions with the correct parent context #854, open PR fix: export visible replay completions #869), export recovery ([OTel]: Recover failed completion exports and isolate invocation finalization errors #855), bounded flushing ([OTel]: Bound provider flush waiting without blocking unrelated providers #856), the remaining core/testing items, and the newer follow-ups below remain open. Passing the current conformance catalog does not close those broader gaps.

OTel runtime correctness — original sub-issues

• #853 — Report terminal failures and keep exception events within span bounds — merged fix: correct terminal telemetry status and timing #865.
• #854 — Export unvisited replay completions with the correct parent context — open PR fix: export visible replay completions #869; broader skipped-parent/unvisited-completion coverage remains outstanding despite case 26 passing.
• #855 — Recover failed completion exports and isolate invocation finalization errors.
• #856 — Bound provider flush waiting without blocking unrelated providers.

Core instrumentation contract — original sub-issues

• #852 — Isolate terminal plugin error snapshots from checkpointed state — merged fix: isolate terminal plugin error snapshots #866.
• #772 — Preserve handler errors in invocation-end hooks when returning PENDING.

Local-runner parity and installed artifacts — original sub-issues

• #858 — Prune completed-context descendants from replay state.
• #859 — Match ReplayChildren lifecycle and wire representation.
• #860 — Record callback terminal completions as updated operations — merged fix(testing): derive updated operation IDs #898.
• #861 — Acknowledge invocation input without losing pending updates — merged fix(testing): derive updated operation IDs #898.
• #862 — Preserve empty callback error history without changing runtime errors — merged fix: preserve empty errors in state and history #870.
• #863 — Validate the built testing-library wheel in OTel compatibility lanes.

#898 derives updated-operation IDs from the runner's existing touch sequences and the last successful invocation watermark. It also makes the duplicate terminal-STEP export reported in #892 reproducible locally; that SDK/OTel defect remains open. The earlier #867/#868 approaches are superseded by #898 and are not additional prerequisites.

Conformance and integration

• #874 — Complete replay and chained-execution conformance coverage — remains open for case 21, the additional case 17 regression, and completion of the remaining qualification work.

Verified cloud evidence: run 41499388644 tested SDK revision de94078 against shared catalog 0eaf734. Invocation and Execution views each passed on Community/S3 and ADOT/X-Ray: 26 passed, 0 failed, 0 optional failed, 1 not implemented, 0 uncovered, 27 total per job. This qualifies the active catalog at those revisions, not case 21 or every open runtime issue. #796 also records 94 passing local conformance/workflow tests.

Remaining conformance work and constraints:

• Resolve the case 21 shared history-order requirement before re-enabling cloud coverage. The recorded failing run observed WaitSucceeded before InvocationCompleted on a successful execution. Preserve saved-step, result, and telemetry assertions when accommodating valid ordering; increasing the wait only reduces the race.
• Preserve case 26's observation that completion has reached the exporter before the target step runs, and its raw export-count assertions at later barriers. test(otel): cover replay and chained execution #796 records observations inside callbacks and asserts on the pytest thread so SDK retries cannot consume assertion failures.
• Detailed case 25 histories in run 40769339007, documented in test(otel): cover replay and chained execution #796, verified Error: {Payload: {}, Truncated: false} against Lambda. The metadata-only Error: {Truncated: true} shape remains unverified against deployed Lambda. Included diagnostic payloads are untruncated; metadata-only history cannot reconstruct omitted error details.
• Shared requirements/orchestration from aws/aws-durable-execution-conformance-tests#131 are merged. Coordinate remaining language-neutral changes in that repository. test(otel): cover replay and chained execution #796 pins the reusable workflow to 0eaf7341c0c7ff937e6a4bf6bda1682ff07e7502; the catalog defaults to shared main, with the manual conformance_test_ref override retained.

Newer OTel and instrumentation follow-ups

These are tracked here by reference and are not included in the 13 native sub-issue count. Broader core/plugin or cross-SDK changes retain the scope and compatibility requirements in their own issues.

Related work with broader ownership

• [Feature]: Add Python LMI cloud tests for lifecycle, concurrency and OpenTelemetry #742 owns deployed Python LMI observability/X-Ray coverage alongside lifecycle tests. It already has Java #798 — Durable Functions on LMI as its parent; retain that hierarchy and track the Python OTel acceptance here by reference.
• Improve the monorepo release process #716 and [Bug]: Fix release publish workflow #487 track cross-package release/publishing behavior, including the OTel package. Their broader release scope remains separate; link any concrete OTel release blocker discovered while delivering this work.
• Plugin conformance tests are flaky when CloudWatch logs are not yet visible #708 tracks log-visibility flakiness in the general plugin conformance suite. It is related test-infrastructure work, not evidence that an OTel runtime fix has passed or failed.

Working independently

Start each implementation from current main, use the issue's focused regression, and keep unrelated runtime/testing changes in their own PRs. Coordinate overlapping files without recreating the closed implementation stack. Test the combined fixes through the public SDK and real exporters after they land. A closed issue must link the merged fix or an explicit resolution; do not close this parent merely because a tracking PR exists or the currently enabled catalog passes.

Completion criteria

• All native sub-issues have merged fixes or an explicitly documented disposition; newer follow-ups have a clear resolution or ownership/scope disposition.
• Replay preserves saved successes/failures, operation identity, and completed-work skipping; completion telemetry has the correct parent, status, event bounds, and export phase in both views, without reconstructing already-terminal STEP work.
• Export-failure cleanup and provider-flush behavior satisfy the bounds and ownership rules in their issues without claiming exactly-once delivery.
• Runner parity is checked across memory/filesystem stores, replay/state reads, callback updates, and history serialization; installed-wheel lanes exercise the built testing library.
• Cases 22–27, including the previously deferred cases 25–26, are delivered and qualified in the four-job cloud matrix above.
• Cloud case 21 and the additional case 17 callback-error regression are restored and qualified, or explicitly dispositioned with evidence; missing coverage is never counted as passing.
• Python LMI OTel acceptance is linked from [Feature]: Add Python LMI cloud tests for lifecycle, concurrency and OpenTelemetry #742, with deployed evidence appropriate to Python's process model.
• Documentation and validation results reflect the final behavior, and external assumptions have evidence or are marked unverified. The metadata-only callback-error service shape remains unverified; detailed empty-error history now has cloud evidence.