Repository navigation
Isolate inbound payload hydration failures to their deliveries - #829
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues block the change to isolate payload hydration failures to individual deliveries.
No open issues 🎉
What changed in this PR
This PR addresses #825 by isolating inbound payload hydration failures to individual deliveries, preserving the Durable Task worker’s shared stream and unrelated in-flight work.
Changes:
- Introduces an internal exception carrying the failed delivery and its completion token.
- Reads work items directly and reuses abandonment logic, tolerating ownership loss while preserving connection-level recovery for other failures.
- Adds consumer and HTTP/2 loopback regression coverage plus an Unreleased changelog entry.
| File | Description |
|---|---|
test/Worker/Grpc.Tests/WorkItemStreamConsumerTests.cs |
Tests delivery recovery, callback ordering, and failure propagation. |
test/Grpc.IntegrationTests/PayloadHydrationFailureTests.cs |
Adds loopback coverage for isolation, redelivery, reconnection, and shutdown. |
src/Worker/Grpc/WorkItemStreamConsumer.cs |
Uses direct stream reads with delivery-local recovery. |
src/Worker/Grpc/Worker.Grpc.csproj |
Grants integration tests access to worker internals. |
src/Worker/Grpc/Logs.cs |
Adds ownership-loss logging. |
src/Worker/Grpc/GrpcDurableTaskWorker.Processor.cs |
Handles hydration failures using shared abandonment logic. |
src/Grpc/WorkItemPayloadException.cs |
Defines the internal failed-delivery exception. |
src/Grpc/Grpc.csproj |
Exposes the internal contract to consuming assemblies and tests. |
src/Extensions/AzureBlobPayloads/Interceptors/PayloadInterceptor.cs |
Wraps work-item hydration failures with delivery context. |
CHANGELOG.md |
Documents delivery-local hydration failure handling. |
Review effort: Balanced
Remove redundant consumer tests and exercise the shared failure-isolation matrix with both gRPC and Azure Managed worker configuration. Preserve retryable inbound delivery semantics and document the recovery boundary. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…hy work Replace the standalone test server with focused coverage in existing harnesses. Preserve ordinary background error handling, validate execution metadata before terminal settlement, and release entity locks for inline and streamed history. Recoverable download failures remain delivery-local and retryable. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ng healthy work" This reverts commit a0f5320 at the user request. Restore retryable delivery-local abandonment for all incoming hydration failures. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Remove the standalone 536-line scripted sidecar and reuse existing worker and integration test harnesses. Leave ordinary background failure handling unchanged, remove unnecessary test friend access and duplicate documentation, and retain delivery-local abandonment for every hydration failure. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use worker shutdown cancellation during delivery recovery and rearm the read watchdog before receiving again. Add gated shutdown and slow-abandon regressions; fix the scoped-payload assertion and prevent test-sidecar concurrency starvation on small CI hosts. Keep all hydration failures abandon-only. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Watch only transport receipt for silent disconnects. Give each active worker payload download its own 120-second cancellation budget while preserving bounded concurrency and drainage. Bound receive-loop abandonment and its existing retries/backoff together to 30 seconds. Keep downloads abandon-only and preserve public unary/history behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
5 open issues
UsingKey=\"$(StrongNamePublicKey)\"is fragile if$(StrongNamePublicKey)can be empty in any… · New This assertion is now overly broad (ThrowAsync<Exception>), which can mask unrelated… · New This allocates a new linkedCancellationTokenSourcefor every hydrated message. Sincelifetime… · New Thisusingis unused in the file. Removing it avoids unnecessary imports and potential analyzer… · New The structured log placeholder names{workItemType}/{completionToken}are lower-cased, which… · New
Review effort: Lite
Move worker-extension infrastructure out of the protobuf assembly and remove its friend access. Preserve protected interceptor overrides and async behavior, simplify call disposal and common-token allocation, and remove timeout test callbacks. Use standard clock scheduling for deterministic deadline, cancellation, and external-subclass regressions; successful transport reads no longer poison later reads with a fired watchdog. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Human assessment is needed because interceptor wrappers can expose healthy payload downloads to watchdog cancellation, while abandon deadlines can be misreported as handshake failures and incorrectly mark channels as poisoned.
No open issues 🎉
5 resolved since last review
This assertion is now overly broad (ThrowAsync<Exception>), which can mask unrelated… UsingKey=\"$(StrongNamePublicKey)\"is fragile if$(StrongNamePublicKey)can be empty in any… The structured log placeholder names{workItemType}/{completionToken}are lower-cased, which… Thisusingis unused in the file. Removing it avoids unnecessary imports and potential analyzer… This allocates a new linkedCancellationTokenSourcefor every hydrated message. Sincelifetime…
Review effort: Balanced
Use Microsoft.Extensions.TimeProvider.Testing 10.0.0 for the net10 test projects. Its native net10 dependency group does not request the uncached Microsoft.Bcl.TimeProvider 8.0.0 package. Verified both test projects restore from an empty package cache using only the checked-in nuget.config. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Remove the added transport/hydration interface, download and abandonment deadlines, TimeProvider plumbing and timeout-only tests. Preserve the upstream watchdog and cancellation policy unchanged. Catch hydration errors within a raw-read iterator, abandon only the failed token, tolerate ownership loss, and continue the same stream. Include current main and resolve its changelog conflict. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Record real failed-delivery receipt without yielding partially hydrated work. Share first-message bookkeeping with healthy deliveries so active streams do not receive empty-drain channel-recreation credit. Give the isolated healthy-work regression two explicit worker/sidecar activity slots so it also runs on one CPU. Preserve abandon-only handling and the existing timeout policy. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Human validation is needed because the loopback sidecar’s abandon RPCs are no-ops, so the tests establish stream isolation but not ownership release or retryability on the intended scheduler.
No open issues 🎉
2 resolved since last review
Review effort: Balanced



Summary
What changed?
Why is this change needed?
Issues / work items
Project checklist
release_notes.mdWorker.Grpc.Internal.WorkItemPayloadExceptionis an infrastructure contract, not an application API. The changed stream-factory signature is internal.AI-assisted code disclosure (required)
Was an AI tool used? (select one)
If AI was used:
fa5acb8f. Human approval remains required.AI verification (required if AI was used):
Testing
Automated tests
fa5acb8f; all four supported product frameworks built successfully, with existing warnings and 0 errors. New-head Validate Build run 43982352555 also passed bothbuildanddts-emulator-tests.Passed: 275 tests. This includes failed-only graceful drain, failed-then-health-ping, truly empty-drain recovery, once-only first-receipt notification, and no dispatch for non-yielded failed deliveries. Existing per-kind token/abandonment, watchdog, shutdown, and genuine transport/reconnect coverage still passes.
Passed: 397 tests.
Passed: 193 tests with one logical processor. All three missing/auth/I/O healthy-work regression cases also passed three further consecutive single-CPU runs with
--no-build --no-restore --filter 'FullyQualifiedName~HydrationFailure_KeepsHealthyActivitiesOnSameStreamAsync'(9 repeated executions). The healthy activity remains held until the other delivery fails hydration; no timeout increase or assertion removal.Passed: netstandard2.0, net6.0, net8.0, net10.0.
Before the receipt fix, the new three-case processor regression produced 1 failure / 2 passing controls: the failed-only stream invoked the first-message callback 0 times instead of 1. The corrected targeted run passed all 18 selected consumer/processor cases. These use deterministic stream contents rather than wall-clock races.
git --no-pager diff --checkandgit --no-pager diff --cached --checkpassed before committing.Manual validation (only if runtime/behavior changed)
IOrchestrationService.MaxConcurrentTaskActivityWorkItems, otherwise based on processor count. The fixture's default behavior and production service are unchanged.43482820750at93426cc3failed only the unchangedOrchestrationPatterns.LongTimerassertion (expected 3, actual 2); restore/compile passed. That test was not modified or its failure hidden by this follow-up; the new-head run above passed.Notes for reviewers
NotFoundon abandon remains item-local; other errors use the existing retry/connection recovery path.MoveNext, including hydration and awaited abandonment. The 119-second receive plus 2-second download interaction is not claimed fixed.