Skip to content

Isolate inbound payload hydration failures to their deliveries - #829

Merged
wangbill (YunchuWang) merged 11 commits into
mainfrom
yunchuwang-payload-hydration-isolation
Nov 17, 2026
Merged

wangbill (YunchuWang) merged 11 commits into
mainfrom
yunchuwang-payload-hydration-isolation

Conversation

@YunchuWang

@YunchuWang wangbill (YunchuWang) commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Summary

What changed?

  • Handle incoming payload hydration exceptions locally: carry the failed delivery, abandon only its token, and continue reading the same stream without dispatching partial work.
  • Record failed deliveries as real stream activity through the consumer's internal receipt callback. First-message notification still fires once per stream, including when every delivery fails hydration; a later graceful drain is not misclassified as empty. No dummy work item or synthetic HealthPing is emitted.
  • Explicitly give the isolated healthy-work regression two worker activity slots and two in-memory-sidecar dispatcher slots, so the existing scenario runs even on a single-CPU host.
  • Retain the user's core-only, abandon-only scope. No added download/abandonment budgets, special reader protocol, fake-clock dependency, production test hook, terminal 404 settlement, or broad refactoring.

Why is this change needed?

  • One failed download must not disconnect unrelated healthy work. Failed deliveries prove that the stream is active even though they are not yielded for execution; losing that evidence could incorrectly contribute to channel recreation.

Issues / work items


Project checklist

  • Release notes are not required for the next release
    • Otherwise: Notes added to release_notes.md
  • Backport is not required
    • Otherwise: Backport tracked by issue/PR #issue_or_pr
  • All required tests have been added/updated (unit tests, E2E tests)
  • Breaking change?
    • If yes:
      • Impact: Existing supported signatures, constructors, protobuf fields and persisted formats are unchanged. The additive Worker.Grpc.Internal.WorkItemPayloadException is an infrastructure contract, not an application API. The changed stream-factory signature is internal.
      • Migration guidance: N/A; timeout/cancellation policy is unchanged.

AI-assisted code disclosure (required)

Was an AI tool used? (select one)

  • No
  • Yes, AI helped write parts of this PR (e.g., GitHub Copilot)
  • Yes, an AI agent generated most of this PR

If AI was used:

  • Tool(s): GitHub Copilot (GPT-6.1 Sol).
  • AI-assisted areas/files: Delivery error wrapper, raw-read recovery/abandonment, receipt bookkeeping, focused tests and fixture capacity, changelog, and PR description.
  • What you changed after AI output: The agent reduced earlier timeout work to the core abandonment fix and addressed both bachuv review comments in fa5acb8f. Human approval remains required.

AI verification (required if AI was used):

  • I understand the code and can explain it
  • I verified referenced APIs/types exist and are correct
  • I reviewed edge cases/failure paths (timeouts, retries, cancellation, exceptions)
  • I reviewed concurrency/async behavior
  • I checked for unintended breaking or behavior changes

Testing

Automated tests

  • Result: 865 selected tests passed, 0 failures/skips, for fa5acb8f; all four supported product frameworks built successfully, with existing warnings and 0 errors. New-head Validate Build run 43982352555 also passed both build and dts-emulator-tests.
dotnet test test\Worker\Grpc.Tests\Worker.Grpc.Tests.csproj --configuration Release --no-restore --verbosity minimal --nologo

Passed: 275 tests. This includes failed-only graceful drain, failed-then-health-ping, truly empty-drain recovery, once-only first-receipt notification, and no dispatch for non-yielded failed deliveries. Existing per-kind token/abandonment, watchdog, shutdown, and genuine transport/reconnect coverage still passes.

dotnet test test\Extensions\AzureBlobPayloads.Tests\AzureBlobPayloads.Tests.csproj --configuration Release --no-restore --verbosity minimal --nologo

Passed: 397 tests.

$env:DOTNET_PROCESSOR_COUNT = '1'
dotnet test test\Grpc.IntegrationTests\Grpc.IntegrationTests.csproj --configuration Release --no-restore --filter 'FullyQualifiedName~LargePayloadTests|FullyQualifiedName~AzureBlobPayloadsSideCarInterceptorTests' --verbosity minimal --nologo

Passed: 193 tests with one logical processor. All three missing/auth/I/O healthy-work regression cases also passed three further consecutive single-CPU runs with --no-build --no-restore --filter 'FullyQualifiedName~HydrationFailure_KeepsHealthyActivitiesOnSameStreamAsync' (9 repeated executions). The healthy activity remains held until the other delivery fails hydration; no timeout increase or assertion removal.

dotnet build src\Extensions\AzureBlobPayloads\AzureBlobPayloads.csproj --configuration Release --no-restore --verbosity minimal --nologo

Passed: netstandard2.0, net6.0, net8.0, net10.0.

Before the receipt fix, the new three-case processor regression produced 1 failure / 2 passing controls: the failed-only stream invoked the first-message callback 0 times instead of 1. The corrected targeted run passed all 18 selected consumer/processor cases. These use deterministic stream contents rather than wall-clock races.

git --no-pager diff --check and git --no-pager diff --cached --check passed before committing.

Manual validation (only if runtime/behavior changed)

  • Environment (OS, .NET version, components): Windows; .NET 10; actual worker/interceptor, existing isolated loopback HTTP/2 fixture, and mocked payload storage.
  • Steps + observed results:
    1. Failed deliveries are observed before abandonment and never dispatched; once-only receipt notification and nonempty-drain classification are preserved.
    2. The test worker explicitly advertises two activity slots. Its test-only in-memory service also exposes two slots because the sidecar dispatcher independently reads IOrchestrationService.MaxConcurrentTaskActivityWorkItems, otherwise based on processor count. The fixture's default behavior and production service are unchanged.
    3. No live-Azure fault injection or full-solution run was performed locally for this follow-up. The new-head standard build/test job and emulator job both passed in CI.
  • Evidence (optional): Logs and TRX results are retained in the coding session. Previous-head run 43482820750 at 93426cc3 failed only the unchanged OrchestrationPatterns.LongTimer assertion (expected 3, actual 2); restore/compile passed. That test was not modified or its failure hidden by this follow-up; the new-head run above passed.

Notes for reviewers

  • Core-only / abandon-only: payload exceptions do not terminally fail an orchestration. NotFound on abandon remains item-local; other errors use the existing retry/connection recovery path.
  • Failed receipts now use the same receipt bookkeeping as yielded work; timer policy is not redesigned. The existing watchdog still measures MoveNext, including hydration and awaited abandonment. The 119-second receive plus 2-second download interaction is not claimed fixed.
  • No public/protected API change, new package, fake clock, special reader interface, production test hook, timeout budget, token snapshot, or standalone test server is added by these review fixes.
  • Ordinary reader wrappers, existing interceptor subclasses, management/unary/history behavior, upload logic and background-task failure handling remain unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 30, 2026 19:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block the change to isolate payload hydration failures to individual deliveries.

No open issues 🎉

What changed in this PR

This PR addresses #825 by isolating inbound payload hydration failures to individual deliveries, preserving the Durable Task worker’s shared stream and unrelated in-flight work.

Changes:

  • Introduces an internal exception carrying the failed delivery and its completion token.
  • Reads work items directly and reuses abandonment logic, tolerating ownership loss while preserving connection-level recovery for other failures.
  • Adds consumer and HTTP/2 loopback regression coverage plus an Unreleased changelog entry.
File Description
test/​Worker/​Grpc.Tests/​WorkItemStreamConsumerTests.cs Tests delivery recovery, callback ordering, and failure propagation.
test/​Grpc.IntegrationTests/​PayloadHydrationFailureTests.cs Adds loopback coverage for isolation, redelivery, reconnection, and shutdown.
src/​Worker/​Grpc/​WorkItemStreamConsumer.cs Uses direct stream reads with delivery-local recovery.
src/​Worker/​Grpc/​Worker.Grpc.csproj Grants integration tests access to worker internals.
src/​Worker/​Grpc/​Logs.cs Adds ownership-loss logging.
src/​Worker/​Grpc/​GrpcDurableTaskWorker.Processor.cs Handles hydration failures using shared abandonment logic.
src/​Grpc/​WorkItemPayloadException.cs Defines the internal failed-delivery exception.
src/​Grpc/​Grpc.csproj Exposes the internal contract to consuming assemblies and tests.
src/​Extensions/​AzureBlobPayloads/​Interceptors/​PayloadInterceptor.cs Wraps work-item hydration failures with delivery context.
CHANGELOG.md Documents delivery-local hydration failure handling.

Review effort: Balanced

Remove redundant consumer tests and exercise the shared failure-isolation matrix with both gRPC and Azure Managed worker configuration. Preserve retryable inbound delivery semantics and document the recovery boundary.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 10, 2026 22:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block the change to isolate payload hydration failures to individual deliveries.

No open issues 🎉

Review effort: Balanced

…hy work

Replace the standalone test server with focused coverage in existing harnesses. Preserve ordinary background error handling, validate execution metadata before terminal settlement, and release entity locks for inline and streamed history. Recoverable download failures remain delivery-local and retryable.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 10, 2026 23:33
@YunchuWang wangbill (YunchuWang) changed the title Isolate inbound payload hydration failures to their deliveries Isolate inbound payload failures and fail missing-payload deliveries Nov 10, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved recovery and replay defects can leave deliveries owned until expiry, disconnect healthy streams, and terminally fail callers before they can handle missing-input failures.

3 open issues

Review effort: Balanced

…ng healthy work"

This reverts commit a0f5320 at the user request. Restore retryable delivery-local abandonment for all incoming hydration failures.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 10, 2026 23:45
@YunchuWang wangbill (YunchuWang) changed the title Isolate inbound payload failures and fail missing-payload deliveries Isolate inbound payload hydration failures to their deliveries Nov 10, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified for the change isolating payload hydration failures to individual deliveries.

2 open issues
1 resolved since last review

Review effort: Balanced

Remove the standalone 536-line scripted sidecar and reuse existing worker and integration test harnesses. Leave ordinary background failure handling unchanged, remove unnecessary test friend access and duplicate documentation, and retain delivery-local abandonment for every hydration failure.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 10, 2026 23:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Missing regression tests for cancellation during hydration and pending abandonment leave the required unchanged shutdown behavior unverified.

3 open issues

Review effort: Balanced

Use worker shutdown cancellation during delivery recovery and rearm the read watchdog before receiving again. Add gated shutdown and slow-abandon regressions; fix the scoped-payload assertion and prevent test-sidecar concurrency starvation on small CI hosts. Keep all hydration failures abandon-only.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 11, 2026 15:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the changes isolating inbound hydration failures to individual deliveries.

No open issues 🎉

3 resolved since last review

Review effort: Balanced

Watch only transport receipt for silent disconnects. Give each active worker payload download its own 120-second cancellation budget while preserving bounded concurrency and drainage. Bound receive-loop abandonment and its existing retries/backoff together to 30 seconds. Keep downloads abandon-only and preserve public unary/history behavior.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 11, 2026 16:19
Comment thread src/Grpc/WorkItemPayloadException.cs Resolved
Comment thread src/Grpc/Grpc.csproj Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

5 open issues

Review effort: Lite

Comment thread src/Grpc/Grpc.csproj Resolved
Comment thread src/Grpc/WorkItemPayloadException.cs Resolved
Comment thread src/Worker/Grpc/Logs.cs Resolved
Move worker-extension infrastructure out of the protobuf assembly and remove its friend access. Preserve protected interceptor overrides and async behavior, simplify call disposal and common-token allocation, and remove timeout test callbacks. Use standard clock scheduling for deterministic deadline, cancellation, and external-subclass regressions; successful transport reads no longer poison later reads with a fired watchdog.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 11, 2026 17:27

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because interceptor wrappers can expose healthy payload downloads to watchdog cancellation, while abandon deadlines can be misreported as handshake failures and incorrectly mark channels as poisoned.

No open issues 🎉

5 resolved since last review

Review effort: Balanced

Use Microsoft.Extensions.TimeProvider.Testing 10.0.0 for the net10 test projects. Its native net10 dependency group does not request the uncached Microsoft.Bcl.TimeProvider 8.0.0 package. Verified both test projects restore from an empty package cache using only the checked-in nuget.config.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 11, 2026 17:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the changes isolating inbound payload hydration failures to individual deliveries.

No open issues 🎉

Review effort: Balanced

Remove the added transport/hydration interface, download and abandonment deadlines, TimeProvider plumbing and timeout-only tests. Preserve the upstream watchdog and cancellation policy unchanged. Catch hydration errors within a raw-read iterator, abandon only the failed token, tolerate ownership loss, and continue the same stream. Include current main and resolve its changelog conflict.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 13, 2026 18:27

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical watchdog misuse and unbounded abandonment calls can trigger unnecessary shared-stream reconnects or block subsequent deliveries until shutdown, disrupting unrelated work.

2 open issues

Review effort: Balanced

Record real failed-delivery receipt without yielding partially hydrated work. Share first-message bookkeeping with healthy deliveries so active streams do not receive empty-drain channel-recreation credit. Give the isolated healthy-work regression two explicit worker/sidecar activity slots so it also runs on one CPU. Preserve abandon-only handling and the existing timeout policy.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 23:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human validation is needed because the loopback sidecar’s abandon RPCs are no-ops, so the tests establish stream isolation but not ownership release or retryability on the intended scheduler.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

@YunchuWang
wangbill (YunchuWang) merged commit 87f32c8 into main Nov 17, 2026
11 checks passed
@YunchuWang
wangbill (YunchuWang) deleted the yunchuwang-payload-hydration-isolation branch November 17, 2026 00:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Payload hydration failure for one work item tears down the shared GetWorkItems stream

3 participants