Skip to content

DurableTask: New beta extension version for 2026-07-01-preview features - #10482

Open
Bernd Verst (berndverst) wants to merge 11 commits into
Azure:mainfrom
berndverst:main
Open

Bernd Verst (berndverst) wants to merge 11 commits into
Azure:mainfrom
berndverst:main

Conversation

@berndverst

@berndverst Bernd Verst (berndverst) commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

🤖 PR Validation — ⚠️ Review suggested

Breaking Changes Suppressions
⚠️ None ️✔️ None
⚠️Azure CLI Extensions Breaking Change Test
⚠️durabletask
rule cmd_name rule_message suggest_message
⚠️ 1011 - SubgroupAdd durabletask scheduler failover-instance sub group durabletask scheduler failover-instance added
⚠️ 1008 - ParaPropAdd durabletask scheduler restart cmd durabletask scheduler restart update parameter resource_group_name: added property aaz_type=string
⚠️ 1008 - ParaPropAdd durabletask scheduler restart cmd durabletask scheduler restart update parameter resource_group_name: added property type=string
⚠️ 1010 - ParaPropUpdate durabletask scheduler restart cmd durabletask scheduler restart update parameter resource_group_name: updated property name from resource_group_name to resource_group
⚠️ 1008 - ParaPropAdd durabletask scheduler restart cmd durabletask scheduler restart update parameter scheduler_name: added property aaz_type=string
⚠️ 1008 - ParaPropAdd durabletask scheduler restart cmd durabletask scheduler restart update parameter scheduler_name: added property id_part=name
⚠️ 1008 - ParaPropAdd durabletask scheduler restart cmd durabletask scheduler restart update parameter scheduler_name: added property type=string
⚠️ 1010 - ParaPropUpdate durabletask scheduler restart cmd durabletask scheduler restart update parameter scheduler_name: updated property options from ['--name', '-n'] to ['--name', '--scheduler-name', '-n']
⚠️ 1006 - ParaAdd durabletask scheduler transparent-data-encryption create cmd durabletask scheduler transparent-data-encryption create added parameter auto_rotation_enabled
⚠️ 1006 - ParaAdd durabletask scheduler transparent-data-encryption update cmd durabletask scheduler transparent-data-encryption update added parameter auto_rotation_enabled
⚠️ 1011 - SubgroupAdd durabletask taskqueue sub group durabletask taskqueue added

This checklist is used to make sure that common guidelines for a pull request are followed.

Related command

General Guidelines

  • Have you run azdev style <YOUR_EXT> locally? (pip install azdev required)
  • Have you run python scripts/ci/test_index.py -q locally? (pip install azdev required)
  • My extension version conforms to the Extension version schema

For new extensions:

About Extension Publish

There is a pipeline to automatically build, upload and publish extension wheels.
Once your pull request is merged into main branch, a new pull request will be created to update src/index.json automatically.
You only need to update the version information in file setup.py and historical information in file HISTORY.rst in your PR but do not modify src/index.json.

Copilot AI balanced review requested due to automatic review settings October 16, 2026 21:51
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Restart registration still overrides the new handler, and encryption creation loses a compatibility-preserving default.

3 open issues
What changed in this PR

Extends DurableTask CLI support for 2026-07-01-preview, adding task queues, scheduler failover operations, and encryption key auto-rotation.

Changes:

  • Adds taskqueue and failover-instance command groups.
  • Updates taskhub and encryption APIs, including auto-rotation support.
  • Adds a generated scheduler restart handler and documents taskhub selection.

Paths below are relative to src/durabletask/.

File Description
README.md Documents taskqueue taskhub selection.
azext_durabletask/​aaz/​latest/​durabletask/​taskqueue/​_wait.py Adds queue condition polling.
azext_durabletask/​aaz/​latest/​durabletask/​taskqueue/​_update.py Adds queue throughput updates.
azext_durabletask/​aaz/​latest/​durabletask/​taskqueue/​_show.py Adds queue retrieval.
azext_durabletask/​aaz/​latest/​durabletask/​taskqueue/​_list.py Adds paginated queue listing.
azext_durabletask/​aaz/​latest/​durabletask/​taskqueue/​_delete.py Adds queue deletion.
azext_durabletask/​aaz/​latest/​durabletask/​taskqueue/​_create.py Adds queue creation.
azext_durabletask/​aaz/​latest/​durabletask/​taskqueue/​__init__.py Imports queue commands.
azext_durabletask/​aaz/​latest/​durabletask/​taskqueue/​__cmd_group.py Registers the queue command group.
azext_durabletask/​aaz/​latest/​durabletask/​taskhub/​_wait.py Updates polling API version.
azext_durabletask/​aaz/​latest/​durabletask/​taskhub/​_show.py Updates retrieval API version.
azext_durabletask/​aaz/​latest/​durabletask/​taskhub/​_list.py Updates listing API version.
azext_durabletask/​aaz/​latest/​durabletask/​taskhub/​_delete.py Updates deletion API version.
azext_durabletask/​aaz/​latest/​durabletask/​taskhub/​_create.py Updates creation API version.
azext_durabletask/​aaz/​latest/​durabletask/​scheduler/​transparent_data_encryption/​_wait.py Updates API and reads auto-rotation state.
azext_durabletask/​aaz/​latest/​durabletask/​scheduler/​transparent_data_encryption/​_update.py Adds auto-rotation updates.
azext_durabletask/​aaz/​latest/​durabletask/​scheduler/​transparent_data_encryption/​_show.py Exposes auto-rotation state.
azext_durabletask/​aaz/​latest/​durabletask/​scheduler/​transparent_data_encryption/​_delete.py Updates API and removes preview marking.
azext_durabletask/​aaz/​latest/​durabletask/​scheduler/​transparent_data_encryption/​_create.py Adds auto-rotation; removes key-source default.
azext_durabletask/​aaz/​latest/​durabletask/​scheduler/​transparent_data_encryption/​__cmd_group.py Removes group preview marking.
azext_durabletask/​aaz/​latest/​durabletask/​scheduler/​failover_instance/​_wait.py Adds failover-instance polling.
azext_durabletask/​aaz/​latest/​durabletask/​scheduler/​failover_instance/​_update.py Adds partner scheduler updates.
azext_durabletask/​aaz/​latest/​durabletask/​scheduler/​failover_instance/​_show.py Adds failover-instance retrieval.
azext_durabletask/​aaz/​latest/​durabletask/​scheduler/​failover_instance/​_force_failover_allow_data_loss.py Adds forced failover.
azext_durabletask/​aaz/​latest/​durabletask/​scheduler/​failover_instance/​_failover.py Adds graceful failover.
azext_durabletask/​aaz/​latest/​durabletask/​scheduler/​failover_instance/​_delete.py Adds failover-instance deletion.
azext_durabletask/​aaz/​latest/​durabletask/​scheduler/​failover_instance/​_create.py Adds failover-instance creation.
azext_durabletask/​aaz/​latest/​durabletask/​scheduler/​failover_instance/​__init__.py Imports failover commands.
azext_durabletask/​aaz/​latest/​durabletask/​scheduler/​failover_instance/​__cmd_group.py Registers the failover command group.
azext_durabletask/​aaz/​latest/​durabletask/​scheduler/​_update.py Removes help examples only.
azext_durabletask/​aaz/​latest/​durabletask/​scheduler/​_restart.py Adds a generated restart handler.
azext_durabletask/​aaz/​latest/​durabletask/​scheduler/​__init__.py Imports the restart handler.

Review effort: Balanced

@a0x1ab

Copy link
Copy Markdown
Member

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 2 pipeline(s).

@a0x1ab

Copy link
Copy Markdown
Member

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 2 pipeline(s).

@a0x1ab

Copy link
Copy Markdown
Member

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 2 pipeline(s).

Bernd Verst (berndverst) pushed a commit to berndverst/aaz that referenced this pull request Oct 21, 2026
Sync missing task queue, scheduler failover, restart, and encryption key rotation examples from Azure/azure-cli-extensions#10482.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 1fa7d6ba-e6f1-457a-8829-9834cec6a58e
@JaysonTaiMicrosoft JaysonTaiMicrosoft added the Request X Engineering Agent Ask the agent to triage this issue or review this PR label Oct 22, 2026
@x-engineering-agent

x-engineering-agent Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

X Engineering Agent Pull request review: Comments

1 high findings 1 medium findings

Bernd Verst (@berndverst) Generated-command provenance remains blocking, and failover outcome assertions need strengthening. The new request tests and seven playback recordings provide substantive evidence for the other command paths.

High Generated code ownership checker

src/durabletask/azext_durabletask/aaz/latest/durabletask/scheduler/__init__.py:15, src/durabletask/azext_durabletask/aaz/latest/durabletask/scheduler/_restart.py:1, src/durabletask/azext_durabletask/aaz/latest/durabletask/scheduler/_update.py, src/durabletask/azext_durabletask/aaz/latest/durabletask/scheduler/failover_instance/__cmd_group.py:1, src/durabletask/azext_durabletask/aaz/latest/durabletask/scheduler/failover_instance/__init__.py:1 and 26 more location(s): A generated file changed without an open or merged Azure/aaz pull request containing the durable command-model change.

Remediation: Make the durable command-model change in Azure/aaz, link that PR in this description, then regenerate the downstream aaz/<profile> output.

Verify: Regenerate from a clean checkout and confirm the resulting diff contains this change and the linked source PR is open or merged.

Medium Failover outcomes are not asserted

src/durabletask/azext_durabletask/tests/latest/test_durabletask_scenario.py:361-373 invokes both failover actions but checks only provisioning completion, never the resulting replication roles. Likewise, src/durabletask/azext_durabletask/tests/latest/test_durabletask_commands.py:484-491 supplies the same pre-failover instance to both actions and asserts only the POST destinations. These assertions cannot distinguish a completed configuration from a failover that actually changes roles.

The README explicitly acknowledges the service limitation: the diagnostic recording reaches provisioningState: Failed during configuration (src/durabletask/azext_durabletask/tests/latest/recordings/diagnostics/failover_westus2_failed.yaml:1864), before either action. This is a specific semantic-test concern; deterministic coverage remains not applicable, and the other recordings are not placeholders.

Fix/verify: Assert the expected before/after replication roles for each action on a functioning deployment and retain a successful playback fixture. Add a focused terminal-failure assertion using the diagnostic response shape. Verify that unchanged roles or a failed operation cannot satisfy the scenario assertions.

Upstream CI

6/6 checks completed; 5 passed, 0 failed, 0 pending. No CI failures require attribution.

Test validation
  • Agent live test: Not run; this is a review-only PR. Check author-provided test evidence and upstream CI separately.
  • Regression coverage: Not applicable to the changed files.
⚖️ Risk assessment — 100/100 · Critical

The Critical rating is driven by security-sensitive behavior, public CLI behavior, dependency or supply chain and 2 additional signal(s).

  • Change scope: 52 changed files, 24532 changed lines (+24428 / -104), including 36 production files.
  • Affected components: durabletask
  • Risk drivers: security-sensitive behavior (+28); public CLI behavior (+18); dependency or supply chain (+18); failure-handling behavior (+12); generated output (+12)
  • Regression evidence: Changed regression tests are included, reducing risk.
  • Confidence: High because changed-line patches were available for every production file.
  • Required review: Owning-squad review is required for durabletask before merge.
Additional notes (not anchored to the diff)

Medium Assert each failover's replication-role outcome — src/durabletask/azext_durabletask/tests/latest/test_durabletask_scenario.py:371-373

The waits and this final check observe only provisioning completion; neither action is checked for its expected replication-role change. The synchronous request tests also reuse the pre-failover instance for both POST responses. Add before/after role assertions for each action and verify they reject unchanged roles. The README correctly identifies the existing recording as a configuration failure, not successful action playback; retain a successful fixture from a functioning deployment and add a terminal-failure assertion using the diagnostic response shape.

X Engineering Agent · Reviewer · react 👍 / 👎 to train the agent


Ask the Agent

Tick a box to queue an action. Requires write access to this repository.

  • 🔁 Re-review this PR
  • 🧪 Run Agent live tests on this PR

@x-engineering-agent x-engineering-agent Bot added the X Engineering Agent Reviewed Agent reviewed this PR label Oct 22, 2026
@JaysonTaiMicrosoft

Copy link
Copy Markdown
Contributor

aaz: Azure/aaz#1138

@JaysonTaiMicrosoft JaysonTaiMicrosoft added Request X Engineering Agent Ask the agent to triage this issue or review this PR and removed Request X Engineering Agent Ask the agent to triage this issue or review this PR X Engineering Agent Reviewed Agent reviewed this PR labels Oct 22, 2026
@JaysonTaiMicrosoft

Copy link
Copy Markdown
Contributor

Bernd Verst (@berndverst) There is very little test coverage for this module. Could you add some life cycle scenario tests for all the commands?

Please refer to https://github.com/Azure/azure-cli/blob/dev/doc/authoring_tests.md#automating-tests-for-azure-cli for guidance

@x-engineering-agent x-engineering-agent Bot added the X Engineering Agent Reviewed Agent reviewed this PR label Oct 22, 2026
@a0x1ab

Copy link
Copy Markdown
Member

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 2 pipeline(s).

@JaysonTaiMicrosoft

Copy link
Copy Markdown
Contributor

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 2 pipeline(s).

Bernd Verst (berndverst) pushed a commit to berndverst/aaz that referenced this pull request Oct 26, 2026
Align 2026-07-01-preview models with Azure/azure-cli-extensions#10482. Remove unused retention-policy, scheduler CRUD/update/list, identity, and taskhub update versions and their unused resource configurations.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@JaysonTaiMicrosoft

Copy link
Copy Markdown
Contributor

Bernd Verst (@berndverst) As mention please refer to https://github.com/Azure/azure-cli/blob/dev/doc/authoring_tests.md#automating-tests-for-azure-cli for authoring scenario tests (integration tests). Scenario tests seek to record the interactions with ARM to validate the schema and to verify proper life cycles of the commands.

Since, many of the commands are code generated from AAZ models, unit testing the generated commands provide little value as their outputs are already deterministic.

Please also address the failing CI issue, a potential credential was detected.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@berndverst

Bernd Verst (berndverst) commented Oct 27, 2026 •

Copy link
Copy Markdown
Member Author

JaysonTaiMicrosoft this has never been required of this before and we are in a time crunch now. Given this is a beta CLI extension I think we should approve and merge this. Understood that we want to have recorded tests all of a sudden for the CLI extension (never required of us before) and we can do so in the future.

Credential leak is a false positive. Pushed a commit that should address that however.

@berndverst

Copy link
Copy Markdown
Member Author

JaysonTaiMicrosoft integration testing also provides no real value - we are a typespec based RP. We use RPaaS - RPaaS validates inbound and outbound requests against the swagger generated from Typespec.

All our management SDKs are auto generated from Typespec as well.

@a0x1ab

Copy link
Copy Markdown
Member

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 2 pipeline(s).

Record real scheduler, identity, task hub, retention, task queue, restart, encryption rotation, and Function App attachment lifecycles. Sanitize account metadata and signed operation links. Keep service-blocked failover coverage explicitly live-only.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@berndverst

Copy link
Copy Markdown
Member Author

Added replayable integration tests and four real VCR recordings in commit 20e5f7b2a.

The ScenarioTest lifecycles use disposable resource groups and randomized names, and cover:

  • schedulers, managed identities, task hubs, and retention policies;
  • task queues, throughput updates, and scheduler restart;
  • Function App attachment, including checking the written app settings;
  • customer-managed encryption and enabling/disabling automatic key rotation.

All four scenarios passed against real Azure resources and in offline playback. The checked-in recordings are under src/durabletask/azext_durabletask/tests/latest/recordings/ and contain 208 sanitized HTTP interactions. Account identifiers, identity metadata, generated endpoints, and signed operation-link parameters were removed; the medium-confidence secret scan reports no secrets.

Validation against Azure CLI 2.92.0: the repository's full pre-push test run passed 61 tests, with 1 live-only test skipped. The pre-push linter, pylint, and flake8 checks also passed.

The remaining limitation is failover: the live attempt produced a failed configuration while its asynchronous ARM operation continued polling. That scenario is explicitly live_only, with bounded waits and cleanup, and no successful failover recording is claimed. The README documents recording prerequisites, cleanup, playback, and this limitation.

@x-engineering-agent x-engineering-agent Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review comments are inline on the changed lines. See the X Engineering Agent Report.

'--scheduler-name {scheduler} --yes')
self.cmd('durabletask scheduler delete -g {rg} -n {scheduler} --yes')

@live_only()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium Failover request/response/output coverage remains unverified

This is the only service failover scenario and is skipped in playback; the four committed recordings contain no failover-instance requests. Lines 243-255 also check provisioning success rather than role/state transitions after graceful and forced actions. The documented unsuccessful deployment explains the missing recording but does not establish these behaviors. On a working deployment, assert the actual transitions, capture sanitized successful traffic and enable playback; add terminal polling-failure coverage. Verify with azdev test --series --live test_durabletask_failover_lifecycle_live, then playback after enabling it.

X Engineering Agent · Reviewer · react 👍 / 👎 to train the reviewer

value, flags=re.IGNORECASE)
value = re.sub(r'([?&](?:c|s|h|t)=)[^&"\s]+', r'\g<1>redacted', value)
return re.sub(
r'[a-z0-9-]+\.([a-z0-9]+)\.durabletask\.io',

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium Recording sanitizer corrupts encoded dashboard endpoints

For an encoded endpoint ending in %2f%2f<hostname>, this regex consumes the second 2f together with the hostname. The committed scheduler recording at tests/latest/recordings/test_durabletask_scheduler_lifecycle_recorded.yaml:1419 already contains endpoint=https%3a%2f%scheduler.northcentralus.durabletask.io, a malformed URL. Preserve encoded URI delimiters when replacing the hostname and repair affected recordings. Extend test_dashboard_links_do_not_reveal_tenant_or_scheduler_host to assert exact encoded/decoded endpoint validity and idempotence, then rerun that test and scheduler/taskqueue playback.

X Engineering Agent · Reviewer · react 👍 / 👎 to train the reviewer

@a0x1ab

Copy link
Copy Markdown
Member

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 2 pipeline(s).

Record independent taskhub, retention-policy and multi-identity lifecycles. Preserve unrelated identities during selective removal. Retry failover with a westus2 partner and retain its sanitized service-failure trace explicitly as diagnostics.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@berndverst

Copy link
Copy Markdown
Member Author

Follow-up in 66a982821: added separate, real recorded lifecycle tests for all three previously combined areas:

  • Task hubs: create/show/list, idempotent replacement, delete, and confirmation of HTTP 404 after deletion.
  • Retention policies: all state-specific periods, replacement with a different policy set, persisted values, delete, and HTTP 404 afterward.
  • Identity: add two identities, remove only one while preserving the other, and remove the final assignment. This caught an AAZ single-ID removal bug; the handwritten adapter now preserves unrelated identities and handles resource IDs case-insensitively. Additional payload regressions cover system identities, multiple removals, remove-all, omitted arguments, and missing IDs.

These three scenarios passed live and offline playback. There are now seven passing service recordings containing 309 sanitized HTTP interactions.

Failover uses a second Dedicated scheduler in westus2, with the primary in northcentralus. Both schedulers reach Succeeded, but failover configuration still transitions to Failed. Retrying with a provisioned primary task hub did not resolve it. A sanitized real trace is checked in at src/durabletask/azext_durabletask/tests/latest/recordings/diagnostics/failover_westus2_failed.yaml. It is explicitly diagnostic evidence, not a passing failover playback fixture; the strict lifecycle remains live-only pending the service issue.

Full validation and pre-push hooks passed: 65 tests passed, 1 service-blocked live-only test skipped, plus linter/pylint/flake8. The medium-confidence secret scan and account-identifier audit also passed.

@a0x1ab

Copy link
Copy Markdown
Member

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 2 pipeline(s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Request X Engineering Agent Ask the agent to triage this issue or review this PR X Engineering Agent Reviewed Agent reviewed this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants