Skip to content

feat!(extstore): add new concurrency controls for external storage. - #1917

Open
cconstable wants to merge 11 commits into
mainfrom
extstore-concurrency-controls
Open

cconstable wants to merge 11 commits into
mainfrom
extstore-concurrency-controls

Conversation

@cconstable

@cconstable cconstable commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

What was changed

ExternalStorage now has a concurrency field with two limits:

  • max_driver_operations caps requests in flight for a single client or worker. Each client and each worker gets its own limit.
  • max_operations_per_message caps requests for a single message (workflow activation, etc). Each message gets its own budget.

Both limits are cooperative. Drivers receive a limiter on the store and retrieve contexts and must wrap each request in permit. Requests made outside a permit are not counted, and a warning is logged when a driver completes without taking one.

async with context.limiter.permit(payload):
    # store/retrieve logic

Removed the max_workflow_task_external_storage_concurrency Worker argument. The payload visitor now walks a message's payload sites max_operations_per_message at a time. The bundled S3 driver takes permits.

Why?

The previous control bounded payload visits rather than driver calls, so setting it to 3 did not limit external storage requests to 3. It also only applied to workflow tasks, leaving client, activity, and Nexus paths unbounded.

max_driver_operations is scoped to a client or worker so that ExternalStorage stays configuration. Holding the counter there meant two clients built from one config silently shared a limit.

Checklist

Added tests

@cconstable
cconstable requested a review from a team as a code owner October 2, 2026 01:13
@cconstable
cconstable force-pushed the extstore-concurrency-controls branch from f88df78 to cf8a627 Compare October 2, 2026 21:05
Comment thread temporalio/converter/_extstore.py Outdated
Comment thread temporalio/bridge/worker.py Outdated
Comment thread temporalio/bridge/_visitor.py
Comment thread temporalio/converter/_extstore.py
Comment thread temporalio/converter/_extstore.py Outdated
@cconstable
cconstable force-pushed the extstore-concurrency-controls branch from cf8a627 to 8d3eb8f Compare October 6, 2026 19:19
Comment thread scripts/gen_payload_visitor.py
Comment thread temporalio/converter/_extstore.py Outdated
Comment thread temporalio/converter/_data_converter.py
Comment thread temporalio/client/_client.py
Comment thread temporalio/converter/_data_converter.py

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants