Skip to content

Implement Adaptive retries - #842

Merged
ubaskota merged 8 commits into
developfrom
adaptive_retries
Oct 15, 2026
Merged

ubaskota merged 8 commits into
developfrom
adaptive_retries

Conversation

@ubaskota

@ubaskota ubaskota commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Issue #, if available:

Description of changes:
Adds the adaptive retry mode. It builds on standard retries by adding client-side rate limiting: once the service starts throttling, the client slows its send rate. The work was reviewed in three PRs and merged into the adaptive_retries feature branch:

  • Token bucket: TokenBucket, an async token bucket whose refill rate can change over time. It controls how fast requests are sent.
  • Client-side rate limiting: ClientRateLimiter coordinates rate limiting. CubicCalculator adjusts the rate with CUBIC: it cuts to 70% on a throttle and grows back on success. RequestRateTracker measures the client's actual send rate.
  • Adaptive retry strategy: AdaptiveRetryStrategy extends StandardRetryStrategy and hooks rate limiting into RequestPipeline.

Behavior:

  • Opt-in. Standard stays the default.
  • Rate limiting stays off until the first throttling response. Every request then waits for a send token before it's sent, including operations that can't be retried (such as input event streams), and reports its result to the rate limiter.
  • If a send token isn't available within 30s, the operation fails without retrying or using up an attempt.
  • DynamoDB and DynamoDB Streams clients keep their retry defaults (4 attempts, 25ms base backoff) in adaptive mode.
  • RetryStrategyResolver now caches strategies per client instead of in a shared 128-entry cache with @lru_cache, so a client's rate limiter state isn't reset when many other clients are created.
  • AWS_RETRY_MODE=adaptive (or retry_mode = adaptive in a config profile) now selects adaptive mode instead of falling back to standard with a warning.
  • The adaptive classes now live in smithy_core.aio.retries

Thread safety: threading.Lock instead of asyncio.Lock

An asyncio.Lock only coordinates tasks on a single event loop. If one client is used from several threads, each running its own loop, the token count, measured send rate and CUBIC state could have a race condition, and calls could fail. The short sections that update this state now use a threading.Lock, which is what StandardRetryQuota already does.

A thread lock is fine here because it only covers a few lines of arithmetic. It's never held across an await, and never while waiting for a token, so it doesn't block for any meaningful time. This follows earlier discussions:

Why the asyncio.Condition stays: waiting for a token is real waiting. A request pauses until tokens refill or the rate changes, and other requests keep running in the meantime. The bucket keeps one Condition per event loop, used only for waiting and wakeups. The thread lock is released before waiting.

Testing:

  • Unit tests for each component, plus functional tests through the real request pipeline.
  • A step-by-step retry sequence test, and a convergence test that runs on a simulated clock: a client attempting 100 r/s settles near a server's 10 r/s limit.
  • Manually verified against a local mock server limited to 10 requests/s, with one client making 50 concurrent requests. The limiter turned on after the first 429s, and the client settled at about 10 requests/s within 5s, with no failed calls.
  • Type checking and linting pass.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

* Implement a token bucket used for adaptive retries
* Implement client-side rate limiting for adaptive retries.
* Implement Adaptive retry strategy for Adaptive retries.
@ubaskota
ubaskota requested a review from a team as a code owner October 9, 2026 18:36
Comment thread packages/smithy-core/src/smithy_core/aio/retries.py Outdated
Comment thread packages/smithy-core/src/smithy_core/aio/retries.py Outdated
Comment thread packages/smithy-core/src/smithy_core/aio/retries.py Outdated
Alan4506
Alan4506 previously approved these changes Oct 15, 2026
Comment thread packages/smithy-core/src/smithy_core/aio/retries.py Outdated

@alexgromero alexgromero left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@ubaskota
ubaskota merged commit 1692e03 into develop Oct 15, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants