[texastony]

Issue #, if available:

Description of changes:

Goal. Build a more complete multi-tenant solution with the AWS KMS Hierarchical Keyring: one that handles a tenant's KMS key becoming unavailable without flooding requests for it. When AWS KMS denies a Branch Key (for example, a 400 AccessDeniedException), the failure is cached. For a configurable timeout, the cache and the components above it stop calling Amazon DynamoDB and AWS KMS for that Branch Key.

Constraint. No changes to the AWS Encryption SDK for Python or the MPL source. Only custom implementations of their public, extendable interfaces are allowed: Keyrings, Master Key Providers, Cryptographic Materials Managers, and Cryptographic Materials Caches. Only the Hierarchical Keyring's behavior is in scope, because it is the multi-tenant solution.

This PR (draft): the design doc only. examples/src/multi_tenant/negative_caching.md records the options considered and the decisions made. Implementation follows in later commits.

• Traces the Hierarchical Keyring's cache and KMS request paths on encrypt and decrypt, through the specification and MPL 1.11.3. Findings:

  • AWS KMS is only called on a cache miss, and always after a DynamoDB GetItem.
  • The cache is never told when a fetch fails.
  • A cache error other than EntryDoesNotExist fails the keyring before the Key Store is called. This is the hook the design relies on.

• A custom cache alone is not enough. It cannot see failures, and cannot tell which Branch Key a cache identifier is for. The design pairs:

  • detection in the Tenant KMS Client, which sees the exact KMS error code
  • enforcement in a per-keyring Cache Guard in front of the cache shared by every tenant. The guard is checked before the shared cache, is a lightweight object, and passes through the KMS error it received, as is.

• Denials are keyed by Branch Key ID, not KMS Key ARN. One KMS key can protect many Branch Keys, and KMS can deny some of them based on their encryption context. A Branch Key's KMS key is also mutable (aws-encryption-sdk-specification#287; the MPL's mutations/mutations TerminalKmsArn). Keying by Branch Key ID also supports the Discovery and MRDiscovery Key Store configurations.

• What counts as a Denial:

  • the KMS key's state: AccessDeniedException, DisabledException, KMSInvalidStateException, NotFoundException, IncorrectKeyException
  • the tenant's credentials: ExpiredTokenException, UnrecognizedClientException, InvalidSignatureException

  botocore only refreshes credentials proactively, never in response to an expired-token error, so a credential failure that reaches KMS will not resolve by retrying. The doc records a probe of AWS KMS that varies one part of the credentials at a time, and confirms that AWS KMS returns HTTP 400 ExpiredTokenException for expired temporary credentials. Only throttling, server errors, and transport failures are transient.

• Encrypt and decrypt Denials are treated the same, for every version of the Branch Key.

• Negative TTL: configured per factory; the default is 5 minutes.

• A Tenant Keyring Factory replaces the Branch Key ID Supplier. It holds the shared transport, cache, DynamoDB client, Key Store settings, and Negative TTL. Given a Branch Key ID and Tenant Credentials, it vends a Tenant Hierarchical Keyring wired with a Cache Guard and a Tenant KMS Client. The factory and everything it vends MUST be thread safe, and this SHOULD be tested.

• The Partition ID is revisited. Refreshed credentials share no constant part, so a Partition ID derived from them changes on every refresh. It should be derived from a stable tenant identity instead. design.md 10.2 now notes this; the example's fix is deferred.

Temporary CI changes (fork only). The ci: commits on this branch exist only to allow development on this fork. Forks have no AWS role, CodeBuild project, or Slack webhooks, and the ubuntu-latest runner no longer provides Python 3.8 or 3.9. These commits are temporary and MUST be reverted before this work is proposed upstream.

• 912ae02:
  • skips the CodeBuild jobs, decrypt_oracle, and the Slack notifications unless the repository owner is aws, and lets pull.yml's gate accept those skips (its daily_ci.yml change is reverted by ded3702)
  • runs ci_tests.yaml on Python 3.12 and Ubuntu only (static analysis stays on 3.10, because the pinned linters predate 3.12)
• 9787429: marks the offline examples tests local and runs them in CI (exampleslocal, mplexampleslocal).
• 46359a2: adds a ddblocal marker and CI job. It runs the Hierarchical Keyring examples against DynamoDB Local (amazon/dynamodb-local:3.3.2, as a service container) with the fake KMS, with no AWS credentials.
• ded3702: only the Pull Request Workflow runs on this fork's PRs.
  • daily_ci.yml is reverted. Editing it made its PR trigger re-run Daily CI on every push; on a fork's schedule, its original guards already skip every job.
  • ci_codebuild-tests.yml keeps only workflow_call, dropping its own push and schedule triggers.

The tests and tox factors from 9787429 and 46359a2 may be worth proposing upstream on their own; the fork guards and the reduced Python and OS matrix should not be.

Out of scope: other keyrings, the KMS Master Key Providers, and the Caching CMM; Amazon DynamoDB failures (the application owns that resource). Propagating a NotFoundException across Branch Key IDs is decided against in Design Question 10.

This branch is based on examples/credentials (aws#827).

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Check any applicable:

• Were any files moved? Moving files changes their URL, which breaks all hyperlinks to the files.

🤖 Generated with Claude Code