Skip to content

sdkharness: repository-owned leaf checks refuse to run outside a loopback simulator - #620

Merged
sophiecarreras merged 2 commits into
masterfrom
sdkharness/loopback-only-leaf-checks
Oct 1, 2026
Merged

sophiecarreras merged 2 commits into
masterfrom
sdkharness/loopback-only-leaf-checks

Conversation

@sophiecarreras

Copy link
Copy Markdown
Contributor

Why

An independent review of the repository-owned sdkharness tests (merged in #611 and #613) found that the 33 conformance leaf checks, which are executable directly, default CONFORMANCE_TARGET to staging, map staging/production realms, and fall back to ambient B2_APPLICATION_KEY_ID / B2_APPLICATION_KEY. The dispatchers (run-conformance, run-resilience) were safe, but a developer running a leaf by hand with real credentials in the environment could create, mutate or delete resources in an external account. This contradicts .sdkharness/README.md ("never use a production B2 endpoint or real credentials").

What

  • New .sdkharness/tests/lib/loopback_guard.py: refusal(level) returns a reason unless the target is simulator, <LEVEL>_SIMULATOR_URL is http://127.0.0.1:<port>, an optional <LEVEL>_SIMULATOR_HTTPS_URL is https://127.0.0.1:<port>, and (resilience) RESILIENCE_CONTROL_URL is a loopback origin.
  • All 33 conformance and 16 resilience leaves call it first and print the standard FAIL (configuration -- ...) line with exit 1. Default target is now simulator; the staging/production realm entries and the ambient-credential fallbacks are removed (the fixed test-key-id / test-key credential is the only one). client.simulator loses unused realm and ambient-credential helpers (it only drives the in-process RawSimulator with the network disabled).
  • README: how to run one health, one conformance and one resilience check through the dispatchers against a local simulator.

Verification

  • 71 unit tests pass (test/unit/test_sdkharness_conformance_resilience_contract.py and test_sdkharness_contract.py); the new tests (54 failing cases) fail on the previous leaves and pass now.
  • Ran through the dispatchers against a local simulator: files.upload, bucket.crud, enc.sse_c, client.simulator PASS; api.backoff_503 PASS; upload.retry_408 FAIL as before (the known Retry-After/408 SDK findings). A direct leaf run with CONFORMANCE_TARGET=staging and ambient keys exits 1 with FAIL (configuration ...).
  • ruff check and ruff format --check clean on .sdkharness and the test file. Not run locally: the full nox matrix and integration suite (read from CI).
  • Changelog fragment: +sdkharness-loopback-guard.infrastructure.md (no related issue; the contributing guide's convention for PRs without an issue).

Review requested from the maintainers who merged recently. Nothing here changes SDK code.

🤖 Generated with Claude Code

The 33 conformance and 16 resilience leaf checks are executable on their own.
Conformance leaves defaulted CONFORMANCE_TARGET to staging, mapped staging and
production realms, and fell back to ambient B2_APPLICATION_KEY_ID/KEY, so a
direct run could mutate an external account. Each leaf now calls a shared guard
first: only the simulator target, only http://127.0.0.1:<port> (and https for
the TLS URL, and the loopback control URL for resilience), and only the fixed
test credential. client.simulator no longer carries unused realm/ambient
credential helpers. Adds unit tests (54 fail on the previous leaves) and a
README section for running one check locally.
@sophiecarreras
sophiecarreras requested review from mpnowacki-reef and ppolewicz and removed request for mpnowacki-reef and ppolewicz October 1, 2026 04:22
@sophiecarreras sophiecarreras self-assigned this Oct 1, 2026
…nd IPv4 loopback; readiness wait in README

Review follow-up for the PR: the health golden path read any non-empty
B2_TEST_APPLICATION_KEY_* pair and passed it to the SDK; a real key could be
disclosed to a loopback listener. It now refuses anything but the fixed
simulator pair before the SDK is touched, accepts only 127.0.0.1 (like the
other leaves), and the README waits for the simulator to be ready.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@sophiecarreras

sophiecarreras commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor Author

Follow-up commit e678272 for the independent review of this PR:

  • Health leaf credential (medium): .sdkharness/tests/health-golden-path accepted any non-empty B2_TEST_APPLICATION_KEY_ID/B2_TEST_APPLICATION_KEY and passed it to authorize_account. It now refuses anything but the simulator's fixed test-key-id / test-key pair before the SDK is touched (the supplied values are never echoed), and uses the fixed pair for the call. I re-read every conformance, resilience and library file in .sdkharness/: the conformance leaves already use only the fixed pair and the resilience leaves hard-code it, so the health leaf was the only path.
  • IPv4-only (consistency): the health leaf also accepted ::1; it now accepts only http://127.0.0.1:<port> like the conformance and resilience leaves (README states this).
  • README startup race (low): the copy-paste sequence now waits (up to ~10 s) until the simulator has printed its three listener lines before reading its URLs. I ran the sequence verbatim against a local b2-simulator main: conformance files.upload and the health check PASS; a real-looking key is refused with configuration: only the fixed simulator credential is accepted.
  • Tests: 3 new cases (real-looking key / id / both; the SDK must never be called) and an ::1 case; they fail on the previous commit (4 failures) and pass now (75 passed in test_sdkharness_*). ruff check clean on the touched files with the pinned ruff 0.8.x. (The two files ruff format --check flags, noxfile.py and doc/sqlite_account_info_schema.py, are unformatted on master too.)

Not run here: the full nox matrix. CI on this PR shows the known unrelated live-encryption failures (#614) — I have not diagnosed them.

@sophiecarreras
sophiecarreras merged commit 481d8fb into master Oct 1, 2026
9 of 28 checks passed
@sophiecarreras
sophiecarreras deleted the sdkharness/loopback-only-leaf-checks branch October 1, 2026 13:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant