Skip to content

sdkharness: fail, not skip, on auth/connection/import errors; scrub proxies - #623

Merged
sophiecarreras merged 1 commit into
masterfrom
sdkharness/fail-not-skip-python
Oct 4, 2026
Merged

sophiecarreras merged 1 commit into
masterfrom
sdkharness/fail-not-skip-python

Conversation

@sophiecarreras

Copy link
Copy Markdown
Contributor

Summary

Review findings for the repo-owned .sdkharness/ Python checks.

  • FAIL, not SKIP. The 33 conformance leaves mapped Unauthorized/InvalidAuthToken/AccessDenied and gaierror/ConnectionError/ServiceError/B2ConnectionError to COULD-NOT-RUN, and any ImportError to missing-runtime. lib/contract.py turned that into SKIP, exit 0. At a loopback simulator the harness supplies the server and the credential, so these are all defects now. Only ModuleNotFoundError for b2sdk itself (and a missing simulator URL) may be COULD-NOT-RUN; a broken or circular SDK import is FAIL. The urls.native_download URLError paths and the client.auth_persistence second-process mapping follow the same rule.
  • Dispatcher SKIP policy. lib/contract.py only accepts no-realm-option, missing-runtime, no-client-option, not-claimed as SKIP; any other COULD-NOT-RUN reason becomes FAIL (the CLI's "SKIP policy FAIL"). The resilience leaves get the same import handling.
  • Proxy scrub. HTTP_PROXY/HTTPS_PROXY/ALL_PROXY/FTP_PROXY (any case) are removed and NO_PROXY is pinned to 127.0.0.1 in the dispatcher child environment, in the health check, and in every leaf (via loopback_guard.refusal()), so a proxy on the machine cannot turn a passing loopback scenario into an "unreachable" one.
  • Docs. README no longer says api.retry_after_503 / upload.retry_408 FAIL (both pass); the garbled files.upload docstring is fixed; references to harness-only paths (bin/conformance/README.md, docs/sdks/.../card.md, bin/resilience/...) are removed.
  • Changelog fragment changelog.d/+sdkharness-fail-not-skip.infrastructure.md.

Test plan

  • New test/unit/test_sdkharness_fail_not_skip.py: a dead loopback port and an SDK ImportError each yield FAIL (leaf and dispatcher translation); a genuinely missing b2sdk is still SKIP; unreachable/unauthorized reasons from the leaf are FAIL in the dispatcher; proxy variables are scrubbed.
  • pytest test/unit/test_sdkharness*.py: 116 passed.
  • Ran all 33 conformance + 16 resilience scenarios and the health check against a local loopback b2-simulator with hostile HTTP_PROXY/HTTPS_PROXY/ALL_PROXY set: no production endpoint was contacted.

Follow-up outside this repo: the central Python conformance adapter (python-repository-conformance-adapter.sh in the harness) still leaves SKIP_POLICY at the amber default and should set SKIP_POLICY=fail.

🤖 Generated with Claude Code

Conformance leaves mapped Unauthorized/AccessDenied/ServiceError/
ConnectionError to COULD-NOT-RUN and any ImportError to missing-runtime,
which the dispatcher turned into SKIP (exit 0). At a loopback simulator the
harness supplies both server and credential, so those are defects. Only a
genuinely absent b2sdk module (or a missing simulator URL) may be amber.

- leaves: auth, access-denied, connection and service errors, URL errors and
  SDK ImportErrors are FAIL; only ModuleNotFoundError for b2sdk itself is
  missing-runtime
- lib/contract.py: SKIP only for no-realm-option, missing-runtime,
  no-client-option, not-claimed; any other COULD-NOT-RUN reason is FAIL
- scrub proxy variables in the dispatcher, the health check and every leaf
  (NO_PROXY pinned to 127.0.0.1) so a stray proxy cannot change a result
- README: api.retry_after_503 and upload.retry_408 pass now
- fix the garbled files.upload docstring and references to harness-only paths
- unit tests: dead simulator port and SDK ImportError yield FAIL

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@sophiecarreras sophiecarreras self-assigned this Oct 4, 2026
@sophiecarreras
sophiecarreras merged commit e4117e3 into master Oct 4, 2026
27 of 28 checks passed
@sophiecarreras
sophiecarreras deleted the sdkharness/fail-not-skip-python branch October 4, 2026 00:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant