Skip to content

sdkharness: fail, not skip, on auth/connection/import errors; scrub proxies #623

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sophiecarreras merged 1 commit into from
Oct 4, 2026
Merged

sdkharness: fail, not skip, on auth/connection/import errors; scrub proxies #623

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 12 additions & 3 deletions .sdkharness/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,9 @@ only the fixed simulator credential `test-key-id` / `test-key`, and the health
check refuses any other `B2_TEST_APPLICATION_KEY*` pair before it reaches the
SDK. The individual conformance and resilience check files refuse to run at all
unless they are given a loopback simulator URL, so run them through the
dispatchers below, never with real `B2_*` values in the environment.
dispatchers below, never with real `B2_*` values in the environment. The
dispatchers also drop `HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY` (any case) and pin
`NO_PROXY` to `127.0.0.1`, so a proxy configured on the machine cannot change a result.

Conformance owns 33 capability checks and resilience owns 16 injected-fault
checks. Their small dispatchers validate the invocation, run the selected
Expand Down Expand Up @@ -83,5 +85,12 @@ HEALTHCHECK_REALM_URL=$SDKHARNESS_SIMULATOR_URL B2_TEST_APPLICATION_KEY_ID=test-
When you are done, stop the simulator with `kill "$SIM_PID"`.

Each prints one `SDKHARNESS_RESULT` line. Scenario ids are in `tests.tsv`.
A `FAIL` for `api.retry_after_503`, `upload.retry_408` is a known SDK finding, not a setup problem.

`api.retry_after_503` and `upload.retry_408` used to FAIL as known SDK findings;
both pass now, so there are no standing known findings listed here. `upload.stall`
reports `SKIP` because b2sdk documents no request-timeout option. A `FAIL` is a
real SDK or harness defect (or a simulator older than the checks expect), not a
setup problem. `SKIP` is reserved for a question that genuinely cannot be asked
(`no-realm-option`, `missing-runtime` when `b2sdk` is not installed at all,
`no-client-option`, `not-claimed`). An unreachable simulator, an authorization
error, an access-denied error, a connection error, or an SDK `ImportError` is
always a `FAIL`.
44 changes: 23 additions & 21 deletions .sdkharness/tests/conformance/bucket.cors
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,12 @@ claims it):
Action set a CORS rule set, then read it back
Assert field-for-field equality

The card answers `partial` for this cell and states the boundary: CORS
The capability record answers `partial` for this cell and states the boundary: CORS
round-trips as an UNTYPED dict with no typed model and no dedicated
accessor, unlike lifecycle and notification rules. That boundary is the
most falsifiable claim in the matrix, so it is asserted here too -- a typed
CorsRule or a get/set_cors_rules accessor appearing upstream FAILS this
check, because the card would then be stale.
check, because the record would then be stale.

EXPECTED VALUES -- B2's published contract, reviewed 2026-09-23.
Every field sent here is a documented rule field (corsRuleName, allowedOrigins,
Expand All @@ -24,21 +24,20 @@ b2_list_buckets returns corsRules.
https://www.backblaze.com/docs/cloud-storage-cross-origin-resource-sharing-rules
https://www.backblaze.com/apidocs/b2-update-bucket
https://www.backblaze.com/apidocs/b2-list-buckets
The `partial` boundary leg tests the card's claim about b2sdk's own surface;
The `partial` boundary leg tests the capability's recorded claim about b2sdk's own surface;
B2's docs do not speak to it.

Written from bin/conformance/b2-sdk-python/files.upload, the reference check,
and bin/conformance/README.md, the contract. It exercises b2sdk from this exact repository
checkout; the API was taken from
docs/sdks/b2-sdk-python/card.md and the published b2sdk docs; the assertion now lives with the SDK code it exercises.
Written from files.upload in this directory, the reference check. It exercises
b2sdk from this exact repository checkout; the API usage is taken from the
published b2sdk documentation.
"""

import os
import sys
import uuid

sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), '..', 'lib'))
from loopback_guard import refusal # noqa: E402
from loopback_guard import missing_runtime, refusal # noqa: E402

SLUG = 'b2-sdk-python'
CAPABILITY = 'bucket.cors'
Expand All @@ -51,8 +50,10 @@ RULE = {
'maxAgeSeconds': 3600,
}

# The closed reason set, mirrored from bin/conformance/README.md. The runner
# enforces it; naming one outside this tuple is a FAIL it manufactures.
# The closed reason set. `lib/contract.py` only turns `no-realm-option`,
# `missing-runtime` and `not-claimed` into a SKIP. `unreachable` and
# `unauthorized` are never reported at the simulator: the runner supplies both
# the server and the credential, so those are FAIL, not amber.
REASONS = (
'no-realm-option',
'unreachable',
Expand All @@ -63,7 +64,7 @@ REASONS = (
)

# A requested target maps to a realm or to amber. `simulator` is the harness's
# own server (bin/simulator/serve.mjs), whose URL the runner exports as
# own loopback server, whose URL the dispatcher exports as
# CONFORMANCE_SIMULATOR_URL; unset, there is nothing to reach: no-realm-option.
REALMS = {
'simulator': os.environ.get('CONFORMANCE_SIMULATOR_URL') or None,
Expand Down Expand Up @@ -92,7 +93,7 @@ class Amber(Exception):


class Failure(Exception):
"""A golden-path step that did not do what the card claims."""
"""A golden-path step that did not do what the capability requires."""

def __init__(self, step: str, detail: str) -> None:
self.step = step
Expand All @@ -116,19 +117,17 @@ def note(text: str) -> None:
def step(name, action):
"""Run one golden-path step, converting the environment's own errors.

A credential NEVER reaches the detail: only the exception class name and,
for a classified network/auth error, the class that classified it.
A credential NEVER reaches the detail: only the exception class name.
Every error here is a FAIL: the runner supplies both the simulator and the
credential, so an auth error, access denied, connection error or SDK error
is a defect, never an amber (COULD-NOT-RUN) result.
"""
try:
return action()
except (Amber, Failure):
raise
except Exception as error: # noqa: BLE001 -- classified below, never echoed
name_of = type(error).__name__
if name_of in ('Unauthorized', 'InvalidAuthToken', 'AccessDenied'):
raise Amber('unauthorized', f"{name_of} at step '{name}'") from error
if name_of in ('gaierror', 'ConnectionError', 'ServiceError', 'B2ConnectionError'):
raise Amber('unreachable', f"{name_of} at step '{name}'") from error
raise Failure(name, name_of) from error


Expand Down Expand Up @@ -173,6 +172,9 @@ def prepare(target):
try:
import b2sdk.v3 as b2
except ImportError as error:
if not missing_runtime(error):
# b2sdk is installed but broken: an SDK regression, not a missing runtime
raise Failure('import', type(error).__name__) from error
raise Amber('missing-runtime', 'b2sdk is not installed') from error

return b2, realm
Expand Down Expand Up @@ -280,18 +282,18 @@ def run(target: str) -> None:
if hasattr(b2, 'CorsRule'):
raise Failure(
'boundary',
"b2sdk.v3 now exports a typed CorsRule; the card's `partial` answer is stale",
"b2sdk.v3 now exports a typed CorsRule; the capability's `partial` answer is stale",
)
for accessor in ('get_cors_rules', 'set_cors_rules'):
if hasattr(b2.Bucket, accessor):
raise Failure(
'boundary',
f"Bucket.{accessor} now exists; the card's `partial` answer is stale",
f"Bucket.{accessor} now exists; the capability's `partial` answer is stale",
)
if type(rule) is not dict:
raise Failure(
'boundary',
f'the CORS rule is a {type(rule).__name__}, not the untyped dict the card records',
f'the CORS rule is a {type(rule).__name__}, not the untyped dict the record names',
)

note('boundary held: CORS round-trips as a plain dict, with no typed model and no accessor')
Expand Down
32 changes: 17 additions & 15 deletions .sdkharness/tests/conformance/bucket.crud
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,24 +20,25 @@ https://www.backblaze.com/apidocs/b2-update-bucket
https://www.backblaze.com/apidocs/b2-delete-bucket
https://www.backblaze.com/apidocs/b2-list-buckets

Written from bin/conformance/b2-sdk-python/files.upload, the reference check,
and bin/conformance/README.md, the contract. It exercises b2sdk from this exact repository
checkout; the API was taken from
docs/sdks/b2-sdk-python/card.md and the published b2sdk docs; the assertion now lives with the SDK code it exercises.
Written from files.upload in this directory, the reference check. It exercises
b2sdk from this exact repository checkout; the API usage is taken from the
published b2sdk documentation.
"""

import os
import sys
import uuid

sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), '..', 'lib'))
from loopback_guard import refusal # noqa: E402
from loopback_guard import missing_runtime, refusal # noqa: E402

SLUG = 'b2-sdk-python'
CAPABILITY = 'bucket.crud'

# The closed reason set, mirrored from bin/conformance/README.md. The runner
# enforces it; naming one outside this tuple is a FAIL it manufactures.
# The closed reason set. `lib/contract.py` only turns `no-realm-option`,
# `missing-runtime` and `not-claimed` into a SKIP. `unreachable` and
# `unauthorized` are never reported at the simulator: the runner supplies both
# the server and the credential, so those are FAIL, not amber.
REASONS = (
'no-realm-option',
'unreachable',
Expand All @@ -48,7 +49,7 @@ REASONS = (
)

# A requested target maps to a realm or to amber. `simulator` is the harness's
# own server (bin/simulator/serve.mjs), whose URL the runner exports as
# own loopback server, whose URL the dispatcher exports as
# CONFORMANCE_SIMULATOR_URL; unset, there is nothing to reach: no-realm-option.
REALMS = {
'simulator': os.environ.get('CONFORMANCE_SIMULATOR_URL') or None,
Expand Down Expand Up @@ -77,7 +78,7 @@ class Amber(Exception):


class Failure(Exception):
"""A golden-path step that did not do what the card claims."""
"""A golden-path step that did not do what the capability requires."""

def __init__(self, step: str, detail: str) -> None:
self.step = step
Expand All @@ -101,19 +102,17 @@ def note(text: str) -> None:
def step(name, action):
"""Run one golden-path step, converting the environment's own errors.

A credential NEVER reaches the detail: only the exception class name and,
for a classified network/auth error, the class that classified it.
A credential NEVER reaches the detail: only the exception class name.
Every error here is a FAIL: the runner supplies both the simulator and the
credential, so an auth error, access denied, connection error or SDK error
is a defect, never an amber (COULD-NOT-RUN) result.
"""
try:
return action()
except (Amber, Failure):
raise
except Exception as error: # noqa: BLE001 -- classified below, never echoed
name_of = type(error).__name__
if name_of in ('Unauthorized', 'InvalidAuthToken', 'AccessDenied'):
raise Amber('unauthorized', f"{name_of} at step '{name}'") from error
if name_of in ('gaierror', 'ConnectionError', 'ServiceError', 'B2ConnectionError'):
raise Amber('unreachable', f"{name_of} at step '{name}'") from error
raise Failure(name, name_of) from error


Expand Down Expand Up @@ -158,6 +157,9 @@ def prepare(target):
try:
import b2sdk.v3 as b2
except ImportError as error:
if not missing_runtime(error):
# b2sdk is installed but broken: an SDK regression, not a missing runtime
raise Failure('import', type(error).__name__) from error
raise Amber('missing-runtime', 'b2sdk is not installed') from error

return b2, realm
Expand Down
32 changes: 17 additions & 15 deletions .sdkharness/tests/conformance/bucket.lifecycle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,26 +18,27 @@ list."
https://www.backblaze.com/docs/cloud-storage-lifecycle-rules
https://www.backblaze.com/apidocs/b2-update-bucket

Written from bin/conformance/b2-sdk-python/files.upload, the reference check,
and bin/conformance/README.md, the contract. It exercises b2sdk from this exact repository
checkout; the API was taken from
docs/sdks/b2-sdk-python/card.md and the published b2sdk docs; the assertion now lives with the SDK code it exercises.
Written from files.upload in this directory, the reference check. It exercises
b2sdk from this exact repository checkout; the API usage is taken from the
published b2sdk documentation.
"""

import os
import sys
import uuid

sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), '..', 'lib'))
from loopback_guard import refusal # noqa: E402
from loopback_guard import missing_runtime, refusal # noqa: E402

SLUG = 'b2-sdk-python'
CAPABILITY = 'bucket.lifecycle'
RULE = {'fileNamePrefix': 'st/', 'daysFromHidingToDeleting': 1}
OTHER_FIELDS = ('daysFromUploadingToHiding', 'daysFromStartingToCancelingUnfinishedLargeFiles')

# The closed reason set, mirrored from bin/conformance/README.md. The runner
# enforces it; naming one outside this tuple is a FAIL it manufactures.
# The closed reason set. `lib/contract.py` only turns `no-realm-option`,
# `missing-runtime` and `not-claimed` into a SKIP. `unreachable` and
# `unauthorized` are never reported at the simulator: the runner supplies both
# the server and the credential, so those are FAIL, not amber.
REASONS = (
'no-realm-option',
'unreachable',
Expand All @@ -48,7 +49,7 @@ REASONS = (
)

# A requested target maps to a realm or to amber. `simulator` is the harness's
# own server (bin/simulator/serve.mjs), whose URL the runner exports as
# own loopback server, whose URL the dispatcher exports as
# CONFORMANCE_SIMULATOR_URL; unset, there is nothing to reach: no-realm-option.
REALMS = {
'simulator': os.environ.get('CONFORMANCE_SIMULATOR_URL') or None,
Expand Down Expand Up @@ -77,7 +78,7 @@ class Amber(Exception):


class Failure(Exception):
"""A golden-path step that did not do what the card claims."""
"""A golden-path step that did not do what the capability requires."""

def __init__(self, step: str, detail: str) -> None:
self.step = step
Expand All @@ -101,19 +102,17 @@ def note(text: str) -> None:
def step(name, action):
"""Run one golden-path step, converting the environment's own errors.

A credential NEVER reaches the detail: only the exception class name and,
for a classified network/auth error, the class that classified it.
A credential NEVER reaches the detail: only the exception class name.
Every error here is a FAIL: the runner supplies both the simulator and the
credential, so an auth error, access denied, connection error or SDK error
is a defect, never an amber (COULD-NOT-RUN) result.
"""
try:
return action()
except (Amber, Failure):
raise
except Exception as error: # noqa: BLE001 -- classified below, never echoed
name_of = type(error).__name__
if name_of in ('Unauthorized', 'InvalidAuthToken', 'AccessDenied'):
raise Amber('unauthorized', f"{name_of} at step '{name}'") from error
if name_of in ('gaierror', 'ConnectionError', 'ServiceError', 'B2ConnectionError'):
raise Amber('unreachable', f"{name_of} at step '{name}'") from error
raise Failure(name, name_of) from error


Expand Down Expand Up @@ -158,6 +157,9 @@ def prepare(target):
try:
import b2sdk.v3 as b2
except ImportError as error:
if not missing_runtime(error):
# b2sdk is installed but broken: an SDK regression, not a missing runtime
raise Failure('import', type(error).__name__) from error
raise Amber('missing-runtime', 'b2sdk is not installed') from error

return b2, realm
Expand Down
Loading
Loading