feat(xtest): cross-SDK TDF validity suite (DSPX-5349) by dmihalcik-virtru · Pull Request #677 · opentdf/tests

dmihalcik-virtru · 2026-10-23T14:32:41Z

What and why

DSPX-5349, under the TDF Validity epic (DSPX-5348).

This adds xtest/test_tdf_validity.py. It asks every SDK's validate command the same question about the same bytes: is this a valid TDF? Each answer is held to one checked-in baseline, xtest/tdfvalidity_baseline.json. Any difference fails the cell, in either direction. A validator that gets more lenient fails, and so does a known divergence that quietly stops reproducing.

Today each SDK carries its own copy of these fixtures; web-sdk#1104 alone has 48 archives and a Go verifier script. Those copies are how the SDKs drifted apart. spec/tdf-validity.md has the details.

CLI contract (all three SDKs)

SDK	command	CLI PR
go	`otdfctl validate <file>`	DSPX-5350, opentdf/platform#4254
js	`@opentdf/ctl validate <file>`	DSPX-5351, opentdf/web-sdk#1119 (stacked on opentdf/web-sdk#1118, opentdf/web-sdk#1117, opentdf/web-sdk#1104)
java	`java -jar cmdline.jar validate <file>`	DSPX-5352, opentdf/java-sdk#428 (stacked on opentdf/java-sdk#418)

stdout is exactly one JSON line: {"valid":true} or {"valid":false,"reason":"..."}.
Exit 0 means valid. Exit 3 means the check ran and the file is not a valid TDF. Any other exit, a signal, or a timeout means the check could not run.
No platform, profile or credentials are needed.

The suite uses 3 for "invalid", not the 1 the tickets say: cobra, Node and picocli all exit 1 on an unhandled error, so 1 would make a crash look like a verdict. This is written down in the shim headers, in tdfvalidity.py and in the spec. The parser is strict: if stdout does not agree with the exit status, the result counts as an error, not a verdict.

What's here

xtest/tdfs.py
- A new tdf_validity feature.
- SDK.validate(path), which returns a parsed ValidateResult (valid, invalid with a reason, or error).
- run_bounded(), which runs the shim in its own process group and kills the whole group on a timeout (XT_VALIDATE_TIMEOUT, default 60 s).
xtest/sdk/{go,java,js}/cli.sh
- validate <file> hands off to the CLI with exec, so stdout and the exit status come through unchanged. It passes no platform or log-level flags.
- supports tdf_validity detects the command, so released CLIs skip. The go shim looks for the command's usage line in help validate, because cobra prints the root usage and exits 0 for an unknown help topic. The js shim looks for it in opentdf help. The java shim asks the jar itself (supports tdf_validity), as it already does for dpop.
- The go shim installs module@version once and runs the binary, instead of using go run. go run turns exit 3 into "exit status 3" plus its own exit 1.
xtest/tdfvalidity_corpus.py: 237 static cases, generated in Python, byte-identical for every SDK:
- the 42 manifest cases and the BOM/duplicate archive cases from web-sdk#1104;
- Go's TestNew_ShouldValidateStandardTdf, TestNew_ShouldNotValidateBadStandardTdf and related fixtures, verbatim, in tdfvalidity_fixtures.py;
- truncation at every structural boundary (zip32 and ZIP64);
- malformed archives;
- the DSPX-5348 divergence table, adapted from the verified generator;
- container-profile cases;
- manifest-name selection;
- one lax-schema keyword at a time, generated from the vendored manifest-lax.schema.json;
- realistic writer manifests;
- xtest/golden files;
- payload scope pins;
- a 10,001-digit number, which java rejects and RFC 8259 permits.
xtest/tdfvalidity.py: the contract parser, the baseline model and its consistency checks, judging, the test-results/tdf-validity.json report, and the baseline refresh command. Judging includes the reason's layer: archive, missing manifest, JSON, or schema, matched by per-SDK patterns.
xtest/test_tdf_validity.py:
- The static corpus. It needs no platform and runs one class per group, so --dist loadscope can spread it across workers.
- Round trips: every encrypt SDK × every validator, for RSA-wrapped, ec-wrapped, 4.2.2, assertions and multi-KAO.
- X-Wing, secp+ML-KEM and ML-KEM round trips, marked tdf_validity_pqc. ML-KEM is recorded as a known divergence (DSPX-4388), not skipped.
xtest/test_tdf_validity_units.py: 125 offline tests. They cover:
- the corpus, read back by zipfile and zipinspect;
- the agreed answers, checked against an independent jsonschema run of the lax schema;
- a mutation analysis: the 42 web-sdk cases leave 105 of 171 single-keyword schema mutants alive, and the full corpus kills every mutant except the 11 provably equivalent ones;
- the contract parser, run_bounded group cleanup, baseline parsing and consistency, judging, the report and refresh;
- all three real cli.sh shims, run against stand-in CLIs.
Workflows
- xtest.yml: adds test_tdf_validity.py next to test_tdfs.py/test_policytypes.py in both standard steps. Adds a step after the extra KAS instances start, for the PQ round trips (they need km1), with its own report file. Uploads test-results/**/*.json.
- check.yml: runs the new unit tests.
spec/tdf-validity.md: what "valid" means, the contract, the classification, current results, and how to refresh or seed the baseline.

Baseline: how it was seeded

Every SDK's baseline comes from its real validate CLI, built locally from the CLI PRs and run over this exact corpus through the shims in this PR:

go: otdfctl from feat(cli): add otdfctl validate command (DSPX-5350) platform#4254 (e33f3cb). It reports platform/sdk v0.35.0 IsValidTdf with err.Error() as the reason.
js: @opentdf/ctl from feat(cli): add validate command (DSPX-5351) web-sdk#1119 (d293e509, stacked on #1118, #1117 and #1104). Its verdicts match #1104's isValidTdf on every case.
java: cmdline.jar from feat(sdk): add isValidTdf with Go lax-schema validation, and tdf validate (DSPX-5352) java-sdk#428 (8e9221d, stacked on fix(xtest): Enable extended EC tests for java >= 0.13.0 #418). It was seeded with the documented tdfvalidity.py ... --seed java refresh, then I added the tickets.

All 711 static cells (237 × 3) match, reason layers included. Every cell that differs from the agreed answer names its ticket:

case	agreed	go	java	js	ticket
archive comment (5 B and 64 KiB)	undecided	invalid	valid	valid	DSPX-5355
deflated payload / method 8 on stored bytes / size mismatch	invalid	valid	valid	invalid	DSPX-5354 (go), DSPX-4924 (java)
deflated manifest	invalid	invalid, at the JSON layer	invalid, at the JSON layer	invalid	DSPX-5354, DSPX-4924
EOCD central-directory size 0xfffffff0	invalid	valid	invalid	invalid	DSPX-5354
stored size that runs into the central directory	invalid	invalid	valid	invalid	DSPX-5103
trailing garbage / second JSON value after the manifest	invalid	valid	invalid	invalid	DSPX-5353
JSON nested 20,000 deep	undecided	invalid	invalid	valid	DSPX-5356
`mlkem-wrapped` manifests and the ML-KEM round trip	valid	invalid	invalid	invalid	DSPX-4388
no `0.payload` entry	undecided	valid	invalid	valid	DSPX-4924
duplicated manifest names	undecided	last copy wins	rejected (archive layer)	last copy wins	DSPX-4924
10,001-digit number	unspecified	valid	invalid	valid

java's STORED-entry cells carry DSPX-4924 because there is no java ticket for them yet. java-sdk#424 would make java read deflated entries, the opposite of what DSPX-5354 asks of Go, so the container-profile decision has to come first. The ML-KEM round-trip cells are inferred, not observed: all three SDKs vendor the same lax schema, and the writer/*-mlkem-wrapped cases show each one rejecting mlkem-wrapped.

Things this turned up:

golden/xstext-java-v0.7.5-* omits method.isStreamable, so all three validators reject it, yet every SDK still decrypts it (test_legacy.py). "Valid" is stricter than "readable" there.
DSPX-5357 (duplicate manifest and payload resolving to different records) only affects decrypt, because validity never reads the payload. The duplicate-pairs case stays in for DSPX-4924.

Base branch

Based on main, not stacked on #624. #624's zipmutate.py rebuilds only the trailer of an existing TDF, and it hardcodes STORED entries with equal sizes. It can't produce most of this corpus: from-scratch archives, deflate and size mismatches, local-header lies, truncation, or data-descriptor and ZIP64 entries. Its runner is a private, decrypt-specific helper in a test module, and zipreport.py reports decrypt conformance, not validity. #624 is also 27 commits behind main. This PR follows #624's patterns instead: user_properties collected in conftest, JSON plus a step summary, and a process-group timeout. It reuses zipinspect.py as the independent oracle in the unit tests. Expect small conflicts with #624 in conftest.py (both add a pytest_runtest_logreport and a call in pytest_sessionfinish) and in the xtest.yml test lists.

#600 rewrites how tdfs.py reads the manifest. The changes here are additive and stay away from those functions. Bounded is a NamedTuple specifically so the import block doesn't touch the line #600 edits.

How to test

cd xtest
uv sync --extra dev
uv run ruff check . && uv run ruff format --check . && uv run pyright
uv run pytest test_tdf_validity_units.py
# with CLIs that have `validate` installed under sdk/*/dist/:
uv run pytest test_tdf_validity.py --sdks "go js" -k TestCorpus     # static corpus, no platform
uv run pytest test_tdf_validity.py --sdks "go js"                   # plus round trips (platform)

What I ran locally:

ruff check, ruff format --check: clean. pyright: 0 errors.
The full check.yml offline list, including test_tdf_validity_units.py (125 tests): 479 passed.
CI's pinned shellcheck v0.8.0 and shfmt v3 (via docker) on the three shims: clean.
The live static suite against the three real CLIs, built from #4254, #1119 and java-sdk#428 and installed as go@pr4254, js@pr1119 and java@pr428. pytest test_tdf_validity.py --sdks "go@pr4254 js@pr1119 java@pr428" -k TestCorpus -n 6: 711 passed, every cell match. That run covers the shims, supports tdf_validity detection, SDK.validate, judging with reason layers, the conftest report hook and tdf-validity.json. Against a deliberately wrong baseline, the drift and reason failures fired as intended.
The go shim's go install fallback against released otdfctl v0.38.1: supports tdf_validity says no, and validate reports an error.
PR CI: all 12 xct jobs are green, and scriptcheck and SonarCloud pass. In every installed build (go/java/js, released and main), the validity cells skip with "doesn't yet support [tdf_validity]": 1,470 skipped in the standard step, and 18 skipped in the new PQ step.

Not run: the platform round trips (test_roundtrip*). They need a platform and KAS, and the CLIs need to be in the builds xtest installs. They collect correctly (--collect-only). Until #4254, #1119 and java-sdk#428 land and reach those builds, every cell in CI skips through the tdf_validity gate, so this PR turns nothing red. To try a CLI branch sooner, use otdf-sdk-mgr install tip --ref ... plus XT_FORCE_SUPPORTS=tdf_validity.

Known gaps

The baseline comes from unmerged CLI branches. If they change before release, the first CI run against them will flag it.
xtest/manifest.schema.json (the strict schema test_tdfs.py checks writer output against) lists only wrapped, remote and ec-wrapped key access types, with no hybrid-wrapped or mlkem-wrapped. This PR doesn't change it.
No >4 GiB ZIP64 round trip. ZIP64 structure is covered by archive/zip64 and by every web-sdk-written file.
Baseline cells are keyed by SDK name, not version. A writer-specific round-trip answer can be pinned as "go:java".

Epic DSPX-5348; divergence tickets DSPX-4388, DSPX-5353, DSPX-5354, DSPX-5355, DSPX-5356, DSPX-5357; container profile DSPX-4924; schema tests DSPX-4369.
feat(cli): add otdfctl validate command (DSPX-5350) platform#4254 (otdfctl validate), feat(sdk): validate TDF manifests using the Go lax schema web-sdk#1104 / #1117 / #1118 / #1119, feat(xtest): cross-SDK structural ZIP conformance tests (DSPX-4591 follow-up) #624 (ZIP conformance), fix(xtest)!: read the manifest entry as manifest.json, per spec #600.
When this lands, web-sdk's in-repo parity corpus and check-go.go can shrink to a smoke test (DSPX-5358).

coderabbitai · 2026-10-23T14:32:50Z

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

Ask every SDK's `validate` command the same question about the same bytes, and hold each answer to one checked-in baseline. - tdfs.py: a `tdf_validity` feature, `SDK.validate()` and `run_bounded()`, which runs the shim in its own process group and kills the group on a timeout, so a hang is reported rather than wedging the run. - cli.sh (go, java, js): `validate <file>` passes the CLI's stdout and exit status through unchanged; `supports tdf_validity` looks for the command in the CLI's help, so released CLIs skip. Contract: one JSON line, exit 0 valid, 3 invalid, anything else could not run. - tdfvalidity_corpus.py: 236 static cases built in Python -- the 42 manifest cases and the archive cases from web-sdk#1104, Go's IsValidTdf fixtures, truncations at every structural boundary, malformed archives, the DSPX-5348 divergences, container-profile questions, and one-keyword-at-a-time lax-schema cases that kill every non-equivalent schema mutant. - tdfvalidity.py: the strict contract parser, the baseline, judging (including the reason's layer, where an SDK reports one), the tdf-validity.json report, and a refresh command. - tdfvalidity_baseline.json: go and js seeded from the libraries their CLIs wrap (platform/sdk v0.35.0 IsValidTdf; web-sdk#1119 validateTdf); every cell that differs from the agreed answer names its ticket. java is pending until DSPX-5352 ships a validate command. - test_tdf_validity.py: the static corpus (no platform) and round trips for every encrypt SDK x validator, including X-Wing, secp+ML-KEM and ML-KEM (a known divergence until DSPX-4388). - test_tdf_validity_units.py: offline tests of all of the above, including the shims against stand-in CLIs. - spec/tdf-validity.md: what "valid" means, the contract, the classification and how to refresh the baseline. Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>

- xtest.yml: test_tdf_validity.py beside test_tdfs.py and test_policytypes.py in both standard-xtest steps; a step after the extra KAS instances start for the PQ round trips (marker tdf_validity_pqc), with its own report file; upload test-results/**/*.json so tdf-validity*.json survives the run. - check.yml: run test_tdf_validity_units.py with the other offline harness tests. Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>

github-actions · 2026-10-23T14:33:16Z

X-Test Failure Report

All three validate CLIs now exist as PRs, so build and run them rather than only the libraries behind them: - java: seeded by running `java -jar cmdline.jar validate`, built from java-sdk#428 (8e9221d), over the corpus with the refresh command. Its divergences carry tickets: STORED and size checks (DSPX-4924), a stored size that runs into the central directory (DSPX-5103), and mlkem-wrapped (DSPX-4388). Its duplicate-name rejections sit at the archive layer. - go and js: otdfctl from platform#4254 (e33f3cb) and @opentdf/ctl from web-sdk#1119 (d293e509) match every cell of the existing baseline, reasons included. - java's shim asks the jar `supports tdf_validity`, as it does for dpop. - A json/number-10001-digits case: java rejects it, go and js do not; RFC 8259 allows the limit, so it is pinned as unspecified. - Keep the start of a reason, not the end: that is where each SDK says which layer rejected the file. Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>

github-actions · 2026-10-23T14:51:48Z

X-Test Failure Report

✅ js@v0.22.0-main
✅ java@main-main
✅ java@main-v0.29.0
✅ js@v0.22.0-v0.29.0
✅ java@v0.19.1-v0.29.0
✅ go@v0.39.0-v0.29.0
✅ js@main-v0.29.0

SonarCloud flags random.Random as a weak PRNG (python:S2245), which failed the security rating on new code. The corpus only needs deterministic, random-looking bytes, so expand SHA-256 in counter mode instead. The payload and zip/random bytes change; all three validate CLIs still match every cell of the baseline. Also use [[ ]] in the new validate branches of the go and java shims (shelldre:S7688). Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>

github-actions · 2026-10-23T14:56:26Z

X-Test Failure Report

github-actions · 2026-10-23T15:19:46Z

X-Test Results

✅ java@v0.19.1-main
✅ js@main-main
✅ java@main-v0.29.0
✅ go@v0.39.0-v0.29.0
✅ js@main-v0.29.0
✅ go@v0.39.0-main
✅ java@main-main
✅ js@v0.22.0-v0.29.0
✅ java@v0.19.1-v0.29.0
✅ js@v0.22.0-main
✅ go@main-v0.29.0
✅ go@main-main

…simplify it (DSPX-5349) Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>

…ps (DSPX-5349) Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>

sonarqubecloud · 2026-10-23T20:18:36Z

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

github-actions · 2026-10-23T20:40:07Z

X-Test Results

✅ java@main-main
✅ java@v0.19.1-main
✅ js@main-v0.29.0
✅ go@main-v0.29.0
✅ java@v0.19.1-v0.29.0
✅ js@main-main
✅ java@main-v0.29.0
✅ go@v0.39.0-v0.29.0
✅ go@main-main
✅ go@v0.39.0-main

dmihalcik-virtru added 2 commits October 23, 2026 10:32

dmihalcik-virtru force-pushed the DSPX-5349-xtest-tdf-validity-suite branch from 26bc6b4 to eadaa65 Compare October 23, 2026 14:32

dmihalcik-virtru added 2 commits October 23, 2026 14:22

refactor(xtest): move the TDF validity suite into xtest/validity and …

2f54e87

…simplify it (DSPX-5349) Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>

fix(xtest): tighten the validity contract, baseline checks and CI ste…

2225899

…ps (DSPX-5349) Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>

Navigation Menu

Repository navigation

feat(xtest): cross-SDK TDF validity suite (DSPX-5349) - #677

feat(xtest): cross-SDK TDF validity suite (DSPX-5349)#677
dmihalcik-virtru wants to merge 6 commits into
mainfrom
DSPX-5349-xtest-tdf-validity-suite

dmihalcik-virtru commented Oct 23, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Oct 23, 2026

Draft PR not reviewed

Uh oh!

github-actions Bot commented Oct 23, 2026

Uh oh!

github-actions Bot commented Oct 23, 2026

Uh oh!

github-actions Bot commented Oct 23, 2026

Uh oh!

github-actions Bot commented Oct 23, 2026

Uh oh!

sonarqubecloud Bot commented Oct 23, 2026

Uh oh!

github-actions Bot commented Oct 23, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

dmihalcik-virtru commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What and why

CLI contract (all three SDKs)

What's here

Baseline: how it was seeded

Base branch

How to test

Known gaps

Related

Uh oh!

coderabbitai Bot commented Oct 23, 2026

Draft PR not reviewed

Uh oh!

github-actions Bot commented Oct 23, 2026

X-Test Failure Report

Uh oh!

github-actions Bot commented Oct 23, 2026

X-Test Failure Report

Uh oh!

github-actions Bot commented Oct 23, 2026

X-Test Failure Report

Uh oh!

github-actions Bot commented Oct 23, 2026

X-Test Results

Uh oh!

sonarqubecloud Bot commented Oct 23, 2026

Quality Gate passed

Uh oh!

github-actions Bot commented Oct 23, 2026

X-Test Results

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

dmihalcik-virtru commented Oct 23, 2026 •

edited

Loading