Skip to content

feat(evidence): retain opt-in test execution receipts and dependency observations - #358

Open
dinakarjs wants to merge 8 commits into
agentrust-io:mainfrom
dinakarjs:feat/execution-evidence-receipts
Open

dinakarjs wants to merge 8 commits into
agentrust-io:mainfrom
dinakarjs:feat/execution-evidence-receipts

Conversation

@dinakarjs

@dinakarjs dinakarjs commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Ready for independent review — 31 October 2026

Readiness assessed at c6e976e0dfe1d4813e545c960df049f61969e832: current metadata is mergeable; all returned PR-triggered technical workflows succeeded, and the public discussion contains no explicit unresolved draft hold or code-change request. Changed-file scope and current CONTRIBUTING requirements were checked. This is a readiness assessment, not a new full source audit, fresh suite run, independent approval or complete branch-protection inventory.

Opt-in unsigned execution receipts and consistency checks only; no attestation, complete dependency discovery or automatic tier promotion.

The earlier draft-for-review wording below is historical. Independent maintainer review and normal merge gates remain required; no merge or release is authorized by this transition. Earlier test counts apply to their stated revisions; obsolete baseline CI failures below are superseded by the successful current-head runs. Prepared with OpenAI Codex.


Current closure refresh - 30 October 2026

Published head: c6e976e. Tested tree: a4e87b4700ef7acd3d8d469e853ae84c65c8a3d4. Current-main history is merged without rewriting earlier contributors' commits; DCO trailers are present on this refresh commit.

Root tests: 69 passed; all 50 integration manifests validate and generated Marketplace catalog is current. This does not establish independently operated receipt custody, complete dependency observation or live acceptance.

The published tree equals the locally tested tree. Whitespace checks on the contribution delta against current main pass (existing CRLF documentation is preserved). Hosted current-head checks and independent maintainer acceptance remain separate; no merge, production deployment or hardware result is claimed. Earlier measurements below describe their original revisions. Prepared with OpenAI Codex.


Compatibility-floor validation currently checks declarations, not execution evidence. This adds an opt-in Python command recorder that retains the source revision/cleanliness, manifest digest, declared and installed package versions, timestamps, command outcome, evidence class, Actions run URL and log digest.

A separate checker rejects missing, stale, unsuccessful or inconsistent evidence. It does not promote community/verified tiers or imply conformance. Receipts are unsigned observations, not adversarial attestations; command coverage and provenance still require review.

The existing LangChain interop command is wrapped and its receipt/log uploaded for 30 days even on failure. Receipt consistency is now a required check for this opt-in job. The first run exposed a stale LangChain declaration, so tested_against was updated from TRACE 0.9.0 to the observed 0.11.0; its tier is unchanged. No vendor code is newly discovered or executed, and no compatibility floors or tier declarations change.

Validation: nine recorder tests cover actual subprocess success/failure/timeout, log substitution, missing evidence, changed manifests and version/source mismatches. Recorder and compatibility tests: 11 passed. Default-rule Ruff passes on both new Python files. The normal validation workflow now runs recorder regression tests.

See docs/test-evidence.md for same-interpreter use, retention, sensitive-log handling and evidence limitations. AI-assisted implementation; DCO sign-offs included.

Historical implementation validation

Full repository validation test suite: 69 passed; manifest validator: 48 integrations, zero failures. At head 044ca19, validation, lint, workflow lint and trace-adapters CI passed. LangChain receipt and log retained at https://github.com/agentrust-io/integrations/actions/runs/39668849253/artifacts/12378777164 . The consistency report detected installed versions differing from the existing declaration.

Historical baseline failures at those earlier revisions included Nobulex HTTP 451 and WCM/OpenSSL imports. Current-head links and example workflows now succeed; those are not current blockers.

Earlier head d213b84 includes the corrected declaration and required consistency gate. Validation, lint, workflow lint, CodeQL and trace-adapters CI passed. The retained receipt reports consistent with no errors: https://github.com/agentrust-io/integrations/actions/runs/39669274550/artifacts/12378772781 . The earlier baseline blockers are superseded by successful technical workflows on current head c6e976e. Independent maintainer acceptance remains pending.

Signed-off-by: Srinivasa Consultants <dinakarjs@gmail.com>
Signed-off-by: Srinivasa Consultants <dinakarjs@gmail.com>
Signed-off-by: Srinivasa Consultants <dinakarjs@gmail.com>
Signed-off-by: Srinivasa Consultants <dinakarjs@gmail.com>
Signed-off-by: Srinivasa Consultants <dinakarjs@gmail.com>
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

🟡 Contributor Check: MEDIUM

Check Result
Profile MEDIUM
Credential LOW
Overall MEDIUM

Automated check by AgenTrust Contributor Check.

@github-actions github-actions Bot added the needs-review:MEDIUM Contributor check flagged MEDIUM risk label Oct 21, 2026
…ce receipt

Signed-off-by: Srinivasa Consultants <dinakarjs@gmail.com>
…ce receipt

Signed-off-by: Srinivasa Consultants <dinakarjs@gmail.com>
Preserve contribution history and evidence boundaries; validate the final merged tree.

Signed-off-by: Srinivasa Dinakar Jainti <dinakarjs@gmail.com>
Signed-off-by: Srinivasa Consultants <78673153+dinakarjs@users.noreply.github.com>
@dinakarjs
dinakarjs marked this pull request as ready for review October 31, 2026 03:43

Copy link
Copy Markdown
Contributor Author

@carloshvp Please review the opt-in execution-receipt recorder, consistency checker and LangChain workflow integration at current head c6e976e0dfe1d4813e545c960df049f61969e832. This PR is now ready for independent review, and the existing reviewer requests are populated. All returned current-head technical workflows succeed (validation evidence); the PR description records the prior local tests and their revisions.

Unsigned observations only; no adversarial attestation, complete dependency discovery or automatic tier promotion.

Please identify any remaining change required or disposition this bounded contribution. This request does not claim independent approval, a new full-suite rerun, or merge readiness. Prepared with OpenAI Codex.

@rajnisht7 rajnisht7 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the overall changes looks good, a few things needs to be looked on:

  • Docs overclaim - docs/test-evidence.md says undeclared dependencies fail the check. They don’t. The recorder only observes packages listed in tested_against. kindly fix the wording to match the actual behaviour.
  • Command is not bound to the log - Validator only checks that command is a non-empty list. one can change the command in receipt.json and keep the same log; check still returns consistent. it would be nice to bind the command (e.g. command_sha256 or include it in a hashed evidence payload).
  • Secrets in the command - full command argv is stored in receipt.json and then uploaded as an artifact. Docs only warn about logs. Please document that secrets must never be passed as command-line arguments.
  • Incomplete dependency observation for LangChain - Receipt only records agentrust-trace. The interop test actually runs against langchain-core==1.6.3 and langgraph==1.2.12. Schema currently forbids putting those in tested_against, so either document clearly that framework versions are out of scope for this receipt, or give the recorder an explicit way to observe extra packages.

also: tested_against is treated as a version floor elsewhere in the repo, but this checker requires exact equality. Worth being explicit about that.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-review:MEDIUM Contributor check flagged MEDIUM risk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants