Skip to content

fix(evaluation): isolate custom simulation template rendering - #714

Merged
notgitika merged 1 commit into
aws:mainfrom
notgitika:fix/isolate-custom-simulation-templates
Oct 26, 2026
Merged

notgitika merged 1 commit into
aws:mainfrom
notgitika:fix/isolate-custom-simulation-templates

Conversation

@notgitika

Copy link
Copy Markdown
Contributor

Summary

  • Render custom simulation prompt strings with a dedicated Jinja2 environment.
  • Keep bundled prompt files on the existing file-backed environment.
  • Require Jinja2 3.1.6 for the simulation extra.

Testing

  • pytest tests/bedrock_agentcore/evaluation/runner/test_simulated_scenario_executor.py -q — 62 passed
  • ruff check and ruff format --check — passed
  • uv lock --check — passed

Render custom simulation prompt strings with a dedicated Jinja2 environment while keeping bundled templates on the existing file-backed environment. Require Jinja2 3.1.6 for custom template rendering.
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Oct 26, 2026
@notgitika

Copy link
Copy Markdown
Contributor Author

CI investigation summary:

  • The PR code checks are healthy: Python 3.10–3.13 tests, lint, packaging, dependency checks, Bandit, secret scanning, and CodeBuild all pass.
  • The nine Secure Integration Test jobs never reached test execution. They fail during checkout because actions/checkout@v6 refuses to check out fork code from a pull_request_target workflow that runs in a trusted context.
  • The breaking-change analysis completed successfully and found no public API break. That job fails afterward with a 403 while attempting to post its result using the restricted fork-PR token.

These failures are caused by the PR originating from a fork, rather than by the proposed code changes. The safest immediate resolution is to move the branch into the aws repository and recreate the PR from that branch. Enabling unsafe fork checkout is not appropriate for this workflow because it loads credentials before running the integration tests.

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 83.33333% with 3 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@969d9eb). Learn more about missing BASE report.

Files with missing lines Patch % Lines
...ck_agentcore/evaluation/runner/prompts/__init__.py 83.33% 3 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #714   +/-   ##
=======================================
  Coverage        ?   89.68%           
=======================================
  Files           ?      125           
  Lines           ?    10885           
  Branches        ?     1694           
=======================================
  Hits            ?     9762           
  Misses          ?      735           
  Partials        ?      388           
Flag Coverage Δ
unittests 89.68% <83.33%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@notgitika
notgitika merged commit 813ab45 into aws:main Oct 26, 2026
32 of 42 checks passed

This branch had an error being deployed

1 failed deployment
auto-approve — 197dca79 Deployed Oct 26, 2026 by notgitika via Test (runtime) #1660
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/s PR size: S

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants