Skip to content

FEAT: Resolve exact dataset example selections for scenarios #2757

Open

Description

@romanlutz

Is your feature request related to a problem? Please describe.

Chosen examples must remain those exact logical examples through preview, estimate and launch. Named-source/filter/cap requests cannot encode that intent, and IDs alone cannot detect changed membership/content.

Part of #2744. Technical prerequisites: completed #2748 browsing identity and #2755 scorer API. #2756 is not a technical dependency. On November 4, 2026, the maintainer explicitly promoted this issue in parallel with #2752 and #2754. It is available for a contributor to claim; check the live assignee and claim thread before starting. Provider loading is not a prerequisite. Later work remains gated until separately promoted.

Describe the solution you'd like

Add the missing bounded typed exact-selection contract and shared resolver. Retain API-provided selection_key plus example_id; group IDs can span datasets, so never widen to another dataset's members.

Define canonical selection freshness from complete membership and relevant stored content/configuration, not IDs or previews alone. A server-provided membership/content fingerprint or equivalent validated manifest must detect removed/added members and changes to values, roles/sequence, media references and objective conditions. Resolve canonical records from memory, never arbitrary browser paths/content.

Exact mode is mutually exclusive with source/filter/sampling/fetch overrides that alter its population. Do not apply default five-item caps, resample, create a persistent selection dataset or regenerate missing records. Invalid/duplicate/stale/missing/unsupported selections need explicit errors, not dropped or replaced rows.

Reuse estimate/run models and objective_scorer: RegisteredComponentReference with registry_name and identifier_hash. Keep stale-instance checks. This issue owns selected inputs/resolution, not a new picker/default policy/compatibility table. #2760 and shared scorer/core policy own selected-population execution compatibility. Retain explicit objective inputs for examples that need them.

Acceptance criteria:

  • N dataset-scoped references resolve to N complete examples with roles/order, source IDs/hashes, typed conditions, media and provenance.
  • Same-ID changed membership/content, cross-dataset group IDs, duplicates/missing references and conflicting modes are tested.
  • More than five selected examples are not capped; a supported request bound errors rather than truncates.
  • Preview/estimate/launch use the same effective selection/freshness and exact explicit scorer. Stale references require refresh or an explicit new choice.
  • Resolution/estimates are read-only: no provider calls, generation, writes, legacy template-file loading or premature rendering.
  • Existing named-dataset/scorer behavior remains compatible; focused resolver/contract tests use portable existing memory APIs, no migrations.

Describe alternatives you've considered, if relevant

No copied persistent dataset, entire-dataset browser transfer, serialized live Python objects, name-only scorer field or DTO-specific scorer rules. #2760 is the later consumer, not a prerequisite for completing this contract.

Additional context

Start with catalog scenario models, ScenarioConfigurationResolver, SeedRecord, dataset-scoped browse reads and inline DatasetAttackConfiguration groups. Inspection records are not already execution-ready components; preserve configuration as configuration and validate supported shapes at setup.

Scorer source resolution, defaults and condition helpers already exist. Refresh from #3206 must invalidate stale selections, not replace them. Reuse normal scenario/result persistence. Scope narrowed and technical dependencies corrected November 4, 2026.

Activity

  1. added
    not ready yetThis issue needs more definition or is blocked by a pending change.
    GUIUmbrella label for all feedback submitted via the Co-PyRIT GUI
    on Sep 22, 2026
  2. changed the title [-]FEAT GUI: Accept exact seed selections in scenario requests[/-] [+]FEAT: Resolve exact dataset example selections for scenarios[/+] on Nov 4, 2026
  3. added
    help wantedExtra attention is needed
    and removed
    not ready yetThis issue needs more definition or is blocked by a pending change.
    on Nov 5, 2026
  4. github-actions commented on Nov 5, 2026

    @github-actions
    Contributor

    Thanks for the feedback! A maintainer will take a look soon.

  5. juliosuas commented on Nov 5, 2026

    @juliosuas
    Contributor

    I'd like to claim this.

    Plan: implement the bounded typed exact-selection contract and shared resolver so chosen dataset examples stay those exact logical examples through preview, estimate, and launch (dataset-scoped selection_key + example_id, freshness from membership/content, mutual exclusion with source/filter/sampling overrides, explicit errors for invalid/stale/duplicate selections, no default five-item cap).

    I'll follow the acceptance criteria in the issue and reuse existing catalog scenario models / ScenarioConfigurationResolver / memory APIs. Please assign me if that works — happy to post a short scope note before coding if preferred.

  6. juliosuas commented on Nov 8, 2026

    @juliosuas
    Contributor

    Plan before coding, split into two PRs: (1) backend contract + shared resolver + fingerprint, (2) frontend request/preview consistency, no new picker. A few questions to make sure the boundary is right:

    1. Launch: should FEAT: Resolve exact dataset example selections for scenarios #2757 already replace the requested scenario's population with the resolved examples (sampling off), leaving prompt-sending-only and baseline suppression to FEAT: Run a scenario over exactly selected dataset examples #2760? Or should launch refuse exact selections until FEAT: Run a scenario over exactly selected dataset examples #2760 lands?
    2. Objectives: should the shared shape check require exactly one stored objective (so estimate and launch fail together), or should objective-less examples resolve and wait for FEAT: Run a scenario over exactly selected dataset examples #2760?
    3. Fingerprint: full SeedRecord content per member plus membership, or only value/value_sha256/role/sequence/media/conditions? I'd lean full record.
    4. Errors: 409 for stale, 400 for missing/duplicate/conflicting mode/unsupported, 422 left to pydantic. OK?
    5. Request bound: is the existing MAX_ITEMS (100) the supported limit for this list?
    6. FIXED / compound / no-override scenarios: explicit up-front "unsupported", or is the existing dataset-constraint error enough?
    7. Frontend split: shared request field + preview branch here, multi-select control and form integration in FEAT GUI: Configure a scenario from selected dataset examples #2759. Correct?
    8. Resume: a stale fingerprint on a saved run fails the resume, and FEAT: Add explicit scenario dataset refresh #3206 refresh never rewrites saved references. Correct?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

GUIUmbrella label for all feedback submitted via the Co-PyRIT GUIfeature-requesthelp wantedExtra attention is needed

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions