Skip to content

FEAT: Explore datasets and use seeds in chat and custom scenarios #2744

Open
8 of 17 issues completed
Open

Description

@romanlutz

Is your feature request related to a problem? Please describe.

People using CoPyRIT need to discover stored seeds, inspect their text/media, optionally load built-in datasets, and use examples without manual copy/paste. One logical example may be a prompt, a multimodal message or a whole stored conversation. Several independent examples belong in a scenario, not concatenated Chat.

The catalog, paginated browsing, media previews, draft preservation and three single-example import destinations are shipped. #3670 merged and completed #2751. This revision focuses remaining work on missing integrations rather than replacement frameworks.

Describe the solution you'd like

Remaining journeys

Shared contracts

  • Keep one Datasets surface and catalog/paging/media APIs, one Scanner configuration/launch flow, and normal scenario progress/history. No second dataset store, batch executor, generated scenario source or memory migration.
  • The catalog already exposes availability, stored origins, counts and opaque selection keys. Add missing provider descriptions/access facts, not duplicate contracts. Stored counts are not provider estimates; unknown is not zero, and unlabeled harm is not safe.
  • Navigation/browsing may make ordinary read-only memory/catalog requests. They must not download providers, generate content, create attacks, send prompts or execute tools merely because a page/callout opened.
  • Loading is explicit, uses backend-configured credentials and writes to shared deployment memory. A configured token is not guaranteed gated access. No browser token entry, credential store, per-user provider authentication or automatic acceptance of terms.
  • Reuse preparation and persistence. Ordinary Load is missing-only and non-destructive, never an implicit always refresh or regeneration. Bounded runtime-local status must handle duplicate requests, failure/partial work, navigation and restart honestly.
  • Default-provider metadata does not describe every subset. VLGuard variants, for example, share vlguard; name presence cannot prove that another variant is loaded. General variant support needs a separate identity/coexistence decision.
  • Preserve selection_key plus example_id, complete membership, source IDs/hashes, roles/order, objective conditions, media and provenance. Capture canonical membership/content freshness, not IDs alone. Missing/changed examples require refresh, not replacement or truncation.
  • Exact selections must not be widened by names/filters, reduced by caps, regenerated, or silently skipped/adapted away. Preview, estimate and confirmed launch use the same effective inputs.
  • Retain source objectives; ask for missing objectives explicitly. No invented goals or unscored mode. Reuse configured source-pinned objective_scorer: RegisteredComponentReference (registry_name and identifier_hash). Shared scorer/core policy owns defaults, condition coverage and result meaning. True/false family or Chat eligibility alone is insufficient; raw refusal=true is not attack success.
  • Whole conversations remain whole examples. Chat imports reuse composer/editor splitting and simulated history roles, respect sent-evidence restrictions, confirm dirty-draft/setup conflicts and never send automatically. Unmaterialized generation configurations and ambiguous shapes remain inspectable with explanations, not flattened or executed on import.
  • Preserve drafts, setup, filters/cursors/selection and return context. Keep bounded paging and lazy media without autoplay, and durable media references distinct from expiring previews.

Technical dependencies and release priority

Native dependencies represent technical prerequisites, not the contributor release queue:

On November 4, 2026, the maintainer explicitly approved #2754 and then #2757 in parallel with #2752 because their technical prerequisites are complete. These three issues may carry help wanted. This is a scoped parallel release, not permission to promote every technically unblocked item. Later releases still require an explicit maintainer decision; all other open children remain gated.

Order Work item
1 #2746 Memory-backed summaries (completed in #2762)
2 #2748 Paginated seed browsing (completed in #2989)
3 #2747 Navigation and cards (completed in #3071)
4 #2745 Prompt table/details (completed in #3163)
5 #2749 Shared media previews (completed in #3251)
6 #2750 Preserve Chat drafts (completed in #3334)
7 #2751 Import prompts/conversations (completed in #3670)
8 #2752 Inspiration and current Chat-journey coverage (assigned to Rohitpro1)
9 #2754 Default-provider descriptions/discovery (promoted in parallel with #2752 and #2757)
10 #2753 Standalone provider Load operation
11 #2756 Load/status/access UI and provider journeys
12 #2755 Scorer API (completed in #2990; former independent track)
13 #2757 Exact-selection request/resolution contract (available in parallel with #2752 and #2754)
14 #2760 Selected-example scenario adapter
15 #2759 Selection into existing Scanner and scenario journeys
16 #2761 Final integrated documentation/cross-flow checkpoint
17 #2758 Optional per-technique converter composition

Contributor readiness

#2752 remains promoted and is assigned to Rohitpro1. #2754 and #2757 are also promoted; contributors should check their live assignees and claim threads before starting. All three can proceed in parallel under the explicit November 4 approvals. All other open children remain not ready yet until separately promoted. This parent remains assigned to Roman Lutz (@romanlutz). Completed issues should have no readiness labels.

The former scorer hold is superseded: #2755 and related #2970 completed in #2990. #2834 closed unmerged, not as delivery. Do not restore that hold or the former #2757 -> #2756 sequencing dependency.

Describe alternatives you've considered, if relevant

Do not rebuild shipped catalogs, scorer APIs, technique editors, provider lifecycle or scenario execution. No batch Chat, persistent copied selection datasets, arbitrary Python/URLs/loader kwargs or unscored execution. General provider variants, generator controls and destructive-refresh UI are optional follow-ups, not hidden prerequisites.

Additional context

Scope approved November 4, 2026 after reviewing main c756e7f8f2b8e24987907ec4691ec0c54a0b0677. All 17 children are retained; the final checkpoint now precedes optional converter work.

Read doc/code/framework.md and applicable instructions. Providers produce typed seeds into memory; scenarios select/package them; techniques configure attacks; scorers evaluate; GUI/backend adapt those contracts.

Reuse #2748/#2989 browsing, #3143 source/count controls, #2956/#3116 preparation/providers and #3206 explicit scenario refresh. The scenario dataset plan is complementary. Stored generated objectives are ordinary data, not permission to regenerate.

Configured scorer discovery/references, core condition/default policy, ChatScorerPicker in Scenario mode, ScenarioDatasetTable, TechniqueEditor and shared estimate/launch controls already exist. Coordinate with open #3526/#2653; do not assume that proposed form decomposition is merged or block independent request-contract work on it.

Each milestone ships its tests and documentation. Real-backend Playwright uses controlled local providers/media/targets/scorers; component tests may mock APIs. No privileged Hugging Face access, real secrets or paid calls. Scheduled monitoring stays read-only and never promotes automatically.

Activity

  1. added
    datasetsPulling in external datasets into PyRIT
    GUIUmbrella label for all feedback submitted via the Co-PyRIT GUI
    on Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

GUIUmbrella label for all feedback submitted via the Co-PyRIT GUIdatasetsPulling in external datasets into PyRITfeature-request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions