Skip to content

FEAT GUI: Load built-in datasets into memory on request #2753

Open

Description

@romanlutz

Is your feature request related to a problem? Please describe.

Explorer Load must work without a Scanner form or selected target. Scenario preparation already loads providers, but not through the standalone operation/status contract needed for safe navigation and clear errors.

Part of #2744. Depends on #2754's supported default-provider identity/descriptor contract. Keep not ready yet until promoted.

Describe the solution you'd like

Add a thin standalone service/API adapter over existing provider preparation and CentralMemory persistence. Accept a supported registered built-in identity using its default configuration. Do not instantiate/launch a scenario merely to load data or accept constructor kwargs, URLs/modules, model instructions or credentials from the browser.

Ordinary Load is missing-only: preserve existing stored data/manual edits. Name presence means data exists, not every possible variant. Report already-present/reused data honestly; retries must not become always refresh, replacement or regeneration.

Reuse runtime/service patterns for bounded process-local operation ID/status. Keep blocking provider work off the API event loop. Expose meaningful stages, terminal outcomes, duplicate protection and honest partial-persistence reporting. Status/duplicate scope follows the actual backend/runtime lifetime, not a promise of distributed exactly-once execution, durable recovery or unsupported cancellation.

Respect authorization and backend credentials. Distinguish missing configuration, gated denial, unsupported providers, upstream/network and persistence failures without secrets. Runtime change/restart must not attach stale results to a new backend or report lost status as success.

Acceptance criteria:

  • Public/gated fake providers use the shared preparation/persistence path; rows become readable through existing APIs.
  • GET/browsing never loads; no scenario execution or target selection is required.
  • Concurrent duplicates, already-present data and retries have documented/tested non-destructive behavior.
  • Long operations, partial failure, runtime replacement and restart/lost status are truthful and tested.
  • Authorization/input validation and errors do not expose credentials.
  • Service/API tests use fake providers/local media, no keys/network datasets; update related API documentation.

Describe alternatives you've considered, if relevant

No memory migration, durable queue, generic loader/code endpoint, browser credentials, configurable variants or automatic refresh. Do not redesign scenario preparation or repair all dataset loaders in this adapter.

Additional context

Reuse DatasetConfiguration.prepare_async, provider lookup, prepared-seed/insertion APIs and backend lifetime patterns. /api/scenarios/catalog/{name}/datasets/load and useScenarioDatasets are references, not a reason to construct a scenario.

Missing-only reuse shipped in #2956/#3116/#3143; explicit scenario refresh shipped in #3206 and is not this operation. #2756 owns explorer UI and real-backend provider journeys. Scope narrowed November 4, 2026.

Metadata

Metadata

Assignees

No one assigned

    Labels

    GUIUmbrella label for all feedback submitted via the Co-PyRIT GUIdatasetsPulling in external datasets into PyRITfeature-requestnot ready yetThis issue needs more definition or is blocked by a pending change.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions