Skip to content

fix(pii): honor abort in quick mode, accept any regex resource, download it per job - #346

Merged
jqnatividad merged 1 commit into
mainfrom
fix-pii-abort-resource-tempdir
Oct 8, 2026
Merged

jqnatividad merged 1 commit into
mainfrom
fix-pii-abort-resource-tempdir

Conversation

@jqnatividad

Copy link
Copy Markdown
Collaborator

Follow-up to #343. A local review of its commits found three more PII-screening problems that were already on main. I confirmed each one against the code.

  1. The abort setting was ignored in quick mode when previews were on. With pii_quick_screen, pii_show_candidates and pii_found_abort all on, found PII matched neither the abort branch (which required previews to be off) nor the preview branch (which required quick screening to be off). Ingestion carried on unless the review gate stopped it.
    • Quick screening stops at the first match and has no candidate rows to preview, so pii_show_candidates can't apply. It now aborts.
    • When quick screening finds PII and abort is off, it now logs a warning. That case was silent before.
  2. A custom regex resource had to have a DataStore table, though its file is then downloaded. A plain uploaded regex file was rejected (since fix(pii): make PII screening work with its default regexes #343, with a clear error; before that, with a crash). The resource is now looked up with resource_show. A missing resource is still a clear JobError.
  3. Every job shared one downloaded regex file. Custom regexes were downloaded to user-pii-regexes.txt in the package directory. Concurrent jobs could overwrite each other's copy mid-scan, and the package directory may not be writable. They now go in the job's own temp_dir.

The pii_show_candidates and pii_regex_resource_id_or_alias entries in config_declaration.yaml now explain these behaviors.

Tests (tests/test_pii_screening_search.py, run against the real qsv):

  • A custom regex resource works without a DataStore table, in both modes. The test checks it's fetched with resource_show, never with the DataStore check, and downloaded into the job's temp dir, not the package dir.
  • Quick mode aborts on PII even with pii_show_candidates on.
  • Quick mode without abort warns and returns (True, 1).
  • The missing-resource test now mocks resource_show raising ObjectNotFound.

Four mutations each made the new tests fail: the old abort condition, no warning, requiring a DataStore table again, and downloading into the package dir. The unit suite passes in dpp-test (342 passed).

🤖 Generated with Claude Code

…oad it per job

Three PII-screening problems found in local review:

- With pii_quick_screen, pii_show_candidates and pii_found_abort all on,
  found PII neither aborted nor produced a preview, so ingestion went on
  despite the abort setting. Quick screening stops at the first match and
  has no candidate rows to preview, so pii_show_candidates can't apply:
  abort. When quick screening finds PII and the job proceeds (abort off),
  log a warning; that case was silent too.
- A custom regex resource had to have a DataStore table, though its file
  is downloaded, so a plain uploaded regex file was rejected. Look it up
  with resource_show instead; a missing resource is still a clear
  JobError.
- The custom regexes were downloaded to user-pii-regexes.txt in the
  package directory, shared by every job: concurrent jobs could overwrite
  each other's copy mid-scan, and the package directory may not be
  writable. Download into the job's temp_dir.

The config declarations now say how pii_show_candidates interacts with
quick screening and what the regex resource must be.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jqnatividad
jqnatividad merged commit fa70966 into main Oct 8, 2026
2 checks passed
@jqnatividad
jqnatividad deleted the fix-pii-abort-resource-tempdir branch October 8, 2026 23:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant