Repository navigation
fix(pii): honor abort in quick mode, accept any regex resource, download it per job - #346
Merged
Merged
Conversation
…oad it per job Three PII-screening problems found in local review: - With pii_quick_screen, pii_show_candidates and pii_found_abort all on, found PII neither aborted nor produced a preview, so ingestion went on despite the abort setting. Quick screening stops at the first match and has no candidate rows to preview, so pii_show_candidates can't apply: abort. When quick screening finds PII and the job proceeds (abort off), log a warning; that case was silent too. - A custom regex resource had to have a DataStore table, though its file is downloaded, so a plain uploaded regex file was rejected. Look it up with resource_show instead; a missing resource is still a clear JobError. - The custom regexes were downloaded to user-pii-regexes.txt in the package directory, shared by every job: concurrent jobs could overwrite each other's copy mid-scan, and the package directory may not be writable. Download into the job's temp_dir. The config declarations now say how pii_show_candidates interacts with quick screening and what the regex resource must be. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #343. A local review of its commits found three more PII-screening problems that were already on
main. I confirmed each one against the code.pii_quick_screen,pii_show_candidatesandpii_found_abortall on, found PII matched neither the abort branch (which required previews to be off) nor the preview branch (which required quick screening to be off). Ingestion carried on unless the review gate stopped it.pii_show_candidatescan't apply. It now aborts.resource_show. A missing resource is still a clearJobError.user-pii-regexes.txtin the package directory. Concurrent jobs could overwrite each other's copy mid-scan, and the package directory may not be writable. They now go in the job's owntemp_dir.The
pii_show_candidatesandpii_regex_resource_id_or_aliasentries inconfig_declaration.yamlnow explain these behaviors.Tests (
tests/test_pii_screening_search.py, run against the real qsv):resource_show, never with the DataStore check, and downloaded into the job's temp dir, not the package dir.pii_show_candidateson.(True, 1).resource_showraisingObjectNotFound.Four mutations each made the new tests fail: the old abort condition, no warning, requiring a DataStore table again, and downloading into the package dir. The unit suite passes in
dpp-test(342 passed).🤖 Generated with Claude Code