Skip to content

Harden incoming archival, overflow recovery, and retention - #419

Merged
TheWitness merged 1 commit into
developfrom
fix/issue-317-pipeline-resilience
Oct 16, 2026
Merged

TheWitness merged 1 commit into
developfrom
fix/issue-317-pipeline-resilience

Conversation

@bmfmancini

Copy link
Copy Markdown
Member

Summary

Incoming cleanup could delete unarchived events, local archival could duplicate events after interruption, and partition recovery and retention could perform unsafe or ineffective maintenance after downtime.

  • Preserve incoming events regardless of age, retry crash-stranded rows, and acknowledge only identities stored in history or the durable remote outbox. Rows missing host/program references remain queued.
  • Make transfers transactional, lock overlapping incoming slices, and skip existing source identities on retry.
  • Defer automatic overflow reorganization above 100,000 estimated rows or 128 MiB, or when size metadata is unavailable. Report the reason through existing maintenance health and preserve the safety partition.
  • Expire partitions using actual UTC upper boundaries rather than partition count. Support epoch and legacy TO_DAYS layouts and retain partitions that span the cutoff.
  • Add unit/database regression coverage, run the database checks in CI, and document recovery behavior.

Partially addresses #317; other design and hardening items remain open. This branch contains one new commit on current develop, excluding the history already squash-merged through #418.

Validation

  • 503 Pest tests passed on the PR branch.
  • 100% measured changed-line coverage (87/87 lines).
  • 44 checks passed against disposable MariaDB while implementing this commit, including connection interruption, commit/deletion failure, overlapping transfer attempts, Aria retries, populated overflow deferral, and sparse retention layouts.
  • PHP syntax, whitespace, and translation-template checks passed.

Limits

Full atomic recovery across database/host crashes requires InnoDB; Aria retries are deduplicated. Overflow statistics are estimates and do not impose an ALTER execution-time limit. Mixed-age partitions retain older rows until the entire partition expires. Existing duplicate history is not removed. No deployment was performed.

Copilot AI balanced review requested due to automatic review settings October 16, 2026 21:12

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@sonarqubecloud

sonarqubecloud Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
C Security Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@TheWitness
TheWitness merged commit a66c493 into develop Oct 16, 2026
7 of 8 checks passed
@TheWitness
TheWitness deleted the fix/issue-317-pipeline-resilience branch October 20, 2026 17:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants