Skip to content

feat: PostgreSQL integration tests and the work queue core (#277, #278) - #289

Merged
bitbiter-dev merged 7 commits into
masterfrom
test/277-pg-integration-tests
Oct 19, 2026
Merged

bitbiter-dev merged 7 commits into
masterfrom
test/277-pg-integration-tests

Conversation

@bitbiter-dev

Copy link
Copy Markdown
Owner

Closes #277. Closes #278. Part of #276; #279, #280 and #281 build on it.

What changes

PostgreSQL integration tests (#277). A new Anichron.Tests.Integration project runs one postgres:17-alpine container per test run through Testcontainers, and gives each test its own database, either empty or with the real migrations applied. A test keeps the image in step with src/docker-compose.yml. Its first callers are code that only real PostgreSQL can exercise and that had no tests:

  • DeleteExpiredAsync;
  • MigrateWithAdvisoryLockAsync: two concurrent migrators both finish, the lock is released afterwards, and with the lock held elsewhere the migrator gives up without migrating.

Test suites by name. Directory.Build.props classifies projects ending in .Tests.Unit and .Tests.Integration. A TestSuite property (Unit by default, Integration or All) decides which of them are test projects. Plain dotnet test and Stryker therefore never run the integration suite.

  • Directory.Build.targets sets IsTestProject for projects in the selected suite.
  • For projects outside it, it also sets IsTestingPlatformApplication=false. Microsoft.NET.Test.Sdk and xUnit v3 set both properties to true after Directory.Build.props, and Stryker treats IsTestingPlatformApplication=true as a test project whatever IsTestProject says. Before this fix the scoped Stryker run found 1,303 tests, the 6 integration tests included; now it finds only the unit tests.
  • A new CI job, Integration tests, runs dotnet test src/ -p:TestSuite=Integration and is added to the gate's needs.

Work queue core (#278). A hand-rolled, durable queue in WorkItems (ADR 0008 records why not a library, citing #273). Refresh-token cleanup is its first caller.

  • Inserting: IWorkQueue.EnqueueAsync (EfWorkQueue, Core) is the only insert. A payload with a DedupKey coalesces: while an item with that key is Queued, enqueueing again returns that item. A partial unique index absorbs racing inserts.
  • Store: EfWorkItemStore (Worker).
    • A claim takes the oldest queued item of a kind this Worker knows, using FOR UPDATE SKIP LOCKED, and increments Attempt.
    • Heartbeat, succeed, fail and release are all fenced by (Id, Attempt), so an attempt that lost its lease changes nothing.
    • Lease recovery counts a stall each time; the third stall fails the item as stalled.
    • An item handed back while another with its key is queued ends Cancelled as superseded.
  • Executor: runs one item at a time under a lease of 60 s, with a heartbeat every 2 s that carries the progress snapshot.
    • A fenced heartbeat cancels the handler, and nothing more is written.
    • A shutdown releases the item without counting a stall.
    • A handler exception fails the item with its error; there is no retry.
  • Scheduler: every Worker polls every 5 s. It recovers expired leases, then enqueues each periodic kind that is due: nothing queued or running with its key, and nothing finished within its interval. No host keeps timer state, so overdue work runs right after downtime.
  • Kinds: TokenCleanup replaces TokenCleanupService and its timer. WorkItemRetention deletes items that finished more than 30 days ago. WorkHandlerRegistry dispatches by kind through registrations made in DI, never a switch. It fails startup unless every WorkKind has exactly one payload type in Core and exactly one handler for it.
  • Settings: Worker:WorkPollIntervalSeconds, WorkHeartbeatSeconds, WorkLeaseSeconds (must be longer than the heartbeat), WorkMaxStalls, WorkItemRetentionDays and WorkItemRetentionIntervalHours, with spec: Work queue — durable API/Worker work items with scope routing, leases and a live admin view #276's defaults.
  • Architecture rules, each shown failing on a deliberate violation:
    rule 'only-work-queue-inserts-work-items' is broken by these types ... {"Anichron.Worker.ViolationInsert"}
    rule 'work-handlers-live-in-worker' is broken by these types ... {"Anichron.API.ViolationHandler"}
    

Decisions made with the maintainer

Deliberately not here

Verification

  • dotnet test -p:TestSuite=All after merging master: 1,416 unit and 31 integration tests, all passing.
  • Each contract test was shown failing with the code it guards broken:
    • the Attempt fence removed;
    • SKIP LOCKED removed (the claim then blocks until the command timeout);
    • the advisory lock bypassed, or its release made a no-op;
    • < changed to <= in the expired-token delete;
    • the store as it was before the review fixes.
  • Mutation testing: one scripts/mutation-changed.sh run over every changed production file reached 100.00%. The PostgreSQL-only members carry // Stryker disable all, with ADR 0005 rows.
    • The full local sweep did not finish. Both attempts stalled in per-test coverage capture while another sweep ran on the same machine, at a load average near 100. The scoped runs on this code logged no coverage timeouts, so CI's mutation job is the first full run.
  • mutation-guard.sh, derive-db-docs.sh --check and dotnet format --verify-no-changes pass.

🤖 Generated with Claude Code

bitbiter-dev and others added 7 commits October 18, 2026 20:47
Adds Anichron.Tests.Integration, which starts one postgres:17-alpine
container per run and gives each test its own database, migrated or
empty. First callers: EfRefreshTokenRepository.DeleteExpiredAsync and
MigrateWithAdvisoryLockAsync (two concurrent migrators, lock released,
lock held elsewhere), both PostgreSQL-only and untested until now.

Test projects are classified by name (*.Tests.Unit, *.Tests.Integration)
and a TestSuite property (Unit by default, Integration, All) selects which
are test projects. Directory.Build.targets sets IsTestProject and, for
projects outside the suite, IsTestingPlatformApplication=false: the test
SDK and xUnit v3 set both to true after Directory.Build.props, and Stryker
treats IsTestingPlatformApplication=true as a test project regardless of
IsTestProject. Verified: Stryker's scoped run finds 1297 tests (unit only)
instead of 1303.

CI gets an "Integration tests" job, added to the gate's needs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A pooled check connection could reuse the migrator's own session, where
an advisory lock is re-entrant, so the lock-released test could pass with
the release broken. Shown failing with ReleaseAsync made a no-op.

CLAUDE.md: a project or filter run of integration tests needs
-p:TestSuite=Integration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The durable work queue's data layer (#276):
- WorkItems, with the claim, queued-dedup, lease and UpdatedAt indexes,
  table and column comments, and its migration.
- IWorkPayload (static abstract kind, opt-in dedup key), IWorkHandler,
  IWorkProgress and WorkJson (System.Text.Json with NodaTime) in Core.
- EfWorkQueue, the only insert path. A queued item with the same key is
  returned instead of a second; a lost race on the partial unique index
  is absorbed and looked up again.
- EfWorkItemStore in the Worker: claim over FOR UPDATE SKIP LOCKED,
  heartbeat, succeed, fail and release fenced by (Id, Attempt), lease
  recovery with the stall limit, due-ness and the retention delete.

StorageConfigId and CancelRequestedAt land with #281 and #280.

Contract tests on real PostgreSQL cover concurrent claims, skip-locked,
zombie attempts, the stall limit, release without a stall and dedup
coalescing. Each was shown failing with its guard removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…up on the queue (#278)

- WorkHandlerRegistry dispatches by kind through registrations built
  in DI, never a switch, and fails startup unless every WorkKind has
  exactly one payload type in Core and exactly one handler, for that
  payload.
- WorkExecutor claims one item at a time (oldest first) and runs it
  under a lease, heartbeating every 2 s with the progress snapshot.
  A fenced heartbeat cancels the handler and writes nothing more; a
  shutdown releases the item without a stall; a handler exception,
  including a cancellation that is neither, fails it with its error.
- WorkScheduler recovers expired leases and enqueues each periodic
  kind that is due, from the table: nothing queued or running with its
  key and nothing finished within its interval.
- TokenCleanup and WorkItemRetention as kinds with handlers and
  schedules. TokenCleanupService and its timer are gone.
- Settings: poll 5 s, heartbeat 2 s, lease 60 s (must exceed the
  heartbeat), 3 stalls, retention 30 days, retention purge 24 h.
- The Worker registers AnichronDbContext through a context factory, so
  the store can open a short-lived context per call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- ADR 0008: a hand-rolled PostgreSQL work queue over a job library,
  with the evaluation from #273.
- Architecture rules only-work-queue-inserts-work-items (only
  EfWorkQueue constructs a WorkItem) and work-handlers-live-in-worker.
  Shown failing on deliberate violations:
    rule 'only-work-queue-inserts-work-items' ... {"Anichron.Worker.ViolationInsert"}
    rule 'work-handlers-live-in-worker' ... {"Anichron.API.ViolationHandler"}
- CLAUDE.md describes the work queue; the Worker no longer has a token
  cleanup timer. The RefreshTokens table comment follows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#278)

Review findings on the store:

- Enqueue coalesces only queued items, so an item can be queued while
  another with its key runs. Releasing or recovering the running one
  then hit the queued-dedup unique index; recovery was one bulk update,
  so a single collision stopped all lease recovery and scheduling.
  Items are now handed back one at a time: requeued when no item with
  the key is queued, otherwise Cancelled as superseded, since the queued
  one does the same work. A racing insert is retried. Recovery re-checks
  the lease per item, so it never requeues one whose owner just beat.
- The claim parsed the kind after claiming the row, so a kind from a
  newer release was left Running until it failed as stalled. The claim
  now takes only the kinds this Worker knows; others stay queued.

Each new contract test fails against the previous store.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bitbiter-dev
bitbiter-dev merged commit b5cc5a5 into master Oct 19, 2026
12 checks passed
@bitbiter-dev
bitbiter-dev deleted the test/277-pg-integration-tests branch October 19, 2026 00:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant