Skip to content

feat: queue release catch-up for already-mapped credly templates - #5172

Merged
ulemons merged 4 commits into
mainfrom
feat/issue-975-credly-release-catchup
Nov 3, 2026
Merged

ulemons merged 4 commits into
mainfrom
feat/issue-975-credly-release-catchup

Conversation

@ulemons

@ulemons ulemons commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Why

The 2026-11-02 go-live mapped 216 Credly badge templates (anchor c3fe78ce) before the per-template release queue (#5166/#5167) existed in deployed code. applyDecisions only queues a release on a new held/excluded/new→mapped transition — these 216 never went through that path, so none of them has a row in credlyTemplateReleases, and their pre-go-live badge history is never recovered by the live poll.

Refs linuxfoundation/lfx-cdp-insights-ops#975

What

  • DAL: listMappedCredlyTemplatesWithoutRelease — mapped templates with no release row, ordered by badgeCount ascending.
  • Resolve logic: unreleasedTemplatesToQueue — same floor rule as a normal release (releaseFloor), applied to the catch-up batch.
  • Script: new --release-unreleased --integration-id <uuid> mode on resolve-credly-templates. Loads the Credly listing (for created_at), reads the unreleased batch, queues them all lightest-first — so the ~200 light templates drain ahead of the dozen high-volume ones (CKA, CKAD, …) in the single-release-at-a-time FIFO walker.
  • Fixed a stale doc note in sys-cdp.md still saying Credly wasn't registered — it's been live in prod since today.

Tests

  • unreleasedTemplatesToQueue: floors from the template's own createdAt, falls back to history start when missing from the listing.
  • listMappedCredlyTemplatesWithoutRelease: lightest-first ordering, drops a template once its release is queued. DAL tests need the local Postgres on :5434 (Docker), not run locally here — same as the rest of this suite; verified by CI.
  • pnpm lint:fix, pnpm format, pnpm lint --deny-warnings --format=agent, pnpm tsc-check all clean.

Manually verified today (before this PR)

Queued and ran the release walk for one already-mapped template (CKAD) live in staging: resumed correctly across the page cap, no errors, real badges/activities emitted. This PR reuses that same mechanism for the other 215.

Rollout

  1. Test --release-unreleased in staging against the full batch.
  2. Run in prod once staging looks right (needs explicit go-ahead — it queues 216 rows at once).

🤖 Generated with Claude Code

The 2026-11-02 go-live mapped 216 templates before the release queue
(#5166/#5167) existed, so none of them have a row in
credlyTemplateReleases and their pre-go-live history is never
recovered by the live poll.

Adds --release-unreleased --integration-id <uuid> to the resolve
script: lists every mapped template missing a release and queues them
all, lightest by badgeCount first, so the long tail drains ahead of
the dozen high-volume ones in the single-release-at-a-time FIFO
walker.

Refs linuxfoundation/lfx-cdp-insights-ops#975

Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
Copilot AI balanced review requested due to automatic review settings November 2, 2026 17:44
@ulemons ulemons self-assigned this Nov 2, 2026
@cursor

cursor Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

PR Summary

Medium Risk
One-time operational mode can enqueue a large batch of release rows in production; mistakes affect Credly history backfill load and queue ordering, though it reuses the existing release mechanism.

Overview
Adds a one-time backfill path for Credly badge templates that were mapped before the per-template release queue existed, so their historical badges can still be walked by the existing FIFO release worker.

Data layer: New listMappedCredlyTemplatesWithoutRelease returns live, routable mapped templates with no row in credlyTemplateReleases, ordered lightest first (badgeCount ascending, nulls first). Deactivated mappings and deleted integrations are excluded.

Resolve helper: unreleasedTemplatesToQueue builds release inputs using the same releaseFloor rules as normal resolve (createdAt from the Credly listing, otherwise CREDLY_HISTORY_START).

CLI: resolve-credly-templates gains --release-unreleased --integration-id <uuid>, which loads the Credly template listing, loads the unreleased batch, and enqueues releases in that lightest-first order. HTTP client setup is extracted to credlyHttpClient for reuse.

Tests cover the new query ordering/filtering and floor fallback behavior.

Reviewed by Cursor Bugbot for commit 7eb1af4. Bugbot is set up for automated code reviews on this repo. Configure here.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical issues allow unroutable releases to be completed without fetching badges and concurrent catch-up runs to overwrite existing release bounds and saved progress.

1 open issue
What changed in this PR

Adds a catch-up path for Credly templates mapped before the release queue existed, reusing the existing release walker to recover historical badges.

Changes:

  • Selects mapped templates without release records and queues them lightest-first.
  • Adds --release-unreleased --integration-id <uuid>, with creation-date floors and tests for selection, ordering, and fallback behavior.
  • Corrects documentation to reflect Credly’s production registration.
File Description
services/​libs/​data-access-layer/​src/​credly/​releases.ts Adds the unreleased mapped-template query.
services/​libs/​data-access-layer/​src/​credly/​releases.test.ts Tests ordering and exclusion after queuing.
services/​apps/​connectors_worker/​src/​credlyTemplateResolve.ts Builds catch-up releases using existing floor rules.
services/​apps/​connectors_worker/​src/​credlyTemplateResolve.test.ts Tests creation-date floors and missing-template fallback.
services/​apps/​connectors_worker/​src/​bin/​resolve-credly-templates.ts Adds bulk catch-up mode and shares HTTP client setup.
docs/​knowledge/​sys-cdp.md Corrects the Credly connector’s registration status.

Review effort: Balanced

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit ab3b500. Configure here.

listMappedCredlyTemplatesWithoutRelease only checked status = 'mapped',
unlike listActiveCredlyTemplateMappings. A deactivated mapping, or one
whose integration was deleted, got queued anyway; the release walker
finds it unmapped in routing and completes it with nothing fetched,
burning the one release a future re-mapping would have queued.

Applies the same live-and-routable predicate used elsewhere: not
deactivated, integration not deleted, segment matches.

Found independently by Copilot and Cursor Bugbot on PR review.

Refs linuxfoundation/lfx-cdp-insights-ops#975

Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
Copilot AI balanced review requested due to automatic review settings November 2, 2026 18:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The catch-up loop can race with another invocation and reset an existing release’s cursor and completion state, so it needs an insert-only operation that leaves existing releases untouched.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings November 3, 2026 08:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The catch-up loop can overwrite a concurrently queued template’s release state, potentially discarding active progress; it needs insert-if-absent behavior while preserving explicit re-queue restart semantics.

No open issues 🎉

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings November 3, 2026 08:34

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because overlapping catch-up or manual release runs can reset existing release progress despite the snapshot-based missing-release check.

No open issues 🎉

Review effort: Balanced

Out of scope for this PR.

Refs linuxfoundation/lfx-cdp-insights-ops#975

Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
Refs linuxfoundation/lfx-cdp-insights-ops#975

Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
@ulemons
ulemons force-pushed the feat/issue-975-credly-release-catchup branch from 331ab80 to 7eb1af4 Compare November 3, 2026 08:52
@ulemons
ulemons merged commit a1df472 into main Nov 3, 2026
13 checks passed
@ulemons
ulemons deleted the feat/issue-975-credly-release-catchup branch November 3, 2026 09:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants