[delthas]

Hand-written description (took me a while). I tried to do much more hand-written and rountrip work with AI. Let me know if it helps :)

This implements the oplog scaling design for the oplog populator (see https://github.com/scality/citadel/pull/375 for details).

The issue with the previous design was that:

• Buckets without lifecycle/... rules weren't exported, but we need this for direct-to-cold
• The number of connectors grew linearly with the number of buckets, which does not work if we have too many buckets

Instead, we listen to all buckets all the time, splitting load across N connectors, with a simple k = hash(bucket) % N.

This is implemented as an opt-in ingestion mode: zenko-operator enables it through a ZenkoVersion feature flag (https://github.com/scality/zenko-operator/pull/656), so each Zenko release decides when to switch. The goal is to drop the previous mode entirely in the short term. (That is why I chose not to have an abstraction over possible modes etc. The goal is to have only one mode left, soon, and delete legacy code.)

This is part of a stack of 8 PRs: the lower ones add the building blocks and the manager, this one wires it in, #2888 adds monitoring, and #2889 fixes existing oplog populator alerts. Review bottom-up.

The general idea is:

• I add a new ingestion mode, hashed, opt-in
• We "reconcile" connectors every 15 seconds, taking care of
  • if a new install, adding our connectors
  • ensuring our connectors are ready/not failed
  • if a migration, properly stopping previous connectors, creating ours, then deleting previous ones. The migration is a bit tricky but properly explained in the code.

The connector name is:

• a prefix (to ignore anything that's not ours at all),
• then N and k
• and a hash such that we can easily spot outdated pipelines

Issue: BB-906

[bert-e]

Hello delthas,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
/after_pull_request	Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval	Bypass the pull request author's approval	⭐
/bypass_build_status	Bypass the build and test status	⭐
/bypass_commit_size	Bypass the check on the size of the changeset TBA	⭐
/bypass_incompatible_branch	Bypass the check on the source branch prefix	⭐
/bypass_jira_check	Bypass the Jira issue check	⭐
/bypass_peer_approval	Bypass the pull request peers' approval	⭐
/bypass_leader_approval	Bypass the pull request leaders' approval	⭐
/bypass_source_branch_lineage	Bypass the cross-branch contamination check	⭐
/approve	Instruct Bert-E that the author has approved the pull request.		✍️
/create_pull_requests	Allow the creation of integration pull requests.
/create_integration_branches	Allow the creation of integration branches.
/no_octopus	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity	Change review acceptance criteria from one reviewer at least to all reviewers
/wait	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
/help	Print Bert-E's manual in the pull request.
/status	Print Bert-E's current status in the pull request.
/clear	Remove all comments from Bert-E from the history TBA
/retry	Re-start a fresh build TBA
/build	Re-start a fresh build TBA
/force_reset	Delete integration branches & pull requests, and restart merge process from the beginning.
/reset	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

[codecov]

Codecov Report

❌ Patch coverage is 93.05556% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.94%. Comparing base (36f5aba) to head (91a9570).
⚠️ Report is 4 commits behind head on development/9.6.

Files with missing lines	Patch %	Lines
extensions/oplogPopulator/OplogPopulator.js	91.66%	5 Missing ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
...ns/oplogPopulator/OplogPopulatorConfigValidator.js	100.00% <100.00%> (ø)
extensions/oplogPopulator/OplogPopulator.js	95.79% <91.66%> (+2.70%)	⬆️

... and 6 files with indirect coverage changes

Components	Coverage Δ
Bucket Notification	85.50% <ø> (ø)
Core Library	87.35% <ø> (+0.29%)	⬆️
Ingestion	83.38% <ø> (ø)
Lifecycle	86.24% <ø> (ø)
Oplog Populator	94.27% <93.05%> (+0.42%)	⬆️
Replication	68.67% <ø> (+0.05%)	⬆️
Bucket Scanner	89.37% <ø> (ø)

@@                 Coverage Diff                 @@
##           development/9.6    #2869      +/-   ##
===================================================
+ Coverage            81.77%   81.94%   +0.17%     
===================================================
  Files                  216      216              
  Lines                45247    45313      +66     
===================================================
+ Hits                 37000    37134     +134     
+ Misses                8247     8179      -68     

Flag	Coverage Δ
api:retry	14.26% <16.66%> (+<0.01%)	⬆️
api:routes	14.28% <16.66%> (+<0.01%)	⬆️
bucket-scanner	89.37% <ø> (ø)
ft_test:queuepopulator	18.05% <16.66%> (+1.04%)	⬆️
ingestion	20.10% <16.66%> (-0.01%)	⬇️
lib	12.63% <16.66%> (+<0.01%)	⬆️
lifecycle	26.20% <16.66%> (-0.02%)	⬇️
notification	1.85% <0.00%> (-0.01%)	⬇️
oplogPopulator	0.58% <0.00%> (-0.01%)	⬇️
replication	24.69% <16.66%> (-0.03%)	⬇️
unit	68.36% <93.05%> (+0.06%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[bert-e]

Waiting for approval

The following approvals are needed before I can proceed with the merge:

• the author

• 2 peers

[bert-e]

Waiting for approval

The following approvals are needed before I can proceed with the merge:

• the author

• 2 peers

[DarkIsDude]

LGTM, check how to manage older version of Mongo but fine otherwise

[delthas]

/approve

[bert-e]

I have successfully merged the changeset of this pull request
into targetted development branches:

• ✔️ development/9.6

The following branches have NOT changed:

• development/7.10
• development/7.4
• development/7.70
• development/8.6
• development/9.0
• development/9.1
• development/9.2
• development/9.3
• development/9.4
• development/9.5

This pull request did not target the following hotfix branch(es) so they
were left untouched:

• hotfix/7.2.0
• hotfix/7.6.0
• hotfix/7.10.3
• hotfix/7.4.1
• hotfix/7.10.0
• hotfix/7.10.2
• hotfix/9.0.4
• hotfix/7.10.1
• hotfix/7.4.10
• hotfix/7.7.0
• hotfix/7.4.7
• hotfix/7.10.12
• hotfix/7.10.17
• hotfix/8.2.12
• hotfix/7.4.6
• hotfix/7.4.9
• hotfix/9.0.7
• hotfix/7.70.12
• hotfix/7.70.15
• hotfix/7.4.5
• hotfix/7.9.0
• hotfix/7.8.0
• hotfix/7.4.4
• hotfix/7.4.8
• hotfix/7.4.3
• hotfix/7.4.0
• hotfix/7.10.4
• hotfix/7.4.2
• hotfix/7.10.8
• hotfix/7.70.1

Please check the status of the associated issue BB-906.

Goodbye delthas.

The following options are set: approve