Skip to content

Exit the oplog populator when the metastore stream is invalidated - #2863

Merged
bert-e merged 2 commits into
development/9.6from
bugfix/BB-897-oplog-populator-invalidate
Oct 1, 2026
Merged

bert-e merged 2 commits into
development/9.6from
bugfix/BB-897-oplog-populator-invalidate

Conversation

@francoisferrand

Copy link
Copy Markdown
Contributor

Dropping __metastore (as a D/R sink bootstrap does) sends drop then invalidate, after which the change stream is closed for good. Since BB-879 these events are just skipped, so the oplog populator stays up and healthy but no longer sees bucket changes: buckets created or restored afterwards are never listened to, and lifecycle, replication and notifications silently stop for them. This is what breaks the Zenko PRA e2e test with 9.6.0-preview.1 (scality/Zenko#2535).

The oplog populator now exits on drop, rename, dropDatabase and invalidate. The restart goes through setup() again: it re-lists the buckets, opens a new stream and stops listening to buckets that are gone. That's what the pre-BB-879 crash used to do, now on purpose and with a clear log line.

MongoConfigManager and LocationStatusStream have the same blind spot after a drop; that will be handled in a separate ticket.

Issue: BB-897

@bert-e

bert-e commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

Hello francoisferrand,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options
name description privileged authored
/after_pull_request Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval Bypass the pull request author's approval ⭐
/bypass_build_status Bypass the build and test status ⭐
/bypass_commit_size Bypass the check on the size of the changeset TBA ⭐
/bypass_incompatible_branch Bypass the check on the source branch prefix ⭐
/bypass_jira_check Bypass the Jira issue check ⭐
/bypass_peer_approval Bypass the pull request peers' approval ⭐
/bypass_leader_approval Bypass the pull request leaders' approval ⭐
/bypass_source_branch_lineage Bypass the cross-branch contamination check ⭐
/approve Instruct Bert-E that the author has approved the pull request. ✍️
/create_pull_requests Allow the creation of integration pull requests.
/create_integration_branches Allow the creation of integration branches.
/no_octopus Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity Change review acceptance criteria from one reviewer at least to all reviewers
/wait Instruct Bert-E not to run until further notice.
Available commands
name description privileged
/help Print Bert-E's manual in the pull request.
/status Print Bert-E's current status in the pull request.
/clear Remove all comments from Bert-E from the history TBA
/retry Re-start a fresh build TBA
/build Re-start a fresh build TBA
/force_reset Delete integration branches & pull requests, and restart merge process from the beginning.
/reset Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

@bert-e

bert-e commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

@codecov

codecov Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.01%. Comparing base (b132204) to head (073f2a1).
⚠️ Report is 2 commits behind head on development/9.6.

Additional details and impacted files

Impacted file tree graph

Files with missing lines Coverage Δ
extensions/oplogPopulator/OplogPopulator.js 89.47% <100.00%> (+0.24%) ⬆️

... and 1 file with indirect coverage changes

Components Coverage Δ
Bucket Notification 80.25% <ø> (ø)
Core Library 84.26% <ø> (+0.01%) ⬆️
Ingestion 78.21% <ø> (ø)
Lifecycle 81.65% <ø> (ø)
Oplog Populator 85.90% <100.00%> (+0.07%) ⬆️
Replication 63.88% <ø> (ø)
Bucket Scanner 85.76% <ø> (ø)
@@                 Coverage Diff                 @@
##           development/9.6    #2863      +/-   ##
===================================================
+ Coverage            77.00%   77.01%   +0.01%     
===================================================
  Files                  211      211              
  Lines                14559    14562       +3     
===================================================
+ Hits                 11211    11215       +4     
+ Misses                3338     3337       -1     
  Partials                10       10              
Flag Coverage Δ
api:retry 9.32% <0.00%> (-0.01%) ⬇️
api:routes 9.09% <0.00%> (-0.01%) ⬇️
bucket-scanner 85.76% <ø> (ø)
ft_test:queuepopulator 11.03% <0.00%> (-0.01%) ⬇️
ingestion 13.40% <0.00%> (-0.01%) ⬇️
lib 9.09% <0.00%> (+<0.01%) ⬆️
lifecycle 20.61% <0.00%> (-0.03%) ⬇️
notification 0.97% <0.00%> (-0.01%) ⬇️
oplogPopulator 0.13% <0.00%> (-0.01%) ⬇️
replication 18.97% <0.00%> (-0.02%) ⬇️
unit 57.93% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@bert-e

bert-e commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

@scality scality deleted a comment from bert-e Oct 1, 2026
Dropping __metastore (as a D/R sink bootstrap does) closes the change
stream for good. Since BB-879 these events are just skipped, so the
process stays up but no longer sees bucket changes: buckets created or
restored afterwards are never listened to, and lifecycle, replication
and notifications silently stop for them.

Exit on drop, rename, dropDatabase and invalidate instead, so that the
restart goes through setup() again: re-list the buckets, open a new
stream and stop listening to buckets that are gone. This is what the
crash used to do before BB-879, now on purpose.

Issue: BB-897
@francoisferrand
francoisferrand force-pushed the bugfix/BB-897-oplog-populator-invalidate branch from 8f7f752 to 073f2a1 Compare October 1, 2026 13:09
@francoisferrand

Copy link
Copy Markdown
Contributor Author

/approve

@bert-e
bert-e merged commit 073f2a1 into development/9.6 Oct 1, 2026
24 checks passed
@bert-e

bert-e commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

I have successfully merged the changeset of this pull request
into targetted development branches:

  • ✔️ development/9.6

The following branches have NOT changed:

  • development/7.10
  • development/7.4
  • development/7.70
  • development/8.6
  • development/9.0
  • development/9.1
  • development/9.2
  • development/9.3
  • development/9.4
  • development/9.5

This pull request did not target the following hotfix branch(es) so they
were left untouched:

  • hotfix/7.8.0
  • hotfix/7.10.17
  • hotfix/7.10.2
  • hotfix/7.70.1
  • hotfix/7.4.9
  • hotfix/7.4.3
  • hotfix/9.0.7
  • hotfix/8.2.12
  • hotfix/7.4.10
  • hotfix/7.4.4
  • hotfix/7.9.0
  • hotfix/7.10.1
  • hotfix/7.4.8
  • hotfix/7.10.12
  • hotfix/7.4.2
  • hotfix/7.10.4
  • hotfix/7.4.1
  • hotfix/7.4.5
  • hotfix/7.6.0
  • hotfix/7.2.0
  • hotfix/7.70.12
  • hotfix/7.10.3
  • hotfix/7.4.7
  • hotfix/7.4.0
  • hotfix/7.4.6
  • hotfix/7.10.0
  • hotfix/7.10.8
  • hotfix/7.70.15
  • hotfix/9.0.4
  • hotfix/7.7.0

Please check the status of the associated issue BB-897.

Goodbye francoisferrand.

The following options are set: approve

@bert-e
bert-e deleted the bugfix/BB-897-oplog-populator-invalidate branch October 1, 2026 13:21
@eve-ci-cd
eve-ci-cd Bot temporarily deployed to zenko/improvement/ZENKO-5370@2.16 October 1, 2026 14:07 Destroyed

This branch was successfully deployed

No deployments
zenko/improvement/ZENKO-5370@2.16 — 073f2a1b Deployed Oct 1, 2026 by eve-ci-cd[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants