Skip to content

Fix oplog populator alerts and panels that never fire - #2889

Merged
bert-e merged 2 commits into
development/9.6from
improvement/BB-906/fix-oplog-populator-alerts
Oct 21, 2026
Merged

bert-e merged 2 commits into
development/9.6from
improvement/BB-906/fix-oplog-populator-alerts

Conversation

@delthas

@delthas delthas commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

All preexisting issues, found when testing the oplog populator dashboard:

  • KafkaConnectFailedConnectors, KafkaConnectFailedConnectorConfiguration and OplogPopulatorMetastoreChangeStreamLagThreshold used 1m windows, which hold a single sample with the 60s scrape interval, so they could never fire. KafkaConnectFailedConnectorConfiguration also queried a metric that does not exist.
  • OplogPopulatorUnavailable fired on a single failed scrape: the up=0 sample stays for the 60s scrape interval, longer than its 30s for. It now waits 2m.
  • The instance stats kept showing their last value when no populator pod exists.

The last commit releases 9.6.0, which closes the hashed oplog ingestion stack.

Part of the hashed oplog ingestion stack, overview in #2869.

Issue: BB-906

@bert-e

bert-e commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

Hello delthas,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options
name description privileged authored
/after_pull_request Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval Bypass the pull request author's approval ⭐
/bypass_build_status Bypass the build and test status ⭐
/bypass_commit_size Bypass the check on the size of the changeset TBA ⭐
/bypass_incompatible_branch Bypass the check on the source branch prefix ⭐
/bypass_jira_check Bypass the Jira issue check ⭐
/bypass_peer_approval Bypass the pull request peers' approval ⭐
/bypass_leader_approval Bypass the pull request leaders' approval ⭐
/bypass_source_branch_lineage Bypass the cross-branch contamination check ⭐
/approve Instruct Bert-E that the author has approved the pull request. ✍️
/create_pull_requests Allow the creation of integration pull requests.
/create_integration_branches Allow the creation of integration branches.
/no_octopus Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity Change review acceptance criteria from one reviewer at least to all reviewers
/wait Instruct Bert-E not to run until further notice.
Available commands
name description privileged
/help Print Bert-E's manual in the pull request.
/status Print Bert-E's current status in the pull request.
/clear Remove all comments from Bert-E from the history TBA
/retry Re-start a fresh build TBA
/build Re-start a fresh build TBA
/force_reset Delete integration branches & pull requests, and restart merge process from the beginning.
/reset Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

@bert-e

bert-e commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

@codecov

codecov Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.93%. Comparing base (3971f39) to head (9ab331d).
⚠️ Report is 2 commits behind head on development/9.6.

Additional details and impacted files

Impacted file tree graph
see 6 files with indirect coverage changes

Components Coverage Δ
Bucket Notification 85.50% <ø> (ø)
Core Library 87.34% <ø> (+0.33%) ⬆️
Ingestion 83.38% <ø> (+0.02%) ⬆️
Lifecycle 86.24% <ø> (ø)
Oplog Populator 94.27% <ø> (ø)
Replication 68.61% <ø> (-0.03%) ⬇️
Bucket Scanner 89.37% <ø> (ø)
@@                 Coverage Diff                 @@
##           development/9.6    #2889      +/-   ##
===================================================
+ Coverage            81.80%   81.93%   +0.13%     
===================================================
  Files                  216      216              
  Lines                45313    45313              
===================================================
+ Hits                 37067    37127      +60     
+ Misses                8246     8186      -60     
Flag Coverage Δ
api:retry 14.26% <ø> (ø)
api:routes 14.28% <ø> (ø)
bucket-scanner 89.37% <ø> (ø)
ft_test:queuepopulator 17.86% <ø> (+0.15%) ⬆️
ingestion 20.10% <ø> (+<0.01%) ⬆️
lib 12.63% <ø> (ø)
lifecycle 26.18% <ø> (ø)
notification 1.85% <ø> (ø)
oplogPopulator 0.58% <ø> (ø)
replication 24.68% <ø> (+0.03%) ⬆️
unit 68.36% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@delthas
delthas force-pushed the improvement/BB-906/fix-oplog-populator-alerts branch from 63d0d53 to 953073a Compare October 16, 2026 08:31
@delthas
delthas changed the base branch from development/9.6 to improvement/BB-906/hashed-oplog-monitoring October 16, 2026 08:31
@delthas
delthas added this pull request to stack #2886 October 16, 2026 08:31
@delthas
delthas force-pushed the improvement/BB-906/fix-oplog-populator-alerts branch from 953073a to 34584ac Compare October 16, 2026 08:41
@delthas
delthas force-pushed the improvement/BB-906/fix-oplog-populator-alerts branch from 34584ac to 9c4d376 Compare October 16, 2026 10:47
@delthas
delthas force-pushed the improvement/BB-906/fix-oplog-populator-alerts branch from 9c4d376 to f299383 Compare October 16, 2026 12:19
@delthas
delthas force-pushed the improvement/BB-906/fix-oplog-populator-alerts branch from f299383 to b454c2e Compare October 16, 2026 12:54
@delthas
delthas force-pushed the improvement/BB-906/fix-oplog-populator-alerts branch from b454c2e to 55b29bb Compare October 16, 2026 13:04
@delthas
delthas force-pushed the improvement/BB-906/fix-oplog-populator-alerts branch from 55b29bb to 8916382 Compare October 16, 2026 14:48
Comment thread monitoring/oplog-populator/alerts.yaml Outdated
Comment thread monitoring/oplog-populator/alerts.yaml Outdated
@delthas
delthas force-pushed the improvement/BB-906/fix-oplog-populator-alerts branch 2 times, most recently from fc7e518 to a3f8931 Compare October 19, 2026 09:24
@delthas
delthas force-pushed the improvement/BB-906/fix-oplog-populator-alerts branch from a3f8931 to f92143b Compare October 19, 2026 10:29
@delthas
delthas requested a review from benzekrimaha October 19, 2026 10:55
@delthas

delthas commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

Requesting @benzekrimaha since Francois Ferrand is on PTO

@delthas
delthas force-pushed the improvement/BB-906/fix-oplog-populator-alerts branch from a15cbaf to c0f6861 Compare October 20, 2026 14:14
Comment thread package.json
@delthas
delthas force-pushed the improvement/BB-906/fix-oplog-populator-alerts branch from c0f6861 to 6115700 Compare October 20, 2026 14:44
@delthas
delthas removed this pull request from stack #2886 October 20, 2026 14:45
@delthas
delthas added this pull request to stack #2895 October 20, 2026 14:45
@delthas
delthas force-pushed the improvement/BB-906/fix-oplog-populator-alerts branch from 6115700 to 38d40a3 Compare October 20, 2026 16:49
@delthas
delthas force-pushed the improvement/BB-906/fix-oplog-populator-alerts branch from 38d40a3 to 7d4c0f0 Compare October 21, 2026 09:24
@delthas
delthas force-pushed the improvement/BB-906/fix-oplog-populator-alerts branch from 7d4c0f0 to 578914d Compare October 21, 2026 13:10
Found while exercising the dashboard on a dev ARTESCA:

- KafkaConnectFailedConnectors, KafkaConnectFailedConnectorConfiguration
  and OplogPopulatorMetastoreChangeStreamLagThreshold used 1m windows,
  which hold a single sample with the 60s scrape interval: rate and
  increase were always empty. KafkaConnectFailedConnectorConfiguration
  also queried a metric that does not exist.
- OplogPopulatorUnavailable fired on a single failed scrape: the up=0
  sample remains for the 60s scrape interval, longer than its 30s for.
  The alert tests now evaluate every 30s, as deployed.
- The instance stats kept showing their last value when no pod exists.

Issue: BB-906
@delthas
delthas force-pushed the improvement/BB-906/fix-oplog-populator-alerts branch from 578914d to 9ab331d Compare October 21, 2026 15:39
@delthas

delthas commented Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

/approve

@delthas

delthas commented Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

!done 5d

@delthas

delthas commented Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

!release

Base automatically changed from improvement/BB-906/hashed-oplog-monitoring to development/9.6 October 21, 2026 21:58
@bert-e

bert-e commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

I have successfully merged the changeset of this pull request
into targetted development branches:

  • ✔️ development/9.6

The following branches have NOT changed:

  • development/7.10
  • development/7.4
  • development/7.70
  • development/8.6
  • development/9.0
  • development/9.1
  • development/9.2
  • development/9.3
  • development/9.4
  • development/9.5

This pull request did not target the following hotfix branch(es) so they
were left untouched:

  • hotfix/7.2.0
  • hotfix/7.6.0
  • hotfix/7.10.3
  • hotfix/7.4.1
  • hotfix/7.10.0
  • hotfix/7.10.2
  • hotfix/9.0.4
  • hotfix/7.10.1
  • hotfix/7.4.10
  • hotfix/7.7.0
  • hotfix/7.4.7
  • hotfix/7.10.12
  • hotfix/7.10.17
  • hotfix/8.2.12
  • hotfix/7.4.6
  • hotfix/7.4.9
  • hotfix/9.0.7
  • hotfix/7.70.12
  • hotfix/7.70.15
  • hotfix/7.4.5
  • hotfix/7.9.0
  • hotfix/7.8.0
  • hotfix/7.4.4
  • hotfix/7.4.8
  • hotfix/7.4.3
  • hotfix/7.4.0
  • hotfix/7.10.4
  • hotfix/7.4.2
  • hotfix/7.10.8
  • hotfix/7.70.1

Please check the status of the associated issue BB-906.

Goodbye delthas.

The following options are set: approve

@bert-e
bert-e merged commit 9ab331d into development/9.6 Oct 21, 2026
25 checks passed
@bert-e
bert-e deleted the improvement/BB-906/fix-oplog-populator-alerts branch October 21, 2026 21:58
@delthas

delthas commented Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

Release summary:

  • development/9.6: releasing 9.6.0

@delthas

delthas commented Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

BB-906: logged 5d; transitioned to Done (Fixed)

@eve-ci-cd
eve-ci-cd Bot deployed to zenko/improvement/ZENKO-5375/hashed-oplog-ingestion@2.16 October 21, 2026 22:10 Active
@eve-ci-cd
eve-ci-cd Bot deployed to zenko/development/2.16 October 22, 2026 02:30 Active
@eve-ci-cd
eve-ci-cd Bot deployed to zenko/2.16.0 October 22, 2026 07:12 Active

This branch was successfully deployed

3 active deployments
zenko/2.16.0 — 9ab331d0 Deployed Oct 22, 2026 by eve-ci-cd[bot]
zenko/development/2.16 — 9ab331d0 Deployed Oct 22, 2026 by eve-ci-cd[bot]
zenko/improvement/ZENKO-5375/hashed-oplog-ingestion@2.16 — 9ab331d0 Deployed Oct 21, 2026 by eve-ci-cd[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants