Skip to content

Monitor hashed connectors - #2888

Merged
bert-e merged 1 commit into
development/9.6from
improvement/BB-906/hashed-oplog-monitoring
Oct 21, 2026
Merged

bert-e merged 1 commit into
development/9.6from
improvement/BB-906/hashed-oplog-monitoring

Conversation

@delthas

@delthas delthas commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Alerts on paused or stopped hashed connectors and on connectors not replaced within 30 minutes. KafkaConnectTooManyConnectors is bounded by the expected number of connectors with hashed ingestion, and keeps the soft limit with per-bucket ingestion.

The dashboard overview shows the connectors by state in both ingestion modes, with the expected number of hashed connectors, and the per-connector oplog lag reported by the mongo-kafka connector. The panels that stay empty with hashed ingestion (bucket counts and changes, acknowledgement lag) move to a collapsed row.

Part of the hashed oplog ingestion stack, overview in #2869.

Dashboard, during a switch from 1 to 2 connectors, two configuration drifts repaired, and a connector paused then resumed:

image

Issue: BB-906

@delthas
delthas added this pull request to stack #2886 October 15, 2026 15:35
@codecov

codecov Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.80%. Comparing base (91a9570) to head (3971f39).
⚠️ Report is 3 commits behind head on development/9.6.

Additional details and impacted files

Impacted file tree graph
see 7 files with indirect coverage changes

Components Coverage Δ
Bucket Notification 85.50% <ø> (ø)
Core Library 87.01% <ø> (-0.35%) ⬇️
Ingestion 83.35% <ø> (-0.03%) ⬇️
Lifecycle 86.24% <ø> (ø)
Oplog Populator 94.27% <ø> (ø)
Replication 68.63% <ø> (-0.04%) ⬇️
Bucket Scanner 89.37% <ø> (ø)
@@                 Coverage Diff                 @@
##           development/9.6    #2888      +/-   ##
===================================================
- Coverage            81.94%   81.80%   -0.15%     
===================================================
  Files                  216      216              
  Lines                45313    45313              
===================================================
- Hits                 37134    37067      -67     
- Misses                8179     8246      +67     
Flag Coverage Δ
api:retry 14.26% <ø> (ø)
api:routes 14.28% <ø> (ø)
bucket-scanner 89.37% <ø> (ø)
ft_test:queuepopulator 17.70% <ø> (-0.35%) ⬇️
ingestion 20.10% <ø> (-0.01%) ⬇️
lib 12.63% <ø> (ø)
lifecycle 26.18% <ø> (-0.02%) ⬇️
notification 1.85% <ø> (ø)
oplogPopulator 0.58% <ø> (ø)
replication 24.65% <ø> (-0.05%) ⬇️
unit 68.36% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@delthas
delthas force-pushed the improvement/BB-906/hashed-oplog-monitoring branch from e2f7ca6 to bc5188c Compare October 15, 2026 17:40
@delthas
delthas force-pushed the improvement/BB-906/hashed-oplog-monitoring branch 2 times, most recently from 50b7ff7 to 5c2bfe2 Compare October 15, 2026 18:18
@delthas
delthas force-pushed the improvement/BB-906/hashed-oplog-monitoring branch from 5c2bfe2 to ad3bdde Compare October 16, 2026 08:29
Comment thread monitoring/oplog-populator/dashboard.py
@delthas
delthas force-pushed the improvement/BB-906/hashed-oplog-monitoring branch from ad3bdde to 8ddee8b Compare October 16, 2026 08:41
Comment thread monitoring/oplog-populator/dashboard.py
Comment thread monitoring/oplog-populator/dashboard.py
@delthas
delthas force-pushed the improvement/BB-906/hashed-oplog-monitoring branch from 8ddee8b to 7386a4a Compare October 16, 2026 10:47
@delthas
delthas force-pushed the improvement/BB-906/hashed-oplog-monitoring branch from 7386a4a to 170e52d Compare October 16, 2026 12:19
@delthas
delthas force-pushed the improvement/BB-906/hashed-oplog-monitoring branch from 170e52d to b9702c0 Compare October 16, 2026 12:54
Comment thread monitoring/oplog-populator/dashboard.py
@delthas
delthas force-pushed the improvement/BB-906/hashed-oplog-monitoring branch from b9702c0 to a800477 Compare October 16, 2026 13:04
@delthas
delthas requested review from a team, francoisferrand and maeldonn October 16, 2026 13:17
@delthas
delthas force-pushed the improvement/BB-906/hashed-oplog-monitoring branch from a800477 to 7e72da3 Compare October 16, 2026 14:48
Comment thread monitoring/oplog-populator/alerts.yaml Outdated
Comment thread monitoring/oplog-populator/dashboard.py Outdated
Comment thread monitoring/oplog-populator/dashboard.py
Comment thread monitoring/oplog-populator/dashboard.py Outdated
Comment thread monitoring/oplog-populator/dashboard.py Outdated
Comment thread monitoring/oplog-populator/dashboard.py
Comment thread monitoring/oplog-populator/dashboard.py Outdated
Comment thread monitoring/oplog-populator/dashboard.py Outdated
Comment thread monitoring/oplog-populator/dashboard.py Outdated
Comment thread monitoring/oplog-populator/dashboard.py
@delthas
delthas force-pushed the improvement/BB-906/hashed-oplog-monitoring branch from 7e72da3 to c1df92b Compare October 19, 2026 09:06
@delthas
delthas force-pushed the improvement/BB-906/hashed-oplog-monitoring branch 2 times, most recently from 7b378c2 to bf3f61e Compare October 20, 2026 14:14
@delthas
delthas requested a review from benzekrimaha October 20, 2026 14:24
@delthas
delthas force-pushed the improvement/BB-906/hashed-oplog-monitoring branch from bf3f61e to 744f8aa Compare October 20, 2026 14:44
@delthas
delthas removed this pull request from stack #2886 October 20, 2026 14:45
@delthas
delthas added this pull request to stack #2895 October 20, 2026 14:45
@delthas
delthas force-pushed the improvement/BB-906/hashed-oplog-monitoring branch from 744f8aa to 439bc24 Compare October 20, 2026 16:49
@delthas
delthas requested review from SylvainSenechal and removed request for benzekrimaha October 21, 2026 09:23
@delthas

delthas commented Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

Requesting @SylvainSenechal since Maha Benzekri is on PTO

@delthas
delthas force-pushed the improvement/BB-906/hashed-oplog-monitoring branch 2 times, most recently from 01d88e6 to 4bf1905 Compare October 21, 2026 13:10

@maeldonn maeldonn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Screenshot and description seem outdated since the review changes. Could you refresh them?

@delthas
delthas requested a review from maeldonn October 21, 2026 15:02
@delthas

delthas commented Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

Screenshot and description seem outdated since the review changes. Could you refresh them?

Done

Alerts on paused or stopped hashed connectors, and on migrations not
completing. KafkaConnectTooManyConnectors is bounded by the number of
hashed connectors with hashed ingestion, and keeps the soft limit with
per-bucket ingestion: hashed connectors do not grow with the number of
buckets.

The dashboard overview shows the connectors by state in both ingestion
modes, with the expected number of hashed connectors, and the
per-connector oplog lag reported by the mongo-kafka connector. The
bucket panels (bucket counts and changes, acknowledgement lag), which
stay empty with hashed ingestion, move to a collapsed row.

dashboard.json is generated with grafanalib 0.6.3, as the rest of the
file was.

Issue: BB-906
@delthas
delthas force-pushed the improvement/BB-906/hashed-oplog-monitoring branch from 4bf1905 to 3971f39 Compare October 21, 2026 15:39
@delthas

delthas commented Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

/approve

Base automatically changed from improvement/BB-906/hashed-oplog-ingestion to development/9.6 October 21, 2026 21:58
@bert-e

bert-e commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Hello delthas,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options
name description privileged authored
/after_pull_request Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval Bypass the pull request author's approval ⭐
/bypass_build_status Bypass the build and test status ⭐
/bypass_commit_size Bypass the check on the size of the changeset TBA ⭐
/bypass_incompatible_branch Bypass the check on the source branch prefix ⭐
/bypass_jira_check Bypass the Jira issue check ⭐
/bypass_peer_approval Bypass the pull request peers' approval ⭐
/bypass_leader_approval Bypass the pull request leaders' approval ⭐
/bypass_source_branch_lineage Bypass the cross-branch contamination check ⭐
/approve Instruct Bert-E that the author has approved the pull request. ✍️
/create_pull_requests Allow the creation of integration pull requests.
/create_integration_branches Allow the creation of integration branches.
/no_octopus Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity Change review acceptance criteria from one reviewer at least to all reviewers
/wait Instruct Bert-E not to run until further notice.
Available commands
name description privileged
/help Print Bert-E's manual in the pull request.
/status Print Bert-E's current status in the pull request.
/clear Remove all comments from Bert-E from the history TBA
/retry Re-start a fresh build TBA
/build Re-start a fresh build TBA
/force_reset Delete integration branches & pull requests, and restart merge process from the beginning.
/reset Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

@bert-e

bert-e commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

I have successfully merged the changeset of this pull request
into targetted development branches:

  • ✔️ development/9.6

The following branches have NOT changed:

  • development/7.10
  • development/7.4
  • development/7.70
  • development/8.6
  • development/9.0
  • development/9.1
  • development/9.2
  • development/9.3
  • development/9.4
  • development/9.5

This pull request did not target the following hotfix branch(es) so they
were left untouched:

  • hotfix/7.2.0
  • hotfix/7.6.0
  • hotfix/7.10.3
  • hotfix/7.4.1
  • hotfix/7.10.0
  • hotfix/7.10.2
  • hotfix/9.0.4
  • hotfix/7.10.1
  • hotfix/7.4.10
  • hotfix/7.7.0
  • hotfix/7.4.7
  • hotfix/7.10.12
  • hotfix/7.10.17
  • hotfix/8.2.12
  • hotfix/7.4.6
  • hotfix/7.4.9
  • hotfix/9.0.7
  • hotfix/7.70.12
  • hotfix/7.70.15
  • hotfix/7.4.5
  • hotfix/7.9.0
  • hotfix/7.8.0
  • hotfix/7.4.4
  • hotfix/7.4.8
  • hotfix/7.4.3
  • hotfix/7.4.0
  • hotfix/7.10.4
  • hotfix/7.4.2
  • hotfix/7.10.8
  • hotfix/7.70.1

Please check the status of the associated issue BB-906.

Goodbye delthas.

The following options are set: approve

@bert-e
bert-e merged commit 3971f39 into development/9.6 Oct 21, 2026
25 checks passed
@bert-e
bert-e deleted the improvement/BB-906/hashed-oplog-monitoring branch October 21, 2026 21:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants