Skip to content

beholder: inject meter in workflow monitoring, gateway, heartbeat (Phase 5) - #23973

Merged
kalverra merged 2 commits into
developfrom
DX-5066/parallel-00d-meters
Oct 26, 2026
Merged

kalverra merged 2 commits into
developfrom
DX-5066/parallel-00d-meters

Conversation

@kalverra

@kalverra kalverra commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Intent

Layer 4 of the beholder emitter-injection stack (DX-5066): engine-metrics tests swapped the process-global beholder client to install a manual meter and the heartbeat test asserted on a writer client's stdout, serializing them and racing parallel engine tests; this PR injects meters, tracers, and emitters explicitly so no core test calls beholder.SetClient.

Notes

  • EngineConfig.Meter is deliberately not defaulted in Validate — newBaseEngine falls back to beholder.GetMeter() at construction, preserving today's read timing.
  • InitMonitoringResources() remains for cre.go; it now delegates to InitMonitoringResourcesWithMeter(beholder.GetMeter()) with identical timing.
  • Heartbeat resolves its old TODO allow override of tracer: WithTracer defaults to beholder.GetTracer() at construction, and the dead discarded cme.WithMapLabels(labels) call is removed.

Changes

Where the engine's meter comes from:

flowchart LR
  T[tests: newTestMeter]:::new --> Cfg[EngineConfig.Meter]:::new
  Cfg --> NB[newBaseEngine]
  NB -->|nil → GetMeter| IM["InitMonitoringResourcesWithMeter(meter)"]:::new
  classDef new fill:#d4f8d4,stroke:#2da44e
Loading
InitMonitoringResourcesWithMeter(meter)
  • All 59 counter/gauge registrations take the injected meter; tests build metrics on a private manual-reader meter.
One newTestMeter helper replaces both global-swap helpers
  • setupTestMeter / setupInitFailureTestMeter become a private-meter helper per test package (example); assertions tighten from "tolerate foreign increments" to exact counts (TestEngine_Init).
Heartbeat asserts on a recorder, not stdout
  • TestHeartbeat_MeterEvents injects WithEmitter(NewLabelerWithEmitter(rec)) and asserts on decoded BaseMessages; the byteCollector/writer client and the restore-to-noop bug are deleted.
Gateway metrics test builds its meter directly
  • NewGatewayMetrics(meterProvider.Meter("gateway-test")) with no global swap (metrics_test.go).

Requires

#23972 (rest of the DX-5066 beholder stack)

Supports

Follow-up DX-5066 parallel-test layers (02–07)

Deployment Validation Steps

Run a node locally with telemetry enabled and confirm engine metrics (platform_engine_*) and heartbeat messages still reach the collector; no metric names or labels changed.


Stack created with GitHub Stacks CLI • Give Feedback 💬

@github-actions

Copy link
Copy Markdown
Contributor

✅ No conflicts with other open PRs targeting develop

@github-actions

Copy link
Copy Markdown
Contributor

I see you updated files related to core. Please run make gocs in the root directory to add a changeset as well as in the text include at least one of the following tags:

  • #added For any new functionality added.
  • #breaking_change For any functionality that requires manual action for the node to boot.
  • #bugfix For bug fixes.
  • #changed For any change to the existing functionality.
  • #db_update For any feature that introduces updates to database schema.
  • #deprecation_notice For any upcoming deprecation functionality.
  • #internal For changesets that need to be excluded from the final changelog.
  • #nops For any feature that is NOP facing and needs to be in the official Release Notes for the release.
  • #removed For any functionality/config that is removed.
  • #updated For any functionality that is updated.
  • #wip For any change that is not ready yet and external communication about it should be held off till it is feature complete.

@trunk-io

trunk-io Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

Static Badge   Static Badge   Static Badge

View Full Report ↗︎ ⋅ Docs

@kalverra
kalverra force-pushed the DX-5066/parallel-00d-meters branch from 00d477b to 4612cc8 Compare October 14, 2026 20:12
@kalverra
kalverra force-pushed the DX-5066/parallel-00d-meters branch from 4612cc8 to 62b7b70 Compare October 14, 2026 20:24
@kalverra
kalverra force-pushed the DX-5066/parallel-00d-meters branch from 62b7b70 to 9f8fdf8 Compare October 14, 2026 20:50
@kalverra
kalverra force-pushed the DX-5066/parallel-00d-meters branch 2 times, most recently from acffd94 to c147c06 Compare October 19, 2026 15:04
@kalverra kalverra added complexity:high pr-triage: high complexity review-tier:3 risk:medium pr-triage: medium risk labels Oct 20, 2026
@kalverra
kalverra requested review from MStreet3 and bolekk October 20, 2026 11:52
@kalverra kalverra added complexity:high pr-triage: high complexity risk:medium pr-triage: medium risk and removed risk:medium pr-triage: medium risk complexity:high pr-triage: high complexity review-tier:3 labels Oct 20, 2026
@kalverra

kalverra commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator Author

PR Triage · Risk 🟡 Medium · Complexity 🔴 High

Suggests 1 approval, including one from an owner. Suggested reviewers: @bolekk, @MStreet3.

Level Why
Risk Medium core/services/workflows/v2/config.go is a hotspot: 11.30 recent commits in 90d (cut 8.32) and max function cognitive complexity 39 (limit 15); 2 recent commits by @kalverra into touched packages within 7d
Complexity High logic added 80 (High at 45); Needs a full human read; may be worth splitting

Start Review Here

  1. core/services/workflows/v2/config.go — hotspot file
  2. InitMonitoringResourcesWithMeter · core/services/workflows/monitoring/monitoring.go:110-528 — new function · +62 logic
  3. newBaseEngine · core/services/workflows/v2/base_engine.go:143-147 — +1 logic
  4. Test_InitMonitoringResourcesWithMeter · core/services/workflows/monitoring/monitoring_test.go:22-66 — new function · +14 logic
  5. TestHeartbeat_MeterEvents · core/services/chainlink/heartbeat_test.go:57-103 — +3 logic
Visualizations

Score map

flowchart LR
  R["Risk: Medium"]
  rs["1 signal → floor Medium"]
  s0["hotspot core/services/workflows/v2/config.go"] --> rs
  rs --> R
  seq["sequential changes (2 in 7d)"] -->|"lifts Low → Medium"| R
  C["Complexity: High"]
  m0["logic added 80 ≥ 45"] --> C
  RT["Suggests 1 approval · one from an owner"]
  R --> RT
  classDef low fill:#dafbe1,stroke:#1a7f37,color:#1f2328;
  classDef medium fill:#fff8c5,stroke:#9a6700,color:#1f2328;
  classDef high fill:#ffebe9,stroke:#cf222e,color:#1f2328;
  class R medium;
  class C high;
Loading

Change shape

pie title Change shape
    "Tests" : 960
    "Source" : 209
Loading

— distributed across 12 files

Files: 12 changed
File Risk Complexity Why
core/services/workflows/monitoring/monitoring.go Low High logic added 62 (High at 45)
core/services/chainlink/heartbeat.go Low Low
core/services/chainlink/heartbeat_test.go Low Low
core/services/gateway/monitoring/metrics_test.go Low Low
core/services/workflows/monitoring/monitoring_test.go Low Low
core/services/workflows/v2/base_engine.go Low Low
core/services/workflows/v2/config.go Low Low
core/services/workflows/v2/engine_init_failure_metrics_test.go Low Low
core/services/workflows/v2/engine_limit_metrics_test.go Low Low
core/services/workflows/v2/engine_org_id_missing_test.go Low Low
core/services/workflows/v2/engine_test.go Low Low
core/services/workflows/v2/init_failures_test.go Low Low
Why these reviewers
Reviewer Share Last commit
@bolekk 31% 2026-10-19
@MStreet3 18% 2026-10-23

Others who know this code: agparadiso 11%, jmank88 8%, mchain0 8%.

Ranked by commits to the changed files. Commits in the 360 days before the head commit count, newer ones more. Requested everyone with at least 20% of the score, up to 4. Added 1 reviewer below the share bar to reach the approval floor. Excluded the PR author, @kalverra. Considered only current organization members.

Metrics and policy
Metric PR value Scale Medium at High at
logic added (cognitive delta) 80 [████│██████│████████] 15 45
syntax change (AST delta) 1,169 [███████████████░░░░│] 1,600 —
new code size (Kolmogorov delta) 2,043 [███████████████│████] 1,600 —
most complex function (max function cognitive complexity) 62 [██████████████████░│] 70 70

Risk signals: change spread ○ · subsystems ○ · hotspot ● · blast radius ○ · author familiarity ○ · floor Medium

pr-triage v0.0.0-20261023193103-09473e7e1439 · policy c6c02b9


What is this? · How is this calculated?

@kalverra
kalverra force-pushed the DX-5066/parallel-00d-meters branch from c147c06 to 41ec02e Compare October 20, 2026 18:30
@kalverra
kalverra force-pushed the DX-5066/parallel-00d-meters branch 2 times, most recently from 6c6fdad to d39df3e Compare October 21, 2026 23:57
@kalverra
kalverra force-pushed the DX-5066/parallel-00d-meters branch 2 times, most recently from ff13d18 to c8047de Compare October 22, 2026 16:55
@kalverra
kalverra force-pushed the DX-5066/parallel-00d-meters branch from c8047de to 8b4e5dc Compare October 22, 2026 17:13
Base automatically changed from DX-5066/parallel-00c-remote-exec to develop October 23, 2026 18:05
@kalverra
kalverra requested a review from a team as a code owner October 23, 2026 18:05
@kalverra
kalverra force-pushed the DX-5066/parallel-00d-meters branch from 8b4e5dc to 6cac96d Compare October 23, 2026 18:05
…ase 5)

- monitoring: InitMonitoringResourcesWithMeter(meter) holds the moved
  body; InitMonitoringResources() wraps it on the global meter (cre.go
  caller unchanged)
- engine: EngineConfig.Meter, nil defaults to beholder.GetMeter() at
  construction in newBaseEngine (not resolved in Validate)
- tests: one newTestMeter helper per test package replaces
  setupTestMeter/setupInitFailureTestMeter global client swaps; all
  listed tests now t.Parallel() with private meters, assertions
  tightened to exact counts
- gateway metrics_test: NewGatewayMetrics(meter) directly
- heartbeat: WithTracer option (resolves tracer override TODO), dead
  WithMapLabels statement dropped; test asserts emitted BaseMessages
  via beholdertest.NewRecorder instead of swapping the global client

After this, no test in core/ calls beholder.SetClient.
@kalverra
kalverra force-pushed the DX-5066/parallel-00d-meters branch from 6cac96d to 689af79 Compare October 23, 2026 18:24
@kalverra
kalverra requested a balanced review from Copilot October 23, 2026 18:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Four workflow metric registrations still bypass the injected meter, sending measurements to the global provider instead.

1 open issue
What changed in this PR

This PR extends explicit telemetry injection to workflow monitoring and heartbeat services, reducing reliance on process-global Beholder state and allowing isolated tests to run in parallel.

Changes:

  • Adds an engine meter option and a meter-aware monitoring constructor while retaining the existing default entry point.
  • Adds heartbeat tracer injection and replaces stdout-based assertions with an injected message recorder.
  • Uses private meters in workflow and gateway tests, enabling parallel execution and exact counter assertions.
File Description
core/​services/​workflows/​v2/​init_failures_test.go Uses private meters for parallel initialization-failure tests.
core/​services/​workflows/​v2/​engine_test.go Injects meters and asserts exact subscription counts.
core/​services/​workflows/​v2/​engine_org_id_missing_test.go Introduces a private-meter helper and parallelizes counter tests.
core/​services/​workflows/​v2/​engine_limit_metrics_test.go Isolates limit-degradation metrics in parallel tests.
core/​services/​workflows/​v2/​engine_init_failure_metrics_test.go Replaces global swaps with injected meters and exact assertions.
core/​services/​workflows/​v2/​config.go Adds the optional engine meter dependency.
core/​services/​workflows/​v2/​base_engine.go Resolves the configured meter at engine construction.
core/​services/​workflows/​monitoring/​monitoring.go Adds meter-aware instrument registration.
core/​services/​gateway/​monitoring/​metrics_test.go Supplies a private meter directly to gateway metrics.
core/​services/​chainlink/​heartbeat.go Adds tracer injection and removes a discarded labeler call.
core/​services/​chainlink/​heartbeat_test.go Replaces global writer-client assertions with a message recorder.

Review effort: Balanced

Comment thread core/services/workflows/monitoring/monitoring.go
@kalverra
kalverra enabled stacked auto-merge October 23, 2026 19:49
@cl-sonarqube-production

Copy link
Copy Markdown

@kalverra
kalverra added this pull request to the merge queue Oct 26, 2026
Merged via the queue into develop with commit 86920ee Oct 26, 2026
234 checks passed
@kalverra
kalverra deleted the DX-5066/parallel-00d-meters branch October 26, 2026 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

complexity:high pr-triage: high complexity risk:medium pr-triage: medium risk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants