Skip to content

MINIFICPP-2877 Fix a lock ordering issue - #2297

Open
fgerlits wants to merge 1 commit into
mainfrom
MINIFICPP-2877_Fix-a-lock-ordering-issue
Open

fgerlits wants to merge 1 commit into
mainfrom
MINIFICPP-2877_Fix-a-lock-ordering-issue

Conversation

@fgerlits

@fgerlits fgerlits commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

ThreadSanitizer has found a mutex lock order issue between FlowController::mutex_ (M1) and C2MetricsPublisher::metrics_mutex_ (M2): during a flow update, we lock M1 first then M2, and during heartbeats, we lock M2 first and then some metrics nodes call executeOnAllComponents(), which locks M1. This is unlikely to happen in practice, because executeOnAllComponents checks if an update is in progress before trying to acquire the M1 lock -- but the flow update could start in the tiny window between these two lines. Also, if executeOnAllComponents finds that an update is in progress, it returns early without doing its stuff, so some metrics could have been missing from the heartbeat.

Claude also found a problem with metrics node types FlowInformation, QueueMetrics and ProcessorMetrics: these store raw pointers to processors or connections inside the current flow, which can become dangling after a flow update.

This PR fixes both issues by removing executeOnAllComponents() and replacing it with a component state cache, which has its own leaf mutex. The cache is cleared before the flow update and resnapped afterwards, which happens under this mutex.

The new MetricsDuringReloadTests unit test exercises these code paths by running several metrics serializations and flow updates in parallel. Before this change, running under ThreadSanitizer, it fails with both kinds of errors (lock order inversion, invalid read); after the change, it passes.


Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.

In order to streamline the review of the contribution we ask you to ensure the following steps have been taken:

For all changes:

  • Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?

  • Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

  • Has your PR been rebased against the latest commit within the target branch (typically main)?

  • Is your initial contribution a single, squashed commit?

For code changes:

  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE file?
  • If applicable, have you updated the NOTICE file?

For documentation related changes:

  • Have you ensured that format looks appropriate for the output in which it is rendered?

Note:

Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible.

ThreadSanitizer has found a mutex lock order issue between FlowController::mutex_ (M1)
and C2MetricsPublisher::metrics_mutex_ (M2): during a flow update, we lock M1 first
then M2, and during heartbeats, we lock M2 first and then some metrics nodes call
executeOnAllComponents(), which locks M1. This is unlikely to happen in practice,
because executeOnAllComponents checks if an update is in progress before trying to acquire
the M1 lock -- but the flow update could start in the tiny window between these two lines.
Also, if executeOnAllComponents finds that an update is in progress, it returns early
without doing its stuff, so some metrics could have been missing from the heartbeat.

Claude also found a problem with metrics node types FlowInformation, QueueMetrics and
ProcessorMetrics: these store raw pointers to processors or connections inside the
current flow, which can become dangling after a flow update.

This PR fixes both issues by removing executeOnAllComponents() and replacing it with
a component state cache, which has its own leaf mutex. The cache is cleared before
the flow update and resnapped afterwards, which happens under this mutex.

The new MetricsDuringReloadTests unit test exercises these code paths by running several
metrics serializations and flow updates in parallel. Before this change, running under
ThreadSanitizer, it fails with both kinds of errors (lock order inversion, invalid read);
after the change, it passes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant