Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions diagnostic_aggregator/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,11 @@ if(BUILD_TESTING)
"${CMAKE_CURRENT_SOURCE_DIR}/test/test_publish_values.py"
TIMEOUT 60
)

ament_add_pytest_test(test_persistent_state
"${CMAKE_CURRENT_SOURCE_DIR}/test/test_persistent_state.py"
TIMEOUT 180
)
endif()

install(
Expand Down
32 changes: 31 additions & 1 deletion diagnostic_aggregator/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,11 +123,41 @@ This must be the name of the class that implements the analyzer.
Additional parameters depend on the type of the analyzer.

Any diagnostic item that is not matched by any analyzer will be published by an "Other" analyzer.
Items created by the "Other" analyzer will go stale after 5 seconds.
Items created by the "Other" analyzer will go stale after 5 seconds, unless they are marked persistent (see below).

The `critical` parameter makes the aggregator react immediately to a degradation in diagnostic state.
This is useful if the toplevel state is parsed by a watchdog for example.

## Persistent items

By default an analyzer decides staleness purely from the age of the last received message: an item that has not been updated within `timeout` is reported as `STALE` (or dropped when `discard_stale` is set), and the "Other" analyzer drops items after 5 seconds.
A publisher can opt out of this for a specific status by adding the reserved key-value pair `persistent_state: True` to `DiagnosticStatus.values` (`DiagnosticStatusWrapper::setPersistentState()` in `diagnostic_updater` does exactly that).

A persistent item

- keeps its last level and message in the aggregated output, so it is never turned `STALE` or discarded,
- gets an extra `last_update_age` value (seconds since the last update), so operators can still tell an old persistent state from a fresh one,
- goes back to the normal timeout behaviour as soon as the publisher sends it again without the key.

This is intended for terminal or latched states, for example the final "Node shutting down" status a node publishes before it exits, or an emergency stop that stays latched.
It is not meant for ordinary operation: a persistent item whose publisher died keeps showing its last level until the aggregator is restarted.
Persistence only covers the lifetime of the aggregator, since `/diagnostics` is not published with transient-local durability.

`GenericAnalyzer` accepts an optional `persistent_timeout` parameter (seconds, default unlimited) to cap how long a persistent item is honoured; after that it is treated like any other item.
Persistent items that end up in the "Other" analyzer are capped by the aggregator-level `other_persistent_timeout` parameter (seconds, default unlimited) instead, since the "Other" analyzer has no configuration of its own.

``` yaml
analyzers:
ros__parameters:
other_persistent_timeout: 3600.0 # Optional, cap for persistent items in "Other"
motors:
type: diagnostic_aggregator/GenericAnalyzer
path: Motors
contains: [ 'motor_driver: ' ]
timeout: 5.0
persistent_timeout: 3600.0 # Optional, keep a persistent item for at most one hour
```

## Launching

You can launch the `aggregator_node` like this (see [example.launch.py.in](example/example.launch.py.in)):
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@ Input (status names):
base_path: My Robot
pub_rate: 1.0
other_as_errors: false
other_persistent_timeout: -1.0
analyzers:
sensors:
type: GenericAnalyzer
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,14 @@ inline bool getParamVals(rclcpp::Parameter param, std::vector<std::string> & out
* The GenericAnalyzer can discard stale items. Use the "discard_stale" parameter to
* remove any items that haven't updated within the timeout. This is "false" by default.
*
* A publisher can opt an item out of the stale handling by adding the reserved
* key-value "persistent_state: True" to the status. Such an item keeps its last level
* and message (plus a "last_update_age" value) instead of going stale or being
* discarded, until it is published again without the key. This is meant for terminal
* or latched states, e.g. the final status a node publishes on shutdown. Use the
* "persistent_timeout" parameter to cap this in seconds; any value <= 0 (the default)
* means persistent items are honoured indefinitely.
*
* Example configurations:
*\verbatim
* hokuyo:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@
#define DIAGNOSTIC_AGGREGATOR__GENERIC_ANALYZER_BASE_HPP_

#include <algorithm>
#include <iomanip>
#include <map>
#include <memory>
#include <sstream>
Expand Down Expand Up @@ -67,6 +68,11 @@ namespace diagnostic_aggregator
*
* The GenericAnalyzerBase holds the state of the analyzer, and tracks if items are stale, and
* if the user has the correct number of items.
*
* Items whose last update carried the reserved "persistent_state" key set to true are
* never marked stale or discarded (see StatusItem::isPersistent()). They are reported
* with their last level and message plus a "last_update_age" value, until the publisher
* sends them again without the key, or until the optional persistent_timeout expires.
*/
class GenericAnalyzerBase : public Analyzer
{
Expand All @@ -76,6 +82,7 @@ class GenericAnalyzerBase : public Analyzer
path_(""),
timeout_(-1.0),
num_items_expected_(-1),
persistent_timeout_(-1.0),
discard_stale_(false),
has_initialized_(false),
has_warned_(false)
Expand All @@ -102,10 +109,11 @@ class GenericAnalyzerBase : public Analyzer
*/
bool init(
const std::string & path, const std::string & breadcrumb, double timeout = -1.0,
int num_items_expected = -1, bool discard_stale = false)
int num_items_expected = -1, bool discard_stale = false, double persistent_timeout = -1.0)
{
num_items_expected_ = num_items_expected;
timeout_ = timeout;
persistent_timeout_ = persistent_timeout;
path_ = path + "/" + nice_name_;
discard_stale_ = discard_stale;
breadcrumb_ = breadcrumb;
Expand Down Expand Up @@ -191,9 +199,14 @@ class GenericAnalyzerBase : public Analyzer
auto name = it->first;
auto item = it->second;

const double age = (clock_->now() - item->getLastUpdateTime()).seconds();
// A persistent item keeps its last level and message no matter how old it is,
// unless this analyzer caps persistence with persistent_timeout.
const bool persistent = item->isPersistent() &&
(persistent_timeout_ <= 0 || age <= persistent_timeout_);
bool stale = false;
if (timeout_ > 0) {
stale = (clock_->now() - item->getLastUpdateTime()).seconds() > timeout_;
if (timeout_ > 0 && !persistent) {
stale = age > timeout_;
}
Comment thread
fujitatomoya marked this conversation as resolved.
Resolved

// Erase item if its stale and we're discarding items
Expand All @@ -213,7 +226,18 @@ class GenericAnalyzerBase : public Analyzer

all_stale = all_stale && ((level == diagnostic_msgs::msg::DiagnosticStatus::STALE) || stale);

processed.push_back(item->toStatusMsg(path_, stale));
auto status = item->toStatusMsg(path_, stale);
if (persistent) {
// Keep the age visible, so a persistent item whose publisher is gone can be
// told apart from one that is still being refreshed.
std::ostringstream age_str;
age_str << std::fixed << std::setprecision(1) << age;
diagnostic_msgs::msg::KeyValue age_kv;
age_kv.key = LAST_UPDATE_AGE_KEY;
age_kv.value = age_str.str();
status->values.push_back(age_kv);
}
processed.push_back(status);

if (stale) {
header_status->level = diagnostic_msgs::msg::DiagnosticStatus::STALE;
Expand Down Expand Up @@ -288,6 +312,8 @@ class GenericAnalyzerBase : public Analyzer

double timeout_;
int num_items_expected_;
/// Upper bound in seconds for honouring persistent items; <= 0 means unlimited.
double persistent_timeout_;

/*!
*\brief Subclasses can add items to analyze
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,10 @@ namespace diagnostic_aggregator
* seperately. The aggregator will call analyze() on any message not handled by other
* analyzers.
*
* Stale items will be discarded after 5 seconds of no updates.
* Stale items will be discarded after 5 seconds of no updates, unless they are marked
* persistent (see StatusItem::isPersistent()), in which case they are kept with their
* last level and message for up to persistent_timeout seconds (unlimited by default,
* configured through the aggregator's "other_persistent_timeout" parameter).
*
* OtherAnalyzer is designed to be used internally by the Aggregator only.
*
Expand All @@ -84,14 +87,18 @@ class OtherAnalyzer : public GenericAnalyzerBase
*
*\param path Base path of Aggregator
*\param breadcrumb Prefix for parameter getter.
*\param persistent_timeout Upper bound in seconds for honouring persistent items,
* <= 0 means unlimited (see GenericAnalyzerBase).
*/
bool init(const std::string & path, const std::string & breadcrumb = "")
bool init(
const std::string & path, const std::string & breadcrumb = "",
double persistent_timeout = -1.0)
{
(void)breadcrumb;

nice_name_ = "Other";
path_ = path;
return GenericAnalyzerBase::init(path_, "", 5.0, -1, true);
return GenericAnalyzerBase::init(path_, "", 5.0, -1, true, persistent_timeout);
}

/*
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,8 @@
#ifndef DIAGNOSTIC_AGGREGATOR__STATUS_ITEM_HPP_
#define DIAGNOSTIC_AGGREGATOR__STATUS_ITEM_HPP_

#include <algorithm>
#include <cctype>
#include <map>
#include <memory>
#include <string>
Expand Down Expand Up @@ -69,6 +71,35 @@ inline std::string getOutputName(const std::string & item_name)
return output_name;
}

/*!
*\brief Reserved key in DiagnosticStatus.values that marks a status as persistent.
*
* A persistent status keeps its last level and message in the aggregated output
* instead of being turned STALE or discarded once it stops being updated. The
* value is parsed case-insensitively; "true" and "1" enable it. The publisher
* restores the normal timeout behaviour simply by sending the status again
* without this key (or with any other value).
*/
inline constexpr const char * PERSISTENT_STATE_KEY = "persistent_state";

/*!
*\brief Key appended to the values of a persistent status in the aggregated
* output, holding the seconds elapsed since the status was last updated.
*/
inline constexpr const char * LAST_UPDATE_AGE_KEY = "last_update_age";

/*!
*\brief Returns true if the given value of PERSISTENT_STATE_KEY enables persistence.
*/
inline bool isPersistentStateValue(const std::string & value)
{
std::string lower = value;
std::transform(
lower.begin(), lower.end(), lower.begin(),
[](unsigned char c) {return static_cast<char>(std::tolower(c));});
return lower == "true" || lower == "1";
}

/*!
*\brief Level of StatusItem. OK, Warn, Error, Stale
*/
Expand Down Expand Up @@ -248,6 +279,11 @@ class StatusItem
*/
const rclcpp::Time getLastUpdateTime() const {return update_time_;}

/*!
*\brief True if the last update carried PERSISTENT_STATE_KEY set to true.
*/
bool isPersistent() const {return persistent_;}

/*!
*\brief Returns true if item has key in values KeyValues
*
Expand Down Expand Up @@ -290,6 +326,7 @@ class StatusItem
std::string message_;
std::string hw_id_;
std::vector<diagnostic_msgs::msg::KeyValue> values_;
bool persistent_;
};

} // namespace diagnostic_aggregator
Expand Down
11 changes: 10 additions & 1 deletion diagnostic_aggregator/src/aggregator.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,7 @@ rcl_interfaces::msg::SetParametersResult Aggregator::onParametersSet(
void Aggregator::initAnalyzers()
{
bool other_as_errors = false;
double other_persistent_timeout = -1.0;

std::map<std::string, rclcpp::Parameter> parameters;
if (!n_->get_parameters("", parameters)) {
Expand All @@ -130,6 +131,11 @@ void Aggregator::initAnalyzers()
base_path_.append(param.second.as_string());
} else if (param.first.compare("other_as_errors") == 0) {
other_as_errors = param.second.as_bool();
} else if (param.first.compare("other_persistent_timeout") == 0) {
// Accept "other_persistent_timeout: 3600" (typed INTEGER by rclcpp) as well as "3600.0".
other_persistent_timeout =
param.second.get_type() == rclcpp::ParameterType::PARAMETER_INTEGER ?
static_cast<double>(param.second.as_int()) : param.second.as_double();
} else if (param.first.compare("history_depth") == 0) {
history_depth_ = param.second.as_int();
} else if (param.first.compare("critical") == 0) {
Expand All @@ -142,6 +148,8 @@ void Aggregator::initAnalyzers()
RCLCPP_DEBUG(logger_, "Aggregator base path configured to: %s", base_path_.c_str());
RCLCPP_DEBUG(
logger_, "Aggregator other_as_errors configured to: %s", (other_as_errors ? "true" : "false"));
RCLCPP_DEBUG(
logger_, "Aggregator other_persistent_timeout configured to: %f", other_persistent_timeout);
RCLCPP_DEBUG(
logger_, "Aggregator critical publisher configured to: %s", (critical_ ? "true" : "false"));
RCLCPP_DEBUG(
Expand All @@ -156,7 +164,8 @@ void Aggregator::initAnalyzers()

// Last analyzer handles remaining data
other_analyzer_ = std::make_unique<OtherAnalyzer>(other_as_errors);
other_analyzer_->init(base_path_); // This always returns true
// This always returns true
other_analyzer_->init(base_path_, "", other_persistent_timeout);
}
}

Expand Down
11 changes: 10 additions & 1 deletion diagnostic_aggregator/src/generic_analyzer.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@ bool GenericAnalyzer::init(
double timeout = 5.0;
int num_items_expected = -1;
bool discard_stale = false;
double persistent_timeout = -1.0;

for (const auto & param : parameters) {
string pname = param.first;
Expand Down Expand Up @@ -147,6 +148,13 @@ bool GenericAnalyzer::init(
rclcpp::get_logger("GenericAnalyzer"), "GenericAnalyzer '%s' found discard_stale: %s",
nice_name_.c_str(), pvalue.value_to_string().c_str());
discard_stale = pvalue.as_bool();
} else if (pname.compare("persistent_timeout") == 0) {
RCLCPP_DEBUG(
rclcpp::get_logger("GenericAnalyzer"), "GenericAnalyzer '%s' found persistent_timeout: %s",
nice_name_.c_str(), pvalue.value_to_string().c_str());
// Accept "persistent_timeout: 3600" (typed INTEGER by rclcpp) as well as "3600.0".
persistent_timeout = pvalue.get_type() == rclcpp::ParameterType::PARAMETER_INTEGER ?
static_cast<double>(pvalue.as_int()) : pvalue.as_double();
}
}

Expand Down Expand Up @@ -178,7 +186,8 @@ bool GenericAnalyzer::init(
my_path = "/" + my_path;
}

return GenericAnalyzerBase::init(path_, breadcrumb_, timeout, num_items_expected, discard_stale);
return GenericAnalyzerBase::init(
path_, breadcrumb_, timeout, num_items_expected, discard_stale, persistent_timeout);
}

GenericAnalyzer::~GenericAnalyzer() {}
Expand Down
17 changes: 17 additions & 0 deletions diagnostic_aggregator/src/status_item.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -38,13 +38,27 @@

#include <memory>
#include <string>
#include <vector>

namespace diagnostic_aggregator
{
using std::string;

using rclcpp::get_logger;

namespace
{
bool hasPersistentState(const std::vector<diagnostic_msgs::msg::KeyValue> & values)
{
for (const auto & kv : values) {
if (kv.key == PERSISTENT_STATE_KEY) {
return isPersistentStateValue(kv.value);
}
}
return false;
}
} // namespace

StatusItem::StatusItem(const diagnostic_msgs::msg::DiagnosticStatus * status)
: clock_(new rclcpp::Clock())
{
Expand All @@ -53,6 +67,7 @@ StatusItem::StatusItem(const diagnostic_msgs::msg::DiagnosticStatus * status)
message_ = status->message;
hw_id_ = status->hardware_id;
values_ = status->values;
persistent_ = hasPersistentState(values_);

output_name_ = getOutputName(name_);

Expand All @@ -67,6 +82,7 @@ StatusItem::StatusItem(const string item_name, const string message, const Diagn
message_ = message;
level_ = level;
hw_id_ = "";
persistent_ = false;

output_name_ = getOutputName(name_);

Expand Down Expand Up @@ -96,6 +112,7 @@ bool StatusItem::update(const diagnostic_msgs::msg::DiagnosticStatus * status)
message_ = status->message;
hw_id_ = status->hardware_id;
values_ = status->values;
persistent_ = hasPersistentState(values_);

update_time_ = clock_->now();

Expand Down
Loading