Skip to content

events/export: Document metric_export_sample_by - #1539

Open
cmouse wants to merge 4 commits into
dovecot:mainfrom
cmouse:optl
Open

cmouse wants to merge 4 commits into
dovecot:mainfrom
cmouse:optl

Conversation

@cmouse

@cmouse cmouse commented May 15, 2026

Copy link
Copy Markdown
Contributor

Adds a Sampling section to the Event Export page, settings entries for metric_export_sample_by / _field / _permille, and a stats_sample_by anchor for cross-linking.

Sampling hashes a configured event field (typically session) and forwards only the fraction set by permille (parts per thousand) to the exporter, while in-process statistics counters keep counting every event. Multiple sample_by blocks combine with AND semantics.

JIRA: DOV-8702

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@cmouse
cmouse force-pushed the optl branch 5 times, most recently from d6613e7 to 8427d12 Compare May 21, 2026 07:13
Comment thread data/settings.js
event_exporter_driver: {
tags: [ 'event-export' ],
values: setting_types.ENUM,
values_enum: [ 'log', 'file', 'unix', 'http-post', 'drop' ],

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  - metric_export_sample_by_permille UINT → _percent STR (0.001..100, DENOM 100000); hash % 1000
    < permille text and examples.
  - event_exporter_opentelemetry_trace_id_field no longer exists; trace_id is random per session
    and propagated, not SHA-1(session); span_id not SHA-1 derived; no Span.Link for <base>:<rest>
    sessions.
  - service.instance.id is UUIDv4 per stats process, not hostname; request timeout default 2s not
    10s; log records also protobuf now.
  - Missing: propagation docs (IMAP ID x-trace-id/x-parent-id, XCLIENT X-TRACE-ID/X-PARENT-ID,
    HTTP headers, auth protocol trace_id/parent_span_id, userdb lookup fields), _-prefixed
    reserved field convention, span-marking rules.

@github-actions

This comment has been minimized.

Comment thread data/settings.js Outdated
},
tags: [ 'metrics' ],
values: setting_types.NAMED_LIST_FILTER,
seealso: [ '[[link,stats_sample_by]]' ],

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

link these 3 settings together

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: metric_export_sample_by, _field and _percent now list each other (and stats_sample_by) in seealso.

Comment thread docs/core/config/events/export.md Outdated
the resulting export usable for tracing: a sampled session has all of
its events, not a random scattering.

The field `trace_id` is special: unless an event has a field named `trace_id`,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

..and it never has now, right? could we even enforce with assert to avoid this extra unnecessary explanation?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, an event field named trace_id is no longer looked up; sampling by trace_id always uses the event's trace id, so there is nothing to assert. The text now just says that.

Comment thread docs/core/config/events/export.md Outdated
The field `trace_id` is special: unless an event has a field named `trace_id`,
the trace id of the event is used. All events of a trace share it, even when
they come from different processes and sessions. This keeps whole traces
together.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really understand this paragraph. When would I use trace_id?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rewritten: trace_id keeps whole traces together, including events of other systems sharing the trace id (traceparent from a trusted network, ID/XCLIENT from a trusted proxy). It also keeps a doveadm command whole, since its events have no session field.

Comment thread data/settings.js Outdated
Path to event unix socket with [[setting,event_exporter_driver,unix]].`
},

event_exporter_opentelemetry_endpoint_url: {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is the "endpoint" important in the name? we don't have it elsewhere either

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed to event_exporter_opentelemetry_url.

Comment thread data/settings.js Outdated
Collector, Jaeger v2, Grafana Tempo, etc.).`
},

event_exporter_opentelemetry_schedule_delay: {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

event_exporter_opentelemetry_flush_interval? event_exporter_opentelemetry_flush_ttl?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed to event_exporter_opentelemetry_flush_interval.

Comment thread data/settings.js Outdated
not be \`0\`.`
},

event_exporter_opentelemetry_max_export_batch_size: {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

event_exporter_opentelemetry_flush_count? In any case, not "size", which is for SIZE typed settings

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed to event_exporter_opentelemetry_flush_count.

Comment thread data/settings.js Outdated
seealso: [ 'event_exporter_opentelemetry_max_export_batch_size' ],
text: `
Limits how many records may wait for the collector. While
\`max_queue_size / max_export_batch_size\` requests are still pending, a

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure what this max_queue_size / max_export_batch_size means.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced: _flush_count is the number of records that triggers a send, and _max_pending_requests (default 4) is how many export requests can wait for the collector's response. When full, the next batch is dropped and an error is logged.

Comment thread data/settings.js Outdated
Must not be \`0\`.`
},

event_exporter_opentelemetry_max_queue_size: {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as above, not a "size"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed to event_exporter_opentelemetry_max_pending_requests; it counts requests, not bytes.

### Sampling traces

Sample by `trace_id` to keep or drop whole traces, see
[[link,stats_sample_by]].

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is saying it differently than in the setting text, but I still don't understand how this differs from sampling by session.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section now points to the sampling text, which explains the difference from session: other systems' events in the same trace, and doveadm commands.

Adds a Sampling section to the Event Export page and settings entries
for metric_export_sample_by / _field / _percent, and a stats_sample_by
anchor for cross-linking.

Sampling hashes a configured event field (typically session, or
trace_id for whole traces) and forwards only the percentage set by
percent to the exporter, while in-process statistics counters keep
counting every event. Multiple sample_by blocks combine with AND
semantics.
Adds settings entries for event_exporter_opentelemetry_url,
_flush_interval, _flush_count and _max_pending_requests, and an
OpenTelemetry section: the span model (one span per span owner), span
kinds, status and timing, trace correlation, resource attributes,
batching, and the protobuf and JSON wire formats.

Whitelist the new spell-check terms in .github/actions/spelling/allow.txt.
Add event_exporter_opentelemetry_emit_spans and
event_exporter_opentelemetry_emit_logs settings, and describe the OTLP
LogRecord output shape, severity mapping and trace/span correlation, and
the reserved _-prefixed event fields.
Add http_client_trace_context and
http_server_trace_context_trusted_networks, and the trace-id /
parent-span-id (IMAP ID) and TRACE-ID / PARENT-SPAN-ID (XCLIENT)
forwarding fields. Describe in the OpenTelemetry section how the trace id
and parent span id are passed between processes and servers.

Add an upgrade note: with imap_id_retain, upgrade proxies before
backends.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants