Skip to content

stats: per-thread counters, log-scaled histogram, interval and JSON output - #41

Open
byrnedj wants to merge 2 commits into
intel:mainfrom
byrnedj:stats-tls
Open

byrnedj wants to merge 2 commits into
intel:mainfrom
byrnedj:stats-tls

Conversation

@byrnedj

@byrnedj byrnedj commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

DTO_COLLECT_STATS kept one global table that every thread updated with atomic adds. Collecting stats got slower as threads were added, so it couldn't be left on to profile a multithreaded workload. This PR changes how stats are collected and adds the output needed to follow a run over time and process it with tools.

Changes

  • Per-thread counters. Each thread counts in its own block, mapped (demand-zero) when it first records a call. When the thread exits, the block is folded into a retired total and unmapped. Output sums the live blocks and the retired total. Increments are relaxed loads and stores to the thread's own block, with no locked instructions.
  • Log-scaled histogram. The old histogram had 512 linear 4 KiB buckets, and everything from 2 MiB up landed in the last one. The new one has a bucket for calls under 64 B, then four buckets per power of two up to 4 GiB, then one for anything larger (106 in all).
  • DTO_STATS_INTERVAL_MS=N. A sampler thread writes what happened in each N ms. The interval records add up to the summary.
  • DTO_STATS_FORMAT=json. Output is JSON Lines: interval and summary records with calls/bytes/ns for each op (memset/memcpy/memmove/memcmp) and path (cpu/dsa/dsa_failed), plus failure reasons. The summary also has the histogram and the configuration at exit (wait method, the auto-tuned min_bytes and cpu_size_fraction, NUMA, number of WQs). Schema is in the README.
  • DTO_STATS_FILE expands %p to the PID and appends instead of truncating, so a child process no longer overwrites its parent's output. Every record carries its pid.
  • fork. The child starts with empty counts and reopens a %p file under its own PID. Locks are taken around fork, so a child can't inherit a lock held by the sampler or a printer. SIGINT output uses trylock.
  • Removed dto_stats_log(). With DTO_LOG_FILE set it wrote each line twice, the second time from a va_list that had already been consumed. Also removed: turning stats on no longer forces the TRACE log level.

Overhead

Extra time per call with stats on, CPU path (DTO_USESTDC_CALLS=1), threads pinned, median of 5 runs, gnr-qual1:

size threads main this PR
4 KiB 1 +55 ns +44 ns
4 KiB 4 +357 ns +46 ns
4 KiB 16 +1319 ns +42 ns
4 KiB 32 +2612 ns +37 ns
64 KiB 32 +1724 ns +37 ns

What's left is the two clock_gettime calls. With stats off, nothing changed. On the DSA path (1 and 4 WQs), runs with stats on were 12–20% faster than main at 4 and 16 threads; 32 threads was too noisy to compare.

Testing

New ctest -R stats (tests/test_stats.py plus a workload of known calls, -fno-builtin). It needs no DSA hardware and checks:

  • totals and the bucket each size lands in
  • interval records summing to the totals
  • calls from 200 short-lived threads all counted, with their blocks unmapped
  • separate parent and child records after fork
  • %p files
  • the text tables

On gnr-qual1, stats passes as root (dsa0 wq0.0) and as a user (CPU path), and functional passes 10/10.

The second commit links libdto with pthread. The per-thread blocks use thread keys, and the sampler uses a thread and a condition variable; before glibc 2.34 these are in libpthread, not libc. Without the link, preloading DTO into a program that doesn't link libpthread itself (sh, true) failed on RHEL 8 with undefined symbol: pthread_key_create. The stats test now checks this case.

Merging with other open PRs

dto.c merges cleanly with #34 and #40. #34 and #38 conflict only in the lines each PR adds to CMakeLists.txt, TESTING.md and README.md.

A follow-up will add dtoctl --stats-run. It runs a program CPU-only and then with DSA, optionally under perf, and reports DSA speedup per operation with an Amdahl's-law estimate, using the JSON output added here.

byrnedj and others added 2 commits October 21, 2026 11:11
DTO_COLLECT_STATS kept one global table that every thread updated with
atomic adds, so the cost of collecting grew with the thread count: a
4 KiB memcpy on the CPU path cost 55 ns more with stats on at one
thread and 2.6 us more at 32 threads. Each thread now counts in its own
block, mapped when the thread first records a call and folded into a
retired total when the thread exits; output sums the live blocks and
the retired total. Stats now cost about 40 ns per call (the two clock
reads) at any thread count:

  extra cost per call, CPU path   1 thr   4 thr   16 thr  32 thr
  4 KiB, before                   +55     +357    +1319   +2612 ns
  4 KiB, after                    +44     +46     +42     +37 ns

The histogram's 512 linear 4 KiB buckets, the last of which held
everything from 2 MiB up, become log-scaled: under 64 bytes, then four
buckets per power of two up to 4 GiB, then one for anything larger.

New settings:
  DTO_STATS_INTERVAL_MS=N  a sampler thread writes what happened in
                           each N ms; the records add up to the summary
  DTO_STATS_FORMAT=json    JSON Lines: interval and summary records with
                           calls/bytes/ns per op and path, failures,
                           histogram, and the configuration at exit
  DTO_STATS_FILE=...%p...  %p becomes the process ID

DTO_STATS_FILE is now appended to rather than truncated, so a forked
or exec'd child's records don't overwrite its parent's; each record
carries its pid. After fork the child starts with empty counts, and a
%p file is reopened under its own pid. The old dto_stats_log() wrote
each line twice when DTO_LOG_FILE was set and reused a consumed
va_list; it is gone, along with the TRACE log level that turning on
stats forced.

Adds tests/test_stats.py (ctest -R stats) with a workload of known
calls: totals and buckets, intervals summing to totals, 200
short-lived threads all counted and their blocks unmapped, fork,
%p, and text output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mv7xKtQPkGoXQUvvuuhAEz
The stats rework uses thread keys, a sampler thread and its condition
variable. Before glibc 2.34 those functions are in libpthread, not libc,
so preloading libdto into a program that doesn't link libpthread itself
(sh, true, most single-threaded tools) failed at load time:

  true: symbol lookup error: libdto.so: undefined symbol: pthread_key_create

Link libdto with pthread in CMake and the Makefile (an empty stub on
newer glibc), and check in the stats test that stats work in a program
without libpthread. Seen on RHEL 8 (glibc 2.28); Ubuntu 24.04 (2.39)
is not affected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mv7xKtQPkGoXQUvvuuhAEz
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant