[Feature]: Low-rate, self-triggering TCs are reported Delayed/Timing-out by definition, even when data is complete #446

Description

Some pictures to help with explanation:

Current state of trigger buffers for a common data-taking mode (using low rate random trigger):

Unpacking the data:

Description:
For a low-rate, self-triggering TC (e.g. Random, CTB), the dataflow request to read back its own TC-buffer entry is, by default, built from that same object's own time range. DefaultRequestHandlerModel::get_fragment_pieces() only reports kFound if the buffer's newest entry is timestamped strictly after the requested window's end. For an object reading back itself, both sides of that comparison come from the same object — so the comparison is unwinnable by construction. Every such request is classified Delayed, then Timing-out once retries exhaust (def-tc-handler-2), throughout the run.

Looking at the data, TC fragments and objects are actually present and complete — n_inputs==0 is expected for these TC types (no TPG->TAG chain) — despite being tagged kIncomplete/kRequestWindowAfterBuffer. This isn't specific to Random TCs: Hardware_Signal fragments show the identical pattern, for the same structural reason.

Potential impact radius

???

Reason for change

Dashboards/alerting built on these counters are currently unusable for these sources — they show permanent failure regardless of actual system health, so real problems in these sources can't be distinguished from this expected-by-construction behavior.

Suggested implementations

• Override the request-handler's completeness check for TC(/TA) (the base classes are virtual and already overridden for TPs) to be object-aware: "does what we found already cover the requested window" instead of "is there something newer in the buffer." Fixes the single-source, no-merging case; doesn't generalize to multi-TC-source/merged-decision configs.
• Per-source watermarking: each source publishes its own "processed up to T" marker independently. Avoids the single-point-of-failure/cross-source-delay problem that broke the earlier in-stream heartbeat/zipper attempt in trigger, since there's no shared ordering/merge point.

• Other options (using time sync, override the readout request 'to take all that is available without retries' for this case, ...)

Testing suggestions

No response

Anything else?

No response