benchmarking/evaluate: a cel source that evaluates CEL over data read from runs by roycaihw · Pull Request #3550 · agent-substrate/substrate

Haowei Cai (Roy) (roycaihw) · 2026-11-02T17:10:59Z

Part of #3298. Second of four: #3551 (merged), #3550 (this), #3552, #3592. Supersedes #2526 (Nishanth's tools/goodput, co-authored here).

A cel source reads run files with generic readers and evaluates standard CEL expressions into scalar values. Its source_cfg (pydantic models; each reader checks its own options) has:

inputs: each naming a generic reader and its options, read from this run or, with test:, from that test's run in the same invocation through BenchmarkRunContext.paired_run:
- json: parses one JSON object file (data, uri).
- json_lines: reads a JSON-lines or prefixed-JSON file (lines, unparsed_lines, uri) with optional prefix, until (stop-marker CEL predicate), select (record-filter CEL predicate), and unique_by (first-seen deduplication) so readers return bounded record lists before CEL runs.
locals: an ordered list of {name, cel} evaluated top-to-bottom. A local that fails is left out, so only what reads it fails, with the local's error.
checks: CEL predicates that must hold for the values to mean anything.
values: scalar outputs (doubles, bools, strings, or null).

The source knows nothing about any benchmark's semantics: what is good, which run to compare against, and every formula live in the test entry. Goodput is expressed purely in config over json and json_lines (benchmarking/evaluate/README.md).

cel_env enables the standard CEL math extension (ext_math, e.g. math.least and math.greatest) and adds one numeric helper, percentile(xs, p) over list(double) (raising on an empty list), alongside round(x, digits). Loading compiles every expression with inferred local return types (including timestamp and duration). At run time, an unreadable input, a false check, or a value error makes the variable invalid.

Changes from #2526:

A SuspendActorAsync failed line (async suspend, the default since benchmarking: rename --lifecycle-mode values to sync/async and default to async #3097) counts as a suspend failure.
config.json's modes become generic cel inputs, locals, checks, and values in the test entry. Strict and relaxed are two evaluations.
No static baseline identity, baseline files, CLI or GCS code.
Every value is present on every valid run, null when there is nothing to measure.

Written this way, goodput matches the patched #2526 prototype on 245 stored Prow sweperf runs (and matches across all 252 stored primary runs in parity checks; runs without a paired 1-user run report invalid).

AI use: I wrote this with an AI coding agent from the design, then reviewed the diff and checked the goodput numbers against the #2526 prototype on stored Prow runs.

…ent-substrate#3551) Part of agent-substrate#3298 ([design](https://docs.google.com/document/d/1wcO8150RH9bjHulYaAvyW0IrbX2raXCREDYOurQavq4/edit?tab=t.v8rzfkkd2f1j&resourcekey=0-COU9pclBVTxrcs4hGnyI7g)). First of four PRs: agent-substrate#3551 (this), agent-substrate#3550, agent-substrate#3552, agent-substrate#3592. Benchmark pass/fail logic is spread over one-off tools. With this, a test entry declares `evaluations`, each a set of variables (source queries) and CEL conditions over their values, and the evaluator judges a run into a `Report`: pass, fail or invalid per evaluation. Nothing calls it yet. agent-substrate#3552 wires it into the runner and writes `evaluation.json` and the `stats.jsonl` rows. Conditions compile with `cel-expr-python` when the evaluations load. Sources are classes with three hooks (`sources/base.py`): `validate(cfg)` when the evaluations load, `init(run)` when the run starts, and `query(cfg)` after it. A variable whose source can't judge the run is invalid and is left out of the CEL data, so every condition that reads it is invalid and the evaluation can't pass. `run_results` reads one `stats.jsonl` row. The config is pydantic models. `source_cfg: null` is an error; leave the key out for `{}`. AI use: I wrote this with an AI coding agent from the design, then reviewed the diff and checked the goodput numbers against the agent-substrate#2526 prototype on 248 Prow runs. --------- Signed-off-by: Haowei Cai <haoweic@google.com>

… from runs A `cel` source reads run files with generic readers and evaluates standard CEL expressions into scalar values. Its `source_cfg` has `inputs` (each naming a reader and its options, read from this run or, with `test:`, from that test's run in the same invocation through `BenchmarkRunContext.paired_run`), `locals` (an ordered list of `{name, cel}` evaluated top-to-bottom), `checks` (what must hold for the values to mean anything), and `values` (doubles, bools, strings, or `null`). The source knows nothing about any benchmark's semantics: what is good, which run to compare against, and every formula live in the test entry. `cel_env` enables the standard CEL math extension (`ext_math`, e.g. `math.least` and `math.greatest`) and adds one numeric helper, `percentile(xs, p)` over `list(double)` (raising on an empty list), alongside `round(x, digits)`. Loading compiles every expression with inferred local return types (including `timestamp` and `duration`). At run time, an unreadable input, a false check, or a local or value error makes the variable `invalid`. Two generic readers expose run files as CEL data: - `json`: parses one JSON object file (`data`, `uri`). - `json_lines`: reads a JSON-lines or prefixed-JSON file (`lines`, `unparsed_lines`, `uri`) with optional `prefix`, `until` (stop-marker CEL predicate), `select` (record-filter CEL predicate), and `unique_by` (first-seen deduplication) so Stage 1 outputs bounded record lists before Stage 2 CEL runs. Goodput is expressed purely in config over `json` and `json_lines` (README). AI use: I wrote this with an AI coding agent from the design, then reviewed the diff. Co-authored-by: Nishanth Kotla <nishanthkotla@google.com> Signed-off-by: Haowei Cai <haoweic@google.com>

Haowei Cai (Roy) (roycaihw) · 2026-11-12T17:00:57Z

Max Smythe (@maxsmythe) Nishanth Kotla (@Nishanth29) This PR is ready for review. PTAL

Max Smythe (maxsmythe) · 2026-11-12T21:09:53Z

+source: cel
+source_cfg:
+  inputs:                    # the data, each a CEL variable
+    stats: {reader: json_lines, file: stats.jsonl}


for consistency, should this be a list of objects with "name" as a key?

Max Smythe (maxsmythe) · 2026-11-12T21:19:32Z

+        return tuple(Expression(kind, name, env.compile(text, f"{kind} {name!r}", gives))
+                     for name, text in section.items())
+
+    return Program(c.inputs, tuple(local_exprs), compiled("check", c.checks, (cel.Type.BOOL, cel.Type.DYN)),


Should this use kwarg assignment for clarity and to help avoid brittleness?

Max Smythe (maxsmythe) · 2026-11-12T21:34:15Z

+    reader: json_lines
+    file: logs.txt
+    prefix: "[boomer] "
+    until: "has(line.msg) && line.msg == 'shutdown'"


We should probably have some kind of typed sub-object config per-reader, for consistency with rest of API

Max Smythe (maxsmythe) · 2026-11-12T21:37:46Z

+                data = reader().read(self._run(i.test), i.options)
+                check_data(reader, data)
+                self._reads[key] = data
+            except (LookupError, ReadError) as e:


🤖 should-fix 🟡 – LookupError is the base of KeyError and IndexError, so a bug in a reader or in check_data is cached and reported as input 'x': 'key' instead of failing loudly. Catch the LookupError where paired_run raises it, in _run, and turn it into a ReadError there.

Max Smythe (maxsmythe) · 2026-11-12T21:37:47Z

+                   compiled("value", c.values, (*VALUE_TYPES, cel.Type.DYN)))
+
+
+def run_program(program: Program, data: dict[str, Any], *, enforce_checks: bool = True) -> dict[str, Scalar]:


🤖 nit 🟢 – enforce_checks is never passed; every caller takes the default. Drop it until something needs it.

Max Smythe (maxsmythe) · 2026-11-12T21:37:49Z

+
+    def query(self, cfg: Mapping[str, Any]) -> Mapping[str, Scalar]:
+        program = compile_cfg(cfg)
+        data = {name: self._read(name, i) for name, i in program.inputs.items()}


🤖 question 🟢 – Every input is read before anything is evaluated, so an input nothing reads still makes the variable invalid when its file or paired run is missing. Locals fail lazily. Is the eager failure intended for inputs shared through an anchor, as the docstring on test_inputs_and_locals_need_not_be_read suggests?

Max Smythe (maxsmythe) · 2026-11-12T21:37:50Z

+
+    # Compile locals top-to-bottom so each local can read inputs and earlier
+    # locals with their inferred CEL return type (including TIMESTAMP/DURATION).
+    var_types: dict[str, cel.Type] = {name: cel.Type.Map(cel.Type.STRING, cel.Type.DYN) for name in c.inputs}


🤖 question 🟢 – Inputs are typed map(string, dyn), so Reader.fields only drives check_data at run time, and a typo like stats.line passes config load. Was the plan to use the declared fields for load-time type checks too? If not, the README's "with their CEL types" overstates what fields does.

This was referenced Nov 2, 2026

benchmarking/evaluate: run evaluations, with a run_results source #3551

Merged

benchmarking: run the test entry's evaluations with each run #3552

Draft

Benchmark Success Gating #3298

Open

Haowei Cai (Roy) (roycaihw) force-pushed the run-evaluations-a branch 2 times, most recently from 875daf8 to 00b641b Compare November 3, 2026 17:56

Haowei Cai (Roy) (roycaihw) mentioned this pull request Nov 3, 2026

benchmarking/evaluate: evaluate stored runs from the command line #3592

Draft

Haowei Cai (Roy) (roycaihw) changed the title ~~benchmarking/evaluate: goodput as the first evaluation source~~ benchmarking/evaluate: goodput as an evaluation source Nov 3, 2026

Haowei Cai (Roy) (roycaihw) force-pushed the run-evaluations-a branch from 00b641b to bf6be15 Compare November 4, 2026 03:56

Nishanth Kotla (Nishanth29) mentioned this pull request Nov 4, 2026

benchmarking/sweperf: optionally retry ResumeActor when no worker has room #3540

Open

2 tasks done

Haowei Cai (Roy) (roycaihw) force-pushed the run-evaluations-a branch 3 times, most recently from 8088919 to 8c37954 Compare November 4, 2026 16:54

Haowei Cai (Roy) (roycaihw) changed the title ~~benchmarking/evaluate: goodput as an evaluation source~~ benchmarking/evaluate: a cel source that judges records with CEL rules Nov 4, 2026

Haowei Cai (Roy) (roycaihw) force-pushed the run-evaluations-a branch 6 times, most recently from 8aaf3bc to 096745b Compare November 10, 2026 21:39

Haowei Cai (Roy) (roycaihw) force-pushed the run-evaluations-a branch from 096745b to 12bdb8a Compare November 12, 2026 06:04

Haowei Cai (Roy) (roycaihw) changed the title ~~benchmarking/evaluate: a cel source that judges records with CEL rules~~ benchmarking/evaluate: a cel source that evaluates CEL over data read from runs Nov 12, 2026

Haowei Cai (Roy) (roycaihw) force-pushed the run-evaluations-a branch from 12bdb8a to b884cd3 Compare November 12, 2026 16:50

Haowei Cai (Roy) (roycaihw) marked this pull request as ready for review November 12, 2026 17:00

Max Smythe (maxsmythe) reviewed Nov 12, 2026

View reviewed changes

Navigation Menu

Repository navigation

benchmarking/evaluate: a cel source that evaluates CEL over data read from runs - #3550

benchmarking/evaluate: a cel source that evaluates CEL over data read from runs#3550
Haowei Cai (Roy) (roycaihw) wants to merge 1 commit into
agent-substrate:mainfrom
roycaihw:run-evaluations-a

Haowei Cai (Roy) (roycaihw) commented Nov 2, 2026 •

edited

Loading

Uh oh!

Haowei Cai (Roy) (roycaihw) commented Nov 12, 2026

Uh oh!

Max Smythe (maxsmythe) Nov 12, 2026

Uh oh!

Max Smythe (maxsmythe) Nov 12, 2026

Uh oh!

Max Smythe (maxsmythe) Nov 12, 2026 •

edited

Loading

Uh oh!

Max Smythe (maxsmythe) Nov 12, 2026

Uh oh!

Max Smythe (maxsmythe) Nov 12, 2026

Uh oh!

Max Smythe (maxsmythe) Nov 12, 2026

Uh oh!

Max Smythe (maxsmythe) Nov 12, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		compiled("value", c.values, (*VALUE_TYPES, cel.Type.DYN)))


		def run_program(program: Program, data: dict[str, Any], *, enforce_checks: bool = True) -> dict[str, Scalar]:

Conversation

Haowei Cai (Roy) (roycaihw) commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Haowei Cai (Roy) (roycaihw) commented Nov 12, 2026

Uh oh!

Max Smythe (maxsmythe) Nov 12, 2026

Choose a reason for hiding this comment

Uh oh!

Max Smythe (maxsmythe) Nov 12, 2026

Choose a reason for hiding this comment

Uh oh!

Max Smythe (maxsmythe) Nov 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Max Smythe (maxsmythe) Nov 12, 2026

Choose a reason for hiding this comment

Uh oh!

Max Smythe (maxsmythe) Nov 12, 2026

Choose a reason for hiding this comment

Uh oh!

Max Smythe (maxsmythe) Nov 12, 2026

Choose a reason for hiding this comment

Uh oh!

Max Smythe (maxsmythe) Nov 12, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Haowei Cai (Roy) (roycaihw) commented Nov 2, 2026 •

edited

Loading

Max Smythe (maxsmythe) Nov 12, 2026 •

edited

Loading