Repository navigation
benchmarking/evaluate: evaluate stored runs from the command line - #3592
Draft
Haowei Cai (Roy) (roycaihw) wants to merge 7 commits into
Draft
Haowei Cai (Roy) (roycaihw) wants to merge 7 commits into
Haowei Cai (Roy) (roycaihw) wants to merge 7 commits into
Conversation
This was referenced Nov 3, 2026
Haowei Cai (Roy) (roycaihw)
force-pushed
the
run-evaluations-d
branch
8 times, most recently
from
November 10, 2026 21:37
f5d2bfd to
ad82a7a
Compare
Benchmark pass/fail logic is spread over one-off tools. With this, a test entry declares `evaluations`, each a set of variables (source queries) and CEL conditions over their values, and the evaluator judges a run into a `Report`: pass, fail or invalid per evaluation. Nothing calls it yet. Conditions compile with `cel-expr-python` when the evaluations load. Sources are classes with three hooks (`sources/base.py`): `validate(cfg)` when the evaluations load, `init(run)` when the run starts, and `query(cfg)` after it. A variable whose source can't judge the run is invalid and is left out of the CEL data, so every condition that reads it is invalid and the evaluation can't pass. `run_results` reads one `stats.jsonl` row. AI use: I wrote this with an AI coding agent from the design, then reviewed the diff and checked the goodput numbers against the tools/goodput prototype on 248 Prow runs. Signed-off-by: Haowei Cai <haoweic@google.com>
Conditions are now plain CEL: integer literals are no longer rewritten, reserved names are left to CEL, and the evaluator no longer tracks which variables a condition reads. An invalid variable is left out of the CEL data and makes its evaluation invalid. The config shape is pydantic models. Signed-off-by: Haowei Cai <haoweic@google.com>
Haowei Cai (Roy) (roycaihw)
force-pushed
the
run-evaluations-d
branch
from
November 10, 2026 21:39
ad82a7a to
266a318
Compare
BenchmarkRunContext now carries the run's identity and artifact(name) instead of a file list, and the Report reads the identity from it. Each source chooses which artifacts to read, or looks the run up in its own store by identity. source_cfg is typed as JSON so non-JSON values or non-string keys fail at load. RunResults skips blank lines in stats.jsonl and marks the variable invalid on any malformed line. Conditions accept DYN so bare bool fields work, and evaluated_at converts to UTC before formatting. AI use: I wrote this with an AI coding agent from the design, then reviewed the diff. Signed-off-by: Haowei Cai <haoweic@google.com>
… from runs
A `cel` source reads run files with generic readers and evaluates standard CEL expressions into scalar values. Its `source_cfg` has `inputs` (each naming a reader and its options, read from this run or, with `test:`, from that test's run in the same invocation through `BenchmarkRunContext.paired_run`), `locals` (an ordered list of `{name, cel}` evaluated top-to-bottom), `checks` (what must hold for the values to mean anything), and `values` (doubles, bools, strings, or `null`). The source knows nothing about any benchmark's semantics: what is good, which run to compare against, and every formula live in the test entry.
`cel_env` enables the standard CEL math extension (`ext_math`, e.g. `math.least` and `math.greatest`) and adds one numeric helper, `percentile(xs, p)` over `list(double)` (raising on an empty list), alongside `round(x, digits)`. Loading compiles every expression with inferred local return types (including `timestamp` and `duration`). At run time, an unreadable input, a false check, or a local or value error makes the variable `invalid`.
Two generic readers expose run files as CEL data:
- `json`: parses one JSON object file (`data`, `uri`).
- `json_lines`: reads a JSON-lines or prefixed-JSON file (`lines`, `unparsed_lines`, `uri`) with optional `prefix`, `until` (stop-marker CEL predicate), `select` (record-filter CEL predicate), and `unique_by` (first-seen deduplication) so Stage 1 outputs bounded record lists before Stage 2 CEL runs.
Goodput is expressed purely in config over `json` and `json_lines` (README).
AI use: I wrote this with an AI coding agent from the design, then reviewed the diff.
Co-authored-by: Nishanth Kotla <nishanthkotla@google.com>
Signed-off-by: Haowei Cai <haoweic@google.com>
The orchestrator passes the full locust test entry as `--test-entry` JSON instead of forwarding only selected fields. Runner features such as observers or evaluations can read their sections directly without adding a new flag per field. AI use: I wrote this with an AI coding agent from the design, then reviewed the diff. Signed-off-by: Haowei Cai <haoweic@google.com>
Prow runs should record evaluations without a separate tool. `runner.py` reads `evaluations` from `--test-entry`, creates the sources before the load test, and queries them as its last post-run step. `outputs.py` writes the `Report` to `evaluation.json` and appends an `evaluation_<name>` row per evaluation and an `evaluations` summary row to `stats.jsonl`. A row keys each value as `<variable>__<value name>`, so a variable name can't contain `__` or end in `_`. When the evaluations don't load, the runner writes `error_report` with the error in `Report.error`. Evaluations never change `status.json` or the exit code. The locust image adds the evaluate package, `cel-expr-python` and `pydantic`. `rundir.py` finds the run a `cel` input's `test` names, such as the suite's 1-user run: the latest run of that test under the same `--dest` with the same tag, at most 6 h older, on the same cluster, and opens it with its own identity; a `gs://` run's artifacts are each downloaded when a source first asks for one. That needs `roles/storage.objectViewer` on the bucket, which a test-infra change grants. Without it, the variables with such an input are invalid, and the reason says why. AI use: I wrote this with an AI coding agent from the design, then reviewed the diff. Signed-off-by: Haowei Cai <haoweic@google.com>
`python -m evaluate` judges stored runs, local or `gs://`, with the runner's code, to try new thresholds or to backfill older runs. A run in the runner's layout gets the same paired runs as in the runner. Outputs go to `--out`, never into the stored run. `--check-config` validates a tests file offline. AI use: I wrote this with an AI coding agent from the design, then reviewed the diff. Signed-off-by: Haowei Cai <haoweic@google.com>
Haowei Cai (Roy) (roycaihw)
force-pushed
the
run-evaluations-d
branch
from
November 12, 2026 06:04
266a318 to
1a778e8
Compare
yerramasu
pushed a commit
to Tracefort-io/substrate
that referenced
this pull request
Nov 12, 2026
…ent-substrate#3551) Part of agent-substrate#3298 ([design](https://docs.google.com/document/d/1wcO8150RH9bjHulYaAvyW0IrbX2raXCREDYOurQavq4/edit?tab=t.v8rzfkkd2f1j&resourcekey=0-COU9pclBVTxrcs4hGnyI7g)). First of four PRs: agent-substrate#3551 (this), agent-substrate#3550, agent-substrate#3552, agent-substrate#3592. Benchmark pass/fail logic is spread over one-off tools. With this, a test entry declares `evaluations`, each a set of variables (source queries) and CEL conditions over their values, and the evaluator judges a run into a `Report`: pass, fail or invalid per evaluation. Nothing calls it yet. agent-substrate#3552 wires it into the runner and writes `evaluation.json` and the `stats.jsonl` rows. Conditions compile with `cel-expr-python` when the evaluations load. Sources are classes with three hooks (`sources/base.py`): `validate(cfg)` when the evaluations load, `init(run)` when the run starts, and `query(cfg)` after it. A variable whose source can't judge the run is invalid and is left out of the CEL data, so every condition that reads it is invalid and the evaluation can't pass. `run_results` reads one `stats.jsonl` row. The config is pydantic models. `source_cfg: null` is an error; leave the key out for `{}`. AI use: I wrote this with an AI coding agent from the design, then reviewed the diff and checked the goodput numbers against the agent-substrate#2526 prototype on 248 Prow runs. --------- Signed-off-by: Haowei Cai <haoweic@google.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #3298. Last of four: #3551, #3550, #3552, #3592 (this). Stacked on #3552, so please review only the last commit.
python -m evaluatejudges stored runs, local orgs://, with the runner's code, to try new thresholds or to backfill older runs. A run in the runner's layout gets the same paired runs as in the runner, andgs://runs download only the artifacts sources request on demand. Outputs go to--out, never into the stored run.--check-configvalidates a tests file offline. CLI unit tests (test_cli.py) use genericjson_linesinputs and ordered listlocals.AI use: I wrote this with an AI coding agent from the design, then reviewed the diff and checked the goodput numbers against the #2526 prototype on 248 Prow runs.