[philrz]

This provides a way to easily run the GitHub Archive Sample queries to help us spot perf regressions early. The perf/canary/README.md has all the details, but in brief:

1. You'll be able to make perf to compare your local work with the main commit your branch started from
2. It'll run automatically in Actions when you put up PRs, comparing the PR merged into main with the current main, in parallel with the existing CI
3. It'll run automatically in Actions when PRs merge to main

These all produce Markdown reports that highlight significant changes in time or memory usage, as well as any change in query results.

It never fails a build and isn't a required check. On PRs it takes about the same time as the existing tests, and it skips changes that only touch book/ or Markdown files.

Examples

I've got examples in a personal fork that show how this would have surfaced some of our recent regressions:

Issue	Report
#7480	report
#7484	report
#7503	report

There's also an open PR philrz#2 that shows how it would look during the PR workflow. Notice the "neutral check" with the link to the report. We could turn this up to a fully "red" failure so it's harder to ignore, but I figured we'd start this out gently until we have confidence that it's not surfacing false positives.

Code

Disclosure: Claude wrote the Go code. I've tested it locally and on my fork (examples above).

[nwt]

Looks fine but I'll leave approval to @mattnibs and @mccanne so they can review.

[philrz]

Thanks @mattnibs. I've pushed a commit that should address your feedback.

Is there a way to run this on a individual query?

I just added that. For instance now you can do make perf SYSTEMS=super-bsup QUERIES=union to query just that "union" query on BSUP input, or make perf QUERIES=union to run the "union" query on all inputs BSUP/Parquet/JSON/etc.

The data set gets cached right (i.e., not download on every run?)

Correct. On macOS for instance, they land in ~/Library/Caches/super-perf-canary/. In Actions it gets cached between runs too. The BSUP is not cached because that can change between builds so it doesn't try to get fancy about keeping that and instead regenerates it in any run where BSUP needs to be queried.

You shouldn't need the -dynamic flag here and if it adversely affects load performance we want to know about this. Also don't need the -f bsup flag since this inferred from the output that name.

I've made those changes. FWIW, the -dynamic was muscle memory from my pre-#7292 benchmark work when we were still doing an extra read through JSON inputs to generate type checking info, so I got into the habit of explicitly invoking -dynamic when converting JSON since I had no query to type check.

---

I also pushed a change that may mark some results as "noisy", which catches situations like what I just bumped into when testing locally on my laptop. I happened to launch my email client in the middle of a run, which made my laptop start running slower, and it basically created a mix of good results and garbage results that made the deltas hard for the scripts to compare, i.e., they looked to me like there might be a regression, but the scripts were not previously flagging it as "red", and it was because it already had logic to ignore that situation but wasn't showing its work. These still aren't flagged as regressions, but they're now marked "noisy" in the results with a tip to re-run. Hopefully these won't come up on calm laptops or Actions runs. Re-checking the ~20 earlier runs from my testing, most of them in Actions, only my laptop run got marked. But Actions runners are VMs, so I guess we can't rule out "noisy neighbor"/infra effects there too.

---

All of the above are also covered in the perf/canary/README.md.