Repository navigation
[ET-VK][test] Run operator benchmarks from one etvk_bench binary - #23725
Merged
Merged
Conversation
Pull Request resolved: #23716 ## Ulterior Motive Keep `VulkanSwiGLUTest` correctness checks usable on Vulkan devices that flush FP16 subnormal values to zero. ## Rationale **What**: Increase the absolute error allowances in `MatchesUnfusedAcrossResizes` to account for FP16 underflow. **Why**: On stable, all 28 FP16 configurations fail on Adreno 750 while FP32 passes. The first expected output is `5.30033e-6`, below FP16's minimum normal value, but the shader returns zero. The unfused path can also flush subnormal inputs and intermediate tensors before later multiplications amplify the difference. ## Details - Fused versus CPU reference: - **Before**: Absolute allowance `1e-7`. - **After**: Absolute allowance `2^-14`, FP16's minimum normal value; relative allowance remains `1e-3`. - Fused versus unfused graph: - **Before**: Absolute allowance `6e-8 * (abs(x*y) + abs(y) + 1)` assumes subnormal preservation. - **After**: Keep the original rounding allowance and add `(1 + 1e-3) * abs(reference)` only when an FP16 input is subnormal or a computed intermediate/output can round below the minimum normal value. Use the existing `3e-3` relative tolerance as the rounding margin at that boundary. - If the unfused result flushes to zero, its relative tolerance vanishes. `abs(reference)` covers the lost output, and the extra `1e-3` covers the fused result's relative error. - Cases without underflow keep the small original absolute allowance: `x=20, y=2` gives `2.58e-6`; the original tiny-output case gives about `6.63e-6`. - FP32 uses its existing assertions. Both FP16 comparisons keep `ASSERT_NEAR`. Differential Revision: [D124833698](https://our.internmc.facebook.com/intern/diff/D124833698/) ghstack-source-id: 446951678
Pull Request resolved: #23490 ## Ulterior Motive Reorganize the flat Vulkan shader directory by operator implementation route before landing additional convolution implementations such as Winograd. ## Rationale **What**: Make shader code generation and build discovery preserve nested source paths. **Why**: The generator discovered nested files but identified and emitted them by basename. Repeated basenames overwrote each other, YAML templates were globally keyed, generated files were flattened, and build inputs excluded nested files. ## Details - Shader identity - **Before**: Source, YAML template, generated source, cache, and SPIR-V paths were basename-based. - **After**: Paths are relative to each shader source root and preserve directories through generation and caching. - Includes - **Before**: Cache invalidation resolved includes only from the generated output root. - **After**: Includes resolve from the including shader's directory, then each ancestor directory up to the generated output root, nearest first. `glslc` receives the same include path list, so route subdirectories (e.g. `convolution/conv2d/im2col/`) can share family-level headers (e.g. `convolution/conv2d/conv2d_common.glslh`). - Build discovery - **Before**: Buck and CMake shader filegroups selected only direct children. - **After**: Runtime, test, custom-op, and tool shader filegroups recursively include nested files. Runtime shader registry names remain flat and unchanged. Duplicate runtime shader names now fail during code generation instead of silently overwriting an earlier source. ghstack-source-id: 446951693 @exported-using-ghexport Differential Revision: [D123101914](https://our.internmc.facebook.com/intern/diff/D123101914/)
Pull Request resolved: #23491 ## Ulterior Motive Organize Vulkan operators by implementation family so operators with multiple routes can use clear, scalable directory layouts. ## Rationale **What**: Move standard binary GLSL, YAML, shared shader definitions, and C++ implementations into dedicated `binary/` directories. **Why**: The flat shader and implementation directories obscure ownership as operator families grow. Binary operators provide a contained validation of nested-source support before reorganizing larger families such as convolution. ## Details - Move tensor-tensor and tensor-scalar shader templates, YAML variant definitions, and `binary_op_defs.glslh` under `glsl/binary/`, and `q8ta_binary` under `glsl/binary/q8ta/`. - Move `BinaryOp.cpp` and `BinaryScalarOp.cpp` under `impl/binary/`, and the quantized `Q8taBinary.{h,cpp}` under `impl/binary/q8ta/`. - Update shader-name validation to address path-qualified template identities, the `Q8taBinary.h` include in `TestQ8taBinary.cpp`, and the binary section labels in `op_registry.py`. - Moved files are recorded as renames. - Preserve generated shader names and runtime dispatch behavior. ghstack-source-id: 446951706 @exported-using-ghexport Differential Revision: [D123101915](https://our.internmc.facebook.com/intern/diff/D123101915/)
Pull Request resolved: #23492 ## Ulterior Motive Organize Vulkan operators by implementation family so operators with multiple routes can use clear, scalable directory layouts. ## Rationale **What**: Move convolution C++ implementations, GLSL templates, YAML variant definitions, and convolution-only shader headers under `convolution/` in both `impl/` and `glsl/`, split into `conv1d/` and `conv2d/`. Split the monolithic `Convolution.cpp` and `Q8taConv2d.{h,cpp}` so each file holds either routing logic or a single implementation route. **Why**: Convolution is implemented through several routes (direct, depthwise, pointwise, im2col/GEMM, transposed), and the q8ta quantized family repeats most of them. In the flat directories their sources were interleaved by name, and single files mixed routing with several route implementations. This prepares for adding further routes such as Winograd. ## Details - Layout (`impl/` and `glsl/` mirror each other) - `convolution/`: `Convolution.{h,cpp}` registers `aten.convolution` and dispatches to conv1d or conv2d by input rank; it also holds `prepack_biases` and `check_conv_args`, used by both. - `convolution/conv1d/`: `Conv1d.{h,cpp}` (`conv1d_impl` routing), `Conv1dDirect.{h,cpp}`, `Conv1dDW.cpp`, `Conv1dPW.cpp`, and the conv1d shaders. - `convolution/conv2d/`: `Conv2d.{h,cpp}` (`conv2d_impl` routing, `Conv2dMethod`, `resize_conv2d_node`), `Conv2dDirect.{h,cpp}` (direct and transposed shader dispatch), `Conv2dDW.cpp`, `Conv2dPW.cpp`, `ConvolutionUtils.h`, `conv2d_common.glslh`, and the direct/depthwise/pointwise shaders. - `convolution/conv2d/{im2col,transposed}/`: im2col/GEMM sources and shaders; transposed shaders. - `convolution/conv2d/q8ta/`: `Q8taConv2d.{h,cpp}` (`q8ta_conv2d` routing and the im2col heuristic, formerly `Q8taConv2dRoute`), `Q8taConv2dCommon.{h,cpp}` (`ActivationType`, packed-dim checks), `Q8taConv2dDirect.{h,cpp}` (general route), `Q8taConv2dDW.{h,cpp}`, `Q8taConv2dPW.{h,cpp}`, plus `im2col/` and `transposed/`. Each route has its own header; consumers include only what they use. - `convolution/conv2d/quantized_legacy/`: `QuantizedConvolution.{h,cpp}` and its shaders and headers. - Weight packing shaders sit in `pack_weights/` subdirectories of the route that dispatches them (`conv2d/`, `conv2d/im2col/`, `conv2d/q8ta/`, `conv2d/transposed/`). - Naming - Functions that add a `DynamicDispatchNode` are named `add_*_node()`; functions that call `add_*_node()` to implement an operator are named `*_impl()`. Registered operator names are unchanged. - `add_conv2d_node` is folded into `conv2d_impl`. Its test-only `force_direct` flag is removed; the conv2d test's `direct` selector calls `conv2d_direct_impl` directly, which is equivalent for its sliding-window cases. - Includes - Route shaders include `conv2d_common.glslh` unchanged through ancestor include resolution. - Root shaders `quantize_and_pack_4w4c` and `unpack_4w4c_and_dequantize` include `convolution/conv2d/conv2d_common.glslh`. - Headers shared with other operator families (`linear_*`, `conv2d_fp_input_tile_load.glslh`, indexing headers) stay at the shader root. - Moved files are recorded as renames; files split from or merged into another file are recorded as copies of their source, so reviews show moved code as unchanged. - Moving code into new files exposes it to CLANGTIDY, so its findings there are fixed: `utils::safe_downcast<int32_t>` for narrowing, `(void)` for unused parameters, an anonymous namespace for the file-local `Conv2dParams`, and removal of an unused `ShaderNameUtils.h` include. Generated shader names and runtime dispatch behavior are unchanged. ghstack-source-id: 446951717 @exported-using-ghexport Differential Revision: [D123450342](https://our.internmc.facebook.com/intern/diff/D123450342/)
Pull Request resolved: #23493 ## Ulterior Motive Organize Vulkan operators by implementation family so operators with multiple routes can use clear, scalable directory layouts. ## Rationale **What**: Move the linear and matrix multiplication C++ implementations, GLSL templates, YAML variant definitions, and GEMM tile headers under `gemm/` in both `impl/` and `glsl/`, split into `linear/` (an activation times a usually prepacked weight) and `matmul/` (multiplication between live activations). Quantized linear sources are grouped by how they are quantized, and the multi-scheme `QuantizedLinear.cpp` and the shared `GemmCoopmat.cpp` are split accordingly. **Why**: Linear and matmul sources were interleaved by name in the flat directories, quantized linear variants for different schemes shared one file, and the shared GEMM tile headers were scattered across the shader root, which obscured which sources implement which operator, route, and scheme. ## Details - Layout (`impl/` and `glsl/` mirror each other) - `gemm/`: `GemmCommon.{h,cpp}` (fp weight prepack, resize helpers, and the cooperative-matrix tile constants and `is_coopmat_eligible()` shared by linear and matmul) and `coopmat_mm`, the cooperative-matrix shader shared by both. - `gemm/tile_utils/`: all GEMM tile headers (tile structs, tile load/store/compute, block packing, accumulators, and scale/sum/bias loaders), also used by convolution, SDPA, and quantize shaders. - `gemm/linear/`: `Linear.{h,cpp}` (`aten.linear`), `LinearCoopmat.{h,cpp}`, and `backward/LinearDw.cpp` (`et_vk.linear_dW`). - `gemm/linear/q8ta/`: `Q8taLinear` and `Q8taLinearGemv`; activations and weights are quantized ahead of time with static parameters (primarily statically quantized convolutional models). - `gemm/linear/qw/`: weight-only quantization, with activations kept in floating point or quantized dynamically (primarily LLMs and transformers). `QuantizedLinear.{h,cpp}` hold the shared weight prepacking, workgroup sizing, the weight-only node, and `quantized_linear_impl`; schemes live in `q4gsw/` (with `backward/` for `et_vk.linear_q4gsw_backward` and `et_vk.q4gsw_requant`), `dq8ca_q4gsw/`, `q8csw/`, and `legacy/` (`QuantizedLinearQCSNW` and the `q8ta_q8csw` implementation that runs through `quantized_linear_impl`). - `gemm/matmul/`: `MatMul.{h,cpp}` (`aten.mm`, `aten.bmm`, `aten.addmm`; renamed from `Matmul.cpp` to match its header) and `MatmulCoopmat.{h,cpp}`. - Weight packing shaders sit in `pack_weights/` subdirectories of the folder that dispatches them (`linear/`, `linear/qw/`, `linear/qw/q4gsw/`, and the shader root for `pack_int4_linear_weight_transposed_interleaved`). - Splits - `QuantizedLinear.cpp`: `q8csw/Q8cswLinear.cpp` (`et_vk.linear_q8csw`, `aten._weight_int8pack_mm`), `legacy/Q8taQ8cswLinear.{h,cpp}` (`add_linear_qa_qw_node`, `et_vk.linear_q8ta_q8csw`), and `dq8ca_q4gsw/Dq8caQ4gswLinear.{h,cpp}` (`add_linear_dqa_qw_node`, `et_vk.linear_dq8ca_q4gsw`). `can_use_q4gsw_coopmat` and the coopmat tile constants are now declared in `QuantizedLinear.h` because the shared and `dq8ca_q4gsw` files both use them. - `GemmCoopmat.cpp`: `linear/LinearCoopmat.cpp` and `matmul/MatmulCoopmat.cpp`. - Functions move unchanged. - Includes - Shaders inside `gemm/` include tile headers as `tile_utils/<header>`; shaders outside `gemm/` include them as `gemm/tile_utils/<header>`. - C++ include paths and `op_registry.py` section labels are updated. - Moved files are recorded as renames, and split files as copies of their source. - Code that moves within or between files is re-linted, so its CLANGTIDY findings are fixed: `(void)` for unused parameters, `utils::safe_downcast` for narrowing in `quantized_linear_gwg` and the coopmat size reads, bounds-checked `.at()` in place of `.back()` in `LinearCoopmat.cpp`, and removal of unused includes (`Softmax.h` in `SDPA.cpp`, `Linear.h` in `Conv1dPW.cpp`, and two utility headers in the coopmat files). Generated shader names and runtime dispatch behavior are unchanged. ghstack-source-id: 446951723 @exported-using-ghexport Differential Revision: [D123523650](https://our.internmc.facebook.com/intern/diff/D123523650/)
Pull Request resolved: #23597 ## Ulterior Motive Restructure the Vulkan operator test+benchmark binaries around a single `etvk_benchmark` entry point with registered, per-operator test case sets, so related cases (such as a model's bottleneck shapes) can be grouped and run in isolation. ## Rationale **What**: Move the test+benchmark binaries, their shared utilities, and the test-only ops and shaders from `test/custom_ops/` into `test/ops/benchmark/`, with one subdirectory per operator under `cases/`, and split the monolithic `utils.{h,cpp}` by topic. **Why**: The flat `custom_ops/` directory mixes benchmark binaries, framework code, test-only ops, and correctness-only gtests, and `utils.{h,cpp}` (3.5k lines) mixes value specs, test cases, execution, and reporting. Grouping the binaries by operator gives the upcoming case-set registration a home for each operator's cases and reference code. ## Details - Layout under `test/ops/benchmark/` - `framework/`: test framework utilities, `conv2d_utils`, `cm_utils`, and `test_coopmat_probe`. - `test_ops/impl/` and `test_ops/glsl/`: test-only ops and shaders, formerly `impl/` and `glsl/`. - `cases/`: one directory per operator, `conv2d/` (direct, depthwise, pointwise), `conv1d/`, `q8csw_conv2d/`, `q4gsw_linear/` (including `test_fpa_q4gsw_linear` and `test_coopmat_linear_bench`), `q8csw_linear/`, `mm/`, `sdpa/`, `embedding_q4gsw/`, `choose_qparams_per_row/`, `add/`; `cases/q8ta/` collects the q8ta operators: `conv2d/`, `conv2d_transposed/`, `linear/`, `binary/`, `unary/`, `clone/`, `pixel_shuffle/`, `qdq/`. - `utils.{h,cpp}` split - `config`: global output and debugging options. - `labels.h`: test case naming helpers. - `value_spec`: `ValueSpec`, data generation, half-precision conversion, and `print_valuespec_data`. - `test_case`: `ReferenceKey`, the shader filter defaults, and `TestCase`. - `results`: `BenchmarkResult`, `TestResult`, and table printing. - `runner`: graph setup, `RepeatedGraphExecutor`, and `execute_test_case(s)`. - `weight_utils`: weight canvases and weight sum helpers. - `utils.h` includes all of the above, so binaries are unchanged. - Moving code into new files exposes it to CLANGTIDY, so its findings are fixed: braces around single-statement bodies, `explicit` on the single-argument `BenchmarkResult`, `TestResult`, and `TestCase` constructors, `const&` for the generator, reference, FLOP calculator, and `op_name` parameters, `utils::safe_downcast<int>` for one narrowing, `(void)` for an unused parameter, and `NOLINTNEXTLINE` on enum `switch`es that list only some enumerators (each has a `default`) and on the `using namespace vkcompute` that binaries rely on. `utils.h` no longer includes `<iostream>`. - Build - `test/ops/benchmark/{BUCK,targets.bzl}` define the binaries, `prototyping_utils`, `operator_implementations`, and `custom_ops_shaderlib`. Binary target names are unchanged; only the package path changes. - The CMake project and `build_and_run.sh` move with the binaries and take source paths from the new subdirectories. - The correctness-only gtests (`utils_test`, `q8ta_conv2d_route_test`, `q8ta_conv2d_stream_plan_test`) stay in `test/custom_ops/`; `utils_test` now depends on `prototyping_utils` in its new package. - Binaries include framework headers by full path (`executorch/backends/vulkan/test/ops/benchmark/framework/utils.h`). - CI references move to the new package: the GPU-host and Eureka Skycastle workflows, and the OSS `vulkan.yml` and `pull.yml` workflows. - Moved files are recorded as renames; the new `BUCK`/`targets.bzl` and the split framework files are recorded as copies. Binary sources, test cases, and test-only ops are otherwise unchanged. ghstack-source-id: 446951727 @exported-using-ghexport Differential Revision: [D123690842](https://our.internmc.facebook.com/intern/diff/D123690842/)
Pull Request resolved: #23598 ## Ulterior Motive Make the Vulkan operator benchmarks easy to drive during optimization work: group related test cases (such as one model's bottleneck shapes) into named sets, run any subset from one binary, and later sweep implementations and dispatch parameters. ## Rationale **What**: Replace the 24 per-operator test+benchmark binaries with one entry point, `etvk_bench`, that runs test case sets registered by files under `test/ops/benchmark/cases/<op>/`. Split each operator's code into shared helpers, CPU references, and one file per set, and split each binary's default case list into correctness and performance sets. **Why**: Each binary hardcoded one mixed list of correctness-sized and performance-sized cases, so running a subset meant editing code, and modes were spread across ad hoc CLI flags, environment variables, and `#ifdef DEBUG_MODE`. ## Details - Framework - `framework/registry.{h,cpp}`: `TestCaseSet` (test cases, reference, FLOP calculator, run counts, or a custom test for checks that are not test cases) and `REGISTER_TEST_CASE_SET("<op>", "<set>") { ... }`. The registered body runs only when the set is selected, since building test cases may query the device. `execute_test_cases()` takes the cases directly instead of a generator. - `etvk_bench.cpp` parses gflags: `--list`, `--op=<op>[,...]` (a parent directory such as `q8ta` selects every operator beneath it), `--set=<set>[,...]`, `--filter='<a>;<b>|<c>'` (cases whose name contains every `;`-separated substring of at least one `|`-separated alternative), `--chain=<n>`, `--nogpu_timestamps`, `--print_output`, `--print_latencies`, `--debug`, `--device_info`, and `--helpshort`. It runs every selected set, continues past failures, and exits nonzero if any set failed. gflags comes from `external_deps` in Buck and, in the standalone CMake project, from the main ExecuTorch build tree (as the Llama runner does). - `--device-info` replaces the `test_coopmat_probe` binary (`framework/device_info.{h,cpp}`, renamed from it). - Case directories: each `cases/<op>/` has `<op>.h` (configs and shared declarations), `<op>.cpp` (config to `TestCase` builders, CPU references, and FLOP calculators; `q8ta/conv2d` splits these by route into `conv2d{,_dw,_pw}.cpp`), and a `test_sets/` folder whose files register the sets. A set file holds the set's test case data (configs, shapes, and the dtypes, storage types, layouts, and route selectors to sweep), the loop that turns it into test cases (including device gating and rules that derive variants from a config), and the returned `TestCaseSet`. A set gets its own `<set>_set.cpp`; sets that share generation code share one `<group>_sets.cpp`, named for their common prefix (`dw_sets.cpp`, `pw_sets.cpp`, `fpa_sets.cpp`, `coopmat_sets.cpp`, `scenex_sets.cpp`) or, for unprefixed sets such as `correctness` and `performance`, for the operator (`linear_sets.cpp`). Each directory's code is in its own namespace (`prototyping::<op>`), since operators reuse names such as `LinearConfig` and now link into one binary. - Sets - Each binary's default list becomes `correctness` (its `ACCU` cases) and `performance` (its `PERF` cases), prefixed by route for secondary binaries in a directory (`dw_`, `pw_`, `fpa_`, `coopmat_`). Every case is registered in exactly one set, so the default lists no longer repeat the cases of the mode sets below. - Modes become sets: - `test_q8ta_conv2d` `--im2col-path=*`, `--narrow-workgroups-only`, `--scenex-regular`, and `--scenex-grouped` → `im2col` (each edge config under the signed, unsigned, and auto selectors), `narrow_workgroup`, `scenex_regular`, `scenex_grouped`. - Its streaming cases and dynamic shrink test → `streaming_im2col` and `streaming_dynamic_shrink`; its startup unsigned dot checks → `unsigned_dot_selection`. - `test_q8ta_conv2d_pw` `--pw-path=*` folds into `pw_correctness` and `pw_performance`, which run each config under the default, signed, unsigned, and auto selectors; `test_q8ta_conv2d_dw` narrow workgroups → `dw_narrow_workgroup`. - `SDPA_DECODE_ONLY` → `sdpa:decode` (`sdpa:performance` keeps only prefill); `COOPMAT_BENCH_CORRECTNESS_ONLY` → `q4gsw_linear:coopmat_correctness`; `SDPA_NO_CHAIN` → `--chain=1`. - `#ifdef DEBUG_MODE` and never-called easy case lists → `debug`. The `qdq` easy list now skips the legacy 4W4C path for 5D/6D shapes, as the default list does. The never-called `q8csw_conv2d` easy list is removed: it could not run, since it never set an operator name. - Dropped: per-binary banners, the coopmat benchmark's tiled-vs-coopmat summary table, and the redundant query pool initialization in `test_add`. - Build: one `etvk_bench` target in Buck and CMake replaces the per-binary targets. - CI: each Skycastle (GPU host, Eureka) and OSS workflow job keeps its name and runs `etvk_bench` with the `--op`/`--set` selection that covers its old binary's run. The default `test_q8ta_conv2d` job adds `im2col` and `narrow_workgroup`, the `_dw` job adds `dw_narrow_workgroup`, and `test_sdpa` adds `decode`; the six `--im2col-path`/`--pw-path` jobs are removed, since `test_q8ta_conv2d` and `test_q8ta_conv2d_pw` now cover them. - New files are recorded as copies of the binary they came from. CLANGTIDY findings in the moved case code are fixed (argument comments that did not match parameter names, a shadowed loop variable, narrowing, a C array). Test case construction, references, tolerances, and case names are otherwise unchanged. ghstack-source-id: 446951726 Differential Revision: [D123707389](https://our.internmc.facebook.com/intern/diff/D123707389/)
pytorchbot
requested review from
SS-JIA,
kirklandsign and
larryliu0820
as code owners
October 13, 2026 01:39
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/23725
Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
SS-JIA
approved these changes
Oct 13, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR was created by the merge bot to help merge the original PR into the main branch.
ghstack PR number: #23598 by @SS-JIA
^ Please use this as the source of truth for the PR details, comments, and reviews
ghstack PR base: https://github.com/pytorch/executorch/tree/gh/SS-JIA/667/base
ghstack PR head: https://github.com/pytorch/executorch/tree/gh/SS-JIA/667/head
Merge bot PR base: https://github.com/pytorch/executorch/tree/gh/SS-JIA/666/orig
Merge bot PR head: https://github.com/pytorch/executorch/tree/gh/SS-JIA/667/orig
Differential Revision: D123707389
@diff-train-skip-merge