From 8321e1d2d6776204c31b8383e773180233c5c51e Mon Sep 17 00:00:00 2001 From: Stephen Jia Date: Mon, 12 Oct 2026 12:45:48 -0700 Subject: [PATCH] [ET-VK][test] Move operator benchmark binaries under ops/benchmark Pull Request resolved: https://github.com/pytorch/executorch/pull/23597 ## Ulterior Motive Restructure the Vulkan operator test+benchmark binaries around a single `etvk_benchmark` entry point with registered, per-operator test case sets, so related cases (such as a model's bottleneck shapes) can be grouped and run in isolation. ## Rationale **What**: Move the test+benchmark binaries, their shared utilities, and the test-only ops and shaders from `test/custom_ops/` into `test/ops/benchmark/`, with one subdirectory per operator under `cases/`, and split the monolithic `utils.{h,cpp}` by topic. **Why**: The flat `custom_ops/` directory mixes benchmark binaries, framework code, test-only ops, and correctness-only gtests, and `utils.{h,cpp}` (3.5k lines) mixes value specs, test cases, execution, and reporting. Grouping the binaries by operator gives the upcoming case-set registration a home for each operator's cases and reference code. ## Details - Layout under `test/ops/benchmark/` - `framework/`: test framework utilities, `conv2d_utils`, `cm_utils`, and `test_coopmat_probe`. - `test_ops/impl/` and `test_ops/glsl/`: test-only ops and shaders, formerly `impl/` and `glsl/`. - `cases/`: one directory per operator, `conv2d/` (direct, depthwise, pointwise), `conv1d/`, `q8csw_conv2d/`, `q4gsw_linear/` (including `test_fpa_q4gsw_linear` and `test_coopmat_linear_bench`), `q8csw_linear/`, `mm/`, `sdpa/`, `embedding_q4gsw/`, `choose_qparams_per_row/`, `add/`; `cases/q8ta/` collects the q8ta operators: `conv2d/`, `conv2d_transposed/`, `linear/`, `binary/`, `unary/`, `clone/`, `pixel_shuffle/`, `qdq/`. - `utils.{h,cpp}` split - `config`: global output and debugging options. - `labels.h`: test case naming helpers. - `value_spec`: `ValueSpec`, data generation, half-precision conversion, and `print_valuespec_data`. - `test_case`: `ReferenceKey`, the shader filter defaults, and `TestCase`. - `results`: `BenchmarkResult`, `TestResult`, and table printing. - `runner`: graph setup, `RepeatedGraphExecutor`, and `execute_test_case(s)`. - `weight_utils`: weight canvases and weight sum helpers. - `utils.h` includes all of the above, so binaries are unchanged. - Moving code into new files exposes it to CLANGTIDY, so its findings are fixed: braces around single-statement bodies, `explicit` on the single-argument `BenchmarkResult`, `TestResult`, and `TestCase` constructors, `const&` for the generator, reference, FLOP calculator, and `op_name` parameters, `utils::safe_downcast` for one narrowing, `(void)` for an unused parameter, and `NOLINTNEXTLINE` on enum `switch`es that list only some enumerators (each has a `default`) and on the `using namespace vkcompute` that binaries rely on. `utils.h` no longer includes ``. - Build - `test/ops/benchmark/{BUCK,targets.bzl}` define the binaries, `prototyping_utils`, `operator_implementations`, and `custom_ops_shaderlib`. Binary target names are unchanged; only the package path changes. - The CMake project and `build_and_run.sh` move with the binaries and take source paths from the new subdirectories. - The correctness-only gtests (`utils_test`, `q8ta_conv2d_route_test`, `q8ta_conv2d_stream_plan_test`) stay in `test/custom_ops/`; `utils_test` now depends on `prototyping_utils` in its new package. - Binaries include framework headers by full path (`executorch/backends/vulkan/test/ops/benchmark/framework/utils.h`). - CI references move to the new package: the GPU-host and Eureka Skycastle workflows, and the OSS `vulkan.yml` and `pull.yml` workflows. - Moved files are recorded as renames; the new `BUCK`/`targets.bzl` and the split framework files are recorded as copies. Binary sources, test cases, and test-only ops are otherwise unchanged. ghstack-source-id: 446951727 @exported-using-ghexport Differential Revision: [D123690842](https://our.internmc.facebook.com/intern/diff/D123690842/) --- .github/workflows/pull.yml | 16 +- .github/workflows/vulkan.yml | 18 +- backends/vulkan/test/custom_ops/targets.bzl | 104 +- backends/vulkan/test/custom_ops/utils.cpp | 2482 ----------------- backends/vulkan/test/custom_ops/utils.h | 1094 -------- .../vulkan/test/custom_ops/utils_test.cpp | 2 +- .../op_tests/conv2d_gemm_dynamic_test.cpp | 13 +- backends/vulkan/test/ops/benchmark/BUCK | 9 + .../benchmark}/CMakeLists.txt | 55 +- .../benchmark}/build_and_run.sh | 14 +- .../benchmark/cases/add}/test_add.cpp | 2 +- .../test_choose_qparams_per_row.cpp | 2 +- .../cases/conv1d}/test_conv1d_dw.cpp | 2 +- .../cases/conv1d}/test_conv1d_pw.cpp | 2 +- .../benchmark/cases/conv2d}/test_conv2d.cpp | 4 +- .../cases/conv2d}/test_conv2d_dw.cpp | 4 +- .../cases/conv2d}/test_conv2d_pw.cpp | 2 +- .../embedding_q4gsw}/test_embedding_q4gsw.cpp | 2 +- .../benchmark/cases/mm}/test_mm.cpp | 2 +- .../test_coopmat_linear_bench.cpp | 2 +- .../q4gsw_linear}/test_fpa_q4gsw_linear.cpp | 2 +- .../cases/q4gsw_linear}/test_q4gsw_linear.cpp | 2 +- .../cases/q8csw_conv2d}/test_q8csw_conv2d.cpp | 4 +- .../cases/q8csw_linear}/test_q8csw_linear.cpp | 2 +- .../cases/q8ta/binary}/test_q8ta_binary.cpp | 2 +- .../cases/q8ta/clone}/test_q8ta_clone.cpp | 2 +- .../cases/q8ta/conv2d}/test_q8ta_conv2d.cpp | 4 +- .../q8ta/conv2d}/test_q8ta_conv2d_dw.cpp | 4 +- .../q8ta/conv2d}/test_q8ta_conv2d_pw.cpp | 4 +- .../test_q8ta_conv2d_transposed.cpp | 4 +- .../cases/q8ta/linear}/test_q8ta_linear.cpp | 2 +- .../test_q8ta_pixel_shuffle.cpp | 2 +- .../cases/q8ta/qdq}/test_q8ta_qdq.cpp | 2 +- .../cases/q8ta/unary}/test_q8ta_unary.cpp | 2 +- .../benchmark/cases/sdpa}/test_sdpa.cpp | 2 +- .../benchmark/framework}/cm_utils.cpp | 0 .../benchmark/framework}/cm_utils.h | 0 .../test/ops/benchmark/framework/config.cpp | 55 + .../test/ops/benchmark/framework/config.h | 31 + .../benchmark/framework}/conv2d_utils.cpp | 0 .../benchmark/framework}/conv2d_utils.h | 0 .../test/ops/benchmark/framework/labels.h | 136 + .../test/ops/benchmark/framework/results.cpp | 468 ++++ .../test/ops/benchmark/framework/results.h | 235 ++ .../test/ops/benchmark/framework/runner.cpp | 760 +++++ .../test/ops/benchmark/framework/runner.h | 123 + .../ops/benchmark/framework/test_case.cpp | 79 + .../test/ops/benchmark/framework/test_case.h | 236 ++ .../framework}/test_coopmat_probe.cpp | 0 .../test/ops/benchmark/framework/utils.h | 27 + .../ops/benchmark/framework/value_spec.cpp | 1024 +++++++ .../test/ops/benchmark/framework/value_spec.h | 404 +++ .../ops/benchmark/framework/weight_utils.cpp | 247 ++ .../ops/benchmark/framework/weight_utils.h | 52 + .../vulkan/test/ops/benchmark/targets.bzl | 129 + .../benchmark/test_ops}/glsl/add.yaml | 0 .../benchmark/test_ops}/glsl/add_buffer.glsl | 0 .../benchmark/test_ops}/glsl/add_texture.glsl | 0 .../test_ops}/glsl/float_canvas.glsl | 0 .../glsl/packed_int32_canvas_buffer.glsl | 0 .../glsl/packed_int32_canvas_texture3d.glsl | 0 .../glsl/q4gsw_linear_gemv__w_4x8.glsl | 0 .../glsl/q4gsw_linear_gemv__w_4x8.yaml | 0 .../benchmark/test_ops}/impl/AddPrototype.cpp | 0 .../benchmark/test_ops}/impl/TestConv1dDW.cpp | 0 .../benchmark/test_ops}/impl/TestConv1dPW.cpp | 0 .../benchmark/test_ops}/impl/TestConv2d.cpp | 0 .../benchmark/test_ops}/impl/TestConv2dDw.cpp | 0 .../benchmark/test_ops}/impl/TestConv2dPw.cpp | 0 .../test_ops}/impl/TestFpaQ4gswLinear.cpp | 8 +- .../test_ops}/impl/TestMatmulLinear.cpp | 0 .../test_ops}/impl/TestQ8taBinary.cpp | 0 .../test_ops}/impl/TestQ8taClone.cpp | 0 .../test_ops}/impl/TestQ8taConv2d.cpp | 0 .../impl/TestQ8taConv2dTransposed.cpp | 0 .../test_ops}/impl/TestQ8taLinear.cpp | 0 .../test_ops}/impl/TestQ8taPixelShuffle.cpp | 0 .../impl/TestQ8taQuantizeDequantize.cpp | 0 .../test_ops}/impl/TestQ8taUnary.cpp | 0 .../benchmark/test_ops}/impl/TestSDPA.cpp | 0 80 files changed, 4117 insertions(+), 3768 deletions(-) delete mode 100644 backends/vulkan/test/custom_ops/utils.cpp delete mode 100644 backends/vulkan/test/custom_ops/utils.h create mode 100644 backends/vulkan/test/ops/benchmark/BUCK rename backends/vulkan/test/{custom_ops => ops/benchmark}/CMakeLists.txt (63%) rename backends/vulkan/test/{custom_ops => ops/benchmark}/build_and_run.sh (89%) rename backends/vulkan/test/{custom_ops => ops/benchmark/cases/add}/test_add.cpp (98%) rename backends/vulkan/test/{custom_ops => ops/benchmark/cases/choose_qparams_per_row}/test_choose_qparams_per_row.cpp (99%) rename backends/vulkan/test/{custom_ops => ops/benchmark/cases/conv1d}/test_conv1d_dw.cpp (99%) rename backends/vulkan/test/{custom_ops => ops/benchmark/cases/conv1d}/test_conv1d_pw.cpp (98%) rename backends/vulkan/test/{custom_ops => ops/benchmark/cases/conv2d}/test_conv2d.cpp (99%) rename backends/vulkan/test/{custom_ops => ops/benchmark/cases/conv2d}/test_conv2d_dw.cpp (98%) rename backends/vulkan/test/{custom_ops => ops/benchmark/cases/conv2d}/test_conv2d_pw.cpp (99%) rename backends/vulkan/test/{custom_ops => ops/benchmark/cases/embedding_q4gsw}/test_embedding_q4gsw.cpp (99%) rename backends/vulkan/test/{custom_ops => ops/benchmark/cases/mm}/test_mm.cpp (99%) rename backends/vulkan/test/{custom_ops => ops/benchmark/cases/q4gsw_linear}/test_coopmat_linear_bench.cpp (99%) rename backends/vulkan/test/{custom_ops => ops/benchmark/cases/q4gsw_linear}/test_fpa_q4gsw_linear.cpp (99%) rename backends/vulkan/test/{custom_ops => ops/benchmark/cases/q4gsw_linear}/test_q4gsw_linear.cpp (99%) rename backends/vulkan/test/{custom_ops => ops/benchmark/cases/q8csw_conv2d}/test_q8csw_conv2d.cpp (99%) rename backends/vulkan/test/{custom_ops => ops/benchmark/cases/q8csw_linear}/test_q8csw_linear.cpp (99%) rename backends/vulkan/test/{custom_ops => ops/benchmark/cases/q8ta/binary}/test_q8ta_binary.cpp (99%) rename backends/vulkan/test/{custom_ops => ops/benchmark/cases/q8ta/clone}/test_q8ta_clone.cpp (99%) rename backends/vulkan/test/{custom_ops => ops/benchmark/cases/q8ta/conv2d}/test_q8ta_conv2d.cpp (99%) rename backends/vulkan/test/{custom_ops => ops/benchmark/cases/q8ta/conv2d}/test_q8ta_conv2d_dw.cpp (99%) rename backends/vulkan/test/{custom_ops => ops/benchmark/cases/q8ta/conv2d}/test_q8ta_conv2d_pw.cpp (99%) rename backends/vulkan/test/{custom_ops => ops/benchmark/cases/q8ta/conv2d_transposed}/test_q8ta_conv2d_transposed.cpp (99%) rename backends/vulkan/test/{custom_ops => ops/benchmark/cases/q8ta/linear}/test_q8ta_linear.cpp (99%) rename backends/vulkan/test/{custom_ops => ops/benchmark/cases/q8ta/pixel_shuffle}/test_q8ta_pixel_shuffle.cpp (99%) rename backends/vulkan/test/{custom_ops => ops/benchmark/cases/q8ta/qdq}/test_q8ta_qdq.cpp (99%) rename backends/vulkan/test/{custom_ops => ops/benchmark/cases/q8ta/unary}/test_q8ta_unary.cpp (99%) rename backends/vulkan/test/{custom_ops => ops/benchmark/cases/sdpa}/test_sdpa.cpp (99%) rename backends/vulkan/test/{custom_ops => ops/benchmark/framework}/cm_utils.cpp (100%) rename backends/vulkan/test/{custom_ops => ops/benchmark/framework}/cm_utils.h (100%) create mode 100644 backends/vulkan/test/ops/benchmark/framework/config.cpp create mode 100644 backends/vulkan/test/ops/benchmark/framework/config.h rename backends/vulkan/test/{custom_ops => ops/benchmark/framework}/conv2d_utils.cpp (100%) rename backends/vulkan/test/{custom_ops => ops/benchmark/framework}/conv2d_utils.h (100%) create mode 100644 backends/vulkan/test/ops/benchmark/framework/labels.h create mode 100644 backends/vulkan/test/ops/benchmark/framework/results.cpp create mode 100644 backends/vulkan/test/ops/benchmark/framework/results.h create mode 100644 backends/vulkan/test/ops/benchmark/framework/runner.cpp create mode 100644 backends/vulkan/test/ops/benchmark/framework/runner.h create mode 100644 backends/vulkan/test/ops/benchmark/framework/test_case.cpp create mode 100644 backends/vulkan/test/ops/benchmark/framework/test_case.h rename backends/vulkan/test/{custom_ops => ops/benchmark/framework}/test_coopmat_probe.cpp (100%) create mode 100644 backends/vulkan/test/ops/benchmark/framework/utils.h create mode 100644 backends/vulkan/test/ops/benchmark/framework/value_spec.cpp create mode 100644 backends/vulkan/test/ops/benchmark/framework/value_spec.h create mode 100644 backends/vulkan/test/ops/benchmark/framework/weight_utils.cpp create mode 100644 backends/vulkan/test/ops/benchmark/framework/weight_utils.h create mode 100644 backends/vulkan/test/ops/benchmark/targets.bzl rename backends/vulkan/test/{custom_ops => ops/benchmark/test_ops}/glsl/add.yaml (100%) rename backends/vulkan/test/{custom_ops => ops/benchmark/test_ops}/glsl/add_buffer.glsl (100%) rename backends/vulkan/test/{custom_ops => ops/benchmark/test_ops}/glsl/add_texture.glsl (100%) rename backends/vulkan/test/{custom_ops => ops/benchmark/test_ops}/glsl/float_canvas.glsl (100%) rename backends/vulkan/test/{custom_ops => ops/benchmark/test_ops}/glsl/packed_int32_canvas_buffer.glsl (100%) rename backends/vulkan/test/{custom_ops => ops/benchmark/test_ops}/glsl/packed_int32_canvas_texture3d.glsl (100%) rename backends/vulkan/test/{custom_ops => ops/benchmark/test_ops}/glsl/q4gsw_linear_gemv__w_4x8.glsl (100%) rename backends/vulkan/test/{custom_ops => ops/benchmark/test_ops}/glsl/q4gsw_linear_gemv__w_4x8.yaml (100%) rename backends/vulkan/test/{custom_ops => ops/benchmark/test_ops}/impl/AddPrototype.cpp (100%) rename backends/vulkan/test/{custom_ops => ops/benchmark/test_ops}/impl/TestConv1dDW.cpp (100%) rename backends/vulkan/test/{custom_ops => ops/benchmark/test_ops}/impl/TestConv1dPW.cpp (100%) rename backends/vulkan/test/{custom_ops => ops/benchmark/test_ops}/impl/TestConv2d.cpp (100%) rename backends/vulkan/test/{custom_ops => ops/benchmark/test_ops}/impl/TestConv2dDw.cpp (100%) rename backends/vulkan/test/{custom_ops => ops/benchmark/test_ops}/impl/TestConv2dPw.cpp (100%) rename backends/vulkan/test/{custom_ops => ops/benchmark/test_ops}/impl/TestFpaQ4gswLinear.cpp (99%) rename backends/vulkan/test/{custom_ops => ops/benchmark/test_ops}/impl/TestMatmulLinear.cpp (100%) rename backends/vulkan/test/{custom_ops => ops/benchmark/test_ops}/impl/TestQ8taBinary.cpp (100%) rename backends/vulkan/test/{custom_ops => ops/benchmark/test_ops}/impl/TestQ8taClone.cpp (100%) rename backends/vulkan/test/{custom_ops => ops/benchmark/test_ops}/impl/TestQ8taConv2d.cpp (100%) rename backends/vulkan/test/{custom_ops => ops/benchmark/test_ops}/impl/TestQ8taConv2dTransposed.cpp (100%) rename backends/vulkan/test/{custom_ops => ops/benchmark/test_ops}/impl/TestQ8taLinear.cpp (100%) rename backends/vulkan/test/{custom_ops => ops/benchmark/test_ops}/impl/TestQ8taPixelShuffle.cpp (100%) rename backends/vulkan/test/{custom_ops => ops/benchmark/test_ops}/impl/TestQ8taQuantizeDequantize.cpp (100%) rename backends/vulkan/test/{custom_ops => ops/benchmark/test_ops}/impl/TestQ8taUnary.cpp (100%) rename backends/vulkan/test/{custom_ops => ops/benchmark/test_ops}/impl/TestSDPA.cpp (100%) diff --git a/.github/workflows/pull.yml b/.github/workflows/pull.yml index 97ca6969318..925d2a6e367 100644 --- a/.github/workflows/pull.yml +++ b/.github/workflows/pull.yml @@ -1680,14 +1680,14 @@ jobs: .ci/scripts/setup-linux.sh --build-tool "cmake" # Custom operator tests - PYTHON_EXECUTABLE=python bash backends/vulkan/test/custom_ops/build_and_run.sh test_add - ./cmake-out/backends/vulkan/test/custom_ops/test_q8csw_linear - ./cmake-out/backends/vulkan/test/custom_ops/test_q8csw_conv2d - ./cmake-out/backends/vulkan/test/custom_ops/test_q4gsw_linear - ./cmake-out/backends/vulkan/test/custom_ops/test_choose_qparams_per_row - ./cmake-out/backends/vulkan/test/custom_ops/test_q8ta_qdq - ./cmake-out/backends/vulkan/test/custom_ops/test_q8ta_clone - ./cmake-out/backends/vulkan/test/custom_ops/test_q8ta_binary + PYTHON_EXECUTABLE=python bash backends/vulkan/test/ops/benchmark/build_and_run.sh test_add + ./cmake-out/backends/vulkan/test/ops/benchmark/test_q8csw_linear + ./cmake-out/backends/vulkan/test/ops/benchmark/test_q8csw_conv2d + ./cmake-out/backends/vulkan/test/ops/benchmark/test_q4gsw_linear + ./cmake-out/backends/vulkan/test/ops/benchmark/test_choose_qparams_per_row + ./cmake-out/backends/vulkan/test/ops/benchmark/test_q8ta_qdq + ./cmake-out/backends/vulkan/test/ops/benchmark/test_q8ta_clone + ./cmake-out/backends/vulkan/test/ops/benchmark/test_q8ta_binary # "Classic" Operator tests PYTHON_EXECUTABLE=python bash backends/vulkan/test/scripts/test_op.sh --build diff --git a/.github/workflows/vulkan.yml b/.github/workflows/vulkan.yml index 223fadebf23..fd9919ade35 100644 --- a/.github/workflows/vulkan.yml +++ b/.github/workflows/vulkan.yml @@ -100,15 +100,15 @@ jobs: # Operator coverage (mirrors test-vulkan-operators-linux, on real hardware). # The custom-op prototyping binaries are GPU microbenchmarks that rely on # GPU timestamp queries, so they require the hardware selected above. - PYTHON_EXECUTABLE=python bash backends/vulkan/test/custom_ops/build_and_run.sh - ./cmake-out/backends/vulkan/test/custom_ops/test_add - ./cmake-out/backends/vulkan/test/custom_ops/test_q8csw_linear - ./cmake-out/backends/vulkan/test/custom_ops/test_q8csw_conv2d - ./cmake-out/backends/vulkan/test/custom_ops/test_q4gsw_linear - ./cmake-out/backends/vulkan/test/custom_ops/test_choose_qparams_per_row - ./cmake-out/backends/vulkan/test/custom_ops/test_q8ta_qdq - ./cmake-out/backends/vulkan/test/custom_ops/test_q8ta_clone - ./cmake-out/backends/vulkan/test/custom_ops/test_q8ta_binary + PYTHON_EXECUTABLE=python bash backends/vulkan/test/ops/benchmark/build_and_run.sh + ./cmake-out/backends/vulkan/test/ops/benchmark/test_add + ./cmake-out/backends/vulkan/test/ops/benchmark/test_q8csw_linear + ./cmake-out/backends/vulkan/test/ops/benchmark/test_q8csw_conv2d + ./cmake-out/backends/vulkan/test/ops/benchmark/test_q4gsw_linear + ./cmake-out/backends/vulkan/test/ops/benchmark/test_choose_qparams_per_row + ./cmake-out/backends/vulkan/test/ops/benchmark/test_q8ta_qdq + ./cmake-out/backends/vulkan/test/ops/benchmark/test_q8ta_clone + ./cmake-out/backends/vulkan/test/ops/benchmark/test_q8ta_binary PYTHON_EXECUTABLE=python bash backends/vulkan/test/scripts/test_op.sh --build vulkan_sdpa_av_test diff --git a/backends/vulkan/test/custom_ops/targets.bzl b/backends/vulkan/test/custom_ops/targets.bzl index e3631298119..2355e6a4f2a 100644 --- a/backends/vulkan/test/custom_ops/targets.bzl +++ b/backends/vulkan/test/custom_ops/targets.bzl @@ -1,89 +1,14 @@ -load("@fbsource//tools/build_defs:platform_defs.bzl", "ANDROID", "CXX") +load("@fbsource//tools/build_defs:platform_defs.bzl", "CXX") load("@fbsource//xplat/executorch/build:runtime_wrapper.bzl", "runtime") load( "@fbsource//xplat/executorch/backends/vulkan:targets.bzl", "get_platforms", - "vulkan_spv_shader_lib", ) -def define_custom_op_test_binary(custom_op_name, extra_deps = [], src_file = None, include_torch = False): - deps_list = [ - ":prototyping_utils", - ":operator_implementations", - ":custom_ops_shaderlib", - "//executorch/backends/vulkan:vulkan_graph_runtime", - ] + ([runtime.external_dep_location("libtorch")] if include_torch else []) + extra_deps - - src_file_str = src_file if src_file else "{}.cpp".format(custom_op_name) - - runtime.cxx_binary( - name = custom_op_name, - srcs = [ - src_file_str, - ], - platforms = get_platforms(), - define_static_target = False, - deps = deps_list, - ) - def define_common_targets(is_fbcode = False): if is_fbcode: return - # Shader library from GLSL files - runtime.filegroup( - name = "custom_ops_shaders", - srcs = native.glob([ - "glsl/**/*", - ]), - visibility = ["PUBLIC"], - ) - - vulkan_spv_shader_lib( - name = "custom_ops_shaderlib", - spv_filegroups = { - ":custom_ops_shaders": "glsl", - }, - is_fbcode = is_fbcode, - ) - - # Prototyping utilities library - runtime.cxx_library( - name = "prototyping_utils", - srcs = [ - "utils.cpp", - "conv2d_utils.cpp", - ], - headers = [ - "utils.h", - "conv2d_utils.h", - ], - exported_headers = [ - "utils.h", - "conv2d_utils.h", - ], - platforms = get_platforms(), - deps = [ - "//executorch/backends/vulkan:vulkan_graph_runtime", - ], - visibility = ["PUBLIC"], - ) - - # Operator implementations library - runtime.cxx_library( - name = "operator_implementations", - srcs = native.glob([ - "impl/*.cpp", - ]), - platforms = get_platforms(), - deps = [ - "//executorch/backends/vulkan:vulkan_graph_runtime", - ":custom_ops_shaderlib", - ], - visibility = ["PUBLIC"], - link_whole = True, - ) - runtime.cxx_test( name = "utils_test", srcs = [ @@ -92,7 +17,7 @@ def define_common_targets(is_fbcode = False): contacts = ["oncall+ai_infra_mobile_platform@xmail.facebook.com"], platforms = [CXX], deps = [ - ":prototyping_utils", + "//executorch/backends/vulkan/test/ops/benchmark:prototyping_utils", "//third-party/googletest:gtest_main", ], ) @@ -116,28 +41,3 @@ def define_common_targets(is_fbcode = False): "//executorch/backends/vulkan:vulkan_graph_runtime", ], ) - - define_custom_op_test_binary("test_add") - define_custom_op_test_binary("test_q8csw_linear") - define_custom_op_test_binary("test_q8csw_conv2d") - define_custom_op_test_binary("test_choose_qparams_per_row") - define_custom_op_test_binary("test_q4gsw_linear") - define_custom_op_test_binary("test_q8ta_qdq") - define_custom_op_test_binary("test_q8ta_clone") - define_custom_op_test_binary("test_q8ta_binary") - define_custom_op_test_binary("test_q8ta_conv2d") - define_custom_op_test_binary("test_q8ta_conv2d_pw") - define_custom_op_test_binary("test_q8ta_conv2d_dw") - define_custom_op_test_binary("test_q8ta_linear") - define_custom_op_test_binary("test_q8ta_conv2d_transposed") - define_custom_op_test_binary("test_q8ta_pixel_shuffle") - define_custom_op_test_binary("test_q8ta_unary") - define_custom_op_test_binary("test_mm") - define_custom_op_test_binary("test_conv2d") - define_custom_op_test_binary("test_conv2d_pw") - define_custom_op_test_binary("test_conv2d_dw") - define_custom_op_test_binary("test_embedding_q4gsw") - define_custom_op_test_binary("test_conv1d_pw") - define_custom_op_test_binary("test_conv1d_dw") - define_custom_op_test_binary("test_fpa_q4gsw_linear") - define_custom_op_test_binary("test_sdpa") diff --git a/backends/vulkan/test/custom_ops/utils.cpp b/backends/vulkan/test/custom_ops/utils.cpp deleted file mode 100644 index 066996de262..00000000000 --- a/backends/vulkan/test/custom_ops/utils.cpp +++ /dev/null @@ -1,2482 +0,0 @@ -// Copyright (c) Meta Platforms, Inc. and affiliates. -// All rights reserved. -// -// This source code is licensed under the BSD-style license found in the -// LICENSE file in the root directory of this source tree. - -#include "utils.h" -#include -#include -#include -#include -#include - -#include - -namespace executorch { -namespace vulkan { -namespace prototyping { - -namespace { - -// Upper bound on the auto-sized chained_dispatches factor. Caps the factor -// when an op is fast enough that target_us / probe_us would otherwise -// exceed this; past this point, stacking more dispatches doesn't materially -// improve measurement precision. -constexpr int kMaxChainedDispatches = 200; - -// Floor for probe_us to avoid division-by-zero / a runaway chained_dispatches -// factor when an op finishes faster than the wall clock can resolve. -constexpr float kMinProbeTimeUs = 1.0f; - -} // namespace - -// Benchmark graphs are immutable after input upload, so the operator nodes can -// be recorded repeatedly. Staging uploads/downloads are encoded once: -// repeating them would redo host-visible copies on every repetition and bias -// per-invocation timings. -RepeatedGraphExecutor::RepeatedGraphExecutor( - ComputeGraph& graph, - int repetitions, - OpNodeRange op_nodes) - : graph_(graph) { - VK_CHECK_COND(repetitions > 0); - VK_CHECK_COND(op_nodes.begin <= op_nodes.end); - VK_CHECK_COND(op_nodes.end <= graph_.execute_nodes().size()); - - api::Context* const context = graph_.context(); - context->flush(); - context->set_cmd(/*reusable=*/true); - context->cmd_reset_querypool(); - - for (size_t i = 0; i < op_nodes.begin; ++i) { - graph_.execute_nodes()[i]->encode(&graph_); - } - for (int i = 0; i < repetitions; ++i) { - for (size_t j = op_nodes.begin; j < op_nodes.end; ++j) { - graph_.execute_nodes()[j]->encode(&graph_); - } - } - for (size_t i = op_nodes.end; i < graph_.execute_nodes().size(); ++i) { - graph_.execute_nodes()[i]->encode(&graph_); - } - - command_ = - std::make_unique(std::move(context->extract_cmd())); -} - -void RepeatedGraphExecutor::execute() { - api::Context* const context = graph_.context(); - command_->end(); - - // Intentionally bypasses Context::submit_cmd_to_gpu(): its submit-count - // bookkeeping and threshold-split path only serve submit_compute_job - // recording, which benchmarks don't use. - vkapi::VulkanFence fence = context->fences().get_fence(); - context->adapter_ptr()->submit_cmd( - context->queue(), - command_->get_submit_handle(/*final_use=*/false), - fence.get_submit_handle()); - fence.wait(); - context->fences().return_fence(fence); -} - -int get_seed() { - static int seed = 42; - return seed++; -} - -int get_seed_or_explicit(int explicit_seed) { - if (explicit_seed >= 0) { - return explicit_seed; - } - return get_seed(); -} - -// Forward declarations for data generation utilities -void generate_random_float_data( - std::vector& data, - float min_val = -1.0f, - float max_val = 1.0f, - int explicit_seed = -1); -void generate_random_int_data( - std::vector& data, - int min_val = -10, - int max_val = 10, - int explicit_seed = -1); -void generate_randint_float_data( - std::vector& data, - int min_val = -10, - int max_val = 10, - int explicit_seed = -1); -void generate_randint_half_data( - std::vector& data, - int min_val = -10, - int max_val = 10, - int explicit_seed = -1); -void generate_random_int8_data( - std::vector& data, - int8_t min_val = -10, - int8_t max_val = 10, - int explicit_seed = -1); -void generate_random_uint8_data( - std::vector& data, - uint8_t min_val = 0, - uint8_t max_val = 255, - int explicit_seed = -1); -void generate_random_2xint4_data( - std::vector& data, - int explicit_seed = -1); -void generate_random_2xint4_data( - std::vector& data, - int explicit_seed = -1); -void generate_random_int4_data( - std::vector& data, - int8_t min_val = -8, - int8_t max_val = 7, - int explicit_seed = -1); -void generate_ones_data(std::vector& data); -void generate_zeros_data(std::vector& data); - -// Convert a float32 value to IEEE 754 half-precision (uint16_t) -uint16_t float_to_half(float value) { - uint32_t float_bits; - std::memcpy(&float_bits, &value, sizeof(float)); - uint32_t sign = (float_bits >> 31) & 0x1; - int32_t exponent = static_cast((float_bits >> 23) & 0xFF) - 127; - uint32_t mantissa = float_bits & 0x7FFFFF; - - uint16_t half_val; - if (exponent > 15) { - half_val = static_cast((sign << 15) | 0x7C00); // Inf - } else if (exponent < -14) { - half_val = static_cast(sign << 15); // Zero / subnormal - } else { - half_val = static_cast( - (sign << 15) | (static_cast(exponent + 15) << 10) | - (mantissa >> 13)); - } - return half_val; -} - -// Convert a IEEE 754 half-precision (uint16_t) value to float32 -float half_to_float(uint16_t half_val) { - uint32_t sign = (half_val >> 15) & 0x1; - uint32_t exponent = (half_val >> 10) & 0x1F; - uint32_t mantissa = half_val & 0x3FF; - - float result; - if (exponent == 0) { - result = std::ldexp(static_cast(mantissa), -24); - } else if (exponent == 31) { - result = mantissa ? std::numeric_limits::quiet_NaN() - : std::numeric_limits::infinity(); - } else { - result = std::ldexp( - 1.0f + static_cast(mantissa) / 1024.0f, exponent - 15); - } - if (sign) { - result = -result; - } - return result; -} - -// Output and latency printing utilities -namespace { -static int print_output_enabled = 0; -static int print_latencies_enabled = 0; -static int gpu_timestamps_enabled = 0; -static int debugging_enabled = 0; -} // namespace - -bool print_output() { - return print_output_enabled > 0; -} - -void set_print_output(bool print_output) { - print_output_enabled = print_output ? 1 : 0; -} - -bool print_latencies() { - return print_latencies_enabled > 0; -} - -void set_print_latencies(bool print_latencies) { - print_latencies_enabled = print_latencies ? 1 : 0; -} - -bool use_gpu_timestamps() { - return gpu_timestamps_enabled > 0; -} - -void set_use_gpu_timestamps(bool use_timestamps) { - gpu_timestamps_enabled = use_timestamps ? 1 : 0; -} - -bool debugging() { - return debugging_enabled > 0; -} - -void set_debugging(bool enable_debugging) { - debugging_enabled = enable_debugging ? 1 : 0; -} - -// ValueSpec implementation -void ValueSpec::ensure_unique_data() const { - if (data_.use_count() != 1) { - data_ = std::make_shared(*data_); - } -} - -void ValueSpec::ensure_unique_reference_data() const { - if (reference_data_.use_count() != 1) { - reference_data_ = std::make_shared(*reference_data_); - } -} - -void ValueSpec::generate_tensor_data(int seed) const { - if (spec_type != SpecType::Tensor) { - return; - } - - ensure_unique_data(); - auto& float_data = data_->float_data; - auto& int32_data = data_->int32_data; - auto& half_data = data_->half_data; - auto& int8_data = data_->int8_data; - auto& uint8_data = data_->uint8_data; - - int64_t num_elements = numel(); - - switch (dtype) { - case vkapi::kFloat: { - float_data.resize(num_elements); - if (data_gen_type == DataGenType::RANDOM) { - generate_random_float_data(float_data, -1.0f, 1.0f, seed); - } else if (data_gen_type == DataGenType::RANDOM_SCALES) { - generate_random_float_data(float_data, 0.005, 0.015, seed); - } else if (data_gen_type == DataGenType::RANDINT) { - generate_randint_float_data(float_data, -10, 10, seed); - } else if (data_gen_type == DataGenType::RANDINT8) { - generate_randint_float_data(float_data, -128, 127, seed); - } else if (data_gen_type == DataGenType::RANDINT4) { - generate_randint_float_data(float_data, -8, 7, seed); - } else if (data_gen_type == DataGenType::ONES) { - generate_ones_data(float_data); - } else if (data_gen_type == DataGenType::ZEROS) { - generate_zeros_data(float_data); - } else { - generate_zeros_data(float_data); - } - break; - } - case vkapi::kHalf: { - half_data.resize(num_elements); - if (data_gen_type == DataGenType::RANDOM) { - // Generate random float data first, then convert to IEEE 754 half. - std::vector temp_data(num_elements); - generate_random_float_data(temp_data, -1.0f, 1.0f, seed); - for (size_t i = 0; i < temp_data.size(); ++i) { - half_data[i] = float_to_half(temp_data[i]); - } - } else if (data_gen_type == DataGenType::RANDOM_SCALES) { - // Generate random scales in float, then convert to proper fp16 - std::vector temp_data(num_elements); - generate_random_float_data(temp_data, 0.005f, 0.015f, seed); - for (size_t i = 0; i < temp_data.size(); ++i) { - half_data[i] = float_to_half(temp_data[i]); - } - } else if (data_gen_type == DataGenType::RANDINT) { - generate_randint_half_data(half_data, -10, 10, seed); - } else if (data_gen_type == DataGenType::RANDINT8) { - generate_randint_half_data(half_data, -128, 127, seed); - } else if (data_gen_type == DataGenType::RANDINT4) { - generate_randint_half_data(half_data, -8, 7, seed); - } else if (data_gen_type == DataGenType::ONES) { - std::fill(half_data.begin(), half_data.end(), float_to_half(1.0f)); - } else if (data_gen_type == DataGenType::ZEROS) { - std::fill( - half_data.begin(), - half_data.end(), - static_cast(0)); // 0.0 in half - } else { - std::fill( - half_data.begin(), - half_data.end(), - static_cast(0)); // 0.0 in half - } - break; - } - case vkapi::kInt: { - int32_data.resize(num_elements); - if (data_gen_type == DataGenType::RANDOM) { - generate_random_int_data(int32_data, -10, 10, seed); - } else if (data_gen_type == DataGenType::RANDINT) { - generate_random_int_data( - int32_data, - -10, - 10, - seed); // For int type, RANDINT is same as RANDOM - } else if (data_gen_type == DataGenType::RANDINT8) { - generate_random_int_data(int32_data, -128, 127, seed); - } else if (data_gen_type == DataGenType::RANDINT4) { - generate_random_int_data(int32_data, -8, 7, seed); - } else if (data_gen_type == DataGenType::ONES) { - std::fill(int32_data.begin(), int32_data.end(), 1); - } else if (data_gen_type == DataGenType::ZEROS) { - std::fill(int32_data.begin(), int32_data.end(), 0); - } else { - std::fill(int32_data.begin(), int32_data.end(), 0); - } - break; - } - case vkapi::kChar: { - int8_data.resize(num_elements); - if (data_gen_type == DataGenType::RANDOM) { - generate_random_int8_data(int8_data, -10, 10, seed); - } else if (data_gen_type == DataGenType::RANDINT) { - generate_random_int8_data(int8_data, -10, 10, seed); - } else if (data_gen_type == DataGenType::RANDINT8) { - generate_random_int8_data(int8_data, -128, 127, seed); - } else if (data_gen_type == DataGenType::RANDINT4) { - generate_random_2xint4_data(int8_data, seed); - } else if (data_gen_type == DataGenType::ONES) { - std::fill(int8_data.begin(), int8_data.end(), 1); - } else if (data_gen_type == DataGenType::ONES_INT4) { - int8_t packed_data = (1 << 4) | 1; - std::fill(int8_data.begin(), int8_data.end(), packed_data); - } else if (data_gen_type == DataGenType::ZEROS) { - std::fill(int8_data.begin(), int8_data.end(), 0); - } else { - std::fill(int8_data.begin(), int8_data.end(), 0); - } - break; - } - case vkapi::kByte: { - uint8_data.resize(num_elements); - if (data_gen_type == DataGenType::RANDOM) { - generate_random_uint8_data(uint8_data, 0, 255, seed); - } else if (data_gen_type == DataGenType::RANDINT) { - generate_random_uint8_data(uint8_data, 0, 255, seed); - } else if (data_gen_type == DataGenType::RANDINT8) { - generate_random_uint8_data(uint8_data, 0, 255, seed); - } else if (data_gen_type == DataGenType::RANDINT4) { - generate_random_2xint4_data(uint8_data, seed); - } else if (data_gen_type == DataGenType::ONES) { - std::fill(uint8_data.begin(), uint8_data.end(), 1); - } else if (data_gen_type == DataGenType::ONES_INT4) { - uint8_t packed_data = (9 << 4) | 9; - std::fill(uint8_data.begin(), uint8_data.end(), packed_data); - } else if (data_gen_type == DataGenType::ZEROS) { - std::fill(uint8_data.begin(), uint8_data.end(), 0); - } else { - std::fill(uint8_data.begin(), uint8_data.end(), 0); - } - break; - } - default: - // Default to float - float_data.resize(num_elements); - if (data_gen_type == DataGenType::RANDOM) { - generate_random_float_data(float_data, -1.0f, 1.0f, seed); - } else if (data_gen_type == DataGenType::RANDINT) { - generate_randint_float_data(float_data, -10, 10, seed); - } else if (data_gen_type == DataGenType::ONES) { - generate_ones_data(float_data); - } else if (data_gen_type == DataGenType::ZEROS) { - generate_zeros_data(float_data); - } else { - generate_zeros_data(float_data); - } - break; - } -} - -int64_t ValueSpec::numel() const { - if (spec_type == SpecType::Int || spec_type == SpecType::Float || - spec_type == SpecType::Bool) { - return 1; - } else if (spec_type == SpecType::IntList) { - return sizes.empty() ? 0 : sizes[0]; - } else { // Tensor - int64_t total = 1; - for (int64_t size : sizes) { - total *= size; - } - return total; - } -} - -size_t ValueSpec::nbytes() const { - size_t element_size = 0; - switch (dtype) { - case vkapi::kFloat: - element_size = sizeof(float); - break; - case vkapi::kHalf: - element_size = sizeof(uint16_t); - break; - case vkapi::kInt: - element_size = sizeof(int32_t); - break; - case vkapi::kChar: - element_size = sizeof(int8_t); - break; - case vkapi::kByte: - element_size = sizeof(uint8_t); - break; - default: - element_size = sizeof(float); // Default fallback - break; - } - return numel() * element_size; -} - -std::string ValueSpec::to_string() const { - std::string result = "ValueSpec("; - - switch (spec_type) { - case SpecType::Tensor: - result += "type=Tensor, sizes=["; - break; - case SpecType::IntList: - result += "type=IntList, count="; - result += std::to_string(sizes.empty() ? 0 : sizes[0]); - result += ", data_gen="; - result += (data_gen_type == DataGenType::FIXED) ? "FIXED" : "RANDOM"; - result += ")"; - return result; - case SpecType::Int: - result += "type=Int, value="; - result += std::to_string(get_int_value()); - result += ", data_gen="; - result += (data_gen_type == DataGenType::FIXED) ? "FIXED" : "RANDOM"; - result += ")"; - return result; - case SpecType::Float: - result += "type=Float, value="; - result += std::to_string(get_float_value()); - result += ", data_gen="; - result += (data_gen_type == DataGenType::FIXED) ? "FIXED" : "RANDOM"; - result += ")"; - return result; - case SpecType::Bool: - result += "type=Bool, value="; - result += get_bool_value() ? "true" : "false"; - result += ", data_gen="; - result += (data_gen_type == DataGenType::FIXED) ? "FIXED" : "RANDOM"; - result += ")"; - return result; - case SpecType::String: - result += "type=String, value=\""; - result += get_string_value(); - result += "\")"; - return result; - } - - for (size_t i = 0; i < sizes.size(); ++i) { - result += std::to_string(sizes[i]); - if (i < sizes.size() - 1) - result += ", "; - } - result += "]"; - - if (spec_type == SpecType::Tensor) { - result += ", dtype="; - switch (dtype) { - case vkapi::kFloat: - result += "float"; - break; - case vkapi::kHalf: - result += "half"; - break; - case vkapi::kInt: - result += "int32"; - break; - case vkapi::kChar: - result += "int8"; - break; - case vkapi::kByte: - result += "uint8"; - break; - default: - result += "unknown"; - break; - } - - result += ", memory_layout="; - switch (memory_layout) { - case utils::kWidthPacked: - result += "WidthPacked"; - break; - case utils::kHeightPacked: - result += "HeightPacked"; - break; - case utils::kChannelsPacked: - result += "ChannelsPacked"; - break; - default: - result += "unknown"; - break; - } - - result += ", storage_type="; - switch (storage_type) { - case utils::kTexture3D: - result += "Texture3D"; - break; - case utils::kBuffer: - result += "Buffer"; - break; - default: - result += "unknown"; - break; - } - } - - result += ", data_gen="; - switch (data_gen_type) { - case DataGenType::FIXED: - result += "FIXED"; - break; - case DataGenType::RANDOM: - result += "RANDOM"; - break; - case DataGenType::RANDINT: - result += "RANDINT"; - break; - case DataGenType::RANDINT8: - result += "RANDINT8"; - break; - case DataGenType::RANDINT4: - result += "RANDINT4"; - break; - case DataGenType::ONES: - result += "ONES"; - break; - case DataGenType::ZEROS: - result += "ZEROS"; - break; - default: - result += "unknown"; - break; - } - result += ")"; - return result; -} - -// Additional ValueSpec methods -void ValueSpec::resize_data(size_t new_size) { - // Generate first so a deferred tensor keeps its data-gen pattern (resized, - // not pinned to zeros by the data_generated_ flag set below). - ensure_data_generated(); - ensure_unique_data(); - switch (dtype) { - case vkapi::kFloat: - data_->float_data.resize(new_size); - break; - case vkapi::kHalf: - data_->half_data.resize(new_size); - break; - case vkapi::kInt: - data_->int32_data.resize(new_size); - break; - case vkapi::kChar: - data_->int8_data.resize(new_size); - break; - case vkapi::kByte: - data_->uint8_data.resize(new_size); - break; - default: - data_->float_data.resize(new_size); - break; - } - data_generated_ = true; -} - -void* ValueSpec::get_mutable_data_ptr() { - ensure_data_generated(); - ensure_unique_data(); - switch (dtype) { - case vkapi::kFloat: - return data_->float_data.data(); - case vkapi::kHalf: - return data_->half_data.data(); - case vkapi::kInt: - return data_->int32_data.data(); - case vkapi::kChar: - return data_->int8_data.data(); - case vkapi::kByte: - return data_->uint8_data.data(); - default: - return data_->float_data.data(); - } -} - -float ValueSpec::get_element(size_t index) const { - ensure_data_generated(); - if (index >= static_cast(numel())) { - return 0.0f; - } - - switch (dtype) { - case vkapi::kFloat: - return index < data_->float_data.size() ? data_->float_data[index] : 0.0f; - case vkapi::kHalf: - return index < data_->half_data.size() - ? half_to_float(data_->half_data[index]) - : 0.0f; - case vkapi::kInt: - return index < data_->int32_data.size() - ? static_cast(data_->int32_data[index]) - : 0.0f; - case vkapi::kChar: - return index < data_->int8_data.size() - ? static_cast(data_->int8_data[index]) - : 0.0f; - case vkapi::kByte: - return index < data_->uint8_data.size() - ? static_cast(data_->uint8_data[index]) - : 0.0f; - default: - return 0.0f; - } -} - -const void* ValueSpec::get_data_ptr() const { - ensure_data_generated(); - switch (dtype) { - case vkapi::kFloat: - return data_->float_data.data(); - case vkapi::kHalf: - return data_->half_data.data(); - case vkapi::kInt: - return data_->int32_data.data(); - case vkapi::kChar: - return data_->int8_data.data(); - case vkapi::kByte: - return data_->uint8_data.data(); - default: - throw std::runtime_error("Unsupported data type for get_data_ptr"); - } -} - -void generate_random_float_data( - std::vector& data, - float min_val, - float max_val, - int explicit_seed) { - std::mt19937 gen(get_seed_or_explicit(explicit_seed)); - std::uniform_real_distribution dis(min_val, max_val); - for (auto& val : data) { - val = dis(gen); - } -} - -void generate_random_int_data( - std::vector& data, - int min_val, - int max_val, - int explicit_seed) { - std::mt19937 gen(get_seed_or_explicit(explicit_seed)); - std::uniform_int_distribution dis(min_val, max_val); - for (auto& val : data) { - val = dis(gen); - } -} - -void generate_randint_float_data( - std::vector& data, - int min_val, - int max_val, - int explicit_seed) { - std::mt19937 gen(get_seed_or_explicit(explicit_seed)); - std::uniform_int_distribution dis(min_val, max_val); - for (auto& val : data) { - val = static_cast(dis(gen)); - } -} - -void generate_randint_half_data( - std::vector& data, - int min_val, - int max_val, - int explicit_seed) { - std::mt19937 gen(get_seed_or_explicit(explicit_seed)); - std::uniform_int_distribution dis(min_val, max_val); - for (auto& val : data) { - val = float_to_half(static_cast(dis(gen))); - } -} - -void generate_ones_data(std::vector& data) { - std::fill(data.begin(), data.end(), 1.0f); -} - -void generate_random_int8_data( - std::vector& data, - int8_t min_val, - int8_t max_val, - int explicit_seed) { - std::mt19937 gen(get_seed_or_explicit(explicit_seed)); - std::uniform_int_distribution dis(min_val, max_val); - for (auto& val : data) { - val = static_cast(dis(gen)); - } -} - -void generate_random_uint8_data( - std::vector& data, - uint8_t min_val, - uint8_t max_val, - int explicit_seed) { - std::mt19937 gen(get_seed_or_explicit(explicit_seed)); - std::uniform_int_distribution dis(min_val, max_val); - for (auto& val : data) { - val = static_cast(dis(gen)); - } -} - -void generate_random_int4_data( - std::vector& data, - int8_t min_val, - int8_t max_val, - int explicit_seed) { - std::mt19937 gen(get_seed_or_explicit(explicit_seed)); - std::uniform_int_distribution dis(min_val, max_val); - for (auto& val : data) { - val = static_cast(dis(gen)); - } -} - -void generate_random_2xint4_data(std::vector& data, int explicit_seed) { - std::mt19937 gen(get_seed_or_explicit(explicit_seed)); - std::uniform_int_distribution dis(-8, 7); // Signed 4-bit range - for (auto& val : data) { - // Generate two separate 4-bit values - int8_t lower_4bits = static_cast(dis(gen)) & 0x0F; - int8_t upper_4bits = static_cast(dis(gen)) & 0x0F; - // Pack them into a single 8-bit value - val = (upper_4bits << 4) | lower_4bits; - } -} - -void generate_random_2xint4_data( - std::vector& data, - int explicit_seed) { - std::mt19937 gen(get_seed_or_explicit(explicit_seed)); - std::uniform_int_distribution dis(0, 15); // Unsigned 4-bit range - for (auto& val : data) { - // Generate two separate 4-bit values - uint8_t lower_4bits = static_cast(dis(gen)) & 0x0F; - uint8_t upper_4bits = static_cast(dis(gen)) & 0x0F; - // Pack them into a single 8-bit value - val = (upper_4bits << 4) | lower_4bits; - } -} - -void generate_zeros_data(std::vector& data) { - std::fill(data.begin(), data.end(), 0.0f); -} - -// Correctness checking against reference data -bool ValueSpec::validate_against_reference( - float abs_tolerance, - float rel_tolerance) const { - // Only validate float and half tensors. For half tensors, convert the - // computed half data to float for comparison against the fp32 reference. - if (!is_tensor() || (dtype != vkapi::kFloat && dtype != vkapi::kHalf)) { - return true; // Skip validation for non-float/half or non-tensor types - } - - // For kHalf, materialize the GPU output as float so the same tolerance - // machinery can compare against the (always-float) reference data. - std::vector half_as_float; - if (dtype == vkapi::kHalf) { - const auto& half_bits = get_half_data(); - half_as_float.resize(half_bits.size()); - for (size_t i = 0; i < half_bits.size(); ++i) { - half_as_float[i] = half_to_float(half_bits[i]); - } - } - // Materialize computed data as float32 for comparison. The dtype is - // guaranteed to be float or half by the early-out above. - const std::vector& computed_data = - (dtype == vkapi::kHalf) ? half_as_float : get_float_data(); - const auto& reference_data = get_ref_float_data(); - - // Skip validation if no reference data is available - if (reference_data.empty()) { - return true; - } - - // Check if sizes match - if (computed_data.size() != reference_data.size()) { - if (debugging()) { - std::cout << "Size mismatch: computed=" << computed_data.size() - << ", reference=" << reference_data.size() << std::endl; - } - return false; - } - - // Element-wise comparison with both absolute and relative tolerance - size_t num_mismatched = 0; - size_t first_mismatch = 0; - for (size_t i = 0; i < computed_data.size(); ++i) { - float diff = std::abs(computed_data[i] - reference_data[i]); - float abs_ref = std::abs(reference_data[i]); - - // Check if either absolute or relative tolerance condition is satisfied - bool abs_tolerance_ok = diff <= abs_tolerance; - bool rel_tolerance_ok = diff <= rel_tolerance * abs_ref; - - if (!abs_tolerance_ok && !rel_tolerance_ok) { - if (num_mismatched == 0) { - first_mismatch = i; - std::cout << "Mismatch at element " << i - << ": computed=" << computed_data[i] - << ", reference=" << reference_data[i] << ", diff=" << diff - << ", abs_tolerance=" << abs_tolerance - << ", rel_tolerance=" << rel_tolerance - << ", rel_threshold=" << (rel_tolerance * abs_ref) - << std::endl; - } - num_mismatched++; - } - } - if (num_mismatched > 0) { - std::cout << " total mismatched: " << num_mismatched << " / " - << computed_data.size() << " (first at " << first_mismatch << ")" - << std::endl; - // For 2D outputs, print a per-16x16-tile mismatch-count map to expose - // the spatial structure of the failure (e.g. zeroed MMA subtiles). - if (sizes.size() == 2) { - const int64_t Mr = sizes[0]; - const int64_t Nc = sizes[1]; - std::cout << " 16x16-tile mismatch counts (rows=M/16, cols=N/16):" - << std::endl; - for (int64_t ti = 0; ti < (Mr + 15) / 16; ++ti) { - std::cout << " "; - for (int64_t tj = 0; tj < (Nc + 15) / 16; ++tj) { - int count = 0; - for (int64_t r = ti * 16; r < std::min(Mr, (ti + 1) * 16); ++r) { - for (int64_t c = tj * 16; c < std::min(Nc, (tj + 1) * 16); ++c) { - float diff = std::abs( - computed_data[r * Nc + c] - reference_data[r * Nc + c]); - float abs_ref = std::abs(reference_data[r * Nc + c]); - if (diff > abs_tolerance && diff > rel_tolerance * abs_ref) { - count++; - } - } - } - std::cout << std::setw(4) << count; - } - std::cout << std::endl; - } - } - return false; - } - - if (debugging()) { - std::cout << "Correctness validation PASSED" << std::endl; - } - return true; -} - -// Ensure data is generated for this ValueSpec -void ValueSpec::ensure_data_generated(int seed) const { - if (data_generated_) { - return; - } - generate_tensor_data(seed); - data_generated_ = true; -} - -void ValueSpec::share_data_from(const ValueSpec& other) { - if (!is_tensor() || !other.is_tensor()) { - return; - } - // Materialize the source first: sharing an ungenerated payload would let a - // later access materialize each spec independently under different seeds. - other.ensure_data_generated(); - data_ = other.data_; - data_generated_ = true; -} - -void ValueSpec::share_reference_from(const ValueSpec& other) { - if (!is_tensor() || !other.is_tensor()) { - return; - } - reference_data_ = other.reference_data_; -} - -// ReferenceKey implementation -ReferenceKey ReferenceKey::from_test_case(const TestCase& tc) { - std::ostringstream oss; - - // Serialize inputs that affect reference computation - // Skip: storage_type, memory_layout, string values (like impl_selector) - for (size_t i = 0; i < tc.inputs().size(); ++i) { - const ValueSpec& input = tc.inputs()[i]; - oss << "i" << i << ":"; - - if (input.is_tensor()) { - // For tensors: sizes, dtype, data_gen_type, is_constant - oss << "T["; - for (size_t j = 0; j < input.sizes.size(); ++j) { - if (j > 0) - oss << ","; - oss << input.sizes[j]; - } - oss << "]d" << static_cast(input.dtype); - oss << "g" << static_cast(input.data_gen_type); - oss << "c" << (input.is_constant() ? 1 : 0); - oss << "n" << (input.is_none() ? 1 : 0); - } else if (input.is_int()) { - oss << "I" << input.get_int_value(); - } else if (input.is_float()) { - oss << "F" << input.get_float_value(); - } else if (input.is_bool()) { - oss << "B" << (input.get_bool_value() ? 1 : 0); - } else if (input.is_int_list()) { - oss << "L["; - const auto& list = input.get_int_list(); - for (size_t j = 0; j < list.size(); ++j) { - if (j > 0) - oss << ","; - oss << list[j]; - } - oss << "]"; - } - // Skip string inputs (like impl_selector) as they don't affect reference - oss << ";"; - } - - // Also include output shapes for completeness - for (size_t i = 0; i < tc.outputs().size(); ++i) { - const ValueSpec& output = tc.outputs()[i]; - oss << "o" << i << ":["; - for (size_t j = 0; j < output.sizes.size(); ++j) { - if (j > 0) - oss << ","; - oss << output.sizes[j]; - } - oss << "]d" << static_cast(output.dtype) << ";"; - } - - ReferenceKey key; - key.key_string = oss.str(); - return key; -} - -// Helper function to collect GPU timing from querypool -float collect_gpu_timing_us( - ComputeGraph& graph, - const std::vector& shader_filter) { - graph.context()->querypool().extract_results(); - const auto results = graph.context()->querypool().get_shader_timestamp_data(); - if (!results.empty()) { - // Sum durations of all shaders that don't match any pattern in - // shader_filter - float total_duration_us = 0.0f; - for (const auto& shader_result : results) { - bool filtered = false; - // Check if this shader matches any filter pattern - for (const auto& filter_pattern : shader_filter) { - if (shader_result.kernel_name.find(filter_pattern) != - std::string::npos) { - filtered = true; - break; - } - } - - if (!filtered) { - // Calculate duration from start and end times, convert from ns to μs - uint64_t duration_ns = - shader_result.end_time_ns - shader_result.start_time_ns; - total_duration_us += static_cast(duration_ns) / 1000.0f; - } - } - return total_duration_us; - } - return 0.0f; -} - -// Helper function to collect per-shader GPU timing from querypool -// Returns a map of shader_name -> timing_us for non-filtered shaders -std::unordered_map collect_per_shader_timing_us( - ComputeGraph& graph, - const std::vector& shader_filter) { - std::unordered_map shader_timings; - - graph.context()->querypool().extract_results(); - const auto results = graph.context()->querypool().get_shader_timestamp_data(); - for (const auto& shader_result : results) { - bool filtered = false; - // Check if this shader matches any filter pattern - for (const auto& filter_pattern : shader_filter) { - if (shader_result.kernel_name.find(filter_pattern) != std::string::npos) { - filtered = true; - break; - } - } - - if (!filtered) { - // Calculate duration from start and end times, convert from ns to μs - uint64_t duration_ns = - shader_result.end_time_ns - shader_result.start_time_ns; - float duration_us = static_cast(duration_ns) / 1000.0f; - // Accumulate timing for shaders with the same name - shader_timings[shader_result.kernel_name] += duration_us; - } - } - return shader_timings; -} - -// BenchmarkResult implementation -void BenchmarkResult::add_iter_timing(float time_us) { - iter_timings.push_back(time_us); -} - -void BenchmarkResult::add_shader_timing( - const std::string& shader_name, - float time_us, - const uint32_t gwg[3], - const uint32_t lwg[3]) { - // Find existing shader timing or create new one - for (auto& st : shader_timings_) { - if (st.shader_name == shader_name) { - st.iter_timings_us.push_back(time_us); - // Work group sizes should be consistent across iterations - return; - } - } - // Not found, create new entry - ShaderTiming new_timing; - new_timing.shader_name = shader_name; - new_timing.iter_timings_us.push_back(time_us); - new_timing.gwg[0] = gwg[0]; - new_timing.gwg[1] = gwg[1]; - new_timing.gwg[2] = gwg[2]; - new_timing.lwg[0] = lwg[0]; - new_timing.lwg[1] = lwg[1]; - new_timing.lwg[2] = lwg[2]; - shader_timings_.push_back(std::move(new_timing)); -} - -float BenchmarkResult::get_avg_time_us() const { - if (iter_timings.empty()) { - return 0.0f; - } - - float sum = 0.0f; - for (float timing : iter_timings) { - sum += timing; - } - return sum / iter_timings.size(); -} - -float BenchmarkResult::get_min_time_us() const { - if (iter_timings.empty()) { - return 0.0f; - } - - return *std::min_element(iter_timings.begin(), iter_timings.end()); -} - -float BenchmarkResult::get_max_time_us() const { - if (iter_timings.empty()) { - return 0.0f; - } - - return *std::max_element(iter_timings.begin(), iter_timings.end()); -} - -float BenchmarkResult::get_std_dev_us() const { - if (iter_timings.size() <= 1) { - return 0.0f; - } - - float mean = get_avg_time_us(); - float sum_sq_diff = 0.0f; - - for (float timing : iter_timings) { - float diff = timing - mean; - sum_sq_diff += diff * diff; - } - - return std::sqrt(sum_sq_diff / (iter_timings.size() - 1)); -} - -void BenchmarkResult::print_summary( - int case_number, - const std::string& size_info, - float total_gflops) const { - static constexpr int OPERATOR_NAME_WIDTH = 50; - static constexpr int GLOBAL_WG_WIDTH = 16; - static constexpr int LOCAL_WG_WIDTH = 12; - static constexpr int KERNEL_NAME_WIDTH = 80; - static constexpr int SIZE_INFO_WIDTH = 20; - static constexpr int TIMING_WIDTH = 16; - static constexpr int GFLOPS_WIDTH = 14; - static constexpr int CORRECTNESS_WIDTH = 8; - - // Helper to truncate shader names longer than 46 chars to 44 chars + ".." - auto truncate_shader_name = [](const std::string& name) -> std::string { - if (name.length() > 46) { - return name.substr(0, 44) + ".."; - } - return name; - }; - - // Helper to format work group size as (x,y,z) - auto format_wg_size = [](const uint32_t wg[3]) -> std::string { - return "(" + std::to_string(wg[0]) + "," + std::to_string(wg[1]) + "," + - std::to_string(wg[2]) + ")"; - }; - - std::string correctness_str; - switch (correctness_status_) { - case CorrectnessStatus::SKIPPED: - correctness_str = "SKIPPED"; - break; - case CorrectnessStatus::PASSED: - correctness_str = "PASSED"; - break; - case CorrectnessStatus::FAILED: - correctness_str = "FAILED"; - break; - } - - // If we have per-shader timing data, print one line per shader plus overall - if (!shader_timings_.empty()) { - // If only one shader, print a single combined row - if (shader_timings_.size() == 1) { - const auto& st = shader_timings_[0]; - std::cout << std::left << std::setw(OPERATOR_NAME_WIDTH) - << truncate_shader_name(st.shader_name) << " " << std::left - << std::setw(GLOBAL_WG_WIDTH) << format_wg_size(st.gwg) - << std::left << std::setw(LOCAL_WG_WIDTH) - << format_wg_size(st.lwg) << std::left - << std::setw(KERNEL_NAME_WIDTH) << get_kernel_name() - << std::right << " " << std::setw(SIZE_INFO_WIDTH) << size_info - << std::setw(TIMING_WIDTH) << std::fixed << std::setprecision(3) - << get_avg_time_us() << " μs " << std::setw(GFLOPS_WIDTH) - << std::fixed << std::setprecision(3) << total_gflops - << " GFLOP/s " << std::setw(CORRECTNESS_WIDTH) - << correctness_str << std::endl; - } else { - // Multiple shaders: print individual shader lines (without GFLOP/s) - for (size_t i = 0; i < shader_timings_.size(); ++i) { - const auto& st = shader_timings_[i]; - float shader_avg_time = st.get_avg_time_us(); - - // Shader lines don't show test case info - std::cout << std::left << std::setw(OPERATOR_NAME_WIDTH) - << truncate_shader_name(st.shader_name) << " " << std::left - << std::setw(GLOBAL_WG_WIDTH) << format_wg_size(st.gwg) - << std::left << std::setw(LOCAL_WG_WIDTH) - << format_wg_size(st.lwg) << std::left - << std::setw(KERNEL_NAME_WIDTH) << "" << std::right << " " - << std::setw(SIZE_INFO_WIDTH) << "" << std::setw(TIMING_WIDTH) - << std::fixed << std::setprecision(3) << shader_avg_time - << " μs " << std::setw(GFLOPS_WIDTH) << "" << " " - << std::setw(CORRECTNESS_WIDTH) << "" << std::endl; - } - - // Print overall row with operator name, test case info, total time, and - // GFLOP/s - std::cout << std::left << std::setw(OPERATOR_NAME_WIDTH) - << get_operator_name() << " " << std::left - << std::setw(GLOBAL_WG_WIDTH) << "" << std::left - << std::setw(LOCAL_WG_WIDTH) << "" << std::left - << std::setw(KERNEL_NAME_WIDTH) << get_kernel_name() - << std::right << " " << std::setw(SIZE_INFO_WIDTH) << size_info - << std::setw(TIMING_WIDTH) << std::fixed << std::setprecision(3) - << get_avg_time_us() << " μs " << std::setw(GFLOPS_WIDTH) - << std::fixed << std::setprecision(3) << total_gflops - << " GFLOP/s " << std::setw(CORRECTNESS_WIDTH) - << correctness_str << std::endl; - } - - // Print separator line between test cases - } else { - // No per-shader timing data, use the original format - std::cout << std::left << std::setw(OPERATOR_NAME_WIDTH) - << get_operator_name() << " " << std::left - << std::setw(GLOBAL_WG_WIDTH) << "" << std::left - << std::setw(LOCAL_WG_WIDTH) << "" << std::left - << std::setw(KERNEL_NAME_WIDTH) << get_kernel_name() << std::right - << " " << std::setw(SIZE_INFO_WIDTH) << size_info - << std::setw(TIMING_WIDTH) << std::fixed << std::setprecision(3) - << get_avg_time_us() << " μs " << std::setw(GFLOPS_WIDTH) - << std::fixed << std::setprecision(3) << total_gflops - << " GFLOP/s " << std::setw(CORRECTNESS_WIDTH) << correctness_str - << std::endl; - } -} - -// TestResult implementation -void TestResult::add_result(const BenchmarkResult& result) { - results_.push_back(result); -} - -void TestResult::add_result(BenchmarkResult&& result) { - results_.push_back(std::move(result)); -} - -void TestResult::print_summary() const { - static constexpr int CASE_WIDTH = 100; - static constexpr int KERNEL_NAME_WIDTH = 20; - static constexpr int TIMING_WIDTH = 12; - static constexpr int PASS_WIDTH = 8; - - if (results_.empty()) { - std::cout << "No results to display" << std::endl; - return; - } - - std::cout << "\n=== " << operation_name_ - << " Performance Summary ===" << std::endl; - print_separator(); - - std::cout << std::left << std::setw(CASE_WIDTH) << "Case" << std::left - << std::setw(KERNEL_NAME_WIDTH) << "Kernel Name" << std::left - << std::setw(TIMING_WIDTH) << "Avg (μs)" << std::left - << std::setw(TIMING_WIDTH) << "Min (μs)" << std::left - << std::setw(TIMING_WIDTH) << "Max (μs)" << std::left - << std::setw(TIMING_WIDTH) << "Std Dev" << std::left - << std::setw(PASS_WIDTH) << "Pass" << std::endl; - print_separator(); - - for (size_t i = 0; i < results_.size(); ++i) { - const auto& result = results_[i]; - bool vulkan_execute_succeeded = - result.get_num_iterations() > 0 && result.get_avg_time_us() > 0.0f; - std::cout << std::left << std::setw(CASE_WIDTH) << i + 1 << std::left - << std::setw(KERNEL_NAME_WIDTH) - << result.get_kernel_name().substr(0, KERNEL_NAME_WIDTH - 1) - << std::left << std::setw(TIMING_WIDTH) << std::fixed - << std::setprecision(3) << result.get_avg_time_us() << std::left - << std::setw(TIMING_WIDTH) << std::fixed << std::setprecision(3) - << result.get_min_time_us() << std::left - << std::setw(TIMING_WIDTH) << std::fixed << std::setprecision(3) - << result.get_max_time_us() << std::left - << std::setw(TIMING_WIDTH) << std::fixed << std::setprecision(3) - << result.get_std_dev_us() << std::left << std::setw(PASS_WIDTH) - << (vulkan_execute_succeeded ? "✓" : "✗") << std::endl; - } - - print_separator(); - std::cout << "Total cases: " << results_.size() - << ", Passed: " << get_passed_count() - << ", Failed: " << get_failed_count() << std::endl; - std::cout << "Overall GFLOP/s: " << std::fixed << std::setprecision(3) - << gflops_ << std::endl; - std::cout << "Overall correctness: " - << (correctness_passed_ ? "PASSED" : "FAILED") << std::endl; -} - -void TestResult::print_detailed_results() const { - if (results_.empty()) { - std::cout << "No results to display" << std::endl; - return; - } - - std::cout << "\n=== " << operation_name_ - << " Detailed Results ===" << std::endl; - - for (size_t i = 0; i < results_.size(); ++i) { - const auto& result = results_[i]; - bool vulkan_execute_succeeded = - result.get_num_iterations() > 0 && result.get_avg_time_us() > 0.0f; - std::cout << "\nCase " << i + 1 << ": " << result.get_kernel_name() - << std::endl; - std::cout << " Iterations: " << result.get_num_iterations() << std::endl; - std::cout << " Average: " << std::fixed << std::setprecision(3) - << result.get_avg_time_us() << " μs" << std::endl; - std::cout << " Min: " << std::fixed << std::setprecision(3) - << result.get_min_time_us() << " μs" << std::endl; - std::cout << " Max: " << std::fixed << std::setprecision(3) - << result.get_max_time_us() << " μs" << std::endl; - std::cout << " Std Dev: " << std::fixed << std::setprecision(3) - << result.get_std_dev_us() << " μs" << std::endl; - std::cout << " Correctness: " - << (vulkan_execute_succeeded ? "PASSED" : "FAILED") << std::endl; - - if (result.get_num_iterations() > 0) { - std::cout << " Individual timings (μs): "; - const auto& timings = result.get_iter_timings(); - for (size_t j = 0; j < std::min(size_t(10), timings.size()); ++j) { - std::cout << std::fixed << std::setprecision(1) << timings[j]; - if (j < std::min(size_t(10), timings.size()) - 1) - std::cout << ", "; - } - if (timings.size() > 10) { - std::cout << " ... (" << (timings.size() - 10) << " more)"; - } - std::cout << std::endl; - } - } - - std::cout << "\nOverall Results:" << std::endl; - std::cout << " Total GFLOP/s: " << std::fixed << std::setprecision(3) - << gflops_ << std::endl; - std::cout << " Overall correctness: " - << (correctness_passed_ ? "PASSED" : "FAILED") << std::endl; -} - -void TestResult::print_statistics() const { - if (results_.empty()) { - std::cout << "No results to display statistics for" << std::endl; - return; - } - - std::cout << "\n=== " << operation_name_ << " Statistics ===" << std::endl; - std::cout << "Total test cases: " << results_.size() << std::endl; - std::cout << "Passed: " << get_passed_count() << std::endl; - std::cout << "Failed: " << get_failed_count() << std::endl; - std::cout << "Success rate: " << std::fixed << std::setprecision(1) - << (100.0f * get_passed_count() / results_.size()) << "%" - << std::endl; - - if (get_passed_count() > 0) { - std::cout << "Total average time: " << std::fixed << std::setprecision(3) - << get_total_avg_time_us() << " μs" << std::endl; - std::cout << "Total GFLOP/s: " << std::fixed << std::setprecision(3) - << get_total_gflops() << std::endl; - - const auto* fastest = get_fastest_result(); - const auto* slowest = get_slowest_result(); - const auto* highest_gflops = get_highest_gflops_result(); - - if (fastest) { - std::cout << "Fastest case: " << fastest->get_kernel_name() << " (" - << std::fixed << std::setprecision(3) - << fastest->get_avg_time_us() << " μs)" << std::endl; - } - - if (slowest) { - std::cout << "Slowest case: " << slowest->get_kernel_name() << " (" - << std::fixed << std::setprecision(3) - << slowest->get_avg_time_us() << " μs)" << std::endl; - } - - if (highest_gflops) { - std::cout << "Best performing case: " << highest_gflops->get_kernel_name() - << " (" << std::fixed << std::setprecision(3) - << highest_gflops->get_avg_time_us() << " μs)" << std::endl; - } - } -} - -void TestResult::print_brief_summary() const { - print_separator(); - std::cout << "Summary Statistics:" << std::endl; - - if (get_passed_count() > 0) { - std::cout << "Average execution time: " << std::fixed - << std::setprecision(3) << get_total_avg_time_us() << " μs" - << std::endl; - std::cout << "Total throughput: " << std::fixed << std::setprecision(3) - << get_gflops() << " GFLOP/s" << std::endl; - std::cout << "Successful test cases: " << get_passed_count() << "/" - << size() << std::endl; - std::cout << "Overall correctness: " - << (get_correctness_passed() ? "PASSED" : "FAILED") << std::endl; - } else { - std::cout << "No successful test cases to report" << std::endl; - } -} - -float TestResult::get_total_avg_time_us() const { - if (results_.empty()) { - return 0.0f; - } - - float sum = 0.0f; - size_t count = 0; - - for (const auto& result : results_) { - bool vulkan_execute_succeeded = - result.get_num_iterations() > 0 && result.get_avg_time_us() > 0.0f; - if (vulkan_execute_succeeded) { - sum += result.get_avg_time_us(); - count++; - } - } - - return count > 0 ? sum / count : 0.0f; -} - -float TestResult::get_total_gflops() const { - return gflops_; -} - -size_t TestResult::get_passed_count() const { - size_t count = 0; - for (const auto& result : results_) { - bool vulkan_execute_succeeded = - result.get_num_iterations() > 0 && result.get_avg_time_us() > 0.0f; - if (vulkan_execute_succeeded) { - count++; - } - } - return count; -} - -size_t TestResult::get_failed_count() const { - return results_.size() - get_passed_count(); -} - -const BenchmarkResult* TestResult::get_fastest_result() const { - const BenchmarkResult* fastest = nullptr; - - for (const auto& result : results_) { - bool vulkan_execute_succeeded = - result.get_num_iterations() > 0 && result.get_avg_time_us() > 0.0f; - if (vulkan_execute_succeeded) { - if (!fastest || result.get_avg_time_us() < fastest->get_avg_time_us()) { - fastest = &result; - } - } - } - - return fastest; -} - -const BenchmarkResult* TestResult::get_slowest_result() const { - const BenchmarkResult* slowest = nullptr; - - for (const auto& result : results_) { - bool vulkan_execute_succeeded = - result.get_num_iterations() > 0 && result.get_avg_time_us() > 0.0f; - if (vulkan_execute_succeeded) { - if (!slowest || result.get_avg_time_us() > slowest->get_avg_time_us()) { - slowest = &result; - } - } - } - - return slowest; -} - -const BenchmarkResult* TestResult::get_highest_gflops_result() const { - // Since GFLOPS is now a TestResult-level metric rather than per-case, - // this method returns the fastest result as a proxy for highest performance - return get_fastest_result(); -} - -// Default FLOP calculation function (assumes 1 FLOP per element) -int64_t default_flop_calculator(const TestCase& test_case) { - // Calculate total elements from the first input tensor - int64_t total_elements = 1; - if (!test_case.empty() && test_case.num_inputs() > 0 && - test_case.inputs()[0].is_tensor()) { - const auto& sizes = test_case.inputs()[0].get_tensor_sizes(); - for (int64_t size : sizes) { - total_elements *= size; - } - } - - // Assume 1 FLOP per element for basic operations - return total_elements; -} - -BenchmarkGraph setup_compute_graph( - TestCase& test_case, - std::string op_name, - int op_invocations_per_execute) { - GraphConfig config; - config.enable_querypool = true; - // Pool sizing takes max(execute, prepack) * factor, so scaling the factor - // also over-reserves the prepack side (encoded once). Accepted: precise - // execute-only sizing would need runtime changes. - config.descriptor_pool_safety_factor *= - std::max(1, op_invocations_per_execute); - // Default-on (opt-out via TestCase::set_force_resize(false)): force every - // DynamicDispatchNode to run its resize function when execute_test_case - // runs propagate_resize() after prepack, exercising the op's resize formula - // even when input shapes are unchanged. - config.force_resize = test_case.get_force_resize(); - config.force_narrow_int4_tile = test_case.get_force_narrow_int4_tile(); - auto graph = std::make_unique(config); - - std::vector input_values; - - // Process input ValueSpecs - for (size_t i = 0; i < test_case.num_inputs(); ++i) { - const ValueSpec& input_spec = test_case.inputs()[i]; - - if (input_spec.is_none()) { - input_values.push_back(graph->add_none()); - } else if (input_spec.is_float()) { - ValueRef input_value = - graph->add_scalar(static_cast(input_spec.get_float_value())); - input_values.push_back(input_value); - } else if (input_spec.is_int()) { - ValueRef input_value = - graph->add_scalar(static_cast(input_spec.get_int_value())); - input_values.push_back(input_value); - } else if (input_spec.is_bool()) { - ValueRef input_value = graph->add_scalar(input_spec.get_bool_value()); - input_values.push_back(input_value); - } else if (input_spec.is_int_list()) { - // Convert int32_t list to int64_t list for ComputeGraph - const auto& int32_list = input_spec.get_int_list(); - std::vector int64_list; - int64_list.reserve(int32_list.size()); - for (int32_t val : int32_list) { - int64_list.push_back(static_cast(val)); - } - ValueRef input_value = graph->add_scalar_list(std::move(int64_list)); - input_values.push_back(input_value); - } else if (input_spec.is_string()) { - std::string str_copy = input_spec.get_string_value(); - ValueRef input_value = graph->add_string(std::move(str_copy)); - input_values.push_back(input_value); - } else if (input_spec.is_constant()) { - ValueRef input_value = graph->add_tensorref( - input_spec.get_tensor_sizes(), - input_spec.dtype, - input_spec.get_data_ptr()); - input_values.push_back(input_value); - } else { - IOValueRef input_io = graph->add_input_tensor( - input_spec.get_tensor_sizes(), - input_spec.dtype, - input_spec.storage_type, - input_spec.memory_layout); - input_values.push_back(input_io.value); - } - } - - std::vector output_values; - - // Process output ValueSpecs - for (size_t i = 0; i < test_case.num_outputs(); ++i) { - const ValueSpec& output_spec = test_case.outputs()[i]; - - if (!output_spec.is_tensor()) { - throw std::runtime_error("All output specifications must be tensors"); - } - - // Create output tensor - ValueRef output_value = graph->add_tensor( - output_spec.get_tensor_sizes(), - output_spec.dtype, - output_spec.storage_type, - output_spec.memory_layout); - - output_values.push_back(output_value); - } - - // Get the operator function and call it - auto opFn = VK_GET_OP_FN(op_name); - - // Create arguments vector for the operator function - std::vector op_args = input_values; - op_args.insert(op_args.end(), output_values.begin(), output_values.end()); - - // Nodes added before the op are staging uploads; nodes added after are - // staging downloads. Only the op's own nodes are repeated by benchmarks. - const size_t op_begin = graph->execute_nodes().size(); - opFn(*graph, op_args); - const size_t op_end = graph->execute_nodes().size(); - - for (size_t i = 0; i < output_values.size(); ++i) { - graph->set_output_value(output_values[i]); - } - return {std::move(graph), {op_begin, op_end}}; -} - -// Test execution utilities -BenchmarkResult execute_test_case( - TestCase& test_case, - int warmup_runs, - int benchmark_runs, - int chained_dispatches, - bool write_outputs) { - BenchmarkResult result( - test_case.name().empty() ? "unnamed_test_case" : test_case.name()); - - // Initialize querypool if using GPU timestamps - if (use_gpu_timestamps()) { - api::context()->initialize_querypool(); - } - - // Build the operator once. Benchmark repetition is encoded separately so - // persistent graph allocations are not duplicated. - BenchmarkGraph benchmark = setup_compute_graph( - test_case, test_case.operator_name(), chained_dispatches); - ComputeGraph& graph = *benchmark.graph; - - // Prepare the graph - graph.prepare(); - graph.prepack(); - - // Run resize functions once so force_resize exercises resize formulas even - // though the record/replay path below never calls propagate_resize(). - graph.propagate_resize(); - - // Copy input data into the graph's staging buffers - size_t graph_input_idx = 0; - for (size_t i = 0; i < test_case.num_inputs(); ++i) { - const ValueSpec& input_spec = test_case.inputs()[i]; - - // Only non-constant tensor inputs correspond to graph.inputs() entries - bool is_graph_input = input_spec.is_tensor() && !input_spec.is_constant() && - !input_spec.is_none(); - if (!is_graph_input) { - continue; - } - - if (graph_input_idx < graph.inputs().size()) { - const auto& input_ref = graph.inputs()[graph_input_idx]; - - // Get the appropriate data based on dtype - const void* data_ptr = nullptr; - size_t data_numel = input_spec.numel(); - - switch (input_spec.dtype) { - case vkapi::kFloat: - data_ptr = input_spec.get_float_data().data(); - break; - case vkapi::kHalf: - data_ptr = input_spec.get_half_data().data(); - break; - case vkapi::kInt: - data_ptr = input_spec.get_int32_data().data(); - break; - case vkapi::kChar: - data_ptr = input_spec.get_int8_data().data(); - break; - case vkapi::kByte: - data_ptr = input_spec.get_uint8_data().data(); - break; - default: - throw std::runtime_error("Unsupported data type for input tensor"); - } - - // Copy data into staging buffer - graph.maybe_cast_and_copy_into_staging( - input_ref.staging, data_ptr, data_numel, input_spec.dtype); - } - ++graph_input_idx; - } - - RepeatedGraphExecutor graph_executor( - graph, chained_dispatches, benchmark.op_nodes); - - // Warmup runs - for (int run = 0; run < warmup_runs; ++run) { - graph_executor.execute(); - } - - // Benchmark runs - collect individual iteration timings - float total_cpu_time_us = 0.0f; - float total_gpu_time_us = 0.0f; - - const float chained_dispatches_f = static_cast(chained_dispatches); - - for (int run = 0; run < benchmark_runs; ++run) { - // Measure CPU time for each execute() call - auto cpu_start = std::chrono::high_resolution_clock::now(); - graph_executor.execute(); - auto cpu_end = std::chrono::high_resolution_clock::now(); - - auto cpu_duration = std::chrono::duration_cast( - cpu_end - cpu_start); - // Each execute() ran chained_dispatches op invocations; report - // per-invocation time. - float cpu_time_us = - static_cast(cpu_duration.count()) / chained_dispatches_f; - total_cpu_time_us += cpu_time_us; - - // Collect per-shader GPU timing - get raw shader results to preserve - // metadata - graph.context()->querypool().extract_results(); - const auto shader_results = - graph.context()->querypool().get_shader_timestamp_data(); - - // Calculate total GPU time from per-shader timings - float gpu_time_us = 0.0f; - for (const auto& shader_result : shader_results) { - // Check if this shader matches any filter pattern - bool filtered = false; - for (const auto& filter_pattern : test_case.get_shader_filter()) { - if (shader_result.kernel_name.find(filter_pattern) != - std::string::npos) { - filtered = true; - break; - } - } - - if (!filtered) { - uint64_t duration_ns = - shader_result.end_time_ns - shader_result.start_time_ns; - float duration_us = static_cast(duration_ns) / 1000.0f; - gpu_time_us += duration_us; - // Per-shader sample is already one querypool entry per dispatch (at - // chained_dispatches>1 we get that many entries per shader), so do NOT - // divide here — the aggregator (get_avg_time_us) averages across them - // naturally. - result.add_shader_timing( - shader_result.kernel_name, - duration_us, - shader_result.metadata.gwg, - shader_result.metadata.lwg); - } - } - // gpu_time_us aggregates chained_dispatches worth of shader runs; divide - // it down to per-invocation time. - gpu_time_us /= chained_dispatches_f; - total_gpu_time_us += gpu_time_us; - - // Add the appropriate timing based on the flag - float iter_time_us = use_gpu_timestamps() ? gpu_time_us : cpu_time_us; - result.add_iter_timing(iter_time_us); - } - - // Calculate averages for display - float avg_cpu_time_us = total_cpu_time_us / benchmark_runs; - float avg_gpu_time_us = total_gpu_time_us / benchmark_runs; - - // Print both timings if latency printing is enabled - if (print_latencies()) { - if (use_gpu_timestamps()) { - graph.context()->querypool().print_results(); - } - std::cout << " CPU timing: " << std::fixed << std::setprecision(3) - << avg_cpu_time_us << " μs" << std::endl; - std::cout << " GPU timing: " << std::fixed << std::setprecision(3) - << avg_gpu_time_us << " μs" << std::endl; - std::cout << " Using " << (use_gpu_timestamps() ? "GPU" : "CPU") - << " timing for result" << std::endl; - } - - // Copy output data from the graph's staging buffers. The benchmarking path - // skips this work since it doesn't need correctness data — the probe pass - // has already populated test_case.outputs() at chained_dispatches=1. - if (write_outputs) { - for (size_t i = 0; i < test_case.num_outputs(); ++i) { - ValueSpec& output_spec = test_case.outputs()[i]; - - if (output_spec.is_tensor() && i < graph.outputs().size()) { - const auto& output_ref = graph.outputs()[i]; - - // Ensure output data vector is properly sized - size_t data_numel = output_spec.numel(); - output_spec.resize_data(data_numel); - - // Get mutable data pointer for the output - void* data_ptr = output_spec.get_mutable_data_ptr(); - - if (data_ptr != nullptr) { - // Copy data from staging buffer to output spec - graph.maybe_cast_and_copy_from_staging( - output_ref.staging, data_ptr, data_numel, output_spec.dtype); - } - - // Print output tensor data if output printing is enabled - if (print_output()) { - std::string output_name = "Output[" + std::to_string(i) + "]"; - print_valuespec_data(output_spec, output_name); - } - } - } - } - - return result; -} - -TestResult execute_test_cases( - std::function()> test_case_generator, - FlopCalculatorFunc flop_calculator, - const std::string& operation_name, - int warmup_runs, - int benchmark_runs, - ReferenceComputeFunc reference_compute_func) { - TestResult results(operation_name); - - // Generate all test cases - std::vector test_cases = test_case_generator(); - - std::cout << "Executing " << test_cases.size() << " test cases for " - << operation_name << std::endl; - print_separator(); - - // Group test cases by ReferenceKey for caching reference computations - // Use a vector to preserve the order in which groups first appear - std::vector group_order; - std::unordered_map, ReferenceKeyHash> - groups; - for (size_t i = 0; i < test_cases.size(); ++i) { - ReferenceKey key = ReferenceKey::from_test_case(test_cases[i]); - if (groups.find(key) == groups.end()) { - group_order.push_back(key); - } - groups[key].push_back(i); - } - - bool any_correctness_failed = false; - float total_gflops = 0.0f; - size_t test_case_counter = 0; - - // Process each group: generate data, compute reference, execute, and print - // Iterate in the order groups first appeared in test_cases - for (const auto& key : group_order) { - const auto& indices = groups[key]; - if (indices.empty()) - continue; - - // Get first test case as the "prototype" - size_t prototype_idx = indices[0]; - TestCase& prototype = test_cases[prototype_idx]; - - // Generate data for prototype with deterministic seed based on key - int group_seed = - static_cast(std::hash{}(key.key_string) % 10000); - for (auto& input : prototype.inputs()) { - input.ensure_data_generated(group_seed++); - } - - // Compute reference once for prototype - bool ref_computed = false; - if (reference_compute_func) { - try { - reference_compute_func(prototype); - ref_computed = true; - } catch (const std::invalid_argument&) { - // Reference computation skipped for this group - } - } - - // Copy data and reference to other test cases in group - for (size_t i = 1; i < indices.size(); ++i) { - size_t tc_idx = indices[i]; - TestCase& tc = test_cases[tc_idx]; - - // Copy input data from prototype - for (size_t j = 0; - j < tc.inputs().size() && j < prototype.inputs().size(); - ++j) { - auto& dest = tc.inputs()[j]; - const auto& src = prototype.inputs()[j]; - if (dest.is_tensor() && src.is_tensor() && dest.sizes == src.sizes && - dest.dtype == src.dtype) { - dest.share_data_from(src); - } - } - - // Copy reference output data if available - if (ref_computed) { - for (size_t j = 0; - j < tc.outputs().size() && j < prototype.outputs().size(); - ++j) { - const auto& src = prototype.outputs()[j]; - auto& dest = tc.outputs()[j]; - if (dest.is_tensor() && src.is_tensor() && dest.sizes == src.sizes && - dest.dtype == src.dtype) { - dest.share_reference_from(src); - } - } - } - } - - // Execute and print results for all test cases in this group - for (size_t tc_idx : indices) { - TestCase& test_case = test_cases[tc_idx]; - ++test_case_counter; - - // Execute single test case - BenchmarkResult result; - bool shader_not_supported = false; - try { - // Always run a probe pass at chained_dispatches=1 with - // write_outputs=true. This populates test_case.outputs() for the - // downstream correctness check with a clean single-dispatch result, - // and also gives us probe_us for adaptive sizing of the measurement - // run's chained_dispatches. - // - // probe_then_scale (Google Benchmark style): size chained_dispatches - // so each measurement execute() takes ~target_us. Tiny ops get a - // large factor (driving GPU governor escalation); heavy ops get a - // small factor (bounded wall time). Caveat: on Adreno 740 the probe - // runs at the DCVS-pinned clock (~220 MHz), so probe_us is inflated - // and the computed factor comes out under-sized for boost clock — but - // the default target is generous enough that an under-sized factor - // still drives sustained activity. - BenchmarkResult probe_result = execute_test_case( - test_case, - /*warmup_runs=*/1, - /*benchmark_runs=*/1, - /*chained_dispatches=*/1, - /*write_outputs=*/true); - float probe_us = probe_result.get_avg_time_us(); - if (probe_us < kMinProbeTimeUs) { - probe_us = kMinProbeTimeUs; - } - - int chained_dispatches; - const int manual_n = test_case.get_op_invocations_per_execute(); - if (manual_n > 0) { - chained_dispatches = manual_n; - } else { - const int target_us = test_case.get_target_execute_time_us(); - chained_dispatches = - std::max(1, static_cast(target_us / probe_us)); - chained_dispatches = - std::min(chained_dispatches, kMaxChainedDispatches); - } - if (debugging()) { - std::cout << "[probe] " << test_case.name() - << ": probe_us=" << probe_us - << ", chained_dispatches=" << chained_dispatches - << std::endl; - } - - // Measurement pass: chained_dispatches stacking, skip output copy - // since the probe already wrote the correctness data. - result = execute_test_case( - test_case, - warmup_runs, - benchmark_runs, - chained_dispatches, - /*write_outputs=*/false); - result.set_operator_name(test_case.operator_name()); - } catch (const vkcompute::vkapi::ShaderNotSupportedError&) { - result = BenchmarkResult( - test_case.name().empty() ? "unnamed_test_case" : test_case.name(), - test_case.operator_name()); - shader_not_supported = true; - } - - // Determine if this test case passed (has valid timing data) - bool vulkan_execute_succeeded = - result.get_num_iterations() > 0 && result.get_avg_time_us() > 0.0f; - - if (shader_not_supported) { - result.set_correctness_status(CorrectnessStatus::SKIPPED); - } else if (!vulkan_execute_succeeded) { - result.set_correctness_status(CorrectnessStatus::FAILED); - } else if (!ref_computed) { - result.set_correctness_status(CorrectnessStatus::SKIPPED); - } else { - // Reference function provided and succeeded - validate outputs - bool correctness_passed = true; - - for (size_t output_idx = 0; output_idx < test_case.num_outputs(); - ++output_idx) { - const ValueSpec& output_spec = test_case.outputs()[output_idx]; - - if (!output_spec.validate_against_reference( - test_case.get_abs_tolerance(), - test_case.get_rel_tolerance())) { - correctness_passed = false; - std::cout << " Correctness validation FAILED for test " - << result.get_kernel_name() << std::endl; - print_valuespec_data(output_spec, "vulkan output"); - print_valuespec_data(output_spec, "ref output", true); - } - } - - if (correctness_passed) { - result.set_correctness_status(CorrectnessStatus::PASSED); - } else { - any_correctness_failed = true; - result.set_correctness_status(CorrectnessStatus::FAILED); - } - } - - // Calculate GFLOPS for this test case using the provided FLOP calculator - float case_gflops = 0.0f; - if (vulkan_execute_succeeded) { - // Use the provided FLOP calculator to get total FLOPs for this test - // case - int64_t total_flops = flop_calculator(test_case); - float flops = static_cast(total_flops); - float avg_time_us = result.get_avg_time_us(); - if (avg_time_us > 0.0f && total_flops > 0) { - case_gflops = (flops / 1e9f) / (avg_time_us / 1e6f); - } - - total_gflops += case_gflops; - } else { - case_gflops = -1.0f; // Indicate failure - } - - // Calculate tensor info for display - std::string size_info = "["; - if (!test_case.empty() && test_case.num_inputs() > 0 && - test_case.inputs()[0].is_tensor()) { - const auto& sizes = test_case.inputs()[0].get_tensor_sizes(); - for (size_t j = 0; j < sizes.size(); ++j) { - size_info += std::to_string(sizes[j]); - if (j < sizes.size() - 1) - size_info += "x"; - } - } - size_info += "]"; - - // Print progress using the BenchmarkResult member function - result.print_summary(test_case_counter, size_info, case_gflops); - - // Add result to collection - results.add_result(std::move(result)); - - test_case.clear(); - } - } - - // Set the overall results on the TestResult - results.set_correctness_passed(!any_correctness_failed); - results.set_gflops(total_gflops); - - print_separator(); - std::cout << "Completed " << results.size() << " test cases" << std::endl; - - // Hard-fail if any correctness check failed. The per-case loop above keeps - // running after a mismatch (so the full pass/fail matrix and per-16x16-tile - // mismatch maps are printed for debugging) rather than aborting on the first - // failure; throwing here preserves the original abort-on-mismatch behavior. - if (any_correctness_failed) { - throw std::runtime_error("Correctness validation failed"); - } - - return results; -} - -// Convenience overload that uses the default FLOP calculator -TestResult execute_test_cases( - std::function()> test_case_generator, - const std::string& operation_name, - int warmup_runs, - int benchmark_runs, - ReferenceComputeFunc reference_compute_func) { - return execute_test_cases( - test_case_generator, - default_flop_calculator, - operation_name, - warmup_runs, - benchmark_runs, - reference_compute_func); -} - -// Utility functions for printing -void print_performance_header() { - std::cout << "\n=== Compute Shader Performance Benchmark ===" << std::endl; -} - -void print_separator() { - std::cout << std::string(70, '-') << std::endl; -} - -// ValueSpec data printing utilities -void print_valuespec_data( - const ValueSpec& spec, - const std::string& name, - const bool print_ref_data, - size_t max_elements, - int precision) { - std::cout << "\n" << name << " Data:" << std::endl; - std::cout << " Type: " << spec.to_string() << std::endl; - - if (!spec.is_tensor()) { - if (spec.is_int()) { - std::cout << " Value: " << spec.get_int_value() << std::endl; - } else if (spec.is_int_list()) { - const auto& int_list = spec.get_int_list(); - std::cout << " Values: ["; - size_t print_count = std::min(max_elements, int_list.size()); - for (size_t i = 0; i < print_count; ++i) { - std::cout << int_list[i]; - if (i < print_count - 1) - std::cout << ", "; - } - if (int_list.size() > max_elements) { - std::cout << ", ... (" << (int_list.size() - max_elements) << " more)"; - } - std::cout << "]" << std::endl; - } - return; - } - - // Print tensor data - size_t total_elements = spec.numel(); - size_t print_count = std::min(max_elements, total_elements); - - std::cout << " Total elements: " << total_elements << std::endl; - std::cout << " Data (first " << print_count << " elements): ["; - - std::cout << std::fixed << std::setprecision(precision); - - switch (spec.dtype) { - case vkapi::kFloat: { - auto data = spec.get_float_data().data(); - if (print_ref_data) { - data = spec.get_ref_float_data().data(); - } - for (size_t i = 0; i < print_count; ++i) { - std::cout << data[i]; - if (i < print_count - 1) - std::cout << ", "; - } - break; - } - case vkapi::kHalf: { - if (print_ref_data) { - const auto& ref = spec.get_ref_float_data(); - for (size_t i = 0; i < print_count; ++i) { - std::cout << ref[i]; - if (i < print_count - 1) - std::cout << ", "; - } - break; - } - const auto& data = spec.get_half_data(); - for (size_t i = 0; i < print_count; ++i) { - // Convert IEEE 754 half-precision bit pattern back to float. - float value = half_to_float(data[i]); - std::cout << value; - if (i < print_count - 1) - std::cout << ", "; - } - break; - } - case vkapi::kInt: { - const auto& data = spec.get_int32_data(); - for (size_t i = 0; i < print_count; ++i) { - std::cout << data[i]; - if (i < print_count - 1) - std::cout << ", "; - } - break; - } - case vkapi::kChar: { - const auto& data = spec.get_int8_data(); - if (spec.is_int4()) { - // Print each 4-bit value individually - size_t element_count = 0; - for (size_t i = 0; i < data.size() && element_count < print_count; - ++i) { - // Extract lower 4 bits (signed) - int8_t lower_4bits = data[i] & 0x0F; - if (lower_4bits > 7) - lower_4bits -= 16; // Convert to signed - std::cout << static_cast(lower_4bits); - element_count++; - - if (element_count < print_count) { - std::cout << ", "; - // Extract upper 4 bits (signed) - int8_t upper_4bits = (data[i] >> 4) & 0x0F; - if (upper_4bits > 7) - upper_4bits -= 16; // Convert to signed - std::cout << static_cast(upper_4bits); - element_count++; - - if (element_count < print_count) - std::cout << ", "; - } - } - } else { - for (size_t i = 0; i < print_count; ++i) { - std::cout << static_cast(data[i]); - if (i < print_count - 1) - std::cout << ", "; - } - } - break; - } - case vkapi::kByte: { - const auto& data = spec.get_uint8_data(); - if (spec.is_int4()) { - // Print each 4-bit value individually - size_t element_count = 0; - for (size_t i = 0; i < data.size() && element_count < print_count; - ++i) { - // Extract lower 4 bits - uint8_t lower_4bits = data[i] & 0x0F; - std::cout << static_cast(lower_4bits); - element_count++; - - if (element_count < print_count) { - std::cout << ", "; - // Extract upper 4 bits - uint8_t upper_4bits = (data[i] >> 4) & 0x0F; - std::cout << static_cast(upper_4bits); - element_count++; - - if (element_count < print_count) - std::cout << ", "; - } - } - } else { - for (size_t i = 0; i < print_count; ++i) { - std::cout << static_cast(data[i]); - if (i < print_count - 1) - std::cout << ", "; - } - } - break; - } - default: - std::cout << "unsupported data type"; - break; - } - - if (total_elements > max_elements) { - std::cout << ", ... (" << (total_elements - max_elements) << " more)"; - } - std::cout << "]" << std::endl; - - // Print some statistics for tensor data - if (total_elements > 0) { - float min_val = 0.0f, max_val = 0.0f, sum = 0.0f; - bool first = true; - - for (size_t i = 0; i < total_elements; ++i) { - float val = spec.get_element(i); - if (first) { - min_val = max_val = val; - first = false; - } else { - min_val = std::min(min_val, val); - max_val = std::max(max_val, val); - } - sum += val; - } - - float mean = sum / total_elements; - std::cout << " Statistics: min=" << std::setprecision(precision) << min_val - << ", max=" << max_val << ", mean=" << mean << ", sum=" << sum - << std::endl; - } -} - -ValueRef quantized_weights_canvas( - ComputeGraph& graph, - const ValueRef weight_ref) { - const auto original_sizes = graph.sizes_of(weight_ref); - - std::vector sorted_sizes = original_sizes; - std::sort(sorted_sizes.begin(), sorted_sizes.end(), std::greater()); - int64_t largest1 = sorted_sizes.size() > 0 ? sorted_sizes[0] : 0; - - std::vector final_sizes = {1, largest1, largest1}; - - // Debug logging if debugging flag is set - if (debugging()) { - std::cout << "Debug: Creating quantized weights canvas tensor" << std::endl; - std::cout << "Debug: Original sizes: ["; - for (size_t i = 0; i < original_sizes.size(); ++i) { - std::cout << original_sizes[i]; - if (i < original_sizes.size() - 1) - std::cout << ", "; - } - std::cout << "]" << std::endl; - std::cout << "Debug: Canvas sizes: ["; - for (size_t i = 0; i < final_sizes.size(); ++i) { - std::cout << final_sizes[i]; - if (i < final_sizes.size() - 1) - std::cout << ", "; - } - std::cout << "]" << std::endl; - } - - ValueRef packed_weight = graph.add_tensor( - final_sizes, vkapi::kInt, utils::kTexture3D, utils::kWidthPacked); - - std::string kernel_name = "packed_int32_canvas"; - add_storage_type_suffix(kernel_name, graph.storage_type_of(packed_weight)); - - graph.prepack_nodes().emplace_back(new PrepackNode( - graph, - VK_KERNEL_FROM_STR(kernel_name), - graph.create_gwg(packed_weight), - graph.create_lwg(packed_weight), - weight_ref, - packed_weight, - // UBOs - {graph.logical_limits_ubo(packed_weight)}, - // Specialization constants - {}, - // Push Constants - {})); - - return packed_weight; -} - -ValueRef float_tensor_canvas(ComputeGraph& graph, const ValueRef weight_ref) { - const auto original_sizes = graph.sizes_of(weight_ref); - - std::vector sorted_sizes = original_sizes; - std::sort(sorted_sizes.begin(), sorted_sizes.end(), std::greater()); - int64_t largest1 = sorted_sizes.size() > 0 ? sorted_sizes[0] : 0; - - std::vector final_sizes = {1, largest1, largest1}; - - // Debug logging if debugging flag is set - if (debugging()) { - std::cout << "Debug: Creating float tensor canvas" << std::endl; - std::cout << "Debug: Original sizes: ["; - for (size_t i = 0; i < original_sizes.size(); ++i) { - std::cout << original_sizes[i]; - if (i < original_sizes.size() - 1) - std::cout << ", "; - } - std::cout << "]" << std::endl; - std::cout << "Debug: Canvas sizes: ["; - for (size_t i = 0; i < final_sizes.size(); ++i) { - std::cout << final_sizes[i]; - if (i < final_sizes.size() - 1) - std::cout << ", "; - } - std::cout << "]" << std::endl; - } - - ValueRef packed_weight = graph.add_tensor( - final_sizes, vkapi::kFloat, utils::kTexture3D, utils::kWidthPacked); - - graph.prepack_nodes().emplace_back(new PrepackNode( - graph, - VK_KERNEL_FROM_STR("float_canvas"), - graph.create_gwg(packed_weight), - graph.create_lwg(packed_weight), - weight_ref, - packed_weight, - // UBOs - {graph.logical_limits_ubo(packed_weight)}, - // Specialization constants - {}, - // Push Constants - {})); - - return packed_weight; -} - -// Compute weight sums for quantized operations (linear and convolution) -void compute_weight_sums( - ValueSpec& weight_sums, - const ValueSpec& quantized_weight, - int64_t out_features, - int64_t elements_per_output_feature) { - auto& weight_sums_data = weight_sums.get_int32_data(); - auto& quantized_weight_data = quantized_weight.get_int8_data(); - - // Don't resize down - the buffer may be pre-allocated with aligned size. - // Only resize up if needed. - if (weight_sums_data.size() < static_cast(out_features)) { - weight_sums_data.resize(out_features); - } - - // For each output feature, compute the sum of quantized weights - for (int64_t out_f = 0; out_f < out_features; ++out_f) { - int32_t sum = 0; - for (int64_t elem = 0; elem < elements_per_output_feature; ++elem) { - // Weight indexing depends on the layout: - // For linear: [out_features, in_features] -> out_f * - // elements_per_output_feature + elem For conv2d: [C_out, C_in * K_h * - // K_w] -> out_f * elements_per_output_feature + elem - int64_t weight_idx = out_f * elements_per_output_feature + elem; - sum += static_cast(quantized_weight_data[weight_idx]); - } - weight_sums_data[out_f] = sum; - } -} - -// Compute weight sums for 4D quantized conv2d operations -// Weight layout: [C_out, K_h, K_w, align_up_4(C_in_per_group)] -void compute_weight_sums_4d( - ValueSpec& weight_sums, - const ValueSpec& quantized_weight, - int64_t out_channels, - int64_t kernel_h, - int64_t kernel_w, - int64_t aligned_in_channels) { - auto& weight_sums_data = weight_sums.get_int32_data(); - auto& quantized_weight_data = quantized_weight.get_int8_data(); - - weight_sums_data.resize(out_channels); - - // For each output channel, compute the sum of quantized weights - for (int64_t out_c = 0; out_c < out_channels; ++out_c) { - int32_t sum = 0; - - for (int64_t kh = 0; kh < kernel_h; ++kh) { - for (int64_t kw = 0; kw < kernel_w; ++kw) { - for (int64_t in_c = 0; in_c < aligned_in_channels; ++in_c) { - // Weight indexing: [out_c, kh, kw, in_c] - int64_t weight_idx = - out_c * (kernel_h * kernel_w * aligned_in_channels) + - kh * (kernel_w * aligned_in_channels) + kw * aligned_in_channels + - in_c; - sum += static_cast(quantized_weight_data[weight_idx]); - } - } - } - - weight_sums_data[out_c] = sum; - } -} - -// Helper function to unpack 4-bit values from uint8 (same as in -// q4gsw_linear.cpp) -std::pair unpack_4bit_utils(uint8_t packed) { - // Extract lower 4 bits and upper 4 bits - int8_t lower = packed & 0x0F; - int8_t upper = (packed >> 4) & 0x0F; - - // Subtract 8 from unpacked 4-bit values - lower -= 8; - upper -= 8; - - return std::make_pair(lower, upper); -} - -// Compute weight sums for 4-bit group symmetric quantized weights -void compute_weight_sums_4bit_grouped( - ValueSpec& weight_sums, - const ValueSpec& quantized_weight, - int64_t num_groups, - int64_t out_features, - int64_t group_size) { - auto& weight_sums_data = weight_sums.get_int32_data(); - auto& quantized_weight_data = quantized_weight.get_uint8_data(); - - // Resize to [num_groups, out_features] - weight_sums_data.resize(num_groups * out_features); - - // For each group and each output feature, compute the sum of quantized - // weights in that group - for (int64_t group_idx = 0; group_idx < num_groups; ++group_idx) { - for (int64_t out_f = 0; out_f < out_features; ++out_f) { - int32_t sum = 0; - - // Sum weights for this group and output feature - for (int64_t in_group = 0; in_group < group_size; ++in_group) { - int64_t in_f = group_idx * group_size + in_group; - - // Get packed weight value - weight matrix is [N, K/2] - int64_t weight_idx = - out_f * ((num_groups * group_size) / 2) + (in_f / 2); - uint8_t packed_weight = quantized_weight_data[weight_idx]; - - // Unpack 4-bit weight - auto unpacked = unpack_4bit_utils(packed_weight); - int8_t weight_4bit = (in_f % 2 == 0) ? unpacked.first : unpacked.second; - - sum += static_cast(weight_4bit); - } - - // Store sum for this group and output feature - int64_t sums_idx = group_idx * out_features + out_f; - weight_sums_data[sums_idx] = sum; - } - } -} - -} // namespace prototyping -} // namespace vulkan -} // namespace executorch diff --git a/backends/vulkan/test/custom_ops/utils.h b/backends/vulkan/test/custom_ops/utils.h deleted file mode 100644 index b94d0d41421..00000000000 --- a/backends/vulkan/test/custom_ops/utils.h +++ /dev/null @@ -1,1094 +0,0 @@ -// Copyright (c) Meta Platforms, Inc. and affiliates. -// All rights reserved. -// -// This source code is licensed under the BSD-style license found in the -// LICENSE file in the root directory of this source tree. - -#pragma once - -#include -#include -#include - -#include -#include -#include -#include -#include -#include -#include -#include - -namespace executorch { -namespace vulkan { -namespace prototyping { - -using namespace vkcompute; - -// -// ReferenceKey for caching reference computations -// - -// Captures the identity of input conditions for test case grouping. -// Test cases with the same ReferenceKey should produce identical reference -// outputs, so reference computation can be cached and reused. -struct ReferenceKey { - std::string key_string; - - static ReferenceKey from_test_case(const class TestCase& tc); - - bool operator==(const ReferenceKey& other) const { - return key_string == other.key_string; - } -}; - -struct ReferenceKeyHash { - size_t operator()(const ReferenceKey& k) const { - return std::hash{}(k.key_string); - } -}; - -// -// Global configuration options -// - -bool print_output(); -void set_print_output(bool print_output); - -bool print_latencies(); -void set_print_latencies(bool print_latencies); - -bool use_gpu_timestamps(); -void set_use_gpu_timestamps(bool use_timestamps); - -bool debugging(); -void set_debugging(bool enable_debugging); - -// -// Shader filter defaults -// - -// Default shader filter: excludes layout conversions and quantization overhead -// This is used for tests where quantize/dequantize are overhead operations -inline const std::vector kDefaultShaderFilter = { - "nchw_to", - "to_nchw", - "quantize_and_pack_4w4c", - "unpack_4w4c_and_dequantize"}; - -// Layout-only filter: only excludes layout conversions -// Use this for tests where quantize/dequantize ARE the operations being tested -inline const std::vector kLayoutOnlyShaderFilter = { - "nchw_to", - "to_nchw"}; - -// -// String utilities -// - -// Helper function to get abbreviated layout names for test case naming -inline std::string layout_abbrev(utils::GPUMemoryLayout layout) { - switch (layout) { - case utils::kWidthPacked: - return "WP"; - case utils::kChannelsPacked: - return "CP"; - case utils::kPackedInt8_4W: - return "4W"; - case utils::kPackedInt8_4C: - return "4C"; - case utils::kPackedInt8_4W4C: - return "4W4C"; - case utils::kPackedInt8_4H4W: - return "4H4W"; - case utils::kPackedInt8_4C1W: - return "4C1W"; - default: - return "UNK"; - } -} - -// Helper function to get abbreviated storage type names for test case naming -inline std::string storage_type_abbrev(utils::StorageType storage_type) { - switch (storage_type) { - case utils::kTexture3D: - return "Tex"; - case utils::kBuffer: - return "Buf"; - default: - return "UNK"; - } -} - -// Helper function to get combined storage type and layout representation -// Example: (kBuffer, kPackedInt8_4W4C) -> "Buf_4W4C" -inline std::string repr_str( - utils::StorageType storage_type, - utils::GPUMemoryLayout layout) { - return storage_type_abbrev(storage_type) + "(" + layout_abbrev(layout) + ")"; -} - -// Helper function to generate comma-separated shape string for test case naming -// Example: {1, 128, 56, 56} -> "1,128,56,56" -inline std::string shape_string(const std::vector& shape) { - std::string result; - for (size_t i = 0; i < shape.size(); ++i) { - if (i > 0) { - result += ","; - } - result += std::to_string(shape[i]); - } - return result; -} - -// Helper function to generate a bracketed shape string for test case naming -// Example: {1, 128, 56, 56} -> "[1,128,56,56]" -inline std::string shape_bracket(const std::vector& shape) { - return "[" + shape_string(shape) + "]"; -} - -// Short dtype symbol used in standardized test case labels -// Example: kFloat -> "f32", kHalf -> "f16", kChar -> "i8", kInt -> "i32" -inline std::string dtype_short(vkapi::ScalarType dtype) { - switch (dtype) { - case vkapi::kFloat: - return "f32"; - case vkapi::kHalf: - return "f16"; - case vkapi::kChar: - return "i8"; - case vkapi::kByte: - return "u8"; - case vkapi::kInt: - return "i32"; - case vkapi::kBool: - return "b"; - default: - return "?"; - } -} - -// Build a standardized test label of the form: -// " [ ]" -// where is "->" when the two differ, or just -// "" when they match. Sections are separated by two spaces. If -// suffix is non-empty it is appended after a single space (allowing callers -// to pass e.g. "[general]" or "[gemv] +bias"). -inline std::string make_test_label( - const std::string& prefix, - const std::string& in_dtype, - const std::string& out_dtype, - const std::string& shape_str, - const std::string& storage_str, - const std::string& suffix = "") { - const std::string dtype_str = - (in_dtype == out_dtype) ? in_dtype : in_dtype + "->" + out_dtype; - std::string label = - prefix + " " + dtype_str + " " + shape_str + " " + storage_str; - if (!suffix.empty()) { - label += " " + suffix; - } - return label; -} - -// -// ValueSpec class -// - -enum class SpecType { Tensor, IntList, Int, Float, Bool, String }; - -// Data generation types -enum class DataGenType { - FIXED, - RANDOM, - RANDOM_SCALES, - RANDINT, - RANDINT8, - RANDINT4, - ONES, - ONES_INT4, - ZEROS -}; - -// Value specification struct -struct ValueSpec { - std::vector sizes; - vkapi::ScalarType dtype; - utils::GPUMemoryLayout memory_layout; - utils::StorageType storage_type; - SpecType spec_type; - DataGenType data_gen_type; - bool is_constant_tensor; - bool is_none_flag; - bool is_int4_tensor; - std::string string_data; - - ValueSpec( - const std::vector& sizes, - vkapi::ScalarType dtype, - utils::StorageType storage_type = utils::kTexture3D, - utils::GPUMemoryLayout memory_layout = utils::kWidthPacked) - : sizes(sizes), - dtype(dtype), - memory_layout(memory_layout), - storage_type(storage_type), - spec_type(SpecType::Tensor), - data_gen_type(DataGenType::ZEROS), - is_constant_tensor(false), - is_none_flag(false), - is_int4_tensor(false), - data_generated_(false) { - // Data generation is deferred until first access (any data getter or - // ensure_data_generated() triggers it). - } - - // Constructor for tensor with custom data generation type - ValueSpec( - const std::vector& sizes, - vkapi::ScalarType dtype, - utils::StorageType storage_type, - utils::GPUMemoryLayout memory_layout, - DataGenType data_gen_type) - : sizes(sizes), - dtype(dtype), - memory_layout(memory_layout), - storage_type(storage_type), - spec_type(SpecType::Tensor), - data_gen_type(data_gen_type), - is_constant_tensor(false), - is_none_flag(false), - is_int4_tensor(false), - data_generated_(false) { - // Data generation is deferred until first access (any data getter or - // ensure_data_generated() triggers it). - } - - // Constructor for single int - ValueSpec(int32_t value) - : sizes({1}), - dtype(vkapi::kInt), - memory_layout(utils::kWidthPacked), - storage_type(utils::kTexture3D), - spec_type(SpecType::Int), - data_gen_type(DataGenType::FIXED), - is_constant_tensor(false), - is_none_flag(false), - is_int4_tensor(false), - data_generated_(true) { - data_->int32_data.push_back(value); - } - - // Constructor for single float - ValueSpec(float value) - : sizes({1}), - dtype(vkapi::kFloat), - memory_layout(utils::kWidthPacked), - storage_type(utils::kTexture3D), - spec_type(SpecType::Float), - data_gen_type(DataGenType::FIXED), - is_constant_tensor(false), - is_none_flag(false), - is_int4_tensor(false), - data_generated_(true) { - data_->float_data.push_back(value); - } - - // Constructor for single bool - ValueSpec(bool value) - : sizes({1}), - dtype(vkapi::kInt), - memory_layout(utils::kWidthPacked), - storage_type(utils::kTexture3D), - spec_type(SpecType::Bool), - data_gen_type(DataGenType::FIXED), - is_constant_tensor(false), - is_none_flag(false), - is_int4_tensor(false), - data_generated_(true) { - data_->int32_data.push_back(value ? 1 : 0); - } - - // Constructor for int list - ValueSpec(const std::vector& values) - : sizes({static_cast(values.size())}), - dtype(vkapi::kInt), - memory_layout(utils::kWidthPacked), - storage_type(utils::kTexture3D), - spec_type(SpecType::IntList), - data_gen_type(DataGenType::FIXED), - is_constant_tensor(false), - is_none_flag(false), - is_int4_tensor(false), - data_generated_(true) { - data_->int32_data = values; - } - - // Factory method for string (avoids ambiguity with vector constructor) - static ValueSpec make_string(const std::string& value) { - ValueSpec spec; - spec.sizes = {1}; - spec.dtype = vkapi::kInt; - spec.memory_layout = utils::kWidthPacked; - spec.storage_type = utils::kTexture3D; - spec.spec_type = SpecType::String; - spec.data_gen_type = DataGenType::FIXED; - spec.is_constant_tensor = false; - spec.is_none_flag = false; - spec.is_int4_tensor = false; - spec.data_generated_ = true; - spec.string_data = value; - return spec; - } - - // Default constructor - ValueSpec() - : dtype(vkapi::kFloat), - memory_layout(utils::kWidthPacked), - storage_type(utils::kTexture3D), - spec_type(SpecType::Tensor), - data_gen_type(DataGenType::ZEROS), - is_constant_tensor(false), - is_none_flag(false), - is_int4_tensor(false), - data_generated_(false) {} - - int64_t numel() const; - size_t nbytes() const; - std::string to_string() const; - - bool is_tensor() const { - return spec_type == SpecType::Tensor; - } - bool is_int_list() const { - return spec_type == SpecType::IntList; - } - bool is_int() const { - return spec_type == SpecType::Int; - } - bool is_float() const { - return spec_type == SpecType::Float; - } - bool is_bool() const { - return spec_type == SpecType::Bool; - } - bool is_string() const { - return spec_type == SpecType::String; - } - - int32_t get_int_value() const { - ensure_data_generated(); - return data_->int32_data.empty() ? 0 : data_->int32_data[0]; - } - float get_float_value() const { - ensure_data_generated(); - return data_->float_data.empty() ? 0.0f : data_->float_data[0]; - } - bool get_bool_value() const { - ensure_data_generated(); - return data_->int32_data.empty() ? false : (data_->int32_data[0] != 0); - } - const std::string& get_string_value() const { - return string_data; - } - const std::vector& get_int_list() const { - ensure_data_generated(); - return data_->int32_data; - } - const std::vector& get_tensor_sizes() const { - return sizes; - } - - // References and pointers into tensor data must not be held across any other - // access to the same spec: a mutable access may detach the shared payload, - // leaving a previously returned reference bound to the old payload. Consume - // immediately. - const std::vector& get_float_data() const { - ensure_data_generated(); - return data_->float_data; - } - const std::vector& get_int32_data() const { - ensure_data_generated(); - return data_->int32_data; - } - const std::vector& get_half_data() const { - ensure_data_generated(); - return data_->half_data; - } - const std::vector& get_int8_data() const { - ensure_data_generated(); - return data_->int8_data; - } - const std::vector& get_uint8_data() const { - ensure_data_generated(); - return data_->uint8_data; - } - - std::vector& get_float_data() { - ensure_data_generated(); - ensure_unique_data(); - return data_->float_data; - } - std::vector& get_int32_data() { - ensure_data_generated(); - ensure_unique_data(); - return data_->int32_data; - } - std::vector& get_half_data() { - ensure_data_generated(); - ensure_unique_data(); - return data_->half_data; - } - std::vector& get_int8_data() { - ensure_data_generated(); - ensure_unique_data(); - return data_->int8_data; - } - std::vector& get_uint8_data() { - ensure_data_generated(); - ensure_unique_data(); - return data_->uint8_data; - } - - const std::vector& get_ref_float_data() const { - return reference_data_->float_data; - } - const std::vector& get_ref_int32_data() const { - return reference_data_->int32_data; - } - const std::vector& get_ref_half_data() const { - return reference_data_->half_data; - } - const std::vector& get_ref_int8_data() const { - return reference_data_->int8_data; - } - const std::vector& get_ref_uint8_data() const { - return reference_data_->uint8_data; - } - - std::vector& get_ref_float_data() { - ensure_unique_reference_data(); - return reference_data_->float_data; - } - std::vector& get_ref_int32_data() { - ensure_unique_reference_data(); - return reference_data_->int32_data; - } - std::vector& get_ref_half_data() { - ensure_unique_reference_data(); - return reference_data_->half_data; - } - std::vector& get_ref_int8_data() { - ensure_unique_reference_data(); - return reference_data_->int8_data; - } - std::vector& get_ref_uint8_data() { - ensure_unique_reference_data(); - return reference_data_->uint8_data; - } - - void resize_data(size_t new_size); - void* get_mutable_data_ptr(); - float get_element(size_t index) const; - - // Data generation methods for deferred generation and caching. - // - // ValueSpec is not thread-safe: lazy materialization and copy-on-write - // detach mutate shared state from const methods. Test cases are built and - // executed on a single thread. - // - // Implicit materialization (any data getter, resize_data) consumes the - // global seed counter. Callers needing deterministic data must call - // ensure_data_generated(explicit_seed) before any other access; a later - // seeded call is a no-op once data is generated. - bool is_data_generated() const { - return data_generated_; - } - void ensure_data_generated(int seed = -1) const; - void share_data_from(const ValueSpec& other); - void share_reference_from(const ValueSpec& other); - - // Set/get constant flag - bool is_constant() const { - return is_constant_tensor; - } - void set_constant(bool is_constant) { - is_constant_tensor = is_constant; - } - - // Set/get none flag - bool is_none() const { - return is_none_flag; - } - - void set_none(bool is_none) { - is_none_flag = is_none; - } - - // Set/get int4 flag - bool is_int4() const { - return is_int4_tensor; - } - void set_int4(bool is_int4) { - is_int4_tensor = is_int4; - } - - const void* get_data_ptr() const; - - // Correctness checking against reference data - // Returns true if computed data matches reference data within tolerance - // Only validates float tensors as specified in requirements - bool validate_against_reference( - float abs_tolerance = 2e-3f, - float rel_tolerance = 1e-3f) const; - - private: - struct TensorData { - std::vector float_data; - std::vector int32_data; - std::vector half_data; - std::vector int8_data; - std::vector uint8_data; - }; - - void ensure_unique_data() const; - void ensure_unique_reference_data() const; - void generate_tensor_data(int seed = -1) const; - - mutable bool data_generated_ = false; - mutable std::shared_ptr data_ = std::make_shared(); - mutable std::shared_ptr reference_data_ = - std::make_shared(); -}; - -// -// TestCase -// - -// Default per-execute() wall-clock target used by the probe-then-scale logic -// in execute_test_cases(). Picked generously enough that even an under-sized -// chained_dispatches factor (the probe runs at governor-pinned clock and -// underestimates the boost-clock latency) still drives sustained GPU -// activity during measurement. -constexpr int kDefaultTargetExecuteTimeUs = 100000; - -class TestCase { - public: - TestCase() - : abs_tolerance_(2e-3f), - rel_tolerance_(1e-3f), - shader_filter_(kDefaultShaderFilter) {} - TestCase(const std::string& name) - : name_(name), - abs_tolerance_(2e-3f), - rel_tolerance_(1e-3f), - shader_filter_(kDefaultShaderFilter) {} - - void set_name(const std::string& name) { - name_ = name; - } - const std::string& name() const { - return name_; - } - - void set_operator_name(const std::string& op_name) { - operator_name_ = op_name; - } - const std::string& operator_name() const { - return operator_name_; - } - - // Tolerance settings - void set_abs_tolerance(float abs_tolerance) { - abs_tolerance_ = abs_tolerance; - } - float get_abs_tolerance() const { - return abs_tolerance_; - } - - void set_rel_tolerance(float rel_tolerance) { - rel_tolerance_ = rel_tolerance; - } - float get_rel_tolerance() const { - return rel_tolerance_; - } - - // Shader filter settings - list of shader name patterns to exclude from - // timing - void set_shader_filter(const std::vector& filter) { - shader_filter_ = filter; - } - const std::vector& get_shader_filter() const { - return shader_filter_; - } - - // Manual override for the number of chained dispatches per measurement - // iteration (a.k.a. chained_dispatches). If > 0, the framework uses this - // directly and skips the probe phase. 0 (the default) means adaptive - // (probe-then-scale). - void set_op_invocations_per_execute(int n) { - op_invocations_per_execute_ = n; - } - int get_op_invocations_per_execute() const { - return op_invocations_per_execute_; - } - - // Target single-execute duration in microseconds. Used only when the - // manual chained_dispatches override is not set. Default - // kDefaultTargetExecuteTimeUs, picked generously enough to mitigate Adreno - // DCVS governor pinning during the probe. - void set_target_execute_time_us(int us) { - target_execute_time_us_ = us; - } - int get_target_execute_time_us() const { - return target_execute_time_us_; - } - - // When true, the ComputeGraph built for this test case sets - // GraphConfig::force_resize, so every DynamicDispatchNode runs its resize - // function once during measurement setup (execute_test_case runs - // propagate_resize() after prepack) even when no input shape changed. - // Because the output is already allocated at the swept shape, the resize - // must recompute the same shape from the current input — a wrong resize - // formula resizes the output to a mismatched shape and surfaces as a test - // failure. Default true (opt-out): every custom_ops test exercises its - // resize formulas across the swept shapes. Call set_force_resize(false) for - // the rare op whose resize fn is intentionally not shape-preserving under a - // fixed output allocation. - void set_force_resize(bool force_resize) { - force_resize_ = force_resize; - } - bool get_force_resize() const { - return force_resize_; - } - - // When true, the ComputeGraph sets GraphConfig::force_narrow_int4_tile, so - // the int4 linear picks its narrow (2-row) output tile on any GPU. That - // variant is otherwise reachable only on Mali, which no CI runner has. - void set_force_narrow_int4_tile(bool force) { - force_narrow_int4_tile_ = force; - } - bool get_force_narrow_int4_tile() const { - return force_narrow_int4_tile_; - } - - void add_input_spec(const ValueSpec& spec) { - inputs_.push_back(spec); - } - - const std::vector& inputs() const { - return inputs_; - } - - std::vector& inputs() { - return inputs_; - } - - size_t num_inputs() const { - return inputs_.size(); - } - - void add_output_spec(const ValueSpec& spec) { - outputs_.push_back(spec); - } - - const std::vector& outputs() const { - return outputs_; - } - - std::vector& outputs() { - return outputs_; - } - - size_t num_outputs() const { - return outputs_.size(); - } - - bool empty() const { - return inputs_.empty() && outputs_.empty(); - } - void clear() { - inputs_.clear(); - outputs_.clear(); - name_.clear(); - operator_name_.clear(); - abs_tolerance_ = 2e-3f; - rel_tolerance_ = 1e-3f; - shader_filter_ = kDefaultShaderFilter; - op_invocations_per_execute_ = 0; - target_execute_time_us_ = kDefaultTargetExecuteTimeUs; - force_resize_ = true; - force_narrow_int4_tile_ = false; - } - - private: - std::string name_; - std::string operator_name_; - std::vector inputs_; - std::vector outputs_; - float abs_tolerance_; - float rel_tolerance_; - std::vector shader_filter_; - int op_invocations_per_execute_ = 0; // 0 = adaptive - int target_execute_time_us_ = kDefaultTargetExecuteTimeUs; - bool force_resize_ = true; - bool force_narrow_int4_tile_ = false; -}; - -// -// BenchmarkResult -// - -enum class CorrectnessStatus { - SKIPPED, // No reference function provided - PASSED, // Reference function provided and validation passed - FAILED // Reference function provided but validation failed -}; - -// Per-shader timing data for detailed reporting -struct ShaderTiming { - std::string shader_name; - std::vector iter_timings_us; // Individual iteration timings - uint32_t gwg[3] = {0, 0, 0}; - uint32_t lwg[3] = {0, 0, 0}; - - float get_avg_time_us() const { - if (iter_timings_us.empty()) { - return 0.0f; - } - float sum = 0.0f; - for (float t : iter_timings_us) { - sum += t; - } - return sum / iter_timings_us.size(); - } -}; - -class BenchmarkResult { - public: - BenchmarkResult() : correctness_status_(CorrectnessStatus::SKIPPED) {} - - BenchmarkResult(const std::string& name) - : kernel_name(name), correctness_status_(CorrectnessStatus::SKIPPED) {} - - BenchmarkResult( - const std::string& kernel_name, - const std::string& operator_name) - : kernel_name(kernel_name), - operator_name(operator_name), - correctness_status_(CorrectnessStatus::SKIPPED) {} - - // Add timing for a single iteration - void add_iter_timing(float time_us); - - // Add per-shader timing for a single iteration - void add_shader_timing( - const std::string& shader_name, - float time_us, - const uint32_t gwg[3], - const uint32_t lwg[3]); - - // Get per-shader timing data - const std::vector& get_shader_timings() const { - return shader_timings_; - } - - // Getters - const std::string& get_kernel_name() const { - return kernel_name; - } - const std::string& get_operator_name() const { - return operator_name; - } - float get_avg_time_us() const; - size_t get_num_iterations() const { - return iter_timings.size(); - } - const std::vector& get_iter_timings() const { - return iter_timings; - } - CorrectnessStatus get_correctness_status() const { - return correctness_status_; - } - - // Setters - void set_kernel_name(const std::string& name) { - kernel_name = name; - } - void set_operator_name(const std::string& name) { - operator_name = name; - } - void set_correctness_status(CorrectnessStatus status) { - correctness_status_ = status; - } - - // Statistics - float get_min_time_us() const; - float get_max_time_us() const; - float get_std_dev_us() const; - - // Clear all timings - void clear_timings() { - iter_timings.clear(); - } - - // Print progress for this benchmark result - void print_summary( - int case_number, - const std::string& size_info, - float total_gflops) const; - - private: - std::string kernel_name; - std::string operator_name; - std::vector - iter_timings; // Individual iteration timings in microseconds - std::vector shader_timings_; // Per-shader timing data - CorrectnessStatus correctness_status_; -}; - -// Test result collection and processing -class TestResult { - public: - TestResult() : gflops_(0.0f), correctness_passed_(true) {} - TestResult(const std::string& operation_name) - : operation_name_(operation_name), - gflops_(0.0f), - correctness_passed_(true) {} - - // Add a benchmark result - void add_result(const BenchmarkResult& result); - void add_result(BenchmarkResult&& result); - - // Getters - const std::string& get_operation_name() const { - return operation_name_; - } - float get_gflops() const { - return gflops_; - } - bool get_correctness_passed() const { - return correctness_passed_; - } - size_t size() const { - return results_.size(); - } - bool empty() const { - return results_.empty(); - } - - // Setters - void set_gflops(float gflops_val) { - gflops_ = gflops_val; - } - void set_correctness_passed(bool passed) { - correctness_passed_ = passed; - } - - // Access results - const BenchmarkResult& operator[](size_t index) const { - return results_[index]; - } - BenchmarkResult& operator[](size_t index) { - return results_[index]; - } - const std::vector& get_results() const { - return results_; - } - - // Iterator support - std::vector::iterator begin() { - return results_.begin(); - } - std::vector::iterator end() { - return results_.end(); - } - std::vector::const_iterator begin() const { - return results_.begin(); - } - std::vector::const_iterator end() const { - return results_.end(); - } - - // Processing and analysis - void print_summary() const; - void print_detailed_results() const; - void print_statistics() const; - void print_brief_summary() const; - - // Get aggregate statistics - float get_total_avg_time_us() const; - float get_total_gflops() const; - size_t get_passed_count() const; - size_t get_failed_count() const; - - // Find best/worst performing results - const BenchmarkResult* get_fastest_result() const; - const BenchmarkResult* get_slowest_result() const; - const BenchmarkResult* get_highest_gflops_result() const; - - // Clear all results - void clear() { - results_.clear(); - } - - // Set operation name - void set_operation_name(const std::string& name) { - operation_name_ = name; - } - - private: - std::string operation_name_; - std::vector results_; - float gflops_; - bool correctness_passed_; -}; - -// -// Test case execution -// - -using FlopCalculatorFunc = std::function; - -// Default FLOP calculation function (assumes 1 FLOP per element) -int64_t default_flop_calculator(const TestCase& test_case); - -using ReferenceComputeFunc = std::function; - -// Half-open index range of the operator's own dispatch nodes within a -// benchmark graph's execute_nodes(). Staging upload nodes precede it, staging -// download nodes follow it. -struct OpNodeRange { - size_t begin = 0; - size_t end = 0; -}; - -// A benchmark graph plus the location of its repeatable operator nodes. The -// graph is heap-held: ComputeGraph owns its Context and must never be moved -// (a moved-from graph's destructor dereferences a null context). -struct BenchmarkGraph { - std::unique_ptr graph; - OpNodeRange op_nodes; -}; - -// Benchmark-only executor that records a graph's execute nodes into a single -// reusable command buffer, then replays it on every execute(). Staging uploads -// are encoded once, operator nodes N times, staging downloads once, so each -// iteration performs the same work as the old stacked-nodes layout (1 upload + -// N ops + 1 download) and the per-invocation divisor is unchanged. Production -// ComputeGraph::execute() behavior is unchanged. -// -// Notes for interpreting benchmark numbers: -// - Submit granularity differs from stacking N distinct nodes: all encodings -// live in one command buffer with one submit per iteration (the old path -// could split across command buffers at the node-count threshold), so -// per-dispatch times may shift systematically against older data. -// - Resize functions run once via propagate_resize() in execute_test_case -// before recording; replay itself never re-triggers resize. -// - Repeated encodings share one node/dispatch id, so per-repetition -// querypool attribution is unavailable (aggregation keys on kernel name). -class RepeatedGraphExecutor final { - public: - RepeatedGraphExecutor( - ComputeGraph& graph, - int repetitions, - OpNodeRange op_nodes); - void execute(); - - private: - ComputeGraph& graph_; - std::unique_ptr command_; -}; - -// Runs a measurement at the given chained_dispatches factor. The operator is -// built once, then its execute nodes are encoded that many times into a -// benchmark-only reusable command buffer. The probe-then-scale orchestration -// lives in execute_test_cases(). -// -// write_outputs controls whether the graph's staging output buffers are copied -// back into test_case.outputs() at the end of the run. The probe path needs -// write_outputs=true so the correctness check has a clean single-dispatch -// reference. The benchmarking path passes write_outputs=false to avoid the -// per-iter GPU->CPU copy cost. -BenchmarkResult execute_test_case( - TestCase& test_case, - int warmup_runs = 1, - int benchmark_runs = 1, - int chained_dispatches = 1, - bool write_outputs = true); - -TestResult execute_test_cases( - std::function()> test_case_generator, - FlopCalculatorFunc flop_calculator, - const std::string& operation_name = "Operation", - int warmup_runs = 1, - int benchmark_runs = 1, - ReferenceComputeFunc reference_compute_func = nullptr); - -TestResult execute_test_cases( - std::function()> test_case_generator, - const std::string& operation_name = "Operation", - int warmup_runs = 1, - int benchmark_runs = 1, - ReferenceComputeFunc reference_compute_func = nullptr); - -// -// Print utilities -// - -void print_performance_header(); -void print_separator(); - -void print_valuespec_data( - const ValueSpec& spec, - const std::string& name = "ValueSpec", - const bool print_ref_data = false, - size_t max_elements = 20, - int precision = 6); - -ValueRef quantized_weights_canvas( - ComputeGraph& graph, - const ValueRef weight_ref); - -ValueRef float_tensor_canvas(ComputeGraph& graph, const ValueRef weight_ref); - -// Compute weight sums for quantized operations (linear and convolution) -void compute_weight_sums( - ValueSpec& weight_sums, - const ValueSpec& quantized_weight, - int64_t out_features, - int64_t elements_per_output_feature); - -// Compute weight sums for 4D quantized conv2d operations -// Weight layout: [C_out, K_h, K_w, align_up_4(C_in_per_group)] -void compute_weight_sums_4d( - ValueSpec& weight_sums, - const ValueSpec& quantized_weight, - int64_t out_channels, - int64_t kernel_h, - int64_t kernel_w, - int64_t aligned_in_channels); - -// Compute weight sums for 4-bit group symmetric quantized weights -void compute_weight_sums_4bit_grouped( - ValueSpec& weight_sums, - const ValueSpec& quantized_weight, - int64_t num_groups, - int64_t out_features, - int64_t group_size); - -// Half-precision conversion utilities -uint16_t float_to_half(float value); -float half_to_float(uint16_t half_val); - -// Setup compute graph based on TestCase and operation name. The op function is -// invoked once. op_invocations_per_execute is used only to reserve enough -// descriptor capacity for benchmark-only repeated command encoding. Returns -// the graph plus the range of the operator's own nodes for repeated encoding. -BenchmarkGraph setup_compute_graph( - TestCase& test_case, - std::string op_name, - int op_invocations_per_execute = 1); - -} // namespace prototyping -} // namespace vulkan -} // namespace executorch diff --git a/backends/vulkan/test/custom_ops/utils_test.cpp b/backends/vulkan/test/custom_ops/utils_test.cpp index 6d2de13d71e..104d585769e 100644 --- a/backends/vulkan/test/custom_ops/utils_test.cpp +++ b/backends/vulkan/test/custom_ops/utils_test.cpp @@ -6,7 +6,7 @@ #include -#include "utils.h" +#include namespace executorch::vulkan::prototyping { namespace { diff --git a/backends/vulkan/test/op_tests/conv2d_gemm_dynamic_test.cpp b/backends/vulkan/test/op_tests/conv2d_gemm_dynamic_test.cpp index f993c453fb0..7b4954113ea 100644 --- a/backends/vulkan/test/op_tests/conv2d_gemm_dynamic_test.cpp +++ b/backends/vulkan/test/op_tests/conv2d_gemm_dynamic_test.cpp @@ -35,12 +35,13 @@ using namespace vkcompute; // // The im2col + GEMM conv path (conv2d_gemm_impl) materializes its im2col matrix // in tiles of output-height rows bounded to a byte budget. The static op-test -// suite (test/custom_ops/test_conv2d.cpp) only validates a single shape per -// graph. This test exercises the path under an actual trigger_resize: it builds -// the graph at an upper-bound input shape that forces MULTIPLE tiles, then -// resizes the input across tile boundaries (so trailing tiles must no-op via -// the shader's `oh < H_out` guard and the scratch must track the smaller shape) -// and verifies the output against a reference at every shape. +// suite (test/ops/benchmark/cases/conv2d/test_conv2d.cpp) only validates a +// single shape per graph. This test exercises the path under an actual +// trigger_resize: it builds the graph at an upper-bound input shape that forces +// MULTIPLE tiles, then resizes the input across tile boundaries (so trailing +// tiles must no-op via the shader's `oh < H_out` guard and the scratch must +// track the smaller shape) and verifies the output against a reference at every +// shape. // // Build-time upper bound: storage and the fixed num_tiles are built at the // initial input shape, so every resized shape MUST be <= the initial shape per diff --git a/backends/vulkan/test/ops/benchmark/BUCK b/backends/vulkan/test/ops/benchmark/BUCK new file mode 100644 index 00000000000..f559a6f1cfe --- /dev/null +++ b/backends/vulkan/test/ops/benchmark/BUCK @@ -0,0 +1,9 @@ +# Any targets that should be shared between fbcode and xplat must be defined in +# targets.bzl. + +load("@fbsource//tools/build_defs:fbsource_utils.bzl", "is_fbcode") +load(":targets.bzl", "define_common_targets") + +oncall("executorch") + +define_common_targets(is_fbcode = is_fbcode()) diff --git a/backends/vulkan/test/custom_ops/CMakeLists.txt b/backends/vulkan/test/ops/benchmark/CMakeLists.txt similarity index 63% rename from backends/vulkan/test/custom_ops/CMakeLists.txt rename to backends/vulkan/test/ops/benchmark/CMakeLists.txt index f29b518ec06..ff9138779ec 100644 --- a/backends/vulkan/test/custom_ops/CMakeLists.txt +++ b/backends/vulkan/test/ops/benchmark/CMakeLists.txt @@ -25,7 +25,7 @@ message(STATUS "VULKAN_CXX_FLAGS: ${VULKAN_CXX_FLAGS}") # Only build if Vulkan was compiled if(TARGET vulkan_backend) if(NOT EXECUTORCH_ROOT) - set(EXECUTORCH_ROOT ${CMAKE_CURRENT_SOURCE_DIR}/../../../..) + set(EXECUTORCH_ROOT ${CMAKE_CURRENT_SOURCE_DIR}/../../../../..) endif() if(NOT PYTHON_EXECUTABLE) @@ -47,23 +47,30 @@ if(TARGET vulkan_backend) ) # Prototyping utility files - set(PROTOTYPING_UTILS_HEADERS ${CMAKE_CURRENT_SOURCE_DIR}) + set(PROTOTYPING_UTILS_HEADERS ${CMAKE_CURRENT_SOURCE_DIR}/framework) set(PROTOTYPING_UTILS_CPP - ${CMAKE_CURRENT_SOURCE_DIR}/utils.cpp - ${CMAKE_CURRENT_SOURCE_DIR}/conv2d_utils.cpp - ${CMAKE_CURRENT_SOURCE_DIR}/cm_utils.cpp + ${CMAKE_CURRENT_SOURCE_DIR}/framework/cm_utils.cpp + ${CMAKE_CURRENT_SOURCE_DIR}/framework/config.cpp + ${CMAKE_CURRENT_SOURCE_DIR}/framework/conv2d_utils.cpp + ${CMAKE_CURRENT_SOURCE_DIR}/framework/results.cpp + ${CMAKE_CURRENT_SOURCE_DIR}/framework/runner.cpp + ${CMAKE_CURRENT_SOURCE_DIR}/framework/test_case.cpp + ${CMAKE_CURRENT_SOURCE_DIR}/framework/value_spec.cpp + ${CMAKE_CURRENT_SOURCE_DIR}/framework/weight_utils.cpp ) # Prototyping shaders message(STATUS "shader stuff") - set(PROTOTYPING_SHADERS_PATH ${CMAKE_CURRENT_SOURCE_DIR}/glsl) + set(PROTOTYPING_SHADERS_PATH ${CMAKE_CURRENT_SOURCE_DIR}/test_ops/glsl) gen_vulkan_shader_lib_cpp(${PROTOTYPING_SHADERS_PATH}) vulkan_shader_lib(prototyping_shaderlib ${generated_spv_cpp}) target_compile_options(prototyping_shaderlib PRIVATE ${VULKAN_CXX_FLAGS}) message(STATUS "done shader stuff") # Operator implementations library - file(GLOB OPERATOR_IMPL_SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/impl/*.cpp) + file(GLOB OPERATOR_IMPL_SOURCES + ${CMAKE_CURRENT_SOURCE_DIR}/test_ops/impl/*.cpp + ) add_library(operator_implementations STATIC ${OPERATOR_IMPL_SOURCES}) target_include_directories( operator_implementations PRIVATE ${COMMON_INCLUDES} @@ -79,9 +86,9 @@ if(TARGET vulkan_backend) executorch_target_link_options_shared_lib(operator_implementations) # Function to create operator prototype binaries - function(add_operator_prototype OPERATOR_NAME) + function(add_operator_prototype OPERATOR_NAME SUBDIR) set(TARGET_NAME ${OPERATOR_NAME}) - set(SOURCE_FILE ${CMAKE_CURRENT_SOURCE_DIR}/${OPERATOR_NAME}.cpp) + set(SOURCE_FILE ${CMAKE_CURRENT_SOURCE_DIR}/${SUBDIR}/${OPERATOR_NAME}.cpp) add_executable(${TARGET_NAME} ${SOURCE_FILE} ${PROTOTYPING_UTILS_CPP}) target_include_directories(${TARGET_NAME} PRIVATE ${COMMON_INCLUDES}) @@ -94,18 +101,20 @@ if(TARGET vulkan_backend) endfunction() # Define operator prototypes - add_operator_prototype(test_add) - add_operator_prototype(test_q8csw_linear) - add_operator_prototype(test_q8csw_conv2d) - add_operator_prototype(test_q4gsw_linear) - add_operator_prototype(test_choose_qparams_per_row) - add_operator_prototype(test_q8ta_qdq) - add_operator_prototype(test_q8ta_clone) - add_operator_prototype(test_q8ta_binary) - add_operator_prototype(test_q8ta_conv2d) - add_operator_prototype(test_q8ta_conv2d_pw) - add_operator_prototype(test_q8ta_conv2d_dw) - add_operator_prototype(test_mm) - add_operator_prototype(test_coopmat_probe) - add_operator_prototype(test_coopmat_linear_bench) + add_operator_prototype(test_add cases/add) + add_operator_prototype(test_q8csw_linear cases/q8csw_linear) + add_operator_prototype(test_q8csw_conv2d cases/q8csw_conv2d) + add_operator_prototype(test_q4gsw_linear cases/q4gsw_linear) + add_operator_prototype( + test_choose_qparams_per_row cases/choose_qparams_per_row + ) + add_operator_prototype(test_q8ta_qdq cases/q8ta/qdq) + add_operator_prototype(test_q8ta_clone cases/q8ta/clone) + add_operator_prototype(test_q8ta_binary cases/q8ta/binary) + add_operator_prototype(test_q8ta_conv2d cases/q8ta/conv2d) + add_operator_prototype(test_q8ta_conv2d_pw cases/q8ta/conv2d) + add_operator_prototype(test_q8ta_conv2d_dw cases/q8ta/conv2d) + add_operator_prototype(test_mm cases/mm) + add_operator_prototype(test_coopmat_probe framework) + add_operator_prototype(test_coopmat_linear_bench cases/q4gsw_linear) endif() diff --git a/backends/vulkan/test/custom_ops/build_and_run.sh b/backends/vulkan/test/ops/benchmark/build_and_run.sh similarity index 89% rename from backends/vulkan/test/custom_ops/build_and_run.sh rename to backends/vulkan/test/ops/benchmark/build_and_run.sh index 2641970faf8..a03aa3032a3 100755 --- a/backends/vulkan/test/custom_ops/build_and_run.sh +++ b/backends/vulkan/test/ops/benchmark/build_and_run.sh @@ -49,28 +49,28 @@ configure_and_build_tests() { local android_args="" if [[ "$ANDROID_MODE" == "true" ]]; then - cmake backends/vulkan/test/custom_ops/ \ + cmake backends/vulkan/test/ops/benchmark/ \ -DCMAKE_INSTALL_PREFIX=$CMAKE_OUT_DIR \ -DCMAKE_BUILD_TYPE=Debug \ -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake \ -DANDROID_ABI=arm64-v8a \ -DANDROID_PLATFORM=android-28 \ -DGLSLC_PATH=$(which glslc) \ - -B$CMAKE_OUT_DIR/backends/vulkan/test/custom_ops + -B$CMAKE_OUT_DIR/backends/vulkan/test/ops/benchmark else - cmake backends/vulkan/test/custom_ops/ \ + cmake backends/vulkan/test/ops/benchmark/ \ -DCMAKE_INSTALL_PREFIX=$CMAKE_OUT_DIR \ -DCMAKE_BUILD_TYPE=Debug \ -DGLSLC_PATH=$(which glslc) \ - -B$CMAKE_OUT_DIR/backends/vulkan/test/custom_ops + -B$CMAKE_OUT_DIR/backends/vulkan/test/ops/benchmark fi - cmake --build $CMAKE_OUT_DIR/backends/vulkan/test/custom_ops -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 )) --target all + cmake --build $CMAKE_OUT_DIR/backends/vulkan/test/ops/benchmark -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 )) --target all } build_tests() { - cmake --build $CMAKE_OUT_DIR/backends/vulkan/test/custom_ops -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 )) --target all + cmake --build $CMAKE_OUT_DIR/backends/vulkan/test/ops/benchmark -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 )) --target all } # Function to rebuild both main and tests @@ -90,7 +90,7 @@ clean_and_rebuild() { execute_binary() { local binary_name="$1" if [[ -n "$binary_name" ]]; then - local binary_path="$CMAKE_OUT_DIR/backends/vulkan/test/custom_ops/$binary_name" + local binary_path="$CMAKE_OUT_DIR/backends/vulkan/test/ops/benchmark/$binary_name" echo "Executing binary: $binary_path" if [[ "$ANDROID_MODE" == "true" ]]; then diff --git a/backends/vulkan/test/custom_ops/test_add.cpp b/backends/vulkan/test/ops/benchmark/cases/add/test_add.cpp similarity index 98% rename from backends/vulkan/test/custom_ops/test_add.cpp rename to backends/vulkan/test/ops/benchmark/cases/add/test_add.cpp index bd7c644c2fd..fed59acedf0 100644 --- a/backends/vulkan/test/custom_ops/test_add.cpp +++ b/backends/vulkan/test/ops/benchmark/cases/add/test_add.cpp @@ -6,9 +6,9 @@ #include #include +#include #include #include -#include "utils.h" using namespace executorch::vulkan::prototyping; diff --git a/backends/vulkan/test/custom_ops/test_choose_qparams_per_row.cpp b/backends/vulkan/test/ops/benchmark/cases/choose_qparams_per_row/test_choose_qparams_per_row.cpp similarity index 99% rename from backends/vulkan/test/custom_ops/test_choose_qparams_per_row.cpp rename to backends/vulkan/test/ops/benchmark/cases/choose_qparams_per_row/test_choose_qparams_per_row.cpp index 69f7812e4ab..48139371d89 100644 --- a/backends/vulkan/test/custom_ops/test_choose_qparams_per_row.cpp +++ b/backends/vulkan/test/ops/benchmark/cases/choose_qparams_per_row/test_choose_qparams_per_row.cpp @@ -6,11 +6,11 @@ #include #include +#include #include #include #include #include -#include "utils.h" #include diff --git a/backends/vulkan/test/custom_ops/test_conv1d_dw.cpp b/backends/vulkan/test/ops/benchmark/cases/conv1d/test_conv1d_dw.cpp similarity index 99% rename from backends/vulkan/test/custom_ops/test_conv1d_dw.cpp rename to backends/vulkan/test/ops/benchmark/cases/conv1d/test_conv1d_dw.cpp index 92539037f75..8e1234611c9 100644 --- a/backends/vulkan/test/custom_ops/test_conv1d_dw.cpp +++ b/backends/vulkan/test/ops/benchmark/cases/conv1d/test_conv1d_dw.cpp @@ -10,7 +10,7 @@ #include #include -#include "utils.h" +#include using namespace executorch::vulkan::prototyping; using namespace vkcompute; diff --git a/backends/vulkan/test/custom_ops/test_conv1d_pw.cpp b/backends/vulkan/test/ops/benchmark/cases/conv1d/test_conv1d_pw.cpp similarity index 98% rename from backends/vulkan/test/custom_ops/test_conv1d_pw.cpp rename to backends/vulkan/test/ops/benchmark/cases/conv1d/test_conv1d_pw.cpp index dc40d1ee795..aab1862c75d 100644 --- a/backends/vulkan/test/custom_ops/test_conv1d_pw.cpp +++ b/backends/vulkan/test/ops/benchmark/cases/conv1d/test_conv1d_pw.cpp @@ -10,7 +10,7 @@ #include #include -#include "utils.h" +#include using namespace executorch::vulkan::prototyping; using namespace vkcompute; diff --git a/backends/vulkan/test/custom_ops/test_conv2d.cpp b/backends/vulkan/test/ops/benchmark/cases/conv2d/test_conv2d.cpp similarity index 99% rename from backends/vulkan/test/custom_ops/test_conv2d.cpp rename to backends/vulkan/test/ops/benchmark/cases/conv2d/test_conv2d.cpp index 2f21f0532fe..83a604345b8 100644 --- a/backends/vulkan/test/custom_ops/test_conv2d.cpp +++ b/backends/vulkan/test/ops/benchmark/cases/conv2d/test_conv2d.cpp @@ -10,8 +10,8 @@ #include #include -#include "conv2d_utils.h" -#include "utils.h" +#include +#include using namespace executorch::vulkan::prototyping; using namespace vkcompute; diff --git a/backends/vulkan/test/custom_ops/test_conv2d_dw.cpp b/backends/vulkan/test/ops/benchmark/cases/conv2d/test_conv2d_dw.cpp similarity index 98% rename from backends/vulkan/test/custom_ops/test_conv2d_dw.cpp rename to backends/vulkan/test/ops/benchmark/cases/conv2d/test_conv2d_dw.cpp index e0e8a55b815..4ba7a3d6286 100644 --- a/backends/vulkan/test/custom_ops/test_conv2d_dw.cpp +++ b/backends/vulkan/test/ops/benchmark/cases/conv2d/test_conv2d_dw.cpp @@ -10,8 +10,8 @@ #include #include -#include "conv2d_utils.h" -#include "utils.h" +#include +#include using namespace executorch::vulkan::prototyping; using namespace vkcompute; diff --git a/backends/vulkan/test/custom_ops/test_conv2d_pw.cpp b/backends/vulkan/test/ops/benchmark/cases/conv2d/test_conv2d_pw.cpp similarity index 99% rename from backends/vulkan/test/custom_ops/test_conv2d_pw.cpp rename to backends/vulkan/test/ops/benchmark/cases/conv2d/test_conv2d_pw.cpp index 30cdb352248..da38a744417 100644 --- a/backends/vulkan/test/custom_ops/test_conv2d_pw.cpp +++ b/backends/vulkan/test/ops/benchmark/cases/conv2d/test_conv2d_pw.cpp @@ -10,7 +10,7 @@ #include #include -#include "utils.h" +#include using namespace executorch::vulkan::prototyping; using namespace vkcompute; diff --git a/backends/vulkan/test/custom_ops/test_embedding_q4gsw.cpp b/backends/vulkan/test/ops/benchmark/cases/embedding_q4gsw/test_embedding_q4gsw.cpp similarity index 99% rename from backends/vulkan/test/custom_ops/test_embedding_q4gsw.cpp rename to backends/vulkan/test/ops/benchmark/cases/embedding_q4gsw/test_embedding_q4gsw.cpp index 7f38b4de77b..20fcce3c5ca 100644 --- a/backends/vulkan/test/custom_ops/test_embedding_q4gsw.cpp +++ b/backends/vulkan/test/ops/benchmark/cases/embedding_q4gsw/test_embedding_q4gsw.cpp @@ -10,7 +10,7 @@ #include #include -#include "utils.h" +#include using namespace executorch::vulkan::prototyping; diff --git a/backends/vulkan/test/custom_ops/test_mm.cpp b/backends/vulkan/test/ops/benchmark/cases/mm/test_mm.cpp similarity index 99% rename from backends/vulkan/test/custom_ops/test_mm.cpp rename to backends/vulkan/test/ops/benchmark/cases/mm/test_mm.cpp index 4a8d9de49b6..608f0fa369a 100644 --- a/backends/vulkan/test/custom_ops/test_mm.cpp +++ b/backends/vulkan/test/ops/benchmark/cases/mm/test_mm.cpp @@ -10,7 +10,7 @@ #include #include -#include "utils.h" +#include using namespace executorch::vulkan::prototyping; using namespace vkcompute; diff --git a/backends/vulkan/test/custom_ops/test_coopmat_linear_bench.cpp b/backends/vulkan/test/ops/benchmark/cases/q4gsw_linear/test_coopmat_linear_bench.cpp similarity index 99% rename from backends/vulkan/test/custom_ops/test_coopmat_linear_bench.cpp rename to backends/vulkan/test/ops/benchmark/cases/q4gsw_linear/test_coopmat_linear_bench.cpp index d57d41266a5..8cd9342f15e 100644 --- a/backends/vulkan/test/custom_ops/test_coopmat_linear_bench.cpp +++ b/backends/vulkan/test/ops/benchmark/cases/q4gsw_linear/test_coopmat_linear_bench.cpp @@ -18,11 +18,11 @@ #include #include #include +#include #include #include #include #include -#include "utils.h" using namespace executorch::vulkan::prototyping; using namespace vkcompute; diff --git a/backends/vulkan/test/custom_ops/test_fpa_q4gsw_linear.cpp b/backends/vulkan/test/ops/benchmark/cases/q4gsw_linear/test_fpa_q4gsw_linear.cpp similarity index 99% rename from backends/vulkan/test/custom_ops/test_fpa_q4gsw_linear.cpp rename to backends/vulkan/test/ops/benchmark/cases/q4gsw_linear/test_fpa_q4gsw_linear.cpp index 212ba03ddaa..ef0249d4267 100644 --- a/backends/vulkan/test/custom_ops/test_fpa_q4gsw_linear.cpp +++ b/backends/vulkan/test/ops/benchmark/cases/q4gsw_linear/test_fpa_q4gsw_linear.cpp @@ -44,9 +44,9 @@ #include #include +#include #include #include -#include "utils.h" using namespace executorch::vulkan::prototyping; diff --git a/backends/vulkan/test/custom_ops/test_q4gsw_linear.cpp b/backends/vulkan/test/ops/benchmark/cases/q4gsw_linear/test_q4gsw_linear.cpp similarity index 99% rename from backends/vulkan/test/custom_ops/test_q4gsw_linear.cpp rename to backends/vulkan/test/ops/benchmark/cases/q4gsw_linear/test_q4gsw_linear.cpp index db188d21c30..cb06f9692e0 100644 --- a/backends/vulkan/test/custom_ops/test_q4gsw_linear.cpp +++ b/backends/vulkan/test/ops/benchmark/cases/q4gsw_linear/test_q4gsw_linear.cpp @@ -6,11 +6,11 @@ #include #include +#include #include #include #include #include -#include "utils.h" #include diff --git a/backends/vulkan/test/custom_ops/test_q8csw_conv2d.cpp b/backends/vulkan/test/ops/benchmark/cases/q8csw_conv2d/test_q8csw_conv2d.cpp similarity index 99% rename from backends/vulkan/test/custom_ops/test_q8csw_conv2d.cpp rename to backends/vulkan/test/ops/benchmark/cases/q8csw_conv2d/test_q8csw_conv2d.cpp index be7a4cab2b8..558f9228355 100644 --- a/backends/vulkan/test/custom_ops/test_q8csw_conv2d.cpp +++ b/backends/vulkan/test/ops/benchmark/cases/q8csw_conv2d/test_q8csw_conv2d.cpp @@ -6,10 +6,10 @@ #include #include +#include +#include #include #include -#include "conv2d_utils.h" -#include "utils.h" #include diff --git a/backends/vulkan/test/custom_ops/test_q8csw_linear.cpp b/backends/vulkan/test/ops/benchmark/cases/q8csw_linear/test_q8csw_linear.cpp similarity index 99% rename from backends/vulkan/test/custom_ops/test_q8csw_linear.cpp rename to backends/vulkan/test/ops/benchmark/cases/q8csw_linear/test_q8csw_linear.cpp index 5e9154b5648..33c6947e167 100644 --- a/backends/vulkan/test/custom_ops/test_q8csw_linear.cpp +++ b/backends/vulkan/test/ops/benchmark/cases/q8csw_linear/test_q8csw_linear.cpp @@ -6,9 +6,9 @@ #include #include +#include #include #include -#include "utils.h" #include diff --git a/backends/vulkan/test/custom_ops/test_q8ta_binary.cpp b/backends/vulkan/test/ops/benchmark/cases/q8ta/binary/test_q8ta_binary.cpp similarity index 99% rename from backends/vulkan/test/custom_ops/test_q8ta_binary.cpp rename to backends/vulkan/test/ops/benchmark/cases/q8ta/binary/test_q8ta_binary.cpp index 4a2654b00db..47b65fa2f1e 100644 --- a/backends/vulkan/test/custom_ops/test_q8ta_binary.cpp +++ b/backends/vulkan/test/ops/benchmark/cases/q8ta/binary/test_q8ta_binary.cpp @@ -6,12 +6,12 @@ #include #include +#include #include #include #include #include #include -#include "utils.h" using namespace executorch::vulkan::prototyping; diff --git a/backends/vulkan/test/custom_ops/test_q8ta_clone.cpp b/backends/vulkan/test/ops/benchmark/cases/q8ta/clone/test_q8ta_clone.cpp similarity index 99% rename from backends/vulkan/test/custom_ops/test_q8ta_clone.cpp rename to backends/vulkan/test/ops/benchmark/cases/q8ta/clone/test_q8ta_clone.cpp index 4936c3702cf..2f6ea11b6c8 100644 --- a/backends/vulkan/test/custom_ops/test_q8ta_clone.cpp +++ b/backends/vulkan/test/ops/benchmark/cases/q8ta/clone/test_q8ta_clone.cpp @@ -6,11 +6,11 @@ #include #include +#include #include #include #include #include -#include "utils.h" #include diff --git a/backends/vulkan/test/custom_ops/test_q8ta_conv2d.cpp b/backends/vulkan/test/ops/benchmark/cases/q8ta/conv2d/test_q8ta_conv2d.cpp similarity index 99% rename from backends/vulkan/test/custom_ops/test_q8ta_conv2d.cpp rename to backends/vulkan/test/ops/benchmark/cases/q8ta/conv2d/test_q8ta_conv2d.cpp index 17045225a2c..683dbf8506c 100644 --- a/backends/vulkan/test/custom_ops/test_q8ta_conv2d.cpp +++ b/backends/vulkan/test/ops/benchmark/cases/q8ta/conv2d/test_q8ta_conv2d.cpp @@ -18,8 +18,8 @@ #include -#include "conv2d_utils.h" -#include "utils.h" +#include +#include // #define DEBUG_MODE diff --git a/backends/vulkan/test/custom_ops/test_q8ta_conv2d_dw.cpp b/backends/vulkan/test/ops/benchmark/cases/q8ta/conv2d/test_q8ta_conv2d_dw.cpp similarity index 99% rename from backends/vulkan/test/custom_ops/test_q8ta_conv2d_dw.cpp rename to backends/vulkan/test/ops/benchmark/cases/q8ta/conv2d/test_q8ta_conv2d_dw.cpp index c573b8b5d39..6ff15f68f81 100644 --- a/backends/vulkan/test/custom_ops/test_q8ta_conv2d_dw.cpp +++ b/backends/vulkan/test/ops/benchmark/cases/q8ta/conv2d/test_q8ta_conv2d_dw.cpp @@ -12,8 +12,8 @@ #include -#include "conv2d_utils.h" -#include "utils.h" +#include +#include // #define DEBUG_MODE diff --git a/backends/vulkan/test/custom_ops/test_q8ta_conv2d_pw.cpp b/backends/vulkan/test/ops/benchmark/cases/q8ta/conv2d/test_q8ta_conv2d_pw.cpp similarity index 99% rename from backends/vulkan/test/custom_ops/test_q8ta_conv2d_pw.cpp rename to backends/vulkan/test/ops/benchmark/cases/q8ta/conv2d/test_q8ta_conv2d_pw.cpp index 5a773f43319..8062b25d31d 100644 --- a/backends/vulkan/test/custom_ops/test_q8ta_conv2d_pw.cpp +++ b/backends/vulkan/test/ops/benchmark/cases/q8ta/conv2d/test_q8ta_conv2d_pw.cpp @@ -14,8 +14,8 @@ #include -#include "conv2d_utils.h" -#include "utils.h" +#include +#include // #define DEBUG_MODE diff --git a/backends/vulkan/test/custom_ops/test_q8ta_conv2d_transposed.cpp b/backends/vulkan/test/ops/benchmark/cases/q8ta/conv2d_transposed/test_q8ta_conv2d_transposed.cpp similarity index 99% rename from backends/vulkan/test/custom_ops/test_q8ta_conv2d_transposed.cpp rename to backends/vulkan/test/ops/benchmark/cases/q8ta/conv2d_transposed/test_q8ta_conv2d_transposed.cpp index b82cb19e614..4ead8123f45 100644 --- a/backends/vulkan/test/custom_ops/test_q8ta_conv2d_transposed.cpp +++ b/backends/vulkan/test/ops/benchmark/cases/q8ta/conv2d_transposed/test_q8ta_conv2d_transposed.cpp @@ -12,8 +12,8 @@ #include -#include "conv2d_utils.h" -#include "utils.h" +#include +#include // #define DEBUG_MODE diff --git a/backends/vulkan/test/custom_ops/test_q8ta_linear.cpp b/backends/vulkan/test/ops/benchmark/cases/q8ta/linear/test_q8ta_linear.cpp similarity index 99% rename from backends/vulkan/test/custom_ops/test_q8ta_linear.cpp rename to backends/vulkan/test/ops/benchmark/cases/q8ta/linear/test_q8ta_linear.cpp index 2b65b53c896..a935508bc19 100644 --- a/backends/vulkan/test/custom_ops/test_q8ta_linear.cpp +++ b/backends/vulkan/test/ops/benchmark/cases/q8ta/linear/test_q8ta_linear.cpp @@ -12,7 +12,7 @@ #include -#include "utils.h" +#include using namespace executorch::vulkan::prototyping; diff --git a/backends/vulkan/test/custom_ops/test_q8ta_pixel_shuffle.cpp b/backends/vulkan/test/ops/benchmark/cases/q8ta/pixel_shuffle/test_q8ta_pixel_shuffle.cpp similarity index 99% rename from backends/vulkan/test/custom_ops/test_q8ta_pixel_shuffle.cpp rename to backends/vulkan/test/ops/benchmark/cases/q8ta/pixel_shuffle/test_q8ta_pixel_shuffle.cpp index b932d1e92d3..948ee347520 100644 --- a/backends/vulkan/test/custom_ops/test_q8ta_pixel_shuffle.cpp +++ b/backends/vulkan/test/ops/benchmark/cases/q8ta/pixel_shuffle/test_q8ta_pixel_shuffle.cpp @@ -6,12 +6,12 @@ #include #include +#include #include #include #include #include #include -#include "utils.h" #include diff --git a/backends/vulkan/test/custom_ops/test_q8ta_qdq.cpp b/backends/vulkan/test/ops/benchmark/cases/q8ta/qdq/test_q8ta_qdq.cpp similarity index 99% rename from backends/vulkan/test/custom_ops/test_q8ta_qdq.cpp rename to backends/vulkan/test/ops/benchmark/cases/q8ta/qdq/test_q8ta_qdq.cpp index af48018d31a..d11988dd192 100644 --- a/backends/vulkan/test/custom_ops/test_q8ta_qdq.cpp +++ b/backends/vulkan/test/ops/benchmark/cases/q8ta/qdq/test_q8ta_qdq.cpp @@ -6,11 +6,11 @@ #include #include +#include #include #include #include #include -#include "utils.h" #include diff --git a/backends/vulkan/test/custom_ops/test_q8ta_unary.cpp b/backends/vulkan/test/ops/benchmark/cases/q8ta/unary/test_q8ta_unary.cpp similarity index 99% rename from backends/vulkan/test/custom_ops/test_q8ta_unary.cpp rename to backends/vulkan/test/ops/benchmark/cases/q8ta/unary/test_q8ta_unary.cpp index a4072c01f02..43eff9bc8ee 100644 --- a/backends/vulkan/test/custom_ops/test_q8ta_unary.cpp +++ b/backends/vulkan/test/ops/benchmark/cases/q8ta/unary/test_q8ta_unary.cpp @@ -6,11 +6,11 @@ #include #include +#include #include #include #include #include -#include "utils.h" #include diff --git a/backends/vulkan/test/custom_ops/test_sdpa.cpp b/backends/vulkan/test/ops/benchmark/cases/sdpa/test_sdpa.cpp similarity index 99% rename from backends/vulkan/test/custom_ops/test_sdpa.cpp rename to backends/vulkan/test/ops/benchmark/cases/sdpa/test_sdpa.cpp index 55d2173e3cd..d71860c5178 100644 --- a/backends/vulkan/test/custom_ops/test_sdpa.cpp +++ b/backends/vulkan/test/ops/benchmark/cases/sdpa/test_sdpa.cpp @@ -13,7 +13,7 @@ #include #include -#include "utils.h" +#include using namespace executorch::vulkan::prototyping; using namespace vkcompute; diff --git a/backends/vulkan/test/custom_ops/cm_utils.cpp b/backends/vulkan/test/ops/benchmark/framework/cm_utils.cpp similarity index 100% rename from backends/vulkan/test/custom_ops/cm_utils.cpp rename to backends/vulkan/test/ops/benchmark/framework/cm_utils.cpp diff --git a/backends/vulkan/test/custom_ops/cm_utils.h b/backends/vulkan/test/ops/benchmark/framework/cm_utils.h similarity index 100% rename from backends/vulkan/test/custom_ops/cm_utils.h rename to backends/vulkan/test/ops/benchmark/framework/cm_utils.h diff --git a/backends/vulkan/test/ops/benchmark/framework/config.cpp b/backends/vulkan/test/ops/benchmark/framework/config.cpp new file mode 100644 index 00000000000..ada88cb840b --- /dev/null +++ b/backends/vulkan/test/ops/benchmark/framework/config.cpp @@ -0,0 +1,55 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. +// All rights reserved. +// +// This source code is licensed under the BSD-style license found in the +// LICENSE file in the root directory of this source tree. + +#include "config.h" + +namespace executorch { +namespace vulkan { +namespace prototyping { + +// Output and latency printing utilities +namespace { +static int print_output_enabled = 0; +static int print_latencies_enabled = 0; +static int gpu_timestamps_enabled = 0; +static int debugging_enabled = 0; +} // namespace + +bool print_output() { + return print_output_enabled > 0; +} + +void set_print_output(bool print_output) { + print_output_enabled = print_output ? 1 : 0; +} + +bool print_latencies() { + return print_latencies_enabled > 0; +} + +void set_print_latencies(bool print_latencies) { + print_latencies_enabled = print_latencies ? 1 : 0; +} + +bool use_gpu_timestamps() { + return gpu_timestamps_enabled > 0; +} + +void set_use_gpu_timestamps(bool use_timestamps) { + gpu_timestamps_enabled = use_timestamps ? 1 : 0; +} + +bool debugging() { + return debugging_enabled > 0; +} + +void set_debugging(bool enable_debugging) { + debugging_enabled = enable_debugging ? 1 : 0; +} + +} // namespace prototyping +} // namespace vulkan +} // namespace executorch diff --git a/backends/vulkan/test/ops/benchmark/framework/config.h b/backends/vulkan/test/ops/benchmark/framework/config.h new file mode 100644 index 00000000000..6b2a3475234 --- /dev/null +++ b/backends/vulkan/test/ops/benchmark/framework/config.h @@ -0,0 +1,31 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. +// All rights reserved. +// +// This source code is licensed under the BSD-style license found in the +// LICENSE file in the root directory of this source tree. + +#pragma once + +namespace executorch { +namespace vulkan { +namespace prototyping { + +// +// Global configuration options +// + +bool print_output(); +void set_print_output(bool print_output); + +bool print_latencies(); +void set_print_latencies(bool print_latencies); + +bool use_gpu_timestamps(); +void set_use_gpu_timestamps(bool use_timestamps); + +bool debugging(); +void set_debugging(bool enable_debugging); + +} // namespace prototyping +} // namespace vulkan +} // namespace executorch diff --git a/backends/vulkan/test/custom_ops/conv2d_utils.cpp b/backends/vulkan/test/ops/benchmark/framework/conv2d_utils.cpp similarity index 100% rename from backends/vulkan/test/custom_ops/conv2d_utils.cpp rename to backends/vulkan/test/ops/benchmark/framework/conv2d_utils.cpp diff --git a/backends/vulkan/test/custom_ops/conv2d_utils.h b/backends/vulkan/test/ops/benchmark/framework/conv2d_utils.h similarity index 100% rename from backends/vulkan/test/custom_ops/conv2d_utils.h rename to backends/vulkan/test/ops/benchmark/framework/conv2d_utils.h diff --git a/backends/vulkan/test/ops/benchmark/framework/labels.h b/backends/vulkan/test/ops/benchmark/framework/labels.h new file mode 100644 index 00000000000..bcae57ccef8 --- /dev/null +++ b/backends/vulkan/test/ops/benchmark/framework/labels.h @@ -0,0 +1,136 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. +// All rights reserved. +// +// This source code is licensed under the BSD-style license found in the +// LICENSE file in the root directory of this source tree. + +#pragma once + +#include + +#include +#include +#include + +namespace executorch { +namespace vulkan { +namespace prototyping { + +// NOLINTNEXTLINE(google-build-using-namespace) +using namespace vkcompute; + +// +// String utilities +// + +// Helper function to get abbreviated layout names for test case naming +inline std::string layout_abbrev(utils::GPUMemoryLayout layout) { + // NOLINTNEXTLINE(clang-diagnostic-switch-enum) + switch (layout) { + case utils::kWidthPacked: + return "WP"; + case utils::kChannelsPacked: + return "CP"; + case utils::kPackedInt8_4W: + return "4W"; + case utils::kPackedInt8_4C: + return "4C"; + case utils::kPackedInt8_4W4C: + return "4W4C"; + case utils::kPackedInt8_4H4W: + return "4H4W"; + case utils::kPackedInt8_4C1W: + return "4C1W"; + default: + return "UNK"; + } +} + +// Helper function to get abbreviated storage type names for test case naming +inline std::string storage_type_abbrev(utils::StorageType storage_type) { + // NOLINTNEXTLINE(clang-diagnostic-switch-enum) + switch (storage_type) { + case utils::kTexture3D: + return "Tex"; + case utils::kBuffer: + return "Buf"; + default: + return "UNK"; + } +} + +// Helper function to get combined storage type and layout representation +// Example: (kBuffer, kPackedInt8_4W4C) -> "Buf_4W4C" +inline std::string repr_str( + utils::StorageType storage_type, + utils::GPUMemoryLayout layout) { + return storage_type_abbrev(storage_type) + "(" + layout_abbrev(layout) + ")"; +} + +// Helper function to generate comma-separated shape string for test case naming +// Example: {1, 128, 56, 56} -> "1,128,56,56" +inline std::string shape_string(const std::vector& shape) { + std::string result; + for (size_t i = 0; i < shape.size(); ++i) { + if (i > 0) { + result += ","; + } + result += std::to_string(shape[i]); + } + return result; +} + +// Helper function to generate a bracketed shape string for test case naming +// Example: {1, 128, 56, 56} -> "[1,128,56,56]" +inline std::string shape_bracket(const std::vector& shape) { + return "[" + shape_string(shape) + "]"; +} + +// Short dtype symbol used in standardized test case labels +// Example: kFloat -> "f32", kHalf -> "f16", kChar -> "i8", kInt -> "i32" +inline std::string dtype_short(vkapi::ScalarType dtype) { + // NOLINTNEXTLINE(clang-diagnostic-switch-enum) + switch (dtype) { + case vkapi::kFloat: + return "f32"; + case vkapi::kHalf: + return "f16"; + case vkapi::kChar: + return "i8"; + case vkapi::kByte: + return "u8"; + case vkapi::kInt: + return "i32"; + case vkapi::kBool: + return "b"; + default: + return "?"; + } +} + +// Build a standardized test label of the form: +// " [ ]" +// where is "->" when the two differ, or just +// "" when they match. Sections are separated by two spaces. If +// suffix is non-empty it is appended after a single space (allowing callers +// to pass e.g. "[general]" or "[gemv] +bias"). +inline std::string make_test_label( + const std::string& prefix, + const std::string& in_dtype, + const std::string& out_dtype, + const std::string& shape_str, + const std::string& storage_str, + const std::string& suffix = "") { + const std::string dtype_str = + (in_dtype == out_dtype) ? in_dtype : in_dtype + "->" + out_dtype; + std::string label = + prefix + " " + dtype_str + " " + shape_str + " " + storage_str; + if (!suffix.empty()) { + label += " " + suffix; + } + return label; +} + +} // namespace prototyping +} // namespace vulkan +} // namespace executorch diff --git a/backends/vulkan/test/ops/benchmark/framework/results.cpp b/backends/vulkan/test/ops/benchmark/framework/results.cpp new file mode 100644 index 00000000000..3a71c50237b --- /dev/null +++ b/backends/vulkan/test/ops/benchmark/framework/results.cpp @@ -0,0 +1,468 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. +// All rights reserved. +// +// This source code is licensed under the BSD-style license found in the +// LICENSE file in the root directory of this source tree. + +#include "results.h" + +#include +#include +#include +#include +#include +#include +#include + +namespace executorch { +namespace vulkan { +namespace prototyping { + +// BenchmarkResult implementation +void BenchmarkResult::add_iter_timing(float time_us) { + iter_timings.push_back(time_us); +} + +void BenchmarkResult::add_shader_timing( + const std::string& shader_name, + float time_us, + const uint32_t gwg[3], + const uint32_t lwg[3]) { + // Find existing shader timing or create new one + for (auto& st : shader_timings_) { + if (st.shader_name == shader_name) { + st.iter_timings_us.push_back(time_us); + // Work group sizes should be consistent across iterations + return; + } + } + // Not found, create new entry + ShaderTiming new_timing; + new_timing.shader_name = shader_name; + new_timing.iter_timings_us.push_back(time_us); + new_timing.gwg[0] = gwg[0]; + new_timing.gwg[1] = gwg[1]; + new_timing.gwg[2] = gwg[2]; + new_timing.lwg[0] = lwg[0]; + new_timing.lwg[1] = lwg[1]; + new_timing.lwg[2] = lwg[2]; + shader_timings_.push_back(std::move(new_timing)); +} + +float BenchmarkResult::get_avg_time_us() const { + if (iter_timings.empty()) { + return 0.0f; + } + + float sum = 0.0f; + for (float timing : iter_timings) { + sum += timing; + } + return sum / iter_timings.size(); +} + +float BenchmarkResult::get_min_time_us() const { + if (iter_timings.empty()) { + return 0.0f; + } + + return *std::min_element(iter_timings.begin(), iter_timings.end()); +} + +float BenchmarkResult::get_max_time_us() const { + if (iter_timings.empty()) { + return 0.0f; + } + + return *std::max_element(iter_timings.begin(), iter_timings.end()); +} + +float BenchmarkResult::get_std_dev_us() const { + if (iter_timings.size() <= 1) { + return 0.0f; + } + + float mean = get_avg_time_us(); + float sum_sq_diff = 0.0f; + + for (float timing : iter_timings) { + float diff = timing - mean; + sum_sq_diff += diff * diff; + } + + return std::sqrt(sum_sq_diff / (iter_timings.size() - 1)); +} + +void BenchmarkResult::print_summary( + int case_number, + const std::string& size_info, + float total_gflops) const { + (void)case_number; + static constexpr int OPERATOR_NAME_WIDTH = 50; + static constexpr int GLOBAL_WG_WIDTH = 16; + static constexpr int LOCAL_WG_WIDTH = 12; + static constexpr int KERNEL_NAME_WIDTH = 80; + static constexpr int SIZE_INFO_WIDTH = 20; + static constexpr int TIMING_WIDTH = 16; + static constexpr int GFLOPS_WIDTH = 14; + static constexpr int CORRECTNESS_WIDTH = 8; + + // Helper to truncate shader names longer than 46 chars to 44 chars + ".." + auto truncate_shader_name = [](const std::string& name) -> std::string { + if (name.length() > 46) { + return name.substr(0, 44) + ".."; + } + return name; + }; + + // Helper to format work group size as (x,y,z) + auto format_wg_size = [](const uint32_t wg[3]) -> std::string { + return "(" + std::to_string(wg[0]) + "," + std::to_string(wg[1]) + "," + + std::to_string(wg[2]) + ")"; + }; + + std::string correctness_str; + switch (correctness_status_) { + case CorrectnessStatus::SKIPPED: + correctness_str = "SKIPPED"; + break; + case CorrectnessStatus::PASSED: + correctness_str = "PASSED"; + break; + case CorrectnessStatus::FAILED: + correctness_str = "FAILED"; + break; + } + + // If we have per-shader timing data, print one line per shader plus overall + if (!shader_timings_.empty()) { + // If only one shader, print a single combined row + if (shader_timings_.size() == 1) { + const auto& st = shader_timings_[0]; + std::cout << std::left << std::setw(OPERATOR_NAME_WIDTH) + << truncate_shader_name(st.shader_name) << " " << std::left + << std::setw(GLOBAL_WG_WIDTH) << format_wg_size(st.gwg) + << std::left << std::setw(LOCAL_WG_WIDTH) + << format_wg_size(st.lwg) << std::left + << std::setw(KERNEL_NAME_WIDTH) << get_kernel_name() + << std::right << " " << std::setw(SIZE_INFO_WIDTH) << size_info + << std::setw(TIMING_WIDTH) << std::fixed << std::setprecision(3) + << get_avg_time_us() << " μs " << std::setw(GFLOPS_WIDTH) + << std::fixed << std::setprecision(3) << total_gflops + << " GFLOP/s " << std::setw(CORRECTNESS_WIDTH) + << correctness_str << std::endl; + } else { + // Multiple shaders: print individual shader lines (without GFLOP/s) + for (size_t i = 0; i < shader_timings_.size(); ++i) { + const auto& st = shader_timings_[i]; + float shader_avg_time = st.get_avg_time_us(); + + // Shader lines don't show test case info + std::cout << std::left << std::setw(OPERATOR_NAME_WIDTH) + << truncate_shader_name(st.shader_name) << " " << std::left + << std::setw(GLOBAL_WG_WIDTH) << format_wg_size(st.gwg) + << std::left << std::setw(LOCAL_WG_WIDTH) + << format_wg_size(st.lwg) << std::left + << std::setw(KERNEL_NAME_WIDTH) << "" << std::right << " " + << std::setw(SIZE_INFO_WIDTH) << "" << std::setw(TIMING_WIDTH) + << std::fixed << std::setprecision(3) << shader_avg_time + << " μs " << std::setw(GFLOPS_WIDTH) << "" << " " + << std::setw(CORRECTNESS_WIDTH) << "" << std::endl; + } + + // Print overall row with operator name, test case info, total time, and + // GFLOP/s + std::cout << std::left << std::setw(OPERATOR_NAME_WIDTH) + << get_operator_name() << " " << std::left + << std::setw(GLOBAL_WG_WIDTH) << "" << std::left + << std::setw(LOCAL_WG_WIDTH) << "" << std::left + << std::setw(KERNEL_NAME_WIDTH) << get_kernel_name() + << std::right << " " << std::setw(SIZE_INFO_WIDTH) << size_info + << std::setw(TIMING_WIDTH) << std::fixed << std::setprecision(3) + << get_avg_time_us() << " μs " << std::setw(GFLOPS_WIDTH) + << std::fixed << std::setprecision(3) << total_gflops + << " GFLOP/s " << std::setw(CORRECTNESS_WIDTH) + << correctness_str << std::endl; + } + + // Print separator line between test cases + } else { + // No per-shader timing data, use the original format + std::cout << std::left << std::setw(OPERATOR_NAME_WIDTH) + << get_operator_name() << " " << std::left + << std::setw(GLOBAL_WG_WIDTH) << "" << std::left + << std::setw(LOCAL_WG_WIDTH) << "" << std::left + << std::setw(KERNEL_NAME_WIDTH) << get_kernel_name() << std::right + << " " << std::setw(SIZE_INFO_WIDTH) << size_info + << std::setw(TIMING_WIDTH) << std::fixed << std::setprecision(3) + << get_avg_time_us() << " μs " << std::setw(GFLOPS_WIDTH) + << std::fixed << std::setprecision(3) << total_gflops + << " GFLOP/s " << std::setw(CORRECTNESS_WIDTH) << correctness_str + << std::endl; + } +} + +// TestResult implementation +void TestResult::add_result(const BenchmarkResult& result) { + results_.push_back(result); +} + +void TestResult::add_result(BenchmarkResult&& result) { + results_.push_back(std::move(result)); +} + +void TestResult::print_summary() const { + static constexpr int CASE_WIDTH = 100; + static constexpr int KERNEL_NAME_WIDTH = 20; + static constexpr int TIMING_WIDTH = 12; + static constexpr int PASS_WIDTH = 8; + + if (results_.empty()) { + std::cout << "No results to display" << std::endl; + return; + } + + std::cout << "\n=== " << operation_name_ + << " Performance Summary ===" << std::endl; + print_separator(); + + std::cout << std::left << std::setw(CASE_WIDTH) << "Case" << std::left + << std::setw(KERNEL_NAME_WIDTH) << "Kernel Name" << std::left + << std::setw(TIMING_WIDTH) << "Avg (μs)" << std::left + << std::setw(TIMING_WIDTH) << "Min (μs)" << std::left + << std::setw(TIMING_WIDTH) << "Max (μs)" << std::left + << std::setw(TIMING_WIDTH) << "Std Dev" << std::left + << std::setw(PASS_WIDTH) << "Pass" << std::endl; + print_separator(); + + for (size_t i = 0; i < results_.size(); ++i) { + const auto& result = results_[i]; + bool vulkan_execute_succeeded = + result.get_num_iterations() > 0 && result.get_avg_time_us() > 0.0f; + std::cout << std::left << std::setw(CASE_WIDTH) << i + 1 << std::left + << std::setw(KERNEL_NAME_WIDTH) + << result.get_kernel_name().substr(0, KERNEL_NAME_WIDTH - 1) + << std::left << std::setw(TIMING_WIDTH) << std::fixed + << std::setprecision(3) << result.get_avg_time_us() << std::left + << std::setw(TIMING_WIDTH) << std::fixed << std::setprecision(3) + << result.get_min_time_us() << std::left + << std::setw(TIMING_WIDTH) << std::fixed << std::setprecision(3) + << result.get_max_time_us() << std::left + << std::setw(TIMING_WIDTH) << std::fixed << std::setprecision(3) + << result.get_std_dev_us() << std::left << std::setw(PASS_WIDTH) + << (vulkan_execute_succeeded ? "✓" : "✗") << std::endl; + } + + print_separator(); + std::cout << "Total cases: " << results_.size() + << ", Passed: " << get_passed_count() + << ", Failed: " << get_failed_count() << std::endl; + std::cout << "Overall GFLOP/s: " << std::fixed << std::setprecision(3) + << gflops_ << std::endl; + std::cout << "Overall correctness: " + << (correctness_passed_ ? "PASSED" : "FAILED") << std::endl; +} + +void TestResult::print_detailed_results() const { + if (results_.empty()) { + std::cout << "No results to display" << std::endl; + return; + } + + std::cout << "\n=== " << operation_name_ + << " Detailed Results ===" << std::endl; + + for (size_t i = 0; i < results_.size(); ++i) { + const auto& result = results_[i]; + bool vulkan_execute_succeeded = + result.get_num_iterations() > 0 && result.get_avg_time_us() > 0.0f; + std::cout << "\nCase " << i + 1 << ": " << result.get_kernel_name() + << std::endl; + std::cout << " Iterations: " << result.get_num_iterations() << std::endl; + std::cout << " Average: " << std::fixed << std::setprecision(3) + << result.get_avg_time_us() << " μs" << std::endl; + std::cout << " Min: " << std::fixed << std::setprecision(3) + << result.get_min_time_us() << " μs" << std::endl; + std::cout << " Max: " << std::fixed << std::setprecision(3) + << result.get_max_time_us() << " μs" << std::endl; + std::cout << " Std Dev: " << std::fixed << std::setprecision(3) + << result.get_std_dev_us() << " μs" << std::endl; + std::cout << " Correctness: " + << (vulkan_execute_succeeded ? "PASSED" : "FAILED") << std::endl; + + if (result.get_num_iterations() > 0) { + std::cout << " Individual timings (μs): "; + const auto& timings = result.get_iter_timings(); + for (size_t j = 0; j < std::min(size_t(10), timings.size()); ++j) { + std::cout << std::fixed << std::setprecision(1) << timings[j]; + if (j < std::min(size_t(10), timings.size()) - 1) { + std::cout << ", "; + } + } + if (timings.size() > 10) { + std::cout << " ... (" << (timings.size() - 10) << " more)"; + } + std::cout << std::endl; + } + } + + std::cout << "\nOverall Results:" << std::endl; + std::cout << " Total GFLOP/s: " << std::fixed << std::setprecision(3) + << gflops_ << std::endl; + std::cout << " Overall correctness: " + << (correctness_passed_ ? "PASSED" : "FAILED") << std::endl; +} + +void TestResult::print_statistics() const { + if (results_.empty()) { + std::cout << "No results to display statistics for" << std::endl; + return; + } + + std::cout << "\n=== " << operation_name_ << " Statistics ===" << std::endl; + std::cout << "Total test cases: " << results_.size() << std::endl; + std::cout << "Passed: " << get_passed_count() << std::endl; + std::cout << "Failed: " << get_failed_count() << std::endl; + std::cout << "Success rate: " << std::fixed << std::setprecision(1) + << (100.0f * get_passed_count() / results_.size()) << "%" + << std::endl; + + if (get_passed_count() > 0) { + std::cout << "Total average time: " << std::fixed << std::setprecision(3) + << get_total_avg_time_us() << " μs" << std::endl; + std::cout << "Total GFLOP/s: " << std::fixed << std::setprecision(3) + << get_total_gflops() << std::endl; + + const auto* fastest = get_fastest_result(); + const auto* slowest = get_slowest_result(); + const auto* highest_gflops = get_highest_gflops_result(); + + if (fastest) { + std::cout << "Fastest case: " << fastest->get_kernel_name() << " (" + << std::fixed << std::setprecision(3) + << fastest->get_avg_time_us() << " μs)" << std::endl; + } + + if (slowest) { + std::cout << "Slowest case: " << slowest->get_kernel_name() << " (" + << std::fixed << std::setprecision(3) + << slowest->get_avg_time_us() << " μs)" << std::endl; + } + + if (highest_gflops) { + std::cout << "Best performing case: " << highest_gflops->get_kernel_name() + << " (" << std::fixed << std::setprecision(3) + << highest_gflops->get_avg_time_us() << " μs)" << std::endl; + } + } +} + +void TestResult::print_brief_summary() const { + print_separator(); + std::cout << "Summary Statistics:" << std::endl; + + if (get_passed_count() > 0) { + std::cout << "Average execution time: " << std::fixed + << std::setprecision(3) << get_total_avg_time_us() << " μs" + << std::endl; + std::cout << "Total throughput: " << std::fixed << std::setprecision(3) + << get_gflops() << " GFLOP/s" << std::endl; + std::cout << "Successful test cases: " << get_passed_count() << "/" + << size() << std::endl; + std::cout << "Overall correctness: " + << (get_correctness_passed() ? "PASSED" : "FAILED") << std::endl; + } else { + std::cout << "No successful test cases to report" << std::endl; + } +} + +float TestResult::get_total_avg_time_us() const { + if (results_.empty()) { + return 0.0f; + } + + float sum = 0.0f; + size_t count = 0; + + for (const auto& result : results_) { + bool vulkan_execute_succeeded = + result.get_num_iterations() > 0 && result.get_avg_time_us() > 0.0f; + if (vulkan_execute_succeeded) { + sum += result.get_avg_time_us(); + count++; + } + } + + return count > 0 ? sum / count : 0.0f; +} + +float TestResult::get_total_gflops() const { + return gflops_; +} + +size_t TestResult::get_passed_count() const { + size_t count = 0; + for (const auto& result : results_) { + bool vulkan_execute_succeeded = + result.get_num_iterations() > 0 && result.get_avg_time_us() > 0.0f; + if (vulkan_execute_succeeded) { + count++; + } + } + return count; +} + +size_t TestResult::get_failed_count() const { + return results_.size() - get_passed_count(); +} + +const BenchmarkResult* TestResult::get_fastest_result() const { + const BenchmarkResult* fastest = nullptr; + + for (const auto& result : results_) { + bool vulkan_execute_succeeded = + result.get_num_iterations() > 0 && result.get_avg_time_us() > 0.0f; + if (vulkan_execute_succeeded) { + if (!fastest || result.get_avg_time_us() < fastest->get_avg_time_us()) { + fastest = &result; + } + } + } + + return fastest; +} + +const BenchmarkResult* TestResult::get_slowest_result() const { + const BenchmarkResult* slowest = nullptr; + + for (const auto& result : results_) { + bool vulkan_execute_succeeded = + result.get_num_iterations() > 0 && result.get_avg_time_us() > 0.0f; + if (vulkan_execute_succeeded) { + if (!slowest || result.get_avg_time_us() > slowest->get_avg_time_us()) { + slowest = &result; + } + } + } + + return slowest; +} + +const BenchmarkResult* TestResult::get_highest_gflops_result() const { + // Since GFLOPS is now a TestResult-level metric rather than per-case, + // this method returns the fastest result as a proxy for highest performance + return get_fastest_result(); +} + +// Utility functions for printing +void print_performance_header() { + std::cout << "\n=== Compute Shader Performance Benchmark ===" << std::endl; +} + +void print_separator() { + std::cout << std::string(70, '-') << std::endl; +} + +} // namespace prototyping +} // namespace vulkan +} // namespace executorch diff --git a/backends/vulkan/test/ops/benchmark/framework/results.h b/backends/vulkan/test/ops/benchmark/framework/results.h new file mode 100644 index 00000000000..e957ddf8bce --- /dev/null +++ b/backends/vulkan/test/ops/benchmark/framework/results.h @@ -0,0 +1,235 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. +// All rights reserved. +// +// This source code is licensed under the BSD-style license found in the +// LICENSE file in the root directory of this source tree. + +#pragma once + +#include +#include +#include + +namespace executorch { +namespace vulkan { +namespace prototyping { + +// +// BenchmarkResult +// + +enum class CorrectnessStatus { + SKIPPED, // No reference function provided + PASSED, // Reference function provided and validation passed + FAILED // Reference function provided but validation failed +}; + +// Per-shader timing data for detailed reporting +struct ShaderTiming { + std::string shader_name; + std::vector iter_timings_us; // Individual iteration timings + uint32_t gwg[3] = {0, 0, 0}; + uint32_t lwg[3] = {0, 0, 0}; + + float get_avg_time_us() const { + if (iter_timings_us.empty()) { + return 0.0f; + } + float sum = 0.0f; + for (float t : iter_timings_us) { + sum += t; + } + return sum / iter_timings_us.size(); + } +}; + +class BenchmarkResult { + public: + BenchmarkResult() : correctness_status_(CorrectnessStatus::SKIPPED) {} + + explicit BenchmarkResult(const std::string& name) + : kernel_name(name), correctness_status_(CorrectnessStatus::SKIPPED) {} + + BenchmarkResult( + const std::string& kernel_name, + const std::string& operator_name) + : kernel_name(kernel_name), + operator_name(operator_name), + correctness_status_(CorrectnessStatus::SKIPPED) {} + + // Add timing for a single iteration + void add_iter_timing(float time_us); + + // Add per-shader timing for a single iteration + void add_shader_timing( + const std::string& shader_name, + float time_us, + const uint32_t gwg[3], + const uint32_t lwg[3]); + + // Get per-shader timing data + const std::vector& get_shader_timings() const { + return shader_timings_; + } + + // Getters + const std::string& get_kernel_name() const { + return kernel_name; + } + const std::string& get_operator_name() const { + return operator_name; + } + float get_avg_time_us() const; + size_t get_num_iterations() const { + return iter_timings.size(); + } + const std::vector& get_iter_timings() const { + return iter_timings; + } + CorrectnessStatus get_correctness_status() const { + return correctness_status_; + } + + // Setters + void set_kernel_name(const std::string& name) { + kernel_name = name; + } + void set_operator_name(const std::string& name) { + operator_name = name; + } + void set_correctness_status(CorrectnessStatus status) { + correctness_status_ = status; + } + + // Statistics + float get_min_time_us() const; + float get_max_time_us() const; + float get_std_dev_us() const; + + // Clear all timings + void clear_timings() { + iter_timings.clear(); + } + + // Print progress for this benchmark result + void print_summary( + int case_number, + const std::string& size_info, + float total_gflops) const; + + private: + std::string kernel_name; + std::string operator_name; + std::vector + iter_timings; // Individual iteration timings in microseconds + std::vector shader_timings_; // Per-shader timing data + CorrectnessStatus correctness_status_; +}; + +// Test result collection and processing +class TestResult { + public: + TestResult() : gflops_(0.0f), correctness_passed_(true) {} + explicit TestResult(const std::string& operation_name) + : operation_name_(operation_name), + gflops_(0.0f), + correctness_passed_(true) {} + + // Add a benchmark result + void add_result(const BenchmarkResult& result); + void add_result(BenchmarkResult&& result); + + // Getters + const std::string& get_operation_name() const { + return operation_name_; + } + float get_gflops() const { + return gflops_; + } + bool get_correctness_passed() const { + return correctness_passed_; + } + size_t size() const { + return results_.size(); + } + bool empty() const { + return results_.empty(); + } + + // Setters + void set_gflops(float gflops_val) { + gflops_ = gflops_val; + } + void set_correctness_passed(bool passed) { + correctness_passed_ = passed; + } + + // Access results + const BenchmarkResult& operator[](size_t index) const { + return results_[index]; + } + BenchmarkResult& operator[](size_t index) { + return results_[index]; + } + const std::vector& get_results() const { + return results_; + } + + // Iterator support + std::vector::iterator begin() { + return results_.begin(); + } + std::vector::iterator end() { + return results_.end(); + } + std::vector::const_iterator begin() const { + return results_.begin(); + } + std::vector::const_iterator end() const { + return results_.end(); + } + + // Processing and analysis + void print_summary() const; + void print_detailed_results() const; + void print_statistics() const; + void print_brief_summary() const; + + // Get aggregate statistics + float get_total_avg_time_us() const; + float get_total_gflops() const; + size_t get_passed_count() const; + size_t get_failed_count() const; + + // Find best/worst performing results + const BenchmarkResult* get_fastest_result() const; + const BenchmarkResult* get_slowest_result() const; + const BenchmarkResult* get_highest_gflops_result() const; + + // Clear all results + void clear() { + results_.clear(); + } + + // Set operation name + void set_operation_name(const std::string& name) { + operation_name_ = name; + } + + private: + std::string operation_name_; + std::vector results_; + float gflops_; + bool correctness_passed_; +}; + +// +// Print utilities +// + +void print_performance_header(); +void print_separator(); + +} // namespace prototyping +} // namespace vulkan +} // namespace executorch diff --git a/backends/vulkan/test/ops/benchmark/framework/runner.cpp b/backends/vulkan/test/ops/benchmark/framework/runner.cpp new file mode 100644 index 00000000000..3d5ce763bb5 --- /dev/null +++ b/backends/vulkan/test/ops/benchmark/framework/runner.cpp @@ -0,0 +1,760 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. +// All rights reserved. +// +// This source code is licensed under the BSD-style license found in the +// LICENSE file in the root directory of this source tree. + +#include "runner.h" +#include "config.h" + +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace executorch { +namespace vulkan { +namespace prototyping { + +namespace { + +// Upper bound on the auto-sized chained_dispatches factor. Caps the factor +// when an op is fast enough that target_us / probe_us would otherwise +// exceed this; past this point, stacking more dispatches doesn't materially +// improve measurement precision. +constexpr int kMaxChainedDispatches = 200; + +// Floor for probe_us to avoid division-by-zero / a runaway chained_dispatches +// factor when an op finishes faster than the wall clock can resolve. +constexpr float kMinProbeTimeUs = 1.0f; + +} // namespace + +// Benchmark graphs are immutable after input upload, so the operator nodes can +// be recorded repeatedly. Staging uploads/downloads are encoded once: +// repeating them would redo host-visible copies on every repetition and bias +// per-invocation timings. +RepeatedGraphExecutor::RepeatedGraphExecutor( + ComputeGraph& graph, + int repetitions, + OpNodeRange op_nodes) + : graph_(graph) { + VK_CHECK_COND(repetitions > 0); + VK_CHECK_COND(op_nodes.begin <= op_nodes.end); + VK_CHECK_COND(op_nodes.end <= graph_.execute_nodes().size()); + + api::Context* const context = graph_.context(); + context->flush(); + context->set_cmd(/*reusable=*/true); + context->cmd_reset_querypool(); + + for (size_t i = 0; i < op_nodes.begin; ++i) { + graph_.execute_nodes()[i]->encode(&graph_); + } + for (int i = 0; i < repetitions; ++i) { + for (size_t j = op_nodes.begin; j < op_nodes.end; ++j) { + graph_.execute_nodes()[j]->encode(&graph_); + } + } + for (size_t i = op_nodes.end; i < graph_.execute_nodes().size(); ++i) { + graph_.execute_nodes()[i]->encode(&graph_); + } + + command_ = + std::make_unique(std::move(context->extract_cmd())); +} + +void RepeatedGraphExecutor::execute() { + api::Context* const context = graph_.context(); + command_->end(); + + // Intentionally bypasses Context::submit_cmd_to_gpu(): its submit-count + // bookkeeping and threshold-split path only serve submit_compute_job + // recording, which benchmarks don't use. + vkapi::VulkanFence fence = context->fences().get_fence(); + context->adapter_ptr()->submit_cmd( + context->queue(), + command_->get_submit_handle(/*final_use=*/false), + fence.get_submit_handle()); + fence.wait(); + context->fences().return_fence(fence); +} + +// Helper function to collect GPU timing from querypool +float collect_gpu_timing_us( + ComputeGraph& graph, + const std::vector& shader_filter) { + graph.context()->querypool().extract_results(); + const auto results = graph.context()->querypool().get_shader_timestamp_data(); + if (!results.empty()) { + // Sum durations of all shaders that don't match any pattern in + // shader_filter + float total_duration_us = 0.0f; + for (const auto& shader_result : results) { + bool filtered = false; + // Check if this shader matches any filter pattern + for (const auto& filter_pattern : shader_filter) { + if (shader_result.kernel_name.find(filter_pattern) != + std::string::npos) { + filtered = true; + break; + } + } + + if (!filtered) { + // Calculate duration from start and end times, convert from ns to μs + uint64_t duration_ns = + shader_result.end_time_ns - shader_result.start_time_ns; + total_duration_us += static_cast(duration_ns) / 1000.0f; + } + } + return total_duration_us; + } + return 0.0f; +} + +// Helper function to collect per-shader GPU timing from querypool +// Returns a map of shader_name -> timing_us for non-filtered shaders +std::unordered_map collect_per_shader_timing_us( + ComputeGraph& graph, + const std::vector& shader_filter) { + std::unordered_map shader_timings; + + graph.context()->querypool().extract_results(); + const auto results = graph.context()->querypool().get_shader_timestamp_data(); + for (const auto& shader_result : results) { + bool filtered = false; + // Check if this shader matches any filter pattern + for (const auto& filter_pattern : shader_filter) { + if (shader_result.kernel_name.find(filter_pattern) != std::string::npos) { + filtered = true; + break; + } + } + + if (!filtered) { + // Calculate duration from start and end times, convert from ns to μs + uint64_t duration_ns = + shader_result.end_time_ns - shader_result.start_time_ns; + float duration_us = static_cast(duration_ns) / 1000.0f; + // Accumulate timing for shaders with the same name + shader_timings[shader_result.kernel_name] += duration_us; + } + } + return shader_timings; +} + +// Default FLOP calculation function (assumes 1 FLOP per element) +int64_t default_flop_calculator(const TestCase& test_case) { + // Calculate total elements from the first input tensor + int64_t total_elements = 1; + if (!test_case.empty() && test_case.num_inputs() > 0 && + test_case.inputs()[0].is_tensor()) { + const auto& sizes = test_case.inputs()[0].get_tensor_sizes(); + for (int64_t size : sizes) { + total_elements *= size; + } + } + + // Assume 1 FLOP per element for basic operations + return total_elements; +} + +BenchmarkGraph setup_compute_graph( + TestCase& test_case, + const std::string& op_name, + int op_invocations_per_execute) { + GraphConfig config; + config.enable_querypool = true; + // Pool sizing takes max(execute, prepack) * factor, so scaling the factor + // also over-reserves the prepack side (encoded once). Accepted: precise + // execute-only sizing would need runtime changes. + config.descriptor_pool_safety_factor *= + std::max(1, op_invocations_per_execute); + // Default-on (opt-out via TestCase::set_force_resize(false)): force every + // DynamicDispatchNode to run its resize function when execute_test_case + // runs propagate_resize() after prepack, exercising the op's resize formula + // even when input shapes are unchanged. + config.force_resize = test_case.get_force_resize(); + config.force_narrow_int4_tile = test_case.get_force_narrow_int4_tile(); + auto graph = std::make_unique(config); + + std::vector input_values; + + // Process input ValueSpecs + for (size_t i = 0; i < test_case.num_inputs(); ++i) { + const ValueSpec& input_spec = test_case.inputs()[i]; + + if (input_spec.is_none()) { + input_values.push_back(graph->add_none()); + } else if (input_spec.is_float()) { + ValueRef input_value = + graph->add_scalar(static_cast(input_spec.get_float_value())); + input_values.push_back(input_value); + } else if (input_spec.is_int()) { + ValueRef input_value = + graph->add_scalar(static_cast(input_spec.get_int_value())); + input_values.push_back(input_value); + } else if (input_spec.is_bool()) { + ValueRef input_value = graph->add_scalar(input_spec.get_bool_value()); + input_values.push_back(input_value); + } else if (input_spec.is_int_list()) { + // Convert int32_t list to int64_t list for ComputeGraph + const auto& int32_list = input_spec.get_int_list(); + std::vector int64_list; + int64_list.reserve(int32_list.size()); + for (int32_t val : int32_list) { + int64_list.push_back(static_cast(val)); + } + ValueRef input_value = graph->add_scalar_list(std::move(int64_list)); + input_values.push_back(input_value); + } else if (input_spec.is_string()) { + std::string str_copy = input_spec.get_string_value(); + ValueRef input_value = graph->add_string(std::move(str_copy)); + input_values.push_back(input_value); + } else if (input_spec.is_constant()) { + ValueRef input_value = graph->add_tensorref( + input_spec.get_tensor_sizes(), + input_spec.dtype, + input_spec.get_data_ptr()); + input_values.push_back(input_value); + } else { + IOValueRef input_io = graph->add_input_tensor( + input_spec.get_tensor_sizes(), + input_spec.dtype, + input_spec.storage_type, + input_spec.memory_layout); + input_values.push_back(input_io.value); + } + } + + std::vector output_values; + + // Process output ValueSpecs + for (size_t i = 0; i < test_case.num_outputs(); ++i) { + const ValueSpec& output_spec = test_case.outputs()[i]; + + if (!output_spec.is_tensor()) { + throw std::runtime_error("All output specifications must be tensors"); + } + + // Create output tensor + ValueRef output_value = graph->add_tensor( + output_spec.get_tensor_sizes(), + output_spec.dtype, + output_spec.storage_type, + output_spec.memory_layout); + + output_values.push_back(output_value); + } + + // Get the operator function and call it + auto opFn = VK_GET_OP_FN(op_name); + + // Create arguments vector for the operator function + std::vector op_args = input_values; + op_args.insert(op_args.end(), output_values.begin(), output_values.end()); + + // Nodes added before the op are staging uploads; nodes added after are + // staging downloads. Only the op's own nodes are repeated by benchmarks. + const size_t op_begin = graph->execute_nodes().size(); + opFn(*graph, op_args); + const size_t op_end = graph->execute_nodes().size(); + + for (size_t i = 0; i < output_values.size(); ++i) { + graph->set_output_value(output_values[i]); + } + return {std::move(graph), {op_begin, op_end}}; +} + +// Test execution utilities +BenchmarkResult execute_test_case( + TestCase& test_case, + int warmup_runs, + int benchmark_runs, + int chained_dispatches, + bool write_outputs) { + BenchmarkResult result( + test_case.name().empty() ? "unnamed_test_case" : test_case.name()); + + // Initialize querypool if using GPU timestamps + if (use_gpu_timestamps()) { + api::context()->initialize_querypool(); + } + + // Build the operator once. Benchmark repetition is encoded separately so + // persistent graph allocations are not duplicated. + BenchmarkGraph benchmark = setup_compute_graph( + test_case, test_case.operator_name(), chained_dispatches); + ComputeGraph& graph = *benchmark.graph; + + // Prepare the graph + graph.prepare(); + graph.prepack(); + + // Run resize functions once so force_resize exercises resize formulas even + // though the record/replay path below never calls propagate_resize(). + graph.propagate_resize(); + + // Copy input data into the graph's staging buffers + size_t graph_input_idx = 0; + for (size_t i = 0; i < test_case.num_inputs(); ++i) { + const ValueSpec& input_spec = test_case.inputs()[i]; + + // Only non-constant tensor inputs correspond to graph.inputs() entries + bool is_graph_input = input_spec.is_tensor() && !input_spec.is_constant() && + !input_spec.is_none(); + if (!is_graph_input) { + continue; + } + + if (graph_input_idx < graph.inputs().size()) { + const auto& input_ref = graph.inputs()[graph_input_idx]; + + // Get the appropriate data based on dtype + const void* data_ptr = nullptr; + size_t data_numel = input_spec.numel(); + + // NOLINTNEXTLINE(clang-diagnostic-switch-enum) + switch (input_spec.dtype) { + case vkapi::kFloat: + data_ptr = input_spec.get_float_data().data(); + break; + case vkapi::kHalf: + data_ptr = input_spec.get_half_data().data(); + break; + case vkapi::kInt: + data_ptr = input_spec.get_int32_data().data(); + break; + case vkapi::kChar: + data_ptr = input_spec.get_int8_data().data(); + break; + case vkapi::kByte: + data_ptr = input_spec.get_uint8_data().data(); + break; + default: + throw std::runtime_error("Unsupported data type for input tensor"); + } + + // Copy data into staging buffer + graph.maybe_cast_and_copy_into_staging( + input_ref.staging, data_ptr, data_numel, input_spec.dtype); + } + ++graph_input_idx; + } + + RepeatedGraphExecutor graph_executor( + graph, chained_dispatches, benchmark.op_nodes); + + // Warmup runs + for (int run = 0; run < warmup_runs; ++run) { + graph_executor.execute(); + } + + // Benchmark runs - collect individual iteration timings + float total_cpu_time_us = 0.0f; + float total_gpu_time_us = 0.0f; + + const float chained_dispatches_f = static_cast(chained_dispatches); + + for (int run = 0; run < benchmark_runs; ++run) { + // Measure CPU time for each execute() call + auto cpu_start = std::chrono::high_resolution_clock::now(); + graph_executor.execute(); + auto cpu_end = std::chrono::high_resolution_clock::now(); + + auto cpu_duration = std::chrono::duration_cast( + cpu_end - cpu_start); + // Each execute() ran chained_dispatches op invocations; report + // per-invocation time. + float cpu_time_us = + static_cast(cpu_duration.count()) / chained_dispatches_f; + total_cpu_time_us += cpu_time_us; + + // Collect per-shader GPU timing - get raw shader results to preserve + // metadata + graph.context()->querypool().extract_results(); + const auto shader_results = + graph.context()->querypool().get_shader_timestamp_data(); + + // Calculate total GPU time from per-shader timings + float gpu_time_us = 0.0f; + for (const auto& shader_result : shader_results) { + // Check if this shader matches any filter pattern + bool filtered = false; + for (const auto& filter_pattern : test_case.get_shader_filter()) { + if (shader_result.kernel_name.find(filter_pattern) != + std::string::npos) { + filtered = true; + break; + } + } + + if (!filtered) { + uint64_t duration_ns = + shader_result.end_time_ns - shader_result.start_time_ns; + float duration_us = static_cast(duration_ns) / 1000.0f; + gpu_time_us += duration_us; + // Per-shader sample is already one querypool entry per dispatch (at + // chained_dispatches>1 we get that many entries per shader), so do NOT + // divide here — the aggregator (get_avg_time_us) averages across them + // naturally. + result.add_shader_timing( + shader_result.kernel_name, + duration_us, + shader_result.metadata.gwg, + shader_result.metadata.lwg); + } + } + // gpu_time_us aggregates chained_dispatches worth of shader runs; divide + // it down to per-invocation time. + gpu_time_us /= chained_dispatches_f; + total_gpu_time_us += gpu_time_us; + + // Add the appropriate timing based on the flag + float iter_time_us = use_gpu_timestamps() ? gpu_time_us : cpu_time_us; + result.add_iter_timing(iter_time_us); + } + + // Calculate averages for display + float avg_cpu_time_us = total_cpu_time_us / benchmark_runs; + float avg_gpu_time_us = total_gpu_time_us / benchmark_runs; + + // Print both timings if latency printing is enabled + if (print_latencies()) { + if (use_gpu_timestamps()) { + graph.context()->querypool().print_results(); + } + std::cout << " CPU timing: " << std::fixed << std::setprecision(3) + << avg_cpu_time_us << " μs" << std::endl; + std::cout << " GPU timing: " << std::fixed << std::setprecision(3) + << avg_gpu_time_us << " μs" << std::endl; + std::cout << " Using " << (use_gpu_timestamps() ? "GPU" : "CPU") + << " timing for result" << std::endl; + } + + // Copy output data from the graph's staging buffers. The benchmarking path + // skips this work since it doesn't need correctness data — the probe pass + // has already populated test_case.outputs() at chained_dispatches=1. + if (write_outputs) { + for (size_t i = 0; i < test_case.num_outputs(); ++i) { + ValueSpec& output_spec = test_case.outputs()[i]; + + if (output_spec.is_tensor() && i < graph.outputs().size()) { + const auto& output_ref = graph.outputs()[i]; + + // Ensure output data vector is properly sized + size_t data_numel = output_spec.numel(); + output_spec.resize_data(data_numel); + + // Get mutable data pointer for the output + void* data_ptr = output_spec.get_mutable_data_ptr(); + + if (data_ptr != nullptr) { + // Copy data from staging buffer to output spec + graph.maybe_cast_and_copy_from_staging( + output_ref.staging, data_ptr, data_numel, output_spec.dtype); + } + + // Print output tensor data if output printing is enabled + if (print_output()) { + std::string output_name = "Output[" + std::to_string(i) + "]"; + print_valuespec_data(output_spec, output_name); + } + } + } + } + + return result; +} + +TestResult execute_test_cases( + const std::function()>& test_case_generator, + const FlopCalculatorFunc& flop_calculator, + const std::string& operation_name, + int warmup_runs, + int benchmark_runs, + const ReferenceComputeFunc& reference_compute_func) { + TestResult results(operation_name); + + // Generate all test cases + std::vector test_cases = test_case_generator(); + + std::cout << "Executing " << test_cases.size() << " test cases for " + << operation_name << std::endl; + print_separator(); + + // Group test cases by ReferenceKey for caching reference computations + // Use a vector to preserve the order in which groups first appear + std::vector group_order; + std::unordered_map, ReferenceKeyHash> + groups; + for (size_t i = 0; i < test_cases.size(); ++i) { + ReferenceKey key = ReferenceKey::from_test_case(test_cases[i]); + if (groups.find(key) == groups.end()) { + group_order.push_back(key); + } + groups[key].push_back(i); + } + + bool any_correctness_failed = false; + float total_gflops = 0.0f; + size_t test_case_counter = 0; + + // Process each group: generate data, compute reference, execute, and print + // Iterate in the order groups first appeared in test_cases + for (const auto& key : group_order) { + const auto& indices = groups[key]; + if (indices.empty()) { + continue; + } + + // Get first test case as the "prototype" + size_t prototype_idx = indices[0]; + TestCase& prototype = test_cases[prototype_idx]; + + // Generate data for prototype with deterministic seed based on key + int group_seed = + static_cast(std::hash{}(key.key_string) % 10000); + for (auto& input : prototype.inputs()) { + input.ensure_data_generated(group_seed++); + } + + // Compute reference once for prototype + bool ref_computed = false; + if (reference_compute_func) { + try { + reference_compute_func(prototype); + ref_computed = true; + } catch (const std::invalid_argument&) { + // Reference computation skipped for this group + } + } + + // Copy data and reference to other test cases in group + for (size_t i = 1; i < indices.size(); ++i) { + size_t tc_idx = indices[i]; + TestCase& tc = test_cases[tc_idx]; + + // Copy input data from prototype + for (size_t j = 0; + j < tc.inputs().size() && j < prototype.inputs().size(); + ++j) { + auto& dest = tc.inputs()[j]; + const auto& src = prototype.inputs()[j]; + if (dest.is_tensor() && src.is_tensor() && dest.sizes == src.sizes && + dest.dtype == src.dtype) { + dest.share_data_from(src); + } + } + + // Copy reference output data if available + if (ref_computed) { + for (size_t j = 0; + j < tc.outputs().size() && j < prototype.outputs().size(); + ++j) { + const auto& src = prototype.outputs()[j]; + auto& dest = tc.outputs()[j]; + if (dest.is_tensor() && src.is_tensor() && dest.sizes == src.sizes && + dest.dtype == src.dtype) { + dest.share_reference_from(src); + } + } + } + } + + // Execute and print results for all test cases in this group + for (size_t tc_idx : indices) { + TestCase& test_case = test_cases[tc_idx]; + ++test_case_counter; + + // Execute single test case + BenchmarkResult result; + bool shader_not_supported = false; + try { + // Always run a probe pass at chained_dispatches=1 with + // write_outputs=true. This populates test_case.outputs() for the + // downstream correctness check with a clean single-dispatch result, + // and also gives us probe_us for adaptive sizing of the measurement + // run's chained_dispatches. + // + // probe_then_scale (Google Benchmark style): size chained_dispatches + // so each measurement execute() takes ~target_us. Tiny ops get a + // large factor (driving GPU governor escalation); heavy ops get a + // small factor (bounded wall time). Caveat: on Adreno 740 the probe + // runs at the DCVS-pinned clock (~220 MHz), so probe_us is inflated + // and the computed factor comes out under-sized for boost clock — but + // the default target is generous enough that an under-sized factor + // still drives sustained activity. + BenchmarkResult probe_result = execute_test_case( + test_case, + /*warmup_runs=*/1, + /*benchmark_runs=*/1, + /*chained_dispatches=*/1, + /*write_outputs=*/true); + float probe_us = probe_result.get_avg_time_us(); + if (probe_us < kMinProbeTimeUs) { + probe_us = kMinProbeTimeUs; + } + + int chained_dispatches; + const int manual_n = test_case.get_op_invocations_per_execute(); + if (manual_n > 0) { + chained_dispatches = manual_n; + } else { + const int target_us = test_case.get_target_execute_time_us(); + chained_dispatches = + std::max(1, static_cast(target_us / probe_us)); + chained_dispatches = + std::min(chained_dispatches, kMaxChainedDispatches); + } + if (debugging()) { + std::cout << "[probe] " << test_case.name() + << ": probe_us=" << probe_us + << ", chained_dispatches=" << chained_dispatches + << std::endl; + } + + // Measurement pass: chained_dispatches stacking, skip output copy + // since the probe already wrote the correctness data. + result = execute_test_case( + test_case, + warmup_runs, + benchmark_runs, + chained_dispatches, + /*write_outputs=*/false); + result.set_operator_name(test_case.operator_name()); + } catch (const vkcompute::vkapi::ShaderNotSupportedError&) { + result = BenchmarkResult( + test_case.name().empty() ? "unnamed_test_case" : test_case.name(), + test_case.operator_name()); + shader_not_supported = true; + } + + // Determine if this test case passed (has valid timing data) + bool vulkan_execute_succeeded = + result.get_num_iterations() > 0 && result.get_avg_time_us() > 0.0f; + + if (shader_not_supported) { + result.set_correctness_status(CorrectnessStatus::SKIPPED); + } else if (!vulkan_execute_succeeded) { + result.set_correctness_status(CorrectnessStatus::FAILED); + } else if (!ref_computed) { + result.set_correctness_status(CorrectnessStatus::SKIPPED); + } else { + // Reference function provided and succeeded - validate outputs + bool correctness_passed = true; + + for (size_t output_idx = 0; output_idx < test_case.num_outputs(); + ++output_idx) { + const ValueSpec& output_spec = test_case.outputs()[output_idx]; + + if (!output_spec.validate_against_reference( + test_case.get_abs_tolerance(), + test_case.get_rel_tolerance())) { + correctness_passed = false; + std::cout << " Correctness validation FAILED for test " + << result.get_kernel_name() << std::endl; + print_valuespec_data(output_spec, "vulkan output"); + print_valuespec_data(output_spec, "ref output", true); + } + } + + if (correctness_passed) { + result.set_correctness_status(CorrectnessStatus::PASSED); + } else { + any_correctness_failed = true; + result.set_correctness_status(CorrectnessStatus::FAILED); + } + } + + // Calculate GFLOPS for this test case using the provided FLOP calculator + float case_gflops = 0.0f; + if (vulkan_execute_succeeded) { + // Use the provided FLOP calculator to get total FLOPs for this test + // case + int64_t total_flops = flop_calculator(test_case); + float flops = static_cast(total_flops); + float avg_time_us = result.get_avg_time_us(); + if (avg_time_us > 0.0f && total_flops > 0) { + case_gflops = (flops / 1e9f) / (avg_time_us / 1e6f); + } + + total_gflops += case_gflops; + } else { + case_gflops = -1.0f; // Indicate failure + } + + // Calculate tensor info for display + std::string size_info = "["; + if (!test_case.empty() && test_case.num_inputs() > 0 && + test_case.inputs()[0].is_tensor()) { + const auto& sizes = test_case.inputs()[0].get_tensor_sizes(); + for (size_t j = 0; j < sizes.size(); ++j) { + size_info += std::to_string(sizes[j]); + if (j < sizes.size() - 1) { + size_info += "x"; + } + } + } + size_info += "]"; + + // Print progress using the BenchmarkResult member function + result.print_summary( + utils::safe_downcast(test_case_counter), size_info, case_gflops); + + // Add result to collection + results.add_result(std::move(result)); + + test_case.clear(); + } + } + + // Set the overall results on the TestResult + results.set_correctness_passed(!any_correctness_failed); + results.set_gflops(total_gflops); + + print_separator(); + std::cout << "Completed " << results.size() << " test cases" << std::endl; + + // Hard-fail if any correctness check failed. The per-case loop above keeps + // running after a mismatch (so the full pass/fail matrix and per-16x16-tile + // mismatch maps are printed for debugging) rather than aborting on the first + // failure; throwing here preserves the original abort-on-mismatch behavior. + if (any_correctness_failed) { + throw std::runtime_error("Correctness validation failed"); + } + + return results; +} + +// Convenience overload that uses the default FLOP calculator +TestResult execute_test_cases( + const std::function()>& test_case_generator, + const std::string& operation_name, + int warmup_runs, + int benchmark_runs, + const ReferenceComputeFunc& reference_compute_func) { + return execute_test_cases( + test_case_generator, + default_flop_calculator, + operation_name, + warmup_runs, + benchmark_runs, + reference_compute_func); +} + +} // namespace prototyping +} // namespace vulkan +} // namespace executorch diff --git a/backends/vulkan/test/ops/benchmark/framework/runner.h b/backends/vulkan/test/ops/benchmark/framework/runner.h new file mode 100644 index 00000000000..b3493aed9cb --- /dev/null +++ b/backends/vulkan/test/ops/benchmark/framework/runner.h @@ -0,0 +1,123 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. +// All rights reserved. +// +// This source code is licensed under the BSD-style license found in the +// LICENSE file in the root directory of this source tree. + +#pragma once + +#include + +#include "results.h" +#include "test_case.h" + +#include +#include +#include +#include +#include + +namespace executorch { +namespace vulkan { +namespace prototyping { + +// +// Test case execution +// + +using FlopCalculatorFunc = std::function; + +// Default FLOP calculation function (assumes 1 FLOP per element) +int64_t default_flop_calculator(const TestCase& test_case); + +using ReferenceComputeFunc = std::function; + +// Half-open index range of the operator's own dispatch nodes within a +// benchmark graph's execute_nodes(). Staging upload nodes precede it, staging +// download nodes follow it. +struct OpNodeRange { + size_t begin = 0; + size_t end = 0; +}; + +// A benchmark graph plus the location of its repeatable operator nodes. The +// graph is heap-held: ComputeGraph owns its Context and must never be moved +// (a moved-from graph's destructor dereferences a null context). +struct BenchmarkGraph { + std::unique_ptr graph; + OpNodeRange op_nodes; +}; + +// Benchmark-only executor that records a graph's execute nodes into a single +// reusable command buffer, then replays it on every execute(). Staging uploads +// are encoded once, operator nodes N times, staging downloads once, so each +// iteration performs the same work as the old stacked-nodes layout (1 upload + +// N ops + 1 download) and the per-invocation divisor is unchanged. Production +// ComputeGraph::execute() behavior is unchanged. +// +// Notes for interpreting benchmark numbers: +// - Submit granularity differs from stacking N distinct nodes: all encodings +// live in one command buffer with one submit per iteration (the old path +// could split across command buffers at the node-count threshold), so +// per-dispatch times may shift systematically against older data. +// - Resize functions run once via propagate_resize() in execute_test_case +// before recording; replay itself never re-triggers resize. +// - Repeated encodings share one node/dispatch id, so per-repetition +// querypool attribution is unavailable (aggregation keys on kernel name). +class RepeatedGraphExecutor final { + public: + RepeatedGraphExecutor( + ComputeGraph& graph, + int repetitions, + OpNodeRange op_nodes); + void execute(); + + private: + ComputeGraph& graph_; + std::unique_ptr command_; +}; + +// Runs a measurement at the given chained_dispatches factor. The operator is +// built once, then its execute nodes are encoded that many times into a +// benchmark-only reusable command buffer. The probe-then-scale orchestration +// lives in execute_test_cases(). +// +// write_outputs controls whether the graph's staging output buffers are copied +// back into test_case.outputs() at the end of the run. The probe path needs +// write_outputs=true so the correctness check has a clean single-dispatch +// reference. The benchmarking path passes write_outputs=false to avoid the +// per-iter GPU->CPU copy cost. +BenchmarkResult execute_test_case( + TestCase& test_case, + int warmup_runs = 1, + int benchmark_runs = 1, + int chained_dispatches = 1, + bool write_outputs = true); + +TestResult execute_test_cases( + const std::function()>& test_case_generator, + const FlopCalculatorFunc& flop_calculator, + const std::string& operation_name = "Operation", + int warmup_runs = 1, + int benchmark_runs = 1, + const ReferenceComputeFunc& reference_compute_func = nullptr); + +TestResult execute_test_cases( + const std::function()>& test_case_generator, + const std::string& operation_name = "Operation", + int warmup_runs = 1, + int benchmark_runs = 1, + const ReferenceComputeFunc& reference_compute_func = nullptr); + +// Setup compute graph based on TestCase and operation name. The op function is +// invoked once. op_invocations_per_execute is used only to reserve enough +// descriptor capacity for benchmark-only repeated command encoding. Returns +// the graph plus the range of the operator's own nodes for repeated encoding. +BenchmarkGraph setup_compute_graph( + TestCase& test_case, + const std::string& op_name, + int op_invocations_per_execute = 1); + +} // namespace prototyping +} // namespace vulkan +} // namespace executorch diff --git a/backends/vulkan/test/ops/benchmark/framework/test_case.cpp b/backends/vulkan/test/ops/benchmark/framework/test_case.cpp new file mode 100644 index 00000000000..550bd10c136 --- /dev/null +++ b/backends/vulkan/test/ops/benchmark/framework/test_case.cpp @@ -0,0 +1,79 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. +// All rights reserved. +// +// This source code is licensed under the BSD-style license found in the +// LICENSE file in the root directory of this source tree. + +#include "test_case.h" + +#include + +namespace executorch { +namespace vulkan { +namespace prototyping { + +// ReferenceKey implementation +ReferenceKey ReferenceKey::from_test_case(const TestCase& tc) { + std::ostringstream oss; + + // Serialize inputs that affect reference computation + // Skip: storage_type, memory_layout, string values (like impl_selector) + for (size_t i = 0; i < tc.inputs().size(); ++i) { + const ValueSpec& input = tc.inputs()[i]; + oss << "i" << i << ":"; + + if (input.is_tensor()) { + // For tensors: sizes, dtype, data_gen_type, is_constant + oss << "T["; + for (size_t j = 0; j < input.sizes.size(); ++j) { + if (j > 0) { + oss << ","; + } + oss << input.sizes[j]; + } + oss << "]d" << static_cast(input.dtype); + oss << "g" << static_cast(input.data_gen_type); + oss << "c" << (input.is_constant() ? 1 : 0); + oss << "n" << (input.is_none() ? 1 : 0); + } else if (input.is_int()) { + oss << "I" << input.get_int_value(); + } else if (input.is_float()) { + oss << "F" << input.get_float_value(); + } else if (input.is_bool()) { + oss << "B" << (input.get_bool_value() ? 1 : 0); + } else if (input.is_int_list()) { + oss << "L["; + const auto& list = input.get_int_list(); + for (size_t j = 0; j < list.size(); ++j) { + if (j > 0) { + oss << ","; + } + oss << list[j]; + } + oss << "]"; + } + // Skip string inputs (like impl_selector) as they don't affect reference + oss << ";"; + } + + // Also include output shapes for completeness + for (size_t i = 0; i < tc.outputs().size(); ++i) { + const ValueSpec& output = tc.outputs()[i]; + oss << "o" << i << ":["; + for (size_t j = 0; j < output.sizes.size(); ++j) { + if (j > 0) { + oss << ","; + } + oss << output.sizes[j]; + } + oss << "]d" << static_cast(output.dtype) << ";"; + } + + ReferenceKey key; + key.key_string = oss.str(); + return key; +} + +} // namespace prototyping +} // namespace vulkan +} // namespace executorch diff --git a/backends/vulkan/test/ops/benchmark/framework/test_case.h b/backends/vulkan/test/ops/benchmark/framework/test_case.h new file mode 100644 index 00000000000..5edd5838ffc --- /dev/null +++ b/backends/vulkan/test/ops/benchmark/framework/test_case.h @@ -0,0 +1,236 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. +// All rights reserved. +// +// This source code is licensed under the BSD-style license found in the +// LICENSE file in the root directory of this source tree. + +#pragma once + +#include "value_spec.h" + +#include +#include +#include + +namespace executorch { +namespace vulkan { +namespace prototyping { + +// +// ReferenceKey for caching reference computations +// + +// Captures the identity of input conditions for test case grouping. +// Test cases with the same ReferenceKey should produce identical reference +// outputs, so reference computation can be cached and reused. +struct ReferenceKey { + std::string key_string; + + static ReferenceKey from_test_case(const class TestCase& tc); + + bool operator==(const ReferenceKey& other) const { + return key_string == other.key_string; + } +}; + +struct ReferenceKeyHash { + size_t operator()(const ReferenceKey& k) const { + return std::hash{}(k.key_string); + } +}; + +// +// Shader filter defaults +// + +// Default shader filter: excludes layout conversions and quantization overhead +// This is used for tests where quantize/dequantize are overhead operations +inline const std::vector kDefaultShaderFilter = { + "nchw_to", + "to_nchw", + "quantize_and_pack_4w4c", + "unpack_4w4c_and_dequantize"}; + +// Layout-only filter: only excludes layout conversions +// Use this for tests where quantize/dequantize ARE the operations being tested +inline const std::vector kLayoutOnlyShaderFilter = { + "nchw_to", + "to_nchw"}; + +// +// TestCase +// + +// Default per-execute() wall-clock target used by the probe-then-scale logic +// in execute_test_cases(). Picked generously enough that even an under-sized +// chained_dispatches factor (the probe runs at governor-pinned clock and +// underestimates the boost-clock latency) still drives sustained GPU +// activity during measurement. +constexpr int kDefaultTargetExecuteTimeUs = 100000; + +class TestCase { + public: + TestCase() + : abs_tolerance_(2e-3f), + rel_tolerance_(1e-3f), + shader_filter_(kDefaultShaderFilter) {} + explicit TestCase(const std::string& name) + : name_(name), + abs_tolerance_(2e-3f), + rel_tolerance_(1e-3f), + shader_filter_(kDefaultShaderFilter) {} + + void set_name(const std::string& name) { + name_ = name; + } + const std::string& name() const { + return name_; + } + + void set_operator_name(const std::string& op_name) { + operator_name_ = op_name; + } + const std::string& operator_name() const { + return operator_name_; + } + + // Tolerance settings + void set_abs_tolerance(float abs_tolerance) { + abs_tolerance_ = abs_tolerance; + } + float get_abs_tolerance() const { + return abs_tolerance_; + } + + void set_rel_tolerance(float rel_tolerance) { + rel_tolerance_ = rel_tolerance; + } + float get_rel_tolerance() const { + return rel_tolerance_; + } + + // Shader filter settings - list of shader name patterns to exclude from + // timing + void set_shader_filter(const std::vector& filter) { + shader_filter_ = filter; + } + const std::vector& get_shader_filter() const { + return shader_filter_; + } + + // Manual override for the number of chained dispatches per measurement + // iteration (a.k.a. chained_dispatches). If > 0, the framework uses this + // directly and skips the probe phase. 0 (the default) means adaptive + // (probe-then-scale). + void set_op_invocations_per_execute(int n) { + op_invocations_per_execute_ = n; + } + int get_op_invocations_per_execute() const { + return op_invocations_per_execute_; + } + + // Target single-execute duration in microseconds. Used only when the + // manual chained_dispatches override is not set. Default + // kDefaultTargetExecuteTimeUs, picked generously enough to mitigate Adreno + // DCVS governor pinning during the probe. + void set_target_execute_time_us(int us) { + target_execute_time_us_ = us; + } + int get_target_execute_time_us() const { + return target_execute_time_us_; + } + + // When true, the ComputeGraph built for this test case sets + // GraphConfig::force_resize, so every DynamicDispatchNode runs its resize + // function once during measurement setup (execute_test_case runs + // propagate_resize() after prepack) even when no input shape changed. + // Because the output is already allocated at the swept shape, the resize + // must recompute the same shape from the current input — a wrong resize + // formula resizes the output to a mismatched shape and surfaces as a test + // failure. Default true (opt-out): every custom_ops test exercises its + // resize formulas across the swept shapes. Call set_force_resize(false) for + // the rare op whose resize fn is intentionally not shape-preserving under a + // fixed output allocation. + void set_force_resize(bool force_resize) { + force_resize_ = force_resize; + } + bool get_force_resize() const { + return force_resize_; + } + + // When true, the ComputeGraph sets GraphConfig::force_narrow_int4_tile, so + // the int4 linear picks its narrow (2-row) output tile on any GPU. That + // variant is otherwise reachable only on Mali, which no CI runner has. + void set_force_narrow_int4_tile(bool force) { + force_narrow_int4_tile_ = force; + } + bool get_force_narrow_int4_tile() const { + return force_narrow_int4_tile_; + } + + void add_input_spec(const ValueSpec& spec) { + inputs_.push_back(spec); + } + + const std::vector& inputs() const { + return inputs_; + } + + std::vector& inputs() { + return inputs_; + } + + size_t num_inputs() const { + return inputs_.size(); + } + + void add_output_spec(const ValueSpec& spec) { + outputs_.push_back(spec); + } + + const std::vector& outputs() const { + return outputs_; + } + + std::vector& outputs() { + return outputs_; + } + + size_t num_outputs() const { + return outputs_.size(); + } + + bool empty() const { + return inputs_.empty() && outputs_.empty(); + } + void clear() { + inputs_.clear(); + outputs_.clear(); + name_.clear(); + operator_name_.clear(); + abs_tolerance_ = 2e-3f; + rel_tolerance_ = 1e-3f; + shader_filter_ = kDefaultShaderFilter; + op_invocations_per_execute_ = 0; + target_execute_time_us_ = kDefaultTargetExecuteTimeUs; + force_resize_ = true; + force_narrow_int4_tile_ = false; + } + + private: + std::string name_; + std::string operator_name_; + std::vector inputs_; + std::vector outputs_; + float abs_tolerance_; + float rel_tolerance_; + std::vector shader_filter_; + int op_invocations_per_execute_ = 0; // 0 = adaptive + int target_execute_time_us_ = kDefaultTargetExecuteTimeUs; + bool force_resize_ = true; + bool force_narrow_int4_tile_ = false; +}; + +} // namespace prototyping +} // namespace vulkan +} // namespace executorch diff --git a/backends/vulkan/test/custom_ops/test_coopmat_probe.cpp b/backends/vulkan/test/ops/benchmark/framework/test_coopmat_probe.cpp similarity index 100% rename from backends/vulkan/test/custom_ops/test_coopmat_probe.cpp rename to backends/vulkan/test/ops/benchmark/framework/test_coopmat_probe.cpp diff --git a/backends/vulkan/test/ops/benchmark/framework/utils.h b/backends/vulkan/test/ops/benchmark/framework/utils.h new file mode 100644 index 00000000000..51d5fd1bd8f --- /dev/null +++ b/backends/vulkan/test/ops/benchmark/framework/utils.h @@ -0,0 +1,27 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. +// All rights reserved. +// +// This source code is licensed under the BSD-style license found in the +// LICENSE file in the root directory of this source tree. + +#pragma once + +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include + +#include "config.h" +#include "labels.h" +#include "results.h" +#include "runner.h" +#include "test_case.h" +#include "value_spec.h" +#include "weight_utils.h" diff --git a/backends/vulkan/test/ops/benchmark/framework/value_spec.cpp b/backends/vulkan/test/ops/benchmark/framework/value_spec.cpp new file mode 100644 index 00000000000..b9752596f89 --- /dev/null +++ b/backends/vulkan/test/ops/benchmark/framework/value_spec.cpp @@ -0,0 +1,1024 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. +// All rights reserved. +// +// This source code is licensed under the BSD-style license found in the +// LICENSE file in the root directory of this source tree. + +#include "value_spec.h" +#include "config.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace executorch { +namespace vulkan { +namespace prototyping { + +int get_seed() { + static int seed = 42; + return seed++; +} + +int get_seed_or_explicit(int explicit_seed) { + if (explicit_seed >= 0) { + return explicit_seed; + } + return get_seed(); +} + +// Forward declarations for data generation utilities +void generate_random_float_data( + std::vector& data, + float min_val = -1.0f, + float max_val = 1.0f, + int explicit_seed = -1); +void generate_random_int_data( + std::vector& data, + int min_val = -10, + int max_val = 10, + int explicit_seed = -1); +void generate_randint_float_data( + std::vector& data, + int min_val = -10, + int max_val = 10, + int explicit_seed = -1); +void generate_randint_half_data( + std::vector& data, + int min_val = -10, + int max_val = 10, + int explicit_seed = -1); +void generate_random_int8_data( + std::vector& data, + int8_t min_val = -10, + int8_t max_val = 10, + int explicit_seed = -1); +void generate_random_uint8_data( + std::vector& data, + uint8_t min_val = 0, + uint8_t max_val = 255, + int explicit_seed = -1); +void generate_random_2xint4_data( + std::vector& data, + int explicit_seed = -1); +void generate_random_2xint4_data( + std::vector& data, + int explicit_seed = -1); +void generate_random_int4_data( + std::vector& data, + int8_t min_val = -8, + int8_t max_val = 7, + int explicit_seed = -1); +void generate_ones_data(std::vector& data); +void generate_zeros_data(std::vector& data); + +// Convert a float32 value to IEEE 754 half-precision (uint16_t) +uint16_t float_to_half(float value) { + uint32_t float_bits; + std::memcpy(&float_bits, &value, sizeof(float)); + uint32_t sign = (float_bits >> 31) & 0x1; + int32_t exponent = static_cast((float_bits >> 23) & 0xFF) - 127; + uint32_t mantissa = float_bits & 0x7FFFFF; + + uint16_t half_val; + if (exponent > 15) { + half_val = static_cast((sign << 15) | 0x7C00); // Inf + } else if (exponent < -14) { + half_val = static_cast(sign << 15); // Zero / subnormal + } else { + half_val = static_cast( + (sign << 15) | (static_cast(exponent + 15) << 10) | + (mantissa >> 13)); + } + return half_val; +} + +// Convert a IEEE 754 half-precision (uint16_t) value to float32 +float half_to_float(uint16_t half_val) { + uint32_t sign = (half_val >> 15) & 0x1; + uint32_t exponent = (half_val >> 10) & 0x1F; + uint32_t mantissa = half_val & 0x3FF; + + float result; + if (exponent == 0) { + result = std::ldexp(static_cast(mantissa), -24); + } else if (exponent == 31) { + result = mantissa ? std::numeric_limits::quiet_NaN() + : std::numeric_limits::infinity(); + } else { + result = std::ldexp( + 1.0f + static_cast(mantissa) / 1024.0f, exponent - 15); + } + if (sign) { + result = -result; + } + return result; +} + +// ValueSpec implementation +void ValueSpec::ensure_unique_data() const { + if (data_.use_count() != 1) { + data_ = std::make_shared(*data_); + } +} + +void ValueSpec::ensure_unique_reference_data() const { + if (reference_data_.use_count() != 1) { + reference_data_ = std::make_shared(*reference_data_); + } +} + +void ValueSpec::generate_tensor_data(int seed) const { + if (spec_type != SpecType::Tensor) { + return; + } + + ensure_unique_data(); + auto& float_data = data_->float_data; + auto& int32_data = data_->int32_data; + auto& half_data = data_->half_data; + auto& int8_data = data_->int8_data; + auto& uint8_data = data_->uint8_data; + + int64_t num_elements = numel(); + + // NOLINTNEXTLINE(clang-diagnostic-switch-enum) + switch (dtype) { + case vkapi::kFloat: { + float_data.resize(num_elements); + if (data_gen_type == DataGenType::RANDOM) { + generate_random_float_data(float_data, -1.0f, 1.0f, seed); + } else if (data_gen_type == DataGenType::RANDOM_SCALES) { + generate_random_float_data(float_data, 0.005, 0.015, seed); + } else if (data_gen_type == DataGenType::RANDINT) { + generate_randint_float_data(float_data, -10, 10, seed); + } else if (data_gen_type == DataGenType::RANDINT8) { + generate_randint_float_data(float_data, -128, 127, seed); + } else if (data_gen_type == DataGenType::RANDINT4) { + generate_randint_float_data(float_data, -8, 7, seed); + } else if (data_gen_type == DataGenType::ONES) { + generate_ones_data(float_data); + } else if (data_gen_type == DataGenType::ZEROS) { + generate_zeros_data(float_data); + } else { + generate_zeros_data(float_data); + } + break; + } + case vkapi::kHalf: { + half_data.resize(num_elements); + if (data_gen_type == DataGenType::RANDOM) { + // Generate random float data first, then convert to IEEE 754 half. + std::vector temp_data(num_elements); + generate_random_float_data(temp_data, -1.0f, 1.0f, seed); + for (size_t i = 0; i < temp_data.size(); ++i) { + half_data[i] = float_to_half(temp_data[i]); + } + } else if (data_gen_type == DataGenType::RANDOM_SCALES) { + // Generate random scales in float, then convert to proper fp16 + std::vector temp_data(num_elements); + generate_random_float_data(temp_data, 0.005f, 0.015f, seed); + for (size_t i = 0; i < temp_data.size(); ++i) { + half_data[i] = float_to_half(temp_data[i]); + } + } else if (data_gen_type == DataGenType::RANDINT) { + generate_randint_half_data(half_data, -10, 10, seed); + } else if (data_gen_type == DataGenType::RANDINT8) { + generate_randint_half_data(half_data, -128, 127, seed); + } else if (data_gen_type == DataGenType::RANDINT4) { + generate_randint_half_data(half_data, -8, 7, seed); + } else if (data_gen_type == DataGenType::ONES) { + std::fill(half_data.begin(), half_data.end(), float_to_half(1.0f)); + } else if (data_gen_type == DataGenType::ZEROS) { + std::fill( + half_data.begin(), + half_data.end(), + static_cast(0)); // 0.0 in half + } else { + std::fill( + half_data.begin(), + half_data.end(), + static_cast(0)); // 0.0 in half + } + break; + } + case vkapi::kInt: { + int32_data.resize(num_elements); + if (data_gen_type == DataGenType::RANDOM) { + generate_random_int_data(int32_data, -10, 10, seed); + } else if (data_gen_type == DataGenType::RANDINT) { + generate_random_int_data( + int32_data, + -10, + 10, + seed); // For int type, RANDINT is same as RANDOM + } else if (data_gen_type == DataGenType::RANDINT8) { + generate_random_int_data(int32_data, -128, 127, seed); + } else if (data_gen_type == DataGenType::RANDINT4) { + generate_random_int_data(int32_data, -8, 7, seed); + } else if (data_gen_type == DataGenType::ONES) { + std::fill(int32_data.begin(), int32_data.end(), 1); + } else if (data_gen_type == DataGenType::ZEROS) { + std::fill(int32_data.begin(), int32_data.end(), 0); + } else { + std::fill(int32_data.begin(), int32_data.end(), 0); + } + break; + } + case vkapi::kChar: { + int8_data.resize(num_elements); + if (data_gen_type == DataGenType::RANDOM) { + generate_random_int8_data(int8_data, -10, 10, seed); + } else if (data_gen_type == DataGenType::RANDINT) { + generate_random_int8_data(int8_data, -10, 10, seed); + } else if (data_gen_type == DataGenType::RANDINT8) { + generate_random_int8_data(int8_data, -128, 127, seed); + } else if (data_gen_type == DataGenType::RANDINT4) { + generate_random_2xint4_data(int8_data, seed); + } else if (data_gen_type == DataGenType::ONES) { + std::fill(int8_data.begin(), int8_data.end(), 1); + } else if (data_gen_type == DataGenType::ONES_INT4) { + int8_t packed_data = (1 << 4) | 1; + std::fill(int8_data.begin(), int8_data.end(), packed_data); + } else if (data_gen_type == DataGenType::ZEROS) { + std::fill(int8_data.begin(), int8_data.end(), 0); + } else { + std::fill(int8_data.begin(), int8_data.end(), 0); + } + break; + } + case vkapi::kByte: { + uint8_data.resize(num_elements); + if (data_gen_type == DataGenType::RANDOM) { + generate_random_uint8_data(uint8_data, 0, 255, seed); + } else if (data_gen_type == DataGenType::RANDINT) { + generate_random_uint8_data(uint8_data, 0, 255, seed); + } else if (data_gen_type == DataGenType::RANDINT8) { + generate_random_uint8_data(uint8_data, 0, 255, seed); + } else if (data_gen_type == DataGenType::RANDINT4) { + generate_random_2xint4_data(uint8_data, seed); + } else if (data_gen_type == DataGenType::ONES) { + std::fill(uint8_data.begin(), uint8_data.end(), 1); + } else if (data_gen_type == DataGenType::ONES_INT4) { + uint8_t packed_data = (9 << 4) | 9; + std::fill(uint8_data.begin(), uint8_data.end(), packed_data); + } else if (data_gen_type == DataGenType::ZEROS) { + std::fill(uint8_data.begin(), uint8_data.end(), 0); + } else { + std::fill(uint8_data.begin(), uint8_data.end(), 0); + } + break; + } + default: + // Default to float + float_data.resize(num_elements); + if (data_gen_type == DataGenType::RANDOM) { + generate_random_float_data(float_data, -1.0f, 1.0f, seed); + } else if (data_gen_type == DataGenType::RANDINT) { + generate_randint_float_data(float_data, -10, 10, seed); + } else if (data_gen_type == DataGenType::ONES) { + generate_ones_data(float_data); + } else if (data_gen_type == DataGenType::ZEROS) { + generate_zeros_data(float_data); + } else { + generate_zeros_data(float_data); + } + break; + } +} + +int64_t ValueSpec::numel() const { + if (spec_type == SpecType::Int || spec_type == SpecType::Float || + spec_type == SpecType::Bool) { + return 1; + } else if (spec_type == SpecType::IntList) { + return sizes.empty() ? 0 : sizes[0]; + } else { // Tensor + int64_t total = 1; + for (int64_t size : sizes) { + total *= size; + } + return total; + } +} + +size_t ValueSpec::nbytes() const { + size_t element_size = 0; + // NOLINTNEXTLINE(clang-diagnostic-switch-enum) + switch (dtype) { + case vkapi::kFloat: + element_size = sizeof(float); + break; + case vkapi::kHalf: + element_size = sizeof(uint16_t); + break; + case vkapi::kInt: + element_size = sizeof(int32_t); + break; + case vkapi::kChar: + element_size = sizeof(int8_t); + break; + case vkapi::kByte: + element_size = sizeof(uint8_t); + break; + default: + element_size = sizeof(float); // Default fallback + break; + } + return numel() * element_size; +} + +std::string ValueSpec::to_string() const { + std::string result = "ValueSpec("; + + switch (spec_type) { + case SpecType::Tensor: + result += "type=Tensor, sizes=["; + break; + case SpecType::IntList: + result += "type=IntList, count="; + result += std::to_string(sizes.empty() ? 0 : sizes[0]); + result += ", data_gen="; + result += (data_gen_type == DataGenType::FIXED) ? "FIXED" : "RANDOM"; + result += ")"; + return result; + case SpecType::Int: + result += "type=Int, value="; + result += std::to_string(get_int_value()); + result += ", data_gen="; + result += (data_gen_type == DataGenType::FIXED) ? "FIXED" : "RANDOM"; + result += ")"; + return result; + case SpecType::Float: + result += "type=Float, value="; + result += std::to_string(get_float_value()); + result += ", data_gen="; + result += (data_gen_type == DataGenType::FIXED) ? "FIXED" : "RANDOM"; + result += ")"; + return result; + case SpecType::Bool: + result += "type=Bool, value="; + result += get_bool_value() ? "true" : "false"; + result += ", data_gen="; + result += (data_gen_type == DataGenType::FIXED) ? "FIXED" : "RANDOM"; + result += ")"; + return result; + case SpecType::String: + result += "type=String, value=\""; + result += get_string_value(); + result += "\")"; + return result; + } + + for (size_t i = 0; i < sizes.size(); ++i) { + result += std::to_string(sizes[i]); + if (i < sizes.size() - 1) { + result += ", "; + } + } + result += "]"; + + if (spec_type == SpecType::Tensor) { + result += ", dtype="; + // NOLINTNEXTLINE(clang-diagnostic-switch-enum) + switch (dtype) { + case vkapi::kFloat: + result += "float"; + break; + case vkapi::kHalf: + result += "half"; + break; + case vkapi::kInt: + result += "int32"; + break; + case vkapi::kChar: + result += "int8"; + break; + case vkapi::kByte: + result += "uint8"; + break; + default: + result += "unknown"; + break; + } + + result += ", memory_layout="; + // NOLINTNEXTLINE(clang-diagnostic-switch-enum) + switch (memory_layout) { + case utils::kWidthPacked: + result += "WidthPacked"; + break; + case utils::kHeightPacked: + result += "HeightPacked"; + break; + case utils::kChannelsPacked: + result += "ChannelsPacked"; + break; + default: + result += "unknown"; + break; + } + + result += ", storage_type="; + // NOLINTNEXTLINE(clang-diagnostic-switch-enum) + switch (storage_type) { + case utils::kTexture3D: + result += "Texture3D"; + break; + case utils::kBuffer: + result += "Buffer"; + break; + default: + result += "unknown"; + break; + } + } + + result += ", data_gen="; + // NOLINTNEXTLINE(clang-diagnostic-switch-enum) + switch (data_gen_type) { + case DataGenType::FIXED: + result += "FIXED"; + break; + case DataGenType::RANDOM: + result += "RANDOM"; + break; + case DataGenType::RANDINT: + result += "RANDINT"; + break; + case DataGenType::RANDINT8: + result += "RANDINT8"; + break; + case DataGenType::RANDINT4: + result += "RANDINT4"; + break; + case DataGenType::ONES: + result += "ONES"; + break; + case DataGenType::ZEROS: + result += "ZEROS"; + break; + default: + result += "unknown"; + break; + } + result += ")"; + return result; +} + +// Additional ValueSpec methods +void ValueSpec::resize_data(size_t new_size) { + // Generate first so a deferred tensor keeps its data-gen pattern (resized, + // not pinned to zeros by the data_generated_ flag set below). + ensure_data_generated(); + ensure_unique_data(); + // NOLINTNEXTLINE(clang-diagnostic-switch-enum) + switch (dtype) { + case vkapi::kFloat: + data_->float_data.resize(new_size); + break; + case vkapi::kHalf: + data_->half_data.resize(new_size); + break; + case vkapi::kInt: + data_->int32_data.resize(new_size); + break; + case vkapi::kChar: + data_->int8_data.resize(new_size); + break; + case vkapi::kByte: + data_->uint8_data.resize(new_size); + break; + default: + data_->float_data.resize(new_size); + break; + } + data_generated_ = true; +} + +void* ValueSpec::get_mutable_data_ptr() { + ensure_data_generated(); + ensure_unique_data(); + // NOLINTNEXTLINE(clang-diagnostic-switch-enum) + switch (dtype) { + case vkapi::kFloat: + return data_->float_data.data(); + case vkapi::kHalf: + return data_->half_data.data(); + case vkapi::kInt: + return data_->int32_data.data(); + case vkapi::kChar: + return data_->int8_data.data(); + case vkapi::kByte: + return data_->uint8_data.data(); + default: + return data_->float_data.data(); + } +} + +float ValueSpec::get_element(size_t index) const { + ensure_data_generated(); + if (index >= static_cast(numel())) { + return 0.0f; + } + + // NOLINTNEXTLINE(clang-diagnostic-switch-enum) + switch (dtype) { + case vkapi::kFloat: + return index < data_->float_data.size() ? data_->float_data[index] : 0.0f; + case vkapi::kHalf: + return index < data_->half_data.size() + ? half_to_float(data_->half_data[index]) + : 0.0f; + case vkapi::kInt: + return index < data_->int32_data.size() + ? static_cast(data_->int32_data[index]) + : 0.0f; + case vkapi::kChar: + return index < data_->int8_data.size() + ? static_cast(data_->int8_data[index]) + : 0.0f; + case vkapi::kByte: + return index < data_->uint8_data.size() + ? static_cast(data_->uint8_data[index]) + : 0.0f; + default: + return 0.0f; + } +} + +const void* ValueSpec::get_data_ptr() const { + ensure_data_generated(); + // NOLINTNEXTLINE(clang-diagnostic-switch-enum) + switch (dtype) { + case vkapi::kFloat: + return data_->float_data.data(); + case vkapi::kHalf: + return data_->half_data.data(); + case vkapi::kInt: + return data_->int32_data.data(); + case vkapi::kChar: + return data_->int8_data.data(); + case vkapi::kByte: + return data_->uint8_data.data(); + default: + throw std::runtime_error("Unsupported data type for get_data_ptr"); + } +} + +void generate_random_float_data( + std::vector& data, + float min_val, + float max_val, + int explicit_seed) { + std::mt19937 gen(get_seed_or_explicit(explicit_seed)); + std::uniform_real_distribution dis(min_val, max_val); + for (auto& val : data) { + val = dis(gen); + } +} + +void generate_random_int_data( + std::vector& data, + int min_val, + int max_val, + int explicit_seed) { + std::mt19937 gen(get_seed_or_explicit(explicit_seed)); + std::uniform_int_distribution dis(min_val, max_val); + for (auto& val : data) { + val = dis(gen); + } +} + +void generate_randint_float_data( + std::vector& data, + int min_val, + int max_val, + int explicit_seed) { + std::mt19937 gen(get_seed_or_explicit(explicit_seed)); + std::uniform_int_distribution dis(min_val, max_val); + for (auto& val : data) { + val = static_cast(dis(gen)); + } +} + +void generate_randint_half_data( + std::vector& data, + int min_val, + int max_val, + int explicit_seed) { + std::mt19937 gen(get_seed_or_explicit(explicit_seed)); + std::uniform_int_distribution dis(min_val, max_val); + for (auto& val : data) { + val = float_to_half(static_cast(dis(gen))); + } +} + +void generate_ones_data(std::vector& data) { + std::fill(data.begin(), data.end(), 1.0f); +} + +void generate_random_int8_data( + std::vector& data, + int8_t min_val, + int8_t max_val, + int explicit_seed) { + std::mt19937 gen(get_seed_or_explicit(explicit_seed)); + std::uniform_int_distribution dis(min_val, max_val); + for (auto& val : data) { + val = static_cast(dis(gen)); + } +} + +void generate_random_uint8_data( + std::vector& data, + uint8_t min_val, + uint8_t max_val, + int explicit_seed) { + std::mt19937 gen(get_seed_or_explicit(explicit_seed)); + std::uniform_int_distribution dis(min_val, max_val); + for (auto& val : data) { + val = static_cast(dis(gen)); + } +} + +void generate_random_int4_data( + std::vector& data, + int8_t min_val, + int8_t max_val, + int explicit_seed) { + std::mt19937 gen(get_seed_or_explicit(explicit_seed)); + std::uniform_int_distribution dis(min_val, max_val); + for (auto& val : data) { + val = static_cast(dis(gen)); + } +} + +void generate_random_2xint4_data(std::vector& data, int explicit_seed) { + std::mt19937 gen(get_seed_or_explicit(explicit_seed)); + std::uniform_int_distribution dis(-8, 7); // Signed 4-bit range + for (auto& val : data) { + // Generate two separate 4-bit values + int8_t lower_4bits = static_cast(dis(gen)) & 0x0F; + int8_t upper_4bits = static_cast(dis(gen)) & 0x0F; + // Pack them into a single 8-bit value + val = (upper_4bits << 4) | lower_4bits; + } +} + +void generate_random_2xint4_data( + std::vector& data, + int explicit_seed) { + std::mt19937 gen(get_seed_or_explicit(explicit_seed)); + std::uniform_int_distribution dis(0, 15); // Unsigned 4-bit range + for (auto& val : data) { + // Generate two separate 4-bit values + uint8_t lower_4bits = static_cast(dis(gen)) & 0x0F; + uint8_t upper_4bits = static_cast(dis(gen)) & 0x0F; + // Pack them into a single 8-bit value + val = (upper_4bits << 4) | lower_4bits; + } +} + +void generate_zeros_data(std::vector& data) { + std::fill(data.begin(), data.end(), 0.0f); +} + +// Correctness checking against reference data +bool ValueSpec::validate_against_reference( + float abs_tolerance, + float rel_tolerance) const { + // Only validate float and half tensors. For half tensors, convert the + // computed half data to float for comparison against the fp32 reference. + if (!is_tensor() || (dtype != vkapi::kFloat && dtype != vkapi::kHalf)) { + return true; // Skip validation for non-float/half or non-tensor types + } + + // For kHalf, materialize the GPU output as float so the same tolerance + // machinery can compare against the (always-float) reference data. + std::vector half_as_float; + if (dtype == vkapi::kHalf) { + const auto& half_bits = get_half_data(); + half_as_float.resize(half_bits.size()); + for (size_t i = 0; i < half_bits.size(); ++i) { + half_as_float[i] = half_to_float(half_bits[i]); + } + } + // Materialize computed data as float32 for comparison. The dtype is + // guaranteed to be float or half by the early-out above. + const std::vector& computed_data = + (dtype == vkapi::kHalf) ? half_as_float : get_float_data(); + const auto& reference_data = get_ref_float_data(); + + // Skip validation if no reference data is available + if (reference_data.empty()) { + return true; + } + + // Check if sizes match + if (computed_data.size() != reference_data.size()) { + if (debugging()) { + std::cout << "Size mismatch: computed=" << computed_data.size() + << ", reference=" << reference_data.size() << std::endl; + } + return false; + } + + // Element-wise comparison with both absolute and relative tolerance + size_t num_mismatched = 0; + size_t first_mismatch = 0; + for (size_t i = 0; i < computed_data.size(); ++i) { + float diff = std::abs(computed_data[i] - reference_data[i]); + float abs_ref = std::abs(reference_data[i]); + + // Check if either absolute or relative tolerance condition is satisfied + bool abs_tolerance_ok = diff <= abs_tolerance; + bool rel_tolerance_ok = diff <= rel_tolerance * abs_ref; + + if (!abs_tolerance_ok && !rel_tolerance_ok) { + if (num_mismatched == 0) { + first_mismatch = i; + std::cout << "Mismatch at element " << i + << ": computed=" << computed_data[i] + << ", reference=" << reference_data[i] << ", diff=" << diff + << ", abs_tolerance=" << abs_tolerance + << ", rel_tolerance=" << rel_tolerance + << ", rel_threshold=" << (rel_tolerance * abs_ref) + << std::endl; + } + num_mismatched++; + } + } + if (num_mismatched > 0) { + std::cout << " total mismatched: " << num_mismatched << " / " + << computed_data.size() << " (first at " << first_mismatch << ")" + << std::endl; + // For 2D outputs, print a per-16x16-tile mismatch-count map to expose + // the spatial structure of the failure (e.g. zeroed MMA subtiles). + if (sizes.size() == 2) { + const int64_t Mr = sizes[0]; + const int64_t Nc = sizes[1]; + std::cout << " 16x16-tile mismatch counts (rows=M/16, cols=N/16):" + << std::endl; + for (int64_t ti = 0; ti < (Mr + 15) / 16; ++ti) { + std::cout << " "; + for (int64_t tj = 0; tj < (Nc + 15) / 16; ++tj) { + int count = 0; + for (int64_t r = ti * 16; r < std::min(Mr, (ti + 1) * 16); ++r) { + for (int64_t c = tj * 16; c < std::min(Nc, (tj + 1) * 16); ++c) { + float diff = std::abs( + computed_data[r * Nc + c] - reference_data[r * Nc + c]); + float abs_ref = std::abs(reference_data[r * Nc + c]); + if (diff > abs_tolerance && diff > rel_tolerance * abs_ref) { + count++; + } + } + } + std::cout << std::setw(4) << count; + } + std::cout << std::endl; + } + } + return false; + } + + if (debugging()) { + std::cout << "Correctness validation PASSED" << std::endl; + } + return true; +} + +// Ensure data is generated for this ValueSpec +void ValueSpec::ensure_data_generated(int seed) const { + if (data_generated_) { + return; + } + generate_tensor_data(seed); + data_generated_ = true; +} + +void ValueSpec::share_data_from(const ValueSpec& other) { + if (!is_tensor() || !other.is_tensor()) { + return; + } + // Materialize the source first: sharing an ungenerated payload would let a + // later access materialize each spec independently under different seeds. + other.ensure_data_generated(); + data_ = other.data_; + data_generated_ = true; +} + +void ValueSpec::share_reference_from(const ValueSpec& other) { + if (!is_tensor() || !other.is_tensor()) { + return; + } + reference_data_ = other.reference_data_; +} + +// ValueSpec data printing utilities +void print_valuespec_data( + const ValueSpec& spec, + const std::string& name, + const bool print_ref_data, + size_t max_elements, + int precision) { + std::cout << "\n" << name << " Data:" << std::endl; + std::cout << " Type: " << spec.to_string() << std::endl; + + if (!spec.is_tensor()) { + if (spec.is_int()) { + std::cout << " Value: " << spec.get_int_value() << std::endl; + } else if (spec.is_int_list()) { + const auto& int_list = spec.get_int_list(); + std::cout << " Values: ["; + size_t print_count = std::min(max_elements, int_list.size()); + for (size_t i = 0; i < print_count; ++i) { + std::cout << int_list[i]; + if (i < print_count - 1) { + std::cout << ", "; + } + } + if (int_list.size() > max_elements) { + std::cout << ", ... (" << (int_list.size() - max_elements) << " more)"; + } + std::cout << "]" << std::endl; + } + return; + } + + // Print tensor data + size_t total_elements = spec.numel(); + size_t print_count = std::min(max_elements, total_elements); + + std::cout << " Total elements: " << total_elements << std::endl; + std::cout << " Data (first " << print_count << " elements): ["; + + std::cout << std::fixed << std::setprecision(precision); + + // NOLINTNEXTLINE(clang-diagnostic-switch-enum) + switch (spec.dtype) { + case vkapi::kFloat: { + auto data = spec.get_float_data().data(); + if (print_ref_data) { + data = spec.get_ref_float_data().data(); + } + for (size_t i = 0; i < print_count; ++i) { + std::cout << data[i]; + if (i < print_count - 1) { + std::cout << ", "; + } + } + break; + } + case vkapi::kHalf: { + if (print_ref_data) { + const auto& ref = spec.get_ref_float_data(); + for (size_t i = 0; i < print_count; ++i) { + std::cout << ref[i]; + if (i < print_count - 1) { + std::cout << ", "; + } + } + break; + } + const auto& data = spec.get_half_data(); + for (size_t i = 0; i < print_count; ++i) { + // Convert IEEE 754 half-precision bit pattern back to float. + float value = half_to_float(data[i]); + std::cout << value; + if (i < print_count - 1) { + std::cout << ", "; + } + } + break; + } + case vkapi::kInt: { + const auto& data = spec.get_int32_data(); + for (size_t i = 0; i < print_count; ++i) { + std::cout << data[i]; + if (i < print_count - 1) { + std::cout << ", "; + } + } + break; + } + case vkapi::kChar: { + const auto& data = spec.get_int8_data(); + if (spec.is_int4()) { + // Print each 4-bit value individually + size_t element_count = 0; + for (size_t i = 0; i < data.size() && element_count < print_count; + ++i) { + // Extract lower 4 bits (signed) + int8_t lower_4bits = data[i] & 0x0F; + if (lower_4bits > 7) { + lower_4bits -= 16; // Convert to signed + } + std::cout << static_cast(lower_4bits); + element_count++; + + if (element_count < print_count) { + std::cout << ", "; + // Extract upper 4 bits (signed) + int8_t upper_4bits = (data[i] >> 4) & 0x0F; + if (upper_4bits > 7) { + upper_4bits -= 16; // Convert to signed + } + std::cout << static_cast(upper_4bits); + element_count++; + + if (element_count < print_count) { + std::cout << ", "; + } + } + } + } else { + for (size_t i = 0; i < print_count; ++i) { + std::cout << static_cast(data[i]); + if (i < print_count - 1) { + std::cout << ", "; + } + } + } + break; + } + case vkapi::kByte: { + const auto& data = spec.get_uint8_data(); + if (spec.is_int4()) { + // Print each 4-bit value individually + size_t element_count = 0; + for (size_t i = 0; i < data.size() && element_count < print_count; + ++i) { + // Extract lower 4 bits + uint8_t lower_4bits = data[i] & 0x0F; + std::cout << static_cast(lower_4bits); + element_count++; + + if (element_count < print_count) { + std::cout << ", "; + // Extract upper 4 bits + uint8_t upper_4bits = (data[i] >> 4) & 0x0F; + std::cout << static_cast(upper_4bits); + element_count++; + + if (element_count < print_count) { + std::cout << ", "; + } + } + } + } else { + for (size_t i = 0; i < print_count; ++i) { + std::cout << static_cast(data[i]); + if (i < print_count - 1) { + std::cout << ", "; + } + } + } + break; + } + default: + std::cout << "unsupported data type"; + break; + } + + if (total_elements > max_elements) { + std::cout << ", ... (" << (total_elements - max_elements) << " more)"; + } + std::cout << "]" << std::endl; + + // Print some statistics for tensor data + if (total_elements > 0) { + float min_val = 0.0f, max_val = 0.0f, sum = 0.0f; + bool first = true; + + for (size_t i = 0; i < total_elements; ++i) { + float val = spec.get_element(i); + if (first) { + min_val = max_val = val; + first = false; + } else { + min_val = std::min(min_val, val); + max_val = std::max(max_val, val); + } + sum += val; + } + + float mean = sum / total_elements; + std::cout << " Statistics: min=" << std::setprecision(precision) << min_val + << ", max=" << max_val << ", mean=" << mean << ", sum=" << sum + << std::endl; + } +} + +} // namespace prototyping +} // namespace vulkan +} // namespace executorch diff --git a/backends/vulkan/test/ops/benchmark/framework/value_spec.h b/backends/vulkan/test/ops/benchmark/framework/value_spec.h new file mode 100644 index 00000000000..ebb87e2c323 --- /dev/null +++ b/backends/vulkan/test/ops/benchmark/framework/value_spec.h @@ -0,0 +1,404 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. +// All rights reserved. +// +// This source code is licensed under the BSD-style license found in the +// LICENSE file in the root directory of this source tree. + +#pragma once + +#include + +#include +#include +#include +#include + +namespace executorch { +namespace vulkan { +namespace prototyping { + +using namespace vkcompute; + +// +// ValueSpec class +// + +enum class SpecType { Tensor, IntList, Int, Float, Bool, String }; + +// Data generation types +enum class DataGenType { + FIXED, + RANDOM, + RANDOM_SCALES, + RANDINT, + RANDINT8, + RANDINT4, + ONES, + ONES_INT4, + ZEROS +}; + +// Value specification struct +struct ValueSpec { + std::vector sizes; + vkapi::ScalarType dtype; + utils::GPUMemoryLayout memory_layout; + utils::StorageType storage_type; + SpecType spec_type; + DataGenType data_gen_type; + bool is_constant_tensor; + bool is_none_flag; + bool is_int4_tensor; + std::string string_data; + + ValueSpec( + const std::vector& sizes, + vkapi::ScalarType dtype, + utils::StorageType storage_type = utils::kTexture3D, + utils::GPUMemoryLayout memory_layout = utils::kWidthPacked) + : sizes(sizes), + dtype(dtype), + memory_layout(memory_layout), + storage_type(storage_type), + spec_type(SpecType::Tensor), + data_gen_type(DataGenType::ZEROS), + is_constant_tensor(false), + is_none_flag(false), + is_int4_tensor(false), + data_generated_(false) { + // Data generation is deferred until first access (any data getter or + // ensure_data_generated() triggers it). + } + + // Constructor for tensor with custom data generation type + ValueSpec( + const std::vector& sizes, + vkapi::ScalarType dtype, + utils::StorageType storage_type, + utils::GPUMemoryLayout memory_layout, + DataGenType data_gen_type) + : sizes(sizes), + dtype(dtype), + memory_layout(memory_layout), + storage_type(storage_type), + spec_type(SpecType::Tensor), + data_gen_type(data_gen_type), + is_constant_tensor(false), + is_none_flag(false), + is_int4_tensor(false), + data_generated_(false) { + // Data generation is deferred until first access (any data getter or + // ensure_data_generated() triggers it). + } + + // Constructor for single int + ValueSpec(int32_t value) + : sizes({1}), + dtype(vkapi::kInt), + memory_layout(utils::kWidthPacked), + storage_type(utils::kTexture3D), + spec_type(SpecType::Int), + data_gen_type(DataGenType::FIXED), + is_constant_tensor(false), + is_none_flag(false), + is_int4_tensor(false), + data_generated_(true) { + data_->int32_data.push_back(value); + } + + // Constructor for single float + ValueSpec(float value) + : sizes({1}), + dtype(vkapi::kFloat), + memory_layout(utils::kWidthPacked), + storage_type(utils::kTexture3D), + spec_type(SpecType::Float), + data_gen_type(DataGenType::FIXED), + is_constant_tensor(false), + is_none_flag(false), + is_int4_tensor(false), + data_generated_(true) { + data_->float_data.push_back(value); + } + + // Constructor for single bool + ValueSpec(bool value) + : sizes({1}), + dtype(vkapi::kInt), + memory_layout(utils::kWidthPacked), + storage_type(utils::kTexture3D), + spec_type(SpecType::Bool), + data_gen_type(DataGenType::FIXED), + is_constant_tensor(false), + is_none_flag(false), + is_int4_tensor(false), + data_generated_(true) { + data_->int32_data.push_back(value ? 1 : 0); + } + + // Constructor for int list + ValueSpec(const std::vector& values) + : sizes({static_cast(values.size())}), + dtype(vkapi::kInt), + memory_layout(utils::kWidthPacked), + storage_type(utils::kTexture3D), + spec_type(SpecType::IntList), + data_gen_type(DataGenType::FIXED), + is_constant_tensor(false), + is_none_flag(false), + is_int4_tensor(false), + data_generated_(true) { + data_->int32_data = values; + } + + // Factory method for string (avoids ambiguity with vector constructor) + static ValueSpec make_string(const std::string& value) { + ValueSpec spec; + spec.sizes = {1}; + spec.dtype = vkapi::kInt; + spec.memory_layout = utils::kWidthPacked; + spec.storage_type = utils::kTexture3D; + spec.spec_type = SpecType::String; + spec.data_gen_type = DataGenType::FIXED; + spec.is_constant_tensor = false; + spec.is_none_flag = false; + spec.is_int4_tensor = false; + spec.data_generated_ = true; + spec.string_data = value; + return spec; + } + + // Default constructor + ValueSpec() + : dtype(vkapi::kFloat), + memory_layout(utils::kWidthPacked), + storage_type(utils::kTexture3D), + spec_type(SpecType::Tensor), + data_gen_type(DataGenType::ZEROS), + is_constant_tensor(false), + is_none_flag(false), + is_int4_tensor(false), + data_generated_(false) {} + + int64_t numel() const; + size_t nbytes() const; + std::string to_string() const; + + bool is_tensor() const { + return spec_type == SpecType::Tensor; + } + bool is_int_list() const { + return spec_type == SpecType::IntList; + } + bool is_int() const { + return spec_type == SpecType::Int; + } + bool is_float() const { + return spec_type == SpecType::Float; + } + bool is_bool() const { + return spec_type == SpecType::Bool; + } + bool is_string() const { + return spec_type == SpecType::String; + } + + int32_t get_int_value() const { + ensure_data_generated(); + return data_->int32_data.empty() ? 0 : data_->int32_data[0]; + } + float get_float_value() const { + ensure_data_generated(); + return data_->float_data.empty() ? 0.0f : data_->float_data[0]; + } + bool get_bool_value() const { + ensure_data_generated(); + return data_->int32_data.empty() ? false : (data_->int32_data[0] != 0); + } + const std::string& get_string_value() const { + return string_data; + } + const std::vector& get_int_list() const { + ensure_data_generated(); + return data_->int32_data; + } + const std::vector& get_tensor_sizes() const { + return sizes; + } + + // References and pointers into tensor data must not be held across any other + // access to the same spec: a mutable access may detach the shared payload, + // leaving a previously returned reference bound to the old payload. Consume + // immediately. + const std::vector& get_float_data() const { + ensure_data_generated(); + return data_->float_data; + } + const std::vector& get_int32_data() const { + ensure_data_generated(); + return data_->int32_data; + } + const std::vector& get_half_data() const { + ensure_data_generated(); + return data_->half_data; + } + const std::vector& get_int8_data() const { + ensure_data_generated(); + return data_->int8_data; + } + const std::vector& get_uint8_data() const { + ensure_data_generated(); + return data_->uint8_data; + } + + std::vector& get_float_data() { + ensure_data_generated(); + ensure_unique_data(); + return data_->float_data; + } + std::vector& get_int32_data() { + ensure_data_generated(); + ensure_unique_data(); + return data_->int32_data; + } + std::vector& get_half_data() { + ensure_data_generated(); + ensure_unique_data(); + return data_->half_data; + } + std::vector& get_int8_data() { + ensure_data_generated(); + ensure_unique_data(); + return data_->int8_data; + } + std::vector& get_uint8_data() { + ensure_data_generated(); + ensure_unique_data(); + return data_->uint8_data; + } + + const std::vector& get_ref_float_data() const { + return reference_data_->float_data; + } + const std::vector& get_ref_int32_data() const { + return reference_data_->int32_data; + } + const std::vector& get_ref_half_data() const { + return reference_data_->half_data; + } + const std::vector& get_ref_int8_data() const { + return reference_data_->int8_data; + } + const std::vector& get_ref_uint8_data() const { + return reference_data_->uint8_data; + } + + std::vector& get_ref_float_data() { + ensure_unique_reference_data(); + return reference_data_->float_data; + } + std::vector& get_ref_int32_data() { + ensure_unique_reference_data(); + return reference_data_->int32_data; + } + std::vector& get_ref_half_data() { + ensure_unique_reference_data(); + return reference_data_->half_data; + } + std::vector& get_ref_int8_data() { + ensure_unique_reference_data(); + return reference_data_->int8_data; + } + std::vector& get_ref_uint8_data() { + ensure_unique_reference_data(); + return reference_data_->uint8_data; + } + + void resize_data(size_t new_size); + void* get_mutable_data_ptr(); + float get_element(size_t index) const; + + // Data generation methods for deferred generation and caching. + // + // ValueSpec is not thread-safe: lazy materialization and copy-on-write + // detach mutate shared state from const methods. Test cases are built and + // executed on a single thread. + // + // Implicit materialization (any data getter, resize_data) consumes the + // global seed counter. Callers needing deterministic data must call + // ensure_data_generated(explicit_seed) before any other access; a later + // seeded call is a no-op once data is generated. + bool is_data_generated() const { + return data_generated_; + } + void ensure_data_generated(int seed = -1) const; + void share_data_from(const ValueSpec& other); + void share_reference_from(const ValueSpec& other); + + // Set/get constant flag + bool is_constant() const { + return is_constant_tensor; + } + void set_constant(bool is_constant) { + is_constant_tensor = is_constant; + } + + // Set/get none flag + bool is_none() const { + return is_none_flag; + } + + void set_none(bool is_none) { + is_none_flag = is_none; + } + + // Set/get int4 flag + bool is_int4() const { + return is_int4_tensor; + } + void set_int4(bool is_int4) { + is_int4_tensor = is_int4; + } + + const void* get_data_ptr() const; + + // Correctness checking against reference data + // Returns true if computed data matches reference data within tolerance + // Only validates float tensors as specified in requirements + bool validate_against_reference( + float abs_tolerance = 2e-3f, + float rel_tolerance = 1e-3f) const; + + private: + struct TensorData { + std::vector float_data; + std::vector int32_data; + std::vector half_data; + std::vector int8_data; + std::vector uint8_data; + }; + + void ensure_unique_data() const; + void ensure_unique_reference_data() const; + void generate_tensor_data(int seed = -1) const; + + mutable bool data_generated_ = false; + mutable std::shared_ptr data_ = std::make_shared(); + mutable std::shared_ptr reference_data_ = + std::make_shared(); +}; + +void print_valuespec_data( + const ValueSpec& spec, + const std::string& name = "ValueSpec", + const bool print_ref_data = false, + size_t max_elements = 20, + int precision = 6); + +// Half-precision conversion utilities +uint16_t float_to_half(float value); +float half_to_float(uint16_t half_val); + +} // namespace prototyping +} // namespace vulkan +} // namespace executorch diff --git a/backends/vulkan/test/ops/benchmark/framework/weight_utils.cpp b/backends/vulkan/test/ops/benchmark/framework/weight_utils.cpp new file mode 100644 index 00000000000..5161178d978 --- /dev/null +++ b/backends/vulkan/test/ops/benchmark/framework/weight_utils.cpp @@ -0,0 +1,247 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. +// All rights reserved. +// +// This source code is licensed under the BSD-style license found in the +// LICENSE file in the root directory of this source tree. + +#include "weight_utils.h" +#include "config.h" + +#include + +#include +#include +#include +#include +#include +#include + +namespace executorch { +namespace vulkan { +namespace prototyping { + +ValueRef quantized_weights_canvas( + ComputeGraph& graph, + const ValueRef weight_ref) { + const auto original_sizes = graph.sizes_of(weight_ref); + + std::vector sorted_sizes = original_sizes; + std::sort(sorted_sizes.begin(), sorted_sizes.end(), std::greater()); + int64_t largest1 = sorted_sizes.size() > 0 ? sorted_sizes[0] : 0; + + std::vector final_sizes = {1, largest1, largest1}; + + // Debug logging if debugging flag is set + if (debugging()) { + std::cout << "Debug: Creating quantized weights canvas tensor" << std::endl; + std::cout << "Debug: Original sizes: ["; + for (size_t i = 0; i < original_sizes.size(); ++i) { + std::cout << original_sizes[i]; + if (i < original_sizes.size() - 1) + std::cout << ", "; + } + std::cout << "]" << std::endl; + std::cout << "Debug: Canvas sizes: ["; + for (size_t i = 0; i < final_sizes.size(); ++i) { + std::cout << final_sizes[i]; + if (i < final_sizes.size() - 1) + std::cout << ", "; + } + std::cout << "]" << std::endl; + } + + ValueRef packed_weight = graph.add_tensor( + final_sizes, vkapi::kInt, utils::kTexture3D, utils::kWidthPacked); + + std::string kernel_name = "packed_int32_canvas"; + add_storage_type_suffix(kernel_name, graph.storage_type_of(packed_weight)); + + graph.prepack_nodes().emplace_back(new PrepackNode( + graph, + VK_KERNEL_FROM_STR(kernel_name), + graph.create_gwg(packed_weight), + graph.create_lwg(packed_weight), + weight_ref, + packed_weight, + // UBOs + {graph.logical_limits_ubo(packed_weight)}, + // Specialization constants + {}, + // Push Constants + {})); + + return packed_weight; +} + +ValueRef float_tensor_canvas(ComputeGraph& graph, const ValueRef weight_ref) { + const auto original_sizes = graph.sizes_of(weight_ref); + + std::vector sorted_sizes = original_sizes; + std::sort(sorted_sizes.begin(), sorted_sizes.end(), std::greater()); + int64_t largest1 = sorted_sizes.size() > 0 ? sorted_sizes[0] : 0; + + std::vector final_sizes = {1, largest1, largest1}; + + // Debug logging if debugging flag is set + if (debugging()) { + std::cout << "Debug: Creating float tensor canvas" << std::endl; + std::cout << "Debug: Original sizes: ["; + for (size_t i = 0; i < original_sizes.size(); ++i) { + std::cout << original_sizes[i]; + if (i < original_sizes.size() - 1) + std::cout << ", "; + } + std::cout << "]" << std::endl; + std::cout << "Debug: Canvas sizes: ["; + for (size_t i = 0; i < final_sizes.size(); ++i) { + std::cout << final_sizes[i]; + if (i < final_sizes.size() - 1) + std::cout << ", "; + } + std::cout << "]" << std::endl; + } + + ValueRef packed_weight = graph.add_tensor( + final_sizes, vkapi::kFloat, utils::kTexture3D, utils::kWidthPacked); + + graph.prepack_nodes().emplace_back(new PrepackNode( + graph, + VK_KERNEL_FROM_STR("float_canvas"), + graph.create_gwg(packed_weight), + graph.create_lwg(packed_weight), + weight_ref, + packed_weight, + // UBOs + {graph.logical_limits_ubo(packed_weight)}, + // Specialization constants + {}, + // Push Constants + {})); + + return packed_weight; +} + +// Compute weight sums for quantized operations (linear and convolution) +void compute_weight_sums( + ValueSpec& weight_sums, + const ValueSpec& quantized_weight, + int64_t out_features, + int64_t elements_per_output_feature) { + auto& weight_sums_data = weight_sums.get_int32_data(); + auto& quantized_weight_data = quantized_weight.get_int8_data(); + + // Don't resize down - the buffer may be pre-allocated with aligned size. + // Only resize up if needed. + if (weight_sums_data.size() < static_cast(out_features)) { + weight_sums_data.resize(out_features); + } + + // For each output feature, compute the sum of quantized weights + for (int64_t out_f = 0; out_f < out_features; ++out_f) { + int32_t sum = 0; + for (int64_t elem = 0; elem < elements_per_output_feature; ++elem) { + // Weight indexing depends on the layout: + // For linear: [out_features, in_features] -> out_f * + // elements_per_output_feature + elem For conv2d: [C_out, C_in * K_h * + // K_w] -> out_f * elements_per_output_feature + elem + int64_t weight_idx = out_f * elements_per_output_feature + elem; + sum += static_cast(quantized_weight_data[weight_idx]); + } + weight_sums_data[out_f] = sum; + } +} + +// Compute weight sums for 4D quantized conv2d operations +// Weight layout: [C_out, K_h, K_w, align_up_4(C_in_per_group)] +void compute_weight_sums_4d( + ValueSpec& weight_sums, + const ValueSpec& quantized_weight, + int64_t out_channels, + int64_t kernel_h, + int64_t kernel_w, + int64_t aligned_in_channels) { + auto& weight_sums_data = weight_sums.get_int32_data(); + auto& quantized_weight_data = quantized_weight.get_int8_data(); + + weight_sums_data.resize(out_channels); + + // For each output channel, compute the sum of quantized weights + for (int64_t out_c = 0; out_c < out_channels; ++out_c) { + int32_t sum = 0; + + for (int64_t kh = 0; kh < kernel_h; ++kh) { + for (int64_t kw = 0; kw < kernel_w; ++kw) { + for (int64_t in_c = 0; in_c < aligned_in_channels; ++in_c) { + // Weight indexing: [out_c, kh, kw, in_c] + int64_t weight_idx = + out_c * (kernel_h * kernel_w * aligned_in_channels) + + kh * (kernel_w * aligned_in_channels) + kw * aligned_in_channels + + in_c; + sum += static_cast(quantized_weight_data[weight_idx]); + } + } + } + + weight_sums_data[out_c] = sum; + } +} + +// Helper function to unpack 4-bit values from uint8 (same as in +// q4gsw_linear.cpp) +std::pair unpack_4bit_utils(uint8_t packed) { + // Extract lower 4 bits and upper 4 bits + int8_t lower = packed & 0x0F; + int8_t upper = (packed >> 4) & 0x0F; + + // Subtract 8 from unpacked 4-bit values + lower -= 8; + upper -= 8; + + return std::make_pair(lower, upper); +} + +// Compute weight sums for 4-bit group symmetric quantized weights +void compute_weight_sums_4bit_grouped( + ValueSpec& weight_sums, + const ValueSpec& quantized_weight, + int64_t num_groups, + int64_t out_features, + int64_t group_size) { + auto& weight_sums_data = weight_sums.get_int32_data(); + auto& quantized_weight_data = quantized_weight.get_uint8_data(); + + // Resize to [num_groups, out_features] + weight_sums_data.resize(num_groups * out_features); + + // For each group and each output feature, compute the sum of quantized + // weights in that group + for (int64_t group_idx = 0; group_idx < num_groups; ++group_idx) { + for (int64_t out_f = 0; out_f < out_features; ++out_f) { + int32_t sum = 0; + + // Sum weights for this group and output feature + for (int64_t in_group = 0; in_group < group_size; ++in_group) { + int64_t in_f = group_idx * group_size + in_group; + + // Get packed weight value - weight matrix is [N, K/2] + int64_t weight_idx = + out_f * ((num_groups * group_size) / 2) + (in_f / 2); + uint8_t packed_weight = quantized_weight_data[weight_idx]; + + // Unpack 4-bit weight + auto unpacked = unpack_4bit_utils(packed_weight); + int8_t weight_4bit = (in_f % 2 == 0) ? unpacked.first : unpacked.second; + + sum += static_cast(weight_4bit); + } + + // Store sum for this group and output feature + int64_t sums_idx = group_idx * out_features + out_f; + weight_sums_data[sums_idx] = sum; + } + } +} + +} // namespace prototyping +} // namespace vulkan +} // namespace executorch diff --git a/backends/vulkan/test/ops/benchmark/framework/weight_utils.h b/backends/vulkan/test/ops/benchmark/framework/weight_utils.h new file mode 100644 index 00000000000..7c9319703b9 --- /dev/null +++ b/backends/vulkan/test/ops/benchmark/framework/weight_utils.h @@ -0,0 +1,52 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. +// All rights reserved. +// +// This source code is licensed under the BSD-style license found in the +// LICENSE file in the root directory of this source tree. + +#pragma once + +#include + +#include "value_spec.h" + +#include + +namespace executorch { +namespace vulkan { +namespace prototyping { + +ValueRef quantized_weights_canvas( + ComputeGraph& graph, + const ValueRef weight_ref); + +ValueRef float_tensor_canvas(ComputeGraph& graph, const ValueRef weight_ref); + +// Compute weight sums for quantized operations (linear and convolution) +void compute_weight_sums( + ValueSpec& weight_sums, + const ValueSpec& quantized_weight, + int64_t out_features, + int64_t elements_per_output_feature); + +// Compute weight sums for 4D quantized conv2d operations +// Weight layout: [C_out, K_h, K_w, align_up_4(C_in_per_group)] +void compute_weight_sums_4d( + ValueSpec& weight_sums, + const ValueSpec& quantized_weight, + int64_t out_channels, + int64_t kernel_h, + int64_t kernel_w, + int64_t aligned_in_channels); + +// Compute weight sums for 4-bit group symmetric quantized weights +void compute_weight_sums_4bit_grouped( + ValueSpec& weight_sums, + const ValueSpec& quantized_weight, + int64_t num_groups, + int64_t out_features, + int64_t group_size); + +} // namespace prototyping +} // namespace vulkan +} // namespace executorch diff --git a/backends/vulkan/test/ops/benchmark/targets.bzl b/backends/vulkan/test/ops/benchmark/targets.bzl new file mode 100644 index 00000000000..ec70bb549f1 --- /dev/null +++ b/backends/vulkan/test/ops/benchmark/targets.bzl @@ -0,0 +1,129 @@ +load("@fbsource//xplat/executorch/build:runtime_wrapper.bzl", "runtime") +load( + "@fbsource//xplat/executorch/backends/vulkan:targets.bzl", + "get_platforms", + "vulkan_spv_shader_lib", +) + +def define_custom_op_test_binary(custom_op_name, subdir, extra_deps = [], include_torch = False): + deps_list = [ + ":prototyping_utils", + ":operator_implementations", + ":custom_ops_shaderlib", + "//executorch/backends/vulkan:vulkan_graph_runtime", + ] + ([runtime.external_dep_location("libtorch")] if include_torch else []) + extra_deps + + runtime.cxx_binary( + name = custom_op_name, + srcs = [ + "{}/{}.cpp".format(subdir, custom_op_name), + ], + platforms = get_platforms(), + define_static_target = False, + deps = deps_list, + ) + +def define_common_targets(is_fbcode = False): + if is_fbcode: + return + + # Shader library from GLSL files + runtime.filegroup( + name = "custom_ops_shaders", + srcs = native.glob([ + "test_ops/glsl/**/*", + ]), + visibility = ["PUBLIC"], + ) + + vulkan_spv_shader_lib( + name = "custom_ops_shaderlib", + spv_filegroups = { + ":custom_ops_shaders": "test_ops/glsl", + }, + is_fbcode = is_fbcode, + ) + + # Prototyping utilities library + runtime.cxx_library( + name = "prototyping_utils", + srcs = [ + "framework/config.cpp", + "framework/conv2d_utils.cpp", + "framework/results.cpp", + "framework/runner.cpp", + "framework/test_case.cpp", + "framework/value_spec.cpp", + "framework/weight_utils.cpp", + ], + headers = [ + "framework/config.h", + "framework/conv2d_utils.h", + "framework/labels.h", + "framework/results.h", + "framework/runner.h", + "framework/test_case.h", + "framework/utils.h", + "framework/value_spec.h", + "framework/weight_utils.h", + ], + exported_headers = [ + "framework/config.h", + "framework/conv2d_utils.h", + "framework/labels.h", + "framework/results.h", + "framework/runner.h", + "framework/test_case.h", + "framework/utils.h", + "framework/value_spec.h", + "framework/weight_utils.h", + ], + platforms = get_platforms(), + deps = [ + "//executorch/backends/vulkan:vulkan_graph_runtime", + ], + visibility = ["PUBLIC"], + ) + + # Operator implementations library + runtime.cxx_library( + name = "operator_implementations", + srcs = native.glob([ + "test_ops/impl/*.cpp", + ]), + platforms = get_platforms(), + deps = [ + "//executorch/backends/vulkan:vulkan_graph_runtime", + ":custom_ops_shaderlib", + ], + visibility = ["PUBLIC"], + link_whole = True, + ) + + + + + define_custom_op_test_binary("test_add", "cases/add") + define_custom_op_test_binary("test_q8csw_linear", "cases/q8csw_linear") + define_custom_op_test_binary("test_q8csw_conv2d", "cases/q8csw_conv2d") + define_custom_op_test_binary("test_choose_qparams_per_row", "cases/choose_qparams_per_row") + define_custom_op_test_binary("test_q4gsw_linear", "cases/q4gsw_linear") + define_custom_op_test_binary("test_q8ta_qdq", "cases/q8ta/qdq") + define_custom_op_test_binary("test_q8ta_clone", "cases/q8ta/clone") + define_custom_op_test_binary("test_q8ta_binary", "cases/q8ta/binary") + define_custom_op_test_binary("test_q8ta_conv2d", "cases/q8ta/conv2d") + define_custom_op_test_binary("test_q8ta_conv2d_pw", "cases/q8ta/conv2d") + define_custom_op_test_binary("test_q8ta_conv2d_dw", "cases/q8ta/conv2d") + define_custom_op_test_binary("test_q8ta_linear", "cases/q8ta/linear") + define_custom_op_test_binary("test_q8ta_conv2d_transposed", "cases/q8ta/conv2d_transposed") + define_custom_op_test_binary("test_q8ta_pixel_shuffle", "cases/q8ta/pixel_shuffle") + define_custom_op_test_binary("test_q8ta_unary", "cases/q8ta/unary") + define_custom_op_test_binary("test_mm", "cases/mm") + define_custom_op_test_binary("test_conv2d", "cases/conv2d") + define_custom_op_test_binary("test_conv2d_pw", "cases/conv2d") + define_custom_op_test_binary("test_conv2d_dw", "cases/conv2d") + define_custom_op_test_binary("test_embedding_q4gsw", "cases/embedding_q4gsw") + define_custom_op_test_binary("test_conv1d_pw", "cases/conv1d") + define_custom_op_test_binary("test_conv1d_dw", "cases/conv1d") + define_custom_op_test_binary("test_fpa_q4gsw_linear", "cases/q4gsw_linear") + define_custom_op_test_binary("test_sdpa", "cases/sdpa") diff --git a/backends/vulkan/test/custom_ops/glsl/add.yaml b/backends/vulkan/test/ops/benchmark/test_ops/glsl/add.yaml similarity index 100% rename from backends/vulkan/test/custom_ops/glsl/add.yaml rename to backends/vulkan/test/ops/benchmark/test_ops/glsl/add.yaml diff --git a/backends/vulkan/test/custom_ops/glsl/add_buffer.glsl b/backends/vulkan/test/ops/benchmark/test_ops/glsl/add_buffer.glsl similarity index 100% rename from backends/vulkan/test/custom_ops/glsl/add_buffer.glsl rename to backends/vulkan/test/ops/benchmark/test_ops/glsl/add_buffer.glsl diff --git a/backends/vulkan/test/custom_ops/glsl/add_texture.glsl b/backends/vulkan/test/ops/benchmark/test_ops/glsl/add_texture.glsl similarity index 100% rename from backends/vulkan/test/custom_ops/glsl/add_texture.glsl rename to backends/vulkan/test/ops/benchmark/test_ops/glsl/add_texture.glsl diff --git a/backends/vulkan/test/custom_ops/glsl/float_canvas.glsl b/backends/vulkan/test/ops/benchmark/test_ops/glsl/float_canvas.glsl similarity index 100% rename from backends/vulkan/test/custom_ops/glsl/float_canvas.glsl rename to backends/vulkan/test/ops/benchmark/test_ops/glsl/float_canvas.glsl diff --git a/backends/vulkan/test/custom_ops/glsl/packed_int32_canvas_buffer.glsl b/backends/vulkan/test/ops/benchmark/test_ops/glsl/packed_int32_canvas_buffer.glsl similarity index 100% rename from backends/vulkan/test/custom_ops/glsl/packed_int32_canvas_buffer.glsl rename to backends/vulkan/test/ops/benchmark/test_ops/glsl/packed_int32_canvas_buffer.glsl diff --git a/backends/vulkan/test/custom_ops/glsl/packed_int32_canvas_texture3d.glsl b/backends/vulkan/test/ops/benchmark/test_ops/glsl/packed_int32_canvas_texture3d.glsl similarity index 100% rename from backends/vulkan/test/custom_ops/glsl/packed_int32_canvas_texture3d.glsl rename to backends/vulkan/test/ops/benchmark/test_ops/glsl/packed_int32_canvas_texture3d.glsl diff --git a/backends/vulkan/test/custom_ops/glsl/q4gsw_linear_gemv__w_4x8.glsl b/backends/vulkan/test/ops/benchmark/test_ops/glsl/q4gsw_linear_gemv__w_4x8.glsl similarity index 100% rename from backends/vulkan/test/custom_ops/glsl/q4gsw_linear_gemv__w_4x8.glsl rename to backends/vulkan/test/ops/benchmark/test_ops/glsl/q4gsw_linear_gemv__w_4x8.glsl diff --git a/backends/vulkan/test/custom_ops/glsl/q4gsw_linear_gemv__w_4x8.yaml b/backends/vulkan/test/ops/benchmark/test_ops/glsl/q4gsw_linear_gemv__w_4x8.yaml similarity index 100% rename from backends/vulkan/test/custom_ops/glsl/q4gsw_linear_gemv__w_4x8.yaml rename to backends/vulkan/test/ops/benchmark/test_ops/glsl/q4gsw_linear_gemv__w_4x8.yaml diff --git a/backends/vulkan/test/custom_ops/impl/AddPrototype.cpp b/backends/vulkan/test/ops/benchmark/test_ops/impl/AddPrototype.cpp similarity index 100% rename from backends/vulkan/test/custom_ops/impl/AddPrototype.cpp rename to backends/vulkan/test/ops/benchmark/test_ops/impl/AddPrototype.cpp diff --git a/backends/vulkan/test/custom_ops/impl/TestConv1dDW.cpp b/backends/vulkan/test/ops/benchmark/test_ops/impl/TestConv1dDW.cpp similarity index 100% rename from backends/vulkan/test/custom_ops/impl/TestConv1dDW.cpp rename to backends/vulkan/test/ops/benchmark/test_ops/impl/TestConv1dDW.cpp diff --git a/backends/vulkan/test/custom_ops/impl/TestConv1dPW.cpp b/backends/vulkan/test/ops/benchmark/test_ops/impl/TestConv1dPW.cpp similarity index 100% rename from backends/vulkan/test/custom_ops/impl/TestConv1dPW.cpp rename to backends/vulkan/test/ops/benchmark/test_ops/impl/TestConv1dPW.cpp diff --git a/backends/vulkan/test/custom_ops/impl/TestConv2d.cpp b/backends/vulkan/test/ops/benchmark/test_ops/impl/TestConv2d.cpp similarity index 100% rename from backends/vulkan/test/custom_ops/impl/TestConv2d.cpp rename to backends/vulkan/test/ops/benchmark/test_ops/impl/TestConv2d.cpp diff --git a/backends/vulkan/test/custom_ops/impl/TestConv2dDw.cpp b/backends/vulkan/test/ops/benchmark/test_ops/impl/TestConv2dDw.cpp similarity index 100% rename from backends/vulkan/test/custom_ops/impl/TestConv2dDw.cpp rename to backends/vulkan/test/ops/benchmark/test_ops/impl/TestConv2dDw.cpp diff --git a/backends/vulkan/test/custom_ops/impl/TestConv2dPw.cpp b/backends/vulkan/test/ops/benchmark/test_ops/impl/TestConv2dPw.cpp similarity index 100% rename from backends/vulkan/test/custom_ops/impl/TestConv2dPw.cpp rename to backends/vulkan/test/ops/benchmark/test_ops/impl/TestConv2dPw.cpp diff --git a/backends/vulkan/test/custom_ops/impl/TestFpaQ4gswLinear.cpp b/backends/vulkan/test/ops/benchmark/test_ops/impl/TestFpaQ4gswLinear.cpp similarity index 99% rename from backends/vulkan/test/custom_ops/impl/TestFpaQ4gswLinear.cpp rename to backends/vulkan/test/ops/benchmark/test_ops/impl/TestFpaQ4gswLinear.cpp index 2da2cd6ceb7..4b41391b2cd 100644 --- a/backends/vulkan/test/custom_ops/impl/TestFpaQ4gswLinear.cpp +++ b/backends/vulkan/test/ops/benchmark/test_ops/impl/TestFpaQ4gswLinear.cpp @@ -212,10 +212,10 @@ LocalWorkGroup pick_q4gsw_coop_lwg( } // Spec-constant LWG values for the q4gsw_linear_gemv__w_4x8_nc[_nosg] shaders -// (now shipped from test/custom_ops/glsl/). The sg variant pins subgroupSize -// to 64 via VK_EXT_subgroup_size_control; the nosg variant uses shared-mem -// reduction so the lane count is purely an LWG choice. Both use 4 subgroups -// per workgroup. +// (now shipped from test/ops/benchmark/test_ops/glsl/). The sg variant pins +// subgroupSize to 64 via VK_EXT_subgroup_size_control; the nosg variant uses +// shared-mem reduction so the lane count is purely an LWG choice. Both use 4 +// subgroups per workgroup. constexpr uint32_t kGemvSubgroupSize = 64u; constexpr uint32_t kGemvNumSubgroups = 4u; diff --git a/backends/vulkan/test/custom_ops/impl/TestMatmulLinear.cpp b/backends/vulkan/test/ops/benchmark/test_ops/impl/TestMatmulLinear.cpp similarity index 100% rename from backends/vulkan/test/custom_ops/impl/TestMatmulLinear.cpp rename to backends/vulkan/test/ops/benchmark/test_ops/impl/TestMatmulLinear.cpp diff --git a/backends/vulkan/test/custom_ops/impl/TestQ8taBinary.cpp b/backends/vulkan/test/ops/benchmark/test_ops/impl/TestQ8taBinary.cpp similarity index 100% rename from backends/vulkan/test/custom_ops/impl/TestQ8taBinary.cpp rename to backends/vulkan/test/ops/benchmark/test_ops/impl/TestQ8taBinary.cpp diff --git a/backends/vulkan/test/custom_ops/impl/TestQ8taClone.cpp b/backends/vulkan/test/ops/benchmark/test_ops/impl/TestQ8taClone.cpp similarity index 100% rename from backends/vulkan/test/custom_ops/impl/TestQ8taClone.cpp rename to backends/vulkan/test/ops/benchmark/test_ops/impl/TestQ8taClone.cpp diff --git a/backends/vulkan/test/custom_ops/impl/TestQ8taConv2d.cpp b/backends/vulkan/test/ops/benchmark/test_ops/impl/TestQ8taConv2d.cpp similarity index 100% rename from backends/vulkan/test/custom_ops/impl/TestQ8taConv2d.cpp rename to backends/vulkan/test/ops/benchmark/test_ops/impl/TestQ8taConv2d.cpp diff --git a/backends/vulkan/test/custom_ops/impl/TestQ8taConv2dTransposed.cpp b/backends/vulkan/test/ops/benchmark/test_ops/impl/TestQ8taConv2dTransposed.cpp similarity index 100% rename from backends/vulkan/test/custom_ops/impl/TestQ8taConv2dTransposed.cpp rename to backends/vulkan/test/ops/benchmark/test_ops/impl/TestQ8taConv2dTransposed.cpp diff --git a/backends/vulkan/test/custom_ops/impl/TestQ8taLinear.cpp b/backends/vulkan/test/ops/benchmark/test_ops/impl/TestQ8taLinear.cpp similarity index 100% rename from backends/vulkan/test/custom_ops/impl/TestQ8taLinear.cpp rename to backends/vulkan/test/ops/benchmark/test_ops/impl/TestQ8taLinear.cpp diff --git a/backends/vulkan/test/custom_ops/impl/TestQ8taPixelShuffle.cpp b/backends/vulkan/test/ops/benchmark/test_ops/impl/TestQ8taPixelShuffle.cpp similarity index 100% rename from backends/vulkan/test/custom_ops/impl/TestQ8taPixelShuffle.cpp rename to backends/vulkan/test/ops/benchmark/test_ops/impl/TestQ8taPixelShuffle.cpp diff --git a/backends/vulkan/test/custom_ops/impl/TestQ8taQuantizeDequantize.cpp b/backends/vulkan/test/ops/benchmark/test_ops/impl/TestQ8taQuantizeDequantize.cpp similarity index 100% rename from backends/vulkan/test/custom_ops/impl/TestQ8taQuantizeDequantize.cpp rename to backends/vulkan/test/ops/benchmark/test_ops/impl/TestQ8taQuantizeDequantize.cpp diff --git a/backends/vulkan/test/custom_ops/impl/TestQ8taUnary.cpp b/backends/vulkan/test/ops/benchmark/test_ops/impl/TestQ8taUnary.cpp similarity index 100% rename from backends/vulkan/test/custom_ops/impl/TestQ8taUnary.cpp rename to backends/vulkan/test/ops/benchmark/test_ops/impl/TestQ8taUnary.cpp diff --git a/backends/vulkan/test/custom_ops/impl/TestSDPA.cpp b/backends/vulkan/test/ops/benchmark/test_ops/impl/TestSDPA.cpp similarity index 100% rename from backends/vulkan/test/custom_ops/impl/TestSDPA.cpp rename to backends/vulkan/test/ops/benchmark/test_ops/impl/TestSDPA.cpp