diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 707496026..ea2f2dbf2 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -15,6 +15,13 @@ This file is the canonical instruction set for Copilot behavior in this reposito - Use source-of-truth files for mutable details. - Do not invent or hardcode versions/flags/matrices. - Avoid `std::iostream` in performance-critical headers. +- Add a file only where the nearest `AGENTS.md` purpose line covers its kind and its role; otherwise it belongs in the directory whose purpose line matches. +- A comment states why the code has to be this way and what breaks if it changes, not what the line does; a documentation block on a public entity states that entity's contract. When a change alters code a comment describes, fix or delete that comment in the same change. +- Where code encodes a constraint a reader cannot infer — a platform-specific guard, a derived numeric bound, a warning suppression — comment the reason and what breaks without it. Standard idioms such as a floating-point tolerance need no justification. +- Delete code a change makes dead rather than committing it disabled: commented-out statements, unused variables, branches the change makes unreachable. A conditional-compilation path is not dead merely because no CI job selects it. Code deliberately left inert carries a comment saying why. +- A path that cannot satisfy a request fails with a diagnostic naming the unsupported input and the bound or set it violated, rather than substituting a different configuration; a preprocessor or dispatch chain ends in a catch-all that errors instead of assuming the last case holds. +- Compute a value once where every use can see it rather than repeating the computation: an expression identical in both arms of a conditional belongs before the branch, and one that does not depend on the iteration belongs before the loop or per-element closure. A repeated copy costs on every pass and drifts when only one of them is edited. +- Derive a computed limit or estimate from the quantity that actually bounds it — not a request-shaped parameter that happens to be in scope, and not a counter that a second, larger counter can exceed. Where the true bound cannot be computed, document the assumption the estimate makes rather than leaving it implicit; a bound built on the wrong quantity is wrong in the direction nobody checks. ## Contribution expectations - Preserve backward compatibility for public API (`include/svs/`) diff --git a/.github/instructions/bindings.instructions.md b/.github/instructions/bindings.instructions.md new file mode 100644 index 000000000..95dc375c6 --- /dev/null +++ b/.github/instructions/bindings.instructions.md @@ -0,0 +1,9 @@ +--- +applyTo: "bindings/**" +--- + +# Bindings Instructions for GitHub Copilot + +- A binding reports failure through its own error channel and lets nothing escape across the boundary: an entry point that cannot propagate an exception is marked `noexcept` and returns its error type. Each binding has its own channel — do not assume another's. +- A binding does not restate a default, policy, or limit the core library already selects; pass the core's own sentinel or value through, so the binding cannot drift from it. +- Give a new member of an existing family the convention its siblings already follow: how a default or "unset" is signalled, which type carries a buffer, where an output argument sits. A divergence that still compiles changes behaviour at call sites that never opted into it. diff --git a/.github/instructions/build-system.instructions.md b/.github/instructions/build-system.instructions.md index f54db4965..185d768c2 100644 --- a/.github/instructions/build-system.instructions.md +++ b/.github/instructions/build-system.instructions.md @@ -1,10 +1,11 @@ --- -applyTo: "{CMakeLists.txt,cmake/**}" +applyTo: "{CMakeLists.txt,cmake/**,.github/workflows/**}" --- # Build System Instructions for GitHub Copilot -- Use existing option/target patterns; avoid introducing parallel build paths. +- Use existing option/target patterns; avoid introducing parallel build paths. Before adding a compile flag, option, or CI setup step, check whether an existing one already covers the condition; where the same block genuinely exists in more than one file, either change every copy or factor it into one included file. - Keep configuration values referenced from existing CMake modules. +- Return a computed value another module consumes from a `function()` via `PARENT_SCOPE`, or publish it as a cache entry; a bare `set()` at file scope is not an interface, and a later module setting the same name collides with it silently. - Do not hardcode versions/toolchain assumptions in instructions or comments. - For behavior changes, suggest updating/validating CI workflows in `.github/workflows/`. diff --git a/.github/instructions/ci.instructions.md b/.github/instructions/ci.instructions.md new file mode 100644 index 000000000..255ff5a72 --- /dev/null +++ b/.github/instructions/ci.instructions.md @@ -0,0 +1,7 @@ +--- +applyTo: "{.github/workflows/**,.github/scripts/**,**/conda-recipe/**,docker/**}" +--- + +# CI and Packaging Instructions for GitHub Copilot + +- Fetch a dependency from its canonical upstream, not a personal fork or feature branch. Where a fork is genuinely still required, name in a comment the upstream pull request or issue that will retire it; an unmarked fork is indistinguishable from an abandoned workaround, and collapsing such a conditional as cleanup silently drops the coverage the fork existed for. diff --git a/.github/instructions/documentation.instructions.md b/.github/instructions/documentation.instructions.md new file mode 100644 index 000000000..6bfbf7118 --- /dev/null +++ b/.github/instructions/documentation.instructions.md @@ -0,0 +1,7 @@ +--- +applyTo: "{bindings/**/*.md,**/docs/**/*.md,README.md}" +--- + +# Documentation Instructions for GitHub Copilot + +- State a policy, requirement, or API description in one place and reference it from the others. A second copy silently drops the parts that matter — a compatibility contract, a constraint — and then diverges from the original. diff --git a/.github/instructions/library-code.instructions.md b/.github/instructions/library-code.instructions.md new file mode 100644 index 000000000..ad6cb5c42 --- /dev/null +++ b/.github/instructions/library-code.instructions.md @@ -0,0 +1,13 @@ +--- +applyTo: "include/svs/**" +--- + +# Library Code Instructions for GitHub Copilot + +- Emit diagnostics through `svs::logging`, never to stdout or stderr. An API whose declared purpose is writing a report to a caller-supplied stream is not a diagnostic and is exempt. +- Document every parameter of a public entity, including what a default resolves to and which level of a nested structure it governs. A placeholder standing for "let the library choose" is used only where the default really is computed at call time, and says so; where the default is a fixed value, state that value. +- Do not widen a type's public surface so another layer can compute something from its internals; the layer that owns a quantity owns the operation on it. Forwarding to an owned member is fine. +- A helper, cast, or constant that more than one component needs belongs in one shared header rather than copied into each user, and a defaulted parameter is preferred over a near-duplicate overload. A type more than one component references is declared where all of them can reach it, not nested inside the first one that needed it. +- A conditional-compilation guard is a claim about what the code inside it depends on, so make the condition exactly that: not a narrower feature flag standing in for the broader platform, and not a higher instruction-set level than the surrounding path targets. Deleting such a guard removes a configuration; do not delete one as cleanup without confirming nothing builds that way. +- Name an entity for the role it plays, not the value it holds or the one algorithm that currently uses it. +- Express a requirement where the compiler enforces it — a `concept` on a template parameter, `constexpr` for a value fixed at compile time — rather than as a runtime check or a comment. diff --git a/.github/instructions/multi-arch.instructions.md b/.github/instructions/multi-arch.instructions.md new file mode 100644 index 000000000..8acd88b81 --- /dev/null +++ b/.github/instructions/multi-arch.instructions.md @@ -0,0 +1,7 @@ +--- +applyTo: "include/svs/multi-arch/**" +--- + +# Multi-Arch Instructions for GitHub Copilot + +- Include the header that defines any macro which expands into a translation unit's own declarations or definitions. This covers code-generating macros, not inline annotations such as a force-inline, unused, or forwarding marker. A generating macro whose defining header is absent hides what the file actually defines, so its contents cannot be determined by reading the file alone. diff --git a/.github/instructions/testing.instructions.md b/.github/instructions/testing.instructions.md index 593df1f4e..f41e3057a 100644 --- a/.github/instructions/testing.instructions.md +++ b/.github/instructions/testing.instructions.md @@ -1,10 +1,12 @@ --- -applyTo: "tests/**" +applyTo: "{tests/**,bindings/*/tests/**}" --- # Testing Instructions for GitHub Copilot - Add regression tests for bug fixes. -- Keep tests deterministic and scoped to changed behavior. -- Prefer smallest test surface that proves correctness. +- Keep tests deterministic. +- Prefer the smallest test surface that proves correctness for a given variant, then cover every variant the subject documents — each index type, each dataset or quantization kind the shared reference helpers accept — rather than the default alone. +- Drive the subject into the state under test rather than only calling the API that would reach it: enough input to cross a batch boundary for per-batch behaviour, a mutation before the save/load round trip for a structure that supports mutation. +- A new test goes through the existing shared fixture or parametrized helper for its subject rather than reimplementing the setup. Where no helper covers the subject, or where reusing one would defeat what the test asserts, bespoke setup is correct. - Refer to repository test/build configuration as source of truth rather than duplicating values. diff --git a/.github/scripts/check_dispatch_surface.sh b/.github/scripts/check_dispatch_surface.sh new file mode 100755 index 000000000..6583ed6ef --- /dev/null +++ b/.github/scripts/check_dispatch_surface.sh @@ -0,0 +1,111 @@ +#!/bin/bash +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Runs cmake/validate-dispatch-surface.cmake over the fixtures in +# tests/cmake/dispatch-surface and checks each verdict. +# +# valid-*.cmake must be accepted. +# invalid-*.cmake must be rejected, with a message containing the substring +# given by that fixture's `# EXPECT-ERROR:` line. +# +# Needs nothing but cmake -- no compiler, no dependencies, no build directory. + +set -uo pipefail + +root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd) +fixtures="${root}/tests/cmake/dispatch-surface" +validator="${root}/cmake/validate-dispatch-surface.cmake" +x86_src_dir="${root}/include/svs/multi-arch/x86" + +if [[ ! -d ${fixtures} ]]; then + echo "no fixture directory: ${fixtures}" >&2 + exit 1 +fi + +# The default declaration must itself be valid -- checked as its own case so that +# a broken default is reported here rather than only at configure time. +run_validator() { + cmake "-DSVS_DISPATCH_SURFACE_FILE=$1" "-DSVS_X86_SRC_DIR=${x86_src_dir}" \ + -P "${validator}" 2>&1 +} + +# CMake indents and line-wraps error text, so compare against a whitespace- +# collapsed copy of the output. +flatten() { tr '\n' ' ' | tr -s '[:space:]' ' '; } + +failures=0 +checked=0 + +check_accepted() { + local fixture=$1 name=$2 output status + output=$(run_validator "${fixture}") + status=$? + if ((status != 0)); then + echo "FAIL ${name}: expected to be accepted, but validation failed:" >&2 + echo "${output}" | sed 's/^/ /' >&2 + ((failures++)) + else + echo "ok ${name}: accepted" + fi + ((checked++)) +} + +check_rejected() { + local fixture=$1 name=$2 expected output status + expected=$(sed -n 's/^# EXPECT-ERROR: *//p' "${fixture}") + if [[ -z ${expected} ]]; then + echo "FAIL ${name}: fixture has no '# EXPECT-ERROR:' line" >&2 + ((failures++)) + ((checked++)) + return + fi + + output=$(run_validator "${fixture}") + status=$? + if ((status == 0)); then + echo "FAIL ${name}: expected rejection, but validation succeeded" >&2 + ((failures++)) + elif [[ $(printf '%s' "${output}" | flatten) != *"${expected}"* ]]; then + echo "FAIL ${name}: rejected, but not for the stated reason." >&2 + echo " expected: ${expected}" >&2 + echo "${output}" | sed 's/^/ actual: /' >&2 + ((failures++)) + else + echo "ok ${name}: rejected (${expected})" + fi + ((checked++)) +} + +check_accepted "${root}/cmake/dispatch-surface.cmake" "dispatch-surface.cmake (default)" + +for fixture in "${fixtures}"/*.cmake; do + name=$(basename "${fixture}") + case ${name} in + valid-*) check_accepted "${fixture}" "${name}" ;; + invalid-*) check_rejected "${fixture}" "${name}" ;; + *) + echo "FAIL ${name}: fixture name must start with valid- or invalid-" >&2 + ((failures++)) + ((checked++)) + ;; + esac +done + +echo +if ((failures != 0)); then + echo "${failures} of ${checked} dispatch-surface checks failed" >&2 + exit 1 +fi +echo "all ${checked} dispatch-surface checks passed" diff --git a/.github/workflows/build-linux.yml b/.github/workflows/build-linux.yml index c57da13eb..a63d82a48 100644 --- a/.github/workflows/build-linux.yml +++ b/.github/workflows/build-linux.yml @@ -116,3 +116,38 @@ jobs: CTEST_OUTPUT_ON_FAILURE: 1 working-directory: ${{ runner.temp }}/build/examples/cpp run: ctest -C ${{ matrix.build_type }} + + # SVS_NO_AVX512=YES is a supported configuration (cmake/options.cmake) that no + # other leg exercises -- the matrix above hardcodes NO -- and it changes which + # instructions each ISA level's object is allowed to contain, so it gets its + # own minimal job rather than a silently-untested option. + dispatch-no-avx512: + name: dispatch surface, SVS_NO_AVX512=YES + runs-on: ubuntu-22.04 + steps: + - uses: actions/checkout@v6 + + - name: Configure build + working-directory: ${{ runner.temp }} + env: + CXX: g++-11 + CC: gcc-11 + TEMP_WORKSPACE: ${{ runner.temp }} + run: | + cmake -B${TEMP_WORKSPACE}/build -S${GITHUB_WORKSPACE} \ + -DCMAKE_BUILD_TYPE=Release \ + -DSVS_BUILD_TESTS=YES \ + -DSVS_NO_AVX512=YES + + # Only the dispatch surface probes, not the full suite: this job exists to + # catch a regression in the instruction-budget check, not to duplicate the + # matrix build above. + - name: Build dispatch surface probes + working-directory: ${{ runner.temp }}/build + run: make -j$(nproc) svs_x86_objects link_probe entry_probe + + - name: Run dispatch surface checks + env: + CTEST_OUTPUT_ON_FAILURE: 1 + working-directory: ${{ runner.temp }}/build/tests/multi-arch + run: ctest -R dispatch diff --git a/.github/workflows/dispatch-surface.yml b/.github/workflows/dispatch-surface.yml new file mode 100644 index 000000000..05f1d8af3 --- /dev/null +++ b/.github/workflows/dispatch-surface.yml @@ -0,0 +1,87 @@ +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# The set of distance kernels compiled ahead of time is declared once, in +# cmake/dispatch-surface.cmake, and generated from there. Two things have to stay +# true for that to be worth anything: the declaration must be checked rather than +# trusted, and it must be genuinely configurable -- a knob nobody turns is a knob +# that quietly stops working. + +name: Dispatch Surface + +on: + push: + branches: + - main + pull_request: + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }} @ ${{ github.event.pull_request.head.label || github.head_ref || github.ref }} + cancel-in-progress: true + +jobs: + declaration: + name: declaration is checked, default surface configures + runs-on: ubuntu-22.04 + steps: + - uses: actions/checkout@v6 + + # Every fixture in tests/cmake/dispatch-surface, plus the default + # declaration. Needs nothing but cmake. + - name: Accept and reject declarations + run: .github/scripts/check_dispatch_surface.sh + + # The header is generated into the build tree only, so this just confirms the + # default declaration configures; the non-default-surface job below covers an + # override. + - name: Configure with the default surface + run: | + cmake -B "${{ runner.temp }}/build" -S "${GITHUB_WORKSPACE}" \ + -DCMAKE_BUILD_TYPE=Release \ + -DSVS_BUILD_TESTS=NO \ + -DSVS_BUILD_BINARIES=NO + + non-default-surface: + name: builds and tests with a non-default surface + runs-on: ubuntu-22.04 + steps: + - uses: actions/checkout@v6 + + # valid-reduced.cmake shares no extent with the default declaration, so a + # build that silently used the default surface instead of the override would + # fail to compile rather than pass by accident. + - name: Configure + run: | + cmake -B "${{ runner.temp }}/build" -S "${GITHUB_WORKSPACE}" \ + -DCMAKE_BUILD_TYPE=Release \ + -DSVS_BUILD_TESTS=YES \ + -DSVS_BUILD_BINARIES=NO \ + -DSVS_DISPATCH_SURFACE_FILE="${GITHUB_WORKSPACE}/tests/cmake/dispatch-surface/valid-reduced.cmake" + + - name: Build + working-directory: ${{ runner.temp }}/build + run: make -j$(nproc) + + # Correctness must not depend on which extents have a fixed-extent kernel: + # an extent without one is served by the svs::Dynamic kernel instead. The + # long-running tests are covered by the default-surface build. + - name: Run tests + env: + CTEST_OUTPUT_ON_FAILURE: 1 + working-directory: ${{ runner.temp }}/build/tests + run: ctest -C Release -LE long diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index baf697990..df9502bd3 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -34,3 +34,21 @@ repos: args: [--markdown-linebreak-ext=md] exclude: .*\.svg$ - id: mixed-line-ending + + - repo: local + hooks: + # Cheap enough to run on every touch of the declaration, and it needs + # nothing but cmake -- no compiler, no build directory. + - id: dispatch-surface + name: dispatch surface declaration + entry: .github/scripts/check_dispatch_surface.sh + language: script + pass_filenames: false + # (?x) ignores the layout below; without it the alternation is one + # unreadable line and every edit to it risks silently matching nothing. + files: | + (?x)^( + cmake/(dispatch-surface|validate-dispatch-surface)\.cmake + |tests/cmake/dispatch-surface/.*\.cmake + |\.github/scripts/check_dispatch_surface\.sh + )$ diff --git a/CMakeLists.txt b/CMakeLists.txt index 62377bef9..1590953a8 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -135,6 +135,16 @@ install( DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}" FILES_MATCHING PATTERN "*.h" ) + +# The generated dispatch-surface header is not part of the include/ tree the +# directory install above copies, so this is declared after it purely to keep +# related install() calls together, not to win a conflict. +if(DEFINED SVS_GENERATED_DISPATCH_HEADER) + install( + FILES "${SVS_GENERATED_DISPATCH_HEADER}" + DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}/svs/core/distance" + ) +endif() install( EXPORT svs-targets NAMESPACE "svs::" diff --git a/cmake/AGENTS.md b/cmake/AGENTS.md index f6f81aecb..b853b8923 100644 --- a/cmake/AGENTS.md +++ b/cmake/AGENTS.md @@ -9,7 +9,8 @@ Build modules, dependency wiring, and feature toggles. ## Intel-specific modules - **`cmake/mkl.cmake`:** MKL linkage (static vs dynamic threading). Do not hardcode MKL versions. When changing linkage mode, validate threading behavior in tests. -- **`cmake/multi-arch.cmake`:** AVX-512 / SIMD ISA dispatch. Do not hardcode `-march` or ISA flags outside this file. Changes must align with `include/svs/multi-arch/` runtime dispatch code. +- **`cmake/multi-arch.cmake`:** AVX-512 / SIMD ISA dispatch wiring. Changes must align with `include/svs/multi-arch/` runtime dispatch code. +- **`cmake/dispatch-surface.cmake`:** The declared x86 dispatch surface — fixed extents, ISA levels, and their `-march` budgets. Edit it, never the generated header (`/generated/include/svs/core/distance/dispatch_surface.h`, build-tree only, not committed). `tests/multi-arch/dispatch-checks/` holds the ctest checkers that hold the built binary to this declaration. - **`cmake/numa.cmake`:** NUMA-aware memory allocation. Respect NUMA topology assumptions in performance-critical code. - **`cmake/openmp.cmake`:** Threading model. Do not assume specific OpenMP version or runtime without checking source-of-truth. diff --git a/cmake/dispatch-levels.cmake b/cmake/dispatch-levels.cmake new file mode 100644 index 000000000..bc2da39cc --- /dev/null +++ b/cmake/dispatch-levels.cmake @@ -0,0 +1,56 @@ +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +include_guard(GLOBAL) + +## A function captures the policy stack where it is defined, so CMP0007 is set here rather +## than by the caller: under OLD an empty field vanishes and a row with one is called malformed. +cmake_policy(PUSH) +cmake_policy(SET CMP0007 NEW) + +# Parses an ISA level spec from SVS_ISA_LEVELS and returns the three fields. +# The spec format is ||. +function(svs_parse_isa_level spec out_level out_arch out_infix) + string(REPLACE "|" ";" fields "${spec}") + list(LENGTH fields n) + if(NOT n EQUAL 3) + message(FATAL_ERROR "Malformed ISA level '${spec}': expected ||.") + endif() + list(GET fields 0 v0) + list(GET fields 1 v1) + list(GET fields 2 v2) + set(${out_level} "${v0}" PARENT_SCOPE) + set(${out_arch} "${v1}" PARENT_SCOPE) + set(${out_infix} "${v2}" PARENT_SCOPE) +endfunction() + +# Parses a translation unit spec from SVS_DISPATCH_TU_SPECS and returns the four fields. +# The spec format is |||. +function(svs_parse_tu_spec spec out_src out_level out_arch out_infix) + string(REPLACE "|" ";" fields "${spec}") + list(LENGTH fields n) + if(NOT n EQUAL 4) + message(FATAL_ERROR "Malformed TU spec '${spec}': expected |||.") + endif() + list(GET fields 0 v0) + list(GET fields 1 v1) + list(GET fields 2 v2) + list(GET fields 3 v3) + set(${out_src} "${v0}" PARENT_SCOPE) + set(${out_level} "${v1}" PARENT_SCOPE) + set(${out_arch} "${v2}" PARENT_SCOPE) + set(${out_infix} "${v3}" PARENT_SCOPE) +endfunction() + +cmake_policy(POP) diff --git a/cmake/dispatch-surface.cmake b/cmake/dispatch-surface.cmake new file mode 100644 index 000000000..db908adff --- /dev/null +++ b/cmake/dispatch-surface.cmake @@ -0,0 +1,118 @@ +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +##### +##### The dispatch surface for the x86 distance kernels. +##### +##### This file is the single place the extent list and the ISA levels are +##### written down. Everything derived from them is generated: +##### +##### - svs/core/distance/dispatch_surface.h, which drives every `extern +##### template` and explicit instantiation, and `supported_dim_list` +##### - the object library each ISA level's translation unit is compiled into, +##### and the instruction budget it is compiled at +##### +##### Edit this file. The header is not in the source tree at all: it is written +##### into the build directory on every configure and installed from there. +##### +##### A build may point somewhere else with -DSVS_DISPATCH_SURFACE_FILE=, +##### in which case only that build tree describes the overridden surface. +##### +##### Bookkeeping +##### +##### To add or remove a fixed extent: +##### Edit `SVS_SUPPORTED_DIMS`. The generated header, every `extern +##### template`, and `supported_dim_list` follow automatically. +##### +##### To add an ISA level: +##### 1. Add a row to `SVS_ISA_LEVELS`. +##### 2. Add a `SVS_TYPE_PAIRS_` list in +##### include/svs/multi-arch/x86/preprocessor.h, listing the element-type +##### pairs that level has kernels for. +##### 3. Add the level's translation unit at +##### include/svs/multi-arch/x86/.cpp. +##### The object library and its compile flags follow from the row here. A +##### level without a type-pair list is a compile error, not an empty +##### instantiation set. +##### +##### To add or remove an element-type pair for a level: +##### Edit that level's `SVS_TYPE_PAIRS_` list in +##### include/svs/multi-arch/x86/preprocessor.h. Not configured here: a type +##### pair exists because an implementation exists for it. +##### +##### To change a level's instruction budget: +##### Edit the middle field of its row in `SVS_ISA_LEVELS`, observing the +##### constraint recorded there. +##### + +# Extents that get their own fixed-extent kernel. +# +# This list is a *performance* choice, not a compatibility one. Any +# dimensionality not listed here still works and is fully supported: it +# dispatches to the `svs::Dynamic` kernel, which takes the length at run time. +# Listing an extent buys a fully unrolled kernel for it, at the cost of one more +# set of instantiations across every ISA level and type pair. +# +# `svs::Dynamic` is required and is appended automatically -- do not list it. +set(SVS_SUPPORTED_DIMS 64 96 100 128 160 200 512 768) + +# Runtime ISA levels. Each has one translation unit, which instantiates every +# extent above at that level. +# +# Only levels with their own translation unit appear here, which is why +# `AVX_AVAILABILITY::NONE` has no row: nothing instantiates its kernels ahead +# of time, so each translation unit that uses them instantiates them itself, at +# its own `-march` -- generic `x86-64` for this project's own build. A row here +# would name a translation unit and an object library that do not exist. +# +# `AVX_AVAILABILITY` mangles positionally, so `NONE` must keep its enumerator +# value despite having no row; renumbering the enumerators is an ABI break. +# +# || +# +# enumerator a value of `svs::distance::AVX_AVAILABILITY` +# instruction budget the -march this level's kernels are compiled to; it must +# match what the level guarantees about the host, because +# anything the compiler is allowed to emit here will run on +# any host that satisfies the level +# TU infix names both the level's translation unit, +# include/svs/multi-arch/x86/.cpp, and the object +# library it is compiled into +# +# Unlike the extent list above, this list is not a knob. A level exists because +# kernels, a translation unit and a runtime check for it exist, so changing it +# means changing code: +# +# - a `SVS_TYPE_PAIRS_` list in +# include/svs/multi-arch/x86/preprocessor.h, saying which element-type pairs +# the level has kernels for +# - the specializations themselves, in the three distance headers +# - a branch in the entry points, and the CPUID check it tests +# +# The generated header defines `SVS_ISA_LEVEL_` for each level listed +# here, which is how the entry points tell a level that is present from one that +# is not; dropping AVX2 or AVX512 from this list is a compile error rather than a +# silent fall back to unvectorized code. +# +# On the instruction budgets in particular: `skylake-avx512` rather than +# `cascadelake` for the AVX512 level, because that level promises AVX-512 F/BW/DQ +# and nothing more. `cascadelake` also enables AVX512-VNNI, which the compiler is +# then free to emit into kernels that run on any AVX512F host -- a Skylake-SP +# among them, where `vpdpbusd` does not exist. VNNI is its own level instead, so +# that the budget and the promise line up. +set(SVS_ISA_LEVELS + "AVX2|haswell|avx2" + "AVX512|skylake-avx512|avx512" + "AVX512_VNNI|cascadelake|avx512_vnni" +) diff --git a/cmake/generate-dispatch-surface.cmake b/cmake/generate-dispatch-surface.cmake new file mode 100644 index 000000000..17a70ce23 --- /dev/null +++ b/cmake/generate-dispatch-surface.cmake @@ -0,0 +1,244 @@ +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +##### +##### Derives the dispatch surface declared in cmake/dispatch-surface.cmake. +##### +##### Defines svs_generate_dispatch_surface(), which produces: +##### /generated/include/svs/core/distance/dispatch_surface.h (build tree only) +##### SVS_DISPATCH_TU_SPECS -- "|||", one per ISA level +##### +##### cmake/multi-arch.cmake calls the function and receives both. +##### + +include_guard(GLOBAL) + +include("${CMAKE_CURRENT_LIST_DIR}/dispatch-levels.cmake") + +# Captured while CMAKE_CURRENT_LIST_DIR unambiguously means this directory, so the +# functions below can reference sibling files after control has passed into them. +set(SVS_DISPATCH_GEN_CMAKE_DIR "${CMAKE_CURRENT_LIST_DIR}") + +set(SVS_DEFAULT_DISPATCH_SURFACE_FILE "${CMAKE_CURRENT_LIST_DIR}/dispatch-surface.cmake") +# Lets validate-dispatch-surface.cmake and check_dispatch_surface.sh point this at a +# tests/cmake/dispatch-surface fixture instead of editing the real declaration. +set(SVS_DISPATCH_SURFACE_FILE "${SVS_DEFAULT_DISPATCH_SURFACE_FILE}" + CACHE FILEPATH + "Declaration of the ahead-of-time distance-kernel dispatch surface" +) + +##### +##### Read and validate the declaration +##### + +# Includes validate-dispatch-surface.cmake, which rejects a malformed declaration +# with FATAL_ERROR and otherwise sets SVS_SUPPORTED_DIMS, SVS_DIM_LIST, +# SVS_DIM_COUNT and SVS_ISA_LEVELS. Also runnable on its own -- see +# .github/scripts/check_dispatch_surface.sh. +function(svs_dispatch_read_declaration surface_file x86_src_dir + out_supported_dims out_dim_list out_dim_count out_isa_levels) + set(SVS_DISPATCH_SURFACE_FILE "${surface_file}") + set(SVS_X86_SRC_DIR "${x86_src_dir}") + include("${SVS_DISPATCH_GEN_CMAKE_DIR}/validate-dispatch-surface.cmake") + set(${out_supported_dims} "${SVS_SUPPORTED_DIMS}" PARENT_SCOPE) + set(${out_dim_list} "${SVS_DIM_LIST}" PARENT_SCOPE) + set(${out_dim_count} "${SVS_DIM_COUNT}" PARENT_SCOPE) + set(${out_isa_levels} "${SVS_ISA_LEVELS}" PARENT_SCOPE) +endfunction() + +##### +##### Build the macro bodies +##### + +# Builds the escaped-newline macro bodies that substitute into the header +# template's @SVS_GEN_*@ placeholders -- see +# cmake/templates/dispatch_surface.h.in for what each one expands into. +function(svs_dispatch_build_expansions dim_list isa_levels + out_dim_loop out_target_loop out_level_loop out_level_defines) + set(dim_loop "\\\n") + foreach(dim IN LISTS dim_list) + string(APPEND dim_loop " M(${dim}) \\\n") + endforeach() + string(APPEND dim_loop " /* end */") + + set(target_loop "\\\n") + set(level_loop "\\\n") + set(level_defines "") + foreach(level_spec IN LISTS isa_levels) + svs_parse_isa_level("${level_spec}" level _ _) + string(APPEND level_loop " M(${level}) \\\n") + string(APPEND level_defines "#define SVS_ISA_LEVEL_${level} 1\n") + foreach(dim IN LISTS dim_list) + string(APPEND target_loop " M(${dim}, ${level}) \\\n") + endforeach() + endforeach() + string(APPEND target_loop " /* end */") + string(APPEND level_loop " /* end */") + string(STRIP "${level_defines}" level_defines) + + set(${out_dim_loop} "${dim_loop}" PARENT_SCOPE) + set(${out_target_loop} "${target_loop}" PARENT_SCOPE) + set(${out_level_loop} "${level_loop}" PARENT_SCOPE) + set(${out_level_defines} "${level_defines}" PARENT_SCOPE) +endfunction() + +##### +##### Derive the translation-unit specs +##### + +# One translation unit per level, committed rather than generated because the +# downstream repository compiles these sources by path (validation checks each +# exists). Also returns the plain level list and the human-readable report lines +# the caller's message(STATUS ...) prints, since both come from this same walk. +function(svs_dispatch_derive_tu_specs isa_levels x86_src_dir + out_tu_specs out_levels out_level_report) + set(tu_specs) + set(levels) + set(level_report) + foreach(level_spec IN LISTS isa_levels) + svs_parse_isa_level("${level_spec}" level arch infix) + list(APPEND tu_specs "${x86_src_dir}/${infix}.cpp|${level}|${arch}|${infix}") + list(APPEND levels "${level}") + list(APPEND level_report "AVX_AVAILABILITY::${level} -march=${arch} ${infix}.cpp") + endforeach() + set(${out_tu_specs} "${tu_specs}" PARENT_SCOPE) + set(${out_levels} "${levels}" PARENT_SCOPE) + set(${out_level_report} "${level_report}" PARENT_SCOPE) +endfunction() + +##### +##### Emit the header +##### + +# Writes the build-tree header from the template and the expansion strings above. +function(svs_dispatch_emit_header template_file dim_count dim_loop target_loop + level_loop level_defines include_dir out_header) + set(SVS_GEN_DIM_COUNT "${dim_count}") + set(SVS_GEN_DIM_LOOP "${dim_loop}") + set(SVS_GEN_TARGET_LOOP "${target_loop}") + set(SVS_GEN_LEVEL_LOOP "${level_loop}") + set(SVS_GEN_LEVEL_DEFINES "${level_defines}") + set(header "${include_dir}/svs/core/distance/dispatch_surface.h") + configure_file("${template_file}" "${header}" @ONLY) + set(${out_header} "${header}" PARENT_SCOPE) +endfunction() + +##### +##### Emit the manifest +##### + +# The ctest checks read this rather than including the declaration, so they depend +# on the surface's contents and not on that file's format, its location, or what +# else it sets. +function(svs_dispatch_emit_manifest supported_dims levels manifest_file) + string(REPLACE ";" " " levels_text "${levels}") + string(REPLACE ";" " " extents_text "${supported_dims}") + file(GENERATE + OUTPUT "${manifest_file}" + CONTENT "set(SVS_MANIFEST_FIXED_EXTENTS ${extents_text}) +set(SVS_MANIFEST_LEVELS ${levels_text}) +" + ) +endfunction() + +##### +##### Orchestrate +##### + +# Reads the declaration, writes the build-tree header and the ctest manifest, and +# reports the surface. Returns the TU specs and the generated header path, which is +# how cmake/multi-arch.cmake receives them; nothing here escapes via a bare +# file-scope set(). +function(svs_generate_dispatch_surface out_tu_specs out_header) + set(x86_src_dir "${PROJECT_SOURCE_DIR}/include/svs/multi-arch/x86") + + svs_dispatch_read_declaration( + "${SVS_DISPATCH_SURFACE_FILE}" "${x86_src_dir}" + supported_dims dim_list dim_count isa_levels + ) + + # Re-run configure when the declaration changes, so the generated header and + # the translation units cannot go stale. + set_property( + DIRECTORY APPEND PROPERTY CMAKE_CONFIGURE_DEPENDS + "${SVS_DISPATCH_SURFACE_FILE}" + ) + + svs_dispatch_build_expansions( + "${dim_list}" "${isa_levels}" + dim_loop target_loop level_loop level_defines + ) + svs_dispatch_derive_tu_specs( + "${isa_levels}" "${x86_src_dir}" + tu_specs levels level_report + ) + + # The build always compiles against the build-tree copy, and it is placed + # ahead of the source include directory so that it wins. + set(generated_include_dir "${CMAKE_CURRENT_BINARY_DIR}/generated/include") + svs_dispatch_emit_header( + "${SVS_DISPATCH_GEN_CMAKE_DIR}/templates/dispatch_surface.h.in" + "${dim_count}" "${dim_loop}" "${target_loop}" "${level_loop}" "${level_defines}" + "${generated_include_dir}" + header + ) + target_include_directories( + ${SVS_LIB} BEFORE INTERFACE $ + ) + + svs_dispatch_emit_manifest( + "${supported_dims}" "${levels}" + "${CMAKE_BINARY_DIR}/dispatch_surface.manifest.cmake" + ) + + ##### + ##### Report the surface + ##### + + list(LENGTH isa_levels level_count) + string(REPLACE ";" " " dims_display "${supported_dims}") + message(STATUS "Dispatch surface: ${dim_count} extents x ${level_count} ISA levels") + message(STATUS " extents: ${dims_display} svs::Dynamic") + foreach(entry IN LISTS level_report) + message(STATUS " level: ${entry}") + endforeach() + + # Every enumerator without a translation unit is still reachable -- the entry + # points fall back to it -- so its kernels are built by each consumer instead. + set(enum_header "${PROJECT_SOURCE_DIR}/include/svs/core/distance/distance_core.h") + if(EXISTS "${enum_header}") + file(READ "${enum_header}" enum_text) + if(enum_text MATCHES "enum class AVX_AVAILABILITY[ \t\r\n]*{([^}]*)}") + string(REPLACE "," ";" enumerators "${CMAKE_MATCH_1}") + set(undeclared) + foreach(enumerator IN LISTS enumerators) + string(STRIP "${enumerator}" enumerator) + if(enumerator AND NOT enumerator IN_LIST levels) + list(APPEND undeclared "${enumerator}") + endif() + endforeach() + if(undeclared) + string(REPLACE ";" ", " undeclared_display "${undeclared}") + message(STATUS " not in the surface: ${undeclared_display}") + message(STATUS + " dispatched to, but compiled by no translation unit, so " + "every consumer instantiates those kernels itself, at its own -march" + ) + endif() + endif() + endif() + + set(${out_tu_specs} "${tu_specs}" PARENT_SCOPE) + set(${out_header} "${header}" PARENT_SCOPE) +endfunction() diff --git a/cmake/multi-arch.cmake b/cmake/multi-arch.cmake index aeb81e693..a33557d29 100644 --- a/cmake/multi-arch.cmake +++ b/cmake/multi-arch.cmake @@ -12,25 +12,30 @@ # See the License for the specific language governing permissions and # limitations under the License. -set(SVS_X86_SRC_DIR "${PROJECT_SOURCE_DIR}/include/svs/multi-arch/x86") -set(SVS_X86 - "${SVS_X86_SRC_DIR}/avx2.cpp,avx2,haswell" - "${SVS_X86_SRC_DIR}/avx512.cpp,avx512,cascadelake" -) +# Writes the generated dispatch-surface header and populates +# SVS_DISPATCH_TU_SPECS -- "|||", one entry per ISA +# level. The extent list and the levels themselves are declared in +# cmake/dispatch-surface.cmake. +include("${CMAKE_CURRENT_LIST_DIR}/dispatch-levels.cmake") +include("${CMAKE_CURRENT_LIST_DIR}/generate-dispatch-surface.cmake") +svs_generate_dispatch_surface(SVS_DISPATCH_TU_SPECS SVS_GENERATED_DISPATCH_HEADER) set(SVS_X86_OBJECT_FILES) -foreach(x86_info IN LISTS SVS_X86) - string(REPLACE "," ";" x86_info "${x86_info}") - list(GET x86_info 0 src) - list(GET x86_info 1 avx) - list(GET x86_info 2 arch) - set(lib_name "svs_x86_${avx}") +foreach(tu_spec IN LISTS SVS_DISPATCH_TU_SPECS) + svs_parse_tu_spec("${tu_spec}" src level arch infix) + + # Carries the instruction budget for this level, and nothing else. + set(lib_name "svs_x86_${infix}") add_library(${lib_name} INTERFACE) target_compile_options(${lib_name} INTERFACE -march=${arch} -mtune=${arch}) - set(obj_name ${arch}_obj) + # Unique only because validate-dispatch-surface.cmake rejects duplicate + # infixes; two levels sharing one would collide on this target name. + set(obj_name ${infix}_obj) add_library(${obj_name} OBJECT ${src}) - target_link_libraries(${obj_name} PRIVATE ${SVS_LIB} svs::compile_options fmt::fmt ${lib_name}) + target_link_libraries( + ${obj_name} PRIVATE ${SVS_LIB} svs::compile_options fmt::fmt ${lib_name} + ) list(APPEND SVS_X86_OBJECT_FILES $) endforeach() diff --git a/cmake/templates/dispatch_surface.h.in b/cmake/templates/dispatch_surface.h.in new file mode 100644 index 000000000..ffbb9c50a --- /dev/null +++ b/cmake/templates/dispatch_surface.h.in @@ -0,0 +1,44 @@ +/* + * Copyright 2026 Intel Corporation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// GENERATED FILE -- DO NOT EDIT. Regenerated on every CMake configure from +// cmake/dispatch-surface.cmake; committed so a bare `-I include` compile works. + +#pragma once + +// clang-format off +// Reflowing would realign the escaped newlines and make the committed copy +// disagree with the one CMake writes -- an endless format/regenerate loop. + +// Number of extents with a fixed-extent kernel, including svs::Dynamic. +#define SVS_SUPPORTED_DIM_COUNT @SVS_GEN_DIM_COUNT@ + +// Invokes M(extent) once per extent. +#define SVS_FOR_EACH_SUPPORTED_DIM(M) @SVS_GEN_DIM_LOOP@ + +// Invokes M(extent, isa_level) once per (extent, ISA level) pair -- that is, +// once per kernel the library compiles ahead of time, modulo type pairs. +#define SVS_FOR_EACH_DISPATCH_TARGET(M) @SVS_GEN_TARGET_LOOP@ + +// Invokes M(isa_level) once per ISA level, weakest first. AVX_AVAILABILITY +// enumerators without a translation unit are absent: this is the surface. +#define SVS_FOR_EACH_ISA_LEVEL(M) @SVS_GEN_LEVEL_LOOP@ + +// One per ISA level in the surface, and only those: a level absent from it has no +// instantiations. Tested with `defined` so a -Wundef build stays quiet. +@SVS_GEN_LEVEL_DEFINES@ + +// clang-format on diff --git a/cmake/validate-dispatch-surface.cmake b/cmake/validate-dispatch-surface.cmake new file mode 100644 index 000000000..3f038565b --- /dev/null +++ b/cmake/validate-dispatch-surface.cmake @@ -0,0 +1,202 @@ +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +##### +##### Reads and checks a dispatch-surface declaration. +##### +##### This is deliberately free of build-system state so that it runs in script +##### mode as well as during configure: +##### +##### cmake -DSVS_DISPATCH_SURFACE_FILE= \ +##### -DSVS_X86_SRC_DIR= \ +##### -P cmake/validate-dispatch-surface.cmake +##### +##### Inputs: +##### SVS_DISPATCH_SURFACE_FILE -- the declaration to read +##### SVS_X86_SRC_DIR -- where per-level translation units live +##### +##### Outputs: +##### SVS_SUPPORTED_DIMS, SVS_ISA_LEVELS -- verbatim from the declaration +##### SVS_DIM_LIST -- extents, with svs::Dynamic appended +##### SVS_DIM_COUNT -- length of SVS_DIM_LIST +##### SVS_FIXED_DIM_COUNT -- length of SVS_SUPPORTED_DIMS +##### + +## In script mode there is no cmake_minimum_required, so policies default to OLD. +## Both of these are load-bearing here: CMP0007 keeps an empty `|`-field from +## vanishing when the entry is split, and CMP0057 enables `IN_LIST`. +cmake_policy(PUSH) +cmake_policy(SET CMP0007 NEW) +cmake_policy(SET CMP0057 NEW) + +include("${CMAKE_CURRENT_LIST_DIR}/dispatch-levels.cmake") + +if(NOT SVS_DISPATCH_SURFACE_FILE) + message(FATAL_ERROR "SVS_DISPATCH_SURFACE_FILE is not set.") +endif() +if(NOT EXISTS "${SVS_DISPATCH_SURFACE_FILE}") + message(FATAL_ERROR + "SVS_DISPATCH_SURFACE_FILE does not exist: ${SVS_DISPATCH_SURFACE_FILE}" + ) +endif() +if(NOT SVS_X86_SRC_DIR) + message(FATAL_ERROR "SVS_X86_SRC_DIR is not set.") +endif() + +# The declaration is plain CMake: it sets SVS_SUPPORTED_DIMS and SVS_ISA_LEVELS +# and does nothing else. Clear them first so that a declaration which forgets one +# is reported as empty rather than inheriting a value from the caller. +set(SVS_SUPPORTED_DIMS) +set(SVS_ISA_LEVELS) +include("${SVS_DISPATCH_SURFACE_FILE}") + +##### +##### The extent list +##### + +if(NOT SVS_SUPPORTED_DIMS) + message(FATAL_ERROR + "SVS_SUPPORTED_DIMS is empty in ${SVS_DISPATCH_SURFACE_FILE}. At least " + "one fixed extent is required." + ) +endif() + +foreach(dim IN LISTS SVS_SUPPORTED_DIMS) + if(NOT dim MATCHES "^[1-9][0-9]*$") + message(FATAL_ERROR + "SVS_SUPPORTED_DIMS contains '${dim}', which is not a positive " + "integer. svs::Dynamic is required and is appended automatically, " + "so it must not be listed." + ) + endif() +endforeach() + +set(svs_dims_deduped ${SVS_SUPPORTED_DIMS}) +list(REMOVE_DUPLICATES svs_dims_deduped) +list(LENGTH SVS_SUPPORTED_DIMS SVS_FIXED_DIM_COUNT) +list(LENGTH svs_dims_deduped svs_dims_unique) +if(NOT SVS_FIXED_DIM_COUNT EQUAL svs_dims_unique) + message(FATAL_ERROR + "SVS_SUPPORTED_DIMS contains duplicate extents. Every extent must " + "appear exactly once." + ) +endif() + +# svs::Dynamic is mandatory: it is what serves every dimensionality without a +# fixed-extent kernel, and the library is incorrect without it. +set(SVS_DIM_LIST ${SVS_SUPPORTED_DIMS} "svs::Dynamic") +list(LENGTH SVS_DIM_LIST SVS_DIM_COUNT) + +##### +##### The ISA levels +##### + +if(NOT SVS_ISA_LEVELS) + message(FATAL_ERROR "SVS_ISA_LEVELS is empty in ${SVS_DISPATCH_SURFACE_FILE}.") +endif() + +# The legal level names are the enumerators of svs::distance::AVX_AVAILABILITY, +# read from the header rather than hand-copied here: a declared level the enum +# does not name would only fail once the generated C++ tries to compile it, +# which is exactly the late failure this validator exists to catch early. +set(svs_distance_core_header + "${CMAKE_CURRENT_LIST_DIR}/../include/svs/core/distance/distance_core.h" +) +if(NOT EXISTS "${svs_distance_core_header}") + message(FATAL_ERROR + "Cannot find ${svs_distance_core_header} to read AVX_AVAILABILITY from." + ) +endif() +file(READ "${svs_distance_core_header}" svs_distance_core_text) +if(NOT svs_distance_core_text MATCHES "enum class AVX_AVAILABILITY[ \t\r\n]*{([^}]*)}") + message(FATAL_ERROR + "Cannot find 'enum class AVX_AVAILABILITY { ... }' in " + "${svs_distance_core_header}." + ) +endif() +string(REPLACE "," ";" svs_legal_levels "${CMAKE_MATCH_1}") +list(TRANSFORM svs_legal_levels STRIP) +# NONE means "no level is present" and is never itself a row: it has no +# translation unit and no object library, so declaring it would name files +# that must not exist. +list(REMOVE_ITEM svs_legal_levels "NONE") + +set(svs_seen_levels) +set(svs_seen_infixes) +set(svs_seen_archs) +foreach(level_spec IN LISTS SVS_ISA_LEVELS) + svs_parse_isa_level("${level_spec}" level arch infix) + foreach(field level arch infix) + if(NOT ${field}) + message(FATAL_ERROR + "Malformed SVS_ISA_LEVELS entry '${level_spec}': ${field} is empty." + ) + endif() + endforeach() + if(level STREQUAL "NONE") + message(FATAL_ERROR + "ISA level 'NONE' in SVS_ISA_LEVELS is not declarable: it means no " + "level is present, so it has no translation unit and no object " + "library for a row to name." + ) + endif() + if(NOT level IN_LIST svs_legal_levels) + string(REPLACE ";" ", " svs_legal_levels_display "${svs_legal_levels}") + message(FATAL_ERROR + "Unknown ISA level '${level}' in SVS_ISA_LEVELS: not an enumerator " + "of svs::distance::AVX_AVAILABILITY in ${svs_distance_core_header} " + "(legal levels: ${svs_legal_levels_display})." + ) + endif() + if(level IN_LIST svs_seen_levels) + message(FATAL_ERROR "Duplicate ISA level '${level}' in SVS_ISA_LEVELS.") + endif() + if(infix IN_LIST svs_seen_infixes) + message(FATAL_ERROR + "Duplicate TU infix '${infix}' in SVS_ISA_LEVELS; infixes name " + "generated files and must be unique." + ) + endif() + if(arch IN_LIST svs_seen_archs) + message(FATAL_ERROR + "Duplicate -march '${arch}' in SVS_ISA_LEVELS; each level's -march " + "is its instruction budget, so two levels sharing one budget compile " + "the weaker level with instructions its runtime predicate does not " + "guarantee, and hosts routed to it fault." + ) + endif() + if(NOT EXISTS "${SVS_X86_SRC_DIR}/${infix}.cpp") + message(FATAL_ERROR + "ISA level '${level}' has no translation unit: expected " + "${SVS_X86_SRC_DIR}/${infix}.cpp. Adding a level to SVS_ISA_LEVELS " + "requires creating that file." + ) + endif() + list(APPEND svs_seen_levels ${level}) + list(APPEND svs_seen_infixes ${infix}) + list(APPEND svs_seen_archs ${arch}) +endforeach() + +# distance_core.h #errors on x86_64 unless both are present, so a surface +# omitting either is not a smaller build: it cannot compile. +foreach(svs_mandatory_level AVX2 AVX512) + if(NOT svs_mandatory_level IN_LIST svs_seen_levels) + message(FATAL_ERROR + "SVS_ISA_LEVELS omits mandatory level '${svs_mandatory_level}'; " + "distance_core.h requires both AVX2 and AVX512 to be present." + ) + endif() +endforeach() + +cmake_policy(POP) diff --git a/include/svs/core/distance/cosine.h b/include/svs/core/distance/cosine.h index 9f4924997..75f617ab1 100644 --- a/include/svs/core/distance/cosine.h +++ b/include/svs/core/distance/cosine.h @@ -47,6 +47,18 @@ class CosineSimilarity { public: template static constexpr float compute(const Ea* a, const Eb* b, float a_norm, size_t N) { + if constexpr (has_vnni_kernel) { + if (__builtin_expect( + svs::detail::avx_runtime_flags.is_avx512vnni_supported(), 1 + )) { + return CosineSimilarityImpl< + Dynamic, + Ea, + Eb, + AVX_AVAILABILITY::AVX512_VNNI>:: + compute(a, b, a_norm, lib::MaybeStatic(N)); + } + } if (__builtin_expect(svs::detail::avx_runtime_flags.is_avx512f_supported(), 1)) { return CosineSimilarityImpl::compute( a, b, a_norm, lib::MaybeStatic(N) @@ -65,6 +77,23 @@ class CosineSimilarity { template static constexpr float compute(const Ea* a, const Eb* b, float a_norm) { + if constexpr (has_vnni_kernel) { + if (__builtin_expect( + svs::detail::avx_runtime_flags.is_avx512vnni_supported(), 1 + )) { + if constexpr (is_dim_supported()) { + return CosineSimilarityImpl:: + compute(a, b, a_norm, lib::MaybeStatic()); + } else { + return CosineSimilarityImpl< + Dynamic, + Ea, + Eb, + AVX_AVAILABILITY::AVX512_VNNI>:: + compute(a, b, a_norm, lib::MaybeStatic(N)); + } + } + } if (__builtin_expect(svs::detail::avx_runtime_flags.is_avx512f_supported(), 1)) { if constexpr (is_dim_supported()) { return CosineSimilarityImpl::compute( @@ -255,38 +284,70 @@ template <> struct CosineFloatOp<16> : public svs::simd::ConvertToFloat<16> { } }; -// Small Integers +// Small Integers, with VNNI SVS_VALIDATE_BOOL_ENV(SVS_AVX512_VNNI) #if SVS_AVX512_VNNI template -struct CosineSimilarityImpl { +struct CosineSimilarityImpl { SVS_NOINLINE static float compute(const int8_t* a, const int8_t* b, float a_norm, lib::MaybeStatic length) { - if (__builtin_expect(svs::detail::avx_runtime_flags.is_avx512vnni_supported(), 1)) { - auto sum = _mm512_setzero_epi32(); - auto bnorm_accum = _mm512_setzero_epi32(); - auto mask = create_mask<32>(length); - auto all = no_mask<32>(); - - for (size_t j = 0; j < length.size(); j += 32) { - auto temp_a = - _mm256_maskz_loadu_epi8(islast<32>(length, j) ? mask : all, a + j); - auto va = _mm512_cvtepi8_epi16(temp_a); - - auto temp_b = - _mm256_maskz_loadu_epi8(islast<32>(length, j) ? mask : all, b + j); - auto vb = _mm512_cvtepi8_epi16(temp_b); - - bnorm_accum = _mm512_dpwssd_epi32(bnorm_accum, vb, vb); - sum = _mm512_dpwssd_epi32(sum, va, vb); - } + auto sum = _mm512_setzero_epi32(); + auto bnorm_accum = _mm512_setzero_epi32(); + auto mask = create_mask<32>(length); + auto all = no_mask<32>(); + + for (size_t j = 0; j < length.size(); j += 32) { + auto temp_a = + _mm256_maskz_loadu_epi8(islast<32>(length, j) ? mask : all, a + j); + auto va = _mm512_cvtepi8_epi16(temp_a); + + auto temp_b = + _mm256_maskz_loadu_epi8(islast<32>(length, j) ? mask : all, b + j); + auto vb = _mm512_cvtepi8_epi16(temp_b); + + bnorm_accum = _mm512_dpwssd_epi32(bnorm_accum, vb, vb); + sum = _mm512_dpwssd_epi32(sum, va, vb); + } + + float b_norm = std::sqrt(static_cast(_mm512_reduce_add_epi32(bnorm_accum))); + return lib::narrow_cast(_mm512_reduce_add_epi32(sum)) / (a_norm * b_norm); + } +}; - float b_norm = - std::sqrt(static_cast(_mm512_reduce_add_epi32(bnorm_accum))); - return lib::narrow_cast(_mm512_reduce_add_epi32(sum)) / - (a_norm * b_norm); +template +struct CosineSimilarityImpl { + SVS_NOINLINE static float + compute(const uint8_t* a, const uint8_t* b, float a_norm, lib::MaybeStatic length) { + auto sum = _mm512_setzero_epi32(); + auto bnorm_accum = _mm512_setzero_epi32(); + auto mask = create_mask<32>(length); + auto all = no_mask<32>(); + + for (size_t j = 0; j < length.size(); j += 32) { + auto temp_a = + _mm256_maskz_loadu_epi8(islast<32>(length, j) ? mask : all, a + j); + auto va = _mm512_cvtepu8_epi16(temp_a); + + auto temp_b = + _mm256_maskz_loadu_epi8(islast<32>(length, j) ? mask : all, b + j); + auto vb = _mm512_cvtepu8_epi16(temp_b); + + bnorm_accum = _mm512_dpwssd_epi32(bnorm_accum, vb, vb); + sum = _mm512_dpwssd_epi32(sum, va, vb); } - // Fallback to AVX512 + float b_norm = std::sqrt(static_cast(_mm512_reduce_add_epi32(bnorm_accum))); + return lib::narrow_cast(_mm512_reduce_add_epi32(sum)) / (a_norm * b_norm); + } +}; + +#endif + +// Must stay outside the SVS_AVX512_VNNI guard: avx512.cpp compiles with that macro +// at 0, and hiding this specialization there would silently select the generic kernel. +template +struct CosineSimilarityImpl { + SVS_NOINLINE static float + compute(const int8_t* a, const int8_t* b, float a_norm, lib::MaybeStatic length) { auto [sum, norm] = simd::generic_simd_op(CosineFloatOp<16>(), a, b, length); return sum / (std::sqrt(norm) * a_norm); } @@ -296,37 +357,11 @@ template struct CosineSimilarityImpl { SVS_NOINLINE static float compute(const uint8_t* a, const uint8_t* b, float a_norm, lib::MaybeStatic length) { - if (__builtin_expect(svs::detail::avx_runtime_flags.is_avx512vnni_supported(), 1)) { - auto sum = _mm512_setzero_epi32(); - auto bnorm_accum = _mm512_setzero_epi32(); - auto mask = create_mask<32>(length); - auto all = no_mask<32>(); - - for (size_t j = 0; j < length.size(); j += 32) { - auto temp_a = - _mm256_maskz_loadu_epi8(islast<32>(length, j) ? mask : all, a + j); - auto va = _mm512_cvtepu8_epi16(temp_a); - - auto temp_b = - _mm256_maskz_loadu_epi8(islast<32>(length, j) ? mask : all, b + j); - auto vb = _mm512_cvtepu8_epi16(temp_b); - - bnorm_accum = _mm512_dpwssd_epi32(bnorm_accum, vb, vb); - sum = _mm512_dpwssd_epi32(sum, va, vb); - } - float b_norm = - std::sqrt(static_cast(_mm512_reduce_add_epi32(bnorm_accum))); - return lib::narrow_cast(_mm512_reduce_add_epi32(sum)) / - (a_norm * b_norm); - } - // Fallback to AVX512 auto [sum, norm] = simd::generic_simd_op(CosineFloatOp<16>(), a, b, length); return sum / (std::sqrt(norm) * a_norm); } }; -#endif - // Floating and Mixed Types template struct CosineSimilarityImpl { SVS_NOINLINE static float @@ -500,26 +535,11 @@ struct CosineSimilarityImpl { #if defined(__x86_64__) #include "svs/multi-arch/x86/preprocessor.h" -// TODO: connect with dim_supported_list -DISTANCE_CS_EXTERN_TEMPLATE(64, AVX_AVAILABILITY::AVX512); -DISTANCE_CS_EXTERN_TEMPLATE(96, AVX_AVAILABILITY::AVX512); -DISTANCE_CS_EXTERN_TEMPLATE(100, AVX_AVAILABILITY::AVX512); -DISTANCE_CS_EXTERN_TEMPLATE(128, AVX_AVAILABILITY::AVX512); -DISTANCE_CS_EXTERN_TEMPLATE(160, AVX_AVAILABILITY::AVX512); -DISTANCE_CS_EXTERN_TEMPLATE(200, AVX_AVAILABILITY::AVX512); -DISTANCE_CS_EXTERN_TEMPLATE(512, AVX_AVAILABILITY::AVX512); -DISTANCE_CS_EXTERN_TEMPLATE(768, AVX_AVAILABILITY::AVX512); -DISTANCE_CS_EXTERN_TEMPLATE(Dynamic, AVX_AVAILABILITY::AVX512); - -DISTANCE_CS_EXTERN_TEMPLATE(64, AVX_AVAILABILITY::AVX2); -DISTANCE_CS_EXTERN_TEMPLATE(96, AVX_AVAILABILITY::AVX2); -DISTANCE_CS_EXTERN_TEMPLATE(100, AVX_AVAILABILITY::AVX2); -DISTANCE_CS_EXTERN_TEMPLATE(128, AVX_AVAILABILITY::AVX2); -DISTANCE_CS_EXTERN_TEMPLATE(160, AVX_AVAILABILITY::AVX2); -DISTANCE_CS_EXTERN_TEMPLATE(200, AVX_AVAILABILITY::AVX2); -DISTANCE_CS_EXTERN_TEMPLATE(512, AVX_AVAILABILITY::AVX2); -DISTANCE_CS_EXTERN_TEMPLATE(768, AVX_AVAILABILITY::AVX2); -DISTANCE_CS_EXTERN_TEMPLATE(Dynamic, AVX_AVAILABILITY::AVX2); +// Declare every kernel the library compiles ahead of time. Missing an entry here +// makes a consumer instantiate it locally, at the consumer's own -march. +#define SVS_CS_EXTERN(DIM, LEVEL) SVS_INSTANTIATE_CS(extern template, DIM, LEVEL) +SVS_FOR_EACH_DISPATCH_TARGET(SVS_CS_EXTERN) +#undef SVS_CS_EXTERN #endif } // namespace svs::distance diff --git a/include/svs/core/distance/distance_core.h b/include/svs/core/distance/distance_core.h index 4f59f9de0..80ba089be 100644 --- a/include/svs/core/distance/distance_core.h +++ b/include/svs/core/distance/distance_core.h @@ -21,16 +21,55 @@ #include "svs/lib/saveload.h" #include "svs/lib/type_traits.h" +// The extent list and the ISA levels, generated from cmake/dispatch-surface.cmake. +#include "svs/core/distance/dispatch_surface.h" + +// Needed here and not only where the kernels are declared: the entry points must +// not dispatch to a level with no kernel for the pair in hand. +#if defined(__x86_64__) +#include "svs/multi-arch/x86/preprocessor.h" + +// Dispatched to for every type pair, so a surface without them would leave each +// consumer to instantiate the kernels itself, from the generic template. +#if !defined(SVS_ISA_LEVEL_AVX2) || !defined(SVS_ISA_LEVEL_AVX512) +#error "the x86 dispatch surface must declare the AVX2 and AVX512 ISA levels" +#endif +#endif + #include #include namespace svs::distance { -enum class AVX_AVAILABILITY { NONE, AVX2, AVX512 }; - -constexpr std::array supported_dim_list{ - 64, 96, 100, 128, 160, 200, 512, 768, svs::Dynamic}; - +/// The runtime ISA levels the library compiles distance kernels for. +/// +/// Each is a promise about the host, checked once in the entry point; the kernels +/// branch on nothing. Append new levels -- renumbering changes mangled names. +enum class AVX_AVAILABILITY { NONE, AVX2, AVX512, AVX512_VNNI }; + +/// Whether (Ea, Eb) has a kernel at AVX_AVAILABILITY::AVX512_VNNI. +/// +/// False where there is no such kernel -- a float-promoting pair, or any pair when +/// the surface omits the level. Dispatching anyway instantiates the generic template. +template inline constexpr bool has_vnni_kernel = false; + +#if defined(__x86_64__) && defined(SVS_ISA_LEVEL_AVX512_VNNI) +#define SVS_MARK_VNNI_PAIR(Ea, Eb, ...) \ + template <> inline constexpr bool has_vnni_kernel = true; +SVS_TYPE_PAIRS_AVX512_VNNI(SVS_MARK_VNNI_PAIR, ) +#undef SVS_MARK_VNNI_PAIR +#endif + +/// The extents that have a fixed-extent kernel, including svs::Dynamic. +#define SVS_DIM_LIST_ENTRY(N) N, +constexpr std::array supported_dim_list{ + SVS_FOR_EACH_SUPPORTED_DIM(SVS_DIM_LIST_ENTRY)}; +#undef SVS_DIM_LIST_ENTRY + +/// Whether N has a fixed-extent kernel. +/// +/// Not a capability test: every dimensionality is supported. An extent answering +/// `false` dispatches to the svs::Dynamic kernel instead of a fully unrolled one. template constexpr bool is_dim_supported() { for (auto i : supported_dim_list) { if (i == N) { diff --git a/include/svs/core/distance/euclidean.h b/include/svs/core/distance/euclidean.h index 08aed5d25..1d3734192 100644 --- a/include/svs/core/distance/euclidean.h +++ b/include/svs/core/distance/euclidean.h @@ -86,6 +86,15 @@ class L2 { public: template static constexpr float compute(const Ea* a, const Eb* b, size_t N) { + if constexpr (has_vnni_kernel) { + if (__builtin_expect( + svs::detail::avx_runtime_flags.is_avx512vnni_supported(), 1 + )) { + return L2Impl::compute( + a, b, lib::MaybeStatic(N) + ); + } + } if (__builtin_expect(svs::detail::avx_runtime_flags.is_avx512f_supported(), 1)) { return L2Impl::compute( a, b, lib::MaybeStatic(N) @@ -103,6 +112,21 @@ class L2 { template static constexpr float compute(const Ea* a, const Eb* b) { + if constexpr (has_vnni_kernel) { + if (__builtin_expect( + svs::detail::avx_runtime_flags.is_avx512vnni_supported(), 1 + )) { + if constexpr (is_dim_supported()) { + return L2Impl::compute( + a, b, lib::MaybeStatic() + ); + } else { + return L2Impl::compute( + a, b, lib::MaybeStatic(N) + ); + } + } + } if (__builtin_expect(svs::detail::avx_runtime_flags.is_avx512f_supported(), 1)) { if constexpr (is_dim_supported()) { return L2Impl::compute( @@ -258,7 +282,7 @@ template <> struct L2FloatOp<16> : public svs::simd::ConvertToFloat<16> { static float reduce(__m512 x) { return _mm512_reduce_add_ps(x); } }; -// Small Integers +// Small Integers, with VNNI SVS_VALIDATE_BOOL_ENV(SVS_AVX512_VNNI) #if SVS_AVX512_VNNI @@ -289,14 +313,27 @@ template <> struct L2VNNIOp : public svs::simd::ConvertForVNNI struct L2Impl { + SVS_NOINLINE static float + compute(const int8_t* a, const int8_t* b, lib::MaybeStatic length) { + return simd::generic_simd_op(L2VNNIOp(), a, b, length); + } +}; + +template struct L2Impl { + SVS_NOINLINE static float + compute(const uint8_t* a, const uint8_t* b, lib::MaybeStatic length) { + return simd::generic_simd_op(L2VNNIOp(), a, b, length); + } +}; + +#endif + +// Must stay outside the SVS_AVX512_VNNI guard: avx512.cpp compiles with that macro +// at 0, and hiding this specialization there would silently select the generic kernel. template struct L2Impl { SVS_NOINLINE static float compute(const int8_t* a, const int8_t* b, lib::MaybeStatic length) { - if (__builtin_expect(svs::detail::avx_runtime_flags.is_avx512vnni_supported(), 1)) { - return simd::generic_simd_op(L2VNNIOp(), a, b, length); - } - // fallback to AVX512 return simd::generic_simd_op(L2FloatOp<16>{}, a, b, length); } }; @@ -304,16 +341,10 @@ template struct L2Impl { template struct L2Impl { SVS_NOINLINE static float compute(const uint8_t* a, const uint8_t* b, lib::MaybeStatic length) { - if (__builtin_expect(svs::detail::avx_runtime_flags.is_avx512vnni_supported(), 1)) { - return simd::generic_simd_op(L2VNNIOp(), a, b, length); - } - // fallback to AVX512 return simd::generic_simd_op(L2FloatOp<16>{}, a, b, length); } }; -#endif - // Floating and Mixed Types template struct L2Impl { SVS_NOINLINE static float @@ -438,26 +469,11 @@ template struct L2Impl { #include "svs/multi-arch/x86/preprocessor.h" -// TODO: connect with dim_supported_list -DISTANCE_L2_EXTERN_TEMPLATE(64, AVX_AVAILABILITY::AVX512); -DISTANCE_L2_EXTERN_TEMPLATE(96, AVX_AVAILABILITY::AVX512); -DISTANCE_L2_EXTERN_TEMPLATE(100, AVX_AVAILABILITY::AVX512); -DISTANCE_L2_EXTERN_TEMPLATE(128, AVX_AVAILABILITY::AVX512); -DISTANCE_L2_EXTERN_TEMPLATE(160, AVX_AVAILABILITY::AVX512); -DISTANCE_L2_EXTERN_TEMPLATE(200, AVX_AVAILABILITY::AVX512); -DISTANCE_L2_EXTERN_TEMPLATE(512, AVX_AVAILABILITY::AVX512); -DISTANCE_L2_EXTERN_TEMPLATE(768, AVX_AVAILABILITY::AVX512); -DISTANCE_L2_EXTERN_TEMPLATE(Dynamic, AVX_AVAILABILITY::AVX512); - -DISTANCE_L2_EXTERN_TEMPLATE(64, AVX_AVAILABILITY::AVX2); -DISTANCE_L2_EXTERN_TEMPLATE(96, AVX_AVAILABILITY::AVX2); -DISTANCE_L2_EXTERN_TEMPLATE(100, AVX_AVAILABILITY::AVX2); -DISTANCE_L2_EXTERN_TEMPLATE(128, AVX_AVAILABILITY::AVX2); -DISTANCE_L2_EXTERN_TEMPLATE(160, AVX_AVAILABILITY::AVX2); -DISTANCE_L2_EXTERN_TEMPLATE(200, AVX_AVAILABILITY::AVX2); -DISTANCE_L2_EXTERN_TEMPLATE(512, AVX_AVAILABILITY::AVX2); -DISTANCE_L2_EXTERN_TEMPLATE(768, AVX_AVAILABILITY::AVX2); -DISTANCE_L2_EXTERN_TEMPLATE(Dynamic, AVX_AVAILABILITY::AVX2); +// Declare every kernel the library compiles ahead of time. Missing an entry here +// makes a consumer instantiate it locally, at the consumer's own -march. +#define SVS_L2_EXTERN(DIM, LEVEL) SVS_INSTANTIATE_L2(extern template, DIM, LEVEL) +SVS_FOR_EACH_DISPATCH_TARGET(SVS_L2_EXTERN) +#undef SVS_L2_EXTERN #endif } // namespace svs::distance diff --git a/include/svs/core/distance/inner_product.h b/include/svs/core/distance/inner_product.h index 0f7837a53..c7d958a0d 100644 --- a/include/svs/core/distance/inner_product.h +++ b/include/svs/core/distance/inner_product.h @@ -46,6 +46,15 @@ class IP { public: template static constexpr float compute(const Ea* a, const Eb* b, size_t N) { + if constexpr (has_vnni_kernel) { + if (__builtin_expect( + svs::detail::avx_runtime_flags.is_avx512vnni_supported(), 1 + )) { + return IPImpl::compute( + a, b, lib::MaybeStatic(N) + ); + } + } if (__builtin_expect(svs::detail::avx_runtime_flags.is_avx512f_supported(), 1)) { return IPImpl::compute( a, b, lib::MaybeStatic(N) @@ -63,6 +72,21 @@ class IP { template static constexpr float compute(const Ea* a, const Eb* b) { + if constexpr (has_vnni_kernel) { + if (__builtin_expect( + svs::detail::avx_runtime_flags.is_avx512vnni_supported(), 1 + )) { + if constexpr (is_dim_supported()) { + return IPImpl::compute( + a, b, lib::MaybeStatic() + ); + } else { + return IPImpl::compute( + a, b, lib::MaybeStatic(N) + ); + } + } + } if (__builtin_expect(svs::detail::avx_runtime_flags.is_avx512f_supported(), 1)) { if constexpr (is_dim_supported()) { return IPImpl::compute( @@ -215,7 +239,7 @@ template <> struct IPFloatOp<16> : public svs::simd::ConvertToFloat<16> { static float reduce(__m512 x) { return _mm512_reduce_add_ps(x); } }; -// Small Integers +// Small Integers, with VNNI SVS_VALIDATE_BOOL_ENV(SVS_AVX512_VNNI) #if SVS_AVX512_VNNI @@ -243,14 +267,27 @@ template <> struct IPVNNIOp : public svs::simd::ConvertForVNNI struct IPImpl { + SVS_NOINLINE static float + compute(const int8_t* a, const int8_t* b, lib::MaybeStatic length) { + return simd::generic_simd_op(IPVNNIOp(), a, b, length); + } +}; + +template struct IPImpl { + SVS_NOINLINE static float + compute(const uint8_t* a, const uint8_t* b, lib::MaybeStatic length) { + return simd::generic_simd_op(IPVNNIOp(), a, b, length); + } +}; + +#endif + +// Must stay outside the SVS_AVX512_VNNI guard: avx512.cpp compiles with that macro +// at 0, and hiding this specialization there would silently select the generic kernel. template struct IPImpl { SVS_NOINLINE static float compute(const int8_t* a, const int8_t* b, lib::MaybeStatic length) { - if (__builtin_expect(svs::detail::avx_runtime_flags.is_avx512vnni_supported(), 1)) { - return simd::generic_simd_op(IPVNNIOp(), a, b, length); - } - // fallback to AVX512 return svs::simd::generic_simd_op(IPFloatOp<16>{}, a, b, length); } }; @@ -258,16 +295,10 @@ template struct IPImpl { template struct IPImpl { SVS_NOINLINE static float compute(const uint8_t* a, const uint8_t* b, lib::MaybeStatic length) { - if (__builtin_expect(svs::detail::avx_runtime_flags.is_avx512vnni_supported(), 1)) { - return simd::generic_simd_op(IPVNNIOp(), a, b, length); - } - // fallback to AVX512 return svs::simd::generic_simd_op(IPFloatOp<16>{}, a, b, length); } }; -#endif - // Floating and Mixed Types template struct IPImpl { SVS_NOINLINE static float @@ -388,26 +419,12 @@ template struct IPImpl { #if defined(__x86_64__) #include "svs/multi-arch/x86/preprocessor.h" -// TODO: connect with dim_supported_list -DISTANCE_IP_EXTERN_TEMPLATE(64, AVX_AVAILABILITY::AVX512); -DISTANCE_IP_EXTERN_TEMPLATE(96, AVX_AVAILABILITY::AVX512); -DISTANCE_IP_EXTERN_TEMPLATE(100, AVX_AVAILABILITY::AVX512); -DISTANCE_IP_EXTERN_TEMPLATE(128, AVX_AVAILABILITY::AVX512); -DISTANCE_IP_EXTERN_TEMPLATE(160, AVX_AVAILABILITY::AVX512); -DISTANCE_IP_EXTERN_TEMPLATE(200, AVX_AVAILABILITY::AVX512); -DISTANCE_IP_EXTERN_TEMPLATE(512, AVX_AVAILABILITY::AVX512); -DISTANCE_IP_EXTERN_TEMPLATE(768, AVX_AVAILABILITY::AVX512); -DISTANCE_IP_EXTERN_TEMPLATE(Dynamic, AVX_AVAILABILITY::AVX512); - -DISTANCE_IP_EXTERN_TEMPLATE(64, AVX_AVAILABILITY::AVX2); -DISTANCE_IP_EXTERN_TEMPLATE(96, AVX_AVAILABILITY::AVX2); -DISTANCE_IP_EXTERN_TEMPLATE(100, AVX_AVAILABILITY::AVX2); -DISTANCE_IP_EXTERN_TEMPLATE(128, AVX_AVAILABILITY::AVX2); -DISTANCE_IP_EXTERN_TEMPLATE(160, AVX_AVAILABILITY::AVX2); -DISTANCE_IP_EXTERN_TEMPLATE(200, AVX_AVAILABILITY::AVX2); -DISTANCE_IP_EXTERN_TEMPLATE(512, AVX_AVAILABILITY::AVX2); -DISTANCE_IP_EXTERN_TEMPLATE(768, AVX_AVAILABILITY::AVX2); -DISTANCE_IP_EXTERN_TEMPLATE(Dynamic, AVX_AVAILABILITY::AVX2); + +// Declare every kernel the library compiles ahead of time. Missing an entry here +// makes a consumer instantiate it locally, at the consumer's own -march. +#define SVS_IP_EXTERN(DIM, LEVEL) SVS_INSTANTIATE_IP(extern template, DIM, LEVEL) +SVS_FOR_EACH_DISPATCH_TARGET(SVS_IP_EXTERN) +#undef SVS_IP_EXTERN #endif } // namespace svs::distance diff --git a/include/svs/multi-arch/x86/avx2.cpp b/include/svs/multi-arch/x86/avx2.cpp index bff53ae10..aa5a2fa75 100644 --- a/include/svs/multi-arch/x86/avx2.cpp +++ b/include/svs/multi-arch/x86/avx2.cpp @@ -16,41 +16,18 @@ #if defined(__x86_64__) #include "svs/core/distance/cosine.h" +#include "svs/core/distance/dispatch_surface.h" #include "svs/core/distance/euclidean.h" #include "svs/core/distance/inner_product.h" +#include "svs/multi-arch/x86/preprocessor.h" namespace svs::distance { -// TODO: connect with dim_supported_list -DISTANCE_L2_INSTANTIATE_TEMPLATE(64, AVX_AVAILABILITY::AVX2); -DISTANCE_L2_INSTANTIATE_TEMPLATE(96, AVX_AVAILABILITY::AVX2); -DISTANCE_L2_INSTANTIATE_TEMPLATE(100, AVX_AVAILABILITY::AVX2); -DISTANCE_L2_INSTANTIATE_TEMPLATE(128, AVX_AVAILABILITY::AVX2); -DISTANCE_L2_INSTANTIATE_TEMPLATE(160, AVX_AVAILABILITY::AVX2); -DISTANCE_L2_INSTANTIATE_TEMPLATE(200, AVX_AVAILABILITY::AVX2); -DISTANCE_L2_INSTANTIATE_TEMPLATE(512, AVX_AVAILABILITY::AVX2); -DISTANCE_L2_INSTANTIATE_TEMPLATE(768, AVX_AVAILABILITY::AVX2); -DISTANCE_L2_INSTANTIATE_TEMPLATE(Dynamic, AVX_AVAILABILITY::AVX2); - -DISTANCE_IP_INSTANTIATE_TEMPLATE(64, AVX_AVAILABILITY::AVX2); -DISTANCE_IP_INSTANTIATE_TEMPLATE(96, AVX_AVAILABILITY::AVX2); -DISTANCE_IP_INSTANTIATE_TEMPLATE(100, AVX_AVAILABILITY::AVX2); -DISTANCE_IP_INSTANTIATE_TEMPLATE(128, AVX_AVAILABILITY::AVX2); -DISTANCE_IP_INSTANTIATE_TEMPLATE(160, AVX_AVAILABILITY::AVX2); -DISTANCE_IP_INSTANTIATE_TEMPLATE(200, AVX_AVAILABILITY::AVX2); -DISTANCE_IP_INSTANTIATE_TEMPLATE(512, AVX_AVAILABILITY::AVX2); -DISTANCE_IP_INSTANTIATE_TEMPLATE(768, AVX_AVAILABILITY::AVX2); -DISTANCE_IP_INSTANTIATE_TEMPLATE(Dynamic, AVX_AVAILABILITY::AVX2); - -DISTANCE_CS_INSTANTIATE_TEMPLATE(64, AVX_AVAILABILITY::AVX2); -DISTANCE_CS_INSTANTIATE_TEMPLATE(96, AVX_AVAILABILITY::AVX2); -DISTANCE_CS_INSTANTIATE_TEMPLATE(100, AVX_AVAILABILITY::AVX2); -DISTANCE_CS_INSTANTIATE_TEMPLATE(128, AVX_AVAILABILITY::AVX2); -DISTANCE_CS_INSTANTIATE_TEMPLATE(160, AVX_AVAILABILITY::AVX2); -DISTANCE_CS_INSTANTIATE_TEMPLATE(200, AVX_AVAILABILITY::AVX2); -DISTANCE_CS_INSTANTIATE_TEMPLATE(512, AVX_AVAILABILITY::AVX2); -DISTANCE_CS_INSTANTIATE_TEMPLATE(768, AVX_AVAILABILITY::AVX2); -DISTANCE_CS_INSTANTIATE_TEMPLATE(Dynamic, AVX_AVAILABILITY::AVX2); +// Define every kernel for this ISA level at every generated extent. Extents come +// from cmake/dispatch-surface.cmake, type pairs from multi-arch/x86/preprocessor.h. +#define SVS_DEFINE_FOR_DIM(DIM) SVS_INSTANTIATE_DISTANCES(template, DIM, AVX2) +SVS_FOR_EACH_SUPPORTED_DIM(SVS_DEFINE_FOR_DIM) +#undef SVS_DEFINE_FOR_DIM } // namespace svs::distance diff --git a/include/svs/multi-arch/x86/avx512.cpp b/include/svs/multi-arch/x86/avx512.cpp index bee150d75..cf187c6cb 100644 --- a/include/svs/multi-arch/x86/avx512.cpp +++ b/include/svs/multi-arch/x86/avx512.cpp @@ -16,41 +16,18 @@ #if defined(__x86_64__) #include "svs/core/distance/cosine.h" +#include "svs/core/distance/dispatch_surface.h" #include "svs/core/distance/euclidean.h" #include "svs/core/distance/inner_product.h" +#include "svs/multi-arch/x86/preprocessor.h" namespace svs::distance { -// TODO: connect with dim_supported_list -DISTANCE_L2_INSTANTIATE_TEMPLATE(64, AVX_AVAILABILITY::AVX512); -DISTANCE_L2_INSTANTIATE_TEMPLATE(96, AVX_AVAILABILITY::AVX512); -DISTANCE_L2_INSTANTIATE_TEMPLATE(100, AVX_AVAILABILITY::AVX512); -DISTANCE_L2_INSTANTIATE_TEMPLATE(128, AVX_AVAILABILITY::AVX512); -DISTANCE_L2_INSTANTIATE_TEMPLATE(160, AVX_AVAILABILITY::AVX512); -DISTANCE_L2_INSTANTIATE_TEMPLATE(200, AVX_AVAILABILITY::AVX512); -DISTANCE_L2_INSTANTIATE_TEMPLATE(512, AVX_AVAILABILITY::AVX512); -DISTANCE_L2_INSTANTIATE_TEMPLATE(768, AVX_AVAILABILITY::AVX512); -DISTANCE_L2_INSTANTIATE_TEMPLATE(Dynamic, AVX_AVAILABILITY::AVX512); - -DISTANCE_IP_INSTANTIATE_TEMPLATE(64, AVX_AVAILABILITY::AVX512); -DISTANCE_IP_INSTANTIATE_TEMPLATE(96, AVX_AVAILABILITY::AVX512); -DISTANCE_IP_INSTANTIATE_TEMPLATE(100, AVX_AVAILABILITY::AVX512); -DISTANCE_IP_INSTANTIATE_TEMPLATE(128, AVX_AVAILABILITY::AVX512); -DISTANCE_IP_INSTANTIATE_TEMPLATE(160, AVX_AVAILABILITY::AVX512); -DISTANCE_IP_INSTANTIATE_TEMPLATE(200, AVX_AVAILABILITY::AVX512); -DISTANCE_IP_INSTANTIATE_TEMPLATE(512, AVX_AVAILABILITY::AVX512); -DISTANCE_IP_INSTANTIATE_TEMPLATE(768, AVX_AVAILABILITY::AVX512); -DISTANCE_IP_INSTANTIATE_TEMPLATE(Dynamic, AVX_AVAILABILITY::AVX512); - -DISTANCE_CS_INSTANTIATE_TEMPLATE(64, AVX_AVAILABILITY::AVX512); -DISTANCE_CS_INSTANTIATE_TEMPLATE(96, AVX_AVAILABILITY::AVX512); -DISTANCE_CS_INSTANTIATE_TEMPLATE(100, AVX_AVAILABILITY::AVX512); -DISTANCE_CS_INSTANTIATE_TEMPLATE(128, AVX_AVAILABILITY::AVX512); -DISTANCE_CS_INSTANTIATE_TEMPLATE(160, AVX_AVAILABILITY::AVX512); -DISTANCE_CS_INSTANTIATE_TEMPLATE(200, AVX_AVAILABILITY::AVX512); -DISTANCE_CS_INSTANTIATE_TEMPLATE(512, AVX_AVAILABILITY::AVX512); -DISTANCE_CS_INSTANTIATE_TEMPLATE(768, AVX_AVAILABILITY::AVX512); -DISTANCE_CS_INSTANTIATE_TEMPLATE(Dynamic, AVX_AVAILABILITY::AVX512); +// Define every kernel for this ISA level at every generated extent. Extents come +// from cmake/dispatch-surface.cmake, type pairs from multi-arch/x86/preprocessor.h. +#define SVS_DEFINE_FOR_DIM(DIM) SVS_INSTANTIATE_DISTANCES(template, DIM, AVX512) +SVS_FOR_EACH_SUPPORTED_DIM(SVS_DEFINE_FOR_DIM) +#undef SVS_DEFINE_FOR_DIM } // namespace svs::distance diff --git a/include/svs/multi-arch/x86/avx512_vnni.cpp b/include/svs/multi-arch/x86/avx512_vnni.cpp new file mode 100644 index 000000000..9c0013753 --- /dev/null +++ b/include/svs/multi-arch/x86/avx512_vnni.cpp @@ -0,0 +1,34 @@ +/* + * Copyright 2026 Intel Corporation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#if defined(__x86_64__) +#include "svs/core/distance/cosine.h" +#include "svs/core/distance/dispatch_surface.h" +#include "svs/core/distance/euclidean.h" +#include "svs/core/distance/inner_product.h" +#include "svs/multi-arch/x86/preprocessor.h" + +namespace svs::distance { + +// Define every kernel for this ISA level at every generated extent. Extents come +// from cmake/dispatch-surface.cmake, type pairs from multi-arch/x86/preprocessor.h. +#define SVS_DEFINE_FOR_DIM(DIM) SVS_INSTANTIATE_DISTANCES(template, DIM, AVX512_VNNI) +SVS_FOR_EACH_SUPPORTED_DIM(SVS_DEFINE_FOR_DIM) +#undef SVS_DEFINE_FOR_DIM + +} // namespace svs::distance + +#endif diff --git a/include/svs/multi-arch/x86/preprocessor.h b/include/svs/multi-arch/x86/preprocessor.h index 4e0cb941d..82eccd846 100644 --- a/include/svs/multi-arch/x86/preprocessor.h +++ b/include/svs/multi-arch/x86/preprocessor.h @@ -16,74 +16,78 @@ #pragma once -#define DISTANCE_L2_TEMPLATE_HELPER(SPEC, N, AVX) \ - SPEC struct L2Impl; \ - SPEC struct L2Impl; \ - SPEC struct L2Impl; \ - SPEC struct L2Impl; \ - SPEC struct L2Impl; \ - SPEC struct L2Impl; \ - SPEC struct L2Impl; \ - SPEC struct L2Impl; \ - SPEC struct L2Impl; \ - SPEC struct L2Impl; \ - SPEC struct L2Impl; \ - SPEC struct L2Impl; \ - SPEC struct L2Impl; \ - SPEC struct L2Impl; \ - SPEC struct L2Impl; \ - SPEC struct L2Impl; +#include "svs/core/distance/dispatch_surface.h" -#define DISTANCE_L2_INSTANTIATE_TEMPLATE(N, AVX) \ - DISTANCE_L2_TEMPLATE_HELPER(template, N, AVX); +///// +///// Element-type pairs. +///// +///// Every (query, dataset) combination of float, int8_t, uint8_t and Float16. +///// Each pair costs one instantiation per extent, ISA level and distance. +///// -#define DISTANCE_L2_EXTERN_TEMPLATE(N, AVX) \ - DISTANCE_L2_TEMPLATE_HELPER(extern template, N, AVX); +// Invokes M(query_type, dataset_type, ...) once per type pair. +#define SVS_FOR_EACH_TYPE_PAIR(M, ...) \ + M(float, float, __VA_ARGS__) \ + M(float, int8_t, __VA_ARGS__) \ + M(float, uint8_t, __VA_ARGS__) \ + M(float, svs::float16::Float16, __VA_ARGS__) \ + M(int8_t, float, __VA_ARGS__) \ + M(int8_t, int8_t, __VA_ARGS__) \ + M(int8_t, uint8_t, __VA_ARGS__) \ + M(int8_t, svs::float16::Float16, __VA_ARGS__) \ + M(uint8_t, float, __VA_ARGS__) \ + M(uint8_t, int8_t, __VA_ARGS__) \ + M(uint8_t, uint8_t, __VA_ARGS__) \ + M(uint8_t, svs::float16::Float16, __VA_ARGS__) \ + M(svs::float16::Float16, float, __VA_ARGS__) \ + M(svs::float16::Float16, int8_t, __VA_ARGS__) \ + M(svs::float16::Float16, uint8_t, __VA_ARGS__) \ + M(svs::float16::Float16, svs::float16::Float16, __VA_ARGS__) -#define DISTANCE_IP_TEMPLATE_HELPER(SPEC, N, AVX) \ - SPEC struct IPImpl; \ - SPEC struct IPImpl; \ - SPEC struct IPImpl; \ - SPEC struct IPImpl; \ - SPEC struct IPImpl; \ - SPEC struct IPImpl; \ - SPEC struct IPImpl; \ - SPEC struct IPImpl; \ - SPEC struct IPImpl; \ - SPEC struct IPImpl; \ - SPEC struct IPImpl; \ - SPEC struct IPImpl; \ - SPEC struct IPImpl; \ - SPEC struct IPImpl; \ - SPEC struct IPImpl; \ - SPEC struct IPImpl; +///// +///// Which type pairs each ISA level has kernels for. +///// +///// One line per level in SVS_ISA_LEVELS. A level with no entry here is a +///// compile error rather than a silently empty instantiation list. +///// -#define DISTANCE_IP_INSTANTIATE_TEMPLATE(N, AVX) \ - DISTANCE_IP_TEMPLATE_HELPER(template, N, AVX); +#define SVS_TYPE_PAIRS_NONE SVS_FOR_EACH_TYPE_PAIR +#define SVS_TYPE_PAIRS_AVX2 SVS_FOR_EACH_TYPE_PAIR +#define SVS_TYPE_PAIRS_AVX512 SVS_FOR_EACH_TYPE_PAIR -#define DISTANCE_IP_EXTERN_TEMPLATE(N, AVX) \ - DISTANCE_IP_TEMPLATE_HELPER(extern template, N, AVX); +// Only these two: every other pair promotes to float, where VNNI has nothing to +// offer. svs::distance::has_vnni_kernel is generated from this same list. +#define SVS_TYPE_PAIRS_AVX512_VNNI(M, ...) \ + M(int8_t, int8_t, __VA_ARGS__) \ + M(uint8_t, uint8_t, __VA_ARGS__) -#define DISTANCE_CS_TEMPLATE_HELPER(SPEC, N, AVX) \ - SPEC struct CosineSimilarityImpl; \ - SPEC struct CosineSimilarityImpl; \ - SPEC struct CosineSimilarityImpl; \ - SPEC struct CosineSimilarityImpl; \ - SPEC struct CosineSimilarityImpl; \ - SPEC struct CosineSimilarityImpl; \ - SPEC struct CosineSimilarityImpl; \ - SPEC struct CosineSimilarityImpl; \ - SPEC struct CosineSimilarityImpl; \ - SPEC struct CosineSimilarityImpl; \ - SPEC struct CosineSimilarityImpl; \ - SPEC struct CosineSimilarityImpl; \ - SPEC struct CosineSimilarityImpl; \ - SPEC struct CosineSimilarityImpl; \ - SPEC struct CosineSimilarityImpl; \ - SPEC struct CosineSimilarityImpl; +///// +///// Instantiation. +///// -#define DISTANCE_CS_INSTANTIATE_TEMPLATE(N, AVX) \ - DISTANCE_CS_TEMPLATE_HELPER(template, N, AVX); +// Resolve LEVEL to its type-pair list. The indirection is required so that +// LEVEL is expanded before being pasted. +#define SVS_TYPE_PAIRS_FOR_(LEVEL) SVS_TYPE_PAIRS_##LEVEL +#define SVS_TYPE_PAIRS_FOR(LEVEL, M, ...) SVS_TYPE_PAIRS_FOR_(LEVEL)(M, __VA_ARGS__) -#define DISTANCE_CS_EXTERN_TEMPLATE(N, AVX) \ - DISTANCE_CS_TEMPLATE_HELPER(extern template, N, AVX); +#define SVS_DECLARE_ONE_L2(Ea, Eb, SPEC, N, LEVEL) \ + SPEC struct L2Impl; +#define SVS_DECLARE_ONE_IP(Ea, Eb, SPEC, N, LEVEL) \ + SPEC struct IPImpl; +#define SVS_DECLARE_ONE_CS(Ea, Eb, SPEC, N, LEVEL) \ + SPEC struct CosineSimilarityImpl; + +// SPEC is `template` for a definition or `extern template` for a declaration. +#define SVS_INSTANTIATE_L2(SPEC, N, LEVEL) \ + SVS_TYPE_PAIRS_FOR(LEVEL, SVS_DECLARE_ONE_L2, SPEC, N, LEVEL) +#define SVS_INSTANTIATE_IP(SPEC, N, LEVEL) \ + SVS_TYPE_PAIRS_FOR(LEVEL, SVS_DECLARE_ONE_IP, SPEC, N, LEVEL) +#define SVS_INSTANTIATE_CS(SPEC, N, LEVEL) \ + SVS_TYPE_PAIRS_FOR(LEVEL, SVS_DECLARE_ONE_CS, SPEC, N, LEVEL) + +// All three distances at once. Only usable where all three are declared, i.e. +// in a generated translation unit. +#define SVS_INSTANTIATE_DISTANCES(SPEC, N, LEVEL) \ + SVS_INSTANTIATE_L2(SPEC, N, LEVEL) \ + SVS_INSTANTIATE_IP(SPEC, N, LEVEL) \ + SVS_INSTANTIATE_CS(SPEC, N, LEVEL) diff --git a/tests/AGENTS.md b/tests/AGENTS.md index bee317116..fe455c4cc 100644 --- a/tests/AGENTS.md +++ b/tests/AGENTS.md @@ -1,6 +1,7 @@ # AGENTS.md — tests/ -Unit/integration/benchmark-adjacent tests. +Tests and test harnesses, in whatever language they are written: unit, integration, +benchmark-adjacent, and build-level checks. - Add regressions for bug fixes. - Keep tests deterministic and scoped. diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 8c812d35a..bb6d2a7d8 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -237,3 +237,9 @@ list(APPEND CMAKE_MODULE_PATH ${catch2_SOURCE_DIR}/extras) include(CTest) include(Catch) catch_discover_tests(tests ADD_TAGS_AS_LABELS SKIP_IS_FAILURE) + +# Checks the x86 dispatch surface. The target only exists where the multi-arch +# build ran, which is the same condition that makes the surface meaningful. +if(TARGET svs_x86_objects) + add_subdirectory(multi-arch) +endif() diff --git a/tests/cmake/dispatch-surface/invalid-duplicate-arch.cmake b/tests/cmake/dispatch-surface/invalid-duplicate-arch.cmake new file mode 100644 index 000000000..92ba4a3e5 --- /dev/null +++ b/tests/cmake/dispatch-surface/invalid-duplicate-arch.cmake @@ -0,0 +1,23 @@ +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Each level's -march is its instruction budget. Two levels cannot share one +# -march: the weaker level is compiled with instructions its runtime predicate +# does not guarantee, and hosts routed to it fault. +# EXPECT-ERROR: Duplicate -march +set(SVS_SUPPORTED_DIMS 128) +set(SVS_ISA_LEVELS + "AVX2|haswell|avx2" + "AVX512|haswell|avx512" +) diff --git a/tests/cmake/dispatch-surface/invalid-duplicate-extent.cmake b/tests/cmake/dispatch-surface/invalid-duplicate-extent.cmake new file mode 100644 index 000000000..fe809c05d --- /dev/null +++ b/tests/cmake/dispatch-surface/invalid-duplicate-extent.cmake @@ -0,0 +1,17 @@ +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# EXPECT-ERROR: SVS_SUPPORTED_DIMS contains duplicate extents +set(SVS_SUPPORTED_DIMS 128 256 128) +set(SVS_ISA_LEVELS "AVX2|haswell|avx2") diff --git a/tests/cmake/dispatch-surface/invalid-duplicate-infix.cmake b/tests/cmake/dispatch-surface/invalid-duplicate-infix.cmake new file mode 100644 index 000000000..8e05c06fe --- /dev/null +++ b/tests/cmake/dispatch-surface/invalid-duplicate-infix.cmake @@ -0,0 +1,22 @@ +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Two levels cannot share one translation unit: it is compiled once, at one +# instruction budget. +# EXPECT-ERROR: Duplicate TU infix 'avx2' +set(SVS_SUPPORTED_DIMS 128) +set(SVS_ISA_LEVELS + "AVX2|haswell|avx2" + "AVX512|cascadelake|avx2" +) diff --git a/tests/cmake/dispatch-surface/invalid-duplicate-level.cmake b/tests/cmake/dispatch-surface/invalid-duplicate-level.cmake new file mode 100644 index 000000000..091109839 --- /dev/null +++ b/tests/cmake/dispatch-surface/invalid-duplicate-level.cmake @@ -0,0 +1,21 @@ +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# One enumerator cannot have two instruction budgets. +# EXPECT-ERROR: Duplicate ISA level 'AVX2' +set(SVS_SUPPORTED_DIMS 128) +set(SVS_ISA_LEVELS + "AVX2|haswell|avx2" + "AVX2|cascadelake|avx512" +) diff --git a/tests/cmake/dispatch-surface/invalid-dynamic-listed.cmake b/tests/cmake/dispatch-surface/invalid-dynamic-listed.cmake new file mode 100644 index 000000000..127527fcf --- /dev/null +++ b/tests/cmake/dispatch-surface/invalid-dynamic-listed.cmake @@ -0,0 +1,18 @@ +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# svs::Dynamic is appended automatically and must not be spelled out. +# EXPECT-ERROR: svs::Dynamic is required and is appended automatically +set(SVS_SUPPORTED_DIMS 128 svs::Dynamic) +set(SVS_ISA_LEVELS "AVX2|haswell|avx2") diff --git a/tests/cmake/dispatch-surface/invalid-level-empty-field.cmake b/tests/cmake/dispatch-surface/invalid-level-empty-field.cmake new file mode 100644 index 000000000..5f548ba90 --- /dev/null +++ b/tests/cmake/dispatch-surface/invalid-level-empty-field.cmake @@ -0,0 +1,17 @@ +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# EXPECT-ERROR: arch is empty +set(SVS_SUPPORTED_DIMS 128) +set(SVS_ISA_LEVELS "AVX2||avx2") diff --git a/tests/cmake/dispatch-surface/invalid-level-missing-field.cmake b/tests/cmake/dispatch-surface/invalid-level-missing-field.cmake new file mode 100644 index 000000000..03092a9f0 --- /dev/null +++ b/tests/cmake/dispatch-surface/invalid-level-missing-field.cmake @@ -0,0 +1,18 @@ +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# The TU infix is absent. +# EXPECT-ERROR: Malformed ISA level 'AVX2|haswell': expected ||. +set(SVS_SUPPORTED_DIMS 128) +set(SVS_ISA_LEVELS "AVX2|haswell") diff --git a/tests/cmake/dispatch-surface/invalid-level-none.cmake b/tests/cmake/dispatch-surface/invalid-level-none.cmake new file mode 100644 index 000000000..bca2af2f9 --- /dev/null +++ b/tests/cmake/dispatch-surface/invalid-level-none.cmake @@ -0,0 +1,22 @@ +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# NONE means "no level is present"; it has no translation unit and no object +# library, so a row for it names files that must not exist. +# EXPECT-ERROR: ISA level 'NONE' in SVS_ISA_LEVELS is not declarable +set(SVS_SUPPORTED_DIMS 128) +set(SVS_ISA_LEVELS + "NONE|haswell|avx2" + "AVX512|skylake-avx512|avx512" +) diff --git a/tests/cmake/dispatch-surface/invalid-level-without-tu.cmake b/tests/cmake/dispatch-surface/invalid-level-without-tu.cmake new file mode 100644 index 000000000..65bbe14d3 --- /dev/null +++ b/tests/cmake/dispatch-surface/invalid-level-without-tu.cmake @@ -0,0 +1,22 @@ +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Adding a level means adding the translation unit that defines its kernels. +# Without this check the level would silently contribute nothing. +# EXPECT-ERROR: has no translation unit +set(SVS_SUPPORTED_DIMS 128) +set(SVS_ISA_LEVELS + "AVX2|haswell|avx2" + "AVX512_VNNI|sapphirerapids|avx512vnni" +) diff --git a/tests/cmake/dispatch-surface/invalid-missing-extents.cmake b/tests/cmake/dispatch-surface/invalid-missing-extents.cmake new file mode 100644 index 000000000..b5cfb55a8 --- /dev/null +++ b/tests/cmake/dispatch-surface/invalid-missing-extents.cmake @@ -0,0 +1,17 @@ +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# The variable is never set at all. +# EXPECT-ERROR: SVS_SUPPORTED_DIMS is empty +set(SVS_ISA_LEVELS "AVX2|haswell|avx2") diff --git a/tests/cmake/dispatch-surface/invalid-missing-mandatory-level.cmake b/tests/cmake/dispatch-surface/invalid-missing-mandatory-level.cmake new file mode 100644 index 000000000..37df38a28 --- /dev/null +++ b/tests/cmake/dispatch-surface/invalid-missing-mandatory-level.cmake @@ -0,0 +1,19 @@ +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# distance_core.h #errors on x86_64 unless both AVX2 and AVX512 are present, so +# a surface declaring only one of them cannot compile. +# EXPECT-ERROR: SVS_ISA_LEVELS omits mandatory level 'AVX512' +set(SVS_SUPPORTED_DIMS 128) +set(SVS_ISA_LEVELS "AVX2|haswell|avx2") diff --git a/tests/cmake/dispatch-surface/invalid-no-extents.cmake b/tests/cmake/dispatch-surface/invalid-no-extents.cmake new file mode 100644 index 000000000..d303cde71 --- /dev/null +++ b/tests/cmake/dispatch-surface/invalid-no-extents.cmake @@ -0,0 +1,17 @@ +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# EXPECT-ERROR: SVS_SUPPORTED_DIMS is empty +set(SVS_SUPPORTED_DIMS) +set(SVS_ISA_LEVELS "AVX2|haswell|avx2") diff --git a/tests/cmake/dispatch-surface/invalid-no-levels.cmake b/tests/cmake/dispatch-surface/invalid-no-levels.cmake new file mode 100644 index 000000000..c3c2b4a98 --- /dev/null +++ b/tests/cmake/dispatch-surface/invalid-no-levels.cmake @@ -0,0 +1,17 @@ +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# EXPECT-ERROR: SVS_ISA_LEVELS is empty +set(SVS_SUPPORTED_DIMS 128) +set(SVS_ISA_LEVELS) diff --git a/tests/cmake/dispatch-surface/invalid-non-numeric-extent.cmake b/tests/cmake/dispatch-surface/invalid-non-numeric-extent.cmake new file mode 100644 index 000000000..2af432451 --- /dev/null +++ b/tests/cmake/dispatch-surface/invalid-non-numeric-extent.cmake @@ -0,0 +1,17 @@ +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# EXPECT-ERROR: SVS_SUPPORTED_DIMS contains '12a' +set(SVS_SUPPORTED_DIMS 128 12a) +set(SVS_ISA_LEVELS "AVX2|haswell|avx2") diff --git a/tests/cmake/dispatch-surface/invalid-unknown-level.cmake b/tests/cmake/dispatch-surface/invalid-unknown-level.cmake new file mode 100644 index 000000000..2ba95e81e --- /dev/null +++ b/tests/cmake/dispatch-surface/invalid-unknown-level.cmake @@ -0,0 +1,22 @@ +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# A level name that is not an AVX_AVAILABILITY enumerator only fails once the +# generated C++ tries to compile it; the validator must reject it sooner. +# EXPECT-ERROR: Unknown ISA level 'BOGUS' +set(SVS_SUPPORTED_DIMS 128) +set(SVS_ISA_LEVELS + "BOGUS|haswell|avx2" + "AVX512|skylake-avx512|avx512" +) diff --git a/tests/cmake/dispatch-surface/invalid-zero-extent.cmake b/tests/cmake/dispatch-surface/invalid-zero-extent.cmake new file mode 100644 index 000000000..c74f51e90 --- /dev/null +++ b/tests/cmake/dispatch-surface/invalid-zero-extent.cmake @@ -0,0 +1,18 @@ +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# A zero-length vector has no kernel to compile. +# EXPECT-ERROR: SVS_SUPPORTED_DIMS contains '0' +set(SVS_SUPPORTED_DIMS 0 128) +set(SVS_ISA_LEVELS "AVX2|haswell|avx2") diff --git a/tests/cmake/dispatch-surface/valid-minimal.cmake b/tests/cmake/dispatch-surface/valid-minimal.cmake new file mode 100644 index 000000000..936776395 --- /dev/null +++ b/tests/cmake/dispatch-surface/valid-minimal.cmake @@ -0,0 +1,23 @@ +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# The smallest declaration that is still a library: one fixed extent and the +# two mandatory ISA levels. Every other dimensionality is served by the +# svs::Dynamic kernel. + +set(SVS_SUPPORTED_DIMS 128) +set(SVS_ISA_LEVELS + "AVX2|haswell|avx2" + "AVX512|skylake-avx512|avx512" +) diff --git a/tests/cmake/dispatch-surface/valid-reduced.cmake b/tests/cmake/dispatch-surface/valid-reduced.cmake new file mode 100644 index 000000000..517e26007 --- /dev/null +++ b/tests/cmake/dispatch-surface/valid-reduced.cmake @@ -0,0 +1,30 @@ +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# A surface that shares no extent with the default declaration, so a build using +# it cannot accidentally pass by reusing a committed header. +# +# The extent list is what this fixture varies. The ISA levels are copied from the +# default declaration verbatim, instruction budgets included: they are not +# configuration, and a build with a different level set would be testing a library +# nobody ships. +# +# Built and tested by the `non-default surface` CI job. + +set(SVS_SUPPORTED_DIMS 32 384) +set(SVS_ISA_LEVELS + "AVX2|haswell|avx2" + "AVX512|skylake-avx512|avx512" + "AVX512_VNNI|cascadelake|avx512_vnni" +) diff --git a/tests/multi-arch/CMakeLists.txt b/tests/multi-arch/CMakeLists.txt new file mode 100644 index 000000000..e8ac89bd3 --- /dev/null +++ b/tests/multi-arch/CMakeLists.txt @@ -0,0 +1,140 @@ +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +##### +##### Tests of the x86 dispatch surface. +##### +##### Not Catch2 tests: what is being checked is a property of an object file or of +##### a process, so each probe has to be its own translation unit, compiled at a +##### known instruction budget and inspected from outside. +##### +##### Not PROJECT_SOURCE_DIR anywhere below: the downstream repository adds this +##### directory to its own project, where that points somewhere else entirely. +##### + +include("${CMAKE_CURRENT_LIST_DIR}/../../cmake/dispatch-levels.cmake") + +set(svs_dispatch_cmake_dir "${CMAKE_CURRENT_LIST_DIR}/dispatch-checks") + +# The object libraries exist so that each probe's single object file can be named +# with $, which is what the symbol-table checks read. +# +# svs::x86_options_base is -march=x86-64 -mtune=generic: a probe must know no more +# about the host than an arbitrary consumer of the headers does. +function(svs_add_dispatch_probe name source) + add_library(${name}_objects OBJECT ${source}) + target_link_libraries( + ${name}_objects PRIVATE svs::svs svs::compile_options svs::x86_options_base + ) + add_executable(${name}) + target_link_libraries( + ${name} + PRIVATE ${name}_objects svs::svs svs::compile_options svs::x86_options_base + ) +endfunction() + +# Names every kernel the surface declares and nothing else, so a declared-but- +# uninstantiated kernel is an undefined symbol here. +svs_add_dispatch_probe(link_probe x86/link_probe.cpp) + +# Reaches the kernels through the entry points instead, so it also sees whether the +# entry points route to the whole surface. +svs_add_dispatch_probe(entry_probe x86/entry_probe.cpp) + +# Calls every kernel whose ISA level this host satisfies. Which kernels those are +# depends on the host, so this covers the whole surface only across the CI matrix. +add_test(NAME dispatch_link_probe COMMAND link_probe) +add_test(NAME dispatch_entry_probe COMMAND entry_probe) + +if(DEFINED CMAKE_NM AND CMAKE_NM) + set(svs_nm "${CMAKE_NM}") +else() + find_program(svs_nm NAMES nm llvm-nm) +endif() +if(DEFINED CMAKE_OBJDUMP AND CMAKE_OBJDUMP) + set(svs_objdump "${CMAKE_OBJDUMP}") +else() + find_program(svs_objdump NAMES objdump llvm-objdump) +endif() +find_program(svs_gdb NAMES gdb) + +if(svs_nm) + # Host-independent, unlike the runs above: it reads the symbol table rather + # than executing anything, so it sees every kernel in the surface everywhere. + add_test( + NAME dispatch_surface_linkage + COMMAND + "${CMAKE_COMMAND}" + "-DSVS_PROBE_OBJECT=$" + "-DSVS_ARCHIVE=$" + "-DSVS_NM=${svs_nm}" + -P "${svs_dispatch_cmake_dir}/check-dispatch-linkage.cmake" + ) + + # The same symbols, counted from the declaration instead of from a probe built + # out of the same generated header -- the one check a generator bug cannot pass + # by being wrong consistently. + add_test( + NAME dispatch_surface_declaration + COMMAND + "${CMAKE_COMMAND}" + "-DSVS_MANIFEST=${CMAKE_BINARY_DIR}/dispatch_surface.manifest.cmake" + "-DSVS_TYPE_PAIR_HEADER=${CMAKE_CURRENT_LIST_DIR}/../../include/svs/multi-arch/x86/preprocessor.h" + "-DSVS_ENUM_HEADER=${CMAKE_CURRENT_LIST_DIR}/../../include/svs/core/distance/distance_core.h" + "-DSVS_ARCHIVE=$" + "-DSVS_CONSUMER_OBJECT=$" + "-DSVS_NM=${svs_nm}" + -P "${svs_dispatch_cmake_dir}/check-dispatch-declaration.cmake" + ) +else() + message(STATUS "nm not found; skipping the dispatch surface symbol-table tests") +endif() + +if(svs_objdump) + # One test per ISA level: a level whose object file carries instructions its + # runtime predicate does not guarantee faults on a host the dispatcher routes + # to it, which no symbol-table check can see. + foreach(tu_spec IN LISTS SVS_DISPATCH_TU_SPECS) + svs_parse_tu_spec("${tu_spec}" svs_src svs_level svs_arch svs_infix) + add_test( + NAME dispatch_instructions_${svs_infix} + COMMAND + "${CMAKE_COMMAND}" + "-DSVS_OBJECT=$" + "-DSVS_LEVEL=${svs_level}" + "-DSVS_ARCH=${svs_arch}" + "-DSVS_OBJDUMP=${svs_objdump}" + "-DSVS_NO_AVX512=${SVS_NO_AVX512}" + -P "${svs_dispatch_cmake_dir}/check-dispatch-instructions.cmake" + ) + endforeach() +else() + message(STATUS "objdump not found; skipping the dispatch surface instruction tests") +endif() + +if(svs_nm AND svs_gdb) + # The only check that observes a kernel run rather than exist: a specialization + # lost behind an `#if` still links and still counts. + add_test( + NAME dispatch_surface_execution + COMMAND + "${CMAKE_COMMAND}" + "-DSVS_PROBE=$" + "-DSVS_NM=${svs_nm}" + "-DSVS_GDB=${svs_gdb}" + -P "${svs_dispatch_cmake_dir}/check-dispatch-execution.cmake" + ) +else() + message(STATUS "nm or gdb not found; skipping the dispatch surface execution test") +endif() diff --git a/tests/multi-arch/dispatch-checks/README.md b/tests/multi-arch/dispatch-checks/README.md new file mode 100644 index 000000000..1a5a48160 --- /dev/null +++ b/tests/multi-arch/dispatch-checks/README.md @@ -0,0 +1,87 @@ + + +# Dispatch surface checks + +These scripts are run by ctest to hold the built binary to the surface +declared in `cmake/dispatch-surface.cmake`. They are not part of building +the library. See `cmake/dispatch-surface.cmake` for documentation on the +declaration format, the current extent list and ISA levels, and how to add +a new level. + +These tests are only added to the build when the x86 object libraries +exist. Run `ctest` from `/tests`, not from the build root. + +## Why these are not Catch2 cases + +Everything else under `tests/` is a Catch2 case registered with ctest by +`catch_discover_tests`. These are registered with `add_test` instead, and the +two probes under `../x86/` contain no Catch2 macros. That is deliberate for the +probes and provisional for the checkers. + +`link_probe.cpp` cannot be a Catch2 case because *its test is the link step*: it +names every kernel the surface declares and nothing else, so a declared but +uninstantiated kernel is an undefined symbol and the probe fails to link. Folding +it into the single `tests` binary would turn a surface bug into a whole-suite link +failure, and would stop `nm --undefined-only` from isolating the surface's symbols +from the hundred other objects. `entry_probe.cpp` is driven under gdb, which needs +its own small executable with predictable breakpoints. + +The four `cmake -P` checkers have no such excuse. They are scripts because they +shell out to `nm` and `objdump` and were easiest to write that way. Porting them +to Catch2 cases that shell out to the same tools is planned separately; it would +get real data structures and unit-testable helpers without adding a Python +dependency to the C++ build. + +## check-dispatch-linkage.cmake + +**Test:** `dispatch_surface_linkage` + +Verifies that every kernel declared in the surface is defined in the +archive, that the archive defines no undeclared kernels, and that a test +probe naming every kernel defines none of its own. Failure means a +declared kernel is missing or an extra kernel was built but not declared. + +## check-dispatch-declaration.cmake + +**Test:** `dispatch_surface_declaration` + +Verifies that the symbol count derived from the surface declaration matches +the symbol count in the archive and in a consumer object that names every +entry point. Failure means the declaration's extent list, ISA levels, type +pairs, or distance enumerator order disagrees with the built kernels. This +catches generator bugs that would pass linkage checks by being wrong +consistently in both the archive and the probe. + +## check-dispatch-instructions.cmake + +**Test:** `dispatch_instructions_` (one test per ISA level) + +Verifies that each level's object file stays within its instruction budget, +emitting only instructions the level's runtime predicate guarantees the host +supports. Failure means an object file contains instructions not guaranteed +by its level's predicate, causing illegal-instruction faults on hosts the +dispatcher routes to that level. + +## check-dispatch-execution.cmake + +**Test:** `dispatch_surface_execution` + +Verifies that a call through the entry points on the test host enters the +ISA level the host satisfies, observed by breaking in gdb on each level's +kernel and reporting which one runs. Failure means runtime dispatch routes +to the wrong level, or a specialization disappeared behind a preprocessor +guard while still linking and counting in the symbol table. diff --git a/tests/multi-arch/dispatch-checks/check-dispatch-declaration.cmake b/tests/multi-arch/dispatch-checks/check-dispatch-declaration.cmake new file mode 100644 index 000000000..7f67daca8 --- /dev/null +++ b/tests/multi-arch/dispatch-checks/check-dispatch-declaration.cmake @@ -0,0 +1,291 @@ +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +##### +##### Checks the built kernels against the declaration, not against the generated +##### header. +##### +##### Run in script mode: +##### +##### cmake -DSVS_MANIFEST=build/dispatch_surface.manifest.cmake \ +##### -DSVS_TYPE_PAIR_HEADER=include/svs/multi-arch/x86/preprocessor.h \ +##### -DSVS_ENUM_HEADER=include/svs/core/distance/distance_core.h \ +##### -DSVS_ARCHIVE= \ +##### -DSVS_CONSUMER_OBJECT= \ +##### -DSVS_NM= \ +##### -P tests/multi-arch/dispatch-checks/check-dispatch-declaration.cmake +##### +##### tests/multi-arch/dispatch-checks/check-dispatch-linkage.cmake compares the archive +##### against a probe, and both are generated from the same header: a generator +##### that dropped an extent would drop it from both and still agree. The +##### expectation here is derived from the three hand-written sources instead -- +##### the extent list and levels from the manifest, the type pairs, and the enumerator +##### order -- so the generated header is not consulted at all and a generator bug +##### has nowhere to hide. +##### + +# Without it, CMP0057 is unset in script mode and the `IN_LIST` tests below are +# rejected as unknown arguments. +cmake_minimum_required(VERSION 3.21) + +include("${CMAKE_CURRENT_LIST_DIR}/lib.cmake") + +svs_require(SVS_MANIFEST SVS_TYPE_PAIR_HEADER SVS_ENUM_HEADER SVS_ARCHIVE SVS_CONSUMER_OBJECT SVS_NM) +svs_require_files(SVS_MANIFEST SVS_TYPE_PAIR_HEADER SVS_ENUM_HEADER SVS_ARCHIVE SVS_CONSUMER_OBJECT) + +##### +##### Ground truth 1: the extents and the ISA levels +##### + +include("${SVS_MANIFEST}") + +if(NOT SVS_MANIFEST_FIXED_EXTENTS OR NOT SVS_MANIFEST_LEVELS) + message(FATAL_ERROR + "${SVS_MANIFEST} is not a dispatch-surface manifest; it does not declare " + "extents or ISA levels." + ) +endif() + +# svs::Dynamic mangles as its numeric value: the extent is a size_t template +# argument, and Dynamic is SIZE_MAX. +set(svs_expected_extents ${SVS_MANIFEST_FIXED_EXTENTS} "18446744073709551615") +set(svs_expected_levels ${SVS_MANIFEST_LEVELS}) + +##### +##### Ground truth 2: the enumerator order, which fixes the mangled digit +##### + +file(READ "${SVS_ENUM_HEADER}" svs_enum_text) +if(NOT svs_enum_text MATCHES "enum class AVX_AVAILABILITY[ \t\r\n]*{([^}]*)}") + message(FATAL_ERROR + "No `enum class AVX_AVAILABILITY` found in ${SVS_ENUM_HEADER}. Its " + "enumerator order is what maps an ISA level onto the digit in a mangled " + "name, so it cannot be inferred." + ) +endif() +string(REPLACE "," ";" svs_enumerators "${CMAKE_MATCH_1}") +set(svs_enum_index 0) +foreach(enumerator IN LISTS svs_enumerators) + string(STRIP "${enumerator}" enumerator) + if(enumerator) + set(svs_digit_of_${enumerator} ${svs_enum_index}) + math(EXPR svs_enum_index "${svs_enum_index} + 1") + endif() +endforeach() + +set(svs_expected_digits) +foreach(level IN LISTS svs_expected_levels) + if(NOT DEFINED svs_digit_of_${level}) + message(FATAL_ERROR + "ISA level '${level}' is declared in ${SVS_MANIFEST} but is not an " + "AVX_AVAILABILITY enumerator." + ) + endif() + list(APPEND svs_expected_digits ${svs_digit_of_${level}}) +endforeach() + +##### +##### Ground truth 3: how many type pairs each level has kernels for +##### + +file(READ "${SVS_TYPE_PAIR_HEADER}" svs_pairs_text) +# Fold line continuations away so each #define is one line to match against. +string(REGEX REPLACE "\\\\[ \t]*\r?\n" " " svs_pairs_text "${svs_pairs_text}") + +# Returns the number of M(...) invocations in a type-pair list macro, following one +# level of aliasing -- `#define SVS_TYPE_PAIRS_AVX2 SVS_FOR_EACH_TYPE_PAIR`. +function(svs_type_pair_count out_var macro_name) + if(NOT svs_pairs_text MATCHES "#define ${macro_name}(\\([^)]*\\))?[ \t]+([^\n]*)") + message(FATAL_ERROR + "No `#define ${macro_name}` in ${SVS_TYPE_PAIR_HEADER}. Every ISA level " + "needs one; without it the level instantiates nothing." + ) + endif() + set(body "${CMAKE_MATCH_2}") + string(STRIP "${body}" body) + if(body MATCHES "^[A-Za-z_][A-Za-z0-9_]*$") + svs_type_pair_count(count "${body}") + set(${out_var} ${count} PARENT_SCOPE) + return() + endif() + string(REGEX MATCHALL "M\\(" hits "${body}") + list(LENGTH hits count) + if(count EQUAL 0) + message(FATAL_ERROR "${macro_name} in ${SVS_TYPE_PAIR_HEADER} invokes M zero times.") + endif() + set(${out_var} ${count} PARENT_SCOPE) +endfunction() + +##### +##### Ground truth 4: the distances +##### + +# Adding a distance means adding its Impl class here; otherwise its kernels are +# counted by nothing and a level could ship without them. +set(svs_distance_classes "L2Impl" "IPImpl" "CosineSimilarityImpl") + +set(svs_mangled_classes) +foreach(class IN LISTS svs_distance_classes) + string(LENGTH "${class}" len) + list(APPEND svs_mangled_classes "${len}${class}") +endforeach() + +##### +##### The expectation +##### + +# Keyed __; the count is the number of type pairs, +# one symbol each. +set(svs_expected_keys) +set(svs_expected_total 0) +foreach(level IN LISTS svs_expected_levels) + svs_type_pair_count(pairs "SVS_TYPE_PAIRS_${level}") + set(svs_pairs_of_${level} ${pairs}) + foreach(class IN LISTS svs_mangled_classes) + foreach(extent IN LISTS svs_expected_extents) + set(key "${class}_${extent}_${svs_digit_of_${level}}") + list(APPEND svs_expected_keys "${key}") + set(svs_want_${key} ${pairs}) + math(EXPR svs_expected_total "${svs_expected_total} + ${pairs}") + endforeach() + endforeach() +endforeach() + +##### +##### What was built +##### + +# Mangled names throughout: demangled ones carry `[clone .isra.0]` suffixes that +# differ between a local instantiation and an explicit one. +set(svs_kernel_regex "^_ZN3svs8distance([0-9]+[A-Za-z0-9_]+)ILm([0-9]+)E.*AVX_AVAILABILITYE([0-9]+)E") +set(svs_distance_regex "_ZN3svs8distance.*AVX_AVAILABILITY") + +# Sets _KEYS, _FOREIGN and _HAVE_ in the caller. +function(svs_tally prefix symbols) + set(keys) + set(foreign) + foreach(symbol IN LISTS symbols) + if(NOT symbol MATCHES "${svs_kernel_regex}") + continue() + endif() + set(key "${CMAKE_MATCH_1}_${CMAKE_MATCH_2}_${CMAKE_MATCH_3}") + if(NOT key IN_LIST svs_expected_keys) + list(APPEND foreign "${symbol}") + continue() + endif() + if(NOT key IN_LIST keys) + list(APPEND keys "${key}") + set(${prefix}_HAVE_${key} 0) + endif() + math(EXPR next "${${prefix}_HAVE_${key}} + 1") + set(${prefix}_HAVE_${key} ${next}) + set(${prefix}_HAVE_${key} ${next} PARENT_SCOPE) + endforeach() + set(${prefix}_KEYS "${keys}" PARENT_SCOPE) + set(${prefix}_FOREIGN "${foreign}" PARENT_SCOPE) +endfunction() + +svs_nm_symbols(archive_defines "${SVS_ARCHIVE}" "${svs_distance_regex}" NM_ARGS --defined-only) +svs_nm_symbols(consumer_defines "${SVS_CONSUMER_OBJECT}" "${svs_distance_regex}" NM_ARGS --defined-only) +svs_nm_symbols(consumer_references "${SVS_CONSUMER_OBJECT}" "${svs_distance_regex}" NM_ARGS --undefined-only) + +svs_tally(ARCHIVE "${archive_defines}") +svs_tally(CONSUMER "${consumer_references}") + +##### +##### Compare +##### + +set(errors 0) + +# Sets _ERRORS in the caller. `what` names the thing being compared, for +# the failure message. +function(svs_compare prefix what) + set(wrong) + foreach(key IN LISTS svs_expected_keys) + set(have 0) + if(DEFINED ${prefix}_HAVE_${key}) + set(have ${${prefix}_HAVE_${key}}) + endif() + if(NOT have EQUAL ${svs_want_${key}}) + list(APPEND wrong "${key}: expected ${svs_want_${key}}, got ${have}") + endif() + endforeach() + list(LENGTH wrong n_wrong) + set(${prefix}_ERRORS 0 PARENT_SCOPE) + if(NOT n_wrong EQUAL 0) + message("${n_wrong} of ${what} disagree with the manifest.") + message("Each entry is __. A count of 0 means") + message("the declaration asks for kernels that were never built; a count") + message("below the number of type pairs means the level is missing some.") + message(" Mismatched:") + svs_report_first(wrong 10) + set(${prefix}_ERRORS 1 PARENT_SCOPE) + endif() +endfunction() + +svs_compare(ARCHIVE "the archive's kernel groups") +svs_compare(CONSUMER "the kernel groups a consumer reaches through the entry points") +math(EXPR errors "${ARCHIVE_ERRORS} + ${CONSUMER_ERRORS}") + +if(ARCHIVE_FOREIGN) + list(LENGTH ARCHIVE_FOREIGN n) + message("${n} kernels in the archive are outside the declared surface.") + message("Nothing in the manifest asks for them, so no consumer reaches") + message("them and they only add to the size of the library.") + message(" Outside the surface:") + svs_report_first(ARCHIVE_FOREIGN 10) + math(EXPR errors "${errors} + 1") +endif() + +# The whole point of the extern declarations. A kernel the consumer defines at a +# declared level came from the generic primary template at the consumer's own +# -march instead of from the library. +set(consumer_leaks) +foreach(symbol IN LISTS consumer_defines) + if(symbol MATCHES "${svs_kernel_regex}") + if(CMAKE_MATCH_3 IN_LIST svs_expected_digits) + list(APPEND consumer_leaks "${symbol}") + endif() + endif() +endforeach() +list(LENGTH consumer_leaks n_leaks) +if(NOT n_leaks EQUAL 0) + message("${n_leaks} kernels at a declared ISA level are instantiated by the consumer.") + message("Each is missing its `extern template` declaration, so every consumer of") + message("the headers builds it locally, from the generic scalar template, at") + message("whatever -march that consumer uses.") + message(" Instantiated locally:") + svs_report_first(consumer_leaks 10) + math(EXPR errors "${errors} + 1") +endif() + +if(NOT errors EQUAL 0) + message(FATAL_ERROR "dispatch declaration check failed") +endif() + +list(LENGTH svs_expected_levels n_levels) +list(LENGTH svs_expected_extents n_extents) +list(LENGTH svs_distance_classes n_distances) +set(pair_display) +foreach(level IN LISTS svs_expected_levels) + list(APPEND pair_display "${level}: ${svs_pairs_of_${level}}") +endforeach() +string(REPLACE ";" ", " pair_display "${pair_display}") +message( + "dispatch declaration: ${svs_expected_total} kernels required by " + "the manifest (${n_levels} levels x ${n_extents} extents x " + "${n_distances} distances x type pairs per level {${pair_display}}); the " + "archive defines exactly those and the entry points reach all of them" +) diff --git a/tests/multi-arch/dispatch-checks/check-dispatch-execution.cmake b/tests/multi-arch/dispatch-checks/check-dispatch-execution.cmake new file mode 100644 index 000000000..6d86e7171 --- /dev/null +++ b/tests/multi-arch/dispatch-checks/check-dispatch-execution.cmake @@ -0,0 +1,142 @@ +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +##### +##### Checks which ISA level a call through the entry points actually enters. +##### +##### Run in script mode: +##### +##### cmake -DSVS_PROBE= -DSVS_NM= -DSVS_GDB= \ +##### -P tests/multi-arch/dispatch-checks/check-dispatch-execution.cmake +##### +##### Everything else about the surface is a property of the symbol table, which a +##### specialization can satisfy while never running: one that disappears behind an +##### `#if` still links and still counts. This breaks on every level's kernel for one +##### extent and type pair, and reports the one the host's run reaches. +##### +##### Only the level this host satisfies is checked. The weaker levels are checked by +##### hosts that satisfy only those, which is what the CI matrix is for. +##### + +include("${CMAKE_CURRENT_LIST_DIR}/lib.cmake") + +svs_require(SVS_PROBE SVS_NM SVS_GDB) +svs_require_files(SVS_PROBE) + +# The probe owns the runtime predicates, so it -- not this script -- decides which +# level the entry points are obliged to choose here. +execute_process( + COMMAND "${SVS_PROBE}" --report + OUTPUT_VARIABLE report + ERROR_VARIABLE err + RESULT_VARIABLE status +) +if(NOT status EQUAL 0) + message(FATAL_ERROR "${SVS_PROBE} --report failed: ${err}") +endif() +if(NOT report MATCHES "expect-level ([0-9]+)") + message(FATAL_ERROR "${SVS_PROBE} --report printed no expected level:\n${report}") +endif() +set(expected_digit "${CMAKE_MATCH_1}") +if(NOT report MATCHES "probe-extent ([0-9]+)") + message(FATAL_ERROR "${SVS_PROBE} --report printed no probe extent:\n${report}") +endif() +set(extent "${CMAKE_MATCH_1}") + +if(expected_digit EQUAL 0) + message( + "dispatch execution: this host satisfies no ISA level in the surface, so the " + "entry points can only reach AVX_AVAILABILITY::NONE; nothing to check" + ) + return() +endif() + +##### +##### The candidate symbols +##### + +# The type pair the probe calls, Itanium-mangled. It must be one with a kernel at +# every level in the surface, or the entry points legitimately route lower. +set(svs_probe_pair "aa") + +# L2 at that pair for the extent the probe reports: one symbol per level, including +# any AVX_AVAILABILITY::NONE fallback the consumer instantiated itself. +set(svs_symbol_regex "_ZN3svs8distance6L2ImplILm${extent}E${svs_probe_pair}.*7computeE") +svs_nm_symbols(raw_symbols "${SVS_PROBE}" "${svs_symbol_regex}" NM_ARGS --defined-only) + +set(candidates) +foreach(symbol IN LISTS raw_symbols) + # Skip GCC's `.isra` clones: gdb reads a dot in a linespec as a file name. + if(symbol MATCHES "\\.") + continue() + endif() + list(APPEND candidates "${symbol}") +endforeach() +list(REMOVE_DUPLICATES candidates) + +list(LENGTH candidates n_candidates) +if(n_candidates EQUAL 0) + message(FATAL_ERROR + "No L2 kernel symbol for extent ${extent} at the mangled type pair " + "'${svs_probe_pair}' is defined in ${SVS_PROBE}. Either the entry points " + "inlined every kernel, in which case the `extern template` declarations are " + "not in effect, or the probe no longer calls that extent and pair." + ) +endif() + +##### +##### Run under the debugger +##### + +set(gdb_args -q -batch) +foreach(symbol IN LISTS candidates) + list(APPEND gdb_args -ex "break ${symbol}") +endforeach() +list(APPEND gdb_args -ex "run --report" -ex "info symbol $pc") + +execute_process( + COMMAND "${SVS_GDB}" ${gdb_args} "${SVS_PROBE}" + OUTPUT_VARIABLE gdb_out + ERROR_VARIABLE gdb_err + RESULT_VARIABLE status +) +if(NOT status EQUAL 0) + message("${gdb_out}") + message(FATAL_ERROR "${SVS_GDB} failed on ${SVS_PROBE}: ${gdb_err}") +endif() + +# gdb demangles for `info symbol`, so the level appears as the enum's underlying +# value: `(svs::distance::AVX_AVAILABILITY)2`. +if(NOT gdb_out MATCHES "AVX_AVAILABILITY\\)([0-9]+)") + message("${gdb_out}") + message(FATAL_ERROR + "The run never entered any of the ${n_candidates} kernels breakpointed for " + "extent ${extent}. A call through the entry points reached none of them, so " + "the dispatch is not routing to the surface." + ) +endif() +set(actual_digit "${CMAKE_MATCH_1}") + +if(NOT actual_digit EQUAL expected_digit) + message("The entry points routed a call to AVX_AVAILABILITY level ${actual_digit},") + message("but this host satisfies level ${expected_digit}. Either the runtime") + message("predicates in the entry points disagree with the ones the probe uses, or") + message("the enumerator order in the surface no longer matches the dispatch order.") + message(FATAL_ERROR "dispatch execution check failed") +endif() + +message( + "dispatch execution: a call to L2 at extent ${extent} entered " + "AVX_AVAILABILITY level ${actual_digit}, the highest this host satisfies" +) diff --git a/tests/multi-arch/dispatch-checks/check-dispatch-instructions.cmake b/tests/multi-arch/dispatch-checks/check-dispatch-instructions.cmake new file mode 100644 index 000000000..812aa4a22 --- /dev/null +++ b/tests/multi-arch/dispatch-checks/check-dispatch-instructions.cmake @@ -0,0 +1,197 @@ +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +##### +##### Checks that an ISA level's object file stays inside its instruction budget. +##### +##### Run in script mode: +##### +##### cmake -DSVS_OBJECT= \ +##### -DSVS_LEVEL=AVX2 -DSVS_ARCH=haswell \ +##### -DSVS_OBJDUMP= \ +##### [-DSVS_NO_AVX512=] \ +##### -P tests/multi-arch/dispatch-checks/check-dispatch-instructions.cmake +##### +##### A level promises the host satisfies its runtime predicate and nothing more, +##### so an instruction the predicate does not guarantee is an illegal-instruction +##### fault on a host the dispatcher considers supported. +##### +##### SVS_NO_AVX512 mirrors the build's own option of that name (cmake/options.cmake): +##### -mno-avx512f beats a level's later -march= on GCC, so an AVX-512-bearing level +##### then gets the generic scalar fallback instead of its wide kernels, and the +##### budget below has to flip from requiring AVX-512 to forbidding it. +##### + +# Without it, CMP0007 is unset in script mode and the empty "forbids nothing" +# field of a budget row below is silently dropped rather than read as empty. +cmake_minimum_required(VERSION 3.21) + +include("${CMAKE_CURRENT_LIST_DIR}/lib.cmake") + +svs_require(SVS_OBJECT SVS_LEVEL SVS_ARCH SVS_OBJDUMP) +svs_require_files(SVS_OBJECT) + +# Optional: absent callers (and any caller predating this option) get the +# default build's budget, matching cmake/options.cmake's own OFF default. +if(NOT DEFINED SVS_NO_AVX512) + set(SVS_NO_AVX512 OFF) +endif() + +# What each instruction class looks like in AT&T disassembly. Register classes are +# matched with their `%` sigil so that a mangled name can never look like one. +set(svs_class_ymm "%ymm") +set(svs_class_zmm "%zmm") +set(svs_class_mask "%k[1-7]") +set(svs_class_vnni "vpdpwssd|vpdpbusd|vpdpwssds|vpdpbusds") + +# "<-march>||", one row per instruction budget +# any ISA level declares. A budget with no row here is a hard error: an +# unrecognized -march silently permitting everything is how a level ends up +# emitting instructions its runtime predicate does not guarantee. +set(svs_budget_table + "x86-64||ymm zmm mask vnni" + "haswell|ymm|zmm mask vnni" + "skylake-avx512|zmm|vnni" + "cascadelake|zmm vnni|" +) + +# Only the arches whose default budget above requires AVX-512: -mno-avx512f +# does not touch haswell, so AVX2 keeps its normal row and has no entry here. +set(svs_noavx512_budget_table + "skylake-avx512||zmm mask vnni" + "cascadelake||zmm mask vnni" +) + +set(svs_budget_found FALSE) +set(svs_noavx512_active FALSE) +if(SVS_NO_AVX512) + foreach(row IN LISTS svs_noavx512_budget_table) + string(REPLACE "|" ";" fields "${row}") + list(GET fields 0 arch) + if(arch STREQUAL SVS_ARCH) + list(GET fields 1 svs_required) + list(GET fields 2 svs_forbidden) + set(svs_budget_found TRUE) + set(svs_noavx512_active TRUE) + break() + endif() + endforeach() +endif() +if(NOT svs_budget_found) + foreach(row IN LISTS svs_budget_table) + string(REPLACE "|" ";" fields "${row}") + list(GET fields 0 arch) + if(arch STREQUAL SVS_ARCH) + list(GET fields 1 svs_required) + list(GET fields 2 svs_forbidden) + set(svs_budget_found TRUE) + break() + endif() + endforeach() +endif() +if(NOT svs_budget_found) + message(FATAL_ERROR + "No instruction budget is known for -march=${SVS_ARCH} (ISA level " + "${SVS_LEVEL}). Add a row to svs_budget_table in this file stating what " + "that budget requires and forbids." + ) +endif() +string(REPLACE " " ";" svs_required "${svs_required}") +string(REPLACE " " ";" svs_forbidden "${svs_forbidden}") + +execute_process( + COMMAND "${SVS_OBJDUMP}" -d --no-show-raw-insn "${SVS_OBJECT}" + OUTPUT_VARIABLE svs_disassembly + ERROR_VARIABLE err + RESULT_VARIABLE status +) +if(NOT status EQUAL 0) + message(FATAL_ERROR "${SVS_OBJDUMP} failed on ${SVS_OBJECT}: ${err}") +endif() + +# Drop the `:` heading lines: they are the only place a mangled name +# appears, and a mangled name must not be mistaken for an instruction. +string(REGEX REPLACE "\n[0-9a-f]+ <[^\n]*>:" "\n" svs_disassembly "${svs_disassembly}") + +function(svs_count_class out_var class) + if(NOT DEFINED svs_class_${class}) + message(FATAL_ERROR "Unknown instruction class '${class}' in the budget table.") + endif() + string(REGEX MATCHALL "${svs_class_${class}}" hits "${svs_disassembly}") + list(LENGTH hits count) + set(${out_var} ${count} PARENT_SCOPE) +endfunction() + +set(errors 0) +set(summary) + +foreach(class IN LISTS svs_required) + svs_count_class(count "${class}") + if(count EQUAL 0) + message("${SVS_OBJECT} contains no ${class} instructions.") + message("ISA level ${SVS_LEVEL} is compiled at -march=${SVS_ARCH}, which is") + message("chosen precisely so the kernels use them. An object without any is") + message("a level built at the wrong budget, or one whose kernels fell back") + message("to the generic scalar template.") + math(EXPR errors "${errors} + 1") + else() + list(APPEND summary "${count} ${class}") + endif() +endforeach() + +foreach(class IN LISTS svs_forbidden) + svs_count_class(count "${class}") + if(NOT count EQUAL 0) + message("${SVS_OBJECT} contains ${count} ${class} instructions.") + if(svs_noavx512_active) + message("SVS_NO_AVX512 is set, so -mno-avx512f is supposed to keep AVX-512") + message("out of every level regardless of -march=${SVS_ARCH}. A hit here means") + message("that flag lost to something -- most likely flag order or a per-TU") + message("override -- and this object still faults on a non-AVX-512 host.") + else() + message("ISA level ${SVS_LEVEL} guarantees only what its runtime predicate") + message("tests, so a host the dispatcher routes here faults on them. Either") + message("lower -march=${SVS_ARCH} in cmake/dispatch-surface.cmake, or give") + message("these kernels their own level with a predicate that covers them.") + endif() + math(EXPR errors "${errors} + 1") + endif() +endforeach() + +if(NOT errors EQUAL 0) + message(FATAL_ERROR "dispatch instruction check failed for level ${SVS_LEVEL}") +endif() + +if(svs_required STREQUAL "") + set(summary_display "nothing required") +else() + string(REPLACE ";" ", " summary_display "${summary}") +endif() +if(svs_forbidden STREQUAL "") + set(forbidden_display "nothing forbidden") +else() + string(REPLACE ";" ", " forbidden_display "${svs_forbidden}") + set(forbidden_display "no ${forbidden_display}") +endif() +if(svs_noavx512_active) + message( + "dispatch instructions: level ${SVS_LEVEL} at -march=${SVS_ARCH} with " + "SVS_NO_AVX512 set requires no wide instructions, and confirms ${forbidden_display}" + ) +else() + message( + "dispatch instructions: level ${SVS_LEVEL} at -march=${SVS_ARCH} has " + "${summary_display}, and ${forbidden_display}" + ) +endif() diff --git a/tests/multi-arch/dispatch-checks/check-dispatch-linkage.cmake b/tests/multi-arch/dispatch-checks/check-dispatch-linkage.cmake new file mode 100644 index 000000000..9fa9ae2cb --- /dev/null +++ b/tests/multi-arch/dispatch-checks/check-dispatch-linkage.cmake @@ -0,0 +1,113 @@ +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +##### +##### Checks the dispatch surface against the symbols that were actually built. +##### +##### Run in script mode: +##### +##### cmake -DSVS_PROBE_OBJECT= \ +##### -DSVS_ARCHIVE= \ +##### -DSVS_NM= \ +##### -P tests/multi-arch/dispatch-checks/check-dispatch-linkage.cmake +##### +##### The probe object names every kernel the surface declares and nothing else +##### (see tests/multi-arch/x86/link_probe.cpp), so the kernels it *references* +##### are exactly the kernels the archive must *define* -- and it must define no +##### others. Both directions are checked, plus that the probe defines none of +##### its own. +##### + +include("${CMAKE_CURRENT_LIST_DIR}/lib.cmake") + +svs_require(SVS_PROBE_OBJECT SVS_ARCHIVE SVS_NM) +svs_require_files(SVS_PROBE_OBJECT SVS_ARCHIVE) + +# Distance kernels, and nothing else in the archive. Mangled names are used +# throughout: demangled ones carry `[clone .isra.0]` suffixes that differ between +# a local instantiation and an explicit one. +set(svs_kernel_regex "_ZN3svs8distance.*Impl") + +svs_nm_symbols(probe_defines "${SVS_PROBE_OBJECT}" "${svs_kernel_regex}" NM_ARGS --defined-only) +svs_nm_symbols(probe_references "${SVS_PROBE_OBJECT}" "${svs_kernel_regex}" NM_ARGS --undefined-only) +svs_nm_symbols(archive_defines "${SVS_ARCHIVE}" "${svs_kernel_regex}" NM_ARGS --defined-only) + +list(LENGTH probe_defines n_probe_defines) +list(LENGTH probe_references n_probe_references) +list(LENGTH archive_defines n_archive_defines) + +set(errors 0) + +# A probe that names nothing is not a passing probe. This is what an LTO build +# looks like here, since the object holds IR rather than symbols. +if(n_probe_references EQUAL 0 AND n_probe_defines EQUAL 0) + message("${SVS_PROBE_OBJECT} names no distance kernels at all.") + message("Nothing can be concluded from it. If this is a link-time-optimized") + message("build, nm cannot see the symbols and this check does not apply.") + message(FATAL_ERROR "dispatch linkage check found no symbols to check") +endif() + +# The probe compiles at -march=x86-64 and guarantees nothing about the host, so a +# kernel it defines itself is a kernel some other consumer would also define +# itself -- at whatever -march that consumer happens to use. +if(NOT n_probe_defines EQUAL 0) + message("${n_probe_defines} kernels are instantiated by the probe itself.") + message("Each is missing its `extern template` declaration, so every consumer") + message("of the headers instantiates it locally, from the generic primary") + message("template, at the consumer's own -march. Declare them: the extern") + message("blocks in the distance headers must cover the whole surface.") + message(" Instantiated locally:") + svs_report_first(probe_defines 10) + math(EXPR errors "${errors} + 1") +endif() + +# Declared but never instantiated. The link should already have failed, so this +# only fires when the object is inspected without being linked. +set(missing ${probe_references}) +if(archive_defines) + list(REMOVE_ITEM missing ${archive_defines}) +endif() +list(LENGTH missing n_missing) +if(NOT n_missing EQUAL 0) + message("${n_missing} kernels are declared but never instantiated.") + message("They are declared `extern template` in the distance headers but no") + message("translation unit defines them, so linking against the library fails.") + message(" Undefined:") + svs_report_first(missing 10) + math(EXPR errors "${errors} + 1") +endif() + +# Instantiated but unreachable through the surface: dead weight in the archive. +set(unreachable ${archive_defines}) +if(probe_references) + list(REMOVE_ITEM unreachable ${probe_references}) +endif() +list(LENGTH unreachable n_unreachable) +if(NOT n_unreachable EQUAL 0) + message("${n_unreachable} kernels are instantiated but are not part of the") + message("dispatch surface. Nothing declares them, so no consumer reaches") + message("them; they only add to the size of the library.") + message(" Unreachable:") + svs_report_first(unreachable 10) + math(EXPR errors "${errors} + 1") +endif() + +if(NOT errors EQUAL 0) + message(FATAL_ERROR "dispatch linkage check failed") +endif() + +message( + "dispatch linkage: ${n_archive_defines} kernels declared, instantiated and " + "reachable; none instantiated by the consumer" +) diff --git a/tests/multi-arch/dispatch-checks/lib.cmake b/tests/multi-arch/dispatch-checks/lib.cmake new file mode 100644 index 000000000..9bcb35446 --- /dev/null +++ b/tests/multi-arch/dispatch-checks/lib.cmake @@ -0,0 +1,71 @@ +# Copyright 2026 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +include_guard(GLOBAL) + +function(svs_require) # svs_require(VAR_A VAR_B ...) + foreach(var IN LISTS ARGN) + if(NOT ${var}) + message(FATAL_ERROR "${var} is not set.") + endif() + endforeach() +endfunction() + +function(svs_require_files) # svs_require_files(VAR_A VAR_B ...) + foreach(var IN LISTS ARGN) + if(NOT EXISTS "${${var}}") + message(FATAL_ERROR "${var} does not exist: ${${var}}") + endif() + endforeach() +endfunction() + +function(svs_nm_symbols out_var file regex) + cmake_parse_arguments(arg "" "" "NM_ARGS" ${ARGN}) + execute_process( + COMMAND "${SVS_NM}" ${arg_NM_ARGS} "${file}" + OUTPUT_VARIABLE raw + ERROR_VARIABLE err + RESULT_VARIABLE status + ) + if(NOT status EQUAL 0) + message(FATAL_ERROR "${SVS_NM} failed on ${file}: ${err}") + endif() + + set(symbols) + string(REPLACE "\n" ";" lines "${raw}") + foreach(line IN LISTS lines) + if(line MATCHES "${regex}") + # The mangled name is the last whitespace-separated field. + string(REGEX MATCH "[^ \t]+$" symbol "${line}") + list(APPEND symbols "${symbol}") + endif() + endforeach() + list(REMOVE_DUPLICATES symbols) + set(${out_var} "${symbols}" PARENT_SCOPE) +endfunction() + +function(svs_report_first list_var limit) + set(shown ${${list_var}}) + list(LENGTH shown count) + if(count GREATER limit) + list(SUBLIST shown 0 ${limit} shown) + endif() + foreach(entry IN LISTS shown) + message(" ${entry}") + endforeach() + if(count GREATER limit) + math(EXPR rest "${count} - ${limit}") + message(" ... and ${rest} more") + endif() +endfunction() diff --git a/tests/multi-arch/x86/entry_probe.cpp b/tests/multi-arch/x86/entry_probe.cpp new file mode 100644 index 000000000..8e22b014c --- /dev/null +++ b/tests/multi-arch/x86/entry_probe.cpp @@ -0,0 +1,133 @@ +/* + * Copyright 2026 Intel Corporation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// A consumer that reaches the kernels the way real code does: through the +// distance entry points, which name every ISA level unconditionally and pick one +// at runtime. Complements link_probe.cpp, which names the kernels directly and so +// checks the surface without depending on the entry points routing to it. + +#include "svs/core/distance/cosine.h" +#include "svs/core/distance/euclidean.h" +#include "svs/core/distance/inner_product.h" +#include "svs/multi-arch/x86/preprocessor.h" + +#include "host_levels.h" + +#include +#include +#include +#include +#include +#include + +namespace { + +// Deliberately not a multiple of any vector width, so the epilogue is exercised. +constexpr size_t entry_dynamic_dim = 97; + +constexpr size_t fixed_extent(bool longest) { + size_t result = longest ? 1 : svs::Dynamic; + for (auto dim : svs::distance::supported_dim_list) { + if (dim == svs::Dynamic) { + continue; + } + result = longest ? std::max(result, dim) : std::min(result, dim); + } + return result; +} + +// The extent the execution check breaks on: the surface's shortest fixed one, so +// the kernel it enters is a fully unrolled specialization rather than the epilogue. +constexpr size_t entry_report_dim = fixed_extent(false); + +constexpr size_t entry_buffer_dim = std::max(fixed_extent(true), entry_dynamic_dim); + +template const E* buffer() { + static const std::array values = []() { + std::array filled{}; + filled.fill(static_cast(1.0F)); + return filled; + }(); + return values.data(); +} + +// Both values are returned together so that the counter cannot drift from the sum it +// describes; the count is what distinguishes a reached entry point from one never emitted. +struct ProbeResult { + float total = 0; + std::size_t kernels = 0; +}; + +// A template rather than a macro body: the svs::Dynamic entry points take a +// length, and `if constexpr` only discards the other branch inside a template. +template float entry_one() { + const Ea* a = buffer(); + const Eb* b = buffer(); + if constexpr (N == svs::Dynamic) { + return svs::distance::L2::compute(a, b, entry_dynamic_dim) + + svs::distance::IP::compute(a, b, entry_dynamic_dim) + + svs::distance::CosineSimilarity::compute(a, b, 1.0F, entry_dynamic_dim); + } else { + return svs::distance::L2::compute(a, b) + svs::distance::IP::compute(a, b) + + svs::distance::CosineSimilarity::compute(a, b, 1.0F); + } +} + +// entry_one reaches L2, IP and cosine; a change to its body must change this constant. +constexpr std::size_t entry_kernels_per_call = 3; + +#define SVS_ENTRY_ONE(Ea, Eb, N) \ + total += entry_one(); \ + count += entry_kernels_per_call; +#define SVS_ENTRY_DIM(N) SVS_FOR_EACH_TYPE_PAIR(SVS_ENTRY_ONE, N) + +ProbeResult entry_all() { + float total = 0; + std::size_t count = 0; + SVS_FOR_EACH_SUPPORTED_DIM(SVS_ENTRY_DIM) + return ProbeResult{total, count}; +} + +#undef SVS_ENTRY_DIM +#undef SVS_ENTRY_ONE + +} // namespace + +int main(int argc, char** argv) { + if (argc == 2 && std::strcmp(argv[1], "--report") == 0) { + // Read by tests/multi-arch/dispatch-checks/check-dispatch-execution.cmake, which + // breaks on this extent's kernels to see which level the call enters. + std::printf("expect-level %d\n", svs_test::expected_level()); + std::printf("probe-extent %zu\n", entry_report_dim); + // int8/int8, mangled `aa`, is one of the only two pairs with an AVX512_VNNI + // kernel, so it is the pair whose reported level distinguishes VNNI from AVX512. + std::printf( + "one-call %f\n", + static_cast(entry_one()) + ); + return 0; + } + + const ProbeResult result = entry_all(); + std::printf( + "dispatch surface entry probe: called %zu kernels that computed a total distance " + "sum of %f (total distance only printed here to avoid calls from getting optimized " + "away)\n", + result.kernels, + static_cast(result.total) + ); + return 0; +} diff --git a/tests/multi-arch/x86/host_levels.h b/tests/multi-arch/x86/host_levels.h new file mode 100644 index 000000000..b8372655e --- /dev/null +++ b/tests/multi-arch/x86/host_levels.h @@ -0,0 +1,60 @@ +/* + * Copyright 2026 Intel Corporation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "svs/core/distance/distance_core.h" +#include "svs/lib/avx_detection.h" + +#include + +namespace svs_test { + +using svs::distance::AVX_AVAILABILITY; + +// The runtime predicate behind each ISA level, and the only place the dispatch +// tests state it: a level added to the surface without a specialization here is +// an undefined symbol, deliberately -- the predicate cannot be guessed. +template bool host_satisfies(); + +template <> inline bool host_satisfies() { + return svs::detail::avx_runtime_flags.is_avx2_supported(); +} + +template <> inline bool host_satisfies() { + return svs::detail::avx_runtime_flags.is_avx512f_supported(); +} + +template <> inline bool host_satisfies() { + return svs::detail::avx_runtime_flags.is_avx512vnni_supported(); +} + +/// The level the distance entry points must choose on this host. +/// +/// The entry points test the strongest level first, so their choice is the +/// highest-numbered satisfied level; AVX_AVAILABILITY::NONE is the fallback. +inline int expected_level() { + int highest = static_cast(AVX_AVAILABILITY::NONE); +#define SVS_HOST_LEVEL_ONE(LEVEL) \ + if (host_satisfies()) { \ + highest = std::max(highest, static_cast(AVX_AVAILABILITY::LEVEL)); \ + } + SVS_FOR_EACH_ISA_LEVEL(SVS_HOST_LEVEL_ONE) +#undef SVS_HOST_LEVEL_ONE + return highest; +} + +} // namespace svs_test diff --git a/tests/multi-arch/x86/link_probe.cpp b/tests/multi-arch/x86/link_probe.cpp new file mode 100644 index 000000000..8eda20974 --- /dev/null +++ b/tests/multi-arch/x86/link_probe.cpp @@ -0,0 +1,129 @@ +/* + * Copyright 2026 Intel Corporation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// A consumer that names every kernel the dispatch surface declares and nothing +// else, so a declared-but-uninstantiated kernel is an undefined symbol here. + +#include "svs/core/distance/cosine.h" +#include "svs/core/distance/euclidean.h" +#include "svs/core/distance/inner_product.h" +#include "svs/lib/static.h" +#include "svs/multi-arch/x86/preprocessor.h" + +#include "host_levels.h" + +#include +#include +#include +#include + +namespace { + +using svs::distance::AVX_AVAILABILITY; +using svs_test::host_satisfies; + +// The longest fixed extent in the surface. +constexpr size_t probe_max_dim = []() { + size_t longest = 1; + for (auto dim : svs::distance::supported_dim_list) { + if (dim != svs::Dynamic && dim > longest) { + longest = dim; + } + } + return longest; +}(); + +// A length for the svs::Dynamic kernels. Deliberately not a multiple of any +// vector width, so the epilogue is exercised too. +constexpr size_t probe_dynamic_dim = 97; + +// One buffer serves every call, whichever extents the surface happens to declare. +constexpr size_t probe_buffer_dim = std::max(probe_max_dim, probe_dynamic_dim); + +template const E* buffer() { + static const std::array values = []() { + std::array filled{}; + filled.fill(static_cast(1.0F)); + return filled; + }(); + return values.data(); +} + +// svs::lib::MaybeStatic has no default constructor: a dynamic +// extent must be told its length. +template svs::lib::MaybeStatic probe_length() { + if constexpr (N == svs::Dynamic) { + return svs::lib::MaybeStatic(probe_dynamic_dim); + } else { + return svs::lib::MaybeStatic(); + } +} + +// Named directly rather than through L2::compute, which also reaches +// AVX_AVAILABILITY::NONE -- not in the surface, so every consumer instantiates it. +#define SVS_PROBE_ONE(Ea, Eb, N, LEVEL) \ + total += svs::distance::L2Impl::compute( \ + buffer(), buffer(), probe_length() \ + ); \ + ++count; \ + total += svs::distance::IPImpl::compute( \ + buffer(), buffer(), probe_length() \ + ); \ + ++count; \ + total += \ + svs::distance::CosineSimilarityImpl::compute( \ + buffer(), buffer(), 1.0F, probe_length() \ + ); \ + ++count; + +// The level's own type-pair list, not the full one: naming a pair the level has no +// kernel for would instantiate the generic template right here. +#define SVS_PROBE_TARGET(N, LEVEL) \ + if (host_satisfies()) { \ + SVS_TYPE_PAIRS_FOR(LEVEL, SVS_PROBE_ONE, N, LEVEL) \ + } + +// Both values are returned together so that the counter cannot drift from the sum it +// describes; the count is what makes a link-only failure distinguishable from a call that +// was never emitted. +struct ProbeResult { + float total = 0; + std::size_t kernels = 0; +}; + +ProbeResult probe_all() { + float total = 0; + std::size_t count = 0; + SVS_FOR_EACH_DISPATCH_TARGET(SVS_PROBE_TARGET) + return ProbeResult{total, count}; +} + +#undef SVS_PROBE_TARGET +#undef SVS_PROBE_ONE + +} // namespace + +int main() { + const ProbeResult result = probe_all(); + std::printf( + "dispatch surface probe: called %zu kernels that computed a total distance sum " + "of %f (total distance only printed here to avoid calls from getting optimized " + "away)\n", + result.kernels, + static_cast(result.total) + ); + return 0; +}