From 5dfa00928ef37cee72ede32cd596dc812f02bdf8 Mon Sep 17 00:00:00 2001 From: Daniel Byrne Date: Tue, 27 Oct 2026 12:19:05 -0400 Subject: [PATCH 01/11] Add dtoctl, a CLI wrapper for running programs with DTO preloaded (picks up #33) (#38) * Add dtoctl wrapper to drive DTO without manual LD_PRELOAD setup Using the DTO library today requires the user to understand two orthogonal mechanisms before a single program can be offloaded to DSA. The library has to be injected into the target through LD_PRELOAD, and its behavior is steered entirely through a set of roughly twenty DTO_* environment variables that must be discovered by reading the manual and then exported by hand. This is easy to get wrong: the LD_PRELOAD path and the variable names are stringly typed, an exported variable silently leaks into every later command in the same shell, and there is no validation or discoverability at the point of use. Provide a small wrapper, dtoctl, that turns this into a single self-describing command line. It follows the same model as hmctl from the hmsdk project: parse options, translate them into the environment the library expects, and then exec the target program so no state lingers in the caller's shell. On top of that model dtoctl also takes care of the injection itself by prepending the resolved libdto.so to LD_PRELOAD, which is possible because DTO works by transparently interposing on the standard memory APIs. Every DTO_* variable is exposed as an option so the mapping stays obvious and greppable. Only options that are actually given are written to the child environment; anything left unset keeps whatever the caller already exported or the library's own default, and an explicit option wins over an inherited variable. The library path is resolved from -l/--library, then $DTO_LIBRARY, then a compiled-in default, and execvp() is used so a bare command name on PATH works as well as an explicit path. Build and install dtoctl with both Make and CMake. Set its compiled-in library path to match the installation directory, since CMake's default /usr/local/lib differs from the Makefile's /usr/lib64. Allow that path to be overridden with DTOCTL_LIBPATH. With this in place the previous incantation $ export LD_PRELOAD=/usr/lib64/libdto.so $ export DTO_WAIT_METHOD=busypoll $ export DTO_CPU_SIZE_FRACTION=0.33 $ ./prog collapses to $ dtoctl -w busypoll -c 0.33 ./prog Signed-off-by: Honggyu Kim * Add a dtoctl man page The dtoctl options mirror the DTO_* environment variables, but a user coming to the tool for the first time still needs a single authoritative place that explains what each option does, what values it accepts, and how the library path and LD_PRELOAD are resolved. Scattering that information across the source help text alone makes it hard to consult without running the tool. Add a manual page in the same troff-friendly markdown form the hmsdk project uses for hmctl, and render it as a section 1 man page for the user command. Besides the per-option reference it documents the library resolution order and the override semantics, and it gathers the common tuning goals - latency, power, and cache-pollution avoidance - as ready-to-run examples such as $ dtoctl -w umwait -c 0.0 --no-auto-adjust ./prog so users can map an intent directly onto a command instead of reverse engineering it from the individual variables. Keep the markdown as the source form and generate dtoctl.1 with pandoc during the Makefile and CMake builds. Both build systems install the generated page under man1, so it can be read with $ man 1 dtoctl Pandoc is optional: when it is unavailable, print a message recommending its installation and skip manual generation. The Makefile can still install an already generated page. Ignore its generated doc/dtoctl.1 and remove it on make clean; CMake keeps its generated page in the build directory. Signed-off-by: Honggyu Kim * Document dtoctl in the README The README is where users first learn how to configure and run DTO, and until now it only described the manual LD_PRELOAD and DTO_* export workflow. A reader following it has no reason to know that dtoctl exists or that it removes most of that ceremony, so the easier path stays hidden precisely where people look for guidance. Introduce dtoctl after the environment variable reference and the common usage models that accompany it, so that discussion stays in one piece and the new section can point back at those usage models. Contrast the old export sequence with the equivalent $ dtoctl -w busypoll -c 0.33 ./prog and spell out the resolution order for libdto.so along with the rule that only the options actually passed are applied. Keep the per-option reference in dtoctl(1) instead of repeating it here. The option list already exists in the --help output and in the manual, and a third copy in the README would only be one more place to forget when an option changes. Signed-off-by: Honggyu Kim * dto: honor DTO_WAIT_METHOD=yield The yield wait method is implemented in dsa_wait() and documented in the README, but dsa_init() never matched "yield" when parsing DTO_WAIT_METHOD, so DTO_WAIT_METHOD=yield silently ran with busypoll. * dtoctl: validate options, resolve the library up front, add --dry-run Make the wrapper catch the mistakes it exists to prevent instead of passing them on to libdto, which silently falls back to a default: - Validate every option value: wait method names, integer ranges for --numa-aware and --log-level, a cpu fraction in [0, 1), a positive umwait delay, and non-empty paths and WQ list. --min-bytes takes an optional K/M/G suffix and is passed on as plain bytes; libdto would read "64K" as 64 and "abc" as 0. - Resolve libdto.so to an absolute path and fail with exit status 125 if it cannot be read. ld.so only warns about a missing preload and runs the program without DTO, and a relative path stopped working as soon as the program changed directory. When built with LIBDTO_REL_PATH, prefer the library installed next to dtoctl so a relocated install finds its own copy. - Drop other libdto.so entries from an inherited LD_PRELOAD, so one process never loads two copies of DTO. - --stats-file implies --stats; libdto ignores DTO_STATS_FILE otherwise. - Add -i/--ignore-env to unset inherited DTO_* variables. The --no-* flags can only turn features off, so this is the way to undo an exported DTO_USESTDC_CALLS=1 or DTO_DSA_MEMCPY=0. - Add --dry-run to print the resolved library, LD_PRELOAD, each DTO_* variable with whether it came from an option or the environment, and the command, without running it. - Add --version, and exit with 126/127 when the program cannot be run or is not found, as env(1) and timeout(1) do. - Make the options local to main() and drop dead argument handling. * dtoctl: follow the install prefix, report the version, add tests CMake cached the default library path on the first configure, so a later CMAKE_INSTALL_PREFIX change was ignored, and "cmake --install --prefix" produced a dtoctl that preloaded /usr/local/lib/libdto.so. Compute the default on every configure, pass dtoctl the library's path relative to its own bindir so a relocated install finds its own copy, and keep DTOCTL_LIBPATH as an explicit override. Pass the project version to both builds for --version, and make the Makefile's install target build dtoctl first. Add tests/test_dtoctl.sh as the "dtoctl" CTest (label functional). It needs no DSA: it checks the option to variable mapping, validation, library resolution, LD_PRELOAD handling and exit statuses through --dry-run and the program's environment, and that libdto applies the settings through its own configuration trace. * docs: document dtoctl's validation, library resolution and --dry-run Update the dtoctl man page and README section for the new behavior: option validation, -i/--ignore-env, --dry-run, --version, size suffixes, --stats-file implying --stats, how the library is resolved, and the exit statuses. Add a note on sudo, which resets the environment and strips LD_PRELOAD when it sits between dtoctl and the program, and on settings being inherited by child processes. Fix stale defaults in the README's variable list: DTO_WAIT_METHOD defaults to busypoll and DTO_MIN_BYTES to 65536, tpause is a valid wait method, and DTO_LOG_FILE gets .. appended. Describe the dtoctl test in TESTING.md. * gitignore: ignore the libdto_explicit build output --------- Signed-off-by: Honggyu Kim Co-authored-by: Honggyu Kim --- .gitignore | 3 + CMakeLists.txt | 49 +++ Makefile | 36 ++- README.md | 61 +++- TESTING.md | 14 + doc/dtoctl.md | 270 ++++++++++++++++ dto.c | 4 + dtoctl.c | 714 +++++++++++++++++++++++++++++++++++++++++++ tests/test_dtoctl.sh | 152 +++++++++ 9 files changed, 1297 insertions(+), 6 deletions(-) create mode 100644 doc/dtoctl.md create mode 100644 dtoctl.c create mode 100755 tests/test_dtoctl.sh diff --git a/.gitignore b/.gitignore index ebb181e..be47746 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,7 @@ dto-test* libdto.so* +libdto_explicit.so* build*/ tests/baselines*/ +dtoctl +/doc/dtoctl.1 diff --git a/CMakeLists.txt b/CMakeLists.txt index 8e8cccd..19f76ce 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -124,6 +124,47 @@ if (HAS_WAITPKG) endforeach() endif() +# Build dtoctl. By default it preloads the libdto.so installed alongside it, +# found relative to the dtoctl binary so "cmake --install --prefix" still works, +# and falls back to the configured install path. Set DTOCTL_LIBPATH to always +# preload a specific library instead. +set(DTOCTL_LIBPATH "" CACHE FILEPATH + "Path to libdto.so that dtoctl preloads by default (empty: the installed libdto.so)") +add_executable(dtoctl dtoctl.c) +target_compile_definitions(dtoctl PRIVATE DTO_VERSION="${PROJECT_VERSION}") +if(DTOCTL_LIBPATH) + target_compile_definitions(dtoctl PRIVATE LIBDTO_PATH="${DTOCTL_LIBPATH}") +else() + # Not cached, so it follows later changes to CMAKE_INSTALL_PREFIX. + get_filename_component(DTOCTL_LIBDIR "${CMAKE_INSTALL_LIBDIR}" ABSOLUTE + BASE_DIR "${CMAKE_INSTALL_PREFIX}") + get_filename_component(DTOCTL_BINDIR "${CMAKE_INSTALL_BINDIR}" ABSOLUTE + BASE_DIR "${CMAKE_INSTALL_PREFIX}") + file(RELATIVE_PATH DTOCTL_REL_LIBPATH "${DTOCTL_BINDIR}" "${DTOCTL_LIBDIR}/libdto.so") + target_compile_definitions(dtoctl PRIVATE + LIBDTO_PATH="${DTOCTL_LIBDIR}/libdto.so" + LIBDTO_REL_PATH="${DTOCTL_REL_LIBPATH}") +endif() +install(TARGETS dtoctl RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}) + +# Build and install the optional section 1 manual. +find_program(PANDOC_EXECUTABLE NAMES pandoc) +if(PANDOC_EXECUTABLE) + set(DTOCTL_MAN_PAGE "${CMAKE_CURRENT_BINARY_DIR}/doc/dtoctl.1") + add_custom_command( + OUTPUT "${DTOCTL_MAN_PAGE}" + COMMAND ${CMAKE_COMMAND} -E make_directory "${CMAKE_CURRENT_BINARY_DIR}/doc" + COMMAND "${PANDOC_EXECUTABLE}" --standalone --to=man + "${CMAKE_CURRENT_SOURCE_DIR}/doc/dtoctl.md" -o "${DTOCTL_MAN_PAGE}" + DEPENDS "${CMAKE_CURRENT_SOURCE_DIR}/doc/dtoctl.md" + COMMENT "Generating dtoctl(1) manual" + VERBATIM) + add_custom_target(dtoctl-man ALL DEPENDS "${DTOCTL_MAN_PAGE}") + install(FILES "${DTOCTL_MAN_PAGE}" DESTINATION ${CMAKE_INSTALL_MANDIR}/man1) +else() + message(STATUS "Skipping dtoctl manual generation and installation: install pandoc to build the man page.") +endif() + # Build dto-test and dto-test-wodto add_executable(dto-test dto-test.c) target_link_libraries(dto-test PRIVATE DTO::dto pthread) @@ -158,6 +199,14 @@ if(DTO_BUILD_TESTS) TIMEOUT 120 ) endif() + # dtoctl option handling and environment setup (no DSA hardware needed) + add_test(NAME dtoctl + COMMAND bash ${CMAKE_CURRENT_SOURCE_DIR}/tests/test_dtoctl.sh + $ $) + set_tests_properties(dtoctl PROPERTIES + LABELS "functional" + TIMEOUT 60 + ) # ---- A/B performance regression test ---- # The baseline object is compiled from unguarded upstream sources that diff --git a/Makefile b/Makefile index 4da8111..25b649a 100644 --- a/Makefile +++ b/Makefile @@ -2,10 +2,19 @@ # # SPDX-License-Identifier: MIT -all: libdto dto-test-wodto +all: libdto dtoctl dto-test-wodto man + +.PHONY: man install-man DML_LIB_CXX=-D_GNU_SOURCE +# Path to libdto.so that dtoctl prepends to LD_PRELOAD (matches "make install"). +DTOCTL_LIBPATH ?= /usr/lib64/libdto.so +PANDOC ?= pandoc +MANDIR ?= /usr/share/man +# Version reported by "dtoctl --version", taken from the CMake project. +DTO_VERSION := $(shell grep -o 'DTO VERSION [0-9.]*' CMakeLists.txt | cut -d' ' -f3) + libdto: dto.c gcc -shared -fPIC -Wl,-soname,libdto.so dto.c $(DML_LIB_CXX) -DDTO_STATS_SUPPORT -DDTO_ACCEL_CONFIG_SUPPORT -DDTO_NUMA_SUPPORT -o libdto.so.1.0 -laccel-config -ldl -lnuma -mwaitpkg @@ -16,11 +25,31 @@ libdto_nostats: dto.c libdto_explicit: dto.c gcc -shared -fPIC -Wl,-soname,libdto_explicit.so dto.c $(DML_LIB_CXX) -DDTO_NO_LIBC_INTERPOSE -DDTO_ACCEL_CONFIG_SUPPORT -DDTO_NUMA_SUPPORT -o libdto_explicit.so.1.0 -laccel-config -ldl -lnuma -mwaitpkg -install: +dtoctl: dtoctl.c + gcc -O2 -Wall dtoctl.c -DLIBDTO_PATH=\"$(DTOCTL_LIBPATH)\" -DDTO_VERSION=\"$(DTO_VERSION)\" -o dtoctl + +man: doc/dtoctl.1 + +doc/dtoctl.1: doc/dtoctl.md + @if command -v "$(PANDOC)" >/dev/null 2>&1; then \ + "$(PANDOC)" --standalone --to=man $< -o $@; \ + else \ + echo "Skipping dtoctl manual generation: install pandoc to build the man page."; \ + fi + +install: dtoctl install-man cp libdto.so.1.0 /usr/lib64/ cp dto.h /usr/include/ ln -sf /usr/lib64/libdto.so.1.0 /usr/lib64/libdto.so.1 ln -sf /usr/lib64/libdto.so.1.0 /usr/lib64/libdto.so + cp dtoctl /usr/bin/ + +install-man: man + @if [ -f doc/dtoctl.1 ]; then \ + install -D -m 644 doc/dtoctl.1 "$(MANDIR)/man1/dtoctl.1"; \ + else \ + echo "Skipping dtoctl manual installation: no generated man page is available."; \ + fi install-local: ln -sf ./libdto.so.1.0 ./libdto.so.1 @@ -33,4 +62,5 @@ dto-test-wodto: dto-test.c gcc -g dto-test.c $(DML_LIB_CXX) -o dto-test-wodto -lpthread clean: - rm -rf *.o *.so dto-test + rm -rf *.o *.so dto-test dtoctl + rm -f doc/dtoctl.1 diff --git a/README.md b/README.md index 42cda0e..ec906cc 100644 --- a/README.md +++ b/README.md @@ -53,8 +53,8 @@ Following environment variables control the behavior of DTO library: Should be enabled for debugging/profiling only, not for perf evaluation (enabling it slows down the workload). Default is 0. DTO_STATS_FILE= when DTO_COLLECT_STATS=1, redirect histogram output to the provided regular file instead of standard output. The file is truncated when DTO loads; failures fall back to standard output and emit an error. - DTO_WAIT_METHOD= (specifies the method to use while waiting for DSA to complete operation, default is yield) - DTO_MIN_BYTES=xxxx (specifies minimum size of API call needed for DSA operation execution, default is 16384 bytes) + DTO_WAIT_METHOD= (specifies the method to use while waiting for DSA to complete operation, default is busypoll) + DTO_MIN_BYTES=xxxx (specifies minimum size of API call needed for DSA operation execution, default is 65536 bytes) DTO_CPU_SIZE_FRACTION=0.xx (specifies fraction of job performed by CPU, in parallel to DSA). Default is 0.00 DTO_AUTO_ADJUST_KNOBS=0/1 (disables/enables auto tuning of cpu_size_fraction and dsa_min_bytes parameters. 0 -- disable, 1 -- enable (default)) DTO_IS_NUMA_AWARE=0/1/2 (disables/buffer-centric/cpu-centric numa awareness. 0 -- disable (default), 1 -- buffer-centric, 2 - cpu-centric) @@ -69,7 +69,7 @@ Following environment variables control the behavior of DTO library: DTO_DSA_CC=0/1, 1 (default) - DTO sets DSA Cache Control flag to 1 if DSA supports cache control, 0 - DTO sets DSA Cache Control flag to 0 DTO_OVERLAPPING_MEMMOVE_ACTION=0/1 0 (default) DTO submits memmove operations with overlapping buffers entirely to CPU, 1 - entirely to DSA DTO_UMWAIT_DELAY=xxxx defines delay for umwait command (check max possible value at: /sys/devices/system/cpu/umwait_control/max_time), default is 100000 - DTO_LOG_FILE= Redirect the DTO output to the specified file instead of std output (useful for debugging and statistics collection). file name is suffixed by process pid. + DTO_LOG_FILE= Redirect the DTO output to the specified file instead of std output (useful for debugging and statistics collection). file name is suffixed by the program name and process pid (..). DTO_LOG_LEVEL=0/1/2 controls the log level. higher value means more verbose logging (default 0). DTO_CRC_MIN_BYTES=xxxx smallest buffer dto_crc32c_with_seed() sends to DSA; smaller ones are checksummed on the CPU (default: DTO_MIN_BYTES). ``` @@ -140,6 +140,61 @@ Programs that only use this API, and must not have their own large libc copies sent to DSA, can link `libdto_explicit` instead: the same library without the memcpy/memset/memmove/memcmp interposers (`-ldto_explicit`, or `DTO::dto_explicit` from CMake). +## Using dtoctl + +Setting `LD_PRELOAD` and exporting the `DTO_*` environment variables by hand is +error prone. The `dtoctl` tool provides a simpler interface: it takes the DTO +settings as command line options, converts them to the corresponding `DTO_*` +environment variables, prepends `libdto.so` to `LD_PRELOAD`, and then runs the +target program. For example, instead of + +```bash +export LD_PRELOAD=/usr/local/lib/libdto.so +export DTO_WAIT_METHOD=busypoll +export DTO_CPU_SIZE_FRACTION=0.33 +export DTO_AUTO_ADJUST_KNOBS=1 +./prog +``` + +you can run + +```bash +dtoctl -w busypoll -c 0.33 ./prog +``` + +Every environment variable listed above has a corresponding option, and option +values are checked before the program runs (DTO itself silently falls back to a +default on a value it does not understand). Only the options you pass are +applied: a `DTO_*` variable you already exported is left untouched, an explicit +option overrides it, and `-i/--ignore-env` drops all inherited `DTO_*` +variables. + +The `libdto.so` path is resolved from `-l/--library`, then `$DTO_LIBRARY`, then +the library installed with `dtoctl`, then the compiled-in default +(`/usr/local/lib/libdto.so` for a CMake build, `/usr/lib64/libdto.so` for the +Makefile build). It is made absolute and prepended to any existing +`LD_PRELOAD`, and `dtoctl` fails if the library cannot be read rather than +running the program without DTO. + +To see what would run without running it, use `--dry-run`: + +```bash +$ dtoctl --dry-run -w umwait -b 16K ./prog +library: /usr/lib64/libdto.so.1.0 (compiled-in default) +LD_PRELOAD=/usr/lib64/libdto.so.1.0 +DTO_WAIT_METHOD=umwait # option +DTO_MIN_BYTES=16384 # option +command: ./prog +``` + +DSA work queues are often accessible to root only. Run `sudo dtoctl ...` +rather than `dtoctl sudo ...`, which loses `LD_PRELOAD` and the `DTO_*` +variables, and pass settings as options: `sudo` does not pass on variables +exported in your shell. + +Run `dtoctl --help` for the option list, or `man 1 dtoctl` for the full manual, +which documents each option and gives a ready-to-run command for each of the +usage models above. ## Build diff --git a/TESTING.md b/TESTING.md index d0e1b37..7dae505 100644 --- a/TESTING.md +++ b/TESTING.md @@ -79,6 +79,20 @@ the work queue has (it reports which), so run the suite once with `block_on_fault` enabled and once without to cover both. It assumes all work queues DTO uses share the same setting. +## dtoctl Tests + +Checks for the `dtoctl` wrapper: every option sets its `DTO_*` variable, +invalid values are rejected, the library path is resolved and validated, +`LD_PRELOAD` and inherited variables are handled, and exit statuses follow +`env(1)`. The last check runs a program under `dtoctl` and reads libdto's own +configuration trace. No DSA hardware is needed. + +```bash +ctest -R dtoctl +# or directly, from the source directory +tests/test_dtoctl.sh build/dtoctl build/libdto.so +``` + ## Performance Tests The performance test (`perf`) detects regressions in libdto by diff --git a/doc/dtoctl.md b/doc/dtoctl.md new file mode 100644 index 0000000..afca4ed --- /dev/null +++ b/doc/dtoctl.md @@ -0,0 +1,270 @@ +% DTOCTL(1) Dtoctl User Manuals +% Honggyu Kim +% 2026 + +NAME +==== +dtoctl - run a program with DSA Transparent Offload (DTO) preloaded + + +SYNOPSIS +======== +dtoctl [_options_] COMMAND [_command-options_] + +dtoctl \--dry-run [_options_] [COMMAND [_command-options_]] + + +DESCRIPTION +=========== +The **dtoctl** tool runs `COMMAND` with the DSA Transparent Offload library +(**libdto.so**) preloaded, so that its `memcpy`, `memmove`, `memset`, and +`memcmp` calls are transparently offloaded to Intel DSA (Data Streaming +Accelerator). + +Without **dtoctl**, users must set **LD_PRELOAD** to point at **libdto.so** and +export a set of **DTO_\*** environment variables to control DTO behavior. +**dtoctl** replaces that manual setup: it checks its command line options, +converts them into the corresponding **DTO_\*** environment variables, prepends +**libdto.so** to **LD_PRELOAD**, and then executes `COMMAND`. Nothing is left +behind in the calling shell. + +Only the options that are explicitly given are applied. Options that are not +given leave any pre-existing **DTO_\*** environment variables (or the DTO +built-in defaults) untouched, and an explicitly given option overrides a +previously exported variable of the same name. Use `-i` to ignore the +inherited **DTO_\*** variables instead. + +Option values are validated before `COMMAND` runs, and an invalid value is an +error. DTO itself silently falls back to a default when a variable holds a +value it does not understand. + +`COMMAND` and every process it starts inherit the settings, so +`dtoctl make` runs each compiler under DTO as well. + + +LIBRARY RESOLUTION +================== +The path of **libdto.so** to preload is resolved in the following order: + +1. The `-l`, `--library` option, if given. +2. The **DTO_LIBRARY** environment variable, if set and not empty. This + variable is read by **dtoctl** only, not by **libdto.so**. +3. For a CMake build, the **libdto.so** installed with **dtoctl**, found + relative to the **dtoctl** executable (`/../lib/libdto.so`), so + that an installation moved with `cmake --install --prefix` finds its own + library. +4. The compiled-in default path (`/usr/local/lib/libdto.so` for a default + CMake build, or `/usr/lib64/libdto.so` for the Makefile build). + +The resolved path is made absolute, so it keeps working when `COMMAND` changes +directory, and **dtoctl** fails with exit status 125 if it cannot read the +library. (**ld.so** would only print a warning and run `COMMAND` without DTO.) +Run `dtoctl --dry-run` to see which library was chosen and why. + +The library is prepended to any existing **LD_PRELOAD** value. Any other +**libdto.so** entry already in **LD_PRELOAD** is removed, with a warning when it +names a different file, so that one process never loads two copies of DTO. + + +OPTIONS +======= + +General +------- +-l _PATH_, \--library=_PATH_ +: Path to **libdto.so** to prepend to **LD_PRELOAD**. Overrides + **DTO_LIBRARY** and the installed library (see LIBRARY RESOLUTION). + +-i, \--ignore-env +: Unset every **DTO_\*** variable inherited from the environment, including + **DTO_LIBRARY**, so that only the options given on this command line and + the DTO built-in defaults apply. This is the way to undo an exported + variable that the `--no-*` options cannot, such as + **DTO_USESTDC_CALLS=1** or **DTO_DSA_MEMCPY=0**. + +\--dry-run +: Print the resolved library and where it came from, the resulting + **LD_PRELOAD**, every **DTO_\*** variable that `COMMAND` would see (each + marked `option` or `inherited`), and the command line, then exit without + running anything. `COMMAND` may be omitted. + +Offload tuning +-------------- +-w _METHOD_, \--wait-method=_METHOD_ +: How to wait for DSA completion: _yield_, _busypoll_, _umwait_, or _tpause_. + _umwait_ and _tpause_ need a CPU with WAITPKG support; DTO falls back + otherwise. Sets **DTO_WAIT_METHOD**. Default: _busypoll_. + +-b _BYTES_, \--min-bytes=_BYTES_ +: Minimum operation size offloaded to DSA. Smaller operations run on the CPU. + _BYTES_ may have a _K_, _M_, or _G_ suffix (powers of 1024), and is passed to + DTO as a plain byte count. With auto tuning on, this is the starting + value. Sets **DTO_MIN_BYTES**. Default: 64K. + +-c _FRACTION_, \--cpu-fraction=_FRACTION_ +: Fraction (0.0 <= f < 1.0) of each operation performed on the CPU in parallel + to DSA. DTO uses two decimal places. With auto tuning on, this is the + starting value. Sets **DTO_CPU_SIZE_FRACTION**. Default: 0.0. + +-n _MODE_, \--numa-aware=_MODE_ +: NUMA awareness: _0_ disable, _1_ buffer-centric, _2_ cpu-centric. Has no + effect if DTO was built without NUMA support. Sets **DTO_IS_NUMA_AWARE**. + Default: _0_. + +-q _LIST_, \--wq-list=_LIST_ +: Semicolon-separated list of DSA work queues to use, e.g. "wq0.0;wq2.0". + Names must match those under /dev/dsa/, and the list can be at most 255 + characters. Required if DTO was built without accel-config support. + Sets **DTO_WQ_LIST**. Default: auto-discover all available WQs. + +\--umwait-delay=_CYCLES_ +: Delay in cycles for the umwait instruction, a positive number. The + maximum is in /sys/devices/system/cpu/umwait_control/max_time. Sets + **DTO_UMWAIT_DELAY**. Default: 100000. + +\--overlapping-memmove=_WHERE_ +: Where to run memmove with overlapping buffers: _cpu_ or _dsa_. + Sets **DTO_OVERLAPPING_MEMMOVE_ACTION** (cpu -> 0, dsa -> 1). Default: _cpu_. + +Disable DSA offload for specific operations +------------------------------------------- +By default every one of these operations is offloaded to DSA; each option below +turns a single one back into its standard C library call on the CPU. + +\--no-memcpy +: Use the system memcpy instead of DSA. Sets **DTO_DSA_MEMCPY=0**. + +\--no-memmove +: Use the system memmove instead of DSA. Sets **DTO_DSA_MEMMOVE=0**. + +\--no-memset +: Use the system memset instead of DSA. Sets **DTO_DSA_MEMSET=0**. + +\--no-memcmp +: Use the system memcmp instead of DSA. Sets **DTO_DSA_MEMCMP=0**. + +Other toggles +------------- +\--no-cache-control +: Clear the DSA cache control flag to avoid cache pollution. + Sets **DTO_DSA_CC=0**. Default: cache control is on. + +\--no-auto-adjust +: Disable auto tuning of cpu-fraction and min-bytes. + Sets **DTO_AUTO_ADJUST_KNOBS=0**. Default: auto tuning is on. + +\--stdc-only +: Use only the standard C memory functions, no DSA offload. + Sets **DTO_USESTDC_CALLS=1**. Default: DSA offload is enabled. + +Statistics and logging +---------------------- +\--stats +: Enable stats collection. For debugging/profiling only, as it slows down the + workload. Sets **DTO_COLLECT_STATS=1**. Default: off. + +\--stats-file=_PATH_ +: Write the stats histogram to _PATH_ instead of standard output. Implies + `--stats`, since DTO ignores the file otherwise. Sets **DTO_STATS_FILE** + and **DTO_COLLECT_STATS=1**. Default: standard output. + +\--log-file=_PATH_ +: Redirect DTO output to _PATH_._program_._pid_, one file per process. + Sets **DTO_LOG_FILE**. Default: standard output. + +\--log-level=_LEVEL_ +: Verbosity of DTO logging: _0_, _1_, or _2_. Sets **DTO_LOG_LEVEL**. + Default: _0_. + +-?, \--help +: Print help message and list of options with description. + +\--usage +: Print usage string. + +-V, \--version +: Print the version. + + +EXIT STATUS +=========== +On success **dtoctl** does not return: `COMMAND` replaces it, so the exit +status is that of `COMMAND`. Otherwise: + +64 +: Invalid option or option value, or no `COMMAND`. + +125 +: **dtoctl** itself failed, for example because the library cannot be read. + +126 +: `COMMAND` was found but cannot be run. + +127 +: `COMMAND` was not found. + +`--dry-run`, `--help`, and `--version` exit with 0. + + +NOTES +===== +DSA work queues under /dev/dsa/ are often accessible to root only. Put +**sudo** in front of **dtoctl**, not between **dtoctl** and `COMMAND`: + + $ sudo dtoctl -w busypoll ./prog # works + $ dtoctl -w busypoll sudo ./prog # sudo drops LD_PRELOAD and DTO_* + +By default **sudo** also resets the environment, so **DTO_\*** variables +exported in the calling shell do not reach `sudo dtoctl`. Pass the settings as +**dtoctl** options instead. + + +EXAMPLES +======== +Run a program with DTO using the installed **libdto.so**: + + $ dtoctl ./prog + +The equivalent of the manual setup + + $ export LD_PRELOAD=/usr/lib64/libdto.so + $ export DTO_WAIT_METHOD=busypoll + $ export DTO_CPU_SIZE_FRACTION=0.33 + $ export DTO_AUTO_ADJUST_KNOBS=1 + $ ./prog + +becomes a single command: + + $ dtoctl -w busypoll -c 0.33 ./prog + +Latency reduction mode (auto tuning on, busy polling): + + $ dtoctl -w busypoll ./prog + +Power reduction mode (offload everything to DSA, wait with umwait): + + $ dtoctl -w umwait -c 0.0 --no-auto-adjust ./prog + +Avoid cache pollution (offload everything, clear cache control): + + $ dtoctl -w yield -c 0.0 --no-auto-adjust --no-cache-control ./prog + +Use a locally built library and a specific set of work queues, and collect +stats into a file: + + $ dtoctl -l ./libdto.so.1.0 -q "wq0.0;wq2.0;wq4.0;wq6.0" \ + --stats-file=/tmp/dto-stats.txt ./prog + +Check what would run, ignoring anything exported in the shell: + + $ dtoctl --dry-run -i -w umwait -b 16K ./prog + library: /usr/lib64/libdto.so.1.0 (compiled-in default) + LD_PRELOAD=/usr/lib64/libdto.so.1.0 + DTO_WAIT_METHOD=umwait # option + DTO_MIN_BYTES=16384 # option + command: ./prog + + +SEE ALSO +======== +**ld.so**(8), **accel-config**(1) diff --git a/dto.c b/dto.c index 24b606b..2ffce76 100644 --- a/dto.c +++ b/dto.c @@ -1294,6 +1294,10 @@ static int dsa_init(void) max_avg_waits = MAX_AVG_POLL_WAITS; } else LOG_ERROR("umwait not supported. Falling back to default wait method\n"); + } else if (!strncmp(env_str, wait_names[WAIT_YIELD], strlen(wait_names[WAIT_YIELD]))) { + wait_method = WAIT_YIELD; + min_avg_waits = MIN_AVG_YIELD_WAITS; + max_avg_waits = MAX_AVG_YIELD_WAITS; } else if (!strncmp(env_str, wait_names[WAIT_TPAUSE], strlen(wait_names[WAIT_TPAUSE]))) { if (waitpkg_support) { wait_method = WAIT_TPAUSE; diff --git a/dtoctl.c b/dtoctl.c new file mode 100644 index 0000000..7c72444 --- /dev/null +++ b/dtoctl.c @@ -0,0 +1,714 @@ +/* Copyright (C) 2026 SK hynix, Inc. */ +/* SPDX-License-Identifier: MIT */ + +/* + * dtoctl -- a CLI wrapper that runs a program with the DSA Transparent Offload + * (DTO) library preloaded. It converts command line options into the DTO_* + * environment variables and prepends libdto.so to LD_PRELOAD so that users do + * not need to export the variables or set up LD_PRELOAD manually. + */ + +#ifndef _GNU_SOURCE +#define _GNU_SOURCE /* for asprintf() and environ */ +#endif + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#ifndef LIBDTO_PATH +#define LIBDTO_PATH "/usr/lib64/libdto.so" +#endif + +#ifndef DTO_VERSION +#define DTO_VERSION "unknown" +#endif + +/* exit statuses for dtoctl's own failures, following env(1) and timeout(1) */ +#define EXIT_DTOCTL_FAILURE 125 +#define EXIT_CANNOT_INVOKE 126 +#define EXIT_ENOENT 127 + +/* libdto copies DTO_WQ_LIST into a buffer of this size */ +#define WQ_LIST_MAX 256 + +/* argp option keys for the long-only options (no short equivalents) */ +enum { + OPT_UMWAIT_DELAY = 0x100, + OPT_OVERLAPPING_MEMMOVE, + OPT_LOG_FILE, + OPT_LOG_LEVEL, + OPT_STATS_FILE, + OPT_NO_MEMCPY, + OPT_NO_MEMMOVE, + OPT_NO_MEMSET, + OPT_NO_MEMCMP, + OPT_NO_CACHE_CONTROL, + OPT_NO_AUTO_ADJUST, + OPT_STDC_ONLY, + OPT_STATS, + OPT_DRY_RUN, +}; + +struct opts { + int idx; + char *exename; + + /* library / preload */ + const char *library; + bool ignore_env; + bool dry_run; + + /* value options (NULL means "not specified") */ + const char *wait_method; + const char *min_bytes; + const char *cpu_fraction; + const char *numa_aware; + const char *wq_list; + const char *umwait_delay; + const char *overlapping_memmove; /* "0" or "1" after validation */ + const char *log_file; + const char *log_level; + const char *stats_file; + + /* flag options (false means "not specified") */ + bool no_memcpy; + bool no_memmove; + bool no_memset; + bool no_memcmp; + bool no_cache_control; + bool no_auto_adjust; + bool stdc_only; + bool stats; +}; + +/* (a part of) output in --help and --version options (generated by argp runtime) */ +const char *argp_program_version = "dtoctl (DTO) " DTO_VERSION; +const char *argp_program_bug_address = "https://github.com/intel/DTO/issues"; + +static const char *const wait_methods[] = { + "busypoll", "umwait", "yield", "tpause", NULL +}; + +/* Option groups are separated using entries with only a .doc field. */ +static struct argp_option dtoctl_options[] = { + {.doc = "General:", .group = 1}, + {.name = "library", + .key = 'l', + .arg = "PATH", + .doc = "Path to libdto.so to prepend to LD_PRELOAD. Overrides $DTO_LIBRARY " + "and the installed library (" LIBDTO_PATH ")"}, + {.name = "ignore-env", + .key = 'i', + .doc = "Unset the DTO_* variables inherited from the environment, so that " + "only the options given here and DTO's defaults apply"}, + {.name = "dry-run", + .key = OPT_DRY_RUN, + .doc = "Print the library, the DTO_* environment and the command that " + "would run, then exit without running it"}, + + {.doc = "Offload tuning:", .group = 2}, + {.name = "wait-method", + .key = 'w', + .arg = "METHOD", + .doc = "How to wait for DSA completion: yield, busypoll, umwait, or tpause " + "(sets DTO_WAIT_METHOD; default: busypoll)"}, + {.name = "min-bytes", + .key = 'b', + .arg = "BYTES", + .doc = "Minimum operation size offloaded to DSA; accepts a K, M or G suffix " + "(sets DTO_MIN_BYTES; default: 64K)"}, + {.name = "cpu-fraction", + .key = 'c', + .arg = "FRACTION", + .doc = "Fraction (0.0 <= f < 1.0) of each operation done on CPU in parallel " + "to DSA (sets DTO_CPU_SIZE_FRACTION; default: 0.0)"}, + {.name = "numa-aware", + .key = 'n', + .arg = "MODE", + .doc = "NUMA awareness: 0 disable, 1 buffer-centric, 2 cpu-centric " + "(sets DTO_IS_NUMA_AWARE; default: 0)"}, + {.name = "wq-list", + .key = 'q', + .arg = "LIST", + .doc = "Semicolon-separated list of DSA WQs to use, e.g. \"wq0.0;wq2.0\" " + "(sets DTO_WQ_LIST; default: auto-discover all WQs)"}, + {.name = "umwait-delay", + .key = OPT_UMWAIT_DELAY, + .arg = "CYCLES", + .doc = "Delay in cycles for the umwait instruction (sets DTO_UMWAIT_DELAY; " + "default: 100000)"}, + {.name = "overlapping-memmove", + .key = OPT_OVERLAPPING_MEMMOVE, + .arg = "WHERE", + .doc = "Where to run memmove with overlapping buffers: cpu or dsa " + "(sets DTO_OVERLAPPING_MEMMOVE_ACTION; default: cpu)"}, + + {.doc = "Disable DSA offload for specific operations (DSA is used by default):", + .group = 3}, + {.name = "no-memcpy", + .key = OPT_NO_MEMCPY, + .doc = "Use system memcpy instead of DSA (sets DTO_DSA_MEMCPY=0)"}, + {.name = "no-memmove", + .key = OPT_NO_MEMMOVE, + .doc = "Use system memmove instead of DSA (sets DTO_DSA_MEMMOVE=0)"}, + {.name = "no-memset", + .key = OPT_NO_MEMSET, + .doc = "Use system memset instead of DSA (sets DTO_DSA_MEMSET=0)"}, + {.name = "no-memcmp", + .key = OPT_NO_MEMCMP, + .doc = "Use system memcmp instead of DSA (sets DTO_DSA_MEMCMP=0)"}, + + {.doc = "Other toggles:", .group = 4}, + {.name = "no-cache-control", + .key = OPT_NO_CACHE_CONTROL, + .doc = "Clear the DSA cache control flag to avoid cache pollution " + "(sets DTO_DSA_CC=0; default: cache control on)"}, + {.name = "no-auto-adjust", + .key = OPT_NO_AUTO_ADJUST, + .doc = "Disable auto tuning of cpu-fraction and min-bytes " + "(sets DTO_AUTO_ADJUST_KNOBS=0; default: auto tuning on)"}, + {.name = "stdc-only", + .key = OPT_STDC_ONLY, + .doc = "Use only the standard C memory functions, no DSA offload " + "(sets DTO_USESTDC_CALLS=1; default: DSA offload enabled)"}, + + {.doc = "Statistics and logging:", .group = 5}, + {.name = "stats", + .key = OPT_STATS, + .doc = "Enable stats collection; for debugging/profiling only " + "(sets DTO_COLLECT_STATS=1; default: off)"}, + {.name = "stats-file", + .key = OPT_STATS_FILE, + .arg = "PATH", + .doc = "Write the stats histogram to PATH instead of standard output; " + "implies --stats (sets DTO_STATS_FILE; default: standard output)"}, + {.name = "log-file", + .key = OPT_LOG_FILE, + .arg = "PATH", + .doc = "Redirect DTO output to PATH.. (sets DTO_LOG_FILE; " + "default: standard output)"}, + {.name = "log-level", + .key = OPT_LOG_LEVEL, + .arg = "LEVEL", + .doc = "Verbosity of DTO logging: 0, 1, or 2 (sets DTO_LOG_LEVEL; " + "default: 0)"}, + + {0}, +}; + +/* Parse a decimal integer, rejecting signs, empty strings and trailing garbage. */ +static bool parse_ulong(const char *s, unsigned long *val) +{ + char *end; + + if (!isdigit((unsigned char)*s)) + return false; + errno = 0; + *val = strtoul(s, &end, 10); + return errno == 0 && *end == '\0'; +} + +/* Parse a byte count with an optional K, M or G suffix (powers of 1024). */ +static bool parse_size(const char *s, unsigned long *val) +{ + unsigned int shift = 0; + unsigned long n; + char *end; + + if (!isdigit((unsigned char)*s)) + return false; + errno = 0; + n = strtoul(s, &end, 10); + if (errno) + return false; + + switch (*end) { + case 'k': case 'K': + shift = 10; + end++; + break; + case 'm': case 'M': + shift = 20; + end++; + break; + case 'g': case 'G': + shift = 30; + end++; + break; + } + if (*end != '\0' || n > (ULONG_MAX >> shift)) + return false; + + *val = n << shift; + return true; +} + +/* Accept an integer in [0, max] for options that select a mode or level. */ +static const char *check_level(struct argp_state *state, const char *name, + char *arg, unsigned long max) +{ + unsigned long val; + + if (!parse_ulong(arg, &val) || val > max) + argp_error(state, "invalid %s '%s': must be an integer from 0 to %lu", + name, arg, max); + return arg; +} + +static const char *check_path(struct argp_state *state, const char *name, char *arg) +{ + if (*arg == '\0') + argp_error(state, "--%s needs a non-empty PATH", name); + return arg; +} + +static error_t parse_option(int key, char *arg, struct argp_state *state) +{ + struct opts *opts = state->input; + unsigned long val = 0; + char *end, *bytes; + double frac; + int i; + + switch (key) { + case 'l': + opts->library = check_path(state, "library", arg); + break; + case 'i': + opts->ignore_env = true; + break; + case OPT_DRY_RUN: + opts->dry_run = true; + break; + case 'w': + for (i = 0; wait_methods[i]; i++) { + if (!strcmp(arg, wait_methods[i])) + break; + } + if (wait_methods[i] == NULL) + argp_error(state, "invalid wait method '%s': must be yield, " + "busypoll, umwait, or tpause", arg); + opts->wait_method = arg; + break; + case 'b': + if (!parse_size(arg, &val)) + argp_error(state, "invalid minimum size '%s': must be a byte " + "count, optionally with a K, M or G suffix", arg); + /* libdto takes plain bytes, so expand any suffix here */ + if (asprintf(&bytes, "%lu", val) < 0) + argp_failure(state, EXIT_DTOCTL_FAILURE, errno, "--min-bytes"); + opts->min_bytes = bytes; + break; + case 'c': + errno = 0; + frac = strtod(arg, &end); + /* the negated range check also rejects NaN */ + if (end == arg || *end != '\0' || errno || !(frac >= 0.0 && frac < 1.0)) + argp_error(state, "invalid cpu fraction '%s': must be a number " + ">= 0.0 and < 1.0", arg); + opts->cpu_fraction = arg; + break; + case 'n': + opts->numa_aware = check_level(state, "NUMA mode", arg, 2); + break; + case 'q': + if (*arg == '\0' || strlen(arg) >= WQ_LIST_MAX) + argp_error(state, "invalid WQ list: must be 1 to %d characters", + WQ_LIST_MAX - 1); + opts->wq_list = arg; + break; + case OPT_UMWAIT_DELAY: + /* libdto treats 0 as "use the default", so reject it */ + if (!parse_ulong(arg, &val) || val == 0) + argp_error(state, "invalid umwait delay '%s': must be a positive " + "number of cycles", arg); + opts->umwait_delay = arg; + break; + case OPT_OVERLAPPING_MEMMOVE: + if (!strcmp(arg, "cpu")) + opts->overlapping_memmove = "0"; + else if (!strcmp(arg, "dsa")) + opts->overlapping_memmove = "1"; + else + argp_error(state, "invalid --overlapping-memmove '%s': must be " + "cpu or dsa", arg); + break; + case OPT_NO_MEMCPY: + opts->no_memcpy = true; + break; + case OPT_NO_MEMMOVE: + opts->no_memmove = true; + break; + case OPT_NO_MEMSET: + opts->no_memset = true; + break; + case OPT_NO_MEMCMP: + opts->no_memcmp = true; + break; + case OPT_NO_CACHE_CONTROL: + opts->no_cache_control = true; + break; + case OPT_NO_AUTO_ADJUST: + opts->no_auto_adjust = true; + break; + case OPT_STDC_ONLY: + opts->stdc_only = true; + break; + case OPT_STATS: + opts->stats = true; + break; + case OPT_LOG_FILE: + opts->log_file = check_path(state, "log-file", arg); + break; + case OPT_LOG_LEVEL: + opts->log_level = check_level(state, "log level", arg, 2); + break; + case OPT_STATS_FILE: + opts->stats_file = check_path(state, "stats-file", arg); + break; + + case ARGP_KEY_ARG: + /* leave the program and its arguments to ARGP_KEY_ARGS */ + return ARGP_ERR_UNKNOWN; + + case ARGP_KEY_ARGS: + /* process remaining non-option arguments */ + opts->exename = state->argv[state->next]; + opts->idx = state->next; + break; + + case ARGP_KEY_END: + /* --dry-run can show the environment without a program */ + if (opts->exename == NULL && !opts->dry_run) + argp_usage(state); + break; + + default: + return ARGP_ERR_UNKNOWN; + } + return 0; +} + +/* DTO_* variables set from options, in the order they were set, for --dry-run */ +static const char *option_vars[32]; +static int nr_option_vars; + +static void set_option_var(const char *name, const char *value) +{ + if (setenv(name, value, 1)) { + perror("dtoctl: setenv"); + exit(EXIT_DTOCTL_FAILURE); + } + option_vars[nr_option_vars++] = name; +} + +static bool is_option_var(const char *name, int len) +{ + for (int i = 0; i < nr_option_vars; i++) { + if ((int)strlen(option_vars[i]) == len && !strncmp(option_vars[i], name, len)) + return true; + } + return false; +} + +/* Unset every DTO_* variable inherited from the caller (--ignore-env). */ +static void clear_dto_environ(void) +{ + char **env, *eq, *name; + bool found; + + /* unsetenv() reshuffles environ, so rescan it after every removal */ + do { + found = false; + for (env = environ; *env; env++) { + eq = strchr(*env, '='); + if (strncmp(*env, "DTO_", 4) || eq == NULL) + continue; + name = strndup(*env, eq - *env); + if (name == NULL || unsetenv(name)) { + perror("dtoctl: unsetenv"); + exit(EXIT_DTOCTL_FAILURE); + } + free(name); + found = true; + break; + } + } while (found); +} + +static void setup_dto_environ(struct opts *opts) +{ + /* value options: only set when explicitly specified */ + if (opts->wait_method) + set_option_var("DTO_WAIT_METHOD", opts->wait_method); + if (opts->min_bytes) + set_option_var("DTO_MIN_BYTES", opts->min_bytes); + if (opts->cpu_fraction) + set_option_var("DTO_CPU_SIZE_FRACTION", opts->cpu_fraction); + if (opts->numa_aware) + set_option_var("DTO_IS_NUMA_AWARE", opts->numa_aware); + if (opts->wq_list) + set_option_var("DTO_WQ_LIST", opts->wq_list); + if (opts->umwait_delay) + set_option_var("DTO_UMWAIT_DELAY", opts->umwait_delay); + if (opts->overlapping_memmove) + set_option_var("DTO_OVERLAPPING_MEMMOVE_ACTION", opts->overlapping_memmove); + if (opts->log_file) + set_option_var("DTO_LOG_FILE", opts->log_file); + if (opts->log_level) + set_option_var("DTO_LOG_LEVEL", opts->log_level); + if (opts->stats_file) + set_option_var("DTO_STATS_FILE", opts->stats_file); + + /* flag options: only set when explicitly specified */ + if (opts->no_memcpy) + set_option_var("DTO_DSA_MEMCPY", "0"); + if (opts->no_memmove) + set_option_var("DTO_DSA_MEMMOVE", "0"); + if (opts->no_memset) + set_option_var("DTO_DSA_MEMSET", "0"); + if (opts->no_memcmp) + set_option_var("DTO_DSA_MEMCMP", "0"); + if (opts->no_cache_control) + set_option_var("DTO_DSA_CC", "0"); + if (opts->no_auto_adjust) + set_option_var("DTO_AUTO_ADJUST_KNOBS", "0"); + if (opts->stdc_only) + set_option_var("DTO_USESTDC_CALLS", "1"); + /* libdto ignores DTO_STATS_FILE unless stats are collected */ + if (opts->stats || opts->stats_file) + set_option_var("DTO_COLLECT_STATS", "1"); +} + +/* The libdto.so installed relative to this dtoctl, if there is one. */ +static char *installed_library(void) +{ +#ifdef LIBDTO_REL_PATH + char exe[PATH_MAX], *path, *real; + ssize_t len; + + len = readlink("/proc/self/exe", exe, sizeof(exe) - 1); + if (len < 0) + return NULL; + exe[len] = '\0'; + + if (asprintf(&path, "%s/%s", dirname(exe), LIBDTO_REL_PATH) < 0) + return NULL; + real = realpath(path, NULL); + free(path); + if (real && access(real, R_OK)) { + free(real); + real = NULL; + } + return real; +#else + return NULL; +#endif +} + +/* + * Resolve the libdto.so to preload: --library, then $DTO_LIBRARY, then the + * library installed alongside dtoctl, then the compiled-in default. The + * result is an absolute path, so it still loads after the program changes + * directory. A missing library is an error here; ld.so would only print a + * warning and run the program without DTO. + */ +static char *resolve_library(struct opts *opts, const char **source) +{ + const char *lib; + char *path; + + if (opts->library) { + lib = opts->library; + *source = "--library"; + } else if ((lib = getenv("DTO_LIBRARY")) && *lib) { + *source = "DTO_LIBRARY"; + } else if ((path = installed_library())) { + *source = "installed with dtoctl"; + return path; + } else { + lib = LIBDTO_PATH; + *source = "compiled-in default"; + } + + path = realpath(lib, NULL); + if (path == NULL || access(path, R_OK)) { + fprintf(stderr, "dtoctl: cannot use DTO library '%s' (%s): %s\n", + lib, *source, strerror(errno)); + if (opts->library == NULL) + fprintf(stderr, "dtoctl: use -l/--library or DTO_LIBRARY to " + "choose a libdto.so\n"); + exit(EXIT_DTOCTL_FAILURE); + } + + /* ld.so splits LD_PRELOAD at colons and spaces */ + if (strpbrk(path, ": ")) { + fprintf(stderr, "dtoctl: DTO library path '%s' cannot be preloaded: " + "it contains a colon or a space\n", path); + exit(EXIT_DTOCTL_FAILURE); + } + return path; +} + +/* true for an LD_PRELOAD entry that names a libdto.so (libdto.so, libdto.so.1, ...) */ +static bool is_libdto(const char *entry) +{ + const char *base = strrchr(entry, '/'); + + base = base ? base + 1 : entry; + return !strncmp(base, "libdto.so", strlen("libdto.so")); +} + +/* + * Put lib at the front of LD_PRELOAD. Other libdto.so entries are dropped: + * two copies of DTO in one process would both initialize and both open the + * work queues. + */ +static void setup_preload(const char *lib) +{ + const char *cur = getenv("LD_PRELOAD"); + char *preload, *copy, *entry, *save, *real; + + /* + * LD_PRELOAD is a list, so it can be much longer than a single path. + * Size the result from the inputs: the kept entries and their colons + * never take more than the old value plus one byte. + */ + preload = malloc(strlen(lib) + (cur ? strlen(cur) : 0) + 2); + copy = strdup(cur ? cur : ""); + if (preload == NULL || copy == NULL) { + perror("dtoctl: cannot build LD_PRELOAD"); + exit(EXIT_DTOCTL_FAILURE); + } + strcpy(preload, lib); + + for (entry = strtok_r(copy, ": ", &save); entry; + entry = strtok_r(NULL, ": ", &save)) { + if (is_libdto(entry)) { + real = realpath(entry, NULL); + if (real == NULL || strcmp(real, lib)) + fprintf(stderr, "dtoctl: dropping '%s' from LD_PRELOAD " + "in favor of '%s'\n", entry, lib); + free(real); + continue; + } + strcat(preload, ":"); + strcat(preload, entry); + } + + if (setenv("LD_PRELOAD", preload, 1)) { + perror("dtoctl: setenv"); + exit(EXIT_DTOCTL_FAILURE); + } + free(copy); + free(preload); +} + +/* Print s, single-quoted if a shell would otherwise split or expand it. */ +static void print_quoted(const char *s) +{ + static const char safe[] = "abcdefghijklmnopqrstuvwxyz" + "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_@%+=:,./-"; + + if (*s && strspn(s, safe) == strlen(s)) { + fputs(s, stdout); + return; + } + + putchar('\''); + for (; *s; s++) { + if (*s == '\'') + fputs("'\\''", stdout); + else + putchar(*s); + } + putchar('\''); +} + +static void print_var(const char *name, int name_len, const char *value, + const char *source) +{ + printf("%.*s=", name_len, name); + print_quoted(value); + if (source) + printf(" # %s", source); + putchar('\n'); +} + +/* --dry-run: show the library, the environment and the command to be run */ +static void print_dry_run(const char *lib, const char *source, char **argv) +{ + char **env; + int len; + + printf("library: %s (%s)\n", lib, source); + print_var("LD_PRELOAD", strlen("LD_PRELOAD"), getenv("LD_PRELOAD"), NULL); + + /* variables from options first, in option order, then inherited ones */ + for (int i = 0; i < nr_option_vars; i++) + print_var(option_vars[i], strlen(option_vars[i]), + getenv(option_vars[i]), "option"); + for (env = environ; *env; env++) { + len = strcspn(*env, "="); + if (!strncmp(*env, "DTO_", 4) && (*env)[len] == '=' && + !is_option_var(*env, len)) + print_var(*env, len, *env + len + 1, "inherited"); + } + + if (argv == NULL) + return; + fputs("command:", stdout); + for (int i = 0; argv[i]; i++) { + putchar(' '); + print_quoted(argv[i]); + } + putchar('\n'); +} + +int main(int argc, char *argv[]) +{ + struct opts opts = {0}; + struct argp argp = { + .options = dtoctl_options, + .parser = parse_option, + .args_doc = " [...]", + .doc = "dtoctl -- run a program with DSA Transparent Offload (DTO) preloaded" + "\vOnly the options given are applied: inherited DTO_* variables " + "are kept unless an option overrides them or -i is given. " + "Exit status is the program's own, or 125 if dtoctl fails, " + "126 if the program cannot be run, 127 if it is not found.", + }; + const char *source; + char *lib; + int err; + + argp_parse(&argp, argc, argv, ARGP_IN_ORDER, NULL, &opts); + + /* clear inherited variables first; this also drops DTO_LIBRARY */ + if (opts.ignore_env) + clear_dto_environ(); + setup_dto_environ(&opts); + + lib = resolve_library(&opts, &source); + setup_preload(lib); + + if (opts.dry_run) { + print_dry_run(lib, source, opts.exename ? argv + opts.idx : NULL); + return 0; + } + + /* pass only the target program and its arguments to execvp() */ + execvp(opts.exename, argv + opts.idx); + err = errno; + fprintf(stderr, "dtoctl: cannot run '%s': %s\n", opts.exename, strerror(err)); + + return err == ENOENT ? EXIT_ENOENT : EXIT_CANNOT_INVOKE; +} diff --git a/tests/test_dtoctl.sh b/tests/test_dtoctl.sh new file mode 100755 index 0000000..8cf4cf0 --- /dev/null +++ b/tests/test_dtoctl.sh @@ -0,0 +1,152 @@ +#!/bin/bash +# SPDX-License-Identifier: MIT +# +# Tests for dtoctl's option handling, library resolution and environment setup. +# No DSA hardware is needed: most checks read dtoctl's --dry-run output or the +# environment the program receives, and the end-to-end check reads libdto's own +# configuration trace, which libdto prints even when it falls back to the CPU. +# +# Usage: test_dtoctl.sh + +DTOCTL="$1" +LIB=$(realpath "$2") +LIBDIR=$(dirname "$LIB") +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT +failures=0 + +# Start from a clean environment so the caller's DTO setup cannot leak in. +unset LD_PRELOAD +for var in $(env | sed -n 's/^\(DTO_[A-Za-z0-9_]*\)=.*/\1/p'); do + unset "$var" +done + +# run : capture the command's output and exit status +run() { + out=$("$@" 2>&1) + rc=$? + cmd="$*" +} + +fail() { + echo "FAIL: $1" + echo " command: $cmd" + printf '%s\n' "$out" | sed 's/^/ | /' + failures=$((failures + 1)) +} + +# has : the last output has a line matching pattern +has() { + if printf '%s\n' "$out" | grep -q -- "$2"; then echo "PASS: $1"; else fail "$1"; fi +} + +# lacks : no line of the last output matches pattern +lacks() { + if printf '%s\n' "$out" | grep -q -- "$2"; then fail "$1"; else echo "PASS: $1"; fi +} + +# status : the last command exited with code +status() { + if [ "$rc" -eq "$2" ]; then echo "PASS: $1"; else fail "$1 (exit $rc, expected $2)"; fi +} + +# --- every option sets its variable --- +run "$DTOCTL" -l "$LIB" --dry-run -w umwait -b 64K -c 0.25 -n 2 -q 'wq0.0;wq2.0' \ + --umwait-delay 5000 --overlapping-memmove=dsa --no-memcpy --no-memmove \ + --no-memset --no-memcmp --no-cache-control --no-auto-adjust --stdc-only \ + --log-file=/tmp/dto.log --log-level 1 --stats-file=/tmp/dto.stats prog 'a b' +status "dry-run succeeds" 0 +for v in DTO_WAIT_METHOD=umwait DTO_MIN_BYTES=65536 DTO_CPU_SIZE_FRACTION=0.25 \ + DTO_IS_NUMA_AWARE=2 "DTO_WQ_LIST='wq0.0;wq2.0'" DTO_UMWAIT_DELAY=5000 \ + DTO_OVERLAPPING_MEMMOVE_ACTION=1 DTO_DSA_MEMCPY=0 DTO_DSA_MEMMOVE=0 \ + DTO_DSA_MEMSET=0 DTO_DSA_MEMCMP=0 DTO_DSA_CC=0 DTO_AUTO_ADJUST_KNOBS=0 \ + DTO_USESTDC_CALLS=1 DTO_LOG_FILE=/tmp/dto.log DTO_LOG_LEVEL=1 \ + DTO_STATS_FILE=/tmp/dto.stats DTO_COLLECT_STATS=1; do + has "option sets $v" "^$v # option\$" +done +has "dry-run shows the library" "^library: $LIB (--library)\$" +has "dry-run shows LD_PRELOAD" "^LD_PRELOAD=$LIB\$" +has "dry-run shows the command" "^command: prog 'a b'\$" + +# --- size suffixes --- +for s in 4096:4096 4k:4096 1M:1048576 2G:2147483648; do + run "$DTOCTL" -l "$LIB" --dry-run -b "${s%%:*}" true + has "-b ${s%%:*} is ${s##*:} bytes" "^DTO_MIN_BYTES=${s##*:} # option\$" +done + +# --- inherited variables --- +run env DTO_WAIT_METHOD=umwait DTO_MIN_BYTES=4096 "$DTOCTL" -l "$LIB" --dry-run -w busypoll true +has "option overrides an inherited variable" "^DTO_WAIT_METHOD=busypoll # option\$" +has "inherited variable is kept" "^DTO_MIN_BYTES=4096 # inherited\$" + +run env DTO_MIN_BYTES=4096 DTO_USESTDC_CALLS=1 "$DTOCTL" -i -l "$LIB" --dry-run true +lacks "-i drops inherited variables" "^DTO_" + +run env DTO_LIBRARY="$LIB" "$DTOCTL" -i --dry-run true +lacks "-i ignores DTO_LIBRARY" "(DTO_LIBRARY)" + +# --- the program receives the environment --- +run "$DTOCTL" -l "$LIB" --stdc-only -w yield env +has "program sees DTO_WAIT_METHOD" "^DTO_WAIT_METHOD=yield\$" +has "program sees LD_PRELOAD" "^LD_PRELOAD=$LIB\$" + +run "$DTOCTL" -l "$LIB" --stdc-only echo -w -l +has "options after the program are the program's" "^-w -l\$" + +# --- library resolution and LD_PRELOAD --- +run env DTO_LIBRARY="$LIB" "$DTOCTL" --dry-run true +has "DTO_LIBRARY selects the library" "^library: $LIB (DTO_LIBRARY)\$" + +run sh -c 'cd "$1" && shift && exec "$@"' sh "$LIBDIR" \ + "$DTOCTL" -l "./$(basename "$LIB")" --dry-run true +has "relative library path becomes absolute" "^LD_PRELOAD=$LIB\$" + +run "$DTOCTL" -l "$TMP/missing/libdto.so" true +status "missing library fails" 125 +has "missing library is reported" "cannot use DTO library" + +run env LD_PRELOAD="$TMP/libother.so $TMP/old/libdto.so.1" "$DTOCTL" -l "$LIB" --dry-run true +has "existing LD_PRELOAD entries are kept after libdto" "^LD_PRELOAD=$LIB:$TMP/libother.so\$" +has "another libdto in LD_PRELOAD is dropped" "dropping '$TMP/old/libdto.so.1'" + +# --- invalid values are rejected --- +for bad in "-w bogus" "-b 64Q" "-b abc" "-b -1" "-c 1.5" "-c nan" "-n 3" \ + "--log-level x" "--umwait-delay 0" "--overlapping-memmove=gpu"; do + run "$DTOCTL" -l "$LIB" $bad true + status "rejects $bad" 64 +done +run "$DTOCTL" -l "$LIB" -q '' true +status "rejects an empty WQ list" 64 + +# --- exit status --- +run "$DTOCTL" -l "$LIB" +status "no program is a usage error" 64 + +run "$DTOCTL" -l "$LIB" --dry-run +status "--dry-run needs no program" 0 +lacks "--dry-run without a program prints no command" "^command:" + +run "$DTOCTL" -l "$LIB" "$TMP/no-such-program" +status "program not found" 127 + +touch "$TMP/not-executable" +run "$DTOCTL" -l "$LIB" "$TMP/not-executable" +status "program not executable" 126 + +run "$DTOCTL" -l "$LIB" --stdc-only sh -c 'exit 7' +status "program's exit status is returned" 7 + +run "$DTOCTL" --version +has "--version" "^dtoctl (DTO) " + +# --- libdto applies the settings (DSA is optional) --- +run "$DTOCTL" -l "$LIB" --log-level 2 -w yield -b 8K -c 0.5 --no-auto-adjust true +has "libdto applies the options" \ + "dsa_min_size: 8192, cpu_size_fraction: 0.50, wait_method: yield, auto_adjust_knobs: 0" + +echo +if [ "$failures" -ne 0 ]; then + echo "$failures dtoctl test(s) failed" + exit 1 +fi +echo "All dtoctl tests passed" From 2d037e6b5d98e08b54dbf5628933ba4d9f64af08 Mon Sep 17 00:00:00 2001 From: Daniel Byrne Date: Tue, 27 Oct 2026 17:19:10 -0400 Subject: [PATCH 02/11] tests: include in test_perf.c (#44) test_perf.c uses fork(), _exit(), sysconf() and _SC_PAGESIZE without including . With GCC 12 the transitive include is missing and the CMake build fails with "_SC_PAGESIZE undeclared". --- tests/test_perf.c | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/test_perf.c b/tests/test_perf.c index 7c93278..e456c5d 100644 --- a/tests/test_perf.c +++ b/tests/test_perf.c @@ -36,6 +36,7 @@ #include "dto_test_utils.h" #include +#include #include #include #include From 7314599f6f145b14d1f91a60bc45c8c6dcedf5ed Mon Sep 17 00:00:00 2001 From: Daniel Byrne Date: Tue, 27 Oct 2026 17:19:26 -0400 Subject: [PATCH 03/11] Fix SIGFPE at startup when CPUID leaf 0x15 is unusable with tpause (#43) With DTO_WAIT_METHOD=tpause, init_dto() converts the wait from ns to TSC cycles using CPUID leaf 0x15 and divides by the denominator unconditionally. If the leaf is absent (__get_cpuid() returns 0 with the outputs uninitialized) or reports den=0, as on many VMs, the constructor dies with SIGFPE before main(). It also ran after dsa_init() had failed and the library was falling back to CPU, where the value is unused. Only do the conversion when DSA is in use and tpause is selected, and require the leaf to be present with non-zero den, num and frequency. Otherwise log an error and fall back to busypoll. Also fix waitpkg detection in dsa_init(), which queried CPUID leaf 0 instead of leaf 7. Leaf 0's ECX is the ASCII "ntel", which has bit 5 set, so waitpkg_support was always 1 and tpause/umwait on a CPU without WAITPKG would SIGILL instead of falling back. --- dto.c | 63 ++++++++++++++++++++++++++++++++++++----------------------- 1 file changed, 39 insertions(+), 24 deletions(-) diff --git a/dto.c b/dto.c index 2ffce76..3514a87 100644 --- a/dto.c +++ b/dto.c @@ -1270,10 +1270,10 @@ static int dsa_init(void) const char *env_str; char wq_list[256]; - /* detect waitpkg support */ - leaf = 7; + /* detect waitpkg support: CPUID.(EAX=7,ECX=0):ECX bit 5 */ + leaf = 0; waitpkg = 0; - if (__get_cpuid(0, &leaf, unused, &waitpkg, unused + 1)) { + if (__get_cpuid_count(7, 0, &leaf, unused, &waitpkg, unused + 1)) { if (waitpkg & 0x20) { LOG_TRACE("waitpkg supported\n"); waitpkg_support = 1; @@ -1567,27 +1567,42 @@ static int init_dto(void) use_std_lib_calls = 1; } - // calculate the wait time for TPAUSE - if (wait_method == WAIT_TPAUSE) { - unsigned int num, den, freq; - unsigned int empty; - unsigned long long tmp; - __get_cpuid( 0x15, &den, &num, &freq, &empty ); - freq /= 1000; - LOG_TRACE( "Core Freq = %u kHz\n", freq ); - LOG_TRACE( "TSC Mult = %u\n", num ); - LOG_TRACE( "TSC Den = %u\n", den ); - freq *= num; - freq /= den; - LOG_TRACE( "CPU freq = %u kHz\n", freq ); - - LOG_TRACE( "Requested wait: %llu nsec\n", tpause_wait_time ); - tmp = tpause_wait_time; - tmp *= freq; - tpause_wait_time = tmp / NSEC_PER_MSEC; - LOG_TRACE( "Requested wait duration: %llu cycles\n", tpause_wait_time ); - } - + /* + * Calculate the wait time for TPAUSE. This converts ns to TSC + * cycles using CPUID leaf 0x15, which may be absent or report a + * zero denominator/crystal frequency (common in VMs and on + * older parts), so verify it before dividing. + */ + if (!use_std_lib_calls && wait_method == WAIT_TPAUSE) { + unsigned int num = 0, den = 0, freq = 0; + unsigned int empty; + unsigned long long tmp; + + if (!__get_cpuid(0x15, &den, &num, &freq, &empty) || + den == 0 || num == 0 || freq == 0) { + LOG_ERROR("CPUID leaf 0x15 unavailable (den=%u, num=%u, freq=%u); " + "cannot compute TSC frequency for tpause. Falling back to busypoll\n", + den, num, freq); + wait_method = WAIT_BUSYPOLL; + min_avg_waits = MIN_AVG_POLL_WAITS; + max_avg_waits = MAX_AVG_POLL_WAITS; + } else { + freq /= 1000; + LOG_TRACE("Core Freq = %u kHz\n", freq); + LOG_TRACE("TSC Mult = %u\n", num); + LOG_TRACE("TSC Den = %u\n", den); + freq *= num; + freq /= den; + LOG_TRACE("CPU freq = %u kHz\n", freq); + + LOG_TRACE("Requested wait: %llu nsec\n", tpause_wait_time); + tmp = tpause_wait_time; + tmp *= freq; + tpause_wait_time = tmp / NSEC_PER_MSEC; + LOG_TRACE("Requested wait duration: %llu cycles\n", tpause_wait_time); + } + } + // display configuration LOG_TRACE("log_level: %d, collect_stats: %d, use_std_lib_calls: %d, dsa_min_size: %lu, " "cpu_size_fraction: %.2f, wait_method: %s, auto_adjust_knobs: %d, numa_awareness: %s, dto_dsa_cc: %d\n", From b7cc9bab7d182f6ffa649a6441d5497bd681afc6 Mon Sep 17 00:00:00 2001 From: Daniel Byrne Date: Tue, 27 Oct 2026 17:19:46 -0400 Subject: [PATCH 04/11] get_wq: round robin per thread instead of through a shared counter (#40) get_wq() picked a work queue with next_wq++ on a global atomic (and on a per-device atomic in NUMA-aware mode), so every DSA submit on every thread did a locked read-modify-write on one shared cache line. Each thread now keeps its own round-robin position. A thread takes its starting position once from a shared counter, so concurrent threads still spread across the queues, and from then on picking a queue touches only thread-local state. The position is 32 bits, so the rotation no longer skews each time the old atomic_uchar counter wrapped at 256 with a queue count that does not divide it. Measured on a Xeon 6787P with four DSA devices on one node, one shared WQ each: N pinned threads each doing back-to-back memcpy of one size through LD_PRELOAD, auto-tuning off, ns per call (median of 3): 4 KiB 16 threads 2295 -> 1432 32 threads 6629 -> 1549 16 KiB 32 threads 5920 -> 2991 64 KiB 32 threads 16970 -> 12124 Single-thread and device-bound cases are unchanged within noise, and the descriptors stay evenly spread: DSA perfmon counted the same number on each of the four devices in every NUMA mode, before and after. Claude-Session: https://claude.ai/code/session_01Mv7xKtQPkGoXQUvvuuhAEz Co-authored-by: Claude Opus 5.5 --- dto.c | 26 ++++++++++++++++++++++---- 1 file changed, 22 insertions(+), 4 deletions(-) diff --git a/dto.c b/dto.c index 3514a87..dd76ef4 100644 --- a/dto.c +++ b/dto.c @@ -96,7 +96,6 @@ struct dto_wq { struct dto_device { struct dto_wq* wqs[MAX_WQS]; uint8_t num_wqs; - atomic_uchar next_wq; }; #endif @@ -132,7 +131,11 @@ static struct dto_wq wqs[MAX_WQS]; static struct dto_device* devices[MAX_NUMA_NODES]; #endif static uint8_t num_wqs; -static atomic_uchar next_wq; +/* round-robin start position for the next thread that picks a WQ (get_wq) */ +static atomic_uint next_thread_wq; +/* this thread's round-robin position */ +static __thread unsigned int thr_wq_rr; +static __thread bool thr_wq_rr_set; static atomic_uchar dto_initialized; static atomic_uchar dto_initializing; static uint8_t use_std_lib_calls; @@ -1646,6 +1649,21 @@ static void cleanup_dto(void) #endif } +/* Next round-robin position for this thread. Each thread keeps its own, so + * picking a WQ writes no cache line shared with other threads; a counter + * incremented by every submit on every thread was measurable on small + * operations. Threads start at successive positions, so concurrent threads + * still spread across the WQs. */ +static __always_inline unsigned int thread_wq_rr(void) +{ + if (unlikely(!thr_wq_rr_set)) { + thr_wq_rr = atomic_fetch_add_explicit(&next_thread_wq, 1, + memory_order_relaxed); + thr_wq_rr_set = true; + } + return thr_wq_rr++; +} + static __always_inline struct dto_wq *get_wq(void* buf) { struct dto_wq* wq = NULL; @@ -1660,14 +1678,14 @@ static __always_inline struct dto_wq *get_wq(void* buf) struct dto_device* dev = devices[numa_node]; if (dev != NULL && dev->num_wqs > 0) { - wq = dev->wqs[dev->next_wq++ % dev->num_wqs]; + wq = dev->wqs[thread_wq_rr() % dev->num_wqs]; } } } #endif if (wq == NULL) { - wq = &wqs[next_wq++ % num_wqs]; + wq = &wqs[thread_wq_rr() % num_wqs]; } return wq; From 74cfb69f5ef6f191f0690d721e72b04b652d4e0c Mon Sep 17 00:00:00 2001 From: Daniel Byrne Date: Tue, 27 Oct 2026 17:20:41 -0400 Subject: [PATCH 05/11] cmake: build only the library when DTO is a subproject (#39) When another project pulls DTO in with add_subdirectory or FetchContent: - The perf test's build steps used CMAKE_SOURCE_DIR and CMAKE_BINARY_DIR, which are the parent's, so tests/build_baseline.sh and dto.c were not found and the parent's build failed -- with DTO's default options. - DTO_BUILD_TESTS defaulted to ON, adding DTO's tests, its dto-test apps and a perf baseline fetched from GitHub to the parent's build. - An unset CMAKE_BUILD_TYPE was forced to RelWithDebInfo in the cache, changing the parent's build type. Use the CMAKE_CURRENT_* directories, default DTO_BUILD_TESTS and the new DTO_BUILD_TEST_APPS (dto-test, dto-test-wodto) to ON only when DTO is the top-level project, and leave the build type alone otherwise. A top-level build is unchanged. --- CMakeLists.txt | 54 +++++++++++++++++++++++++++++++------------------- README.md | 13 ++++++++++++ TESTING.md | 4 +++- 3 files changed, 50 insertions(+), 21 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 19f76ce..b7a9139 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1,10 +1,20 @@ cmake_minimum_required(VERSION 3.5...3.31) project(DTO VERSION 1.0 LANGUAGES C) -# Default to RelWithDebInfo when no build type is specified. +# DTO may be built on its own or as part of another project (add_subdirectory +# or FetchContent). As a subproject only the library is built by default, and +# the parent's settings are left alone. +if(CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR) + set(DTO_IS_TOP_LEVEL ON) +else() + set(DTO_IS_TOP_LEVEL OFF) +endif() + +# Default to RelWithDebInfo when no build type is specified. Only at the top +# level: CMAKE_BUILD_TYPE is the parent's when DTO is a subproject. # (Skip for multi-config generators, which ignore CMAKE_BUILD_TYPE.) get_property(_is_multi_config GLOBAL PROPERTY GENERATOR_IS_MULTI_CONFIG) -if(NOT _is_multi_config AND NOT CMAKE_BUILD_TYPE) +if(DTO_IS_TOP_LEVEL AND NOT _is_multi_config AND NOT CMAKE_BUILD_TYPE) set(CMAKE_BUILD_TYPE "RelWithDebInfo" CACHE STRING "Choose the type of build" FORCE) set_property(CACHE CMAKE_BUILD_TYPE PROPERTY STRINGS @@ -166,14 +176,18 @@ else() endif() # Build dto-test and dto-test-wodto -add_executable(dto-test dto-test.c) -target_link_libraries(dto-test PRIVATE DTO::dto pthread) - -add_executable(dto-test-wodto dto-test.c) -target_link_libraries(dto-test-wodto PRIVATE pthread) +option(DTO_BUILD_TEST_APPS "Build the dto-test and dto-test-wodto applications" + ${DTO_IS_TOP_LEVEL}) +if(DTO_BUILD_TEST_APPS) + add_executable(dto-test dto-test.c) + target_link_libraries(dto-test PRIVATE DTO::dto pthread) + + add_executable(dto-test-wodto dto-test.c) + target_link_libraries(dto-test-wodto PRIVATE pthread) +endif() # ---- Test Suite ---- -option(DTO_BUILD_TESTS "Build the DTO test suite" ON) +option(DTO_BUILD_TESTS "Build the DTO test suite" ${DTO_IS_TOP_LEVEL}) if(DTO_BUILD_TESTS) enable_testing() @@ -225,20 +239,20 @@ if(DTO_BUILD_TESTS) endif() # Build current dto.c as object file with renamed symbols (cur_memcpy, etc.) - set(PERF_BASELINE_DIR "${CMAKE_BINARY_DIR}/perf_baseline") + set(PERF_BASELINE_DIR "${CMAKE_CURRENT_BINARY_DIR}/perf_baseline") add_custom_command( - OUTPUT ${CMAKE_BINARY_DIR}/dto_current.o + OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/dto_current.o COMMAND ${CMAKE_C_COMPILER} -c ${PERF_OBJ_CFLAGS} - ${CMAKE_SOURCE_DIR}/dto.c - -o ${CMAKE_BINARY_DIR}/dto_current_raw.o + ${CMAKE_CURRENT_SOURCE_DIR}/dto.c + -o ${CMAKE_CURRENT_BINARY_DIR}/dto_current_raw.o COMMAND objcopy --redefine-sym memcpy=cur_memcpy --redefine-sym memset=cur_memset --redefine-sym memcmp=cur_memcmp --redefine-sym memmove=cur_memmove - ${CMAKE_BINARY_DIR}/dto_current_raw.o - ${CMAKE_BINARY_DIR}/dto_current.o - DEPENDS ${CMAKE_SOURCE_DIR}/dto.c + ${CMAKE_CURRENT_BINARY_DIR}/dto_current_raw.o + ${CMAKE_CURRENT_BINARY_DIR}/dto_current.o + DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/dto.c COMMENT "Building dto_current.o with renamed symbols" ) @@ -249,8 +263,8 @@ if(DTO_BUILD_TESTS) # object is still fresh, and a custom command would never re-run once # the object exists. add_custom_target(dto_baseline_build - COMMAND bash ${CMAKE_SOURCE_DIR}/tests/build_baseline.sh - ${CMAKE_SOURCE_DIR} ${PERF_BASELINE_DIR} ${CMAKE_C_COMPILER} + COMMAND bash ${CMAKE_CURRENT_SOURCE_DIR}/tests/build_baseline.sh + ${CMAKE_CURRENT_SOURCE_DIR} ${PERF_BASELINE_DIR} ${CMAKE_C_COMPILER} ${PERF_OBJ_CFLAGS} BYPRODUCTS ${PERF_BASELINE_DIR}/dto_baseline.o COMMENT "Building dto_baseline.o from baseline branch" @@ -258,7 +272,7 @@ if(DTO_BUILD_TESTS) add_executable(dto-test-perf tests/test_perf.c - ${CMAKE_BINARY_DIR}/dto_current.o + ${CMAKE_CURRENT_BINARY_DIR}/dto_current.o ${PERF_BASELINE_DIR}/dto_baseline.o ) add_dependencies(dto-test-perf dto_baseline_build) @@ -290,7 +304,7 @@ if(DTO_BUILD_TESTS) LABELS "perf" TIMEOUT 2400 RUN_SERIAL TRUE - ENVIRONMENT "RESULTS_DIR=${CMAKE_BINARY_DIR}/perf_results;${PERF_COMMON_ENV}" + ENVIRONMENT "RESULTS_DIR=${CMAKE_CURRENT_BINARY_DIR}/perf_results;${PERF_COMMON_ENV}" ) # short sanity variant: memcpy 4k/64k/1m + memset 64k, @@ -301,7 +315,7 @@ if(DTO_BUILD_TESTS) LABELS "perf" TIMEOUT 120 RUN_SERIAL TRUE - ENVIRONMENT "PERF_SHORT=1;RESULTS_DIR=${CMAKE_BINARY_DIR}/perf_results_sanity;${PERF_COMMON_ENV}" + ENVIRONMENT "PERF_SHORT=1;RESULTS_DIR=${CMAKE_CURRENT_BINARY_DIR}/perf_results_sanity;${PERF_COMMON_ENV}" ) else() message(STATUS "perf test: disabled (requires DTO_ACCEL_CONFIG_SUPPORT and " diff --git a/README.md b/README.md index ec906cc..71e1424 100644 --- a/README.md +++ b/README.md @@ -226,6 +226,8 @@ The accel-config and libnuma dependencies are optional and can be disabled at co | `DTO_STATIC_ACCEL_CONFIG` | `OFF` | Statically link libaccel-config into libdto.so instead of depending on the shared library at runtime. Requires `libaccel-config.a` to be installed. | | `DTO_STATIC_NUMA` | `OFF` | Statically link libnuma into libdto.so instead of depending on the shared library at runtime. Requires a `libnuma.a` compiled with `-fPIC`. | | `DTO_BUILD_EXPLICIT` | `ON` | Also build `libdto_explicit`, the explicit API (`dto.h`) without the libc interposers. | +| `DTO_BUILD_TESTS` | `ON` (`OFF` as a subproject) | Build the test suite (see [TESTING.md](TESTING.md)). | +| `DTO_BUILD_TEST_APPS` | `ON` (`OFF` as a subproject) | Build the `dto-test` and `dto-test-wodto` applications. | Example: build without either dependency: ```bash @@ -240,6 +242,17 @@ you need to build without them. The CMake build also produces the test applications in the build directory: `dto-test` (for the -ldto method) and `dto-test-wodto` (for the LD_PRELOAD method). +DTO can also be built as part of another CMake project, with `add_subdirectory` +or `FetchContent`, and linked as `DTO::dto`. Only the library is built then, +unless `DTO_BUILD_TESTS` or `DTO_BUILD_TEST_APPS` is set, and the parent +project's build type is left as it is: +```cmake +include(FetchContent) +FetchContent_Declare(dto GIT_REPOSITORY https://github.com/intel/DTO.git GIT_TAG main) +FetchContent_MakeAvailable(dto) +target_link_libraries(myapp PRIVATE DTO::dto) +``` + CMake installs the library to `/usr/local/lib` by default, while the deprecated Makefile installs to `/usr/lib64`. Pass `-DCMAKE_INSTALL_PREFIX=` at configure time to change the install location. diff --git a/TESTING.md b/TESTING.md index 7dae505..db6b543 100644 --- a/TESTING.md +++ b/TESTING.md @@ -20,7 +20,9 @@ cmake --build build cd build ``` -To disable tests: `-DDTO_BUILD_TESTS=OFF` +To disable tests: `-DDTO_BUILD_TESTS=OFF`. When DTO is built as part of +another CMake project (`add_subdirectory` or `FetchContent`) the tests are off +unless `DTO_BUILD_TESTS` is set. ## Running Tests From 18b789b5a7c73d4eeab47c5e8e878f0175db3de1 Mon Sep 17 00:00:00 2001 From: Daniel Byrne Date: Wed, 21 Oct 2026 11:07:14 -0700 Subject: [PATCH 06/11] stats: per-thread counters, log-scaled histogram, intervals, JSON DTO_COLLECT_STATS kept one global table that every thread updated with atomic adds, so the cost of collecting grew with the thread count: a 4 KiB memcpy on the CPU path cost 55 ns more with stats on at one thread and 2.6 us more at 32 threads. Each thread now counts in its own block, mapped when the thread first records a call and folded into a retired total when the thread exits; output sums the live blocks and the retired total. Stats now cost about 40 ns per call (the two clock reads) at any thread count: extra cost per call, CPU path 1 thr 4 thr 16 thr 32 thr 4 KiB, before +55 +357 +1319 +2612 ns 4 KiB, after +44 +46 +42 +37 ns The histogram's 512 linear 4 KiB buckets, the last of which held everything from 2 MiB up, become log-scaled: under 64 bytes, then four buckets per power of two up to 4 GiB, then one for anything larger. New settings: DTO_STATS_INTERVAL_MS=N a sampler thread writes what happened in each N ms; the records add up to the summary DTO_STATS_FORMAT=json JSON Lines: interval and summary records with calls/bytes/ns per op and path, failures, histogram, and the configuration at exit DTO_STATS_FILE=...%p... %p becomes the process ID DTO_STATS_FILE is now appended to rather than truncated, so a forked or exec'd child's records don't overwrite its parent's; each record carries its pid. After fork the child starts with empty counts, and a %p file is reopened under its own pid. The old dto_stats_log() wrote each line twice when DTO_LOG_FILE was set and reused a consumed va_list; it is gone, along with the TRACE log level that turning on stats forced. Adds tests/test_stats.py (ctest -R stats) with a workload of known calls: totals and buckets, intervals summing to totals, 200 short-lived threads all counted and their blocks unmapped, fork, %p, and text output. --- CMakeLists.txt | 17 + README.md | 114 ++--- TESTING.md | 13 + dto.c | 840 ++++++++++++++++++++++++++++++------ tests/test_stats.py | 141 ++++++ tests/test_stats_workload.c | 132 ++++++ 6 files changed, 1078 insertions(+), 179 deletions(-) create mode 100644 tests/test_stats.py create mode 100644 tests/test_stats_workload.c diff --git a/CMakeLists.txt b/CMakeLists.txt index b7a9139..4419a70 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -221,6 +221,23 @@ if(DTO_BUILD_TESTS) LABELS "functional" TIMEOUT 60 ) + # Stats collection (DTO_COLLECT_STATS): a workload of known calls, checked + # by a Python script. Runs without DSA (everything is counted as CPU). + find_package(Python3 COMPONENTS Interpreter QUIET) + add_executable(dto-test-stats-workload tests/test_stats_workload.c) + target_compile_options(dto-test-stats-workload PRIVATE -fno-builtin) + target_link_libraries(dto-test-stats-workload PRIVATE DTO::dto pthread) + if(Python3_Interpreter_FOUND) + add_test(NAME stats + COMMAND ${Python3_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/tests/test_stats.py + $) + set_tests_properties(stats PROPERTIES + LABELS "functional" + TIMEOUT 120 + ) + else() + message(STATUS "stats test: disabled (python3 not found)") + endif() # ---- A/B performance regression test ---- # The baseline object is compiled from unguarded upstream sources that diff --git a/README.md b/README.md index 71e1424..cb1a842 100644 --- a/README.md +++ b/README.md @@ -50,9 +50,12 @@ dto-4-dsa.conf: An example json config file for configuring DSAs Following environment variables control the behavior of DTO library: DTO_USESTDC_CALLS=0/1, 1 (uses std c memory functions only), 0 (uses DSA along with std c lib call; in case of DSA page fault - reverts to std c lib call). Default is 0. DTO_COLLECT_STATS=0/1, 1 (enables stats collection - #of operations, avg latency for each API, etc.>, 0 (disables stats collection). - Should be enabled for debugging/profiling only, not for perf evaluation (enabling it slows down the workload). Default is 0. - DTO_STATS_FILE= when DTO_COLLECT_STATS=1, redirect histogram output to the provided regular file instead of standard output. - The file is truncated when DTO loads; failures fall back to standard output and emit an error. + Adds about 40 ns per call (two clock reads), whatever the number of threads. Default is 0. See Statistics below. + DTO_STATS_FILE= when DTO_COLLECT_STATS=1, write stats to the provided regular file instead of standard output. Each %p in + the path is replaced by the process ID. Records are appended, not truncated: processes started from this one + append to the same file unless the path has a %p. Failures fall back to standard output and emit an error. + DTO_STATS_FORMAT=text/json text (default): the tables shown below; json: JSON Lines, one record per line (see Statistics). + DTO_STATS_INTERVAL_MS=N when N > 0, also write what happened in every N ms, from a background thread. Default is 0. DTO_WAIT_METHOD= (specifies the method to use while waiting for DSA to complete operation, default is busypoll) DTO_MIN_BYTES=xxxx (specifies minimum size of API call needed for DSA operation execution, default is 65536 bytes) DTO_CPU_SIZE_FRACTION=0.xx (specifies fraction of job performed by CPU, in parallel to DSA). Default is 0.00 @@ -310,63 +313,72 @@ to the DSA device. /cachebench -json_test_config --progress_stats_file=dto.log --report_api_latency -4. Sample histogram given below (generated using DTO_COLLECT_STATS=1) +4. Sample stats output (DTO_COLLECT_STATS=1; see [Statistics](#statistics)) i. Numbers under columns set, cpy, mov, and cmp show number of API calls or per-API completion latency for memset, memcpy, memmove, and memcmp respectively. ii. Numbers in bytes column show total bytes processed across all 4 API calls ------------------------------------------------------------------------------------------------------------------------------------------ -DTO Run Time: 22826 ms +``` +DTO Run Time: 8 ms ******** Number of Memory Operations ******** - <*************** stdc calls ***************> <*************** dsa (success) ***************> <*************** dsa (failed) ***************> <***** failure reason *****> -Byte Range -- set cpy mov cmp bytes set cpy mov cmp bytes set cpy mov cmp bytes Retries PFs Others - 0-4095 -- 24616626 12319310 7676731 244777209 4531255821 0 0 0 0 0 0 0 0 0 0 0 0 0 - 4096-8191 -- 3 290 14 0 1842507 0 0 0 0 0 0 0 0 0 0 0 0 0 - 8192-12287 -- 0 2 6 0 45579 64 176 2 0 2545968 0 2 6 0 25276 0 8 0 - 12288-16383 -- 1 1 0 0 20756 1 137 0 0 1965343 1 1 0 0 11172 0 2 0 - 16384-20479 -- 0 1 3 0 41974 0 126 2 0 2328434 0 1 3 0 24337 0 4 0 - 32768-36863 -- 0 0 0 0 0 0 67 2 0 2378165 0 0 0 0 0 0 0 0 - 53248-57343 -- 0 0 0 0 0 0 46 0 0 2545890 0 0 0 0 0 0 0 0 - 98304-102399 -- 0 0 0 0 0 0 13 0 0 1309627 0 0 0 0 0 0 0 0 - 106496-110591 -- 0 1 0 0 31637 0 8 0 0 869880 0 1 0 0 77586 0 1 0 - 118784-122879 -- 0 0 0 0 0 0 9 0 0 1083519 0 0 0 0 0 0 0 0 - 131072-135167 -- 0 0 2 0 175638 0 5 0 0 665475 0 0 2 0 86506 0 2 0 - 143360-147455 -- 0 0 0 0 0 0 1 0 0 143399 0 0 0 0 0 0 0 0 - 147456-151551 -- 0 0 0 0 0 0 1 0 0 148647 0 0 0 0 0 0 0 0 - 159744-163839 -- 0 0 0 0 0 0 6 0 0 970186 0 0 0 0 0 0 0 0 - 212992-217087 -- 0 0 11 0 1580248 0 4 2 0 1292252 0 0 11 0 779384 0 11 0 - 278528-282623 -- 0 0 0 0 0 0 1 0 0 279719 0 0 0 0 0 0 0 0 - 466944-471039 -- 0 0 0 0 0 0 1 0 0 467783 0 0 0 0 0 0 0 0 - 528384-532479 -- 0 0 0 0 0 0 1 0 0 529927 0 0 0 0 0 0 0 0 - 720896-724991 -- 0 0 0 0 0 1 0 0 0 720896 0 0 0 0 0 0 0 0 - 999424-1003519 -- 0 0 0 0 0 0 1 0 0 1002439 0 0 0 0 0 0 0 0 - >=2093056 -- 0 1 0 0 1975911 0 0 0 0 0 0 1 0 0 973209 0 1 0 + <*************** stdc calls ***************> <*************** dsa (success) ***************> <*************** dsa (failed) ***************> <***** failure reason *****> +Byte Range -- set cpy mov cmp bytes set cpy mov cmp bytes set cpy mov cmp bytes Retry PFs Others + 65536-81919 -- 0 0 0 0 0 200 0 0 0 13926400 0 0 0 0 0 0 0 0 + 98304-114687 -- 0 0 200 0 19662200 0 0 0 0 0 0 0 0 0 0 0 0 0 + 131072-163839 -- 0 0 0 0 0 0 200 0 0 26239000 0 0 0 0 0 0 0 0 + 262144-327679 -- 0 0 0 0 0 0 0 0 200 52441600 0 0 0 0 0 0 0 0 ******** Average Memory Operation Latency (us) ******** - <******** stdc calls ********> <******** dsa (success) ********> <******** dsa (failed) ********> -Byte Range -- set cpy mov cmp set cpy mov cmp set cpy mov cmp - 0-4095 -- 0.01 0.02 0.01 0.04 0 0 0 0 0 0 0 0 - 4096-8191 -- 0.07 0.42 0.47 0 0 0 0 0 0 0 0 0 - 8192-12287 -- 0 1.29 0.47 0 0.84 1.19 1.82 0 0 56.18 2.54 0 - 12288-16383 -- 2.90 3.34 0 0 7.49 1.26 0 0 2.07 1.93 0 0 - 16384-20479 -- 0 0.27 3.24 0 0 1.37 1.86 0 0 84.92 2.47 0 - 32768-36863 -- 0 0 0 0 0 1.87 2.91 0 0 0 0 0 - 53248-57343 -- 0 0 0 0 0 2.45 0 0 0 0 0 0 - 98304-102399 -- 0 0 0 0 0 3.76 0 0 0 0 0 0 - 106496-110591 -- 0 229.18 0 0 0 4.19 0 0 0 3.65 0 0 - 118784-122879 -- 0 0 0 0 0 5.18 0 0 0 0 0 0 - 131072-135167 -- 0 0 25.00 0 0 5.92 0 0 0 0 14.39 0 - 143360-147455 -- 0 0 0 0 0 5.11 0 0 0 0 0 0 - 147456-151551 -- 0 0 0 0 0 6.16 0 0 0 0 0 0 - 159744-163839 -- 0 0 0 0 0 7.65 0 0 0 0 0 0 - 212992-217087 -- 0 0 36.10 0 0 8.15 6.59 0 0 0 18.07 0 - 278528-282623 -- 0 0 0 0 0 15.47 0 0 0 0 0 0 - 466944-471039 -- 0 0 0 0 0 15.91 0 0 0 0 0 0 - 528384-532479 -- 0 0 0 0 0 18.65 0 0 0 0 0 0 - 720896-724991 -- 0 0 0 0 17.71 0 0 0 0 0 0 0 - 999424-1003519 -- 0 0 0 0 0 28.42 0 0 0 0 0 0 - >=2093056 -- 0 422.15 0 0 0 0 0 0 0 272.87 0 0 + <******** stdc calls ********> <******** dsa (success) ********> <******** dsa (failed) ********> +Byte Range -- set cpy mov cmp set cpy mov cmp set cpy mov cmp + 65536-81919 -- 0 0 0 0 8.06 0 0 0 0 0 0 0 + 98304-114687 -- 0 0 4.18 0 0 0 0 0 0 0 0 0 + 131072-163839 -- 0 0 0 0 0 17.02 0 0 0 0 0 0 + 262144-327679 -- 0 0 0 0 0 0 0 34.50 0 0 0 0 +``` + +## Statistics + +With `DTO_COLLECT_STATS=1`, DTO times and counts every memset, memcpy, +memmove and memcmp it intercepts, by operation, by path (CPU, DSA, or DSA +failed and finished on the CPU) and by size, and writes a summary when the +process exits (or on Ctrl-C). Each thread counts on its own, so collection +costs about 40 ns per call however many threads there are, and the counts +are summed for output. + +The size histogram has a bucket for calls under 64 bytes, then four buckets +per power of two up to 4 GiB (65536-81919, 81920-98303, ...), and one for +anything larger. A call that DSA completed only partly (for example on a +page fault) is counted once as `dsa (failed)` and once as a CPU call for the +remainder. + +With `DTO_STATS_INTERVAL_MS=N`, a background thread also writes, every N ms, +what happened since the previous record, so activity can be followed over +time. The interval records add up to the summary's totals. + +With `DTO_STATS_FORMAT=json` the output is JSON Lines, one record per line: + +``` +{"type":"interval","pid":183654,"t_ms":14.333,"dt_ms":14.333,"ops":{"memcpy":{"dsa":{"calls":8,"bytes":1049560,"ns":96273}},"memmove":{"cpu":{"calls":8,"bytes":786488,"ns":31601}}},"failures":{"retry":0,"page_fault":0,"other":0}} +{"type":"summary","pid":183654,"program":"prog","run_ms":23.395, + "config":{"stdc_only":0,"wait_method":"busypoll","min_bytes":65536,"cpu_size_fraction":0.00,"auto_adjust":1,"numa_aware":0,"wqs":1}, + "totals":{"memcpy":{"dsa":{"calls":12,"bytes":1574340,"ns":138591}},...}, + "failures":{"retry":0,"page_fault":0,"other":0}, + "histogram":[{"min":131072,"max":163839,"ops":{"memcpy":{"dsa":{"calls":12,"bytes":1574340,"ns":138591}}},"failures":{...}},...]} +``` + +(The summary is a single line; it is wrapped here.) `ops` maps `memset`, +`memcpy`, `memmove` and `memcmp` to the paths they took, `cpu`, `dsa` and +`dsa_failed`, each with the number of calls, bytes and total nanoseconds; +operations and paths with no calls are left out. `t_ms` and `run_ms` count +from DTO's start, and `config` holds the settings at exit, including the +auto-tuned `min_bytes` and `cpu_size_fraction`. + +Every process that loads DTO with stats enabled writes its own records, +tagged with its `pid`. They share `DTO_STATS_FILE` (records are appended +whole) unless the path contains `%p`, which gives each process its own file. ## Usage Notes diff --git a/TESTING.md b/TESTING.md index db6b543..ab8a41f 100644 --- a/TESTING.md +++ b/TESTING.md @@ -95,6 +95,19 @@ ctest -R dtoctl tests/test_dtoctl.sh build/dtoctl build/libdto.so ``` +## Stats Tests + +`stats` runs `dto-test-stats-workload`, which makes a known number of calls +of known sizes, with `DTO_COLLECT_STATS` set, and checks the output: totals +and histogram buckets, interval records adding up to the totals, calls from +200 short-lived threads all counted without their stats memory piling up, +separate records for a forked child, `%p` file names, and the text format. +It needs Python 3 and no DSA hardware. + +```bash +ctest -R stats +``` + ## Performance Tests The performance test (`perf`) detects regressions in libdto by diff --git a/dto.c b/dto.c index dd76ef4..51899d0 100644 --- a/dto.c +++ b/dto.c @@ -4,6 +4,7 @@ * SPDX-License-Identifier: MIT ******************************************************************************/ +#include #include #include #include @@ -187,9 +188,13 @@ static const char * const memop_names[] = { [MEMCMP] = "cmp" }; -// memory stats -#define HIST_BUCKET_SIZE 4096 -#define HIST_NO_BUCKETS 512 +// memory stats: a histogram by call size, with bucket 0 for calls under +// 64 bytes, then four buckets per power of two up to 4 GiB, and a last +// bucket for anything larger +#define HIST_MIN_SHIFT 6 +#define HIST_MAX_SHIFT 32 +#define HIST_SUBS 4 +#define HIST_NO_BUCKETS (2 + (HIST_MAX_SHIFT - HIST_MIN_SHIFT) * HIST_SUBS) enum stat_group { STDC_CALL = 0x0, DSA_CALL_SUCCESS, @@ -268,10 +273,57 @@ static struct timespec dto_start_time; } \ } while (0) \ -static atomic_int op_counter[HIST_NO_BUCKETS][MAX_STAT_GROUP][MAX_MEMOP]; -static atomic_ullong bytes_counter[HIST_NO_BUCKETS][MAX_STAT_GROUP]; -static atomic_ullong lat_counter[HIST_NO_BUCKETS][MAX_STAT_GROUP][MAX_MEMOP]; -static atomic_int fail_counter[HIST_NO_BUCKETS][MAX_FAILURES]; +/* the groups that have per-operation counters */ +#define STAT_GROUPS DSA_FAIL_CODES + +struct dto_op_stats { + uint64_t calls; + uint64_t bytes; + uint64_t ns; +}; + +struct dto_bucket_stats { + struct dto_op_stats op[STAT_GROUPS][MAX_MEMOP]; + uint64_t fails[MAX_FAILURES]; +}; + +struct dto_stats { + struct dto_op_stats total[STAT_GROUPS][MAX_MEMOP]; + uint64_t fails[MAX_FAILURES]; + struct dto_bucket_stats hist[HIST_NO_BUCKETS]; +}; + +/* one per counting thread (see update_stats) */ +struct dto_thread_stats { + struct dto_thread_stats *prev, *next; + struct dto_stats s; +}; + +static struct dto_thread_stats *stats_threads; /* live threads' blocks */ +static struct dto_stats stats_retired; /* exited threads, summed */ +static pthread_mutex_t stats_lock = PTHREAD_MUTEX_INITIALIZER; +static pthread_key_t stats_key; +static bool stats_key_created; +static __thread struct dto_thread_stats *thr_stats; +static __thread bool thr_stats_off; /* don't count this thread's calls */ + +/* output: one formatter at a time (sampler, exit, SIGINT) */ +static pthread_mutex_t stats_out_lock = PTHREAD_MUTEX_INITIALIZER; +static struct dto_stats stats_scratch; +enum { STATS_FORMAT_TEXT, STATS_FORMAT_JSON }; +static int stats_format; + +static bool stats_file_per_pid; /* DTO_STATS_FILE has a %p */ + +/* DTO_STATS_INTERVAL_MS */ +static unsigned long stats_interval_ms; +static atomic_uchar sampler_started; +static struct dto_stats sampler_prev; /* totals at the previous record */ +static uint64_t sampler_prev_ns; + +static void print_stats(void); +static void print_stats_from_signal(void); +static void stats_fork_child(void); #endif /* call initialize/cleanup functions when library is loaded/unloaded */ @@ -292,29 +344,7 @@ static enum { static unsigned int log_level = LOG_LEVEL_FATAL; -static void dto_stats_log(const char *fmt, ...) -{ - char buf[512]; - va_list args; - - if (stats_fd == -1 && LOG_LEVEL_TRACE > log_level) - return; - - va_start(args, fmt); - if (stats_fd != -1) { - if (vsnprintf(buf, sizeof(buf), fmt, args) > 0) - write(stats_fd, buf, strlen(buf)); - } else if (log_fd == -1) { - vprintf(fmt, args); - } else { - vsnprintf(buf, sizeof(buf), fmt, args); - write(log_fd, buf, strlen(buf)); - } - va_end(args); -} - #ifdef DTO_STATS_SUPPORT -static void print_stats(void); static struct sigaction previous_sigint_action; static volatile sig_atomic_t sigint_handler_installed; static int previous_sigint_action_valid; @@ -338,8 +368,7 @@ static void restore_sigint_handler(void) static void dto_handle_sigint(int signo) { - if (collect_stats) - print_stats(); + print_stats_from_signal(); restore_sigint_handler(); @@ -418,20 +447,7 @@ static void dto_log(int req_log_level, const char *fmt, ...) static void child (void) { #ifdef DTO_STATS_SUPPORT - int i, j, k; - - /* Reset the counters */ - for (i = 0; i < HIST_NO_BUCKETS; i++) { - for (j = 0; j < MAX_STAT_GROUP; j++) { - for (k = 0; k < MAX_MEMOP; k++) { - op_counter[i][j][k] = 0; - lat_counter[i][j][k] = 0; - } - bytes_counter[i][j] = 0; - } - for (j = 0; j < MAX_FAILURES; j++) - fail_counter[i][j] = 0; - } + stats_fork_child(); #endif dto_initializing = 0; dto_initialized = 0; @@ -672,116 +688,662 @@ static __always_inline int dsa_execute(struct dto_wq *wq, } #ifdef DTO_STATS_SUPPORT +/* + * Statistics (DTO_COLLECT_STATS). + * + * Each thread counts into its own block, so an update is a few plain loads + * and stores with no shared cache line; the blocks are summed when stats are + * printed. A block is mmap'd on the thread's first update (only the pages it + * touches are ever allocated), registered in a list, and on thread exit + * folded into stats_retired and unmapped. The totals sit at the start of the + * block, so the interval sampler reads one page per thread. + */ + +/* Histogram bucket of a call of n bytes. */ +static __always_inline int stats_bucket(size_t n) +{ + int e; + + if (n < (1UL << HIST_MIN_SHIFT)) + return 0; + e = 63 - __builtin_clzl(n); + if (e >= HIST_MAX_SHIFT) + return HIST_NO_BUCKETS - 1; + return 1 + (e - HIST_MIN_SHIFT) * HIST_SUBS + + (int)((n >> (e - 2)) & (HIST_SUBS - 1)); +} + +/* Smallest size counted in bucket b. */ +static uint64_t stats_bucket_min(int b) +{ + int e; + + if (b == 0) + return 0; + if (b == HIST_NO_BUCKETS - 1) + return 1ULL << HIST_MAX_SHIFT; + e = HIST_MIN_SHIFT + (b - 1) / HIST_SUBS; + return (uint64_t)(HIST_SUBS + (b - 1) % HIST_SUBS) << (e - 2); +} + +/* Relaxed load-add-store: the owning thread is the only writer, and + * readers summing blocks only need each counter to be read whole. */ +#define STAT_ADD(field, v) \ + __atomic_store_n(&(field), __atomic_load_n(&(field), __ATOMIC_RELAXED) + (v), \ + __ATOMIC_RELAXED) +#define STAT_GET(field) __atomic_load_n(&(field), __ATOMIC_RELAXED) + +static __always_inline void stats_add_op(struct dto_op_stats *s, uint64_t bytes, + uint64_t ns) +{ + STAT_ADD(s->calls, 1); + STAT_ADD(s->bytes, bytes); + STAT_ADD(s->ns, ns); +} + +/* dst += src for the totals, or for everything with hist */ +static void stats_sum(struct dto_stats *dst, struct dto_stats *src, bool hist) +{ + for (int g = 0; g < STAT_GROUPS; g++) + for (int o = 0; o < MAX_MEMOP; o++) { + dst->total[g][o].calls += STAT_GET(src->total[g][o].calls); + dst->total[g][o].bytes += STAT_GET(src->total[g][o].bytes); + dst->total[g][o].ns += STAT_GET(src->total[g][o].ns); + } + for (int f = 0; f < MAX_FAILURES; f++) + dst->fails[f] += STAT_GET(src->fails[f]); + if (!hist) + return; + for (int b = 0; b < HIST_NO_BUCKETS; b++) { + struct dto_bucket_stats *d = &dst->hist[b], *s = &src->hist[b]; + + for (int g = 0; g < STAT_GROUPS; g++) + for (int o = 0; o < MAX_MEMOP; o++) { + d->op[g][o].calls += STAT_GET(s->op[g][o].calls); + d->op[g][o].bytes += STAT_GET(s->op[g][o].bytes); + d->op[g][o].ns += STAT_GET(s->op[g][o].ns); + } + for (int f = 0; f < MAX_FAILURES; f++) + d->fails[f] += STAT_GET(s->fails[f]); + } +} + +/* Sum of every thread's stats, live and exited, into *out (zeroed first). + * The caller holds stats_lock. */ +static void stats_collect(struct dto_stats *out, bool hist) +{ + size_t len = hist ? sizeof(*out) : offsetof(struct dto_stats, hist); + + orig_memset(out, 0, len); + stats_sum(out, &stats_retired, hist); + for (struct dto_thread_stats *t = stats_threads; t; t = t->next) + stats_sum(out, &t->s, hist); +} + +static void stats_snapshot(struct dto_stats *out, bool hist) +{ + pthread_mutex_lock(&stats_lock); + stats_collect(out, hist); + pthread_mutex_unlock(&stats_lock); +} + +static void stats_start_sampler(void); + +static void stats_thread_exit(void *arg) +{ + struct dto_thread_stats *ts = arg; + + /* memcpy and friends may still run during the rest of thread exit */ + thr_stats_off = true; + thr_stats = NULL; + pthread_mutex_lock(&stats_lock); + stats_sum(&stats_retired, &ts->s, true); + if (ts->prev) + ts->prev->next = ts->next; + else + stats_threads = ts->next; + if (ts->next) + ts->next->prev = ts->prev; + pthread_mutex_unlock(&stats_lock); + munmap(ts, sizeof(*ts)); +} + +/* First update on this thread: give it a block. NULL leaves the thread + * uncounted (thr_stats_off stays set, so it is not retried every call). */ +static struct dto_thread_stats *stats_thread_register(void) +{ + struct dto_thread_stats *ts; + + /* mmap, the registry and pthread_setspecific must not count themselves */ + thr_stats_off = true; + ts = mmap(NULL, sizeof(*ts), PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (ts == MAP_FAILED) + return NULL; + + pthread_mutex_lock(&stats_lock); + ts->prev = NULL; + ts->next = stats_threads; + if (stats_threads) + stats_threads->prev = ts; + stats_threads = ts; + pthread_mutex_unlock(&stats_lock); + pthread_setspecific(stats_key, ts); + thr_stats = ts; + + if (stats_interval_ms && !atomic_load_explicit(&sampler_started, + memory_order_acquire)) + stats_start_sampler(); + thr_stats_off = false; + return ts; +} + static void update_stats(int op, size_t n, bool overlapping, size_t bytes_completed, uint64_t elapsed_ns, int group, int error_code) { + struct dto_thread_stats *ts; + struct dto_bucket_stats *b; + // dto_memcpymove didn't actually submit the request to DSA, so there is nothing to log. This will be captured by a second call if (op == MEMMOVE && overlapping && dto_overlapping_memmove_action == OVERLAPPING_CPU && group == DSA_CALL_SUCCESS) { return; } + if (unlikely(thr_stats_off)) + return; + ts = thr_stats; + if (unlikely(ts == NULL)) { + ts = stats_thread_register(); + if (ts == NULL) + return; + } - int bucket = (n / HIST_BUCKET_SIZE); + b = &ts->s.hist[stats_bucket(n)]; + stats_add_op(&ts->s.total[group][op], bytes_completed, elapsed_ns); + stats_add_op(&b->op[group][op], bytes_completed, elapsed_ns); + if (group == DSA_CALL_FAILED) { + STAT_ADD(ts->s.fails[error_code], 1); + STAT_ADD(b->fails[error_code], 1); + } +} - if (bucket >= HIST_NO_BUCKETS) /* last bucket includes remaining sizes */ - bucket = HIST_NO_BUCKETS-1; - ++op_counter[bucket][group][op]; - bytes_counter[bucket][group] += bytes_completed; - lat_counter[bucket][group][op] += elapsed_ns; - if (group == DSA_CALL_FAILED) - ++fail_counter[bucket][error_code]; +/* ---- Output ---- + * + * Records are formatted into stats_out and written with as few write()s as + * possible: to DTO_STATS_FILE (opened O_APPEND, so records from several + * processes sharing it stay whole), else DTO_LOG_FILE, else stdout. */ + +static char stats_out[1 << 16]; +static size_t stats_out_len; + +static void stats_flush(void) +{ + int fd = stats_fd != -1 ? stats_fd : log_fd != -1 ? log_fd : STDOUT_FILENO; + size_t off = 0; + + while (off < stats_out_len) { + ssize_t r = write(fd, stats_out + off, stats_out_len - off); + if (r <= 0) + break; + off += r; + } + stats_out_len = 0; } -static void print_stats(void) +__attribute__((format(printf, 1, 2))) +static void stats_printf(const char *fmt, ...) { - struct timespec dto_end_time; + va_list args; + int len; - if (likely(!collect_stats)) + va_start(args, fmt); + len = vsnprintf(stats_out + stats_out_len, sizeof(stats_out) - stats_out_len, + fmt, args); + va_end(args); + if (len < 0) return; + if ((size_t)len >= sizeof(stats_out) - stats_out_len) { + /* no room: write out what is there and format again */ + stats_flush(); + va_start(args, fmt); + len = vsnprintf(stats_out, sizeof(stats_out), fmt, args); + va_end(args); + if (len < 0) + return; + if ((size_t)len >= sizeof(stats_out)) + len = sizeof(stats_out) - 1; + } + stats_out_len += len; +} + +/* DTO_STATS_FILE, with each %p replaced by the process ID. It is opened + * O_APPEND, not truncated: processes started from this one (fork, exec) + * inherit the setting, and with %p each gets a file of its own, without it + * they append whole records to a shared one. */ +static void stats_open_file(const char *name) +{ + char path[PATH_MAX]; + size_t j = 0; + struct stat st; + + for (const char *c = name; *c && j < sizeof(path) - 1; c++) { + if (c[0] == '%' && c[1] == 'p') { + stats_file_per_pid = true; + int n = snprintf(path + j, sizeof(path) - j, "%d", getpid()); + + if (n < 0 || (size_t)n >= sizeof(path) - j) + break; + j += n; + c++; + } else { + path[j++] = *c; + } + } + path[j] = '\0'; + + if (lstat(path, &st) == 0 && (st.st_mode & S_IFMT) != S_IFREG) { + LOG_ERROR("DTO_STATS_FILE '%s' must reference a regular file\n", path); + return; + } + stats_fd = open(path, O_WRONLY | O_CREAT | O_APPEND | O_CLOEXEC, 0600); + if (stats_fd == -1) + LOG_ERROR("Failed to open DTO_STATS_FILE '%s': %s\n", path, strerror(errno)); +} + +static const char * const memop_json[] = { + [MEMSET] = "memset", + [MEMCOPY] = "memcpy", + [MEMMOVE] = "memmove", + [MEMCMP] = "memcmp", +}; - clock_gettime(CLOCK_BOOTTIME, &dto_end_time); +static const char * const group_json[] = { + [STDC_CALL] = "cpu", + [DSA_CALL_SUCCESS] = "dsa", + [DSA_CALL_FAILED] = "dsa_failed", +}; - dto_stats_log("DTO Run Time: %ld ms\n", TS_NS(dto_start_time, dto_end_time)/1000000); +static const char * const failure_json[] = { + [RETRY] = "retry", + [PAGE_FAULT] = "page_fault", + [FAIL_OTHERS] = "other", +}; + +static uint64_t stats_now_ns(void) +{ + struct timespec now; + + clock_gettime(CLOCK_BOOTTIME, &now); + return TS_NS(dto_start_time, now); +} + +/* {"memcpy":{"dsa":{"calls":..,"bytes":..,"ns":..},..},..} without zeroes */ +static void stats_json_ops(struct dto_op_stats op[STAT_GROUPS][MAX_MEMOP]) +{ + bool first_op = true; + + stats_printf("{"); + for (int o = 0; o < MAX_MEMOP; o++) { + bool first_group = true; + + for (int g = 0; g < STAT_GROUPS; g++) { + if (!op[g][o].calls) + continue; + if (first_group) + stats_printf("%s\"%s\":{", first_op ? "" : ",", memop_json[o]); + stats_printf("%s\"%s\":{\"calls\":%lu,\"bytes\":%lu,\"ns\":%lu}", + first_group ? "" : ",", group_json[g], op[g][o].calls, + op[g][o].bytes, op[g][o].ns); + first_group = false; + first_op = false; + } + if (!first_group) + stats_printf("}"); + } + stats_printf("}"); +} + +static void stats_json_fails(uint64_t fails[MAX_FAILURES]) +{ + stats_printf("{"); + for (int f = 1; f < MAX_FAILURES; f++) + stats_printf("%s\"%s\":%lu", f == 1 ? "" : ",", failure_json[f], fails[f]); + stats_printf("}"); +} + +/* Text: "memcpy dsa 120 calls 7864320 B 421.3 us, cpu ..." */ +static void stats_text_ops(struct dto_op_stats op[STAT_GROUPS][MAX_MEMOP]) +{ + bool any = false; + + for (int o = 0; o < MAX_MEMOP; o++) + for (int g = 0; g < STAT_GROUPS; g++) { + if (!op[g][o].calls) + continue; + stats_printf("%s%s %s %lu calls %lu B %.1f us", any ? ", " : "", + memop_json[o], group_json[g], op[g][o].calls, + op[g][o].bytes, op[g][o].ns / 1000.0); + any = true; + } + if (!any) + stats_printf("no calls"); +} + +/* One interval record: the change in the totals since the previous one. + * The caller holds stats_out_lock. */ +static void stats_emit_interval(void) +{ + struct dto_stats *now = &stats_scratch; /* totals part only */ + struct dto_op_stats delta[STAT_GROUPS][MAX_MEMOP]; + uint64_t fails[MAX_FAILURES]; + uint64_t t = stats_now_ns(); + + stats_snapshot(now, false); + for (int g = 0; g < STAT_GROUPS; g++) + for (int o = 0; o < MAX_MEMOP; o++) { + delta[g][o].calls = now->total[g][o].calls - sampler_prev.total[g][o].calls; + delta[g][o].bytes = now->total[g][o].bytes - sampler_prev.total[g][o].bytes; + delta[g][o].ns = now->total[g][o].ns - sampler_prev.total[g][o].ns; + } + for (int f = 0; f < MAX_FAILURES; f++) + fails[f] = now->fails[f] - sampler_prev.fails[f]; + orig_memcpy(&sampler_prev, now, offsetof(struct dto_stats, hist)); + + if (stats_format == STATS_FORMAT_JSON) { + stats_printf("{\"type\":\"interval\",\"pid\":%d,\"t_ms\":%.3f,\"dt_ms\":%.3f,\"ops\":", + getpid(), t / 1e6, (t - sampler_prev_ns) / 1e6); + stats_json_ops(delta); + stats_printf(",\"failures\":"); + stats_json_fails(fails); + stats_printf("}\n"); + } else { + stats_printf("DTO stats t=%.3f ms (+%.3f ms): ", t / 1e6, (t - sampler_prev_ns) / 1e6); + stats_text_ops(delta); + if (fails[RETRY] || fails[PAGE_FAULT] || fails[FAIL_OTHERS]) + stats_printf("; failures retry %lu page_fault %lu other %lu", + fails[RETRY], fails[PAGE_FAULT], fails[FAIL_OTHERS]); + stats_printf("\n"); + } + sampler_prev_ns = t; + stats_flush(); +} + +static bool stats_bucket_empty(struct dto_bucket_stats *b) +{ + for (int g = 0; g < STAT_GROUPS; g++) + for (int o = 0; o < MAX_MEMOP; o++) + if (b->op[g][o].calls) + return false; + return true; +} + +static void stats_print_json(struct dto_stats *s, uint64_t run_ns) +{ + char prog[64]; + bool first = true; + size_t j = 0; + + /* the program name as a JSON string: drop quotes, backslashes and + * control characters rather than escape them */ + for (const char *c = __progname; *c && j < sizeof(prog) - 1; c++) + if (*c != '"' && *c != '\\' && (unsigned char)*c >= 0x20) + prog[j++] = *c; + prog[j] = '\0'; + stats_printf("{\"type\":\"summary\",\"pid\":%d,\"program\":\"%s\",\"run_ms\":%.3f,", + getpid(), prog, run_ns / 1e6); + stats_printf("\"config\":{\"stdc_only\":%d,\"wait_method\":\"%s\",\"min_bytes\":%lu," + "\"cpu_size_fraction\":%.2f,\"auto_adjust\":%d,\"numa_aware\":%d,\"wqs\":%d},", + use_std_lib_calls, wait_names[wait_method], dsa_min_size, + cpu_size_fraction / 100.0, auto_adjust_knobs, (int)is_numa_aware, num_wqs); + stats_printf("\"totals\":"); + stats_json_ops(s->total); + stats_printf(",\"failures\":"); + stats_json_fails(s->fails); + stats_printf(",\"histogram\":["); + for (int b = 0; b < HIST_NO_BUCKETS; b++) { + if (stats_bucket_empty(&s->hist[b])) + continue; + stats_printf("%s{\"min\":%lu,", first ? "" : ",", stats_bucket_min(b)); + if (b < HIST_NO_BUCKETS - 1) + stats_printf("\"max\":%lu,", stats_bucket_min(b + 1) - 1); + stats_printf("\"ops\":"); + stats_json_ops(s->hist[b].op); + stats_printf(",\"failures\":"); + stats_json_fails(s->hist[b].fails); + stats_printf("}"); + first = false; + } + stats_printf("]}\n"); +} + +static void stats_print_text(struct dto_stats *s, uint64_t run_ns) +{ + stats_printf("DTO Run Time: %lu ms\n", run_ns / 1000000); - // display stats for (int t = 0; t < 2; ++t) { if (t == 0) - dto_stats_log("\n******** Number of Memory Operations ********\n"); + stats_printf("\n******** Number of Memory Operations ********\n"); else - dto_stats_log("\n******** Average Memory Operation Latency (us) ********\n"); + stats_printf("\n******** Average Memory Operation Latency (us) ********\n"); - dto_stats_log("%17s ", ""); + stats_printf("%23s ", ""); for (int g = 0; g < MAX_STAT_GROUP; ++g) { if (t == 0) { if (g == DSA_FAIL_CODES) - dto_stats_log("<***** %-13s *****> ", stat_group_names[g]); + stats_printf("<***** %-13s *****> ", stat_group_names[g]); else - dto_stats_log("<*************** %-13s ***************> ", stat_group_names[g]); - } else { - if (g != DSA_FAIL_CODES) - dto_stats_log("<******** %-13s ********> ", stat_group_names[g]); - + stats_printf("<*************** %-13s ***************> ", stat_group_names[g]); + } else if (g != DSA_FAIL_CODES) { + stats_printf("<******** %-13s ********> ", stat_group_names[g]); } } - dto_stats_log("\n"); - - dto_stats_log("%-17s -- ", "Byte Range"); - for (int g = 0; g < MAX_STAT_GROUP - 1; ++g) { + stats_printf("\n%-23s -- ", "Byte Range"); + for (int g = 0; g < STAT_GROUPS; ++g) { for (int o = 0; o < MAX_MEMOP; ++o) - dto_stats_log("%-8s ", memop_names[o]); + stats_printf("%-8s ", memop_names[o]); if (t == 0) - dto_stats_log("%-12s ", "bytes"); + stats_printf("%-12s ", "bytes"); } if (t == 0) for (int o = 1; o < MAX_FAILURES; ++o) - dto_stats_log("%-6s ", failure_names[o]); - dto_stats_log("\n"); + stats_printf("%-6s ", failure_names[o]); + stats_printf("\n"); for (int b = 0; b < HIST_NO_BUCKETS; ++b) { - bool empty = true; + struct dto_bucket_stats *h = &s->hist[b]; - for (int g = 0; g < MAX_STAT_GROUP; ++g) { - for (int o = 0; o < MAX_MEMOP; ++o) { - if (op_counter[b][g][o] != 0) { - empty = false; - break; - } - } - if (!empty) - break; - } - if (empty) + if (stats_bucket_empty(h)) continue; - - if (b < (HIST_NO_BUCKETS-1)) - dto_stats_log("% 8d-%-8d -- ", b*4096, ((b+1)*4096)-1); + if (b < HIST_NO_BUCKETS - 1) + stats_printf("%11lu-%-11lu -- ", stats_bucket_min(b), + stats_bucket_min(b + 1) - 1); else - dto_stats_log(" >=%-12d -- ", b*4096); + stats_printf(" >=%-18lu -- ", stats_bucket_min(b)); + + for (int g = 0; g < STAT_GROUPS; ++g) { + uint64_t bytes = 0; - for (int g = 0; g < MAX_STAT_GROUP - 1; ++g) { for (int o = 0; o < MAX_MEMOP; ++o) { - if (t == 0) { - dto_stats_log("%-8d ", op_counter[b][g][o]); - continue; - } - if (op_counter[b][g][o] != 0) { - double avg_us = ((double) lat_counter[b][g][o])/(((double) op_counter[b][g][o]) * 1000.0); + struct dto_op_stats *op = &h->op[g][o]; - dto_stats_log("%-8.2f ", avg_us); - } else { - dto_stats_log("%-8d ", 0); - } + bytes += op->bytes; + if (t == 0) + stats_printf("%-8lu ", op->calls); + else if (op->calls) + stats_printf("%-8.2f ", op->ns / (op->calls * 1000.0)); + else + stats_printf("%-8d ", 0); } if (t == 0) - dto_stats_log("%-12lld ", bytes_counter[b][g]); + stats_printf("%-12lu ", bytes); } if (t == 0) for (int o = 1; o < MAX_FAILURES; ++o) - dto_stats_log("%-6d ", fail_counter[b][o]); - dto_stats_log("\n"); + stats_printf("%-6lu ", h->fails[o]); + stats_printf("\n"); } } } + +/* The end-of-run summary of *s. The caller holds stats_out_lock. */ +static void stats_print_summary(struct dto_stats *s) +{ + if (stats_format == STATS_FORMAT_JSON) + stats_print_json(s, stats_now_ns()); + else + stats_print_text(s, stats_now_ns()); + stats_flush(); +} + +static void print_stats(void) +{ + bool off = thr_stats_off; + + if (likely(!collect_stats)) + return; + thr_stats_off = true; + pthread_mutex_lock(&stats_out_lock); + stats_snapshot(&stats_scratch, true); + stats_print_summary(&stats_scratch); + pthread_mutex_unlock(&stats_out_lock); + thr_stats_off = off; +} + +/* From the SIGINT handler: the interrupted thread may hold a stats lock, so + * print only if both can be taken right away. */ +static void print_stats_from_signal(void) +{ + bool off = thr_stats_off; + + if (!collect_stats) + return; + thr_stats_off = true; + if (pthread_mutex_trylock(&stats_out_lock) == 0) { + if (pthread_mutex_trylock(&stats_lock) == 0) { + stats_collect(&stats_scratch, true); + pthread_mutex_unlock(&stats_lock); + stats_print_summary(&stats_scratch); + } + pthread_mutex_unlock(&stats_out_lock); + } + thr_stats_off = off; +} + +/* ---- Interval sampler (DTO_STATS_INTERVAL_MS) ---- */ + +static pthread_t sampler_thread; +static pthread_mutex_t sampler_lock = PTHREAD_MUTEX_INITIALIZER; +static pthread_cond_t sampler_cv; +static bool sampler_stop; + +static void *stats_sampler(void *arg) +{ + struct timespec next; + + thr_stats_off = true; /* its own formatting is not the program's */ + clock_gettime(CLOCK_MONOTONIC, &next); + pthread_mutex_lock(&sampler_lock); + while (!sampler_stop) { + next.tv_nsec += (long)(stats_interval_ms % 1000) * 1000000; + next.tv_sec += stats_interval_ms / 1000 + next.tv_nsec / 1000000000; + next.tv_nsec %= 1000000000; + while (!sampler_stop && + pthread_cond_timedwait(&sampler_cv, &sampler_lock, &next) != ETIMEDOUT) + ; + if (sampler_stop) + break; + pthread_mutex_unlock(&sampler_lock); + pthread_mutex_lock(&stats_out_lock); + stats_emit_interval(); + pthread_mutex_unlock(&stats_out_lock); + pthread_mutex_lock(&sampler_lock); + } + pthread_mutex_unlock(&sampler_lock); + return NULL; +} + +/* Started by the first counted call (not from the constructor or the fork + * handler, where creating a thread is not safe), at most once per process. */ +static void stats_start_sampler(void) +{ + unsigned char expected = 0; + pthread_condattr_t attr; + sigset_t all, old; + + if (!atomic_compare_exchange_strong(&sampler_started, &expected, 1)) + return; + pthread_condattr_init(&attr); + pthread_condattr_setclock(&attr, CLOCK_MONOTONIC); + pthread_cond_init(&sampler_cv, &attr); + pthread_condattr_destroy(&attr); + sampler_stop = false; + /* the sampler must not take the program's signals */ + sigfillset(&all); + pthread_sigmask(SIG_SETMASK, &all, &old); + if (pthread_create(&sampler_thread, NULL, stats_sampler, NULL)) + atomic_store(&sampler_started, 0); + pthread_sigmask(SIG_SETMASK, &old, NULL); +} + +static void stats_stop_sampler(void) +{ + if (!atomic_load(&sampler_started)) + return; + pthread_mutex_lock(&sampler_lock); + sampler_stop = true; + pthread_cond_signal(&sampler_cv); + pthread_mutex_unlock(&sampler_lock); + pthread_join(sampler_thread, NULL); + atomic_store(&sampler_started, 0); + /* the part of the last interval since the sampler's final record */ + pthread_mutex_lock(&stats_out_lock); + stats_emit_interval(); + pthread_mutex_unlock(&stats_out_lock); +} + +/* ---- fork ---- + * + * The forking thread holds both stats locks across fork() so the child never + * inherits one held by a thread that does not exist there. */ +static void stats_fork_prepare(void) +{ + pthread_mutex_lock(&stats_out_lock); + pthread_mutex_lock(&stats_lock); +} + +static void stats_fork_parent(void) +{ + pthread_mutex_unlock(&stats_lock); + pthread_mutex_unlock(&stats_out_lock); +} + +/* In the child: drop every block (their threads are gone, and the child + * counts from zero) and forget the parent's sampler. */ +static void stats_fork_child(void) +{ + struct dto_thread_stats *t = stats_threads, *next; + + pthread_mutex_unlock(&stats_lock); + pthread_mutex_unlock(&stats_out_lock); + for (; t; t = next) { + next = t->next; + munmap(t, sizeof(*t)); + } + stats_threads = NULL; + orig_memset(&stats_retired, 0, sizeof(stats_retired)); + orig_memset(&sampler_prev, 0, sizeof(sampler_prev)); + sampler_prev_ns = 0; + pthread_setspecific(stats_key, NULL); + thr_stats = NULL; + thr_stats_off = false; + atomic_store(&sampler_started, 0); + stats_out_len = 0; + /* with %p the child writes a file of its own: init_dto opens it */ + if (stats_file_per_pid && stats_fd != -1) { + close(stats_fd); + stats_fd = -1; + } +} #endif #define DTO_MAX_PARAM_LEN 16 @@ -1454,28 +2016,43 @@ static int init_dto(void) } if (collect_stats) { + env_str = getenv("DTO_STATS_FORMAT"); + if (env_str != NULL && !strcmp(env_str, "json")) + stats_format = STATS_FORMAT_JSON; + else if (env_str != NULL && strcmp(env_str, "text")) + LOG_ERROR("Unknown DTO_STATS_FORMAT '%s', using text\n", env_str); + + env_str = getenv("DTO_STATS_INTERVAL_MS"); + if (env_str != NULL) { + errno = 0; + stats_interval_ms = strtoul(env_str, NULL, 10); + if (errno) + stats_interval_ms = 0; + } + + /* A forked child keeps the parent's (O_APPEND) file, + * unless the name has a %p (see stats_fork_child). */ env_str = getenv("DTO_STATS_FILE"); - if (env_str != NULL) { - struct stat st; - - if (lstat(env_str, &st) == -1 || - (st.st_mode & S_IFMT) == S_IFREG) { - stats_fd = open(env_str, O_WRONLY | O_CREAT | O_TRUNC | O_CLOEXEC, 0600); - if (stats_fd == -1) - LOG_ERROR("Failed to open DTO_STATS_FILE '%s': %s\n", env_str, strerror(errno)); - } else { - LOG_ERROR("DTO_STATS_FILE '%s' must reference a regular file\n", env_str); - } - } - - clock_gettime(CLOCK_BOOTTIME, &dto_start_time); - /* Change the log level to 'trace' so that the - * stats can be logged - */ - log_level = LOG_LEVEL_TRACE; - dto_stats_log("Stats collection enabled, some latencies may be higher due to collection overhead.\n"); - install_sigint_handler(); - } + if (env_str != NULL && stats_fd == -1) + stats_open_file(env_str); + + if (!stats_key_created && + pthread_key_create(&stats_key, stats_thread_exit) == 0) + stats_key_created = true; + if (!stats_key_created) { + LOG_ERROR("Stats collection disabled: no thread key\n"); + collect_stats = 0; + } + } + + if (collect_stats) { + clock_gettime(CLOCK_BOOTTIME, &dto_start_time); + if (stats_format == STATS_FORMAT_TEXT) { + stats_printf("Stats collection enabled, some latencies may be higher due to collection overhead.\n"); + stats_flush(); + } + install_sigint_handler(); + } #endif /* Register fork handler for the child process */ @@ -1485,7 +2062,11 @@ static int init_dto(void) * take that risk and disable DSA offload for parent * as well. */ +#ifdef DTO_STATS_SUPPORT + if (!pthread_atfork(stats_fork_prepare, stats_fork_parent, child)) +#else if (!pthread_atfork(NULL, NULL, child)) +#endif fork_handler_registered = 1; else { LOG_ERROR("Setting fork() handler failed. " @@ -1634,7 +2215,10 @@ static void cleanup_dto(void) close(wqs[i].wq_fd); } #ifdef DTO_STATS_SUPPORT - print_stats(); + if (collect_stats) { + stats_stop_sampler(); + print_stats(); + } restore_sigint_handler(); #endif if (log_fd != -1) diff --git a/tests/test_stats.py b/tests/test_stats.py new file mode 100644 index 0000000..fff0449 --- /dev/null +++ b/tests/test_stats.py @@ -0,0 +1,141 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: MIT +# +# Checks DTO_COLLECT_STATS output against test_stats_workload's known calls. +# Needs no DSA hardware: without it every call is counted on the CPU path. +# +# Usage: test_stats.py + +import json +import os +import subprocess +import sys +import tempfile + +WORKLOAD = sys.argv[1] +SIZES = {"memcpy": 131072 + 123, "memset": 65536 + 4096, + "memmove": 98304 + 7, "memcmp": 262144 + 64} +SIZE_CHILD = 524288 + 1 +failures = 0 + + +def check(desc, ok, detail=""): + global failures + print(("PASS: " if ok else "FAIL: ") + desc + ("" if ok else " " + detail)) + failures += not ok + + +def run(args, env_extra, stats_path): + env = dict(os.environ, DTO_COLLECT_STATS="1", DTO_STATS_FILE=stats_path, + **env_extra) + out = subprocess.run([WORKLOAD] + args, env=env, check=True, + capture_output=True, text=True).stdout + with open(stats_path) as f: + return out, f.read() + + +def records(text): + return [json.loads(line) for line in text.splitlines() if line.strip()] + + +def op_sum(ops, op, field): + return sum(g[field] for g in ops.get(op, {}).values()) + + +def bucket_for(summary, size): + for b in summary["histogram"]: + if b["min"] <= size and ("max" not in b or size <= b["max"]): + return b + return None + + +tmp = tempfile.mkdtemp() + +# --- totals and histogram, JSON --- +threads, iters = 4, 50 +_, text = run(["basic", str(threads), str(iters)], {"DTO_STATS_FORMAT": "json"}, + os.path.join(tmp, "basic.json")) +recs = records(text) +summaries = [r for r in recs if r["type"] == "summary"] +check("one summary record", len(summaries) == 1, str(len(summaries))) +s = summaries[0] +n = threads * iters +for op, size in SIZES.items(): + b = bucket_for(s, size) + check(f"{op} lands in the bucket for {size} bytes", b is not None and op in b["ops"], + json.dumps(b)) + if b is None or op not in b["ops"]: + continue + calls = op_sum(b["ops"], op, "calls") + failed = b["ops"][op].get("dsa_failed", {}).get("calls", 0) + # a call the device left partly done is counted as dsa_failed plus cpu + check(f"{op} calls ({n})", calls - failed == n, f"calls {calls} failed {failed}") + check(f"{op} bytes ({n * size})", op_sum(b["ops"], op, "bytes") == n * size, + str(op_sum(b["ops"], op, "bytes"))) + check(f"{op} time recorded", op_sum(b["ops"], op, "ns") > 0) +check("summary carries the configuration", "wait_method" in s["config"]) + +# --- interval records add up to the totals --- +_, text = run(["basic", "2", "40", "2"], + {"DTO_STATS_FORMAT": "json", "DTO_STATS_INTERVAL_MS": "10"}, + os.path.join(tmp, "interval.json")) +recs = records(text) +intervals = [r for r in recs if r["type"] == "interval"] +s = [r for r in recs if r["type"] == "summary"][0] +check("interval records emitted", len(intervals) >= 5, str(len(intervals))) +check("interval times increase", + all(b["t_ms"] > a["t_ms"] for a, b in zip(intervals, intervals[1:]))) +for op in SIZES: + for field in ("calls", "bytes"): + total = sum(op_sum(r["ops"], op, field) for r in intervals) + check(f"intervals sum to the {op} {field} total", + total == op_sum(s["totals"], op, field), + f"{total} vs {op_sum(s['totals'], op, field)}") + +# --- short-lived threads: all counted, blocks released --- +out, text = run(["churn", "200", "5"], {"DTO_STATS_FORMAT": "json"}, + os.path.join(tmp, "churn.json")) +s = [r for r in records(text) if r["type"] == "summary"][0] +b = bucket_for(s, SIZES["memcpy"]) +calls = op_sum(b["ops"], "memcpy", "calls") - \ + b["ops"]["memcpy"].get("dsa_failed", {}).get("calls", 0) +check("calls from 200 exited threads counted", calls == 200 * 5, str(calls)) +before, after = (int(x) for x in out.split()[1::2]) +check("exited threads' stats blocks are unmapped", after - before < 20, + f"mappings {before} -> {after}") + +# --- fork: separate records per process, appended to one file --- +out, text = run(["fork", "30"], {"DTO_STATS_FORMAT": "json"}, + os.path.join(tmp, "fork.json")) +child = int(out.split()[1]) +by_pid = {r["pid"]: r for r in records(text) if r["type"] == "summary"} +check("parent and child both report, in one file", len(by_pid) == 2, str(list(by_pid))) +if child in by_pid and len(by_pid) == 2: + parent = [p for p in by_pid if p != child][0] + cb = bucket_for(by_pid[child], SIZE_CHILD) + check("child counts only its own copies", + cb is not None and bucket_for(by_pid[child], SIZES["memcpy"]) is None) + pb = bucket_for(by_pid[parent], SIZES["memcpy"]) + check("parent counts its copies from before and after the fork", + pb is not None and op_sum(pb["ops"], "memcpy", "bytes") == 60 * SIZES["memcpy"]) + +# --- %p: a file per process --- +pattern = os.path.join(tmp, "pp.%p.json") +env = dict(os.environ, DTO_COLLECT_STATS="1", DTO_STATS_FILE=pattern, + DTO_STATS_FORMAT="json") +subprocess.run([WORKLOAD, "fork", "5"], env=env, check=True, capture_output=True) +files = [f for f in os.listdir(tmp) if f.startswith("pp.")] +check("%p gives each process its own file", len(files) == 2, str(files)) + +# --- text format --- +_, text = run(["basic", "1", "10"], {"DTO_STATS_FORMAT": "text"}, + os.path.join(tmp, "basic.txt")) +check("text summary has the operation table", "Number of Memory Operations" in text + and "Byte Range" in text) +check("text summary lists the memcpy bucket", " 131072-163839" in text) + +print() +if failures: + print(f"{failures} stats test(s) failed") + sys.exit(1) +print("All stats tests passed") diff --git a/tests/test_stats_workload.c b/tests/test_stats_workload.c new file mode 100644 index 0000000..d6f257a --- /dev/null +++ b/tests/test_stats_workload.c @@ -0,0 +1,132 @@ +/******************************************************************************* + * Copyright (C) 2026 Intel Corporation + * + * SPDX-License-Identifier: MIT + * + * test_stats_workload.c - known memory operations for tests/test_stats.py, + * which runs this with DTO_COLLECT_STATS set and checks what DTO reports. + * + * basic [sleep_ms] each thread, per iteration: memcpy, + * memset, memmove, memcmp of + * SIZE_{CPY,SET,MOV,CMP} bytes + * churn the same, from short-lived threads + * started one after another; prints + * the mapping count before and after + * fork memcpy SIZE_CPY in the parent and + * SIZE_CHILD in a forked child + * + * Built with -fno-builtin so every call reaches DTO. + ******************************************************************************/ + +#include +#include +#include +#include +#include +#include +#include +#include + +/* distinctive sizes, each in its own histogram bucket */ +#define SIZE_CPY (131072 + 123) +#define SIZE_SET (65536 + 4096) +#define SIZE_MOV (98304 + 7) +#define SIZE_CMP (262144 + 64) +#define SIZE_CHILD (524288 + 1) +#define BUF (1 << 20) + +static int iters, sleep_ms; + +static void *work(void *arg) +{ + char *a = mmap(NULL, BUF, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_POPULATE, -1, 0); + char *b = mmap(NULL, BUF, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_POPULATE, -1, 0); + struct timespec pause = { 0, sleep_ms * 1000000L }; + + for (int i = 0; i < iters; i++) { + memcpy(b, a, SIZE_CPY); + memset(a, i, SIZE_SET); + memmove(b + 64, b, SIZE_MOV); + if (memcmp(a, a + 0, SIZE_CMP)) + abort(); + if (sleep_ms) + nanosleep(&pause, NULL); + } + munmap(a, BUF); + munmap(b, BUF); + return NULL; +} + +static int count_maps(void) +{ + char line[512]; + int n = 0; + FILE *f = fopen("/proc/self/maps", "r"); + + while (f && fgets(line, sizeof(line), f)) + n++; + if (f) + fclose(f); + return n; +} + +static void copies(size_t size, int n) +{ + char *a = mmap(NULL, BUF, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_POPULATE, -1, 0); + char *b = mmap(NULL, BUF, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_POPULATE, -1, 0); + + for (int i = 0; i < n; i++) + memcpy(b, a, size); + munmap(a, BUF); + munmap(b, BUF); +} + +int main(int argc, char **argv) +{ + pthread_t t[256]; + + if (argc < 3) + return 2; + if (!strcmp(argv[1], "basic")) { + int nt = atoi(argv[2]); + + iters = atoi(argv[3]); + sleep_ms = argc > 4 ? atoi(argv[4]) : 0; + for (int i = 0; i < nt; i++) + pthread_create(&t[i], NULL, work, NULL); + for (int i = 0; i < nt; i++) + pthread_join(t[i], NULL); + } else if (!strcmp(argv[1], "churn")) { + int nt = atoi(argv[2]), before, after; + + iters = atoi(argv[3]); + /* the first thread's block and allocations, so they don't count */ + pthread_create(&t[0], NULL, work, NULL); + pthread_join(t[0], NULL); + before = count_maps(); + for (int i = 1; i < nt; i++) { + pthread_create(&t[0], NULL, work, NULL); + pthread_join(t[0], NULL); + } + after = count_maps(); + printf("maps_before %d maps_after %d\n", before, after); + } else if (!strcmp(argv[1], "fork")) { + int n = atoi(argv[2]), st; + pid_t pid; + + copies(SIZE_CPY, n); + pid = fork(); + if (pid == 0) { + copies(SIZE_CHILD, n); + exit(0); + } + waitpid(pid, &st, 0); + copies(SIZE_CPY, n); + printf("child %d\n", pid); + } + return 0; +} From fddac82cc72aefc9768b23e197bbbd9651486775 Mon Sep 17 00:00:00 2001 From: Daniel Byrne Date: Wed, 21 Oct 2026 11:29:57 -0700 Subject: [PATCH 07/11] stats: link libdto with pthread The stats rework uses thread keys, a sampler thread and its condition variable. Before glibc 2.34 those functions are in libpthread, not libc, so preloading libdto into a program that doesn't link libpthread itself (sh, true, most single-threaded tools) failed at load time: true: symbol lookup error: libdto.so: undefined symbol: pthread_key_create Link libdto with pthread in CMake and the Makefile (an empty stub on newer glibc), and check in the stats test that stats work in a program without libpthread. Seen on RHEL 8 (glibc 2.28); Ubuntu 24.04 (2.39) is not affected. --- CMakeLists.txt | 7 +++++-- Makefile | 2 +- tests/test_stats.py | 9 ++++++++- 3 files changed, 14 insertions(+), 4 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 4419a70..a383cfe 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -65,7 +65,10 @@ target_compile_definitions(dto PRIVATE DTO_STATS_SUPPORT) # Link libraries set(DTO_STATIC_LIBS "") -set(DTO_DYNAMIC_LIBS dl) +# pthread: stats use thread keys and a sampler thread; before glibc 2.34 these +# are not in libc, so a program that doesn't link libpthread itself would fail +# to load libdto. +set(DTO_DYNAMIC_LIBS dl pthread) # Handle accel-config support if(DTO_ACCEL_CONFIG_SUPPORT) @@ -230,7 +233,7 @@ if(DTO_BUILD_TESTS) if(Python3_Interpreter_FOUND) add_test(NAME stats COMMAND ${Python3_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/tests/test_stats.py - $) + $ $) set_tests_properties(stats PROPERTIES LABELS "functional" TIMEOUT 120 diff --git a/Makefile b/Makefile index 25b649a..9c809be 100644 --- a/Makefile +++ b/Makefile @@ -16,7 +16,7 @@ MANDIR ?= /usr/share/man DTO_VERSION := $(shell grep -o 'DTO VERSION [0-9.]*' CMakeLists.txt | cut -d' ' -f3) libdto: dto.c - gcc -shared -fPIC -Wl,-soname,libdto.so dto.c $(DML_LIB_CXX) -DDTO_STATS_SUPPORT -DDTO_ACCEL_CONFIG_SUPPORT -DDTO_NUMA_SUPPORT -o libdto.so.1.0 -laccel-config -ldl -lnuma -mwaitpkg + gcc -shared -fPIC -Wl,-soname,libdto.so dto.c $(DML_LIB_CXX) -DDTO_STATS_SUPPORT -DDTO_ACCEL_CONFIG_SUPPORT -DDTO_NUMA_SUPPORT -o libdto.so.1.0 -laccel-config -ldl -lnuma -lpthread -mwaitpkg libdto_nostats: dto.c gcc -shared -fPIC -Wl,-soname,libdto.so dto.c $(DML_LIB_CXX) -DDTO_ACCEL_CONFIG_SUPPORT -DDTO_NUMA_SUPPORT -o libdto.so.1.0 -laccel-config -ldl -lnuma -mwaitpkg diff --git a/tests/test_stats.py b/tests/test_stats.py index fff0449..0043159 100644 --- a/tests/test_stats.py +++ b/tests/test_stats.py @@ -4,7 +4,7 @@ # Checks DTO_COLLECT_STATS output against test_stats_workload's known calls. # Needs no DSA hardware: without it every call is counted on the CPU path. # -# Usage: test_stats.py +# Usage: test_stats.py import json import os @@ -13,6 +13,7 @@ import tempfile WORKLOAD = sys.argv[1] +LIBDTO = os.path.abspath(sys.argv[2]) SIZES = {"memcpy": 131072 + 123, "memset": 65536 + 4096, "memmove": 98304 + 7, "memcmp": 262144 + 64} SIZE_CHILD = 524288 + 1 @@ -127,6 +128,12 @@ def bucket_for(summary, size): files = [f for f in os.listdir(tmp) if f.startswith("pp.")] check("%p gives each process its own file", len(files) == 2, str(files)) +# --- a program that doesn't link libpthread (before glibc 2.34, it isn't libc) --- +env = dict(os.environ, LD_PRELOAD=LIBDTO, DTO_COLLECT_STATS="1", + DTO_STATS_INTERVAL_MS="1", DTO_STATS_FILE=os.path.join(tmp, "true.txt")) +p = subprocess.run(["true"], env=env, capture_output=True, text=True) +check("stats work in a program without libpthread", p.returncode == 0, p.stderr.strip()) + # --- text format --- _, text = run(["basic", "1", "10"], {"DTO_STATS_FORMAT": "text"}, os.path.join(tmp, "basic.txt")) From c1ffec8944c9f412455cda240b49fc7f9d470e36 Mon Sep 17 00:00:00 2001 From: Daniel Byrne Date: Wed, 28 Oct 2026 09:24:47 -0700 Subject: [PATCH 08/11] stats: fix fork and exit safety of the stats machinery - stats_fork_child() called pthread_setspecific(stats_key, NULL) even when stats were off and the key had never been created, i.e. on key 0, which belongs to whichever library created it first. Only touch the key when it exists. - sampler_lock was not held across fork(), so a child could inherit it locked by the parent's sampler thread and hang in its first stats_start_sampler(). Take it in the prepare handler with the other two and release it in parent and child. - stats_stop_sampler() reset sampler_started, so a thread whose first counted call landed after cleanup_dto() restarted the sampler during process exit, writing to closed descriptors. Set stats_shutdown in cleanup_dto() and check it before starting the sampler. - stats_flush() gave up on EINTR and EAGAIN and discarded the rest of the record. Retry on EINTR and poll() on EAGAIN. - The 64 KiB output buffer was smaller than a worst-case JSON summary, so a record could go out in two write()s and interleave with another process's in a shared O_APPEND file. Make it 256 KiB. - DTO_STATS_INTERVAL_MS accepted negative, non-numeric and absurd values silently; reject them with an error like DTO_STATS_FORMAT does. --- dto.c | 36 +++++++++++++++++++++++++++++++----- 1 file changed, 31 insertions(+), 5 deletions(-) diff --git a/dto.c b/dto.c index 51899d0..8c7f4dd 100644 --- a/dto.c +++ b/dto.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include #include @@ -318,6 +319,7 @@ static bool stats_file_per_pid; /* DTO_STATS_FILE has a %p */ /* DTO_STATS_INTERVAL_MS */ static unsigned long stats_interval_ms; static atomic_uchar sampler_started; +static bool stats_shutdown; /* cleanup_dto ran: no new sampler */ static struct dto_stats sampler_prev; /* totals at the previous record */ static uint64_t sampler_prev_ns; @@ -831,7 +833,8 @@ static struct dto_thread_stats *stats_thread_register(void) pthread_setspecific(stats_key, ts); thr_stats = ts; - if (stats_interval_ms && !atomic_load_explicit(&sampler_started, + if (stats_interval_ms && !stats_shutdown && + !atomic_load_explicit(&sampler_started, memory_order_acquire)) stats_start_sampler(); thr_stats_off = false; @@ -872,7 +875,10 @@ static void update_stats(int op, size_t n, bool overlapping, size_t bytes_comple * possible: to DTO_STATS_FILE (opened O_APPEND, so records from several * processes sharing it stay whole), else DTO_LOG_FILE, else stdout. */ -static char stats_out[1 << 16]; +/* Holds a whole record, so a summary (up to ~1 KB for each of the 106 + * histogram buckets in JSON) goes out in one write() and records from + * several processes sharing an O_APPEND file stay whole. */ +static char stats_out[1 << 18]; static size_t stats_out_len; static void stats_flush(void) @@ -883,6 +889,14 @@ static void stats_flush(void) while (off < stats_out_len) { ssize_t r = write(fd, stats_out + off, stats_out_len - off); + if (r < 0 && errno == EINTR) + continue; + if (r < 0 && errno == EAGAIN) { + struct pollfd pfd = { .fd = fd, .events = POLLOUT }; + + if (poll(&pfd, 1, -1) > 0 || errno == EINTR) + continue; + } if (r <= 0) break; off += r; @@ -1309,10 +1323,12 @@ static void stats_fork_prepare(void) { pthread_mutex_lock(&stats_out_lock); pthread_mutex_lock(&stats_lock); + pthread_mutex_lock(&sampler_lock); } static void stats_fork_parent(void) { + pthread_mutex_unlock(&sampler_lock); pthread_mutex_unlock(&stats_lock); pthread_mutex_unlock(&stats_out_lock); } @@ -1323,6 +1339,7 @@ static void stats_fork_child(void) { struct dto_thread_stats *t = stats_threads, *next; + pthread_mutex_unlock(&sampler_lock); pthread_mutex_unlock(&stats_lock); pthread_mutex_unlock(&stats_out_lock); for (; t; t = next) { @@ -1333,9 +1350,11 @@ static void stats_fork_child(void) orig_memset(&stats_retired, 0, sizeof(stats_retired)); orig_memset(&sampler_prev, 0, sizeof(sampler_prev)); sampler_prev_ns = 0; - pthread_setspecific(stats_key, NULL); + if (stats_key_created) + pthread_setspecific(stats_key, NULL); thr_stats = NULL; thr_stats_off = false; + sampler_stop = false; atomic_store(&sampler_started, 0); stats_out_len = 0; /* with %p the child writes a file of its own: init_dto opens it */ @@ -2024,10 +2043,16 @@ static int init_dto(void) env_str = getenv("DTO_STATS_INTERVAL_MS"); if (env_str != NULL) { + char *end; + errno = 0; - stats_interval_ms = strtoul(env_str, NULL, 10); - if (errno) + stats_interval_ms = strtoul(env_str, &end, 10); + /* a day: beyond that it is a typo, and the timespec math stays exact */ + if (errno || end == env_str || *end || *env_str == '-' || + stats_interval_ms > 86400000UL) { + LOG_ERROR("Invalid DTO_STATS_INTERVAL_MS '%s', no interval records\n", env_str); stats_interval_ms = 0; + } } /* A forked child keeps the parent's (O_APPEND) file, @@ -2216,6 +2241,7 @@ static void cleanup_dto(void) } #ifdef DTO_STATS_SUPPORT if (collect_stats) { + stats_shutdown = true; stats_stop_sampler(); print_stats(); } From c2a4d5482fe8152e76a4a0cae578e848b732ed7a Mon Sep 17 00:00:00 2001 From: Daniel Byrne Date: Wed, 21 Oct 2026 11:30:11 -0700 Subject: [PATCH 09/11] dtoctl: add --stats-run to measure the speedup from DSA "dtoctl --stats-run[=DIR] [--repeat=N] [--perf[=PERF]] prog" runs the program CPU-only (DTO_USESTDC_CALLS=1) and then with DSA, N times each, alternating, with DTO_STATS_FORMAT=json and each run's stats in its own file in DIR. dtoctl records each run's wall time, user and system CPU time (wait4, so including children) and exit status in DIR/runs.jsonl, then runs dto-stats-report on DIR. dto-stats-report (Python 3, standard library only) sums the stats of every process a run started, takes medians over the runs, and prints: wall, CPU and mem* time each way; Amdahl's law from f (share of the CPU-only run's CPU time in mem* calls) and s (mem* speedup per byte), 1 / ((1 - f) + f / s), against the measured speedup and the 1 / (1 - f) limit; each operation and the sizes with the most time; and notes: no call reached DSA (and why), runs did different amounts of work, sizes where DSA was slower (with a DTO_MIN_BYTES to try), runs that failed or wrote no stats. --json prints the numbers instead. With --perf, one more run each way goes through "PERF record -e cpu-clock" (cpu-clock works without hardware counters, and on a test kernel here cycles hardly sampled). perf runs dtoctl again, so DTO is preloaded into the program and not into perf. The report splits each profile into copies, memset, memcmp, CRC, kernel copies, DTO itself and the rest, and notes mem* time DTO did not intercept and CRC time DSA could take. Where libc has no symbol table (Ubuntu), perf shows its memcpy as an address; on the machine that made the profile the report asks its own libc, through dlsym, which implementation of each function this CPU uses, and places samples by address, after checking the profile's libc has the same build ID. Ctrl-C and Ctrl-\ go to the program, as with system(); a run that fails stops the rest, and dtoctl exits with its status. Also adds --stats-format and --stats-interval for DTO_STATS_FORMAT and DTO_STATS_INTERVAL_MS, and documents them with --stats-run in the man page, README and TESTING.md. Tests: test_dtoctl.sh checks the new options, the --stats-run plan (--dry-run), a real stats run of a shell command and a failing one; tests/test_stats_report.py checks the report's arithmetic on made-up runs with known answers, sorts made-up perf lines, and finds memcpy, memset and memcmp by address in this machine's libc. --- CMakeLists.txt | 15 + Makefile | 1 + README.md | 83 +++++ TESTING.md | 12 +- doc/dtoctl.md | 95 +++++- dto-stats-report | 676 +++++++++++++++++++++++++++++++++++++ dtoctl.c | 551 +++++++++++++++++++++++++++++- tests/test_dtoctl.sh | 57 +++- tests/test_stats_report.py | 219 ++++++++++++ 9 files changed, 1692 insertions(+), 17 deletions(-) create mode 100755 dto-stats-report create mode 100755 tests/test_stats_report.py diff --git a/CMakeLists.txt b/CMakeLists.txt index a383cfe..2c83353 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -160,6 +160,11 @@ else() endif() install(TARGETS dtoctl RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}) +# dto-stats-report summarizes "dtoctl --stats-run"; dtoctl looks for it next to +# itself first, so copy it beside dtoctl in the build tree too. +configure_file(dto-stats-report ${CMAKE_CURRENT_BINARY_DIR}/dto-stats-report COPYONLY) +install(PROGRAMS dto-stats-report DESTINATION ${CMAKE_INSTALL_BINDIR}) + # Build and install the optional section 1 manual. find_program(PANDOC_EXECUTABLE NAMES pandoc) if(PANDOC_EXECUTABLE) @@ -241,6 +246,16 @@ if(DTO_BUILD_TESTS) else() message(STATUS "stats test: disabled (python3 not found)") endif() + # dto-stats-report's arithmetic, on made-up runs + if(Python3_Interpreter_FOUND) + add_test(NAME stats-report + COMMAND ${Python3_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/tests/test_stats_report.py + ${CMAKE_CURRENT_BINARY_DIR}/dto-stats-report) + set_tests_properties(stats-report PROPERTIES + LABELS "functional" + TIMEOUT 60 + ) + endif() # ---- A/B performance regression test ---- # The baseline object is compiled from unguarded upstream sources that diff --git a/Makefile b/Makefile index 9c809be..15467f5 100644 --- a/Makefile +++ b/Makefile @@ -43,6 +43,7 @@ install: dtoctl install-man ln -sf /usr/lib64/libdto.so.1.0 /usr/lib64/libdto.so.1 ln -sf /usr/lib64/libdto.so.1.0 /usr/lib64/libdto.so cp dtoctl /usr/bin/ + cp dto-stats-report /usr/bin/ install-man: man @if [ -f doc/dtoctl.1 ]; then \ diff --git a/README.md b/README.md index cb1a842..a7a116e 100644 --- a/README.md +++ b/README.md @@ -195,6 +195,85 @@ rather than `dtoctl sudo ...`, which loses `LD_PRELOAD` and the `DTO_*` variables, and pass settings as options: `sudo` does not pass on variables exported in your shell. +### Measuring the speedup from DSA + +`dtoctl --stats-run` runs a program CPU-only (`DTO_USESTDC_CALLS=1`) and then +with DSA, collecting DTO's JSON stats (see [Statistics](#statistics)) from each +run into a directory, and `dto-stats-report` (installed with `dtoctl`) compares +them. With `--repeat=N` each way runs N times, alternating, and the report uses +the medians; `--perf` adds one more run each way under `perf record`. + +```bash +sudo dtoctl -w busypoll --stats-run=app-stats --repeat=3 --perf ./app +``` + +``` +DTO stats run: app-stats + command: ./app + settings: DTO_WAIT_METHOD=busypoll + timed runs: 3 CPU-only, 3 with DSA (medians) + DTO in the DSA run: 1 WQs, wait busypoll, min_bytes 65536, cpu_size_fraction 0.5 (auto-adjusted) + + CPU only with DSA speedup + wall time 972.66 ms 1.55 s 0.63x + CPU time (user+sys) 968.31 ms 1.54 s 0.63x + in mem* calls 304.86 ms 794.05 ms 0.38x + outside mem* calls 663.45 ms 747.81 ms 0.89x + wall time, min-max 972.24-973.64 ms 1.26-1.58 s + + Amdahl's law + f, share of CPU time in mem* calls (CPU-only) 31.5% + s, mem* speedup with DSA (time per byte) 0.38x + predicted speedup, 1 / ((1 - f) + f / s) 0.66x + measured speedup, wall time 0.63x + limit if mem* took no time, 1 / (1 - f) 1.46x + + per operation calls bytes avg size CPU only with DSA speedup on DSA f + memcpy 20.00k 4.9 GiB 256.0 KiB 115.50 ms 266.45 ms 0.43x 100.0% 11.9% + memset 20.00k 4.9 GiB 256.0 KiB 96.36 ms 83.32 ms 1.16x 100.0% 10.0% + memcmp 20.00k 4.9 GiB 256.0 KiB 93.00 ms 444.27 ms 0.21x 100.0% 9.6% + (on DSA: share of the bytes in calls sent to DSA, of which the CPU does cpu_size_fraction; + f: share of the CPU-only run's CPU time) + calls DSA left to the CPU, per run: page fault 1, retry 0, other 0 + + by size (most time) calls CPU only with DSA speedup on DSA + 256K-320K 60.00k 5.08 us 13.23 us 0.38x 100.0% + (average time per call; on DSA: share of calls sent to DSA) + + profile (perf, % of CPU time) CPU only with DSA + memcpy/memmove 13.0% 12.2% + memset 10.0% 1.6% + memcmp 10.2% 0.0% + CRC 14.6% 11.7% + DTO library 0.0% 34.4% + libc, unknown function 0.0% 0.1% + other user code 51.5% 39.6% + other kernel code 0.7% 0.5% + + note: mem* calls that went to DSA were slower than on the CPU at every size this program used + note: 15% of the CPU-only run is CRC code: DSA can compute CRC32C, but DTO doesn't intercept CRC functions, so offloading it needs an explicit call +``` + +Here a single thread copies, sets and compares 256 KiB buffers that stay in +the CPU caches, computes a CRC32C in software, and does other work. The CPU-only +run spent f = 31.5% of its CPU time in mem\* calls, and those calls ran +s = 0.38x as fast with DSA, so Amdahl's law predicts 1 / ((1 - f) + f / s) = +0.66x for the whole program; it measured 0.63x. The rest of the gap is the +time outside mem\* calls, which grew with DSA. At best, mem\* calls taking no +time at all, this program could run 1.46x faster. + +The report also shows each operation and the sizes with the most time, so you +can see where DSA wins and set `DTO_MIN_BYTES` accordingly, and notes anything +that makes the comparison doubtful. Among them: no call reaching DSA (no usable +work queue: run with `sudo`), and runs that did different amounts of work, as +a benchmark that runs for a fixed time does (speedups are then per byte). The +perf profile splits each run's CPU time into copies, memset, memcmp, CRC, +kernel copies, DTO itself (here, busy-polling for completions) and the rest. It +also counts mem\* calls DTO cannot intercept, such as those made inside libc. +Run `dto-stats-report app-stats` again at any time, or with `--json` for the +numbers. Amdahl's estimate assumes the program keeps its CPUs busy; one that +waits for I/O gains less in wall time. + Run `dtoctl --help` for the option list, or `man 1 dtoctl` for the full manual, which documents each option and gives a ready-to-run command for each of the usage models above. @@ -376,6 +455,10 @@ operations and paths with no calls are left out. `t_ms` and `run_ms` count from DTO's start, and `config` holds the settings at exit, including the auto-tuned `min_bytes` and `cpu_size_fraction`. +To compare a run with DSA against a CPU-only run, and see how close the speedup +comes to what Amdahl's law predicts, use `dtoctl --stats-run` (see +[Measuring the speedup from DSA](#measuring-the-speedup-from-dsa)). + Every process that loads DTO with stats enabled writes its own records, tagged with its `pid`. They share `DTO_STATS_FILE` (records are appended whole) unless the path contains `%p`, which gives each process its own file. diff --git a/TESTING.md b/TESTING.md index ab8a41f..bf2ef7b 100644 --- a/TESTING.md +++ b/TESTING.md @@ -86,8 +86,10 @@ queues DTO uses share the same setting. Checks for the `dtoctl` wrapper: every option sets its `DTO_*` variable, invalid values are rejected, the library path is resolved and validated, `LD_PRELOAD` and inherited variables are handled, and exit statuses follow -`env(1)`. The last check runs a program under `dtoctl` and reads libdto's own -configuration trace. No DSA hardware is needed. +`env(1)`. `--stats-run` is checked from its `--dry-run` plan and by running a +small shell command: each run's environment and stats file, `runs.jsonl`, the +report, and stopping at a failed run. The last check runs a program under +`dtoctl` and reads libdto's own configuration trace. No DSA hardware is needed. ```bash ctest -R dtoctl @@ -108,6 +110,12 @@ It needs Python 3 and no DSA hardware. ctest -R stats ``` +`stats-report` checks `dto-stats-report` on made-up runs with known answers +(Amdahl's law, per-operation and per-size speedups, medians, the notes it +makes), sorts made-up perf samples into categories, and checks that it finds +memcpy, memset and memcmp by address in this machine's libc. It needs Python 3, +and neither DSA nor perf. + ## Performance Tests The performance test (`perf`) detects regressions in libdto by diff --git a/doc/dtoctl.md b/doc/dtoctl.md index afca4ed..8452bff 100644 --- a/doc/dtoctl.md +++ b/doc/dtoctl.md @@ -13,6 +13,8 @@ dtoctl [_options_] COMMAND [_command-options_] dtoctl \--dry-run [_options_] [COMMAND [_command-options_]] +dtoctl \--stats-run[=_DIR_] [\--repeat=_N_] [\--perf[=_PERF_]] [_options_] COMMAND [_command-options_] + DESCRIPTION =========== @@ -160,13 +162,24 @@ Other toggles Statistics and logging ---------------------- \--stats -: Enable stats collection. For debugging/profiling only, as it slows down the - workload. Sets **DTO_COLLECT_STATS=1**. Default: off. +: Enable stats collection: calls, bytes and time for each operation, by + path (CPU, DSA, DSA failed) and by size. Costs about 40 ns per call. + Sets **DTO_COLLECT_STATS=1**. Default: off. \--stats-file=_PATH_ -: Write the stats histogram to _PATH_ instead of standard output. Implies - `--stats`, since DTO ignores the file otherwise. Sets **DTO_STATS_FILE** - and **DTO_COLLECT_STATS=1**. Default: standard output. +: Write the stats to _PATH_ instead of standard output; each `%p` in _PATH_ + becomes the process ID. The file is appended to. Implies `--stats`, + since DTO ignores the file otherwise. Sets **DTO_STATS_FILE** and + **DTO_COLLECT_STATS=1**. Default: standard output. + +\--stats-format=_FORMAT_ +: _text_ for tables, or _json_ for JSON Lines (one record per line). + Implies `--stats`. Sets **DTO_STATS_FORMAT**. Default: _text_. + +\--stats-interval=_MS_ +: Also write the stats for every _MS_ milliseconds, from a background + thread, as well as the summary at exit. Implies `--stats`. Sets + **DTO_STATS_INTERVAL_MS**. Default: summary only. \--log-file=_PATH_ : Redirect DTO output to _PATH_._program_._pid_, one file per process. @@ -176,6 +189,27 @@ Statistics and logging : Verbosity of DTO logging: _0_, _1_, or _2_. Sets **DTO_LOG_LEVEL**. Default: _0_. +Stats run +--------- +\--stats-run[=_DIR_] +: Measure how much DSA speeds `COMMAND` up. Run it CPU-only + (**DTO_USESTDC_CALLS=1**), then with DSA (**DTO_USESTDC_CALLS=0**), + collecting JSON stats from each run into _DIR_, then summarize them with + **dto-stats-report** (see STATS RUN). _DIR_ must be new or empty; the + default is a new `dto-stats--