Doc updates for 1.71.0

docs/conceptual/transferbench-data-validation.rst

@@ -85,8 +85,9 @@ How validation is timed
 
 The timing of validation is controlled by the ``alwaysValidate`` option. By default
 (``alwaysValidate = 0``), validation runs once after all timed iterations complete,
-minimizing overhead during benchmarking. When ``alwaysValidate = 1``, validation is
-performed after every iteration; any detected error immediately stops the run.
+minimizing overhead during benchmarking. When ``alwaysValidate > 0``, validation is
+performed after every iteration; any detected error immediately stops the run. Setting
+``alwaysValidate`` to a negative value disables validation entirely.
 
 .. list-table::
     :header-rows: 1
@@ -95,11 +96,15 @@ performed after every iteration; any detected error immediately stops the run.
       - When
       - Behavior
 
+    * - ``alwaysValidate < 0``
+      - Never
+      - Validation is disabled entirely; ``ValidateAllTransfers`` is never called.
+
     * - ``alwaysValidate = 0`` (default)
       - Once at the end of all iterations
       - ``ValidateAllTransfers`` called after the iteration loop.
 
-    * - ``alwaysValidate = 1``
+    * - ``alwaysValidate > 0``
       - After every timed iteration
       - ``ValidateAllTransfers`` called inside the loop; any error stops the run.
 
@@ -112,8 +117,9 @@ For each transfer and each DST, the following steps are performed:
 
 2. **Get the actual output:**
 
+   - **GPU destination** and ``validateOnDevice = 1``: Compare on the device using a validation kernel; no host copy is made.
    - **CPU destination** or ``validateDirect = 1``: Point directly at the destination memory.
-   - **GPU destination** and ``validateDirect = 0``: Copy destination to a host ``outputBuffer``
+   - **GPU destination**, ``validateOnDevice = 0``, and ``validateDirect = 0``: Copy destination to a host ``outputBuffer``
      via ``hipMemcpy``, then compare against ``outputBuffer``.
 
 3. **Comparison:** Performed using ``memcmp(output, expected, numBytes)``. On mismatch, the code finds the first differing index and returns an error with the index, expected value, and actual value.
@@ -135,7 +141,7 @@ environment variables or in a configuration file.
 
     * - ``alwaysValidate``
       - ``ALWAYS_VALIDATE``
-      - To validate after each iteration, set to ``1``. To validate once at the end, set to ``0``.
+      - To validate after each iteration, set to a value greater than ``0``. To validate once at the end (default), set to ``0``. To disable validation entirely, set to a negative value.
 
     * - ``validateDirect``
       - ``VALIDATE_DIRECT``

docs/index.rst

@@ -10,6 +10,10 @@ TransferBench is a utility for benchmarking simultaneous memory transfers betwee
 
 The source code is hosted at `<https://github.com/ROCm/TransferBench>`_.
 
+.. note::
+
+   TransferBench is part of the `ROCmValidationSuite (RVS) <https://rocm.docs.amd.com/projects/ROCmValidationSuite/en/latest/index.html>`_ and is installed with it.
+
 .. grid:: 2
   :gutter: 3
 

docs/install/build_from_source.rst

@@ -8,6 +8,10 @@
 Building TransferBench from source
 ===================================
 
+.. note::
+
+   TransferBench is part of the `ROCmValidationSuite (RVS) <https://rocm.docs.amd.com/projects/ROCmValidationSuite/en/latest/index.html>`_ and is installed with it.
+
 To build TransferBench from source, install the following required dependencies first:
 
 Required dependencies
@@ -347,6 +351,8 @@ To modify the CMake behavior, use the following environment variables:
   cmake .. -DENABLE_NIC_EXEC=ON -DENABLE_MPI_COMM=ON
   make
 
+.. _build-troubleshooting:
+
 Troubleshooting common build errors
 ====================================
 

docs/reference/environment-variables.rst

@@ -69,6 +69,18 @@ General options
       - Controls how many transfer sizes are sampled per power-of-2 range when ``num_bytes`` is set to ``0``. For example, ``SAMPLING_FACTOR=4`` produces four evenly spaced sizes between each power of 2.
       - ``1``
 
+    * - ``SHOW_PERCENTILES``
+      - Comma-separated percentiles of iteration duration to report. Each value must be between 1 and 99.
+      - —
+
+    * - ``SWEEP_MIN_POW2``
+      - When ``0`` is specified for data size, the starting power-of-two exponent.
+      - ``10``
+
+    * - ``SWEEP_MAX_POW2``
+      - When ``0`` is specified for data size, the ending power-of-two exponent.
+      - ``29``
+
 .. _data-validation-var:
 
 Data and validation options
@@ -225,6 +237,10 @@ GFX and kernel options
       - Preferred XCC per source-by-destination GPU pair, specified as a comma-separated list. Supported on AMD hardware only.
       - —
 
+    * - ``GFX_KERNEL``
+      - Selects the GFX kernel. ``-1`` = auto, ``0`` = force ``GpuReduceKernel``, ``1`` = force ``GpuCopyKernel`` (might error if the transfer is ineligible for this kernel).
+      - ``0``
+
 DMA options
 -----------
 
@@ -239,6 +255,28 @@ DMA options
       - DMA implementation. ``1`` = use ``hsa_amd_async_copy`` for direct HSA control, ``0`` = use ``hipMemcpy`` for the standard HIP path.
       - ``0``
 
+TDM options
+-----------
+
+.. list-table::
+    :header-rows: 1
+
+    * - Environment variable
+      - Description
+      - Default value
+
+    * - ``TDM_BLOCK_ORDER``
+      - How blocks for TDM transfers are ordered. ``0`` = sequential, ``1`` = interleaved.
+      - ``0``
+
+    * - ``TDM_BLOCK_SIZE``
+      - Threads per threadblock for TDM (async tensor) kernels. Must be a multiple of 32.
+      - ``256``
+
+    * - ``TDM_LDS_BYTES``
+      - LDS bytes per workgroup for TDM kernels. ``0`` = device max. Accepts ``K``, ``M``, and ``G`` suffixes.
+      - ``0``
+
 Variable SubExecutor options
 ------------------------------
 
@@ -293,6 +331,18 @@ The following environment variables apply only when NIC support is enabled. To e
       - RoCE version.
       - ``2``
 
+    * - ``NIC_CQ_POLL_BATCH``
+      - Number of completion queue entries to poll per ``ibv_poll_cq`` call.
+      - ``4``
+
+    * - ``NIC_SERVICE_LEVEL``
+      - InfiniBand service level (``sl``) for IB queue pairs. Must be between 0 and 15.
+      - ``0``
+
+    * - ``NIC_TRAFFIC_CLASS``
+      - DSCP / traffic-class byte for the RoCE GRH. Must be between 0 and 255.
+      - ``0``
+
 HIP runtime variables (client-read)
 -------------------------------------
 
@@ -338,6 +388,12 @@ Socket-related variables
     * - ``TB_SINGLE_LOG``
       - When set, only rank 0 produces output. Useful for multinode socket mode.
 
+    * - ``TB_MASTER_IFACE``
+      - When ``TB_MASTER_ADDR`` is unset on rank 0, the optional network interface used for IPv4 address detection (for example, ``eth0``).
+
+    * - ``TB_SEND_USLEEP``
+      - Microseconds to sleep after each socket ``SendData`` call. Default: ``0``. Useful for diagnosing socket stability.
+
 Backend variables
 -----------------
 
@@ -407,6 +463,12 @@ Backend variables
     * - ``TB_FORCE_SINGLE_POD``
       - Forces single pod mode, skipping AMD-SMI and NVML pod queries. This assumes that all GPUs are in the same pod and skips cluster membership API calls.
 
+    * - ``TB_SHOW_ALL_NUMA``
+      - Shows all CPU NUMA nodes, including those with no cores. By default, coreless nodes are hidden, which changes the reported CPU indices.
+
+    * - ``TB_WALLCLOCK_RATE``
+      - Overrides the queried GPU wallclock rate when the query returns ``0``.
+
 HSA runtime variables
 ---------------------
 

docs/reference/faq.rst

@@ -1,6 +1,6 @@
 .. meta::
   :description: Frequently asked questions about TransferBench, covering common errors, warnings, and configuration issues including IOMMU, memory types, and XGMI.
-  :keywords: TransferBench FAQ, TransferBench errors, TransferBench warnings, IOMMU, GPU_MAX_HW_QUEUES, GFX_UNROLL, validation, XGMI, UALoE, memory types
+  :keywords: TransferBench FAQ, TransferBench errors, TransferBench warnings, IOMMU, GPU_MAX_HW_QUEUES, GFX_UNROLL, validation, XGMI, UALoE, memory types, cmath, numactl, cpuset
 
 .. _faq:
 
@@ -15,6 +15,18 @@ Error and warning messages
 
 This section describes common TransferBench error and warning messages and how to resolve them.
 
+Could not find standard C++ header 'cmath'
+--------------------------------------------
+
+This occurs when the standard C++ headers aren't installed. Try installing ``g++-12`` or
+``g++-14`` based on the OS version. For example:
+
+.. code-block:: shell
+
+  apt-get install g++-12
+
+For more build troubleshooting, see :ref:`build-troubleshooting`.
+
 Unexpected mismatch at index
 -----------------------------
 
@@ -154,6 +166,57 @@ Setting ``GPU_MAX_HW_QUEUES=8``:
   ``GPU_MAX_HW_QUEUES=4`` case the CPU-timed aggregate bandwidth is roughly half the
   GPU-timed aggregate, while in the ``GPU_MAX_HW_QUEUES=8`` case the two are much closer.
 
+.. _cpuset-allocation-error:
+
+Cannot allocate memory due to process memory policy/cpuset
+-------------------------------------------------------------
+
+For example:
+
+.. code-block:: text
+
+  [ERROR] ERROR: Transfer 0: SRC 0: CPU 0 on rank 0 cannot allocate memory due to process memory policy/cpuset
+
+This indicates that the memory policy on the node doesn't allow CPU memory allocations on
+a particular NUMA node. Check the current policy:
+
+.. code-block:: shell
+
+  numactl --show
+
+.. code-block:: text
+
+  policy: default
+  preferred node: current
+  physcpubind: 0 1 2 3 4 5 6 7 8 9 10 ... 255
+  cpubind: 1       # <-- 0 is not listed
+  nodebind: 1      # <-- 0 is not listed
+  membind: 1       # <-- 0 is not listed
+  preferred:
+
+If the NUMA node that TransferBench is trying to allocate on isn't listed under
+``cpubind``, ``nodebind``, or ``membind``, the process is restricted from allocating memory
+there.
+
+This can also be caused by a hardware misconfiguration, or by a NUMA node having no RAM
+installed. Check installed memory per node with:
+
+.. code-block:: shell
+
+  numactl --hardware
+
+.. code-block:: text
+
+  available: 11 nodes (0-10)
+  node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 ... 191
+  node 0 size: 0 MB      # <-- No RAM installed on NUMA node 0
+  node 0 free: 0 MB
+  node 1 cpus: 64 65 66 67 68 69 70 71 72 73 74 ... 255
+  node 1 size: 257227 MB
+  node 1 free: 242799 MB
+
+A node with ``size: 0 MB`` has no RAM installed and can't be used for allocations.
+
 Feature questions
 ==================