OffloadTest

Experimental Runtime test suite for HLSL

Current Status

Testing Machine	DXC	Clang
Tier 1 Targets
Windows DirectX12 Intel GPU
Windows DirectX12 NVIDIA GPU
Windows DirectX12 Warp (x64 LKG)
Windows DirectX12 Warp (arm64 LKG)
Windows Vulkan Intel GPU
Tier 2 Targets
macOS Metal Apple Silicon
Experimental Targets
Windows DirectX12 AMD GPU
Windows DirectX12 Qualcomm GPU
Windows Vulkan AMD GPU
Windows Vulkan NVIDIA GPU
Windows Vulkan Qualcomm GPU
Windows Vulkan Lavapipe (x64)
Windows Vulkan Lavapipe (arm64)

See the Continuous Integration documentation for the description of support tiers and builder hardware.

Prerequisites

Requires the Vulkan 1.4 SDK.

This project requires being able to locally build LLVM and leverages LLVM's build infrastructure. It also requires installing the pyyaml Python package. You can install pyyaml by running:

pip3 install pyyaml

On Windows, the Graphics Tools optional feature is additionally required to run the test suite.

Windows builds use the app-local DirectX 12 Agility SDK runtime from the Microsoft.Direct3D.D3D12 NuGet package. The default AGILITY_SDK_VERSION=LKG selects the repository's known-good version; System, Latest, or an explicit NuGet version may be selected instead. See Direct3D on Windows for details.

Building

The LLVM project provides a CMake cache file, clang/cmake/caches/HLSL.cmake, that configures the required projects and targets for HLSL development. You can use it with -C to set up a build that includes the offload test suite:

cmake -G Ninja -Bbuild \
-C <path to llvm-project>/clang/cmake/caches/HLSL.cmake \
-C <path to OffloadTest>/cmake/caches/OffloadTest.cmake \
<path to llvm-project>/llvm

The OffloadTest.cmake cache file automatically sets LLVM_EXTERNAL_OFFLOADTEST_SOURCE_DIR and LLVM_EXTERNAL_PROJECTS based on its location in the source tree. If you already have an LLVM build configured, you can add the offload test suite to it by passing the same -C flag or by adding the following to your CMake options:

-DLLVM_EXTERNAL_OFFLOADTEST_SOURCE_DIR=<path to OffloadTest> -DLLVM_EXTERNAL_PROJECTS="OffloadTest"

If you do not have a build of dxc on your path you'll need to specify the shader compiler to use by passing:

-DDXC_DIR=<path to folder containing dxc & dxv>

Running Tests

cmake --build build --target check-hlsl

The check-hlsl target builds all required tools and runs the full test suite. You can also run tests for a specific platform with check-hlsl-<platform> (e.g. check-hlsl-vk, check-hlsl-d3d12). To only run clang-based tests (without requiring DXC), use check-hlsl-clang-<platform> (e.g. check-hlsl-clang-mtl). Subdirectories of the test suite are also available as targets with check-hlsl-<platform>-<path> where the path is lowercased with directory separators replaced by - (e.g. check-hlsl-d3d12-feature-hlsllib).

Enabling clang-tidy

The offload test suite's code is clang-tidy clean for a limited ruleset. If you have clang-tidy installed locally you can enable clang-tidy by adding -DOFFLOADTEST_USE_CLANG_TIDY=On to your CMake invocation. You can also add -DOFFLOADTEST_CLANG_TIDY_APPLY_FIX=On to enable automatically applying the clang-tidy fix-its for any warnings that have automated fixes.

Failing tests

Tests which are failing can be prevented from running using XFAIL and UNSUPPORTED. When XFAILing a test make sure to add a comment above linking the appropriate issue and whether the failure is due to a bug or an unimplemented feature.

# Bug/Unimplemented <link to issue>
# XFAIL: Clang && Vulkan

YAML Pipeline Format

This framework provides a YAML representation for describing GPU pipelines and buffers. The format is implemented by the API/Pipeline.{h|cpp} sources. The following is an example pipeline YAML description:

---
Shaders:
- Stage: ComputeEntry: mainDispatchParameters:
DispatchGroupCount: [2, 1, 1] # Define how many groups to dispatch, if omitted one group is launched.Buffers:
- Name: ConstantsFormat: Int32Data: [ 1, 2, 3, 4, 5, 6, 7, 8]
- Name: In1Format: Float32Data: [ 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8]
- Name: In2Format: Hex16Data: [ 0x1, 0x2, 0x3, 0x4, 0x5, 0x6, 0x7, 0x8]
- Name: TexFormat: Float32Channels: 4OutputProps:
Width: 2Height: 2Depth: 1MipLevels: 2Data: [ 1.0, 0.0, 0.0, 1.0, # Mip 0 (2x2)0.0, 1.0, 0.0, 1.0,0.0, 0.0, 1.0, 1.0,1.0, 1.0, 1.0, 1.0,1.0, 1.0, 0.0, 1.0 ] # Mip 1 (1x1)
- Name: Out1 # Buffer where our output will goFormat: Float32Stride: 4FillSize: 8FillValue: 0.0# The FillValue is optional and defaults to zero
- Name: Expected1 # Buffer which stores the expected result of our testFormat: Float32Stride: 4Data: [ 0.0, 1.0 ]
- Name: Out2 # Buffer where our output will goFormat: Float16Stride: 2FillSize: 4# FillSize needs to be 4 bytes minimum
- Name: Expected2 # Buffer which stores the expected result of our testFormat: Float16Stride: 2Data: [ 0x1, 0x2 ]Results: # Using Result can verify test values without filecheck
- Result: Test1Rule: BufferFloatULP # Rule which can be used to compare Float Buffers; They are compared within a ULP rangeULPT: 1# ULP to useDenormMode: Any # if DenormMode Field is not Specified, 'Any' is the default; FTZ and Preserve are the other options.ZeroTolerance: 0.001# Optional absolute range for values near zero where ULP-based compares often fail due to tiny rounding errors.Actual: Out1 # First buffer to compareExpected: Expected1 # Second buffer to compare against first
- Result: Test2Rule: BufferExact # Compares Two Buffers for == equality between each value elementwiseActual: Out1Expected: Expected1
- Result: Test3Rule: BufferFloatEpsilon # Rule which can be used to compare Float Buffers; They are compared within an epsilon differenceEpsilon: 0.0008Actual: Out1Expected: Expected1DescriptorSets:
- Resources:
- Name: ConstantsKind: ConstantBufferDirectXBinding:
Register: 0# implies b0 due to Access being ConstantSpace: 0VulkanBinding:
Binding: 0# [[vk::binding(0, 0)]]
- Name: In1Kind: BufferDirectXBinding:
Register: 0# implies t0 due to Access being ROSpace: 0VulkanBinding:
Binding: 10
- Resources:
- Name: In2Kind: BufferDirectXBinding:
Register: 1# implies t1 due to Access being ROSpace: 0VulkanBinding:
Binding: 0# [[vk::binding(0, 1)]]
...