Unable to access streaming counters with Vulkan?

[xoofx]

Hello there,

Calling GpaGetSupportedSampleTypes on my GPU (RX 7600 XT) with Vulkan only returns kGpaContextSampleTypeDiscreteCounter but not kGpaContextSampleTypeStreamingCounter, so I'm not able to access streaming counters like WaveOccupancyPct.

Is there anything special to deploy or to tweak in order to access such counters?

Edit: Looking at the code, it is indeed hardcoded to only support discrete:

gpu_performance_api/source/gpu_perf_api_vk/vk_gpa_context.cc

Line 29 in a0852bc

supported_sample_types_ = kGpaContextSampleTypeDiscreteCounter;

Seems removed here:

gpu_performance_api/source/public_counter_compiler/CounterCompiler.cs

Lines 1150 to 1161 in a0852bc

	if ("dx12" != api.ToLower())
	{
	// Non-DX12 only supports discrete counters. Enforce this.
	// Remove SPM-only counters if they happen to exist.
	publicCounterDerivedList.RemoveAll(x => (x.IsSpmCounter && !x.IsDiscreteCounter));
	// Force the counters to not indicate they are SPM counters. This simplifies a lot of logic later.
	foreach (var counter in publicCounterDerivedList)
	{
	counter.IsSpmCounter = false;
	}
	}

While the VK extensions seem to have a few stuffs related to spm:

gpu_performance_api/source/third_party/AmdVkExt/vk_amd_gpa_interface.h

Lines 137 to 146 in a0852bc

	typedef struct VkPhysicalDeviceGpaFeaturesAMD
	{
	VkStructureType sType;
	const void* pNext;
	VkBool32 perfCounters;
	VkBool32 streamingPerfCounters;
	VkBool32 sqThreadTracing;
	VkBool32 clockModes;
	} VkPhysicalDeviceGpaFeaturesAMD;

gpu_performance_api/source/third_party/AmdVkExt/vk_amd_gpa_interface.h

Lines 190 to 214 in a0852bc

	typedef struct VkGpaSampleBeginInfoAMD
	{
	VkStructureType sType;
	const void* pNext;
	VkGpaSampleTypeAMD sampleType;
	VkBool32 sampleInternalOperations;
	VkBool32 cacheFlushOnCounterCollection;
	VkBool32 sqShaderMaskEnable;
	VkGpaSqShaderStageFlagsAMD sqShaderMask;
	uint32_t perfCounterCount;
	const VkGpaPerfCounterAMD* pPerfCounters;
	uint32_t streamingPerfTraceSampleInterval;
	VkDeviceSize perfCounterDeviceMemoryLimit;
	VkBool32 sqThreadTraceEnable;
	VkBool32 sqThreadTraceSuppressInstructionTokens;
	VkDeviceSize sqThreadTraceDeviceMemoryLimit;
	VkPipelineStageFlagBits timingPreSample;
	VkPipelineStageFlagBits timingPostSample;
	} VkGpaSampleBeginInfoAMD;

So, what needs to be done to support streaming? Is it possible with the internal codebase by replicating what is done for d3d12 to Vulkan or it requires internal changes to some AMD drivers?

[PLohrmannAMD]

At this time GPUPerfAPI does not support streaming counters on Vulkan API. I would not recommend attempting to implement it, because we have also not released any information about how you can process the resulting buffer of data. GPUPerfAPI only supports Streaming counters on DX12 for the use of our Microsoft PIX plugin.

As noted in our README.md:

"GPUPerfAPI now includes additional entrypoints and enums to support querying streaming counters and SQTT data. These are only intended to support the AmdRadeonPlugin in Microsoft PIX on Windows. No support will be offered on how to use these entrypoints."

We can look into whether we can create an alternative equation for WaveOccupancyPct that works as a Discrete counter. The current equation only works for Streaming counters due to the way the underlying hardware counts those events.

Alternative Recommendation - Use our Radeon GPU Profiler to capture & analyze a frame of your Vulkan application. You'll get a very nice visual of the occupancy throughout your frame, plus a lot more insight than what GPUPerfAPI can provide!

[xoofx]

We can look into whether we can create an alternative equation for WaveOccupancyPct that works as a Discrete counter. The current equation only works for Streaming counters due to the way the underlying hardware counts those events.

I would be definitely interested in these counters being exposed (all the CS limiters...etc.). Hope that your team can find the time/priorities for this!

Alternative Recommendation - Use our Radeon GPU Profiler to capture & analyze a frame of your Vulkan application. You'll get a very nice visual of the occupancy throughout your frame, plus a lot more insight than what GPUPerfAPI can provide!

Thanks, yeah, I tried it, but unfortunately, it doesn't work as per the issue here: GPUOpen-Tools/radeon_gpu_profiler#111 (if you can help nudge someone at AMD, I would appreciate ☺️)

Nonetheless, using GPUPerfAPI, I was able to get all counters straight into my benchmark and I can iterate a lot more quickly:

It is particularly interesting in CI/CD scenarios as well.

[PLohrmannAMD]

I absolutely love that you've incorporated it into your benchmark and are using that information to iterate on your development! As a side comment, if you move to Radeon RX 9000 Series, be aware that the L1 counters do not exist. The L1 on that hardware is not actually a cache, so there are no Hits or Misses.

Noted on the usefulness of the CS Limiters counters as well. We'll see what we can do with them!

I've passed along your RGP bug report to the RGP lead. RGP 2.4 specifically made improvements for Compute-only Vulkan apps.
https://github.com/GPUOpen-Tools/radeon_gpu_profiler/releases/tag/v2.4
"3. Support for profiling non-frame-based pure compute DirectX® 12 and Vulkan® applications (requires a 24.30-based driver or newer)"