ogles2_vk: fix proc-addr ABI, SPIR-V endianness, varying linkage, VkFormat 37, VMA limits

[afxgroup]

AmigaOS4 Vulkan OGLES2 ICD — fixes for downstream rendering applications
Working through getting a real Vulkan application (f.e with Dear ImGui) running on top of ogles2_vk.library, I hit a series of issues in the driver. This is a write-up of the fixes.

1. vkGetDeviceProcAddr returned APICALL trampolines instead of raw functions

Bug. ogles2vk_LookupRawProcAddr() fell back to ogles2vk_LookupProcAddr() for every name not present in its small RAW(...) table. The fallback returns the t* APICALL trampolines that expect Self in r3 and the real arguments in r4..r10.

The loader explicitly documents (Loader_vkGetDeviceProcAddr, loader_dispatch.c) that the ICD's vkGetDeviceProcAddr is supposed to hand back raw, standard-Vulkan-ABI pointers because the application calls them directly as PFN_vkXxx. With the trampolines on that path, every argument slid by one register slot.

Most visible symptom: vkCreateSwapchainKHR(device, &createInfo, alloc, &swap) arrived inside the bare function as (device = createInfo, pCreateInfo = NULL/alloc, pSwapchain = garbage), so the NULL parameter check failed and the swapchain creation aborted.

Fix. Return NULL for unknown names in ogles2vk_LookupRawProcAddr() — same behaviour as software_vk.library. The application's loader wrapper can then fall back to its own dispatch through IVulkan with proper Self setup.

2. VkPhysicalDeviceLimits left mostly zero — VMA bailed with VK_ERROR_OUT_OF_DEVICE_MEMORY

Bug. _discover_gpu() zeroes g_physDevice and then only assigns a handful of limits (texture/render dimensions, descriptor counts, vertex inputs). Everything else — including bufferImageGranularity, nonCoherentAtomSize, min*Alignment, maxMemoryAllocationCount, maxSamplerAllocationCount, max{Uniform,Storage}BufferRange — stayed 0.

VMA reads bufferImageGranularity during its findMemoryTypeIndex/budget logic; align(x, 0) is undefined and in practice VMA preemptively returned VK_ERROR_OUT_OF_DEVICE_MEMORY without ever calling vkAllocateMemory. Same effect for nonCoherentAtomSize == 0.

Fix. Populate sane conservative defaults in _discover_gpu():

bufferImageGranularity = 1024
nonCoherentAtomSize = 64
minMemoryMapAlignment = 64
min{Uniform,Storage}BufferOffsetAlignment = 256
minTexelBufferOffsetAlignment = 16
optimalBufferCopy{Offset,RowPitch}Alignment = 4
maxMemoryAllocationCount = 4096
maxSamplerAllocationCount = 4096
maxStorageBufferRange = 2 GiB
maxUniformBufferRange = 64 KiB

Also bumped the reported heap sizes from 256 MB / 512 MB to 1 GiB / 1 GiB. VMA derives its preferredLargeHeapBlockSize from the heap size; with a 256 MB heap it kept choosing 32 MB blocks and was needlessly conservative.

3. SPIR-V endianness assumed to be little-endian

Bug. swvk_CreateShaderModule (and the parallel path in ogles2_vk reusing the same logic) byte-swapped every SPIR-V word unconditionally on the assumption that the stream is always LE. That breaks for callers that produce host-byte-order SPIR-V — e.g. ImGui's _glsl_shader*_spv arrays, which are uint32_t literals compiled by GCC on the PowerPC big-endian host. After the unconditional swap they became 0x03022307 and the magic check rejected them.

Fix. Detect endianness by reading the first four bytes of pCode:

03 02 23 07 → SPIR-V stream is LE, swap each word.
07 23 02 03 → already in host byte order, copy verbatim.

Anything else → invalid magic, return VK_ERROR_INITIALIZATION_FAILED.
The Vulkan spec explicitly allows either byte order, so this is the right behaviour.

4. SPIRV-Cross emitted varyings with stage-prefixed names → GLSL ES linker mismatch

Bug. Default SPIRV-Cross GLSL ES output for ImGui shaders produced:

// vertex
out struct { vec4 Color; vec2 UV; } Out;
Out.Color = aColor;

// fragment
in struct { vec4 Color; vec2 UV; } In;
fColor = In.Color * texture(sTexture, In.UV);

Warp3D Nova's GLSL ES linker matches varyings by name, so Out.Color and In.Color never connected and link failed:

W3DN_SI.library (0): Fragment shader has input In.Color, but the previous shader pipeline stage doesn't.
[ogles2_vk] Program link error
[ogles2_vk] Pipeline not ready, draw skipped

Fix. In ogles2vk_spirv2glsl.c:

Enable SPVC_COMPILER_OPTION_GLSL_FORCE_FLATTENED_IO_BLOCKS so the struct interface block is decomposed into individual varyings.
Before compiling, iterate STAGE_INPUT and STAGE_OUTPUT reflected resources and call spvc_compiler_set_name(compiler, id, "vary") on every struct-typed interface variable. SPIRV-Cross uses that name as the prefix for flattened fields, so both stages now emit matching vary_Color / vary_UV symbols and the linker is happy.

5. R8G8B8A8_UNORM rejected as a vertex attribute format

Bug. ogles2vk_FormatToAttrib() only handled R32_SFLOAT … R32G32B32A32_SFLOAT. ImGui packs its per-vertex colour as VK_FORMAT_R8G8B8A8_UNORM (= 37), so every ImGui draw call hit the default branch:

[ogles2_vk] Unsupported vertex format 37
with the attribute silently dropped — the UI received colour (0,0,0,0) and was rendered fully transparent black.

Fix. Extend FormatToAttrib to also return a normalized flag, add a case for VK_FORMAT_R8G8B8A8_UNORM → (size=4, type=GL_UNSIGNED_BYTE, normalized=GL_TRUE), and pass that flag through to glVertexAttribPointer instead of the hard-coded GL_FALSE. (Only this format and the float formats are exposed in the project's reduced vulkan_core.h; SNORM and 16-bit format cases were intentionally not added — they would not compile.)