MoltenVK 1.2.11 breaks vkdoom

[DUOLabs333]

This used to be a comment under #2146, but I thought it would be better as its own issue:

1.2.11 (up to 78396b7) breaks vkdoom, regardless of whether Metal argument buffers are used or not --- the error messages are different though (vkdoom worked on 1.2.9 with MVK_CONFIG_USE_METAL_ARGUMENT_BUFFERS=1). From using Actions artifacts, I have determined that the problem existed since at least 96f9d89:
MVK_CONFIG_USE_METAL_ARGUMENT_BUFFERS=0:

[mvk-error] VK_ERROR_INITIALIZATION_FAILED: Shader library compile failed (Error code 3):
Compilation failed: 

program_source:429:12: warning: unused variable 'ray_end'
    float3 ray_end = origin + (dir * tmax);
           ^
program_source:587:464: error: cannot reserve 'texture' resource locations at index 0
fragment main0_out main0(main0_in in [[stage_in]], device SurfaceIndexBuffer& _608 [[buffer(1)]], device SurfaceBuffer& _604 [[buffer(2)]], device LightBuffer& _941 [[buffer(3)]], device LightIndexBuffer& _945 [[buffer(4)]], device PortalBuffer& _696 [[buffer(5)]], device ConstantsBuffer& _904 [[buffer(6)]], device NodeBuffer& _131 [[buffer(7)]], device VertexBuffer& _285 [[buffer(8)]], device ElementBuffer& _289 [[buffer(9)]], array<texture2d<float>, 16536> textures [[texture(0)]], array<sampler, 16536> texturesSmplr [[sampler(0)]])
                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ^
program_source:587:511: error: cannot reserve 'sampler' resource locations at index 0
fragment main0_out main0(main0_in in [[stage_in]], device SurfaceIndexBuffer& _608 [[buffer(1)]], device SurfaceBuffer& _604 [[buffer(2)]], device LightBuffer& _941 [[buffer(3)]], device LightIndexBuffer& _945 [[buffer(4)]], device PortalBuffer& _696 [[buffer(5)]], device ConstantsBuffer& _904 [[buffer(6)]], device NodeBuffer& _131 [[buffer(7)]], device VertexBuffer& _285 [[buffer(8)]], device ElementBuffer& _289 [[buffer(9)]], array<texture2d<float>, 16536> textures [[texture(0)]], array<sampler, 16536> texturesSmplr [[sampler(0)]])
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              ^
.
[mvk-error] VK_ERROR_INVALID_SHADER_NV: Fragment shader function could not be compiled into pipeline. See previous logged error.

MVK_CONFIG_USE_METAL_ARGUMENT_BUFFERS=1:

[mvk-error] SPIR-V to MSL conversion error: Argument buffer resource base type could not be determined. When padding argument buffer elements, all descriptor set resources must be supplied with a base type by the app.
[mvk-error] VK_ERROR_INVALID_SHADER_NV: Fragment shader function could not be compiled into pipeline. See previous logged error.

[DUOLabs333]

However, to be fair, vkdoom at no point passes validation, so the fact it worked before was probably a lucky accident.

The validation errors:

VUID-VkPipelineLayoutCreateInfo-descriptorType-03022(ERROR / SPEC): msgNum: -658548338 - Validation Error: [ VUID-VkPipelineLayoutCreateInfo-descriptorType-03022 ] | MessageID = 0xd8bf598e | vkCreatePipelineLayout():  max per-stage sampler bindings count (16536) exceeds device maxPerStageDescriptorUpdateAfterBindSamplers limit (1024). The Vulkan spec states: The total number of descriptors with a descriptorType of VK_DESCRIPTOR_TYPE_SAMPLER and VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER accessible to any given shader stage across all elements of pSetLayouts must be less than or equal to VkPhysicalDeviceDescriptorIndexingProperties::maxPerStageDescriptorUpdateAfterBindSamplers (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-VkPipelineLayoutCreateInfo-descriptorType-03022)
    Objects: 0
VUID-VkPipelineLayoutCreateInfo-pSetLayouts-03036(ERROR / SPEC): msgNum: -662943472 - Validation Error: [ VUID-VkPipelineLayoutCreateInfo-pSetLayouts-03036 ] | MessageID = 0xd87c4910 | vkCreatePipelineLayout():  sum of sampler bindings among all stages (16536) exceeds device maxDescriptorSetUpdateAfterBindSamplers limit (1024). The Vulkan spec states: The total number of descriptors of the type VK_DESCRIPTOR_TYPE_SAMPLER and VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER accessible across all shader stages and across all elements of pSetLayouts must be less than or equal to VkPhysicalDeviceDescriptorIndexingProperties::maxDescriptorSetUpdateAfterBindSamplers (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-VkPipelineLayoutCreateInfo-pSetLayouts-03036)
    Objects: 0
VUID-VkPipelineLayoutCreateInfo-descriptorType-03022(ERROR / SPEC): msgNum: -658548338 - Validation Error: [ VUID-VkPipelineLayoutCreateInfo-descriptorType-03022 ] | MessageID = 0xd8bf598e | vkCreatePipelineLayout():  max per-stage sampler bindings count (16536) exceeds device maxPerStageDescriptorUpdateAfterBindSamplers limit (1024). The Vulkan spec states: The total number of descriptors with a descriptorType of VK_DESCRIPTOR_TYPE_SAMPLER and VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER accessible to any given shader stage across all elements of pSetLayouts must be less than or equal to VkPhysicalDeviceDescriptorIndexingProperties::maxPerStageDescriptorUpdateAfterBindSamplers (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-VkPipelineLayoutCreateInfo-descriptorType-03022)
    Objects: 0
VUID-VkPipelineLayoutCreateInfo-pSetLayouts-03036(ERROR / SPEC): msgNum: -662943472 - Validation Error: [ VUID-VkPipelineLayoutCreateInfo-pSetLayouts-03036 ] | MessageID = 0xd87c4910 | vkCreatePipelineLayout():  sum of sampler bindings among all stages (16536) exceeds device maxDescriptorSetUpdateAfterBindSamplers limit (1024). The Vulkan spec states: The total number of descriptors of the type VK_DESCRIPTOR_TYPE_SAMPLER and VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER accessible across all shader stages and across all elements of pSetLayouts must be less than or equal to VkPhysicalDeviceDescriptorIndexingProperties::maxDescriptorSetUpdateAfterBindSamplers (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-VkPipelineLayoutCreateInfo-pSetLayouts-03036)
    Objects: 0
VUID-VkPipelineLayoutCreateInfo-descriptorType-03022(ERROR / SPEC): msgNum: -658548338 - Validation Error: [ VUID-VkPipelineLayoutCreateInfo-descriptorType-03022 ] | MessageID = 0xd8bf598e | vkCreatePipelineLayout():  max per-stage sampler bindings count (16539) exceeds device maxPerStageDescriptorUpdateAfterBindSamplers limit (1024). The Vulkan spec states: The total number of descriptors with a descriptorType of VK_DESCRIPTOR_TYPE_SAMPLER and VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER accessible to any given shader stage across all elements of pSetLayouts must be less than or equal to VkPhysicalDeviceDescriptorIndexingProperties::maxPerStageDescriptorUpdateAfterBindSamplers (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-VkPipelineLayoutCreateInfo-descriptorType-03022)
    Objects: 0
VUID-VkPipelineLayoutCreateInfo-pSetLayouts-03036(ERROR / SPEC): msgNum: -662943472 - Validation Error: [ VUID-VkPipelineLayoutCreateInfo-pSetLayouts-03036 ] | MessageID = 0xd87c4910 | vkCreatePipelineLayout():  sum of sampler bindings among all stages (16539) exceeds device maxDescriptorSetUpdateAfterBindSamplers limit (1024). The Vulkan spec states: The total number of descriptors of the type VK_DESCRIPTOR_TYPE_SAMPLER and VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER accessible across all shader stages and across all elements of pSetLayouts must be less than or equal to VkPhysicalDeviceDescriptorIndexingProperties::maxDescriptorSetUpdateAfterBindSamplers (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-VkPipelineLayoutCreateInfo-pSetLayouts-03036)

[billhollings]

Please run vkDoom with the following environment variables set:

MVK_CONFIG_DEBUG=1
MVK_CONFIG_LOG_LEVEL=4
MVK_CONFIG_USE_METAL_ARGUMENT_BUFFERS=1

This will log the shaders, descriptor sets, and errors.

Please ZIP up that log into a file, and post it here.

---

Also, to try to replicate this here, we'll need help getting vkDoom set up here to test it.

A Google search leads to the following repos:

https://github.com/dpjudas/VkDoom
https://github.com/ZDoom/zdoom-macos-deps

Is this correct? Or is vkDoom found somewhere else?

Is a macOS prebuilt app somewhere? The VkDoom repo mentions nightly builds, but there are no macOS binaries archived there.

If these are the correct repos, we'll need some more detail on how to work with this:

• The deps repo talks about identifying targets for the build, but it's not clear (even from the build.py help text) which targets need to be built.

• What should be done with the built deps from the deps repo? How do they integrate into the VkDoom repo when it is built?

• After it is built, how do we run VkDoom?

• At runtime, where in VkDoom is the error identified in this issue occurring? How can the error be replicated here?

[DUOLabs333]

• Once you have cloned zdoom-macos-deps, you should be able to build with ./build.py --target vkdoom. This requires a lot of space (~3 GB), and it takes some time. This should build vkdoom (ie, cloning vkDoom separately shouldn't be required).

• There are (unfortunately) no prebuilt binaries for macOS available (maybe you should be able to run the Windows binaries under Wine, but I haven't tried this, and may introduce a confounding variable).

• The error prevents vkDoom from starting, so at runtime, just before the title screen loads, you should get an error message about a pipeline shader failing to compile.

[billhollings]

Thanks. I've got it built now, but hitting:

Cannot find a game IWAD (doom.wad, doom2.wad, heretic.wad, etc.).

Did I miss an install step? I can't find a WAD in the repo dir. Where can we get one?

[DUOLabs333]

Oh, right forgot about that --- usually you can get them off of the Internet Archive, but they're down right now. I'm not at my computer right now, so I'll send the WAD to you when I'm in front of it.

[billhollings]

NM. I sourced on online. I'm able to replicate the error now.

[DUOLabs333]

One thing I have noticed is that on 1.2.9, the structs in the converted MSL used packed_floats, while on 1.2.11, it's just float.