MSAA + mipmaps fails on D3D11/D3D12/Vulkan (bgfx)

[bghgary]

Summary

bgfx's D3D11, D3D12, and Vulkan backends fail to create render-target textures when both MSAA (samples > 1) and mipmaps (mips > 1) are requested. Symptom on D3D11 is E_INVALIDARG (0x80070057) from ID3D11Device::CreateTexture2D. The bug is in bgfx, but it surfaces in BabylonNative whenever JS code sets a non-1 sample count on an existing mipped render-target texture.

Root cause

D3D11 (and D3D12/Vulkan analogs) explicitly forbid MipLevels > 1 when SampleDesc.Count > 1. The MSAA target is rendered to with samples > 1 and MipLevels = 1; the resolved sample target gets MipLevels = N and SampleDesc.Count = 1. bgfx's D3D11 backend builds a single D3D11_TEXTURE2D_DESC and uses it for both targets — only resetting SampleDesc between them, not MipLevels. D3D12 (renderer_d3d12.cpp:5754 creating m_ptr from resourceDesc with both fields set) and Vulkan (renderer_vk.cpp:6450-6451 ici.samples + ici.mipLevels) have the same architectural issue. OpenGL/GLES/WebGL avoid the bug by using a separate non-mipped multisample renderbuffer for rendering and a mipped texture for sampling.

Reproduction

Minimal playground (run via Playground.exe --headless --once -- file:///path/to/repro.js):

const createScene = function () {
    const scene = new BABYLON.Scene(engine);
    const cam = new BABYLON.FreeCamera("cam", new BABYLON.Vector3(0, 0, -3), scene);
    cam.setTarget(BABYLON.Vector3.Zero());
    const box = BABYLON.MeshBuilder.CreateBox("box", { size: 1 }, scene);

    // 4th arg: generateMipMaps = true   <-- key to reproducing the bug
    const rtt = new BABYLON.RenderTargetTexture("mippedRtt", 256, scene, true);
    rtt.activeCamera = cam;
    rtt.renderList.push(box);
    scene.customRenderTargets.push(rtt);

    rtt.samples = 4;   // crashes with the BJS native MSAA-RTT impl from #18469 sans guard
    return scene;
};

Expected output on Win32 D3D11 Debug (using a BJS UMD where ThinNativeEngine.updateRenderTargetTextureSampleCount actually re-issues the texture):

BGFX FATAL 0x00000000 at .../bgfx/src/renderer_d3d11.cpp(4601):
    s_renderD3D11->m_device->CreateTexture2D(&desc, 0, &m_rt2d) FAILED 0x80070057
--- BN: ABORT ---
SIGABRT raised.

Stack frame: bgfx::d3d11::TextureD3D11::create → RendererContextD3D11::createTexture.

How it surfaced

Babylon.js PR #18469 replaced the previously-no-op ThinNativeEngine.updateRenderTargetTextureSampleCount with a real implementation that recreates the bgfx texture handle with the new MSAA flag. CI hit the FATAL during GLTF Texture Linear Interpolation Test, where scene.createDefaultEnvironment(...) creates a mipped IBL prefilter RTT and downstream PostProcess infrastructure calls setSamples(N) on it.

Affected backends

Backend	Status	Source location
D3D11	confirmed via local repro	renderer_d3d11.cpp:4516, 4601
D3D12	confirmed by code review	renderer_d3d12.cpp:5581, 5583, 5754
Vulkan	confirmed by code review	renderer_vk.cpp:6450-6451 (also VUID-VkImageCreateInfo-samples-02257)
Metal	likely (per Apple docs, mipmapLevelCount > 1 invalid with TextureType2DMultisample)	renderer_mtl.cpp:3393, 3440
OpenGL / GLES / WebGL	not affected (separate m_rbo renderbuffer for MSAA)	renderer_gl.cpp:5688+
WebGPU	not investigated