Expose backend-agnostic 3D rendering, reusing the textured-triangle path every backend already implements #413

Tracking issue. Answers #412, which posed three options without picking one; this is option 2 — an in-frame rendering surface that keeps the backend abstraction — worked out far enough to schedule.

The observation that makes this small

IRendererBackend.RenderDrawData(ImDrawDataPtr) already takes vertex and index buffers with a texture id and a clip rect, and draws them. Every backend implements it:

• ImGuiController on OpenGL via Silk.NET
• MetalRendererBackend on iOS
• SoftwareRenderer in ImGui.App.Testing, which bottoms out at SoftwareRasterizer.FillTriangle(target, a, b, c, texture, scissor)

Every backend in this repository is already a textured-triangle rasterizer. The 2D-ness lives in exactly three places: the projection is orthographic, there is no depth buffer, and ImDrawVert.pos is a Vector2. Make those three things parameters and the same backend draws 3D.

That is the whole proposal. It is not a rendering engine, it is the existing draw path with a matrix and a depth buffer.

API

An optional extension interface, so no existing backend breaks and none is forced to implement it:

public interface IRenderer3D
{
    nint CreateRenderTarget(int width, int height, bool depth);
    bool ResizeRenderTarget(nint target, int width, int height);
    void DeleteRenderTarget(nint target);

    /// <summary>The colour attachment, usable as an ImGui texture id.</summary>
    nint GetTargetTexture(nint target);

    void Clear(nint target, Vector4 color, float depth);
    void Draw(nint target, ReadOnlySpan<Vertex3D> vertices, ReadOnlySpan<uint> indices, in DrawState3D state);
}

public readonly record struct Vertex3D(Vector3 Position, Vector2 Uv, uint Color);

public readonly record struct DrawState3D
{
    public Matrix4x4 ModelViewProjection { get; init; }
    public nint TextureId { get; init; }               // 0 = untextured, vertex colour only
    public DepthMode Depth { get; init; }              // None | Test | TestAndWrite
    public CullMode Cull { get; init; }                // None | Back | Front
    public BlendMode Blend { get; init; }              // Opaque | Alpha
    public PrimitiveTopology Topology { get; init; }   // TriangleList | LineList
}

Discovery, with no change to IRendererBackend:

public static bool TryGetRenderer3D(out IRenderer3D renderer);

A backend opts in by also implementing IRenderer3D. Callers that need 3D check once and fall back to their CPU raster path otherwise, which is what the existing CreateTexture/UpdateTexture route already gives them.

What is deliberately not in v1

Shaders, materials, lighting models, instancing, compute, MSAA, mipmaps, cubemaps, render-target formats other than RGBA8 + D24. That list is where a rendering abstraction goes to die, and every item on it forces the abstraction to grow a backend-specific escape hatch.

Vertex colour plus one optional texture plus a matrix covers what consumers have actually asked for: a globe, a model preview, a node-editor background, a gizmo.

No lighting, and therefore no vertex normal. The caller bakes shading into Vertex3D.Color. This is the one call most likely to be argued with, so the reasoning: a caller who wants real lighting wants shaders, and shaders are the thing that cannot be made backend-agnostic without inventing a shading language. Per-vertex Lambert on the CPU is a few lines and costs nothing at these vertex counts. If it proves insufficient, a single directional light in DrawState3D — direction, colour, ambient, against a normal added to Vertex3D — is the smallest next step and can be added without breaking the v1 surface.

Two contract hazards worth designing around now

Texture id namespace collision. GetTargetTexture returns something usable as an ImGui texture id, which on OpenGL is a texture name drawn from the same namespace as CreateTexture. A caller who passes it to DeleteTexture would free a live render target's colour attachment out from under it. Render targets must be released through DeleteRenderTarget only, and DeleteTexture should reject a handle it did not issue rather than silently corrupting state. The same applies in reverse.

Resize thrash. A viewport sized to ImGui.GetContentRegionAvail() changes size every frame while a splitter is being dragged, and reallocating GPU storage per frame is the obvious way to make this feature look slow. Round the target size up to a granularity (64 px) and resize on a change of bucket, not a change of pixel. That belongs in the widget rather than the backend.

Why the software backend is a first-class deliverable, not polish

This repository has six headless UI suites, a one-class-per-widget isolation rule for ImGui.Widgets, and a WidgetTest base offering Snapshot / PixelsChangedSince / BoundsOfDifference. All of it rests on ImGui.App.Testing's SoftwareRenderer.

A 3D API that only works on OpenGL would be the first thing in the suite that cannot be rendered headlessly — no Viewport3D isolation test could exist, and the pixel-difference helpers would have nothing to compare. So SoftwareRenderer implementing IRenderer3D is what makes the feature testable at all, and it is not speculative work: the CPU rasterizer already fills textured triangles, and needs a depth buffer and perspective-correct interpolation added to it.

Sub-issues

1. IRenderer3D, Vertex3D, DrawState3D, TryGetRenderer3D, and the OpenGL implementation
2. SoftwareRenderer implementation — depth buffer and perspective-correct interpolation in SoftwareRasterizer
3. MetalRendererBackend implementation
4. ImGuiWidgets.Viewport3D — the widget that owns a target, sizes it to the content region, and handles orbit / pan / zoom

Relationship to existing work

• An application under ImGuiApp.Start() has no way to issue its own GL, so a custom 3D view must be CPU-rasterized and uploaded #412 is the problem statement this answers; it should be read first for why BeginExternalFrameSession is not the hook.
• Separate ImGui context lifecycle from GL rendering in ImGuiController #313 (separating ImGui context lifecycle from GL rendering in ImGuiController) is adjacent and probably wants to land first, since sub-issue 1 adds a second pipeline to that same class.