(release/25.1) rootless: Keep the screen pixmap header consistent with its allocation

[metux]

RootlessUpdateScreenPixmap freed the old backing store before attempting the replacement allocation and returned early
on failure, leaving the screen pixmap's devPrivate.ptr pointing at freed memory.

The published geometry was also only refreshed when the buffer grew, so a screen that shrinks kept advertising its
previous, larger size.

Signed-off-by: Jeremy Huddleston Sequoia jeremyhu@apple.com

[metux]

🤖 Automated review — generated by Starfleet ship Voyager on behalf of @metux. Not a human review.

Two real bugs fixed, both correct:

1. Use-after-free (critical): Old code called free(s->pixmap_data) before attempting calloc, returning early on failure — leaving devPrivate.ptr pointing at freed memory. The fix allocates the new buffer first (malloc), only frees the old buffer after success, and returns early before the free on failure. The pixmap stays valid with its existing (still-live) backing store.

2. Stale geometry on shrink: ModifyPixmapHeader was only called inside the growth branch, so a screen that shrunk kept advertising its previous, larger size. Moving it outside the branch means geometry is republished on every call.

Also adds a null-pixmap guard after CreatePixmap failure — the old code would dereference NULL.

The devKind = 0 assignment and its comment are well-explained: it makes every row alias row 0, keeping a one-scanline allocation vertically in bounds. The ModifyPixmapHeader call with zero devKind leaves the field untouched, so the explicit assignment after is load-bearing, not cosmetic.

ABI: RootlessUpdateScreenPixmap is internal to rootless. Not exported, not referenced by any NVIDIA blob. ABI-safe.

Looks good.