Integer overflow in maxRawSize calculation (rre.c, corre.c, zlib.c)

[afldl]

What I found

Three encoding modules independently compute the maximum raw buffer size with the same pattern:

int maxRawSize = (cl->scaledScreen->width * cl->scaledScreen->height
                  * (cl->format.bitsPerPixel / 8));

int is 32-bit on every platform I know of. When screen dimensions get large enough, this overflows. For example, 65535 × 65535 × 4 (32bpp) = ~17.2 billion, which wraps to a negative value in int.

The overflowed value then gets used in a comparison like cl->beforeEncBufSize < maxRawSize. When maxRawSize is negative and promoted to size_t for the comparison, it becomes a huge positive number, so the check always passes. The subsequent realloc(cl->beforeEncBuf, (size_t)maxRawSize) gets that huge value, returns NULL, and the encoding fails with a disconnect.

Where

All three files have the same code:

• libvncserver/rre.c:62-63
• libvncserver/corre.c:96-97
• libvncserver/zlib.c:66-67

When it triggers

By default, setDesktopSizeHook rejects client resize requests, so this is locked behind custom server configuration. But servers that call rfbScalingSetup with large scale factors, or custom servers that accept desktop size changes, would hit this.

To trigger it you need width × height × (bpp/8) > 2^31. With 32bpp that means roughly width > 23170 and height > 23170 (or any combination whose product exceeds ~537M pixels). Not unrealistic for remote desktop scenarios with high-resolution displays.

Fix

Use size_t and check for overflow before the multiply:

size_t maxRawSize;
size_t bpp_bytes = cl->format.bitsPerPixel / 8;
if (cl->scaledScreen->width > 0 &&
    cl->scaledScreen->height <= SIZE_MAX / bpp_bytes / cl->scaledScreen->width) {
    maxRawSize = (size_t)cl->scaledScreen->width *
                 cl->scaledScreen->height * bpp_bytes;
} else {
    return FALSE;
}

---

Found during security review of protocol parsing libraries.