enc: clamp max_base64_regions to avoid allocation-size overflow

[adilburaksen]

Summary

BROTLI_ALLOC (c/enc/memory.h) computes N * sizeof(T) with no overflow
check. BROTLI_PARAM_MAX_BASE64_REGIONS (added in 037b70e, part of the new
BROTLI_PARAM_BASE64_MODE feature) sets params.max_base64_regions from an
unvalidated uint32_t with no upper bound. When that value is close to
SIZE_MAX / sizeof(Base64Region), the multiplication in HasherSetup's
base64_regions allocation (c/enc/hash.h) wraps, producing a buffer far
smaller than every subsequent num_base64_regions < params->max_base64_regions
guard (c/enc/backward_references_inc.h) assumes.

Verified with a 32-bit ASan build (sizeof(Base64Region) == 8 there):
max_base64_regions = 1u << 29 makes the allocation wrap to 0, and the first
detected Base64 region (a plain ;base64,-triggered match in the compression
input -- ordinary public-API input, not internal misuse) writes past the
resulting allocation.

Fix

Clamp max_base64_regions to the largest value that cannot overflow the
allocation, right before it's used in HasherSetup. Since every downstream
read goes through the same params pointer, this keeps the allocation size
and every write-guard consistent. Values above the clamp had no realistic
chance of a successful allocation anyway (multiple gigabytes), so this
doesn't change behavior for any value that could previously succeed -- it
routes the overflow case through the library's existing
BROTLI_IS_OOM/IS_NULL failure handling instead of wrapping silently.

Reachability

BROTLI_PARAM_BASE64_MODE defaults to disabled
(BROTLI_DEFAULT_BASE64_MODE = BROTLI_BASE64_MODE_DISABLED). The bundled
brotli CLI and the Python/Java/Go bindings in this repository don't expose
either new parameter. This is a defensive fix for the public C API contract
of a two-week-old experimental feature, not a demonstrated issue in any
current consumer.

Testing

• Built c/common/*.c + c/enc/*.c with gcc -m32 -fsanitize=address
  (32-bit, since the overflow needs a 32-bit size_t).
• Before this change: BrotliEncoderSetParameter(..., BROTLI_PARAM_BASE64_MODE, 1)
  • BrotliEncoderSetParameter(..., BROTLI_PARAM_MAX_BASE64_REGIONS, 1u << 29)
  • BrotliEncoderCompressStream on input containing a ;base64, region
    reproducibly triggers an ASan heap-buffer-overflow.
• After this change: the same input either compresses normally (small
  max_base64_regions after clamping still permits legitimate use) or fails
  cleanly via the existing OOM path for the deliberately-oversized case --
  no overflow, verified under the same ASan build.
• Re-ran with the default max_base64_regions (16) before and after the
  change: identical, unaffected output.

Happy to share the full PoC/ASan trace via g.co/vulnz or here, whichever the
maintainers prefer.