With the default ModeMask (0xCF, excluding modes 4/5) and -AlphaRestrict 1 set, a block containing hard alpha cutout pixels (values of exactly 0 or 255 alongside real alpha) can end up with a fully empty valid-mode set. When that happens, the encoder never writes that block's 16 output bytes at all and whatever was already in that memory (an uninitialized malloc'd buffer) ends up in the file.(cmp_mips.cpp:1095)
Reproduction
compressonatorcli -fd BC7 -EncodeWith CPU -Quality 0.5 -AlphaRestrict 1 <RGBA image with mixed 0/255 alpha content>
Root cause
The default ModeMask excludes modes 4/5 (the separate-alpha modes).
-AlphaRestrict 1 additionally excludes modes 6/7 for blocks matching blockNeedsAlpha && blockAlphaZeroOne (bc7_encode.cpp:1398–1431).
For a block matching that condition, the intersection of both restrictions is zero valid modes.
The mode-search loop (bc7_encode.cpp:1458) skips every mode via if (!(validModeMask & Mode)) continue;, never sets encodedBlock = TRUE, and the !encodedBlock fallback (:1521) returns without writing anything to that block's output bytes.
pOutBuffer points directly into the destination texture buffer (codec_bc7.cpp:634–635), which is allocated per-file via plain malloc (cmp_mips.cpp:1095) The unwritten bytes are uninitialized memory, not a deterministic fallback value.
There's a debug-only assert(validModeMask != 0) (:1433) that could catch this.
Suggested fix
Right now, this fails silently and non-deterministically. Any of the following would be a real improvement over that:
Promote the existing debug-only assert(validModeMask != 0) to something that also fires in release builds such as an stderr warning naming the affected block, without changing the output.
Write a deterministic, visually distinct sentinel for unencodable blocks (a fixed solid color, similar to how many engines flag missing textures) rather than leaving stale/uninitialized bytes.
Fall back to the unrestricted mode set for just that block. We've implemented and verified this specific approach in a downstream fork, although it does mean the restrict flags silently don't apply in that one case. Probably suboptimal.
Happy to share more detail on any of these, or open a PR for whichever direction you'd prefer.
With the default ModeMask (0xCF, excluding modes 4/5) and -AlphaRestrict 1 set, a block containing hard alpha cutout pixels (values of exactly 0 or 255 alongside real alpha) can end up with a fully empty valid-mode set. When that happens, the encoder never writes that block's 16 output bytes at all and whatever was already in that memory (an uninitialized malloc'd buffer) ends up in the file.(cmp_mips.cpp:1095)
Reproduction
compressonatorcli -fd BC7 -EncodeWith CPU -Quality 0.5 -AlphaRestrict 1 <RGBA image with mixed 0/255 alpha content>
Root cause
The default ModeMask excludes modes 4/5 (the separate-alpha modes).
-AlphaRestrict 1 additionally excludes modes 6/7 for blocks matching blockNeedsAlpha && blockAlphaZeroOne (bc7_encode.cpp:1398–1431).
For a block matching that condition, the intersection of both restrictions is zero valid modes.
The mode-search loop (bc7_encode.cpp:1458) skips every mode via if (!(validModeMask & Mode)) continue;, never sets encodedBlock = TRUE, and the !encodedBlock fallback (:1521) returns without writing anything to that block's output bytes.
pOutBuffer points directly into the destination texture buffer (codec_bc7.cpp:634–635), which is allocated per-file via plain malloc (cmp_mips.cpp:1095) The unwritten bytes are uninitialized memory, not a deterministic fallback value.
There's a debug-only assert(validModeMask != 0) (:1433) that could catch this.
Suggested fix
Right now, this fails silently and non-deterministically. Any of the following would be a real improvement over that:
Promote the existing debug-only assert(validModeMask != 0) to something that also fires in release builds such as an stderr warning naming the affected block, without changing the output.
Write a deterministic, visually distinct sentinel for unencodable blocks (a fixed solid color, similar to how many engines flag missing textures) rather than leaving stale/uninitialized bytes.
Fall back to the unrestricted mode set for just that block. We've implemented and verified this specific approach in a downstream fork, although it does mean the restrict flags silently don't apply in that one case. Probably suboptimal.
Happy to share more detail on any of these, or open a PR for whichever direction you'd prefer.