Skip to content

Do not read ISUNCOMPRESSED after a metadata meta-block - #1518

Open
mohass1927 wants to merge 1 commit into
google:masterfrom
mohass1927:fix/metadata-metablock-phantom-isuncompressed-bit
Open

Do not read ISUNCOMPRESSED after a metadata meta-block#1518
mohass1927 wants to merge 1 commit into
google:masterfrom
mohass1927:fix/metadata-metablock-phantom-isuncompressed-bit

Conversation

@mohass1927

Copy link
Copy Markdown

decodeMetaBlockLength reads the ISUNCOMPRESSED bit after a metadata meta-block, which the format does not contain. RFC 7932 §9.2 places that bit only in the non-metadata branch, and the C decoder agrees: BROTLI_STATE_METABLOCK_HEADER_METADATA increments meta_block_remaining_len and returns, so it never reaches BROTLI_STATE_METABLOCK_HEADER_UNCOMPRESSED.

The Java metadata branch returns early only when MSKIPBYTES == 0. With MSKIPBYTES >= 1 it falls through to the header tail shared with the non-metadata branch and reads the bit.

Both branches then run if (isMetadata || isUncompressed) jumpToByteBoundary(), which is why this is usually invisible — when the metadata header ends unaligned, the jump absorbs the extra bit and both decoders land on the same byte. It stops being invisible when the header ends exactly on a byte boundary: C jumps zero bits, Java has already consumed one and jumps seven more, so Java is a full byte ahead for the rest of the stream and reads every later meta-block header from a different offset.

Because a meta-block header starts wherever the previous one ended, that alignment is reachable, and a stream can be arranged so the subsequent headers are individually valid under both offsets. Then the two decoders produce different output from the same bytes and both report success rather than one of them erroring.

What this changes

The missing early return, applied to Decode.java and to each copy transpiled from it — kt/Decode.kt, go/brotli/decode.go, js/decode.ts, js/decode.js and csharp/org/brotli/dec/Decode.cs. The generated files are edited directly since they are checked in; the edit is mechanically identical in each. js/decode.min.js is left alone as a minified bundle that is not in js/package.json's files list — regenerating it is better done by your pipeline. Same for the C# copy if you would rather re-run csharp/transpile.sh; mono/nuget were not available here.

Test

SynthTest.testMetadataEndingOnByteBoundary and the Go mirror TestMetadataEndingOnByteBoundary, both on a 14-byte stream whose metadata header ends at bit 96:

bit  0      WBITS, window 16
bit  1-81   compressed meta-block, MLEN 4, simple literal code {'A','B'}, emits "ABAB"
bit 82-95   metadata header: ISLAST 0, MNIBBLES 3, reserved 0, MSKIPBYTES 1, MSKIPLEN byte 0x00
bit 96      header ends, 96 % 8 == 0
byte 12     the skipped metadata byte
byte 13     final empty meta-block

The existing metadata coverage in testPeculiarWrap does not catch this: it uses MSKIPBYTES == 1, so it does reach the offending line, but its header ends unaligned so the jump hides the bit.

Verified locally:

before after
SynthTest (Java, 47 tests) 1 failure, testMetadataEndingOnByteBoundary, error code -4 OK (47 tests)
go test ./go/brotli synth --- FAIL: TestMetadataEndingOnByteBoundary ok
reference C on the same 14 bytes 4 bytes, ABAB unchanged

On a longer crafted stream the Java decoder returned 8453 bytes where the C decoder returned 5, both reporting success; after the change both return the same 5 bytes (4142414209). DecodeTest, DictionaryTest, EagerStreamTest and CompoundDictionaryTest pass. BitReaderTest and SetDictionaryTest each have one failure that reproduces identically on unpatched master in this environment, and go test ./go/brotli cannot build at master because brotli_test.go references cbrotli.NewPreparedDictionary and cbrotli.WriterOptions.Dictionary, which the resolved cbrotli does not export — both unrelated to this change.

RFC 7932 section 9.2 places the ISUNCOMPRESSED bit only in the non-metadata
branch of the meta-block header. The C decoder matches the format: the
BROTLI_STATE_METABLOCK_HEADER_METADATA state increments
meta_block_remaining_len and returns, so it never reaches
BROTLI_STATE_METABLOCK_HEADER_UNCOMPRESSED and never reads the bit.

decodeMetaBlockLength in the Java decoder, and in the ports transpiled from
it, returns early for a metadata meta-block only when MSKIPBYTES is 0. With
MSKIPBYTES >= 1 it falls through to the header tail shared with the
non-metadata branch and reads ISUNCOMPRESSED anyway.

Usually this is invisible, because both branches then jump to the next byte
boundary and that jump absorbs the extra bit. It is not invisible when the
metadata header ends exactly on a byte boundary: the C decoder jumps zero
bits while the Java decoder has already consumed one and jumps seven more,
so it is a whole byte ahead for the remainder of the stream and reads every
later meta-block header from a different offset. A stream can be built so
that the headers are valid under both offsets, in which case the two
decoders return different output and both report success.

Add the missing early return to Decode.java and to the transpiled Java,
Kotlin, Go, TypeScript, JavaScript and C# copies, and add a synth test whose
metadata header ends on a byte boundary. Before this change the new test
fails in Java with error code -4 and in Go; the existing metadata coverage
in testPeculiarWrap does not catch it because that header ends unaligned.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant