Do not read ISUNCOMPRESSED after a metadata meta-block - #1518
Open
mohass1927 wants to merge 1 commit into
Open
Conversation
RFC 7932 section 9.2 places the ISUNCOMPRESSED bit only in the non-metadata branch of the meta-block header. The C decoder matches the format: the BROTLI_STATE_METABLOCK_HEADER_METADATA state increments meta_block_remaining_len and returns, so it never reaches BROTLI_STATE_METABLOCK_HEADER_UNCOMPRESSED and never reads the bit. decodeMetaBlockLength in the Java decoder, and in the ports transpiled from it, returns early for a metadata meta-block only when MSKIPBYTES is 0. With MSKIPBYTES >= 1 it falls through to the header tail shared with the non-metadata branch and reads ISUNCOMPRESSED anyway. Usually this is invisible, because both branches then jump to the next byte boundary and that jump absorbs the extra bit. It is not invisible when the metadata header ends exactly on a byte boundary: the C decoder jumps zero bits while the Java decoder has already consumed one and jumps seven more, so it is a whole byte ahead for the remainder of the stream and reads every later meta-block header from a different offset. A stream can be built so that the headers are valid under both offsets, in which case the two decoders return different output and both report success. Add the missing early return to Decode.java and to the transpiled Java, Kotlin, Go, TypeScript, JavaScript and C# copies, and add a synth test whose metadata header ends on a byte boundary. Before this change the new test fails in Java with error code -4 and in Go; the existing metadata coverage in testPeculiarWrap does not catch it because that header ends unaligned.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
decodeMetaBlockLengthreads theISUNCOMPRESSEDbit after a metadata meta-block, which the format does not contain. RFC 7932 §9.2 places that bit only in the non-metadata branch, and the C decoder agrees:BROTLI_STATE_METABLOCK_HEADER_METADATAincrementsmeta_block_remaining_lenand returns, so it never reachesBROTLI_STATE_METABLOCK_HEADER_UNCOMPRESSED.The Java metadata branch returns early only when
MSKIPBYTES == 0. WithMSKIPBYTES >= 1it falls through to the header tail shared with the non-metadata branch and reads the bit.Both branches then run
if (isMetadata || isUncompressed) jumpToByteBoundary(), which is why this is usually invisible — when the metadata header ends unaligned, the jump absorbs the extra bit and both decoders land on the same byte. It stops being invisible when the header ends exactly on a byte boundary: C jumps zero bits, Java has already consumed one and jumps seven more, so Java is a full byte ahead for the rest of the stream and reads every later meta-block header from a different offset.Because a meta-block header starts wherever the previous one ended, that alignment is reachable, and a stream can be arranged so the subsequent headers are individually valid under both offsets. Then the two decoders produce different output from the same bytes and both report success rather than one of them erroring.
What this changes
The missing early return, applied to
Decode.javaand to each copy transpiled from it —kt/Decode.kt,go/brotli/decode.go,js/decode.ts,js/decode.jsandcsharp/org/brotli/dec/Decode.cs. The generated files are edited directly since they are checked in; the edit is mechanically identical in each.js/decode.min.jsis left alone as a minified bundle that is not injs/package.json'sfileslist — regenerating it is better done by your pipeline. Same for the C# copy if you would rather re-runcsharp/transpile.sh; mono/nuget were not available here.Test
SynthTest.testMetadataEndingOnByteBoundaryand the Go mirrorTestMetadataEndingOnByteBoundary, both on a 14-byte stream whose metadata header ends at bit 96:The existing metadata coverage in
testPeculiarWrapdoes not catch this: it usesMSKIPBYTES == 1, so it does reach the offending line, but its header ends unaligned so the jump hides the bit.Verified locally:
SynthTest(Java, 47 tests)testMetadataEndingOnByteBoundary, error code -4go test ./go/brotlisynth--- FAIL: TestMetadataEndingOnByteBoundaryABABOn a longer crafted stream the Java decoder returned 8453 bytes where the C decoder returned 5, both reporting success; after the change both return the same 5 bytes (
4142414209).DecodeTest,DictionaryTest,EagerStreamTestandCompoundDictionaryTestpass.BitReaderTestandSetDictionaryTesteach have one failure that reproduces identically on unpatchedmasterin this environment, andgo test ./go/brotlicannot build atmasterbecausebrotli_test.goreferencescbrotli.NewPreparedDictionaryandcbrotli.WriterOptions.Dictionary, which the resolvedcbrotlidoes not export — both unrelated to this change.