Skip to content

Fix BrotliCompressor.CopyToAsync buffer is not enough#249

Merged
hadashiA merged 3 commits intomainfrom
hadashiA/fix-brotli-encoder
Mar 15, 2024
Merged

Fix BrotliCompressor.CopyToAsync buffer is not enough#249
hadashiA merged 3 commits intomainfrom
hadashiA/fix-brotli-encoder

Conversation

@hadashiA
Copy link
Copy Markdown
Contributor

#243

BrotoliEncoder.Compress seems to return OperationState.DestinationTooSmall even when the destination buffer is short.

If the length of one BufferSegment is very large (more than the bufferSize parameter), finalState was DestinationTooSmall, the method would complete.
( In #243, MemoryStream is used).

@hadashiA
Copy link
Copy Markdown
Contributor Author

Here is the code to reproduce the issue.
#243 (comment)

I have written a slightly shortened version,

const string path1 = "/path/to/test.compressed";

var inputBytes = await File.ReadAllBytesAsync(path1);
System.Console.WriteLine($"Compressed file size (input) : {inputBytes.Length}");

var decompressor = new BrotliDecompressor();
var inputDecompressedBytes = decompressor.Decompress(inputBytes);
System.Console.WriteLine($"Decompressed size (input) : {inputDecompressedBytes.Length}");
var inputDeserialized = MemoryPackSerializer.Deserialize<TradeLog>(inputDecompressedBytes)!;
System.Console.WriteLine($"Decompressed data count (input) : {inputDeserialized.Data.Count}");
decompressor.Dispose();
System.Console.WriteLine("---------------");

var compressor = new BrotliCompressor(CompressionLevel.SmallestSize);
MemoryPackSerializer.Serialize(compressor, inputDeserialized);

// var buffer = new ArrayBufferWriter<byte>(inputBytes.Length);
// compressor.CopyTo(buffer);
var memoryStream = new MemoryStream(inputBytes.Length);
await compressor.CopyToAsync(memoryStream);
memoryStream.Flush();
memoryStream.Seek(0, SeekOrigin.Begin);
// System.Console.WriteLine($"Compressed size (serialized) : {buffer.WrittenSpan.Length}");
System.Console.WriteLine($"Compressed size (serialized) : {memoryStream.Length}");
compressor.Dispose();

var decompressor2 = new BrotliDecompressor();
var decompressedBytes = decompressor.Decompress(memoryStream.ToArray());
System.Console.WriteLine($"Decompressed size (serialized) : {decompressedBytes.Length}");
var deserialized = MemoryPackSerializer.Deserialize<TradeLog>(decompressedBytes)!;
System.Console.WriteLine($"Decompressed data count (serialized) : {deserialized.Data.Count}");
decompressor2.Dispose();

test.compressed data is available from the above comment.

}

// call BrotliEncoderOperation.Finish
var finalStatus = encoder.Compress(ReadOnlySpan<byte>.Empty, buffer, out var consumed, out var written, isFinalBlock: true);
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

スクリーンショット 2024-03-15 14 36 43

@hadashiA hadashiA merged commit f1fc4fc into main Mar 15, 2024
@hadashiA hadashiA deleted the hadashiA/fix-brotli-encoder branch March 15, 2024 06:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant