GZipStream regression - Much worse compression for sparse data when specifying CompressionLevel.Optimal

[SeanFeldman]

Expected behaviour

To produce identical compressed result for the same original byte array on .NET Framework and .NET Core.

Actual behaviour

Results of GZipStream on .NET Framework and .NET Core are different.

Repro

1. An array of 1500 characters 'A'
2. Compressed using GZipStream
3. Verified results on .NET Framework (4.6.1) and .NET Core (2.1) that should be identical, but they are not.

Code available here.

Results

Platform	Compressed result in hexa
.NET Framework 4.6.1	1f8b080000000000040073741c05a360148c825130dc00004d6f6cebdc050000
.NET Core 2.2.100 Commit b9f2fa0ca8 Host 2.2.0 Commit 1249f08fed	1f8b080000000000000b73741c05a321301a02a321301a028ec30c00004d6f6cebdc050000

Executed on Windows 10 Pro 64bit 1809 build 17763.134

[Wraith2]

Are they both valid and decompress to the correct data?
Is binary compatibility of the stream expected? It doesn't seem that doing so would be sensible overall because you'd never be able to improve compression performance if it were.

[SeanFeldman]

Are they both valid and decompress to the correct data?

Yes.

Is binary compatibility of the stream expected?

Yes. Sending side and receiving side could be implemented different frameworks, running on different platforms.

The question here is why using the same platform results are different. For instance, running a 1MB byte array with all values identical through compression on .NET Framework has yielded ~x5 smaller result that with .NET Core (2K vs 10K).

[BhaaLseN]

💭 Different CompressionLevel perhaps?
Could you try and see if specifying a compression level yields a consistent result? It would be kinda odd if they used different defaults, but there are both managed and ZLib implementations which may differ somewhere.

[SeanFeldman]

Thank you @BhaaLseN. CompressionLevel did help to identify what seems to be inconsistency in the implementation.

This is what I was able to find playing with different CompressionLevel options.

Platform	Tested version	CompressionLevel.Fastest	CompressionLevel.Optimal
Windows	.NET Framework	9218	2104
Windows	.NET Core	9218	9180
WSL (Ubuntu)	.NET Core	9218	2104

Note: to emphasize the difference, I've used 1MB (1024x1024) array filled with the same character, not the original 1500 size from the issue description.

[SeanFeldman]

I've also tested against Linux (Ubuntu 16.04.2 LTS) and it looks like .NET Core on Windows is the outlier.
@karelz should the issue be re-tagged as a bug instead of question?

[karelz]

I let area owners decide if it is a bug or question. Given that nothing seems to be wrong, I would not be surprised if we won't treat it as a bug.

[SeanFeldman]

Given that nothing seems to be wrong

Compression far from optimal for .NET Core on Windows when Optimal flag is used. Saying that, agree, let's hear from owners first 🙂 Thank you.

[stephentoub]

Given that nothing seems to be wrong, I would not be surprised if we won't treat it as a bug.

We don't expect/require that the compressed data will match across runtimes/versions, as long as it's valid. But there does appear to be a size regression here.

using System;
using System.IO;
using System.IO.Compression;

class Program
{
    static void Main()
    {
        byte[] data = new byte[1024*1024];
        for (int i = 0; i < data.Length; i++) data[i] = (byte)'a';

        Compress(data, CompressionLevel.NoCompression);
        Compress(data, CompressionLevel.Optimal);
        Compress(data, CompressionLevel.Fastest);
    }

    static void Compress(byte[] data, CompressionLevel level)
    {
        using (MemoryStream ms = new MemoryStream())
        {
            using (GZipStream gz = new GZipStream(ms, level, leaveOpen: true))
            {
                gz.Write(data, 0, data.Length);
            }
            Console.WriteLine(level + ":\t" + ms.Length);
        }
    }
}

results in the following for me on Windows:

C:\Users\stoub\Desktop\tmpapp>dotnet run -c Release -f net472
NoCompression: 1048759
Optimal: 1052
Fastest: 4609

C:\Users\stoub\Desktop\tmpapp>dotnet run -c Release -f netcoreapp3.0
NoCompression:  1048759
Optimal:        4590
Fastest:        4609

Note that the values for NoCompression and Fastest are the same, but the values for Optimal differ by 4x. Since Optimal is supposed to mean best compression ratio, there's obviously an issue when it's possible to be 4x smaller.

[MarcoRossignoli]

I spent some time here(I'm on Intel).
I tried to understand if it's a regression after PR dotnet/corefx#32732 so I reverted to before that commit (git reset --hard 32f43a75ff51ca633bb310ca3afe577c54a75d91) and tested with:

        [Theory]
        [InlineData(CompressionLevel.Fastest, 5000)]
        [InlineData(CompressionLevel.Optimal, 2000)]
        public void CompressRatio(CompressionLevel level, int lessThan)
        {
            byte[] data = new byte[1024 * 1024];
            for (int i = 0; i < data.Length; i++)
                data[i] = (byte)'a';

            using (MemoryStream ms = new MemoryStream())
            {
                using (GZipStream gz = new GZipStream(ms, level, leaveOpen: true))
                {
                    gz.Write(data, 0, data.Length);
                }
                Assert.True(ms.Length < lessThan, $"Compression level {level} result not less than {lessThan}, {ms.Length}");
                Console.WriteLine(ms.Length);
            }
        }

But it fail also before Compression level Optimal result not less than 2000, 4590

So I went deep in code and I found that there is a symbol defined here https://github.com/dotnet/corefx/blob/master/src/Native/Windows/clrcompression/zlib-intel/x86.h#L25 that affect level -> alg type relationship https://github.com/dotnet/corefx/blob/master/src/Native/Windows/clrcompression/zlib-intel/deflate.c#L134.
I cannot see which alg is used on .NET full https://referencesource.microsoft.com/#System/parent/parent/parent/InternalApis/NDP_FX/inc/ZLibNative.cs,261

However I found that USE_MEDIUM was added in version 1.2.11 intel/zlib@d6a0d5f#diff-6512e2889bbced8d558793c317fa04c4 but .NET full version seems oldest https://referencesource.microsoft.com/#System/parent/parent/parent/InternalApis/NDP_FX/inc/ZLibNative.cs,73 (1.2.3)
I tried to compile without USE_MEDIUM and test pass with deflate_slow alg
Maybe to be consistent with .NET full we need to remove #define USE_MEDIUM.
I hope it helps

/cc @vkvenkat

[joshudson]

This is one of those "Fix or Document" kind of cases. If CompressionLevel.Optimal is intended to be machine-specific, document as such.

[SeanFeldman]

@joshudson not quite sure what do you mean by "machine-specific" given verification tests running on the identical hardware configuration with different framework targets.

Also, not sure how badly performinng Optimal case can be considered an acceptable result.

[MarcoRossignoli]

/cc @ahsonkhan, @ViktorHofer https://github.com/dotnet/corefx/issues/34157#issuecomment-449645039

[joshfree]

Keeping bug active to track the secondary issue that @stephentoub identified with Optimal compression on the latest zlib (.NET Core version) being less-optimal than the older zlib in .NET Framework 4.x

[SeanFeldman]

@joshfree secondary issue? Are you referring to https://github.com/dotnet/corefx/issues/34157#issuecomment-449452688 or there's another issue raised?

[joshfree]

@SeanFeldman Yup. Though I was specifically referring to this subsection in @stephentoub 's comment:

results in the following for me on Windows:

C:\Users\stoub\Desktop\tmpapp>dotnet run -c Release -f net472
NoCompression: 1048759
Optimal: 1052
Fastest: 4609

C:\Users\stoub\Desktop\tmpapp>dotnet run -c Release -f netcoreapp3.0
NoCompression:  1048759
Optimal:        4590
Fastest:        4609

Note that the values for NoCompression and Fastest are the same, but the values for Optimal differ by 4x. Since Optimal is supposed to mean best compression ratio, there's obviously an issue when it's possible to be 4x smaller.