System.IO.Packaging part stream has a memory leak when writing a large stream

[twsouthwick]

This was found through the DocumentFormat.OpenXML library which uses System.IO.Packaging extensively (original issue: dotnet/Open-XML-SDK#244). The issue logged there is trying to generate a large Excel document which uses a working set of around 10mb on .NET 4.7, while it grows quite quickly until hitting a OutOfMemoryException. I've simplified the issue to remove the dependency on DocumentFormat.OpenXML and it appears to be isolated to writing to a Part within a Package.

Source

using System;
using System.IO;
using System.IO.Packaging;

namespace MemoryRepro
{
    class Program
    {
        static void Main(string[] args)
        {
            using (var fs = new FileStream(Path.GetTempFileName(), FileMode.Create, FileAccess.ReadWrite))
            using (var package = Package.Open(fs, FileMode.Create))
            {
                var part = package.CreatePart(new Uri("/part", UriKind.Relative), "something/sometype");

                using (var stream = part.GetStream())
                using (var writer = new StreamWriter(stream))
                {
                    for (var i = 0; i < int.MaxValue; i++)
                    {
                        writer.Write("hello");
                    }
                }
            }
        }
    }
}

Project

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net47</TargetFramework>
    <!--<TargetFramework>netcoreapp2.0</TargetFramework>-->
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="System.IO.Packaging" Version="4.4.0" />
  </ItemGroup>

</Project>

This repro code appears to have a working set of around 60mb running on .NET 4.7, while it grows very quickly on .NET Core 2.0

The error on .NET Core 2.0 is:

Unhandled Exception: System.IO.IOException: Stream was too long.
   at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.Compression.WrappedStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.IO.StreamWriter.Write(String value)
   at MemoryRepro.Program.Main(String[] args) in c:\users\tasou\source\repos\MemoryRepro\MemoryRepro\Program.cs:line 21

[twsouthwick]

I did some more investigating, and it's only seen on .NET Core because the .NET 4.6 build just contains type forwarders to the System.IO.Packaging classes in WindowsBase.

It appears that the reason the memory is growing is due to the ZipArchiveMode passed into the ZipArchive. The following is what the packaging library is doing (simplified to show the cause):

static void Main(string[] args)
{
    // Causes memory leak
    var mode = ZipArchiveMode.Update;
    // Does not leak
    //var mode = ZipArchiveMode.Create;

    using (var fs = new FileStream(Path.GetTempFileName(), FileMode.Create, FileAccess.ReadWrite))
    using (var archive = new ZipArchive(fs, mode, true, System.Text.Encoding.UTF8))
    {
        var entry = archive.CreateEntry("entry", CompressionLevel.NoCompression);

        using (var stream = entry.Open())
        using (var writer = new StreamWriter(stream))
        {
            for (var i = 0; i < int.MaxValue; i++)
            {
                writer.Write("hello");
            }
        }
    }
}

The Package creates the ZipArchive here: https://source.dot.net/#System.IO.Packaging/System/IO/Packaging/ZipPackage.cs,349.

[twsouthwick]

Looks like there are a few other open issues with performance issues for ZipArchive. @ianhays Any thoughts?

[stephentoub]

The following is what the packaging library is doing (simplified to show the cause)

@twsouthwick, it's doing the exact same thing on both .NET Framework and .NET Core?

[twsouthwick]

The distinction between .NET Framework and .NET Core was a red herring; the .NET Framework build was using a System.IO.Packaging that redirects to the implementation in WindowsBase. When I ran the code in corefx manually against .NET Framework, it repros the same. So, yes, it is doing the same thing on both .NET Framework and .NET Core.

[stephentoub]

So, yes, it is doing the same thing on both .NET Framework and .NET Core.

I'm a little confused :) What is doing the same thing? Is ZipArchive behaving the same between .NET Framework and .NET Core, but System.IO.Packaging is behaving differently between the two? Or is the issue you cited with Packaging also reproing with .NET Framework?

[twsouthwick]

Sorry about that.

• ZipArchvie is behaving the same between .NET Framework and .NET Core. The ZipArchive snippet is used in the .NET Core implementation of System.IO.Packaging; I don't know what is used in the .NET Framework implementation since it's in WindowsBase.
• System.IO.Packaging is behaving differently between .NET Framework and .NET Core. On .NET Framework, the original code snippet works; I can create a package and write an arbitrarily large number of items to it. On .NET Core, the original snippet will crash due to memory usage.

[stephentoub]

Thanks, @twsouthwick. I took a quick look. It looks like Package in WindowsBase uses its own internal ZipArchive that has streaming support.

[twsouthwick]

That would explain it. Are there plans to add streaming support to ZipArchive in CoreFX?

[ianhays]

There's an issue open for it that's had some good discussion but hasn't moved forward due to it being a lower priority addition.

[cmerat]

This has caused different behavior when using the OpenXml SDK library between NetFX and Core. It makes it impossible to create huge spreadsheets without busting memory limits. The implications are rather big as the library can no longer be used if the dataset is large enough in .NET Core.

[twsouthwick]

@ianhays Is there a timeline as to when a design may be available? What can be done to help move that forward? This issue is blocking for people who generate large office documents on the fly and it would be nice to know what steps need to be taken.

[ianhays]

There is not a timeline currently. More "likes" on the issue always helps as does discussion of the API on the issue and comments explaining use cases.

It's pretty high up on the priority list as far as compression investments go.

[LarinLive]

IGNOREME: I am commenting here so I can easily find this issue again in the future, since I can't do that by subscribing alone.

[igorko]

Voting for this to be fixed.