Failed to read stream from HttpClient.Get with option HttpCompletionOption.ResponseHeadersRead

[jiabiao]

The following code works fine under full framework, but get exception under .net core 1.0 and 2.0.

var res = new HttpClient().GetAsync("https://github.com/adamhathcock/sharpcompress/raw/master/tests/TestArchives/Archives/Tar.tar",
	HttpCompletionOption.ResponseHeadersRead).Result;
var stream = res.Content.ReadAsStreamAsync().Result;
var reader = ReaderFactory.Open(stream);
reader.MoveToNextEntry();
reader.WriteEntryTo(new MemoryStream());  //System.IO.IOException: 'Unable to read data from the transport connection. The connection was closed before all data could be read. Expected 105472 bytes, read 512 bytes.'

Changing the HttpClient.Get option to HttpCompletionOption.ResponseContentRead also works. But for large files, it can not start decompressing until the entire file is downloaded.

[adamhathcock]

Worked for me

    class Program
    {
        static void Main(string[] args)
        {
            Test().Wait();
        }

        public static async Task Test()
        {
            using(var client = new HttpClient())
            using (var response = await client.GetAsync("https://github.com/adamhathcock/sharpcompress/raw/master/tests/TestArchives/Archives/Tar.tar",
                    HttpCompletionOption.ResponseHeadersRead))
            {
                var stream = await response.Content.ReadAsStreamAsync();
                using (var reader = ReaderFactory.Open(stream))
                {
                    reader.MoveToNextEntry();
                    reader.WriteEntryTo(new MemoryStream());  //System
                }
            }
        }
    }

Always use async/await and using statements please.

[adamhathcock]

This netcoreapp1.1 haven't tried 2.0

[jiabiao]

Weird, your code throw the same exception on my end.

I did use async/await and using statements in my actual code. The code in the original post is simplified for demonstration purposes.

[adamhathcock]

Weird.

Just in case I put my code here: https://github.com/adamhathcock/sc-http-test

[jiabiao]

git clone & dotnet run still get the same exception.
I will debug with source to see what happened.

D:\Code\netcore>git clone https://github.com/adamhathcock/sc-http-test.git
Cloning into 'sc-http-test'...
remote: Counting objects: 6, done.
remote: Compressing objects: 100% (6/6), done.
Unpacking objects: 100% (6/6), done.
remote: Total 6 (delta 0), reused 6 (delta 0), pack-reused 0
Checking connectivity... done.

D:\Code\netcore>cd sc-http-test

D:\Code\netcore\sc-http-test>dotnet restore
  Restoring packages for D:\Code\netcore\sc-http-test\test.csproj...
  Generating MSBuild file D:\Code\netcore\sc-http-test\obj\test.csproj.nuget.g.props.
  Generating MSBuild file D:\Code\netcore\sc-http-test\obj\test.csproj.nuget.g.targets.
  Writing lock file to disk. Path: D:\Code\netcore\sc-http-test\obj\project.assets.json
  Restore completed in 1.46 sec for D:\Code\netcore\sc-http-test\test.csproj.

  NuGet Config files used:
      C:\Users\jiabiao\AppData\Roaming\NuGet\NuGet.Config
      C:\Program Files (x86)\NuGet\Config\Microsoft.VisualStudio.Offline.config

  Feeds used:
      https://api.nuget.org/v3/index.json
      C:\Program Files (x86)\Microsoft SDKs\NuGetPackages\

D:\Code\netcore\sc-http-test>dotnet run

Unhandled Exception: System.AggregateException: One or more errors occurred. (Unable to read data from the transport connection. The connection was closed before all data could be read. Expected 105472 bytes, read 512 bytes.) ---> System.IO.IOException: Unable to read data from the transport connection. The connection was closed before all data could be read. Expected 105472 bytes, read 512 bytes.
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at System.Net.Http.WinHttpResponseStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Http.DelegatingStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at SharpCompress.IO.RewindableStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at SharpCompress.IO.RewindableStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at SharpCompress.Common.Tar.TarReadOnlySubStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at SharpCompress.Common.EntryStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at SharpCompress.Common.EntryStream.SkipEntry()
   at SharpCompress.Common.EntryStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at SharpCompress.Readers.AbstractReader`2.Write(Stream writeStream)
   at SharpCompress.Readers.AbstractReader`2.WriteEntryTo(Stream writableStream)
   at test.Program.<Test>d__1.MoveNext() in D:\Code\netcore\sc-http-test\Program.cs:line 26
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at test.Program.Main(String[] args) in D:\Code\netcore\sc-http-test\Program.cs:line 13

D:\Code\netcore\sc-http-test>

[jiabiao]

I created an issue https://github.com/dotnet/corefx/issues/20676.
Reading 0 byte from network stream will cause this exception.
So if the count parameter is 0, the RewindableStream.Read method should return 0 directly.

        public override int Read(byte[] buffer, int offset, int count)
        {
            if (count == 0)
            {
                return 0;
            }

[adamhathcock]

I don't get how I didn't get the exception but you did. Just happened to read zero bytes and I didn't?

[jiabiao]

This issue occurs only on windows, Are you using macOS or Linux?
Otherwise, I don't know why you didn't get the exception. You can add a conditional breakpoints at the beginning of RewindableStream.Read method to investigate.

[adamhathcock]

You're right. I get this on Windows but not macOS

[adamhathcock]

https://www.nuget.org/packages/sharpcompress/0.17.1

[jiabiao]

thanks.