Performance issue with static file server #304
Description
Our service uses the owin static file server (packages\Microsoft.Owin.StaticFiles.4.0.1\lib\net45\Microsoft.Owin.StaticFiles.dll) to serve static files.
We discovered that it took more than 15 seconds to retrieve a file of 3M size from our service in east US, and it took less than 2 seconds to get the same file in west US data center (The test client is located in west US.
From network capture, we found that the TCP traffic always stopped after 16438 bytes were sent, waiting for the ACK from the http client. So, basically, it took about 200 RTT to get the whole file. Wireshark indicates this is a TCP reassembly of application data. So it seems that the file server always waits for an ACK after 16438 bytes are sent.
It turns out that the 16k buffer size is the root cause of the performance issue.
The current code sends 16k data to the stream, then waits for the async call to finish before sending another 16k.
If I bumped the buffer size to 128k, and now it only took less than 2.5 seconds to retrieve the same file (It used to be more than 15 seconds.)
The static file server needs to make the buffer size configurable and/or auto-tuning based on file size/network bandwidth.