Last Chunk of Upload not getting split

[Anna-Rehm]

We have observed an upload error in which the last chunk of a file does not get split according to the resumableChunkSize logged in the Qiita logs. This only happens if the last "chunk" is bigger than 3MB (1 chunk) but smaller than 6MB (2 chunks). We were able to solve this by setting the nginx max client body size to 7MB, but are still confused as to why this would happen.

We wondered if you have encountered this problem before?

In the example below the file only gets divided into 2 chunks instead of 3.

[W 231206 16:56:06 web:2161] 400 GET /upload/?study_id=2&alias_sites=undefined&resumableChunkNumber=1&resumableChunkSize=3145728&resumableCurrentChunkSize=3145728&resumableTotalSize=9186788&resumableType=application%2Fgzip&resumableIdentifier=9186788-210skin_S177_L001_R2_001fastqgz&resumableFilename=210skin_S177_L001_R2_001.fastq.gz&resumableRelativePath=210skin_S177_L001_R2_001.fastq.gz&resumableTotalChunks=2 (::1) 9.66ms
[W 231206 16:56:06 web:2161] 400 GET /upload/?study_id=2&alias_sites=undefined&resumableChunkNumber=2&resumableChunkSize=3145728&resumableCurrentChunkSize=6041060&resumableTotalSize=9186788&resumableType=application%2Fgzip&resumableIdentifier=9186788-210skin_S177_L001_R2_001fastqgz&resumableFilename=210skin_S177_L001_R2_001.fastq.gz&resumableRelativePath=210skin_S177_L001_R2_001.fastq.gz&resumableTotalChunks=2 (::1) 12.61ms

[antgonza]

Short answer: no, we haven't seen before.

Background: the qiita system configuration uses chunks = 3*1024*1024 (~3M) and our smallest nginx entry is client_max_body_size 300M;. Note that this parameter is important and large because it will also determine the maximum reply size you will accept between plugins and the main system; which could be a bottleneck for large project or datasets.

Hope this helps.

[sjanssen2]

The same thing happens without using nginx, but it seems to be a known behaviour. From https://github.com/23/resumable.js/blob/master/README.md#how-do-i-set-it-up-with-my-server : Please note that the size of the data received in the HTTP might be higher than resumableChunkSize for the last chunk for a file.

Here is the discussion: 23/resumable.js#51
And a way to enforce chunk size: 23/resumable.js#56

Now that we know that chuck size can be max. 2 times the predefined one, I think we can leave code as is, but maybe add a hint to the documentation?

[sjanssen2]

alternatively, we could add the line forceChunkSize:true, here:

qiita/qiita_pet/static/vendor/js/resumable-uploader.js

Line 56 in ac62aba

prioritizeFirstAndLastChunk:false,

[antgonza]

Thank you @Anna-Rehm and @sjanssen2 !

[antgonza]

Closed by #3338