Multipart uploads of large files produces partially corrupted data when upload chunk size

[Licenser]

LeoFS is configured with 5MB chunks.

Upload a file with multipart upload put into 1MB chunks

• Download the file with multipat download in 5MB chunks -> the file is corrupted
• Download as one file -> File is not corrupted

Upload a file with 5MB chunks.

• Download the file with multipat download in 5MB chunks -> File is fine
• Download as one file -> File is fine

Uploading:

[root@go-test ~/fifo_s3]# md5sum 4b6c9c1e-ab43-11e3-b6af-0799fb0203af
735b95a9640a67f7295a6665dbdee8d1  4b6c9c1e-ab43-11e3-b6af-0799fb0203af

[root@go-test ~/fifo_s3]# ./fifo_s3 -c 1 -b fifo-images -h ny4.storage.lucerahq.com put test 4b6c9c1e-ab43-11e3-b6af-0799fb0203af
Done

[root@go-test ~/fifo_s3]# ./fifo_s3 -s 1048576 -c 1 -b fifo-images -h ny4.storage.lucerahq.com put test-1M 4b6c9c1e-ab43-11e3-b6af-0799fb0203af
Done

Checking md5:

[root@go-test ~/fifo_s3]# ./fifo_s3 -s 1048576 -b fifo-images -h ny4.storage.lucerahq.com md5 test-1M
MD5 Sum: 5d05ec45a8788e940b3245bd5c196901

[root@go-test ~/fifo_s3]# ./fifo_s3 -b fifo-images -h ny4.storage.lucerahq.com md5 test-1M
MD5 Sum: 5d05ec45a8788e940b3245bd5c196901

[root@go-test ~/fifo_s3]# ./fifo_s3 -s 1048576 -b fifo-images -h ny4.storage.lucerahq.com md5 test
MD5 Sum: 735b95a9640a67f7295a6665dbdee8d1

[root@go-test ~/fifo_s3]# ./fifo_s3 -b fifo-images -h ny4.storage.lucerahq.com md5 test
MD5 Sum: 735b95a9640a67f7295a6665dbdee8d1

Chross check with s3cmd (S3cmd does not use multipart):

[root@go-test ~/fifo_s3]# s3cmd get s3://fifo-images/test --force
s3://fifo-images/test -> ./test  [1 of 1]
 227597244 of 227597244   100% in    2s    90.74 MB/s  done
WARNING: MD5 signatures do not match: computed=735b95a9640a67f7295a6665dbdee8d1, received="cc4525ef9f8e8fe1f7304f383adf3086"

[root@go-test ~/fifo_s3]# s3cmd get s3://fifo-images/test-1M --force
s3://fifo-images/test-1M -> ./test-1M  [1 of 1]
 227597244 of 227597244   100% in    1s   125.88 MB/s  done
WARNING: MD5 signatures do not match: computed=735b95a9640a67f7295a6665dbdee8d1, received="f0dea482c247937112a1890f8265c6a6"

[1] fifo_s3: https://github.com/project-fifo/fifo_s3

[Licenser]

A possible solution would be for LeoFS to refuse multipart uploads that do not match the chunk size.

[yosukehara]

Thank you for your report. We'll check this issue.

related issue: #177

[yosukehara]

We're planning to fix this issue with v1.2 later because we need to improve the internal architecture.
Please wait.

[windkit]

This issue is related to range get queries, issue #376 #382
Bugs are fixed on develop branch.

File MD5 (50M /dev/urandom)

/tmp/fifo_s3$ md5sum testfile
3cd55bbe40389beac4d493d3f91cd687  testfile

Multipart Upload (1MB Chunk Size)

/tmp/fifo_s3$ ./fifo_s3 -c 1 -s 1048576 -b test -h localhost -p 8443 put test-1M2 testfile
Done
/tmp/fifo_s3$ ./fifo_s3 -c 1 -s 5242880 -b test -h localhost -p 8443 put test-1M3 testfile
Done

/tmp/fifo_s3$ ./fifo_s3 -b test -h localhost -p 8443 md5 test-1M2
MD5 Sum: 3cd55bbe40389beac4d493d3f91cd687
/tmp/fifo_s3$ ./fifo_s3 -b test -h localhost -p 8443 md5 test-1M3
MD5 Sum: 3cd55bbe40389beac4d493d3f91cd687

OT:
Part Size should be at least 5MB according to Amazon S3 Documentation
http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html
But I guess it is no harm to support small part size

[shooding]

Hi, what is the maximum number of parts in a upload job?

[mocchira]

@shooding you can configure the max number of parts at leo_gateway.conf as below.
http://leo-project.net/leofs/docs/configuration/configuration_3.html?highlight=max_chunked_objs
So there are no limits.