DataStore.SetSegmented() Maximum segment size is not optimal for big files > 1 MB

[jianmingyong]

I have tested uploading a 2MB worth of junk data. At first, I used the default method it offers.

using (var fileStream = new FileStream(string.Format(DirectoryService.MusicContentDirectoryFormat, DirectoryService.ContentDirectory).ResolveFullPath() + "/Some2MBMusicFile.ogg", FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
    using (var memoryStream = new MemoryStream())
    {
        fileStream.CopyTo(memoryStream);
        var data = Convert.ToBase64String(memoryStream.ToArray());

        DataStore.SetSegmented("test", data, true, a => Debug.Log(a), b => Debug.Log(b));
    }
}

I edited the response error handler a little to get me a better error unity throws: (Unity really need to improve that lol) response.downloadHandler.text > Instead of response.error

It would still throw "413 (Request Entity Too Large)". However, now that I specify a smaller segment size, let say DataStore.SoftLimit / 2. It works!

The method still works for smaller data less than 1MB but it seems like the segment size isn't optimal enough to be used for bigger files without specifying a smaller default segment size.

So in summary, this soft limit could be improved further. If I can find an optimal value, I would let you know.

[jianmingyong]

Upon inspection of Unity3D API of uploading data...
LogHelper.Warning(System.Text.Encoding.UTF8.GetString(response.uploadHandler.data).Length);

Unfortunately they actually encode the data string. That encoding made it go past the limit. Sorry but guess you have to actually account for the encoding unity does this automatically.

[InfectedBytes]

Good catch! I'll update the api such that it will take the url encoding into account.

[jianmingyong]

The keys do have encoding too. A simple "Test|V2|Abc" would generate 15 bytes rather than 11. Max segment size would then fails to work again xD

Of course who would have made such keys but since I know some games do... This is rather important as well.

[jianmingyong]

"Datset is larger than 16MB!" - A small typo.

[InfectedBytes]

thanks, I've fixed this on a separate branch (SetSegmentedBugfix). If you like you could try my fix 😄

[jianmingyong]

oh, I will test it now.

[jianmingyong]

[Edit v2] Something fails... not sure which exact segment. Need more time.

[InfectedBytes]

that's strange, I'll tested it with several MB of randomly generated data. I'll look into it

[InfectedBytes]

I got it. I've moved the key parameter for the update operation into the POST-request payload in order to allow larger keys, but because the update operation needs one additional byte, the request is now one byte too large 😄

[jianmingyong]

oh lol.

[jianmingyong]

My test data is now working.