React Native Issue uploading smaller files

[paulsizer]

I have added this library to my ReactNative project. It works great for larger file sizes but trying to use it for smaller files doesn't seem to be working.

I get the following error when trying to upload smaller files:

UploadClientError: [400] Request does not contain files.

This works fine with 3-minute videos, I have no issues with it at all.

This is the code I am using to upload the files:

import UploadClient from '@uploadcare/upload-client'

const client = new UploadClient({ publicKey: 'XXX' })

const resp = await fetch(image.uri);
const blob = await resp.blob();
const uriParts = image.uri.split('.');
const ext = uriParts[uriParts.length - 1];
const filename = `file.${ext}`;

const uploadResponse = await client.uploadFile(blob, { fileName: filename });

Expo CLI 4.1.4 environment info:
System:
OS: macOS 10.15.7
Shell: 5.7.1 - /bin/zsh
Binaries:
Node: 12.16.3 - /usr/local/bin/node
Yarn: 1.22.4 - /usr/local/bin/yarn
npm: 6.14.4 - /usr/local/bin/npm
SDKs:
iOS SDK:
Platforms: iOS 14.4, DriverKit 20.2, macOS 11.1, tvOS 14.3, watchOS 7.2
IDEs:
Android Studio: 3.6 AI-192.7142.36.36.6392135
Xcode: 12.4/12D4e - /usr/bin/xcodebuild
npmPackages:
expo: ^40.0.1 => 40.0.1
react: 16.13.1 => 16.13.1
react-dom: 16.13.1 => 16.13.1
react-native: https://github.com/expo/react-native/archive/sdk-40.0.1.tar.gz => 0.63.2
react-native-web: ~0.13.12 => 0.13.14
npmGlobalPackages:
expo-cli: 4.1.4
Expo Workflow: managed

[paulsizer]

We have changed to using the direct upload API for smaller file uploads (smaller than 100mb).

We now need a way to upload files greater than 100mb. I have tried to use the Javascript library above but that now also appears to be blowing up in my ReactNative app. I have tried changing the maxConcurrentRequests to 1 thinking it might be a resource issue but that still hasn't fixed it.

[paulsizer]

I am now getting file size mismatch when calling complete on the multipart upload. Not sure if this is something to do with the way I do the chunking in my ReactNative (Expo) app.

const resp = await fetch(image.uri);
const blob = await resp.blob();
const uriParts = image.uri.split('.');
const ext = uriParts[uriParts.length - 1];
const filename = `file.${ext}`;

if (blob.size > 99000000) {
  multipartUpload(blob, filename);
} else {
  directUpload(image, filename);
}

const getChunk = (file, index, filesize, chunkSize) => {
  const start = chunkSize * index
  const end = Math.min(start + chunkSize, filesize)

  return file.slice(start, end)
}

const multipartStart = async(blob, filename) => {

  const body = new FormData
  body.append("UPLOADCARE_PUB_KEY", "XXX")
  body.append("filename", filename)
  body.append("size", blob.data.size)
  body.append("content_type", blob.type)
  body.append("UPLOADCARE_STORE", "auto")

  const response = await fetch("https://upload.uploadcare.com/multipart/start/?jsonerrors=1", {
    method: "POST",
    body,
    headers: {
      "Content-Type": "multipart/form-data"
    }
  })

  return response.json();

};

const uploadParts = async(blob, parts, uuid) => {
  const CHUNK_SIZE = 5 * 1024 * 1024;
  const chunks = [];
  for (let i = 0; i < parts.length; i++) {
    chunks.push(getChunk(blob, i, blob.data.size, CHUNK_SIZE));
  }
  console.log(chunks);
  let completed = 0;
  for (let i = 0; i < parts.length; i++) {
    let endpoint = parts[i];
    axios(endpoint, {
      method: 'PUT',
      headers: {
        'Content-Type': blob.type
      },
      data: chunks[i]
    })
    .then(res => {
      if (res.status == 200) {
        completed++;
      }
      console.log('Parts uploaded: ', completed);
      if (completed == parts.length) {
        multipartComplete(uuid);
      }
    })
    .catch(function (error) {
      console.log(error);
    });
  }

};

const multipartComplete = async(uuid) => {

  const body = new FormData
  body.append("UPLOADCARE_PUB_KEY", "XXX")
  body.append("uuid", uuid)

  const response = await fetch("https://upload.uploadcare.com/multipart/complete/?jsonerrors=1", {
    method: "POST",
    body,
    headers: {
      "Content-Type": "multipart/form-data"
    }
  })

  const result = await response.json();
  console.log(result);
};

[nd0ut]

I'm not quite understand why you're doing your own implementation?

Until we release a fixed version, you can import needed methods directly and use react-native file-like object ({ name, type, uri, size }). It will work for both direct and multipart uploads.

Example:

import defaultSettings from '@uploadcare/upload-client/lib/defaultSettings';
import uploadFile from '@uploadcare/upload-client/lib/uploadFile';
import uploadMultipart from '@uploadcare/upload-client/lib/uploadFile/uploadMultipart';

const uri = image.uri;
const resp = await fetch(uri);
const blob = await resp.blob();
const name = 'filename.ext';
const size = blob.size
const type = blob.type

const populateOptionsWithSettings = (options) => ({
  ...defaultSettings,
  ...options,
});

const options = { publicKey: 'demopublickey' }
const asset = { size, name, type, uri }

uploadFile(asset, populateOptionsWithSettings(options));
uploadMultipart(asset, populateOptionsWithSettings(options))

[paulsizer]

@nd0ut using that approach the call to getChuck within uploadMultipart doesn't like the file.

This is the error I get using your example above.

Error - TypeError: file.slice is not a function

[paulsizer]

@nd0ut trying your approach, passing in the blob and extra options seems to do what I want but I get issues like a mentioned previously.

The whole reason for me trying my own approach was that it seems to be hit and miss and the app just completely crashes during an upload, doesn't drop in the the catch block just dumps out.

Like I said above I didn't know if it was something to do with the concurrent requests, this is why I get a failure without anyway to catch the error.

const options = {publicKey: 'XXX', fileName: name, contentType: type};

    uploadMultipart(blob, populateOptionsWithSettings(options))
    .then(value => 
      console.log(`Success - ${value}`)
    )
    .catch(error => 
      console.log(`Error - ${error}`)
    )

[nd0ut]

@paulsizer

using that approach the call to getChuck within uploadMultipart doesn't like the file.

Yep, I was wrong, uploadMultipart requires a Blob, sorry.

I was able to reproduce both problems with size mismatch and crashes on large files, I'll take a look into it. Thanks!

[paulsizer]

@nd0ut

I'm glad you were able to reproduce the errors and it wasn't me doing something wrong my end. I have been banging my head against this for a while now!

Any timescales when you think the fix can be rolled out? We are looking at getting this in the app asap as we were doing our own transcoding and uploading on our own server but it makes no sense for us to do it. We are better offloading this onto your service.

[nd0ut]

@paulsizer
This is the first time I've encountered with react-native and I can't figure out what's causing these crashes. There is nothing that can highlight the cause - no errors in the system log or in the console. Since your simplified approach works without crashes, the cause is clearly somewhere in our codebase. We need a lot more time to find out the problem.

Now to the good stuff. I ran your code and it seems that axios doesn't work well in react-native and doesn't upload the Blob to the server, it's just sends an empty request instead, which resulted in a size mismatch. I replaced it with fetch and that's worked for me.

fetch(endpoint, {
  method: 'PUT',
  headers: {
    'Content-Type': 'image/jpeg'
  },
  body: chunks[i]
})

[nd0ut]

Ha, I turned off the queue and retry mechanics and the crashes are gone. I'm close to a clue.

[nd0ut]

It will not crash if you set maxConcurrentRequests to value greater than parts number, it will upload all the parts at once without concurrency. Still can't figure out why concurrent queue causes crashes at react-native.

[paulsizer]

@nd0ut What would be the best way to do this? Is there a way to set that on the options for that you pass into populateOptionsWithSettings?

[nd0ut]

Catched it.

facebook/react-native#27543

When any of sliced blobs get collected, react-native will deallocate the whole blob, which is leading to the app crash.

@paulsizer for now you can temporarily set maxConcurrentRequests to some high enough value like 100 anywhere you want. I'll prepare a PR with the fix soon.