Adding 50Mb+ Files to a browser js-ipfs crashes Chrome

[ya7ya]

• Version: 0.25.1
• Platform: Ubuntu 16.04 4.4.0-42-generic , Chrome Version 57.0.2987.133 (64-bit) and Win 10 (64-bit) Chrome Version 60.0.3112.90 (Official Build) (64-bit)
• Subsystem: files.add

Type: Bug

Severity: Critical

Description: When trying out the exchange-files-in-browser example, uploading a file larger than 50MB crashes the browser, The Ram gradually starts filling up till it crashes.

Steps to reproduce the error:

• go to the exchange-files-in-browser example directory.
• run npm install
• npm start
• follow along with the README.md , but upload a large file, around 90MB.

Edit: Version fix.

[daviddias]

@ya7ya Thanks for the report!

I believe this issue to be related to the use of WebRTC and not necessarily with large files.

[daviddias]

@ya7ya wanna try disabling WebRTC and adding a large file?

[ya7ya]

@diasdavid hey, I'll try that now 👍

[ya7ya]

Hey @diasdavid , I tried to disable WebRTC and it still crashed, However I had an idea and it kinda worked.

instead of adding the file buffer using files.add , I used a createAddStream and chunked the file into a Readable stream.

readFileContents(file).then((buf) => {
      // create Readable Stream from buffer. requires 'stream-buffers'
      let myReadableStreamBuffer = new streamBuffers.ReadableStreamBuffer({
        chunkSize: 2048  // in bytes.
      })

      window.ipfs.files.createAddStream((err, stream) => {
        if (err) throw err

        stream.on('data', (file) => {
          console.log('FILE : ', file)
        })

        myReadableStreamBuffer.on('data', (chunk) => {
          myReadableStreamBuffer.resume()
        })

        // add ReadableStream to content
        stream.write({
          path: file.name,
          content: myReadableStreamBuffer
        })
       // add file buffer to the stream
        myReadableStreamBuffer.put(Buffer.from(buf))
        myReadableStreamBuffer.stop()

        myReadableStreamBuffer.on('end', () => {
          console.log('stream ended.')
          stream.end()
        })

        myReadableStreamBuffer.resume()
      })
    })

the browser didn't crash, but chrome got super resource hungry and ate up alotta ram. What I did is instead of adding the content as 1 big buffer, i chunked it into a stream to throttle it. it took alot longer but it didn't crash.

[daviddias]

That tells me that we need stress tests sooner than ever. Thank you @ya7ya!

[pyromaticx]

Having this same issue. I watched resources go to nearly 12GB of swap and 6GB of ram in about 20 minutes before killing the Chrome Helper process. Running js-ipfs in browser.

How could a 80mb file possibly cause all of this without some sort of severe memory leak occurring?