Add method to read file into user given Uint8Array

Reading file contents into user supplied array avoids unnecessary copies. For example you can just get a pointer to an area in a WASM memory and directly read the file instead of first reading it into an useless ArrayBuffer and then doing a slow copy:

// file is a File or Blob
// wasm is a WebAssembly module instance wrapper with some additional methods like malloc added
const reader = new FileReader();
const ptr = wasm.malloc(file.size);
const array = new Uint8Array(wasm.memory.buffer, ptr, file.size);
try {
    let bytesRead = await reader.readIntoUint8Array(file, array);
    // Done
} catch (e) {
   // Error
}

It also allows reading files while using minimal memory and allocations:

const reader = new FileReader();
const bufferPtr = wasm.malloc(8192);
const buffer = new Uint8Array(wasm.memory.buffer, bufferPtr, 8192);
let totalBytesRead = 0;

while (totalBytesRead < file.size) {
    const start = totalBytesRead;
    const end = Math.min(start + 8192, file.size);
    const bytesRead = await readIntoUint8Array(file.slice(start, end), buffer);
    wasm.process(bufferPtr, bytesRead);
    totalBytesRead += bytesRead;
}

Doing the same with current APIs would allocate a ton of small ArrayBuffers...

This sounds very similar to what you would be able to do with a https://streams.spec.whatwg.org/#byob-reader-class right? I'm not necessarily against having similar functionality in FileReader as well, but I think eventually the streams API should be the one way to stream data from whatever source, so I'd be hesitant to add more functionality to FileReader. A somewhat sensible API to get a ReadableStream out of a Blob would seem like a much better solution to improving the ways you can read from blobs/files (note that you can already get a ReadableStream for a blob by wrapping the blob in a Response and calling its body method).

See also #40

Readable stream with byob sounds kinda like what I'm looking for but since it's not a specific API I fear it will still internally perform copy.

Until BYOB reader is supported by more browsers, this would be nice to have for use with SharedArrayBuffer.

For me it is not clear how byob-streams will solve this problem.
It is not possible to get a stream from a file isnt it? As far as I understand the streaming api (https://streams.spec.whatwg.org/) there is no plan on creating a stream from a file-input.
The FileReader still is the only option to create a buffer from the file-input. Even if you slice the file into chunks and stream it via a byob-stream, you still need to copy the contents of the FileReader-Buffer over to the buffer from the byob-stream.
It would be far easier, to just pass the byob-buffer to readAsArrayBuffer(file, buf).
The required changes for something like that should be very small!

Please enlighten me on this issue!

@Fuzzyma

@Fuzzyma #117 adds blobOrFile.stream() (among some nice-to-haves).

Fixed by #188 I'd say.

Navigation Menu

Repository navigation

Add method to read file into user given Uint8Array #83

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Add method to read file into user given Uint8Array #83

Description

Activity

mkruisselbrink commented on Jun 19, 2017

domenic commented on Jun 19, 2017

petkaantonov commented on Jun 20, 2017

thiccar commented on Oct 12, 2017

Fuzzyma commented on Apr 6, 2019

annevk commented on Apr 8, 2019

saschanaz commented on May 14, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions