TextEncoder#encode - write to existing Uint8Array

[maciejhirsz]

For binary protocols, such as msgpack, performance of writing/reading strings can be crucial, however those strings rarely (virtually never) exist on a binary buffer on their own. When decoding a string from a binary buffer of Uint8Array (obtained over, say, websockets) it's possible to cheaply create a subarray and pass it over to TextDecoder#decode, as such the decoding story is pretty straight forward.

The writing story is much more complicated.

In order to write a string into an existing buffer, one must take the result Uint8Array from TextEncoder#encode and TypedArray#set it at a desired offset into another buffer, which introduces unnecessary copying and, as result, is slower than plain JavaScript encoding implementations (at least for UTF-8).

A much better solution would be analog to the Node.js Buffer#write implementation where the encoder can write a string to a buffer at a desired offset. Example signature could be something like:

TextEncoder#write(DOMString source, Uint8Array target [, Number offset [, Number length]]) -> Number bytesWritten

[annevk]

Thank you for reporting this, I've heard a similar request aired for fetch() at some point and there might be other APIs where folks want buffer reuse. I wonder if we should adopt some kind of pattern for this.

@domenic? @inexorabletash?

[inexorabletash]

Yep, legit request we've heard from potential users before. I don't know what the pattern should be like, or if it should be done at this level (new/extended method) or if we should add Stream support and there should be a generic stream-into-buffer mechanism.

With the write() proposal above, we'd want to return the amount of data consumed (since the encoder won't buffer arbitrary amounts itself) and amount written. That also sounds like a common element of this sort of pattern. (And also the kind of thing streams are good at, so hoping there's a generic solution.)

[domenic]

Right, if we get transform streams for text encoding, then you'd be able to use a BYOB reader to target specific segments of your destination buffer.

A lower-level method might make sense. If so, don't have separate offset/length parameters; those are already part of the Uint8Array (which is a { byteOffset, byteLength, buffer } tuple).

[shaunc]

To my mind writing to/reading from a specified position in an existing ArrayBuffer would be a better interface. Alternately DataView.getString/setString methods could be added in conjunction with this interface.

[hsivonen]

Adding this feature would be problematic, because it would expose to the Web the encoder's behavior when approaching the end of the buffer.

Considering the experience with encoding_rs, it's very useful not to try to fill the output buffer exactly but to

1. Never let a byte sequence representing a single code point be split across output buffer boundaries
2. Be allowed to signal that the output buffer is full if the worst-case output length for a code point (4 bytes in the case of UTF-8) doesn't fit regardless of what the next input code point is.

Specifying one particular behavior is likely to lead to problems with encoding back ends that don't exhibit the specified behavior already. encoding_rs doesn't even have a single behavior: the ASCII fast path is more willing to write near the end of the buffer while the handling for non-ASCII want to see output space for the worst case (an astral character).

Allowing implementation-defined behavior, on the other hand, could easily be an interop problem if Web devs test with an implementation that e.g. is willing to convert a final BMP code point when there are 3 bytes of output space remaining but another implementation wants to see 4 bytes of space even if the remaining input would be a BMP code point.

[hsivonen]

I believe we could avoid exposing implementation details (while making the feature less useful) by specifying the the conversion fails immediately if the available output space in bytes is not greater or equal to the number of input UTF-16 code units times three plus one.

In other words if we didn't support incremental encode but required the worst-case (assuming an implementation that wants to see worst-case available space on a per input code point basis, hence the plus one) output space and always consumed the entire input string if it didn't throw right away.

[hsivonen]

I've thought about this some more. If we are doing the bring your own ArrayBuffer thing only for UTF-8 encode, it's not too onerous to wrap encoding_rs's UTF-8 encoder with something that performs slow but exact buffer filling for the last couple of bytes.

I'd be unhappy to have to do that for all encodings, though, so let's be careful if we ever get a request for decoding to ArrayBuffers.

In any case, the exact buffer filling behavior in the last 3 bytes of the output buffer is an area that is going to need special interop attention to avoid compatibility problems in cases where Web devs misuse a streaming API as a non-streaming API.

[ricea]

Strawman API proposal: encodeInto(view, string, offset = 0). Example use:

let offset = 0;
while (offset < string.length) {
  offset = encoder.encodeInto(view, string, offset);
  if (offset < string.length) {
    view = getANewViewSomehow();
  }
}

encodeInto() would always write the maximum number of code points that it can without overflowing view. Incomplete utf-8 sequences would not be written. Unused bytes at the end of view would not be touched.

As @hsivonen noted, implementations would probably fall back to a slower algorithm when they get close to the end of the output buffer.

[fitzgen]

This would be very useful for getting JS strings into and out of webassembly memory.

[domenic]

@ricea given that a view is already a { buffer, byteOffset, byteLength } tuple, I would think the argument should just be a view, not a (view, offset) tuple.

[ricea]

@domenic Sorry, I should have made it clear. offset is an offset into the string, not into the view. It would be inefficient for callers to chop the front of the string after every call, so instead we include the ability to ignore offset code units.

Argument summary:
view: a BufferSource to accept the output of the conversion
string: A USVString to be converted
offset: The number of code units at the start of string to ignore.

Return value is the index into the string which the encoder reached (but it might be better to return the number of code units consumed?). If the entire string was converted, the return value will be equal to string.length.

If the output buffer is less than 4 bytes long, it is possible for examples like I wrote above to loop forever. Maybe encodeInto() should throw if it can't make progress?