Suggestion: stream.read(buffer, offset, length, callback)

[jorangreef]

I am not sure if this is possible with the design of libuv and Node, but it would be great to be able to do:

stream.read(buffer, offset, length, callback)

buffer is the buffer that the data will be written to.

offset is the offset in the buffer to start writing at.

length is an integer specifying the number of bytes to read.

Especially for net and tls sockets.

I am not sure if Node itself currently manages and re-uses the underlying buffers for net sockets? But if not, then this would make it possible to do static memory allocation in a server. In the past, streams did not support pull mode at all, now they support pull mode, but the user still has no control over the allocation of the underlying buffers, so this creates more allocations and more work for the GC.

I am working on a client-side program at the moment and am finding that V8's GC is just too lazy. Things work and eventually get collected but memory usage climbs and climbs. I would like to have more control and rather statically allocate buffers and re-use them. This would also have a positive side-effect of controlling queue sizes and workloads in various parts of the program more explicitly.

[Fishrock123]

cc @nodejs/streams

[chrisdickinson]

Something like this would be cool. WHATWG currently has "getByobReader", which implements something similar. I would advise we watch how that pans out closely to see if there are any rough edges we should know about before implementing such a thing ourselves.

On Oct 20, 2015, at 6:08 AM, Jeremiah Senkpiel notifications@github.com wrote:

cc @nodejs/streams

—
Reply to this email directly or view it on GitHub.

[bnoordhuis]

Apropos nothing:

I am not sure if this is possible with the design of libuv

Request-driven reading is not the model that libuv currently uses but it can be modeled reasonably efficiently by selectively calling uv_read_start() and uv_read_stop().

That's what libuv's socks5-proxy example does in fact and it's the most natural model for most applications. The plan is to implement it natively in libuv v2.x.

[Trott]

Anything new to add to this issue in the past 7 months?

[jhamhader]

Looking into it.
Need to think on how this API makes sense in the case of objectMode or stringMode (encoding set) on the readable stream.

[vkurchatkin]

#6923

[Trott]

This issue has been inactive for sufficiently long that it seems like perhaps it should be closed. Feel free to re-open (or leave a comment requesting that it be re-opened) if you disagree. I'm just tidying up and not acting on a super-strong opinion or anything like that.

[jorangreef]

This issue is blocked on libuv v2.x and nodejs/node-eps#27 which may take some time.

Can we re-open?

We definitely need a way to do more static allocation in Node specifically when working with TCP.

[refack]

Put into https://github.com/nodejs/node/projects/13 backlog