Feature request: file pread and pwrite

[tomjridge]

I have the following scenario:

I have many writes to make to a file. Each write is of a fixed length (eg 4096 bytes). Each write occurs at a different offset in the file. No writes overlap. The data for each write is held in a bigstring buffer. Essentially I have batched updates to different regions in a file.

I would like these writes to happen "in parallel", preferably with only one system call.

Let's focus on the "in parallel" part.

At the moment, for each write, I can lseek to the position and then write the data. But this doesn't parallelize because the lseek of one write will interfere with the lseek of another.

If the file based pread/pwrite were supported (ie the call documented here: https://pubs.opengroup.org/onlinepubs/9699919799/functions/read.html and write.html , with explicit file offset), I could at least issue the writes in parallel, without them interfering with each other.

Ideally it would be possible to submit a single write-family syscall, with a list of buffers, where each buffer also includes the offset in the file at which the buffer should be written. But I don't know if Linux aio, or some other mechanism, can accomplish this. Perhaps the basic mechanisms are supported in libuv or similar?

Anyway, any advice appreciated.

[aantron]

Ideally it would be possible to submit a single write-family syscall, with a list of buffers, where each buffer also includes the offset in the file at which the buffer should be written.

AFAIK, the closest that we can get to this at a system call level is pwritev, which will take a list of buffers, but only one offset, and write the buffers contiguously starting at that offset. AIO may provide the kind of interface you are suggesting; however, when I last checked Linux AIO around 2017, it was a very slow wrapper around synchronous I/O and threads, so I have been disregarding it ever since. I think AIO should be avoided unless/until there is news that it has been worked on :)

It should be possible and relatively straightforward to support pwrite and/or pwritev in Lwt.

[aantron]

To clarify, I was benchmarking write performance, comparing several methods, including synchronous write, libuv, Lwt, AIO, using a worker thread, and some others. I couldn't easily find a link to the results just now, but IIRC AIO had the worst performance by a significant margin.

(edit: and I briefly glanced in the AIO sources to try to understand why, but I quickly realized that libuv or manual threads will outperform what I seemed to see in the AIO implementation)

[aantron]

Reading about it in more detail, I may have been looking at POSIX AIO in 2017. Perhaps libaio needs a look.

[tomjridge]

Thanks for the prompt reply! It is a bit strange that the scatter/gather interfaces can only take 1 offset. Oh well. pread and pwrite support would be great. pwritev looks like it won't help me that much (since there is only one offset). But perhaps it is easier to add this than pwrite, and presumably some people will find it useful. Anyway, thanks for your help.

I looked at the doc here:

https://github.com/littledan/linux-aio

And indeed it does seem to match my use case (each iocb supports its own offset within the file).

There is an ocaml lib here: https://opam.ocaml.org/packages/aio/

But I'm not sure how maintained it is. So, plenty of options for further investigation anyway.

Thanks for your help

Tom

[chambart]

From what I have read libaio does basically the same thing as Lwt in detach mode. This won't allows you to do less system calls. If you want to minimize the number of system calls, you might want to take a look at the brand new io_uring interface. Notice that this is only available since Linux 5.1 (hence not in debian stable for instance).

[aantron]

@chambart I read that POSIX AIO does the same thing as Lwt in detach mode (though even slower). libaio takes advantage of some kind of kernel support for AIO. I'm not sure if the kernel then also behaves like Lwt in detach mode, is that what you are referring to?

[tomjridge]

I think there are various "aio" interfaces (POSIX, Linux etc). At least the Linux native aio seemed to support my use case. However...

A quote from https://kernel-recipes.org/en/2019/talks/faster-io-through-io_uring/ :

"Native Linux AIO supports only a niche application class (O_DIRECT), and even for that use case, it’s far too slow for modern storage."

Which is not encouraging.

[tomjridge]

But the talk about io_uring is very encouraging!

[aantron]

It also looks like libuv may switch to io_uring under the hood for APIs where it is beneficial (libuv/libuv#1947), so we would automatically get that if we used libuv to implement lwt.unix (#328).