Figure out the future of nix

[carllerche]

tl;dr Should nix be split up into multiple crates named nix-* and what should the API style be?

The situation

Nix originated as a posix binding dumping ground for various crates that I had been working on. I tended to need the same bindings over and over again, so I just moved them into Nix. I thought this would be useful for other people, so I accepted PRs for various system APIs as they have been submitted.

Everything has been fine so far, but a few issues have arisen.

As the API coverage has increased, whenever a breaking change is made, the semver must be incremented. So, if the ioctl module changes an API as it figures out the best way to expose functionality, the version must be bumped and downstream crates must change their dependencies. For example, Mio depends on nix, but not on any ioctl features, so bumping the dependency is pretty annoying.

Secondly, the API style has been somewhat ad hoc, especially with more complicated system APIs like fcntl.

Crate structure

There is value in having a single place (nix) to find system APIs that have a coherent API style and a single documentation site. I think that each specific API silo, like sockets, ioctl, mqueue, signals, etc... could be in their own crate named nix_*. Common features like errors, NixPath, etc... would live in nix_core. The nix crate would be a facade depending on all the sub crates with liberal dependencies (probably "*") which would allow the consumer to optionally be able to pin each sub crate to whatever.

Libraries like Mio could then depend only on the features that are needed.

Also, it would allow delegating responsibility better. I, for one, know nothing about ioctl. I would like to stop being the blocker.

API style

This is still something that I haven't figured out. I do know that the goal of nix is *comprehensive, safe, zero cost bindings to system APIs on nix OSes.

So, what does that mean for nix? As of now, there are currently a variety of styles. For example, most APIs are 1-1 bindings to the equivalent OS api. This leads to APIs like fcntl which uses a convoluted FcntlArg strategy. ioctl on the other hand uses a (fairly impressive) macro...

The question is, would it be better to expand beyond the 1-1 rule. For example, in the fnctl case, the API would most likely be much cleaner and more "rusty" if each "fcntl arg" was moved to an individual function. There could, for example, be a public fcntl mod with a dupfd function. Using it would be:

let res = fcntl::dup_fd(fd);

Then, should it go further? Should there be an Fd type that wraps std::os::unix::RawFd and implement functions like dup directly on that type?

These are all open questions that I pose to the users of nix.

cc @posborne @cmr @geofft @utkarshkukreti @MarkusJais and whoever else :)

[aturon]

cc @alexcrichton (possibly relevant for liblibc plans?)

[joekain]

I’m not sure what the best approach is here but here are few thoughts I had:

Also, it would allow delegating responsibility better. I, for one, know nothing about ioctl. I would like to stop being the blocker.

This is possible within a single crate if you wanted to give others ability to merge pull requests. Multiple crates would give finer control e.g. someone can have permission to merge ioctl changes but not fcntl changes.

I do know that the goal of nix is comprehensive, safe, zero cost bindings to system APIs on *nix OSes.

Awesome.

As of now, there are currently a variety of styles

This is something to discuss in terms of developing nix but I think single crate or multiple crates this still imposes multiple styles on users of the crate.

[carllerche]

@joekain My proposal would be to keep all the code in the same git repository, but to release individual crates to crates.io. This would only be to allow the versions to progress independently. There should be a consistent style across all the individual crates.

[carllerche]

@alexcrichton also mentioned that he had a goal of generating all the constants / unsafe FFI definitions needed for nix in the the libc crate, which would allow nix to purge all the unsafe / const bindings in favor of focusing entirely on the Rust API.

[alexcrichton]

Ah and to clarify, I'm not sure we'll be generating the libc crate, but we're going to start verifying it much more rigorously, so it should be much easier to add to but will still require additions.

[posborne]

Thanks for bringing this up @carllerche.

I am in favor of your proposal to retain a single git repository for the "official" nix libraries in order to provide consistency but breaking out subsystems into discrete crates that are largely independent (with the exception of nix_core).

The level of abstraction question is one that I have been thinking about more as changes have been made with the ioctl system. With that change, I feel that it may have been a better (in hindsight) to expose the functionality in layers:

• Layer 1: The raw ioctl call exposed by libc/liblibc
• Layer 2: Something similar to (the old ioctl API)[https://github.com/carllerche/nix-rust/blob/c4a9598ec16a01e11e66bea1eb34cea9b4646326/src/sys/ioctl.rs] I wrote (Unsafe)
• Layer 3: Higher level, macros (API present today)https://github.com/carllerche/nix-rust/blob/master/src/sys/ioctl/mod.rs
• Layer 4: Safe APIs for specific drivers based on top of layer 2/3. Examples of that would be crates like https://github.com/posborne/rust-spidev and https://github.com/posborne/rust-i2cdev

I think all layers up to 3 could have a good spot within nix, especially with discrete crates but that designing in terms of layers and exposing each layer to users probably makes sense in most cases as well. The higher level the abstraction, the more likely it is that a use case might be prevented.

I believe Nix should aim to expose the highest level (most safe) API that presents an unlimited (a user can do everything needed), zero-cost abstraction as a start. To benefit users, nix crates may also expose APIs that sacrifice flexibility in favor of safety for select use cases.

[droundy]

My vote (as a user who is only starting to use nix) would be to keep nix as a monolithic crate, and just let users bump their version dependencies. They shouldn't be forced to bump the version that frequently, right? i.e. it should be safe for them to continue using an older version for some time.

I also think that developing a rusty (and complete) API would be ideal. e.g. the ptrace signature is currently unsafe (although not marked so?!), and could productively be replaced with multiple helper functions that are actually safe, since the types of the later arguments depend on the flag.

[kamalmarhubi]

FYI I filed #214 about exporting all functions / types at the crate root as is now done in the libc crate. It's probably relevant to this discussion.

[carllerche]

@droundy the main driver around splitting up the crate is that a breaking change in a less thought out part of the API isn't a breaking change in other parts...

@kamalmarhubi Yep, thanks for filing it.

In general, this issue is on hold until libc stabilizes a bit and pulls in the required symbols.

[MarkusJais]

I think the most important thing is to have a consistent style for APIs.

Putting the low-level bindings in libc is a good idea and then nix could just wrap them so no unsafe stuff is necessary for the user of nix. And we could add convenience functions on top of the low-level bindings (e.g. the mq_set_nonblock/mq_remove_nonblock I've added to mqueue.rs) that makes using the APIs easier (but avoiding any serious overhead!)

I am happy to remove all the low-level mqueue stuff once it is in libc.

[kamalmarhubi]

I like the idea of moving as much of the current ffi module stuff to libc as possible. The main things attracting me to nix are:

• no unsafe necessary in user code
• Rustified error handling

I don't have enough experience in any of the areas where nix has fancier bindings, so I can't comment on that at all. I can say I do like things mapping directly to man pages for what I'm trying to do at the moment.