RFC: Expand the scope of `std::fs` #1044

alexcrichton · 2015-04-07T23:07:16Z

Expand the scope of the std::fs module by enhancing existing functionality,
exposing lower level representations, and adding a few more new functions.

Rendered

Expand the scope of the `std::fs` module by enhancing existing functionality, exposing lower level representations, and adding a few more new functions.

retep998 · 2015-04-08T00:51:25Z

text/0000-io-fs-2.1.md

+    fn metadata(&self) -> io::Result<Metadata>;
+    fn soft_link_metadata(&self) -> io::Result<Metadata>;
+    fn canonicalize(&self) -> io::Result<PathBuf>;
+    fn read_link(&self) -> io::Result<PathBuf>;


Currently on Windows read_link recursively resolves all intermediate symbolic links. Is the behavior going to be changed so that it only reads a single layer?

I would love for it to do so! You wouldn't happen to know the API off-hand to do this? (I feel like I've asked this before...)

I suspect you need to call CreateFile[Ex] with FILE_FLAG_OPEN_REPARSE_POINT and use DeviceIoControl with FSCTL_GET_REPARSE_POINT

Also potentially useful:

Symbolic Link Effects on File Systems Functions

Python example code

https://github.com/retep998/wio-rs/blob/master/src/reparse.rs

Thanks for the pointers @andrew-d and @retep998! I hope to make a PR soon as it should be able to land separately from this RFC.

alexcrichton · 2015-04-08T17:09:31Z

I've added two more pieces of information (thanks to @lambda for the suggestions!):

There is now a unix extension trait to access the d_ino field of DirEntry on unix.
There is now a FileType abstraction in std::fs to represent the type of a file, which is also exported on a DirEntry.

lambda · 2015-04-08T18:43:17Z

Yep, those two changes address my concerns. 👍

codyps · 2015-04-08T18:48:09Z

text/0000-io-fs-2.1.md

+impl Permissions {
+    /// Creates a new set of permissions appropriate for being placed on a file.
+    ///
+    /// On unix platforms this corresponds to the permission bits `0o666`


Perhaps this should use umask() as the default for linux? Or do we specifically want a certain type of perms for the default here?

Either way, we should document the reasoning.

Hm that's a good point! The goal here was to add some method of creating a Permissions in a cross platform manner, but the exact meaning of the permissions being returned is tough to nail down, and it may mean that we don't want this constructor at all. I've personally only ever needed this when calling set_permissions on unix with a known set of bits ahead of time, in which case the from_mode constructor below would suffice.

I think I'm somewhat tempted to just remove this constructor altogether for now while we continue to hammer down what Permissions means on windows.

I think that holding off on trying to handle permissions in a cross platform manner would be a good idea. Permissions semantics are very different between platforms, besides the basic UID, GID, and modes which are common across Unix like platforms, and wrapping them in a portable manner could be quite tricky. The rest of this RFC would be useful without trying to solve how to represent or construct permissions in a portable manner.

The best default is likely to just be the permissions that are created on a file or directory by the current user under the current circumstances. This can be influenced by the umask, the setgid bit and group on the directory, the default ACLs on the directory, and that's all just on Unix-like platforms, I don't know the Windows permission model well enough to say. Thus, I think that most of the time when creating permissions, you will either know exactly what you want, which will be platform dependent, or you will want to copy the permissions from an existing object and apply them to a new one.

lambda · 2015-04-08T23:28:30Z

Thanks for the updates!

One more bikeshed after reading it through one more time. This RFC refers to symbolic links as soft links, but on both Unix and Windows, the preferred term seems to be symbolic link, or abbreviated as symlink. I notice that there's already a stabilized std::fs::soft_link function to create one, so that may be sufficient inertia to keep the soft link terminology, but if it's still possible to deprecate that and change to using the term "symlink" or "symbolic link", I think that that would be less surprising to most people.

Many sources say "symbolic link (or soft link)" or "soft link (or symbolic link)" when referring to them, but the actual APIs are symlink on Unix, CreateSymbolicLink on Windows, and the official documentation and standards for most platforms I looked at (Mac OS X, FreeBSD, Linux, Windows, POSIX/Single Unix Specification) uses "symbolic link" exclusively in prose, never "soft link".

Quick poll of other languages as well: Java has createSymbolicLink, C# CreateSymbolicLink, C++ (Boost/draft standard) create_symlink, Ruby symlink, Python symlink, Objective-C/Swift createSymbolicLinkAtPath:withDestinationPath:.

alexcrichton · 2015-04-08T23:56:41Z

I do agree that I sometimes have to do a double-take whenever I see "soft_link" as opposed to symlink, but it does tend to agree with the current conventions of the I/O modules:

The term "symlink" is arguably a unix-ism (not mentioned on Windows), and we don't tend to favor one platform's naming conventions over another.
The term "symbolic link" is a little wordy for the standard library, it tends to stick to some smaller terms wherever possible.

With these two the term "soft link" seemed like a good fit in the middle. It is possible that the fs::soft_link function could be deprecated (if this change happened soon), but another possibility would also be to modify accessors to something like is_link instead of is_soft_link.

alexcrichton · 2015-04-09T00:25:41Z

I've pushed a commit that removes the recommendation of Permissions::new

lambda · 2015-04-09T00:58:02Z

Yeah, Windows does use SymbolicLink fairly consistently in their API, but I have found no API that uses "soft link"; every API I have found uses some variant of "symlink" or "symbolic link" in snake or camel case. Even other Windows-centric languages do use symlink; Delphi has FileCreateSymLink, and while it looks like PowerShell has no native way to do it, various people have written PowerShell cmdlets called New-SymLink (one example, another example). And Microsoft also includes the proposed C++ filesystem standard that uses create_symlink in Visual Studio.

I feel like the surprise of being the only API (that I could find) that uses soft_link may outweigh either the length of symbolic_link or the slightly Unix-centric nature of symlink; and one thing to recall is that symbolic links themselves are a concept that originated on Unix and were implemented on Windows mostly for compatibility with Unix:

Symbolic links are designed to aid in migration and application compatibility with UNIX operating systems. Microsoft has implemented its symbolic links to function just like UNIX links.

Anyhow, it is a relatively minor bikeshed, and given the existing soft_link function, I'm wondering whether the discussion is appropriate for this RFC, or whether I should file another RFC to discuss just this one naming point and possible deprecation of the already stabilized soft_link in favor of a more standard naming convention.

codyps · 2015-04-09T04:11:48Z

I've also never been a fan of soft_link for the similar reasons. Avoiding symlink as a precieved unix-ism seemed a bit knee-jerky.

alexcrichton · 2015-04-09T06:43:58Z

@lambda I think it's definitely relevant to this RFC as it's expanding the surface area of the "soft link" terminology, but I also think that you may want to open a separate RFC issue and/or discuss post about it as we probably shouldn't centralize discussion about that topic here.

alexcrichton · 2015-04-09T06:44:25Z

er, it appears you did!

#1048

lambda · 2015-04-09T06:49:04Z

Indeed, and you beat me to cross-referencing to this issue or cc'ing you on it!

nagisa · 2015-04-09T10:04:08Z

text/0000-io-fs-2.1.md

+
+    pub struct Metadata(raw::stat);
+    impl Metadata {
+        // Accessors for fields available in `raw::stat` for *all* platforms


all unix platforms.

l0kod · 2015-04-09T22:23:43Z

👍 for the enum FileType.

Don't forget to include POSIX ACL in Permissions (cf. acl(5)).

The PathExt should include all functions from FileType (i.e. is_soft_link()).

PathExt should inherit from a common trait to file descriptors and paths (e.g. AsIo, cf. rust-lang/rust#21936 (comment)) including this functions:

FileType functions
metadata()
read_link()
read_dir()

About all the stat(2) related cache (e.g. is_dir, is_file…), FileType should cache all metadata but provide a refresh() function to explicitly reload the path/fd info when needed and avoid unneeded syscall calls.

I'm not sure the PathBuf is the better return value for read_link(). Some symlinks are virtual like /proc/self/ns/* (but they can be interpreted as path too).

👍 for std::os::MetadataExt: creation_time(), read_time() and write_time() functions for both Windows and UNIX to return an unified time representation + time precision.

This commit is an implementation of [RFC 1044][rfc] which adds additional surface area to the `std::fs` module. All new APIs are `#[unstable]` behind assorted feature names for each one. [rfc]: rust-lang/rfcs#1044 The new APIs added are: * `fs::canonicalize` - bindings to `realpath` on unix and `GetFinalPathNameByHandle` on windows. * `fs::symlink_metadata` - similar to `lstat` on unix * `fs::FileType` and accessor methods as `is_{file,dir,symlink}` * `fs::Metadata::file_type` - accessor for the raw file type * `fs::DirEntry::metadata` - acquisition of metadata which is free on Windows but requires a syscall on unix. * `fs::DirEntry::file_type` - access the file type which may not require a syscall on most platforms. * `fs::DirEntry::file_name` - access just the file name without leading components. * `fs::PathExt::symlink_metadata` - convenience method for the top-level function. * `fs::PathExt::canonicalize` - convenience method for the top-level function. * `fs::PathExt::read_link` - convenience method for the top-level function. * `fs::PathExt::read_dir` - convenience method for the top-level function. * `std::os::raw` - type definitions for raw OS/C types available on all platforms. * `std::os::$platform` - new modules have been added for all currently supported platforms (e.g. those more specific than just `unix`). * `std::os::$platform::raw` - platform-specific type definitions. These modules are populated with the bare essentials necessary for lowing I/O types into their raw representations, and currently largely consist of the `stat` definition for unix platforms. This commit also deprecates `Metadata::{modified, accessed}` in favor of inspecting the raw representations via the lowering methods of `Metadata`.

alexcrichton · 2015-04-23T01:24:31Z

I've posted a PR with a sample implementation of this RFC.

Exa now uses the new IO, Path, and Filesystem libraries that have been out for a while now. Unfortunately, the new libraries don't *entirely* cover the range of the old libraries just yet: in particular, to become more cross-platform, the data in `UnstableFileStat` isn't available in the Unix `MetadataExt` yet. Much of this is contained in rust-lang/rfcs#1044 (which is due to be implemented in rust-lang/rust#14711), but it's not *entirely* there yet. As such, this commits a serious loss of functionality: no symlink viewing, no hard links or blocks, or users or groups. Also, some of the code could now be optimised. I just wanted to commit this to sort out most of the 'teething problems' of having a different path system in advance. Here's an example problem that took ages to fix for you, just because you read this far: when I first got exa to compile, it worked mostly fine, except calling `exa` by itself didn't list the current directory. I traced where the command-line options were being generated, to where files and directories were sorted, to where the threads were spawned... and the problem turned out to be that it was using the full path as the file name, rather than just the last component, and these paths happened to begin with `.`, so it thought they were dotfiles.

This commit is an implementation of [RFC 1044][rfc] which adds additional surface area to the `std::fs` module. All new APIs are `#[unstable]` behind assorted feature names for each one. [rfc]: rust-lang/rfcs#1044 The new APIs added are: * `fs::canonicalize` - bindings to `realpath` on unix and `GetFinalPathNameByHandle` on windows. * `fs::symlink_metadata` - similar to `lstat` on unix * `fs::FileType` and accessor methods as `is_{file,dir,symlink}` * `fs::Metadata::file_type` - accessor for the raw file type * `fs::DirEntry::metadata` - acquisition of metadata which is free on Windows but requires a syscall on unix. * `fs::DirEntry::file_type` - access the file type which may not require a syscall on most platforms. * `fs::DirEntry::file_name` - access just the file name without leading components. * `fs::PathExt::symlink_metadata` - convenience method for the top-level function. * `fs::PathExt::canonicalize` - convenience method for the top-level function. * `fs::PathExt::read_link` - convenience method for the top-level function. * `fs::PathExt::read_dir` - convenience method for the top-level function. * `std::os::raw` - type definitions for raw OS/C types available on all platforms. * `std::os::$platform` - new modules have been added for all currently supported platforms (e.g. those more specific than just `unix`). * `std::os::$platform::raw` - platform-specific type definitions. These modules are populated with the bare essentials necessary for lowing I/O types into their raw representations, and currently largely consist of the `stat` definition for unix platforms. This commit also deprecates `Metadata::{modified, accessed}` in favor of inspecting the raw representations via the lowering methods of `Metadata`.

aturon · 2015-04-24T23:44:37Z

Thanks for the fantastic discussion, everyone! I've merged the RFC.

Much of the feedback ended up targeting symlinks, which got moved off into their own RFC. Otherwise, I believe that all of the feedback has been taken into account, and overall we have a strong consensus that these APIs are both reasonable and needed.

Note that, as always, these features will be landing as unstable (for 1.1 beta) and we can continue tweaking details of the design prior to stabilization.

GuillaumeGomez · 2015-04-27T14:11:35Z

text/0000-io-fs-2.1.md

+    pub fn file_type(&self) -> io::Result<FileType>;
+
+    /// Returns the file name for this directory entry.
+    pub fn file_name(&self) -> OsString;


What happen if this is a directory or something else ? Do we call this function ?

This commit is an implementation of [RFC 1044][rfc] which adds additional surface area to the `std::fs` module. All new APIs are `#[unstable]` behind assorted feature names for each one. [rfc]: rust-lang/rfcs#1044 The new APIs added are: * `fs::canonicalize` - bindings to `realpath` on unix and `GetFinalPathNameByHandle` on windows. * `fs::symlink_metadata` - similar to `lstat` on unix * `fs::FileType` and accessor methods as `is_{file,dir,symlink}` * `fs::Metadata::file_type` - accessor for the raw file type * `fs::DirEntry::metadata` - acquisition of metadata which is free on Windows but requires a syscall on unix. * `fs::DirEntry::file_type` - access the file type which may not require a syscall on most platforms. * `fs::DirEntry::file_name` - access just the file name without leading components. * `fs::PathExt::symlink_metadata` - convenience method for the top-level function. * `fs::PathExt::canonicalize` - convenience method for the top-level function. * `fs::PathExt::read_link` - convenience method for the top-level function. * `fs::PathExt::read_dir` - convenience method for the top-level function. * `std::os::raw` - type definitions for raw OS/C types available on all platforms. * `std::os::$platform` - new modules have been added for all currently supported platforms (e.g. those more specific than just `unix`). * `std::os::$platform::raw` - platform-specific type definitions. These modules are populated with the bare essentials necessary for lowing I/O types into their raw representations, and currently largely consist of the `stat` definition for unix platforms. This commit also deprecates `Metadata::{modified, accessed}` in favor of inspecting the raw representations via the lowering methods of `Metadata`.

haxney · 2015-04-28T01:12:02Z

As a general comment, I'd recommend looking into user-defined filesystems like what Java has with its Path and FileSystem classes. Essentially, rather than being able to turn a String into a Path directly, you need to ask some FileSystem object to give you a Path for a String.

It is a bit more complex (but you can do FileSystems.getDefault().getPath("thefile.txt") to use the default file system), but allows you to create things like Jimfs (an in-memory file system for testing) without worrying about things like FUSE.

This commit is an implementation of [RFC 1044][rfc] which adds additional surface area to the `std::fs` module. All new APIs are `#[unstable]` behind assorted feature names for each one. [rfc]: rust-lang/rfcs#1044 The new APIs added are: * `fs::canonicalize` - bindings to `realpath` on unix and `GetFinalPathNameByHandle` on windows. * `fs::symlink_metadata` - similar to `lstat` on unix * `fs::FileType` and accessor methods as `is_{file,dir,symlink}` * `fs::Metadata::file_type` - accessor for the raw file type * `fs::DirEntry::metadata` - acquisition of metadata which is free on Windows but requires a syscall on unix. * `fs::DirEntry::file_type` - access the file type which may not require a syscall on most platforms. * `fs::DirEntry::file_name` - access just the file name without leading components. * `fs::PathExt::symlink_metadata` - convenience method for the top-level function. * `fs::PathExt::canonicalize` - convenience method for the top-level function. * `fs::PathExt::read_link` - convenience method for the top-level function. * `fs::PathExt::read_dir` - convenience method for the top-level function. * `std::os::raw` - type definitions for raw OS/C types available on all platforms. * `std::os::$platform` - new modules have been added for all currently supported platforms (e.g. those more specific than just `unix`). * `std::os::$platform::raw` - platform-specific type definitions. These modules are populated with the bare essentials necessary for lowing I/O types into their raw representations, and currently largely consist of the `stat` definition for unix platforms. This commit also deprecates `Metadata::{modified, accessed}` in favor of inspecting the raw representations via the lowering methods of `Metadata`. Closes #24796

This commit is an implementation of [RFC 1044][rfc] which adds additional surface area to the `std::fs` module. All new APIs are `#[unstable]` behind assorted feature names for each one. [rfc]: rust-lang/rfcs#1044 The new APIs added are: * `fs::canonicalize` - bindings to `realpath` on unix and `GetFinalPathNameByHandle` on windows. * `fs::symlink_metadata` - similar to `lstat` on unix * `fs::FileType` and accessor methods as `is_{file,dir,symlink}` * `fs::Metadata::file_type` - accessor for the raw file type * `fs::DirEntry::metadata` - acquisition of metadata which is free on Windows but requires a syscall on unix. * `fs::DirEntry::file_type` - access the file type which may not require a syscall on most platforms. * `fs::DirEntry::file_name` - access just the file name without leading components. * `fs::PathExt::symlink_metadata` - convenience method for the top-level function. * `fs::PathExt::canonicalize` - convenience method for the top-level function. * `fs::PathExt::read_link` - convenience method for the top-level function. * `fs::PathExt::read_dir` - convenience method for the top-level function. * `std::os::raw` - type definitions for raw OS/C types available on all platforms. * `std::os::$platform` - new modules have been added for all currently supported platforms (e.g. those more specific than just `unix`). * `std::os::$platform::raw` - platform-specific type definitions. These modules are populated with the bare essentials necessary for lowing I/O types into their raw representations, and currently largely consist of the `stat` definition for unix platforms. This commit also deprecates `Metadata::{modified, accessed}` in favor of inspecting the raw representations via the lowering methods of `Metadata`. Closes rust-lang#24796

alexcrichton and others added 3 commits April 6, 2015 10:56

RFC: Expand the scope of std::fs

f3153e6

Expand the scope of the `std::fs` module by enhancing existing functionality, exposing lower level representations, and adding a few more new functions.

Add vision for std::os, a few other tweaks

b3d1ad6

More feedback and tweaks

91c0df5

alexcrichton mentioned this pull request Apr 7, 2015

Expand the scope of std::fs #939

Open

retep998 reviewed Apr 8, 2015
View reviewed changes

alexcrichton self-assigned this Apr 8, 2015

alexcrichton added 2 commits April 8, 2015 09:58

Add an accessor for d_ino on unix

e6c4854

Add a FileType structure

ebf5514

codyps reviewed Apr 8, 2015
View reviewed changes

alexcrichton added 2 commits April 8, 2015 15:53

Remove a half-worded alternative

bc8759d

Bind remaining unix mode constants

fedf7fd

Don't add Permissions::new

11265b4

nagisa reviewed Apr 9, 2015
View reviewed changes

alexcrichton mentioned this pull request Apr 9, 2015

Add Builder guidelines #1036

Closed

alexcrichton mentioned this pull request Apr 20, 2015

Unable to build in stable (beta) channel alexcrichton/tar-rs#17

Closed

alexcrichton mentioned this pull request Apr 23, 2015

std: Expand the area of std::fs rust-lang/rust#24711

Merged

aturon mentioned this pull request Apr 24, 2015

Tracking issue for std::fs extensions (RFC #1044) rust-lang/rust#24796

Closed

aturon merged commit d8bc0fe into rust-lang:master Apr 24, 2015

GuillaumeGomez reviewed Apr 27, 2015
View reviewed changes

carols10cents mentioned this pull request Sep 6, 2015

cargo login should ensure ~/.cargo/config is not world-readable rust-lang/cargo#930

Closed

Centril added T-libs-api Relevant to the library API team, which will review and decide on the RFC. A-file Proposals relating to file systems. labels Nov 23, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC: Expand the scope of `std::fs` #1044

RFC: Expand the scope of `std::fs` #1044

alexcrichton commented Apr 7, 2015

retep998 Apr 8, 2015

alexcrichton Apr 8, 2015

andrew-d Apr 8, 2015

retep998 Apr 8, 2015

alexcrichton Apr 8, 2015

alexcrichton commented Apr 8, 2015

lambda commented Apr 8, 2015

codyps Apr 8, 2015

alexcrichton Apr 8, 2015

lambda Apr 9, 2015

lambda commented Apr 8, 2015

alexcrichton commented Apr 8, 2015

alexcrichton commented Apr 9, 2015

lambda commented Apr 9, 2015

codyps commented Apr 9, 2015

alexcrichton commented Apr 9, 2015

alexcrichton commented Apr 9, 2015

lambda commented Apr 9, 2015

nagisa Apr 9, 2015

l0kod commented Apr 9, 2015

alexcrichton commented Apr 23, 2015

aturon commented Apr 24, 2015

GuillaumeGomez Apr 27, 2015

haxney commented Apr 28, 2015

RFC: Expand the scope of std::fs #1044

RFC: Expand the scope of std::fs #1044

Conversation

alexcrichton commented Apr 7, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexcrichton commented Apr 8, 2015

lambda commented Apr 8, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lambda commented Apr 8, 2015

alexcrichton commented Apr 8, 2015

alexcrichton commented Apr 9, 2015

lambda commented Apr 9, 2015

codyps commented Apr 9, 2015

alexcrichton commented Apr 9, 2015

alexcrichton commented Apr 9, 2015

lambda commented Apr 9, 2015

Choose a reason for hiding this comment

l0kod commented Apr 9, 2015

alexcrichton commented Apr 23, 2015

aturon commented Apr 24, 2015

Choose a reason for hiding this comment

haxney commented Apr 28, 2015

RFC: Expand the scope of `std::fs` #1044

RFC: Expand the scope of `std::fs` #1044