No cross-platform way to close an fs::File and check for errors

[reem]

There should be an: fn close(self) -> io::Result<()> that does the same as Drop but does not swallow any errors. Currently the only way to do this is with into_raw_fd and libc::close, which is both unergonomic and non-cross-platform.

[retep998]

into_raw_handle and winapi::CloseHandle as well. So you can implement your own external version that works cross platform. Still would be nice to have it directly in std though.

[ranma42]

Even when invoking libc::close, one has to tackle platform-specific behaviour, as not all *nix operating systems conform to the latest POSIX specification .
For additional information, see https://lwn.net/Articles/576478/ (or search for the keywords EINTR and close).

[bluss]

I think should have a policy to provide these methods whenever fallible close / fallible drop can occur. Harder question is maybe if there should be an I/O trait for close.

[ticki]

Note: This issue was linked here.

[retep998]

I think should have a policy to provide these methods whenever fallible close / fallible drop can occur.

HeapFree on Windows can return an error (and in fact at one point actually did during CI because we have a debug assertion checking it). Should everything that frees memory provide these methods too?

[dpc]

I have a practical need for this File::close(). I'm working on rdedup which is kind of backup software with deduplication. Data chunks are written to disk in a temporary file, then File::sync_data is done to make sure write happened and there were no errors, file is closed, and renamed atomically to make sure only completely written, uncorrupted data can ever be visible in the destination path. However all these sync_data take precious time as I do it for thousands of files. I'd like to try writing all files, then syncing whole filesystem, then doing the rename in the batch. Maybe it would be faster. However it's not possible (without resorting to libc at least), as not doing data_sync means, there is no way to close the file and still make sure data was actually written. I'm still validating the whole approach (especially close() syscall not returning errors etc.), and I'm not sure if it's going to be faster, but I just want to point out that it's currently impossible.

[retep998]

@dpc Some platforms support unbuffered writes, which can be faster than buffered writes + flushing. On Windows this can be done with the FILE_FLAG_NO_BUFFERING and FILE_FLAG_WRITE_THROUGH flags when opening the file (OpenOptionsExt).

[alexcrichton]

@dpc if we were to add this to the standard library, this is the implementation that I would prefer. Which is also to say that this is not impossible to do today, it's just not as ergonomic as it could be.

(you can also imagine a similar implementation for TcpStream and UdpSocket and friends.

[federicomenaquintero]

On #42619 I was doing some investigation of what the Linux kernel does when closing files depending on the file system.

close(2) can return EINTR; also, it can return other things for the following file systems: afs, cifs, ecryptfs, exofs, fuse, nfs. None of the local file systems, as far as I can tell, go through the code path that would cause filp_close() to return another kind of error (i.e. they don't implement the file_operations->flush method). The remote filesystems listed here implement flush, and can return other errors.

Many moons ago in GNOME we were having a related discussion, around fsync() and saving files reliably. There are various ways in which user-facing programs have tried to save files:

1. Plain open(), write(), close() - easy, not terribly safe.

2. fsync() before close(), with error checking. Slow if done frequently.

3. Write to a temporary file, fsync() before close(), and then atomically rename() the temp file to the final file.

(3) is the preferred way for File/Save for "precious" data: if the user has been typing for hours, you don't want to:

• Overwrite an old version of the file and leave a half-written file if you get errors (out of space, I/O error, etc.)
• Lose data because the network went down while the save was in progress (or even the close() for it). The app can tell you, "I couldn't save", and you can take extra measures like saving to another location.
• Leave a non-atomically-replaced halfway-written version of the file. With (3), what remains on disk after a system crash is the old file, or the new file, or both, but not halfway or zero-byte versions of either (... if the file system cooperates). (It's also what Emacs does, FWIW; it's saving code is reassuringly careful.)

(Then, there is debate on whether one needs to fsync() the directory after the rename()...)

Conclusion: maybe we could have a recommendation to File.sync_all() for important data. We could have an extra crate for doing the rename() dance for user-initiated saves (we had something like that in GNOME at some point in libgnome-office; there was clearly a need for it). I don't know if we should expose close() errors; it's nice to have @alexcrichton's code to do that if needed.

[the8472]

(3) is the preferred way for File/Save for "precious" data: if the user has been typing for hours, you don't want to:

That not sufficient. You need to write, fsync, close, rename, fsync dir to ensure the rename went through. The rust stdlib currently provides no way to fsync a directory.

https://lwn.net/Articles/457667/

[mzabaluev]

There are several issues that often get mixed up in the discussion on proper closing of files:

1. There are errors provided by the OS to the application, which do not get reported when the result of close is ignored. Currently, calling close in drop is the 'natural' way to close files in the Rust standard library.
2. Some applications need a way to make sure that the written data have been reliably stored.
3. Some applications require atomic replacement of contents of the file under the same name.

The filesystem operations necessary for case 2 cause significant performance hits, and furthermore they don't necessarily guarantee data persistency; that also depends on the storage hardware. Case 3, on Unix, requires creation of temporary files, which may be left in the filesystem in case the replacement sequence is aborted. I don't think std::fs::File should address those concerns with its basic functions.

[federicomenaquintero]

The filesystem operations necessary for case 2 cause significant performance hits, and furthermore they don't necessarily guarantee data persistency; that also depends on the storage hardware. Case 3, on Unix, requires creation of temporary files, which may be left in the filesystem in case the replacement sequence is aborted. I don't think std::fs::File should address those concerns with its basic functions.

Yes; this is why I suggested having a different crate for high-level, application-type atomic saves with rename(), creation of backup files, and everything.