Abstract filepath proposal (request for help)

[hasufell]

Motivation

The abstract filepath proposal strives to improve filepath handling in Haskell. The original proposal is written down in the GHC wiki and suggested implementing this in base.

Since this would break a lot of real world code, we (CLC and filepath maintainers) decided to implement this in "user-space", as in: only in filepath (a core library shipped with GHC). That means base will not use these new filepath types for the foreseeable future, but other core-libraries can.

The main benefits compared to type FilePath = String are:

1. it is more efficient and avoids memory fragmentation (uses unpinned 'ShortByteString' under the hood)
2. it is more type-safe (newtype over 'ShortByteString')
3. avoids round-tripping issues by not converting to String (which is not total and loses the encoding)
4. abstracts over unix and windows while keeping the original bytes

Previous discussions

• https://discourse.haskell.org/t/reviving-the-abstract-filepath-proposal-afpp-in-user-space/2344
• https://mail.haskell.org/pipermail/libraries/2021-August/031427.html
• https://groups.google.com/g/haskell-core-libraries/c/WzapcSvvfQM/m/oDhGbf9wCwAJ?pli=1

Previous work

1. extending bytestring API wrt ShortByteString: Merge shortbytestring package back into bytestring wrt #444 haskell/bytestring#471
2. AFPP merged into filepath: https://gitlab.haskell.org/haskell/filepath/-/merge_requests/103

Current state

1. a release candidate is available: https://hackage.haskell.org/package/filepath-2.0.0.3/candidate
   • extensive test suite: https://gitlab.haskell.org/haskell/filepath/-/tree/master/tests
   • benchmarks: https://gitlab.haskell.org/haskell/filepath/-/tree/master/bench
   • memory fragmentation stats: https://github.com/hasufell/filepath-debug/blob/master/result.txt
2. a companion library that allows file reading/writing is available here: https://github.com/hasufell/file-io
3. pending Win32 PR: Add WindwowsString/WindowsFilePath support wrt AFPP haskell/win32#198
4. pending unix PR: Add PosixFilePath and friends support (for AFPP) haskell/unix#202
5. GHC issue that needs to be solved: https://gitlab.haskell.org/ghc/ghc/-/issues/21738

The code is ready for release, PRs 3. and 4. can be merged in a timely manner (those are the minimum requirements to be able to use it).

Next steps

The idea is to get this release into GHC 9.4 or 9.6 (@Bodigrim I actually forgot which one). Users on older GHC will however be able to upgrade to the newer filepath anyway, because it's not tied to base.

After that, we need to market this new API and request core libraries maintainers to support AFPP (either as additional API or as the main API).

Some dicussions and PRs:

1. Add AFPP support haskell/directory#136 (has gone stale)
2. Support AbstractFilePath haskell/directory#138
3. Support OsString haskell/process#252
4. AbstractFilePath support kowainik/relude#401
5. Support AbstractFilePath commercialhaskell/path#189

This isn't a formal tech proposal, since the new API has been merged and will be part of the next filepath release. However, migrating the ecosystem can only be a joint effort. I'd imagine the next packages that need adoption are directory and process at least. I can't do all these migrations on my own, unless I intend to burn out.

I'm not sure what exactly I'm asking, but I talked with @david-christiansen during ZuriHac that this needs wider support. So I'm dropping this thread here.

I intend to write a blog post after the release that summarizes the work done, outlines how to use the new API and suggests migration strategies.

[Bodigrim]

The idea is to get this release into GHC 9.4 or 9.6 (@Bodigrim I actually forgot which one).

It's GHC 9.6.

[Bodigrim]

FWIW IMO AFPP is "the next big thing" for Haskell core libraries, huge thanks to @hasufell for leading this project.

[frasertweedale]

Love your work, this looks great. I think there's a need for a AbstractFilePath -> Data.ByteString.Builder.Builder in the main module, and I filed a ticket to track that.

[Ericson2314]

filesystem has a copy of ShortByteString inlined into it? Err I see it still depends on bytestring, but System.AbstractFilePath.Data.ByteString.Short is no mere reexport. I am confused.

Also I opened up https://gitlab.haskell.org/haskell/filepath/-/issues/117 with some questions that might be good to resolve before the release.

[hasufell]

filesystem has a copy of ShortByteString inlined into it? Err I see it still depends on bytestring, but System.AbstractFilePath.Data.ByteString.Short is no mere reexport. I am confused.

It is a re-export. What you're looking at is a Word16 based API variant that was not accepted in bytestring, because it's too specific. This one is needed to deal with wide character bytestrings (UCS-2LE) on windows: https://gitlab.haskell.org/haskell/filepath/-/blob/master/System/AbstractFilePath/Data/ByteString/Short/Word16.hs

[Ericson2314]

Mmm IMO having to conform to bytestring's scope is another reason why it was a mistake for ShortByteString to be merged back into bytestring.

If it was a separate repo, it would be very easy to use some build time tricks to get e.g. short-bytestring and short-bytestring-16 libraries built deduplicated code, and then bytestring can only only depend on short-bytestring not short-bytestring-16.

[mpickering]

Has it been considered not doing a major version bump to filepath-2.0?

It is clearly an improvement to be able to use this abstract datatype for FilePath manipulation but I am worried that it will cause disruption in the ecosystem for a project which doesn't yet have significant buy-in when the same feature could be introduced with a minor version bump.

The API of FilePath has been largely stable for a number of years so disrupting this I think requires great care.

[Ericson2314]

I agree major bumps without breaking changes to force them are probably not worth the churn. It would be nice if there was a separate mechanism to signal "exciting release!" than that to signal "breaking release!", but that doesn't yet exist.,

[hasufell]

Has it been considered not doing a major version bump to filepath-2.0?

It's possible. My reasoning was mainly to get more attention. And a major bump will definitely do that.

PVP doesn't disallow this, at least. @Bodigrim opinions?

[hasufell]

Mmm IMO having to conform to bytestring's scope is another reason why it was a mistake for ShortByteString to be merged back into bytestring.

If it was a separate repo, it would be very easy to use some build time tricks to get e.g. short-bytestring and short-bytestring-16 libraries built deduplicated code, and then bytestring can only only depend on short-bytestring not short-bytestring-16.

Yes, I suggested something along those lines here haskell/bytestring#444

This was rejected and I think we're fine for now.

[Ericson2314]

Yes I said "another" because I had already offered a different reason there :). I would strongly advocate revisiting that decision.

[hasufell]

I've added some examples that prove that the current base library is unsound wrt filepaths when interacting with CP932 (which is the most common japanese encoding it seems): https://gist.github.com/hasufell/c600d318bdbe010a7841cc351c835f92

This also demonstrates that abstract filepath fixes these issues.