Allow installing from main branches of fsspec and s3fs

[maxrjones]

Pinning fsspec at 2025.5.0 means that it is not possible to install from dev branches of both fsspec and s3fs, which is useful for downstream libraries testing upstream compatibility. Would you consider instead pinning to >=2025.5.0 or 2025.5.* instead?

[martindurant]

We do not want to accidentally pick up a potentially incompatible future release of fsspec, though. A previous suggestion, which does work for this use case (and we used in the past) was "2025.5.0.*", but this confuses some environment managers like poetry.
If you use git+http in the requirements for both fsspec and s3fs, the dependency here should not matter.

[maxrjones]

We do not want to accidentally pick up a potentially incompatible future release of fsspec, though. A previous suggestion, which does work for this use case (and we used in the past) was "2025.5.0.*", but this confuses some environment managers like poetry.
If you use git+http in the requirements for both fsspec and s3fs, the dependency here should not matter.

Makes sense. It's too bad that the solution to #957 seems to fix some environment managers like poetry but breaks others like uv and pixi. E.g., I get the following error when using git+http:

    × failed to solve the pypi requirements of 'upstream' 'osx-64'
    ├─▶ failed to resolve pypi dependencies
    ╰─▶ Because only s3fs==2025.5.0 is available and s3fs==2025.5.0 depends
        on fsspec==2025.5.0, we can conclude that all versions of s3fs depend
        on fsspec==2025.5.0.
        And because there is no version of fsspec==2025.5.0 and you require
        s3fs, we can conclude that your requirements are unsatisfiable.

with the specification in https://github.com/zarr-developers/VirtualiZarr/blob/43209b32b8d84f7f57f91942c5b8d3b6cf2db0f4/pyproject.toml#L92-L104.

We can just only test against upstream s3fs moving forward - zarr-developers/VirtualiZarr#602

[martindurant]

What tool is reporting that failure?

Another option, of course, it to install in two steps or with --no-deps (having previously installed all deps or the release versions).

[maxrjones]

What tool is reporting that failure?

uv is reporting the failure by way of pixi (which uses uv). Here's an MVCE:

touch example.py
uv add --script example.py 'fsspec @ git+https://github.com/fsspec/filesystem_spec' 's3fs @ git+https://github.com/fsspec/s3fs'
uv run example.py

Another option, of course, it to install in two steps or with --no-deps (having previously installed all deps or the release versions).

I'm not sure that would work since pixi and uv are lockfile based such that they're aware of versions installed by previous steps.

[martindurant]

Anecdotally, I have heard the sentiment "great for projects, difficult for library dev" of pixi and uv.

[maxrjones]

Anecdotally, I have heard the sentiment "great for projects, difficult for library dev" of pixi and uv.

fwiw I've been quite happy generally with pixi and uv for library development. I think that this error is accurate to the relationship between fsspec and s3fs right now. The requirements.txt in s3fs states that I should only be able to use the main branch with an exact tag of fsspec. I am trying to do something different in my pyproject.toml. uv is raising an error, which seems appropriate. Either I should request that s3fs changes its expectations (this closed PR) or change my expectations (e.g., zarr-developers/VirtualiZarr#602).

[d-v-b]

We do not want to accidentally pick up a potentially incompatible future release of fsspec, though. A previous suggestion, which does work for this use case (and we used in the past) was "2025.5.0.*", but this confuses some environment managers like poetry. If you use git+http in the requirements for both fsspec and s3fs, the dependency here should not matter.

What about picking up a compatible future release of fsspec? Presumably this is much more likely than an incompatible release, at least in the short time horizon, but we cannot do that as long as fsspec is pinned to an exact version.

And dealing with an incompatible future release of fsspec is relatively simple, from a library POV: you exclude exactly the incompatible version (up to a range, if you know that all future versions will be incompatible).

In the short term I will fork s3fs and change the deps myself

[martindurant]

And dealing with an incompatible future release of fsspec is relatively simple, from a library POV: you exclude exactly the incompatible version (up to a range, if you know that all future versions will be incompatible).

How can you possibly know that? For the longest time people were getting many-years-old s3fs 0.6.x because it had a ">=" dep on fsspec and something else in their environment prevented the latest version of [aiobotocore or whatever]. Fortunately, that particular case has ages out as python versions deprecated.

In the short term I will fork s3fs

better would be to change your install script. After all, s3fs is perfectly able to test itself against dev fsspec. Obviously, it's fine in my local envs too.

[maxrjones]

Does 2025.5.x (note difference from the previously used 2025.5.0.x) also cause issues with package managers? I thought the final x in CalVer usually indicates bug fixes such that there wouldn't be breaking changes sneaking through with that syntax.

[martindurant]

I thought the final x in CalVer usually indicates bug fixes

Usually, but it's not a guarantee. I don't tend to release more than once a month, but ...

[d-v-b]

i found this post helpful for explaining why an upper bound for a version can be problematic.

[martindurant]

The post is very long it will take some time for me to have a look. I am not doubting that ">=" has some advantages, but I am also pointing out that it definitely also has disadvantages, and this repo in particular has previously been hit by them.

[d-v-b]

in any case, our tests are running normally again (my fork was unnecessary in the end) :)

[d-v-b]

this is a problem for zarr's upstream tests again. I learned a bit more about how pip defines a version for a git dependency, and it seems that for versioneer-based versioning (which s3fs uses), pip will resolve a version based on tags and the latest commit.

So, as I understand it, since s3fs declares a dependency on exactly 1 version of fsspec, any commits to fsspec after that version is released will result in a newer, unsupported fsspec version from pip's POV, and therefore be incompatible with s3fs.

Relaxing the upper bound on the fsspec dependency would solve this, at the risk that some future version of fsspec might be incompatible with s3fs, but in this case, people affected by that problem can do something: exclude the incompatible version of fsspec.

[martindurant]

Relaxing the upper bound on the fsspec dependency would solve this, at the risk that some future version of fsspec might be incompatible with s3fs, but in this case, people affected by that problem can do something: exclude the incompatible version of fsspec.

Unfortunately, the size of the community using released s3fs is far far bigger than those who need dev s3fs and dev fsspec, so this is not a risk that we can take.

[d-v-b]

isn't it possible to test for incompatibility between s3fs and fsspec, and then update s3fs' dependencies accordingly?

[martindurant]

No, you cannot change the upper bound of the released version once it is on pypi. Did I misunderstand you? Perhaps you have a concrete suggestion of what to do.

[d-v-b]

right, i wasn't thinking about the inability to retroactively restrict dependencies for released versions.

I guess this issue turns on the likelihood that a future fsspec release breaks s3fs -- you seem to assign a very high likelihood to this. Over in zarr, all of our dependencies have an open upper bound. This means we are completely exposed to breaking changes in all those libraries, but this has not amounted to much work, because the rate of breaking changes has been low. So maybe that approach doesn't work here.

[martindurant]

I guess this issue turns on the likelihood that a future fsspec release breaks s3fs -- you seem to assign a very high likelihood to this

Rather, I would say that the cost of such a breakage, even for a couple of days until a new patch, is very high.

[dstansby]

One solution here would be if the version s3fs reported from a checkout of git always contained a suffix indicating that source code is from git instead of a source distribution or wheel. Looking at available versioneer styles it's not obvious to me that this is possible without some custom logic for getting the version in __init__.py though.

[martindurant]

Yes, it would be totally fine to use .dev1 versions in s3fs; and indeed to ditch versioneer in favour of something else. But I think it's the version of fsspec that's at issue, not s3fs. Actually, you want the source install not to add a suffix to the version string.

[d-v-b]

i think it's important and correct to track the difference between version X and X+commit hash, and to treat the latter as a newer version than the former. We can solve this on the zarr side by respecting the fact that the dev versions of s3fs and fsspec are not generally compatible, and testing them separately as needed.