Consider moving `sway-lib-std` and `sway-lib-core` back into the `sway` repo to make synchronising under CI easier

[mitchmindtree]

Currently, we have a cyclic dependency between the sway repo and the sway-lib-core/sway-lib-std repos. std and core are used throughout the examples and tests under sway, and the two libraries implicitly depend on sway via the sway language and forc.

We should consider the possibility of moving the std and core libs back into the sway repo to make keeping everything in sync under CI easier and to avoid the need to open two PRs each time we make a change to either the libs or sway.

forc dependencies in git repo sub-directories

Seeing as Sway doesn't yet have an implicitly included prelude through which we can provide core and std, we'll need to solve how to refer to the sub-directory of a repo in forc's git depedencies to make this possible. This will be necessary to allow downstream projects to depend on std and core via git by pointing at the sway repo.

Cargo generally seems to "solve" this via their [workspace] feature, as crates within subdirectories are almost always a part of some top-level workspace. That said, workspaces are a much broader feature which aims to solve a lot of other problems too, so I'm not sure we should rely on waiting to land a similar feature in forc just to solve this.

An alternative might be to add something like a root attribute to forc's git dependency type, where root is a relative path from the repo to the subdirectory containing the manifest of the dependency. We could potentially re-use the existing path field for this rather than adding a root field, as long as we clarify the behaviour that when both path and git are specified, path is relative to the root of the specified git repo rather than the root of the current project.

In the sway repo examples and tests we could solve this by specifying the core and std dependencies via path, however this wouldn't solve anything for downstream projects.

[otrho]

Couldn't it be resolved with pinning? The libs don't necessarily require the latest Sway and could stick with 'stable' (though while it's still early days this isn't as true, but it should settle down).

Keeping the repositories separate and referenced simply via a git repo URL in the TOML is nicer organisationally, as is having separate repo meta, like issues and PRs.

[mitchmindtree]

Couldn't it be resolved with pinning?

Kind of, I think this is currently the idea behind pinning the std and core deps to a particular version in the E2E tests.

However, this only solves depending on sway-lib-std and sway-lib-core inside of sway, but doesn't solve the issue where updating sway or forc may break the ability to compile the master branch of sway-lib-core or sway-lib-std (e.g. due to a language change, or forc manifest change).

It also allows for the E2E test core and std deps to fall out of date as is currently the case, but perhaps that doesn't matter so much if those tests are specifically for sway the language, and not the libraries. Ideally it would be nice if we could use them to check for breakage in both the language and the core/std libs.

I think from the perspective of a user, I would assume that I should be able to download the current master of both sway, sway-lib-std and sway-lib-core and know that all three will be compatible, however this is difficult to guarantee while each are split across 3 repositories. If all are synchronised under one repo, I think it's much easier to offer that guarantee.

Keeping the repositories separate and referenced simply via a git repo URL in the TOML is nicer organisationally, as is having separate repo meta, like issues and PRs.

I think we can get the issue and PR organisational benefits via labels, but otherwise I'm not so sure I see much benefit to keeping core and std separate. It does make specifying them via git dependencies nice and easy at the moment, and this is partly what I was aiming to address in the second part of the OP, but I think this will only be an issue until we have a prelude anyway.

For the record, rust-lang keep their core and std libs under the main repo too, but they've been shipping them alongside the compiler for a long time now, and we're not doing that just yet.

[nfurfaro]

It also allows for the E2E test core and std deps to fall out of date as is currently the case, but perhaps that doesn't matter so much if those tests are specifically for sway the language, and not the libraries.

Indeed, now that we have a test harness in lib-std, all new tests added for the std lib will be added in that repo.
Additionally, existing std-lib tests in the sway repo could be migrated to the new harness for consistency.

[sezna]

I think perhaps moving them back is a good idea, as long as we are still able to reference them directly from a Forc.toml or introduce a prelude, as is discussed above. It seems like those are the only two options. I'm thinking the former is less work, as a prelude has the whole chicken-and-egg problem of compiling the standard library in order to compile the compiler... etc.

Cargo's [workspace] feature does somewhat rely on having a working package manager, as consumers of crates that exist in a workspace refer to the crate by its name on crates.io.

My 🪙 🪙 : I think the easiest way forward right now is to allow both subdirectories in git and local path dependencies which would take precedence over the git dependency when it is available. This solves all of the pain points I believe?

pain points that would be solved:

1. Painful iteration across multiple repos
2. Lack of enforced compatibility among the latest compiler, core, and std
3. lack of ability to specify a dependency that is not the top-level project in a github repository
4. unified testing

P.S. re: repo metadata. If we do this, I think the only way to solve the repo-meta issue Toby described is to use labels more.

[Voxelot]

The risk of overriding the path behavior when git is enabled is that local dev workflows could be broken if you have multiple forc projects in the same repo. For example, if I make a change in crate A and consume that in crate B, I'd have to push crate A to a branch before seeing those changes locally in crate B.

Although, it seems like they would use the same path in either case whether a git repo is involved or not?

[emilyaherbert]

I think from the perspective of a user, I would assume that I should be able to download the current master of both sway, sway-lib-std and sway-lib-core and know that all three will be compatible, however this is difficult to guarantee while each are split across 3 repositories. If all are synchronised under one repo, I think it's much easier to offer that guarantee.

This hits the nail on the head of what I think our focus should be.

I think perhaps moving them back is a good idea, as long as we are still able to reference them directly from a Forc.toml or introduce a prelude, as is discussed above. It seems like those are the only two options. I'm thinking the former is less work, as a prelude has the whole chicken-and-egg problem of compiling the standard library in order to compile the compiler... etc.

Sure, but I do think that we should plan to support prelude in the future.

My 🪙 🪙 : I think the easiest way forward right now is to allow both subdirectories in git and local path dependencies which would take precedence over the git dependency when it is available.

To clarify, are you saying that the user could have core and std in their [dependencies] both pointing to GitHub, and forc still would still elect to use either the local version (or the git subdir) version instead? I think that this is fine for now while waiting for a solution like prelude. However, what I want to avoid is a situation in which the user thinks that they are indeed using the Github versions (in the event that we just added a new feature or something that hasn't been pushed to crates.io) and the compiler is failing and producing error messages that aren't relevant. I think if forc does decide to prefer a different version, it should tell the user that it has found a local version (or a git subdir) and is compiling with that version, in the compiler messages.