Don't implement support for multiple workspace

[tengstrand]

I've been working on supporting multiple workspaces (issue #431) for several months now, by pushing 20 snapshot releases to the master branch, and now I'm thinking of reverting all that work.

There are two main reasons for this:

• The codebase is now several times harder to understand and maintain, and is a pain to work with.
• We thought this was an important thing to do, but now I'm not so sure any more. I think that many people will choose to use a single workspace anyway. Frontend and backend artifacts will end up as different projects in the projects directory. All the bricks will be mixed in the components and bases directories.

The complexity price just got too high. See also discussion #301.

My number one design rule for the last twenty years has been "If it's ugly then it's wrong" ("Är det fult så är det fel" in Swedish). This time, that rule struck.

[seancorfield]

Well, I was very skeptical of the multiple workspaces (in a single repo) right from the get-go so I'm glad to hear you now think this is unreasonably complex to support. I was never convinced that mixed Clojure/Script projects really needed separate workspaces that shared code -- either a single workspace (for shared code) or completely separate repos (for no shared code).

I do accept that something needs to be done to support .cljs in a workspace alongside .cljc and .clj files, but it seems like that should just be a "dialect" in a project that controls how files are read (reader conditionals).

[mitchelkuijpers]

We never were looking for multiple workspaces but mainly for support of ClojureScript to be honest. I don't really see why you would need separate workspaces for backend and frontend. To me that seem like different projects.

[jasonjckn]

The only real need for us would be CLJS support in polylith, and even that, it's just 'nice to have'.

I think if I ever leveraged multiple workspaces feature, my team would think i've complected our codebase.

I haven't had time to dive deep yet into polylith support for publishing libraries, but that's something I would use, e.g. we have a sister java project that can reuse some of our clojure code packaged in a lib, and maybe that is also a good way to share code with a CLJS project, that's located in a different git repo.

[marksto]

Wow, this is terrible news! 😞 And it seems like I'm the only one who sees real value in how this feature can become a lever to benefit a real business use case. I've been waiting for it for all these months.

For my undertaking, this solution was a workable way to split a monorepo into multiple workspaces, most of which are for end products (with deliverables), and the remaining code is allocated to the common platform. This approach allows us to achieve the following goals:

1. Allow for a better distribution of work across the organization among engineers, with a separate platform team and independent product teams.
2. Allow product teams to stay focused on their part of the codebase and only the platform components they need.
3. Allow using the practice of incremental testing across all projects of the organization (on a dedicated CI server).

and probably even more. All these while preserving the main advantages that the Polylith architecture provides, i.e allowing the teams, especially the platform one, to make changes to shared components in a predictable and controlled manner, with immediate feedback from the entire codebase (thanks to the fact that it's still a monorepo and not a set of interdependent versioned libraries).

Leaving aside the organizational aspect, we can also talk about the goals related to structuring / factoring:

1. Allow to differentiate and delimit bricks in the workspace according to their relationship to the infrastructure, individual subject areas, different "ends" (FE, BE), etc.
   • This applies to "All the bricks will be mixed in the components and bases directories", which is already a pain point for some larger Polylith codebases.
   • Although this can be achieved in other ways, for example by introducing a tree structure into the components and bases subdirectories.
2. Allow extraction of entire groups of closely related blocks into their own homomorphic (sub)workspaces.
   • This opens the door to building (your systems around) "bigger things" such as reusable subsystems, frameworks, etc. that can be published (open-sourced) in their own right.

I understand that this level of complexity is not what an average Polylith workspace requires. Even the largest ones get away with what is available OOTB (salute, @seancorfield). And, of course, I will accept any decision you make, Joaquim @tengstrand.

[marksto]

My number one design rule for the last twenty years has been "If it's ugly then it's wrong" ...

Complex life forms definitely cross this line. For example, the human body is a sheer engineer nightmare. 😄

[tengstrand]

Hi @marksto!

Allow for a better distribution of work across the organization among engineers + Allow product teams to stay focused on their part of the codebase and only the platform components they need.

• I guess your concern is that if you have 200 components, and one part of the organisation is responsible for 30 of them, then they are less interested in the other 170. One possible solution could be to use views/projections, so that when you execute the info, deps, and libs commands, only the bricks that are part of the active view(s) are listed.
• Another solution could be to add support for sub directories in the components and bases directory, as you suggest.

Allow using the practice of incremental testing across all projects of the organization

This is already supported by a single workspace.

Allow to differentiate and delimit bricks in the workspace according to their relationship to the infrastructure, individual subject areas, different "ends" (FE, BE), etc.

• The simplest solution is of course to keep all components in a single components directory and prefix some with e.g. "fe-" and "be-".
• Could also be solved by using views or sub directories under the components directory, as described above.

that can be published (open-sourced) in their own right

• Polylith is mainly designed to be used within an organisation, and there are challenges associated with using Polylith for public libraries, discussed in issue Build isolated jar files out of projects by changing the top namespace #212.

Complex life forms definitely cross this line. For example, the human body is a sheer engineer nightmare. 😄

Yes, that's true, but fortunately we don't have to maintain the body as code!

[tengstrand]

We have discussed the idea of having sub directories in the components and bases directory, and our believe right now is that it's not the right way to go. Instead I think we can have views, that limits which components and maybe projects, to include in each command. When starting a shell, you could then give the active view like this poly view:team-x (or set it from the shell).

[edenworky]

In my view, this feature is critical both to my needs and to Polylith's future. Without this feature, sure, modules (components, bases, projects, views, etc.) can be broken down into bricks, so that as long as everything fits within a single repo everything's fine, but producing an source code artifact is limited to pulling together repo-internal bricks and external projects, but even the docs discourage generating more than one source code artifact from a workspace and recommends using multiple workspaces instead, which naturally without this feature is difficult to do.

While I agree that supporting (and necessitating this) is overkill just for cljs support, with this feature, workspaces are self-similar enough to where we won't be limited in composing and splitting workspaces even across repos, allowing e.g.:

• a single private repo to expose public workspaces as a source code artifact which is a public repo that can accept PRs independently and be pulled seamlessly into the main private workspace,
• or have multiple standalone public non-polylith repos which behind the scenes are all generated from small wrapping workspaces on some central private workspace which all generate the standalone repos as source code artifacts,
• as well as (perhaps most importantly) we could publish open-source workspaces that could be "plugged into" a local workspace as dependencies, rather than the clumsy indirection of having to first wrap a workspace into a more standard library dependency (which also loses brick granularity, so less useful editing, debugging, tree shaking, test incrementality, etc.).

Not that we usually have these things for external dependencies, but this feature would allow us to enjoy these benefits for external dependencies which happen to be polylith workspaces, which would enable a polylith-all-the-way-down situation as long as you stay within polylith land, so to speak.

All of this would require decoupling the concepts of 'repo' and 'workspace' thoroughly throughout the code, or else allow composability with some arbitrary restriction like "only one workspace per repo", and though I haven't went over the code I'm presuming most of aforementioned "ugly complexity" this feature necessitated is related to this, in particular as it relates to the CLI, so perhaps it is pertinent to make some concessions, but some form of this feature is critical to my using polylith and, I believe, to the future of polylith.

[seancorfield]

Just to clarify: you can have a Polylith workspace as a source dependency today. After all, https://github.com/seancorfield/polylith-external-test-runner is a Polylith workspace that is used as a (library) dependency. And Polylith itself, another workspace, documents how to use it as a source (git) dependency: https://cljdoc.org/d/polylith/clj-poly/0.2.21/doc/install#github-dependency

You can also use a local workspace as a :local/root dependency if you don't want to go the git route. I do this while developing Polylith projects and wanting to try them out in other local Polylith projects.

[tengstrand]

There is a possibility that we will add support for including bricks in projects from other workspaces, in the future. The question remains whether it's worth the extra complexity that this entails. I have ideas about how it could be implemented more simply than the attempt I made, but whether the need for such a solution is great enough and could be implemented simple enough, the future will tell.
Currently the main focus is to support cljs.

[marksto]

You can also use a local workspace as a :local/root dependency if you don't want to go the git route.

While this is true, be warned that tools.deps doesn't support "transitive" aliases. Which is kinda necessary for Polylith wss, since there's a convention to have all development dependencies under the :dev alias.