Preferred file architecture for our apps

[Mohammer5]

I've talked to @ismay about what we think is the cleanest way to design our apps in regards to file/folder architecture.
What we've agreed on so far is:

Pages should be organized similarly to what next.js does

On the root level there should be a src/pages (or routes or views) folder.
Inside the components the organization is similarly to what the navigation tree looks like:

In Next.js, a page is a React Component exported from a .js, .jsx, .ts, or .tsx file in the pages directory. Each page is associated with a route based on its file name.
Example: If you create pages/about.js that exports a React component like below, it will be accessible at /about.
...
Next.js supports pages with dynamic routes. For example, if you create a file called pages/posts/[id].js, then it will be accessible at posts/1, posts/2, etc.

In our case, there's no magic that does that for us, but we like the structure. It's clear, scalable and easy to reason about.
Additionally this is where integration tests can be done as the components in the pages folder are MVC-Controller-like (composing functionality from building blocks)

src/components and src/utils

In order to be able to compose the pages, all our apps need reusable components. @ismay's and my preferred approach would be to have a ducks-like structure inside the components folder (group things by namespace).

The src/utils folder is an optional third folder, which can be used when code doesn't fit into one of the component-folder's namespaces (e. g. array helpers, etc)

Hooks

With graphql it's good practice that every component defines its own query when it needs data. Abstracting the data fetching into shared / re-usable hooks can have undesired side effects (like loading data that's not needed when changing the query slightly for a different area of the app), so instead of also adding a src/hooks folder, it's better to co-locate components with their query/mutation definitions.

Forms

Instead of using configs, we prefer if forms are implemented using components. This way forms are lot easier to implement and to reason about, dynamic forms always have been a pain in my experience.
Using components instead of configs, forms become scalable

[ghost]

To elaborate a little on the above, from my perspective:

• The src/components and src/pages distinction. I find this really helps with organisation. You can start building in pages without caring about abstractions, and then if reusable patterns emerge, they can be abstracted to generic components. This saves you from abstracting too early. You can just starting building out pages for the app, and then only when a proper pattern has emerged do you abstract it. Code that is page specific can just be left in pages.
• Organizing by feature instead of by type (for redux this is called ducks). I've used something similar quite a while ago with ember. They call it pods: https://cli.emberjs.com/release/advanced-use/project-layouts/#podslayout. And this article delves into organizing by feature from a react perspective: https://jaysoo.ca/2016/02/28/organizing-redux-application (you can ignore the redux specific parts). I find it tends to lead to a more maintainable project structure.
• Keep the folder structure flat. I usually try to have just one level under src/components, src/pages, etc. I personally find it horrible to work on an app and discover mindbending levels of nesting under a component that initially seemed rather innocuous.
• The jaysoo.ca article I mentioned above mentioned it as well, but I think it deserves an explicit mention. I like creating an explicit boundary between components, pages, etc. by creating an index.js for each component (and page, etc), and only importing from there. You can enforce this with linting. It makes it very clear what the external API is for a component and prevents you from inadvertently creating react spaghetti.
• Declarative app structure. I feel overly intelligent components have been a major source of complexity in the dhis2 apps I've worked on. Trying to make the apps dynamic by adapting to what we receive from the backend is a nice idea, but I think apps with a declarative structure are a lot easier to work on. I think we should forego the idea of an app that adapts to whatever the backend throws at it, I think our apps are too complex for that (or I am too dumb for it). Plus, I personally feel it fits better with react's paradigm.

Also, I think we shouldn't be too dogmatic about the above. There are always exceptions, and I think following architectural rules to the letter can end up worse than making a practical exception. The end goal is to make anyone working on your code hate you as little as possible.

[jenniferarnesen]

• (you can ignore the redux specific parts)

@ismay I'm curious about your thoughts on how to structure the redux. I've been thinking about doing a restructure to feature-based architecture in the dashboard app, and after reading the above articles, I decided to give it a quick go just to get an idea. I like the result so far, but I'm not sure what to do with the redux, since actions and selectors are often used by several components/pages (that's the idea of redux - sharing state across the app, isn't it?). I could at least co-locate the reducers and actions/operations, but it seems like it still needs to be separate from the pages and components.

[jenniferarnesen]

Also, I remember hearing hints from @amcgee and others that some apps might phase out redux (preferring context or component state?). I'd like to talk more about how that idea fits into this file architecture discussion.

[Mohammer5]

Also, I remember hearing hints from @amcgee and others that some apps might phase out redux (preferring context or component state?). I'd like to talk more about how that idea fits into this file architecture discussion.

I'm not sure if we already have apps that transitioned from redux to app-runtime or from centralized state to declarative data fetching on the component level, but we already have a couple of apps with slightly different implementations.
Some apps don't need global/shared state at all (like the sms configuration app) while others use a bit of shared state for UI or caching improvements (the latter should not be that much, of an issue soon, I hope) while the former can be done with context quite easily (and additionally with useReducer if complexity increases a little bit).

I think we have to differentiate between apps that display database state or allow the user to manipulate database state (like the sms configuration app) and apps that have actual client side state that will never be stored in the backend's database.
With the app runtime the actual database becomes the centralized state manager, while redux might still make a lot of sense if complex client side state needs to be handled.

I think - at least on team platform's side - we have not encountered an app yet where redux would be benefitial because most apps actually just display database state and allow the user to manipulate it through forms. I'm wondering about what would be best for data capture and data presentation apps

@ismay I'm curious about your thoughts on how to structure the redux.

I'm not ismay 😅 Ismay and I were talking about what structure we'd prefer inside the pages folder and we had different ideas, so wie thought we just try them in fairly small apps and compare them. I'm talking about the current status here. I actually like the approach right now because it allows the project to stay quite clean without too many abstraction. I think with redux, I'd follow the same approach and would just add a redux folder to the modules that exist (in this case there's sms-command, sms-gateway, sms-received, sms-sent & shared, each of these folders could have a redux folder as well, with their reducers, actions (& middleware helpers, like thunks).
But that's not really a recommendation, as I just want to add my current approach and I'm curious what others think about it.

[ghost]

I'm not sure if we already have apps that transitioned from redux to app-runtime or from centralized state to declarative data fetching on the component level

The scheduler used to use redux. Most of it was for form state, which can be handled with final-form (or something like that). The rest could be handled with local state.

[ghost]

• (you can ignore the redux specific parts)

@ismay I'm curious about your thoughts on how to structure the redux. I've been thinking about doing a restructure to feature-based architecture in the dashboard app, and after reading the above articles, I decided to give it a quick go just to get an idea. I like the result so far, but I'm not sure what to do with the redux, since actions and selectors are often used by several components/pages (that's the idea of redux - sharing state across the app, isn't it?). I could at least co-locate the reducers and actions/operations, but it seems like it still needs to be separate from the pages and components.

@jenniferarnesen I don't really have a general answer. A lot of it depends on how redux is used in the app. In the scheduler the form state was stored in redux, as was data from the backend, and I believe some app state.

Everything that came from the backend was refactored to use the data-query hook. And the form state handled with final-form. For everything else local state was good enough. There was a tiny amount of state that needed to be shared throughout the app, that was done with an app wide store that shares state through context. This state here, which is accessed here via these hooks. Note that this store currently also fetches and distributes all the data, but that's just until we have caching in the data-query hook.

Factoring out redux can be quite a lot of work, as it's usually threaded throughout the app and seems to have quite an influence on the app's architecture. Though, if you have a ducks redux architecture I think it should be a lot easier to migrate, as that concept is quite close to the kind of organization that seems to work well with the app-platform.

If the app really depends on a centralized store for it's state, and not just the data, you could create a store abstraction with the app-platform (similar to what I've done for the scheduler), and get the same functionality. If you want to keep it very close you could even use useReducer hooks. Maybe that would help transitioning apps that are too complex to transition in one go?

I think it's probably good for us all to take a look at the actual codebases, get involved with the migration process. Until now we've mostly been figuring it out as we go. Probably good to get some consensus and formulate some general recommendations.

[amcgee]

This is a great discussion 👆, you should all read it. There are at least 10 things you wouldn't believe in there

• @ismay

I couldn't agree more, great salient points! Succinct and direct, not too many rules to follow.

[ghost]

This is a minor thing, but we use different standards of folder and filename naming (extensions, capitalizations, etc.). Would be nice to settle on a single convention and just lint that. Say with something like https://ls-lint.org, https://github.com/sindresorhus/eslint-plugin-unicorn/blob/main/docs/rules/filename-case.md or similar.

[varl]

ls-lint looks great.

I think we should keep ESLint's job very specific to avoid extremely verbose ESLint logs.

[ghost]

Yeah I agree, ls-lint seems better suited for the job.

[ghost]

@Mohammer5 and I were talking about this, and thought that once we've settled on something we all like, we could test it out on:

• https://github.com/dhis2/sms-configuration-app
• https://github.com/dhis2/usage-analytics-app
• https://github.com/dhis2/scheduler-app

Those are all a little different, but all modern apps. Could make for a good way to test whether what we've decided on is practical.

[Mohammer5]

@ismay, @HendrikThePendric and I talked about this and what we figured out deviates from the current proposal:

Why are we talking about this in the first place?

Before talking about the solutions, we have to have a problem in the first place.
Our consensus was: When switching from app to app, often we find ourselves a bit lost until we understood the organizational aspects of the files- & folders-architecture

We found the following problems

• Existing apps are not in sync (in terms of how they're organized)
• Organization is hard to understand in some cases
• We don't have a guide for how to organize new apps

Issues with defining a clear rule set

Ideal organization within an app depends on the size of an app

@ismay found some good articles about this topic while doing some research.

https://www.robinwieruch.de/react-folder-structure

This article highlights an important aspect of organizing apps: It really depends on the scale of the project.
Of course we don't really have 1-file apps, but we still have apps that vary in complexity.
Good examples are:

• Scheduler app - Handles one dhis2 domain: Cron jobs
• Sms Config app - Handles multiple, but few dhis2 domains: sms gateways, sms commands & in-/outbound sms
• Maintenance app - Handles an awful lot of dhis2 domains; too many to list

The reason why this is important is: There's no one size fits all solution

It's hard to find a definitive distinction between different variations of complexity

As described above, different levels of complexity require different approaches to organizing the code.
There are many blurry lines between the levels of complexity of our apps, it's more or less impossible to have a complexity to organization mapping that developers could refer to.

If we want to define rules how to organize an app, we need to find a rule set that's unambiguous. And this is very things get really tricky. It was quite straight forward to us to extract rules from existing apps, but we quickly figured out that it doesn't necessarily make sense to apply these rules to different apps.

Some other apps with a recent overhaul that have different levels of complexities are:

• Cache cleaner app (1 route only)
• Usage analytics app

Alternative approaches to having a strict rule set

The actual organization is not that important, it "just" has to follow a few abstract rules

To me it always feels like react components are more a vertical slice of the MVC pattern (as in, by concern or function), as opposed to being organized horizontally (by technology, as in: model, view or controller).
(source)

As our apps or basically react component (the index file is a react component), the organization should be by "concern" or "function"

It doesn't matter as long as it's clear & consistent.
(me 😎)

As different developers do things differently anyway, we shouldn't enforce a static app organization. Instead we should focus on having examples for different degrees of complexity and instead keep in mind that apps are also "living" and are bound to change over time, so it's really about being able to adapt easily to new requirements, also on the organizational level

Maybe it's not as much about formulating rules as it is about creating a shared consensus.
I do think it's a good topic to share opinions on.
Maybe the end goal shouldn't be to end up with rules. Maybe it should be more about creating app structures that we're all happy with.
And then perhaps rules would follow from that (or not)
(Ismay)

I guess what we could do is to highlight different solutions, everyone can share opinions on the solutions, add their own solution to the discussion so we have both a catalog of solutions for different problems as well as a discussion about what we like the most. We might come to a conclusion that some guidelines and/or rules apply to all scales, we might find some complexity levels that we can generalize and turn into guidelines and/or rules.

[HendrikThePendric]

Thanks for that @Mohammer5. This summarises what we discussed yesterday very well.

We established that defining organisation rules is probably not feasible at this point in time, but outlining some guidelines should already be possible. So, I think we could I think we can define the following stages:

1. We declare some very generic guidelines about project structure
2. We try it our in various new-builds and re-writes
3. We adjust guidelines according to our findings and try to extract rules

When we talked yesterday, we already identified a few potential guidelines. I'll paste them here:

1. Have similar organisation strategies between all our apps
2. Organise by-function, not by-type
3. Avoid ambiguity (i.e. no conceptual overlap in folder names)
4. Avoid generic folder names, and attempt to chose names that immediately reveal what’s inside
5. (Accept that a project-structure is not a static thing you can define up-front, the structure should evolve with the app)

[ghost]

Nice summary 👍. I agree, this kind of approach seems like the most appropriate. I feel that it hinges on us sharing our views on organization often, and not just one to one, but with the group. I wonder how doable that is, as we're all remote, and how best to help foster communication on the topic?