Folder structure redesign

[haixuanTao]

As our API is mainly based on YAML. We can use this to our advantage in our folder structure as both are graph-based.

I think that we can have a folder structure as:

dora/
├─ communication/
│  ├─ zenoh/
├─ nodes/
│  ├─ c/
│  ├─ python/
│  ├─ rust/
├─ runtime/
│  ├─ operator/
│  │  ├─ c/
│  │  ├─ rust/

To this we can add the necessary complementary folder:

dora/
├─ communication/
│  ├─ zenoh/
├─ nodes/
│  ├─ c/
│  ├─ python/
│  ├─ rust/
├─ runtime/
│  ├─ operator/
│  │  ├─ c/
│  │  ├─ rust/
├─ common/
│  ├─ graph/
├─ daemon/
│  ├─ coordinator/
├─ examples/
├─ extensions/
│  ├─ serialisation/
│  ├─ telemetry/

I feel like this is pretty readable. What do you guys think @phil-opp @heyong4725 ?

[heyong4725]

Overall looks good to me, I made some updates:

dora/
├─ apis/ (dora sdk)
│  ├─ c/
│  ├─ rust/
│  ├─ python
├─ cli/
│  ├─ doractl/
│  ├─ .../
├─ docs
│  ├─ overview/
│  ├─ design/
│  ├─ .../
├─ communication/ (this is for generic MLA)
│  ├─ apis
│  ├─ pub-sub/
│  ├─ req-reply/
│  ├─ client-server/
│  ├─ point-to-point/
│  ├─ ...
├─ src/
│  ├─ nodes/
│  │  ├─ c/
│  │  ├─ python/
│  │  ├─ rust/
│  ├─ nodelet/
│  │  ├─ operator/
│  │  │  ├─ c/
│  │  │  ├─ rust/
│  ├─ dataflow/
│  │  ├─ graph/
│  │  ├─ streams/
│  │  ├─ policies/
│  │  ├─ .../
├─ daemons/
│  ├─ coordinator/
│  ├─ dora (dora-runtime)/
│  ├─ monitor/
│  ├─ e2e policy/
│  ├─ ...
├─ examples/
├─ extensions/
│  ├─ serialisation/
│  ├─ telemetry/

What do you guys think @phil-opp @haixuanTao ?

[phil-opp]

Looks good to me overall. Some things that I'm not sure about:

• The coordinator, runtime, etc. are our main executables, so I would not call them "deamon". Maybe we should name the directory something like bin/binaries/executables instead?
• What functionality do you expect in src/nodes and src/nodelet? The SDK/APIs are already under api/...
• We don't really have a "main crate" since dora's functionality is split across various libraries and executables. The top-level src folder makes it look like the current common crate is now the main crate, but I'm not sure if that'a a good idea since it is mostly an implementation detail.

[haixuanTao]

I feel like there is a bit of overlap between dora/apis, dora/communication/apis, and src/nodes and we should probably try to focus on a single API crate. Those API crates are particularly error-prone and maintenance burden. Especially if we convolute those API by the number of languages supported.

Those 3 APIs are all related to the communication to the dataflow. And therefore for the first iteration of dora we should probably only keep a dora/api crate which provides different levels of integration into the dataflow.

I would probably also rename src to something like core as src make me think of a compilation code source.

The rest looks great to me :)

[phil-opp]

My preference would be something like this:

dora/
├─ apis/ (dora sdk)
│  ├─ c/
│  ├─ rust/
│  ├─ python
├─ binaries/
│  ├─ dora-cli/
│  ├─ coordinator/
│  ├─ dora (dora-runtime)/
│  ├─ monitor/
│  ├─ e2e policy/
│  ├─ ...
├─ docs
│  ├─ overview/
│  ├─ design/
│  ├─ .../
├─ examples/
├─ libraries/
│  ├─ communication/ (this is for generic MLA)
│  │  ├─ pub-sub/
│  │  ├─ req-reply/
│  │  ├─ client-server/
│  │  ├─ point-to-point/
│  │  ├─ ...
│  ├─ core/
│  │  ├─ graph/
│  │  ├─ streams/
│  │  ├─ policies/
│  │  ├─ .../
│  ├─ extensions/
│  │  ├─ serialisation/
│  │  ├─ telemetry/

Splitting the crates into binaries and libraries is a bit technical, but I think this distinction is useful. The binaries provide the main functionality, while the libraries are mostly implementation details. The other folders have clear purposes as well: the apis provide the user-facing interface and make it possible to create operators and custom nodes, the docs explain the design of dora, and the examples show how it can be used.

[phil-opp]

Implementation in #46