Update DISCO documentation

.github/workflows/lint-test-build.yml

@@ -23,7 +23,7 @@ jobs:
         with:
           path: example_training_data
           key: training_data
-      - run: ./get_training_data
+      - run: ./get_training_data.sh
         working-directory: ./
 
   lint-lib-core:

DEV.md

@@ -1,10 +1,5 @@
 <div align="center">
-  <h1><code>Disco</code></h1>
-
-  <p>
-    <strong>Distributed collaborative learning</strong>
-  </p>
-
+  <h1>DISCO <code>developer guide</code></h1>
   <p>
     <a href="https://github.com/epfml/disco/actions/workflows/lint-test-build.yml"><img src="https://github.com/epfml/disco/actions/workflows/lint-test-build.yml/badge.svg" alt="build status" /></a>
     <a href="https://github.com/epfml/disco/actions/workflows/deploy-server.yml"><img src="https://github.com/epfml/disco/actions/workflows/deploy-server.yml/badge.svg" alt="build status" /></a>
@@ -13,44 +8,158 @@
 
 </div>
 
-Welcome to the Disco🔮 developer guide. 
-
-If you want to get an in depth guide of how to run things (and why and how they work) you can have a look at our [on boarding](./docs/ONBOARDING.md) document
+Welcome to the DISCO developer guide. 
+Here you will have a first overview of the project, how to install and run an instance of DISCO and links to further documentation.
 
-To quickly get up and running you can find some relevant information here as well as in the [server](./server/README.md) document.
-
-If you run into any sort of trouble then hopefully you can find an answer in our [faq](./docs/FAQ.md); otherwise please create a new issue. If you want to contribute to Disco🔮, then please have a look at our [contributing](./docs/CONTRIBUTING.md) guide; and if you are curious about our architecture you can find information [here](./docs/ARCHITECTURE.md).
-
-## Quick-start guide
-
-The following command lines will install the required dependencies, build Disco.js and start the Disco server (on localhost:8080) and the Disco web client (on localhost:8081). We recommend using (and assume you are) [nvm](https://github.com/nvm-sh/nvm) (the Node Version Manager).
+## Structure
 
+The DISCO project is composed of multiple parts. At the root level, there are four main folders: `discojs`, `server`, `web-client` and `cli`.
+
+- `discojs`, or Disco.js, is the TypeScript library that contains federated and decentralized learning logic. The library allows to train and use machine learning models in a distributed fashion. The library itself is composed of the `disco-node` and `disco-web` modules, both of them extending the platform-agnostic code in `disco-core`. In other words, `disco-core` contains most of the implementation but can't be used by itself, while `disco-web` and `disco-node` allow using `disco-core` via different architectures. To some extents, you can think of `disco-core` as an abstract class extended by `disco-web`and `disco-node`.
+    - `disco-node` lets you use Disco.js with Node.js. For example, the `server` and the `cli` rely on `disco-node`. A user can also directly import the `disco-node` package in their Node.js programs.
+    - `disco-web` allows using Disco.js through a browser. The `web-client`, discussed below, relies on `disco-web` to implement a browser UI.
+
+  The main difference between the two is how they handle storage: a browser doesn't have access to the file system (for security reasons) while a Node.js application does.
+- `server` contains the server implementation necessary to use Disco.js. Indeed, while the federated and decentralized learning logic is implemented by Disco.js, we still need a server to orchestrate users in both paradigms. In decentralized learning, the server exposes an API for users to query the necessary information to train models in a decentralized fashion, such as the list of other peers. Thus, the server never receives training data or model parameters. In federated learning, the server receives model updates but never training data. It keeps track of participants and updates the model weights. A `server` instance is **always** necessary to use DISCO, whether one is using a browser UI, the CLI or directly programming with `disco-node`.
+- `web-client` implements a browser User Interface. In other words, it implements a website allowing users to use DISCO without coding. Via the browser, a user can create and participate in federated and decentralized training sessions, evaluate models, etc.
+- `cli` contains the Command Line Interface for Disco.js. For example, the CLI allows a user to create and join training sessions from the command line, benchmark performance by emulating multiple clients, etc.
+
+Here is a summary diagram:
+
+```mermaid
+flowchart LR
+  subgraph discojs
+    discojs-node-->|extends|discojs-core;
+    discojs-web-->|extends|discojs-core;
+  end
+  subgraph User Interface
+    web-client-->|uses|discojs-web;
+    custom_browser["custom browser implementation"]-->|uses|discojs-web;
+    server-->| uses |discojs-node;
+    cli-->|uses|discojs-node;
+    custom_node["custom Node.js scripts"]-->|uses|discojs-node;
+  end
+``` 
+
+## Installation guide
+
+The following instructions will install the required dependencies, build Disco.js and launch a DISCO server and a web client. If you run into any sort of trouble check our [FAQ](./docs/FAQ.md); otherwise please create a new issue or feel free to ask on [our slack](https://join.slack.com/t/disco-decentralized/shared_invite/zt-fpsb7c9h-1M9hnbaSonZ7lAgJRTyNsw). 
+
+**1.** We recommend using [nvm](https://github.com/nvm-sh/nvm) (Node Version Manager) to handle multiple Node.js versions. Start by installing `nvm` by following [their installation instructions](https://github.com/nvm-sh/nvm).
+After installation, you should be able to run
+```
+nvm -v
+0.39.7 # my nvm version at the time
+```
+**2.** Install Node.js version 16
+```
+nvm install 16
+```
+You can now choose which Node.js version to use:
+```
+nvm use 16
+```
+Using Node.js v16 should automatically set your [npm](https://docs.npmjs.com/about-npm) (Node Package Manager, different from n**v**m) version to 8:
+```
+npm --version
+8.xx.xx
+```
+`nvm` manages your different Node.js versions while `npm` handles your different Node.js project packages within one version.
 
+**3.** Clone the repository
 ```
 git clone git@github.com:epfml/disco.git
 cd disco
-nvm use
-cd discojs && npm ci && npm run build
-cd ..
-cd server && npm ci && npm run dev
-cd ..
-cd web-client && npm ci && npm run dev
 ```
 
-For full details, see the respective README files, that is [discojs-core](./discojs/discojs-core/README.md), [discojs](./discojs/discojs/README.md), [discojs-node](./discojs/discojs-node/README.md), [server](./server/README.md), [web-client](./web-client/README.md) and [cli](./cli/README.md).
+**4.** Run the installation script
+```
+./install.sh
+```
+The [Installation Script](#installation-script) Section goes over what <code>install.sh</code> does. If you are planning to contribute to DISCO, it will certainly prove useful to understand what the script does.
 
-## Structure
+**5.** Launch DISCO
 
-The Disco GitHub repository is composed of four main directories:
+As you may have seen, there are many ways to use DISCO. Here we will run a server and a web client. From there, a user can use DISCO from their browser.
+* First launch a `server` instance, which is used for federated and decentralized learning tasks, e.g. to list peers participating in a decentralized task.
+```
+cd server
+npm run dev
+```
+The server should be listening on `http://localhost:8080/`.
+* Secondly, start a web client, which will allow you to use DISCO from your browser. You may have to do so **from another terminal** since the previous one is now used by the server.
+```
+cd web-client # from another terminal
+npm run dev
+```
+The web client should be running on `http://localhost:8081`, if not first restart the server and then the web client.
+
+> [!IMPORTANT]
+> Make sure to first start the server to ensure that it is listening to port 8080.
+
+**You can now access DISCO at http://localhost:8081/**
+
+
+## Further documentation
+
+* Next you may want to read our [onboarding guide](./docs/ONBOARDING.md) which lists the following steps to onboard DISCO. 
+* If you are only planning to use DISCO in your own scripts, you can find a stand-alone example relying on `discojs-node` [here](./docs/node_example). The example runs with Node.js outside any browser, using the `@epfml/discojs-node` NPM package and the `server` module. A DISCO server is launched by the script itself and the data is already available in the repo.
+
+#### Table of contents
+As there are many guides in the project, here is a table of contents referencing them all:
+* [DISCO README](./README.md)
+* [Developer guide](./DEV.md)
+* The `docs` folder contains in-depth documentation on the project:
+	* [Onboarding guide](./docs/ONBOARDING.md)
+	* [Contributing guide](./docs/CONTRIBUTING.md)
+	* [Disco.js under the hood](./docs/DISCOJS.md)
+	* [FAQ](./docs/FAQ.md)
+	* [Example: using `discojs-node` in a script](./docs/node_example/README.md)
+	* [Privacy in DISCO](./docs/PRIVACY.md)
+	* [How to create a DISCO Task](./docs/TASK.md)
+	* [Vue.js architecture](./docs/VUEJS.md)
+* Respective `README` files contain installation and packaging instructions relevant to the module
+	* [`discojs` README](./discojs/README.md)
+		* [`discojs-core` README](./discojs/discojs-core/README.md)
+		* [`discojs-node` README](./discojs/discojs-node/README.md)
+		* [`discojs-web` README](./discojs/discojs-web/README.md)
+	* [`server` README](./server/README.md)
+	* [`web-client` README](./web-client/README.md)
+	* [`cli` README](./cli/README.md)
+
+## Installation Script
+
+The installation script installs the dependencies required by the different parts of the project, which are described in the [Structure Section](#structure).
+It first installs the Disco.js library dependencies, notably, `TensorFlow.js`, and anything else required for federated and decentralized learning logic. 
+The script then builds the library, a step necessary to compile TypeScript into JavaScript.
 
-- [discojs](./discojs/README.md) contains Disco.js, the main library used by the other projects within this repo. It is also available as standalone libraries on NPM, for both the browser and Node.js.
-  - [discojs-web](./discojs/discojs-web/README.md) contains the browser implementation of the Disco.js library. It is available on NPM as `@epmfl/discojs`.
-  - [discojs-node](./discojs/discojs-node/README.md) contains the Node.js implementatino of the Disco.js library. It is available on NPM as `@epfml/discojs-node`.
-  - [discojs-core](./discojs/discojs-core/README.md) contains code common to `discojs-web` and `discojs-node`. Little source code changes between the two implementations, hence the need for a "core" directory.
-- [server](./server/README.md) contains the Node.js server for Disco, enabling the decentralized (with peer-to-peer coordination) and federated learning of Disco.js nodes. It uses `discojs-node` under the hood.
-- [web-client](./web-client/README.md) contains a web client to interact with Disco.js and the Disco server via a user-friendly UI. This basically emulates a Disco.js node in the network. It uses `discojs-web` as backend.
-- [cli](./cli/README.md) contains the CLI tool for Disco.js. Just like `web-client`, it emulates a Disco.js node in the network. Furthermore, it can actually emulate multiple nodes and even manages its own Disco server instance. It uses `discojs-node` under the hood.
+```
+cd discojs
+npm ci # stands for `clean install`, to ensure that only expected dependencies are being installed.
+npm run build
+```
+The script then installs dependencies for the web client, which implements a browser UI.
+By default, the project points to the [@epfml/disco-web](https://www.npmjs.com/package/@epfml/discojs) package published on the `npm` remote repository. In a development environment, we want to use the local web client in the `web-client` folder. To do so, we need to link the local folder as the actual dependency.
 
-## Example
+```
+cd ../web-client
+npm ci
+npm link ../discojs/discojs-web
+```
+You can verify than the link is effective by checking that `npm ls` lists `@epfml/discojs@x.x.x -> ./../discojs/discojs-web`.
 
-A full -- self-contained -- examples of the Disco.js API running two federated users can be found [here](./docs/node_example). This runs on Node.js outside of any browser, using the `@epfml/discojs-node` NPM package and the `server` module of this repo. A Disco server is run from the script itself and the data is already available in the repo.
+Similarly, we install the server dependencies, and then the `discojs-node` dependency to the local folder rather than the remote npm package [@epfml/disco-node](https://www.npmjs.com/package/@epfml/discojs-node):
+```
+cd ../server
+npm ci
+npm link ../discojs/discojs-node
+```
+Install the CLI dependencies:
+```
+cd ../cli
+npm ci
+```
+Download and extract the sample training datasets. These datasets are used in the automated tests.
+```
+./get_training_data.sh
+```

README.md

@@ -1,4 +1,4 @@
-# **DISCO** - _Dis_​tributed _Co_​llaborative Machine Learning
+# **DISCO** - DIStributed COllaborative Machine Learning
 
 
 
@@ -9,7 +9,7 @@ The latest version is always running on the following link, [directly in your br
  :man_dancing: [https://epfml.github.io/disco/](https://epfml.github.io/disco/) :man_dancing:
 
 ___
-:magic_wand: DEVELOPERS: Contribute or customize DISCO [HERE](DEV.md)
+:magic_wand: DEVELOPERS: Have a look at our [developer guide](DEV.md)
 ___
 
 :question: **WHY DISCO?** 
@@ -30,7 +30,7 @@ ___
 :question: **DISCO TECHNOLOGY** 
 - DISCO supports arbitrary deep learning tasks and model architectures, via [TF.js](https://www.tensorflow.org/js)
 - :sparkles: relies on [peer2peer](https://peerjs.com/) communication
-- Learn more about secure aggregation and differential privacy for privacy-respecting training [HERE](docs/PRIVACY.md)
+- Have a look at how DISCO ensures privacy and confidentiality [HERE](docs/PRIVACY.md)
 
 ___
 
@@ -43,23 +43,22 @@ DISCO aims to enable open-access and easy-use distributed training which is
 - :ninja: robust to malicious actors and data poisoning ([R6](https://arxiv.org/abs/2012.10333), [R7](https://arxiv.org/abs/2006.09365))
 - :apple: :banana: interpretable in imperfectly interoperable data distributions ([R8](https://arxiv.org/abs/2107.06580))
 - :mirror: personalizable  ([R9](https://arxiv.org/abs/2103.00710))
-- :carrot: fairly incentivizes participation
+- :carrot: fairly incentivize participation
 
 
 ___
 
 
 :checkered_flag: **HOW TO USE DISCO**
-- Start by exploring our example *DISCOllaboratives* in the `Tasks` tab. 
+- Start by exploring our example *DISCOllaboratives* in the [`Tasks` page](https://epfml.github.io/disco/#/list). 
 - The example models are based on popular datasets such as [Titanic](https://www.kaggle.com/c/titanic), [MNIST](https://www.kaggle.com/c/digit-recognizer) or [CIFAR-10](https://www.kaggle.com/pankrzysiu/cifar10-python)
-- It is also possible to create a custom task without coding. Just upload the following 2 files:
-	- A `TensorFlow.js` model file in JSON format (useful links to [create](https://www.tensorflow.org/js/guide/models_and_layers) and [save](https://www.tensorflow.org/js/guide/save_load) your model)
-	- A weight file in `.bin` format
-		- These are the initial weights provided to new users joining your task (pre-trained or random initialisation) 
+- It is also possible to create your own task without coding on the [custom training page](https://epfml.github.io/disco/#/create):
+	- Upload the initial model
 	- You can choose from several existing dataloaders
-	- Then...select your DISCO training scheme (:star2: or :sparkles:) ... connect your data and... :bar_chart:
+	- Choose between federated and decentralized for your DISCO training scheme ... connect your data and... done! :bar_chart:
+	- For more details on ML tasks and custom training have a look at [this guide](./docs/TASK.md)
 
-> **Note**: Currently only `CSV` and `Image` data types are supported. Adding new data types, preprocessing code or dataloaders, is accessible in developer mode (see [developer guide](https://github.com/epfml/disco/blob/develop/DEV.md)). Specific instructions on how to build a custom task can be found [HERE](docs/TASK.md)
+> **Note**: Currently only `CSV` and `Image` data types are supported. Adding new data types, preprocessing code or dataloaders, is accessible in developer mode (see [developer guide](https://github.com/epfml/disco/blob/develop/DEV.md)).
 
 __
 

cli/README.md

@@ -23,7 +23,7 @@ npm start -- -t cifar10 -u 4 -e 15 -r 5
 - install node 16 and ensure it is activated on opening any new terminal (e.g. `nvm use 16`)
 - `git clone git@github.com:epfml/disco.git`
 - download the `example_training_data.tar.gz` file and extract it into the root of the repository
-  - simply execute [get_training_data.sh](../get_training_data)
+  - simply execute [get_training_data.sh](../get_training_data.sh)
 - `npm ci` within `discojs`, `server` and `cli`
 - `cd discojs/discojs-node && npm run build`
 

discojs/discojs-core/src/training/trainer/local_trainer.ts

@@ -1,7 +1,8 @@
 import { tf } from '../..'
 import { Trainer } from './trainer'
 
-/** Class whose role is to locally (alone) train a model on a given dataset, without any collaborators.
+/** Class whose role is to locally (alone) train a model on a given dataset,
+ * without any collaborators.
  */
 export class LocalTrainer extends Trainer {
   async onRoundBegin (accuracy: number): Promise<void> {}

discojs/discojs-core/src/weights/weights_container.ts

@@ -5,7 +5,7 @@ import { tf, Weights } from '..'
 export type TensorLike = tf.Tensor | ArrayLike<number>
 
 /**
- * Convenient wrapper object reprensenting an immutable list of TF.js tensors.
+ * Convenient wrapper object representing an immutable list of TF.js tensors.
  */
 export class WeightsContainer {
   private readonly _weights: List<tf.Tensor>
@@ -113,18 +113,18 @@ export class WeightsContainer {
   }
 
   /**
-   * Instanciates a new weights container from the given tensors or arrays of numbers.
+   * Instantiates a new weights container from the given tensors or arrays of numbers.
    * @param weights The tensors or number arrays
-   * @returns The instanciated weights container
+   * @returns The instantiated weights container
    */
   static of (...weights: TensorLike[]): WeightsContainer {
     return new this(weights)
   }
 
   /**
-   * Instanciates a new weights container from the given model's weights.
+   * Instantiates a new weights container from the given model's weights.
    * @param model The TF.js model
-   * @returns The instanciated weights container
+   * @returns The instantiated weights container
    */
   static from (model: tf.LayersModel): WeightsContainer {
     return new this(model.weights.map((w) => w.read()))

discojs/test.sh

@@ -1,5 +1,21 @@
-DIR="$( cd "$( dirname "$0" )" ; pwd -P )"
+#!/bin/sh
+set -e # Exit script on error
+DIR="$( cd "$( dirname "$0" )" ; pwd -P )" # Fetch current directory
 
-cd $DIR/discojs-node && npm test &&
-cd $DIR/discojs-core && npm test &&
-cd $DIR/discojs-web && npm test
+printf '\n>>> Make sure a server instance is running and reachable at http://localhost:8080! <<<'
+
+printf '\n>>> Rebuilding discojs\n'
+cd $DIR
+npm run build
+
+printf '\n>>> Testing disco-core\n'
+cd $DIR/discojs-core
+npm run test
+
+printf '\n>>> Testing disco-node\n'
+cd $DIR/discojs-node
+npm run test
+
+printf '\n>>> Testing disco-web\n'
+cd $DIR/discojs-web
+npm run test