Updates for wis2box-training for Jakarta (#84)

* updated MQTT exercise * mqtt exercise update * update mqtt section * update student-vm, new script for creating users * updated wis2box-introduction * maaike-wip * commit current maaike-wip for review * Update connecting-to-mqtt.md * Update README.md * Update accessing-your-student-vm.md * Update configuring-station-metadata.md * Update connecting-to-mqtt.md * Update wis2box-introduction.md --------- Co-authored-by: Tom Kralidis <tomkralidis@gmail.com>
wmo-im · Oct 1, 2023 · 646f05e · 646f05e
1 parent 8f76c70
commit 646f05e
Show file tree

Hide file tree

Showing 29 changed files with 607 additions and 303 deletions.
diff --git a/documentation/docs/assets/img/minio-ui.png b/documentation/docs/assets/img/minio-ui.png
diff --git a/documentation/docs/assets/img/mqtt-explorer-global-broker-connection-china.png b/documentation/docs/assets/img/mqtt-explorer-global-broker-connection-china.png
diff --git a/documentation/docs/assets/img/mqtt-explorer-global-broker-topics-exercise2.png b/documentation/docs/assets/img/mqtt-explorer-global-broker-topics-exercise2.png
diff --git a/documentation/docs/assets/img/mqtt-explorer-new-connection.png b/documentation/docs/assets/img/mqtt-explorer-new-connection.png
diff --git a/documentation/docs/assets/img/mqtt-explorer-topics.png b/documentation/docs/assets/img/mqtt-explorer-topics.png
diff --git a/documentation/docs/assets/img/mqtt-explorer-wis2box-broker.png b/documentation/docs/assets/img/mqtt-explorer-wis2box-broker.png
diff --git a/documentation/docs/assets/img/wis2box-api-collections-initial.png b/documentation/docs/assets/img/wis2box-api-collections-initial.png
diff --git a/documentation/docs/assets/img/wis2box-api-stations.png b/documentation/docs/assets/img/wis2box-api-stations.png
diff --git a/documentation/docs/assets/img/wis2box-grafana-ui.png b/documentation/docs/assets/img/wis2box-grafana-ui.png
diff --git a/documentation/docs/assets/img/wis2box-mini-buckets.png b/documentation/docs/assets/img/wis2box-mini-buckets.png
diff --git a/documentation/docs/assets/img/wis2box-minio-buckets.png b/documentation/docs/assets/img/wis2box-minio-buckets.png
diff --git a/documentation/docs/assets/img/wis2box-ui-empty.png b/documentation/docs/assets/img/wis2box-ui-empty.png
diff --git a/documentation/docs/assets/img/wis2box-ui-explore-stations.png b/documentation/docs/assets/img/wis2box-ui-explore-stations.png
diff --git a/documentation/docs/assets/img/wis2box-webapp-create-station-save.png b/documentation/docs/assets/img/wis2box-webapp-create-station-save.png
diff --git a/documentation/docs/assets/img/wis2box-webapp-create-station-select-topic.png b/documentation/docs/assets/img/wis2box-webapp-create-station-select-topic.png
diff --git a/documentation/docs/assets/img/wis2box-webapp-create-station-success.png b/documentation/docs/assets/img/wis2box-webapp-create-station-success.png
diff --git a/documentation/docs/assets/img/wis2box-webapp-import-station-from-oscar.png b/documentation/docs/assets/img/wis2box-webapp-import-station-from-oscar.png
diff --git a/documentation/docs/assets/img/wis2box-webapp-select-stations.png b/documentation/docs/assets/img/wis2box-webapp-select-stations.png
diff --git a/documentation/docs/assets/img/wis2box-webapp-stations-with-one-station.png b/documentation/docs/assets/img/wis2box-webapp-stations-with-one-station.png
diff --git a/documentation/docs/assets/img/wis2box-webapp.png b/documentation/docs/assets/img/wis2box-webapp.png
diff --git a/documentation/docs/practical-sessions/accessing-your-student-vm.md b/documentation/docs/practical-sessions/accessing-your-student-vm.md
@@ -4,10 +4,22 @@ title: Accessing your student VM
 
 # Accessing your student VM
 
+!!! abstract
+
+    In this session you will practice accessing your student VM using SSH and WinSCP.
+
 ## Introduction
 
 As part of locally run wis2box training sessions, you can access your personal student VM on the local training network named "WIS2-training".
 
+Your student VM has the following software pre-installed:
+
+- Ubuntu 22.0.4.3 LTS [ubuntu-22.04.3-live-server-amd64.iso](https://releases.ubuntu.com/jammy/ubuntu-22.04.3-live-server-amd64.iso)
+- Python 3.10.12
+- Docker 24.0.6
+- Docker Compose 2.21.0
+- Text editors: vim, nano
+
 !!! note
 
     If you want to run this training outside of a local training session, you can provide your own instance using any Cloud Provider:
@@ -16,18 +28,25 @@ As part of locally run wis2box training sessions, you can access your personal s
     - AWS (Amazon Web Services)  ec2-instance `t3a.medium` 
     - Azure (Microsoft) Azure Virtual Machine `standard_b2s`
 
-    Select Ubuntu Server 20.0.4 LTS as OS and run the setup script available in [student-vm-setup.zip](https://training.wis2box.wis.wmo.int/student-vm-setup.zip) on your instance to ensure you have all required software.
+    Select Ubuntu Server 22.0.4 LTS as OS and run setup instructions as described at https://docs.wis2box.wis.wmo.int/en/latest/user/getting-started.html#software-dependencies
+
+    The release archive for wis2box as used in this training can be downloaded as follows:
 
-    If you are using the student VM provided during local WIS2 training sessions, the required software will already be installed.
+    ```bash
+    wget https://github.com/wmo-im/wis2box/releases/download/1.0b5/wis2box-setup-1.0b5.zip
+    unzip wis2box-setup-1.0b5.zip
+    ```
+
+    You can always find the latest 'wis2box-setup' archive at [https://github.com/wmo-im/wis2box/releases](https://github.com/wmo-im/wis2box/releases).
 
-!!! note
+    The exercise material as used in this training can be downloaded as follows:
 
-    The student-VMs provided during WIS2 local training sessions have the following command-line editors pre-installed:
+    ```bash
+    wget https://training.wis2box.wis.wmo.int/exercise-materials.zip
+    unzip exercise-materials.zip
+    ```
 
-    - vi
-    - vim
-    - nano
-    - emacs
+    If you are using the student VM provided during local WIS2 training sessions, the required software will already be installed.
 
 ## Connect to your student VM on the local training network
 
@@ -59,64 +78,49 @@ passwd: password updated successfully
 
 ## Verify software versions
 
-To be able to practice conversion to BUFR, the student VM comes with ecCodes, synop2bufr and csv2bufr pre-installed:
+To be able to practice data conversion to BUFR, the student VM comes with ecCodes, synop2bufr and csv2bufr pre-installed.
 
-Check the ecCodes version via the `bufr_dump` command:
+To be able to run wis2box, the student VM is provided with with Python, Docker and Docker Compose pre-installed. 
 
+Check Python version:
 ```bash
-bufr_dump -V
-```
-returns:
-```console
-
-ecCodes Version 2.28.0
-```
-
-Check synop2bufr version:
-```bash
-synop2bufr --version
-```
-returns:
-```console
-synop2bufr, version 0.4.1
-```
-
-Check csv2bufr version:
-```bash
-csv2bufr --version
+python3 --version
 ```
 returns:
 ```console
-csv2bufr, version 0.6.3
+Python 3.10.12
 ```
 
-To be able to run wis2box, the student VM also comes with Python Docker and Docker Compose pre-installed. 
-
 Check docker version:
 ```bash
 docker --version
 ```
 returns:
 ```console
-Docker version 20.10.17, build 100c701
+Docker version 24.0.6, build ed223bc
 ```
 
 Check Docker Compose version:
 ```bash
-docker-compose --version
+docker compose version
 ```
 returns:
 ```console
-docker-compose version 1.29.0, build unknown
+Docker Compose version v2.21.0
 ```
 
-Check Python version:
+To ensure your user can run Docker commands your user has been added to the `docker` group. 
+
+To test that your user can run docker hello-world, run the following command:
 ```bash
-python3 --version
+docker run hello-world
 ```
+
 returns:
 ```console
-Python 3.8.10
+Hello from Docker!
+This message shows that your installation appears to be working correctly.
+...
 ```
 
 ## Inspect the exercise materials
@@ -128,34 +132,19 @@ ls ~/
 ```
 returns:
 ```console
-exercise-materials  wis2box-1.0b4
+exercise-materials  wis2box-1.0b5
 ```
 
-To access the material on your local machine rather than from the command line, you can use SCP. Using WinSCP, you can create a new SCP connection to your VM as follows:
+You can use WinSCP to connect to your instance and inspect the contents of your home directory and download or upload files between your VM and your local PC.
+
+Using WinSCP, you can create a new SCP connection to your VM as follows:
 
 <img alt="winscp-student-vm-scp.png" src="../../assets/img/winscp-student-vm-scp.png" width="400">
 
 And you should be able to see the following content:
 
 <img alt="winscp-student-vm-exercise-materials.png" src="../../assets/img/winscp-student-vm-exercise-materials.png" width="600">
 
-## Exercise 1: Editing files on your Student VM
-
-Connect to your Student VM using WinSCP and browse into the directory: `~/exercise-materials/accessing-your-student-vm/`
-
-Right-click on the file `hello_world.txt` and select **Edit -> Internal editor**.  Edit this file by adding a message of your own and save your changes.
-
-<img alt="winscp_internal_editor.png" src="../../assets/img/winscp_internal_editor.png" width="600">
-
-From within your SSH client check the content of the file `~/exercise-materials/accessing-your-student-vm/hello_world.txt`:
-
-```bash
-cat ~/exercise-materials/accessing-your-student-vm/hello_world.txt
-```
-And confirm you see the changes you made in the file.
-
-During the exercises you will be asked to edit files. It is up to you if you prefer to edit files from the command line in your SSH client (using `vi`/`vim`/`nano`/`emacs`) or using WinSCP.
-
 ## Conclusion
 
 !!! success "Congratulations!"

diff --git a/documentation/docs/practical-sessions/configuring-station-metadata.md b/documentation/docs/practical-sessions/configuring-station-metadata.md
@@ -4,96 +4,155 @@ title: Configuring station metadata
 
 # Configuring station metadata
 
+!!! abstract
+
+    In this session you will practice configuring station metadata for your dataset.
+
 ## Introduction
 
-wis2box uses a fixed station metadata list that is used as part of its runtime operation. 
+wis2box has a collection of station metadata that is used to publish data on WIS2.
 Only data for stations configured in the wis2box station list will be published on your wis2box broker.
 The **WIGOS Station Identifier (WSI)** is used as the unique reference of the station which produced a specific set of observation data.
 
-## Preparation
+## Create an authorization token for collections/stations
 
-Login to your student VM using SSH.
+To edit stations via the **wis2box-webapp** you will first to need create an authorization token.
 
-Ensure wis2box is running:
+Login to your student VM and ensure you are in the `wis2box-1.0b5` directory:
 
 ```bash
-cd ~/wis2box-1.0b4
-python3 wis2box-ctl.py start
-python3 wis2box-ctl.py status
+cd ~/wis2box-1.0b5
 ```
 
-## Update the wis2box station list
-
-Pre-select a few stations in your country that you would consider for data publishing on WIS2. If you want to ingest your own data sample later during the WIS2 training, make sure to add the stations corresponding to your data.
+Then login into the **wis2box-management** container with the following command:
 
-Edit the file `~/wis2box-data/metadata/station/station_list.csv`:
+```bash
+python3 wis2box-ctl.py login
+```
 
-For each new station, add a row to the end of the file with the following values:
+Within the wis2box-container your can create an authorization token for a specific endpoint using the command: `wis2box auth add-token --path <my-endpoint>`.
 
-- `station_name`: the human readable name of the station
-- `wigos_station_identifier`: the WSI issued for the station
-- `traditional_station_identifier`: the traditional station identifier if a WSI does not exist
-- `facility_type`: the station/platform type (use **Land (fixed)** for land stations)
-- `latitude`: the latitude, in decimal degrees
-- `longitude`: the longitude, in decimal degrees
-- `elevation`: station elevation, in metres above sea level
-- `territory_name`: the human readable country name
-- `wmo_region`: the Roman numeral of your country based on WMO Regional Associations
+For example, to use a random automatically generated token for the `collections/stations` endpoint:
 
-!!! tip
-    Ensure that latitude and longitude values are correctly signed (for example, use the minus sign [`-`] for southern or western hemispheres.
+```{.copy}
+wis2box auth add-token --path collections/stations
+```	
 
-### Using data from OSCAR
+The output will look like this:
 
-It is recommended to use station information from the [WMO OSCAR/Surface](https://oscar.wmo.int/surface) system where available.
+```{.copy}
+Continue with token: 7ca20386a131f0de384e6ffa288eb1ae385364b3694e47e3b451598c82e899d1 [y/N]? y
+Token successfully created
+```
 
-The script `~/exercise-materials/create-station-list/oscar2wis2box.py` can be used to add stations to your station list if they are available in OSCAR/Surface.
+Or, if you want to define your own token for the `collections/stations` endpoint, you can use the following example:
 
-For example to add the stations with WIGOS-IDs=0-20000-0-78970, 0-20000-0-78969 and 0-20000-0-78962 to your `station_list.csv`, run the following commands: 
+```{.copy}
+wis2box auth add-token --path collections/stations MyCatIsCalledJessie!
+```
 
-```bash
-python3 ~/exercise-materials/station-list/oscar2wis2box.py 0-20000-0-78970 >> ~/wis2box-data/metadata/station/station_list.csv
-python3 ~/exercise-materials/station-list/oscar2wis2box.py 0-20000-0-78969 >> ~/wis2box-data/metadata/station/station_list.csv
-python3 ~/exercise-materials/station-list/oscar2wis2box.py 0-20000-0-78962 >> ~/wis2box-data/metadata/station/station_list.csv
+Output:
+
+```{.copy}
+Continue with token: MyCatIsCalledJessie! [y/N]? y
+Token successfully created
 ```
 
-### Review your station list
+!!! note "Exercise 1: Create an authorization token for collections/stations"
 
-Check the content of your station list from the command line as follows:
+    Please create an authorization token for the `collections/stations` endpoint using the instructions above.
 
-```bash
-cat ~/wis2box-data/metadata/station/station_list.csv
-```
+## add station metadata using the wis2box-webapp
 
-Or open the file in WinSCP.
+The **wis2box-webapp** provides a graphical user interface to edit station metadata.
 
-Keep adding lines to `station_list.csv` and ensure you have at least three stations defined.
+Open the **wis2box-webapp** in your browser by navigating to `http://<your-host>/wis2box-app`:
 
-## Publishing station metadata
+<img alt="wis2box-webapp" src="../../assets/img/wis2box-webapp.png" width="800">
 
-Login in to the **wis2box-management** container:
+And select stations:
 
-```bash
-cd ~/wis2box-1.0b4/
-python3 wis2box-ctl.py login
-```
+<img alt="wis2box-webapp-select-stations" src="../../assets/img/wis2box-webapp-select-stations.png" width="250">
+
+When you click add 'add new station' you are asked to provide the WIGOS-station-identifier for the station you want to add:
+
+<img alt="wis2box-webapp-import-station-from-oscar" src="../../assets/img/wis2box-webapp-import-station-from-oscar.png" width="800">
+
+When you click search the station data is retrieved from OSCAR/Surface, please note that this can take a few seconds.
+
+Review the data returned by OSCAR/Surface and add missing data where required. Select a topic for the station and provide your authorization token for the `collections/stations` endpoint and click 'save':
 
-Run the following command to publish your station metadata:
+<img alt="wis2box-webapp-create-station-save" src="../../assets/img/wis2box-webapp-create-station-save.png" width="800">
+
+<img alt="wis2box-webapp-create-station-success" src="../../assets/img/wis2box-webapp-create-station-success.png" width="500">
+
+Go back to the station list and you will see the station you added:
+
+<img alt="wis2box-webapp-stations-with-one-station" src="../../assets/img/wis2box-webapp-stations-with-one-station.png" width="800">
+
+!!! note "Exercise 2: Add station metadata"
+
+    Please add three or more stations to the wis2box station metadata collection of your wis2box. 
+
+    Please use stations from your country if possible, especially if you brought your own data.
+
+    Otherwise, you can use the following WIGOS-station-identifiers for testing purposes:
+
+    - 0-20000-0-91334
+    - 0-20000-0-96323 (note missing station elevation in OSCAR)
+    - 0-20000-0-96749 (note missing station elevation in OSCAR)
+
+!!! note "Deriving elevation information"
+
+If your station elevation is missing, there are online services that allow the lookup of the elevation using open elevation data. One such example is the [Open Topo Data API](https://www.opentopodata.org).
+
+For example, to get the elevation of the BMKG auditorium one would query the Open Topo Data API as follows:
 
 ```bash
-wis2box metadata station publish-collection
+ wget -q -O - "https://api.opentopodata.org/v1/aster30m?locations=-6.15558,106.84204"
+{
+  "results": [
+    {
+      "dataset": "aster30m", 
+      "elevation": 7.0, 
+      "location": {
+        "lat": -6.15558, 
+        "lng": 106.84204
+      }
+    }
+  ], 
+  "status": "OK"
+}
 ```
 
-Ensure that your new station metadata was published to the API, by navigating to `http://<your-host>.wis2.training/oapi/collections/stations/items`:
+## Review your station metadata
+
+After saving your station metadata, you can review the content of your station metadata in the **wis2box-webapp**:
+
+You can verify the updated stations are available in the **wis2box-api**:
+
+<img alt="wis2box-api-stations" src="../../assets/img/wis2box-api-stations.png" width="800">
+
+You can also visit the **wis2box-ui** at `http://<your-host>` and select "EXPLORE" on your dataset and you will see the stations you added:
+
+<img alt="wis2box-ui-explore-stations" src="../../assets/img/wis2box-ui-explore-stations.png" width="800">
+
+!!! note "Exercise 3: Review your station metadata"
+
+    Verify the stations you added are associated to your dataset by visiting the **wis2box-api** and **wis2box-ui** endpoints for your host in your browser.
+
+You also have the option to view/update/delete the station in the **wis2box-webapp**. Note that you are required to provide your authorization token for the `collections/stations` endpoint to update/delete the station.
 
-<img alt="stationlist_from_oscar" src="../../assets/img/stationlist_from_oscar.png" width="800">
+!!! note "Exercise 4: Update/delete station metadata"
 
-Click on your station metadata record and inspect the content, noting how it relates to the content of the `station_list.csv` you have updated.
+    Please update/delete the station metadata for one of the stations you added using the **wis2box-webapp**.
 
 ## Conclusion
 
 !!! success "Congratulations!"
     In this practical session, you learned how to:
 
-    - update station metadata
-    - publish station metadata
+    - create an authorization token for the `collections/stations` endpoint
+    - add station metadata to the wis2box
+    - review stations associated to datasets in the **wis2box-ui**
+    - update/delete station metadata using the **wis2box-webapp**