Improve README.md

README.md

@@ -1,6 +1,122 @@
-# Galileo Client
+# Galileo Experiments
+
+This repository contains code that invokes the galileo shell to start experiments.
+It also includes deployment files for necessary components and describes in detail which other services are 
+required to start an experiment.
+Further, this project aims to unify different sub-projects of `edgerun` and make deployment and experiment setup easy.
+The main goals and functionalities are:
+
+* A framework for distributed load testing experiments
+* Fine-grained telemetry data collection
+* HTTP Trace recording for any service
+* A container orchestration adaption for ease of use
+
+## Who is this project for?
+
+For everyone that wants to see resource usage and application performance with easy configurable workload creation.
+All components are tailored and suited to run on low performance devices (i.e., Raspberry Pi) but can run on default server VMs too.
+
+Common questions that can be answered by performing and analyzing Galileo experiments:
+
+* How much CPU usage does my application use?
+* What is the average execution time of my application across the cluster?
+* What are the differences in terms of resource usage between two nodes hosting the same application?
+* What is the impact of having multiple applications running on one node?
+
+All these questions can be easily answered and have a simple flow in common:
+1. Deploy base infrastructure
+2. Deploy my application
+3. Start requests
+4. Analyze in Jupyter Notebooks
+
+In summary: simple profiling tasks.
+But, this framework also targets full end-to-end tests to evaluate important cluster components (i.e, load balancer, scheduler and scaling).
+
+Therefore, experiments can be done to evaluate new implementations for  the aforementioned components.
+
+
+# High level overview
+
+
+
+# Kubernetes cluster setup
+
+![Cluster components](figures/cluster.drawio.png)
+
+## Overview
+
+### Main components
+The cluster setup consists of the following main components:
+
+* Kubernetes
+* Galileo (for clients and experiment shell)
+* Telemd
+* Controller (i.e., the load balancer)
+* MySQL (i.e., MariaDB)
+* InfluxDB v2
+* Etcd (Kubernetes requires an instance to run!)
+
+Kubernetes is used to host the clients (galileo running in a Pod), telemetry agents (telemd), load balancer and the applications to test.
+Redis is used as a pub/sub system through which all data is sent (i.e., telemetry) and recorded by the Galileo Shell (i.e., the program that prepares and executes an experiment).
+The Galileo Shell persists data in MariaDB and InfluxDB.
+The provided load balancer implementation uses etcd to watch for weights for the round-robin algorithm and galileo uses redis to provide the clients with routing rules (`rtbl`).
+
+
+
+## Main interactions
+
+The figure above depicts all components and also highlights important interactions.
+Those interactions are in short:
+* Client nodes send HTTP requests to the Controller (load balancer), which forwards requests to the worker nodes which host the application pods.
+* Client nodes report the results of each request (i.e., trace) via Redis. 
+* The Go-based load balancer implementation fetches weights and ip addresses from the etcd instance.
+* The clients get the routing rules from the Redis instance (set via `rtbl` from Galileo)
+* Worker nodes report resource usage (i.e., telemd) via Redis, which is saved in InfluxDB
+* The Galileo Shell starts the experiments and saves metadata (i.e., the cluster hosts, misc. data) in the MariaDB
+
+
+## Deployment 
+
+This project provides deployment files for the following components:
+* Galileo (for clients and experiment shell)
+* Telemd
+* Controller (i.e., the load balancer)
+
+Which leaves the following components to be additionally deployed:
+
+* Kubernetes
+* MySQL (i.e., MariaDB)
+* InfluxDB v2
+* Etcd (Kubernetes requires an instance to run!)
+
+Deployment files can be found in `deployment/kubernetes`.
+Note, that we use Kubernetes node labels to schedule the workers (i.e., clients).
+On nodes that should act as clients execute the following command:
+
+    kubectl label node <node> node-role.kubernetes.io/client=true
+
+The following label is used to identify nodes with hosting capabilities (i.e., workers):
+
+    kubectl label node <node> node-role.kubernetes.io/worker=true
+
+If you have multiple zones (i.e., clusters) in which you want to have seperate clients, adapt the `zone`  arguments (default is `main`).
+You can easily group your nodes by labelling it with the following command:
+
+    kubectl label node <node> ether.edgerun.io/zone=main
+
+## Galileo Workers
+
+
+
+# Data storage
+
+Galileo requires the following data components that are either deployed in the cluster or externally:
+
+* Redis (pub/sub for telemetry and traces)
+* MySQL (i.e., MariaDB) (persistent storage for experiment metadata)
+* InfluxDB v2 (stores runtime data - telemetry and traces)
+
 
-This repo contains code that invokes the galileo shell to start experiments.
 
 Environment variables
 =====================

figures/cluster.drawio

@@ -0,0 +1 @@
+<mxfile host="app.diagrams.net" modified="2022-03-23T15:56:24.943Z" agent="5.0 (X11)" etag="MOkwXHxhnsPw1ln5YWjS" version="16.6.6" type="device"><diagram id="hr-8L9cHcijfdLjV0N6L" name="Page-1">7V1bd5s4EP41Pqf7EB8Q98fGSS+77ban6dluH2WQbRqCvCA3dn/9SiAwCNkmDjenuA+FkRCg75sZaTQiE232sH0bwfXqI/ZQMAGKt51oNxMAgKMC+h+T7FKJqupOKllGvsdle8Gd/wtxocKlG99DcakiwTgg/rosdHEYIpeUZDCK8GO52gIH5buu4RJVBHcuDKrSb75HVqnUNpS9/B3yl6vszqrCSx5gVpkL4hX08GNBpN1OtFmEMUmPHrYzFLDey/olve7NgdL8wSIUkjoXbH1/45Mfd/if24X1489daD1ur0wjbeYnDDb8jfnTkl3WBRHehB5irSgT7fpx5RN0t4YuK32kqFPZijwE9EylhwscEo6iqrNzPwhmOMBR0pa2MNg/Xq8gT39UHpMI36NCiZn8aAl/UBQRtD3YBWresZSSCD8gEu1oFX6BmcHG2ahl5497aDWLy1YFWIHDhZDTaZm3ve9xesA7/QkAZC9R6G/kUQbyUxyRFV7iEAa3e+l1GZF9nQ8YrzkOPxAhOw4E3BBcRikmMCKvmXpQgRvAOPbdTPzGD/bVGBgZ70EOAnvC4xDQF8KbyEVH3lzn2gyjJSJH6llySCMUQOL/LD9H4/Bodh/w0L6Mdv8WT76zxqZGdnqz5Y2nZzt+5sF4ldz30jBWewVZl9hAMyCsQ/2f9HDJDmeBz16fF9D7FMok1V8tYeAHCP+Rlc0jsbbYhsCzE4YWxuvU3y38LcNctLQeRPbClVpU10bzRWar+Q3VBsyrqpfNq6VWzastsa52a8ZV71V7Cwr7vVh2QHuHoLDGJRhl2ZilcX19cdoJtKFppzFq59O007wE7TRH7TxDOzUwMO1UzX6182kj37O1szRkblBVrbojX6NPXbVq6SqleoSDAEX19fUDhh5t+BoGMHRR1N8o2EC2p8s02QZzLYkrNO5nVaOkyQaoarKqSlQ5FzbvaXvW5bKnVdrQZa1Z9bU78rTJpfRd4a5QYY39kMSFlj8zwZ5hwjRLt4TAn1AdWEfr04P0Afb8yt/kGYGTXqZeg3MDdXn07ADIWUS6EqYEmnOcSWJ93baP1j9evR3i2XVcWu6OMsE3HN2X/Nv8lJ863ebf2ENH/d7JezTrCxeLBXClo1rPnJvGc2PsR2Lq3U4rrYuxPA07rWz8MEyvpYrWYIhuK+vC0Xx0bT74BY7gkfq2Jqo6EqJXQqj20BgBRkb0ygjRMfTPCG1kRK+M0MyhMaLWkvLIiNYYodfI9OmWEYcXLXMAEHE9OTACBE/LyCr3P1I9A1my/ndMS4NN9b8QozJ0o9L/KpAAYLYGwOFYd97RX5Dnxy8EATFWMwAEDodm8o5+Hy6CzfbmgH26PBDyPNbhoOCcRuEjjHz4ckEwtaoz6BYErZfUh8GlB2Z9PvD8QK2X5bPDaIU4RAJUynCgcnqFqpfgbwNQodCrKiMVFlSxFzTtZ6J53rpUJVfNOLEwJV5gWIbApRaCxVkvHp/lvU1Tl2rP4O5Wya3PzJd4+maNPPVBMlVD5oGpmuXMFaWdgaoly5fo1juDPozImC5R05qczIfQBGKkj88vOmJFxMBinrmTNZS+X6WhF5m/IZsmiEoh81kFRpdVQnCS545Vquwp2AlDYiYy2TNJZorkEENFdVlmmScaOsCyppDVq17rBhJIJV8jGMYLFoBUXvlTNJ2AGVNqFCDeN8lpRL1J/EeFDNQBECESGPjLkHGDAkgb1a6Zm/BdGLzmBQ++56W2E8X+LzhPmmJU4QpE2zWuJ8YNa4uay5hvK6w4JD7IKnqvTCQ6OGF3YgPuyzKFdD+r6r5yV1XaU9iW+9JB06p7+dpqir19rrZWGmpbW6trS+++fv1MJV/QfxsUMz+jvGILBwGG3qiWOW5OWS1NST49UDpVy/PWhBJNvIrw3A+bXRh6TF46PjrTeOra0HDZ1iy7NE0w+pKIoiMhl94auWosL6Weu1ZQ93fBUa0RGe7Wdx/ePbXHcT8gG6HchxFELHUJlrIl3/awrLHeFRMcIVrHS0bg9SzxbwuwkBuYD816A/jwUtoxxxtv5rEb+XPUrD8n+Ik8Gj16kV1CwohZM2GkNY+e3f+kJ/BqWYl4Bdfs0N3MWQ+eCAzP0+nfh3kugO79MhmKftqQwA8zxDwY3X+izfgkCUdOFaMsBIm0xoeB1DlUEZhUPgy0SH4VbtASRTFvX79hJbzVhuLRhiL4EUk8WjYmEKOOzTHhcAryyIQ2maCLI4remXB4CjkyoU0miLvz+2dCnfneyITmmVDZ09g7E+rOGEcmNMsE8WtX/TOhRobryIRWRowDI0KNwMM7HLNVjvdsJYH2D1D+ojBHISIsQqh8xh4vHEMSR3ciCYsMehV5IPvUR2shCaNGdu+I9POR1u0q0rlD6ATpzKJUl/FZTJF91VjE6nnf8T2eKtZNJrct7g6WRAClqWKtmVqzOjdPN3HFnXZ/N5uqxO7Xld67H1S6v/i1qg4RqPGhqTYQkDicjhGo5iek3/brVgFqfLGvje6XeYFuu78aEcrzjRWeSdypHeoiY1iEQfqFtW5hqDEJ/4CWKKw39fpdRlWV7/FL/Ln0c/yt4SibQovz4ntE3BXvXdkmkFyt4rJaCboCbMu4BTJdyWewZeyu6WSZzpbBTJkyqztTGJxglhSoiRgIUkcqTZoQazoHGraSq2m5pBFVkAFbqJvQLYsjPGyX7G+ETO/z6cfUd3HIXjN6nxzcrLHXDtGuhEi+lqX4FydqzjTbI1zimt0W12Sz9JFrF881W5na2uDYZlWniyPbXgDb6Dhnml2V5cVIMtA1Z5pFBUsDotboJluzHul28XQDwkLoILhWDQKMXHsBXNPUAXJN9qmvkWsXzzWg0KdTnP3PLlFPtnukOerR0/2fAUw3Du3/mqJ2+z8=</diagram></mxfile>