Update some docs

README.md

@@ -1,14 +1,14 @@
-# GitHub Actions Runner Manager (garm)
+# GitHub Actions Runner Manager (GARM)
 
 [![Go Tests](https://github.com/cloudbase/garm/actions/workflows/go-tests.yml/badge.svg)](https://github.com/cloudbase/garm/actions/workflows/go-tests.yml)
 
-Welcome to garm!
+Welcome to GARM!
 
 Garm enables you to create and automatically maintain pools of [self-hosted GitHub runners](https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners), with autoscaling that can be used inside your github workflow runs.
 
-The goal of ```garm``` is to be simple to set up, simple to configure and simple to use. It is a single binary that can run on any GNU/Linux machine without any other requirements other than the providers it creates the runners in. It is intended to be easy to deploy in any environment and can create runners in any system you can write a provider for. There is no complicated setup process and no extremely complex concepts to understand. Once set up, it's meant to stay out of your way.
+The goal of ```GARM``` is to be simple to set up, simple to configure and simple to use. It is a single binary that can run on any GNU/Linux machine without any other requirements other than the providers it creates the runners in. It is intended to be easy to deploy in any environment and can create runners in any system you can write a provider for. There is no complicated setup process and no extremely complex concepts to understand. Once set up, it's meant to stay out of your way.
 
-Garm supports creating pools on either GitHub itself or on your own deployment of [GitHub Enterprise Server](https://docs.github.com/en/enterprise-server@3.5/admin/overview/about-github-enterprise-server). For instructions on how to use ```garm``` with GHE, see the [credentials](/doc/github_credentials.md) section of the documentation.
+Garm supports creating pools on either GitHub itself or on your own deployment of [GitHub Enterprise Server](https://docs.github.com/en/enterprise-server@3.5/admin/overview/about-github-enterprise-server). For instructions on how to use ```GARM``` with GHE, see the [credentials](/doc/github_credentials.md) section of the documentation.
 
 ## Join us on slack
 
@@ -18,121 +18,36 @@ Whether you're running into issues or just want to drop by and say "hi", feel fr
 
 ## Installing
 
-## Build from source
+Check out the [quickstart](/doc/quickstart.md) document for instructions on how to install ```GARM```. If you'd like to build from source, check out the [building from source](/doc/building_from_source.md) document.
 
-You need to have Go installed, then run:
+## Installing external providers
 
-  ```bash
-  git clone https://github.com/cloudbase/garm
-  cd garm
-  go install ./...
-  ```
+External providers are binaries that GARM calls into to create runners in a particular IaaS. There are currently two external providers available:
 
-You should now have both ```garm``` and ```garm-cli``` in your ```$GOPATH/bin``` folder.
+* [OpenStack](https://github.com/cloudbase/garm-provider-openstack)
+* [Azure](https://github.com/cloudbase/garm-provider-azure)
 
-If you have docker/podman installed, you can also build statically linked binaries by running:
-
-  ```bash
-  make build-static
-  ```
-
-The ```garm``` and ```garm-cli``` binaries will be built and copied to the ```bin/``` folder in your current working directory.
-
-## Install the service
-
-Add a new system user:
-
-  ```bash
-  useradd --shell /usr/bin/false \
-      --system \
-      --groups lxd \
-      --no-create-home garm
-  ```
-
-The ```lxd``` group is only needed if you have a local LXD install and want to connect to the unix socket to use it. If you're connecting to a remote LXD server over TCP, you can skip adding the ```garm``` user to the ```lxd``` group.
-
-Copy the binary to somewhere in the system ```$PATH```:
-
-  ```bash
-  sudo cp $(go env GOPATH)/bin/garm /usr/local/bin/garm
-  ```
-
-Or if you built garm using ```make```:
-
-  ```bash
-  sudo cp ./bin/garm /usr/local/bin/garm
-  ```
-
-Create the config folder:
-
-  ```bash
-  sudo mkdir -p /etc/garm
-  ```
-
-Copy the config template:
-
-  ```bash
-  sudo cp ./testdata/config.toml /etc/garm/
-  ```
-
-Copy the external provider (optional):
-
-  ```bash
-  sudo cp -a ./contrib/providers.d /etc/garm/
-  ```
-
-Copy the systemd service file:
-
-  ```bash
-  sudo cp ./contrib/garm.service /etc/systemd/system/
-  ```
-
-Change permissions on config folder:
-
-  ```bash
-  sudo chown -R garm:garm /etc/garm
-  sudo chmod 750 -R /etc/garm
-  ```
-
-Enable the service:
-
-  ```bash
-  sudo systemctl enable garm
-  ```
-
-Customize the config in ```/etc/garm/config.toml```, and start the service:
-
-  ```bash
-  sudo systemctl start garm
-  ```
+Follow the instructions in the README of each provider to install them. 
 
 ## Configuration
 
-The ```garm``` configuration is a simple ```toml```. A sample of the config file can be found in [the testdata folder](/testdata/config.toml).
-
-There are 3 major sections of the config that require your attention:
-
-* [Github credentials section](/doc/github_credentials.md)
-* [Providers section](/doc/providers.md)
-* [The database section](/doc/database.md)
-
-Once you've configured your database, providers and github credentials, you'll need to configure your [webhooks and the callback_url](/doc/webhooks_and_callbacks.md).
-
-At this point, you should be done. Have a look at the [running garm document](/doc/running_garm.md) for usage instructions and available features.
-
-If you would like to use ```garm``` with a different IaaS than the ones already available, have a look at the [writing an external provider](/doc/external_provider.md) page.
-
-If you like to optimize the startup time of new instance, take a look at the [performance considerations](/doc/performance_considerations.md) page.
+The ```GARM``` configuration is a simple ```toml```. The sample config file in [the testdata folder](/testdata/config.toml) is fairly well commented and should be enough to get you started. The configuration file is split into several sections, each of which is documented in its own page. The sections are:
 
-## Security considerations
+* [The default section](/doc/config_default.md)
+* [Database](/doc/database.md)
+* [Github credentials](/doc/github_credentials.md)
+* [Providers](/doc/providers.md)
+* [Metrics](/doc/config_metrics.md)
+* [JWT authentication](/doc/config_jwt_auth.md)
+* [API server](/doc/config_api_server.md)
 
-Garm does not apply any ACLs of any kind to the instances it creates. That task remains in the responsibility of the user. [Here is a guide for creating ACLs in LXD](https://linuxcontainers.org/lxd/docs/master/howto/network_acls/). You can of course use ```iptables``` or ```nftables``` to create any rules you wish. I recommend you create a separate isolated lxd bridge for runners, and secure it using ACLs/iptables/nftables.
+## Optimizing your runners
 
-You must make sure that the code that runs as part of the workflows is trusted, and if that cannot be done, you must make sure that any malicious code that will be pulled in by the actions and run as part of a workload, is as contained as possible. There is a nice article about [securing your workflow runs here](https://blog.gitguardian.com/github-actions-security-cheat-sheet/).
+If you would like to optimize the startup time of new instance, take a look at the [performance considerations](/doc/performance_considerations.md) page.
 
 ## Write your own provider
 
-The providers are interfaces between ```garm``` and a particular IaaS in which we spin up GitHub Runners. These providers can be either **native** or **external**. The **native** providers are written in ```Go```, and must implement [the interface defined here](https://github.com/cloudbase/garm/blob/main/runner/common/provider.go#L22-L39). **External** providers can be written in any language, as they are in the form of an external executable that ```garm``` calls into.
+The providers are interfaces between ```GARM``` and a particular IaaS in which we spin up GitHub Runners. These providers can be either **native** or **external**. The **native** providers are written in ```Go```, and must implement [the interface defined here](https://github.com/cloudbase/garm/blob/main/runner/common/provider.go#L22-L39). **External** providers can be written in any language, as they are in the form of an external executable that ```GARM``` calls into.
 
 There is currently one **native** provider for [LXD](https://linuxcontainers.org/lxd/) and two **external** providers for [Openstack and Azure](/contrib/providers.d/).
 

cloudconfig/templates.go

@@ -40,6 +40,10 @@ GITHUB_TOKEN=$(curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X G
 
 function call() {
 	PAYLOAD="$1"
+	[[ $CALLBACK_URL =~ ^(.*)/status$ ]]
+	if [ -z "$BASH_REMATCH" ];then
+		CALLBACK_URL="${CALLBACK_URL}/status"
+	fi
 	curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X POST -d "${PAYLOAD}" -H 'Accept: application/json' -H "Authorization: Bearer ${BEARER_TOKEN}" "${CALLBACK_URL}" || echo "failed to call home: exit code ($?)"
 }
 
@@ -350,6 +354,10 @@ $GHRunnerGroup = "{{.GitHubRunnerGroup}}"
 
 function Install-Runner() {
 	$CallbackURL="{{.CallbackURL}}"
+	if (!$CallbackURL.EndsWith("/status")) {
+		$CallbackURL = "$CallbackURL/status"
+	}
+
 	if ($Token.Length -eq 0) {
 		Throw "missing callback authentication token"
 	}

config/config.go

@@ -47,9 +47,6 @@ func NewConfig(cfgFile string) (*Config, error) {
 	if _, err := toml.DecodeFile(cfgFile, &config); err != nil {
 		return nil, errors.Wrap(err, "decoding toml")
 	}
-	if config.Default.ConfigDir == "" {
-		config.Default.ConfigDir = appdefaults.DefaultConfigDir
-	}
 	if err := config.Validate(); err != nil {
 		return nil, errors.Wrap(err, "validating config")
 	}
@@ -108,10 +105,6 @@ func (c *Config) Validate() error {
 }
 
 type Default struct {
-	// ConfigDir is the folder where the runner may save any aditional files
-	// or configurations it may need. Things like auto-generated SSH keys that
-	// may be used to access the runner instances.
-	ConfigDir string `toml:"config_dir,omitempty" json:"config-dir,omitempty"`
 	// CallbackURL is the URL where the instances can send back status reports.
 	CallbackURL string `toml:"callback_url" json:"callback-url"`
 	// MetadataURL is the URL where instances can fetch information they may need
@@ -139,14 +132,6 @@ func (d *Default) Validate() error {
 		return errors.Wrap(err, "validating metadata_url")
 	}
 
-	if d.ConfigDir == "" {
-		return fmt.Errorf("config_dir cannot be empty")
-	}
-
-	if _, err := os.Stat(d.ConfigDir); err != nil {
-		return errors.Wrap(err, "accessing config dir")
-	}
-
 	return nil
 }
 

config/config_test.go

@@ -32,7 +32,6 @@ var (
 
 func getDefaultSectionConfig(configDir string) Default {
 	return Default{
-		ConfigDir:   configDir,
 		CallbackURL: "https://garm.example.com/",
 		MetadataURL: "https://garm.example.com/api/v1/metadata",
 		LogFile:     filepath.Join(configDir, "garm.log"),
@@ -152,7 +151,6 @@ func TestDefaultSectionConfig(t *testing.T) {
 			cfg: Default{
 				CallbackURL: "",
 				MetadataURL: cfg.MetadataURL,
-				ConfigDir:   cfg.ConfigDir,
 			},
 			errString: "missing callback_url",
 		},
@@ -161,28 +159,9 @@ func TestDefaultSectionConfig(t *testing.T) {
 			cfg: Default{
 				CallbackURL: cfg.CallbackURL,
 				MetadataURL: "",
-				ConfigDir:   cfg.ConfigDir,
 			},
 			errString: "missing metadata-url",
 		},
-		{
-			name: "ConfigDir cannot be empty",
-			cfg: Default{
-				CallbackURL: cfg.CallbackURL,
-				MetadataURL: cfg.MetadataURL,
-				ConfigDir:   "",
-			},
-			errString: "config_dir cannot be empty",
-		},
-		{
-			name: "config_dir must exist and be accessible",
-			cfg: Default{
-				CallbackURL: cfg.CallbackURL,
-				MetadataURL: cfg.MetadataURL,
-				ConfigDir:   "/i/do/not/exist",
-			},
-			errString: "accessing config dir: stat /i/do/not/exist:.*",
-		},
 	}
 
 	for _, tc := range tests {
@@ -560,7 +539,6 @@ func TestNewConfig(t *testing.T) {
 	require.Nil(t, err)
 	require.NotNil(t, cfg)
 	require.Equal(t, "https://garm.example.com/", cfg.Default.CallbackURL)
-	require.Equal(t, "./testdata", cfg.Default.ConfigDir)
 	require.Equal(t, "0.0.0.0", cfg.APIServer.Bind)
 	require.Equal(t, 9998, cfg.APIServer.Port)
 	require.Equal(t, false, cfg.APIServer.UseTLS)
@@ -574,31 +552,6 @@ func TestNewConfig(t *testing.T) {
 	require.Equal(t, timeToLive("48h"), cfg.JWTAuth.TimeToLive)
 }
 
-func TestNewConfigEmptyConfigDir(t *testing.T) {
-	dirPath, err := os.MkdirTemp("", "garm-config-test")
-	if err != nil {
-		t.Fatalf("failed to create temporary directory: %s", err)
-	}
-	defer os.RemoveAll(dirPath)
-	appdefaults.DefaultConfigDir = dirPath
-
-	cfg, err := NewConfig("testdata/test-empty-config-dir.toml")
-	require.Nil(t, err)
-	require.NotNil(t, cfg)
-	require.Equal(t, cfg.Default.ConfigDir, dirPath)
-	require.Equal(t, "https://garm.example.com/", cfg.Default.CallbackURL)
-	require.Equal(t, "0.0.0.0", cfg.APIServer.Bind)
-	require.Equal(t, 9998, cfg.APIServer.Port)
-	require.Equal(t, false, cfg.APIServer.UseTLS)
-	require.Equal(t, DBBackendType("mysql"), cfg.Database.DbBackend)
-	require.Equal(t, "test", cfg.Database.MySQL.Username)
-	require.Equal(t, "test", cfg.Database.MySQL.Password)
-	require.Equal(t, "127.0.0.1", cfg.Database.MySQL.Hostname)
-	require.Equal(t, "garm", cfg.Database.MySQL.DatabaseName)
-	require.Equal(t, "bocyasicgatEtenOubwonIbsudNutDom", cfg.JWTAuth.Secret)
-	require.Equal(t, timeToLive("48h"), cfg.JWTAuth.TimeToLive)
-}
-
 func TestNewConfigInvalidTomlPath(t *testing.T) {
 	cfg, err := NewConfig("this is not a file path")
 	require.Nil(t, cfg)

doc/building_from_source.md

@@ -0,0 +1,25 @@
+# Building GARM from source
+
+The procedure is simple. You will need to gave [go](https://golang.org/) installed as well as `make`.
+
+First, clone the repository:
+
+```bash
+git clone https://github.com/cloudbase/garm
+```
+
+Then build garm:
+
+```bash
+make
+```
+
+You should now have both `garm` and `garm-cli` available in the `./bin` folder.
+
+If you have docker/podman installed, you can also build a static binary against `musl`:
+
+```bash
+make build-static
+```
+
+This command will also build for both AMD64 and ARM64. Resulting binaries will be in the `./bin` folder.

doc/config_api_server.md

@@ -0,0 +1,34 @@
+# The API server config section
+
+This section allows you to configure the GARM API server. The API server is responsible for serving all the API endpoints used by the `garm-cli`, the runners that phone home their status and by GitHub when it sends us webhooks.
+
+The config options are fairly straight forward.
+
+```toml
+[apiserver]
+  # Bind the API to this IP
+  bind = "0.0.0.0"
+  # Bind the API to this port
+  port = 9997
+  # Whether or not to set up TLS for the API endpoint. If this is set to true,
+  # you must have a valid apiserver.tls section.
+  use_tls = false
+  # Set a list of allowed origins
+  # By default, if this option is ommited or empty, we will check
+  # only that the origin is the same as the originating server.
+  # A literal of "*" will allow any origin
+  cors_origins = ["*"]
+  [apiserver.tls]
+    # Path on disk to a x509 certificate bundle.
+    # NOTE: if your certificate is signed by an intermediary CA, this file
+    # must contain the entire certificate bundle needed for clients to validate
+    # the certificate. This usually means concatenating the certificate and the
+    # CA bundle you received.
+    certificate = ""
+    # The path on disk to the corresponding private key for the certificate.
+    key = ""
+```
+
+The GARM API server has the option to enable TLS, but I suggest you use a reverse proxy and enable TLS termination in that reverse proxy. There is an `nginx` sample in this repository with TLS termination enabled.
+
+You can of course enable TLS in both garm and the reverse proxy. The choice is yours.