Skip to content

Commit

Permalink
Merge pull request #21 from vshn/postgres
Browse files Browse the repository at this point in the history
Add Postgres to metrics collector
  • Loading branch information
zugao authored Dec 1, 2022
2 parents 303d2d7 + 840eccb commit b299ea8
Show file tree
Hide file tree
Showing 27 changed files with 1,374 additions and 386 deletions.
29 changes: 0 additions & 29 deletions .codeclimate.yml

This file was deleted.

8 changes: 0 additions & 8 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,14 +34,6 @@ jobs:
- name: Run tests
run: make test

- name: Upload code coverage report to Code Climate
uses: paambaati/codeclimate-action@v3.0.0
env:
CC_TEST_REPORTER_ID: 562205f0a6edaf974f08b29e0eecd479e6b3efd6fa40eee105e03c56e5efa4c5
with:
coverageLocations: cover.out:gocov
prefix: github.com/${{ github.repository }}

golden:
runs-on: ubuntu-latest
steps:
Expand Down
67 changes: 61 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,16 @@
[![Build](https://img.shields.io/github/workflow/status/vshn/exoscale-metrics-collector/Test)][build]
![Go version](https://img.shields.io/github/go-mod/go-version/vshn/exoscale-metrics-collector)
[![Version](https://img.shields.io/github/v/release/vshn/exoscale-metrics-collector)][releases]
[![Maintainability](https://img.shields.io/codeclimate/maintainability/vshn/exoscale-metrics-collector)][codeclimate]
[![Coverage](https://img.shields.io/codeclimate/coverage/vshn/exoscale-metrics-collector)][codeclimate]
[![GitHub downloads](https://img.shields.io/github/downloads/vshn/exoscale-metrics-collector/total)][releases]

[build]: https://github.com/vshn/exoscale-metrics-collector/actions?query=workflow%3ATest
[releases]: https://github.com/vshn/exoscale-metrics-collector/releases
[codeclimate]: https://codeclimate.com/github/vshn/exoscale-metrics-collector

Batch job to sync usage data from the Exoscale API to the [APPUiO Cloud reporting](https://github.com/appuio/appuio-cloud-reporting/) database.

See the [component documentation](https://hub.syn.tools/exoscale-metrics-collector/index.html) for more information.
Metrics are collected taking into account product (e.g. `object-storage-storage:exoscale`), source (e.g. `exoscale:namespace`), tenant (as organization) and date time.

See the [component documentation](https://hub.syn.tools/exoscale-metrics-collector/index.html) for more information.

## Getting started for developers

Expand All @@ -38,7 +36,18 @@ Then source the env file and run the client:
```
$ . ./env
$ make build
$ ./exoscale-metrics-collector objectstorage
```

Then, run one of the available commands:

* Object Storage:
```
$ ./exoscale-metrics-collector objectstorage
```

* DBaaS (runs metrics collector for all supported databases):
```
$ ./exoscale-metrics-collector dbaas
```

### Billing Database
Expand All @@ -50,6 +59,52 @@ $ cd appuio-cloud-reporting
$ make docker-compose-up
```

For the first time you start the database locally, check `Local Development` on the Readme in [appuio-cloud-reporting](https://github.com/appuio/appuio-cloud-reporting/blob/master/README.md#local-development) or follow these steps:
* Next command asks for a password, it's "reporting":
```
$ createdb --username=reporting -h localhost -p 5432 appuio-cloud-reporting-test
```

* Then (no need to seed the database):
```
$ export ACR_DB_URL="postgres://reporting:reporting@localhost/appuio-cloud-reporting-test?sslmode=disable"
$ go run . migrate
```

### Create Resources in Lab Cluster to test metrics collector

You can first connect to your cluster and then create a claim for Postgres Database by applying a claim, for example:

```
apiVersion: appcat.vshn.io/v1
kind: ExoscalePostgreSQL
metadata:
namespace: default
name: exoscale-postgres-lab-test-1
spec:
parameters:
backup:
timeOfDay: '13:00:00'
maintenance:
dayOfWeek: monday
timeOfDay: "12:00:00"
size:
plan: hobbyist-2
service:
majorVersion: "14"
writeConnectionSecretToRef:
name: postgres-connection-details
```

Once the database is created and `Ready`, you can run locally the command:
```
$ ./exoscale-metrics-collector dbaas
```

The same works for other resources. Just apply the right claim and run the proper command.

And don't forget to delete the resource(s) you created once you're done testing.

### Exoscale API key and secret

You can get your Exoscale API key and secret from the Exoscale web UI. Be sure to select the correct project.
Expand All @@ -72,7 +127,7 @@ $ oc -n default --as cluster-admin apply -f clusterrole-secret.yaml
$ oc -n default --as cluster-admin get secret vshn-exoscale-metrics-collector-secret -o jsonpath='{.data.token}' | base64 -d
```

Instuctions for OpenShift <=4.10:
Instructions for OpenShift <=4.10:
```
$ cd exoscale-metrics-collector
$ oc -n default --as cluster-admin apply -f clusterrole.yaml
Expand Down
76 changes: 76 additions & 0 deletions command.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
package main

import (
"github.com/urfave/cli/v2"
)

const (
keyEnvVariable = "EXOSCALE_API_KEY"
secretEnvVariable = "EXOSCALE_API_SECRET"
dbURLEnvVariable = "ACR_DB_URL"
k8sServerURLEnvVariable = "K8S_SERVER_URL"
k8sTokenEnvVariable = "K8S_TOKEN"
)

type command struct {
clusterURL string
clusterToken string
databaseURL string
exoscaleKey string
exoscaleSecret string
}

func getExoscaleSecretFlag(exoscaleSecret *string) *cli.StringFlag {
return &cli.StringFlag{
Name: "exoscale-secret",
Aliases: []string{"s"},
EnvVars: []string{secretEnvVariable},
Required: true,
Usage: "The secret which has unrestricted SOS service access in an Exoscale organization",
Destination: exoscaleSecret,
}
}

func getExoscaleAccessKeyFlag(exoscaleKey *string) *cli.StringFlag {
return &cli.StringFlag{
Name: "exoscale-access-key",
Aliases: []string{"k"},
EnvVars: []string{keyEnvVariable},
Required: true,
Usage: "A key which has unrestricted SOS service access in an Exoscale organization",
Destination: exoscaleKey,
}
}

func getDatabaseURLFlag(databaseURL *string) *cli.StringFlag {
return &cli.StringFlag{
Name: "database-url",
Aliases: []string{"d"},
EnvVars: []string{dbURLEnvVariable},
Required: true,
Usage: "A PostgreSQL database URL where to save relevant metrics",
Destination: databaseURL,
}
}

func getK8sServerTokenURLFlag(clusterToken *string) *cli.StringFlag {
return &cli.StringFlag{
Name: "k8s-server-token",
Aliases: []string{"t"},
EnvVars: []string{k8sTokenEnvVariable},
Required: true,
Usage: "A Kubernetes server token which can view buckets.exoscale.crossplane.io resources",
Destination: clusterToken,
}
}

func getClusterURLFlag(clusterURL *string) *cli.StringFlag {
return &cli.StringFlag{
Name: "k8s-server-url",
Aliases: []string{"u"},
EnvVars: []string{k8sServerURLEnvVariable},
Required: true,
Usage: "A Kubernetes server URL from where to get the data from",
Destination: clusterURL,
}
}
53 changes: 53 additions & 0 deletions dbaas_command.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
package main

import (
"github.com/urfave/cli/v2"
"github.com/vshn/exoscale-metrics-collector/pkg/clients/cluster"
"github.com/vshn/exoscale-metrics-collector/pkg/clients/exoscale"
"github.com/vshn/exoscale-metrics-collector/pkg/service/dbaas"
ctrl "sigs.k8s.io/controller-runtime"
)

const (
dbaasName = "dbaas"
)

type dbaasCommand struct {
command
}

func newDBaasSCommand() *cli.Command {
command := &dbaasCommand{}
return &cli.Command{
Name: dbaasName,
Usage: "Get metrics from database service",
Action: command.execute,
Flags: []cli.Flag{
getClusterURLFlag(&command.clusterURL),
getK8sServerTokenURLFlag(&command.clusterToken),
getDatabaseURLFlag(&command.databaseURL),
getExoscaleAccessKeyFlag(&command.exoscaleKey),
getExoscaleSecretFlag(&command.exoscaleSecret),
},
}
}

func (c *dbaasCommand) execute(ctx *cli.Context) error {
log := AppLogger(ctx).WithName(dbaasName)
ctrl.SetLogger(log)

log.Info("Creating Exoscale client")
exoscaleClient, err := exoscale.InitClient(c.exoscaleKey, c.exoscaleSecret)
if err != nil {
return err
}

log.Info("Creating k8s client")
k8sClient, err := cluster.InitK8sClientDynamic(c.clusterURL, c.clusterToken)
if err != nil {
return err
}

d := dbaas.NewDBaaSService(exoscaleClient, k8sClient, c.databaseURL)
return d.Execute(ctx.Context)
}
6 changes: 3 additions & 3 deletions docs/modules/ROOT/pages/explanations/data-usage.adoc
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
= Data Usage

This page gives a brief overview how buckets data usage is saved to the postgres billing database.
This page gives a brief overview how resources data usage (e.g. buckets) is saved to the postgres billing database.

== Data flow

image::application-logic.drawio.svg[]

== Data source
- Buckets are fetched from Exoscale Provider and K8s Cluster.
- The bucket names in Exoscale are unique across organisation which prevents clusters having same bucket names.
- Resources (e.g. buckets, DBaaS, ...) are fetched from Exoscale Provider and K8s Cluster.
- The resource names in Exoscale are unique across organisation which prevents clusters having same resource names.

== Data saving

Expand Down
10 changes: 7 additions & 3 deletions docs/modules/ROOT/pages/how-tos/installation.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,17 @@

This component requires https://github.com/appuio/component-appuio-cloud-reporting[component-appuio-cloud-reporting] and is installed into the same namespace.
This is required for this component to be able to access the billing database and its connection secrets.
It also requires an Exoscale IAMKey and a Kubernetes/OpenShift Service Account token in the target cluster to get `buckets.exoscale.crossplane.io` resources.
It also requires an Exoscale IAMKey and a Kubernetes/OpenShift Service Account token in the target cluster to get resources (e.g. SOS buckets, Postgres, etc).

== Sources

The data is matched from k8s cluster and Exoscale organization.
The Kubernetes Service Account token is required to have `get` permissions on `Namespaces` and `buckets.exoscale.crossplane.io` resources.
The Access Key (IAM Key) from an Exoscale organization is required to have read access across all SOS buckets.
The Kubernetes Service Account token is required to have `get` permissions on `Namespaces` and to the following managed resources:

* `buckets.exoscale.crossplane.io`
* `postgresqls.exoscale.crossplane.io`

The Access Key (IAM Key) from an Exoscale organization is required to have read access across all managed resources (e.g. SOS buckets, Postgres, etc).

== Example

Expand Down
4 changes: 4 additions & 0 deletions docs/modules/ROOT/pages/index.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,8 @@

Batch job to sync usage data from the Exoscale metrics API to the https://github.com/appuio/appuio-cloud-reporting/[APPUiO Cloud Reporting] database.

Metrics are collected taking into account product (e.g. `object-storage-storage:exoscale`), source (e.g. `exoscale:namespace`), tenant (organization) and date time.

On DBaaS, we also gather information by Plan. That can be seeing in the product. For example, in the case of a PostgreSQL database service, product may look like `postgres:exoscale:*:*:hobbyist-2`.

See the xref:references/parameters.adoc[parameters] reference for further details.
15 changes: 13 additions & 2 deletions docs/modules/ROOT/pages/references/parameters.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ See xref:how-tos/installation.adoc[Installation] for a full example.
[horizontal]
type:: dictionary
default:: https://github.com/vshn/exoscale-metrics-collector/blob/master/component/class/defaults.yml[See `class/defaults.yml].
default:: https://github.com/vshn/exoscale-metrics-collector/blob/master/component/class/defaults.yml[See class/defaults.yml].
Dictionary containing the container images used by this component.
Expand Down Expand Up @@ -65,4 +65,15 @@ default:: Required.
The token to connect to a Kubernetes cluster.
The Service Account connected to this token should have `get` and `list` permissions to `buckets.exoscale.crossplane.io` managed resource.
The Service Account connected to this token should have `get` and `list` permissions to the following managed resources:
* `buckets.exoscale.crossplane.io`
* `postgresqls.exoscale.crossplane.io`
== `secrets.credentials.stringData.ACR_DB_URL`
[horizontal]
type:: string
default:: Required.
The Billing Database URL.
1 change: 1 addition & 0 deletions docs/modules/ROOT/partials/nav.adoc
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
* xref:index.adoc[Home]
* xref:how-tos/installation.adoc[Installation]
* xref:how-tos/multi-instance.adoc[Multi Instance]
* xref:references/parameters.adoc[Parameters]
7 changes: 0 additions & 7 deletions dummy_test.go

This file was deleted.

Loading

0 comments on commit b299ea8

Please sign in to comment.