diff --git a/README.md b/README.md index c2d31a3..8562edd 100644 --- a/README.md +++ b/README.md @@ -39,6 +39,7 @@ kubectl create namespace schema | **postgres.deployment.dbPassword** | Password of the DB user | | **postgres.deployment.dbName** | Name of the DB | | **cluster\_endpoint** | Endpoint of the Kubernetes api server (e.g. https://xxx.xxx.xxx.xxx:443) +| **registry** | URL of the private registry | | **registry.data\_volume.deploy\_volume** | Whether to deploy a storage volume for the registry data | | **registry.data\_volume.size** | same as schema.volume.size for the registry data volume | | **registry.data\_volume.storageClass** | Same as schema.volume.storageClass for the registry data volume | @@ -49,7 +50,11 @@ kubectl create namespace schema | **registry.deployment.password** | Your registry password | | **ftp.deployment.username** | Your FTP username | | **ftp.deployment.password** | Your FTP password | - +| **tesk.url** | The URL of your TESK installation | +| **wes.url** | The URL of your cwl-WES installation | +| **standalone.isStandalone** | Leave to "true" (unless you are running the [CLIMA](https://github.com/athenarc/clima) project management system.) | +| **standalone.Resources** | Maximum resources for job pods when running in standalone mode | +| **metrics.url** | Link to a metrics server dashboard of your choice (leave blank if not available) | Note: you can either create Persistent Volume Claims (PVC) with the appropriate names in ```values.yaml``` or you can allow the helm chart to create them automatically. @@ -57,66 +62,12 @@ Note: you can either create Persistent Volume Claims (PVC) with the appropriate ```bash helm install schema-app deployment -f deployment/values.yaml ``` -4. Using the registry credentials you entered in ```values.yaml``` create a secret in Kubernetes for SCHeMa to be able to pull images from the private registry (Helm will create the registry at https://registry.schema.your-domain.com): -```bash -kubectl create secret docker-registry registry-creds --docker-server= --docker-username= --docker-password= -n schema -``` -5. Get the id of the SCHeMa pod: -```bash -kubectl get pods -n schema -l app=schema -``` -6. Edit ```deployment/config-files/configuration.json``` and fill the appropriate values: - -| Name | Description | -| ------ | ----------- | -| **registry** | URL of the private registry | -| **registryAuth.username** | Private registry username (same as ```values.yaml```) | -| **registryAuth.password** | Private registry password (same as ```values.yaml```) | -| **database.host** | Host of the database (leave unchanged unless you are using an external database server) | -| **database.username** | Username of the database user | -| **database.password** | Password of the database user | -| **database.database** | Name of the database | -| **localftp.domain** | Leave unchanged | -| **localftp.username** | FTP username (same as ```values.yaml```) | -| **localftp.password** | FTP password (same as ```values.yaml```) | -| **imagePullSecrets** | Array of K8s secrets for pulling images (at least "registry-creds" is required) | -| **ftp-creds** | Array of FTP credentials used for remote TES-like API (at least the local FTP credentials) | -| **namespaces** | Leave unchanged | - - -Then, copy the file to the pod: - -```bash -kubectl -n schema cp deployment/config-files/configuration.json :/app/web/schema/scheduler_files -``` - -7. Edit ```deployment/config-files/db.php``` and add the database credentials. Leave the hostname unchanged. Copy to the pod: -```bash -kubectl -n schema cp deployment/config-files/db.php :/app/web/schema/config/ -``` -8. Edit ```deployment/config-files/params.php``` and fill the appropriate values: - -| Name | Description | -| ------ | ----------- | -| **ftpIp** | Leave unchanged | -| **teskEndpoint** | The URL of your TESK installation | -| **wesEndpoint** | The URL of your cwl-WES installation | -| **standalone** | Leave to "true" (unless you are running the [CLIMA](https://github.com/athenarc/clima) project management system.) | -| **standaloneResources** | Maximum resources for job pods when running in standalone mode | -| **metrics\_url** | Link to a metrics server dashboard of your choice (leave blank if not available) | -| **namespaces** | Leave unchanged | - - -and copy the file to the pod: -```bash -kubectl -n schema cp deployment/config-files/params.php :/app/web/schema/config/ -``` -9. Create the database structure and add required data: +4. Create the database structure and add required data: ```bash kubectl -n schema exec -it psql -h postgres.schema.svc.cluster.local -U -d -f /app/web/schema/database_schema/schema_db.sql ``` -10. Run the same command for all migration files ```/app/web/schema/database-schema/migration-xx.sql``` in order. If you are upgrading to the latest version of SCHeMa, please run the miagration files that have been published since the last version. +5. Run the same command for all migration files ```/app/web/schema/database-schema/migration-xx.sql``` in order. If you are upgrading to the latest version of SCHeMa, please run the migration files that have been published since the last version. After all steps have been completed the app should be running as expected. By default a superadministrator account is created and you can login using "superadmin" as username and password. Please change it as soon as possible after logging in. diff --git a/deployment/config-files/configuration.json b/deployment/config-files/configuration.json index c04bdb0..d0896ed 100644 --- a/deployment/config-files/configuration.json +++ b/deployment/config-files/configuration.json @@ -1,23 +1,23 @@ { - "registry": "", + "registry": "{{ .Values.registry.url }}", "registryAuth": { - "username": "***", - "password": "***" + "username": "{{ .Values.registry.deployment.username }}", + "password": "{{ .Values.registry.deployment.password }}" }, "database": { "host":"postgres.schema.svc.cluster.local", - "username": "***", - "password": "***", - "database": "***" + "username": "{{ .Values.postgres.deployment.dbUsername }}", + "password": "{{ .Values.postgres.deployment.dbPassword }}", + "database": "{{ .Values.postgres.deployment.dbName }}" }, "localftp": { "domain": "ftp.schema.svc.cluster.local", - "username": "***", - "password": "***" + "username": "{{ .Values.ftp.deployment.username }}", + "password": "{{ .Values.ftp.deployment.password }}" }, "imagePullSecrets": [ @@ -25,10 +25,10 @@ ], "ftp-creds": { - "postgres.schema.svc.cluster.local": + "ftp.schema.svc.cluster.local": { - "username": "****", - "password": "****" + "username": "{{ .Values.ftp.deployment.username }}", + "password": "{{ .Values.ftp.deployment.password }}" } }, "namespaces": diff --git a/deployment/config-files/db.php b/deployment/config-files/db.php index 3033183..aa3036e 100644 --- a/deployment/config-files/db.php +++ b/deployment/config-files/db.php @@ -2,9 +2,9 @@ return [ 'class' => 'yii\db\Connection', - 'dsn' => 'pgsql:host=postgres.schema.svc.cluster.local;dbname=***', - 'username' => '***', - 'password' => '***', + 'dsn' => 'pgsql:host=postgres.schema.svc.cluster.local;dbname={{ .Values.postgres.deployment.dbName }}', + 'username' => '{{ .Values.postgres.deployment.dbUsername }}', + 'password' => '{{ .Values.postgres.deployment.dbPassword }}', 'charset' => 'utf8', // Schema cache options (for production environment) diff --git a/deployment/config-files/params.php b/deployment/config-files/params.php index 3c0020e..f8ed367 100644 --- a/deployment/config-files/params.php +++ b/deployment/config-files/params.php @@ -16,21 +16,21 @@ 'workflowsFolder' => '/data/docker/workflows', 'archivedWorkflowsFolder' => '/data/docker/archived_workflows', 'systemUser' => 'root', - 'nfsIp' => '****', + 'nfsIp' => '{{ .Values.nfsIp }}', /* * Change the following parameters according to your installation */ 'ftpIp' => 'ftp.schema.svc.cluster.local', - 'teskEndpoint' => '', - 'wesEndpoint' => '', - 'standalone' => true, + 'teskEndpoint' => '{{ .Values.tesk.url }}', + 'wesEndpoint' => '{{ .Values.wes.url }}', + 'standalone' => {{ .Values.standalone.isStandalone }}, 'standaloneResources'=> [ - 'maxCores'=> 8, - 'maxRam' => 16, + 'maxCores'=> {{ .Values.standalone.resources.maxCores }}, + 'maxRam' => {{ .Values.standalone.resources.maxRam }}, ], 'classifierMemLimit'=>8, - 'metrics_url' => '*******', + 'metrics_url' => '{{ .Values.metrics.url }}', 'namespaces' => [ 'jobs'=>'schema' ] diff --git a/deployment/templates/_helpers.tpl b/deployment/templates/_helpers.tpl new file mode 100644 index 0000000..07c36da --- /dev/null +++ b/deployment/templates/_helpers.tpl @@ -0,0 +1,6 @@ +{{/* +Create secret to access docker registry +*/}} +{{- define "imagePullSecret" }} +{{- printf "{\"auths\": {\"%s\": {\"username\":\"%s\",\"password\":\"%s\",\"auth\": \"%s\"}}}" .Values.registry.url .Values.registry.deployment.username .Values.registry.deployment.password (printf "%s:%s" .Values.registry.deployment.username .Values.registry.deployment.password | b64enc) | b64enc }} +{{- end }} diff --git a/deployment/templates/registry/registry-creds.yaml b/deployment/templates/registry/registry-creds.yaml new file mode 100644 index 0000000..cc251e8 --- /dev/null +++ b/deployment/templates/registry/registry-creds.yaml @@ -0,0 +1,7 @@ +apiVersion: v1 +stringData: + .dockerconfigjson: {{ template "imagePullSecret" . }} +kind: Secret +metadata: + name: registry-creds +type: kubernetes.io/dockerconfigjson diff --git a/deployment/templates/registry/registry-ingress.yaml b/deployment/templates/registry/registry-ingress.yaml index 6f4d61b..7cdb524 100644 --- a/deployment/templates/registry/registry-ingress.yaml +++ b/deployment/templates/registry/registry-ingress.yaml @@ -1,3 +1,4 @@ +{{- if .Capabilities.APIVersions.Has "networking.k8s.io/v1/Ingress" }} {{- if .Values.registry.ingress.enabled }} apiVersion: networking.k8s.io/v1 kind: Ingress @@ -28,4 +29,5 @@ spec: - {{ .Values.registry.app_name }}.{{ .Values.domain }} secretName: {{ .Values.registry.app_name }}-ingress-secret {{- end }} -{{- end }} \ No newline at end of file +{{- end }} +{{- end }} diff --git a/deployment/templates/schema/schema-deployment.yaml b/deployment/templates/schema/schema-deployment.yaml index 1927efc..bc7a9f5 100644 --- a/deployment/templates/schema/schema-deployment.yaml +++ b/deployment/templates/schema/schema-deployment.yaml @@ -23,10 +23,30 @@ spec: volumeMounts: - mountPath: /data name: schema-volume + - name: schema-config + mountPath: /app/web/schema/config/db.php + subPath: db.php + - name: schema-config + mountPath: /app/web/schema/config/params.php + subPath: params.php + - name: schema-config + mountPath: /app/web/schema/scheduler_files/configuration.json + subPath: configuration.json volumes: - name: schema-volume persistentVolumeClaim: claimName: {{ .Values.schema.volume.name }} + - name: configuration + configMap: + defaultMode: 420 + items: + - key: configuration.json + path: configuration.json + - key: db.php + path: db.php + - key: params.php + path: params.php + name: configuration restartPolicy: Always imagePullSecrets: - name: schema-regcreds diff --git a/deployment/templates/schema/schema-ingress.yaml b/deployment/templates/schema/schema-ingress.yaml index 3ee1ff6..5b7b4a2 100644 --- a/deployment/templates/schema/schema-ingress.yaml +++ b/deployment/templates/schema/schema-ingress.yaml @@ -1,3 +1,4 @@ +{{- if .Capabilities.APIVersions.Has "networking.k8s.io/v1/Ingress" }} {{- if .Values.schema.ingress.enabled }} apiVersion: networking.k8s.io/v1 kind: Ingress @@ -28,4 +29,5 @@ spec: - {{ .Values.schema.app_name }}.{{ .Values.domain }} secretName: {{ .Values.schema.app_name }}-ingress-secret {{- end }} -{{- end }} \ No newline at end of file +{{- end }} +{{- end }} diff --git a/deployment/templates/schema/schema-secret.yaml b/deployment/templates/schema/schema-secret.yaml new file mode 100644 index 0000000..9597319 --- /dev/null +++ b/deployment/templates/schema/schema-secret.yaml @@ -0,0 +1,65 @@ +apiVersion: v1 +stringData: + configuration.json: "{\n\t\n\t\"registry\": \"{{ .Values.registry.url }}\",\n\t\"registryAuth\":\n\t{\n\t\t\"username\":\"{{ .Values.registry.deployment.username }}\",\n\t\t\"password\": \"{{ .Values.registry.deployment.password }}\"\n\t},\n\t\"database\": \n\t{\n\t\t\"host\":\"postgres.schema.svc.cluster.local\",\n\t\t\"username\":\"{{ .Values.postgres.deployment.dbUsername }}\",\n\t\t\"password\": \"{{ .Values.postgres.deployment.dbPassword }}\",\n\t\t\"database\": \"{{ .Values.postgres.deployment.dbName }}\"\n\t},\n\t\"localftp\":\n\t{\n\t\t\"domain\":\"ftp.schema.svc.cluster.local\",\n\t\t\"username\": \"{{ .Values.ftp.deployment.username }}\",\n\t\t\"password\": \"{{ .Values.ftp.deployment.password }}\"\n\t},\n\t\"imagePullSecrets\":\n\t[\n\t\t{\"name\":\"registry-creds\"}\n\t],\n\t\"ftp-creds\":\n\t{\n\t\t\"ftp.schema.svc.cluster.local\":\n\t\t{\n\t\t\t\"username\":\"{{ .Values.ftp.deployment.username }}\",\n\t\t\t\"password\": \"{{ .Values.ftp.deployment.password }}\"\n\t\t}\n\t},\n\t\"namespaces\":\n\t{\n\t\t\"registry\":\"schema\",\n\t\t\"jobs\":\"schema\",\n\t\t\"tesk\": \"tesk\"\n\t}\n}" + db.php: | + 'yii\db\Connection', + 'dsn' => 'pgsql:host=postgres.schema.svc.cluster.local;dbname={{ .Values.postgres.deployment.dbName }}', + 'username' => '{{ .Values.postgres.deployment.dbUsername }}', + 'password' => '{{ .Values.postgres.deployment.dbPassword }}', + 'charset' => 'utf8', + + // Schema cache options (for production environment) + //'enableSchemaCache' => true, + //'schemaCacheDuration' => 60, + //'schemaCache' => 'cache', + ]; + params.php: | + false, + 'adminEmail' => 'admin@example.com', + 'userDataPath' => '/data/docker/user-data/', + 'tmpFolderPath' => '/data/docker/tmp/', + 'ROCratesFolder' => '/data/docker/RO-crates/', + 'profilesFolderPath' => '/data/docker/profiles', + 'tmpImagePath' => '/data/docker/tmp-images/', + 'tmpWorkflowPath' => '/data/docker/workflows/', + 'scriptsFolder' => '/app/web/schema/scheduler_files/', + 'workflowsFolder' => '/data/docker/workflows', + 'archivedWorkflowsFolder' => '/data/docker/archived_workflows', + 'systemUser' => 'root', + 'nfsIp' => '{{ .Values.nfsIp }}', + /* + * Change the following parameters according to your installation + */ + 'ftpIp' => 'ftp.schema.svc.cluster.local', + 'teskEndpoint' => '{{ .Values.tesk.url }}', + 'wesEndpoint' => '{{ .Values.wes.url }}', + 'standalone' => {{ .Values.standalone.isStandalone }}, + 'standaloneResources'=> + [ + 'maxCores'=> {{ .Values.standalone.resources.maxCores }}, + 'maxRam' => {{ .Values.standalone.resources.maxRam }}, + ], + 'classifierMemLimit'=>8, + 'metrics_url' => '{{ .Values.metrics.url }}', + 'namespaces' => [ + 'jobs'=>'schema' + ] + + + + ]; + + ?> +kind: Secret +metadata: + name: schema-config +type: Opaque diff --git a/deployment/values.yaml b/deployment/values.yaml index 80f4b80..69897d6 100644 --- a/deployment/values.yaml +++ b/deployment/values.yaml @@ -28,10 +28,14 @@ schema: enabled: true issuer: letsencrypt-prod # Insert your certificate cluster issuer (see https://dev.to/chrisme/setting-up-nginx-ingress-w-automatically-generated-letsencrypt-certificates-on-kubernetes-4f1k ) +#IP of the local NFS exposing dir +nfsIp: + # This is the docker registry deployed along with schema registry: app_name: registry + url: data_volume: deploy_volume: true @@ -92,3 +96,18 @@ ftp: image: diwis/schema-ftp:1 username: tesk-1 password: Z6fsH6MG + +tesk: + url: + +wes: + url: + +standalone: + isStandalone: true + resources: + maxCores: 8 + maxRam: 16 + +metrics: + url: