Skip to content

Commit 4040c4f

Browse files
drizzenticdrizzentic
authored
Merge branch 'main' into fix-minio-clickhouse-network
2 parents 8645a5b + bb22490 commit 4040c4f

File tree

8 files changed

+137
-114
lines changed

8 files changed

+137
-114
lines changed

‎analytics-datastore-clickhouse/docker-compose.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ services:
2525
datalake:
2626

2727

28+
2829
volumes:
2930
clickhouse-data:
3031

‎database-postgres/package-metadata.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@
99
"REPMGR_PRIMARY_HOST": "postgres-1",
1010
"REPMGR_PARTNER_NODES": "postgres-1",
1111
"REPMGR_PASSWORD": "instant101",
12-
"POSTGRES_IMAGE": "bitnami/postgresql-repmgr:14",
12+
"POSTGRES_IMAGE": "bitnami/postgresql-repmgr:14@sha256:bdf1e4903710c4e0b465664b886d4556897e1b18c07d7c513a4fc1ceba929e02",
1313
"POSTGRES_1_PLACEMENT": "node-1",
1414
"POSTGRES_2_PLACEMENT": "node-2",
1515
"POSTGRES_3_PLACEMENT": "node-3",

‎datalake/docker-compose.yml‎

Lines changed: 19 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -24,16 +24,31 @@ services:
2424
labels:
2525
- traefik.enable=true
2626
- traefik.docker.network=reverse-proxy-traefik_public
27-
- traefik.http.routers.minio.rule=${DOMAIN_NAME_HOST_TRAEFIK} && PathPrefix(`/minio`)
28-
- traefik.http.services.minio.loadbalancer.server.port=9001
29-
- traefik.http.middlewares.minio-stripprefix.stripprefix.prefixes=/minio
30-
- traefik.http.routers.minio.middlewares=minio-stripprefix
27+
- traefik.http.routers.minio-console.rule=Host(`${DOMAIN_NAME_HOST_TRAEFIK}`) && PathPrefix(`/minio-console`)
28+
- traefik.http.routers.minio-console.priority=100
29+
- traefik.http.routers.minio-console.service=minio-console-service
30+
- traefik.http.services.minio-console-service.loadbalancer.server.port=9001
31+
- traefik.http.middlewares.minio-console-stripprefix.stripprefix.prefixes=/minio-console/
32+
- traefik.http.routers.minio-console.middlewares=minio-console-stripprefix
33+
- traefik.http.routers.minio-console.tls=${TLS}
34+
35+
- traefik.http.routers.minio-api.rule=Host(`${DOMAIN_NAME_HOST_TRAEFIK}`) && PathPrefix(`/minio-api`)
36+
- traefik.http.routers.minio-api.priority=100
37+
- traefik.http.routers.minio-api.service=minio-api-service
38+
- traefik.http.services.minio-api-service.loadbalancer.server.port=9090
39+
- traefik.http.middlewares.minio-api-stripprefix.stripprefix.prefixes=/minio-api/
40+
- traefik.http.routers.minio-api.middlewares=minio-api-stripprefix
41+
- traefik.http.routers.minio-api.tls=${TLS}
3142
networks:
3243
public:
44+
traefik:
3345
networks:
3446
public:
3547
name: minio_public
3648
external: true
49+
traefik:
50+
name: reverse-proxy-traefik_public
51+
external: true
3752

3853
volumes:
3954
minio-01-data1:

‎documentation/SUMMARY.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,10 @@
5252
- [Reverse Proxy Nginx](packages/reverse-proxy-nginx/README.md)
5353
- [Local Development](packages/reverse-proxy-nginx/local-development.md)
5454
- [Environment Variables](packages/reverse-proxy-nginx/environment-variables.md)
55+
- [OpenFn](packages/openfn/README.md)
56+
- [Environment Variables](packages/openfn/environment-variables.md)
57+
- [Reverse Proxy Traefik](packages/reverse-proxy-traefik/README.md)
58+
- [Environment Variables](packages/reverse-proxy-traefik/environment-variables.md)
5559
- [🗒️ Cheat sheet](cheat-sheet.md)
5660
- [Architecture](architecture.md)
5761
- [Guides](guides/README.md)

‎documentation/packages/openfn/environment-variables.md‎

Lines changed: 90 additions & 106 deletions
Original file line numberDiff line numberDiff line change
@@ -5,180 +5,164 @@
55
<tr>
66
<th>Variable Name</th>
77
<th>Description</th>
8-
<th>Type</th>
9-
<th>Relevance</th>
10-
<th>Required</th>
118
<th>Default</th>
129
</tr>
1310
</thead>
1411
<tbody>
1512
<tr>
16-
<td>DATABASE_URL</td>
13+
<td>OPENFN_DATABASE_URL</td>
1714
<td>The URL of the PostgreSQL database</td>
18-
<td></td>
19-
<td></td>
20-
<td></td>
21-
<td></td>
15+
<td>postgresql://openfn:instant101@postgres-1:5432/lightning_dev</td>
2216
</tr>
2317
<tr>
24-
<td>DISABLE_DB_SSL</td>
18+
<td>OPENFN_DISABLE_DB_SSL</td>
2519
<td>Whether to disable SSL for the database connection</td>
26-
<td></td>
27-
<td></td>
28-
<td></td>
29-
<td></td>
20+
<td>true</td>
3021
</tr>
3122
<tr>
32-
<td>IS_RESETTABLE_DEMO</td>
23+
<td>OPENFN_IS_RESETTABLE_DEMO</td>
3324
<td>Whether the application is running in resettable demo mode</td>
34-
<td></td>
35-
<td></td>
36-
<td></td>
37-
<td></td>
25+
<td>true</td>
3826
</tr>
3927
<tr>
40-
<td>LISTEN_ADDRESS</td>
28+
<td>OPENFN_LISTEN_ADDRESS</td>
4129
<td>The IP address to listen on</td>
42-
<td></td>
43-
<td></td>
44-
<td></td>
45-
<td></td>
30+
<td>0.0.0.0</td>
4631
</tr>
4732
<tr>
48-
<td>LOG_LEVEL</td>
33+
<td>OPENFN_LOG_LEVEL</td>
4934
<td>The log level for the application</td>
50-
<td></td>
51-
<td></td>
52-
<td></td>
53-
<td></td>
35+
<td>debug</td>
5436
</tr>
5537
<tr>
56-
<td>ORIGINS</td>
38+
<td>OPENFN_ORIGINS</td>
5739
<td>The allowed origins for CORS</td>
58-
<td></td>
59-
<td></td>
60-
<td></td>
61-
<td></td>
40+
<td>http://localhost:4000</td>
6241
</tr>
6342
<tr>
64-
<td>PRIMARY_ENCRYPTION_KEY</td>
43+
<td>OPENFN_PRIMARY_ENCRYPTION_KEY</td>
6544
<td>The primary encryption key</td>
66-
<td></td>
67-
<td></td>
68-
<td></td>
69-
<td></td>
45+
<td>KLu/IoZuaf+baDECd8wG4Z6auwNe6VAmwh9N8lWdJ1A=</td>
7046
</tr>
7147
<tr>
72-
<td>SECRET_KEY_BASE</td>
48+
<td>OPENFN_SECRET_KEY_BASE</td>
7349
<td>The secret key base</td>
74-
<td></td>
75-
<td></td>
76-
<td></td>
77-
<td></td>
50+
<td>jGDxZj2O+Qzegm5wcZ940RfWO4D6RyU8thNCr5BUpHNwa7UNV52M1/Sn+7RxiP+f</td>
7851
</tr>
7952
<tr>
80-
<td>WORKER_RUNS_PRIVATE_KEY</td>
53+
<td>OPENFN_WORKER_RUNS_PRIVATE_KEY</td>
8154
<td>The private key for worker runs</td>
82-
<td></td>
83-
<td></td>
84-
<td></td>
85-
<td></td>
55+
<td>LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2Z0lCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktnd2dnU2tBZ0VBQW9JQkFRREVtR3drUW5pT0hqVCsKMnkyRHFvRUhyT3dLZFI2RW9RWG9DeDE4MytXZ3hNcGthTFZyOFViYVVVQWNISGgzUFp2Z2UwcEIzTWlCWWR5Kwp1ajM1am5uK2JIdk9OZGRldWxOUUdpczdrVFFHRU1nTSs0Njhldm5RS0h6R29DRUhabDlZV0s0MUd5SEZCZXppCnJiOGx2T1A1NEtSTS90aE5pVGtHaUIvTGFLMldLcTh0VmtoSHBvaFE3OGIyR21vNzNmcWtuSGZNWnc0ZE43d1MKdldOamZIN3QwSmhUdW9mTXludUxSWmdFYUhmTDlnbytzZ0thc0ZUTmVvdEZIQkYxQTJjUDJCakwzaUxad0hmdQozTzEwZzg0aGZlTzJqTWlsZlladHNDdmxDTE1EZWNrdFJGWFl6V0dWc25FcFNiOStjcWJWUXRvdEU4QklON09GClRmaEx2MG9uQWdNQkFBRUNnZ0VBV3dmZyt5RTBSVXBEYThiOVdqdzNKdUN4STE1NzFSbmliRUhKVTZzdzNyS0EKck9HM0w5WTI0cHhBdlVPSm5GMFFzbThrUVQ4RU1MU3B6RDdjdDVON2RZMngvaGY4TThhL0VSWXM4cFlYcXI5Vwpnbnh3NldGZ0R6elFHZ0RIaW0raXNudk5ucFdEbTRGVTRObG02d2g5MzVSZlA2KzVaSjJucEJpZjhFWDJLdE9rCklOSHRVbFcwNFlXeDEwS0pIWWhYNFlydXVjL3MraXBORzBCSDZEdlJaQzQxSWw0N1luaTg1OERaL0FaeVNZN1kKWTlTamNKQ0QvUHBENTlNQjlSanJDQjhweDBjWGlsVXBVZUJSYndGalVwbWZuVmhIa1hiYlM1U0hXWWM4K3pLRQp2ajFqSEpxc2UyR0hxK2lHL1V3NTZvcHNyM2x3dHBRUXpVcEJGblhMMFFLQmdRRDM5bkV3L1NNVGhCallSd1JGCkY2a2xOYmltU2RGOVozQlZleXhrT0dUeU5NSCtYckhsQjFpOXBRRHdtMit3V2RvcWg1ZFRFbEU5K1crZ0FhN0YKbXlWc2xPTW4wdnZ2cXY2Wkp5SDRtNTVKU0lWSzBzRjRQOTRMYkpNSStHUW5VNnRha3Y0V0FSMkpXaURabGxPdAp3R01EQWZqRVIrSEFZeUJDKzNDL25MNHF5d0tCZ1FESzk3NERtV0c4VDMzNHBiUFVEYnpDbG9oTlQ2UldxMXVwCmJSWng4ZGpzZU0vQ09kZnBUcmJuMnk5dVc3Q1pBNFVPQ2s4REcxZ3ZENVVDYlpEUVdMaUp5RzZGdG5OdGgvaU8KT1dJM0UyczZOS0VMMU1NVzh5QWZwNzV4Ung5cnNaQzI2UEtqQ0pWL2lTVjcyNlQ1ZTFzRG5sZUtBb0JFZnlDRgpvbEhhMmhybWxRS0JnUURHT1YyOWd1K1NmMng1SVRTWm8xT1ZxbitGZDhlZno1d3V5YnZ3Rm1Fa2V1YUdXZDh1CnJ4UFM3MkJ6K0Y1dUJUWngvMWtLa0w4Zm94TUlQN0FleW1zOWhUeWVybnkyMk9TVlBJSmN3dExqMUxTeDN3L0kKK0kyaVpsYVl1akVlZXpXbHY1S2R0cUNORjk3Zzh0ck1NTnMySVZKa1h1NXFwUk82V0ZXRzZGL2h4d0tCZ0hnNApHYUpFSFhIT204ekZTU2lYSW5FWGZKQmVWZmJIOUxqNzFrbVRlR3RJZTdhTlVHZnVxY1BYUGRiZUZGSHRsY2ZsCkx6dWwzS3V6VFExdEhGTnIyWkl5MTlQM1o1TSs4R2c5Y1FFeVRWYmlpV2xha2x0cmttRnRtQTI4bE0zVEZPWmkKUUNWMUZpZStjaWRVeC9qRnFma1F0c1VXQ2llSUxSazZOY1d0WGpXcEFvR0JBTGN6Y210VGlUUEFvWnk0MFV1QQpTOXpUd3RsamhmUWJEVTVjb21EcnlKcnFRU0VOdmQ2VW5HdW0zYVNnNk13dDc0NGxidDAyMC9mSGI0WTJkTGhMCmx4YWJ5b1dQUElRRUpLL1NNOGtURFEvYTRyME5tZzhuV3h5bGFLcHQ5WUhmZ2NYMkYzSzUrc0VSUGNFcVZlWFMKdWZkYXdYQVlFampZK3V2UHZ2YzU3RU1aCi0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K</td>
56+
</tr>
57+
<tr>
58+
<td>OPENFN_WORKER_SECRET</td>
59+
<td>The secret key for the worker</td>
60+
<td>secret_here</td>
8661
</tr>
8762
<tr>
8863
<td>POSTGRES_USER</td>
8964
<td>The username for the PostgreSQL database</td>
90-
<td></td>
91-
<td></td>
92-
<td></td>
93-
<td></td>
65+
<td>postgres</td>
9466
</tr>
9567
<tr>
9668
<td>POSTGRES_SERVICE</td>
9769
<td>The service name for the PostgreSQL database</td>
98-
<td></td>
99-
<td></td>
100-
<td></td>
101-
<td></td>
70+
<td>postgres-1</td>
10271
</tr>
10372
<tr>
10473
<td>POSTGRES_DATABASE</td>
10574
<td>The name of the PostgreSQL database</td>
106-
<td></td>
107-
<td></td>
108-
<td></td>
109-
<td></td>
75+
<td>postgres</td>
11076
</tr>
11177
<tr>
11278
<td>POSTGRES_PASSWORD</td>
11379
<td>The password for the PostgreSQL database</td>
114-
<td></td>
115-
<td></td>
116-
<td></td>
117-
<td></td>
80+
<td>instant101</td>
11881
</tr>
11982
<tr>
12083
<td>POSTGRES_PORT</td>
12184
<td>The port number for the PostgreSQL database</td>
122-
<td></td>
123-
<td></td>
124-
<td></td>
125-
<td></td>
85+
<td>5432</td>
12686
</tr>
12787
<tr>
128-
<td>OpenFn_POSTGRESQL_DB</td>
88+
<td>OPENFN_POSTGRESQL_DB</td>
12989
<td>The name of the OpenFn PostgreSQL database</td>
130-
<td></td>
131-
<td></td>
132-
<td></td>
133-
<td></td>
90+
<td>lightning_dev</td>
13491
</tr>
13592
<tr>
136-
<td>OpenFn_POSTGRESQL_USERNAME</td>
93+
<td>OPENFN_POSTGRESQL_USERNAME</td>
13794
<td>The username for the OpenFn PostgreSQL database</td>
138-
<td></td>
139-
<td></td>
140-
<td></td>
141-
<td></td>
95+
<td>openfn</td>
14296
</tr>
14397
<tr>
144-
<td>OpenFn_POSTGRESQL_PASSWORD</td>
98+
<td>OPENFN_POSTGRESQL_PASSWORD</td>
14599
<td>The password for the OpenFn PostgreSQL database</td>
146-
<td></td>
147-
<td></td>
148-
<td></td>
149-
<td></td>
100+
<td>instant101</td>
150101
</tr>
151102
<tr>
152-
<td>WORKER_LIGHTNING_PUBLIC_KEY</td>
103+
<td>OPENFN_WORKER_LIGHTNING_PUBLIC_KEY</td>
153104
<td>The public key for the worker lightning</td>
154-
<td></td>
155-
<td></td>
156-
<td></td>
157-
<td></td>
105+
<td>LS0tLS1CRUdJTiBQVUJMSUMgS0VZLS0tLS0KTUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF4SmhzSkVKNGpoNDAvdHN0ZzZxQgpCNnpzQ25VZWhLRUY2QXNkZk4vbG9NVEtaR2kxYS9GRzJsRkFIQng0ZHoyYjRIdEtRZHpJZ1dIY3ZybzkrWTU1Ci9teDd6alhYWHJwVFVCb3JPNUUwQmhESURQdU92SHI1MENoOHhxQWhCMlpmV0ZpdU5Sc2h4UVhzNHEyL0piemoKK2VDa1RQN1lUWWs1Qm9nZnkyaXRsaXF2TFZaSVI2YUlVTy9HOWhwcU85MzZwSngzekdjT0hUZThFcjFqWTN4Kwo3ZENZVTdxSHpNcDdpMFdZQkdoM3kvWUtQcklDbXJCVXpYcUxSUndSZFFObkQ5Z1l5OTRpMmNCMzd0enRkSVBPCklYM2p0b3pJcFgyR2JiQXI1UWl6QTNuSkxVUlYyTTFobGJKeEtVbS9mbkttMVVMYUxSUEFTRGV6aFUzNFM3OUsKSndJREFRQUIKLS0tLS1FTkQgUFVCTElDIEtFWS0tLS0tCg</td>
158106
</tr>
159107
<tr>
160-
<td>WORKER_SECRET</td>
161-
<td>The secret key for the worker</td>
162-
<td></td>
163-
<td></td>
164-
<td></td>
165-
<td></td>
166-
</tr>
167-
<tr>
168-
<td>OpenFn_IMAGE</td>
108+
<td>OPENFN_IMAGE</td>
169109
<td>The image name for OpenFn</td>
170-
<td></td>
171-
<td></td>
172-
<td></td>
173-
<td></td>
110+
<td>openfn/lightning:v2.9.5</td>
174111
</tr>
175112
<tr>
176-
<td>OpenFn_WORKER_IMAGE</td>
113+
<td>OPENFN_WORKER_IMAGE</td>
177114
<td>The image name for OpenFn worker</td>
178-
<td></td>
179-
<td></td>
180-
<td></td>
181-
<td></td>
115+
<td>openfn/ws-worker:latest</td>
116+
</tr>
117+
<tr>
118+
<td>OPENFN_KAFKA_TRIGGERS_ENABLED</td>
119+
<td>Whether Kafka triggers are enabled</td>
120+
<td>true</td>
121+
</tr>
122+
<tr>
123+
<td>OPENFN_API_KEY</td>
124+
<td>The API key for OpenFn</td>
125+
<td>apiKey</td>
126+
</tr>
127+
<tr>
128+
<td>OPENFN_ENDPOINT</td>
129+
<td>The endpoint for OpenFn</td>
130+
<td>http://localhost:4000</td>
131+
</tr>
132+
<tr>
133+
<td>OPENFN_DOCKER_WEB_CPUS</td>
134+
<td>The number of CPUs allocated to the web container</td>
135+
<td>2</td>
136+
</tr>
137+
<tr>
138+
<td>OPENFN_DOCKER_WEB_MEMORY</td>
139+
<td>The amount of memory allocated to the web container</td>
140+
<td>4G</td>
141+
</tr>
142+
<tr>
143+
<td>OPENFN_DOCKER_WORKER_CPUS</td>
144+
<td>The number of CPUs allocated to the worker container</td>
145+
<td>2</td>
146+
</tr>
147+
<tr>
148+
<td>OPENFN_DOCKER_WORKER_MEMORY</td>
149+
<td>The amount of memory allocated to the worker container</td>
150+
<td>4G</td>
151+
</tr>
152+
<tr>
153+
<td>FHIR_SERVER_BASE_URL</td>
154+
<td>The base URL for the FHIR server</td>
155+
<td>http://openhim-core:5001</td>
156+
</tr>
157+
<tr>
158+
<td>FHIR_SERVER_USERNAME</td>
159+
<td>The username for the FHIR server</td>
160+
<td>openfn_client</td>
161+
</tr>
162+
<tr>
163+
<td>FHIR_SERVER_PASSWORD</td>
164+
<td>The password for the FHIR server</td>
165+
<td>openfn_client_password</td>
182166
</tr>
183167
</tbody>
184168
</table>

‎documentation/packages/reverse-proxy-traefik/README.md‎

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -92,14 +92,30 @@ Set the following environment variables in the package-metadata.json in the "./d
9292
Set the following environment variables in the package-metadata.json in the "monitoring" directory
9393

9494
```bash
95-
9695
"environmentVariables":
9796
{
9897
# Other Configurations
9998
...
100-
"MINIO_BROWSER_REDIRECT_URL": "https://domain/minio/"
99+
"MINIO_BROWSER_REDIRECT_URL": "https://domain/minio-console/"
101100
}
101+
```
102+
103+
### MinIO Configuration
104+
105+
The MinIO server is configured to run with the following port settings:
106+
107+
- **API Port**: 9090
108+
- **Console Port**: 9001
109+
110+
Ensure that your Traefik configuration reflects these ports to properly route traffic to the MinIO services. The API can be accessed at `https://<domain>/minio` and the Console at `https://<domain>/minio-console`.
111+
112+
Update your Traefik labels in the `docker-compose.yml` to match these settings:
102113

114+
```yaml
115+
# API Configuration
116+
- traefik.http.services.minio.loadbalancer.server.port=9090
117+
# Console Configuration
118+
- traefik.http.services.minio-console.loadbalancer.server.port=9001
103119
```
104120
105121
### Enabling Grafana

‎monitoring/package-metadata.json‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,6 @@
2929
"MO_SECURITY_ADMIN_PASSWORD": "dev_password_only",
3030
"MO_RETENTION_TIME": "15d",
3131
"GF_SERVER_SERVE_FROM_SUB_PATH": "false",
32-
"MINIO_BROWSER_REDIRECT_URL": "",
3332
"DOCKER_SOCK_FOLDER": "/var/run/docker.sock",
3433
"DOCKER_LIB_FOLDER": "/var/lib/docker/"
3534
}

‎reverse-proxy-traefik/docker-compose.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,10 @@ services:
4949
- traefik.http.middlewares.to-https.redirectscheme.permanent=${REDIRECT_TO_HTTPS}
5050
- traefik.http.middlewares.auth.basicauth.users=${USERNAME}:${PASSWORD}
5151

52+
- "traefik.http.middlewares.bigfiles.buffering.maxRequestBodyBytes=100000000"
53+
- "traefik.http.service.traefik.loadbalancer.server.forwardingTimeouts.dialTimeout=120s"
54+
- "traefik.http.service.traefik.loadbalancer.server.forwardingTimeouts.responseHeaderTimeout=120s"
55+
5256
placement:
5357
max_replicas_per_node: 1
5458
constraints:

0 commit comments

Comments
 (0)