Skip to content

feat(api,sdk,ui): Add user-configurable secrets for deployments and ensembling jobs #403

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 29 commits into from
Feb 21, 2025
Merged

feat(api,sdk,ui): Add user-configurable secrets for deployments and ensembling jobs #403

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
a3e966f
Update openapi specs
deadlycoconuts Feb 14, 2025
84881e5
Update autogenerated golang client files
deadlycoconuts Feb 14, 2025
ffd4337
Update autogenerated python client files
deadlycoconuts Feb 14, 2025
adedc79
Add user-configured secrets to batch jobs
deadlycoconuts Feb 14, 2025
6830375
Add missing python client file
deadlycoconuts Feb 14, 2025
5b18c20
Fix lint comments
deadlycoconuts Feb 14, 2025
a9bf075
Add user-configured secrets to enricher and ensembler deployments
deadlycoconuts Feb 17, 2025
8d18641
Black turing sdk files
deadlycoconuts Feb 17, 2025
0d0780f
Add db migration scripts
deadlycoconuts Feb 17, 2025
0376643
Fix lint comments
deadlycoconuts Feb 17, 2025
87cb961
Add sdk changes to allow user-secrets to be mounted
deadlycoconuts Feb 17, 2025
cccd99e
Update e2e tests
deadlycoconuts Feb 18, 2025
1c190fd
Fix enricher secrets parsing
deadlycoconuts Feb 18, 2025
f51dd47
Fix broken validator test
deadlycoconuts Feb 18, 2025
2afcfc4
Fix broken validator tests and add required tag to secrets field in p…
deadlycoconuts Feb 18, 2025
73aaed5
Fix e2e tests
deadlycoconuts Feb 18, 2025
f3a973f
Fix secret map key bug
deadlycoconuts Feb 18, 2025
76194ba
Update jsonb column to have empty list as default value
deadlycoconuts Feb 19, 2025
d50bd56
Update db migration scripts
deadlycoconuts Feb 19, 2025
6bff918
Add missing step in api server to add enricher and ensembler secrets
deadlycoconuts Feb 19, 2025
f7a8c65
Update api specs and autogenerated client files
deadlycoconuts Feb 19, 2025
4b9da38
Replace isnumeric check with check that passes floats
deadlycoconuts Feb 19, 2025
c8406f5
Update react-lazylog with published version
deadlycoconuts Feb 19, 2025
99af479
Add panels to display secrets
deadlycoconuts Feb 19, 2025
fe8fe8d
Add steps to configure secrets in forms
deadlycoconuts Feb 19, 2025
5e1ed57
Add new unit test for autoscaling policy
deadlycoconuts Feb 19, 2025
f488f51
Add fix to unit tests to prevent race conditions
deadlycoconuts Feb 19, 2025
e728486
Update openapi specs and autogenerated files
deadlycoconuts Feb 21, 2025
c13bb6c
Remove redundant variable assignment
deadlycoconuts Feb 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
115 changes: 115 additions & 0 deletions api/api/openapi.bundle.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1617,6 +1617,11 @@ components:
value: value
- name: name
value: value
secrets:
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
monitoring_url: monitoring_url
environment_name: environment_name
properties:
Expand Down Expand Up @@ -1672,6 +1677,11 @@ components:
value: value
- name: name
value: value
secrets:
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
properties:
artifact_uri:
type: string
Expand All @@ -1680,6 +1690,10 @@ components:
service_account_name:
type: string
x-go-custom-tag: validate:"required"
secrets:
items:
$ref: '#/components/schemas/MountedMLPSecret'
type: array
resources:
$ref: '#/components/schemas/EnsemblingResources'
run_id:
Expand Down Expand Up @@ -2136,6 +2150,11 @@ components:
value: value
- name: name
value: value
secrets:
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
timeout: timeout
routes:
- endpoint: endpoint
Expand Down Expand Up @@ -2186,6 +2205,11 @@ components:
value: value
- name: name
value: value
secrets:
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
timeout: timeout
updated_at: 2000-01-23T04:56:07.000+00:00
standard_config:
Expand Down Expand Up @@ -2220,6 +2244,11 @@ components:
value: value
- name: name
value: value
secrets:
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
timeout: timeout
properties:
id:
Expand Down Expand Up @@ -2438,6 +2467,11 @@ components:
value: value
- name: name
value: value
secrets:
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
timeout: timeout
properties:
id:
Expand All @@ -2460,6 +2494,10 @@ components:
items:
$ref: '#/components/schemas/EnvVar'
type: array
secrets:
items:
$ref: '#/components/schemas/MountedMLPSecret'
type: array
service_account:
description: |
(Optional) Name of the secret registered in the current MLP project that contains the Google service account JSON key. This secret will be mounted as a file inside the container and the environment variable GOOGLE_APPLICATION_CREDENTIALS will point to the service account file."
Expand All @@ -2479,6 +2517,7 @@ components:
- image
- port
- resource_request
- secrets
- timeout
type: object
RouterEnsemblerConfig:
Expand All @@ -2500,6 +2539,11 @@ components:
value: value
- name: name
value: value
secrets:
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
timeout: timeout
updated_at: 2000-01-23T04:56:07.000+00:00
standard_config:
Expand Down Expand Up @@ -2534,6 +2578,11 @@ components:
value: value
- name: name
value: value
secrets:
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
timeout: timeout
properties:
id:
Expand Down Expand Up @@ -2609,6 +2658,11 @@ components:
value: value
- name: name
value: value
secrets:
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
timeout: timeout
nullable: true
properties:
Expand All @@ -2630,6 +2684,10 @@ components:
items:
$ref: '#/components/schemas/EnvVar'
type: array
secrets:
items:
$ref: '#/components/schemas/MountedMLPSecret'
type: array
service_account:
description: |
(Optional) Name of the secret registered in the current MLP project that contains the Google service account JSON key. This secret will be mounted as a file inside the container and the environment variable GOOGLE_APPLICATION_CREDENTIALS will point to the service account file."
Expand All @@ -2641,6 +2699,7 @@ components:
- image
- port
- resource_request
- secrets
- timeout
type: object
EnsemblerPyfuncConfig:
Expand All @@ -2662,6 +2721,11 @@ components:
value: value
- name: name
value: value
secrets:
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
timeout: timeout
nullable: true
properties:
Expand All @@ -2680,10 +2744,16 @@ components:
items:
$ref: '#/components/schemas/EnvVar'
type: array
secrets:
items:
$ref: '#/components/schemas/MountedMLPSecret'
type: array
required:
- ensembler_id
- env
- project_id
- resource_request
- secrets
- timeout
type: object
TrafficRule:
Expand Down Expand Up @@ -2794,6 +2864,11 @@ components:
value: value
- name: name
value: value
secrets:
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
timeout: timeout
routes:
- endpoint: endpoint
Expand Down Expand Up @@ -2867,6 +2942,11 @@ components:
value: value
- name: name
value: value
secrets:
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
timeout: timeout
updated_at: 2000-01-23T04:56:07.000+00:00
standard_config:
Expand Down Expand Up @@ -2901,6 +2981,11 @@ components:
value: value
- name: name
value: value
secrets:
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
timeout: timeout
log_config:
bigquery_config:
Expand Down Expand Up @@ -2957,6 +3042,11 @@ components:
value: value
- name: name
value: value
secrets:
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
timeout: timeout
routes:
- endpoint: endpoint
Expand Down Expand Up @@ -3030,6 +3120,11 @@ components:
value: value
- name: name
value: value
secrets:
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
timeout: timeout
updated_at: 2000-01-23T04:56:07.000+00:00
standard_config:
Expand Down Expand Up @@ -3064,6 +3159,11 @@ components:
value: value
- name: name
value: value
secrets:
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
- mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
timeout: timeout
log_config:
bigquery_config:
Expand Down Expand Up @@ -3476,6 +3576,21 @@ components:
format: int32
type: integer
type: object
MountedMLPSecret:
example:
mlp_secret_name: mlp_secret_name
env_var_name: env_var_name
properties:
mlp_secret_name:
pattern: ^[-._a-zA-Z0-9]+$
type: string
env_var_name:
pattern: ^[a-zA-Z0-9_]*$
type: string
required:
- env_var_name
- mlp_secret_name
type: object
EnvVar:
example:
name: name
Expand Down
13 changes: 13 additions & 0 deletions api/api/specs/common.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,19 @@ components:
value:
type: "string"

MountedMLPSecret:
type: "object"
required:
- mlp_secret_name
- env_var_name
properties:
mlp_secret_name:
type: "string"
pattern: '^[-._a-zA-Z0-9]+$'
env_var_name:
type: "string"
pattern: '^[a-zA-Z0-9_]*$'

pagination.Paging:
type: "object"
properties:
Expand Down
4 changes: 4 additions & 0 deletions api/api/specs/jobs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -417,6 +417,10 @@ components:
service_account_name:
type: string
x-go-custom-tag: validate:"required"
secrets:
type: array
items:
$ref: "common.yaml#/components/schemas/MountedMLPSecret"
resources:
$ref: "#/components/schemas/EnsemblingResources"
run_id:
Expand Down
16 changes: 16 additions & 0 deletions api/api/specs/routers.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -782,6 +782,7 @@ components:
- timeout
- port
- env
- secrets
properties:
id:
$ref: "common.yaml#/components/schemas/Id"
Expand All @@ -801,6 +802,10 @@ components:
type: "array"
items:
$ref: "common.yaml#/components/schemas/EnvVar"
secrets:
type: "array"
items:
$ref: "common.yaml#/components/schemas/MountedMLPSecret"
service_account:
type: "string"
description: >
Expand Down Expand Up @@ -888,6 +893,7 @@ components:
- timeout
- port
- env
- secrets
properties:
image:
type: "string"
Expand All @@ -906,6 +912,10 @@ components:
type: "array"
items:
$ref: "common.yaml#/components/schemas/EnvVar"
secrets:
type: "array"
items:
$ref: "common.yaml#/components/schemas/MountedMLPSecret"
service_account:
type: "string"
description: >
Expand All @@ -923,6 +933,8 @@ components:
- ensembler_id
- resource_request
- timeout
- env
- secrets
properties:
project_id:
type: "integer"
Expand All @@ -938,6 +950,10 @@ components:
type: "array"
items:
$ref: "common.yaml#/components/schemas/EnvVar"
secrets:
type: "array"
items:
$ref: "common.yaml#/components/schemas/MountedMLPSecret"

ResourceRequest:
type: "object"
Expand Down
7 changes: 7 additions & 0 deletions api/db-migrations/000016_add_secrets_columns.down.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
-- Remove secrets column for enrichers
ALTER TABLE enrichers DROP COLUMN secrets;

-- Remove secrets field in docker_config and pyfunc_config columns for ensemblers
UPDATE ensembler_configs set docker_config = docker_config - 'secrets' WHERE docker_config IS NOT NULL;

UPDATE ensembler_configs set pyfunc_config = pyfunc_config - 'secrets' WHERE pyfunc_config IS NOT NULL;
7 changes: 7 additions & 0 deletions api/db-migrations/000016_add_secrets_columns.up.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
-- Create secrets column for enrichers
ALTER TABLE enrichers ADD COLUMN secrets jsonb NOT NULL DEFAULT '[]'::jsonb;

-- Create secrets field in docker_config and pyfunc_config columns for ensemblers
UPDATE ensembler_configs SET docker_config = jsonb_set(docker_config, '{secrets}', '[]'::jsonb) WHERE docker_config IS NOT NULL AND docker_config->'secrets' IS NULL;

UPDATE ensembler_configs SET pyfunc_config = jsonb_set(pyfunc_config, '{secrets}', '[]'::jsonb) WHERE pyfunc_config IS NOT NULL AND pyfunc_config->'secrets' IS NULL;
Loading
Loading