Skip to content

Commit 9b0ccd2

Browse files
deadlycoconutsdeadlycoconuts
authored
feat(api): Add user-configurable secrets for deployments and ensembling jobs (#403)
* Update openapi specs * Update autogenerated golang client files * Update autogenerated python client files * Add user-configured secrets to batch jobs * Add missing python client file * Fix lint comments * Add user-configured secrets to enricher and ensembler deployments * Black turing sdk files * Add db migration scripts * Fix lint comments * Add sdk changes to allow user-secrets to be mounted * Update e2e tests * Fix enricher secrets parsing * Fix broken validator test * Fix broken validator tests and add required tag to secrets field in pyfunc ensembler config * Fix e2e tests * Fix secret map key bug * Update jsonb column to have empty list as default value * Update db migration scripts * Add missing step in api server to add enricher and ensembler secrets * Update api specs and autogenerated client files * Replace isnumeric check with check that passes floats * Update react-lazylog with published version * Add panels to display secrets * Add steps to configure secrets in forms * Add new unit test for autoscaling policy * Add fix to unit tests to prevent race conditions * Update openapi specs and autogenerated files * Remove redundant variable assignment
1 parent 1175212 commit 9b0ccd2

File tree

76 files changed

+2053
-421
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

76 files changed

+2053
-421
lines changed

‎api/api/openapi.bundle.yaml‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1617,6 +1617,11 @@ components:
16171617
value: value
16181618
- name: name
16191619
value: value
1620+
secrets:
1621+
- mlp_secret_name: mlp_secret_name
1622+
env_var_name: env_var_name
1623+
- mlp_secret_name: mlp_secret_name
1624+
env_var_name: env_var_name
16201625
monitoring_url: monitoring_url
16211626
environment_name: environment_name
16221627
properties:
@@ -1672,6 +1677,11 @@ components:
16721677
value: value
16731678
- name: name
16741679
value: value
1680+
secrets:
1681+
- mlp_secret_name: mlp_secret_name
1682+
env_var_name: env_var_name
1683+
- mlp_secret_name: mlp_secret_name
1684+
env_var_name: env_var_name
16751685
properties:
16761686
artifact_uri:
16771687
type: string
@@ -1680,6 +1690,10 @@ components:
16801690
service_account_name:
16811691
type: string
16821692
x-go-custom-tag: validate:"required"
1693+
secrets:
1694+
items:
1695+
$ref: '#/components/schemas/MountedMLPSecret'
1696+
type: array
16831697
resources:
16841698
$ref: '#/components/schemas/EnsemblingResources'
16851699
run_id:
@@ -2136,6 +2150,11 @@ components:
21362150
value: value
21372151
- name: name
21382152
value: value
2153+
secrets:
2154+
- mlp_secret_name: mlp_secret_name
2155+
env_var_name: env_var_name
2156+
- mlp_secret_name: mlp_secret_name
2157+
env_var_name: env_var_name
21392158
timeout: timeout
21402159
routes:
21412160
- endpoint: endpoint
@@ -2186,6 +2205,11 @@ components:
21862205
value: value
21872206
- name: name
21882207
value: value
2208+
secrets:
2209+
- mlp_secret_name: mlp_secret_name
2210+
env_var_name: env_var_name
2211+
- mlp_secret_name: mlp_secret_name
2212+
env_var_name: env_var_name
21892213
timeout: timeout
21902214
updated_at: 2000-01-23T04:56:07.000+00:00
21912215
standard_config:
@@ -2220,6 +2244,11 @@ components:
22202244
value: value
22212245
- name: name
22222246
value: value
2247+
secrets:
2248+
- mlp_secret_name: mlp_secret_name
2249+
env_var_name: env_var_name
2250+
- mlp_secret_name: mlp_secret_name
2251+
env_var_name: env_var_name
22232252
timeout: timeout
22242253
properties:
22252254
id:
@@ -2438,6 +2467,11 @@ components:
24382467
value: value
24392468
- name: name
24402469
value: value
2470+
secrets:
2471+
- mlp_secret_name: mlp_secret_name
2472+
env_var_name: env_var_name
2473+
- mlp_secret_name: mlp_secret_name
2474+
env_var_name: env_var_name
24412475
timeout: timeout
24422476
properties:
24432477
id:
@@ -2460,6 +2494,10 @@ components:
24602494
items:
24612495
$ref: '#/components/schemas/EnvVar'
24622496
type: array
2497+
secrets:
2498+
items:
2499+
$ref: '#/components/schemas/MountedMLPSecret'
2500+
type: array
24632501
service_account:
24642502
description: |
24652503
(Optional) Name of the secret registered in the current MLP project that contains the Google service account JSON key. This secret will be mounted as a file inside the container and the environment variable GOOGLE_APPLICATION_CREDENTIALS will point to the service account file."
@@ -2479,6 +2517,7 @@ components:
24792517
- image
24802518
- port
24812519
- resource_request
2520+
- secrets
24822521
- timeout
24832522
type: object
24842523
RouterEnsemblerConfig:
@@ -2500,6 +2539,11 @@ components:
25002539
value: value
25012540
- name: name
25022541
value: value
2542+
secrets:
2543+
- mlp_secret_name: mlp_secret_name
2544+
env_var_name: env_var_name
2545+
- mlp_secret_name: mlp_secret_name
2546+
env_var_name: env_var_name
25032547
timeout: timeout
25042548
updated_at: 2000-01-23T04:56:07.000+00:00
25052549
standard_config:
@@ -2534,6 +2578,11 @@ components:
25342578
value: value
25352579
- name: name
25362580
value: value
2581+
secrets:
2582+
- mlp_secret_name: mlp_secret_name
2583+
env_var_name: env_var_name
2584+
- mlp_secret_name: mlp_secret_name
2585+
env_var_name: env_var_name
25372586
timeout: timeout
25382587
properties:
25392588
id:
@@ -2609,6 +2658,11 @@ components:
26092658
value: value
26102659
- name: name
26112660
value: value
2661+
secrets:
2662+
- mlp_secret_name: mlp_secret_name
2663+
env_var_name: env_var_name
2664+
- mlp_secret_name: mlp_secret_name
2665+
env_var_name: env_var_name
26122666
timeout: timeout
26132667
nullable: true
26142668
properties:
@@ -2630,6 +2684,10 @@ components:
26302684
items:
26312685
$ref: '#/components/schemas/EnvVar'
26322686
type: array
2687+
secrets:
2688+
items:
2689+
$ref: '#/components/schemas/MountedMLPSecret'
2690+
type: array
26332691
service_account:
26342692
description: |
26352693
(Optional) Name of the secret registered in the current MLP project that contains the Google service account JSON key. This secret will be mounted as a file inside the container and the environment variable GOOGLE_APPLICATION_CREDENTIALS will point to the service account file."
@@ -2641,6 +2699,7 @@ components:
26412699
- image
26422700
- port
26432701
- resource_request
2702+
- secrets
26442703
- timeout
26452704
type: object
26462705
EnsemblerPyfuncConfig:
@@ -2662,6 +2721,11 @@ components:
26622721
value: value
26632722
- name: name
26642723
value: value
2724+
secrets:
2725+
- mlp_secret_name: mlp_secret_name
2726+
env_var_name: env_var_name
2727+
- mlp_secret_name: mlp_secret_name
2728+
env_var_name: env_var_name
26652729
timeout: timeout
26662730
nullable: true
26672731
properties:
@@ -2680,10 +2744,16 @@ components:
26802744
items:
26812745
$ref: '#/components/schemas/EnvVar'
26822746
type: array
2747+
secrets:
2748+
items:
2749+
$ref: '#/components/schemas/MountedMLPSecret'
2750+
type: array
26832751
required:
26842752
- ensembler_id
2753+
- env
26852754
- project_id
26862755
- resource_request
2756+
- secrets
26872757
- timeout
26882758
type: object
26892759
TrafficRule:
@@ -2794,6 +2864,11 @@ components:
27942864
value: value
27952865
- name: name
27962866
value: value
2867+
secrets:
2868+
- mlp_secret_name: mlp_secret_name
2869+
env_var_name: env_var_name
2870+
- mlp_secret_name: mlp_secret_name
2871+
env_var_name: env_var_name
27972872
timeout: timeout
27982873
routes:
27992874
- endpoint: endpoint
@@ -2867,6 +2942,11 @@ components:
28672942
value: value
28682943
- name: name
28692944
value: value
2945+
secrets:
2946+
- mlp_secret_name: mlp_secret_name
2947+
env_var_name: env_var_name
2948+
- mlp_secret_name: mlp_secret_name
2949+
env_var_name: env_var_name
28702950
timeout: timeout
28712951
updated_at: 2000-01-23T04:56:07.000+00:00
28722952
standard_config:
@@ -2901,6 +2981,11 @@ components:
29012981
value: value
29022982
- name: name
29032983
value: value
2984+
secrets:
2985+
- mlp_secret_name: mlp_secret_name
2986+
env_var_name: env_var_name
2987+
- mlp_secret_name: mlp_secret_name
2988+
env_var_name: env_var_name
29042989
timeout: timeout
29052990
log_config:
29062991
bigquery_config:
@@ -2957,6 +3042,11 @@ components:
29573042
value: value
29583043
- name: name
29593044
value: value
3045+
secrets:
3046+
- mlp_secret_name: mlp_secret_name
3047+
env_var_name: env_var_name
3048+
- mlp_secret_name: mlp_secret_name
3049+
env_var_name: env_var_name
29603050
timeout: timeout
29613051
routes:
29623052
- endpoint: endpoint
@@ -3030,6 +3120,11 @@ components:
30303120
value: value
30313121
- name: name
30323122
value: value
3123+
secrets:
3124+
- mlp_secret_name: mlp_secret_name
3125+
env_var_name: env_var_name
3126+
- mlp_secret_name: mlp_secret_name
3127+
env_var_name: env_var_name
30333128
timeout: timeout
30343129
updated_at: 2000-01-23T04:56:07.000+00:00
30353130
standard_config:
@@ -3064,6 +3159,11 @@ components:
30643159
value: value
30653160
- name: name
30663161
value: value
3162+
secrets:
3163+
- mlp_secret_name: mlp_secret_name
3164+
env_var_name: env_var_name
3165+
- mlp_secret_name: mlp_secret_name
3166+
env_var_name: env_var_name
30673167
timeout: timeout
30683168
log_config:
30693169
bigquery_config:
@@ -3476,6 +3576,21 @@ components:
34763576
format: int32
34773577
type: integer
34783578
type: object
3579+
MountedMLPSecret:
3580+
example:
3581+
mlp_secret_name: mlp_secret_name
3582+
env_var_name: env_var_name
3583+
properties:
3584+
mlp_secret_name:
3585+
pattern: ^[-._a-zA-Z0-9]+$
3586+
type: string
3587+
env_var_name:
3588+
pattern: ^[a-zA-Z0-9_]*$
3589+
type: string
3590+
required:
3591+
- env_var_name
3592+
- mlp_secret_name
3593+
type: object
34793594
EnvVar:
34803595
example:
34813596
name: name

‎api/api/specs/common.yaml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,19 @@ components:
3535
value:
3636
type: "string"
3737

38+
MountedMLPSecret:
39+
type: "object"
40+
required:
41+
- mlp_secret_name
42+
- env_var_name
43+
properties:
44+
mlp_secret_name:
45+
type: "string"
46+
pattern: '^[-._a-zA-Z0-9]+$'
47+
env_var_name:
48+
type: "string"
49+
pattern: '^[a-zA-Z0-9_]*$'
50+
3851
pagination.Paging:
3952
type: "object"
4053
properties:

‎api/api/specs/jobs.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -417,6 +417,10 @@ components:
417417
service_account_name:
418418
type: string
419419
x-go-custom-tag: validate:"required"
420+
secrets:
421+
type: array
422+
items:
423+
$ref: "common.yaml#/components/schemas/MountedMLPSecret"
420424
resources:
421425
$ref: "#/components/schemas/EnsemblingResources"
422426
run_id:

‎api/api/specs/routers.yaml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -782,6 +782,7 @@ components:
782782
- timeout
783783
- port
784784
- env
785+
- secrets
785786
properties:
786787
id:
787788
$ref: "common.yaml#/components/schemas/Id"
@@ -801,6 +802,10 @@ components:
801802
type: "array"
802803
items:
803804
$ref: "common.yaml#/components/schemas/EnvVar"
805+
secrets:
806+
type: "array"
807+
items:
808+
$ref: "common.yaml#/components/schemas/MountedMLPSecret"
804809
service_account:
805810
type: "string"
806811
description: >
@@ -888,6 +893,7 @@ components:
888893
- timeout
889894
- port
890895
- env
896+
- secrets
891897
properties:
892898
image:
893899
type: "string"
@@ -906,6 +912,10 @@ components:
906912
type: "array"
907913
items:
908914
$ref: "common.yaml#/components/schemas/EnvVar"
915+
secrets:
916+
type: "array"
917+
items:
918+
$ref: "common.yaml#/components/schemas/MountedMLPSecret"
909919
service_account:
910920
type: "string"
911921
description: >
@@ -923,6 +933,8 @@ components:
923933
- ensembler_id
924934
- resource_request
925935
- timeout
936+
- env
937+
- secrets
926938
properties:
927939
project_id:
928940
type: "integer"
@@ -938,6 +950,10 @@ components:
938950
type: "array"
939951
items:
940952
$ref: "common.yaml#/components/schemas/EnvVar"
953+
secrets:
954+
type: "array"
955+
items:
956+
$ref: "common.yaml#/components/schemas/MountedMLPSecret"
941957

942958
ResourceRequest:
943959
type: "object"

‎api/db-migrations/000016_add_secrets_columns.down.sql‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
-- Remove secrets column for enrichers
2+
ALTER TABLE enrichers DROP COLUMN secrets;
3+
4+
-- Remove secrets field in docker_config and pyfunc_config columns for ensemblers
5+
UPDATE ensembler_configs set docker_config = docker_config - 'secrets' WHERE docker_config IS NOT NULL;
6+
7+
UPDATE ensembler_configs set pyfunc_config = pyfunc_config - 'secrets' WHERE pyfunc_config IS NOT NULL;

‎api/db-migrations/000016_add_secrets_columns.up.sql‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
-- Create secrets column for enrichers
2+
ALTER TABLE enrichers ADD COLUMN secrets jsonb NOT NULL DEFAULT '[]'::jsonb;
3+
4+
-- Create secrets field in docker_config and pyfunc_config columns for ensemblers
5+
UPDATE ensembler_configs SET docker_config = jsonb_set(docker_config, '{secrets}', '[]'::jsonb) WHERE docker_config IS NOT NULL AND docker_config->'secrets' IS NULL;
6+
7+
UPDATE ensembler_configs SET pyfunc_config = jsonb_set(pyfunc_config, '{secrets}', '[]'::jsonb) WHERE pyfunc_config IS NOT NULL AND pyfunc_config->'secrets' IS NULL;

0 commit comments

Comments
 (0)