Project domain attribute for workflow Execution config

[pmahindrakar-oss]

Signed-off-by: Prafulla Mahindrakar prafulla.mahindrakar@gmail.com

TL;DR

Refer the flyteadmin PR for how this feature is used
flyteorg/flyteadmin#378

No existing project domain attribute for execution config

flytectl get workflow-execution-config -p flytesnacks -d development
Error: rpc error: code = NotFound desc = Resource [{Project:flytesnacks Domain:development Workflow: LaunchPlan: ResourceType:WORKFLOW_EXECUTION_CONFIG}] not found

Generate a sample file for project domain attribute for execution config

 flytectl get workflow-execution-config -p flytesnacks -d development --attrFile wec.yaml --gen
Generating a sample workflow execution config file
warning file wec.yaml will be overwritten [y/n]: y
wrote the config to file wec.yaml%      

Sample file contents

cat wec.yaml
annotations:
  values:
    cliAnnotationKey: cliAnnotationValue
domain: development
labels:
  values:
    cliLabelKey: cliLabelValue
max_parallelism: 10
project: flytesnacks
raw_output_data_config:
  output_location_prefix: cliOutputLocationPrefix
security_context:
  run_as:
    k8s_service_account: default

Update project domain attribute for execution config

flytectl update workflow-execution-config --attrFile wec.yaml
Updated attributes from flytesnacks project and domain development

GET an existing project domain attribute for execution config

flytectl get workflow-execution-config -p flytesnacks -d development                     
{
  "project": "flytesnacks",
  "domain": "development",
  "max_parallelism": 10,
  "security_context": {
    "run_as": {
      "k8s_service_account": "default"
    }
  },
  "raw_output_data_config": {
    "output_location_prefix": "cliOutputLocationPrefix"
  },
  "labels": {
    "values": {
      "cliLabelKey": "cliLabelValue"
    }
  },
  "annotations": {
    "values": {
      "cliAnnotationKey": "cliAnnotationValue"
    }
  }
}%   

Delete project domain attribute for workflow exeuction config.

flytectl delete workflow-execution-config --attrFile wec.yaml
Deleted matchable resources from flytesnacks project and domain development

Type

• Bug Fix
• Feature
• Plugin

Are all requirements met?

• Code completed
• Smoke tested
• Unit tests added
• Code documentation added
• Any pending items have an associated Issue

Complete description

How did you fix the bug, make the feature etc. Link to any design docs etc

Tracking Issue

flyteorg/flyte#2070

Follow-up issue

NA

[katrogan]

looks good! do you mind updating the PR title and linking the issue

[katrogan]

love this!

[katrogan]

this is great, but it's too bad we have to manually update this if the proto structure changes. is there anyways to use reflection somehow to populate all the fields from the proto definition?

[pmahindrakar-oss]

So we can dump it in following way if we enable EmitDefaults in jsnonpb.Marshaller.
The user still has to know the types for each of the fields when modifying it.

annotations: null
labels: null
maxParallelism: 0
rawOutputDataConfig: null
securityContext: null

Having a customized default lets us give a sample structure with the populated types which is something like this.
And also this gives us to go beyond one level and define deeper customized objects instead of default which would be nil.

annotations:
  values:
    cliAnnotationKey: cliAnnotationValue
domain: development
labels:
  values:
    cliLabelKey: cliLabelValue
max_parallelism: 10
project: flytesnacks
raw_output_data_config:
  output_location_prefix: cliOutputLocationPrefix
security_context:
  run_as:
    k8s_service_account: default

I understand its going to be maintenance to add a new field and requires a corresponding change in the default but doesn't this provide a more user friendly option.
Let me know what you think.

[katrogan]

prioritizing user-friendliness definitely is the superior option! I don't imagine we'll update the proto structure too frequently so this seems like a superior choice 👍

[katrogan]

one more nit: please update the PR title!

[pmahindrakar-oss]

@katrogan need your re-approval. Also please have a look at this issue flyteorg/flyte#2287
as it block workflow matchable attributes for execution config