Deploy Airflow DAGs after OGC API deployment is complete

[nikki-t]

Purpose

• Register Airflow DAGs (OGC Processes) after the SPS deployment is complete.

Proposed Changes

• [ADD] null_resource Terraform resource that executes the modified post_deployment_terraform.sh to deploy DAGs after the OGC API has been successfully deployed.

Issues

• #410 - [New Feature]: Register DAGs upon SPS deployment

Testing

• Executed Terraform on local laptop and confirmed the DAGs were deployed to the nikki-3 deployment on unity-venue-dev.

Pending question

Do we want to trigger the DAG deployment everytime we run terraform?

[LucaCinquini]

@nikki-t : I think it would be better to not add new variables in .tfvars, but instead read those values from these SSM parameters:

/sps/processing/workflows/unity_username
/sps/processing/workflows/unity_password
/sps/processing/workflows/unity_client_id

The last parameter key/value does not currently exists - I just created that for the unity-dev-venue. But we could ask @jl-0 to create it manually together with the other "/sps/processing/" parameters?

[LucaCinquini]

You could also create those values manually for now in unity-venue-test and unity-venue-prod

[LucaCinquini]

I get these errors when trying to deploy:
unity-sps-initiators.module.unity_initiator.aws_lambda_event_source_mapping.initiator_queue_event_source_mapping: Creation complete after 25s [id=4b0f5bf1-1ee1-47a9-804c-f095357f3c55]
╷
│ Error: Incorrect attribute value type
│
│ on modules/terraform-unity-sps-ogc-processes-api/main.tf line 517, in resource "null_resource" "register_ogc_processes":
│ 517: environment = {
│ 518: OGC_PROCESSES_API = nonsensitive(aws_ssm_parameter.ogc_processes_api_url.value)
│ 519: TOKEN_URL = "https://cognito-idp.${local.region}.amazonaws.com"
│ 520: UNITY_CLIENTID = data.aws_ssm_parameter.unity_client_id
│ 521: UNITY_PASSWORD = data.aws_ssm_parameter.unity_password
│ 522: UNITY_USERNAME = data.aws_ssm_parameter.unity_username
│ 523: }
│ ├────────────────
│ │ data.aws_ssm_parameter.unity_client_id is object with 8 attributes
│ │ data.aws_ssm_parameter.unity_password is object with 8 attributes
│ │ data.aws_ssm_parameter.unity_username is object with 8 attributes
│ │ local.region is "us-west-2"
│
│ Inappropriate value for attribute "environment": element "UNITY_CLIENTID": string required.
╵
(.venv) cinquini@MT-509305 unity-sps %

[nikki-t]

@LucaCinquini - I updated the SSM parameters to retrieve their value from the data sources but may be running into a race condition where the OGC API is not quite available when the Terraform deployment goes to deploy the Airflow DAGs. And therefore the DAGs do not get deployed an the following response is returned: {"message": "forbidden").

@jpl-btlunsfo - Am I looking at the right resource to make sure the OGC API is fully deployed:

unity-sps/terraform-unity/modules/terraform-unity-sps-ogc-processes-api/main.tf

Line 525 in 0fc0fb6

depends_on = [aws_api_gateway_deployment.ogc-api-gateway-deployment, aws_ssm_parameter.ogc_processes_api_url]

It looks like the ogc-api-gateway-deployment resource does wait for the proxy method and integration.

[LucaCinquini]

Well if this ticket cannot be completed so be it... It's still ok to deploy the DAGs as a separate step. Let me know what you and Brad fond out about the dependencies.

[LucaCinquini]

After the fix for the script that waits for the OGC API, this works very well: I was able to redeploy the SPS and the DAGs were automatically published. Note that I had to remove all previous snapshots because there was somewhat a messy state between DAGs that had been published via OGC, and DAGs that were manually copied into the Airflow pods. Restarting from scratch solved that problem.