Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GH-1035] Add terra module to WFL #197

Merged
merged 5 commits into from
Oct 28, 2020
Merged

Conversation

samanehsan
Copy link
Contributor

@samanehsan samanehsan commented Oct 22, 2020

Purpose

Started as a part of https://broadinstitute.atlassian.net/browse/GH-1018

Changes

  • Adds a terra module with functions to:
    • Create a submission
    • Get a submission by uuid

A submission is like Terra's concept of a workload. It includes what data need to get processed with what pipeline and when you make a request to get a submission it includes a list of the workflows that were started in Cromwell.

Review Instructions

  • No instructions.

@samanehsan samanehsan changed the title [WIP] Add terra module to WFL [GH-1035] Add terra module to WFL Oct 22, 2020
Copy link
Member

@ehigham ehigham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add integration tests for these?

api/src/wfl/service/terra.clj Outdated Show resolved Hide resolved
(defn get-submission
"Get information about a Terra Cromwell submission."
[terra-url workspace-namespace workspace-name submission-id]
(let [workspace-url (workspace-api-url terra-url workspace-namespace workspace-name)]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not call http/get directly?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's some slight benefit to making the code here the same as that for create-submission. If there's a http/post we can call there then sure I guess but this is pretty readable to me as-is.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No particular reason...I referenced another module when writing this one and it used http/request but I think http/get is a bit clearer!

Copy link
Contributor Author

@samanehsan samanehsan Oct 22, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I didn't see Jack's comment until I refreshed. I updated these functions to use http/get and http/post so let me know if you all think this is more readable!

api/src/wfl/service/terra.clj Outdated Show resolved Hide resolved
Copy link
Contributor

@rexwangcc rexwangcc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a good start especially after today's eng discussion. It seems the test is failing and it's crying about the workspace does not exist.

@samanehsan
Copy link
Contributor Author

samanehsan commented Oct 27, 2020

It seems the test is failing and it's crying about the workspace does not exist.

@rexwangcc I just added the wfl service account to the Terra workspace and the test passes now! I also added the rest of the Hornet team.

Copy link
Contributor

@rexwangcc rexwangcc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, LGTM!


(deftest test-terra-submission
(testing "A workflow is created for the entity"
(let [submission-id (terra/create-submission terra-url workspace-namespace workspace-name
Copy link
Member

@ehigham ehigham Oct 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding the test. Makes it easier to see how it's meant to be used.

What's the difference between the namespace and the name? Does it make sense to combine these into a fully-qualified name? Thinking this would simplify the call site for both the workspace and the method configuration?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The workspace-namespace corresponds to a gcloud billing project and can contain multiple workspaces. Here's one I set up in dev using the "general-dev-billing-account" namespace: https://bvdp-saturn-dev.appspot.com/#workspaces. I like the idea of combining the two parameters into one since they're just used in the submission url.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also it seems like the method-configuration-namespace will always be the same as the workspace-namespace. But they can be specified individually through the Terra API so I'm assuming there's a case where they can be different.

@samanehsan samanehsan merged commit e3eb22e into master Oct 28, 2020
@samanehsan samanehsan deleted the se-wfl-terra-integration branch November 9, 2020 17:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants