Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GH-1035] Add terra module to WFL #197

Merged
merged 5 commits into from
Oct 28, 2020
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions api/src/wfl/service/terra.clj
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
(ns wfl.service.terra
"Analyze data in Terra using the Firecloud/Terra API."
(:require [clojure.data.json :as json]
[clojure.string :as str]
[clj-http.client :as http]
[wfl.once :as once]
[wfl.util :as util]))

(defn workspace-api-url
[terra-url workspace-namespace workspace-name]
(str terra-url "/api/workspaces/" workspace-namespace "/" workspace-name))

(defn create-submission
"Submit samples in a workspace for analysis with a method configuration in Terra."
[terra-url workspace-namespace workspace-name method-configuration-name
method-configuration-namespace entity-type entity-name]
(let [workspace-url (workspace-api-url terra-url workspace-namespace workspace-name)
submission-url (str workspace-url "/submissions")]
(->
(http/post
submission-url
{:content-type :application/json
:headers (once/get-auth-header)
:body (json/write-str
{
:methodConfigurationNamespace method-configuration-namespace
:methodConfigurationName method-configuration-name
:entityType entity-type
:entityName entity-name
:useCallCache true
} :escape-slash false)})
:body
(util/parse-json)
:submissionId)))

(defn get-submission
"Get information about a Terra Cromwell submission."
[terra-url workspace-namespace workspace-name submission-id]
(let [workspace-url (workspace-api-url terra-url workspace-namespace workspace-name)
submission-url (str workspace-url "/submissions/" submission-id)
response (http/get submission-url {:headers (once/get-auth-header)})]
(util/parse-json (:body response))))

5 changes: 5 additions & 0 deletions api/src/wfl/util.clj
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,11 @@ vault.client.http/http-client ; Keep :clint eastwo
`(try (do ~@body)
(catch Exception x#)))

(defn parse-json
"Parse the json string STR into a keyword-string map"
[str]
(json/read-str str :key-fn keyword))

(defn slurp-json
"Nil or the JSON in FILE."
[file]
Expand Down
34 changes: 34 additions & 0 deletions api/test/wfl/integration/terra_test.clj
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
(ns wfl.integration.terra-test
(:require [clojure.test :refer [deftest is testing]]
[wfl.service.terra :as terra]
[wfl.tools.fixtures :refer [with-temporary-gcs-folder]]))

(def terra-url
"https://firecloud-orchestration.dsde-dev.broadinstitute.org")

(def workspace-namespace
"general-dev-billing-account")

(def workspace-name
"wfl-integration")

(def method-configuration-name
"ExternalWholeGenomeReprocessing")

(def method-configuration-namespace
"general-dev-billing-account")

(def entity-type
"sample")

(def entity-name
"NA12878_PLUMBING")

(deftest test-terra-submission
(testing "A workflow is created for the entity"
(let [submission-id (terra/create-submission terra-url workspace-namespace workspace-name
Copy link
Member

@ehigham ehigham Oct 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding the test. Makes it easier to see how it's meant to be used.

What's the difference between the namespace and the name? Does it make sense to combine these into a fully-qualified name? Thinking this would simplify the call site for both the workspace and the method configuration?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The workspace-namespace corresponds to a gcloud billing project and can contain multiple workspaces. Here's one I set up in dev using the "general-dev-billing-account" namespace: https://bvdp-saturn-dev.appspot.com/#workspaces. I like the idea of combining the two parameters into one since they're just used in the submission url.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also it seems like the method-configuration-namespace will always be the same as the workspace-namespace. But they can be specified individually through the Terra API so I'm assuming there's a case where they can be different.

method-configuration-name method-configuration-namespace
entity-type entity-name)
submission (terra/get-submission terra-url workspace-namespace workspace-name submission-id)
workflow (first (:workflows submission))]
(is (= entity-name (get-in workflow [:workflowEntity :entityName]))))))