[GH-1181] Ingest Workflow Outputs into TDR

[ehigham]

Purpose

https://broadinstitute.atlassian.net/browse/GH-1181

This change is a demonstrator for generic workflow output ingest into TDR. The main idea is:

• Use womtool to describe the outputs of the WDL. Using this description, build a type environment (a mapping from name -> type).
• When ingesting outputs, lookup the type of the output and dispatch on the type to a specific handler to perform any work needed to ingest that object
  • e.g. ingest a file to get its TDR resource identifier
• Transform the result into some form that can be ingested as a row in a dataset table (rename-gather)
  • map output names onto dataset table names (rename)
  • collect together the outputs (gather)
• Upload result and have TDR ingest it.

The main file to look at is api/test/wfl/integration/datarepo_test.clj.
In the test, I've made use of thunks to delay work that can be performed later (e.g. polling). I've tried to add comments to clarify the intent. Let me know if you have questions.

[ehigham]

Not sure if this should be a macro or a function.
If I made it a function, it would itself be a fixture (by symmetry) and passed to other with-fixtures calls which might be kinda nice. On the other hand, we'd have to use more partial applications.

[ehigham]

Guess you can do this as a macro?

(fixtures/with-fixtures
     [(fixtures/with-fixtures 
        [(fixtures/with-fixtures 
           [(fixtures/with-fixtures [])])])]
    identity)
=> [[[[]]]]

[tbl3rd]

Thank you for upload-content.
Not sure how with-fixtures differs from use-fixtures,
but confident using it will bring understanding.

[ehigham]

Thank you for upload-content.
Not sure how with-fixtures differs from use-fixtures,
but confident using it will bring understanding.

The difference comes from what I've called a fixture and what clojure test calls a fixture. In clojure test, a fixture has the form [1]:

(defn my-test-fixture [f]
  (create-resource)
  (f)
  (destroy-resource))

And it's applied to the testing environment via use-fixtures:

(use-fixtures my-test-fixture)

The thunk f passed to the fixture cannot have state passed into it - it is the test itself. Thus fixtures in the clojure test sense have to modify a global variable.
I've introduced a different kind of fixture that allows values to be passed into the function. They can be applied within the test itself have the form:

(defn with-my-fixture [data use]
  (let [resource (acquire data)]
    (try
      (use resource)
      (finally
        (release resource)))))

It's used like so

(deftest my-test
  (with-my-fixture state
    (fn [resource] ... )))

with-fixtures is a combinator for using multiple of these fixtures without excess indentation:

(deftest my-test
  (with-my-fixture0 state0
    (fn [resource0]
      (with-my-fixture1 state1
        (fn [resource1]
          (with-my-fixture2 state2
            (fn [resource2] ... )))))))

becomes:

(deftest my-test
  (with-fixtures [(with-my-fixture0 state0)
                 (with-my-fixture1 state1) 
                 ...
                 (with-my-fixtureN stateN))
    (fn [[resource0 resource1 ... resourceN]] ...)))

[1] https://clojuredocs.org/clojure.test/use-fixtures

[rexwangcc]

with-fixtures is really interesting, I can see macro makes sense here.