GH-986: Remove dsde-pipelines as WFL's dependency and fix what breaks. #195
Conversation
|
Looking good so far! In case you weren't aware, be sure to remove the dependence on dsde-pipelines in api/module.mk! |
tbl3rd
force-pushed
the
tbl/GH-986-goodbye-dsde-pipelines
branch
from
e7b76b9
to
77440f0
Compare
|
Thanks. Might have missed that. |
| @@ -111,7 +95,7 @@ | |||
| (io/make-parents out) | |||
| (io/copy file out)))] | |||
| (let [environments (clone "pipeline-config" "wfl/environments.clj")] | |||
| (stage resources (clone "dsde-pipelines" "tasks/CopyFilesFromCloudToCloud.wdl")) | |||
| (stage resources (clone "warp" "tasks/broad/CopyFilesFromCloudToCloud.wdl")) | |||
| (let [hg38 "gs://gcp-public-data--broad-references/hg38/v0/" | ||
| cram_ref_fasta (str hg38 "Homo_sapiens_assembly38.fasta")] | ||
| {:cram_ref_fasta cram_ref_fasta | ||
| :cram_ref_fasta_index (str cram_ref_fasta ".fai")})) |
There was a problem hiding this comment.
2 parts of me have been struggling on "explicit hard-coded consts" v.s. "dynamically generate data" in Clojure for a year, I preferred the former simply because it's more friendly for full text search, but I guess this (latter) is a de facto in clj?
There was a problem hiding this comment.
I'd prefer these were just literals too now, but the code isn't consistent.
Wanted to just factor the data, but discovered wgs.clj now has a synthetic
reference_fasta_prefix parameter that necessitates something like a
(defn reference_fasta [prefix] …) to support it. Tackling that in this PR
is a design change (and definitely a bridge too far).
There was a problem hiding this comment.
That said, for things like foo and foo.index, I think we should do something to emphasize that the index is just the root with a suffix added.
api/src/wfl/module/xx.clj
Outdated
| (let [hg38 (partial str "gs://gcp-public-data--broad-references/hg38/v0/")] | ||
| (let [hg38 "gs://gcp-public-data--broad-references/hg38/v0/" | ||
| hsa "Homo_sapiens_assembly38" | ||
| bait_set_name "whole_exome_illumina_coding_v1" | ||
| iv1 (str "HybSelOligos/" bait_set_name "/" "." hsa)] |
There was a problem hiding this comment.
I really don't like this - I wrote it this way to be explicit. I can look at the design doc and automatically recognise the defaults without juggling temporaries.
There was a problem hiding this comment.
I don't know what you mean about juggling temporaries.
I agree with you about the scatter-settings.
They are back where they belong.
There was a problem hiding this comment.
FWIW: Wanted to emphasize that inputs depend on the chosen bait set.
tbl3rd
force-pushed
the
tbl/GH-986-goodbye-dsde-pipelines
branch
from
8158502
to
e025041
Compare
| @@ -401,8 +388,7 @@ vault.client.http/http-client ; Keep :clint eastwo | |||
| (defn to-quoted-comma-separated-list | |||
| "Return the sequence `xs` composed into a comma-separated list string. | |||
| Example: | |||
| (to-quoted-comma-separated-list ['x 'y 'z]) | |||
There was a problem hiding this comment.
FYI the extra line is what you get when evaluating in a repl
There was a problem hiding this comment.
Didn't even realize I'd closed that. More interested in the simpler quoting.
Purpose
Changes
Review Instructions