* Snapshot creation uses datetime rather than date

* Making the name of the column used for datetime interval variable

Co-authored-by: rhiananthony <ranthony@broadinstitute.org>

…ultiple times (#358)

[GH-1278] Fix SG update-workload! implementation from trying to update the
clio BAM records for each Succeeded workflow repeatedly by restoring the
:finished guard for register-workload-in-clio.
Introducted in https://broadinstitute.atlassian.net/browse/GH-1277.

…Release (#351)

* Restore inputs processing hacks.
* GDCWholeGenomeSomaticSingleSample moved.
* Run off of /develop/ branch instead.
* clojure -M:format
* Use the GDCWholeGenomeSomaticSingleSample_v1.1.0 release.
* Remove the Rich Comment.

Update test actions to only run on PRs.
Update release action to only run on new commits into main.
Update + format docs about branching off develop and releasing into main.
Restore tag-and-push-images in cli.py

* Draft doc module for SG.
* Update navigation Camel^H^H^H^H^HYAML.

cherry-pick from main

The `Dockerfile` is being invoked in the root directory - we don't need
to add the parent directory, just the current one.

Catch all throwables to make sure a workload failure doesn't bring down
workflow-launcher.

* util/do-or-nil-silently should move to build.clj

* Fixed indentation to match standard

* First pass: create-submissions generates one submission per entity.

* Cleaned up TODOs for first pass attempt, reformatted tests to pass lint step in build.

* Second pass: support entity set creation when specifying >1 entity for a submission.

Refactored bigquery table dump to pull out now-common code.

* Address PR feedback

To simplify, will generate an entity set even for the singleton input.
Removed firecloud/consolidate-entities-to-set as a result.
Need to pass in 'expression' to submission creation payload when specifying an entity set.
Supplemented integration and unit tests.

* Create an interface all hornets work on top of.

* Apply suggestions from code review

* Update readme.md

Build Boad.

…ty (#368)

* Boad -=> Board
* Still doesn't round-trip on doubles and values like "19A".
* Unit test wfl.tsv.
* Ensure TSV mappability.
* Ensure TSV is tabulatable.
* Fix bug in assert-mapulatable!.

We create the snapshot in the TDR, and link to it in the workspace via Rawls.

…ema (#375)

* Add snapshot_reference_id to Sarscov2IlluminaFull table schema

Because schema update hasn't run, altering existing table definition in place.

* Documentation changes from local Postgres debugging, onboarding

- Clarified order of operations when installing, using local Postgres
- Fixed broken intra-doc links
- Added docs readme with instruction for launching local documentation site

* Add instruction for recreating wfl DB to development docs

* Remove references to undefined documentation files

Simply getting the submission again is not sufficient to guarantee that
the workflow has been queued for execution. Poll instead.

* Update documentation and infrastructure for release candidates
- creating release candiates
- bashing release candidates

Restore version override in cli.py.
Re-tag latest docker images in cli.py so that version can be
overwritten.

Update IMAGES target in Makefile to add the :latest tag.
Clean up :latest images on distclean target.

* fix whitespace

Bumps [ssri](https://github.com/npm/ssri) from 6.0.1 to 6.0.2.
- [Release notes](https://github.com/npm/ssri/releases)
- [Changelog](https://github.com/npm/ssri/blob/v6.0.2/CHANGELOG.md)
- [Commits](npm/ssri@v6.0.1...v6.0.2)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* add interactive script ahead of covid-19 demo tomorrow

* fix rawls test failure

* reformat test

* share demo remources with cdc-covid-surveillance firecloud group

* corrections from review

* format for consistency

* clone a dev workspace instead of the prod one

* fix test-import-snapshot - workspace changed

* fix arrays test failures

* changelog changes

* Link workload with source, sink and executor tables.

* Update the p-keys to bigint.

* SQL has reserved table.

* Initial creat covid workload commit

* Verifies dataset as well

* Update DB schema.

* Address comments.

* v0.6.0 Release (#359)

* [GH-1278] Fix SG update-workload! implementation from updating Clio multiple times (#358)

[GH-1278] Fix SG update-workload! implementation from trying to update the
clio BAM records for each Succeeded workflow repeatedly by restoring the
:finished guard for register-workload-in-clio.
Introducted in https://broadinstitute.atlassian.net/browse/GH-1277.

* GH-1188: Bump GDCWholeGenomeSomaticSingleSample Version When Lantern Release (#351)

* Restore inputs processing hacks.
* GDCWholeGenomeSomaticSingleSample moved.
* Run off of /develop/ branch instead.
* clojure -M:format
* Use the GDCWholeGenomeSomaticSingleSample_v1.1.0 release.
* Remove the Rich Comment.

* GitHub `develop` infrastructure + release changes (#356)

Update test actions to only run on PRs.
Update release action to only run on new commits into main.
Update + format docs about branching off develop and releasing into main.
Restore tag-and-push-images in cli.py

* GH-1282: Document WFL's support for Somatic Genomes. (#361)

* Draft doc module for SG.
* Update navigation Camel^H^H^H^H^HYAML.

* Update CHANGELOG.md for v0.6.0, including previous patches

Co-authored-by: Tom Lyons <tbl3rd@gmail.com>
Co-authored-by: Tom Lyons <tbl@broadinstitute.org>

* Resolve merge conflicts.

* Fix issues generated from rebasing.

* Fix the failing integration tests.

Co-authored-by: rhiananthony <ranthony@broadinstitute.org>
Co-authored-by: Edmund Higham <ehigham@users.noreply.github.com>
Co-authored-by: Tom Lyons <tbl3rd@gmail.com>
Co-authored-by: Tom Lyons <tbl@broadinstitute.org>

* Add multimethods for source, executor and sink operations
Add skeleton implementations for tdr-source

* make use of `!` more consistent?

* add create operations for source, executor and sink

- Data model change: a snapshot_reference_id is linked to a workspace and should be associated with the executor, not the source (TDR in our use case)
- Added covid/get-imported-snapshot-reference - nil or snapshot reference from Rawls for snapshot_reference_id in executor details instance (to be created from TerraExecutorDetails)
- Added covid/import-snapshot! - import snapshot to workspace, writing to DB if successful
- Added integration tests (incomplete coverage)

RR: https://broadinstitute.atlassian.net/browse/GH-1295

firecloud's flexibleImportEntities has a size limit on the TSV file you POST. Unfortunately, one sarscov2_illumina_full workflow's outputs exceeds this. We can work around this issue by using Rawl's batchUpsert. This is slightly different as it takes a list of operations on how to construct the entity rather than the entity serialised to TSV.

In this PR, I've demonstrated how we can use Rawls to import the workflow's outputs into the workspace. I've also updated the demo to do this for a workflow that has already passed.

…them (#372)

* Simplify day intervals.
* Document wfl.tools.snapshots.
* Map name over keyword arguments.
* Force the production Rawls.
* Checkpoint groups getters.
* Patch rebase conflict.
* Implement start-covid-workload! kinda.
* Checkpoint start-covid-workload! test.
* Add covid-workload-request to support unit.
* clojure -M:format
* Move Rich comment into a unit test.
* Respond to comments and tidy up a little
* Remove unit test now covered by integration.
* Document clojure.test/test-vars to remind me.

Bumps [url-parse](https://github.com/unshiftio/url-parse) from 1.4.7 to 1.5.1.
- [Release notes](https://github.com/unshiftio/url-parse/releases)
- [Commits](unshiftio/url-parse@1.4.7...1.5.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Update db schemas.

* Make the names more clear.

* Address comments.

* Switch to SQL syntax from XML.

* Address more comments.

* one more update.

Bumps [y18n](https://github.com/yargs/y18n) from 4.0.0 to 4.0.1.
- [Release notes](https://github.com/yargs/y18n/releases)
- [Changelog](https://github.com/yargs/y18n/blob/master/CHANGELOG.md)
- [Commits](https://github.com/yargs/y18n/commits)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ur (#386)

RR: https://broadinstitute.atlassian.net/browse/GH-1301
Introduce wfl.util.UserVisibleException - a new type of exception that we should handle to throw and handle errors that users are meant to see.
Re-wire the background loop to handle UserVisibleException and also be more exception safe. Start to plumb in email notifications.

Bumps [hosted-git-info](https://github.com/npm/hosted-git-info) from 2.8.8 to 2.8.9.
- [Release notes](https://github.com/npm/hosted-git-info/releases)
- [Changelog](https://github.com/npm/hosted-git-info/blob/v2.8.9/CHANGELOG.md)
- [Commits](npm/hosted-git-info@v2.8.8...v2.8.9)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [lodash](https://github.com/lodash/lodash) from 4.17.19 to 4.17.21.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](lodash/lodash@4.17.19...4.17.21)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

RR: https://broadinstitute.atlassian.net/browse/GH-1311
Add support for loading and creating COVID workloads
Add PR test hook for database/ directory
Tweak database schema to make it a bit easier to create workloads

* Add implementation for TerraWorkspaceSink/update-sink!
Rework temporary-postresql-database fixture to override wfl-db-config

* integration test passes

* add make-queue-from-list to be clearer

* fix test

* use max(id) + 1 for new output row instead of table length + 1

Added update-terra-executor and required helpers:
- coerce an available snapshot from source queue to a snapshot reference
- add entry point for submission creation
- write new workflows to executor details instance

Refined queue peek and pop for source and executor.  Notably, queue peeks return a meaningful object (snapshot for source, workflow for executor) rather than a database record.

Removed now-deprecated code from earlier update loop design.
Added integration test.

RR: https://broadinstitute.atlassian.net/browse/GH-1306
Add workflows multimethod
Change all locations where workflows are accessed via the :workflows keyword to use this function.
Implement workflows for COVID

I've added a simple test to make sure the SQL is correct
Awaiting executor implementation to test with more than 0 workflows.

* v0.6.0 Release (#359)

* [GH-1278] Fix SG update-workload! implementation from updating Clio multiple times (#358)

[GH-1278] Fix SG update-workload! implementation from trying to update the
clio BAM records for each Succeeded workflow repeatedly by restoring the
:finished guard for register-workload-in-clio.
Introducted in https://broadinstitute.atlassian.net/browse/GH-1277.

* GH-1188: Bump GDCWholeGenomeSomaticSingleSample Version When Lantern Release (#351)

* Restore inputs processing hacks.
* GDCWholeGenomeSomaticSingleSample moved.
* Run off of /develop/ branch instead.
* clojure -M:format
* Use the GDCWholeGenomeSomaticSingleSample_v1.1.0 release.
* Remove the Rich Comment.

* GitHub `develop` infrastructure + release changes (#356)

Update test actions to only run on PRs.
Update release action to only run on new commits into main.
Update + format docs about branching off develop and releasing into main.
Restore tag-and-push-images in cli.py

* GH-1282: Document WFL's support for Somatic Genomes. (#361)

* Draft doc module for SG.
* Update navigation Camel^H^H^H^H^HYAML.

* Update CHANGELOG.md for v0.6.0, including previous patches

Co-authored-by: Tom Lyons <tbl3rd@gmail.com>
Co-authored-by: Tom Lyons <tbl@broadinstitute.org>

* Release v0.6.1 Patch (#365)

* [GH-1284] Fix docker image generation (#363)

The `Dockerfile` is being invoked in the root directory - we don't need
to add the parent directory, just the current one.

* [GH-1277] Prevent update loop from terminating (#364)

Catch all throwables to make sure a workload failure doesn't bring down
workflow-launcher.

* update changelog + bump patch to v0.6.1

* Remove outdated test code.

* Remove unused utilities code.

* Checkpoint.

* Bump timeout for system tests, as it takes longer to finish now.

* Update docs to include how to override envs for local testing.

* Commit what I had before surgery.

* Wrap up.

* Add a protocol extension for psql array.

* Update DB schema to store row ids for TDR.

* Also store TDR row-ids and query interval info in DB.

* Move protocol test to integration.

* Update utc-now util also randomize the snapshot name with request datetime.

* Resolve merge conflicts.

* Fix weirdness during rebasing...

* Move the logic into the multimethod interface.

* Take out the utc-now util function.

* Move protocol test to jdbc_test and reformat code.

* Remove duplicate code.

* Remove comment blocks.

* Fix broken MD during rebasing.

* Crazy formats.

* Fix a bug with the snapshot naming.

* Add the snapshot_id back to DB schema.

* Include job-status in Source Details table.

* update TDR jobs that are still running as part of source update loop.

* Fix timestamp formatting.

* Exclude the problematic test.

* Fix a hiccup.

* Fix formatting and a small issue with job-result from TDR.

* Minor fixes around covid module.

* Add unit and integration tests.

* Lint.

* Let DB control the primary key auto increment.

* Fix the issues identified by the tests.

* Fix an issue with the jdbc protocol test.

* Update unit tests.

* Call the multi-method in covid test to make sure everything is plumbed correctly.

* Log when TDR snapshot creation job failed.

* Address comments.

* Increase the interval of system test polling.

Co-authored-by: Edmund Higham <ehigham@users.noreply.github.com>
Co-authored-by: Tom Lyons <tbl3rd@gmail.com>
Co-authored-by: Tom Lyons <tbl@broadinstitute.org>

#397)

RR: https://broadinstitute.atlassian.net/browse/GH-1314
Adding a separate workflows selector will allow us to decouple the workload representation from serialisation artefacts that aren’t useful for a user.

In this PR, I'm changing the workload data model such that `GET /api/v1/workloads`
to only return workload metadata (everything but the workflows) and adding
`GET /api/v1/workloads/{uuid}/workflows`
to fetch the workflows in the workload.

RR: https://broadinstitute.atlassian.net/browse/GH-1317

The source of the workload needs to be told that the workload has been started so it can start looking in he datarepo for new data.
I've added a new multi-method for the source to start it, called on start-workload!. This writes last_checked to be the time when we should start listening for new data.

…tatuses (#398)

1. Within a single tx, fetch executor details workflow records with active or failed status.
2. For each record, use Firecloud to fetch up-to-date workflow object. Update the record with the corresponding new workflow status.
3. Within a single tx, write new record statuses to DB.

Updated integration tests.

…402)

In new method `update-method-configuration!`:

1. GET the method configuration from Firecloud
2. If the Firecloud-derived mc version doesn't match our DB record, throw
3. Update the method configuration with the snapshot reference name and POST to Firecloud
4. Increment our DB record's mc version

Supplemented integration tests.

RR: https://broadinstitute.atlassian.net/browse/GH-1322
Adding a new source to workflow-launcher that's a list of snapshots.

RR: https://broadinstitute.atlassian.net/browse/GH-1318
Add `stop-source` multimethod and implement for the TerraDataRepoSource.
When stopped, the source will stop looking for new rows to snaphot in
the TDR.

Also added logic for when the workload is finished. In this case, once
started the workload is finished when it has been stopped and both
source and executor queues are empty. To do this, I added another
multimethod to return the length of the queues.

For the executor, the length of the queue is the number of workflows
that have not been consumed or aborted.

For the source, the length of the queue is the number of unconsumed
records whose snapshot job is not 'failed'.

Note that this is slightly at odds with peek, where peek is meant to
return objects that are ready to be consumed.

…ssing (#395)

* Update vault to work around NoClassDefFoundError.
* Checkpoint failing with PSQLException: FATAL: database ... does not exist 
   database "wfltest147f1cbf976046e68e6beb5eef728674" does not exist
* Spec validation still losing sink and source keys.
* Update most dependencies.
* Restore batch or covid request.
* Checkpoint "working" spec.clj.
* Lift strip-internals out of endpoint API transaction.
* New system test passing again.
* Use Ed's strip-internals fix.
* clojure -M:format
* Respond to comments but now there is no :name in response.
* clojure -M:format
* Make ::name optional for now.

RR: https://broadinstitute.atlassian.net/browse/GH-1320
Adding a list of people who want to access to the snapshots we create to the source request.

* Update the specs.

* Commit pair programming results.

* Commit updated implementation (WIP).

* organizing the keyworrds in spec.clj

* It seems spec/or requires ky-value pairs! Fix it.

* Added start of workflow-request for create-covid-workload function

* changelog changes

* Link workload with source, sink and executor tables.

* Update the p-keys to bigint.

* SQL has reserved table.

* We cannot use table as col name in SQL.

* Checkpoint.

* More.

* Initial creat covid workload commit

* Verifies dataset as well

* Verifications of the source, executor and sink

* More verification of sink and executor

* Verification of source, sink, executor pre review

* Fixing broken test and docstrings instead of comments

* Update api/src/wfl/service/rawls.clj

* Fix and rewrite the test.

* Fix the test part II.

* Adding more tests for the create-covid-workload and adding messages for the cause of a thrown exception

* Updated tests and create covid workload verifications

* removing commente3d out code

* removing get-workspace function from rawls code since we use firecloud to make this call

* Adding lint changes

* split up tests. Added verification of dataset table and columns

* Lint changes

* Fixing merge conflict and lint issue

* Fixing import

* Updated to base off of develop changes

* Adding validation to create workflow logic

* Updates to test, spec and create method

* Working validation tests

* linted. Also removed old Liquibase files no longer used

* cleanup of testing namespace

* removing try/catch since in verify source function

* Moving validation to top of function

* Moving docstrings for two functions

* Fixing two broken integration tests for covid

* Using variables bound in the TDR section for validation

* Moving some code within file to group liike code together

* Fixed all integration tests. Reorganized location of workload methods in covid module

* Fixed all integration tests. Reorganized location of workload methods in covid module

* Removing extraneous description for integration tests relating to create workload method

* Removing extraneous description for integration tests relating to create workload method

* linted

* Converted verification methods to multimethods dispatched off name

* Updated name of validate source, executor, sink functions

* Fixed merge conflicts and linted

* removing unnecessary changeset

* reverted spec.clj to develop branch. Also cleaned up some cruft

* reverted spec.clj to develop branch. Also cleaned up some cruft

* Making get-workspace-json function private and adding a public function which checks the workload. Also cleaned up the create verification methods and removed some unnecessary variables. Also moved source/sink/executor verification functions to the source/sink/executor sections

* Adding default implementations for the create validation functions

* linted

* fixing firecloud method and tests

* Fixing merge conflicts with develop branch

* Adding flags for skipping validation of source, executor and sink in requests (#404)

* Adding flags for skipping validation to source, executor and sink

* linted

* Fixing merge conflict

Co-authored-by: rhiananthony <ranthony@broadinstitute.org>

* linting fixes

* linted and fixes some nits

* Fixing nits

* Fixing nits

* lint fixes

Co-authored-by: Rex <rexwangcc@gmail.com>
Co-authored-by: rhiananthony <ranthony@broadinstitute.org>
Co-authored-by: Chengchen(Rex) Wang <14366016+rexwangcc@users.noreply.github.com>

GH-1316 Create submission within TerraExecutor update

Fully implemented `create-submission!` stub, pulling in existing `update-method-configuration!`.
Simplified code which previously passed around updated executor with incremented method configuration version.
Docstrings converted to use backticked parameter names to preserve camel case.
Expanded integration testing.

RR: https://broadinstitute.atlassian.net/browse/GH-1323 https://broadinstitute.atlassian.net/browse/GH-1321 

Added util/to-edn a general :type dispatch function to coerce an object into a user-friendly EDN view that gets returned to the user.
Allow terra executor to update details table with workflow IDs between update calls
Introduce `done?` to test when the workload should be marked `:finished`
Fetch workflow inputs/outputs from firecloud and transform them into a more workable form
Fix jdbc double exaludations

* Worried that :missing will get swamped by :attributes though.

[GH-1327] Fix Start After Stop
In this change I'm fixing the strange behaviour were we can start a
covid workload after it was stopped. To do this, I've made it illegal to
stop a workload before it's been started.

I've done this because stop before start seems strange to me too. The
purpose of stop was to stop getting new data from the source but allow
all data in the workload pipeline to get flushed (ie. consume all
created snapshots and write all worklfows back). Stop was not meant for
aborting a workload and cancelling/aborting all active prossessing. If
there is a use case for such a thing (and i think there is) - I think we
should expose a new endpoint called `POST /api/v1/abort` or something.

Of course this PR is pretty small so if the team feels strongly about
stop before start I can find another solution/

[GH-1327] update system tests after /stop changes removing verification that we can stop before starting and no workflows are run,

RR: https://broadinstitute.atlassian.net/browse/GH-1333
Caused by database changes not propagated to source code. Destructing
old keys for the source lead to null values.

Our BigQuery query was wong as we were simply calling `toString` on a
java.sql.Timestamp. This lead to intervals like
`['2021-05-26 15:31:18.287641' '2021-05-26T20:00:13']`
The space was causing the problem and BigQuery interpreted that as a
date. The fix is to ensure the lower and upper bound of our query-between
interval is of the same type by coercing java.sql.Timestamp `last_checked`
into a java.time.OffsetDateTime.

Bumps [browserslist](https://github.com/browserslist/browserslist) from 4.13.0 to 4.16.6.
- [Release notes](https://github.com/browserslist/browserslist/releases)
- [Changelog](https://github.com/browserslist/browserslist/blob/main/CHANGELOG.md)
- [Commits](browserslist/browserslist@4.13.0...4.16.6)

Signed-off-by: dependabot[bot] <support@github.com>

RR: https://broadinstitute.atlassian.net/browse/GH-1337
Caused by destructuring the tdr job response with the wrong key.

RR: https://broadinstitute.atlassian.net/browse/GH-1338
This isn't a complete fix sadly but it at least solves the problem for
low ingest rate projects like COVID. For now, only updating
`last_checked` when we successfully found new rows in the dataset.

RR: https://broadinstitute.atlassian.net/browse/GH-1339
When we update the status of snapshot creation jobs, we only get the
job metadata when the job is complete.As a consequence, `nil`s are
propagated up the call stack and the database record is clobbered with
nils, including the `snapshot_creation_job_id` column.

To fix this, always return the metadata and log the failure should one occur.

[GH-1331] Document `/stop` and `/workflows` endpoints
RR: https://broadinstitute.atlassian.net/browse/GH-1331
Updating documentation for changes in v0.7.0:
- workload-response no longer returning workflows
- `GET /api/v1/workload/{uuid}/workflows`
- `POST /api/v1/stop`

Co-authored-by: Chengchen(Rex) Wang <14366016+rexwangcc@users.noreply.github.com>

… and fix method configuration namespace.

RR: https://broadinstitute.atlassian.net/browse/GH-1341
Enforce labels are of the form "name:value" where
- name starts with a letter followed by any combination of letters,
  numbers, underscores and dashes
- value is any non-blank string not containing `:`

I had to include labels in the batch workload request as the coercion
layer got confused when you gave it bad labels.

RR: https://broadinstitute.atlassian.net/browse/GH-1348
The :pipeline attribute of the workload request is meaningless for COVID-type workloads. Note that these workloads aren't even specific to COVID-19 processing, rather a generalisation.
In this PR, I've removed the :pipeline attribute. To do this, all workload operations on maps with a nil :pipeline will be implemented by the functions in the covid namespace.

RR: https://broadinstitute.atlassian.net/browse/GH-1350
The "Terra Workspace" sink validation did not validate that the
attributes listed in fromOutputs exist in the specified entity
type in the workspace. This PR fixes this.

Bumps [dns-packet](https://github.com/mafintosh/dns-packet) from 1.3.1 to 1.3.4.
- [Release notes](https://github.com/mafintosh/dns-packet/releases)
- [Changelog](https://github.com/mafintosh/dns-packet/blob/master/CHANGELOG.md)
- [Commits](mafintosh/dns-packet@v1.3.1...v1.3.4)

Signed-off-by: dependabot[bot] <support@github.com>

Within rawls namespace:
- get-snapshot-references: lazily paginate through all snapshot references in a workspace.
- create-or-get-snapshot-reference: if importing the snapshot fails with a 409, find the first existing snapshot reference matching the snapshot id.

RR: https://broadinstitute.atlassian.net/browse/GH-1354
This guards against erroring while creating snapshots if a user
makes a typo.

RR: https://broadinstitute.atlassian.net/browse/GH-1356
Caused by metosin/reitit#494
Fix as per https://broadinstitute.atlassian.net/browse/GH-1348

Included integration test for reitit coercions to catch this sooner.

RR: https://broadinstitute.atlassian.net/browse/GH-1357
I assumed (wrongly) the way Rawls represented workflows in a submission
was the same as how cromwell represented workflows. I also mocked these
representations incorrectly in tests. To fix this, only use the submission
to fetch workflow IDs, then use firecloud's workflow and workflow/outputs
endpoints so we don't have to adapt/mock multiple data models.

RR: https://broadinstitute.atlassian.net/browse/GH-1359
When sinking a resubmitted workflow, the TerraWorkspaceSink would
append to the existing entity in the workspace. This leaves the entity
in an even more wrong state than the reason the sample was reanalysed.
Our fix (as agreed with cloreth) is to clobber the entity in the
workspace by deleting the entity if it exists then upserting the new one.

The PR check fails in the test that Ed excludes on another branch.

RR: https://broadinstitute.atlassian.net/browse/GH-1363
Caused by firecloud returning "Launching" as a workflow status - we were
testing that the status was in #{"Queued" "Submitted"}.
Fix util/poll so logical FALSE can be returned from the action.

Bumps [ws](https://github.com/websockets/ws) from 6.2.1 to 6.2.2.
- [Release notes](https://github.com/websockets/ws/releases)
- [Commits](https://github.com/websockets/ws/commits)

---
updated-dependencies:
- dependency-name: ws
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…rror from invalid creator (#442)

* Don't spec the batch creator as an email address.

If checking that all workloads are finished, also check that the workload list is nonempty.
Fixed incorrect reference to static workload object from workload creation: we instead want to examine the latest version of the workload.

* Removing the arrays module and the corresponding integraiton test

* Removed references to the deleted arrays module. Also ran all of the integration tests and verified that they pass

* Updating the doc for the aou arrays module to have the appropriate name

Co-authored-by: rhiananthony <ranthony@broadinstitute.org>

…stage ns (#444)

RR: https://broadinstitute.atlassian.net/browse/GH-1388
Inline the `version` key and add a `spec` for the response from the
endpoint. Added system tests.

RR: https://broadinstitute.atlassian.net/browse/GH-1389
These tests were missed in
#446

* Fixing dependabot reported security issues  as well as build reported npm audit security issues

* Fixing lib for is-glob. Part of UI npm packages

* Some updates. Some left to do

* adding package.lock

* Fix by using npm update.

Co-authored-by: rhiananthony <ranthony@broadinstitute.org>
Co-authored-by: Rex <rexwangcc@gmail.com>

* Reduce noise while kibit'zing.
* Add the Eastwood cLinter.
* cLint Eastwood api/src and dampen noise.
* Lint tests too.
* Squash bugs and squelch noise.
* Fix tests now that they "work".
* nil-if-empty -=> not-empty
* Disable some eastwood warnings.
* :staus -=> :status
* Suppress more warnings.
* Expand namespace aliases in keywords.
* Update Kondo.
* LINT -=> FORMAT
* Continue on LINT errors.
* Checkpoint some kondo advice.
* dotimes -=> repeatedly
* Revert "Expand namespace aliases in keywords."
* Build UserException to support Eastwood.
* repeatedly -=> dotimes
* Restore code deleted for debugging.
* Work around: unused-fn-args: Function arg p2__11676# never used
* Maybe explain foldl better.

* Update default values.

* Use a dataset in production for integration tests.

* Lint.

* Remove unused ENV VAR WFL_TDR_SA.

* Address comments.

* Add sleep to avoid the transient 404s from TDR.

RR: https://broadinstitute.atlassian.net/browse/GH-1345
Implementations include:
- `Terra DataRepo` Source
- `TDR Snapshots` Source

…lementation (#449)

[GH-1347] Document the workload `Sink` and `Terra Workspace` sink implementation
RR: https://broadinstitute.atlassian.net/browse/GH-1347

…space (#454)

* [GH-1334] Move `source` Interface and Implementation into new namespace
RR: https://broadinstitute.atlassian.net/browse/GH-1334
Start to move tests too.

* [GH-1335] Move `Executor` Code to new Namespace
RR: https://broadinstitute.atlassian.net/browse/GH-1335
Doc-strings from #451

* [GH-1408] Don't modify source code on build
Generate UserException on prebuild
Make linters depend on prebuild. They're run as part of the PR action
and can be run in parallel locally. Remove formatting target - use deps
for that.

* [GH-1393]: Add `/retry` Endpoint (#452)
RR: https://broadinstitute.atlassian.net/browse/GH-1393
Add `POST /api/v1/workload/{uuid}/retry` route, taking the status as the
request json body.
Add `retry` multimethod to the workload interface. This operation
takes the workload as well as as list of workflows to retry.
Add `workflows-by-status` multimethod to take a status to filter by. No
workloads support `retry` as yet so all implementations throw a 501.
Added system and integration tests to lock down this behaviour.

* Add test for [GH-1385]
RR: https://broadinstitute.atlassian.net/browse/GH-1385