[GH-1591] Release WFL v0.13.0 #563
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Snapshot creation uses datetime rather than date * Making the name of the column used for datetime interval variable Co-authored-by: rhiananthony <ranthony@broadinstitute.org>
…ultiple times (#358) [GH-1278] Fix SG update-workload! implementation from trying to update the clio BAM records for each Succeeded workflow repeatedly by restoring the :finished guard for register-workload-in-clio. Introducted in https://broadinstitute.atlassian.net/browse/GH-1277.
…Release (#351) * Restore inputs processing hacks. * GDCWholeGenomeSomaticSingleSample moved. * Run off of /develop/ branch instead. * clojure -M:format * Use the GDCWholeGenomeSomaticSingleSample_v1.1.0 release. * Remove the Rich Comment.
* Draft doc module for SG. * Update navigation Camel^H^H^H^H^HYAML.
cherry-pick from main
The `Dockerfile` is being invoked in the root directory - we don't need to add the parent directory, just the current one.
Catch all throwables to make sure a workload failure doesn't bring down workflow-launcher.
* util/do-or-nil-silently should move to build.clj * Fixed indentation to match standard
* First pass: create-submissions generates one submission per entity. * Cleaned up TODOs for first pass attempt, reformatted tests to pass lint step in build. * Second pass: support entity set creation when specifying >1 entity for a submission. Refactored bigquery table dump to pull out now-common code. * Address PR feedback To simplify, will generate an entity set even for the singleton input. Removed firecloud/consolidate-entities-to-set as a result. Need to pass in 'expression' to submission creation payload when specifying an entity set. Supplemented integration and unit tests.
* Create an interface all hornets work on top of. * Apply suggestions from code review * Update readme.md Build Boad.
…ty (#368) * Boad -=> Board * Still doesn't round-trip on doubles and values like "19A". * Unit test wfl.tsv. * Ensure TSV mappability. * Ensure TSV is tabulatable. * Fix bug in assert-mapulatable!.
We create the snapshot in the TDR, and link to it in the workspace via Rawls.
…ema (#375) * Add snapshot_reference_id to Sarscov2IlluminaFull table schema Because schema update hasn't run, altering existing table definition in place. * Documentation changes from local Postgres debugging, onboarding - Clarified order of operations when installing, using local Postgres - Fixed broken intra-doc links - Added docs readme with instruction for launching local documentation site * Add instruction for recreating wfl DB to development docs * Remove references to undefined documentation files
Simply getting the submission again is not sufficient to guarantee that the workflow has been queued for execution. Poll instead.
* Update documentation and infrastructure for release candidates - creating release candiates - bashing release candidates Restore version override in cli.py. Re-tag latest docker images in cli.py so that version can be overwritten. Update IMAGES target in Makefile to add the :latest tag. Clean up :latest images on distclean target. * fix whitespace
Bumps [ssri](https://github.com/npm/ssri) from 6.0.1 to 6.0.2. - [Release notes](https://github.com/npm/ssri/releases) - [Changelog](https://github.com/npm/ssri/blob/v6.0.2/CHANGELOG.md) - [Commits](npm/ssri@v6.0.1...v6.0.2) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* add interactive script ahead of covid-19 demo tomorrow * fix rawls test failure * reformat test * share demo remources with cdc-covid-surveillance firecloud group * corrections from review * format for consistency * clone a dev workspace instead of the prod one * fix test-import-snapshot - workspace changed * fix arrays test failures
* changelog changes * Link workload with source, sink and executor tables. * Update the p-keys to bigint. * SQL has reserved table. * Initial creat covid workload commit * Verifies dataset as well * Update DB schema. * Address comments. * v0.6.0 Release (#359) * [GH-1278] Fix SG update-workload! implementation from updating Clio multiple times (#358) [GH-1278] Fix SG update-workload! implementation from trying to update the clio BAM records for each Succeeded workflow repeatedly by restoring the :finished guard for register-workload-in-clio. Introducted in https://broadinstitute.atlassian.net/browse/GH-1277. * GH-1188: Bump GDCWholeGenomeSomaticSingleSample Version When Lantern Release (#351) * Restore inputs processing hacks. * GDCWholeGenomeSomaticSingleSample moved. * Run off of /develop/ branch instead. * clojure -M:format * Use the GDCWholeGenomeSomaticSingleSample_v1.1.0 release. * Remove the Rich Comment. * GitHub `develop` infrastructure + release changes (#356) Update test actions to only run on PRs. Update release action to only run on new commits into main. Update + format docs about branching off develop and releasing into main. Restore tag-and-push-images in cli.py * GH-1282: Document WFL's support for Somatic Genomes. (#361) * Draft doc module for SG. * Update navigation Camel^H^H^H^H^HYAML. * Update CHANGELOG.md for v0.6.0, including previous patches Co-authored-by: Tom Lyons <tbl3rd@gmail.com> Co-authored-by: Tom Lyons <tbl@broadinstitute.org> * Resolve merge conflicts. * Fix issues generated from rebasing. * Fix the failing integration tests. Co-authored-by: rhiananthony <ranthony@broadinstitute.org> Co-authored-by: Edmund Higham <ehigham@users.noreply.github.com> Co-authored-by: Tom Lyons <tbl3rd@gmail.com> Co-authored-by: Tom Lyons <tbl@broadinstitute.org>
* Add multimethods for source, executor and sink operations Add skeleton implementations for tdr-source * make use of `!` more consistent? * add create operations for source, executor and sink
- Data model change: a snapshot_reference_id is linked to a workspace and should be associated with the executor, not the source (TDR in our use case) - Added covid/get-imported-snapshot-reference - nil or snapshot reference from Rawls for snapshot_reference_id in executor details instance (to be created from TerraExecutorDetails) - Added covid/import-snapshot! - import snapshot to workspace, writing to DB if successful - Added integration tests (incomplete coverage)
RR: https://broadinstitute.atlassian.net/browse/GH-1295 firecloud's flexibleImportEntities has a size limit on the TSV file you POST. Unfortunately, one sarscov2_illumina_full workflow's outputs exceeds this. We can work around this issue by using Rawl's batchUpsert. This is slightly different as it takes a list of operations on how to construct the entity rather than the entity serialised to TSV. In this PR, I've demonstrated how we can use Rawls to import the workflow's outputs into the workspace. I've also updated the demo to do this for a workflow that has already passed.
…them (#372) * Simplify day intervals. * Document wfl.tools.snapshots. * Map name over keyword arguments. * Force the production Rawls. * Checkpoint groups getters. * Patch rebase conflict. * Implement start-covid-workload! kinda. * Checkpoint start-covid-workload! test. * Add covid-workload-request to support unit. * clojure -M:format * Move Rich comment into a unit test. * Respond to comments and tidy up a little * Remove unit test now covered by integration. * Document clojure.test/test-vars to remind me.
Bumps [url-parse](https://github.com/unshiftio/url-parse) from 1.4.7 to 1.5.1. - [Release notes](https://github.com/unshiftio/url-parse/releases) - [Commits](unshiftio/url-parse@1.4.7...1.5.1) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Update db schemas. * Make the names more clear. * Address comments. * Switch to SQL syntax from XML. * Address more comments. * one more update.
Bumps [y18n](https://github.com/yargs/y18n) from 4.0.0 to 4.0.1. - [Release notes](https://github.com/yargs/y18n/releases) - [Changelog](https://github.com/yargs/y18n/blob/master/CHANGELOG.md) - [Commits](https://github.com/yargs/y18n/commits) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ur (#386) RR: https://broadinstitute.atlassian.net/browse/GH-1301 Introduce wfl.util.UserVisibleException - a new type of exception that we should handle to throw and handle errors that users are meant to see. Re-wire the background loop to handle UserVisibleException and also be more exception safe. Start to plumb in email notifications.
In preparation for the next release.
* add null check * Couple of small changes
* pass workload into update functions * Update retry method to take only workload * fix lint error * Couple of fixes * update documentation * add host to user comment * pr updates * boo lint * Add snapshot id to user comment * Change back to space
…543) The returned values are records from the TerraExecutor details table (with the executor's workspace and method configuration, to make it easier to map back to Terra UI). While here, refactored down TerraExecutor workflow fetches to a single fetch with optional filters. Additionally: - Only fetch status when polling Firecloud for updates to active workflows - Moved executor-specific spec def'ns out of common-use all module - Refactored executor JDBC queries for better readability when logged - SQL params when logging prior to JDBC execution -- otherwise can log something like LazySeq without its contents. - Update GitHub Pages documentation
Snapshot deletion obtains an exclusive lock on the underlying dataset. Performing this action on an existing dataset can drive failures in concurrent runs of the same test.
Poll TDR row metadata table for new rows with optional load tag specification Verify that row metadata table exists on workload creation Remove TerraDataRepoSource.table_column_name and all references Include ACCESS_INFORMATION when requesting TDR datasets, snapshots
Previously, each full system test run added 3 active AoU workloads.
tbl3rd
approved these changes
Feb 2, 2022
okotsopoulos
added a commit
that referenced
this pull request
Feb 9, 2022
GH-1539 Stop started workloads in system tests (#562) GH-1563 GH-1573 TerraDataRepoSource polls row metadata table (#559) GH-1433 Test TDR snapshot creation within temporary dataset (#561) GH-1578 TerraDataRepoSource filters old rows before reducing (#560) GH-1542 TerraExecutor notifies Slack watchers on workflow completion (#553)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
Changes
System Tests
Pass locally:
Review Instructions