feature : Add version tracking to 'import-to-loculus' metadata #195
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Add option to not include git version here. Otherwise CI must fail. |
added 11 commits
April 7, 2025 16:28
required manual updating of some of the test data
|
removing poetry dynamic versioning - some installation conflict with brotli |
added 3 commits
April 14, 2025 13:01
gordonkoehn
added a commit
that referenced
this pull request
Apr 14, 2025
* feature: add paired read merging on alignments (#125) * copy over the script * update by manual copy * WIP * ignore .bai index files - should be freshly generated * add smallgenomutils conda package * add read pair merger * add sam * WIP * WIP: adding checks * move to temp * move to temp * move constants * Reset wokrflows * fix test * add proper test with check fo sorting * add test coverage * up test coverage * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * lower the test coverage * lower the test coverage --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * test: fix in CI for test coverage (#153) * test: test coverage in CI (#152) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * Update .github/workflows/test.yml praying this is correct. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fixing CI --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fixing CI * test: test coverage in CI (#152) (#154) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * Update .github/workflows/test.yml praying this is correct. --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix CI test coverage (#155) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * backtack: the CI issues are bad (#156) * fix CI test coverage * fix aciton * backtrack on code coverage --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * test: adjusting pytest coverage in CI (#159) * fix CI test coverage * fix aciton * backtrack on code coverage * fixing format of pipeline * fixing format of pipeline * fixing format of pipeline --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * feature: integrate paired-end read merging into CLI (#160) * adding sam_to_bam * add conversion tests * remove comments * specify sorting * rought workflow * add sorting by coordinate * WIP move aux functions to convert * working start-to-end with merging * black it * fix workflwotest * fix workflwotest * fix ruff * ruff ruff --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * history cleanup for dev --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fixing CI test: test coverage in CI (#152) (#154) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * Update .github/workflows/test.yml praying this is correct. --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fix CI test coverage (#155) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> backtack: the CI issues are bad (#156) * fix CI test coverage * fix aciton * backtrack on code coverage --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> test: adjusting pytest coverage in CI (#159) * fix CI test coverage * fix aciton * backtrack on code coverage * fixing format of pipeline * fixing format of pipeline * fixing format of pipeline --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * test: test coverage in CI (#152) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fix test to temp dir * declutter temp dir handling in main (#166) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * add sr2silo in namespace for workflow setup (#165) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * centralize workflow config (#167) * feature / fix: temporary directory handling for amino acid aligment (#179) * Introduces TMPDIR support in both main and translate_align modules * Refactors enrich_read_with_metadata into curry_read_with_metadata and updates related tests * Updates documentation and configuration files for resource requirements and CI environment handling --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * test: validate one/zero based coords (#181) * WUP * WIP tests * support for skipped char * add support for soft clipping * add support for hardclipping * adding Padding support * fix test read 02 * fix last read * move to script * note on coord system * fix black * cleanup test to tmp_path * remove trash * bam_to_fasta_query - support for soft clippings + validation * add test for create index and move test_bam_to_fasta_query to temp * update parse_translate_align text fixture after corrections * reformat ruff --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * fix: Insertions JSON format (#183) * change insertion strings * add validator to pydantic model * update front to end tests * Update src/sr2silo/silo_read_schema.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * satisfy ruff --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * refactor/feature: remove nextclade | add eqn to reads (#187) * layout orthogonal test * Salvage Nextclade Orth Test Changes get insertions form Nextclade get insertions form Nextclade get insertions form Nextclade WIP get nuc sequence WIP: parsesd nextclade to read objects remove trash WIP: failing tests Fix: indexing of bam black formating WIP - got the results to compare WIP all __eq__ adding Nuc comparison WIP making Nextclade use the reference from sr2silo WIP changed sr2silo ref to nextclade - still not workign add sorting by coordinate config feature: add paired read merging on alignments (#125) * copy over the script * update by manual copy * WIP * ignore .bai index files - should be freshly generated * add smallgenomutils conda package * add read pair merger * add sam * WIP * WIP: adding checks * move to temp * move to temp * move constants * Reset wokrflows * fix test * add proper test with check fo sorting * add test coverage * up test coverage * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * lower the test coverage * lower the test coverage --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> test: fix in CI for test coverage (#153) * test: test coverage in CI (#152) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * Update .github/workflows/test.yml praying this is correct. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fixing CI --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fixing CI test: test coverage in CI (#152) (#154) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * Update .github/workflows/test.yml praying this is correct. --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fix CI test coverage (#155) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> backtack: the CI issues are bad (#156) * fix CI test coverage * fix aciton * backtrack on code coverage --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> test: adjusting pytest coverage in CI (#159) * fix CI test coverage * fix aciton * backtrack on code coverage * fixing format of pipeline * fixing format of pipeline * fixing format of pipeline --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> feature: integrate paired-end read merging into CLI (#160) * adding sam_to_bam * add conversion tests * remove comments * specify sorting * rought workflow * add sorting by coordinate * WIP move aux functions to convert * working start-to-end with merging * black it * fix workflwotest * fix workflwotest * fix ruff * ruff ruff --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> test: test coverage in CI (#152) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * Update .github/workflows/test.yml praying this is correct. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> history cleanup for dev --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fixing CI test: test coverage in CI (#152) (#154) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * Update .github/workflows/test.yml praying this is correct. --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fix CI test coverage (#155) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> backtack: the CI issues are bad (#156) * fix CI test coverage * fix aciton * backtrack on code coverage --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> test: adjusting pytest coverage in CI (#159) * fix CI test coverage * fix aciton * backtrack on code coverage * fixing format of pipeline * fixing format of pipeline * fixing format of pipeline --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> test: test coverage in CI (#152) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fix test to temp dir declutter temp dir handling in main (#166) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> add sr2silo in namespace for workflow setup (#165) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> centralize workflow config (#167) feature / fix: temporary directory handling for amino acid aligment (#179) * Introduces TMPDIR support in both main and translate_align modules * Refactors enrich_read_with_metadata into curry_read_with_metadata and updates related tests * Updates documentation and configuration files for resource requirements and CI environment handling --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> test: validate one/zero based coords (#181) * WUP * WIP tests * support for skipped char * add support for soft clipping * add support for hardclipping * adding Padding support * fix test read 02 * fix last read * move to script * note on coord system * fix black * cleanup test to tmp_path * remove trash * bam_to_fasta_query - support for soft clippings + validation * add test for create index and move test_bam_to_fasta_query to temp * update parse_translate_align text fixture after corrections * reformat ruff --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> resolve changing base issues ram ref nextclade change ref to same as V-Pipe fix: Insertions JSON format (#183) * change insertion strings * add validator to pydantic model * update front to end tests * Update src/sr2silo/silo_read_schema.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * satisfy ruff --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> reformat black nextclade test cannot work skip * remove nextclade add tests for insertions Update tests/conftest.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * format black * double code bug * accidental removal * accidental removal --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update resources/sars-cov-2/README.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * remove notebooks * add option to skip paired-end read merging `skip_merge` and tests (#189) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * Add conda packaging and GitHub Actions workflows (#191) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * remove brnach protection (#188) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * Update version to 0.0.4 and prepare for conda packaging * Fix version handling in GitHub release workflow * GitHub Actions(deps): Bump actions/setup-python from 4 to 5 (#193) Bumps [actions/setup-python](https://github.com/actions/setup-python) from 4 to 5. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@v4...v5) --- updated-dependencies: - dependency-name: actions/setup-python dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * GitHub Actions(deps): Bump actions/checkout from 3 to 4 (#192) Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v3...v4) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * docs: specify usage of import-to-silo (#198) * validate silo insertions pydantic schema * add note to docs not to use untrimmed * add tests and allow N in insertiosn --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * fix: snakemake workflow / multiple batch ids clash (#202) * fix combination bug * fix multiple batch issue * fix config path * change path --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * fix: correct unknown / skipped char for amino acid padding (#200) * validate silo insertions pydantic schema * add note to docs not to use untrimmed * add tests and allow N in insertiosn * chang the Amio Acid Validation * changing unknown char for amino acid paddings * update expexte aligned reads fixture * fix test fixture aa insertions * Update src/sr2silo/silo_read_schema.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * update snakemake fixture --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * workflows: adding conda envs to rule (#194) * Adding conda rule support - (not tested) * remove redundant dependncies * adapt submission script --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * changing AA skipped chars in Readme (#205) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * update Readme with conda install (#206) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * feature : Add version tracking to 'import-to-loculus' metadata (#195) * WIP * make version info mandatory * adding version to all of test suits * update database config file * missing version * remove expected comparrions from smk rule * adjust test data * remove unused * generate index sr2silo_version * poetry dynamic versioning * adding transient dependency * install self-verisoning * adjust python version * remove brotli * adjsut * update fallback * simpligf --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR would like to add the current sr2silo version to the metadata output for each processed file.
This would be good practical as in the future we may want to change, adjust how the amino acid alignment is done.
This unfortunately is not quite exact, hence version changes may change output.