-
Notifications
You must be signed in to change notification settings - Fork 0
feature: integrate merge into CLI #160
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
3bb8318
to
4dda8e4
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR integrates the paired-end read merging functionality into the CLI import workflow and updates various test modules and process functions to support the new workflow. Key changes include:
- Adding integration tests for sorting, merging, and BAM-to-SAM conversion with paired-end merging.
- Updating logging levels and error handling in the CLI submission workflow.
- Refactoring functions in the process and convert modules to improve consistency and integration.
Reviewed Changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated no comments.
Show a summary per file
File | Description |
---|---|
tests/process/test_convert.py | Updated tests to use Path fixtures and validate new sorting and merging functionality. |
src/sr2silo/import_to_loculus.py | Integrated paired-end merging into the import-to-loculus workflow with temporary file handling. |
tests/conftest.py | Updated test fixtures, including adding a contextmanager-based temporary directory fixture. |
tests/snakemake/test_process_sample.py | Modified test job execution and output validation for process_sample rule. |
src/sr2silo/process/init.py | Refactored module exports to include new functions and maintain consistency. |
src/sr2silo/process/merge.py | Replaced deprecated function calls with updated ones from the convert module. |
src/sr2silo/process/convert.py | Refactored sorting and conversion functions, including updating bam_to_sam to write to a specified file. |
Comments suppressed due to low confidence (2)
tests/snakemake/test_process_sample.py:21
- Consider using the TemporaryDirectory context manager instead of a static 'tmpdir' Path to ensure proper test isolation and cleanup of temporary files.
tmpdir = Path("tmpdir")
tests/conftest.py:87
- [nitpick] Using the @contextmanager decorator on the 'temp_dir' fixture may lead to unexpected usage patterns; consider renaming the fixture or revisiting its interface to ensure consistency with other temporary directory fixtures.
@contextmanager
get insertions form Nextclade get insertions form Nextclade get insertions form Nextclade WIP get nuc sequence WIP: parsesd nextclade to read objects remove trash WIP: failing tests Fix: indexing of bam black formating WIP - got the results to compare WIP all __eq__ adding Nuc comparison WIP making Nextclade use the reference from sr2silo WIP changed sr2silo ref to nextclade - still not workign add sorting by coordinate config feature: add paired read merging on alignments (#125) * copy over the script * update by manual copy * WIP * ignore .bai index files - should be freshly generated * add smallgenomutils conda package * add read pair merger * add sam * WIP * WIP: adding checks * move to temp * move to temp * move constants * Reset wokrflows * fix test * add proper test with check fo sorting * add test coverage * up test coverage * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * lower the test coverage * lower the test coverage --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> test: fix in CI for test coverage (#153) * test: test coverage in CI (#152) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * Update .github/workflows/test.yml praying this is correct. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fixing CI --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fixing CI test: test coverage in CI (#152) (#154) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * Update .github/workflows/test.yml praying this is correct. --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fix CI test coverage (#155) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> backtack: the CI issues are bad (#156) * fix CI test coverage * fix aciton * backtrack on code coverage --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> test: adjusting pytest coverage in CI (#159) * fix CI test coverage * fix aciton * backtrack on code coverage * fixing format of pipeline * fixing format of pipeline * fixing format of pipeline --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> feature: integrate paired-end read merging into CLI (#160) * adding sam_to_bam * add conversion tests * remove comments * specify sorting * rought workflow * add sorting by coordinate * WIP move aux functions to convert * working start-to-end with merging * black it * fix workflwotest * fix workflwotest * fix ruff * ruff ruff --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> test: test coverage in CI (#152) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * Update .github/workflows/test.yml praying this is correct. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> history cleanup for dev --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fixing CI test: test coverage in CI (#152) (#154) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * Update .github/workflows/test.yml praying this is correct. --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fix CI test coverage (#155) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> backtack: the CI issues are bad (#156) * fix CI test coverage * fix aciton * backtrack on code coverage --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> test: adjusting pytest coverage in CI (#159) * fix CI test coverage * fix aciton * backtrack on code coverage * fixing format of pipeline * fixing format of pipeline * fixing format of pipeline --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> test: test coverage in CI (#152) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fix test to temp dir declutter temp dir handling in main (#166) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> add sr2silo in namespace for workflow setup (#165) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> centralize workflow config (#167) feature / fix: temporary directory handling for amino acid aligment (#179) * Introduces TMPDIR support in both main and translate_align modules * Refactors enrich_read_with_metadata into curry_read_with_metadata and updates related tests * Updates documentation and configuration files for resource requirements and CI environment handling --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> test: validate one/zero based coords (#181) * WUP * WIP tests * support for skipped char * add support for soft clipping * add support for hardclipping * adding Padding support * fix test read 02 * fix last read * move to script * note on coord system * fix black * cleanup test to tmp_path * remove trash * bam_to_fasta_query - support for soft clippings + validation * add test for create index and move test_bam_to_fasta_query to temp * update parse_translate_align text fixture after corrections * reformat ruff --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> resolve changing base issues ram ref nextclade change ref to same as V-Pipe fix: Insertions JSON format (#183) * change insertion strings * add validator to pydantic model * update front to end tests * Update src/sr2silo/silo_read_schema.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * satisfy ruff --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> reformat black nextclade test cannot work skip
* layout orthogonal test * Salvage Nextclade Orth Test Changes get insertions form Nextclade get insertions form Nextclade get insertions form Nextclade WIP get nuc sequence WIP: parsesd nextclade to read objects remove trash WIP: failing tests Fix: indexing of bam black formating WIP - got the results to compare WIP all __eq__ adding Nuc comparison WIP making Nextclade use the reference from sr2silo WIP changed sr2silo ref to nextclade - still not workign add sorting by coordinate config feature: add paired read merging on alignments (#125) * copy over the script * update by manual copy * WIP * ignore .bai index files - should be freshly generated * add smallgenomutils conda package * add read pair merger * add sam * WIP * WIP: adding checks * move to temp * move to temp * move constants * Reset wokrflows * fix test * add proper test with check fo sorting * add test coverage * up test coverage * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * lower the test coverage * lower the test coverage --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> test: fix in CI for test coverage (#153) * test: test coverage in CI (#152) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * Update .github/workflows/test.yml praying this is correct. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fixing CI --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fixing CI test: test coverage in CI (#152) (#154) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * Update .github/workflows/test.yml praying this is correct. --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fix CI test coverage (#155) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> backtack: the CI issues are bad (#156) * fix CI test coverage * fix aciton * backtrack on code coverage --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> test: adjusting pytest coverage in CI (#159) * fix CI test coverage * fix aciton * backtrack on code coverage * fixing format of pipeline * fixing format of pipeline * fixing format of pipeline --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> feature: integrate paired-end read merging into CLI (#160) * adding sam_to_bam * add conversion tests * remove comments * specify sorting * rought workflow * add sorting by coordinate * WIP move aux functions to convert * working start-to-end with merging * black it * fix workflwotest * fix workflwotest * fix ruff * ruff ruff --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> test: test coverage in CI (#152) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * Update .github/workflows/test.yml praying this is correct. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> history cleanup for dev --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fixing CI test: test coverage in CI (#152) (#154) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * Update .github/workflows/test.yml praying this is correct. --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fix CI test coverage (#155) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> backtack: the CI issues are bad (#156) * fix CI test coverage * fix aciton * backtrack on code coverage --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> test: adjusting pytest coverage in CI (#159) * fix CI test coverage * fix aciton * backtrack on code coverage * fixing format of pipeline * fixing format of pipeline * fixing format of pipeline --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> test: test coverage in CI (#152) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fix test to temp dir declutter temp dir handling in main (#166) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> add sr2silo in namespace for workflow setup (#165) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> centralize workflow config (#167) feature / fix: temporary directory handling for amino acid aligment (#179) * Introduces TMPDIR support in both main and translate_align modules * Refactors enrich_read_with_metadata into curry_read_with_metadata and updates related tests * Updates documentation and configuration files for resource requirements and CI environment handling --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> test: validate one/zero based coords (#181) * WUP * WIP tests * support for skipped char * add support for soft clipping * add support for hardclipping * adding Padding support * fix test read 02 * fix last read * move to script * note on coord system * fix black * cleanup test to tmp_path * remove trash * bam_to_fasta_query - support for soft clippings + validation * add test for create index and move test_bam_to_fasta_query to temp * update parse_translate_align text fixture after corrections * reformat ruff --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> resolve changing base issues ram ref nextclade change ref to same as V-Pipe fix: Insertions JSON format (#183) * change insertion strings * add validator to pydantic model * update front to end tests * Update src/sr2silo/silo_read_schema.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * satisfy ruff --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> reformat black nextclade test cannot work skip * remove nextclade add tests for insertions Update tests/conftest.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * format black * double code bug * accidental removal * accidental removal --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* feature: add paired read merging on alignments (#125) * copy over the script * update by manual copy * WIP * ignore .bai index files - should be freshly generated * add smallgenomutils conda package * add read pair merger * add sam * WIP * WIP: adding checks * move to temp * move to temp * move constants * Reset wokrflows * fix test * add proper test with check fo sorting * add test coverage * up test coverage * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * lower the test coverage * lower the test coverage --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * test: fix in CI for test coverage (#153) * test: test coverage in CI (#152) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * Update .github/workflows/test.yml praying this is correct. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fixing CI --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fixing CI * test: test coverage in CI (#152) (#154) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * Update .github/workflows/test.yml praying this is correct. --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix CI test coverage (#155) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * backtack: the CI issues are bad (#156) * fix CI test coverage * fix aciton * backtrack on code coverage --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * test: adjusting pytest coverage in CI (#159) * fix CI test coverage * fix aciton * backtrack on code coverage * fixing format of pipeline * fixing format of pipeline * fixing format of pipeline --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * feature: integrate paired-end read merging into CLI (#160) * adding sam_to_bam * add conversion tests * remove comments * specify sorting * rought workflow * add sorting by coordinate * WIP move aux functions to convert * working start-to-end with merging * black it * fix workflwotest * fix workflwotest * fix ruff * ruff ruff --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * history cleanup for dev --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fixing CI test: test coverage in CI (#152) (#154) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * Update .github/workflows/test.yml praying this is correct. --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fix CI test coverage (#155) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> backtack: the CI issues are bad (#156) * fix CI test coverage * fix aciton * backtrack on code coverage --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> test: adjusting pytest coverage in CI (#159) * fix CI test coverage * fix aciton * backtrack on code coverage * fixing format of pipeline * fixing format of pipeline * fixing format of pipeline --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * test: test coverage in CI (#152) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fix test to temp dir * declutter temp dir handling in main (#166) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * add sr2silo in namespace for workflow setup (#165) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * centralize workflow config (#167) * feature / fix: temporary directory handling for amino acid aligment (#179) * Introduces TMPDIR support in both main and translate_align modules * Refactors enrich_read_with_metadata into curry_read_with_metadata and updates related tests * Updates documentation and configuration files for resource requirements and CI environment handling --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * test: validate one/zero based coords (#181) * WUP * WIP tests * support for skipped char * add support for soft clipping * add support for hardclipping * adding Padding support * fix test read 02 * fix last read * move to script * note on coord system * fix black * cleanup test to tmp_path * remove trash * bam_to_fasta_query - support for soft clippings + validation * add test for create index and move test_bam_to_fasta_query to temp * update parse_translate_align text fixture after corrections * reformat ruff --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * fix: Insertions JSON format (#183) * change insertion strings * add validator to pydantic model * update front to end tests * Update src/sr2silo/silo_read_schema.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * satisfy ruff --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * refactor/feature: remove nextclade | add eqn to reads (#187) * layout orthogonal test * Salvage Nextclade Orth Test Changes get insertions form Nextclade get insertions form Nextclade get insertions form Nextclade WIP get nuc sequence WIP: parsesd nextclade to read objects remove trash WIP: failing tests Fix: indexing of bam black formating WIP - got the results to compare WIP all __eq__ adding Nuc comparison WIP making Nextclade use the reference from sr2silo WIP changed sr2silo ref to nextclade - still not workign add sorting by coordinate config feature: add paired read merging on alignments (#125) * copy over the script * update by manual copy * WIP * ignore .bai index files - should be freshly generated * add smallgenomutils conda package * add read pair merger * add sam * WIP * WIP: adding checks * move to temp * move to temp * move constants * Reset wokrflows * fix test * add proper test with check fo sorting * add test coverage * up test coverage * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * lower the test coverage * lower the test coverage --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> test: fix in CI for test coverage (#153) * test: test coverage in CI (#152) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * Update .github/workflows/test.yml praying this is correct. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fixing CI --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fixing CI test: test coverage in CI (#152) (#154) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * Update .github/workflows/test.yml praying this is correct. --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fix CI test coverage (#155) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> backtack: the CI issues are bad (#156) * fix CI test coverage * fix aciton * backtrack on code coverage --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> test: adjusting pytest coverage in CI (#159) * fix CI test coverage * fix aciton * backtrack on code coverage * fixing format of pipeline * fixing format of pipeline * fixing format of pipeline --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> feature: integrate paired-end read merging into CLI (#160) * adding sam_to_bam * add conversion tests * remove comments * specify sorting * rought workflow * add sorting by coordinate * WIP move aux functions to convert * working start-to-end with merging * black it * fix workflwotest * fix workflwotest * fix ruff * ruff ruff --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> test: test coverage in CI (#152) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * Update .github/workflows/test.yml praying this is correct. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> history cleanup for dev --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fixing CI test: test coverage in CI (#152) (#154) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * Update .github/workflows/test.yml praying this is correct. --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fix CI test coverage (#155) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> backtack: the CI issues are bad (#156) * fix CI test coverage * fix aciton * backtrack on code coverage --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> test: adjusting pytest coverage in CI (#159) * fix CI test coverage * fix aciton * backtrack on code coverage * fixing format of pipeline * fixing format of pipeline * fixing format of pipeline --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> test: test coverage in CI (#152) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fix test to temp dir declutter temp dir handling in main (#166) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> add sr2silo in namespace for workflow setup (#165) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> centralize workflow config (#167) feature / fix: temporary directory handling for amino acid aligment (#179) * Introduces TMPDIR support in both main and translate_align modules * Refactors enrich_read_with_metadata into curry_read_with_metadata and updates related tests * Updates documentation and configuration files for resource requirements and CI environment handling --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> test: validate one/zero based coords (#181) * WUP * WIP tests * support for skipped char * add support for soft clipping * add support for hardclipping * adding Padding support * fix test read 02 * fix last read * move to script * note on coord system * fix black * cleanup test to tmp_path * remove trash * bam_to_fasta_query - support for soft clippings + validation * add test for create index and move test_bam_to_fasta_query to temp * update parse_translate_align text fixture after corrections * reformat ruff --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> resolve changing base issues ram ref nextclade change ref to same as V-Pipe fix: Insertions JSON format (#183) * change insertion strings * add validator to pydantic model * update front to end tests * Update src/sr2silo/silo_read_schema.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * satisfy ruff --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> reformat black nextclade test cannot work skip * remove nextclade add tests for insertions Update tests/conftest.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * format black * double code bug * accidental removal * accidental removal --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update resources/sars-cov-2/README.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * remove notebooks * add option to skip paired-end read merging `skip_merge` and tests (#189) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * Add conda packaging and GitHub Actions workflows (#191) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * remove brnach protection (#188) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * Update version to 0.0.4 and prepare for conda packaging * Fix version handling in GitHub release workflow --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* feature: add paired read merging on alignments (#125) * copy over the script * update by manual copy * WIP * ignore .bai index files - should be freshly generated * add smallgenomutils conda package * add read pair merger * add sam * WIP * WIP: adding checks * move to temp * move to temp * move constants * Reset wokrflows * fix test * add proper test with check fo sorting * add test coverage * up test coverage * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * lower the test coverage * lower the test coverage --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * test: fix in CI for test coverage (#153) * test: test coverage in CI (#152) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * Update .github/workflows/test.yml praying this is correct. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fixing CI --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fixing CI * test: test coverage in CI (#152) (#154) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * Update .github/workflows/test.yml praying this is correct. --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix CI test coverage (#155) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * backtack: the CI issues are bad (#156) * fix CI test coverage * fix aciton * backtrack on code coverage --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * test: adjusting pytest coverage in CI (#159) * fix CI test coverage * fix aciton * backtrack on code coverage * fixing format of pipeline * fixing format of pipeline * fixing format of pipeline --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * feature: integrate paired-end read merging into CLI (#160) * adding sam_to_bam * add conversion tests * remove comments * specify sorting * rought workflow * add sorting by coordinate * WIP move aux functions to convert * working start-to-end with merging * black it * fix workflwotest * fix workflwotest * fix ruff * ruff ruff --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * history cleanup for dev --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fixing CI test: test coverage in CI (#152) (#154) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * Update .github/workflows/test.yml praying this is correct. --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fix CI test coverage (#155) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> backtack: the CI issues are bad (#156) * fix CI test coverage * fix aciton * backtrack on code coverage --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> test: adjusting pytest coverage in CI (#159) * fix CI test coverage * fix aciton * backtrack on code coverage * fixing format of pipeline * fixing format of pipeline * fixing format of pipeline --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * test: test coverage in CI (#152) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fix test to temp dir * declutter temp dir handling in main (#166) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * add sr2silo in namespace for workflow setup (#165) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * centralize workflow config (#167) * feature / fix: temporary directory handling for amino acid aligment (#179) * Introduces TMPDIR support in both main and translate_align modules * Refactors enrich_read_with_metadata into curry_read_with_metadata and updates related tests * Updates documentation and configuration files for resource requirements and CI environment handling --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * test: validate one/zero based coords (#181) * WUP * WIP tests * support for skipped char * add support for soft clipping * add support for hardclipping * adding Padding support * fix test read 02 * fix last read * move to script * note on coord system * fix black * cleanup test to tmp_path * remove trash * bam_to_fasta_query - support for soft clippings + validation * add test for create index and move test_bam_to_fasta_query to temp * update parse_translate_align text fixture after corrections * reformat ruff --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * fix: Insertions JSON format (#183) * change insertion strings * add validator to pydantic model * update front to end tests * Update src/sr2silo/silo_read_schema.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * satisfy ruff --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * refactor/feature: remove nextclade | add eqn to reads (#187) * layout orthogonal test * Salvage Nextclade Orth Test Changes get insertions form Nextclade get insertions form Nextclade get insertions form Nextclade WIP get nuc sequence WIP: parsesd nextclade to read objects remove trash WIP: failing tests Fix: indexing of bam black formating WIP - got the results to compare WIP all __eq__ adding Nuc comparison WIP making Nextclade use the reference from sr2silo WIP changed sr2silo ref to nextclade - still not workign add sorting by coordinate config feature: add paired read merging on alignments (#125) * copy over the script * update by manual copy * WIP * ignore .bai index files - should be freshly generated * add smallgenomutils conda package * add read pair merger * add sam * WIP * WIP: adding checks * move to temp * move to temp * move constants * Reset wokrflows * fix test * add proper test with check fo sorting * add test coverage * up test coverage * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * lower the test coverage * lower the test coverage --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> test: fix in CI for test coverage (#153) * test: test coverage in CI (#152) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * Update .github/workflows/test.yml praying this is correct. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fixing CI --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fixing CI test: test coverage in CI (#152) (#154) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * Update .github/workflows/test.yml praying this is correct. --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fix CI test coverage (#155) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> backtack: the CI issues are bad (#156) * fix CI test coverage * fix aciton * backtrack on code coverage --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> test: adjusting pytest coverage in CI (#159) * fix CI test coverage * fix aciton * backtrack on code coverage * fixing format of pipeline * fixing format of pipeline * fixing format of pipeline --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> feature: integrate paired-end read merging into CLI (#160) * adding sam_to_bam * add conversion tests * remove comments * specify sorting * rought workflow * add sorting by coordinate * WIP move aux functions to convert * working start-to-end with merging * black it * fix workflwotest * fix workflwotest * fix ruff * ruff ruff --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> test: test coverage in CI (#152) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * Update .github/workflows/test.yml praying this is correct. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> history cleanup for dev --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fixing CI test: test coverage in CI (#152) (#154) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge * Update .github/workflows/test.yml praying this is correct. --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fix CI test coverage (#155) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> backtack: the CI issues are bad (#156) * fix CI test coverage * fix aciton * backtrack on code coverage --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> test: adjusting pytest coverage in CI (#159) * fix CI test coverage * fix aciton * backtrack on code coverage * fixing format of pipeline * fixing format of pipeline * fixing format of pipeline --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> test: test coverage in CI (#152) * Update workflow to conditionally run coverage tests based on PR or push to main/dev branch * Add check for changed files before running coverage * remove template / badge Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fix test to temp dir declutter temp dir handling in main (#166) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> add sr2silo in namespace for workflow setup (#165) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> centralize workflow config (#167) feature / fix: temporary directory handling for amino acid aligment (#179) * Introduces TMPDIR support in both main and translate_align modules * Refactors enrich_read_with_metadata into curry_read_with_metadata and updates related tests * Updates documentation and configuration files for resource requirements and CI environment handling --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> test: validate one/zero based coords (#181) * WUP * WIP tests * support for skipped char * add support for soft clipping * add support for hardclipping * adding Padding support * fix test read 02 * fix last read * move to script * note on coord system * fix black * cleanup test to tmp_path * remove trash * bam_to_fasta_query - support for soft clippings + validation * add test for create index and move test_bam_to_fasta_query to temp * update parse_translate_align text fixture after corrections * reformat ruff --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> resolve changing base issues ram ref nextclade change ref to same as V-Pipe fix: Insertions JSON format (#183) * change insertion strings * add validator to pydantic model * update front to end tests * Update src/sr2silo/silo_read_schema.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * satisfy ruff --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> reformat black nextclade test cannot work skip * remove nextclade add tests for insertions Update tests/conftest.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * format black * double code bug * accidental removal * accidental removal --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update resources/sars-cov-2/README.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * remove notebooks * add option to skip paired-end read merging `skip_merge` and tests (#189) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * Add conda packaging and GitHub Actions workflows (#191) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * remove brnach protection (#188) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * Update version to 0.0.4 and prepare for conda packaging * Fix version handling in GitHub release workflow * GitHub Actions(deps): Bump actions/setup-python from 4 to 5 (#193) Bumps [actions/setup-python](https://github.com/actions/setup-python) from 4 to 5. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@v4...v5) --- updated-dependencies: - dependency-name: actions/setup-python dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * GitHub Actions(deps): Bump actions/checkout from 3 to 4 (#192) Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v3...v4) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * docs: specify usage of import-to-silo (#198) * validate silo insertions pydantic schema * add note to docs not to use untrimmed * add tests and allow N in insertiosn --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * fix: snakemake workflow / multiple batch ids clash (#202) * fix combination bug * fix multiple batch issue * fix config path * change path --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * fix: correct unknown / skipped char for amino acid padding (#200) * validate silo insertions pydantic schema * add note to docs not to use untrimmed * add tests and allow N in insertiosn * chang the Amio Acid Validation * changing unknown char for amino acid paddings * update expexte aligned reads fixture * fix test fixture aa insertions * Update src/sr2silo/silo_read_schema.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * update snakemake fixture --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * workflows: adding conda envs to rule (#194) * Adding conda rule support - (not tested) * remove redundant dependncies * adapt submission script --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * changing AA skipped chars in Readme (#205) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * update Readme with conda install (#206) Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> * feature : Add version tracking to 'import-to-loculus' metadata (#195) * WIP * make version info mandatory * adding version to all of test suits * update database config file * missing version * remove expected comparrions from smk rule * adjust test data * remove unused * generate index sr2silo_version * poetry dynamic versioning * adding transient dependency * install self-verisoning * adjust python version * remove brotli * adjsut * update fallback * simpligf --------- Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Gordon J. Köhn <gordon.koehn@dbsse.ethz.ch> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This PR integrates the
paired_end_merger
funcitonailtiy as a default into thesr2silo --import-to-loculus
command.