scGPT integration workflow #794

dorien-er · 2024-04-25T08:57:39Z

Changelog

Workflow for scGPT integration followed by Leiden clustering
Allow for passing fine-tuned models in embedding component

Issue ticket number and link

Checklist before requesting a review

… scgpt

scGPT integration preproc module

* add script to download scgpt test resources * Update resources_test_scripts/scgpt.sh Co-authored-by: Dries Schaumont <5946712+DriesSchaumont@users.noreply.github.com> * add drive folders containing data and model * chmod +x --------- Co-authored-by: Dries Schaumont <5946712+DriesSchaumont@users.noreply.github.com>

merge scgpt-dev into scgpt

CHANGELOG.md

Co-authored-by: Dries Schaumont <5946712+DriesSchaumont@users.noreply.github.com>

DriesSchaumont

I cannot seem to build the workflow:

$ viash ns build -q scgpt_leiden
Reading file 'src/workflows/integration/scgpt_leiden/config.vsh.yaml' failed
Not all configs built successfully
  131 configs were disabled
  1/1 configs encountered parse errors

CHANGELOG.md

src/scgpt/embedding/script.py

src/workflows/integration/scgpt_leiden/config.vsh.yaml

Address requested PR changes Co-authored-by: Dries Schaumont <5946712+DriesSchaumont@users.noreply.github.com>

dorien-er · 2024-08-15T06:47:03Z

PR requested changes are addressed :-)

rcannood

Minor remark in the code.

Could you post the output of the integration test? 🙇

src/scgpt/embedding/script.py

Co-authored-by: Robrecht Cannoodt <rcannood@gmail.com>

dorien-er · 2024-09-06T08:29:28Z

Minor remark in the code.

Could you post the output of the integration test? 🙇

Exporting scgpt_leiden (workflows/integration) =nextflow=> .../openpipeline/target/nextflow/workflows/integration/scgpt_leiden
Not all configs built successfully
  139 configs were disabled
  1/1 configs built successfully
Nextflow 24.04.4 is available - Please consider updating your version to it
N E X T F L O W  ~  version 21.10.6
Launching src/workflows/integration/scgpt_leiden/test.nf` [serene_chandrasekhar] - revision: cf3a3008c8
executor >  local (18)
[1c/78a5d3] process > test_wf:scgpt_leiden:run_wf:highly_variable_features_scanpy:processWf:highly_variable_features_scanpy_process (no_leiden_resolutions_test) [100%] 2 of 2 ✔
[ef/9cc287] process > test_wf:scgpt_leiden:run_wf:do_filter:processWf:do_filter_process (no_leiden_resolutions_test)                                             [100%] 2 of 2 ✔
[20/8ebf3f] process > test_wf:scgpt_leiden:run_wf:cross_check_genes:processWf:cross_check_genes_process (no_leiden_resolutions_test)                             [100%] 2 of 2 ✔
[51/fd44cc] process > test_wf:scgpt_leiden:run_wf:binning:processWf:binning_process (no_leiden_resolutions_test)                                                 [100%] 2 of 2 ✔
[39/36f2c6] process > test_wf:scgpt_leiden:run_wf:pad_tokenize:processWf:pad_tokenize_process (no_leiden_resolutions_test)                                       [100%] 2 of 2 ✔
[d4/37a03e] process > test_wf:scgpt_leiden:run_wf:embedding:processWf:embedding_process (no_leiden_resolutions_test)                                             [100%] 2 of 2 ✔
[3b/ff6912] process > test_wf:scgpt_leiden:run_wf:find_neighbors:processWf:find_neighbors_process (no_leiden_resolutions_test)                                   [100%] 2 of 2 ✔
[45/1a083f] process > test_wf:scgpt_leiden:run_wf:leiden:processWf:leiden_process (simple_execution_test)                                                        [100%] 1 of 1 ✔
[27/16ab50] process > test_wf:scgpt_leiden:run_wf:move_obsm_to_obs:processWf:move_obsm_to_obs_process (simple_execution_test)                                    [100%] 1 of 1 ✔
[07/48a516] process > test_wf:scgpt_leiden:run_wf:umap:processWf:umap_process (no_leiden_resolutions_test)                                                       [100%] 2 of 2 ✔
Output: [simple_execution_test, [output:.../openpipeline/work/15/827e8f2d834d905980ffaa5f0f3cbe/simple_execution_test.umap.output.h5mu]]
Output: [no_leiden_resolutions_test, [output:.../openpipeline/work/07/48a516c120091818b9f0bbba5b285b/no_leiden_resolutions_test.umap.output.h5mu]]
Completed at: 06-Sep-2024 10:23:35
Duration    : 5m 16s
CPU hours   : 0.2
Succeeded   : 18


Nextflow 24.04.4 is available - Please consider updating your version to it
N E X T F L O W  ~  version 21.10.6
Launching `src/workflows/integration/scgpt_leiden/test.nf` [modest_fermi] - revision: cf3a3008c8
executor >  local (10)
[7c/648a99] process > test_wf2:scgpt_leiden:run_wf:highly_variable_features_scanpy:processWf:highly_variable_features_scanpy_process (test_output_arg) [100%] 1 of 1 ✔
[46/b703df] process > test_wf2:scgpt_leiden:run_wf:do_filter:processWf:do_filter_process (test_output_arg)                                             [100%] 1 of 1 ✔
[23/d42f40] process > test_wf2:scgpt_leiden:run_wf:cross_check_genes:processWf:cross_check_genes_process (test_output_arg)                             [100%] 1 of 1 ✔
[d6/daf018] process > test_wf2:scgpt_leiden:run_wf:binning:processWf:binning_process (test_output_arg)                                                 [100%] 1 of 1 ✔
[d1/5cdffa] process > test_wf2:scgpt_leiden:run_wf:pad_tokenize:processWf:pad_tokenize_process (test_output_arg)                                       [100%] 1 of 1 ✔
[fd/1cfe12] process > test_wf2:scgpt_leiden:run_wf:embedding:processWf:embedding_process (test_output_arg)                                             [100%] 1 of 1 ✔
[ab/c8d4dc] process > test_wf2:scgpt_leiden:run_wf:find_neighbors:processWf:find_neighbors_process (test_output_arg)                                   [100%] 1 of 1 ✔
[6f/8e7535] process > test_wf2:scgpt_leiden:run_wf:leiden:processWf:leiden_process (test_output_arg)                                                   [100%] 1 of 1 ✔
[86/bbaec4] process > test_wf2:scgpt_leiden:run_wf:move_obsm_to_obs:processWf:move_obsm_to_obs_process (test_output_arg)                               [100%] 1 of 1 ✔
[09/d0b70c] process > test_wf2:scgpt_leiden:run_wf:umap:processWf:umap_process (test_output_arg)                                                       [100%] 1 of 1 ✔
Output: [test_output_arg, [output:.../openpipeline/work/09/d0b70c48b14bfd4bcfd6b023489b95/test.h5mu]]

Completed at: 06-Sep-2024 10:26:43
Duration    : 3m 1s
CPU hours   : 0.1
Succeeded   : 10

rcannood

LGTM!

dorien-er and others added 30 commits March 6, 2024 11:16

preproc script

78954c0

preproc script

9068e7a

tokenize and pad script

dbe5204

tokenize and pad script

89a9c6a

embedding script

9e446f8

test resourcers and evaluation script

94dd10c

cross check gene set

3edf3c0

pad_tokenize module

085cdc4

updat image

724427e

remove test resources, update inputs

f9aadfa

use pytorch image

33c9ffe

remove integration component

0c6316d

remove nvidia reqs

47f5dda

Merge branch 'main' of github.com:openpipelines-bio/openpipeline into…

9d2ffd0

… scgpt

remove load_model option

0f74ebd

Fix retag for viash-hub not using correct namespace separator (#745)

52fb38c

CI - Build: Fix second occurance of namespace separator (#746)

accf980

script to download scgpt test data

b1dd6ce

remove test resources script

18db6d6

adjust preprocessing script

6c3fec0

add scgpt full preproc module

acd3600

integration submodule

3e31204

integration submodule and add normalize_total flag

b5d1970

add params

ec326f8

Merge pull request #751 from openpipelines-bio/scgpt-preprocessor

2dddc1c

scGPT integration preproc module

embedding module

adcd6f0

Merge pull request #755 from openpipelines-bio/scgpt-dev

bd7a32f

merge scgpt-dev into scgpt

add unit tests

154ef26

undo subsampling test data

a7e08bc

dorien-er added 2 commits June 19, 2024 16:45

add scgpt resources script

d1a7d32

add scgpt resources script

731d019

DriesSchaumont reviewed Jun 21, 2024

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

dorien-er and others added 5 commits July 1, 2024 08:57

Update CHANGELOG.md

d3e862f

Co-authored-by: Dries Schaumont <5946712+DriesSchaumont@users.noreply.github.com>

allow for finetuned models

7ecdbb7

enable finetuned models in workflow

454a32e

add variable for integrated obsm

e84ff78

Merge remote-tracking branch 'origin/main' into scgpt-integration

3f25c3b

DriesSchaumont requested changes Aug 2, 2024

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

src/scgpt/embedding/script.py Outdated Show resolved Hide resolved

src/workflows/integration/scgpt_leiden/config.vsh.yaml Outdated Show resolved Hide resolved

dorien-er and others added 3 commits August 14, 2024 16:13

update authorship, fix parsing error

cd7410e

remove compression

9f06e34

Apply suggestions from code review

21bcbf7

Address requested PR changes Co-authored-by: Dries Schaumont <5946712+DriesSchaumont@users.noreply.github.com>

dorien-er and others added 3 commits August 15, 2024 08:47

Merge branch 'main' into scgpt-integration

0742316

fix unit test

9d0225e

fix unit test

1a40003

dorien-er requested a review from DriesSchaumont August 15, 2024 09:03

DriesSchaumont and others added 4 commits August 23, 2024 14:53

Merge remote-tracking branch 'origin/main' into scgpt-integration

1cbe506

Merge branch 'main' into scgpt-integration

8be4858

update to viash 9

c9fc39d

fix test

c2b11a2

rcannood reviewed Sep 4, 2024

View reviewed changes

src/scgpt/embedding/script.py Outdated Show resolved Hide resolved

dorien-er and others added 3 commits September 4, 2024 08:53

Update src/scgpt/embedding/script.py

9871631

Co-authored-by: Robrecht Cannoodt <rcannood@gmail.com>

update integration_test.sh

c194078

update integration test

72df593

dorien-er requested a review from rcannood September 6, 2024 09:00

rcannood approved these changes Sep 6, 2024

View reviewed changes

dorien-er merged commit ae0e1d6 into main Sep 6, 2024
5 checks passed

dorien-er deleted the scgpt-integration branch September 6, 2024 12:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

scGPT integration workflow #794

scGPT integration workflow #794

Uh oh!

dorien-er commented Apr 25, 2024 •

edited

Loading

Uh oh!

Uh oh!

DriesSchaumont left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dorien-er commented Aug 15, 2024

Uh oh!

rcannood left a comment

Uh oh!

Uh oh!

dorien-er commented Sep 6, 2024 •

edited

Loading

Uh oh!

rcannood left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

scGPT integration workflow #794

scGPT integration workflow #794

Uh oh!

Conversation

dorien-er commented Apr 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog

Issue ticket number and link

Checklist before requesting a review

Uh oh!

Uh oh!

DriesSchaumont left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dorien-er commented Aug 15, 2024

Uh oh!

rcannood left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dorien-er commented Sep 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rcannood left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dorien-er commented Apr 25, 2024 •

edited

Loading

dorien-er commented Sep 6, 2024 •

edited

Loading