Skip to content
This repository was archived by the owner on Apr 28, 2022. It is now read-only.

MTS-merino pipeline code review #2

Draft
wants to merge 1,005 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1005 commits
Select commit Hold shift + click to select a range
10e61cf
Triggering pipeline docker dev on Jenkins
maha-naim Jun 13, 2023
8b86302
added comments
ashishge1 Jun 13, 2023
bdbe77a
Merge branch 'robust_DRC' into dev
ashishge1 Jun 15, 2023
b5e984c
updated sign of slope in drc fit to match downstream plots
ashishge1 Jun 20, 2023
0de9118
updated drc fit documentation
ashishge1 Jun 22, 2023
7f23fb7
Changes fit constraints based on type of screen
ashishge1 Jun 28, 2023
0d03578
updated functions for readability and performance
ashishge1 Jul 5, 2023
b456b0b
fixed typo
ashishge1 Jul 6, 2023
e43eedc
Adding screen type argument
maha-naim Jul 13, 2023
23b59af
Adding shortcut notation for screen type argument
maha-naim Jul 13, 2023
44cfb1d
Removing extra 't' from '-st' argument
maha-naim Jul 13, 2023
83dccbb
Adjusting screen type argument notation
maha-naim Jul 14, 2023
2404a27
Added Riemann AUC and robustness measures
ashishge1 Jul 18, 2023
f6e1688
removed hard-coded column indexing
ashishge1 Jul 18, 2023
20abfd4
update to display NA rows
ashishge1 Jul 18, 2023
83634fc
update table to display AUC =NA
ashishge1 Jul 18, 2023
fca014f
drop rows without fits
ashishge1 Jul 18, 2023
6ca43cc
removed harcoded column indices
ashishge1 Jul 18, 2023
4301983
fixed bug in subsetting rows
ashishge1 Jul 19, 2023
a31288b
Removed rows with dose=NA and resolved ties in fit
ashishge1 Jul 20, 2023
c3520fd
fixed case_when and drop_na errors
ashishge1 Jul 26, 2023
3a8c147
bug fix
ashishge1 Jul 27, 2023
95fac1b
added messages for debugging
ashishge1 Jul 27, 2023
10eac26
Update compound_report.Rmd
ashishge1 Jul 31, 2023
cce0154
Converting pert_time to character in DRC and LFC
maha-naim Jul 31, 2023
4f24ef9
updated convergence flag
ashishge1 Aug 1, 2023
6e1e45e
added arguments to script
AnupJonchhe Aug 1, 2023
2d5841f
Commenting to debug SEQ-005 in MTS
maha-naim Aug 3, 2023
382179b
renamed files
AnupJonchhe Aug 3, 2023
31f3a7c
Passing screen type argument if not none
maha-naim Aug 3, 2023
c5e3f76
fixed directory structure
AnupJonchhe Aug 3, 2023
f9306ff
dockerized compound summary script
AnupJonchhe Aug 3, 2023
13caf32
Merge branch 'compound_metrics' into dev
AnupJonchhe Aug 3, 2023
bc1da5b
Merge branch 'dev' of github.com:cmap/dockerized_mts into dev
AnupJonchhe Aug 3, 2023
4a713c0
added README.md
AnupJonchhe Aug 3, 2023
94a1442
fixed clashing short flag for combinations
AnupJonchhe Aug 7, 2023
4e34cf4
Merge branch 'master' into dev
AnupJonchhe Aug 7, 2023
da0ae36
removed hardcoded reference
AnupJonchhe Aug 7, 2023
bf5ed7b
changed name of output file
AnupJonchhe Aug 7, 2023
928420b
fixed from mkocak
AnupJonchhe Aug 8, 2023
a5b84cd
Adding flag for levels
maha-naim Aug 8, 2023
9200e1a
updated logo, slight resizing
AnupJonchhe Aug 11, 2023
d9eb300
Merge branch 'change_logo'
AnupJonchhe Aug 11, 2023
ec2548d
Merge branch 'change_logo' into dev
AnupJonchhe Aug 11, 2023
f0679c1
Create README.md
shiker1738 Aug 18, 2023
7210bb9
Add files via upload
shiker1738 Aug 18, 2023
65fd075
init of filter wells module
AnupJonchhe Aug 21, 2023
5460125
adding CSL search_patterns
AnupJonchhe Aug 21, 2023
fe635ea
adding file_prefix
AnupJonchhe Aug 24, 2023
a6fb9a7
Merge branch 'collate_build' into dev
AnupJonchhe Aug 24, 2023
fc35b58
working filter_skipped_wells module
AnupJonchhe Aug 25, 2023
e5b3045
Merge branch 'filter_skipped_wells' into dev
AnupJonchhe Aug 25, 2023
be2ea81
Update landing_page.Rmd
shiker1738 Aug 30, 2023
4501176
Update drc_report.Rmd
shiker1738 Aug 30, 2023
020af36
Merge branch 'unify_combination_solo' into dev
AnupJonchhe Sep 7, 2023
b66baa6
Merge pull request #37 from cmap/dev
AnupJonchhe Sep 7, 2023
7065c78
Initial todos
jdavis3141 Sep 7, 2023
e02eaf4
build docker
AnupJonchhe Sep 7, 2023
0213193
patch for prep_portal_data
NoopDawg Sep 8, 2023
4f59574
Merge branch 'prep_portal_data_fix'
NoopDawg Sep 8, 2023
bdd3bc7
merging drc text change into dev
NoopDawg Sep 8, 2023
fe07d69
Including count in level 3 table
jdavis3141 Sep 12, 2023
2a4ecd7
Merge pull request #39 from cmap/metadata_change
jdavis3141 Sep 12, 2023
4aa7da7
added old drc function in to compare results
ashishge1 Sep 18, 2023
01e2a4a
Merge branch 'dev' of https://github.com/cmap/dockerized_mts into dev
ashishge1 Sep 18, 2023
0df0bef
bigquery check table columns
AnupJonchhe Sep 21, 2023
f75f91f
renamed df to lineage_df
AnupJonchhe Sep 21, 2023
8003b76
added code to debug issues with drc
ashishge1 Sep 21, 2023
234f313
output lineage_enrichment_boxplot_v2 table
AnupJonchhe Sep 21, 2023
e71f3b9
catch NA results in fits to prevent failure
ashishge1 Sep 21, 2023
ae7074d
pushing testing
AnupJonchhe Sep 21, 2023
2ee5c32
Update drc_functions.R
ashishge1 Sep 21, 2023
6cf4485
Merge branch 'unify_combination_solo' into dev
AnupJonchhe Sep 22, 2023
c6305e4
Merge branch 'dev' of github.com:cmap/dockerized_mts into dev
AnupJonchhe Sep 22, 2023
c16853f
Check slope bound in result
ashishge1 Sep 22, 2023
aedcbbf
changed UL _bound
ashishge1 Sep 22, 2023
761f345
fixed merge conflict
AnupJonchhe Sep 25, 2023
3b25366
finds screen type based on number of compounds
ashishge1 Sep 25, 2023
adae436
check for second pert in CPS
ashishge1 Sep 25, 2023
4000b98
handle na fit_results.
ashishge1 Sep 25, 2023
61d7d2b
Adding flexibility to deal to only expect files that exist and changi…
maha-naim Sep 26, 2023
64cce5d
Merge conflict with landing page changes
maha-naim Sep 26, 2023
ddc2890
check columns in biqquery before outputting file
AnupJonchhe Sep 26, 2023
5e49d71
fix DRC case
AnupJonchhe Sep 27, 2023
78d8ffa
adding case insensitivity to subset columns
AnupJonchhe Sep 27, 2023
b3c4614
removed old combo_solo file
AnupJonchhe Sep 27, 2023
b5632ed
Merge branch 'dev' of github.com:cmap/dockerized_mts into dev
AnupJonchhe Sep 27, 2023
e05d27e
Merge branch 'prep_portal_check_bq_cols' into dev
AnupJonchhe Sep 27, 2023
0374a95
outputting files from combination report
AnupJonchhe Sep 27, 2023
c0b3d05
detects CPS automatically
ashishge1 Sep 27, 2023
e3e252e
Merge branch 'dev' of https://github.com/cmap/dockerized_mts into dev
ashishge1 Sep 27, 2023
4f2fb14
Robust fit and manual initialization
ashishge1 Sep 27, 2023
05ec74d
adding neg_log10_q calculation to synergy table
AnupJonchhe Oct 2, 2023
a5cf639
Merge branch 'dev' of github.com:cmap/dockerized_mts into dev
AnupJonchhe Oct 2, 2023
f439971
exceptions to filenames equalling BQ table name
AnupJonchhe Oct 10, 2023
2a449ad
Update biomarkers.R
shiker1738 Oct 10, 2023
8b509f0
if no curve is fit fill with placeholder -666 for upload into Data Wa…
AnupJonchhe Oct 10, 2023
14da5cd
Updated Column name and comments
ashishge1 Oct 11, 2023
bf1ae25
Merge branch 'dev' of https://github.com/cmap/dockerized_mts into dev
ashishge1 Oct 11, 2023
c34cd86
combo_rf_models refactors to separate out_df object, fixing column issue
AnupJonchhe Oct 12, 2023
f01b156
added changelog
AnupJonchhe Oct 12, 2023
a630384
added changelog
AnupJonchhe Oct 12, 2023
3c3a66c
Update changelog.md
AnupJonchhe Oct 12, 2023
ddee303
Update changelog.md with DRC description
ashishge1 Oct 13, 2023
6a803c1
filter wells comment
AnupJonchhe Oct 16, 2023
79d4ebf
Update compound_report.Rmd
shiker1738 Oct 16, 2023
d6e6ec5
Update combination_solo.Rmd
shiker1738 Oct 16, 2023
52df2dc
Update biomarkers.R
shiker1738 Oct 17, 2023
a2a957e
Update changelog.md
AnupJonchhe Oct 18, 2023
a0cda7f
Update changelog.md
shiker1738 Oct 18, 2023
0517a6d
script to summarize differences between dev and master branches
AnupJonchhe Oct 18, 2023
1d42d8f
Merge branch 'dev' of github.com:cmap/dockerized_mts into dev
AnupJonchhe Oct 18, 2023
0fd987e
confirmed removal of combination_solo
AnupJonchhe Oct 18, 2023
2289c77
deleted old file
AnupJonchhe Oct 25, 2023
07bc013
conda creation removal, env already created in base image
AnupJonchhe Oct 31, 2023
30d9558
filter cell_set_definition view by beadset
AnupJonchhe Oct 31, 2023
c6db199
changed split to take search pattern
AnupJonchhe Nov 2, 2023
2eb3d92
Adjusting keys passed for extended builds during stack
maha-naim Nov 2, 2023
968e504
Adding LFC columns as arguments to avoid dependency on combat correct…
maha-naim Nov 3, 2023
7ec7b21
Update aws_batch.sh - Commenting out -s argument
maha-naim Nov 7, 2023
382f89c
Removing screen type argument being passed into DRC, added LFC file a…
maha-naim Nov 9, 2023
bab0ae2
Merge pull request #41 from cmap/dev
jdavis3141 Nov 9, 2023
568cc8f
add 1 to raw matrix to prevent log(0)
jdavis3141 Nov 9, 2023
c459723
Adding ignore missing flag
maha-naim Nov 9, 2023
e715b57
Removing comments
maha-naim Nov 9, 2023
9f3c4a5
Update changelog.md
jdavis3141 Nov 9, 2023
3c1e543
Adding normalized counts recognition in stack, and search pattern arg…
maha-naim Nov 9, 2023
4022420
Fixed condition where skipped wells df is empty
jdavis3141 Nov 9, 2023
75af364
Add ability to filter by instance_ids
jdavis3141 Nov 9, 2023
00c0748
Fixed typo
jdavis3141 Nov 9, 2023
6fb383e
Detecting LFC_cb, otherwise setting to LFC in reports.
maha-naim Nov 9, 2023
5e26a06
Get inst/bcids to skip from API
jdavis3141 Nov 14, 2023
7fc55d4
removed unnecessary args
jdavis3141 Nov 14, 2023
835b59f
Get api_url from env
jdavis3141 Nov 15, 2023
674ef59
Fix request URL in normalize
jdavis3141 Nov 15, 2023
aba836c
Fix request_url bug
jdavis3141 Nov 15, 2023
87e6b80
Fixing typo in filter_skipped_wells
jdavis3141 Nov 15, 2023
c4190d1
create eps-prep module that removes ic50 references from downloads
AnupJonchhe Nov 16, 2023
8eede1b
Added arg to query API
jdavis3141 Nov 17, 2023
6f82737
Merge remote-tracking branch 'origin/dev' into dev
jdavis3141 Nov 17, 2023
4084765
protected from failure when re-running builds
AnupJonchhe Nov 21, 2023
108b1d9
Merge branch 'dev' of github.com:cmap/dockerized_mts into dev
AnupJonchhe Nov 21, 2023
adec142
changelog
AnupJonchhe Nov 21, 2023
2342596
commenting out GE_noLin
shiker1738 Nov 21, 2023
e062737
uncomment removal of ic50
AnupJonchhe Nov 21, 2023
55177ac
Merge branch 'dev' of github.com:cmap/dockerized_mts into dev
AnupJonchhe Nov 21, 2023
de926c6
Fix removed inst req, check for dup analytes
jdavis3141 Nov 29, 2023
378beaf
Added logging for analyte dup error
jdavis3141 Nov 29, 2023
a448e11
Show all duplicated analytes
jdavis3141 Nov 29, 2023
a83514b
Revert "Show all duplicated analytes"
jdavis3141 Nov 29, 2023
fc0e597
Fixing dup analytes
jdavis3141 Nov 29, 2023
b92e0e8
fixed typo
jdavis3141 Nov 29, 2023
0255662
Fail if requested remove instance does not exist
jdavis3141 Nov 29, 2023
c303a09
Fixed bcid removal in normalize
jdavis3141 Nov 30, 2023
cf7469e
Merge pull request #42 from cmap/dev
jdavis3141 Nov 30, 2023
1bc45b6
Set api_url in normalization
jdavis3141 Nov 30, 2023
25a4703
Remove instances/well based on oncref thresholds
jdavis3141 Dec 6, 2023
023b59a
Merge pull request #43 from cmap/dev
jdavis3141 Dec 7, 2023
a664b7c
Group samples by pert_vehicle for qc, lfc, norm
jdavis3141 Dec 7, 2023
dd55695
Removing S3 specific lineage pathway
shiker1738 Dec 9, 2023
94804f3
Removing S3 specific lineages pathway
shiker1738 Dec 9, 2023
80f365b
protecting from space string characters, adding verbosity flag
AnupJonchhe Dec 15, 2023
c62d11f
protecting from space string characters, adding verbosity flag
AnupJonchhe Dec 15, 2023
a80ae6a
updated pipeline documentation
shiker1738 Jan 4, 2024
cd99171
Add files via upload
shiker1738 Jan 4, 2024
9f994c0
Update README.md
shiker1738 Jan 4, 2024
76dc90e
Add files via upload
shiker1738 Jan 4, 2024
1b4bf9e
Add files via upload
shiker1738 Jan 4, 2024
611bcfa
Update README.md
shiker1738 Jan 4, 2024
b9fcee3
Creating a new EPS_QC_TABLE key specific to EPS builds. Will build of…
maha-naim Jan 10, 2024
a385aeb
adding new module to flexibly remove lines of files, for use with EPS
AnupJonchhe Jan 18, 2024
6910d4c
cleaned up data a little
AnupJonchhe Jan 18, 2024
ce41395
Merge pull request #45 from cmap/mts_documentation
AnupJonchhe Jan 18, 2024
6b7e88f
Merge pull request #44 from cmap/dev
AnupJonchhe Jan 18, 2024
a4cbeff
fix issue with EPS_QC_TABLE tables being stripped
AnupJonchhe Jan 18, 2024
d3328b7
Merge branch 'dev' of github.com:cmap/dockerized_mts into dev
AnupJonchhe Jan 18, 2024
e15a0c9
Merge pull request #47 from cmap/feature-remove-data-module
AnupJonchhe Jan 18, 2024
6742a05
fix bug and prep for AWS
AnupJonchhe Jan 18, 2024
6049c4d
update to MTS024 changelog documentation
shiker1738 Jan 23, 2024
f55b7c9
Merge pull request #49 from cmap/hotfix2_changelog
shiker1738 Jan 23, 2024
06f59ec
Merge pull request #48 from cmap/hotfix2_changelog
shiker1738 Jan 23, 2024
ac69a66
Update changelog.md
ashishge1 Jan 23, 2024
ddbdc97
removing index in QC_table
AnupJonchhe Feb 1, 2024
41184cd
changelog update
AnupJonchhe Feb 1, 2024
e691946
Resolving merge conflicts
maha-naim Feb 1, 2024
1fe242c
Merge pull request #46 from cmap/ext_test
AnupJonchhe Feb 2, 2024
918bf2e
Fixing filter_lowcounts bug
jdavis3141 Feb 14, 2024
38bb218
Update normalization_functions.R
jdavis3141 Apr 18, 2024
8751a85
Merge pull request #50 from cmap/jdavis3141-patch-countfilter
jdavis3141 Apr 18, 2024
b61fe0b
adding floor_range metric calculation to QC table and removing failur…
jdavis3141 Apr 22, 2024
6155dc8
Fixed merge and columns
jdavis3141 Apr 22, 2024
1a9f262
1. Add todo -> accept list of analyte_ids instead of specifying 6
jdavis3141 Apr 23, 2024
b8c51bb
mutation boxplot significance filtering
shiker1738 Apr 23, 2024
3eaeeab
reverting screen info commenting
shiker1738 Apr 24, 2024
c3a8082
Merge pull request #51 from cmap/new_qc_metric
jdavis3141 Apr 26, 2024
4d4b26f
deprecating extract biomarkers module from build process"
AnupJonchhe Apr 29, 2024
c1bdbf0
adding debug message"
AnupJonchhe Apr 29, 2024
6e21933
pinning r docker with version
AnupJonchhe Apr 30, 2024
f45eb71
merge
AnupJonchhe Apr 30, 2024
7542a89
Adding code to remove poorly correlated replicates at the pool+well l…
jdavis3141 Apr 30, 2024
35793f8
Chnage pass_rc logic and add flag to run poor replicate correlation r…
jdavis3141 Apr 30, 2024
a4d9755
Merge pull request #52 from cmap/master
jdavis3141 Apr 30, 2024
b6f7ed0
fix invalid version issue
AnupJonchhe Apr 30, 2024
8ea237c
Merge branch 'master' into dev
AnupJonchhe Apr 30, 2024
fecebe6
Initial commit on new delta_replicate_corr_ctls branch
jdavis3141 May 3, 2024
9c36b96
Include controls in poor correlation data removal
jdavis3141 May 3, 2024
eace55b
Merge pull request #53 from cmap/master
jdavis3141 May 3, 2024
519a283
Merge pull request #54 from cmap/delta_replicate_corr_ctls
jdavis3141 May 3, 2024
81d4db4
Update changelog.md
jdavis3141 May 3, 2024
46df4d4
Merge pull request #56 from cmap/mut_boxplot
maha-naim May 6, 2024
e54db10
adding pert_plate to bliss and synergy tables for data download
shiker1738 May 10, 2024
a33e39e
adding LEVEL3 to split
AnupJonchhe May 13, 2024
f211484
minor improvements for local runnability
AnupJonchhe May 20, 2024
911a064
Merge branch 'dev' into master
AnupJonchhe May 20, 2024
8b1e1bf
Merge pull request #57 from cmap/master
AnupJonchhe May 20, 2024
124d011
biomarkers to point to consistent location on s3
AnupJonchhe May 21, 2024
f756a7c
chore: changelog
AnupJonchhe May 21, 2024
7d2d28b
Delete PRISM MTS data processing.pdf
ashishge1 May 21, 2024
e414889
Merge branch 'dev' into cps_export
shiker1738 May 21, 2024
25f2fa9
qc fail all replicates if 2/3 fail change
shiker1738 May 21, 2024
7fa97d7
removed out of date documentation
ashishge1 May 21, 2024
2fa92d3
updating s3 paths for annotations and features for local running
shiker1738 May 23, 2024
4ac86e7
Merge pull request #58 from cmap/s3_path_changes
shiker1738 May 23, 2024
28bcc55
Update changelog.md
ashishge1 May 24, 2024
bee2237
deactivate build process for remove-data (not used)
AnupJonchhe May 24, 2024
db738e3
fixing remove-data module, used in EPS pipeline
AnupJonchhe May 24, 2024
085b620
Merge branch 'dev' of github.com:cmap/dockerized_mts into dev
AnupJonchhe May 24, 2024
0d232a6
Merge pull request #59 from cmap/master
jdavis3141 May 29, 2024
d768332
Merge pull request #60 from cmap/cps_export
jdavis3141 Jun 3, 2024
eb685b5
Update README.md
ashishge1 Jun 3, 2024
32fc9c6
updating default biomarker dir to static location
AnupJonchhe Jun 18, 2024
e9027a3
merging to master
AnupJonchhe Jun 18, 2024
f7122be
improvements to local running
AnupJonchhe Jun 18, 2024
7594eae
fix path to metadata
AnupJonchhe Jun 18, 2024
dc8df25
Merge branch 'dev'
AnupJonchhe Jun 18, 2024
bcfc8fd
fixing path of annotations files
maha-naim Jun 21, 2024
f4f0a32
combination drc to output viability data like single drc report does
AnupJonchhe Jun 24, 2024
53e2db1
protecting from trailing slash
NoopDawg Jun 25, 2024
74e4d24
fix: can't update params variables, TIL
AnupJonchhe Jun 25, 2024
1e0a102
Merge branch 'master' into dev
AnupJonchhe Jun 25, 2024
a590796
fixing biomarker paths
AnupJonchhe Jun 25, 2024
1126703
Merge pull request #61 from cmap/dev
AnupJonchhe Jun 25, 2024
5f2af26
feat: merging files in a stream processed to reduce memory requirements
AnupJonchhe Jun 26, 2024
4f7ca7d
Merge pull request #62 from cmap/low_memory_merge_csvs
AnupJonchhe Jun 27, 2024
36f5c04
Merge pull request #63 from cmap/dev
AnupJonchhe Jul 8, 2024
1000c58
only using LFC column if LFC_cb is not run
AnupJonchhe Jul 26, 2024
2caf744
Merge branch 'hot-fix-LFC-cb-field'
AnupJonchhe Jul 26, 2024
e071499
fix: bug causing missing LFC_cb not to be caught
AnupJonchhe Aug 20, 2024
e4f3dac
fixing logical mutation boxplot error
AnupJonchhe Sep 24, 2024
373d020
simplifying split
AnupJonchhe Sep 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 34 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
#Python compiled
*.pyc

# History files
.Rhistory
.Rapp.history
Expand All @@ -13,11 +16,12 @@

# Output files from R CMD build
/*.tar.gz

.idea/
# Output files from R CMD check
/*.Rcheck/

# RStudio files
**.Rproj
.Rproj.user/

# produced vignettes
Expand All @@ -40,3 +44,32 @@ vignettes/*.pdf

# pkgdown site
docs/
.idea/
.ipynb_checkpoints/
foo.json
foo.sh
enter_container.sh
stack/enter_stack.sh
stack/run*.sh
extract-biomarkers/docker_run_*.sh

# report files
reports/rmarkdown/*.html
screen_qc/rmarkdown/*.html
landing_pages/rmarkdown/*.html
node_modules
__pycache__/

# biomarker feature files
biomarker_files/*.csv
biomarker_data/*.csv

#debug notebooks
#*.ipynb

#scratch folder
scratch/
wkdir/
wkdir/*

.DS_Store
22 changes: 0 additions & 22 deletions Dockerfile

This file was deleted.

Loading