- 📖 Methylation Pipline Overview
- ⚡️ Quickstart
- Input Paths
- Default Working Directory
- Output Paths - ⚙️ Executing Methylation CLI
- 🧪 Run the Test Case
⚠️ Troubleshooting
Download and install the following packages:
- R 4.1 or higher from CRAN: https://cran.r-project.org/bin/macosx/
- Gfortran from GitHub use dmg installer: https://github.com/fxcoudert/gfortran-for-macOS/releases/tag/11.2-bigsur-intel
- RStudio 1.4 or later: https://www.rstudio.com/products/rstudio/download/#download
- XQuartz: https://www.xquartz.org/
- LaTeX for Mac: https://www.tug.org/mactex/mactex-download.html
- Pandoc: https://pandoc.org/installing.html
- XCode command line tools for Mac OS:
xcode-select --install - Java 8 JDK: https://www.oracle.com/java/technologies/downloads/#java8-mac
- Rswitch: https://rud.is/rswitch/
- Homebrew: https://brew.sh/ you can install using the following line in terminal:
/bin/bash -c $(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh - Library Magic, Sqlite and Proj:
brew install libmagic sqlite proj - Compilers and Unpackers:
brew install llvm aspell gdal autoconf automake gcc libgit2 openssl@3 zlib go pandoc - Additional Libraries:
brew install texinfo pango cairo open-mpi poppler-qt5 graphviz libopenmpt java11 - Additional Library for M1 Mac:
brew install libffi - Additional OpenGL:
brew install --from-source glfw3
brew install cmake
brew uninstall glfw
git clone https://github.com/glfw/glfw.git && \
cd glfw && \
cmake -DCMAKE_OSX_ARCHITECTURES=arm64 . && \
make && \
sudo make install
NOTE
- R v4.1 includes compile and Tckl dependencies. brew can install libomp and cairo if needed. - After downloading R & RStudio unlocked the [System Preferences Privacy & Security Panel](https://github.com/NYU-Molecular-Pathology/Methylation/blob/main/Notes/SystemPermissions.md) before installing packages.- To install & run the pipeline, it is critical to mount the following network smb shared drives:
- Open Finder and press ⌘(CMD) + K then paste each of the directories below, using NYUMC\KerberosID as the login name and password is your kerberos password.
cifs://research-cifs.nyumc.org/Research/CBioinformatics/
cifs://research-cifs.nyumc.org/Research/snudem01lab/snudem01labspace
cifs://shares-cifs.nyumc.org/apps/acc_pathology/molecular
- Download the shell script to your home folder or another directory:
- You can download runMeth.sh in this repo under Methylation/Meth_Scripts/ or use curl/wget:
curl -# -L https://raw.githubusercontent.com/NYU-Molecular-Pathology/Methylation/main/Meth_Scripts/runMeth.sh >$HOME/runMeth.sh
- Open the shell script, paste your REDCap API token in the methAPI field, and save it.
You can usenano $HOME/runMeth.sh
#!/bin/bash
methAPI="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" #Paste your API Token here- Add permissions to the script to be executable:
chmod +rwx $HOME/runMeth.shTo install the pipeline from your terminal, simply execute the following command:
$HOME/runMeth.sh 21-MGDM_TEST- If install of any packages fail, be sure to check the troubleshooting section at the bottom of this page
Files are copied to the work directory by their RUNID name and YEAR, including the worksheet and idats for example:
- Worksheets
/Volumes/molecular/MOLECULAR LAB ONLY/NYU-METHYLATION/WORKSHEETS/2022/22-MGDM17.xlsm - .idat files
/Volumes/molecular/Molecular/iScan/
- Input files are copied and report files are generated on the Cbioinformatics drive:
/Volumes/CBioinformatics/Methylation/Clinical_Runs/22-MGDM17
- Html report files saved to the working directory are copied to the Z-drive
For example, run 22-MGDM17 report files would be output in the following directories:
/Volumes/molecular/Molecular/MethylationClassifier/2022/22-MGDM17
/Volumes/molecular/MOLECULAR LAB ONLY/NYU-METHYLATION/Results/2022/22-MGDM17
To run the Clinical or Research Methylation pipeline, simply use the locally stored Shell Script in:
/Volumes/CBioinformatics/Methylation/runMeth.sh
- This shell script uses Curl to download the files from this repo and takes four positional argument inputs to execute methylExpress.R in the terminal.
- The bash script stores your REDCap API token locally and only requires the methylation run ID to be entered.
- You can copy runMeth.sh and create an alias or symlink to execute more easily. For example:
alias runmeth='bash $HOME/runMeth.sh'orecho "alias runmeth='bash $HOME/runMeth.sh'" >> ~/.bashrc
The shell script takes the following positional arguments:
methAPI='XXXXXXXX' # (hardcoded) Your REDCap API Token
methRun=${1-NULL} # (required argument) methylation run id e.g. 22-MGDM17 @Default is NULL
PRIORITY=${2-NULL} # (optional) string list of prioritized RD-numbers to run
runPath=${3-NULL} # (optional) any directory to copy/run the idat files if not Clinical_Runs
redcapUp=${4-NULL} # (optional) character "T" or "F": Flag to upload REDCap data or notFirst, runmeth.sh downloads methylExpress.R and other files using curl:
curl -o methylExpress.R -L https://raw.githubusercontent.com/NYU-Molecular-Pathology/Methylation/main/R/methylExpress.RNext, runmeth.sh passes the values input as args to methylExpress.R:
Rscript --verbose methylExpress.R $methAPI $methRun $PRIORITY $runPath
The four positional arguments from runmeth.sh are passed to the Rscript methylExpress.R:
arg[1] is the token for the API call ('#######################')
arg[2] is the RunID which if NULL runs the latest Clinical Worksheet 22-MGDM17
arg[3] is the selectRds parameter which is to prioritize samples being run (NULL)
arg[4] is the baseFolder parameter which is optional if you want to run/save output to a different directory (NULL)
Alternatively, instead of passing the RunID to runmeth.sh, you can source and download this repository and then locally edit args in methylExpress.R to run manually.
After installation, you can use the following run command to test the pipeline.
/Volumes/CBioinformatics/Methylation/runMeth.sh 21-MGDM_TESTPipeline Installation Issues
If you have issues with package installation or dependencies:Make sure compilers are installed by opening Xcode.app or executing `sudo xcode-select --install`
Then, execute the all_installer.R script by copy and pasting the raw contents of the script below into Rstudio before running runmeth.sh again
https://raw.githubusercontent.com/NYU-Molecular-Pathology/Methylation/main/Research/all_installer.R
To resolve any problems during automation, you can open methylExpress.R in RStudio which is downladed by runmeth.sh to your home directory.
Try to run `sudo xcode-select -s /Library/Developer/CommandLineTools` and `brew install gdal proj` then install the package **rgdal** in Rstudio.
Download the libraries below from their sources:
(a) sqlite-autoconf-3330000.tar.gz from "https://www.sqlite.org/download.html".
(b) tiff-4.1.0.tar.gz from "https://download.osgeo.org/libtiff/"
(c) proj-7.2.0.tar.gz from "https://proj.org/download.html#current-release"
(d) libgeotiff-1.6.0.tar.gz from "https://download.osgeo.org/geotiff/libgeotiff/"
(e) geos-3.8.1.tar.bz2 from "https://trac.osgeo.org/geos"
(f) gdal-3.2.0.tar.gz from "https://gdal.org/download.html"
REDCap errors
Once your run completes check in your run directory if there is any *upload_log.tsv* file or *redcaperrors.txt*. If these files exist, they may note any files or data which would have been over-written in the database. Check with the wet lab if any RD-numbers were duplicated or previously used for the samples listed in the upload_log.tsv file.**NOTE** When running the test case run *21-MGDM_TEST* you may notice an error with the upload log as these reports would already exist in REDCap. It is normal for the test case to fail uploading since the REDCap database already contains the data and files for the test run.