Merge pull request ejseqera#1 from ejseqera/add_demo_docs

Add demo docs, seqerakit updates

.gitignore

@@ -0,0 +1,3 @@
+node_modules/*
+package*
+package-lock.json

README.md

@@ -1,27 +1,107 @@
-# `seqeralabs/showcase` Workspace on Nextflow Tower
+# `seqeralabs/showcase` Workspace on Seqera Platform
 
 ## Infrastructure as code
 
-Most [Nextflow Tower](https://cloud.tower.nf/) entities such as Pipelines, Compute Environments and Datasets can be exported in JSON format via the [Nextflow Tower CLI](https://github.com/seqeralabs/tower-cli#nextflow-tower-cli). This is very useful for creating infrastructure as code to store the exact configuration options used to create these entities, and to share and track changes over time.
+Infrastructure as Code (IaC) provides the ability to provision and manage infrastructure through configuration files. Having IaC enables automation in set up, consistency and standardization in infrastructure, and documentation of how resources are configured.
 
-This repository contains JSON representation of the Nextflow Tower entitites that were used to create the `seqeralabs/showcase` Workspace:
-- [Compute Environments](compute-envs)
-- [Pipelines](pipelines)
-- [Datasets](datasets)
+This guide describes how to create resources on Seqera Platform through IaC using two methods: (1) Seqera Platform CLI, and (2) seqerakit.
 
-## Scripts
+### 1. Using Seqera Platform CLI
 
-The bash scripts required to programatically import and export these entities into Tower via the Tower CLI have also been provided in the [`scripts`](scripts) directory. 
+Most [Seqera Platform](https://seqera.io/platform/) entities such as Pipelines, Compute Environments and Datasets can be exported in JSON format via the [Seqera Platform CLI](https://github.com/seqeralabs/tower-cli#nextflow-tower-cli). This is very useful for creating IaC to store the exact configuration options used to create these entities, and to share and track changes over time.
 
-See Tower CLI [usage docs](https://github.com/seqeralabs/tower-cli/blob/master/USAGE.md#usage-examples) for more examples.
+This repository contains JSON representation of the Seqera Platform entitites generated by the Platform CLI that were used to create the `seqeralabs/showcase` Workspace:
 
-### Prerequisites
+- [Compute Environments](compute-envs) for AWS, Azure, Google Cloud, and SLURM
+- [Pipelines](pipelines) from a select set of [nf-core pipelines](https://nf-co.re/pipelines)
+- [Datasets](datasets) for running the pipelines with minimal test data
 
-1. [Nextflow Tower CLI](https://github.com/seqeralabs/tower-cli#1-installation)
-2. [`jq`](https://stedolan.github.io/jq/)
+#### Running the Bash scripts
 
-### Assumptions
+The bash scripts required to programatically import and export these entities from Seqera Platform via the CLI have also been provided in the [`scripts`](scripts) directory.
 
-- Tower CLI has been installed and `tw info` has been configured
-- `TOWER_API_ENDPOINT` and `TOWER_ACCESS_TOKEN` have been injected into the executing environment
-- Credentials have already been set-up in Tower
+To create the Compute Environments:
+
+```bash
+bash scripts/bash/tw_computeenvs_import.sh
+```
+
+To add the Datasets:
+
+```bash
+bash scripts/bash/tw_datasets_add.sh
+```
+
+To add the Pipelines:
+
+```bash
+bash scripts/bash/tw_pipelines_import.sh
+```
+
+See Seqera Platform CLI [usage docs](https://github.com/seqeralabs/tower-cli/blob/master/USAGE.md#usage-examples) for more examples of interacting with the Platform.
+
+#### Software Prerequisites
+
+1. [Seqera Platform CLI](https://github.com/seqeralabs/tower-cli#1-installation) installed
+2. [`jq`](https://stedolan.github.io/jq/) installed
+
+#### Assumptions
+
+- [Seqera Platform CLI](https://github.com/seqeralabs/tower-cli#1-installation) installed and `tw info` has been configured
+- `TOWER_API_ENDPOINT` and `TOWER_ACCESS_TOKEN` have been injected into the executing environment (see the [Seqera Platform CLI documentation](https://github.com/seqeralabs/tower-cli/tree/master?tab=readme-ov-file#2-configuration) on configuring these)
+- Credentials for AWS, Azure, and Google have already been set-up in Seqera Platform
+
+### 2. Using `seqerakit`
+
+The [Seqera Platform](https://https://seqera.io/platform/) entities represented above in JSON can also be represented as YAML configuration files. These configuration YAMLs can be used by `seqerakit`, a Python package wrapping the Seqera Platform CLI, to define and automate how entities should be created.
+
+This repository also contains YAML configuration files defining creation of Seqera Platform entities on the `seqeralabs/showcase` Workspace with `seqerakit`:
+
+- [Compute Environments](seqerakit/compute-envs/)
+- [Pipelines](seqerakit/pipelines/)
+- [Datasets](seqerakit/datasets/)
+
+#### Prerequisites
+
+To use `seqerakit`, you will need to have:
+
+- The [Seqera Platform CLI](https://github.com/seqeralabs/tower-cli#1-installation) installed and `tw info` has been configured as described above
+- `TOWER_API_ENDPOINT` and `TOWER_ACCESS_TOKEN` environment variables set
+  Credentials for AWS, Azure, and Google have already been set-up in Seqera Platform
+- The package installed via:
+
+```bash
+conda install seqerakit
+```
+
+Or:
+
+```bash
+pip install seqerakit
+```
+
+Please refer to the package's [installation guide](https://github.com/seqeralabs/seqera-kit?tab=readme-ov-file#installation) for more details on the required dependencies.
+
+#### Running `seqerakit`
+
+Similar to executing the scripts above to import and create Pipelines, Compute Environments, and Datasets that are defined in JSON onto Seqera Platform, the [YAML files](#using-seqerakit) provided in this repository can also be used to create `seqeralabs/showcase` entities on the Platform. For example:
+
+To create the Compute Environments:
+
+```bash
+seqerakit seqerakit/compute-envs/*
+```
+
+To add the Datasets:
+
+```bash
+seqerakit seqerakit/datasets/*
+```
+
+To add the Pipelines:
+
+```bash
+seqerakit seqerakit/pipelines/*
+```
+
+See the [seqerakit documentation](https://github.com/seqeralabs/seqera-kit#-seqerakit) for more usage examples and options for running `seqerakit`.

datasets/SRP459762-carcinoma-samplesheet.csv

@@ -0,0 +1,6 @@
+sample,fastq_1,fastq_2,strandedness,run_accession,experiment_accession,sample_accession,secondary_sample_accession,study_accession,secondary_study_accession,submission_accession,run_alias,experiment_alias,sample_alias,study_alias,library_layout,library_selection,library_source,library_strategy,library_name,instrument_model,instrument_platform,base_count,read_count,tax_id,scientific_name,sample_title,experiment_title,study_title,sample_description,fastq_md5,fastq_bytes,fastq_ftp,fastq_galaxy,fastq_aspera
+SRX21750941,s3://seqeralabs-showcase/nf-core-fetchngs/SRP459762/results/fastq/SRX21750941_SRR26033656_1.fastq.gz,s3://seqeralabs-showcase/nf-core-fetchngs/SRP459762/results/fastq/SRX21750941_SRR26033656_2.fastq.gz,auto,SRR26033656,SRX21750941,SAMN37353910,SRS18858468,PRJNA1014965,SRP459762,SRA1710039,MEC_T_07_1.fastq.gz,MEC_T_07,,PRJNA1014965,PAIRED,unspecified,TRANSCRIPTOMIC,RNA-Seq,MEC_T_07,Illumina HiSeq 2500,ILLUMINA,9105149798,45074999,9606,Homo sapiens,,Illumina HiSeq 2500 sequencing: RNA-Seq of mucoepidermoid carcinoma,Immunological subtyping of salivary gland cancer identifies histological origin-specific tumor immune microenvironment,,a4e3385ac9b33a12a2648687af5a9eda;1cba4d7de8569bc84216d0d6f5ba19a1,1734457262;1749369782,ftp.sra.ebi.ac.uk/vol1/fastq/SRR260/056/SRR26033656/SRR26033656_1.fastq.gz;ftp.sra.ebi.ac.uk/vol1/fastq/SRR260/056/SRR26033656/SRR26033656_2.fastq.gz,ftp.sra.ebi.ac.uk/vol1/fastq/SRR260/056/SRR26033656/SRR26033656_1.fastq.gz;ftp.sra.ebi.ac.uk/vol1/fastq/SRR260/056/SRR26033656/SRR26033656_2.fastq.gz,fasp.sra.ebi.ac.uk:/vol1/fastq/SRR260/056/SRR26033656/SRR26033656_1.fastq.gz;fasp.sra.ebi.ac.uk:/vol1/fastq/SRR260/056/SRR26033656/SRR26033656_2.fastq.gz
+SRX21750942,s3://seqeralabs-showcase/nf-core-fetchngs/SRP459762/results/fastq/SRX21750942_SRR26033655_1.fastq.gz,s3://seqeralabs-showcase/nf-core-fetchngs/SRP459762/results/fastq/SRX21750942_SRR26033655_2.fastq.gz,auto,SRR26033655,SRX21750942,SAMN37353911,SRS18858470,PRJNA1014965,SRP459762,SRA1710039,MEC_T_08_1.fastq.gz,MEC_T_08,,PRJNA1014965,PAIRED,unspecified,TRANSCRIPTOMIC,RNA-Seq,MEC_T_08,Illumina HiSeq 2500,ILLUMINA,7291499060,36096530,9606,Homo sapiens,,Illumina HiSeq 2500 sequencing: RNA-Seq of mucoepidermoid carcinoma,Immunological subtyping of salivary gland cancer identifies histological origin-specific tumor immune microenvironment,,37c4046ba9128693ff685fd1a8f867da;2aa0babe90e17ed58636909b11d0a4eb,1331451936;1345825378,ftp.sra.ebi.ac.uk/vol1/fastq/SRR260/055/SRR26033655/SRR26033655_1.fastq.gz;ftp.sra.ebi.ac.uk/vol1/fastq/SRR260/055/SRR26033655/SRR26033655_2.fastq.gz,ftp.sra.ebi.ac.uk/vol1/fastq/SRR260/055/SRR26033655/SRR26033655_1.fastq.gz;ftp.sra.ebi.ac.uk/vol1/fastq/SRR260/055/SRR26033655/SRR26033655_2.fastq.gz,fasp.sra.ebi.ac.uk:/vol1/fastq/SRR260/055/SRR26033655/SRR26033655_1.fastq.gz;fasp.sra.ebi.ac.uk:/vol1/fastq/SRR260/055/SRR26033655/SRR26033655_2.fastq.gz
+SRX21750943,s3://seqeralabs-showcase/nf-core-fetchngs/SRP459762/results/fastq/SRX21750943_SRR26033654_1.fastq.gz,s3://seqeralabs-showcase/nf-core-fetchngs/SRP459762/results/fastq/SRX21750943_SRR26033654_2.fastq.gz,auto,SRR26033654,SRX21750943,SAMN37353912,SRS18858471,PRJNA1014965,SRP459762,SRA1710039,MEC_T_09_1.fastq.gz,MEC_T_09,,PRJNA1014965,PAIRED,unspecified,TRANSCRIPTOMIC,RNA-Seq,MEC_T_09,Illumina HiSeq 2500,ILLUMINA,7761595884,38423742,9606,Homo sapiens,,Illumina HiSeq 2500 sequencing: RNA-Seq of mucoepidermoid carcinoma,Immunological subtyping of salivary gland cancer identifies histological origin-specific tumor immune microenvironment,,d7a91ce3b764c79722d4f36ec5713efa;e42d3c993a18108a5258af65141c580f,1170104300;1203799314,ftp.sra.ebi.ac.uk/vol1/fastq/SRR260/054/SRR26033654/SRR26033654_1.fastq.gz;ftp.sra.ebi.ac.uk/vol1/fastq/SRR260/054/SRR26033654/SRR26033654_2.fastq.gz,ftp.sra.ebi.ac.uk/vol1/fastq/SRR260/054/SRR26033654/SRR26033654_1.fastq.gz;ftp.sra.ebi.ac.uk/vol1/fastq/SRR260/054/SRR26033654/SRR26033654_2.fastq.gz,fasp.sra.ebi.ac.uk:/vol1/fastq/SRR260/054/SRR26033654/SRR26033654_1.fastq.gz;fasp.sra.ebi.ac.uk:/vol1/fastq/SRR260/054/SRR26033654/SRR26033654_2.fastq.gz
+SRX21750944,s3://seqeralabs-showcase/nf-core-fetchngs/SRP459762/results/fastq/SRX21750944_SRR26033653_1.fastq.gz,s3://seqeralabs-showcase/nf-core-fetchngs/SRP459762/results/fastq/SRX21750944_SRR26033653_2.fastq.gz,auto,SRR26033653,SRX21750944,SAMN37353913,SRS18858472,PRJNA1014965,SRP459762,SRA1710039,MEC_T_10_1.fastq.gz,MEC_T_10,,PRJNA1014965,PAIRED,unspecified,TRANSCRIPTOMIC,RNA-Seq,MEC_T_10,Illumina HiSeq 2500,ILLUMINA,9238355264,45734432,9606,Homo sapiens,,Illumina HiSeq 2500 sequencing: RNA-Seq of mucoepidermoid carcinoma,Immunological subtyping of salivary gland cancer identifies histological origin-specific tumor immune microenvironment,,1a63023afeca02cd10271a20385424b1;0939823df564a6d14b61eb45581994e8,1768219705;1783586223,ftp.sra.ebi.ac.uk/vol1/fastq/SRR260/053/SRR26033653/SRR26033653_1.fastq.gz;ftp.sra.ebi.ac.uk/vol1/fastq/SRR260/053/SRR26033653/SRR26033653_2.fastq.gz,ftp.sra.ebi.ac.uk/vol1/fastq/SRR260/053/SRR26033653/SRR26033653_1.fastq.gz;ftp.sra.ebi.ac.uk/vol1/fastq/SRR260/053/SRR26033653/SRR26033653_2.fastq.gz,fasp.sra.ebi.ac.uk:/vol1/fastq/SRR260/053/SRR26033653/SRR26033653_1.fastq.gz;fasp.sra.ebi.ac.uk:/vol1/fastq/SRR260/053/SRR26033653/SRR26033653_2.fastq.gz
+

datasets/SRP459762-carcinoma-sra-ids.csv

@@ -0,0 +1,4 @@
+SRR26033653
+SRR26033654
+SRR26033655
+SRR26033656

demo/docs/add_a_dataset.md

@@ -0,0 +1,36 @@
+# Datasets
+
+Most bioinformatics pipelines will require an input of some sort, typically a samplesheet where each row consists of a sample, the location of files for that sample (such as fastq files), and other sample details.
+
+Datasets in Seqera Platform are CSV (comma-separated values) and TSV (tab-separated values) files stored in a workspace. They are used as inputs to pipelines to simplify data management, minimize user data-input errors, and facilitate reproducible workflows.
+
+When running pipelines on the Cloud, this samplesheet has to be made available in Cloud storage or a remote location. Instead of doing this, we can upload a samplesheet we have locally, as a Dataset to the Platform to specify as input to our pipeline.
+
+## 1. Download the nf-core/rnaseq test samplesheet
+
+The [nf-core/rnaseq](https://github.com/nf-core/rnaseq) pipeline works with input datasets (samplesheets) containing sample names, fastq file locations, and indications of strandedness. The Seqera Community Showcase sample dataset for _nf-core/rnaseq_ looks like this:
+
+**Example rnaseq dataset**
+<center>
+
+| sample              | fastq_1                              | fastq_2                              | strandedness |
+| ------------------- | ------------------------------------ | ------------------------------------ | ------------ |
+| WT_REP1             | s3://nf-core-awsmegatests/rnaseq/... | s3://nf-core-awsmegatests/rnaseq/... | reverse      |
+| WT_REP1             | s3://nf-core-awsmegatests/rnaseq/... | s3://nf-core-awsmegatests/rnaseq/... | reverse      |
+| WT_REP2             | s3://nf-core-awsmegatests/rnaseq/... | s3://nf-core-awsmegatests/rnaseq/... | reverse      |
+| RAP1_UNINDUCED_REP1 | s3://nf-core-awsmegatests/rnaseq/... |                                      | reverse      |
+| RAP1_UNINDUCED_REP2 | s3://nf-core-awsmegatests/rnaseq/... |                                      | reverse      |
+| RAP1_UNINDUCED_REP2 | s3://nf-core-awsmegatests/rnaseq/... |                                      | reverse      |
+| RAP1_IAA_30M_REP1   | s3://nf-core-awsmegatests/rnaseq/... | s3://nf-core-awsmegatests/rnaseq/... | reverse      |
+
+</center>
+
+Download the nf-core/rnaseq [samplesheet_test.csv](samplesheet_test.csv) provided in this repository on to your computer.
+
+## 2. Add the Dataset
+
+Go to the 'Datasets' tab and click 'Add Dataset'.
+
+![Adding a Dataset](assets/sp-cloud-add-a-dataset.gif)
+
+Specify a name for the dataset such as 'nf-core-rnaseq-test-dataset', description, include the first row as header, and upload the CSV file provided in this repository. This CSV file specifies the paths to 7 small FASTQ files for a sub-sampled Yeast RNAseq dataset.

demo/docs/add_a_pipeline.md

@@ -0,0 +1,35 @@
+# Add a Pipeline to the Launchpad
+
+The Launchpad allows you to launch and manage Nextflow pipelines and associated compute that your pipelines will be executed on. Using the Launchpad, you can create a curated set of pipelines (including variations of the same pipeline) that are ready to be executed on the associated compute environments, while allowing the user to customize the pipeline-level parameters if needed.
+
+## 1. Add a Pipeline
+
+To add a pipeline, click on the **'Add Pipeline'** button. As an example, we will add the [nf-core/rnaseq](https://github.com/nf-core/rnaseq) pipeline to the Launchpad.
+
+![Adding nf-core/rnaseq pipeline](assets/sp-cloud-add-rnaseq.gif)
+
+Specify a name, description, and click on pre-existing AWS compute environment to execute on.
+
+## 2. Specify a repository URL and revision
+
+In the repository URL, specify the nf-core/rnaseq repository:
+
+```bash
+https://github.com/nf-core/rnaseq
+```
+
+Additionally, specify a version of the pipeline as the 'Revision number'. You can use `3.12.0`.
+
+## 3. Parameters and Nextflow Configuration
+
+Pipeline parameters and Nextflow configuration settings can also be specified as you add the pipeline to the Launchpad.
+
+For example, a pipeline can be pre-populated to run with specific parameters on the Launchpad.
+![Adding pipeline parameters](assets/sp-cloud-pipeline-params.gif)
+
+## 4. Pre-run script and additional options
+
+You can run custom code either before or after the execution of the Nextflow script. These text fields allow you to enter shell commands.
+
+Pre-run scripts are executed in the nf-launch script prior to invoking Nextflow processes. Pre-run scripts are useful for executor setup (e.g., use a specific version of Nextflow) and troubleshooting.
+![Specify NF version in pre-run script](assets/sp-cloud-pre-run-options.gif)

demo/docs/data_explorer.md

@@ -0,0 +1,33 @@
+# Data Explorer
+
+With Data Explorer, you can browse and interact with remote data repositories from organization workspaces in Seqera Platform. It supports AWS S3, Azure Blob Storage, and Google Cloud Storage repositories.
+
+## 1. Data Explorer features
+
+- View bucket details
+  To view bucket details such as the cloud provider, bucket address, and credentials, select the information icon next to a bucket in the Data Explorer list.
+
+- Search and filter buckets
+  Search for buckets by name and region (e.g., `region:eu-west-2`) in the search field, and filter by provider.
+
+- Hide buckets from list view
+  Workspace maintainers can hide buckets from the Data Explorer list view. Select multiple buckets, then select Hide in the Data Explorer toolbar. To hide buckets individually, select Hide from the options menu of a bucket in the list.
+
+The Data Explorer list filter defaults to Only visible. Select Only hidden or All from the filtering menu to view hidden buckets in the list. You can Unhide a bucket from its options menu in the list view.
+
+- View bucket contents
+  Select a bucket name from the Data Explorer list to view the contents of that bucket. From the View cloud bucket page, you can browse directories and search for objects by name in a particular directory. The file type, size, and path of objects are displayed in columns to the right of the object name. To view bucket details such as the cloud provider, bucket address, and credentials, select the information icon.
+
+- Preview and download files
+  From the View cloud bucket page, you can preview and download files. Select the download icon in the Actions column to download a file directly from the list view. Select a file to open a preview window that includes a Download button.
+
+## 2. View Run outputs in Data Explorer
+
+Data Explorer can be used to view the outputs of your pipelines.
+
+From the View cloud bucket page, you can:
+
+1. Preview and download files: Select the download icon in the 'Actions' column to download a file directly from the list view. Select a file to open a preview window that includes a Download button.
+2. Copy bucket/object paths: Select the Path of an object on the cloud bucket page to copy its absolute path to the clipboard. Use these object paths to specify input data locations during pipeline launch, or add them to a dataset for pipeline input.
+
+![Data Explorer bucket](assets/sp-cloud-data-explorer.gif)

demo/docs/demo_overview.md

@@ -0,0 +1,18 @@
+### 1. Login to seqera.io
+
+Log into Seqera Platform, either through a GitHub account, Google account, or an email address.
+
+If an email address is provided, Seqera Cloud will send an authentication link to the email address to login with.
+
+![Seqera Platform Cloud login](assets/sp-cloud-signin.gif)
+
+### 2. Navigate into the seqeralabs/showcase Workspace
+
+All resources in Seqera Platform live inside a Workspace, which in turn belong to an Organisation. Typically, teams of colleagues or collaborators will share one or more workspaces. All resources in a Workspace (i.e. pipelines, compute environments, datasets) are shared by members of that workspace.
+
+Navigate into the `seqeralabs/showcase` Workspace.
+
+![Seqera Labs Showcase Workspace](assets/go-to-workspace.gif)
+
+### 3. TODO User settings
+