EVA-3833: Address user feedback on documentation

.github/workflows/publish_docs.yml

@@ -0,0 +1,50 @@
+name: Deploy documentation with Jekyll / GitHub Pages
+
+on:
+  push:
+    tags:
+      - '**'
+
+  # Allows you to run this workflow manually from the Actions tab
+  workflow_dispatch:
+
+# Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages
+permissions:
+  contents: read
+  pages: write
+  id-token: write
+
+# Allow only one concurrent deployment, skipping runs queued between the run in-progress and latest queued.
+# However, do NOT cancel in-progress runs as we want to allow these production deployments to complete.
+concurrency:
+  group: "pages"
+  cancel-in-progress: false
+
+jobs:
+  # Build job
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+      - name: Setup Pages
+        uses: actions/configure-pages@v5
+      - name: Build with Jekyll
+        uses: actions/jekyll-build-pages@v1
+        with:
+          source: ./docs
+          destination: ./_site
+      - name: Upload artifact
+        uses: actions/upload-pages-artifact@v3
+
+  # Deployment job
+  deploy:
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    runs-on: ubuntu-latest
+    needs: build
+    steps:
+      - name: Deploy to GitHub Pages
+        id: deployment
+        uses: actions/deploy-pages@v4

README.md

@@ -1,184 +1,3 @@
 # eva-sub-cli
-The eva-sub-cli tool is a command line interface tool for data validation and upload. The tool transforms the submission process at EVA by enabling users to take control of data validation process. Previously handled by our helpdesk team, validation can now be performed directly by users, streamlining and improving the overall submission workflow at the EVA. 
 
-
-## Installation
-
-There are currently three ways to install and run the tool : 
-- Using conda
-- From source using Docker
-- From source natively (i.e. installing dependencies yourself)
-
-### 1. Conda
-
-The most straightforward way to install eva-sub-cli and its dependencies is through conda.
-For instance, the following commands install eva-sub-cli in a new environment called `eva`, activate the environment, and print
-the help message:
-```bash
-conda create -n eva -c conda-forge -c bioconda eva-sub-cli
-conda activate eva
-eva-sub-cli.py --help
-````
-
-### 2. From source using Docker
-
-Docker provides an easy way to run eva-sub-cli without installing dependencies separately.
-This method requires just Python 3.8+ and [Docker](https://docs.docker.com/engine/install/) to be installed.
-Then you can install the most recent version from [PyPI](https://pypi.org/project/eva-sub-cli/) in a virtual environment:
-```bash
-pip install eva-sub-cli
-```
-
-To verify that the cli tool is installed correctly, run the following command, and you should see the help message displayed : 
-```bash
-eva-sub-cli.py -h
-```
-
-### 3. From source natively
-
-This installation method requires the following :
-* Python 3.8+
-* [Nextflow](https://www.nextflow.io/docs/latest/getstarted.html) 21.10+
-* [biovalidator](https://github.com/elixir-europe/biovalidator) 2.1.0+
-* [vcf-validator](https://github.com/EBIvariation/vcf-validator) 0.9.7+
-
-Install each of these and ensure they are included in your PATH. Then install the latest release as previously described.
-
-## Getting started with the eva-sub-cli tool 
-
-The ["Getting Started" guide](docs/Getting_Started_with_eva_sub_cli.md) serves as an introduction for users of the eva-sub-cli tool. It includes instructions on how to prepare your data and metadata, ensuring that users are equipped with the necessary information to successfully submit variant data. This guide is essential for new users, offering practical advice and tips for a smooth onboarding experience with the eva-sub-cli tool.
-
-## Options and parameters guide
-
-The eva-sub-cli tool provides several options and parameters that you can use to tailor its functionality to your needs.
-You can view all the available parameters with the command `eva-sub-cli.py -h` and view detailed explanations for the
-input file requirements in the ["Getting Started" guide](docs/Getting_Started_with_eva_sub_cli.md).
-Below is an overview of the key parameters.
-
-### Submission directory
-
-This is the directory where all processing will take place, and where configuration and reports will be saved.
-Crucially, the eva-sub-cli tool requires that there be **only one submission per directory** and that the submission directory not be reused.
-Running multiple submissions from a single directory can result in data loss during validation and submission.
-
-### Metadata file
-
-Metadata can be provided in one of two files.
-
-#### The metadata spreadsheet
-
-The metadata template can be found within the [etc folder](eva_sub_cli/etc/EVA_Submission_template.xlsx). It should be populated following the instructions provided within the template.
-This is passed using the option `--metadata_xlsx`.
-
-#### The metadata JSON
-
-The metadata can also be provided via a JSON file, which should conform to the schema located [here](eva_sub_cli/etc/eva_schema.json).
-This is passed using the option `--metadata_json`.
-
-### VCF files and Reference FASTA
-
-These can be provided either in the metadata file directly, or on the command line using the `--vcf_files` and `--reference_fata` options.
-Note that if you are using more than one reference FASTA, you **cannot** use the command line options; you must specify which VCF files use which FASTA files in the metadata.
-
-VCF files can be either uncompressed or compressed using bgzip.
-Other types of compression are not allowed and will result in errors during validation.
-FASTA files must be uncompressed.
-
-## Execution
-
-### Validate only
-
-To validate and not submit, run the following command:
-
-```shell
-eva-sub-cli.py --metadata_xlsx metadata_spreadsheet.xlsx --submission_dir submission_dir --tasks VALIDATE
-```
-
-**Note for Docker users:** 
-
-Make sure that Docker is running in the background, e.g. by opening Docker Desktop.
-For each of the below commands, add the command line option `--executor docker`, which will
-fetch and manage the Docker container for you. 
-
-```shell
-eva-sub-cli.py --metadata_xlsx metadata_spreadsheet.xlsx --submission_dir submission_dir --tasks VALIDATE --executor docker 
-```
-
-### Validate and submit your dataset
-
-To validate and submit, run the following command:
-
-```shell
-eva-sub-cli.py --metadata_xlsx metadata_spreadsheet.xlsx \
-               --vcf_files vcf_file1.vcf vcf_file2.vcf --reference_fasta assembly.fa --submission_dir submission_dir
-```
-
-
-### Submit only
-
-All submissions must have been validated. You cannot run the submission without validation. Once validated, execute the following command:
-
-```shell
-eva-sub-cli.py --metadata_xlsx metadata_spreadsheet.xlsx --submission_dir submission_dir
-```
-or 
-```shell
-eva-sub-cli.py --metadata_xlsx metadata_spreadsheet.xlsx --submission_dir submission_dir --tasks SUBMIT
-```
-This will only submit the data and not validate.
-
-### Shallow validation
-
-If you are working with large VCF files and find that validation takes a very long time, you can add the
-argument `--shallow` to the command, which will validate only the first 10,000 lines in each VCF. Note that running
-shallow validation will **not** be sufficient for actual submission.
-
-
-## Leveraging Nextflow to parallelize the validation process  
-
-Nextflow is a common workflow management system that helps orchestrate tasks and interface with the execution engine (like HPC or cloud). When running natively (i.e. not using Docker), eva-sub-cli will use Nextflow to run all the validation steps. In this section we'll see how it can be parameterised to work with your compute infrastructure.
-
-When no options are provided, Nextflow will run as many tasks as there are available CPUs on the machine executing it. To modify how many tasks can start and how Nextflow will process each one, you can provide a Nextflow configuration file in several ways.
-
-From the command line you can use `--nextflow_config <path>` to specify the Nextflow config file you want to apply. The configuration can also be picked up from other places directly by Nextflow. Please refer to [the nextflow documentation](https://www.nextflow.io/docs/latest/config.html) for more details.
-
-### Basic Nextflow configuration.
-
-There are many options to configure Nextflow so we will not provide them all. Please refer to [the documentation](https://www.nextflow.io/docs/latest/reference/config.html) for advanced features.
-Below is a very basic Nextflow configuration file that will request 2 cpus for each process, essentially limiting the number of process to half the number of available CPUs 
-```
-process {
-    executor="local"
-    cpus=2
-}
-```
-In this configuration, all the process will be running on the same machine where eva-sub-cli was started as described in the schema below.
-```
-(Local machine)
-eva-sub-cli
-  |_ nextflow
-      |_ task1
-      |_ task2
-```
-
-### Basic Nextflow configuration for HPC use.
-
-If you have access to High Performance Compute (HPC) environment, Nextflow supports the main resource managers such as [SLURM](https://www.nextflow.io/docs/latest/executor.html#slurm), [SGE](https://www.nextflow.io/docs/latest/executor.html#sge), [LSF](https://www.nextflow.io/docs/latest/executor.html#lsf) and others.
-In the configuration below, we're assuming that you are using SLURM. It would work similarly with other resource managers.
-```
-process {
-    executor="slurm"
-    queue="my_production_queue"
-}
-```
-
-In this configuration, the subtasks will be performed in other machines as specified by your SLURM resource manager as described in the schema below.
-```
-(Local machine)
-eva-sub-cli
-  |_ nextflow
-(Other compute node)
-task1
-(Other compute node)
-task2
-```
+Please view our [documentation website](https://ebivariation.github.io/eva-sub-cli) for help installing and using eva-sub-cli.

docs/README.md

@@ -0,0 +1,10 @@
+# About eva-sub-cli
+The eva-sub-cli tool is a command line interface tool for data validation and upload. The tool enables users to perform
+validation and submission to the EVA directly, without requiring interaction with our helpdesk team.
+
+For instructions on how to install eva-sub-cli, see [Installation](installation.md).
+
+For instructions on how to use eva-sub-cli to submit data, see [How to Submit](how_to_submit.md).
+
+## Contact us
+For help or feedback, please get in touch with us at [eva-helpdesk@ebi.ac.uk](mailto:eva-helpdesk@ebi.ac.uk).

docs/_config.yml

@@ -0,0 +1 @@
+remote_theme: just-the-docs/just-the-docs

docs/how_to_submit.md

@@ -0,0 +1,89 @@
+# How to Submit
+
+## 1. File preparation
+You should have **one or more VCF files** to submit, as well as a **completed metadata spreadsheet** following [our 
+template](https://raw.githubusercontent.com/EBIvariation/eva-sub-cli/main/eva_sub_cli/etc/EVA_Submission_template.xlsx).
+More guidance on these files can be found in the [inputs overview](input_file_overview.md) as well as in the metadata 
+template itself.
+
+You will also need the **reference sequences in FASTA format** that you used to generate the VCF files. This is required for
+validation, though it will not be submitted.
+
+## 2. Setting up
+First [**install eva-sub-cli**](installation.md), or check that you have the latest version installed.
+
+You will need an [**ENA Webin account**](https://www.ebi.ac.uk/ena/submit/webin/login) in order to submit, though you
+can run validation without one.
+
+Finally, you need to specify a **submission directory**, which is a specific directory associated with each submission.
+This is where all processing will take place, and where configuration and reports will be saved.
+
+Crucially, the eva-sub-cli tool requires that there be **only one submission per directory** and
+that the submission directory not be reused. Running multiple submissions from a single directory can result in data 
+loss during validation and submission.
+
+## 3. Running eva-sub-cli
+The basic command to run validation and submission is as follows:
+```shell
+eva-sub-cli.py --metadata_xlsx <metadata file> --submission_dir <submission directory> --username <webin username> --password <webin password>
+```
+This will run validation and generate a report with the results. The report can be viewed as HTML or plain text. More 
+description of the validation checks being performed can be found in the [validation overview](validation_overview.md).
+
+**NOTE:** If the validation is successful, this command will automatically submit your data and metadata to EVA. If you 
+don't want this, see [below](#running-only-validation-or-only-submission).
+
+If your files are large, we also encourage you to use [shallow validation](#shallow-validation) before running full
+validation and submission.
+
+## 4. Post submission
+Once you've successfully validated and submitted your data, you will receive an automatic email stating that your data
+has been uploaded. If you don't get this email within a short time, we have very likely **not received your data**, so
+you should get in touch with our [helpdesk](mailto:eva-helpdesk@ebi.ac.uk).
+
+If your submission contains **human genotype data**, then you will also need to provide a signed copy of our
+[consent statement](https://docs.google.com/document/d/1UaRmimAe919IZYIH55mDXMe-4sHsRVqaP4moqx2IYE4) via email.
+For more information please see our [submission FAQs](https://www.ebi.ac.uk/eva/?Help#submissionPanel&link=consent-statement-for-human-genotype-data).
+
+## Other options
+The above steps are the most common, but there are some other options that you can use to tailor its functionality to
+your needs. A full listing is available by running `eva-sub-cli.py -h`.
+
+### Running only validation or only submission
+If you want to run validation without submitting automatically once the validation passes, you can use the `--tasks` 
+option. To run only validation:
+```shell
+eva-sub-cli.py --metadata_xlsx <metadata file> --submission_dir <submission directory> --tasks VALIDATE
+```
+To run only submission - note this will **not submit** if you've not yet validated successfully:
+```shell
+eva-sub-cli.py --metadata_xlsx <metadata file> --submission_dir <submission directory> --tasks SUBMIT --username <webin username> --password <webin password>
+```
+
+### Shallow validation
+If you are working with large VCF files and find that validation takes a very long time, you can add the
+argument `--shallow` to the command, which will validate only the first 10,000 lines in each VCF. Note that running
+shallow validation will **not** be sufficient for actual submission but will allow you to identify most validation concerns without long run time.
+
+### VCF files and reference FASTA
+These can be provided either in the metadata file directly, or on the command line using the `--vcf_files` and
+`--reference_fasta` options. Note that if you are using more than one reference FASTA, you **cannot** use the command 
+line options; you must specify which VCF files use which FASTA files in the metadata.
+
+VCF files can be either uncompressed or compressed using bgzip.
+Other types of compression are not allowed and will result in errors during validation.
+FASTA files must be uncompressed.
+
+### Metadata JSON
+Frequent submitters may be interested in using our [metadata JSON schema](https://github.com/EBIvariation/eva-sub-cli/blob/main/eva_sub_cli/etc/eva_schema.json)
+instead of our spreadsheet template. The metadata requirements are the same regardless of which format you use, you will
+just need to use the `--metadata_json` option instead of the `--metadata_xlsx` option.
+
+### Running using Docker
+If you've installed eva-sub-cli using Docker, make sure that Docker is running in the background before running 
+eva-sub-cli, e.g. by opening Docker Desktop. Additionally, for each eva-sub-cli command, add the command line option 
+`--executor docker`, which will fetch and manage the Docker container for you.
+
+### Using Nextflow
+Under the hood, eva-sub-cli uses Nextflow to run validation, which you can configure on the command line using the 
+`--nextflow_config` option. For more information, please see [here](using_nextflow.md).