Update docs to v0.2.0 (#213)

This commit updates the StreamFlow documentation to reflect API changes
occurred in version `v0.2.x`. Plus, it better structures the
documentation, adding more sections and a better description of the
extension points.

.github/workflows/ci-tests.yaml

@@ -72,6 +72,8 @@ jobs:
           node-version: "15"
       - name: "Install Docker (MacOS X)"
         uses: douglascamata/setup-docker-macos-action@v1-alpha.9
+        env:
+          HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK: 1
         if: ${{ startsWith(matrix.on, 'macos-') }}
       - uses: docker/setup-qemu-action@v2
       - name: "Install Apptainer"
@@ -211,6 +213,8 @@ jobs:
           node-version: "15"
       - name: "Install Docker (MacOs X)"
         uses: douglascamata/setup-docker-macos-action@v1-alpha.9
+        env:
+          HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK: 1
         if: ${{ startsWith(matrix.on, 'macos-') }}
       - uses: docker/setup-qemu-action@v2
       - name: "Install Apptainer"

README.md

@@ -46,7 +46,8 @@ docker run -d \
     --mount type=bind,source="$(pwd)"/my-project,target=/streamflow/project \
     --mount type=bind,source="$(pwd)"/results,target=/streamflow/results \
     --mount type=bind,source="$(pwd)"/tmp,target=/tmp/streamflow \
-    alphaunito/streamflow run /streamflow/project/streamflow.yml
+    alphaunito/streamflow \
+    streamflow run /streamflow/project/streamflow.yml
 ```
 
 #### Kubernetes

docs/source/advanced/multiple-targets.rst

@@ -0,0 +1,27 @@
+================
+Multiple targets
+================
+
+StreamFlow lets users to map steps and ports to multiple targets in the :ref:`StreamFlow file <Put it all together>`. A step bound to multiple locations can be scheduled on each of them at runtime. Plus, if a step encloses multiple instances (e.g., in a CWL ``scatter`` operation), they can run on different targets.
+
+The `filters` directive defines one or more strategies to select a target among the set of available ones (or among a subset) at runtime. By default, all available targets will be evaluated in the order of appearance in the ``target`` directive.
+
+Users can select a given :ref:`BindingFilter <BindingFilter>` implementation by specifying its name in the ``filters`` directive of a ``binding`` object. If multiple filters are declared, they are applied to the target list in the order of appearance. For example, to evaluate targets in random order at every allocation request, users can specify the following:
+
+.. code-block:: yaml
+
+   workflows:
+     example:
+       type: cwl
+       config:
+         file: main.cwl
+         settings: config.yml
+       bindings:
+        - step: /compile
+          target:
+            - deployment: first-deployment
+            - deployment: second-deployment
+          filters:
+            - shuffle
+
+Conversely, a file or directory port bound to multiple locations can be retrieved from each of them at runtime. StreamFlow will always try to minimize the overhead of data transfers, using local data whenever possible.

docs/source/advanced/port-targets.rst

@@ -0,0 +1,35 @@
+============
+Port targets
+============
+
+In the default case, when a workflow receives files or folders as initial input objects, StreamFlow looks for them in the local file system. Along the same line, whenever a workflow step produces input files or folders, StreamFlow searches them in the location where the step was executed.
+
+However, there are cases in which these assumptions are not valid. To correctly handle these cases, the user can specify port targets in the ``bindings`` list of a workflow. Port targets are similar to step targets described :ref:`here <Binding steps and deployments>`, but bind ports instead of steps.
+
+In particular, a port binding contains a ``port`` directive referring to a specific input/output port in the workflow, a ``target`` directive referring to a deployment entry in the ``deployments`` section of the StreamFlow file, and a (mandatory) ``workdir`` entry identifies the base path where the data should be placed.
+
+Similarly to steps, ports are uniquely identified using a Posix-like path, where the port is mapped to a file, and the related step is mapped to a folder. Consider the following example, which refers to :ref:`this <Write your workflow>` workflow:
+
+.. code-block:: yaml
+
+   version: v1.0
+   workflows:
+     extract-and-compile:
+       type: cwl
+       config:
+         file: main.cwl
+         settings: config.yml
+       bindings:
+         - step: /compile/src
+           target:
+             deployment: hpc-slurm
+             workdir: /archive/home/myuser
+
+   deployments:
+     hpc-slurm:
+       type: slurm
+       config:
+         ...
+
+Here, the ``/compile/src`` path refers to the ``src`` port of the ``/compile`` step. StreamFlow will search for the file the ``src`` port requires directly on the remote ``hpc-slurm`` location in the ``/archive/home/myuser`` path.
+

docs/source/advanced/stacked-locations.rst

@@ -0,0 +1,28 @@
+=================
+Stacked locations
+=================
+
+StreamFlow supports the concept of stacked locations, adhering to the separation of concerns principle. This allows the user to describe complex execution environments, e.g., a :ref:`Singularity container <SingularityConnector>` launched by a :ref:`Slurm queue manager <SlurmConnector>` called through an :ref:`SSH connection <SSHConnector>`.
+
+Users can define stacked locations using the ``wraps`` property in the :ref:`StreamFlow file <Put it all together>`. For example, consider a remote Slurm queue manager that can be contacted by connecting to the login node of an HPC facility using SSH. This is a typical configuration for HPC systems. Then a user can write:
+
+.. code-block:: yaml
+
+   deployments:
+     ssh-hpc:
+       type: ssh
+       config:
+         ...
+     slurm-hpc:
+       type: slurm
+       config:
+         ...
+       wraps: ssh-hpc
+
+.. warning::
+
+   Note that in StreamFlow ``v0.1``, the queue manager connectors (:ref:`Slurm <SlurmConnector>` and :ref:`PBS <PBSConnector>`) are inherited from the :ref:`SSHConnector <SSHConnector>` at the implementation level. Consequently, all the properties needed to open an SSH connection to the HPC login node (e.g., ``hostname``, ``username``, and ``sshKey``) were defined directly in the ``config`` section of the queue manager deployment. This path is still supported by StreamFlow ``v0.2``, but it is deprecated and will be removed in StreamFlow ``v0.3``.
+
+Note that not all deployment types can wrap other locations. Indeed, only connectors extending the :ref:`ConnectorWrapper <ConnectorWrapper>` interface support the ``wraps`` directive. Specifying the ``wraps`` directive on a container type that does not support it will result in an error during StreamFlow initialization. Conversely, if no explicit ``wraps`` directive is specified for a :ref:`ConnectorWrapper <ConnectorWrapper>`, it wraps the :ref:`LocalConnector <LocalConnector>`.
+
+The ``wraps`` directive only supports wrapping a single inner location. However, a single location can be wrapped by multiple deployment definitions. The :ref:`DeploymentManager <DeploymentManager>` component is responsible for guaranteeing the correct order of deployment and undeployment for stacked locations.

docs/source/conf.py

@@ -17,12 +17,13 @@
 
 # -- Project information -----------------------------------------------------
 import importlib
+import os.path
 
 project = 'StreamFlow'
-copyright = '2021, Alpha Research Group, Computer Science Dept., University of Torino'
+copyright = '2023, Alpha Research Group, Computer Science Dept., University of Torino'
 author = 'Iacopo Colonnelli'
-version = '0.1'
-release = '0.1.0'
+version = '0.2'
+release = '0.2.0'
 
 # -- General configuration ---------------------------------------------------
 
@@ -80,6 +81,7 @@ def setup(app):
 }
 
 # JSONSchema extensions
+sjs = importlib.import_module("sphinx-jsonschema")
 sjs_wide_format = importlib.import_module("sphinx-jsonschema.wide_format")
 
 
@@ -208,3 +210,27 @@ def patched_run(self, schema, pointer=''):
 
 original_run = sjs_wide_format.WideFormat.run
 sjs_wide_format.WideFormat.run = patched_run
+
+
+def patched_get_json_data(self):
+    schema, source, pointer = original_get_json_data(self)
+
+    if self.arguments:
+        filename, pointer = self._splitpointer(self.arguments[0])
+    else:
+        filename, pointer = None, ''
+
+    if 'allOf' in schema:
+        for obj in schema['allOf']:
+            if '$ref' in obj:
+                target_file = os.path.join(os.path.dirname(filename), obj['$ref'])
+                target_schema, _ = self.from_file(target_file)
+                target_schema = self.ordered_load(target_schema)
+                schema['properties'] = {**target_schema.get('properties', {}), **schema['properties']}
+        del schema['allOf']
+        schema['properties'] = dict(sorted(schema['properties'].items()))
+    return schema, source, pointer
+
+
+original_get_json_data = sjs.JsonSchema.get_json_data
+sjs.JsonSchema.get_json_data = patched_get_json_data

docs/source/connector/docker-compose.rst

@@ -4,5 +4,5 @@ DockerComposeConnector
 
 The `DockerCompose <https://docs.docker.com/compose/>`_ connector can spawn complex, multi-container environments described in a Docker Compose file locally on the StreamFlow node. The entire set of ``services`` in the Docker Compose file contitutes the unit of deployment, while a single service is the unit of binding. Finally, the single instance of a potentially replicated service is the unit of scheduling.
 
-.. jsonschema:: ../../../streamflow/config/schemas/v1.0/docker-compose.json
+.. jsonschema:: ../../../streamflow/deployment/connector/schemas/docker-compose.json
     :lift_description: true

docs/source/connector/docker.rst

@@ -4,5 +4,5 @@ DockerConnector
 
 The `Docker <https://www.docker.com/>`_ connector can spawn one or more instances of a Docker container locally on the StreamFlow node. The units of deployment and binding for this connector correspond to the set of homogeneous container instances, while the unit of scheduling is the single instance.
 
-.. jsonschema:: ../../../streamflow/config/schemas/v1.0/docker.json
+.. jsonschema:: ../../../streamflow/deployment/connector/schemas/docker.json
     :lift_description: true

docs/source/connector/flux.rst

@@ -2,9 +2,60 @@
 FluxConnector
 =============
 
-The `Flux Framework <https://flux-framework.org/>`_ connector allows running jobs on a cluster with Flux Framework in a High Performance Computing Context. Although Flux can work in a local testing container or a cloud environment and has a Python SDK, to match the design here, we follow suit and use a :ref:`SSHConnection <SSHConnection>` pointing to a login node.
+The `Flux Framework <https://flux-framework.org/>`_ connector allows running jobs on a cluster with Flux Framework in a High Performance Computing Context. Although Flux can work in a local testing container or a cloud environment and has a Python SDK, to match the design here, we follow suit and inherit from the :ref:`QueueManagerConnector <QueueManagerConnector>`. In this way, users can offload jobs to local or remote PBS controllers using the :ref:`stacked locations <Stacked locations>` mechanism. The HPC facility is supposed to be constantly active, reducing the deployment phase to deploy the inner connector (e.g., to create an :ref:`SSHConnection <SSHConnection>` pointing to an HPC login node).
+
+.. warning::
+
+   Note that in StreamFlow ``v0.1``, the ``QueueManagerConnector`` directly inherited from the :ref:`SSHConnector <SSHConnector>` at the implementation level. Consequently, all the properties needed to open an SSH connection to the HPC login node (e.g., ``hostname``, ``username``, and ``sshKey``) were defined directly in the ``QueueManagerConnector``. This path is still supported by StreamFlow ``v0.2``, but it is deprecated and will be removed in StreamFlow ``v0.3``.
+
+Interaction with the Flux scheduler happens through a Bash script with ``#flux`` directives. Users can pass the path of a custom script to the connector using the ``file`` attribute of the :ref:`FluxService <FluxService>` configuration. This file is interpreted as a `Jinja2 <https://jinja.palletsprojects.com/>`_ template and populated at runtime by the connector. Alternatively, users can pass PBS options directly from YAML using the other options of a :ref:`FluxService <FluxService>` object.
+
+As an example, suppose to have a Flux template script called ``batch.sh``, with the following content:
+
+.. code-block:: bash
+
+    #!/bin/bash
+
+    #flux --nodes=1
+    #flux --queue=queue_name
+
+    {{streamflow_command}}
+
+A PBS deployment configuration which uses the ``batch.sh`` file to spawn jobs can be written as follows:
+
+.. code-block:: yaml
+
+   deployments:
+     flux-example:
+       type: pbs
+       config:
+         services:
+           example:
+             file: batch.sh
+
+Alternatively, the same behaviour can be recreated by directly passing options through the YAML configuration, as follows:
+
+.. code-block:: yaml
+
+   deployments:
+     flux-example:
+       type: pbs
+       config:
+         services:
+           example:
+             nodes: 1
+             queue: queue_name
+
+Being passed directly to the ``flux batch`` command line, the YAML options have higher priority than the file-based ones.
+
+.. warning::
+
+    Note that the ``file`` property in the upper configuration level, i.e., outside a ``service`` definition, is still supported in Streamflow ``v0.2``, but it is deprecated and will be removed in StreamFlow ``v0.3``.
 
 For a quick demo or tutorial, see our `example workflow <https://github.com/alpha-unito/streamflow/tree/master/examples/flux>`_.
 
-.. jsonschema:: ../../../streamflow/config/schemas/v1.0/queue_manager.json
+.. jsonschema:: ../../../streamflow/deployment/connector/schemas/flux.json
     :lift_description: true
+    :lift_definitions: true
+    :auto_reference: true
+    :auto_target: true

docs/source/connector/helm3.rst

@@ -4,5 +4,5 @@ Helm3Connector
 
 The `Helm v3 <https://helm.sh/>`_ connector can spawn complex, multi-container environments on a `Kubernetes <https://kubernetes.io/>`_ cluster. The deployment unit is the entire Helm release, while the binding unit is the single container in a ``Pod``. StreamFlow requires each container in a Helm release to have a unique ``name`` attribute, allowing an unambiguous identification. Finally, the scheduling unit is the single instance of a potentially replicated container in a ``ReplicaSet``.
 
-.. jsonschema:: ../../../streamflow/config/schemas/v1.0/helm3.json
+.. jsonschema:: ../../../streamflow/deployment/connector/schemas/helm3.json
     :lift_description: true

docs/source/connector/kubernetes.rst

@@ -0,0 +1,8 @@
+===================
+KubernetesConnector
+===================
+
+The `Kubernetes <https://kubernetes.io/>`_ connector can spawn complex, multi-container environments on a Kubernetes cluster. The deployment unit is a set of Kubernetes YAML files, which are deployed in the order they are written in the ``config`` section and undeployed in the reverse order. The binding unit is the single container in a ``Pod``. StreamFlow requires each container in a Kubernetes namespace to have a unique ``name`` attribute, allowing an unambiguous identification. Finally, the scheduling unit is the single instance of a potentially replicated container in a ``ReplicaSet``.
+
+.. jsonschema:: ../../../streamflow/deployment/connector/schemas/kubernetes.json
+    :lift_description: true

docs/source/connector/occam.rst

@@ -8,5 +8,5 @@ It is different from standard HPC facilities for two main reasons. First, users
 
 This connector allows StreamFlow to offload computation to multi-container environments deployed on the Occam facility. The deployment unit is a multi-container environment deployed on one or more computing nodes. Multi-container environments are described in a YAML file with a syntax similar to the ``service`` section of `Docker Compose <https://docs.docker.com/compose/>`_. Users can pass this file to the connector through the ``file`` parameter. The unit of binding is the single top-level entry in the file, while the scheduling unit is the single container instance.
 
-.. jsonschema:: ../../../streamflow/config/schemas/v1.0/occam.json
+.. jsonschema:: ../../../streamflow/deployment/connector/schemas/occam.json
     :lift_description: true

docs/source/connector/pbs.rst

@@ -2,11 +2,63 @@
 PBSConnector
 =====================
 
-The `PBS <https://www.openpbs.org/>`_ connector allows offloading execution to High-Performance Computing (HPC) facilities orchestrated by the PBS queue manager. The HPC facility is supposed to be constantly active, reducing the deployment phase to creating an :ref:`SSHConnection <SSHConnection>` pointing to its login node.
+The `PBS <https://www.openpbs.org/>`_ connector allows offloading execution to High-Performance Computing (HPC) facilities orchestrated by the PBS queue manager. It extends the :ref:`QueueManagerConnector <QueueManagerConnector>`, which inherits from the :ref:`ConnectorWrapper <ConnectorWrapper>` interface, allowing users to offload jobs to local or remote PBS controllers using the :ref:`stacked locations <Stacked locations>` mechanism. The HPC facility is supposed to be constantly active, reducing the deployment phase to deploy the inner connector (e.g., to create an :ref:`SSHConnection <SSHConnection>` pointing to an HPC login node).
 
-Interaction with the PBS scheduler happens through a Bash script with ``#PBS`` directives. Users can pass the path of a custom script to the connector using the ``file`` attribute. This file is interpreted as a `Jinja2 <https://jinja.palletsprojects.com/>`_ template and populated at runtime by the connector.
+.. warning::
+
+   Note that in StreamFlow ``v0.1``, the ``QueueManagerConnector`` directly inherited from the :ref:`SSHConnector <SSHConnector>` at the implementation level. Consequently, all the properties needed to open an SSH connection to the HPC login node (e.g., ``hostname``, ``username``, and ``sshKey``) were defined directly in the ``QueueManagerConnector``. This path is still supported by StreamFlow ``v0.2``, but it is deprecated and will be removed in StreamFlow ``v0.3``.
+
+Interaction with the PBS scheduler happens through a Bash script with ``#PBS`` directives. Users can pass the path of a custom script to the connector using the ``file`` attribute of the :ref:`PBSService <PBSService>` configuration. This file is interpreted as a `Jinja2 <https://jinja.palletsprojects.com/>`_ template and populated at runtime by the connector. Alternatively, users can pass PBS options directly from YAML using the other options of a :ref:`PBSService <PBSService>` object.
+
+As an example, suppose to have a PBS template script called ``qsub.sh``, with the following content:
+
+.. code-block:: bash
+
+    #!/bin/bash
+
+    #PBS -l nodes=1
+    #PBS -q queue_name
+    #PBS -l mem=1gb
+
+    {{streamflow_command}}
+
+A PBS deployment configuration which uses the ``qsub.sh`` file to spawn jobs can be written as follows:
+
+.. code-block:: yaml
+
+   deployments:
+     pbs-example:
+       type: pbs
+       config:
+         services:
+           example:
+             file: qsub.sh
+
+Alternatively, the same behaviour can be recreated by directly passing options through the YAML configuration, as follows:
+
+.. code-block:: yaml
+
+   deployments:
+     pbs-example:
+       type: pbs
+       config:
+         services:
+           example:
+             destination: queue_name
+             resources:
+               mem: 1gb
+               nodes: 1
+
+Being passed directly to the ``qsub`` command line, the YAML options have higher priority than the file-based ones.
+
+.. warning::
+
+    Note that the ``file`` property in the upper configuration level, i.e., outside a ``service`` definition, is still supported in Streamflow ``v0.2``, but it is deprecated and will be removed in StreamFlow ``v0.3``.
 
 The unit of binding is the entire HPC facility. In contrast, the scheduling unit is a single job placement in the PBS queue. Users can limit the maximum number of concurrently placed jobs by setting the ``maxConcurrentJobs`` parameter.
 
-.. jsonschema:: ../../../streamflow/config/schemas/v1.0/queue_manager.json
-    :lift_description: true
+.. jsonschema:: ../../../streamflow/deployment/connector/schemas/pbs.json
+    :lift_description: true
+    :lift_definitions: true
+    :auto_reference: true
+    :auto_target: true