-
Notifications
You must be signed in to change notification settings - Fork 13
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
This commit updates the StreamFlow documentation to reflect API changes occurred in version `v0.2.x`. Plus, it better structures the documentation, adding more sections and a better description of the extension points.
- Loading branch information
1 parent
96fb7d9
commit 4efa208
Showing
68 changed files
with
2,080 additions
and
626 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
================ | ||
Multiple targets | ||
================ | ||
|
||
StreamFlow lets users to map steps and ports to multiple targets in the :ref:`StreamFlow file <Put it all together>`. A step bound to multiple locations can be scheduled on each of them at runtime. Plus, if a step encloses multiple instances (e.g., in a CWL ``scatter`` operation), they can run on different targets. | ||
|
||
The `filters` directive defines one or more strategies to select a target among the set of available ones (or among a subset) at runtime. By default, all available targets will be evaluated in the order of appearance in the ``target`` directive. | ||
|
||
Users can select a given :ref:`BindingFilter <BindingFilter>` implementation by specifying its name in the ``filters`` directive of a ``binding`` object. If multiple filters are declared, they are applied to the target list in the order of appearance. For example, to evaluate targets in random order at every allocation request, users can specify the following: | ||
|
||
.. code-block:: yaml | ||
workflows: | ||
example: | ||
type: cwl | ||
config: | ||
file: main.cwl | ||
settings: config.yml | ||
bindings: | ||
- step: /compile | ||
target: | ||
- deployment: first-deployment | ||
- deployment: second-deployment | ||
filters: | ||
- shuffle | ||
Conversely, a file or directory port bound to multiple locations can be retrieved from each of them at runtime. StreamFlow will always try to minimize the overhead of data transfers, using local data whenever possible. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,35 @@ | ||
============ | ||
Port targets | ||
============ | ||
|
||
In the default case, when a workflow receives files or folders as initial input objects, StreamFlow looks for them in the local file system. Along the same line, whenever a workflow step produces input files or folders, StreamFlow searches them in the location where the step was executed. | ||
|
||
However, there are cases in which these assumptions are not valid. To correctly handle these cases, the user can specify port targets in the ``bindings`` list of a workflow. Port targets are similar to step targets described :ref:`here <Binding steps and deployments>`, but bind ports instead of steps. | ||
|
||
In particular, a port binding contains a ``port`` directive referring to a specific input/output port in the workflow, a ``target`` directive referring to a deployment entry in the ``deployments`` section of the StreamFlow file, and a (mandatory) ``workdir`` entry identifies the base path where the data should be placed. | ||
|
||
Similarly to steps, ports are uniquely identified using a Posix-like path, where the port is mapped to a file, and the related step is mapped to a folder. Consider the following example, which refers to :ref:`this <Write your workflow>` workflow: | ||
|
||
.. code-block:: yaml | ||
version: v1.0 | ||
workflows: | ||
extract-and-compile: | ||
type: cwl | ||
config: | ||
file: main.cwl | ||
settings: config.yml | ||
bindings: | ||
- step: /compile/src | ||
target: | ||
deployment: hpc-slurm | ||
workdir: /archive/home/myuser | ||
deployments: | ||
hpc-slurm: | ||
type: slurm | ||
config: | ||
... | ||
Here, the ``/compile/src`` path refers to the ``src`` port of the ``/compile`` step. StreamFlow will search for the file the ``src`` port requires directly on the remote ``hpc-slurm`` location in the ``/archive/home/myuser`` path. | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
================= | ||
Stacked locations | ||
================= | ||
|
||
StreamFlow supports the concept of stacked locations, adhering to the separation of concerns principle. This allows the user to describe complex execution environments, e.g., a :ref:`Singularity container <SingularityConnector>` launched by a :ref:`Slurm queue manager <SlurmConnector>` called through an :ref:`SSH connection <SSHConnector>`. | ||
|
||
Users can define stacked locations using the ``wraps`` property in the :ref:`StreamFlow file <Put it all together>`. For example, consider a remote Slurm queue manager that can be contacted by connecting to the login node of an HPC facility using SSH. This is a typical configuration for HPC systems. Then a user can write: | ||
|
||
.. code-block:: yaml | ||
deployments: | ||
ssh-hpc: | ||
type: ssh | ||
config: | ||
... | ||
slurm-hpc: | ||
type: slurm | ||
config: | ||
... | ||
wraps: ssh-hpc | ||
.. warning:: | ||
|
||
Note that in StreamFlow ``v0.1``, the queue manager connectors (:ref:`Slurm <SlurmConnector>` and :ref:`PBS <PBSConnector>`) are inherited from the :ref:`SSHConnector <SSHConnector>` at the implementation level. Consequently, all the properties needed to open an SSH connection to the HPC login node (e.g., ``hostname``, ``username``, and ``sshKey``) were defined directly in the ``config`` section of the queue manager deployment. This path is still supported by StreamFlow ``v0.2``, but it is deprecated and will be removed in StreamFlow ``v0.3``. | ||
|
||
Note that not all deployment types can wrap other locations. Indeed, only connectors extending the :ref:`ConnectorWrapper <ConnectorWrapper>` interface support the ``wraps`` directive. Specifying the ``wraps`` directive on a container type that does not support it will result in an error during StreamFlow initialization. Conversely, if no explicit ``wraps`` directive is specified for a :ref:`ConnectorWrapper <ConnectorWrapper>`, it wraps the :ref:`LocalConnector <LocalConnector>`. | ||
|
||
The ``wraps`` directive only supports wrapping a single inner location. However, a single location can be wrapped by multiple deployment definitions. The :ref:`DeploymentManager <DeploymentManager>` component is responsible for guaranteeing the correct order of deployment and undeployment for stacked locations. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
=================== | ||
KubernetesConnector | ||
=================== | ||
|
||
The `Kubernetes <https://kubernetes.io/>`_ connector can spawn complex, multi-container environments on a Kubernetes cluster. The deployment unit is a set of Kubernetes YAML files, which are deployed in the order they are written in the ``config`` section and undeployed in the reverse order. The binding unit is the single container in a ``Pod``. StreamFlow requires each container in a Kubernetes namespace to have a unique ``name`` attribute, allowing an unambiguous identification. Finally, the scheduling unit is the single instance of a potentially replicated container in a ``ReplicaSet``. | ||
|
||
.. jsonschema:: ../../../streamflow/deployment/connector/schemas/kubernetes.json | ||
:lift_description: true |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.