Merge pull request #15 from fractal-analytics-platform/task_spec

Refactor documentation

docs/build_your_own_fractal_task.md

@@ -1,4 +1,4 @@
-# Build a Fractal task
+# Create a Fractal task
 
 Fractal tasks are the core processing units of to build your workflows. Each Fractal task loads the data from one (or many) OME-Zarr(s) and applies processing to them. Fractal tasks are Linux command line executables. For the purpose of this demo, we will look at the Python implementation. You can think of a Fractal task as a Python function that knows how to process an OME-Zarr image and save the results back into that OME-Zarr image. With a bit of syntax sugar, this becomes a Fractal task you can then run from the web interface. To understand the types of tasks, their API & how they provide information to Fractal server, check out the [V2 Tasks page](https://fractal-analytics-platform.github.io/version_2/tasks/). 
 

docs/version_2/image_list.md → docs/image_list.md

@@ -2,17 +2,19 @@
 layout: default
 ---
 
-While applying a processing workflow to a given dataset, Fractal keeps a list of all the OME-Zarr images it is processing. In this page we describe the concepts of [images](#images) and [filters](#filters) - see also the [examples section](#examples).
+# Image List
+
+While applying a processing workflow to a given dataset, Fractal keeps a list of all the OME-Zarr images it is processing and their metadata. In this page we describe the concepts of [images](#images) and [filters](#filters) - see also the [examples section](#examples).
 
 ## Images
 
 Each entry in the image list is defined by a unique `zarr_url` property (the full path to the OME-Zarr image), and it may also include image types and image attributes.
 
 ### Image types
 
-Image types are boolean properties that allow to split the image list into different sub-lists (e.g. the `is_3D` type for 3D/2D images, or the `illumination_corrected` type for raw/corrected images when illumination correction was not run in-place). Types can be set both by the task manifest (e.g. after an MIP task, the resulting images always have the type `is_3D` set to `False` - see [task-manifest section](#dataset-filters)) as well as from within an individual task (see [task-API/output section](./tasks.md#output-api)).
+Image types are boolean properties that allow to split the image list into different sub-lists (e.g. the `is_3D` type for 3D/2D images, or the `illumination_corrected` type for raw/corrected images when illumination correction was not run in-place). Types can be set both by the task manifest (e.g. after an MIP task, the resulting images always have the type `is_3D` set to `False` - see [task-manifest section](#dataset-filters)) as well as from within an individual task (see [task-API/output section](./tasks_spec.md#output-api)).
 
-Note: whenever applying filters to the image list, the absence of a type corresponds to false by default.
+Note: when applying filters to the image list, the absence of a type corresponds to false by default.
 
 ### Image attributes
 
@@ -23,7 +25,7 @@ Fractal server uses the image list combined with filters (see [below](#dataset-f
 
 ## Filters
 
-Before running a given task, Fractal prepares an appropriate image list by extracting the images that match with a given set of filters (that is, a set of specific values assigned to image types and/or image attributes). Filters can be defined for a dataset and/or for a workflow task. If a specific filter is set both for the dataset and for the workflow task, the workflow-task one takes priority.
+Before running a given task, Fractal prepares an appropriate image list by extracting the images that match with a given set of filters (that is, a set of specific values assigned to image types and/or image attributes). Filters can be defined for a dataset and/or for a workflow task. If a specific filter is set both for the dataset and for the workflow task, the workflow-task filter takes priority.
 
 
 ### Dataset filters
@@ -33,9 +35,9 @@ There are multiple ways a dataset may have a given filter set:
 1. I manually set it, by modifying the dataset `filters` property.
 2. While writing the Fractal manifest for a task package, I include the `output_types` attribute for a given task. These types are automatically included in the dataset filters after the task is run. 
 Examples:
-    * An MIP task would have `output_types = {"is_3D": False}`: from this task onwards, the 2D images are processed (not the raw 3D images).
-    * An illumination-correction task would have `output_types = {"illumination_corrected": True}`: from this task onwards, the registered images are processed (not the raw images).
-4. When writing the code for a specific task, the task output, I can include a `filters` property, for either image attributes and/or types - see the [section on task outputs](./tasks.md#output-api).
+    * An MIP task would set `output_types = {"is_3D": False}` in its output arguments: from this task onwards, the 2D images are processed (not the raw 3D images).
+    * An illumination-correction task would set `output_types = {"illumination_corrected": True}`: from this task onwards, the registered images are processed (not the raw images).
+3. When writing the code for a specific task, the task output can include a `filters` property, for either image attributes and/or types - see the [section on task outputs](./tasks_spec.md#output-api).
 
 Examples:
 
@@ -62,22 +64,20 @@ Examples:
 * The Apply Registration to Image task has `input_types={"registered": False}`, which means it cannot run on images with type `registered=True`.
 
 
-> Note: as part of an [upcoming `fractal-web` update](https://github.com/fractal-analytics-platform/fractal-web/issues/442), it may become possible to see/edit the current filters upon job submission.
-
 
 ## Examples
 
 After running a converter task, I may have an OME-Zarr HCS plate with 2 wells that contain one image each. In this case, the image list has 2 entries and each image has attributes for plate and well, as well as a true/fals `is_3D` type.
 
-![Image List 1](../assets/image_list_x_1_two_wells_two_images.png)
+![Image List 1](assets/image_list_x_1_two_wells_two_images.png)
 
  If I then run an illumination-correction task that does not overwrite its input images, the image list includes the two original images (without the `illumination_corrected` type) and two new ones (with `illumination_corrected` type set to true). Note that this task also sets the dataset type filters to `{"illumination_correction": True}`.
 
-![Image List 2](../assets/image_list_x_2_two_wells_four_images.png)
+![Image List 2](assets/image_list_x_2_two_wells_four_images.png)
 
 If I then run an MIP task, this will act on the two images with `illumination_corrected` set to true, due to the dataset filters. After the task has run, two new images are added to the list (with type `is_3D` set to false).
 
-![Image list 3](../assets/image_list_x_3_two_wells_six_images.png)
+![Image list 3](assets/image_list_x_3_two_wells_six_images.png)
 
 
 Another example is that if I have an OME-Zarr HCS plate with 3 wells and each well has 3 multiplexing acquisition, then the image list includes 9 OME-Zarr images (and those entries should have the acquisition attribute set).

docs/run_fractal.md

@@ -1,21 +1,14 @@
-# Deploy Fractal Server & Fractal Web
+# Deploy Fractal
 
-Fractal runs locally on a laptop (tested both Linux, macOS and Windows with subsystem Linux) or on a Linux server that submits jobs to a SLURM cluster. The [`fractal-server` documentation](https://fractal-analytics-platform.github.io/fractal-server/) describes the preconditions and the different configurations that can be changed.
+Fractal is meant to be deployed to manage workflows on large cluster and currently has support for different modes of running on slurm clusters. It is deployed on Linux servers and also runs on macOS or Windows (by using Windows Subsystem Linux).
+You can run a fully containerized local example that is useful for demos and testing purposes by following the instructions in the [fractal containers repository](https://github.com/fractal-analytics-platform/fractal-containers/tree/main/examples/full-stack) or by following along this walkthrough:
 
-Fractal can be used via a command line client, as well as via a web client. To get started with Fractal, you can follow the setup in the [fractal-demos repository](https://github.com/fractal-analytics-platform/fractal-demos).
-
-Here is a video walk-through for how to set up a local Fractal server:
-
-<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/mEDHh9Kkdmk" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
+<iframe width="560" height="315" src="https://www.youtube.com/embed/g0s0a1F-2K4?si=-nE51acRS1Vnn2Vk" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
 
 <br/>
 
-Once you have a Fractal server running, you can also access it via Fractal web. To do so, set up a Fractal web server as shown here:
+More detailed documentation about the configuration of the different Fractal components can be found in the [`fractal-server` documentation](https://fractal-analytics-platform.github.io/fractal-server/) and the [`fractal web` documentation](https://fractal-analytics-platform.github.io/fractal-web/).
 
-<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/f_HaiOVH-ig" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
+-----
 
-<br/>
-
-If you prefer to follow a written guide, follow the instructions in the [server folder](https://github.com/fractal-analytics-platform/fractal-demos/tree/main/examples/server) to set up Fractal server. Once you have successfully installed and started the Fractal server, you can [install a fractal-client environment](https://github.com/fractal-analytics-platform/fractal-demos/tree/main/examples/00_user_setup) and interact with the Fractal server from there. To do so, follow the [instructions for the 01_cardio_tiny_dataset example](https://github.com/fractal-analytics-platform/fractal-demos/tree/main/examples/01_cardio_tiny_dataset). This also includes a link to a tiny dataset and instructions on how to run a full Fractal workflow on this dataset (which should run in under a minute).
-To set up Fractal web, follow the instructions in the [Fractal web README](https://github.com/fractal-analytics-platform/fractal-web).
-<br/>
+Fractal can also be deployed by manually setting up the server in a Python environment, configuring your own postgres database & setting up Fractal web from source. You can find some helpful material for this in the  [fractal-demos repository](https://github.com/fractal-analytics-platform/fractal-demos) (especially the examples/server section). We also have older video walkthroughs on manual setups available for both the [fractal-server](https://www.youtube.com/watch?v=mEDHh9Kkdmk) as well as [fractal-web](https://www.youtube.com/watch?v=f_HaiOVH-ig).