Skip to content

Commit

Permalink
Update docs.
Browse files Browse the repository at this point in the history
  • Loading branch information
joaander committed May 13, 2024
1 parent 5d247b9 commit 26d64da
Show file tree
Hide file tree
Showing 9 changed files with 108 additions and 16 deletions.
7 changes: 4 additions & 3 deletions doc/src/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,9 @@
- [Directory status](guide/concepts/status.md)
- [JSON pointers](guide/concepts/json-pointers.md)
- [Cache files](guide/concepts/cache.md)
- [Advanced topics]()
- [Submit the same action to different groups/resources]()
- [How-to topics](guide/howto/index.md)
- [Set the cluster account](guide/howto/account.md)
- [Submit the same action to different groups/resources](guide/howto/same.md)
- [Use an action to summarize many directories]()

# Reference
Expand All @@ -38,13 +39,13 @@
- [show launchers](row/show/launchers.md)
- [scan](row/scan.md)
- [clean](row/clean.md)

- [`workflow.toml`](workflow/index.md)
- [workspace](workflow/workspace.md)
- [action](workflow/action/index.md)
- [group](workflow/action/group.md)
- [resources](workflow/action/resources.md)
- [submit_options](workflow/action/submit-options.md)
- [default](workflow/default.md)
- [`clusters.toml`](clusters/index.md)
- [cluster](clusters/cluster.md)
- [Built-in clusters](clusters/built-in.md)
Expand Down
21 changes: 21 additions & 0 deletions doc/src/guide/howto/account.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Set the cluster account

Use the default action to conveniently set the account (or accounts) once in your
`workflow.toml`. It will apply to all actions that do not override the account

```toml
[default.action.submit_options.cluster1]
account "cluster1-account"
[default.action.submit_options.cluster2]
account "cluster2-account"

[[action]]
# Will use the defaults above.

[[action]]
# Will use the defaults above.

[[action]]
submit_options.cluster1.account = "alternate-account"
# Will use the "alternate-account" on cluster1 and "cluster2-account" on cluster2.
```
4 changes: 4 additions & 0 deletions doc/src/guide/howto/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# How-to

This section shows how to accomplish various tasks in Row that are not covered in the
tutorial.
31 changes: 31 additions & 0 deletions doc/src/guide/howto/same.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Submit the same action to different groups/resources

You can submit the same action to different groups and resources. To do so,
create multiple elements in the action *array with the same name*. Each must use
[`group.include`](../../workflow/action/group.md#include) to select *non-overlapping
subsets*. You can use [`action.from`](../../workflow/action/index.md#from) to copy all
fields from one action and selectively override others.

For example, this `workflow.toml` uses 4 processors on small systems and 8 on large
ones.

```toml
[default.action]
command = "python actions.py --action $ACTION_NAME {directories}"

[[action]]
name = "compute"
products = ["results.out"]
[action.resources]
walltime.per_submission = "12:00:00"
processes.per_directory = 4
[action.group]
include = [["/N", "<=", "4096"]]
maximum_size = 32

[[action]]
from = "compute"
resources.processes.per_directory = 8
group.include = [["/N", ">", "4096"]]
group.maximum_size = 16
```
21 changes: 10 additions & 11 deletions doc/src/guide/python/actions.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,21 +69,21 @@ Next, replace the contents of `workflow.toml` with the corresponding workflow:
```

You should be familiar with all of these options from previous tutorials. The main point
of interest here is that *both* actions have the same **command**:
of interest here is that *both* actions have the same **command**, set once by the
[**default action**](../../workflow/default.md):
```toml
{{#include signac-workflow.toml:6}}
{{#include signac-workflow.toml:5}}
```

`python actions.py` executes the `actions.py` file above. It is given the argument
`--action $ACTION_NAME` which selects the Python function to call. Here `$ACTION_NAME`
is an [environment variable](../../env.md) that **row** sets in job scripts. You could
type the action name explicitly here, but then you should take extra care when copying
and pasting commands to avoid executing the wrong action! The last arguments are given
by `{directories}`. Unlike `{directory}` shown in previous tutorials, `{directories}`
expands to *ALL* directories in the submitted **group**. In this way, `action.py` is
executed only once and is free to process the list of directories in any way it chooses
(e.g. in serial, with multiprocessing parallelism, using MPI parallelism, multiple
threads, ...).
is an [environment variable](../../env.md) that **row** sets in job scripts. The
last arguments are given by `{directories}`. Unlike `{directory}` shown in previous
tutorials, `{directories}` expands to *ALL* directories in the submitted **group**.
In this way, `action.py` is executed only once and is free to process the list of
directories in any way it chooses (e.g. in serial, with
[multiprocessing parallelism, multiple threads](../concepts/thread-parallelism.md),
using [MPI parallelism](../concepts/process-parallelism.md), ...).

## Execute the workflow

Expand Down Expand Up @@ -121,7 +121,6 @@ Then add:
```toml
[[action]]
name = "action"
command = "python actions.py --action $ACTION_NAME {directories}"
# And other relevant options
```
to your `workflow.toml` file.
Expand Down
5 changes: 3 additions & 2 deletions doc/src/guide/python/signac-workflow.toml
Original file line number Diff line number Diff line change
@@ -1,14 +1,15 @@
[workspace]
value_file = "signac_statepoint.json"

[default.action]
command = "python actions.py --action $ACTION_NAME {directories}"

[[action]]
name = "square"
command = "python actions.py --action $ACTION_NAME {directories}"
products = ["square.out"]
resources.walltime.per_directory = "00:00:01"

[[action]]
name = "compute_sum"
command = "python actions.py --action $ACTION_NAME {directories}"
previous_actions = ["square"]
resources.walltime.per_directory = "00:00:01"
31 changes: 31 additions & 0 deletions doc/src/workflow/default.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# default

The `default` table sets default options.

Example:

```toml
[default.action.submit_options.cluster1]
account = "my_account"
```

## action

`default.action`: **table** - accepts *any* key that is valid in
an [action array element](action/index.md). When an action array element omits a key,
the default key is used. When both the action **and** the default action omit a key,
the individually documented "when omitted" behavior takes effect.

> Note: This rule applies to all sub-keys as well. For example:
> ```toml
> [default.action.resources]
> processes.per_submission = 8
> walltime.per_directory = "02:00:00"
>
> [[action]]
> name = "action"
> command = "command {directory}"
> resources.processes.per_submission = 16
> ```
> Will result in an action that sets `processes.per_submission == 16` and
> `walltime.per_directory == "02:00:00"`.
2 changes: 2 additions & 0 deletions src/cli/directories.rs
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,8 @@ pub fn directories<W: Write>(
let query_directories =
cli::parse_directories(args.directories, || Ok(project.state().list_directories()))?;

// TODO: handle multiple actions with the same name

let action = project
.workflow()
.action_by_name(&args.action)
Expand Down
2 changes: 2 additions & 0 deletions src/cli/submit.rs
Original file line number Diff line number Diff line change
Expand Up @@ -225,6 +225,8 @@ pub fn submit<W: Write>(
flag::register(SIGTERM, Arc::clone(&should_terminate))?;
let instant = Instant::now();

// TODO: Error on multiple submissions

for (index, (action, directories)) in action_directories.iter().enumerate() {
let scheduler = project.scheduler();
let mut message = format!(
Expand Down

0 comments on commit 26d64da

Please sign in to comment.