Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .docusaurus_site/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,8 @@ module.exports = {
"azure",
"google",
"kubernetes",
"fusion"
"fusion",
"seqera-filesystem"
]
},
{
Expand Down
3 changes: 3 additions & 0 deletions docs/migrations/26-04.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -396,6 +396,8 @@ workflow {

A Platform access token with appropriate permissions is required to download non-public datasets. It can be specified using the `TOWER_ACCESS_TOKEN` environment variable or the `tower.accessToken` config option.

See [Seqera file system][seqera-filesystem-page] for details.

## Breaking changes

- The [strict syntax parser][strict-syntax-page] is now enabled by default. The legacy parser can be enabled by setting the `NXF_SYNTAX_PARSER` environment variable to `v1`.
Expand Down Expand Up @@ -439,6 +441,7 @@ A Platform access token with appropriate permissions is required to download non
[process-hints]: ../reference/process#hints
[process-typed-page]: ../process-typed
[script-records]: ../script#records
[seqera-filesystem-page]: ../seqera-filesystem
[static-typing-first-preview]: ./25-10#static-typing-preview
[stdlib-listDirectory]: ../reference/stdlib-types#listdirectory---iterablepath
[strict-syntax-page]: ../strict-syntax
Expand Down
2 changes: 0 additions & 2 deletions docs/reference/config.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1784,8 +1784,6 @@ The `seqera.executor` scope configures the Seqera scheduler service for the [Seq

The following settings are available:

The following settings are available:

###### `seqera.executor.autoLabels`

When `true`, automatically adds workflow metadata labels to the session with the `nextflow.io/` prefix (default: `false`). The following labels are added: `projectName`, `userName`, `runName`, `sessionId`, `resume`, `revision`, `commitId`, `repository`, `manifestName`, `runtimeVersion`. A `seqera.io/runId` label is also added, computed as a SipHash of the session ID and run name.
Expand Down
130 changes: 130 additions & 0 deletions docs/seqera-filesystem.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
---
title: Seqera file system
description: Read Seqera Platform datasets and data links in Nextflow pipelines using the seqera:// URI scheme.
---

<AddedInVersion version="26.04" />

Nextflow can read Seqera Platform datasets and data links using the `seqera` URI scheme.
Use a `seqera://` path anywhere Nextflow reads an input file.
Your pipeline then reads data registered in a Seqera Platform workspace without hard-coding the storage location or the credentials.

The `nf-tower` plugin, bundled with Nextflow, provides the Seqera file system.
The file system is read-only. A `seqera://` path cannot be a work directory, an output directory, or a publish target.

See [Working with files][working-with-files] and the [Path][stdlib-types-path] reference for the available file operations.

## Authentication

Reading a `seqera://` path requires a [Seqera Platform](https://seqera.io) access token with permission to read the target datasets or data links. See [Authentication](https://docs.seqera.io/platform-cloud/api/overview#authentication) in the Seqera Platform documentation to create one.

Provide the token using the `TOWER_ACCESS_TOKEN` environment variable:

```bash
export TOWER_ACCESS_TOKEN='<PLATFORM_ACCESS_TOKEN>'
```

Alternatively, specify the token in your configuration with the [`tower.accessToken`][config-tower-accessToken] setting, along with the plugin declaration:

```nextflow
plugins {
id 'nf-tower'
}

tower.accessToken = '<PLATFORM_ACCESS_TOKEN>'
```

Nextflow loads the `nf-tower` plugin automatically when the `TOWER_ACCESS_TOKEN` environment variable is set, or when [`tower.enabled`][config-tower-enabled] or [`fusion.enabled`][config-fusion-enabled] is `true`. Otherwise, declare the plugin explicitly. Without it, Nextflow does not register the `seqera` scheme.

:::note
Setting `tower.enabled = true` also registers the scheme, but it turns on run monitoring with Seqera Platform. To read data without reporting your runs, declare the plugin instead.
:::

For Seqera Platform Enterprise, set the API endpoint of your instance with the [`tower.endpoint`][config-tower-endpoint] setting or the `TOWER_API_ENDPOINT` environment variable.

## Dataset paths

A [dataset](https://docs.seqera.io/platform-cloud/data/datasets) is a versioned CSV or TSV file stored in a Seqera Platform workspace, typically a samplesheet. Dataset paths take the following form:

```
seqera://<organization>/<workspace>/datasets/<name>[@<version>]
```

For example, the following pipeline reads a samplesheet from the `showcase` workspace of the `seqeralabs` organization:

```nextflow
params.dataset = 'seqera://seqeralabs/showcase/datasets/sarek_samples'

workflow {
channel.fromPath(params.dataset)
| splitCsv(header: true)
| view
}
```

Omit the version to read the latest enabled version of the dataset. Append `@<version>` to pin a specific version. A pinned path keeps a run reproducible after someone uploads a new version:

```nextflow
params.dataset = 'seqera://seqeralabs/showcase/datasets/sarek_samples@2'
```

## Data link paths

<AddedInVersion version="26.05.0-edge" />

A [data link](https://docs.seqera.io/platform-cloud/data/data-explorer) is a cloud storage bucket or container registered in a Seqera Platform workspace, along with the credentials that grant access to it. Data link paths take the following form:

```
seqera://<organization>/<workspace>/data-links/<provider>/<name>/<path>
```

The segments are:

- `<provider>`: the cloud provider of the data link, such as `aws`, `azure`, or `google`.
- `<name>`: the name of the data link in the workspace, as shown in **Data Explorer**. This is not the name of the underlying bucket or container.
- `<path>`: the path of the file or directory within the data link.

For example, the following pipeline reads a file from the `inputs` data link:

```nextflow
workflow {
println file('seqera://acme/research/data-links/aws/inputs/data/sequences.fa').text
}
```

Because the data link supplies the credentials, your pipeline does not need its own cloud credentials to read the data.

:::note
To write pipeline outputs to the storage behind a data link, use the native path of the bucket or container, such as `s3://my-bucket/results`.
:::

## List available data

Every level of a `seqera://` path is a directory. Use [`listDirectory()`][stdlib-listDirectory] to see the organizations, workspaces, datasets, and data links that your access token can read:

```nextflow
workflow {
// organizations
file('seqera://').listDirectory().each { println it }

// workspaces in an organization
file('seqera://acme').listDirectory().each { println it }

// datasets in a workspace
file('seqera://acme/research/datasets').listDirectory().each { println it }

// providers with data links in a workspace
file('seqera://acme/research/data-links').listDirectory().each { println it }

// data links for a provider
file('seqera://acme/research/data-links/aws').listDirectory().each { println it }
}
```

[config-fusion-enabled]: ./reference/config#fusionenabled
[config-tower-accessToken]: ./reference/config#toweraccesstoken
[config-tower-enabled]: ./reference/config#towerenabled
[config-tower-endpoint]: ./reference/config#towerendpoint
[stdlib-listDirectory]: ./reference/stdlib-types#listdirectory---iterablepath
[stdlib-types-path]: ./reference/stdlib-types#path
[working-with-files]: ./working-with-files
Loading