Allow the concurrent run of multiple pipeline revisions

[pditommaso]

Summary

Nextflow relies on built-in integration with Git to pull and run a workflow.

When the user specifies the Git repository URL on then run command line, Nextflow carry out a Git clone command, stores the pipeline code into the $HOME/.nextflow/assets directory and launch the execution from there.

When the user specifies the -r (revision) CLI option, the repository is checked out at the specified revision ie. branch, tag or even commit id.

This however poses a problem when if two or more users run different versions at the same time, because the last performing the operation would override the previous repository code, which could be a disruptive operation.

This is not such an unlikely event considering a pipeline execution can last for hours or even days.

To mitigate this problem nextflow refuses to perform a run if the project is currently checkout to a non-default version and the run does not specify the revision to be executed in an explicit manner. However, this is the cause of other unexpected side effects. See here.

Goal

The goal of this enhancement is to allow the concurrent use of multiple pipeline revision in the same computer and deprecated the need for the stick revision check.

This could be achieved by downloading the Git repository with bare clone instead of a normal clone, and checkout the work tree into a separate subdirectory named as the commit id associated with the specified revision.

For example, if the user runs

nextflow run https://github.com/nextflow-io/hello

nextflow should clone the repo above with the bare option and store in the path $HOME/.nextflow/assets/nextflow-io/hello.git

Then implicitly the default branch is checkout, therefore the associate commit should be retrieved e.g. 4eab81bd42eed592f4371cd91b755ec78df25fe9, therefore the following path should be created containing the work tree accessible for the execution

$HOME/.nextflow/assets/nextflow-io/hello.git/.nextflow/revs/4eab81bd42eed592f4371cd91b755ec78df25fe9

When the user-specified a different revision e.g.

nextflow run https://github.com/nextflow-io/hello -r dev

A new subdirectory with the corresponding commit id should be created.

The commit id should be resolved against the local git clone, unless the -latest option is specified.

[jorgeaguileraseqera]

@pditommaso one question:

do we need to check if the repo is present in the asset directory with the "old" format (no bare) and in this case no use the bare feature? i.e. some kind of retro compatibility or we'll force to remove and recreate local repos

[pditommaso]

Good point. If already exists think should report a warning message maybe?

[jorgeaguileraseqera]

so, report with a warning message (maybe with some instructions to remove the current repo) and stop the command, right?

[pditommaso]

No I mean, show a warning message i.e. log.warn and do not stop. Usually nextflow only stops on error.

[jorgeaguileraseqera]

ah ok, so if I understand correctly we try to identify which kind of repo we are working on at startup

• if it's a working git (no bare) we use it as usual (with a warning message)
• if it's a bare git we "activate" this feature

[pditommaso]

I see your point. In principle the bare should have been created when the feature has been enabled with a config flag or env variable, right?

If so, I think when this option does not match the repo format a warning should be reported

[pditommaso]

@jorgeaguileraseqera any ETA for this?

[jorgeaguileraseqera]

Hope to have in these days

(it's a little tedious due to the API rate limit breaks sometimes to run all tests )

[pditommaso]

Can you please open at least a draft PR asap?

[pditommaso]

due to the API rate limit breaks sometimes to run all tests

Do you mean Github rate limits? Are you using your GITHUB_TOKEN for tests?

[jorgeaguileraseqera]

yes, I've created one and configured the env to run the tests