Add Nextclade v3 alpha; remove Nextclade v1

[ivan-aksamentov]

Description of proposed changes

• Add Nextclade v3 alpha release as nextclade3. We can start upgrading pipelines to v3. The downloaded version is currently hardcoded.
• Freeze Nextclade v2 to the exact version 2.14.0. This is the latest version with major 2 and there will be no more updates. The binaries are called nextclade2 and nextalign2. They are symlinked to nextclade and nextalign respectively, so that this version is still the default.
• Remove Nextclade v1 (binaries nextclade1 and nextalign1) - it is obsolete and it is not adequate scientifically anymore, so if something happens to break missing v1 it should be fixed by upgrading.

TODO after final v3 release:

• replace download link with hardcoded alpha version to the link to latest version
• symlink nextclade to nextclade3, making it default
• either symlink nextalign to nextclade3, or, preferably, consider removing nextalign symlink.

Useful links:

• GitHub release: https://github.com/nextstrain/nextclade/releases/tag/3.0.0-alpha.0
• Slack thread: https://bedfordlab.slack.com/archives/C01LCTT7JNN/p1701151846934669

Related issue(s)

Checklist

• Checks pass

• Binary versions are as expected on PR branch's preview image:

  $ docker run --rm -it docker.io/nextstrain/base:branch-nextclade-v3 nextclade3 --version
  nextclade 3.0.0-alpha.0
  
  $ docker run --rm -it docker.io/nextstrain/base:branch-nextclade-v3 nextclade --version
  nextclade 2.14.0
  
  $ docker run --rm -it docker.io/nextstrain/base:branch-nextclade-v3 nextalign --version
  nextalign 2.14.0
  
  $ docker run --rm -it docker.io/nextstrain/base:branch-nextclade-v3 nextclade2 --version
  nextclade 2.14.0
  
  $ docker run --rm -it docker.io/nextstrain/base:branch-nextclade-v3 nextalign2 --version
  nextalign 2.14.0
  

[ivan-aksamentov]

I had to move v2 downloads from stage 2 (before CACHE_DATE) to stage 3 (after CACHE_DATE) because /builder-scripts/target-triple is no available in stage 2. This was an easy solution, but it might not be the optimal one. These downloads are immutable. Please comment which solution is preferable.

[corneliusroemer]

Looks like a good migration strategy to me, @victorlin is best placed to comment on the Dockerfile details.

One challenge is that the symlinking strategy doesn't work for bioconda, unless we tweak the bioconda recipes to not just add nextclade/nextalign to $PATH but also nextclade2 and then nextclade3 post release.

This would be similar to python2/3, and should work in principle, but there might be constraints in bioconda that prevent use from doing this.

Something like this should work for bioconda: bioconda/bioconda-recipes@master...corneliusroemer-patch-2, there's probably a better way including a link (soft or hard) to not duplicate the file, but I don't know enough conda for that.

Apparently symlinking works: see https://github.com/conda-forge/python-feedstock/blob/07b5f66b338c611a60610283ac52b239038b1b3d/recipe/build_base.sh#L536

[ivan-aksamentov]

@corneliusroemer I don't see why we'd include multiple versions in the conda package. Conda packages are already versioned, so for sticking to v2 you'd just conda install nextclade=2.14.0 (or whatever the conda syntax is). I did not think to include multiple binaries there.

What I am talking about (on Slack) is that I cannot release 3.0.0-alpha.0 to conda without breaking conda install nextclade (latest version) right now.

If you mean that we can release conda package versioned 3.0.0-alpha.0 and it would have nextclade default binary 2.14.0, as well as nextclade3 binary 3.0.0-alpha.0, then I find it misleading. That's too "smart" and unexpected. I would not like anyone to pull this trick on me with the packages I use.

Anyways, this PR is about docker-base and the conda problems are to be solved in the places where it is used, if any. Or these places can just wait the final release.

[corneliusroemer]

@ivan-aksamentov Yes this is not important, but for context, the reason I brought up conda is that we try to keep nextstrain/docker-base and nextstrain/conda-base aligned so that users can switch seamlessly between runtimes.

I don't suggest adding multiple versions into the conda package, just exporting an extra binary name, so that if we add nextclade2 to a workflow, it can still be used with the conda runtime.

IIUC, the point of adding v3 to this docker image is that we can use nextclade3 in workflows (or nextclade2 for those that don't want to upgrade yet).

When we did the last upgrade there was no conda runtime to think about yet. ;)

[victorlin]

Changes seem reasonable. I think eeaedcf can be reworked to leverage caching for v2 - I'll push a change.

[corneliusroemer]

Thanks a lot @victorlin! Did you manage to resolve things? It says you're requesting changes but it's unclear which. Would be great to have that nextclade3 binary to be able to start testing migrations.

[victorlin]

@corneliusroemer thanks for following up. I pushed the requested changes. Will run a quick test now on the preview image then merge when it looks good.

[victorlin]

• Binary versions are as expected on PR branch's preview image:

  $ docker run --rm -it docker.io/nextstrain/base:branch-nextclade-v3 nextclade3 --version
  nextclade 3.0.0-alpha.0
  
  $ docker run --rm -it docker.io/nextstrain/base:branch-nextclade-v3 nextclade --version
  nextclade 2.14.0
  
  $ docker run --rm -it docker.io/nextstrain/base:branch-nextclade-v3 nextalign --version
  nextalign 2.14.0
  
  $ docker run --rm -it docker.io/nextstrain/base:branch-nextclade-v3 nextclade2 --version
  nextclade 2.14.0
  
  $ docker run --rm -it docker.io/nextstrain/base:branch-nextclade-v3 nextalign2 --version
  nextalign 2.14.0
  

[victorlin]

@corneliusroemer nextclade3 is available as of docker.io/nextstrain/base:build-20231130T175113Z.

[corneliusroemer]

I already tested the image using the branch tag, nextclade worked as before, that is without 2 it was 2.

So now we can transition to 3, as long as we use v3, but should add version checks in snakemake to warn conda users.