Add tutorials for ingest pipelines

[joverlee521]

Initially proposed by @trvrb in our meeting with WA DOH on 2024-01-17.
The tentative due date for this objective is 2024-02-29.

We should have tutorials for running and creating ingest pipelines from NCBI data in parallel with the Running a pathogen workflow and Creating a pathogen workflow tutorials.

The ultimate goal would be for an external user to be able to follow the tutorials to set up their own ingest pipeline for any viral pathogen using NCBI data.

[joverlee521]

As @kimandrews works through setting up ingest for measles, we can work on putting together a step by step tutorial for creating the ingest workflow.

[joverlee521]

Jotting down some general thoughts to keep track of from conversations with @kimandrews

• who is our intended audience? What is the expected level of experience with Nextstrain?
• will need guide for installing git subrepo for managing the ingest/vendored scripts
• document commands to get the first fetch from NCBI to give users a view of the raw data
• document commands to run full ingest pipeline to see curated output
• provide guidance on how to assess data to make specific config changes
• pointing to an existing ingest workflow is really helpful to understanding
• need pathogen specific knowledge (e.g. measles strain name schema)

[tsibley]

• will need guide for installing git subrepo for managing the ingest/vendored scripts

How about we vendor git subrepo so that our vendored stuff is bootstrapped?

I know the installation docs for git subrepo have you clone its repo and source some shell init, but taking a look deeper there's only three files we strictly need (git-subrepo, help-functions.bash, and bash+.bash) and it'd be pretty easy to vendor just those or even pack them into a single file (I just did it).

(Hell, we could git subrepo the git-subrepo repo into ingest/vendored/, but I worry I might not be welcome anymore if I seriously suggest that…)

(…the other bad suggestion is that git subtree doesn't have this issue either…)

Bootstrapping doesn't solve the git subrepo usage hurdles, and I know the longer term solution is ditching vendoring entirely in favor of augur curate et al., but it would solve the installation hurdle.

[victorlin]

…the other bad suggestion is that git subtree doesn't have this issue either…

I don't think it's a bad suggestion. Wrote more in nextstrain/ingest#3 (comment).

[tsibley]

"bad" because I'm not convinced its worth it to re-open the decision there and switch stuff over. Maybe it is!

[joverlee521]

Another option is to bypass setting up git subrepo in the tutorial and just have people use the vendored scripts in the pathogen-repo-guide.

The ingest scripts will not be updated that frequently so only mention the need for using git subrepo if the user wants to update the vendored scripts.