Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ingest nextclade updates #54

Merged
merged 3 commits into from
Jul 16, 2024
Merged

Ingest nextclade updates #54

merged 3 commits into from
Jul 16, 2024

Conversation

joverlee521
Copy link
Contributor

@joverlee521 joverlee521 commented Jul 13, 2024

Description of proposed changes

Various small fixes for the Nextclade rules of the ingest workflow.
See commits for details.

Related issue(s)

Resolves #47
Related to https://github.com/nextstrain/private/issues/97

Checklist

  • Checks pass

I guess I've never tested running this path of the ingest workflow.
Error showed up when I tried to run the workflow with the Nextclade
config.

```
nextstrain build ingest results/metadata.tsv --config ncbi_taxon_id='64320' --configfile defaults/nextclade_config.yaml --forceall -n
Config file defaults/config.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
MissingInputException in rule join_metadata_and_nextclade in file /nextstrain/build/ingest/rules/nextclade.smk, line 64:
Missing input files for rule join_metadata_and_nextclade:
    output: results/metadata.tsv
    affected files:
        config/nextclade_field_map.tsv
```
This was originally copied from mpox and `lineage` is a very mpox
specific field that is not usually included in the Nextclade output TSV.
This was written before Nextclade v3 was officially released.
Nextclade v3 been out for >6 months now and all new Nextclade datasets
are expected to be v3.
@joverlee521
Copy link
Contributor Author

joverlee521 commented Jul 15, 2024

I was running into a weird error when testing this PR with mpox:

$ nextstrain build ingest --config ncbi_taxon_id=10244 --configfile defaults/nextclade_config.yaml
[...]
[INFO] read key-value file: defaults/nextclade_field_map.tsv
[INFO] 12 pairs of key-value loaded
[ERRO] bufio.Scanner: token too long
[ERRO] record on line 440: wrong number of fields
[WARN] csvtk rename2: skipping empty input file: -
[WARN] csvtk csv2tab: skipping empty input file: -
[tsv-join] Error processing command line arguments: [--k|key-fields] Field not found in file header: 'seqName'.
[tsv-select] Error processing command line arguments: [--e|exclude] Field not found in file header: 'seqName'.

Turns out the root error is coming from csvtk fix-quotes

nextstrain shell .
csvtk fix-quotes -Ht ingest/results/nextclade.tsv > ingest/data/fix-quotes.tsv 
[ERRO] bufio.Scanner: token too long

I'm going to drop dc9ac25 for now and follow up in a separate change -> see

@joverlee521 joverlee521 merged commit 89b3c5d into main Jul 16, 2024
1 check passed
@joverlee521 joverlee521 deleted the ingest-nextclade-updates branch July 16, 2024 17:12
@corneliusroemer
Copy link
Member

corneliusroemer commented Jul 17, 2024

@joverlee521 Oh no, so fix-quotes, which is a workaround for csvtk's lack of TSV support has its own quirks? This strengthens my opinion that we should just never use csvtk for TSVs 🙃

I see that this is exactly what you're doing in #55

@joverlee521 joverlee521 mentioned this pull request Oct 17, 2024
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Updates to use of Nextclade in ingest workflow
3 participants