Repository navigation
Conversation
Removed 'conda' from the profile options in the workflow.
modules PR #247 is merged. The two subworkflow copies in this repo were carrying the padding edits as uncommitted working-tree changes against stale modules.json pins, so 'nf-core subworkflows update' would have silently reverted them. Both files are byte-identical to origin/develop, so this is a sync, not a reconstruction: generate_mutated_peptides cd48156 (neoantigen) -> 262bc0f (develop) netmhcstabandpan 42c230a (develop) -> f8e7239 (develop) The two are coupled: generate_mutated_peptides pre-pads SV-less samples with [meta, []] so netmhcstabandpan can use a plain inner join instead of join(remainder: true), which previously buffered every sample until the slowest GENERATEMUTFASTA in the cohort finished.
All five configs pointed at .../test-datasets/neoantigen/... where the 'neoantigen' segment is a mutable BRANCH ref, so the IEDB reference set could change under a running analysis. R, and therefore every neoantigen quality score, is only reproducible against a frozen IEDB snapshot. Pinned to 0617e432a0d048bc45fd3998d07b27a4c036e9b5 (current branch tip). Verified score-neutral: the pinned and branch URLs return byte-identical files, sha1 0ab27d04a5249d22f900612bd68b480e8f308f37, 5080 epitopes. Air-gapped nodes still cannot run: workflows/neoantigenpipeline.nf:124 does a bare file(params.iedbfasta) with no offline fallback. Vendoring the file to the local resource dirs already used by prod/juno is the follow-up; not done here because those paths could not be verified from outside the cluster and a wrong path breaks prod.
Two independent ceilings were cutting multievolve short:
1. conf/juno.config resourceLimits clamped time at 100.h (~4d4h).
2. Even unclamped, the process_high default 72.h * task.attempt tops
out at 216.h = 9 days across all three retries.
Observed production runtimes span 5 to 14 days, and one 22k-mutation
run was TIMEOUT-killed at 14d 14h while 99% converged -- so raising the
clamp alone would not have saved it.
Give the process its own escalation (120/240/360.h over three attempts)
and raise the juno clamp to 360.h so attempt 3 is not cut.
Only 'time' is set. ext.args is deliberately untouched: it IS part of
the task hash, so editing it would invalidate every stored PhyloWGS
result and force the ~4,122-pair MCMC to recompute. time/cpus/memory
are not hashed. Verified via 'nextflow config -profile juno' that
ext.args is unchanged.
…pport Pull in module updates currently open/merged upstream in mskcc-omics-workflows/modules: - generatehlastring: pin to develop (modules PR #256, merged) — fixes 3-digit HLA-allele field truncation, adds HLA-HD support, sources generateHLAString.sh from the neoantigen-utils pip package instead of a bundled copy. - generatemutfasta, mutalyzer/retriever, neoantigenediting/aligntoiedb, neoantigenediting/computefitness, neoantigenutils/convertannotjson, neoantigenutils/formatnetmhcpan, neoantigenutils/neoantigeninput, neosv: pin to feature/update_base_image_and_command (modules PR #258) — bumps neoantigen-utils-base 1.4.0 -> 1.6.1 and drops bundled versioning/CLI boilerplate now owned by the shared package. - phylowgs/multievolve: pin to feature/seperate_phylowgs_with_resume_support (modules PR #254) — checkpoints chains/ under workDir/.scratch so a walltime/attempt failure resumes the MCMC instead of restarting from scratch. PRs #253 and #257 are superseded by #254 and #258 respectively and are intentionally not synced. Not run: full nextflow test-profile execution (local Docker on this Mac can't collect task metrics for the arm64-emulated amd64 images — an environment limitation, not a module issue). nf-test module-level tests are excluded from this repo's suite by nf-test.config and are covered by the modules repo's own CI on each PR.
…anifest The plain neoantigen-utils-base:1.6.1 tag is missing the unzip binary that this module shells out to (see containers commit 3d2a0ad, which added it). create-ghcr-manifest -- the CI step that re-stitches the per-arch :1.6.1-amd64/-arm64 images into the plain :1.6.1 tag -- was skipped in both the original and rerun of containers workflow run 32375210816, because an unrelated gbcms push (permission_denied: read_package -- the gbcms GHCR package isn't linked for this repo's Actions token) failed in the same matrix and blocked the downstream manifest job for every image, including this one. The :1.6.1-amd64 tag itself built and pushed fine (confirmed: pulled it directly, unzip is present, dated after the fix). Pointing at it directly unblocks iris runs without waiting on the gbcms fix. iris.config already pins process.arch to linux/x86_64, so this isn't a platform regression -- it's the same image iris would resolve to via the manifest anyway. Revert to the bare :1.6.1 tag once mskcc-omics-workflows/containers re-creates the manifest (either by someone with GHCR org admin manually running 'docker buildx imagetools create', or once the gbcms package-linking issue is fixed and the workflow rerun goes green).
generate_input.py's ensembl_load() calls pyensembl's Genome.index(), which defaults its cache directory to $HOME (via datacache) when no explicit cache_directory_path is passed. $HOME is read-only on iris compute nodes, so indexing fails: OSError: [Errno 30] Read-only file system: '/home/orgeraj' pyensembl already supports overriding this via the PYENSEMBL_CACHE_DIR env var (pyensembl/download_cache.py), so export it to the task's own work dir before calling generate_input.py, rather than $HOME. Done as a plain export in the script block instead of a config-level beforeScript override -- iris.config already sets beforeScript at the process scope (SINGULARITY_TMPDIR setup), and a withName override would have replaced it wholesale instead of composing with it.
…bl index Lets a shared, persistent directory be used for pyensembl's GTF/cDNA index instead of rebuilding it fresh in every task's work dir. Build it once (e.g. via a single non-concurrent test run pointed at the shared path), then point subsequent multi-sample runs at the same directory so every task just reads the already-built index. Not safe to point multiple *first-time* builds at the same directory concurrently -- datacache's Database.create() opens a plain sqlite3 connection on the final cache path and runs CREATE TABLE/INSERT with no locking or atomic rename, so concurrent builders racing to create the same db can hit 'database is locked' or a corrupt/incomplete file. Once a build has finished (metadata table written), concurrent reads of the finished db are safe. Defaults to '' (falls back to the existing task-local $PWD/.pyensembl_cache behavior) so this doesn't change default pipeline behavior.
…anifest Same root cause as the neoantigen-utils-base:1.6.1 unzip issue: the containers repo's neoantigen-editing:1.1 hotfix (6e233b4, Aug 18) adds /usr/bin/NeoantigenEditing to PATH and chmod +x's the bundled align_neoantigens_to_IEDB.py / compute_fitness.py scripts -- needed since modules#258 removed these modules' local resources/usr/bin/ copies in favor of the ones baked into this image. The per-arch :1.1-amd64/-arm64 tags built correctly on Aug 20 with the fix, but create-ghcr-manifest was skipped for the whole matrix (same unrelated gbcms push failure), so the plain :1.1 tag people actually pull is still the pre-fix Aug 13 image: .command.sh: line 3: align_neoantigens_to_IEDB.py: command not found Confirmed via direct pull: :1.1 lacks the fix, :1.1-amd64 has it (both scripts resolve on PATH). iris.config already pins process.arch to linux/x86_64, so amd64-only isn't a platform regression here. Revert to the bare :1.1 tag once mskcc-omics-workflows/containers re-creates the manifest.
…in script's CLI align_neoantigens_to_IEDB.py inside neoantigen-editing:1.1 (installed via modules#258's switch from a module-local resource script to the image's bundled copy) takes --fasta plus exactly one of --sample_file (a single patient JSON) or --patient_folder (a directory of them) -- there is no --input flag: align_neoantigens_to_IEDB: error: unrecognized arguments: --input ... This module always passes a single patient_data file, so --sample_file is the correct flag. Confirmed by reading the script's argparse block directly out of the neoantigen-editing:1.1-amd64 image; output still writes to iedb_alignments_<patient>.txt in cwd either way, matching this module's existing output glob -- no other change needed.
…mpute_fitness_kd_cutoff modules#258 switched NEOANTIGENEDITING_COMPUTEFITNESS from a module-local resources/usr/bin/compute_fitness.py to the copy baked into neoantigen-editing:1.1, which has a materially different CLI (confirmed by reading both versions directly, old from modules repo git history at 70f2c5d, new from the neoantigen-editing:1.1-amd64 image): - --input -> --sample_file/--patient_folder (same rename as the aligntoiedb fix). - --a_param/--k_param/--w_param no longer exist -- a/k/w are now hardcoded in the script to the same values as this pipeline's current compute_fitness_a/k/w defaults, so today's scoring is unaffected, but those params are now a no-op: passing the old flags errors, and changing the params has no effect until the script regains them. Documented in nextflow_schema.json rather than silently dropping the params, since restoring the flags upstream is a plausible follow-up. - A new required --kd_cutoff_fitness controls which neoantigens are eligible to be attached to a clone for fitness purposes at all (previously unconditional) -- distinct from params.kd_cutoff, which filters weak binders earlier at NEOANTIGENUTILS_NEOANTIGENINPUT. Added as its own params.compute_fitness_kd_cutoff per team decision (kept separate from kd_cutoff, not reusing it), deliberately with no default -- must be supplied explicitly via --compute_fitness_kd_cutoff, or compute_fitness.py fails loudly on an invalid float value rather than silently running with a guessed cutoff.
MafParser._parse_maf fed the MAF straight to csv.DictReader, which
takes line 1 as the header with no allowance for comment/metadata
lines. GenomeNexus's own MAF writer prepends two such lines
('#genome_nexus_version: 2.2.2', '#isoform: mskcc') before the real
tab-separated header, so DictReader treated the first comment line as
the header and every column lookup below (single_line["Chromosome"],
etc.) raised KeyError: 'Chromosome' on real GenomeNexus/VEP111-
annotated input:
File "create_phylowgs_inputs.py", line 328, in _parse_maf
chrom = single_line["Chromosome"]
KeyError: 'Chromosome'
Not specific to one bad file -- any GenomeNexus-annotated MAF hits
this the same way, which is exactly the annotation step this
production samplesheet's MAFs went through
(*.genomenexus.vep111.maf).
Fix: peek lines with readline()/tell(), skip any that start with '#',
then seek back to the first non-comment line before handing the file
to DictReader. Verified against both a MAF with leading '#' lines and
one without -- both parse the Chromosome column correctly.
…pu's 7d MaxTime Adds multievolve_walltime.config (passed via -c), setting PHYLOWGS_MULTIEVOLVE's walltime per sample via a quantile-regression estimate on ssm_data.txt/cnv_data.txt size instead of a flat duration. mv_max_hours and resourceLimits.time are capped at 168h (7d) to match cmobic_cpu's actual MaxTime (confirmed via `scontrol show partition cmobic_cpu`). Requesting above the partition's MaxTime gets sbatch to reject the submission outright (empty workdir, no .command.log) rather than fail after running -- hit this on MSK-05/07/09/37 with a prior 384h request before the cap was corrected.
… IO usage on the cluster
…e GHCR manifest" This reverts commit 0df7184.
…e GHCR manifest" This reverts commit 1a01686.
|
…feature/release_candidate_2.0.0
…oantigen-pipeline into feature/release_candidate_2.0.0
…oantigen-pipeline into feature/release_candidate_2.0.0
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR checklist
nf-core pipelines lint).nextflow run . -profile test,docker --outdir <OUTDIR>).nextflow run . -profile debug,test,docker --outdir <OUTDIR>).docs/usage.mdis updated.docs/output.mdis updated.CHANGELOG.mdis updated.README.mdis updated (including new tool citations and authors/contributors).