vg giraffe hangs or stalls when reading large gzipped FASTQ files

[linsson]

1. What were you trying to do?

I was trying to map paired-end reads from .fastp.fastq.gz files to a tomato pangenome graph using vg giraffe.
Command structure (called inside a Bash job submitted via bsub):

vg giraffe -t 48 -Z graph.gbz -d graph.dist -m graph.min \
  -f sample_R1.fastp.fastq.gz -f sample_R2.fastp.fastq.gz \
  -N SAMPLE -R SAMPLE

2. What did you want to happen?

I expected vg giraffe to process the reads and output a .gam file with the alignments.

3. What actually happened?

In many jobs, the process stalls indefinitely. Symptoms include:

• .gam file remains empty (0 bytes) even after 1+ hour.
• The vg giraffe process appears in ps output with status Dl, indicating uninterruptible I/O wait.
• In some cases, vg giraffe hangs immediately after start; in others, it starts writing but then stops progressing.
• Manual inspection with lsof confirms that both FASTQ files are open.

Files are on a GPFS (IBM Spectrum Scale) filesystem, but manual read with zcat is instantaneous.

4. If you got a line like Stack trace path: /somewhere/on/your/computer/stacktrace.txt, please copy-paste the contents of that file here:

No crash or stack trace is produced, only hanging.

5. What data and command can the vg dev team use to make the problem happen?

While I cannot share the full tomato data due to size/privacy, the problem appears reproducible when:

• Reading gzipped .fastq.gz files > 5 GB each (paired-end).
• Files are stored on a GPFS filesystem.
• The command is run with high thread count (e.g. -t 48).
• The command is launched inside a batch job (e.g. via bsub).

6. What does running vg version say?

v> vg version
vg [warning]: System's vm.overcommit_memory setting is 2 (never overcommit). vg does not work well under these conditions; you may appear to run out of memory with plenty of memory left. Attempting to unsafely reconfigure jemalloc to deal better with this situation.
vg version v1.66.0 "Navetta"
Compiled with g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 on Linux
Linked against libstd++ 20230528
Using HTSlib headers 101990, library 1.19.1-29-g3cfe8769
Built by fokamoto@mustard

[jltsiren]

What do the stalled instances output if you run them with option -p / --progress? Without knowing anything else, my first guess would be an issue with the file system or memory-mapped files.

[faithokamoto]

Wait what, you're using my vg installation? I really can't recommend that - one of them is kept on a branch that is actively developing vg giraffe's chaining algorithm (and thus is prone to having bugs all the time while I work on it). The other, while usually close to up-to-date, constantly swaps between branches as I tinker around the edges.

[faithokamoto]

As for the actual issue, if you unzip the FASTQs does it still hang?

[faithokamoto]

Oh wait maybe you're using the vg that I released. That makes more sense. Sorry, I sort of panicked on seeing my username.

[linsson]

Oh wait maybe you're using the vg that I released. That makes more sense. Sorry, I sort of panicked on seeing my username.

I downloaded the first asset (vg bynary file) from https://github.com/vgteam/vg/releases/tag/v1.66.0. Actually I tried to build from source it but it seems not super easy for me.

[linsson]

What do the stalled instances output if you run them with option -p / --progress? Without knowing anything else, my first guess would be an issue with the file system or memory-mapped files.

Hi. I added "-p" option and launched again the job. Again, il looks hanging with a zero byte output file created 45 minutes ago. This is the result from stderr:

vg [warning]: System's vm.overcommit_memory setting is 2 (never overcommit). vg does not work well under these conditions; you may appear to run out of memory with plenty of memory left. Attempting to unsafely reconfigure jemalloc to deal better with this situation.
Guessing that /*********************************/pan/35_tomato_cactus_graph_noch0.d2.shortread.zipcodes is Short Read Zipcodes
Preparing Indexes
Loading Minimizer Index
Loading Zipcodes
Loading GBZ
Loading Distance Index v2
Paging in Distance Index v2
Initializing MinimizerMapper
Loading and initialization: 253.177 seconds
Of which Distance Index v2 paging: 194.359 seconds
Mapping reads to "-" (GAM)
--watchdog-timeout 10
--batch-size 512
--match 1
--mismatch 4
--gap-open 6
--gap-extend 1
--full-l-bonus 5
--max-multimaps 1
--hit-cap 10
--hard-hit-cap 500
--score-fraction 0.9
--max-min 500
--min-coverage-flank 250
--num-bp-per-min 1000
--downsample-window-length 18446744073709551615
--downsample-window-count 0
--distance-limit 200
--max-extensions 800
--max-alignments 8
--cluster-score 50
--pad-cluster-score 20
--cluster-coverage 0.3
--max-extension-mismatches 4
--extension-score 1
--extension-set 20
--rescue-attempts 15
--max-fragment-length 2000
--paired-distance-limit 2
--rescue-subgraph-size 4
--rescue-seed-limit 100
--mapq-score-window 0
--mapq-score-scale 1
--zipcode-tree-score-threshold 50
--pad-zipcode-tree-score-threshold 20
--zipcode-tree-coverage-threshold 0.3
--zipcode-tree-scale 2
--min-to-fragment 4
--max-to-fragment 10
--max-direct-chain 0
--gapless-extension-limit 0
--fragment-max-graph-lookback-bases 300
--fragment-max-graph-lookback-bases-per-base 0.03
--fragment-max-read-lookback-bases 18446744073709551615
--fragment-max-read-lookback-bases-per-base 1
--max-fragments 18446744073709551615
--fragment-max-indel-bases 2000
--fragment-max-indel-bases-per-base 0.2
--fragment-gap-scale 1
--fragment-points-per-possible-match 0
--fragment-score-fraction 0.1
--fragment-max-min-score 1.79769e+308
--fragment-min-score 60
--fragment-set-score-threshold 0
--min-chaining-problems 1
--max-chaining-problems 2147483647
--max-graph-lookback-bases 3000
--max-graph-lookback-bases-per-base 0.3
--max-read-lookback-bases 18446744073709551615
--max-read-lookback-bases-per-base 1
--max-indel-bases 2000
--max-indel-bases-per-base 0.2
--item-bonus 0
--item-scale 1
--gap-scale 1
--points-per-possible-match 0
--chain-score-threshold 100
--min-chains 4
--min-chain-score-per-base 0.01
--max-min-chain-score 200
--max-skipped-bases 0
--max-chains-per-tree 1
--max-chain-connection 100
--max-tail-length 100
--max-dp-cells 18446744073709551615
--max-tail-gap 18446744073709551615
--max-middle-gap 18446744073709551615
--max-tail-dp-length 30000
--max-middle-dp-length 2147483647
--wfa-max-mismatches 2
--wfa-max-mismatches-per-base 0.1
--wfa-max-max-mismatches 20
--wfa-distance 10
--wfa-distance-per-base 0.1
--wfa-max-distance 200
--min-unique-node-fraction 0
--rescue-algorithm dozeu
warning[vg::Watchdog]: Thread 21 has been checked in for 10 seconds processing: FP100002664BRL1C001R00300026832/1, FP100002664BRL1C001R00300026832/2
warning[vg::Watchdog]: Thread 21 finally checked out after 25 seconds and 0 kb memory growth processing: FP100002664BRL1C001R00300026832/1, FP100002664BRL1C001R00300026832/2
warning[vg::Watchdog]: Thread 21 has been checked in for 10 seconds processing: FP100002664BRL1C001R00300031368/1, FP100002664BRL1C001R00300031368/2
warning[vg::Watchdog]: Thread 21 finally checked out after 54 seconds and 0 kb memory growth processing: FP100002664BRL1C001R00300031368/1, FP100002664BRL1C001R00300031368/2
warning[vg::Watchdog]: Thread 21 has been checked in for 10 seconds processing: FP100002664BRL1C001R00300040655/1, FP100002664BRL1C001R00300040655/2
warning[vg::Watchdog]: Thread 21 finally checked out after 20 seconds and 0 kb memory growth processing: FP100002664BRL1C001R00300040655/1, FP100002664BRL1C001R00300040655/2
warning[vg::Watchdog]: Thread 21 has been checked in for 10 seconds processing: FP100002664BRL1C001R00300042273/1, FP100002664BRL1C001R00300042273/2
warning[vg::Watchdog]: Thread 21 finally checked out after 73 seconds and 0 kb memory growth processing: FP100002664BRL1C001R00300042273/1, FP100002664BRL1C001R00300042273/2
warning[vg::Watchdog]: Thread 21 has been checked in for 10 seconds processing: FP100002664BRL1C001R00300044788/1, FP100002664BRL1C001R00300044788/2
warning[vg::Watchdog]: Thread 21 finally checked out after 18 seconds and 0 kb memory growth processing: FP100002664BRL1C001R00300044788/1, FP100002664BRL1C001R00300044788/2
warning[vg::Watchdog]: Thread 21 has been checked in for 10 seconds processing: FP100002664BRL1C001R00300044893/1, FP100002664BRL1C001R00300044893/2
warning[vg::Watchdog]: Thread 21 finally checked out after 654 seconds and 0 kb memory growth processing: FP100002664BRL1C001R00300044893/1, FP100002664BRL1C001R00300044893/2
warning[vg::Watchdog]: Thread 21 has been checked in for 10 seconds processing: FP100002664BRL1C001R00300048836/1, FP100002664BRL1C001R00300048836/2
warning[vg::Watchdog]: Thread 21 finally checked out after 15 seconds and 0 kb memory growth processing: FP100002664BRL1C001R00300048836/1, FP100002664BRL1C001R00300048836/2
warning[vg::Watchdog]: Thread 21 has been checked in for 10 seconds processing: FP100002664BRL1C001R00300052462/1, FP100002664BRL1C001R00300052462/2
warning[vg::Watchdog]: Thread 21 finally checked out after 13 seconds and 0 kb memory growth processing: FP100002664BRL1C001R00300052462/1, FP100002664BRL1C001R00300052462/2
warning[vg::Watchdog]: Thread 21 has been checked in for 10 seconds processing: FP100002664BRL1C001R00300055097/1, FP100002664BRL1C001R00300055097/2
warning[vg::Watchdog]: Thread 21 finally checked out after 30 seconds and 0 kb memory growth processing: FP100002664BRL1C001R00300055097/1, FP100002664BRL1C001R00300055097/2
warning[vg::Watchdog]: Thread 21 has been checked in for 10 seconds processing: FP100002664BRL1C001R00300059638/1, FP100002664BRL1C001R00300059638/2
warning[vg::Watchdog]: Thread 21 finally checked out after 14 seconds and 0 kb memory growth processing: FP100002664BRL1C001R00300059638/1, FP100002664BRL1C001R00300059638/2
warning[vg::Watchdog]: Thread 21 has been checked in for 10 seconds processing: FP100002664BRL1C001R00300060987/1, FP100002664BRL1C001R00300060987/2
warning[vg::Watchdog]: Thread 21 finally checked out after 36 seconds and 0 kb memory growth processing: FP100002664BRL1C001R00300060987/1, FP100002664BRL1C001R00300060987/2
warning[vg::Watchdog]: Thread 21 has been checked in for 10 seconds processing: FP100002664BRL1C001R00300061236/1, FP100002664BRL1C001R00300061236/2
warning[vg::Watchdog]: Thread 21 finally checked out after 14 seconds and 0 kb memory growth processing: FP100002664BRL1C001R00300061236/1, FP100002664BRL1C001R00300061236/2
warning[vg::Watchdog]: Thread 21 has been checked in for 10 seconds processing: FP100002664BRL1C001R00300061623/1, FP100002664BRL1C001R00300061623/2
warning[vg::Watchdog]: Thread 21 finally checked out after 18 seconds and 0 kb memory growth processing: FP100002664BRL1C001R00300061623/1, FP100002664BRL1C001R00300061623/2
warning[vg::Watchdog]: Thread 21 has been checked in for 10 seconds processing: FP100002664BRL1C001R00300063652/1, FP100002664BRL1C001R00300063652/2
warning[vg::Watchdog]: Thread 21 finally checked out after 20 seconds and 0 kb memory growth processing: FP100002664BRL1C001R00300063652/1, FP100002664BRL1C001R00300063652/2
warning[vg::Watchdog]: Thread 21 has been checked in for 10 seconds processing: FP100002664BRL1C001R00300072579/1, FP100002664BRL1C001R00300072579/2
warning[vg::Watchdog]: Thread 21 finally checked out after 16 seconds and 0 kb memory growth processing: FP100002664BRL1C001R00300072579/1, FP100002664BRL1C001R00300072579/2
warning[vg::Watchdog]: Thread 21 has been checked in for 10 seconds processing: FP100002664BRL1C001R00300074448/1, FP100002664BRL1C001R00300074448/2
warning[vg::Watchdog]: Thread 21 finally checked out after 20 seconds and 0 kb memory growth processing: FP100002664BRL1C001R00300074448/1, FP100002664BRL1C001R00300074448/2
warning[vg::Watchdog]: Thread 21 has been checked in for 10 seconds processing: FP100002664BRL1C001R00300075177/1, FP100002664BRL1C001R00300075177/2
warning[vg::Watchdog]: Thread 21 finally checked out after 14 seconds and 0 kb memory growth processing: FP100002664BRL1C001R00300075177/1, FP100002664BRL1C001R00300075177/2
warning[vg::Watchdog]: Thread 21 has been checked in for 10 seconds processing: FP100002664BRL1C001R00300082207/1, FP100002664BRL1C001R00300082207/2
warning[vg::Watchdog]: Thread 21 finally checked out after 18 seconds and 0 kb memory growth processing: FP100002664BRL1C001R00300082207/1, FP100002664BRL1C001R00300082207/2
warning[vg::Watchdog]: Thread 21 has been checked in for 10 seconds processing: FP100002664BRL1C001R00300082936/1, FP100002664BRL1C001R00300082936/2
warning[vg::Watchdog]: Thread 21 finally checked out after 105 seconds and 0 kb memory growth processing: FP100002664BRL1C001R00300082936/1, FP100002664BRL1C001R00300082936/2
warning[vg::Watchdog]: Thread 21 has been checked in for 10 seconds processing: FP100002664BRL1C001R00300091332/1, FP100002664BRL1C001R00300091332/2
warning[vg::Watchdog]: Thread 21 finally checked out after 67 seconds and 0 kb memory growth processing: FP100002664BRL1C001R00300091332/1, FP100002664BRL1C001R00300091332/2
warning[vg::Watchdog]: Thread 21 has been checked in for 10 seconds processing: FP100002664BRL1C001R02104050204/1, FP100002664BRL1C001R02104050204/2
warning[vg::Watchdog]: Thread 21 finally checked out after 380 seconds and 0 kb memory growth processing: FP100002664BRL1C001R02104050204/1, FP100002664BRL1C001R02104050204/2
warning[vg::Watchdog]: Thread 21 has been checked in for 10 seconds processing: FP100002664BRL1C001R00300091627/1, FP100002664BRL1C001R00300091627/2
warning[vg::Watchdog]: Thread 21 finally checked out after 30 seconds and 0 kb memory growth processing: FP100002664BRL1C001R00300091627/1, FP100002664BRL1C001R00300091627/2
warning[vg::Watchdog]: Thread 21 has been checked in for 10 seconds processing: FP100002664BRL1C001R00300093215/1, FP100002664BRL1C001R00300093215/2
warning[vg::Watchdog]: Thread 21 finally checked out after 35 seconds and 0 kb memory growth processing: FP100002664BRL1C001R00300093215/1, FP100002664BRL1C001R00300093215/2
warning[vg::Watchdog]: Thread 21 has been checked in for 10 seconds processing: FP100002664BRL1C001R02104050291/1, FP100002664BRL1C001R02104050291/2
warning[vg::Watchdog]: Thread 21 finally checked out after 24 seconds and 0 kb memory growth processing: FP100002664BRL1C001R02104050291/1, FP100002664BRL1C001R02104050291/2
warning[vg::Watchdog]: Thread 21 has been checked in for 10 seconds processing: FP100002664BRL1C001R02104050381/1, FP100002664BRL1C001R02104050381/2
warning[vg::Watchdog]: Thread 21 finally checked out after 200 seconds and 0 kb memory growth processing: FP100002664BRL1C001R02104050381/1, FP100002664BRL1C001R02104050381/2
warning[vg::Watchdog]: Thread 21 has been checked in for 10 seconds processing: FP100002664BRL1C001R00300093437/1, FP100002664BRL1C001R00300093437/2
warning[vg::Watchdog]: Thread 21 finally checked out after 17 seconds and 0 kb memory growth processing: FP100002664BRL1C001R00300093437/1, FP100002664BRL1C001R00300093437/2
warning[vg::Watchdog]: Thread 21 has been checked in for 10 seconds processing: FP100002664BRL1C001R00300094401/1, FP100002664BRL1C001R00300094401/2

[jltsiren]

Normally distance index paging should take only a few seconds, but here it's taking more than the rest of loading and initialization. As you are using a distributed file system, this could be the old issue with memory-mapped files on such file systems. If you make the distance index (.dist) file read-only, it could fix the issue. If not, copying the distance index to a local drive and using the local copy might.

If neither of those help, the issue has probably something to do with the graph, the indexes, or the reads. In that case, more information on what is exactly in the graph and which commands were used to build and index it could help.

[linsson]

Hi! Things are getting little better (maybe) but still not working, unfortunately.

Just to be complete, I changed (chmod 444) .dist file only without killing and relaunching the jobs from the queue since they were pending. Appparently that was not enough. Then I tried to kill all jobs and go with

> ls -lhrt
total 23G
-rw-rw-r--. 1 **** **** 1.7G Jun 20 20:21 35_tomato_cactus_graph_noch0.d2.gbz
-rw-rw-r--. 1 **** ****  52M Jun 20 20:21 35_tomato_cactus_graph_noch0.d2.snarls
-rw-rw-r--. 1 **** **** 322M Jun 20 20:22 35_tomato_cactus_graph_noch0.d2.shortread.zipcodes
-r--r--r--. 1 **** **** 1.5G Jun 20 20:22 35_tomato_cactus_graph_noch0.d2.dist
-rw-rw-r--. 1 **** **** 9.4G Jun 25 19:41 35_tomato_cactus_graph_noch0.d2.shortread.withzip.min
-rw-rw-r--. 1 **** **** 9.4G Jun 25 19:47 35_tomato_cactus_graph_noch0.d2.min
> chmod 444 *
> ls -lhrt
total 23G
-r--r--r--. 1 **** **** 1.7G Jun 20 20:21 35_tomato_cactus_graph_noch0.d2.gbz
-r--r--r--. 1 **** ****  52M Jun 20 20:21 35_tomato_cactus_graph_noch0.d2.snarls
-r--r--r--. 1 **** **** 322M Jun 20 20:22 35_tomato_cactus_graph_noch0.d2.shortread.zipcodes
-r--r--r--. 1 **** ****1.5G Jun 20 20:22 35_tomato_cactus_graph_noch0.d2.dist
-r--r--r--. 1 **** **** 9.4G Jun 25 19:41 35_tomato_cactus_graph_noch0.d2.shortread.withzip.min
-r--r--r--. 1 **** **** 9.4G Jun 25 19:47 35_tomato_cactus_graph_noch0.d2.min

Then I submitted my jobs again and the output files this time moved from zero bytes. Unfortunately a crush seem to occur very often (not sure if I should say ALWAYS):

vg [warning]: System's vm.overcommit_memory setting is 2 (never overcommit). vg does not work well under these conditions; you may appear to run out of memory with plenty of memory left. Attempting to unsafely reconfigure jemalloc to deal better with this situation.
Guessing that ***********/35_tomato_cactus_graph_noch0.d2.shortread.zipcodes is Short Read Zipcodes
Preparing Indexes
Loading Minimizer Index
Loading Zipcodes
Loading GBZ
Loading Distance Index v2
Paging in Distance Index v2
Initializing MinimizerMapper
Loading and initialization: 56.3655 seconds
Of which Distance Index v2 paging: 3.43938 seconds
Mapping reads to "-" (GAM)
--watchdog-timeout 10
--batch-size 512
--match 1
--mismatch 4
--gap-open 6
--gap-extend 1
--full-l-bonus 5
--max-multimaps 1
--hit-cap 10
--hard-hit-cap 500
--score-fraction 0.9
--max-min 500
--min-coverage-flank 250
--num-bp-per-min 1000
--downsample-window-length 18446744073709551615
--downsample-window-count 0
--distance-limit 200
--max-extensions 800
--max-alignments 8
--cluster-score 50
--pad-cluster-score 20
--cluster-coverage 0.3
--max-extension-mismatches 4
--extension-score 1
--extension-set 20
--rescue-attempts 15
--max-fragment-length 2000
--paired-distance-limit 2
--rescue-subgraph-size 4
--rescue-seed-limit 100
--mapq-score-window 0
--mapq-score-scale 1
--zipcode-tree-score-threshold 50
--pad-zipcode-tree-score-threshold 20
--zipcode-tree-coverage-threshold 0.3
--zipcode-tree-scale 2
--min-to-fragment 4
--max-to-fragment 10
--max-direct-chain 0
--gapless-extension-limit 0
--fragment-max-graph-lookback-bases 300
--fragment-max-graph-lookback-bases-per-base 0.03
--fragment-max-read-lookback-bases 18446744073709551615
--fragment-max-read-lookback-bases-per-base 1
--max-fragments 18446744073709551615
--fragment-max-indel-bases 2000
--fragment-max-indel-bases-per-base 0.2
--fragment-gap-scale 1
--fragment-points-per-possible-match 0
--fragment-score-fraction 0.1
--fragment-max-min-score 1.79769e+308
--fragment-min-score 60
--fragment-set-score-threshold 0
--min-chaining-problems 1
--max-chaining-problems 2147483647
--max-graph-lookback-bases 3000
--max-graph-lookback-bases-per-base 0.3
--max-read-lookback-bases 18446744073709551615
--max-read-lookback-bases-per-base 1
--max-indel-bases 2000
--max-indel-bases-per-base 0.2
--item-bonus 0
--item-scale 1
--gap-scale 1
--points-per-possible-match 0
--chain-score-threshold 100
--min-chains 4
--min-chain-score-per-base 0.01
--max-min-chain-score 200
--max-skipped-bases 0
--max-chains-per-tree 1
--max-chain-connection 100
--max-tail-length 100
--max-dp-cells 18446744073709551615
--max-tail-gap 18446744073709551615
--max-middle-gap 18446744073709551615
--max-tail-dp-length 30000
--max-middle-dp-length 2147483647
--wfa-max-mismatches 2
--wfa-max-mismatches-per-base 0.1
--wfa-max-max-mismatches 20
--wfa-distance 10
--wfa-distance-per-base 0.1
--wfa-max-distance 200
--min-unique-node-fraction 0
--rescue-algorithm dozeu
Using fragment length estimate: 325.663 +/- 76.919
warning[vg::giraffe]: Refusing to perform too-large rescue alignment of 150 bp against 12218 bp ordered subgraph for read FP100002360BRL1C001R00604220712/1 which would use more than 1572864 cells and might exhaust Dozeu's allocator; suppressing further warnings.
━━━━━━━━━━━━━━━━━━━━
Crash report for vg v1.66.0 "Navetta"
Caught signal 11 raised at address 0x1326dff; tracing with backward-cpp
Stack trace (most recent call last) in thread 371760:
#12   Object "", at 0xffffffffffffffff, in 
#11   Object "/gporq3/store_1/project/aprea_software/cresco4/gcc/8.5.0/vg/1.66.0/bin/vg", at 0x2393cf3, in __clone
#10   Object "/gporq3/store_1/project/aprea_software/cresco4/gcc/8.5.0/vg/1.66.0/bin/vg", at 0x22ed24a, in start_thread
#9    Object "/gporq3/store_1/project/aprea_software/cresco4/gcc/8.5.0/vg/1.66.0/bin/vg", at 0x228fadd, in gomp_thread_start
#8    Object "/gporq3/store_1/project/aprea_software/cresco4/gcc/8.5.0/vg/1.66.0/bin/vg", at 0x2292427, in gomp_team_barrier_wait_end
#7    Object "/gporq3/store_1/project/aprea_software/cresco4/gcc/8.5.0/vg/1.66.0/bin/vg", at 0x2289d2a, in gomp_barrier_handle_tasks
#6    Object "/gporq3/store_1/project/aprea_software/cresco4/gcc/8.5.0/vg/1.66.0/bin/vg", at 0x101394a, in unsigned long vg::io::paired_for_each_parallel_after_wait<vg::Alignment>(std::function<bool (vg::Alignment&, vg::Alignment&)>, std::function<void (vg::Alignment&, vg::Alignment&)>, std::function<bool ()>, unsi
gned long) [clone ._omp_fn.1]
#5    Object "/gporq3/store_1/project/aprea_software/cresco4/gcc/8.5.0/vg/1.66.0/bin/vg", at 0xe53a43, in std::_Function_handler<void (vg::Alignment&, vg::Alignment&), main_giraffe(int, char**)::{lambda()#1}::operator()() const::{lambda(vg::Alignment&, vg::Alignment&)#9}>::_M_invoke(std::_Any_data const&, vg::Align
ment&, vg::Alignment&)
#4    Object "/gporq3/store_1/project/aprea_software/cresco4/gcc/8.5.0/vg/1.66.0/bin/vg", at 0x13187e6, in vg::MinimizerMapper::map_paired(vg::Alignment&, vg::Alignment&, std::vector<std::pair<vg::Alignment, vg::Alignment>, std::allocator<std::pair<vg::Alignment, vg::Alignment> > >&)
#3    Object "/gporq3/store_1/project/aprea_software/cresco4/gcc/8.5.0/vg/1.66.0/bin/vg", at 0x131438f, in vg::MinimizerMapper::map_paired(vg::Alignment&, vg::Alignment&)
#2    Object "/gporq3/store_1/project/aprea_software/cresco4/gcc/8.5.0/vg/1.66.0/bin/vg", at 0x130986f, in void vg::MinimizerMapper::process_until_threshold_c<double>(unsigned long, std::function<double (unsigned long)> const&, std::function<bool (unsigned long, unsigned long)> const&, double, unsigned long, unsign
ed long, vg::LazyRNG&, std::function<bool (unsigned long, unsigned long)> const&, std::function<void (unsigned long)> const&, std::function<void (unsigned long)> const&) const [clone .constprop.0]
#1    Object "/gporq3/store_1/project/aprea_software/cresco4/gcc/8.5.0/vg/1.66.0/bin/vg", at 0x1301edc, in std::_Function_handler<void (unsigned long), vg::MinimizerMapper::map_paired(vg::Alignment&, vg::Alignment&)::{lambda(unsigned long)#21}>::_M_invoke(std::_Any_data const&, unsigned long&&)
#0    Object "/gporq3/store_1/project/aprea_software/cresco4/gcc/8.5.0/vg/1.66.0/bin/vg", at 0x1326dff, in void std::vector<long, std::allocator<long> >::_M_realloc_insert<long&>(__gnu_cxx::__normal_iterator<long*, std::vector<long, std::allocator<long> > >, long&)

Library locations:
ERROR: Signal 11 occurred. VG has crashed. Visit https://github.com/vgteam/vg/issues/new/choose to report a bug.
━━━━━━━━━━━━━━━━━━━━
Context dump:
        Thread 0: FP100002360BRL1C003R01204923921/1, FP100002360BRL1C003R01204923921/2
        Thread 1: FP100002360BRL1C003R01204727447/1, FP100002360BRL1C003R01204727447/2
        Thread 2: FP100002360BRL1C003R01205137067/1, FP100002360BRL1C003R01205137067/2
        Thread 3: FP100002360BRL1C003R01205045084/1, FP100002360BRL1C003R01205045084/2
        Thread 4: FP100002360BRL1C003R00900869849/1, FP100002360BRL1C003R00900869849/2
        Thread 5: FP100002360BRL1C003R00900815933/1, FP100002360BRL1C003R00900815933/2
        Thread 6: FP100002360BRL1C003R01204955962/1, FP100002360BRL1C003R01204955962/2
        Thread 7: FP100002360BRL1C003R00900622907/1, FP100002360BRL1C003R00900622907/2
        Thread 8: FP100002360BRL1C003R00900745129/1, FP100002360BRL1C003R00900745129/2
        Thread 9: FP100002360BRL1C003R00900972533/1, FP100002360BRL1C003R00900972533/2
        Thread 10: FP100002360BRL1C003R00900780540/1, FP100002360BRL1C003R00900780540/2
        Thread 11: FP100002360BRL1C003R00900560016/1, FP100002360BRL1C003R00900560016/2
        Thread 12: FP100002360BRL1C003R00900655702/1, FP100002360BRL1C003R00900655702/2
        Thread 13: FP100002360BRL1C003R00900398065/1, FP100002360BRL1C003R00900398065/2
        Thread 14: FP100002360BRL1C003R01205302967/1, FP100002360BRL1C003R01205302967/2
        Thread 15: FP100002360BRL1C003R01204760322/1, FP100002360BRL1C003R01204760322/2
        Thread 16: FP100002360BRL1C003R01204828858/1, FP100002360BRL1C003R01204828858/2
        Thread 17: FP100002360BRL1C003R00900941420/1, FP100002360BRL1C003R00900941420/2
        Thread 18: FP100002360BRL1C003R00900464689/1, FP100002360BRL1C003R00900464689/2
        Thread 19: FP100002360BRL1C003R01205080161/1, FP100002360BRL1C003R01205080161/2
        Thread 20: FP100002360BRL1C003R01205238659/1, FP100002360BRL1C003R01205238659/2
        Thread 21: FP100002360BRL1C003R00900432062/1, FP100002360BRL1C003R00900432062/2
        Thread 22: FP100002360BRL1C003R00900843161/1, FP100002360BRL1C003R00900843161/2
        Thread 23: FP100002360BRL1C003R00900686806/1, FP100002360BRL1C003R00900686806/2
        Thread 24: FP100002360BRL1C003R00900877992/1, FP100002360BRL1C003R00900877992/2
        Thread 25: FP100002360BRL1C003R00900706123/1, FP100002360BRL1C003R00900706123/2
        Thread 26: FP100002360BRL1C003R01205242600/1, FP100002360BRL1C003R01205242600/2
        Thread 27: FP100002360BRL1C003R00900739593/1, FP100002360BRL1C003R00900739593/2
        Thread 28: FP100002360BRL1C003R00900427036/1, FP100002360BRL1C003R00900427036/2
        Thread 29: FP100002360BRL1C003R00900526317/1, FP100002360BRL1C003R00900526317/2
        Thread 31: FP100002360BRL1C003R00900495952/1, FP100002360BRL1C003R00900495952/2
        Thread 32: FP100002360BRL1C003R01205174491/1, FP100002360BRL1C003R01205174491/2
        Thread 33: FP100002360BRL1C003R00900679592/1, FP100002360BRL1C003R00900679592/2
        Thread 34: FP100002360BRL1C003R01205021795/1, FP100002360BRL1C003R01205021795/2
        Thread 35: FP100002360BRL1C003R01204792506/1, FP100002360BRL1C003R01204792506/2
        Thread 36: FP100002360BRL1C003R00900904380/1, FP100002360BRL1C003R00900904380/2
        Thread 37: FP100002360BRL1C003R01205208257/1, FP100002360BRL1C003R01205208257/2
        Thread 38: FP100002360BRL1C003R01205113798/1, FP100002360BRL1C003R01205113798/2
        Thread 39: FP100002360BRL1C003R01205111628/1, FP100002360BRL1C003R01205111628/2
        Thread 40: FP100002360BRL1C003R01204663282/1, FP100002360BRL1C003R01204663282/2
        Thread 41: FP100002360BRL1C003R01204920483/1, FP100002360BRL1C003R01204920483/2
        Thread 42: FP100002360BRL1C003R01204857056/1, FP100002360BRL1C003R01204857056/2
        Thread 43: FP100002360BRL1C003R01204797716/1, FP100002360BRL1C003R01204797716/2
        Thread 44: FP100002360BRL1C003R01204889962/1, FP100002360BRL1C003R01204889962/2
        Thread 45: FP100002360BRL1C003R01205273840/1, FP100002360BRL1C003R01205273840/2
        Thread 46: FP100002360BRL1C003R00900660607/1, FP100002360BRL1C003R00900660607/2
        Thread 47: FP100002360BRL1C003R00900588442/1, FP100002360BRL1C003R00900588442/2
Found 47 threads with context.
━━━━━━━━━━━━━━━━━━━━
Please include this entire error log in your bug report!
━━━━━━━━━━━━━━━━━━━━
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Crash report for vg v1.66.0
Caught signal 11 raised at address 0x16a6f4e; tracing with backward-cpp
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Crash report for vg ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Crash report for vg v1.66.0
Caught signal 11 raised at address 0x16a6f4e━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Crash report for vg v1.66.0
Caught signal 11 raised at address ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Crash report for vg ━━━━━━━━━━━━━
━━Stack trace (most recent call last) in thread 0x5ac39━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Crash report for vg ━━━
Crash report for vg v1.66.0━:
━; tracing with backward-cpp━━━━━━━━
Crash report for vg v1.66.0
Caught signal 0xb raised at address 0x16a6f4e; tracing with backward-cpp
━━━━━Stack trace (most recent call last) in thread 0x5ac52:
#━━━━━━━0x16a6f4e━━━━━━━━
Crash report for vg v1.66.0
Caught signal 11 raised at address 0x16a6f4e; tracing with backward-cpp
━━━━━━━━━
Crash report for vg v1.66.0
Caught signal 11 raised at address 0x16a6f4e; tracing with backward-cppStack trace (most recent call last)━━━━━━━━v1.66.0
Caught signal 11━━━━━━━━
Crash report for vg v1.66.0
Caught signal 11 raised at address 0x16a6f4e; tracing with backward-cpp
#37Stack trace (most recent call last) in thread 371798:
━━━━━
Crash report for vg ━━━━━━━━━━━
Crash report for vg v1.66.0
Caught signal 11 raised at address ━
Crash report for vg v1.66.0
Caught signal 0xb raised at address 0x16a6f4e; tracing with backward-cpp; tracing with backward-cpp━━━━━━━━━━━━━━━━━━━━━
Crash report for vg ━━━━━━━━━━━
Crash report for vg v1.66.0
Caught signal 0xb raised at address 0x16a6f4e; tracing with backward-cpp

━
Crash report for vg ━Stack trace (most recent call last) in thread 0x5ac3d:
#━━ raised at address v1.66.0
Caught signal 0xb raised at address 0x16a6f4e; tracing with backward-cpp
━━━━━
Crash report for vg ━━━━━━━━━━━━━━━━━━
Crash report for vg v1.66.0
Caught signal 0xb raised at address 0x16a7888; tracing with backward-cpp
━━Stack trace (most recent call last) in thread 0x5ac4b:
#0xd:
0xf   Object "", at 0xffffffffffffffff, in 
━━━━━0x16a6f4e; tracing with backward-cpp
━Stack trace (most recent call last) in thread v1.66.0━   Object "", at 0, in 
#0    Object "", at 0, in 
#0    Object "", at 0, in 
#0    Object "", at 0, in 
#0    Object "", at ━━━━━━Crash report for vg Stack trace (most recent call last)0x5ac3b:
#0xd   Object "", at 0xffffffffffffffff, in 
━━━━━━━━━━━━━━━━
Crash report for vg v1.66.0
Caught signal 0xb raised at address 0x16a6f4e; tracing with backward-cpp
━━━━━━━━━━━Stack trace (most recent call last) in thread 0x5ac3d:
#0xd   Object "", at 0xffffffffffffffff, in 
0xc/afs/enea.it/por/system/lsbatch/1751438908.256274: line 1: 371672 Segmentation fault      (core dumped) vg giraffe -p -t ${THREADS} -Z ${PAN_REF}.gbz -d ${PAN_REF}.dist -m ${PAN_REF}.min -f ${FASTP1} -f ${FASTP2} -N ${SAMPLE} -R ${SAMPLE} > ${PAN_GAM_PREFIX}/${SAMPLE}/${FC}/${LANE}/${SAMPLE}.gam

[jltsiren]

You have two copies of the minimizer index: .shortread.withzip.min and .min. Are those files identical? Which index do you specify in the vg giraffe command? It's possible that the crashes are caused by a mismatch between the minimizer index and the .zipcodes file.

[linsson]

Thanks for your message. Unfortunately I did not realized I was using the wrong minimizer index. I apologize. Now I have completed vg giraffe for one sample:

vg [warning]: System's vm.overcommit_memory setting is 2 (never overcommit). vg does not work well under these conditions; you may appear to run out of memory with plenty of memory left. Attempting to unsafely reconfigure jemalloc to deal better with this situation.
Guessing that ****/35_tomato_cactus_graph_noch0.d2.shortread.zipcodes is Short Read Zipcodes
Preparing Indexes
Loading Minimizer Index
Loading Zipcodes
Loading GBZ
Loading Distance Index v2
Paging in Distance Index v2
Initializing MinimizerMapper
Loading and initialization: 72.1768 seconds
Of which Distance Index v2 paging: 5.30046 seconds
Mapping reads to "-" (GAM)
--watchdog-timeout 10
--batch-size 512
--match 1
--mismatch 4
--gap-open 6
--gap-extend 1
--full-l-bonus 5
--max-multimaps 1
--hit-cap 10
--hard-hit-cap 500
--score-fraction 0.9
--max-min 500
--min-coverage-flank 250
--num-bp-per-min 1000
--downsample-window-length 18446744073709551615
--downsample-window-count 0
--distance-limit 200
--max-extensions 800
--max-alignments 8
--cluster-score 50
--pad-cluster-score 20
--cluster-coverage 0.3
--max-extension-mismatches 4
--extension-score 1
--extension-set 20
--rescue-attempts 15
--max-fragment-length 2000
--paired-distance-limit 2
--rescue-subgraph-size 4
--rescue-seed-limit 100
--mapq-score-window 0
--mapq-score-scale 1
--zipcode-tree-score-threshold 50
--pad-zipcode-tree-score-threshold 20
--zipcode-tree-coverage-threshold 0.3
--zipcode-tree-scale 2
--min-to-fragment 4
--max-to-fragment 10
--max-direct-chain 0
--gapless-extension-limit 0
--fragment-max-graph-lookback-bases 300
--fragment-max-graph-lookback-bases-per-base 0.03
--fragment-max-read-lookback-bases 18446744073709551615
--fragment-max-read-lookback-bases-per-base 1
--max-fragments 18446744073709551615
--fragment-max-indel-bases 2000
--fragment-max-indel-bases-per-base 0.2
--fragment-gap-scale 1
--fragment-points-per-possible-match 0
--fragment-score-fraction 0.1
--fragment-max-min-score 1.79769e+308
--fragment-min-score 60
--fragment-set-score-threshold 0
--min-chaining-problems 1
--max-chaining-problems 2147483647
--max-graph-lookback-bases 3000
--max-graph-lookback-bases-per-base 0.3
--max-read-lookback-bases 18446744073709551615
--max-read-lookback-bases-per-base 1
--max-indel-bases 2000
--max-indel-bases-per-base 0.2
--item-bonus 0
--item-scale 1
--gap-scale 1
--points-per-possible-match 0
--chain-score-threshold 100
--min-chains 4
--min-chain-score-per-base 0.01
--max-min-chain-score 200
--max-skipped-bases 0
--max-chains-per-tree 1
--max-chain-connection 100
--max-tail-length 100
--max-dp-cells 18446744073709551615
--max-tail-gap 18446744073709551615
--max-middle-gap 18446744073709551615
--max-tail-dp-length 30000
--max-middle-dp-length 2147483647
--wfa-max-mismatches 2
--wfa-max-mismatches-per-base 0.1
--wfa-max-max-mismatches 20
--wfa-distance 10
--wfa-distance-per-base 0.1
--wfa-max-distance 200
--min-unique-node-fraction 0
--rescue-algorithm dozeu
Using fragment length estimate: 281.329 +/- 60.8255
warning[vg::giraffe]: Refusing to perform too-large rescue alignment of 150 bp against 12074 bp ordered subgraph for read E150029594L1C001R02402699313/1 which would use more than 1572864 cells and might exhaust Dozeu's allocator; suppressing further warnings.
Mapped 80150792 reads across 48 threads in 5279.17 seconds with 6.22422 additional single-threaded seconds.
Mapping speed: 316.293 reads per second per thread
Used 148657 CPU-seconds (including output).
Achieved 539.166 reads per CPU-second (including output)
Used 475407200159736 CPU instructions (not including output).
Mapping slowness: 5.93141 M instructions per read at 3198.02 M mapping instructions per inclusive CPU-second
Memory footprint: 18.891 GB
vg [warning]: System's vm.overcommit_memory setting is 2 (never overcommit). vg does not work well under these conditions; you may appear to run out of memory with plenty of memory left. Attempting to unsafely reconfigure jemalloc to deal better with this situation.

Nevertheless, for other two samples I have what looks like a similar error:

vg [warning]: System's vm.overcommit_memory setting is 2 (never overcommit). vg does not work well under these conditions; you may appear to run out of memory with plenty of memory left. Attempting to unsafely reconfigure jemalloc to deal better with this situation.
Guessing that ****/35_tomato_cactus_graph_noch0.d2.shortread.zipcodes is Short Read Zipcodes
Preparing Indexes
Loading Minimizer Index
Loading Zipcodes
Loading GBZ
Loading Distance Index v2
Paging in Distance Index v2
Initializing MinimizerMapper
Loading and initialization: 59.1753 seconds
Of which Distance Index v2 paging: 4.05367 seconds
Mapping reads to "-" (GAM)
--watchdog-timeout 10
--batch-size 512
--match 1
--mismatch 4
--gap-open 6
--gap-extend 1
--full-l-bonus 5
--max-multimaps 1
--hit-cap 10
--hard-hit-cap 500
--score-fraction 0.9
--max-min 500
--min-coverage-flank 250
--num-bp-per-min 1000
--downsample-window-length 18446744073709551615
--downsample-window-count 0
--distance-limit 200
--max-extensions 800
--max-alignments 8
--cluster-score 50
--pad-cluster-score 20
--cluster-coverage 0.3
--max-extension-mismatches 4
--extension-score 1
--extension-set 20
--rescue-attempts 15
--max-fragment-length 2000
--paired-distance-limit 2
--rescue-subgraph-size 4
--rescue-seed-limit 100
--mapq-score-window 0
--mapq-score-scale 1
--zipcode-tree-score-threshold 50
--pad-zipcode-tree-score-threshold 20
--zipcode-tree-coverage-threshold 0.3
--zipcode-tree-scale 2
--min-to-fragment 4
--max-to-fragment 10
--max-direct-chain 0
--gapless-extension-limit 0
--fragment-max-graph-lookback-bases 300
--fragment-max-graph-lookback-bases-per-base 0.03
--fragment-max-read-lookback-bases 18446744073709551615
--fragment-max-read-lookback-bases-per-base 1
--max-fragments 18446744073709551615
--fragment-max-indel-bases 2000
--fragment-max-indel-bases-per-base 0.2
--fragment-gap-scale 1
--fragment-points-per-possible-match 0
--fragment-score-fraction 0.1
--fragment-max-min-score 1.79769e+308
--fragment-min-score 60
--fragment-set-score-threshold 0
--min-chaining-problems 1
--max-chaining-problems 2147483647
--max-graph-lookback-bases 3000
--max-graph-lookback-bases-per-base 0.3
--max-read-lookback-bases 18446744073709551615
--max-read-lookback-bases-per-base 1
--max-indel-bases 2000
--max-indel-bases-per-base 0.2
--item-bonus 0
--item-scale 1
--gap-scale 1
--points-per-possible-match 0
--chain-score-threshold 100
--min-chains 4
--min-chain-score-per-base 0.01
--max-min-chain-score 200
--max-skipped-bases 0
--max-chains-per-tree 1
--max-chain-connection 100
--max-tail-length 100
--max-dp-cells 18446744073709551615
--max-tail-gap 18446744073709551615
--max-middle-gap 18446744073709551615
--max-tail-dp-length 30000
--max-middle-dp-length 2147483647
--wfa-max-mismatches 2
--wfa-max-mismatches-per-base 0.1
--wfa-max-max-mismatches 20
--wfa-distance 10
--wfa-distance-per-base 0.1
--wfa-max-distance 200
--min-unique-node-fraction 0
--rescue-algorithm dozeu
Using fragment length estimate: 314.851 +/- 66.712
━━━━━━━━━━━━━━━━━━━━
Crash report for vg v1.66.0 "Navetta"
Caught signal 11 raised at address 0x1326dff; tracing with backward-cpp
Stack trace (most recent call last) in thread 2187742:
#12   Object "", at 0xffffffffffffffff, in 
#11   Object "****/gcc/8.5.0/vg/1.66.0/bin/vg", at 0x2393cf3, in __clone
#10   Object "****/gcc/8.5.0/vg/1.66.0/bin/vg", at 0x22ed24a, in start_thread
#9    Object "****/8.5.0/vg/1.66.0/bin/vg", at 0x228fadd, in gomp_thread_start
#8    Object "****/8.5.0/vg/1.66.0/bin/vg", at 0x2292427, in gomp_team_barrier_wait_end
#7    Object "****/8.5.0/vg/1.66.0/bin/vg", at 0x2289d2a, in gomp_barrier_handle_tasks
#6    Object "****/8.5.0/vg/1.66.0/bin/vg", at 0x101394a, in unsigned long vg::io::paired_for_each_parallel_after_wait<vg::Alignment>(std::function<bool (vg::Alignment&, vg::Alignment&)>, std::function<void (vg::Alignment&, vg::Alignment&)>, std::function<bool ()>, unsi
gned long) [clone ._omp_fn.1]
#5    Object "****/8.5.0/vg/1.66.0/bin/vg", at 0xe53a43, in std::_Function_handler<void (vg::Alignment&, vg::Alignment&), main_giraffe(int, char**)::{lambda()#1}::operator()() const::{lambda(vg::Alignment&, vg::Alignment&)#9}>::_M_invoke(std::_Any_data const&, vg::Align
ment&, vg::Alignment&)
#4    Object "****/8.5.0/vg/1.66.0/bin/vg", at 0x13187e6, in vg::MinimizerMapper::map_paired(vg::Alignment&, vg::Alignment&, std::vector<std::pair<vg::Alignment, vg::Alignment>, std::allocator<std::pair<vg::Alignment, vg::Alignment> > >&)
#3    Object "****/8.5.0/vg/1.66.0/bin/vg", at 0x131438f, in vg::MinimizerMapper::map_paired(vg::Alignment&, vg::Alignment&)
#2    Object "****/8.5.0/vg/1.66.0/bin/vg", at 0x130986f, in void vg::MinimizerMapper::process_until_threshold_c<double>(unsigned long, std::function<double (unsigned long)> const&, std::function<bool (unsigned long, unsigned long)> const&, double, unsigned long, unsign
ed long, vg::LazyRNG&, std::function<bool (unsigned long, unsigned long)> const&, std::function<void (unsigned long)> const&, std::function<void (unsigned long)> const&) const [clone .constprop.0]
#1    Object "****/8.5.0/vg/1.66.0/bin/vg", at 0x1301edc, in std::_Function_handler<void (unsigned long), vg::MinimizerMapper::map_paired(vg::Alignment&, vg::Alignment&)::{lambda(unsigned long)#21}>::_M_invoke(std::_Any_data const&, unsigned long&&)
#0    Object "****/8.5.0/vg/1.66.0/bin/vg", at 0x1326dff, in void std::vector<long, std::allocator<long> >::_M_realloc_insert<long&>(__gnu_cxx::__normal_iterator<long*, std::vector<long, std::allocator<long> > >, long&)

Library locations:
ERROR: Signal 11 occurred. VG has crashed. Visit https://github.com/vgteam/vg/issues/new/choose to report a bug.
━━━━━━━━━━━━━━━━━━━━
Context dump:
        Thread 0: FP100002273TRL1C002R02801988469/1, FP100002273TRL1C002R02801988469/2
        Thread 1: FP100002273TRL1C002R02803000359/1, FP100002273TRL1C002R02803000359/2
        Thread 2: FP100002273TRL1C002R02802186194/1, FP100002273TRL1C002R02802186194/2
        Thread 3: FP100002273TRL1C002R02201435765/1, FP100002273TRL1C002R02201435765/2
        Thread 4: FP100002273TRL1C002R02201222181/1, FP100002273TRL1C002R02201222181/2
        Thread 5: FP100002273TRL1C002R02200853237/1, FP100002273TRL1C002R02200853237/2
        Thread 6: FP100002273TRL1C002R02200958280/1, FP100002273TRL1C002R02200958280/2
        Thread 7: FP100002273TRL1C002R02803105522/1, FP100002273TRL1C002R02803105522/2
        Thread 8: FP100002273TRL1C002R02200894056/1, FP100002273TRL1C002R02200894056/2
        Thread 9: FP100002273TRL1C002R02201371766/1, FP100002273TRL1C002R02201371766/2
        Thread 10: FP100002273TRL1C002R02201648076/1, FP100002273TRL1C002R02201648076/2
        Thread 11: FP100002273TRL1C002R02802467045/1, FP100002273TRL1C002R02802467045/2
        Thread 12: FP100002273TRL1C002R02201759642/1, FP100002273TRL1C002R02201759642/2
        Thread 13: FP100002273TRL1C002R02802834059/1, FP100002273TRL1C002R02802834059/2
        Thread 14: FP100002273TRL1C002R02201042598/1, FP100002273TRL1C002R02201042598/2
        Thread 15: FP100002273TRL1C002R02201148191/1, FP100002273TRL1C002R02201148191/2
        Thread 16: FP100002273TRL1C002R02802693253/1, FP100002273TRL1C002R02802693253/2
        Thread 17: FP100002273TRL1C002R02201817008/1, FP100002273TRL1C002R02201817008/2
        Thread 18: FP100002273TRL1C002R02201766519/1, FP100002273TRL1C002R02201766519/2
        Thread 19: FP100002273TRL1C002R02201656133/1, FP100002273TRL1C002R02201656133/2
        Thread 20: FP100002273TRL1C002R02802902527/1, FP100002273TRL1C002R02802902527/2
        Thread 21: FP100002273TRL1C002R02201066894/1, FP100002273TRL1C002R02201066894/2
        Thread 22: FP100002273TRL1C002R02201345134/1, FP100002273TRL1C002R02201345134/2
        Thread 23: FP100002273TRL1C002R02802997475/1, FP100002273TRL1C002R02802997475/2
        Thread 24: FP100002273TRL1C002R02201293816/1, FP100002273TRL1C002R02201293816/2
        Thread 25: FP100002273TRL1C002R02802840563/1, FP100002273TRL1C002R02802840563/2
        Thread 26: FP100002273TRL1C002R02802441968/1, FP100002273TRL1C002R02802441968/2
        Thread 27: FP100002273TRL1C002R02200864243/1, FP100002273TRL1C002R02200864243/2
        Thread 28: FP100002273TRL1C002R02201276397/1, FP100002273TRL1C002R02201276397/2
        Thread 29: FP100002273TRL1C002R02802952612/1, FP100002273TRL1C002R02802952612/2
        Thread 30: FP100002273TRL1C002R03703415279/1, FP100002273TRL1C002R03703415279/2
        Thread 31: FP100002273TRL1C002R02201462248/1, FP100002273TRL1C002R02201462248/2
        Thread 32: FP100002273TRL1C002R02803159214/1, FP100002273TRL1C002R02803159214/2
        Thread 33: FP100002273TRL1C002R02801964057/1, FP100002273TRL1C002R02801964057/2
        Thread 34: FP100002273TRL1C002R02802791963/1, FP100002273TRL1C002R02802791963/2
        Thread 35: FP100002273TRL1C002R02201594551/1, FP100002273TRL1C002R02201594551/2
        Thread 36: FP100002273TRL1C002R02201117089/1, FP100002273TRL1C002R02201117089/2
        Thread 37: FP100002273TRL1C002R02201186825/1, FP100002273TRL1C002R02201186825/2
        Thread 38: FP100002273TRL1C002R02802273252/1, FP100002273TRL1C002R02802273252/2
        Thread 39: FP100002273TRL1C002R02802565223/1, FP100002273TRL1C002R02802565223/2
        Thread 40: FP100002273TRL1C002R02201097753/1, FP100002273TRL1C002R02201097753/2
        Thread 41: FP100002273TRL1C002R02803213839/1, FP100002273TRL1C002R02803213839/2
        Thread 42: FP100002273TRL1C002R02201712464/1, FP100002273TRL1C002R02201712464/2
        Thread 43: FP100002273TRL1C002R02803261933/1, FP100002273TRL1C002R02803261933/2
        Thread 44: FP100002273TRL1C002R02803056756/1, FP100002273TRL1C002R02803056756/2
        Thread 45: FP100002273TRL1C002R02201489049/1, FP100002273TRL1C002R02201489049/2
        Thread 46: FP100002273TRL1C002R02201543500/1, FP100002273TRL1C002R02201543500/2
        Thread 47: FP100002273TRL1C002R02200766266/1, FP100002273TRL1C002R02200766266/2
Found 48 threads with context.
━━━━━━━━━━━━━━━━━━━━
Please include this entire error log in your bug report!
━━━━━━━━━━━━━━━━━━━━
error[MinimizerMapper::faster_cap]: Minimizers seem impossible to disrupt in region 75 76 4 10
T29:    000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000011111111111111111111111111111111111111111111111111
T29:    000000000011111111112222222222333333333344444444445555555555666666666677777777778888888888999999999900000000001111111111222222222233333333334444444444
T29:    012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789
T29:    GAAAAATGATAACATTACTAAAAAGACTATAAGTATTGGGTTTCCTCCTAACAAGCGCCTTATTTAATGTCGCGGCACGACTAGGGACACTTGATTACTCAGACTTCATCAAGATGGTGTGCCTCTATCACTTCATTCCCCGATGCTTTA
T29:    ----------AACATTACTAAAAAGACTATAAGTATTGG------- (#3, 1 hits)
T29:                       -AAAAGACTATAAGTATTGGGTTTCCTCCT---------- (#8, 4 hits)
T29:                         ----------AGTATTGGGTTTCCTCCTAACAAGCGCCT-- (#7, 3 hits)
T29:                            ----------ATAAGGCGCTTGTTAGGAGGAAACCCAAT---------- (#2, 1 hits)
T29:                                       --GGGTTTCCTCCTAACAAGCGCCTTATTTA---------- (#1, 1 hits)
T29:                                          --TTTCCTCCTAACAAGCGCCTTATTTAATG-------- (#0, 1 hits)
T29:                                                     ----------TTATTTAATGTCGCGGCACGACTAGGGAC-- (#9, 4 hits)
T29:                                                                   -----GTCGCGGCACGACTAGGGACACTTGATTA----- (#14, 16 hits)
T29:                                                                    ----------TCTGAGTAATCAAGTGTCCCTAGTCGTGC---------- (#11, 10 hits)
T29:                                                                               -AGTCTGAGTAATCAAGTGTCCCTAGTCGT---------- (#12, 11 hits)
T29:                                                                                 ---ATGAAGTCTGAGTAATCAAGTGTCCCTAG---------- (#13, 13 hits)
T29:                                                                                      ----------CACACCATCTTGATGAAGTCTGAGTAATC-------- (#10, 7 hits)
T29:                                                                                               ----------GACTTCATCAAGATGGTGTGCCTCTATCA---------- (#5, 2 hits)
T29:                                                                                                          -----ATCAAGATGGTGTGCCTCTATCACTTCAT---------- (#4, 2 hits)
T29:                                                                                                                ----GATGGTGTGCCTCTATCACTTCATTCCCC--------- (#6, 2 hits)
Mininizer 3 agg start 0 length 46 core start 10 length 29
Mininizer 8 agg start 19 length 40 core start 20 length 29━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Crash report for vg v1.66.0
Caught signal 11 raised at address 0x16a6f4e; tracing with backward-cpp
━━━━━━━━━━━━━━━
━━━━━━━━━━━━━━━━━━━━━━━━━━━━Mininizer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Crash report for vg ━━━━━━━━━━━━━━━━━━━━
Crash report for vg v1.66.0
Caught signal 11━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━Stack trace (most recent call last) in thread 2187716:
#14   Object "", at 0xffffffffffffffff, in 
━━━/afs/enea.it/por/system/lsbatch/1751468751.258017: line 1: 2187617 Segmentation fault      (core dumped) vg giraffe -p -t ${THREADS} -Z ${PAN_REF}.gbz -d ${PAN_REF}.dist -m ${PAN_REF}.shortread.withzip.min -f ${FASTP1} -f ${FASTP2} -N ${SAMPLE} -R ${SAMPLE} > ${PAN_GAM_PREFIX}/${SAMPLE}/${FC}/${LANE}/${SAMPLE}.g
am

and

vg [warning]: System's vm.overcommit_memory setting is 2 (never overcommit). vg does not work well under these conditions; you may appear to run out of memory with plenty of memory left. Attempting to unsafely reconfigure jemalloc to deal better with this situation.
Guessing that ****/35_tomato_cactus_graph_noch0.d2.shortread.zipcodes is Short Read Zipcodes
Preparing Indexes
Loading Minimizer Index
Loading Zipcodes
Loading GBZ
Loading Distance Index v2
Paging in Distance Index v2
Initializing MinimizerMapper
Loading and initialization: 60.9567 seconds
Of which Distance Index v2 paging: 4.65921 seconds
Mapping reads to "-" (GAM)
--watchdog-timeout 10
--batch-size 512
--match 1
--mismatch 4
--gap-open 6
--gap-extend 1
--full-l-bonus 5
--max-multimaps 1
--hit-cap 10
--hard-hit-cap 500
--score-fraction 0.9
--max-min 500
--min-coverage-flank 250
--num-bp-per-min 1000
--downsample-window-length 18446744073709551615
--downsample-window-count 0
--distance-limit 200
--max-extensions 800
--max-alignments 8
--cluster-score 50
--pad-cluster-score 20
--cluster-coverage 0.3
--max-extension-mismatches 4
--extension-score 1
--extension-set 20
--rescue-attempts 15
--max-fragment-length 2000
--paired-distance-limit 2
--rescue-subgraph-size 4
--rescue-seed-limit 100
--mapq-score-window 0
--mapq-score-scale 1
--zipcode-tree-score-threshold 50
--pad-zipcode-tree-score-threshold 20
--zipcode-tree-coverage-threshold 0.3
--zipcode-tree-scale 2
--min-to-fragment 4
--max-to-fragment 10
--max-direct-chain 0
--gapless-extension-limit 0
--fragment-max-graph-lookback-bases 300
--fragment-max-graph-lookback-bases-per-base 0.03
--fragment-max-read-lookback-bases 18446744073709551615
--fragment-max-read-lookback-bases-per-base 1
--max-fragments 18446744073709551615
--fragment-max-indel-bases 2000
--fragment-max-indel-bases-per-base 0.2
--fragment-gap-scale 1
--fragment-points-per-possible-match 0
--fragment-score-fraction 0.1
--fragment-max-min-score 1.79769e+308
--fragment-min-score 60
--fragment-set-score-threshold 0
--min-chaining-problems 1
--max-chaining-problems 2147483647
--max-graph-lookback-bases 3000
--max-graph-lookback-bases-per-base 0.3
--max-read-lookback-bases 18446744073709551615
--max-read-lookback-bases-per-base 1
--max-indel-bases 2000
--max-indel-bases-per-base 0.2
--item-bonus 0
--item-scale 1
--gap-scale 1
--points-per-possible-match 0
--chain-score-threshold 100
--min-chains 4
--min-chain-score-per-base 0.01
--max-min-chain-score 200
--max-skipped-bases 0
--max-chains-per-tree 1
--max-chain-connection 100
--max-tail-length 100
--max-dp-cells 18446744073709551615
--max-tail-gap 18446744073709551615
--max-middle-gap 18446744073709551615
--max-tail-dp-length 30000
--max-middle-dp-length 2147483647
--wfa-max-mismatches 2
--wfa-max-mismatches-per-base 0.1
--wfa-max-max-mismatches 20
--wfa-distance 10
--wfa-distance-per-base 0.1
--wfa-max-distance 200
--min-unique-node-fraction 0
--rescue-algorithm dozeu
Using fragment length estimate: 359.669 +/- 87.6472
warning[vg::giraffe]: Refusing to perform too-large rescue alignment of 150 bp against 10890 bp ordered subgraph for read FP100002664BRL1C001R00205615706/2 which would use more than 1572864 cells and might exhaust Dozeu's allocator; suppressing further warnings.
━━━━━━━━━━━━━━━━━━━━
Crash report for vg v1.66.0 "Navetta"
Caught signal 11 raised at address 0x1301e44; tracing with backward-cpp
Stack trace (most recent call last) in thread 3767322:
#11   Object "", at 0xffffffffffffffff, in 
#10   Object "****/8.5.0/vg/1.66.0/bin/vg", at 0x2393cf3, in __clone
#9    Object "****/8.5.0/vg/1.66.0/bin/vg", at 0x22ed24a, in start_thread
#8    Object "****/8.5.0/vg/1.66.0/bin/vg", at 0x228fadd, in gomp_thread_start
#7    Object "****/8.5.0/vg/1.66.0/bin/vg", at 0x2292427, in gomp_team_barrier_wait_end
#6    Object "****/8.5.0/vg/1.66.0/bin/vg", at 0x2289d2a, in gomp_barrier_handle_tasks
#5    Object "****/8.5.0/vg/1.66.0/bin/vg", at 0x101394a, in unsigned long vg::io::paired_for_each_parallel_after_wait<vg::Alignment>(std::function<bool (vg::Alignment&, vg::Alignment&)>, std::function<void (vg::Alignment&, vg::Alignment&)>, std::function<bool ()>, unsi
gned long) [clone ._omp_fn.1]
#4    Object "****/8.5.0/vg/1.66.0/bin/vg", at 0xe53a43, in std::_Function_handler<void (vg::Alignment&, vg::Alignment&), main_giraffe(int, char**)::{lambda()#1}::operator()() const::{lambda(vg::Alignment&, vg::Alignment&)#9}>::_M_invoke(std::_Any_data const&, vg::Align
ment&, vg::Alignment&)
#3    Object "****/8.5.0/vg/1.66.0/bin/vg", at 0x13187e6, in vg::MinimizerMapper::map_paired(vg::Alignment&, vg::Alignment&, std::vector<std::pair<vg::Alignment, vg::Alignment>, std::allocator<std::pair<vg::Alignment, vg::Alignment> > >&)
#2    Object "****/8.5.0/vg/1.66.0/bin/vg", at 0x131438f, in vg::MinimizerMapper::map_paired(vg::Alignment&, vg::Alignment&)
#1    Object "****/8.5.0/vg/1.66.0/bin/vg", at 0x130986f, in void vg::MinimizerMapper::process_until_threshold_c<double>(unsigned long, std::function<double (unsigned long)> const&, std::function<bool (unsigned long, unsigned long)> const&, double, unsigned long, unsign
ed long, vg::LazyRNG&, std::function<bool (unsigned long, unsigned long)> const&, std::function<void (unsigned long)> const&, std::function<void (unsigned long)> const&) const [clone .constprop.0]
#0    Object "****/8.5.0/vg/1.66.0/bin/vg", at 0x1301e44, in std::_Function_handler<void (unsigned long), vg::MinimizerMapper::map_paired(vg::Alignment&, vg::Alignment&)::{lambda(unsigned long)#21}>::_M_invoke(std::_Any_data const&, unsigned long&&)

Library locations:
ERROR: Signal 11 occurred. VG has crashed. Visit https://github.com/vgteam/vg/issues/new/choose to report a bug.
━━━━━━━━━━━━━━━━━━━━
Context dump:
        Thread 0: FP100002664BRL1C015R04001683383/1, FP100002664BRL1C015R04001683383/2
        Thread 1: FP100002664BRL1C015R04001392525/1, FP100002664BRL1C015R04001392525/2
        Thread 2: FP100002664BRL1C016R00704376902/1, FP100002664BRL1C016R00704376902/2
        Thread 3: FP100002664BRL1C015R04001854046/1, FP100002664BRL1C015R04001854046/2
        Thread 4: FP100002664BRL1C015R04001701143/1, FP100002664BRL1C015R04001701143/2
        Thread 5: FP100002664BRL1C016R00704405890/1, FP100002664BRL1C016R00704405890/2
        Thread 6: FP100002664BRL1C016R00705112421/1, FP100002664BRL1C016R00705112421/2
        Thread 7: FP100002664BRL1C016R00705187271/1, FP100002664BRL1C016R00705187271/2
        Thread 8: FP100002664BRL1C015R04001620991/1, FP100002664BRL1C015R04001620991/2
        Thread 9: FP100002664BRL1C016R00704663841/1, FP100002664BRL1C016R00704663841/2
        Thread 10: FP100002664BRL1C015R04001378993/1, FP100002664BRL1C015R04001378993/2
        Thread 11: FP100002664BRL1C015R04000904017/1, FP100002664BRL1C015R04000904017/2
        Thread 12: FP100002664BRL1C015R04001223519/1, FP100002664BRL1C015R04001223519/2
        Thread 13: FP100002664BRL1C015R04001523057/1, FP100002664BRL1C015R04001523057/2
        Thread 14: FP100002664BRL1C016R00703446769/1, FP100002664BRL1C016R00703446769/2
        Thread 15: FP100002664BRL1C015R04001603032/1, FP100002664BRL1C015R04001603032/2
        Thread 16: FP100002664BRL1C015R04001996125/1, FP100002664BRL1C015R04001996125/2
        Thread 17: FP100002664BRL1C015R04001570878/1, FP100002664BRL1C015R04001570878/2
        Thread 18: FP100002664BRL1C015R04001945078/1, FP100002664BRL1C015R04001945078/2
        Thread 19: FP100002664BRL1C016R00705381037/1, FP100002664BRL1C016R00705381037/2
        Thread 20: FP100002664BRL1C015R04000698092/1, FP100002664BRL1C015R04000698092/2
        Thread 21: FP100002664BRL1C016R00704577890/1, FP100002664BRL1C016R00704577890/2
        Thread 22: FP100002664BRL1C015R04001742564/1, FP100002664BRL1C015R04001742564/2
        Thread 23: FP100002664BRL1C016R00703996707/1, FP100002664BRL1C016R00703996707/2
        Thread 24: FP100002664BRL1C015R04001158071/1, FP100002664BRL1C015R04001158071/2
        Thread 25: FP100002664BRL1C016R00704959185/1, FP100002664BRL1C016R00704959185/2
        Thread 26: FP100002664BRL1C016R00702735174/1, FP100002664BRL1C016R00702735174/2
        Thread 27: FP100002664BRL1C016R00704476289/1, FP100002664BRL1C016R00704476289/2
        Thread 28: FP100002664BRL1C015R04001540515/1, FP100002664BRL1C015R04001540515/2
        Thread 29: FP100002664BRL1C015R04105203431/1, FP100002664BRL1C015R04105203431/2
        Thread 30: FP100002664BRL1C016R00705274217/1, FP100002664BRL1C016R00705274217/2
        Thread 31: FP100002664BRL1C016R00704206299/1, FP100002664BRL1C016R00704206299/2
        Thread 32: FP100002664BRL1C016R00704711788/1, FP100002664BRL1C016R00704711788/2
        Thread 33: FP100002664BRL1C016R00704839061/1, FP100002664BRL1C016R00704839061/2
        Thread 34: FP100002664BRL1C015R04001832979/1, FP100002664BRL1C015R04001832979/2
        Thread 35: FP100002664BRL1C016R00705350763/1, FP100002664BRL1C016R00705350763/2
        Thread 36: FP100002664BRL1C016R00704933590/1, FP100002664BRL1C016R00704933590/2
        Thread 37: FP100002664BRL1C015R04001924264/1, FP100002664BRL1C015R04001924264/2
        Thread 38: FP100002664BRL1C015R04001447108/1, FP100002664BRL1C015R04001447108/2
        Thread 39: FP100002664BRL1C015R04001722668/1, FP100002664BRL1C015R04001722668/2
        Thread 40: FP100002664BRL1C016R00704989474/1, FP100002664BRL1C016R00704989474/2
        Thread 41: FP100002664BRL1C016R00704691637/1, FP100002664BRL1C016R00704691637/2
        Thread 42: FP100002664BRL1C015R04002016170/1, FP100002664BRL1C015R04002016170/2
        Thread 43: FP100002664BRL1C016R00704622351/1, FP100002664BRL1C016R00704622351/2
        Thread 44: FP100002664BRL1C016R00705162702/1, FP100002664BRL1C016R00705162702/2
        Thread 45: FP100002664BRL1C015R04001323018/1, FP100002664BRL1C015R04001323018/2
        Thread 46: FP100002664BRL1C016R00705252584/1, FP100002664BRL1C016R00705252584/2
        Thread 47: FP100002664BRL1C016R00705138621/1, FP100002664BRL1C016R00705138621/2
Found 48 threads with context.
━━━━━━━━━━━━━━━━━━━━
Please include this entire error log in your bug report!
━━━━━━━━━━━━━━━━━━━━
error[MinimizerMapper::faster_cap]: Minimizers seem impossible to disrupt in region 49 51 1 9
T13:    000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000011111111111111111111111111111111111111111111111111
T13:    000000000011111111112222222222333333333344444444445555555555666666666677777777778888888888999999999900000000001111111111222222222233333333334444444444
T13:    012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789
T13:    GGCAACACAAGACATGTTTGGCACTAGTTGCAGATGAGACAGCTGCGGTTCACTTTCAAATGTGGGGTGATGAATGTGATGTTTTTGAGCCTGGGGATATCATCCGGTTAGAAAATGGCATCTTTTCTTATATTCGCAATAACCGTAATA
T13:    ---TGCAACTAGTGCCAAACATGTCTTGTGTT---------- (#14, 1 hits)
T13:        ----------GCTGTCTCATCTGCAACTAGTGCCAAACA---------- (#0, 1 hits)
T13:                   AGCTGTCTCATCTGCAACTAGTGCCAAAC---------- (#8, 1 hits)
T13:                    -----CACTAGTTGCAGATGAGACAGCTGCGGTT---------- (#6, 1 hits)
T13:                          ----AAGTGAACCGCAGCTGTCTCATCTGCAAC-------- (#1, 1 hits)
T13:                             ----------CACATTTGAAAGTGAACCGCAGCTGTCTC--- (#12, 1 hits)
T13:                                 ----------ACCCCACATTTGAAAGTGAACCGCAGCTG-------- (#4, 1 hits)
T13:                                          ----------TTCACTTTCAAATGTGGGGTGATGAATGT---------- (#3, 1 hits)
T13:                                                     -CACTTTCAAATGTGGGGTGATGAATGTGA---------- (#10, 1 hits)
T13:                                                       ---AACATCACATTCATCACCCCACATTTGAA---------- (#5, 1 hits)
T13:                                                                                     ----------TGCCATTTTCTAACCGGATGATATCCCCA------ (#7, 1 hits)
T13:                                                                                            ----------ATCATCCGGTTAGAAAATGGCATCTTTTC---------- (#13, 1 hits)
T13:                                                                                                       ---TCCGGTTAGAAAATGGCATCTTTTCTTAT---------- (#15, 1 hits)
T13:                                                                                                           TATAAGAAAAGATGCCATTTTCTAACCGG---------- (#11, 1 hits)
T13:                                                                                                            CGGTTAGAAAATGGCATCTTTTCTTATAT---------- (#2, 1 hits)
T13:                                                                                                             ---GCGAATATAAGAAAAGATGCCATTTTCTA-------- (#9, 1 hits)
Mininizer 14 agg start 0 length 42 core start ━━━━━31 length 29
Mininizer 0 agg start 4 length 49 core start 42 length 29
Mininizer 8 agg start 15 length 39 core start 43 length 29
Mininizer 6 agg start 16 length 44 core start 21 length 29
Mininizer 1 agg start 22 length 41 core start 54 length 29
Mininizer 12 agg start 25 length 42 core start 63 length 29
Mininizer 4 agg start 29 length 47 core start 67 length 29
Mininizer 3 agg start 38━━━━━━━━━━━━━━━━━━━━━━━
Crash report for vg v1.66.0
Caught signal 11 raised at address 0x16a6f4e; tracing with backward-cpp
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Crash report for vg ━━━━━━━━━━━━━━━━━━━━━━━━━━
Crash report for vg v1.66.0
Caught signal 11 raised at address 0x16a6f4e; tracing with backward-cpp
━━━━Stack trace (most recent call last) in thread 3767335:
#15   Object "", at 0xffffffffffffffff, in 
━━━━━━━/afs/enea.it/por/system/lsbatch/1751468751.258018: line 1: 3767223 Segmentation fault      (core dumped) vg giraffe -p -t ${THREADS} -Z ${PAN_REF}.gbz -d ${PAN_REF}.dist -m ${PAN_REF}.shortread.withzip.min -f ${FASTP1} -f ${FASTP2} -N ${SAMPLE} -R ${SAMPLE} > ${PAN_GAM_PREFIX}/${SAMPLE}/${FC}/${LANE}/${SAMPL
E}.gam

[adamnovak]

That "Minimizers seem impossible to disrupt" message is mine; I'll have to take a look at it in depth tomorrow, since looking at the little diagram it draws there doesn't look to be anything particularly wrong with the minimizer layout in the reads.

@linsson I've never seen read names that look like FP100002273TRL1C002R02801988469 before. Are these Illumina reads, or something else? Do they maybe somehow have unusually high base qualities, which might be making Giraffe think that sequencing errors at some locations are effectively impossible?

[linsson]

That "Minimizers seem impossible to disrupt" message is mine; I'll have to take a look at it in depth tomorrow, since looking at the little diagram it draws there doesn't look to be anything particularly wrong with the minimizer layout in the reads.

@linsson I've never seen read names that look like FP100002273TRL1C002R02801988469 before. Are these Illumina reads, or something else? Do they maybe somehow have unusually high base qualities, which might be making Giraffe think that sequencing errors at some locations are effectively impossible?

It's dnbseq by bgi, not illumina. I also used fastp for cleaning.

[adamnovak]

I'm unable to replicate this problem with faster_cap() with either read when using all the minimizers, or with the second read when using exactly the minimizers shown in the crash report, when I set all the base qualities to Phred 255 (which is the highest quality vg can contemplate). This is with 2e75e2a (which adds some tests to try and trigger this) on top of the mainline commit d5b0008. The differences between 1.66.0 and there don't really seem to touch the relevant code. Switching form ARM/MacOS/Clang to x86_64/Linux/GCC didn't seem to induce the crash either.

[adamnovak]

I now think the Minimizers seem impossible to disrupt error might be a red herring. It happens after we've already reported a signal 11 from the signal handler, and generated a stack trace for that. We've already got undefined behavior kicking around in the process's memory space, so for all I know the crashing thread has been busily writing 0s over some other thread's scratch space for probability calculations and it causing it to get wrong answers from its math that it then complains about. The real problem is the signal 11 crash.

It looks like that's happening inside an unremarkable vector append/reallocation, from what's in the stack trace and from looking at the blamed addresses/symbols on a debug build of vg.

I'd look into the warning next: vg [warning]: System's vm.overcommit_memory setting is 2 (never overcommit). vg does not work well under these conditions; you may appear to run out of memory with plenty of memory left. Attempting to unsafely reconfigure jemalloc to deal better with this situation.

Maybe whatever we're doing to jemalloc that's unsafe in theory is also unsafe in practice on your system (or just is broken, since that code isn't often exercised), and something bad happens to the internal memory arenas that shows up as threads stepping on each others' memory. I can look into a workaround for turning it off, or you can try a build built with make clean && make -j16 jemalloc=off to see if the problem goes away when using the normal allocator.

Alternatively, it could be that an out-of-memory issue is causing failed allocations. I don't think vg is guaranteed to crash cleanly when memory allocations fail, because it contains unchecked allocations we haven't gotten around to removing. I wouldn't think a vector insert would be one of those, but maybe that's not the real original source of the misbehavior and it's a result of some earlier memory corruption caused by writing into a failed allocation. @linsson how much of your allocated memory limit did the crashing jobs use?