Regarding --low_ram_primary_clustering, the error message shows OSError: [Errno 7] Argument list too long

[QiaofanLi]

Dear Matt Olm,

I have a dataset of 169k MAGs. I can't use the default dereplicate parameters in dRep (it would exceed memory). Therefore, I used the following parameters:
dRep dereplicate f__Oscillospiraceae.res -g f__Oscillospiraceae.list -comp 70 -con 5 -d --genomeInfo quality_report_without_0.csv -p 96 --skip_plots -pa 0.8 --low_ram_primary_clustering --primary_chunksize 20000

When running with these parameters, I get the following error:

---

..:: dRep dereplicate Step 1. Filter ::..

---

Will filter the genome list
Loading genomes from a list
169,453 genomes were input to dRep
Calculating genome info of genomes
100.00% of genomes passed length filtering
409.86% of genomes passed checkM filtering

---

..:: dRep dereplicate Step 2. Cluster ::..

---

Running primary clustering
Running pair-wise MASH clustering
Will split genomes into 9 groups for primary clustering
Traceback (most recent call last):
File "/usr/local/bin/dRep", line 32, in
Controller().parseArguments(args)
File "/usr/local/lib/python3.10/dist-packages/drep/controller.py", line 100, in parseArguments
self.dereplicate_operation(**vars(args))
File "/usr/local/lib/python3.10/dist-packages/drep/controller.py", line 48, in dereplicate_operation
drep.d_workflows.dereplicate_wrapper(kwargs['work_directory'],**kwargs)
File "/usr/local/lib/python3.10/dist-packages/drep/d_workflows.py", line 37, in dereplicate_wrapper
drep.d_cluster.controller.d_cluster_wrapper(wd, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/drep/d_cluster/controller.py", line 184, in d_cluster_wrapper
GenomeClusterController(workDirectory, **kwargs).main()
File "/usr/local/lib/python3.10/dist-packages/drep/d_cluster/controller.py", line 32, in main
self.run_primary_clustering()
File "/usr/local/lib/python3.10/dist-packages/drep/d_cluster/controller.py", line 100, in run_primary_clustering
Mdb, Cdb, cluster_ret = drep.d_cluster.compare_utils.all_vs_all_MASH(self.Bdb, self.wd.get_dir('MASH'), **self.kwargs)
File "/usr/local/lib/python3.10/dist-packages/drep/d_cluster/compare_utils.py", line 110, in all_vs_all_MASH
genome_chunks = run_mash_on_genome_chunks(genome_chunks, mash_exe, sketch_folder, MASH_folder, logdir, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/drep/d_cluster/compare_utils.py", line 180, in run_mash_on_genome_chunks
drep.thread_cmds(cmds, logdir=logdir, t=int(p))
File "/usr/local/lib/python3.10/dist-packages/drep/init.py", line 56, in thread_cmds
pool.map(thread_cmd_wrapper, tups)
File "/usr/lib/python3.10/multiprocessing/pool.py", line 367, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/lib/python3.10/multiprocessing/pool.py", line 774, in get
raise self._value
File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/usr/local/lib/python3.10/dist-packages/drep/init.py", line 51, in thread_cmd_wrapper
run_cmd(*tup)
File "/usr/local/lib/python3.10/dist-packages/drep/init.py", line 47, in run_cmd
call(cmd,stdout=sto, stderr=ste)
File "/usr/lib/python3.10/subprocess.py", line 345, in call
with Popen(*popenargs, **kwargs) as p:
File "/usr/lib/python3.10/subprocess.py", line 971, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.10/subprocess.py", line 1863, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 7] Argument list too long: '/home/mash-Linux64-v2.3/mash'

How should this situation be resolved? What confuses me is that if I have 700,000 MAGs, how should I input them all at once into dRep for dereplication?

Best Regards!

[MrOlm]

Hello,

I think this is something I addressed in later versions of dRep. Can you confirm you’re using the latest version?

Thanks,
Matt

[QiaofanLi]

Hello,

Thanks for your reply!

I think I have the latest version installed:

dRep -h
...::: dRep v3.6.2 :::...

Matt Olm. MIT License. Banfield Lab, UC Berkeley. 2017 (last updated 2025)

See https://drep.readthedocs.io/en/latest/index.html for documentation
Choose one of the operations below for more detailed help.

Example: dRep dereplicate -h

Commands:
compare -> Compare and cluster a set of genomes
dereplicate -> De-replicate a set of genomes
check_dependencies -> Check which dependencies are properly installed

And dRep check_dependencies shows the following:

mash.................................... all good (location = /home/mash-Linux64-v2.3/mash)
nucmer.................................. all good (location = /home/mummer-4.0.1/nucmer)
checkm.................................. all good (location = /usr/local/bin/checkm)
ANIcalculator........................... !!! ERROR !!! (location = None)
prodigal................................ all good (location = /usr/local/bin/prodigal)
centrifuge.............................. !!! ERROR !!! (location = None)
nsimscan................................ !!! ERROR !!! (location = None)
fastANI................................. all good (location = /home/fastANI)
skani................................... !!! ERROR !!! (location = None)

Does this mean I need to use different parameters, or avoid using certain parameters?

By the way, I also want to ask a stupid question: if I have 700,000 genomes, which is too many, how can I dereplicate them all at once?

Best Regards!

[QiaofanLi]

Hello,

I think this is something I addressed in later versions of dRep. Can you confirm you’re using the latest version?

Thanks, Matt

Dear Matt Olm,

I have tried using conda to create a fresh environment and install the latest version of dRep, but this problem still occurs. Could you please tell me how to avoid/solve this problem?

Best Regards!

[MrOlm]

I forget the argument name, but you need to increase the primary clustering chunk size. Lmk if you can’t find it

[QiaofanLi]

Thanks for your quick reply!

I'm not sure whether I should modify this via a command-line argument or directly in the source code. I've also searched the codebase for keywords like chunk size and mash, but since I'm not familiar with the project structure, I couldn't pinpoint which file and variable to change.

Could you please point me to the specific command-line argument, or perhaps the file or something that needs to be modified?

I'd also like to double-check, are you suggesting I increase the --primary_chunksize value from 20000? My initial thought was that it might need to be decreased, so I just want to be sure I'm on the right track before proceeding.

Thanks again for your help!

[MrOlm]

OK- --low_ram_primary_clustering doesn't work (it's a deprecated function); you just want to run --multiround_primary_clustering instead. I'll warn you, though, that this can still take lots of RAM when doing that many genomes; it just depends on how similar they are (if all 700,000 are E. coli, it'll work no problem; if each genome is a different species, this will use lots and lots of RAM)

Best,
Matt

[QiaofanLi]

OK- --low_ram_primary_clustering doesn't work (it's a deprecated function); you just want to run --multiround_primary_clustering instead. I'll warn you, though, that this can still take lots of RAM when doing that many genomes; it just depends on how similar they are (if all 700,000 are E. coli, it'll work no problem; if each genome is a different species, this will use lots and lots of RAM)

Best, Matt

Hi Matt,

Thank you very much for your concern and helpful reply regarding the deprecated --low_ram_primary_clustering function. Following your advice, I've taken the following steps:

1. I set up a fresh conda virtual environment to ensure there were no conflicts.
2. I installed the latest version of dRep to make sure I'm using the most up-to-date code.
3. I tested the --multiround_primary_clustering parameter as you suggested, using a random subset of 100,000 of my MAGs.
4. Initially, I set the --primary_chunksize to 50,000, but the run failed. I then reverted the --primary_chunksize to the default value (it should be 5000), but unfortunately, I'm still encountering the same error.

The error message I'm receiving is:

dRep dereplicate test_multiround -comp 50 -con 10 -d --genomeInfo quality_report_without_0.csv -p 96 --skip_plots -pa 0.8 --multiround_primary_clustering
***************************************************
    ..:: dRep dereplicate Step 1. Filter ::..
***************************************************

Will filter Bdb
100,000 genomes were input to dRep
Calculating genome info of genomes
100.00% of genomes passed length filtering
858.41% of genomes passed checkM filtering
***************************************************
    ..:: dRep dereplicate Step 2. Cluster ::..
***************************************************

Running primary clustering
Running pair-wise MASH clustering
  Will split genomes into 20 groups for primary clustering
Traceback (most recent call last):
  File "/data/lqf/miniconda3/envs/drep/bin/dRep", line 32, in <module>
    Controller().parseArguments(args)
  File "/data/lqf/miniconda3/envs/drep/lib/python3.11/site-packages/drep/controller.py", line 100, in parseArguments
    self.dereplicate_operation(**vars(args))
  File "/data/lqf/miniconda3/envs/drep/lib/python3.11/site-packages/drep/controller.py", line 48, in dereplicate_operation
    drep.d_workflows.dereplicate_wrapper(kwargs['work_directory'],**kwargs)
  File "/data/lqf/miniconda3/envs/drep/lib/python3.11/site-packages/drep/d_workflows.py", line 37, in dereplicate_wrapper
    drep.d_cluster.controller.d_cluster_wrapper(wd, **kwargs)
  File "/data/lqf/miniconda3/envs/drep/lib/python3.11/site-packages/drep/d_cluster/controller.py", line 184, in d_cluster_wrapper
    GenomeClusterController(workDirectory, **kwargs).main()
  File "/data/lqf/miniconda3/envs/drep/lib/python3.11/site-packages/drep/d_cluster/controller.py", line 32, in main
    self.run_primary_clustering()
  File "/data/lqf/miniconda3/envs/drep/lib/python3.11/site-packages/drep/d_cluster/controller.py", line 100, in run_primary_clustering
    Mdb, Cdb, cluster_ret = drep.d_cluster.compare_utils.all_vs_all_MASH(self.Bdb, self.wd.get_dir('MASH'), **self.kwargs)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/lqf/miniconda3/envs/drep/lib/python3.11/site-packages/drep/d_cluster/compare_utils.py", line 110, in all_vs_all_MASH
    genome_chunks = run_mash_on_genome_chunks(genome_chunks, mash_exe, sketch_folder, MASH_folder, logdir,  **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/lqf/miniconda3/envs/drep/lib/python3.11/site-packages/drep/d_cluster/compare_utils.py", line 180, in run_mash_on_genome_chunks
    drep.thread_cmds(cmds, logdir=logdir, t=int(p))
  File "/data/lqf/miniconda3/envs/drep/lib/python3.11/site-packages/drep/__init__.py", line 56, in thread_cmds
    pool.map(thread_cmd_wrapper, tups)
  File "/data/lqf/miniconda3/envs/drep/lib/python3.11/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/lqf/miniconda3/envs/drep/lib/python3.11/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/data/lqf/miniconda3/envs/drep/lib/python3.11/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "/data/lqf/miniconda3/envs/drep/lib/python3.11/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
           ^^^^^^^^^^^^^^^^
  File "/data/lqf/miniconda3/envs/drep/lib/python3.11/site-packages/drep/__init__.py", line 51, in thread_cmd_wrapper
    run_cmd(*tup)
  File "/data/lqf/miniconda3/envs/drep/lib/python3.11/site-packages/drep/__init__.py", line 47, in run_cmd
    call(cmd,stdout=sto, stderr=ste)
  File "/data/lqf/miniconda3/envs/drep/lib/python3.11/subprocess.py", line 389, in call
    with Popen(*popenargs, **kwargs) as p:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/lqf/miniconda3/envs/drep/lib/python3.11/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/data/lqf/miniconda3/envs/drep/lib/python3.11/subprocess.py", line 1955, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 7] Argument list too long: '/data/lqf/miniconda3/envs/drep/bin/mash'

Could you please take a look at this? I'm wondering if there might be an underlying issue that's causing this behavior, even with the recommended parameters.

And the complete log file generated is as follows:

09-30 03:43 DEBUG    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
09-30 03:43 DEBUG    ***Logger started up at /data/part8/test_multiround/log/logger.log***
09-30 03:43 DEBUG    Command to run dRep was: /data/lqf/miniconda3/envs/drep/bin/dRep dereplicate test_multiround -g file_list_path.tsv -comp 50 -con 10 -d --genomeInfo quality_report_without_0.csv -p 96 --skip_plots -pa 0.8 --multiround_primary_clustering --primary_chunksize 50000

09-30 03:43 DEBUG    dRep version 3.6.2 was run

09-30 03:43 DEBUG    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

09-30 03:43 DEBUG    Namespace(operation='dereplicate', work_directory='test_multiround', processors=96, debug=True, genomes=['file_list_path.tsv'], length=50000, completeness=50.0, contamination=10.0, ignoreGenomeQuality=False, genomeInfo='quality_report_without_0.csv', checkM_method='lineage_wf', set_recursion='0', checkm_group_size=2000, S_algorithm='fastANI', MASH_sketch=1000, SkipMash=False, SkipSecondary=False, skani_extra='', n_PRESET='normal', P_ani=0.8, S_ani=0.95, cov_thresh=0.1, coverage_method='larger', clusterAlg='average', low_ram_primary_clustering=False, multiround_primary_clustering=True, primary_chunksize=50000, greedy_secondary_clustering=False, run_tertiary_clustering=False, completeness_weight=1, contamination_weight=5, strain_heterogeneity_weight=1, N50_weight=0.5, size_weight=0, centrality_weight=1, extra_weight_table=None, gen_warnings=False, warn_dist=0.25, warn_sim=0.98, warn_aln=0.25, skip_plots=True)
09-30 03:43 DEBUG    Starting the dereplicate operation
09-30 03:43 INFO     ***************************************************
    ..:: dRep dereplicate Step 1. Filter ::..
***************************************************

09-30 03:43 DEBUG    Loading work directory in filter
09-30 03:43 DEBUG    Located: /data/part8/test_multiround
Datatables: []
Cluster files: []
Arguments: []
09-30 03:43 DEBUG    Validating filter arguments
09-30 03:43 INFO     Will filter the genome list
09-30 03:43 INFO     Loading genomes from a list
09-30 03:43 INFO     100,000 genomes were input to dRep
09-30 03:43 INFO     Calculating genome info of genomes
09-30 03:58 DEBUG    Filtering genomes by size
09-30 03:58 INFO     100.00% of genomes passed length filtering
09-30 03:58 DEBUG    Loading provided genome quality information
09-30 03:58 DEBUG    HERE IS GENOME INFO:
09-30 03:58 DEBUG
                     genome  completeness  contamination          completeness_model_used  ...  genome_size  gc_content  total_coding_sequences  additional_notes
0  107205891_SemiBin_102.fa         87.96           5.57  Neural Network (Specific Model)  ...      2538633        0.60                    2073               NaN
1  107205891_SemiBin_104.fa         70.36           1.47  Neural Network (Specific Model)  ...      2153662        0.51                    2104               NaN
2  107205891_SemiBin_105.fa         99.99           1.27  Neural Network (Specific Model)  ...      2627346        0.48                    2401               NaN
3  107205891_SemiBin_108.fa         69.00           2.34  Neural Network (Specific Model)  ...      1784475        0.61                    1719               NaN
4  107205891_SemiBin_109.fa         71.53           1.03  Neural Network (Specific Model)  ...      1480515        0.59                    1475               NaN

[5 rows x 12 columns]
09-30 03:58 DEBUG    There are the columns: ['genome', 'completeness', 'contamination', 'completeness_model_used', 'translation_table_used', 'coding_density', 'contig_n50', 'average_gene_length', 'genome_size', 'gc_content', 'total_coding_sequences', 'additional_notes']
09-30 05:19 DEBUG    Filtering genomes
09-30 05:19 INFO     858.41% of genomes passed checkM filtering
09-30 05:19 DEBUG    Storing resulting files
09-30 05:19 INFO     ***************************************************
    ..:: dRep dereplicate Step 2. Cluster ::..
***************************************************

09-30 05:23 INFO     Running primary clustering
09-30 05:23 INFO     Running pair-wise MASH clustering
09-30 05:23 INFO       Will split genomes into 2 groups for primary clustering
09-30 06:01 DEBUG    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
09-30 06:01 DEBUG    ***Logger started up at /data/part8/test_multiround/log/logger.log***
09-30 06:01 DEBUG    Command to run dRep was: /data/lqf/miniconda3/envs/drep/bin/dRep dereplicate test_multiround -g file_list_path.tsv -comp 50 -con 10 -d --genomeInfo quality_report_without_0.csv -p 96 --skip_plots -pa 0.8 --multiround_primary_clustering

09-30 06:01 DEBUG    dRep version 3.6.2 was run

09-30 06:01 DEBUG    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

09-30 06:01 DEBUG    Namespace(operation='dereplicate', work_directory='test_multiround', processors=96, debug=True, genomes=['file_list_path.tsv'], length=50000, completeness=50.0, contamination=10.0, ignoreGenomeQuality=False, genomeInfo='quality_report_without_0.csv', checkM_method='lineage_wf', set_recursion='0', checkm_group_size=2000, S_algorithm='fastANI', MASH_sketch=1000, SkipMash=False, SkipSecondary=False, skani_extra='', n_PRESET='normal', P_ani=0.8, S_ani=0.95, cov_thresh=0.1, coverage_method='larger', clusterAlg='average', low_ram_primary_clustering=False, multiround_primary_clustering=True, primary_chunksize=5000, greedy_secondary_clustering=False, run_tertiary_clustering=False, completeness_weight=1, contamination_weight=5, strain_heterogeneity_weight=1, N50_weight=0.5, size_weight=0, centrality_weight=1, extra_weight_table=None, gen_warnings=False, warn_dist=0.25, warn_sim=0.98, warn_aln=0.25, skip_plots=True)
09-30 06:01 DEBUG    Starting the dereplicate operation
09-30 06:01 INFO     ***************************************************
    ..:: dRep dereplicate Step 1. Filter ::..
***************************************************

09-30 06:01 DEBUG    Loading work directory in filter
09-30 06:01 DEBUG    Located: /data/part8/test_multiround
Datatables: ['genomeInfo', 'Bdb']
Cluster files: []
Arguments: []
09-30 06:01 DEBUG    Validating filter arguments
09-30 06:01 ERROR    Both Bdb and a genome list are found- either don't include a genome list or start a new work directory!
09-30 06:01 DEBUG    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
09-30 06:01 DEBUG    ***Logger started up at /data/part8/test_multiround/log/logger.log***
09-30 06:01 DEBUG    Command to run dRep was: /data/lqf/miniconda3/envs/drep/bin/dRep dereplicate test_multiround -comp 50 -con 10 -d --genomeInfo quality_report_without_0.csv -p 96 --skip_plots -pa 0.8 --multiround_primary_clustering

09-30 06:01 DEBUG    dRep version 3.6.2 was run

09-30 06:01 DEBUG    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

09-30 06:01 DEBUG    Namespace(operation='dereplicate', work_directory='test_multiround', processors=96, debug=True, genomes=None, length=50000, completeness=50.0, contamination=10.0, ignoreGenomeQuality=False, genomeInfo='quality_report_without_0.csv', checkM_method='lineage_wf', set_recursion='0', checkm_group_size=2000, S_algorithm='fastANI', MASH_sketch=1000, SkipMash=False, SkipSecondary=False, skani_extra='', n_PRESET='normal', P_ani=0.8, S_ani=0.95, cov_thresh=0.1, coverage_method='larger', clusterAlg='average', low_ram_primary_clustering=False, multiround_primary_clustering=True, primary_chunksize=5000, greedy_secondary_clustering=False, run_tertiary_clustering=False, completeness_weight=1, contamination_weight=5, strain_heterogeneity_weight=1, N50_weight=0.5, size_weight=0, centrality_weight=1, extra_weight_table=None, gen_warnings=False, warn_dist=0.25, warn_sim=0.98, warn_aln=0.25, skip_plots=True)
09-30 06:01 DEBUG    Starting the dereplicate operation
09-30 06:01 INFO     ***************************************************
    ..:: dRep dereplicate Step 1. Filter ::..
***************************************************

09-30 06:01 DEBUG    Loading work directory in filter
09-30 06:01 DEBUG    Located: /data/part8/test_multiround
Datatables: ['genomeInfo', 'Bdb']
Cluster files: []
Arguments: []
09-30 06:01 DEBUG    Validating filter arguments
09-30 06:01 INFO     Will filter Bdb
09-30 06:01 INFO     100,000 genomes were input to dRep
09-30 06:01 INFO     Calculating genome info of genomes
09-30 06:06 DEBUG    Filtering genomes by size
09-30 06:06 INFO     100.00% of genomes passed length filtering
09-30 06:06 DEBUG    Loading provided genome quality information
09-30 06:06 DEBUG    HERE IS GENOME INFO:
09-30 06:06 DEBUG
                     genome  completeness  contamination          completeness_model_used  ...  genome_size  gc_content  total_coding_sequences  additional_notes
0  107205891_SemiBin_102.fa         87.96           5.57  Neural Network (Specific Model)  ...      2538633        0.60                    2073               NaN
1  107205891_SemiBin_104.fa         70.36           1.47  Neural Network (Specific Model)  ...      2153662        0.51                    2104               NaN
2  107205891_SemiBin_105.fa         99.99           1.27  Neural Network (Specific Model)  ...      2627346        0.48                    2401               NaN
3  107205891_SemiBin_108.fa         69.00           2.34  Neural Network (Specific Model)  ...      1784475        0.61                    1719               NaN
4  107205891_SemiBin_109.fa         71.53           1.03  Neural Network (Specific Model)  ...      1480515        0.59                    1475               NaN

[5 rows x 12 columns]
09-30 06:06 DEBUG    There are the columns: ['genome', 'completeness', 'contamination', 'completeness_model_used', 'translation_table_used', 'coding_density', 'contig_n50', 'average_gene_length', 'genome_size', 'gc_content', 'total_coding_sequences', 'additional_notes']
09-30 07:28 DEBUG    Filtering genomes
09-30 07:28 INFO     858.41% of genomes passed checkM filtering
09-30 07:28 DEBUG    Storing resulting files
09-30 07:28 INFO     ***************************************************
    ..:: dRep dereplicate Step 2. Cluster ::..
***************************************************

09-30 07:32 INFO     Running primary clustering
09-30 07:32 INFO     Running pair-wise MASH clustering
09-30 07:32 INFO       Will split genomes into 20 groups for primary clustering

Please let me know if you need any more detailed information from my end, such as the full input/output or anything else that might help with debugging.

Thank you again for your time and assistance.

Best regards,
Qiaofan

[MrOlm]

Hi @QiaofanLi -

This is an odd error that I haven't encountered before. I'll look into when I have time.

I the meantime, shorting the path names should help. The limit is about total command length. Move/symlink your FASTAs into a very short path:

mkdir /g
# create short symlinks with short names
awk '{printf "ln -s %s /g/g%06d.fa\n", $0, NR}' file_list_path.tsv | bash
# then point dRep to a new list with /g/g000001.fa ... entries
ls /g/*.fa > /g/list.tsv

Re-run with /g/list.tsv. This alone can be the difference.

Sorry about this bug!
Matt