Dark matter detector followups

[dkoslicki]

Ideas to validate that we can detect dark matter (DMI=Dark matter index):

1. Rig "dark matter" simulations. Eg. take a bunch of human gut metagenomes, toss out frequently known "major players" from the reference DB, see if DMI goes up, while DMI stays about the same for environments very different from human gut
2. Pull down ~100 samples, get indep. measure of DM, compare against DMI. Eg. align reads to big ref. DB, quantify % unmapped. Assemble and quantify contigs not taxonomically binned

Connect to AVA approach: After defining the DMI, run the all-vs-all Jaccard on these, look for ones with high overlap. Do co-assembly and cross fingers that this helps pull out novel genomes while individual assemblies do not.

[dkoslicki]

When running it on the /scratch/dmk333_new/Logan/Logan_Analyses/verify_diversity_correlation/data/hash_diversity_results_full_cov_0.0/data/hash_diversity_data.parquet, this is missing a lot of the metadata that is in the analysis part of the filtering. Will need to have a helper script or the like to copy the DMI column over to things like /scratch/dmk333_new/Logan/Logan_Analyses/verify_diversity_correlation/data/hash_diversity_results_full_cov_0.0/analysis/filtered_analysis_min_mbases_1620_min_diversity_10/filtered_data.parquet

Done in the next commit

[dkoslicki]

Found that we will need to remove host contamination. Working on that now