Guidelines for large sample sizes

[igordot]

Do you have updated guidance for handling large datasets (>100 samples and/or >500,000 cells)?

A previous issue addressed this topic nearly 4 years ago (#108), but I'm hoping you have more insights now. The clearest recommendation from that discussion was: "for very large datasets with many samples, use large k~[50, 100] and small prop~[0.01, 0.1] to reduce neighborhood redundancy."

The main concerns are:

• Sample representation: How do I ensure each neighborhood captures enough cells from each sample? Should k scale with sample size?
• Computational constraints: Does adjusting prop address memory/computation limits, or are other strategies needed?

[MikeDMorgan]

Milo2.0 handles exactly these cases: https://doi.org/10.1101/2023.11.08.566176

The latest version on Bioconductor is 2.6.0 which implements the graph-based approach described in the above pre-print. N=100 and 500k cells is well within Milo's capabilities. As with any computational analysis, parameters need to be tuned to the specific data set. The recommendation of refining k between 50 and 100 stands, as does tweaking the prop and d parameters to ensure good representation of samples and conditions in neighbourhoods. As an example the above preprint performs several analyses using N~90-120 on a standard laptop.

[igordot]

Yes, I've used Milo in the past and the graph-based approach is a great enhancement.

Thank you for confirming that k should be 50-100. Do you still recommend the average neighborhood size to be around 50-100 cells regardless of the number of samples? I am concerned that then the neighborhoods become more sparse as the number of samples increases.

[MikeDMorgan]

The recommended neighbourhood size is the average (median) size is at least 100. I would also recommend check that you have neighbourhoods with non-zero counts from the majority of samples as well. Inspecting the histogram from plotNhoodSizeHist can help guide you there.

[igordot]

The vignette mentions "we want to have an average neighbourhood size over 5 x N_samples". That would be 500 if you have 100 samples. How would you achieve that if k is capped at 100? Or should the average neighborhood size not go over 100?

[MikeDMorgan]

The original recommendation of 5xN_samples is for smaller studies - in reality you can still achieve neighbourhood sizes > k. The k is for the NN finding algorithm, but once the graph is constructed each cell can have > k edges because it is not a shared nearest-neighbour graph. The guideline for k=50-100 balances the computational demand of finding 100's NNs for each cell when there are hundreds of thousands of cells.

Note that I said the median nhood size should be at least 100 not that this is an upper bound.