Find a better way to generate a null distribution

[axiomcura]

Shuffle data may be not enough. What we can do is change how we shuffle the data instead of full column w can just shuffle the labels (which is much easier and computational less expensive)

Might be impossible.

[axiomcura]

It seem we might have found some issue with the shuffling itself. Based on this comment in #64 #64 (comment)

also find the shuffling part to be quite confusing now i see the full picture. If my understanding is correct, there we are using the previously generated clusters and then first compute the on and off signatures using references (which are also shuffled in terms of features).
In the end some the signatures from shuffled reference, previously generated fixed clusters, and shuffled profiles come together and produce the scores. The hybrid-ness of the shuffle eval kind of defeats the shuffling purpose where you can't attribute the robustness against false discover to any component of Buscar. Potentially better ways to shuffle are:
shuffle end to end, generate both clustering, feature selection and scores from shuffled profiles (this would be the most robust but can be compute intensive and not feasible).
shuffle and test every component individually:
a) generate one fixed clustering and one fixed set of on and off features per pair of reference and treatment, compute measure_phenotypic_activity on shuffled profiles but the real clustering and real features and expect scores and compound hit to be random-ish against moa.
b) generate fixed clustering and one fixed set of on and off features per pair of reference and treatment, randomly exchange features between the on and off set, run measure_phenotypic_activity on the real cluster, real profile but exchanged/shuffled on and off feature sets and expect drug hits to be not bringing cells closer to the negative reference
c) fix on and off features, shuffle clustering labels around, run measure_phenotypic_activity on the exchanged cluster labels, real profile and real off feature sets and expect scores to be random-sh
I think all of the 2) option are individually feasible for compute and each would make individual sense. If possible do all three of them but its probably okay to do only two of the three.

I agree that this might be a better approach and might explain the weird waterfall plots we are getting with the shuffle data.

[axiomcura]

This is also related to #69