Machine learning plots overwrite each other under some condition #98
Description
As first reported in #80 (review), sometimes when running the Snakemake workflow with multiple cores the ML plots overwrite each other. I observed behaviors such as:
- the PCA plot for data 1 actually being the PCA plot for data 0
- a hierarchical clustering plot only being partially drawn
- plots being the wrong size
I suspect there could be some sort of race condition happening due to the parallelism and the randomness of the types of errors I see. Perhaps somehow the pyplot current figure is shared across multiple Snakemake rules executing in parallel?
What makes this confusing and hard to debug is that I can consistently reproduce this behavior and Neha cannot trigger it. She is on macOS, and I'm on Windows, so the platform could have to do with it. Our plan is to merge #80 and then ask more of the SPRAS team to help us try to reproduce the problem.