[python] Fast CSR loading for to_anndata
#83
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 8 commits
January 18, 2023 17:44
added 2 commits
January 18, 2023 20:11
|
@thetorpedodog - OK, I think I hit all of your (excellent) feedback. |
thetorpedodog
approved these changes
Jan 18, 2023
added 2 commits
January 18, 2023 20:55
|
Still looks good. |
johnkerl
requested changes
Jan 18, 2023
johnkerl
approved these changes
Jan 19, 2023
| indices = np.empty((nnz,), dtype=index_dtype) | ||
| data = np.empty((nnz,), dtype=self.coo_chunks[0][2].dtype) | ||
|
|
||
| # empirically determined value. Needs to be large enough for reasonable |
There was a problem hiding this comment.
Super-nit: capitalize Empirically at start of sentence
There was a problem hiding this comment.
super nit, but super easy - fixed!
|
Related issue with scipy.sparse performance: scipy/scipy#11496 |
johnkerl
changed the title
to_anndata
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses the peformance issues noted in single-cell-data/TileDB-SOMA#719. In particular, loading large AnnData requires CSR sparse matrices, which are very slow to convert from COO to CSR. This PR adds a fast-path converter for Python, implementing two changes:
In addition, this change uses substantially less memory for the conversion, allowing larger datasets to be loaded into CSR, and therefore into AnnData.
Before/after benchmarks show 1.5-4X speed-ups when the work fits in RAM (tested on r6i instance types with data on S3). In cases where paging occured, speed-ups were dramatically larger (e.g., 20X).
See also, related PR single-cell-data/TileDB-SOMA#745