(feat): sc.get.aggregate via dask

[ilan-gold]

See #3723 for a follow-up where we would go algorithm-by-algorithm and implement column-wise operations as well

• Closes sc.get.aggregate with dask #3659
• Tests included or not required because:

• Release notes not necessary because:

[codecov]

Codecov Report

❌ Patch coverage is 85.71429% with 8 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@77dd43c). Learn more about missing BASE report.
⚠️ Report is 42 commits behind head on main.

Files with missing lines	Patch %	Lines
src/scanpy/get/_aggregated.py	84.90%	8 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3700   +/-   ##
=======================================
  Coverage        ?   76.00%           
=======================================
  Files           ?      117           
  Lines           ?    12600           
  Branches        ?        0           
=======================================
  Hits            ?     9576           
  Misses          ?     3024           
  Partials        ?        0           

Files with missing lines	Coverage Δ
src/scanpy/preprocessing/_qc.py	95.80% <100.00%> (ø)
src/scanpy/get/_aggregated.py	90.04% <84.90%> (ø)

[ilan-gold]

@Intron7 It won't let me request your review, but you're welcome to have a look

[ilan-gold]

TODOs:

• CSC aggregation, i.e., aggregate by groups of features and then concatenate
• Similar to the above, try out adding (or reusing) some sort of chunks parameter that allows doing the aggregation on a subset of the requested groupby-categories and then concatenate at the end to potentially memory.

[flying-sheep]

I added @Intron7 to https://github.com/orgs/scverse/teams/scanpy, now you can request his review (I also did so)

[flying-sheep]

Awesome! I’m not 100% sure how it works though.

Could you explain how it works in respect to using the categories in the chunking? Are you making heterogeneous chunks to collect each subset according to a category?

[flying-sheep]

Can this info be centralized? this looks fragile because it’s an inline check based on the currently available functions.

[ilan-gold]

This really should be a TODO. To be correct, we would need to handle the overflow potential of sum i.e., jumping from uint16 to uint32 or something because the summation gets bigger. I wonder how dask or fau handles this internally? For example https://github.com/scverse/fast-array-utils/blob/b3c7c9829b2fe4e6d23915a2c6bab1b025484b10/src/fast_array_utils/stats/_sum.py#L89-L98 actually seems wrong for this exact reason, no?

[flying-sheep]

First, I still stand by my comment. Please don’t bi-partition an extensible set by specifying one partition. This should be a lookup, either central or by defining and accessing an inline dict/switch-case that spells out all options.

regarding the rest: hmm, I guess I conceived fau to be more low-level, i.e. if you fear that something can overflow, set dtype manually, otherwise it preserves wherever possible, e.g. it doesn’t make sense to preserve int dtypes in mean, but fau doesn’t know what how big the numbers you sum up are so it makes no assumptions.

does that make sense or do you think it should be more opinionated?

[ilan-gold]

Fair point, the two things are separate. I will add a TODO as well then. Have a look and let me know if this is what you had in mind or if I misunderstood

[ilan-gold]

Awesome! I’m not 100% sure how it works though.

The idea is that we go row_chunk-by-row_chunk and aggregate those individually before doing a sum-like operation over all of them to get the final full-dataset aggregation. For example, for sum, the aggregation of rows 0-100 into 10 categories and 15 features produces a 10x15 matrix of that subset's summation aggregation, but the same is true of any subset. If we do these repeatedly over subsets, we can then do a sum operation over these 10x15 matrices to get the final aggregation result.

So each map_blocks operation outputs a 1x10x15 matrix giving an intermediary n_chunks x 10 x 15 matrix that is then summed over the first axis (i.e., the .sum(axis=0) call after map_blocks) to give the final result, a 10x15 matrix.

This architecture allows us to reuse our already existing implementation as a subroutine and paves the way hopefully for #3723 applied here as well where we just have an additional concat operation over the feature space.

Does this clarify things?

[flying-sheep]

First, I still stand by my comment. Please don’t bi-partition an extensible set by specifying one partition. This should be a lookup, either central or by defining and accessing an inline dict/switch-case that spells out all options.

regarding the rest: hmm, I guess I conceived fau to be more low-level, i.e. if you fear that something can overflow, set dtype manually, otherwise it preserves wherever possible, e.g. it doesn’t make sense to preserve int dtypes in mean, but fau doesn’t know what how big the numbers you sum up are so it makes no assumptions.

does that make sense or do you think it should be more opinionated?

[flying-sheep]

beautiful!