Skip to content

[Potential] Add sparse / backed matrix support for omics-scale inputs #492

Description

@breimanntools

Status: Potential (speculative). Candidate for v1.5, not yet committed. Filed so the
design is captured; needs a go/no-go before implementation.

Problem

Omics-scale feature/label matrices are often sparse or too large to hold densely in
memory, but AAanalysis components assume dense in-memory arrays. Passing a SciPy sparse
matrix or a backed AnnData today either densifies silently (blowing up memory) or fails
in an opaque place, with no declared contract for what each component accepts.

Goal

Add declared support for sparse and backed matrix inputs where it is scientifically
sound, with explicit failure (not silent densification) where a component cannot support
them, keeping feature names and sample order stable.

Requirements

  • Components declare whether they accept sparse / backed input; a .densify-style
    boundary is explicit and logged (ut.print_out), never silent.
  • Unsupported sparse/backed operations raise a bare ValueError naming the component
    and the required dense form — not a deep numpy error.
  • Accidental densification is detected/flagged where feasible (a guard in the hot path).
  • Feature names and sample order remain stable through sparse handling.
  • numpydoc docstring updates on affected signatures; example notebook; unit tests +
    a sparse omics-scale benchmark fixture.

KPIs / Acceptance criteria

  • A supported component consumes a SciPy sparse input and produces the same result as
    its dense equivalent (asserted within tolerance).
  • An unsupported component raises a clear bare ValueError rather than densifying
    silently (asserted).
  • Feature-name / sample-order stability asserted across dense vs sparse paths.
  • Benchmark fixture demonstrates bounded memory on a sparse omics-scale input.

Scope / non-goals

  • Core / pro depending on the component; no new required dependency (SciPy already
    present).
  • AAanalysis operates on the matrices it is given. Large-data storage, object stores and
    cluster memory management stay in ProtXplain / infrastructure.

Dependencies

Standards checklist

  • Frontend/backend split honored; validation block; backend trusts frontend
  • CONFIRM-FIRST? — signature changes on public methods flagged
  • numpydoc docstring (named Returns, per-method Examples include)
  • tests (unit; sparse==dense; unsupported raises; name/order stable); reproducibility
  • no print() (use ut.print_out); bare ValueError/RuntimeError; no aaanalysis._utils.* outside utils.py

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions