Long Integer for Non-Zero Elements of Sparse Matrices

[ArmoredCat]

I use OjAlgo a lot and greatly appreciate its robustness and speed, particularly with matrix operations.

I'm writing to ask about the current implementation of sparse matrices and the limits on the number of non-zero elements.

Currently, it appears the maximum number of non-zero elements is constrained by an int. In my large-scale numerical simulations I often encounter matrices with billions of cells, even a small density can result in a non-zero count that exceeds 2,147,483,647.

My question is: Are there any plans to expand the capacity of the sparse matrix implementation to handle a maximum number of non-zero elements defined by a long? Maybe an alternate implementation where the user can chose between the two matrix types for people who don't need such large matrices?

Any insight into the feasibility, roadmap, or technical challenges of making this change would be highly valued. Thank you for your work on this excellent library!

[apete]

In principal the org.ojalgo.matrix package supports int indices as Integer.MAX_VALUE x Integer.MAX_VALUE matrices, but in reality not all implementations can store that many elements.

For SparseStore the limitation is the underlying SparseArray which can only store Integer.MAX_VALUE non-zero elements. (I believe this was a relatively new restriction, introduced to make things more efficient as it was assumed to be "enough".) SparseStore stores all the non-zeros in a single SparseArray.

Without changing anything in ojAlgo you have some alternatives:

Use RowsSupplier or ColumnsSupplier instead. They are also backed by SparseArray but has one instance per row/column.

Use Array2D instead – it supports long indices. It doesn't do linear algebra, but is quite flexible in terms of how things are actually stored.

Implementing a custom MatrixStore to match some specific requirements is actually easy.

https://www.ojalgo.org/2020/09/sparse-and-special-structure-matrices/