Migrating from MAGMA: question about matrix storage

[mawxcarroll]

Hello,

I've been using MAGMA with great success for a few years for some quantum dynamics simulations. I started work on a new class of problems and realized that my matrices were now quite sparse. It was suggested to me that Ginkgo might be a good solution.

One of the most useful features of MAGMA (for me) is that one can store a matrix in system memory (512 GB in my case) while diagonalizing it on the GPUs (4 x A100 with total 160 GB). This is, of course, slower that pure GPU computation, but in many cases I wish to maximize the size of the problem that I can solve.

I wonder if this is possible using Ginkgo as well?

Thanks for the help!

Cheers,
tom

[MarcelKoch]

Hi Tom,

could you elaborate on how you want to store your matrix? Do you want to store a matrix in the CPU memory of a single CPU and then distribute it to multiple GPUs attached to this one CPU? This might not be straight forward in Ginkgo, but we are working on making this possible in the not-too-distant future.
However, if you have your matrix already distributed among CPU ranks, you can immediately transfer the matrix to as many GPU ranks.

Best,
Marcel.

[pauleonix]

Shouldn't it be possible to oversubscribe device memory with the managed/unified memory allocator? Not sure if performance will be anywhere near what is needed, but it might be worth a try. As an optimization one could manually split the matrix into blocks that fit and prefetch them to device memory. Depending on the structure, off-diagonal blocks might be small enough to process the on the host in parallel.

[mawxcarroll]

Hi Marcel and Paul,

Thanks for your replies.

I've heard/read enough to know that it's worth moving forward, so I am in the process of installing Ginkgo so I can actually try these things!

[mawxcarroll]

Hi Marcel,

I have four GPUs in a single node, but I am also interested in having my code run in an environment where the GPUs are distributed across multiple nodes. To my understanding, Ginkgo handles both cases the same way, with MPI. Do I have that right?

Thanks again!

Cheers,
tom

[MarcelKoch]

Yes, that's correct. You can find more details on how to use MPI with Ginkgo in this example: https://ginkgo-project.github.io/ginkgo-generated-documentation/doc/develop/distributed_solver.html, or you can also take a look at the necessary classes directly:
https://ginkgo-project.github.io/ginkgo-generated-documentation/doc/develop/classgko_1_1experimental_1_1distributed_1_1Vector.html
https://ginkgo-project.github.io/ginkgo-generated-documentation/doc/develop/classgko_1_1experimental_1_1distributed_1_1Matrix.html

[mawxcarroll]

Thanks, Marcel!

I got the distributed solver up and running and using all four of my GPUs, so this is great.

Also, I'm loving the whole Ginkgo package so far, so thanks to you and the rest of the team.

Cheers,
tom