Multiple uses of svd or svd_inplace in the same program cause it to stall

[rennis250]

I've found that calling SVD functions more than once in the same program cause it to stall and start eating up lots of memory. It seems to be independent of whether you use svd or svd_inplace. It is also independent of whether you use the same array repeatedly or not. An example program:

extern crate arrayfire as af;

use af::*;

fn main() {
    set_backend(Backend::CPU);

    for x in 40000..50000 {
        let w = 400u64;
        let h = x as u64;

        let dims = Dim4::new(&[w, h, 1, 1]);
        let mut d: Array = randu::<f32>(dims);
        let mut e: Array = randu::<f32>(dims);
        if x == 40000 {
            println!("{}", d.dims());
        } else {
            println!("{}", e.dims());
        }

        println!("before svd");
        if x == 40000 {
            let (u, s, vt) = svd(&mut d);
        } else {
            let (u, s, vt) = svd(&mut e);
        }
        println!("after svd");
    }
}

On my Mac with 8GB of RAM, the second iteration of the loop hangs and starts eating up RAM. I'm using ArrayFire from the installer, Rust 1.16.0, and the latest version of this crate from crates.io. I will be able to test on a Linux machine later.

[rennis250]

On the Linux machine, which has 256GB of RAM, the loop hangs after 5 iterations. I'm also finding that SVD on the Linux machine cannot handle matrices of about 800x200x400 elements. The limit is much less on my laptop with 8GB of RAM. A matrix of this size would be considered a small data file for us. I assume that the memory limit is coming from the underlying clBLAS library, but the problems with iteration could potentially be a Rust problem?

[rennis250]

It seems part of the size limitation is coming from Rust(??), not from the underlying libraries. If I translate a similar version of this to C++, then I can make the matrices larger and SVD works correctly, but it runs rather slowly and there is still an upper limit that prevents us from using ArrayFire with any of our data.

See here for the C++ version: arrayfire/arrayfire#1746

[rennis250]

And the problem of multiple SVDs in one block of code is coming from the underlying ArrayFire libraries: arrayfire/arrayfire#1746

[9prady9]

@rennis250 I did run the code you shared on both CPU and CUDA backend. CPU backend is eating up almost all of the RAM although it runs. However, CUDA is running out of memory.

Note: I have 8GB RAM machine with GT650M card with 2GM GPU memory.

I think the core issue stems from either upstream ArrayFire or the libraries ArrayFire uses for SVD implementation. I am fairly confident that is the case because svd implementation in rust wrapper looks like below - it is pure call delegation to ArrayFire's C API, nothing else.

pub fn svd(input: &Array) -> (Array, Array, Array) {
    unsafe {
        let mut u: i64 = 0;
        let mut s: i64 = 0;
        let mut vt: i64 = 0;
        let err_val = af_svd(&mut u as MutAfArray, &mut s as MutAfArray, &mut vt as MutAfArray,
                             input.get() as AfArray);
        HANDLE_ERROR(AfError::from(err_val));
        (Array::from(u), Array::from(s), Array::from(vt))
    }
}

[9prady9]

Closing this issue as this very likely upstream or upstream's dependency related. Upstream already has an active issue whose progress can be tracked here arrayfire/arrayfire#1746