datasets - deserialize_data mismatched types error

[proohit]

Hi, first of all thanks for your amazing works on bringing ML to Rust!

While compiling a very simple function to train and predict a linear regression model, I encountered an error from the datasets module:

#[wasm_bindgen]
pub fn basic_prediction() -> f64 {
    let x = DenseMatrix::from_2d_array(&[
        &[234.289, 235.6, 159.0, 107.608, 1947., 60.323],
        &[259.426, 232.5, 145.6, 108.632, 1948., 61.122],
        &[258.054, 368.2, 161.6, 109.773, 1949., 60.171],
        &[284.599, 335.1, 165.0, 110.929, 1950., 61.187],
        &[328.975, 209.9, 309.9, 112.075, 1951., 63.221],
        &[346.999, 193.2, 359.4, 113.270, 1952., 63.639],
        &[365.385, 187.0, 354.7, 115.094, 1953., 64.989],
        &[363.112, 357.8, 335.0, 116.219, 1954., 63.761],
        &[397.469, 290.4, 304.8, 117.388, 1955., 66.019],
        &[419.180, 282.2, 285.7, 118.734, 1956., 67.857],
        &[442.769, 293.6, 279.8, 120.445, 1957., 68.169],
        &[444.546, 468.1, 263.7, 121.950, 1958., 66.513],
        &[482.704, 381.3, 255.2, 123.366, 1959., 68.655],
        &[502.601, 393.1, 251.4, 125.368, 1960., 69.564],
        &[518.173, 480.6, 257.2, 127.852, 1961., 69.331],
        &[554.894, 400.7, 282.7, 130.081, 1962., 70.551],
    ]);

    let y: Vec<f64> = vec![
        83.0, 88.5, 88.2, 89.5, 96.2, 98.1, 99.0, 100.0, 101.2, 104.6, 108.4, 110.8, 112.6, 114.2,
        115.7, 116.9,
    ];
    let (x_train, x_test, y_train, y_test) = train_test_split(&x, &y, 0.2, true);
    let y_hat_lr = LinearRegression::fit(&x_train, &y_train, Default::default())
    .and_then(|lr| lr.predict(&x_test)).unwrap();
    let mse = mean_squared_error(&y_test, &y_hat_lr);

    return mse;
}

direnc@direnc-VirtualBox:~/workspace/nodejs-rust$ wasm-pack build --target nodejs
[INFO]: Checking for the Wasm target...
[INFO]: Compiling to Wasm...
   Compiling smartcore v0.2.0
error[E0308]: mismatched types
  --> /home/direnc/.cargo/registry/src/github.com-1ecc6299db9ec823/smartcore-0.2.0/src/dataset/mod.rs:88:49
   |
88 |         let num_features = usize::from_le_bytes(buffer);
   |                                                 ^^^^^^ expected an array with a fixed size of 4 elements, found one with 8 elements

error[E0308]: mismatched types
  --> /home/direnc/.cargo/registry/src/github.com-1ecc6299db9ec823/smartcore-0.2.0/src/dataset/mod.rs:90:48
   |
90 |         let num_samples = usize::from_le_bytes(buffer);
   |                                                ^^^^^^ expected an array with a fixed size of 4 elements, found one with 8 elements

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0308`.
error: could not compile `smartcore`

To learn more, run the command again with --verbose.
Error: Compiling your crate to WebAssembly failed
Caused by: failed to execute `cargo build`: exited with exit code: 101
  full command: "cargo" "build" "--lib" "--release" "--target" "wasm32-unknown-unknown"

The error occurs in mod.rs inside the deserialize_data function. When changing the buffers from 8 to 4 as shown below, the code builds, but throws a RuntimeError somewhere.

...
 let mut buffer = [0u8; 4];
 buffer.copy_from_slice(&bytes[0..4]);
...

const nodejsrust = require('nodejs-rust')
console.log(nodejsrust.basic_prediction())

wasm://wasm/0003e286:1
RuntimeError: unreachable
    at <anonymous>:wasm-function[56]:0xb419
    at <anonymous>:wasm-function[73]:0xbf1e
    at <anonymous>:wasm-function[128]:0xd073
    at <anonymous>:wasm-function[117]:0xce92
    at <anonymous>:wasm-function[124]:0xcfd6
    at <anonymous>:wasm-function[101]:0xca19
    at <anonymous>:wasm-function[20]:0x89ea
    at <anonymous>:wasm-function[9]:0x68ac
    at <anonymous>:wasm-function[15]:0x7cad
    at basic_prediction (<anonymous>:wasm-function[189]:0xd537)

EDIT1: It was supposed to be fixed with #88 , but when applying the changes locally, the RuntimeError still occurs.
EDIT2:
It seems to be an issue with the wasm-bindings... I tried disabling the dataset feature by specifying smartcore = {version = "0.2.0", default-features = false} and the function runs via main, but still panics the RuntimeError..

Additional Info

I'm compiling to WASM with wasm-pack for nodejs.

[dependencies]
wasm-bindgen = "0.2.63"
smartcore = "0.2.0"

Ubuntu 20.04.2 LTS
cargo 1.51.0 (43b129a20 2021-03-16)
wasm-pack 0.9.1

[morenol]

@proohit Thanks for your report. Are these RuntimeError logs generated by the console_error_panic_hook? If that is not happening could you share the logs of the error after you enable that for debugging?

https://github.com/rustwasm/console_error_panic_hook

[proohit]

Thanks for your advice! With included error hook, this error is generated:

panicked at 'could not initialize thread_rng: getrandom: this target is not supported', /home/direnc/.cargo/registry/src/github.com-1ecc6299db9ec823/rand-0.7.3/src/rngs/thread.rs:65:17

Stack:

Error
    at module.exports.__wbg_new_59cb74e423758ede (/home/direnc/workspace/nodejs-rust/smartcore-wasmpack/pkg/nodejs_rust.js:139:15)
    at <anonymous>:wasm-function[40]:0xd16b
    at <anonymous>:wasm-function[275]:0x139af
    at <anonymous>:wasm-function[86]:0x109d0
    at <anonymous>:wasm-function[107]:0x1179e
    at <anonymous>:wasm-function[192]:0x13266
    at <anonymous>:wasm-function[173]:0x12eed
    at <anonymous>:wasm-function[186]:0x13167
    at <anonymous>:wasm-function[148]:0x127ec
    at <anonymous>:wasm-function[41]:0xd4a5

Looks like some incompatibility with rand? I build the binaries with wasm-pack build --target nodejs, so maybe there is some incompatibility.

That's the called function from thread.rs of rand

thread_local!(
    static THREAD_RNG_KEY: UnsafeCell<ReseedingRng<Core, OsRng>> = {
        let r = Core::from_rng(OsRng).unwrap_or_else(|err|
                panic!("could not initialize thread_rng: {}", err));
        let rng = ReseedingRng::new(r,
                                    THREAD_RNG_RESEED_THRESHOLD,
                                    OsRng);
        UnsafeCell::new(rng)
    }
);

[morenol]

I will open a PR that makes the CI runs the tests with wasm-pack. That will make us be clear in which part of this library we need to make changes to add support for wasm.

[proohit]

Interestingly, changing the wasm toolchain to SSVM with ssvmup build which needs 1.50.0 > rust > 1.30.0 seems to work and provides an output. I checked the compilation logs and it compiles rand v.0.7.3 successfully

[morenol]

@proohit could you try with the changes from this PR?

#93

The tests are now running and passing using wasm-pack.

I pushed that commit to this repository too

In particular, for the error with rand adding this to Cargo.toml solves the problem:

[target.'cfg(target_arch = "wasm32")'.dependencies]
rand = { version = "0.7.3", features = ["wasm-bindgen"] }

[proohit]

Thank you for your efforts! Can I just clone this repo and build smartcore myself and then use it locally or is there an online crate distributed for development which I can tell cargo to fetch?

[morenol]

You can install from git repositoyies i.e

smartcore = { git = "https://github.com/smartcorelib/smartcore", rev = "1cf8c7941feeb514344678ddaf1073743ce49607" }

I suggest you to create a fork of smartcore and use

smartcore = { git = "https://github.com/proohit/smartcore", rev = "1cf8c7941feeb514344678ddaf1073743ce49607" }

I think that I will remove that branch from this repository once this gets merged (hopefully soon). Once that is merged you could use.

smartcore = { git = "https://github.com/proohit/smartcore", branch = "development" }

The only problem with that is that you will not be able to publish in crates.io if you have a git dependency.

[proohit]

Wow that's a thing? Cargo is just amazing.

Anyway, I tried smartcore = { git = "https://github.com/proohit/smartcore", branch="lmm/wasm_tests" } and the error is gone. It compiles correctly and js bindings work too. Thank's a lot!

[morenol]

@proohit that change was merged in the development branch! you could now change your requirements again to this repo or wait for the next release of smartcore, to get that directly from crates.io.