Thoughts on moving the linalg and math abstractions into a standalone crate?

[sfriedowitz]

Hello Smartcore team,

I'm pretty new to the rust language, and have been working on a personal machine learning project to try to learn the language a bit better. In the process of looking for good code examples, I came upon this library, and am impressed with the organization and level of abstraction present in some modules. Particularly, I keep thinking about how useful it would be to develop my personal project against the interface provided by the n-dimensional array/vector/real number abstractions present in the linalg and math modules of this crate.

What are your thoughts on making those modules, linalg and math, part of a standalone crate that Smartcore then depends on? I feel like as those abstractions continue to be refined (e.g. #108), they could become an invaluable part of the ML/AI ecosystem in rust.

If this isn't something you're all interested in, no worries! Just a thought and figured I'd put it out there.

Cheers, from an aspiring rust developer!
-Sean

[morenol]

I think that we should move in this direction

[Mec-iS]

I don't know. I have picked up #108 and it looks good enough to give it a shot and see, I will push new commits in the PR as soon as I solved all the conflicts with current branch. Before thinking about this we need to have the PR finalised.
There are already stand-alone libraries that do linear algebra in Rust, what would be the extra provided but our package?

[morenol]

I think that we can do this after #108, my idea is not that the lineal algebra crate is needed but that we could separate smartcore in several crates so people can install only what they need. For instance, smartcore-naive-bayes, smartcore-dataset, etc. We always could re-import all in a main crate smartcore. This could improve compilation times for people that don't need everything

[Mec-iS]

I understand, that is how Linfa is structured. I think this is not the time as the library is quite under-development, maybe when we reach version ~0.5? I mean this sounds adding some overhead in terms of organisation and operations and we have to focus on implementation I think. I would prefer to wait for some fixes and the big refactoring to be done before even thinking about this; for now I would just add a label like "postponed" and pick it up later.

[montanalow]

Part of the reason I chose Smartcore over Linfa was because having everything contained in a single package made browsing the documentation and code a lot easier to get started with. There are parts of the Linfa documentation that are now out of sync across packages, and all the packages are interdependent anyway between the core, datasets, and whatever algorithm you want you need to be on matching versions, and having to specify each feature that I need in each package (inter-package co-dependencies aren't well documented) was a lot of asking the compiler what am I missing now...

[cmccomb]

Just want to second @montanalow on consistency across crates. I actually chose smartcore for exactly the same reason! Maybe feature-gating would be worth considering? I could imagine one for classification, one for regression, etc.

[morenol]

Thanks for the feedback.

I will close this issue then since that seems to be one of the selling points of smartcore.
I agree that we could then use more feature-gating