UMAP Roadmap

[lmcinnes]

A rough roadmap of things to be done for UMAP. Some of these tasks are easy, some are hard, and some require deeper knowledge of UMAP. Short and medium term tasks should be approachable for many people. Reply to this issue if you are interested in taking up any of them.

Short term items

• Support for sparse matrix input
• Add random seed as an user option
• Support for cosine distance RP-trees
• Allow non-RP-tree initialisation of NN-descent
• Better document (via docstrings) all the support functions
• "Custom" initialisation with a predefined positioning.

Medium term items

• Generate notebook for basic usage demonstration
• Generate notebook explaining parameter options and their effects
• Set up CI and build a basic test suite
• Start building basic documentation and integrate with readthedocs

Longer term items

• Generate notebook for "How UMAP works"
• Add code (and devise API(?)) for UMAP on general pandas dataframes
• Add support for semi-supervised dimension reduction via UMAP
• UMAP as a generative model (code + demo)
• UMAP for text data (similar to word2vec)
• A transform function for new previously unseen data (see issue UMAP as a dimensionality reduction (umap.transform()) #40)
• Model persistence for UMAP models

No priority

• GPU support for UMAP
• Conda-forge UMAP package
• Improve numba usage (better numba expertise required)
• Concurrency via Dask for multicore and distributed support

[Fil]

Here's the smallest notebook I could think of for basic usage demonstration
https://nbviewer.jupyter.org/gist/Fil/cce232583907035b65686cdec7d4cc92

[lmcinnes]

Thanks! I was hoping to have some further description in Markdown in the notebook, but this is an excellent beginning.

[KeithTheEE]

Would you mind if we pulled direct quotes from your README for the notebook ('basic usage demonstration' and 'explaining parameter options and their effects')?

I'm also currently wrapping the two ideas as one notebook, with a basic usage section at the top, and more in-depth information after that. Thoughts?

[lmcinnes]

Go ahead and pull whatever you need. It's helpful if you can explore the parameter effects in a little detail. Kyle McDonald had some nice min_dist comparisons here https://twitter.com/kcimc/status/930180473262919685 . Exploring some of the other effects similarly as well (metric, n_components, spread, n_neighbors) would be beneficial. But certainly any contributions are welcome.

[Fil]

Here's another version, exploring some of parameters
https://nbviewer.jupyter.org/gist/Fil/5c48475e88a0e1a8f56eaadaebff0544

[lmcinnes]

I love the metric exploration! The custom metrics nicely show off what can be done, and the effects (e.g. the pure red metric has a clear linear embedding etc.)

[lmcinnes]

@Fil I have added in a version of your parameter exploration notebook (with some minor changes and added text commentary and explanation) in the notebooks directory. Have a look and let me know if it looks okay to you. I really appreciate your work on this, so let me know how you would like to be acknowledged within the notebook.

[Fil]

Wow! I wanted to do the hue and HSL metrics, but didn't think they would turn out that splendid. Thank you! For the credit you should remove "excellent", and can add "for visionscarto.net" after my name for affiliation.

I'm preparing another example, will follow up when it's ready :)