MAINT: Vendoring policy

[larsoner]

I think we should perhaps discuss/revisit our policy of vendoring, at least for some dependencies we might want to try using install_requires nowadays. Even matplotlib nowadays has some dependencies:

https://github.com/matplotlib/matplotlib/blob/806f00aa8faa03c19adb66560272e956bb7ddc09/setup.py#L315-L324

I know the numpy/scipy/sklearn stack does not require anything (other than setuptools maybe?) but I see us as closer to matplotlib than these core (very C-based) scientific libraries. Also the packaging ecosystem (pip wheels, conda/mamba) has matured quite a bit I think since we made the conscious decision to vendor (5+ years ago IIRC?).

I bring this up because it would be nice to vendor fewer libraries (ideally zero?), and in #10197 we're discussing possibly vendoring another library (packaging).

We can also discuss this at our next dev meeting if it's easier.

[larsoner]

From @cbrnr in #10197:

I'd also try to vendor as few packages as possible, especially when we're talking about core packages provided by pypa.

[cbrnr]

To add to my comment, could you summarize what the reasons for vendoring those packages were? It might well be the case that a lot has changed, and I would try to vendor zero packages nowadays. We have a lot of tests, so we should catch any breaking changes before any new release. BTW, matplotlib depends on packaging but not setuptools.

[larsoner]

could you summarize what the reasons for vendoring those packages were?

@agramfort feel free to elaborate or correct me here, but I think it had to do with making so that you only need numpy/scipy/matplotlib installed, then pip install mne "just works" -- no need to pull anything else in or risk pip resolvers failing (for example by trying to build things -- maybe mayavi? -- from source and failing).

I think for a long time we also didn't even list numpy or scipy in our install_requires -- looking back I guess @cbrnr you added it in #6083. So my sense is that removing some of our vendored stuff would continue the incremental progress we've made over the years. And so far I don't think we've suffered any negative consequences from stuff like #6083, so that adds some evidence that maybe it's safe to move ahead with things like removing vendored libs in favor of proper standard dependencies.

[cbrnr]

It is also worth noting that packages available to pip (on PyPI) are almost always available as wheels nowadays. In my opinion, I don't even see any reason to use conda over pip, at least not for the standard user - even on Windows installing Python from python.org and then pip installing packages just works.

[agramfort]

what packages would you recommend to add to install_requires? tqdm? scholar_parser? h5io? pymatreader? the tempita library we vendor is not really maintained anymore.

…

Message ID: ***@***.***>

[larsoner]

what packages would you recommend to add to install_requires?

Okay let's look through the seven things we keep in mne/externals, plus the one (packaging) that we need for #10197 ...

the tempita library we vendor is not really maintained anymore.

... And good point -- maybe we only vendor things that are no longer maintained. So my proposal is:

Remove

• FieldTrip.py -- it's for realtime buffer support, we shouldn't even have this anymore, we should put it in mne-realtime

Add to install_requires:

• decorator: pure python, maintained
• h5io: pure python, only strict dep is numpy IIRC, maintained
• packaging: will be maintained by pypa folks, needed for Use packaging.version for comparing versions #10197
• pymatreader: pure python, only depends on scipy IIRC, maintained
• tqdm: already an optional dependency with [data], but we use it elsewhere (like for progress bars) so I think we might just want a hard dep here

Move

• tempita: move to mne/reports for now where it's needed, and maybe @hoechenberger can see how much work it would be to move to jinja2 or some other templating system rather than sticking with something that is not maintained
• doccer.py: originally adapted from SciPy, it's only ~100 lines so we should just move it to utils/docs.py or wherever our doc-filling code lives

[cbrnr]

This sounds great!

[hoechenberger]

• tempita: move to mne/reports for now where it's needed, and maybe @hoechenberger can see how much work it would be to move to jinja2 or some other templating system rather than sticking with something that is not maintained

In fact this is something I felt was worth doing when refactoring the reports. So yes, can do.

[jasmainak]

could pooch also be added and we do away with "data dependencies" while we're at it? I've stumbled upon this multiple times in packages that use MNE as dependency. The distinction seems a bit artificial to me as I've never seen anyone use MNE without downloading some sample data and trying an example first ...

[hoechenberger]

+1

But please please ensure that all dependencies can easily get pulled in automatically when installing MNE on a simple Python installation in Colab, as this is what many people use for teaching.

[cbrnr]

import pooch works on Colab. Not sure how this works, but there is probably a way to (pip) install missing packages too.

Edit: yes, just !pip install mne for example.