Document the lack of an API

[pradyunsg]

pip does not provide any API. Despite this, there are many people who do import pip and try to use pip, either by using pip.main or even invoking pip in different ways.

It might be beneficial to document the fact that pip does not provide an API in some part of the documentation.

The better solution is to list alternatives to using pip for the specific use cases that are out there.

For example, distlib.

[pfmoore]

I'll be writing a PR for this in the next few days, hopefully. You can see what I have in mind then. But basically, I expect to explain:

1. That users should not import pip in their own code, or use APIs from the pip module.
2. Why we take that position - I'll cover some of the real issues that exist in the API (it's not thread-safe or even safe to call twice in the same process, it assumes it is the sole user of the logging subsystem, etc) as well as pointing out that it's not that we wouldn't like to provide a supported API, but the amount of work required, and the constraints on pip that would come from having an API that remains stable, make it essentially impossible at the current time.
3. What people who feel they would like to import pip should actually do. They can use pip in a subprocess. Or they could investigate other libraries - I'll mention options like packaging, pkg_resources and distlib, but it'll remain the user's responsibility to decide how suitable they are.

If I can work out how to explain it nicely, I'll also clarify what "not supported" means. But I'm not yet sure what to say on that given the reality is "you got the program for free, so why do you feel you are entitled to anything?" 😄

[dstufft]

What I normally say is something along the lines of:

Not supported means that everything inside of pip is considered an implementation detail, including the fact that the import name is pip (other than pip.__main__) and while we generally try to not break things if we reasonably can, these apis can change at any time, for any reason. It also means that we generally won't make changes to accommodate issues that only stem from using pip in an unsupported way.

[dstufft]

One thing we could do, that I've avoided for a while because it would break a lot of things is to restructure our code to make this more obvious, and do something like:

.
├── _pip
│   ├── ...
│   └── __init__.py
└── pip.py

Essentially just moving all of pip/ into _pip/ and then moving _pip/__main__.py to pip.py. That would keep python -m pip working although it might break some people who were doing python -m pip.__main__ to work around Python 2.6 not supporting python -m on packages. We could also just keep pip/__main__.py and move everything else to _pip/.

Another, similar option is to do:

.
└── pip
    ├── __init__.py
    ├── __main__.py
    └── _internal
        ├── ...
        └── __init__.py

Which is really the same basic idea, it just shifts everything down into pip/_internal instead. It would result in longer paths but keeps everything contained inside the pip/ directory still.

This would be a big flag day that would break all uses of the internal API (although pretty easy to fix) and would make it pretty obvious that somebody is doing something that isn't supported. It would also probably invalidate every open PR and require a rebase. This might be a better idea than documentation though, since it would become impossible-ish to use pip's internal APIs without importing from a underscore name (possibly one that literally says the word "internal").

Of the two, I think I prefer the pip._internal spelling, simply because it keeps things tidy in the pip/ directory still AND puts it right in the name that these are internal APIs.

[pradyunsg]

A big +1 on pip._internal from me.

I think that would be well complemented by addition of a section within the documentation as well.

I'll be happy to do what's needed to get this happening, if that's what we decide to do.

[pfmoore]

I like this idea, in general, although I too thought it was likely too big a change to be worthwhile. Specifically, I'm in favour of the pip._internal colour for the bikeshed.

I still think having some documentation would be useful, and I'm happy to write that. Specific things I would like to cover:

1. What "unsupported" means - I like the way @dstufft put it above.
2. The recommended approach of running pip in a subprocess.
3. The fact that it's not simply pip that doesn't support this - modifying site-packages from within a running Python program is something you should only ever do if you really know what you're doing, whether or not pip is involved (noting that this means that even running pip in a subprocess has possible issues).
4. Some pointers as to libraries you can use to implement stuff that pip does, if you want to. And pointers to relevant PEPs for people who want to write their own code.

[dstufft]

I went ahead and opened up #4700. I suspect there will be some test failures that I didn't catch, but it seems to work fine.

[pradyunsg]

#4700

🎉 Nice round number for this change. ^.^