Remove the pip search command

[dstufft]

As of today (Oct 7, 2022), pip search does not return results with PyPI (pypi.org). Please use pypi.org/search instead.

Relevant context on this that GitHub hides by default:

#5216 (comment)

PyPI is shutting down the API that pip search used, because the design of that API was such that it made it easy for people to overwhelm and take down PyPI as a whole using that API. The pip search command doesn't work without that API, and we don't have a way to restore 1:1 functionality without it.

#5216 (comment)

Here's the inevitable truth. The XMLRPC based search API either needs major surgery to somehow move the load off of the PyPI backends so we can just ignore if it breaks, massive clusters need to stop hammering it, or we need to deprecate it. [snip graph + few sentences] The fact that we've managed to keep it working to this point is frankly something that should provide us an immense amount of pride in engineering and shame in foresight.

The status.python.org incident that (eventually) led to the XMLPRC search (which powers pip search) being disabled on pypi.org

---

(original description below)

Currently pip allows you to search a repository by running pip search, which will then print out a bunch of packages that match, see for example:

$ pip search requests
negotiator-3k (1.0.0)                                                  - Proper Content Negotiation for Python
    
    The Negotiator is a library for decision making over Content Negotiation requests.
    It takes the standard HTTP Accept headers (Accept, Accept-Language, Accept-Charset,
    Accept-Encoding) and rationalises them against the parameters acceptable by the
    server; it then makes a recommendation as to the appropriate response format.
    
    This version of the Negotiator also supports the SWORDv2 extensions to HTTP Accept
    in the form of Accept-Packaging.
odoo10-addon-sql-request-abstract (10.0.1.0.0.99.dev1)                 - Abstract Model to manage SQL Requests
odoo9-addon-sql-request-abstract (9.0.1.0.0.99.dev6)                   - Abstract Model to manage SQL Requests
odoo8-addon-sql-request-abstract (8.0.1.0.0.99.dev7)                   - Abstract Model to manage SQL Requests
zenodo-accessrequests (1.0.0a2)                                        - Zenodo module for providing access request feature.
requests-wsgi-adapter (0.4.0)                                          - WSGI Transport Adapter for Requests
requests-celery-adapters (2.0.9)                                       - Requests lib adapters to send Celery messages (tasks)
odoo9-addon-hr-holiday-notify-employee-manager (9.0.1.0.0.99.dev1)     - Notify employee's manager by mail on Leave Requests creation.
odoo9-addon-purchase-request-operating-unit (9.0.1.0.0)                - Operating Unit in Purchase Requests
odoo10-addon-hr-holidays-notify-employee-manager (10.0.1.0.0.99.dev4)  - Notify employee's manager by mail on Leave Requests creation.
odoo9-addon-sql-export (9.0.1.0.0.99.dev12)                            - Export data in csv file with SQL requests
odoo8-addon-sql-export (8.0.1.0.0.99.dev9)                             - Export data in csv file with SQL requests

...

The output here goes on for ~900 lines and the results are just complete trash. This is a better on Warehouse:

$ pip search --index https://pypi.org/pypi requests
requests (2.18.4)                         - Python HTTP for Humans.
  INSTALLED: 2.18.4 (latest)
aiohttp-requests (0.1.0)                  - A thin wrapper for aiohttp client with Requests simplicity
anonymous-requests (0.2)                  - 
apiclient-requests (0.1.2)                - A simple python base package for building good api clients on
careful-requests (0.1.4)                  - Requests for header-sensitive servers (like Accept-Encoding)
crawl-requests (2.2.8)                    - crawl_requests(like requests) can update ua and proxy automatically.
gcloud-requests (1.1.9)                   - Thread-safe client functionality for gcloud-python via requests.
jsonapi-requests (0.6.0)                  - Python client implementation for json api. http://jsonapi.org/
jsonrpc-requests (0.4.0)                  - A JSON-RPC client library, backed by requests
nav-requests (1.1.4)                      - Renamed to `nav`
parse-requests (1.0.7)                    - parse-rest-python - A fast and simple Python library to interact with Parse.com REST API
play-requests (0.0.3)                     - pytest-play plugin driving the famous Python requests library for making HTTP calls
PyGithub-requests (1.26.0)                - Use the full Github API v3
Randomized-Requests (1.0.2)               - Python package that makes post and get request with random proxy and user agent
requests-aeaweb (0.0.1)                   - Requests wrapper to log onto AEAweb.org.
requests-aliyun (0.3.1)                   - authentication for aliyun service
requests-auth (1.0.2)                     - Easy Authentication for Requests
requests-aws (0.1.8)                      - AWS authentication for Amazon S3 for the python requests module
requests-aws4auth (0.9)                   - AWS4 authentication for Requests
requests-bce (0.0.5)                      - authentication for bce service

...

Which gives us ~111 lines of output, and which actually returns some meaningful output.

I believe that this command is a fairly regular source of confusion for users, primarily because it uses a different source of truth than pip install does, which means they need to configure the location to search at differently than they need to configure the location to install from (and the search API is not standardized, and to the best of my knowledge, very few alternative implementations support it.

There has been a long standing idea of switching search to use the PackageFinder() class to try and resolve these issues, but I don't think that is going to work reasonably either. The problem is that while that would reconcile the differences between, PEP 503 doesn't provide any mechanism to pass information like the summary that we print alongside each result above. Speaking with my PyPI hat on, I would be very opposed to adding such information to the PEP 503 repository API, because it would bloat the responses and have them take up more bandwidth for a very minority edge case. The other problem is that the PackageFinder() API itself doesn't fall back to /simple/ anymore, but that's resolvable but the larger issue with that is that /simple/ is 7MB large as of right now, and that is likely to continue to grow, having pip search issue a 7MB http request is a pretty crummy experience.

So that leaves us in a bit of a sticky situation. The current implementation is confusing and practically speaking only searches PyPI and not anywhere else, but our best path forward for resolving that is a non-starter due to other concerns.

So I think we should just rip the bandaid off and deprecate and eventually remove the pip search command. The only other alternative I can really think of that would actually resolve this, is to switch to using /simple/, but that would then mean getting hit with a 7MB download just to try and search.

Thoughts @pypa/pip-committers?

[pfmoore]

I've rarely used pip search, and never really got useful data from it. But then again, I've never got much use out of PyPI's search facilities anyway.

I'd like to see a good search facility on PyPI, and if there were one, I'd want to be able to use it on the command line as pip search. I'd be happy for pip search to only work on indexes that support a (yet to be defined in a PEP 😄) standard search API, assuming that PyPI is one such index.

So I guess I'm -1 on removing pip search, but happy to acknowledge that it's currently useless - possibly even to the extent of having it simply report "Search is currently disabled because there is no search API defined for package indexes - please use the index search page directly". Long term I do think it's worth having though.

(Another option is that we could develop a plugin API and delegate producing a good pip search to 3rd party contributors. But that's a whole other debate ;-))

[dstufft]

@pfmoore Is there a functional difference of deprecating/removing the search command now, and if a standardized search API gets designed and implemented, adding it back at a later point? Having the command there but always returning an error seems the worst of all possible outcomes.

[pfmoore]

Two, in my view. Both minor.

1. It signals our intent to have a search command, not simply to dump it. (Assuming that is our intent...)
2. If there are people using it, even in its current bad state, leaving it alone avoids harming them, and costs us little. Obviously going as far as a "Currently disabled" message removes this benefit (so this one only applies if we stop at the "acknowledge it's useless" level 😄).

But honestly, I don't care enough to fight for its retention. I do think it deserves a deprecation cycle, but you included that in your proposal anyway, so that's fine.

[cjerdonek]

I tried a couple examples (pip search websockets and pip search trio), and the results seem reasonable / useful.

For the "requests" example, it looks like the reason it's picking up ~900 instead of ~100 is that the first implementation is also searching the summary string instead of just the project name, and "requests" is a common English word. For example, it picks up this:

odoo8-addon-sql-export (8.0.1.0.0.99.dev9)         - Export data in csv file with SQL requests

[pradyunsg]

Do we expect the XML-RPC API to be removed before we manage to make a search API?

If that's the case, we should go ahead and pull out the band aid. Otherwise, what Paul suggests (making it loudly known that we expect pip search to "lie" and be no-so-useful) is also fine.

[pradyunsg]

Any interest in this? I don't mind deprecating it for removal.

[xnuinside]

If pip search will be deprecated, which tool need to use for packages search in PyPi (not in warehouse, in public)? I ask, because, we have several internal cases when we use "pip search". Thanks in advance for the answer!

[di]

@xnuinside I'm not sure I understand your question, specifically the "(not in warehouse, in public)" part, can you elaborate?

The pip search command uses PyPI's XML-RPC API that is available for anyone to use: https://warehouse.readthedocs.io/api-reference/xml-rpc/

[xnuinside]

@di, Dustin, PyPI's API it is what I need) thx