Implement "hook" support for package signature verification.

[nejucomo]

Synopsis

Some people want package signature verification during their pip installs. Other people think relying on authenticated package repository connections (such as over TLS) is sufficient for their needs.

Of those who want package signature verification, there is disagreement about how to tell PIP which signatures to trust (and how users will manage package signing public keys).

Rationale

The rationale for this ticket is to provide a mechanism in mainline pip for signature verification enthusiasts to experiment with different approaches. If a particular approach becomes popular, pip could consider incorporating that particular approach.

In the meantime, rather than have endless committee-style-arguments about how to do package verification, we should have a system that lets users choose for themselves, but only if they opt in.

Also, it keeps package verification cleanly separate from the pip codebase.

Criteria

This ticket may be marked as wontfix, or some other status to indicate that the pip developers reject this proposal.

This ticket may be marked closed, only when these conditions are met:

• A user can configure an arbitrary "hook" to process downloaded packages prior to proceeding.
  • By default, this hook is configured to nothing, and pip with this feature behaves like pip without this feature.
  • A hook provides an interface which:
    • takes two inputs:
      1. a path to a local file, which is a newly downloaded package
      2. a URL which the file was retrieved from
    • and returns a single boolean indicating "accept/reject".
  • When a user has a hook configured, it is invoked on every downloaded package file before any other processing. If it returns true, pip proceeds as normal. If it returns false, pip logs an error and exits.

Implementation Details

I prefer a hook api where the config specifies a path to an executable in pip's config file. The inputs are passed as commandline arguments to a subprocess which invokes that command. The hook's stdout & stderr are the same as the parent pip process. The exit status is 0 to indicate "accept package" and non-zero to indicate "reject package".

-but I'd be happy with any system that fulfills the Criteria above.

Related Issues

Note, there is a less-well-specified ticket #425. I made this ticket because the vagueness of that ticket makes it difficult to close. (Is #425 satisfied by TLS authentication to package repositories based on a standard OS or user trust root? Does it imply or require package signature verification?)

[dstufft]

Thanks for this ticket!

Immediate thoughts are that I think package signing at this stage in the game is premature as there are other avenues of very serious attacks available... However the proposed system is not really related strictly to signing. It could also be used to implement something like https://pypi.python.org/pypi/peep. So personally I'm going to think about this ticket a little bit before hand to figure out if I believe it's going to provide a useful feature without serious short comings in the near term.

[dstufft]

This has another useful purpose too, companies or organizations could use it to disallow installing items that haven't been through a security audit or license review or what have you. For instance OpenStack could potentially use it to help ensure that an unapproved dependency isn't added.

[westurner]

A syntax like the following would be convenient:

pip install --verify-<sig> -e git+https://github.com/pypa/pip#egg=pip

...

• http://www.pip-installer.org/en/latest/usage.html#pip-install
• http://www.pip-installer.org/en/latest/usage.html#pip-verify ?
• https://python-packaging-user-guide.readthedocs.org/en/latest/packaging_tutorial.html#create-your-first-release

These may be helpful for creating documentation on this feature and how it relates to other components of a secure python packaging process:

Source Repository GPG

• https://en.wikipedia.org/wiki/GNU_Privacy_Guard (PGP)
• http://stackoverflow.com/questions/10077996/sign-git-commits-with-gpg
• http://stackoverflow.com/questions/11556184/whats-the-purpose-of-signing-changesets-in-mercurial

Python Package GPG (./<package>.asc)

• http://pythonhosted.org/distlib/tutorial.html#signing-a-distribution

• http://pythonhosted.org/distlib/tutorial.html#verifying-signatures

  For any archive downloaded from an index, you can retrieve any signature by just appending .asc to the path portion of the download URL for the archive, and downloading that.

• https://pypi.python.org/packages/source/p/pip/pip-1.3.1.tar.gz.asc#md5=cbb27a191cebc58997c4da8513863153

Python Wheel JWS S/MIME (PEP 427)

• http://www.python.org/dev/peps/pep-0427/#signed-wheel-files
• https://tools.ietf.org/html/draft-ietf-jose-json-web-signature-11 (html)
• https://tools.ietf.org/html/draft-ietf-jose-json-web-key-11 (html)
• https://en.wikipedia.org/wiki/X.509
• https://bitbucket.org/dholth/wheel/src/tip/wheel/signatures/__init__.py
• https://distlib.readthedocs.org/en/latest/internals.html#the-wheel-api
• https://payswarm.com/specs/source/web-keys/ (http://json-ld.org)

Index Mirror DSA (PEP 381)

• http://www.python.org/dev/peps/pep-0381/#mirror-authenticity
• https://en.wikipedia.org/wiki/Digital_Signature_Algorithm

Package Signatures for .deb, .rpm, ...

• https://en.wikipedia.org/wiki/List_of_software_package_management_systems
• http://man.he.net/man8/apt-key
• http://wiki.debian.org/SecureApt
• http://linux.die.net/man/5/yum.conf # gpgcheck, localpkg_gpgcheck, repo_gpgcheck
• http://iuscommunity.org/pages/CreatingAGPGKeyandSigningRPMs.html

Python Package Configuration Management Systems

• https://github.com/puppetlabs/puppet/blob/master/lib/puppet/provider/package/pip.rb
• https://github.com/opscode/chef/blob/master/lib/chef/provider/package/easy_install.rb
• https://github.com/saltstack/salt/blob/develop/salt/modules/pip.py
• https://github.com/ansible/ansible/blob/devel/library/packaging/pip
• http://docs.bcfg2.org/server/plugins/generators/packages.html#handling-gpg-keys

[Cryptographic] Hash Functions

• https://en.wikipedia.org/wiki/Cryptographic_hash_function#Verifying_the_integrity_of_files_or_messages
• https://en.wikipedia.org/wiki/Cryptographic_hash_function#Hash_functions_based_on_block_ciphers
• https://en.wikipedia.org/wiki/Category:Cryptographic_hash_functions
• https://en.wikipedia.org/wiki/Hash_function_security_summary
• https://en.wikipedia.org/wiki/Category:Broken_hash_functions
  • https://en.wikipedia.org/wiki/MD5
  • https://en.wikipedia.org/wiki/SHA-1
• http://pythonhosted.org/passlib/lib/passlib.hash.html#unix-modular-crypt-hashes
• http://pythonhosted.org/passlib/modular_crypt_format.html

seeAlso: #425 (this comment)

[westurner]

This has another useful purpose too, companies or organizations could use it to disallow installing items that haven't been through a security audit or license review or what have you. For instance OpenStack could potentially use it to help ensure that an unapproved dependency isn't added.

So would a use case be something like verifying a dependency graph of packages' checksums and metadata?

[pnasrat]

I'm pretty interested in implementing this. Should I knock up a strawman Pull Request?

[dstufft]

So I've thought about this some more, and it's really started to grow on me.

Some thoughts on what I'd like to see:

I think the hook should be a python hook, that allows us to pass data about the thing we are trying to install into the hook easily, and receive more complex return types than pass/fail. If someone wants a simple call a command and the subprocess module is simple to use so the python portion of the hook in that case would be a small shim.

I think there needs to be more return types than Pass/Fail. In my mind there are four distinct return values. They are Pass, Warn, Retry, Fail. The defintions of them (again in my mind) would be:

Pass: The installation looks fine, go ahead and install it
Warn: The installation is ok, but there is a warning that should be presented to the user (this one is possibly not needed and warning could be done via the logging system).
Retry: This particular package is unsuitable, but pip can attempt to locate another package that fulfils this dependency (either from a different location, a different type, or a different version).
Fail: This package is unsuitable. Pip should not attempt to satisfy it and should throw an error.

At least that's what I think :) I'd love a PR that implements this hook feature.

[westurner]

I think there needs to be more return types than Pass/Fail. In my mind there are four distinct return values. They are Pass, Warn, Retry, Fail. The defintions of them (again in my mind) would be:

Pass: The installation looks fine, go ahead and install it
Warn: The installation is ok, but there is a warning that should be presented to the user (this one is possibly not needed and warning could be done via the logging system).
Retry: This particular package is unsuitable, but pip can attempt to locate another package that fulfils this dependency (either from a different location, a different type, or a different version).
Fail: This package is unsuitable. Pip should not attempt to satisfy it and should throw an error.

What are the os.exit() codes for each of these?

So would a use case be something like verifying a dependency graph of packages' checksums and metadata?

Pip package lists are specified as requirement specifiers in requirements.txt files.

So, in order to verify a list (a topologically sorted dependency graph) of python packages required for an environment, it is/will/would_be necessary to determine the path to the .asc file (for each/every/most package listed in a requirements description format).

[dstufft]

I don't think it should be shelling out to executable by default. It should call a python function as a hook and use a python return value. If people want their particular instance of the hook to shell out that's a simple python wrapper that they can shell out on their own.

[westurner]

takes two inputs:

• a path to a local file, which is a newly downloaded package
• a URL which the file was retrieved from

source_url = "https://pypi.python.org/packages/source/p/pip/pip-1.3.1.tar.gz#md5=cbb27a191cebc58997c4da8513863153"
asc_url = "https://pypi.python.org/packages/source/p/pip/pip-1.3.1.tar.gz.asc#md5=cbb27a191cebc58997c4da8513863153"
pkg_file = "./path/to/pip-1.3.1.tar.gz"
asc_file = "./path/to/pip-1.3.1.tar.gz.asc"

def verify(pkg_file, asc_file, source_url, asc_url):
    return [distlib].index.verify_signature(asc_file, pkg_file)

... http://pythonhosted.org/distlib/tutorial.html#verifying-signatures

[dstufft]

Distlib's Signature support is inherently broken. You cannot just pipe out to GPG and trust whatever keys are in the trustdb. Just because you trust me for X does not mean you trust me for Y.

[westurner]

[Distlib Signature Support]

So remove [distlib].index? I guess the question I was trying to ask was: what is the minimal python function call signature necessary to most correctly verify what it is we are trying to verify here.

I prefer a hook api where the config specifies a path to an executable in pip's config file.

http://stevedore.readthedocs.org/en/latest/ may be useful for adding hooks / plugins / extension points and/or as a reference for [setuptools entry_point configuration]

• http://pythonhosted.org/setuptools/pkg_resources.html#entry-points
• "dotted.python:name"
• example: https://github.com/okfn/ckan/blob/master/setup.py#L106

Mercurial hooks and extensions pass something like a context dict instead of positional arguments as with the verify() interface listed above.

The inputs are passed as commandline arguments to a subprocess which invokes that command.

How and when should I sanitize this input? What is the best way to specify the command arguments?

cmd = ("bash",string_downlaoded_from_the_internets) , shell=False
# NOT
cmd = "bash %s" % string_downlaoded_from_the_internets

The hook's stdout & stderr are the same as the parent pip process. The exit status is 0 to indicate "accept package" and non-zero to indicate "reject package".

From a shell script, is there then a way to differentiate between failed and sig-check-failed for an install -r requirements.txt?

I would be in favor of either and/or both:

• Python argspec for verify(): e.g. verify(pkg_file, asc_file, source_url, asc_url)
• a context dict keyset for the argspec parameters: e.g. dict.fromkeys(('pkg_file', 'asc_file', 'source_url', 'asc_url'), None)

[westurner]

... These were the stevedore documentation links I was looking for:

• http://stevedore.readthedocs.org/en/latest/patterns_loading.html#hooks-single-name-many-entry-points
• http://stevedore.readthedocs.org/en/latest/patterns_enabling.html

• Enabled through Installation
• Enabled explicitly
• Self-Enabled

[westurner]

Is the package signature hook called for .zip, .egg, and .whl packages AND for editable distributions?

There are new metadata attributes for package source locations.

# PEP 345 Metadata 1.2
download_url = str
# PEP 426 Metadata 2.0
source_url = {
    'key': http_path,
    'key_2': 'git+https://editable/path@version'
}

• http://www.python.org/dev/peps/pep-0345/#download-url
• http://www.python.org/dev/peps/pep-0426/#source-url