Add option to build charm with binary wheels (#620)

The reactive framework was born in a time of the now legacy charm
store.  The legacy charm store did not have awareness for binary
compatibility with various CPU architectures.

In order to enable the use of Python virtual environments, and at
the same time support multiple CPU architectures, the charm-tools
build action currently uses `pip download` to retrieve the source
code of Python dependencies.  On charm install layer-basic will
subsequently use pip to build and install the dependencies on the
target.

Over the recent years this approach has become increasingly
difficult to maintain.  The Python ecosystem has a plethora of
build automation and package mangement projects, and each and
every Python dependency makes their own choices about what to use.

Adding to that, pip does not really support automatic discovery
of build dependencies on download (ref pypa/pip#7863).

Today the legacy charm store has been replaced by charmhub, which
does have awareness of CPU architectures.  The Launchpad build
automation service has also gained support for charm recipes,
which allow developers to build their charms accross a broad
range of CPU architectures.

To leverage these capabilities and relieve reactive charm
maintainers of the duty of manually hunting down and compiling
requirement files for build dependencies, this patch adds
support for building charms using binary wheels.

Allowing the user to specify that wheels should be built from
source could be useful for detecting any missing build environment
binary dependencies (C/Rust library packages etc).

Signed-off-by: Frode Nordahl <frode.nordahl@canonical.com>

charmtools/build/builder.py

@@ -1163,6 +1163,18 @@ def main(args=None):
                         help="Force raw output (color)")
     parser.add_argument('--charm-file', '-F', action='store_true',
                         help='Create a .charm file in the current directory')
+    parser.add_argument('--binary-wheels', action='store_true',
+                        help='Populate the charm wheelhouse with binary '
+                             'wheels that matches the Python version and '
+                             'architecture the charm is built on. Note that '
+                             'to use this option you must ensure that you '
+                             'have a series/architecture aware build and '
+                             'distribution infrastructure configured for '
+                             'your charm (such as Launchpad and Charmhub).')
+    parser.add_argument('--binary-wheels-from-source', action='store_true',
+                        help='Same as --binary-wheels but build all the '
+                             'wheels from source code, even if a binary '
+                             'distribution is available on PyPi.')
     parser.add_argument('charm', nargs="?", default=".", type=path,
                         help='Source directory for charm layer to build '
                              '(default: .)')
@@ -1181,6 +1193,8 @@ def main(args=None):
     LayerFetcher.NO_LOCAL_LAYERS = build.no_local_layers
 
     WheelhouseTactic.per_layer = build.wheelhouse_per_layer
+    WheelhouseTactic.binary_build = build.binary_wheels
+    WheelhouseTactic.binary_build_from_source = build.binary_wheels_from_source
 
     configLogging(build)
 

charmtools/build/tactics.py

@@ -1044,6 +1044,8 @@ class WheelhouseTactic(ExactMatch, Tactic):
     FILENAME = 'wheelhouse.txt'
     removed = []  # has to be class level to affect all tactics during signing
     per_layer = False
+    binary_build = False
+    binary_build_from_source = False
 
     def __init__(self, *args, **kwargs):
         super(WheelhouseTactic, self).__init__(*args, **kwargs)
@@ -1110,8 +1112,14 @@ def read(self):
     def _add(self, wheelhouse, *reqs):
         with utils.tempdir(chdir=False) as temp_dir:
             # put in a temp dir first to ensure we track all of the files
-            self._pip('download', '--no-binary', ':all:', '-d', temp_dir,
-                      *reqs)
+            _no_binary_opts = ('--no-binary', ':all:')
+            if self.binary_build_from_source or self.binary_build:
+                self._pip('wheel',
+                          *_no_binary_opts
+                          if self.binary_build_from_source else tuple(),
+                          '-w', temp_dir, *reqs)
+            else:
+                self._pip('download', *_no_binary_opts, '-d', temp_dir, *reqs)
             log.debug('Copying wheels:')
             for wheel in temp_dir.files():
                 log.debug('  ' + wheel.name)