[build-script] LLVM Python builder + infrastructure.

LLVM builder and all the surrounding necessary infrastructure. Because
other builders make use of some pieces of LLVM, LLVM should be the first
builder available.

- Add new options to the Python arguments which were part of the
  build-script-impl and not of build-script. Where the options have been
  used in presets, the naming has been maintained, but in some cases the
  naming of the underlaying variables should explain better what the
  arguments do. The descriptions have been copied where possible.
- Added functionality in the CMake type to generate and build targets.
  The generation uses the source and build directories, and the given
  options, while the building generates CMake arguments and build
  arguments depending on the build variant. The targets will be modified
  for Xcode in all cases (in build-script-impl this step wasn't doing
  uniformily).
- BuilderHostInstallMixin is a mixin for different builder (here it is
  only used in the CMake builder), but contains pieces of code that are
  shared between different builder. It deals with paths when doing
  cross-compiled targets.
- CMakeProductBuilder is a builder that uses CMake to build the given
  product. Subclasses are supposed to customize this builder with their
  own pieces (usually the protected methods). The do_build
  implementation does the generation of the CMake build directory and
  the build; do_test is supposed to build the test targets and
  execute them; while do_install will invoke the install target. For
  most templated methods I found good defaults or common values, but
  there are pieces that are abstract (like build_variant).
- LLVMBase is a mixin for product builders that hold pieces of code used
  from several different builders, but related to LLVM. The Swift
  product will use the values of LTO defined for LLVM, and other
  products like libcxx use the same CMake options as LLVM.
- LLVMBuilder is an specialization of CMakeProductBuilder for LLVM. Most
  of the builders will have to specialize the initializer (to provide
  its product to CMakeProductBuilder and the initial CMake options), and
  some of the template methods. However LLVMBuilder also overrides the
  do_build because it needs to symlink the C++ headers into the build
  directory. The builder also offers a series of public read-only
  properties to recover some of the interesting paths of the results.
- The shell module gets a new helper function to get the SDK path, which
  is used by several products.

Author: drodriguez

utils/build_swift/driver_arguments.py

@@ -341,6 +341,8 @@ def create_argument_parser():
            help='the path to install debug symbols into')
     option('--install-destdir', store_path,
            help='the path to use as the filesystem root for the installation')
+    option('--build-toolchain-only', toggle_true,
+           help='Only build the necessary tools to build an external toolchain')
 
     option(['-j', '--jobs'], store_int('build_jobs'),
            default=multiprocessing.cpu_count(),
@@ -568,12 +570,25 @@ def create_argument_parser():
     option(['--build-libparser-only'], store_true('build_libparser_only'),
            help='build only libParser for SwiftSyntax')
 
+    option('--build-llvm', toggle_false('clean_llvm'),
+           default=False,
+           help='If set to false, cleans the LLVM build result.')
+
+    option('--skip-build-llvm', store_false('build_llvm'),
+           help='Skip building LLVM/Clang')
+
+    option('--skip-build-compiler-rt', store_false('build_compiler_rt'),
+           help='Skip building Compiler-RT')
+
     # -------------------------------------------------------------------------
     in_group('Extra actions to perform before or in addition to building')
 
     option(['-c', '--clean'], store_true,
            help='do a clean build')
 
+    option('--reconfigure', store_true,
+           help='force a CMake configuration run even if CMakeCache.txt already exists')
+
     option('--export-compile-commands', toggle_true,
            help='generate compilation databases in addition to building')
 
@@ -924,6 +939,10 @@ def create_argument_parser():
            default='X86;ARM;AArch64;PowerPC;SystemZ;Mips',
            help='LLVM target generators to build')
 
+    option('--llvm-include-tests', toggle_true,
+           default=True,
+           help='Generate testing targets for LLVM. Set to true by default.')
+
     # -------------------------------------------------------------------------
     in_group('Build settings for Android')
 

utils/build_swift/tests/expected_options.py

@@ -56,6 +56,7 @@
     'build_android': False,
     'build_args': [],
     'build_benchmarks': True,
+    'build_compiler_rt': True,
     'build_cygwin': True,
     'build_external_benchmarks': False,
     'build_foundation': False,
@@ -69,6 +70,7 @@
     'build_linux': True,
     'build_llbuild': False,
     'build_lldb': False,
+    'build_llvm': True,
     'build_libcxx': False,
     'build_ninja': False,
     'build_osx': True,
@@ -88,6 +90,7 @@
     'build_swiftevolve': False,
     'build_indexstoredb': False,
     'build_sourcekitlsp': False,
+    'build_toolchain_only': False,
     'build_tvos': True,
     'build_tvos_device': False,
     'build_tvos_simulator': False,
@@ -102,6 +105,7 @@
     'clang_profile_instr_use': None,
     'clang_user_visible_version': defaults.CLANG_USER_VISIBLE_VERSION,
     'clean': False,
+    'clean_llvm': False,
     'cmake': None,
     'cmake_generator': 'Ninja',
     'cmark_assertions': True,
@@ -154,11 +158,13 @@
     'lldb_build_with_xcode': '1',
     'llvm_assertions': True,
     'llvm_build_variant': 'Debug',
+    'llvm_include_tests': True,
     'llvm_max_parallel_lto_link_jobs':
         host.max_lto_link_job_counts()['llvm'],
     'llvm_targets_to_build': 'X86;ARM;AArch64;PowerPC;SystemZ;Mips',
     'long_test': False,
     'lto_type': None,
+    'reconfigure': False,
     'show_sdks': False,
     'skip_build': False,
     'stdlib_deployment_targets': None,
@@ -397,6 +403,9 @@ class IgnoreOption(_BaseOption):
     SetOption('--skip-tvos', dest='tvos', value=False),
     SetOption('--skip-watchos', dest='watchos', value=False),
 
+    SetFalseOption('--skip-build-compiler-rt', dest='build_compiler_rt'),
+    SetFalseOption('--skip-build-llvm', dest='build_llvm'),
+
     SetTrueOption('--benchmark'),
     SetTrueOption('--clean'),
     SetTrueOption('--dry-run'),
@@ -408,6 +417,7 @@ class IgnoreOption(_BaseOption):
     SetTrueOption('--lldb', dest='build_lldb'),
     SetTrueOption('--libcxx', dest='build_libcxx'),
     SetTrueOption('--playgroundsupport', dest='build_playgroundsupport'),
+    SetTrueOption('--reconfigure'),
     SetTrueOption('--skip-build'),
     SetTrueOption('--swiftpm', dest='build_swiftpm'),
     SetTrueOption('--swiftsyntax', dest='build_swiftsyntax'),
@@ -434,6 +444,7 @@ class IgnoreOption(_BaseOption):
     EnableOption('--build-swift-static-sdk-overlay'),
     EnableOption('--build-swift-static-stdlib'),
     EnableOption('--build-swift-stdlib-unittest-extra'),
+    EnableOption('--build-toolchain-only'),
     EnableOption('--distcc'),
     EnableOption('--enable-asan'),
     EnableOption('--enable-lsan'),
@@ -446,6 +457,7 @@ class IgnoreOption(_BaseOption):
     EnableOption('--host-test'),
     EnableOption('--libdispatch', dest='build_libdispatch'),
     EnableOption('--libicu', dest='build_libicu'),
+    EnableOption('--llvm-include-tests'),
     EnableOption('--indexstore-db', dest='build_indexstoredb'),
     EnableOption('--sourcekit-lsp', dest='build_sourcekitlsp'),
     EnableOption('--long-test'),
@@ -461,6 +473,7 @@ class IgnoreOption(_BaseOption):
     EnableOption('--watchos'),
     EnableOption('--xctest', dest='build_xctest'),
 
+    DisableOption('--build-llvm', dest='clean_llvm'),
     DisableOption('--skip-build-android', dest='build_android'),
     DisableOption('--skip-build-benchmarks', dest='build_benchmarks'),
     DisableOption('--skip-build-cygwin', dest='build_cygwin'),

utils/swift_build_support/swift_build_support/cmake.py

@@ -16,6 +16,8 @@
 
 from __future__ import absolute_import
 
+import itertools
+import os
 from numbers import Number
 
 from . import shell
@@ -128,7 +130,7 @@ def common_options(self):
         define("CMAKE_CXX_COMPILER:PATH", toolchain.cxx)
         define("CMAKE_LIBTOOL:PATH", toolchain.libtool)
 
-        if args.cmake_generator == 'Xcode':
+        if self.__is_xcode_generator:
             define("CMAKE_CONFIGURATION_TYPES",
                    "Debug;Release;MinSizeRel;RelWithDebInfo")
 
@@ -142,9 +144,8 @@ def common_options(self):
             define("CLANG_VERSION_MINOR:STRING", minor)
             define("CLANG_VERSION_PATCH:STRING", patch)
 
-        if args.build_ninja and args.cmake_generator == 'Ninja':
-            define('CMAKE_MAKE_PROGRAM', toolchain.ninja)
-        elif args.cmake_generator == 'Ninja' and toolchain.ninja is not None:
+        if self.__is_ninja_generator and \
+                (args.build_ninja or toolchain.ninja is not None):
             define('CMAKE_MAKE_PROGRAM', toolchain.ninja)
 
         return options
@@ -161,18 +162,109 @@ def build_args(self):
 
         build_args = list(args.build_args)
 
-        if args.cmake_generator == 'Ninja':
+        if self.__is_ninja_generator:
             build_args += ['-j%s' % jobs]
             if args.verbose_build:
                 build_args += ['-v']
 
-        elif args.cmake_generator == 'Unix Makefiles':
+        elif self.__is_makefile_generator:
             build_args += ['-j%s' % jobs]
             if args.verbose_build:
                 build_args += ['VERBOSE=1']
 
-        elif args.cmake_generator == 'Xcode':
+        elif self.__is_xcode_generator:
             build_args += ['-parallelizeTargets',
                            '-jobs', str(jobs)]
 
         return build_args
+
+    def generate(self, source_dir, build_dir, cmake_options=None):
+        cmake_options = [] if cmake_options is None else cmake_options
+        shell.makedirs(build_dir)
+        extra_environment = {}
+        if self.args.distcc:
+            extra_environment['DISTCC_HOSTS'] = 'localhost,lzo,cpp'
+        with shell.pushd(build_dir):
+            shell.call([self.toolchain.cmake] + self.common_options() +
+                       cmake_options + self.args.extra_cmake_options +
+                       [source_dir],
+                       env=extra_environment)
+
+    def generate_if_needed(self, source_dir, build_dir, cmake_options=None):
+        # Compute the generator output file to check for, to determine if we
+        # must reconfigure. We only handle Ninja for now.
+        #
+        # This is important for ensuring that if a Cmake configuration fails in
+        # CI, that we will still be willing to rerun the configuration process.
+        cmake_options = [] if cmake_options is None else cmake_options
+        generator_output_path = None
+        if self.__is_ninja_generator:
+            generator_output_path = os.path.join(build_dir, 'build.ninja')
+
+        # Configure if necessary
+        cmake_cache_path = os.path.join(build_dir, 'CMakeCache.txt')
+        if self.args.reconfigure or \
+                not os.path.exists(cmake_cache_path) or \
+                (generator_output_path is not None and
+                    not os.path.exists(generator_output_path)):
+            self.generate(source_dir, build_dir, cmake_options)
+
+    def build_targets(self, build_dir, build_variant, targets=None):
+        targets = ['all'] if targets is None else targets
+        self.__cmake_build(build_dir,
+                           cmake_arguments=self.__cmake_config(build_variant),
+                           build_arguments=self.build_args(),
+                           targets=targets)
+
+    def install_targets(self, build_dir, targets=None, environment=None):
+        targets = [] if targets is None else targets
+        self.__cmake_build(build_dir,
+                           targets=targets,
+                           environment=environment)
+
+    def __cmake_build(self, build_dir, cmake_arguments=None,
+                      build_arguments=None, targets=None, environment=None):
+        cmake_arguments = [] if cmake_arguments is None else cmake_arguments
+        build_arguments = [] if build_arguments is None else build_arguments
+        targets = [] if targets is None else targets
+        distcc_pump = []
+        if self.args.distcc:
+            distcc_pump = [self.toolchain.distcc_pump]
+
+        if self.__is_xcode_generator:
+            targets = self.__transform_targets_for_xcode(targets)
+
+        shell.call(distcc_pump + [self.toolchain.cmake, '--build', build_dir] +
+                   cmake_arguments + ['--'] + build_arguments + targets,
+                   env=environment)
+
+    def __cmake_config(self, build_variant):
+        if self.__is_xcode_generator:
+            return [
+                '--target', 'ZERO_CHECK',
+                '--config', build_variant
+            ]
+        else:
+            return []
+
+    @property
+    def __is_xcode_generator(self):
+        return self.args.cmake_generator == 'Xcode'
+
+    @property
+    def __is_ninja_generator(self):
+        return self.args.cmake_generator == 'Ninja'
+
+    @property
+    def __is_makefile_generator(self):
+        return self.args.cmake_generator == 'Unix Makefiles'
+
+    def __transform_targets_for_xcode(self, targets):
+        # Xcode generator uses "ALL_BUILD" instead of "all".
+        # Also, xcodebuild uses -target instead of bare names.
+        targets = ['ALL_BUILD'
+                   if t == 'all' else t
+                   for t in targets]
+        targets = itertools.chain.from_iterable(
+            [['-target', t] for t in targets])
+        return targets

utils/swift_build_support/swift_build_support/products/build_script_impl_builder.py

@@ -141,6 +141,9 @@ def __convert_to_impl_arguments(self):
                 pipes.quote(arg) for arg in cmake.build_args()),
         ]
 
+        if args.reconfigure:
+            impl_args += ['--reconfigure']
+
         # Compute any product specific cmake arguments.
         for product_class in self.__product_classes:
             if not product_class.is_build_script_impl_product():
@@ -183,6 +186,8 @@ def __convert_to_impl_arguments(self):
         if args.cross_compile_hosts:
             impl_args += [
                 "--cross-compile-hosts", " ".join(args.cross_compile_hosts)]
+        if args.build_toolchain_only:
+            impl_args += ['--build-toolchain-only']
 
         if args.test_paths:
             impl_args += ["--test-paths", " ".join(args.test_paths)]
@@ -227,7 +232,6 @@ def __convert_to_impl_arguments(self):
 
         if args.skip_build:
             impl_args += ["--skip-build-cmark",
-                          "--skip-build-llvm",
                           "--skip-build-swift"]
         if not args.build_benchmarks:
             impl_args += ["--skip-build-benchmarks"]
@@ -238,6 +242,11 @@ def __convert_to_impl_arguments(self):
             impl_args += ["--skip-build-foundation"]
         if not args.build_xctest:
             impl_args += ["--skip-build-xctest"]
+        if not args.build_llvm:
+            impl_args += ['--skip-build-llvm']
+        impl_args += ['--build-llvm=%s' % args.clean_llvm]
+        if not args.build_compiler_rt:
+            impl_args += ["--skip-build-compiler-rt"]
         if not args.build_lldb:
             impl_args += ["--skip-build-lldb"]
         if not args.build_llbuild:
@@ -425,6 +434,8 @@ def __convert_to_impl_arguments(self):
                 os.path.abspath(args.clang_profile_instr_use)
             ]
 
+        impl_args += ['--llvm-include-tests=%s' % args.llvm_include_tests]
+
         if args.lit_args:
             impl_args += ["--llvm-lit-args=%s" % args.lit_args]