Bug 784841 - Part 2: Implement sandboxing for Python build files; r=t…

…ed,glandium

This is the beginning of Mozilla's new build system.

In this patch, we have a Python sandbox tailored for execution
of Python scripts which will define the build system. We also have a
build reader that traverses a linked set of scripts.

More details are available in the thorough README.rst files as part of
this patch.
* * *
Bug 784841 - Part 2b: Option to not descend into child moz.build files; r=ted

mach

@@ -42,6 +42,7 @@ MACH_MODULES = [
     'python/mozboot/mozboot/mach_commands.py',
     'python/mozbuild/mozbuild/config.py',
     'python/mozbuild/mozbuild/mach_commands.py',
+    'python/mozbuild/mozbuild/frontend/mach_commands.py',
     'testing/mochitest/mach_commands.py',
     'testing/xpcshell/mach_commands.py',
 ]

python/Makefile.in

@@ -12,6 +12,7 @@ include $(DEPTH)/config/autoconf.mk
 test_dirs := \
   mozbuild/mozbuild/test \
   mozbuild/mozbuild/test/compilation \
+  mozbuild/mozbuild/test/frontend \
   $(NULL)
 
 PYTHON_UNIT_TESTS := $(foreach dir,$(test_dirs),$(wildcard $(srcdir)/$(dir)/*.py))

python/mozbuild/README.rst

@@ -10,4 +10,30 @@ Modules Overview
 
 * mozbuild.compilation -- Functionality related to compiling. This
   includes managing compiler warnings.
+* mozbuild.frontend -- Functionality for reading build frontend files
+  (what defines the build system) and converting them to data structures
+  which are fed into build backends to produce backend configurations.
+
+Overview
+========
+
+The build system consists of frontend files that define what to do. They
+say things like "compile X" "copy Y."
+
+The mozbuild.frontend package contains code for reading these frontend
+files and converting them to static data structures. The set of produced
+static data structures for the tree constitute the current build
+configuration.
+
+There exist entities called build backends. From a high level, build
+backends consume the build configuration and do something with it. They
+typically produce tool-specific files such as make files which can be used
+to build the tree.
+
+Builders are entities that build the tree. They typically have high
+cohesion with a specific build backend.
+
+Piecing it all together, we have frontend files that are parsed into data
+structures. These data structures are fed into a build backend. The output
+from build backends is used by builders to build the tree.
 

python/mozbuild/TODO

@@ -0,0 +1,3 @@
+dom/imptests Makefile.in's are autogenerated. See
+dom/imptests/writeMakefile.py and bug 782651. We will need to update
+writeMakefile.py to produce mozbuild files.

python/mozbuild/mozbuild/frontend/README.rst

@@ -0,0 +1,137 @@
+=================
+mozbuild.frontend
+=================
+
+The mozbuild.frontend package is of sufficient importance and complexity
+to warrant its own README file. If you are looking for documentation on
+how the build system gets started, you've come to the right place.
+
+Overview
+========
+
+The build system is defined by a bunch of files in the source tree called
+*mozbuild* files. Each *mozbuild* file defines a unique part of the overall
+build system. This includes information like "compile file X," "copy this
+file here," "link these files together to form a library." Together,
+all the *mozbuild* files define how the entire build system works.
+
+*mozbuild* files are actually Python scripts. However, their execution
+is governed by special rules. This will be explained later.
+
+Once a *mozbuild* file has executed, it is converted into a set of static
+data structures.
+
+The set of all data structures from all relevant *mozbuild* files
+constitutes the current build configuration.
+
+How *mozbuild* Files Work
+=========================
+
+As stated above, *mozbuild* files are actually Python scripts. However,
+their behavior is very different from what you would expect if you executed
+the file using the standard Python interpreter from the command line.
+
+There are two properties that make execution of *mozbuild* files special:
+
+1. They are evaluated in a sandbox which exposes a limited subset of Python
+2. There is a special set of global variables which hold the output from
+   execution.
+
+The limited subset of Python is actually an extremely limited subset.
+Only a few built-ins are exposed. These include *True*, *False*, and
+*None*. Global functions like *import*, *print*, and *open* aren't defined.
+Without these, *mozbuild* files can do very little. This is by design.
+
+The side-effects of the execution of a *mozbuild* file are used to define
+the build configuration. Specifically, variables set during the execution
+of a *mozbuild* file are examined and their values are used to populate
+data structures.
+
+The enforced convention is that all UPPERCASE names inside a sandbox are
+reserved and it is the value of these variables post-execution that is
+examined. Furthermore, the set of allowed UPPERCASE variable names and
+their types is statically defined. If you attempt to reference or assign
+to an UPPERCASE variable name that isn't known to the build system or
+attempt to assign a value of the wrong type (e.g. a string when it wants a
+list), an error will be raised during execution of the *mozbuild* file.
+This strictness is to ensure that assignment to all UPPERCASE variables
+actually does something. If things weren't this way, *mozbuild* files
+might think they were doing something but in reality wouldn't be. We don't
+want to create false promises, so we validate behavior strictly.
+
+If a variable is not UPPERCASE, you can do anything you want with it,
+provided it isn't a function or other built-in. In other words, normal
+Python rules apply.
+
+All of the logic for loading and evaluating *mozbuild* files is in the
+*reader* module. Of specific interest is the *MozbuildSandbox* class. The
+*BuildReader* class is also important, as it is in charge of
+instantiating *MozbuildSandbox* instances and traversing a tree of linked
+*mozbuild* files. Unless you are a core component of the build system,
+*BuildReader* is probably the only class you care about in this module.
+
+The set of variables and functions *exported* to the sandbox is defined by
+the *sandbox_symbols* module. These data structures are actually used to
+populate MozbuildSandbox instances. And, there are tests to ensure that the
+sandbox doesn't add new symbols without those symbols being added to the
+module. And, since the module contains documentation, this ensures the
+documentation is up to date (at least in terms of symbol membership).
+
+How Sandboxes are Converted into Data Structures
+================================================
+
+The output of a *mozbuild* file execution is essentially a dict of all
+the special UPPERCASE variables populated during its execution. While these
+dicts are data structures, they aren't the final data structures that
+represent the build configuration.
+
+We feed the *mozbuild* execution output (actually *reader.MozbuildSandbox*
+instances) into a *BuildDefinitionEmitter* class instance. This class is
+defined in the *emitter* module. *BuildDefinitionEmitter* converts the
+*MozbuildSandbox* instances into instances of the *BuildDefinition*-derived
+classes from the *data* module.
+
+All the classes in the *data* module define a domain-specific
+component of the build configuration. File compilation and IDL generation
+are separate classes, for example. The only thing these classes have in
+common is that they inherit from *BuildDefinition*, which is merely an
+abstract base class.
+
+The set of all emitted *BuildDefinition* instances (converted from executed
+*mozbuild* files) constitutes the aggregate build configuration. This is
+the authoritative definition of the build system and is what's used by
+all downstream consumers, such as backends. There is no monolithic build
+system configuration class. Instead, the build system configuration is
+modeled as a collection/iterable of *BuildDefinition*.
+
+There is no defined mapping between the number of
+*MozbuildSandbox*/*moz.build* instances and *BuildDefinition* instances.
+Some *mozbuild* files will emit only 1 *BuildDefinition* instance. Some
+will emit 7. Some may even emit 0!
+
+The purpose of this *emitter* layer between the raw *mozbuild* execution
+result and *BuildDefinition* is to facilitate additional normalization and
+verification of the output. The downstream consumer of the build
+configuration are build backends. And, there are several of these. There
+are common functions shared by backends related to examining the build
+configuration. It makes sense to move this functionality upstream as part
+of a shared pipe. Thus, *BuildDefinitionEmitter* exists.
+
+Other Notes
+===========
+
+*reader.BuildReader* and *emitter.BuildDefinitionEmitter* have a nice
+stream-based API courtesy of generators. When you hook them up properly,
+*BuildDefinition* instances can be consumed before all *mozbuild* files have
+been read. This means that errors down the pipe can trigger before all
+upstream tasks (such as executing and converting) are complete. This should
+reduce the turnaround time in the event of errors. This likely translates to
+a more rapid pace for implementing backends, which require lots of iterative
+runs through the entire system.
+
+In theory, the frontend to the build system is generic and could be used
+by any project. In practice, parts are specifically tailored towards
+Mozilla's needs. With a little work, the core build system bits could be
+separated into its own package, independent of the Mozilla bits. Or, one
+could simply replace the Mozilla-specific pieces in the *variables*, *data*,
+and *emitter* modules to reuse the core logic.

python/mozbuild/mozbuild/frontend/mach_commands.py

@@ -0,0 +1,171 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+from __future__ import print_function, unicode_literals
+
+import textwrap
+
+from mach.decorators import (
+    CommandArgument,
+    CommandProvider,
+    Command
+)
+
+from mozbuild.frontend.sandbox_symbols import (
+    FUNCTIONS,
+    SPECIAL_VARIABLES,
+    VARIABLES,
+    doc_to_paragraphs,
+)
+
+
+def get_doc(doc):
+    """Split documentation into summary line and everything else."""
+    paragraphs = doc_to_paragraphs(doc)
+
+    summary = paragraphs[0]
+    extra = paragraphs[1:]
+
+    return summary, extra
+
+def print_extra(extra):
+    """Prints the 'everything else' part of documentation intelligently."""
+    for para in extra:
+        for line in textwrap.wrap(para):
+            print(line)
+
+        print('')
+
+    if not len(extra):
+        print('')
+
+
+@CommandProvider
+class MozbuildFileCommands(object):
+    @Command('mozbuild-reference',
+        help='View reference documentation on mozbuild files.')
+    @CommandArgument('symbol', default=None, nargs='*',
+        help='Symbol to view help on. If not specified, all will be shown.')
+    @CommandArgument('--name-only', '-n', default=False, action='store_true',
+        help='Print symbol names only.')
+    def reference(self, symbol, name_only=False):
+        if name_only:
+            for s in sorted(VARIABLES.keys()):
+                print(s)
+
+            for s in sorted(FUNCTIONS.keys()):
+                print(s)
+
+            for s in sorted(SPECIAL_VARIABLES.keys()):
+                print(s)
+
+            return 0
+
+        if len(symbol):
+            for s in symbol:
+                if s in VARIABLES:
+                    self.variable_reference(s)
+                    continue
+                elif s in FUNCTIONS:
+                    self.function_reference(s)
+                    continue
+                elif s in SPECIAL_VARIABLES:
+                    self.special_reference(s)
+                    continue
+
+                print('Could not find symbol: %s' % s)
+                return 1
+
+            return 0
+
+        print('=========')
+        print('VARIABLES')
+        print('=========')
+        print('')
+        print('This section lists all the variables that may be set ')
+        print('in moz.build files.')
+        print('')
+
+        for v in sorted(VARIABLES.keys()):
+            self.variable_reference(v)
+
+        print('=========')
+        print('FUNCTIONS')
+        print('=========')
+        print('')
+        print('This section lists all the functions that may be called ')
+        print('in moz.build files.')
+        print('')
+
+        for f in sorted(FUNCTIONS.keys()):
+            self.function_reference(f)
+
+        print('=================')
+        print('SPECIAL VARIABLES')
+        print('=================')
+        print('')
+
+        for v in sorted(SPECIAL_VARIABLES.keys()):
+            self.special_reference(v)
+
+        return 0
+
+    def variable_reference(self, v):
+        typ, default, doc = VARIABLES[v]
+
+        print(v)
+        print('=' * len(v))
+        print('')
+
+        summary, extra = get_doc(doc)
+
+        print(summary)
+        print('')
+        print('Type: %s' % typ.__name__)
+        print('Default Value: %s' % default)
+        print('')
+        print_extra(extra)
+
+    def function_reference(self, f):
+        attr, args, doc = FUNCTIONS[f]
+
+        print(f)
+        print('=' * len(f))
+        print('')
+
+        summary, extra = get_doc(doc)
+
+        print(summary)
+        print('')
+
+        arg_types = []
+
+        for t in args:
+            if isinstance(t, list):
+                inner_types = [t2.__name__ for t2 in t]
+                arg_types.append(' | ' .join(inner_types))
+                continue
+
+            arg_types.append(t.__name__)
+
+        arg_s = '(%s)' % ', '.join(arg_types)
+
+        print('Arguments: %s' % arg_s)
+        print('')
+        print_extra(extra)
+
+    def special_reference(self, v):
+        typ, doc = SPECIAL_VARIABLES[v]
+
+        print(v)
+        print('=' * len(v))
+        print('')
+
+        summary, extra = get_doc(doc)
+
+        print(summary)
+        print('')
+        print('Type: %s' % typ.__name__)
+        print('')
+        print_extra(extra)