[build-script] Add support for using isolated -impl actions.

[ddunbar]

What's in this pull request?

This adopts the feature in #2880 in build-script, and is part of SR-237.

---

Before merging this pull request to apple/swift repository:

• Test pull request on Swift continuous integration.

Triggering Swift CI

The swift-ci is triggered by writing a comment on this PR addressed to the GitHub user @swift-ci. Different tests will run depending on the specific comment that you use. The currently available comments are:

Smoke Testing

Platform	Comment
All supported platforms	@swift-ci Please smoke test
All supported platforms	@swift-ci Please smoke test and merge
OS X platform	@swift-ci Please smoke test OS X platform
Linux platform	@swift-ci Please smoke test Linux platform

Validation Testing

Platform	Comment
All supported platforms	@swift-ci Please test
All supported platforms	@swift-ci Please test and merge
OS X platform	@swift-ci Please test OS X platform
OS X platform	@swift-ci Please benchmark
Linux platform	@swift-ci Please test Linux platform

Lint Testing

Language	Comment
Python	@swift-ci Please Python lint

Note: Only members of the Apple organization can trigger swift-ci.

• This change moves the top-level invocation driver loop into build-script
  and uses the -impl script to perform each individual action. Once landed
  and enabled, this will enable us to migrate the individual pieces of the
  -impl script into Python code in an incremental fashion.
• This also introduces stub product definitions for each of the different projects
  we manage.
• This works, but is disabled by default (--no-legacy-impl) because it
  severely impacts the performance of null builds (4x slower, currently) due to
  the build-script-impl parsing overhead. If only we had a JITing bash
  implementation...

[ddunbar]

@swift-ci please python lint

[ddunbar]

@swift-ci please smoke test

[ddunbar]

/cc @gribozavr @rintaro @karwa

[ddunbar]

Note that this mode drops echoing subcommands by default, it is too verbose until we finish killing the -impl script.

[ddunbar]

@swift-ci please smoke test

[rintaro]

Continued from: #2988 (comment)

@ddunbar

I want all the products to have the same initializer signature so that the build-script can work with the classes generically.

According to my own experience, difficult part for that is "dependency": Swift depends on CMark and LLVM, Foundation depends on Swift, and so on.

FYI, Here's my outdated attempt from #2190:
https://github.com/rintaro/swift/blob/SR-237-build-script-rewrite/utils-experimental/build-script-impl/build_script/main_driver.py#L321-L483

From an object-oriented perspective, I think, only LLVM object should knows the path of its deliverables.
That means, Swift object must somehow reference LLVM object for the host, or receive deliverable paths from the main control flow.
Maybe, we can pass structured object that holds all product object, but that leads the same problem of args: it easy to lose track of which products depends on which.
Or should we invent some dependency manager?

@karwa As for args problem

it makes it easy to lose track of which options are required and for what.

I agree with your concern. But, for example, Swift requires over 30 properties of args.
Nobody wants to do like this:

swift_obj = products.Swift(
    target=target,

    # args
    native_llvm_tools_path=args.native_llvm_tools_path,
    native_clang_tools_path=args.native_clang_tools_path,
    native_swift_tools_path=args.native_tools_swift_path,
    build_type=args.swift_build_type,
    enable_assertions=args.swift_enable_assertions,
    ...
    ... )

Do you have ideas?

[ddunbar]

One possible way to manage that, and I'm not familiar enough with argparse to know how easy this is, but if we could get the highly product dependent arguments to simply be parsed into a more structured product-specific object (SwiftProductArgs, e.g.) that would at least be an improvement.

Is it possible to get an argparse argument dest to map to a subobject?

[rintaro]

Is it possible to get an argparse argument dest to map to a subobject?

Like this?

parser = argparse.ArgumentParser()
parser.add_argument('--swift-opt', dest="swift.opt")

args = parser.parse_args(['--swift-opt', 'foobar'])
assert args.swift.opt == 'foobar'

It's possible if we use custom Namespace object.
Roughly implemented demo:
https://gist.github.com/rintaro/9f99f4b1627aa9daf91e643f7e5e6a7c

[karwa]

I don't think it's such a big problem. It's only the Swift project that really takes a lot of parameters. Many of them can be given reasonable defaults for the purpose of keeping the initialiser manageable. The build script would then parse the args and set the appropriate properties.

Many of those arguments could also be folded - e.g. the native_{project}_tools_path arguments could be derived from a native HostSpecificConfiguration argument. Actually, in this case the native toolchain is only required for cross-compile builds, so we can leave it out of the initialiser and default the properties to None. In general, any build settings Swift gets from, e.g, LLVM, should come from an already-parsed LLVM object. Thats also easier to reason about and makes semantic sense.

[ddunbar]

@swift-ci please test

[ddunbar]

@swift-ci please smoke test

[ddunbar]

I am going to go ahead and land this change, in the interest of continuing the SR-237 conversion. I'll continue to iterate on the feedback to simplify and cleanup the build-script, but I want to split that into separate PRs.

[ddunbar]

@rintaro

From an object-oriented perspective, I think, only LLVM object should knows the path of its deliverables.
That means, Swift object must somehow reference LLVM object for the host, or receive deliverable paths from the main control flow.
Maybe, we can pass structured object that holds all product object, but that leads the same problem of args: it easy to lose track of which products depends on which

I think the relationships between the products are simple enough that this is a good pragmatic approach, we have one structured object to hold all the product objects, and pass that to each product so that it can get connection information from that.

[karwa]

I would say the opposite: our dependencies are so simple they can be passed manually.

Especially if we split the build step in to tools/target libraries steps as was proposed for the 'toolchain-based build' thingy.

[karwa]

By manually, I mean initialising the Swift product object with a configured LLVM object.

(Sorry, GitHub doesn't have an edit button on mobile)

[ddunbar]

I agree, I think the tradeoff just depends on how generic we can make the build-script in terms of products. I'm hoping that we can get rid of a lot of the per-product duplication, and if have a single config object helps that I think it would be a win.