[RFC][build-script] SR-237: Merge build-script-impl into build-script

[rintaro]

Resolved bug number: (SR-237)

Complete rewrite in Python.
Some - bad English - design memo are available here: https://gist.github.com/rintaro/ff6b5a063850a0efe535ede03585b592

Not well tested for now, so please try it and tell me if you found any problems.
Currently, for easy to compare with original utils/build-script,
all codes are in utils-experimental/ directory.
With new --dry-run, -n mode, you can safely preview what commands to be executed.

utils-experimental/build-script -n -RT --reconfigure
utils-experimental/build-script -n --preset=buildbot_incremental,tools=RA,stdlib=RA
utils-experimental/build-toolchain -n local.swift

No change in command line interface, including --preset mode.
Compatible with current build-presets.ini.

---

Before merging this pull request to apple/swift repository:

• Move to utils/ after some basic design acceptance.
• Test pull request on Swift continuous integration.

[rintaro]

@gribozavr @modocache
Please tell me anything to move forward this PR.
I'm aware 7000+ lines of code are not easy to review...

[modocache]

I've been thinking about this for a few days, but am torn:

• On the one hand, I think a huge pull request like this might be the only feasible way to achieve SR-237. build-script-impl grows every day, and moving things piecemeal is hard. And I think SR-237 is important--for example, were we to ever consider allowing Swift to be built on Windows, the shellscript-based system would need to go. So this work seems like a big step in the right direction.
• On the other hand, it's really hard to review such a large pull request. How can we be sure all the builds still work as they should?

I defer to @gribozavr and other maintainers on whether we should make one huge change, or whether we should migrate little by little. If I absolutely had to take a side, I personally feel moving little by little, although frustrating, is a more reviewable/feasible approach.

[gottesmm]

@modocache We really want a change like this to be incremental.

This is how I would do this:

1. Study the recent changes to build-script-impl in the past 3 months.
2. Provide facilities to migrate those changes to build-script and move them.
3. Whenever someone wants to add something to build-script-impl, we push back and say lets discuss what is needed to move it into build-script.

The reason I believe we need 1,2 is that I think there are fundamental capabilities missing from build-script that is making people want to use build-script-impl (of course there is also the cultural aspect). But I am not 100% sure about my statement, that is why we really need to understand what are the changes that are going into build-script-impl before we continue.

[gribozavr]

@rintaro First of all, I want to thank you for taking on this colossal project.

If I absolutely had to take a side, I personally feel moving little by little, although frustrating, is a more reviewable/feasible approach.

I very much agree with @modocache. It is hard to review such a big change. It is not big just in terms of code lines, but also in terms of build logic. Swift builds in many diverse environments, and no single person has access to all of them. It will be much easier to find issues if we took in smaller changes and checked at each point that we haven't broken any of our usecases.

[gottesmm]

@modocache @rintaro After rereading my post, I realized I want to be a little clearer: Before we can work on the cultural aspects, we need to ensure that build-script has enough "coverage" of build-script-impl's functionality so that when we push back the tech is there and we can "teach" people how to make these changes.

[modocache]

We do provide one way to migrate individual build-script-impl arguments to build-script, but I agree that there are many more reasons people choose build-script-impl over build-script. The biggest reason I've used impl before is for access to variables like LLVM_BUILD_DIR and SWIFT_BUILD_DIR -- if we could move things like build_directory() over that would be a big help.

[rintaro]

@modocache @gribozavr @gottesmm
Thanks for the comments!
I understand the reviewing is too hard, or even impossible, and that is within my expectations :)
Since I'm not good at writing in English, I'm very sorry that I cannot join such advanced discussion.
Anyway, I have some - not yet implemented - idea in my mind to do incremental, little by little, migrations. I'm willing to post "reviewable" PR one by one when ready.

Can I close this PR?

[modocache]

@rintaro Yes, feel free to close this.

As for the end result of the pull request: I love it! I especially appreciate the --dry-run mode. It would be amazing to see the exact CMake invocations that build the project.

[rintaro]

@modocache Thanks!

closing.