Enable our own CI

[rhuanjl]

Microsoft have transferred the repository into our control. We now need to activate our own CI.

In the first instance this will be largely based on the check in CI microsoft was running.

(though we need our own instance of it all as the existing webhooks no longer function)

[ppenzin]

Oh, I managed to miss the moment of transition, will do as soon as I get to my laptop.

[ppenzin]

I started, but ended up with two commits to master before opening a PR to batch changes up.

Basic build:

• Windows
  • Debug
  • Test
  • Release
• Linux
  • Use libicu-dev
    • Debug
    • RelWithDebInfo (aka Test)
    • Release
• OSX
  • With icu4c
    • Debug - static library mode
    • RelWithDebInfo (aka Test)
    • Release
    • Debug with Jit disabled - static library mode

Important miscellanea:

• Style
  • ASCII check
  • EOL check
• Copyright check
• Remove obsolete checks from PR pages

Extra credit:

• Cache build artifacts

@rhuanjl, feel free to add anything.

Edit: I might have enabled Linux Debug CI in #6554, but running tests takes a while, will come back and check on it early tomorrow.

[rhuanjl]

A question is whether we want to target different versions of windows: (7 8 and 10 were what the old CI did) also processors.

Currently Linux and macOS builds only work with x64 BUT windows can be done with x86 and ARM as well - we probably want to keep testing those?

Also could/should things like style checks be done with github actions or the like instead of using Azure for everything? (I don't know what limits/caps our azure provision may have?)

[rhuanjl]

On the cleanup bit the triggers that are still firing appear to be in the github settings for this repository.

Also there's the now defunct jenkins folder in the source tree - though some of the information in those scripts may be useful.

[ppenzin]

On the cleanup bit the triggers that are still firing appear to be in the github settings for this repository.

I think I figured out how disable old checks. There are also numerous webhooks configured - we might want to check if they are still needed.

Also there's the now defunct jenkins folder in the source tree - though some of the information in those scripts may be useful.

Yep, we need to review its contents and reorganize if we want to keep them.

A question is whether we want to target different versions of windows: (7 8 and 10 were what the old CI did) also processors.

Let's see what Azure has in store, we can try older versions if they are available.

Currently Linux and macOS builds only work with x64 BUT windows can be done with x86 and ARM as well - we probably want to keep testing those?

Agree. Did we have an explicit Arm job? It would be good to try anyway.

Also could/should things like style checks be done with github actions or the like instead of using Azure for everything? (I don't know what limits/caps our azure provision may have?)

I think they can be, Azure Pipelines set up is getting more complicated, and keeping simple checks out of it would help. However, we are not yet at the limit - they allow 10 concurrent jobs, with run time up to 6 hours.

[rhuanjl]

I just looked through the old CI, here's the set of builds that were done for every checkin, each heading showed as one check on github but the bullets under it were separate builds that had to be completed before it reported as done, in addition to these there were the style checks and the CLA agreement check.

Windows 7

• x64 Debug; build + run tests

Windows 8

• x64 Debug; build + run tests
• arm Debug; build + run tests
• arm Test (RelWithDebugINfo); build + run tests
• arm Release; build only

Windows 10

• x64 Debug; build + run tests
• x64 Test (RelWithDebugInfo); build + run tests
• x64 Release; build only
• x64 codecoverage - build with "codecoverage" flag - unsure what this does, didn't seem to involve any tests
• x86 Debug; build + run tests
• x86 Test (RelWithDebugInfo); build + run tests
• x86 Release; build only
• x64 Debug Lite variant; build + run tests ("Lite" is a build time flag that omits the JIT and a few over components to make CC smaller)
• x64 Debug Disable Jit variant; build + run tests ("Disable Jit" is a build time flag to omit the Jit from CC but not anything else, this is slightly bigger than a "Lite" build)
• x86 Debug Lite variant, build + run tests
• x86 Debug Disable Jit variant, build + run tests

macOS

• static Debug: build + run tests (statically linked library)
• static Test (RelWithDebugInfo): build + run tests
• static Releast; build only
• shared Disable Jit variant; build + run tests (build a shared library but with the disable jit flag)

Linux

• static Debug; build + run tests
• static test (RelWithDebugInfo); build + run tests
• static Release; build only
• shared Debug; build + run tests
• shared test (RelWithDebugInfo); build + run tests
• shared Release; build only
• static no icu Debug; build + run tests (no icu flag to build without ICU)
• shared disable jit Test (RelWithDebugInfo); build + run tests

[ppenzin]

OK, I am going to try to follow what the old CI did where possible.

Code coverage should be an instrumented test run, which would show how much of the code base is 'hit' by tests. May be that was disabled at some point. We had a request for that recently.

@rhuanjl do you know what 'disable JIT' builds are for? There are also jobs marked "slow" (despite finishing quite fast) in the old CI, do you know what are those?

Also, to speed up the builds it would be great to cache build files and lunch multiple test jobs at once, since testing seems to be the slowest part of the build.

[rhuanjl]

@rhuanjl do you know what 'disable JIT' builds are for? There are also jobs marked "slow" (despite finishing quite fast) in the old CI, do you know what are those?

"Disable JIT" is a build time flag to remove the JIT from CC and build it to run with the interpreter only - it works and supports all features though performance will drop, it's a smaller binary and needs less memory though. There's also the "Lite build" flag which does disable JIT and a few other features are removed too for an even smaller binary.

"Slow" tests are a few specific tests in the test suite that are expected to take a long time (more than a minute each perhaps), these are omitted from normal test runs, I didn't think the old CI ran these at all - I thought there were builds for them but they were auto skipped hence the rapid finish - they didn't actually do anything.