Parallel WL Tests on CI

[aokellermann]

Changes

• Adds 4x parallelism to WL tests on CI
• Closes Parallelize Circle CI jobs #448

Comments

Currently, we do not have WL test metadata compatible with CircleCI. Therefore, we can't use CircleCI's built-in split-by timing to distribute tests efficiently. This PR simply runs matching and WolframModel (which take the most time by far) each in separate executors, and splits the rest among the last two.

In order to optimize this further, we could:

1. Distribute tests from matching and WolframModel into separate files.
2. Implement JUnit XML or Cucumber JSON test result reports for WL and enable split-by timing.

Examples

Parallel executors:

Test time is reduced by ~10 minutes (over ~2x reduction):

compared to master:

---

This change is 

[maxitg]

This is incredible! Thanks!

Reviewed 2 of 2 files at r1.
Reviewable status: all files reviewed, 3 unresolved discussions (waiting on @aokellermann)

---

.circleci/config.yml, line 9 at r1 (raw file):

  wolfram-language-paclet-test:
    docker:
      - image: maxitg/set-replace-wl-ci:12.1.1

It seems redundant that we are downloading the image 4 times separately. There is definitely a way around it, but we might have to download WL outside of the Docker image and put it into the workspace (not in this PR, obviously).

---

.circleci/config.yml, line 23 at r1 (raw file):

      - store_artifacts:
          path: ./LibraryResources/

Similarly, we don't need to store artifacts from every parallel run. The way to fix that would be to build in a separate job, however, doing that will cost us ~60 secs for another WL download, so it will need to be done after the previous comment.

---

.circleci/config.yml, line 65 at r1 (raw file):

      - run:
          name: Performance Test
          command: ./performanceTest.wls master @HEAD 2

If we only run this on the zeroth or the first node, we will save another ~30 secs.

[aokellermann]

Reviewable status: 1 of 2 files reviewed, 1 unresolved discussion (waiting on @maxitg)

---

.circleci/config.yml, line 65 at r1 (raw file):

Previously, maxitg (Max Piskunov) wrote…

If we only run this on the zeroth or the first node, we will save another ~30 secs.

Done.

[maxitg]

Reviewed 1 of 1 files at r2.
Reviewable status: complete! all files reviewed, all discussions resolved