Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parallelize Circle CI jobs #448

Closed
maxitg opened this issue Oct 11, 2020 · 0 comments · Fixed by #578
Closed

Parallelize Circle CI jobs #448

maxitg opened this issue Oct 11, 2020 · 0 comments · Fixed by #578
Assignees
@aokellermann
Labels
infrastructure Has to do with changes to the development process, e.g., build scripts, CI, testing utilities optimization Does not change functionality but makes code faster

Comments

@maxitg
Copy link
Owner

maxitg commented Oct 11, 2020

The problem

We currently have the ability to use up to 4 parallel runs per CI job in Circle CI. However, we are not using that at the moment.

Possible solution

We can split WL and C++ tests into groups that we can run in parallel. Take a look at Circle CI docs.
@maxitg maxitg added optimization Does not change functionality but makes code faster infrastructure Has to do with changes to the development process, e.g., build scripts, CI, testing utilities labels Oct 11, 2020
@maxitg maxitg changed the title Parallelize Circle CI Parallelize Circle CI jobs Oct 11, 2020
@aokellermann aokellermann self-assigned this Dec 17, 2020
@aokellermann aokellermann mentioned this issue Dec 17, 2020
Merged
aokellermann added a commit that referenced this issue Dec 18, 2020
@aokellermann
Parallel WL Tests on CI (#578)
2a95d9e 
## Changes
* Adds 4x parallelism to WL tests on CI
* Closes #448 

## Comments
Currently, we do not have WL [test metadata](https://circleci.com/docs/2.0/collect-test-data/) compatible with CircleCI. Therefore, we can't use CircleCI's built-in [split-by timing](https://circleci.com/docs/2.0/parallelism-faster-jobs/#splitting-by-timing-data) to distribute tests efficiently. This PR simply runs `matching` and `WolframModel` (which take the most time by far) each in separate executors, and splits the rest among the last two. 

In order to optimize this further, we could:
1. Distribute tests from `matching` and `WolframModel` into separate files.
2. Implement JUnit XML or Cucumber JSON test result reports for WL and enable `split-by timing`.

## Examples
Parallel executors:
![image](https://user-images.githubusercontent.com/26678747/102551793-1b26e680-408e-11eb-8301-79c441f71da0.png)

Test time is reduced by ~10 minutes (over ~2x reduction):
![image](https://user-images.githubusercontent.com/26678747/102552754-b7052200-408f-11eb-97d1-7cba3b151416.png)

compared to master:
![image](https://user-images.githubusercontent.com/26678747/102552708-9fc63480-408f-11eb-87ef-592c572eb111.png)


<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/maxitg/setreplace/578)
<!-- Reviewable:end -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aokellermann aokellermann

Labels
infrastructure Has to do with changes to the development process, e.g., build scripts, CI, testing utilities optimization Does not change functionality but makes code faster
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Parallel WL Tests on CI maxitg/SetReplace
2 participants
@maxitg @aokellermann