Skip to content

Checking CircleCI tests run once and only once #2749

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
algojohnlee merged 54 commits into from
Sep 8, 2021
Merged

Checking CircleCI tests run once and only once #2749

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
9442a8c
Adding gotestsum to integration tests
algojack Aug 16, 2021
68e791c
Removed a testing block
algojack Aug 16, 2021
f20b6d0
Adding test results dir
algojack Aug 17, 2021
ec23752
Undoing shellcheck improvements
algojack Aug 17, 2021
58a3b56
Undoing shellcheck improvements
algojack Aug 17, 2021
12bb942
Added junit file for test results
algojack Aug 17, 2021
64907c0
Merge branch 'master' into jack/1489-checking-circleci-tests-run-only…
algojack Aug 18, 2021
3e167d4
Added new check job and result_path var
algojack Aug 18, 2021
d35acb0
Fixed typo
algojack Aug 18, 2021
6c6a3b2
Fixed parameter reference
algojack Aug 18, 2021
e4aaf76
Added script to check for tests
algojack Aug 18, 2021
cff740f
Added partition id folders and recursive file list
algojack Aug 19, 2021
eb1e87e
Creating node index folders
algojack Aug 19, 2021
47eee52
Fixing workspace paths
algojack Aug 19, 2021
744eca3
Fixing workspace paths
algojack Aug 19, 2021
b73ee2b
Combining result files into one
algojack Aug 19, 2021
784072c
Making it run for all needed jobs
algojack Aug 19, 2021
25cd3dd
Fixing naming
algojack Aug 19, 2021
2dbac42
Merge remote-tracking branch 'origin/master' into jack/1489-checking-…
algojack Aug 20, 2021
69e438d
Separated skipped due to partition, and made docker small
algojack Aug 20, 2021
61986f5
Restricted to amd64_test and nightly for testing
algojack Aug 20, 2021
aed014b
Restricted to mac_amd64_test and nightly for testing
algojack Aug 20, 2021
965c61d
Added color and switched to nightly
algojack Aug 20, 2021
7884fe7
Enabled all nightly tests to check that they all work
algojack Aug 23, 2021
464a39a
Merge remote-tracking branch 'origin/master' into jack/1489-checking-…
algojack Aug 23, 2021
2855341
Switched back to normal (not nightlies)
algojack Aug 23, 2021
b6d3b4d
Added comment about gotestsum usage
algojack Aug 23, 2021
11d153a
Add double sharding to the expect tests
cce Aug 25, 2021
f4e5e60
Merge branch 'master' into jack/1489-checking-circleci-tests-run-only…
algojack Aug 30, 2021
7f93ef4
Replaced lists with one dictionary and better messages
algojack Aug 30, 2021
08fd3b5
Fixed typos and formatting
algojack Aug 31, 2021
a749734
Added verification to e2e expect jobs
algojack Aug 31, 2021
08cfb40
Fixed exit error code
algojack Aug 31, 2021
a7ef7c7
Switching to nightly tests to check them too
algojack Aug 31, 2021
fa5f92c
Revert "Switching to nightly tests to check them too"
algojack Aug 31, 2021
fff3913
increase parallelism: 2 and reduce resource_class used by expect jobs
cce Sep 1, 2021
c4b6da3
Fixing how we detect intentional vs unintentional skips
algojack Sep 1, 2021
aa5576d
Fixed message when no reason found
algojack Sep 1, 2021
30c9613
Fixed formatting of skipped reasons
algojack Sep 2, 2021
5772416
Fixing wording of some things
algojack Sep 3, 2021
bb2fa57
Revert "Add double sharding to the expect tests"
cce Sep 7, 2021
4929222
Fixing double partitioned tests
algojack Sep 7, 2021
67abb84
Add PartitionTest call to some tests that were not partitioned spotte…
cce Sep 7, 2021
04ed1bc
Switching to nightly tests for testing only
algojack Sep 7, 2021
732eb5b
Merge branch 'jack/1489-checking-circleci-tests-run-only-once' of git…
algojack Sep 7, 2021
87e0cba
Fixing expect tests to have partition
algojack Sep 7, 2021
017a5ae
Paritioning unpartitioned multiple run tests
algojack Sep 7, 2021
8c09cb5
Revert "Switching to nightly tests for testing only"
algojack Sep 7, 2021
dcbca18
Fixed automatic VSCode corrections
algojack Sep 7, 2021
7a0c2b5
Fixing vscode bug
algojack Sep 7, 2021
4ae2738
Fixing double partitioned tests
algojack Sep 8, 2021
efd73b6
Switching to nightly tests for testing only
algojack Sep 7, 2021
d083c52
Fixing double partitioning
algojack Sep 8, 2021
54a03e5
Revert "Switching to nightly tests for testing only"
algojack Sep 8, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
137 changes: 99 additions & 38 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -217,6 +217,14 @@ workflows:
- /rel\/.*/
- /hotfix\/.*/
context: slack-secrets
- tests_verification_job:
name: << matrix.job_type >>_<< matrix.job_version >>_verification
matrix:
parameters:
job_type: ["amd64", "arm64", "mac_amd64"]
job_version: ["test", "test_nightly", "integration", "integration_nightly", "e2e_expect", "e2e_expect_nightly"]
requires:
- << matrix.job_type >>_<< matrix.job_version >>
#- windows_x64_build

commands:
Expand Down Expand Up @@ -327,10 +335,13 @@ commands:
short_test_flag:
type: string
default: ""
result_path:
type: string
default: "/tmp/results"
steps:
- attach_workspace:
at: << parameters.circleci_home >>
- run: mkdir -p /tmp/results/<< parameters.result_subdir >>
- run: mkdir -p << parameters.result_path >>/<< parameters.result_subdir >>/$CIRCLE_NODE_INDEX
- restore_cache:
keys:
- 'go-cache-{{ .Environment.CIRCLE_STAGE }}-'
Expand All @@ -354,12 +365,16 @@ commands:
export PARTITION_TOTAL=$CIRCLE_NODE_TOTAL
export PARTITION_ID=$CIRCLE_NODE_INDEX
export PARALLEL_FLAG="-p 1"
gotestsum --format pkgname --junitfile /tmp/results/<< parameters.result_subdir >>/results.xml --jsonfile /tmp/results/<< parameters.result_subdir >>/testresults.json -- --tags "sqlite_unlock_notify sqlite_omit_load_extension" << parameters.short_test_flag >> -race -timeout 1h -coverprofile=coverage.txt -covermode=atomic -p 1 $PACKAGE_NAMES
gotestsum --format testname --junitfile << parameters.result_path >>/<< parameters.result_subdir >>/${CIRCLE_NODE_INDEX}/results.xml --jsonfile << parameters.result_path >>/<< parameters.result_subdir >>/${CIRCLE_NODE_INDEX}/testresults.json -- --tags "sqlite_unlock_notify sqlite_omit_load_extension" << parameters.short_test_flag >> -race -timeout 1h -coverprofile=coverage.txt -covermode=atomic -p 1 $PACKAGE_NAMES
- store_artifacts:
path: /tmp/results
path: << parameters.result_path >>
destination: test-results
- store_test_results:
path: /tmp/results
path: << parameters.result_path >>
- persist_to_workspace:
root: << parameters.result_path >>
paths:
- << parameters.result_subdir >>
- save_cache:
key: 'go-cache-{{ .Environment.CIRCLE_STAGE }}-{{ .Environment.CIRCLE_BUILD_NUM }}'
paths:
Expand Down Expand Up @@ -388,10 +403,13 @@ commands:
short_test_flag:
type: string
default: ""
result_path:
type: string
default: "/tmp/results"
steps:
- attach_workspace:
at: << parameters.circleci_home >>
- run: mkdir -p /tmp/results/<< parameters.result_subdir >>
- run: mkdir -p << parameters.result_path >>/<< parameters.result_subdir >>/$CIRCLE_NODE_INDEX
- run:
name: Run integration tests
no_output_timeout: << parameters.no_output_timeout >>
Expand All @@ -409,15 +427,39 @@ commands:
scripts/buildtools/install_buildtools.sh -o "gotest.tools/gotestsum"
export ALGOTEST=1
export SHORTTEST=<< parameters.short_test_flag >>
export TEST_RESULTS=/tmp/results/<< parameters.result_subdir >>
export TEST_RESULTS=<< parameters.result_path >>/<< parameters.result_subdir >>/$CIRCLE_NODE_INDEX
export PARTITION_TOTAL=$CIRCLE_NODE_TOTAL
export PARTITION_ID=$CIRCLE_NODE_INDEX
test/scripts/run_integration_tests.sh
- store_artifacts:
path: /tmp/results
path: << parameters.result_path >>
destination: test-results
- store_test_results:
path: /tmp/results
path: << parameters.result_path >>
- persist_to_workspace:
root: << parameters.result_path >>
paths:
- << parameters.result_subdir >>

tests_verification_command:
description: Check if all tests were run at least once and only once across all parallel runs
parameters:
result_path:
type: string
default: "/tmp/results"
result_subdir:
type: string
steps:
- attach_workspace:
at: << parameters.result_path >>
- run:
name: Check if all tests were run
command: |
cat << parameters.result_path >>/<< parameters.result_subdir >>/**/testresults.json > << parameters.result_path >>/<< parameters.result_subdir >>/combined_testresults.json
python3 scripts/buildtools/check_tests.py << parameters.result_path >>/<< parameters.result_subdir >>/combined_testresults.json
- store_artifacts:
path: << parameters.result_path >>/<< parameters.result_subdir >>
destination: << parameters.result_subdir >>/combined-test-results

jobs:
codegen_verification:
Expand Down Expand Up @@ -449,7 +491,7 @@ jobs:
steps:
- prepare_go
- generic_buildtest:
result_subdir: amd64-short
result_subdir: amd64_test
short_test_flag: "-short"
- upload_coverage

Expand All @@ -461,7 +503,7 @@ jobs:
steps:
- prepare_go
- generic_buildtest:
result_subdir: amd64-nightly
result_subdir: amd64_test_nightly
no_output_timeout: 45m
- upload_coverage
- slack/notify:
Expand All @@ -478,7 +520,7 @@ jobs:
steps:
- prepare_go
- generic_integration:
result_subdir: amd64-integration
result_subdir: amd64_integration
short_test_flag: "-short"

amd64_integration_nightly:
Expand All @@ -491,7 +533,7 @@ jobs:
steps:
- prepare_go
- generic_integration:
result_subdir: amd64-integrationnightly
result_subdir: amd64_integration_nightly
no_output_timeout: 45m
- slack/notify:
event: fail
Expand All @@ -506,7 +548,7 @@ jobs:
steps:
- prepare_go
- generic_integration:
result_subdir: amd64-e2e_subs
result_subdir: amd64_e2e_subs
short_test_flag: "-short"

amd64_e2e_subs_nightly:
Expand All @@ -518,7 +560,7 @@ jobs:
steps:
- prepare_go
- generic_integration:
result_subdir: amd64-e2e_subs_nightly
result_subdir: amd64_e2e_subs_nightly
no_output_timeout: 45m
- slack/notify:
event: fail
Expand All @@ -527,25 +569,27 @@ jobs:
amd64_e2e_expect:
machine:
image: ubuntu-2004:202104-01
resource_class: large
resource_class: medium
parallelism: 2
environment:
E2E_TEST_FILTER: "EXPECT"
steps:
- prepare_go
- generic_integration:
result_subdir: amd64-e2e_subs
result_subdir: amd64_e2e_expect
short_test_flag: "-short"

amd64_e2e_expect_nightly:
machine:
image: ubuntu-2004:202104-01
resource_class: large
resource_class: medium
parallelism: 2
environment:
E2E_TEST_FILTER: "EXPECT"
steps:
- prepare_go
- generic_integration:
result_subdir: amd64-e2e_subs_nightly
result_subdir: amd64_e2e_expect_nightly
no_output_timeout: 45m
- slack/notify:
event: fail
Expand All @@ -569,7 +613,7 @@ jobs:
- checkout
- prepare_go
- generic_buildtest:
result_subdir: arm64-short
result_subdir: arm64_test
short_test_flag: "-short"
- upload_coverage

Expand All @@ -582,7 +626,7 @@ jobs:
- checkout
- prepare_go
- generic_buildtest:
result_subdir: arm64-nightly
result_subdir: arm64_test_nightly
no_output_timeout: 45m
- upload_coverage
- slack/notify:
Expand All @@ -600,7 +644,7 @@ jobs:
- checkout
- prepare_go
- generic_integration:
result_subdir: arm64-integration
result_subdir: arm64_integration
short_test_flag: "-short"

arm64_integration_nightly:
Expand All @@ -614,7 +658,7 @@ jobs:
- checkout
- prepare_go
- generic_integration:
result_subdir: arm64-integration-nightly
result_subdir: arm64_integration_nightly
no_output_timeout: 45m
- slack/notify:
event: fail
Expand All @@ -630,7 +674,7 @@ jobs:
- checkout
- prepare_go
- generic_integration:
result_subdir: arm64-e2e_subs
result_subdir: arm64_e2e_subs
short_test_flag: "-short"

arm64_e2e_subs_nightly:
Expand All @@ -643,7 +687,7 @@ jobs:
- checkout
- prepare_go
- generic_integration:
result_subdir: arm64-e2e_subs-nightly
result_subdir: arm64_e2e_subs_nightly
no_output_timeout: 45m
- slack/notify:
event: fail
Expand All @@ -652,27 +696,29 @@ jobs:
arm64_e2e_expect:
machine:
image: ubuntu-2004:202101-01
resource_class: arm.large
resource_class: arm.medium
parallelism: 2
environment:
E2E_TEST_FILTER: "EXPECT"
steps:
- checkout
- prepare_go
- generic_integration:
result_subdir: arm64-e2e_subs
result_subdir: arm64_e2e_expect
short_test_flag: "-short"

arm64_e2e_expect_nightly:
machine:
image: ubuntu-2004:202101-01
resource_class: arm.large
resource_class: arm.medium
parallelism: 2
environment:
E2E_TEST_FILTER: "EXPECT"
steps:
- checkout
- prepare_go
- generic_integration:
result_subdir: arm64-e2e_subs-nightly
result_subdir: arm64_e2e_expect_nightly
no_output_timeout: 45m
- slack/notify:
event: fail
Expand Down Expand Up @@ -702,7 +748,7 @@ jobs:
#- run: git -C /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core fetch --unshallow
- prepare_go
- generic_buildtest:
result_subdir: mac-amd64-short
result_subdir: mac_amd64_test
circleci_home: /Users/distiller
short_test_flag: "-short"
- upload_coverage
Expand All @@ -718,7 +764,7 @@ jobs:
#- run: git -C /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core fetch --unshallow
- prepare_go
- generic_buildtest:
result_subdir: mac-amd64-short
result_subdir: mac_amd64_test_nightly
circleci_home: /Users/distiller
no_output_timeout: 45m
- upload_coverage
Expand All @@ -738,7 +784,7 @@ jobs:
#- run: git -C /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core fetch --unshallow
- prepare_go
- generic_integration:
result_subdir: mac-amd64-integration
result_subdir: mac_amd64_integration
circleci_home: /Users/distiller
short_test_flag: "-short"

Expand All @@ -754,7 +800,7 @@ jobs:
#- run: git -C /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core fetch --unshallow
- prepare_go
- generic_integration:
result_subdir: mac-amd64-integration-nightly
result_subdir: mac_amd64_integration_nightly
circleci_home: /Users/distiller
no_output_timeout: 45m
- slack/notify:
Expand All @@ -772,7 +818,7 @@ jobs:
#- run: git -C /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core fetch --unshallow
- prepare_go
- generic_integration:
result_subdir: mac-amd64-e2e_subs
result_subdir: mac_amd64_e2e_subs
circleci_home: /Users/distiller
short_test_flag: "-short"

Expand All @@ -787,7 +833,7 @@ jobs:
#- run: git -C /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core fetch --unshallow
- prepare_go
- generic_integration:
result_subdir: mac-amd64-e2e_subs-nightly
result_subdir: mac_amd64_e2e_subs_nightly
circleci_home: /Users/distiller
no_output_timeout: 45m
- slack/notify:
Expand All @@ -797,30 +843,32 @@ jobs:
mac_amd64_e2e_expect:
macos:
xcode: 12.0.1
resource_class: large
resource_class: medium
parallelism: 2
environment:
E2E_TEST_FILTER: "EXPECT"
HOMEBREW_NO_AUTO_UPDATE: "true"
steps:
#- run: git -C /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core fetch --unshallow
- prepare_go
- generic_integration:
result_subdir: mac-amd64-e2e_subs
result_subdir: mac_amd64_e2e_expect
circleci_home: /Users/distiller
short_test_flag: "-short"

mac_amd64_e2e_expect_nightly:
macos:
xcode: 12.0.1
resource_class: large
resource_class: medium
parallelism: 2
environment:
E2E_TEST_FILTER: "EXPECT"
HOMEBREW_NO_AUTO_UPDATE: "true"
steps:
#- run: git -C /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core fetch --unshallow
- prepare_go
- generic_integration:
result_subdir: mac-amd64-e2e_subs-nightly
result_subdir: mac_amd64_e2e_expect_nightly
circleci_home: /Users/distiller
no_output_timeout: 45m
- slack/notify:
Expand All @@ -845,3 +893,16 @@ jobs:
export MAKE=mingw32-make
$msys2 scripts/travis/build_test.sh
shell: bash.exe
tests_verification_job:
docker:
- image: python:3.9.6-alpine
resource_class: small
parameters:
job_type: # job_type: ["amd64", "arm64", "mac_amd64"]
type: string
job_version: # job_version: ["test", "test_nightly", "integration", "integration_nightly", "e2e_expect", "e2e_expect_nightly"]
type: string
steps:
- checkout
- tests_verification_command:
result_subdir: << parameters.job_type >>_<< parameters.job_version >>
Loading