implementation to run kube acceptance tests on GKE #4961
Conversation
# Conflicts: # airbyte-tests/src/acceptanceTests/java/io/airbyte/test/acceptance/AcceptanceTests.java
subodh1810
changed the title
There was a problem hiding this comment.
Looks good! Some minor comments.
This is currently configured to run on every build now. One thing to call out is our tiny GKE cluster probably won't support more than 3 tests running in parallel. I think this is fine if we are going to merge in on demand testing in the next few days. If not we should probably either make this scheduled instead of every build or merge in the on-demand command with this PR.
.github/workflows/gradle.yml
Outdated
| sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl | ||
|
|
||
| # Set up kustomize | ||
| - name: Set up Kustomize |
There was a problem hiding this comment.
why do we need to install Kustomize separately? shouldn't this come with Kubectl?
| } | ||
|
|
||
| @AfterAll | ||
| public static void end() { | ||
| sourcePsql.stop(); | ||
| if (IS_GKE && !IS_KUBE) { | ||
| throw new RuntimeException("KUBE Flag should also be enabled if GKE flag is enabled"); |
There was a problem hiding this comment.
should this check be in the @BeforeAll method?
|
|
||
| kubectl port-forward svc/postgres-destination-svc 3000:5432 --namespace=$NAMESPACE & | ||
|
|
||
| sleep 10s |
There was a problem hiding this comment.
This is cause I observed that even after starting the port-forwarding, it was taking a couple of seconds to re-route the traffic so I put 10 seconds just to be safe
airbyte-tests/src/acceptanceTests/java/io/airbyte/test/acceptance/GKEPostgresConfig.java
Show resolved
Hide resolved
airbyte-tests/src/acceptanceTests/java/io/airbyte/test/acceptance/AcceptanceTests.java
Show resolved
Hide resolved
| } | ||
| return Jsons.jsonNode(dbConfig); | ||
| } catch (Exception e) { | ||
| throw new RuntimeException(e); |
There was a problem hiding this comment.
curious, why was this removed?
There was a problem hiding this comment.
@davinchia Oh no this was not removed, I just refactored methods and broke this into two. The new method localConfig returns a map and the JSON conversion is done in getDbConfig
.github/workflows/gradle.yml
Outdated
| @@ -487,3 +487,116 @@ jobs: | |||
| github-token: ${{ secrets.SELF_RUNNER_GITHUB_ACCESS_TOKEN }} | |||
| label: ${{ needs.start-kube-acceptance-test-runner.outputs.label }} | |||
| ec2-instance-id: ${{ needs.start-kube-acceptance-test-runner.outputs.ec2-instance-id }} | |||
|
|
|||
| start-gke-kube-acceptance-test-runner: | |||
There was a problem hiding this comment.
something that will go away once we move to on demand testing - running this on every build might overload the tiny GKE cluster we have now.
There was a problem hiding this comment.
Oh am going to add the master and cron condition in here. I have not added it cause I wanted to run this on the CI and see how it goes
|
@davinchia I wanted to run the test on the CI thats why I have not made this scheduled yet. I plan to make this scheduled and only run on master branch
|
# Conflicts: # airbyte-tests/src/acceptanceTests/java/io/airbyte/test/acceptance/AcceptanceTests.java
Issue : #4884
Tech spec: https://docs.google.com/document/d/170JgBmRJwkuGWNZxy41bkyd6z4hAkhuKbNhZtxFtaJo/edit#
Pre-merge Checklist
Expand the checklist which is relevant for this PR.
Connector checklist
airbyte_secretin the connector's spec./gradlew :airbyte-integrations:connectors:<name>:integrationTest./test connector=connectors/<name>command as documented here is passing.README.mddocs/SUMMARY.mdif it's a new connectordocs/integrations/<source or destination>/<name>.docs/integrations/.... See changelog exampledocs/integrations/README.mdcontains a reference to the new connector/publishcommand described hereConnector Generator checklist
-scaffoldin their name) have been updated with the latest scaffold by running./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplatesthen checking in your changes