run graphql tests on both http and websocket; add parallelism (close #1868) #1921

nizar-m · 2019-04-01T11:25:41Z

Description

Affected components

Tests

Related Issues

#1868

Invocation

Examples

pytest --hge-urls "http://127.0.0.1:8080" --pg-urls "postgresql://admin@127.0.0.1:5432/hge_tests" -vv
pytest --hge-urls "http://127.0.0.1:8080" "http://127.0.0.1:8081" --pg-urls "postgresql://admin@127.0.0.1:5432/hge_tests" "postgresql://admin@127.0.0.1:5432/hge_tests2" -vv

Solution and Design

Reducing execution time of tests

The Schema setup and teardown, which were earlier done per test method, usually takes around 1 sec.
For mutations, the model has now been changed to only do schema setup and teardown once per test class.
A data setup and teardown will be done once per test instead (usually takes ~10ms).
For the test class to get this behaviour, one can can extend the class DefaultTestMutations.
- The function dir() should be define which returns the location of the configuration folder.
- Inside the configuration folder, there should be
  - Files <conf_dir>/schema_setup.yaml and <conf_dir>/schema_teardown.yaml, which has the metadata query executed during schema setup and teardown respectively
  - Files named <conf_dir>/values_setup.yaml and <conf_dir>/values_teardown.yaml. These files are executed to setup and remove data from the tables respectively.

Running Graphql queries on both http and websockets

Each GraphQL query/mutation is run on the both HTTP and websocket protocols
Pytests test parameterisation is used to achieve this
The errors over websockets are slightly different from that on HTTP
- The code takes care of converting the errors in HTTP to errors in websockets

Parallel executation of tests.

The plugin pytest-xdist helps in running tests on parallel workers.
We are using this plugin to group tests by file and run on different workers.
Parallel test worker processes operate on separate postgres databases(and separate graphql-engines connected to these databases). Thus tests on one worker will not affect the tests on the other worker.
With two workers, this decreases execution times by half, as the tests on event triggers usually takes a long time, but does not consume much CPU.

Steps to test and verify

Limitations, known bugs & workarounds

…-faster

netlify · 2019-04-01T11:26:58Z

Deploy preview for hasura-docs ready!

Built with commit f2d8363

https://deploy-preview-1921--hasura-docs.netlify.com

hasura-bot · 2019-04-01T11:48:45Z

Review app for commit 0b24b30 deployed to Heroku: https://hge-ci-pull-1921.herokuapp.com
Docker image for server: hasura/graphql-engine:pull1921-0b24b30

hasura-bot · 2019-04-02T22:11:17Z

Review app for commit 6810ae0 deployed to Heroku: https://hge-ci-pull-1921.herokuapp.com
Docker image for server: hasura/graphql-engine:pull1921-6810ae0

hasura-bot · 2019-04-03T05:14:44Z

Review app for commit 9d093c7 deployed to Heroku: https://hge-ci-pull-1921.herokuapp.com
Docker image for server: hasura/graphql-engine:pull1921-9d093c7

2) readme about structure of pytests 3) Update Contributing.md

ecthiender

Looks good. We just need to just document why we split the schema setup and the data setup into separate steps/scopes, before merging so that other ppl writing tests knows how to do this.

hasura-bot · 2019-04-03T13:26:30Z

Review app for commit 68c5269 deployed to Heroku: https://hge-ci-pull-1921.herokuapp.com
Docker image for server: hasura/graphql-engine:pull1921-68c5269

hasura-bot · 2019-04-08T05:32:56Z

Review app for commit 4a53cd6 deployed to Heroku: https://hge-ci-pull-1921.herokuapp.com
Docker image for server: hasura/graphql-engine:pull1921-4a53cd6

hasura-bot · 2019-04-08T07:19:08Z

Review app for commit f2d8363 deployed to Heroku: https://hge-ci-pull-1921.herokuapp.com
Docker image for server: hasura/graphql-engine:pull1921-f2d8363

shahidhk

LGTM

hasura-bot · 2019-04-08T07:22:41Z

Review app https://hge-ci-pull-1921.herokuapp.com is deleted

…asura#1868) (hasura#1921) Examples 1) ` pytest --hge-urls "http://127.0.0.1:8080" --pg-urls "postgresql://admin@127.0.0.1:5432/hge_tests" -vv ` 2) `pytest --hge-urls "http://127.0.0.1:8080" "http://127.0.0.1:8081" --pg-urls "postgresql://admin@127.0.0.1:5432/hge_tests" "postgresql://admin@127.0.0.1:5432/hge_tests2" -vv ` ### Solution and Design   #### Reducing execution time of tests - The Schema setup and teardown, which were earlier done per test method, usually takes around 1 sec. - For mutations, the model has now been changed to only do schema setup and teardown once per test class. - A data setup and teardown will be done once per test instead (usually takes ~10ms). - For the test class to get this behaviour, one can can extend the class `DefaultTestMutations`. - The function `dir()` should be define which returns the location of the configuration folder. - Inside the configuration folder, there should be - Files `<conf_dir>/schema_setup.yaml` and `<conf_dir>/schema_teardown.yaml`, which has the metadata query executed during schema setup and teardown respectively - Files named `<conf_dir>/values_setup.yaml` and `<conf_dir>/values_teardown.yaml`. These files are executed to setup and remove data from the tables respectively. #### Running Graphql queries on both http and websockets - Each GraphQL query/mutation is run on the both HTTP and websocket protocols - Pytests test parameterisation is used to achieve this - The errors over websockets are slightly different from that on HTTP - The code takes care of converting the errors in HTTP to errors in websockets #### Parallel executation of tests. - The plugin pytest-xdist helps in running tests on parallel workers. - We are using this plugin to group tests by file and run on different workers. - Parallel test worker processes operate on separate postgres databases(and separate graphql-engines connected to these databases). Thus tests on one worker will not affect the tests on the other worker. - With two workers, this decreases execution times by half, as the tests on event triggers usually takes a long time, but does not consume much CPU.

* added feedback on migration commands (close hasura#1410) (hasura#1931) * firebase2graphql (v0.0.2): support array types in firebase json (close hasura#1829) (hasura#1900) * improve console error page (close hasura#272) (hasura#1890) * add nextjs 8 jwt sample app (hasura#1943) * persist queries in GraphiQL (closes hasura#1894) (hasura#1903) * add realtime chat sample app with vue (hasura#1885) * fix update_event_trigger query not persisting the changes (hasura#1950) fix hasura#1949 * mention required key length for HMAC encryption to user (hasura#1956) Following on from a discussion in discord, a small tweak to make the required key length for a HMAC key more obvious * fix input argument types for custom SQL functions (close hasura#1952) (hasura#1953) * run graphql tests on both http and websocket; add parallelism (close hasura#1868) (hasura#1921) Examples 1) ` pytest --hge-urls "http://127.0.0.1:8080" --pg-urls "postgresql://admin@127.0.0.1:5432/hge_tests" -vv ` 2) `pytest --hge-urls "http://127.0.0.1:8080" "http://127.0.0.1:8081" --pg-urls "postgresql://admin@127.0.0.1:5432/hge_tests" "postgresql://admin@127.0.0.1:5432/hge_tests2" -vv ` ### Solution and Design   #### Reducing execution time of tests - The Schema setup and teardown, which were earlier done per test method, usually takes around 1 sec. - For mutations, the model has now been changed to only do schema setup and teardown once per test class. - A data setup and teardown will be done once per test instead (usually takes ~10ms). - For the test class to get this behaviour, one can can extend the class `DefaultTestMutations`. - The function `dir()` should be define which returns the location of the configuration folder. - Inside the configuration folder, there should be - Files `<conf_dir>/schema_setup.yaml` and `<conf_dir>/schema_teardown.yaml`, which has the metadata query executed during schema setup and teardown respectively - Files named `<conf_dir>/values_setup.yaml` and `<conf_dir>/values_teardown.yaml`. These files are executed to setup and remove data from the tables respectively. #### Running Graphql queries on both http and websockets - Each GraphQL query/mutation is run on the both HTTP and websocket protocols - Pytests test parameterisation is used to achieve this - The errors over websockets are slightly different from that on HTTP - The code takes care of converting the errors in HTTP to errors in websockets #### Parallel executation of tests. - The plugin pytest-xdist helps in running tests on parallel workers. - We are using this plugin to group tests by file and run on different workers. - Parallel test worker processes operate on separate postgres databases(and separate graphql-engines connected to these databases). Thus tests on one worker will not affect the tests on the other worker. - With two workers, this decreases execution times by half, as the tests on event triggers usually takes a long time, but does not consume much CPU. * update manifests to v1.0.0-alpha42 * update regex to delete event triggers in edit-pg-dump (close hasura#1959) (hasura#1960) * support composite foreign keys and primary keys through console (close hasura#1707) (hasura#1854) * edit-pg-dump: fix a missing semi-colon * fix styles in browse rows, relationship and permission tables (hasura#1969) * update console permissions ui (hasura#1924) * Make permissions sections as collapsibles with tooltips * Remove 'use same permission as select' type options from row permissions section * Added clone section to apply same permission to any table-role-action * Disable other subsections till row permissions are set * re-add same as other action options in permissions (hasura#1970) * fix excessive memory consumption with bulk queries (hasura#1972) Use 'Strict' State monad instead of 'Lazy' to avoid unevaluated memory thunks * add placeholder for bool type fields while inserting rows (hasura#1962) * Use UTF-8 encoding for stack hpc report generation (hasura#1974) * fix save function in console's retry config editor (fix hasura#1975) (hasura#1976) * Wip * add scroll to literal blocks in docs (close hasura#1954) (hasura#1978) * remote relationship ui v1 * wip * ui for creating remote relationships * wip * ui fixes * ui fixes * functional ui * namespace ui wip * add functionality for dropping relationship * show exact error in notification * fix state inconsistencies * update reldef for namespaced fields * lint

…asura#1868) (hasura#1921) Examples 1) ` pytest --hge-urls "http://127.0.0.1:8080" --pg-urls "postgresql://admin@127.0.0.1:5432/hge_tests" -vv ` 2) `pytest --hge-urls "http://127.0.0.1:8080" "http://127.0.0.1:8081" --pg-urls "postgresql://admin@127.0.0.1:5432/hge_tests" "postgresql://admin@127.0.0.1:5432/hge_tests2" -vv ` ### Solution and Design   #### Reducing execution time of tests - The Schema setup and teardown, which were earlier done per test method, usually takes around 1 sec. - For mutations, the model has now been changed to only do schema setup and teardown once per test class. - A data setup and teardown will be done once per test instead (usually takes ~10ms). - For the test class to get this behaviour, one can can extend the class `DefaultTestMutations`. - The function `dir()` should be define which returns the location of the configuration folder. - Inside the configuration folder, there should be - Files `<conf_dir>/schema_setup.yaml` and `<conf_dir>/schema_teardown.yaml`, which has the metadata query executed during schema setup and teardown respectively - Files named `<conf_dir>/values_setup.yaml` and `<conf_dir>/values_teardown.yaml`. These files are executed to setup and remove data from the tables respectively. #### Running Graphql queries on both http and websockets - Each GraphQL query/mutation is run on the both HTTP and websocket protocols - Pytests test parameterisation is used to achieve this - The errors over websockets are slightly different from that on HTTP - The code takes care of converting the errors in HTTP to errors in websockets #### Parallel executation of tests. - The plugin pytest-xdist helps in running tests on parallel workers. - We are using this plugin to group tests by file and run on different workers. - Parallel test worker processes operate on separate postgres databases(and separate graphql-engines connected to these databases). Thus tests on one worker will not affect the tests on the other worker. - With two workers, this decreases execution times by half, as the tests on event triggers usually takes a long time, but does not consume much CPU.

nizar-m added 7 commits March 26, 2019 12:04

Do schema setup and teardown only once for a test class

940aad9

Support running with pytest-xdist with --dist=loadfile

ad7d846

for /v1/query updates & inserts, do schema setup and teardown only once

95f503f

Use yaml.safe_load()

5b7177c

Reconnect if websocket is not connected

46832d4

Run graphQL queries on both http and websocket

7db997c

Merge branch 'master' of github.com:hasura/graphql-engine into pytest…

054a97a

…-faster

nizar-m requested review from 0x777 and shahidhk as code owners April 1, 2019 11:25

nizar-m force-pushed the issue-1868 branch from da4518a to 0b24b30 Compare April 1, 2019 11:31

nizar-m changed the title ~~Run GraphQL queries on both http and websocket (close #1868)~~ Run GraphQL tests on both http and websocket (close #1868) Apr 1, 2019

shahidhk added c/server Related to server c/build-system Related to the build-release system labels Apr 2, 2019

Handle Haskell program coverage parse errors

6810ae0

nizar-m force-pushed the issue-1868 branch from 0b24b30 to 6810ae0 Compare April 2, 2019 21:53

Test parallel live queries

9d093c7

1) Set default values for pytest-dist

68c5269

2) readme about structure of pytests 3) Update Contributing.md

ecthiender previously approved these changes Apr 3, 2019

View reviewed changes

nizar-m dismissed ecthiender’s stale review via 68c5269 April 3, 2019 13:09

nizar-m added the s/ok-to-merge Status: This pull request can be merged to master label Apr 8, 2019

Merge branch 'master' into issue-1868

4a53cd6

shahidhk changed the title ~~Run GraphQL tests on both http and websocket (close #1868)~~ run graphql tests on both http and websocket (close #1868) Apr 8, 2019

shahidhk changed the title ~~run graphql tests on both http and websocket (close #1868)~~ run graphql tests on both http and websocket; adds parallelism (close #1868) Apr 8, 2019

shahidhk changed the title ~~run graphql tests on both http and websocket; adds parallelism (close #1868)~~ run graphql tests on both http and websocket; add parallelism (close #1868) Apr 8, 2019

Merge branch 'master' into issue-1868

f2d8363

shahidhk approved these changes Apr 8, 2019

View reviewed changes

shahidhk merged commit a40bf10 into hasura:master Apr 8, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

run graphql tests on both http and websocket; add parallelism (close #1868) #1921

run graphql tests on both http and websocket; add parallelism (close #1868) #1921

nizar-m commented Apr 1, 2019 •

edited

Loading

netlify bot commented Apr 1, 2019 •

edited

Loading

hasura-bot commented Apr 1, 2019

hasura-bot commented Apr 2, 2019

hasura-bot commented Apr 3, 2019

ecthiender left a comment

hasura-bot commented Apr 3, 2019

hasura-bot commented Apr 8, 2019

hasura-bot commented Apr 8, 2019

shahidhk left a comment

hasura-bot commented Apr 8, 2019

run graphql tests on both http and websocket; add parallelism (close #1868) #1921

run graphql tests on both http and websocket; add parallelism (close #1868) #1921

Conversation

nizar-m commented Apr 1, 2019 • edited Loading

Description

Affected components

Related Issues

Invocation

Solution and Design

Reducing execution time of tests

Running Graphql queries on both http and websockets

Parallel executation of tests.

Steps to test and verify

Limitations, known bugs & workarounds

netlify bot commented Apr 1, 2019 • edited Loading

hasura-bot commented Apr 1, 2019

hasura-bot commented Apr 2, 2019

hasura-bot commented Apr 3, 2019

ecthiender left a comment

Choose a reason for hiding this comment

hasura-bot commented Apr 3, 2019

hasura-bot commented Apr 8, 2019

hasura-bot commented Apr 8, 2019

shahidhk left a comment

Choose a reason for hiding this comment

hasura-bot commented Apr 8, 2019

nizar-m commented Apr 1, 2019 •

edited

Loading

netlify bot commented Apr 1, 2019 •

edited

Loading