test: Add blue/green deployment test #24142
Conversation
def-
requested review from
benesch,
philip-stoev,
ggnall,
ParkMyCar and
teskje
| @@ -205,7 +205,7 @@ def workflow_test_github_12251(c: Composition) -> None: | |||
| c.down(destroy_volumes=True) | |||
| c.up("materialized") | |||
|
|
|||
| start_time = time.process_time() | |||
| start_time = time.time() | |||
There was a problem hiding this comment.
process_time doesn't include the time waiting for Mz to return results: https://docs.python.org/3/library/time.html#time.process_time
def-
force-pushed
the
pr-blue-green-deployment-test
branch
from
ac8e1a0
to
e8f6247
Compare
|
I now got the test green by bumping the testdrive timeout to 5 minutes and the load generator interval to 1s, with 100ms was often seeing dataflows pending. I'm wondering if the way we measure dataflows pending doesn't work for cases where data comes faster than once a second. |
| FOR ALL TABLES | ||
| > CREATE MATERIALIZED VIEW prod_deploy.tpch_mv | ||
| IN CLUSTER prod_deploy AS | ||
| SELECT |
There was a problem hiding this comment.
You may wish to have a dedicated constant column that distinguishes the original view from the swapped-in one. Then check that the value of that column is as expected post-swap.
| STORAGE ADDRESSES ['clusterd1:2103'], | ||
| COMPUTECTL ADDRESSES ['clusterd1:2101'], | ||
| COMPUTE ADDRESSES ['clusterd1:2102'], | ||
| WORKERS 1)) |
There was a problem hiding this comment.
Would more workers and/or more storageds make this test more stessful/realistic?
For example, have the two clusters have different sizes/shapes?
There was a problem hiding this comment.
I'm not sure. My understanding is that one of the motivations is also changing the schema without any downtime. I'll try a different cluster though.
| while running: | ||
| total_runtime = 0 | ||
| queries = [ | ||
| "SELECT * FROM prod.counter_mv", |
There was a problem hiding this comment.
do you need to direct those SELECTs to a particular cluster, or the default one is good enough?
There was a problem hiding this comment.
In my understanding default should be good enough.
test/cluster/mzcompose.py
Outdated
| threads = [PropagatingThread(target=fn) for fn in (selects, subscribe)] | ||
| for thread in threads: | ||
| thread.start() | ||
| time.sleep(3) # some time to make sure the queries run fine |
There was a problem hiding this comment.
I think this interval may need to be longer. Or, alternatively, some check is needed to confirm that every type of query ran at least once post-swap.
| # by the Apache License, Version 2.0. | ||
|
|
||
| > CREATE SCHEMA prod_deploy | ||
| > CREATE SOURCE prod_deploy.counter IN CLUSTER prod_deploy FROM LOAD GENERATOR counter (TICK INTERVAL '1s') |
There was a problem hiding this comment.
You can use a sub-second TICK INTERVAL here, like 0.01s, to make sure stuff actively happens in the system all the time while the swap is in progress.
There was a problem hiding this comment.
I tried this before and it always led to the pending dataflows never becoming considered ready, since their lag would stay at > 1 s. I'm not sure if this is a limitation of Mz or the way that we check the pending dataflows.
|
Won't have time to review, sorry! |
def-
force-pushed
the
pr-blue-green-deployment-test
branch
from
e8f6247
to
323a66b
Compare
Following https://www.notion.so/materialize/Testing-Plan-Blue-Green-Deployments-01528a1eec3b42c3a25d5faaff7a9bf9#f53b51b110b044859bf954afc771c63a
I've seen latencies go to 3 seconds, so hope the 5 seconds limit is ok.
There isn't really a useful check for correctness at the moment.
I'm wondering if the > 60 s when waiting for pending dataflows is acceptable.
Checklist
$T ⇔ Proto$Tmapping (possibly in a backwards-incompatible way), then it is tagged with aT-protolabel.