-
Notifications
You must be signed in to change notification settings - Fork 207
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Hardening for 1.3 Release #1470
Comments
RC1 - Migration TestingSome pre-RC testing has already been done in this space, so the aim of this testing is to double check the existing testing and other permutations of FireFly configurations which have not yet been checked. From the original comment we know we don't have coverage for migration testing in these areas:
Additionally, we'll also need to do migration testing in the areas we have covered pre-RC:
Using this issue to track permutations of testing... General steps to migrate a FireFly stack
To upgrade the batch pin contract:
...or alternatively use the script from the original comment to automate this process. Very quick and dirty migration scriptWill contribute this formally into a script somewhere when it's not thrown-together hackiness but this script does a semi-automatic migration. #!/bin/bash
export STACK_NAME=migration
export CREATE_STACK=false
ff init migration
ff start migration
sleep 10
cd ./firefly
make e2e
cd ..
ff stop migration
cat >>~/.firefly/stacks/migration <<EOL
...
EOL
services:
dataexchange_0:
image: localhost:7000/firefly-dataexchange-https:latest
dataexchange_1:
image: localhost:7000/firefly-dataexchange-https:latest
evmconnect_0:
image: localhost:7000/firefly-evmconnect:latest
evmconnect_1:
image: localhost:7000/firefly-evmconnect:latest
firefly_core_0:
image: localhost:7000/firefly:latest
firefly_core_1:
image: localhost:7000/firefly:latest
tokens_0_0:
image: localhost:7000/firefly-tokens-erc20-erc721:latest
tokens_1_0:
image: localhost:7000/firefly-tokens-erc20-erc721:latest
ff start migration
./contract-migration migration ~/.firefly/stacks ~/firefly
cd ~/firefly
git checkout main
git pull
make e2e
|
RC1 - Functionality Check |
RC1 - Performance TestingWill start looking at the performance of RC1 soon, but in the mean time, I've kicked a test of 1.2.2 to gather some performance metrics as a reference point, I'll put the configuration and results below. 1.2.2 Release Commit Reference performance testing for 1.2.2
core-config.yml
ethconnect.yml
instances.yml
FireFly git commit:
|
Steps that I am following for validating migration scenarios:Multiparty Tests
Tokens Tests
|
RC4 - Performance TestingHave started a RC4 performance testing with the below options Reference Performance test options
core-config.yml
ethconnect.yml
instances.yml
FireFly git commit:
|
Run Report RC4Started: 10/04/24 Node Configuration Reference Performance test optionscore-config.yml
ethconnect.yml
instances.yml
FireFly git commit:
NOTE: confirmations set to 0 Results Broadcast messages: 199,448 Summary result:
Grafana results: (Note I modified the Grafana dashboard to add the transfer submitted to the broadcast submitted ahead of #1490 getting merged) I find the heatmap not particularly useful, so this a view with histograms to see on average how long it takes to confirm: Compared to the testing from 1.2, the number look slightly better based on the above testing. I did notice that the TPS and time to confirm grows overtime |
Run Report RC2Started: 24/05/24 Node Configuration Reference Performance test optionscore-config.yml
ethconnect.yml
instances.yml
FireFly git commit:
NOTE: confirmations set to 0 Results Broadcast messages: 1,000,039 Summary result:
Grafana results: I find the heatmap not particularly useful, so this a view with histograms to see on average how long it takes to confirm: |
This is great! Thank you so much for all the work on this, @EnriqueL8 |
Had an awesome chat with @nguyer and come up with what feels like the remaining items required for hardening so that we have confidence in FireFly 1.3.
Pre-requisite: We have an RC for 1.3, realistically it looks like we might a couple of weeks away from that due to a number of in-flight PRs requiring review from specific people.
High-level items:
Migration 1.2.2 -> 1.3 RC
One of the largest items in this release is the change from having a single event stream per plugin (even across multiple namespaces) to having a single event stream per namespace in the network. There has been a whole bunch of testing from branches to verify that the migration should work as expected but there's gaps in the coverage and it's worth re-testing anyway.
In discussion w/ @nguyer we think the areas to be covered here are:
Existing testing has covered:
Script for contract migration
Performing contract migration should be able to be done using this script (note: this only works on Stacks created using the FireFly
ff
CLI.)Major functionality check
There's a draft PR here #1461 with some draft release notes covering the major features being added in this release, as part of hardening here we should go through with the release candidate and check that all of the major function area have been covered by some testing.
Performance testing
Once we have a new RC available we can start to conduct performance regression testing against the previous release. Some very preliminary testing has been done under this issue #1465 so we should be able to use the same configuration for the testing there.
The text was updated successfully, but these errors were encountered: