[ Stabilize ] Optimization and bug fixes #1312

astaphobia · 2020-10-22T07:59:23Z

Description

Start from Sep 10 - Oct 22 2020 we had alot bug fixes and enhancements to optimizing speed and memory usages.

Enhancements:

Account type format, this is the biggest one 1204 account type bytes in db #1245
Refactor signature bytes:
- Remove 4 bytes before signature: signature length is now a property of (sender) account type
- Remove first 4 bytes 'in' the signature: signature type is now a property of (sender) account type
Cache storage of some part/data that can speed up thread and some process in zoobc-core, some of them is:
1. Last block state
2. Mempool transactions
3. Scrambled nodes
4. Block objects only some fields that needed
5. Backup mempool when rollback happened
6. Node shards snapshot process part
7. Node registries
8. Node address infos
9. Receipts pool that was received
10. Last Merkle Root
Merkle root in SpineBlock
Allow all transaction types with escrow part, added new event ledgers as well.
Enhancement receipt validation
SpineBlock - completing validate merkle root
New coinbase winner selection algorithm
Add metric for time consumed by PushBlock
Transaction has message fields:
- tx message is expressed in bytes, not as string: core node is agnostic about the message encoding and stores tx messages as BLOB in db
- to build/parse a transaction:
  - add 4 bytes before transaction signature (message length) when creating a transaction and read them to get the message length when parsing transaction bytes into a transaction object/model
  - write the message as bytes (when creating a transaction) and read them as bytes when parsing (logic to encode the message into its original format is demanded to the client application)
Sync node address info send addresses in bulk
Optimize nodeAddressInfo validation by avoiding signature check before other light validation done.
Validate NodePublicKey on decrypting certificate.json file
MerkleTree now accept any n > 1 number of variable.
GetNodeTime API for checking time between client and node.
Remove BadgerDB, shrink down memory usages
Reduce workload when checking length & existing peers node on P2P
Remove package github.com/mohae/deepcopy and use marshal from encoding/json as a substitute

Bug fixes:

Fix ticker leaks on the scheduler
Node receipts query rollback
Fix database transaction called in a big loop
Security-Fix: use zed for a block seed value.
Fix: sort Spinechain genesis spine public keys
Fix: remove backup mempool when fail to restore

Reference schema

zoobc/zoobc-schema#167

* #1145 inject nodeAuthValidation to typeSwitcher * #1145 remove duplicate initialization * #1145 call transaction general validation in multisig inner transaction * #1145 add missing feescale division

* initiate receipt reminder storage * remove unused codes * fix nil allocation of map * receipt reminder only need receipt key and chaintype * fnished mempool backup and receipt reminder cache storage * fnished mempool backup and receipt reminder cache storage * remove println code and remove badger db from go.mod * fix warning line exceded * remove badger key from config.toml * Update loadConfig to let nomad pass a full path to resource folder * Node status (#1170) * update healthcheck service : add node status endpoint * Fix case query between usage (#1173) * sort genesis transactions for consistent genesis block hash and id (#1174) * Genesis generate command refactoring (#1169) * Add genesisTimestamp flag and updated genesis templates * Add logic to validate input data files used to generate genesis.go * Exclude from node seed verification all entries that don't have a node seed, but only node public key and account address (we can't verify them using genesis generate)

* #1160 finish preparation for scramble node storage * #1160 handle scramble node stack limit * #1160 push scramble information by value instead of reference * #1160 add EmptyCache blocker * #1160 add GetTop method for stack storage * #1160 initialize scramble list on startup * #1160 initialize scramble list on finish applying snapshot * #1160 add database lookup in case looking for scramble outside of cache

* move batch receipt to cache storage * initiate batch receipt cache on main in a correct way * check duplicated batch receipt send or receipt tx and block * reminder batch receipt * remove not used fields on p2p service client

* initialize Node Address Info cache * implement node address info cache * fix wrong use address info service * add node Address info on Block main service & fix test * add check error on SetItem storage * fix parameter NewBlockMainService on unite test * add error handler

fix failed st receipt reminders (#1185)

* #1186 rename mempool storage file * #1186 validate receipt based on cached result * #1186 remove dead code

* add cache storage metrics * update mutex mechanism on get size * better usage gauge vector metrics * add node address info cache into metrics

* get size in safety way, no more game

* Remove NodeAddress from node registration tx generator (cmd) * Remove (unused) NodeAddress from struct in genesis generator (cmd) * Fixed escrow=false in generate register node tx command * Updated readme.md Co-authored-by: stefano galassi <stefano.galassi@blockchainzoo.com>

* move function from node registry service to node address info * remove unused mock variable

* #1189 add node registry cache layer * #1189 handle reset index when resorting pending node

…1208)

…registry cache (#1211)

* update node proto to have node public key

* fix: returning wrong address update status * fix: concurent read-write map in storage transactional * log node_id-address:port of failed nodeAddressInfo update

* do not update nodeAddress info if less than or same as latest height in database; * avoid updating multiple node address info that could cause unique constraint error by adding extra status filter * fix unit tests

* fix collide naming and no need to req txs when peer nil * fix posible to nil pointer * need to clearing receipt cache when rollback happened * fix failed test cases that was affected

* Update well known peers for beta Co-authored-by: stefano galassi <stefano.galassi@blockchainzoo.com>

* remove active node registry when participation score reach 0 * calculateNodeOrder divide by 1, it is not used anymore, will be removed in new blocksmith selection implementation #1190 * omit division * fix tests * remove outdated test

* Add arm architecture in Makefile for cross compiling with xgo Co-authored-by: stefano galassi <stefano.galassi@blockchainzoo.com>

* add missing reinitialize cache on import & insert snapshot * remove unneeded intilize snapshot

* fix missing dependencies on node address info * add missing depedencies on cmd

…- this fix will be omitted in next beta since blocksmith strategy will be revamped in #1190 (#1334)

* update readme after enhancement

* add checker for Node ID 0 & Implement blocks storage on generate node address info * use cache to get activeRegisteredNode in InsertNextNodeAdmissionTimestamp

* use BlocksStorage in mempool & optimeze validate request p2p * remove priority peers from blacklist

…mat in Blocks Service (#1341) * add GetBlockByHeightCacheFormat & implement more blocks cache

* fix double import package

andy-shi88 and others added 30 commits September 10, 2020 09:45

Bugfix mempool cache (#1141)

d258ad9

1145 tidy node auth validation (#1151)

5d1375c

* #1145 inject nodeAuthValidation to typeSwitcher * #1145 remove duplicate initialization * #1145 call transaction general validation in multisig inner transaction * #1145 add missing feescale division

update experimental pre-registeredNodes json file (#1156)

6f8b092

Merge branch 'develop' of github.com:zoobc/zoobc-core into experimental

972279a

Merge branch 'develop' of github.com:zoobc/zoobc-core into experimental

52c31cd

Merge branch 'develop' of github.com:zoobc/zoobc-core into experimental

59945d2

Move batch receipt to cache storage (#1178)

5ece9fe

* move batch receipt to cache storage * initiate batch receipt cache on main in a correct way * check duplicated batch receipt send or receipt tx and block * reminder batch receipt * remove not used fields on p2p service client

Merge branch 'develop' of github.com:zoobc/zoobc-core into experimental

a088dc3

fix failed st receipt reminders (#1185)

c915925

fix failed st receipt reminders (#1185)

#348 handle odd and non power2 number of element (#1180)

d056d64

1186 enhance receipt validation (#1188)

7a87666

* #1186 rename mempool storage file * #1186 validate receipt based on cached result * #1186 remove dead code

store map[string] key in hex form (#1192)

6010d1d

add cache storage metrics (#1195)

46ce337

* add cache storage metrics * update mutex mechanism on get size * better usage gauge vector metrics * add node address info cache into metrics

fix conflicts (#1198)

fbdb04c

fix failed when get interface size, need to marshal first: (#1199)

236f3af

Fix failed to get size of variable inside mutex (#1200)

1a71ffa

* get size in safety way, no more game

1187- Move related function to node address info Service (#1196)

41cdeb3

* move function from node registry service to node address info * remove unused mock variable

use manual copy to speed up cache access (#1202)

d75e712

fix index out of range issue (#1203)

8844392

1189 cache node registry (#1201)

0a983ba

* #1189 add node registry cache layer * #1189 handle reset index when resorting pending node

add missing dependency on generate genesis command (#1207)

e6f2132

fix memory leak and better memory allocation on node registry cache (#…

a1a5a8e

…1208)

bugfix: only fetch latest participation score when initializing node …

9005450

…registry cache (#1211)

update monitoring data on commit cache update (#1212)

a5faa3d

Optimize node address info (#1215)

3cb66d8

* update node proto to have node public key

sign hash of txbytes instead of the txbytes directly (#1216)

9c9cdf5

andy-shi88 and others added 20 commits October 26, 2020 11:00

fix: returning wrong address update status (#1320)

affa611

* fix: returning wrong address update status * fix: concurent read-write map in storage transactional * log node_id-address:port of failed nodeAddressInfo update

Fix unique constraint update node address info (#1322)

f5145fe

* do not update nodeAddress info if less than or same as latest height in database; * avoid updating multiple node address info that could cause unique constraint error by adding extra status filter * fix unit tests

fix injection AuthPoown field on node registry tx (#1323)

11ae162

[Bugfix] - No need to request txs when peer nil (#1324)

16c59f5

* fix collide naming and no need to req txs when peer nil * fix posible to nil pointer * need to clearing receipt cache when rollback happened * fix failed test cases that was affected

Code Refactoring (#1325)

1fdb845

* Update well known peers for beta Co-authored-by: stefano galassi <stefano.galassi@blockchainzoo.com>

bugfix: node registration cache error (#1326)

c9bb2f8

Code Refactoring (#1329)

a2f3b17

* Add arm architecture in Makefile for cross compiling with xgo Co-authored-by: stefano galassi <stefano.galassi@blockchainzoo.com>

Add missing reinitialize cache on Snapshot (#1330)

68410a8

* add missing reinitialize cache on import & insert snapshot * remove unneeded intilize snapshot

print error details of scramble node storage (#1331)

5a17626

Fix missing depedencies Node address info (#1332)

bc128de

* fix missing dependencies on node address info * add missing depedencies on cmd

temporary fix: use write lock when reading to avoid concurrent write …

6405d68

…- this fix will be omitted in next beta since blocksmith strategy will be revamped in #1190 (#1334)

Update readme after enhancement (#1321)

425263b

* update readme after enhancement

P2P: Add Node ID 0 checker & update Own Node ID (#1335)

5f4e843

* add checker for Node ID 0 & Implement blocks storage on generate node address info * use cache to get activeRegisteredNode in InsertNextNodeAdmissionTimestamp

pointer caused wrong node chosen to be expelled (#1336)

7d3a948

bugfix: build scramble node after block pushed and cache updated (#1338)

248a6ea

use BlocksStorage in mempool & optimeze validate request p2p (#1339)

ab8f858

* use BlocksStorage in mempool & optimeze validate request p2p * remove priority peers from blacklist

Add GetTotalItems in CacheStorageInterface & GetBlockByHeightCacheFor…

5fc9acc

…mat in Blocks Service (#1341) * add GetBlockByHeightCacheFormat & implement more blocks cache

Fix Duplicate package (#1345)

e497184

* fix double import package

fix multi signature transaction bug (#1344)

1a7a34a

abiman7u modified the milestones: Milestone 20, V.0.0.2.W.02 Nov 12, 2020

astaphobia added 5 commits November 13, 2020 10:15

unstage circleci config

3806153

unstage circleci config

1b4d5d5

fix conflict schema

8023b19

fix conflict schema, pointing to v0.0.1-beta

1d6095c

Merge branch 'develop' into experimental

2265f8e

andy-shi88 merged commit e1fc323 into develop Nov 13, 2020

andy-shi88 removed their assignment Dec 10, 2020

bzpython unassigned sukrawidhyawan Dec 29, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[ Stabilize ] Optimization and bug fixes #1312

[ Stabilize ] Optimization and bug fixes #1312

Uh oh!

astaphobia commented Oct 22, 2020 •

edited

Loading

Uh oh!

Uh oh!

[ Stabilize ] Optimization and bug fixes #1312

[ Stabilize ] Optimization and bug fixes #1312

Uh oh!

Conversation

astaphobia commented Oct 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Enhancements:

Bug fixes:

Reference schema

Uh oh!

Uh oh!

astaphobia commented Oct 22, 2020 •

edited

Loading