fix(core-transaction-pool): rebuild vs update race

[rainydio]

Summary

There are two kind of operations executed in pool:

• addTransaction, removeTransaction, acceptForgedTransaction, and cleanUp are update operations evolving pool.
• flush and readdTransactions are rebuild operations that work on pool as a whole.

These two kind of operations cannot happen simultaneously. The key difference is that flush and readdTransaction are wiping mempool, storage, and in general require full rebuild of senders wallet repositories.

While addTransaction, removeTransaction, acceptForgedTransaction, and cleanUp were synchronized on sender level, they were not synchronized with flush and readdTransactions. This caused problem in entity transaction handler after pool started handling milestone change. I wasn't able to pinpoint the actual race cause, but got rough idea what sequence of events caused it:

1. Transaction is added to pool.
2. Transaction is forged into new block.
3. While forged block is chained, transaction is accepted.
4. While forged block is chained, milestone change is triggered.
5. Milestone change attempts to re-add transaction, while it's being removed due to accept.

In general solution is to allow addTransaction, removeTransaction, acceptForgedTransaction, and cleanUp to execute in parallel, but readdTransactions and flush to lock pool completely until finished:

• readdTransactions and flush set themself into rebuildLock.
• addTransaction, removeTransaction, acceptForgedTransaction, and cleanUp wait for rebuildLock before adding themself into updateLocks
• readdTransactions and flush wait for any pending updateLocks to finish before touching storage or mempool.

Checklist

• Tests
• Ready to be merged

[codecov]

Codecov Report

Merging #3800 into develop will increase coverage by 10.53%.
The diff coverage is 84.61%.

@@             Coverage Diff              @@
##           develop    #3800       +/-   ##
============================================
+ Coverage    86.39%   96.92%   +10.53%     
============================================
  Files          616      616               
  Lines        13925    13999       +74     
  Branches      1686     1688        +2     
============================================
+ Hits         12030    13569     +1539     
+ Misses        1538      137     -1401     
+ Partials       357      293       -64     

Flag	Coverage Δ
#functional	6.39% <0.00%> (-0.04%)	⬇️
#integration	10.10% <0.00%> (-0.06%)	⬇️
#unit	94.92% <84.61%> (+12.70%)	⬆️

Impacted Files	Coverage Δ
packages/core-transaction-pool/src/service.ts	86.55% <83.78%> (+76.82%)	⬆️
...es/core-transaction-pool/src/expiration-service.ts	100.00% <100.00%> (+61.90%)	⬆️
packages/core-transaction-pool/src/sender-state.ts	100.00% <100.00%> (+55.26%)	⬆️
...ore-magistrate-transactions/src/handlers/entity.ts	68.29% <0.00%> (+4.87%)	⬆️
.../core-api/src/resources/block-with-transactions.ts	84.21% <0.00%> (+5.26%)	⬆️
packages/core-api/src/controllers/locks.ts	100.00% <0.00%> (+9.52%)	⬆️
packages/core-api/src/controllers/delegates.ts	100.00% <0.00%> (+10.81%)	⬆️
packages/core-api/src/plugins/rate-limit.ts	93.54% <0.00%> (+12.90%)	⬆️
...core-magistrate-transactions/src/wallet-indexes.ts	94.73% <0.00%> (+15.78%)	⬆️
packages/core-api/src/controllers/controller.ts	96.42% <0.00%> (+17.85%)	⬆️
... and 70 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3525e86...aa27402. Read the comment docs.