New gas price estimation + HashDB + Fork ID 5 SMC #2196

Psykepro · 2023-06-15T16:06:40Z

Closes #1763
Closes #2067
Closes #2178
Closes #2180
Closes #2098

New config parameters:

[Pool]
IntervalToRefreshGasPrices = "5s"

[Pool.EffectiveGasPrice]
L1GasPriceFactor = 0.25
ByteGasCost = 16
MarginFactor = 1

[Sequencer]
[Sequencer.EffectiveGasPrice]
MaxBreakEvenGasPriceDeviationPercentage = 10
Enabled = false

What does this PR do?

Replaces StateDB for HashDB
Uses flush id to improve sequencer performance
Uses suggested gas price
Updates SMC for fork id 5

Pre-execution of Transactions

Carry out transaction pre-execution
Acquire the Zero-Knowledge (ZK) Counters and Used Gas Price
Do not use ZKCounters in the calculation formula
Include an additional field in the transaction (RLP)

Formula for Calculation (In Pool)

TotalTxPrice = GasUsed * L2MinGasPrice + DataLengthInBytes * byteGasCost * L1GasPrice
BreakEvenGasPrice = TotalTxPrice / GasUsed * marginFactor
(The sequencer is to use the lower value among regular GasPrice and BreakEvenGasPrice.)
Store breakEvenGasPrice in the pool transactions table
L2MinGasPrice = Configuration parameter % L1 Gas price
byteGasCost = 16 (gas per byte in the storage)
MarginFactor: configurable constant = 1.1
Guarantee this price for 30 seconds
- Store the estimation for the tx hash
- Consider this in the RPC when a new tx is added
- If it is ‘>=’ to the estimation, accept it
- If tx hash is not in the table, use the “5 minutes” gas price (current implementation)

Transaction Execution (in Sequencer)

Execute transaction using the BreakEvenGasPrice and recalculate TotalTxPrice using the actual GasUsed. Use this information to recalculate the BreakEvenGasPrice and compare it with the estimated BreakEvenGasPrice in the pre-execution
If the difference is substantial, re-execute with the newly calculated BreakEvenGasPrice
Log an event if a re-execution occurs
Update ZkCounters

Attack Protection

After the second execution, compare again the execution and if there is still a large difference, log an event
Possibly keep the most expensive execution. (We have to decide what to do, for the moment, do nothing, just log the event)

BatchL2Data

Old Format
- RLP(tx) # r # s # v
New Format
- RLP(tx) # r # s # v # effectivePercentage
- effectivePercentage = 1 byte
- effectiveGasPrice = Floor(gasPrice * (effectivePercentage + 1) / 256)
- The protocol team will provide the final formula to calculate effectivePercentage

Sequencer

Add effectivePercentage to the batchl2data when sending txs to the executor
func calcEffectivePercentage (gasPrice, breakevenGasPrice) returns effectivePercentage

Example GasPrice >= BreakEvenGasPRice

gasPrice = 100 Wei
breakEvenGasPRice = 70 wei
Table formula gasPrice * (b + 1)/256
- 255 --> 100 %
- 254 --> 99.6 %
- 253 --> 99.2 %
- ....
- 3 --> 1.56 %
- 2 --> 1.17 %
- 1 --> 0.78 %
- 0 --> 0.39 %
computePercentatge(gasPrice, breakEvenGasPRice) return bytes1 --> [0, 255]
- (breakEvenGasPRice - gasPricegasPrice) / gasPrice = % --> 45.56 % [45% or 46%]
- 46 --> gasPrice * (b - 1)/255 71 Wei
compute transaction

Example GasPrice < BreakEvenGasPrice

select --> 255
----> increase eth_gasPrice RPC endpoint

Reviewers

Signed-off-by: Nikolay Nedkov <nikolai_nedkov@yahoo.com>

* update executor proto * gas price as string Signed-off-by: Nikolay Nedkov <nikolai_nedkov@yahoo.com>

Signed-off-by: Nikolay Nedkov <nikolai_nedkov@yahoo.com>

…timation Signed-off-by: Nikolay Nedkov <nikolai_nedkov@yahoo.com>

Signed-off-by: Nikolay Nedkov <nikolai_nedkov@yahoo.com>

config/config_test.go

pool/pool.go

sequencer/finalizer.go

pool/pool.go

sequencer/finalizer.go

Signed-off-by: Nikolay Nedkov <nikolai_nedkov@yahoo.com>

* add effective_gas_used to receipt * fix * fix * fix sql * fix

* effective gas price comparation * rename function * fix * refactor * linter * fix * update prover image * update prover image * refactor * refactor gas prices * refactor gas prices * fix * fix test * fix test * fix test * fix test * fix test * fix test * fix test * fix test * fix test * fix test * fix test * fix test * fix test * fix test * mutex * fix test * do not return error on l1gas=0 * remove commented code * refactor

ARR552 · 2023-06-30T07:55:27Z

gasprice/default.go

+func newDefaultGasPriceSuggester(ctx context.Context, cfg Config, pool poolInterface) *DefaultGasPricer {
+	// Apply factor to calculate l1 gasPrice
+	factorAsPercentage := int64(cfg.Factor * 100) // nolint:gomnd
+	factor := big.NewInt(factorAsPercentage)
+	defaultGasPriceDivByFactor := new(big.Int).Div(new(big.Int).SetUint64(cfg.DefaultGasPriceWei), factor)
+
+	gpe := &DefaultGasPricer{
+		ctx:        ctx,
+		cfg:        cfg,
+		pool:       pool,
+		l1GasPrice: new(big.Int).Mul(defaultGasPriceDivByFactor, big.NewInt(100)).Uint64(), // nolint:gomnd
+	}


I thin it is not neccessary do the multiplication by 100 because later we divide by 100 and multiply by 100 to remove the percentage.

This is needed to be able to do big.Int arithmetics

ARR552 · 2023-06-30T07:55:50Z

gasprice/default_test.go

+	factorAsPercentage := int64(cfg.Factor * 100) // nolint:gomnd
+	factor := big.NewInt(factorAsPercentage)
+	defaultGasPriceDivByFactor := new(big.Int).Div(new(big.Int).SetUint64(cfg.DefaultGasPriceWei), factor)
+	l1GasPrice := new(big.Int).Mul(defaultGasPriceDivByFactor, big.NewInt(100)).Uint64() // nolint:gomnd
+


I thin it is not neccessary do the multiplication by 100 because later we divide by 100 and multiply by 100 to remove the percentage.

This is needed to be able to do big.Int arithmetics

ARR552 · 2023-06-30T07:56:16Z

gasprice/lastnbatches.go

+	// Store gasPrices
+	factorAsPercentage := int64(g.cfg.Factor * 100) // nolint:gomnd
+	factor := big.NewInt(factorAsPercentage)
+	l1GasPrice := new(big.Int).Div(g.lastPrice, factor)
+	err = g.pool.SetGasPrices(g.ctx, g.lastPrice.Uint64(), l1GasPrice.Uint64())


Here I think the l1GasPrice value that we store on db is incorrect because is 100 times less than the real value

Psykepro and others added 6 commits May 23, 2023 18:25

wip

238841b

Signed-off-by: Nikolay Nedkov <nikolai_nedkov@yahoo.com>

update executor proto (#2122)

6b25dea

* update executor proto * gas price as string Signed-off-by: Nikolay Nedkov <nikolai_nedkov@yahoo.com>

new smc

db73917

smc docker image

8231e94

merging feature/HashDB

e037fb1

Signed-off-by: Nikolay Nedkov <nikolai_nedkov@yahoo.com>

Merge branch 'feature/#2180_update_smc' into feature/new-gas-price-es…

0d2e667

…timation Signed-off-by: Nikolay Nedkov <nikolai_nedkov@yahoo.com>

Psykepro added sequencer rpc e2e pool config labels Jun 15, 2023

Psykepro requested review from tclemos, ARR552, ToniRamirezM and agnusmor June 15, 2023 16:06

Psykepro requested a review from arnaubennassar as a code owner June 15, 2023 16:06

Psykepro self-assigned this Jun 15, 2023

cla-bot bot added the cla-signed label Jun 15, 2023

wip

005bfcc

Signed-off-by: Nikolay Nedkov <nikolai_nedkov@yahoo.com>

Psykepro force-pushed the feature/new-gas-price-estimation branch from a892877 to 005bfcc Compare June 15, 2023 16:10

agnusmor reviewed Jun 19, 2023

View reviewed changes

fixing comments.

104809e

Signed-off-by: Nikolay Nedkov <nikolai_nedkov@yahoo.com>

Psykepro requested a review from agnusmor June 19, 2023 13:57

Merge branch 'develop' into feature/new-gas-price-estimation

42dd793

Signed-off-by: Nikolay Nedkov <nikolai_nedkov@yahoo.com>

Psykepro force-pushed the feature/new-gas-price-estimation branch 5 times, most recently from d438ffa to bec2576 Compare June 20, 2023 06:34

improve: adding missing safe check for L2 Min Gas Price.

652f87c

Signed-off-by: Nikolay Nedkov <nikolai_nedkov@yahoo.com>

ARR552 added 2 commits June 21, 2023 11:51

smc

52ae6c0

genBlockNumb

277cada

ToniRamirezM mentioned this pull request Jun 22, 2023

new smc v2.0.0 forkId5 #2181

Closed

ToniRamirezM changed the title ~~New gas price estimation + HashDB~~ New gas price estimation + HashDB + Fork ID 5 SMC Jun 22, 2023

ToniRamirezM and others added 17 commits June 23, 2023 12:28

use forkID while encoding/decoding txs (#2219)

c44883f

Add effective_gas_price to receipt (#2222)

6eca7e0

* add effective_gas_used to receipt * fix * fix * fix sql * fix

merge develop

0097eb9

fix sql

77cc055

fix sql

4fbb91a

fix

9fb166f

fix

2c6cd24

fix helper

ebfbb51

fix sql

6e39665

change sql

3454ca8

update prover image

f63c4d9

update prover image

dd8a654

solve conflicts

837f81c

fix conflicts

7eee697

fix conflicts

c190c7e

make factors floats

e69a85c

ARR552 reviewed Jun 30, 2023

View reviewed changes

ToniRamirezM added 4 commits June 30, 2023 11:14

fix gasprice

2d866e3

fix gasprice

3a28e89

Merge branch 'develop' into feature/new-gas-price-estimation

22648f1

merge develop and fix conflicts

e8a5389

ToniRamirezM approved these changes Jun 30, 2023

View reviewed changes

ARR552 approved these changes Jun 30, 2023

View reviewed changes

ToniRamirezM merged commit cf20a24 into develop Jun 30, 2023

ToniRamirezM deleted the feature/new-gas-price-estimation branch June 30, 2023 10:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New gas price estimation + HashDB + Fork ID 5 SMC #2196

New gas price estimation + HashDB + Fork ID 5 SMC #2196

Psykepro commented Jun 15, 2023 •

edited by ToniRamirezM

Loading

ARR552 Jun 30, 2023

ToniRamirezM Jun 30, 2023

ARR552 Jun 30, 2023

ToniRamirezM Jun 30, 2023

ARR552 Jun 30, 2023

New gas price estimation + HashDB + Fork ID 5 SMC #2196

New gas price estimation + HashDB + Fork ID 5 SMC #2196

Conversation

Psykepro commented Jun 15, 2023 • edited by ToniRamirezM Loading

New config parameters:

What does this PR do?

Pre-execution of Transactions

Formula for Calculation (In Pool)

Transaction Execution (in Sequencer)

Attack Protection

BatchL2Data

Sequencer

Example GasPrice >= BreakEvenGasPRice

Example GasPrice < BreakEvenGasPrice

Reviewers

ARR552 Jun 30, 2023

Choose a reason for hiding this comment

ToniRamirezM Jun 30, 2023

Choose a reason for hiding this comment

ARR552 Jun 30, 2023

Choose a reason for hiding this comment

ToniRamirezM Jun 30, 2023

Choose a reason for hiding this comment

ARR552 Jun 30, 2023

Choose a reason for hiding this comment

Psykepro commented Jun 15, 2023 •

edited by ToniRamirezM

Loading