Limit size of transactions included in ChunkStateWitness

[jancionear]

ChunkStateWitness contains transactions which were included in the last two chunks:

pub struct ChunkStateWitness {
    ...
    pub transactions: Vec<SignedTransaction>,
    ...
    pub new_transactions: Vec<SignedTransaction>,
    pub new_transactions_validation_state: PartialState,
}

We must ensure that the size of these fields isn't too big. A malicious chunk producer could potentially try to construct a ChunkStateWitness with so many transactions that the size of the witness would explode. Large witnesses are known to cause problems, so we must implement some sort of limit.

Refs: #10259 (comment)

It's a security issue, so I'd say that it's necessary for stateless validation MVP.

[jancionear]

From what I read it seems that a single Action is always less than ~4MB in size.
Action::DeployContract contains a contract which is limited to 4MB. Action::FunctionCall contains function arguments which are limited to 4MB.
Despite that, a single transaction can contain multiple actions (up to 100), so a single transaction could potentially be as large as 400MB!

Because of that I think that we should introduce a hard per-transaction limit, which could be set to 4MB. We can't make the limit much smaller than that because there are already contracts which are larger than 2MB, so the limit would have to be at least 2MB to allow deploying such contracts. 4MB is in line with the previous limitation.

[jancionear]

It's a bit sad that ChunkStateWitness contains transactions from both the current and the previous chunk. That pretty much halves the size limit that we're able to afford, as every transaction is included in two ChunkStateWitnesses:

pub struct ChunkStateWitness {
    /// The transactions to apply. These must be in the correct order in which
    /// they are to be applied.
    pub transactions: Vec<SignedTransaction>,
    /// Finally, we need to be able to verify that the new transitions proposed
    /// by the chunk (that this witness is for) are valid. For that, we need
    /// the transactions as well as another partial storage (based on the
    /// pre-state-root of this chunk) in order to verify that the sender
    /// accounts have appropriate balances, access keys, nonces, etc.
    pub new_transactions: Vec<SignedTransaction>,
    pub new_transactions_validation_state: PartialState,
}

If there was one 4MB transaction in the previous chunk and one 4MB transaction in the current one, that'll cause the witness to be 8MB, and that's before the storage proof which could also be as large as 7MB :/

AFAIR this was done because it was easier to implement transaction processing this way. We could consider modifying this so that every transaction is included only once, that'd give us more breathing room.

[jancionear]

To deal with transaction doubling I'd propose to enforce a limit on the sum of all transaction fields:

For all valid ChunkStateWitnesses sizeof(transactions) + sizeof(new_transactions) + sizeof(new_transactions_validation_state) must be smaller than 5MB, otherwise the witness is invalid.

One worry is that the previous chunk producer could prevent the next one from adding transactions by taking up the whole allocated 5MB, leaving zero space for new_transactions. IMO this isn't a big problem, as the order of chunk producers is random.

The "proper" way to deal with this would be to include transactions once, but that'd probably require heavy refactoring.

[jancionear]

I collected some metrics to see how big all these fields of ChunkStateWitness really are.
Things generally look like this (mainnet traffic with shadow validation):

link to the dashboard

• new_transactions_size is usually under 200kB
• new_transactions_validation_state_size is usually under 500kB

This means that:

• We don't need size limits for "normal" mainnet traffic, the limits are mainly needed to protect against attacks
• 5MB limit for previous and new transactions is more than enough
• big transactions are extremely rare, I haven't seen a single one above 200kB during the 30min period.

I'm a bit surprised that the storage proof is this big, afaik validating a transactions requires only reading the account and access key, which are rather small o-0. I guess things add up.

[jancionear]

Adding a simple 5MB limit on transaction size might not be enough on its own. 5MB of transactions and 7MB of receipt storage proof would add up to 12MB, which is quite big.
Maybe 12MB is acceptable, it would be nice if it were as we could go with a simpler solution, but it might not be. I opened #11184 to figure out what's the actual maximum size that the network can handle.

If it turns out that 12MB is too large, we'll have to think of resolving the the conflict between transactions and receipts somehow. What to do when there's a 4MB transaction waiting in the pool, as well as 7MB of receipts to process?
Currently we reserve 300TGas for processing new transactions, and the rest is spent on processing receipts. Plus we stop adding new transactions when the delayed receipt queue gets too big. The problem is that this approach doesn't really translate to size limits. We must accept a 4MB transaction, but with low limits that 4MB would take up the whole chunk, so it's like spending 1PGas on processing a single transaction :/ So if we gave priority to transactions, having a lot of 4MB transactions would mean that we stop processing receipts. Giving priority to receipts also doesn't work, as it would starve big transactions.
One way to solve it would be to accept big transaction on every n-th block. For example we could accept transactions larger than 1MB on every 10th block. This "big transaction blocks" would contain the big transaction, but in exchange they would contain less receipts, we could lower the per-chunk soft limit on those blocks.

[jancionear]

Idea from @Longarithm:

We could apply the receipts first, and then include as many transactions as'll fit under the limit. The big transactions would be included in chunks that don't have a lot of receipts, we have those every now and then.

[jancionear]

Idea from @bowenwang1996

We could lower the size limit for a single transaction. Disallow contracts larger than e.g 1MB.
Ethereum has a 24kB limit for contracts, why is it 4MB on NEAR?

[Longarithm]

From my discussion with @jancionear:

There is check of max args length of function call.
You can generate really small tx, but inside a contract it'll generate child receipt with 4MB garbage args.
And you can probably generate plenty of them
And this itself may make source receipt fields for state witness so big that it can't get distributed.
Limiting tx size doesn't help but actually we can include this to "upper bound" and interrupt the contract if it happens.

Another concern - we can get unlucky and receive large txs/receipts from all shard ids simultaneously, so we'd have to multiply worst case by 6, currently.
So I'm afraid we will need to distribute source receipts separately, for each source shard, and reconstruct full source_receipt_proofs on the target node.

[jancionear]

Collected some data:

In the current state there are ~170 accounts with contracts larger than 1MB, the largest ones reach 4MB:
top-200.txt

To support those contracts we'd have to support 4MB DeployContract transactions :/

Applying compression helps, only 6 accounts have contracts with compressed size larger than 1MB:

Top 200 accounts by compressed contract size:
1.3 MB: myriad.octopus-registry.near
1.3 MB: discovol.octopus-registry.near
1.3 MB: gateway-routerprotocol.near
1.1 MB: mint.exverse.near
1.1 MB: client.bsc-bridge.near
1038.8 KB: nearpay-portals.near
931.1 KB: moodev.near
...

top-200-compressed.txt

The only problem is that it'd make zstd compression part of the consensus protocol, we'd have to keep this exact version of zstd forever :/

I also scanned the last year of blockchain history to find the largest transactions: top-transactions-1year.txt
There were ~600 transactions with size >= 1MB: ~400 DeployContracts, ~100 FunctionCalls and ~100 other transactions.
Out of those, about 400 were >= 1.5MB

[jancionear]

zulip discussion about lowering the size limit: https://near.zulipchat.com/#narrow/stream/295306-contract-runtime/topic/Lowering.20the.20limit.20for.20contract.20code.20size