svm: interleave transaction validation and processing

[2501babe]

Problem

simd83 intends to relax entry-level constraints, namely that a transaction which takes a write lock on an account cannot be batched with any other transaction which takes a read or write lock on it

before the locking code can be changed, however, svm must be able to support such batches. presently, if one transaction were to write to an account, and a subsequent transaction read from or wrote to that account, svm would not pass the updated account state to the second transaction; it gets them from accounts-db, which is not updated until after all transaction execution finishes

previously, in #3146, we attempted to implement a jit account loader with intermediate account cache to pass updated accounts forward to future transaction in the batch. however, due to existing patterns in how the program cache is used, this has proven quite difficult to get right, and the volume of code changes required is quite large

Summary of Changes

this pr extracts all code from #3146 which does not modify behavior. we would like to commit it standalone for ease of review, and focus specifically on the cache behaviors in a following pr

the existing transaction processing loop in master validates all transaction fee-payers together, loads all transaction accounts together, and then executes each transaction in serial. the new transaction processing loop validates one fee-payer, loads accounts for one transaction, and then executes that transaction before proceeding to the next. this prepares us for an svm where accounts can change in between transactions

we also validate nonce accounts before each transaction, because these accounts can also change in the future svm. a future pr may choose to eliminate most nonce validation code from the check_transactions, but that is out of scope here

futhermore, we create a new type, AccountLoader, which encapsulates the batch-local program cache, account overrides, and the accounts-db callback. it provides the method load_account, which is opaque to its caller, returning a LoadedTransactionAccount according to the exact same rules as the current load_transaction_account

this pr changes nothing about account loading, but introducing AccountLoader prepares us to add the internal account cache in support of simd83

[2501babe]

the first commit is the two files copied exactly from the previous pr, and the second commit deletes the intermediate account cache. this way you can look at the full changeset, or the difference between what you already reviewed, easily

[jstarry]

This change has the side effect of collecting rent from writable accounts that come from account overrides. Probably fine? I'm not too sure yet

[2501babe]

this was intentional:

• the current code actually collects rent from an override if its the fee-payer, otherwise it doesnt. so this is at least consistent
• with simd83, we have to consider what happens to a writable overridden account after the first transaction: does it stay frozen, and all changes to it are ignored? or does it behave like an ordinary account that happens to have an overridden starting value? the latter is more in line with its current behavior, where it is freely allowed to change in the transaction, and then the mutated value is retained in the LoadedTransactionAccount returned from the svm entrypoint
• if we let an overriden account change intrabatch then rent collection should happen on first use, otherwise we collect on second use, which is odd
• account overrides is only intended to be used for simulation. actually giving it a writable account for a real execution would be highly unwise because it would make its way all the way back to accounts-db to be committed

in fact, account overrides is an overly general mechanism for transaction simulation to pass svm a SlotHistory sysvar from the previously frozen bank, and nothing else. andrew was suggesting we could change it if/when we change the load_and_execute_sanitized_transactions interface

[jstarry]

Great breakdown, completely onboard now

[jstarry]

nit: can you introduce a new function called validate_transaction which wraps validate_transaction_nonce and validate_transaction_fee_payer? There's a lot of nesting here that makes the code hard to read IMO

[jstarry]

I love that all the inspect_account call sites are inside AccountLoader now

[2501babe]

im putting this back in draft because alexander and alessandro are planning on fundamentally changing how the program cache interacts with account loading and putting it in 2.1 so we need to coordinate with them

[2501babe]

ok. this will be back in review soon. i spent a while talking with them and alexander thought he had an elegant fix to decouple account loading from program cache but it is now scuttled due to the bugs in how the account loader uses the program cache. but he is making some changes to the program cache tests that i want to rebase on

[2501babe]

once i rebase on alexander, i am removing this get_account_shared_data() call:

• its wrong. the changes to the flow means if it were ever hit, account loading would believe there is a new account to create rather than aborting
• it is impossible to ever trigger because program cache is built directly from accounts-db

alexander is removing the test that artificially triggers this condition