FRAME: why not refund the weight used for storage access accurately with a new DataBaseAccessExt

[gui1117]

Context:

Transactions are paying for their weight. Weight has 2 part: ref_time and proof_size.

ref_time is calculated using benchmarks. It benchmarks for compute time and also database access. Database access are then converted into some ref_time.

We usually provide benchmark for the worst case and for the best cases. And the call give some refund in case transaction execute in the best cases.

But benchmarks are hard. And benchmarking accurate best cases is even harder considering Database cache.

PVM will give us accurate compute time. But we can already get the accurate database usage after the transaction.

Proposition

I propose we have a new runtime extension similar to ProofSizeExt: DataBaseAccessExt

And DispatchInfo is modified to be more precise: ref_time is split into ref_time_compute and ref_time_db.

So that:

• we can give way more accurate refund due to less database access and more cache hits than expected.
• the refund is more accurate even when no specific best-cases benchmark are provided.

Additionally

• Some extrinsic could want to reserve some weight to be used in on_finalize hook, I guess we can provide some add_reserved_weight in frame-system.
  also such reserved weight would be good for StorageWeightReclaim extension.

[kianenigma]

This would require us to somehow count the number of db cache hits on the client side, and do some kind of weight reclaim based on that?

I am a bit hesitant if this is worth it at this point. PoV was always easier to reclaim, as it is a measurable, objective number. The database access is, in reality, as subjective as the compute time. My disk might be faster or slower than your disk for whatever reason. This is why we never even tried to reclaim the ref_time part: nondeterminism.

Your proposition, to use cache hits, is like a proxy to make a part of ref_time objectively measure-able.

If that is correct, I am not sure if we can even count on this. Some nodes might run their node with smaller or bigger caches, and therefore their database performance and cache-thit-rate will be objectively different from that of other nodes. This will then be nondeterministic.

Perhaps there is a more "deterministic" cache somewhere in the substrate stack that I am not aware of. But this cache will then need to be imposed on all validators.

Even if I am wrong about the above and this is feasible, I would spend my time already implementing PVM in Substrate instead of this + word toward removing all the weight and benchmarking abstractions from FRAME 🙈

[gui1117]

Does PVM measure the time spent in the Trie, it is only the runtime isn't it?

Then with PVM we should implement a cache inside the runtime to avoid hitting the sp-io interface.

Actually we could already implement a cache inside the runtime and then just ask the number of call to the client database access and compare it the benchmarked one.

We could do this for any expensive sp-io call:

• benchmark the amount and have a precise weight formula
• compare the benchmarked amount to the actual amount and reclaim.

anyway I'm ok to wait for PVM

EDIT: if memory is still limited in PVM then having an assumption of a cache of a fixed size on the DB would still make sense IMO

[bkchr]

Goes a little bit into the direction of polkadot-fellows/RFCs#49?