[ZIP 402] Specify database formats with witness-related data in a separate file, no longer storing it in wallet.dat

[bitcartel]

Consider saving witness related data to a file, e.g. witness.dat instead of storing it in wallet.dat.

• Separation of data, to isolate and only store keys in wallet.dat.
• Performance by avoiding contention between different subsystems trying to write to wallet.dat
• Avoid frequent writing of data to wallet.dat on block rescan and reindex.
• Reduce risk that when a system crash occurs, wallet.dat is being written to

Related, see zcash/zcash#1749

[str4d]

Currently, we store witness-related data with the wallet transaction it belongs to. IIRC we considered having a single witness data entity and then decided against it (possibly to try and reduce the chance of that data getting out-of-sync with the transaction data, I don't remember). We could decide to reverse the decision and split out witness data from the wallet file.

An alternative proposal is to split apart recoverable and non-recoverable information. That is, have wallet.dat only contain keys, and then have e.g. transactions.dat contain the transactions themselves, witness data etc. I think this is more in line with the considerations @bitcartel listed in the OP. If we do the split simply on transactions, then the only non-recoverable data in transactions.dat would be the transaction labels, but a) that's not a huge loss, and b) private transactions contain memo fields which would therefore be recoverable. We could also just leave a skeleton map in wallet.dat of `txid -> label'.

Another point to consider is that when we implement viewing keys, if we use the current in-net architecture then we will need to save the per-transaction ephemeral private keys for the sender in wallet.dat, for transactions stored in transactions.dat Thus we again return to having the possibility of inconsistency between two data files, although once a transaction is mined the problem is reduced, because the viewing keys will be usable any time after a reindex (unlike for witness data, which needs to be reset or carefully handled).

[daira]

+1 on the recoverable/non-recoverable split — including storing the txid -> label map and optional per-transaction ephemeral private keys in the non-recoverable db.

Furthermore I think we should do this at the same time as the SQLite migration, so that the only possibilities are the old BDB wallet.dat, or two new SQLite files wallet.sqlite and transactions.sqlite (supporting encryption).

As resident "forward secrecy is important damnit" curmudgeon, I'm compelled to point out that storing the ephemeral private keys breaks forward secrecy for memo contents. (Forward privacy of when particular notes were sent is already broken by storing spending keys, or by storing full viewing keys in the case of Sapling.)

[daira]

ZIP number changed to avoid a clash.