Partkey Interface: Avoid mutex and DB operation in Register/Record

[winder]

Description:

The ParticipationRegistry attempts to DB access for functions called on the consensus goroutine. It does this using a cache, and a read/write mutex. Because of the mutex, there are cases where a long-running operation on another goroutine could block consensus.

Acceptance Criteria:

• use a buffered channel to pass Register/Record events to a separate goroutine.
• do not write registrations to the DB right away, use the cache and write them along with the record events. This is safe because we attempt to re-register keys at the start of every round.
• Use a different mutex for Get and Flush

[yaovi-a]

Hey team! Please add your planning poker estimate with ZenHub @winder @tsachizehub @nicholasguoalgorand @algonautshant @algobolson @AlgoStephenAkiki @tolikzinovyev

[brianolson]

.Record() doesn't do any IO, but just touches .dirty{} and .cache{}, so I think the mutex shadow on that should not be too bad.
Making .Register() async isn't hard and I have that in local code now.
Should .Insert() be async? Probably .Flush() should still by synchronous?

[winder]

Yes, I think the ideal solution would be to have everything except Flush be asynchronous. I think the way the mutex is currently organized would mean a long .Flush() would cause the otherwise fast operations to block. Does mutex shadow mean you have a 2nd mutex for the long operation so that the critical path mutex is only used for fast things?