fix(batch): Improve batch verifier async, correctness, and performance

[teor2345]

Motivation

Zebra's tower-batch has some async performance issues, and contains some buggy and risky code.
We should replace it with more robust code.

Close #4738
Close #4740
Close #4729 (by using a channel that never lags)
Close #4728 (by removing the log entirely)

API Reference

tokio::spawn_blocking():
https://docs.rs/tokio/latest/tokio/task/fn.spawn_blocking.html

Solution

Improve tower-batch async fairness:

• Prefer the batch timer if there are also new batch requests
• Allow other tasks to run after each batch
• Label each batch worker with the verifier's type in tokio-console

Improve error handling in tower-batch:

• Check batch worker tasks for panics and task termination
• Rename Handle to ErrorHandle, and fix up some docs

Improve channel limit handling in tower-batch:

• Use a new channel for each batch
• Use tokio's PollSemaphore instead of an outdated Semaphore impl

Improve channel limit handling in zebra-consensus:

• Replace batch verifier broadcast channels with watch channels

Run all verifier cryptography on blocking CPU-bound threads

• Run batch verification on blocking tokio threads
• Run fallback verification on blocking tokio threads
• Make flush and drop behaviour consistent for all verifiers
• Switch batch verifier tests to the multi-threaded runtime

Related fixes:

• Fix a bug in the v5_with_sapling_spends test (but it's still missing NU5 test vectors)

Review

This is blocking halo2 batch verification, so it would be good to get it in soon.
This might fix some bugs we're seeing during syncing.

Reviewer Checklist

• Code implements Specs and Designs
• CI passes

[codecov]

Codecov Report

Merging #4750 (14d8e78) into main (61eeeb0) will increase coverage by 0.14%.
The diff coverage is 88.77%.

@@            Coverage Diff             @@
##             main    #4750      +/-   ##
==========================================
+ Coverage   78.86%   79.00%   +0.14%     
==========================================
  Files         306      305       -1     
  Lines       37664    37897     +233     
==========================================
+ Hits        29704    29941     +237     
+ Misses       7960     7956       -4     

[teor2345]

I'm running a full sync here, to see if these fixes help:
https://github.com/ZcashFoundation/zebra/actions/runs/2619584632

[oxarbitrage]

I think this looks ok but i am not approving yet in case you want to do #4776 on top of this branch first. Feel free to admin merge or let me know how you will want to handle it and ill approve where needed.

[teor2345]

I admin-merged this separately from PR #4776, just in case we need to debug them as separate commits.