[WIP] server, ui: return replica-level QPS and leaseholder info; render in replica matrix

[vilterp]

(Video)

Example: bin/roachtest bench tpccbench/nodes=30/cpu=16 (30 node x 16 CPU cluster; ~58k ranges)

This uses a lot of memory and CPU in the browser (and on the server, probably), so needs a fair amount of perf investigation& work before merging.

Current perf concerns:

• Server
  • New endpoint LeaseholdersAndQPS calls RaftDebug range internally, and then copies only a few things from that response into its own response. How much is to be gained by removing that intermediate call, and having LeaseholdersAndQPS iterating through stores and ranges itself?
  • RaftDebug iterates through all stores and all ranges they have. How much memory and CPU does this take? How much disk IO? (I assume those range descriptors aren't all stored in memory at all times)
  • returns about 3 megs of data for the example above (58k ranges, 30 nodes); usually takes around 3-10 seconds

TODO

• fix sorting in UI
• figure out why sizes (live bytes) are missing for some replicas

[cockroach-teamcity]

This change is 

[bdarnell]

RaftDebug iterates through all stores and all ranges they have. How much memory and CPU does this take? How much disk IO? (I assume those range descriptors aren't all stored in memory at all times)

Range descriptors are stored in memory, so no disk IO. The CPU and memory cost can be substantial, though.

---

Reviewed 8 of 8 files at r2, 2 of 4 files at r3, 1 of 3 files at r5, 2 of 2 files at r6, 1 of 1 files at r7.
Review status: complete! 0 of 0 LGTMs obtained

---

pkg/server/status.go, line 1595 at r7 (raw file):

			replicaInfo.Stats = replInfo.Range.Stats
			// this seems complicated... each node has its own idea of the lease
			// they might disagree.

Leases have sequence numbers; this should use the one with the highest sequence.

---

pkg/server/serverpb/status.proto, line 613 at r7 (raw file):

message LeaseholdersAndQPSResponse {
  message ReplicaInfo {
    // TODO(vilterp): I am pretty confused about why this is called RangeStatistics

Writes per second is a range-scoped stat; reads per second is replica-scoped (for now it's zero on followers, so the leaseholder's replica stats tell you everything about the range, but this will change when we have follower reads).

---

Comments from Reviewable

[vilterp]

Note to self: the UI should just hit each node independently to ask it for which replicas it has. That way, unhappy/slow nodes don't completely block the rendering of the page.

[bobvawter]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

Pete Vilter seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}