Log 13434 leader (logdna#388)

* Summary:
This adds leader election to the agent - currently it is just wrapped around the metrics reporter.
One leader is chosen in a cluster, and will hold on to that leader indefinitely. The only way a leader should be lost is if a pod goes down, or if for some reason execution hangs long enough that it is unable to update the leader lease.
Pods not elected leader will poll with a duration of the leaseDurationInSeconds. Or 5 minutes if that for whatever reason is not set.
Polling to become the leader has a built in jitter used by a random interval of ten seconds.
See:
leader-election-leases.yaml
for lease definition.
Leases don't have a concept of expire - this must be done with manually checking.
The metrics reporter stream is set to be created, and then blocked until such time it becomes the leader.

Test plan:
Integration Test
Unit Test
Manual Testing via observing leader logs

Ref: LOG-13434

* removed a file

* pr feedback

* fixed log level

* lint

Co-authored-by: James <james.bloom@mezmo.com>

bin/Cargo.toml

@@ -50,6 +50,7 @@ pin-utils = "0.1"
 regex = "1"
 auditable = { version = "0.1", optional = true }
 miniz_oxide = "0.5"
+rand = "0.8.5"
 
 [target.'cfg(any(unix, macos))'.dependencies]
 capctl = "0.2"
@@ -106,6 +107,7 @@ prometheus-parse = { git = "https://github.com/ccakes/prometheus-parse-rs", rev
 float-cmp = "0.9.0"
 once_cell = "1.10"
 serial_test = "0.8"
+chrono = { version = "0.4", features = ["serde"] }
 
 [target.'cfg(target_os="linux")'.dev-dependencies]
 pnet = "0.28"