doc: loki + canary

docs/README.md

@@ -0,0 +1,42 @@
+<p align="center"> <img src="logo_and_name.png" alt="Loki Logo"> <br>
+  <small>Like Prometheus, but for logs!</small> </p>
+
+Grafana Loki is a set of components, that can be composed into a fully featured
+logging stack.
+
+It builds around the idea of treating a single log line as-is. This means that
+instead of full-text indexing them, related logs are grouped using the same
+labels as in Prometheus. This is much more efficient and scales better.
+
+## Components
+- **[Loki](loki/overview.md)**: The main server component is called Loki. It is
+  responsible for permanently storing the logs it is being shipped and it
+  executes the LogQL
+  queries from clients.  
+  Loki shares its high-level architecture with Cortex, a highly scalable
+  Prometheus backend.
+- **[Promtail](promtail/overview.md)**: To ship logs to a central place, an
+  agent is required. Promtail
+  is deployed to every node that should be monitored and sends the logs to Loki.  
+  It also does important task of pre-processing the log lines, including
+  attaching labels to them for easier querying.
+- *Grafana*: The *Explore* feature of Grafana 6.0+ is the primary place of
+  contact between a human and Loki. It is used for discovering and analyzing
+  logs.
+
+Alongside these main components, there are some other ones as well:
+
+- **[LogCLI](logcli.md)**: A command line interface to query logs and labels
+  from Loki
+- **[Canary](canary.md)**: An audit utility to analyze the log-capturing
+  performance of Loki. Ingests data into Loki and immediately reads it back to
+  check for latency and loss.
+- **[Docker
+  Driver](https://github.com/grafana/loki/tree/master/cmd/docker-driver)**: A
+  Docker [log
+  driver](https://docs.docker.com/config/containers/logging/configure/) to ship
+  logs captured by Docker directly to Loki, without the need of an agent.
+- **[Fluentd
+  Plugin](https://github.com/grafana/loki/tree/master/fluentd/fluent-plugin-grafana-loki)**:
+  An Fluentd [output plugin](https://docs.fluentd.org/output), to use Fluentd
+  for shipping logs into Loki

docs/canary.md

@@ -0,0 +1,166 @@
+# Canary
+
+A standalone app to audit the log capturing performance of Loki.
+
+## How it works
+
+![block_diagram](block.png)
+
+loki-canary writes a log to a file and stores the timestamp in an internal
+array, the contents look something like this:
+
+```nohighlight
+1557935669096040040 ppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp
+```
+
+The relevant part is the timestamp, the `p`'s are just filler bytes to make the
+size of the log configurable.
+
+Promtail (or another agent) then reads the log file and ships it to Loki.
+
+Meanwhile loki-canary opens a websocket connection to loki and listens for logs
+it creates
+
+When a log is received on the websocket, the timestamp in the log message is
+compared to the internal array.
+
+If the received log is:
+
+  * The next in the array to be received, it is removed from the array and the
+    (current time - log timestamp) is recorded in the `response_latency`
+    histogram, this is the expected behavior for well behaving logs
+  * Not the next in the array received, is is removed from the array, the
+    response time is recorded in the `response_latency` histogram, and the
+    `out_of_order_entries` counter is incremented
+  * Not in the array at all, it is checked against a separate list of received
+    logs to either increment the `duplicate_entries` counter or the
+    `unexpected_entries` counter.
+
+In the background, loki-canary also runs a timer which iterates through all the
+entries in the internal array, if any are older than the duration specified by
+the `-wait` flag (default 60s), they are removed from the array and the
+`websocket_missing_entries` counter is incremented. Then an additional query is
+made directly to loki for these missing entries to determine if they were
+actually missing or just didn't make it down the websocket. If they are not
+found in the followup query the `missing_entries` counter is incremented.
+
+## Installation
+
+### Binary
+Loki Canary is provided as a pre-compiled binary as part of the
+[Releases](https://github.com/grafana/loki/releases) on GitHub.
+
+### Docker
+Loki Canary is also provided as a Docker container image:
+```bash
+# change tag to the most recent release
+$ docker pull grafana/loki-canary:v0.2.0
+```
+
+### Kubernetes
+To run on Kubernetes, you can do something simple like:
+
+`kubectl run loki-canary --generator=run-pod/v1
+--image=grafana/loki-canary:latest --restart=Never --image-pull-policy=Never
+--labels=name=loki-canary -- -addr=loki:3100`
+
+Or you can do something more complex like deploy it as a daemonset, there is a
+ksonnet setup for this in the `production` folder, you can import it using
+jsonnet-bundler:
+
+```shell
+jb install github.com/grafana/loki-canary/production/ksonnet/loki-canary
+```
+
+Then in your ksonnet environments `main.jsonnet` you'll want something like
+this:
+
+```jsonnet
+local loki_canary = import 'loki-canary/loki-canary.libsonnet';
+
+loki_canary {
+  loki_canary_args+:: {
+    addr: "loki:3100",
+    port: 80,
+    labelname: "instance",
+    interval: "100ms",
+    size: 1024,
+    wait: "3m",
+  },
+  _config+:: {
+    namespace: "default",
+  }
+}
+
+```
+
+### From Source
+If the other options are not sufficient for your use-case, you can compile
+`loki-canary` yourself:
+
+```bash
+# clone the source tree
+$ git clone https://github.com/grafana/loki
+
+# build the binary
+$ make loki-canary
+
+# (optionally build the container image)
+$ make loki-canary-image
+```
+
+## Configuration
+
+It is required to pass in the Loki address with the `-addr` flag, if your server
+uses TLS, also pass `-tls=true` (this will create a `wss://` instead of `ws://`
+connection)
+
+You should also pass the `-labelname` and `-labelvalue` flags, these are used by
+loki-canary to filter the log stream to only process logs for this instance of
+loki-canary, so they must be unique per each of your loki-canary instances. The
+ksonnet config in this project accomplishes this by passing in the pod name as
+the labelvalue
+
+If you get a high number of `unexpected_entries` you may not be waiting long
+enough and should increase `-wait` from 60s to something larger.
+
+__Be cognizant__ of the relationship between `pruneinterval` and the `interval`.
+For example, with an interval of 10ms (100 logs per second) and a prune interval
+of 60s, you will write 6000 logs per minute, if those logs were not received
+over the websocket, the canary will attempt to query loki directly to see if
+they are completely lost. __However__ the query return is limited to 1000
+results so you will not be able to return all the logs even if they did make it
+to Loki.
+
+__Likewise__, if you lower the `pruneinterval` you risk causing a denial of
+service attack as all your canaries attempt to query for missing logs at
+whatever your `pruneinterval` is defined at.
+
+All options:
+
+```nohighlight
+  -addr string
+        The Loki server URL:Port, e.g. loki:3100
+  -buckets int
+        Number of buckets in the response_latency histogram (default 10)
+  -interval duration
+        Duration between log entries (default 1s)
+  -labelname string
+        The label name for this instance of loki-canary to use in the log selector (default "name")
+  -labelvalue string
+        The unique label value for this instance of loki-canary to use in the log selector (default "loki-canary")
+  -pass string
+        Loki password
+  -port int
+        Port which loki-canary should expose metrics (default 3500)
+  -pruneinterval duration
+        Frequency to check sent vs received logs, also the frequency which queries for missing logs will be dispatched to loki (default 1m0s)
+  -size int
+        Size in bytes of each log line (default 100)
+  -tls
+        Does the loki connection use TLS?
+  -user string
+        Loki username
+  -wait duration
+        Duration to wait for log entries before reporting them lost (default 1m0s)
+```