Expand documentation (#261)

* Clarify links to other sections

* Expanded setup sections

* Clarified usage docs

* Extracted contributing.md

* Extracted provisioning.md

* Operations docs

* Troubleshooting docs started

* Review feedback.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

CONTRIBUTING.md

@@ -0,0 +1,20 @@
+# Contribute
+
+Loki uses GitHub to manage reviews of pull requests:
+
+- If you have a trivial fix or improvement, go ahead and create a pull request.
+- If you plan to do something more involved, discuss your ideas on the relevant GitHub issue.
+
+## Steps to contribute
+
+For now, you need to add your fork as a remote on the original **\$GOPATH**/src/github.com/grafana/loki clone, so:
+
+```bash
+
+$ go get github.com/grafana/loki
+$ cd $GOPATH/src/github.com/grafana/loki # GOPATH is $HOME/go by default.
+
+$ git remote add <FORK_NAME> <FORK_URL>
+```
+
+Notice: `go get` return `package github.com/grafana/loki: no Go files in /go/src/github.com/grafana/loki` is normal.

README.md

@@ -6,29 +6,37 @@
 
 # Loki: like Prometheus, but for logs.
 
-Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by [Prometheus](https://prometheus.io/).  It is designed to be very cost effective and easy to operate, as it does not index the contents of the logs, but rather a set of labels for each log stream.
+Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by [Prometheus](https://prometheus.io/).
+It is designed to be very cost effective and easy to operate.
+It does not index the contents of the logs, but rather a set of labels for each log stream.
 
 Compared to other log aggregation systems, Loki:
 
 - does not do full text indexing on logs. By storing compressed, unstructured logs and only indexing metadata, Loki is simpler to operate and cheaper to run.
 - indexes and groups log streams using the same labels you’re already using with Prometheus, enabling you to seamlessly switch between metrics and logs using the same labels that you’re already using with Prometheus.
 - is an especially good fit for storing [Kubernetes](https://kubernetes.io/) Pod logs. Metadata such as Pod labels is automatically scraped and indexed.
-- has native support in Grafana (already in the nightly builds, will be included in Grafana 6.0).
+- has native support in Grafana (needs Grafana v6.0).
 
 A Loki-based logging stack consists of 3 components:
 
 - `promtail` is the agent, responsible for gathering logs and sending them to Loki.
 - `loki` is the main server, responsible for storing logs and processing queries.
-- [Grafana](https://github.com/grafana/grafana) for the UI.
+- [Grafana](https://github.com/grafana/grafana) for querying and displaying the logs.
 
 Loki is like Prometheus, but for logs: we prefer a multidimensional label-based approach to indexing, and want a single-binary, easy to operate system with no dependencies.
 Loki differs from Prometheus by focussing on logs instead of metrics, and delivering logs via push, instead of pull.
 
 ## Getting started
 
-For instructions on getting started with Loki, see [our getting started docs](./production/README.md).
+The [getting started docs](./production/README.md) have instructions on how to install Loki via Docker images, Helm charts, Jsonnet, or from source.
 
-For the beginnings of documentation on how to use Loki, see [our usage docs](./docs/usage.md).  [API documentation](./docs/api.md) is also available.
+Once you have promtail, Loki, and Grafana running, continue with [our usage docs](./docs/usage.md) on how to query your logs.
+
+### Documentation
+
+- [API documentation](./docs/api.md) for alternative ways of getting logs into Loki.
+- [Operations](./docs/operations.md) for important aspects of running Loki.
+- [Troubleshooting](./docs/troubleshooting.md) for help around frequent error messages.
 
 ## Getting Help
 
@@ -44,23 +52,19 @@ Your feedback is always welcome.
 ## Further Reading
 
 - The original [design doc](https://docs.google.com/document/d/11tjK_lvp1-SVsFZjgOTr1vV3-q6vBAsZYIQ5ZeYBkyM/view) for Loki is a good source for discussion of the motivation and design decisions.
-- David Kaltschmidt's OSMC 2018 talk "[Logging is coming to Grafana](https://www.youtube.com/watch?v=vDXAVZY7rB0)"
-- David Kaltschmidt's KubeCon 2018 talk "[On the OSS Path to Full Observability with Grafana][kccna18-event]" ([slides][kccna18-slides], [video][kccna18-video])
-- Goutham Veeramachaneni's blog post "[Loki: Prometheus-inspired, open source logging for cloud natives](https://grafana.com/blog/2018/12/12/loki-prometheus-inspired-open-source-logging-for-cloud-natives/)".
+- Tom Wilkie's Jan 2019 CNCF Paris Meetup talk "[Grafana Loki: like Prometheus, but for logs](https://speakerdeck.com/grafana/grafana-loki-like-prometheus-but-for-logs)".
+- David Kaltschmidt's KubeCon 2018 talk "[On the OSS Path to Full Observability with Grafana][kccna18-event]" ([slides][kccna18-slides], [video][kccna18-video]) on how Loki fits into a cloud-native environment.
+- Goutham Veeramachaneni's blog post "[Loki: Prometheus-inspired, open source logging for cloud natives](https://grafana.com/blog/2018/12/12/loki-prometheus-inspired-open-source-logging-for-cloud-natives/)" on details of the Loki architectire.
+- David Kaltschmidt's blog post "[Closer look at Grafana's user interface for Loki](https://grafana.com/blog/2019/01/02/closer-look-at-grafanas-user-interface-for-loki/)" on the ideas that went into the logging user interface.
 
 [kccna18-event]: https://kccna18.sched.com/event/GrXC/on-the-oss-path-to-full-observability-with-grafana-david-kaltschmidt-grafana-labs
 [kccna18-slides]: https://speakerdeck.com/davkal/on-the-path-to-full-observability-with-oss-and-launch-of-loki
 [kccna18-video]: https://www.youtube.com/watch?v=U7C5SpRtK74&list=PLj6h78yzYM2PZf9eA7bhWnIh_mK1vyOfU&index=346
 
 ## Contributing
 
-For now, you need to add your fork as a remote on the original **$GOPATH**/src/github.com/grafana/loki clone, so:
-
-```bash
+Refer to [CONTRIBUTING.md](CONTRIBUTING.md)
 
-$ go get github.com/grafana/loki
-$ cd $GOPATH/src/github.com/grafana/loki # GOPATH is $HOME/go by default.
+## License
 
-$ git remote add <FORK_NAME> <FORK_URL>
-```
-Notice: `go get` return `package github.com/grafana/loki: no Go files in /go/src/github.com/grafana/loki` is normal.
+Apache License 2.0, see [LICENSE](LICENSE).

docs/api.md

@@ -1,20 +1,6 @@
 # Loki API
 
-## Authentication
-
-*nb* Authentication is out of scope for this project.
-You are expected to run an authenticating reverse proxy in front of our services, such as an Nginx with basic auth or an OAuth2 proxy.
-
-Loki is a multitenant system; requests and data for tenant A are isolated from tenant B.
-Requests to the Loki API should include an HTTP header (`X-Scope-OrgID`) identifying the tenant for the request.
-Tenant IDs can be any alphanumeric string; limiting them to 20 bytes is reasonable.
-
-Loki can be run in "single-tenant" mode where the `X-Scope-OrgID` header is not required.
-In this situation, the tenant ID is defaulted to be `fake`.
-
-## REST API
-
-There are 4 API endpoints:
+The Loki server has the following API endpoints (_Note:_ Authentication is out of scope for this project):
 
 - `POST /api/prom/push`
 
@@ -23,24 +9,23 @@ There are 4 API endpoints:
   - [ProtoBuffer definition](/pkg/logproto/logproto.proto)
   - [Golang client library](/pkg/promtail/client.go)
 
-  Also accepts JSON formatted requests when the header `Content-Type: application/json` is sent.  Example of the JSON format:
+  Also accepts JSON formatted requests when the header `Content-Type: application/json` is sent. Example of the JSON format:
 
   ```json
   {
-      "streams": [
-          {
-              "labels": "{foo=\"bar\"}",
-              "entries": [
-                  {"ts": "2018-12-18T08:28:06.801064-04:00", "line": "baz"}
-              ]
-          }
-      ]
+    "streams": [
+      {
+        "labels": "{foo=\"bar\"}",
+        "entries": [{ "ts": "2018-12-18T08:28:06.801064-04:00", "line": "baz" }]
+      }
+    ]
   }
   ```
 
 - `GET /api/prom/query`
 
   For doing queries, accepts the following parameters in the query-string:
+
   - `query`: a logQL query
   - `limit`: max number of entries to return
   - `start`: the start time for the query, as a nanosecond Unix epoch (nanoseconds since 1970)
@@ -49,6 +34,7 @@ There are 4 API endpoints:
   - `regexp`: a regex to filter the returned results, will eventually be rolled into the query language
 
   Responses looks like this:
+
   ```
   {
     "streams": [
@@ -72,6 +58,7 @@ There are 4 API endpoints:
   For retrieving the names of the labels one can query on.
 
   Responses looks like this:
+
   ```
   {
     "values": [
@@ -86,6 +73,7 @@ There are 4 API endpoints:
   For retrieving the label values one can query on.
 
   Responses looks like this:
+
   ```
   {
     "values": [

docs/grafana-provisioning.md

@@ -0,0 +1,39 @@
+# Configuring Grafana via Provisioning
+
+It is possible to configure Grafana datasources using config files with Grafana’s provisioning system. You can read more about how it works in the [Grafana documentation](http://docs.grafana.org/administration/provisioning/#datasources).
+
+Here is a simple example of the provisioning yaml config for the Grafana Loki datasource:
+
+```yaml
+apiVersion: 1
+
+datasources:
+  - name: Loki
+    type: loki
+    access: proxy
+    url: http://localhost:3100
+    editable: false
+```
+
+Example with basic auth:
+
+```yaml
+apiVersion: 1
+
+datasources:
+  - name: Loki
+    type: loki
+    access: proxy
+    url: http://localhost:3100
+    editable: false
+    basicAuth: true
+    basicAuthUser: my_user
+    basicAuthPassword: test_password
+```
+
+Make sure to adjust the url and authentication to your needs, the `url` should be:
+
+- `http://localhost:3100` when run Loki locally
+- `http://loki:3100` when run Loki with docker-compose
+
+`basicAuthUser` and `basicAuthPassword` should same as your Grafana setting.

docs/logcli.md

@@ -1,6 +1,6 @@
 # Log CLI usage Instructions
 
-Loki's main UI is Grafana; however, a basic CLI is provided as a proof of concept.
+Loki's main query interface is Grafana; however, a basic CLI is provided as a proof of concept.
 
 Once you have Loki running in a cluster, you can query logs from that cluster using the following commands:
 

docs/operations.md

@@ -0,0 +1,82 @@
+# Operations
+
+This page lists operational aspects of running Loki in alphabetical order:
+
+## Authentication
+
+Loki does not have an authentication layer.
+You are expected to run an authenticating reverse proxy in front of your services, such as an Nginx with basic auth or an OAuth2 proxy.
+
+### Multi-tenancy
+
+Loki is a multitenant system; requests and data for tenant A are isolated from tenant B.
+Requests to the Loki API should include an HTTP header (`X-Scope-OrgID`) identifying the tenant for the request.
+Tenant IDs can be any alphanumeric string; limiting them to 20 bytes is reasonable.
+
+Loki can be run in "single-tenant" mode where the `X-Scope-OrgID` header is not required.
+In this situation, the tenant ID is defaulted to be `fake`.
+
+## Observability
+
+### Metrics
+
+Both Loki and promtail expose a `/metrics` endpoint for Prometheus metrics.
+
+Loki metrics:
+
+- `log_messages_total` Total number of log messages.
+- `loki_distributor_bytes_received_total` The total number of uncompressed bytes received per tenant.
+- `loki_distributor_lines_received_total` The total number of lines received per tenant.
+- `loki_ingester_streams_created_total` The total number of streams created per tenant.
+- `loki_request_duration_seconds_count` Number of received HTTP requests.
+
+Promtail metrics:
+
+- `promtail_read_bytes_total` Number of bytes read.
+- `promtail_read_lines_total` Number of lines read.
+- `promtail_request_duration_seconds_count` Number of send requests.
+- `promtail_sent_bytes_total` Number of bytes sent.
+
+Most of these metrics are counters and should continuously increase during normal operations:
+
+1. Your app emits a log line to a file tracked by promtail.
+2. Promtail reads the new line and increases its counters.
+3. Promtail forwards the line to a Loki distributor, its received counters should increase.
+4. The Loki distributor forwards it to a Loki ingester, its request duration counter increases.
+
+### Monitoring Mixins
+
+Check out our [Loki mixin](../production/loki-mixin) for a set of dashboards, recording rules, and alerts.
+These give you a comprehensive package on how to monitor Loki in production.
+
+For more information about mixins, take a look at the [mixins project docs](https://github.com/monitoring-mixins/docs).
+
+## Scalability
+
+See this [blog post](https://grafana.com/blog/2018/12/12/loki-prometheus-inspired-open-source-logging-for-cloud-natives/) on a discussion about Loki's scalability.
+
+When scaling Loki, consider running several Loki processes with their respective roles of ingestor, distributor, and querier.
+Take a look at their respective `.libsonnet` files in [our production setup](../production/ksonnet/loki) to get an idea about resource usage.
+
+We're happy to get feedback about your resource usage.
+
+## Storage
+
+Loki needs two stores: an index store and a chunk store.
+Loki receives logs in separate streams.
+Each stream is identified by a set of labels.
+As the log entries from a stream arrive, they are gzipped as chunks and saved in the chunks store.
+The index then stores the stream's label set, and links them to the chunks.
+
+### Local storage
+
+By default, Loki stores everything on disk.
+The index is stored in a BoltDB under `/tmp/loki/index`.
+The chunks are stored under `/tmp/loki/chunks`.
+
+### Cloud storage
+
+Loki has support for Google Cloud storage.
+Take a look at our [production setup](https://github.com/grafana/loki/blob/a422f394bb4660c98f7d692e16c3cc28747b7abd/production/ksonnet/loki/config.libsonnet#L55) for the relevant configuration fields.
+
+Support for AWS and Azure is also available, but not yet documented.

docs/troubleshooting.md

@@ -0,0 +1,16 @@
+# Troubleshooting
+
+## "Data source connected, but no labels received. Verify that Loki and Promtail is configured properly."
+
+This error can appear in Grafana when you add Loki as a datasource.
+It means that Grafana can connect to Loki, but Loki has not received any logs from promtail.
+This can have several reasons:
+
+- Promtail cannot reach Loki, check promtails output.
+- Promtail started sending logs before Loki was ready. This can happen in test environments where promtail already read all logs and sent them off. You have two options:
+  - Start promtail later, e.g., 60 seconds
+  - Make sure new log messages are written after both promtail and Loki have started
+
+## Failed to create target, "ioutil.ReadDir: readdirent: not a directory"
+
+The promtail configuration contains a `__path__` entry to a directory that promtail cannot find.