Documentation, examples?

[setevoy2]

Hi!
First of all - thanks for the Loki - really great.
But still few questions, please.

1. "federation" analog?

Is there any way to configure Loki's instances in the same way as Prometheus uses its federation ability?

I.e. - I have a Stage environment, with 3 AWS EC2 instances running.
On each EC2 - I have Prom's exporters running + Prometheus itself to collect metrics from local exporters.
Then Prometheus exposes them via its :9090/federation to the central Prometheus instance.

So I'd like to make the same with Loki - collect logs with promtail and Loki on some environment's instance, and pass them to the "central" Loki on the host with Prometheus server and Grafana.

Is it possible? Can't see any docs about such approach, unfortunately.

UPD: "Solved" this: Loki and Grafana runs on one host, promtail "worker" - on another, and just has url: http://loki.example.com:3100/api/prom/push.

2. Loki's storages

Found only one doc where storages configuration was mentioned - here.

Can I use local filesystem instead of AWS S3? I think so - just map new volume via Docker like:

  loki:
    image: grafana/loki:master
    volumes:
      - /some/local/storage/:/tmp/loki/chunks/

Is it correct? And the same for indexes.

And is there any way to use MySQL/Postgres etc as chunks storage?
And is there any way to use something else instead of DynamoDB etc as indexes storage?

3. Documentation, examples?

And finally - the main question... Docs :-)

Where can I see storage_config description, available options etc for the Loki, for example? Or - scrape_configs for the promtail? The only way I found since yesterday was an NGINX's "example" here.

Or it is absolutely similar to the Prometheus?

Thanks in advance and sorry for posting all question to the same issue.

[base698]

Would be nice to get some promtail examples. I was unclear that was even the log agent. Can you pull labels out of the log files themselves? Or would that require preprocessing?

[yubozhao]

@setevoy2 Traditional database is not a good solution for chunk storage, typically cloud object storage is better for this task. You can use local file(boltdb) as index table, or Google's big table. Any reason you are not looking for DynamoDB?

[setevoy2]

@yubozhao Yah, I agreed about traditional databases. Just wondering if Loki can (will can) deal with something else as storages.

[Bfoster-melrok]

I want to second the request for better documentation and examples.

I've read through this blog post about loki's scalability but cant any documentation on how to actually achieve this.
documentation around using s3 and dynamodb are great, but what about users using clusters not on aws? I'd love to use my own local object store and cassandra(this is mentioned in the blog post) for my storage if possible.
I know there is the "our production setup" Doc, but not everyone likes or uses ksonnet, and heptio isn't even working on it anymore since they were acquired by vmware.

[yubozhao]

hi @Bfoster-melrok I agree with more documentations. I am not very clear what documentations you are looking for. From what you wrote, I think you are looking for :

1. a deep dive of how Loki achieve scalability
2. non k8s setup for Loki/Promtail, and with different storage options
3. a different deployment setup? I assume for k8s.

Is that correct?

[Bfoster-melrok]

@yubozhao Almost,

1. an example of setting up a production loki cluster (including distributor/ingester/querier) inside kubernetes without ksonnet
2. documentation on promtail and loki config options (including different storage options)

[yubozhao]

For example setup without using ksonnet, you could use helm --dry-run to print out the raw YAML files. That should be a good starting point. You can also use helm as well.

Documentation for configuration is definitely lagging behind. ATM you would have to dig in the code and find what you want.

[Bfoster-melrok]

I have tried the helm chart and it only seems to be able to do a single loki instance setup with local storage

[Sniliev]

It would be nice to get a bit more information about how Loki scales. As far as I understand you are supposed to run multiple instances of Loki. Each of these instances taking on one(or possibly more) of the roles of distributor, ingestor or querier. However I cannot find how this should be achieved.

[rrfeng]

Need docs

[sciffer]

It's worse than that, Loki cluster requires also hashicorp consul cluster + an AWS storage (S3 or dynamodb). S3 can be emulated using multiple opensource solutions(like minio, edgefs, etc..) - but not really tested to the best of my knowledge. Anyhow - that means lots more overhead than should be!!
Docs are definitely missing, but it should be fairly simple to figure it out from the ksonnet docs if you are familiar with kubernetes manifest files.

[yubozhao]

If you don't have heavy workload, I would advice not use the consul to setup a multi instance Loki. As far as S3/Dynamo those stuff, refer to the upstream support. Those open source projects are not part of the support AFAIK.

[rrfeng]

Detailed docs help users try Loki easy and developers support the development.