Description
openedon Oct 13, 2019
Describe the bug
Fluent-bit reports errors when pushing to Loki:
level=error caller=client.go:206 component=client host=cloud-services-qa-loki:3100 msg="final error sending batch" status=404 error="server returned HTTP status 404 Not Found (404): 404 page not found"
To Reproduce
All I did was set up a Helm chart that used loki-stack as a requirement:
requirements.yaml
dependencies:
- name: "loki-stack"
repository: "https://grafana.github.io/loki/charts"
version: "^0.6.0"
values.yaml
fluent-bit:
enabled: true
promtail:
enabled: false
grafana:
enabled: true
prometheus:
enabled: true
alertmanager:
persistentVolume:
enabled: false
server:
persistentVolume:
enabled: false
service:
servicePort: web
Then installed it with a release name
helm upgrade --install some-release-name ./my-chart
Expected behavior
I would have expected loki-stack to configure fluent-bit to correctly push logs to loki.
Environment:
- Infrastructure: Kubernetes (AWS EKS)
- Deployment tool: Helm
Screenshots, Promtail config, or terminal output
If applicable, add any output to help explain your problem.
Additional Notes
If I run helm template then the fluent-bit output configuration is:
[Output]
Name loki
Match *
Url http://some-release-name-loki:3100/loki/api/v1/push
Labels {job="fluent-bit"}
RemoveKeys kubernetes,stream
LabelMapPath /fluent-bit/etc/labelmap.json
LineFormat json
LogLevel warn
If I shell into a pod on the node and make a curl request to that url:
# curl -H "Content-Type: application/json" -XPOST -s "http://some-release-name-loki:3100/loki/api/v1/push" --data-raw '{"streams": [{ "labels": "{foo=\"bar\"}", "entries": [{ "ts": "2018-12-18T08:28:06.801064-04:00", "line": "fizzbuzz" }] }]}'
404 page not found
However the /ready and /metrics endpoints are there
# curl "http://some-release-name-loki:3100/ready"
Ready
According to the helm lock file, loki-stack is at version 0.17.1
and according to the pod info, the deployed container images are
some-release-name-fluent-bit-loki-w2tnw: grafana/fluent-bit-plugin-loki:0.1,
some-release-name-grafana-59f6f55ddc-rpcmm: grafana/grafana:6.4.1,
some-release-name-loki-0: grafana/loki:v0.3.0,
Additional Notes 2
It looks like the API endpoint loki/api/v1/push was just added 19 days ago by #1049 and is not yet tagged in any version of Loki. I'm guessing this is due to mismatched releases.
It looks like loki-stack 0.17.1 has the updated API path for fluent-bit that is in master, but still uses loki 0.3.0 which didn't have that endpoint yet. 😞
This is broken starting in loki-stack 0.16.5 which seems to be the first version that supports fluent-bit