Promtail kubernetes host log & drop namespace

[prasenforu]

I am running Loki with promtail in kubernetes platform.

Promtail running as a daemonset and promtail configuration as follows ..

    scrape_configs:
      - job_name: kubernetes-pods
        kubernetes_sd_configs:
        - role: pod
        relabel_configs:
        - source_labels:
          - __meta_kubernetes_pod_node_name
          target_label: __host__
        - action: drop
          regex: ^$
          source_labels:
          - __meta_kubernetes_pod_label_name
        - action: replace
          replacement: $1
          separator: /
          source_labels:
          - __meta_kubernetes_namespace
          - __meta_kubernetes_pod_label_name
          target_label: job
        - action: drop
          regex: default
          source_labels:
          - __meta_kubernetes_namespace
        - action: replace
          source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - action: replace
          source_labels:
          - __meta_kubernetes_pod_name
          target_label: instance
        - action: replace
          source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container_name
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - replacement: /var/log/pods/$1/*.log
          separator: /
          source_labels:
          - __meta_kubernetes_pod_uid
          - __meta_kubernetes_pod_container_name
          target_label: __path__
      - job_name: kubernetes-pods-app
        kubernetes_sd_configs:
        - role: pod
        relabel_configs:
        - source_labels:
          - __meta_kubernetes_pod_node_name
          target_label: __host__
        - action: drop
          regex: .+
          source_labels:
          - __meta_kubernetes_pod_label_name
        - action: replace
          replacement: $1
          separator: /
          source_labels:
          - __meta_kubernetes_namespace
          - __meta_kubernetes_pod_label_app
          target_label: job
        - action: drop
          regex: default
          source_labels:
          - __meta_kubernetes_namespace
        - action: replace
          source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - action: replace
          source_labels:
          - __meta_kubernetes_pod_name
          target_label: instance
        - action: replace
          source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container_name
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - replacement: /var/log/pods/$1/*.log
          separator: /
          source_labels:
          - __meta_kubernetes_pod_uid
          - __meta_kubernetes_pod_container_name
          target_label: __path__

I have a two question?

Q1# How can I capture kubernetes host log (/var/log/messages) along with hostname and what will be config file definition?

I checked with https://github.com/grafana/loki/blob/master/docs/promtail-examples.md

But looks like its for static configuration.

scrape_configs:
 - job_name: system
   pipeline_stages:
   - docker:
   static_configs:
   - targets:
      - localhost
     labels:
      job: varlogs
      host: yourhost
      __path__: /var/log/*.log

Q2# How can I DROP all logs from particular namespace (default)

I tried with following but no luck, not sure if I missing anything ...
I added below text in job "kubernetes-pods" & "kubernetes-pods-app"

        - action: drop
          regex: default
          source_labels:
          - __meta_kubernetes_namespace

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[minhdanh]

@prasenforu Could you have your questions answered? I'm wondering the same.

[zhushilu]

Take a look at this article. https://itnext.io/grafana-logging-using-loki-45665916aec9

[prasenforu]

I followed that URL but not working.

Finally I moved to fluent-bit

[cyriltovena]

Q1 - We don't add the host automatically but you can use external label flags see #510

Q2 - Labels after relabeling are accessible in pipeline stage, we support dropping via the match stage so that should work now.

[imranrazakhan]

@prasenforu how you able to exclude namespace in fluent-bit?

[cloudcafetech]

@imranrazakhan

in fluent-bit conf add in INPUT section

    [INPUT]
        Name           tail
        Tag            kube.*
        Path           /var/log/containers/*.log
        Exclude_Path   /var/log/containers/*_monitoring_*.log,/var/log/containers/*_logging_*.log,/var/log/containers/*_kube-system_*.log,/var/log/containers/*_kube-node-lease_*.log,/var/log/containers/*_kube-public_*.log
        Parser         docker
        DB             /run/fluent-bit/flb_kube.db
        Mem_Buf_Limit  5MB

[cloudcafetech]

@cyriltovena

Q2 - Labels after relabeling are accessible in pipeline stage, we support dropping via the match stage so that should work now.

Can you please help me to get sample configuration for dorp namespace

[eranbetzalel]

Drop all namespaces but "default" on promtail

    - match:
        pipeline_name: "drop-all"
        selector: '{namespace!="default"}'
        action: drop

This is a working example.

[aroldobossoni]

Drop all namespaces but "default" on promtail

    - match:
        pipeline_name: "drop-all"
        selector: '{namespace!="default"}'
        action: drop

This is a working example.

This setting only works on newer versions of Promtail in Chart version: 3.6.1 Promtail version: 2.2.1 does not work.

[mhoyer]

It might be helpful to also point out that there is a probably more efficient way to filter per namespace. Thus, this only works when it's about an include list of valid namespaces.

There is the kubernetes_sd_config section which enables you to configure namespaces. Promtail will then only discover and tail on log files of pods/containers for that given set of namespaces. Ergo, less work (IOps) to do for Promtail.

scrape_configs:
- job_name: kubernetes-pods
  pipeline_stages:
    - cri: {}
  kubernetes_sd_configs:
    - role: pod
      namespaces:
        names:
          - default
          # - another-namespace

[mhoyer]

ℹ️ With my recent comment I just trapped into the issue mentioned here: #808 (comment)

Filtering log files by e.g. just the "default" namespace led to zero matching files. Thus, nothing to be tailed by promtail. Which resulted in a broken readiness check by Kubernetes. I managed to simply make this ready check pass again by setting server.health_check_target: false (ref #1919).