Fluent log shipping app

Fluent log shipping app is a managed app used to help customers forward their logs to any supported storage backends. It use fluent-bit DaemonSet, a lightweight log collector used to collect and forward containers and audit logs your chosen output

Requirements

• You can install only one release of this chart per kubernetes cluster
• By default, no forwarding is active so make sure you check configuration document before deploying it in your cluster.

Currently supported storage backends

AWS

• CloudWatch
• S3

Azure

• Log Analytics

Others

• Elasticsearch

Installation

The log shipping app is built to be installed in AWS or Azure.

Sample command for installing it on AWS with cloudwatch enabled:

helm install --namespace logging giantswarm-playground-catalog/fluent-logshipping-app --set fluentd.aws.cloudWatch.enabled=true

Exported logs

The app currently exports the following logs:

Log type	Location	Format
Container Logs	/var/log/containers/*.log	json
Kubernetes Audit Log	/var/log/apiserver/audit.log	json
SSH Access Logs	syslog	^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$

Configuration

Configuration options are documented in Configuration.md document.

AWS

Prerequisites

• When using Cloudwatch or S3 a new role has to be created upfront which allows pushing of the logs to the backend(s). More info about permissions in official docs (S3, CloudWatch).

Configuration

• For clusters running kiam you must enable the values.aws.kiam configuration option.

Compatibility

Tested on Giant Swarm release 11.0.0 on AWS and Azure (Kubernetes 1.16.3).