This article explains how to install stable versions of fluent-package
rpm packages, the stable Fluentd distribution packages maintained by Fluentd Project and calyptia-fluentd
which is maintained by Chronosphere after its acquisition of Calyptia.
Fluentd is written in Ruby for flexibility, with performance-sensitive parts in C. However, some users may have difficulty installing and operating a Ruby daemon.
That is why Fluentd Project provides the stable distribution of Fluentd, called fluent-package
(formerly known as td-agent
). The differences between Fluentd and fluent-package
can be found here.
Fluentd is written in Ruby for flexibility, with performance-sensitive parts in C. However, some users may have difficulty installing and operating a Ruby daemon. And td-agent
is still seated on Ruby 2.7 due to compatibility reasons and Ruby versioning policy, calyptia-fluentd
uses Ruby 3 instead of Ruby 2.7 for now.
That is why Chronosphere (formerly Calyptia) provides the alternative stable distribution of Fluentd, called calyptia-fluentd
. The differences between td-agent
and calyptia-fluentd
are bundled and running Ruby versions for now.
This installation guide is for fluent-package
v5 and calyptia-fluentd
v1. fluent-package
v5 and calyptia-fluentd
use fluentd v1 in the core. See fluent-package-v5-vs-td-agent or td-agent-v2-vs-v3-vs-v4 for the comparison and supported OS.
{% hint style='info' %} NOTE:
- About deprecated Treasure Agent (td-agent) v4 (EOL), See Install by RPM Package v4.
- About deprecated Treasure Agent (td-agent) 3 will not be maintained anymore, see Install by DEB Package v3.
fluent-package
will be shipped in two flavors - normal release version and LTS (Long Term Support) version. See Scheduled support lifecycle announcement about Fluent Package about difference between this two flavors.- If you upgrade from
td-agent
v4, See Upgrade to fluent-package v5. - Do not directly upgrade from v3 to v5. Such a workflow is not supported. It causes a trouble. Upgrade in stages. (v3 to v4, then v4 to v5) {% endhint %}
Please follow the Pre-installation Guide to configure your OS properly.
It is highly recommended to set up ntpd
on the node to prevent invalid timestamps in the logs. See Pre-installation Guide.
NOTE: If your OS is not supported, consider gem installation instead.
Download and execute the install script with curl
:
curl -fsSL https://toolbelt.treasuredata.com/sh/install-redhat-fluent-package5-lts.sh | sh
curl -fsSL https://toolbelt.treasuredata.com/sh/install-redhat-fluent-package5.sh | sh
Executing this script will automatically install fluent-package
on your machine. This shell script registers a new rpm
repository at /etc/yum.repos.d/fluent-package.repo
(or /etc/yum.repos.d/fluent-package-lts.repo
) and installs fluent-package
.
We use $releasever
for repository path in the script and $releasever
should be the major version only like "9"
. If your environment uses some other format like "9.2"
, change it to the major version only or set up .repo file manually.
{% hint style='danger' %} Since v5.0.4, RHEL 7 / CentOS 7 is not supported anymore because CentOS 7 has reached EOL (June, 2024). Please consider migrating to another release. {% endhint %}
curl -fsSL https://toolbelt.treasuredata.com/sh/install-amazon2023-fluent-package5-lts.sh | sh
curl -fsSL https://toolbelt.treasuredata.com/sh/install-amazon2023-fluent-package5.sh | sh
curl -fsSL https://toolbelt.treasuredata.com/sh/install-amazon2-fluent-package5-lts.sh | sh
curl -fsSL https://toolbelt.treasuredata.com/sh/install-amazon2-fluent-package5.sh | sh
Use /usr/lib/systemd/system/fluentd
service to start
, stop
, or restart
the agent:
$ sudo systemctl start fluentd.service
$ sudo systemctl status fluentd.service
* fluentd.service - fluentd: All in one package of Fluentd
Loaded: loaded (/usr/lib/systemd/system/fluentd.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2023-08-02 05:35:16 UTC; 41s ago
Docs: https://docs.fluentd.org/
Process: 1901 ExecStart=/opt/fluent/bin/fluentd --log $FLUENT_PACKAGE_LOG_FILE --daemon /var/run/fluent/fluentd.pid $FLUENT_PACKAGE_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 1907 (fluentd)
CGroup: /system.slice/fluentd.service
|-1907 /opt/fluent/bin/ruby /opt/fluent/bin/fluentd --log /var/log/fluent/fluentd.log --daemon /var/run/fluent/fluentd.pid
`-1910 /opt/fluent/bin/ruby -Eascii-8bit:ascii-8bit /opt/fluent/bin/fluentd --log /var/log/fluent/fluentd.log --daemon /var/ru...
To customize systemd
behavior, put your fluentd.service
in /etc/systemd/system
.
NOTE: In fluent-package
v5, the path is different i.e. /opt/fluent/bin
instead of /opt/td-agent/bin
.
The default configuration (/etc/fluent/fluentd.conf
) is to receive logs at an HTTP endpoint and route them to stdout
. For fluentd
logs, see /var/log/fluent/fluentd.log
.
You can post sample log records with curl
command:
$ curl -X POST -d 'json={"json":"message"}' http://localhost:8888/debug.test
$ tail -n 1 /var/log/fluent/fluentd.log
2023-08-02 05:37:29.185634777 +0000 debug.test: {"json":"message"}
Please follow the Pre-installation Guide to configure your OS properly.
It is highly recommended to set up ntpd
on the node to prevent invalid timestamps in the logs. See Pre-installation Guide.
NOTE: If your OS is not supported, consider gem installation instead.
Download and execute the install script with curl
:
# calyptia-fluentd 1
$ curl -L https://calyptia-fluentd.s3.us-east-2.amazonaws.com/calyptia-fluentd-1-redhat.sh | sh
Executing this script will automatically install calyptia-fluentd
on your machine. This shell script registers a new rpm
repository at /etc/yum.repos.d/Calyptia-Fluentd.repo
and installs calyptia-fluentd
.
We use $releasever
for repository path in the script and $releasever
should be the major version only like "7"
. If your environment uses some other format like "7.2"
, change it to the major version only or set up TD repository manually.
For CentOS Stream 8:
# calyptia-fluentd 1
$ curl -L https://calyptia-fluentd.s3.us-east-2.amazonaws.com/calyptia-fluentd-1-centos-stream.sh
For Amazon Linux 2:
# calyptia-fluentd 1
$ curl -L https://calyptia-fluentd.s3.us-east-2.amazonaws.com/calyptia-fluentd-1-amazon-2.sh | sh
calyptia-fluentd
only provides systemd's unit file:
Use /usr/lib/systemd/system/calyptia-fluentd
script to start
, stop
, or restart
the agent:
$ sudo systemctl start calyptia-fluentd.service
$ sudo systemctl status calyptia-fluentd.service
● calyptia-fluentd.service - calyptia-fluentd: Fluentd based data collector for Calyptia Services
Loaded: loaded (/usr/lib/systemd/system/calyptia-fluentd.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2021-05-31 01:37:47 UTC; 4h 38min ago
Docs: https://docs.fluentd.org/
Process: 694 ExecStart=/opt/calyptia-fluentd/bin/fluentd --log $CALYPTIA_FLUENTD_LOG_FILE --daemon /var/run/calyptia-fluentd/calyptia-fluentd.pid $CALYPTIA_FLUENTD_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 1365 (fluentd)
CGroup: /system.slice/calyptia-fluentd.service
├─1365 /opt/calyptia-fluentd/bin/ruby /opt/calyptia-fluentd/bin/fluentd --log /var/log/calyptia-fluentd/calyptia-fluentd.log --daem...
└─1368 /opt/calyptia-fluentd/bin/ruby -Eascii-8bit:ascii-8bit /opt/calyptia-fluentd/bin/fluentd --log /var/log/calyptia-fluentd/cal...
To customize systemd
behavior, put your calyptia-fluentd.service
in /etc/systemd/system
.
Please make sure your configuration file path is:
/etc/calyptia-fluentd/calyptia-fluentd.conf
The default configuration (/etc/calyptia-fluentd/calyptia-fluentd.conf
) is to receive logs at an HTTP endpoint and route them to stdout
. For calyptia-fluentd
logs, see /var/log/calyptia-fluentd/calyptia-fluentd.log
.
You can post sample log records with curl
command:
$ curl -X POST -d 'json={"json":"message"}' http://localhost:8888/debug.test
$ sudo tail -n 1 /var/log/calyptia-fluentd/calyptia-fluentd.log
2021-05-31 06:19:04.415878392 +0000 debug.test: {"json":"message"}
You are now ready to collect real logs with Fluentd. Refer to the following tutorials on how to collect data from various sources:
- Basic Configuration
- Application Logs
- Examples
For further steps, follow these:
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.