-
Notifications
You must be signed in to change notification settings - Fork 5.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Telegraf tail & logparser should handle deleted & newly created files #1829
Comments
+1 on this one. |
It's working as intended. Jetty should be configured so that it always writes to the same log file and |
@dgnorton Unfortunately this is not something that is out of the box standard to jetty. Please refer to here: Either way, logparser should detect new files. There is not only 1 way to accomplish 1 task. |
@akan1979 what happens if |
This seems to be a problem when a file is (re)moved as well, e.g. given this config:
I see this behavior:
On the plus side, truncation detection seems to work, e.g.
... but I suspect more than a few logging tools (e.g. logback) don't support rolling by truncation. |
Internally telegraf has to spinup a thread for each file that it needs to tail. Currently it only does that once at startup, which is why deleted & created files are not handled. I think it's worthwhile to handle these two cases in the tailer & logparser so I'm going to reopen this issue and put it into the 1.3 milestone. |
Sounds good. FWIW, it looks like this lib solves addresses these problems. I haven't dug into the source yet, but it has a lot of ⭐️s so must be good :) |
@njhartwell what lib are you talking about? |
Sorry, forgot to paste it https://github.com/hpcloud/tail |
we are already using that library, it doesn't solve all of this |
Is this still slated to go into the next release of Telegraf? (1.3) Other than restarting the Telegraf service after each log rotation, are there any other workarounds you can suggest for this? We are trying to replace Logstash (which handles this case) with Telegraf, we can not proceed until this issue is resolved. |
Our particular use case is we have Telegraf pointing to a test machine where we periodically reinstall our software with newer versions. The deployment is fairly simplistic, we shutdown our servers, rename the old deployment directory to backup.directory (which includes the log file we are monitoring), install the new software, and bring our services back up. We can add a restart of the Telegraf service to this process until 1.3 is released. Interestingly enough when I went to look into this this morning, I noticed that Telegraf was still holding on to a deleted file, our software had been reinstalled twice since I had last looked so the file was renamed into the backup directory in the first round, and deleted in the second.
Note the deleted Than could be another issue you may want to investigate. |
👍 The tail plugin Readme explicitly states that it should behave like a
but this is not the case as it keeps open the renamed/deleted file. |
this was closed by #2141, it will be available in 1.3 |
I tried testing this today using the same technique that was listed at the beginning of this issue. I used the latest nightly build of Telegraf. Here is what I found: telegraf version:
Configuration File
Test Script
Test Execution
Conclusion Assuming that this PR is actually in the build I used, it does not appear that this fix accomplishes what was requested e.g. discovery of newly created files with the same name of the original file. We are currently using LogStash to parse our log that has this behavior. This is the last remaining issue we have that will allow us to use a pure TICK stack. |
I can duplicate your report @mottati. It seems that recreated files are not detected, but truncated files are. There is also a bug where if the file is truncated with a file of the same size it is not detected. |
I merged a fix for the problem into our tail fork. |
@danielnelson Thanks! Do you know when this fix will make it into a released version of Telegraf? |
The fix for this was in the 1.3.0 release, though I have heard that there are remaining issues. #2847 |
This still seems like an issue for [inputs.tail] for |
Yes, got this bug at telegraf 1.17.0 with [inputs.tail] plugin on ubuntu 20.04 |
I have the same issue when using Telegraf (1.17.0)
After restarting the telegraf service all works fine again until the next log rotation. As workaround I restart the telegraf service after the log rotation, but thats not a clean solution. |
Bug report
Relevant telegraf.conf:
[[inputs.logparser]]
files to tail.
files = ["/u0/log/jetty/*.request.log"]
Read file from beginning.
from_beginning = false
System info:
Telegraf 1.0.0
Ubuntu 14.04.4
Jetty
Steps to reproduce:
Expected behavior:
Telegraf logparser picks up new file on the same pattern defined in the telegraf.conf
Actual behavior:
Telegraf does not pick up new rotated file automatically
Additional info:
Here's an fuser output of the log directory. The 7692 pid is Telegraf. It was started on 9/28.
/u0/log/jetty/2016_09_23.request.log: 7692
/u0/log/jetty/2016_09_24.request.log: 7692
/u0/log/jetty/2016_09_25.request.log: 7692
/u0/log/jetty/2016_09_26.request.log: 7692
/u0/log/jetty/2016_09_27.request.log: 1081 7692
/u0/log/jetty/2016_09_28.request.log: 1081 7692
/u0/log/jetty/2016_09_29.request.log: 1081 8769
The 2016_09_29.request.log was created by the 8769 process with is jetty. notice that 7692 does not hold a handle of that file.
Interestingly the pid 1081 belongs to logstash-forwarder, which is able to pick up the newly created file automatically.
The text was updated successfully, but these errors were encountered: