Consider combining directory_monitor, file, tail plugins

[reimda]

@Hipska noticed that there is functional overlap across these plugins. It might make sense to combine them in some way, either sharing code or consolidating functionality then deprecating/removing one or more of the plugins.

Plugin / Feature	directory_monitor	file	tail
Read newly appeared files	✔️	✔️	✔️
Parsing	line-by-line, complete-file	complete-file	line-by-line
Move/Remove file after parsing	✔️	❌	❌
Supports gzipped file	✔️	❌	❌

[powersj]

next steps: review as part of 2.0 milestone

[Hipska]

My preference on this would be to merge directory_monitor functionality into file and then deprecate directory_monitor. The tail plugin can stay as is.

Other scenario could be to deprecate both file and tail plugin in favour of directory_monitor.

[Hipska]

@powersj next steps?

[powersj]

I think we are ok removing older plugins that have long been deprecated, but removing either directory_monitor, file, or tail will break a lot of users and cause a lot of pain for them. So I do not see us removing any of these.

Does that only then leave the gzip support and moving/delete a file?

[Hipska]

We could start with adding the functionalities to 1 plugin and marking the others as deprecated..

I think if we do it smart, we can even make the others ones a link/rename, so it is seamless..

[Hipska]

@powersj your thoughts?

[powersj]

We could start with adding the functionalities to 1 plugin and marking the others as deprecated..

I am not entirely sure what we as maintainers and our users gain from doing this. Yes, there is duplication but since these plugins are so widely used it will impact a wide group of users.

Doing this will ultimately lead to lots of questions from users about moving over or we find out that there is different or unexpected behavior that users were depending on which suddenly disappears by consolidating the three plugins.

As I mentioned in my earlier comment, I could see us adding features like the gzip support and moving files possibly if users would find it useful.

[powersj]

Sven and I met to further discussion this again and agree this is not something we would do as it would interrupt existing users and only cause confusion.