Rolling file is not flexible

[jods4]

I was looking into moving our code from NLog to Serilog, but the rolling file sink is severly limited in comparison and probably a showstopper for us. (Even considering the alternative rolling file package)

First there are very few options, like choosing the period for a date-based rolling. E.g. having monthly files for a log that has few activity and long retention.

The lack of concurrent writes support is bad. As noted in the 2014 blog post, there was the IIS recycling issue. The split-file _001 trick is a hack but the worse is that retention is based on file count! So if one day I have issues and my app pool recycles like crazy, I'm going to loose all my log files! This doesn't seem ok to me.

I am tempted by all the cool Serilog structured logging features, but the rolling file sink is one (of two) blockers for me. It would be nice to port file target of NLog to Serilog, because it has plenty of options.

[merbla]

Hi @jods4,

I am not very familiar with NLog anymore. Do you have a compete list of features you are interested in or somewhere you can point me to on the NLog doco?

Thanks!

[merbla]

Just saw this serilog/serilog#820 (comment) to add to the list

[jods4]

@merbla For my use cases, the blockers are:

• Support iis recycling;
• Never delete an archived file before the retention period;
• More flexible file config (e.g. by month);
• Optional non-buffered writes;
• Fatal failures when a logging is not successful.

More generally, NLog has tons of options itself, several interesting wrapper targets (async, buffering, fail-over, and more). Here's just the file target config: https://github.com/nlog/NLog/wiki/File-target

[rdavisunr]

I would really like to see support for "Sized-based file archival" (see the section in that NLog link from @jods4. I guess this is similar to what the authors of the alternative rolling file sink tried to achieve, but I don't think they support "maxArchiveFiles". So, I would be looking to specify a size limit, roll when it hits that limit and then keep a maximum of n files around. Seems like the semantics of the existing retainedFileCountLimit would work for specifying the max...

[merbla]

There is some work over at https://github.com/serilog/serilog-sinks-async that may be of interest as well.

[nblumhardt]

Thanks all for chiming in :-)

@jods4 @rdavisunr just out of curiosity, are you guys using the file-based targets solely in your applications at the moment? Serilog came a bit later to the party, so a lot of work over here has been focused on supporting network-based log collectors, so for many users the file sinks are closer to a fallback target than the primary logging mechanism. Any details you can share about the kinds of apps you're building, and the industries they're in, would be really useful.

I am on board with the idea of improving file logging support here; serilog/serilog-sinks-file#7 was recently opened as a proposal to deal with multi-process logging scenarios, and I think either that approach or one like it would be an awesome addition to the file sink (and thus the rolling file). Assistance on that ticket would be greatly appreciated (and fun - it's a cool OS API to hook into 😉).

To keep the effort of maintaining the core sinks manageable I'd like to avoid expanding File and RollingFile much beyond what we currently have. They serve the role of simple file-based logging well for many people with a minimum of complexity. But if the "alternate" rolling file is still leaving scenarios on the table I'd be interested in working on yet-another alternative that addresses all of the requirements listed in this thread. I am not confident I have the bandwidth currently to get all the way there myself. The best way to kick this off, if anyone is keen, would be to spike something up that we could iterate on in a new repository (FlexibleRollingFile?). I'm able to put effort in on it if someone else can take the lead.

[jods4]

@nblumhardt My situation is that I'm looking at Serilog mostly because I want to get into structured logging and investigate Seq, which looks wicked cool.
That being said, there is no way my current customer will agree to completely drop the file stuff. So we would certainly have files in parallel for some logs.

[rdavisunr]

@nblumhardt - My industry is casino gaming. Many of our services and applications can take advantage of network-based sinks. However, we sometimes deploy services to isolated hardware where network connectivity is limited or non-existent. Disk resources are more constrained in these cases as well. Thus, we have a need for file-based logging that can be capped in size.

The current rolling sink doesn't work too well for that case because, (if memory serves) new events are truncated if the max size is exceeded before the end of the day rollover occurs. Increasing the max file size to a large number is one possibility to alleviate that, but it comes with drawbacks. Namely, large files are harder to deal with and our testing showed high CPU utilization when large files are rolled at the end-of-day.

Perhaps the more esoteric features from other loggers would be a good fit for a new FlexibleRollingFile sink. But, IMO, size based rolling should be core functionality in the existing RollingFile sink

I wish I could dedicate some time to helping out with this, but my bandwidth is also limited at this time. Great work all around though, Serilog and Seq are great products!

[nblumhardt]

@rdavisunr thanks for all the info Rob; we'll keep you posted!

[nblumhardt]

Just a quick update on a couple of items from @jods4 's list:

Support iis recycling;

Multi-process shared files has just landed in the dev branch of the underlying file sink (serilog/serilog-sinks-file#14); #15 will build on that to add rolling file support.

Never delete an archived file before the retention period;

The item above mitigates the common scenario where, using date-based rolling, this could occur.

Optional non-buffered writes;

The buffered parameter is already exposed on WriteTo.RollingFile() and defaults to false. Note this will flush writes through to the OS page cache (to survive process crashes) not necessarily to the physical hardware (to survive power events, which is tricky and hideously slow).

Fatal failures when a logging is not successful.

This just landed in Serilog's dev via AuditTo in serilog/serilog#826.

The feedback on this ticket has been really valuable, I don't think the ticket itself should stay open at this point however as it's not one we can act wholesale on in the near future. In particular, I think putting some of our energies behind supporting https://github.com/BedeGaming/sinks-rollingfile would be a good way to make sure the size-rolling case is covered.

Closing for now, but feel free to jump back in :-) Thanks again, everyone!