Feature to roll on file size and date

[akshetty9]

Question : Is there any plan to include Rolling file based on size/date in this sink. AlternateRollingFileSink does not have option to specify the retainedFileCount and there is no development/pr happening on that sink. I am curious to know if this will be included here.

Update : AlternateRollingFileSink now supports Retenation policy and also ITextFormatter if anyone is looking for the same

[hmvs]

Will author accept PR if i will implement that?
Will be nice to have rolling by size. Now it just stops writing to file.

[nblumhardt]

This particular request has come up enough times now that I think it's time we made some progress on it 👍

From a PR, I think we would need:

• A fairly surgical/low-impact approach, unless a big gain can be found in further refactoring (e.g. to "roll policies")
• Consistent with the current file naming approach: appending _001, _002 etc as the files roll on a particular day
• Some basic benchmarks showing performance consistent with the current implementation for the current configuration defaults

@hmvs are you interested in taking a shot at this? Also CC'ing @jose3m, @Wedvich who have worked in this area recently and may have some additional points to consider.

Cheers!

[hmvs]

Let's imagine:

• rolling by size is enabled
• maximum numbers of files is set to 5
• limit is 10k per file

What should happen when we have more than 50k of logs?
Variants:

• 001 is overwritten, then 002 is overwritten... (content should be truncated)
• Just stops writing (i don't like that)
• 001 is deleted and start writing to 006
  • Problems in case when we will have more than 999
    • Should we increment it to _1001?

Yep, i am interested. Will do in my spare time.
Cheers.

[nblumhardt]

Great, thanks for the help.

I think the answer to the first question above should be "001 is deleted and start writing to 006".

The second question - hmm. _1001 is ugly because it breaks the lexicographic sort order, but it's probably the least surprising. Thoughts?

(> 1000 log files per day is probably a rare situation, I'd suppose.)

[hmvs]

Hmmm. Dilemma :)

[hmvs]

@nblumhardt i had a quick look into code and didn't find a way how we can determine that file is full and it's time to roll.
I think we should make changes to FileSink package first. For example we can make an interface IFileSink or ISinkWithSize or something similar.
public interface ISinkWithSize{ long CurrentSize {get;} }
After it we can make changes to this package. Without it we could hit performance badly by checking filesize everytime.

Another way is to throw an exception from FileSync when file is full. I am new to Serilog but i think it's not good.

[nblumhardt]

Agreed on the exception, too much interference with other exception-related debugging that might be going on.

The file sinks (two classes, IIRC) both support size limits already, so exposing this via an interface seems reasonable. Could alternatively shape it as ISizeLimitedFileSink.SizeLimitReached so that the logic that determines whether the sink will accept an event, and the logic that determines whether an event will be sent to the sink, are the same?

The IFlushableFileSink interface was added for a similar purpose just a little while ago - the downstream changes here were a bit inconvenient due to the lack of union types (ISink + IFlushableFileSink) but otherwise it came out fairly cleanly.

[hmvs]

Hmm. We can go even further and make it more generic. So we can call it
something like ILimitedSink or similar. So that any sink that can decline
messages can inherit it

IRejectingSink... ?

[nblumhardt]

We could, though I don't think there's another (non-file) place we'd use this at the moment. Making a general interface, not specific to the file sink, would mean adding it to Serilog.dll which would only really make the cut if multiple packages need it.

This was the reason why IFlushableFileSink is not IFlushableSink - using File in the name means that if, in the future, we decide to add the more general interface, the names won't collide :-)

[hmvs]

Ok. We can go with ISizeLimitedFileSink

[optical]

How do we feel about also supporting automatic archival (compressing) of rotated/rolled logs? I figure this is likely to be a useful feature for this sink and generally goes along with the idea of rolling based on file size. Theres probably a lot more to discuss, so might be more appropriate to discuss in a separate issue alongside this and #26 since it can be done after the work has been done here.

[narendrachava]

Is the development still active to implement this feature ??