Support Compression of Rolled Log Files

[lajoie]

It would be nice if the File sink supported compressing files as they're rolled out in order to help save on disk space.

[nblumhardt]

Thanks for the suggestion! We don't have any plans to attempt this at present - the sink is heavily text-oriented. It would be interesting to see if there are any extension points we could provide that would enable it, but I suspect they would need to be quite intricate.

[cocowalla]

If I'm not mistaken, all the actual writing is centralised in SharedFileSink - if we put aside rolling for the sake of simplifying this discussion, a compressed output can be achieved by simply wrapping the stream in a GZipStream and BufferedStream.

A couple of notes:

1. You have no control over when data is flushed to disk - flushing is handled by the GZIP stream
2. In the event of a crash, using compression is still fairly 'safe' - as GZIP is a streaming algorithm, you're not going to lose much data. Recovery is a simple gunzip < mylog.gz > mylog.recovered.txt

An alternative is to have some kind of hook for when files are rolled, allowing the application to do something like ZIP the file. But I'm not sure if there is any 'precedent' for hooks in any of the other Serilog sinks? Personally, in any case I'd favour a streaming implementation for its more predictable CPU and disk usage.

[cocowalla]

@nblumhardt have you any thoughts on this?

[nblumhardt]

Thanks for the nudge! Personally, and this might not be an informed point of view :-), I don't find compressed-file logging appealing. On Windows at least, it's likely easier and more ergonomic just to mark a folder as "compressed" and let the OS/filesystem deal with it, if you're stuck with low-disk-space local file logging.

With disk being at least as cheap to expand as CPU, it seems like adding storage is also probably a better bang-for-the-buck solution.

Adding something like a Stream-to-Stream extension point is an interesting possibility, though; by plugging in a factory from Stream to Stream, wrapper streams could be created, and these might be useful for implementing other cases like headers/footers (write header if length == 0 on wrapping, write the footer inside Dispose(), etc.), as well as this one.

There are likely quite a few gory details to figure out, though - it'd require some dedicated time and effort. If anyone's interested in enumerating the use cases, how they'd be achieved, and the rough API/internal design, it'd be a worthwhile discussion.

[cocowalla]

My particular use case is for logging audit logs straight to an archive - the expectation is that these will be needed only rarely, so it makes sense to compress them from the start. I also need it to work on both Windows and Linux. While storage is cheap, we also expect to log a lot of data, and with excellent compression ratios achievable with almost no impact on CPU or memory, it's hard to think of a reason not to compress files.

Have you ever used Windows compressed folders before? I ask, because if you had, you likely wouldn't have suggested it ;) They have unfortunately always had terrible compression ratios and awful performance.

If I get some time, I'll come up with a couple of suggestions, for both the streaming (some kind of Stream extension point) and static (likely some kind of rolling callback) scenarios.

[nblumhardt]

this might not be an informed point of view

Don't say I didn't warn you ;-). We use Snappy compression in Seq 👍 .. I should have further qualified:

if you're stuck with low-disk-space local file logging

It's been a long time since I thought of a local log file as anything more than a temporary destination while logs make their way off to a central archive, where compression is pretty normal. Auditing directly to the archive is an interesting twist on this. Will be interesting to see what you come up with!

[AdamUdovich]

I have started to work on resolving this issue with @lajoie for our specific use case.

We are basing the compression choice parameters off Java's logback library.

As of now I am planning on checking parameters at the end of a roll period for compression, then compressing the log file using dotnet's FileZip to create a zip for each file. Once the compressed version is created, the original file will be deleted. I am working on gzip compression as well.

Do you have any suggestions related to this method or things I should consider?

[cocowalla]

@AdamUdovich I've just got round to thinking about this again too. I'm interested to see what you do, and was wondering if it's extensible?

In particular, it would be nice to easily switch out the compression algorithm - for example by specifying your own implementation of an IFileRolledHandler (or even just an Action<string>, where the string is the filename). As a bonus, this would also allow performing arbitrary actions on rolled files, such as moving them to another folder, encrypting them, signing them or writing them to read-only media.

@nblumhardt what do you think about this? If we put aside the option of streaming compression and focus on compressing rolled files, these are the options I had been considering:

1. Build support for popular compression algorithms (e.g. ZIP, GZIP), with the ability to configure compression parameters
2. Build a user-extensible way to plugin any compression algorithm (or indeed any action on a rolled file), such as providing an implementation of an interface

[AdamUdovich]

@cocowalla I do not currently have plans to make the compression system extensible, but I can check with @lajoie to see if it is something we would want to work on.

One question you might be able to answer is how unit tests should be structured for testing the rolling compression. Do you have specific testing goals for a basic Zip and GZip system that serilog would like to see?

[cocowalla]

@AdamUdovich I think there are 2 parts:

1. Test that the compression code produces compressed files. So you're checking for the existence of the expected file, and checking that it has the correct magic byte sequence for whichever algorithm is in use

2. Test that compressed files are created when files are rolled (I guess you also optionally delete the plaintext file?)

There are existing tests for file rolling, so it makes sense to build on them.

Also, just so it's clear: I don't own this repo, I'm just a Serilog user who has written a few sinks, and would like compression to be added :)

@nblumhardt, would be great to get your thoughts on compression, especially with regards to the last few comments here?

[AdamUdovich]

Ok thank you @cocowalla ! I do not have an option to not delete non compressed files a the moment. I can check if this is something I will build in or if a repo admin says it should be included.

[cocowalla]

@AdamUdovich Linux' logrotate is probably the most well known log rotating tool, and it has this feature. I can't think why you wouldn't want to delete the original if you now have a compressed copy? :)

[AdamUdovich]

@cocowalla do you think Seri log would like to see functionality based on logrotate for this type of compression fucntionality? Is this a question more for @nblumhardt on what the repo would like to see done structure wise in general?

[cocowalla]

@AdamUdovich yes, I'd consider the option to delete compressed files as essential for any log rolling feature.

Would be good tho to get some input from @nblumhardt on the questions of extensibility, algorithm support and the like.