Implement CRC control

[Gsantomaggio]

Implement CRC32 check before creating the Chunk

I can't find a standard library. A lot of people use https://github.com/damieng/DamienGKit

Any thought?

[acogoluegnes]

Nothing in the standard SDK?

[Gsantomaggio]

Can't find anything.
There is also:

• https://github.com/force-net/Crc32.NET

since we will need some compress library there is also:

• https://github.com/icsharpcode/SharpZipLib

[kjnilsson]

I wouldn't worry too much about the CRC check if there isn't a function available in the standard library. best to try to avoid taking dependencies where ever possible and instead allow users to plug in stuff like compression.

[acogoluegnes]

CRC calculation can matter at high throughput and with "large" messages. I started to see unbalanced publishing/consuming rates with stream perf test with reasonably sized messages. The consuming rate was lagging and it happened to be the CRC calculation on the client side. The Java client was using the JDK CRC library but with a non-optimal method, to keep compatibility with Java 8.

I ended up using some reflection at runtime to detect if a Java 11 faster CRC method was available and the unbalance went away.

Needless to say the unbalance does not appear if you don't do any CRC at all (it's possible with the Java client as the CRC strategy is pluggable).

[Tornhoof]

In .NET 6 there is System.IO.Hashing:
https://www.nuget.org/packages/System.IO.Hashing/6.0.0
it includes CRC16, CRC32

[PaoloLaurenti]

In .NET 6 there is System.IO.Hashing: https://www.nuget.org/packages/System.IO.Hashing/6.0.0 it includes CRC16, CRC32

Unfortunately this is not available on the standard library of .NET 6.

Is it OK to include the package mentioned by @Tornhoof ?
Or do we prefer to provide only a way to add a custom implementation with no checksum calculation available inside the library (as @kjnilsson suggested if I understood correctly)?