Skip to content

Enable setting compression via the KafkaAttribute #175

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: dev
Choose a base branch
from
Open

Enable setting compression via the KafkaAttribute #175

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
466e804
Enable setting compression via the KafkaAttribute
gliljas Oct 31, 2020
8520962
Merge branch 'dev' into compression
shrohilla Dec 13, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -323,6 +323,8 @@ The settings exposed here are targeted to more advanced users that want to custo
|LibkafkaDebug|debug|Both
|MetadataMaxAgeMs|metadata.max.age.ms|Both
|SocketKeepaliveEnable|socket.keepalive.enable|Both
|CompressionType|compression.codec|Output
|CompressionLevel|compression.level|Output

**NOTE:** `MetadataMaxAgeMs` default is `180000` `SocketKeepaliveEnable` default is `true` otherwise, the default value is the same as the [Configuration properties](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md). The reason of the default settings, refer to this [issue](https://github.com/Azure/azure-functions-kafka-extension/issues/187).
**NOTE:** `AutoOffsetReset` default is Earliest. Allowed Values are `Earliest` and `Latest`.
Expand Down
18 changes: 18 additions & 0 deletions src/Microsoft.Azure.WebJobs.Extensions.Kafka/Config/MessageCompressionType.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
// Copyright (c) .NET Foundation. All rights reserved.
// Licensed under the MIT License. See License.txt in the project root for license information.

namespace Microsoft.Azure.WebJobs.Extensions.Kafka
{
/// <summary>
/// Defines the message compression type
/// </summary>
public enum MessageCompressionType
{
NotSet = -1,
None = 0,
Gzip = 1,
Snappy = 2,
Lz4 = 3,
Zstd = 4
}
}
12 changes: 12 additions & 0 deletions src/Microsoft.Azure.WebJobs.Extensions.Kafka/Output/KafkaAttribute.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,5 +139,17 @@ public KafkaAttribute()
/// ssl.key.password in librdkafka
/// </summary>
public string SslKeyPassword { get; set; }

/// <summary>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Properties should have getters, according .NET guidelines.

I can imagine that we might want to log/write the value and not having the setters prevents it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my latest commit I have used one possible implementation. The getters have been reintroduced, in order to comply with guidelines, but the backing fields are still nullable. The values from the attribute are then merged (coalesced) with the values from the global options (can be also done as a one-liner using JsonConvert, among many ways of doing it), and the default value is used if neither supplies it.

The same approach could definitely be used for consumers as well. It may seem a bit convoluted, and there may well be better ways of doing it, but I think it provides maximum flexibility, while maintaining backwards compatibility.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The merging could also look like this:

var mergedOptions = DefaultProducerOptions.CreateNew()
                    .MergeWithNonNullPropertiesFrom(kafkaOptions)
                    .MergeWithNonNullPropertiesFrom(entity.Attribute);

using a simple extension like this:

public static class MergeExtensions
{
    public static T MergeWithNonNullPropertiesFrom<T>(this T instance, T other)
    {
        if (other != null)
        {
            JsonConvert.PopulateObject(JsonConvert.SerializeObject(other, new JsonSerializerSettings { NullValueHandling = NullValueHandling.Ignore }), instance);
        }
        return instance;
    }
}

/// Compression level parameter for algorithm selected by configuration property <see cref="CompressionType"/>
/// compression.level in librdkafka
/// </summary>
public int CompressionLevel { get; set; } = -1;

/// <summary>
/// Compression codec to use for compressing message sets.
/// compression.codec in librdkafka
/// </summary>
public MessageCompressionType CompressionType { get; set; } = MessageCompressionType.NotSet;
}
}
8 changes: 7 additions & 1 deletion src/Microsoft.Azure.WebJobs.Extensions.Kafka/Output/KafkaProducerFactory.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,9 +131,15 @@ public ProducerConfig GetProducerConfig(KafkaProducerEntity entity)
SslCaLocation = resolvedSslCaLocation,
Debug = kafkaOptions?.LibkafkaDebug,
MetadataMaxAgeMs = kafkaOptions?.MetadataMaxAgeMs,
SocketKeepaliveEnable = kafkaOptions?.SocketKeepaliveEnable
SocketKeepaliveEnable = kafkaOptions?.SocketKeepaliveEnable,
CompressionLevel = entity.Attribute.CompressionLevel
};

if (entity.Attribute.CompressionType != MessageCompressionType.NotSet)
{
conf.CompressionType = (CompressionType)entity.Attribute.CompressionType;
}

if (entity.Attribute.AuthenticationMode != BrokerAuthenticationMode.NotSet)
{
conf.SaslMechanism = (SaslMechanism)entity.Attribute.AuthenticationMode;
Expand Down
25 changes: 25 additions & 0 deletions test/Microsoft.Azure.WebJobs.Extensions.Kafka.UnitTests/KafkaProducerFactoryTest.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -307,5 +307,30 @@ public void GetProducerConfig_When_Ssl_Locations_Resolve_InAzure_Should_Contain_
Assert.Equal(sslCa.FullName, config.SslCaLocation);
Assert.Equal(sslKeyLocation.FullName, config.SslKeyLocation);
}

[Theory]
[InlineData(MessageCompressionType.NotSet, null)]
[InlineData(MessageCompressionType.None, CompressionType.None)]
[InlineData(MessageCompressionType.Gzip, CompressionType.Gzip)]
[InlineData(MessageCompressionType.Snappy, CompressionType.Snappy)]
[InlineData(MessageCompressionType.Lz4, CompressionType.Lz4)]
[InlineData(MessageCompressionType.Zstd, CompressionType.Zstd)]
public void GetProducerConfig_When_CompressionType_Defined_Should_Set_CompressionType(MessageCompressionType sourceType, CompressionType? targetType)
{
var attribute = new KafkaAttribute("brokers:9092", "myTopic")
{
CompressionType = sourceType
};

var entity = new KafkaProducerEntity()
{
Attribute = attribute,
ValueType = typeof(ProtoUser),
};

var factory = new KafkaProducerFactory(emptyConfiguration, new DefaultNameResolver(emptyConfiguration), NullLoggerProvider.Instance);
var config = factory.GetProducerConfig(entity);
Assert.Equal(targetType, config.CompressionType);
}
}
}