Support kafka trigger

[eric-gonzalez-tfs]

I was unable to find documentation how to configure the librdkafka settings in JS with the v4 runtime.

1. How can we configure properties such as AutoOffsetReset/auto.offset.reset?
2. Is there a types package that we can leverage to type the options since there is no kafka specific trigger and we must use the generic trigger?

Thank you in advance for your insight

Investigative information

Please provide the following:

• Timestamp: N/A
• Function App name: N/A
• Function name(s) (as appropriate): N/A
• Invocation ID: N/A
• Region: N/A

Repro steps

Provide the steps required to reproduce the problem: N/A

Expected behavior

Provide a description of the expected behavior.

N/A

Actual behavior

Provide a description of the actual behavior observed.

Known workarounds

Provide a description of any known workarounds.

N/A

Related information

Provide any related information

• Programming language used : Typescript
• Links to source
• Bindings used: Kafka
• Runtime: "@azure/functions": "^4.0.1",
• host.json:

{
  "version": "2.0",
  "logging": {
    "applicationInsights": {
      "samplingSettings": {
        "isEnabled": true,
        "excludedTypes": "Request"
      }
    }
  },
  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "[4.*, 5.0.0)"
  }
}

[eric-gonzalez-tfs]

Hi Team, do you have an update here? Is there any information you're missing from me? TIA for the help

[ejizba]

Hi @eric-gonzalez-tfs, sorry for the delay. Unfortunately I'm not super familiar with kafka. I don't see AutoOffsetReset or auto.offset.reset listed in the docs here. Can you give more context on those settings?

We add types for triggers based on usage and customer feedback, so the fact that you filed an issue increased the chances we will add it for Kafka. That being said, we would normally base the types on the docs I just linked, so if those docs aren't useful in the first place we would need to reach out to the kafka team for help

[eric-gonzalez-tfs]

Hi @eric-gonzalez-tfs, sorry for the delay. Unfortunately I'm not super familiar with kafka. I don't see AutoOffsetReset or auto.offset.reset listed in the docs here. Can you give more context on those settings?

We add types for triggers based on usage and customer feedback, so the fact that you filed an issue increased the chances we will add it for Kafka. That being said, we would normally base the types on the docs I just linked, so if those docs aren't useful in the first place we would need to reach out to the kafka team for help

No problem - I'll try my best to elaborate here. One of the features of Kafka is that it can ensure each message is processed based on its unique consumer group name. Every time a message is processed, it's offset is committed so the consumer(s) know where to pick up from when more messages are produced. In the current implementation and the parameters included in my screenshot above, if I were to change the consumerGroup name the Azure Function would begin reading all of the messages on that topic since it's inception. This behavior can be configured by setting AutoOffsetReset or auto.offset.reset to a value other than the default. More info here.

Thanks for the info - It seems like the docs are up to date. However - I have noticed it does not get into the nitty-gritty of the librdkafka configuration. I did find these docs that have a little more info but do not go into how to configure them in the Javascript implementation. It would be nice to include those in the JS docs (assuming they're supported).

Let me know if you need any addt'l info @ejizba, appreciate the help.

[eric-gonzalez-tfs]

Hi @ejizba - following up here.

It does seem like azure-functions-kafka-extension supports setting AutoOffsetReset...

Is this something that is inherited by the Azure Function runtime?

I even see the configuration being logged in the Filesystem Logs of my Function App

[jgaph]

This would be a great feature for the Kafka trigger. The fact that it is impossible to control offset commits using the trigger is very frustrating.