NGINX common variables not passed to kafka topic

[qbadx]

Hi,

Would it be possible to pass some extra NGINX common variables to the data sent to the Kafka topic?

Currently, only the data of theNGINX variable $request_body is passed but , for instance, data from variables like $remote_addr would be great .

The goal is to have the IP of the client that is sending the data.

[brg-liuwei]

I have add this issue into TODO LIST.

[lifehacking]

nginx common var is very usefull

[brg-liuwei]

@lifehacking yep, I cannot agree more. PR is welcomed :)

[dkinon]

If anyone here is looking for a workaround in the interim, I am successfully injecting nginx variables into the $request_body using the lua nginx module.

[brg-liuwei]

@dkinon Yes, using lua_nginx_module injecting ngx variables into $request_body is easy to work, and I am an openresty user. But I still want to do some works to pass variables to kafka topic without lua module, cause not everyone use lua_nginx_module. :)

[adridi]

what about http headers - would it be possible to add them to the message body?

[dkinon]

@adridi http headers are actually the ngx variables (ngx.req.get_headers()) I'm injecting into the $request_body. Since they are just another ngx variable, I don't see why they couldn't technically be included. The bigger issue with including ngx variables into a kafka payload in some standard way is that kafka messages have no standard structure. It's just a text payload.

I prefer JSON so I make sure the incoming $request_body is JSON, I inject the ngx variables into the appropriate JSON key in $request_body, pass the resulting text payload of $request_body to the appropriate kafka topic using this module, and then parse that JSON structure on the other end with my kafka consumer. This works great for me but what if someone doesn't want to, or can't, use JSON... now the complexity of implementing this support in this module explodes.

[adridi]

@dkinon Totally Agree with your statement! I was referring to the existing module, I believe it should have set kafka message as = request header + request body. Both are equally important.

[dkinon]

@adridi I was also referring to the existing module, I merely used my lua_ngx module pseudocode example to illustrate one of the many formatted ways this could be done within the confines of the module. The larger point of my previous statement was discussing what format the headers in $request_header + $request_body would look like. Whatever the format, we wouldn't want this to be the default as it would definitely break downstream consumers as they exist today. I'm curious if anyone has suggestions on what those headers would look like?

[1349236893]

When I was testing,crul localhost/ kafka-d "this is a test data",kafka consumers could not accept the data

[1349236893]

当我测试时，crul localhost / kafka-d“这是一个测试数据”，kafka消费者无法接受数据
kafka and nginx No errors were reported