Issue when a broker is down

[ArtemisMucaj]

Hi,

My setup is as follows, I have two instances of Kafka running in docker containers, a producer and a consumer script.
When I kill the first broker and restart the producer / consumer script they timeout and "ready" event is never emitted.
{"message":"Local: Timed out","code":-185,"errno":-185,"origin":"kafka"}

The issue is that when calling rd_kafka_metadata, there is a moment when both brokers' state is NOT up and the function that decides which broker to talk to, rd_kafka_broker_any, returns the first broker, rd_kafka_metadata returns a timeout error (because that broker is down).
In the JS code's connect method we return and never emit "ready" event.

This error seems to be fixed on librdkafka's side in the current master version as rd_kafka_metadata's function was partly rewritten.

Did you observer this behavior during your tests or am I doing something wrong?

[webmakersteve]

Ready event is for JS level connections only. It will only ever get emitted one time - that is, when you make the initial connection, unless I'm misunderstanding what you're trying to test.

Librdkafka manages connections internally. Getting a time out is not a non-recoverable error and it will do its best to reconnect you. It batches and queues messages so you can continue writing with no problem, and when Kafka reconnects it will send off what you gave it, or continue reading if you are producing.

What .connect does, on the node side, is just tell librdkafka to go ahead and open a socket. .disconnect tells it to close the socket and stop working.

It's a little hard to follow exactly what you're doing to get this to happen though. Can you give me a sequential list so I can see whether it's working as intended, a bug in the node.js code, or a bug in librdkafka?

Thanks!

[ArtemisMucaj]

• metadata.broker.list equals `kafka1:9092,kafka2:9092'
• I shutdown kafka1 and start my producer script gets a timeout and will never produce as I wait for ready event to start producing
• I restart kafka1 and shutdown kafka2 and start my producer, it connects successfuly
• If both my brokers are down and I start my producer it can never recover and produce when I restart a broker

For my case getting a time out is non-recoverable as I read the messages I produce from files and after it was successfully produced I delete those files.

To be honest I'm really confused with the error messages I get when one of my brokers is down
{"origin":"local","message":"broker transport failure","code":-1,"errno":-1,"stack":"Error: Local: Broker transport failure\n at Error (native)"}
{"origin":"local","message":"all broker connections are down","code":-1,"errno":-1,"stack":"Error: Local: All broker connections are down\n at Error (native)"}

These two errors have the same code which corresponds to ERR__UNKNOWN but their message makes it quite clear it should be ERR__TRANSPORT and ERR__ALL_BROKERS_DOWN

[webmakersteve]

That's a bug if they're both getting that error code. I can adjust that so it uses the proper error code.

I'm still a little confused about what you're doing. It looks like you're explaining two separate flows. Correct me if I'm wrong:

Situation 1

1. Provide node-rdkafka two brokers in a comma separated list. One of them is down before startup.
2. Call .connect to attempt to connect to the metadata.broker.list, one of which is down.
3. ready event is not called.
4. Start broker that was previously unavailable and shut down broker that was previously available.
5. node-rdkafka successfully connects and emits ready event.

Situation 2

1. Provide node-rdkafka two brokers in a comma separated list. Both brokers are down.
2. Call .connect to attempt to connect to the metadata.broker.list, one of which is down.
3. ready event is not called.
4. Start broker both brokers that were previously down.
5. node-rdkafka never attempts to reconnect.

Am I correct in breaking up your two cases?

[ArtemisMucaj]

I'm sorry I wasn't clear in my explaination, there are indeed multiple situations but they're slightly different from what you wrote down ^^

Situation 1

1. Provide node-rdkafka two brokers in a comma separated list. One of them is down before startup
2. Call .connect to attempt to connect to the metadata.broker.list, first broker from the list is down
3. ready event is not called
4. Start broker that was previously unavailable
5. node-rdkafka never attempts to reconnect

Situation 2

1. Provide node-rdkafka two brokers in a comma separated list. One of them is down before startup
2. Call .connect to attempt to connect to the metadata.broker.list, second broker from the list is down
3. ready event is called and everything works just fine

Situation 3

1. Provide node-rdkafka two brokers in a comma separated list. Both are down before startup
2. Call .connect to attempt to connect to the metadata.broker.list, both broker from the list are down
3. ready event is not called
4. Start a broker that was previously unavailable
5. node-rdkafka never attempts to reconnect

[webmakersteve]

As far as I can tell, this would be a bug or an expected behavior of librdkafka itself. I'm going to do some testing using the c++ client for librdkafka to ensure it follows the same pattern and I'll follow up

[ArtemisMucaj]

Hey,

I just stumbled upon a new case, not sure if this is related at all to this issue but it kinda happens in the same context so I didn't want to create a new issue!

1. Provide node-rdkafka two brokers in a comma separeted list. Both are up before startup
2. Call .connect, ready event is called
3. Shutdown both brokers, in some cases an unassignment event is triggered right after {"origin":"local","message":"all broker connections are down","code":-1,"errno":-1,"stack":"Error: Local: All broker connections are down\n at Error (native)"} error message
4. Restart a / both brokers, if unassignment event is emitted we never receive a reassignment event, otherwise everything works just fine!

This is a real bummer as I need to restart my consumer script by hand / shutdown when I get an "all brokers are down" error!

Any advice ?

[webmakersteve]

It looks like if it gets an unassignment rebalance, librdkafka stops checking assignments internally so it never actually starts reading messages again.

Can you listen to that error and then just .disconnect and .connect again to fix the issue? That should initialize a new librdkafka client so it should fix the problem.

[ArtemisMucaj]

The last case is actually the same issue as #29

I fixed Situations 1 and 2 by using librdkafka's master branch as rd_kafka_metadata function was rewritten not long ago.

[webmakersteve]

Okay. So I think we may just need to wait until that is released for this bug to be resolved. Do you agree?

Situation 3 seems to be working as intended from my perspective. You should receive an error in the callback provided to connect.

[ArtemisMucaj]

Agreed, I'll be closing the issue then.

Thanks!

[vinaybhor]

Hi,
I am trying to configure kafka for High availability with two nodes(both given in server.properties with comma separated), but while testing the fail over is failing in below situation:
Start zookeeper and kafka on both nodes,
Start Producer
Start consumer
Then to test the failover:
kill services of 1st Kafka broker
Then the messages pushed from Producer doesn't appear on consumer until the kafka broker is restarted.
Can you please suggest what I'm missing and can check.

[webmakersteve]

If you kill a broker and have acks not set to -1, the broker acknowledges before all ISRs are in sync. That means when the broker restarts it will give it to the other nodes in the cluster and then they will replicate it.

You will not get the message in case of failure until the broker has replicated it out to ISRs in your situation unless you do more aggressive acknowledgements. But this is out of scope of node-rdkafka as it is more just general kafka support.

Additionally, it does not relate to the issue you commented on, so if in the future you feel it requires node-rdkafka changes to be addressed please open a new issue.

Thank you.