Cannot subscribe to new topic

[paambaati]

@tulios @Nevon I ran into another case where I cannot subscribe to a topic —

1. Stand up app subscribed to ['topic1', 'topic2'].
2. Create a new topic topic1_v2 with a different number of topics from topic1.
3. Kill app, update config to subscribe from ['topic1_v2', 'topic2'] and restart; no change to groupId.
4. The app now does not consume from topic1_v2.

Relevant logs

{"level":"WARN","timestamp":"2018-12-03T04:47:55.265Z","logger":"kafkajs","message":"[ConsumerGroup] Consumer group received unsubscribed topics","groupId":"kafka-reader","generationId":134,"memberId":"kafka-reader-0ef556dd-88fd-455f-b651-83e64a70cbac","assignedTopics":["topic1","topic2"],"topicsSubscribed":["topic1_v2","topic2"],"topicsNotSubscribed":["topic1"]}
{"level":"ERROR","timestamp":"2018-12-03T04:48:00.286Z","logger":"kafkajs","message":"[Connection] Response Heartbeat(key: 12, version: 0)","broker":"kafka1.com:9092","clientId":"kafka-reader","error":"The group is rebalancing, so a rejoin is needed","correlationId":9,"size":6}
{"level":"ERROR","timestamp":"2018-12-03T04:48:00.286Z","logger":"kafkajs","message":"[Runner] The group is rebalancing, re-joining","groupId":"kafka-reader","memberId":"kafka-reader-0ef556dd-88fd-455f-b651-83e64a70cbac","error":"The group is rebalancing, so a rejoin is needed","retryCount":0,"retryTime":259}
{"level":"ERROR","timestamp":"2018-12-03T04:49:55.264Z","logger":"kafkajs","message":"[Connection] Response SyncGroup(key: 14, version: 0)","broker":"kafka1.com:9092","clientId":"kafka-reader","error":"The group is rebalancing, so a rejoin is needed","correlationId":11,"size":10}
{"level":"WARN","timestamp":"2018-12-03T04:49:55.651Z","logger":"kafkajs","message":"[ConsumerGroup] Consumer group received unsubscribed topics","groupId":"kafka-reader","generationId":136,"memberId":"kafka-reader-0ef556dd-88fd-455f-b651-83e64a70cbac","assignedTopics":["topic1_v2","topic2"],"topicsSubscribed":["topic2"],"topicsNotSubscribed":["topic1_v2"]}

[Nevon]

I have managed to reproduce this bug. It's related to #221. We'll be trying to get a fix sorted. Currently, just doing a rolling restart won't fix things. I suspect you'll need to bring all your consumers down before starting them back up again, but if it's impacting production, I would just wait for the fix.

[paambaati]

I have managed to reproduce this bug.

@Nevon That's great. I've temporarily changed the groupId and redeployed so I can unblock the app, so there's no hurry.

[tulios]

@paambaati v1.4.6 is out with the fix. Thanks for the reporting