Event Notifications not working over Event Bus for Kafka or RabbitMQ Configurations #7589
Description
ISSUE TYPE
- Bug Report
COMPONENT NAME
Event Notification Framework
CLOUDSTACK VERSION
4.17.2.0
CONFIGURATION
- Kafka Cluster on same network (can connect from management server)
- RabbitMQ Server on same network (can connect from management server)
- Management server config:
publish.action.events = true
publish.alert.events = true
publish.async.job.events = true
publish.resource.state.events = true
publish.usage.events = true
OS / ENVIRONMENT
N/A
SUMMARY
Event notifications were set up according to the Events documentation. We tried both the Kafka and AMQP implementations, but neither one generated events in our Kafka or RabbitMQ queues.
Our guess is that there is a problem with the file configuration, since this documentation does not appear to have been updated in the past 5 years.
In order to create the files as instructed, we had to manually create the META-INF directory as it did not exist. This makes us suspicious since we are not confident this is the correct place for the configuration. The management logs also did not contain any references to kafka or rabbitmq.
mkdir -p /etc/cloudstack/management/META-INF/cloudstack/core
The management server was restarted any time the config changed.
This issue seems to indicate that the feature is working in relatively recent versions of Cloudstack, which conflicts with the "wrong directory" theory.
We also suspect it might not be working due to some other service consuming event notifications, since there is not support for having more than one event consumer. We however do not have other event consumers set up, unless the internal Event Log is somehow consuming these events.
STEPS TO REPRODUCE
File setup for Kafka:
cat /etc/cloudstack/management/kafka.producer.properties
bootstrap.servers=kafka1.mydomain.org:29092,kafka2.mydomain.org:39092
acks=1
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer
topic=cloudstack(we also tried acks=all, no difference)
cat /etc/cloudstack/management/META-INF/cloudstack/core/spring-event-bus-context.xml
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:aop="http://www.springframework.org/schema/aop"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-3.0.xsd
http://www.springframework.org/schema/context
http://www.springframework.org/schema/context/spring-context-3.0.xsd">
<bean id="eventNotificationBus" class="org.apache.cloudstack.mom.kafka.KafkaEventBus">
<property name="name" value="eventNotificationBus"/>
</bean>
</beans>RabbitMQ Setup:
cat /etc/cloudstack/management/META-INF/cloudstack/core/spring-event-bus-context.xml
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:aop="http://www.springframework.org/schema/aop"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-3.0.xsd
http://www.springframework.org/schema/context
http://www.springframework.org/schema/context/spring-context-3.0.xsd">
<bean id="eventNotificationBus" class="org.apache.cloudstack.mom.rabbitmq.RabbitMQEventBus">
<property name="name" value="eventNotificationBus"/>
<property name="server" value="kafka1.mydomain.org"/>
<property name="port" value="5672"/>
<property name="username" value="guest"/>
<property name="password" value="guest"/>
<property name="exchange" value="cloudstack-events"/>
</bean>
</beans>EXPECTED RESULTS
We expected to see events published to the Kafka cluster or RabbitMQ servers.
ACTUAL RESULTS
We did not receive any events published to the queue.