snuba-subscription-consumers are running but are not consuming

[m0nikasingh]

Hello Sentry team,

I have upgraded the sentry from 21.8.0 to 22.10.0
I am facing some issues with the snuba-subscription-consumer-events and snuba-subscription-consumer-transactions consumers. The consumers are running without any errors but are not consuming any messages. It is causing the messages to get queued and increase in redis memory and redis eventually crashes.

consumers are run as below:

/usr/local/bin/python /usr/local/bin/snuba subscriptions-scheduler-executor --dataset events --entity events --auto-offset-reset latest --no-strict-offset-reset --consumer-group snuba-events-subscriptions-consumers --followed-consumer-group snuba-consumers --delay-seconds 60 --schedule-ttl 60 --stale-threshold-seconds 900 --log-level debug

/usr/local/bin/python /usr/local/bin/snuba subscriptions-scheduler-executor --dataset transactions --entity transactions --auto-offset-reset latest --no-strict-offset-reset --consumer-group snuba-transactions-subscriptions-consumers --followed-consumer-group transactions_group --delay-seconds 60 --schedule-ttl 60 --stale-threshold-seconds 900

I am getting the events but not the transaction events.

I see messages in transactions topic and redis transaction type events piling up but nothing is consuming from it.

I would really appreciate any help on this.

[m0nikasingh]

It would be helpful if I can get a consumer:consumer_group:topic mapping.

[fpacifici]

Hi,
may I ask what problem you would solve with that?
Generally we do not put consumer groups in settings as they are passed through the CLI. This is because they tend to change more often than other parameters with respect to the default.

[m0nikasingh]

Thank you for the quick response.

Since all the consumers are running without any error, I suspect if the consumers are trying to consume from some non-existent (/wrong) topic.

The consumer-groups are passed in the CLI (/consumer), but with the recent release, the CLI "topic" option has been removed. So I am having hard time figuring out which topic maps to which consumer and if it configured properly.

Any pointers for troubleshooting this?

[lynnagara]

Yep. For transaction events the default topic subscriptions will consume from is snuba-transactions-commit-log. In code this is defined via https://github.com/getsentry/snuba/blob/master/snuba/datasets/storages/transactions.py#L36 and https://github.com/getsentry/snuba/blob/master/snuba/utils/streams/topics.py#L12. These values can be overridden by users via a custom settings file.

If you want to change this default topic it should be done via the settings.KAFKA_TOPIC_MAP mechanism.

The main transactions-consumer should be the one producing values to that topic so the subscriptions consumer can read from there. Is that somehow being routed to a different topic in your case?

[m0nikasingh]

Thanks for your reply.

There was an oversight, snuba-transaction-consumer --raw-events-topic was set to events topic, after removing this the transactions are working fine.

But the Redis memory continues to be the problem - The keys are piling up in the Redis. I believe I am seeing the same issue as mentioned in getsentry/self-hosted/issues/1787 and getsentry/self-hosted/issues/1796.

I also do not see any messages for topics - production.events-subscription-results, production.transactions-subscription-results, production.snuba-transactions-commit-log, production.scheduled-subscriptions-events, production.scheduled-subscriptions-events, production.scheduled-subscriptions-transactions. I have verified all the topics/config, everything looks fine to me.

In Redis, I see 2 KEYSPACEs
0 - which has most of the keys in the format "c:1:e:<event_id>:project_num" and ts:0: KEYS
1 - which has only snuba-ratelimit:int keys

Most of the keys which have piled up in Redis are of transaction type. I can see these events/traces in Sentry even after deleting the key from the Redis, so could it be that these Redis keys are stale and are not getting cleaned up?

The PR #3219 was added in 22.10.0 release where use of multiple Redis instances was introduced. Is there any chance that the the Redis memory issue could be related to this change?

Please let me know if you need any more information.

[lynnagara]

@m0nikasingh The keys get removed in the post processing phase. Please ensure that the transactions post process forwarder is running and correctly configured to the right topic as well.

[klboke]

When we upgraded to 22.10.0, we encountered a similar problem to yours: debugging the code revealed that it could be caused by the following two issues：

• 1、Fix commit-log-topic parameter configuration problem self-hosted#1817
• 2、ref(symbolicator): Avoid large messages (crash logs for ios) being thrown into celery queues sentry#41597

As @lynnagara said, the problem is ultimately caused by post-process-forwarder(errors) \ post-process-forwarder(transactions) not working properly, the first point above will lead to a failure in getting snuba-transactions-commit The first point above will result in a failure to get the snuba-transactions-commit -log subject information, which will affect the Redis delete of transactions, and the second point will result in a failure to write to RabbitMQ, resulting in a Redis delete failure

[klboke]

1. After the upgrade to version 22.10.0, the event data of type=transactions and the events data of type=errors are handled separately from the data flow, and after the upgrade, the kafka consumer task sentry-post-process that cleans up the data of type=transactions -forwarder-transactions because it relies on the commit-log-topic=snuba-transactions-commit-log commit log of the sentry-snuba-transactions-consumer consumer task, and the sentry-snuba-transactions-commit-log commit log of the sentry-snuba-transactions-consumer consumer task. -transactions-consumer was upgraded to 22.10.0, the commit-log-topic record was not modified as expected, but remained snuba-commit-log, resulting in the sentry-post-process-forwarder-transactions task never consuming data from kafka, and thus the data with type=transactions was never cleaned

Solution: Modify the kafka commit commit-log-topic of sentry-snuba-transactions-consumer to snuba-transactions-commit-log.

2, ios crash event data, after version 22.1.0 updated the data flow in the worker, when submitting asynchronous tasks, the data data will be passed directly to rabbitmq, and our AliCloud rabbitmq limits the size of 64kb message body, resulting in ios crash event in the worker does not run properly, and the asynchronous scheduling task framework. and the asynchronous scheduling task framework celery swallowed the exception message directly.

Temporary solution: modify the code so that woker tasks only pass through the cache_key and use the cache_key to fetch the raw data from redis when running tasks. Create a fix image and deploy the fix image.

Final solution: Three options are available: 1. sync the modified code to the community, 2. build rabbitmq, 3. switch the celee queue to redis

@m0nikasingh These are my summary notes from troubleshooting this problem, I hope you find them useful. The pr fixes I submitted are now available in 22.12.0, perhaps if you upgrade to this version you will have solved the problem.