Upgrade 9.1.2 -> 10 fails (Kafka error)

[fbhdk]

Important Details

How are you running Sentry?

• On-Premise docker [Version 10]
• Saas (sentry.io)
• Other [briefly describe your environment]

Description

Ran into this bug: getsentry/sentry#17422
Now that it has been fixed thi shappens:

Applying sentry.0024_auto_20191230_2052...Events to process: 22961

06:54:05 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=MSG_SIZE_TOO_LARGE,val=10,str="Broker: Message size too large"}): <cimpl.Message object at 0x7fd6a4ab35d0>
06:54:05 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=MSG_SIZE_TOO_LARGE,val=10,str="Broker: Message size too large"}): <cimpl.Message object at 0x7fd6a4ab35d0>

The same Kafka-error shows up many times and after quite a while it will continue:
Event migration done. Migrated 22961 of 22961 events.

• and then eventually finish migrations.

Steps to Reproduce

1. git checkout master --force
2. git pull
3. mv sentry/config.example.yml sentry/config.yml (insert security key)
4. ./install.sh
5. Wait

There are no special configurations. Dataset is in an existing postgres 9.6 container.

What you expected to happen

No errors

Possible Solution

Maybe some limits in terms of message size limit should be altered somewhere?

[BYK]

I'll increase this limit in the default configuration as @buffcode suggested over at #356 (comment)

[BYK]

@buffcode - would you like to submit the PR yourself to get the credit for it? :)

[tm1000]

@BYK if we run into this issue (Message size too large) but just wait it out. Is everything ok? or should we re-run the migrations on a backup?

[fbhdk]

@BYK if we run into this issue (Message size too large) but just wait it out. Is everything ok? or should we re-run the migrations on a backup?

If you wait it out the elements that are too large will not be migrated as far as I could tell. I fixed the errors by modifying docker-compose.yml under the kafka service by adding the KAFKA_MESSAGE_MAX_BYTES environment variable.

[BYK]

Seems like we use 50MB for this in a few places, I wonder if we should apply it here:

KAFKA_MESSAGE_MAX_BYTES=50000000 #50MB or bust
KAFKA_MAX_REQUEST_SIZE=50000000 #50MB on requests apparently too

cc @mattrobenolt @JTCunning

[mattrobenolt]

Correct. I would.

[AkshayNarayan777]

Hi
Can someone help me out on this sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out".
This error is shown in kafka container
Thanks