DBZ-2499: Debezium Connectors are failing while reading binlog: Unknown event type 100

[morozov]

AWS Aurora writes some non-standard ignorable (LOG_EVENT_IGNORABLE_F) events of types 100 and 101 to the binlog (no documentation reference so far). If the binlog reader reads an event of an unknown type, it stops at the position right after the event type.

A subsequent call to reader.readEvent() will resume from this position and misinterpret the remaining portion of the previous event's header as a new event header. In the best case, it will return some other unknown-type event (see the test). In the worst case (if it interprets the remaining part as ROTATE), it will read the rest of the binlog in memory:

mysql-binlog-connector-java/src/main/java/com/github/shyiko/mysql/binlog/event/deserialization/RotateEventDataDeserializer.java

Lines 29 to 34 in c9c8903

	public RotateEventData deserialize(ByteArrayInputStream inputStream) throws IOException {
	RotateEventData eventData = new RotateEventData();
	eventData.setBinlogPosition(inputStream.readLong(8));
	eventData.setBinlogFilename(inputStream.readString(inputStream.available()));
	return eventData;
	}

It's not clear how this issue should be solved properly since EventHeader#getType() only allows to return a value of the EventType type which by design cannot contain any non-standard values.

mysql-binlog-connector-java/src/main/java/com/github/shyiko/mysql/binlog/event/EventHeader.java

Line 26 in 61482c5

EventType getEventType();

The proposed solution is to represent an unknown event with the UNKNOWN type and let the consumer decide how to handle it.

[gunnarmorling]

So I think that's a good fix. We then can interpret EventType.UNKNOWN in Debezium as per our "event.processing.failure.handling.mode" setting.

[osheroff]

yeah this'll work for maxwell too. thx! merging.

[xaka]

@osheroff could you please apply it to maxwell by any chance? thx

[barryoneill]

@morozov I just hit this using the predecessor for this project - thank you so much for fixing it!