You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Problem description
Make it possible to create a FlinkPravegaReader that provides event metadata (e.g. EventPosition) for each event. This would facilitate the development of jobs that care about the stream position of the event data, e.g. for indexing purposes.
Suggestions for an improvement
Normally, the data stream that is produced is of type DataStream<T> to convey the deserialized event data of type T. To also convey metadata implies a data stream of type DataStream<EventRead<T>>. For example:
The EventRead class might not be ideal; we could introduce a similar type based on discussion.
One complication is that the DeserializationSchema<T> that is provided to the reader is typed to the event data type. Some cleverness in the builder would be needed to disambiguate the deserialized type <T> from the data stream type EventRead<T>. May also need to leverage Flink's OutputTypeConfigurable.
The text was updated successfully, but these errors were encountered:
Problem description
Make it possible to create a
FlinkPravegaReader
that provides event metadata (e.g.EventPosition
) for each event. This would facilitate the development of jobs that care about the stream position of the event data, e.g. for indexing purposes.Suggestions for an improvement
Normally, the data stream that is produced is of type
DataStream<T>
to convey the deserialized event data of typeT
. To also convey metadata implies a data stream of typeDataStream<EventRead<T>>
. For example:The
EventRead
class might not be ideal; we could introduce a similar type based on discussion.One complication is that the
DeserializationSchema<T>
that is provided to the reader is typed to the event data type. Some cleverness in the builder would be needed to disambiguate the deserialized type<T>
from the data stream typeEventRead<T>
. May also need to leverage Flink'sOutputTypeConfigurable
.The text was updated successfully, but these errors were encountered: