Skip to content

Flink: Improvements for iceberg sink connector. #1403

Closed as not planned
Closed as not planned
@openinx

Description

@openinx

In this patch, we have several minor issues to follow up for flink sink connector:

  • Support multiple iceberg sinks in the same flink job. discussion.
  • provide a method to set user-provided write parallelism in FlinkSink#Builder.
  • The netflix team have improved their flink sink by buffering all the uncommitted data files in a separate manifest file to reduce the state size. It would be really helpful if users put their data files and metstore in two different cloud region, because we could keep the job going even if we fail to commit to iceberg sometimes. discussion

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions