Use Iceberg tables as sources for Spark Structured Streaming

[aokolnychyi]

We can use snapshot history to support reading from an Iceberg table as a streaming source.

[prodeezy]

Sounds like something we might be interested in as well. Can you add more detail as to how you envision this will be used?

[aokolnychyi]

I think it is possible to leverage the snapshot log in Iceberg to use Iceberg tables as streaming sources for Structured Streaming. This would mean streaming out data from all of the files present in the table as well as from new files that arrive after the stream is started.

There are some things to think about:

• What should happen if we detect deleted files while we are streaming data out?
• What do we consider as one batch? For example, we will need to stream out data from all of the files present in the table when we start a query. This might be a lot of data, so we need to split it into multiple batches. The split can be based on either the number of files/records or even estimated size.

[aokolnychyi]

@prodeezy would you be interested to work on this?

[prodeezy]

Looks interesting to me to work on! I'm currently discussing this internally for approval.

[aokolnychyi]

@prodeezy sure, let me know the outcome.

[aokolnychyi]

Another point to consider is how we compute diffs between snapshots. This operation should be efficient. For example, we consumed all of the data at a given point in time and then a new batch came in. We should be able to identify new files easily.

[goldentriangle]

@prodeezy is there an ETA for this feature to be completed?

[prodeezy]

Haven't really looked much into this. I'v been off this past week and otherwise busy with vectorization work. So don't have an plan or ETA as yet.

…

On Wed, Jul 3, 2019 at 8:16 AM Kai ***@***.***> wrote: @prodeezy <https://github.com/prodeezy> is there an ETA for this feature to be completed? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#179>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AALIAH2ZANF4F6QZPTL567LP5TUJDANCNFSM4HLGNXSQ> .

[prabcs]

👋, I'm going through iceberg and it seems quite interesting. I have couple of questions on this issue.

What do we consider as one batch?

With Spark's Structured Streaming, need to consider the size of a microbatch as well. The size of a microbatch can be defined at the query level. Typically, that would mean reading that number of records from the stream (FileStream) in this case. The File Source in Spark Structured Streaming can read new files per batch. So, the design of a streaming Iceberg File Source can be based on that.

[aokolnychyi]

I am thinking to pick this up in the near future. Let me know if there is anybody already working on this.

[timmylicheng]

We at Tencent are also looking in it.

…

On Tue, Aug 6, 2019 at 6:20 AM Anton Okolnychyi ***@***.***> wrote: I am thinking to pick this up in the near future. Let me know if there is anybody already working on this. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#179>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABNTOUPT3PSKMVGJZDROEATQDCRSPANCNFSM4HLGNXSQ> .

-- [NIV] Corinthians 13:8 Love never fails. But where there are prophecies, they will cease; where there are tongues,they will be stilled; where there is knowledge, it will pass away. Li Cheng SDE in Amazon Web Service/ Simple Storage Solution (S3) Electrical & System Engineering Washington University in St. Louis

[aokolnychyi]

@timmylicheng do you already have a prototype? Want to submit a PR?

[aokolnychyi]

@prabcs, yeah, I also raised a similar question. I believe the micro-batch size should be configurable.

[timmylicheng]

No work done yet. Feel free to take this up if you already have something done. Just keep us posted. :)

…

On Tue, Aug 6, 2019 at 8:06 PM Anton Okolnychyi ***@***.***> wrote: @timmylicheng <https://github.com/timmylicheng> do you already have a prototype? Want to submit a PR? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#179>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABNTOUM46VLL3XZHBLVAS6LQDFSMPANCNFSM4HLGNXSQ> .

-- [NIV] Corinthians 13:8 Love never fails. But where there are prophecies, they will cease; where there are tongues,they will be stilled; where there is knowledge, it will pass away. Li Cheng SDE in Amazon Web Service/ Simple Storage Solution (S3) Electrical & System Engineering Washington University in St. Louis