Core: Store schema and spec in TaskContext to avoid unnecessary deserialization (#11235) #11280
Conversation
|
This works well when table's schema is over 1k, and when there is a need to read table specs & schema after table scan. In our case(2k column, 25.9TB, 460,000 files), this can reduce the time required to get all specs after tableScan from over 10 min to 30s. @aokolnychyi @RussellSpitzer please take a look when you have time~ |
|
This sounds interesting, is it just the ser-de that causes this ? what about the increase in memory pressure to hold this in memory ? |
Yes for the first question, in my SDK service, I need to get all the spec id (do ser-de) for every FileScanTask, this is one reason why it is so slow. Sure for the increase in memory pressure, it's a good question. Is there another way to solve ser-de problem? I collect all spec id to check if one column was previously used as a partition field. |
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
No description provided.