-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Destination CDK: Skip flush of zero byte records. #39106
Conversation
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Ignored Deployment
|
This stack of pull requests is managed by Graphite. Learn more about stacking. |
9a3742e
to
aa96c15
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🤦 so it really was that easy
does it.flush
need to be before the byteCount check? Or can we skip that too?
we still need to do flush because weirdly that call does close of the file stream and all. otherwise we'll end up with open fds and open byte streams 🤦♂️ |
aa96c15
to
5997f8c
Compare
What
Related: #31425
Fixes: https://github.com/airbytehq/airbyte-internal-issues/issues/6728
In the old code, the below trace was the culprit in
AsyncFlush.kt
, fixing it for good in new flow.This seems to happen when sources are slow and Async framework issues a flush irrespective of the number of records, leading to the Flush function just initializing empty buffer and doing UPLOAD/COPY with 0 byte files.
How
Review guide
User Impact
Can this PR be safely reverted and rolled back?