[Improvement] flushing buffer if the memoryUsed of the first record of WriterBuffer larger than bufferSize #1497
Comments
zuston
pushed a commit
that referenced
this issue
Feb 1, 2024
…first record of `WriterBuffer` larger than bufferSize (#1485) ### What changes were proposed in this pull request? Flushing buffer if the memoryUsed of the first record of `WriterBuffer` larger than bufferSize. ### Why are the changes needed? More accurate. fix #1497 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? UT.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Code of Conduct
Search before asking
What would you like to be improved?
Currently in
org.apache.spark.shuffle.writer.WriteBufferManager#insertIntoBuffer, if the memoryUsed of WriterBuffer larger than bufferSize, the WriterBuffer will be flush. But if the memoryUsed of the first record larger than bufferSize, the WriterBuffer will not be flush.How should we improve?
Flushing buffer if the memoryUsed of the first record of WriterBuffer larger than bufferSize
Are you willing to submit PR?
The text was updated successfully, but these errors were encountered: