You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
A couple of developers might discover that it's not that easy to decide when to use ensureLoaded to load native data to JVM as well as ensureOffloaded. The existing of the methods were to implement lazy load/offload on data to reduce the overall JNI overhead running queires.
Describe the solution you'd like
Spark 3.3 made class ColumnarBatch inheritable. We can implement the class by (e.g.) GlutenLoadedColumnarBatch and GlutenOffloadedColumnarBatch, then we don't need developers to explicitly use ensureLoaded and ensureOffloaded.
This is a placeholder issue since we are now still on Spark 3.2 and ColumnarBatch is still not allowed to be extended.
The text was updated successfully, but these errors were encountered:
zhztheplayer
changed the title
Remove explicit use of utility methods ensureLoaded() and ensureOffloaded
Remove explicit use of utility methods ensureLoaded() and ensureOffloaded()Nov 10, 2022
zhztheplayer
changed the title
Remove explicit use of utility methods ensureLoaded() and ensureOffloaded()
[VL] Remove explicit use of utility methods ensureLoaded() and ensureOffloaded()Apr 4, 2023
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
A couple of developers might discover that it's not that easy to decide when to use
ensureLoaded
to load native data to JVM as well asensureOffloaded
. The existing of the methods were to implement lazy load/offload on data to reduce the overall JNI overhead running queires.Describe the solution you'd like
Spark 3.3 made class
ColumnarBatch
inheritable. We can implement the class by (e.g.)GlutenLoadedColumnarBatch
andGlutenOffloadedColumnarBatch
, then we don't need developers to explicitly useensureLoaded
andensureOffloaded
.This is a placeholder issue since we are now still on Spark 3.2 and
ColumnarBatch
is still not allowed to be extended.The text was updated successfully, but these errors were encountered: