Arrow Executor #5647
Arrow Executor #5647
Conversation
Signed-off-by: Nicholas Gates <nick@nickgates.com>
vortex-array/src/arrow/executor.rs
Outdated
| use crate::arrays::VarBinVTable; | ||
|
|
||
| /// Trait for executing a Vortex array to produce an Arrow array. | ||
| pub trait ArrowArrayExecutor { |
There was a problem hiding this comment.
Is the use of "executor" here 100% necessary? I think ArrowConverter or ArrowConvertible makes more sense to me and then the trait function would just be convert. More of a nit, so feel free to ignore.
There was a problem hiding this comment.
No not necessarily. I agree the name isn't great, but we want to move towards making it such that the only way to get data out of a Vortex array is via some form of "execution".
By default this turns an array into a vector, but we have MaskExecutor that checks for constant true/false (and should also check for run-end in order to construct slices), and obviously ArrowExecutor checks for lots of things.
Maybe exporter is a better name? Or maybe just into_arrow...
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Deploying vortex-bench with
|
| Latest commit: |
d8d129a
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://672ad3a8.vortex-93b.pages.dev |
| Branch Preview URL: | https://ngates-arrow-executor.vortex-93b.pages.dev |
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
CodSpeed Performance ReportMerging #5647 will improve performances by 17.69%Comparing Summary
Benchmarks breakdownFootnotes
|
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
connortsui20
left a comment
connortsui20
left a comment
There was a problem hiding this comment.
Just a few things.
I think there's a lot of other stuff that can be cleaned up but since we're on time pressure let's get this over the line.
Signed-off-by: Nicholas Gates <nick@nickgates.com>
| // NOTE(ngates): we assume that a cast operation will produce the narrowest possible type that | ||
| // fits the requested precision. | ||
| let fake_precision = precision.max(N::MAX_PRECISION); | ||
| let array = array.cast(DType::Decimal( | ||
| DecimalDType::new(fake_precision, scale), | ||
| Nullability::Nullable, | ||
| ))?; |
There was a problem hiding this comment.
We document the cast to do this
Signed-off-by: Nicholas Gates <nick@nickgates.com>
We eventually need to port all of our execution logic over to executing using Vectors + VortexSession. We can release this new API prior to deprecating the existing one. We should figure out what this API should be. Once the full operators change has finished, we should be able to pull this module out into vortex-arrow. --------- Signed-off-by: Nicholas Gates <nick@nickgates.com>
We eventually need to port all of our execution logic over to executing using Vectors + VortexSession.
We can release this new API prior to deprecating the existing one.
We should figure out what this API should be.
Once the full operators change has finished, we should be able to pull this module out into vortex-arrow.