-
Notifications
You must be signed in to change notification settings - Fork 367
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Formalize API for column vectors #2567
Comments
This is a great idea; in particular, it would be great to document which functions/methods are expected to work along with how they're used in DataFrames in different operations. Happy to help with this effort. |
The first candidates that would break are fast aggregations like Having said that I think it should be doable to add "distributed" support to DataFrames.jl in the long run. However, probably we would need to have some API that would communicate to DataFrames.jl how distribution is performed (as if you have distributed vectors most likely you want to process them in a way that takes this into account). |
Yeah I have no idea how distributed computing works, or threading for that matter. Still I will put this on the to-do list for winter break / procrastination from school. |
Somewhat related is whether we preserve the container types of input columns: #2569 I don't think DataFrames has very specific requirements for columns: apart from the issue of one-based indexing, which we should investigate if somebody cares, things should work as long as the |
I feel like the question about datframes with distributed arrays comes up a lot. My impression is that we don't know, for sure, if a Dagger array etc. can "just work" as a column in a DataFrame.
I think I might try to write a custom vector type and then put it in a data frame and see how many functions I can call for it before it becomes a normal vector. Then we can assess to what extent DataFrames can support Dask-like operations just by changing the vector type.
The text was updated successfully, but these errors were encountered: