Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: Add select methods for IPC #8

Merged
merged 1 commit into from
Jun 28, 2017

Conversation

TomAugspurger
Copy link
Contributor

Allows for select statements that returns memory handles, rather than a result
set. ATM this is implemented on the connection object, rather than the cursor.
I'm not sure which is more appropriate. For it to belong on the Cursor object,
I think we would need to bring the parsing code from pygdf into pymapd, so we
can parse the result into a result set, and return an Iterator over that.

@TomAugspurger
Copy link
Contributor Author

@sklam could you take a look at this on a machine with a GPU when you get a chance? I'm still working on getting access to one.

This script should work assuming you have a mapd server running on 9091 with the default login info, and the flights_2008_10k dataset loaded.

import numpy as np
from numba import cuda
from numba.cuda.cudadrv import drvapi
from pygdf.gpuarrow import GpuArrowReader
import pymapd

table = "flights_2008_10k"
con = pymapd.connect(user="mapd", password="HyperInteractive",
                     host="localhost", dbname="mapd")
statement = "select depdelay from {}".format(table)

results = con.select_ipc_gpu(statement)


# Now for pygdf

ipc_handle = drvapi.cu_ipc_mem_handle(*results.df_handle)
ipch = cuda.driver.IpcHandle(None, ipc_handle, size=results.df_size)
ctx = cuda.current_context()
dptr = ipch.open(ctx)

dtype = np.dtype(np.byte)
darr = cuda.devicearray.DeviceNDArray(shape=dptr.size, strides=dtype.itemsize,
                                      dtype=dtype, gpu_data=dptr)
reader = GpuArrowReader(darr)
reader.to_dict()

Some questions you may be able to answer

  • I've put these on the Connection object, instead of Cursor, which is more about building up results sets and then fetching them. For it to make sense on a Cursor, I think we would have to move the GpuArrowReader code to pymapd (or pymapd could import it and call it?)
  • Do the method names sound OK to you?

Allows for select statements that returns memory handles, rather than a result
set. ATM this is implemented on the `connection` object, rather than the cursor.
I'm not sure which is more appropriate. For it to belong on the `Cursor` object,
I think we would need to bring the parsing code from pygdf into pymapd, so we
can parse the result into a result set, and return an Iterator over that.
@TomAugspurger TomAugspurger merged commit 853f550 into heavyai:master Jun 28, 2017
@TomAugspurger TomAugspurger deleted the ipc-handles branch June 28, 2017 20:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant