Add ak.array() and pdarray.to_ndarray()

[reuster986]

We need a function on the python side that accepts a list-like object and sends it to the chapel backend to create a pdarray. The complementary method, pdarray.to_ndarray(), would also be useful, but with some safety checks to disallow sending back huge arrays. Both of these functions will require a means of bulk data serialization over ZMQ between python and chapel.

[reuster986]

I'm trying to get Python and Chapel to pass arrays over ZMQ, but it's surprisingly difficult. I have passed binary arrays between Python and Chapel before via the filesystem, and it was relatively easy with Chapel's channel IO interface and Python's struct module. However, Chapel has no channel interface for (binary) strings. If Chapel's ZMQ sockets could send/receive binary data (e.g. as a uint8 array), I could use a tempfile proxy to hack together a channel interface to a ZMQ socket, but I don't think they can. If chapel could encode/decode base64, I could send binary data as a string over ZMQ, but alas.

My latest attempt was to use Chapel's builtin JSON formatting, which works fine for the Python->Chapel direction, but breaks in the other direction. It seems that "%jt".format(array) gives something that's trying to be JSON but isn't: the separators are spaces, not commas; the internal array elements are quoted, but the first and last elements are missing their bookend quotes and square brackets; and floats are converted with not enough precision for round-trip.

I think serialization beyond files is likely to be important for Chapel in general, not just arkouda, so I am suggesting three potential (non-exclusive) solutions to this issue:

1. Implement binary strings/buffers as first-class objects, with support for channel IO and ZMQ read/write
2. Fix Chapel's array to JSON conversion via "%jt".format(array)
3. Add a Base64 module

The only one I can really work on is 3, so that's probably what I'll do in the short term for arkouda, but I wanted to catalog raise these issues to hopefully improve Chapel in the long run.

[bradcray]

@mppf, @lydia-duncan : Any thoughts or experiences here that would help Bill out?

[mppf]

However, Chapel has no channel interface for (binary) strings.

1. Implement binary strings/buffers as first-class objects, with support for channel IO and ZMQ read/write

Sure it does. In the future, we're expecting to add a separate bytes type (a-la Python 3) but for now, you can store arbitrary bytes in a string (and then just be careful what you do with it). (Of course you'll have to migrate this over to bytes in the future if you do this).

To read such a binary string you could use

• https://chapel-lang.org/docs/master/modules/standard/IO.html#IO.channel.readstring (but then you'd have to know the length first)
• https://chapel-lang.org/docs/master/modules/standard/IO/FormattedIO.html#string-and-bytes-conversions (e.g. the format specifier %|8S uses a binary format for a string and prefixes the string with 8 bytes of length; or you could use %|*s to put the length argument right before the string argument in the formatted I/O call - the length argument will be the length in bytes you want to read/write).
• opening a file in binary mode should result in the array data just getting stored directly as bytes, with no length information or any delimiters or anything

Channels also have a binary mode, and perhaps using that is important in this setting.

But, the ZMQ module just knows how to read/write data in a string. AFAIK it does nothing with the string contents (and it could be any bytes). The string encompasses the whole message.

You can open a memory buffer (openmem), use binary I/O operations to write an array to it, and then use a reader channel to read the entire contents into a string, and then send that with ZMQ. Going the other way, you could receive an entire (binary) string with ZMQ, write it to a memory buffer (openmem), and then use binary I/O operations to read it into an array (which needs to have a size you already established - the size has to be communicated separately in this scheme).

2. Fix Chapel's array to JSON conversion via "%jt".format(array)

I don't expect this to be hard at all. If you can show a demo program with all the bad behaviors, I could probably fix it tomorrow. It's probably specific to Block arrays (e.g.).

[reuster986]

@mppf Thanks for the pointers! I had been trying something like what you suggested -- creating a Python bytes from an array with struct.pack, sending it to Chapel via ZMQ, writing the resulting Chapel string to a tempfile (I will use openmem from now on), and reading that with a channel in binary mode. However, it was giving me Unicode errors, so I assumed Chapel was treating it like an encoded string as opposed to bytes. This was a while ago, so I'll try again with your comments in mind, and I'll either get it right this time or reproduce the errors here.

The JSON formatting problem indeed has something to do with a Block array. Checkout the JSONError branch of this repo and compile/run JSONtest.chpl for an example. It only triggers with a SymEntry (which is backed by a Block array), not with a normal array, as far as I can tell. The JSON portion of the test string in that file was created using Python:

import json, numpy
json.dumps(numpy.linspace(0, 1, 10, dtype=numpy.float64))

[mppf]

@reuster986 - I was able to reproduce the issue with a very simple standalone Block array JSON print. I've created this PR: chapel-lang/chapel#12767 to fix the issue, but I havn't tried it with your tester. After that PR, you'll be able to increase the floating point precision when doing JSON array output in the format string, e.g. writef("%.20jt\n", A);.

[mppf]

However, it was giving me Unicode errors, so I assumed Chapel was treating it like an encoded string as opposed to bytes

You probably need to specify that your channels are doing binary I/O (rather than text I/O). See https://chapel-lang.org/docs/master/primers/fileIO.html#binary-i-o and https://chapel-lang.org/docs/master/modules/standard/IO.html#IO.iokind

[reuster986]

Ah, that was the problem: I was using binary IO for read/writing the arrays, but not the strings. It looks like I'm able to convert numpy arrays to pdarrays now; just need to try the other direction. Thanks for setting me straight!

[mhmerrill]

I looked at the code, please add some simple error checking for isinstance(a, np.ndarray) to the code and raise a TypeError if it's not the case.

[reuster986]

That kind of type-checking is not very pythonic... instead, I propose:

def array(a):
    try:
        a = np.array(a)
    catch:
        raise TypeError("Argument must be array-like")
    ...

This way, we could accept any array-like argument and take advantage of numpy's conversions and error handling.

The only problem is I won't be able to work on this until Thursday at the earliest, unless I just edit the code here through the browser. But then I can't test it... please advise.

[mhmerrill]

Thanks for the advise, I can add the code to the master branch.
I just felt a little naked with no error checking.