How do I get involved?

[jpivarski]

I've been on the fence for a while about whether I should be involved in this effort. I'm the lead developer of Awkward Array, which is an array library for complex data structures. Its interface was designed to be a strict generalization of a subset of NumPy's, so in that sense, I'm very much involved in array library APIs.

On the other hand, permitting arrays like

>>> import awkward1 as ak
>>> array = ak.Array([[{"x": 0.0, "y": []}, None, {"x": 1.1, "y": [1]}], [], [{"x": 2.2, "y": [1, 2]}]])

>>> ak.type(array)     # using Datashape to describe non-rectilinear data types
3 * var * ?{"x": float64, "y": var * int64}

>>> ak.to_list(array)
[[{'x': 0.0, 'y': []}, None, {'x': 1.1, 'y': [1]}], [], [{'x': 2.2, 'y': [1, 2]}]]

would break some of the standardization goals you're trying to achieve. For instance, these arrays can't have dtypes or shapes:

>>> array.dtype
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jpivarski/irishep/awkward-1.0/awkward1/highlevel.py", line 1082, in __getattr__
    raise AttributeError(
AttributeError: no field named 'dtype'
>>> array.shape
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jpivarski/irishep/awkward-1.0/awkward1/highlevel.py", line 1082, in __getattr__
    raise AttributeError(
AttributeError: no field named 'shape'

However, the same would be true of Apache Arrow, and I see that they're represented in the list of members.

>>> ak.to_arrow(array)
<pyarrow.lib.ListArray object at 0x7f7b1edbfca0>
[
  -- is_valid:
      [
      true,
      false,
      true
    ]
  -- child 0 type: double
    [
      0,
      0,
      1.1
    ]
  -- child 1 type: list<item: int64>
    [
      [],
      [],
      [
        1
      ]
    ],
  -- is_valid: all not null
  -- child 0 type: double
    []
  -- child 1 type: list<item: int64>
    [],
  -- is_valid: all not null
  -- child 0 type: double
    [
      2.2
    ]
  -- child 1 type: list<item: int64>
    [
      [
        1,
        2
      ]
    ]
]

Maybe the role that projects like Awkward Array and Apache Arrow can play would be to define subsets of array APIs that can apply to non-rectilinear arrays? In today's announcement, you also mention mutability; Awkward Array (and maybe Apache Arrow?) would fall into the immutable subset. Perhaps "allowing for irregularity" is another subset?

[asmeurer]

Do awkward arrays have a shape and a dtype when they represent basic NumPy-like arrays? A big part of the standard is that it defines a minimal subset of APIs that are required for array libraries. But a lot of things are explicitly left out of the spec because the libraries are allowed to do different things there. For something like awkward, an option would be to make it so that "standard" arrays work as the spec says, but if the array is one of the more generalized types, then it doesn't have some of the properties like shape and dtype, and other things that the spec says might not apply either.

[jpivarski]

There are two degrees in which an Awkward array could correspond to rectilinear arrays: (1) if the lengths of all nested lists happen to be equal, and (2) if the data type also ensures that they must be equal.

>>> case1 = ak.Array([[1, 2, 3], [4, 5, 6]])
>>> case1               # the list type is "var" (variable), but those lists happen to have the same length
<Array [[1, 2, 3], [4, 5, 6]] type='2 * var * int64'>
>>> np.asarray(case1)   # so converting it as an array doesn't raise an exception
array([[1, 2, 3],
       [4, 5, 6]])

>>> case2 = ak.Array(np.array([[1, 2, 3], [4, 5, 6]]))
>>> case2               # the list type is "3", meaning that they must have the same length
<Array [[1, 2, 3], [4, 5, 6]] type='2 * 3 * int64'>
>>> np.asarray(case2)   # so it can be cast (zero-copy) as a NumPy array
array([[1, 2, 3],
       [4, 5, 6]])

>>> case3 = ak.Array([[1, None, 3], None, [4, 5, 6]])
>>> case3               # missing values translate to NumPy's masked arrays
<Array [[1, None, 3], None, [4, 5, 6]] type='3 * option[var * ?int64]'>
>>> ak.to_numpy(case3, allow_missing=True)   # with an extra argument
masked_array(
  data=[[1, --, 3],
        [--, --, --],
        [4, 5, 6]],
  mask=[[False,  True, False],
        [ True,  True,  True],
        [False, False, False]],
  fill_value=999999)

>>> case4 = ak.Array([{"x": 1, "y": 1.1}, {"x": 2, "y": 2.2}, {"x": 3, "y": 3.3}])
>>> case4               # and JSON-like objects map to structured arrays, if regular
<Array [{x: 1, y: 1.1}, ... {x: 3, y: 3.3}] type='3 * {"x": int64, "y": float64}'>
>>> np.asarray(case4)
array([(1, 1.1), (2, 2.2), (3, 3.3)], dtype=[('x', '<i8'), ('y', '<f8')])

>>> case5 = ak.Array([[1, 2, 3], [], [4, 5]])
>>> case5               # but if they're not rectangular, there is no NumPy translation
<Array [[1, 2, 3], [], [4, 5]] type='3 * var * int64'>
>>> np.asarray(case5)   # because dtype="O" has been excluded
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jpivarski/miniconda3/lib/python3.8/site-packages/numpy/core/_asarray.py", line 83, in asarray
    return array(a, dtype, copy=False, order=order)
  File "/home/jpivarski/irishep/awkward-1.0/awkward1/highlevel.py", line 1310, in __array__
    return awkward1._connect._numpy.convert_to_array(self._layout, args, kwargs)
  File "/home/jpivarski/irishep/awkward-1.0/awkward1/_connect/_numpy.py", line 16, in convert_to_array
    out = awkward1.operations.convert.to_numpy(layout, allow_missing=False)
  File "/home/jpivarski/irishep/awkward-1.0/awkward1/operations/convert.py", line 313, in to_numpy
    return to_numpy(array.toRegularArray(), allow_missing=allow_missing)
ValueError: in ListOffsetArray64, cannot convert to RegularArray because subarray lengths are not regular

So these arrays could have dtype and shape properties, which would correspond to their NumPy meanings when these arrays have NumPy equivalents. But I'm wary of defining properties that can raise exceptions: such a thing would look like a bug, even though it's intentional (to avoid accidentally converting to Python objects and getting a surprising performance drop).

Apache Arrow must have this issue as well...

>>> np.asarray(ak.to_arrow(case5))
array([array([1, 2, 3]), array([], dtype=int64), array([4, 5])],
      dtype=object)

Aha—I wasn't expecting that. Converting to dtype="O" arrays is a tradeoff, and not everyone's going to want to make it.

[asmeurer]

Speaking from the position of writing the test suite, the dtype and shape attributes are needed to check if the library conforms to the type promotion rules and broadcasting rules, respectively. Being able to create arrays with the dtype keyword argument and dtype literals like mod.ones(dtype=mod.float64) is also important for testing purposes (I don't know if awkward has those).

(an aside: why does the library name for Awkward have a number in it? I thought that was a typo the first time I saw it, until I noticed it was like that everywhere)

[jpivarski]

It doesn't have a dtype constructor keyword or (as yet) a ones function or method, though the latter is possible and has been requested: scikit-hep/awkward#493. But incidentally, we do have tests verifying that it agrees with NumPy's type promotion and broadcasting. These are things that can be made consistent, even without a dtype and shape.

This was why I wasn't sure how relevant Awkward Array would be to this project: some basic expectations can't be met. But maybe it could be useful to define a subset of the API you have in mind?

As for the number in the name, the original Awkward Array had an entirely different implementation and somewhat different interface (before strict adherence with NumPy was a goal), so the "1.0" version was developed beside it, to let users transition before the interface changes. But now appearances of import awkward1 have exceeded import awkward in GitHub, so we've scheduled the name change for Dec 1. In less than a month, the "1" will drop and users will have to opt into the old library with import awkward0 as awkward, if they still need it.

[rgommers]

This was why I wasn't sure how relevant Awkward Array would be to this project: some basic expectations can't be met. But maybe it could be useful to define a subset of the API you have in mind?

I think Awkward Array is too different to really be able to adopt the API standard - properties like shape and dtype are quite fundamental. That said, the API standard itself may still be relevant to Awkward Array, in a similar way as for array libraries written in other languages - it provides a good selection of what a "core array API" should contain, with reasonable choices for syntax and semantics that are largely numpy-compatible but deviate where needed.

However, the same would be true of Apache Arrow, and I see that they're represented in the list of members. [...] Maybe the role that projects like Awkward Array and Apache Arrow can play would be to define subsets of array APIs that can apply to non-rectilinear arrays?

We listed Apache Arrow indeed, it's relevant to this effort but more a set of technology on which a dataframe library can be built (and that library could adopt a standard dataframe API) rather than a dataframe library itself. ONNX falls also in that bucket I'd say.

Should we have listed them - maybe, maybe not - it's not completely black and white. We have members who are maintainers, and also are Pandas/MXNet maintainers. It seemed like a relevant affiliation to list at that point.

Perhaps "allowing for irregularity" is another subset?

While I appreciate Awkward Array, I doubt this would be helpful given that there are few (no?) other implementations providing irregular arrays, certainly not any widely used ones I think.

Also note that "immutable arrays" are not a subset; instead the full API standard was designed with the requirement in mind that it must work for immutable array implementations.

[jpivarski]

Follow up on this topic at scikit-hep/ragged#6