memoryview doesn't allow tuple-indexing

[pitrou]

BPO	23632
Nosy	@pitrou, @bitdancer, @skrah, @serhiy-storchaka
Files	memoryview_tuple_indexing.patch memoryview_tuple_indexing2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-03-19.23:11:26.897>
created_at = <Date 2015-03-10.21:46:41.107>
labels = ['type-feature']
title = "memoryview doesn't allow tuple-indexing"
updated_at = <Date 2015-03-19.23:11:26.895>
user = 'https://github.com/pitrou'

bugs.python.org fields:

activity = <Date 2015-03-19.23:11:26.895>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2015-03-19.23:11:26.897>
closer = 'pitrou'
components = []
creation = <Date 2015-03-10.21:46:41.107>
creator = 'pitrou'
dependencies = []
files = ['38431', '38550']
hgrepos = []
issue_num = 23632
keywords = ['patch']
message_count = 18.0
messages = ['237814', '237820', '237926', '237927', '237928', '237930', '237931', '237932', '237933', '237934', '237935', '238093', '238471', '238475', '238479', '238492', '238580', '238581']
nosy_count = 5.0
nosy_names = ['pitrou', 'r.david.murray', 'skrah', 'python-dev', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue23632'
versions = ['Python 3.5']

[pitrou]

It is a bit of pity. A least non-sliced indexing should be able to work:

>>> import numpy as np
>>> a = np.arange(10)
>>> memoryview(a)[0]
0
>>> a = np.arange(10).reshape((2,5))
>>> a[0,1]
1
>>> memoryview(a)[0,1]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: memoryview: invalid slice key

[pitrou]

Here is a patch.

[bitdancer]

Aren't there wider implications of python starting to support tuple indexing? If we make this work, aren't people going to expect [[1,2], [3,4]][0,1] to work?

[pitrou]

If we make this work, aren't people going to expect [[1,2], [3,4]][0,1] to work?

Or even [[1,2], [3,4]][*(0,1)] :-)

But seriously, I don't know. memorview is a pretty specialized object, its semantics have more to do with Numpy (which has been a major inspiration and use case for the buffer protocol's design) than standard Python containers. I wouldn't expect a list of lists to support tuple indexing.

[skrah]

Multi-dimensional slicing is explicitly mentioned in PEP-3118, so
I guess the intention was to cover this use case as well.

[skrah]

To be sure, the PEP contains some odd proposals like "Unpacking a long-double will return a decimal object", but the one in this issue seems reasonable.

[serhiy-storchaka]

May be add support of multi-dimensional sub-views?

>>> m = memoryview(bytearray(range(12)))
>>> m2 = m.cast('B', (3,4))
>>> m2[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NotImplementedError: multi-dimensional sub-views are not implemented

Then we could use m2[0][1].

[pitrou]

Sub-views are not as easily implemented. Besides, they create an intermediate object which is immediately thrown away. And tuple indexing is a very common idiom in the Numpy universe.

[skrah]

sub-views and multi-dimensional slicing are not that bad: They're already implemented in _testbuffer (indeed with intermediate objects though).

[pitrou]

Well, we could certainly do both :) I don't think we should weigh this issue with separate features, though.

[skrah]

Yes, to be clear I'm +1 on this specific feature -- and separate issues. :)

[skrah]

It turns out that msg237933 was too simplistic: There are some
intricacies of numpy array indexing that *are* very complex:

http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html

In particular, x[::2, 1, ::2] is not necessarily equal to
x[::2][1][::2].

[pitrou]

Do you want to review the patch?

[serhiy-storchaka]

I left few nitpicks on Rietveld, but in general the patch LGTM.

Oh, needed a documentation.

[pitrou]

Updated patch addressing Serhiy's comments.

[serhiy-storchaka]

It would be good to add examples with multidimensional memoryviews. For example:

>>> a = array.array('l', [-11111111, 22222222, -33333333, 44444444])
>>> m = memoryview(a).cast('B').cast('l', [2, 2])
>>> m[0, 0]
-11111111
>>> m[1, 0]
-33333333
>>> m[-1, -1]
44444444

[python-dev]

New changeset 7d4eb5902f82 by Antoine Pitrou in branch 'default':
Issue bpo-23632: Memoryviews now allow tuple indexing (including for multi-dimensional memoryviews).
https://hg.python.org/cpython/rev/7d4eb5902f82

[pitrou]

I didn't add the 2d example as the double cast is quite ugly. Probably we should allow casting when the format stays the same.