__len__ can't return big numbers

[cool-RR]

BPO	21444
Nosy	@rhettinger, @jcea, @mdickinson, @bitdancer, @cool-RR, @4kir4
PRs	bpo-12159: document sys.maxsize limit in len() function reference #17934

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-05-08.21:17:54.444>
created_at = <Date 2014-05-05.23:11:11.092>
labels = ['interpreter-core', 'type-feature']
title = "__len__ can't return big numbers"
updated_at = <Date 2020-01-10.12:13:55.203>
user = 'https://github.com/cool-RR'

bugs.python.org fields:

activity = <Date 2020-01-10.12:13:55.203>
actor = 'Zac Hatfield-Dodds'
assignee = 'none'
closed = True
closed_date = <Date 2014-05-08.21:17:54.444>
closer = 'benjamin.peterson'
components = ['Interpreter Core']
creation = <Date 2014-05-05.23:11:11.092>
creator = 'cool-RR'
dependencies = []
files = []
hgrepos = []
issue_num = 21444
keywords = []
message_count = 7.0
messages = ['217957', '217958', '217974', '217983', '217988', '218002', '218124']
nosy_count = 6.0
nosy_names = ['rhettinger', 'jcea', 'mark.dickinson', 'r.david.murray', 'cool-RR', 'akira']
pr_nums = ['17934']
priority = 'low'
resolution = 'wont fix'
stage = None
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue21444'
versions = ['Python 3.5']

[cool-RR]

I want to use big numbers for length.

>>> class A:
...     __len__ = lambda self: 10 ** 20
>>> len(A())
Traceback (most recent call last):
  File "<pyshell#5>", line 1, in <module>
    len(A())
OverflowError: cannot fit 'int' into an index-sized integer

[bitdancer]

While this is classed as a CPython implementation detail (see bpo-15718) it doesn't sound like it is likely to be changed (see bpo-2723).

[mdickinson]

Whoops; sorry -- accidental title change by typing __len__ into something that wasn't the search box.

Stupid fingers...

(I suspect this issue is a duplicate of an existing issue.)

[bitdancer]

Mark: I thought it was too, but the two I noted were the closest I could find. Maybe you'll find something even more on point :)

[4kir4]

If len() signature can't be changed to return Python int objects (unlimited) then the OverflowError may contain the actual .length
property instead (based on msg66459 by Antoine Pitrou)

operator.length():

    def length(sized):
        """Return the true (possibly large) length of `sized` object.

    It is equivalent to len(sized) if len doesn't raise 
    OverflowError i.e., if the length is less than sys.maxsize on
    CPython; otherwise return OverflowError.length attribute
    """
    try:
        return len(sized)
    except OverflowError as e:
        return e.length

[mdickinson]

That's pretty evil. :-)

[rhettinger]

I recommend this be closed: too much impact on existing code for too little benefit.

CPython has historically imposed some artificial implementation specific details in order make the implementation cleaner and faster internally (i.e. a limit on the number of function arguments, sys.maxsize limits, etc.)

[rhettinger]

If people really care about this (I'm not sure they do), there is a possible general solution.

1. Add an OverflowError fallback code path to the builtin len() function. That path can then compute arbitrarily large sizes from objects that have integer start, stop, and step attributes. The fixes cases like len(range(1 << 100)).

2. The builtin len() could also be built out with a fallback that calls __len__ method directly. This won't help with builtin types like rangeobject, but it would help with user defined types:

class A:
    def __len__(self):
        return 1 << 66

>>> a.__len__()
73786976294838206464

These two measures would mitigate the limitations imposed by CPython's system of internal slots like mp_length and sq_length. However, I don't think any work should be done until we've ascertained whether we're partially solving a real problem or just addressing a theoretical concern.