Nesting of containers

[OriolAbril]

I am a bit confused by the behaviour of natural language containers. Currently, list of list of str is not valid, but arguably more convoluted alternatives like list of (int, list of int)->list[int, list[int]] (not sure this is a valid type hint) or list of (str or list of str)->list[str | list[str]] are.

I think this comes from the "of qualname" piece in qualname "of" qualname _PLURAL_S? as opposed to "of type" but then there is also qualname "of" "(" union ")" with union being defined with type, not qualname and also the tuple version qualname "of" "(" type ("," type)+ ")" also with type instead of qualname.

I do have some list of list of str but I completely understand that nesting these types too much becomes incomprehensible so we want to discourage users from such patterns. I am not sure what some good alternatives could be, but I think the current situation has the potential to be quite confusing.

[lagru]

Oje, thanks for the feedback.

list of list of str – I'm not sure that I'd want to support this. list of (list of str) might not be so bad. But even then, I explicitly tried to limit nesting of natural language containers in #37. Natural language is nice to use on the first level. But I don't think nesting is them is readable or common in our ecosystem? Do you think I should nevertheless support these? I considered allowing this and raising a warning. But restricting the grammar seemed like the fastest way forward.

list of (int, list of int) should definitely not work. But checking with the current grammar, it does. So I'd consider that a bug. I don't think that
qualname "of" qualname _PLURAL_S? is the culprit. qualname only allows dotted names. But I think the other variants using union and type are the ones letting it through.

[OriolAbril]

I do use one level of nesting somewhat often, e.g. list of (str or list of str), dict of {hashable : int or array-like of int} (does currently work with docstub) or the list of list of str with or without parenthesis for the inner list of str, but happy to enforce parenthesis for this one if needed.

I haven't explicitly checked this yet but answering here made me think about it, I'll try to check docstub behaviour and community usage next week. In many of these nested types, the structure and content of the argument is more/as important as the type itself, so I do think it is quite common to use the type info for the first level and then explain in the description the full type+structure. For example, I think sequence of sequence is more common than sequence of sequence of hashable but then the description would explain what the hashable "leaves" mean, what they can be and the lenghts/nesting structure if there are restrictions.

I also do seem to remember that I had one method with a type hint and an equivalent through aliases version in the docstring and I got a warning from docstub. How would you envision support for these cases in docstub, having the full type hint in the docstring and no type hint or would having a superset of the type in the docstring plus type hint with full detail?

---

I don't think that qualname "of" qualname _PLURAL_S? is the culprit. qualname only allows dotted names. But I think the other variants using union and type are the ones letting it through.

Me too, I am quite sure the tuple one is the culprit. I think given the specialness of tuple among the its cases might need to use "tuple" instead of qualname. I think ellipsis wouldn't be allowed with other containers either but the current syntax would parse them as correct.

[lagru]

I also do seem to remember that I had one method with a type hint and an equivalent through aliases version in the docstring and I got a warning from docstub.

Right now, docstub always gives priority to the type annotation in the docstring if it exits. If an inline type also exists, it warns and replaces. Though, that was just a default I started with. The case of floats in literals pointed out in #44 makes me think that the other way round might actually be smarter. That way, one can always fallback to inline literals if docstub isn't expressive enough. 🤔

[lagru]

Okay, after thinking about this some more, I'm content with leaving the current behavior. If users really want to nest natural language containers it actually works. And I guess we can introduce warnings down the line if users seem to abuse it.

[lagru]

Feel very welcome to reopen if there's more to discuss here or if you don't agree!

[OriolAbril]

I think there are two things that need addressing,but I can open specific issues for them:

• The main one is allowing list of (int, str) and generating invalid type hints from that list[int, str], without even printing a warning. I would propose changing the qualname in the tuple literals to strictly tuple (or _TUPLE similar to _OR in case there are multiple types that allow it)
• The other is more about consistency. Personally I am perfectly fine with two levels of nesting, but I also understand they get confusing quickly and already never go over 2 levels, so I would be fine with enforcing 1 level natural language or type hints. However, that is not the case, through the qualname of ( union ) infinite nesting is allowed as long as the inner types are a union.
  That means that list of (list of int) (the type signature of axes argument in numpy ufuncs, I also use it with string to mean the same but for xarray input using named dimensions) is not allowed whereas list of (int or list of (int or list of int)) is perfectly valid

[lagru]

list of (int, str)

That's allowed because docstub isn't taking the container name into account in the grammar. We have no robust way of really knowing if some_container should be allowed to use more than one parameter. If we did, we'd have re-implemented a type checker. I'd rather encourage users to check their generated stubs with a proper type checker afterwards.

I would propose changing the qualname in the tuple literals to strictly tuple (or _TUPLE similar to _OR in case there are multiple types that allow it)

I don't think that approach scales well. That is currently done for the array expression rule. But what if people want to use xarray? I think the type system allows declaring new types with as many parameters as desired. So we'd have to give users the option to allow or forbid names for certain rules. I'm not sure it's worth giving users the option to configure that. It would be easiest to just let type checkers catch that.

That said, I'm not opposed to raising warnings for common pitfalls or bad practices. E.g. I think it could make sense to treat the array expression in a special way. Though, it's just not my biggest priority to work on right now. 😅

list of (list of int)

After having come around to allowing nesting natural language expressions, this should definitely work! Using

natlang_container: ...
    | qualname "of" "(" (type | union) ")"

Seems to fix that. We should probably extend the test suite significantly since you seem to find new cases that should work but don't. 😅