Question regarding typing a values_list with (flat=True)

[rvanlaar]

Bug report

What's wrong

I can't set the correct type on the return value of a values list.
Simplified example:

class Question:
    category = models.TextField()

def get_result() -> QuerySet[Any]:
    result = Exam.objects.get().questions.values_list("category", flat=True)
    reveal_type(result) # Revealed type is "django.db.models.query._QuerySet[backend.models.Question, Union[builtins.str, None]]"
    return result
reveal_type(get_result()) # Revealed type is "django.db.models.query._QuerySet[Any, Any]"

How is that should be

The calling code needs to loop over the categories, i.e. it needs a sequence of type str.
What is the best way to annotate the get_result function?

System information

• OS: debian 11
• python version: 3.9.7
• django version: 3.2.13
• mypy version: 0.961
• django-stubs version: 1.12.0
• django-stubs-ext version: 0.4.0

[sobolevn]

You can use Sequence[str] or List[str] instead.

I am going to close this as non issue. Please, feel free to reopen if something in my answer does not work for you.

[rvanlaar]

Thank you for the response. Setting Sequence or List as a return type doesn't resolve the issue.

I've create a new more contained example:

models.py:

from django.db import models
from typing import Sequence

class Exam(models.Model):
    name = models.TextField()

    @property
    def category(self) -> Sequence[str]:
        return self.question_set.values_list("category", flat=True)

class Question(models.Model):
    category = models.TextField()
    exam = models.ForeignKey(Exam, on_delete=models.CASCADE)

Which has the following error:
error: Incompatible return value type (got "_QuerySet[Question, str]", expected "Sequence[str]")

[sobolevn]

Hm, looks like our _QuerySet is not compatible with Sequence protocol 🤔

[UnknownPlatypus]

I'm having a similar issue so I did some digging:

The custom queryset representation is currently:

class _QuerySet(Generic[_T, _Row], Collection[_Row], Reversible[_Row], Sized):

To allow your syntax, we would need:

class _QuerySet(Generic[_T, _Row], Sequence[_Row]):

From the docs, a Sequence inherit from Collection and Reversible and adds __getitem__, count and index methods. (See Sequence implementation in python source.) Because Django Queryset implement the __getitem__ hook but nor the count neither the index ones, It's not strictly matching the Sequence protocol but it's quite close.

I see a few options to solve your case:

1. Use ValuesQuerySet from django_stubs_ext instead :

.values_list("category", flat=True) -> ValuesQuerySet[Exam, str]
.values_list("category") -> ValuesQuerySet[Exam, Tuple[str]]
.values("category") -> ValuesQuerySet[Exam, TypedDict(...)]

PS: Took me some time to understand I could use that, and this issue confirm my feeling. Maybe a doc section about type annotation for .values methods using the above syntax would be nice ? Or I guess better error message could point user in the right direction (I've seen this issue but I'm not sure I understand if the proposed idea is doable) . I'm happy to send a MR if you're interested.

2. Change the stubs to use a Sequence: It would allow the above syntax but is not accurate, mypy would not detect wrong .index() or .count(val) call on a QuerySet and following .filter()/.order_by()/... etc would raise issues while they are allowed by django. It doesn't seems like a good idea to me.

[jorenham]

I experimented a bit, and quickly realised that the T in QuerySet[T] shouldn't be bound by Model:

>>> type(Food.objects.all()) is type(Food.objects.values_list('spam', flat=True))
True
>>> set(dir(Food.objects.all())) == set(dir(Food.objects.values_list('spam', flat=True)))
True
>>> from django.db.models import QuerySet
>>> isinstance(Food.objects.values_list('spam', flat=True), QuerySet)
True

Clearly, they are exactly the same type (i.e. QuerySet). And, to my surprise, there appears to be none of that runtime magic type voodoo that Django usually practises

So IMHO, the obvious way to type Food.objects.values_list('spam', flat=True) here, is as a QuerySet[Spam], where food.spam: Spam and food: Food.
Unless I'm missing something, this issue can be resolved by simply removing the bound=Model of the TypeVar in _BaseQuerySet and her kids -- no need to bother with the wobbly type-pyramid of collections.abc 🙃.

[flaeppe]

I'm thinking this was closed via suggestion 1. in #1047 (comment). Nowadays ValuesQuerySet is simply QuerySet (via #2104), also see #2194 (comment)

Let me know if I this is an incorrect assumption.

[jorenham]

@flaeppe Unless I'm missing something, I believe that the QuerySet type parameter is still bound to Model. See #1047 (comment)

[flaeppe]

Yes, it's still bound to Model but the second type argument isn't bound. So what I think should work is to type the following signature

@property
def category(self) -> QuerySet[Question, str]: ...

[jorenham]

Yes, it's still bound to Model but the second type argument isn't bound. So what I think should work is to type the following signature

@property
def category(self) -> QuerySet[Question, str]: ...

Ok great, than I suppose it's indeed solved 🎉