PyJWT allows for the optional installation of the cryptography package via pip install pyjwt[crypto]. This package is required for decoding RSA256-encoded tokens. Other packages like django-allauth specify the pyjwt[crypto] dependency directly.
It's currently necessary to install this package separately, but maybe it would be a good idea to allow for optional installation just like PyJWT.
Something like pip install djangorestframework-simplejwt[crypto]. I'm willing to work on a PR for this if it seems a good idea.