OAuth Server Error 500

[edpomacedo]

Hi there !

I've managed to set it up following the Deploying on WMF Toolforge, but locally.

Now, at the stage I'm creating the User, after creating the OAuth keys and passing them to the secret.py, when I try to login for the first time (before creating the staff user within the shell) I get the following error (with DEBUG = True):

Django Version: | 2.1.15
-- | --
TypeError
Direct assignment to the forward side of a many-to-many set is prohibited. Use groups.set() instead.
/var/www/venv/lib/python3.6/site-packages/django/db/models/fields/related_descriptors.py in __set__, line 537
/var/www/venv/bin/python
3.6.9
['/var/www/src',  '/usr/lib/python36.zip',  '/usr/lib/python3.6',  '/usr/lib/python3.6/lib-dynload',  '/var/www/venv/lib/python3.6/site-packages']
Sat, 16 Apr 2022 02:23:54 +0000

Here is the Traceback:

Django Version: 2.1.15
Python Version: 3.6.9
Installed Applications:
('django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'social_django',
 'django_filters',
 'rest_framework',
 'store',
 'revert',
 'tagging',
 'django_extensions')
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'social_django.middleware.SocialAuthExceptionMiddleware']



Traceback:

File "/var/www/venv/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
  34.             response = get_response(request)

File "/var/www/venv/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  126.                 response = self.process_exception_by_middleware(e, request)

File "/var/www/venv/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  124.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/var/www/venv/lib/python3.6/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
  44.         response = view_func(request, *args, **kwargs)

File "/var/www/venv/lib/python3.6/site-packages/django/views/decorators/csrf.py" in wrapped_view
  54.         return view_func(*args, **kwargs)

File "/var/www/venv/lib/python3.6/site-packages/social_django/utils.py" in wrapper
  46.             return func(request, backend, *args, **kwargs)

File "/var/www/venv/lib/python3.6/site-packages/social_django/views.py" in complete
  33.                        *args, **kwargs)

File "/var/www/venv/lib/python3.6/site-packages/social_core/actions.py" in do_complete
  45.         user = backend.complete(user=user, *args, **kwargs)

File "/var/www/venv/lib/python3.6/site-packages/social_core/backends/base.py" in complete
  40.         return self.auth_complete(*args, **kwargs)

File "/var/www/venv/lib/python3.6/site-packages/social_core/utils.py" in wrapper
  247.             return func(*args, **kwargs)

File "/var/www/venv/lib/python3.6/site-packages/social_core/backends/oauth.py" in auth_complete
  181.         return self.do_auth(access_token, *args, **kwargs)

File "/var/www/venv/lib/python3.6/site-packages/social_core/utils.py" in wrapper
  247.             return func(*args, **kwargs)

File "/var/www/venv/lib/python3.6/site-packages/social_core/backends/oauth.py" in do_auth
  192.         return self.strategy.authenticate(*args, **kwargs)

File "/var/www/venv/lib/python3.6/site-packages/social_django/strategy.py" in authenticate
  105.         return authenticate(*args, **kwargs)

File "/var/www/venv/lib/python3.6/site-packages/django/contrib/auth/__init__.py" in authenticate
  73.             user = backend.authenticate(request, **credentials)

File "/var/www/venv/lib/python3.6/site-packages/social_core/backends/base.py" in authenticate
  80.         return self.pipeline(pipeline, *args, **kwargs)

File "/var/www/venv/lib/python3.6/site-packages/social_core/backends/base.py" in pipeline
  83.         out = self.run_pipeline(pipeline, pipeline_index, *args, **kwargs)

File "/var/www/venv/lib/python3.6/site-packages/social_core/backends/base.py" in run_pipeline
  113.             result = func(*args, **out) or {}

File "/var/www/venv/lib/python3.6/site-packages/social_core/pipeline/user.py" in user_details
  119.         setattr(user, name, value)

File "/var/www/venv/lib/python3.6/site-packages/django/db/models/fields/related_descriptors.py" in __set__
  537.             % self._get_set_deprecation_msg_params(),

Exception Type: TypeError at /oauth/complete/mediawiki/
Exception Value: Direct assignment to the forward side of a many-to-many set is prohibited. Use groups.set() instead.

Do you guys have any idea how to fix it?

Thanks in advance

[edpomacedo]

Fixed !

Add the following to the secret.py, as stated at social-app-django:

SOCIAL_AUTH_PROTECTED_USER_FIELDS = ['groups']

And under try:at __init__.py, add the following:

from .secret import SOCIAL_AUTH_PROTECTED_USER_FIELDS

[wetneb]

Thanks a lot! Perhaps that is worth adding to the documentation?
Also, I am curious if you are deploying this for a non-WMF wiki, if you are deploying it outside of the toolforge?

[edpomacedo]

You're welcome!

I'm working on a Wikibase to store Brazilian legal data. I had to set up my own in order to have strings and multilingual texts with more than 250 characters, so the content makes sense (i.e. case abstract).

After setting up the QuickStatements on the server and perform a few tests, I was able to import some batches to the Wikibase, but they were not recorded within QuickStatements. Also, I had to keep clicking the "Execute"/"Stop" button until the last item of the batch.

I saw that it has connections to the EditGroups tool, so I'm trying to deploy it too and see if things works out.

Right now I'm facing nginx/permissions problems regarding the /admin/store/tool. As soon as I succeed, I'll paste the gist here and you see if it fits within the documentation. This is the Traceback:

[16/Apr/2022 19:04:24] "GET / HTTP/1.0" 200 3933
Forbidden (Permission denied): /admin/store/tool/
Traceback (most recent call last):
  File "/var/www/venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/var/www/venv/lib/python3.6/site-packages/django/core/handlers/base.py", line 126, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/var/www/venv/lib/python3.6/site-packages/django/core/handlers/base.py", line 124, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/var/www/venv/lib/python3.6/site-packages/django/contrib/admin/options.py", line 604, in wrapper
    return self.admin_site.admin_view(view)(*args, **kwargs)
  File "/var/www/venv/lib/python3.6/site-packages/django/utils/decorators.py", line 142, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/var/www/venv/lib/python3.6/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/var/www/venv/lib/python3.6/site-packages/django/contrib/admin/sites.py", line 223, in inner
    return view(request, *args, **kwargs)
  File "/var/www/venv/lib/python3.6/site-packages/django/utils/decorators.py", line 45, in _wrapper
    return bound_method(*args, **kwargs)
  File "/var/www/venv/lib/python3.6/site-packages/django/utils/decorators.py", line 142, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/var/www/venv/lib/python3.6/site-packages/django/contrib/admin/options.py", line 1683, in changelist_view
    raise PermissionDenied
django.core.exceptions.PermissionDenied
[16/Apr/2022 19:04:33] "GET /admin/store/tool/ HTTP/1.0" 403 22

Any tips or guidance will be very much appreciated!

[wetneb]

If it is just the admin interface that you fail to reach, then the user you are connected with probably does not have admin/superuser rights (you could try setting both).
Otherwise if it is for the entire app, then it could be that you have not set up your ALLOWED_HOSTS properly (in your Django settings).

But I should warn you that this tool relies on the WMF Event Stream to stay in sync with the Wikibase instance and I am not aware of any Wikibase instance that also deployed something similar - I suspect it is quite some effort to replicate Wikimedia's infrastructure.

[wetneb]

(See #5)

[edpomacedo]

Thank you for the heads up!

You are right. I'm afraid it will not be possible to make it work. Even with the ./listener.sh and the ./celery.sh running, I did not see any changes publicly.

Regarding the permission, I have access to the admin panel, but unable to edit or view anything there.

I would love to try an alternative. Btw, I asked for membership at toolforge.

Do you think the gist is still necessary? I've installed virtualenv, redis and some other dependencies to resume the requirements.txt that were returning errors. If you think so, I'll leave it here.

[wetneb]

Ok! Well I do not think it would be that hard to add support for ingesting changes by polling the recent changes feed. But I just have not done it because I have no need for it myself. But if you are motivated I am keen to help you!

I have access to the admin panel, but unable to edit or view anything there.

Initially there should not be much to see as the database is empty, but normally you should be able to create a new tool.

Do you think the gist is still necessary?

Which gist?

[edpomacedo]

Awesome! Here it is: EditGroups. Something is off because of my tweaks. I'm considering to restore the VM and start again.

When I'm logged in, I try to go to /admin/store/tool/, it returns 403 Forbidden. I've tried to ./manage.py createsuperuser and pass my username, it says "is already taken". So I'm wondering: what if I create the superuser before I log in via OAuth for the first time? Maybe something related to my nginx .conf? I can't really tell.

The gist I'm referring to is the one that records my steps to reproduce the application, more or less like this:

# REDIS

sudo apt-get install redis-server

sudo nano /etc/redis/redis.conf

supervised systemd
bind 127.0.0.1 ::1
requirepass mypass

sudo systemctl restart redis.service

sudo systemctl status redis

...

[edpomacedo]

Turning on DEBUG mode and logged out, it looks like it's something regarding redis:

Authentication required.

Request Method: | GET
-- | --
http://weg.ordenamento.com.br/admin/store/tool/
2.1.15
AuthenticationError
Authentication required.
/var/www/venv/lib/python3.6/site-packages/redis/connection.py in read_response, line 340
/var/www/venv/bin/python
3.6.9
['/var/www/venv/src',  '/usr/lib/python36.zip',  '/usr/lib/python3.6',  '/usr/lib/python3.6/lib-dynload',  '/var/www/venv/lib/python3.6/site-packages']
Sun, 17 Apr 2022 12:49:18 +0000

Traceback:

Request Method: GET
Request URL: http://weg.ordenamento.com.br/admin/store/tool/

File "/var/www/venv/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
  34.             response = get_response(request)

File "/var/www/venv/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  126.                 response = self.process_exception_by_middleware(e, request)

File "/var/www/venv/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  124.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/var/www/venv/lib/python3.6/site-packages/django/contrib/admin/options.py" in wrapper
  604.                 return self.admin_site.admin_view(view)(*args, **kwargs)

File "/var/www/venv/lib/python3.6/site-packages/django/utils/decorators.py" in _wrapped_view
  142.                     response = view_func(request, *args, **kwargs)

File "/var/www/venv/lib/python3.6/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
  44.         response = view_func(request, *args, **kwargs)

File "/var/www/venv/lib/python3.6/site-packages/django/contrib/admin/sites.py" in inner
  223.             return view(request, *args, **kwargs)

File "/var/www/venv/lib/python3.6/site-packages/django/utils/decorators.py" in _wrapper
  45.         return bound_method(*args, **kwargs)

File "/var/www/venv/lib/python3.6/site-packages/django/utils/decorators.py" in _wrapped_view
  142.                     response = view_func(request, *args, **kwargs)

File "/var/www/venv/lib/python3.6/site-packages/django/contrib/admin/options.py" in changelist_view
  1804.             'selection_note': _('0 of %(cnt)s selected') % {'cnt': len(cl.result_list)},

File "/var/www/venv/lib/python3.6/site-packages/django/db/models/query.py" in __len__
  250.         self._fetch_all()

File "/var/www/venv/lib/python3.6/site-packages/django/db/models/query.py" in _fetch_all
  1186.             self._result_cache = list(self._iterable_class(self))

File "/var/www/venv/lib/python3.6/site-packages/caching/base.py" in __iter__
  124.         cached = cache.get(query_key)

File "/var/www/venv/lib/python3.6/site-packages/redis_cache/backends/base.py" in wrapped
  30.             return method(self, client, key, *args, **kwargs)

File "/var/www/venv/lib/python3.6/site-packages/redis_cache/backends/base.py" in get
  267.         return self._get(client, key, default)

File "/var/www/venv/lib/python3.6/site-packages/redis_cache/backends/base.py" in _get
  255.         value = client.get(key)

File "/var/www/venv/lib/python3.6/site-packages/redis/client.py" in get
  1606.         return self.execute_command('GET', name)

File "/var/www/venv/lib/python3.6/site-packages/redis/client.py" in execute_command
  898.         conn = self.connection or pool.get_connection(command_name, **options)

File "/var/www/venv/lib/python3.6/site-packages/redis/connection.py" in get_connection
  1192.             connection.connect()

File "/var/www/venv/lib/python3.6/site-packages/redis/connection.py" in connect
  567.             self.on_connect()

File "/var/www/venv/lib/python3.6/site-packages/redis/connection.py" in on_connect
  664.             if nativestr(self.read_response()) != 'OK':

File "/var/www/venv/lib/python3.6/site-packages/redis/connection.py" in read_response
  739.             response = self._parser.read_response()

File "/var/www/venv/lib/python3.6/site-packages/redis/connection.py" in read_response
  340.                 raise error

Exception Type: AuthenticationError at /admin/store/tool/
Exception Value: Authentication required.

When I'm logged in, the error related to the /admin/store/tool/ is the same reproduced above. And the admin panel returns the following:

Site administration
You don't have permission to view or edit anything.

My actions
None available

[wetneb]

So, if you log in via OAuth, that will create a user account that will be different than any previously created account.
Now that you are in this situation, there are two solutions:

1. Log out of EditGroups and of the MediaWiki instance. Go to /admin/ in the EditGroups instance. There you should see a login form that is not the OAuth one (Django's native one). There you can use credentials that you have created with ./manage.py createsuperuser

or

2. Open a django shell with ./manage.py shell and input the following lines:

from django.contrib.auth.models import User
User.objects.all().update(is_superuser=True, is_admin=True)

This will make all existing users in EditGroups admins and superuser (obviously you only want to do that if no non-admin users have logged in via OAuth so far, you can check which users exist before executing the second line with User.objects.all()).

[edpomacedo]

Good news! With the root user created through step 1, I was able to navigate through the panel and even saw the tool creation form. Thats's great! But it looks like "pure html", nothing like the django admin panel.

Then, for step 2, i got the following error:

Traceback (most recent call last):
  File "/var/www/venv/lib/python3.6/site-packages/django/db/models/options.py", line 564, in get_field
    return self.fields_map[field_name]
KeyError: 'is_admin'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/var/www/venv/lib/python3.6/site-packages/django/db/models/query.py", line 689, in update
    query.add_update_values(kwargs)
  File "/var/www/venv/lib/python3.6/site-packages/django/db/models/sql/subqueries.py", line 117, in add_update_values
    field = self.get_meta().get_field(name)
  File "/var/www/venv/lib/python3.6/site-packages/django/db/models/options.py", line 566, in get_field
    raise FieldDoesNotExist("%s has no field named '%s'" % (self.object_name, field_name))
django.core.exceptions.FieldDoesNotExist: User has no field named 'is_admin'

Nonetheless, checking the admin/auth/user/ with the root user, I can see that my MediaWiki/OAuth user has Staff status true:

Username
Email address
First name
Last name
Staff status
	EdpoMacedo	mail@mail.com	 	 	True
	root	mail@mail.com	 	 	True

Sadly, the admin/store/tool/ for the root user returns Server Error (500). But when I point to admin/store/tool/add/, I can see the form described at Configuring the supported tools.

Should I start all over from scratch? Because with my MediaWiki user, I still have no "permission to view or edit anything".

[wetneb]

Sorry, it was is_staff, not is_admin. I am not sure about your 500 error, it is probably something we can debug if you set DEBUG = True in your Django settings.

[edpomacedo]

NICE! Now my MediaWiki user has full access to the admin panel!

Now, browsing to the /admin/store/tool/ with DEBUG on, here's the error:

AuthenticationError at /admin/store/tool/
Authentication required.
Request Method:	GET
Request URL:	http://weg.ordenamento.com.br/admin/store/tool/
Django Version:	2.1.15
Exception Type:	AuthenticationError
Exception Value:	
Authentication required.
Exception Location:	/var/www/venv/lib/python3.6/site-packages/redis/connection.py in read_response, line 340
Python Executable:	/var/www/venv/bin/python
Python Version:	3.6.9
Python Path:	
['/var/www/venv/src',
 '/usr/lib/python36.zip',
 '/usr/lib/python3.6',
 '/usr/lib/python3.6/lib-dynload',
 '/var/www/venv/lib/python3.6/site-packages']
Server time:	Sun, 17 Apr 2022 14:25:41 +0000

Traceback:

Environment:


Request Method: GET
Request URL: http://weg.ordenamento.com.br/admin/store/tool/

Django Version: 2.1.15
Python Version: 3.6.9
Installed Applications:
('django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'social_django',
 'django_filters',
 'rest_framework',
 'store',
 'revert',
 'tagging',
 'django_extensions')
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'social_django.middleware.SocialAuthExceptionMiddleware']



Traceback:

File "/var/www/venv/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
  34.             response = get_response(request)

File "/var/www/venv/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  126.                 response = self.process_exception_by_middleware(e, request)

File "/var/www/venv/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  124.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/var/www/venv/lib/python3.6/site-packages/django/contrib/admin/options.py" in wrapper
  604.                 return self.admin_site.admin_view(view)(*args, **kwargs)

File "/var/www/venv/lib/python3.6/site-packages/django/utils/decorators.py" in _wrapped_view
  142.                     response = view_func(request, *args, **kwargs)

File "/var/www/venv/lib/python3.6/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
  44.         response = view_func(request, *args, **kwargs)

File "/var/www/venv/lib/python3.6/site-packages/django/contrib/admin/sites.py" in inner
  223.             return view(request, *args, **kwargs)

File "/var/www/venv/lib/python3.6/site-packages/django/utils/decorators.py" in _wrapper
  45.         return bound_method(*args, **kwargs)

File "/var/www/venv/lib/python3.6/site-packages/django/utils/decorators.py" in _wrapped_view
  142.                     response = view_func(request, *args, **kwargs)

File "/var/www/venv/lib/python3.6/site-packages/django/contrib/admin/options.py" in changelist_view
  1804.             'selection_note': _('0 of %(cnt)s selected') % {'cnt': len(cl.result_list)},

File "/var/www/venv/lib/python3.6/site-packages/django/db/models/query.py" in __len__
  250.         self._fetch_all()

File "/var/www/venv/lib/python3.6/site-packages/django/db/models/query.py" in _fetch_all
  1186.             self._result_cache = list(self._iterable_class(self))

File "/var/www/venv/lib/python3.6/site-packages/caching/base.py" in __iter__
  124.         cached = cache.get(query_key)

File "/var/www/venv/lib/python3.6/site-packages/redis_cache/backends/base.py" in wrapped
  30.             return method(self, client, key, *args, **kwargs)

File "/var/www/venv/lib/python3.6/site-packages/redis_cache/backends/base.py" in get
  267.         return self._get(client, key, default)

File "/var/www/venv/lib/python3.6/site-packages/redis_cache/backends/base.py" in _get
  255.         value = client.get(key)

File "/var/www/venv/lib/python3.6/site-packages/redis/client.py" in get
  1606.         return self.execute_command('GET', name)

File "/var/www/venv/lib/python3.6/site-packages/redis/client.py" in execute_command
  898.         conn = self.connection or pool.get_connection(command_name, **options)

File "/var/www/venv/lib/python3.6/site-packages/redis/connection.py" in get_connection
  1192.             connection.connect()

File "/var/www/venv/lib/python3.6/site-packages/redis/connection.py" in connect
  567.             self.on_connect()

File "/var/www/venv/lib/python3.6/site-packages/redis/connection.py" in on_connect
  664.             if nativestr(self.read_response()) != 'OK':

File "/var/www/venv/lib/python3.6/site-packages/redis/connection.py" in read_response
  739.             response = self._parser.read_response()

File "/var/www/venv/lib/python3.6/site-packages/redis/connection.py" in read_response
  340.                 raise error

Exception Type: AuthenticationError at /admin/store/tool/
Exception Value: Authentication required.

I see it is making the request through http: instead of https:. Is that related?

[wetneb]

hmm that is curious, I am not sure where to go from there. Perhaps we should upgrade to a newer version of Django and other libraries.

[edpomacedo]

I'm ready to do so. For the requirements.txt, I had to declare celery==5.*. Also, to the mysqlclient==2.0.3, I had to install libmysqlclient-dev, build-essential and python3-dev in order to migrate without any errors.

Do you have any suggestions on which libraries to use?

[wetneb]

I would just try to upgrade them one by one to higher versions and fix any incompatibilities that I encounter on the way. Hopefully the CI can be of some help for this.

[edpomacedo]

Ok! I will try again next weekend. Hopefully it works out.

I saw that you think it's possible to make it work with EventLogging... I read some of the documentation, and I learned that there is a backend to it too. Once I install both, the EditGroups will ingest the batches? Does it require further coding?

[wetneb]

If you install EventLogging then it should be reasonably straightforward to adapt EditGroups to connect to the underlying cluster.
It will make it possible to undo any batch run after you start running the listener.
To enable reversion of batches you have already done, there could be multiple options:

• backfill the recent changes feed into the kafka instance (I do not know if EventLogging supports that) and then ingest it again from EditGroups (should not require coding on EditGroups side)
• or implement another ingester from dumps of recent changes in EditGroups (or focused on the contributions of a specific user), which requires coding