Draft: add `rdmo.config` app for `Plugin` model (plugin managament) #1436

MyPyDavid · 2025-09-30T08:36:24Z

Description

adds a rdmo.config app
- with a Plugin model
- the Plugin behaves similar to any other element (Catalog, Taks, View)
added a new setting PLUGINS = [ .. ] which replaces the legacy settings (e.g. PROJECT_IMPORT).

Deprecations and legacy

removed rdmo.core.plugins
- moved Plugin base class to rdmo.config.plugins and renamed to PluginBase

Related issue: #1413

jochenklar

Great! Lets discuss in the meeting.

jochenklar · 2025-10-09T08:49:28Z

rdmo/config/models.py

+        help_text=_('Designates whether this plugin is generally available for projects.')
+    )
+
+    class PluginType(models.TextChoices):


I think we can do without this, the Plugin(-class) should know this.

I have now something like this as a utils.py function, to detect the plugin type from the class inheritance.

PLUGIN_BASES = { 'optionset_provider': OptionsetProvider, 'export': Export, 'import': Import, 'issue_provider': IssueProvider, } def detect_plugin_type(plugin_class): if not issubclass(plugin_class, Plugin): return "not_an_rdmo_plugin" for type_name, base_cls in PLUGIN_BASES.items(): if issubclass(plugin_class, base_cls): return type_name return 'rdmo_plugin_unknown_type'

I kept it as a property in the Plugin model:

@property def plugin_type(self) -> str: try: plugin_class = self.get_class() except Exception as e: return e.__class__.__qualname__.lower() return detect_plugin_type(plugin_class)

jochenklar · 2025-10-09T08:52:30Z

rdmo/config/models.py

+        verbose_name=_("Python path"),
+    )
+    plugin_type = models.CharField(max_length=32, choices=PluginType.choices)
+


But what we need to add is plugin_settings which is used to set the settings for this plugin.

the plugin_settings field is there now:

plugin_settings = models.JSONField( blank=True, default=dict, verbose_name=_("Plugin settings"), help_text=_("Contains the settings for this plugin in JSON format."), )

MyPyDavid · 2025-10-15T15:42:54Z

I'm getting some strange DB errors for mysql in ci(and sqlite in local testing).
In my local pytest with sqlite:

django.db.utils.OperationalError: foreign key mismatch - "questions_question" referencing "domain_attribute"

From the ci logs:

# at
query = b'ALTER TABLE `domain_attribute` DROP COLUMN `attributeentity_ptr_id`'

 django.db.utils.OperationalError:
  (1829, "Cannot drop column 'attributeentity_ptr_id': 
 needed in a foreign key constraint 'questions_questionse_attribute_id_30ee3ea9_fk_domain_at' of table 'questions_questionset'")

The problem can be resolved by removing the app 'rdmo.config' from the installed apps again..
Trying to see it is a bug somewhere in the migrations (0037_remove_attribute.py) but can't really find it so far..

MyPyDavid · 2025-10-21T12:15:51Z

I'm getting some strange DB errors for mysql in ci(and sqlite in local testing). In my local pytest with sqlite:
django.db.utils.OperationalError: foreign key mismatch - "questions_question" referencing "domain_attribute"

Think that I found the bug with help of GPT and could fix it by adding a single dependency to the ('domain', '0038_rename_attributeentity_to_attribute') in the questions.migrations.0032_meta.py file.

GPT:

You’ve got a classic multi-table-inheritance gotcha: the questions app points a FK at the child table’s primary key just before the domain app drops that child PK. SQLite complains with a vague “foreign key mismatch,” while MySQL is blunt: you can’t drop attributeentity_ptr_id on domain_attribute because it’s referenced by questions_questionset / questions_question.

Why this happens (in your tree)

In domain 0037, the child model Attribute (multi-table inheritance) is dismantled: it first adds a dummy column, then drops the attributeentity_ptr field (which is the PK of the child table!), and finally deletes the model.

Immediately after, domain 0038 renames the base class AttributeEntity → Attribute, so from here on “Attribute” refers to the base table (with a normal id PK), not the child.

In the questions app, migration 0031 renames the field attribute_entity → attribute, and 0032 makes that FK point to to='domain.Attribute'. At the time 0032 runs (without extra dependencies), “domain.Attribute” still means the child table (pre-rename), so the FK targets the child table’s PK (attributeentity_ptr_id).

Then domain 0037 tries to drop attributeentity_ptr on domain_attribute, but the FK from questions still points to that column → MySQL error 1829; SQLite “FK mismatch”.
You also noticed removing rdmo.config “fixes” it. That’s consistent: the extra app changes the topological order so domain 0037 sneaks in before questions flips its FKs to the renamed/base model, exposing the ordering bug.

jochenklar · 2025-10-21T15:16:51Z

Ok got it! Good explanation by ChatGPT. This is why you don't do multi-table inheritance and why past-Jochen dismantled it.

rdmo/config/apps.py

MyPyDavid · 2025-11-14T14:50:15Z

need to rebase to 2.4.0 now
PS it ran without any conflicts

MyPyDavid · 2025-11-27T11:38:50Z

rdmo/core/managers.py

+        return self.filter(Q(groups=None) | Q(groups__in=groups))
+
+
+class ForSiteQuerySetMixin:


Could implement this already in #1488, see #1488 (comment)

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

MyPyDavid · 2025-12-22T10:05:00Z

rebased to main , in expectation of new 2.5.0 Release PR

coveralls · 2025-12-22T10:30:14Z

coverage: 94.693% (-0.1%) from 94.814%
when pulling 0ffb48a on add-config-app-for-plugin-model
into cda25ed on main.

jochenklar

Thanks a lot for the work! I like the implementation, but I have some requests, in particular with the management scripts.

Additionally, the integration into the old front-end seems to be missing.

I have not tested yet with real plugins, I will do so in the next iteration.

jochenklar · 2025-12-22T09:45:26Z

rdmo/services/validators.py

+            plugin_type=PluginType.PROJECT_ISSUE_PROVIDER.value,
+            format=provider_key
+        )
+        provider = plugins.first().initialize_class() if plugins else None


I think this should be try: ... except DoesNotExist:

jochenklar · 2025-12-22T09:47:00Z

rdmo/questions/migrations/0032_meta.py


    dependencies = [
        ('questions', '0031_rename_attribute_entity_to_attribute'),
+        ('domain', '0038_rename_attributeentity_to_attribute'),  # post hoc fix for mysql/sqlite


Didn't we fix this before?

Yes, before in this PR #1436 (comment)

jochenklar · 2025-12-22T10:21:57Z

rdmo/options/viewsets.py

-    queryset = settings.OPTIONSET_PROVIDERS
+
+    def get_queryset(self):
+        return getattr(settings, 'OPTIONSET_PROVIDERS', [])


settings in RDMO should have a default so that we don't need those checks. But I guess OPTIONSET_PROVIDERS is now a queryset anyway so this might be an oversight.

Yes, it's an oversight, it needs to query the Plugin objects.

jochenklar · 2025-12-22T10:24:27Z

rdmo/options/models.py

        verbose_name=_('Conditions'),
        help_text=_('The list of conditions evaluated for this option set.')
    )
+    plugins = models.ManyToManyField(  # can not import Plugin due to circular import


Interesting, we now have multiple plugins for an optionset, lets see what crasy things people will do with it.

This could be crazy?! 😏 https://github.com/rdmorganiser/rdmo-plugins-generic-search

jochenklar · 2025-12-22T10:29:53Z

rdmo/core/utils.py

 from django.utils.dateparse import parse_date
 from django.utils.encoding import force_str
 from django.utils.formats import get_format
+from django.utils.module_loading import import_string


this exists since Django 1.7? 🤯

jochenklar · 2025-12-22T12:00:41Z

rdmo/management/assets/js/components/edit/EditPlugin.js

+        <Tabs id="#plugin-tabs" defaultActiveKey={0} animation={false}>
+          {
+            config.settings && config.settings.languages.map(([lang_code, lang], index) => (
+              <Tab className="pt-10" key={index} eventKey={index} title={lang}>


help is missing here.

jochenklar · 2025-12-22T12:04:40Z

rdmo/management/assets/js/components/edit/common/JsonField.js

+  const [value, setValue] = useState(formatValue(element[field]))
+  const [parseError, setParseError] = useState(false)
+
+  useEffect(() => {


This is buggy, if I type something in the field it (almost) instantly runs the validation, which is different from the other fields. I would prefer that the indenting happens only after pressing save.

rdmo/config/management/commands/setup_plugins.py

jochenklar · 2025-12-22T12:19:32Z

rdmo/config/models.py

+        verbose_name_plural = _('Plugins')
+
+    def __str__(self):
+        return f"({self.python_path}, {self.plugin_type}) {self.uri}"


This should be just the URI.

I could add that f-string to a __repr__ right?!

jochenklar · 2025-12-22T12:27:29Z

rdmo/config/managers.py

+    models.QuerySet):
+
+    def filter_for_project(self, project):
+        return (


This needs to check for availability. Right now, I can export using an unavailable Plugin.

I still need to look into this, I though I'm using only the for_context query (it has a check for availability) for the Plugins and this filter_for_project is used inside of it.

jochenklar · 2025-12-22T12:32:23Z

The test plugins can be used in the rdmo-app by adding:

import importlib.util
import sys
from pathlib import Path

rdmo_path = Path(importlib.util.find_spec('rdmo').origin).parent.parent
sys.path.append(str(rdmo_path / 'testing'))

INSTALLED_APPS += ['plugins']

to the settings.

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

MyPyDavid · 2025-12-23T10:23:55Z

rdmo/projects/models/integration.py

+                format=self.provider_key)
+        )
+        if plugins.exists():
+            return plugins.first().initialize_class()


Currently this will "arbitrarily" only return the first plugin instance as provider. At some other places in the code I also simply return plugins.first().initialize_class(), this was a temporary hack to make a working version.
How should it be handled when there are multiple plugin instances available as providers? Should a new property like providers be added?

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

MyPyDavid self-assigned this Sep 30, 2025

jochenklar requested changes Oct 9, 2025

View reviewed changes

jochenklar reviewed Oct 9, 2025

View reviewed changes

MyPyDavid changed the title ~~Draft: add config app for Plugin model~~ Draft: add rdmo.config app for Plugin model Oct 9, 2025

MyPyDavid added the status:work-in-progress label Oct 20, 2025

MyPyDavid added this to the RDMO 2.4.0 milestone Oct 20, 2025

MyPyDavid changed the title ~~Draft: add rdmo.config app for Plugin model~~ Draft: add rdmo.config app for Plugin model Oct 21, 2025

jochenklar reviewed Oct 21, 2025

View reviewed changes

rdmo/config/apps.py Outdated Show resolved Hide resolved

MyPyDavid linked an issue Oct 27, 2025 that may be closed by this pull request

config app: Allow managers to configure Plugin settings via GUI #1413

Open

jochenklar modified the milestones: RDMO 2.4.0, RDMO 2.5.0 Oct 30, 2025

MyPyDavid force-pushed the add-config-app-for-plugin-model branch from 107f379 to f5eae22 Compare November 7, 2025 13:08

MyPyDavid force-pushed the add-config-app-for-plugin-model branch from 7031a4b to 6875f9a Compare November 14, 2025 14:52

MyPyDavid changed the base branch from 2.3.3 to 2.4.0 November 14, 2025 14:52

MyPyDavid force-pushed the add-config-app-for-plugin-model branch from 6875f9a to 9791229 Compare November 19, 2025 08:16

MyPyDavid changed the base branch from 2.4.0 to 2.5.0 November 19, 2025 08:17

MyPyDavid force-pushed the add-config-app-for-plugin-model branch from dceca19 to ae887db Compare November 24, 2025 14:45

MyPyDavid commented Nov 27, 2025

View reviewed changes

MyPyDavid mentioned this pull request Nov 27, 2025

2.4.0 fix multisite not all sites #1488

Merged

MyPyDavid force-pushed the 2.5.0 branch from e3b2b72 to 57a75b0 Compare December 5, 2025 16:12

MyPyDavid force-pushed the add-config-app-for-plugin-model branch from 27516be to 4026f9a Compare December 5, 2025 16:12

MyPyDavid changed the title ~~Draft: add rdmo.config app for Plugin model~~ Draft: add rdmo.config app for Plugin model (plugin managament) Dec 9, 2025

MyPyDavid marked this pull request as ready for review December 9, 2025 12:23

MyPyDavid requested a review from jochenklar December 9, 2025 12:23

MyPyDavid added the type:feature-request label Dec 10, 2025

initial add config app with Plugin model

f90b6e3

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

MyPyDavid added 2 commits December 22, 2025 11:01

Use enum constant for internal plugin types

6f3a3bd

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Clean up use of plugin types

0ffb48a

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

MyPyDavid force-pushed the add-config-app-for-plugin-model branch from e9b3b51 to 0ffb48a Compare December 22, 2025 10:02

MyPyDavid changed the base branch from 2.5.0 to main December 22, 2025 10:02

MyPyDavid added the status:hold label Dec 22, 2025

jochenklar requested changes Dec 22, 2025

View reviewed changes

MyPyDavid added 2 commits December 22, 2025 17:37

Update setup plugins script

35b3bdd

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Rename function

8f66bac

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

MyPyDavid changed the base branch from main to 2.5.0/release December 22, 2025 16:41

MyPyDavid added 4 commits December 22, 2025 20:42

Use variables directly from options kwargs

9108f28

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Clean up comment

0ffdc3b

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Clean up newline

d4e590a

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Add newline

ed97b32

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

MyPyDavid commented Dec 23, 2025

View reviewed changes

MyPyDavid added 13 commits December 23, 2025 11:53

Clean up and refactor config app

3445ee2

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Clean up and refactor plugins in projects

b176e23

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Remove queryset that uses OPTIONSET_PROVIDERS

92ee088

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Clean up and refactor plugins in services

7b7b2f5

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Add queryset from plugins to ProviderViewSet and add test

06a503e

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Clean up in config.legacy

5c1ae6e

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Add support for default uri prefix to plugins

9ecf292

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Use meta model name instead

ce6e187

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Remove CSV and JSON exports from PLUGINS

3f7b5aa

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Clean up config.helpers and imported annotations

4d46f64

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Do not raise from Exception in config

366c7ad

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Make str return just uri for Plugin

1b178d1

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Improve error handling for setup plugins

af00d09

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

		return self.filter(Q(groups=None) \| Q(groups__in=groups))


		class ForSiteQuerySetMixin:

Draft: add rdmo.config app for Plugin model (plugin managament) #1436

Are you sure you want to change the base?

Draft: add rdmo.config app for Plugin model (plugin managament) #1436

Uh oh!

Conversation

MyPyDavid commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Deprecations and legacy

Uh oh!

jochenklar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MyPyDavid Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MyPyDavid commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MyPyDavid commented Oct 21, 2025

Uh oh!

jochenklar commented Oct 21, 2025

Uh oh!

Uh oh!

MyPyDavid commented Nov 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MyPyDavid commented Dec 22, 2025

Uh oh!

coveralls commented Dec 22, 2025

Uh oh!

jochenklar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jochenklar commented Dec 22, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Draft: add `rdmo.config` app for `Plugin` model (plugin managament) #1436

Draft: add `rdmo.config` app for `Plugin` model (plugin managament) #1436

MyPyDavid commented Sep 30, 2025 •

edited

Loading

MyPyDavid Oct 27, 2025 •

edited

Loading

MyPyDavid commented Oct 15, 2025 •

edited

Loading

MyPyDavid commented Nov 14, 2025 •

edited

Loading