Optimize some management endpoints via the querysets and prefetches by MyPyDavid · Pull Request #1600 · rdmorganiser/rdmo

MyPyDavid · 2026-05-07T07:18:02Z

Description

Related issue: "I think we had an issue from even before the migration to the React interface"

Some managment pages were still loading slowly, one prefetch field default_option for the questions.Models was missing. This may improve some queries.

…and exports Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

…or catalog index Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

coveralls · 2026-05-07T08:02:14Z

Coverage is 95.101% — 2.4.5/fix/optimize-management-querysets into 2.4.5/release. No base build found for 2.4.5/release.

…sets Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

MyPyDavid · 2026-05-07T13:13:45Z

+        "queries": {
+            "index": 3,
+            "list": 9,
+            "detail": 9,


this mapper defines the max query counts per api endpoint action during a test.
Very simple test at the moment. They could be expanded by creating 10000 Options, Attributes, etc dynamically during test as well..

MyPyDavid · 2026-05-07T13:14:29Z

        'editors'
    )

+    def get_queryset(self):


nearly forgot the Tasks and Views ;)

jochenklar

I am a bit puzzled, why I cannot follow part of your optimizations, maybe we should talks next week.

jochenklar · 2026-05-10T09:35:04Z

        'catalog_sections__section__section_pages__page__page_questions__question__attribute',
        'catalog_sections__section__section_pages__page__page_questions__question__conditions',
        'catalog_sections__section__section_pages__page__page_questions__question__optionsets',
+        'catalog_sections__section__section_pages__page__page_questions__question__default_option',


I am not convinced prefetching the default_option is needed, since we only use its id in the serializer and do not resolve the instance.

And what about the QuestionExportSerializer? It uses default_option = serializers.CharField(source='default_option.uri', default=None, read_only=True) so default_option.uri needs to be resolved for exports right?! It would be used by all Export serializers up to the CatalogExport. Would it be worth to add the prefetch field only for export or apply it only for the export actions? Otherwise we can leave it out like you said.

rdmo/rdmo/questions/serializers/export.py

Line 26 in e77e2ca

default_option = serializers.CharField(source='default_option.uri', default=None, read_only=True)

So actually, the export of a Catalog via the management interface can make almost 3000(!) queries and debug toolbar crashed my machine a bit.
Is it because of a missing prefetch_elements call in the detail_export?

rdmo/rdmo/questions/viewsets.py

Line 97 in e77e2ca

serializer = CatalogExportSerializer(self.get_object())

jochenklar · 2026-05-10T09:38:36Z

                'questionsets',
                'editors'
-            ).select_related('attribute')
+            ).select_related('attribute', 'default_option')


Same as above, I don't think we use this.

…elated Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

…unctinos and select_related calls Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

MyPyDavid · 2026-05-13T09:09:18Z

so, these were the query counts from before the optimizations of the export and detail_export actions:

 Error: test_page_endpoints_query_counts[detail_export-13-url_kwargs4]

Failed: Expected to perform 13 queries or less but 20 were done (add -v option to show queries)
Error: test_views_endpoints_query_counts[detail_export-10-url_kwargs4]

Failed: Expected to perform 10 queries or less but 12 were done (add -v option to show queries)
Error: test_page_endpoints_query_counts[export-25-url_kwargs2]

Failed: Expected to perform 25 queries or less but 409 were done (add -v option to show queries)
Error: test_question_endpoints_query_counts[export-12-url_kwargs2]

Failed: Expected to perform 12 queries or less but 398 were done (add -v option to show queries)
Error: test_optionset_endpoints_query_counts[detail_export-10-url_kwargs4]

Failed: Expected to perform 10 queries or less but 14 were done (add -v option to show queries)
Error: test_actions_max_query_counts[export-11-url_kwargs2]

Failed: Expected to perform 11 queries or less but 26 were done (add -v option to show queries)
Error: test_actions_max_query_counts[detail_export-9-url_kwargs4]

Failed: Expected to perform 9 queries or less but 17 were done (add -v option to show queries)
Error: test_catalog_endpoints_query_counts[detail_export-29-url_kwargs4]

Failed: Expected to perform 29 queries or less but 425 were done (add -v option to show queries)
Error: test_catalog_endpoints_query_counts[export-30-url_kwargs2]

Failed: Expected to perform 30 queries or less but 408 were done (add -v option to show queries)
.F.........F..F..............FF........................F.FF.........F... [ 99%]
Error: test_questionset_endpoints_query_counts[export-17-url_kwargs2]

Failed: Expected to perform 17 queries or less but 70 were done (add -v option to show queries)
Error: test_questionset_endpoints_query_counts[detail_export-13-url_kwargs4]

Failed: Expected to perform 13 queries or less but 26 were done (add -v option to show queries)
Error: test_section_endpoints_query_counts[export-27-url_kwargs2]

Failed: Expected to perform 27 queries or less but 406 were done (add -v option to show queries)
Error: test_section_endpoints_query_counts[detail_export-12-url_kwargs4]

Failed: Expected to perform 12 queries or less but 59 were done (add -v option to show queries)
Error: test_domain_endpoints_query_counts[detail_export-12-url_kwargs4]

Failed: Expected to perform 12 queries or less but 30 were done (add -v option to show queries)
Error: test_domain_endpoints_query_counts[export-12-url_kwargs2]

Failed: Expected to perform 12 queries or less but 160 were done (add -v option to show queries)
Error: test_domain_endpoints_query_counts[index-3-None]

KeyError: 'index'
Error: test_domain_endpoints_query_counts[list-10-None]

Failed: Expected to perform 10 queries or less but 114 were done (add -v option to show queries)
Error: test_tasks_endpoints_query_counts[detail_export-12-url_kwargs4]

Failed: Expected to perform 12 queries or less but 16 were done (add -v option to show queries)
...F..F.FF.F.F.FF...........F........................................... [ 99%]

For a catalog 😅 :
Expected to perform 29 queries or less but 425 were done

jochenklar · 2026-05-15T11:09:50Z

 from .serializers.v1 import ConditionIndexSerializer, ConditionSerializer


+def get_attributes_by_id():


This method is not needed because of .in_bulk(), but if it should go into utils.py or managers.py.

Also this prefetches all (!) attributes, which is overkill.

jochenklar · 2026-05-15T11:11:13Z

            )

-    def get_export_renderer_context(self, request):
+    def get_export_renderer_context(self, request, attributes_by_id=False):


There should be a seperate get_export_serializer_context to for the serializer context.

jochenklar · 2026-05-15T11:12:55Z

    def get_parent_data(self, obj):
-        if obj.parent is not None:
-            return AttributeExportSerializer(obj.parent).data
+        parent = self.get_parent_object_from_context(obj)


Those to methods are unnecessary complex, part because AI slob and part because the original code with the parent_data was not great. I refactored the whole thing in 2.4.5/fix/optimize-management-querysets+refactor.

where is this branch 2.4.5/fix/optimize-management-querysets+refactor that you speak of?? 👀 👀

Ah, sorry, I forgot to push: #1616.

jochenklar · 2026-05-15T12:06:04Z

 )


+def get_attributes_by_id():


Same as above.

jochenklar · 2026-05-16T06:43:10Z

            raise RuntimeError('uri_path is missing')
        return join_url(uri_prefix or settings.DEFAULT_URI_PREFIX, '/questions/', uri_path)
+
+


This part (same for the other question models) needs more work. I get that the recursion solves the problem with nested questionsets. However for the full export we need the same attribute_map I implemented for the other models in 2.4.5/fix/optimize-management-querysets+refactor.

Nevertheless, all this code does not belong in models.py. Please create an extra prefetch.py for it and refactor the prefetch_elements methods accordingly. Catalog.prefetch_lookups is not needed anymore. child_questionset_prefetch should be questionset_questionset_prefetch to match the other naming in RDMO.

After this we need to see if we need a special case for default_option since it is only needed for the export and would maybe slow down the answer tree.

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

…ect page Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

…ent-querysets+refactor Refactor queryset optimizations

MyPyDavid · 2026-05-20T10:00:11Z

I've made some changes according to your suggestions and included the refactor PR..

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

…osen export tests Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

jochenklar · 2026-05-26T08:24:34Z

Alright, this was quite a ride, but I think/hope/pray that we have a good result now. I did not get why the Prefect -> select_related pattern is better than .prefetch_related('foo__bar__baz') at first, but it is just one prefetch query less.

I refactored prefetch.py to make some of the prefetching optional (the answer tree needs less than the page in the interview and that needs less than the export). I also made it a bit more readable, with the "public" methods on top.
get_export_renderer_context is now get_export_flags.
The attribute_map pattern is now also used in the questions export.
get_section_for_page did something wrong (since a long time ago), which prevented the use of prefetching.
I renamed the news tests to test_queries_*.
The nested serializers needed some more work, since they inherited from the regular full serializer, which lead to a lot of queries. I created new, stand-alone serializers. I thought about creating a CatalogBaseSerializer, but I think its cleaner this way, since we don't need to change/break the existing serializers.
I also moved some getters from the serializers to the models.

@MyPyDavid please check and then merge.

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

MyPyDavid · 2026-05-27T08:26:49Z

ok, thanks @jochenklar! 🙌
I've added the is_collection field to these serializers, that was still missing from them right? bc94b27

jochenklar · 2026-05-27T14:13:30Z

Yes, thanks!

MyPyDavid added 2 commits May 7, 2026 09:13

Add select_related for Attribute queryset and split for index,nested …

aa71f3a

…and exports Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Add default_option to questions models prefetch and do not annotate f…

f5582c3

…or catalog index Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

MyPyDavid added this to the RDMO 2.4.5 milestone May 7, 2026

MyPyDavid self-assigned this May 7, 2026

MyPyDavid changed the title ~~Optimize management querysets~~ Optimize some management querysets May 7, 2026

MyPyDavid changed the base branch from main to 2.4.5/release May 7, 2026 07:47

MyPyDavid added 3 commits May 7, 2026 14:23

Do not prefetch for the index action on conditions,options and option…

ad813f2

…sets Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Also do not prefetch for the index action on views and tasks

f1ec6c9

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Add simple performance tests for query counts on api actions

1a38bf6

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

MyPyDavid commented May 7, 2026

View reviewed changes

jochenklar requested changes May 10, 2026

View reviewed changes

Remove editors from AttributeViewSet prefetch and do not use select_r…

757e139

…elated Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

MyPyDavid changed the title ~~Optimize some management querysets~~ Optimize some management endpoints via the querysets May 12, 2026

MyPyDavid added 5 commits May 13, 2026 10:30

Add tests for api performance of all elements

248e5bb

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Clean up tests

4bf0ba3

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Optimize queries for export and detail_export actions on elements

0ef1b74

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Add test for index attributes action

5f4c0a8

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Refactor and optimize prefetch lookups for questions with recursive f…

1140b53

…unctinos and select_related calls Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

MyPyDavid requested a review from jochenklar May 13, 2026 18:57

jochenklar added 2 commits May 15, 2026 12:05

Refactor tests

e76676a

Refactor Attribute exports

733ba22

jochenklar requested changes May 16, 2026

View reviewed changes

MyPyDavid changed the title ~~Optimize some management endpoints via the querysets~~ Optimize some management endpoints via the querysets and prefetches May 18, 2026

MyPyDavid added 4 commits May 20, 2026 09:51

Add context to AttributeExportSerializer calls

81bf755

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Add export serializer context to questions export actions

b4ee4cf

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Also refactor catalog performance test

1d95756

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Refactor lookups into questions/prefetch.py and add a lookup for proj…

2d4903a

…ect page Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

MyPyDavid and others added 2 commits May 20, 2026 10:42

Add api performance test for project answers

8797201

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Merge pull request #1616 from rdmorganiser/2.4.5/fix/optimize-managem…

8239f2c

…ent-querysets+refactor Refactor queryset optimizations

MyPyDavid requested a review from jochenklar May 20, 2026 10:00

MyPyDavid and others added 10 commits May 20, 2026 17:32

Add AnswerTree related endpoints to projects api perf test

2671d31

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Remove default_option from select related in Question prefetch and lo…

c4cc1b3

…osen export tests Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Refactor prefetching

830f0d9

Improve get_section_for_page

da59051

Rename and adjust query tests

6e78ab3

Refactor nested serializers

63f10b3

Refactor OptionSetNestedSerializer and add OptionNestedSerializer

1492909

Rename remaining get_export_renderer_context methods

2f89b1b

Move get_condition_uris and get_condition_uris to models

3d993d8

Fix nested serializers

7407b31

Add is_collection to PageNested and QuestionSetNested serializers

bc94b27

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

jochenklar approved these changes May 27, 2026

View reviewed changes

MyPyDavid merged commit 38ca003 into 2.4.5/release May 27, 2026
19 checks passed

MyPyDavid deleted the 2.4.5/fix/optimize-management-querysets branch May 27, 2026 15:02

		from .serializers.v1 import ConditionIndexSerializer, ConditionSerializer


		def get_attributes_by_id():

		raise RuntimeError('uri_path is missing')
		return join_url(uri_prefix or settings.DEFAULT_URI_PREFIX, '/questions/', uri_path)

Conversation

MyPyDavid commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

coveralls commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MyPyDavid May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jochenklar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MyPyDavid May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MyPyDavid commented May 13, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jochenklar May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MyPyDavid commented May 20, 2026

Uh oh!

jochenklar commented May 26, 2026

Uh oh!

MyPyDavid commented May 27, 2026

Uh oh!

jochenklar commented May 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

MyPyDavid commented May 7, 2026 •

edited

Loading

coveralls commented May 7, 2026 •

edited

Loading

MyPyDavid May 7, 2026 •

edited

Loading

MyPyDavid May 11, 2026 •

edited

Loading

jochenklar May 19, 2026 •

edited

Loading