Roadmap

[bckohan]

This is an update for everyone on how I'm thinking about and prioritizing work on this library so you may make decisions about continuing to use it based on more than nothing.

The first version of django-polymorphic was released 15 years ago in 2011. There are many users of this library and it runs in production systems around the world - per PyPi statistics it is not unreasonable to assume that somewhere approaching 5% of production Django systems depend on this library. During that time it has been supported by 6 different maintainers (@bconstantin @vdboor @chrisglass @jleclanche @meshy with me being the most recent) and over 125 individual contributors.

Maintaining open source is difficult. Volunteers burn out, get busy or have life changes that pull them away. There was a period of ~4 years where this library was essentially unmaintained. The Django/Python ecosystem marched on and things started to break. When I was granted maintainership there were 163 open issues and 16 open PRs - the oldest dating back 14 years.

Move to django-commons

This project was moved into jazzband not that long ago. Unfortunately the move took place without an identified maintainer so it languished for a few years without a successful update. The jazzband has been a true boon to the Django ecosystem over the years but I think this move is necessary for the health of this library for several reasons:

1. I maintain several other packages at django-commons so its just going to be easier for me to unify all my Django package workflows under a common organization.
2. I still have not been able to get a successful release through the jazzband process. and moving to django-commons would enable trusted publishing through backbone infrastructure in a way that I believe is just more secure. Right now I am publishing directly through my own PyPi account which is not ideal.
3. If something happened to me this project would be stuck again with nobody in position to issue releases. django-commons has 5 active admins to jazzband's overstretched single admin. This means if something happened to me there would be 5 people able to grant maintainership to someone else.

I do not have admin rights to this repo, nor do I have full admin rights to the pypi repository. This move will be hard to coordinate as we will need to get help from at least @jezdez @vdboor or @chrisglass . Please be patient.

Additional Maintainer

I would like to find at least one other additional maintainer with publish privileges. This will likely not move forward until the move to django-commons is complete. Given where I would like to take this library (see below) if you have experience in the guts of Django's ORM that would be extremely helpful! Otherwise - for trust - you would need an established record in the open source Django ecosystem. The best way to help right now is to pick off issues by opening PRs - I promise to look at them within a reasonable amount of time!

Progress So Far

• I have closed around 70 issues and merged around 15 outstanding PRs. Most of these issues were longstanding bug fixes. Many issues were duplicates or were addressing bugs that were fixed without the issue having been closed. For specific bug reports I have been adding matching tests and closing the issue if the tests pass. I ask for your continued patience because this is a very laborious process - no PR was merged without changes. Most merged PRs have included bug fixes. See the changelog for a comprehensive list. A few notable improvements include:
  1. Polymorphic QuerySet iteration now produces dramatically fewer SQL queries.
  2. A create_from_super method was added to Polymorphic managers that provides a shortcut for "promoting" an existing model row to a direct subclass of that model. This was the longest standing open PR (from 2013!).
  3. Q objects are no longer deepcopied by the PolymorphicQueryset. This eliminates spurious failures that some users were encountering.
  4. Many admin bug fixes.
• Code coverage has increased by ~7%.
• I have enabled discussions and for issues that were how-to-like questions I have been moving them into Q&A. If I have answered your question satisfactorily please accept the answer. The intention is for Q&A to function as an FAQ.
• Packaging has been modernized:
  1. uv is now the package manager.
  2. just has implemented many package management shortcuts.
  3. CI runs on all Django supported RDBMS and more combinations of supported dependencies.
  4. Admin tests now test full integration (including the delivered javascript) using playwright

Roadmap

I am cognizant that there are many outstanding bugs in this library and for users that have invested in its existing functionality fixing these should and will be be my top priority. The focus of the version 4.x series will be fixing those bugs while maintaining backwards compatibility. For version 5.x I would like to take a deeper look at how django-polymorphic works and determine if there are some fundamental changes that might notably increase its performance or simplify the implementation in ways that address the longstanding lack of support for selected_related and prefetch_related.

The current design of django-polymorphic is mostly a manager/queryset extension. It basically works by storing the most-derived leaf class ContentType on the parent table(s). When queries are made a second step uses those ContentTypes to run an additional query (or queries) if there are more derived tables for any model in the queryset. This works, is performant enough for most workflows, and is aligned with how most users would implement polymorphism themselves - but there may be deeper tie-ins to the Django ORM that could replace the 2-phase process with a single step process.

I want to acknowledge that of all the outstanding PRs those by @pgammans (@676 and @545) were the most thorough and impactful but they are the only two outstanding PRs I have not merged yet. I am delaying proxy model changes (@676) to version 5.x because there will be breaking changes. The current proxy model support is unintuitive and, I think, incongruous with the intent of proxy models in Django. I did not see a way to offer backwards compatibility with the current behavior without requiring users to change their code. I am delaying select_related and prefetch_related support (@676) until I can do a more thorough review of the library's architecture and I do not want to publish an interface that might be immediately deprecated in the next release. Thank you @pgammans - I promise these PRs will not languish forever!

Version 4.x

The last 4.x release will coincide with a Django LTS release (hopefully 6.2) to give users a solid 3 years to migrate to 5.x.

In priority order, with some issue links these are my areas of focus. Each block will be a release containing fixes in a related area. This should help users more easily identify the source of regressions if they arise and help keep me sane as I work through the backlog.

• Deletion Woes (there are many longstanding issues with Polymorphic deletion - punting some of these issues to 5.x may be necessary but getting deletion to work as well as possible in the 4.x series is my top priority) (4.5)
  • Deletion fails when polymorphic models have different pk fields at different levels of the hierarchy. #686
  • objects.all().delete() non-deterministically fails on oracle #673
  • CASCADE deletion of model with nested polymorphic models fail with xxx_ptr does not exist error #608
  • Error during deleting #547
  • Deletion doesn't function with self referential foreign key #540
  • AttributeError | 'NoneType' object has no attribute 'attname' #481
  • Exception on delete of polymorphic collection #357
  • orphaned polymorphic_ctype_id rows #352
  • Getting a specific element from a queryset in an post_delete returns None #347
  • Multiple inheritance bug on deletion (Django 1.10) #274
  • Another cascading bug on CASCADE deletion #229
  • Cascading bug on deletion (Django 1.8) #160
  • Cannot delete or update a parent row: a foreign key constraint fails  #34
• ORM Bugs (4.6)
  • AttributeError Using .alias() On Polymorphic Querysets #658
  • GenericPrefetch does not work correctly with Polymorphic Models #613 (may be pushed to 5.x)
  • can the instance_of Q-object be made to accept a string? #505
  • Model .save() with strange behaviour and not persisting on database #495
  • How to use update_or_create with django-polymorphic ? #494
  • ForeignKeyViolation when trying to save an entity when using a non-default db #486
  • refresh_from_db() got an unexpected keyword argument 'fields' #447
  • Content types, multiple databases and database routers #446
  • How to annotate with F()? #423
  • Filtering on fields which start with _ #295
  • 1.2 breaks Prefetch compatibility for Django 1.10- #290
  • Model object accessible via old class after class changed raises TypeError #280
• Admin Bugs (4.7)
  • Admin: add view popup breaks if initial submit has validation error #612
  • child models show in index regardless of show_in_index settings #532
  • PolymorphicChildModelAdmin with show_in_index=False in Django Admin with nav_sidebar #497
  • Calculated Field not found #479
  • regression ? custom admin urls #381
  • Polymorphic inline form ignoring multipart for FileFields from Django 2.1 #380
  • admin doesn't seem to work with intermediate join table for manytomany #375
  • Filters are not preserved in polymorphic parent admin #356
  • How to handle non-admin polymorphic forms? #346
  • StackedPolymorphicInline & Grappelli support #234
  • M2M field in model admin  #182
  • PolymorphicChildModelAdmin list displays base model of foreignkey fields #158
  • Can django-polymorphic filter the popup change list for raw ID fields? #144
  • Admin change form doesn't preserve changelist filter #125
• Formset Bugs (4.8)
  • PolymorphicFormSetChild overrides form exclude #578
  • Issue with polymorphic_ctype when populating polymorphic inline formsets. #549
  • Override multiple fieldsets of base_fieldsets #472
  • error when tring to access a=children formset of a model #409
  • Nested polymorphic_inline_formsets gives AttributeError: 'NoneType' object has no attribute 'get_real_instance_class' #363
  • M2M with inline forms and Polymorphic #294
• Serialization bugs (4.9)
  • please document how to deserialize by natural key #517
  • fixtures #513
  • Instance attributes missing in post save signal triggered by loaddata #502
  • Using django-polymorphic with django-import-export #342
  • dumpdata --natural management command missing parent fields to restore polymorphic child models #175
  • Wrong manager when using call_command('dumpdata') in stead of python manage.py dumpdata #146
• Miscellaneous Issues - anything else thats open and not slated for 5.x (4.x)
  • Spurious polymorphic related test failure in python.org CI #618
  • Other options are ignored by migration algorithm #570
  • Integration with django-filter: 'Meta.fields' must not contain non-model field names #520
  • Integration with django-reversion error #476
  • IntegrityError for polymorphic_ctype_id while running unit tests #470
  • contribute_to_class issue with specific model inheritance scenarios #431
  • PolymorphicTypeInvalid if primary key is the same... #420
  • Copying Polymorphic objects for ModelC not working #414
  • Reverse relation for all instances of abstract Polymorphic superclass #398
  • Polymorphic custom user model with the django-helpdesk app, migration error #397
  • Renaming superclass causes migrations to produce invalid SQL #396
  • Creating polymorphic instances in migrations results in invalid model instances (polymorphic_ctype_id is null) #388
  • First makemigrations fails with polymorphic and DRF #383
  • Django-polymorphic does not update the PolymorphicSerializer's validated_data after running the child's validation method #378
  • How to use abstract base model classes with intermediate abstract classes #366
  • refresh_from_db RecursionError #334
  • IndexError: list index out of range #252
  • select_related() from a plain model with a ForeignKey to a polymorphic model #244
  • get_real_instance_class throws TypeError #202
  • reversion or polymorphic bug? #137
  • Reverse related object descriptor is overwritten on class #71
• Test coverage needs to be 100% branch #660

The best way for folks to help right now is to pick out open issues and open PRs that have tests and fixes! I will happily accept and work on PRs that have tests illustrating bugs without the fixes in place yet!

Version 5.x

Version 5.x will include a significant scope of refactoring and potential breaking changes.

• Single-step queries
• Intuitive Proxy Model Support
  • Fix proxy model support. #676
  • Issue with the base_objects deprecation and proxy model QuerySets #466
  • Proxy Model Support - Creating base class and querying for proxy model does not return results #390
  • Proxy model needs manager definition #376
  • Use of proxy models broke polymorphic / some bad ctypes resulted in no results at all #358
  • Probable issue on proxy submodels and polymorphic inlines #256
• Support for select_related and prefetch_related
  • Related polymorphic query support inc select_related and prefetch_related #545
  • select_related issue #641
  • hacks for getting around the select_related issue from parent on inherited models to avoid n+1 queries? #436
  • select_related for a polymorphic model only returns base class objects #410
  • Can't prefetch base instance #359
  • select_related and prefetch_related for inherited models #198
• Bring in popular third party extensions.
  • Bring django-rest-polymorphic into this package. #655
• Non-backwards compat misc
  • Change default column name polymorphic_ctype_id #429
  • Polymorphic Filtering (for fields in inherited classes) enhance #233
  • NOT NULL Constraint for polymorphic_ctype_id? #148
• New Features
  • GeoDjango support. #172
• Type hints
  • Add type hints for django-polymorphic #647
  • Mypy stubs #579
  • Possible issue when using django-polymorphic along with typeddjango #529
  • Prefetch's to_attr field is not respected for polymorphic models #498
• Documentation overhaul
• Test Suite Reorg/Refactor

Please hold some grace for me and the other contributors to this library. Coming updates will likely trigger regressions. This is unavoidable to make progress and that progress lies at the feet of volunteers who are not remunerated for this work.

[JohananOppongAmoateng]

@bckohan i want to start working on adding the stub files do i need to wait until the lib is getting updated to v5?

[bckohan]

Hi @JohananOppongAmoateng! Thank you for volunteering! You might want to checkout #647 Huy had mentioned wanting to work on this but I'm not sure anything has happened yet. He mentioned maybe putting these types in typeshed instead of django-polymorphic... what do you think?

[JohananOppongAmoateng]

Honestly i think having the types in typeshed isnt bad but i am looking at long term the fact that we need to always sync the types as they change in the lib with typeshed. I think for start we can have them there if the need arises to include them in the lib we then move them here

[bckohan]

I'm also partial to keeping them in this library. I also don't think we need to wait for 5.0 - who knows how long it will be before we get to that...

If you want to start a branch we can just do our best to stay in sync until the merge!

[JohananOppongAmoateng]

Okay i will start a branch for this

[bckohan]

I would want to keep all the types in stub files though. I think the rebasing will be waaaay easier that way.

[JohananOppongAmoateng]

Noted

[JohananOppongAmoateng]

Just to confirm something do you want the stub files living alongside the code

[bckohan]

Yes I believe that would be ideal