fix: Fixed _pickle.UnpicklingError

[felipe3dfx]

Error Details
The migration process attempted to apply the migration 0003_drop_pickle but failed with the following traceback:

Operations to perform:
  Target specific migration: 0003_drop_pickle, from constance
Running migrations:
  Applying constance.0003_drop_pickle...Traceback (most recent call last):
  File "/code/manage.py", line 23, in <module>
    main()
  File "/code/manage.py", line 19, in main
    execute_from_command_line(sys.argv)
  File "/code/.venv/lib/python3.12/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/code/.venv/lib/python3.12/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/code/.venv/lib/python3.12/site-packages/django/core/management/base.py", line 413, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/code/.venv/lib/python3.12/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.12/site-packages/django/core/management/base.py", line 107, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.12/site-packages/django/core/management/commands/migrate.py", line 357, in handle
    post_migrate_state = executor.migrate(
                         ^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.12/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.12/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
            ^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.12/site-packages/django/db/migrations/executor.py", line 255, in apply_migration
    state = migration.apply(state, schema_editor)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.12/site-packages/django/db/migrations/migration.py", line 132, in apply
    operation.database_forwards(
  File "/code/.venv/lib/python3.12/site-packages/django/db/migrations/operations/special.py", line 196, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/code/.venv/lib/python3.12/site-packages/constance/migrations/0003_drop_pickle.py", line 43, in migrate_pickled_data
    print(pickle.loads(value))
          ^^^^^^^^^^^^^^^^^^^
_pickle.UnpicklingError: invalid load key, '{'.

[Mogost]

Not sure that this is a right fix of the problem. This fix only ignores the problem. @camilonova please let a chance for review before merging

[Mogost]

@felipe3dfx what value is stored in the problem key? I guess some collection list or dict?

[ivan-klass]

This isn't a fix but a careless suppress of an underlying error. Chances are high that this can lead to much more serious troubles - users got exceptions much more later, when there's no clear context of the migration issues.

It would be better to understand the problem. The exception here seems like some kind of re-applying the migration, as { is faced where pickle data is expected.

picle.loads(b'{"__key__": "default", "__value__": 1}')
...
_pickle.UnpicklingError: invalid load key, '{'.

Was it a second attempt of the migration? If so, what was a first error?
Anyway, if reapplying the migration was the case, maybe better to check whether a value is already in a new format and only suppress in that specific case? That would be much more safe

[hvdklauw]

Can this be released please?

The issue happens when migrate fails after constance has done it's thing and redis contains the new data while the migration (when run again) still expects the old format.

[felipe3dfx]

@Mogost The value that raised the issue was: b'{"__type__": "default", "__value__": "We are in a testing stage"}' and was raised when running all migrations of an application using Docker containers. In other words, when the database is empty and applying all migrations.

[Mogost]

This fix should not be released in this form. I fully support @ivan-klass's

Anyway, if reapplying the migration was the case, maybe better to check whether a value is already in a new format and only suppress in that specific case? That would be much more safe

[ivan-klass]

@hvdklauw thank you for the context. Not sure where redis get the value of new format, but if there's a rush on this, I would still recommend to limit the suppress to that case of re-applying the conversion

[ivan-klass]

Working on the PR right now