django-dynamic-storage

Have you ever wanted not to store every instance of FileFileds or ImageFileds of a model in one storage or one bucket of a storage?

Now you can, because I wanted that for my project.

prerequisites:

If your django version is earlier than 3.1 (<=3.0) then your database should be PostgreSQL otherwise you are good to go.

Usage:

pip install django-dynamic-storage

in storage.py:

from django.utils.deconstruct import deconstructible
from dynamic_storage.storage import DynamicStorageMixin
from dynamic_storage.storage import AbstractBaseStorageDispatcher

class MyStorageDispatcher(AbstractBaseStorageDispatcher):
    @staticmethod
    def get_storage(instance, field, **kwargs):
        if kwargs.get("my_storage_identifier") == "storage1":
            return MyDynamicStorage(named_param1=kwargs["named_param1"], named_param2=kwargs["named_param2"])
        elif isinstance(instance, models.Profile) and field.name == "profile_pic":
            return MyDynamicStorage(named_param1="my_hard_coded_var", named_param2="my_other_hard_coded_var")
        # elif ...
        raise NotImplementedError


@deconstructible
class MyDynamicStorage(DynamicStorageMixin, AnyStorage):
    def __init__(self, named_param1, named_param2):
        # AnyStorage stuff
        super().__init__(named_param1, named_param2)
        
	def init_params(self) -> dict:
		"""
		here you should return a dictionary of key value pairs that 
		later are passed to MyStorageDispatcher.
		should be json serializable!!!
		"""
		return {"my_storage_identifier": "storage1", "named_param1": self.named_param1, "named_param2": self.named_param2, ...}

AnyStorage can be a storage that you define yourself or import from django-storages.

in settings.py:

# path to your storage dispatcher
STORAGE_DISPATCHER = "myapp.storage.MyStorageDispatcher"

in models.py:

from dynamic_storage.models import DynamicFileField, DynamicImageField

class MyModel(models.Model):
	"""
	DynamicFileField and DynamicImageField accept any options that django's native FileField and ImageField accept
	"""
	file = DynamicFileField()
	image = DynamicImageField()

note that there is no Storage specified here!😎

Now your logic to take control of the storage where your content is going to be saved to:

obj = MyModel(file=file, image=image)
obj.file.destination_storage = MyDynamicStorage(named_param1="something", named_param2="another_thing")
obj.image.destination_storage = MyDynamicStorage(named_param1="foo", named_param2="bar")
obj.save()

or using signals:

(new to signals? learn how to connect them)

from dynamic_storage.signals import pre_dynamic_file_save

@receiver(pre_dynamic_file_save, sender=models.MyModel)
def decide_the_storage_to_save(
instance
, field_file
, to_storage
, *args,
 **kwargs
):
    if not to_storage:
    	# destination_storage is not set, so we set it here
        field_file.destination_storage = MyDynamicStorage(named_param1="something", named_param2="another_thing")
    elif to_storage == wrong_storage:
		# override the destination_storage set earlier
		field_file.destination_storage = MyAnotherDynamicStorage(named_param1="foo", named_param2="bar")

Performance penalty?!

Not even a bit!

HOW?

We are just using the django's built in JsonField instead of CharField  to store more data (init_params output) in addition to the path to the file.

so no extra queries, no extra steps, no performance penalty.

How to migrate from django's native FileField and ImageField?

the schema saved to the JSONField is like this:

{
  "name":  "this/is/the-path/to-the-file", 
  "storage": {
    "constructor": {
      // here is the key values that passed to MyStorageDispatcher.get_storage as **kwargs
    }
  }
}

so just write a custom migration that satisfies this schema