Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[gcloud] Add configurable blob parameters #970

Merged
merged 2 commits into from
Sep 20, 2021
Merged

[gcloud] Add configurable blob parameters #970

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9e5a3dc
[gcloud] Clean up documentation
jschneier Sep 12, 2021
1a8812f
[gcloud] Add configurable blob parameters
jschneier Sep 12, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 63 additions & 24 deletions docs/backends/gcloud.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,24 +38,29 @@ Alternatively, you can use the setting `GS_CREDENTIALS` as described below.

Getting Started
---------------
Set the default storage and bucket name in your settings.py file:
Set the default storage and bucket name in your settings.py file

::
.. code-block:: python

DEFAULT_FILE_STORAGE = 'storages.backends.gcloud.GoogleCloudStorage'
GS_BUCKET_NAME = 'YOUR_BUCKET_NAME_GOES_HERE'


To allow ``django-admin`` collectstatic to automatically put your static files in your bucket set the following in your settings.py::

STATICFILES_STORAGE = 'storages.backends.gcloud.GoogleCloudStorage'

Once you're done, default_storage will be Google Cloud Storage::
Once you're done, default_storage will be Google Cloud Storage

.. code-block:: python

>>> from django.core.files.storage import default_storage
>>> print(default_storage.__class__)
<class 'storages.backends.gcloud.GoogleCloudStorage'>

This way, if you define a new FileField, it will use the Google Cloud Storage::
This way, if you define a new FileField, it will use the Google Cloud Storage

.. code-block:: python

>>> from django.db import models
>>> class Resume(models.Model):
Expand All @@ -69,10 +74,6 @@ This way, if you define a new FileField, it will use the Google Cloud Storage::
Settings
--------

To use gcloud set::

DEFAULT_FILE_STORAGE = 'storages.backends.gcloud.GoogleCloudStorage'

``GS_BUCKET_NAME``

Your Google Storage bucket name, as a string. Required.
Expand All @@ -82,6 +83,8 @@ Your Google Storage bucket name, as a string. Required.
Your Google Cloud project ID. If unset, falls back to the default
inferred from the environment.

.. _gs-creds:

``GS_CREDENTIALS`` (optional)

The OAuth 2 credentials to use for the connection. If unset, falls
Expand All @@ -96,6 +99,8 @@ back to the default inferred from the environment
"path/to/credentials.json"
)

.. _gs-default-acl:

``GS_DEFAULT_ACL`` (optional, default is None)

ACL used when creating a new blob, from the
Expand All @@ -104,7 +109,7 @@ ACL used when creating a new blob, from the
translated.)

For most cases, the blob will need to be set to the ``publicRead`` ACL in order for the file to be viewed.
If GS_DEFAULT_ACL is not set, the blob will have the default permissions set by the bucket.
If ``GS_DEFAULT_ACL`` is not set, the blob will have the default permissions set by the bucket.

``publicRead`` files will return a public, non-expiring url. All other files return
a signed (expiring) url.
Expand All @@ -125,10 +130,6 @@ If set to ``False`` it forces the url not to be signed. This setting is useful i
bucket configured with ``Uniform`` access control configured with public read. In that case you should
force the flag ``GS_QUERYSTRING_AUTH = False`` and ``GS_DEFAULT_ACL = None``

``GS_FILE_CHARSET`` (optional)

Allows overriding the character set used in filenames.

``GS_FILE_OVERWRITE`` (optional: default is ``True``)

By default files with the same name will overwrite each other. Set this to ``False`` to have extra characters appended.
Expand All @@ -147,9 +148,31 @@ must fit in memory. Recommended if you are going to be uploading large files.

This must be a multiple of 256K (1024 * 256)

``GS_CACHE_CONTROL`` (optional: default is ``None``)
``GS_OBJECT_PARAMETERS`` (optional: default is ``{}``)

Dictionary of key-value pairs mapping from blob property name to value.

Use this to set parameters on all objects. To set these on a per-object
basis, subclass the backend and override ``GoogleCloudStorage.get_object_parameters``.

Sets Cache-Control HTTP header for the file, more about HTTP caching can be found `here <https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/http-caching#cache-control>`_
The valid property names are ::

acl
cache_control
content_disposition
content_encoding
content_language
content_type
metadata
storage_class

If not set, the ``content_type`` property will be guessed.

If set, ``acl`` overrides :ref:`GS_DEFAULT_ACL <gs-default-acl>`.

.. warning::

Do not set ``name``. This is set automatically based on the filename.

``GS_CUSTOM_ENDPOINT`` (optional: default is ``None``)

Expand All @@ -165,7 +188,7 @@ Defaults to the root of the bucket.

The time that a generated URL is valid before expiration. The default is 1 day.
Public files will return a url that does not expire. Files will be signed by
the credentials provided to django-storages (See GS_CREDENTIALS).
the credentials provided to django-storages (See :ref:`GS Credentials <gs-creds>`).

Note: Default Google Compute Engine (GCE) Service accounts are
`unable to sign urls <https://googlecloudplatform.github.io/google-cloud-python/latest/storage/blobs.html#google.cloud.storage.blob.Blob.generate_signed_url>`_.
Expand All @@ -180,13 +203,17 @@ Usage
Fields
^^^^^^

Once you're done, default_storage will be Google Cloud Storage::
Once you're done, default_storage will be Google Cloud Storage

.. code-block:: python

>>> from django.core.files.storage import default_storage
>>> print(default_storage.__class__)
<class 'storages.backends.gcloud.GoogleCloudStorage'>

This way, if you define a new FileField, it will use the Google Cloud Storage::
This way, if you define a new FileField, it will use the Google Cloud Storage

.. code-block:: python

>>> from django.db import models
>>> class Resume(models.Model):
Expand All @@ -200,7 +227,9 @@ This way, if you define a new FileField, it will use the Google Cloud Storage::
Storage
^^^^^^^

Standard file access options are available, and work as expected::
Standard file access options are available, and work as expected

.. code-block:: python

>>> default_storage.exists('storage_test')
False
Expand All @@ -222,7 +251,9 @@ Standard file access options are available, and work as expected::
Model
^^^^^

An object without a file has limited functionality::
An object without a file has limited functionality

.. code-block:: python

>>> obj1 = Resume()
>>> obj1.pdf
Expand All @@ -232,7 +263,9 @@ An object without a file has limited functionality::
...
ValueError: The 'pdf' attribute has no file associated with it.

Saving a file enables full functionality::
Saving a file enables full functionality

.. code-block:: python

>>> obj1.pdf.save('django_test.txt', ContentFile('content'))
>>> obj1.pdf
Expand All @@ -242,7 +275,9 @@ Saving a file enables full functionality::
>>> obj1.pdf.read()
'content'

Files can be read in a little at a time, if necessary::
Files can be read in a little at a time, if necessary

.. code-block:: python

>>> obj1.pdf.open()
>>> obj1.pdf.read(3)
Expand All @@ -252,7 +287,9 @@ Files can be read in a little at a time, if necessary::
>>> '-'.join(obj1.pdf.chunks(chunk_size=2))
'co-nt-en-t'

Save another file with the same name::
Save another file with the same name

.. code-block:: python

>>> obj2 = Resume()
>>> obj2.pdf.save('django_test.txt', ContentFile('more content'))
Expand All @@ -261,7 +298,9 @@ Save another file with the same name::
>>> obj2.pdf.size
12

Push the objects into the cache to make sure they pickle properly::
Push the objects into the cache to make sure they pickle properly

.. code-block:: python

>>> cache.set('obj1', obj1)
>>> cache.set('obj2', obj2)
Expand Down
35 changes: 31 additions & 4 deletions storages/backends/gcloud.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import mimetypes
import warnings
from datetime import timedelta
from tempfile import SpooledTemporaryFile

Expand All @@ -23,6 +24,9 @@
"See https://github.com/GoogleCloudPlatform/gcloud-python")


CONTENT_TYPE = 'content_type'


class GoogleCloudFile(File):
def __init__(self, name, mode, storage):
self.name = name
Expand Down Expand Up @@ -106,6 +110,7 @@ def get_default_settings(self):
"expiration": setting('GS_EXPIRATION', timedelta(seconds=86400)),
"file_overwrite": setting('GS_FILE_OVERWRITE', True),
"cache_control": setting('GS_CACHE_CONTROL'),
"object_parameters": setting('GS_OBJECT_PARAMETERS', {}),
# The max amount of memory a returned file can take up before being
# rolled over into a temporary file on disk. Default is 0: Do not
# roll over.
Expand Down Expand Up @@ -154,12 +159,34 @@ def _save(self, name, content):

content.name = cleaned_name
file = GoogleCloudFile(name, 'rw', self)
file.blob.cache_control = self.cache_control
file.blob.upload_from_file(
content, rewind=True, size=content.size,
content_type=file.mime_type, predefined_acl=self.default_acl)

upload_params = {}
blob_params = self.get_object_parameters(name, content)
if 'cache_control' not in blob_params and self.cache_control:
warnings.warn(
'The GS_CACHE_CONTROL setting is deprecated. Use GS_OBJECT_PARAMETERS to set any '
'writable blob property or override GoogleCloudStorage.get_object_parameters to '
'vary the parameters per object.', DeprecationWarning
)
blob_params['cache_control'] = self.cache_control

upload_params['predefined_acl'] = blob_params.pop('acl', self.default_acl)
if CONTENT_TYPE not in blob_params:
upload_params[CONTENT_TYPE] = file.mime_type

for prop, val in blob_params.items():
setattr(file.blob, prop, val)

file.blob.upload_from_file(content, rewind=True, size=content.size, **upload_params)
return cleaned_name

def get_object_parameters(self, name, content):
"""Override this to return a dictionary of overwritable blob-property to value.

Returns GS_OBJECT_PARAMETRS by default. See the docs for all possible options.
"""
return self.object_parameters.copy()

def delete(self, name):
name = self._normalize_name(clean_name(name))
try:
Expand Down