-
-
Notifications
You must be signed in to change notification settings - Fork 865
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Google Cloud Storage backend using the gcloud-python library #236
Merged
Merged
Changes from 25 commits
Commits
Show all changes
28 commits
Select commit
Hold shift + click to select a range
2f6f6e7
Add Google Cloud Storage backend using the gcloud-python library
eirsyl 85c53b5
Rename to gcloud, and import google.cloud
scjody b3ec8b5
Add @deconstructible to GoogleCloudStorage
scjody 1ddc32b
Remove MIME type guessing
scjody 8105cba
Fix name in error message
scjody 7cb67d5
Move clean_name to utils
scjody 159b003
Move clean_name tests to test_utils
scjody 621391a
Use utils.clean_name()
scjody 058972f
Remove _normalize_name
scjody cbe2530
Remove unused import and class variable
scjody 189233a
Move safe_join to utils
scjody 3829f9a
Add and use _normalize_name() like in s3boto
scjody e16800e
Remove unused function
scjody 0d61513
Only create a Blob in write mode
scjody dfe90b5
Add tests of Google Cloud Storage
scjody 9ad54cb
Add documentation for Google Cloud Storage
scjody b37a9cb
Import Storage directly
scjody 74f3841
Use byte string for test read
scjody e8fc9fb
Add Google Cloud Storage authors
scjody 737ac57
Address review comments
scjody 44f39cd
Fix modified_time; add get_modified_time
scjody bb9307f
Test and fix unicode handling
scjody acbe31d
Address further review comments
scjody 02e5829
Remove *args and **kwargs
scjody 1d2e206
Add deprecation notice to 'gs' backend
scjody bbb203e
Print deprecation notice as warning
scjody 598753b
Simplify ACL options and improve ACL documentation
scjody 7ee116b
Address final PR comments
scjody File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,190 @@ | ||
Google Cloud Storage | ||
==================== | ||
|
||
Usage | ||
***** | ||
|
||
This backend provides support for Google Cloud Storage using the | ||
library provided by Google. | ||
|
||
It's possible to access Google Cloud Storage in S3 compatibility mode | ||
using other libraries in django-storages, but this is the only library | ||
offering native support. | ||
|
||
By default this library will use the credentials associated with the | ||
current instance for authentication. To override this, see the | ||
settings below. | ||
|
||
|
||
Settings | ||
-------- | ||
|
||
To use gcloud set:: | ||
|
||
DEFAULT_FILE_STORAGE = 'storages.backends.gcloud.GoogleCloudStorage' | ||
|
||
``GS_BUCKET_NAME`` | ||
|
||
Your Google Storage bucket name, as a string. | ||
|
||
``GS_PROJECT_ID`` (optional) | ||
|
||
Your Google Cloud project ID. If unset, falls back to the default | ||
inferred from the environment. | ||
|
||
``GS_CREDENTIALS`` (optional) | ||
|
||
The OAuth 2 credentials to use for the connection. If unset, falls | ||
back to the default inferred from the environment. | ||
|
||
``GS_AUTO_CREATE_BUCKET`` (optional, default is ``False``) | ||
|
||
If True, attempt to create the bucket if it does not exist. | ||
|
||
``GS_DEFAULT_ACL`` (optional) | ||
|
||
If set to ``private`` changes uploaded file's Access Control List from the default permission ``public-read`` to give owner full control and remove read access from everyone else. | ||
|
||
``GS_BUCKET_ACL`` (optional) | ||
|
||
ACL used when creating a new bucket; defaults to ``GS_DEFAULT_ACL``. | ||
|
||
``GS_FILE_CHARSET`` (optional) | ||
|
||
Allows overriding the character set used in filenames. | ||
|
||
``GS_FILE_OVERWRITE`` (optional: default is ``True``) | ||
|
||
By default files with the same name will overwrite each other. Set this to ``False`` to have extra characters appended. | ||
|
||
``GS_MAX_MEMORY_SIZE`` (optional) | ||
|
||
The maximum amount of memory a returned file can take up before being | ||
rolled over into a temporary file on disk. Default is 0: Do not roll over. | ||
|
||
Fields | ||
------ | ||
|
||
Once you're done, default_storage will be Google Cloud Storage:: | ||
|
||
>>> from django.core.files.storage import default_storage | ||
>>> print default_storage.__class__ | ||
<class 'storages.backends.gcloud.GoogleCloudStorage'> | ||
|
||
This way, if you define a new FileField, it will use the Google Cloud Storage:: | ||
|
||
>>> from django.db import models | ||
>>> class Resume(models.Model): | ||
... pdf = models.FileField(upload_to='pdfs') | ||
... photos = models.ImageField(upload_to='photos') | ||
... | ||
>>> resume = Resume() | ||
>>> print resume.pdf.storage | ||
<storages.backends.gcloud.GoogleCloudStorage object at ...> | ||
|
||
Storage | ||
------- | ||
|
||
Standard file access options are available, and work as expected:: | ||
|
||
>>> default_storage.exists('storage_test') | ||
False | ||
>>> file = default_storage.open('storage_test', 'w') | ||
>>> file.write('storage contents') | ||
>>> file.close() | ||
|
||
>>> default_storage.exists('storage_test') | ||
True | ||
>>> file = default_storage.open('storage_test', 'r') | ||
>>> file.read() | ||
'storage contents' | ||
>>> file.close() | ||
|
||
>>> default_storage.delete('storage_test') | ||
>>> default_storage.exists('storage_test') | ||
False | ||
|
||
Model | ||
----- | ||
|
||
An object without a file has limited functionality:: | ||
|
||
>>> obj1 = MyStorage() | ||
>>> obj1.normal | ||
<FieldFile: None> | ||
>>> obj1.normal.size | ||
Traceback (most recent call last): | ||
... | ||
ValueError: The 'normal' attribute has no file associated with it. | ||
|
||
Saving a file enables full functionality:: | ||
|
||
>>> obj1.normal.save('django_test.txt', ContentFile('content')) | ||
>>> obj1.normal | ||
<FieldFile: tests/django_test.txt> | ||
>>> obj1.normal.size | ||
7 | ||
>>> obj1.normal.read() | ||
'content' | ||
|
||
Files can be read in a little at a time, if necessary:: | ||
|
||
>>> obj1.normal.open() | ||
>>> obj1.normal.read(3) | ||
'con' | ||
>>> obj1.normal.read() | ||
'tent' | ||
>>> '-'.join(obj1.normal.chunks(chunk_size=2)) | ||
'co-nt-en-t' | ||
|
||
Save another file with the same name:: | ||
|
||
>>> obj2 = MyStorage() | ||
>>> obj2.normal.save('django_test.txt', ContentFile('more content')) | ||
>>> obj2.normal | ||
<FieldFile: tests/django_test_.txt> | ||
>>> obj2.normal.size | ||
12 | ||
|
||
Push the objects into the cache to make sure they pickle properly:: | ||
|
||
>>> cache.set('obj1', obj1) | ||
>>> cache.set('obj2', obj2) | ||
>>> cache.get('obj2').normal | ||
<FieldFile: tests/django_test_.txt> | ||
|
||
Deleting an object deletes the file it uses, if there are no other objects still using that file:: | ||
|
||
>>> obj2.delete() | ||
>>> obj2.normal.save('django_test.txt', ContentFile('more content')) | ||
>>> obj2.normal | ||
<FieldFile: tests/django_test_.txt> | ||
|
||
Default values allow an object to access a single file:: | ||
|
||
>>> obj3 = MyStorage.objects.create() | ||
>>> obj3.default | ||
<FieldFile: tests/default.txt> | ||
>>> obj3.default.read() | ||
'default content' | ||
|
||
But it shouldn't be deleted, even if there are no more objects using it:: | ||
|
||
>>> obj3.delete() | ||
>>> obj3 = MyStorage() | ||
>>> obj3.default.read() | ||
'default content' | ||
|
||
Verify the fix for #5655, making sure the directory is only determined once:: | ||
|
||
>>> obj4 = MyStorage() | ||
>>> obj4.random.save('random_file', ContentFile('random content')) | ||
>>> obj4.random | ||
<FieldFile: .../random_file> | ||
|
||
Clean up the temporary files:: | ||
|
||
>>> obj1.normal.delete() | ||
>>> obj2.normal.delete() | ||
>>> obj3.default.delete() | ||
>>> obj4.random.delete() |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps list the options here or reference to https://cloud.google.com/storage/docs/access-control/lists#predefined-acl.
In addition, the predefined ACL aliases for JSON API are different than XML API.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
From my understanding of the docs, you should be able to use either.
Unfortunately while testing this I ran into what's either a bug in the
google-cloud
python library or a huge misunderstanding on my part. I'm going to submit a support request to Google and see where that goes, and I'll update the PR when I have more information.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems like
google-cloud
is using JSON API and does transform predefined ACL aliases from XML to JSON format here. I would suggest only reference to JSON format here, e.g.s/public-read/publicRead/g
.