Fixing issue with UTF-8 conversion in Python3 for GCS uploads

[bucklander]

SUMMARY

Fixing issue with UTF-8 conversion in Python3 for GCS uploads (#266)

ISSUE TYPE

• Bugfix Pull Request

COMPONENT NAME

gcp_storage_object

ADDITIONAL INFORMATION

This bug can be resolved by simply changing gcp_storage_object.py#L254 from ...

with open(src, "r") as file_obj: to with open(src, "rb") as file_obj:

Hence, this PR. This small change ensures there's no decoding attempt (or subsequent error) by forcing the file to be read as binary via the open func.

This approach appears similar to download_file's use of the open func at gcp_storage_object.py#L243, where it specifically is configured with the b (binary) mode specifier. Not only does this resolve the issue with uploading binary files, but the change doesn't appear to harm text uploads. Unsure though if this is a safe strategy overall and should be probably be reviewed carefully by Google Ansible module maintainers.

$ ansible-playbook --connection=local --inventory localhost, upload-text-file.yml -e ansible_python_interpreter=`which python2`


PLAY [Upload a file to GCS using gcp_storage_object module] ****************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************
ok: [localhost]

TASK [Upload Latest Backup(s) to GCS Bucket] *******************************************************************************************************************************************************
changed: [localhost]

PLAY RECAP *****************************************************************************************************************************************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

$ ansible-playbook --connection=local --inventory localhost, upload-text-file.yml -e ansible_python_interpreter=`which python3`


PLAY [Upload a file to GCS using gcp_storage_object module] ****************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************
ok: [localhost]

TASK [Upload Latest Backup(s) to GCS Bucket] *******************************************************************************************************************************************************
changed: [localhost]

PLAY RECAP *****************************************************************************************************************************************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

$ ansible-playbook --connection=local --inventory localhost, upload-bin-file.yml -e ansible_python_interpreter=`which python2`

PLAY [Upload a binary file to Google Cloud Storage bucket using gcp_storage_object module] *********************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************
ok: [localhost]

TASK [Upload Latest Backup(s) to GCS Bucket] *******************************************************************************************************************************************************
changed: [localhost]

PLAY RECAP *****************************************************************************************************************************************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

$ ansible-playbook --connection=local --inventory localhost, upload-bin-file.yml -e ansible_python_interpreter=`which python3`

PLAY [Upload a binary file to Google Cloud Storage bucket using gcp_storage_object module] *********************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************
ok: [localhost]

TASK [Upload Latest Backup(s) to GCS Bucket] *******************************************************************************************************************************************************
changed: [localhost]

PLAY RECAP *****************************************************************************************************************************************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

[bucklander]

Additionally, according to this Google Cloud doc, by using curl's --data-binary arg the preference appears to be to read and use the given file in binary exactly as given (I.e. no conversion) before attempting upload:

curl -X POST --data-binary @[OBJECT_LOCATION] \
-H "Authorization: Bearer [OAUTH2_TOKEN]" \
-H "Content-Type: [OBJECT_CONTENT_TYPE]" \
"https://storage.googleapis.com/upload/storage/v1/b/[BUCKET_NAME]/o?uploadType=media&name=[OBJECT_NAME]"

It'd seem that this change introduces consistent behavior.

[bucklander]

[alvaro-gh]

I think I spent two hours until finding this. Good old Ansible.

[bucklander]

I've since given up here, doesn't seem to me that Google Cloud is investing much in their Ansible module collection. If anyone else runs across this issue, I recommend interacting with Google's Cloud APIs as much as possible simply using Ansible's URI module instead.

Simple play set example for GCE instance uploading to GCS bucket:

- name: Retrieve GCS Access Token
        uri:
          url: http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token
          method: GET
          headers:
            Metadata-Flavor: Google
          return_content: yes
        run_once: true
        register: gcs_token_resp

- name: Upload Tarball to GCS Bucket
        uri:
          url: https://storage.googleapis.com/upload/storage/v1/b/{{ gcs_bucket_name }}/o?uploadType=media&name={{ filename }}.tar
          timeout: 720
          method: POST
          src: {{ filename }}.tar
          remote_src: yes
          return_content: yes
          headers:
            Authorization: Bearer {{ gcs_token_resp.json.access_token }}
            Content-Type: application/x-tar
        ignore_errors: yes

[dperezro]

Hello! When will this be merged?

[dperezro]

[levonet]

Maybe there is some alternative to google.cloud collection?

[dperezro]

@levonet there is... but it's a shame they do not fix this and abandon it.

[dalbani]

@dperezro
What is the alternative that you known of?

[levonet]

[dperezro]

@dperezro
What is the alternative that you known of?

Google's Cloud APIs, using gcloud commands (although it's interactive so you need to run gcloud login first) or make this correction locally (the PR change) :-\

[ikauzak]

Still facing the same issue. Hope someone fixes this asap D:

[bucklander]

hail marry attempt for merge - @rambleraptor

[levonet]

I finally realized it was a conspiracy between google and ansible. The purpose of the conspiracy is to migrate everyone to k8s, helm, and terraform. It happened to me :/

[alvaro-gh]

@rambleraptor mentioning you since you seem to be main contributor. Any chance this can be merged? Pretty please?

[nkvojvodic]

I find GCP's lack of interest in merging simple, badly needed fixes...disturbing.

@rmoriar1
Any chance of getting this merged?

[alvaro-gh]

Thanks for merging that, I noticed that the last release for this collection was on February 2020. I spent all day doing a manual fix to get this specific commit as a requirement in AWX, turns out that AWX with Ansible 2.9 can't use requirements.yml with a git version.
So, here's hoping for a 1.0.3 release.
Thanks again, I first noticed this issue on 09/10/2021.

[jessequinn]

Should help someone

requirements.yml

collections:
  - name: git+https://github.com/ansible-collections/google.cloud.git