Storage: Cannot download brotli compressed/encoded files from cloud storage

[m3o7]

Environment details

Issue caused when deployed as cloud function (with google-cloud-storage==1.15.0):

gcloud functions deploy generate_file --runtime python37 --trigger-http

Steps to reproduce

1. generate string content
2. compress content with brotli
3. create new blob
4. set blob encoding to br for brotli
5. upload blob via upload_from_string
6. download blob via download_as_string

Code example

https://gist.github.com/marcopashkov/9175be7c72f96ffc8ea984d892e07f98

Stack trace

textPayload: "Traceback (most recent call last):
  File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 346, in run_http_function
    result = _function_handler.invoke_user_function(flask.request)
  File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 217, in invoke_user_function
    return call_user_function(request_or_event)
  File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 210, in call_user_function
    return self._user_function(request_or_event)
  File "/user_code/main.py", line 26, in generate_file
    download_content = blob.download_as_string()
  File "/env/local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 697, in download_as_string
    self.download_to_file(string_buffer, client=client, start=start, end=end)
  File "/env/local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 636, in download_to_file
    self._do_download(transport, file_obj, download_url, headers, start, end)
  File "/env/local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 574, in _do_download
    download.consume(transport)
  File "/env/local/lib/python3.7/site-packages/google/resumable_media/requests/download.py", line 174, in consume
    self._write_to_stream(result)
  File "/env/local/lib/python3.7/site-packages/google/resumable_media/requests/download.py", line 137, in _write_to_stream
    raise common.DataCorruption(response, msg)
google.resumable_media.common.DataCorruption: Checksum mismatch while downloading:

  https://www.googleapis.com/download/storage/v1/b/REDACTED.appspot.com/o/brotli_test%2F2019-08-09.html?generation=1565385647421287&alt=media

The X-Goog-Hash header indicated an MD5 checksum of:

  O0XeX+jfZFb9nQYmC6DU2Q==

but the actual MD5 checksum of the downloaded contents was:

  qawBTvkgTjJkJklqx11tRw==

" 

[m3o7]

Notes:

• Using gzip instead of brotli works just fine.
• I also tried to calculated the md5 hash and set it manually for the blob before uploading (see below) - to no avail
• the exact same code runs flawlessly on my local machine (my full pip freeze output), but as soon as I deploy it to cloud storage it breaks

md5 = hashlib.md5()
md5.update(content)
blob.md5_hash = base64.b64encode(md5.digest()).decode('utf-8')

[m3o7]

I found a temporary workaround: After [binary] searching my local pip packages I found that using urllib3==1.23 will fix the issue. It appears by default urllib3=1.25.3 is currently installed, potentially breaking the MD5 calculation.

[tseaver]

@marcopashkov I believe this issue will be fixed in the upcoming 0.4.0 release of google-resumable-media-python: could you check against the current master of that repository.

[HemangChothani]

@marcopashkov The bug is related to google-resumable-media library.
While not working with the versions that you have mentioned, the issue has been resolved in the updated version of google-resumable-media which is going to be released soon.
When tested with google-cloud-storage >1.17.0 and the new version of google-resumable-media library, the function works fine with both, brotli compressed and gzip compressed files.

[HemangChothani]

google-resumable-media 0.4.0 released, feel free to reopen if new release of google-resumable-media doesn't solve your issue completely.

[william-silversmith]

Hi all,

Unfortunately this was broken in google-resumable-media 0.4.1... The backwards compatibility note in 0.4.0 negatively impacted google-cloud-python storage.

Backward-Compatibility Note
The change to use raw response data (PR
#87)
might break the hypothetical usecase of downloading a blob marked with
Content-Encoding: gzip and expecting to get the expanded data.

I discussed this issue in #9188

I still think it would be a good idea regardless to implement the patch I suggested there. This enables users to acquire the content encoding and the data in one request (which is required by the HPC application I work on) which could be especially useful if odd encodings are used.

In this case, I could manually apply brotli decompression even if GCP didn't support it.

Thanks for reading!

[tseaver]

@william-silversmith FYI I just merged googleapis/google-resumable-media-python#109, which adds back "raw downloads" as an opt-in feature, which will be avaialabe in google-resumable-media 0.5.0 (release PR here).

I will be working this week on adding opt-in support for such downloads in google-cloud-storage.