Add retry (exponential backoff) for storage api functions #108
Description
While using the python-storage library, we were getting 503 or 504 errors from time to time when deleting or moving objects.
I looked up the docs for cloud storage and came across this:
Clients should use truncated exponential backoff for all requests to Cloud Storage that return HTTP 5xx and 429 response codes, including uploads and downloads of data or metadata.
At the bottom of the page there is also:
Examples of backoff implemented in client libraries that you can use with Cloud Storage include:
- Retrying library for Python.
- Google Cloud Client Libraries for Node.js can automatically use backoff strategies to retry requests with the autoRetry parameter.
It would be super useful and in my opinion necessary to include retrying functionality into the python-storage library.
It was also a bit confusing to me that the official docs are recommending a third party retrying library while already having google.api.core.retry inside google cloud python sdk.
The google.api.core.retry docs mention this:
Some client library methods apply retry automatically. These methods can accept a retry parameter that allows you to configure the behavior
A potential solution could be to add the retry parameter of type google.api.core.retry to public api functions or while creating the storage client. Developers could specify if they want exponential backoff and set the parameters as they like (maximum delay, deadline, etc.)
Java and Node.js libraries already have similar functionality and it seems that this can be useful when cloud storage is facing some issues or high traffic load, or in the case where users are hitting the limits of 1k writes or 5k reads per second and GCS is autoscaling.
If this is already supported in the library, please direct me to some docs, I wasn't able to find it...