Skip to content

feat: Add multiprocessing and chunked downloading to transfer manager #1002

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
andrewsg merged 14 commits into from
Mar 21, 2023
Merged

feat: Add multiprocessing and chunked downloading to transfer manager #1002

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
e970531
multiprocessing intitial commit
andrewsg Feb 15, 2023
52375c7
added multiprocessing, fixed tests; tests for multiprocessing chunks …
andrewsg Mar 6, 2023
91ed03f
unit tests for multiprocessing
andrewsg Mar 7, 2023
c2eff05
add system tests, fix off-by-one error detected by system tests
andrewsg Mar 7, 2023
286f38a
lint
andrewsg Mar 7, 2023
d7f5acf
exclude subprocess mocks from coverage
andrewsg Mar 7, 2023
5f94115
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Mar 8, 2023
527bcbf
add deadline to system tests
andrewsg Mar 9, 2023
00e48c5
ban broken grpcio version
andrewsg Mar 13, 2023
c6b1a4a
Clarify docstring for threads param
andrewsg Mar 14, 2023
0be3fe0
fix coverage issues and catch warning
andrewsg Mar 14, 2023
07e1b3d
respond to feedback
andrewsg Mar 21, 2023
6cc0d4d
more feedback
andrewsg Mar 21, 2023
f778bee
Merge branch 'main' into multiprocessing
andrewsg Mar 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions google/cloud/storage/client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,11 @@ def __init__(
if project is _marker:
project = None

# Save the initial value of client_info and client_options before they
# are passed along, for use in __reduce__ defined elsewhere.
self._initial_client_info = client_info
self._initial_client_options = client_options

kw_args = {"client_info": client_info}

# `api_endpoint` should be only set by the user via `client_options`,
Expand Down
Loading