BigTable: Modify system test for new GAPIC code #5302

aneepct · 2018-05-07T16:31:30Z

Modify system test for new GAPIC code

…thub.com/zakons/google-cloud-python into feature/gapic

…into feature/read_rows_retry

aneepct · 2018-05-07T16:32:59Z

@jonparrott @crwilcox I am getting error on create_cluster on system tests.

grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.INVALID_ARGUMENT, Error in field 'clusters' : Need to have at least one cluster map element in CreateInstanceRequest.)>

bigtable/google/cloud/bigtable/instance.py

+        )
+        clusters = []
+        clusters.append(instance_pb2.Cluster(name=cluster_path,
+                                             location=location_path))


bigtable/tests/system.py

 from test_utils.system import unique_resource_id


+PROJECT_ID = 'grass-clump-479'


bigtable/tests/system.py

        Config.CLIENT = Client(admin=True, credentials=credentials)
    else:
-        Config.CLIENT = Client(admin=True)
+        Config.CLIENT = Client(project=PROJECT_ID, admin=True)


bigtable/tests/system.py

+        #     raise ValueError('List instances failed in module set up.')

-        EXISTING_INSTANCES[:] = instances
+        EXISTING_INSTANCES[:] = instances_response.instances


tseaver · 2018-05-08T16:45:38Z

@aneepct Also, can you please rebase against the current master to dump all the redundant commits?

zakons · 2018-05-10T04:27:56Z

@jonparrott @tseaver Hoping this closes the loop on a rather large series of merges on master to support GAPIC. Together with #5275, #5315 and #5283 - which add support for app profile, truncate table and initial splits on top of the GAPIC interface - this should complete the series of changes for the python beta. All have been updated to resolve the review issues found. Please let us know if any issues remain so we can turn these around quickly. Thanks again for the excellent reviews.

bigtable/google/cloud/bigtable/client.py

        """
-        return bigtable_v2.BigtableClient(channel=self._channel,
-                                          credentials=self._credentials)
+        return bigtable_v2.BigtableClient(channel=self._channel)


bigtable/google/cloud/bigtable/client.py

            raise ValueError('Client is not an admin client.')
        return bigtable_admin_v2.BigtableTableAdminClient(
-            channel=self._channel, credentials=self._credentials)
+            channel=self._channel)


bigtable/google/cloud/bigtable/client.py

            raise ValueError('Client is not an admin client.')
        return bigtable_admin_v2.BigtableInstanceAdminClient(
-            channel=self._channel, credentials=self._credentials)
+            channel=self._channel)


bigtable/tests/system.py


-        number_of_bytes = 10 * 1024 * 1024
+        number_of_bytes = 10 * 1024
        data = b'1' * number_of_bytes  # 10MB of 1's.


bigtable/google/cloud/bigtable/instance.py

+        location = self._client._instance_admin_client.location_path(
+            self._client.project, self._cluster_location_id)
+        cluster = instance_pb2.Cluster(name=cluster_name, location=location,
+                                       serve_nodes=DEFAULT_SERVE_NODES)


bigtable/tests/unit/test_client.py

        client = self._make_one(
-            project=project, credentials=credentials, admin=True)
+            project=project, channel=channel, credentials=credentials,
+            admin=True)


bigtable/tests/unit/test_client.py

-        channel = _make_channel()
-        client = self._make_one(project=self.PROJECT, channel=channel)
+        credentials = _make_credentials()
+        client = self._make_one(project=self.PROJECT, credentials=credentials)


bigtable/tests/unit/test_cluster.py

+            google.auth.credentials.Scoped):
+        pass
+
+    return mock.Mock(spec=_CredentialsWithScopes)


aneepct · 2018-05-15T11:48:23Z

@tseaver Please review the recent changes

aneepct · 2018-05-17T04:16:27Z

@tseaver Thank you for all your help.

crwilcox · 2018-05-23T16:33:57Z

@aneepct the tests are failing after this merge. nox -s "system(py='3.6')" run from the bigtable directory will show the failures.

Here is a circle build failing if you want to see what the errors are. https://circleci.com/gh/GoogleCloudPlatform/google-cloud-python/6646

aneepct · 2018-05-25T04:52:41Z

@crwilcox The system test already fixed on different PR #5315

* Provide new auto-generated layer for Bigtable. * Change bigtable_pb2 imports to use from gapic library. * Add retry for read rows * Add parameter start_inclusive to _create_row_request * Add retry for Deadline Exceeded on read rows * Refactor yield_rows retry * Add grpc google iam v1 on setup.py on bigtable * Change routing_header to use to_grpc_metadata

Luke Sneeringer and others added 30 commits January 4, 2018 10:29

Provide new auto-generated layer for Bigtable.

3edca4c

Merge branch 'master' into bigtable-regen

a677959

Add retry for read rows

d540349

Fix line lengths

4aa5702

Add parameter start_inclusive to _create_row_request

f7d4c69

Update unit test case for yield rows retry

28892ce

Fix line lengths

8d141f0

Edit table.py as requested changes

902f11b

Added code review changes from @jonparrott.

4afabb1

Fix parameter name on _RetryableReadRows class

79698b1

Add retry for Deadline Exceeded on read rows

9b778be

Fix line over-indented

cafc5ad

Fix line over-indented

f974752

Merge commit '3edca4c3299e85d4346dae49232d540a2d34002c' of https://gi…

f7242f4

…thub.com/zakons/google-cloud-python into feature/gapic

Merge commit 'a67795951bc9f490265eda7cf511e17d2cc70921' of https://gi…

180d1d5

…thub.com/zakons/google-cloud-python into feature/gapic

Working on gapic integration

f2e39e0

Refactore yield_rows retry

13adfd6

1. Moved last_scanned_row_key into YieldRowsData.

ca69a2f

Fix line over-indented

02c154e

Fix row_data.py

b6da7fa

Change row.py as requested

f1a2d92

Create new test case for yield_rows iterator failure

65c1c4b

Fix for start_inclusive on retry read rows.

170a173

Merge branch 'master' of https://github.com/zakons/google-cloud-python …

5da3fd3

…into feature/read_rows_retry

Change bigtable_pb2 imports to use from gapic library.

f7a58b5

Merge branch 'feature/read_rows_retry' of https://github.com/zakons/g…

c8f664b

…oogle-cloud-python into feature/gapic

Working on gapic

d5f0d98

Integrate bigtable GAPIC client

32df7bd

Refactoring unit tests

77ec950

Refactor to use GAPIC client

efbf2f5

crwilcox mentioned this pull request May 7, 2018

Grpcio pre system tests #5304

Merged

aneepct added 3 commits May 8, 2018 07:40

Remove unused imports

b0646af

Added ClientWithProject on Client

316e59c

Add credentials on unit test cases

1118493

tseaver suggested changes May 8, 2018

View reviewed changes

tseaver added the api: bigtable Issues related to the Bigtable API. label May 8, 2018

aneepct added 9 commits May 9, 2018 07:13

Add ClientWithProject on Client

6e9f95e

Rebase with Master

0f880b1

Rebase with master and add pull client with project

3de1ca3

Make requested changes

4765e3f

Remove unused import

5f03036

Fixing system teest

17f6ac3

Fix instance and system test

b45004f

Create test case for updated instace

c3c9857

Update instance and system tests

35d3199

tseaver suggested changes May 10, 2018

View reviewed changes

aneepct added 3 commits May 11, 2018 12:27

Fixed required changes

8749b94

Fix py lint errors

efef394

Move _make_credentials to _testing

9e2df00

tseaver approved these changes May 16, 2018

View reviewed changes

tseaver merged commit 1b4cf40 into googleapis:master May 16, 2018

crwilcox mentioned this pull request May 23, 2018

Fix Failing BigTable System tests #5362

Closed

		from test_utils.system import unique_resource_id


		PROJECT_ID = 'grass-clump-479'

BigTable: Modify system test for new GAPIC code #5302

BigTable: Modify system test for new GAPIC code #5302

Uh oh!

Conversation

aneepct commented May 7, 2018

Uh oh!

aneepct commented May 7, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

tseaver commented May 8, 2018

Uh oh!

zakons commented May 10, 2018

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

aneepct commented May 15, 2018

Uh oh!

aneepct commented May 17, 2018

Uh oh!

crwilcox commented May 23, 2018

Uh oh!

aneepct commented May 25, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

aneepct commented May 7, 2018 •

edited

Loading