Bigtable: Create table in list_table test, deadline exceed error fix

[sangramql]

[issue 8479]
Bigtable: 'test_bigtable_list_tables' snippet flakes with '504 Deadline Exceeded'.

[sangramql]

@mf2199 please review

[mf2199]

Two remarks before reviewing this PR:

1. Like we discussed a few weeks ago, could you please clone our Q-logic repository and male PRs from/to it? This way you will still be seen as a contributor, while all our PRs will be traceable in one place. Hemang, Ilya, Leonid and I are already onboard with this.

2. Could you please include a description stating the gist of the issue, your proposed way of solving it, and if there are multiple ways then why yours is the most optimal. You can do it either, here, in the OpenProject task page, on our Wiki site or even in an email. We will need this late for our Wiki site. All the extra time needed for the write-up is the same billable time and should be logged same as working on the issue. And don't worry about perfecting the language, as this will probably be rephrased by our technical writer(s) anyway.

Thanks!

[mf2199]

Like mentioned before, we need to describe your reasoning behind these changes. Especially when it comes to test flakes, it is important to know and record their root cause as they may happen again with a different product.

Also, have you tested these changes enough to make sure there are no more flakes?

[sangramql]

Like we discussed a few weeks ago, could you please clone our Q-logic repository and male PRs from/to it? This way you will still be seen as a contributor, while all our PRs will be traceable in one place. Hemang, Ilya, Leonid and I are already onboard with this.

Cloned it. Will raise PR using this repo.

[sangramql]

Could you please include a description stating the gist of the issue, your proposed way of solving it, and if there are multiple ways then why yours is the most optimal. You can do it either, here, in the OpenProject task page, on our Wiki site or even in an email. We will need this late for our Wiki site. All the extra time needed for the write-up is the same billable time and should be logged same as working on the issue. And don't worry about perfecting the language, as this will probably be rephrased by our technical writer(s) anyway.

This issue is not consistent in production i suppose. And I was not able to reproduce even once.
I tried different approach to hit this issue,

1. Created multiple tables(>100) in test function.
2. Created instance and immediately created multiple tables (> 100) in class setup.
3. Tried to created multiple instances (>10) and table in an instance, but got the following error after creating 6 instances. But didn’t get ‘deadline exceeded’ error though.

—error—
E grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
E status = StatusCode.RESOURCE_EXHAUSTED
E details = "Insufficient node quota. You requested a node count of 3 nodes for your cluster, but this request would exceed your project's node quota of 30 nodes total across all clusters in this zone. Contact us to request a quota increase: https://goo.gl/pf2Su8"
E debug_error_string = "{"created":"@1564490401.065480000","description":"Error received from peer ipv4:216.58.203.138:443","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Insufficient node quota. You requested a node count of 3 nodes for your cluster, but this request would exceed your project's node quota of 30 nodes total across all clusters in this zone. Contact us to request a quota increase: https://goo.gl/pf2Su8","grpc_status":8}"
E >
.nox/snippets-3-7/lib/python3.7/site-packages/grpc/_channel.py:467: _Rendezvous
—/error—

[sangramql]

Like mentioned before, we need to describe your reasoning behind these changes. Especially when it comes to test flakes, it is important to know and record their root cause as they may happen again with a different product.

This fix is added to avoid creating and deleting table for single test. Instead, common table can be created and used in tests which require table specific tests, except table creation test.

Also, have you tested these changes enough to make sure there are no more flakes?

Like mentioned in above comment, this is been tested with various approaches and with multiple tables(>100) in single instance as well as multiple instances(10) with single table.
Tables were created immediately after creating instances w/o waiting for cool down time .

Cool down time (timeout=100) code was avoided to hit the issue, but was not hit.
---code---
operation = Config.INSTANCE.create(clusters=[cluster])
#We want to make sure the operation completes.
#operation.result(timeout=100) #this code is commented to reproduce issue.
Config.TABLE = Config.INSTANCE.table(TABLE_ID) Config.TABLE.create()
---/code---

[sangramql]

Like we discussed a few weeks ago, could you please clone our Q-logic repository and male PRs from/to it? This way you will still be seen as a contributor, while all our PRs will be traceable in one place. Hemang, Ilya, Leonid and I are already onboard with this.

Cloned it. Will raise PR using this repo.

Unable to raise PR on qlogic fork.
remote: Permission to q-logic/google-cloud-python.git denied to sangramql.

[sangramql]

Closing this PR as this opened in googleapis#8879