feat(datastore): Add Datastore Admin API samples

[IlyaFaer]

Fixes#4120

[IlyaFaer]

@BenWhitehead, @crwilcox, I've used the code generated by Alex Vaskman (see the generation configurations PR), to run tests. They are okay:

We need an access for Datastore Admin and Storage simultaneously to write tests for entities export/import on Python and Go. I'll ask Deepika about this on our meeting today.

I've written the code in Datastore fashion for now. Ready for any changes requests.

[BenWhitehead]

I think it's okay to not have tests for the export/import calls, since those are LongRunningOperations and depend on bucket access/config.

[tmatsuo]

@BenWhitehead

I think we should add a test.

For bucket access and config, please tell us the requirements. We can set it up.
For long running operations, you can still test it. Wait for the result with a timeout that gives you around 90% success rate, then retry the test 2 or 3 times.

[BenWhitehead]

Hey @tmatsuo,

This is the quick list of what I believe would be needed to run import/export integration tests:

1. A storage bucket (ideally located in the same region is the datastore instance)
   • Some way of passing this bucket into the tests (env variable?)
2. Service Account used to run the tests will need
   • Cloud Datastore Import Export Admin, or Cloud Datastore Owner or Owner (least to most permissive)
   • Storage Admin, or Owner (least to most permissive)
3. The App Engine default service account of GCP Project used for integration tests needs to exist (it is the service account used by the backend jobs to do the import/export)
   • Cloud Datastore Import Export Admin, or Cloud Datastore Owner or Owner (least to most permissive)
   • Storage Admin, or Owner (least to most permissive) on the bucket from 1

Datastore Exporting and importing entities Permissions

[tmatsuo]

@BenWhitehead

A storage bucket (ideally located in the same region is the datastore instance)

We're already using a storage bucket specified in CLOUD_STORAGE_BUCKET envvar. On Kokoro, this is set to python-docs-samples-tests bucket. I hope you can just use it.

Re: permission

I added those permissions to the service accounts.

I think Storage Object Admin is enough, instead of Storage Admin, so I only applied Storage Object Admin on the bucket to the app engine default service account. Let me know if we actually need Storage Admin.

[tmatsuo]

@IlyaFaer

Can you also update the required env vars and permissions to run the tests in the README file? Actually you'll need to update README.rst.in and run the readmegen nox session to re-generate the README.rst file.

I think you can use required_role and other_required_steps field in that file (Example usage).

[IlyaFaer]

@BenWhitehead, @tmatsuo, looks like we were provided with all the necessary permissions in our project, thanks. I've run the tests locally - they're okay (I assume on kokoro they'll be failing until the next Datastore release):

I've unioned import and export tests as it's easier to run them sequentially and use the results of the export as an argument for import. Hope it's okay.

And, if I understood correctly, this CLOUD_STORAGE_BUCKET already exists in samples kokoro environment, so I don't need to create/delete it on every test run.

Can you also update the required env vars and permissions to run the tests in the README file? Actually you'll need to update README.rst.in and run the readmegen nox session to re-generate the README.rst file

Pushed. There is no noxfile.py in datastore directory by the way. Should be generated I guess.

[IlyaFaer]

Thanks, @crwilcox! All the files are there now. I suppose only renovate PR is left, and kokoro checks will become green.

[tmatsuo]

@IlyaFaer FYI, I think the test will automatically executed if you join googlecloudplatform org.

[BenWhitehead]

@IlyaFaer looks like there are some test bootstrapping issues

==================================== ERRORS ====================================
____________ ERROR collecting datastore/cloud-client/admin_test.py _____________
ImportError while importing test module '/workspace/datastore/cloud-client/admin_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
admin_test.py:19: in <module>
import admin
admin.py:15: in <module>
from google.cloud.datastore_admin_v1.gapic import datastore_admin_client
E ModuleNotFoundError: No module named 'google.cloud.datastore_admin_v1.gapic'
----- generated xml file: /workspace/datastore/cloud-client/sponge_log.xml -----
=========================== short test summary info ============================
ERROR admin_test.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.51s ===============================

[IlyaFaer]

@BenWhitehead, @crwilcox, I've updated the branch with renovate PR changes. Now another error appeared:

____________ERRORcollectingdatastore/cloud-client/admin_test.py_____________Traceback (mostrecentcalllast):
File"/workspace/datastore/cloud-client/admin_test.py", line19, in<module>importadminFile"/workspace/datastore/cloud-client/admin.py", line15, in<module>fromgoogle.cloud.datastore_admin_v1.gapicimportdatastore_admin_clientFile"/workspace/datastore/cloud-client/.nox/py-3-6/lib/python3.6/site-packages/google/cloud/datastore_admin_v1/__init__.py", line23, in<module>fromgoogle.cloud.datastore_admin_v1.gapicimportdatastore_admin_clientFile"/workspace/datastore/cloud-client/.nox/py-3-6/lib/python3.6/site-packages/google/cloud/datastore_admin_v1/gapic/datastore_admin_client.py", line48, in<module>"google-cloud-datastore-admin",
File"/workspace/datastore/cloud-client/.nox/py-3-6/lib/python3.6/site-packages/pkg_resources/__init__.py", line481, inget_distributiondist=get_provider(dist)
File"/workspace/datastore/cloud-client/.nox/py-3-6/lib/python3.6/site-packages/pkg_resources/__init__.py", line357, inget_providerreturnworking_set.find(moduleOrReq) orrequire(str(moduleOrReq))[0]
File"/workspace/datastore/cloud-client/.nox/py-3-6/lib/python3.6/site-packages/pkg_resources/__init__.py", line900, inrequireneeded=self.resolve(parse_requirements(requirements))
File"/workspace/datastore/cloud-client/.nox/py-3-6/lib/python3.6/site-packages/pkg_resources/__init__.py", line786, inresolveraiseDistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The'google-cloud-datastore-admin'distributionwasnotfoundandisrequiredbytheapplication

I suspect the problem caused with the generated lines:
https://github.com/googleapis/python-datastore/blob/592c7aa636990533a9c02f73a413de44b71dcc98/google/cloud/datastore_admin_v1/gapic/datastore_admin_client.py#L47-L49

There is no such a Python package google-cloud-datastore-admin.

[IlyaFaer]

@tmatsuo, @crwilcox, @BenWhitehead, all the checks are green 🎉
Thanks for the re-generation and dependencies!