diff --git a/.coveragerc b/.coveragerc
index 6a18f761d2c4..8aff2b6744ca 100644
--- a/.coveragerc
+++ b/.coveragerc
@@ -6,7 +6,8 @@ omit =
*/_generated/*.py
# Packages in the "google.cloud" package that we don't own.
*/google/cloud/gapic/*
-fail_under = 100
+ */google/cloud/grpc/*
+ */google/cloud/proto/*
show_missing = True
exclude_lines =
# Re-enable the standard pragma
diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS
new file mode 100644
index 000000000000..fcf34e569555
--- /dev/null
+++ b/.github/CODEOWNERS
@@ -0,0 +1,52 @@
+# Code owners file.
+# This file controls who is tagged for review for any given pull request.
+
+# Bu Sun Kim is the default owner for anything not explicitly
+# taken by someone else.
+* @busunkim96
+
+##################################
+# Storage and Database (SoDa) team
+##################################
+bigtable/ @tseaver @crwilcox @frankyn
+docs/bigtable/ @tseaver @crwilcox @frankyn
+
+datastore/ @tseaver @crwilcox @frankyn
+docs/datastore @tseaver @crwilcox @frankyn
+
+storage/ @tseaver @crwilcox @frankyn
+docs/storage @tseaver @crwilcox @frankyn
+
+firestore/ @tseaver @crwilcox @frankyn
+docs/firestore/ @tseaver @crwilcox @frankyn
+
+spanner/ @tseaver @crwilcox @frankyn
+docs/spanner @tseaver @crwilcox @frankyn
+
+###############################
+# Analytics and ML (AniML) team
+###############################
+
+# BigQuery suite of APIs
+bigquery/ @googleapis/api-bigquery
+docs/bigquery/ @googleapis/api-bigquery
+
+bigquery_datatransfer/ @googleapis/api-bigquery
+docs/bigquery_datatransfer/ @googleapis/api-bigquery
+
+bigquery_storage/ @googleapis/api-bigquery
+docs/bigquery_storage/ @googleapis/api-bigquery
+
+# Pubsub
+pubsub/ @anguillanneuf @plamut
+docs/pubsub @anguillanneuf @plamut
+
+
+# Tim Swast is the primary author of Runtime Config.
+runtimeconfig/ @tswast
+docs/runtimeconfig/ @tswast
+
+
+# Doug Greiman and Angela Li are the owners of trace.
+trace/ @liyanhui1228 @duggelz
+docs/trace/ @liyanhui1228 @duggelz
diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md
new file mode 100644
index 000000000000..939e5341e74d
--- /dev/null
+++ b/.github/CONTRIBUTING.md
@@ -0,0 +1,28 @@
+# How to Contribute
+
+We'd love to accept your patches and contributions to this project. There are
+just a few small guidelines you need to follow.
+
+## Contributor License Agreement
+
+Contributions to this project must be accompanied by a Contributor License
+Agreement. You (or your employer) retain the copyright to your contribution;
+this simply gives us permission to use and redistribute your contributions as
+part of the project. Head over to to see
+your current agreements on file or to sign a new one.
+
+You generally only need to submit a CLA once, so if you've already submitted one
+(even if it was for a different project), you probably don't need to do it
+again.
+
+## Code reviews
+
+All submissions, including submissions by project members, require review. We
+use GitHub pull requests for this purpose. Consult
+[GitHub Help](https://help.github.com/articles/about-pull-requests/) for more
+information on using pull requests.
+
+## Community Guidelines
+
+This project follows [Google's Open Source Community
+Guidelines](https://opensource.google.com/conduct/).
diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md
new file mode 100644
index 000000000000..1ca956493631
--- /dev/null
+++ b/.github/ISSUE_TEMPLATE/bug_report.md
@@ -0,0 +1,43 @@
+---
+name: Bug report
+about: Create a report to help us improve
+
+---
+
+Thanks for stopping by to let us know something could be better!
+
+**PLEASE READ**: If you have a support contract with Google, please create an issue in the [support console](https://cloud.google.com/support/) instead of filing on GitHub. This will ensure a timely response.
+
+Please run down the following list and make sure you've tried the usual "quick fixes":
+
+ - Search the issues already opened: https://github.com/googleapis/google-cloud-python/issues
+ - Check for answers on StackOverflow: http://stackoverflow.com/questions/tagged/google-cloud-python
+
+If you are still having issues, please be sure to include as much information as possible:
+
+#### Environment details
+
+1. Specify the API at the beginning of the title (for example, "BigQuery: ...")
+ General, Core, and Other are also allowed as types
+2. OS type and version
+3. Python version and virtual environment information: `python --version`
+4. google-cloud- version: `pip show google-` or `pip freeze`
+
+#### Steps to reproduce
+
+ 1. ?
+
+#### Code example
+
+```python
+# example
+```
+
+#### Stack trace
+```
+# example
+```
+
+Making sure to follow these steps will guarantee the quickest resolution possible.
+
+Thanks!
diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md
new file mode 100644
index 000000000000..6365857f33c6
--- /dev/null
+++ b/.github/ISSUE_TEMPLATE/feature_request.md
@@ -0,0 +1,18 @@
+---
+name: Feature request
+about: Suggest an idea for this library
+
+---
+
+Thanks for stopping by to let us know something could be better!
+
+**PLEASE READ**: If you have a support contract with Google, please create an issue in the [support console](https://cloud.google.com/support/) instead of filing on GitHub. This will ensure a timely response.
+
+ **Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+ **Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+ **Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+ **Additional context**
+Add any other context or screenshots about the feature request here.
diff --git a/.github/ISSUE_TEMPLATE/support_request.md b/.github/ISSUE_TEMPLATE/support_request.md
new file mode 100644
index 000000000000..995869032125
--- /dev/null
+++ b/.github/ISSUE_TEMPLATE/support_request.md
@@ -0,0 +1,7 @@
+---
+name: Support request
+about: If you have a support contract with Google, please create an issue in the Google Cloud Support console.
+
+---
+
+**PLEASE READ**: If you have a support contract with Google, please create an issue in the [support console](https://cloud.google.com/support/) instead of filing on GitHub. This will ensure a timely response.
diff --git a/.gitignore b/.gitignore
index e2eb2ef7a2e5..87a4d1890975 100644
--- a/.gitignore
+++ b/.gitignore
@@ -25,8 +25,11 @@ pip-log.txt
# Unit test / coverage reports
.coverage
+.nox
.tox
.cache
+.pytest_cache
+htmlcov
# Translations
*.mo
@@ -42,8 +45,14 @@ pip-log.txt
# JetBrains
.idea
+# VS Code
+.vscode
+
# Built documentation
docs/_build
+*/docs/_build
+docs/_build_doc2dash
+bigquery/docs/generated
# Virtual environment
env/
@@ -53,10 +62,12 @@ coverage.xml
system_tests/local_test_setup
# Make sure a generated file isn't accidentally committed.
-scripts/pylintrc_reduced
+pylintrc
+pylintrc.test
# Directories used for creating generated PB2 files
generated_python/
cloud-bigtable-client/
googleapis-pb/
grpc_python_venv/
+pytype_output/
diff --git a/.kokoro/build.sh b/.kokoro/build.sh
new file mode 100755
index 000000000000..39634b7d6d52
--- /dev/null
+++ b/.kokoro/build.sh
@@ -0,0 +1,56 @@
+#!/bin/bash
+
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -eo pipefail
+
+cd github/google-cloud-python
+
+# Disable buffering, so that the logs stream through.
+export PYTHONUNBUFFERED=1
+
+# Debug: show build environment
+env | grep KOKORO
+
+# Setup firestore account credentials
+export FIRESTORE_APPLICATION_CREDENTIALS=${KOKORO_GFILE_DIR}/firebase-credentials.json
+
+# Setup service account credentials.
+export GOOGLE_APPLICATION_CREDENTIALS=${KOKORO_GFILE_DIR}/service-account.json
+
+# Setup project id.
+export PROJECT_ID=$(cat "${KOKORO_GFILE_DIR}/project-id.json")
+
+# Find out if this package was modified.
+# Temporarily use Thea's fork of ci-diff-helper w/ Kokoro support.
+python3.6 -m pip install --quiet git+https://github.com/theacodes/ci-diff-helper.git
+python3.6 test_utils/scripts/get_target_packages_kokoro.py > ~/target_packages
+cat ~/target_packages
+
+if [[ ! -n $(grep -x "$PACKAGE" ~/target_packages) ]]; then
+ echo "$PACKAGE was not modified, returning."
+ exit;
+fi
+
+cd "$PACKAGE"
+
+# Remove old nox
+python3.6 -m pip uninstall --yes --quiet nox-automation
+
+# Install nox
+python3.6 -m pip install --upgrade --quiet nox
+python3.6 -m nox --version
+
+python3.6 -m nox
diff --git a/.kokoro/continuous/api_core.cfg b/.kokoro/continuous/api_core.cfg
new file mode 100644
index 000000000000..00ae35f46e2f
--- /dev/null
+++ b/.kokoro/continuous/api_core.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "api_core"
+}
diff --git a/.kokoro/continuous/asset.cfg b/.kokoro/continuous/asset.cfg
new file mode 100644
index 000000000000..61c3fa9c5b4b
--- /dev/null
+++ b/.kokoro/continuous/asset.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "asset"
+}
diff --git a/.kokoro/continuous/automl.cfg b/.kokoro/continuous/automl.cfg
new file mode 100644
index 000000000000..5081664abb56
--- /dev/null
+++ b/.kokoro/continuous/automl.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "automl"
+}
diff --git a/.kokoro/continuous/bigquery.cfg b/.kokoro/continuous/bigquery.cfg
new file mode 100644
index 000000000000..69eabb20803c
--- /dev/null
+++ b/.kokoro/continuous/bigquery.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "bigquery"
+}
diff --git a/.kokoro/continuous/bigquery_datatransfer.cfg b/.kokoro/continuous/bigquery_datatransfer.cfg
new file mode 100644
index 000000000000..47e510d21a44
--- /dev/null
+++ b/.kokoro/continuous/bigquery_datatransfer.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "bigquery_datatransfer"
+}
diff --git a/.kokoro/continuous/bigquery_storage.cfg b/.kokoro/continuous/bigquery_storage.cfg
new file mode 100644
index 000000000000..7f72bfed0587
--- /dev/null
+++ b/.kokoro/continuous/bigquery_storage.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "bigquery_storage"
+}
diff --git a/.kokoro/continuous/bigtable.cfg b/.kokoro/continuous/bigtable.cfg
new file mode 100644
index 000000000000..de67c4844de9
--- /dev/null
+++ b/.kokoro/continuous/bigtable.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "bigtable"
+}
diff --git a/.kokoro/continuous/common.cfg b/.kokoro/continuous/common.cfg
new file mode 100644
index 000000000000..5dec06191476
--- /dev/null
+++ b/.kokoro/continuous/common.cfg
@@ -0,0 +1,27 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Build logs will be here
+action {
+ define_artifacts {
+ regex: "**/*sponge_log.xml"
+ }
+}
+
+# Download trampoline resources.
+gfile_resources: "/bigstore/cloud-devrel-kokoro-resources/trampoline"
+
+# Download resources for system tests (service account key, etc.)
+gfile_resources: "/bigstore/cloud-devrel-kokoro-resources/google-cloud-python"
+
+# Use the trampoline script to run in docker.
+build_file: "google-cloud-python/.kokoro/trampoline.sh"
+
+# Configure the docker image for kokoro-trampoline.
+env_vars: {
+ key: "TRAMPOLINE_IMAGE"
+ value: "gcr.io/cloud-devrel-kokoro-resources/python-multi"
+}
+env_vars: {
+ key: "TRAMPOLINE_BUILD_FILE"
+ value: "github/google-cloud-python/.kokoro/build.sh"
+}
diff --git a/.kokoro/continuous/container.cfg b/.kokoro/continuous/container.cfg
new file mode 100644
index 000000000000..8d06b5285bf2
--- /dev/null
+++ b/.kokoro/continuous/container.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "container"
+}
diff --git a/.kokoro/continuous/containeranalysis.cfg b/.kokoro/continuous/containeranalysis.cfg
new file mode 100644
index 000000000000..ae80ed8d8bc6
--- /dev/null
+++ b/.kokoro/continuous/containeranalysis.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "containeranalysis"
+}
diff --git a/.kokoro/continuous/core.cfg b/.kokoro/continuous/core.cfg
new file mode 100644
index 000000000000..68b0d504429c
--- /dev/null
+++ b/.kokoro/continuous/core.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "core"
+}
diff --git a/.kokoro/continuous/datacatalog.cfg b/.kokoro/continuous/datacatalog.cfg
new file mode 100644
index 000000000000..fc0371e5eb11
--- /dev/null
+++ b/.kokoro/continuous/datacatalog.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "datacatalog"
+}
diff --git a/.kokoro/continuous/datalabeling.cfg b/.kokoro/continuous/datalabeling.cfg
new file mode 100644
index 000000000000..a1c53b81884b
--- /dev/null
+++ b/.kokoro/continuous/datalabeling.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "datalabeling"
+}
diff --git a/.kokoro/continuous/dataproc.cfg b/.kokoro/continuous/dataproc.cfg
new file mode 100644
index 000000000000..d9ea2e670aea
--- /dev/null
+++ b/.kokoro/continuous/dataproc.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "dataproc"
+}
diff --git a/.kokoro/continuous/datastore.cfg b/.kokoro/continuous/datastore.cfg
new file mode 100644
index 000000000000..7cf6a9c0dc8f
--- /dev/null
+++ b/.kokoro/continuous/datastore.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "datastore"
+}
diff --git a/.kokoro/continuous/dlp.cfg b/.kokoro/continuous/dlp.cfg
new file mode 100644
index 000000000000..931126d3e719
--- /dev/null
+++ b/.kokoro/continuous/dlp.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "dlp"
+}
diff --git a/.kokoro/continuous/dns.cfg b/.kokoro/continuous/dns.cfg
new file mode 100644
index 000000000000..845e3e148060
--- /dev/null
+++ b/.kokoro/continuous/dns.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "dns"
+}
diff --git a/.kokoro/continuous/error_reporting.cfg b/.kokoro/continuous/error_reporting.cfg
new file mode 100644
index 000000000000..99a86c522b9a
--- /dev/null
+++ b/.kokoro/continuous/error_reporting.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "error_reporting"
+}
diff --git a/.kokoro/continuous/firestore.cfg b/.kokoro/continuous/firestore.cfg
new file mode 100644
index 000000000000..9c5b0c53ab2d
--- /dev/null
+++ b/.kokoro/continuous/firestore.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "firestore"
+}
diff --git a/.kokoro/continuous/grafeas.cfg b/.kokoro/continuous/grafeas.cfg
new file mode 100644
index 000000000000..8f0e2f271518
--- /dev/null
+++ b/.kokoro/continuous/grafeas.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "grafeas"
+}
diff --git a/.kokoro/continuous/iam.cfg b/.kokoro/continuous/iam.cfg
new file mode 100644
index 000000000000..812d891c07f8
--- /dev/null
+++ b/.kokoro/continuous/iam.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "iam"
+}
diff --git a/.kokoro/continuous/iot.cfg b/.kokoro/continuous/iot.cfg
new file mode 100644
index 000000000000..dd388a8323e7
--- /dev/null
+++ b/.kokoro/continuous/iot.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "iot"
+}
diff --git a/.kokoro/continuous/irm.cfg b/.kokoro/continuous/irm.cfg
new file mode 100644
index 000000000000..ec847dd64e83
--- /dev/null
+++ b/.kokoro/continuous/irm.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "irm"
+}
diff --git a/.kokoro/continuous/kms.cfg b/.kokoro/continuous/kms.cfg
new file mode 100644
index 000000000000..ba634573085e
--- /dev/null
+++ b/.kokoro/continuous/kms.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "kms"
+}
diff --git a/.kokoro/continuous/language.cfg b/.kokoro/continuous/language.cfg
new file mode 100644
index 000000000000..c7ee4e650ca3
--- /dev/null
+++ b/.kokoro/continuous/language.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "language"
+}
diff --git a/.kokoro/continuous/logging.cfg b/.kokoro/continuous/logging.cfg
new file mode 100644
index 000000000000..cf274e2d626b
--- /dev/null
+++ b/.kokoro/continuous/logging.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "logging"
+}
diff --git a/.kokoro/continuous/monitoring.cfg b/.kokoro/continuous/monitoring.cfg
new file mode 100644
index 000000000000..80d261740e6f
--- /dev/null
+++ b/.kokoro/continuous/monitoring.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "monitoring"
+}
diff --git a/.kokoro/continuous/oslogin.cfg b/.kokoro/continuous/oslogin.cfg
new file mode 100644
index 000000000000..9ca9766f6e3a
--- /dev/null
+++ b/.kokoro/continuous/oslogin.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "oslogin"
+}
diff --git a/.kokoro/continuous/phishingprotection.cfg b/.kokoro/continuous/phishingprotection.cfg
new file mode 100644
index 000000000000..e21bd50ab4f1
--- /dev/null
+++ b/.kokoro/continuous/phishingprotection.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "phishingprotection"
+}
diff --git a/.kokoro/continuous/pubsub.cfg b/.kokoro/continuous/pubsub.cfg
new file mode 100644
index 000000000000..16c7ed3c175c
--- /dev/null
+++ b/.kokoro/continuous/pubsub.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "pubsub"
+}
diff --git a/.kokoro/continuous/redis.cfg b/.kokoro/continuous/redis.cfg
new file mode 100644
index 000000000000..268bc06a7101
--- /dev/null
+++ b/.kokoro/continuous/redis.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "redis"
+}
diff --git a/.kokoro/continuous/resource_manager.cfg b/.kokoro/continuous/resource_manager.cfg
new file mode 100644
index 000000000000..7a3976aef6cd
--- /dev/null
+++ b/.kokoro/continuous/resource_manager.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "resource_manager"
+}
diff --git a/.kokoro/continuous/runtimeconfig.cfg b/.kokoro/continuous/runtimeconfig.cfg
new file mode 100644
index 000000000000..2900b3c7e91a
--- /dev/null
+++ b/.kokoro/continuous/runtimeconfig.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "runtimeconfig"
+}
diff --git a/.kokoro/continuous/scheduler.cfg b/.kokoro/continuous/scheduler.cfg
new file mode 100644
index 000000000000..0701399853ef
--- /dev/null
+++ b/.kokoro/continuous/scheduler.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "scheduler"
+}
diff --git a/.kokoro/continuous/securitycenter.cfg b/.kokoro/continuous/securitycenter.cfg
new file mode 100644
index 000000000000..17dc427dae15
--- /dev/null
+++ b/.kokoro/continuous/securitycenter.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "securitycenter"
+}
diff --git a/.kokoro/continuous/spanner.cfg b/.kokoro/continuous/spanner.cfg
new file mode 100644
index 000000000000..d986f9d137f9
--- /dev/null
+++ b/.kokoro/continuous/spanner.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "spanner"
+}
diff --git a/.kokoro/continuous/speech.cfg b/.kokoro/continuous/speech.cfg
new file mode 100644
index 000000000000..b178b790be8d
--- /dev/null
+++ b/.kokoro/continuous/speech.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "speech"
+}
diff --git a/.kokoro/continuous/storage.cfg b/.kokoro/continuous/storage.cfg
new file mode 100644
index 000000000000..6bd9371ff99d
--- /dev/null
+++ b/.kokoro/continuous/storage.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "storage"
+}
diff --git a/.kokoro/continuous/talent.cfg b/.kokoro/continuous/talent.cfg
new file mode 100644
index 000000000000..abafead62769
--- /dev/null
+++ b/.kokoro/continuous/talent.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "talent"
+}
diff --git a/.kokoro/continuous/tasks.cfg b/.kokoro/continuous/tasks.cfg
new file mode 100644
index 000000000000..2ae5c581b88d
--- /dev/null
+++ b/.kokoro/continuous/tasks.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "tasks"
+}
diff --git a/.kokoro/continuous/texttospeech.cfg b/.kokoro/continuous/texttospeech.cfg
new file mode 100644
index 000000000000..9e7d6d7f7980
--- /dev/null
+++ b/.kokoro/continuous/texttospeech.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "texttospeech"
+}
diff --git a/.kokoro/continuous/trace.cfg b/.kokoro/continuous/trace.cfg
new file mode 100644
index 000000000000..9169296e4753
--- /dev/null
+++ b/.kokoro/continuous/trace.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "trace"
+}
diff --git a/.kokoro/continuous/translate.cfg b/.kokoro/continuous/translate.cfg
new file mode 100644
index 000000000000..807c96195b41
--- /dev/null
+++ b/.kokoro/continuous/translate.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "translate"
+}
diff --git a/.kokoro/continuous/videointelligence.cfg b/.kokoro/continuous/videointelligence.cfg
new file mode 100644
index 000000000000..77d1d3fc49e6
--- /dev/null
+++ b/.kokoro/continuous/videointelligence.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "videointelligence"
+}
diff --git a/.kokoro/continuous/vision.cfg b/.kokoro/continuous/vision.cfg
new file mode 100644
index 000000000000..4f686b3abf42
--- /dev/null
+++ b/.kokoro/continuous/vision.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "vision"
+}
diff --git a/.kokoro/continuous/webrisk.cfg b/.kokoro/continuous/webrisk.cfg
new file mode 100644
index 000000000000..36b06f0b0b55
--- /dev/null
+++ b/.kokoro/continuous/webrisk.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "webrisk"
+}
diff --git a/.kokoro/continuous/websecurityscanner.cfg b/.kokoro/continuous/websecurityscanner.cfg
new file mode 100644
index 000000000000..221d1fc23d6b
--- /dev/null
+++ b/.kokoro/continuous/websecurityscanner.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "websecurityscanner"
+}
diff --git a/.kokoro/docs/api_core.cfg b/.kokoro/docs/api_core.cfg
new file mode 100644
index 000000000000..00ae35f46e2f
--- /dev/null
+++ b/.kokoro/docs/api_core.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "api_core"
+}
diff --git a/.kokoro/docs/asset.cfg b/.kokoro/docs/asset.cfg
new file mode 100644
index 000000000000..61c3fa9c5b4b
--- /dev/null
+++ b/.kokoro/docs/asset.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "asset"
+}
diff --git a/.kokoro/docs/automl.cfg b/.kokoro/docs/automl.cfg
new file mode 100644
index 000000000000..093775c51065
--- /dev/null
+++ b/.kokoro/docs/automl.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "automl"
+}
\ No newline at end of file
diff --git a/.kokoro/docs/bigquery.cfg b/.kokoro/docs/bigquery.cfg
new file mode 100644
index 000000000000..69eabb20803c
--- /dev/null
+++ b/.kokoro/docs/bigquery.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "bigquery"
+}
diff --git a/.kokoro/docs/bigquery_datatransfer.cfg b/.kokoro/docs/bigquery_datatransfer.cfg
new file mode 100644
index 000000000000..47e510d21a44
--- /dev/null
+++ b/.kokoro/docs/bigquery_datatransfer.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "bigquery_datatransfer"
+}
diff --git a/.kokoro/docs/bigquery_storage.cfg b/.kokoro/docs/bigquery_storage.cfg
new file mode 100644
index 000000000000..7f72bfed0587
--- /dev/null
+++ b/.kokoro/docs/bigquery_storage.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "bigquery_storage"
+}
diff --git a/.kokoro/docs/bigtable.cfg b/.kokoro/docs/bigtable.cfg
new file mode 100644
index 000000000000..de67c4844de9
--- /dev/null
+++ b/.kokoro/docs/bigtable.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "bigtable"
+}
diff --git a/.kokoro/docs/common.cfg b/.kokoro/docs/common.cfg
new file mode 100644
index 000000000000..96e9740dd889
--- /dev/null
+++ b/.kokoro/docs/common.cfg
@@ -0,0 +1,48 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Build logs will be here
+action {
+ define_artifacts {
+ regex: "**/*sponge_log.xml"
+ }
+}
+
+# Download trampoline resources.
+gfile_resources: "/bigstore/cloud-devrel-kokoro-resources/trampoline"
+
+# Use the trampoline script to run in docker.
+build_file: "google-cloud-python/.kokoro/trampoline.sh"
+
+# Configure the docker image for kokoro-trampoline.
+env_vars: {
+ key: "TRAMPOLINE_IMAGE"
+ value: "gcr.io/cloud-devrel-kokoro-resources/python-multi"
+}
+env_vars: {
+ key: "TRAMPOLINE_BUILD_FILE"
+ value: "github/google-cloud-python/.kokoro/publish-docs.sh"
+}
+
+env_vars: {
+ key: "STAGING_BUCKET"
+ value: "docs-staging"
+}
+
+# Fetch the token needed for reporting release status to GitHub
+before_action {
+ fetch_keystore {
+ keystore_resource {
+ keystore_config_id: 73713
+ keyname: "yoshi-automation-github-key"
+ }
+ }
+}
+
+before_action {
+ fetch_keystore {
+ keystore_resource {
+ keystore_config_id: 73713
+ keyname: "docuploader_service_account"
+ }
+ }
+}
\ No newline at end of file
diff --git a/.kokoro/docs/container.cfg b/.kokoro/docs/container.cfg
new file mode 100644
index 000000000000..8d06b5285bf2
--- /dev/null
+++ b/.kokoro/docs/container.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "container"
+}
diff --git a/.kokoro/docs/containeranalysis.cfg b/.kokoro/docs/containeranalysis.cfg
new file mode 100644
index 000000000000..ae80ed8d8bc6
--- /dev/null
+++ b/.kokoro/docs/containeranalysis.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "containeranalysis"
+}
diff --git a/.kokoro/docs/core.cfg b/.kokoro/docs/core.cfg
new file mode 100644
index 000000000000..68b0d504429c
--- /dev/null
+++ b/.kokoro/docs/core.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "core"
+}
diff --git a/.kokoro/docs/datacatalog.cfg b/.kokoro/docs/datacatalog.cfg
new file mode 100644
index 000000000000..fc0371e5eb11
--- /dev/null
+++ b/.kokoro/docs/datacatalog.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "datacatalog"
+}
diff --git a/.kokoro/docs/datalabeling.cfg b/.kokoro/docs/datalabeling.cfg
new file mode 100644
index 000000000000..a1c53b81884b
--- /dev/null
+++ b/.kokoro/docs/datalabeling.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "datalabeling"
+}
diff --git a/.kokoro/docs/dataproc.cfg b/.kokoro/docs/dataproc.cfg
new file mode 100644
index 000000000000..d9ea2e670aea
--- /dev/null
+++ b/.kokoro/docs/dataproc.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "dataproc"
+}
diff --git a/.kokoro/docs/datastore.cfg b/.kokoro/docs/datastore.cfg
new file mode 100644
index 000000000000..7cf6a9c0dc8f
--- /dev/null
+++ b/.kokoro/docs/datastore.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "datastore"
+}
diff --git a/.kokoro/docs/dlp.cfg b/.kokoro/docs/dlp.cfg
new file mode 100644
index 000000000000..931126d3e719
--- /dev/null
+++ b/.kokoro/docs/dlp.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "dlp"
+}
diff --git a/.kokoro/docs/dns.cfg b/.kokoro/docs/dns.cfg
new file mode 100644
index 000000000000..845e3e148060
--- /dev/null
+++ b/.kokoro/docs/dns.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "dns"
+}
diff --git a/.kokoro/docs/error_reporting.cfg b/.kokoro/docs/error_reporting.cfg
new file mode 100644
index 000000000000..99a86c522b9a
--- /dev/null
+++ b/.kokoro/docs/error_reporting.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "error_reporting"
+}
diff --git a/.kokoro/docs/firestore.cfg b/.kokoro/docs/firestore.cfg
new file mode 100644
index 000000000000..9c5b0c53ab2d
--- /dev/null
+++ b/.kokoro/docs/firestore.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "firestore"
+}
diff --git a/.kokoro/docs/grafeas.cfg b/.kokoro/docs/grafeas.cfg
new file mode 100644
index 000000000000..8f0e2f271518
--- /dev/null
+++ b/.kokoro/docs/grafeas.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "grafeas"
+}
diff --git a/.kokoro/docs/iam.cfg b/.kokoro/docs/iam.cfg
new file mode 100644
index 000000000000..812d891c07f8
--- /dev/null
+++ b/.kokoro/docs/iam.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "iam"
+}
diff --git a/.kokoro/docs/iot.cfg b/.kokoro/docs/iot.cfg
new file mode 100644
index 000000000000..114c8da20d77
--- /dev/null
+++ b/.kokoro/docs/iot.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "iot"
+}
\ No newline at end of file
diff --git a/.kokoro/docs/irm.cfg b/.kokoro/docs/irm.cfg
new file mode 100644
index 000000000000..ec847dd64e83
--- /dev/null
+++ b/.kokoro/docs/irm.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "irm"
+}
diff --git a/.kokoro/docs/kms.cfg b/.kokoro/docs/kms.cfg
new file mode 100644
index 000000000000..ba634573085e
--- /dev/null
+++ b/.kokoro/docs/kms.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "kms"
+}
diff --git a/.kokoro/docs/language.cfg b/.kokoro/docs/language.cfg
new file mode 100644
index 000000000000..c7ee4e650ca3
--- /dev/null
+++ b/.kokoro/docs/language.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "language"
+}
diff --git a/.kokoro/docs/logging.cfg b/.kokoro/docs/logging.cfg
new file mode 100644
index 000000000000..cf274e2d626b
--- /dev/null
+++ b/.kokoro/docs/logging.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "logging"
+}
diff --git a/.kokoro/docs/monitoring.cfg b/.kokoro/docs/monitoring.cfg
new file mode 100644
index 000000000000..80d261740e6f
--- /dev/null
+++ b/.kokoro/docs/monitoring.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "monitoring"
+}
diff --git a/.kokoro/docs/oslogin.cfg b/.kokoro/docs/oslogin.cfg
new file mode 100644
index 000000000000..9ca9766f6e3a
--- /dev/null
+++ b/.kokoro/docs/oslogin.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "oslogin"
+}
diff --git a/.kokoro/docs/phishingprotection.cfg b/.kokoro/docs/phishingprotection.cfg
new file mode 100644
index 000000000000..e21bd50ab4f1
--- /dev/null
+++ b/.kokoro/docs/phishingprotection.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "phishingprotection"
+}
diff --git a/.kokoro/docs/pubsub.cfg b/.kokoro/docs/pubsub.cfg
new file mode 100644
index 000000000000..16c7ed3c175c
--- /dev/null
+++ b/.kokoro/docs/pubsub.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "pubsub"
+}
diff --git a/.kokoro/docs/redis.cfg b/.kokoro/docs/redis.cfg
new file mode 100644
index 000000000000..268bc06a7101
--- /dev/null
+++ b/.kokoro/docs/redis.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "redis"
+}
diff --git a/.kokoro/docs/resource_manager.cfg b/.kokoro/docs/resource_manager.cfg
new file mode 100644
index 000000000000..7a3976aef6cd
--- /dev/null
+++ b/.kokoro/docs/resource_manager.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "resource_manager"
+}
diff --git a/.kokoro/docs/runtimeconfig.cfg b/.kokoro/docs/runtimeconfig.cfg
new file mode 100644
index 000000000000..2900b3c7e91a
--- /dev/null
+++ b/.kokoro/docs/runtimeconfig.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "runtimeconfig"
+}
diff --git a/.kokoro/docs/scheduler.cfg b/.kokoro/docs/scheduler.cfg
new file mode 100644
index 000000000000..0701399853ef
--- /dev/null
+++ b/.kokoro/docs/scheduler.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "scheduler"
+}
diff --git a/.kokoro/docs/securitycenter.cfg b/.kokoro/docs/securitycenter.cfg
new file mode 100644
index 000000000000..17dc427dae15
--- /dev/null
+++ b/.kokoro/docs/securitycenter.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "securitycenter"
+}
diff --git a/.kokoro/docs/spanner.cfg b/.kokoro/docs/spanner.cfg
new file mode 100644
index 000000000000..d986f9d137f9
--- /dev/null
+++ b/.kokoro/docs/spanner.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "spanner"
+}
diff --git a/.kokoro/docs/speech.cfg b/.kokoro/docs/speech.cfg
new file mode 100644
index 000000000000..b178b790be8d
--- /dev/null
+++ b/.kokoro/docs/speech.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "speech"
+}
diff --git a/.kokoro/docs/storage.cfg b/.kokoro/docs/storage.cfg
new file mode 100644
index 000000000000..6bd9371ff99d
--- /dev/null
+++ b/.kokoro/docs/storage.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "storage"
+}
diff --git a/.kokoro/docs/talent.cfg b/.kokoro/docs/talent.cfg
new file mode 100644
index 000000000000..abafead62769
--- /dev/null
+++ b/.kokoro/docs/talent.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "talent"
+}
diff --git a/.kokoro/docs/tasks.cfg b/.kokoro/docs/tasks.cfg
new file mode 100644
index 000000000000..2ae5c581b88d
--- /dev/null
+++ b/.kokoro/docs/tasks.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "tasks"
+}
diff --git a/.kokoro/docs/texttospeech.cfg b/.kokoro/docs/texttospeech.cfg
new file mode 100644
index 000000000000..9e7d6d7f7980
--- /dev/null
+++ b/.kokoro/docs/texttospeech.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "texttospeech"
+}
diff --git a/.kokoro/docs/trace.cfg b/.kokoro/docs/trace.cfg
new file mode 100644
index 000000000000..9169296e4753
--- /dev/null
+++ b/.kokoro/docs/trace.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "trace"
+}
diff --git a/.kokoro/docs/translate.cfg b/.kokoro/docs/translate.cfg
new file mode 100644
index 000000000000..807c96195b41
--- /dev/null
+++ b/.kokoro/docs/translate.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "translate"
+}
diff --git a/.kokoro/docs/videointelligence.cfg b/.kokoro/docs/videointelligence.cfg
new file mode 100644
index 000000000000..77d1d3fc49e6
--- /dev/null
+++ b/.kokoro/docs/videointelligence.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "videointelligence"
+}
diff --git a/.kokoro/docs/vision.cfg b/.kokoro/docs/vision.cfg
new file mode 100644
index 000000000000..4f686b3abf42
--- /dev/null
+++ b/.kokoro/docs/vision.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "vision"
+}
diff --git a/.kokoro/docs/webrisk.cfg b/.kokoro/docs/webrisk.cfg
new file mode 100644
index 000000000000..36b06f0b0b55
--- /dev/null
+++ b/.kokoro/docs/webrisk.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "webrisk"
+}
diff --git a/.kokoro/docs/websecurityscanner.cfg b/.kokoro/docs/websecurityscanner.cfg
new file mode 100644
index 000000000000..221d1fc23d6b
--- /dev/null
+++ b/.kokoro/docs/websecurityscanner.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "websecurityscanner"
+}
diff --git a/.kokoro/presubmit/api_core.cfg b/.kokoro/presubmit/api_core.cfg
new file mode 100644
index 000000000000..00ae35f46e2f
--- /dev/null
+++ b/.kokoro/presubmit/api_core.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "api_core"
+}
diff --git a/.kokoro/presubmit/asset.cfg b/.kokoro/presubmit/asset.cfg
new file mode 100644
index 000000000000..61c3fa9c5b4b
--- /dev/null
+++ b/.kokoro/presubmit/asset.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "asset"
+}
diff --git a/.kokoro/presubmit/automl.cfg b/.kokoro/presubmit/automl.cfg
new file mode 100644
index 000000000000..5081664abb56
--- /dev/null
+++ b/.kokoro/presubmit/automl.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "automl"
+}
diff --git a/.kokoro/presubmit/bigquery.cfg b/.kokoro/presubmit/bigquery.cfg
new file mode 100644
index 000000000000..69eabb20803c
--- /dev/null
+++ b/.kokoro/presubmit/bigquery.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "bigquery"
+}
diff --git a/.kokoro/presubmit/bigquery_datatransfer.cfg b/.kokoro/presubmit/bigquery_datatransfer.cfg
new file mode 100644
index 000000000000..47e510d21a44
--- /dev/null
+++ b/.kokoro/presubmit/bigquery_datatransfer.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "bigquery_datatransfer"
+}
diff --git a/.kokoro/presubmit/bigquery_storage.cfg b/.kokoro/presubmit/bigquery_storage.cfg
new file mode 100644
index 000000000000..7f72bfed0587
--- /dev/null
+++ b/.kokoro/presubmit/bigquery_storage.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "bigquery_storage"
+}
diff --git a/.kokoro/presubmit/bigtable.cfg b/.kokoro/presubmit/bigtable.cfg
new file mode 100644
index 000000000000..de67c4844de9
--- /dev/null
+++ b/.kokoro/presubmit/bigtable.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "bigtable"
+}
diff --git a/.kokoro/presubmit/common.cfg b/.kokoro/presubmit/common.cfg
new file mode 100644
index 000000000000..5dec06191476
--- /dev/null
+++ b/.kokoro/presubmit/common.cfg
@@ -0,0 +1,27 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Build logs will be here
+action {
+ define_artifacts {
+ regex: "**/*sponge_log.xml"
+ }
+}
+
+# Download trampoline resources.
+gfile_resources: "/bigstore/cloud-devrel-kokoro-resources/trampoline"
+
+# Download resources for system tests (service account key, etc.)
+gfile_resources: "/bigstore/cloud-devrel-kokoro-resources/google-cloud-python"
+
+# Use the trampoline script to run in docker.
+build_file: "google-cloud-python/.kokoro/trampoline.sh"
+
+# Configure the docker image for kokoro-trampoline.
+env_vars: {
+ key: "TRAMPOLINE_IMAGE"
+ value: "gcr.io/cloud-devrel-kokoro-resources/python-multi"
+}
+env_vars: {
+ key: "TRAMPOLINE_BUILD_FILE"
+ value: "github/google-cloud-python/.kokoro/build.sh"
+}
diff --git a/.kokoro/presubmit/container.cfg b/.kokoro/presubmit/container.cfg
new file mode 100644
index 000000000000..8d06b5285bf2
--- /dev/null
+++ b/.kokoro/presubmit/container.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "container"
+}
diff --git a/.kokoro/presubmit/containeranalysis.cfg b/.kokoro/presubmit/containeranalysis.cfg
new file mode 100644
index 000000000000..ae80ed8d8bc6
--- /dev/null
+++ b/.kokoro/presubmit/containeranalysis.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "containeranalysis"
+}
diff --git a/.kokoro/presubmit/core.cfg b/.kokoro/presubmit/core.cfg
new file mode 100644
index 000000000000..68b0d504429c
--- /dev/null
+++ b/.kokoro/presubmit/core.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "core"
+}
diff --git a/.kokoro/presubmit/datacatalog.cfg b/.kokoro/presubmit/datacatalog.cfg
new file mode 100644
index 000000000000..fc0371e5eb11
--- /dev/null
+++ b/.kokoro/presubmit/datacatalog.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "datacatalog"
+}
diff --git a/.kokoro/presubmit/datalabeling.cfg b/.kokoro/presubmit/datalabeling.cfg
new file mode 100644
index 000000000000..a1c53b81884b
--- /dev/null
+++ b/.kokoro/presubmit/datalabeling.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "datalabeling"
+}
diff --git a/.kokoro/presubmit/dataproc.cfg b/.kokoro/presubmit/dataproc.cfg
new file mode 100644
index 000000000000..d9ea2e670aea
--- /dev/null
+++ b/.kokoro/presubmit/dataproc.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "dataproc"
+}
diff --git a/.kokoro/presubmit/datastore.cfg b/.kokoro/presubmit/datastore.cfg
new file mode 100644
index 000000000000..7cf6a9c0dc8f
--- /dev/null
+++ b/.kokoro/presubmit/datastore.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "datastore"
+}
diff --git a/.kokoro/presubmit/dlp.cfg b/.kokoro/presubmit/dlp.cfg
new file mode 100644
index 000000000000..931126d3e719
--- /dev/null
+++ b/.kokoro/presubmit/dlp.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "dlp"
+}
diff --git a/.kokoro/presubmit/dns.cfg b/.kokoro/presubmit/dns.cfg
new file mode 100644
index 000000000000..845e3e148060
--- /dev/null
+++ b/.kokoro/presubmit/dns.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "dns"
+}
diff --git a/.kokoro/presubmit/error_reporting.cfg b/.kokoro/presubmit/error_reporting.cfg
new file mode 100644
index 000000000000..99a86c522b9a
--- /dev/null
+++ b/.kokoro/presubmit/error_reporting.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "error_reporting"
+}
diff --git a/.kokoro/presubmit/firestore.cfg b/.kokoro/presubmit/firestore.cfg
new file mode 100644
index 000000000000..9c5b0c53ab2d
--- /dev/null
+++ b/.kokoro/presubmit/firestore.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "firestore"
+}
diff --git a/.kokoro/presubmit/grafeas.cfg b/.kokoro/presubmit/grafeas.cfg
new file mode 100644
index 000000000000..8f0e2f271518
--- /dev/null
+++ b/.kokoro/presubmit/grafeas.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "grafeas"
+}
diff --git a/.kokoro/presubmit/iam.cfg b/.kokoro/presubmit/iam.cfg
new file mode 100644
index 000000000000..812d891c07f8
--- /dev/null
+++ b/.kokoro/presubmit/iam.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "iam"
+}
diff --git a/.kokoro/presubmit/iot.cfg b/.kokoro/presubmit/iot.cfg
new file mode 100644
index 000000000000..dd388a8323e7
--- /dev/null
+++ b/.kokoro/presubmit/iot.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "iot"
+}
diff --git a/.kokoro/presubmit/irm.cfg b/.kokoro/presubmit/irm.cfg
new file mode 100644
index 000000000000..ec847dd64e83
--- /dev/null
+++ b/.kokoro/presubmit/irm.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "irm"
+}
diff --git a/.kokoro/presubmit/kms.cfg b/.kokoro/presubmit/kms.cfg
new file mode 100644
index 000000000000..ba634573085e
--- /dev/null
+++ b/.kokoro/presubmit/kms.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "kms"
+}
diff --git a/.kokoro/presubmit/language.cfg b/.kokoro/presubmit/language.cfg
new file mode 100644
index 000000000000..c7ee4e650ca3
--- /dev/null
+++ b/.kokoro/presubmit/language.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "language"
+}
diff --git a/.kokoro/presubmit/logging.cfg b/.kokoro/presubmit/logging.cfg
new file mode 100644
index 000000000000..cf274e2d626b
--- /dev/null
+++ b/.kokoro/presubmit/logging.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "logging"
+}
diff --git a/.kokoro/presubmit/monitoring.cfg b/.kokoro/presubmit/monitoring.cfg
new file mode 100644
index 000000000000..80d261740e6f
--- /dev/null
+++ b/.kokoro/presubmit/monitoring.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "monitoring"
+}
diff --git a/.kokoro/presubmit/oslogin.cfg b/.kokoro/presubmit/oslogin.cfg
new file mode 100644
index 000000000000..9ca9766f6e3a
--- /dev/null
+++ b/.kokoro/presubmit/oslogin.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "oslogin"
+}
diff --git a/.kokoro/presubmit/phishingprotection.cfg b/.kokoro/presubmit/phishingprotection.cfg
new file mode 100644
index 000000000000..e21bd50ab4f1
--- /dev/null
+++ b/.kokoro/presubmit/phishingprotection.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "phishingprotection"
+}
diff --git a/.kokoro/presubmit/pubsub.cfg b/.kokoro/presubmit/pubsub.cfg
new file mode 100644
index 000000000000..16c7ed3c175c
--- /dev/null
+++ b/.kokoro/presubmit/pubsub.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "pubsub"
+}
diff --git a/.kokoro/presubmit/redis.cfg b/.kokoro/presubmit/redis.cfg
new file mode 100644
index 000000000000..268bc06a7101
--- /dev/null
+++ b/.kokoro/presubmit/redis.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "redis"
+}
diff --git a/.kokoro/presubmit/resource_manager.cfg b/.kokoro/presubmit/resource_manager.cfg
new file mode 100644
index 000000000000..7a3976aef6cd
--- /dev/null
+++ b/.kokoro/presubmit/resource_manager.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "resource_manager"
+}
diff --git a/.kokoro/presubmit/runtimeconfig.cfg b/.kokoro/presubmit/runtimeconfig.cfg
new file mode 100644
index 000000000000..2900b3c7e91a
--- /dev/null
+++ b/.kokoro/presubmit/runtimeconfig.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "runtimeconfig"
+}
diff --git a/.kokoro/presubmit/scheduler.cfg b/.kokoro/presubmit/scheduler.cfg
new file mode 100644
index 000000000000..0701399853ef
--- /dev/null
+++ b/.kokoro/presubmit/scheduler.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "scheduler"
+}
diff --git a/.kokoro/presubmit/securitycenter.cfg b/.kokoro/presubmit/securitycenter.cfg
new file mode 100644
index 000000000000..17dc427dae15
--- /dev/null
+++ b/.kokoro/presubmit/securitycenter.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "securitycenter"
+}
diff --git a/.kokoro/presubmit/spanner.cfg b/.kokoro/presubmit/spanner.cfg
new file mode 100644
index 000000000000..d986f9d137f9
--- /dev/null
+++ b/.kokoro/presubmit/spanner.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "spanner"
+}
diff --git a/.kokoro/presubmit/speech.cfg b/.kokoro/presubmit/speech.cfg
new file mode 100644
index 000000000000..b178b790be8d
--- /dev/null
+++ b/.kokoro/presubmit/speech.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "speech"
+}
diff --git a/.kokoro/presubmit/storage.cfg b/.kokoro/presubmit/storage.cfg
new file mode 100644
index 000000000000..6bd9371ff99d
--- /dev/null
+++ b/.kokoro/presubmit/storage.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "storage"
+}
diff --git a/.kokoro/presubmit/talent.cfg b/.kokoro/presubmit/talent.cfg
new file mode 100644
index 000000000000..abafead62769
--- /dev/null
+++ b/.kokoro/presubmit/talent.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "talent"
+}
diff --git a/.kokoro/presubmit/tasks.cfg b/.kokoro/presubmit/tasks.cfg
new file mode 100644
index 000000000000..2ae5c581b88d
--- /dev/null
+++ b/.kokoro/presubmit/tasks.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "tasks"
+}
diff --git a/.kokoro/presubmit/texttospeech.cfg b/.kokoro/presubmit/texttospeech.cfg
new file mode 100644
index 000000000000..9e7d6d7f7980
--- /dev/null
+++ b/.kokoro/presubmit/texttospeech.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "texttospeech"
+}
diff --git a/.kokoro/presubmit/trace.cfg b/.kokoro/presubmit/trace.cfg
new file mode 100644
index 000000000000..9169296e4753
--- /dev/null
+++ b/.kokoro/presubmit/trace.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "trace"
+}
diff --git a/.kokoro/presubmit/translate.cfg b/.kokoro/presubmit/translate.cfg
new file mode 100644
index 000000000000..807c96195b41
--- /dev/null
+++ b/.kokoro/presubmit/translate.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "translate"
+}
diff --git a/.kokoro/presubmit/videointelligence.cfg b/.kokoro/presubmit/videointelligence.cfg
new file mode 100644
index 000000000000..77d1d3fc49e6
--- /dev/null
+++ b/.kokoro/presubmit/videointelligence.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "videointelligence"
+}
diff --git a/.kokoro/presubmit/vision.cfg b/.kokoro/presubmit/vision.cfg
new file mode 100644
index 000000000000..4f686b3abf42
--- /dev/null
+++ b/.kokoro/presubmit/vision.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "vision"
+}
diff --git a/.kokoro/presubmit/webrisk.cfg b/.kokoro/presubmit/webrisk.cfg
new file mode 100644
index 000000000000..36b06f0b0b55
--- /dev/null
+++ b/.kokoro/presubmit/webrisk.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "webrisk"
+}
diff --git a/.kokoro/presubmit/websecurityscanner.cfg b/.kokoro/presubmit/websecurityscanner.cfg
new file mode 100644
index 000000000000..221d1fc23d6b
--- /dev/null
+++ b/.kokoro/presubmit/websecurityscanner.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "websecurityscanner"
+}
diff --git a/.kokoro/publish-docs.sh b/.kokoro/publish-docs.sh
new file mode 100755
index 000000000000..d6762b5ed6f9
--- /dev/null
+++ b/.kokoro/publish-docs.sh
@@ -0,0 +1,42 @@
+#!/bin/bash
+
+set -eo pipefail
+
+# Disable buffering, so that the logs stream through.
+export PYTHONUNBUFFERED=1
+
+cd github/google-cloud-python/${PACKAGE}
+
+# Remove old nox
+python3.6 -m pip uninstall --yes --quiet nox-automation
+
+# Install nox
+python3.6 -m pip install --upgrade --quiet nox
+python3.6 -m nox --version
+
+# build docs
+nox -s docs
+
+python3 -m pip install gcp-docuploader
+
+# install a json parser
+sudo apt-get update
+sudo apt-get -y install software-properties-common
+sudo add-apt-repository universe
+sudo apt-get update
+sudo apt-get -y install jq
+
+# create metadata
+python3 -m docuploader create-metadata \
+ --name=$(jq --raw-output '.name // empty' .repo-metadata.json) \
+ --version=$(python3 setup.py --version) \
+ --language=$(jq --raw-output '.language // empty' .repo-metadata.json) \
+ --distribution-name=$(python3 setup.py --name) \
+ --product-page=$(jq --raw-output '.product_documentation // empty' .repo-metadata.json) \
+ --github-repository=$(jq --raw-output '.repo // empty' .repo-metadata.json) \
+ --issue-tracker=$(jq --raw-output '.issue_tracker // empty' .repo-metadata.json)
+
+cat docs.metadata
+
+# upload docs
+python3 -m docuploader upload docs/_build/html --metadata-file docs.metadata --staging-bucket docs-staging
\ No newline at end of file
diff --git a/.kokoro/release.sh b/.kokoro/release.sh
new file mode 100755
index 000000000000..6909ae880d8b
--- /dev/null
+++ b/.kokoro/release.sh
@@ -0,0 +1,19 @@
+#!/bin/bash
+
+set -eo pipefail
+
+# Start the releasetool reporter
+python3 -m pip install gcp-releasetool
+python3 -m releasetool publish-reporter-script > /tmp/publisher-script; source /tmp/publisher-script
+
+# Ensure that we have the latest versions of Twine, Wheel, and Setuptools.
+python3 -m pip install --upgrade twine wheel setuptools
+
+# Disable buffering, so that the logs stream through.
+export PYTHONUNBUFFERED=1
+
+# Move into the package, build the distribution and upload.
+TWINE_PASSWORD=$(cat "${KOKORO_KEYSTORE_DIR}/73713_google_cloud_pypi_password")
+cd github/google-cloud-python/${PACKAGE}
+python3 setup.py sdist bdist_wheel
+twine upload --username gcloudpypi --password "${TWINE_PASSWORD}" dist/*
diff --git a/.kokoro/release/api_core.cfg b/.kokoro/release/api_core.cfg
new file mode 100644
index 000000000000..00ae35f46e2f
--- /dev/null
+++ b/.kokoro/release/api_core.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "api_core"
+}
diff --git a/.kokoro/release/asset.cfg b/.kokoro/release/asset.cfg
new file mode 100644
index 000000000000..61c3fa9c5b4b
--- /dev/null
+++ b/.kokoro/release/asset.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "asset"
+}
diff --git a/.kokoro/release/automl.cfg b/.kokoro/release/automl.cfg
new file mode 100644
index 000000000000..5081664abb56
--- /dev/null
+++ b/.kokoro/release/automl.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "automl"
+}
diff --git a/.kokoro/release/bigquery.cfg b/.kokoro/release/bigquery.cfg
new file mode 100644
index 000000000000..69eabb20803c
--- /dev/null
+++ b/.kokoro/release/bigquery.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "bigquery"
+}
diff --git a/.kokoro/release/bigquery_datatransfer.cfg b/.kokoro/release/bigquery_datatransfer.cfg
new file mode 100644
index 000000000000..47e510d21a44
--- /dev/null
+++ b/.kokoro/release/bigquery_datatransfer.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "bigquery_datatransfer"
+}
diff --git a/.kokoro/release/bigquery_storage.cfg b/.kokoro/release/bigquery_storage.cfg
new file mode 100644
index 000000000000..7f72bfed0587
--- /dev/null
+++ b/.kokoro/release/bigquery_storage.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "bigquery_storage"
+}
diff --git a/.kokoro/release/bigtable.cfg b/.kokoro/release/bigtable.cfg
new file mode 100644
index 000000000000..de67c4844de9
--- /dev/null
+++ b/.kokoro/release/bigtable.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "bigtable"
+}
diff --git a/.kokoro/release/common.cfg b/.kokoro/release/common.cfg
new file mode 100644
index 000000000000..7456ab5604ef
--- /dev/null
+++ b/.kokoro/release/common.cfg
@@ -0,0 +1,64 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Build logs will be here
+action {
+ define_artifacts {
+ regex: "**/*sponge_log.xml"
+ }
+}
+
+# Download trampoline resources.
+gfile_resources: "/bigstore/cloud-devrel-kokoro-resources/trampoline"
+
+# Use the trampoline script to run in docker.
+build_file: "google-cloud-python/.kokoro/trampoline.sh"
+
+# Configure the docker image for kokoro-trampoline.
+env_vars: {
+ key: "TRAMPOLINE_IMAGE"
+ value: "gcr.io/cloud-devrel-kokoro-resources/python-multi"
+}
+env_vars: {
+ key: "TRAMPOLINE_BUILD_FILE"
+ value: "github/google-cloud-python/.kokoro/release.sh"
+}
+
+# Fetch the token needed for reporting release status to GitHub
+before_action {
+ fetch_keystore {
+ keystore_resource {
+ keystore_config_id: 73713
+ keyname: "yoshi-automation-github-key"
+ }
+ }
+}
+
+# Fetch PyPI password
+before_action {
+ fetch_keystore {
+ keystore_resource {
+ keystore_config_id: 73713
+ keyname: "google_cloud_pypi_password"
+ }
+ }
+}
+
+# Fetch magictoken to use with Magic Github Proxy
+before_action {
+ fetch_keystore {
+ keystore_resource {
+ keystore_config_id: 73713
+ keyname: "releasetool-magictoken"
+ }
+ }
+}
+
+# Fetch api key to use with Magic Github Proxy
+before_action {
+ fetch_keystore {
+ keystore_resource {
+ keystore_config_id: 73713
+ keyname: "magic-github-proxy-api-key"
+ }
+ }
+}
diff --git a/.kokoro/release/container.cfg b/.kokoro/release/container.cfg
new file mode 100644
index 000000000000..8d06b5285bf2
--- /dev/null
+++ b/.kokoro/release/container.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "container"
+}
diff --git a/.kokoro/release/containeranalysis.cfg b/.kokoro/release/containeranalysis.cfg
new file mode 100644
index 000000000000..ae80ed8d8bc6
--- /dev/null
+++ b/.kokoro/release/containeranalysis.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "containeranalysis"
+}
diff --git a/.kokoro/release/core.cfg b/.kokoro/release/core.cfg
new file mode 100644
index 000000000000..68b0d504429c
--- /dev/null
+++ b/.kokoro/release/core.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "core"
+}
diff --git a/.kokoro/release/datacatalog.cfg b/.kokoro/release/datacatalog.cfg
new file mode 100644
index 000000000000..fc0371e5eb11
--- /dev/null
+++ b/.kokoro/release/datacatalog.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "datacatalog"
+}
diff --git a/.kokoro/release/datalabeling.cfg b/.kokoro/release/datalabeling.cfg
new file mode 100644
index 000000000000..a1c53b81884b
--- /dev/null
+++ b/.kokoro/release/datalabeling.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "datalabeling"
+}
diff --git a/.kokoro/release/dataproc.cfg b/.kokoro/release/dataproc.cfg
new file mode 100644
index 000000000000..d9ea2e670aea
--- /dev/null
+++ b/.kokoro/release/dataproc.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "dataproc"
+}
diff --git a/.kokoro/release/datastore.cfg b/.kokoro/release/datastore.cfg
new file mode 100644
index 000000000000..7cf6a9c0dc8f
--- /dev/null
+++ b/.kokoro/release/datastore.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "datastore"
+}
diff --git a/.kokoro/release/dlp.cfg b/.kokoro/release/dlp.cfg
new file mode 100644
index 000000000000..931126d3e719
--- /dev/null
+++ b/.kokoro/release/dlp.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "dlp"
+}
diff --git a/.kokoro/release/dns.cfg b/.kokoro/release/dns.cfg
new file mode 100644
index 000000000000..845e3e148060
--- /dev/null
+++ b/.kokoro/release/dns.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "dns"
+}
diff --git a/.kokoro/release/error_reporting.cfg b/.kokoro/release/error_reporting.cfg
new file mode 100644
index 000000000000..99a86c522b9a
--- /dev/null
+++ b/.kokoro/release/error_reporting.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "error_reporting"
+}
diff --git a/.kokoro/release/firestore.cfg b/.kokoro/release/firestore.cfg
new file mode 100644
index 000000000000..9c5b0c53ab2d
--- /dev/null
+++ b/.kokoro/release/firestore.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "firestore"
+}
diff --git a/.kokoro/release/grafeas.cfg b/.kokoro/release/grafeas.cfg
new file mode 100644
index 000000000000..8f0e2f271518
--- /dev/null
+++ b/.kokoro/release/grafeas.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "grafeas"
+}
diff --git a/.kokoro/release/iam.cfg b/.kokoro/release/iam.cfg
new file mode 100644
index 000000000000..812d891c07f8
--- /dev/null
+++ b/.kokoro/release/iam.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "iam"
+}
diff --git a/.kokoro/release/iot.cfg b/.kokoro/release/iot.cfg
new file mode 100644
index 000000000000..dd388a8323e7
--- /dev/null
+++ b/.kokoro/release/iot.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "iot"
+}
diff --git a/.kokoro/release/irm.cfg b/.kokoro/release/irm.cfg
new file mode 100644
index 000000000000..ec847dd64e83
--- /dev/null
+++ b/.kokoro/release/irm.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "irm"
+}
diff --git a/.kokoro/release/kms.cfg b/.kokoro/release/kms.cfg
new file mode 100644
index 000000000000..ba634573085e
--- /dev/null
+++ b/.kokoro/release/kms.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "kms"
+}
diff --git a/.kokoro/release/language.cfg b/.kokoro/release/language.cfg
new file mode 100644
index 000000000000..c7ee4e650ca3
--- /dev/null
+++ b/.kokoro/release/language.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "language"
+}
diff --git a/.kokoro/release/logging.cfg b/.kokoro/release/logging.cfg
new file mode 100644
index 000000000000..cf274e2d626b
--- /dev/null
+++ b/.kokoro/release/logging.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "logging"
+}
diff --git a/.kokoro/release/monitoring.cfg b/.kokoro/release/monitoring.cfg
new file mode 100644
index 000000000000..80d261740e6f
--- /dev/null
+++ b/.kokoro/release/monitoring.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "monitoring"
+}
diff --git a/.kokoro/release/oslogin.cfg b/.kokoro/release/oslogin.cfg
new file mode 100644
index 000000000000..9ca9766f6e3a
--- /dev/null
+++ b/.kokoro/release/oslogin.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "oslogin"
+}
diff --git a/.kokoro/release/phishingprotection.cfg b/.kokoro/release/phishingprotection.cfg
new file mode 100644
index 000000000000..e21bd50ab4f1
--- /dev/null
+++ b/.kokoro/release/phishingprotection.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "phishingprotection"
+}
diff --git a/.kokoro/release/pubsub.cfg b/.kokoro/release/pubsub.cfg
new file mode 100644
index 000000000000..16c7ed3c175c
--- /dev/null
+++ b/.kokoro/release/pubsub.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "pubsub"
+}
diff --git a/.kokoro/release/redis.cfg b/.kokoro/release/redis.cfg
new file mode 100644
index 000000000000..268bc06a7101
--- /dev/null
+++ b/.kokoro/release/redis.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "redis"
+}
diff --git a/.kokoro/release/resource_manager.cfg b/.kokoro/release/resource_manager.cfg
new file mode 100644
index 000000000000..7a3976aef6cd
--- /dev/null
+++ b/.kokoro/release/resource_manager.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "resource_manager"
+}
diff --git a/.kokoro/release/runtimeconfig.cfg b/.kokoro/release/runtimeconfig.cfg
new file mode 100644
index 000000000000..2900b3c7e91a
--- /dev/null
+++ b/.kokoro/release/runtimeconfig.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "runtimeconfig"
+}
diff --git a/.kokoro/release/scheduler.cfg b/.kokoro/release/scheduler.cfg
new file mode 100644
index 000000000000..0701399853ef
--- /dev/null
+++ b/.kokoro/release/scheduler.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "scheduler"
+}
diff --git a/.kokoro/release/securitycenter.cfg b/.kokoro/release/securitycenter.cfg
new file mode 100644
index 000000000000..17dc427dae15
--- /dev/null
+++ b/.kokoro/release/securitycenter.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "securitycenter"
+}
diff --git a/.kokoro/release/spanner.cfg b/.kokoro/release/spanner.cfg
new file mode 100644
index 000000000000..d986f9d137f9
--- /dev/null
+++ b/.kokoro/release/spanner.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "spanner"
+}
diff --git a/.kokoro/release/speech.cfg b/.kokoro/release/speech.cfg
new file mode 100644
index 000000000000..b178b790be8d
--- /dev/null
+++ b/.kokoro/release/speech.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "speech"
+}
diff --git a/.kokoro/release/storage.cfg b/.kokoro/release/storage.cfg
new file mode 100644
index 000000000000..6bd9371ff99d
--- /dev/null
+++ b/.kokoro/release/storage.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "storage"
+}
diff --git a/.kokoro/release/talent.cfg b/.kokoro/release/talent.cfg
new file mode 100644
index 000000000000..abafead62769
--- /dev/null
+++ b/.kokoro/release/talent.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "talent"
+}
diff --git a/.kokoro/release/tasks.cfg b/.kokoro/release/tasks.cfg
new file mode 100644
index 000000000000..2ae5c581b88d
--- /dev/null
+++ b/.kokoro/release/tasks.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "tasks"
+}
diff --git a/.kokoro/release/texttospeech.cfg b/.kokoro/release/texttospeech.cfg
new file mode 100644
index 000000000000..9e7d6d7f7980
--- /dev/null
+++ b/.kokoro/release/texttospeech.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "texttospeech"
+}
diff --git a/.kokoro/release/trace.cfg b/.kokoro/release/trace.cfg
new file mode 100644
index 000000000000..9169296e4753
--- /dev/null
+++ b/.kokoro/release/trace.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "trace"
+}
diff --git a/.kokoro/release/translate.cfg b/.kokoro/release/translate.cfg
new file mode 100644
index 000000000000..807c96195b41
--- /dev/null
+++ b/.kokoro/release/translate.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "translate"
+}
diff --git a/.kokoro/release/videointelligence.cfg b/.kokoro/release/videointelligence.cfg
new file mode 100644
index 000000000000..77d1d3fc49e6
--- /dev/null
+++ b/.kokoro/release/videointelligence.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "videointelligence"
+}
diff --git a/.kokoro/release/vision.cfg b/.kokoro/release/vision.cfg
new file mode 100644
index 000000000000..4f686b3abf42
--- /dev/null
+++ b/.kokoro/release/vision.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "vision"
+}
diff --git a/.kokoro/release/webrisk.cfg b/.kokoro/release/webrisk.cfg
new file mode 100644
index 000000000000..36b06f0b0b55
--- /dev/null
+++ b/.kokoro/release/webrisk.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "webrisk"
+}
diff --git a/.kokoro/release/websecurityscanner.cfg b/.kokoro/release/websecurityscanner.cfg
new file mode 100644
index 000000000000..221d1fc23d6b
--- /dev/null
+++ b/.kokoro/release/websecurityscanner.cfg
@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Tell the trampoline which build file to use.
+env_vars: {
+ key: "PACKAGE"
+ value: "websecurityscanner"
+}
diff --git a/.kokoro/trampoline.sh b/.kokoro/trampoline.sh
new file mode 100755
index 000000000000..e8c4251f3ed4
--- /dev/null
+++ b/.kokoro/trampoline.sh
@@ -0,0 +1,23 @@
+#!/bin/bash
+# Copyright 2017 Google Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -eo pipefail
+
+python3 "${KOKORO_GFILE_DIR}/trampoline_v1.py" || ret_code=$?
+
+chmod +x ${KOKORO_GFILE_DIR}/trampoline_cleanup.sh
+${KOKORO_GFILE_DIR}/trampoline_cleanup.sh || true
+
+exit ${ret_code}
diff --git a/.readthedocs.yml b/.readthedocs.yml
new file mode 100644
index 000000000000..607de497679a
--- /dev/null
+++ b/.readthedocs.yml
@@ -0,0 +1,5 @@
+requirements_file: docs/requirements.txt
+build:
+ image: latest
+python:
+ version: 3.6
diff --git a/.settings/org.eclipse.core.resources.prefs b/.settings/org.eclipse.core.resources.prefs
new file mode 100644
index 000000000000..86179f70d44f
--- /dev/null
+++ b/.settings/org.eclipse.core.resources.prefs
@@ -0,0 +1,2 @@
+eclipse.preferences.version=1
+encoding//bigtable/docs/conf.py=utf-8
diff --git a/.travis.yml b/.travis.yml
deleted file mode 100644
index b222923baf11..000000000000
--- a/.travis.yml
+++ /dev/null
@@ -1,33 +0,0 @@
-language: python
-sudo: false
-
-install:
- - pip install --upgrade pip tox
-
-script:
- - tox -e py27
- - tox -e py34
- - tox -e lint
- - tox -e cover
- - tox -e system-tests
- - tox -e system-tests3
- - scripts/update_docs.sh
-
-after_success:
- - tox -e coveralls
-
-deploy:
- provider: pypi
- user: gcloudpypi
- password:
- secure: LR0i9Oeu6kpLTYS5xK/zCng4gmdtPvFfD/XYdQhyY5jBibQkC2WUQU6nJA9bDXRxhBP5bUwXFGkbhOcOJgHNrUfmyPzpDbM8BR29KfY0WfdYv72gsGZOaekqCReFmHbqLE7qOQtHR5U3ey6ivcgw+hZO72Uu6qDCc9B8qwoBfAs=
- on:
- tags: true
- repo: GoogleCloudPlatform/google-cloud-python
- all_branches: true
- # 'bdist_wheel' builds disabled until #1879 et al. are resolved.
- distributions: "sdist"
-
-cache:
- directories:
- - ${HOME}/.cache/pip
diff --git a/CHANGELOG.md b/CHANGELOG.md
new file mode 100644
index 000000000000..7301a860a45c
--- /dev/null
+++ b/CHANGELOG.md
@@ -0,0 +1,73 @@
+# Changelog
+
+[PyPI History][1]
+
+[1]: https://pypi.org/project/google-cloud/#history
+
+## 0.32.0
+
+Packages newly added to the `google-cloud` umbrella package:
+
+- [`google-cloud-bigquery-datatransfer==0.1.0`](https://pypi.org/project/google-cloud-bigquery-datatransfer/0.1.0/)
+- [`google-cloud-container==0.1.0`](https://pypi.org/project/google-cloud-container/0.1.0/)
+
+Feature / bugfix release for the following packages:
+
+- [`google-api-core==0.1.2`](https://pypi.org/project/google-api-core/0.1.2/)
+- [`google-cloud-bigtable==0.28.1`](https://pypi.org/project/google-cloud-bigtable/0.28.1/)
+- [`google-cloud-pubsub==0.30.0`](https://pypi.org/project/google-cloud-pubsub/0.30.0/)
+- [`google-cloud-trace==0.17.0`](https://pypi.org/project/google-cloud-trace/0.17.0/)
+- [`google-cloud-vision==0.29.0`](https://pypi.org/project/google-cloud-vision/0.29.0/)
+
+## 0.31.0
+
+Feature / bugfix release for the following packages:
+
+ * [`google-cloud-pubsub==0.29.1`](https://pypi.org/project/google-cloud-pubsub/0.29.1/)
+ * [`google-cloud-videointelligence==1.0.0`](https://pypi.org/project/google-cloud-videointelligence/1.0.0/)
+
+## 0.30.0
+
+Feature / bugfix release for the following package:
+
+ * [`google-cloud-language==1.0.0`](https://pypi.org/project/google-cloud-language/1.0.0/)
+
+## 0.29.0
+
+Feature / bugfix release for the following package:
+
+- [`google-cloud-language==0.31.0`](https://pypi.org/project/google-cloud-language/0.31.0/)
+
+## 0.28.0
+
+Packages newly added to the `google-cloud` umbrella package:
+
+- [`google-api-core==0.1.1`](https://pypi.org/project/google-api-core/0.1.1/)
+- [`google-cloud-firestore==0.28.0`](https://pypi.org/project/google-cloud-firestore/0.28.0/)
+- [`google-cloud-trace==0.16.0`](https://pypi.org/project/google-cloud-trace/0.16.0/)
+- [`google-cloud-videointelligence=0.28.0`](https://pypi.org/project/google-cloud-videointelligence/0.28.0/)
+
+Feature / bugfix releases for the following packages:
+
+- [`google-cloud-bigquery==0.28.0`](https://pypi.org/project/google-cloud-bigquery/0.28.0/)
+- [`google-cloud-core==0.28.0`](https://pypi.org/project/google-cloud-core/0.28.0/)
+- [`google-cloud-datastore==1.4.0`](https://pypi.org/project/google-cloud-datastore/1.4.0/)
+- [`google-cloud-logging==1.4.0`](https://pypi.org/project/google-cloud-logging/1.4.0/)
+- [`google-cloud-pubsub==0.29.0`](https://pypi.org/project/google-cloud-pubsub/0.29.0/)
+- [`google-cloud-spanner==0.29.0`](https://pypi.org/project/google-cloud-spanner/0.29.0/)
+- [`google-cloud-translate==1.3.0`](https://pypi.org/project/google-cloud-translate/1.3.0/)
+- [`google-cloud-vision==0.28.0`](https://pypi.org/project/google-cloud-vision/0.28.0/)
+
+Dependency-only / user-invisible changes to the following packages:
+
+- [`google-cloud-bigtable==0.28.0`](https://pypi.org/project/google-cloud-bigtable/0.28.0/)
+- [`google-cloud-dns==0.28.0`](https://pypi.org/project/google-cloud-dns/0.28.0/)
+- [`google-cloud-error-reporting==0.28.0`](https://pypi.org/project/google-cloud-error-reporting/0.28.0/)
+- [`google-cloud-language==0.30.0`](https://pypi.org/project/google-cloud-language/0.30.0/)
+- [`google-cloud-monitoring==0.28.0`](https://pypi.org/project/google-cloud-monitoring/0.28.0/)
+- [`google-cloud-resource-manager==0.28.0`](https://pypi.org/project/google-cloud-resource-manager/0.28.0/)
+- [`google-cloud-runtimeconfig==0.28.0`](https://pypi.org/project/google-cloud-runtimeconfig/0.28.0/)
+- [`google-cloud-speech==0.30.0`](https://pypi.org/project/google-cloud-speech/0.30.0/)
+- [`google-cloud-storage==1.6.0`](https://pypi.org/project/google-cloud-storage/1.6.0/)
+
+PyPI: https://pypi.org/project/google-cloud/0.28.0/
diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
index 9cb07618962e..b05875c89e81 100644
--- a/CONTRIBUTING.rst
+++ b/CONTRIBUTING.rst
@@ -1,37 +1,55 @@
+############
Contributing
-============================
+############
#. **Please sign one of the contributor license agreements below.**
#. Fork the repo, develop and test your code changes, add docs.
#. Make sure that your commit messages clearly describe the changes.
-#. Send a pull request.
+#. Send a pull request. (Please Read: `Faster Pull Request Reviews`_)
-Here are some guidelines for hacking on ``google-cloud-python``.
+.. _Faster Pull Request Reviews: https://github.com/kubernetes/community/blob/master/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
-Using a Development Checkout
-----------------------------
+.. contents:: Here are some guidelines for hacking on ``google-cloud-python``.
+
+***************
+Adding Features
+***************
+
+In order to add a feature to ``google-cloud-python``:
+
+- The feature must be documented in both the API and narrative
+ documentation (in ``docs/``).
+
+- The feature must work fully on the following CPython versions: 2.7,
+ 3.5, 3.6, and 3.7 on both UNIX and Windows.
-You'll have to create a development environment to hack on ``google-cloud-python``,
-using a Git checkout:
+- The feature must not add unnecessary dependencies (where
+ "unnecessary" is of course subjective, but new dependencies should
+ be discussed).
-- While logged into your GitHub account, navigate to the ``google-cloud-python`` repo
- on GitHub.
+****************************
+Using a Development Checkout
+****************************
- https://github.com/GoogleCloudPlatform/google-cloud-python
+You'll have to create a development environment to hack on
+``google-cloud-python``, using a Git checkout:
+
+- While logged into your GitHub account, navigate to the
+ ``google-cloud-python`` `repo`_ on GitHub.
- Fork and clone the ``google-cloud-python`` repository to your GitHub account by
clicking the "Fork" button.
- Clone your fork of ``google-cloud-python`` from your GitHub account to your local
computer, substituting your account username and specifying the destination
- as "hack-on-google-cloud-python". E.g.::
+ as ``hack-on-google-cloud-python``. E.g.::
- $ cd ~
+ $ cd ${HOME}
$ git clone git@github.com:USERNAME/google-cloud-python.git hack-on-google-cloud-python
$ cd hack-on-google-cloud-python
# Configure remotes such that you can pull changes from the google-cloud-python
# repository into your local repository.
- $ git remote add upstream https://github.com:GoogleCloudPlatform/google-cloud-python
+ $ git remote add upstream git@github.com:GoogleCloudPlatform/google-cloud-python.git
# fetch and merge changes from upstream into master
$ git fetch upstream
$ git merge upstream/master
@@ -39,69 +57,82 @@ using a Git checkout:
Now your local repo is set up such that you will push changes to your GitHub
repo, from which you can submit a pull request.
-- Create a virtualenv in which to install ``google-cloud-python``::
+To work on the codebase and run the tests, we recommend using ``nox``,
+but you can also use a ``virtualenv`` of your own creation.
- $ cd ~/hack-on-google-cloud-python
- $ virtualenv --python python2.7 env
+.. _repo: https://github.com/GoogleCloudPlatform/google-cloud-python
- Note that very old versions of virtualenv (virtualenv versions below, say,
- 1.10 or thereabouts) require you to pass a ``--no-site-packages`` flag to
- get a completely isolated environment.
+Using ``nox``
+=============
- You can choose which Python version you want to use by passing a ``--python``
- flag to ``virtualenv``. For example, ``virtualenv --python python2.7``
- chooses the Python 2.7 interpreter to be installed.
+We use `nox `__ to instrument our tests.
- From here on in within these instructions, the ``~/hack-on-google-cloud-python/env``
- virtual environment you created above will be referred to as ``$VENV``.
- To use the instructions in the steps that follow literally, use the
- ``export VENV=~/hack-on-google-cloud-python/env`` command.
+- To test your changes, run unit tests with ``nox``::
-- Install ``google-cloud-python`` from the checkout into the virtualenv using
- ``setup.py develop``. Running ``setup.py develop`` *must* be done while
- the current working directory is the ``google-cloud-python`` checkout directory::
+ $ nox -f datastore/noxfile.py -s unit-2.7
+ $ nox -f datastore/noxfile.py -s unit-3.7
+ $ ...
- $ cd ~/hack-on-google-cloud-python
- $ $VENV/bin/python setup.py develop
+ .. note::
-I'm getting weird errors... Can you help?
------------------------------------------
+ The unit tests and system tests are contained in the individual
+ ``nox.py`` files in each directory; substitute ``datastore`` in the
+ example above with the package of your choice.
-If the error mentions ``Python.h`` not being found,
-install ``python-dev`` and try again.
-On Debian/Ubuntu::
- $ sudo apt-get install python-dev
+ Alternatively, you can just navigate directly to the package you are
+ currently developing and run tests there::
-Adding Features
----------------
+ $ export GIT_ROOT=$(pwd)
+ $ cd ${GIT_ROOT}/datastore/
+ $ nox -s "unit(py='3.7')"
-In order to add a feature to ``google-cloud-python``:
+.. nox: https://pypi.org/project/nox-automation/
-- The feature must be documented in both the API and narrative
- documentation (in ``docs/``).
+Note on Editable Installs / Develop Mode
+========================================
-- The feature must work fully on the following CPython versions: 2.7,
- 3.4, and 3.5 on both UNIX and Windows.
+- As mentioned previously, using ``setuptools`` in `develop mode`_
+ or a ``pip`` `editable install`_ is not possible with this
+ library. This is because this library uses `namespace packages`_.
+ For context see `Issue #2316`_ and the relevant `PyPA issue`_.
-- The feature must not add unnecessary dependencies (where
- "unnecessary" is of course subjective, but new dependencies should
- be discussed).
+ Since ``editable`` / ``develop`` mode can't be used, packages
+ need to be installed directly. Hence your changes to the source
+ tree don't get incorporated into the **already installed**
+ package.
+
+.. _namespace packages: https://www.python.org/dev/peps/pep-0420/
+.. _Issue #2316: https://github.com/GoogleCloudPlatform/google-cloud-python/issues/2316
+.. _PyPA issue: https://github.com/pypa/packaging-problems/issues/12
+.. _develop mode: https://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode
+.. _editable install: https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs
+*****************************************
+I'm getting weird errors... Can you help?
+*****************************************
+
+If the error mentions ``Python.h`` not being found,
+install ``python-dev`` and try again.
+On Debian/Ubuntu::
+
+ $ sudo apt-get install python-dev
+
+************
Coding Style
-------------
+************
-- PEP8 compliance, with exceptions defined in ``tox.ini``.
- If you have ``tox`` installed, you can test that you have not introduced
+- PEP8 compliance, with exceptions defined in the linter configuration.
+ If you have ``nox`` installed, you can test that you have not introduced
any non-compliant code via::
- $ tox -e lint
+ $ nox -s lint
-- In order to make ``tox -e lint`` run faster, you can set some environment
+- In order to make ``nox -s lint`` run faster, you can set some environment
variables::
- export GOOGLE_CLOUD_REMOTE_FOR_LINT="upstream"
- export GOOGLE_CLOUD_BRANCH_FOR_LINT="master"
+ export GOOGLE_CLOUD_TESTING_REMOTE="upstream"
+ export GOOGLE_CLOUD_TESTING_BRANCH="master"
By doing this, you are specifying the location of the most up-to-date
version of ``google-cloud-python``. The the suggested remote name ``upstream``
@@ -110,47 +141,24 @@ Coding Style
Exceptions to PEP8:
-- Many unit tests use a helper method, ``_callFUT`` ("FUT" is short for
+- Many unit tests use a helper method, ``_call_fut`` ("FUT" is short for
"Function-Under-Test"), which is PEP8-incompliant, but more readable.
Some also use a local variable, ``MUT`` (short for "Module-Under-Test").
-Running Tests
---------------
-
-- To run all tests for ``google-cloud-python`` on a single Python version, run
- ``py.test`` from your development virtualenv (See
- *Using a Development Checkout* above).
-
-- To run the full set of ``google-cloud-python`` tests on all platforms, install
- ``tox`` (https://testrun.org/tox/) into a system Python. The ``tox`` console
- script will be installed into the scripts location for that Python. While
- ``cd``'ed to the ``google-cloud-python`` checkout root directory (it contains
- ``tox.ini``), invoke the ``tox`` console script. This will read the
- ``tox.ini`` file and execute the tests on multiple Python versions and
- platforms; while it runs, it creates a virtualenv for each version/platform
- combination. For example::
-
- $ sudo /usr/bin/pip install tox
- $ cd ~/hack-on-google-cloud-python/
- $ /usr/bin/tox
-
+********************
Running System Tests
---------------------
+********************
-- To run system tests you can execute::
+- To run system tests for a given package, you can execute::
- $ tox -e system-tests
- $ tox -e system-tests3
+ $ nox -f datastore/noxfile.py -s system-3.7
+ $ nox -f datastore/noxfile.py -s system-2.7
- or run only system tests for a particular package via::
+ .. note::
- $ python system_tests/run_system_test.py --package {package}
- $ python3 system_tests/run_system_test.py --package {package}
-
- To run a subset of the system tests::
-
- $ tox -e system-tests -- datastore storage
- $ python system_tests/attempt_system_tests.py datastore storage
+ System tests are only configured to run under Python 2.7 and
+ Python 3.7. For expediency, we do not run them in older versions
+ of Python 3.
This alone will not run the tests. You'll need to change some local
auth settings and change some configuration in your project to
@@ -161,20 +169,20 @@ Running System Tests
authentication to your project:
- ``GOOGLE_APPLICATION_CREDENTIALS``: The path to a JSON key file;
- see ``system_tests/app_credentials.json.sample`` as an example. Such a file
- can be downloaded directly from the developer's console by clicking
+ Such a file can be downloaded directly from the developer's console by clicking
"Generate new JSON key". See private key
`docs `__
- for more details. In order for Logging system tests to work, the Service Account
- will also have to be made a project Owner. This can be changed under "IAM & Admin".
- - ``GOOGLE_CLOUD_TESTS_API_KEY``: The API key for your project with
- the Google Translate API (and others) enabled.
+ for more details.
-- Examples of these can be found in ``system_tests/local_test_setup.sample``. We
- recommend copying this to ``system_tests/local_test_setup``, editing the
- values and sourcing them into your environment::
+ - In order for Logging system tests to work, the Service Account
+ will also have to be made a project ``Owner``. This can be changed under
+ "IAM & Admin". Additionally, ``cloud-logs@google.com`` must be given
+ ``Editor`` permissions on the project.
- $ source system_tests/local_test_setup
+- Once you have downloaded your json keys, set the environment variable
+ ``GOOGLE_APPLICATION_CREDENTIALS`` to the absolute path of the json file::
+
+ $ export GOOGLE_APPLICATION_CREDENTIALS="/Users//path/to/app_credentials.json"
- For datastore tests, you'll need to create composite
`indexes `__
@@ -182,11 +190,7 @@ Running System Tests
`tool `__::
# Install the app (App Engine Command Line Interface) component.
- $ gcloud components update app
-
- # See https://cloud.google.com/sdk/crypto for details on PyOpenSSL and
- # http://stackoverflow.com/a/25067729/1068170 for why we must persist.
- $ export CLOUDSDK_PYTHON_SITEPACKAGES=1
+ $ gcloud components install app-engine-python
# Authenticate the gcloud tool with your account.
$ GOOGLE_APPLICATION_CREDENTIALS="path/to/app_credentials.json"
@@ -194,132 +198,66 @@ Running System Tests
> --key-file=${GOOGLE_APPLICATION_CREDENTIALS}
# Create the indexes
- $ gcloud preview datastore create-indexes system_tests/data/index.yaml
-
- # Restore your environment to its previous state.
- $ unset CLOUDSDK_PYTHON_SITEPACKAGES
+ $ gcloud datastore indexes create datastore/tests/system/index.yaml
- For datastore query tests, you'll need stored data in your dataset.
To populate this data, run::
- $ python system_tests/populate_datastore.py
+ $ python datastore/tests/system/utils/populate_datastore.py
- If you make a mistake during development (i.e. a failing test that
prevents clean-up) you can clear all system test data from your
datastore instance via::
- $ python system_tests/clear_datastore.py
-
-- System tests can also be run against local `emulators`_ that mock
- the production services. To run the system tests with the
- ``datastore`` emulator::
-
- $ tox -e datastore-emulator
- $ GOOGLE_CLOUD_DISABLE_GRPC=true tox -e datastore-emulator
-
- This also requires that the ``gcloud`` command line tool is
- installed. If you'd like to run them directly (outside of a
- ``tox`` environment), first start the emulator and
- take note of the process ID::
-
- $ gcloud beta emulators datastore start --no-legacy 2>&1 > log.txt &
- [1] 33333
-
- then determine the environment variables needed to interact with
- the emulator::
-
- $ gcloud beta emulators datastore env-init
- export DATASTORE_LOCAL_HOST=localhost:8417
- export DATASTORE_HOST=http://localhost:8417
- export DATASTORE_DATASET=google-cloud-settings-app-id
- export DATASTORE_PROJECT_ID=google-cloud-settings-app-id
-
- using these environment variables run the emulator::
-
- $ DATASTORE_HOST=http://localhost:8471 \
- > DATASTORE_DATASET=google-cloud-settings-app-id \
- > GOOGLE_CLOUD_NO_PRINT=true \
- > python system_tests/run_system_test.py \
- > --package=datastore --ignore-requirements
+ $ python datastore/tests/system/utils/clear_datastore.py
- and after completion stop the emulator and any child
- processes it spawned::
-
- $ kill -- -33333
-
-.. _emulators: https://cloud.google.com/sdk/gcloud/reference/beta/emulators/
-
-- To run the system tests with the ``pubsub`` emulator::
-
- $ tox -e pubsub-emulator
- $ GOOGLE_CLOUD_DISABLE_GRPC=true tox -e pubsub-emulator
-
- If you'd like to run them directly (outside of a ``tox`` environment), first
- start the emulator and take note of the process ID::
-
- $ gcloud beta emulators pubsub start 2>&1 > log.txt &
- [1] 44444
-
- then determine the environment variables needed to interact with
- the emulator::
-
- $ gcloud beta emulators pubsub env-init
- export PUBSUB_EMULATOR_HOST=localhost:8897
-
- using these environment variables run the emulator::
-
- $ PUBSUB_EMULATOR_HOST=localhost:8897 \
- > python system_tests/run_system_test.py \
- > --package=pubsub
-
- and after completion stop the emulator and any child
- processes it spawned::
-
- $ kill -- -44444
+*************
Test Coverage
--------------
+*************
- The codebase *must* have 100% test statement coverage after each commit.
- You can test coverage via ``tox -e cover``.
+ You can test coverage via ``nox -s cover``.
+******************************************************
Documentation Coverage and Building HTML Documentation
-------------------------------------------------------
+******************************************************
If you fix a bug, and the bug requires an API or behavior modification, all
documentation in this package which references that API or behavior must be
changed to reflect the bug fix, ideally in the same commit that fixes the bug
or adds the feature.
-To build and review docs (where ``$VENV`` refers to the virtualenv you're
+To build and review docs (where ``${VENV}`` refers to the virtualenv you're
using to develop ``google-cloud-python``):
-1. After following the steps above in "Using a Development Checkout", install
+#. After following the steps above in "Using a Development Checkout", install
Sphinx and all development requirements in your virtualenv::
- $ cd ~/hack-on-google-cloud-python
- $ $VENV/bin/pip install Sphinx
+ $ cd ${HOME}/hack-on-google-cloud-python
+ $ ${VENV}/bin/pip install Sphinx
-2. Change into the ``docs`` directory within your ``google-cloud-python`` checkout and
+#. Change into the ``docs`` directory within your ``google-cloud-python`` checkout and
execute the ``make`` command with some flags::
- $ cd ~/hack-on-google-cloud-python/google-cloud-python/docs
- $ make clean html SPHINXBUILD=$VENV/bin/sphinx-build
+ $ cd ${HOME}/hack-on-google-cloud-python/google-cloud-python/docs
+ $ make clean html SPHINXBUILD=${VENV}/bin/sphinx-build
The ``SPHINXBUILD=...`` argument tells Sphinx to use the virtualenv Python,
which will have both Sphinx and ``google-cloud-python`` (for API documentation
generation) installed.
-3. Open the ``docs/_build/html/index.html`` file to see the resulting HTML
+#. Open the ``docs/_build/html/index.html`` file to see the resulting HTML
rendering.
-As an alternative to 1. and 2. above, if you have ``tox`` installed, you
+As an alternative to 1. and 2. above, if you have ``nox`` installed, you
can build the docs via::
- $ tox -e docs
+ $ nox -s docs
+********************************************
Note About ``README`` as it pertains to PyPI
---------------------------------------------
+********************************************
The `description on PyPI`_ for the project comes directly from the
``README``. Due to the reStructuredText (``rst``) parser used by
@@ -328,58 +266,51 @@ instead of
``https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/CONTRIBUTING.rst``)
may cause problems creating links or rendering the description.
-.. _description on PyPI: https://pypi.python.org/pypi/google-cloud
-
-Travis Configuration and Build Optimizations
---------------------------------------------
+.. _description on PyPI: https://pypi.org/project/google-cloud/
-All build scripts in the ``.travis.yml`` configuration file which have
-Python dependencies are specified in the ``tox.ini`` configuration.
-They are executed in the Travis build via ``tox -e {ENV}`` where
-``{ENV}`` is the environment being tested.
+**********************
+CircleCI Configuration
+**********************
-If new ``tox`` environments are added to be run in a Travis build, they
-should be listed in ``[tox].envlist`` as a default environment.
+All build scripts in the ``.circleci/config.yml`` configuration file which have
+Python dependencies are specified in the ``nox.py`` configuration.
+They are executed in the Travis build via ``nox -s ${ENV}`` where
+``${ENV}`` is the environment being tested.
-We speed up builds by using the Travis `caching feature`_.
-
-.. _caching feature: https://docs.travis-ci.com/user/caching/#pip-cache
-
-We intentionally **do not** cache the ``.tox/`` directory. Instead, we
-allow the ``tox`` environments to be re-built for every build. This
-way, we'll always get the latest versions of our dependencies and any
-caching or wheel optimization to be done will be handled automatically
-by ``pip``.
+*************************
Supported Python Versions
--------------------------
+*************************
We support:
-- `Python 2.7`_
-- `Python 3.4`_
- `Python 3.5`_
+- `Python 3.6`_
+- `Python 3.7`_
-.. _Python 2.7: https://docs.python.org/2.7/
-.. _Python 3.4: https://docs.python.org/3.4/
.. _Python 3.5: https://docs.python.org/3.5/
+.. _Python 3.6: https://docs.python.org/3.6/
+.. _Python 3.7: https://docs.python.org/3.7/
-Supported versions can be found in our ``tox.ini`` `config`_.
-.. _config: https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/tox.ini
+Supported versions can be found in our ``noxfile.py`` `config`_.
+
+.. _config: https://github.com/googleapis/google-cloud-python/blob/master/noxfile.py
We explicitly decided not to support `Python 2.5`_ due to `decreased usage`_
and lack of continuous integration `support`_.
.. _Python 2.5: https://docs.python.org/2.5/
.. _decreased usage: https://caremad.io/2013/10/a-look-at-pypi-downloads/
-.. _support: http://blog.travis-ci.com/2013-11-18-upcoming-build-environment-updates/
+.. _support: https://blog.travis-ci.com/2013-11-18-upcoming-build-environment-updates/
We have `dropped 2.6`_ as a supported version as well since Python 2.6 is no
longer supported by the core development team.
+Python 2.7 support is deprecated. All code changes should maintain Python 2.7 compatibility until January 1, 2020.
+
We also explicitly decided to support Python 3 beginning with version
-3.4. Reasons for this include:
+3.5. Reasons for this include:
- Encouraging use of newest versions of Python 3
- Taking the lead of `prominent`_ open-source `projects`_
@@ -389,25 +320,33 @@ We also explicitly decided to support Python 3 beginning with version
.. _prominent: https://docs.djangoproject.com/en/1.9/faq/install/#what-python-version-can-i-use-with-django
.. _projects: http://flask.pocoo.org/docs/0.10/python3/
.. _Unicode literal support: https://www.python.org/dev/peps/pep-0414/
-.. _dropped 2.6: https://github.com/GoogleCloudPlatform/google-cloud-python/issues/995
+.. _dropped 2.6: https://github.com/googleapis/google-cloud-python/issues/995
+**********
Versioning
-----------
+**********
This library follows `Semantic Versioning`_.
.. _Semantic Versioning: http://semver.org/
-It is currently in major version zero (``0.y.z``), which means that anything
-may change at any time and the public API should not be considered
+Some packages are currently in major version zero (``0.y.z``), which means that
+anything may change at any time and the public API should not be considered
stable.
+******************************
Contributor License Agreements
-------------------------------
+******************************
-Before we can accept your pull requests you'll need to sign a Contributor License Agreement (CLA):
+Before we can accept your pull requests you'll need to sign a Contributor
+License Agreement (CLA):
-- **If you are an individual writing original source code** and **you own the intellectual property**, then you'll need to sign an `individual CLA `__.
-- **If you work for a company that wants to allow you to contribute your work**, then you'll need to sign a `corporate CLA `__.
+- **If you are an individual writing original source code** and **you own the
+ intellectual property**, then you'll need to sign an
+ `individual CLA `__.
+- **If you work for a company that wants to allow you to contribute your work**,
+ then you'll need to sign a
+ `corporate CLA `__.
-You can sign these electronically (just scroll to the bottom). After that, we'll be able to accept your pull requests.
+You can sign these electronically (just scroll to the bottom). After that,
+we'll be able to accept your pull requests.
diff --git a/MANIFEST.in b/MANIFEST.in
deleted file mode 100644
index ae20b86a84fd..000000000000
--- a/MANIFEST.in
+++ /dev/null
@@ -1,3 +0,0 @@
-include README.rst
-graft google
-global-exclude *.pyc
diff --git a/Makefile.bigtable_v2 b/Makefile.bigtable_v2
index 46a958cf4234..b739053e0116 100644
--- a/Makefile.bigtable_v2
+++ b/Makefile.bigtable_v2
@@ -1,15 +1,15 @@
GRPCIO_VIRTUALENV=$(shell pwd)/grpc_python_venv
GENERATED_DIR=$(shell pwd)/generated_python
GENERATED_SUBDIR=_generated
-BIGTABLE_DIR=$(shell pwd)/google/cloud/bigtable/$(GENERATED_SUBDIR)
+BIGTABLE_DIR=$(shell pwd)/bigtable/google/cloud/bigtable/$(GENERATED_SUBDIR)
PROTOC_CMD=$(GRPCIO_VIRTUALENV)/bin/python -m grpc.tools.protoc
GOOGLEAPIS_PROTOS_DIR=$(shell pwd)/googleapis-pb
help:
- @echo 'Makefile for google-cloud-python Bigtable protos '
- @echo ' '
- @echo ' make generate Generates the protobuf modules '
- @echo ' make clean Clean generated files '
+ @echo 'Makefile for google-cloud-python Bigtable protos '
+ @echo ' '
+ @echo ' make generate Generates the protobuf modules'
+ @echo ' make clean Clean generated files '
generate:
# Ensure we have a virtualenv w/ up-to-date grpcio/grpcio-tools
@@ -44,7 +44,6 @@ generate:
# Copy over the *.proto files into our library.
cp $(GOOGLEAPIS_PROTOS_DIR)/google/bigtable/v2/*.proto $(BIGTABLE_DIR)
cp $(GOOGLEAPIS_PROTOS_DIR)/google/bigtable/admin/v2/*.proto $(BIGTABLE_DIR)
- cp $(GOOGLEAPIS_PROTOS_DIR)/google/longrunning/operations.proto $(BIGTABLE_DIR)
# Rename all *.proto files in our library with an
# underscore and remove executable bit.
cd $(BIGTABLE_DIR) && \
@@ -52,12 +51,7 @@ generate:
chmod -x $$filename ; \
mv $$filename _$$filename ; \
done
- # Separate the gRPC parts of the operations service from the
- # non-gRPC parts so that the protos from `googleapis-common-protos`
- # can be used without gRPC.
GRPCIO_VIRTUALENV="$(GRPCIO_VIRTUALENV)" \
- GENERATED_SUBDIR=$(GENERATED_SUBDIR) \
- python scripts/make_operations_grpc.py
# Rewrite the imports in the generated *_pb2.py files.
python scripts/rewrite_imports.py $(BIGTABLE_DIR)/*pb2.py
diff --git a/Makefile.datastore b/Makefile.datastore
deleted file mode 100644
index 837f5ba26390..000000000000
--- a/Makefile.datastore
+++ /dev/null
@@ -1,53 +0,0 @@
-GRPCIO_VIRTUALENV=$(shell pwd)/grpc_python_venv
-GENERATED_DIR=$(shell pwd)/generated_python
-DATASTORE_DIR=$(shell pwd)/google/cloud/datastore/_generated
-PROTOC_CMD=$(GRPCIO_VIRTUALENV)/bin/python -m grpc.tools.protoc
-GOOGLEAPIS_PROTOS_DIR=$(shell pwd)/googleapis-pb
-
-help:
- @echo 'Makefile for google-cloud-python Bigtable protos '
- @echo ' '
- @echo ' make generate Generates the protobuf modules '
- @echo ' make clean Clean generated files '
-
-generate:
- # Ensure we have a virtualenv w/ up-to-date grpcio/grpcio-tools
- [ -d $(GRPCIO_VIRTUALENV) ] || python2.7 -m virtualenv $(GRPCIO_VIRTUALENV)
- $(GRPCIO_VIRTUALENV)/bin/pip install --upgrade grpcio grpcio-tools
- # Retrieve git repos that have our *.proto files.
- [ -d googleapis-pb ] || git clone https://github.com/googleapis/googleapis googleapis-pb --depth=1
- cd googleapis-pb && git pull origin master
- # Make the directory where our *_pb2.py files will go.
- mkdir -p $(GENERATED_DIR)
- # Generate all *_pb2.py files that do not require gRPC.
- $(PROTOC_CMD) \
- --proto_path=$(GOOGLEAPIS_PROTOS_DIR) \
- --python_out=$(GENERATED_DIR) \
- $(GOOGLEAPIS_PROTOS_DIR)/google/datastore/v1/datastore.proto \
- $(GOOGLEAPIS_PROTOS_DIR)/google/datastore/v1/entity.proto \
- $(GOOGLEAPIS_PROTOS_DIR)/google/datastore/v1/query.proto
- # Move the newly generated *_pb2.py files into our library.
- cp $(GENERATED_DIR)/google/datastore/v1/* $(DATASTORE_DIR)
- # Remove all existing *.proto files before we replace
- rm -f $(DATASTORE_DIR)/*.proto
- # Copy over the *.proto files into our library.
- cp $(GOOGLEAPIS_PROTOS_DIR)/google/datastore/v1/*.proto $(DATASTORE_DIR)
- # Rename all *.proto files in our library with an
- # underscore and remove executable bit.
- cd $(DATASTORE_DIR) && \
- for filename in *.proto; do \
- chmod -x $$filename ; \
- mv $$filename _$$filename ; \
- done
- # Separate the gRPC parts of the datastore service from the
- # non-gRPC parts so that the protos can be used without gRPC.
- GRPCIO_VIRTUALENV="$(GRPCIO_VIRTUALENV)" \
- GENERATED_SUBDIR=$(GENERATED_SUBDIR) \
- python scripts/make_datastore_grpc.py
- # Rewrite the imports in the generated *_pb2.py files.
- python scripts/rewrite_imports.py $(DATASTORE_DIR)/*pb2.py
-
-clean:
- rm -fr $(GENERATED_DIR)
-
-.PHONY: generate clean
diff --git a/README.rst b/README.rst
index 39568ec8a5a8..f0820498cc21 100644
--- a/README.rst
+++ b/README.rst
@@ -1,47 +1,170 @@
Google Cloud Python Client
==========================
- Python idiomatic client for `Google Cloud Platform`_ services.
+Python idiomatic clients for `Google Cloud Platform`_ services.
.. _Google Cloud Platform: https://cloud.google.com/
-|pypi| |build| |appveyor| |coverage| |versions|
+**Heads up**! These libraries are supported on App Engine standard's `Python 3 runtime`_ but are *not* supported on App Engine's `Python 2 runtime`_.
-- `Homepage`_
-- `API Documentation`_
+.. _Python 3 runtime: https://cloud.google.com/appengine/docs/standard/python3
+.. _Python 2 runtime: https://cloud.google.com/appengine/docs/standard/python
-.. _Homepage: https://googlecloudplatform.github.io/google-cloud-python/
-.. _API Documentation: http://googlecloudplatform.github.io/google-cloud-python/
-
-This client supports the following Google Cloud Platform services:
+General Availability
+--------------------
-- `Google Cloud Datastore`_
-- `Google Cloud Storage`_
-- `Google Cloud Pub/Sub`_
-- `Google BigQuery`_
-- `Google Cloud Resource Manager`_
-- `Google Stackdriver Logging`_
-- `Google Stackdriver Monitoring`_
+**GA** (general availability) indicates that the client library for a
+particular service is stable, and that the code surface will not change in
+backwards-incompatible ways unless either absolutely necessary (e.g. because
+of critical security issues) or with an extensive deprecation period.
+Issues and requests against GA libraries are addressed with the highest
+priority.
+
+.. note::
+
+ Sub-components of GA libraries explicitly marked as beta in the
+ import path (e.g. ``google.cloud.language_v1beta2``) should be considered
+ to be beta.
+
+The following client libraries have **GA** support:
+
+- `Google BigQuery`_ (`BigQuery README`_)
+- `Google Cloud Datastore`_ (`Datastore README`_)
+- `Google Cloud KMS`_ (`KMS README`_)
+- `Google Cloud Natural Language`_ (`Natural Language README`_)
+- `Google Cloud Scheduler`_ (`Scheduler README`_)
+- `Google Cloud Spanner`_ (`Spanner README`_)
+- `Google Cloud Speech`_ (`Speech README`_)
+- `Google Cloud Storage`_ (`Storage README`_)
+- `Google Cloud Tasks`_ (`Tasks README`_)
+- `Google Cloud Translation`_ (`Translation README`_)
+- `Stackdriver Logging`_ (`Logging README`_)
+
+.. _Google BigQuery: https://pypi.org/project/google-cloud-bigquery/
+.. _BigQuery README: https://github.com/googleapis/google-cloud-python/tree/master/bigquery
+.. _Google Cloud Datastore: https://pypi.org/project/google-cloud-datastore/
+.. _Datastore README: https://github.com/googleapis/google-cloud-python/tree/master/datastore
+.. _Google Cloud KMS: https://pypi.org/project/google-cloud-kms/
+.. _KMS README: https://github.com/googleapis/google-cloud-python/tree/master/kms
+.. _Google Cloud Natural Language: https://pypi.org/project/google-cloud-language/
+.. _Natural Language README: https://github.com/googleapis/google-cloud-python/tree/master/language
+.. _Google Cloud Spanner: https://pypi.org/project/google-cloud-spanner
+.. _Spanner README: https://github.com/googleapis/google-cloud-python/tree/master/spanner
+.. _Google Cloud Speech: https://pypi.org/project/google-cloud-speech/
+.. _Speech README: https://github.com/googleapis/google-cloud-python/tree/master/speech
+.. _Google Cloud Storage: https://pypi.org/project/google-cloud-storage/
+.. _Storage README: https://github.com/googleapis/google-cloud-python/tree/master/storage
+.. _Google Cloud Tasks: https://pypi.org/project/google-cloud-tasks/
+.. _Tasks README: https://github.com/googleapis/google-cloud-python/tree/master/tasks
+.. _Google Cloud Translation: https://pypi.org/project/google-cloud-translate/
+.. _Translation README: https://github.com/googleapis/google-cloud-python/tree/master/translate
+.. _Google Cloud Scheduler: https://pypi.org/project/google-cloud-scheduler/
+.. _Scheduler README: https://github.com/googleapis/google-cloud-python/tree/master/scheduler
+.. _Stackdriver Logging: https://pypi.org/project/google-cloud-logging/
+.. _Logging README: https://github.com/googleapis/google-cloud-python/tree/master/logging
+
+Beta Support
+------------
-.. _Google Cloud Datastore: https://github.com/GoogleCloudPlatform/google-cloud-python#google-cloud-datastore
-.. _Google Cloud Storage: https://github.com/GoogleCloudPlatform/google-cloud-python#google-cloud-storage
-.. _Google Cloud Pub/Sub: https://github.com/GoogleCloudPlatform/google-cloud-python#google-cloud-pubsub
-.. _Google BigQuery: https://github.com/GoogleCloudPlatform/google-cloud-python#google-bigquery
-.. _Google Cloud Resource Manager: https://github.com/GoogleCloudPlatform/google-cloud-python#google-cloud-resource-manager
-.. _Google Stackdriver Logging: https://github.com/GoogleCloudPlatform/google-cloud-python#google-stackdriver-logging
-.. _Google Stackdriver Monitoring: https://github.com/GoogleCloudPlatform/google-cloud-python#google-stackdriver-monitoring
+**Beta** indicates that the client library for a particular service is
+mostly stable and is being prepared for release. Issues and requests
+against beta libraries are addressed with a higher priority.
+
+The following client libraries have **beta** support:
+
+- `Google Cloud Bigtable`_ (`Bigtable README`_)
+- `Google Cloud Firestore`_ (`Firestore README`_)
+- `Google Cloud Pub/Sub`_ (`Pub/Sub README`_)
+- `Google Cloud Video Intelligence`_ (`Video Intelligence README`_)
+- `Google Cloud Vision`_ (`Vision README`_)
+
+.. _Google Cloud Bigtable: https://pypi.org/project/google-cloud-bigtable/
+.. _Bigtable README: https://github.com/googleapis/google-cloud-python/tree/master/bigtable
+.. _Google Cloud Firestore: https://pypi.org/project/google-cloud-firestore/
+.. _Firestore README: https://github.com/googleapis/google-cloud-python/tree/master/firestore
+.. _Google Cloud Pub/Sub: https://pypi.org/project/google-cloud-pubsub/
+.. _Pub/Sub README: https://github.com/googleapis/google-cloud-python/tree/master/pubsub
+.. _Google Cloud Video Intelligence: https://pypi.org/project/google-cloud-videointelligence
+.. _Video Intelligence README: https://github.com/googleapis/google-cloud-python/tree/master/videointelligence
+.. _Google Cloud Vision: https://pypi.org/project/google-cloud-vision/
+.. _Vision README: https://github.com/googleapis/google-cloud-python/tree/master/vision
+
+
+Alpha Support
+-------------
+
+**Alpha** indicates that the client library for a particular service is
+still a work-in-progress and is more likely to get backwards-incompatible
+updates. See `versioning`_ for more details.
+
+The following client libraries have **alpha** support:
+
+- `Google Cloud Asset`_ (`Asset README`_)
+- `Google Cloud AutoML`_ (`AutoML README`_)
+- `Google BigQuery Data Transfer`_ (`BigQuery Data Transfer README`_)
+- `Google Cloud Bigtable - HappyBase`_ (`HappyBase README`_)
+- `Google Cloud Container`_ (`Container README`_)
+- `Google Cloud Container Analysis`_ (`Container Analysis README`_)
+- `Google Cloud Dataproc`_ (`Dataproc README`_)
+- `Google Cloud DLP`_ (`DLP README`_)
+- `Google Cloud DNS`_ (`DNS README`_)
+- `Google Cloud IoT`_ (`IoT README`_)
+- `Google Cloud Memorystore for Redis`_ (`Redis README`_)
+- `Google Cloud Resource Manager`_ (`Resource Manager README`_)
+- `Google Cloud Runtime Configuration`_ (`Runtime Config README`_)
+- `Google Cloud Security Scanner`_ (`Security Scanner README`_ )
+- `Google Cloud Trace`_ (`Trace README`_)
+- `Google Cloud Text-to-Speech`_ (`Text-to-Speech README`_)
+- `Grafeas`_ (`Grafeas README`_)
+- `Stackdriver Error Reporting`_ (`Error Reporting README`_)
+- `Stackdriver Monitoring`_ (`Monitoring README`_)
+
+.. _Google Cloud Asset: https://pypi.org/project/google-cloud-asset/
+.. _Asset README: https://github.com/googleapis/google-cloud-python/blob/master/asset
+.. _Google Cloud AutoML: https://pypi.org/project/google-cloud-automl/
+.. _AutoML README: https://github.com/googleapis/google-cloud-python/blob/master/automl
+.. _Google BigQuery Data Transfer: https://pypi.org/project/google-cloud-bigquery-datatransfer/
+.. _BigQuery Data Transfer README: https://github.com/googleapis/google-cloud-python/tree/master/bigquery_datatransfer
+.. _Google Cloud Bigtable - HappyBase: https://pypi.org/project/google-cloud-happybase/
+.. _HappyBase README: https://github.com/googleapis/google-cloud-python-happybase
+.. _Google Cloud Container: https://pypi.org/project/google-cloud-container/
+.. _Container README: https://github.com/googleapis/google-cloud-python/tree/master/container
+.. _Google Cloud Container Analysis: https://pypi.org/project/google-cloud-containeranalysis/
+.. _Container Analysis README: https://github.com/googleapis/google-cloud-python/tree/master/containeranalysis
+.. _Google Cloud Dataproc: https://pypi.org/project/google-cloud-dataproc/
+.. _Dataproc README: https://github.com/googleapis/google-cloud-python/tree/master/dataproc
+.. _Google Cloud DLP: https://pypi.org/project/google-cloud-dlp/
+.. _DLP README: https://github.com/googleapis/google-cloud-python/tree/master/dlp
+.. _Google Cloud DNS: https://pypi.org/project/google-cloud-dns/
+.. _DNS README: https://github.com/googleapis/google-cloud-python/tree/master/dns
+.. _Google Cloud IoT: https://pypi.org/project/google-cloud-iot/
+.. _IoT README: https://github.com/googleapis/google-cloud-python/tree/master/iot
+.. _Google Cloud Memorystore for Redis: https://pypi.org/project/google-cloud-redis/
+.. _Redis README: https://github.com/googleapis/google-cloud-python/tree/master/redis
+.. _Google Cloud Resource Manager: https://pypi.org/project/google-cloud-resource-manager/
+.. _Resource Manager README: https://github.com/googleapis/google-cloud-python/tree/master/resource_manager
+.. _Google Cloud Runtime Configuration: https://pypi.org/project/google-cloud-runtimeconfig/
+.. _Runtime Config README: https://github.com/googleapis/google-cloud-python/tree/master/runtimeconfig
+.. _Google Cloud Security Scanner: https://pypi.org/project/google-cloud-websecurityscanner/
+.. _Security Scanner README: https://github.com/googleapis/google-cloud-python/blob/master/websecurityscanner
+.. _Google Cloud Text-to-Speech: https://pypi.org/project/google-cloud-texttospeech/
+.. _Text-to-Speech README: https://github.com/googleapis/google-cloud-python/tree/master/texttospeech
+.. _Google Cloud Trace: https://pypi.org/project/google-cloud-trace/
+.. _Trace README: https://github.com/googleapis/google-cloud-python/tree/master/trace
+.. _Grafeas: https://pypi.org/project/grafeas/
+.. _Grafeas README: https://github.com/googleapis/google-cloud-python/tree/master/grafeas
+.. _Stackdriver Error Reporting: https://pypi.org/project/google-cloud-error-reporting/
+.. _Error Reporting README: https://github.com/googleapis/google-cloud-python/tree/master/error_reporting
+.. _Stackdriver Monitoring: https://pypi.org/project/google-cloud-monitoring/
+.. _Monitoring README: https://github.com/googleapis/google-cloud-python/tree/master/monitoring
+
+.. _versioning: https://github.com/googleapis/google-cloud-python/blob/master/CONTRIBUTING.rst#versioning
If you need support for other Google APIs, check out the
`Google APIs Python Client library`_.
.. _Google APIs Python Client library: https://github.com/google/google-api-python-client
-Quick Start
------------
-
-::
-
- $ pip install --upgrade google-cloud
Example Applications
--------------------
@@ -53,6 +176,7 @@ Example Applications
.. _tutorial: https://cloud.google.com/python
.. _google-cloud-python-expenses-demo: https://github.com/GoogleCloudPlatform/google-cloud-python-expenses-demo
+
Authentication
--------------
@@ -61,276 +185,32 @@ Check out the `Authentication section`_ in our documentation to learn more.
You may also find the `authentication document`_ shared by all the
``google-cloud-*`` libraries to be helpful.
-.. _Authentication section: http://google-cloud-python.readthedocs.io/en/latest/google-cloud-auth.html
-.. _authentication document: https://github.com/GoogleCloudPlatform/gcloud-common/tree/master/authentication
-
-Google Cloud Datastore
-----------------------
-
-Google `Cloud Datastore`_ (`Datastore API docs`_) is a fully managed, schemaless
-database for storing non-relational data. Cloud Datastore automatically scales
-with your users and supports ACID transactions, high availability of reads and
-writes, strong consistency for reads and ancestor queries, and eventual
-consistency for all other queries.
-
-.. _Cloud Datastore: https://cloud.google.com/datastore/docs
-.. _Datastore API docs: https://cloud.google.com/datastore/docs/
-
-See the ``google-cloud-python`` API `datastore documentation`_ to learn how to
-interact with the Cloud Datastore using this Client Library.
-
-.. _datastore documentation: https://googlecloudplatform.github.io/google-cloud-python/stable/datastore-client.html
-
-See the `official Google Cloud Datastore documentation`_ for more details on how
-to activate Cloud Datastore for your project.
-
-.. _official Google Cloud Datastore documentation: https://cloud.google.com/datastore/docs/activate
-
-.. code:: python
-
- from google.cloud import datastore
- # Create, populate and persist an entity
- entity = datastore.Entity(key=datastore.Key('EntityKind'))
- entity.update({
- 'foo': u'bar',
- 'baz': 1337,
- 'qux': False,
- })
- # Then query for entities
- query = datastore.Query(kind='EntityKind')
- for result in query.fetch():
- print result
-
-Google Cloud Storage
---------------------
-
-Google `Cloud Storage`_ (`Storage API docs`_) allows you to store data on Google
-infrastructure with very high reliability, performance and availability, and can
-be used to distribute large data objects to users via direct download.
-
-.. _Cloud Storage: https://cloud.google.com/storage/docs
-.. _Storage API docs: https://cloud.google.com/storage/docs/json_api/v1
-
-See the ``google-cloud-python`` API `storage documentation`_ to learn how to connect
-to Cloud Storage using this Client Library.
-
-.. _storage documentation: https://googlecloudplatform.github.io/google-cloud-python/stable/storage-client.html
-
-You need to create a Google Cloud Storage bucket to use this client library.
-Follow along with the `official Google Cloud Storage documentation`_ to learn
-how to create a bucket.
-
-.. _official Google Cloud Storage documentation: https://cloud.google.com/storage/docs/cloud-console#_creatingbuckets
-
-.. code:: python
-
- from google.cloud import storage
- client = storage.Client()
- bucket = client.get_bucket('bucket-id-here')
- # Then do other things...
- blob = bucket.get_blob('remote/path/to/file.txt')
- print blob.download_as_string()
- blob.upload_from_string('New contents!')
- blob2 = bucket.blob('remote/path/storage.txt')
- blob2.upload_from_filename(filename='/local/path.txt')
-
-Google Cloud Pub/Sub
---------------------
-
-Google `Cloud Pub/Sub`_ (`Pub/Sub API docs`_) is designed to provide reliable,
-many-to-many, asynchronous messaging between applications. Publisher
-applications can send messages to a ``topic`` and other applications can
-subscribe to that topic to receive the messages. By decoupling senders and
-receivers, Google Cloud Pub/Sub allows developers to communicate between
-independently written applications.
-
-.. _Cloud Pub/Sub: https://cloud.google.com/pubsub/docs
-.. _Pub/Sub API docs: https://cloud.google.com/pubsub/reference/rest/
-
-See the ``google-cloud-python`` API `Pub/Sub documentation`_ to learn how to connect
-to Cloud Pub/Sub using this Client Library.
-
-.. _Pub/Sub documentation: https://googlecloudplatform.github.io/google-cloud-python/stable/pubsub-usage.html
-
-To get started with this API, you'll need to create
-
-.. code:: python
-
- from google.cloud import pubsub
-
- client = pubsub.Client()
- topic = client.topic('topic_name')
- topic.create()
-
- topic.publish('this is the message_payload',
- attr1='value1', attr2='value2')
-
-Google BigQuery
----------------
-
-Querying massive datasets can be time consuming and expensive without the
-right hardware and infrastructure. Google `BigQuery`_ (`BigQuery API docs`_)
-solves this problem by enabling super-fast, SQL-like queries against
-append-only tables, using the processing power of Google's infrastructure.
-
-.. _BigQuery: https://cloud.google.com/bigquery/what-is-bigquery
-.. _BigQuery API docs: https://cloud.google.com/bigquery/docs/reference/v2/
-
-This package is still being implemented, but it is almost complete!
-
-Load data from CSV
-~~~~~~~~~~~~~~~~~~
-
-.. code:: python
+.. _Authentication section: https://google-cloud-python.readthedocs.io/en/latest/core/auth.html
+.. _authentication document: https://github.com/googleapis/google-cloud-common/tree/master/authentication
- import csv
-
- from google.cloud import bigquery
- from google.cloud.bigquery import SchemaField
-
- client = bigquery.Client()
-
- dataset = client.dataset('dataset_name')
- dataset.create() # API request
-
- SCHEMA = [
- SchemaField('full_name', 'STRING', mode='required'),
- SchemaField('age', 'INTEGER', mode='required'),
- ]
- table = dataset.table('table_name', SCHEMA)
- table.create()
-
- with open('csv_file', 'rb') as readable:
- table.upload_from_file(
- readable, source_format='CSV', skip_leading_rows=1)
-
-Perform a synchronous query
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-.. code:: python
-
- # Perform a synchronous query.
- QUERY = (
- 'SELECT name FROM [bigquery-public-data:usa_names.usa_1910_2013] '
- 'WHERE state = "TX"')
- query = client.run_sync_query('%s LIMIT 100' % QUERY)
- query.timeout_ms = TIMEOUT_MS
- query.run()
-
- for row in query.rows:
- print row
-
-
-See the ``google-cloud-python`` API `BigQuery documentation`_ to learn how to connect
-to BigQuery using this Client Library.
-
-.. _BigQuery documentation: https://googlecloudplatform.github.io/google-cloud-python/stable/bigquery-usage.html
-
-Google Cloud Resource Manager
------------------------------
-
-The Cloud `Resource Manager`_ API (`Resource Manager API docs`_) provides
-methods that you can use to programmatically manage your projects in the
-Google Cloud Platform.
-
-.. _Resource Manager: https://cloud.google.com/resource-manager/
-.. _Resource Manager API docs: https://cloud.google.com/resource-manager/reference/rest/
-
-See the ``google-cloud-python`` API `Resource Manager documentation`_ to learn how to
-manage projects using this Client Library.
-
-.. _Resource Manager documentation: https://googlecloudplatform.github.io/google-cloud-python/stable/resource-manager-api.html
-
-Google Stackdriver Logging
---------------------------
-
-`Stackdriver Logging`_ API (`Logging API docs`_) allows you to store, search,
-analyze, monitor, and alert on log data and events from Google Cloud Platform.
-
-.. _Stackdriver Logging: https://cloud.google.com/logging/
-.. _Logging API docs: https://cloud.google.com/logging/docs/
-
-.. code:: python
-
- from google.cloud import logging
- client = logging.Client()
- logger = client.logger('log_name')
- logger.log_text("A simple entry") # API call
-
-Example of fetching entries:
-
-.. code:: python
-
- entries, token = logger.list_entries()
- for entry in entries:
- print entry.payload
-
-See the ``google-cloud-python`` API `logging documentation`_ to learn how to connect
-to Stackdriver Logging using this Client Library.
-
-.. _logging documentation: https://googlecloudplatform.github.io/google-cloud-python/stable/logging-usage.html
-
-Google Stackdriver Monitoring
------------------------------
-
-`Stackdriver Monitoring`_ (`Monitoring API docs`_) collects metrics,
-events, and metadata from Google Cloud Platform, Amazon Web Services (AWS),
-hosted uptime probes, application instrumentation, and a variety of common
-application components including Cassandra, Nginx, Apache Web Server,
-Elasticsearch and many others. Stackdriver ingests that data and generates
-insights via dashboards, charts, and alerts.
-
-This package currently supports all Monitoring API operations other than
-writing custom metrics.
-
-.. _Stackdriver Monitoring: https://cloud.google.com/monitoring/
-.. _Monitoring API docs: https://cloud.google.com/monitoring/api/ref_v3/rest/
-
-List available metric types:
-
-.. code:: python
-
- from google.cloud import monitoring
- client = monitoring.Client()
- for descriptor in client.list_metric_descriptors():
- print(descriptor.type)
-
-Display CPU utilization across your GCE instances during the last five minutes:
+Contributing
+------------
-.. code:: python
+Contributions to this library are always welcome and highly encouraged.
- metric = 'compute.googleapis.com/instance/cpu/utilization'
- query = client.query(metric, minutes=5)
- print(query.as_dataframe())
+See the `CONTRIBUTING doc`_ for more information on how to get started.
-See the ``google-cloud-python`` API `monitoring documentation`_ to learn how to connect
-to Stackdriver Monitoring using this Client Library.
+.. _CONTRIBUTING doc: https://github.com/googleapis/google-cloud-python/blob/master/CONTRIBUTING.rst
-.. _monitoring documentation: https://googlecloudplatform.github.io/google-cloud-python/stable/monitoring-usage.html
-Contributing
-------------
+Community
+---------
-Contributions to this library are always welcome and highly encouraged.
+Google Cloud Platform Python developers hang out in `Slack`_ in the ``#python``
+channel, click here to `get an invitation`_.
-See `CONTRIBUTING`_ for more information on how to get started.
+.. _Slack: https://googlecloud-community.slack.com
+.. _get an invitation: https://gcp-slack.appspot.com/
-.. _CONTRIBUTING: https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/CONTRIBUTING.rst
License
-------
-Apache 2.0 - See `LICENSE`_ for more information.
-
-.. _LICENSE: https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/LICENSE
+Apache 2.0 - See `the LICENSE`_ for more information.
-.. |build| image:: https://travis-ci.org/GoogleCloudPlatform/google-cloud-python.svg?branch=master
- :target: https://travis-ci.org/GoogleCloudPlatform/google-cloud-python
-.. |appveyor| image:: https://ci.appveyor.com/api/projects/status/github/googlecloudplatform/google-cloud-python?branch=master&svg=true
- :target: https://ci.appveyor.com/project/GoogleCloudPlatform/google-cloud-python
-.. |coverage| image:: https://coveralls.io/repos/GoogleCloudPlatform/google-cloud-python/badge.png?branch=master
- :target: https://coveralls.io/r/GoogleCloudPlatform/google-cloud-python?branch=master
-.. |pypi| image:: https://img.shields.io/pypi/v/google-cloud.svg
- :target: https://pypi.python.org/pypi/google-cloud
-.. |versions| image:: https://img.shields.io/pypi/pyversions/google-cloud.svg
- :target: https://pypi.python.org/pypi/google-cloud
+.. _the LICENSE: https://github.com/googleapis/google-cloud-python/blob/master/LICENSE
diff --git a/api_core/.coveragerc b/api_core/.coveragerc
new file mode 100644
index 000000000000..d097511c3124
--- /dev/null
+++ b/api_core/.coveragerc
@@ -0,0 +1,13 @@
+[run]
+branch = True
+
+[report]
+fail_under = 100
+show_missing = True
+exclude_lines =
+ # Re-enable the standard pragma
+ pragma: NO COVER
+ # Ignore debug-only repr
+ def __repr__
+ # Ignore abstract methods
+ raise NotImplementedError
diff --git a/api_core/.flake8 b/api_core/.flake8
new file mode 100644
index 000000000000..3da787cdfbf3
--- /dev/null
+++ b/api_core/.flake8
@@ -0,0 +1,11 @@
+[flake8]
+import-order-style=google
+# Note: this forces all google imports to be in the third group. See
+# https://github.com/PyCQA/flake8-import-order/issues/111
+application-import-names=google
+ignore = E203, E266, E501, W503
+exclude =
+ __pycache__,
+ .git,
+ *.pyc,
+ conf.py
diff --git a/api_core/.repo-metadata.json b/api_core/.repo-metadata.json
new file mode 100644
index 000000000000..6a5f3f53359f
--- /dev/null
+++ b/api_core/.repo-metadata.json
@@ -0,0 +1,9 @@
+{
+ "name": "google-api-core",
+ "name_pretty": "Google API client core library",
+ "client_documentation": "https://googleapis.dev/python/google-api-core/latest",
+ "release_level": "ga",
+ "language": "python",
+ "repo": "googleapis/google-cloud-python",
+ "distribution_name": "google-api-core"
+}
\ No newline at end of file
diff --git a/api_core/CHANGELOG.md b/api_core/CHANGELOG.md
new file mode 100644
index 000000000000..74ab16564dbc
--- /dev/null
+++ b/api_core/CHANGELOG.md
@@ -0,0 +1,349 @@
+# Changelog
+
+[PyPI History][1]
+
+[1]: https://pypi.org/project/google-api-core/#history
+
+## 1.14.2
+
+07-30-2019 14:08 PDT
+
+
+### Documentation
+- Add client_options documentation. ([#8834](https://github.com/googleapis/google-cloud-python/pull/8834))
+
+## 1.14.1
+
+07-30-2019 12:24 PDT
+
+
+### Implementation Changes
+- Remove error log entry on clean BiDi shutdown. ([#8806](https://github.com/googleapis/google-cloud-python/pull/8806))
+- Forward 'timeout' arg from 'exception' to `_blocking_poll`. ([#8735](https://github.com/googleapis/google-cloud-python/pull/8735))
+
+### Documentation
+- Update intersphinx mapping for requests. ([#8805](https://github.com/googleapis/google-cloud-python/pull/8805))
+- Link to googleapis.dev documentation in READMEs. ([#8705](https://github.com/googleapis/google-cloud-python/pull/8705))
+
+## 1.14.0
+
+07-17-2019 13:16 PDT
+
+
+### New Features
+- Firestore: Add `should_terminate` predicate for clean BiDi shutdown. ([#8650](https://github.com/googleapis/google-cloud-python/pull/8650))
+
+### Dependencies
+- Update pins of 'googleapis-common-protos. ([#8688](https://github.com/googleapis/google-cloud-python/pull/8688))
+
+### Documentation
+- Add compatibility check badges to READMEs. ([#8288](https://github.com/googleapis/google-cloud-python/pull/8288))
+
+### Internal / Testing Changes
+- All: Add docs job to publish to googleapis.dev. ([#8464](https://github.com/googleapis/google-cloud-python/pull/8464))
+
+## 1.13.0
+
+06-24-2019 10:34 PDT
+
+### New Features
+- Add `client_options.ClientOptions` object. ([#8265](https://github.com/googleapis/google-cloud-python/pull/8265))
+
+## 1.12.0
+
+06-18-2019 12:37 PDT
+
+
+### New Features
+- Add Throttling to Bidi Reopening. Mitigates ResumableBidiRpc consuming 100% CPU ([#8193](https://github.com/googleapis/google-cloud-python/pull/8193))
+
+## 1.11.1
+
+05-28-2019 11:19 PDT
+
+
+### Implementation Changes
+- Classify 503 Service Unavailable errors as transient. ([#8182](https://github.com/googleapis/google-cloud-python/pull/8182))
+
+### Dependencies
+- Pin `grpcio < 2.0dev`. ([#8182](https://github.com/googleapis/google-cloud-python/pull/8182))
+
+### Internal / Testing Changes
+- Add parameterized test for `from_rfc3339` with nanos ([#7675](https://github.com/googleapis/google-cloud-python/pull/7675))
+- Unbreak pytype by silencing a false positive. ([#8106](https://github.com/googleapis/google-cloud-python/pull/8106))
+
+## 1.11.0
+
+05-15-2019 10:29 PDT
+
+### New Features
+
+- Refactor 'client_info' support. ([#7849](https://github.com/googleapis/google-cloud-python/pull/7849))
+
+## 1.10.0
+
+04-29-2019 10:12 PDT
+
+### Implementation Changes
+
+- Append leading zeros for nanosecond precision DateTimes
+ ([#7663](https://github.com/googleapis/google-cloud-python/pull/7663))
+
+### New Features
+
+- Add `user_agent` property to `ClientInfo`
+ ([#7799](https://github.com/googleapis/google-cloud-python/pull/7799))
+
+## 1.9.0
+
+04-05-2019 10:38 PDT
+
+
+### Implementation Changes
+- Allow passing metadata as part of creating a bidi ([#7514](https://github.com/googleapis/google-cloud-python/pull/7514))
+
+### Internal / Testing Changes
+- Update setup.py
+- API Core: specify a pytype output directory in setup.cfg. ([#7639](https://github.com/googleapis/google-cloud-python/pull/7639))
+
+## 1.8.2
+
+03-22-2019 16:27 PDT
+
+
+### Implementation Changes
+- Remove classifier for Python 3.4 for end-of-life. ([#7535](https://github.com/googleapis/google-cloud-python/pull/7535))
+
+### Internal / Testing Changes
+- When re-opening a `ResumableBidiRPC` set `_request_queue_generator` to `None`. ([#7548](https://github.com/googleapis/google-cloud-python/pull/7548))
+
+## 1.8.1
+
+03-12-2019 12:45 PDT
+
+### Implementation Changes
+- Protect the creation of a background thread in BackgroundConsumer and wait on it starting. ([#7499](https://github.com/googleapis/google-cloud-python/pull/7499))
+
+## 1.8.0
+
+02-23-2019 15:46 PST
+
+
+### New Features
+- Add support to unwrap Anys into wrapped pb2 objects. ([#7430](https://github.com/googleapis/google-cloud-python/pull/7430))
+- Add `Operation.deserialize`. ([#7427](https://github.com/googleapis/google-cloud-python/pull/7427))
+
+### Documentation
+- Updated client library documentation URLs. ([#7307](https://github.com/googleapis/google-cloud-python/pull/7307))
+
+### Internal / Testing Changes
+- Fix new lint failure. ([#7382](https://github.com/googleapis/google-cloud-python/pull/7382))
+
+## 1.7.0
+
+12-17-2018 13:56 PST
+
+### New Features
+- Support converting `DatetimeWithNanos` to / from `google.protobuf.timestamp_pb2.Timestamp`. ([#6919](https://github.com/googleapis/google-cloud-python/pull/6919))
+
+### Documentation
+- Document Python 2 deprecation. ([#6910](https://github.com/googleapis/google-cloud-python/pull/6910))
+- Add usage example for `google.api_core.iam.Polcy`. ([#6855](https://github.com/googleapis/google-cloud-python/pull/6855))
+
+### Internal / Testing Changes
+- Work around pytype big for `ABCMeta.register`. ([#6873](https://github.com/googleapis/google-cloud-python/pull/6873))
+
+## 1.6.0
+
+11-30-2018 12:45 PST
+
+
+### Implementation Changes
+- Import stdlib ABCs from 'collections.abc' rather than 'collections'. ([#6451](https://github.com/googleapis/google-cloud-python/pull/6451))
+
+### New Features
+- Move google.cloud.iam (core) to google.api_core.iam ([#6740](https://github.com/googleapis/google-cloud-python/pull/6740))
+- Add bidi support to api_core. ([#6191](https://github.com/googleapis/google-cloud-python/pull/6191))
+
+### Documentation
+- Fix typo ([#6532](https://github.com/googleapis/google-cloud-python/pull/6532))
+
+### Internal / Testing Changes
+- blacken api_core and core ([#6668](https://github.com/googleapis/google-cloud-python/pull/6668))
+
+## 1.5.2
+
+11-09-2018 14:22 PST
+
+
+### Implementation Changes
+- Retry transient errors in 'PollingFuture.result'. ([#6305](https://github.com/googleapis/google-cloud-python/pull/6305))
+
+### Dependencies
+- Remove hyphen from named extra in api_core. ([#6468](https://github.com/googleapis/google-cloud-python/pull/6468))
+- Bump minimum 'api_core' version for all GAPIC libs to 1.4.1. ([#6391](https://github.com/googleapis/google-cloud-python/pull/6391))
+- Avoid broken 'google-common-apis 1.5.4' release. ([#6355](https://github.com/googleapis/google-cloud-python/pull/6355))
+
+## 1.5.1
+
+10-29-2018 13:29 PDT
+
+### Implementation Changes
+- Don't URL-encode slashes in gRPC request headers. ([#6310](https://github.com/googleapis/google-cloud-python/pull/6310))
+
+### Internal / Testing Changes
+- Back out changes from [#6267](https://github.com/googleapis/google-cloud-python/pull/6267) / `api_core-1.6.0a1` release. ([#6328](https://github.com/googleapis/google-cloud-python/pull/6328))
+
+## 1.5.0
+
+### New Features
+- Add bidi, Bidirection Streaming, to api-core ([#6211](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/6211))
+
+### Internal / Testing Changes
+- Use new Nox ([#6175](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/6175))
+
+## 1.4.1
+
+### Dependencies
+- Pin minimum protobuf dependency to 3.4.0. ([#6132](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/6132))
+
+### Internal / Testing Changes
+- Add type-checking via pytype to api_core. ([#6116](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/6116))
+
+## 1.4.0
+
+### Dependencies
+
+- Add support for gRPC connection management (available when using optional grpc_gcp dependency) ([#5553](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/5553)) ([#5904](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/5904))
+- Update classifiers to drop Python 3.4 and add Python 3.7 ([#5702](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/5702))
+
+## 1.3.0
+
+### New Features
+
+- Add protobuf_helpers.field_mask to calculate a field mask from two messages (#5320)
+
+## 1.2.1
+
+### Implementation Changes
+- Make client_info work without gRPC installed. (#5075)
+- Rename `x-goog-header-params` to `x-goog-request-params` (#5495)
+
+## 1.2.0
+
+### Implementation Changes
+- Add close method to grpc Channel (#5333)
+
+### Internal / Testing Changes
+- Fix tests after grpcio update (#5333)
+- Add Test runs for Python 3.7 and remove 3.4 (#5295)
+
+## 1.1.2
+
+### Packaging
+- Update setuptools before packaging (#5265)
+
+## 1.1.1
+
+### Internal / Testing Changes
+- Use `install_requires` for platform dependencies instead of `extras_require` (#4991)
+- Update trove classifer to '5 - Production/Stable'
+
+## 1.1.0
+
+### Interface additions
+
+- Add `datetime_helpers.DatetimeWithNanoSeconds` (#4979)
+
+### Implementation changes
+
+- Use a class to wrap grpc streaming errors instead of monkey-patching (#4995)
+
+## 1.0.0
+
+This is the stable v1.0.0 release of google-api-core for Python. Releases after
+this will not contain breaking changes.
+
+### Interface changes and additions
+
+- Made `api_core.page_iterator.PageIterator.item_to_value` public
+- Added ability to specify retry for `Operation` and `polling.Future`. (#4922)
+
+## 0.1.4
+
+### New Features
+
+- Add `ChannelStub` to `grpc_helpers` for testing gRPC-based clients. (#4705)
+
+### Notable Implementation Changes
+
+- Fix handling of gapic metadata when specified as `None`. (#4701)
+
+## 0.1.3
+
+### Notable Implementation Changes
+
+- Apply scopes to explicitly provided credentials if needed (#4594).
+- Removing `google.api_core.gapic_v1.method.METRICS_METADATA_KEY`. It
+ can be accessed via
+ `google.api_core.gapic_v1.client_info.METRICS_METADATA_KEY` (#4588).
+
+### Dependencies
+
+- Upgrading to latest `grpcio==1.8.2` (#4642). For details, see
+ related gRPC [bug](https://github.com/grpc/grpc/issues/9688)
+ and [fix](https://github.com/grpc/grpc/pull/13665).
+
+PyPI: https://pypi.org/project/google-api-core/0.1.3/
+
+## 0.1.2
+
+- Upgrading `concurrent.futures` backport from `>= 3.0.0`
+ to `>= 3.2.0` (#4521).
+- Moved `datetime`-related helpers from `google.cloud.core` to
+ `google.api_core.datetime_helpers` (#4399).
+- Added missing `client_info` to `gapic_v1/__init__.py`'s
+ `__all__` (#4567).
+- Added helpers for routing headers to `gapic_v1` (#4336).
+
+PyPI: https://pypi.org/project/google-api-core/0.1.2/
+
+## 0.1.1
+
+### Dependencies
+
+- Upgrading `grpcio` dependency from `1.2.0, < 1.6dev` to `>= 1.7.0` (#4280)
+
+PyPI: https://pypi.org/project/google-api-core/0.1.1/
+
+## 0.1.0
+
+Initial release
+
+Prior to being separated, this package was developed in `google-cloud-core`, so
+relevant changes from that package are included here.
+
+- Add google.api.core.gapic_v1.config (#4022)
+- Add google.api.core.helpers.grpc_helpers (#4041)
+- Add google.api.core.gapic_v1.method (#4057)
+- Add wrap_with_paging (#4067)
+- Add grpc_helpers.create_channel (#4069)
+- Add DEFAULT sentinel for gapic_v1.method (#4079)
+- Remove `googleapis-common-protos` from deps in non-`core` packages. (#4098)
+- Add google.api.core.operations_v1 (#4081)
+- Fix test assertion in test_wrap_method_with_overriding_retry_deadline (#4131)
+- Add google.api.core.helpers.general_helpers.wraps (#4166)
+- Update Docs with Python Setup Guide (#4187)
+- Move modules in google.api.core.helpers up one level, delete google.api.core.helpers. (#4196)
+- Clarify that PollingFuture timeout is in seconds. (#4201)
+- Add api_core package (#4210)
+- Replace usage of google.api.core with google.api_core (#4221)
+- Add google.api_core.gapic_v2.client_info (#4225)
+- Fix how api_core.operation populates exception errors (#4231)
+- Fix bare except (#4250)
+- Fix parsing of API errors with Unicode err message (#4251)
+- Port gax proto helper methods (#4249)
+- Remove gapic_v1.method.wrap_with_paging (#4257)
+- Add final set of protobuf helpers to api_core (#4259)
+
+PyPI: https://pypi.org/project/google-api-core/0.1.0/
diff --git a/api_core/LICENSE b/api_core/LICENSE
new file mode 100644
index 000000000000..d64569567334
--- /dev/null
+++ b/api_core/LICENSE
@@ -0,0 +1,202 @@
+
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [yyyy] [name of copyright owner]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
diff --git a/api_core/MANIFEST.in b/api_core/MANIFEST.in
new file mode 100644
index 000000000000..1fbc0d0b321e
--- /dev/null
+++ b/api_core/MANIFEST.in
@@ -0,0 +1,3 @@
+include README.rst LICENSE
+recursive-include tests *
+global-exclude *.pyc __pycache__
diff --git a/api_core/README.rst b/api_core/README.rst
new file mode 100644
index 000000000000..244043ea61b3
--- /dev/null
+++ b/api_core/README.rst
@@ -0,0 +1,23 @@
+Core Library for Google Client Libraries
+========================================
+
+|pypi| |versions|
+
+This library is not meant to stand-alone. Instead it defines
+common helpers used by all Google API clients. For more information, see the
+`documentation`_.
+
+.. |pypi| image:: https://img.shields.io/pypi/v/google-api_core.svg
+ :target: https://pypi.org/project/google-api_core/
+.. |versions| image:: https://img.shields.io/pypi/pyversions/google-api_core.svg
+ :target: https://pypi.org/project/google-api_core/
+.. _documentation: https://googleapis.dev/python/google-api-core/latest
+
+
+Supported Python Versions
+-------------------------
+Python >= 3.5
+
+Deprecated Python Versions
+--------------------------
+Python == 2.7. Python 2.7 support will be removed on January 1, 2020.
diff --git a/api_core/docs/auth.rst b/api_core/docs/auth.rst
new file mode 100644
index 000000000000..f309a3a018ec
--- /dev/null
+++ b/api_core/docs/auth.rst
@@ -0,0 +1,301 @@
+Authentication
+**************
+
+.. _Overview:
+
+Overview
+========
+
+* **If you're running in Compute Engine or App Engine**,
+ authentication should "just work".
+
+* **If you're developing locally**,
+ the easiest way to authenticate is using the `Google Cloud SDK`_:
+
+ .. code-block:: bash
+
+ $ gcloud auth application-default login
+
+ Note that this command generates credentials for client libraries. To authenticate the CLI itself, use:
+
+ .. code-block:: bash
+
+ $ gcloud auth login
+
+ Previously, ``gcloud auth login`` was used for both use cases. If
+ your ``gcloud`` installation does not support the new command,
+ please update it:
+
+ .. code-block:: bash
+
+ $ gcloud components update
+
+.. _Google Cloud SDK: http://cloud.google.com/sdk
+
+
+* **If you're running your application elsewhere**,
+ you should download a `service account`_ JSON keyfile
+ and point to it using an environment variable:
+
+ .. code-block:: bash
+
+ $ export GOOGLE_APPLICATION_CREDENTIALS="/path/to/keyfile.json"
+
+.. _service account: https://cloud.google.com/storage/docs/authentication#generating-a-private-key
+
+Client-Provided Authentication
+==============================
+
+Every package uses a :class:`Client `
+as a base for interacting with an API.
+For example:
+
+.. code-block:: python
+
+ from google.cloud import datastore
+ client = datastore.Client()
+
+Passing no arguments at all will "just work" if you've followed the
+instructions in the :ref:`Overview`.
+The credentials are inferred from your local environment by using
+Google `Application Default Credentials`_.
+
+.. _Application Default Credentials: https://developers.google.com/identity/protocols/application-default-credentials
+
+.. _Precedence:
+
+Credential Discovery Precedence
+-------------------------------
+
+When loading the `Application Default Credentials`_,
+the library will check for credentials in your environment by following the
+precedence outlined by :func:`google.auth.default`.
+
+Explicit Credentials
+====================
+
+The Application Default Credentials discussed above can be useful
+if your code needs to run in many different environments or
+if you just don't want authentication to be a focus in your code.
+
+However, you may want to be explicit because
+
+* your code will only run in one place
+* you may have code which needs to be run as a specific service account
+ every time (rather than with the locally inferred credentials)
+* you may want to use two separate accounts to simultaneously access data
+ from different projects
+
+In these situations, you can create an explicit
+:class:`~google.auth.credentials.Credentials` object suited to your environment.
+After creation, you can pass it directly to a :class:`Client `:
+
+.. code:: python
+
+ client = Client(credentials=credentials)
+
+.. tip::
+ To create a credentials object, follow the `google-auth-guide`_.
+
+.. _google-auth-guide: https://google-auth.readthedocs.io/en/latest/user-guide.html#service-account-private-key-files
+
+
+Google App Engine Environment
+-----------------------------
+
+To create
+:class:`credentials `
+just for Google App Engine:
+
+.. code:: python
+
+ from google.auth import app_engine
+ credentials = app_engine.Credentials()
+
+Google Compute Engine Environment
+---------------------------------
+
+To create
+:class:`credentials `
+just for Google Compute Engine:
+
+.. code:: python
+
+ from google.auth import compute_engine
+ credentials = compute_engine.Credentials()
+
+Service Accounts
+----------------
+
+A `service account`_ is stored in a JSON keyfile.
+
+The
+:meth:`from_service_account_json() `
+factory can be used to create a :class:`Client ` with
+service account credentials.
+
+For example, with a JSON keyfile:
+
+.. code:: python
+
+ client = Client.from_service_account_json('/path/to/keyfile.json')
+
+.. tip::
+
+ Previously the Google Cloud Console would issue a PKCS12/P12 key for your
+ service account. This library does not support that key format. You can
+ generate a new JSON key for the same service account from the console.
+
+User Accounts (3-legged OAuth 2.0) with a refresh token
+-------------------------------------------------------
+
+The majority of cases are intended to authenticate machines or
+workers rather than actual user accounts. However, it's also
+possible to call Google Cloud APIs with a user account via
+`OAuth 2.0`_.
+
+.. _OAuth 2.0: https://developers.google.com/identity/protocols/OAuth2
+
+.. tip::
+
+ A production application should **use a service account**,
+ but you may wish to use your own personal user account when first
+ getting started with the ``google-cloud-python`` library.
+
+The simplest way to use credentials from a user account is via
+Application Default Credentials using ``gcloud auth login``
+(as mentioned above) and :func:`google.auth.default`:
+
+.. code:: python
+
+ import google.auth
+
+ credentials, project = google.auth.default()
+
+This will still follow the :ref:`precedence `
+described above,
+so be sure none of the other possible environments conflict
+with your user provided credentials.
+
+Advanced users of `oauth2client`_ can also use custom flows to
+create credentials using `client secrets`_ or using a
+`webserver flow`_.
+After creation, :class:`Credentials `
+can be serialized with
+:meth:`to_json() `
+and stored in a file and then and deserialized with
+:meth:`from_json() `. In order
+to use ``oauth2client``'s credentials with this library, you'll need to
+`convert them`_.
+
+.. _oauth2client: https://github.com/Google/oauth2client.
+.. _client secrets: https://developers.google.com/api-client-library/python/guide/aaa_oauth#flow_from_clientsecrets
+.. _webserver flow: https://developers.google.com/api-client-library/python/guide/aaa_oauth#OAuth2WebServerFlow
+.. _convert them: http://google-auth.readthedocs.io/en/stable/user-guide.html#user-credentials
+
+Troubleshooting
+===============
+
+Setting up a Service Account
+----------------------------
+
+If your application is not running on Google Compute Engine,
+you need a `Google Developers Service Account`_.
+
+#. Visit the `Google Developers Console`_.
+
+#. Create a new project or click on an existing project.
+
+#. Navigate to **APIs & auth** > **APIs** and enable the APIs
+ that your application requires.
+
+ .. raw:: html
+
+
+
+ .. note::
+
+ You may need to enable billing in order to use these services.
+
+ * **BigQuery**
+
+ * BigQuery API
+
+ * **Datastore**
+
+ * Google Cloud Datastore API
+
+ * **Pub/Sub**
+
+ * Google Cloud Pub/Sub
+
+ * **Storage**
+
+ * Google Cloud Storage
+ * Google Cloud Storage JSON API
+
+#. Navigate to **APIs & auth** > **Credentials**.
+
+ You should see a screen like one of the following:
+
+ .. raw:: html
+
+
+
+ .. raw:: html
+
+
+
+ Find the "Add credentials" drop down and select "Service account" to be
+ guided through downloading a new JSON keyfile.
+
+ If you want to re-use an existing service account,
+ you can easily generate a new keyfile.
+ Just select the account you wish to re-use,
+ and click **Generate new JSON key**:
+
+ .. raw:: html
+
+
+
+.. _Google Developers Console: https://console.developers.google.com/project
+.. _Google Developers Service Account: https://developers.google.com/accounts/docs/OAuth2ServiceAccount
+
+Using Google Compute Engine
+---------------------------
+
+If your code is running on Google Compute Engine,
+using the inferred Google `Application Default Credentials`_
+will be sufficient for retrieving credentials.
+
+However, by default your credentials may not grant you
+access to the services you intend to use.
+Be sure when you `set up the GCE instance`_,
+you add the correct scopes for the APIs you want to access:
+
+* **All APIs**
+
+ * ``https://www.googleapis.com/auth/cloud-platform``
+ * ``https://www.googleapis.com/auth/cloud-platform.read-only``
+
+* **BigQuery**
+
+ * ``https://www.googleapis.com/auth/bigquery``
+ * ``https://www.googleapis.com/auth/bigquery.insertdata``
+
+* **Datastore**
+
+ * ``https://www.googleapis.com/auth/datastore``
+ * ``https://www.googleapis.com/auth/userinfo.email``
+
+* **Pub/Sub**
+
+ * ``https://www.googleapis.com/auth/pubsub``
+
+* **Storage**
+
+ * ``https://www.googleapis.com/auth/devstorage.full_control``
+ * ``https://www.googleapis.com/auth/devstorage.read_only``
+ * ``https://www.googleapis.com/auth/devstorage.read_write``
+
+.. _set up the GCE instance: https://cloud.google.com/compute/docs/authentication#using
diff --git a/api_core/docs/changelog.md b/api_core/docs/changelog.md
new file mode 120000
index 000000000000..04c99a55caae
--- /dev/null
+++ b/api_core/docs/changelog.md
@@ -0,0 +1 @@
+../CHANGELOG.md
\ No newline at end of file
diff --git a/api_core/docs/client_info.rst b/api_core/docs/client_info.rst
new file mode 100644
index 000000000000..e976b1863c7c
--- /dev/null
+++ b/api_core/docs/client_info.rst
@@ -0,0 +1,11 @@
+Client Information Helpers
+==========================
+
+.. automodule:: google.api_core.client_info
+ :members:
+ :show-inheritance:
+
+.. automodule:: google.api_core.gapic_v1.client_info
+ :members:
+ :show-inheritance:
+
diff --git a/api_core/docs/client_options.rst b/api_core/docs/client_options.rst
new file mode 100644
index 000000000000..da7c9a339bbd
--- /dev/null
+++ b/api_core/docs/client_options.rst
@@ -0,0 +1,6 @@
+Client Options
+==========================
+
+.. automodule:: google.api_core.client_options
+ :members:
+ :show-inheritance:
diff --git a/api_core/docs/conf.py b/api_core/docs/conf.py
new file mode 100644
index 000000000000..60b1f7a3f0b9
--- /dev/null
+++ b/api_core/docs/conf.py
@@ -0,0 +1,357 @@
+# -*- coding: utf-8 -*-
+#
+# google-api-core documentation build configuration file
+#
+# This file is execfile()d with the current directory set to its
+# containing dir.
+#
+# Note that not all possible configuration values are present in this
+# autogenerated file.
+#
+# All configuration values have a default; values that are commented out
+# serve to show the default.
+
+import sys
+import os
+import shlex
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+sys.path.insert(0, os.path.abspath(".."))
+
+__version__ = "0.1.0"
+
+# -- General configuration ------------------------------------------------
+
+# If your documentation needs a minimal Sphinx version, state it here.
+needs_sphinx = "1.6.3"
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+ "sphinx.ext.autodoc",
+ "sphinx.ext.autosummary",
+ "sphinx.ext.intersphinx",
+ "sphinx.ext.coverage",
+ "sphinx.ext.napoleon",
+ "sphinx.ext.todo",
+ "sphinx.ext.viewcode",
+]
+
+# autodoc/autosummary flags
+autoclass_content = "both"
+autodoc_default_flags = ["members"]
+autosummary_generate = True
+
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ["_templates"]
+
+# Allow markdown includes (so releases.md can include CHANGLEOG.md)
+# http://www.sphinx-doc.org/en/master/markdown.html
+source_parsers = {".md": "recommonmark.parser.CommonMarkParser"}
+
+# The suffix(es) of source filenames.
+# You can specify multiple suffix as a list of string:
+# source_suffix = ['.rst', '.md']
+source_suffix = [".rst", ".md"]
+
+# The encoding of source files.
+# source_encoding = 'utf-8-sig'
+
+# The master toctree document.
+master_doc = "index"
+
+# General information about the project.
+project = u"google-api-core"
+copyright = u"2017, Google"
+author = u"Google APIs"
+
+# The version info for the project you're documenting, acts as replacement for
+# |version| and |release|, also used in various other places throughout the
+# built documents.
+#
+# The full version, including alpha/beta/rc tags.
+release = __version__
+# The short X.Y version.
+version = ".".join(release.split(".")[0:2])
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#
+# This is also used if you do content translation via gettext catalogs.
+# Usually you set "language" from the command line for these cases.
+language = None
+
+# There are two options for replacing |today|: either, you set today to some
+# non-false value, then it is used:
+# today = ''
+# Else, today_fmt is used as the format for a strftime call.
+# today_fmt = '%B %d, %Y'
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+exclude_patterns = ["_build"]
+
+# The reST default role (used for this markup: `text`) to use for all
+# documents.
+# default_role = None
+
+# If true, '()' will be appended to :func: etc. cross-reference text.
+# add_function_parentheses = True
+
+# If true, the current module name will be prepended to all description
+# unit titles (such as .. function::).
+# add_module_names = True
+
+# If true, sectionauthor and moduleauthor directives will be shown in the
+# output. They are ignored by default.
+# show_authors = False
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = "sphinx"
+
+# A list of ignored prefixes for module index sorting.
+# modindex_common_prefix = []
+
+# If true, keep warnings as "system message" paragraphs in the built documents.
+# keep_warnings = False
+
+# If true, `todo` and `todoList` produce output, else they produce nothing.
+todo_include_todos = True
+
+
+# -- Options for HTML output ----------------------------------------------
+
+# The theme to use for HTML and HTML Help pages. See the documentation for
+# a list of builtin themes.
+html_theme = "alabaster"
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further. For a list of options available for each theme, see the
+# documentation.
+html_theme_options = {
+ "description": "Google Cloud Client Libraries for Python",
+ "github_user": "googleapis",
+ "github_repo": "google-cloud-python",
+ "github_banner": True,
+ "font_family": "'Roboto', Georgia, sans",
+ "head_font_family": "'Roboto', Georgia, serif",
+ "code_font_family": "'Roboto Mono', 'Consolas', monospace",
+}
+
+# Add any paths that contain custom themes here, relative to this directory.
+# html_theme_path = []
+
+# The name for this set of Sphinx documents. If None, it defaults to
+# " v documentation".
+# html_title = None
+
+# A shorter title for the navigation bar. Default is the same as html_title.
+# html_short_title = None
+
+# The name of an image file (relative to this directory) to place at the top
+# of the sidebar.
+# html_logo = None
+
+# The name of an image file (within the static path) to use as favicon of the
+# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
+# pixels large.
+# html_favicon = None
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+# html_static_path = []
+
+# Add any extra paths that contain custom files (such as robots.txt or
+# .htaccess) here, relative to this directory. These files are copied
+# directly to the root of the documentation.
+# html_extra_path = []
+
+# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
+# using the given strftime format.
+# html_last_updated_fmt = '%b %d, %Y'
+
+# If true, SmartyPants will be used to convert quotes and dashes to
+# typographically correct entities.
+# html_use_smartypants = True
+
+# Custom sidebar templates, maps document names to template names.
+# html_sidebars = {}
+
+# Additional templates that should be rendered to pages, maps page names to
+# template names.
+# html_additional_pages = {}
+
+# If false, no module index is generated.
+# html_domain_indices = True
+
+# If false, no index is generated.
+# html_use_index = True
+
+# If true, the index is split into individual pages for each letter.
+# html_split_index = False
+
+# If true, links to the reST sources are added to the pages.
+# html_show_sourcelink = True
+
+# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
+# html_show_sphinx = True
+
+# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
+# html_show_copyright = True
+
+# If true, an OpenSearch description file will be output, and all pages will
+# contain a tag referring to it. The value of this option must be the
+# base URL from which the finished HTML is served.
+# html_use_opensearch = ''
+
+# This is the file name suffix for HTML files (e.g. ".xhtml").
+# html_file_suffix = None
+
+# Language to be used for generating the HTML full-text search index.
+# Sphinx supports the following languages:
+# 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja'
+# 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr'
+# html_search_language = 'en'
+
+# A dictionary with options for the search language support, empty by default.
+# Now only 'ja' uses this config value
+# html_search_options = {'type': 'default'}
+
+# The name of a javascript file (relative to the configuration directory) that
+# implements a search results scorer. If empty, the default will be used.
+# html_search_scorer = 'scorer.js'
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = "google-api-core-doc"
+
+# -- Options for warnings ------------------------------------------------------
+
+
+suppress_warnings = [
+ # Temporarily suppress this to avoid "more than one target found for
+ # cross-reference" warning, which are intractable for us to avoid while in
+ # a mono-repo.
+ # See https://github.com/sphinx-doc/sphinx/blob
+ # /2a65ffeef5c107c19084fabdd706cdff3f52d93c/sphinx/domains/python.py#L843
+ "ref.python"
+]
+
+# -- Options for LaTeX output ---------------------------------------------
+
+latex_elements = {
+ # The paper size ('letterpaper' or 'a4paper').
+ #'papersize': 'letterpaper',
+ # The font size ('10pt', '11pt' or '12pt').
+ #'pointsize': '10pt',
+ # Additional stuff for the LaTeX preamble.
+ #'preamble': '',
+ # Latex figure (float) alignment
+ #'figure_align': 'htbp',
+}
+
+# Grouping the document tree into LaTeX files. List of tuples
+# (source start file, target name, title,
+# author, documentclass [howto, manual, or own class]).
+latex_documents = [
+ (
+ master_doc,
+ "google-api-core.tex",
+ u"google-api-core Documentation",
+ author,
+ "manual",
+ )
+]
+
+# The name of an image file (relative to this directory) to place at the top of
+# the title page.
+# latex_logo = None
+
+# For "manual" documents, if this is true, then toplevel headings are parts,
+# not chapters.
+# latex_use_parts = False
+
+# If true, show page references after internal links.
+# latex_show_pagerefs = False
+
+# If true, show URL addresses after external links.
+# latex_show_urls = False
+
+# Documents to append as an appendix to all manuals.
+# latex_appendices = []
+
+# If false, no module index is generated.
+# latex_domain_indices = True
+
+
+# -- Options for manual page output ---------------------------------------
+
+# One entry per manual page. List of tuples
+# (source start file, name, description, authors, manual section).
+man_pages = [
+ (master_doc, "google-api-core", u"google-api-core Documentation", [author], 1)
+]
+
+# If true, show URL addresses after external links.
+# man_show_urls = False
+
+
+# -- Options for Texinfo output -------------------------------------------
+
+# Grouping the document tree into Texinfo files. List of tuples
+# (source start file, target name, title, author,
+# dir menu entry, description, category)
+texinfo_documents = [
+ (
+ master_doc,
+ "google-api-core",
+ u"google-api-core Documentation",
+ author,
+ "google-api-core",
+ "GAPIC library for the {metadata.shortName} v1beta1 service",
+ "APIs",
+ )
+]
+
+# Documents to append as an appendix to all manuals.
+# texinfo_appendices = []
+
+# If false, no module index is generated.
+# texinfo_domain_indices = True
+
+# How to display URL addresses: 'footnote', 'no', or 'inline'.
+# texinfo_show_urls = 'footnote'
+
+# If true, do not generate a @detailmenu in the "Top" node's menu.
+# texinfo_no_detailmenu = False
+
+
+# Example configuration for intersphinx: refer to the Python standard library.
+intersphinx_mapping = {
+ "python": ("http://python.readthedocs.org/en/latest/", None),
+ "gax": ("https://gax-python.readthedocs.org/en/latest/", None),
+ "google-auth": ("https://google-auth.readthedocs.io/en/stable", None),
+ "google-gax": ("https://gax-python.readthedocs.io/en/latest/", None),
+ "google.api_core": ("https://googleapis.dev/python/google-api-core/latest", None),
+ "grpc": ("https://grpc.io/grpc/python/", None),
+ "requests": ("https://2.python-requests.org/en/master/", None),
+ "fastavro": ("https://fastavro.readthedocs.io/en/stable/", None),
+ "pandas": ("https://pandas.pydata.org/pandas-docs/stable/", None),
+}
+
+
+# Napoleon settings
+napoleon_google_docstring = True
+napoleon_numpy_docstring = True
+napoleon_include_private_with_doc = False
+napoleon_include_special_with_doc = True
+napoleon_use_admonition_for_examples = False
+napoleon_use_admonition_for_notes = False
+napoleon_use_admonition_for_references = False
+napoleon_use_ivar = False
+napoleon_use_param = True
+napoleon_use_rtype = True
diff --git a/api_core/docs/exceptions.rst b/api_core/docs/exceptions.rst
new file mode 100644
index 000000000000..d671f4e02046
--- /dev/null
+++ b/api_core/docs/exceptions.rst
@@ -0,0 +1,6 @@
+Exceptions
+==========
+
+.. automodule:: google.api_core.exceptions
+ :members:
+ :show-inheritance:
diff --git a/api_core/docs/futures.rst b/api_core/docs/futures.rst
new file mode 100644
index 000000000000..7a43da9de291
--- /dev/null
+++ b/api_core/docs/futures.rst
@@ -0,0 +1,10 @@
+Futures
+=======================
+
+.. automodule:: google.api_core.future
+ :members:
+ :show-inheritance:
+
+.. automodule:: google.api_core.future.polling
+ :members:
+ :show-inheritance:
\ No newline at end of file
diff --git a/api_core/docs/helpers.rst b/api_core/docs/helpers.rst
new file mode 100644
index 000000000000..6f72df9561ef
--- /dev/null
+++ b/api_core/docs/helpers.rst
@@ -0,0 +1,26 @@
+Helpers
+=======
+
+
+General Helpers
+---------------
+
+.. automodule:: google.api_core.general_helpers
+ :members:
+ :show-inheritance:
+
+
+Datetime Helpers
+----------------
+
+.. automodule:: google.api_core.datetime_helpers
+ :members:
+ :show-inheritance:
+
+
+gRPC Helpers
+------------
+
+.. automodule:: google.api_core.grpc_helpers
+ :members:
+ :show-inheritance:
diff --git a/api_core/docs/iam.rst b/api_core/docs/iam.rst
new file mode 100644
index 000000000000..bb80ae3c5798
--- /dev/null
+++ b/api_core/docs/iam.rst
@@ -0,0 +1,7 @@
+Identity and Access Management
+==============================
+
+.. automodule:: google.api_core.iam
+ :members:
+ :show-inheritance:
+ :member-order: bysource
diff --git a/api_core/docs/index.rst b/api_core/docs/index.rst
new file mode 100644
index 000000000000..67572a0bbf96
--- /dev/null
+++ b/api_core/docs/index.rst
@@ -0,0 +1,30 @@
+The ``google-cloud-core`` package contains helpers common to all
+``google-cloud-*`` packages. In an attempt to reach a stable API,
+much of the functionality has been split out into this package,
+``google-api-core``.
+
+Core
+====
+
+.. toctree::
+ auth
+ client_info
+ client_options
+ exceptions
+ futures
+ helpers
+ iam
+ operation
+ operations_client
+ page_iterator
+ path_template
+ retry
+ timeout
+
+Changelog
+~~~~~~~~~
+
+.. toctree::
+ :maxdepth: 2
+
+ changelog
diff --git a/api_core/docs/operation.rst b/api_core/docs/operation.rst
new file mode 100644
index 000000000000..c5e676621318
--- /dev/null
+++ b/api_core/docs/operation.rst
@@ -0,0 +1,6 @@
+Long-Running Operations
+=======================
+
+.. automodule:: google.api_core.operation
+ :members:
+ :show-inheritance:
diff --git a/api_core/docs/operations_client.rst b/api_core/docs/operations_client.rst
new file mode 100644
index 000000000000..be466d3fbf43
--- /dev/null
+++ b/api_core/docs/operations_client.rst
@@ -0,0 +1,6 @@
+Long-Running Operations Client
+==============================
+
+.. automodule:: google.api_core.operations_v1
+ :members:
+ :show-inheritance:
diff --git a/api_core/docs/page_iterator.rst b/api_core/docs/page_iterator.rst
new file mode 100644
index 000000000000..28842da2a76d
--- /dev/null
+++ b/api_core/docs/page_iterator.rst
@@ -0,0 +1,6 @@
+Page Iterators
+==============
+
+.. automodule:: google.api_core.page_iterator
+ :members:
+ :show-inheritance:
diff --git a/api_core/docs/path_template.rst b/api_core/docs/path_template.rst
new file mode 100644
index 000000000000..220779e3ba9f
--- /dev/null
+++ b/api_core/docs/path_template.rst
@@ -0,0 +1,6 @@
+Path Templates
+==============
+
+.. automodule:: google.api_core.path_template
+ :members:
+ :show-inheritance:
diff --git a/api_core/docs/retry.rst b/api_core/docs/retry.rst
new file mode 100644
index 000000000000..23a7d70f3431
--- /dev/null
+++ b/api_core/docs/retry.rst
@@ -0,0 +1,6 @@
+Retry
+=====
+
+.. automodule:: google.api_core.retry
+ :members:
+ :show-inheritance:
diff --git a/api_core/docs/timeout.rst b/api_core/docs/timeout.rst
new file mode 100644
index 000000000000..943d42508997
--- /dev/null
+++ b/api_core/docs/timeout.rst
@@ -0,0 +1,6 @@
+Timeout
+=======
+
+.. automodule:: google.api_core.timeout
+ :members:
+ :show-inheritance:
diff --git a/api_core/google/__init__.py b/api_core/google/__init__.py
new file mode 100644
index 000000000000..0d0a4c3ab273
--- /dev/null
+++ b/api_core/google/__init__.py
@@ -0,0 +1,24 @@
+# Copyright 2016 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Google namespace package."""
+
+try:
+ import pkg_resources
+
+ pkg_resources.declare_namespace(__name__)
+except ImportError:
+ import pkgutil
+
+ __path__ = pkgutil.extend_path(__path__, __name__)
diff --git a/api_core/google/api_core/__init__.py b/api_core/google/api_core/__init__.py
new file mode 100644
index 000000000000..c762e18309db
--- /dev/null
+++ b/api_core/google/api_core/__init__.py
@@ -0,0 +1,23 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Google API Core.
+
+This package contains common code and utilties used by Google client libraries.
+"""
+
+from pkg_resources import get_distribution
+
+
+__version__ = get_distribution("google-api-core").version
diff --git a/api_core/google/api_core/bidi.py b/api_core/google/api_core/bidi.py
new file mode 100644
index 000000000000..f73c7c9dfabc
--- /dev/null
+++ b/api_core/google/api_core/bidi.py
@@ -0,0 +1,727 @@
+# Copyright 2017, Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Bi-directional streaming RPC helpers."""
+
+import collections
+import datetime
+import logging
+import threading
+import time
+
+from six.moves import queue
+
+from google.api_core import exceptions
+
+_LOGGER = logging.getLogger(__name__)
+_BIDIRECTIONAL_CONSUMER_NAME = "Thread-ConsumeBidirectionalStream"
+
+
+class _RequestQueueGenerator(object):
+ """A helper for sending requests to a gRPC stream from a Queue.
+
+ This generator takes requests off a given queue and yields them to gRPC.
+
+ This helper is useful when you have an indeterminate, indefinite, or
+ otherwise open-ended set of requests to send through a request-streaming
+ (or bidirectional) RPC.
+
+ The reason this is necessary is because gRPC takes an iterator as the
+ request for request-streaming RPCs. gRPC consumes this iterator in another
+ thread to allow it to block while generating requests for the stream.
+ However, if the generator blocks indefinitely gRPC will not be able to
+ clean up the thread as it'll be blocked on `next(iterator)` and not be able
+ to check the channel status to stop iterating. This helper mitigates that
+ by waiting on the queue with a timeout and checking the RPC state before
+ yielding.
+
+ Finally, it allows for retrying without swapping queues because if it does
+ pull an item off the queue when the RPC is inactive, it'll immediately put
+ it back and then exit. This is necessary because yielding the item in this
+ case will cause gRPC to discard it. In practice, this means that the order
+ of messages is not guaranteed. If such a thing is necessary it would be
+ easy to use a priority queue.
+
+ Example::
+
+ requests = request_queue_generator(q)
+ call = stub.StreamingRequest(iter(requests))
+ requests.call = call
+
+ for response in call:
+ print(response)
+ q.put(...)
+
+ Note that it is possible to accomplish this behavior without "spinning"
+ (using a queue timeout). One possible way would be to use more threads to
+ multiplex the grpc end event with the queue, another possible way is to
+ use selectors and a custom event/queue object. Both of these approaches
+ are significant from an engineering perspective for small benefit - the
+ CPU consumed by spinning is pretty minuscule.
+
+ Args:
+ queue (queue.Queue): The request queue.
+ period (float): The number of seconds to wait for items from the queue
+ before checking if the RPC is cancelled. In practice, this
+ determines the maximum amount of time the request consumption
+ thread will live after the RPC is cancelled.
+ initial_request (Union[protobuf.Message,
+ Callable[None, protobuf.Message]]): The initial request to
+ yield. This is done independently of the request queue to allow fo
+ easily restarting streams that require some initial configuration
+ request.
+ """
+
+ def __init__(self, queue, period=1, initial_request=None):
+ self._queue = queue
+ self._period = period
+ self._initial_request = initial_request
+ self.call = None
+
+ def _is_active(self):
+ # Note: there is a possibility that this starts *before* the call
+ # property is set. So we have to check if self.call is set before
+ # seeing if it's active.
+ if self.call is not None and not self.call.is_active():
+ return False
+ else:
+ return True
+
+ def __iter__(self):
+ if self._initial_request is not None:
+ if callable(self._initial_request):
+ yield self._initial_request()
+ else:
+ yield self._initial_request
+
+ while True:
+ try:
+ item = self._queue.get(timeout=self._period)
+ except queue.Empty:
+ if not self._is_active():
+ _LOGGER.debug(
+ "Empty queue and inactive call, exiting request " "generator."
+ )
+ return
+ else:
+ # call is still active, keep waiting for queue items.
+ continue
+
+ # The consumer explicitly sent "None", indicating that the request
+ # should end.
+ if item is None:
+ _LOGGER.debug("Cleanly exiting request generator.")
+ return
+
+ if not self._is_active():
+ # We have an item, but the call is closed. We should put the
+ # item back on the queue so that the next call can consume it.
+ self._queue.put(item)
+ _LOGGER.debug(
+ "Inactive call, replacing item on queue and exiting "
+ "request generator."
+ )
+ return
+
+ yield item
+
+
+class _Throttle(object):
+ """A context manager limiting the total entries in a sliding time window.
+
+ If more than ``access_limit`` attempts are made to enter the context manager
+ instance in the last ``time window`` interval, the exceeding requests block
+ until enough time elapses.
+
+ The context manager instances are thread-safe and can be shared between
+ multiple threads. If multiple requests are blocked and waiting to enter,
+ the exact order in which they are allowed to proceed is not determined.
+
+ Example::
+
+ max_three_per_second = _Throttle(
+ access_limit=3, time_window=datetime.timedelta(seconds=1)
+ )
+
+ for i in range(5):
+ with max_three_per_second as time_waited:
+ print("{}: Waited {} seconds to enter".format(i, time_waited))
+
+ Args:
+ access_limit (int): the maximum number of entries allowed in the time window
+ time_window (datetime.timedelta): the width of the sliding time window
+ """
+
+ def __init__(self, access_limit, time_window):
+ if access_limit < 1:
+ raise ValueError("access_limit argument must be positive")
+
+ if time_window <= datetime.timedelta(0):
+ raise ValueError("time_window argument must be a positive timedelta")
+
+ self._time_window = time_window
+ self._access_limit = access_limit
+ self._past_entries = collections.deque(maxlen=access_limit) # least recent first
+ self._entry_lock = threading.Lock()
+
+ def __enter__(self):
+ with self._entry_lock:
+ cutoff_time = datetime.datetime.now() - self._time_window
+
+ # drop the entries that are too old, as they are no longer relevant
+ while self._past_entries and self._past_entries[0] < cutoff_time:
+ self._past_entries.popleft()
+
+ if len(self._past_entries) < self._access_limit:
+ self._past_entries.append(datetime.datetime.now())
+ return 0.0 # no waiting was needed
+
+ to_wait = (self._past_entries[0] - cutoff_time).total_seconds()
+ time.sleep(to_wait)
+
+ self._past_entries.append(datetime.datetime.now())
+ return to_wait
+
+ def __exit__(self, *_):
+ pass
+
+ def __repr__(self):
+ return "{}(access_limit={}, time_window={})".format(
+ self.__class__.__name__,
+ self._access_limit,
+ repr(self._time_window),
+ )
+
+
+class BidiRpc(object):
+ """A helper for consuming a bi-directional streaming RPC.
+
+ This maps gRPC's built-in interface which uses a request iterator and a
+ response iterator into a socket-like :func:`send` and :func:`recv`. This
+ is a more useful pattern for long-running or asymmetric streams (streams
+ where there is not a direct correlation between the requests and
+ responses).
+
+ Example::
+
+ initial_request = example_pb2.StreamingRpcRequest(
+ setting='example')
+ rpc = BidiRpc(
+ stub.StreamingRpc,
+ initial_request=initial_request,
+ metadata=[('name', 'value')]
+ )
+
+ rpc.open()
+
+ while rpc.is_active():
+ print(rpc.recv())
+ rpc.send(example_pb2.StreamingRpcRequest(
+ data='example'))
+
+ This does *not* retry the stream on errors. See :class:`ResumableBidiRpc`.
+
+ Args:
+ start_rpc (grpc.StreamStreamMultiCallable): The gRPC method used to
+ start the RPC.
+ initial_request (Union[protobuf.Message,
+ Callable[None, protobuf.Message]]): The initial request to
+ yield. This is useful if an initial request is needed to start the
+ stream.
+ metadata (Sequence[Tuple(str, str)]): RPC metadata to include in
+ the request.
+ """
+
+ def __init__(self, start_rpc, initial_request=None, metadata=None):
+ self._start_rpc = start_rpc
+ self._initial_request = initial_request
+ self._rpc_metadata = metadata
+ self._request_queue = queue.Queue()
+ self._request_generator = None
+ self._is_active = False
+ self._callbacks = []
+ self.call = None
+
+ def add_done_callback(self, callback):
+ """Adds a callback that will be called when the RPC terminates.
+
+ This occurs when the RPC errors or is successfully terminated.
+
+ Args:
+ callback (Callable[[grpc.Future], None]): The callback to execute.
+ It will be provided with the same gRPC future as the underlying
+ stream which will also be a :class:`grpc.Call`.
+ """
+ self._callbacks.append(callback)
+
+ def _on_call_done(self, future):
+ for callback in self._callbacks:
+ callback(future)
+
+ def open(self):
+ """Opens the stream."""
+ if self.is_active:
+ raise ValueError("Can not open an already open stream.")
+
+ request_generator = _RequestQueueGenerator(
+ self._request_queue, initial_request=self._initial_request
+ )
+ call = self._start_rpc(iter(request_generator), metadata=self._rpc_metadata)
+
+ request_generator.call = call
+
+ # TODO: api_core should expose the future interface for wrapped
+ # callables as well.
+ if hasattr(call, "_wrapped"): # pragma: NO COVER
+ call._wrapped.add_done_callback(self._on_call_done)
+ else:
+ call.add_done_callback(self._on_call_done)
+
+ self._request_generator = request_generator
+ self.call = call
+
+ def close(self):
+ """Closes the stream."""
+ if self.call is None:
+ return
+
+ self._request_queue.put(None)
+ self.call.cancel()
+ self._request_generator = None
+ # Don't set self.call to None. Keep it around so that send/recv can
+ # raise the error.
+
+ def send(self, request):
+ """Queue a message to be sent on the stream.
+
+ Send is non-blocking.
+
+ If the underlying RPC has been closed, this will raise.
+
+ Args:
+ request (protobuf.Message): The request to send.
+ """
+ if self.call is None:
+ raise ValueError("Can not send() on an RPC that has never been open()ed.")
+
+ # Don't use self.is_active(), as ResumableBidiRpc will overload it
+ # to mean something semantically different.
+ if self.call.is_active():
+ self._request_queue.put(request)
+ else:
+ # calling next should cause the call to raise.
+ next(self.call)
+
+ def recv(self):
+ """Wait for a message to be returned from the stream.
+
+ Recv is blocking.
+
+ If the underlying RPC has been closed, this will raise.
+
+ Returns:
+ protobuf.Message: The received message.
+ """
+ if self.call is None:
+ raise ValueError("Can not recv() on an RPC that has never been open()ed.")
+
+ return next(self.call)
+
+ @property
+ def is_active(self):
+ """bool: True if this stream is currently open and active."""
+ return self.call is not None and self.call.is_active()
+
+ @property
+ def pending_requests(self):
+ """int: Returns an estimate of the number of queued requests."""
+ return self._request_queue.qsize()
+
+
+def _never_terminate(future_or_error):
+ """By default, no errors cause BiDi termination."""
+ return False
+
+
+class ResumableBidiRpc(BidiRpc):
+ """A :class:`BidiRpc` that can automatically resume the stream on errors.
+
+ It uses the ``should_recover`` arg to determine if it should re-establish
+ the stream on error.
+
+ Example::
+
+ def should_recover(exc):
+ return (
+ isinstance(exc, grpc.RpcError) and
+ exc.code() == grpc.StatusCode.UNVAILABLE)
+
+ initial_request = example_pb2.StreamingRpcRequest(
+ setting='example')
+
+ metadata = [('header_name', 'value')]
+
+ rpc = ResumableBidiRpc(
+ stub.StreamingRpc,
+ should_recover=should_recover,
+ initial_request=initial_request,
+ metadata=metadata
+ )
+
+ rpc.open()
+
+ while rpc.is_active():
+ print(rpc.recv())
+ rpc.send(example_pb2.StreamingRpcRequest(
+ data='example'))
+
+ Args:
+ start_rpc (grpc.StreamStreamMultiCallable): The gRPC method used to
+ start the RPC.
+ initial_request (Union[protobuf.Message,
+ Callable[None, protobuf.Message]]): The initial request to
+ yield. This is useful if an initial request is needed to start the
+ stream.
+ should_recover (Callable[[Exception], bool]): A function that returns
+ True if the stream should be recovered. This will be called
+ whenever an error is encountered on the stream.
+ should_terminate (Callable[[Exception], bool]): A function that returns
+ True if the stream should be terminated. This will be called
+ whenever an error is encountered on the stream.
+ metadata Sequence[Tuple(str, str)]: RPC metadata to include in
+ the request.
+ throttle_reopen (bool): If ``True``, throttling will be applied to
+ stream reopen calls. Defaults to ``False``.
+ """
+
+ def __init__(
+ self,
+ start_rpc,
+ should_recover,
+ should_terminate=_never_terminate,
+ initial_request=None,
+ metadata=None,
+ throttle_reopen=False,
+ ):
+ super(ResumableBidiRpc, self).__init__(start_rpc, initial_request, metadata)
+ self._should_recover = should_recover
+ self._should_terminate = should_terminate
+ self._operational_lock = threading.RLock()
+ self._finalized = False
+ self._finalize_lock = threading.Lock()
+
+ if throttle_reopen:
+ self._reopen_throttle = _Throttle(
+ access_limit=5, time_window=datetime.timedelta(seconds=10),
+ )
+ else:
+ self._reopen_throttle = None
+
+ def _finalize(self, result):
+ with self._finalize_lock:
+ if self._finalized:
+ return
+
+ for callback in self._callbacks:
+ callback(result)
+
+ self._finalized = True
+
+ def _on_call_done(self, future):
+ # Unlike the base class, we only execute the callbacks on a terminal
+ # error, not for errors that we can recover from. Note that grpc's
+ # "future" here is also a grpc.RpcError.
+ with self._operational_lock:
+ if self._should_terminate(future):
+ self._finalize(future)
+ elif not self._should_recover(future):
+ self._finalize(future)
+ else:
+ _LOGGER.debug("Re-opening stream from gRPC callback.")
+ self._reopen()
+
+ def _reopen(self):
+ with self._operational_lock:
+ # Another thread already managed to re-open this stream.
+ if self.call is not None and self.call.is_active():
+ _LOGGER.debug("Stream was already re-established.")
+ return
+
+ self.call = None
+ # Request generator should exit cleanly since the RPC its bound to
+ # has exited.
+ self._request_generator = None
+
+ # Note: we do not currently do any sort of backoff here. The
+ # assumption is that re-establishing the stream under normal
+ # circumstances will happen in intervals greater than 60s.
+ # However, it is possible in a degenerative case that the server
+ # closes the stream rapidly which would lead to thrashing here,
+ # but hopefully in those cases the server would return a non-
+ # retryable error.
+
+ try:
+ if self._reopen_throttle:
+ with self._reopen_throttle:
+ self.open()
+ else:
+ self.open()
+ # If re-opening or re-calling the method fails for any reason,
+ # consider it a terminal error and finalize the stream.
+ except Exception as exc:
+ _LOGGER.debug("Failed to re-open stream due to %s", exc)
+ self._finalize(exc)
+ raise
+
+ _LOGGER.info("Re-established stream")
+
+ def _recoverable(self, method, *args, **kwargs):
+ """Wraps a method to recover the stream and retry on error.
+
+ If a retryable error occurs while making the call, then the stream will
+ be re-opened and the method will be retried. This happens indefinitely
+ so long as the error is a retryable one. If an error occurs while
+ re-opening the stream, then this method will raise immediately and
+ trigger finalization of this object.
+
+ Args:
+ method (Callable[..., Any]): The method to call.
+ args: The args to pass to the method.
+ kwargs: The kwargs to pass to the method.
+ """
+ while True:
+ try:
+ return method(*args, **kwargs)
+
+ except Exception as exc:
+ with self._operational_lock:
+ _LOGGER.debug("Call to retryable %r caused %s.", method, exc)
+
+ if self._should_terminate(exc):
+ self.close()
+ _LOGGER.debug("Terminating %r due to %s.", method, exc)
+ self._finalize(exc)
+ break
+
+ if not self._should_recover(exc):
+ self.close()
+ _LOGGER.debug("Not retrying %r due to %s.", method, exc)
+ self._finalize(exc)
+ raise exc
+
+ _LOGGER.debug("Re-opening stream from retryable %r.", method)
+ self._reopen()
+
+ def _send(self, request):
+ # Grab a reference to the RPC call. Because another thread (notably
+ # the gRPC error thread) can modify self.call (by invoking reopen),
+ # we should ensure our reference can not change underneath us.
+ # If self.call is modified (such as replaced with a new RPC call) then
+ # this will use the "old" RPC, which should result in the same
+ # exception passed into gRPC's error handler being raised here, which
+ # will be handled by the usual error handling in retryable.
+ with self._operational_lock:
+ call = self.call
+
+ if call is None:
+ raise ValueError("Can not send() on an RPC that has never been open()ed.")
+
+ # Don't use self.is_active(), as ResumableBidiRpc will overload it
+ # to mean something semantically different.
+ if call.is_active():
+ self._request_queue.put(request)
+ pass
+ else:
+ # calling next should cause the call to raise.
+ next(call)
+
+ def send(self, request):
+ return self._recoverable(self._send, request)
+
+ def _recv(self):
+ with self._operational_lock:
+ call = self.call
+
+ if call is None:
+ raise ValueError("Can not recv() on an RPC that has never been open()ed.")
+
+ return next(call)
+
+ def recv(self):
+ return self._recoverable(self._recv)
+
+ @property
+ def is_active(self):
+ """bool: True if this stream is currently open and active."""
+ # Use the operational lock. It's entirely possible for something
+ # to check the active state *while* the RPC is being retried.
+ # Also, use finalized to track the actual terminal state here.
+ # This is because if the stream is re-established by the gRPC thread
+ # it's technically possible to check this between when gRPC marks the
+ # RPC as inactive and when gRPC executes our callback that re-opens
+ # the stream.
+ with self._operational_lock:
+ return self.call is not None and not self._finalized
+
+
+class BackgroundConsumer(object):
+ """A bi-directional stream consumer that runs in a separate thread.
+
+ This maps the consumption of a stream into a callback-based model. It also
+ provides :func:`pause` and :func:`resume` to allow for flow-control.
+
+ Example::
+
+ def should_recover(exc):
+ return (
+ isinstance(exc, grpc.RpcError) and
+ exc.code() == grpc.StatusCode.UNVAILABLE)
+
+ initial_request = example_pb2.StreamingRpcRequest(
+ setting='example')
+
+ rpc = ResumeableBidiRpc(
+ stub.StreamingRpc,
+ initial_request=initial_request,
+ should_recover=should_recover)
+
+ def on_response(response):
+ print(response)
+
+ consumer = BackgroundConsumer(rpc, on_response)
+ consumer.start()
+
+ Note that error handling *must* be done by using the provided
+ ``bidi_rpc``'s ``add_done_callback``. This helper will automatically exit
+ whenever the RPC itself exits and will not provide any error details.
+
+ Args:
+ bidi_rpc (BidiRpc): The RPC to consume. Should not have been
+ ``open()``ed yet.
+ on_response (Callable[[protobuf.Message], None]): The callback to
+ be called for every response on the stream.
+ """
+
+ def __init__(self, bidi_rpc, on_response):
+ self._bidi_rpc = bidi_rpc
+ self._on_response = on_response
+ self._paused = False
+ self._wake = threading.Condition()
+ self._thread = None
+ self._operational_lock = threading.Lock()
+
+ def _on_call_done(self, future):
+ # Resume the thread if it's paused, this prevents blocking forever
+ # when the RPC has terminated.
+ self.resume()
+
+ def _thread_main(self, ready):
+ try:
+ ready.set()
+ self._bidi_rpc.add_done_callback(self._on_call_done)
+ self._bidi_rpc.open()
+
+ while self._bidi_rpc.is_active:
+ # Do not allow the paused status to change at all during this
+ # section. There is a condition where we could be resumed
+ # between checking if we are paused and calling wake.wait(),
+ # which means that we will miss the notification to wake up
+ # (oops!) and wait for a notification that will never come.
+ # Keeping the lock throughout avoids that.
+ # In the future, we could use `Condition.wait_for` if we drop
+ # Python 2.7.
+ with self._wake:
+ while self._paused:
+ _LOGGER.debug("paused, waiting for waking.")
+ self._wake.wait()
+ _LOGGER.debug("woken.")
+
+ _LOGGER.debug("waiting for recv.")
+ response = self._bidi_rpc.recv()
+ _LOGGER.debug("recved response.")
+ self._on_response(response)
+
+ except exceptions.GoogleAPICallError as exc:
+ _LOGGER.debug(
+ "%s caught error %s and will exit. Generally this is due to "
+ "the RPC itself being cancelled and the error will be "
+ "surfaced to the calling code.",
+ _BIDIRECTIONAL_CONSUMER_NAME,
+ exc,
+ exc_info=True,
+ )
+
+ except Exception as exc:
+ _LOGGER.exception(
+ "%s caught unexpected exception %s and will exit.",
+ _BIDIRECTIONAL_CONSUMER_NAME,
+ exc,
+ )
+
+ _LOGGER.info("%s exiting", _BIDIRECTIONAL_CONSUMER_NAME)
+
+ def start(self):
+ """Start the background thread and begin consuming the thread."""
+ with self._operational_lock:
+ ready = threading.Event()
+ thread = threading.Thread(
+ name=_BIDIRECTIONAL_CONSUMER_NAME,
+ target=self._thread_main,
+ args=(ready,),
+ )
+ thread.daemon = True
+ thread.start()
+ # Other parts of the code rely on `thread.is_alive` which
+ # isn't sufficient to know if a thread is active, just that it may
+ # soon be active. This can cause races. Further protect
+ # against races by using a ready event and wait on it to be set.
+ ready.wait()
+ self._thread = thread
+ _LOGGER.debug("Started helper thread %s", thread.name)
+
+ def stop(self):
+ """Stop consuming the stream and shutdown the background thread."""
+ with self._operational_lock:
+ self._bidi_rpc.close()
+
+ if self._thread is not None:
+ # Resume the thread to wake it up in case it is sleeping.
+ self.resume()
+ self._thread.join()
+
+ self._thread = None
+
+ @property
+ def is_active(self):
+ """bool: True if the background thread is active."""
+ return self._thread is not None and self._thread.is_alive()
+
+ def pause(self):
+ """Pauses the response stream.
+
+ This does *not* pause the request stream.
+ """
+ with self._wake:
+ self._paused = True
+
+ def resume(self):
+ """Resumes the response stream."""
+ with self._wake:
+ self._paused = False
+ self._wake.notifyAll()
+
+ @property
+ def is_paused(self):
+ """bool: True if the response stream is paused."""
+ return self._paused
diff --git a/api_core/google/api_core/client_info.py b/api_core/google/api_core/client_info.py
new file mode 100644
index 000000000000..b196b7a987e4
--- /dev/null
+++ b/api_core/google/api_core/client_info.py
@@ -0,0 +1,96 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Helpers for providing client information.
+
+Client information is used to send information about the calling client,
+such as the library and Python version, to API services.
+"""
+
+import platform
+
+import pkg_resources
+
+_PY_VERSION = platform.python_version()
+_API_CORE_VERSION = pkg_resources.get_distribution("google-api-core").version
+
+try:
+ _GRPC_VERSION = pkg_resources.get_distribution("grpcio").version
+except pkg_resources.DistributionNotFound: # pragma: NO COVER
+ _GRPC_VERSION = None
+
+
+class ClientInfo(object):
+ """Client information used to generate a user-agent for API calls.
+
+ This user-agent information is sent along with API calls to allow the
+ receiving service to do analytics on which versions of Python and Google
+ libraries are being used.
+
+ Args:
+ python_version (str): The Python interpreter version, for example,
+ ``'2.7.13'``.
+ grpc_version (Optional[str]): The gRPC library version.
+ api_core_version (str): The google-api-core library version.
+ gapic_version (Optional[str]): The sversion of gapic-generated client
+ library, if the library was generated by gapic.
+ client_library_version (Optional[str]): The version of the client
+ library, generally used if the client library was not generated
+ by gapic or if additional functionality was built on top of
+ a gapic client library.
+ user_agent (Optional[str]): Prefix to the user agent header. This is
+ used to supply information such as application name or partner tool.
+ Recommended format: ``application-or-tool-ID/major.minor.version``.
+ """
+
+ def __init__(
+ self,
+ python_version=_PY_VERSION,
+ grpc_version=_GRPC_VERSION,
+ api_core_version=_API_CORE_VERSION,
+ gapic_version=None,
+ client_library_version=None,
+ user_agent=None,
+ ):
+ self.python_version = python_version
+ self.grpc_version = grpc_version
+ self.api_core_version = api_core_version
+ self.gapic_version = gapic_version
+ self.client_library_version = client_library_version
+ self.user_agent = user_agent
+
+ def to_user_agent(self):
+ """Returns the user-agent string for this client info."""
+
+ # Note: the order here is important as the internal metrics system
+ # expects these items to be in specific locations.
+ ua = ""
+
+ if self.user_agent is not None:
+ ua += "{user_agent} "
+
+ ua += "gl-python/{python_version} "
+
+ if self.grpc_version is not None:
+ ua += "grpc/{grpc_version} "
+
+ ua += "gax/{api_core_version} "
+
+ if self.gapic_version is not None:
+ ua += "gapic/{gapic_version} "
+
+ if self.client_library_version is not None:
+ ua += "gccl/{client_library_version} "
+
+ return ua.format(**self.__dict__).strip()
diff --git a/api_core/google/api_core/client_options.py b/api_core/google/api_core/client_options.py
new file mode 100644
index 000000000000..1144d2f62a37
--- /dev/null
+++ b/api_core/google/api_core/client_options.py
@@ -0,0 +1,69 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Client options class.
+
+Client options provide a consistent interface for user options to be defined
+across clients.
+
+You can pass a client options object to a client.
+
+.. code-block:: python
+
+ from google.api_core.client_options import ClientOptions
+ from google.cloud.vision_v1 import ImageAnnotatorClient
+
+ options = ClientOptions(api_endpoint="foo.googleapis.com")
+
+ client = ImageAnnotatorClient(client_options=options)
+
+You can also pass a dictionary.
+
+.. code-block:: python
+
+ from google.cloud.vision_v1 import ImageAnnotatorClient
+
+ client = ImageAnnotatorClient(client_options={"api_endpoint": "foo.googleapis.com"})
+
+
+"""
+
+
+class ClientOptions(object):
+ """Client Options used to set options on clients.
+
+ Args:
+ api_endpoint (str): The desired API endpoint, e.g., compute.googleapis.com
+ """
+
+ def __init__(self, api_endpoint=None):
+ self.api_endpoint = api_endpoint
+
+
+def from_dict(options):
+ """Construct a client options object from a dictionary.
+
+ Args:
+ options (dict): A dictionary with client options.
+ """
+
+ client_options = ClientOptions()
+
+ for key, value in options.items():
+ if hasattr(client_options, key):
+ setattr(client_options, key, value)
+ else:
+ raise ValueError("ClientOptions does not accept an option '" + key + "'")
+
+ return client_options
diff --git a/api_core/google/api_core/datetime_helpers.py b/api_core/google/api_core/datetime_helpers.py
new file mode 100644
index 000000000000..84c1bb7f512c
--- /dev/null
+++ b/api_core/google/api_core/datetime_helpers.py
@@ -0,0 +1,303 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Helpers for :mod:`datetime`."""
+
+import calendar
+import datetime
+import re
+
+import pytz
+
+from google.protobuf import timestamp_pb2
+
+
+_UTC_EPOCH = datetime.datetime.utcfromtimestamp(0).replace(tzinfo=pytz.utc)
+_RFC3339_MICROS = "%Y-%m-%dT%H:%M:%S.%fZ"
+_RFC3339_NO_FRACTION = "%Y-%m-%dT%H:%M:%S"
+# datetime.strptime cannot handle nanosecond precision: parse w/ regex
+_RFC3339_NANOS = re.compile(
+ r"""
+ (?P
+ \d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2} # YYYY-MM-DDTHH:MM:SS
+ )
+ ( # Optional decimal part
+ \. # decimal point
+ (?P\d{1,9}) # nanoseconds, maybe truncated
+ )?
+ Z # Zulu
+""",
+ re.VERBOSE,
+)
+
+
+def utcnow():
+ """A :meth:`datetime.datetime.utcnow()` alias to allow mocking in tests."""
+ return datetime.datetime.utcnow()
+
+
+def to_milliseconds(value):
+ """Convert a zone-aware datetime to milliseconds since the unix epoch.
+
+ Args:
+ value (datetime.datetime): The datetime to covert.
+
+ Returns:
+ int: Milliseconds since the unix epoch.
+ """
+ micros = to_microseconds(value)
+ return micros // 1000
+
+
+def from_microseconds(value):
+ """Convert timestamp in microseconds since the unix epoch to datetime.
+
+ Args:
+ value (float): The timestamp to convert, in microseconds.
+
+ Returns:
+ datetime.datetime: The datetime object equivalent to the timestamp in
+ UTC.
+ """
+ return _UTC_EPOCH + datetime.timedelta(microseconds=value)
+
+
+def to_microseconds(value):
+ """Convert a datetime to microseconds since the unix epoch.
+
+ Args:
+ value (datetime.datetime): The datetime to covert.
+
+ Returns:
+ int: Microseconds since the unix epoch.
+ """
+ if not value.tzinfo:
+ value = value.replace(tzinfo=pytz.utc)
+ # Regardless of what timezone is on the value, convert it to UTC.
+ value = value.astimezone(pytz.utc)
+ # Convert the datetime to a microsecond timestamp.
+ return int(calendar.timegm(value.timetuple()) * 1e6) + value.microsecond
+
+
+def from_iso8601_date(value):
+ """Convert a ISO8601 date string to a date.
+
+ Args:
+ value (str): The ISO8601 date string.
+
+ Returns:
+ datetime.date: A date equivalent to the date string.
+ """
+ return datetime.datetime.strptime(value, "%Y-%m-%d").date()
+
+
+def from_iso8601_time(value):
+ """Convert a zoneless ISO8601 time string to a time.
+
+ Args:
+ value (str): The ISO8601 time string.
+
+ Returns:
+ datetime.time: A time equivalent to the time string.
+ """
+ return datetime.datetime.strptime(value, "%H:%M:%S").time()
+
+
+def from_rfc3339(value):
+ """Convert a microsecond-precision timestamp to datetime.
+
+ Args:
+ value (str): The RFC3339 string to convert.
+
+ Returns:
+ datetime.datetime: The datetime object equivalent to the timestamp in
+ UTC.
+ """
+ return datetime.datetime.strptime(value, _RFC3339_MICROS).replace(tzinfo=pytz.utc)
+
+
+def from_rfc3339_nanos(value):
+ """Convert a nanosecond-precision timestamp to a native datetime.
+
+ .. note::
+ Python datetimes do not support nanosecond precision; this function
+ therefore truncates such values to microseconds.
+
+ Args:
+ value (str): The RFC3339 string to convert.
+
+ Returns:
+ datetime.datetime: The datetime object equivalent to the timestamp in
+ UTC.
+
+ Raises:
+ ValueError: If the timestamp does not match the RFC 3339
+ regular expression.
+ """
+ with_nanos = _RFC3339_NANOS.match(value)
+
+ if with_nanos is None:
+ raise ValueError(
+ "Timestamp: {!r}, does not match pattern: {!r}".format(
+ value, _RFC3339_NANOS.pattern
+ )
+ )
+
+ bare_seconds = datetime.datetime.strptime(
+ with_nanos.group("no_fraction"), _RFC3339_NO_FRACTION
+ )
+ fraction = with_nanos.group("nanos")
+
+ if fraction is None:
+ micros = 0
+ else:
+ scale = 9 - len(fraction)
+ nanos = int(fraction) * (10 ** scale)
+ micros = nanos // 1000
+
+ return bare_seconds.replace(microsecond=micros, tzinfo=pytz.utc)
+
+
+def to_rfc3339(value, ignore_zone=True):
+ """Convert a datetime to an RFC3339 timestamp string.
+
+ Args:
+ value (datetime.datetime):
+ The datetime object to be converted to a string.
+ ignore_zone (bool): If True, then the timezone (if any) of the
+ datetime object is ignored and the datetime is treated as UTC.
+
+ Returns:
+ str: The RFC3339 formated string representing the datetime.
+ """
+ if not ignore_zone and value.tzinfo is not None:
+ # Convert to UTC and remove the time zone info.
+ value = value.replace(tzinfo=None) - value.utcoffset()
+
+ return value.strftime(_RFC3339_MICROS)
+
+
+class DatetimeWithNanoseconds(datetime.datetime):
+ """Track nanosecond in addition to normal datetime attrs.
+
+ Nanosecond can be passed only as a keyword argument.
+ """
+
+ __slots__ = ("_nanosecond",)
+
+ # pylint: disable=arguments-differ
+ def __new__(cls, *args, **kw):
+ nanos = kw.pop("nanosecond", 0)
+ if nanos > 0:
+ if "microsecond" in kw:
+ raise TypeError("Specify only one of 'microsecond' or 'nanosecond'")
+ kw["microsecond"] = nanos // 1000
+ inst = datetime.datetime.__new__(cls, *args, **kw)
+ inst._nanosecond = nanos or 0
+ return inst
+
+ # pylint: disable=arguments-differ
+
+ @property
+ def nanosecond(self):
+ """Read-only: nanosecond precision."""
+ return self._nanosecond
+
+ def rfc3339(self):
+ """Return an RFC 3339-compliant timestamp.
+
+ Returns:
+ (str): Timestamp string according to RFC 3339 spec.
+ """
+ if self._nanosecond == 0:
+ return to_rfc3339(self)
+ nanos = str(self._nanosecond).rjust(9, '0').rstrip("0")
+ return "{}.{}Z".format(self.strftime(_RFC3339_NO_FRACTION), nanos)
+
+ @classmethod
+ def from_rfc3339(cls, stamp):
+ """Parse RFC 3339-compliant timestamp, preserving nanoseconds.
+
+ Args:
+ stamp (str): RFC 3339 stamp, with up to nanosecond precision
+
+ Returns:
+ :class:`DatetimeWithNanoseconds`:
+ an instance matching the timestamp string
+
+ Raises:
+ ValueError: if `stamp` does not match the expected format
+ """
+ with_nanos = _RFC3339_NANOS.match(stamp)
+ if with_nanos is None:
+ raise ValueError(
+ "Timestamp: {}, does not match pattern: {}".format(
+ stamp, _RFC3339_NANOS.pattern
+ )
+ )
+ bare = datetime.datetime.strptime(
+ with_nanos.group("no_fraction"), _RFC3339_NO_FRACTION
+ )
+ fraction = with_nanos.group("nanos")
+ if fraction is None:
+ nanos = 0
+ else:
+ scale = 9 - len(fraction)
+ nanos = int(fraction) * (10 ** scale)
+ return cls(
+ bare.year,
+ bare.month,
+ bare.day,
+ bare.hour,
+ bare.minute,
+ bare.second,
+ nanosecond=nanos,
+ tzinfo=pytz.UTC,
+ )
+
+ def timestamp_pb(self):
+ """Return a timestamp message.
+
+ Returns:
+ (:class:`~google.protobuf.timestamp_pb2.Timestamp`): Timestamp message
+ """
+ inst = self if self.tzinfo is not None else self.replace(tzinfo=pytz.UTC)
+ delta = inst - _UTC_EPOCH
+ seconds = int(delta.total_seconds())
+ nanos = self._nanosecond or self.microsecond * 1000
+ return timestamp_pb2.Timestamp(seconds=seconds, nanos=nanos)
+
+ @classmethod
+ def from_timestamp_pb(cls, stamp):
+ """Parse RFC 3339-compliant timestamp, preserving nanoseconds.
+
+ Args:
+ stamp (:class:`~google.protobuf.timestamp_pb2.Timestamp`): timestamp message
+
+ Returns:
+ :class:`DatetimeWithNanoseconds`:
+ an instance matching the timestamp message
+ """
+ microseconds = int(stamp.seconds * 1e6)
+ bare = from_microseconds(microseconds)
+ return cls(
+ bare.year,
+ bare.month,
+ bare.day,
+ bare.hour,
+ bare.minute,
+ bare.second,
+ nanosecond=stamp.nanos,
+ tzinfo=pytz.UTC,
+ )
diff --git a/api_core/google/api_core/exceptions.py b/api_core/google/api_core/exceptions.py
new file mode 100644
index 000000000000..eed4ee40eee5
--- /dev/null
+++ b/api_core/google/api_core/exceptions.py
@@ -0,0 +1,462 @@
+# Copyright 2014 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Exceptions raised by Google API core & clients.
+
+This module provides base classes for all errors raised by libraries based
+on :mod:`google.api_core`, including both HTTP and gRPC clients.
+"""
+
+from __future__ import absolute_import
+from __future__ import unicode_literals
+
+import six
+from six.moves import http_client
+
+try:
+ import grpc
+except ImportError: # pragma: NO COVER
+ grpc = None
+
+# Lookup tables for mapping exceptions from HTTP and gRPC transports.
+# Populated by _APICallErrorMeta
+_HTTP_CODE_TO_EXCEPTION = {}
+_GRPC_CODE_TO_EXCEPTION = {}
+
+
+class GoogleAPIError(Exception):
+ """Base class for all exceptions raised by Google API Clients."""
+
+ pass
+
+
+@six.python_2_unicode_compatible
+class RetryError(GoogleAPIError):
+ """Raised when a function has exhausted all of its available retries.
+
+ Args:
+ message (str): The exception message.
+ cause (Exception): The last exception raised when retring the
+ function.
+ """
+
+ def __init__(self, message, cause):
+ super(RetryError, self).__init__(message)
+ self.message = message
+ self._cause = cause
+
+ @property
+ def cause(self):
+ """The last exception raised when retrying the function."""
+ return self._cause
+
+ def __str__(self):
+ return "{}, last exception: {}".format(self.message, self.cause)
+
+
+class _GoogleAPICallErrorMeta(type):
+ """Metaclass for registering GoogleAPICallError subclasses."""
+
+ def __new__(mcs, name, bases, class_dict):
+ cls = type.__new__(mcs, name, bases, class_dict)
+ if cls.code is not None:
+ _HTTP_CODE_TO_EXCEPTION.setdefault(cls.code, cls)
+ if cls.grpc_status_code is not None:
+ _GRPC_CODE_TO_EXCEPTION.setdefault(cls.grpc_status_code, cls)
+ return cls
+
+
+@six.python_2_unicode_compatible
+@six.add_metaclass(_GoogleAPICallErrorMeta)
+class GoogleAPICallError(GoogleAPIError):
+ """Base class for exceptions raised by calling API methods.
+
+ Args:
+ message (str): The exception message.
+ errors (Sequence[Any]): An optional list of error details.
+ response (Union[requests.Request, grpc.Call]): The response or
+ gRPC call metadata.
+ """
+
+ code = None
+ """Optional[int]: The HTTP status code associated with this error.
+
+ This may be ``None`` if the exception does not have a direct mapping
+ to an HTTP error.
+
+ See http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
+ """
+
+ grpc_status_code = None
+ """Optional[grpc.StatusCode]: The gRPC status code associated with this
+ error.
+
+ This may be ``None`` if the exception does not match up to a gRPC error.
+ """
+
+ def __init__(self, message, errors=(), response=None):
+ super(GoogleAPICallError, self).__init__(message)
+ self.message = message
+ """str: The exception message."""
+ self._errors = errors
+ self._response = response
+
+ def __str__(self):
+ return "{} {}".format(self.code, self.message)
+
+ @property
+ def errors(self):
+ """Detailed error information.
+
+ Returns:
+ Sequence[Any]: A list of additional error details.
+ """
+ return list(self._errors)
+
+ @property
+ def response(self):
+ """Optional[Union[requests.Request, grpc.Call]]: The response or
+ gRPC call metadata."""
+ return self._response
+
+
+class Redirection(GoogleAPICallError):
+ """Base class for for all redirection (HTTP 3xx) responses."""
+
+
+class MovedPermanently(Redirection):
+ """Exception mapping a ``301 Moved Permanently`` response."""
+
+ code = http_client.MOVED_PERMANENTLY
+
+
+class NotModified(Redirection):
+ """Exception mapping a ``304 Not Modified`` response."""
+
+ code = http_client.NOT_MODIFIED
+
+
+class TemporaryRedirect(Redirection):
+ """Exception mapping a ``307 Temporary Redirect`` response."""
+
+ code = http_client.TEMPORARY_REDIRECT
+
+
+class ResumeIncomplete(Redirection):
+ """Exception mapping a ``308 Resume Incomplete`` response.
+
+ .. note:: :attr:`http_client.PERMANENT_REDIRECT` is ``308``, but Google
+ APIs differ in their use of this status code.
+ """
+
+ code = 308
+
+
+class ClientError(GoogleAPICallError):
+ """Base class for all client error (HTTP 4xx) responses."""
+
+
+class BadRequest(ClientError):
+ """Exception mapping a ``400 Bad Request`` response."""
+
+ code = http_client.BAD_REQUEST
+
+
+class InvalidArgument(BadRequest):
+ """Exception mapping a :attr:`grpc.StatusCode.INVALID_ARGUMENT` error."""
+
+ grpc_status_code = grpc.StatusCode.INVALID_ARGUMENT if grpc is not None else None
+
+
+class FailedPrecondition(BadRequest):
+ """Exception mapping a :attr:`grpc.StatusCode.FAILED_PRECONDITION`
+ error."""
+
+ grpc_status_code = grpc.StatusCode.FAILED_PRECONDITION if grpc is not None else None
+
+
+class OutOfRange(BadRequest):
+ """Exception mapping a :attr:`grpc.StatusCode.OUT_OF_RANGE` error."""
+
+ grpc_status_code = grpc.StatusCode.OUT_OF_RANGE if grpc is not None else None
+
+
+class Unauthorized(ClientError):
+ """Exception mapping a ``401 Unauthorized`` response."""
+
+ code = http_client.UNAUTHORIZED
+
+
+class Unauthenticated(Unauthorized):
+ """Exception mapping a :attr:`grpc.StatusCode.UNAUTHENTICATED` error."""
+
+ grpc_status_code = grpc.StatusCode.UNAUTHENTICATED if grpc is not None else None
+
+
+class Forbidden(ClientError):
+ """Exception mapping a ``403 Forbidden`` response."""
+
+ code = http_client.FORBIDDEN
+
+
+class PermissionDenied(Forbidden):
+ """Exception mapping a :attr:`grpc.StatusCode.PERMISSION_DENIED` error."""
+
+ grpc_status_code = grpc.StatusCode.PERMISSION_DENIED if grpc is not None else None
+
+
+class NotFound(ClientError):
+ """Exception mapping a ``404 Not Found`` response or a
+ :attr:`grpc.StatusCode.NOT_FOUND` error."""
+
+ code = http_client.NOT_FOUND
+ grpc_status_code = grpc.StatusCode.NOT_FOUND if grpc is not None else None
+
+
+class MethodNotAllowed(ClientError):
+ """Exception mapping a ``405 Method Not Allowed`` response."""
+
+ code = http_client.METHOD_NOT_ALLOWED
+
+
+class Conflict(ClientError):
+ """Exception mapping a ``409 Conflict`` response."""
+
+ code = http_client.CONFLICT
+
+
+class AlreadyExists(Conflict):
+ """Exception mapping a :attr:`grpc.StatusCode.ALREADY_EXISTS` error."""
+
+ grpc_status_code = grpc.StatusCode.ALREADY_EXISTS if grpc is not None else None
+
+
+class Aborted(Conflict):
+ """Exception mapping a :attr:`grpc.StatusCode.ABORTED` error."""
+
+ grpc_status_code = grpc.StatusCode.ABORTED if grpc is not None else None
+
+
+class LengthRequired(ClientError):
+ """Exception mapping a ``411 Length Required`` response."""
+
+ code = http_client.LENGTH_REQUIRED
+
+
+class PreconditionFailed(ClientError):
+ """Exception mapping a ``412 Precondition Failed`` response."""
+
+ code = http_client.PRECONDITION_FAILED
+
+
+class RequestRangeNotSatisfiable(ClientError):
+ """Exception mapping a ``416 Request Range Not Satisfiable`` response."""
+
+ code = http_client.REQUESTED_RANGE_NOT_SATISFIABLE
+
+
+class TooManyRequests(ClientError):
+ """Exception mapping a ``429 Too Many Requests`` response."""
+
+ # http_client does not define a constant for this in Python 2.
+ code = 429
+
+
+class ResourceExhausted(TooManyRequests):
+ """Exception mapping a :attr:`grpc.StatusCode.RESOURCE_EXHAUSTED` error."""
+
+ grpc_status_code = grpc.StatusCode.RESOURCE_EXHAUSTED if grpc is not None else None
+
+
+class Cancelled(ClientError):
+ """Exception mapping a :attr:`grpc.StatusCode.CANCELLED` error."""
+
+ # This maps to HTTP status code 499. See
+ # https://github.com/googleapis/googleapis/blob/master/google/rpc\
+ # /code.proto
+ code = 499
+ grpc_status_code = grpc.StatusCode.CANCELLED if grpc is not None else None
+
+
+class ServerError(GoogleAPICallError):
+ """Base for 5xx responses."""
+
+
+class InternalServerError(ServerError):
+ """Exception mapping a ``500 Internal Server Error`` response. or a
+ :attr:`grpc.StatusCode.INTERNAL` error."""
+
+ code = http_client.INTERNAL_SERVER_ERROR
+ grpc_status_code = grpc.StatusCode.INTERNAL if grpc is not None else None
+
+
+class Unknown(ServerError):
+ """Exception mapping a :attr:`grpc.StatusCode.UNKNOWN` error."""
+
+ grpc_status_code = grpc.StatusCode.UNKNOWN if grpc is not None else None
+
+
+class DataLoss(ServerError):
+ """Exception mapping a :attr:`grpc.StatusCode.DATA_LOSS` error."""
+
+ grpc_status_code = grpc.StatusCode.DATA_LOSS if grpc is not None else None
+
+
+class MethodNotImplemented(ServerError):
+ """Exception mapping a ``501 Not Implemented`` response or a
+ :attr:`grpc.StatusCode.UNIMPLEMENTED` error."""
+
+ code = http_client.NOT_IMPLEMENTED
+ grpc_status_code = grpc.StatusCode.UNIMPLEMENTED if grpc is not None else None
+
+
+class BadGateway(ServerError):
+ """Exception mapping a ``502 Bad Gateway`` response."""
+
+ code = http_client.BAD_GATEWAY
+
+
+class ServiceUnavailable(ServerError):
+ """Exception mapping a ``503 Service Unavailable`` response or a
+ :attr:`grpc.StatusCode.UNAVAILABLE` error."""
+
+ code = http_client.SERVICE_UNAVAILABLE
+ grpc_status_code = grpc.StatusCode.UNAVAILABLE if grpc is not None else None
+
+
+class GatewayTimeout(ServerError):
+ """Exception mapping a ``504 Gateway Timeout`` response."""
+
+ code = http_client.GATEWAY_TIMEOUT
+
+
+class DeadlineExceeded(GatewayTimeout):
+ """Exception mapping a :attr:`grpc.StatusCode.DEADLINE_EXCEEDED` error."""
+
+ grpc_status_code = grpc.StatusCode.DEADLINE_EXCEEDED if grpc is not None else None
+
+
+def exception_class_for_http_status(status_code):
+ """Return the exception class for a specific HTTP status code.
+
+ Args:
+ status_code (int): The HTTP status code.
+
+ Returns:
+ :func:`type`: the appropriate subclass of :class:`GoogleAPICallError`.
+ """
+ return _HTTP_CODE_TO_EXCEPTION.get(status_code, GoogleAPICallError)
+
+
+def from_http_status(status_code, message, **kwargs):
+ """Create a :class:`GoogleAPICallError` from an HTTP status code.
+
+ Args:
+ status_code (int): The HTTP status code.
+ message (str): The exception message.
+ kwargs: Additional arguments passed to the :class:`GoogleAPICallError`
+ constructor.
+
+ Returns:
+ GoogleAPICallError: An instance of the appropriate subclass of
+ :class:`GoogleAPICallError`.
+ """
+ error_class = exception_class_for_http_status(status_code)
+ error = error_class(message, **kwargs)
+
+ if error.code is None:
+ error.code = status_code
+
+ return error
+
+
+def from_http_response(response):
+ """Create a :class:`GoogleAPICallError` from a :class:`requests.Response`.
+
+ Args:
+ response (requests.Response): The HTTP response.
+
+ Returns:
+ GoogleAPICallError: An instance of the appropriate subclass of
+ :class:`GoogleAPICallError`, with the message and errors populated
+ from the response.
+ """
+ try:
+ payload = response.json()
+ except ValueError:
+ payload = {"error": {"message": response.text or "unknown error"}}
+
+ error_message = payload.get("error", {}).get("message", "unknown error")
+ errors = payload.get("error", {}).get("errors", ())
+
+ message = "{method} {url}: {error}".format(
+ method=response.request.method, url=response.request.url, error=error_message
+ )
+
+ exception = from_http_status(
+ response.status_code, message, errors=errors, response=response
+ )
+ return exception
+
+
+def exception_class_for_grpc_status(status_code):
+ """Return the exception class for a specific :class:`grpc.StatusCode`.
+
+ Args:
+ status_code (grpc.StatusCode): The gRPC status code.
+
+ Returns:
+ :func:`type`: the appropriate subclass of :class:`GoogleAPICallError`.
+ """
+ return _GRPC_CODE_TO_EXCEPTION.get(status_code, GoogleAPICallError)
+
+
+def from_grpc_status(status_code, message, **kwargs):
+ """Create a :class:`GoogleAPICallError` from a :class:`grpc.StatusCode`.
+
+ Args:
+ status_code (grpc.StatusCode): The gRPC status code.
+ message (str): The exception message.
+ kwargs: Additional arguments passed to the :class:`GoogleAPICallError`
+ constructor.
+
+ Returns:
+ GoogleAPICallError: An instance of the appropriate subclass of
+ :class:`GoogleAPICallError`.
+ """
+ error_class = exception_class_for_grpc_status(status_code)
+ error = error_class(message, **kwargs)
+
+ if error.grpc_status_code is None:
+ error.grpc_status_code = status_code
+
+ return error
+
+
+def from_grpc_error(rpc_exc):
+ """Create a :class:`GoogleAPICallError` from a :class:`grpc.RpcError`.
+
+ Args:
+ rpc_exc (grpc.RpcError): The gRPC error.
+
+ Returns:
+ GoogleAPICallError: An instance of the appropriate subclass of
+ :class:`GoogleAPICallError`.
+ """
+ if isinstance(rpc_exc, grpc.Call):
+ return from_grpc_status(
+ rpc_exc.code(), rpc_exc.details(), errors=(rpc_exc,), response=rpc_exc
+ )
+ else:
+ return GoogleAPICallError(str(rpc_exc), errors=(rpc_exc,), response=rpc_exc)
diff --git a/api_core/google/api_core/future/__init__.py b/api_core/google/api_core/future/__init__.py
new file mode 100644
index 000000000000..3768b2c53f53
--- /dev/null
+++ b/api_core/google/api_core/future/__init__.py
@@ -0,0 +1,19 @@
+# Copyright 2017, Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Futures for dealing with asynchronous operations."""
+
+from google.api_core.future.base import Future
+
+__all__ = ["Future"]
diff --git a/api_core/google/api_core/future/_helpers.py b/api_core/google/api_core/future/_helpers.py
new file mode 100644
index 000000000000..9e88ca9d561d
--- /dev/null
+++ b/api_core/google/api_core/future/_helpers.py
@@ -0,0 +1,39 @@
+# Copyright 2017, Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Private helpers for futures."""
+
+import logging
+import threading
+
+
+_LOGGER = logging.getLogger(__name__)
+
+
+def start_daemon_thread(*args, **kwargs):
+ """Starts a thread and marks it as a daemon thread."""
+ thread = threading.Thread(*args, **kwargs)
+ thread.daemon = True
+ thread.start()
+ return thread
+
+
+def safe_invoke_callback(callback, *args, **kwargs):
+ """Invoke a callback, swallowing and logging any exceptions."""
+ # pylint: disable=bare-except
+ # We intentionally want to swallow all exceptions.
+ try:
+ return callback(*args, **kwargs)
+ except Exception:
+ _LOGGER.exception("Error while executing Future callback.")
diff --git a/api_core/google/api_core/future/base.py b/api_core/google/api_core/future/base.py
new file mode 100644
index 000000000000..e7888ca3b07b
--- /dev/null
+++ b/api_core/google/api_core/future/base.py
@@ -0,0 +1,67 @@
+# Copyright 2017, Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Abstract and helper bases for Future implementations."""
+
+import abc
+
+import six
+
+
+@six.add_metaclass(abc.ABCMeta)
+class Future(object):
+ # pylint: disable=missing-docstring
+ # We inherit the interfaces here from concurrent.futures.
+
+ """Future interface.
+
+ This interface is based on :class:`concurrent.futures.Future`.
+ """
+
+ @abc.abstractmethod
+ def cancel(self):
+ raise NotImplementedError()
+
+ @abc.abstractmethod
+ def cancelled(self):
+ raise NotImplementedError()
+
+ @abc.abstractmethod
+ def running(self):
+ raise NotImplementedError()
+
+ @abc.abstractmethod
+ def done(self):
+ raise NotImplementedError()
+
+ @abc.abstractmethod
+ def result(self, timeout=None):
+ raise NotImplementedError()
+
+ @abc.abstractmethod
+ def exception(self, timeout=None):
+ raise NotImplementedError()
+
+ @abc.abstractmethod
+ def add_done_callback(self, fn):
+ # pylint: disable=invalid-name
+ raise NotImplementedError()
+
+ @abc.abstractmethod
+ def set_result(self, result):
+ raise NotImplementedError()
+
+ @abc.abstractmethod
+ def set_exception(self, exception):
+ raise NotImplementedError()
diff --git a/api_core/google/api_core/future/polling.py b/api_core/google/api_core/future/polling.py
new file mode 100644
index 000000000000..4266e9e721e3
--- /dev/null
+++ b/api_core/google/api_core/future/polling.py
@@ -0,0 +1,183 @@
+# Copyright 2017, Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Abstract and helper bases for Future implementations."""
+
+import abc
+import concurrent.futures
+
+from google.api_core import exceptions
+from google.api_core import retry
+from google.api_core.future import _helpers
+from google.api_core.future import base
+
+
+class _OperationNotComplete(Exception):
+ """Private exception used for polling via retry."""
+
+ pass
+
+
+RETRY_PREDICATE = retry.if_exception_type(
+ _OperationNotComplete,
+ exceptions.TooManyRequests,
+ exceptions.InternalServerError,
+ exceptions.BadGateway,
+)
+DEFAULT_RETRY = retry.Retry(predicate=RETRY_PREDICATE)
+
+
+class PollingFuture(base.Future):
+ """A Future that needs to poll some service to check its status.
+
+ The :meth:`done` method should be implemented by subclasses. The polling
+ behavior will repeatedly call ``done`` until it returns True.
+
+ .. note: Privacy here is intended to prevent the final class from
+ overexposing, not to prevent subclasses from accessing methods.
+
+ Args:
+ retry (google.api_core.retry.Retry): The retry configuration used
+ when polling. This can be used to control how often :meth:`done`
+ is polled. Regardless of the retry's ``deadline``, it will be
+ overridden by the ``timeout`` argument to :meth:`result`.
+ """
+
+ def __init__(self, retry=DEFAULT_RETRY):
+ super(PollingFuture, self).__init__()
+ self._retry = retry
+ self._result = None
+ self._exception = None
+ self._result_set = False
+ """bool: Set to True when the result has been set via set_result or
+ set_exception."""
+ self._polling_thread = None
+ self._done_callbacks = []
+
+ @abc.abstractmethod
+ def done(self):
+ """Checks to see if the operation is complete.
+
+ Returns:
+ bool: True if the operation is complete, False otherwise.
+ """
+ # pylint: disable=redundant-returns-doc, missing-raises-doc
+ raise NotImplementedError()
+
+ def _done_or_raise(self):
+ """Check if the future is done and raise if it's not."""
+ if not self.done():
+ raise _OperationNotComplete()
+
+ def running(self):
+ """True if the operation is currently running."""
+ return not self.done()
+
+ def _blocking_poll(self, timeout=None):
+ """Poll and wait for the Future to be resolved.
+
+ Args:
+ timeout (int):
+ How long (in seconds) to wait for the operation to complete.
+ If None, wait indefinitely.
+ """
+ if self._result_set:
+ return
+
+ retry_ = self._retry.with_deadline(timeout)
+
+ try:
+ retry_(self._done_or_raise)()
+ except exceptions.RetryError:
+ raise concurrent.futures.TimeoutError(
+ "Operation did not complete within the designated " "timeout."
+ )
+
+ def result(self, timeout=None):
+ """Get the result of the operation, blocking if necessary.
+
+ Args:
+ timeout (int):
+ How long (in seconds) to wait for the operation to complete.
+ If None, wait indefinitely.
+
+ Returns:
+ google.protobuf.Message: The Operation's result.
+
+ Raises:
+ google.api_core.GoogleAPICallError: If the operation errors or if
+ the timeout is reached before the operation completes.
+ """
+ self._blocking_poll(timeout=timeout)
+
+ if self._exception is not None:
+ # pylint: disable=raising-bad-type
+ # Pylint doesn't recognize that this is valid in this case.
+ raise self._exception
+
+ return self._result
+
+ def exception(self, timeout=None):
+ """Get the exception from the operation, blocking if necessary.
+
+ Args:
+ timeout (int): How long to wait for the operation to complete.
+ If None, wait indefinitely.
+
+ Returns:
+ Optional[google.api_core.GoogleAPICallError]: The operation's
+ error.
+ """
+ self._blocking_poll(timeout=timeout)
+ return self._exception
+
+ def add_done_callback(self, fn):
+ """Add a callback to be executed when the operation is complete.
+
+ If the operation is not already complete, this will start a helper
+ thread to poll for the status of the operation in the background.
+
+ Args:
+ fn (Callable[Future]): The callback to execute when the operation
+ is complete.
+ """
+ if self._result_set:
+ _helpers.safe_invoke_callback(fn, self)
+ return
+
+ self._done_callbacks.append(fn)
+
+ if self._polling_thread is None:
+ # The polling thread will exit on its own as soon as the operation
+ # is done.
+ self._polling_thread = _helpers.start_daemon_thread(
+ target=self._blocking_poll
+ )
+
+ def _invoke_callbacks(self, *args, **kwargs):
+ """Invoke all done callbacks."""
+ for callback in self._done_callbacks:
+ _helpers.safe_invoke_callback(callback, *args, **kwargs)
+
+ def set_result(self, result):
+ """Set the Future's result."""
+ self._result = result
+ self._result_set = True
+ self._invoke_callbacks(self)
+
+ def set_exception(self, exception):
+ """Set the Future's exception."""
+ self._exception = exception
+ self._result_set = True
+ self._invoke_callbacks(self)
diff --git a/api_core/google/api_core/gapic_v1/__init__.py b/api_core/google/api_core/gapic_v1/__init__.py
new file mode 100644
index 000000000000..e7a7a686f58e
--- /dev/null
+++ b/api_core/google/api_core/gapic_v1/__init__.py
@@ -0,0 +1,20 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from google.api_core.gapic_v1 import client_info
+from google.api_core.gapic_v1 import config
+from google.api_core.gapic_v1 import method
+from google.api_core.gapic_v1 import routing_header
+
+__all__ = ["client_info", "config", "method", "routing_header"]
diff --git a/api_core/google/api_core/gapic_v1/client_info.py b/api_core/google/api_core/gapic_v1/client_info.py
new file mode 100644
index 000000000000..bdc2ce440de3
--- /dev/null
+++ b/api_core/google/api_core/gapic_v1/client_info.py
@@ -0,0 +1,55 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Helpers for providing client information.
+
+Client information is used to send information about the calling client,
+such as the library and Python version, to API services.
+"""
+
+from google.api_core import client_info
+
+
+METRICS_METADATA_KEY = "x-goog-api-client"
+
+
+class ClientInfo(client_info.ClientInfo):
+ """Client information used to generate a user-agent for API calls.
+
+ This user-agent information is sent along with API calls to allow the
+ receiving service to do analytics on which versions of Python and Google
+ libraries are being used.
+
+ Args:
+ python_version (str): The Python interpreter version, for example,
+ ``'2.7.13'``.
+ grpc_version (Optional[str]): The gRPC library version.
+ api_core_version (str): The google-api-core library version.
+ gapic_version (Optional[str]): The sversion of gapic-generated client
+ library, if the library was generated by gapic.
+ client_library_version (Optional[str]): The version of the client
+ library, generally used if the client library was not generated
+ by gapic or if additional functionality was built on top of
+ a gapic client library.
+ user_agent (Optional[str]): Prefix to the user agent header. This is
+ used to supply information such as application name or partner tool.
+ Recommended format: ``application-or-tool-ID/major.minor.version``.
+ """
+
+ def to_grpc_metadata(self):
+ """Returns the gRPC metadata for this client info."""
+ return (METRICS_METADATA_KEY, self.to_user_agent())
+
+
+DEFAULT_CLIENT_INFO = ClientInfo()
diff --git a/api_core/google/api_core/gapic_v1/config.py b/api_core/google/api_core/gapic_v1/config.py
new file mode 100644
index 000000000000..3a3eb15fdd44
--- /dev/null
+++ b/api_core/google/api_core/gapic_v1/config.py
@@ -0,0 +1,167 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Helpers for loading gapic configuration data.
+
+The Google API generator creates supplementary configuration for each RPC
+method to tell the client library how to deal with retries and timeouts.
+"""
+
+import collections
+
+import grpc
+import six
+
+from google.api_core import exceptions
+from google.api_core import retry
+from google.api_core import timeout
+
+
+_MILLIS_PER_SECOND = 1000.0
+
+
+def _exception_class_for_grpc_status_name(name):
+ """Returns the Google API exception class for a gRPC error code name.
+
+ Args:
+ name (str): The name of the gRPC status code, for example,
+ ``UNAVAILABLE``.
+
+ Returns:
+ :func:`type`: The appropriate subclass of
+ :class:`google.api_core.exceptions.GoogleAPICallError`.
+ """
+ return exceptions.exception_class_for_grpc_status(getattr(grpc.StatusCode, name))
+
+
+def _retry_from_retry_config(retry_params, retry_codes):
+ """Creates a Retry object given a gapic retry configuration.
+
+ Args:
+ retry_params (dict): The retry parameter values, for example::
+
+ {
+ "initial_retry_delay_millis": 1000,
+ "retry_delay_multiplier": 2.5,
+ "max_retry_delay_millis": 120000,
+ "initial_rpc_timeout_millis": 120000,
+ "rpc_timeout_multiplier": 1.0,
+ "max_rpc_timeout_millis": 120000,
+ "total_timeout_millis": 600000
+ }
+
+ retry_codes (sequence[str]): The list of retryable gRPC error code
+ names.
+
+ Returns:
+ google.api_core.retry.Retry: The default retry object for the method.
+ """
+ exception_classes = [
+ _exception_class_for_grpc_status_name(code) for code in retry_codes
+ ]
+ return retry.Retry(
+ retry.if_exception_type(*exception_classes),
+ initial=(retry_params["initial_retry_delay_millis"] / _MILLIS_PER_SECOND),
+ maximum=(retry_params["max_retry_delay_millis"] / _MILLIS_PER_SECOND),
+ multiplier=retry_params["retry_delay_multiplier"],
+ deadline=retry_params["total_timeout_millis"] / _MILLIS_PER_SECOND,
+ )
+
+
+def _timeout_from_retry_config(retry_params):
+ """Creates a ExponentialTimeout object given a gapic retry configuration.
+
+ Args:
+ retry_params (dict): The retry parameter values, for example::
+
+ {
+ "initial_retry_delay_millis": 1000,
+ "retry_delay_multiplier": 2.5,
+ "max_retry_delay_millis": 120000,
+ "initial_rpc_timeout_millis": 120000,
+ "rpc_timeout_multiplier": 1.0,
+ "max_rpc_timeout_millis": 120000,
+ "total_timeout_millis": 600000
+ }
+
+ Returns:
+ google.api_core.retry.ExponentialTimeout: The default time object for
+ the method.
+ """
+ return timeout.ExponentialTimeout(
+ initial=(retry_params["initial_rpc_timeout_millis"] / _MILLIS_PER_SECOND),
+ maximum=(retry_params["max_rpc_timeout_millis"] / _MILLIS_PER_SECOND),
+ multiplier=retry_params["rpc_timeout_multiplier"],
+ deadline=(retry_params["total_timeout_millis"] / _MILLIS_PER_SECOND),
+ )
+
+
+MethodConfig = collections.namedtuple("MethodConfig", ["retry", "timeout"])
+
+
+def parse_method_configs(interface_config):
+ """Creates default retry and timeout objects for each method in a gapic
+ interface config.
+
+ Args:
+ interface_config (Mapping): The interface config section of the full
+ gapic library config. For example, If the full configuration has
+ an interface named ``google.example.v1.ExampleService`` you would
+ pass in just that interface's configuration, for example
+ ``gapic_config['interfaces']['google.example.v1.ExampleService']``.
+
+ Returns:
+ Mapping[str, MethodConfig]: A mapping of RPC method names to their
+ configuration.
+ """
+ # Grab all the retry codes
+ retry_codes_map = {
+ name: retry_codes
+ for name, retry_codes in six.iteritems(interface_config.get("retry_codes", {}))
+ }
+
+ # Grab all of the retry params
+ retry_params_map = {
+ name: retry_params
+ for name, retry_params in six.iteritems(
+ interface_config.get("retry_params", {})
+ )
+ }
+
+ # Iterate through all the API methods and create a flat MethodConfig
+ # instance for each one.
+ method_configs = {}
+
+ for method_name, method_params in six.iteritems(
+ interface_config.get("methods", {})
+ ):
+ retry_params_name = method_params.get("retry_params_name")
+
+ if retry_params_name is not None:
+ retry_params = retry_params_map[retry_params_name]
+ retry_ = _retry_from_retry_config(
+ retry_params, retry_codes_map[method_params["retry_codes_name"]]
+ )
+ timeout_ = _timeout_from_retry_config(retry_params)
+
+ # No retry config, so this is a non-retryable method.
+ else:
+ retry_ = None
+ timeout_ = timeout.ConstantTimeout(
+ method_params["timeout_millis"] / _MILLIS_PER_SECOND
+ )
+
+ method_configs[method_name] = MethodConfig(retry=retry_, timeout=timeout_)
+
+ return method_configs
diff --git a/api_core/google/api_core/gapic_v1/method.py b/api_core/google/api_core/gapic_v1/method.py
new file mode 100644
index 000000000000..49982c0306cb
--- /dev/null
+++ b/api_core/google/api_core/gapic_v1/method.py
@@ -0,0 +1,242 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Helpers for wrapping low-level gRPC methods with common functionality.
+
+This is used by gapic clients to provide common error mapping, retry, timeout,
+pagination, and long-running operations to gRPC methods.
+"""
+
+from google.api_core import general_helpers
+from google.api_core import grpc_helpers
+from google.api_core import timeout
+from google.api_core.gapic_v1 import client_info
+
+USE_DEFAULT_METADATA = object()
+DEFAULT = object()
+"""Sentinel value indicating that a retry or timeout argument was unspecified,
+so the default should be used."""
+
+
+def _is_not_none_or_false(value):
+ return value is not None and value is not False
+
+
+def _apply_decorators(func, decorators):
+ """Apply a list of decorators to a given function.
+
+ ``decorators`` may contain items that are ``None`` or ``False`` which will
+ be ignored.
+ """
+ decorators = filter(_is_not_none_or_false, reversed(decorators))
+
+ for decorator in decorators:
+ func = decorator(func)
+
+ return func
+
+
+def _determine_timeout(default_timeout, specified_timeout, retry):
+ """Determines how timeout should be applied to a wrapped method.
+
+ Args:
+ default_timeout (Optional[Timeout]): The default timeout specified
+ at method creation time.
+ specified_timeout (Optional[Timeout]): The timeout specified at
+ invocation time. If :attr:`DEFAULT`, this will be set to
+ the ``default_timeout``.
+ retry (Optional[Retry]): The retry specified at invocation time.
+
+ Returns:
+ Optional[Timeout]: The timeout to apply to the method or ``None``.
+ """
+ if specified_timeout is DEFAULT:
+ specified_timeout = default_timeout
+
+ if specified_timeout is default_timeout:
+ # If timeout is the default and the default timeout is exponential and
+ # a non-default retry is specified, make sure the timeout's deadline
+ # matches the retry's. This handles the case where the user leaves
+ # the timeout default but specifies a lower deadline via the retry.
+ if (
+ retry
+ and retry is not DEFAULT
+ and isinstance(default_timeout, timeout.ExponentialTimeout)
+ ):
+ return default_timeout.with_deadline(retry._deadline)
+ else:
+ return default_timeout
+
+ # If timeout is specified as a number instead of a Timeout instance,
+ # convert it to a ConstantTimeout.
+ if isinstance(specified_timeout, (int, float)):
+ return timeout.ConstantTimeout(specified_timeout)
+ else:
+ return specified_timeout
+
+
+class _GapicCallable(object):
+ """Callable that applies retry, timeout, and metadata logic.
+
+ Args:
+ target (Callable): The low-level RPC method.
+ retry (google.api_core.retry.Retry): The default retry for the
+ callable. If ``None``, this callable will not retry by default
+ timeout (google.api_core.timeout.Timeout): The default timeout
+ for the callable. If ``None``, this callable will not specify
+ a timeout argument to the low-level RPC method by default.
+ metadata (Sequence[Tuple[str, str]]): Additional metadata that is
+ provided to the RPC method on every invocation. This is merged with
+ any metadata specified during invocation. If ``None``, no
+ additional metadata will be passed to the RPC method.
+ """
+
+ def __init__(self, target, retry, timeout, metadata=None):
+ self._target = target
+ self._retry = retry
+ self._timeout = timeout
+ self._metadata = metadata
+
+ def __call__(self, *args, **kwargs):
+ """Invoke the low-level RPC with retry, timeout, and metadata."""
+ # Note: Due to Python 2 lacking keyword-only arguments we use kwargs to
+ # extract the retry and timeout params.
+ timeout_ = _determine_timeout(
+ self._timeout,
+ kwargs.pop("timeout", self._timeout),
+ # Use only the invocation-specified retry only for this, as we only
+ # want to adjust the timeout deadline if the *user* specified
+ # a different retry.
+ kwargs.get("retry", None),
+ )
+
+ retry = kwargs.pop("retry", self._retry)
+
+ if retry is DEFAULT:
+ retry = self._retry
+
+ # Apply all applicable decorators.
+ wrapped_func = _apply_decorators(self._target, [retry, timeout_])
+
+ # Add the user agent metadata to the call.
+ if self._metadata is not None:
+ metadata = kwargs.get("metadata", [])
+ # Due to the nature of invocation, None should be treated the same
+ # as not specified.
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ metadata.extend(self._metadata)
+ kwargs["metadata"] = metadata
+
+ return wrapped_func(*args, **kwargs)
+
+
+def wrap_method(
+ func,
+ default_retry=None,
+ default_timeout=None,
+ client_info=client_info.DEFAULT_CLIENT_INFO,
+):
+ """Wrap an RPC method with common behavior.
+
+ This applies common error wrapping, retry, and timeout behavior a function.
+ The wrapped function will take optional ``retry`` and ``timeout``
+ arguments.
+
+ For example::
+
+ import google.api_core.gapic_v1.method
+ from google.api_core import retry
+ from google.api_core import timeout
+
+ # The original RPC method.
+ def get_topic(name, timeout=None):
+ request = publisher_v2.GetTopicRequest(name=name)
+ return publisher_stub.GetTopic(request, timeout=timeout)
+
+ default_retry = retry.Retry(deadline=60)
+ default_timeout = timeout.Timeout(deadline=60)
+ wrapped_get_topic = google.api_core.gapic_v1.method.wrap_method(
+ get_topic, default_retry)
+
+ # Execute get_topic with default retry and timeout:
+ response = wrapped_get_topic()
+
+ # Execute get_topic without doing any retying but with the default
+ # timeout:
+ response = wrapped_get_topic(retry=None)
+
+ # Execute get_topic but only retry on 5xx errors:
+ my_retry = retry.Retry(retry.if_exception_type(
+ exceptions.InternalServerError))
+ response = wrapped_get_topic(retry=my_retry)
+
+ The way this works is by late-wrapping the given function with the retry
+ and timeout decorators. Essentially, when ``wrapped_get_topic()`` is
+ called:
+
+ * ``get_topic()`` is first wrapped with the ``timeout`` into
+ ``get_topic_with_timeout``.
+ * ``get_topic_with_timeout`` is wrapped with the ``retry`` into
+ ``get_topic_with_timeout_and_retry()``.
+ * The final ``get_topic_with_timeout_and_retry`` is called passing through
+ the ``args`` and ``kwargs``.
+
+ The callstack is therefore::
+
+ method.__call__() ->
+ Retry.__call__() ->
+ Timeout.__call__() ->
+ wrap_errors() ->
+ get_topic()
+
+ Note that if ``timeout`` or ``retry`` is ``None``, then they are not
+ applied to the function. For example,
+ ``wrapped_get_topic(timeout=None, retry=None)`` is more or less
+ equivalent to just calling ``get_topic`` but with error re-mapping.
+
+ Args:
+ func (Callable[Any]): The function to wrap. It should accept an
+ optional ``timeout`` argument. If ``metadata`` is not ``None``, it
+ should accept a ``metadata`` argument.
+ default_retry (Optional[google.api_core.Retry]): The default retry
+ strategy. If ``None``, the method will not retry by default.
+ default_timeout (Optional[google.api_core.Timeout]): The default
+ timeout strategy. Can also be specified as an int or float. If
+ ``None``, the method will not have timeout specified by default.
+ client_info
+ (Optional[google.api_core.gapic_v1.client_info.ClientInfo]):
+ Client information used to create a user-agent string that's
+ passed as gRPC metadata to the method. If unspecified, then
+ a sane default will be used. If ``None``, then no user agent
+ metadata will be provided to the RPC method.
+
+ Returns:
+ Callable: A new callable that takes optional ``retry`` and ``timeout``
+ arguments and applies the common error mapping, retry, timeout,
+ and metadata behavior to the low-level RPC method.
+ """
+ func = grpc_helpers.wrap_errors(func)
+
+ if client_info is not None:
+ user_agent_metadata = [client_info.to_grpc_metadata()]
+ else:
+ user_agent_metadata = None
+
+ return general_helpers.wraps(func)(
+ _GapicCallable(
+ func, default_retry, default_timeout, metadata=user_agent_metadata
+ )
+ )
diff --git a/api_core/google/api_core/gapic_v1/routing_header.py b/api_core/google/api_core/gapic_v1/routing_header.py
new file mode 100644
index 000000000000..3fb12a6f8e26
--- /dev/null
+++ b/api_core/google/api_core/gapic_v1/routing_header.py
@@ -0,0 +1,62 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Helpers for constructing routing headers.
+
+These headers are used by Google infrastructure to determine how to route
+requests, especially for services that are regional.
+
+Generally, these headers are specified as gRPC metadata.
+"""
+
+import sys
+
+from six.moves.urllib.parse import urlencode
+
+ROUTING_METADATA_KEY = "x-goog-request-params"
+
+
+def to_routing_header(params):
+ """Returns a routing header string for the given request parameters.
+
+ Args:
+ params (Mapping[str, Any]): A dictionary containing the request
+ parameters used for routing.
+
+ Returns:
+ str: The routing header string.
+ """
+ if sys.version_info[0] < 3:
+ # Python 2 does not have the "safe" parameter for urlencode.
+ return urlencode(params).replace("%2F", "/")
+ return urlencode(
+ params,
+ # Per Google API policy (go/api-url-encoding), / is not encoded.
+ safe="/",
+ )
+
+
+def to_grpc_metadata(params):
+ """Returns the gRPC metadata containing the routing headers for the given
+ request parameters.
+
+ Args:
+ params (Mapping[str, Any]): A dictionary containing the request
+ parameters used for routing.
+
+ Returns:
+ Tuple(str, str): The gRPC metadata containing the routing header key
+ and value.
+ """
+ return (ROUTING_METADATA_KEY, to_routing_header(params))
diff --git a/api_core/google/api_core/general_helpers.py b/api_core/google/api_core/general_helpers.py
new file mode 100644
index 000000000000..d2d0c4402311
--- /dev/null
+++ b/api_core/google/api_core/general_helpers.py
@@ -0,0 +1,33 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Helpers for general Python functionality."""
+
+import functools
+
+import six
+
+
+# functools.partial objects lack several attributes present on real function
+# objects. In Python 2 wraps fails on this so use a restricted set instead.
+_PARTIAL_VALID_ASSIGNMENTS = ("__doc__",)
+
+
+def wraps(wrapped):
+ """A functools.wraps helper that handles partial objects on Python 2."""
+ # https://github.com/google/pytype/issues/322
+ if isinstance(wrapped, functools.partial): # pytype: disable=wrong-arg-types
+ return six.wraps(wrapped, assigned=_PARTIAL_VALID_ASSIGNMENTS)
+ else:
+ return six.wraps(wrapped)
diff --git a/api_core/google/api_core/grpc_helpers.py b/api_core/google/api_core/grpc_helpers.py
new file mode 100644
index 000000000000..4d63beb36910
--- /dev/null
+++ b/api_core/google/api_core/grpc_helpers.py
@@ -0,0 +1,384 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Helpers for :mod:`grpc`."""
+
+import collections
+
+import grpc
+import six
+
+from google.api_core import exceptions
+from google.api_core import general_helpers
+import google.auth
+import google.auth.credentials
+import google.auth.transport.grpc
+import google.auth.transport.requests
+
+try:
+ import grpc_gcp
+
+ HAS_GRPC_GCP = True
+except ImportError:
+ HAS_GRPC_GCP = False
+
+# The list of gRPC Callable interfaces that return iterators.
+_STREAM_WRAP_CLASSES = (grpc.UnaryStreamMultiCallable, grpc.StreamStreamMultiCallable)
+
+
+def _patch_callable_name(callable_):
+ """Fix-up gRPC callable attributes.
+
+ gRPC callable lack the ``__name__`` attribute which causes
+ :func:`functools.wraps` to error. This adds the attribute if needed.
+ """
+ if not hasattr(callable_, "__name__"):
+ callable_.__name__ = callable_.__class__.__name__
+
+
+def _wrap_unary_errors(callable_):
+ """Map errors for Unary-Unary and Stream-Unary gRPC callables."""
+ _patch_callable_name(callable_)
+
+ @six.wraps(callable_)
+ def error_remapped_callable(*args, **kwargs):
+ try:
+ return callable_(*args, **kwargs)
+ except grpc.RpcError as exc:
+ six.raise_from(exceptions.from_grpc_error(exc), exc)
+
+ return error_remapped_callable
+
+
+class _StreamingResponseIterator(grpc.Call):
+ def __init__(self, wrapped):
+ self._wrapped = wrapped
+
+ def __iter__(self):
+ """This iterator is also an iterable that returns itself."""
+ return self
+
+ def next(self):
+ """Get the next response from the stream.
+
+ Returns:
+ protobuf.Message: A single response from the stream.
+ """
+ try:
+ return six.next(self._wrapped)
+ except grpc.RpcError as exc:
+ six.raise_from(exceptions.from_grpc_error(exc), exc)
+
+ # Alias needed for Python 2/3 support.
+ __next__ = next
+
+ # grpc.Call & grpc.RpcContext interface
+
+ def add_callback(self, callback):
+ return self._wrapped.add_callback(callback)
+
+ def cancel(self):
+ return self._wrapped.cancel()
+
+ def code(self):
+ return self._wrapped.code()
+
+ def details(self):
+ return self._wrapped.details()
+
+ def initial_metadata(self):
+ return self._wrapped.initial_metadata()
+
+ def is_active(self):
+ return self._wrapped.is_active()
+
+ def time_remaining(self):
+ return self._wrapped.time_remaining()
+
+ def trailing_metadata(self):
+ return self._wrapped.trailing_metadata()
+
+
+def _wrap_stream_errors(callable_):
+ """Wrap errors for Unary-Stream and Stream-Stream gRPC callables.
+
+ The callables that return iterators require a bit more logic to re-map
+ errors when iterating. This wraps both the initial invocation and the
+ iterator of the return value to re-map errors.
+ """
+ _patch_callable_name(callable_)
+
+ @general_helpers.wraps(callable_)
+ def error_remapped_callable(*args, **kwargs):
+ try:
+ result = callable_(*args, **kwargs)
+ return _StreamingResponseIterator(result)
+ except grpc.RpcError as exc:
+ six.raise_from(exceptions.from_grpc_error(exc), exc)
+
+ return error_remapped_callable
+
+
+def wrap_errors(callable_):
+ """Wrap a gRPC callable and map :class:`grpc.RpcErrors` to friendly error
+ classes.
+
+ Errors raised by the gRPC callable are mapped to the appropriate
+ :class:`google.api_core.exceptions.GoogleAPICallError` subclasses.
+ The original `grpc.RpcError` (which is usually also a `grpc.Call`) is
+ available from the ``response`` property on the mapped exception. This
+ is useful for extracting metadata from the original error.
+
+ Args:
+ callable_ (Callable): A gRPC callable.
+
+ Returns:
+ Callable: The wrapped gRPC callable.
+ """
+ if isinstance(callable_, _STREAM_WRAP_CLASSES):
+ return _wrap_stream_errors(callable_)
+ else:
+ return _wrap_unary_errors(callable_)
+
+
+def create_channel(
+ target, credentials=None, scopes=None, ssl_credentials=None, **kwargs
+):
+ """Create a secure channel with credentials.
+
+ Args:
+ target (str): The target service address in the format 'hostname:port'.
+ credentials (google.auth.credentials.Credentials): The credentials. If
+ not specified, then this function will attempt to ascertain the
+ credentials from the environment using :func:`google.auth.default`.
+ scopes (Sequence[str]): A optional list of scopes needed for this
+ service. These are only used when credentials are not specified and
+ are passed to :func:`google.auth.default`.
+ ssl_credentials (grpc.ChannelCredentials): Optional SSL channel
+ credentials. This can be used to specify different certificates.
+ kwargs: Additional key-word args passed to
+ :func:`grpc_gcp.secure_channel` or :func:`grpc.secure_channel`.
+
+ Returns:
+ grpc.Channel: The created channel.
+ """
+ if credentials is None:
+ credentials, _ = google.auth.default(scopes=scopes)
+ else:
+ credentials = google.auth.credentials.with_scopes_if_required(
+ credentials, scopes
+ )
+
+ request = google.auth.transport.requests.Request()
+
+ # Create the metadata plugin for inserting the authorization header.
+ metadata_plugin = google.auth.transport.grpc.AuthMetadataPlugin(
+ credentials, request
+ )
+
+ # Create a set of grpc.CallCredentials using the metadata plugin.
+ google_auth_credentials = grpc.metadata_call_credentials(metadata_plugin)
+
+ if ssl_credentials is None:
+ ssl_credentials = grpc.ssl_channel_credentials()
+
+ # Combine the ssl credentials and the authorization credentials.
+ composite_credentials = grpc.composite_channel_credentials(
+ ssl_credentials, google_auth_credentials
+ )
+
+ if HAS_GRPC_GCP:
+ # If grpc_gcp module is available use grpc_gcp.secure_channel,
+ # otherwise, use grpc.secure_channel to create grpc channel.
+ return grpc_gcp.secure_channel(target, composite_credentials, **kwargs)
+ else:
+ return grpc.secure_channel(target, composite_credentials, **kwargs)
+
+
+_MethodCall = collections.namedtuple(
+ "_MethodCall", ("request", "timeout", "metadata", "credentials")
+)
+
+_ChannelRequest = collections.namedtuple("_ChannelRequest", ("method", "request"))
+
+
+class _CallableStub(object):
+ """Stub for the grpc.*MultiCallable interfaces."""
+
+ def __init__(self, method, channel):
+ self._method = method
+ self._channel = channel
+ self.response = None
+ """Union[protobuf.Message, Callable[protobuf.Message], exception]:
+ The response to give when invoking this callable. If this is a
+ callable, it will be invoked with the request protobuf. If it's an
+ exception, the exception will be raised when this is invoked.
+ """
+ self.responses = None
+ """Iterator[
+ Union[protobuf.Message, Callable[protobuf.Message], exception]]:
+ An iterator of responses. If specified, self.response will be populated
+ on each invocation by calling ``next(self.responses)``."""
+ self.requests = []
+ """List[protobuf.Message]: All requests sent to this callable."""
+ self.calls = []
+ """List[Tuple]: All invocations of this callable. Each tuple is the
+ request, timeout, metadata, and credentials."""
+
+ def __call__(self, request, timeout=None, metadata=None, credentials=None):
+ self._channel.requests.append(_ChannelRequest(self._method, request))
+ self.calls.append(_MethodCall(request, timeout, metadata, credentials))
+ self.requests.append(request)
+
+ response = self.response
+ if self.responses is not None:
+ if response is None:
+ response = next(self.responses)
+ else:
+ raise ValueError(
+ "{method}.response and {method}.responses are mutually "
+ "exclusive.".format(method=self._method)
+ )
+
+ if callable(response):
+ return response(request)
+
+ if isinstance(response, Exception):
+ raise response
+
+ if response is not None:
+ return response
+
+ raise ValueError('Method stub for "{}" has no response.'.format(self._method))
+
+
+def _simplify_method_name(method):
+ """Simplifies a gRPC method name.
+
+ When gRPC invokes the channel to create a callable, it gives a full
+ method name like "/google.pubsub.v1.Publisher/CreateTopic". This
+ returns just the name of the method, in this case "CreateTopic".
+
+ Args:
+ method (str): The name of the method.
+
+ Returns:
+ str: The simplified name of the method.
+ """
+ return method.rsplit("/", 1).pop()
+
+
+class ChannelStub(grpc.Channel):
+ """A testing stub for the grpc.Channel interface.
+
+ This can be used to test any client that eventually uses a gRPC channel
+ to communicate. By passing in a channel stub, you can configure which
+ responses are returned and track which requests are made.
+
+ For example:
+
+ .. code-block:: python
+
+ channel_stub = grpc_helpers.ChannelStub()
+ client = FooClient(channel=channel_stub)
+
+ channel_stub.GetFoo.response = foo_pb2.Foo(name='bar')
+
+ foo = client.get_foo(labels=['baz'])
+
+ assert foo.name == 'bar'
+ assert channel_stub.GetFoo.requests[0].labels = ['baz']
+
+ Each method on the stub can be accessed and configured on the channel.
+ Here's some examples of various configurations:
+
+ .. code-block:: python
+
+ # Return a basic response:
+
+ channel_stub.GetFoo.response = foo_pb2.Foo(name='bar')
+ assert client.get_foo().name == 'bar'
+
+ # Raise an exception:
+ channel_stub.GetFoo.response = NotFound('...')
+
+ with pytest.raises(NotFound):
+ client.get_foo()
+
+ # Use a sequence of responses:
+ channel_stub.GetFoo.responses = iter([
+ foo_pb2.Foo(name='bar'),
+ foo_pb2.Foo(name='baz'),
+ ])
+
+ assert client.get_foo().name == 'bar'
+ assert client.get_foo().name == 'baz'
+
+ # Use a callable
+
+ def on_get_foo(request):
+ return foo_pb2.Foo(name='bar' + request.id)
+
+ channel_stub.GetFoo.response = on_get_foo
+
+ assert client.get_foo(id='123').name == 'bar123'
+ """
+
+ def __init__(self, responses=[]):
+ self.requests = []
+ """Sequence[Tuple[str, protobuf.Message]]: A list of all requests made
+ on this channel in order. The tuple is of method name, request
+ message."""
+ self._method_stubs = {}
+
+ def _stub_for_method(self, method):
+ method = _simplify_method_name(method)
+ self._method_stubs[method] = _CallableStub(method, self)
+ return self._method_stubs[method]
+
+ def __getattr__(self, key):
+ try:
+ return self._method_stubs[key]
+ except KeyError:
+ raise AttributeError
+
+ def unary_unary(self, method, request_serializer=None, response_deserializer=None):
+ """grpc.Channel.unary_unary implementation."""
+ return self._stub_for_method(method)
+
+ def unary_stream(self, method, request_serializer=None, response_deserializer=None):
+ """grpc.Channel.unary_stream implementation."""
+ return self._stub_for_method(method)
+
+ def stream_unary(self, method, request_serializer=None, response_deserializer=None):
+ """grpc.Channel.stream_unary implementation."""
+ return self._stub_for_method(method)
+
+ def stream_stream(
+ self, method, request_serializer=None, response_deserializer=None
+ ):
+ """grpc.Channel.stream_stream implementation."""
+ return self._stub_for_method(method)
+
+ def subscribe(self, callback, try_to_connect=False):
+ """grpc.Channel.subscribe implementation."""
+ pass
+
+ def unsubscribe(self, callback):
+ """grpc.Channel.unsubscribe implementation."""
+ pass
+
+ def close(self):
+ """grpc.Channel.close implementation."""
+ pass
diff --git a/api_core/google/api_core/iam.py b/api_core/google/api_core/iam.py
new file mode 100644
index 000000000000..0e108a30679a
--- /dev/null
+++ b/api_core/google/api_core/iam.py
@@ -0,0 +1,273 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Non-API-specific IAM policy definitions
+
+For allowed roles / permissions, see:
+https://cloud.google.com/iam/docs/understanding-roles
+
+Example usage:
+
+.. code-block:: python
+
+ # ``get_iam_policy`` returns a :class:'~google.api_core.iam.Policy`.
+ policy = resource.get_iam_policy()
+
+ phred = policy.user("phred@example.com")
+ admin_group = policy.group("admins@groups.example.com")
+ account = policy.service_account("account-1234@accounts.example.com")
+ policy["roles/owner"] = [phred, admin_group, account]
+ policy["roles/editor"] = policy.authenticated_users()
+ policy["roles/viewer"] = policy.all_users()
+
+ resource.set_iam_policy(policy)
+"""
+
+import collections
+try:
+ from collections import abc as collections_abc
+except ImportError: # Python 2.7
+ import collections as collections_abc
+import warnings
+
+# Generic IAM roles
+
+OWNER_ROLE = "roles/owner"
+"""Generic role implying all rights to an object."""
+
+EDITOR_ROLE = "roles/editor"
+"""Generic role implying rights to modify an object."""
+
+VIEWER_ROLE = "roles/viewer"
+"""Generic role implying rights to access an object."""
+
+_ASSIGNMENT_DEPRECATED_MSG = """\
+Assigning to '{}' is deprecated. Replace with 'policy[{}] = members."""
+
+
+class Policy(collections_abc.MutableMapping):
+ """IAM Policy
+
+ See
+ https://cloud.google.com/iam/reference/rest/v1/Policy
+
+ Args:
+ etag (Optional[str]): ETag used to identify a unique of the policy
+ version (Optional[int]): unique version of the policy
+ """
+
+ _OWNER_ROLES = (OWNER_ROLE,)
+ """Roles mapped onto our ``owners`` attribute."""
+
+ _EDITOR_ROLES = (EDITOR_ROLE,)
+ """Roles mapped onto our ``editors`` attribute."""
+
+ _VIEWER_ROLES = (VIEWER_ROLE,)
+ """Roles mapped onto our ``viewers`` attribute."""
+
+ def __init__(self, etag=None, version=None):
+ self.etag = etag
+ self.version = version
+ self._bindings = collections.defaultdict(set)
+
+ def __iter__(self):
+ return iter(self._bindings)
+
+ def __len__(self):
+ return len(self._bindings)
+
+ def __getitem__(self, key):
+ return self._bindings[key]
+
+ def __setitem__(self, key, value):
+ self._bindings[key] = set(value)
+
+ def __delitem__(self, key):
+ del self._bindings[key]
+
+ @property
+ def owners(self):
+ """Legacy access to owner role.
+
+ DEPRECATED: use ``policy["roles/owners"]`` instead."""
+ result = set()
+ for role in self._OWNER_ROLES:
+ for member in self._bindings.get(role, ()):
+ result.add(member)
+ return frozenset(result)
+
+ @owners.setter
+ def owners(self, value):
+ """Update owners.
+
+ DEPRECATED: use ``policy["roles/owners"] = value`` instead."""
+ warnings.warn(
+ _ASSIGNMENT_DEPRECATED_MSG.format("owners", OWNER_ROLE), DeprecationWarning
+ )
+ self[OWNER_ROLE] = value
+
+ @property
+ def editors(self):
+ """Legacy access to editor role.
+
+ DEPRECATED: use ``policy["roles/editors"]`` instead."""
+ result = set()
+ for role in self._EDITOR_ROLES:
+ for member in self._bindings.get(role, ()):
+ result.add(member)
+ return frozenset(result)
+
+ @editors.setter
+ def editors(self, value):
+ """Update editors.
+
+ DEPRECATED: use ``policy["roles/editors"] = value`` instead."""
+ warnings.warn(
+ _ASSIGNMENT_DEPRECATED_MSG.format("editors", EDITOR_ROLE),
+ DeprecationWarning,
+ )
+ self[EDITOR_ROLE] = value
+
+ @property
+ def viewers(self):
+ """Legacy access to viewer role.
+
+ DEPRECATED: use ``policy["roles/viewers"]`` instead
+ """
+ result = set()
+ for role in self._VIEWER_ROLES:
+ for member in self._bindings.get(role, ()):
+ result.add(member)
+ return frozenset(result)
+
+ @viewers.setter
+ def viewers(self, value):
+ """Update viewers.
+
+ DEPRECATED: use ``policy["roles/viewers"] = value`` instead.
+ """
+ warnings.warn(
+ _ASSIGNMENT_DEPRECATED_MSG.format("viewers", VIEWER_ROLE),
+ DeprecationWarning,
+ )
+ self[VIEWER_ROLE] = value
+
+ @staticmethod
+ def user(email):
+ """Factory method for a user member.
+
+ Args:
+ email (str): E-mail for this particular user.
+
+ Returns:
+ str: A member string corresponding to the given user.
+ """
+ return "user:%s" % (email,)
+
+ @staticmethod
+ def service_account(email):
+ """Factory method for a service account member.
+
+ Args:
+ email (str): E-mail for this particular service account.
+
+ Returns:
+ str: A member string corresponding to the given service account.
+ """
+ return "serviceAccount:%s" % (email,)
+
+ @staticmethod
+ def group(email):
+ """Factory method for a group member.
+
+ Args:
+ email (str): An id or e-mail for this particular group.
+
+ Returns:
+ str: A member string corresponding to the given group.
+ """
+ return "group:%s" % (email,)
+
+ @staticmethod
+ def domain(domain):
+ """Factory method for a domain member.
+
+ Args:
+ domain (str): The domain for this member.
+
+ Returns:
+ str: A member string corresponding to the given domain.
+ """
+ return "domain:%s" % (domain,)
+
+ @staticmethod
+ def all_users():
+ """Factory method for a member representing all users.
+
+ Returns:
+ str: A member string representing all users.
+ """
+ return "allUsers"
+
+ @staticmethod
+ def authenticated_users():
+ """Factory method for a member representing all authenticated users.
+
+ Returns:
+ str: A member string representing all authenticated users.
+ """
+ return "allAuthenticatedUsers"
+
+ @classmethod
+ def from_api_repr(cls, resource):
+ """Factory: create a policy from a JSON resource.
+
+ Args:
+ resource (dict): policy resource returned by ``getIamPolicy`` API.
+
+ Returns:
+ :class:`Policy`: the parsed policy
+ """
+ version = resource.get("version")
+ etag = resource.get("etag")
+ policy = cls(etag, version)
+ for binding in resource.get("bindings", ()):
+ role = binding["role"]
+ members = sorted(binding["members"])
+ policy[role] = members
+ return policy
+
+ def to_api_repr(self):
+ """Render a JSON policy resource.
+
+ Returns:
+ dict: a resource to be passed to the ``setIamPolicy`` API.
+ """
+ resource = {}
+
+ if self.etag is not None:
+ resource["etag"] = self.etag
+
+ if self.version is not None:
+ resource["version"] = self.version
+
+ if self._bindings:
+ bindings = resource["bindings"] = []
+ for role, members in sorted(self._bindings.items()):
+ if members:
+ bindings.append({"role": role, "members": sorted(set(members))})
+
+ if not bindings:
+ del resource["bindings"]
+
+ return resource
diff --git a/api_core/google/api_core/operation.py b/api_core/google/api_core/operation.py
new file mode 100644
index 000000000000..87f42a973e1b
--- /dev/null
+++ b/api_core/google/api_core/operation.py
@@ -0,0 +1,320 @@
+# Copyright 2016 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Futures for long-running operations returned from Google Cloud APIs.
+
+These futures can be used to synchronously wait for the result of a
+long-running operation using :meth:`Operation.result`:
+
+
+.. code-block:: python
+
+ operation = my_api_client.long_running_method()
+ result = operation.result()
+
+Or asynchronously using callbacks and :meth:`Operation.add_done_callback`:
+
+.. code-block:: python
+
+ operation = my_api_client.long_running_method()
+
+ def my_callback(future):
+ result = future.result()
+
+ operation.add_done_callback(my_callback)
+
+"""
+
+import functools
+import threading
+
+from google.api_core import exceptions
+from google.api_core import protobuf_helpers
+from google.api_core.future import polling
+from google.longrunning import operations_pb2
+from google.protobuf import json_format
+from google.rpc import code_pb2
+
+
+class Operation(polling.PollingFuture):
+ """A Future for interacting with a Google API Long-Running Operation.
+
+ Args:
+ operation (google.longrunning.operations_pb2.Operation): The
+ initial operation.
+ refresh (Callable[[], ~.api_core.operation.Operation]): A callable that
+ returns the latest state of the operation.
+ cancel (Callable[[], None]): A callable that tries to cancel
+ the operation.
+ result_type (func:`type`): The protobuf type for the operation's
+ result.
+ metadata_type (func:`type`): The protobuf type for the operation's
+ metadata.
+ retry (google.api_core.retry.Retry): The retry configuration used
+ when polling. This can be used to control how often :meth:`done`
+ is polled. Regardless of the retry's ``deadline``, it will be
+ overridden by the ``timeout`` argument to :meth:`result`.
+ """
+
+ def __init__(
+ self,
+ operation,
+ refresh,
+ cancel,
+ result_type,
+ metadata_type=None,
+ retry=polling.DEFAULT_RETRY,
+ ):
+ super(Operation, self).__init__(retry=retry)
+ self._operation = operation
+ self._refresh = refresh
+ self._cancel = cancel
+ self._result_type = result_type
+ self._metadata_type = metadata_type
+ self._completion_lock = threading.Lock()
+ # Invoke this in case the operation came back already complete.
+ self._set_result_from_operation()
+
+ @property
+ def operation(self):
+ """google.longrunning.Operation: The current long-running operation."""
+ return self._operation
+
+ @property
+ def metadata(self):
+ """google.protobuf.Message: the current operation metadata."""
+ if not self._operation.HasField("metadata"):
+ return None
+
+ return protobuf_helpers.from_any_pb(
+ self._metadata_type, self._operation.metadata
+ )
+
+ @classmethod
+ def deserialize(self, payload):
+ """Deserialize a ``google.longrunning.Operation`` protocol buffer.
+
+ Args:
+ payload (bytes): A serialized operation protocol buffer.
+
+ Returns:
+ ~.operations_pb2.Operation: An Operation protobuf object.
+ """
+ return operations_pb2.Operation.FromString(payload)
+
+ def _set_result_from_operation(self):
+ """Set the result or exception from the operation if it is complete."""
+ # This must be done in a lock to prevent the polling thread
+ # and main thread from both executing the completion logic
+ # at the same time.
+ with self._completion_lock:
+ # If the operation isn't complete or if the result has already been
+ # set, do not call set_result/set_exception again.
+ # Note: self._result_set is set to True in set_result and
+ # set_exception, in case those methods are invoked directly.
+ if not self._operation.done or self._result_set:
+ return
+
+ if self._operation.HasField("response"):
+ response = protobuf_helpers.from_any_pb(
+ self._result_type, self._operation.response
+ )
+ self.set_result(response)
+ elif self._operation.HasField("error"):
+ exception = exceptions.GoogleAPICallError(
+ self._operation.error.message,
+ errors=(self._operation.error,),
+ response=self._operation,
+ )
+ self.set_exception(exception)
+ else:
+ exception = exceptions.GoogleAPICallError(
+ "Unexpected state: Long-running operation had neither "
+ "response nor error set."
+ )
+ self.set_exception(exception)
+
+ def _refresh_and_update(self):
+ """Refresh the operation and update the result if needed."""
+ # If the currently cached operation is done, no need to make another
+ # RPC as it will not change once done.
+ if not self._operation.done:
+ self._operation = self._refresh()
+ self._set_result_from_operation()
+
+ def done(self):
+ """Checks to see if the operation is complete.
+
+ Returns:
+ bool: True if the operation is complete, False otherwise.
+ """
+ self._refresh_and_update()
+ return self._operation.done
+
+ def cancel(self):
+ """Attempt to cancel the operation.
+
+ Returns:
+ bool: True if the cancel RPC was made, False if the operation is
+ already complete.
+ """
+ if self.done():
+ return False
+
+ self._cancel()
+ return True
+
+ def cancelled(self):
+ """True if the operation was cancelled."""
+ self._refresh_and_update()
+ return (
+ self._operation.HasField("error")
+ and self._operation.error.code == code_pb2.CANCELLED
+ )
+
+
+def _refresh_http(api_request, operation_name):
+ """Refresh an operation using a JSON/HTTP client.
+
+ Args:
+ api_request (Callable): A callable used to make an API request. This
+ should generally be
+ :meth:`google.cloud._http.Connection.api_request`.
+ operation_name (str): The name of the operation.
+
+ Returns:
+ google.longrunning.operations_pb2.Operation: The operation.
+ """
+ path = "operations/{}".format(operation_name)
+ api_response = api_request(method="GET", path=path)
+ return json_format.ParseDict(api_response, operations_pb2.Operation())
+
+
+def _cancel_http(api_request, operation_name):
+ """Cancel an operation using a JSON/HTTP client.
+
+ Args:
+ api_request (Callable): A callable used to make an API request. This
+ should generally be
+ :meth:`google.cloud._http.Connection.api_request`.
+ operation_name (str): The name of the operation.
+ """
+ path = "operations/{}:cancel".format(operation_name)
+ api_request(method="POST", path=path)
+
+
+def from_http_json(operation, api_request, result_type, **kwargs):
+ """Create an operation future using a HTTP/JSON client.
+
+ This interacts with the long-running operations `service`_ (specific
+ to a given API) via `HTTP/JSON`_.
+
+ .. _HTTP/JSON: https://cloud.google.com/speech/reference/rest/\
+ v1beta1/operations#Operation
+
+ Args:
+ operation (dict): Operation as a dictionary.
+ api_request (Callable): A callable used to make an API request. This
+ should generally be
+ :meth:`google.cloud._http.Connection.api_request`.
+ result_type (:func:`type`): The protobuf result type.
+ kwargs: Keyword args passed into the :class:`Operation` constructor.
+
+ Returns:
+ ~.api_core.operation.Operation: The operation future to track the given
+ operation.
+ """
+ operation_proto = json_format.ParseDict(operation, operations_pb2.Operation())
+ refresh = functools.partial(_refresh_http, api_request, operation_proto.name)
+ cancel = functools.partial(_cancel_http, api_request, operation_proto.name)
+ return Operation(operation_proto, refresh, cancel, result_type, **kwargs)
+
+
+def _refresh_grpc(operations_stub, operation_name):
+ """Refresh an operation using a gRPC client.
+
+ Args:
+ operations_stub (google.longrunning.operations_pb2.OperationsStub):
+ The gRPC operations stub.
+ operation_name (str): The name of the operation.
+
+ Returns:
+ google.longrunning.operations_pb2.Operation: The operation.
+ """
+ request_pb = operations_pb2.GetOperationRequest(name=operation_name)
+ return operations_stub.GetOperation(request_pb)
+
+
+def _cancel_grpc(operations_stub, operation_name):
+ """Cancel an operation using a gRPC client.
+
+ Args:
+ operations_stub (google.longrunning.operations_pb2.OperationsStub):
+ The gRPC operations stub.
+ operation_name (str): The name of the operation.
+ """
+ request_pb = operations_pb2.CancelOperationRequest(name=operation_name)
+ operations_stub.CancelOperation(request_pb)
+
+
+def from_grpc(operation, operations_stub, result_type, **kwargs):
+ """Create an operation future using a gRPC client.
+
+ This interacts with the long-running operations `service`_ (specific
+ to a given API) via gRPC.
+
+ .. _service: https://github.com/googleapis/googleapis/blob/\
+ 050400df0fdb16f63b63e9dee53819044bffc857/\
+ google/longrunning/operations.proto#L38
+
+ Args:
+ operation (google.longrunning.operations_pb2.Operation): The operation.
+ operations_stub (google.longrunning.operations_pb2.OperationsStub):
+ The operations stub.
+ result_type (:func:`type`): The protobuf result type.
+ kwargs: Keyword args passed into the :class:`Operation` constructor.
+
+ Returns:
+ ~.api_core.operation.Operation: The operation future to track the given
+ operation.
+ """
+ refresh = functools.partial(_refresh_grpc, operations_stub, operation.name)
+ cancel = functools.partial(_cancel_grpc, operations_stub, operation.name)
+ return Operation(operation, refresh, cancel, result_type, **kwargs)
+
+
+def from_gapic(operation, operations_client, result_type, **kwargs):
+ """Create an operation future from a gapic client.
+
+ This interacts with the long-running operations `service`_ (specific
+ to a given API) via a gapic client.
+
+ .. _service: https://github.com/googleapis/googleapis/blob/\
+ 050400df0fdb16f63b63e9dee53819044bffc857/\
+ google/longrunning/operations.proto#L38
+
+ Args:
+ operation (google.longrunning.operations_pb2.Operation): The operation.
+ operations_client (google.api_core.operations_v1.OperationsClient):
+ The operations client.
+ result_type (:func:`type`): The protobuf result type.
+ kwargs: Keyword args passed into the :class:`Operation` constructor.
+
+ Returns:
+ ~.api_core.operation.Operation: The operation future to track the given
+ operation.
+ """
+ refresh = functools.partial(operations_client.get_operation, operation.name)
+ cancel = functools.partial(operations_client.cancel_operation, operation.name)
+ return Operation(operation, refresh, cancel, result_type, **kwargs)
diff --git a/api_core/google/api_core/operations_v1/__init__.py b/api_core/google/api_core/operations_v1/__init__.py
new file mode 100644
index 000000000000..f0549561b6ff
--- /dev/null
+++ b/api_core/google/api_core/operations_v1/__init__.py
@@ -0,0 +1,19 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Package for interacting with the google.longrunning.operations meta-API."""
+
+from google.api_core.operations_v1.operations_client import OperationsClient
+
+__all__ = ["OperationsClient"]
diff --git a/api_core/google/api_core/operations_v1/operations_client.py b/api_core/google/api_core/operations_v1/operations_client.py
new file mode 100644
index 000000000000..cd2923bb9d09
--- /dev/null
+++ b/api_core/google/api_core/operations_v1/operations_client.py
@@ -0,0 +1,288 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""A client for the google.longrunning.operations meta-API.
+
+This is a client that deals with long-running operations that follow the
+pattern outlined by the `Google API Style Guide`_.
+
+When an API method normally takes long time to complete, it can be designed to
+return ``Operation`` to the client, and the client can use this interface to
+receive the real response asynchronously by polling the operation resource to
+receive the response.
+
+It is not a separate service, but rather an interface implemented by a larger
+service. The protocol-level definition is available at
+`google/longrunning/operations.proto`_. Typically, this will be constructed
+automatically by another client class to deal with operations.
+
+.. _Google API Style Guide:
+ https://cloud.google.com/apis/design/design_pattern
+ s#long_running_operations
+.. _google/longrunning/operations.proto:
+ https://github.com/googleapis/googleapis/blob/master/google/longrunning
+ /operations.proto
+"""
+
+import functools
+
+from google.api_core import gapic_v1
+from google.api_core import page_iterator
+from google.api_core.operations_v1 import operations_client_config
+from google.longrunning import operations_pb2
+
+
+class OperationsClient(object):
+ """Client for interacting with long-running operations within a service.
+
+ Args:
+ channel (grpc.Channel): The gRPC channel associated with the service
+ that implements the ``google.longrunning.operations`` interface.
+ client_config (dict):
+ A dictionary of call options for each method. If not specified
+ the default configuration is used.
+ """
+
+ def __init__(self, channel, client_config=operations_client_config.config):
+ # Create the gRPC client stub.
+ self.operations_stub = operations_pb2.OperationsStub(channel)
+
+ # Create all wrapped methods using the interface configuration.
+ # The interface config contains all of the default settings for retry
+ # and timeout for each RPC method.
+ interfaces = client_config["interfaces"]
+ interface_config = interfaces["google.longrunning.Operations"]
+ method_configs = gapic_v1.config.parse_method_configs(interface_config)
+
+ self._get_operation = gapic_v1.method.wrap_method(
+ self.operations_stub.GetOperation,
+ default_retry=method_configs["GetOperation"].retry,
+ default_timeout=method_configs["GetOperation"].timeout,
+ )
+
+ self._list_operations = gapic_v1.method.wrap_method(
+ self.operations_stub.ListOperations,
+ default_retry=method_configs["ListOperations"].retry,
+ default_timeout=method_configs["ListOperations"].timeout,
+ )
+
+ self._cancel_operation = gapic_v1.method.wrap_method(
+ self.operations_stub.CancelOperation,
+ default_retry=method_configs["CancelOperation"].retry,
+ default_timeout=method_configs["CancelOperation"].timeout,
+ )
+
+ self._delete_operation = gapic_v1.method.wrap_method(
+ self.operations_stub.DeleteOperation,
+ default_retry=method_configs["DeleteOperation"].retry,
+ default_timeout=method_configs["DeleteOperation"].timeout,
+ )
+
+ # Service calls
+ def get_operation(
+ self, name, retry=gapic_v1.method.DEFAULT, timeout=gapic_v1.method.DEFAULT
+ ):
+ """Gets the latest state of a long-running operation.
+
+ Clients can use this method to poll the operation result at intervals
+ as recommended by the API service.
+
+ Example:
+ >>> from google.api_core import operations_v1
+ >>> api = operations_v1.OperationsClient()
+ >>> name = ''
+ >>> response = api.get_operation(name)
+
+ Args:
+ name (str): The name of the operation resource.
+ retry (google.api_core.retry.Retry): The retry strategy to use
+ when invoking the RPC. If unspecified, the default retry from
+ the client configuration will be used. If ``None``, then this
+ method will not retry the RPC at all.
+ timeout (float): The amount of time in seconds to wait for the RPC
+ to complete. Note that if ``retry`` is used, this timeout
+ applies to each individual attempt and the overall time it
+ takes for this method to complete may be longer. If
+ unspecified, the the default timeout in the client
+ configuration is used. If ``None``, then the RPC method will
+ not time out.
+
+ Returns:
+ google.longrunning.operations_pb2.Operation: The state of the
+ operation.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If an error occurred
+ while invoking the RPC, the appropriate ``GoogleAPICallError``
+ subclass will be raised.
+ """
+ request = operations_pb2.GetOperationRequest(name=name)
+ return self._get_operation(request, retry=retry, timeout=timeout)
+
+ def list_operations(
+ self,
+ name,
+ filter_,
+ retry=gapic_v1.method.DEFAULT,
+ timeout=gapic_v1.method.DEFAULT,
+ ):
+ """
+ Lists operations that match the specified filter in the request.
+
+ Example:
+ >>> from google.api_core import operations_v1
+ >>> api = operations_v1.OperationsClient()
+ >>> name = ''
+ >>>
+ >>> # Iterate over all results
+ >>> for operation in api.list_operations(name):
+ >>> # process operation
+ >>> pass
+ >>>
+ >>> # Or iterate over results one page at a time
+ >>> iter = api.list_operations(name)
+ >>> for page in iter.pages:
+ >>> for operation in page:
+ >>> # process operation
+ >>> pass
+
+ Args:
+ name (str): The name of the operation collection.
+ filter_ (str): The standard list filter.
+ retry (google.api_core.retry.Retry): The retry strategy to use
+ when invoking the RPC. If unspecified, the default retry from
+ the client configuration will be used. If ``None``, then this
+ method will not retry the RPC at all.
+ timeout (float): The amount of time in seconds to wait for the RPC
+ to complete. Note that if ``retry`` is used, this timeout
+ applies to each individual attempt and the overall time it
+ takes for this method to complete may be longer. If
+ unspecified, the the default timeout in the client
+ configuration is used. If ``None``, then the RPC method will
+ not time out.
+
+ Returns:
+ google.api_core.page_iterator.Iterator: An iterator that yields
+ :class:`google.longrunning.operations_pb2.Operation` instances.
+
+ Raises:
+ google.api_core.exceptions.MethodNotImplemented: If the server
+ does not support this method. Services are not required to
+ implement this method.
+ google.api_core.exceptions.GoogleAPICallError: If an error occurred
+ while invoking the RPC, the appropriate ``GoogleAPICallError``
+ subclass will be raised.
+ """
+ # Create the request object.
+ request = operations_pb2.ListOperationsRequest(name=name, filter=filter_)
+
+ # Create the method used to fetch pages
+ method = functools.partial(self._list_operations, retry=retry, timeout=timeout)
+
+ iterator = page_iterator.GRPCIterator(
+ client=None,
+ method=method,
+ request=request,
+ items_field="operations",
+ request_token_field="page_token",
+ response_token_field="next_page_token",
+ )
+
+ return iterator
+
+ def cancel_operation(
+ self, name, retry=gapic_v1.method.DEFAULT, timeout=gapic_v1.method.DEFAULT
+ ):
+ """Starts asynchronous cancellation on a long-running operation.
+
+ The server makes a best effort to cancel the operation, but success is
+ not guaranteed. Clients can use :meth:`get_operation` or service-
+ specific methods to check whether the cancellation succeeded or whether
+ the operation completed despite cancellation. On successful
+ cancellation, the operation is not deleted; instead, it becomes an
+ operation with an ``Operation.error`` value with a
+ ``google.rpc.Status.code`` of ``1``, corresponding to
+ ``Code.CANCELLED``.
+
+ Example:
+ >>> from google.api_core import operations_v1
+ >>> api = operations_v1.OperationsClient()
+ >>> name = ''
+ >>> api.cancel_operation(name)
+
+ Args:
+ name (str): The name of the operation resource to be cancelled.
+ retry (google.api_core.retry.Retry): The retry strategy to use
+ when invoking the RPC. If unspecified, the default retry from
+ the client configuration will be used. If ``None``, then this
+ method will not retry the RPC at all.
+ timeout (float): The amount of time in seconds to wait for the RPC
+ to complete. Note that if ``retry`` is used, this timeout
+ applies to each individual attempt and the overall time it
+ takes for this method to complete may be longer. If
+ unspecified, the the default timeout in the client
+ configuration is used. If ``None``, then the RPC method will
+ not time out.
+
+ Raises:
+ google.api_core.exceptions.MethodNotImplemented: If the server
+ does not support this method. Services are not required to
+ implement this method.
+ google.api_core.exceptions.GoogleAPICallError: If an error occurred
+ while invoking the RPC, the appropriate ``GoogleAPICallError``
+ subclass will be raised.
+ """
+ # Create the request object.
+ request = operations_pb2.CancelOperationRequest(name=name)
+ self._cancel_operation(request, retry=retry, timeout=timeout)
+
+ def delete_operation(
+ self, name, retry=gapic_v1.method.DEFAULT, timeout=gapic_v1.method.DEFAULT
+ ):
+ """Deletes a long-running operation.
+
+ This method indicates that the client is no longer interested in the
+ operation result. It does not cancel the operation.
+
+ Example:
+ >>> from google.api_core import operations_v1
+ >>> api = operations_v1.OperationsClient()
+ >>> name = ''
+ >>> api.delete_operation(name)
+
+ Args:
+ name (str): The name of the operation resource to be deleted.
+ retry (google.api_core.retry.Retry): The retry strategy to use
+ when invoking the RPC. If unspecified, the default retry from
+ the client configuration will be used. If ``None``, then this
+ method will not retry the RPC at all.
+ timeout (float): The amount of time in seconds to wait for the RPC
+ to complete. Note that if ``retry`` is used, this timeout
+ applies to each individual attempt and the overall time it
+ takes for this method to complete may be longer. If
+ unspecified, the the default timeout in the client
+ configuration is used. If ``None``, then the RPC method will
+ not time out.
+
+ Raises:
+ google.api_core.exceptions.MethodNotImplemented: If the server
+ does not support this method. Services are not required to
+ implement this method.
+ google.api_core.exceptions.GoogleAPICallError: If an error occurred
+ while invoking the RPC, the appropriate ``GoogleAPICallError``
+ subclass will be raised.
+ """
+ # Create the request object.
+ request = operations_pb2.DeleteOperationRequest(name=name)
+ self._delete_operation(request, retry=retry, timeout=timeout)
diff --git a/api_core/google/api_core/operations_v1/operations_client_config.py b/api_core/google/api_core/operations_v1/operations_client_config.py
new file mode 100644
index 000000000000..6cf95753f4be
--- /dev/null
+++ b/api_core/google/api_core/operations_v1/operations_client_config.py
@@ -0,0 +1,59 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""gapic configuration for the googe.longrunning.operations client."""
+
+config = {
+ "interfaces": {
+ "google.longrunning.Operations": {
+ "retry_codes": {
+ "idempotent": ["DEADLINE_EXCEEDED", "UNAVAILABLE"],
+ "non_idempotent": [],
+ },
+ "retry_params": {
+ "default": {
+ "initial_retry_delay_millis": 100,
+ "retry_delay_multiplier": 1.3,
+ "max_retry_delay_millis": 60000,
+ "initial_rpc_timeout_millis": 20000,
+ "rpc_timeout_multiplier": 1.0,
+ "max_rpc_timeout_millis": 600000,
+ "total_timeout_millis": 600000,
+ }
+ },
+ "methods": {
+ "GetOperation": {
+ "timeout_millis": 60000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "ListOperations": {
+ "timeout_millis": 60000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "CancelOperation": {
+ "timeout_millis": 60000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "DeleteOperation": {
+ "timeout_millis": 60000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ },
+ }
+ }
+}
diff --git a/api_core/google/api_core/page_iterator.py b/api_core/google/api_core/page_iterator.py
new file mode 100644
index 000000000000..3ac5904399b0
--- /dev/null
+++ b/api_core/google/api_core/page_iterator.py
@@ -0,0 +1,549 @@
+# Copyright 2015 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Iterators for paging through paged API methods.
+
+These iterators simplify the process of paging through API responses
+where the request takes a page token and the response is a list of results with
+a token for the next page. See `list pagination`_ in the Google API Style Guide
+for more details.
+
+.. _list pagination:
+ https://cloud.google.com/apis/design/design_patterns#list_pagination
+
+API clients that have methods that follow the list pagination pattern can
+return an :class:`.Iterator`. You can use this iterator to get **all** of
+the results across all pages::
+
+ >>> results_iterator = client.list_resources()
+ >>> list(results_iterator) # Convert to a list (consumes all values).
+
+Or you can walk your way through items and call off the search early if
+you find what you're looking for (resulting in possibly fewer requests)::
+
+ >>> for resource in results_iterator:
+ ... print(resource.name)
+ ... if not resource.is_valid:
+ ... break
+
+At any point, you may check the number of items consumed by referencing the
+``num_results`` property of the iterator::
+
+ >>> for my_item in results_iterator:
+ ... if results_iterator.num_results >= 10:
+ ... break
+
+When iterating, not every new item will send a request to the server.
+To iterate based on each page of items (where a page corresponds to
+a request)::
+
+ >>> for page in results_iterator.pages:
+ ... print('=' * 20)
+ ... print(' Page number: {:d}'.format(iterator.page_number))
+ ... print(' Items in page: {:d}'.format(page.num_items))
+ ... print(' First item: {!r}'.format(next(page)))
+ ... print('Items remaining: {:d}'.format(page.remaining))
+ ... print('Next page token: {}'.format(iterator.next_page_token))
+ ====================
+ Page number: 1
+ Items in page: 1
+ First item:
+ Items remaining: 0
+ Next page token: eav1OzQB0OM8rLdGXOEsyQWSG
+ ====================
+ Page number: 2
+ Items in page: 19
+ First item:
+ Items remaining: 18
+ Next page token: None
+
+Then, for each page you can get all the resources on that page by iterating
+through it or using :func:`list`::
+
+ >>> list(page)
+ [
+ ,
+ ,
+ ,
+ ]
+"""
+
+import abc
+
+import six
+
+
+class Page(object):
+ """Single page of results in an iterator.
+
+ Args:
+ parent (google.api_core.page_iterator.Iterator): The iterator that owns
+ the current page.
+ items (Sequence[Any]): An iterable (that also defines __len__) of items
+ from a raw API response.
+ item_to_value (Callable[google.api_core.page_iterator.Iterator, Any]):
+ Callable to convert an item from the type in the raw API response
+ into the native object. Will be called with the iterator and a
+ single item.
+ """
+
+ def __init__(self, parent, items, item_to_value):
+ self._parent = parent
+ self._num_items = len(items)
+ self._remaining = self._num_items
+ self._item_iter = iter(items)
+ self._item_to_value = item_to_value
+
+ @property
+ def num_items(self):
+ """int: Total items in the page."""
+ return self._num_items
+
+ @property
+ def remaining(self):
+ """int: Remaining items in the page."""
+ return self._remaining
+
+ def __iter__(self):
+ """The :class:`Page` is an iterator of items."""
+ return self
+
+ def next(self):
+ """Get the next value in the page."""
+ item = six.next(self._item_iter)
+ result = self._item_to_value(self._parent, item)
+ # Since we've successfully got the next value from the
+ # iterator, we update the number of remaining.
+ self._remaining -= 1
+ return result
+
+ # Alias needed for Python 2/3 support.
+ __next__ = next
+
+
+def _item_to_value_identity(iterator, item):
+ """An item to value transformer that returns the item un-changed."""
+ # pylint: disable=unused-argument
+ # We are conforming to the interface defined by Iterator.
+ return item
+
+
+@six.add_metaclass(abc.ABCMeta)
+class Iterator(object):
+ """A generic class for iterating through API list responses.
+
+ Args:
+ client(google.cloud.client.Client): The API client.
+ item_to_value (Callable[google.api_core.page_iterator.Iterator, Any]):
+ Callable to convert an item from the type in the raw API response
+ into the native object. Will be called with the iterator and a
+ single item.
+ page_token (str): A token identifying a page in a result set to start
+ fetching results from.
+ max_results (int): The maximum number of results to fetch.
+ """
+
+ def __init__(
+ self,
+ client,
+ item_to_value=_item_to_value_identity,
+ page_token=None,
+ max_results=None,
+ ):
+ self._started = False
+ self.client = client
+ """Optional[Any]: The client that created this iterator."""
+ self.item_to_value = item_to_value
+ """Callable[Iterator, Any]: Callable to convert an item from the type
+ in the raw API response into the native object. Will be called with
+ the iterator and a
+ single item.
+ """
+ self.max_results = max_results
+ """int: The maximum number of results to fetch."""
+
+ # The attributes below will change over the life of the iterator.
+ self.page_number = 0
+ """int: The current page of results."""
+ self.next_page_token = page_token
+ """str: The token for the next page of results. If this is set before
+ the iterator starts, it effectively offsets the iterator to a
+ specific starting point."""
+ self.num_results = 0
+ """int: The total number of results fetched so far."""
+
+ @property
+ def pages(self):
+ """Iterator of pages in the response.
+
+ returns:
+ types.GeneratorType[google.api_core.page_iterator.Page]: A
+ generator of page instances.
+
+ raises:
+ ValueError: If the iterator has already been started.
+ """
+ if self._started:
+ raise ValueError("Iterator has already started", self)
+ self._started = True
+ return self._page_iter(increment=True)
+
+ def _items_iter(self):
+ """Iterator for each item returned."""
+ for page in self._page_iter(increment=False):
+ for item in page:
+ self.num_results += 1
+ yield item
+
+ def __iter__(self):
+ """Iterator for each item returned.
+
+ Returns:
+ types.GeneratorType[Any]: A generator of items from the API.
+
+ Raises:
+ ValueError: If the iterator has already been started.
+ """
+ if self._started:
+ raise ValueError("Iterator has already started", self)
+ self._started = True
+ return self._items_iter()
+
+ def _page_iter(self, increment):
+ """Generator of pages of API responses.
+
+ Args:
+ increment (bool): Flag indicating if the total number of results
+ should be incremented on each page. This is useful since a page
+ iterator will want to increment by results per page while an
+ items iterator will want to increment per item.
+
+ Yields:
+ Page: each page of items from the API.
+ """
+ page = self._next_page()
+ while page is not None:
+ self.page_number += 1
+ if increment:
+ self.num_results += page.num_items
+ yield page
+ page = self._next_page()
+
+ @abc.abstractmethod
+ def _next_page(self):
+ """Get the next page in the iterator.
+
+ This does nothing and is intended to be over-ridden by subclasses
+ to return the next :class:`Page`.
+
+ Raises:
+ NotImplementedError: Always, this method is abstract.
+ """
+ raise NotImplementedError
+
+
+def _do_nothing_page_start(iterator, page, response):
+ """Helper to provide custom behavior after a :class:`Page` is started.
+
+ This is a do-nothing stand-in as the default value.
+
+ Args:
+ iterator (Iterator): An iterator that holds some request info.
+ page (Page): The page that was just created.
+ response (Any): The API response for a page.
+ """
+ # pylint: disable=unused-argument
+ pass
+
+
+class HTTPIterator(Iterator):
+ """A generic class for iterating through HTTP/JSON API list responses.
+
+ To make an iterator work, you'll need to provide a way to convert a JSON
+ item returned from the API into the object of your choice (via
+ ``item_to_value``). You also may need to specify a custom ``items_key`` so
+ that a given response (containing a page of results) can be parsed into an
+ iterable page of the actual objects you want.
+
+ Args:
+ client (google.cloud.client.Client): The API client.
+ api_request (Callable): The function to use to make API requests.
+ Generally, this will be
+ :meth:`google.cloud._http.JSONConnection.api_request`.
+ path (str): The method path to query for the list of items.
+ item_to_value (Callable[google.api_core.page_iterator.Iterator, Any]):
+ Callable to convert an item from the type in the JSON response into
+ a native object. Will be called with the iterator and a single
+ item.
+ items_key (str): The key in the API response where the list of items
+ can be found.
+ page_token (str): A token identifying a page in a result set to start
+ fetching results from.
+ max_results (int): The maximum number of results to fetch.
+ extra_params (dict): Extra query string parameters for the
+ API call.
+ page_start (Callable[
+ google.api_core.page_iterator.Iterator,
+ google.api_core.page_iterator.Page, dict]): Callable to provide
+ any special behavior after a new page has been created. Assumed
+ signature takes the :class:`.Iterator` that started the page,
+ the :class:`.Page` that was started and the dictionary containing
+ the page response.
+ next_token (str): The name of the field used in the response for page
+ tokens.
+
+ .. autoattribute:: pages
+ """
+
+ _DEFAULT_ITEMS_KEY = "items"
+ _PAGE_TOKEN = "pageToken"
+ _MAX_RESULTS = "maxResults"
+ _NEXT_TOKEN = "nextPageToken"
+ _RESERVED_PARAMS = frozenset([_PAGE_TOKEN])
+ _HTTP_METHOD = "GET"
+
+ def __init__(
+ self,
+ client,
+ api_request,
+ path,
+ item_to_value,
+ items_key=_DEFAULT_ITEMS_KEY,
+ page_token=None,
+ max_results=None,
+ extra_params=None,
+ page_start=_do_nothing_page_start,
+ next_token=_NEXT_TOKEN,
+ ):
+ super(HTTPIterator, self).__init__(
+ client, item_to_value, page_token=page_token, max_results=max_results
+ )
+ self.api_request = api_request
+ self.path = path
+ self._items_key = items_key
+ self.extra_params = extra_params
+ self._page_start = page_start
+ self._next_token = next_token
+ # Verify inputs / provide defaults.
+ if self.extra_params is None:
+ self.extra_params = {}
+ self._verify_params()
+
+ def _verify_params(self):
+ """Verifies the parameters don't use any reserved parameter.
+
+ Raises:
+ ValueError: If a reserved parameter is used.
+ """
+ reserved_in_use = self._RESERVED_PARAMS.intersection(self.extra_params)
+ if reserved_in_use:
+ raise ValueError("Using a reserved parameter", reserved_in_use)
+
+ def _next_page(self):
+ """Get the next page in the iterator.
+
+ Returns:
+ Optional[Page]: The next page in the iterator or :data:`None` if
+ there are no pages left.
+ """
+ if self._has_next_page():
+ response = self._get_next_page_response()
+ items = response.get(self._items_key, ())
+ page = Page(self, items, self.item_to_value)
+ self._page_start(self, page, response)
+ self.next_page_token = response.get(self._next_token)
+ return page
+ else:
+ return None
+
+ def _has_next_page(self):
+ """Determines whether or not there are more pages with results.
+
+ Returns:
+ bool: Whether the iterator has more pages.
+ """
+ if self.page_number == 0:
+ return True
+
+ if self.max_results is not None:
+ if self.num_results >= self.max_results:
+ return False
+
+ return self.next_page_token is not None
+
+ def _get_query_params(self):
+ """Getter for query parameters for the next request.
+
+ Returns:
+ dict: A dictionary of query parameters.
+ """
+ result = {}
+ if self.next_page_token is not None:
+ result[self._PAGE_TOKEN] = self.next_page_token
+ if self.max_results is not None:
+ result[self._MAX_RESULTS] = self.max_results - self.num_results
+ result.update(self.extra_params)
+ return result
+
+ def _get_next_page_response(self):
+ """Requests the next page from the path provided.
+
+ Returns:
+ dict: The parsed JSON response of the next page's contents.
+
+ Raises:
+ ValueError: If the HTTP method is not ``GET`` or ``POST``.
+ """
+ params = self._get_query_params()
+ if self._HTTP_METHOD == "GET":
+ return self.api_request(
+ method=self._HTTP_METHOD, path=self.path, query_params=params
+ )
+ elif self._HTTP_METHOD == "POST":
+ return self.api_request(
+ method=self._HTTP_METHOD, path=self.path, data=params
+ )
+ else:
+ raise ValueError("Unexpected HTTP method", self._HTTP_METHOD)
+
+
+class _GAXIterator(Iterator):
+ """A generic class for iterating through Cloud gRPC APIs list responses.
+
+ Any:
+ client (google.cloud.client.Client): The API client.
+ page_iter (google.gax.PageIterator): A GAX page iterator to be wrapped
+ to conform to the :class:`Iterator` interface.
+ item_to_value (Callable[Iterator, Any]): Callable to convert an item
+ from the the protobuf response into a native object. Will
+ be called with the iterator and a single item.
+ max_results (int): The maximum number of results to fetch.
+
+ .. autoattribute:: pages
+ """
+
+ def __init__(self, client, page_iter, item_to_value, max_results=None):
+ super(_GAXIterator, self).__init__(
+ client,
+ item_to_value,
+ page_token=page_iter.page_token,
+ max_results=max_results,
+ )
+ self._gax_page_iter = page_iter
+
+ def _next_page(self):
+ """Get the next page in the iterator.
+
+ Wraps the response from the :class:`~google.gax.PageIterator` in a
+ :class:`Page` instance and captures some state at each page.
+
+ Returns:
+ Optional[Page]: The next page in the iterator or :data:`None` if
+ there are no pages left.
+ """
+ try:
+ items = six.next(self._gax_page_iter)
+ page = Page(self, items, self.item_to_value)
+ self.next_page_token = self._gax_page_iter.page_token or None
+ return page
+ except StopIteration:
+ return None
+
+
+class GRPCIterator(Iterator):
+ """A generic class for iterating through gRPC list responses.
+
+ .. note:: The class does not take a ``page_token`` argument because it can
+ just be specified in the ``request``.
+
+ Args:
+ client (google.cloud.client.Client): The API client. This unused by
+ this class, but kept to satisfy the :class:`Iterator` interface.
+ method (Callable[protobuf.Message]): A bound gRPC method that should
+ take a single message for the request.
+ request (protobuf.Message): The request message.
+ items_field (str): The field in the response message that has the
+ items for the page.
+ item_to_value (Callable[GRPCIterator, Any]): Callable to convert an
+ item from the type in the JSON response into a native object. Will
+ be called with the iterator and a single item.
+ request_token_field (str): The field in the request message used to
+ specify the page token.
+ response_token_field (str): The field in the response message that has
+ the token for the next page.
+ max_results (int): The maximum number of results to fetch.
+
+ .. autoattribute:: pages
+ """
+
+ _DEFAULT_REQUEST_TOKEN_FIELD = "page_token"
+ _DEFAULT_RESPONSE_TOKEN_FIELD = "next_page_token"
+
+ def __init__(
+ self,
+ client,
+ method,
+ request,
+ items_field,
+ item_to_value=_item_to_value_identity,
+ request_token_field=_DEFAULT_REQUEST_TOKEN_FIELD,
+ response_token_field=_DEFAULT_RESPONSE_TOKEN_FIELD,
+ max_results=None,
+ ):
+ super(GRPCIterator, self).__init__(
+ client, item_to_value, max_results=max_results
+ )
+ self._method = method
+ self._request = request
+ self._items_field = items_field
+ self._request_token_field = request_token_field
+ self._response_token_field = response_token_field
+
+ def _next_page(self):
+ """Get the next page in the iterator.
+
+ Returns:
+ Page: The next page in the iterator or :data:`None` if
+ there are no pages left.
+ """
+ if not self._has_next_page():
+ return None
+
+ if self.next_page_token is not None:
+ setattr(self._request, self._request_token_field, self.next_page_token)
+
+ response = self._method(self._request)
+
+ self.next_page_token = getattr(response, self._response_token_field)
+ items = getattr(response, self._items_field)
+ page = Page(self, items, self.item_to_value)
+
+ return page
+
+ def _has_next_page(self):
+ """Determines whether or not there are more pages with results.
+
+ Returns:
+ bool: Whether the iterator has more pages.
+ """
+ if self.page_number == 0:
+ return True
+
+ if self.max_results is not None:
+ if self.num_results >= self.max_results:
+ return False
+
+ # Note: intentionally a falsy check instead of a None check. The RPC
+ # can return an empty string indicating no more pages.
+ return True if self.next_page_token else False
diff --git a/api_core/google/api_core/path_template.py b/api_core/google/api_core/path_template.py
new file mode 100644
index 000000000000..bb549356d284
--- /dev/null
+++ b/api_core/google/api_core/path_template.py
@@ -0,0 +1,197 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Expand and validate URL path templates.
+
+This module provides the :func:`expand` and :func:`validate` functions for
+interacting with Google-style URL `path templates`_ which are commonly used
+in Google APIs for `resource names`_.
+
+.. _path templates: https://github.com/googleapis/googleapis/blob
+ /57e2d376ac7ef48681554204a3ba78a414f2c533/google/api/http.proto#L212
+.. _resource names: https://cloud.google.com/apis/design/resource_names
+"""
+
+from __future__ import unicode_literals
+
+import functools
+import re
+
+import six
+
+# Regular expression for extracting variable parts from a path template.
+# The variables can be expressed as:
+#
+# - "*": a single-segment positional variable, for example: "books/*"
+# - "**": a multi-segment positional variable, for example: "shelf/**/book/*"
+# - "{name}": a single-segment wildcard named variable, for example
+# "books/{name}"
+# - "{name=*}: same as above.
+# - "{name=**}": a multi-segment wildcard named variable, for example
+# "shelf/{name=**}"
+# - "{name=/path/*/**}": a multi-segment named variable with a sub-template.
+_VARIABLE_RE = re.compile(
+ r"""
+ ( # Capture the entire variable expression
+ (?P\*\*?) # Match & capture * and ** positional variables.
+ |
+ # Match & capture named variables {name}
+ {
+ (?P[^/]+?)
+ # Optionally match and capture the named variable's template.
+ (?:=(?P.+?))?
+ }
+ )
+ """,
+ re.VERBOSE,
+)
+
+# Segment expressions used for validating paths against a template.
+_SINGLE_SEGMENT_PATTERN = r"([^/]+)"
+_MULTI_SEGMENT_PATTERN = r"(.+)"
+
+
+def _expand_variable_match(positional_vars, named_vars, match):
+ """Expand a matched variable with its value.
+
+ Args:
+ positional_vars (list): A list of positonal variables. This list will
+ be modified.
+ named_vars (dict): A dictionary of named variables.
+ match (re.Match): A regular expression match.
+
+ Returns:
+ str: The expanded variable to replace the match.
+
+ Raises:
+ ValueError: If a positional or named variable is required by the
+ template but not specified or if an unexpected template expression
+ is encountered.
+ """
+ positional = match.group("positional")
+ name = match.group("name")
+ if name is not None:
+ try:
+ return six.text_type(named_vars[name])
+ except KeyError:
+ raise ValueError(
+ "Named variable '{}' not specified and needed by template "
+ "`{}` at position {}".format(name, match.string, match.start())
+ )
+ elif positional is not None:
+ try:
+ return six.text_type(positional_vars.pop(0))
+ except IndexError:
+ raise ValueError(
+ "Positional variable not specified and needed by template "
+ "`{}` at position {}".format(match.string, match.start())
+ )
+ else:
+ raise ValueError("Unknown template expression {}".format(match.group(0)))
+
+
+def expand(tmpl, *args, **kwargs):
+ """Expand a path template with the given variables.
+
+ ..code-block:: python
+
+ >>> expand('users/*/messages/*', 'me', '123')
+ users/me/messages/123
+ >>> expand('/v1/{name=shelves/*/books/*}', name='shelves/1/books/3')
+ /v1/shelves/1/books/3
+
+ Args:
+ tmpl (str): The path template.
+ args: The positional variables for the path.
+ kwargs: The named variables for the path.
+
+ Returns:
+ str: The expanded path
+
+ Raises:
+ ValueError: If a positional or named variable is required by the
+ template but not specified or if an unexpected template expression
+ is encountered.
+ """
+ replacer = functools.partial(_expand_variable_match, list(args), kwargs)
+ return _VARIABLE_RE.sub(replacer, tmpl)
+
+
+def _replace_variable_with_pattern(match):
+ """Replace a variable match with a pattern that can be used to validate it.
+
+ Args:
+ match (re.Match): A regular expression match
+
+ Returns:
+ str: A regular expression pattern that can be used to validate the
+ variable in an expanded path.
+
+ Raises:
+ ValueError: If an unexpected template expression is encountered.
+ """
+ positional = match.group("positional")
+ name = match.group("name")
+ template = match.group("template")
+ if name is not None:
+ if not template:
+ return _SINGLE_SEGMENT_PATTERN.format(name)
+ elif template == "**":
+ return _MULTI_SEGMENT_PATTERN.format(name)
+ else:
+ return _generate_pattern_for_template(template)
+ elif positional == "*":
+ return _SINGLE_SEGMENT_PATTERN
+ elif positional == "**":
+ return _MULTI_SEGMENT_PATTERN
+ else:
+ raise ValueError("Unknown template expression {}".format(match.group(0)))
+
+
+def _generate_pattern_for_template(tmpl):
+ """Generate a pattern that can validate a path template.
+
+ Args:
+ tmpl (str): The path template
+
+ Returns:
+ str: A regular expression pattern that can be used to validate an
+ expanded path template.
+ """
+ return _VARIABLE_RE.sub(_replace_variable_with_pattern, tmpl)
+
+
+def validate(tmpl, path):
+ """Validate a path against the path template.
+
+ .. code-block:: python
+
+ >>> validate('users/*/messages/*', 'users/me/messages/123')
+ True
+ >>> validate('users/*/messages/*', 'users/me/drafts/123')
+ False
+ >>> validate('/v1/{name=shelves/*/books/*}', /v1/shelves/1/books/3)
+ True
+ >>> validate('/v1/{name=shelves/*/books/*}', /v1/shelves/1/tapes/3)
+ False
+
+ Args:
+ tmpl (str): The path template.
+ path (str): The expanded path.
+
+ Returns:
+ bool: True if the path matches.
+ """
+ pattern = _generate_pattern_for_template(tmpl) + "$"
+ return True if re.match(pattern, path) is not None else False
diff --git a/api_core/google/api_core/protobuf_helpers.py b/api_core/google/api_core/protobuf_helpers.py
new file mode 100644
index 000000000000..aec6b3620743
--- /dev/null
+++ b/api_core/google/api_core/protobuf_helpers.py
@@ -0,0 +1,368 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Helpers for :mod:`protobuf`."""
+
+import collections
+try:
+ from collections import abc as collections_abc
+except ImportError: # Python 2.7
+ import collections as collections_abc
+import copy
+import inspect
+
+from google.protobuf import field_mask_pb2
+from google.protobuf import message
+from google.protobuf import wrappers_pb2
+
+_SENTINEL = object()
+_WRAPPER_TYPES = (
+ wrappers_pb2.BoolValue,
+ wrappers_pb2.BytesValue,
+ wrappers_pb2.DoubleValue,
+ wrappers_pb2.FloatValue,
+ wrappers_pb2.Int32Value,
+ wrappers_pb2.Int64Value,
+ wrappers_pb2.StringValue,
+ wrappers_pb2.UInt32Value,
+ wrappers_pb2.UInt64Value,
+)
+
+
+def from_any_pb(pb_type, any_pb):
+ """Converts an ``Any`` protobuf to the specified message type.
+
+ Args:
+ pb_type (type): the type of the message that any_pb stores an instance
+ of.
+ any_pb (google.protobuf.any_pb2.Any): the object to be converted.
+
+ Returns:
+ pb_type: An instance of the pb_type message.
+
+ Raises:
+ TypeError: if the message could not be converted.
+ """
+ msg = pb_type()
+
+ # Unwrap proto-plus wrapped messages.
+ if callable(getattr(pb_type, "pb", None)):
+ msg_pb = pb_type.pb(msg)
+ else:
+ msg_pb = msg
+
+ # Unpack the Any object and populate the protobuf message instance.
+ if not any_pb.Unpack(msg_pb):
+ raise TypeError(
+ "Could not convert {} to {}".format(
+ any_pb.__class__.__name__, pb_type.__name__
+ )
+ )
+
+ # Done; return the message.
+ return msg
+
+
+def check_oneof(**kwargs):
+ """Raise ValueError if more than one keyword argument is not ``None``.
+
+ Args:
+ kwargs (dict): The keyword arguments sent to the function.
+
+ Raises:
+ ValueError: If more than one entry in ``kwargs`` is not ``None``.
+ """
+ # Sanity check: If no keyword arguments were sent, this is fine.
+ if not kwargs:
+ return
+
+ not_nones = [val for val in kwargs.values() if val is not None]
+ if len(not_nones) > 1:
+ raise ValueError(
+ "Only one of {fields} should be set.".format(
+ fields=", ".join(sorted(kwargs.keys()))
+ )
+ )
+
+
+def get_messages(module):
+ """Discovers all protobuf Message classes in a given import module.
+
+ Args:
+ module (module): A Python module; :func:`dir` will be run against this
+ module to find Message subclasses.
+
+ Returns:
+ dict[str, google.protobuf.message.Message]: A dictionary with the
+ Message class names as keys, and the Message subclasses themselves
+ as values.
+ """
+ answer = collections.OrderedDict()
+ for name in dir(module):
+ candidate = getattr(module, name)
+ if inspect.isclass(candidate) and issubclass(candidate, message.Message):
+ answer[name] = candidate
+ return answer
+
+
+def _resolve_subkeys(key, separator="."):
+ """Resolve a potentially nested key.
+
+ If the key contains the ``separator`` (e.g. ``.``) then the key will be
+ split on the first instance of the subkey::
+
+ >>> _resolve_subkeys('a.b.c')
+ ('a', 'b.c')
+ >>> _resolve_subkeys('d|e|f', separator='|')
+ ('d', 'e|f')
+
+ If not, the subkey will be :data:`None`::
+
+ >>> _resolve_subkeys('foo')
+ ('foo', None)
+
+ Args:
+ key (str): A string that may or may not contain the separator.
+ separator (str): The namespace separator. Defaults to `.`.
+
+ Returns:
+ Tuple[str, str]: The key and subkey(s).
+ """
+ parts = key.split(separator, 1)
+
+ if len(parts) > 1:
+ return parts
+ else:
+ return parts[0], None
+
+
+def get(msg_or_dict, key, default=_SENTINEL):
+ """Retrieve a key's value from a protobuf Message or dictionary.
+
+ Args:
+ mdg_or_dict (Union[~google.protobuf.message.Message, Mapping]): the
+ object.
+ key (str): The key to retrieve from the object.
+ default (Any): If the key is not present on the object, and a default
+ is set, returns that default instead. A type-appropriate falsy
+ default is generally recommended, as protobuf messages almost
+ always have default values for unset values and it is not always
+ possible to tell the difference between a falsy value and an
+ unset one. If no default is set then :class:`KeyError` will be
+ raised if the key is not present in the object.
+
+ Returns:
+ Any: The return value from the underlying Message or dict.
+
+ Raises:
+ KeyError: If the key is not found. Note that, for unset values,
+ messages and dictionaries may not have consistent behavior.
+ TypeError: If ``msg_or_dict`` is not a Message or Mapping.
+ """
+ # We may need to get a nested key. Resolve this.
+ key, subkey = _resolve_subkeys(key)
+
+ # Attempt to get the value from the two types of objects we know about.
+ # If we get something else, complain.
+ if isinstance(msg_or_dict, message.Message):
+ answer = getattr(msg_or_dict, key, default)
+ elif isinstance(msg_or_dict, collections_abc.Mapping):
+ answer = msg_or_dict.get(key, default)
+ else:
+ raise TypeError(
+ "get() expected a dict or protobuf message, got {!r}.".format(
+ type(msg_or_dict)
+ )
+ )
+
+ # If the object we got back is our sentinel, raise KeyError; this is
+ # a "not found" case.
+ if answer is _SENTINEL:
+ raise KeyError(key)
+
+ # If a subkey exists, call this method recursively against the answer.
+ if subkey is not None and answer is not default:
+ return get(answer, subkey, default=default)
+
+ return answer
+
+
+def _set_field_on_message(msg, key, value):
+ """Set helper for protobuf Messages."""
+ # Attempt to set the value on the types of objects we know how to deal
+ # with.
+ if isinstance(value, (collections_abc.MutableSequence, tuple)):
+ # Clear the existing repeated protobuf message of any elements
+ # currently inside it.
+ while getattr(msg, key):
+ getattr(msg, key).pop()
+
+ # Write our new elements to the repeated field.
+ for item in value:
+ if isinstance(item, collections_abc.Mapping):
+ getattr(msg, key).add(**item)
+ else:
+ # protobuf's RepeatedCompositeContainer doesn't support
+ # append.
+ getattr(msg, key).extend([item])
+ elif isinstance(value, collections_abc.Mapping):
+ # Assign the dictionary values to the protobuf message.
+ for item_key, item_value in value.items():
+ set(getattr(msg, key), item_key, item_value)
+ elif isinstance(value, message.Message):
+ getattr(msg, key).CopyFrom(value)
+ else:
+ setattr(msg, key, value)
+
+
+def set(msg_or_dict, key, value):
+ """Set a key's value on a protobuf Message or dictionary.
+
+ Args:
+ msg_or_dict (Union[~google.protobuf.message.Message, Mapping]): the
+ object.
+ key (str): The key to set.
+ value (Any): The value to set.
+
+ Raises:
+ TypeError: If ``msg_or_dict`` is not a Message or dictionary.
+ """
+ # Sanity check: Is our target object valid?
+ if not isinstance(msg_or_dict, (collections_abc.MutableMapping, message.Message)):
+ raise TypeError(
+ "set() expected a dict or protobuf message, got {!r}.".format(
+ type(msg_or_dict)
+ )
+ )
+
+ # We may be setting a nested key. Resolve this.
+ basekey, subkey = _resolve_subkeys(key)
+
+ # If a subkey exists, then get that object and call this method
+ # recursively against it using the subkey.
+ if subkey is not None:
+ if isinstance(msg_or_dict, collections_abc.MutableMapping):
+ msg_or_dict.setdefault(basekey, {})
+ set(get(msg_or_dict, basekey), subkey, value)
+ return
+
+ if isinstance(msg_or_dict, collections_abc.MutableMapping):
+ msg_or_dict[key] = value
+ else:
+ _set_field_on_message(msg_or_dict, key, value)
+
+
+def setdefault(msg_or_dict, key, value):
+ """Set the key on a protobuf Message or dictionary to a given value if the
+ current value is falsy.
+
+ Because protobuf Messages do not distinguish between unset values and
+ falsy ones particularly well (by design), this method treats any falsy
+ value (e.g. 0, empty list) as a target to be overwritten, on both Messages
+ and dictionaries.
+
+ Args:
+ msg_or_dict (Union[~google.protobuf.message.Message, Mapping]): the
+ object.
+ key (str): The key on the object in question.
+ value (Any): The value to set.
+
+ Raises:
+ TypeError: If ``msg_or_dict`` is not a Message or dictionary.
+ """
+ if not get(msg_or_dict, key, default=None):
+ set(msg_or_dict, key, value)
+
+
+def field_mask(original, modified):
+ """Create a field mask by comparing two messages.
+
+ Args:
+ original (~google.protobuf.message.Message): the original message.
+ If set to None, this field will be interpretted as an empty
+ message.
+ modified (~google.protobuf.message.Message): the modified message.
+ If set to None, this field will be interpretted as an empty
+ message.
+
+ Returns:
+ google.protobuf.field_mask_pb2.FieldMask: field mask that contains
+ the list of field names that have different values between the two
+ messages. If the messages are equivalent, then the field mask is empty.
+
+ Raises:
+ ValueError: If the ``original`` or ``modified`` are not the same type.
+ """
+ if original is None and modified is None:
+ return field_mask_pb2.FieldMask()
+
+ if original is None and modified is not None:
+ original = copy.deepcopy(modified)
+ original.Clear()
+
+ if modified is None and original is not None:
+ modified = copy.deepcopy(original)
+ modified.Clear()
+
+ if type(original) != type(modified):
+ raise ValueError(
+ "expected that both original and modified should be of the "
+ 'same type, received "{!r}" and "{!r}".'.format(
+ type(original), type(modified)
+ )
+ )
+
+ return field_mask_pb2.FieldMask(paths=_field_mask_helper(original, modified))
+
+
+def _field_mask_helper(original, modified, current=""):
+ answer = []
+
+ for name in original.DESCRIPTOR.fields_by_name:
+ field_path = _get_path(current, name)
+
+ original_val = getattr(original, name)
+ modified_val = getattr(modified, name)
+
+ if _is_message(original_val) or _is_message(modified_val):
+ if original_val != modified_val:
+ # Wrapper types do not need to include the .value part of the
+ # path.
+ if _is_wrapper(original_val) or _is_wrapper(modified_val):
+ answer.append(field_path)
+ elif not modified_val.ListFields():
+ answer.append(field_path)
+ else:
+ answer.extend(
+ _field_mask_helper(original_val, modified_val, field_path)
+ )
+ else:
+ if original_val != modified_val:
+ answer.append(field_path)
+
+ return answer
+
+
+def _get_path(current, name):
+ if not current:
+ return name
+ return "%s.%s" % (current, name)
+
+
+def _is_message(value):
+ return isinstance(value, message.Message)
+
+
+def _is_wrapper(value):
+ return type(value) in _WRAPPER_TYPES
diff --git a/api_core/google/api_core/retry.py b/api_core/google/api_core/retry.py
new file mode 100644
index 000000000000..79343bad3471
--- /dev/null
+++ b/api_core/google/api_core/retry.py
@@ -0,0 +1,351 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Helpers for retrying functions with exponential back-off.
+
+The :class:`Retry` decorator can be used to retry functions that raise
+exceptions using exponential backoff. Because a exponential sleep algorithm is
+used, the retry is limited by a `deadline`. The deadline is the maxmimum amount
+of time a method can block. This is used instead of total number of retries
+because it is difficult to ascertain the amount of time a function can block
+when using total number of retries and exponential backoff.
+
+By default, this decorator will retry transient
+API errors (see :func:`if_transient_error`). For example:
+
+.. code-block:: python
+
+ @retry.Retry()
+ def call_flaky_rpc():
+ return client.flaky_rpc()
+
+ # Will retry flaky_rpc() if it raises transient API errors.
+ result = call_flaky_rpc()
+
+You can pass a custom predicate to retry on different exceptions, such as
+waiting for an eventually consistent item to be available:
+
+.. code-block:: python
+
+ @retry.Retry(predicate=if_exception_type(exceptions.NotFound))
+ def check_if_exists():
+ return client.does_thing_exist()
+
+ is_available = check_if_exists()
+
+Some client library methods apply retry automatically. These methods can accept
+a ``retry`` parameter that allows you to configure the behavior:
+
+.. code-block:: python
+
+ my_retry = retry.Retry(deadline=60)
+ result = client.some_method(retry=my_retry)
+
+"""
+
+from __future__ import unicode_literals
+
+import datetime
+import functools
+import logging
+import random
+import time
+
+import six
+
+from google.api_core import datetime_helpers
+from google.api_core import exceptions
+from google.api_core import general_helpers
+
+_LOGGER = logging.getLogger(__name__)
+_DEFAULT_INITIAL_DELAY = 1.0 # seconds
+_DEFAULT_MAXIMUM_DELAY = 60.0 # seconds
+_DEFAULT_DELAY_MULTIPLIER = 2.0
+_DEFAULT_DEADLINE = 60.0 * 2.0 # seconds
+
+
+def if_exception_type(*exception_types):
+ """Creates a predicate to check if the exception is of a given type.
+
+ Args:
+ exception_types (Sequence[:func:`type`]): The exception types to check
+ for.
+
+ Returns:
+ Callable[Exception]: A predicate that returns True if the provided
+ exception is of the given type(s).
+ """
+
+ def if_exception_type_predicate(exception):
+ """Bound predicate for checking an exception type."""
+ return isinstance(exception, exception_types)
+
+ return if_exception_type_predicate
+
+
+# pylint: disable=invalid-name
+# Pylint sees this as a constant, but it is also an alias that should be
+# considered a function.
+if_transient_error = if_exception_type(
+ exceptions.InternalServerError,
+ exceptions.TooManyRequests,
+ exceptions.ServiceUnavailable,
+)
+"""A predicate that checks if an exception is a transient API error.
+
+The following server errors are considered transient:
+
+- :class:`google.api_core.exceptions.InternalServerError` - HTTP 500, gRPC
+ ``INTERNAL(13)`` and its subclasses.
+- :class:`google.api_core.exceptions.TooManyRequests` - HTTP 429
+- :class:`google.api_core.exceptions.ServiceUnavailable` - HTTP 503
+- :class:`google.api_core.exceptions.ResourceExhausted` - gRPC
+ ``RESOURCE_EXHAUSTED(8)``
+"""
+# pylint: enable=invalid-name
+
+
+def exponential_sleep_generator(initial, maximum, multiplier=_DEFAULT_DELAY_MULTIPLIER):
+ """Generates sleep intervals based on the exponential back-off algorithm.
+
+ This implements the `Truncated Exponential Back-off`_ algorithm.
+
+ .. _Truncated Exponential Back-off:
+ https://cloud.google.com/storage/docs/exponential-backoff
+
+ Args:
+ initial (float): The minimum amout of time to delay. This must
+ be greater than 0.
+ maximum (float): The maximum amout of time to delay.
+ multiplier (float): The multiplier applied to the delay.
+
+ Yields:
+ float: successive sleep intervals.
+ """
+ delay = initial
+ while True:
+ # Introduce jitter by yielding a delay that is uniformly distributed
+ # to average out to the delay time.
+ yield min(random.uniform(0.0, delay * 2.0), maximum)
+ delay = delay * multiplier
+
+
+def retry_target(target, predicate, sleep_generator, deadline, on_error=None):
+ """Call a function and retry if it fails.
+
+ This is the lowest-level retry helper. Generally, you'll use the
+ higher-level retry helper :class:`Retry`.
+
+ Args:
+ target(Callable): The function to call and retry. This must be a
+ nullary function - apply arguments with `functools.partial`.
+ predicate (Callable[Exception]): A callable used to determine if an
+ exception raised by the target should be considered retryable.
+ It should return True to retry or False otherwise.
+ sleep_generator (Iterable[float]): An infinite iterator that determines
+ how long to sleep between retries.
+ deadline (float): How long to keep retrying the target.
+ on_error (Callable): A function to call while processing a retryable
+ exception. Any error raised by this function will *not* be
+ caught.
+
+ Returns:
+ Any: the return value of the target function.
+
+ Raises:
+ google.api_core.RetryError: If the deadline is exceeded while retrying.
+ ValueError: If the sleep generator stops yielding values.
+ Exception: If the target raises a method that isn't retryable.
+ """
+ if deadline is not None:
+ deadline_datetime = datetime_helpers.utcnow() + datetime.timedelta(
+ seconds=deadline
+ )
+ else:
+ deadline_datetime = None
+
+ last_exc = None
+
+ for sleep in sleep_generator:
+ try:
+ return target()
+
+ # pylint: disable=broad-except
+ # This function explicitly must deal with broad exceptions.
+ except Exception as exc:
+ if not predicate(exc):
+ raise
+ last_exc = exc
+ if on_error is not None:
+ on_error(exc)
+
+ now = datetime_helpers.utcnow()
+ if deadline_datetime is not None and deadline_datetime < now:
+ six.raise_from(
+ exceptions.RetryError(
+ "Deadline of {:.1f}s exceeded while calling {}".format(
+ deadline, target
+ ),
+ last_exc,
+ ),
+ last_exc,
+ )
+
+ _LOGGER.debug(
+ "Retrying due to {}, sleeping {:.1f}s ...".format(last_exc, sleep)
+ )
+ time.sleep(sleep)
+
+ raise ValueError("Sleep generator stopped yielding sleep values.")
+
+
+@six.python_2_unicode_compatible
+class Retry(object):
+ """Exponential retry decorator.
+
+ This class is a decorator used to add exponential back-off retry behavior
+ to an RPC call.
+
+ Although the default behavior is to retry transient API errors, a
+ different predicate can be provided to retry other exceptions.
+
+ Args:
+ predicate (Callable[Exception]): A callable that should return ``True``
+ if the given exception is retryable.
+ initial (float): The minimum a,out of time to delay in seconds. This
+ must be greater than 0.
+ maximum (float): The maximum amout of time to delay in seconds.
+ multiplier (float): The multiplier applied to the delay.
+ deadline (float): How long to keep retrying in seconds.
+ """
+
+ def __init__(
+ self,
+ predicate=if_transient_error,
+ initial=_DEFAULT_INITIAL_DELAY,
+ maximum=_DEFAULT_MAXIMUM_DELAY,
+ multiplier=_DEFAULT_DELAY_MULTIPLIER,
+ deadline=_DEFAULT_DEADLINE,
+ on_error=None
+ ):
+ self._predicate = predicate
+ self._initial = initial
+ self._multiplier = multiplier
+ self._maximum = maximum
+ self._deadline = deadline
+ self._on_error = on_error
+
+ def __call__(self, func, on_error=None):
+ """Wrap a callable with retry behavior.
+
+ Args:
+ func (Callable): The callable to add retry behavior to.
+ on_error (Callable): A function to call while processing a
+ retryable exception. Any error raised by this function will
+ *not* be caught.
+
+ Returns:
+ Callable: A callable that will invoke ``func`` with retry
+ behavior.
+ """
+ if self._on_error is not None:
+ on_error = self._on_error
+
+ @general_helpers.wraps(func)
+ def retry_wrapped_func(*args, **kwargs):
+ """A wrapper that calls target function with retry."""
+ target = functools.partial(func, *args, **kwargs)
+ sleep_generator = exponential_sleep_generator(
+ self._initial, self._maximum, multiplier=self._multiplier
+ )
+ return retry_target(
+ target,
+ self._predicate,
+ sleep_generator,
+ self._deadline,
+ on_error=on_error,
+ )
+
+ return retry_wrapped_func
+
+ def with_deadline(self, deadline):
+ """Return a copy of this retry with the given deadline.
+
+ Args:
+ deadline (float): How long to keep retrying.
+
+ Returns:
+ Retry: A new retry instance with the given deadline.
+ """
+ return Retry(
+ predicate=self._predicate,
+ initial=self._initial,
+ maximum=self._maximum,
+ multiplier=self._multiplier,
+ deadline=deadline,
+ on_error=self._on_error,
+ )
+
+ def with_predicate(self, predicate):
+ """Return a copy of this retry with the given predicate.
+
+ Args:
+ predicate (Callable[Exception]): A callable that should return
+ ``True`` if the given exception is retryable.
+
+ Returns:
+ Retry: A new retry instance with the given predicate.
+ """
+ return Retry(
+ predicate=predicate,
+ initial=self._initial,
+ maximum=self._maximum,
+ multiplier=self._multiplier,
+ deadline=self._deadline,
+ on_error=self._on_error,
+ )
+
+ def with_delay(self, initial=None, maximum=None, multiplier=None):
+ """Return a copy of this retry with the given delay options.
+
+ Args:
+ initial (float): The minimum amout of time to delay. This must
+ be greater than 0.
+ maximum (float): The maximum amout of time to delay.
+ multiplier (float): The multiplier applied to the delay.
+
+ Returns:
+ Retry: A new retry instance with the given predicate.
+ """
+ return Retry(
+ predicate=self._predicate,
+ initial=initial if initial is not None else self._initial,
+ maximum=maximum if maximum is not None else self._maximum,
+ multiplier=multiplier if maximum is not None else self._multiplier,
+ deadline=self._deadline,
+ on_error=self._on_error,
+ )
+
+ def __str__(self):
+ return (
+ "".format(
+ self._predicate,
+ self._initial,
+ self._maximum,
+ self._multiplier,
+ self._deadline,
+ self._on_error,
+ )
+ )
diff --git a/api_core/google/api_core/timeout.py b/api_core/google/api_core/timeout.py
new file mode 100644
index 000000000000..17c1beabb3c5
--- /dev/null
+++ b/api_core/google/api_core/timeout.py
@@ -0,0 +1,224 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Decorators for applying timeout arguments to functions.
+
+These decorators are used to wrap API methods to apply either a constant
+or exponential timeout argument.
+
+For example, imagine an API method that can take a while to return results,
+such as one that might block until a resource is ready:
+
+.. code-block:: python
+
+ def is_thing_ready(timeout=None):
+ response = requests.get('https://example.com/is_thing_ready')
+ response.raise_for_status()
+ return response.json()
+
+This module allows a function like this to be wrapped so that timeouts are
+automatically determined, for example:
+
+.. code-block:: python
+
+ timeout_ = timeout.ExponentialTimeout()
+ is_thing_ready_with_timeout = timeout_(is_thing_ready)
+
+ for n in range(10):
+ try:
+ is_thing_ready_with_timeout({'example': 'data'})
+ except:
+ pass
+
+In this example the first call to ``is_thing_ready`` will have a relatively
+small timeout (like 1 second). If the resource is available and the request
+completes quickly, the loop exits. But, if the resource isn't yet available
+and the request times out, it'll be retried - this time with a larger timeout.
+
+In the broader context these decorators are typically combined with
+:mod:`google.api_core.retry` to implement API methods with a signature that
+matches ``api_method(request, timeout=None, retry=None)``.
+"""
+
+from __future__ import unicode_literals
+
+import datetime
+
+import six
+
+from google.api_core import datetime_helpers
+from google.api_core import general_helpers
+
+_DEFAULT_INITIAL_TIMEOUT = 5.0 # seconds
+_DEFAULT_MAXIMUM_TIMEOUT = 30.0 # seconds
+_DEFAULT_TIMEOUT_MULTIPLIER = 2.0
+# If specified, must be in seconds. If none, deadline is not used in the
+# timeout calculation.
+_DEFAULT_DEADLINE = None
+
+
+@six.python_2_unicode_compatible
+class ConstantTimeout(object):
+ """A decorator that adds a constant timeout argument.
+
+ This is effectively equivalent to
+ ``functools.partial(func, timeout=timeout)``.
+
+ Args:
+ timeout (Optional[float]): the timeout (in seconds) to applied to the
+ wrapped function. If `None`, the target function is expected to
+ never timeout.
+ """
+
+ def __init__(self, timeout=None):
+ self._timeout = timeout
+
+ def __call__(self, func):
+ """Apply the timeout decorator.
+
+ Args:
+ func (Callable): The function to apply the timeout argument to.
+ This function must accept a timeout keyword argument.
+
+ Returns:
+ Callable: The wrapped function.
+ """
+
+ @general_helpers.wraps(func)
+ def func_with_timeout(*args, **kwargs):
+ """Wrapped function that adds timeout."""
+ kwargs["timeout"] = self._timeout
+ return func(*args, **kwargs)
+
+ return func_with_timeout
+
+ def __str__(self):
+ return "".format(self._timeout)
+
+
+def _exponential_timeout_generator(initial, maximum, multiplier, deadline):
+ """A generator that yields exponential timeout values.
+
+ Args:
+ initial (float): The initial timeout.
+ maximum (float): The maximum timeout.
+ multiplier (float): The multiplier applied to the timeout.
+ deadline (float): The overall deadline across all invocations.
+
+ Yields:
+ float: A timeout value.
+ """
+ if deadline is not None:
+ deadline_datetime = datetime_helpers.utcnow() + datetime.timedelta(
+ seconds=deadline
+ )
+ else:
+ deadline_datetime = datetime.datetime.max
+
+ timeout = initial
+ while True:
+ now = datetime_helpers.utcnow()
+ yield min(
+ # The calculated timeout based on invocations.
+ timeout,
+ # The set maximum timeout.
+ maximum,
+ # The remaining time before the deadline is reached.
+ float((deadline_datetime - now).seconds),
+ )
+ timeout = timeout * multiplier
+
+
+@six.python_2_unicode_compatible
+class ExponentialTimeout(object):
+ """A decorator that adds an exponentially increasing timeout argument.
+
+ This is useful if a function is called multiple times. Each time the
+ function is called this decorator will calculate a new timeout parameter
+ based on the the number of times the function has been called.
+
+ For example
+
+ .. code-block:: python
+
+ Args:
+ initial (float): The initial timeout to pass.
+ maximum (float): The maximum timeout for any one call.
+ multiplier (float): The multiplier applied to the timeout for each
+ invocation.
+ deadline (Optional[float]): The overall deadline across all
+ invocations. This is used to prevent a very large calculated
+ timeout from pushing the overall execution time over the deadline.
+ This is especially useful in conjuction with
+ :mod:`google.api_core.retry`. If ``None``, the timeouts will not
+ be adjusted to accomodate an overall deadline.
+ """
+
+ def __init__(
+ self,
+ initial=_DEFAULT_INITIAL_TIMEOUT,
+ maximum=_DEFAULT_MAXIMUM_TIMEOUT,
+ multiplier=_DEFAULT_TIMEOUT_MULTIPLIER,
+ deadline=_DEFAULT_DEADLINE,
+ ):
+ self._initial = initial
+ self._maximum = maximum
+ self._multiplier = multiplier
+ self._deadline = deadline
+
+ def with_deadline(self, deadline):
+ """Return a copy of this teimout with the given deadline.
+
+ Args:
+ deadline (float): The overall deadline across all invocations.
+
+ Returns:
+ ExponentialTimeout: A new instance with the given deadline.
+ """
+ return ExponentialTimeout(
+ initial=self._initial,
+ maximum=self._maximum,
+ multiplier=self._multiplier,
+ deadline=deadline,
+ )
+
+ def __call__(self, func):
+ """Apply the timeout decorator.
+
+ Args:
+ func (Callable): The function to apply the timeout argument to.
+ This function must accept a timeout keyword argument.
+
+ Returns:
+ Callable: The wrapped function.
+ """
+ timeouts = _exponential_timeout_generator(
+ self._initial, self._maximum, self._multiplier, self._deadline
+ )
+
+ @general_helpers.wraps(func)
+ def func_with_timeout(*args, **kwargs):
+ """Wrapped function that adds timeout."""
+ kwargs["timeout"] = next(timeouts)
+ return func(*args, **kwargs)
+
+ return func_with_timeout
+
+ def __str__(self):
+ return (
+ "".format(
+ self._initial, self._maximum, self._multiplier, self._deadline
+ )
+ )
diff --git a/api_core/noxfile.py b/api_core/noxfile.py
new file mode 100644
index 000000000000..5e70db20dcd1
--- /dev/null
+++ b/api_core/noxfile.py
@@ -0,0 +1,129 @@
+# Copyright 2016 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import absolute_import
+import os
+import shutil
+
+# https://github.com/google/importlab/issues/25
+import nox # pytype: disable=import-error
+
+
+def default(session):
+ """Default unit test session.
+
+ This is intended to be run **without** an interpreter set, so
+ that the current ``python`` (on the ``PATH``) or the version of
+ Python corresponding to the ``nox`` binary the ``PATH`` can
+ run the tests.
+ """
+ # Install all test dependencies, then install this package in-place.
+ session.install("mock", "pytest", "pytest-cov", "grpcio >= 1.0.2")
+ session.install("-e", ".")
+
+ # Run py.test against the unit tests.
+ session.run(
+ "py.test",
+ "--quiet",
+ "--cov=google.api_core",
+ "--cov=tests.unit",
+ "--cov-append",
+ "--cov-config=.coveragerc",
+ "--cov-report=",
+ "--cov-fail-under=97",
+ os.path.join("tests", "unit"),
+ *session.posargs
+ )
+
+
+@nox.session(python=["2.7", "3.5", "3.6", "3.7"])
+def unit(session):
+ """Run the unit test suite."""
+ default(session)
+
+
+@nox.session(python=["2.7", "3.5", "3.6", "3.7"])
+def unit_grpc_gcp(session):
+ """Run the unit test suite with grpcio-gcp installed."""
+
+ # Install grpcio-gcp
+ session.install("grpcio-gcp")
+
+ default(session)
+
+
+@nox.session(python="3.6")
+def lint(session):
+ """Run linters.
+
+ Returns a failure if the linters find linting errors or sufficiently
+ serious code quality issues.
+ """
+ session.install("flake8", "flake8-import-order")
+ session.install(".")
+ session.run("flake8", "google", "tests")
+
+
+@nox.session(python="3.6")
+def lint_setup_py(session):
+ """Verify that setup.py is valid (including RST check)."""
+
+ session.install("docutils", "Pygments")
+ session.run("python", "setup.py", "check", "--restructuredtext", "--strict")
+
+
+# No 2.7 due to https://github.com/google/importlab/issues/26.
+# No 3.7 because pytype supports up to 3.6 only.
+@nox.session(python="3.6")
+def pytype(session):
+ """Run type-checking."""
+ session.install(
+ ".", "grpcio >= 1.8.2", "grpcio-gcp >= 0.2.2", "pytype >= 2019.3.21"
+ )
+ session.run("pytype")
+
+
+@nox.session(python="3.6")
+def cover(session):
+ """Run the final coverage report.
+
+ This outputs the coverage report aggregating coverage from the unit
+ test runs (not system test runs), and then erases coverage data.
+ """
+ session.install("coverage", "pytest-cov")
+ session.run("coverage", "report", "--show-missing", "--fail-under=100")
+ session.run("coverage", "erase")
+
+
+@nox.session(python="3.7")
+def docs(session):
+ """Build the docs for this library."""
+
+ session.install(".", "grpcio >= 1.8.2", "grpcio-gcp >= 0.2.2")
+ session.install("-e", ".")
+ session.install("sphinx", "alabaster", "recommonmark")
+
+ shutil.rmtree(os.path.join("docs", "_build"), ignore_errors=True)
+ session.run(
+ "sphinx-build",
+ "-W", # warnings as errors
+ "-T", # show full traceback on exception
+ "-N", # no colors
+ "-b",
+ "html",
+ "-d",
+ os.path.join("docs", "_build", "doctrees", ""),
+ os.path.join("docs", ""),
+ os.path.join("docs", "_build", "html", ""),
+ )
\ No newline at end of file
diff --git a/api_core/setup.cfg b/api_core/setup.cfg
new file mode 100644
index 000000000000..5c32e1664d2c
--- /dev/null
+++ b/api_core/setup.cfg
@@ -0,0 +1,12 @@
+[bdist_wheel]
+universal = 1
+
+[pytype]
+python_version = 3.6
+inputs =
+ google/
+exclude =
+ tests/
+output = pytype_output/
+# Workaround for https://github.com/google/pytype/issues/150
+disable = pyi-error
diff --git a/api_core/setup.py b/api_core/setup.py
new file mode 100644
index 000000000000..25b7072d91f2
--- /dev/null
+++ b/api_core/setup.py
@@ -0,0 +1,99 @@
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import io
+import os
+
+import setuptools
+
+
+# Package metadata.
+
+name = "google-api-core"
+description = "Google API client core library"
+version = "1.14.2"
+# Should be one of:
+# 'Development Status :: 3 - Alpha'
+# 'Development Status :: 4 - Beta'
+# 'Development Status :: 5 - Production/Stable'
+release_status = "Development Status :: 5 - Production/Stable"
+dependencies = [
+ "googleapis-common-protos >= 1.6.0, < 2.0dev",
+ "protobuf >= 3.4.0",
+ "google-auth >= 0.4.0, < 2.0dev",
+ "requests >= 2.18.0, < 3.0.0dev",
+ "setuptools >= 34.0.0",
+ "six >= 1.10.0",
+ "pytz",
+ 'futures >= 3.2.0; python_version < "3.2"',
+]
+extras = {
+ "grpc": "grpcio >= 1.8.2, < 2.0dev",
+ "grpcgcp": "grpcio-gcp >= 0.2.2",
+ "grpcio-gcp": "grpcio-gcp >= 0.2.2",
+}
+
+
+# Setup boilerplate below this line.
+
+package_root = os.path.abspath(os.path.dirname(__file__))
+
+readme_filename = os.path.join(package_root, "README.rst")
+with io.open(readme_filename, encoding="utf-8") as readme_file:
+ readme = readme_file.read()
+
+# Only include packages under the 'google' namespace. Do not include tests,
+# benchmarks, etc.
+packages = [
+ package for package in setuptools.find_packages() if package.startswith("google")
+]
+
+# Determine which namespaces are needed.
+namespaces = ["google"]
+if "google.cloud" in packages:
+ namespaces.append("google.cloud")
+
+
+setuptools.setup(
+ name=name,
+ version=version,
+ description=description,
+ long_description=readme,
+ author="Google LLC",
+ author_email="googleapis-packages@google.com",
+ license="Apache 2.0",
+ url="https://github.com/GoogleCloudPlatform/google-cloud-python",
+ classifiers=[
+ release_status,
+ "Intended Audience :: Developers",
+ "License :: OSI Approved :: Apache Software License",
+ "Programming Language :: Python",
+ "Programming Language :: Python :: 2",
+ "Programming Language :: Python :: 2.7",
+ "Programming Language :: Python :: 3",
+ "Programming Language :: Python :: 3.5",
+ "Programming Language :: Python :: 3.6",
+ "Programming Language :: Python :: 3.7",
+ "Operating System :: OS Independent",
+ "Topic :: Internet",
+ ],
+ platforms="Posix; MacOS X; Windows",
+ packages=packages,
+ namespace_packages=namespaces,
+ install_requires=dependencies,
+ extras_require=extras,
+ python_requires='>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*',
+ include_package_data=True,
+ zip_safe=False,
+)
diff --git a/api_core/tests/__init__.py b/api_core/tests/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/api_core/tests/unit/__init__.py b/api_core/tests/unit/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/api_core/tests/unit/future/__init__.py b/api_core/tests/unit/future/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/api_core/tests/unit/future/test__helpers.py b/api_core/tests/unit/future/test__helpers.py
new file mode 100644
index 000000000000..98afc599f84e
--- /dev/null
+++ b/api_core/tests/unit/future/test__helpers.py
@@ -0,0 +1,37 @@
+# Copyright 2017, Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import mock
+
+from google.api_core.future import _helpers
+
+
+@mock.patch("threading.Thread", autospec=True)
+def test_start_deamon_thread(unused_thread):
+ deamon_thread = _helpers.start_daemon_thread(target=mock.sentinel.target)
+ assert deamon_thread.daemon is True
+
+
+def test_safe_invoke_callback():
+ callback = mock.Mock(spec=["__call__"], return_value=42)
+ result = _helpers.safe_invoke_callback(callback, "a", b="c")
+ assert result == 42
+ callback.assert_called_once_with("a", b="c")
+
+
+def test_safe_invoke_callback_exception():
+ callback = mock.Mock(spec=["__call__"], side_effect=ValueError())
+ result = _helpers.safe_invoke_callback(callback, "a", b="c")
+ assert result is None
+ callback.assert_called_once_with("a", b="c")
diff --git a/api_core/tests/unit/future/test_polling.py b/api_core/tests/unit/future/test_polling.py
new file mode 100644
index 000000000000..c67de064add5
--- /dev/null
+++ b/api_core/tests/unit/future/test_polling.py
@@ -0,0 +1,194 @@
+# Copyright 2017, Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import concurrent.futures
+import threading
+import time
+
+import mock
+import pytest
+
+from google.api_core import exceptions
+from google.api_core.future import polling
+
+
+class PollingFutureImpl(polling.PollingFuture):
+ def done(self):
+ return False
+
+ def cancel(self):
+ return True
+
+ def cancelled(self):
+ return False
+
+ def running(self):
+ return True
+
+
+def test_polling_future_constructor():
+ future = PollingFutureImpl()
+ assert not future.done()
+ assert not future.cancelled()
+ assert future.running()
+ assert future.cancel()
+
+
+def test_set_result():
+ future = PollingFutureImpl()
+ callback = mock.Mock()
+
+ future.set_result(1)
+
+ assert future.result() == 1
+ future.add_done_callback(callback)
+ callback.assert_called_once_with(future)
+
+
+def test_set_exception():
+ future = PollingFutureImpl()
+ exception = ValueError("meep")
+
+ future.set_exception(exception)
+
+ assert future.exception() == exception
+ with pytest.raises(ValueError):
+ future.result()
+
+ callback = mock.Mock()
+ future.add_done_callback(callback)
+ callback.assert_called_once_with(future)
+
+
+def test_invoke_callback_exception():
+ future = PollingFutureImplWithPoll()
+ future.set_result(42)
+
+ # This should not raise, despite the callback causing an exception.
+ callback = mock.Mock(side_effect=ValueError)
+ future.add_done_callback(callback)
+ callback.assert_called_once_with(future)
+
+
+class PollingFutureImplWithPoll(PollingFutureImpl):
+ def __init__(self):
+ super(PollingFutureImplWithPoll, self).__init__()
+ self.poll_count = 0
+ self.event = threading.Event()
+
+ def done(self):
+ self.poll_count += 1
+ self.event.wait()
+ self.set_result(42)
+ return True
+
+
+def test_result_with_polling():
+ future = PollingFutureImplWithPoll()
+
+ future.event.set()
+ result = future.result()
+
+ assert result == 42
+ assert future.poll_count == 1
+ # Repeated calls should not cause additional polling
+ assert future.result() == result
+ assert future.poll_count == 1
+
+
+class PollingFutureImplTimeout(PollingFutureImplWithPoll):
+ def done(self):
+ time.sleep(1)
+ return False
+
+
+def test_result_timeout():
+ future = PollingFutureImplTimeout()
+ with pytest.raises(concurrent.futures.TimeoutError):
+ future.result(timeout=1)
+
+
+def test_exception_timeout():
+ future = PollingFutureImplTimeout()
+ with pytest.raises(concurrent.futures.TimeoutError):
+ future.exception(timeout=1)
+
+
+class PollingFutureImplTransient(PollingFutureImplWithPoll):
+ def __init__(self, errors):
+ super(PollingFutureImplTransient, self).__init__()
+ self._errors = errors
+
+ def done(self):
+ if self._errors:
+ error, self._errors = self._errors[0], self._errors[1:]
+ raise error("testing")
+ self.poll_count += 1
+ self.set_result(42)
+ return True
+
+
+def test_result_transient_error():
+ future = PollingFutureImplTransient(
+ (
+ exceptions.TooManyRequests,
+ exceptions.InternalServerError,
+ exceptions.BadGateway,
+ )
+ )
+ result = future.result()
+ assert result == 42
+ assert future.poll_count == 1
+ # Repeated calls should not cause additional polling
+ assert future.result() == result
+ assert future.poll_count == 1
+
+
+def test_callback_background_thread():
+ future = PollingFutureImplWithPoll()
+ callback = mock.Mock()
+
+ future.add_done_callback(callback)
+
+ assert future._polling_thread is not None
+
+ # Give the thread a second to poll
+ time.sleep(1)
+ assert future.poll_count == 1
+
+ future.event.set()
+ future._polling_thread.join()
+
+ callback.assert_called_once_with(future)
+
+
+def test_double_callback_background_thread():
+ future = PollingFutureImplWithPoll()
+ callback = mock.Mock()
+ callback2 = mock.Mock()
+
+ future.add_done_callback(callback)
+ current_thread = future._polling_thread
+ assert current_thread is not None
+
+ # only one polling thread should be created.
+ future.add_done_callback(callback2)
+ assert future._polling_thread is current_thread
+
+ future.event.set()
+ future._polling_thread.join()
+
+ assert future.poll_count == 1
+ callback.assert_called_once_with(future)
+ callback2.assert_called_once_with(future)
diff --git a/api_core/tests/unit/gapic/test_client_info.py b/api_core/tests/unit/gapic/test_client_info.py
new file mode 100644
index 000000000000..64080ffdbccd
--- /dev/null
+++ b/api_core/tests/unit/gapic/test_client_info.py
@@ -0,0 +1,24 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from google.api_core.gapic_v1 import client_info
+
+
+def test_to_grpc_metadata():
+ info = client_info.ClientInfo()
+
+ metadata = info.to_grpc_metadata()
+
+ assert metadata == (client_info.METRICS_METADATA_KEY, info.to_user_agent())
diff --git a/api_core/tests/unit/gapic/test_config.py b/api_core/tests/unit/gapic/test_config.py
new file mode 100644
index 000000000000..1c15261da0dd
--- /dev/null
+++ b/api_core/tests/unit/gapic/test_config.py
@@ -0,0 +1,87 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from google.api_core import exceptions
+from google.api_core.gapic_v1 import config
+
+
+INTERFACE_CONFIG = {
+ "retry_codes": {
+ "idempotent": ["DEADLINE_EXCEEDED", "UNAVAILABLE"],
+ "other": ["FAILED_PRECONDITION"],
+ "non_idempotent": [],
+ },
+ "retry_params": {
+ "default": {
+ "initial_retry_delay_millis": 1000,
+ "retry_delay_multiplier": 2.5,
+ "max_retry_delay_millis": 120000,
+ "initial_rpc_timeout_millis": 120000,
+ "rpc_timeout_multiplier": 1.0,
+ "max_rpc_timeout_millis": 120000,
+ "total_timeout_millis": 600000,
+ },
+ "other": {
+ "initial_retry_delay_millis": 1000,
+ "retry_delay_multiplier": 1,
+ "max_retry_delay_millis": 1000,
+ "initial_rpc_timeout_millis": 1000,
+ "rpc_timeout_multiplier": 1,
+ "max_rpc_timeout_millis": 1000,
+ "total_timeout_millis": 1000,
+ },
+ },
+ "methods": {
+ "AnnotateVideo": {
+ "timeout_millis": 60000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "Other": {
+ "timeout_millis": 60000,
+ "retry_codes_name": "other",
+ "retry_params_name": "other",
+ },
+ "Plain": {"timeout_millis": 30000},
+ },
+}
+
+
+def test_create_method_configs():
+ method_configs = config.parse_method_configs(INTERFACE_CONFIG)
+
+ retry, timeout = method_configs["AnnotateVideo"]
+ assert retry._predicate(exceptions.DeadlineExceeded(None))
+ assert retry._predicate(exceptions.ServiceUnavailable(None))
+ assert retry._initial == 1.0
+ assert retry._multiplier == 2.5
+ assert retry._maximum == 120.0
+ assert retry._deadline == 600.0
+ assert timeout._initial == 120.0
+ assert timeout._multiplier == 1.0
+ assert timeout._maximum == 120.0
+
+ retry, timeout = method_configs["Other"]
+ assert retry._predicate(exceptions.FailedPrecondition(None))
+ assert retry._initial == 1.0
+ assert retry._multiplier == 1.0
+ assert retry._maximum == 1.0
+ assert retry._deadline == 1.0
+ assert timeout._initial == 1.0
+ assert timeout._multiplier == 1.0
+ assert timeout._maximum == 1.0
+
+ retry, timeout = method_configs["Plain"]
+ assert retry is None
+ assert timeout._timeout == 30.0
diff --git a/api_core/tests/unit/gapic/test_method.py b/api_core/tests/unit/gapic/test_method.py
new file mode 100644
index 000000000000..0f9bee934924
--- /dev/null
+++ b/api_core/tests/unit/gapic/test_method.py
@@ -0,0 +1,216 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import datetime
+
+import mock
+
+from google.api_core import exceptions
+from google.api_core import retry
+from google.api_core import timeout
+import google.api_core.gapic_v1.client_info
+import google.api_core.gapic_v1.method
+import google.api_core.page_iterator
+
+
+def _utcnow_monotonic():
+ curr_value = datetime.datetime.min
+ delta = datetime.timedelta(seconds=0.5)
+ while True:
+ yield curr_value
+ curr_value += delta
+
+
+def test_wrap_method_basic():
+ method = mock.Mock(spec=["__call__"], return_value=42)
+
+ wrapped_method = google.api_core.gapic_v1.method.wrap_method(method)
+
+ result = wrapped_method(1, 2, meep="moop")
+
+ assert result == 42
+ method.assert_called_once_with(1, 2, meep="moop", metadata=mock.ANY)
+
+ # Check that the default client info was specified in the metadata.
+ metadata = method.call_args[1]["metadata"]
+ assert len(metadata) == 1
+ client_info = google.api_core.gapic_v1.client_info.DEFAULT_CLIENT_INFO
+ user_agent_metadata = client_info.to_grpc_metadata()
+ assert user_agent_metadata in metadata
+
+
+def test_wrap_method_with_no_client_info():
+ method = mock.Mock(spec=["__call__"])
+
+ wrapped_method = google.api_core.gapic_v1.method.wrap_method(
+ method, client_info=None
+ )
+
+ wrapped_method(1, 2, meep="moop")
+
+ method.assert_called_once_with(1, 2, meep="moop")
+
+
+def test_wrap_method_with_custom_client_info():
+ client_info = google.api_core.gapic_v1.client_info.ClientInfo(
+ python_version=1,
+ grpc_version=2,
+ api_core_version=3,
+ gapic_version=4,
+ client_library_version=5,
+ )
+ method = mock.Mock(spec=["__call__"])
+
+ wrapped_method = google.api_core.gapic_v1.method.wrap_method(
+ method, client_info=client_info
+ )
+
+ wrapped_method(1, 2, meep="moop")
+
+ method.assert_called_once_with(1, 2, meep="moop", metadata=mock.ANY)
+
+ # Check that the custom client info was specified in the metadata.
+ metadata = method.call_args[1]["metadata"]
+ assert client_info.to_grpc_metadata() in metadata
+
+
+def test_invoke_wrapped_method_with_metadata():
+ method = mock.Mock(spec=["__call__"])
+
+ wrapped_method = google.api_core.gapic_v1.method.wrap_method(method)
+
+ wrapped_method(mock.sentinel.request, metadata=[("a", "b")])
+
+ method.assert_called_once_with(mock.sentinel.request, metadata=mock.ANY)
+ metadata = method.call_args[1]["metadata"]
+ # Metadata should have two items: the client info metadata and our custom
+ # metadata.
+ assert len(metadata) == 2
+ assert ("a", "b") in metadata
+
+
+def test_invoke_wrapped_method_with_metadata_as_none():
+ method = mock.Mock(spec=["__call__"])
+
+ wrapped_method = google.api_core.gapic_v1.method.wrap_method(method)
+
+ wrapped_method(mock.sentinel.request, metadata=None)
+
+ method.assert_called_once_with(mock.sentinel.request, metadata=mock.ANY)
+ metadata = method.call_args[1]["metadata"]
+ # Metadata should have just one items: the client info metadata.
+ assert len(metadata) == 1
+
+
+@mock.patch("time.sleep")
+def test_wrap_method_with_default_retry_and_timeout(unusued_sleep):
+ method = mock.Mock(
+ spec=["__call__"], side_effect=[exceptions.InternalServerError(None), 42]
+ )
+ default_retry = retry.Retry()
+ default_timeout = timeout.ConstantTimeout(60)
+ wrapped_method = google.api_core.gapic_v1.method.wrap_method(
+ method, default_retry, default_timeout
+ )
+
+ result = wrapped_method()
+
+ assert result == 42
+ assert method.call_count == 2
+ method.assert_called_with(timeout=60, metadata=mock.ANY)
+
+
+@mock.patch("time.sleep")
+def test_wrap_method_with_default_retry_and_timeout_using_sentinel(unusued_sleep):
+ method = mock.Mock(
+ spec=["__call__"], side_effect=[exceptions.InternalServerError(None), 42]
+ )
+ default_retry = retry.Retry()
+ default_timeout = timeout.ConstantTimeout(60)
+ wrapped_method = google.api_core.gapic_v1.method.wrap_method(
+ method, default_retry, default_timeout
+ )
+
+ result = wrapped_method(
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ )
+
+ assert result == 42
+ assert method.call_count == 2
+ method.assert_called_with(timeout=60, metadata=mock.ANY)
+
+
+@mock.patch("time.sleep")
+def test_wrap_method_with_overriding_retry_and_timeout(unusued_sleep):
+ method = mock.Mock(spec=["__call__"], side_effect=[exceptions.NotFound(None), 42])
+ default_retry = retry.Retry()
+ default_timeout = timeout.ConstantTimeout(60)
+ wrapped_method = google.api_core.gapic_v1.method.wrap_method(
+ method, default_retry, default_timeout
+ )
+
+ result = wrapped_method(
+ retry=retry.Retry(retry.if_exception_type(exceptions.NotFound)),
+ timeout=timeout.ConstantTimeout(22),
+ )
+
+ assert result == 42
+ assert method.call_count == 2
+ method.assert_called_with(timeout=22, metadata=mock.ANY)
+
+
+@mock.patch("time.sleep")
+@mock.patch(
+ "google.api_core.datetime_helpers.utcnow",
+ side_effect=_utcnow_monotonic(),
+ autospec=True,
+)
+def test_wrap_method_with_overriding_retry_deadline(utcnow, unused_sleep):
+ method = mock.Mock(
+ spec=["__call__"],
+ side_effect=([exceptions.InternalServerError(None)] * 4) + [42],
+ )
+ default_retry = retry.Retry()
+ default_timeout = timeout.ExponentialTimeout(deadline=60)
+ wrapped_method = google.api_core.gapic_v1.method.wrap_method(
+ method, default_retry, default_timeout
+ )
+
+ # Overriding only the retry's deadline should also override the timeout's
+ # deadline.
+ result = wrapped_method(retry=default_retry.with_deadline(30))
+
+ assert result == 42
+ timeout_args = [call[1]["timeout"] for call in method.call_args_list]
+ assert timeout_args == [5.0, 10.0, 20.0, 26.0, 25.0]
+ assert utcnow.call_count == (
+ 1
+ + 5 # First to set the deadline.
+ + 5 # One for each min(timeout, maximum, (DEADLINE - NOW).seconds)
+ )
+
+
+def test_wrap_method_with_overriding_timeout_as_a_number():
+ method = mock.Mock(spec=["__call__"], return_value=42)
+ default_retry = retry.Retry()
+ default_timeout = timeout.ConstantTimeout(60)
+ wrapped_method = google.api_core.gapic_v1.method.wrap_method(
+ method, default_retry, default_timeout
+ )
+
+ result = wrapped_method(timeout=22)
+
+ assert result == 42
+ method.assert_called_once_with(timeout=22, metadata=mock.ANY)
diff --git a/api_core/tests/unit/gapic/test_routing_header.py b/api_core/tests/unit/gapic/test_routing_header.py
new file mode 100644
index 000000000000..77300e8739d7
--- /dev/null
+++ b/api_core/tests/unit/gapic/test_routing_header.py
@@ -0,0 +1,34 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from google.api_core.gapic_v1 import routing_header
+
+
+def test_to_routing_header():
+ params = [("name", "meep"), ("book.read", "1")]
+ value = routing_header.to_routing_header(params)
+ assert value == "name=meep&book.read=1"
+
+
+def test_to_routing_header_with_slashes():
+ params = [("name", "me/ep"), ("book.read", "1&2")]
+ value = routing_header.to_routing_header(params)
+ assert value == "name=me/ep&book.read=1%262"
+
+
+def test_to_grpc_metadata():
+ params = [("name", "meep"), ("book.read", "1")]
+ metadata = routing_header.to_grpc_metadata(params)
+ assert metadata == (routing_header.ROUTING_METADATA_KEY, "name=meep&book.read=1")
diff --git a/api_core/tests/unit/operations_v1/__init__.py b/api_core/tests/unit/operations_v1/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/api_core/tests/unit/operations_v1/test_operations_client.py b/api_core/tests/unit/operations_v1/test_operations_client.py
new file mode 100644
index 000000000000..cc5746123cc0
--- /dev/null
+++ b/api_core/tests/unit/operations_v1/test_operations_client.py
@@ -0,0 +1,75 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from google.api_core import grpc_helpers
+from google.api_core import operations_v1
+from google.api_core import page_iterator
+from google.longrunning import operations_pb2
+from google.protobuf import empty_pb2
+
+
+def test_get_operation():
+ channel = grpc_helpers.ChannelStub()
+ client = operations_v1.OperationsClient(channel)
+ channel.GetOperation.response = operations_pb2.Operation(name="meep")
+
+ response = client.get_operation("name")
+
+ assert len(channel.GetOperation.requests) == 1
+ assert channel.GetOperation.requests[0].name == "name"
+ assert response == channel.GetOperation.response
+
+
+def test_list_operations():
+ channel = grpc_helpers.ChannelStub()
+ client = operations_v1.OperationsClient(channel)
+ operations = [
+ operations_pb2.Operation(name="1"),
+ operations_pb2.Operation(name="2"),
+ ]
+ list_response = operations_pb2.ListOperationsResponse(operations=operations)
+ channel.ListOperations.response = list_response
+
+ response = client.list_operations("name", "filter")
+
+ assert isinstance(response, page_iterator.Iterator)
+ assert list(response) == operations
+
+ assert len(channel.ListOperations.requests) == 1
+ request = channel.ListOperations.requests[0]
+ assert isinstance(request, operations_pb2.ListOperationsRequest)
+ assert request.name == "name"
+ assert request.filter == "filter"
+
+
+def test_delete_operation():
+ channel = grpc_helpers.ChannelStub()
+ client = operations_v1.OperationsClient(channel)
+ channel.DeleteOperation.response = empty_pb2.Empty()
+
+ client.delete_operation("name")
+
+ assert len(channel.DeleteOperation.requests) == 1
+ assert channel.DeleteOperation.requests[0].name == "name"
+
+
+def test_cancel_operation():
+ channel = grpc_helpers.ChannelStub()
+ client = operations_v1.OperationsClient(channel)
+ channel.CancelOperation.response = empty_pb2.Empty()
+
+ client.cancel_operation("name")
+
+ assert len(channel.CancelOperation.requests) == 1
+ assert channel.CancelOperation.requests[0].name == "name"
diff --git a/api_core/tests/unit/test_bidi.py b/api_core/tests/unit/test_bidi.py
new file mode 100644
index 000000000000..4d185d3158e4
--- /dev/null
+++ b/api_core/tests/unit/test_bidi.py
@@ -0,0 +1,847 @@
+# Copyright 2018, Google LLC All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import datetime
+import logging
+import threading
+
+import grpc
+import mock
+import pytest
+from six.moves import queue
+
+from google.api_core import bidi
+from google.api_core import exceptions
+
+
+class Test_RequestQueueGenerator(object):
+ def test_bounded_consume(self):
+ call = mock.create_autospec(grpc.Call, instance=True)
+ call.is_active.return_value = True
+
+ def queue_generator(rpc):
+ yield mock.sentinel.A
+ yield queue.Empty()
+ yield mock.sentinel.B
+ rpc.is_active.return_value = False
+ yield mock.sentinel.C
+
+ q = mock.create_autospec(queue.Queue, instance=True)
+ q.get.side_effect = queue_generator(call)
+
+ generator = bidi._RequestQueueGenerator(q)
+ generator.call = call
+
+ items = list(generator)
+
+ assert items == [mock.sentinel.A, mock.sentinel.B]
+
+ def test_yield_initial_and_exit(self):
+ q = mock.create_autospec(queue.Queue, instance=True)
+ q.get.side_effect = queue.Empty()
+ call = mock.create_autospec(grpc.Call, instance=True)
+ call.is_active.return_value = False
+
+ generator = bidi._RequestQueueGenerator(q, initial_request=mock.sentinel.A)
+ generator.call = call
+
+ items = list(generator)
+
+ assert items == [mock.sentinel.A]
+
+ def test_yield_initial_callable_and_exit(self):
+ q = mock.create_autospec(queue.Queue, instance=True)
+ q.get.side_effect = queue.Empty()
+ call = mock.create_autospec(grpc.Call, instance=True)
+ call.is_active.return_value = False
+
+ generator = bidi._RequestQueueGenerator(
+ q, initial_request=lambda: mock.sentinel.A
+ )
+ generator.call = call
+
+ items = list(generator)
+
+ assert items == [mock.sentinel.A]
+
+ def test_exit_when_inactive_with_item(self):
+ q = mock.create_autospec(queue.Queue, instance=True)
+ q.get.side_effect = [mock.sentinel.A, queue.Empty()]
+ call = mock.create_autospec(grpc.Call, instance=True)
+ call.is_active.return_value = False
+
+ generator = bidi._RequestQueueGenerator(q)
+ generator.call = call
+
+ items = list(generator)
+
+ assert items == []
+ # Make sure it put the item back.
+ q.put.assert_called_once_with(mock.sentinel.A)
+
+ def test_exit_when_inactive_empty(self):
+ q = mock.create_autospec(queue.Queue, instance=True)
+ q.get.side_effect = queue.Empty()
+ call = mock.create_autospec(grpc.Call, instance=True)
+ call.is_active.return_value = False
+
+ generator = bidi._RequestQueueGenerator(q)
+ generator.call = call
+
+ items = list(generator)
+
+ assert items == []
+
+ def test_exit_with_stop(self):
+ q = mock.create_autospec(queue.Queue, instance=True)
+ q.get.side_effect = [None, queue.Empty()]
+ call = mock.create_autospec(grpc.Call, instance=True)
+ call.is_active.return_value = True
+
+ generator = bidi._RequestQueueGenerator(q)
+ generator.call = call
+
+ items = list(generator)
+
+ assert items == []
+
+
+class Test_Throttle(object):
+ def test_repr(self):
+ delta = datetime.timedelta(seconds=4.5)
+ instance = bidi._Throttle(access_limit=42, time_window=delta)
+ assert repr(instance) == \
+ "_Throttle(access_limit=42, time_window={})".format(repr(delta))
+
+ def test_raises_error_on_invalid_init_arguments(self):
+ with pytest.raises(ValueError) as exc_info:
+ bidi._Throttle(
+ access_limit=10, time_window=datetime.timedelta(seconds=0.0)
+ )
+ assert "time_window" in str(exc_info.value)
+ assert "must be a positive timedelta" in str(exc_info.value)
+
+ with pytest.raises(ValueError) as exc_info:
+ bidi._Throttle(
+ access_limit=0, time_window=datetime.timedelta(seconds=10)
+ )
+ assert "access_limit" in str(exc_info.value)
+ assert "must be positive" in str(exc_info.value)
+
+ def test_does_not_delay_entry_attempts_under_threshold(self):
+ throttle = bidi._Throttle(
+ access_limit=3, time_window=datetime.timedelta(seconds=1)
+ )
+ entries = []
+
+ for _ in range(3):
+ with throttle as time_waited:
+ entry_info = {
+ "entered_at": datetime.datetime.now(),
+ "reported_wait": time_waited,
+ }
+ entries.append(entry_info)
+
+ # check the reported wait times ...
+ assert all(entry["reported_wait"] == 0.0 for entry in entries)
+
+ # .. and the actual wait times
+ delta = entries[1]["entered_at"] - entries[0]["entered_at"]
+ assert delta.total_seconds() < 0.1
+ delta = entries[2]["entered_at"] - entries[1]["entered_at"]
+ assert delta.total_seconds() < 0.1
+
+ def test_delays_entry_attempts_above_threshold(self):
+ throttle = bidi._Throttle(
+ access_limit=3, time_window=datetime.timedelta(seconds=1)
+ )
+ entries = []
+
+ for _ in range(6):
+ with throttle as time_waited:
+ entry_info = {
+ "entered_at": datetime.datetime.now(),
+ "reported_wait": time_waited,
+ }
+ entries.append(entry_info)
+
+ # For each group of 4 consecutive entries the time difference between
+ # the first and the last entry must have been greater than time_window,
+ # because a maximum of 3 are allowed in each time_window.
+ for i, entry in enumerate(entries[3:], start=3):
+ first_entry = entries[i - 3]
+ delta = entry["entered_at"] - first_entry["entered_at"]
+ assert delta.total_seconds() > 1.0
+
+ # check the reported wait times
+ # (NOTE: not using assert all(...), b/c the coverage check would complain)
+ for i, entry in enumerate(entries):
+ if i != 3:
+ assert entry["reported_wait"] == 0.0
+
+ # The delayed entry is expected to have been delayed for a significant
+ # chunk of the full second, and the actual and reported delay times
+ # should reflect that.
+ assert entries[3]["reported_wait"] > 0.7
+ delta = entries[3]["entered_at"] - entries[2]["entered_at"]
+ assert delta.total_seconds() > 0.7
+
+
+class _CallAndFuture(grpc.Call, grpc.Future):
+ pass
+
+
+def make_rpc():
+ """Makes a mock RPC used to test Bidi classes."""
+ call = mock.create_autospec(_CallAndFuture, instance=True)
+ rpc = mock.create_autospec(grpc.StreamStreamMultiCallable, instance=True)
+
+ def rpc_side_effect(request, metadata=None):
+ call.is_active.return_value = True
+ call.request = request
+ call.metadata = metadata
+ return call
+
+ rpc.side_effect = rpc_side_effect
+
+ def cancel_side_effect():
+ call.is_active.return_value = False
+
+ call.cancel.side_effect = cancel_side_effect
+
+ return rpc, call
+
+
+class ClosedCall(object):
+ # NOTE: This is needed because defining `.next` on an **instance**
+ # rather than the **class** will not be iterable in Python 2.
+ # This is problematic since a `Mock` just sets members.
+
+ def __init__(self, exception):
+ self.exception = exception
+
+ def __next__(self):
+ raise self.exception
+
+ next = __next__ # Python 2
+
+ def is_active(self):
+ return False
+
+
+class TestBidiRpc(object):
+ def test_initial_state(self):
+ bidi_rpc = bidi.BidiRpc(None)
+
+ assert bidi_rpc.is_active is False
+
+ def test_done_callbacks(self):
+ bidi_rpc = bidi.BidiRpc(None)
+ callback = mock.Mock(spec=["__call__"])
+
+ bidi_rpc.add_done_callback(callback)
+ bidi_rpc._on_call_done(mock.sentinel.future)
+
+ callback.assert_called_once_with(mock.sentinel.future)
+
+ def test_metadata(self):
+ rpc, call = make_rpc()
+ bidi_rpc = bidi.BidiRpc(rpc, metadata=mock.sentinel.A)
+ assert bidi_rpc._rpc_metadata == mock.sentinel.A
+
+ bidi_rpc.open()
+ assert bidi_rpc.call == call
+ assert bidi_rpc.call.metadata == mock.sentinel.A
+
+ def test_open(self):
+ rpc, call = make_rpc()
+ bidi_rpc = bidi.BidiRpc(rpc)
+
+ bidi_rpc.open()
+
+ assert bidi_rpc.call == call
+ assert bidi_rpc.is_active
+ call.add_done_callback.assert_called_once_with(bidi_rpc._on_call_done)
+
+ def test_open_error_already_open(self):
+ rpc, _ = make_rpc()
+ bidi_rpc = bidi.BidiRpc(rpc)
+
+ bidi_rpc.open()
+
+ with pytest.raises(ValueError):
+ bidi_rpc.open()
+
+ def test_close(self):
+ rpc, call = make_rpc()
+ bidi_rpc = bidi.BidiRpc(rpc)
+ bidi_rpc.open()
+
+ bidi_rpc.close()
+
+ call.cancel.assert_called_once()
+ assert bidi_rpc.call == call
+ assert bidi_rpc.is_active is False
+ # ensure the request queue was signaled to stop.
+ assert bidi_rpc.pending_requests == 1
+ assert bidi_rpc._request_queue.get() is None
+
+ def test_close_no_rpc(self):
+ bidi_rpc = bidi.BidiRpc(None)
+ bidi_rpc.close()
+
+ def test_send(self):
+ rpc, call = make_rpc()
+ bidi_rpc = bidi.BidiRpc(rpc)
+ bidi_rpc.open()
+
+ bidi_rpc.send(mock.sentinel.request)
+
+ assert bidi_rpc.pending_requests == 1
+ assert bidi_rpc._request_queue.get() is mock.sentinel.request
+
+ def test_send_not_open(self):
+ rpc, call = make_rpc()
+ bidi_rpc = bidi.BidiRpc(rpc)
+
+ with pytest.raises(ValueError):
+ bidi_rpc.send(mock.sentinel.request)
+
+ def test_send_dead_rpc(self):
+ error = ValueError()
+ bidi_rpc = bidi.BidiRpc(None)
+ bidi_rpc.call = ClosedCall(error)
+
+ with pytest.raises(ValueError) as exc_info:
+ bidi_rpc.send(mock.sentinel.request)
+
+ assert exc_info.value == error
+
+ def test_recv(self):
+ bidi_rpc = bidi.BidiRpc(None)
+ bidi_rpc.call = iter([mock.sentinel.response])
+
+ response = bidi_rpc.recv()
+
+ assert response == mock.sentinel.response
+
+ def test_recv_not_open(self):
+ rpc, call = make_rpc()
+ bidi_rpc = bidi.BidiRpc(rpc)
+
+ with pytest.raises(ValueError):
+ bidi_rpc.recv()
+
+
+class CallStub(object):
+ def __init__(self, values, active=True):
+ self.values = iter(values)
+ self._is_active = active
+ self.cancelled = False
+
+ def __next__(self):
+ item = next(self.values)
+ if isinstance(item, Exception):
+ self._is_active = False
+ raise item
+ return item
+
+ next = __next__ # Python 2
+
+ def is_active(self):
+ return self._is_active
+
+ def add_done_callback(self, callback):
+ pass
+
+ def cancel(self):
+ self.cancelled = True
+
+
+class TestResumableBidiRpc(object):
+ def test_ctor_defaults(self):
+ start_rpc = mock.Mock()
+ should_recover = mock.Mock()
+ bidi_rpc = bidi.ResumableBidiRpc(start_rpc, should_recover)
+
+ assert bidi_rpc.is_active is False
+ assert bidi_rpc._finalized is False
+ assert bidi_rpc._start_rpc is start_rpc
+ assert bidi_rpc._should_recover is should_recover
+ assert bidi_rpc._should_terminate is bidi._never_terminate
+ assert bidi_rpc._initial_request is None
+ assert bidi_rpc._rpc_metadata is None
+ assert bidi_rpc._reopen_throttle is None
+
+ def test_ctor_explicit(self):
+ start_rpc = mock.Mock()
+ should_recover = mock.Mock()
+ should_terminate = mock.Mock()
+ initial_request = mock.Mock()
+ metadata = {"x-foo": "bar"}
+ bidi_rpc = bidi.ResumableBidiRpc(
+ start_rpc,
+ should_recover,
+ should_terminate=should_terminate,
+ initial_request=initial_request,
+ metadata=metadata,
+ throttle_reopen=True,
+ )
+
+ assert bidi_rpc.is_active is False
+ assert bidi_rpc._finalized is False
+ assert bidi_rpc._should_recover is should_recover
+ assert bidi_rpc._should_terminate is should_terminate
+ assert bidi_rpc._initial_request is initial_request
+ assert bidi_rpc._rpc_metadata == metadata
+ assert isinstance(bidi_rpc._reopen_throttle, bidi._Throttle)
+
+ def test_done_callbacks_terminate(self):
+ cancellation = mock.Mock()
+ start_rpc = mock.Mock()
+ should_recover = mock.Mock(spec=["__call__"], return_value=True)
+ should_terminate = mock.Mock(spec=["__call__"], return_value=True)
+ bidi_rpc = bidi.ResumableBidiRpc(
+ start_rpc, should_recover, should_terminate=should_terminate
+ )
+ callback = mock.Mock(spec=["__call__"])
+
+ bidi_rpc.add_done_callback(callback)
+ bidi_rpc._on_call_done(cancellation)
+
+ should_terminate.assert_called_once_with(cancellation)
+ should_recover.assert_not_called()
+ callback.assert_called_once_with(cancellation)
+ assert not bidi_rpc.is_active
+
+ def test_done_callbacks_recoverable(self):
+ start_rpc = mock.create_autospec(grpc.StreamStreamMultiCallable, instance=True)
+ should_recover = mock.Mock(spec=["__call__"], return_value=True)
+ bidi_rpc = bidi.ResumableBidiRpc(start_rpc, should_recover)
+ callback = mock.Mock(spec=["__call__"])
+
+ bidi_rpc.add_done_callback(callback)
+ bidi_rpc._on_call_done(mock.sentinel.future)
+
+ callback.assert_not_called()
+ start_rpc.assert_called_once()
+ should_recover.assert_called_once_with(mock.sentinel.future)
+ assert bidi_rpc.is_active
+
+ def test_done_callbacks_non_recoverable(self):
+ start_rpc = mock.create_autospec(grpc.StreamStreamMultiCallable, instance=True)
+ should_recover = mock.Mock(spec=["__call__"], return_value=False)
+ bidi_rpc = bidi.ResumableBidiRpc(start_rpc, should_recover)
+ callback = mock.Mock(spec=["__call__"])
+
+ bidi_rpc.add_done_callback(callback)
+ bidi_rpc._on_call_done(mock.sentinel.future)
+
+ callback.assert_called_once_with(mock.sentinel.future)
+ should_recover.assert_called_once_with(mock.sentinel.future)
+ assert not bidi_rpc.is_active
+
+ def test_send_terminate(self):
+ cancellation = ValueError()
+ call_1 = CallStub([cancellation], active=False)
+ call_2 = CallStub([])
+ start_rpc = mock.create_autospec(
+ grpc.StreamStreamMultiCallable, instance=True, side_effect=[call_1, call_2]
+ )
+ should_recover = mock.Mock(spec=["__call__"], return_value=False)
+ should_terminate = mock.Mock(spec=["__call__"], return_value=True)
+ bidi_rpc = bidi.ResumableBidiRpc(start_rpc, should_recover, should_terminate=should_terminate)
+
+ bidi_rpc.open()
+
+ bidi_rpc.send(mock.sentinel.request)
+
+ assert bidi_rpc.pending_requests == 1
+ assert bidi_rpc._request_queue.get() is None
+
+ should_recover.assert_not_called()
+ should_terminate.assert_called_once_with(cancellation)
+ assert bidi_rpc.call == call_1
+ assert bidi_rpc.is_active is False
+ assert call_1.cancelled is True
+
+ def test_send_recover(self):
+ error = ValueError()
+ call_1 = CallStub([error], active=False)
+ call_2 = CallStub([])
+ start_rpc = mock.create_autospec(
+ grpc.StreamStreamMultiCallable, instance=True, side_effect=[call_1, call_2]
+ )
+ should_recover = mock.Mock(spec=["__call__"], return_value=True)
+ bidi_rpc = bidi.ResumableBidiRpc(start_rpc, should_recover)
+
+ bidi_rpc.open()
+
+ bidi_rpc.send(mock.sentinel.request)
+
+ assert bidi_rpc.pending_requests == 1
+ assert bidi_rpc._request_queue.get() is mock.sentinel.request
+
+ should_recover.assert_called_once_with(error)
+ assert bidi_rpc.call == call_2
+ assert bidi_rpc.is_active is True
+
+ def test_send_failure(self):
+ error = ValueError()
+ call = CallStub([error], active=False)
+ start_rpc = mock.create_autospec(
+ grpc.StreamStreamMultiCallable, instance=True, return_value=call
+ )
+ should_recover = mock.Mock(spec=["__call__"], return_value=False)
+ bidi_rpc = bidi.ResumableBidiRpc(start_rpc, should_recover)
+
+ bidi_rpc.open()
+
+ with pytest.raises(ValueError) as exc_info:
+ bidi_rpc.send(mock.sentinel.request)
+
+ assert exc_info.value == error
+ should_recover.assert_called_once_with(error)
+ assert bidi_rpc.call == call
+ assert bidi_rpc.is_active is False
+ assert call.cancelled is True
+ assert bidi_rpc.pending_requests == 1
+ assert bidi_rpc._request_queue.get() is None
+
+ def test_recv_terminate(self):
+ cancellation = ValueError()
+ call = CallStub([cancellation])
+ start_rpc = mock.create_autospec(
+ grpc.StreamStreamMultiCallable, instance=True, return_value=call
+ )
+ should_recover = mock.Mock(spec=["__call__"], return_value=False)
+ should_terminate = mock.Mock(spec=["__call__"], return_value=True)
+ bidi_rpc = bidi.ResumableBidiRpc(start_rpc, should_recover, should_terminate=should_terminate)
+
+ bidi_rpc.open()
+
+ bidi_rpc.recv()
+
+ should_recover.assert_not_called()
+ should_terminate.assert_called_once_with(cancellation)
+ assert bidi_rpc.call == call
+ assert bidi_rpc.is_active is False
+ assert call.cancelled is True
+
+ def test_recv_recover(self):
+ error = ValueError()
+ call_1 = CallStub([1, error])
+ call_2 = CallStub([2, 3])
+ start_rpc = mock.create_autospec(
+ grpc.StreamStreamMultiCallable, instance=True, side_effect=[call_1, call_2]
+ )
+ should_recover = mock.Mock(spec=["__call__"], return_value=True)
+ bidi_rpc = bidi.ResumableBidiRpc(start_rpc, should_recover)
+
+ bidi_rpc.open()
+
+ values = []
+ for n in range(3):
+ values.append(bidi_rpc.recv())
+
+ assert values == [1, 2, 3]
+ should_recover.assert_called_once_with(error)
+ assert bidi_rpc.call == call_2
+ assert bidi_rpc.is_active is True
+
+ def test_recv_recover_already_recovered(self):
+ call_1 = CallStub([])
+ call_2 = CallStub([])
+ start_rpc = mock.create_autospec(
+ grpc.StreamStreamMultiCallable, instance=True, side_effect=[call_1, call_2]
+ )
+ callback = mock.Mock()
+ callback.return_value = True
+ bidi_rpc = bidi.ResumableBidiRpc(start_rpc, callback)
+
+ bidi_rpc.open()
+
+ bidi_rpc._reopen()
+
+ assert bidi_rpc.call is call_1
+ assert bidi_rpc.is_active is True
+
+ def test_recv_failure(self):
+ error = ValueError()
+ call = CallStub([error])
+ start_rpc = mock.create_autospec(
+ grpc.StreamStreamMultiCallable, instance=True, return_value=call
+ )
+ should_recover = mock.Mock(spec=["__call__"], return_value=False)
+ bidi_rpc = bidi.ResumableBidiRpc(start_rpc, should_recover)
+
+ bidi_rpc.open()
+
+ with pytest.raises(ValueError) as exc_info:
+ bidi_rpc.recv()
+
+ assert exc_info.value == error
+ should_recover.assert_called_once_with(error)
+ assert bidi_rpc.call == call
+ assert bidi_rpc.is_active is False
+ assert call.cancelled is True
+
+ def test_reopen_failure_on_rpc_restart(self):
+ error1 = ValueError("1")
+ error2 = ValueError("2")
+ call = CallStub([error1])
+ # Invoking start RPC a second time will trigger an error.
+ start_rpc = mock.create_autospec(
+ grpc.StreamStreamMultiCallable, instance=True, side_effect=[call, error2]
+ )
+ should_recover = mock.Mock(spec=["__call__"], return_value=True)
+ callback = mock.Mock(spec=["__call__"])
+
+ bidi_rpc = bidi.ResumableBidiRpc(start_rpc, should_recover)
+ bidi_rpc.add_done_callback(callback)
+
+ bidi_rpc.open()
+
+ with pytest.raises(ValueError) as exc_info:
+ bidi_rpc.recv()
+
+ assert exc_info.value == error2
+ should_recover.assert_called_once_with(error1)
+ assert bidi_rpc.call is None
+ assert bidi_rpc.is_active is False
+ callback.assert_called_once_with(error2)
+
+ def test_using_throttle_on_reopen_requests(self):
+ call = CallStub([])
+ start_rpc = mock.create_autospec(
+ grpc.StreamStreamMultiCallable, instance=True, return_value=call
+ )
+ should_recover = mock.Mock(spec=["__call__"], return_value=True)
+ bidi_rpc = bidi.ResumableBidiRpc(
+ start_rpc, should_recover, throttle_reopen=True
+ )
+
+ patcher = mock.patch.object(bidi_rpc._reopen_throttle.__class__, "__enter__")
+ with patcher as mock_enter:
+ bidi_rpc._reopen()
+
+ mock_enter.assert_called_once()
+
+ def test_send_not_open(self):
+ bidi_rpc = bidi.ResumableBidiRpc(None, lambda _: False)
+
+ with pytest.raises(ValueError):
+ bidi_rpc.send(mock.sentinel.request)
+
+ def test_recv_not_open(self):
+ bidi_rpc = bidi.ResumableBidiRpc(None, lambda _: False)
+
+ with pytest.raises(ValueError):
+ bidi_rpc.recv()
+
+ def test_finalize_idempotent(self):
+ error1 = ValueError("1")
+ error2 = ValueError("2")
+ callback = mock.Mock(spec=["__call__"])
+ should_recover = mock.Mock(spec=["__call__"], return_value=False)
+
+ bidi_rpc = bidi.ResumableBidiRpc(mock.sentinel.start_rpc, should_recover)
+
+ bidi_rpc.add_done_callback(callback)
+
+ bidi_rpc._on_call_done(error1)
+ bidi_rpc._on_call_done(error2)
+
+ callback.assert_called_once_with(error1)
+
+
+class TestBackgroundConsumer(object):
+ def test_consume_once_then_exit(self):
+ bidi_rpc = mock.create_autospec(bidi.BidiRpc, instance=True)
+ bidi_rpc.is_active = True
+ bidi_rpc.recv.side_effect = [mock.sentinel.response_1]
+ recved = threading.Event()
+
+ def on_response(response):
+ assert response == mock.sentinel.response_1
+ bidi_rpc.is_active = False
+ recved.set()
+
+ consumer = bidi.BackgroundConsumer(bidi_rpc, on_response)
+
+ consumer.start()
+
+ recved.wait()
+
+ bidi_rpc.recv.assert_called_once()
+ assert bidi_rpc.is_active is False
+
+ consumer.stop()
+
+ bidi_rpc.close.assert_called_once()
+ assert consumer.is_active is False
+
+ def test_pause_resume_and_close(self):
+ # This test is relatively complex. It attempts to start the consumer,
+ # consume one item, pause the consumer, check the state of the world,
+ # then resume the consumer. Doing this in a deterministic fashion
+ # requires a bit more mocking and patching than usual.
+
+ bidi_rpc = mock.create_autospec(bidi.BidiRpc, instance=True)
+ bidi_rpc.is_active = True
+
+ def close_side_effect():
+ bidi_rpc.is_active = False
+
+ bidi_rpc.close.side_effect = close_side_effect
+
+ # These are used to coordinate the two threads to ensure deterministic
+ # execution.
+ should_continue = threading.Event()
+ responses_and_events = {
+ mock.sentinel.response_1: threading.Event(),
+ mock.sentinel.response_2: threading.Event(),
+ }
+ bidi_rpc.recv.side_effect = [mock.sentinel.response_1, mock.sentinel.response_2]
+
+ recved_responses = []
+ consumer = None
+
+ def on_response(response):
+ if response == mock.sentinel.response_1:
+ consumer.pause()
+
+ recved_responses.append(response)
+ responses_and_events[response].set()
+ should_continue.wait()
+
+ consumer = bidi.BackgroundConsumer(bidi_rpc, on_response)
+
+ consumer.start()
+
+ # Wait for the first response to be recved.
+ responses_and_events[mock.sentinel.response_1].wait()
+
+ # Ensure only one item has been recved and that the consumer is paused.
+ assert recved_responses == [mock.sentinel.response_1]
+ assert consumer.is_paused is True
+ assert consumer.is_active is True
+
+ # Unpause the consumer, wait for the second item, then close the
+ # consumer.
+ should_continue.set()
+ consumer.resume()
+
+ responses_and_events[mock.sentinel.response_2].wait()
+
+ assert recved_responses == [mock.sentinel.response_1, mock.sentinel.response_2]
+
+ consumer.stop()
+
+ assert consumer.is_active is False
+
+ def test_wake_on_error(self):
+ should_continue = threading.Event()
+
+ bidi_rpc = mock.create_autospec(bidi.BidiRpc, instance=True)
+ bidi_rpc.is_active = True
+ bidi_rpc.add_done_callback.side_effect = lambda _: should_continue.set()
+
+ consumer = bidi.BackgroundConsumer(bidi_rpc, mock.sentinel.on_response)
+
+ # Start the consumer paused, which should immediately put it into wait
+ # state.
+ consumer.pause()
+ consumer.start()
+
+ # Wait for add_done_callback to be called
+ should_continue.wait()
+ bidi_rpc.add_done_callback.assert_called_once_with(consumer._on_call_done)
+
+ # The consumer should now be blocked on waiting to be unpaused.
+ assert consumer.is_active
+ assert consumer.is_paused
+
+ # Trigger the done callback, it should unpause the consumer and cause
+ # it to exit.
+ bidi_rpc.is_active = False
+ consumer._on_call_done(bidi_rpc)
+
+ # It may take a few cycles for the thread to exit.
+ while consumer.is_active:
+ pass
+
+ def test_consumer_expected_error(self, caplog):
+ caplog.set_level(logging.DEBUG)
+
+ bidi_rpc = mock.create_autospec(bidi.BidiRpc, instance=True)
+ bidi_rpc.is_active = True
+ bidi_rpc.recv.side_effect = exceptions.ServiceUnavailable("Gone away")
+
+ on_response = mock.Mock(spec=["__call__"])
+
+ consumer = bidi.BackgroundConsumer(bidi_rpc, on_response)
+
+ consumer.start()
+
+ # Wait for the consumer's thread to exit.
+ while consumer.is_active:
+ pass
+
+ on_response.assert_not_called()
+ bidi_rpc.recv.assert_called_once()
+ assert "caught error" in caplog.text
+
+ def test_consumer_unexpected_error(self, caplog):
+ caplog.set_level(logging.DEBUG)
+
+ bidi_rpc = mock.create_autospec(bidi.BidiRpc, instance=True)
+ bidi_rpc.is_active = True
+ bidi_rpc.recv.side_effect = ValueError()
+
+ on_response = mock.Mock(spec=["__call__"])
+
+ consumer = bidi.BackgroundConsumer(bidi_rpc, on_response)
+
+ consumer.start()
+
+ # Wait for the consumer's thread to exit.
+ while consumer.is_active:
+ pass
+
+ on_response.assert_not_called()
+ bidi_rpc.recv.assert_called_once()
+ assert "caught unexpected exception" in caplog.text
+
+ def test_double_stop(self, caplog):
+ caplog.set_level(logging.DEBUG)
+ bidi_rpc = mock.create_autospec(bidi.BidiRpc, instance=True)
+ bidi_rpc.is_active = True
+ on_response = mock.Mock(spec=["__call__"])
+
+ def close_side_effect():
+ bidi_rpc.is_active = False
+
+ bidi_rpc.close.side_effect = close_side_effect
+
+ consumer = bidi.BackgroundConsumer(bidi_rpc, on_response)
+
+ consumer.start()
+ assert consumer.is_active is True
+
+ consumer.stop()
+ assert consumer.is_active is False
+
+ # calling stop twice should not result in an error.
+ consumer.stop()
diff --git a/api_core/tests/unit/test_client_info.py b/api_core/tests/unit/test_client_info.py
new file mode 100644
index 000000000000..0eb17c5feb1c
--- /dev/null
+++ b/api_core/tests/unit/test_client_info.py
@@ -0,0 +1,69 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from google.api_core import client_info
+
+
+def test_constructor_defaults():
+ info = client_info.ClientInfo()
+
+ assert info.python_version is not None
+ assert info.grpc_version is not None
+ assert info.api_core_version is not None
+ assert info.gapic_version is None
+ assert info.client_library_version is None
+
+
+def test_constructor_options():
+ info = client_info.ClientInfo(
+ python_version="1",
+ grpc_version="2",
+ api_core_version="3",
+ gapic_version="4",
+ client_library_version="5",
+ user_agent="6"
+ )
+
+ assert info.python_version == "1"
+ assert info.grpc_version == "2"
+ assert info.api_core_version == "3"
+ assert info.gapic_version == "4"
+ assert info.client_library_version == "5"
+ assert info.user_agent == "6"
+
+
+def test_to_user_agent_minimal():
+ info = client_info.ClientInfo(
+ python_version="1", api_core_version="2", grpc_version=None
+ )
+
+ user_agent = info.to_user_agent()
+
+ assert user_agent == "gl-python/1 gax/2"
+
+
+def test_to_user_agent_full():
+ info = client_info.ClientInfo(
+ python_version="1",
+ grpc_version="2",
+ api_core_version="3",
+ gapic_version="4",
+ client_library_version="5",
+ user_agent="app-name/1.0",
+ )
+
+ user_agent = info.to_user_agent()
+
+ assert user_agent == "app-name/1.0 gl-python/1 grpc/2 gax/3 gapic/4 gccl/5"
diff --git a/api_core/tests/unit/test_client_options.py b/api_core/tests/unit/test_client_options.py
new file mode 100644
index 000000000000..14cae9f24bfb
--- /dev/null
+++ b/api_core/tests/unit/test_client_options.py
@@ -0,0 +1,36 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import pytest
+
+from google.api_core import client_options
+
+
+def test_constructor():
+ options = client_options.ClientOptions(api_endpoint="foo.googleapis.com")
+
+ assert options.api_endpoint == "foo.googleapis.com"
+
+
+def test_from_dict():
+ options = client_options.from_dict({"api_endpoint": "foo.googleapis.com"})
+
+ assert options.api_endpoint == "foo.googleapis.com"
+
+
+def test_from_dict_bad_argument():
+ with pytest.raises(ValueError):
+ client_options.from_dict(
+ {"api_endpoint": "foo.googleapis.com", "bad_arg": "1234"}
+ )
diff --git a/api_core/tests/unit/test_datetime_helpers.py b/api_core/tests/unit/test_datetime_helpers.py
new file mode 100644
index 000000000000..4d138c88a80c
--- /dev/null
+++ b/api_core/tests/unit/test_datetime_helpers.py
@@ -0,0 +1,351 @@
+# Copyright 2017, Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import calendar
+import datetime
+
+import pytest
+import pytz
+
+from google.api_core import datetime_helpers
+from google.protobuf import timestamp_pb2
+
+
+ONE_MINUTE_IN_MICROSECONDS = 60 * 1e6
+
+
+def test_utcnow():
+ result = datetime_helpers.utcnow()
+ assert isinstance(result, datetime.datetime)
+
+
+def test_to_milliseconds():
+ dt = datetime.datetime(1970, 1, 1, 0, 0, 1, tzinfo=pytz.utc)
+ assert datetime_helpers.to_milliseconds(dt) == 1000
+
+
+def test_to_microseconds():
+ microseconds = 314159
+ dt = datetime.datetime(1970, 1, 1, 0, 0, 0, microsecond=microseconds)
+ assert datetime_helpers.to_microseconds(dt) == microseconds
+
+
+def test_to_microseconds_non_utc():
+ zone = pytz.FixedOffset(-1)
+ dt = datetime.datetime(1970, 1, 1, 0, 0, 0, tzinfo=zone)
+ assert datetime_helpers.to_microseconds(dt) == ONE_MINUTE_IN_MICROSECONDS
+
+
+def test_to_microseconds_naive():
+ microseconds = 314159
+ dt = datetime.datetime(1970, 1, 1, 0, 0, 0, microsecond=microseconds, tzinfo=None)
+ assert datetime_helpers.to_microseconds(dt) == microseconds
+
+
+def test_from_microseconds():
+ five_mins_from_epoch_in_microseconds = 5 * ONE_MINUTE_IN_MICROSECONDS
+ five_mins_from_epoch_datetime = datetime.datetime(
+ 1970, 1, 1, 0, 5, 0, tzinfo=pytz.utc
+ )
+
+ result = datetime_helpers.from_microseconds(five_mins_from_epoch_in_microseconds)
+
+ assert result == five_mins_from_epoch_datetime
+
+
+def test_from_iso8601_date():
+ today = datetime.date.today()
+ iso_8601_today = today.strftime("%Y-%m-%d")
+
+ assert datetime_helpers.from_iso8601_date(iso_8601_today) == today
+
+
+def test_from_iso8601_time():
+ assert datetime_helpers.from_iso8601_time("12:09:42") == datetime.time(12, 9, 42)
+
+
+def test_from_rfc3339():
+ value = "2009-12-17T12:44:32.123456Z"
+ assert datetime_helpers.from_rfc3339(value) == datetime.datetime(
+ 2009, 12, 17, 12, 44, 32, 123456, pytz.utc
+ )
+
+
+def test_from_rfc3339_with_bad_tz():
+ value = "2009-12-17T12:44:32.123456BAD"
+
+ with pytest.raises(ValueError):
+ datetime_helpers.from_rfc3339(value)
+
+
+def test_from_rfc3339_with_nanos():
+ value = "2009-12-17T12:44:32.123456789Z"
+
+ with pytest.raises(ValueError):
+ datetime_helpers.from_rfc3339(value)
+
+
+def test_from_rfc3339_nanos_without_nanos():
+ value = "2009-12-17T12:44:32Z"
+ assert datetime_helpers.from_rfc3339_nanos(value) == datetime.datetime(
+ 2009, 12, 17, 12, 44, 32, 0, pytz.utc
+ )
+
+
+def test_from_rfc3339_nanos_with_bad_tz():
+ value = "2009-12-17T12:44:32.123456789BAD"
+
+ with pytest.raises(ValueError):
+ datetime_helpers.from_rfc3339_nanos(value)
+
+
+@pytest.mark.parametrize(
+ "truncated, micros",
+ [
+ ("12345678", 123456),
+ ("1234567", 123456),
+ ("123456", 123456),
+ ("12345", 123450),
+ ("1234", 123400),
+ ("123", 123000),
+ ("12", 120000),
+ ("1", 100000),
+ ],
+)
+def test_from_rfc3339_nanos_with_truncated_nanos(truncated, micros):
+ value = "2009-12-17T12:44:32.{}Z".format(truncated)
+ assert datetime_helpers.from_rfc3339_nanos(value) == datetime.datetime(
+ 2009, 12, 17, 12, 44, 32, micros, pytz.utc
+ )
+
+
+def test_to_rfc3339():
+ value = datetime.datetime(2016, 4, 5, 13, 30, 0)
+ expected = "2016-04-05T13:30:00.000000Z"
+ assert datetime_helpers.to_rfc3339(value) == expected
+
+
+def test_to_rfc3339_with_utc():
+ value = datetime.datetime(2016, 4, 5, 13, 30, 0, tzinfo=pytz.utc)
+ expected = "2016-04-05T13:30:00.000000Z"
+ assert datetime_helpers.to_rfc3339(value, ignore_zone=False) == expected
+
+
+def test_to_rfc3339_with_non_utc():
+ zone = pytz.FixedOffset(-60)
+ value = datetime.datetime(2016, 4, 5, 13, 30, 0, tzinfo=zone)
+ expected = "2016-04-05T14:30:00.000000Z"
+ assert datetime_helpers.to_rfc3339(value, ignore_zone=False) == expected
+
+
+def test_to_rfc3339_with_non_utc_ignore_zone():
+ zone = pytz.FixedOffset(-60)
+ value = datetime.datetime(2016, 4, 5, 13, 30, 0, tzinfo=zone)
+ expected = "2016-04-05T13:30:00.000000Z"
+ assert datetime_helpers.to_rfc3339(value, ignore_zone=True) == expected
+
+
+class Test_DateTimeWithNanos(object):
+
+ @staticmethod
+ def test_ctor_wo_nanos():
+ stamp = datetime_helpers.DatetimeWithNanoseconds(2016, 12, 20, 21, 13, 47, 123456)
+ assert stamp.year == 2016
+ assert stamp.month == 12
+ assert stamp.day == 20
+ assert stamp.hour == 21
+ assert stamp.minute == 13
+ assert stamp.second == 47
+ assert stamp.microsecond == 123456
+ assert stamp.nanosecond == 0
+
+ @staticmethod
+ def test_ctor_w_nanos():
+ stamp = datetime_helpers.DatetimeWithNanoseconds(
+ 2016, 12, 20, 21, 13, 47, nanosecond=123456789
+ )
+ assert stamp.year == 2016
+ assert stamp.month == 12
+ assert stamp.day == 20
+ assert stamp.hour == 21
+ assert stamp.minute == 13
+ assert stamp.second == 47
+ assert stamp.microsecond == 123456
+ assert stamp.nanosecond == 123456789
+
+ @staticmethod
+ def test_ctor_w_micros_positional_and_nanos():
+ with pytest.raises(TypeError):
+ datetime_helpers.DatetimeWithNanoseconds(
+ 2016, 12, 20, 21, 13, 47, 123456, nanosecond=123456789
+ )
+
+ @staticmethod
+ def test_ctor_w_micros_keyword_and_nanos():
+ with pytest.raises(TypeError):
+ datetime_helpers.DatetimeWithNanoseconds(
+ 2016, 12, 20, 21, 13, 47, microsecond=123456, nanosecond=123456789
+ )
+
+ @staticmethod
+ def test_rfc3339_wo_nanos():
+ stamp = datetime_helpers.DatetimeWithNanoseconds(2016, 12, 20, 21, 13, 47, 123456)
+ assert stamp.rfc3339() == "2016-12-20T21:13:47.123456Z"
+
+ @staticmethod
+ def test_rfc3339_wo_nanos_w_leading_zero():
+ stamp = datetime_helpers.DatetimeWithNanoseconds(2016, 12, 20, 21, 13, 47, 1234)
+ assert stamp.rfc3339() == "2016-12-20T21:13:47.001234Z"
+
+ @staticmethod
+ def test_rfc3339_w_nanos():
+ stamp = datetime_helpers.DatetimeWithNanoseconds(
+ 2016, 12, 20, 21, 13, 47, nanosecond=123456789
+ )
+ assert stamp.rfc3339() == "2016-12-20T21:13:47.123456789Z"
+
+ @staticmethod
+ def test_rfc3339_w_nanos_w_leading_zero():
+ stamp = datetime_helpers.DatetimeWithNanoseconds(
+ 2016, 12, 20, 21, 13, 47, nanosecond=1234567
+ )
+ assert stamp.rfc3339() == "2016-12-20T21:13:47.001234567Z"
+
+ @staticmethod
+ def test_rfc3339_w_nanos_no_trailing_zeroes():
+ stamp = datetime_helpers.DatetimeWithNanoseconds(
+ 2016, 12, 20, 21, 13, 47, nanosecond=100000000
+ )
+ assert stamp.rfc3339() == "2016-12-20T21:13:47.1Z"
+
+ @staticmethod
+ def test_rfc3339_w_nanos_w_leading_zero_and_no_trailing_zeros():
+ stamp = datetime_helpers.DatetimeWithNanoseconds(
+ 2016, 12, 20, 21, 13, 47, nanosecond=1234500
+ )
+ assert stamp.rfc3339() == "2016-12-20T21:13:47.0012345Z"
+
+ @staticmethod
+ def test_from_rfc3339_w_invalid():
+ stamp = "2016-12-20T21:13:47"
+ with pytest.raises(ValueError):
+ datetime_helpers.DatetimeWithNanoseconds.from_rfc3339(stamp)
+
+ @staticmethod
+ def test_from_rfc3339_wo_fraction():
+ timestamp = "2016-12-20T21:13:47Z"
+ expected = datetime_helpers.DatetimeWithNanoseconds(
+ 2016, 12, 20, 21, 13, 47, tzinfo=pytz.UTC
+ )
+ stamp = datetime_helpers.DatetimeWithNanoseconds.from_rfc3339(timestamp)
+ assert stamp == expected
+
+ @staticmethod
+ def test_from_rfc3339_w_partial_precision():
+ timestamp = "2016-12-20T21:13:47.1Z"
+ expected = datetime_helpers.DatetimeWithNanoseconds(
+ 2016, 12, 20, 21, 13, 47, microsecond=100000, tzinfo=pytz.UTC
+ )
+ stamp = datetime_helpers.DatetimeWithNanoseconds.from_rfc3339(timestamp)
+ assert stamp == expected
+
+ @staticmethod
+ def test_from_rfc3339_w_full_precision():
+ timestamp = "2016-12-20T21:13:47.123456789Z"
+ expected = datetime_helpers.DatetimeWithNanoseconds(
+ 2016, 12, 20, 21, 13, 47, nanosecond=123456789, tzinfo=pytz.UTC
+ )
+ stamp = datetime_helpers.DatetimeWithNanoseconds.from_rfc3339(timestamp)
+ assert stamp == expected
+
+ @staticmethod
+ @pytest.mark.parametrize(
+ "fractional, nanos",
+ [
+ ("12345678", 123456780),
+ ("1234567", 123456700),
+ ("123456", 123456000),
+ ("12345", 123450000),
+ ("1234", 123400000),
+ ("123", 123000000),
+ ("12", 120000000),
+ ("1", 100000000),
+ ],
+ )
+ def test_from_rfc3339_test_nanoseconds(fractional, nanos):
+ value = "2009-12-17T12:44:32.{}Z".format(fractional)
+ assert datetime_helpers.DatetimeWithNanoseconds.from_rfc3339(value).nanosecond == nanos
+
+ @staticmethod
+ def test_timestamp_pb_wo_nanos_naive():
+ stamp = datetime_helpers.DatetimeWithNanoseconds(
+ 2016, 12, 20, 21, 13, 47, 123456)
+ delta = stamp.replace(tzinfo=pytz.UTC) - datetime_helpers._UTC_EPOCH
+ seconds = int(delta.total_seconds())
+ nanos = 123456000
+ timestamp = timestamp_pb2.Timestamp(seconds=seconds, nanos=nanos)
+ assert stamp.timestamp_pb() == timestamp
+
+ @staticmethod
+ def test_timestamp_pb_w_nanos():
+ stamp = datetime_helpers.DatetimeWithNanoseconds(
+ 2016, 12, 20, 21, 13, 47, nanosecond=123456789, tzinfo=pytz.UTC
+ )
+ delta = stamp - datetime_helpers._UTC_EPOCH
+ timestamp = timestamp_pb2.Timestamp(
+ seconds=int(delta.total_seconds()), nanos=123456789)
+ assert stamp.timestamp_pb() == timestamp
+
+ @staticmethod
+ def test_from_timestamp_pb_wo_nanos():
+ when = datetime.datetime(2016, 12, 20, 21, 13, 47, 123456, tzinfo=pytz.UTC)
+ delta = when - datetime_helpers._UTC_EPOCH
+ seconds = int(delta.total_seconds())
+ timestamp = timestamp_pb2.Timestamp(seconds=seconds)
+
+ stamp = datetime_helpers.DatetimeWithNanoseconds.from_timestamp_pb(
+ timestamp)
+
+ assert _to_seconds(when) == _to_seconds(stamp)
+ assert stamp.microsecond == 0
+ assert stamp.nanosecond == 0
+ assert stamp.tzinfo == pytz.UTC
+
+ @staticmethod
+ def test_from_timestamp_pb_w_nanos():
+ when = datetime.datetime(2016, 12, 20, 21, 13, 47, 123456, tzinfo=pytz.UTC)
+ delta = when - datetime_helpers._UTC_EPOCH
+ seconds = int(delta.total_seconds())
+ timestamp = timestamp_pb2.Timestamp(seconds=seconds, nanos=123456789)
+
+ stamp = datetime_helpers.DatetimeWithNanoseconds.from_timestamp_pb(
+ timestamp)
+
+ assert _to_seconds(when) == _to_seconds(stamp)
+ assert stamp.microsecond == 123456
+ assert stamp.nanosecond == 123456789
+ assert stamp.tzinfo == pytz.UTC
+
+
+def _to_seconds(value):
+ """Convert a datetime to seconds since the unix epoch.
+
+ Args:
+ value (datetime.datetime): The datetime to covert.
+
+ Returns:
+ int: Microseconds since the unix epoch.
+ """
+ assert value.tzinfo is pytz.UTC
+ return calendar.timegm(value.timetuple())
diff --git a/api_core/tests/unit/test_exceptions.py b/api_core/tests/unit/test_exceptions.py
new file mode 100644
index 000000000000..040ac8ac1d7c
--- /dev/null
+++ b/api_core/tests/unit/test_exceptions.py
@@ -0,0 +1,214 @@
+# Copyright 2014 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import json
+
+import grpc
+import mock
+import requests
+from six.moves import http_client
+
+from google.api_core import exceptions
+
+
+def test_create_google_cloud_error():
+ exception = exceptions.GoogleAPICallError("Testing")
+ exception.code = 600
+ assert str(exception) == "600 Testing"
+ assert exception.message == "Testing"
+ assert exception.errors == []
+ assert exception.response is None
+
+
+def test_create_google_cloud_error_with_args():
+ error = {
+ "domain": "global",
+ "location": "test",
+ "locationType": "testing",
+ "message": "Testing",
+ "reason": "test",
+ }
+ response = mock.sentinel.response
+ exception = exceptions.GoogleAPICallError("Testing", [error], response=response)
+ exception.code = 600
+ assert str(exception) == "600 Testing"
+ assert exception.message == "Testing"
+ assert exception.errors == [error]
+ assert exception.response == response
+
+
+def test_from_http_status():
+ message = "message"
+ exception = exceptions.from_http_status(http_client.NOT_FOUND, message)
+ assert exception.code == http_client.NOT_FOUND
+ assert exception.message == message
+ assert exception.errors == []
+
+
+def test_from_http_status_with_errors_and_response():
+ message = "message"
+ errors = ["1", "2"]
+ response = mock.sentinel.response
+ exception = exceptions.from_http_status(
+ http_client.NOT_FOUND, message, errors=errors, response=response
+ )
+
+ assert isinstance(exception, exceptions.NotFound)
+ assert exception.code == http_client.NOT_FOUND
+ assert exception.message == message
+ assert exception.errors == errors
+ assert exception.response == response
+
+
+def test_from_http_status_unknown_code():
+ message = "message"
+ status_code = 156
+ exception = exceptions.from_http_status(status_code, message)
+ assert exception.code == status_code
+ assert exception.message == message
+
+
+def make_response(content):
+ response = requests.Response()
+ response._content = content
+ response.status_code = http_client.NOT_FOUND
+ response.request = requests.Request(
+ method="POST", url="https://example.com"
+ ).prepare()
+ return response
+
+
+def test_from_http_response_no_content():
+ response = make_response(None)
+
+ exception = exceptions.from_http_response(response)
+
+ assert isinstance(exception, exceptions.NotFound)
+ assert exception.code == http_client.NOT_FOUND
+ assert exception.message == "POST https://example.com/: unknown error"
+ assert exception.response == response
+
+
+def test_from_http_response_text_content():
+ response = make_response(b"message")
+
+ exception = exceptions.from_http_response(response)
+
+ assert isinstance(exception, exceptions.NotFound)
+ assert exception.code == http_client.NOT_FOUND
+ assert exception.message == "POST https://example.com/: message"
+
+
+def test_from_http_response_json_content():
+ response = make_response(
+ json.dumps({"error": {"message": "json message", "errors": ["1", "2"]}}).encode(
+ "utf-8"
+ )
+ )
+
+ exception = exceptions.from_http_response(response)
+
+ assert isinstance(exception, exceptions.NotFound)
+ assert exception.code == http_client.NOT_FOUND
+ assert exception.message == "POST https://example.com/: json message"
+ assert exception.errors == ["1", "2"]
+
+
+def test_from_http_response_bad_json_content():
+ response = make_response(json.dumps({"meep": "moop"}).encode("utf-8"))
+
+ exception = exceptions.from_http_response(response)
+
+ assert isinstance(exception, exceptions.NotFound)
+ assert exception.code == http_client.NOT_FOUND
+ assert exception.message == "POST https://example.com/: unknown error"
+
+
+def test_from_http_response_json_unicode_content():
+ response = make_response(
+ json.dumps(
+ {"error": {"message": u"\u2019 message", "errors": ["1", "2"]}}
+ ).encode("utf-8")
+ )
+
+ exception = exceptions.from_http_response(response)
+
+ assert isinstance(exception, exceptions.NotFound)
+ assert exception.code == http_client.NOT_FOUND
+ assert exception.message == u"POST https://example.com/: \u2019 message"
+ assert exception.errors == ["1", "2"]
+
+
+def test_from_grpc_status():
+ message = "message"
+ exception = exceptions.from_grpc_status(grpc.StatusCode.OUT_OF_RANGE, message)
+ assert isinstance(exception, exceptions.BadRequest)
+ assert isinstance(exception, exceptions.OutOfRange)
+ assert exception.code == http_client.BAD_REQUEST
+ assert exception.grpc_status_code == grpc.StatusCode.OUT_OF_RANGE
+ assert exception.message == message
+ assert exception.errors == []
+
+
+def test_from_grpc_status_with_errors_and_response():
+ message = "message"
+ response = mock.sentinel.response
+ errors = ["1", "2"]
+ exception = exceptions.from_grpc_status(
+ grpc.StatusCode.OUT_OF_RANGE, message, errors=errors, response=response
+ )
+
+ assert isinstance(exception, exceptions.OutOfRange)
+ assert exception.message == message
+ assert exception.errors == errors
+ assert exception.response == response
+
+
+def test_from_grpc_status_unknown_code():
+ message = "message"
+ exception = exceptions.from_grpc_status(grpc.StatusCode.OK, message)
+ assert exception.grpc_status_code == grpc.StatusCode.OK
+ assert exception.message == message
+
+
+def test_from_grpc_error():
+ message = "message"
+ error = mock.create_autospec(grpc.Call, instance=True)
+ error.code.return_value = grpc.StatusCode.INVALID_ARGUMENT
+ error.details.return_value = message
+
+ exception = exceptions.from_grpc_error(error)
+
+ assert isinstance(exception, exceptions.BadRequest)
+ assert isinstance(exception, exceptions.InvalidArgument)
+ assert exception.code == http_client.BAD_REQUEST
+ assert exception.grpc_status_code == grpc.StatusCode.INVALID_ARGUMENT
+ assert exception.message == message
+ assert exception.errors == [error]
+ assert exception.response == error
+
+
+def test_from_grpc_error_non_call():
+ message = "message"
+ error = mock.create_autospec(grpc.RpcError, instance=True)
+ error.__str__.return_value = message
+
+ exception = exceptions.from_grpc_error(error)
+
+ assert isinstance(exception, exceptions.GoogleAPICallError)
+ assert exception.code is None
+ assert exception.grpc_status_code is None
+ assert exception.message == message
+ assert exception.errors == [error]
+ assert exception.response == error
diff --git a/api_core/tests/unit/test_general_helpers.py b/api_core/tests/unit/test_general_helpers.py
new file mode 100644
index 000000000000..027d4892ba09
--- /dev/null
+++ b/api_core/tests/unit/test_general_helpers.py
@@ -0,0 +1,41 @@
+# Copyright 2017, Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import functools
+
+from google.api_core import general_helpers
+
+
+def test_wraps_normal_func():
+ def func():
+ return 42
+
+ @general_helpers.wraps(func)
+ def replacement():
+ return func()
+
+ assert replacement() == 42
+
+
+def test_wraps_partial():
+ def func():
+ return 42
+
+ partial = functools.partial(func)
+
+ @general_helpers.wraps(partial)
+ def replacement():
+ return func()
+
+ assert replacement() == 42
diff --git a/api_core/tests/unit/test_grpc_helpers.py b/api_core/tests/unit/test_grpc_helpers.py
new file mode 100644
index 000000000000..c37c3eedbe83
--- /dev/null
+++ b/api_core/tests/unit/test_grpc_helpers.py
@@ -0,0 +1,460 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import grpc
+import mock
+import pytest
+
+from google.api_core import exceptions
+from google.api_core import grpc_helpers
+import google.auth.credentials
+from google.longrunning import operations_pb2
+
+
+def test__patch_callable_name():
+ callable = mock.Mock(spec=["__class__"])
+ callable.__class__ = mock.Mock(spec=["__name__"])
+ callable.__class__.__name__ = "TestCallable"
+
+ grpc_helpers._patch_callable_name(callable)
+
+ assert callable.__name__ == "TestCallable"
+
+
+def test__patch_callable_name_no_op():
+ callable = mock.Mock(spec=["__name__"])
+ callable.__name__ = "test_callable"
+
+ grpc_helpers._patch_callable_name(callable)
+
+ assert callable.__name__ == "test_callable"
+
+
+class RpcErrorImpl(grpc.RpcError, grpc.Call):
+ def __init__(self, code):
+ super(RpcErrorImpl, self).__init__()
+ self._code = code
+
+ def code(self):
+ return self._code
+
+ def details(self):
+ return None
+
+
+def test_wrap_unary_errors():
+ grpc_error = RpcErrorImpl(grpc.StatusCode.INVALID_ARGUMENT)
+ callable_ = mock.Mock(spec=["__call__"], side_effect=grpc_error)
+
+ wrapped_callable = grpc_helpers._wrap_unary_errors(callable_)
+
+ with pytest.raises(exceptions.InvalidArgument) as exc_info:
+ wrapped_callable(1, 2, three="four")
+
+ callable_.assert_called_once_with(1, 2, three="four")
+ assert exc_info.value.response == grpc_error
+
+
+def test_wrap_stream_okay():
+ expected_responses = [1, 2, 3]
+ callable_ = mock.Mock(spec=["__call__"], return_value=iter(expected_responses))
+
+ wrapped_callable = grpc_helpers._wrap_stream_errors(callable_)
+
+ got_iterator = wrapped_callable(1, 2, three="four")
+
+ responses = list(got_iterator)
+
+ callable_.assert_called_once_with(1, 2, three="four")
+ assert responses == expected_responses
+
+
+def test_wrap_stream_iterable_iterface():
+ response_iter = mock.create_autospec(grpc.Call, instance=True)
+ callable_ = mock.Mock(spec=["__call__"], return_value=response_iter)
+
+ wrapped_callable = grpc_helpers._wrap_stream_errors(callable_)
+
+ got_iterator = wrapped_callable()
+
+ callable_.assert_called_once_with()
+
+ # Check each aliased method in the grpc.Call interface
+ got_iterator.add_callback(mock.sentinel.callback)
+ response_iter.add_callback.assert_called_once_with(mock.sentinel.callback)
+
+ got_iterator.cancel()
+ response_iter.cancel.assert_called_once_with()
+
+ got_iterator.code()
+ response_iter.code.assert_called_once_with()
+
+ got_iterator.details()
+ response_iter.details.assert_called_once_with()
+
+ got_iterator.initial_metadata()
+ response_iter.initial_metadata.assert_called_once_with()
+
+ got_iterator.is_active()
+ response_iter.is_active.assert_called_once_with()
+
+ got_iterator.time_remaining()
+ response_iter.time_remaining.assert_called_once_with()
+
+ got_iterator.trailing_metadata()
+ response_iter.trailing_metadata.assert_called_once_with()
+
+
+def test_wrap_stream_errors_invocation():
+ grpc_error = RpcErrorImpl(grpc.StatusCode.INVALID_ARGUMENT)
+ callable_ = mock.Mock(spec=["__call__"], side_effect=grpc_error)
+
+ wrapped_callable = grpc_helpers._wrap_stream_errors(callable_)
+
+ with pytest.raises(exceptions.InvalidArgument) as exc_info:
+ wrapped_callable(1, 2, three="four")
+
+ callable_.assert_called_once_with(1, 2, three="four")
+ assert exc_info.value.response == grpc_error
+
+
+class RpcResponseIteratorImpl(object):
+ def __init__(self, exception):
+ self._exception = exception
+
+ def next(self):
+ raise self._exception
+
+ __next__ = next
+
+
+def test_wrap_stream_errors_iterator():
+ grpc_error = RpcErrorImpl(grpc.StatusCode.UNAVAILABLE)
+ response_iter = RpcResponseIteratorImpl(grpc_error)
+ callable_ = mock.Mock(spec=["__call__"], return_value=response_iter)
+
+ wrapped_callable = grpc_helpers._wrap_stream_errors(callable_)
+
+ got_iterator = wrapped_callable(1, 2, three="four")
+
+ with pytest.raises(exceptions.ServiceUnavailable) as exc_info:
+ next(got_iterator)
+
+ callable_.assert_called_once_with(1, 2, three="four")
+ assert exc_info.value.response == grpc_error
+
+
+@mock.patch("google.api_core.grpc_helpers._wrap_unary_errors")
+def test_wrap_errors_non_streaming(wrap_unary_errors):
+ callable_ = mock.create_autospec(grpc.UnaryUnaryMultiCallable)
+
+ result = grpc_helpers.wrap_errors(callable_)
+
+ assert result == wrap_unary_errors.return_value
+ wrap_unary_errors.assert_called_once_with(callable_)
+
+
+@mock.patch("google.api_core.grpc_helpers._wrap_stream_errors")
+def test_wrap_errors_streaming(wrap_stream_errors):
+ callable_ = mock.create_autospec(grpc.UnaryStreamMultiCallable)
+
+ result = grpc_helpers.wrap_errors(callable_)
+
+ assert result == wrap_stream_errors.return_value
+ wrap_stream_errors.assert_called_once_with(callable_)
+
+
+@mock.patch("grpc.composite_channel_credentials")
+@mock.patch(
+ "google.auth.default",
+ return_value=(mock.sentinel.credentials, mock.sentinel.projet),
+)
+@mock.patch("grpc.secure_channel")
+def test_create_channel_implicit(grpc_secure_channel, default, composite_creds_call):
+ target = "example.com:443"
+ composite_creds = composite_creds_call.return_value
+
+ channel = grpc_helpers.create_channel(target)
+
+ assert channel is grpc_secure_channel.return_value
+ default.assert_called_once_with(scopes=None)
+ if grpc_helpers.HAS_GRPC_GCP:
+ grpc_secure_channel.assert_called_once_with(target, composite_creds, None)
+ else:
+ grpc_secure_channel.assert_called_once_with(target, composite_creds)
+
+
+@mock.patch("grpc.composite_channel_credentials")
+@mock.patch(
+ "google.auth.default",
+ return_value=(mock.sentinel.credentials, mock.sentinel.projet),
+)
+@mock.patch("grpc.secure_channel")
+def test_create_channel_implicit_with_ssl_creds(
+ grpc_secure_channel, default, composite_creds_call
+):
+ target = "example.com:443"
+
+ ssl_creds = grpc.ssl_channel_credentials()
+
+ grpc_helpers.create_channel(target, ssl_credentials=ssl_creds)
+
+ default.assert_called_once_with(scopes=None)
+ composite_creds_call.assert_called_once_with(ssl_creds, mock.ANY)
+ composite_creds = composite_creds_call.return_value
+ if grpc_helpers.HAS_GRPC_GCP:
+ grpc_secure_channel.assert_called_once_with(target, composite_creds, None)
+ else:
+ grpc_secure_channel.assert_called_once_with(target, composite_creds)
+
+
+@mock.patch("grpc.composite_channel_credentials")
+@mock.patch(
+ "google.auth.default",
+ return_value=(mock.sentinel.credentials, mock.sentinel.projet),
+)
+@mock.patch("grpc.secure_channel")
+def test_create_channel_implicit_with_scopes(
+ grpc_secure_channel, default, composite_creds_call
+):
+ target = "example.com:443"
+ composite_creds = composite_creds_call.return_value
+
+ channel = grpc_helpers.create_channel(target, scopes=["one", "two"])
+
+ assert channel is grpc_secure_channel.return_value
+ default.assert_called_once_with(scopes=["one", "two"])
+ if grpc_helpers.HAS_GRPC_GCP:
+ grpc_secure_channel.assert_called_once_with(target, composite_creds, None)
+ else:
+ grpc_secure_channel.assert_called_once_with(target, composite_creds)
+
+
+@mock.patch("grpc.composite_channel_credentials")
+@mock.patch("google.auth.credentials.with_scopes_if_required")
+@mock.patch("grpc.secure_channel")
+def test_create_channel_explicit(grpc_secure_channel, auth_creds, composite_creds_call):
+ target = "example.com:443"
+ composite_creds = composite_creds_call.return_value
+
+ channel = grpc_helpers.create_channel(target, credentials=mock.sentinel.credentials)
+
+ auth_creds.assert_called_once_with(mock.sentinel.credentials, None)
+ assert channel is grpc_secure_channel.return_value
+ if grpc_helpers.HAS_GRPC_GCP:
+ grpc_secure_channel.assert_called_once_with(target, composite_creds, None)
+ else:
+ grpc_secure_channel.assert_called_once_with(target, composite_creds)
+
+
+@mock.patch("grpc.composite_channel_credentials")
+@mock.patch("grpc.secure_channel")
+def test_create_channel_explicit_scoped(grpc_secure_channel, composite_creds_call):
+ target = "example.com:443"
+ scopes = ["1", "2"]
+ composite_creds = composite_creds_call.return_value
+
+ credentials = mock.create_autospec(google.auth.credentials.Scoped, instance=True)
+ credentials.requires_scopes = True
+
+ channel = grpc_helpers.create_channel(
+ target, credentials=credentials, scopes=scopes
+ )
+
+ credentials.with_scopes.assert_called_once_with(scopes)
+ assert channel is grpc_secure_channel.return_value
+ if grpc_helpers.HAS_GRPC_GCP:
+ grpc_secure_channel.assert_called_once_with(target, composite_creds, None)
+ else:
+ grpc_secure_channel.assert_called_once_with(target, composite_creds)
+
+
+@pytest.mark.skipif(
+ not grpc_helpers.HAS_GRPC_GCP, reason="grpc_gcp module not available"
+)
+@mock.patch("grpc_gcp.secure_channel")
+def test_create_channel_with_grpc_gcp(grpc_gcp_secure_channel):
+ target = "example.com:443"
+ scopes = ["test_scope"]
+
+ credentials = mock.create_autospec(google.auth.credentials.Scoped, instance=True)
+ credentials.requires_scopes = True
+
+ grpc_helpers.create_channel(target, credentials=credentials, scopes=scopes)
+ grpc_gcp_secure_channel.assert_called()
+ credentials.with_scopes.assert_called_once_with(scopes)
+
+
+@pytest.mark.skipif(grpc_helpers.HAS_GRPC_GCP, reason="grpc_gcp module not available")
+@mock.patch("grpc.secure_channel")
+def test_create_channel_without_grpc_gcp(grpc_secure_channel):
+ target = "example.com:443"
+ scopes = ["test_scope"]
+
+ credentials = mock.create_autospec(google.auth.credentials.Scoped, instance=True)
+ credentials.requires_scopes = True
+
+ grpc_helpers.create_channel(target, credentials=credentials, scopes=scopes)
+ grpc_secure_channel.assert_called()
+ credentials.with_scopes.assert_called_once_with(scopes)
+
+
+class TestChannelStub(object):
+ def test_single_response(self):
+ channel = grpc_helpers.ChannelStub()
+ stub = operations_pb2.OperationsStub(channel)
+ expected_request = operations_pb2.GetOperationRequest(name="meep")
+ expected_response = operations_pb2.Operation(name="moop")
+
+ channel.GetOperation.response = expected_response
+
+ response = stub.GetOperation(expected_request)
+
+ assert response == expected_response
+ assert channel.requests == [("GetOperation", expected_request)]
+ assert channel.GetOperation.requests == [expected_request]
+
+ def test_no_response(self):
+ channel = grpc_helpers.ChannelStub()
+ stub = operations_pb2.OperationsStub(channel)
+ expected_request = operations_pb2.GetOperationRequest(name="meep")
+
+ with pytest.raises(ValueError) as exc_info:
+ stub.GetOperation(expected_request)
+
+ assert exc_info.match("GetOperation")
+
+ def test_missing_method(self):
+ channel = grpc_helpers.ChannelStub()
+
+ with pytest.raises(AttributeError):
+ channel.DoesNotExist.response
+
+ def test_exception_response(self):
+ channel = grpc_helpers.ChannelStub()
+ stub = operations_pb2.OperationsStub(channel)
+ expected_request = operations_pb2.GetOperationRequest(name="meep")
+
+ channel.GetOperation.response = RuntimeError()
+
+ with pytest.raises(RuntimeError):
+ stub.GetOperation(expected_request)
+
+ def test_callable_response(self):
+ channel = grpc_helpers.ChannelStub()
+ stub = operations_pb2.OperationsStub(channel)
+ expected_request = operations_pb2.GetOperationRequest(name="meep")
+ expected_response = operations_pb2.Operation(name="moop")
+
+ on_get_operation = mock.Mock(spec=("__call__",), return_value=expected_response)
+
+ channel.GetOperation.response = on_get_operation
+
+ response = stub.GetOperation(expected_request)
+
+ assert response == expected_response
+ on_get_operation.assert_called_once_with(expected_request)
+
+ def test_multiple_responses(self):
+ channel = grpc_helpers.ChannelStub()
+ stub = operations_pb2.OperationsStub(channel)
+ expected_request = operations_pb2.GetOperationRequest(name="meep")
+ expected_responses = [
+ operations_pb2.Operation(name="foo"),
+ operations_pb2.Operation(name="bar"),
+ operations_pb2.Operation(name="baz"),
+ ]
+
+ channel.GetOperation.responses = iter(expected_responses)
+
+ response1 = stub.GetOperation(expected_request)
+ response2 = stub.GetOperation(expected_request)
+ response3 = stub.GetOperation(expected_request)
+
+ assert response1 == expected_responses[0]
+ assert response2 == expected_responses[1]
+ assert response3 == expected_responses[2]
+ assert channel.requests == [("GetOperation", expected_request)] * 3
+ assert channel.GetOperation.requests == [expected_request] * 3
+
+ with pytest.raises(StopIteration):
+ stub.GetOperation(expected_request)
+
+ def test_multiple_responses_and_single_response_error(self):
+ channel = grpc_helpers.ChannelStub()
+ stub = operations_pb2.OperationsStub(channel)
+ channel.GetOperation.responses = []
+ channel.GetOperation.response = mock.sentinel.response
+
+ with pytest.raises(ValueError):
+ stub.GetOperation(operations_pb2.GetOperationRequest())
+
+ def test_call_info(self):
+ channel = grpc_helpers.ChannelStub()
+ stub = operations_pb2.OperationsStub(channel)
+ expected_request = operations_pb2.GetOperationRequest(name="meep")
+ expected_response = operations_pb2.Operation(name="moop")
+ expected_metadata = [("red", "blue"), ("two", "shoe")]
+ expected_credentials = mock.sentinel.credentials
+ channel.GetOperation.response = expected_response
+
+ response = stub.GetOperation(
+ expected_request,
+ timeout=42,
+ metadata=expected_metadata,
+ credentials=expected_credentials,
+ )
+
+ assert response == expected_response
+ assert channel.requests == [("GetOperation", expected_request)]
+ assert channel.GetOperation.calls == [
+ (expected_request, 42, expected_metadata, expected_credentials)
+ ]
+
+ def test_unary_unary(self):
+ channel = grpc_helpers.ChannelStub()
+ method_name = "GetOperation"
+ callable_stub = channel.unary_unary(method_name)
+ assert callable_stub._method == method_name
+ assert callable_stub._channel == channel
+
+ def test_unary_stream(self):
+ channel = grpc_helpers.ChannelStub()
+ method_name = "GetOperation"
+ callable_stub = channel.unary_stream(method_name)
+ assert callable_stub._method == method_name
+ assert callable_stub._channel == channel
+
+ def test_stream_unary(self):
+ channel = grpc_helpers.ChannelStub()
+ method_name = "GetOperation"
+ callable_stub = channel.stream_unary(method_name)
+ assert callable_stub._method == method_name
+ assert callable_stub._channel == channel
+
+ def test_stream_stream(self):
+ channel = grpc_helpers.ChannelStub()
+ method_name = "GetOperation"
+ callable_stub = channel.stream_stream(method_name)
+ assert callable_stub._method == method_name
+ assert callable_stub._channel == channel
+
+ def test_subscribe_unsubscribe(self):
+ channel = grpc_helpers.ChannelStub()
+ assert channel.subscribe(None) is None
+ assert channel.unsubscribe(None) is None
+
+ def test_close(self):
+ channel = grpc_helpers.ChannelStub()
+ assert channel.close() is None
diff --git a/api_core/tests/unit/test_iam.py b/api_core/tests/unit/test_iam.py
new file mode 100644
index 000000000000..199c38907983
--- /dev/null
+++ b/api_core/tests/unit/test_iam.py
@@ -0,0 +1,288 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import pytest
+
+
+class TestPolicy:
+ @staticmethod
+ def _get_target_class():
+ from google.api_core.iam import Policy
+
+ return Policy
+
+ def _make_one(self, *args, **kw):
+ return self._get_target_class()(*args, **kw)
+
+ def test_ctor_defaults(self):
+ empty = frozenset()
+ policy = self._make_one()
+ assert policy.etag is None
+ assert policy.version is None
+ assert policy.owners == empty
+ assert policy.editors == empty
+ assert policy.viewers == empty
+ assert len(policy) == 0
+ assert dict(policy) == {}
+
+ def test_ctor_explicit(self):
+ VERSION = 17
+ ETAG = "ETAG"
+ empty = frozenset()
+ policy = self._make_one(ETAG, VERSION)
+ assert policy.etag == ETAG
+ assert policy.version == VERSION
+ assert policy.owners == empty
+ assert policy.editors == empty
+ assert policy.viewers == empty
+ assert len(policy) == 0
+ assert dict(policy) == {}
+
+ def test___getitem___miss(self):
+ policy = self._make_one()
+ assert policy["nonesuch"] == set()
+
+ def test___setitem__(self):
+ USER = "user:phred@example.com"
+ PRINCIPALS = set([USER])
+ policy = self._make_one()
+ policy["rolename"] = [USER]
+ assert policy["rolename"] == PRINCIPALS
+ assert len(policy) == 1
+ assert dict(policy) == {"rolename": PRINCIPALS}
+
+ def test___delitem___hit(self):
+ policy = self._make_one()
+ policy._bindings["rolename"] = ["phred@example.com"]
+ del policy["rolename"]
+ assert len(policy) == 0
+ assert dict(policy) == {}
+
+ def test___delitem___miss(self):
+ policy = self._make_one()
+ with pytest.raises(KeyError):
+ del policy["nonesuch"]
+
+ def test_owners_getter(self):
+ from google.api_core.iam import OWNER_ROLE
+
+ MEMBER = "user:phred@example.com"
+ expected = frozenset([MEMBER])
+ policy = self._make_one()
+ policy[OWNER_ROLE] = [MEMBER]
+ assert policy.owners == expected
+
+ def test_owners_setter(self):
+ import warnings
+ from google.api_core.iam import OWNER_ROLE
+
+ MEMBER = "user:phred@example.com"
+ expected = set([MEMBER])
+ policy = self._make_one()
+
+ with warnings.catch_warnings(record=True) as warned:
+ policy.owners = [MEMBER]
+
+ warning, = warned
+ assert warning.category is DeprecationWarning
+ assert policy[OWNER_ROLE] == expected
+
+ def test_editors_getter(self):
+ from google.api_core.iam import EDITOR_ROLE
+
+ MEMBER = "user:phred@example.com"
+ expected = frozenset([MEMBER])
+ policy = self._make_one()
+ policy[EDITOR_ROLE] = [MEMBER]
+ assert policy.editors == expected
+
+ def test_editors_setter(self):
+ import warnings
+ from google.api_core.iam import EDITOR_ROLE
+
+ MEMBER = "user:phred@example.com"
+ expected = set([MEMBER])
+ policy = self._make_one()
+
+ with warnings.catch_warnings(record=True) as warned:
+ policy.editors = [MEMBER]
+
+ warning, = warned
+ assert warning.category is DeprecationWarning
+ assert policy[EDITOR_ROLE] == expected
+
+ def test_viewers_getter(self):
+ from google.api_core.iam import VIEWER_ROLE
+
+ MEMBER = "user:phred@example.com"
+ expected = frozenset([MEMBER])
+ policy = self._make_one()
+ policy[VIEWER_ROLE] = [MEMBER]
+ assert policy.viewers == expected
+
+ def test_viewers_setter(self):
+ import warnings
+ from google.api_core.iam import VIEWER_ROLE
+
+ MEMBER = "user:phred@example.com"
+ expected = set([MEMBER])
+ policy = self._make_one()
+
+ with warnings.catch_warnings(record=True) as warned:
+ policy.viewers = [MEMBER]
+
+ warning, = warned
+ assert warning.category is DeprecationWarning
+ assert policy[VIEWER_ROLE] == expected
+
+ def test_user(self):
+ EMAIL = "phred@example.com"
+ MEMBER = "user:%s" % (EMAIL,)
+ policy = self._make_one()
+ assert policy.user(EMAIL) == MEMBER
+
+ def test_service_account(self):
+ EMAIL = "phred@example.com"
+ MEMBER = "serviceAccount:%s" % (EMAIL,)
+ policy = self._make_one()
+ assert policy.service_account(EMAIL) == MEMBER
+
+ def test_group(self):
+ EMAIL = "phred@example.com"
+ MEMBER = "group:%s" % (EMAIL,)
+ policy = self._make_one()
+ assert policy.group(EMAIL) == MEMBER
+
+ def test_domain(self):
+ DOMAIN = "example.com"
+ MEMBER = "domain:%s" % (DOMAIN,)
+ policy = self._make_one()
+ assert policy.domain(DOMAIN) == MEMBER
+
+ def test_all_users(self):
+ policy = self._make_one()
+ assert policy.all_users() == "allUsers"
+
+ def test_authenticated_users(self):
+ policy = self._make_one()
+ assert policy.authenticated_users() == "allAuthenticatedUsers"
+
+ def test_from_api_repr_only_etag(self):
+ empty = frozenset()
+ RESOURCE = {"etag": "ACAB"}
+ klass = self._get_target_class()
+ policy = klass.from_api_repr(RESOURCE)
+ assert policy.etag == "ACAB"
+ assert policy.version is None
+ assert policy.owners == empty
+ assert policy.editors == empty
+ assert policy.viewers == empty
+ assert dict(policy) == {}
+
+ def test_from_api_repr_complete(self):
+ from google.api_core.iam import OWNER_ROLE, EDITOR_ROLE, VIEWER_ROLE
+
+ OWNER1 = "group:cloud-logs@google.com"
+ OWNER2 = "user:phred@example.com"
+ EDITOR1 = "domain:google.com"
+ EDITOR2 = "user:phred@example.com"
+ VIEWER1 = "serviceAccount:1234-abcdef@service.example.com"
+ VIEWER2 = "user:phred@example.com"
+ RESOURCE = {
+ "etag": "DEADBEEF",
+ "version": 17,
+ "bindings": [
+ {"role": OWNER_ROLE, "members": [OWNER1, OWNER2]},
+ {"role": EDITOR_ROLE, "members": [EDITOR1, EDITOR2]},
+ {"role": VIEWER_ROLE, "members": [VIEWER1, VIEWER2]},
+ ],
+ }
+ klass = self._get_target_class()
+ policy = klass.from_api_repr(RESOURCE)
+ assert policy.etag == "DEADBEEF"
+ assert policy.version == 17
+ assert policy.owners, frozenset([OWNER1 == OWNER2])
+ assert policy.editors, frozenset([EDITOR1 == EDITOR2])
+ assert policy.viewers, frozenset([VIEWER1 == VIEWER2])
+ assert dict(policy) == {
+ OWNER_ROLE: set([OWNER1, OWNER2]),
+ EDITOR_ROLE: set([EDITOR1, EDITOR2]),
+ VIEWER_ROLE: set([VIEWER1, VIEWER2]),
+ }
+
+ def test_from_api_repr_unknown_role(self):
+ USER = "user:phred@example.com"
+ GROUP = "group:cloud-logs@google.com"
+ RESOURCE = {
+ "etag": "DEADBEEF",
+ "version": 17,
+ "bindings": [{"role": "unknown", "members": [USER, GROUP]}],
+ }
+ klass = self._get_target_class()
+ policy = klass.from_api_repr(RESOURCE)
+ assert policy.etag == "DEADBEEF"
+ assert policy.version == 17
+ assert dict(policy), {"unknown": set([GROUP == USER])}
+
+ def test_to_api_repr_defaults(self):
+ policy = self._make_one()
+ assert policy.to_api_repr() == {}
+
+ def test_to_api_repr_only_etag(self):
+ policy = self._make_one("DEADBEEF")
+ assert policy.to_api_repr() == {"etag": "DEADBEEF"}
+
+ def test_to_api_repr_binding_wo_members(self):
+ policy = self._make_one()
+ policy["empty"] = []
+ assert policy.to_api_repr() == {}
+
+ def test_to_api_repr_binding_w_duplicates(self):
+ import warnings
+ from google.api_core.iam import OWNER_ROLE
+
+ OWNER = "group:cloud-logs@google.com"
+ policy = self._make_one()
+ with warnings.catch_warnings(record=True):
+ policy.owners = [OWNER, OWNER]
+ assert policy.to_api_repr() == {
+ "bindings": [{"role": OWNER_ROLE, "members": [OWNER]}]
+ }
+
+ def test_to_api_repr_full(self):
+ import operator
+ import warnings
+ from google.api_core.iam import OWNER_ROLE, EDITOR_ROLE, VIEWER_ROLE
+
+ OWNER1 = "group:cloud-logs@google.com"
+ OWNER2 = "user:phred@example.com"
+ EDITOR1 = "domain:google.com"
+ EDITOR2 = "user:phred@example.com"
+ VIEWER1 = "serviceAccount:1234-abcdef@service.example.com"
+ VIEWER2 = "user:phred@example.com"
+ BINDINGS = [
+ {"role": OWNER_ROLE, "members": [OWNER1, OWNER2]},
+ {"role": EDITOR_ROLE, "members": [EDITOR1, EDITOR2]},
+ {"role": VIEWER_ROLE, "members": [VIEWER1, VIEWER2]},
+ ]
+ policy = self._make_one("DEADBEEF", 17)
+ with warnings.catch_warnings(record=True):
+ policy.owners = [OWNER1, OWNER2]
+ policy.editors = [EDITOR1, EDITOR2]
+ policy.viewers = [VIEWER1, VIEWER2]
+ resource = policy.to_api_repr()
+ assert resource["etag"] == "DEADBEEF"
+ assert resource["version"] == 17
+ key = operator.itemgetter("role")
+ assert sorted(resource["bindings"], key=key) == sorted(BINDINGS, key=key)
diff --git a/api_core/tests/unit/test_operation.py b/api_core/tests/unit/test_operation.py
new file mode 100644
index 000000000000..a5346a706422
--- /dev/null
+++ b/api_core/tests/unit/test_operation.py
@@ -0,0 +1,241 @@
+# Copyright 2017, Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import mock
+
+from google.api_core import operation
+from google.api_core import operations_v1
+from google.longrunning import operations_pb2
+from google.protobuf import struct_pb2
+from google.rpc import code_pb2
+from google.rpc import status_pb2
+
+TEST_OPERATION_NAME = "test/operation"
+
+
+def make_operation_proto(
+ name=TEST_OPERATION_NAME, metadata=None, response=None, error=None, **kwargs
+):
+ operation_proto = operations_pb2.Operation(name=name, **kwargs)
+
+ if metadata is not None:
+ operation_proto.metadata.Pack(metadata)
+
+ if response is not None:
+ operation_proto.response.Pack(response)
+
+ if error is not None:
+ operation_proto.error.CopyFrom(error)
+
+ return operation_proto
+
+
+def make_operation_future(client_operations_responses=None):
+ if client_operations_responses is None:
+ client_operations_responses = [make_operation_proto()]
+
+ refresh = mock.Mock(spec=["__call__"], side_effect=client_operations_responses)
+ refresh.responses = client_operations_responses
+ cancel = mock.Mock(spec=["__call__"])
+ operation_future = operation.Operation(
+ client_operations_responses[0],
+ refresh,
+ cancel,
+ result_type=struct_pb2.Struct,
+ metadata_type=struct_pb2.Struct,
+ )
+
+ return operation_future, refresh, cancel
+
+
+def test_constructor():
+ future, refresh, _ = make_operation_future()
+
+ assert future.operation == refresh.responses[0]
+ assert future.operation.done is False
+ assert future.operation.name == TEST_OPERATION_NAME
+ assert future.metadata is None
+ assert future.running()
+
+
+def test_metadata():
+ expected_metadata = struct_pb2.Struct()
+ future, _, _ = make_operation_future(
+ [make_operation_proto(metadata=expected_metadata)]
+ )
+
+ assert future.metadata == expected_metadata
+
+
+def test_cancellation():
+ responses = [
+ make_operation_proto(),
+ # Second response indicates that the operation was cancelled.
+ make_operation_proto(
+ done=True, error=status_pb2.Status(code=code_pb2.CANCELLED)
+ ),
+ ]
+ future, _, cancel = make_operation_future(responses)
+
+ assert future.cancel()
+ assert future.cancelled()
+ cancel.assert_called_once_with()
+
+ # Cancelling twice should have no effect.
+ assert not future.cancel()
+ cancel.assert_called_once_with()
+
+
+def test_result():
+ expected_result = struct_pb2.Struct()
+ responses = [
+ make_operation_proto(),
+ # Second operation response includes the result.
+ make_operation_proto(done=True, response=expected_result),
+ ]
+ future, _, _ = make_operation_future(responses)
+
+ result = future.result()
+
+ assert result == expected_result
+ assert future.done()
+
+
+def test_exception():
+ expected_exception = status_pb2.Status(message="meep")
+ responses = [
+ make_operation_proto(),
+ # Second operation response includes the error.
+ make_operation_proto(done=True, error=expected_exception),
+ ]
+ future, _, _ = make_operation_future(responses)
+
+ exception = future.exception()
+
+ assert expected_exception.message in "{!r}".format(exception)
+
+
+def test_unexpected_result():
+ responses = [
+ make_operation_proto(),
+ # Second operation response is done, but has not error or response.
+ make_operation_proto(done=True),
+ ]
+ future, _, _ = make_operation_future(responses)
+
+ exception = future.exception()
+
+ assert "Unexpected state" in "{!r}".format(exception)
+
+
+def test__refresh_http():
+ api_request = mock.Mock(return_value={"name": TEST_OPERATION_NAME, "done": True})
+
+ result = operation._refresh_http(api_request, TEST_OPERATION_NAME)
+
+ assert result.name == TEST_OPERATION_NAME
+ assert result.done is True
+ api_request.assert_called_once_with(
+ method="GET", path="operations/{}".format(TEST_OPERATION_NAME)
+ )
+
+
+def test__cancel_http():
+ api_request = mock.Mock()
+
+ operation._cancel_http(api_request, TEST_OPERATION_NAME)
+
+ api_request.assert_called_once_with(
+ method="POST", path="operations/{}:cancel".format(TEST_OPERATION_NAME)
+ )
+
+
+def test_from_http_json():
+ operation_json = {"name": TEST_OPERATION_NAME, "done": True}
+ api_request = mock.sentinel.api_request
+
+ future = operation.from_http_json(
+ operation_json, api_request, struct_pb2.Struct, metadata_type=struct_pb2.Struct
+ )
+
+ assert future._result_type == struct_pb2.Struct
+ assert future._metadata_type == struct_pb2.Struct
+ assert future.operation.name == TEST_OPERATION_NAME
+ assert future.done
+
+
+def test__refresh_grpc():
+ operations_stub = mock.Mock(spec=["GetOperation"])
+ expected_result = make_operation_proto(done=True)
+ operations_stub.GetOperation.return_value = expected_result
+
+ result = operation._refresh_grpc(operations_stub, TEST_OPERATION_NAME)
+
+ assert result == expected_result
+ expected_request = operations_pb2.GetOperationRequest(name=TEST_OPERATION_NAME)
+ operations_stub.GetOperation.assert_called_once_with(expected_request)
+
+
+def test__cancel_grpc():
+ operations_stub = mock.Mock(spec=["CancelOperation"])
+
+ operation._cancel_grpc(operations_stub, TEST_OPERATION_NAME)
+
+ expected_request = operations_pb2.CancelOperationRequest(name=TEST_OPERATION_NAME)
+ operations_stub.CancelOperation.assert_called_once_with(expected_request)
+
+
+def test_from_grpc():
+ operation_proto = make_operation_proto(done=True)
+ operations_stub = mock.sentinel.operations_stub
+
+ future = operation.from_grpc(
+ operation_proto,
+ operations_stub,
+ struct_pb2.Struct,
+ metadata_type=struct_pb2.Struct,
+ )
+
+ assert future._result_type == struct_pb2.Struct
+ assert future._metadata_type == struct_pb2.Struct
+ assert future.operation.name == TEST_OPERATION_NAME
+ assert future.done
+
+
+def test_from_gapic():
+ operation_proto = make_operation_proto(done=True)
+ operations_client = mock.create_autospec(
+ operations_v1.OperationsClient, instance=True
+ )
+
+ future = operation.from_gapic(
+ operation_proto,
+ operations_client,
+ struct_pb2.Struct,
+ metadata_type=struct_pb2.Struct,
+ )
+
+ assert future._result_type == struct_pb2.Struct
+ assert future._metadata_type == struct_pb2.Struct
+ assert future.operation.name == TEST_OPERATION_NAME
+ assert future.done
+
+
+def test_deserialize():
+ op = make_operation_proto(name="foobarbaz")
+ serialized = op.SerializeToString()
+ deserialized_op = operation.Operation.deserialize(serialized)
+ assert op.name == deserialized_op.name
+ assert type(op) is type(deserialized_op)
diff --git a/api_core/tests/unit/test_page_iterator.py b/api_core/tests/unit/test_page_iterator.py
new file mode 100644
index 000000000000..6335001bcf41
--- /dev/null
+++ b/api_core/tests/unit/test_page_iterator.py
@@ -0,0 +1,570 @@
+# Copyright 2015 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import types
+
+import mock
+import pytest
+import six
+
+from google.api_core import page_iterator
+
+
+def test__do_nothing_page_start():
+ assert page_iterator._do_nothing_page_start(None, None, None) is None
+
+
+class TestPage(object):
+ def test_constructor(self):
+ parent = mock.sentinel.parent
+ item_to_value = mock.sentinel.item_to_value
+
+ page = page_iterator.Page(parent, (1, 2, 3), item_to_value)
+
+ assert page.num_items == 3
+ assert page.remaining == 3
+ assert page._parent is parent
+ assert page._item_to_value is item_to_value
+
+ def test___iter__(self):
+ page = page_iterator.Page(None, (), None)
+ assert iter(page) is page
+
+ def test_iterator_calls_parent_item_to_value(self):
+ parent = mock.sentinel.parent
+
+ item_to_value = mock.Mock(
+ side_effect=lambda iterator, value: value, spec=["__call__"]
+ )
+
+ page = page_iterator.Page(parent, (10, 11, 12), item_to_value)
+ page._remaining = 100
+
+ assert item_to_value.call_count == 0
+ assert page.remaining == 100
+
+ assert six.next(page) == 10
+ assert item_to_value.call_count == 1
+ item_to_value.assert_called_with(parent, 10)
+ assert page.remaining == 99
+
+ assert six.next(page) == 11
+ assert item_to_value.call_count == 2
+ item_to_value.assert_called_with(parent, 11)
+ assert page.remaining == 98
+
+ assert six.next(page) == 12
+ assert item_to_value.call_count == 3
+ item_to_value.assert_called_with(parent, 12)
+ assert page.remaining == 97
+
+
+class PageIteratorImpl(page_iterator.Iterator):
+ def _next_page(self):
+ return mock.create_autospec(page_iterator.Page, instance=True)
+
+
+class TestIterator(object):
+ def test_constructor(self):
+ client = mock.sentinel.client
+ item_to_value = mock.sentinel.item_to_value
+ token = "ab13nceor03"
+ max_results = 1337
+
+ iterator = PageIteratorImpl(
+ client, item_to_value, page_token=token, max_results=max_results
+ )
+
+ assert not iterator._started
+ assert iterator.client is client
+ assert iterator.item_to_value == item_to_value
+ assert iterator.max_results == max_results
+ # Changing attributes.
+ assert iterator.page_number == 0
+ assert iterator.next_page_token == token
+ assert iterator.num_results == 0
+
+ def test_pages_property_starts(self):
+ iterator = PageIteratorImpl(None, None)
+
+ assert not iterator._started
+
+ assert isinstance(iterator.pages, types.GeneratorType)
+
+ assert iterator._started
+
+ def test_pages_property_restart(self):
+ iterator = PageIteratorImpl(None, None)
+
+ assert iterator.pages
+
+ # Make sure we cannot restart.
+ with pytest.raises(ValueError):
+ assert iterator.pages
+
+ def test__page_iter_increment(self):
+ iterator = PageIteratorImpl(None, None)
+ page = page_iterator.Page(
+ iterator, ("item",), page_iterator._item_to_value_identity
+ )
+ iterator._next_page = mock.Mock(side_effect=[page, None])
+
+ assert iterator.num_results == 0
+
+ page_iter = iterator._page_iter(increment=True)
+ next(page_iter)
+
+ assert iterator.num_results == 1
+
+ def test__page_iter_no_increment(self):
+ iterator = PageIteratorImpl(None, None)
+
+ assert iterator.num_results == 0
+
+ page_iter = iterator._page_iter(increment=False)
+ next(page_iter)
+
+ # results should still be 0 after fetching a page.
+ assert iterator.num_results == 0
+
+ def test__items_iter(self):
+ # Items to be returned.
+ item1 = 17
+ item2 = 100
+ item3 = 211
+
+ # Make pages from mock responses
+ parent = mock.sentinel.parent
+ page1 = page_iterator.Page(
+ parent, (item1, item2), page_iterator._item_to_value_identity
+ )
+ page2 = page_iterator.Page(
+ parent, (item3,), page_iterator._item_to_value_identity
+ )
+
+ iterator = PageIteratorImpl(None, None)
+ iterator._next_page = mock.Mock(side_effect=[page1, page2, None])
+
+ items_iter = iterator._items_iter()
+
+ assert isinstance(items_iter, types.GeneratorType)
+
+ # Consume items and check the state of the iterator.
+ assert iterator.num_results == 0
+
+ assert six.next(items_iter) == item1
+ assert iterator.num_results == 1
+
+ assert six.next(items_iter) == item2
+ assert iterator.num_results == 2
+
+ assert six.next(items_iter) == item3
+ assert iterator.num_results == 3
+
+ with pytest.raises(StopIteration):
+ six.next(items_iter)
+
+ def test___iter__(self):
+ iterator = PageIteratorImpl(None, None)
+ iterator._next_page = mock.Mock(side_effect=[(1, 2), (3,), None])
+
+ assert not iterator._started
+
+ result = list(iterator)
+
+ assert result == [1, 2, 3]
+ assert iterator._started
+
+ def test___iter__restart(self):
+ iterator = PageIteratorImpl(None, None)
+
+ iter(iterator)
+
+ # Make sure we cannot restart.
+ with pytest.raises(ValueError):
+ iter(iterator)
+
+ def test___iter___restart_after_page(self):
+ iterator = PageIteratorImpl(None, None)
+
+ assert iterator.pages
+
+ # Make sure we cannot restart after starting the page iterator
+ with pytest.raises(ValueError):
+ iter(iterator)
+
+
+class TestHTTPIterator(object):
+ def test_constructor(self):
+ client = mock.sentinel.client
+ path = "/foo"
+ iterator = page_iterator.HTTPIterator(
+ client, mock.sentinel.api_request, path, mock.sentinel.item_to_value
+ )
+
+ assert not iterator._started
+ assert iterator.client is client
+ assert iterator.path == path
+ assert iterator.item_to_value is mock.sentinel.item_to_value
+ assert iterator._items_key == "items"
+ assert iterator.max_results is None
+ assert iterator.extra_params == {}
+ assert iterator._page_start == page_iterator._do_nothing_page_start
+ # Changing attributes.
+ assert iterator.page_number == 0
+ assert iterator.next_page_token is None
+ assert iterator.num_results == 0
+
+ def test_constructor_w_extra_param_collision(self):
+ extra_params = {"pageToken": "val"}
+
+ with pytest.raises(ValueError):
+ page_iterator.HTTPIterator(
+ mock.sentinel.client,
+ mock.sentinel.api_request,
+ mock.sentinel.path,
+ mock.sentinel.item_to_value,
+ extra_params=extra_params,
+ )
+
+ def test_iterate(self):
+ path = "/foo"
+ item1 = {"name": "1"}
+ item2 = {"name": "2"}
+ api_request = mock.Mock(return_value={"items": [item1, item2]})
+ iterator = page_iterator.HTTPIterator(
+ mock.sentinel.client,
+ api_request,
+ path=path,
+ item_to_value=page_iterator._item_to_value_identity,
+ )
+
+ assert iterator.num_results == 0
+
+ items_iter = iter(iterator)
+
+ val1 = six.next(items_iter)
+ assert val1 == item1
+ assert iterator.num_results == 1
+
+ val2 = six.next(items_iter)
+ assert val2 == item2
+ assert iterator.num_results == 2
+
+ with pytest.raises(StopIteration):
+ six.next(items_iter)
+
+ api_request.assert_called_once_with(method="GET", path=path, query_params={})
+
+ def test__has_next_page_new(self):
+ iterator = page_iterator.HTTPIterator(
+ mock.sentinel.client,
+ mock.sentinel.api_request,
+ mock.sentinel.path,
+ mock.sentinel.item_to_value,
+ )
+
+ # The iterator should *always* indicate that it has a next page
+ # when created so that it can fetch the initial page.
+ assert iterator._has_next_page()
+
+ def test__has_next_page_without_token(self):
+ iterator = page_iterator.HTTPIterator(
+ mock.sentinel.client,
+ mock.sentinel.api_request,
+ mock.sentinel.path,
+ mock.sentinel.item_to_value,
+ )
+
+ iterator.page_number = 1
+
+ # The iterator should not indicate that it has a new page if the
+ # initial page has been requested and there's no page token.
+ assert not iterator._has_next_page()
+
+ def test__has_next_page_w_number_w_token(self):
+ iterator = page_iterator.HTTPIterator(
+ mock.sentinel.client,
+ mock.sentinel.api_request,
+ mock.sentinel.path,
+ mock.sentinel.item_to_value,
+ )
+
+ iterator.page_number = 1
+ iterator.next_page_token = mock.sentinel.token
+
+ # The iterator should indicate that it has a new page if the
+ # initial page has been requested and there's is a page token.
+ assert iterator._has_next_page()
+
+ def test__has_next_page_w_max_results_not_done(self):
+ iterator = page_iterator.HTTPIterator(
+ mock.sentinel.client,
+ mock.sentinel.api_request,
+ mock.sentinel.path,
+ mock.sentinel.item_to_value,
+ max_results=3,
+ page_token=mock.sentinel.token,
+ )
+
+ iterator.page_number = 1
+
+ # The iterator should indicate that it has a new page if there
+ # is a page token and it has not consumed more than max_results.
+ assert iterator.num_results < iterator.max_results
+ assert iterator._has_next_page()
+
+ def test__has_next_page_w_max_results_done(self):
+
+ iterator = page_iterator.HTTPIterator(
+ mock.sentinel.client,
+ mock.sentinel.api_request,
+ mock.sentinel.path,
+ mock.sentinel.item_to_value,
+ max_results=3,
+ page_token=mock.sentinel.token,
+ )
+
+ iterator.page_number = 1
+ iterator.num_results = 3
+
+ # The iterator should not indicate that it has a new page if there
+ # if it has consumed more than max_results.
+ assert iterator.num_results == iterator.max_results
+ assert not iterator._has_next_page()
+
+ def test__get_query_params_no_token(self):
+ iterator = page_iterator.HTTPIterator(
+ mock.sentinel.client,
+ mock.sentinel.api_request,
+ mock.sentinel.path,
+ mock.sentinel.item_to_value,
+ )
+
+ assert iterator._get_query_params() == {}
+
+ def test__get_query_params_w_token(self):
+ iterator = page_iterator.HTTPIterator(
+ mock.sentinel.client,
+ mock.sentinel.api_request,
+ mock.sentinel.path,
+ mock.sentinel.item_to_value,
+ )
+ iterator.next_page_token = "token"
+
+ assert iterator._get_query_params() == {"pageToken": iterator.next_page_token}
+
+ def test__get_query_params_w_max_results(self):
+ max_results = 3
+ iterator = page_iterator.HTTPIterator(
+ mock.sentinel.client,
+ mock.sentinel.api_request,
+ mock.sentinel.path,
+ mock.sentinel.item_to_value,
+ max_results=max_results,
+ )
+
+ iterator.num_results = 1
+ local_max = max_results - iterator.num_results
+
+ assert iterator._get_query_params() == {"maxResults": local_max}
+
+ def test__get_query_params_extra_params(self):
+ extra_params = {"key": "val"}
+ iterator = page_iterator.HTTPIterator(
+ mock.sentinel.client,
+ mock.sentinel.api_request,
+ mock.sentinel.path,
+ mock.sentinel.item_to_value,
+ extra_params=extra_params,
+ )
+
+ assert iterator._get_query_params() == extra_params
+
+ def test__get_next_page_response_with_post(self):
+ path = "/foo"
+ page_response = {"items": ["one", "two"]}
+ api_request = mock.Mock(return_value=page_response)
+ iterator = page_iterator.HTTPIterator(
+ mock.sentinel.client,
+ api_request,
+ path=path,
+ item_to_value=page_iterator._item_to_value_identity,
+ )
+ iterator._HTTP_METHOD = "POST"
+
+ response = iterator._get_next_page_response()
+
+ assert response == page_response
+
+ api_request.assert_called_once_with(method="POST", path=path, data={})
+
+ def test__get_next_page_bad_http_method(self):
+ iterator = page_iterator.HTTPIterator(
+ mock.sentinel.client,
+ mock.sentinel.api_request,
+ mock.sentinel.path,
+ mock.sentinel.item_to_value,
+ )
+ iterator._HTTP_METHOD = "NOT-A-VERB"
+
+ with pytest.raises(ValueError):
+ iterator._get_next_page_response()
+
+
+class TestGRPCIterator(object):
+ def test_constructor(self):
+ client = mock.sentinel.client
+ items_field = "items"
+ iterator = page_iterator.GRPCIterator(
+ client, mock.sentinel.method, mock.sentinel.request, items_field
+ )
+
+ assert not iterator._started
+ assert iterator.client is client
+ assert iterator.max_results is None
+ assert iterator.item_to_value is page_iterator._item_to_value_identity
+ assert iterator._method == mock.sentinel.method
+ assert iterator._request == mock.sentinel.request
+ assert iterator._items_field == items_field
+ assert (
+ iterator._request_token_field
+ == page_iterator.GRPCIterator._DEFAULT_REQUEST_TOKEN_FIELD
+ )
+ assert (
+ iterator._response_token_field
+ == page_iterator.GRPCIterator._DEFAULT_RESPONSE_TOKEN_FIELD
+ )
+ # Changing attributes.
+ assert iterator.page_number == 0
+ assert iterator.next_page_token is None
+ assert iterator.num_results == 0
+
+ def test_constructor_options(self):
+ client = mock.sentinel.client
+ items_field = "items"
+ request_field = "request"
+ response_field = "response"
+ iterator = page_iterator.GRPCIterator(
+ client,
+ mock.sentinel.method,
+ mock.sentinel.request,
+ items_field,
+ item_to_value=mock.sentinel.item_to_value,
+ request_token_field=request_field,
+ response_token_field=response_field,
+ max_results=42,
+ )
+
+ assert iterator.client is client
+ assert iterator.max_results == 42
+ assert iterator.item_to_value is mock.sentinel.item_to_value
+ assert iterator._method == mock.sentinel.method
+ assert iterator._request == mock.sentinel.request
+ assert iterator._items_field == items_field
+ assert iterator._request_token_field == request_field
+ assert iterator._response_token_field == response_field
+
+ def test_iterate(self):
+ request = mock.Mock(spec=["page_token"], page_token=None)
+ response1 = mock.Mock(items=["a", "b"], next_page_token="1")
+ response2 = mock.Mock(items=["c"], next_page_token="2")
+ response3 = mock.Mock(items=["d"], next_page_token="")
+ method = mock.Mock(side_effect=[response1, response2, response3])
+ iterator = page_iterator.GRPCIterator(
+ mock.sentinel.client, method, request, "items"
+ )
+
+ assert iterator.num_results == 0
+
+ items = list(iterator)
+ assert items == ["a", "b", "c", "d"]
+
+ method.assert_called_with(request)
+ assert method.call_count == 3
+ assert request.page_token == "2"
+
+ def test_iterate_with_max_results(self):
+ request = mock.Mock(spec=["page_token"], page_token=None)
+ response1 = mock.Mock(items=["a", "b"], next_page_token="1")
+ response2 = mock.Mock(items=["c"], next_page_token="2")
+ response3 = mock.Mock(items=["d"], next_page_token="")
+ method = mock.Mock(side_effect=[response1, response2, response3])
+ iterator = page_iterator.GRPCIterator(
+ mock.sentinel.client, method, request, "items", max_results=3
+ )
+
+ assert iterator.num_results == 0
+
+ items = list(iterator)
+
+ assert items == ["a", "b", "c"]
+ assert iterator.num_results == 3
+
+ method.assert_called_with(request)
+ assert method.call_count == 2
+ assert request.page_token == "1"
+
+
+class GAXPageIterator(object):
+ """Fake object that matches gax.PageIterator"""
+
+ def __init__(self, pages, page_token=None):
+ self._pages = iter(pages)
+ self.page_token = page_token
+
+ def next(self):
+ return six.next(self._pages)
+
+ __next__ = next
+
+
+class TestGAXIterator(object):
+ def test_constructor(self):
+ client = mock.sentinel.client
+ token = "zzzyy78kl"
+ page_iter = GAXPageIterator((), page_token=token)
+ item_to_value = page_iterator._item_to_value_identity
+ max_results = 1337
+ iterator = page_iterator._GAXIterator(
+ client, page_iter, item_to_value, max_results=max_results
+ )
+
+ assert not iterator._started
+ assert iterator.client is client
+ assert iterator.item_to_value is item_to_value
+ assert iterator.max_results == max_results
+ assert iterator._gax_page_iter is page_iter
+ # Changing attributes.
+ assert iterator.page_number == 0
+ assert iterator.next_page_token == token
+ assert iterator.num_results == 0
+
+ def test__next_page(self):
+ page_items = (29, 31)
+ page_token = "2sde98ds2s0hh"
+ page_iter = GAXPageIterator([page_items], page_token=page_token)
+ iterator = page_iterator._GAXIterator(
+ mock.sentinel.client, page_iter, page_iterator._item_to_value_identity
+ )
+
+ page = iterator._next_page()
+
+ assert iterator.next_page_token == page_token
+ assert isinstance(page, page_iterator.Page)
+ assert list(page) == list(page_items)
+
+ next_page = iterator._next_page()
+
+ assert next_page is None
diff --git a/api_core/tests/unit/test_path_template.py b/api_core/tests/unit/test_path_template.py
new file mode 100644
index 000000000000..4c8a7c5e7e7f
--- /dev/null
+++ b/api_core/tests/unit/test_path_template.py
@@ -0,0 +1,115 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import unicode_literals
+
+import mock
+import pytest
+
+from google.api_core import path_template
+
+
+@pytest.mark.parametrize(
+ "tmpl, args, kwargs, expected_result",
+ [
+ # Basic positional params
+ ["/v1/*", ["a"], {}, "/v1/a"],
+ ["/v1/**", ["a/b"], {}, "/v1/a/b"],
+ ["/v1/*/*", ["a", "b"], {}, "/v1/a/b"],
+ ["/v1/*/*/**", ["a", "b", "c/d"], {}, "/v1/a/b/c/d"],
+ # Basic named params
+ ["/v1/{name}", [], {"name": "parent"}, "/v1/parent"],
+ ["/v1/{name=**}", [], {"name": "parent/child"}, "/v1/parent/child"],
+ # Named params with a sub-template
+ ["/v1/{name=parent/*}", [], {"name": "parent/child"}, "/v1/parent/child"],
+ [
+ "/v1/{name=parent/**}",
+ [],
+ {"name": "parent/child/object"},
+ "/v1/parent/child/object",
+ ],
+ # Combining positional and named params
+ ["/v1/*/{name}", ["a"], {"name": "parent"}, "/v1/a/parent"],
+ ["/v1/{name}/*", ["a"], {"name": "parent"}, "/v1/parent/a"],
+ [
+ "/v1/{parent}/*/{child}/*",
+ ["a", "b"],
+ {"parent": "thor", "child": "thorson"},
+ "/v1/thor/a/thorson/b",
+ ],
+ ["/v1/{name}/**", ["a/b"], {"name": "parent"}, "/v1/parent/a/b"],
+ # Combining positional and named params with sub-templates.
+ [
+ "/v1/{name=parent/*}/*",
+ ["a"],
+ {"name": "parent/child"},
+ "/v1/parent/child/a",
+ ],
+ [
+ "/v1/*/{name=parent/**}",
+ ["a"],
+ {"name": "parent/child/object"},
+ "/v1/a/parent/child/object",
+ ],
+ ],
+)
+def test_expand_success(tmpl, args, kwargs, expected_result):
+ result = path_template.expand(tmpl, *args, **kwargs)
+ assert result == expected_result
+ assert path_template.validate(tmpl, result)
+
+
+@pytest.mark.parametrize(
+ "tmpl, args, kwargs, exc_match",
+ [
+ # Missing positional arg.
+ ["v1/*", [], {}, "Positional"],
+ # Missing named arg.
+ ["v1/{name}", [], {}, "Named"],
+ ],
+)
+def test_expanded_failure(tmpl, args, kwargs, exc_match):
+ with pytest.raises(ValueError, match=exc_match):
+ path_template.expand(tmpl, *args, **kwargs)
+
+
+@pytest.mark.parametrize(
+ "tmpl, path",
+ [
+ # Single segment template, but multi segment value
+ ["v1/*", "v1/a/b"],
+ ["v1/*/*", "v1/a/b/c"],
+ # Single segement named template, but multi segment value
+ ["v1/{name}", "v1/a/b"],
+ ["v1/{name}/{value}", "v1/a/b/c"],
+ # Named value with a sub-template but invalid value
+ ["v1/{name=parent/*}", "v1/grandparent/child"],
+ ],
+)
+def test_validate_failure(tmpl, path):
+ assert not path_template.validate(tmpl, path)
+
+
+def test__expand_variable_match_unexpected():
+ match = mock.Mock(spec=["group"])
+ match.group.return_value = None
+ with pytest.raises(ValueError, match="Unknown"):
+ path_template._expand_variable_match([], {}, match)
+
+
+def test__replace_variable_with_pattern():
+ match = mock.Mock(spec=["group"])
+ match.group.return_value = None
+ with pytest.raises(ValueError, match="Unknown"):
+ path_template._replace_variable_with_pattern(match)
diff --git a/api_core/tests/unit/test_protobuf_helpers.py b/api_core/tests/unit/test_protobuf_helpers.py
new file mode 100644
index 000000000000..db972383120a
--- /dev/null
+++ b/api_core/tests/unit/test_protobuf_helpers.py
@@ -0,0 +1,474 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import pytest
+
+from google.api import http_pb2
+from google.api_core import protobuf_helpers
+from google.longrunning import operations_pb2
+from google.protobuf import any_pb2
+from google.protobuf import message
+from google.protobuf import source_context_pb2
+from google.protobuf import struct_pb2
+from google.protobuf import timestamp_pb2
+from google.protobuf import type_pb2
+from google.protobuf import wrappers_pb2
+from google.type import color_pb2
+from google.type import date_pb2
+from google.type import timeofday_pb2
+
+
+def test_from_any_pb_success():
+ in_message = date_pb2.Date(year=1990)
+ in_message_any = any_pb2.Any()
+ in_message_any.Pack(in_message)
+ out_message = protobuf_helpers.from_any_pb(date_pb2.Date, in_message_any)
+
+ assert in_message == out_message
+
+
+def test_from_any_pb_wrapped_success():
+ # Declare a message class conforming to wrapped messages.
+ class WrappedDate(object):
+ def __init__(self, **kwargs):
+ self._pb = date_pb2.Date(**kwargs)
+
+ def __eq__(self, other):
+ return self._pb == other
+
+ @classmethod
+ def pb(cls, msg):
+ return msg._pb
+
+ # Run the same test as `test_from_any_pb_success`, but using the
+ # wrapped class.
+ in_message = date_pb2.Date(year=1990)
+ in_message_any = any_pb2.Any()
+ in_message_any.Pack(in_message)
+ out_message = protobuf_helpers.from_any_pb(WrappedDate, in_message_any)
+
+ assert out_message == in_message
+
+
+def test_from_any_pb_failure():
+ in_message = any_pb2.Any()
+ in_message.Pack(date_pb2.Date(year=1990))
+
+ with pytest.raises(TypeError):
+ protobuf_helpers.from_any_pb(timeofday_pb2.TimeOfDay, in_message)
+
+
+def test_check_protobuf_helpers_ok():
+ assert protobuf_helpers.check_oneof() is None
+ assert protobuf_helpers.check_oneof(foo="bar") is None
+ assert protobuf_helpers.check_oneof(foo="bar", baz=None) is None
+ assert protobuf_helpers.check_oneof(foo=None, baz="bacon") is None
+ assert protobuf_helpers.check_oneof(foo="bar", spam=None, eggs=None) is None
+
+
+def test_check_protobuf_helpers_failures():
+ with pytest.raises(ValueError):
+ protobuf_helpers.check_oneof(foo="bar", spam="eggs")
+ with pytest.raises(ValueError):
+ protobuf_helpers.check_oneof(foo="bar", baz="bacon", spam="eggs")
+ with pytest.raises(ValueError):
+ protobuf_helpers.check_oneof(foo="bar", spam=0, eggs=None)
+
+
+def test_get_messages():
+ answer = protobuf_helpers.get_messages(date_pb2)
+
+ # Ensure that Date was exported properly.
+ assert answer["Date"] is date_pb2.Date
+
+ # Ensure that no non-Message objects were exported.
+ for value in answer.values():
+ assert issubclass(value, message.Message)
+
+
+def test_get_dict_absent():
+ with pytest.raises(KeyError):
+ assert protobuf_helpers.get({}, "foo")
+
+
+def test_get_dict_present():
+ assert protobuf_helpers.get({"foo": "bar"}, "foo") == "bar"
+
+
+def test_get_dict_default():
+ assert protobuf_helpers.get({}, "foo", default="bar") == "bar"
+
+
+def test_get_dict_nested():
+ assert protobuf_helpers.get({"foo": {"bar": "baz"}}, "foo.bar") == "baz"
+
+
+def test_get_dict_nested_default():
+ assert protobuf_helpers.get({}, "foo.baz", default="bacon") == "bacon"
+ assert protobuf_helpers.get({"foo": {}}, "foo.baz", default="bacon") == "bacon"
+
+
+def test_get_msg_sentinel():
+ msg = timestamp_pb2.Timestamp()
+ with pytest.raises(KeyError):
+ assert protobuf_helpers.get(msg, "foo")
+
+
+def test_get_msg_present():
+ msg = timestamp_pb2.Timestamp(seconds=42)
+ assert protobuf_helpers.get(msg, "seconds") == 42
+
+
+def test_get_msg_default():
+ msg = timestamp_pb2.Timestamp()
+ assert protobuf_helpers.get(msg, "foo", default="bar") == "bar"
+
+
+def test_invalid_object():
+ with pytest.raises(TypeError):
+ protobuf_helpers.get(object(), "foo", "bar")
+
+
+def test_set_dict():
+ mapping = {}
+ protobuf_helpers.set(mapping, "foo", "bar")
+ assert mapping == {"foo": "bar"}
+
+
+def test_set_msg():
+ msg = timestamp_pb2.Timestamp()
+ protobuf_helpers.set(msg, "seconds", 42)
+ assert msg.seconds == 42
+
+
+def test_set_dict_nested():
+ mapping = {}
+ protobuf_helpers.set(mapping, "foo.bar", "baz")
+ assert mapping == {"foo": {"bar": "baz"}}
+
+
+def test_set_invalid_object():
+ with pytest.raises(TypeError):
+ protobuf_helpers.set(object(), "foo", "bar")
+
+
+def test_set_list():
+ list_ops_response = operations_pb2.ListOperationsResponse()
+
+ protobuf_helpers.set(
+ list_ops_response,
+ "operations",
+ [{"name": "foo"}, operations_pb2.Operation(name="bar")],
+ )
+
+ assert len(list_ops_response.operations) == 2
+
+ for operation in list_ops_response.operations:
+ assert isinstance(operation, operations_pb2.Operation)
+
+ assert list_ops_response.operations[0].name == "foo"
+ assert list_ops_response.operations[1].name == "bar"
+
+
+def test_set_list_clear_existing():
+ list_ops_response = operations_pb2.ListOperationsResponse(
+ operations=[{"name": "baz"}]
+ )
+
+ protobuf_helpers.set(
+ list_ops_response,
+ "operations",
+ [{"name": "foo"}, operations_pb2.Operation(name="bar")],
+ )
+
+ assert len(list_ops_response.operations) == 2
+ for operation in list_ops_response.operations:
+ assert isinstance(operation, operations_pb2.Operation)
+ assert list_ops_response.operations[0].name == "foo"
+ assert list_ops_response.operations[1].name == "bar"
+
+
+def test_set_msg_with_msg_field():
+ rule = http_pb2.HttpRule()
+ pattern = http_pb2.CustomHttpPattern(kind="foo", path="bar")
+
+ protobuf_helpers.set(rule, "custom", pattern)
+
+ assert rule.custom.kind == "foo"
+ assert rule.custom.path == "bar"
+
+
+def test_set_msg_with_dict_field():
+ rule = http_pb2.HttpRule()
+ pattern = {"kind": "foo", "path": "bar"}
+
+ protobuf_helpers.set(rule, "custom", pattern)
+
+ assert rule.custom.kind == "foo"
+ assert rule.custom.path == "bar"
+
+
+def test_set_msg_nested_key():
+ rule = http_pb2.HttpRule(custom=http_pb2.CustomHttpPattern(kind="foo", path="bar"))
+
+ protobuf_helpers.set(rule, "custom.kind", "baz")
+
+ assert rule.custom.kind == "baz"
+ assert rule.custom.path == "bar"
+
+
+def test_setdefault_dict_unset():
+ mapping = {}
+ protobuf_helpers.setdefault(mapping, "foo", "bar")
+ assert mapping == {"foo": "bar"}
+
+
+def test_setdefault_dict_falsy():
+ mapping = {"foo": None}
+ protobuf_helpers.setdefault(mapping, "foo", "bar")
+ assert mapping == {"foo": "bar"}
+
+
+def test_setdefault_dict_truthy():
+ mapping = {"foo": "bar"}
+ protobuf_helpers.setdefault(mapping, "foo", "baz")
+ assert mapping == {"foo": "bar"}
+
+
+def test_setdefault_pb2_falsy():
+ operation = operations_pb2.Operation()
+ protobuf_helpers.setdefault(operation, "name", "foo")
+ assert operation.name == "foo"
+
+
+def test_setdefault_pb2_truthy():
+ operation = operations_pb2.Operation(name="bar")
+ protobuf_helpers.setdefault(operation, "name", "foo")
+ assert operation.name == "bar"
+
+
+def test_field_mask_invalid_args():
+ with pytest.raises(ValueError):
+ protobuf_helpers.field_mask("foo", any_pb2.Any())
+ with pytest.raises(ValueError):
+ protobuf_helpers.field_mask(any_pb2.Any(), "bar")
+ with pytest.raises(ValueError):
+ protobuf_helpers.field_mask(any_pb2.Any(), operations_pb2.Operation())
+
+
+def test_field_mask_equal_values():
+ assert protobuf_helpers.field_mask(None, None).paths == []
+
+ original = struct_pb2.Value(number_value=1.0)
+ modified = struct_pb2.Value(number_value=1.0)
+ assert protobuf_helpers.field_mask(original, modified).paths == []
+
+ original = color_pb2.Color(alpha=wrappers_pb2.FloatValue(value=1.0))
+ modified = color_pb2.Color(alpha=wrappers_pb2.FloatValue(value=1.0))
+ assert protobuf_helpers.field_mask(original, modified).paths == []
+
+ original = struct_pb2.ListValue(values=[struct_pb2.Value(number_value=1.0)])
+ modified = struct_pb2.ListValue(values=[struct_pb2.Value(number_value=1.0)])
+ assert protobuf_helpers.field_mask(original, modified).paths == []
+
+ original = struct_pb2.Struct(fields={"bar": struct_pb2.Value(number_value=1.0)})
+ modified = struct_pb2.Struct(fields={"bar": struct_pb2.Value(number_value=1.0)})
+ assert protobuf_helpers.field_mask(original, modified).paths == []
+
+
+def test_field_mask_zero_values():
+ # Singular Values
+ original = color_pb2.Color(red=0.0)
+ modified = None
+ assert protobuf_helpers.field_mask(original, modified).paths == []
+
+ original = None
+ modified = color_pb2.Color(red=0.0)
+ assert protobuf_helpers.field_mask(original, modified).paths == []
+
+ # Repeated Values
+ original = struct_pb2.ListValue(values=[])
+ modified = None
+ assert protobuf_helpers.field_mask(original, modified).paths == []
+
+ original = None
+ modified = struct_pb2.ListValue(values=[])
+ assert protobuf_helpers.field_mask(original, modified).paths == []
+
+ # Maps
+ original = struct_pb2.Struct(fields={})
+ modified = None
+ assert protobuf_helpers.field_mask(original, modified).paths == []
+
+ original = None
+ modified = struct_pb2.Struct(fields={})
+ assert protobuf_helpers.field_mask(original, modified).paths == []
+
+ # Oneofs
+ original = struct_pb2.Value(number_value=0.0)
+ modified = None
+ assert protobuf_helpers.field_mask(original, modified).paths == []
+
+ original = None
+ modified = struct_pb2.Value(number_value=0.0)
+ assert protobuf_helpers.field_mask(original, modified).paths == []
+
+
+def test_field_mask_singular_field_diffs():
+ original = type_pb2.Type(name="name")
+ modified = type_pb2.Type()
+ assert protobuf_helpers.field_mask(original, modified).paths == ["name"]
+
+ original = type_pb2.Type(name="name")
+ modified = type_pb2.Type()
+ assert protobuf_helpers.field_mask(original, modified).paths == ["name"]
+
+ original = None
+ modified = type_pb2.Type(name="name")
+ assert protobuf_helpers.field_mask(original, modified).paths == ["name"]
+
+ original = type_pb2.Type(name="name")
+ modified = None
+ assert protobuf_helpers.field_mask(original, modified).paths == ["name"]
+
+
+def test_field_mask_message_diffs():
+ original = type_pb2.Type()
+ modified = type_pb2.Type(
+ source_context=source_context_pb2.SourceContext(file_name="name")
+ )
+ assert protobuf_helpers.field_mask(original, modified).paths == [
+ "source_context.file_name"
+ ]
+
+ original = type_pb2.Type(
+ source_context=source_context_pb2.SourceContext(file_name="name")
+ )
+ modified = type_pb2.Type()
+ assert protobuf_helpers.field_mask(original, modified).paths == ["source_context"]
+
+ original = type_pb2.Type(
+ source_context=source_context_pb2.SourceContext(file_name="name")
+ )
+ modified = type_pb2.Type(
+ source_context=source_context_pb2.SourceContext(file_name="other_name")
+ )
+ assert protobuf_helpers.field_mask(original, modified).paths == [
+ "source_context.file_name"
+ ]
+
+ original = None
+ modified = type_pb2.Type(
+ source_context=source_context_pb2.SourceContext(file_name="name")
+ )
+ assert protobuf_helpers.field_mask(original, modified).paths == [
+ "source_context.file_name"
+ ]
+
+ original = type_pb2.Type(
+ source_context=source_context_pb2.SourceContext(file_name="name")
+ )
+ modified = None
+ assert protobuf_helpers.field_mask(original, modified).paths == ["source_context"]
+
+
+def test_field_mask_wrapper_type_diffs():
+ original = color_pb2.Color()
+ modified = color_pb2.Color(alpha=wrappers_pb2.FloatValue(value=1.0))
+ assert protobuf_helpers.field_mask(original, modified).paths == ["alpha"]
+
+ original = color_pb2.Color(alpha=wrappers_pb2.FloatValue(value=1.0))
+ modified = color_pb2.Color()
+ assert protobuf_helpers.field_mask(original, modified).paths == ["alpha"]
+
+ original = color_pb2.Color(alpha=wrappers_pb2.FloatValue(value=1.0))
+ modified = color_pb2.Color(alpha=wrappers_pb2.FloatValue(value=2.0))
+ assert protobuf_helpers.field_mask(original, modified).paths == ["alpha"]
+
+ original = None
+ modified = color_pb2.Color(alpha=wrappers_pb2.FloatValue(value=2.0))
+ assert protobuf_helpers.field_mask(original, modified).paths == ["alpha"]
+
+ original = color_pb2.Color(alpha=wrappers_pb2.FloatValue(value=1.0))
+ modified = None
+ assert protobuf_helpers.field_mask(original, modified).paths == ["alpha"]
+
+
+def test_field_mask_repeated_diffs():
+ original = struct_pb2.ListValue()
+ modified = struct_pb2.ListValue(
+ values=[struct_pb2.Value(number_value=1.0), struct_pb2.Value(number_value=2.0)]
+ )
+ assert protobuf_helpers.field_mask(original, modified).paths == ["values"]
+
+ original = struct_pb2.ListValue(
+ values=[struct_pb2.Value(number_value=1.0), struct_pb2.Value(number_value=2.0)]
+ )
+ modified = struct_pb2.ListValue()
+ assert protobuf_helpers.field_mask(original, modified).paths == ["values"]
+
+ original = None
+ modified = struct_pb2.ListValue(
+ values=[struct_pb2.Value(number_value=1.0), struct_pb2.Value(number_value=2.0)]
+ )
+ assert protobuf_helpers.field_mask(original, modified).paths == ["values"]
+
+ original = struct_pb2.ListValue(
+ values=[struct_pb2.Value(number_value=1.0), struct_pb2.Value(number_value=2.0)]
+ )
+ modified = None
+ assert protobuf_helpers.field_mask(original, modified).paths == ["values"]
+
+ original = struct_pb2.ListValue(
+ values=[struct_pb2.Value(number_value=1.0), struct_pb2.Value(number_value=2.0)]
+ )
+ modified = struct_pb2.ListValue(
+ values=[struct_pb2.Value(number_value=2.0), struct_pb2.Value(number_value=1.0)]
+ )
+ assert protobuf_helpers.field_mask(original, modified).paths == ["values"]
+
+
+def test_field_mask_map_diffs():
+ original = struct_pb2.Struct()
+ modified = struct_pb2.Struct(fields={"foo": struct_pb2.Value(number_value=1.0)})
+ assert protobuf_helpers.field_mask(original, modified).paths == ["fields"]
+
+ original = struct_pb2.Struct(fields={"foo": struct_pb2.Value(number_value=1.0)})
+ modified = struct_pb2.Struct()
+ assert protobuf_helpers.field_mask(original, modified).paths == ["fields"]
+
+ original = None
+ modified = struct_pb2.Struct(fields={"foo": struct_pb2.Value(number_value=1.0)})
+ assert protobuf_helpers.field_mask(original, modified).paths == ["fields"]
+
+ original = struct_pb2.Struct(fields={"foo": struct_pb2.Value(number_value=1.0)})
+ modified = None
+ assert protobuf_helpers.field_mask(original, modified).paths == ["fields"]
+
+ original = struct_pb2.Struct(fields={"foo": struct_pb2.Value(number_value=1.0)})
+ modified = struct_pb2.Struct(fields={"foo": struct_pb2.Value(number_value=2.0)})
+ assert protobuf_helpers.field_mask(original, modified).paths == ["fields"]
+
+ original = struct_pb2.Struct(fields={"foo": struct_pb2.Value(number_value=1.0)})
+ modified = struct_pb2.Struct(fields={"bar": struct_pb2.Value(number_value=1.0)})
+ assert protobuf_helpers.field_mask(original, modified).paths == ["fields"]
+
+
+def test_field_mask_different_level_diffs():
+ original = color_pb2.Color(alpha=wrappers_pb2.FloatValue(value=1.0))
+ modified = color_pb2.Color(alpha=wrappers_pb2.FloatValue(value=2.0), red=1.0)
+ assert sorted(protobuf_helpers.field_mask(original, modified).paths) == [
+ "alpha",
+ "red",
+ ]
diff --git a/api_core/tests/unit/test_retry.py b/api_core/tests/unit/test_retry.py
new file mode 100644
index 000000000000..5b5e59b8eb40
--- /dev/null
+++ b/api_core/tests/unit/test_retry.py
@@ -0,0 +1,312 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import datetime
+import itertools
+import re
+
+import mock
+import pytest
+
+from google.api_core import exceptions
+from google.api_core import retry
+
+
+def test_if_exception_type():
+ predicate = retry.if_exception_type(ValueError)
+
+ assert predicate(ValueError())
+ assert not predicate(TypeError())
+
+
+def test_if_exception_type_multiple():
+ predicate = retry.if_exception_type(ValueError, TypeError)
+
+ assert predicate(ValueError())
+ assert predicate(TypeError())
+ assert not predicate(RuntimeError())
+
+
+def test_if_transient_error():
+ assert retry.if_transient_error(exceptions.InternalServerError(""))
+ assert retry.if_transient_error(exceptions.TooManyRequests(""))
+ assert retry.if_transient_error(exceptions.ServiceUnavailable(""))
+ assert not retry.if_transient_error(exceptions.InvalidArgument(""))
+
+
+# Make uniform return half of its maximum, which will be the calculated
+# sleep time.
+@mock.patch("random.uniform", autospec=True, side_effect=lambda m, n: n / 2.0)
+def test_exponential_sleep_generator_base_2(uniform):
+ gen = retry.exponential_sleep_generator(1, 60, multiplier=2)
+
+ result = list(itertools.islice(gen, 8))
+ assert result == [1, 2, 4, 8, 16, 32, 60, 60]
+
+
+@mock.patch("time.sleep", autospec=True)
+@mock.patch(
+ "google.api_core.datetime_helpers.utcnow",
+ return_value=datetime.datetime.min,
+ autospec=True,
+)
+def test_retry_target_success(utcnow, sleep):
+ predicate = retry.if_exception_type(ValueError)
+ call_count = [0]
+
+ def target():
+ call_count[0] += 1
+ if call_count[0] < 3:
+ raise ValueError()
+ return 42
+
+ result = retry.retry_target(target, predicate, range(10), None)
+
+ assert result == 42
+ assert call_count[0] == 3
+ sleep.assert_has_calls([mock.call(0), mock.call(1)])
+
+
+@mock.patch("time.sleep", autospec=True)
+@mock.patch(
+ "google.api_core.datetime_helpers.utcnow",
+ return_value=datetime.datetime.min,
+ autospec=True,
+)
+def test_retry_target_w_on_error(utcnow, sleep):
+ predicate = retry.if_exception_type(ValueError)
+ call_count = {"target": 0}
+ to_raise = ValueError()
+
+ def target():
+ call_count["target"] += 1
+ if call_count["target"] < 3:
+ raise to_raise
+ return 42
+
+ on_error = mock.Mock()
+
+ result = retry.retry_target(target, predicate, range(10), None, on_error=on_error)
+
+ assert result == 42
+ assert call_count["target"] == 3
+
+ on_error.assert_has_calls([mock.call(to_raise), mock.call(to_raise)])
+ sleep.assert_has_calls([mock.call(0), mock.call(1)])
+
+
+@mock.patch("time.sleep", autospec=True)
+@mock.patch(
+ "google.api_core.datetime_helpers.utcnow",
+ return_value=datetime.datetime.min,
+ autospec=True,
+)
+def test_retry_target_non_retryable_error(utcnow, sleep):
+ predicate = retry.if_exception_type(ValueError)
+ exception = TypeError()
+ target = mock.Mock(side_effect=exception)
+
+ with pytest.raises(TypeError) as exc_info:
+ retry.retry_target(target, predicate, range(10), None)
+
+ assert exc_info.value == exception
+ sleep.assert_not_called()
+
+
+@mock.patch("time.sleep", autospec=True)
+@mock.patch("google.api_core.datetime_helpers.utcnow", autospec=True)
+def test_retry_target_deadline_exceeded(utcnow, sleep):
+ predicate = retry.if_exception_type(ValueError)
+ exception = ValueError("meep")
+ target = mock.Mock(side_effect=exception)
+ # Setup the timeline so that the first call takes 5 seconds but the second
+ # call takes 6, which puts the retry over the deadline.
+ utcnow.side_effect = [
+ # The first call to utcnow establishes the start of the timeline.
+ datetime.datetime.min,
+ datetime.datetime.min + datetime.timedelta(seconds=5),
+ datetime.datetime.min + datetime.timedelta(seconds=11),
+ ]
+
+ with pytest.raises(exceptions.RetryError) as exc_info:
+ retry.retry_target(target, predicate, range(10), deadline=10)
+
+ assert exc_info.value.cause == exception
+ assert exc_info.match("Deadline of 10.0s exceeded")
+ assert exc_info.match("last exception: meep")
+ assert target.call_count == 2
+
+
+def test_retry_target_bad_sleep_generator():
+ with pytest.raises(ValueError, match="Sleep generator"):
+ retry.retry_target(mock.sentinel.target, mock.sentinel.predicate, [], None)
+
+
+class TestRetry(object):
+ def test_constructor_defaults(self):
+ retry_ = retry.Retry()
+ assert retry_._predicate == retry.if_transient_error
+ assert retry_._initial == 1
+ assert retry_._maximum == 60
+ assert retry_._multiplier == 2
+ assert retry_._deadline == 120
+ assert retry_._on_error is None
+
+ def test_constructor_options(self):
+ _some_function = mock.Mock()
+
+ retry_ = retry.Retry(
+ predicate=mock.sentinel.predicate,
+ initial=1,
+ maximum=2,
+ multiplier=3,
+ deadline=4,
+ on_error=_some_function,
+ )
+ assert retry_._predicate == mock.sentinel.predicate
+ assert retry_._initial == 1
+ assert retry_._maximum == 2
+ assert retry_._multiplier == 3
+ assert retry_._deadline == 4
+ assert retry_._on_error is _some_function
+
+ def test_with_deadline(self):
+ retry_ = retry.Retry()
+ new_retry = retry_.with_deadline(42)
+ assert retry_ is not new_retry
+ assert new_retry._deadline == 42
+
+ def test_with_predicate(self):
+ retry_ = retry.Retry()
+ new_retry = retry_.with_predicate(mock.sentinel.predicate)
+ assert retry_ is not new_retry
+ assert new_retry._predicate == mock.sentinel.predicate
+
+ def test_with_delay_noop(self):
+ retry_ = retry.Retry()
+ new_retry = retry_.with_delay()
+ assert retry_ is not new_retry
+ assert new_retry._initial == retry_._initial
+ assert new_retry._maximum == retry_._maximum
+ assert new_retry._multiplier == retry_._multiplier
+
+ def test_with_delay(self):
+ retry_ = retry.Retry()
+ new_retry = retry_.with_delay(initial=1, maximum=2, multiplier=3)
+ assert retry_ is not new_retry
+ assert new_retry._initial == 1
+ assert new_retry._maximum == 2
+ assert new_retry._multiplier == 3
+
+ def test___str__(self):
+ retry_ = retry.Retry()
+ assert re.match(
+ (
+ r", "
+ r"initial=1.0, maximum=60.0, multiplier=2.0, deadline=120.0, "
+ r"on_error=None>"
+ ),
+ str(retry_),
+ )
+
+ @mock.patch("time.sleep", autospec=True)
+ def test___call___and_execute_success(self, sleep):
+ retry_ = retry.Retry()
+ target = mock.Mock(spec=["__call__"], return_value=42)
+ # __name__ is needed by functools.partial.
+ target.__name__ = "target"
+
+ decorated = retry_(target)
+ target.assert_not_called()
+
+ result = decorated("meep")
+
+ assert result == 42
+ target.assert_called_once_with("meep")
+ sleep.assert_not_called()
+
+ # Make uniform return half of its maximum, which is the calculated sleep time.
+ @mock.patch("random.uniform", autospec=True, side_effect=lambda m, n: n / 2.0)
+ @mock.patch("time.sleep", autospec=True)
+ def test___call___and_execute_retry(self, sleep, uniform):
+
+ on_error = mock.Mock(spec=["__call__"], side_effect=[None])
+ retry_ = retry.Retry(predicate=retry.if_exception_type(ValueError))
+
+ target = mock.Mock(spec=["__call__"], side_effect=[ValueError(), 42])
+ # __name__ is needed by functools.partial.
+ target.__name__ = "target"
+
+ decorated = retry_(target, on_error=on_error)
+ target.assert_not_called()
+
+ result = decorated("meep")
+
+ assert result == 42
+ assert target.call_count == 2
+ target.assert_has_calls([mock.call("meep"), mock.call("meep")])
+ sleep.assert_called_once_with(retry_._initial)
+ assert on_error.call_count == 1
+
+ @mock.patch("time.sleep", autospec=True)
+ def test___init___without_retry_executed(self, sleep):
+ _some_function = mock.Mock()
+
+ retry_ = retry.Retry(
+ predicate=retry.if_exception_type(ValueError), on_error=_some_function
+ )
+ # check the proper creation of the class
+ assert retry_._on_error is _some_function
+
+ target = mock.Mock(spec=["__call__"], side_effect=[42])
+ # __name__ is needed by functools.partial.
+ target.__name__ = "target"
+
+ wrapped = retry_(target)
+
+ result = wrapped("meep")
+
+ assert result == 42
+ target.assert_called_once_with("meep")
+ sleep.assert_not_called()
+ _some_function.assert_not_called()
+
+ # Make uniform return half of its maximum, which is the calculated sleep time.
+ @mock.patch("random.uniform", autospec=True, side_effect=lambda m, n: n / 2.0)
+ @mock.patch("time.sleep", autospec=True)
+ def test___init___when_retry_is_executed(self, sleep, uniform):
+ _some_function = mock.Mock()
+
+ retry_ = retry.Retry(
+ predicate=retry.if_exception_type(ValueError), on_error=_some_function
+ )
+ # check the proper creation of the class
+ assert retry_._on_error is _some_function
+
+ target = mock.Mock(
+ spec=["__call__"], side_effect=[ValueError(), ValueError(), 42]
+ )
+ # __name__ is needed by functools.partial.
+ target.__name__ = "target"
+
+ wrapped = retry_(target)
+ target.assert_not_called()
+
+ result = wrapped("meep")
+
+ assert result == 42
+ assert target.call_count == 3
+ assert _some_function.call_count == 2
+ target.assert_has_calls([mock.call("meep"), mock.call("meep")])
+ sleep.assert_any_call(retry_._initial)
diff --git a/api_core/tests/unit/test_timeout.py b/api_core/tests/unit/test_timeout.py
new file mode 100644
index 000000000000..30d624e2c0c4
--- /dev/null
+++ b/api_core/tests/unit/test_timeout.py
@@ -0,0 +1,129 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import datetime
+import itertools
+
+import mock
+
+from google.api_core import timeout
+
+
+def test__exponential_timeout_generator_base_2():
+ gen = timeout._exponential_timeout_generator(1.0, 60.0, 2.0, deadline=None)
+
+ result = list(itertools.islice(gen, 8))
+ assert result == [1, 2, 4, 8, 16, 32, 60, 60]
+
+
+@mock.patch("google.api_core.datetime_helpers.utcnow", autospec=True)
+def test__exponential_timeout_generator_base_deadline(utcnow):
+ # Make each successive call to utcnow() advance one second.
+ utcnow.side_effect = [
+ datetime.datetime.min + datetime.timedelta(seconds=n) for n in range(15)
+ ]
+
+ gen = timeout._exponential_timeout_generator(1.0, 60.0, 2.0, deadline=30.0)
+
+ result = list(itertools.islice(gen, 14))
+ # Should grow until the cumulative time is > 30s, then start decreasing as
+ # the cumulative time approaches 60s.
+ assert result == [1, 2, 4, 8, 16, 24, 23, 22, 21, 20, 19, 18, 17, 16]
+
+
+class TestConstantTimeout(object):
+ def test_constructor(self):
+ timeout_ = timeout.ConstantTimeout()
+ assert timeout_._timeout is None
+
+ def test_constructor_args(self):
+ timeout_ = timeout.ConstantTimeout(42.0)
+ assert timeout_._timeout == 42.0
+
+ def test___str__(self):
+ timeout_ = timeout.ConstantTimeout(1)
+ assert str(timeout_) == ""
+
+ def test_apply(self):
+ target = mock.Mock(spec=["__call__", "__name__"], __name__="target")
+ timeout_ = timeout.ConstantTimeout(42.0)
+ wrapped = timeout_(target)
+
+ wrapped()
+
+ target.assert_called_once_with(timeout=42.0)
+
+ def test_apply_passthrough(self):
+ target = mock.Mock(spec=["__call__", "__name__"], __name__="target")
+ timeout_ = timeout.ConstantTimeout(42.0)
+ wrapped = timeout_(target)
+
+ wrapped(1, 2, meep="moop")
+
+ target.assert_called_once_with(1, 2, meep="moop", timeout=42.0)
+
+
+class TestExponentialTimeout(object):
+ def test_constructor(self):
+ timeout_ = timeout.ExponentialTimeout()
+ assert timeout_._initial == timeout._DEFAULT_INITIAL_TIMEOUT
+ assert timeout_._maximum == timeout._DEFAULT_MAXIMUM_TIMEOUT
+ assert timeout_._multiplier == timeout._DEFAULT_TIMEOUT_MULTIPLIER
+ assert timeout_._deadline == timeout._DEFAULT_DEADLINE
+
+ def test_constructor_args(self):
+ timeout_ = timeout.ExponentialTimeout(1, 2, 3, 4)
+ assert timeout_._initial == 1
+ assert timeout_._maximum == 2
+ assert timeout_._multiplier == 3
+ assert timeout_._deadline == 4
+
+ def test_with_timeout(self):
+ original_timeout = timeout.ExponentialTimeout()
+ timeout_ = original_timeout.with_deadline(42)
+ assert original_timeout is not timeout_
+ assert timeout_._initial == timeout._DEFAULT_INITIAL_TIMEOUT
+ assert timeout_._maximum == timeout._DEFAULT_MAXIMUM_TIMEOUT
+ assert timeout_._multiplier == timeout._DEFAULT_TIMEOUT_MULTIPLIER
+ assert timeout_._deadline == 42
+
+ def test___str__(self):
+ timeout_ = timeout.ExponentialTimeout(1, 2, 3, 4)
+ assert str(timeout_) == (
+ ""
+ )
+
+ def test_apply(self):
+ target = mock.Mock(spec=["__call__", "__name__"], __name__="target")
+ timeout_ = timeout.ExponentialTimeout(1, 10, 2)
+ wrapped = timeout_(target)
+
+ wrapped()
+ target.assert_called_with(timeout=1)
+
+ wrapped()
+ target.assert_called_with(timeout=2)
+
+ wrapped()
+ target.assert_called_with(timeout=4)
+
+ def test_apply_passthrough(self):
+ target = mock.Mock(spec=["__call__", "__name__"], __name__="target")
+ timeout_ = timeout.ExponentialTimeout(42.0, 100, 2)
+ wrapped = timeout_(target)
+
+ wrapped(1, 2, meep="moop")
+
+ target.assert_called_once_with(1, 2, meep="moop", timeout=42.0)
diff --git a/appveyor.yml b/appveyor.yml
index e552cb428de5..fcb43d498ad1 100644
--- a/appveyor.yml
+++ b/appveyor.yml
@@ -1,102 +1,75 @@
version: 1.0.{build}.{branch}
-environment:
- global:
- # SDK v7.0 MSVC Express 2008's SetEnv.cmd script will fail if the
- # /E:ON and /V:ON options are not enabled in the batch script intepreter
- # See: http://stackoverflow.com/a/13751649/163740
- CMD_IN_ENV: "cmd /E:ON /V:ON /C .\\appveyor\\run_with_env.cmd"
+build: off
+environment:
matrix:
# Pre-installed Python versions, which Appveyor may upgrade to
# a later point release.
# See: http://www.appveyor.com/docs/installed-software#python
- # Python 2.7.11 is the latest Python 2.7 with a Windows installer
- # Python 2.7.11 is the overall latest
- # https://www.python.org/ftp/python/2.7.11/
- - PYTHON: "C:\\Python27"
- PYTHON_VERSION: "2.7.11"
- PYTHON_ARCH: "32"
- TOX_ENV: "py27"
-
- - PYTHON: "C:\\Python27-x64"
- PYTHON_VERSION: "2.7.11"
- PYTHON_ARCH: "64"
- TOX_ENV: "py27"
-
- # Python 3.4.4 is the latest Python 3.4 with a Windows installer
- # Python 3.4.4 is the overall latest
- # https://www.python.org/ftp/python/3.4.4/
- - PYTHON: "C:\\Python34"
- PYTHON_VERSION: "3.4.4"
- PYTHON_ARCH: "32"
- TOX_ENV: "py34"
-
- - PYTHON: "C:\\Python34-x64"
- PYTHON_VERSION: "3.4.4"
- PYTHON_ARCH: "64"
- TOX_ENV: "py34"
-
- # Python 3.5.1 is the latest Python 3.5 with a Windows installer
- # Python 3.5.1 is the overall latest
- # https://www.python.org/ftp/python/3.5.1/
- - PYTHON: "C:\\Python35"
- PYTHON_VERSION: "3.5.1"
- PYTHON_ARCH: "32"
- TOX_ENV: "py35"
+ # We test on 32-bit Python 2.7, and 64-bit Python 3.x.
+ # This is because it takes too long to serially go through six to eight
+ # iterations.
+ - PYTHON: "C:\\Python27"
- PYTHON: "C:\\Python35-x64"
- PYTHON_VERSION: "3.5.1"
- PYTHON_ARCH: "64"
- TOX_ENV: "py35"
install:
- - ECHO "Filesystem root:"
- - ps: "ls \"C:/\""
-
- - ECHO "Installed SDKs:"
- - ps: "ls \"C:/Program Files/Microsoft SDKs/Windows\""
+ - cmd: echo "Filesystem root:"
+ - dir C:\
- # Install Python (from the official .msi of http://python.org) and pip when
- # not already installed.
- - ps: if (-not(Test-Path($env:PYTHON))) { & appveyor\install.ps1 }
+ - echo "Installed SDKs:"
+ - dir "C:\Program Files\Microsoft SDKs\Windows"
- # Prepend newly installed Python to the PATH of this build (this cannot be
+ # Prepend "current" Python to the PATH of this build (this cannot be
# done from inside the powershell script as it would require to restart
# the parent CMD process).
- - "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%"
+ - SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%
- # Check that we have the expected version and architecture for Python
- - "python --version"
- - "python -c \"import struct; print(struct.calcsize('P') * 8)\""
+ # https://stackoverflow.com/a/4509885/1068170
+ # Set `PYTHON_ARCH` (either 32 or 64)
+ - set PY_CMD=python -c "import platform; print(platform.architecture()[0][:2])"
+ - for /f "tokens=1 delims=" %%i in ('%PY_CMD%') do set PYTHON_ARCH=%%i
+ # Set `PYTHON_VERSION`
+ - set PY_CMD=python -c "import platform; print(platform.python_version())"
+ - for /f "tokens=1 delims=" %%i in ('%PY_CMD%') do set PYTHON_VERSION=%%i
- # Upgrade to the latest version of pip to avoid it displaying warnings
- # about it being out of date.
- - "pip install --disable-pip-version-check --user --upgrade pip"
+ # Check that we have the expected version and architecture for Python
+ - python appveyor\env_info.py
- # Install the build dependencies of the project. If some dependencies contain
- # compiled extensions and are not provided as pre-built wheel packages,
- # pip will build them from source using the MSVC compiler matching the
- # target Python version and architecture
- - "%CMD_IN_ENV% pip install -r appveyor/requirements.txt"
+ # Upgrade to the latest version of pip.
+ - python -m pip install --upgrade pip setuptools
-build_script:
- # Build the compiled extension
- - "%CMD_IN_ENV% python setup.py build"
+ # Install the build dependencies of the project (assumes all extensions
+ # are pure Python or have wheels, so they don't need to be built).
+ - python -m pip install -r appveyor\requirements.txt
test_script:
- - "%CMD_IN_ENV% pip list"
# Run the project tests
- - "%CMD_IN_ENV% tox -e %TOX_ENV%"
-
-after_test:
- # If tests are successful, create binary packages for the project.
- - "%CMD_IN_ENV% python setup.py bdist_wininst"
- - "%CMD_IN_ENV% python setup.py bdist_msi"
- - ps: "ls dist"
-
-artifacts:
- # Archive the generated packages in the ci.appveyor.com build report.
- - path: dist\*
+ - "%PYTHON%\\Scripts\\nox.exe -f api_core\\nox.py -s default"
+ - "%PYTHON%\\Scripts\\nox.exe -f asset\\nox.py -s default"
+ - "%PYTHON%\\Scripts\\nox.exe -f bigquery\\nox.py -s default"
+ - "%PYTHON%\\Scripts\\nox.exe -f bigquery_datatransfer\\nox.py -s default"
+ - "%PYTHON%\\Scripts\\nox.exe -f bigtable\\nox.py -s default"
+ - "%PYTHON%\\Scripts\\nox.exe -f container\\nox.py -s default"
+ - "%PYTHON%\\Scripts\\nox.exe -f core\\nox.py -s default"
+ - "%PYTHON%\\Scripts\\nox.exe -f dataproc\\nox.py -s default"
+ - "%PYTHON%\\Scripts\\nox.exe -f datastore\\nox.py -s default"
+ - "%PYTHON%\\Scripts\\nox.exe -f dns\\nox.py -s default"
+ - "%PYTHON%\\Scripts\\nox.exe -f error_reporting\\nox.py -s default"
+ - "%PYTHON%\\Scripts\\nox.exe -f firestore\\nox.py -s default"
+ - "%PYTHON%\\Scripts\\nox.exe -f language\\nox.py -s default"
+ - "%PYTHON%\\Scripts\\nox.exe -f logging\\nox.py -s default"
+ - "%PYTHON%\\Scripts\\nox.exe -f monitoring\\nox.py -s default"
+ - "%PYTHON%\\Scripts\\nox.exe -f pubsub\\nox.py -s default"
+ - "%PYTHON%\\Scripts\\nox.exe -f resource_manager\\nox.py -s default"
+ - "%PYTHON%\\Scripts\\nox.exe -f runtimeconfig\\nox.py -s default"
+ - "%PYTHON%\\Scripts\\nox.exe -f spanner\\nox.py -s default"
+ - "%PYTHON%\\Scripts\\nox.exe -f speech\\nox.py -s default"
+ - "%PYTHON%\\Scripts\\nox.exe -f storage\\nox.py -s default"
+ - "%PYTHON%\\Scripts\\nox.exe -f trace\\nox.py -s default"
+ - "%PYTHON%\\Scripts\\nox.exe -f translate\\nox.py -s default"
+ - "%PYTHON%\\Scripts\\nox.exe -f videointelligence\\nox.py -s default"
+ - "%PYTHON%\\Scripts\\nox.exe -f vision\\nox.py -s default"
diff --git a/appveyor/env_info.py b/appveyor/env_info.py
new file mode 100644
index 000000000000..c27b686dd199
--- /dev/null
+++ b/appveyor/env_info.py
@@ -0,0 +1,48 @@
+# Copyright 2017 Google LLC All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Get environment information."""
+
+from __future__ import print_function
+
+import os
+import struct
+import sys
+
+
+def env_var(var_name):
+ value = os.environ.get(var_name)
+ print("os.environ[{!r}]: {}".format(var_name, value))
+
+
+def main():
+ print("os.name: {}".format(os.name))
+ env_var("PYTHON_ARCH")
+ env_var("PYTHON_VERSION")
+ print("sys.platform: {}".format(sys.platform))
+
+ if sys.maxsize == 2 ** 63 - 1:
+ print("sys.maxsize: 2^(63) - 1")
+ elif sys.maxsize == 2 ** 31 - 1:
+ print("sys.maxsize: 2^(31) - 1")
+ else:
+ print("sys.maxsize: {}".format(sys.maxsize))
+
+ print("sys.version:\n{}".format(sys.version))
+ bitness = struct.calcsize("P") * 8
+ print("struct.calcsize('P') * 8: {}".format(bitness))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/appveyor/install.ps1 b/appveyor/install.ps1
deleted file mode 100644
index 160ba55c0737..000000000000
--- a/appveyor/install.ps1
+++ /dev/null
@@ -1,229 +0,0 @@
-# Sample script to install Python and pip under Windows
-# Authors: Olivier Grisel, Jonathan Helmus, Kyle Kastner, and Alex Willmer
-# License: CC0 1.0 Universal: http://creativecommons.org/publicdomain/zero/1.0/
-
-$MINICONDA_URL = "http://repo.continuum.io/miniconda/"
-$BASE_URL = "https://www.python.org/ftp/python/"
-$GET_PIP_URL = "https://bootstrap.pypa.io/get-pip.py"
-$GET_PIP_PATH = "C:\get-pip.py"
-
-$PYTHON_PRERELEASE_REGEX = @"
-(?x)
-(?\d+)
-\.
-(?\d+)
-\.
-(?\d+)
-(?[a-z]{1,2}\d+)
-"@
-
-
-function Download ($filename, $url) {
- $webclient = New-Object System.Net.WebClient
-
- $basedir = $pwd.Path + "\"
- $filepath = $basedir + $filename
- if (Test-Path $filename) {
- Write-Host "Reusing" $filepath
- return $filepath
- }
-
- # Download and retry up to 3 times in case of network transient errors.
- Write-Host "Downloading" $filename "from" $url
- $retry_attempts = 2
- for ($i = 0; $i -lt $retry_attempts; $i++) {
- try {
- $webclient.DownloadFile($url, $filepath)
- break
- }
- Catch [Exception]{
- Start-Sleep 1
- }
- }
- if (Test-Path $filepath) {
- Write-Host "File saved at" $filepath
- } else {
- # Retry once to get the error message if any at the last try
- $webclient.DownloadFile($url, $filepath)
- }
- return $filepath
-}
-
-
-function ParsePythonVersion ($python_version) {
- if ($python_version -match $PYTHON_PRERELEASE_REGEX) {
- return ([int]$matches.major, [int]$matches.minor, [int]$matches.micro,
- $matches.prerelease)
- }
- $version_obj = [version]$python_version
- return ($version_obj.major, $version_obj.minor, $version_obj.build, "")
-}
-
-
-function DownloadPython ($python_version, $platform_suffix) {
- $major, $minor, $micro, $prerelease = ParsePythonVersion $python_version
-
- if (($major -le 2 -and $micro -eq 0) `
- -or ($major -eq 3 -and $minor -le 2 -and $micro -eq 0) `
- ) {
- $dir = "$major.$minor"
- $python_version = "$major.$minor$prerelease"
- } else {
- $dir = "$major.$minor.$micro"
- }
-
- if ($prerelease) {
- if (($major -le 2) `
- -or ($major -eq 3 -and $minor -eq 1) `
- -or ($major -eq 3 -and $minor -eq 2) `
- -or ($major -eq 3 -and $minor -eq 3) `
- ) {
- $dir = "$dir/prev"
- }
- }
-
- if (($major -le 2) -or ($major -le 3 -and $minor -le 4)) {
- $ext = "msi"
- if ($platform_suffix) {
- $platform_suffix = ".$platform_suffix"
- }
- } else {
- $ext = "exe"
- if ($platform_suffix) {
- $platform_suffix = "-$platform_suffix"
- }
- }
-
- $filename = "python-$python_version$platform_suffix.$ext"
- $url = "$BASE_URL$dir/$filename"
- $filepath = Download $filename $url
- return $filepath
-}
-
-
-function InstallPython ($python_version, $architecture, $python_home) {
- Write-Host "Installing Python" $python_version "for" $architecture "bit architecture to" $python_home
- if (Test-Path $python_home) {
- Write-Host $python_home "already exists, skipping."
- return $false
- }
- if ($architecture -eq "32") {
- $platform_suffix = ""
- } else {
- $platform_suffix = "amd64"
- }
- $installer_path = DownloadPython $python_version $platform_suffix
- $installer_ext = [System.IO.Path]::GetExtension($installer_path)
- Write-Host "Installing $installer_path to $python_home"
- $install_log = $python_home + ".log"
- if ($installer_ext -eq '.msi') {
- InstallPythonMSI $installer_path $python_home $install_log
- } else {
- InstallPythonEXE $installer_path $python_home $install_log
- }
- if (Test-Path $python_home) {
- Write-Host "Python $python_version ($architecture) installation complete"
- } else {
- Write-Host "Failed to install Python in $python_home"
- Get-Content -Path $install_log
- Exit 1
- }
-}
-
-
-function InstallPythonEXE ($exepath, $python_home, $install_log) {
- $install_args = "/quiet InstallAllUsers=1 TargetDir=$python_home"
- RunCommand $exepath $install_args
-}
-
-
-function InstallPythonMSI ($msipath, $python_home, $install_log) {
- $install_args = "/qn /log $install_log /i $msipath TARGETDIR=$python_home"
- $uninstall_args = "/qn /x $msipath"
- RunCommand "msiexec.exe" $install_args
- if (-not(Test-Path $python_home)) {
- Write-Host "Python seems to be installed else-where, reinstalling."
- RunCommand "msiexec.exe" $uninstall_args
- RunCommand "msiexec.exe" $install_args
- }
-}
-
-function RunCommand ($command, $command_args) {
- Write-Host $command $command_args
- Start-Process -FilePath $command -ArgumentList $command_args -Wait -Passthru
-}
-
-
-function InstallPip ($python_home) {
- $pip_path = $python_home + "\Scripts\pip.exe"
- $python_path = $python_home + "\python.exe"
- if (-not(Test-Path $pip_path)) {
- Write-Host "Installing pip..."
- $webclient = New-Object System.Net.WebClient
- $webclient.DownloadFile($GET_PIP_URL, $GET_PIP_PATH)
- Write-Host "Executing:" $python_path $GET_PIP_PATH
- & $python_path $GET_PIP_PATH
- } else {
- Write-Host "pip already installed."
- }
-}
-
-
-function DownloadMiniconda ($python_version, $platform_suffix) {
- if ($python_version -eq "3.4") {
- $filename = "Miniconda3-3.5.5-Windows-" + $platform_suffix + ".exe"
- } else {
- $filename = "Miniconda-3.5.5-Windows-" + $platform_suffix + ".exe"
- }
- $url = $MINICONDA_URL + $filename
- $filepath = Download $filename $url
- return $filepath
-}
-
-
-function InstallMiniconda ($python_version, $architecture, $python_home) {
- Write-Host "Installing Python" $python_version "for" $architecture "bit architecture to" $python_home
- if (Test-Path $python_home) {
- Write-Host $python_home "already exists, skipping."
- return $false
- }
- if ($architecture -eq "32") {
- $platform_suffix = "x86"
- } else {
- $platform_suffix = "x86_64"
- }
- $filepath = DownloadMiniconda $python_version $platform_suffix
- Write-Host "Installing" $filepath "to" $python_home
- $install_log = $python_home + ".log"
- $args = "/S /D=$python_home"
- Write-Host $filepath $args
- Start-Process -FilePath $filepath -ArgumentList $args -Wait -Passthru
- if (Test-Path $python_home) {
- Write-Host "Python $python_version ($architecture) installation complete"
- } else {
- Write-Host "Failed to install Python in $python_home"
- Get-Content -Path $install_log
- Exit 1
- }
-}
-
-
-function InstallMinicondaPip ($python_home) {
- $pip_path = $python_home + "\Scripts\pip.exe"
- $conda_path = $python_home + "\Scripts\conda.exe"
- if (-not(Test-Path $pip_path)) {
- Write-Host "Installing pip..."
- $args = "install --yes pip"
- Write-Host $conda_path $args
- Start-Process -FilePath "$conda_path" -ArgumentList $args -Wait -Passthru
- } else {
- Write-Host "pip already installed."
- }
-}
-
-function main () {
- InstallPython $env:PYTHON_VERSION $env:PYTHON_ARCH $env:PYTHON
- InstallPip $env:PYTHON
-}
-
-main
diff --git a/appveyor/requirements.txt b/appveyor/requirements.txt
index 390d11078312..45a4e4c25627 100644
--- a/appveyor/requirements.txt
+++ b/appveyor/requirements.txt
@@ -3,4 +3,4 @@
# pip will build them from source using the MSVC compiler matching the
# target Python version and architecture
wheel
-tox
+nox-automation>=0.17.0
diff --git a/appveyor/run_with_env.cmd b/appveyor/run_with_env.cmd
deleted file mode 100644
index 5da547c499ee..000000000000
--- a/appveyor/run_with_env.cmd
+++ /dev/null
@@ -1,88 +0,0 @@
-:: To build extensions for 64 bit Python 3, we need to configure environment
-:: variables to use the MSVC 2010 C++ compilers from GRMSDKX_EN_DVD.iso of:
-:: MS Windows SDK for Windows 7 and .NET Framework 4 (SDK v7.1)
-::
-:: To build extensions for 64 bit Python 2, we need to configure environment
-:: variables to use the MSVC 2008 C++ compilers from GRMSDKX_EN_DVD.iso of:
-:: MS Windows SDK for Windows 7 and .NET Framework 3.5 (SDK v7.0)
-::
-:: 32 bit builds, and 64-bit builds for 3.5 and beyond, do not require specific
-:: environment configurations.
-::
-:: Note: this script needs to be run with the /E:ON and /V:ON flags for the
-:: cmd interpreter, at least for (SDK v7.0)
-::
-:: More details at:
-:: https://github.com/cython/cython/wiki/64BitCythonExtensionsOnWindows
-:: http://stackoverflow.com/a/13751649/163740
-::
-:: Author: Olivier Grisel
-:: License: CC0 1.0 Universal: http://creativecommons.org/publicdomain/zero/1.0/
-::
-:: Notes about batch files for Python people:
-::
-:: Quotes in values are literally part of the values:
-:: SET FOO="bar"
-:: FOO is now five characters long: " b a r "
-:: If you don't want quotes, don't include them on the right-hand side.
-::
-:: The CALL lines at the end of this file look redundant, but if you move them
-:: outside of the IF clauses, they do not run properly in the SET_SDK_64==Y
-:: case, I don't know why.
-@ECHO OFF
-
-SET COMMAND_TO_RUN=%*
-SET WIN_SDK_ROOT=C:\Program Files\Microsoft SDKs\Windows
-SET WIN_WDK=c:\Program Files (x86)\Windows Kits\10\Include\wdf
-
-:: Extract the major and minor versions, and allow for the minor version to be
-:: more than 9. This requires the version number to have two dots in it.
-SET MAJOR_PYTHON_VERSION=%PYTHON_VERSION:~0,1%
-IF "%PYTHON_VERSION:~3,1%" == "." (
- SET MINOR_PYTHON_VERSION=%PYTHON_VERSION:~2,1%
-) ELSE (
- SET MINOR_PYTHON_VERSION=%PYTHON_VERSION:~2,2%
-)
-
-:: Based on the Python version, determine what SDK version to use, and whether
-:: to set the SDK for 64-bit.
-IF %MAJOR_PYTHON_VERSION% == 2 (
- SET WINDOWS_SDK_VERSION="v7.0"
- SET SET_SDK_64=Y
-) ELSE (
- IF %MAJOR_PYTHON_VERSION% == 3 (
- SET WINDOWS_SDK_VERSION="v7.1"
- IF %MINOR_PYTHON_VERSION% LEQ 4 (
- SET SET_SDK_64=Y
- ) ELSE (
- SET SET_SDK_64=N
- IF EXIST "%WIN_WDK%" (
- :: See: https://connect.microsoft.com/VisualStudio/feedback/details/1610302/
- REN "%WIN_WDK%" 0wdf
- )
- )
- ) ELSE (
- ECHO Unsupported Python version: "%MAJOR_PYTHON_VERSION%"
- EXIT 1
- )
-)
-
-IF %PYTHON_ARCH% == 64 (
- IF %SET_SDK_64% == Y (
- ECHO Configuring Windows SDK %WINDOWS_SDK_VERSION% for Python %MAJOR_PYTHON_VERSION% on a 64 bit architecture
- SET DISTUTILS_USE_SDK=1
- SET MSSdk=1
- "%WIN_SDK_ROOT%\%WINDOWS_SDK_VERSION%\Setup\WindowsSdkVer.exe" -q -version:%WINDOWS_SDK_VERSION%
- "%WIN_SDK_ROOT%\%WINDOWS_SDK_VERSION%\Bin\SetEnv.cmd" /x64 /release
- ECHO Executing: %COMMAND_TO_RUN%
- call %COMMAND_TO_RUN% || EXIT 1
- ) ELSE (
- ECHO Using default MSVC build environment for 64 bit architecture
- ECHO Executing: %COMMAND_TO_RUN%
- call %COMMAND_TO_RUN% || EXIT 1
- )
-) ELSE (
- ECHO Using default MSVC build environment for 32 bit architecture
- ECHO Executing: %COMMAND_TO_RUN%
- call %COMMAND_TO_RUN% || EXIT 1
-)
diff --git a/asset/.coveragerc b/asset/.coveragerc
new file mode 100644
index 000000000000..b178b094aa1d
--- /dev/null
+++ b/asset/.coveragerc
@@ -0,0 +1,19 @@
+# Generated by synthtool. DO NOT EDIT!
+[run]
+branch = True
+
+[report]
+fail_under = 100
+show_missing = True
+exclude_lines =
+ # Re-enable the standard pragma
+ pragma: NO COVER
+ # Ignore debug-only repr
+ def __repr__
+ # Ignore abstract methods
+ raise NotImplementedError
+omit =
+ */gapic/*.py
+ */proto/*.py
+ */core/*.py
+ */site-packages/*.py
\ No newline at end of file
diff --git a/asset/.flake8 b/asset/.flake8
new file mode 100644
index 000000000000..0268ecc9c55c
--- /dev/null
+++ b/asset/.flake8
@@ -0,0 +1,14 @@
+# Generated by synthtool. DO NOT EDIT!
+[flake8]
+ignore = E203, E266, E501, W503
+exclude =
+ # Exclude generated code.
+ **/proto/**
+ **/gapic/**
+ *_pb2.py
+
+ # Standard linting exemptions.
+ __pycache__,
+ .git,
+ *.pyc,
+ conf.py
diff --git a/asset/.repo-metadata.json b/asset/.repo-metadata.json
new file mode 100644
index 000000000000..cc9ac3cd4b1d
--- /dev/null
+++ b/asset/.repo-metadata.json
@@ -0,0 +1,13 @@
+{
+ "name": "cloudasset",
+ "name_pretty": "Cloud Asset Inventory",
+ "product_documentation": "https://cloud.google.com/resource-manager/docs/cloud-asset-inventory/overview",
+ "client_documentation": "https://googleapis.dev/python/cloudasset/latest",
+ "issue_tracker": "https://issuetracker.google.com/savedsearches/559757",
+ "release_level": "alpha",
+ "language": "python",
+ "repo": "googleapis/google-cloud-python",
+ "distribution_name": "google-cloud-asset",
+ "api_id": "cloudasset.googleapis.com",
+ "requires_billing": true
+}
\ No newline at end of file
diff --git a/asset/CHANGELOG.md b/asset/CHANGELOG.md
new file mode 100644
index 000000000000..049d0f2c3017
--- /dev/null
+++ b/asset/CHANGELOG.md
@@ -0,0 +1,126 @@
+# Changelog
+
+[PyPI History][1]
+
+[1]: https://pypi.org/project/google-cloud-asset/#history
+
+## 0.4.1
+
+08-12-2019 13:44 PDT
+
+### Documentation
+- Fix links to googleapis.dev ([#8998](https://github.com/googleapis/google-cloud-python/pull/8998))
+
+## 0.4.0
+
+08-01-2019 14:24 PDT
+
+### New Features
+- Generate asset v1p2beta1. ([#8888](https://github.com/googleapis/google-cloud-python/pull/8888))
+
+### Internal / Testing Changes
+- Update intersphinx mapping for requests. ([#8805](https://github.com/googleapis/google-cloud-python/pull/8805))
+
+## 0.3.0
+
+07-22-2019 17:42 PDT
+
+
+### Implementation Changes
+- Allow kwargs to be passed to create_channel (via synth). ([#8382](https://github.com/googleapis/google-cloud-python/pull/8382))
+- Add nox session docs, add routing header to method metadata (via synth). ([#7919](https://github.com/googleapis/google-cloud-python/pull/7919))
+- Remove classifier for Python 3.4 for end-of-life. ([#7535](https://github.com/googleapis/google-cloud-python/pull/7535))
+
+### New Features
+- Add 'client_options' support (via synth). ([#8498](https://github.com/googleapis/google-cloud-python/pull/8498))
+
+### Dependencies
+- Bump minimum version for google-api-core to 1.14.0. ([#8709](https://github.com/googleapis/google-cloud-python/pull/8709))
+- Update pin for 'grpc-google-iam-v1' to 0.12.3+. ([#8647](https://github.com/googleapis/google-cloud-python/pull/8647))
+
+### Documentation
+- Link to googleapis.dev documentation in READMEs. ([#8705](https://github.com/googleapis/google-cloud-python/pull/8705))
+- Add compatibility check badges to READMEs. ([#8288](https://github.com/googleapis/google-cloud-python/pull/8288))
+
+### Internal / Testing Changes
+- Update vpcsc test settings. ([#8627](https://github.com/googleapis/google-cloud-python/pull/8627))
+- Pin black version (via synth) ([#8572](https://github.com/googleapis/google-cloud-python/pull/8572))
+- Add VPCSC tests. ([#8613](https://github.com/googleapis/google-cloud-python/pull/8613))
+- All: Add docs job to publish to googleapis.dev. ([#8464](https://github.com/googleapis/google-cloud-python/pull/8464))
+- Add disclaimer to auto-generated template files (via synth). ([#8306](https://github.com/googleapis/google-cloud-python/pull/8306))
+- Suppress checking 'cov-fail-under' in nox default session (via synth). ([#8232](https://github.com/googleapis/google-cloud-python/pull/8232))
+- Fix coverage in 'types.py'. ([#8144](https://github.com/googleapis/google-cloud-python/pull/8144))
+- Blacken noxfile.py, setup.py (via synth). ([#8114](https://github.com/googleapis/google-cloud-python/pull/8114))
+- Declare encoding as utf-8 in pb2 files (via synth). ([#8343](https://github.com/googleapis/google-cloud-python/pull/8343))
+- Add empty lines (via synth). ([#8047](https://github.com/googleapis/google-cloud-python/pull/8047))
+
+## 0.2.0
+
+03-19-2019 12:17 PDT
+
+
+### Implementation Changes
+- Rename 'GcsDestination.uri' -> 'object_uri', docstring changes . ([#7202](https://github.com/googleapis/google-cloud-python/pull/7202))
+- Protoc-generated serialization update.. ([#7073](https://github.com/googleapis/google-cloud-python/pull/7073))
+
+### New Features
+- Generate v1. ([#7513](https://github.com/googleapis/google-cloud-python/pull/7513))
+
+### Documentation
+- Fix broken links to Cloud Asset API ([#7524](https://github.com/googleapis/google-cloud-python/pull/7524))
+- Updated client library documentation URLs. ([#7307](https://github.com/googleapis/google-cloud-python/pull/7307))
+- Update copyright headers
+- Pick up stub docstring fix in GAPIC generator.[#6963](https://github.com/googleapis/google-cloud-python/pull/6963))
+
+### Internal / Testing Changes
+- Add protos as an artifact to library ([#7205](https://github.com/googleapis/google-cloud-python/pull/7205))
+- Add support for including protos in synth ([#7114](https://github.com/googleapis/google-cloud-python/pull/7114))
+
+## 0.1.2
+
+12-17-2018 16:15 PST
+
+
+### Implementation Changes
+- Use moved iam.policy now at google.api_core.iam.policy ([#6741](https://github.com/googleapis/google-cloud-python/pull/6741))
+- Pick up enum fixes in the GAPIC generator. ([#6607](https://github.com/googleapis/google-cloud-python/pull/6607))
+- Pick up fixes in GAPIC generator. ([#6489](https://github.com/googleapis/google-cloud-python/pull/6489))
+- Fix client_info bug, update docstrings. ([#6403](https://github.com/googleapis/google-cloud-python/pull/6403))
+- Synth docstring changes generated from updated protos ([#6349](https://github.com/googleapis/google-cloud-python/pull/6349))
+- Generated cloud asset client files are under asset-[version] ([#6341](https://github.com/googleapis/google-cloud-python/pull/6341))
+
+### New Features
+
+### Dependencies
+- Add 'mock' to unit test dependencies for autogen libs. ([#6402](https://github.com/googleapis/google-cloud-python/pull/6402))
+- Bump minimum `api_core` version for all GAPIC libs to 1.4.1. ([#6391](https://github.com/googleapis/google-cloud-python/pull/6391))
+- Avoid broken 'google-common-apis 1.5.4' release. ([#6355](https://github.com/googleapis/google-cloud-python/pull/6355))
+
+### Documentation
+- Document Python 2 deprecation ([#6910](https://github.com/googleapis/google-cloud-python/pull/6910))
+- Fix docs build. ([#6351](https://github.com/googleapis/google-cloud-python/pull/6351))
+- Normalize use of support level badges ([#6159](https://github.com/googleapis/google-cloud-python/pull/6159))
+
+### Internal / Testing Changes
+- Add templating to asset synth.py ([#6606](https://github.com/googleapis/google-cloud-python/pull/6606))
+- Add synth metadata. ([#6560](https://github.com/googleapis/google-cloud-python/pull/6560))
+- Update noxfile.
+- Blacken all gen'd libs ([#6792](https://github.com/googleapis/google-cloud-python/pull/6792))
+- Omit local deps ([#6701](https://github.com/googleapis/google-cloud-python/pull/6701))
+- Run black at end of synth.py ([#6698](https://github.com/googleapis/google-cloud-python/pull/6698))
+- Unblack gapic and protos.
+- Run Black on Generated libraries ([#6666](https://github.com/googleapis/google-cloud-python/pull/6666))
+- Add templates for flake8, coveragerc, noxfile, and black. ([#6642](https://github.com/googleapis/google-cloud-python/pull/6642))
+- Add / fix badges for PyPI / versions. ([#6158](https://github.com/googleapis/google-cloud-python/pull/6158))
+- Use new Nox ([#6175](https://github.com/googleapis/google-cloud-python/pull/6175))
+
+## 0.1.1
+
+### Packaging
+- Release as `google-cloud-asset`, rather than `google-cloud-cloudasset`.
+ (#5998)
+
+## 0.1.0
+
+Initial release.
+
diff --git a/asset/LICENSE b/asset/LICENSE
new file mode 100644
index 000000000000..a8ee855de2aa
--- /dev/null
+++ b/asset/LICENSE
@@ -0,0 +1,201 @@
+ Apache License
+ Version 2.0, January 2004
+ https://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [yyyy] [name of copyright owner]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
diff --git a/asset/MANIFEST.in b/asset/MANIFEST.in
new file mode 100644
index 000000000000..9cbf175afe6b
--- /dev/null
+++ b/asset/MANIFEST.in
@@ -0,0 +1,5 @@
+include README.rst LICENSE
+recursive-include google *.json *.proto
+recursive-include tests *
+global-exclude *.py[co]
+global-exclude __pycache__
diff --git a/asset/README.rst b/asset/README.rst
new file mode 100644
index 000000000000..e0d579c20035
--- /dev/null
+++ b/asset/README.rst
@@ -0,0 +1,86 @@
+Python Client for Cloud Asset API
+=================================
+
+|alpha| |pypi| |versions|
+
+`Cloud Asset API`_: The cloud asset API manages the history and inventory of cloud resources.
+
+- `Client Library Documentation`_
+- `Product Documentation`_
+
+.. |alpha| image:: https://img.shields.io/badge/support-alpha-orange.svg
+ :target: https://github.com/googleapis/google-cloud-python/blob/master/README.rst#alpha-support
+.. |pypi| image:: https://img.shields.io/pypi/v/google-cloud-asset.svg
+ :target: https://pypi.org/project/google-cloud-asset/
+.. |versions| image:: https://img.shields.io/pypi/pyversions/google-cloud-asset.svg
+ :target: https://pypi.org/project/google-cloud-asset/
+.. _Cloud Asset API: https://cloud.google.com/resource-manager/docs/cloud-asset-inventory/reference/rest/
+.. _Client Library Documentation: https://googleapis.dev/python/cloudasset/latest
+.. _Product Documentation: https://cloud.google.com/resource-manager/docs/cloud-asset-inventory/overview
+
+Quick Start
+-----------
+
+In order to use this library, you first need to go through the following steps:
+
+1. `Select or create a Cloud Platform project.`_
+2. `Enable billing for your project.`_
+3. `Enable the Cloud Asset API.`_
+4. `Setup Authentication.`_
+
+.. _Select or create a Cloud Platform project.: https://console.cloud.google.com/project
+.. _Enable billing for your project.: https://cloud.google.com/billing/docs/how-to/modify-project#enable_billing_for_a_project
+.. _Enable the Cloud Asset API.: https://console.cloud.google.com/apis/library/cloudasset.googleapis.com
+.. _Setup Authentication.: https://googleapis.dev/python/google-api-core/latest/auth.html
+
+Installation
+~~~~~~~~~~~~
+
+Install this library in a `virtualenv`_ using pip. `virtualenv`_ is a tool to
+create isolated Python environments. The basic problem it addresses is one of
+dependencies and versions, and indirectly permissions.
+
+With `virtualenv`_, it's possible to install this library without needing system
+install permissions, and without clashing with the installed system
+dependencies.
+
+.. _`virtualenv`: https://virtualenv.pypa.io/en/latest/
+
+
+Supported Python Versions
+^^^^^^^^^^^^^^^^^^^^^^^^^
+Python >= 3.5
+
+Deprecated Python Versions
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+Python == 2.7. Python 2.7 support will be removed on January 1, 2020.
+
+
+Mac/Linux
+^^^^^^^^^
+
+.. code-block:: console
+
+ pip install virtualenv
+ virtualenv
+ source /bin/activate
+ /bin/pip install google-cloud-asset
+
+
+Windows
+^^^^^^^
+
+.. code-block:: console
+
+ pip install virtualenv
+ virtualenv
+ \Scripts\activate
+ \Scripts\pip.exe install google-cloud-asset
+
+Next Steps
+~~~~~~~~~~
+
+- Read the `Client Library Documentation`_ for Cloud Asset API
+ API to see other available methods on the client.
+- Read the `Product documentation`_ to learn
+ more about the product and see How-to Guides.
diff --git a/asset/docs/README.rst b/asset/docs/README.rst
new file mode 120000
index 000000000000..89a0106941ff
--- /dev/null
+++ b/asset/docs/README.rst
@@ -0,0 +1 @@
+../README.rst
\ No newline at end of file
diff --git a/asset/docs/changelog.md b/asset/docs/changelog.md
new file mode 120000
index 000000000000..04c99a55caae
--- /dev/null
+++ b/asset/docs/changelog.md
@@ -0,0 +1 @@
+../CHANGELOG.md
\ No newline at end of file
diff --git a/asset/docs/conf.py b/asset/docs/conf.py
new file mode 100644
index 000000000000..cb7ef3bdb856
--- /dev/null
+++ b/asset/docs/conf.py
@@ -0,0 +1,350 @@
+# -*- coding: utf-8 -*-
+#
+# google-cloud-asset documentation build configuration file
+#
+# This file is execfile()d with the current directory set to its
+# containing dir.
+#
+# Note that not all possible configuration values are present in this
+# autogenerated file.
+#
+# All configuration values have a default; values that are commented out
+# serve to show the default.
+
+import sys
+import os
+import shlex
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+sys.path.insert(0, os.path.abspath(".."))
+
+__version__ = "0.1.0"
+
+# -- General configuration ------------------------------------------------
+
+# If your documentation needs a minimal Sphinx version, state it here.
+needs_sphinx = "1.6.3"
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+ "sphinx.ext.autodoc",
+ "sphinx.ext.autosummary",
+ "sphinx.ext.intersphinx",
+ "sphinx.ext.coverage",
+ "sphinx.ext.napoleon",
+ "sphinx.ext.todo",
+ "sphinx.ext.viewcode",
+]
+
+# autodoc/autosummary flags
+autoclass_content = "both"
+autodoc_default_flags = ["members"]
+autosummary_generate = True
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ["_templates"]
+
+# Allow markdown includes (so releases.md can include CHANGLEOG.md)
+# http://www.sphinx-doc.org/en/master/markdown.html
+source_parsers = {".md": "recommonmark.parser.CommonMarkParser"}
+
+# The suffix(es) of source filenames.
+# You can specify multiple suffix as a list of string:
+# source_suffix = ['.rst', '.md']
+source_suffix = [".rst", ".md"]
+
+# The encoding of source files.
+# source_encoding = 'utf-8-sig'
+
+# The master toctree document.
+master_doc = "index"
+
+# General information about the project.
+project = u"google-cloud-asset"
+copyright = u"2017, Google"
+author = u"Google APIs"
+
+# The version info for the project you're documenting, acts as replacement for
+# |version| and |release|, also used in various other places throughout the
+# built documents.
+#
+# The full version, including alpha/beta/rc tags.
+release = __version__
+# The short X.Y version.
+version = ".".join(release.split(".")[0:2])
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#
+# This is also used if you do content translation via gettext catalogs.
+# Usually you set "language" from the command line for these cases.
+language = None
+
+# There are two options for replacing |today|: either, you set today to some
+# non-false value, then it is used:
+# today = ''
+# Else, today_fmt is used as the format for a strftime call.
+# today_fmt = '%B %d, %Y'
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+exclude_patterns = ["_build"]
+
+# The reST default role (used for this markup: `text`) to use for all
+# documents.
+# default_role = None
+
+# If true, '()' will be appended to :func: etc. cross-reference text.
+# add_function_parentheses = True
+
+# If true, the current module name will be prepended to all description
+# unit titles (such as .. function::).
+# add_module_names = True
+
+# If true, sectionauthor and moduleauthor directives will be shown in the
+# output. They are ignored by default.
+# show_authors = False
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = "sphinx"
+
+# A list of ignored prefixes for module index sorting.
+# modindex_common_prefix = []
+
+# If true, keep warnings as "system message" paragraphs in the built documents.
+# keep_warnings = False
+
+# If true, `todo` and `todoList` produce output, else they produce nothing.
+todo_include_todos = True
+
+# -- Options for HTML output ----------------------------------------------
+
+# The theme to use for HTML and HTML Help pages. See the documentation for
+# a list of builtin themes.
+html_theme = "alabaster"
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further. For a list of options available for each theme, see the
+# documentation.
+html_theme_options = {
+ "description": "Google Cloud Client Libraries for Python",
+ "github_user": "googleapis",
+ "github_repo": "google-cloud-python",
+ "github_banner": True,
+ "font_family": "'Roboto', Georgia, sans",
+ "head_font_family": "'Roboto', Georgia, serif",
+ "code_font_family": "'Roboto Mono', 'Consolas', monospace",
+}
+
+# Add any paths that contain custom themes here, relative to this directory.
+# html_theme_path = []
+
+# The name for this set of Sphinx documents. If None, it defaults to
+# " v documentation".
+# html_title = None
+
+# A shorter title for the navigation bar. Default is the same as html_title.
+# html_short_title = None
+
+# The name of an image file (relative to this directory) to place at the top
+# of the sidebar.
+# html_logo = None
+
+# The name of an image file (within the static path) to use as favicon of the
+# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
+# pixels large.
+# html_favicon = None
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+# html_static_path = []
+
+# Add any extra paths that contain custom files (such as robots.txt or
+# .htaccess) here, relative to this directory. These files are copied
+# directly to the root of the documentation.
+# html_extra_path = []
+
+# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
+# using the given strftime format.
+# html_last_updated_fmt = '%b %d, %Y'
+
+# If true, SmartyPants will be used to convert quotes and dashes to
+# typographically correct entities.
+# html_use_smartypants = True
+
+# Custom sidebar templates, maps document names to template names.
+# html_sidebars = {}
+
+# Additional templates that should be rendered to pages, maps page names to
+# template names.
+# html_additional_pages = {}
+
+# If false, no module index is generated.
+# html_domain_indices = True
+
+# If false, no index is generated.
+# html_use_index = True
+
+# If true, the index is split into individual pages for each letter.
+# html_split_index = False
+
+# If true, links to the reST sources are added to the pages.
+# html_show_sourcelink = True
+
+# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
+# html_show_sphinx = True
+
+# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
+# html_show_copyright = True
+
+# If true, an OpenSearch description file will be output, and all pages will
+# contain a tag referring to it. The value of this option must be the
+# base URL from which the finished HTML is served.
+# html_use_opensearch = ''
+
+# This is the file name suffix for HTML files (e.g. ".xhtml").
+# html_file_suffix = None
+
+# Language to be used for generating the HTML full-text search index.
+# Sphinx supports the following languages:
+# 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja'
+# 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr'
+# html_search_language = 'en'
+
+# A dictionary with options for the search language support, empty by default.
+# Now only 'ja' uses this config value
+# html_search_options = {'type': 'default'}
+
+# The name of a javascript file (relative to the configuration directory) that
+# implements a search results scorer. If empty, the default will be used.
+# html_search_scorer = 'scorer.js'
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = "google-cloud-asset-doc"
+
+# -- Options for warnings ------------------------------------------------------
+
+suppress_warnings = [
+ # Temporarily suppress this to avoid "more than one target found for
+ # cross-reference" warning, which are intractable for us to avoid while in
+ # a mono-repo.
+ # See https://github.com/sphinx-doc/sphinx/blob
+ # /2a65ffeef5c107c19084fabdd706cdff3f52d93c/sphinx/domains/python.py#L843
+ "ref.python"
+]
+
+# -- Options for LaTeX output ---------------------------------------------
+
+latex_elements = {
+ # The paper size ('letterpaper' or 'a4paper').
+ #'papersize': 'letterpaper',
+ # The font size ('10pt', '11pt' or '12pt').
+ #'pointsize': '10pt',
+ # Additional stuff for the LaTeX preamble.
+ #'preamble': '',
+ # Latex figure (float) alignment
+ #'figure_align': 'htbp',
+}
+
+# Grouping the document tree into LaTeX files. List of tuples
+# (source start file, target name, title,
+# author, documentclass [howto, manual, or own class]).
+latex_documents = [
+ (
+ master_doc,
+ "google-cloud-asset.tex",
+ u"google-cloud-asset Documentation",
+ author,
+ "manual",
+ )
+]
+
+# The name of an image file (relative to this directory) to place at the top of
+# the title page.
+# latex_logo = None
+
+# For "manual" documents, if this is true, then toplevel headings are parts,
+# not chapters.
+# latex_use_parts = False
+
+# If true, show page references after internal links.
+# latex_show_pagerefs = False
+
+# If true, show URL addresses after external links.
+# latex_show_urls = False
+
+# Documents to append as an appendix to all manuals.
+# latex_appendices = []
+
+# If false, no module index is generated.
+# latex_domain_indices = True
+
+# -- Options for manual page output ---------------------------------------
+
+# One entry per manual page. List of tuples
+# (source start file, name, description, authors, manual section).
+man_pages = [
+ (master_doc, "google-cloud-asset", u"google-cloud-asset Documentation", [author], 1)
+]
+
+# If true, show URL addresses after external links.
+# man_show_urls = False
+
+# -- Options for Texinfo output -------------------------------------------
+
+# Grouping the document tree into Texinfo files. List of tuples
+# (source start file, target name, title, author,
+# dir menu entry, description, category)
+texinfo_documents = [
+ (
+ master_doc,
+ "google-cloud-asset",
+ u"google-cloud-asset Documentation",
+ author,
+ "google-cloud-asset",
+ "GAPIC library for the {metadata.shortName} v1 service",
+ "APIs",
+ )
+]
+
+# Documents to append as an appendix to all manuals.
+# texinfo_appendices = []
+
+# If false, no module index is generated.
+# texinfo_domain_indices = True
+
+# How to display URL addresses: 'footnote', 'no', or 'inline'.
+# texinfo_show_urls = 'footnote'
+
+# If true, do not generate a @detailmenu in the "Top" node's menu.
+# texinfo_no_detailmenu = False
+
+# Example configuration for intersphinx: refer to the Python standard library.
+intersphinx_mapping = {
+ "python": ("http://python.readthedocs.org/en/latest/", None),
+ "gax": ("https://gax-python.readthedocs.org/en/latest/", None),
+ "google-auth": ("https://google-auth.readthedocs.io/en/stable", None),
+ "google-gax": ("https://gax-python.readthedocs.io/en/latest/", None),
+ "google.api_core": ("https://googleapis.dev/python/google-api-core/latest", None),
+ "grpc": ("https://grpc.io/grpc/python/", None),
+ "requests": ("https://2.python-requests.org/en/master/", None),
+ "fastavro": ("https://fastavro.readthedocs.io/en/stable/", None),
+ "pandas": ("https://pandas.pydata.org/pandas-docs/stable/", None),
+}
+
+# Napoleon settings
+napoleon_google_docstring = True
+napoleon_numpy_docstring = True
+napoleon_include_private_with_doc = False
+napoleon_include_special_with_doc = True
+napoleon_use_admonition_for_examples = False
+napoleon_use_admonition_for_notes = False
+napoleon_use_admonition_for_references = False
+napoleon_use_ivar = False
+napoleon_use_param = True
+napoleon_use_rtype = True
diff --git a/asset/docs/gapic/v1/api.rst b/asset/docs/gapic/v1/api.rst
new file mode 100644
index 000000000000..12fc69a59923
--- /dev/null
+++ b/asset/docs/gapic/v1/api.rst
@@ -0,0 +1,6 @@
+Client for Cloud Asset API
+==========================
+
+.. automodule:: google.cloud.asset_v1
+ :members:
+ :inherited-members:
\ No newline at end of file
diff --git a/asset/docs/gapic/v1/types.rst b/asset/docs/gapic/v1/types.rst
new file mode 100644
index 000000000000..962b18ff2902
--- /dev/null
+++ b/asset/docs/gapic/v1/types.rst
@@ -0,0 +1,5 @@
+Types for Cloud Asset API Client
+================================
+
+.. automodule:: google.cloud.asset_v1.types
+ :members:
\ No newline at end of file
diff --git a/asset/docs/gapic/v1beta1/api.rst b/asset/docs/gapic/v1beta1/api.rst
new file mode 100644
index 000000000000..f5b1195aef13
--- /dev/null
+++ b/asset/docs/gapic/v1beta1/api.rst
@@ -0,0 +1,6 @@
+Client for Cloud Asset API
+==========================
+
+.. automodule:: google.cloud.asset_v1beta1
+ :members:
+ :inherited-members:
\ No newline at end of file
diff --git a/asset/docs/gapic/v1beta1/types.rst b/asset/docs/gapic/v1beta1/types.rst
new file mode 100644
index 000000000000..550bd11428af
--- /dev/null
+++ b/asset/docs/gapic/v1beta1/types.rst
@@ -0,0 +1,5 @@
+Types for Cloud Asset API Client
+================================
+
+.. automodule:: google.cloud.asset_v1beta1.types
+ :members:
\ No newline at end of file
diff --git a/asset/docs/gapic/v1p2beta1/api.rst b/asset/docs/gapic/v1p2beta1/api.rst
new file mode 100644
index 000000000000..ca2696c24291
--- /dev/null
+++ b/asset/docs/gapic/v1p2beta1/api.rst
@@ -0,0 +1,6 @@
+Client for Cloud Asset API
+==========================
+
+.. automodule:: google.cloud.asset_v1p2beta1
+ :members:
+ :inherited-members:
\ No newline at end of file
diff --git a/asset/docs/gapic/v1p2beta1/types.rst b/asset/docs/gapic/v1p2beta1/types.rst
new file mode 100644
index 000000000000..914469b3224f
--- /dev/null
+++ b/asset/docs/gapic/v1p2beta1/types.rst
@@ -0,0 +1,5 @@
+Types for Cloud Asset API Client
+================================
+
+.. automodule:: google.cloud.asset_v1p2beta1.types
+ :members:
\ No newline at end of file
diff --git a/asset/docs/index.rst b/asset/docs/index.rst
new file mode 100644
index 000000000000..cc1b49a90e90
--- /dev/null
+++ b/asset/docs/index.rst
@@ -0,0 +1,46 @@
+.. include:: README.rst
+
+Api Reference
+-------------
+
+This package includes clients for multiple versions of the Cloud Asset API. By default, you will get ``v1``
+the latest stable version.
+
+.. toctree::
+ :maxdepth: 2
+
+ gapic/v1/api
+ gapic/v1/types
+
+A new beta release with additional features over the current stable version, spelled ``v1p2beta1``, is provided
+to allow you to use these new features. These are expected to move into the stable release soon; until
+then, the usual beta admonishment (changes are possible, etc.) applies.
+
+An API and type reference is provided for this beta:
+
+.. toctree::
+ :maxdepth: 2
+
+ gapic/v1p2beta1/api
+ gapic/v1p2beta1/types
+
+The previous beta release, spelled ``v1p2beta1`` is also provided to continue to support code
+previously written against it. In order to use it, you will want to import from
+``google.cloud.asset_v1beta1`` in lieu of ``google.cloud.asset_v1``.
+
+.. toctree::
+ :maxdepth: 2
+
+ gapic/v1beta1/api
+ gapic/v1beta1/types
+
+
+Changelog
+---------
+
+For a list of all ``google-cloud-asset`` releases:
+
+.. toctree::
+ :maxdepth: 2
+
+ changelog
diff --git a/asset/google/__init__.py b/asset/google/__init__.py
new file mode 100644
index 000000000000..8fcc60e2b9c6
--- /dev/null
+++ b/asset/google/__init__.py
@@ -0,0 +1,24 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+try:
+ import pkg_resources
+
+ pkg_resources.declare_namespace(__name__)
+except ImportError:
+ import pkgutil
+
+ __path__ = pkgutil.extend_path(__path__, __name__)
diff --git a/asset/google/cloud/__init__.py b/asset/google/cloud/__init__.py
new file mode 100644
index 000000000000..8fcc60e2b9c6
--- /dev/null
+++ b/asset/google/cloud/__init__.py
@@ -0,0 +1,24 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+try:
+ import pkg_resources
+
+ pkg_resources.declare_namespace(__name__)
+except ImportError:
+ import pkgutil
+
+ __path__ = pkgutil.extend_path(__path__, __name__)
diff --git a/asset/google/cloud/asset.py b/asset/google/cloud/asset.py
new file mode 100644
index 000000000000..24c7e7d24e40
--- /dev/null
+++ b/asset/google/cloud/asset.py
@@ -0,0 +1,25 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from __future__ import absolute_import
+
+from google.cloud.asset_v1 import AssetServiceClient
+from google.cloud.asset_v1 import enums
+from google.cloud.asset_v1 import types
+
+
+__all__ = ("enums", "types", "AssetServiceClient")
diff --git a/asset/google/cloud/asset_v1/__init__.py b/asset/google/cloud/asset_v1/__init__.py
new file mode 100644
index 000000000000..a3dfa14c692b
--- /dev/null
+++ b/asset/google/cloud/asset_v1/__init__.py
@@ -0,0 +1,30 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from __future__ import absolute_import
+
+from google.cloud.asset_v1 import types
+from google.cloud.asset_v1.gapic import asset_service_client
+from google.cloud.asset_v1.gapic import enums
+
+
+class AssetServiceClient(asset_service_client.AssetServiceClient):
+ __doc__ = asset_service_client.AssetServiceClient.__doc__
+ enums = enums
+
+
+__all__ = ("enums", "types", "AssetServiceClient")
diff --git a/asset/google/cloud/asset_v1/gapic/__init__.py b/asset/google/cloud/asset_v1/gapic/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/asset/google/cloud/asset_v1/gapic/asset_service_client.py b/asset/google/cloud/asset_v1/gapic/asset_service_client.py
new file mode 100644
index 000000000000..ab9078cc62c7
--- /dev/null
+++ b/asset/google/cloud/asset_v1/gapic/asset_service_client.py
@@ -0,0 +1,431 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Accesses the google.cloud.asset.v1 AssetService API."""
+
+import pkg_resources
+import warnings
+
+from google.oauth2 import service_account
+import google.api_core.client_options
+import google.api_core.gapic_v1.client_info
+import google.api_core.gapic_v1.config
+import google.api_core.gapic_v1.method
+import google.api_core.gapic_v1.routing_header
+import google.api_core.grpc_helpers
+import google.api_core.operation
+import google.api_core.operations_v1
+import google.api_core.path_template
+import grpc
+
+from google.cloud.asset_v1.gapic import asset_service_client_config
+from google.cloud.asset_v1.gapic import enums
+from google.cloud.asset_v1.gapic.transports import asset_service_grpc_transport
+from google.cloud.asset_v1.proto import asset_service_pb2
+from google.cloud.asset_v1.proto import asset_service_pb2_grpc
+from google.cloud.asset_v1.proto import assets_pb2
+from google.longrunning import operations_pb2
+from google.protobuf import timestamp_pb2
+
+
+_GAPIC_LIBRARY_VERSION = pkg_resources.get_distribution("google-cloud-asset").version
+
+
+class AssetServiceClient(object):
+ """Asset service definition."""
+
+ SERVICE_ADDRESS = "cloudasset.googleapis.com:443"
+ """The default address of the service."""
+
+ # The name of the interface for this client. This is the key used to
+ # find the method configuration in the client_config dictionary.
+ _INTERFACE_NAME = "google.cloud.asset.v1.AssetService"
+
+ @classmethod
+ def from_service_account_file(cls, filename, *args, **kwargs):
+ """Creates an instance of this client using the provided credentials
+ file.
+
+ Args:
+ filename (str): The path to the service account private key json
+ file.
+ args: Additional arguments to pass to the constructor.
+ kwargs: Additional arguments to pass to the constructor.
+
+ Returns:
+ AssetServiceClient: The constructed client.
+ """
+ credentials = service_account.Credentials.from_service_account_file(filename)
+ kwargs["credentials"] = credentials
+ return cls(*args, **kwargs)
+
+ from_service_account_json = from_service_account_file
+
+ @classmethod
+ def project_path(cls, project):
+ """Return a fully-qualified project string."""
+ return google.api_core.path_template.expand(
+ "projects/{project}", project=project
+ )
+
+ def __init__(
+ self,
+ transport=None,
+ channel=None,
+ credentials=None,
+ client_config=None,
+ client_info=None,
+ client_options=None,
+ ):
+ """Constructor.
+
+ Args:
+ transport (Union[~.AssetServiceGrpcTransport,
+ Callable[[~.Credentials, type], ~.AssetServiceGrpcTransport]): A transport
+ instance, responsible for actually making the API calls.
+ The default transport uses the gRPC protocol.
+ This argument may also be a callable which returns a
+ transport instance. Callables will be sent the credentials
+ as the first argument and the default transport class as
+ the second argument.
+ channel (grpc.Channel): DEPRECATED. A ``Channel`` instance
+ through which to make calls. This argument is mutually exclusive
+ with ``credentials``; providing both will raise an exception.
+ credentials (google.auth.credentials.Credentials): The
+ authorization credentials to attach to requests. These
+ credentials identify this application to the service. If none
+ are specified, the client will attempt to ascertain the
+ credentials from the environment.
+ This argument is mutually exclusive with providing a
+ transport instance to ``transport``; doing so will raise
+ an exception.
+ client_config (dict): DEPRECATED. A dictionary of call options for
+ each method. If not specified, the default configuration is used.
+ client_info (google.api_core.gapic_v1.client_info.ClientInfo):
+ The client info used to send a user-agent string along with
+ API requests. If ``None``, then default info will be used.
+ Generally, you only need to set this if you're developing
+ your own client library.
+ client_options (Union[dict, google.api_core.client_options.ClientOptions]):
+ Client options used to set user options on the client. API Endpoint
+ should be set through client_options.
+ """
+ # Raise deprecation warnings for things we want to go away.
+ if client_config is not None:
+ warnings.warn(
+ "The `client_config` argument is deprecated.",
+ PendingDeprecationWarning,
+ stacklevel=2,
+ )
+ else:
+ client_config = asset_service_client_config.config
+
+ if channel:
+ warnings.warn(
+ "The `channel` argument is deprecated; use " "`transport` instead.",
+ PendingDeprecationWarning,
+ stacklevel=2,
+ )
+
+ api_endpoint = self.SERVICE_ADDRESS
+ if client_options:
+ if type(client_options) == dict:
+ client_options = google.api_core.client_options.from_dict(
+ client_options
+ )
+ if client_options.api_endpoint:
+ api_endpoint = client_options.api_endpoint
+
+ # Instantiate the transport.
+ # The transport is responsible for handling serialization and
+ # deserialization and actually sending data to the service.
+ if transport:
+ if callable(transport):
+ self.transport = transport(
+ credentials=credentials,
+ default_class=asset_service_grpc_transport.AssetServiceGrpcTransport,
+ address=api_endpoint,
+ )
+ else:
+ if credentials:
+ raise ValueError(
+ "Received both a transport instance and "
+ "credentials; these are mutually exclusive."
+ )
+ self.transport = transport
+ else:
+ self.transport = asset_service_grpc_transport.AssetServiceGrpcTransport(
+ address=api_endpoint, channel=channel, credentials=credentials
+ )
+
+ if client_info is None:
+ client_info = google.api_core.gapic_v1.client_info.ClientInfo(
+ gapic_version=_GAPIC_LIBRARY_VERSION
+ )
+ else:
+ client_info.gapic_version = _GAPIC_LIBRARY_VERSION
+ self._client_info = client_info
+
+ # Parse out the default settings for retry and timeout for each RPC
+ # from the client configuration.
+ # (Ordinarily, these are the defaults specified in the `*_config.py`
+ # file next to this one.)
+ self._method_configs = google.api_core.gapic_v1.config.parse_method_configs(
+ client_config["interfaces"][self._INTERFACE_NAME]
+ )
+
+ # Save a dictionary of cached API call functions.
+ # These are the actual callables which invoke the proper
+ # transport methods, wrapped with `wrap_method` to add retry,
+ # timeout, and the like.
+ self._inner_api_calls = {}
+
+ # Service calls
+ def export_assets(
+ self,
+ parent,
+ output_config,
+ read_time=None,
+ asset_types=None,
+ content_type=None,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Exports assets with time and resource types to a given Cloud Storage
+ location. The output format is newline-delimited JSON. This API
+ implements the ``google.longrunning.Operation`` API allowing you to keep
+ track of the export.
+
+ Example:
+ >>> from google.cloud import asset_v1
+ >>>
+ >>> client = asset_v1.AssetServiceClient()
+ >>>
+ >>> # TODO: Initialize `parent`:
+ >>> parent = ''
+ >>>
+ >>> # TODO: Initialize `output_config`:
+ >>> output_config = {}
+ >>>
+ >>> response = client.export_assets(parent, output_config)
+ >>>
+ >>> def callback(operation_future):
+ ... # Handle result.
+ ... result = operation_future.result()
+ >>>
+ >>> response.add_done_callback(callback)
+ >>>
+ >>> # Handle metadata.
+ >>> metadata = response.metadata()
+
+ Args:
+ parent (str): Required. The relative name of the root asset. This can only be an
+ organization number (such as "organizations/123"), a project ID (such as
+ "projects/my-project-id"), or a project number (such as "projects/12345"),
+ or a folder number (such as "folders/123").
+ output_config (Union[dict, ~google.cloud.asset_v1.types.OutputConfig]): Required. Output configuration indicating where the results will be output
+ to. All results will be in newline delimited JSON format.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.asset_v1.types.OutputConfig`
+ read_time (Union[dict, ~google.cloud.asset_v1.types.Timestamp]): Timestamp to take an asset snapshot. This can only be set to a timestamp
+ between 2018-10-02 UTC (inclusive) and the current time. If not specified,
+ the current time will be used. Due to delays in resource data collection
+ and indexing, there is a volatile window during which running the same
+ query may get different results.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.asset_v1.types.Timestamp`
+ asset_types (list[str]): A list of asset types of which to take a snapshot for. For example:
+ "compute.googleapis.com/Disk". If specified, only matching assets will
+ be returned. See `Introduction to Cloud Asset
+ Inventory `__
+ for all supported asset types.
+ content_type (~google.cloud.asset_v1.types.ContentType): Asset content type. If not specified, no content but the asset name will be
+ returned.
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.asset_v1.types._OperationFuture` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "export_assets" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "export_assets"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.export_assets,
+ default_retry=self._method_configs["ExportAssets"].retry,
+ default_timeout=self._method_configs["ExportAssets"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = asset_service_pb2.ExportAssetsRequest(
+ parent=parent,
+ output_config=output_config,
+ read_time=read_time,
+ asset_types=asset_types,
+ content_type=content_type,
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("parent", parent)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ operation = self._inner_api_calls["export_assets"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+ return google.api_core.operation.from_gapic(
+ operation,
+ self.transport._operations_client,
+ asset_service_pb2.ExportAssetsResponse,
+ metadata_type=asset_service_pb2.ExportAssetsRequest,
+ )
+
+ def batch_get_assets_history(
+ self,
+ parent,
+ content_type,
+ read_time_window,
+ asset_names=None,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Batch gets the update history of assets that overlap a time window. For
+ RESOURCE content, this API outputs history with asset in both non-delete
+ or deleted status. For IAM\_POLICY content, this API outputs history
+ when the asset and its attached IAM POLICY both exist. This can create
+ gaps in the output history. If a specified asset does not exist, this
+ API returns an INVALID\_ARGUMENT error.
+
+ Example:
+ >>> from google.cloud import asset_v1
+ >>> from google.cloud.asset_v1 import enums
+ >>>
+ >>> client = asset_v1.AssetServiceClient()
+ >>>
+ >>> # TODO: Initialize `parent`:
+ >>> parent = ''
+ >>>
+ >>> # TODO: Initialize `content_type`:
+ >>> content_type = enums.ContentType.CONTENT_TYPE_UNSPECIFIED
+ >>>
+ >>> # TODO: Initialize `read_time_window`:
+ >>> read_time_window = {}
+ >>>
+ >>> response = client.batch_get_assets_history(parent, content_type, read_time_window)
+
+ Args:
+ parent (str): Required. The relative name of the root asset. It can only be an
+ organization number (such as "organizations/123"), a project ID (such as
+ "projects/my-project-id")", or a project number (such as "projects/12345").
+ content_type (~google.cloud.asset_v1.types.ContentType): Required. The content type.
+ read_time_window (Union[dict, ~google.cloud.asset_v1.types.TimeWindow]): Optional. The time window for the asset history. Both start\_time and
+ end\_time are optional and if set, it must be after 2018-10-02 UTC. If
+ end\_time is not set, it is default to current timestamp. If start\_time
+ is not set, the snapshot of the assets at end\_time will be returned.
+ The returned results contain all temporal assets whose time window
+ overlap with read\_time\_window.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.asset_v1.types.TimeWindow`
+ asset_names (list[str]): A list of the full names of the assets. For example:
+ ``//compute.googleapis.com/projects/my_project_123/zones/zone1/instances/instance1``.
+ See `Resource
+ Names `__
+ and `Resource Name
+ Format `__
+ for more info.
+
+ The request becomes a no-op if the asset name list is empty, and the max
+ size of the asset name list is 100 in one request.
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.asset_v1.types.BatchGetAssetsHistoryResponse` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "batch_get_assets_history" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "batch_get_assets_history"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.batch_get_assets_history,
+ default_retry=self._method_configs["BatchGetAssetsHistory"].retry,
+ default_timeout=self._method_configs["BatchGetAssetsHistory"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = asset_service_pb2.BatchGetAssetsHistoryRequest(
+ parent=parent,
+ content_type=content_type,
+ read_time_window=read_time_window,
+ asset_names=asset_names,
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("parent", parent)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ return self._inner_api_calls["batch_get_assets_history"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
diff --git a/asset/google/cloud/asset_v1/gapic/asset_service_client_config.py b/asset/google/cloud/asset_v1/gapic/asset_service_client_config.py
new file mode 100644
index 000000000000..885ad4f0968c
--- /dev/null
+++ b/asset/google/cloud/asset_v1/gapic/asset_service_client_config.py
@@ -0,0 +1,33 @@
+config = {
+ "interfaces": {
+ "google.cloud.asset.v1.AssetService": {
+ "retry_codes": {
+ "idempotent": ["DEADLINE_EXCEEDED", "UNAVAILABLE"],
+ "non_idempotent": [],
+ },
+ "retry_params": {
+ "default": {
+ "initial_retry_delay_millis": 100,
+ "retry_delay_multiplier": 1.3,
+ "max_retry_delay_millis": 60000,
+ "initial_rpc_timeout_millis": 20000,
+ "rpc_timeout_multiplier": 1.0,
+ "max_rpc_timeout_millis": 20000,
+ "total_timeout_millis": 600000,
+ }
+ },
+ "methods": {
+ "ExportAssets": {
+ "timeout_millis": 600000,
+ "retry_codes_name": "non_idempotent",
+ "retry_params_name": "default",
+ },
+ "BatchGetAssetsHistory": {
+ "timeout_millis": 600000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ },
+ }
+ }
+}
diff --git a/asset/google/cloud/asset_v1/gapic/enums.py b/asset/google/cloud/asset_v1/gapic/enums.py
new file mode 100644
index 000000000000..780beae4ddef
--- /dev/null
+++ b/asset/google/cloud/asset_v1/gapic/enums.py
@@ -0,0 +1,48 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Wrappers for protocol buffer enum types."""
+
+import enum
+
+
+class ContentType(enum.IntEnum):
+ """
+ Asset content type.
+
+ Attributes:
+ CONTENT_TYPE_UNSPECIFIED (int): Unspecified content type.
+ RESOURCE (int): Resource metadata.
+ IAM_POLICY (int): The actual IAM policy set on a resource.
+ """
+
+ CONTENT_TYPE_UNSPECIFIED = 0
+ RESOURCE = 1
+ IAM_POLICY = 2
+
+
+class NullValue(enum.IntEnum):
+ """
+ ``NullValue`` is a singleton enumeration to represent the null value for
+ the ``Value`` type union.
+
+ The JSON representation for ``NullValue`` is JSON ``null``.
+
+ Attributes:
+ NULL_VALUE (int): Null value.
+ """
+
+ NULL_VALUE = 0
diff --git a/asset/google/cloud/asset_v1/gapic/transports/__init__.py b/asset/google/cloud/asset_v1/gapic/transports/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/asset/google/cloud/asset_v1/gapic/transports/asset_service_grpc_transport.py b/asset/google/cloud/asset_v1/gapic/transports/asset_service_grpc_transport.py
new file mode 100644
index 000000000000..b6849225d4f5
--- /dev/null
+++ b/asset/google/cloud/asset_v1/gapic/transports/asset_service_grpc_transport.py
@@ -0,0 +1,150 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import google.api_core.grpc_helpers
+import google.api_core.operations_v1
+
+from google.cloud.asset_v1.proto import asset_service_pb2_grpc
+
+
+class AssetServiceGrpcTransport(object):
+ """gRPC transport class providing stubs for
+ google.cloud.asset.v1 AssetService API.
+
+ The transport provides access to the raw gRPC stubs,
+ which can be used to take advantage of advanced
+ features of gRPC.
+ """
+
+ # The scopes needed to make gRPC calls to all of the methods defined
+ # in this service.
+ _OAUTH_SCOPES = ("https://www.googleapis.com/auth/cloud-platform",)
+
+ def __init__(
+ self, channel=None, credentials=None, address="cloudasset.googleapis.com:443"
+ ):
+ """Instantiate the transport class.
+
+ Args:
+ channel (grpc.Channel): A ``Channel`` instance through
+ which to make calls. This argument is mutually exclusive
+ with ``credentials``; providing both will raise an exception.
+ credentials (google.auth.credentials.Credentials): The
+ authorization credentials to attach to requests. These
+ credentials identify this application to the service. If none
+ are specified, the client will attempt to ascertain the
+ credentials from the environment.
+ address (str): The address where the service is hosted.
+ """
+ # If both `channel` and `credentials` are specified, raise an
+ # exception (channels come with credentials baked in already).
+ if channel is not None and credentials is not None:
+ raise ValueError(
+ "The `channel` and `credentials` arguments are mutually " "exclusive."
+ )
+
+ # Create the channel.
+ if channel is None:
+ channel = self.create_channel(
+ address=address,
+ credentials=credentials,
+ options={
+ "grpc.max_send_message_length": -1,
+ "grpc.max_receive_message_length": -1,
+ }.items(),
+ )
+
+ self._channel = channel
+
+ # gRPC uses objects called "stubs" that are bound to the
+ # channel and provide a basic method for each RPC.
+ self._stubs = {
+ "asset_service_stub": asset_service_pb2_grpc.AssetServiceStub(channel)
+ }
+
+ # Because this API includes a method that returns a
+ # long-running operation (proto: google.longrunning.Operation),
+ # instantiate an LRO client.
+ self._operations_client = google.api_core.operations_v1.OperationsClient(
+ channel
+ )
+
+ @classmethod
+ def create_channel(
+ cls, address="cloudasset.googleapis.com:443", credentials=None, **kwargs
+ ):
+ """Create and return a gRPC channel object.
+
+ Args:
+ address (str): The host for the channel to use.
+ credentials (~.Credentials): The
+ authorization credentials to attach to requests. These
+ credentials identify this application to the service. If
+ none are specified, the client will attempt to ascertain
+ the credentials from the environment.
+ kwargs (dict): Keyword arguments, which are passed to the
+ channel creation.
+
+ Returns:
+ grpc.Channel: A gRPC channel object.
+ """
+ return google.api_core.grpc_helpers.create_channel(
+ address, credentials=credentials, scopes=cls._OAUTH_SCOPES, **kwargs
+ )
+
+ @property
+ def channel(self):
+ """The gRPC channel used by the transport.
+
+ Returns:
+ grpc.Channel: A gRPC channel object.
+ """
+ return self._channel
+
+ @property
+ def export_assets(self):
+ """Return the gRPC stub for :meth:`AssetServiceClient.export_assets`.
+
+ Exports assets with time and resource types to a given Cloud Storage
+ location. The output format is newline-delimited JSON. This API
+ implements the ``google.longrunning.Operation`` API allowing you to keep
+ track of the export.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["asset_service_stub"].ExportAssets
+
+ @property
+ def batch_get_assets_history(self):
+ """Return the gRPC stub for :meth:`AssetServiceClient.batch_get_assets_history`.
+
+ Batch gets the update history of assets that overlap a time window. For
+ RESOURCE content, this API outputs history with asset in both non-delete
+ or deleted status. For IAM\_POLICY content, this API outputs history
+ when the asset and its attached IAM POLICY both exist. This can create
+ gaps in the output history. If a specified asset does not exist, this
+ API returns an INVALID\_ARGUMENT error.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["asset_service_stub"].BatchGetAssetsHistory
diff --git a/asset/google/cloud/asset_v1/proto/__init__.py b/asset/google/cloud/asset_v1/proto/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/asset/google/cloud/asset_v1/proto/asset_service.proto b/asset/google/cloud/asset_v1/proto/asset_service.proto
new file mode 100644
index 000000000000..0dfc2898b131
--- /dev/null
+++ b/asset/google/cloud/asset_v1/proto/asset_service.proto
@@ -0,0 +1,171 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.asset.v1;
+
+import "google/api/annotations.proto";
+import "google/cloud/asset/v1/assets.proto";
+import "google/longrunning/operations.proto";
+import "google/protobuf/timestamp.proto";
+
+option csharp_namespace = "Google.Cloud.Asset.V1";
+option go_package = "google.golang.org/genproto/googleapis/cloud/asset/v1;asset";
+option java_multiple_files = true;
+option java_outer_classname = "AssetServiceProto";
+option java_package = "com.google.cloud.asset.v1";
+option php_namespace = "Google\\Cloud\\Asset\\V1";
+
+
+// Asset service definition.
+service AssetService {
+ // Exports assets with time and resource types to a given Cloud Storage
+ // location. The output format is newline-delimited JSON.
+ // This API implements the [google.longrunning.Operation][google.longrunning.Operation] API allowing you
+ // to keep track of the export.
+ rpc ExportAssets(ExportAssetsRequest) returns (google.longrunning.Operation) {
+ option (google.api.http) = {
+ post: "/v1/{parent=*/*}:exportAssets"
+ body: "*"
+ };
+ }
+
+ // Batch gets the update history of assets that overlap a time window.
+ // For RESOURCE content, this API outputs history with asset in both
+ // non-delete or deleted status.
+ // For IAM_POLICY content, this API outputs history when the asset and its
+ // attached IAM POLICY both exist. This can create gaps in the output history.
+ // If a specified asset does not exist, this API returns an INVALID_ARGUMENT
+ // error.
+ rpc BatchGetAssetsHistory(BatchGetAssetsHistoryRequest) returns (BatchGetAssetsHistoryResponse) {
+ option (google.api.http) = {
+ get: "/v1/{parent=*/*}:batchGetAssetsHistory"
+ };
+ }
+}
+
+// Export asset request.
+message ExportAssetsRequest {
+ // Required. The relative name of the root asset. This can only be an
+ // organization number (such as "organizations/123"), a project ID (such as
+ // "projects/my-project-id"), or a project number (such as "projects/12345"),
+ // or a folder number (such as "folders/123").
+ string parent = 1;
+
+ // Timestamp to take an asset snapshot. This can only be set to a timestamp
+ // between 2018-10-02 UTC (inclusive) and the current time. If not specified,
+ // the current time will be used. Due to delays in resource data collection
+ // and indexing, there is a volatile window during which running the same
+ // query may get different results.
+ google.protobuf.Timestamp read_time = 2;
+
+ // A list of asset types of which to take a snapshot for. For example:
+ // "compute.googleapis.com/Disk". If specified, only matching assets will be returned.
+ // See [Introduction to Cloud Asset
+ // Inventory](https://cloud.google.com/resource-manager/docs/cloud-asset-inventory/overview)
+ // for all supported asset types.
+ repeated string asset_types = 3;
+
+ // Asset content type. If not specified, no content but the asset name will be
+ // returned.
+ ContentType content_type = 4;
+
+ // Required. Output configuration indicating where the results will be output
+ // to. All results will be in newline delimited JSON format.
+ OutputConfig output_config = 5;
+}
+
+// The export asset response. This message is returned by the
+// [google.longrunning.Operations.GetOperation][google.longrunning.Operations.GetOperation] method in the returned
+// [google.longrunning.Operation.response][google.longrunning.Operation.response] field.
+message ExportAssetsResponse {
+ // Time the snapshot was taken.
+ google.protobuf.Timestamp read_time = 1;
+
+ // Output configuration indicating where the results were output to.
+ // All results are in JSON format.
+ OutputConfig output_config = 2;
+}
+
+// Batch get assets history request.
+message BatchGetAssetsHistoryRequest {
+ // Required. The relative name of the root asset. It can only be an
+ // organization number (such as "organizations/123"), a project ID (such as
+ // "projects/my-project-id")", or a project number (such as "projects/12345").
+ string parent = 1;
+
+ // A list of the full names of the assets. For example:
+ // `//compute.googleapis.com/projects/my_project_123/zones/zone1/instances/instance1`.
+ // See [Resource
+ // Names](https://cloud.google.com/apis/design/resource_names#full_resource_name)
+ // and [Resource Name Format](https://cloud.google.com/resource-manager/docs/cloud-asset-inventory/resource-name-format)
+ // for more info.
+ //
+ // The request becomes a no-op if the asset name list is empty, and the max
+ // size of the asset name list is 100 in one request.
+ repeated string asset_names = 2;
+
+ // Required. The content type.
+ ContentType content_type = 3;
+
+ // Optional. The time window for the asset history. Both start_time and
+ // end_time are optional and if set, it must be after 2018-10-02 UTC. If
+ // end_time is not set, it is default to current timestamp. If start_time is
+ // not set, the snapshot of the assets at end_time will be returned. The
+ // returned results contain all temporal assets whose time window overlap with
+ // read_time_window.
+ TimeWindow read_time_window = 4;
+}
+
+// Batch get assets history response.
+message BatchGetAssetsHistoryResponse {
+ // A list of assets with valid time windows.
+ repeated TemporalAsset assets = 1;
+}
+
+// Output configuration for export assets destination.
+message OutputConfig {
+ // Asset export destination.
+ oneof destination {
+ // Destination on Cloud Storage.
+ GcsDestination gcs_destination = 1;
+ }
+}
+
+// A Cloud Storage location.
+message GcsDestination {
+ // Required.
+ oneof object_uri {
+ // The uri of the Cloud Storage object. It's the same uri that is used by
+ // gsutil. For example: "gs://bucket_name/object_name". See [Viewing and
+ // Editing Object
+ // Metadata](https://cloud.google.com/storage/docs/viewing-editing-metadata)
+ // for more information.
+ string uri = 1;
+ }
+}
+
+// Asset content type.
+enum ContentType {
+ // Unspecified content type.
+ CONTENT_TYPE_UNSPECIFIED = 0;
+
+ // Resource metadata.
+ RESOURCE = 1;
+
+ // The actual IAM policy set on a resource.
+ IAM_POLICY = 2;
+}
diff --git a/asset/google/cloud/asset_v1/proto/asset_service_pb2.py b/asset/google/cloud/asset_v1/proto/asset_service_pb2.py
new file mode 100644
index 000000000000..869d4dc3fee1
--- /dev/null
+++ b/asset/google/cloud/asset_v1/proto/asset_service_pb2.py
@@ -0,0 +1,726 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/asset_v1/proto/asset_service.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf.internal import enum_type_wrapper
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+from google.cloud.asset_v1.proto import (
+ assets_pb2 as google_dot_cloud_dot_asset__v1_dot_proto_dot_assets__pb2,
+)
+from google.longrunning import (
+ operations_pb2 as google_dot_longrunning_dot_operations__pb2,
+)
+from google.protobuf import timestamp_pb2 as google_dot_protobuf_dot_timestamp__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/asset_v1/proto/asset_service.proto",
+ package="google.cloud.asset.v1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\031com.google.cloud.asset.v1B\021AssetServiceProtoP\001Z:google.golang.org/genproto/googleapis/cloud/asset/v1;asset\252\002\025Google.Cloud.Asset.V1\312\002\025Google\\Cloud\\Asset\\V1"
+ ),
+ serialized_pb=_b(
+ '\n/google/cloud/asset_v1/proto/asset_service.proto\x12\x15google.cloud.asset.v1\x1a\x1cgoogle/api/annotations.proto\x1a(google/cloud/asset_v1/proto/assets.proto\x1a#google/longrunning/operations.proto\x1a\x1fgoogle/protobuf/timestamp.proto"\xdf\x01\n\x13\x45xportAssetsRequest\x12\x0e\n\x06parent\x18\x01 \x01(\t\x12-\n\tread_time\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12\x13\n\x0b\x61sset_types\x18\x03 \x03(\t\x12\x38\n\x0c\x63ontent_type\x18\x04 \x01(\x0e\x32".google.cloud.asset.v1.ContentType\x12:\n\routput_config\x18\x05 \x01(\x0b\x32#.google.cloud.asset.v1.OutputConfig"\x81\x01\n\x14\x45xportAssetsResponse\x12-\n\tread_time\x18\x01 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12:\n\routput_config\x18\x02 \x01(\x0b\x32#.google.cloud.asset.v1.OutputConfig"\xba\x01\n\x1c\x42\x61tchGetAssetsHistoryRequest\x12\x0e\n\x06parent\x18\x01 \x01(\t\x12\x13\n\x0b\x61sset_names\x18\x02 \x03(\t\x12\x38\n\x0c\x63ontent_type\x18\x03 \x01(\x0e\x32".google.cloud.asset.v1.ContentType\x12;\n\x10read_time_window\x18\x04 \x01(\x0b\x32!.google.cloud.asset.v1.TimeWindow"U\n\x1d\x42\x61tchGetAssetsHistoryResponse\x12\x34\n\x06\x61ssets\x18\x01 \x03(\x0b\x32$.google.cloud.asset.v1.TemporalAsset"_\n\x0cOutputConfig\x12@\n\x0fgcs_destination\x18\x01 \x01(\x0b\x32%.google.cloud.asset.v1.GcsDestinationH\x00\x42\r\n\x0b\x64\x65stination"-\n\x0eGcsDestination\x12\r\n\x03uri\x18\x01 \x01(\tH\x00\x42\x0c\n\nobject_uri*I\n\x0b\x43ontentType\x12\x1c\n\x18\x43ONTENT_TYPE_UNSPECIFIED\x10\x00\x12\x0c\n\x08RESOURCE\x10\x01\x12\x0e\n\nIAM_POLICY\x10\x02\x32\xc9\x02\n\x0c\x41ssetService\x12\x83\x01\n\x0c\x45xportAssets\x12*.google.cloud.asset.v1.ExportAssetsRequest\x1a\x1d.google.longrunning.Operation"(\x82\xd3\xe4\x93\x02""\x1d/v1/{parent=*/*}:exportAssets:\x01*\x12\xb2\x01\n\x15\x42\x61tchGetAssetsHistory\x12\x33.google.cloud.asset.v1.BatchGetAssetsHistoryRequest\x1a\x34.google.cloud.asset.v1.BatchGetAssetsHistoryResponse".\x82\xd3\xe4\x93\x02(\x12&/v1/{parent=*/*}:batchGetAssetsHistoryB\x9c\x01\n\x19\x63om.google.cloud.asset.v1B\x11\x41ssetServiceProtoP\x01Z:google.golang.org/genproto/googleapis/cloud/asset/v1;asset\xaa\x02\x15Google.Cloud.Asset.V1\xca\x02\x15Google\\Cloud\\Asset\\V1b\x06proto3'
+ ),
+ dependencies=[
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_asset__v1_dot_proto_dot_assets__pb2.DESCRIPTOR,
+ google_dot_longrunning_dot_operations__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_timestamp__pb2.DESCRIPTOR,
+ ],
+)
+
+_CONTENTTYPE = _descriptor.EnumDescriptor(
+ name="ContentType",
+ full_name="google.cloud.asset.v1.ContentType",
+ filename=None,
+ file=DESCRIPTOR,
+ values=[
+ _descriptor.EnumValueDescriptor(
+ name="CONTENT_TYPE_UNSPECIFIED",
+ index=0,
+ number=0,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="RESOURCE", index=1, number=1, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="IAM_POLICY", index=2, number=2, serialized_options=None, type=None
+ ),
+ ],
+ containing_type=None,
+ serialized_options=None,
+ serialized_start=994,
+ serialized_end=1067,
+)
+_sym_db.RegisterEnumDescriptor(_CONTENTTYPE)
+
+ContentType = enum_type_wrapper.EnumTypeWrapper(_CONTENTTYPE)
+CONTENT_TYPE_UNSPECIFIED = 0
+RESOURCE = 1
+IAM_POLICY = 2
+
+
+_EXPORTASSETSREQUEST = _descriptor.Descriptor(
+ name="ExportAssetsRequest",
+ full_name="google.cloud.asset.v1.ExportAssetsRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="parent",
+ full_name="google.cloud.asset.v1.ExportAssetsRequest.parent",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="read_time",
+ full_name="google.cloud.asset.v1.ExportAssetsRequest.read_time",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="asset_types",
+ full_name="google.cloud.asset.v1.ExportAssetsRequest.asset_types",
+ index=2,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="content_type",
+ full_name="google.cloud.asset.v1.ExportAssetsRequest.content_type",
+ index=3,
+ number=4,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="output_config",
+ full_name="google.cloud.asset.v1.ExportAssetsRequest.output_config",
+ index=4,
+ number=5,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=217,
+ serialized_end=440,
+)
+
+
+_EXPORTASSETSRESPONSE = _descriptor.Descriptor(
+ name="ExportAssetsResponse",
+ full_name="google.cloud.asset.v1.ExportAssetsResponse",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="read_time",
+ full_name="google.cloud.asset.v1.ExportAssetsResponse.read_time",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="output_config",
+ full_name="google.cloud.asset.v1.ExportAssetsResponse.output_config",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=443,
+ serialized_end=572,
+)
+
+
+_BATCHGETASSETSHISTORYREQUEST = _descriptor.Descriptor(
+ name="BatchGetAssetsHistoryRequest",
+ full_name="google.cloud.asset.v1.BatchGetAssetsHistoryRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="parent",
+ full_name="google.cloud.asset.v1.BatchGetAssetsHistoryRequest.parent",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="asset_names",
+ full_name="google.cloud.asset.v1.BatchGetAssetsHistoryRequest.asset_names",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="content_type",
+ full_name="google.cloud.asset.v1.BatchGetAssetsHistoryRequest.content_type",
+ index=2,
+ number=3,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="read_time_window",
+ full_name="google.cloud.asset.v1.BatchGetAssetsHistoryRequest.read_time_window",
+ index=3,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=575,
+ serialized_end=761,
+)
+
+
+_BATCHGETASSETSHISTORYRESPONSE = _descriptor.Descriptor(
+ name="BatchGetAssetsHistoryResponse",
+ full_name="google.cloud.asset.v1.BatchGetAssetsHistoryResponse",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="assets",
+ full_name="google.cloud.asset.v1.BatchGetAssetsHistoryResponse.assets",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=763,
+ serialized_end=848,
+)
+
+
+_OUTPUTCONFIG = _descriptor.Descriptor(
+ name="OutputConfig",
+ full_name="google.cloud.asset.v1.OutputConfig",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="gcs_destination",
+ full_name="google.cloud.asset.v1.OutputConfig.gcs_destination",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="destination",
+ full_name="google.cloud.asset.v1.OutputConfig.destination",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=850,
+ serialized_end=945,
+)
+
+
+_GCSDESTINATION = _descriptor.Descriptor(
+ name="GcsDestination",
+ full_name="google.cloud.asset.v1.GcsDestination",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="uri",
+ full_name="google.cloud.asset.v1.GcsDestination.uri",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="object_uri",
+ full_name="google.cloud.asset.v1.GcsDestination.object_uri",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=947,
+ serialized_end=992,
+)
+
+_EXPORTASSETSREQUEST.fields_by_name[
+ "read_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_EXPORTASSETSREQUEST.fields_by_name["content_type"].enum_type = _CONTENTTYPE
+_EXPORTASSETSREQUEST.fields_by_name["output_config"].message_type = _OUTPUTCONFIG
+_EXPORTASSETSRESPONSE.fields_by_name[
+ "read_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_EXPORTASSETSRESPONSE.fields_by_name["output_config"].message_type = _OUTPUTCONFIG
+_BATCHGETASSETSHISTORYREQUEST.fields_by_name["content_type"].enum_type = _CONTENTTYPE
+_BATCHGETASSETSHISTORYREQUEST.fields_by_name[
+ "read_time_window"
+].message_type = google_dot_cloud_dot_asset__v1_dot_proto_dot_assets__pb2._TIMEWINDOW
+_BATCHGETASSETSHISTORYRESPONSE.fields_by_name[
+ "assets"
+].message_type = google_dot_cloud_dot_asset__v1_dot_proto_dot_assets__pb2._TEMPORALASSET
+_OUTPUTCONFIG.fields_by_name["gcs_destination"].message_type = _GCSDESTINATION
+_OUTPUTCONFIG.oneofs_by_name["destination"].fields.append(
+ _OUTPUTCONFIG.fields_by_name["gcs_destination"]
+)
+_OUTPUTCONFIG.fields_by_name[
+ "gcs_destination"
+].containing_oneof = _OUTPUTCONFIG.oneofs_by_name["destination"]
+_GCSDESTINATION.oneofs_by_name["object_uri"].fields.append(
+ _GCSDESTINATION.fields_by_name["uri"]
+)
+_GCSDESTINATION.fields_by_name["uri"].containing_oneof = _GCSDESTINATION.oneofs_by_name[
+ "object_uri"
+]
+DESCRIPTOR.message_types_by_name["ExportAssetsRequest"] = _EXPORTASSETSREQUEST
+DESCRIPTOR.message_types_by_name["ExportAssetsResponse"] = _EXPORTASSETSRESPONSE
+DESCRIPTOR.message_types_by_name[
+ "BatchGetAssetsHistoryRequest"
+] = _BATCHGETASSETSHISTORYREQUEST
+DESCRIPTOR.message_types_by_name[
+ "BatchGetAssetsHistoryResponse"
+] = _BATCHGETASSETSHISTORYRESPONSE
+DESCRIPTOR.message_types_by_name["OutputConfig"] = _OUTPUTCONFIG
+DESCRIPTOR.message_types_by_name["GcsDestination"] = _GCSDESTINATION
+DESCRIPTOR.enum_types_by_name["ContentType"] = _CONTENTTYPE
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+ExportAssetsRequest = _reflection.GeneratedProtocolMessageType(
+ "ExportAssetsRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_EXPORTASSETSREQUEST,
+ __module__="google.cloud.asset_v1.proto.asset_service_pb2",
+ __doc__="""Export asset request.
+
+
+ Attributes:
+ parent:
+ Required. The relative name of the root asset. This can only
+ be an organization number (such as "organizations/123"), a
+ project ID (such as "projects/my-project-id"), or a project
+ number (such as "projects/12345"), or a folder number (such as
+ "folders/123").
+ read_time:
+ Timestamp to take an asset snapshot. This can only be set to a
+ timestamp between 2018-10-02 UTC (inclusive) and the current
+ time. If not specified, the current time will be used. Due to
+ delays in resource data collection and indexing, there is a
+ volatile window during which running the same query may get
+ different results.
+ asset_types:
+ A list of asset types of which to take a snapshot for. For
+ example: "compute.googleapis.com/Disk". If specified, only
+ matching assets will be returned. See `Introduction to Cloud
+ Asset Inventory `__ for all
+ supported asset types.
+ content_type:
+ Asset content type. If not specified, no content but the asset
+ name will be returned.
+ output_config:
+ Required. Output configuration indicating where the results
+ will be output to. All results will be in newline delimited
+ JSON format.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1.ExportAssetsRequest)
+ ),
+)
+_sym_db.RegisterMessage(ExportAssetsRequest)
+
+ExportAssetsResponse = _reflection.GeneratedProtocolMessageType(
+ "ExportAssetsResponse",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_EXPORTASSETSRESPONSE,
+ __module__="google.cloud.asset_v1.proto.asset_service_pb2",
+ __doc__="""The export asset response. This message is returned by the
+ [google.longrunning.Operations.GetOperation][google.longrunning.Operations.GetOperation]
+ method in the returned
+ [google.longrunning.Operation.response][google.longrunning.Operation.response]
+ field.
+
+
+ Attributes:
+ read_time:
+ Time the snapshot was taken.
+ output_config:
+ Output configuration indicating where the results were output
+ to. All results are in JSON format.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1.ExportAssetsResponse)
+ ),
+)
+_sym_db.RegisterMessage(ExportAssetsResponse)
+
+BatchGetAssetsHistoryRequest = _reflection.GeneratedProtocolMessageType(
+ "BatchGetAssetsHistoryRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_BATCHGETASSETSHISTORYREQUEST,
+ __module__="google.cloud.asset_v1.proto.asset_service_pb2",
+ __doc__="""Batch get assets history request.
+
+
+ Attributes:
+ parent:
+ Required. The relative name of the root asset. It can only be
+ an organization number (such as "organizations/123"), a
+ project ID (such as "projects/my-project-id")", or a project
+ number (such as "projects/12345").
+ asset_names:
+ A list of the full names of the assets. For example: ``//compu
+ te.googleapis.com/projects/my_project_123/zones/zone1/instance
+ s/instance1``. See `Resource Names `__ and `Resource
+ Name Format `__
+ for more info. The request becomes a no-op if the asset name
+ list is empty, and the max size of the asset name list is 100
+ in one request.
+ content_type:
+ Required. The content type.
+ read_time_window:
+ Optional. The time window for the asset history. Both
+ start\_time and end\_time are optional and if set, it must be
+ after 2018-10-02 UTC. If end\_time is not set, it is default
+ to current timestamp. If start\_time is not set, the snapshot
+ of the assets at end\_time will be returned. The returned
+ results contain all temporal assets whose time window overlap
+ with read\_time\_window.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1.BatchGetAssetsHistoryRequest)
+ ),
+)
+_sym_db.RegisterMessage(BatchGetAssetsHistoryRequest)
+
+BatchGetAssetsHistoryResponse = _reflection.GeneratedProtocolMessageType(
+ "BatchGetAssetsHistoryResponse",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_BATCHGETASSETSHISTORYRESPONSE,
+ __module__="google.cloud.asset_v1.proto.asset_service_pb2",
+ __doc__="""Batch get assets history response.
+
+
+ Attributes:
+ assets:
+ A list of assets with valid time windows.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1.BatchGetAssetsHistoryResponse)
+ ),
+)
+_sym_db.RegisterMessage(BatchGetAssetsHistoryResponse)
+
+OutputConfig = _reflection.GeneratedProtocolMessageType(
+ "OutputConfig",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_OUTPUTCONFIG,
+ __module__="google.cloud.asset_v1.proto.asset_service_pb2",
+ __doc__="""Output configuration for export assets destination.
+
+
+ Attributes:
+ destination:
+ Asset export destination.
+ gcs_destination:
+ Destination on Cloud Storage.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1.OutputConfig)
+ ),
+)
+_sym_db.RegisterMessage(OutputConfig)
+
+GcsDestination = _reflection.GeneratedProtocolMessageType(
+ "GcsDestination",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_GCSDESTINATION,
+ __module__="google.cloud.asset_v1.proto.asset_service_pb2",
+ __doc__="""A Cloud Storage location.
+
+
+ Attributes:
+ object_uri:
+ Required.
+ uri:
+ The uri of the Cloud Storage object. It's the same uri that is
+ used by gsutil. For example: "gs://bucket\_name/object\_name".
+ See `Viewing and Editing Object Metadata
+ `__ for more information.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1.GcsDestination)
+ ),
+)
+_sym_db.RegisterMessage(GcsDestination)
+
+
+DESCRIPTOR._options = None
+
+_ASSETSERVICE = _descriptor.ServiceDescriptor(
+ name="AssetService",
+ full_name="google.cloud.asset.v1.AssetService",
+ file=DESCRIPTOR,
+ index=0,
+ serialized_options=None,
+ serialized_start=1070,
+ serialized_end=1399,
+ methods=[
+ _descriptor.MethodDescriptor(
+ name="ExportAssets",
+ full_name="google.cloud.asset.v1.AssetService.ExportAssets",
+ index=0,
+ containing_service=None,
+ input_type=_EXPORTASSETSREQUEST,
+ output_type=google_dot_longrunning_dot_operations__pb2._OPERATION,
+ serialized_options=_b(
+ '\202\323\344\223\002""\035/v1/{parent=*/*}:exportAssets:\001*'
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="BatchGetAssetsHistory",
+ full_name="google.cloud.asset.v1.AssetService.BatchGetAssetsHistory",
+ index=1,
+ containing_service=None,
+ input_type=_BATCHGETASSETSHISTORYREQUEST,
+ output_type=_BATCHGETASSETSHISTORYRESPONSE,
+ serialized_options=_b(
+ "\202\323\344\223\002(\022&/v1/{parent=*/*}:batchGetAssetsHistory"
+ ),
+ ),
+ ],
+)
+_sym_db.RegisterServiceDescriptor(_ASSETSERVICE)
+
+DESCRIPTOR.services_by_name["AssetService"] = _ASSETSERVICE
+
+# @@protoc_insertion_point(module_scope)
diff --git a/asset/google/cloud/asset_v1/proto/asset_service_pb2_grpc.py b/asset/google/cloud/asset_v1/proto/asset_service_pb2_grpc.py
new file mode 100644
index 000000000000..3795a401928c
--- /dev/null
+++ b/asset/google/cloud/asset_v1/proto/asset_service_pb2_grpc.py
@@ -0,0 +1,78 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
+
+from google.cloud.asset_v1.proto import (
+ asset_service_pb2 as google_dot_cloud_dot_asset__v1_dot_proto_dot_asset__service__pb2,
+)
+from google.longrunning import (
+ operations_pb2 as google_dot_longrunning_dot_operations__pb2,
+)
+
+
+class AssetServiceStub(object):
+ """Asset service definition.
+ """
+
+ def __init__(self, channel):
+ """Constructor.
+
+ Args:
+ channel: A grpc.Channel.
+ """
+ self.ExportAssets = channel.unary_unary(
+ "/google.cloud.asset.v1.AssetService/ExportAssets",
+ request_serializer=google_dot_cloud_dot_asset__v1_dot_proto_dot_asset__service__pb2.ExportAssetsRequest.SerializeToString,
+ response_deserializer=google_dot_longrunning_dot_operations__pb2.Operation.FromString,
+ )
+ self.BatchGetAssetsHistory = channel.unary_unary(
+ "/google.cloud.asset.v1.AssetService/BatchGetAssetsHistory",
+ request_serializer=google_dot_cloud_dot_asset__v1_dot_proto_dot_asset__service__pb2.BatchGetAssetsHistoryRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_asset__v1_dot_proto_dot_asset__service__pb2.BatchGetAssetsHistoryResponse.FromString,
+ )
+
+
+class AssetServiceServicer(object):
+ """Asset service definition.
+ """
+
+ def ExportAssets(self, request, context):
+ """Exports assets with time and resource types to a given Cloud Storage
+ location. The output format is newline-delimited JSON.
+ This API implements the [google.longrunning.Operation][google.longrunning.Operation] API allowing you
+ to keep track of the export.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def BatchGetAssetsHistory(self, request, context):
+ """Batch gets the update history of assets that overlap a time window.
+ For RESOURCE content, this API outputs history with asset in both
+ non-delete or deleted status.
+ For IAM_POLICY content, this API outputs history when the asset and its
+ attached IAM POLICY both exist. This can create gaps in the output history.
+ If a specified asset does not exist, this API returns an INVALID_ARGUMENT
+ error.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+
+def add_AssetServiceServicer_to_server(servicer, server):
+ rpc_method_handlers = {
+ "ExportAssets": grpc.unary_unary_rpc_method_handler(
+ servicer.ExportAssets,
+ request_deserializer=google_dot_cloud_dot_asset__v1_dot_proto_dot_asset__service__pb2.ExportAssetsRequest.FromString,
+ response_serializer=google_dot_longrunning_dot_operations__pb2.Operation.SerializeToString,
+ ),
+ "BatchGetAssetsHistory": grpc.unary_unary_rpc_method_handler(
+ servicer.BatchGetAssetsHistory,
+ request_deserializer=google_dot_cloud_dot_asset__v1_dot_proto_dot_asset__service__pb2.BatchGetAssetsHistoryRequest.FromString,
+ response_serializer=google_dot_cloud_dot_asset__v1_dot_proto_dot_asset__service__pb2.BatchGetAssetsHistoryResponse.SerializeToString,
+ ),
+ }
+ generic_handler = grpc.method_handlers_generic_handler(
+ "google.cloud.asset.v1.AssetService", rpc_method_handlers
+ )
+ server.add_generic_rpc_handlers((generic_handler,))
diff --git a/asset/google/cloud/asset_v1/proto/assets.proto b/asset/google/cloud/asset_v1/proto/assets.proto
new file mode 100644
index 000000000000..f6a8108c0bd2
--- /dev/null
+++ b/asset/google/cloud/asset_v1/proto/assets.proto
@@ -0,0 +1,118 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.asset.v1;
+
+import "google/api/annotations.proto";
+import "google/iam/v1/policy.proto";
+import "google/protobuf/any.proto";
+import "google/protobuf/struct.proto";
+import "google/protobuf/timestamp.proto";
+
+option csharp_namespace = "Google.Cloud.Asset.V1";
+option go_package = "google.golang.org/genproto/googleapis/cloud/asset/v1;asset";
+option java_multiple_files = true;
+option java_outer_classname = "AssetProto";
+option java_package = "com.google.cloud.asset.v1";
+option php_namespace = "Google\\Cloud\\Asset\\V1";
+
+
+// Temporal asset. In addition to the asset, the temporal asset includes the
+// status of the asset and valid from and to time of it.
+message TemporalAsset {
+ // The time window when the asset data and state was observed.
+ TimeWindow window = 1;
+
+ // If the asset is deleted or not.
+ bool deleted = 2;
+
+ // Asset.
+ Asset asset = 3;
+}
+
+// A time window of (start_time, end_time].
+message TimeWindow {
+ // Start time of the time window (exclusive).
+ google.protobuf.Timestamp start_time = 1;
+
+ // End time of the time window (inclusive).
+ // Current timestamp if not specified.
+ google.protobuf.Timestamp end_time = 2;
+}
+
+// Cloud asset. This includes all Google Cloud Platform resources,
+// Cloud IAM policies, and other non-GCP assets.
+message Asset {
+ // The full name of the asset. For example:
+ // `//compute.googleapis.com/projects/my_project_123/zones/zone1/instances/instance1`.
+ // See [Resource
+ // Names](https://cloud.google.com/apis/design/resource_names#full_resource_name)
+ // for more information.
+ string name = 1;
+
+ // Type of the asset. Example: "compute.googleapis.com/Disk".
+ string asset_type = 2;
+
+ // Representation of the resource.
+ Resource resource = 3;
+
+ // Representation of the actual Cloud IAM policy set on a cloud resource. For
+ // each resource, there must be at most one Cloud IAM policy set on it.
+ google.iam.v1.Policy iam_policy = 4;
+}
+
+// Representation of a cloud resource.
+message Resource {
+ // The API version. Example: "v1".
+ string version = 1;
+
+ // The URL of the discovery document containing the resource's JSON schema.
+ // For example:
+ // `"https://www.googleapis.com/discovery/v1/apis/compute/v1/rest"`.
+ // It will be left unspecified for resources without a discovery-based API,
+ // such as Cloud Bigtable.
+ string discovery_document_uri = 2;
+
+ // The JSON schema name listed in the discovery document.
+ // Example: "Project". It will be left unspecified for resources (such as
+ // Cloud Bigtable) without a discovery-based API.
+ string discovery_name = 3;
+
+ // The REST URL for accessing the resource. An HTTP GET operation using this
+ // URL returns the resource itself.
+ // Example:
+ // `https://cloudresourcemanager.googleapis.com/v1/projects/my-project-123`.
+ // It will be left unspecified for resources without a REST API.
+ string resource_url = 4;
+
+ // The full name of the immediate parent of this resource. See
+ // [Resource
+ // Names](https://cloud.google.com/apis/design/resource_names#full_resource_name)
+ // for more information.
+ //
+ // For GCP assets, it is the parent resource defined in the [Cloud IAM policy
+ // hierarchy](https://cloud.google.com/iam/docs/overview#policy_hierarchy).
+ // For example:
+ // `"//cloudresourcemanager.googleapis.com/projects/my_project_123"`.
+ //
+ // For third-party assets, it is up to the users to define.
+ string parent = 5;
+
+ // The content of the resource, in which some sensitive fields are scrubbed
+ // away and may not be present.
+ google.protobuf.Struct data = 6;
+}
diff --git a/asset/google/cloud/asset_v1/proto/assets_pb2.py b/asset/google/cloud/asset_v1/proto/assets_pb2.py
new file mode 100644
index 000000000000..99fba5cde79d
--- /dev/null
+++ b/asset/google/cloud/asset_v1/proto/assets_pb2.py
@@ -0,0 +1,544 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/asset_v1/proto/assets.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+from google.iam.v1 import iam_policy_pb2 as google_dot_iam_dot_v1_dot_policy__pb2
+from google.protobuf import any_pb2 as google_dot_protobuf_dot_any__pb2
+from google.protobuf import struct_pb2 as google_dot_protobuf_dot_struct__pb2
+from google.protobuf import timestamp_pb2 as google_dot_protobuf_dot_timestamp__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/asset_v1/proto/assets.proto",
+ package="google.cloud.asset.v1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\031com.google.cloud.asset.v1B\nAssetProtoP\001Z:google.golang.org/genproto/googleapis/cloud/asset/v1;asset\252\002\025Google.Cloud.Asset.V1\312\002\025Google\\Cloud\\Asset\\V1"
+ ),
+ serialized_pb=_b(
+ '\n(google/cloud/asset_v1/proto/assets.proto\x12\x15google.cloud.asset.v1\x1a\x1cgoogle/api/annotations.proto\x1a\x1agoogle/iam/v1/policy.proto\x1a\x19google/protobuf/any.proto\x1a\x1cgoogle/protobuf/struct.proto\x1a\x1fgoogle/protobuf/timestamp.proto"\x80\x01\n\rTemporalAsset\x12\x31\n\x06window\x18\x01 \x01(\x0b\x32!.google.cloud.asset.v1.TimeWindow\x12\x0f\n\x07\x64\x65leted\x18\x02 \x01(\x08\x12+\n\x05\x61sset\x18\x03 \x01(\x0b\x32\x1c.google.cloud.asset.v1.Asset"j\n\nTimeWindow\x12.\n\nstart_time\x18\x01 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12,\n\x08\x65nd_time\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.Timestamp"\x87\x01\n\x05\x41sset\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x12\n\nasset_type\x18\x02 \x01(\t\x12\x31\n\x08resource\x18\x03 \x01(\x0b\x32\x1f.google.cloud.asset.v1.Resource\x12)\n\niam_policy\x18\x04 \x01(\x0b\x32\x15.google.iam.v1.Policy"\xa0\x01\n\x08Resource\x12\x0f\n\x07version\x18\x01 \x01(\t\x12\x1e\n\x16\x64iscovery_document_uri\x18\x02 \x01(\t\x12\x16\n\x0e\x64iscovery_name\x18\x03 \x01(\t\x12\x14\n\x0cresource_url\x18\x04 \x01(\t\x12\x0e\n\x06parent\x18\x05 \x01(\t\x12%\n\x04\x64\x61ta\x18\x06 \x01(\x0b\x32\x17.google.protobuf.StructB\x95\x01\n\x19\x63om.google.cloud.asset.v1B\nAssetProtoP\x01Z:google.golang.org/genproto/googleapis/cloud/asset/v1;asset\xaa\x02\x15Google.Cloud.Asset.V1\xca\x02\x15Google\\Cloud\\Asset\\V1b\x06proto3'
+ ),
+ dependencies=[
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ google_dot_iam_dot_v1_dot_policy__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_any__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_struct__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_timestamp__pb2.DESCRIPTOR,
+ ],
+)
+
+
+_TEMPORALASSET = _descriptor.Descriptor(
+ name="TemporalAsset",
+ full_name="google.cloud.asset.v1.TemporalAsset",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="window",
+ full_name="google.cloud.asset.v1.TemporalAsset.window",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="deleted",
+ full_name="google.cloud.asset.v1.TemporalAsset.deleted",
+ index=1,
+ number=2,
+ type=8,
+ cpp_type=7,
+ label=1,
+ has_default_value=False,
+ default_value=False,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="asset",
+ full_name="google.cloud.asset.v1.TemporalAsset.asset",
+ index=2,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=216,
+ serialized_end=344,
+)
+
+
+_TIMEWINDOW = _descriptor.Descriptor(
+ name="TimeWindow",
+ full_name="google.cloud.asset.v1.TimeWindow",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="start_time",
+ full_name="google.cloud.asset.v1.TimeWindow.start_time",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="end_time",
+ full_name="google.cloud.asset.v1.TimeWindow.end_time",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=346,
+ serialized_end=452,
+)
+
+
+_ASSET = _descriptor.Descriptor(
+ name="Asset",
+ full_name="google.cloud.asset.v1.Asset",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.asset.v1.Asset.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="asset_type",
+ full_name="google.cloud.asset.v1.Asset.asset_type",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="resource",
+ full_name="google.cloud.asset.v1.Asset.resource",
+ index=2,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="iam_policy",
+ full_name="google.cloud.asset.v1.Asset.iam_policy",
+ index=3,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=455,
+ serialized_end=590,
+)
+
+
+_RESOURCE = _descriptor.Descriptor(
+ name="Resource",
+ full_name="google.cloud.asset.v1.Resource",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="version",
+ full_name="google.cloud.asset.v1.Resource.version",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="discovery_document_uri",
+ full_name="google.cloud.asset.v1.Resource.discovery_document_uri",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="discovery_name",
+ full_name="google.cloud.asset.v1.Resource.discovery_name",
+ index=2,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="resource_url",
+ full_name="google.cloud.asset.v1.Resource.resource_url",
+ index=3,
+ number=4,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="parent",
+ full_name="google.cloud.asset.v1.Resource.parent",
+ index=4,
+ number=5,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="data",
+ full_name="google.cloud.asset.v1.Resource.data",
+ index=5,
+ number=6,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=593,
+ serialized_end=753,
+)
+
+_TEMPORALASSET.fields_by_name["window"].message_type = _TIMEWINDOW
+_TEMPORALASSET.fields_by_name["asset"].message_type = _ASSET
+_TIMEWINDOW.fields_by_name[
+ "start_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_TIMEWINDOW.fields_by_name[
+ "end_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_ASSET.fields_by_name["resource"].message_type = _RESOURCE
+_ASSET.fields_by_name[
+ "iam_policy"
+].message_type = (
+ google_dot_iam_dot_v1_dot_policy__pb2.google_dot_iam_dot_v1_dot_policy__pb2._POLICY
+)
+_RESOURCE.fields_by_name[
+ "data"
+].message_type = google_dot_protobuf_dot_struct__pb2._STRUCT
+DESCRIPTOR.message_types_by_name["TemporalAsset"] = _TEMPORALASSET
+DESCRIPTOR.message_types_by_name["TimeWindow"] = _TIMEWINDOW
+DESCRIPTOR.message_types_by_name["Asset"] = _ASSET
+DESCRIPTOR.message_types_by_name["Resource"] = _RESOURCE
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+TemporalAsset = _reflection.GeneratedProtocolMessageType(
+ "TemporalAsset",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TEMPORALASSET,
+ __module__="google.cloud.asset_v1.proto.assets_pb2",
+ __doc__="""Temporal asset. In addition to the asset, the temporal asset includes
+ the status of the asset and valid from and to time of it.
+
+
+ Attributes:
+ window:
+ The time window when the asset data and state was observed.
+ deleted:
+ If the asset is deleted or not.
+ asset:
+ Asset.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1.TemporalAsset)
+ ),
+)
+_sym_db.RegisterMessage(TemporalAsset)
+
+TimeWindow = _reflection.GeneratedProtocolMessageType(
+ "TimeWindow",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TIMEWINDOW,
+ __module__="google.cloud.asset_v1.proto.assets_pb2",
+ __doc__="""A time window of (start\_time, end\_time].
+
+
+ Attributes:
+ start_time:
+ Start time of the time window (exclusive).
+ end_time:
+ End time of the time window (inclusive). Current timestamp if
+ not specified.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1.TimeWindow)
+ ),
+)
+_sym_db.RegisterMessage(TimeWindow)
+
+Asset = _reflection.GeneratedProtocolMessageType(
+ "Asset",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_ASSET,
+ __module__="google.cloud.asset_v1.proto.assets_pb2",
+ __doc__="""Cloud asset. This includes all Google Cloud Platform resources, Cloud
+ IAM policies, and other non-GCP assets.
+
+
+ Attributes:
+ name:
+
+ The full name of the asset. For example:
+ ``//compute.googleapis.com/projects/my_project_123/zones/zone1/instances/instance1``.
+ See https://cloud.google.com/apis/design/resource_names#full_resource_name
+ for more information.
+ asset_type:
+ Type of the asset. Example: "compute.googleapis.com/Disk".
+ resource:
+ Representation of the resource.
+ iam_policy:
+ Representation of the actual Cloud IAM policy set on a cloud
+ resource. For each resource, there must be at most one Cloud
+ IAM policy set on it.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1.Asset)
+ ),
+)
+_sym_db.RegisterMessage(Asset)
+
+Resource = _reflection.GeneratedProtocolMessageType(
+ "Resource",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_RESOURCE,
+ __module__="google.cloud.asset_v1.proto.assets_pb2",
+ __doc__="""Representation of a cloud resource.
+
+
+ Attributes:
+ version:
+ The API version. Example: "v1".
+ discovery_document_uri:
+ The URL of the discovery document containing the resource's
+ JSON schema. For example: ``"https://www.googleapis.com/discov
+ ery/v1/apis/compute/v1/rest"``. It will be left unspecified
+ for resources without a discovery-based API, such as Cloud
+ Bigtable.
+ discovery_name:
+ The JSON schema name listed in the discovery document.
+ Example: "Project". It will be left unspecified for resources
+ (such as Cloud Bigtable) without a discovery-based API.
+ resource_url:
+ The REST URL for accessing the resource. An HTTP GET operation
+ using this URL returns the resource itself. Example:
+ ``https://cloudresourcemanager.googleapis.com/v1/projects/my-
+ project-123``. It will be left unspecified for resources
+ without a REST API.
+ parent:
+ The full name of the immediate parent of this resource. See
+ `Resource Names `__ for more information. For GCP
+ assets, it is the parent resource defined in the `Cloud IAM
+ policy hierarchy `__. For example: ``"//cloudresourcemanager.go
+ ogleapis.com/projects/my_project_123"``. For third-party
+ assets, it is up to the users to define.
+ data:
+ The content of the resource, in which some sensitive fields
+ are scrubbed away and may not be present.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1.Resource)
+ ),
+)
+_sym_db.RegisterMessage(Resource)
+
+
+DESCRIPTOR._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/asset/google/cloud/asset_v1/proto/assets_pb2_grpc.py b/asset/google/cloud/asset_v1/proto/assets_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/asset/google/cloud/asset_v1/proto/assets_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/asset/google/cloud/asset_v1/types.py b/asset/google/cloud/asset_v1/types.py
new file mode 100644
index 000000000000..6dc0ded7c413
--- /dev/null
+++ b/asset/google/cloud/asset_v1/types.py
@@ -0,0 +1,59 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from __future__ import absolute_import
+import sys
+
+from google.api_core.protobuf_helpers import get_messages
+
+from google.cloud.asset_v1.proto import asset_service_pb2
+from google.cloud.asset_v1.proto import assets_pb2
+from google.iam.v1 import policy_pb2
+from google.longrunning import operations_pb2
+from google.protobuf import any_pb2
+from google.protobuf import struct_pb2
+from google.protobuf import timestamp_pb2
+from google.rpc import status_pb2
+from google.type import expr_pb2
+
+
+_shared_modules = [
+ policy_pb2,
+ operations_pb2,
+ any_pb2,
+ struct_pb2,
+ timestamp_pb2,
+ status_pb2,
+ expr_pb2,
+]
+
+_local_modules = [asset_service_pb2, assets_pb2]
+
+names = []
+
+for module in _shared_modules: # pragma: NO COVER
+ for name, message in get_messages(module).items():
+ setattr(sys.modules[__name__], name, message)
+ names.append(name)
+for module in _local_modules:
+ for name, message in get_messages(module).items():
+ message.__module__ = "google.cloud.asset_v1.types"
+ setattr(sys.modules[__name__], name, message)
+ names.append(name)
+
+
+__all__ = tuple(sorted(names))
diff --git a/asset/google/cloud/asset_v1beta1/__init__.py b/asset/google/cloud/asset_v1beta1/__init__.py
new file mode 100644
index 000000000000..bd6e93391183
--- /dev/null
+++ b/asset/google/cloud/asset_v1beta1/__init__.py
@@ -0,0 +1,30 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from __future__ import absolute_import
+
+from google.cloud.asset_v1beta1 import types
+from google.cloud.asset_v1beta1.gapic import asset_service_client
+from google.cloud.asset_v1beta1.gapic import enums
+
+
+class AssetServiceClient(asset_service_client.AssetServiceClient):
+ __doc__ = asset_service_client.AssetServiceClient.__doc__
+ enums = enums
+
+
+__all__ = ("enums", "types", "AssetServiceClient")
diff --git a/asset/google/cloud/asset_v1beta1/gapic/__init__.py b/asset/google/cloud/asset_v1beta1/gapic/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/asset/google/cloud/asset_v1beta1/gapic/asset_service_client.py b/asset/google/cloud/asset_v1beta1/gapic/asset_service_client.py
new file mode 100644
index 000000000000..32b796af82ec
--- /dev/null
+++ b/asset/google/cloud/asset_v1beta1/gapic/asset_service_client.py
@@ -0,0 +1,426 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Accesses the google.cloud.asset.v1beta1 AssetService API."""
+
+import pkg_resources
+import warnings
+
+from google.oauth2 import service_account
+import google.api_core.client_options
+import google.api_core.gapic_v1.client_info
+import google.api_core.gapic_v1.config
+import google.api_core.gapic_v1.method
+import google.api_core.gapic_v1.routing_header
+import google.api_core.grpc_helpers
+import google.api_core.operation
+import google.api_core.operations_v1
+import google.api_core.path_template
+import grpc
+
+from google.cloud.asset_v1beta1.gapic import asset_service_client_config
+from google.cloud.asset_v1beta1.gapic import enums
+from google.cloud.asset_v1beta1.gapic.transports import asset_service_grpc_transport
+from google.cloud.asset_v1beta1.proto import asset_service_pb2
+from google.cloud.asset_v1beta1.proto import asset_service_pb2_grpc
+from google.cloud.asset_v1beta1.proto import assets_pb2
+from google.longrunning import operations_pb2
+from google.protobuf import timestamp_pb2
+
+
+_GAPIC_LIBRARY_VERSION = pkg_resources.get_distribution("google-cloud-asset").version
+
+
+class AssetServiceClient(object):
+ """Asset service definition."""
+
+ SERVICE_ADDRESS = "cloudasset.googleapis.com:443"
+ """The default address of the service."""
+
+ # The name of the interface for this client. This is the key used to
+ # find the method configuration in the client_config dictionary.
+ _INTERFACE_NAME = "google.cloud.asset.v1beta1.AssetService"
+
+ @classmethod
+ def from_service_account_file(cls, filename, *args, **kwargs):
+ """Creates an instance of this client using the provided credentials
+ file.
+
+ Args:
+ filename (str): The path to the service account private key json
+ file.
+ args: Additional arguments to pass to the constructor.
+ kwargs: Additional arguments to pass to the constructor.
+
+ Returns:
+ AssetServiceClient: The constructed client.
+ """
+ credentials = service_account.Credentials.from_service_account_file(filename)
+ kwargs["credentials"] = credentials
+ return cls(*args, **kwargs)
+
+ from_service_account_json = from_service_account_file
+
+ @classmethod
+ def project_path(cls, project):
+ """Return a fully-qualified project string."""
+ return google.api_core.path_template.expand(
+ "projects/{project}", project=project
+ )
+
+ def __init__(
+ self,
+ transport=None,
+ channel=None,
+ credentials=None,
+ client_config=None,
+ client_info=None,
+ client_options=None,
+ ):
+ """Constructor.
+
+ Args:
+ transport (Union[~.AssetServiceGrpcTransport,
+ Callable[[~.Credentials, type], ~.AssetServiceGrpcTransport]): A transport
+ instance, responsible for actually making the API calls.
+ The default transport uses the gRPC protocol.
+ This argument may also be a callable which returns a
+ transport instance. Callables will be sent the credentials
+ as the first argument and the default transport class as
+ the second argument.
+ channel (grpc.Channel): DEPRECATED. A ``Channel`` instance
+ through which to make calls. This argument is mutually exclusive
+ with ``credentials``; providing both will raise an exception.
+ credentials (google.auth.credentials.Credentials): The
+ authorization credentials to attach to requests. These
+ credentials identify this application to the service. If none
+ are specified, the client will attempt to ascertain the
+ credentials from the environment.
+ This argument is mutually exclusive with providing a
+ transport instance to ``transport``; doing so will raise
+ an exception.
+ client_config (dict): DEPRECATED. A dictionary of call options for
+ each method. If not specified, the default configuration is used.
+ client_info (google.api_core.gapic_v1.client_info.ClientInfo):
+ The client info used to send a user-agent string along with
+ API requests. If ``None``, then default info will be used.
+ Generally, you only need to set this if you're developing
+ your own client library.
+ client_options (Union[dict, google.api_core.client_options.ClientOptions]):
+ Client options used to set user options on the client. API Endpoint
+ should be set through client_options.
+ """
+ # Raise deprecation warnings for things we want to go away.
+ if client_config is not None:
+ warnings.warn(
+ "The `client_config` argument is deprecated.",
+ PendingDeprecationWarning,
+ stacklevel=2,
+ )
+ else:
+ client_config = asset_service_client_config.config
+
+ if channel:
+ warnings.warn(
+ "The `channel` argument is deprecated; use " "`transport` instead.",
+ PendingDeprecationWarning,
+ stacklevel=2,
+ )
+
+ api_endpoint = self.SERVICE_ADDRESS
+ if client_options:
+ if type(client_options) == dict:
+ client_options = google.api_core.client_options.from_dict(
+ client_options
+ )
+ if client_options.api_endpoint:
+ api_endpoint = client_options.api_endpoint
+
+ # Instantiate the transport.
+ # The transport is responsible for handling serialization and
+ # deserialization and actually sending data to the service.
+ if transport:
+ if callable(transport):
+ self.transport = transport(
+ credentials=credentials,
+ default_class=asset_service_grpc_transport.AssetServiceGrpcTransport,
+ address=api_endpoint,
+ )
+ else:
+ if credentials:
+ raise ValueError(
+ "Received both a transport instance and "
+ "credentials; these are mutually exclusive."
+ )
+ self.transport = transport
+ else:
+ self.transport = asset_service_grpc_transport.AssetServiceGrpcTransport(
+ address=api_endpoint, channel=channel, credentials=credentials
+ )
+
+ if client_info is None:
+ client_info = google.api_core.gapic_v1.client_info.ClientInfo(
+ gapic_version=_GAPIC_LIBRARY_VERSION
+ )
+ else:
+ client_info.gapic_version = _GAPIC_LIBRARY_VERSION
+ self._client_info = client_info
+
+ # Parse out the default settings for retry and timeout for each RPC
+ # from the client configuration.
+ # (Ordinarily, these are the defaults specified in the `*_config.py`
+ # file next to this one.)
+ self._method_configs = google.api_core.gapic_v1.config.parse_method_configs(
+ client_config["interfaces"][self._INTERFACE_NAME]
+ )
+
+ # Save a dictionary of cached API call functions.
+ # These are the actual callables which invoke the proper
+ # transport methods, wrapped with `wrap_method` to add retry,
+ # timeout, and the like.
+ self._inner_api_calls = {}
+
+ # Service calls
+ def export_assets(
+ self,
+ parent,
+ output_config,
+ read_time=None,
+ asset_types=None,
+ content_type=None,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Exports assets with time and resource types to a given Cloud Storage
+ location. The output format is newline-delimited JSON. This API
+ implements the ``google.longrunning.Operation`` API allowing you to keep
+ track of the export.
+
+ Example:
+ >>> from google.cloud import asset_v1beta1
+ >>>
+ >>> client = asset_v1beta1.AssetServiceClient()
+ >>>
+ >>> parent = client.project_path('[PROJECT]')
+ >>>
+ >>> # TODO: Initialize `output_config`:
+ >>> output_config = {}
+ >>>
+ >>> response = client.export_assets(parent, output_config)
+ >>>
+ >>> def callback(operation_future):
+ ... # Handle result.
+ ... result = operation_future.result()
+ >>>
+ >>> response.add_done_callback(callback)
+ >>>
+ >>> # Handle metadata.
+ >>> metadata = response.metadata()
+
+ Args:
+ parent (str): Required. The relative name of the root asset. This can only be an
+ organization number (such as "organizations/123"), a project ID (such as
+ "projects/my-project-id"), a project number (such as "projects/12345"), or
+ a folder number (such as "folders/123").
+ output_config (Union[dict, ~google.cloud.asset_v1beta1.types.OutputConfig]): Required. Output configuration indicating where the results will be output
+ to. All results will be in newline delimited JSON format.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.asset_v1beta1.types.OutputConfig`
+ read_time (Union[dict, ~google.cloud.asset_v1beta1.types.Timestamp]): Timestamp to take an asset snapshot. This can only be set to a timestamp
+ between 2018-10-02 UTC (inclusive) and the current time. If not specified,
+ the current time will be used. Due to delays in resource data collection
+ and indexing, there is a volatile window during which running the same
+ query may get different results.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.asset_v1beta1.types.Timestamp`
+ asset_types (list[str]): A list of asset types of which to take a snapshot for. For example:
+ "google.compute.Disk". If specified, only matching assets will be
+ returned. See `Introduction to Cloud Asset
+ Inventory `__
+ for all supported asset types.
+ content_type (~google.cloud.asset_v1beta1.types.ContentType): Asset content type. If not specified, no content but the asset name will be
+ returned.
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.asset_v1beta1.types._OperationFuture` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "export_assets" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "export_assets"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.export_assets,
+ default_retry=self._method_configs["ExportAssets"].retry,
+ default_timeout=self._method_configs["ExportAssets"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = asset_service_pb2.ExportAssetsRequest(
+ parent=parent,
+ output_config=output_config,
+ read_time=read_time,
+ asset_types=asset_types,
+ content_type=content_type,
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("parent", parent)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ operation = self._inner_api_calls["export_assets"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+ return google.api_core.operation.from_gapic(
+ operation,
+ self.transport._operations_client,
+ asset_service_pb2.ExportAssetsResponse,
+ metadata_type=asset_service_pb2.ExportAssetsRequest,
+ )
+
+ def batch_get_assets_history(
+ self,
+ parent,
+ content_type,
+ read_time_window,
+ asset_names=None,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Batch gets the update history of assets that overlap a time window. For
+ RESOURCE content, this API outputs history with asset in both non-delete
+ or deleted status. For IAM\_POLICY content, this API outputs history
+ when the asset and its attached IAM POLICY both exist. This can create
+ gaps in the output history.
+
+ Example:
+ >>> from google.cloud import asset_v1beta1
+ >>> from google.cloud.asset_v1beta1 import enums
+ >>>
+ >>> client = asset_v1beta1.AssetServiceClient()
+ >>>
+ >>> parent = client.project_path('[PROJECT]')
+ >>>
+ >>> # TODO: Initialize `content_type`:
+ >>> content_type = enums.ContentType.CONTENT_TYPE_UNSPECIFIED
+ >>>
+ >>> # TODO: Initialize `read_time_window`:
+ >>> read_time_window = {}
+ >>>
+ >>> response = client.batch_get_assets_history(parent, content_type, read_time_window)
+
+ Args:
+ parent (str): Required. The relative name of the root asset. It can only be an
+ organization number (such as "organizations/123"), a project ID (such as
+ "projects/my-project-id")", or a project number (such as "projects/12345").
+ content_type (~google.cloud.asset_v1beta1.types.ContentType): Required. The content type.
+ read_time_window (Union[dict, ~google.cloud.asset_v1beta1.types.TimeWindow]): Optional. The time window for the asset history. Both start\_time and
+ end\_time are optional and if set, it must be after 2018-10-02 UTC. If
+ end\_time is not set, it is default to current timestamp. If start\_time
+ is not set, the snapshot of the assets at end\_time will be returned.
+ The returned results contain all temporal assets whose time window
+ overlap with read\_time\_window.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.asset_v1beta1.types.TimeWindow`
+ asset_names (list[str]): A list of the full names of the assets. For example:
+ ``//compute.googleapis.com/projects/my_project_123/zones/zone1/instances/instance1``.
+ See `Resource
+ Names `__
+ for more info.
+
+ The request becomes a no-op if the asset name list is empty, and the max
+ size of the asset name list is 100 in one request.
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.asset_v1beta1.types.BatchGetAssetsHistoryResponse` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "batch_get_assets_history" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "batch_get_assets_history"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.batch_get_assets_history,
+ default_retry=self._method_configs["BatchGetAssetsHistory"].retry,
+ default_timeout=self._method_configs["BatchGetAssetsHistory"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = asset_service_pb2.BatchGetAssetsHistoryRequest(
+ parent=parent,
+ content_type=content_type,
+ read_time_window=read_time_window,
+ asset_names=asset_names,
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("parent", parent)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ return self._inner_api_calls["batch_get_assets_history"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
diff --git a/asset/google/cloud/asset_v1beta1/gapic/asset_service_client_config.py b/asset/google/cloud/asset_v1beta1/gapic/asset_service_client_config.py
new file mode 100644
index 000000000000..340e89de38b2
--- /dev/null
+++ b/asset/google/cloud/asset_v1beta1/gapic/asset_service_client_config.py
@@ -0,0 +1,33 @@
+config = {
+ "interfaces": {
+ "google.cloud.asset.v1beta1.AssetService": {
+ "retry_codes": {
+ "idempotent": ["DEADLINE_EXCEEDED", "UNAVAILABLE"],
+ "non_idempotent": [],
+ },
+ "retry_params": {
+ "default": {
+ "initial_retry_delay_millis": 100,
+ "retry_delay_multiplier": 1.3,
+ "max_retry_delay_millis": 60000,
+ "initial_rpc_timeout_millis": 20000,
+ "rpc_timeout_multiplier": 1.0,
+ "max_rpc_timeout_millis": 20000,
+ "total_timeout_millis": 600000,
+ }
+ },
+ "methods": {
+ "ExportAssets": {
+ "timeout_millis": 600000,
+ "retry_codes_name": "non_idempotent",
+ "retry_params_name": "default",
+ },
+ "BatchGetAssetsHistory": {
+ "timeout_millis": 600000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ },
+ }
+ }
+}
diff --git a/asset/google/cloud/asset_v1beta1/gapic/enums.py b/asset/google/cloud/asset_v1beta1/gapic/enums.py
new file mode 100644
index 000000000000..780beae4ddef
--- /dev/null
+++ b/asset/google/cloud/asset_v1beta1/gapic/enums.py
@@ -0,0 +1,48 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Wrappers for protocol buffer enum types."""
+
+import enum
+
+
+class ContentType(enum.IntEnum):
+ """
+ Asset content type.
+
+ Attributes:
+ CONTENT_TYPE_UNSPECIFIED (int): Unspecified content type.
+ RESOURCE (int): Resource metadata.
+ IAM_POLICY (int): The actual IAM policy set on a resource.
+ """
+
+ CONTENT_TYPE_UNSPECIFIED = 0
+ RESOURCE = 1
+ IAM_POLICY = 2
+
+
+class NullValue(enum.IntEnum):
+ """
+ ``NullValue`` is a singleton enumeration to represent the null value for
+ the ``Value`` type union.
+
+ The JSON representation for ``NullValue`` is JSON ``null``.
+
+ Attributes:
+ NULL_VALUE (int): Null value.
+ """
+
+ NULL_VALUE = 0
diff --git a/asset/google/cloud/asset_v1beta1/gapic/transports/__init__.py b/asset/google/cloud/asset_v1beta1/gapic/transports/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/asset/google/cloud/asset_v1beta1/gapic/transports/asset_service_grpc_transport.py b/asset/google/cloud/asset_v1beta1/gapic/transports/asset_service_grpc_transport.py
new file mode 100644
index 000000000000..9e92ba565ce3
--- /dev/null
+++ b/asset/google/cloud/asset_v1beta1/gapic/transports/asset_service_grpc_transport.py
@@ -0,0 +1,149 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import google.api_core.grpc_helpers
+import google.api_core.operations_v1
+
+from google.cloud.asset_v1beta1.proto import asset_service_pb2_grpc
+
+
+class AssetServiceGrpcTransport(object):
+ """gRPC transport class providing stubs for
+ google.cloud.asset.v1beta1 AssetService API.
+
+ The transport provides access to the raw gRPC stubs,
+ which can be used to take advantage of advanced
+ features of gRPC.
+ """
+
+ # The scopes needed to make gRPC calls to all of the methods defined
+ # in this service.
+ _OAUTH_SCOPES = ("https://www.googleapis.com/auth/cloud-platform",)
+
+ def __init__(
+ self, channel=None, credentials=None, address="cloudasset.googleapis.com:443"
+ ):
+ """Instantiate the transport class.
+
+ Args:
+ channel (grpc.Channel): A ``Channel`` instance through
+ which to make calls. This argument is mutually exclusive
+ with ``credentials``; providing both will raise an exception.
+ credentials (google.auth.credentials.Credentials): The
+ authorization credentials to attach to requests. These
+ credentials identify this application to the service. If none
+ are specified, the client will attempt to ascertain the
+ credentials from the environment.
+ address (str): The address where the service is hosted.
+ """
+ # If both `channel` and `credentials` are specified, raise an
+ # exception (channels come with credentials baked in already).
+ if channel is not None and credentials is not None:
+ raise ValueError(
+ "The `channel` and `credentials` arguments are mutually " "exclusive."
+ )
+
+ # Create the channel.
+ if channel is None:
+ channel = self.create_channel(
+ address=address,
+ credentials=credentials,
+ options={
+ "grpc.max_send_message_length": -1,
+ "grpc.max_receive_message_length": -1,
+ }.items(),
+ )
+
+ self._channel = channel
+
+ # gRPC uses objects called "stubs" that are bound to the
+ # channel and provide a basic method for each RPC.
+ self._stubs = {
+ "asset_service_stub": asset_service_pb2_grpc.AssetServiceStub(channel)
+ }
+
+ # Because this API includes a method that returns a
+ # long-running operation (proto: google.longrunning.Operation),
+ # instantiate an LRO client.
+ self._operations_client = google.api_core.operations_v1.OperationsClient(
+ channel
+ )
+
+ @classmethod
+ def create_channel(
+ cls, address="cloudasset.googleapis.com:443", credentials=None, **kwargs
+ ):
+ """Create and return a gRPC channel object.
+
+ Args:
+ address (str): The host for the channel to use.
+ credentials (~.Credentials): The
+ authorization credentials to attach to requests. These
+ credentials identify this application to the service. If
+ none are specified, the client will attempt to ascertain
+ the credentials from the environment.
+ kwargs (dict): Keyword arguments, which are passed to the
+ channel creation.
+
+ Returns:
+ grpc.Channel: A gRPC channel object.
+ """
+ return google.api_core.grpc_helpers.create_channel(
+ address, credentials=credentials, scopes=cls._OAUTH_SCOPES, **kwargs
+ )
+
+ @property
+ def channel(self):
+ """The gRPC channel used by the transport.
+
+ Returns:
+ grpc.Channel: A gRPC channel object.
+ """
+ return self._channel
+
+ @property
+ def export_assets(self):
+ """Return the gRPC stub for :meth:`AssetServiceClient.export_assets`.
+
+ Exports assets with time and resource types to a given Cloud Storage
+ location. The output format is newline-delimited JSON. This API
+ implements the ``google.longrunning.Operation`` API allowing you to keep
+ track of the export.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["asset_service_stub"].ExportAssets
+
+ @property
+ def batch_get_assets_history(self):
+ """Return the gRPC stub for :meth:`AssetServiceClient.batch_get_assets_history`.
+
+ Batch gets the update history of assets that overlap a time window. For
+ RESOURCE content, this API outputs history with asset in both non-delete
+ or deleted status. For IAM\_POLICY content, this API outputs history
+ when the asset and its attached IAM POLICY both exist. This can create
+ gaps in the output history.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["asset_service_stub"].BatchGetAssetsHistory
diff --git a/asset/google/cloud/asset_v1beta1/proto/__init__.py b/asset/google/cloud/asset_v1beta1/proto/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/asset/google/cloud/asset_v1beta1/proto/asset_service.proto b/asset/google/cloud/asset_v1beta1/proto/asset_service.proto
new file mode 100644
index 000000000000..8e2d1b0a387e
--- /dev/null
+++ b/asset/google/cloud/asset_v1beta1/proto/asset_service.proto
@@ -0,0 +1,182 @@
+// Copyright 2018 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.asset.v1beta1;
+
+import "google/api/annotations.proto";
+import "google/cloud/asset/v1beta1/assets.proto";
+import "google/longrunning/operations.proto";
+import "google/protobuf/timestamp.proto";
+
+option csharp_namespace = "Google.Cloud.Asset.V1Beta1";
+option go_package = "google.golang.org/genproto/googleapis/cloud/asset/v1beta1;asset";
+option java_multiple_files = true;
+option java_outer_classname = "AssetServiceProto";
+option java_package = "com.google.cloud.asset.v1beta1";
+option php_namespace = "Google\\Cloud\\Asset\\V1beta1";
+
+// Asset service definition.
+service AssetService {
+ // Exports assets with time and resource types to a given Cloud Storage
+ // location. The output format is newline-delimited JSON.
+ // This API implements the
+ // [google.longrunning.Operation][google.longrunning.Operation] API allowing
+ // you to keep track of the export.
+ rpc ExportAssets(ExportAssetsRequest) returns (google.longrunning.Operation) {
+ option (google.api.http) = {
+ post: "/v1beta1/{parent=projects/*}:exportAssets"
+ body: "*"
+ additional_bindings {
+ post: "/v1beta1/{parent=folders/*}:exportAssets"
+ body: "*"
+ }
+ additional_bindings {
+ post: "/v1beta1/{parent=organizations/*}:exportAssets"
+ body: "*"
+ }
+ };
+ }
+
+ // Batch gets the update history of assets that overlap a time window.
+ // For RESOURCE content, this API outputs history with asset in both
+ // non-delete or deleted status.
+ // For IAM_POLICY content, this API outputs history when the asset and its
+ // attached IAM POLICY both exist. This can create gaps in the output history.
+ rpc BatchGetAssetsHistory(BatchGetAssetsHistoryRequest)
+ returns (BatchGetAssetsHistoryResponse) {
+ option (google.api.http) = {
+ get: "/v1beta1/{parent=projects/*}:batchGetAssetsHistory"
+ additional_bindings {
+ get: "/v1beta1/{parent=organizations/*}:batchGetAssetsHistory"
+ }
+ };
+ }
+}
+
+// Export asset request.
+message ExportAssetsRequest {
+ // Required. The relative name of the root asset. This can only be an
+ // organization number (such as "organizations/123"), a project ID (such as
+ // "projects/my-project-id"), a project number (such as "projects/12345"), or
+ // a folder number (such as "folders/123").
+ string parent = 1;
+
+ // Timestamp to take an asset snapshot. This can only be set to a timestamp
+ // between 2018-10-02 UTC (inclusive) and the current time. If not specified,
+ // the current time will be used. Due to delays in resource data collection
+ // and indexing, there is a volatile window during which running the same
+ // query may get different results.
+ google.protobuf.Timestamp read_time = 2;
+
+ // A list of asset types of which to take a snapshot for. For example:
+ // "google.compute.Disk". If specified, only matching assets will be returned.
+ // See [Introduction to Cloud Asset
+ // Inventory](https://cloud.google.com/resource-manager/docs/cloud-asset-inventory/overview)
+ // for all supported asset types.
+ repeated string asset_types = 3;
+
+ // Asset content type. If not specified, no content but the asset name will be
+ // returned.
+ ContentType content_type = 4;
+
+ // Required. Output configuration indicating where the results will be output
+ // to. All results will be in newline delimited JSON format.
+ OutputConfig output_config = 5;
+}
+
+// The export asset response. This message is returned by the
+// [google.longrunning.Operations.GetOperation][google.longrunning.Operations.GetOperation]
+// method in the returned
+// [google.longrunning.Operation.response][google.longrunning.Operation.response]
+// field.
+message ExportAssetsResponse {
+ // Time the snapshot was taken.
+ google.protobuf.Timestamp read_time = 1;
+
+ // Output configuration indicating where the results were output to.
+ // All results are in JSON format.
+ OutputConfig output_config = 2;
+}
+
+// Batch get assets history request.
+message BatchGetAssetsHistoryRequest {
+ // Required. The relative name of the root asset. It can only be an
+ // organization number (such as "organizations/123"), a project ID (such as
+ // "projects/my-project-id")", or a project number (such as "projects/12345").
+ string parent = 1;
+
+ // A list of the full names of the assets. For example:
+ // `//compute.googleapis.com/projects/my_project_123/zones/zone1/instances/instance1`.
+ // See [Resource
+ // Names](https://cloud.google.com/apis/design/resource_names#full_resource_name)
+ // for more info.
+ //
+ // The request becomes a no-op if the asset name list is empty, and the max
+ // size of the asset name list is 100 in one request.
+ repeated string asset_names = 2;
+
+ // Required. The content type.
+ ContentType content_type = 3;
+
+ // Optional. The time window for the asset history. Both start_time and
+ // end_time are optional and if set, it must be after 2018-10-02 UTC. If
+ // end_time is not set, it is default to current timestamp. If start_time is
+ // not set, the snapshot of the assets at end_time will be returned. The
+ // returned results contain all temporal assets whose time window overlap with
+ // read_time_window.
+ TimeWindow read_time_window = 4;
+}
+
+// Batch get assets history response.
+message BatchGetAssetsHistoryResponse {
+ // A list of assets with valid time windows.
+ repeated TemporalAsset assets = 1;
+}
+
+// Output configuration for export assets destination.
+message OutputConfig {
+ // Asset export destination.
+ oneof destination {
+ // Destination on Cloud Storage.
+ GcsDestination gcs_destination = 1;
+ }
+}
+
+// A Cloud Storage location.
+message GcsDestination {
+ // Required.
+ oneof object_uri {
+ // The uri of the Cloud Storage object. It's the same uri that is used by
+ // gsutil. For example: "gs://bucket_name/object_name". See [Viewing and
+ // Editing Object
+ // Metadata](https://cloud.google.com/storage/docs/viewing-editing-metadata)
+ // for more information.
+ string uri = 1;
+ }
+}
+
+// Asset content type.
+enum ContentType {
+ // Unspecified content type.
+ CONTENT_TYPE_UNSPECIFIED = 0;
+
+ // Resource metadata.
+ RESOURCE = 1;
+
+ // The actual IAM policy set on a resource.
+ IAM_POLICY = 2;
+}
diff --git a/asset/google/cloud/asset_v1beta1/proto/asset_service_pb2.py b/asset/google/cloud/asset_v1beta1/proto/asset_service_pb2.py
new file mode 100644
index 000000000000..bd97c5aa8137
--- /dev/null
+++ b/asset/google/cloud/asset_v1beta1/proto/asset_service_pb2.py
@@ -0,0 +1,728 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/asset_v1beta1/proto/asset_service.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf.internal import enum_type_wrapper
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+from google.cloud.asset_v1beta1.proto import (
+ assets_pb2 as google_dot_cloud_dot_asset__v1beta1_dot_proto_dot_assets__pb2,
+)
+from google.longrunning import (
+ operations_pb2 as google_dot_longrunning_dot_operations__pb2,
+)
+from google.protobuf import timestamp_pb2 as google_dot_protobuf_dot_timestamp__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/asset_v1beta1/proto/asset_service.proto",
+ package="google.cloud.asset.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\036com.google.cloud.asset.v1beta1B\021AssetServiceProtoP\001Z?google.golang.org/genproto/googleapis/cloud/asset/v1beta1;asset\252\002\032Google.Cloud.Asset.V1Beta1\312\002\032Google\\Cloud\\Asset\\V1beta1"
+ ),
+ serialized_pb=_b(
+ '\n4google/cloud/asset_v1beta1/proto/asset_service.proto\x12\x1agoogle.cloud.asset.v1beta1\x1a\x1cgoogle/api/annotations.proto\x1a-google/cloud/asset_v1beta1/proto/assets.proto\x1a#google/longrunning/operations.proto\x1a\x1fgoogle/protobuf/timestamp.proto"\xe9\x01\n\x13\x45xportAssetsRequest\x12\x0e\n\x06parent\x18\x01 \x01(\t\x12-\n\tread_time\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12\x13\n\x0b\x61sset_types\x18\x03 \x03(\t\x12=\n\x0c\x63ontent_type\x18\x04 \x01(\x0e\x32\'.google.cloud.asset.v1beta1.ContentType\x12?\n\routput_config\x18\x05 \x01(\x0b\x32(.google.cloud.asset.v1beta1.OutputConfig"\x86\x01\n\x14\x45xportAssetsResponse\x12-\n\tread_time\x18\x01 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12?\n\routput_config\x18\x02 \x01(\x0b\x32(.google.cloud.asset.v1beta1.OutputConfig"\xc4\x01\n\x1c\x42\x61tchGetAssetsHistoryRequest\x12\x0e\n\x06parent\x18\x01 \x01(\t\x12\x13\n\x0b\x61sset_names\x18\x02 \x03(\t\x12=\n\x0c\x63ontent_type\x18\x03 \x01(\x0e\x32\'.google.cloud.asset.v1beta1.ContentType\x12@\n\x10read_time_window\x18\x04 \x01(\x0b\x32&.google.cloud.asset.v1beta1.TimeWindow"Z\n\x1d\x42\x61tchGetAssetsHistoryResponse\x12\x39\n\x06\x61ssets\x18\x01 \x03(\x0b\x32).google.cloud.asset.v1beta1.TemporalAsset"d\n\x0cOutputConfig\x12\x45\n\x0fgcs_destination\x18\x01 \x01(\x0b\x32*.google.cloud.asset.v1beta1.GcsDestinationH\x00\x42\r\n\x0b\x64\x65stination"-\n\x0eGcsDestination\x12\r\n\x03uri\x18\x01 \x01(\tH\x00\x42\x0c\n\nobject_uri*I\n\x0b\x43ontentType\x12\x1c\n\x18\x43ONTENT_TYPE_UNSPECIFIED\x10\x00\x12\x0c\n\x08RESOURCE\x10\x01\x12\x0e\n\nIAM_POLICY\x10\x02\x32\x91\x04\n\x0c\x41ssetService\x12\xfa\x01\n\x0c\x45xportAssets\x12/.google.cloud.asset.v1beta1.ExportAssetsRequest\x1a\x1d.google.longrunning.Operation"\x99\x01\x82\xd3\xe4\x93\x02\x92\x01")/v1beta1/{parent=projects/*}:exportAssets:\x01*Z-"(/v1beta1/{parent=folders/*}:exportAssets:\x01*Z3"./v1beta1/{parent=organizations/*}:exportAssets:\x01*\x12\x83\x02\n\x15\x42\x61tchGetAssetsHistory\x12\x38.google.cloud.asset.v1beta1.BatchGetAssetsHistoryRequest\x1a\x39.google.cloud.asset.v1beta1.BatchGetAssetsHistoryResponse"u\x82\xd3\xe4\x93\x02o\x12\x32/v1beta1/{parent=projects/*}:batchGetAssetsHistoryZ9\x12\x37/v1beta1/{parent=organizations/*}:batchGetAssetsHistoryB\xb0\x01\n\x1e\x63om.google.cloud.asset.v1beta1B\x11\x41ssetServiceProtoP\x01Z?google.golang.org/genproto/googleapis/cloud/asset/v1beta1;asset\xaa\x02\x1aGoogle.Cloud.Asset.V1Beta1\xca\x02\x1aGoogle\\Cloud\\Asset\\V1beta1b\x06proto3'
+ ),
+ dependencies=[
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_asset__v1beta1_dot_proto_dot_assets__pb2.DESCRIPTOR,
+ google_dot_longrunning_dot_operations__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_timestamp__pb2.DESCRIPTOR,
+ ],
+)
+
+_CONTENTTYPE = _descriptor.EnumDescriptor(
+ name="ContentType",
+ full_name="google.cloud.asset.v1beta1.ContentType",
+ filename=None,
+ file=DESCRIPTOR,
+ values=[
+ _descriptor.EnumValueDescriptor(
+ name="CONTENT_TYPE_UNSPECIFIED",
+ index=0,
+ number=0,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="RESOURCE", index=1, number=1, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="IAM_POLICY", index=2, number=2, serialized_options=None, type=None
+ ),
+ ],
+ containing_type=None,
+ serialized_options=None,
+ serialized_start=1044,
+ serialized_end=1117,
+)
+_sym_db.RegisterEnumDescriptor(_CONTENTTYPE)
+
+ContentType = enum_type_wrapper.EnumTypeWrapper(_CONTENTTYPE)
+CONTENT_TYPE_UNSPECIFIED = 0
+RESOURCE = 1
+IAM_POLICY = 2
+
+
+_EXPORTASSETSREQUEST = _descriptor.Descriptor(
+ name="ExportAssetsRequest",
+ full_name="google.cloud.asset.v1beta1.ExportAssetsRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="parent",
+ full_name="google.cloud.asset.v1beta1.ExportAssetsRequest.parent",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="read_time",
+ full_name="google.cloud.asset.v1beta1.ExportAssetsRequest.read_time",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="asset_types",
+ full_name="google.cloud.asset.v1beta1.ExportAssetsRequest.asset_types",
+ index=2,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="content_type",
+ full_name="google.cloud.asset.v1beta1.ExportAssetsRequest.content_type",
+ index=3,
+ number=4,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="output_config",
+ full_name="google.cloud.asset.v1beta1.ExportAssetsRequest.output_config",
+ index=4,
+ number=5,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=232,
+ serialized_end=465,
+)
+
+
+_EXPORTASSETSRESPONSE = _descriptor.Descriptor(
+ name="ExportAssetsResponse",
+ full_name="google.cloud.asset.v1beta1.ExportAssetsResponse",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="read_time",
+ full_name="google.cloud.asset.v1beta1.ExportAssetsResponse.read_time",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="output_config",
+ full_name="google.cloud.asset.v1beta1.ExportAssetsResponse.output_config",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=468,
+ serialized_end=602,
+)
+
+
+_BATCHGETASSETSHISTORYREQUEST = _descriptor.Descriptor(
+ name="BatchGetAssetsHistoryRequest",
+ full_name="google.cloud.asset.v1beta1.BatchGetAssetsHistoryRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="parent",
+ full_name="google.cloud.asset.v1beta1.BatchGetAssetsHistoryRequest.parent",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="asset_names",
+ full_name="google.cloud.asset.v1beta1.BatchGetAssetsHistoryRequest.asset_names",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="content_type",
+ full_name="google.cloud.asset.v1beta1.BatchGetAssetsHistoryRequest.content_type",
+ index=2,
+ number=3,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="read_time_window",
+ full_name="google.cloud.asset.v1beta1.BatchGetAssetsHistoryRequest.read_time_window",
+ index=3,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=605,
+ serialized_end=801,
+)
+
+
+_BATCHGETASSETSHISTORYRESPONSE = _descriptor.Descriptor(
+ name="BatchGetAssetsHistoryResponse",
+ full_name="google.cloud.asset.v1beta1.BatchGetAssetsHistoryResponse",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="assets",
+ full_name="google.cloud.asset.v1beta1.BatchGetAssetsHistoryResponse.assets",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=803,
+ serialized_end=893,
+)
+
+
+_OUTPUTCONFIG = _descriptor.Descriptor(
+ name="OutputConfig",
+ full_name="google.cloud.asset.v1beta1.OutputConfig",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="gcs_destination",
+ full_name="google.cloud.asset.v1beta1.OutputConfig.gcs_destination",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="destination",
+ full_name="google.cloud.asset.v1beta1.OutputConfig.destination",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=895,
+ serialized_end=995,
+)
+
+
+_GCSDESTINATION = _descriptor.Descriptor(
+ name="GcsDestination",
+ full_name="google.cloud.asset.v1beta1.GcsDestination",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="uri",
+ full_name="google.cloud.asset.v1beta1.GcsDestination.uri",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="object_uri",
+ full_name="google.cloud.asset.v1beta1.GcsDestination.object_uri",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=997,
+ serialized_end=1042,
+)
+
+_EXPORTASSETSREQUEST.fields_by_name[
+ "read_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_EXPORTASSETSREQUEST.fields_by_name["content_type"].enum_type = _CONTENTTYPE
+_EXPORTASSETSREQUEST.fields_by_name["output_config"].message_type = _OUTPUTCONFIG
+_EXPORTASSETSRESPONSE.fields_by_name[
+ "read_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_EXPORTASSETSRESPONSE.fields_by_name["output_config"].message_type = _OUTPUTCONFIG
+_BATCHGETASSETSHISTORYREQUEST.fields_by_name["content_type"].enum_type = _CONTENTTYPE
+_BATCHGETASSETSHISTORYREQUEST.fields_by_name[
+ "read_time_window"
+].message_type = (
+ google_dot_cloud_dot_asset__v1beta1_dot_proto_dot_assets__pb2._TIMEWINDOW
+)
+_BATCHGETASSETSHISTORYRESPONSE.fields_by_name[
+ "assets"
+].message_type = (
+ google_dot_cloud_dot_asset__v1beta1_dot_proto_dot_assets__pb2._TEMPORALASSET
+)
+_OUTPUTCONFIG.fields_by_name["gcs_destination"].message_type = _GCSDESTINATION
+_OUTPUTCONFIG.oneofs_by_name["destination"].fields.append(
+ _OUTPUTCONFIG.fields_by_name["gcs_destination"]
+)
+_OUTPUTCONFIG.fields_by_name[
+ "gcs_destination"
+].containing_oneof = _OUTPUTCONFIG.oneofs_by_name["destination"]
+_GCSDESTINATION.oneofs_by_name["object_uri"].fields.append(
+ _GCSDESTINATION.fields_by_name["uri"]
+)
+_GCSDESTINATION.fields_by_name["uri"].containing_oneof = _GCSDESTINATION.oneofs_by_name[
+ "object_uri"
+]
+DESCRIPTOR.message_types_by_name["ExportAssetsRequest"] = _EXPORTASSETSREQUEST
+DESCRIPTOR.message_types_by_name["ExportAssetsResponse"] = _EXPORTASSETSRESPONSE
+DESCRIPTOR.message_types_by_name[
+ "BatchGetAssetsHistoryRequest"
+] = _BATCHGETASSETSHISTORYREQUEST
+DESCRIPTOR.message_types_by_name[
+ "BatchGetAssetsHistoryResponse"
+] = _BATCHGETASSETSHISTORYRESPONSE
+DESCRIPTOR.message_types_by_name["OutputConfig"] = _OUTPUTCONFIG
+DESCRIPTOR.message_types_by_name["GcsDestination"] = _GCSDESTINATION
+DESCRIPTOR.enum_types_by_name["ContentType"] = _CONTENTTYPE
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+ExportAssetsRequest = _reflection.GeneratedProtocolMessageType(
+ "ExportAssetsRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_EXPORTASSETSREQUEST,
+ __module__="google.cloud.asset_v1beta1.proto.asset_service_pb2",
+ __doc__="""Export asset request.
+
+
+ Attributes:
+ parent:
+ Required. The relative name of the root asset. This can only
+ be an organization number (such as "organizations/123"), a
+ project ID (such as "projects/my-project-id"), a project
+ number (such as "projects/12345"), or a folder number (such as
+ "folders/123").
+ read_time:
+ Timestamp to take an asset snapshot. This can only be set to a
+ timestamp between 2018-10-02 UTC (inclusive) and the current
+ time. If not specified, the current time will be used. Due to
+ delays in resource data collection and indexing, there is a
+ volatile window during which running the same query may get
+ different results.
+ asset_types:
+ A list of asset types of which to take a snapshot for. For
+ example: "google.compute.Disk". If specified, only matching
+ assets will be returned. See `Introduction to Cloud Asset
+ Inventory `__ for all
+ supported asset types.
+ content_type:
+ Asset content type. If not specified, no content but the asset
+ name will be returned.
+ output_config:
+ Required. Output configuration indicating where the results
+ will be output to. All results will be in newline delimited
+ JSON format.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1beta1.ExportAssetsRequest)
+ ),
+)
+_sym_db.RegisterMessage(ExportAssetsRequest)
+
+ExportAssetsResponse = _reflection.GeneratedProtocolMessageType(
+ "ExportAssetsResponse",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_EXPORTASSETSRESPONSE,
+ __module__="google.cloud.asset_v1beta1.proto.asset_service_pb2",
+ __doc__="""The export asset response. This message is returned by the
+ [google.longrunning.Operations.GetOperation][google.longrunning.Operations.GetOperation]
+ method in the returned
+ [google.longrunning.Operation.response][google.longrunning.Operation.response]
+ field.
+
+
+ Attributes:
+ read_time:
+ Time the snapshot was taken.
+ output_config:
+ Output configuration indicating where the results were output
+ to. All results are in JSON format.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1beta1.ExportAssetsResponse)
+ ),
+)
+_sym_db.RegisterMessage(ExportAssetsResponse)
+
+BatchGetAssetsHistoryRequest = _reflection.GeneratedProtocolMessageType(
+ "BatchGetAssetsHistoryRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_BATCHGETASSETSHISTORYREQUEST,
+ __module__="google.cloud.asset_v1beta1.proto.asset_service_pb2",
+ __doc__="""Batch get assets history request.
+
+
+ Attributes:
+ parent:
+ Required. The relative name of the root asset. It can only be
+ an organization number (such as "organizations/123"), a
+ project ID (such as "projects/my-project-id")", or a project
+ number (such as "projects/12345").
+ asset_names:
+ A list of the full names of the assets. For example: ``//compu
+ te.googleapis.com/projects/my_project_123/zones/zone1/instance
+ s/instance1``. See `Resource Names `__ for more
+ info. The request becomes a no-op if the asset name list is
+ empty, and the max size of the asset name list is 100 in one
+ request.
+ content_type:
+ Required. The content type.
+ read_time_window:
+ Optional. The time window for the asset history. Both
+ start\_time and end\_time are optional and if set, it must be
+ after 2018-10-02 UTC. If end\_time is not set, it is default
+ to current timestamp. If start\_time is not set, the snapshot
+ of the assets at end\_time will be returned. The returned
+ results contain all temporal assets whose time window overlap
+ with read\_time\_window.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1beta1.BatchGetAssetsHistoryRequest)
+ ),
+)
+_sym_db.RegisterMessage(BatchGetAssetsHistoryRequest)
+
+BatchGetAssetsHistoryResponse = _reflection.GeneratedProtocolMessageType(
+ "BatchGetAssetsHistoryResponse",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_BATCHGETASSETSHISTORYRESPONSE,
+ __module__="google.cloud.asset_v1beta1.proto.asset_service_pb2",
+ __doc__="""Batch get assets history response.
+
+
+ Attributes:
+ assets:
+ A list of assets with valid time windows.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1beta1.BatchGetAssetsHistoryResponse)
+ ),
+)
+_sym_db.RegisterMessage(BatchGetAssetsHistoryResponse)
+
+OutputConfig = _reflection.GeneratedProtocolMessageType(
+ "OutputConfig",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_OUTPUTCONFIG,
+ __module__="google.cloud.asset_v1beta1.proto.asset_service_pb2",
+ __doc__="""Output configuration for export assets destination.
+
+
+ Attributes:
+ destination:
+ Asset export destination.
+ gcs_destination:
+ Destination on Cloud Storage.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1beta1.OutputConfig)
+ ),
+)
+_sym_db.RegisterMessage(OutputConfig)
+
+GcsDestination = _reflection.GeneratedProtocolMessageType(
+ "GcsDestination",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_GCSDESTINATION,
+ __module__="google.cloud.asset_v1beta1.proto.asset_service_pb2",
+ __doc__="""A Cloud Storage location.
+
+
+ Attributes:
+ object_uri:
+ Required.
+ uri:
+ The uri of the Cloud Storage object. It's the same uri that is
+ used by gsutil. For example: "gs://bucket\_name/object\_name".
+ See `Viewing and Editing Object Metadata
+ `__ for more information.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1beta1.GcsDestination)
+ ),
+)
+_sym_db.RegisterMessage(GcsDestination)
+
+
+DESCRIPTOR._options = None
+
+_ASSETSERVICE = _descriptor.ServiceDescriptor(
+ name="AssetService",
+ full_name="google.cloud.asset.v1beta1.AssetService",
+ file=DESCRIPTOR,
+ index=0,
+ serialized_options=None,
+ serialized_start=1120,
+ serialized_end=1649,
+ methods=[
+ _descriptor.MethodDescriptor(
+ name="ExportAssets",
+ full_name="google.cloud.asset.v1beta1.AssetService.ExportAssets",
+ index=0,
+ containing_service=None,
+ input_type=_EXPORTASSETSREQUEST,
+ output_type=google_dot_longrunning_dot_operations__pb2._OPERATION,
+ serialized_options=_b(
+ '\202\323\344\223\002\222\001")/v1beta1/{parent=projects/*}:exportAssets:\001*Z-"(/v1beta1/{parent=folders/*}:exportAssets:\001*Z3"./v1beta1/{parent=organizations/*}:exportAssets:\001*'
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="BatchGetAssetsHistory",
+ full_name="google.cloud.asset.v1beta1.AssetService.BatchGetAssetsHistory",
+ index=1,
+ containing_service=None,
+ input_type=_BATCHGETASSETSHISTORYREQUEST,
+ output_type=_BATCHGETASSETSHISTORYRESPONSE,
+ serialized_options=_b(
+ "\202\323\344\223\002o\0222/v1beta1/{parent=projects/*}:batchGetAssetsHistoryZ9\0227/v1beta1/{parent=organizations/*}:batchGetAssetsHistory"
+ ),
+ ),
+ ],
+)
+_sym_db.RegisterServiceDescriptor(_ASSETSERVICE)
+
+DESCRIPTOR.services_by_name["AssetService"] = _ASSETSERVICE
+
+# @@protoc_insertion_point(module_scope)
diff --git a/asset/google/cloud/asset_v1beta1/proto/asset_service_pb2_grpc.py b/asset/google/cloud/asset_v1beta1/proto/asset_service_pb2_grpc.py
new file mode 100644
index 000000000000..35e7c536eae5
--- /dev/null
+++ b/asset/google/cloud/asset_v1beta1/proto/asset_service_pb2_grpc.py
@@ -0,0 +1,77 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
+
+from google.cloud.asset_v1beta1.proto import (
+ asset_service_pb2 as google_dot_cloud_dot_asset__v1beta1_dot_proto_dot_asset__service__pb2,
+)
+from google.longrunning import (
+ operations_pb2 as google_dot_longrunning_dot_operations__pb2,
+)
+
+
+class AssetServiceStub(object):
+ """Asset service definition.
+ """
+
+ def __init__(self, channel):
+ """Constructor.
+
+ Args:
+ channel: A grpc.Channel.
+ """
+ self.ExportAssets = channel.unary_unary(
+ "/google.cloud.asset.v1beta1.AssetService/ExportAssets",
+ request_serializer=google_dot_cloud_dot_asset__v1beta1_dot_proto_dot_asset__service__pb2.ExportAssetsRequest.SerializeToString,
+ response_deserializer=google_dot_longrunning_dot_operations__pb2.Operation.FromString,
+ )
+ self.BatchGetAssetsHistory = channel.unary_unary(
+ "/google.cloud.asset.v1beta1.AssetService/BatchGetAssetsHistory",
+ request_serializer=google_dot_cloud_dot_asset__v1beta1_dot_proto_dot_asset__service__pb2.BatchGetAssetsHistoryRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_asset__v1beta1_dot_proto_dot_asset__service__pb2.BatchGetAssetsHistoryResponse.FromString,
+ )
+
+
+class AssetServiceServicer(object):
+ """Asset service definition.
+ """
+
+ def ExportAssets(self, request, context):
+ """Exports assets with time and resource types to a given Cloud Storage
+ location. The output format is newline-delimited JSON.
+ This API implements the
+ [google.longrunning.Operation][google.longrunning.Operation] API allowing
+ you to keep track of the export.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def BatchGetAssetsHistory(self, request, context):
+ """Batch gets the update history of assets that overlap a time window.
+ For RESOURCE content, this API outputs history with asset in both
+ non-delete or deleted status.
+ For IAM_POLICY content, this API outputs history when the asset and its
+ attached IAM POLICY both exist. This can create gaps in the output history.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+
+def add_AssetServiceServicer_to_server(servicer, server):
+ rpc_method_handlers = {
+ "ExportAssets": grpc.unary_unary_rpc_method_handler(
+ servicer.ExportAssets,
+ request_deserializer=google_dot_cloud_dot_asset__v1beta1_dot_proto_dot_asset__service__pb2.ExportAssetsRequest.FromString,
+ response_serializer=google_dot_longrunning_dot_operations__pb2.Operation.SerializeToString,
+ ),
+ "BatchGetAssetsHistory": grpc.unary_unary_rpc_method_handler(
+ servicer.BatchGetAssetsHistory,
+ request_deserializer=google_dot_cloud_dot_asset__v1beta1_dot_proto_dot_asset__service__pb2.BatchGetAssetsHistoryRequest.FromString,
+ response_serializer=google_dot_cloud_dot_asset__v1beta1_dot_proto_dot_asset__service__pb2.BatchGetAssetsHistoryResponse.SerializeToString,
+ ),
+ }
+ generic_handler = grpc.method_handlers_generic_handler(
+ "google.cloud.asset.v1beta1.AssetService", rpc_method_handlers
+ )
+ server.add_generic_rpc_handlers((generic_handler,))
diff --git a/asset/google/cloud/asset_v1beta1/proto/assets.proto b/asset/google/cloud/asset_v1beta1/proto/assets.proto
new file mode 100644
index 000000000000..84db977b97d7
--- /dev/null
+++ b/asset/google/cloud/asset_v1beta1/proto/assets.proto
@@ -0,0 +1,117 @@
+// Copyright 2018 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.asset.v1beta1;
+
+import "google/api/annotations.proto";
+import "google/iam/v1/policy.proto";
+import "google/protobuf/any.proto";
+import "google/protobuf/struct.proto";
+import "google/protobuf/timestamp.proto";
+
+option csharp_namespace = "Google.Cloud.Asset.V1Beta1";
+option go_package = "google.golang.org/genproto/googleapis/cloud/asset/v1beta1;asset";
+option java_multiple_files = true;
+option java_outer_classname = "AssetProto";
+option java_package = "com.google.cloud.asset.v1beta1";
+option php_namespace = "Google\\Cloud\\Asset\\V1beta1";
+
+// Temporal asset. In addition to the asset, the temporal asset includes the
+// status of the asset and valid from and to time of it.
+message TemporalAsset {
+ // The time window when the asset data and state was observed.
+ TimeWindow window = 1;
+
+ // If the asset is deleted or not.
+ bool deleted = 2;
+
+ // Asset.
+ Asset asset = 3;
+}
+
+// A time window of (start_time, end_time].
+message TimeWindow {
+ // Start time of the time window (exclusive).
+ google.protobuf.Timestamp start_time = 1;
+
+ // End time of the time window (inclusive).
+ // Current timestamp if not specified.
+ google.protobuf.Timestamp end_time = 2;
+}
+
+// Cloud asset. This includes all Google Cloud Platform resources,
+// Cloud IAM policies, and other non-GCP assets.
+message Asset {
+ // The full name of the asset. For example:
+ // `//compute.googleapis.com/projects/my_project_123/zones/zone1/instances/instance1`.
+ // See [Resource
+ // Names](https://cloud.google.com/apis/design/resource_names#full_resource_name)
+ // for more information.
+ string name = 1;
+
+ // Type of the asset. Example: "google.compute.Disk".
+ string asset_type = 2;
+
+ // Representation of the resource.
+ Resource resource = 3;
+
+ // Representation of the actual Cloud IAM policy set on a cloud resource. For
+ // each resource, there must be at most one Cloud IAM policy set on it.
+ google.iam.v1.Policy iam_policy = 4;
+}
+
+// Representation of a cloud resource.
+message Resource {
+ // The API version. Example: "v1".
+ string version = 1;
+
+ // The URL of the discovery document containing the resource's JSON schema.
+ // For example:
+ // `"https://www.googleapis.com/discovery/v1/apis/compute/v1/rest"`.
+ // It will be left unspecified for resources without a discovery-based API,
+ // such as Cloud Bigtable.
+ string discovery_document_uri = 2;
+
+ // The JSON schema name listed in the discovery document.
+ // Example: "Project". It will be left unspecified for resources (such as
+ // Cloud Bigtable) without a discovery-based API.
+ string discovery_name = 3;
+
+ // The REST URL for accessing the resource. An HTTP GET operation using this
+ // URL returns the resource itself.
+ // Example:
+ // `https://cloudresourcemanager.googleapis.com/v1/projects/my-project-123`.
+ // It will be left unspecified for resources without a REST API.
+ string resource_url = 4;
+
+ // The full name of the immediate parent of this resource. See
+ // [Resource
+ // Names](https://cloud.google.com/apis/design/resource_names#full_resource_name)
+ // for more information.
+ //
+ // For GCP assets, it is the parent resource defined in the [Cloud IAM policy
+ // hierarchy](https://cloud.google.com/iam/docs/overview#policy_hierarchy).
+ // For example:
+ // `"//cloudresourcemanager.googleapis.com/projects/my_project_123"`.
+ //
+ // For third-party assets, it is up to the users to define.
+ string parent = 5;
+
+ // The content of the resource, in which some sensitive fields are scrubbed
+ // away and may not be present.
+ google.protobuf.Struct data = 6;
+}
diff --git a/asset/google/cloud/asset_v1beta1/proto/assets_pb2.py b/asset/google/cloud/asset_v1beta1/proto/assets_pb2.py
new file mode 100644
index 000000000000..6a04ad0a8fdd
--- /dev/null
+++ b/asset/google/cloud/asset_v1beta1/proto/assets_pb2.py
@@ -0,0 +1,544 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/asset_v1beta1/proto/assets.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+from google.iam.v1 import iam_policy_pb2 as google_dot_iam_dot_v1_dot_policy__pb2
+from google.protobuf import any_pb2 as google_dot_protobuf_dot_any__pb2
+from google.protobuf import struct_pb2 as google_dot_protobuf_dot_struct__pb2
+from google.protobuf import timestamp_pb2 as google_dot_protobuf_dot_timestamp__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/asset_v1beta1/proto/assets.proto",
+ package="google.cloud.asset.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\036com.google.cloud.asset.v1beta1B\nAssetProtoP\001Z?google.golang.org/genproto/googleapis/cloud/asset/v1beta1;asset\252\002\032Google.Cloud.Asset.V1Beta1\312\002\032Google\\Cloud\\Asset\\V1beta1"
+ ),
+ serialized_pb=_b(
+ '\n-google/cloud/asset_v1beta1/proto/assets.proto\x12\x1agoogle.cloud.asset.v1beta1\x1a\x1cgoogle/api/annotations.proto\x1a\x1agoogle/iam/v1/policy.proto\x1a\x19google/protobuf/any.proto\x1a\x1cgoogle/protobuf/struct.proto\x1a\x1fgoogle/protobuf/timestamp.proto"\x8a\x01\n\rTemporalAsset\x12\x36\n\x06window\x18\x01 \x01(\x0b\x32&.google.cloud.asset.v1beta1.TimeWindow\x12\x0f\n\x07\x64\x65leted\x18\x02 \x01(\x08\x12\x30\n\x05\x61sset\x18\x03 \x01(\x0b\x32!.google.cloud.asset.v1beta1.Asset"j\n\nTimeWindow\x12.\n\nstart_time\x18\x01 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12,\n\x08\x65nd_time\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.Timestamp"\x8c\x01\n\x05\x41sset\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x12\n\nasset_type\x18\x02 \x01(\t\x12\x36\n\x08resource\x18\x03 \x01(\x0b\x32$.google.cloud.asset.v1beta1.Resource\x12)\n\niam_policy\x18\x04 \x01(\x0b\x32\x15.google.iam.v1.Policy"\xa0\x01\n\x08Resource\x12\x0f\n\x07version\x18\x01 \x01(\t\x12\x1e\n\x16\x64iscovery_document_uri\x18\x02 \x01(\t\x12\x16\n\x0e\x64iscovery_name\x18\x03 \x01(\t\x12\x14\n\x0cresource_url\x18\x04 \x01(\t\x12\x0e\n\x06parent\x18\x05 \x01(\t\x12%\n\x04\x64\x61ta\x18\x06 \x01(\x0b\x32\x17.google.protobuf.StructB\xa9\x01\n\x1e\x63om.google.cloud.asset.v1beta1B\nAssetProtoP\x01Z?google.golang.org/genproto/googleapis/cloud/asset/v1beta1;asset\xaa\x02\x1aGoogle.Cloud.Asset.V1Beta1\xca\x02\x1aGoogle\\Cloud\\Asset\\V1beta1b\x06proto3'
+ ),
+ dependencies=[
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ google_dot_iam_dot_v1_dot_policy__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_any__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_struct__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_timestamp__pb2.DESCRIPTOR,
+ ],
+)
+
+
+_TEMPORALASSET = _descriptor.Descriptor(
+ name="TemporalAsset",
+ full_name="google.cloud.asset.v1beta1.TemporalAsset",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="window",
+ full_name="google.cloud.asset.v1beta1.TemporalAsset.window",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="deleted",
+ full_name="google.cloud.asset.v1beta1.TemporalAsset.deleted",
+ index=1,
+ number=2,
+ type=8,
+ cpp_type=7,
+ label=1,
+ has_default_value=False,
+ default_value=False,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="asset",
+ full_name="google.cloud.asset.v1beta1.TemporalAsset.asset",
+ index=2,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=226,
+ serialized_end=364,
+)
+
+
+_TIMEWINDOW = _descriptor.Descriptor(
+ name="TimeWindow",
+ full_name="google.cloud.asset.v1beta1.TimeWindow",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="start_time",
+ full_name="google.cloud.asset.v1beta1.TimeWindow.start_time",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="end_time",
+ full_name="google.cloud.asset.v1beta1.TimeWindow.end_time",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=366,
+ serialized_end=472,
+)
+
+
+_ASSET = _descriptor.Descriptor(
+ name="Asset",
+ full_name="google.cloud.asset.v1beta1.Asset",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.asset.v1beta1.Asset.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="asset_type",
+ full_name="google.cloud.asset.v1beta1.Asset.asset_type",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="resource",
+ full_name="google.cloud.asset.v1beta1.Asset.resource",
+ index=2,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="iam_policy",
+ full_name="google.cloud.asset.v1beta1.Asset.iam_policy",
+ index=3,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=475,
+ serialized_end=615,
+)
+
+
+_RESOURCE = _descriptor.Descriptor(
+ name="Resource",
+ full_name="google.cloud.asset.v1beta1.Resource",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="version",
+ full_name="google.cloud.asset.v1beta1.Resource.version",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="discovery_document_uri",
+ full_name="google.cloud.asset.v1beta1.Resource.discovery_document_uri",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="discovery_name",
+ full_name="google.cloud.asset.v1beta1.Resource.discovery_name",
+ index=2,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="resource_url",
+ full_name="google.cloud.asset.v1beta1.Resource.resource_url",
+ index=3,
+ number=4,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="parent",
+ full_name="google.cloud.asset.v1beta1.Resource.parent",
+ index=4,
+ number=5,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="data",
+ full_name="google.cloud.asset.v1beta1.Resource.data",
+ index=5,
+ number=6,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=618,
+ serialized_end=778,
+)
+
+_TEMPORALASSET.fields_by_name["window"].message_type = _TIMEWINDOW
+_TEMPORALASSET.fields_by_name["asset"].message_type = _ASSET
+_TIMEWINDOW.fields_by_name[
+ "start_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_TIMEWINDOW.fields_by_name[
+ "end_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_ASSET.fields_by_name["resource"].message_type = _RESOURCE
+_ASSET.fields_by_name[
+ "iam_policy"
+].message_type = (
+ google_dot_iam_dot_v1_dot_policy__pb2.google_dot_iam_dot_v1_dot_policy__pb2._POLICY
+)
+_RESOURCE.fields_by_name[
+ "data"
+].message_type = google_dot_protobuf_dot_struct__pb2._STRUCT
+DESCRIPTOR.message_types_by_name["TemporalAsset"] = _TEMPORALASSET
+DESCRIPTOR.message_types_by_name["TimeWindow"] = _TIMEWINDOW
+DESCRIPTOR.message_types_by_name["Asset"] = _ASSET
+DESCRIPTOR.message_types_by_name["Resource"] = _RESOURCE
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+TemporalAsset = _reflection.GeneratedProtocolMessageType(
+ "TemporalAsset",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TEMPORALASSET,
+ __module__="google.cloud.asset_v1beta1.proto.assets_pb2",
+ __doc__="""Temporal asset. In addition to the asset, the temporal asset includes
+ the status of the asset and valid from and to time of it.
+
+
+ Attributes:
+ window:
+ The time window when the asset data and state was observed.
+ deleted:
+ If the asset is deleted or not.
+ asset:
+ Asset.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1beta1.TemporalAsset)
+ ),
+)
+_sym_db.RegisterMessage(TemporalAsset)
+
+TimeWindow = _reflection.GeneratedProtocolMessageType(
+ "TimeWindow",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TIMEWINDOW,
+ __module__="google.cloud.asset_v1beta1.proto.assets_pb2",
+ __doc__="""A time window of (start\_time, end\_time].
+
+
+ Attributes:
+ start_time:
+ Start time of the time window (exclusive).
+ end_time:
+ End time of the time window (inclusive). Current timestamp if
+ not specified.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1beta1.TimeWindow)
+ ),
+)
+_sym_db.RegisterMessage(TimeWindow)
+
+Asset = _reflection.GeneratedProtocolMessageType(
+ "Asset",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_ASSET,
+ __module__="google.cloud.asset_v1beta1.proto.assets_pb2",
+ __doc__="""Cloud asset. This includes all Google Cloud Platform resources, Cloud
+ IAM policies, and other non-GCP assets.
+
+
+ Attributes:
+ name:
+
+ The full name of the asset. For example:
+ ``//compute.googleapis.com/projects/my_project_123/zones/zone1/instances/instance1``.
+ See https://cloud.google.com/apis/design/resource_names#full_resource_name
+ for more information.
+ asset_type:
+ Type of the asset. Example: "google.compute.Disk".
+ resource:
+ Representation of the resource.
+ iam_policy:
+ Representation of the actual Cloud IAM policy set on a cloud
+ resource. For each resource, there must be at most one Cloud
+ IAM policy set on it.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1beta1.Asset)
+ ),
+)
+_sym_db.RegisterMessage(Asset)
+
+Resource = _reflection.GeneratedProtocolMessageType(
+ "Resource",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_RESOURCE,
+ __module__="google.cloud.asset_v1beta1.proto.assets_pb2",
+ __doc__="""Representation of a cloud resource.
+
+
+ Attributes:
+ version:
+ The API version. Example: "v1".
+ discovery_document_uri:
+ The URL of the discovery document containing the resource's
+ JSON schema. For example: ``"https://www.googleapis.com/discov
+ ery/v1/apis/compute/v1/rest"``. It will be left unspecified
+ for resources without a discovery-based API, such as Cloud
+ Bigtable.
+ discovery_name:
+ The JSON schema name listed in the discovery document.
+ Example: "Project". It will be left unspecified for resources
+ (such as Cloud Bigtable) without a discovery-based API.
+ resource_url:
+ The REST URL for accessing the resource. An HTTP GET operation
+ using this URL returns the resource itself. Example:
+ ``https://cloudresourcemanager.googleapis.com/v1/projects/my-
+ project-123``. It will be left unspecified for resources
+ without a REST API.
+ parent:
+ The full name of the immediate parent of this resource. See
+ `Resource Names `__ for more information. For GCP
+ assets, it is the parent resource defined in the `Cloud IAM
+ policy hierarchy `__. For example: ``"//cloudresourcemanager.go
+ ogleapis.com/projects/my_project_123"``. For third-party
+ assets, it is up to the users to define.
+ data:
+ The content of the resource, in which some sensitive fields
+ are scrubbed away and may not be present.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1beta1.Resource)
+ ),
+)
+_sym_db.RegisterMessage(Resource)
+
+
+DESCRIPTOR._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/asset/google/cloud/asset_v1beta1/proto/assets_pb2_grpc.py b/asset/google/cloud/asset_v1beta1/proto/assets_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/asset/google/cloud/asset_v1beta1/proto/assets_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/asset/google/cloud/asset_v1beta1/types.py b/asset/google/cloud/asset_v1beta1/types.py
new file mode 100644
index 000000000000..cf051722e176
--- /dev/null
+++ b/asset/google/cloud/asset_v1beta1/types.py
@@ -0,0 +1,59 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from __future__ import absolute_import
+import sys
+
+from google.api_core.protobuf_helpers import get_messages
+
+from google.cloud.asset_v1beta1.proto import asset_service_pb2
+from google.cloud.asset_v1beta1.proto import assets_pb2
+from google.iam.v1 import policy_pb2
+from google.longrunning import operations_pb2
+from google.protobuf import any_pb2
+from google.protobuf import struct_pb2
+from google.protobuf import timestamp_pb2
+from google.rpc import status_pb2
+from google.type import expr_pb2
+
+
+_shared_modules = [
+ policy_pb2,
+ operations_pb2,
+ any_pb2,
+ struct_pb2,
+ timestamp_pb2,
+ status_pb2,
+ expr_pb2,
+]
+
+_local_modules = [asset_service_pb2, assets_pb2]
+
+names = []
+
+for module in _shared_modules: # pragma: NO COVER
+ for name, message in get_messages(module).items():
+ setattr(sys.modules[__name__], name, message)
+ names.append(name)
+for module in _local_modules:
+ for name, message in get_messages(module).items():
+ message.__module__ = "google.cloud.asset_v1beta1.types"
+ setattr(sys.modules[__name__], name, message)
+ names.append(name)
+
+
+__all__ = tuple(sorted(names))
diff --git a/asset/google/cloud/asset_v1p2beta1/__init__.py b/asset/google/cloud/asset_v1p2beta1/__init__.py
new file mode 100644
index 000000000000..07b349a3269b
--- /dev/null
+++ b/asset/google/cloud/asset_v1p2beta1/__init__.py
@@ -0,0 +1,30 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from __future__ import absolute_import
+
+from google.cloud.asset_v1p2beta1 import types
+from google.cloud.asset_v1p2beta1.gapic import asset_service_client
+from google.cloud.asset_v1p2beta1.gapic import enums
+
+
+class AssetServiceClient(asset_service_client.AssetServiceClient):
+ __doc__ = asset_service_client.AssetServiceClient.__doc__
+ enums = enums
+
+
+__all__ = ("enums", "types", "AssetServiceClient")
diff --git a/asset/google/cloud/asset_v1p2beta1/gapic/__init__.py b/asset/google/cloud/asset_v1p2beta1/gapic/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/asset/google/cloud/asset_v1p2beta1/gapic/asset_service_client.py b/asset/google/cloud/asset_v1p2beta1/gapic/asset_service_client.py
new file mode 100644
index 000000000000..b4cdc2b0da2d
--- /dev/null
+++ b/asset/google/cloud/asset_v1p2beta1/gapic/asset_service_client.py
@@ -0,0 +1,827 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Accesses the google.cloud.asset.v1p2beta1 AssetService API."""
+
+import pkg_resources
+import warnings
+
+from google.oauth2 import service_account
+import google.api_core.client_options
+import google.api_core.gapic_v1.client_info
+import google.api_core.gapic_v1.config
+import google.api_core.gapic_v1.method
+import google.api_core.gapic_v1.routing_header
+import google.api_core.grpc_helpers
+import google.api_core.operation
+import google.api_core.operations_v1
+import google.api_core.path_template
+import grpc
+
+from google.cloud.asset_v1p2beta1.gapic import asset_service_client_config
+from google.cloud.asset_v1p2beta1.gapic import enums
+from google.cloud.asset_v1p2beta1.gapic.transports import asset_service_grpc_transport
+from google.cloud.asset_v1p2beta1.proto import asset_service_pb2
+from google.cloud.asset_v1p2beta1.proto import asset_service_pb2_grpc
+from google.cloud.asset_v1p2beta1.proto import assets_pb2
+from google.longrunning import operations_pb2
+from google.protobuf import empty_pb2
+from google.protobuf import field_mask_pb2
+from google.protobuf import timestamp_pb2
+
+
+_GAPIC_LIBRARY_VERSION = pkg_resources.get_distribution("google-cloud-asset").version
+
+
+class AssetServiceClient(object):
+ """Asset service definition."""
+
+ SERVICE_ADDRESS = "cloudasset.googleapis.com:443"
+ """The default address of the service."""
+
+ # The name of the interface for this client. This is the key used to
+ # find the method configuration in the client_config dictionary.
+ _INTERFACE_NAME = "google.cloud.asset.v1p2beta1.AssetService"
+
+ @classmethod
+ def from_service_account_file(cls, filename, *args, **kwargs):
+ """Creates an instance of this client using the provided credentials
+ file.
+
+ Args:
+ filename (str): The path to the service account private key json
+ file.
+ args: Additional arguments to pass to the constructor.
+ kwargs: Additional arguments to pass to the constructor.
+
+ Returns:
+ AssetServiceClient: The constructed client.
+ """
+ credentials = service_account.Credentials.from_service_account_file(filename)
+ kwargs["credentials"] = credentials
+ return cls(*args, **kwargs)
+
+ from_service_account_json = from_service_account_file
+
+ @classmethod
+ def feed_path(cls, project, feed):
+ """Return a fully-qualified feed string."""
+ return google.api_core.path_template.expand(
+ "projects/{project}/feeds/{feed}", project=project, feed=feed
+ )
+
+ def __init__(
+ self,
+ transport=None,
+ channel=None,
+ credentials=None,
+ client_config=None,
+ client_info=None,
+ client_options=None,
+ ):
+ """Constructor.
+
+ Args:
+ transport (Union[~.AssetServiceGrpcTransport,
+ Callable[[~.Credentials, type], ~.AssetServiceGrpcTransport]): A transport
+ instance, responsible for actually making the API calls.
+ The default transport uses the gRPC protocol.
+ This argument may also be a callable which returns a
+ transport instance. Callables will be sent the credentials
+ as the first argument and the default transport class as
+ the second argument.
+ channel (grpc.Channel): DEPRECATED. A ``Channel`` instance
+ through which to make calls. This argument is mutually exclusive
+ with ``credentials``; providing both will raise an exception.
+ credentials (google.auth.credentials.Credentials): The
+ authorization credentials to attach to requests. These
+ credentials identify this application to the service. If none
+ are specified, the client will attempt to ascertain the
+ credentials from the environment.
+ This argument is mutually exclusive with providing a
+ transport instance to ``transport``; doing so will raise
+ an exception.
+ client_config (dict): DEPRECATED. A dictionary of call options for
+ each method. If not specified, the default configuration is used.
+ client_info (google.api_core.gapic_v1.client_info.ClientInfo):
+ The client info used to send a user-agent string along with
+ API requests. If ``None``, then default info will be used.
+ Generally, you only need to set this if you're developing
+ your own client library.
+ client_options (Union[dict, google.api_core.client_options.ClientOptions]):
+ Client options used to set user options on the client. API Endpoint
+ should be set through client_options.
+ """
+ # Raise deprecation warnings for things we want to go away.
+ if client_config is not None:
+ warnings.warn(
+ "The `client_config` argument is deprecated.",
+ PendingDeprecationWarning,
+ stacklevel=2,
+ )
+ else:
+ client_config = asset_service_client_config.config
+
+ if channel:
+ warnings.warn(
+ "The `channel` argument is deprecated; use " "`transport` instead.",
+ PendingDeprecationWarning,
+ stacklevel=2,
+ )
+
+ api_endpoint = self.SERVICE_ADDRESS
+ if client_options:
+ if type(client_options) == dict:
+ client_options = google.api_core.client_options.from_dict(
+ client_options
+ )
+ if client_options.api_endpoint:
+ api_endpoint = client_options.api_endpoint
+
+ # Instantiate the transport.
+ # The transport is responsible for handling serialization and
+ # deserialization and actually sending data to the service.
+ if transport:
+ if callable(transport):
+ self.transport = transport(
+ credentials=credentials,
+ default_class=asset_service_grpc_transport.AssetServiceGrpcTransport,
+ address=api_endpoint,
+ )
+ else:
+ if credentials:
+ raise ValueError(
+ "Received both a transport instance and "
+ "credentials; these are mutually exclusive."
+ )
+ self.transport = transport
+ else:
+ self.transport = asset_service_grpc_transport.AssetServiceGrpcTransport(
+ address=api_endpoint, channel=channel, credentials=credentials
+ )
+
+ if client_info is None:
+ client_info = google.api_core.gapic_v1.client_info.ClientInfo(
+ gapic_version=_GAPIC_LIBRARY_VERSION
+ )
+ else:
+ client_info.gapic_version = _GAPIC_LIBRARY_VERSION
+ self._client_info = client_info
+
+ # Parse out the default settings for retry and timeout for each RPC
+ # from the client configuration.
+ # (Ordinarily, these are the defaults specified in the `*_config.py`
+ # file next to this one.)
+ self._method_configs = google.api_core.gapic_v1.config.parse_method_configs(
+ client_config["interfaces"][self._INTERFACE_NAME]
+ )
+
+ # Save a dictionary of cached API call functions.
+ # These are the actual callables which invoke the proper
+ # transport methods, wrapped with `wrap_method` to add retry,
+ # timeout, and the like.
+ self._inner_api_calls = {}
+
+ # Service calls
+ def export_assets(
+ self,
+ parent,
+ output_config,
+ read_time=None,
+ asset_types=None,
+ content_type=None,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Exports assets with time and resource types to a given Cloud Storage
+ location. The output format is newline-delimited JSON. This API
+ implements the ``google.longrunning.Operation`` API allowing you to keep
+ track of the export.
+
+ Example:
+ >>> from google.cloud import asset_v1p2beta1
+ >>>
+ >>> client = asset_v1p2beta1.AssetServiceClient()
+ >>>
+ >>> # TODO: Initialize `parent`:
+ >>> parent = ''
+ >>>
+ >>> # TODO: Initialize `output_config`:
+ >>> output_config = {}
+ >>>
+ >>> response = client.export_assets(parent, output_config)
+ >>>
+ >>> def callback(operation_future):
+ ... # Handle result.
+ ... result = operation_future.result()
+ >>>
+ >>> response.add_done_callback(callback)
+ >>>
+ >>> # Handle metadata.
+ >>> metadata = response.metadata()
+
+ Args:
+ parent (str): Required. The relative name of the root asset. This can only be an
+ organization number (such as "organizations/123"), a project ID (such as
+ "projects/my-project-id"), or a project number (such as "projects/12345").
+ output_config (Union[dict, ~google.cloud.asset_v1p2beta1.types.OutputConfig]): Required. Output configuration indicating where the results will be output
+ to. All results will be in newline delimited JSON format.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.asset_v1p2beta1.types.OutputConfig`
+ read_time (Union[dict, ~google.cloud.asset_v1p2beta1.types.Timestamp]): Timestamp to take an asset snapshot. This can only be set to a timestamp
+ between 2018-10-02 UTC (inclusive) and the current time. If not specified,
+ the current time will be used. Due to delays in resource data collection
+ and indexing, there is a volatile window during which running the same
+ query may get different results.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.asset_v1p2beta1.types.Timestamp`
+ asset_types (list[str]): A list of asset types of which to take a snapshot for. For example:
+ "compute.googleapis.com/Disk". If specified, only matching assets will
+ be returned. See `Introduction to Cloud Asset
+ Inventory `__
+ for all supported asset types.
+ content_type (~google.cloud.asset_v1p2beta1.types.ContentType): Asset content type. If not specified, no content but the asset name will be
+ returned.
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.asset_v1p2beta1.types._OperationFuture` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "export_assets" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "export_assets"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.export_assets,
+ default_retry=self._method_configs["ExportAssets"].retry,
+ default_timeout=self._method_configs["ExportAssets"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = asset_service_pb2.ExportAssetsRequest(
+ parent=parent,
+ output_config=output_config,
+ read_time=read_time,
+ asset_types=asset_types,
+ content_type=content_type,
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("parent", parent)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ operation = self._inner_api_calls["export_assets"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+ return google.api_core.operation.from_gapic(
+ operation,
+ self.transport._operations_client,
+ asset_service_pb2.ExportAssetsResponse,
+ metadata_type=asset_service_pb2.ExportAssetsRequest,
+ )
+
+ def batch_get_assets_history(
+ self,
+ parent,
+ asset_names,
+ content_type,
+ read_time_window=None,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Batch gets the update history of assets that overlap a time window. For
+ RESOURCE content, this API outputs history with asset in both non-delete
+ or deleted status. For IAM\_POLICY content, this API outputs history
+ when the asset and its attached IAM POLICY both exist. This can create
+ gaps in the output history.
+
+ Example:
+ >>> from google.cloud import asset_v1p2beta1
+ >>> from google.cloud.asset_v1p2beta1 import enums
+ >>>
+ >>> client = asset_v1p2beta1.AssetServiceClient()
+ >>>
+ >>> # TODO: Initialize `parent`:
+ >>> parent = ''
+ >>>
+ >>> # TODO: Initialize `asset_names`:
+ >>> asset_names = []
+ >>>
+ >>> # TODO: Initialize `content_type`:
+ >>> content_type = enums.ContentType.CONTENT_TYPE_UNSPECIFIED
+ >>>
+ >>> response = client.batch_get_assets_history(parent, asset_names, content_type)
+
+ Args:
+ parent (str): Required. The relative name of the root asset. It can only be an
+ organization number (such as "organizations/123"), a project ID (such as
+ "projects/my-project-id")", or a project number (such as "projects/12345").
+ asset_names (list[str]): A list of the full names of the assets. For example:
+ ``//compute.googleapis.com/projects/my_project_123/zones/zone1/instances/instance1``.
+ See `Resource
+ Names `__
+ and `Resource Name
+ Format `__
+ for more info.
+
+ The request becomes a no-op if the asset name list is empty, and the max
+ size of the asset name list is 100 in one request.
+ content_type (~google.cloud.asset_v1p2beta1.types.ContentType): Required. The content type.
+ read_time_window (Union[dict, ~google.cloud.asset_v1p2beta1.types.TimeWindow]): Optional. The time window for the asset history. Both start\_time and
+ end\_time are optional and if set, it must be after 2018-10-02 UTC. If
+ end\_time is not set, it is default to current timestamp. If start\_time
+ is not set, the snapshot of the assets at end\_time will be returned.
+ The returned results contain all temporal assets whose time window
+ overlap with read\_time\_window.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.asset_v1p2beta1.types.TimeWindow`
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.asset_v1p2beta1.types.BatchGetAssetsHistoryResponse` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "batch_get_assets_history" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "batch_get_assets_history"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.batch_get_assets_history,
+ default_retry=self._method_configs["BatchGetAssetsHistory"].retry,
+ default_timeout=self._method_configs["BatchGetAssetsHistory"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = asset_service_pb2.BatchGetAssetsHistoryRequest(
+ parent=parent,
+ asset_names=asset_names,
+ content_type=content_type,
+ read_time_window=read_time_window,
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("parent", parent)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ return self._inner_api_calls["batch_get_assets_history"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+
+ def create_feed(
+ self,
+ parent,
+ feed_id,
+ feed,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Creates a feed in a parent project/folder/organization to listen to its
+ asset updates.
+
+ Example:
+ >>> from google.cloud import asset_v1p2beta1
+ >>>
+ >>> client = asset_v1p2beta1.AssetServiceClient()
+ >>>
+ >>> # TODO: Initialize `parent`:
+ >>> parent = ''
+ >>>
+ >>> # TODO: Initialize `feed_id`:
+ >>> feed_id = ''
+ >>>
+ >>> # TODO: Initialize `feed`:
+ >>> feed = {}
+ >>>
+ >>> response = client.create_feed(parent, feed_id, feed)
+
+ Args:
+ parent (str): Required. The name of the project/folder/organization where this feed
+ should be created in. It can only be an organization number (such as
+ "organizations/123"), a folder number (such as "folders/123"), a project ID
+ (such as "projects/my-project-id")", or a project number (such as
+ "projects/12345").
+ feed_id (str): Required. This is the client-assigned asset feed identifier and it needs to
+ be unique under a specific parent project/folder/organization.
+ feed (Union[dict, ~google.cloud.asset_v1p2beta1.types.Feed]): The feed details. The field ``name`` must be empty and it will be
+ generated in the format of: projects/project\_number/feeds/feed\_id
+ folders/folder\_number/feeds/feed\_id
+ organizations/organization\_number/feeds/feed\_id
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.asset_v1p2beta1.types.Feed`
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.asset_v1p2beta1.types.Feed` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "create_feed" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "create_feed"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.create_feed,
+ default_retry=self._method_configs["CreateFeed"].retry,
+ default_timeout=self._method_configs["CreateFeed"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = asset_service_pb2.CreateFeedRequest(
+ parent=parent, feed_id=feed_id, feed=feed
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("parent", parent)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ return self._inner_api_calls["create_feed"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+
+ def get_feed(
+ self,
+ name,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Gets details about an asset feed.
+
+ Example:
+ >>> from google.cloud import asset_v1p2beta1
+ >>>
+ >>> client = asset_v1p2beta1.AssetServiceClient()
+ >>>
+ >>> name = client.feed_path('[PROJECT]', '[FEED]')
+ >>>
+ >>> response = client.get_feed(name)
+
+ Args:
+ name (str): The name of the Feed and it must be in the format of:
+ projects/project\_number/feeds/feed\_id
+ folders/folder\_number/feeds/feed\_id
+ organizations/organization\_number/feeds/feed\_id
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.asset_v1p2beta1.types.Feed` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "get_feed" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "get_feed"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.get_feed,
+ default_retry=self._method_configs["GetFeed"].retry,
+ default_timeout=self._method_configs["GetFeed"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = asset_service_pb2.GetFeedRequest(name=name)
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("name", name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ return self._inner_api_calls["get_feed"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+
+ def list_feeds(
+ self,
+ parent,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Lists all asset feeds in a parent project/folder/organization.
+
+ Example:
+ >>> from google.cloud import asset_v1p2beta1
+ >>>
+ >>> client = asset_v1p2beta1.AssetServiceClient()
+ >>>
+ >>> # TODO: Initialize `parent`:
+ >>> parent = ''
+ >>>
+ >>> response = client.list_feeds(parent)
+
+ Args:
+ parent (str): Required. The parent project/folder/organization whose feeds are to be
+ listed. It can only be using project/folder/organization number (such as
+ "folders/12345")", or a project ID (such as "projects/my-project-id").
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.asset_v1p2beta1.types.ListFeedsResponse` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "list_feeds" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "list_feeds"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.list_feeds,
+ default_retry=self._method_configs["ListFeeds"].retry,
+ default_timeout=self._method_configs["ListFeeds"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = asset_service_pb2.ListFeedsRequest(parent=parent)
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("parent", parent)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ return self._inner_api_calls["list_feeds"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+
+ def update_feed(
+ self,
+ feed,
+ update_mask,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Updates an asset feed configuration.
+
+ Example:
+ >>> from google.cloud import asset_v1p2beta1
+ >>>
+ >>> client = asset_v1p2beta1.AssetServiceClient()
+ >>>
+ >>> # TODO: Initialize `feed`:
+ >>> feed = {}
+ >>>
+ >>> # TODO: Initialize `update_mask`:
+ >>> update_mask = {}
+ >>>
+ >>> response = client.update_feed(feed, update_mask)
+
+ Args:
+ feed (Union[dict, ~google.cloud.asset_v1p2beta1.types.Feed]): The new values of feed details. It must match an existing feed and the
+ field ``name`` must be in the format of:
+ projects/project\_number/feeds/feed\_id or
+ folders/folder\_number/feeds/feed\_id or
+ organizations/organization\_number/feeds/feed\_id.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.asset_v1p2beta1.types.Feed`
+ update_mask (Union[dict, ~google.cloud.asset_v1p2beta1.types.FieldMask]): Only updates the ``feed`` fields indicated by this mask. The field mask
+ must not be empty, and it must not contain fields that are immutable or
+ only set by the server.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.asset_v1p2beta1.types.FieldMask`
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.asset_v1p2beta1.types.Feed` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "update_feed" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "update_feed"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.update_feed,
+ default_retry=self._method_configs["UpdateFeed"].retry,
+ default_timeout=self._method_configs["UpdateFeed"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = asset_service_pb2.UpdateFeedRequest(
+ feed=feed, update_mask=update_mask
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("feed.name", feed.name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ return self._inner_api_calls["update_feed"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+
+ def delete_feed(
+ self,
+ name,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Deletes an asset feed.
+
+ Example:
+ >>> from google.cloud import asset_v1p2beta1
+ >>>
+ >>> client = asset_v1p2beta1.AssetServiceClient()
+ >>>
+ >>> name = client.feed_path('[PROJECT]', '[FEED]')
+ >>>
+ >>> client.delete_feed(name)
+
+ Args:
+ name (str): The name of the feed and it must be in the format of:
+ projects/project\_number/feeds/feed\_id
+ folders/folder\_number/feeds/feed\_id
+ organizations/organization\_number/feeds/feed\_id
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "delete_feed" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "delete_feed"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.delete_feed,
+ default_retry=self._method_configs["DeleteFeed"].retry,
+ default_timeout=self._method_configs["DeleteFeed"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = asset_service_pb2.DeleteFeedRequest(name=name)
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("name", name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ self._inner_api_calls["delete_feed"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
diff --git a/asset/google/cloud/asset_v1p2beta1/gapic/asset_service_client_config.py b/asset/google/cloud/asset_v1p2beta1/gapic/asset_service_client_config.py
new file mode 100644
index 000000000000..f3eab9e21e3b
--- /dev/null
+++ b/asset/google/cloud/asset_v1p2beta1/gapic/asset_service_client_config.py
@@ -0,0 +1,58 @@
+config = {
+ "interfaces": {
+ "google.cloud.asset.v1p2beta1.AssetService": {
+ "retry_codes": {
+ "idempotent": ["DEADLINE_EXCEEDED", "UNAVAILABLE"],
+ "non_idempotent": [],
+ },
+ "retry_params": {
+ "default": {
+ "initial_retry_delay_millis": 100,
+ "retry_delay_multiplier": 1.3,
+ "max_retry_delay_millis": 60000,
+ "initial_rpc_timeout_millis": 20000,
+ "rpc_timeout_multiplier": 1.0,
+ "max_rpc_timeout_millis": 20000,
+ "total_timeout_millis": 600000,
+ }
+ },
+ "methods": {
+ "ExportAssets": {
+ "timeout_millis": 600000,
+ "retry_codes_name": "non_idempotent",
+ "retry_params_name": "default",
+ },
+ "BatchGetAssetsHistory": {
+ "timeout_millis": 600000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "CreateFeed": {
+ "timeout_millis": 30000,
+ "retry_codes_name": "non_idempotent",
+ "retry_params_name": "default",
+ },
+ "GetFeed": {
+ "timeout_millis": 10000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "ListFeeds": {
+ "timeout_millis": 10000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "UpdateFeed": {
+ "timeout_millis": 30000,
+ "retry_codes_name": "non_idempotent",
+ "retry_params_name": "default",
+ },
+ "DeleteFeed": {
+ "timeout_millis": 30000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ },
+ }
+ }
+}
diff --git a/asset/google/cloud/asset_v1p2beta1/gapic/enums.py b/asset/google/cloud/asset_v1p2beta1/gapic/enums.py
new file mode 100644
index 000000000000..a9541ab2c818
--- /dev/null
+++ b/asset/google/cloud/asset_v1p2beta1/gapic/enums.py
@@ -0,0 +1,54 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Wrappers for protocol buffer enum types."""
+
+import enum
+
+
+class ContentType(enum.IntEnum):
+ """
+ Asset content type.
+
+ Attributes:
+ CONTENT_TYPE_UNSPECIFIED (int): Unspecified content type.
+ RESOURCE (int): Resource metadata.
+ IAM_POLICY (int): The actual IAM policy set on a resource.
+ IAM_POLICY_NAME (int): The IAM policy name for the IAM policy set on a resource.
+ ORG_POLICY (int)
+ ACCESS_POLICY (int): The Cloud Access context mananger Policy set on an asset.
+ """
+
+ CONTENT_TYPE_UNSPECIFIED = 0
+ RESOURCE = 1
+ IAM_POLICY = 2
+ IAM_POLICY_NAME = 3
+ ORG_POLICY = 4
+ ACCESS_POLICY = 5
+
+
+class NullValue(enum.IntEnum):
+ """
+ ``NullValue`` is a singleton enumeration to represent the null value for
+ the ``Value`` type union.
+
+ The JSON representation for ``NullValue`` is JSON ``null``.
+
+ Attributes:
+ NULL_VALUE (int): Null value.
+ """
+
+ NULL_VALUE = 0
diff --git a/asset/google/cloud/asset_v1p2beta1/gapic/transports/__init__.py b/asset/google/cloud/asset_v1p2beta1/gapic/transports/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/asset/google/cloud/asset_v1p2beta1/gapic/transports/asset_service_grpc_transport.py b/asset/google/cloud/asset_v1p2beta1/gapic/transports/asset_service_grpc_transport.py
new file mode 100644
index 000000000000..d7c33e9f0f5f
--- /dev/null
+++ b/asset/google/cloud/asset_v1p2beta1/gapic/transports/asset_service_grpc_transport.py
@@ -0,0 +1,215 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import google.api_core.grpc_helpers
+import google.api_core.operations_v1
+
+from google.cloud.asset_v1p2beta1.proto import asset_service_pb2_grpc
+
+
+class AssetServiceGrpcTransport(object):
+ """gRPC transport class providing stubs for
+ google.cloud.asset.v1p2beta1 AssetService API.
+
+ The transport provides access to the raw gRPC stubs,
+ which can be used to take advantage of advanced
+ features of gRPC.
+ """
+
+ # The scopes needed to make gRPC calls to all of the methods defined
+ # in this service.
+ _OAUTH_SCOPES = ("https://www.googleapis.com/auth/cloud-platform",)
+
+ def __init__(
+ self, channel=None, credentials=None, address="cloudasset.googleapis.com:443"
+ ):
+ """Instantiate the transport class.
+
+ Args:
+ channel (grpc.Channel): A ``Channel`` instance through
+ which to make calls. This argument is mutually exclusive
+ with ``credentials``; providing both will raise an exception.
+ credentials (google.auth.credentials.Credentials): The
+ authorization credentials to attach to requests. These
+ credentials identify this application to the service. If none
+ are specified, the client will attempt to ascertain the
+ credentials from the environment.
+ address (str): The address where the service is hosted.
+ """
+ # If both `channel` and `credentials` are specified, raise an
+ # exception (channels come with credentials baked in already).
+ if channel is not None and credentials is not None:
+ raise ValueError(
+ "The `channel` and `credentials` arguments are mutually " "exclusive."
+ )
+
+ # Create the channel.
+ if channel is None:
+ channel = self.create_channel(
+ address=address,
+ credentials=credentials,
+ options={
+ "grpc.max_send_message_length": -1,
+ "grpc.max_receive_message_length": -1,
+ }.items(),
+ )
+
+ self._channel = channel
+
+ # gRPC uses objects called "stubs" that are bound to the
+ # channel and provide a basic method for each RPC.
+ self._stubs = {
+ "asset_service_stub": asset_service_pb2_grpc.AssetServiceStub(channel)
+ }
+
+ # Because this API includes a method that returns a
+ # long-running operation (proto: google.longrunning.Operation),
+ # instantiate an LRO client.
+ self._operations_client = google.api_core.operations_v1.OperationsClient(
+ channel
+ )
+
+ @classmethod
+ def create_channel(
+ cls, address="cloudasset.googleapis.com:443", credentials=None, **kwargs
+ ):
+ """Create and return a gRPC channel object.
+
+ Args:
+ address (str): The host for the channel to use.
+ credentials (~.Credentials): The
+ authorization credentials to attach to requests. These
+ credentials identify this application to the service. If
+ none are specified, the client will attempt to ascertain
+ the credentials from the environment.
+ kwargs (dict): Keyword arguments, which are passed to the
+ channel creation.
+
+ Returns:
+ grpc.Channel: A gRPC channel object.
+ """
+ return google.api_core.grpc_helpers.create_channel(
+ address, credentials=credentials, scopes=cls._OAUTH_SCOPES, **kwargs
+ )
+
+ @property
+ def channel(self):
+ """The gRPC channel used by the transport.
+
+ Returns:
+ grpc.Channel: A gRPC channel object.
+ """
+ return self._channel
+
+ @property
+ def export_assets(self):
+ """Return the gRPC stub for :meth:`AssetServiceClient.export_assets`.
+
+ Exports assets with time and resource types to a given Cloud Storage
+ location. The output format is newline-delimited JSON. This API
+ implements the ``google.longrunning.Operation`` API allowing you to keep
+ track of the export.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["asset_service_stub"].ExportAssets
+
+ @property
+ def batch_get_assets_history(self):
+ """Return the gRPC stub for :meth:`AssetServiceClient.batch_get_assets_history`.
+
+ Batch gets the update history of assets that overlap a time window. For
+ RESOURCE content, this API outputs history with asset in both non-delete
+ or deleted status. For IAM\_POLICY content, this API outputs history
+ when the asset and its attached IAM POLICY both exist. This can create
+ gaps in the output history.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["asset_service_stub"].BatchGetAssetsHistory
+
+ @property
+ def create_feed(self):
+ """Return the gRPC stub for :meth:`AssetServiceClient.create_feed`.
+
+ Creates a feed in a parent project/folder/organization to listen to its
+ asset updates.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["asset_service_stub"].CreateFeed
+
+ @property
+ def get_feed(self):
+ """Return the gRPC stub for :meth:`AssetServiceClient.get_feed`.
+
+ Gets details about an asset feed.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["asset_service_stub"].GetFeed
+
+ @property
+ def list_feeds(self):
+ """Return the gRPC stub for :meth:`AssetServiceClient.list_feeds`.
+
+ Lists all asset feeds in a parent project/folder/organization.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["asset_service_stub"].ListFeeds
+
+ @property
+ def update_feed(self):
+ """Return the gRPC stub for :meth:`AssetServiceClient.update_feed`.
+
+ Updates an asset feed configuration.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["asset_service_stub"].UpdateFeed
+
+ @property
+ def delete_feed(self):
+ """Return the gRPC stub for :meth:`AssetServiceClient.delete_feed`.
+
+ Deletes an asset feed.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["asset_service_stub"].DeleteFeed
diff --git a/asset/google/cloud/asset_v1p2beta1/proto/__init__.py b/asset/google/cloud/asset_v1p2beta1/proto/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/asset/google/cloud/asset_v1p2beta1/proto/asset_service.proto b/asset/google/cloud/asset_v1p2beta1/proto/asset_service.proto
new file mode 100644
index 000000000000..0d461d0fa81c
--- /dev/null
+++ b/asset/google/cloud/asset_v1p2beta1/proto/asset_service.proto
@@ -0,0 +1,384 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.asset.v1p2beta1;
+
+import "google/api/annotations.proto";
+import "google/cloud/asset/v1p2beta1/assets.proto";
+import "google/longrunning/operations.proto";
+import "google/protobuf/empty.proto";
+import "google/protobuf/field_mask.proto";
+import "google/protobuf/timestamp.proto";
+import "google/api/client.proto";
+
+option csharp_namespace = "Google.Cloud.Asset.V1p2Beta1";
+option go_package = "google.golang.org/genproto/googleapis/cloud/asset/v1p2beta1;asset";
+option java_multiple_files = true;
+option java_outer_classname = "AssetServiceProto";
+option java_package = "com.google.cloud.asset.v1p2beta1";
+option php_namespace = "Google\\Cloud\\Asset\\V1p2Beta1";
+
+// Asset service definition.
+service AssetService {
+ option (google.api.default_host) = "cloudasset.googleapis.com";
+ option (google.api.oauth_scopes) = "https://www.googleapis.com/auth/cloud-platform";
+
+ // Exports assets with time and resource types to a given Cloud Storage
+ // location. The output format is newline-delimited JSON.
+ // This API implements the [google.longrunning.Operation][google.longrunning.Operation] API allowing you
+ // to keep track of the export.
+ rpc ExportAssets(ExportAssetsRequest) returns (google.longrunning.Operation) {
+ option (google.api.http) = {
+ post: "/v1p2beta1/{parent=*/*}:exportAssets"
+ body: "*"
+ };
+ }
+
+ // Batch gets the update history of assets that overlap a time window.
+ // For RESOURCE content, this API outputs history with asset in both
+ // non-delete or deleted status.
+ // For IAM_POLICY content, this API outputs history when the asset and its
+ // attached IAM POLICY both exist. This can create gaps in the output history.
+ rpc BatchGetAssetsHistory(BatchGetAssetsHistoryRequest) returns (BatchGetAssetsHistoryResponse) {
+ option (google.api.http) = {
+ get: "/v1p2beta1/{parent=*/*}:batchGetAssetsHistory"
+ };
+ }
+
+ // Creates a feed in a parent project/folder/organization to listen to its
+ // asset updates.
+ rpc CreateFeed(CreateFeedRequest) returns (Feed) {
+ option (google.api.http) = {
+ post: "/v1p2beta1/{parent=*/*}/feeds"
+ body: "*"
+ };
+ }
+
+ // Gets details about an asset feed.
+ rpc GetFeed(GetFeedRequest) returns (Feed) {
+ option (google.api.http) = {
+ get: "/v1p2beta1/{name=*/*/feeds/*}"
+ };
+ }
+
+ // Lists all asset feeds in a parent project/folder/organization.
+ rpc ListFeeds(ListFeedsRequest) returns (ListFeedsResponse) {
+ option (google.api.http) = {
+ get: "/v1p2beta1/{parent=*/*}/feeds"
+ };
+ }
+
+ // Updates an asset feed configuration.
+ rpc UpdateFeed(UpdateFeedRequest) returns (Feed) {
+ option (google.api.http) = {
+ patch: "/v1p2beta1/{feed.name=*/*/feeds/*}"
+ body: "*"
+ };
+ }
+
+ // Deletes an asset feed.
+ rpc DeleteFeed(DeleteFeedRequest) returns (google.protobuf.Empty) {
+ option (google.api.http) = {
+ delete: "/v1p2beta1/{name=*/*/feeds/*}"
+ };
+ }
+}
+
+// Export asset request.
+message ExportAssetsRequest {
+ // Required. The relative name of the root asset. This can only be an
+ // organization number (such as "organizations/123"), a project ID (such as
+ // "projects/my-project-id"), or a project number (such as "projects/12345").
+ string parent = 1;
+
+ // Timestamp to take an asset snapshot. This can only be set to a timestamp
+ // between 2018-10-02 UTC (inclusive) and the current time. If not specified,
+ // the current time will be used. Due to delays in resource data collection
+ // and indexing, there is a volatile window during which running the same
+ // query may get different results.
+ google.protobuf.Timestamp read_time = 2;
+
+ // A list of asset types of which to take a snapshot for. For example:
+ // "compute.googleapis.com/Disk". If specified, only matching assets will be
+ // returned. See [Introduction to Cloud Asset
+ // Inventory](https://cloud.google.com/resource-manager/docs/cloud-asset-inventory/overview)
+ // for all supported asset types.
+ repeated string asset_types = 3;
+
+ // Asset content type. If not specified, no content but the asset name will be
+ // returned.
+ ContentType content_type = 4;
+
+ // Required. Output configuration indicating where the results will be output
+ // to. All results will be in newline delimited JSON format.
+ OutputConfig output_config = 5;
+}
+
+// The export asset response. This message is returned by the
+// [google.longrunning.Operations.GetOperation][google.longrunning.Operations.GetOperation] method in the returned
+// [google.longrunning.Operation.response][google.longrunning.Operation.response] field.
+message ExportAssetsResponse {
+ // Time the snapshot was taken.
+ google.protobuf.Timestamp read_time = 1;
+
+ // Output configuration indicating where the results were output to.
+ // All results are in JSON format.
+ OutputConfig output_config = 2;
+}
+
+// Batch get assets history request.
+message BatchGetAssetsHistoryRequest {
+ // Required. The relative name of the root asset. It can only be an
+ // organization number (such as "organizations/123"), a project ID (such as
+ // "projects/my-project-id")", or a project number (such as "projects/12345").
+ string parent = 1;
+
+ // A list of the full names of the assets. For example:
+ // `//compute.googleapis.com/projects/my_project_123/zones/zone1/instances/instance1`.
+ // See [Resource
+ // Names](https://cloud.google.com/apis/design/resource_names#full_resource_name)
+ // and [Resource Name
+ // Format](https://cloud.google.com/resource-manager/docs/cloud-asset-inventory/resource-name-format)
+ // for more info.
+ //
+ // The request becomes a no-op if the asset name list is empty, and the max
+ // size of the asset name list is 100 in one request.
+ repeated string asset_names = 2;
+
+ // Required. The content type.
+ ContentType content_type = 3;
+
+ // Optional. The time window for the asset history. Both start_time and
+ // end_time are optional and if set, it must be after 2018-10-02 UTC. If
+ // end_time is not set, it is default to current timestamp. If start_time is
+ // not set, the snapshot of the assets at end_time will be returned. The
+ // returned results contain all temporal assets whose time window overlap with
+ // read_time_window.
+ TimeWindow read_time_window = 4;
+}
+
+// Batch get assets history response.
+message BatchGetAssetsHistoryResponse {
+ // A list of assets with valid time windows.
+ repeated TemporalAsset assets = 1;
+}
+
+// Create asset feed request.
+message CreateFeedRequest {
+ // Required. The name of the project/folder/organization where this feed
+ // should be created in. It can only be an organization number (such as
+ // "organizations/123"), a folder number (such as "folders/123"), a project ID
+ // (such as "projects/my-project-id")", or a project number (such as
+ // "projects/12345").
+ string parent = 1;
+
+ // Required. This is the client-assigned asset feed identifier and it needs to
+ // be unique under a specific parent project/folder/organization.
+ string feed_id = 2;
+
+ // The feed details. The field `name` must be empty and it will be generated
+ // in the format of:
+ // projects/project_number/feeds/feed_id
+ // folders/folder_number/feeds/feed_id
+ // organizations/organization_number/feeds/feed_id
+ Feed feed = 3;
+}
+
+// Get asset feed request.
+message GetFeedRequest {
+ // The name of the Feed and it must be in the format of:
+ // projects/project_number/feeds/feed_id
+ // folders/folder_number/feeds/feed_id
+ // organizations/organization_number/feeds/feed_id
+ string name = 1;
+}
+
+// List asset feeds request.
+message ListFeedsRequest {
+ // Required. The parent project/folder/organization whose feeds are to be
+ // listed. It can only be using project/folder/organization number (such as
+ // "folders/12345")", or a project ID (such as "projects/my-project-id").
+ string parent = 1;
+}
+
+// List asset feeds response.
+message ListFeedsResponse {
+ // A list of feeds.
+ repeated Feed feeds = 1;
+}
+
+// Update asset feed request.
+message UpdateFeedRequest {
+ // The new values of feed details. It must match an existing feed and the
+ // field `name` must be in the format of:
+ // projects/project_number/feeds/feed_id or
+ // folders/folder_number/feeds/feed_id or
+ // organizations/organization_number/feeds/feed_id.
+ Feed feed = 1;
+
+ // Only updates the `feed` fields indicated by this mask.
+ // The field mask must not be empty, and it must not contain fields that
+ // are immutable or only set by the server.
+ google.protobuf.FieldMask update_mask = 2;
+}
+
+// Delete asset feed request.
+message DeleteFeedRequest {
+ // The name of the feed and it must be in the format of:
+ // projects/project_number/feeds/feed_id
+ // folders/folder_number/feeds/feed_id
+ // organizations/organization_number/feeds/feed_id
+ string name = 1;
+}
+
+// Output configuration for export assets destination.
+message OutputConfig {
+ // Asset export destination.
+ oneof destination {
+ // Destination on Cloud Storage.
+ GcsDestination gcs_destination = 1;
+
+ // Destination on Bigquery. The output table stores the fields in asset
+ // proto as columns in BigQuery. The resource/iam_policy field is converted
+ // to a record with each field to a column, except metadata to a single JSON
+ // string.
+ BigQueryDestination bigquery_destination = 2;
+ }
+}
+
+// A Cloud Storage location.
+message GcsDestination {
+ // Required.
+ oneof object_uri {
+ // The uri of the Cloud Storage object. It's the same uri that is used by
+ // gsutil. For example: "gs://bucket_name/object_name". See [Viewing and
+ // Editing Object
+ // Metadata](https://cloud.google.com/storage/docs/viewing-editing-metadata)
+ // for more information.
+ string uri = 1;
+
+ // The uri prefix of all generated Cloud Storage objects. For example:
+ // "gs://bucket_name/object_name_prefix". Each object uri is in format:
+ // "gs://bucket_name/object_name_prefix// and only
+ // contains assets for that type. starts from 0. For example:
+ // "gs://bucket_name/object_name_prefix/compute.googleapis.com/Disk/0" is
+ // the first shard of output objects containing all
+ // compute.googleapis.com/Disk assets. An INVALID_ARGUMENT error will be
+ // returned if file with the same name "gs://bucket_name/object_name_prefix"
+ // already exists.
+ string uri_prefix = 2;
+ }
+}
+
+// A Bigquery destination.
+message BigQueryDestination {
+ // Required. The BigQuery dataset in format
+ // "projects/projectId/datasets/datasetId", to which the snapshot result
+ // should be exported. If this dataset does not exist, the export call returns
+ // an error.
+ string dataset = 1;
+
+ // Required. The BigQuery table to which the snapshot result should be
+ // written. If this table does not exist, a new table with the given name
+ // will be created.
+ string table = 2;
+
+ // If the destination table already exists and this flag is `TRUE`, the
+ // table will be overwritten by the contents of assets snapshot. If the flag
+ // is not set and the destination table already exists, the export call
+ // returns an error.
+ bool force = 3;
+}
+
+// A Cloud Pubsub destination.
+message PubsubDestination {
+ // The name of the Cloud Pub/Sub topic to publish to.
+ // For example: `projects/PROJECT_ID/topics/TOPIC_ID`.
+ string topic = 1;
+}
+
+// Asset content type.
+enum ContentType {
+ // Unspecified content type.
+ CONTENT_TYPE_UNSPECIFIED = 0;
+
+ // Resource metadata.
+ RESOURCE = 1;
+
+ // The actual IAM policy set on a resource.
+ IAM_POLICY = 2;
+
+ // The IAM policy name for the IAM policy set on a resource.
+ IAM_POLICY_NAME = 3;
+
+ ORG_POLICY = 4;
+
+ // The Cloud Access context mananger Policy set on an asset.
+ ACCESS_POLICY = 5;
+}
+
+// Output configuration for asset feed destination.
+message FeedOutputConfig {
+ // Asset feed destination.
+ oneof destination {
+ // Destination on Cloud Pubsub.
+ PubsubDestination pubsub_destination = 1;
+ }
+}
+
+// An asset feed used to export asset updates to a destinations.
+// An asset feed filter controls what updates are exported.
+// The asset feed must be created within a project, organization, or
+// folder. Supported destinations are:
+// Cloud Pub/Sub topics.
+message Feed {
+ // Required. The format will be
+ // projects/{project_number}/feeds/{client-assigned_feed_identifier} or
+ // folders/{folder_number}/feeds/{client-assigned_feed_identifier} or
+ // organizations/{organization_number}/feeds/{client-assigned_feed_identifier}
+ //
+ // The client-assigned feed identifier must be unique within the parent
+ // project/folder/organization.
+ string name = 1;
+
+ // A list of the full names of the assets to receive updates. You must specify
+ // either or both of asset_names and asset_types. Only asset updates matching
+ // specified asset_names and asset_types are exported to the feed. For
+ // example:
+ // `//compute.googleapis.com/projects/my_project_123/zones/zone1/instances/instance1`.
+ // See [Resource
+ // Names](https://cloud.google.com/apis/design/resource_names#full_resource_name)
+ // for more info.
+ repeated string asset_names = 2;
+
+ // A list of types of the assets to receive updates. You must specify either
+ // or both of asset_names and asset_types. Only asset updates matching
+ // specified asset_names and asset_types are exported to the feed.
+ // For example:
+ // "compute.googleapis.com/Disk" See [Introduction to Cloud Asset
+ // Inventory](https://cloud.google.com/resource-manager/docs/cloud-asset-inventory/overview)
+ // for all supported asset types.
+ repeated string asset_types = 3;
+
+ // Asset content type. If not specified, no content but the asset name and
+ // type will be returned.
+ ContentType content_type = 4;
+
+ // Required. Feed output configuration defining where the asset updates are
+ // published to.
+ FeedOutputConfig feed_output_config = 5;
+}
diff --git a/asset/google/cloud/asset_v1p2beta1/proto/asset_service_pb2.py b/asset/google/cloud/asset_v1p2beta1/proto/asset_service_pb2.py
new file mode 100644
index 000000000000..0275b6737a02
--- /dev/null
+++ b/asset/google/cloud/asset_v1p2beta1/proto/asset_service_pb2.py
@@ -0,0 +1,1717 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/asset_v1p2beta1/proto/asset_service.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf.internal import enum_type_wrapper
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+from google.cloud.asset_v1p2beta1.proto import (
+ assets_pb2 as google_dot_cloud_dot_asset__v1p2beta1_dot_proto_dot_assets__pb2,
+)
+from google.longrunning import (
+ operations_pb2 as google_dot_longrunning_dot_operations__pb2,
+)
+from google.protobuf import empty_pb2 as google_dot_protobuf_dot_empty__pb2
+from google.protobuf import field_mask_pb2 as google_dot_protobuf_dot_field__mask__pb2
+from google.protobuf import timestamp_pb2 as google_dot_protobuf_dot_timestamp__pb2
+from google.api import client_pb2 as google_dot_api_dot_client__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/asset_v1p2beta1/proto/asset_service.proto",
+ package="google.cloud.asset.v1p2beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n com.google.cloud.asset.v1p2beta1B\021AssetServiceProtoP\001ZAgoogle.golang.org/genproto/googleapis/cloud/asset/v1p2beta1;asset\252\002\034Google.Cloud.Asset.V1p2Beta1\312\002\034Google\\Cloud\\Asset\\V1p2Beta1"
+ ),
+ serialized_pb=_b(
+ '\n6google/cloud/asset_v1p2beta1/proto/asset_service.proto\x12\x1cgoogle.cloud.asset.v1p2beta1\x1a\x1cgoogle/api/annotations.proto\x1a/google/cloud/asset_v1p2beta1/proto/assets.proto\x1a#google/longrunning/operations.proto\x1a\x1bgoogle/protobuf/empty.proto\x1a google/protobuf/field_mask.proto\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x17google/api/client.proto"\xed\x01\n\x13\x45xportAssetsRequest\x12\x0e\n\x06parent\x18\x01 \x01(\t\x12-\n\tread_time\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12\x13\n\x0b\x61sset_types\x18\x03 \x03(\t\x12?\n\x0c\x63ontent_type\x18\x04 \x01(\x0e\x32).google.cloud.asset.v1p2beta1.ContentType\x12\x41\n\routput_config\x18\x05 \x01(\x0b\x32*.google.cloud.asset.v1p2beta1.OutputConfig"\x88\x01\n\x14\x45xportAssetsResponse\x12-\n\tread_time\x18\x01 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12\x41\n\routput_config\x18\x02 \x01(\x0b\x32*.google.cloud.asset.v1p2beta1.OutputConfig"\xc8\x01\n\x1c\x42\x61tchGetAssetsHistoryRequest\x12\x0e\n\x06parent\x18\x01 \x01(\t\x12\x13\n\x0b\x61sset_names\x18\x02 \x03(\t\x12?\n\x0c\x63ontent_type\x18\x03 \x01(\x0e\x32).google.cloud.asset.v1p2beta1.ContentType\x12\x42\n\x10read_time_window\x18\x04 \x01(\x0b\x32(.google.cloud.asset.v1p2beta1.TimeWindow"\\\n\x1d\x42\x61tchGetAssetsHistoryResponse\x12;\n\x06\x61ssets\x18\x01 \x03(\x0b\x32+.google.cloud.asset.v1p2beta1.TemporalAsset"f\n\x11\x43reateFeedRequest\x12\x0e\n\x06parent\x18\x01 \x01(\t\x12\x0f\n\x07\x66\x65\x65\x64_id\x18\x02 \x01(\t\x12\x30\n\x04\x66\x65\x65\x64\x18\x03 \x01(\x0b\x32".google.cloud.asset.v1p2beta1.Feed"\x1e\n\x0eGetFeedRequest\x12\x0c\n\x04name\x18\x01 \x01(\t""\n\x10ListFeedsRequest\x12\x0e\n\x06parent\x18\x01 \x01(\t"F\n\x11ListFeedsResponse\x12\x31\n\x05\x66\x65\x65\x64s\x18\x01 \x03(\x0b\x32".google.cloud.asset.v1p2beta1.Feed"v\n\x11UpdateFeedRequest\x12\x30\n\x04\x66\x65\x65\x64\x18\x01 \x01(\x0b\x32".google.cloud.asset.v1p2beta1.Feed\x12/\n\x0bupdate_mask\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.FieldMask"!\n\x11\x44\x65leteFeedRequest\x12\x0c\n\x04name\x18\x01 \x01(\t"\xb9\x01\n\x0cOutputConfig\x12G\n\x0fgcs_destination\x18\x01 \x01(\x0b\x32,.google.cloud.asset.v1p2beta1.GcsDestinationH\x00\x12Q\n\x14\x62igquery_destination\x18\x02 \x01(\x0b\x32\x31.google.cloud.asset.v1p2beta1.BigQueryDestinationH\x00\x42\r\n\x0b\x64\x65stination"C\n\x0eGcsDestination\x12\r\n\x03uri\x18\x01 \x01(\tH\x00\x12\x14\n\nuri_prefix\x18\x02 \x01(\tH\x00\x42\x0c\n\nobject_uri"D\n\x13\x42igQueryDestination\x12\x0f\n\x07\x64\x61taset\x18\x01 \x01(\t\x12\r\n\x05table\x18\x02 \x01(\t\x12\r\n\x05\x66orce\x18\x03 \x01(\x08""\n\x11PubsubDestination\x12\r\n\x05topic\x18\x01 \x01(\t"p\n\x10\x46\x65\x65\x64OutputConfig\x12M\n\x12pubsub_destination\x18\x01 \x01(\x0b\x32/.google.cloud.asset.v1p2beta1.PubsubDestinationH\x00\x42\r\n\x0b\x64\x65stination"\xcb\x01\n\x04\x46\x65\x65\x64\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x13\n\x0b\x61sset_names\x18\x02 \x03(\t\x12\x13\n\x0b\x61sset_types\x18\x03 \x03(\t\x12?\n\x0c\x63ontent_type\x18\x04 \x01(\x0e\x32).google.cloud.asset.v1p2beta1.ContentType\x12J\n\x12\x66\x65\x65\x64_output_config\x18\x05 \x01(\x0b\x32..google.cloud.asset.v1p2beta1.FeedOutputConfig*\x81\x01\n\x0b\x43ontentType\x12\x1c\n\x18\x43ONTENT_TYPE_UNSPECIFIED\x10\x00\x12\x0c\n\x08RESOURCE\x10\x01\x12\x0e\n\nIAM_POLICY\x10\x02\x12\x13\n\x0fIAM_POLICY_NAME\x10\x03\x12\x0e\n\nORG_POLICY\x10\x04\x12\x11\n\rACCESS_POLICY\x10\x05\x32\xf5\x08\n\x0c\x41ssetService\x12\x91\x01\n\x0c\x45xportAssets\x12\x31.google.cloud.asset.v1p2beta1.ExportAssetsRequest\x1a\x1d.google.longrunning.Operation"/\x82\xd3\xe4\x93\x02)"$/v1p2beta1/{parent=*/*}:exportAssets:\x01*\x12\xc7\x01\n\x15\x42\x61tchGetAssetsHistory\x12:.google.cloud.asset.v1p2beta1.BatchGetAssetsHistoryRequest\x1a;.google.cloud.asset.v1p2beta1.BatchGetAssetsHistoryResponse"5\x82\xd3\xe4\x93\x02/\x12-/v1p2beta1/{parent=*/*}:batchGetAssetsHistory\x12\x8b\x01\n\nCreateFeed\x12/.google.cloud.asset.v1p2beta1.CreateFeedRequest\x1a".google.cloud.asset.v1p2beta1.Feed"(\x82\xd3\xe4\x93\x02""\x1d/v1p2beta1/{parent=*/*}/feeds:\x01*\x12\x82\x01\n\x07GetFeed\x12,.google.cloud.asset.v1p2beta1.GetFeedRequest\x1a".google.cloud.asset.v1p2beta1.Feed"%\x82\xd3\xe4\x93\x02\x1f\x12\x1d/v1p2beta1/{name=*/*/feeds/*}\x12\x93\x01\n\tListFeeds\x12..google.cloud.asset.v1p2beta1.ListFeedsRequest\x1a/.google.cloud.asset.v1p2beta1.ListFeedsResponse"%\x82\xd3\xe4\x93\x02\x1f\x12\x1d/v1p2beta1/{parent=*/*}/feeds\x12\x90\x01\n\nUpdateFeed\x12/.google.cloud.asset.v1p2beta1.UpdateFeedRequest\x1a".google.cloud.asset.v1p2beta1.Feed"-\x82\xd3\xe4\x93\x02\'2"/v1p2beta1/{feed.name=*/*/feeds/*}:\x01*\x12|\n\nDeleteFeed\x12/.google.cloud.asset.v1p2beta1.DeleteFeedRequest\x1a\x16.google.protobuf.Empty"%\x82\xd3\xe4\x93\x02\x1f*\x1d/v1p2beta1/{name=*/*/feeds/*}\x1aM\xca\x41\x19\x63loudasset.googleapis.com\xd2\x41.https://www.googleapis.com/auth/cloud-platformB\xb8\x01\n com.google.cloud.asset.v1p2beta1B\x11\x41ssetServiceProtoP\x01ZAgoogle.golang.org/genproto/googleapis/cloud/asset/v1p2beta1;asset\xaa\x02\x1cGoogle.Cloud.Asset.V1p2Beta1\xca\x02\x1cGoogle\\Cloud\\Asset\\V1p2Beta1b\x06proto3'
+ ),
+ dependencies=[
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_asset__v1p2beta1_dot_proto_dot_assets__pb2.DESCRIPTOR,
+ google_dot_longrunning_dot_operations__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_empty__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_field__mask__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_timestamp__pb2.DESCRIPTOR,
+ google_dot_api_dot_client__pb2.DESCRIPTOR,
+ ],
+)
+
+_CONTENTTYPE = _descriptor.EnumDescriptor(
+ name="ContentType",
+ full_name="google.cloud.asset.v1p2beta1.ContentType",
+ filename=None,
+ file=DESCRIPTOR,
+ values=[
+ _descriptor.EnumValueDescriptor(
+ name="CONTENT_TYPE_UNSPECIFIED",
+ index=0,
+ number=0,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="RESOURCE", index=1, number=1, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="IAM_POLICY", index=2, number=2, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="IAM_POLICY_NAME",
+ index=3,
+ number=3,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="ORG_POLICY", index=4, number=4, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="ACCESS_POLICY", index=5, number=5, serialized_options=None, type=None
+ ),
+ ],
+ containing_type=None,
+ serialized_options=None,
+ serialized_start=2084,
+ serialized_end=2213,
+)
+_sym_db.RegisterEnumDescriptor(_CONTENTTYPE)
+
+ContentType = enum_type_wrapper.EnumTypeWrapper(_CONTENTTYPE)
+CONTENT_TYPE_UNSPECIFIED = 0
+RESOURCE = 1
+IAM_POLICY = 2
+IAM_POLICY_NAME = 3
+ORG_POLICY = 4
+ACCESS_POLICY = 5
+
+
+_EXPORTASSETSREQUEST = _descriptor.Descriptor(
+ name="ExportAssetsRequest",
+ full_name="google.cloud.asset.v1p2beta1.ExportAssetsRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="parent",
+ full_name="google.cloud.asset.v1p2beta1.ExportAssetsRequest.parent",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="read_time",
+ full_name="google.cloud.asset.v1p2beta1.ExportAssetsRequest.read_time",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="asset_types",
+ full_name="google.cloud.asset.v1p2beta1.ExportAssetsRequest.asset_types",
+ index=2,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="content_type",
+ full_name="google.cloud.asset.v1p2beta1.ExportAssetsRequest.content_type",
+ index=3,
+ number=4,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="output_config",
+ full_name="google.cloud.asset.v1p2beta1.ExportAssetsRequest.output_config",
+ index=4,
+ number=5,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=326,
+ serialized_end=563,
+)
+
+
+_EXPORTASSETSRESPONSE = _descriptor.Descriptor(
+ name="ExportAssetsResponse",
+ full_name="google.cloud.asset.v1p2beta1.ExportAssetsResponse",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="read_time",
+ full_name="google.cloud.asset.v1p2beta1.ExportAssetsResponse.read_time",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="output_config",
+ full_name="google.cloud.asset.v1p2beta1.ExportAssetsResponse.output_config",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=566,
+ serialized_end=702,
+)
+
+
+_BATCHGETASSETSHISTORYREQUEST = _descriptor.Descriptor(
+ name="BatchGetAssetsHistoryRequest",
+ full_name="google.cloud.asset.v1p2beta1.BatchGetAssetsHistoryRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="parent",
+ full_name="google.cloud.asset.v1p2beta1.BatchGetAssetsHistoryRequest.parent",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="asset_names",
+ full_name="google.cloud.asset.v1p2beta1.BatchGetAssetsHistoryRequest.asset_names",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="content_type",
+ full_name="google.cloud.asset.v1p2beta1.BatchGetAssetsHistoryRequest.content_type",
+ index=2,
+ number=3,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="read_time_window",
+ full_name="google.cloud.asset.v1p2beta1.BatchGetAssetsHistoryRequest.read_time_window",
+ index=3,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=705,
+ serialized_end=905,
+)
+
+
+_BATCHGETASSETSHISTORYRESPONSE = _descriptor.Descriptor(
+ name="BatchGetAssetsHistoryResponse",
+ full_name="google.cloud.asset.v1p2beta1.BatchGetAssetsHistoryResponse",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="assets",
+ full_name="google.cloud.asset.v1p2beta1.BatchGetAssetsHistoryResponse.assets",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=907,
+ serialized_end=999,
+)
+
+
+_CREATEFEEDREQUEST = _descriptor.Descriptor(
+ name="CreateFeedRequest",
+ full_name="google.cloud.asset.v1p2beta1.CreateFeedRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="parent",
+ full_name="google.cloud.asset.v1p2beta1.CreateFeedRequest.parent",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="feed_id",
+ full_name="google.cloud.asset.v1p2beta1.CreateFeedRequest.feed_id",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="feed",
+ full_name="google.cloud.asset.v1p2beta1.CreateFeedRequest.feed",
+ index=2,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1001,
+ serialized_end=1103,
+)
+
+
+_GETFEEDREQUEST = _descriptor.Descriptor(
+ name="GetFeedRequest",
+ full_name="google.cloud.asset.v1p2beta1.GetFeedRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.asset.v1p2beta1.GetFeedRequest.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1105,
+ serialized_end=1135,
+)
+
+
+_LISTFEEDSREQUEST = _descriptor.Descriptor(
+ name="ListFeedsRequest",
+ full_name="google.cloud.asset.v1p2beta1.ListFeedsRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="parent",
+ full_name="google.cloud.asset.v1p2beta1.ListFeedsRequest.parent",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1137,
+ serialized_end=1171,
+)
+
+
+_LISTFEEDSRESPONSE = _descriptor.Descriptor(
+ name="ListFeedsResponse",
+ full_name="google.cloud.asset.v1p2beta1.ListFeedsResponse",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="feeds",
+ full_name="google.cloud.asset.v1p2beta1.ListFeedsResponse.feeds",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1173,
+ serialized_end=1243,
+)
+
+
+_UPDATEFEEDREQUEST = _descriptor.Descriptor(
+ name="UpdateFeedRequest",
+ full_name="google.cloud.asset.v1p2beta1.UpdateFeedRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="feed",
+ full_name="google.cloud.asset.v1p2beta1.UpdateFeedRequest.feed",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="update_mask",
+ full_name="google.cloud.asset.v1p2beta1.UpdateFeedRequest.update_mask",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1245,
+ serialized_end=1363,
+)
+
+
+_DELETEFEEDREQUEST = _descriptor.Descriptor(
+ name="DeleteFeedRequest",
+ full_name="google.cloud.asset.v1p2beta1.DeleteFeedRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.asset.v1p2beta1.DeleteFeedRequest.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1365,
+ serialized_end=1398,
+)
+
+
+_OUTPUTCONFIG = _descriptor.Descriptor(
+ name="OutputConfig",
+ full_name="google.cloud.asset.v1p2beta1.OutputConfig",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="gcs_destination",
+ full_name="google.cloud.asset.v1p2beta1.OutputConfig.gcs_destination",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="bigquery_destination",
+ full_name="google.cloud.asset.v1p2beta1.OutputConfig.bigquery_destination",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="destination",
+ full_name="google.cloud.asset.v1p2beta1.OutputConfig.destination",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=1401,
+ serialized_end=1586,
+)
+
+
+_GCSDESTINATION = _descriptor.Descriptor(
+ name="GcsDestination",
+ full_name="google.cloud.asset.v1p2beta1.GcsDestination",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="uri",
+ full_name="google.cloud.asset.v1p2beta1.GcsDestination.uri",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="uri_prefix",
+ full_name="google.cloud.asset.v1p2beta1.GcsDestination.uri_prefix",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="object_uri",
+ full_name="google.cloud.asset.v1p2beta1.GcsDestination.object_uri",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=1588,
+ serialized_end=1655,
+)
+
+
+_BIGQUERYDESTINATION = _descriptor.Descriptor(
+ name="BigQueryDestination",
+ full_name="google.cloud.asset.v1p2beta1.BigQueryDestination",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="dataset",
+ full_name="google.cloud.asset.v1p2beta1.BigQueryDestination.dataset",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="table",
+ full_name="google.cloud.asset.v1p2beta1.BigQueryDestination.table",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="force",
+ full_name="google.cloud.asset.v1p2beta1.BigQueryDestination.force",
+ index=2,
+ number=3,
+ type=8,
+ cpp_type=7,
+ label=1,
+ has_default_value=False,
+ default_value=False,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1657,
+ serialized_end=1725,
+)
+
+
+_PUBSUBDESTINATION = _descriptor.Descriptor(
+ name="PubsubDestination",
+ full_name="google.cloud.asset.v1p2beta1.PubsubDestination",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="topic",
+ full_name="google.cloud.asset.v1p2beta1.PubsubDestination.topic",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1727,
+ serialized_end=1761,
+)
+
+
+_FEEDOUTPUTCONFIG = _descriptor.Descriptor(
+ name="FeedOutputConfig",
+ full_name="google.cloud.asset.v1p2beta1.FeedOutputConfig",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="pubsub_destination",
+ full_name="google.cloud.asset.v1p2beta1.FeedOutputConfig.pubsub_destination",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="destination",
+ full_name="google.cloud.asset.v1p2beta1.FeedOutputConfig.destination",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=1763,
+ serialized_end=1875,
+)
+
+
+_FEED = _descriptor.Descriptor(
+ name="Feed",
+ full_name="google.cloud.asset.v1p2beta1.Feed",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.asset.v1p2beta1.Feed.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="asset_names",
+ full_name="google.cloud.asset.v1p2beta1.Feed.asset_names",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="asset_types",
+ full_name="google.cloud.asset.v1p2beta1.Feed.asset_types",
+ index=2,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="content_type",
+ full_name="google.cloud.asset.v1p2beta1.Feed.content_type",
+ index=3,
+ number=4,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="feed_output_config",
+ full_name="google.cloud.asset.v1p2beta1.Feed.feed_output_config",
+ index=4,
+ number=5,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1878,
+ serialized_end=2081,
+)
+
+_EXPORTASSETSREQUEST.fields_by_name[
+ "read_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_EXPORTASSETSREQUEST.fields_by_name["content_type"].enum_type = _CONTENTTYPE
+_EXPORTASSETSREQUEST.fields_by_name["output_config"].message_type = _OUTPUTCONFIG
+_EXPORTASSETSRESPONSE.fields_by_name[
+ "read_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_EXPORTASSETSRESPONSE.fields_by_name["output_config"].message_type = _OUTPUTCONFIG
+_BATCHGETASSETSHISTORYREQUEST.fields_by_name["content_type"].enum_type = _CONTENTTYPE
+_BATCHGETASSETSHISTORYREQUEST.fields_by_name[
+ "read_time_window"
+].message_type = (
+ google_dot_cloud_dot_asset__v1p2beta1_dot_proto_dot_assets__pb2._TIMEWINDOW
+)
+_BATCHGETASSETSHISTORYRESPONSE.fields_by_name[
+ "assets"
+].message_type = (
+ google_dot_cloud_dot_asset__v1p2beta1_dot_proto_dot_assets__pb2._TEMPORALASSET
+)
+_CREATEFEEDREQUEST.fields_by_name["feed"].message_type = _FEED
+_LISTFEEDSRESPONSE.fields_by_name["feeds"].message_type = _FEED
+_UPDATEFEEDREQUEST.fields_by_name["feed"].message_type = _FEED
+_UPDATEFEEDREQUEST.fields_by_name[
+ "update_mask"
+].message_type = google_dot_protobuf_dot_field__mask__pb2._FIELDMASK
+_OUTPUTCONFIG.fields_by_name["gcs_destination"].message_type = _GCSDESTINATION
+_OUTPUTCONFIG.fields_by_name["bigquery_destination"].message_type = _BIGQUERYDESTINATION
+_OUTPUTCONFIG.oneofs_by_name["destination"].fields.append(
+ _OUTPUTCONFIG.fields_by_name["gcs_destination"]
+)
+_OUTPUTCONFIG.fields_by_name[
+ "gcs_destination"
+].containing_oneof = _OUTPUTCONFIG.oneofs_by_name["destination"]
+_OUTPUTCONFIG.oneofs_by_name["destination"].fields.append(
+ _OUTPUTCONFIG.fields_by_name["bigquery_destination"]
+)
+_OUTPUTCONFIG.fields_by_name[
+ "bigquery_destination"
+].containing_oneof = _OUTPUTCONFIG.oneofs_by_name["destination"]
+_GCSDESTINATION.oneofs_by_name["object_uri"].fields.append(
+ _GCSDESTINATION.fields_by_name["uri"]
+)
+_GCSDESTINATION.fields_by_name["uri"].containing_oneof = _GCSDESTINATION.oneofs_by_name[
+ "object_uri"
+]
+_GCSDESTINATION.oneofs_by_name["object_uri"].fields.append(
+ _GCSDESTINATION.fields_by_name["uri_prefix"]
+)
+_GCSDESTINATION.fields_by_name[
+ "uri_prefix"
+].containing_oneof = _GCSDESTINATION.oneofs_by_name["object_uri"]
+_FEEDOUTPUTCONFIG.fields_by_name["pubsub_destination"].message_type = _PUBSUBDESTINATION
+_FEEDOUTPUTCONFIG.oneofs_by_name["destination"].fields.append(
+ _FEEDOUTPUTCONFIG.fields_by_name["pubsub_destination"]
+)
+_FEEDOUTPUTCONFIG.fields_by_name[
+ "pubsub_destination"
+].containing_oneof = _FEEDOUTPUTCONFIG.oneofs_by_name["destination"]
+_FEED.fields_by_name["content_type"].enum_type = _CONTENTTYPE
+_FEED.fields_by_name["feed_output_config"].message_type = _FEEDOUTPUTCONFIG
+DESCRIPTOR.message_types_by_name["ExportAssetsRequest"] = _EXPORTASSETSREQUEST
+DESCRIPTOR.message_types_by_name["ExportAssetsResponse"] = _EXPORTASSETSRESPONSE
+DESCRIPTOR.message_types_by_name[
+ "BatchGetAssetsHistoryRequest"
+] = _BATCHGETASSETSHISTORYREQUEST
+DESCRIPTOR.message_types_by_name[
+ "BatchGetAssetsHistoryResponse"
+] = _BATCHGETASSETSHISTORYRESPONSE
+DESCRIPTOR.message_types_by_name["CreateFeedRequest"] = _CREATEFEEDREQUEST
+DESCRIPTOR.message_types_by_name["GetFeedRequest"] = _GETFEEDREQUEST
+DESCRIPTOR.message_types_by_name["ListFeedsRequest"] = _LISTFEEDSREQUEST
+DESCRIPTOR.message_types_by_name["ListFeedsResponse"] = _LISTFEEDSRESPONSE
+DESCRIPTOR.message_types_by_name["UpdateFeedRequest"] = _UPDATEFEEDREQUEST
+DESCRIPTOR.message_types_by_name["DeleteFeedRequest"] = _DELETEFEEDREQUEST
+DESCRIPTOR.message_types_by_name["OutputConfig"] = _OUTPUTCONFIG
+DESCRIPTOR.message_types_by_name["GcsDestination"] = _GCSDESTINATION
+DESCRIPTOR.message_types_by_name["BigQueryDestination"] = _BIGQUERYDESTINATION
+DESCRIPTOR.message_types_by_name["PubsubDestination"] = _PUBSUBDESTINATION
+DESCRIPTOR.message_types_by_name["FeedOutputConfig"] = _FEEDOUTPUTCONFIG
+DESCRIPTOR.message_types_by_name["Feed"] = _FEED
+DESCRIPTOR.enum_types_by_name["ContentType"] = _CONTENTTYPE
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+ExportAssetsRequest = _reflection.GeneratedProtocolMessageType(
+ "ExportAssetsRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_EXPORTASSETSREQUEST,
+ __module__="google.cloud.asset_v1p2beta1.proto.asset_service_pb2",
+ __doc__="""Export asset request.
+
+
+ Attributes:
+ parent:
+ Required. The relative name of the root asset. This can only
+ be an organization number (such as "organizations/123"), a
+ project ID (such as "projects/my-project-id"), or a project
+ number (such as "projects/12345").
+ read_time:
+ Timestamp to take an asset snapshot. This can only be set to a
+ timestamp between 2018-10-02 UTC (inclusive) and the current
+ time. If not specified, the current time will be used. Due to
+ delays in resource data collection and indexing, there is a
+ volatile window during which running the same query may get
+ different results.
+ asset_types:
+ A list of asset types of which to take a snapshot for. For
+ example: "compute.googleapis.com/Disk". If specified, only
+ matching assets will be returned. See `Introduction to Cloud
+ Asset Inventory `__ for all
+ supported asset types.
+ content_type:
+ Asset content type. If not specified, no content but the asset
+ name will be returned.
+ output_config:
+ Required. Output configuration indicating where the results
+ will be output to. All results will be in newline delimited
+ JSON format.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1p2beta1.ExportAssetsRequest)
+ ),
+)
+_sym_db.RegisterMessage(ExportAssetsRequest)
+
+ExportAssetsResponse = _reflection.GeneratedProtocolMessageType(
+ "ExportAssetsResponse",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_EXPORTASSETSRESPONSE,
+ __module__="google.cloud.asset_v1p2beta1.proto.asset_service_pb2",
+ __doc__="""The export asset response. This message is returned by the
+ [google.longrunning.Operations.GetOperation][google.longrunning.Operations.GetOperation]
+ method in the returned
+ [google.longrunning.Operation.response][google.longrunning.Operation.response]
+ field.
+
+
+ Attributes:
+ read_time:
+ Time the snapshot was taken.
+ output_config:
+ Output configuration indicating where the results were output
+ to. All results are in JSON format.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1p2beta1.ExportAssetsResponse)
+ ),
+)
+_sym_db.RegisterMessage(ExportAssetsResponse)
+
+BatchGetAssetsHistoryRequest = _reflection.GeneratedProtocolMessageType(
+ "BatchGetAssetsHistoryRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_BATCHGETASSETSHISTORYREQUEST,
+ __module__="google.cloud.asset_v1p2beta1.proto.asset_service_pb2",
+ __doc__="""Batch get assets history request.
+
+
+ Attributes:
+ parent:
+ Required. The relative name of the root asset. It can only be
+ an organization number (such as "organizations/123"), a
+ project ID (such as "projects/my-project-id")", or a project
+ number (such as "projects/12345").
+ asset_names:
+ A list of the full names of the assets. For example: ``//compu
+ te.googleapis.com/projects/my_project_123/zones/zone1/instance
+ s/instance1``. See `Resource Names `__ and `Resource
+ Name Format `__
+ for more info. The request becomes a no-op if the asset name
+ list is empty, and the max size of the asset name list is 100
+ in one request.
+ content_type:
+ Required. The content type.
+ read_time_window:
+ Optional. The time window for the asset history. Both
+ start\_time and end\_time are optional and if set, it must be
+ after 2018-10-02 UTC. If end\_time is not set, it is default
+ to current timestamp. If start\_time is not set, the snapshot
+ of the assets at end\_time will be returned. The returned
+ results contain all temporal assets whose time window overlap
+ with read\_time\_window.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1p2beta1.BatchGetAssetsHistoryRequest)
+ ),
+)
+_sym_db.RegisterMessage(BatchGetAssetsHistoryRequest)
+
+BatchGetAssetsHistoryResponse = _reflection.GeneratedProtocolMessageType(
+ "BatchGetAssetsHistoryResponse",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_BATCHGETASSETSHISTORYRESPONSE,
+ __module__="google.cloud.asset_v1p2beta1.proto.asset_service_pb2",
+ __doc__="""Batch get assets history response.
+
+
+ Attributes:
+ assets:
+ A list of assets with valid time windows.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1p2beta1.BatchGetAssetsHistoryResponse)
+ ),
+)
+_sym_db.RegisterMessage(BatchGetAssetsHistoryResponse)
+
+CreateFeedRequest = _reflection.GeneratedProtocolMessageType(
+ "CreateFeedRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_CREATEFEEDREQUEST,
+ __module__="google.cloud.asset_v1p2beta1.proto.asset_service_pb2",
+ __doc__="""Create asset feed request.
+
+
+ Attributes:
+ parent:
+ Required. The name of the project/folder/organization where
+ this feed should be created in. It can only be an organization
+ number (such as "organizations/123"), a folder number (such as
+ "folders/123"), a project ID (such as "projects/my-project-
+ id")", or a project number (such as "projects/12345").
+ feed_id:
+ Required. This is the client-assigned asset feed identifier
+ and it needs to be unique under a specific parent
+ project/folder/organization.
+ feed:
+ The feed details. The field ``name`` must be empty and it will
+ be generated in the format of:
+ projects/project\_number/feeds/feed\_id
+ folders/folder\_number/feeds/feed\_id
+ organizations/organization\_number/feeds/feed\_id
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1p2beta1.CreateFeedRequest)
+ ),
+)
+_sym_db.RegisterMessage(CreateFeedRequest)
+
+GetFeedRequest = _reflection.GeneratedProtocolMessageType(
+ "GetFeedRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_GETFEEDREQUEST,
+ __module__="google.cloud.asset_v1p2beta1.proto.asset_service_pb2",
+ __doc__="""Get asset feed request.
+
+
+ Attributes:
+ name:
+ The name of the Feed and it must be in the format of:
+ projects/project\_number/feeds/feed\_id
+ folders/folder\_number/feeds/feed\_id
+ organizations/organization\_number/feeds/feed\_id
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1p2beta1.GetFeedRequest)
+ ),
+)
+_sym_db.RegisterMessage(GetFeedRequest)
+
+ListFeedsRequest = _reflection.GeneratedProtocolMessageType(
+ "ListFeedsRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_LISTFEEDSREQUEST,
+ __module__="google.cloud.asset_v1p2beta1.proto.asset_service_pb2",
+ __doc__="""List asset feeds request.
+
+
+ Attributes:
+ parent:
+ Required. The parent project/folder/organization whose feeds
+ are to be listed. It can only be using
+ project/folder/organization number (such as "folders/12345")",
+ or a project ID (such as "projects/my-project-id").
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1p2beta1.ListFeedsRequest)
+ ),
+)
+_sym_db.RegisterMessage(ListFeedsRequest)
+
+ListFeedsResponse = _reflection.GeneratedProtocolMessageType(
+ "ListFeedsResponse",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_LISTFEEDSRESPONSE,
+ __module__="google.cloud.asset_v1p2beta1.proto.asset_service_pb2",
+ __doc__="""List asset feeds response.
+
+
+ Attributes:
+ feeds:
+ A list of feeds.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1p2beta1.ListFeedsResponse)
+ ),
+)
+_sym_db.RegisterMessage(ListFeedsResponse)
+
+UpdateFeedRequest = _reflection.GeneratedProtocolMessageType(
+ "UpdateFeedRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_UPDATEFEEDREQUEST,
+ __module__="google.cloud.asset_v1p2beta1.proto.asset_service_pb2",
+ __doc__="""Update asset feed request.
+
+
+ Attributes:
+ feed:
+ The new values of feed details. It must match an existing feed
+ and the field ``name`` must be in the format of:
+ projects/project\_number/feeds/feed\_id or
+ folders/folder\_number/feeds/feed\_id or
+ organizations/organization\_number/feeds/feed\_id.
+ update_mask:
+ Only updates the ``feed`` fields indicated by this mask. The
+ field mask must not be empty, and it must not contain fields
+ that are immutable or only set by the server.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1p2beta1.UpdateFeedRequest)
+ ),
+)
+_sym_db.RegisterMessage(UpdateFeedRequest)
+
+DeleteFeedRequest = _reflection.GeneratedProtocolMessageType(
+ "DeleteFeedRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_DELETEFEEDREQUEST,
+ __module__="google.cloud.asset_v1p2beta1.proto.asset_service_pb2",
+ __doc__="""Delete asset feed request.
+
+
+ Attributes:
+ name:
+ The name of the feed and it must be in the format of:
+ projects/project\_number/feeds/feed\_id
+ folders/folder\_number/feeds/feed\_id
+ organizations/organization\_number/feeds/feed\_id
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1p2beta1.DeleteFeedRequest)
+ ),
+)
+_sym_db.RegisterMessage(DeleteFeedRequest)
+
+OutputConfig = _reflection.GeneratedProtocolMessageType(
+ "OutputConfig",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_OUTPUTCONFIG,
+ __module__="google.cloud.asset_v1p2beta1.proto.asset_service_pb2",
+ __doc__="""Output configuration for export assets destination.
+
+
+ Attributes:
+ destination:
+ Asset export destination.
+ gcs_destination:
+ Destination on Cloud Storage.
+ bigquery_destination:
+ Destination on Bigquery. The output table stores the fields in
+ asset proto as columns in BigQuery. The resource/iam\_policy
+ field is converted to a record with each field to a column,
+ except metadata to a single JSON string.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1p2beta1.OutputConfig)
+ ),
+)
+_sym_db.RegisterMessage(OutputConfig)
+
+GcsDestination = _reflection.GeneratedProtocolMessageType(
+ "GcsDestination",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_GCSDESTINATION,
+ __module__="google.cloud.asset_v1p2beta1.proto.asset_service_pb2",
+ __doc__="""A Cloud Storage location.
+
+
+ Attributes:
+ object_uri:
+ Required.
+ uri:
+ The uri of the Cloud Storage object. It's the same uri that is
+ used by gsutil. For example: "gs://bucket\_name/object\_name".
+ See `Viewing and Editing Object Metadata
+ `__ for more information.
+ uri_prefix:
+ The uri prefix of all generated Cloud Storage objects. For
+ example: "gs://bucket\_name/object\_name\_prefix". Each object
+ uri is in format: "gs://bucket\_name/object\_name\_prefix//
+ and only contains assets for that type. starts from 0. For
+ example: "gs://bucket\_name/object\_name\_prefix/compute.googl
+ eapis.com/Disk/0" is the first shard of output objects
+ containing all compute.googleapis.com/Disk assets. An
+ INVALID\_ARGUMENT error will be returned if file with the same
+ name "gs://bucket\_name/object\_name\_prefix" already exists.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1p2beta1.GcsDestination)
+ ),
+)
+_sym_db.RegisterMessage(GcsDestination)
+
+BigQueryDestination = _reflection.GeneratedProtocolMessageType(
+ "BigQueryDestination",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_BIGQUERYDESTINATION,
+ __module__="google.cloud.asset_v1p2beta1.proto.asset_service_pb2",
+ __doc__="""A Bigquery destination.
+
+
+ Attributes:
+ dataset:
+ Required. The BigQuery dataset in format
+ "projects/projectId/datasets/datasetId", to which the snapshot
+ result should be exported. If this dataset does not exist, the
+ export call returns an error.
+ table:
+ Required. The BigQuery table to which the snapshot result
+ should be written. If this table does not exist, a new table
+ with the given name will be created.
+ force:
+ If the destination table already exists and this flag is
+ ``TRUE``, the table will be overwritten by the contents of
+ assets snapshot. If the flag is not set and the destination
+ table already exists, the export call returns an error.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1p2beta1.BigQueryDestination)
+ ),
+)
+_sym_db.RegisterMessage(BigQueryDestination)
+
+PubsubDestination = _reflection.GeneratedProtocolMessageType(
+ "PubsubDestination",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_PUBSUBDESTINATION,
+ __module__="google.cloud.asset_v1p2beta1.proto.asset_service_pb2",
+ __doc__="""A Cloud Pubsub destination.
+
+
+ Attributes:
+ topic:
+ The name of the Cloud Pub/Sub topic to publish to. For
+ example: ``projects/PROJECT_ID/topics/TOPIC_ID``.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1p2beta1.PubsubDestination)
+ ),
+)
+_sym_db.RegisterMessage(PubsubDestination)
+
+FeedOutputConfig = _reflection.GeneratedProtocolMessageType(
+ "FeedOutputConfig",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_FEEDOUTPUTCONFIG,
+ __module__="google.cloud.asset_v1p2beta1.proto.asset_service_pb2",
+ __doc__="""Output configuration for asset feed destination.
+
+
+ Attributes:
+ destination:
+ Asset feed destination.
+ pubsub_destination:
+ Destination on Cloud Pubsub.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1p2beta1.FeedOutputConfig)
+ ),
+)
+_sym_db.RegisterMessage(FeedOutputConfig)
+
+Feed = _reflection.GeneratedProtocolMessageType(
+ "Feed",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_FEED,
+ __module__="google.cloud.asset_v1p2beta1.proto.asset_service_pb2",
+ __doc__="""An asset feed used to export asset updates to a destinations. An asset
+ feed filter controls what updates are exported. The asset feed must be
+ created within a project, organization, or folder. Supported
+ destinations are: Cloud Pub/Sub topics.
+
+
+ Attributes:
+ name:
+ Required. The format will be
+ projects/{project\_number}/feeds/{client-
+ assigned\_feed\_identifier} or
+ folders/{folder\_number}/feeds/{client-
+ assigned\_feed\_identifier} or
+ organizations/{organization\_number}/feeds/{client-
+ assigned\_feed\_identifier} The client-assigned feed
+ identifier must be unique within the parent
+ project/folder/organization.
+ asset_names:
+ A list of the full names of the assets to receive updates. You
+ must specify either or both of asset\_names and asset\_types.
+ Only asset updates matching specified asset\_names and
+ asset\_types are exported to the feed. For example: ``//comput
+ e.googleapis.com/projects/my_project_123/zones/zone1/instances
+ /instance1``. See `Resource Names `__ for more info.
+ asset_types:
+ A list of types of the assets to receive updates. You must
+ specify either or both of asset\_names and asset\_types. Only
+ asset updates matching specified asset\_names and asset\_types
+ are exported to the feed. For example:
+ "compute.googleapis.com/Disk" See `Introduction to Cloud Asset
+ Inventory `__ for all
+ supported asset types.
+ content_type:
+ Asset content type. If not specified, no content but the asset
+ name and type will be returned.
+ feed_output_config:
+ Required. Feed output configuration defining where the asset
+ updates are published to.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1p2beta1.Feed)
+ ),
+)
+_sym_db.RegisterMessage(Feed)
+
+
+DESCRIPTOR._options = None
+
+_ASSETSERVICE = _descriptor.ServiceDescriptor(
+ name="AssetService",
+ full_name="google.cloud.asset.v1p2beta1.AssetService",
+ file=DESCRIPTOR,
+ index=0,
+ serialized_options=_b(
+ "\312A\031cloudasset.googleapis.com\322A.https://www.googleapis.com/auth/cloud-platform"
+ ),
+ serialized_start=2216,
+ serialized_end=3357,
+ methods=[
+ _descriptor.MethodDescriptor(
+ name="ExportAssets",
+ full_name="google.cloud.asset.v1p2beta1.AssetService.ExportAssets",
+ index=0,
+ containing_service=None,
+ input_type=_EXPORTASSETSREQUEST,
+ output_type=google_dot_longrunning_dot_operations__pb2._OPERATION,
+ serialized_options=_b(
+ '\202\323\344\223\002)"$/v1p2beta1/{parent=*/*}:exportAssets:\001*'
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="BatchGetAssetsHistory",
+ full_name="google.cloud.asset.v1p2beta1.AssetService.BatchGetAssetsHistory",
+ index=1,
+ containing_service=None,
+ input_type=_BATCHGETASSETSHISTORYREQUEST,
+ output_type=_BATCHGETASSETSHISTORYRESPONSE,
+ serialized_options=_b(
+ "\202\323\344\223\002/\022-/v1p2beta1/{parent=*/*}:batchGetAssetsHistory"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="CreateFeed",
+ full_name="google.cloud.asset.v1p2beta1.AssetService.CreateFeed",
+ index=2,
+ containing_service=None,
+ input_type=_CREATEFEEDREQUEST,
+ output_type=_FEED,
+ serialized_options=_b(
+ '\202\323\344\223\002""\035/v1p2beta1/{parent=*/*}/feeds:\001*'
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="GetFeed",
+ full_name="google.cloud.asset.v1p2beta1.AssetService.GetFeed",
+ index=3,
+ containing_service=None,
+ input_type=_GETFEEDREQUEST,
+ output_type=_FEED,
+ serialized_options=_b(
+ "\202\323\344\223\002\037\022\035/v1p2beta1/{name=*/*/feeds/*}"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="ListFeeds",
+ full_name="google.cloud.asset.v1p2beta1.AssetService.ListFeeds",
+ index=4,
+ containing_service=None,
+ input_type=_LISTFEEDSREQUEST,
+ output_type=_LISTFEEDSRESPONSE,
+ serialized_options=_b(
+ "\202\323\344\223\002\037\022\035/v1p2beta1/{parent=*/*}/feeds"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="UpdateFeed",
+ full_name="google.cloud.asset.v1p2beta1.AssetService.UpdateFeed",
+ index=5,
+ containing_service=None,
+ input_type=_UPDATEFEEDREQUEST,
+ output_type=_FEED,
+ serialized_options=_b(
+ "\202\323\344\223\002'2\"/v1p2beta1/{feed.name=*/*/feeds/*}:\001*"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="DeleteFeed",
+ full_name="google.cloud.asset.v1p2beta1.AssetService.DeleteFeed",
+ index=6,
+ containing_service=None,
+ input_type=_DELETEFEEDREQUEST,
+ output_type=google_dot_protobuf_dot_empty__pb2._EMPTY,
+ serialized_options=_b(
+ "\202\323\344\223\002\037*\035/v1p2beta1/{name=*/*/feeds/*}"
+ ),
+ ),
+ ],
+)
+_sym_db.RegisterServiceDescriptor(_ASSETSERVICE)
+
+DESCRIPTOR.services_by_name["AssetService"] = _ASSETSERVICE
+
+# @@protoc_insertion_point(module_scope)
diff --git a/asset/google/cloud/asset_v1p2beta1/proto/asset_service_pb2_grpc.py b/asset/google/cloud/asset_v1p2beta1/proto/asset_service_pb2_grpc.py
new file mode 100644
index 000000000000..05ec4e44be0f
--- /dev/null
+++ b/asset/google/cloud/asset_v1p2beta1/proto/asset_service_pb2_grpc.py
@@ -0,0 +1,163 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
+
+from google.cloud.asset_v1p2beta1.proto import (
+ asset_service_pb2 as google_dot_cloud_dot_asset__v1p2beta1_dot_proto_dot_asset__service__pb2,
+)
+from google.longrunning import (
+ operations_pb2 as google_dot_longrunning_dot_operations__pb2,
+)
+from google.protobuf import empty_pb2 as google_dot_protobuf_dot_empty__pb2
+
+
+class AssetServiceStub(object):
+ """Asset service definition.
+ """
+
+ def __init__(self, channel):
+ """Constructor.
+
+ Args:
+ channel: A grpc.Channel.
+ """
+ self.ExportAssets = channel.unary_unary(
+ "/google.cloud.asset.v1p2beta1.AssetService/ExportAssets",
+ request_serializer=google_dot_cloud_dot_asset__v1p2beta1_dot_proto_dot_asset__service__pb2.ExportAssetsRequest.SerializeToString,
+ response_deserializer=google_dot_longrunning_dot_operations__pb2.Operation.FromString,
+ )
+ self.BatchGetAssetsHistory = channel.unary_unary(
+ "/google.cloud.asset.v1p2beta1.AssetService/BatchGetAssetsHistory",
+ request_serializer=google_dot_cloud_dot_asset__v1p2beta1_dot_proto_dot_asset__service__pb2.BatchGetAssetsHistoryRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_asset__v1p2beta1_dot_proto_dot_asset__service__pb2.BatchGetAssetsHistoryResponse.FromString,
+ )
+ self.CreateFeed = channel.unary_unary(
+ "/google.cloud.asset.v1p2beta1.AssetService/CreateFeed",
+ request_serializer=google_dot_cloud_dot_asset__v1p2beta1_dot_proto_dot_asset__service__pb2.CreateFeedRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_asset__v1p2beta1_dot_proto_dot_asset__service__pb2.Feed.FromString,
+ )
+ self.GetFeed = channel.unary_unary(
+ "/google.cloud.asset.v1p2beta1.AssetService/GetFeed",
+ request_serializer=google_dot_cloud_dot_asset__v1p2beta1_dot_proto_dot_asset__service__pb2.GetFeedRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_asset__v1p2beta1_dot_proto_dot_asset__service__pb2.Feed.FromString,
+ )
+ self.ListFeeds = channel.unary_unary(
+ "/google.cloud.asset.v1p2beta1.AssetService/ListFeeds",
+ request_serializer=google_dot_cloud_dot_asset__v1p2beta1_dot_proto_dot_asset__service__pb2.ListFeedsRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_asset__v1p2beta1_dot_proto_dot_asset__service__pb2.ListFeedsResponse.FromString,
+ )
+ self.UpdateFeed = channel.unary_unary(
+ "/google.cloud.asset.v1p2beta1.AssetService/UpdateFeed",
+ request_serializer=google_dot_cloud_dot_asset__v1p2beta1_dot_proto_dot_asset__service__pb2.UpdateFeedRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_asset__v1p2beta1_dot_proto_dot_asset__service__pb2.Feed.FromString,
+ )
+ self.DeleteFeed = channel.unary_unary(
+ "/google.cloud.asset.v1p2beta1.AssetService/DeleteFeed",
+ request_serializer=google_dot_cloud_dot_asset__v1p2beta1_dot_proto_dot_asset__service__pb2.DeleteFeedRequest.SerializeToString,
+ response_deserializer=google_dot_protobuf_dot_empty__pb2.Empty.FromString,
+ )
+
+
+class AssetServiceServicer(object):
+ """Asset service definition.
+ """
+
+ def ExportAssets(self, request, context):
+ """Exports assets with time and resource types to a given Cloud Storage
+ location. The output format is newline-delimited JSON.
+ This API implements the [google.longrunning.Operation][google.longrunning.Operation] API allowing you
+ to keep track of the export.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def BatchGetAssetsHistory(self, request, context):
+ """Batch gets the update history of assets that overlap a time window.
+ For RESOURCE content, this API outputs history with asset in both
+ non-delete or deleted status.
+ For IAM_POLICY content, this API outputs history when the asset and its
+ attached IAM POLICY both exist. This can create gaps in the output history.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def CreateFeed(self, request, context):
+ """Creates a feed in a parent project/folder/organization to listen to its
+ asset updates.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def GetFeed(self, request, context):
+ """Gets details about an asset feed.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def ListFeeds(self, request, context):
+ """Lists all asset feeds in a parent project/folder/organization.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def UpdateFeed(self, request, context):
+ """Updates an asset feed configuration.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def DeleteFeed(self, request, context):
+ """Deletes an asset feed.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+
+def add_AssetServiceServicer_to_server(servicer, server):
+ rpc_method_handlers = {
+ "ExportAssets": grpc.unary_unary_rpc_method_handler(
+ servicer.ExportAssets,
+ request_deserializer=google_dot_cloud_dot_asset__v1p2beta1_dot_proto_dot_asset__service__pb2.ExportAssetsRequest.FromString,
+ response_serializer=google_dot_longrunning_dot_operations__pb2.Operation.SerializeToString,
+ ),
+ "BatchGetAssetsHistory": grpc.unary_unary_rpc_method_handler(
+ servicer.BatchGetAssetsHistory,
+ request_deserializer=google_dot_cloud_dot_asset__v1p2beta1_dot_proto_dot_asset__service__pb2.BatchGetAssetsHistoryRequest.FromString,
+ response_serializer=google_dot_cloud_dot_asset__v1p2beta1_dot_proto_dot_asset__service__pb2.BatchGetAssetsHistoryResponse.SerializeToString,
+ ),
+ "CreateFeed": grpc.unary_unary_rpc_method_handler(
+ servicer.CreateFeed,
+ request_deserializer=google_dot_cloud_dot_asset__v1p2beta1_dot_proto_dot_asset__service__pb2.CreateFeedRequest.FromString,
+ response_serializer=google_dot_cloud_dot_asset__v1p2beta1_dot_proto_dot_asset__service__pb2.Feed.SerializeToString,
+ ),
+ "GetFeed": grpc.unary_unary_rpc_method_handler(
+ servicer.GetFeed,
+ request_deserializer=google_dot_cloud_dot_asset__v1p2beta1_dot_proto_dot_asset__service__pb2.GetFeedRequest.FromString,
+ response_serializer=google_dot_cloud_dot_asset__v1p2beta1_dot_proto_dot_asset__service__pb2.Feed.SerializeToString,
+ ),
+ "ListFeeds": grpc.unary_unary_rpc_method_handler(
+ servicer.ListFeeds,
+ request_deserializer=google_dot_cloud_dot_asset__v1p2beta1_dot_proto_dot_asset__service__pb2.ListFeedsRequest.FromString,
+ response_serializer=google_dot_cloud_dot_asset__v1p2beta1_dot_proto_dot_asset__service__pb2.ListFeedsResponse.SerializeToString,
+ ),
+ "UpdateFeed": grpc.unary_unary_rpc_method_handler(
+ servicer.UpdateFeed,
+ request_deserializer=google_dot_cloud_dot_asset__v1p2beta1_dot_proto_dot_asset__service__pb2.UpdateFeedRequest.FromString,
+ response_serializer=google_dot_cloud_dot_asset__v1p2beta1_dot_proto_dot_asset__service__pb2.Feed.SerializeToString,
+ ),
+ "DeleteFeed": grpc.unary_unary_rpc_method_handler(
+ servicer.DeleteFeed,
+ request_deserializer=google_dot_cloud_dot_asset__v1p2beta1_dot_proto_dot_asset__service__pb2.DeleteFeedRequest.FromString,
+ response_serializer=google_dot_protobuf_dot_empty__pb2.Empty.SerializeToString,
+ ),
+ }
+ generic_handler = grpc.method_handlers_generic_handler(
+ "google.cloud.asset.v1p2beta1.AssetService", rpc_method_handlers
+ )
+ server.add_generic_rpc_handlers((generic_handler,))
diff --git a/asset/google/cloud/asset_v1p2beta1/proto/assets.proto b/asset/google/cloud/asset_v1p2beta1/proto/assets.proto
new file mode 100644
index 000000000000..a9cda898a357
--- /dev/null
+++ b/asset/google/cloud/asset_v1p2beta1/proto/assets.proto
@@ -0,0 +1,135 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.asset.v1p2beta1;
+
+import "google/api/annotations.proto";
+import "google/iam/v1/policy.proto";
+import "google/protobuf/any.proto";
+import "google/protobuf/struct.proto";
+import "google/protobuf/timestamp.proto";
+
+option cc_enable_arenas = true;
+option csharp_namespace = "Google.Cloud.Asset.v1p2beta1";
+option go_package = "google.golang.org/genproto/googleapis/cloud/asset/v1p2beta1;asset";
+option java_multiple_files = true;
+option java_outer_classname = "AssetProto";
+option java_package = "com.google.cloud.asset.v1p2beta1";
+option php_namespace = "Google\\Cloud\\Asset\\v1p2beta1";
+
+// Temporal asset. In addition to the asset, the temporal asset includes the
+// status of the asset and valid from and to time of it.
+message TemporalAsset {
+ // The time window when the asset data and state was observed.
+ TimeWindow window = 1;
+
+ // If the asset is deleted or not.
+ bool deleted = 2;
+
+ // Asset.
+ Asset asset = 3;
+}
+
+// A time window of (start_time, end_time].
+message TimeWindow {
+ // Start time of the time window (exclusive).
+ google.protobuf.Timestamp start_time = 1;
+
+ // End time of the time window (inclusive).
+ // Current timestamp if not specified.
+ google.protobuf.Timestamp end_time = 2;
+}
+
+// Cloud asset. This includes all Google Cloud Platform resources,
+// Cloud IAM policies, and other non-GCP assets.
+message Asset {
+ // The full name of the asset. For example:
+ // `//compute.googleapis.com/projects/my_project_123/zones/zone1/instances/instance1`.
+ // See [Resource
+ // Names](https://cloud.google.com/apis/design/resource_names#full_resource_name)
+ // for more information.
+ string name = 1;
+
+ // Type of the asset. Example: "compute.googleapis.com/Disk".
+ string asset_type = 2;
+
+ // Representation of the resource.
+ Resource resource = 3;
+
+ // Representation of the actual Cloud IAM policy set on a cloud resource. For
+ // each resource, there must be at most one Cloud IAM policy set on it.
+ google.iam.v1.Policy iam_policy = 4;
+
+ // Cloud IAM policy name of the Cloud IAM policy set on a cloud resource. For
+ // each resource, there must be at most one Cloud IAM policy name associated
+ // with it.
+ bytes iam_policy_name = 5;
+
+ // Asset's ancestry path in Cloud Resource Manager (CRM) hierarchy,
+ // represented as a list of relative resource names. Ancestry path starts with
+ // the closest CRM ancestor and ending at a visible root. If the asset is a
+ // CRM project/ folder/organization, this starts from the asset itself.
+ //
+ // Example: ["projects/123456789", "folders/5432", "organizations/1234"]
+ repeated string ancestors = 6;
+}
+
+// Representation of a cloud resource.
+message Resource {
+ // The API version. Example: "v1".
+ string version = 1;
+
+ // The URL of the discovery document containing the resource's JSON schema.
+ // For example:
+ // `"https://www.googleapis.com/discovery/v1/apis/compute/v1/rest"`.
+ // It will be left unspecified for resources without a discovery-based API,
+ // such as Cloud Bigtable.
+ string discovery_document_uri = 2;
+
+ // The JSON schema name listed in the discovery document.
+ // Example: "Project". It will be left unspecified for resources (such as
+ // Cloud Bigtable) without a discovery-based API.
+ string discovery_name = 3;
+
+ // The REST URL for accessing the resource. An HTTP GET operation using this
+ // URL returns the resource itself.
+ // Example:
+ // `https://cloudresourcemanager.googleapis.com/v1/projects/my-project-123`.
+ // It will be left unspecified for resources without a REST API.
+ string resource_url = 4;
+
+ // The full name of the immediate parent of this resource. See
+ // [Resource
+ // Names](https://cloud.google.com/apis/design/resource_names#full_resource_name)
+ // for more information.
+ //
+ // For GCP assets, it is the parent resource defined in the [Cloud IAM policy
+ // hierarchy](https://cloud.google.com/iam/docs/overview#policy_hierarchy).
+ // For example:
+ // `"//cloudresourcemanager.googleapis.com/projects/my_project_123"`.
+ //
+ // For third-party assets, it is up to the users to define.
+ string parent = 5;
+
+ // The content of the resource, in which some sensitive fields are scrubbed
+ // away and may not be present.
+ google.protobuf.Struct data = 6;
+
+ // The actual metadata content for the resource, only visible for internal
+ // users.
+ google.protobuf.Any internal_data = 7;
+}
diff --git a/asset/google/cloud/asset_v1p2beta1/proto/assets_pb2.py b/asset/google/cloud/asset_v1p2beta1/proto/assets_pb2.py
new file mode 100644
index 000000000000..0e086f63c197
--- /dev/null
+++ b/asset/google/cloud/asset_v1p2beta1/proto/assets_pb2.py
@@ -0,0 +1,616 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/asset_v1p2beta1/proto/assets.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+from google.iam.v1 import iam_policy_pb2 as google_dot_iam_dot_v1_dot_policy__pb2
+from google.protobuf import any_pb2 as google_dot_protobuf_dot_any__pb2
+from google.protobuf import struct_pb2 as google_dot_protobuf_dot_struct__pb2
+from google.protobuf import timestamp_pb2 as google_dot_protobuf_dot_timestamp__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/asset_v1p2beta1/proto/assets.proto",
+ package="google.cloud.asset.v1p2beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n com.google.cloud.asset.v1p2beta1B\nAssetProtoP\001ZAgoogle.golang.org/genproto/googleapis/cloud/asset/v1p2beta1;asset\370\001\001\252\002\034Google.Cloud.Asset.v1p2beta1\312\002\034Google\\Cloud\\Asset\\v1p2beta1"
+ ),
+ serialized_pb=_b(
+ '\n/google/cloud/asset_v1p2beta1/proto/assets.proto\x12\x1cgoogle.cloud.asset.v1p2beta1\x1a\x1cgoogle/api/annotations.proto\x1a\x1agoogle/iam/v1/policy.proto\x1a\x19google/protobuf/any.proto\x1a\x1cgoogle/protobuf/struct.proto\x1a\x1fgoogle/protobuf/timestamp.proto"\x8e\x01\n\rTemporalAsset\x12\x38\n\x06window\x18\x01 \x01(\x0b\x32(.google.cloud.asset.v1p2beta1.TimeWindow\x12\x0f\n\x07\x64\x65leted\x18\x02 \x01(\x08\x12\x32\n\x05\x61sset\x18\x03 \x01(\x0b\x32#.google.cloud.asset.v1p2beta1.Asset"j\n\nTimeWindow\x12.\n\nstart_time\x18\x01 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12,\n\x08\x65nd_time\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.Timestamp"\xba\x01\n\x05\x41sset\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x12\n\nasset_type\x18\x02 \x01(\t\x12\x38\n\x08resource\x18\x03 \x01(\x0b\x32&.google.cloud.asset.v1p2beta1.Resource\x12)\n\niam_policy\x18\x04 \x01(\x0b\x32\x15.google.iam.v1.Policy\x12\x17\n\x0fiam_policy_name\x18\x05 \x01(\x0c\x12\x11\n\tancestors\x18\x06 \x03(\t"\xcd\x01\n\x08Resource\x12\x0f\n\x07version\x18\x01 \x01(\t\x12\x1e\n\x16\x64iscovery_document_uri\x18\x02 \x01(\t\x12\x16\n\x0e\x64iscovery_name\x18\x03 \x01(\t\x12\x14\n\x0cresource_url\x18\x04 \x01(\t\x12\x0e\n\x06parent\x18\x05 \x01(\t\x12%\n\x04\x64\x61ta\x18\x06 \x01(\x0b\x32\x17.google.protobuf.Struct\x12+\n\rinternal_data\x18\x07 \x01(\x0b\x32\x14.google.protobuf.AnyB\xb4\x01\n com.google.cloud.asset.v1p2beta1B\nAssetProtoP\x01ZAgoogle.golang.org/genproto/googleapis/cloud/asset/v1p2beta1;asset\xf8\x01\x01\xaa\x02\x1cGoogle.Cloud.Asset.v1p2beta1\xca\x02\x1cGoogle\\Cloud\\Asset\\v1p2beta1b\x06proto3'
+ ),
+ dependencies=[
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ google_dot_iam_dot_v1_dot_policy__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_any__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_struct__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_timestamp__pb2.DESCRIPTOR,
+ ],
+)
+
+
+_TEMPORALASSET = _descriptor.Descriptor(
+ name="TemporalAsset",
+ full_name="google.cloud.asset.v1p2beta1.TemporalAsset",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="window",
+ full_name="google.cloud.asset.v1p2beta1.TemporalAsset.window",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="deleted",
+ full_name="google.cloud.asset.v1p2beta1.TemporalAsset.deleted",
+ index=1,
+ number=2,
+ type=8,
+ cpp_type=7,
+ label=1,
+ has_default_value=False,
+ default_value=False,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="asset",
+ full_name="google.cloud.asset.v1p2beta1.TemporalAsset.asset",
+ index=2,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=230,
+ serialized_end=372,
+)
+
+
+_TIMEWINDOW = _descriptor.Descriptor(
+ name="TimeWindow",
+ full_name="google.cloud.asset.v1p2beta1.TimeWindow",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="start_time",
+ full_name="google.cloud.asset.v1p2beta1.TimeWindow.start_time",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="end_time",
+ full_name="google.cloud.asset.v1p2beta1.TimeWindow.end_time",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=374,
+ serialized_end=480,
+)
+
+
+_ASSET = _descriptor.Descriptor(
+ name="Asset",
+ full_name="google.cloud.asset.v1p2beta1.Asset",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.asset.v1p2beta1.Asset.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="asset_type",
+ full_name="google.cloud.asset.v1p2beta1.Asset.asset_type",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="resource",
+ full_name="google.cloud.asset.v1p2beta1.Asset.resource",
+ index=2,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="iam_policy",
+ full_name="google.cloud.asset.v1p2beta1.Asset.iam_policy",
+ index=3,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="iam_policy_name",
+ full_name="google.cloud.asset.v1p2beta1.Asset.iam_policy_name",
+ index=4,
+ number=5,
+ type=12,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b(""),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="ancestors",
+ full_name="google.cloud.asset.v1p2beta1.Asset.ancestors",
+ index=5,
+ number=6,
+ type=9,
+ cpp_type=9,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=483,
+ serialized_end=669,
+)
+
+
+_RESOURCE = _descriptor.Descriptor(
+ name="Resource",
+ full_name="google.cloud.asset.v1p2beta1.Resource",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="version",
+ full_name="google.cloud.asset.v1p2beta1.Resource.version",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="discovery_document_uri",
+ full_name="google.cloud.asset.v1p2beta1.Resource.discovery_document_uri",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="discovery_name",
+ full_name="google.cloud.asset.v1p2beta1.Resource.discovery_name",
+ index=2,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="resource_url",
+ full_name="google.cloud.asset.v1p2beta1.Resource.resource_url",
+ index=3,
+ number=4,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="parent",
+ full_name="google.cloud.asset.v1p2beta1.Resource.parent",
+ index=4,
+ number=5,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="data",
+ full_name="google.cloud.asset.v1p2beta1.Resource.data",
+ index=5,
+ number=6,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="internal_data",
+ full_name="google.cloud.asset.v1p2beta1.Resource.internal_data",
+ index=6,
+ number=7,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=672,
+ serialized_end=877,
+)
+
+_TEMPORALASSET.fields_by_name["window"].message_type = _TIMEWINDOW
+_TEMPORALASSET.fields_by_name["asset"].message_type = _ASSET
+_TIMEWINDOW.fields_by_name[
+ "start_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_TIMEWINDOW.fields_by_name[
+ "end_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_ASSET.fields_by_name["resource"].message_type = _RESOURCE
+_ASSET.fields_by_name[
+ "iam_policy"
+].message_type = (
+ google_dot_iam_dot_v1_dot_policy__pb2.google_dot_iam_dot_v1_dot_policy__pb2._POLICY
+)
+_RESOURCE.fields_by_name[
+ "data"
+].message_type = google_dot_protobuf_dot_struct__pb2._STRUCT
+_RESOURCE.fields_by_name[
+ "internal_data"
+].message_type = google_dot_protobuf_dot_any__pb2._ANY
+DESCRIPTOR.message_types_by_name["TemporalAsset"] = _TEMPORALASSET
+DESCRIPTOR.message_types_by_name["TimeWindow"] = _TIMEWINDOW
+DESCRIPTOR.message_types_by_name["Asset"] = _ASSET
+DESCRIPTOR.message_types_by_name["Resource"] = _RESOURCE
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+TemporalAsset = _reflection.GeneratedProtocolMessageType(
+ "TemporalAsset",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TEMPORALASSET,
+ __module__="google.cloud.asset_v1p2beta1.proto.assets_pb2",
+ __doc__="""Temporal asset. In addition to the asset, the temporal asset includes
+ the status of the asset and valid from and to time of it.
+
+
+ Attributes:
+ window:
+ The time window when the asset data and state was observed.
+ deleted:
+ If the asset is deleted or not.
+ asset:
+ Asset.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1p2beta1.TemporalAsset)
+ ),
+)
+_sym_db.RegisterMessage(TemporalAsset)
+
+TimeWindow = _reflection.GeneratedProtocolMessageType(
+ "TimeWindow",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TIMEWINDOW,
+ __module__="google.cloud.asset_v1p2beta1.proto.assets_pb2",
+ __doc__="""A time window of (start\_time, end\_time].
+
+
+ Attributes:
+ start_time:
+ Start time of the time window (exclusive).
+ end_time:
+ End time of the time window (inclusive). Current timestamp if
+ not specified.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1p2beta1.TimeWindow)
+ ),
+)
+_sym_db.RegisterMessage(TimeWindow)
+
+Asset = _reflection.GeneratedProtocolMessageType(
+ "Asset",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_ASSET,
+ __module__="google.cloud.asset_v1p2beta1.proto.assets_pb2",
+ __doc__="""Cloud asset. This includes all Google Cloud Platform resources, Cloud
+ IAM policies, and other non-GCP assets.
+
+
+ Attributes:
+ name:
+
+ The full name of the asset. For example:
+ ``//compute.googleapis.com/projects/my_project_123/zones/zone1/instances/instance1``.
+ See https://cloud.google.com/apis/design/resource_names#full_resource_name
+ for more information.
+ asset_type:
+ Type of the asset. Example: "compute.googleapis.com/Disk".
+ resource:
+ Representation of the resource.
+ iam_policy:
+ Representation of the actual Cloud IAM policy set on a cloud
+ resource. For each resource, there must be at most one Cloud
+ IAM policy set on it.
+ iam_policy_name:
+ Cloud IAM policy name of the Cloud IAM policy set on a cloud
+ resource. For each resource, there must be at most one Cloud
+ IAM policy name associated with it.
+ ancestors:
+ Asset's ancestry path in Cloud Resource Manager (CRM)
+ hierarchy, represented as a list of relative resource names.
+ Ancestry path starts with the closest CRM ancestor and ending
+ at a visible root. If the asset is a CRM project/
+ folder/organization, this starts from the asset itself.
+ Example: ["projects/123456789", "folders/5432",
+ "organizations/1234"]
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1p2beta1.Asset)
+ ),
+)
+_sym_db.RegisterMessage(Asset)
+
+Resource = _reflection.GeneratedProtocolMessageType(
+ "Resource",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_RESOURCE,
+ __module__="google.cloud.asset_v1p2beta1.proto.assets_pb2",
+ __doc__="""Representation of a cloud resource.
+
+
+ Attributes:
+ version:
+ The API version. Example: "v1".
+ discovery_document_uri:
+ The URL of the discovery document containing the resource's
+ JSON schema. For example: ``"https://www.googleapis.com/discov
+ ery/v1/apis/compute/v1/rest"``. It will be left unspecified
+ for resources without a discovery-based API, such as Cloud
+ Bigtable.
+ discovery_name:
+ The JSON schema name listed in the discovery document.
+ Example: "Project". It will be left unspecified for resources
+ (such as Cloud Bigtable) without a discovery-based API.
+ resource_url:
+ The REST URL for accessing the resource. An HTTP GET operation
+ using this URL returns the resource itself. Example:
+ ``https://cloudresourcemanager.googleapis.com/v1/projects/my-
+ project-123``. It will be left unspecified for resources
+ without a REST API.
+ parent:
+ The full name of the immediate parent of this resource. See
+ `Resource Names `__ for more information. For GCP
+ assets, it is the parent resource defined in the `Cloud IAM
+ policy hierarchy `__. For example: ``"//cloudresourcemanager.go
+ ogleapis.com/projects/my_project_123"``. For third-party
+ assets, it is up to the users to define.
+ data:
+ The content of the resource, in which some sensitive fields
+ are scrubbed away and may not be present.
+ internal_data:
+ The actual metadata content for the resource, only visible for
+ internal users.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.asset.v1p2beta1.Resource)
+ ),
+)
+_sym_db.RegisterMessage(Resource)
+
+
+DESCRIPTOR._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/asset/google/cloud/asset_v1p2beta1/proto/assets_pb2_grpc.py b/asset/google/cloud/asset_v1p2beta1/proto/assets_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/asset/google/cloud/asset_v1p2beta1/proto/assets_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/asset/google/cloud/asset_v1p2beta1/types.py b/asset/google/cloud/asset_v1p2beta1/types.py
new file mode 100644
index 000000000000..adaff8cd6862
--- /dev/null
+++ b/asset/google/cloud/asset_v1p2beta1/types.py
@@ -0,0 +1,63 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from __future__ import absolute_import
+import sys
+
+from google.api_core.protobuf_helpers import get_messages
+
+from google.cloud.asset_v1p2beta1.proto import asset_service_pb2
+from google.cloud.asset_v1p2beta1.proto import assets_pb2
+from google.iam.v1 import policy_pb2
+from google.longrunning import operations_pb2
+from google.protobuf import any_pb2
+from google.protobuf import empty_pb2
+from google.protobuf import field_mask_pb2
+from google.protobuf import struct_pb2
+from google.protobuf import timestamp_pb2
+from google.rpc import status_pb2
+from google.type import expr_pb2
+
+
+_shared_modules = [
+ policy_pb2,
+ operations_pb2,
+ any_pb2,
+ empty_pb2,
+ field_mask_pb2,
+ struct_pb2,
+ timestamp_pb2,
+ status_pb2,
+ expr_pb2,
+]
+
+_local_modules = [asset_service_pb2, assets_pb2]
+
+names = []
+
+for module in _shared_modules: # pragma: NO COVER
+ for name, message in get_messages(module).items():
+ setattr(sys.modules[__name__], name, message)
+ names.append(name)
+for module in _local_modules:
+ for name, message in get_messages(module).items():
+ message.__module__ = "google.cloud.asset_v1p2beta1.types"
+ setattr(sys.modules[__name__], name, message)
+ names.append(name)
+
+
+__all__ = tuple(sorted(names))
diff --git a/asset/noxfile.py b/asset/noxfile.py
new file mode 100644
index 000000000000..393f4b64a33c
--- /dev/null
+++ b/asset/noxfile.py
@@ -0,0 +1,169 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Generated by synthtool. DO NOT EDIT!
+
+from __future__ import absolute_import
+import os
+import shutil
+
+import nox
+
+
+LOCAL_DEPS = (os.path.join("..", "api_core"), os.path.join("..", "core"))
+BLACK_VERSION = "black==19.3b0"
+BLACK_PATHS = ["docs", "google", "tests", "noxfile.py", "setup.py"]
+
+if os.path.exists("samples"):
+ BLACK_PATHS.append("samples")
+
+
+@nox.session(python="3.7")
+def lint(session):
+ """Run linters.
+
+ Returns a failure if the linters find linting errors or sufficiently
+ serious code quality issues.
+ """
+ session.install("flake8", BLACK_VERSION, *LOCAL_DEPS)
+ session.run("black", "--check", *BLACK_PATHS)
+ session.run("flake8", "google", "tests")
+
+
+@nox.session(python="3.6")
+def blacken(session):
+ """Run black.
+
+ Format code to uniform standard.
+
+ This currently uses Python 3.6 due to the automated Kokoro run of synthtool.
+ That run uses an image that doesn't have 3.6 installed. Before updating this
+ check the state of the `gcp_ubuntu_config` we use for that Kokoro run.
+ """
+ session.install(BLACK_VERSION)
+ session.run("black", *BLACK_PATHS)
+
+
+@nox.session(python="3.7")
+def lint_setup_py(session):
+ """Verify that setup.py is valid (including RST check)."""
+ session.install("docutils", "pygments")
+ session.run("python", "setup.py", "check", "--restructuredtext", "--strict")
+
+
+def default(session):
+ # Install all test dependencies, then install this package in-place.
+ session.install("mock", "pytest", "pytest-cov")
+ for local_dep in LOCAL_DEPS:
+ session.install("-e", local_dep)
+ session.install("-e", ".")
+
+ # Run py.test against the unit tests.
+ session.run(
+ "py.test",
+ "--quiet",
+ "--cov=google.cloud",
+ "--cov=tests.unit",
+ "--cov-append",
+ "--cov-config=.coveragerc",
+ "--cov-report=",
+ "--cov-fail-under=0",
+ os.path.join("tests", "unit"),
+ *session.posargs,
+ )
+
+
+@nox.session(python=["2.7", "3.5", "3.6", "3.7"])
+def unit(session):
+ """Run the unit test suite."""
+ default(session)
+
+
+@nox.session(python=["2.7", "3.7"])
+def system(session):
+ """Run the system test suite."""
+ system_test_path = os.path.join("tests", "system.py")
+ system_test_folder_path = os.path.join("tests", "system")
+ # Sanity check: Only run tests if the environment variable is set.
+ if not os.environ.get("GOOGLE_APPLICATION_CREDENTIALS", ""):
+ session.skip("Credentials must be set via environment variable")
+
+ system_test_exists = os.path.exists(system_test_path)
+ system_test_folder_exists = os.path.exists(system_test_folder_path)
+ # Sanity check: only run tests if found.
+ if not system_test_exists and not system_test_folder_exists:
+ session.skip("System tests were not found")
+
+ # Use pre-release gRPC for system tests.
+ session.install("--pre", "grpcio")
+
+ # Install all test dependencies, then install this package into the
+ # virtualenv's dist-packages.
+ session.install("mock", "pytest")
+ for local_dep in LOCAL_DEPS:
+ session.install("-e", local_dep)
+ session.install("-e", "../test_utils/")
+ session.install("-e", ".")
+
+ # Additional setup for VPCSC system tests
+ env = {
+ "PROJECT_ID": os.environ.get("PROJECT_ID"),
+ "GOOGLE_CLOUD_TESTS_VPCSC_OUTSIDE_PERIMETER_PROJECT": "secure-gcp-test-project-4",
+ "GOOGLE_CLOUD_TESTS_IN_VPCSC": "true",
+ }
+
+ # Run py.test against the system tests.
+ if system_test_exists:
+ session.run("py.test", "--quiet", system_test_path, env=env, *session.posargs)
+ if system_test_folder_exists:
+ session.run(
+ "py.test", "--quiet", system_test_folder_path, env=env, *session.posargs
+ )
+
+
+@nox.session(python="3.7")
+def cover(session):
+ """Run the final coverage report.
+
+ This outputs the coverage report aggregating coverage from the unit
+ test runs (not system test runs), and then erases coverage data.
+ """
+ session.install("coverage", "pytest-cov")
+ session.run("coverage", "report", "--show-missing", "--fail-under=80")
+
+ session.run("coverage", "erase")
+
+
+@nox.session(python="3.7")
+def docs(session):
+ """Build the docs for this library."""
+
+ session.install("-e", ".")
+ session.install("sphinx", "alabaster", "recommonmark")
+
+ shutil.rmtree(os.path.join("docs", "_build"), ignore_errors=True)
+ session.run(
+ "sphinx-build",
+ "-W", # warnings as errors
+ "-T", # show full traceback on exception
+ "-N", # no colors
+ "-b",
+ "html",
+ "-d",
+ os.path.join("docs", "_build", "doctrees", ""),
+ os.path.join("docs", ""),
+ os.path.join("docs", "_build", "html", ""),
+ )
diff --git a/asset/setup.cfg b/asset/setup.cfg
new file mode 100644
index 000000000000..3bd555500e37
--- /dev/null
+++ b/asset/setup.cfg
@@ -0,0 +1,3 @@
+# Generated by synthtool. DO NOT EDIT!
+[bdist_wheel]
+universal = 1
diff --git a/asset/setup.py b/asset/setup.py
new file mode 100644
index 000000000000..a70843c8e1de
--- /dev/null
+++ b/asset/setup.py
@@ -0,0 +1,85 @@
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import io
+import os
+
+import setuptools
+
+# Package metadata.
+
+name = "google-cloud-asset"
+description = "Cloud Asset API API client library"
+version = "0.4.1"
+# Should be one of:
+# 'Development Status :: 3 - Alpha'
+# 'Development Status :: 4 - Beta'
+# 'Development Status :: 5 - Production/Stable'
+release_status = "Development Status :: 3 - Alpha"
+dependencies = [
+ "google-api-core[grpc] >= 1.14.0, < 2.0.0dev",
+ 'enum34; python_version < "3.4"',
+ "grpc-google-iam-v1 >= 0.12.3, < 0.13dev",
+]
+
+# Setup boilerplate below this line.
+
+package_root = os.path.abspath(os.path.dirname(__file__))
+
+readme_filename = os.path.join(package_root, "README.rst")
+with io.open(readme_filename, encoding="utf-8") as readme_file:
+ readme = readme_file.read()
+
+# Only include packages under the 'google' namespace. Do not include tests,
+# benchmarks, etc.
+packages = [
+ package for package in setuptools.find_packages() if package.startswith("google")
+]
+
+# Determine which namespaces are needed.
+namespaces = ["google"]
+if "google.cloud" in packages:
+ namespaces.append("google.cloud")
+
+setuptools.setup(
+ name=name,
+ version=version,
+ description=description,
+ long_description=readme,
+ author="Google LLC",
+ author_email="googleapis-packages@google.com",
+ license="Apache 2.0",
+ url="https://github.com/GoogleCloudPlatform/google-cloud-python",
+ classifiers=[
+ release_status,
+ "Intended Audience :: Developers",
+ "License :: OSI Approved :: Apache Software License",
+ "Programming Language :: Python",
+ "Programming Language :: Python :: 2",
+ "Programming Language :: Python :: 2.7",
+ "Programming Language :: Python :: 3",
+ "Programming Language :: Python :: 3.5",
+ "Programming Language :: Python :: 3.6",
+ "Programming Language :: Python :: 3.7",
+ "Operating System :: OS Independent",
+ "Topic :: Internet",
+ ],
+ platforms="Posix; MacOS X; Windows",
+ packages=packages,
+ namespace_packages=namespaces,
+ install_requires=dependencies,
+ python_requires=">=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*",
+ include_package_data=True,
+ zip_safe=False,
+)
diff --git a/asset/synth.metadata b/asset/synth.metadata
new file mode 100644
index 000000000000..2a27fdfa4cf0
--- /dev/null
+++ b/asset/synth.metadata
@@ -0,0 +1,59 @@
+{
+ "updateTime": "2019-08-06T12:11:34.528326Z",
+ "sources": [
+ {
+ "generator": {
+ "name": "artman",
+ "version": "0.32.1",
+ "dockerImage": "googleapis/artman@sha256:a684d40ba9a4e15946f5f2ca6b4bd9fe301192f522e9de4fff622118775f309b"
+ }
+ },
+ {
+ "git": {
+ "name": "googleapis",
+ "remote": "https://github.com/googleapis/googleapis.git",
+ "sha": "e699b0cba64ffddfae39633417180f1f65875896",
+ "internalRef": "261759677"
+ }
+ },
+ {
+ "template": {
+ "name": "python_library",
+ "origin": "synthtool.gcp",
+ "version": "2019.5.2"
+ }
+ }
+ ],
+ "destinations": [
+ {
+ "client": {
+ "source": "googleapis",
+ "apiName": "asset",
+ "apiVersion": "v1beta1",
+ "language": "python",
+ "generator": "gapic",
+ "config": "google/cloud/asset/artman_cloudasset_v1beta1.yaml"
+ }
+ },
+ {
+ "client": {
+ "source": "googleapis",
+ "apiName": "asset",
+ "apiVersion": "v1p2beta1",
+ "language": "python",
+ "generator": "gapic",
+ "config": "google/cloud/asset/artman_cloudasset_v1p2beta1.yaml"
+ }
+ },
+ {
+ "client": {
+ "source": "googleapis",
+ "apiName": "asset",
+ "apiVersion": "v1",
+ "language": "python",
+ "generator": "gapic",
+ "config": "google/cloud/asset/artman_cloudasset_v1.yaml"
+ }
+ }
+ ]
+}
\ No newline at end of file
diff --git a/asset/synth.py b/asset/synth.py
new file mode 100644
index 000000000000..9f4498bdbb58
--- /dev/null
+++ b/asset/synth.py
@@ -0,0 +1,89 @@
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""This script is used to synthesize generated parts of this library."""
+
+import synthtool as s
+from synthtool import gcp
+
+gapic = gcp.GAPICGenerator()
+common = gcp.CommonTemplates()
+versions = ["v1beta1", "v1p2beta1", "v1"]
+
+excludes = ["setup.py", "nox*.py", "README.rst", "docs/conf.py", "docs/index.rst"]
+
+# ----------------------------------------------------------------------------
+# Generate asset GAPIC layer
+# ----------------------------------------------------------------------------
+for version in versions:
+ library = gapic.py_library(
+ "asset",
+ version,
+ config_path=f"/google/cloud/asset/artman_cloudasset_{version}.yaml",
+ artman_output_name=f"asset-{version}",
+ include_protos=True,
+ )
+
+ s.move(library, excludes=excludes)
+
+s.replace(
+ "google/cloud/asset_v*/proto/assets_pb2.py",
+ "from google.iam.v1 import policy_pb2 as",
+ "from google.iam.v1 import iam_policy_pb2_grpc as",
+)
+
+s.replace(
+ "google/cloud/asset_v*/proto/assets_pb2.py",
+ "from google.iam.v1 import iam_policy_pb2_grpc "
+ "as google_dot_iam_dot_v1_dot_policy__pb2",
+ "from google.iam.v1 import iam_policy_pb2 "
+ "as google_dot_iam_dot_v1_dot_policy__pb2",
+)
+
+s.replace(
+ "google/cloud/asset_v*/proto/assets_pb2.py",
+ "_ASSET.fields_by_name\['iam_policy'\].message_type "
+ "= google_dot_iam_dot_v1_dot_policy__pb2._POLICY",
+ "_ASSET.fields_by_name['iam_policy'].message_type = google_dot_iam_dot"
+ "_v1_dot_policy__pb2.google_dot_iam_dot_v1_dot_policy__pb2._POLICY",
+)
+
+
+_BORKED_ASSET_DOCSTRING = """\
+ The full name of the asset. For example: ``//compute.googleapi
+ s.com/projects/my_project_123/zones/zone1/instances/instance1`
+ `. See `Resource Names `__ for more information.
+"""
+
+_FIXED_ASSET_DOCSTRING = """
+ The full name of the asset. For example:
+ ``//compute.googleapis.com/projects/my_project_123/zones/zone1/instances/instance1``.
+ See https://cloud.google.com/apis/design/resource_names#full_resource_name
+ for more information.
+"""
+
+s.replace(
+ "google/cloud/asset_v*/proto/assets_pb2.py",
+ _BORKED_ASSET_DOCSTRING,
+ _FIXED_ASSET_DOCSTRING,
+)
+
+# ----------------------------------------------------------------------------
+# Add templated files
+# ----------------------------------------------------------------------------
+templated_files = gcp.CommonTemplates().py_library(unit_cov_level=79, cov_level=80)
+s.move(templated_files, excludes=["noxfile.py"])
+
+s.shell.run(["nox", "-s", "blacken"], hide_output=False)
diff --git a/asset/tests/system/test_vpcsc.py b/asset/tests/system/test_vpcsc.py
new file mode 100644
index 000000000000..e3336bfe4f4f
--- /dev/null
+++ b/asset/tests/system/test_vpcsc.py
@@ -0,0 +1,88 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# flake8: noqa
+
+import os
+import pytest
+
+from google.api_core import exceptions
+from google.cloud import asset_v1
+from google.cloud.asset_v1 import enums
+
+PROJECT_INSIDE = os.environ.get("PROJECT_ID", None)
+PROJECT_OUTSIDE = os.environ.get(
+ "GOOGLE_CLOUD_TESTS_VPCSC_OUTSIDE_PERIMETER_PROJECT", None
+)
+IS_INSIDE_VPCSC = os.environ.get("GOOGLE_CLOUD_TESTS_IN_VPCSC", "true")
+
+
+class TestVPCServiceControl(object):
+ @staticmethod
+ def _is_rejected(call):
+ try:
+ responses = call()
+ except exceptions.PermissionDenied as e:
+ return e.message == "Request is prohibited by organization's policy"
+ except:
+ pass
+ return False
+
+ @staticmethod
+ def _do_test(delayed_inside, delayed_outside):
+ if IS_INSIDE_VPCSC.lower() == "true":
+ assert TestVPCServiceControl._is_rejected(delayed_outside)
+ assert not (TestVPCServiceControl._is_rejected(delayed_inside))
+ else:
+ assert not (TestVPCServiceControl._is_rejected(delayed_outside))
+ assert TestVPCServiceControl._is_rejected(delayed_inside)
+
+ @pytest.mark.skipif(
+ PROJECT_INSIDE is None, reason="Missing environment variable: PROJECT_ID"
+ )
+ @pytest.mark.skipif(
+ PROJECT_OUTSIDE is None,
+ reason="Missing environment variable: GOOGLE_CLOUD_TESTS_VPCSC_OUTSIDE_PERIMETER_PROJECT",
+ )
+ def test_export_assets(self):
+ client = asset_v1.AssetServiceClient()
+ output_config = {}
+ parent_inside = "projects/" + PROJECT_INSIDE
+ delayed_inside = lambda: client.export_assets(parent_inside, output_config)
+ parent_outside = "projects/" + PROJECT_OUTSIDE
+ delayed_outside = lambda: client.export_assets(parent_outside, output_config)
+ TestVPCServiceControl._do_test(delayed_inside, delayed_outside)
+
+ @pytest.mark.skipif(
+ PROJECT_INSIDE is None, reason="Missing environment variable: PROJECT_ID"
+ )
+ @pytest.mark.skipif(
+ PROJECT_OUTSIDE is None,
+ reason="Missing environment variable: GOOGLE_CLOUD_TESTS_VPCSC_OUTSIDE_PERIMETER_PROJECT",
+ )
+ def test_batch_get_assets_history(self):
+ client = asset_v1.AssetServiceClient()
+ content_type = enums.ContentType.CONTENT_TYPE_UNSPECIFIED
+ read_time_window = {}
+ parent_inside = "projects/" + PROJECT_INSIDE
+ delayed_inside = lambda: client.batch_get_assets_history(
+ parent_inside, content_type, read_time_window
+ )
+ parent_outside = "projects/" + PROJECT_OUTSIDE
+ delayed_outside = lambda: client.batch_get_assets_history(
+ parent_outside, content_type, read_time_window
+ )
+ TestVPCServiceControl._do_test(delayed_inside, delayed_outside)
diff --git a/asset/tests/unit/gapic/v1/test_asset_service_client_v1.py b/asset/tests/unit/gapic/v1/test_asset_service_client_v1.py
new file mode 100644
index 000000000000..0afe1cf92033
--- /dev/null
+++ b/asset/tests/unit/gapic/v1/test_asset_service_client_v1.py
@@ -0,0 +1,167 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Unit tests."""
+
+import mock
+import pytest
+
+from google.rpc import status_pb2
+
+from google.cloud import asset_v1
+from google.cloud.asset_v1 import enums
+from google.cloud.asset_v1.proto import asset_service_pb2
+from google.cloud.asset_v1.proto import assets_pb2
+from google.longrunning import operations_pb2
+
+
+class MultiCallableStub(object):
+ """Stub for the grpc.UnaryUnaryMultiCallable interface."""
+
+ def __init__(self, method, channel_stub):
+ self.method = method
+ self.channel_stub = channel_stub
+
+ def __call__(self, request, timeout=None, metadata=None, credentials=None):
+ self.channel_stub.requests.append((self.method, request))
+
+ response = None
+ if self.channel_stub.responses:
+ response = self.channel_stub.responses.pop()
+
+ if isinstance(response, Exception):
+ raise response
+
+ if response:
+ return response
+
+
+class ChannelStub(object):
+ """Stub for the grpc.Channel interface."""
+
+ def __init__(self, responses=[]):
+ self.responses = responses
+ self.requests = []
+
+ def unary_unary(self, method, request_serializer=None, response_deserializer=None):
+ return MultiCallableStub(method, self)
+
+
+class CustomException(Exception):
+ pass
+
+
+class TestAssetServiceClient(object):
+ def test_export_assets(self):
+ # Setup Expected Response
+ expected_response = {}
+ expected_response = asset_service_pb2.ExportAssetsResponse(**expected_response)
+ operation = operations_pb2.Operation(
+ name="operations/test_export_assets", done=True
+ )
+ operation.response.Pack(expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[operation])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = asset_v1.AssetServiceClient()
+
+ # Setup Request
+ parent = "parent-995424086"
+ output_config = {}
+
+ response = client.export_assets(parent, output_config)
+ result = response.result()
+ assert expected_response == result
+
+ assert len(channel.requests) == 1
+ expected_request = asset_service_pb2.ExportAssetsRequest(
+ parent=parent, output_config=output_config
+ )
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_export_assets_exception(self):
+ # Setup Response
+ error = status_pb2.Status()
+ operation = operations_pb2.Operation(
+ name="operations/test_export_assets_exception", done=True
+ )
+ operation.error.CopyFrom(error)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[operation])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = asset_v1.AssetServiceClient()
+
+ # Setup Request
+ parent = "parent-995424086"
+ output_config = {}
+
+ response = client.export_assets(parent, output_config)
+ exception = response.exception()
+ assert exception.errors[0] == error
+
+ def test_batch_get_assets_history(self):
+ # Setup Expected Response
+ expected_response = {}
+ expected_response = asset_service_pb2.BatchGetAssetsHistoryResponse(
+ **expected_response
+ )
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = asset_v1.AssetServiceClient()
+
+ # Setup Request
+ parent = "parent-995424086"
+ content_type = enums.ContentType.CONTENT_TYPE_UNSPECIFIED
+ read_time_window = {}
+
+ response = client.batch_get_assets_history(
+ parent, content_type, read_time_window
+ )
+ assert expected_response == response
+
+ assert len(channel.requests) == 1
+ expected_request = asset_service_pb2.BatchGetAssetsHistoryRequest(
+ parent=parent, content_type=content_type, read_time_window=read_time_window
+ )
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_batch_get_assets_history_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = asset_v1.AssetServiceClient()
+
+ # Setup request
+ parent = "parent-995424086"
+ content_type = enums.ContentType.CONTENT_TYPE_UNSPECIFIED
+ read_time_window = {}
+
+ with pytest.raises(CustomException):
+ client.batch_get_assets_history(parent, content_type, read_time_window)
diff --git a/asset/tests/unit/gapic/v1beta1/test_asset_service_client_v1beta1.py b/asset/tests/unit/gapic/v1beta1/test_asset_service_client_v1beta1.py
new file mode 100644
index 000000000000..7ade3e8d5402
--- /dev/null
+++ b/asset/tests/unit/gapic/v1beta1/test_asset_service_client_v1beta1.py
@@ -0,0 +1,167 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Unit tests."""
+
+import mock
+import pytest
+
+from google.rpc import status_pb2
+
+from google.cloud import asset_v1beta1
+from google.cloud.asset_v1beta1 import enums
+from google.cloud.asset_v1beta1.proto import asset_service_pb2
+from google.cloud.asset_v1beta1.proto import assets_pb2
+from google.longrunning import operations_pb2
+
+
+class MultiCallableStub(object):
+ """Stub for the grpc.UnaryUnaryMultiCallable interface."""
+
+ def __init__(self, method, channel_stub):
+ self.method = method
+ self.channel_stub = channel_stub
+
+ def __call__(self, request, timeout=None, metadata=None, credentials=None):
+ self.channel_stub.requests.append((self.method, request))
+
+ response = None
+ if self.channel_stub.responses:
+ response = self.channel_stub.responses.pop()
+
+ if isinstance(response, Exception):
+ raise response
+
+ if response:
+ return response
+
+
+class ChannelStub(object):
+ """Stub for the grpc.Channel interface."""
+
+ def __init__(self, responses=[]):
+ self.responses = responses
+ self.requests = []
+
+ def unary_unary(self, method, request_serializer=None, response_deserializer=None):
+ return MultiCallableStub(method, self)
+
+
+class CustomException(Exception):
+ pass
+
+
+class TestAssetServiceClient(object):
+ def test_export_assets(self):
+ # Setup Expected Response
+ expected_response = {}
+ expected_response = asset_service_pb2.ExportAssetsResponse(**expected_response)
+ operation = operations_pb2.Operation(
+ name="operations/test_export_assets", done=True
+ )
+ operation.response.Pack(expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[operation])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = asset_v1beta1.AssetServiceClient()
+
+ # Setup Request
+ parent = client.project_path("[PROJECT]")
+ output_config = {}
+
+ response = client.export_assets(parent, output_config)
+ result = response.result()
+ assert expected_response == result
+
+ assert len(channel.requests) == 1
+ expected_request = asset_service_pb2.ExportAssetsRequest(
+ parent=parent, output_config=output_config
+ )
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_export_assets_exception(self):
+ # Setup Response
+ error = status_pb2.Status()
+ operation = operations_pb2.Operation(
+ name="operations/test_export_assets_exception", done=True
+ )
+ operation.error.CopyFrom(error)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[operation])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = asset_v1beta1.AssetServiceClient()
+
+ # Setup Request
+ parent = client.project_path("[PROJECT]")
+ output_config = {}
+
+ response = client.export_assets(parent, output_config)
+ exception = response.exception()
+ assert exception.errors[0] == error
+
+ def test_batch_get_assets_history(self):
+ # Setup Expected Response
+ expected_response = {}
+ expected_response = asset_service_pb2.BatchGetAssetsHistoryResponse(
+ **expected_response
+ )
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = asset_v1beta1.AssetServiceClient()
+
+ # Setup Request
+ parent = client.project_path("[PROJECT]")
+ content_type = enums.ContentType.CONTENT_TYPE_UNSPECIFIED
+ read_time_window = {}
+
+ response = client.batch_get_assets_history(
+ parent, content_type, read_time_window
+ )
+ assert expected_response == response
+
+ assert len(channel.requests) == 1
+ expected_request = asset_service_pb2.BatchGetAssetsHistoryRequest(
+ parent=parent, content_type=content_type, read_time_window=read_time_window
+ )
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_batch_get_assets_history_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = asset_v1beta1.AssetServiceClient()
+
+ # Setup request
+ parent = client.project_path("[PROJECT]")
+ content_type = enums.ContentType.CONTENT_TYPE_UNSPECIFIED
+ read_time_window = {}
+
+ with pytest.raises(CustomException):
+ client.batch_get_assets_history(parent, content_type, read_time_window)
diff --git a/asset/tests/unit/gapic/v1p2beta1/test_asset_service_client_v1p2beta1.py b/asset/tests/unit/gapic/v1p2beta1/test_asset_service_client_v1p2beta1.py
new file mode 100644
index 000000000000..b893a5d278af
--- /dev/null
+++ b/asset/tests/unit/gapic/v1p2beta1/test_asset_service_client_v1p2beta1.py
@@ -0,0 +1,358 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Unit tests."""
+
+import mock
+import pytest
+
+from google.rpc import status_pb2
+
+from google.cloud import asset_v1p2beta1
+from google.cloud.asset_v1p2beta1 import enums
+from google.cloud.asset_v1p2beta1.proto import asset_service_pb2
+from google.longrunning import operations_pb2
+from google.protobuf import empty_pb2
+from google.protobuf import field_mask_pb2
+
+
+class MultiCallableStub(object):
+ """Stub for the grpc.UnaryUnaryMultiCallable interface."""
+
+ def __init__(self, method, channel_stub):
+ self.method = method
+ self.channel_stub = channel_stub
+
+ def __call__(self, request, timeout=None, metadata=None, credentials=None):
+ self.channel_stub.requests.append((self.method, request))
+
+ response = None
+ if self.channel_stub.responses:
+ response = self.channel_stub.responses.pop()
+
+ if isinstance(response, Exception):
+ raise response
+
+ if response:
+ return response
+
+
+class ChannelStub(object):
+ """Stub for the grpc.Channel interface."""
+
+ def __init__(self, responses=[]):
+ self.responses = responses
+ self.requests = []
+
+ def unary_unary(self, method, request_serializer=None, response_deserializer=None):
+ return MultiCallableStub(method, self)
+
+
+class CustomException(Exception):
+ pass
+
+
+class TestAssetServiceClient(object):
+ def test_export_assets(self):
+ # Setup Expected Response
+ expected_response = {}
+ expected_response = asset_service_pb2.ExportAssetsResponse(**expected_response)
+ operation = operations_pb2.Operation(
+ name="operations/test_export_assets", done=True
+ )
+ operation.response.Pack(expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[operation])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = asset_v1p2beta1.AssetServiceClient()
+
+ # Setup Request
+ parent = "parent-995424086"
+ output_config = {}
+
+ response = client.export_assets(parent, output_config)
+ result = response.result()
+ assert expected_response == result
+
+ assert len(channel.requests) == 1
+ expected_request = asset_service_pb2.ExportAssetsRequest(
+ parent=parent, output_config=output_config
+ )
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_export_assets_exception(self):
+ # Setup Response
+ error = status_pb2.Status()
+ operation = operations_pb2.Operation(
+ name="operations/test_export_assets_exception", done=True
+ )
+ operation.error.CopyFrom(error)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[operation])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = asset_v1p2beta1.AssetServiceClient()
+
+ # Setup Request
+ parent = "parent-995424086"
+ output_config = {}
+
+ response = client.export_assets(parent, output_config)
+ exception = response.exception()
+ assert exception.errors[0] == error
+
+ def test_batch_get_assets_history(self):
+ # Setup Expected Response
+ expected_response = {}
+ expected_response = asset_service_pb2.BatchGetAssetsHistoryResponse(
+ **expected_response
+ )
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = asset_v1p2beta1.AssetServiceClient()
+
+ # Setup Request
+ parent = "parent-995424086"
+ asset_names = []
+ content_type = enums.ContentType.CONTENT_TYPE_UNSPECIFIED
+
+ response = client.batch_get_assets_history(parent, asset_names, content_type)
+ assert expected_response == response
+
+ assert len(channel.requests) == 1
+ expected_request = asset_service_pb2.BatchGetAssetsHistoryRequest(
+ parent=parent, asset_names=asset_names, content_type=content_type
+ )
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_batch_get_assets_history_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = asset_v1p2beta1.AssetServiceClient()
+
+ # Setup request
+ parent = "parent-995424086"
+ asset_names = []
+ content_type = enums.ContentType.CONTENT_TYPE_UNSPECIFIED
+
+ with pytest.raises(CustomException):
+ client.batch_get_assets_history(parent, asset_names, content_type)
+
+ def test_create_feed(self):
+ # Setup Expected Response
+ name = "name3373707"
+ expected_response = {"name": name}
+ expected_response = asset_service_pb2.Feed(**expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = asset_v1p2beta1.AssetServiceClient()
+
+ # Setup Request
+ parent = "parent-995424086"
+ feed_id = "feedId-976011428"
+ feed = {}
+
+ response = client.create_feed(parent, feed_id, feed)
+ assert expected_response == response
+
+ assert len(channel.requests) == 1
+ expected_request = asset_service_pb2.CreateFeedRequest(
+ parent=parent, feed_id=feed_id, feed=feed
+ )
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_create_feed_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = asset_v1p2beta1.AssetServiceClient()
+
+ # Setup request
+ parent = "parent-995424086"
+ feed_id = "feedId-976011428"
+ feed = {}
+
+ with pytest.raises(CustomException):
+ client.create_feed(parent, feed_id, feed)
+
+ def test_get_feed(self):
+ # Setup Expected Response
+ name_2 = "name2-1052831874"
+ expected_response = {"name": name_2}
+ expected_response = asset_service_pb2.Feed(**expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = asset_v1p2beta1.AssetServiceClient()
+
+ # Setup Request
+ name = client.feed_path("[PROJECT]", "[FEED]")
+
+ response = client.get_feed(name)
+ assert expected_response == response
+
+ assert len(channel.requests) == 1
+ expected_request = asset_service_pb2.GetFeedRequest(name=name)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_get_feed_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = asset_v1p2beta1.AssetServiceClient()
+
+ # Setup request
+ name = client.feed_path("[PROJECT]", "[FEED]")
+
+ with pytest.raises(CustomException):
+ client.get_feed(name)
+
+ def test_list_feeds(self):
+ # Setup Expected Response
+ expected_response = {}
+ expected_response = asset_service_pb2.ListFeedsResponse(**expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = asset_v1p2beta1.AssetServiceClient()
+
+ # Setup Request
+ parent = "parent-995424086"
+
+ response = client.list_feeds(parent)
+ assert expected_response == response
+
+ assert len(channel.requests) == 1
+ expected_request = asset_service_pb2.ListFeedsRequest(parent=parent)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_list_feeds_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = asset_v1p2beta1.AssetServiceClient()
+
+ # Setup request
+ parent = "parent-995424086"
+
+ with pytest.raises(CustomException):
+ client.list_feeds(parent)
+
+ def test_update_feed(self):
+ # Setup Expected Response
+ name = "name3373707"
+ expected_response = {"name": name}
+ expected_response = asset_service_pb2.Feed(**expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = asset_v1p2beta1.AssetServiceClient()
+
+ # Setup Request
+ feed = {}
+ update_mask = {}
+
+ response = client.update_feed(feed, update_mask)
+ assert expected_response == response
+
+ assert len(channel.requests) == 1
+ expected_request = asset_service_pb2.UpdateFeedRequest(
+ feed=feed, update_mask=update_mask
+ )
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_update_feed_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = asset_v1p2beta1.AssetServiceClient()
+
+ # Setup request
+ feed = {}
+ update_mask = {}
+
+ with pytest.raises(CustomException):
+ client.update_feed(feed, update_mask)
+
+ def test_delete_feed(self):
+ channel = ChannelStub()
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = asset_v1p2beta1.AssetServiceClient()
+
+ # Setup Request
+ name = client.feed_path("[PROJECT]", "[FEED]")
+
+ client.delete_feed(name)
+
+ assert len(channel.requests) == 1
+ expected_request = asset_service_pb2.DeleteFeedRequest(name=name)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_delete_feed_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = asset_v1p2beta1.AssetServiceClient()
+
+ # Setup request
+ name = client.feed_path("[PROJECT]", "[FEED]")
+
+ with pytest.raises(CustomException):
+ client.delete_feed(name)
diff --git a/automl/.coveragerc b/automl/.coveragerc
new file mode 100644
index 000000000000..b178b094aa1d
--- /dev/null
+++ b/automl/.coveragerc
@@ -0,0 +1,19 @@
+# Generated by synthtool. DO NOT EDIT!
+[run]
+branch = True
+
+[report]
+fail_under = 100
+show_missing = True
+exclude_lines =
+ # Re-enable the standard pragma
+ pragma: NO COVER
+ # Ignore debug-only repr
+ def __repr__
+ # Ignore abstract methods
+ raise NotImplementedError
+omit =
+ */gapic/*.py
+ */proto/*.py
+ */core/*.py
+ */site-packages/*.py
\ No newline at end of file
diff --git a/automl/.flake8 b/automl/.flake8
new file mode 100644
index 000000000000..0268ecc9c55c
--- /dev/null
+++ b/automl/.flake8
@@ -0,0 +1,14 @@
+# Generated by synthtool. DO NOT EDIT!
+[flake8]
+ignore = E203, E266, E501, W503
+exclude =
+ # Exclude generated code.
+ **/proto/**
+ **/gapic/**
+ *_pb2.py
+
+ # Standard linting exemptions.
+ __pycache__,
+ .git,
+ *.pyc,
+ conf.py
diff --git a/automl/.repo-metadata.json b/automl/.repo-metadata.json
new file mode 100644
index 000000000000..6d94acb2d745
--- /dev/null
+++ b/automl/.repo-metadata.json
@@ -0,0 +1,13 @@
+{
+ "name": "automl",
+ "name_pretty": "Cloud AutoML",
+ "product_documentation": "https://cloud.google.com/automl/docs/",
+ "client_documentation": "https://googleapis.dev/python/automl/latest",
+ "issue_tracker": "https://issuetracker.google.com/savedsearches/559744",
+ "release_level": "alpha",
+ "language": "python",
+ "repo": "googleapis/google-cloud-python",
+ "distribution_name": "google-cloud-automl",
+ "api_id": "automl.googleapis.com",
+ "requires_billing": true
+}
\ No newline at end of file
diff --git a/automl/CHANGELOG.md b/automl/CHANGELOG.md
new file mode 100644
index 000000000000..e991a616462e
--- /dev/null
+++ b/automl/CHANGELOG.md
@@ -0,0 +1,139 @@
+# Changelog
+
+[PyPI History][1]
+
+[1]: https://pypi.org/project/google-cloud-automl/#history
+
+## 0.5.0
+
+08-28-2019 14:07 PDT
+
+### Implementation Changes
+- Catch exceptions in GcsClient when a same name bucket already exists in a different project ([#9139](https://github.com/googleapis/google-cloud-python/pull/9139))
+- Log when LROs are kicked off. ([#9058](https://github.com/googleapis/google-cloud-python/pull/9058))
+- Throw a ValueError when an ambiguous display_name is used ([#9089](https://github.com/googleapis/google-cloud-python/pull/9089))
+- Remove send/recv msg size limit (via synth). ([#8944](https://github.com/googleapis/google-cloud-python/pull/8944))
+
+### New Features
+- Enable users to pass in Pandas Dataframe when calling import_data() and batch_predict() from AutoML Tables client ([#9116](https://github.com/googleapis/google-cloud-python/pull/9116))
+- Add support for documents (via synth). ([#9039](https://github.com/googleapis/google-cloud-python/pull/9039))
+- Add a TablesClient for automl-tables specific behavior. ([#8720](https://github.com/googleapis/google-cloud-python/pull/8720))
+- Add 'ClassificationEvaluationMetrics.display_name'/'BatchPredictResult.metadata'/'TableSpec.valid_row_count' (via synth) ([#9004](https://github.com/googleapis/google-cloud-python/pull/9004))
+
+### Documentation
+- Remove compatability badges from READMEs. ([#9035](https://github.com/googleapis/google-cloud-python/pull/9035))
+
+### Internal / Testing Changes
+- Remove CI for gh-pages, use googleapis.dev for api_core refs. ([#9085](https://github.com/googleapis/google-cloud-python/pull/9085))
+- Skip relevant system tests if in VPCSC ([#9111](https://github.com/googleapis/google-cloud-python/pull/9111))
+- Fix synth replace to add TablesClient. ([#9033](https://github.com/googleapis/google-cloud-python/pull/9033))
+- Update intersphinx mapping for requests. ([#8805](https://github.com/googleapis/google-cloud-python/pull/8805))
+
+## 0.4.0
+
+07-22-2019 17:39 PDT
+
+
+### New Features
+- Add support for 'TablesModelMetadata.{optimization_objective_recall_value,optimization_objective_precision_value}' (via synth). ([#8643](https://github.com/googleapis/google-cloud-python/pull/8643))
+- Add 'client_options' support, update list method docstrings (via synth). ([#8533](https://github.com/googleapis/google-cloud-python/pull/8533))
+
+### Dependencies
+- Bump minimum version for google-api-core to 1.14.0. ([#8709](https://github.com/googleapis/google-cloud-python/pull/8709))
+
+### Documentation
+- Link to googleapis.dev documentation in READMEs. ([#8705](https://github.com/googleapis/google-cloud-python/pull/8705))
+- Add compatibility check badges to READMEs. ([#8288](https://github.com/googleapis/google-cloud-python/pull/8288))
+
+### Internal / Testing Changes
+- Pin black version (via synth). ([#8573](https://github.com/googleapis/google-cloud-python/pull/8573))
+- Update Ruby package name (via synth). ([#8485](https://github.com/googleapis/google-cloud-python/pull/8485))
+- All: Add docs job to publish to googleapis.dev. ([#8464](https://github.com/googleapis/google-cloud-python/pull/8464))
+
+## 0.3.0
+
+06-20-2019 14:47 PDT
+
+
+### New Features
+- Add support for video object tracking (via synth). ([#8278](https://github.com/googleapis/google-cloud-python/pull/8278))
+
+### Documentation
+- Add proto files; add 'docs' session to 'nox'; update docstrings (via synth). ([#8029](https://github.com/googleapis/google-cloud-python/pull/8029))
+
+### Internal / Testing Changes
+- Suppress checking 'cov-fail-under' in nox default session (via synth). ([#8233](https://github.com/googleapis/google-cloud-python/pull/8233))
+- Fix coverage in 'types.py'. ([#8145](https://github.com/googleapis/google-cloud-python/pull/8145))
+- Blacken noxfile.py, setup.py (via synth). ([#8115](https://github.com/googleapis/google-cloud-python/pull/8115))
+- Add empty lines (via synth). ([#8048](https://github.com/googleapis/google-cloud-python/pull/8048))
+- Use alabaster theme everwhere. ([#8021](https://github.com/googleapis/google-cloud-python/pull/8021))
+- Include protos in synth. ([#8000](https://github.com/googleapis/google-cloud-python/pull/8000))
+
+## 0.2.0
+
+04-03-2019 09:16 PDT
+
+
+### Implementation Changes
+- Remove classifier for Python 3.4 for end-of-life. ([#7535](https://github.com/googleapis/google-cloud-python/pull/7535))
+- Sort generated classes (via synth). ([#7256](https://github.com/googleapis/google-cloud-python/pull/7256))
+- Protoc-generated serialization update. ([#7074](https://github.com/googleapis/google-cloud-python/pull/7074))
+
+### New Features
+- Video Classification, Text Extraction, Text Sentiment, Tables support. ([#7650](https://github.com/googleapis/google-cloud-python/pull/7650))
+
+### Documentation
+- Updated client library documentation URLs. ([#7307](https://github.com/googleapis/google-cloud-python/pull/7307))
+- AutoML: pick up stub docstring fix in GAPIC generator. ([#6964](https://github.com/googleapis/google-cloud-python/pull/6964))
+
+### Internal / Testing Changes
+- Update copyright headers
+
+## 0.1.2
+
+12-17-2018 16:27 PST
+
+
+### Implementation Changes
+- Add protoc-generated descriptor changes from updated .proto files. ([#6899](https://github.com/googleapis/google-cloud-python/pull/6899))
+- Import `iam.policy` from `google.api_core.iam.policy`. ([#6741](https://github.com/googleapis/google-cloud-python/pull/6741))
+- Pick up fixes in GAPIC generator. ([#6490](https://github.com/googleapis/google-cloud-python/pull/6490))
+- Assorted synth fixups / cleanups ([#6400](https://github.com/googleapis/google-cloud-python/pull/6400))
+- Fix client_info bug, update docstrings. ([#6404](https://github.com/googleapis/google-cloud-python/pull/6404))
+- Re-generate library using automl/synth.py ([#5972](https://github.com/googleapis/google-cloud-python/pull/5972))
+- Re-generate library using automl/synth.py ([#5946](https://github.com/googleapis/google-cloud-python/pull/5946))
+
+### Dependencies
+- Add 'mock' to unit test dependencies for autogen libs. ([#6402](https://github.com/googleapis/google-cloud-python/pull/6402))
+- Bump minimum `api_core`' version for all GAPIC libs to 1.4.1. ([#6391](https://github.com/googleapis/google-cloud-python/pull/6391))
+- Avoid broken 'google-common-apis 1.5.4' release. ([#6355](https://github.com/googleapis/google-cloud-python/pull/6355))
+
+### Documentation
+- Document Python 2 deprecation ([#6910](https://github.com/googleapis/google-cloud-python/pull/6910))
+- Harmonize / DRY 'README.rst' / 'docs/index.rst'. ([#6005](https://github.com/googleapis/google-cloud-python/pull/6005))
+- Replace links to '/stable/' with '/latest/'. ([#5901](https://github.com/googleapis/google-cloud-python/pull/5901))
+
+### Internal / Testing Changes
+- Update noxfile.
+- Blacken all gen'd libs ([#6792](https://github.com/googleapis/google-cloud-python/pull/6792))
+- Omit local deps ([#6701](https://github.com/googleapis/google-cloud-python/pull/6701))
+- Run black at end of synth.py ([#6698](https://github.com/googleapis/google-cloud-python/pull/6698))
+- Unblack automl gapic and protos.
+- Run Black on Generated libraries ([#6666](https://github.com/googleapis/google-cloud-python/pull/6666))
+- Add templates for flake8, coveragerc, noxfile, and black. ([#6642](https://github.com/googleapis/google-cloud-python/pull/6642))
+- Add synth metadata. ([#6561](https://github.com/googleapis/google-cloud-python/pull/6561))
+- Normalize use of support level badges ([#6159](https://github.com/googleapis/google-cloud-python/pull/6159))
+- Add / fix badges for PyPI / versions. ([#6158](https://github.com/googleapis/google-cloud-python/pull/6158))
+- Use new Nox ([#6175](https://github.com/googleapis/google-cloud-python/pull/6175))
+
+## 0.1.1
+
+### Documentation
+- Fix broken links (#5675)
+- bad trove classifier (#5648)
+
+## 0.1.0
+
+### New Features
+- Initial Release of AutoML v1beta1
+
diff --git a/automl/LICENSE b/automl/LICENSE
new file mode 100644
index 000000000000..a8ee855de2aa
--- /dev/null
+++ b/automl/LICENSE
@@ -0,0 +1,201 @@
+ Apache License
+ Version 2.0, January 2004
+ https://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [yyyy] [name of copyright owner]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
diff --git a/automl/MANIFEST.in b/automl/MANIFEST.in
new file mode 100644
index 000000000000..9cbf175afe6b
--- /dev/null
+++ b/automl/MANIFEST.in
@@ -0,0 +1,5 @@
+include README.rst LICENSE
+recursive-include google *.json *.proto
+recursive-include tests *
+global-exclude *.py[co]
+global-exclude __pycache__
diff --git a/automl/README.rst b/automl/README.rst
new file mode 100644
index 000000000000..ecf4052479a1
--- /dev/null
+++ b/automl/README.rst
@@ -0,0 +1,131 @@
+Python Client for Cloud AutoML API
+==================================
+
+|alpha| |pypi| |versions|
+
+The `Cloud AutoML API`_ is a suite of machine learning products that enables
+developers with limited machine learning expertise to train high-quality models
+specific to their business needs, by leveraging Google’s state-of-the-art
+transfer learning, and Neural Architecture Search technology.
+
+- `Client Library Documentation`_
+- `Product Documentation`_
+
+.. |alpha| image:: https://img.shields.io/badge/support-alpha-orange.svg
+ :target: https://github.com/googleapis/google-cloud-python/blob/master/README.rst#alpha-support
+.. |pypi| image:: https://img.shields.io/pypi/v/google-cloud-automl.svg
+ :target: https://pypi.org/project/google-cloud-automl/
+.. |versions| image:: https://img.shields.io/pypi/pyversions/google-cloud-automl.svg
+ :target: https://pypi.org/project/google-cloud-automl/
+.. _Cloud AutoML API: https://cloud.google.com/automl
+.. _Client Library Documentation: https://googleapis.dev/python/automl/latest
+.. _Product Documentation: https://cloud.google.com/automl
+
+Quick Start
+-----------
+
+In order to use this library, you first need to go through the following steps:
+
+1. `Select or create a Cloud Platform project.`_
+2. `Enable billing for your project.`_
+3. `Enable the Cloud AutoML API.`_
+4. `Setup Authentication.`_
+
+.. _Select or create a Cloud Platform project.: https://console.cloud.google.com/project
+.. _Enable billing for your project.: https://cloud.google.com/billing/docs/how-to/modify-project#enable_billing_for_a_project
+.. _Enable the Cloud AutoML API.: https://cloud.google.com/automl
+.. _Setup Authentication.: https://googleapis.dev/python/google-api-core/latest/auth.html
+
+Installation
+~~~~~~~~~~~~
+
+Install this library in a `virtualenv`_ using pip. `virtualenv`_ is a tool to
+create isolated Python environments. The basic problem it addresses is one of
+dependencies and versions, and indirectly permissions.
+
+With `virtualenv`_, it's possible to install this library without needing system
+install permissions, and without clashing with the installed system
+dependencies.
+
+.. _`virtualenv`: https://virtualenv.pypa.io/en/latest/
+
+
+Supported Python Versions
+^^^^^^^^^^^^^^^^^^^^^^^^^
+Python >= 3.5
+
+Deprecated Python Versions
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+Python == 2.7. Python 2.7 support will be removed on January 1, 2020.
+
+
+Mac/Linux
+^^^^^^^^^
+
+.. code-block:: console
+
+ pip install virtualenv
+ virtualenv
+ source /bin/activate
+ /bin/pip install google-cloud-automl
+
+
+Windows
+^^^^^^^
+
+.. code-block:: console
+
+ pip install virtualenv
+ virtualenv
+ \Scripts\activate
+ \Scripts\pip.exe install google-cloud-automl
+
+Example Usage
+~~~~~~~~~~~~~
+
+.. code-block:: python
+
+ from google.cloud.automl_v1beta1 import PredictionServiceClient
+
+ client = PredictionServiceClient()
+ model_path = client.model_path('my-project-123', 'us-central', 'model-name')
+ payload = {...}
+ params = {'foo': 1}
+ response = client.predict(model_path, payload, params=params)
+
+Next Steps
+~~~~~~~~~~
+
+- Read the `Client Library Documentation`_ for Cloud AutoML API
+ API to see other available methods on the client.
+- Read the `Product documentation`_ to learn
+ more about the product and see How-to Guides.
+
+Making & Testing Local Changes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you want to make changes to this library, here is how to set up your
+development environment:
+
+1. Make sure you have `virtualenv`_ installed and activated as shown above.
+2. Run the following one-time setup (it will be persisted in your virtualenv):
+
+ .. code-block:: console
+
+ pip install -r ../docs/requirements.txt
+ pip install -U nox mock pytest
+
+3. If you want to run all tests, you will need a billing-enabled
+ `GCP project`_, and a `service account`_ with access to the AutoML APIs.
+ Note: the first time the tests run in a new project it will take a _long_
+ time, on the order of 2-3 hours. This is one-time setup that will be skipped
+ in future runs.
+
+.. _service account: https://cloud.google.com/iam/docs/creating-managing-service-accounts
+.. _GCP project: https://cloud.google.com/resource-manager/docs/creating-managing-projects
+
+.. code-block:: console
+
+ export PROJECT_ID= GOOGLE_APPLICATION_CREDENTIALS=
+ nox
+
diff --git a/automl/docs/README.rst b/automl/docs/README.rst
new file mode 120000
index 000000000000..89a0106941ff
--- /dev/null
+++ b/automl/docs/README.rst
@@ -0,0 +1 @@
+../README.rst
\ No newline at end of file
diff --git a/automl/docs/changelog.md b/automl/docs/changelog.md
new file mode 120000
index 000000000000..04c99a55caae
--- /dev/null
+++ b/automl/docs/changelog.md
@@ -0,0 +1 @@
+../CHANGELOG.md
\ No newline at end of file
diff --git a/automl/docs/conf.py b/automl/docs/conf.py
new file mode 100644
index 000000000000..4eb565b9bf75
--- /dev/null
+++ b/automl/docs/conf.py
@@ -0,0 +1,363 @@
+# -*- coding: utf-8 -*-
+#
+# google-cloud-automl documentation build configuration file
+#
+# This file is execfile()d with the current directory set to its
+# containing dir.
+#
+# Note that not all possible configuration values are present in this
+# autogenerated file.
+#
+# All configuration values have a default; values that are commented out
+# serve to show the default.
+
+import sys
+import os
+import shlex
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+sys.path.insert(0, os.path.abspath(".."))
+
+__version__ = "0.1.0"
+
+# -- General configuration ------------------------------------------------
+
+# If your documentation needs a minimal Sphinx version, state it here.
+needs_sphinx = "1.6.3"
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+ "sphinx.ext.autodoc",
+ "sphinx.ext.autosummary",
+ "sphinx.ext.intersphinx",
+ "sphinx.ext.coverage",
+ "sphinx.ext.napoleon",
+ "sphinx.ext.todo",
+ "sphinx.ext.viewcode",
+]
+
+# autodoc/autosummary flags
+autoclass_content = "both"
+autodoc_default_flags = ["members"]
+autosummary_generate = True
+
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ["_templates"]
+
+# Allow markdown includes (so releases.md can include CHANGLEOG.md)
+# http://www.sphinx-doc.org/en/master/markdown.html
+source_parsers = {".md": "recommonmark.parser.CommonMarkParser"}
+
+# The suffix(es) of source filenames.
+# You can specify multiple suffix as a list of string:
+# source_suffix = ['.rst', '.md']
+source_suffix = [".rst", ".md"]
+
+# The encoding of source files.
+# source_encoding = 'utf-8-sig'
+
+# The master toctree document.
+master_doc = "index"
+
+# General information about the project.
+project = u"google-cloud-automl"
+copyright = u"2017, Google"
+author = u"Google APIs"
+
+# The version info for the project you're documenting, acts as replacement for
+# |version| and |release|, also used in various other places throughout the
+# built documents.
+#
+# The full version, including alpha/beta/rc tags.
+release = __version__
+# The short X.Y version.
+version = ".".join(release.split(".")[0:2])
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#
+# This is also used if you do content translation via gettext catalogs.
+# Usually you set "language" from the command line for these cases.
+language = None
+
+# There are two options for replacing |today|: either, you set today to some
+# non-false value, then it is used:
+# today = ''
+# Else, today_fmt is used as the format for a strftime call.
+# today_fmt = '%B %d, %Y'
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+exclude_patterns = ["_build"]
+
+# The reST default role (used for this markup: `text`) to use for all
+# documents.
+# default_role = None
+
+# If true, '()' will be appended to :func: etc. cross-reference text.
+# add_function_parentheses = True
+
+# If true, the current module name will be prepended to all description
+# unit titles (such as .. function::).
+# add_module_names = True
+
+# If true, sectionauthor and moduleauthor directives will be shown in the
+# output. They are ignored by default.
+# show_authors = False
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = "sphinx"
+
+# A list of ignored prefixes for module index sorting.
+# modindex_common_prefix = []
+
+# If true, keep warnings as "system message" paragraphs in the built documents.
+# keep_warnings = False
+
+# If true, `todo` and `todoList` produce output, else they produce nothing.
+todo_include_todos = True
+
+
+# -- Options for HTML output ----------------------------------------------
+
+# The theme to use for HTML and HTML Help pages. See the documentation for
+# a list of builtin themes.
+html_theme = "alabaster"
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further. For a list of options available for each theme, see the
+# documentation.
+html_theme_options = {
+ "description": "Google Cloud Client Libraries for Python",
+ "github_user": "googleapis",
+ "github_repo": "google-cloud-python",
+ "github_banner": True,
+ "font_family": "'Roboto', Georgia, sans",
+ "head_font_family": "'Roboto', Georgia, serif",
+ "code_font_family": "'Roboto Mono', 'Consolas', monospace",
+}
+
+# Add any paths that contain custom themes here, relative to this directory.
+# html_theme_path = []
+
+# The name for this set of Sphinx documents. If None, it defaults to
+# " v documentation".
+# html_title = None
+
+# A shorter title for the navigation bar. Default is the same as html_title.
+# html_short_title = None
+
+# The name of an image file (relative to this directory) to place at the top
+# of the sidebar.
+# html_logo = None
+
+# The name of an image file (within the static path) to use as favicon of the
+# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
+# pixels large.
+# html_favicon = None
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+# html_static_path = []
+
+# Add any extra paths that contain custom files (such as robots.txt or
+# .htaccess) here, relative to this directory. These files are copied
+# directly to the root of the documentation.
+# html_extra_path = []
+
+# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
+# using the given strftime format.
+# html_last_updated_fmt = '%b %d, %Y'
+
+# If true, SmartyPants will be used to convert quotes and dashes to
+# typographically correct entities.
+# html_use_smartypants = True
+
+# Custom sidebar templates, maps document names to template names.
+# html_sidebars = {}
+
+# Additional templates that should be rendered to pages, maps page names to
+# template names.
+# html_additional_pages = {}
+
+# If false, no module index is generated.
+# html_domain_indices = True
+
+# If false, no index is generated.
+# html_use_index = True
+
+# If true, the index is split into individual pages for each letter.
+# html_split_index = False
+
+# If true, links to the reST sources are added to the pages.
+# html_show_sourcelink = True
+
+# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
+# html_show_sphinx = True
+
+# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
+# html_show_copyright = True
+
+# If true, an OpenSearch description file will be output, and all pages will
+# contain a tag referring to it. The value of this option must be the
+# base URL from which the finished HTML is served.
+# html_use_opensearch = ''
+
+# This is the file name suffix for HTML files (e.g. ".xhtml").
+# html_file_suffix = None
+
+# Language to be used for generating the HTML full-text search index.
+# Sphinx supports the following languages:
+# 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja'
+# 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr'
+# html_search_language = 'en'
+
+# A dictionary with options for the search language support, empty by default.
+# Now only 'ja' uses this config value
+# html_search_options = {'type': 'default'}
+
+# The name of a javascript file (relative to the configuration directory) that
+# implements a search results scorer. If empty, the default will be used.
+# html_search_scorer = 'scorer.js'
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = "google-cloud-automl-doc"
+
+# -- Options for warnings ------------------------------------------------------
+
+
+suppress_warnings = [
+ # Temporarily suppress this to avoid "more than one target found for
+ # cross-reference" warning, which are intractable for us to avoid while in
+ # a mono-repo.
+ # See https://github.com/sphinx-doc/sphinx/blob
+ # /2a65ffeef5c107c19084fabdd706cdff3f52d93c/sphinx/domains/python.py#L843
+ "ref.python"
+]
+
+# -- Options for LaTeX output ---------------------------------------------
+
+latex_elements = {
+ # The paper size ('letterpaper' or 'a4paper').
+ #'papersize': 'letterpaper',
+ # The font size ('10pt', '11pt' or '12pt').
+ #'pointsize': '10pt',
+ # Additional stuff for the LaTeX preamble.
+ #'preamble': '',
+ # Latex figure (float) alignment
+ #'figure_align': 'htbp',
+}
+
+# Grouping the document tree into LaTeX files. List of tuples
+# (source start file, target name, title,
+# author, documentclass [howto, manual, or own class]).
+latex_documents = [
+ (
+ master_doc,
+ "google-cloud-automl.tex",
+ u"google-cloud-automl Documentation",
+ author,
+ "manual",
+ )
+]
+
+# The name of an image file (relative to this directory) to place at the top of
+# the title page.
+# latex_logo = None
+
+# For "manual" documents, if this is true, then toplevel headings are parts,
+# not chapters.
+# latex_use_parts = False
+
+# If true, show page references after internal links.
+# latex_show_pagerefs = False
+
+# If true, show URL addresses after external links.
+# latex_show_urls = False
+
+# Documents to append as an appendix to all manuals.
+# latex_appendices = []
+
+# If false, no module index is generated.
+# latex_domain_indices = True
+
+
+# -- Options for manual page output ---------------------------------------
+
+# One entry per manual page. List of tuples
+# (source start file, name, description, authors, manual section).
+man_pages = [
+ (
+ master_doc,
+ "google-cloud-automl",
+ u"google-cloud-automl Documentation",
+ [author],
+ 1,
+ )
+]
+
+# If true, show URL addresses after external links.
+# man_show_urls = False
+
+
+# -- Options for Texinfo output -------------------------------------------
+
+# Grouping the document tree into Texinfo files. List of tuples
+# (source start file, target name, title, author,
+# dir menu entry, description, category)
+texinfo_documents = [
+ (
+ master_doc,
+ "google-cloud-automl",
+ u"google-cloud-automl Documentation",
+ author,
+ "google-cloud-automl",
+ "GAPIC library for the {metadata.shortName} v1beta1 service",
+ "APIs",
+ )
+]
+
+# Documents to append as an appendix to all manuals.
+# texinfo_appendices = []
+
+# If false, no module index is generated.
+# texinfo_domain_indices = True
+
+# How to display URL addresses: 'footnote', 'no', or 'inline'.
+# texinfo_show_urls = 'footnote'
+
+# If true, do not generate a @detailmenu in the "Top" node's menu.
+# texinfo_no_detailmenu = False
+
+
+# Example configuration for intersphinx: refer to the Python standard library.
+intersphinx_mapping = {
+ "python": ("http://python.readthedocs.org/en/latest/", None),
+ "gax": ("https://gax-python.readthedocs.org/en/latest/", None),
+ "google-auth": ("https://google-auth.readthedocs.io/en/stable", None),
+ "google-gax": ("https://gax-python.readthedocs.io/en/latest/", None),
+ "google.api_core": ("https://googleapis.dev/python/google-api-core/latest", None),
+ "grpc": ("https://grpc.io/grpc/python/", None),
+ "requests": ("https://2.python-requests.org/en/master/", None),
+ "fastavro": ("https://fastavro.readthedocs.io/en/stable/", None),
+ "pandas": ("https://pandas.pydata.org/pandas-docs/stable/", None),
+}
+
+
+# Napoleon settings
+napoleon_google_docstring = True
+napoleon_numpy_docstring = True
+napoleon_include_private_with_doc = False
+napoleon_include_special_with_doc = True
+napoleon_use_admonition_for_examples = False
+napoleon_use_admonition_for_notes = False
+napoleon_use_admonition_for_references = False
+napoleon_use_ivar = False
+napoleon_use_param = True
+napoleon_use_rtype = True
diff --git a/automl/docs/gapic/v1beta1/api.rst b/automl/docs/gapic/v1beta1/api.rst
new file mode 100644
index 000000000000..268baa5b798c
--- /dev/null
+++ b/automl/docs/gapic/v1beta1/api.rst
@@ -0,0 +1,6 @@
+Client for Cloud AutoML API
+===========================
+
+.. automodule:: google.cloud.automl_v1beta1
+ :members:
+ :inherited-members:
\ No newline at end of file
diff --git a/automl/docs/gapic/v1beta1/tables.rst b/automl/docs/gapic/v1beta1/tables.rst
new file mode 100644
index 000000000000..54ed6a203805
--- /dev/null
+++ b/automl/docs/gapic/v1beta1/tables.rst
@@ -0,0 +1,5 @@
+A tables-specific client for AutoML
+===================================
+
+.. automodule:: google.cloud.automl_v1beta1.tables.tables_client
+ :members:
diff --git a/automl/docs/gapic/v1beta1/types.rst b/automl/docs/gapic/v1beta1/types.rst
new file mode 100644
index 000000000000..27ce6644bcc5
--- /dev/null
+++ b/automl/docs/gapic/v1beta1/types.rst
@@ -0,0 +1,5 @@
+Types for Cloud AutoML API Client
+=================================
+
+.. automodule:: google.cloud.automl_v1beta1.types
+ :members:
\ No newline at end of file
diff --git a/automl/docs/index.rst b/automl/docs/index.rst
new file mode 100644
index 000000000000..01f577642cb1
--- /dev/null
+++ b/automl/docs/index.rst
@@ -0,0 +1,22 @@
+.. include:: README.rst
+
+
+Api Reference
+-------------
+.. toctree::
+ :maxdepth: 2
+
+ gapic/v1beta1/api
+ gapic/v1beta1/types
+ gapic/v1beta1/tables
+
+
+Changelog
+---------
+
+For a list of all ``google-cloud-automl`` releases:
+
+.. toctree::
+ :maxdepth: 2
+
+ changelog
diff --git a/automl/google/__init__.py b/automl/google/__init__.py
new file mode 100644
index 000000000000..dd3a9f485275
--- /dev/null
+++ b/automl/google/__init__.py
@@ -0,0 +1,22 @@
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+try:
+ import pkg_resources
+
+ pkg_resources.declare_namespace(__name__)
+except ImportError:
+ import pkgutil
+
+ __path__ = pkgutil.extend_path(__path__, __name__)
diff --git a/automl/google/cloud/__init__.py b/automl/google/cloud/__init__.py
new file mode 100644
index 000000000000..dd3a9f485275
--- /dev/null
+++ b/automl/google/cloud/__init__.py
@@ -0,0 +1,22 @@
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+try:
+ import pkg_resources
+
+ pkg_resources.declare_namespace(__name__)
+except ImportError:
+ import pkgutil
+
+ __path__ = pkgutil.extend_path(__path__, __name__)
diff --git a/automl/google/cloud/automl.py b/automl/google/cloud/automl.py
new file mode 100644
index 000000000000..77528b3d1dbf
--- /dev/null
+++ b/automl/google/cloud/automl.py
@@ -0,0 +1,26 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from __future__ import absolute_import
+
+from google.cloud.automl_v1beta1 import AutoMlClient
+from google.cloud.automl_v1beta1 import PredictionServiceClient
+from google.cloud.automl_v1beta1 import enums
+from google.cloud.automl_v1beta1 import types
+
+
+__all__ = ("enums", "types", "AutoMlClient", "PredictionServiceClient")
diff --git a/automl/google/cloud/automl_v1beta1/__init__.py b/automl/google/cloud/automl_v1beta1/__init__.py
new file mode 100644
index 000000000000..113b55279b85
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/__init__.py
@@ -0,0 +1,53 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from __future__ import absolute_import
+
+from google.cloud.automl_v1beta1 import types
+from google.cloud.automl_v1beta1.gapic import auto_ml_client
+from google.cloud.automl_v1beta1.gapic import enums
+from google.cloud.automl_v1beta1.gapic import prediction_service_client
+from google.cloud.automl_v1beta1.tables import gcs_client
+from google.cloud.automl_v1beta1.tables import tables_client
+
+
+class TablesClient(tables_client.TablesClient):
+ __doc__ = tables_client.TablesClient.__doc__
+
+
+class AutoMlClient(auto_ml_client.AutoMlClient):
+ __doc__ = auto_ml_client.AutoMlClient.__doc__
+ enums = enums
+
+
+class PredictionServiceClient(prediction_service_client.PredictionServiceClient):
+ __doc__ = prediction_service_client.PredictionServiceClient.__doc__
+ enums = enums
+
+
+class GcsClient(gcs_client.GcsClient):
+ __doc__ = gcs_client.GcsClient.__doc__
+
+
+__all__ = (
+ "enums",
+ "types",
+ "AutoMlClient",
+ "PredictionServiceClient",
+ "TablesClient",
+ "GcsClient",
+)
diff --git a/automl/google/cloud/automl_v1beta1/gapic/__init__.py b/automl/google/cloud/automl_v1beta1/gapic/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/automl/google/cloud/automl_v1beta1/gapic/auto_ml_client.py b/automl/google/cloud/automl_v1beta1/gapic/auto_ml_client.py
new file mode 100644
index 000000000000..e0e10d8735e4
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/gapic/auto_ml_client.py
@@ -0,0 +1,2457 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Accesses the google.cloud.automl.v1beta1 AutoMl API."""
+
+import functools
+import pkg_resources
+import warnings
+
+from google.oauth2 import service_account
+import google.api_core.client_options
+import google.api_core.gapic_v1.client_info
+import google.api_core.gapic_v1.config
+import google.api_core.gapic_v1.method
+import google.api_core.gapic_v1.routing_header
+import google.api_core.grpc_helpers
+import google.api_core.operation
+import google.api_core.operations_v1
+import google.api_core.page_iterator
+import google.api_core.path_template
+import google.api_core.protobuf_helpers
+import grpc
+
+from google.cloud.automl_v1beta1.gapic import auto_ml_client_config
+from google.cloud.automl_v1beta1.gapic import enums
+from google.cloud.automl_v1beta1.gapic.transports import auto_ml_grpc_transport
+from google.cloud.automl_v1beta1.proto import annotation_spec_pb2
+from google.cloud.automl_v1beta1.proto import column_spec_pb2
+from google.cloud.automl_v1beta1.proto import dataset_pb2
+from google.cloud.automl_v1beta1.proto import image_pb2
+from google.cloud.automl_v1beta1.proto import io_pb2
+from google.cloud.automl_v1beta1.proto import model_evaluation_pb2
+from google.cloud.automl_v1beta1.proto import model_pb2
+from google.cloud.automl_v1beta1.proto import operations_pb2 as proto_operations_pb2
+from google.cloud.automl_v1beta1.proto import service_pb2
+from google.cloud.automl_v1beta1.proto import service_pb2_grpc
+from google.cloud.automl_v1beta1.proto import table_spec_pb2
+from google.longrunning import operations_pb2 as longrunning_operations_pb2
+from google.protobuf import empty_pb2
+from google.protobuf import field_mask_pb2
+
+
+_GAPIC_LIBRARY_VERSION = pkg_resources.get_distribution("google-cloud-automl").version
+
+
+class AutoMlClient(object):
+ """
+ AutoML Server API.
+
+ The resource names are assigned by the server. The server never reuses
+ names that it has created after the resources with those names are
+ deleted.
+
+ An ID of a resource is the last element of the item's resource name. For
+ ``projects/{project_id}/locations/{location_id}/datasets/{dataset_id}``,
+ then the id for the item is ``{dataset_id}``.
+
+ Currently the only supported ``location_id`` is "us-central1".
+
+ On any input that is documented to expect a string parameter in
+ snake\_case or kebab-case, either of those cases is accepted.
+ """
+
+ SERVICE_ADDRESS = "automl.googleapis.com:443"
+ """The default address of the service."""
+
+ # The name of the interface for this client. This is the key used to
+ # find the method configuration in the client_config dictionary.
+ _INTERFACE_NAME = "google.cloud.automl.v1beta1.AutoMl"
+
+ @classmethod
+ def from_service_account_file(cls, filename, *args, **kwargs):
+ """Creates an instance of this client using the provided credentials
+ file.
+
+ Args:
+ filename (str): The path to the service account private key json
+ file.
+ args: Additional arguments to pass to the constructor.
+ kwargs: Additional arguments to pass to the constructor.
+
+ Returns:
+ AutoMlClient: The constructed client.
+ """
+ credentials = service_account.Credentials.from_service_account_file(filename)
+ kwargs["credentials"] = credentials
+ return cls(*args, **kwargs)
+
+ from_service_account_json = from_service_account_file
+
+ @classmethod
+ def annotation_spec_path(cls, project, location, dataset, annotation_spec):
+ """Return a fully-qualified annotation_spec string."""
+ return google.api_core.path_template.expand(
+ "projects/{project}/locations/{location}/datasets/{dataset}/annotationSpecs/{annotation_spec}",
+ project=project,
+ location=location,
+ dataset=dataset,
+ annotation_spec=annotation_spec,
+ )
+
+ @classmethod
+ def column_spec_path(cls, project, location, dataset, table_spec, column_spec):
+ """Return a fully-qualified column_spec string."""
+ return google.api_core.path_template.expand(
+ "projects/{project}/locations/{location}/datasets/{dataset}/tableSpecs/{table_spec}/columnSpecs/{column_spec}",
+ project=project,
+ location=location,
+ dataset=dataset,
+ table_spec=table_spec,
+ column_spec=column_spec,
+ )
+
+ @classmethod
+ def dataset_path(cls, project, location, dataset):
+ """Return a fully-qualified dataset string."""
+ return google.api_core.path_template.expand(
+ "projects/{project}/locations/{location}/datasets/{dataset}",
+ project=project,
+ location=location,
+ dataset=dataset,
+ )
+
+ @classmethod
+ def location_path(cls, project, location):
+ """Return a fully-qualified location string."""
+ return google.api_core.path_template.expand(
+ "projects/{project}/locations/{location}",
+ project=project,
+ location=location,
+ )
+
+ @classmethod
+ def model_path(cls, project, location, model):
+ """Return a fully-qualified model string."""
+ return google.api_core.path_template.expand(
+ "projects/{project}/locations/{location}/models/{model}",
+ project=project,
+ location=location,
+ model=model,
+ )
+
+ @classmethod
+ def model_evaluation_path(cls, project, location, model, model_evaluation):
+ """Return a fully-qualified model_evaluation string."""
+ return google.api_core.path_template.expand(
+ "projects/{project}/locations/{location}/models/{model}/modelEvaluations/{model_evaluation}",
+ project=project,
+ location=location,
+ model=model,
+ model_evaluation=model_evaluation,
+ )
+
+ @classmethod
+ def table_spec_path(cls, project, location, dataset, table_spec):
+ """Return a fully-qualified table_spec string."""
+ return google.api_core.path_template.expand(
+ "projects/{project}/locations/{location}/datasets/{dataset}/tableSpecs/{table_spec}",
+ project=project,
+ location=location,
+ dataset=dataset,
+ table_spec=table_spec,
+ )
+
+ def __init__(
+ self,
+ transport=None,
+ channel=None,
+ credentials=None,
+ client_config=None,
+ client_info=None,
+ client_options=None,
+ ):
+ """Constructor.
+
+ Args:
+ transport (Union[~.AutoMlGrpcTransport,
+ Callable[[~.Credentials, type], ~.AutoMlGrpcTransport]): A transport
+ instance, responsible for actually making the API calls.
+ The default transport uses the gRPC protocol.
+ This argument may also be a callable which returns a
+ transport instance. Callables will be sent the credentials
+ as the first argument and the default transport class as
+ the second argument.
+ channel (grpc.Channel): DEPRECATED. A ``Channel`` instance
+ through which to make calls. This argument is mutually exclusive
+ with ``credentials``; providing both will raise an exception.
+ credentials (google.auth.credentials.Credentials): The
+ authorization credentials to attach to requests. These
+ credentials identify this application to the service. If none
+ are specified, the client will attempt to ascertain the
+ credentials from the environment.
+ This argument is mutually exclusive with providing a
+ transport instance to ``transport``; doing so will raise
+ an exception.
+ client_config (dict): DEPRECATED. A dictionary of call options for
+ each method. If not specified, the default configuration is used.
+ client_info (google.api_core.gapic_v1.client_info.ClientInfo):
+ The client info used to send a user-agent string along with
+ API requests. If ``None``, then default info will be used.
+ Generally, you only need to set this if you're developing
+ your own client library.
+ client_options (Union[dict, google.api_core.client_options.ClientOptions]):
+ Client options used to set user options on the client. API Endpoint
+ should be set through client_options.
+ """
+ # Raise deprecation warnings for things we want to go away.
+ if client_config is not None:
+ warnings.warn(
+ "The `client_config` argument is deprecated.",
+ PendingDeprecationWarning,
+ stacklevel=2,
+ )
+ else:
+ client_config = auto_ml_client_config.config
+
+ if channel:
+ warnings.warn(
+ "The `channel` argument is deprecated; use " "`transport` instead.",
+ PendingDeprecationWarning,
+ stacklevel=2,
+ )
+
+ api_endpoint = self.SERVICE_ADDRESS
+ if client_options:
+ if type(client_options) == dict:
+ client_options = google.api_core.client_options.from_dict(
+ client_options
+ )
+ if client_options.api_endpoint:
+ api_endpoint = client_options.api_endpoint
+
+ # Instantiate the transport.
+ # The transport is responsible for handling serialization and
+ # deserialization and actually sending data to the service.
+ if transport:
+ if callable(transport):
+ self.transport = transport(
+ credentials=credentials,
+ default_class=auto_ml_grpc_transport.AutoMlGrpcTransport,
+ address=api_endpoint,
+ )
+ else:
+ if credentials:
+ raise ValueError(
+ "Received both a transport instance and "
+ "credentials; these are mutually exclusive."
+ )
+ self.transport = transport
+ else:
+ self.transport = auto_ml_grpc_transport.AutoMlGrpcTransport(
+ address=api_endpoint, channel=channel, credentials=credentials
+ )
+
+ if client_info is None:
+ client_info = google.api_core.gapic_v1.client_info.ClientInfo(
+ gapic_version=_GAPIC_LIBRARY_VERSION
+ )
+ else:
+ client_info.gapic_version = _GAPIC_LIBRARY_VERSION
+ self._client_info = client_info
+
+ # Parse out the default settings for retry and timeout for each RPC
+ # from the client configuration.
+ # (Ordinarily, these are the defaults specified in the `*_config.py`
+ # file next to this one.)
+ self._method_configs = google.api_core.gapic_v1.config.parse_method_configs(
+ client_config["interfaces"][self._INTERFACE_NAME]
+ )
+
+ # Save a dictionary of cached API call functions.
+ # These are the actual callables which invoke the proper
+ # transport methods, wrapped with `wrap_method` to add retry,
+ # timeout, and the like.
+ self._inner_api_calls = {}
+
+ # Service calls
+ def create_dataset(
+ self,
+ parent,
+ dataset,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Creates a dataset.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> client = automl_v1beta1.AutoMlClient()
+ >>>
+ >>> parent = client.location_path('[PROJECT]', '[LOCATION]')
+ >>>
+ >>> # TODO: Initialize `dataset`:
+ >>> dataset = {}
+ >>>
+ >>> response = client.create_dataset(parent, dataset)
+
+ Args:
+ parent (str): The resource name of the project to create the dataset for.
+ dataset (Union[dict, ~google.cloud.automl_v1beta1.types.Dataset]): The dataset to create.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.automl_v1beta1.types.Dataset`
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types.Dataset` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "create_dataset" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "create_dataset"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.create_dataset,
+ default_retry=self._method_configs["CreateDataset"].retry,
+ default_timeout=self._method_configs["CreateDataset"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = service_pb2.CreateDatasetRequest(parent=parent, dataset=dataset)
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("parent", parent)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ return self._inner_api_calls["create_dataset"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+
+ def update_dataset(
+ self,
+ dataset,
+ update_mask=None,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Updates a dataset.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> client = automl_v1beta1.AutoMlClient()
+ >>>
+ >>> # TODO: Initialize `dataset`:
+ >>> dataset = {}
+ >>>
+ >>> response = client.update_dataset(dataset)
+
+ Args:
+ dataset (Union[dict, ~google.cloud.automl_v1beta1.types.Dataset]): The dataset which replaces the resource on the server.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.automl_v1beta1.types.Dataset`
+ update_mask (Union[dict, ~google.cloud.automl_v1beta1.types.FieldMask]): The update mask applies to the resource.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.automl_v1beta1.types.FieldMask`
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types.Dataset` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "update_dataset" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "update_dataset"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.update_dataset,
+ default_retry=self._method_configs["UpdateDataset"].retry,
+ default_timeout=self._method_configs["UpdateDataset"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = service_pb2.UpdateDatasetRequest(
+ dataset=dataset, update_mask=update_mask
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("dataset.name", dataset.name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ return self._inner_api_calls["update_dataset"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+
+ def get_dataset(
+ self,
+ name,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Gets a dataset.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> client = automl_v1beta1.AutoMlClient()
+ >>>
+ >>> name = client.dataset_path('[PROJECT]', '[LOCATION]', '[DATASET]')
+ >>>
+ >>> response = client.get_dataset(name)
+
+ Args:
+ name (str): The resource name of the dataset to retrieve.
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types.Dataset` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "get_dataset" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "get_dataset"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.get_dataset,
+ default_retry=self._method_configs["GetDataset"].retry,
+ default_timeout=self._method_configs["GetDataset"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = service_pb2.GetDatasetRequest(name=name)
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("name", name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ return self._inner_api_calls["get_dataset"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+
+ def list_datasets(
+ self,
+ parent,
+ filter_=None,
+ page_size=None,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Lists datasets in a project.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> client = automl_v1beta1.AutoMlClient()
+ >>>
+ >>> parent = client.location_path('[PROJECT]', '[LOCATION]')
+ >>>
+ >>> # Iterate over all results
+ >>> for element in client.list_datasets(parent):
+ ... # process element
+ ... pass
+ >>>
+ >>>
+ >>> # Alternatively:
+ >>>
+ >>> # Iterate over results one page at a time
+ >>> for page in client.list_datasets(parent).pages:
+ ... for element in page:
+ ... # process element
+ ... pass
+
+ Args:
+ parent (str): The resource name of the project from which to list datasets.
+ filter_ (str): An expression for filtering the results of the request.
+
+ - ``dataset_metadata`` - for existence of the case (e.g.
+ image\_classification\_dataset\_metadata:\*). Some examples of using
+ the filter are:
+
+ - ``translation_dataset_metadata:*`` --> The dataset has
+ translation\_dataset\_metadata.
+ page_size (int): The maximum number of resources contained in the
+ underlying API response. If page streaming is performed per-
+ resource, this parameter does not affect the return value. If page
+ streaming is performed per-page, this determines the maximum number
+ of resources in a page.
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.api_core.page_iterator.PageIterator` instance.
+ An iterable of :class:`~google.cloud.automl_v1beta1.types.Dataset` instances.
+ You can also iterate over the pages of the response
+ using its `pages` property.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "list_datasets" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "list_datasets"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.list_datasets,
+ default_retry=self._method_configs["ListDatasets"].retry,
+ default_timeout=self._method_configs["ListDatasets"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = service_pb2.ListDatasetsRequest(
+ parent=parent, filter=filter_, page_size=page_size
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("parent", parent)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ iterator = google.api_core.page_iterator.GRPCIterator(
+ client=None,
+ method=functools.partial(
+ self._inner_api_calls["list_datasets"],
+ retry=retry,
+ timeout=timeout,
+ metadata=metadata,
+ ),
+ request=request,
+ items_field="datasets",
+ request_token_field="page_token",
+ response_token_field="next_page_token",
+ )
+ return iterator
+
+ def delete_dataset(
+ self,
+ name,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Deletes a dataset and all of its contents. Returns empty response in the
+ ``response`` field when it completes, and ``delete_details`` in the
+ ``metadata`` field.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> client = automl_v1beta1.AutoMlClient()
+ >>>
+ >>> name = client.dataset_path('[PROJECT]', '[LOCATION]', '[DATASET]')
+ >>>
+ >>> response = client.delete_dataset(name)
+ >>>
+ >>> def callback(operation_future):
+ ... # Handle result.
+ ... result = operation_future.result()
+ >>>
+ >>> response.add_done_callback(callback)
+ >>>
+ >>> # Handle metadata.
+ >>> metadata = response.metadata()
+
+ Args:
+ name (str): The resource name of the dataset to delete.
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types._OperationFuture` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "delete_dataset" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "delete_dataset"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.delete_dataset,
+ default_retry=self._method_configs["DeleteDataset"].retry,
+ default_timeout=self._method_configs["DeleteDataset"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = service_pb2.DeleteDatasetRequest(name=name)
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("name", name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ operation = self._inner_api_calls["delete_dataset"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+ return google.api_core.operation.from_gapic(
+ operation,
+ self.transport._operations_client,
+ empty_pb2.Empty,
+ metadata_type=proto_operations_pb2.OperationMetadata,
+ )
+
+ def import_data(
+ self,
+ name,
+ input_config,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Imports data into a dataset. For Tables this method can only be called
+ on an empty Dataset.
+
+ For Tables:
+
+ - A ``schema_inference_version`` parameter must be explicitly set.
+ Returns an empty response in the ``response`` field when it
+ completes.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> client = automl_v1beta1.AutoMlClient()
+ >>>
+ >>> name = client.dataset_path('[PROJECT]', '[LOCATION]', '[DATASET]')
+ >>>
+ >>> # TODO: Initialize `input_config`:
+ >>> input_config = {}
+ >>>
+ >>> response = client.import_data(name, input_config)
+ >>>
+ >>> def callback(operation_future):
+ ... # Handle result.
+ ... result = operation_future.result()
+ >>>
+ >>> response.add_done_callback(callback)
+ >>>
+ >>> # Handle metadata.
+ >>> metadata = response.metadata()
+
+ Args:
+ name (str): Required. Dataset name. Dataset must already exist. All imported
+ annotations and examples will be added.
+ input_config (Union[dict, ~google.cloud.automl_v1beta1.types.InputConfig]): Required. The desired input location and its domain specific semantics,
+ if any.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.automl_v1beta1.types.InputConfig`
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types._OperationFuture` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "import_data" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "import_data"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.import_data,
+ default_retry=self._method_configs["ImportData"].retry,
+ default_timeout=self._method_configs["ImportData"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = service_pb2.ImportDataRequest(name=name, input_config=input_config)
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("name", name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ operation = self._inner_api_calls["import_data"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+ return google.api_core.operation.from_gapic(
+ operation,
+ self.transport._operations_client,
+ empty_pb2.Empty,
+ metadata_type=proto_operations_pb2.OperationMetadata,
+ )
+
+ def export_data(
+ self,
+ name,
+ output_config,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Exports dataset's data to the provided output location. Returns an empty
+ response in the ``response`` field when it completes.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> client = automl_v1beta1.AutoMlClient()
+ >>>
+ >>> name = client.dataset_path('[PROJECT]', '[LOCATION]', '[DATASET]')
+ >>>
+ >>> # TODO: Initialize `output_config`:
+ >>> output_config = {}
+ >>>
+ >>> response = client.export_data(name, output_config)
+ >>>
+ >>> def callback(operation_future):
+ ... # Handle result.
+ ... result = operation_future.result()
+ >>>
+ >>> response.add_done_callback(callback)
+ >>>
+ >>> # Handle metadata.
+ >>> metadata = response.metadata()
+
+ Args:
+ name (str): Required. The resource name of the dataset.
+ output_config (Union[dict, ~google.cloud.automl_v1beta1.types.OutputConfig]): Required. The desired output location.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.automl_v1beta1.types.OutputConfig`
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types._OperationFuture` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "export_data" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "export_data"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.export_data,
+ default_retry=self._method_configs["ExportData"].retry,
+ default_timeout=self._method_configs["ExportData"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = service_pb2.ExportDataRequest(name=name, output_config=output_config)
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("name", name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ operation = self._inner_api_calls["export_data"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+ return google.api_core.operation.from_gapic(
+ operation,
+ self.transport._operations_client,
+ empty_pb2.Empty,
+ metadata_type=proto_operations_pb2.OperationMetadata,
+ )
+
+ def create_model(
+ self,
+ parent,
+ model,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Creates a model. Returns a Model in the ``response`` field when it
+ completes. When you create a model, several model evaluations are
+ created for it: a global evaluation, and one evaluation for each
+ annotation spec.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> client = automl_v1beta1.AutoMlClient()
+ >>>
+ >>> parent = client.location_path('[PROJECT]', '[LOCATION]')
+ >>>
+ >>> # TODO: Initialize `model`:
+ >>> model = {}
+ >>>
+ >>> response = client.create_model(parent, model)
+ >>>
+ >>> def callback(operation_future):
+ ... # Handle result.
+ ... result = operation_future.result()
+ >>>
+ >>> response.add_done_callback(callback)
+ >>>
+ >>> # Handle metadata.
+ >>> metadata = response.metadata()
+
+ Args:
+ parent (str): Resource name of the parent project where the model is being created.
+ model (Union[dict, ~google.cloud.automl_v1beta1.types.Model]): The model to create.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.automl_v1beta1.types.Model`
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types._OperationFuture` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "create_model" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "create_model"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.create_model,
+ default_retry=self._method_configs["CreateModel"].retry,
+ default_timeout=self._method_configs["CreateModel"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = service_pb2.CreateModelRequest(parent=parent, model=model)
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("parent", parent)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ operation = self._inner_api_calls["create_model"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+ return google.api_core.operation.from_gapic(
+ operation,
+ self.transport._operations_client,
+ model_pb2.Model,
+ metadata_type=proto_operations_pb2.OperationMetadata,
+ )
+
+ def get_model(
+ self,
+ name,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Gets a model.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> client = automl_v1beta1.AutoMlClient()
+ >>>
+ >>> name = client.model_path('[PROJECT]', '[LOCATION]', '[MODEL]')
+ >>>
+ >>> response = client.get_model(name)
+
+ Args:
+ name (str): Resource name of the model.
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types.Model` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "get_model" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "get_model"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.get_model,
+ default_retry=self._method_configs["GetModel"].retry,
+ default_timeout=self._method_configs["GetModel"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = service_pb2.GetModelRequest(name=name)
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("name", name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ return self._inner_api_calls["get_model"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+
+ def list_models(
+ self,
+ parent,
+ filter_=None,
+ page_size=None,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Lists models.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> client = automl_v1beta1.AutoMlClient()
+ >>>
+ >>> parent = client.location_path('[PROJECT]', '[LOCATION]')
+ >>>
+ >>> # Iterate over all results
+ >>> for element in client.list_models(parent):
+ ... # process element
+ ... pass
+ >>>
+ >>>
+ >>> # Alternatively:
+ >>>
+ >>> # Iterate over results one page at a time
+ >>> for page in client.list_models(parent).pages:
+ ... for element in page:
+ ... # process element
+ ... pass
+
+ Args:
+ parent (str): Resource name of the project, from which to list the models.
+ filter_ (str): An expression for filtering the results of the request.
+
+ - ``model_metadata`` - for existence of the case (e.g.
+ video\_classification\_model\_metadata:\*).
+
+ - ``dataset_id`` - for = or !=. Some examples of using the filter are:
+
+ - ``image_classification_model_metadata:*`` --> The model has
+ image\_classification\_model\_metadata.
+
+ - ``dataset_id=5`` --> The model was created from a dataset with ID 5.
+ page_size (int): The maximum number of resources contained in the
+ underlying API response. If page streaming is performed per-
+ resource, this parameter does not affect the return value. If page
+ streaming is performed per-page, this determines the maximum number
+ of resources in a page.
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.api_core.page_iterator.PageIterator` instance.
+ An iterable of :class:`~google.cloud.automl_v1beta1.types.Model` instances.
+ You can also iterate over the pages of the response
+ using its `pages` property.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "list_models" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "list_models"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.list_models,
+ default_retry=self._method_configs["ListModels"].retry,
+ default_timeout=self._method_configs["ListModels"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = service_pb2.ListModelsRequest(
+ parent=parent, filter=filter_, page_size=page_size
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("parent", parent)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ iterator = google.api_core.page_iterator.GRPCIterator(
+ client=None,
+ method=functools.partial(
+ self._inner_api_calls["list_models"],
+ retry=retry,
+ timeout=timeout,
+ metadata=metadata,
+ ),
+ request=request,
+ items_field="model",
+ request_token_field="page_token",
+ response_token_field="next_page_token",
+ )
+ return iterator
+
+ def delete_model(
+ self,
+ name,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Deletes a model. Returns ``google.protobuf.Empty`` in the ``response``
+ field when it completes, and ``delete_details`` in the ``metadata``
+ field.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> client = automl_v1beta1.AutoMlClient()
+ >>>
+ >>> name = client.model_path('[PROJECT]', '[LOCATION]', '[MODEL]')
+ >>>
+ >>> response = client.delete_model(name)
+ >>>
+ >>> def callback(operation_future):
+ ... # Handle result.
+ ... result = operation_future.result()
+ >>>
+ >>> response.add_done_callback(callback)
+ >>>
+ >>> # Handle metadata.
+ >>> metadata = response.metadata()
+
+ Args:
+ name (str): Resource name of the model being deleted.
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types._OperationFuture` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "delete_model" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "delete_model"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.delete_model,
+ default_retry=self._method_configs["DeleteModel"].retry,
+ default_timeout=self._method_configs["DeleteModel"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = service_pb2.DeleteModelRequest(name=name)
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("name", name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ operation = self._inner_api_calls["delete_model"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+ return google.api_core.operation.from_gapic(
+ operation,
+ self.transport._operations_client,
+ empty_pb2.Empty,
+ metadata_type=proto_operations_pb2.OperationMetadata,
+ )
+
+ def deploy_model(
+ self,
+ name,
+ image_object_detection_model_deployment_metadata=None,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Deploys a model. If a model is already deployed, deploying it with the
+ same parameters has no effect. Deploying with different parametrs (as
+ e.g. changing
+
+ ``node_number``) will reset the deployment state without pausing the
+ model's availability.
+
+ Only applicable for Text Classification, Image Object Detection and
+ Tables; all other domains manage deployment automatically.
+
+ Returns an empty response in the ``response`` field when it completes.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> client = automl_v1beta1.AutoMlClient()
+ >>>
+ >>> name = client.model_path('[PROJECT]', '[LOCATION]', '[MODEL]')
+ >>>
+ >>> response = client.deploy_model(name)
+ >>>
+ >>> def callback(operation_future):
+ ... # Handle result.
+ ... result = operation_future.result()
+ >>>
+ >>> response.add_done_callback(callback)
+ >>>
+ >>> # Handle metadata.
+ >>> metadata = response.metadata()
+
+ Args:
+ name (str): Resource name of the model to deploy.
+ image_object_detection_model_deployment_metadata (Union[dict, ~google.cloud.automl_v1beta1.types.ImageObjectDetectionModelDeploymentMetadata]): Model deployment metadata specific to Image Object Detection.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.automl_v1beta1.types.ImageObjectDetectionModelDeploymentMetadata`
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types._OperationFuture` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "deploy_model" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "deploy_model"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.deploy_model,
+ default_retry=self._method_configs["DeployModel"].retry,
+ default_timeout=self._method_configs["DeployModel"].timeout,
+ client_info=self._client_info,
+ )
+
+ # Sanity check: We have some fields which are mutually exclusive;
+ # raise ValueError if more than one is sent.
+ google.api_core.protobuf_helpers.check_oneof(
+ image_object_detection_model_deployment_metadata=image_object_detection_model_deployment_metadata
+ )
+
+ request = service_pb2.DeployModelRequest(
+ name=name,
+ image_object_detection_model_deployment_metadata=image_object_detection_model_deployment_metadata,
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("name", name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ operation = self._inner_api_calls["deploy_model"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+ return google.api_core.operation.from_gapic(
+ operation,
+ self.transport._operations_client,
+ empty_pb2.Empty,
+ metadata_type=proto_operations_pb2.OperationMetadata,
+ )
+
+ def undeploy_model(
+ self,
+ name,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Undeploys a model. If the model is not deployed this method has no
+ effect.
+
+ Only applicable for Text Classification, Image Object Detection and
+ Tables; all other domains manage deployment automatically.
+
+ Returns an empty response in the ``response`` field when it completes.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> client = automl_v1beta1.AutoMlClient()
+ >>>
+ >>> name = client.model_path('[PROJECT]', '[LOCATION]', '[MODEL]')
+ >>>
+ >>> response = client.undeploy_model(name)
+ >>>
+ >>> def callback(operation_future):
+ ... # Handle result.
+ ... result = operation_future.result()
+ >>>
+ >>> response.add_done_callback(callback)
+ >>>
+ >>> # Handle metadata.
+ >>> metadata = response.metadata()
+
+ Args:
+ name (str): Resource name of the model to undeploy.
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types._OperationFuture` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "undeploy_model" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "undeploy_model"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.undeploy_model,
+ default_retry=self._method_configs["UndeployModel"].retry,
+ default_timeout=self._method_configs["UndeployModel"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = service_pb2.UndeployModelRequest(name=name)
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("name", name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ operation = self._inner_api_calls["undeploy_model"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+ return google.api_core.operation.from_gapic(
+ operation,
+ self.transport._operations_client,
+ empty_pb2.Empty,
+ metadata_type=proto_operations_pb2.OperationMetadata,
+ )
+
+ def get_model_evaluation(
+ self,
+ name,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Gets a model evaluation.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> client = automl_v1beta1.AutoMlClient()
+ >>>
+ >>> name = client.model_evaluation_path('[PROJECT]', '[LOCATION]', '[MODEL]', '[MODEL_EVALUATION]')
+ >>>
+ >>> response = client.get_model_evaluation(name)
+
+ Args:
+ name (str): Resource name for the model evaluation.
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types.ModelEvaluation` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "get_model_evaluation" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "get_model_evaluation"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.get_model_evaluation,
+ default_retry=self._method_configs["GetModelEvaluation"].retry,
+ default_timeout=self._method_configs["GetModelEvaluation"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = service_pb2.GetModelEvaluationRequest(name=name)
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("name", name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ return self._inner_api_calls["get_model_evaluation"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+
+ def export_model(
+ self,
+ name,
+ output_config,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Exports a trained, "export-able", model to a user specified Google Cloud
+ Storage location. A model is considered export-able if and only if it
+ has an export format defined for it in
+
+ ``ModelExportOutputConfig``.
+
+ Returns an empty response in the ``response`` field when it completes.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> client = automl_v1beta1.AutoMlClient()
+ >>>
+ >>> name = client.model_path('[PROJECT]', '[LOCATION]', '[MODEL]')
+ >>>
+ >>> # TODO: Initialize `output_config`:
+ >>> output_config = {}
+ >>>
+ >>> response = client.export_model(name, output_config)
+ >>>
+ >>> def callback(operation_future):
+ ... # Handle result.
+ ... result = operation_future.result()
+ >>>
+ >>> response.add_done_callback(callback)
+ >>>
+ >>> # Handle metadata.
+ >>> metadata = response.metadata()
+
+ Args:
+ name (str): Required. The resource name of the model to export.
+ output_config (Union[dict, ~google.cloud.automl_v1beta1.types.ModelExportOutputConfig]): Required. The desired output location and configuration.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.automl_v1beta1.types.ModelExportOutputConfig`
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types._OperationFuture` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "export_model" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "export_model"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.export_model,
+ default_retry=self._method_configs["ExportModel"].retry,
+ default_timeout=self._method_configs["ExportModel"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = service_pb2.ExportModelRequest(name=name, output_config=output_config)
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("name", name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ operation = self._inner_api_calls["export_model"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+ return google.api_core.operation.from_gapic(
+ operation,
+ self.transport._operations_client,
+ empty_pb2.Empty,
+ metadata_type=proto_operations_pb2.OperationMetadata,
+ )
+
+ def export_evaluated_examples(
+ self,
+ name,
+ output_config,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Exports examples on which the model was evaluated (i.e. which were in
+ the TEST set of the dataset the model was created from), together with
+ their ground truth annotations and the annotations created (predicted)
+ by the model. The examples, ground truth and predictions are exported in
+ the state they were at the moment the model was evaluated.
+
+ This export is available only for 30 days since the model evaluation is
+ created.
+
+ Currently only available for Tables.
+
+ Returns an empty response in the ``response`` field when it completes.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> client = automl_v1beta1.AutoMlClient()
+ >>>
+ >>> name = client.model_path('[PROJECT]', '[LOCATION]', '[MODEL]')
+ >>>
+ >>> # TODO: Initialize `output_config`:
+ >>> output_config = {}
+ >>>
+ >>> response = client.export_evaluated_examples(name, output_config)
+ >>>
+ >>> def callback(operation_future):
+ ... # Handle result.
+ ... result = operation_future.result()
+ >>>
+ >>> response.add_done_callback(callback)
+ >>>
+ >>> # Handle metadata.
+ >>> metadata = response.metadata()
+
+ Args:
+ name (str): Required. The resource name of the model whose evaluated examples are to
+ be exported.
+ output_config (Union[dict, ~google.cloud.automl_v1beta1.types.ExportEvaluatedExamplesOutputConfig]): Required. The desired output location and configuration.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.automl_v1beta1.types.ExportEvaluatedExamplesOutputConfig`
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types._OperationFuture` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "export_evaluated_examples" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "export_evaluated_examples"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.export_evaluated_examples,
+ default_retry=self._method_configs["ExportEvaluatedExamples"].retry,
+ default_timeout=self._method_configs["ExportEvaluatedExamples"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = service_pb2.ExportEvaluatedExamplesRequest(
+ name=name, output_config=output_config
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("name", name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ operation = self._inner_api_calls["export_evaluated_examples"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+ return google.api_core.operation.from_gapic(
+ operation,
+ self.transport._operations_client,
+ empty_pb2.Empty,
+ metadata_type=proto_operations_pb2.OperationMetadata,
+ )
+
+ def list_model_evaluations(
+ self,
+ parent,
+ filter_=None,
+ page_size=None,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Lists model evaluations.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> client = automl_v1beta1.AutoMlClient()
+ >>>
+ >>> parent = client.model_path('[PROJECT]', '[LOCATION]', '[MODEL]')
+ >>>
+ >>> # Iterate over all results
+ >>> for element in client.list_model_evaluations(parent):
+ ... # process element
+ ... pass
+ >>>
+ >>>
+ >>> # Alternatively:
+ >>>
+ >>> # Iterate over results one page at a time
+ >>> for page in client.list_model_evaluations(parent).pages:
+ ... for element in page:
+ ... # process element
+ ... pass
+
+ Args:
+ parent (str): Resource name of the model to list the model evaluations for.
+ If modelId is set as "-", this will list model evaluations from across all
+ models of the parent location.
+ filter_ (str): An expression for filtering the results of the request.
+
+ - ``annotation_spec_id`` - for =, != or existence. See example below
+ for the last.
+
+ Some examples of using the filter are:
+
+ - ``annotation_spec_id!=4`` --> The model evaluation was done for
+ annotation spec with ID different than 4.
+ - ``NOT annotation_spec_id:*`` --> The model evaluation was done for
+ aggregate of all annotation specs.
+ page_size (int): The maximum number of resources contained in the
+ underlying API response. If page streaming is performed per-
+ resource, this parameter does not affect the return value. If page
+ streaming is performed per-page, this determines the maximum number
+ of resources in a page.
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.api_core.page_iterator.PageIterator` instance.
+ An iterable of :class:`~google.cloud.automl_v1beta1.types.ModelEvaluation` instances.
+ You can also iterate over the pages of the response
+ using its `pages` property.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "list_model_evaluations" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "list_model_evaluations"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.list_model_evaluations,
+ default_retry=self._method_configs["ListModelEvaluations"].retry,
+ default_timeout=self._method_configs["ListModelEvaluations"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = service_pb2.ListModelEvaluationsRequest(
+ parent=parent, filter=filter_, page_size=page_size
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("parent", parent)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ iterator = google.api_core.page_iterator.GRPCIterator(
+ client=None,
+ method=functools.partial(
+ self._inner_api_calls["list_model_evaluations"],
+ retry=retry,
+ timeout=timeout,
+ metadata=metadata,
+ ),
+ request=request,
+ items_field="model_evaluation",
+ request_token_field="page_token",
+ response_token_field="next_page_token",
+ )
+ return iterator
+
+ def get_annotation_spec(
+ self,
+ name,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Gets an annotation spec.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> client = automl_v1beta1.AutoMlClient()
+ >>>
+ >>> name = client.annotation_spec_path('[PROJECT]', '[LOCATION]', '[DATASET]', '[ANNOTATION_SPEC]')
+ >>>
+ >>> response = client.get_annotation_spec(name)
+
+ Args:
+ name (str): The resource name of the annotation spec to retrieve.
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types.AnnotationSpec` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "get_annotation_spec" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "get_annotation_spec"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.get_annotation_spec,
+ default_retry=self._method_configs["GetAnnotationSpec"].retry,
+ default_timeout=self._method_configs["GetAnnotationSpec"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = service_pb2.GetAnnotationSpecRequest(name=name)
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("name", name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ return self._inner_api_calls["get_annotation_spec"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+
+ def get_table_spec(
+ self,
+ name,
+ field_mask=None,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Gets a table spec.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> client = automl_v1beta1.AutoMlClient()
+ >>>
+ >>> name = client.table_spec_path('[PROJECT]', '[LOCATION]', '[DATASET]', '[TABLE_SPEC]')
+ >>>
+ >>> response = client.get_table_spec(name)
+
+ Args:
+ name (str): The resource name of the table spec to retrieve.
+ field_mask (Union[dict, ~google.cloud.automl_v1beta1.types.FieldMask]): Mask specifying which fields to read.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.automl_v1beta1.types.FieldMask`
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types.TableSpec` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "get_table_spec" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "get_table_spec"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.get_table_spec,
+ default_retry=self._method_configs["GetTableSpec"].retry,
+ default_timeout=self._method_configs["GetTableSpec"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = service_pb2.GetTableSpecRequest(name=name, field_mask=field_mask)
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("name", name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ return self._inner_api_calls["get_table_spec"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+
+ def list_table_specs(
+ self,
+ parent,
+ field_mask=None,
+ filter_=None,
+ page_size=None,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Lists table specs in a dataset.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> client = automl_v1beta1.AutoMlClient()
+ >>>
+ >>> parent = client.dataset_path('[PROJECT]', '[LOCATION]', '[DATASET]')
+ >>>
+ >>> # Iterate over all results
+ >>> for element in client.list_table_specs(parent):
+ ... # process element
+ ... pass
+ >>>
+ >>>
+ >>> # Alternatively:
+ >>>
+ >>> # Iterate over results one page at a time
+ >>> for page in client.list_table_specs(parent).pages:
+ ... for element in page:
+ ... # process element
+ ... pass
+
+ Args:
+ parent (str): The resource name of the dataset to list table specs from.
+ field_mask (Union[dict, ~google.cloud.automl_v1beta1.types.FieldMask]): Mask specifying which fields to read.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.automl_v1beta1.types.FieldMask`
+ filter_ (str): Filter expression, see go/filtering.
+ page_size (int): The maximum number of resources contained in the
+ underlying API response. If page streaming is performed per-
+ resource, this parameter does not affect the return value. If page
+ streaming is performed per-page, this determines the maximum number
+ of resources in a page.
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.api_core.page_iterator.PageIterator` instance.
+ An iterable of :class:`~google.cloud.automl_v1beta1.types.TableSpec` instances.
+ You can also iterate over the pages of the response
+ using its `pages` property.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "list_table_specs" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "list_table_specs"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.list_table_specs,
+ default_retry=self._method_configs["ListTableSpecs"].retry,
+ default_timeout=self._method_configs["ListTableSpecs"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = service_pb2.ListTableSpecsRequest(
+ parent=parent, field_mask=field_mask, filter=filter_, page_size=page_size
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("parent", parent)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ iterator = google.api_core.page_iterator.GRPCIterator(
+ client=None,
+ method=functools.partial(
+ self._inner_api_calls["list_table_specs"],
+ retry=retry,
+ timeout=timeout,
+ metadata=metadata,
+ ),
+ request=request,
+ items_field="table_specs",
+ request_token_field="page_token",
+ response_token_field="next_page_token",
+ )
+ return iterator
+
+ def update_table_spec(
+ self,
+ table_spec,
+ update_mask=None,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Updates a table spec.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> client = automl_v1beta1.AutoMlClient()
+ >>>
+ >>> # TODO: Initialize `table_spec`:
+ >>> table_spec = {}
+ >>>
+ >>> response = client.update_table_spec(table_spec)
+
+ Args:
+ table_spec (Union[dict, ~google.cloud.automl_v1beta1.types.TableSpec]): The table spec which replaces the resource on the server.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.automl_v1beta1.types.TableSpec`
+ update_mask (Union[dict, ~google.cloud.automl_v1beta1.types.FieldMask]): The update mask applies to the resource.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.automl_v1beta1.types.FieldMask`
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types.TableSpec` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "update_table_spec" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "update_table_spec"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.update_table_spec,
+ default_retry=self._method_configs["UpdateTableSpec"].retry,
+ default_timeout=self._method_configs["UpdateTableSpec"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = service_pb2.UpdateTableSpecRequest(
+ table_spec=table_spec, update_mask=update_mask
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("table_spec.name", table_spec.name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ return self._inner_api_calls["update_table_spec"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+
+ def get_column_spec(
+ self,
+ name,
+ field_mask=None,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Gets a column spec.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> client = automl_v1beta1.AutoMlClient()
+ >>>
+ >>> name = client.column_spec_path('[PROJECT]', '[LOCATION]', '[DATASET]', '[TABLE_SPEC]', '[COLUMN_SPEC]')
+ >>>
+ >>> response = client.get_column_spec(name)
+
+ Args:
+ name (str): The resource name of the column spec to retrieve.
+ field_mask (Union[dict, ~google.cloud.automl_v1beta1.types.FieldMask]): Mask specifying which fields to read.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.automl_v1beta1.types.FieldMask`
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types.ColumnSpec` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "get_column_spec" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "get_column_spec"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.get_column_spec,
+ default_retry=self._method_configs["GetColumnSpec"].retry,
+ default_timeout=self._method_configs["GetColumnSpec"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = service_pb2.GetColumnSpecRequest(name=name, field_mask=field_mask)
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("name", name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ return self._inner_api_calls["get_column_spec"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+
+ def list_column_specs(
+ self,
+ parent,
+ field_mask=None,
+ filter_=None,
+ page_size=None,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Lists column specs in a table spec.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> client = automl_v1beta1.AutoMlClient()
+ >>>
+ >>> parent = client.table_spec_path('[PROJECT]', '[LOCATION]', '[DATASET]', '[TABLE_SPEC]')
+ >>>
+ >>> # Iterate over all results
+ >>> for element in client.list_column_specs(parent):
+ ... # process element
+ ... pass
+ >>>
+ >>>
+ >>> # Alternatively:
+ >>>
+ >>> # Iterate over results one page at a time
+ >>> for page in client.list_column_specs(parent).pages:
+ ... for element in page:
+ ... # process element
+ ... pass
+
+ Args:
+ parent (str): The resource name of the table spec to list column specs from.
+ field_mask (Union[dict, ~google.cloud.automl_v1beta1.types.FieldMask]): Mask specifying which fields to read.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.automl_v1beta1.types.FieldMask`
+ filter_ (str): Filter expression, see go/filtering.
+ page_size (int): The maximum number of resources contained in the
+ underlying API response. If page streaming is performed per-
+ resource, this parameter does not affect the return value. If page
+ streaming is performed per-page, this determines the maximum number
+ of resources in a page.
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.api_core.page_iterator.PageIterator` instance.
+ An iterable of :class:`~google.cloud.automl_v1beta1.types.ColumnSpec` instances.
+ You can also iterate over the pages of the response
+ using its `pages` property.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "list_column_specs" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "list_column_specs"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.list_column_specs,
+ default_retry=self._method_configs["ListColumnSpecs"].retry,
+ default_timeout=self._method_configs["ListColumnSpecs"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = service_pb2.ListColumnSpecsRequest(
+ parent=parent, field_mask=field_mask, filter=filter_, page_size=page_size
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("parent", parent)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ iterator = google.api_core.page_iterator.GRPCIterator(
+ client=None,
+ method=functools.partial(
+ self._inner_api_calls["list_column_specs"],
+ retry=retry,
+ timeout=timeout,
+ metadata=metadata,
+ ),
+ request=request,
+ items_field="column_specs",
+ request_token_field="page_token",
+ response_token_field="next_page_token",
+ )
+ return iterator
+
+ def update_column_spec(
+ self,
+ column_spec,
+ update_mask=None,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Updates a column spec.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> client = automl_v1beta1.AutoMlClient()
+ >>>
+ >>> # TODO: Initialize `column_spec`:
+ >>> column_spec = {}
+ >>>
+ >>> response = client.update_column_spec(column_spec)
+
+ Args:
+ column_spec (Union[dict, ~google.cloud.automl_v1beta1.types.ColumnSpec]): The column spec which replaces the resource on the server.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.automl_v1beta1.types.ColumnSpec`
+ update_mask (Union[dict, ~google.cloud.automl_v1beta1.types.FieldMask]): The update mask applies to the resource.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.automl_v1beta1.types.FieldMask`
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types.ColumnSpec` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "update_column_spec" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "update_column_spec"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.update_column_spec,
+ default_retry=self._method_configs["UpdateColumnSpec"].retry,
+ default_timeout=self._method_configs["UpdateColumnSpec"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = service_pb2.UpdateColumnSpecRequest(
+ column_spec=column_spec, update_mask=update_mask
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("column_spec.name", column_spec.name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ return self._inner_api_calls["update_column_spec"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
diff --git a/automl/google/cloud/automl_v1beta1/gapic/auto_ml_client_config.py b/automl/google/cloud/automl_v1beta1/gapic/auto_ml_client_config.py
new file mode 100644
index 000000000000..d127ce6f478b
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/gapic/auto_ml_client_config.py
@@ -0,0 +1,143 @@
+config = {
+ "interfaces": {
+ "google.cloud.automl.v1beta1.AutoMl": {
+ "retry_codes": {
+ "idempotent": ["DEADLINE_EXCEEDED", "UNAVAILABLE"],
+ "non_idempotent": [],
+ },
+ "retry_params": {
+ "default": {
+ "initial_retry_delay_millis": 100,
+ "retry_delay_multiplier": 1.3,
+ "max_retry_delay_millis": 60000,
+ "initial_rpc_timeout_millis": 20000,
+ "rpc_timeout_multiplier": 1.0,
+ "max_rpc_timeout_millis": 20000,
+ "total_timeout_millis": 600000,
+ }
+ },
+ "methods": {
+ "CreateDataset": {
+ "timeout_millis": 5000,
+ "retry_codes_name": "non_idempotent",
+ "retry_params_name": "default",
+ },
+ "UpdateDataset": {
+ "timeout_millis": 5000,
+ "retry_codes_name": "non_idempotent",
+ "retry_params_name": "default",
+ },
+ "GetDataset": {
+ "timeout_millis": 5000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "ListDatasets": {
+ "timeout_millis": 50000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "DeleteDataset": {
+ "timeout_millis": 5000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "ImportData": {
+ "timeout_millis": 20000,
+ "retry_codes_name": "non_idempotent",
+ "retry_params_name": "default",
+ },
+ "ExportData": {
+ "timeout_millis": 5000,
+ "retry_codes_name": "non_idempotent",
+ "retry_params_name": "default",
+ },
+ "CreateModel": {
+ "timeout_millis": 20000,
+ "retry_codes_name": "non_idempotent",
+ "retry_params_name": "default",
+ },
+ "GetModel": {
+ "timeout_millis": 5000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "ListModels": {
+ "timeout_millis": 50000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "DeleteModel": {
+ "timeout_millis": 5000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "DeployModel": {
+ "timeout_millis": 5000,
+ "retry_codes_name": "non_idempotent",
+ "retry_params_name": "default",
+ },
+ "UndeployModel": {
+ "timeout_millis": 5000,
+ "retry_codes_name": "non_idempotent",
+ "retry_params_name": "default",
+ },
+ "GetModelEvaluation": {
+ "timeout_millis": 5000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "ExportModel": {
+ "timeout_millis": 5000,
+ "retry_codes_name": "non_idempotent",
+ "retry_params_name": "default",
+ },
+ "ExportEvaluatedExamples": {
+ "timeout_millis": 5000,
+ "retry_codes_name": "non_idempotent",
+ "retry_params_name": "default",
+ },
+ "ListModelEvaluations": {
+ "timeout_millis": 50000,
+ "retry_codes_name": "non_idempotent",
+ "retry_params_name": "default",
+ },
+ "GetAnnotationSpec": {
+ "timeout_millis": 5000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "GetTableSpec": {
+ "timeout_millis": 5000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "ListTableSpecs": {
+ "timeout_millis": 5000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "UpdateTableSpec": {
+ "timeout_millis": 5000,
+ "retry_codes_name": "non_idempotent",
+ "retry_params_name": "default",
+ },
+ "GetColumnSpec": {
+ "timeout_millis": 5000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "ListColumnSpecs": {
+ "timeout_millis": 5000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "UpdateColumnSpec": {
+ "timeout_millis": 5000,
+ "retry_codes_name": "non_idempotent",
+ "retry_params_name": "default",
+ },
+ },
+ }
+ }
+}
diff --git a/automl/google/cloud/automl_v1beta1/gapic/enums.py b/automl/google/cloud/automl_v1beta1/gapic/enums.py
new file mode 100644
index 000000000000..9d817f8ec807
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/gapic/enums.py
@@ -0,0 +1,158 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Wrappers for protocol buffer enum types."""
+
+import enum
+
+
+class ClassificationType(enum.IntEnum):
+ """
+ Type of the classification problem.
+
+ Attributes:
+ CLASSIFICATION_TYPE_UNSPECIFIED (int): An un-set value of this enum.
+ MULTICLASS (int): At most one label is allowed per example.
+ MULTILABEL (int): Multiple labels are allowed for one example.
+ """
+
+ CLASSIFICATION_TYPE_UNSPECIFIED = 0
+ MULTICLASS = 1
+ MULTILABEL = 2
+
+
+class NullValue(enum.IntEnum):
+ """
+ ``NullValue`` is a singleton enumeration to represent the null value for
+ the ``Value`` type union.
+
+ The JSON representation for ``NullValue`` is JSON ``null``.
+
+ Attributes:
+ NULL_VALUE (int): Null value.
+ """
+
+ NULL_VALUE = 0
+
+
+class TypeCode(enum.IntEnum):
+ """
+ ``TypeCode`` is used as a part of ``DataType``.
+
+ Attributes:
+ TYPE_CODE_UNSPECIFIED (int): Not specified. Should not be used.
+ FLOAT64 (int): Encoded as ``number``, or the strings ``"NaN"``, ``"Infinity"``, or
+ ``"-Infinity"``.
+ TIMESTAMP (int): Must be between 0AD and 9999AD. Encoded as ``string`` according to
+ ``time_format``, or, if that format is not set, then in RFC 3339
+ ``date-time`` format, where ``time-offset`` = ``"Z"`` (e.g.
+ 1985-04-12T23:20:50.52Z).
+ STRING (int): Encoded as ``string``.
+ ARRAY (int): Encoded as ``list``, where the list elements are represented according
+ to
+
+ ``list_element_type``.
+ STRUCT (int): Encoded as ``struct``, where field values are represented according to
+ ``struct_type``.
+ CATEGORY (int): Values of this type are not further understood by AutoML, e.g. AutoML is
+ unable to tell the order of values (as it could with FLOAT64), or is
+ unable to say if one value contains another (as it could with STRING).
+ Encoded as ``string`` (bytes should be base64-encoded, as described in
+ RFC 4648, section 4).
+ """
+
+ TYPE_CODE_UNSPECIFIED = 0
+ FLOAT64 = 3
+ TIMESTAMP = 4
+ STRING = 6
+ ARRAY = 8
+ STRUCT = 9
+ CATEGORY = 10
+
+
+class Document(object):
+ class Layout(object):
+ class TextSegmentType(enum.IntEnum):
+ """
+ The type of TextSegment in the context of the original document.
+
+ Attributes:
+ TEXT_SEGMENT_TYPE_UNSPECIFIED (int): Should not be used.
+ TOKEN (int): The text segment is a token. e.g. word.
+ PARAGRAPH (int): The text segment is a paragraph.
+ FORM_FIELD (int): The text segment is a form field.
+ FORM_FIELD_NAME (int): The text segment is the name part of a form field. It will be treated as
+ child of another FORM\_FIELD TextSegment if its span is subspan of
+ another TextSegment with type FORM\_FIELD.
+ FORM_FIELD_CONTENTS (int): The text segment is the text content part of a form field. It will be
+ treated as child of another FORM\_FIELD TextSegment if its span is
+ subspan of another TextSegment with type FORM\_FIELD.
+ TABLE (int): The text segment is a whole table, including headers, and all rows.
+ TABLE_HEADER (int): The text segment is a table's headers. It will be treated as child of
+ another TABLE TextSegment if its span is subspan of another TextSegment
+ with type TABLE.
+ TABLE_ROW (int): The text segment is a row in table. It will be treated as child of
+ another TABLE TextSegment if its span is subspan of another TextSegment
+ with type TABLE.
+ TABLE_CELL (int): The text segment is a cell in table. It will be treated as child of
+ another TABLE\_ROW TextSegment if its span is subspan of another
+ TextSegment with type TABLE\_ROW.
+ """
+
+ TEXT_SEGMENT_TYPE_UNSPECIFIED = 0
+ TOKEN = 1
+ PARAGRAPH = 2
+ FORM_FIELD = 3
+ FORM_FIELD_NAME = 4
+ FORM_FIELD_CONTENTS = 5
+ TABLE = 6
+ TABLE_HEADER = 7
+ TABLE_ROW = 8
+ TABLE_CELL = 9
+
+
+class DocumentDimensions(object):
+ class DocumentDimensionUnit(enum.IntEnum):
+ """
+ Unit of the document dimension.
+
+ Attributes:
+ DOCUMENT_DIMENSION_UNIT_UNSPECIFIED (int): Should not be used.
+ INCH (int): Document dimension is measured in inches.
+ CENTIMETER (int): Document dimension is measured in centimeters.
+ POINT (int): Document dimension is measured in points. 72 points = 1 inch.
+ """
+
+ DOCUMENT_DIMENSION_UNIT_UNSPECIFIED = 0
+ INCH = 1
+ CENTIMETER = 2
+ POINT = 3
+
+
+class Model(object):
+ class DeploymentState(enum.IntEnum):
+ """
+ Deployment state of the model.
+
+ Attributes:
+ DEPLOYMENT_STATE_UNSPECIFIED (int): Should not be used, an un-set enum has this value by default.
+ DEPLOYED (int): Model is deployed.
+ UNDEPLOYED (int): Model is not deployed.
+ """
+
+ DEPLOYMENT_STATE_UNSPECIFIED = 0
+ DEPLOYED = 1
+ UNDEPLOYED = 2
diff --git a/automl/google/cloud/automl_v1beta1/gapic/prediction_service_client.py b/automl/google/cloud/automl_v1beta1/gapic/prediction_service_client.py
new file mode 100644
index 000000000000..3342fecd621f
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/gapic/prediction_service_client.py
@@ -0,0 +1,517 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Accesses the google.cloud.automl.v1beta1 PredictionService API."""
+
+import pkg_resources
+import warnings
+
+from google.oauth2 import service_account
+import google.api_core.client_options
+import google.api_core.gapic_v1.client_info
+import google.api_core.gapic_v1.config
+import google.api_core.gapic_v1.method
+import google.api_core.gapic_v1.routing_header
+import google.api_core.grpc_helpers
+import google.api_core.operation
+import google.api_core.operations_v1
+import google.api_core.path_template
+import grpc
+
+from google.cloud.automl_v1beta1.gapic import enums
+from google.cloud.automl_v1beta1.gapic import prediction_service_client_config
+from google.cloud.automl_v1beta1.gapic.transports import (
+ prediction_service_grpc_transport,
+)
+from google.cloud.automl_v1beta1.proto import annotation_spec_pb2
+from google.cloud.automl_v1beta1.proto import column_spec_pb2
+from google.cloud.automl_v1beta1.proto import data_items_pb2
+from google.cloud.automl_v1beta1.proto import dataset_pb2
+from google.cloud.automl_v1beta1.proto import image_pb2
+from google.cloud.automl_v1beta1.proto import io_pb2
+from google.cloud.automl_v1beta1.proto import model_evaluation_pb2
+from google.cloud.automl_v1beta1.proto import model_pb2
+from google.cloud.automl_v1beta1.proto import operations_pb2 as proto_operations_pb2
+from google.cloud.automl_v1beta1.proto import prediction_service_pb2
+from google.cloud.automl_v1beta1.proto import prediction_service_pb2_grpc
+from google.cloud.automl_v1beta1.proto import service_pb2
+from google.cloud.automl_v1beta1.proto import service_pb2_grpc
+from google.cloud.automl_v1beta1.proto import table_spec_pb2
+from google.longrunning import operations_pb2 as longrunning_operations_pb2
+from google.protobuf import empty_pb2
+from google.protobuf import field_mask_pb2
+
+
+_GAPIC_LIBRARY_VERSION = pkg_resources.get_distribution("google-cloud-automl").version
+
+
+class PredictionServiceClient(object):
+ """
+ AutoML Prediction API.
+
+ On any input that is documented to expect a string parameter in
+ snake\_case or kebab-case, either of those cases is accepted.
+ """
+
+ SERVICE_ADDRESS = "automl.googleapis.com:443"
+ """The default address of the service."""
+
+ # The name of the interface for this client. This is the key used to
+ # find the method configuration in the client_config dictionary.
+ _INTERFACE_NAME = "google.cloud.automl.v1beta1.PredictionService"
+
+ @classmethod
+ def from_service_account_file(cls, filename, *args, **kwargs):
+ """Creates an instance of this client using the provided credentials
+ file.
+
+ Args:
+ filename (str): The path to the service account private key json
+ file.
+ args: Additional arguments to pass to the constructor.
+ kwargs: Additional arguments to pass to the constructor.
+
+ Returns:
+ PredictionServiceClient: The constructed client.
+ """
+ credentials = service_account.Credentials.from_service_account_file(filename)
+ kwargs["credentials"] = credentials
+ return cls(*args, **kwargs)
+
+ from_service_account_json = from_service_account_file
+
+ @classmethod
+ def model_path(cls, project, location, model):
+ """Return a fully-qualified model string."""
+ return google.api_core.path_template.expand(
+ "projects/{project}/locations/{location}/models/{model}",
+ project=project,
+ location=location,
+ model=model,
+ )
+
+ def __init__(
+ self,
+ transport=None,
+ channel=None,
+ credentials=None,
+ client_config=None,
+ client_info=None,
+ client_options=None,
+ ):
+ """Constructor.
+
+ Args:
+ transport (Union[~.PredictionServiceGrpcTransport,
+ Callable[[~.Credentials, type], ~.PredictionServiceGrpcTransport]): A transport
+ instance, responsible for actually making the API calls.
+ The default transport uses the gRPC protocol.
+ This argument may also be a callable which returns a
+ transport instance. Callables will be sent the credentials
+ as the first argument and the default transport class as
+ the second argument.
+ channel (grpc.Channel): DEPRECATED. A ``Channel`` instance
+ through which to make calls. This argument is mutually exclusive
+ with ``credentials``; providing both will raise an exception.
+ credentials (google.auth.credentials.Credentials): The
+ authorization credentials to attach to requests. These
+ credentials identify this application to the service. If none
+ are specified, the client will attempt to ascertain the
+ credentials from the environment.
+ This argument is mutually exclusive with providing a
+ transport instance to ``transport``; doing so will raise
+ an exception.
+ client_config (dict): DEPRECATED. A dictionary of call options for
+ each method. If not specified, the default configuration is used.
+ client_info (google.api_core.gapic_v1.client_info.ClientInfo):
+ The client info used to send a user-agent string along with
+ API requests. If ``None``, then default info will be used.
+ Generally, you only need to set this if you're developing
+ your own client library.
+ client_options (Union[dict, google.api_core.client_options.ClientOptions]):
+ Client options used to set user options on the client. API Endpoint
+ should be set through client_options.
+ """
+ # Raise deprecation warnings for things we want to go away.
+ if client_config is not None:
+ warnings.warn(
+ "The `client_config` argument is deprecated.",
+ PendingDeprecationWarning,
+ stacklevel=2,
+ )
+ else:
+ client_config = prediction_service_client_config.config
+
+ if channel:
+ warnings.warn(
+ "The `channel` argument is deprecated; use " "`transport` instead.",
+ PendingDeprecationWarning,
+ stacklevel=2,
+ )
+
+ api_endpoint = self.SERVICE_ADDRESS
+ if client_options:
+ if type(client_options) == dict:
+ client_options = google.api_core.client_options.from_dict(
+ client_options
+ )
+ if client_options.api_endpoint:
+ api_endpoint = client_options.api_endpoint
+
+ # Instantiate the transport.
+ # The transport is responsible for handling serialization and
+ # deserialization and actually sending data to the service.
+ if transport:
+ if callable(transport):
+ self.transport = transport(
+ credentials=credentials,
+ default_class=prediction_service_grpc_transport.PredictionServiceGrpcTransport,
+ address=api_endpoint,
+ )
+ else:
+ if credentials:
+ raise ValueError(
+ "Received both a transport instance and "
+ "credentials; these are mutually exclusive."
+ )
+ self.transport = transport
+ else:
+ self.transport = prediction_service_grpc_transport.PredictionServiceGrpcTransport(
+ address=api_endpoint, channel=channel, credentials=credentials
+ )
+
+ if client_info is None:
+ client_info = google.api_core.gapic_v1.client_info.ClientInfo(
+ gapic_version=_GAPIC_LIBRARY_VERSION
+ )
+ else:
+ client_info.gapic_version = _GAPIC_LIBRARY_VERSION
+ self._client_info = client_info
+
+ # Parse out the default settings for retry and timeout for each RPC
+ # from the client configuration.
+ # (Ordinarily, these are the defaults specified in the `*_config.py`
+ # file next to this one.)
+ self._method_configs = google.api_core.gapic_v1.config.parse_method_configs(
+ client_config["interfaces"][self._INTERFACE_NAME]
+ )
+
+ # Save a dictionary of cached API call functions.
+ # These are the actual callables which invoke the proper
+ # transport methods, wrapped with `wrap_method` to add retry,
+ # timeout, and the like.
+ self._inner_api_calls = {}
+
+ # Service calls
+ def predict(
+ self,
+ name,
+ payload,
+ params=None,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Perform an online prediction. The prediction result will be directly
+ returned in the response. Available for following ML problems, and their
+ expected request payloads:
+
+ - Image Classification - Image in .JPEG, .GIF or .PNG format,
+ image\_bytes up to 30MB.
+ - Image Object Detection - Image in .JPEG, .GIF or .PNG format,
+ image\_bytes up to 30MB.
+ - Text Classification - TextSnippet, content up to 60,000 characters,
+ UTF-8 encoded.
+ - Text Extraction - TextSnippet, content up to 30,000 characters, UTF-8
+ NFC encoded.
+ - Translation - TextSnippet, content up to 25,000 characters, UTF-8
+ encoded.
+ - Tables - Row, with column values matching the columns of the model,
+ up to 5MB. Not available for FORECASTING
+
+ ``prediction_type``.
+
+ - Text Sentiment - TextSnippet, content up 500 characters, UTF-8
+ encoded.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> client = automl_v1beta1.PredictionServiceClient()
+ >>>
+ >>> name = client.model_path('[PROJECT]', '[LOCATION]', '[MODEL]')
+ >>>
+ >>> # TODO: Initialize `payload`:
+ >>> payload = {}
+ >>>
+ >>> response = client.predict(name, payload)
+
+ Args:
+ name (str): Name of the model requested to serve the prediction.
+ payload (Union[dict, ~google.cloud.automl_v1beta1.types.ExamplePayload]): Required. Payload to perform a prediction on. The payload must match the
+ problem type that the model was trained to solve.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.automl_v1beta1.types.ExamplePayload`
+ params (dict[str -> str]): Additional domain-specific parameters, any string must be up to 25000
+ characters long.
+
+ - For Image Classification:
+
+ ``score_threshold`` - (float) A value from 0.0 to 1.0. When the model
+ makes predictions for an image, it will only produce results that
+ have at least this confidence score. The default is 0.5.
+
+ - For Image Object Detection: ``score_threshold`` - (float) When Model
+ detects objects on the image, it will only produce bounding boxes
+ which have at least this confidence score. Value in 0 to 1 range,
+ default is 0.5. ``max_bounding_box_count`` - (int64) No more than
+ this number of bounding boxes will be returned in the response.
+ Default is 100, the requested value may be limited by server.
+
+ - For Tables: ``feature_importance`` - (boolean) Whether
+
+ [feature\_importance][[google.cloud.automl.v1beta1.TablesModelColumnInfo.feature\_importance]
+ should be populated in the returned
+
+ [TablesAnnotation(-s)][[google.cloud.automl.v1beta1.TablesAnnotation].
+ The default is false.
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types.PredictResponse` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "predict" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "predict"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.predict,
+ default_retry=self._method_configs["Predict"].retry,
+ default_timeout=self._method_configs["Predict"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = prediction_service_pb2.PredictRequest(
+ name=name, payload=payload, params=params
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("name", name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ return self._inner_api_calls["predict"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+
+ def batch_predict(
+ self,
+ name,
+ input_config,
+ output_config,
+ params=None,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Perform a batch prediction. Unlike the online ``Predict``, batch
+ prediction result won't be immediately available in the response.
+ Instead, a long running operation object is returned. User can poll the
+ operation result via ``GetOperation`` method. Once the operation is
+ done, ``BatchPredictResult`` is returned in the ``response`` field.
+ Available for following ML problems:
+
+ - Image Classification
+ - Image Object Detection
+ - Video Classification
+ - Video Object Tracking \* Text Extraction
+ - Tables
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> client = automl_v1beta1.PredictionServiceClient()
+ >>>
+ >>> name = client.model_path('[PROJECT]', '[LOCATION]', '[MODEL]')
+ >>>
+ >>> # TODO: Initialize `input_config`:
+ >>> input_config = {}
+ >>>
+ >>> # TODO: Initialize `output_config`:
+ >>> output_config = {}
+ >>>
+ >>> response = client.batch_predict(name, input_config, output_config)
+ >>>
+ >>> def callback(operation_future):
+ ... # Handle result.
+ ... result = operation_future.result()
+ >>>
+ >>> response.add_done_callback(callback)
+ >>>
+ >>> # Handle metadata.
+ >>> metadata = response.metadata()
+
+ Args:
+ name (str): Name of the model requested to serve the batch prediction.
+ input_config (Union[dict, ~google.cloud.automl_v1beta1.types.BatchPredictInputConfig]): Required. The input configuration for batch prediction.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.automl_v1beta1.types.BatchPredictInputConfig`
+ output_config (Union[dict, ~google.cloud.automl_v1beta1.types.BatchPredictOutputConfig]): Required. The Configuration specifying where output predictions should
+ be written.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.automl_v1beta1.types.BatchPredictOutputConfig`
+ params (dict[str -> str]): Additional domain-specific parameters for the predictions, any string
+ must be up to 25000 characters long.
+
+ - For Text Classification:
+
+ ``score_threshold`` - (float) A value from 0.0 to 1.0. When the model
+ makes predictions for a text snippet, it will only produce results
+ that have at least this confidence score. The default is 0.5.
+
+ - For Image Classification:
+
+ ``score_threshold`` - (float) A value from 0.0 to 1.0. When the model
+ makes predictions for an image, it will only produce results that
+ have at least this confidence score. The default is 0.5.
+
+ - For Image Object Detection:
+
+ ``score_threshold`` - (float) When Model detects objects on the
+ image, it will only produce bounding boxes which have at least this
+ confidence score. Value in 0 to 1 range, default is 0.5.
+ ``max_bounding_box_count`` - (int64) No more than this number of
+ bounding boxes will be produced per image. Default is 100, the
+ requested value may be limited by server.
+
+ - For Video Classification : ``score_threshold`` - (float) A value from
+ 0.0 to 1.0. When the model makes predictions for a video, it will
+ only produce results that have at least this confidence score. The
+ default is 0.5. ``segment_classification`` - (boolean) Set to true to
+ request segment-level classification. AutoML Video Intelligence
+ returns labels and their confidence scores for the entire segment of
+ the video that user specified in the request configuration. The
+ default is "true". ``shot_classification`` - (boolean) Set to true to
+ request shot-level classification. AutoML Video Intelligence
+ determines the boundaries for each camera shot in the entire segment
+ of the video that user specified in the request configuration. AutoML
+ Video Intelligence then returns labels and their confidence scores
+ for each detected shot, along with the start and end time of the
+ shot. WARNING: Model evaluation is not done for this classification
+ type, the quality of it depends on training data, but there are no
+ metrics provided to describe that quality. The default is "false".
+ ``1s_interval_classification`` - (boolean) Set to true to request
+ classification for a video at one-second intervals. AutoML Video
+ Intelligence returns labels and their confidence scores for each
+ second of the entire segment of the video that user specified in the
+ request configuration. WARNING: Model evaluation is not done for this
+ classification type, the quality of it depends on training data, but
+ there are no metrics provided to describe that quality. The default
+ is "false".
+
+ - For Video Object Tracking: ``score_threshold`` - (float) When Model
+ detects objects on video frames, it will only produce bounding boxes
+ which have at least this confidence score. Value in 0 to 1 range,
+ default is 0.5. ``max_bounding_box_count`` - (int64) No more than
+ this number of bounding boxes will be returned per frame. Default is
+ 100, the requested value may be limited by server.
+ ``min_bounding_box_size`` - (float) Only bounding boxes with shortest
+ edge at least that long as a relative value of video frame size will
+ be returned. Value in 0 to 1 range. Default is 0.
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types._OperationFuture` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "batch_predict" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "batch_predict"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.batch_predict,
+ default_retry=self._method_configs["BatchPredict"].retry,
+ default_timeout=self._method_configs["BatchPredict"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = prediction_service_pb2.BatchPredictRequest(
+ name=name,
+ input_config=input_config,
+ output_config=output_config,
+ params=params,
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("name", name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ operation = self._inner_api_calls["batch_predict"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+ return google.api_core.operation.from_gapic(
+ operation,
+ self.transport._operations_client,
+ prediction_service_pb2.BatchPredictResult,
+ metadata_type=proto_operations_pb2.OperationMetadata,
+ )
diff --git a/automl/google/cloud/automl_v1beta1/gapic/prediction_service_client_config.py b/automl/google/cloud/automl_v1beta1/gapic/prediction_service_client_config.py
new file mode 100644
index 000000000000..d93ca92f8ed2
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/gapic/prediction_service_client_config.py
@@ -0,0 +1,33 @@
+config = {
+ "interfaces": {
+ "google.cloud.automl.v1beta1.PredictionService": {
+ "retry_codes": {
+ "idempotent": ["DEADLINE_EXCEEDED", "UNAVAILABLE"],
+ "non_idempotent": [],
+ },
+ "retry_params": {
+ "default": {
+ "initial_retry_delay_millis": 100,
+ "retry_delay_multiplier": 1.3,
+ "max_retry_delay_millis": 60000,
+ "initial_rpc_timeout_millis": 60000,
+ "rpc_timeout_multiplier": 1.0,
+ "max_rpc_timeout_millis": 60000,
+ "total_timeout_millis": 600000,
+ }
+ },
+ "methods": {
+ "Predict": {
+ "timeout_millis": 60000,
+ "retry_codes_name": "non_idempotent",
+ "retry_params_name": "default",
+ },
+ "BatchPredict": {
+ "timeout_millis": 20000,
+ "retry_codes_name": "non_idempotent",
+ "retry_params_name": "default",
+ },
+ },
+ }
+ }
+}
diff --git a/automl/google/cloud/automl_v1beta1/gapic/transports/__init__.py b/automl/google/cloud/automl_v1beta1/gapic/transports/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/automl/google/cloud/automl_v1beta1/gapic/transports/auto_ml_grpc_transport.py b/automl/google/cloud/automl_v1beta1/gapic/transports/auto_ml_grpc_transport.py
new file mode 100644
index 000000000000..0efa164f5de5
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/gapic/transports/auto_ml_grpc_transport.py
@@ -0,0 +1,474 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import google.api_core.grpc_helpers
+import google.api_core.operations_v1
+
+from google.cloud.automl_v1beta1.proto import service_pb2_grpc
+
+
+class AutoMlGrpcTransport(object):
+ """gRPC transport class providing stubs for
+ google.cloud.automl.v1beta1 AutoMl API.
+
+ The transport provides access to the raw gRPC stubs,
+ which can be used to take advantage of advanced
+ features of gRPC.
+ """
+
+ # The scopes needed to make gRPC calls to all of the methods defined
+ # in this service.
+ _OAUTH_SCOPES = ("https://www.googleapis.com/auth/cloud-platform",)
+
+ def __init__(
+ self, channel=None, credentials=None, address="automl.googleapis.com:443"
+ ):
+ """Instantiate the transport class.
+
+ Args:
+ channel (grpc.Channel): A ``Channel`` instance through
+ which to make calls. This argument is mutually exclusive
+ with ``credentials``; providing both will raise an exception.
+ credentials (google.auth.credentials.Credentials): The
+ authorization credentials to attach to requests. These
+ credentials identify this application to the service. If none
+ are specified, the client will attempt to ascertain the
+ credentials from the environment.
+ address (str): The address where the service is hosted.
+ """
+ # If both `channel` and `credentials` are specified, raise an
+ # exception (channels come with credentials baked in already).
+ if channel is not None and credentials is not None:
+ raise ValueError(
+ "The `channel` and `credentials` arguments are mutually " "exclusive."
+ )
+
+ # Create the channel.
+ if channel is None:
+ channel = self.create_channel(
+ address=address,
+ credentials=credentials,
+ options={
+ "grpc.max_send_message_length": -1,
+ "grpc.max_receive_message_length": -1,
+ }.items(),
+ )
+
+ self._channel = channel
+
+ # gRPC uses objects called "stubs" that are bound to the
+ # channel and provide a basic method for each RPC.
+ self._stubs = {"auto_ml_stub": service_pb2_grpc.AutoMlStub(channel)}
+
+ # Because this API includes a method that returns a
+ # long-running operation (proto: google.longrunning.Operation),
+ # instantiate an LRO client.
+ self._operations_client = google.api_core.operations_v1.OperationsClient(
+ channel
+ )
+
+ @classmethod
+ def create_channel(
+ cls, address="automl.googleapis.com:443", credentials=None, **kwargs
+ ):
+ """Create and return a gRPC channel object.
+
+ Args:
+ address (str): The host for the channel to use.
+ credentials (~.Credentials): The
+ authorization credentials to attach to requests. These
+ credentials identify this application to the service. If
+ none are specified, the client will attempt to ascertain
+ the credentials from the environment.
+ kwargs (dict): Keyword arguments, which are passed to the
+ channel creation.
+
+ Returns:
+ grpc.Channel: A gRPC channel object.
+ """
+ return google.api_core.grpc_helpers.create_channel(
+ address, credentials=credentials, scopes=cls._OAUTH_SCOPES, **kwargs
+ )
+
+ @property
+ def channel(self):
+ """The gRPC channel used by the transport.
+
+ Returns:
+ grpc.Channel: A gRPC channel object.
+ """
+ return self._channel
+
+ @property
+ def create_dataset(self):
+ """Return the gRPC stub for :meth:`AutoMlClient.create_dataset`.
+
+ Creates a dataset.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["auto_ml_stub"].CreateDataset
+
+ @property
+ def update_dataset(self):
+ """Return the gRPC stub for :meth:`AutoMlClient.update_dataset`.
+
+ Updates a dataset.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["auto_ml_stub"].UpdateDataset
+
+ @property
+ def get_dataset(self):
+ """Return the gRPC stub for :meth:`AutoMlClient.get_dataset`.
+
+ Gets a dataset.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["auto_ml_stub"].GetDataset
+
+ @property
+ def list_datasets(self):
+ """Return the gRPC stub for :meth:`AutoMlClient.list_datasets`.
+
+ Lists datasets in a project.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["auto_ml_stub"].ListDatasets
+
+ @property
+ def delete_dataset(self):
+ """Return the gRPC stub for :meth:`AutoMlClient.delete_dataset`.
+
+ Deletes a dataset and all of its contents. Returns empty response in the
+ ``response`` field when it completes, and ``delete_details`` in the
+ ``metadata`` field.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["auto_ml_stub"].DeleteDataset
+
+ @property
+ def import_data(self):
+ """Return the gRPC stub for :meth:`AutoMlClient.import_data`.
+
+ Imports data into a dataset. For Tables this method can only be called
+ on an empty Dataset.
+
+ For Tables:
+
+ - A ``schema_inference_version`` parameter must be explicitly set.
+ Returns an empty response in the ``response`` field when it
+ completes.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["auto_ml_stub"].ImportData
+
+ @property
+ def export_data(self):
+ """Return the gRPC stub for :meth:`AutoMlClient.export_data`.
+
+ Exports dataset's data to the provided output location. Returns an empty
+ response in the ``response`` field when it completes.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["auto_ml_stub"].ExportData
+
+ @property
+ def create_model(self):
+ """Return the gRPC stub for :meth:`AutoMlClient.create_model`.
+
+ Creates a model. Returns a Model in the ``response`` field when it
+ completes. When you create a model, several model evaluations are
+ created for it: a global evaluation, and one evaluation for each
+ annotation spec.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["auto_ml_stub"].CreateModel
+
+ @property
+ def get_model(self):
+ """Return the gRPC stub for :meth:`AutoMlClient.get_model`.
+
+ Gets a model.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["auto_ml_stub"].GetModel
+
+ @property
+ def list_models(self):
+ """Return the gRPC stub for :meth:`AutoMlClient.list_models`.
+
+ Lists models.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["auto_ml_stub"].ListModels
+
+ @property
+ def delete_model(self):
+ """Return the gRPC stub for :meth:`AutoMlClient.delete_model`.
+
+ Deletes a model. Returns ``google.protobuf.Empty`` in the ``response``
+ field when it completes, and ``delete_details`` in the ``metadata``
+ field.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["auto_ml_stub"].DeleteModel
+
+ @property
+ def deploy_model(self):
+ """Return the gRPC stub for :meth:`AutoMlClient.deploy_model`.
+
+ Deploys a model. If a model is already deployed, deploying it with the
+ same parameters has no effect. Deploying with different parametrs (as
+ e.g. changing
+
+ ``node_number``) will reset the deployment state without pausing the
+ model's availability.
+
+ Only applicable for Text Classification, Image Object Detection and
+ Tables; all other domains manage deployment automatically.
+
+ Returns an empty response in the ``response`` field when it completes.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["auto_ml_stub"].DeployModel
+
+ @property
+ def undeploy_model(self):
+ """Return the gRPC stub for :meth:`AutoMlClient.undeploy_model`.
+
+ Undeploys a model. If the model is not deployed this method has no
+ effect.
+
+ Only applicable for Text Classification, Image Object Detection and
+ Tables; all other domains manage deployment automatically.
+
+ Returns an empty response in the ``response`` field when it completes.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["auto_ml_stub"].UndeployModel
+
+ @property
+ def get_model_evaluation(self):
+ """Return the gRPC stub for :meth:`AutoMlClient.get_model_evaluation`.
+
+ Gets a model evaluation.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["auto_ml_stub"].GetModelEvaluation
+
+ @property
+ def export_model(self):
+ """Return the gRPC stub for :meth:`AutoMlClient.export_model`.
+
+ Exports a trained, "export-able", model to a user specified Google Cloud
+ Storage location. A model is considered export-able if and only if it
+ has an export format defined for it in
+
+ ``ModelExportOutputConfig``.
+
+ Returns an empty response in the ``response`` field when it completes.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["auto_ml_stub"].ExportModel
+
+ @property
+ def export_evaluated_examples(self):
+ """Return the gRPC stub for :meth:`AutoMlClient.export_evaluated_examples`.
+
+ Exports examples on which the model was evaluated (i.e. which were in
+ the TEST set of the dataset the model was created from), together with
+ their ground truth annotations and the annotations created (predicted)
+ by the model. The examples, ground truth and predictions are exported in
+ the state they were at the moment the model was evaluated.
+
+ This export is available only for 30 days since the model evaluation is
+ created.
+
+ Currently only available for Tables.
+
+ Returns an empty response in the ``response`` field when it completes.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["auto_ml_stub"].ExportEvaluatedExamples
+
+ @property
+ def list_model_evaluations(self):
+ """Return the gRPC stub for :meth:`AutoMlClient.list_model_evaluations`.
+
+ Lists model evaluations.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["auto_ml_stub"].ListModelEvaluations
+
+ @property
+ def get_annotation_spec(self):
+ """Return the gRPC stub for :meth:`AutoMlClient.get_annotation_spec`.
+
+ Gets an annotation spec.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["auto_ml_stub"].GetAnnotationSpec
+
+ @property
+ def get_table_spec(self):
+ """Return the gRPC stub for :meth:`AutoMlClient.get_table_spec`.
+
+ Gets a table spec.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["auto_ml_stub"].GetTableSpec
+
+ @property
+ def list_table_specs(self):
+ """Return the gRPC stub for :meth:`AutoMlClient.list_table_specs`.
+
+ Lists table specs in a dataset.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["auto_ml_stub"].ListTableSpecs
+
+ @property
+ def update_table_spec(self):
+ """Return the gRPC stub for :meth:`AutoMlClient.update_table_spec`.
+
+ Updates a table spec.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["auto_ml_stub"].UpdateTableSpec
+
+ @property
+ def get_column_spec(self):
+ """Return the gRPC stub for :meth:`AutoMlClient.get_column_spec`.
+
+ Gets a column spec.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["auto_ml_stub"].GetColumnSpec
+
+ @property
+ def list_column_specs(self):
+ """Return the gRPC stub for :meth:`AutoMlClient.list_column_specs`.
+
+ Lists column specs in a table spec.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["auto_ml_stub"].ListColumnSpecs
+
+ @property
+ def update_column_spec(self):
+ """Return the gRPC stub for :meth:`AutoMlClient.update_column_spec`.
+
+ Updates a column spec.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["auto_ml_stub"].UpdateColumnSpec
diff --git a/automl/google/cloud/automl_v1beta1/gapic/transports/prediction_service_grpc_transport.py b/automl/google/cloud/automl_v1beta1/gapic/transports/prediction_service_grpc_transport.py
new file mode 100644
index 000000000000..143711923f54
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/gapic/transports/prediction_service_grpc_transport.py
@@ -0,0 +1,175 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import google.api_core.grpc_helpers
+import google.api_core.operations_v1
+
+from google.cloud.automl_v1beta1.proto import prediction_service_pb2_grpc
+
+
+class PredictionServiceGrpcTransport(object):
+ """gRPC transport class providing stubs for
+ google.cloud.automl.v1beta1 PredictionService API.
+
+ The transport provides access to the raw gRPC stubs,
+ which can be used to take advantage of advanced
+ features of gRPC.
+ """
+
+ # The scopes needed to make gRPC calls to all of the methods defined
+ # in this service.
+ _OAUTH_SCOPES = ("https://www.googleapis.com/auth/cloud-platform",)
+
+ def __init__(
+ self, channel=None, credentials=None, address="automl.googleapis.com:443"
+ ):
+ """Instantiate the transport class.
+
+ Args:
+ channel (grpc.Channel): A ``Channel`` instance through
+ which to make calls. This argument is mutually exclusive
+ with ``credentials``; providing both will raise an exception.
+ credentials (google.auth.credentials.Credentials): The
+ authorization credentials to attach to requests. These
+ credentials identify this application to the service. If none
+ are specified, the client will attempt to ascertain the
+ credentials from the environment.
+ address (str): The address where the service is hosted.
+ """
+ # If both `channel` and `credentials` are specified, raise an
+ # exception (channels come with credentials baked in already).
+ if channel is not None and credentials is not None:
+ raise ValueError(
+ "The `channel` and `credentials` arguments are mutually " "exclusive."
+ )
+
+ # Create the channel.
+ if channel is None:
+ channel = self.create_channel(
+ address=address,
+ credentials=credentials,
+ options={
+ "grpc.max_send_message_length": -1,
+ "grpc.max_receive_message_length": -1,
+ }.items(),
+ )
+
+ self._channel = channel
+
+ # gRPC uses objects called "stubs" that are bound to the
+ # channel and provide a basic method for each RPC.
+ self._stubs = {
+ "prediction_service_stub": prediction_service_pb2_grpc.PredictionServiceStub(
+ channel
+ )
+ }
+
+ # Because this API includes a method that returns a
+ # long-running operation (proto: google.longrunning.Operation),
+ # instantiate an LRO client.
+ self._operations_client = google.api_core.operations_v1.OperationsClient(
+ channel
+ )
+
+ @classmethod
+ def create_channel(
+ cls, address="automl.googleapis.com:443", credentials=None, **kwargs
+ ):
+ """Create and return a gRPC channel object.
+
+ Args:
+ address (str): The host for the channel to use.
+ credentials (~.Credentials): The
+ authorization credentials to attach to requests. These
+ credentials identify this application to the service. If
+ none are specified, the client will attempt to ascertain
+ the credentials from the environment.
+ kwargs (dict): Keyword arguments, which are passed to the
+ channel creation.
+
+ Returns:
+ grpc.Channel: A gRPC channel object.
+ """
+ return google.api_core.grpc_helpers.create_channel(
+ address, credentials=credentials, scopes=cls._OAUTH_SCOPES, **kwargs
+ )
+
+ @property
+ def channel(self):
+ """The gRPC channel used by the transport.
+
+ Returns:
+ grpc.Channel: A gRPC channel object.
+ """
+ return self._channel
+
+ @property
+ def predict(self):
+ """Return the gRPC stub for :meth:`PredictionServiceClient.predict`.
+
+ Perform an online prediction. The prediction result will be directly
+ returned in the response. Available for following ML problems, and their
+ expected request payloads:
+
+ - Image Classification - Image in .JPEG, .GIF or .PNG format,
+ image\_bytes up to 30MB.
+ - Image Object Detection - Image in .JPEG, .GIF or .PNG format,
+ image\_bytes up to 30MB.
+ - Text Classification - TextSnippet, content up to 60,000 characters,
+ UTF-8 encoded.
+ - Text Extraction - TextSnippet, content up to 30,000 characters, UTF-8
+ NFC encoded.
+ - Translation - TextSnippet, content up to 25,000 characters, UTF-8
+ encoded.
+ - Tables - Row, with column values matching the columns of the model,
+ up to 5MB. Not available for FORECASTING
+
+ ``prediction_type``.
+
+ - Text Sentiment - TextSnippet, content up 500 characters, UTF-8
+ encoded.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["prediction_service_stub"].Predict
+
+ @property
+ def batch_predict(self):
+ """Return the gRPC stub for :meth:`PredictionServiceClient.batch_predict`.
+
+ Perform a batch prediction. Unlike the online ``Predict``, batch
+ prediction result won't be immediately available in the response.
+ Instead, a long running operation object is returned. User can poll the
+ operation result via ``GetOperation`` method. Once the operation is
+ done, ``BatchPredictResult`` is returned in the ``response`` field.
+ Available for following ML problems:
+
+ - Image Classification
+ - Image Object Detection
+ - Video Classification
+ - Video Object Tracking \* Text Extraction
+ - Tables
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["prediction_service_stub"].BatchPredict
diff --git a/automl/google/cloud/automl_v1beta1/proto/__init__.py b/automl/google/cloud/automl_v1beta1/proto/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/automl/google/cloud/automl_v1beta1/proto/annotation_payload.proto b/automl/google/cloud/automl_v1beta1/proto/annotation_payload.proto
new file mode 100644
index 000000000000..7cc2860f1597
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/annotation_payload.proto
@@ -0,0 +1,78 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.automl.v1beta1;
+
+import "google/cloud/automl/v1beta1/classification.proto";
+import "google/cloud/automl/v1beta1/detection.proto";
+import "google/cloud/automl/v1beta1/tables.proto";
+import "google/cloud/automl/v1beta1/text_extraction.proto";
+import "google/cloud/automl/v1beta1/text_sentiment.proto";
+import "google/cloud/automl/v1beta1/translation.proto";
+import "google/protobuf/any.proto";
+import "google/api/annotations.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl";
+option java_multiple_files = true;
+option java_package = "com.google.cloud.automl.v1beta1";
+option php_namespace = "Google\\Cloud\\AutoMl\\V1beta1";
+option ruby_package = "Google::Cloud::AutoML::V1beta1";
+
+// Contains annotation information that is relevant to AutoML.
+message AnnotationPayload {
+ // Output only . Additional information about the annotation
+ // specific to the AutoML domain.
+ oneof detail {
+ // Annotation details for translation.
+ TranslationAnnotation translation = 2;
+
+ // Annotation details for content or image classification.
+ ClassificationAnnotation classification = 3;
+
+ // Annotation details for image object detection.
+ ImageObjectDetectionAnnotation image_object_detection = 4;
+
+ // Annotation details for video classification.
+ // Returned for Video Classification predictions.
+ VideoClassificationAnnotation video_classification = 9;
+
+ // Annotation details for video object tracking.
+ VideoObjectTrackingAnnotation video_object_tracking = 8;
+
+ // Annotation details for text extraction.
+ TextExtractionAnnotation text_extraction = 6;
+
+ // Annotation details for text sentiment.
+ TextSentimentAnnotation text_sentiment = 7;
+
+ // Annotation details for Tables.
+ TablesAnnotation tables = 10;
+ }
+
+ // Output only . The resource ID of the annotation spec that
+ // this annotation pertains to. The annotation spec comes from either an
+ // ancestor dataset, or the dataset that was used to train the model in use.
+ string annotation_spec_id = 1;
+
+ // Output only. The value of
+ // [display_name][google.cloud.automl.v1beta1.AnnotationSpec.display_name]
+ // when the model was trained. Because this field returns a value at model
+ // training time, for different models trained using the same dataset, the
+ // returned value could be different as model owner could update the
+ // `display_name` between any two model training.
+ string display_name = 5;
+}
diff --git a/automl/google/cloud/automl_v1beta1/proto/annotation_payload_pb2.py b/automl/google/cloud/automl_v1beta1/proto/annotation_payload_pb2.py
new file mode 100644
index 000000000000..f05435b900ab
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/annotation_payload_pb2.py
@@ -0,0 +1,412 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/automl_v1beta1/proto/annotation_payload.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.cloud.automl_v1beta1.proto import (
+ classification_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_classification__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ detection_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_detection__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ tables_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_tables__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ text_extraction_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_text__extraction__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ text_sentiment_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_text__sentiment__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ translation_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_translation__pb2,
+)
+from google.protobuf import any_pb2 as google_dot_protobuf_dot_any__pb2
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/automl_v1beta1/proto/annotation_payload.proto",
+ package="google.cloud.automl.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\037com.google.cloud.automl.v1beta1P\001ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\312\002\033Google\\Cloud\\AutoMl\\V1beta1\352\002\036Google::Cloud::AutoML::V1beta1"
+ ),
+ serialized_pb=_b(
+ '\n:google/cloud/automl_v1beta1/proto/annotation_payload.proto\x12\x1bgoogle.cloud.automl.v1beta1\x1a\x36google/cloud/automl_v1beta1/proto/classification.proto\x1a\x31google/cloud/automl_v1beta1/proto/detection.proto\x1a.google/cloud/automl_v1beta1/proto/tables.proto\x1a\x37google/cloud/automl_v1beta1/proto/text_extraction.proto\x1a\x36google/cloud/automl_v1beta1/proto/text_sentiment.proto\x1a\x33google/cloud/automl_v1beta1/proto/translation.proto\x1a\x19google/protobuf/any.proto\x1a\x1cgoogle/api/annotations.proto"\xe6\x05\n\x11\x41nnotationPayload\x12I\n\x0btranslation\x18\x02 \x01(\x0b\x32\x32.google.cloud.automl.v1beta1.TranslationAnnotationH\x00\x12O\n\x0e\x63lassification\x18\x03 \x01(\x0b\x32\x35.google.cloud.automl.v1beta1.ClassificationAnnotationH\x00\x12]\n\x16image_object_detection\x18\x04 \x01(\x0b\x32;.google.cloud.automl.v1beta1.ImageObjectDetectionAnnotationH\x00\x12Z\n\x14video_classification\x18\t \x01(\x0b\x32:.google.cloud.automl.v1beta1.VideoClassificationAnnotationH\x00\x12[\n\x15video_object_tracking\x18\x08 \x01(\x0b\x32:.google.cloud.automl.v1beta1.VideoObjectTrackingAnnotationH\x00\x12P\n\x0ftext_extraction\x18\x06 \x01(\x0b\x32\x35.google.cloud.automl.v1beta1.TextExtractionAnnotationH\x00\x12N\n\x0etext_sentiment\x18\x07 \x01(\x0b\x32\x34.google.cloud.automl.v1beta1.TextSentimentAnnotationH\x00\x12?\n\x06tables\x18\n \x01(\x0b\x32-.google.cloud.automl.v1beta1.TablesAnnotationH\x00\x12\x1a\n\x12\x61nnotation_spec_id\x18\x01 \x01(\t\x12\x14\n\x0c\x64isplay_name\x18\x05 \x01(\tB\x08\n\x06\x64\x65tailB\xa5\x01\n\x1f\x63om.google.cloud.automl.v1beta1P\x01ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\xca\x02\x1bGoogle\\Cloud\\AutoMl\\V1beta1\xea\x02\x1eGoogle::Cloud::AutoML::V1beta1b\x06proto3'
+ ),
+ dependencies=[
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_classification__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_detection__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_tables__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_text__extraction__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_text__sentiment__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_translation__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_any__pb2.DESCRIPTOR,
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ ],
+)
+
+
+_ANNOTATIONPAYLOAD = _descriptor.Descriptor(
+ name="AnnotationPayload",
+ full_name="google.cloud.automl.v1beta1.AnnotationPayload",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="translation",
+ full_name="google.cloud.automl.v1beta1.AnnotationPayload.translation",
+ index=0,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="classification",
+ full_name="google.cloud.automl.v1beta1.AnnotationPayload.classification",
+ index=1,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="image_object_detection",
+ full_name="google.cloud.automl.v1beta1.AnnotationPayload.image_object_detection",
+ index=2,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="video_classification",
+ full_name="google.cloud.automl.v1beta1.AnnotationPayload.video_classification",
+ index=3,
+ number=9,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="video_object_tracking",
+ full_name="google.cloud.automl.v1beta1.AnnotationPayload.video_object_tracking",
+ index=4,
+ number=8,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="text_extraction",
+ full_name="google.cloud.automl.v1beta1.AnnotationPayload.text_extraction",
+ index=5,
+ number=6,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="text_sentiment",
+ full_name="google.cloud.automl.v1beta1.AnnotationPayload.text_sentiment",
+ index=6,
+ number=7,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="tables",
+ full_name="google.cloud.automl.v1beta1.AnnotationPayload.tables",
+ index=7,
+ number=10,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="annotation_spec_id",
+ full_name="google.cloud.automl.v1beta1.AnnotationPayload.annotation_spec_id",
+ index=8,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="display_name",
+ full_name="google.cloud.automl.v1beta1.AnnotationPayload.display_name",
+ index=9,
+ number=5,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="detail",
+ full_name="google.cloud.automl.v1beta1.AnnotationPayload.detail",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=470,
+ serialized_end=1212,
+)
+
+_ANNOTATIONPAYLOAD.fields_by_name[
+ "translation"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_translation__pb2._TRANSLATIONANNOTATION
+)
+_ANNOTATIONPAYLOAD.fields_by_name[
+ "classification"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_classification__pb2._CLASSIFICATIONANNOTATION
+)
+_ANNOTATIONPAYLOAD.fields_by_name[
+ "image_object_detection"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_detection__pb2._IMAGEOBJECTDETECTIONANNOTATION
+)
+_ANNOTATIONPAYLOAD.fields_by_name[
+ "video_classification"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_classification__pb2._VIDEOCLASSIFICATIONANNOTATION
+)
+_ANNOTATIONPAYLOAD.fields_by_name[
+ "video_object_tracking"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_detection__pb2._VIDEOOBJECTTRACKINGANNOTATION
+)
+_ANNOTATIONPAYLOAD.fields_by_name[
+ "text_extraction"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_text__extraction__pb2._TEXTEXTRACTIONANNOTATION
+)
+_ANNOTATIONPAYLOAD.fields_by_name[
+ "text_sentiment"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_text__sentiment__pb2._TEXTSENTIMENTANNOTATION
+)
+_ANNOTATIONPAYLOAD.fields_by_name[
+ "tables"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_tables__pb2._TABLESANNOTATION
+)
+_ANNOTATIONPAYLOAD.oneofs_by_name["detail"].fields.append(
+ _ANNOTATIONPAYLOAD.fields_by_name["translation"]
+)
+_ANNOTATIONPAYLOAD.fields_by_name[
+ "translation"
+].containing_oneof = _ANNOTATIONPAYLOAD.oneofs_by_name["detail"]
+_ANNOTATIONPAYLOAD.oneofs_by_name["detail"].fields.append(
+ _ANNOTATIONPAYLOAD.fields_by_name["classification"]
+)
+_ANNOTATIONPAYLOAD.fields_by_name[
+ "classification"
+].containing_oneof = _ANNOTATIONPAYLOAD.oneofs_by_name["detail"]
+_ANNOTATIONPAYLOAD.oneofs_by_name["detail"].fields.append(
+ _ANNOTATIONPAYLOAD.fields_by_name["image_object_detection"]
+)
+_ANNOTATIONPAYLOAD.fields_by_name[
+ "image_object_detection"
+].containing_oneof = _ANNOTATIONPAYLOAD.oneofs_by_name["detail"]
+_ANNOTATIONPAYLOAD.oneofs_by_name["detail"].fields.append(
+ _ANNOTATIONPAYLOAD.fields_by_name["video_classification"]
+)
+_ANNOTATIONPAYLOAD.fields_by_name[
+ "video_classification"
+].containing_oneof = _ANNOTATIONPAYLOAD.oneofs_by_name["detail"]
+_ANNOTATIONPAYLOAD.oneofs_by_name["detail"].fields.append(
+ _ANNOTATIONPAYLOAD.fields_by_name["video_object_tracking"]
+)
+_ANNOTATIONPAYLOAD.fields_by_name[
+ "video_object_tracking"
+].containing_oneof = _ANNOTATIONPAYLOAD.oneofs_by_name["detail"]
+_ANNOTATIONPAYLOAD.oneofs_by_name["detail"].fields.append(
+ _ANNOTATIONPAYLOAD.fields_by_name["text_extraction"]
+)
+_ANNOTATIONPAYLOAD.fields_by_name[
+ "text_extraction"
+].containing_oneof = _ANNOTATIONPAYLOAD.oneofs_by_name["detail"]
+_ANNOTATIONPAYLOAD.oneofs_by_name["detail"].fields.append(
+ _ANNOTATIONPAYLOAD.fields_by_name["text_sentiment"]
+)
+_ANNOTATIONPAYLOAD.fields_by_name[
+ "text_sentiment"
+].containing_oneof = _ANNOTATIONPAYLOAD.oneofs_by_name["detail"]
+_ANNOTATIONPAYLOAD.oneofs_by_name["detail"].fields.append(
+ _ANNOTATIONPAYLOAD.fields_by_name["tables"]
+)
+_ANNOTATIONPAYLOAD.fields_by_name[
+ "tables"
+].containing_oneof = _ANNOTATIONPAYLOAD.oneofs_by_name["detail"]
+DESCRIPTOR.message_types_by_name["AnnotationPayload"] = _ANNOTATIONPAYLOAD
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+AnnotationPayload = _reflection.GeneratedProtocolMessageType(
+ "AnnotationPayload",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_ANNOTATIONPAYLOAD,
+ __module__="google.cloud.automl_v1beta1.proto.annotation_payload_pb2",
+ __doc__="""Contains annotation information that is relevant to AutoML.
+
+
+ Attributes:
+ detail:
+ Output only . Additional information about the annotation
+ specific to the AutoML domain.
+ translation:
+ Annotation details for translation.
+ classification:
+ Annotation details for content or image classification.
+ image_object_detection:
+ Annotation details for image object detection.
+ video_classification:
+ Annotation details for video classification. Returned for
+ Video Classification predictions.
+ video_object_tracking:
+ Annotation details for video object tracking.
+ text_extraction:
+ Annotation details for text extraction.
+ text_sentiment:
+ Annotation details for text sentiment.
+ tables:
+ Annotation details for Tables.
+ annotation_spec_id:
+ Output only . The resource ID of the annotation spec that this
+ annotation pertains to. The annotation spec comes from either
+ an ancestor dataset, or the dataset that was used to train the
+ model in use.
+ display_name:
+ Output only. The value of [display\_name][google.cloud.automl.
+ v1beta1.AnnotationSpec.display\_name] when the model was
+ trained. Because this field returns a value at model training
+ time, for different models trained using the same dataset, the
+ returned value could be different as model owner could update
+ the ``display_name`` between any two model training.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.AnnotationPayload)
+ ),
+)
+_sym_db.RegisterMessage(AnnotationPayload)
+
+
+DESCRIPTOR._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/automl/google/cloud/automl_v1beta1/proto/annotation_payload_pb2_grpc.py b/automl/google/cloud/automl_v1beta1/proto/annotation_payload_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/annotation_payload_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/automl/google/cloud/automl_v1beta1/proto/annotation_spec.proto b/automl/google/cloud/automl_v1beta1/proto/annotation_spec.proto
new file mode 100644
index 000000000000..483792b6678a
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/annotation_spec.proto
@@ -0,0 +1,45 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.automl.v1beta1;
+
+import "google/api/annotations.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl";
+option java_multiple_files = true;
+option java_package = "com.google.cloud.automl.v1beta1";
+option php_namespace = "Google\\Cloud\\AutoMl\\V1beta1";
+option ruby_package = "Google::Cloud::AutoML::V1beta1";
+
+// A definition of an annotation spec.
+message AnnotationSpec {
+ // Output only. Resource name of the annotation spec.
+ // Form:
+ //
+ // 'projects/{project_id}/locations/{location_id}/datasets/{dataset_id}/annotationSpecs/{annotation_spec_id}'
+ string name = 1;
+
+ // Required.
+ // The name of the annotation spec to show in the interface. The name can be
+ // up to 32 characters long and must match the regexp `[a-zA-Z0-9_]+`.
+ // (_), and ASCII digits 0-9.
+ string display_name = 2;
+
+ // Output only. The number of examples in the parent dataset
+ // labeled by the annotation spec.
+ int32 example_count = 9;
+}
diff --git a/automl/google/cloud/automl_v1beta1/proto/annotation_spec_pb2.py b/automl/google/cloud/automl_v1beta1/proto/annotation_spec_pb2.py
new file mode 100644
index 000000000000..80ae0a8af98a
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/annotation_spec_pb2.py
@@ -0,0 +1,142 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/automl_v1beta1/proto/annotation_spec.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/automl_v1beta1/proto/annotation_spec.proto",
+ package="google.cloud.automl.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\037com.google.cloud.automl.v1beta1P\001ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\312\002\033Google\\Cloud\\AutoMl\\V1beta1\352\002\036Google::Cloud::AutoML::V1beta1"
+ ),
+ serialized_pb=_b(
+ '\n7google/cloud/automl_v1beta1/proto/annotation_spec.proto\x12\x1bgoogle.cloud.automl.v1beta1\x1a\x1cgoogle/api/annotations.proto"K\n\x0e\x41nnotationSpec\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x14\n\x0c\x64isplay_name\x18\x02 \x01(\t\x12\x15\n\rexample_count\x18\t \x01(\x05\x42\xa5\x01\n\x1f\x63om.google.cloud.automl.v1beta1P\x01ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\xca\x02\x1bGoogle\\Cloud\\AutoMl\\V1beta1\xea\x02\x1eGoogle::Cloud::AutoML::V1beta1b\x06proto3'
+ ),
+ dependencies=[google_dot_api_dot_annotations__pb2.DESCRIPTOR],
+)
+
+
+_ANNOTATIONSPEC = _descriptor.Descriptor(
+ name="AnnotationSpec",
+ full_name="google.cloud.automl.v1beta1.AnnotationSpec",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.automl.v1beta1.AnnotationSpec.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="display_name",
+ full_name="google.cloud.automl.v1beta1.AnnotationSpec.display_name",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="example_count",
+ full_name="google.cloud.automl.v1beta1.AnnotationSpec.example_count",
+ index=2,
+ number=9,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=118,
+ serialized_end=193,
+)
+
+DESCRIPTOR.message_types_by_name["AnnotationSpec"] = _ANNOTATIONSPEC
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+AnnotationSpec = _reflection.GeneratedProtocolMessageType(
+ "AnnotationSpec",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_ANNOTATIONSPEC,
+ __module__="google.cloud.automl_v1beta1.proto.annotation_spec_pb2",
+ __doc__="""A definition of an annotation spec.
+
+
+ Attributes:
+ name:
+ Output only. Resource name of the annotation spec. Form: 'pro
+ jects/{project\_id}/locations/{location\_id}/datasets/{dataset
+ \_id}/annotationSpecs/{annotation\_spec\_id}'
+ display_name:
+ Required. The name of the annotation spec to show in the
+ interface. The name can be up to 32 characters long and must
+ match the regexp ``[a-zA-Z0-9_]+``. (\_), and ASCII digits
+ 0-9.
+ example_count:
+ Output only. The number of examples in the parent dataset
+ labeled by the annotation spec.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.AnnotationSpec)
+ ),
+)
+_sym_db.RegisterMessage(AnnotationSpec)
+
+
+DESCRIPTOR._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/automl/google/cloud/automl_v1beta1/proto/annotation_spec_pb2_grpc.py b/automl/google/cloud/automl_v1beta1/proto/annotation_spec_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/annotation_spec_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/automl/google/cloud/automl_v1beta1/proto/classification.proto b/automl/google/cloud/automl_v1beta1/proto/classification.proto
new file mode 100644
index 000000000000..7e54238e0333
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/classification.proto
@@ -0,0 +1,217 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.automl.v1beta1;
+
+import "google/cloud/automl/v1beta1/temporal.proto";
+import "google/api/annotations.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl";
+option java_outer_classname = "ClassificationProto";
+option java_package = "com.google.cloud.automl.v1beta1";
+option php_namespace = "Google\\Cloud\\AutoMl\\V1beta1";
+option ruby_package = "Google::Cloud::AutoML::V1beta1";
+
+// Type of the classification problem.
+enum ClassificationType {
+ // An un-set value of this enum.
+ CLASSIFICATION_TYPE_UNSPECIFIED = 0;
+
+ // At most one label is allowed per example.
+ MULTICLASS = 1;
+
+ // Multiple labels are allowed for one example.
+ MULTILABEL = 2;
+}
+
+// Contains annotation details specific to classification.
+message ClassificationAnnotation {
+ // Output only. A confidence estimate between 0.0 and 1.0. A higher value
+ // means greater confidence that the annotation is positive. If a user
+ // approves an annotation as negative or positive, the score value remains
+ // unchanged. If a user creates an annotation, the score is 0 for negative or
+ // 1 for positive.
+ float score = 1;
+}
+
+// Contains annotation details specific to video classification.
+message VideoClassificationAnnotation {
+ // Output only. Expresses the type of video classification. Possible values:
+ //
+ // * `segment` - Classification done on a specified by user
+ // time segment of a video. AnnotationSpec is answered to be present
+ // in that time segment, if it is present in any part of it. The video
+ // ML model evaluations are done only for this type of classification.
+ //
+ // * `shot`- Shot-level classification.
+ // AutoML Video Intelligence determines the boundaries
+ // for each camera shot in the entire segment of the video that user
+ // specified in the request configuration. AutoML Video Intelligence
+ // then returns labels and their confidence scores for each detected
+ // shot, along with the start and end time of the shot.
+ // WARNING: Model evaluation is not done for this classification type,
+ // the quality of it depends on training data, but there are no
+ // metrics provided to describe that quality.
+ //
+ // * `1s_interval` - AutoML Video Intelligence returns labels and their
+ // confidence scores for each second of the entire segment of the video
+ // that user specified in the request configuration.
+ // WARNING: Model evaluation is not done for this classification type,
+ // the quality of it depends on training data, but there are no
+ // metrics provided to describe that quality.
+ string type = 1;
+
+ // Output only . The classification details of this annotation.
+ ClassificationAnnotation classification_annotation = 2;
+
+ // Output only . The time segment of the video to which the
+ // annotation applies.
+ TimeSegment time_segment = 3;
+}
+
+// Model evaluation metrics for classification problems.
+// Note: For Video Classification this metrics only describe quality of the
+// Video Classification predictions of "segment_classification" type.
+message ClassificationEvaluationMetrics {
+ // Metrics for a single confidence threshold.
+ message ConfidenceMetricsEntry {
+ // Output only. Metrics are computed with an assumption that the model
+ // never returns predictions with score lower than this value.
+ float confidence_threshold = 1;
+
+ // Output only. Metrics are computed with an assumption that the model
+ // always returns at most this many predictions (ordered by their score,
+ // descendingly), but they all still need to meet the confidence_threshold.
+ int32 position_threshold = 14;
+
+ // Output only. Recall (True Positive Rate) for the given confidence
+ // threshold.
+ float recall = 2;
+
+ // Output only. Precision for the given confidence threshold.
+ float precision = 3;
+
+ // Output only. False Positive Rate for the given confidence threshold.
+ float false_positive_rate = 8;
+
+ // Output only. The harmonic mean of recall and precision.
+ float f1_score = 4;
+
+ // Output only. The Recall (True Positive Rate) when only considering the
+ // label that has the highest prediction score and not below the confidence
+ // threshold for each example.
+ float recall_at1 = 5;
+
+ // Output only. The precision when only considering the label that has the
+ // highest prediction score and not below the confidence threshold for each
+ // example.
+ float precision_at1 = 6;
+
+ // Output only. The False Positive Rate when only considering the label that
+ // has the highest prediction score and not below the confidence threshold
+ // for each example.
+ float false_positive_rate_at1 = 9;
+
+ // Output only. The harmonic mean of [recall_at1][google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfidenceMetricsEntry.recall_at1] and [precision_at1][google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfidenceMetricsEntry.precision_at1].
+ float f1_score_at1 = 7;
+
+ // Output only. The number of model created labels that match a ground truth
+ // label.
+ int64 true_positive_count = 10;
+
+ // Output only. The number of model created labels that do not match a
+ // ground truth label.
+ int64 false_positive_count = 11;
+
+ // Output only. The number of ground truth labels that are not matched
+ // by a model created label.
+ int64 false_negative_count = 12;
+
+ // Output only. The number of labels that were not created by the model,
+ // but if they would, they would not match a ground truth label.
+ int64 true_negative_count = 13;
+ }
+
+ // Confusion matrix of the model running the classification.
+ message ConfusionMatrix {
+ // Output only. A row in the confusion matrix.
+ message Row {
+ // Output only. Value of the specific cell in the confusion matrix.
+ // The number of values each row has (i.e. the length of the row) is equal
+ // to the length of the `annotation_spec_id` field or, if that one is not
+ // populated, length of the [display_name][google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfusionMatrix.display_name] field.
+ repeated int32 example_count = 1;
+ }
+
+ // Output only. IDs of the annotation specs used in the confusion matrix.
+ // For Tables CLASSIFICATION
+ //
+ // [prediction_type][google.cloud.automl.v1beta1.TablesModelMetadata.prediction_type]
+ // only list of [annotation_spec_display_name-s][] is populated.
+ repeated string annotation_spec_id = 1;
+
+ // Output only. Display name of the annotation specs used in the confusion
+ // matrix, as they were at the moment of the evaluation. For Tables
+ // CLASSIFICATION
+ //
+ // [prediction_type-s][google.cloud.automl.v1beta1.TablesModelMetadata.prediction_type],
+ // distinct values of the target column at the moment of the model
+ // evaluation are populated here.
+ repeated string display_name = 3;
+
+ // Output only. Rows in the confusion matrix. The number of rows is equal to
+ // the size of `annotation_spec_id`.
+ // `row[i].value[j]` is the number of examples that have ground truth of the
+ // `annotation_spec_id[i]` and are predicted as `annotation_spec_id[j]` by
+ // the model being evaluated.
+ repeated Row row = 2;
+ }
+
+ // Output only. The Area Under Precision-Recall Curve metric. Micro-averaged
+ // for the overall evaluation.
+ float au_prc = 1;
+
+ // Output only. The Area Under Precision-Recall Curve metric based on priors.
+ // Micro-averaged for the overall evaluation.
+ // Deprecated.
+ float base_au_prc = 2 [deprecated = true];
+
+ // Output only. The Area Under Receiver Operating Characteristic curve metric.
+ // Micro-averaged for the overall evaluation.
+ float au_roc = 6;
+
+ // Output only. The Log Loss metric.
+ float log_loss = 7;
+
+ // Output only. Metrics for each confidence_threshold in
+ // 0.00,0.05,0.10,...,0.95,0.96,0.97,0.98,0.99 and
+ // position_threshold = INT32_MAX_VALUE.
+ // ROC and precision-recall curves, and other aggregated metrics are derived
+ // from them. The confidence metrics entries may also be supplied for
+ // additional values of position_threshold, but from these no aggregated
+ // metrics are computed.
+ repeated ConfidenceMetricsEntry confidence_metrics_entry = 3;
+
+ // Output only. Confusion matrix of the evaluation.
+ // Only set for MULTICLASS classification problems where number
+ // of labels is no more than 10.
+ // Only set for model level evaluation, not for evaluation per label.
+ ConfusionMatrix confusion_matrix = 4;
+
+ // Output only. The annotation spec ids used for this evaluation.
+ repeated string annotation_spec_id = 5;
+}
diff --git a/automl/google/cloud/automl_v1beta1/proto/classification_pb2.py b/automl/google/cloud/automl_v1beta1/proto/classification_pb2.py
new file mode 100644
index 000000000000..496a4a2844b9
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/classification_pb2.py
@@ -0,0 +1,991 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/automl_v1beta1/proto/classification.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf.internal import enum_type_wrapper
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.cloud.automl_v1beta1.proto import (
+ temporal_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_temporal__pb2,
+)
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/automl_v1beta1/proto/classification.proto",
+ package="google.cloud.automl.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\037com.google.cloud.automl.v1beta1B\023ClassificationProtoZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\312\002\033Google\\Cloud\\AutoMl\\V1beta1\352\002\036Google::Cloud::AutoML::V1beta1"
+ ),
+ serialized_pb=_b(
+ '\n6google/cloud/automl_v1beta1/proto/classification.proto\x12\x1bgoogle.cloud.automl.v1beta1\x1a\x30google/cloud/automl_v1beta1/proto/temporal.proto\x1a\x1cgoogle/api/annotations.proto")\n\x18\x43lassificationAnnotation\x12\r\n\x05score\x18\x01 \x01(\x02"\xc7\x01\n\x1dVideoClassificationAnnotation\x12\x0c\n\x04type\x18\x01 \x01(\t\x12X\n\x19\x63lassification_annotation\x18\x02 \x01(\x0b\x32\x35.google.cloud.automl.v1beta1.ClassificationAnnotation\x12>\n\x0ctime_segment\x18\x03 \x01(\x0b\x32(.google.cloud.automl.v1beta1.TimeSegment"\xa9\x07\n\x1f\x43lassificationEvaluationMetrics\x12\x0e\n\x06\x61u_prc\x18\x01 \x01(\x02\x12\x17\n\x0b\x62\x61se_au_prc\x18\x02 \x01(\x02\x42\x02\x18\x01\x12\x0e\n\x06\x61u_roc\x18\x06 \x01(\x02\x12\x10\n\x08log_loss\x18\x07 \x01(\x02\x12u\n\x18\x63onfidence_metrics_entry\x18\x03 \x03(\x0b\x32S.google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfidenceMetricsEntry\x12\x66\n\x10\x63onfusion_matrix\x18\x04 \x01(\x0b\x32L.google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfusionMatrix\x12\x1a\n\x12\x61nnotation_spec_id\x18\x05 \x03(\t\x1a\xfc\x02\n\x16\x43onfidenceMetricsEntry\x12\x1c\n\x14\x63onfidence_threshold\x18\x01 \x01(\x02\x12\x1a\n\x12position_threshold\x18\x0e \x01(\x05\x12\x0e\n\x06recall\x18\x02 \x01(\x02\x12\x11\n\tprecision\x18\x03 \x01(\x02\x12\x1b\n\x13\x66\x61lse_positive_rate\x18\x08 \x01(\x02\x12\x10\n\x08\x66\x31_score\x18\x04 \x01(\x02\x12\x12\n\nrecall_at1\x18\x05 \x01(\x02\x12\x15\n\rprecision_at1\x18\x06 \x01(\x02\x12\x1f\n\x17\x66\x61lse_positive_rate_at1\x18\t \x01(\x02\x12\x14\n\x0c\x66\x31_score_at1\x18\x07 \x01(\x02\x12\x1b\n\x13true_positive_count\x18\n \x01(\x03\x12\x1c\n\x14\x66\x61lse_positive_count\x18\x0b \x01(\x03\x12\x1c\n\x14\x66\x61lse_negative_count\x18\x0c \x01(\x03\x12\x1b\n\x13true_negative_count\x18\r \x01(\x03\x1a\xc0\x01\n\x0f\x43onfusionMatrix\x12\x1a\n\x12\x61nnotation_spec_id\x18\x01 \x03(\t\x12\x14\n\x0c\x64isplay_name\x18\x03 \x03(\t\x12]\n\x03row\x18\x02 \x03(\x0b\x32P.google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfusionMatrix.Row\x1a\x1c\n\x03Row\x12\x15\n\rexample_count\x18\x01 \x03(\x05*Y\n\x12\x43lassificationType\x12#\n\x1f\x43LASSIFICATION_TYPE_UNSPECIFIED\x10\x00\x12\x0e\n\nMULTICLASS\x10\x01\x12\x0e\n\nMULTILABEL\x10\x02\x42\xb8\x01\n\x1f\x63om.google.cloud.automl.v1beta1B\x13\x43lassificationProtoZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\xca\x02\x1bGoogle\\Cloud\\AutoMl\\V1beta1\xea\x02\x1eGoogle::Cloud::AutoML::V1beta1b\x06proto3'
+ ),
+ dependencies=[
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_temporal__pb2.DESCRIPTOR,
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ ],
+)
+
+_CLASSIFICATIONTYPE = _descriptor.EnumDescriptor(
+ name="ClassificationType",
+ full_name="google.cloud.automl.v1beta1.ClassificationType",
+ filename=None,
+ file=DESCRIPTOR,
+ values=[
+ _descriptor.EnumValueDescriptor(
+ name="CLASSIFICATION_TYPE_UNSPECIFIED",
+ index=0,
+ number=0,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="MULTICLASS", index=1, number=1, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="MULTILABEL", index=2, number=2, serialized_options=None, type=None
+ ),
+ ],
+ containing_type=None,
+ serialized_options=None,
+ serialized_start=1352,
+ serialized_end=1441,
+)
+_sym_db.RegisterEnumDescriptor(_CLASSIFICATIONTYPE)
+
+ClassificationType = enum_type_wrapper.EnumTypeWrapper(_CLASSIFICATIONTYPE)
+CLASSIFICATION_TYPE_UNSPECIFIED = 0
+MULTICLASS = 1
+MULTILABEL = 2
+
+
+_CLASSIFICATIONANNOTATION = _descriptor.Descriptor(
+ name="ClassificationAnnotation",
+ full_name="google.cloud.automl.v1beta1.ClassificationAnnotation",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="score",
+ full_name="google.cloud.automl.v1beta1.ClassificationAnnotation.score",
+ index=0,
+ number=1,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=167,
+ serialized_end=208,
+)
+
+
+_VIDEOCLASSIFICATIONANNOTATION = _descriptor.Descriptor(
+ name="VideoClassificationAnnotation",
+ full_name="google.cloud.automl.v1beta1.VideoClassificationAnnotation",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="type",
+ full_name="google.cloud.automl.v1beta1.VideoClassificationAnnotation.type",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="classification_annotation",
+ full_name="google.cloud.automl.v1beta1.VideoClassificationAnnotation.classification_annotation",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="time_segment",
+ full_name="google.cloud.automl.v1beta1.VideoClassificationAnnotation.time_segment",
+ index=2,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=211,
+ serialized_end=410,
+)
+
+
+_CLASSIFICATIONEVALUATIONMETRICS_CONFIDENCEMETRICSENTRY = _descriptor.Descriptor(
+ name="ConfidenceMetricsEntry",
+ full_name="google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfidenceMetricsEntry",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="confidence_threshold",
+ full_name="google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfidenceMetricsEntry.confidence_threshold",
+ index=0,
+ number=1,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="position_threshold",
+ full_name="google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfidenceMetricsEntry.position_threshold",
+ index=1,
+ number=14,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="recall",
+ full_name="google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfidenceMetricsEntry.recall",
+ index=2,
+ number=2,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="precision",
+ full_name="google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfidenceMetricsEntry.precision",
+ index=3,
+ number=3,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="false_positive_rate",
+ full_name="google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfidenceMetricsEntry.false_positive_rate",
+ index=4,
+ number=8,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="f1_score",
+ full_name="google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfidenceMetricsEntry.f1_score",
+ index=5,
+ number=4,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="recall_at1",
+ full_name="google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfidenceMetricsEntry.recall_at1",
+ index=6,
+ number=5,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="precision_at1",
+ full_name="google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfidenceMetricsEntry.precision_at1",
+ index=7,
+ number=6,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="false_positive_rate_at1",
+ full_name="google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfidenceMetricsEntry.false_positive_rate_at1",
+ index=8,
+ number=9,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="f1_score_at1",
+ full_name="google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfidenceMetricsEntry.f1_score_at1",
+ index=9,
+ number=7,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="true_positive_count",
+ full_name="google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfidenceMetricsEntry.true_positive_count",
+ index=10,
+ number=10,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="false_positive_count",
+ full_name="google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfidenceMetricsEntry.false_positive_count",
+ index=11,
+ number=11,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="false_negative_count",
+ full_name="google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfidenceMetricsEntry.false_negative_count",
+ index=12,
+ number=12,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="true_negative_count",
+ full_name="google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfidenceMetricsEntry.true_negative_count",
+ index=13,
+ number=13,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=775,
+ serialized_end=1155,
+)
+
+_CLASSIFICATIONEVALUATIONMETRICS_CONFUSIONMATRIX_ROW = _descriptor.Descriptor(
+ name="Row",
+ full_name="google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfusionMatrix.Row",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="example_count",
+ full_name="google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfusionMatrix.Row.example_count",
+ index=0,
+ number=1,
+ type=5,
+ cpp_type=1,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1322,
+ serialized_end=1350,
+)
+
+_CLASSIFICATIONEVALUATIONMETRICS_CONFUSIONMATRIX = _descriptor.Descriptor(
+ name="ConfusionMatrix",
+ full_name="google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfusionMatrix",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="annotation_spec_id",
+ full_name="google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfusionMatrix.annotation_spec_id",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="display_name",
+ full_name="google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfusionMatrix.display_name",
+ index=1,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="row",
+ full_name="google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfusionMatrix.row",
+ index=2,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[_CLASSIFICATIONEVALUATIONMETRICS_CONFUSIONMATRIX_ROW],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1158,
+ serialized_end=1350,
+)
+
+_CLASSIFICATIONEVALUATIONMETRICS = _descriptor.Descriptor(
+ name="ClassificationEvaluationMetrics",
+ full_name="google.cloud.automl.v1beta1.ClassificationEvaluationMetrics",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="au_prc",
+ full_name="google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.au_prc",
+ index=0,
+ number=1,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="base_au_prc",
+ full_name="google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.base_au_prc",
+ index=1,
+ number=2,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=_b("\030\001"),
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="au_roc",
+ full_name="google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.au_roc",
+ index=2,
+ number=6,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="log_loss",
+ full_name="google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.log_loss",
+ index=3,
+ number=7,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="confidence_metrics_entry",
+ full_name="google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.confidence_metrics_entry",
+ index=4,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="confusion_matrix",
+ full_name="google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.confusion_matrix",
+ index=5,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="annotation_spec_id",
+ full_name="google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.annotation_spec_id",
+ index=6,
+ number=5,
+ type=9,
+ cpp_type=9,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[
+ _CLASSIFICATIONEVALUATIONMETRICS_CONFIDENCEMETRICSENTRY,
+ _CLASSIFICATIONEVALUATIONMETRICS_CONFUSIONMATRIX,
+ ],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=413,
+ serialized_end=1350,
+)
+
+_VIDEOCLASSIFICATIONANNOTATION.fields_by_name[
+ "classification_annotation"
+].message_type = _CLASSIFICATIONANNOTATION
+_VIDEOCLASSIFICATIONANNOTATION.fields_by_name[
+ "time_segment"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_temporal__pb2._TIMESEGMENT
+)
+_CLASSIFICATIONEVALUATIONMETRICS_CONFIDENCEMETRICSENTRY.containing_type = (
+ _CLASSIFICATIONEVALUATIONMETRICS
+)
+_CLASSIFICATIONEVALUATIONMETRICS_CONFUSIONMATRIX_ROW.containing_type = (
+ _CLASSIFICATIONEVALUATIONMETRICS_CONFUSIONMATRIX
+)
+_CLASSIFICATIONEVALUATIONMETRICS_CONFUSIONMATRIX.fields_by_name[
+ "row"
+].message_type = _CLASSIFICATIONEVALUATIONMETRICS_CONFUSIONMATRIX_ROW
+_CLASSIFICATIONEVALUATIONMETRICS_CONFUSIONMATRIX.containing_type = (
+ _CLASSIFICATIONEVALUATIONMETRICS
+)
+_CLASSIFICATIONEVALUATIONMETRICS.fields_by_name[
+ "confidence_metrics_entry"
+].message_type = _CLASSIFICATIONEVALUATIONMETRICS_CONFIDENCEMETRICSENTRY
+_CLASSIFICATIONEVALUATIONMETRICS.fields_by_name[
+ "confusion_matrix"
+].message_type = _CLASSIFICATIONEVALUATIONMETRICS_CONFUSIONMATRIX
+DESCRIPTOR.message_types_by_name["ClassificationAnnotation"] = _CLASSIFICATIONANNOTATION
+DESCRIPTOR.message_types_by_name[
+ "VideoClassificationAnnotation"
+] = _VIDEOCLASSIFICATIONANNOTATION
+DESCRIPTOR.message_types_by_name[
+ "ClassificationEvaluationMetrics"
+] = _CLASSIFICATIONEVALUATIONMETRICS
+DESCRIPTOR.enum_types_by_name["ClassificationType"] = _CLASSIFICATIONTYPE
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+ClassificationAnnotation = _reflection.GeneratedProtocolMessageType(
+ "ClassificationAnnotation",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_CLASSIFICATIONANNOTATION,
+ __module__="google.cloud.automl_v1beta1.proto.classification_pb2",
+ __doc__="""Contains annotation details specific to classification.
+
+
+ Attributes:
+ score:
+ Output only. A confidence estimate between 0.0 and 1.0. A
+ higher value means greater confidence that the annotation is
+ positive. If a user approves an annotation as negative or
+ positive, the score value remains unchanged. If a user creates
+ an annotation, the score is 0 for negative or 1 for positive.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ClassificationAnnotation)
+ ),
+)
+_sym_db.RegisterMessage(ClassificationAnnotation)
+
+VideoClassificationAnnotation = _reflection.GeneratedProtocolMessageType(
+ "VideoClassificationAnnotation",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_VIDEOCLASSIFICATIONANNOTATION,
+ __module__="google.cloud.automl_v1beta1.proto.classification_pb2",
+ __doc__="""Contains annotation details specific to video classification.
+
+
+ Attributes:
+ type:
+ Output only. Expresses the type of video classification.
+ Possible values: - ``segment`` - Classification done on a
+ specified by user time segment of a video. AnnotationSpec
+ is answered to be present in that time segment, if it is
+ present in any part of it. The video ML model evaluations
+ are done only for this type of classification. - ``shot``-
+ Shot-level classification. AutoML Video Intelligence
+ determines the boundaries for each camera shot in the entire
+ segment of the video that user specified in the request
+ configuration. AutoML Video Intelligence then returns
+ labels and their confidence scores for each detected shot,
+ along with the start and end time of the shot. WARNING:
+ Model evaluation is not done for this classification type,
+ the quality of it depends on training data, but there are no
+ metrics provided to describe that quality. - ``1s_interval``
+ - AutoML Video Intelligence returns labels and their
+ confidence scores for each second of the entire segment of the
+ video that user specified in the request configuration.
+ WARNING: Model evaluation is not done for this
+ classification type, the quality of it depends on training
+ data, but there are no metrics provided to describe that
+ quality.
+ classification_annotation:
+ Output only . The classification details of this annotation.
+ time_segment:
+ Output only . The time segment of the video to which the
+ annotation applies.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.VideoClassificationAnnotation)
+ ),
+)
+_sym_db.RegisterMessage(VideoClassificationAnnotation)
+
+ClassificationEvaluationMetrics = _reflection.GeneratedProtocolMessageType(
+ "ClassificationEvaluationMetrics",
+ (_message.Message,),
+ dict(
+ ConfidenceMetricsEntry=_reflection.GeneratedProtocolMessageType(
+ "ConfidenceMetricsEntry",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_CLASSIFICATIONEVALUATIONMETRICS_CONFIDENCEMETRICSENTRY,
+ __module__="google.cloud.automl_v1beta1.proto.classification_pb2",
+ __doc__="""Metrics for a single confidence threshold.
+
+
+ Attributes:
+ confidence_threshold:
+ Output only. Metrics are computed with an assumption that the
+ model never returns predictions with score lower than this
+ value.
+ position_threshold:
+ Output only. Metrics are computed with an assumption that the
+ model always returns at most this many predictions (ordered by
+ their score, descendingly), but they all still need to meet
+ the confidence\_threshold.
+ recall:
+ Output only. Recall (True Positive Rate) for the given
+ confidence threshold.
+ precision:
+ Output only. Precision for the given confidence threshold.
+ false_positive_rate:
+ Output only. False Positive Rate for the given confidence
+ threshold.
+ f1_score:
+ Output only. The harmonic mean of recall and precision.
+ recall_at1:
+ Output only. The Recall (True Positive Rate) when only
+ considering the label that has the highest prediction score
+ and not below the confidence threshold for each example.
+ precision_at1:
+ Output only. The precision when only considering the label
+ that has the highest prediction score and not below the
+ confidence threshold for each example.
+ false_positive_rate_at1:
+ Output only. The False Positive Rate when only considering the
+ label that has the highest prediction score and not below the
+ confidence threshold for each example.
+ f1_score_at1:
+ Output only. The harmonic mean of [recall\_at1][google.cloud.a
+ utoml.v1beta1.ClassificationEvaluationMetrics.ConfidenceMetric
+ sEntry.recall\_at1] and [precision\_at1][google.cloud.automl.v
+ 1beta1.ClassificationEvaluationMetrics.ConfidenceMetricsEntry.
+ precision\_at1].
+ true_positive_count:
+ Output only. The number of model created labels that match a
+ ground truth label.
+ false_positive_count:
+ Output only. The number of model created labels that do not
+ match a ground truth label.
+ false_negative_count:
+ Output only. The number of ground truth labels that are not
+ matched by a model created label.
+ true_negative_count:
+ Output only. The number of labels that were not created by the
+ model, but if they would, they would not match a ground truth
+ label.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfidenceMetricsEntry)
+ ),
+ ),
+ ConfusionMatrix=_reflection.GeneratedProtocolMessageType(
+ "ConfusionMatrix",
+ (_message.Message,),
+ dict(
+ Row=_reflection.GeneratedProtocolMessageType(
+ "Row",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_CLASSIFICATIONEVALUATIONMETRICS_CONFUSIONMATRIX_ROW,
+ __module__="google.cloud.automl_v1beta1.proto.classification_pb2",
+ __doc__="""Output only. A row in the confusion matrix.
+
+
+ Attributes:
+ example_count:
+ Output only. Value of the specific cell in the confusion
+ matrix. The number of values each row has (i.e. the length of
+ the row) is equal to the length of the ``annotation_spec_id``
+ field or, if that one is not populated, length of the [display
+ \_name][google.cloud.automl.v1beta1.ClassificationEvaluationMe
+ trics.ConfusionMatrix.display\_name] field.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfusionMatrix.Row)
+ ),
+ ),
+ DESCRIPTOR=_CLASSIFICATIONEVALUATIONMETRICS_CONFUSIONMATRIX,
+ __module__="google.cloud.automl_v1beta1.proto.classification_pb2",
+ __doc__="""Confusion matrix of the model running the classification.
+
+
+ Attributes:
+ annotation_spec_id:
+ Output only. IDs of the annotation specs used in the confusion
+ matrix. For Tables CLASSIFICATION [prediction\_type][google.c
+ loud.automl.v1beta1.TablesModelMetadata.prediction\_type] only
+ list of [annotation\_spec\_display\_name-s][] is populated.
+ display_name:
+ Output only. Display name of the annotation specs used in the
+ confusion matrix, as they were at the moment of the
+ evaluation. For Tables CLASSIFICATION [prediction\_type-s][go
+ ogle.cloud.automl.v1beta1.TablesModelMetadata.prediction\_type
+ ], distinct values of the target column at the moment of the
+ model evaluation are populated here.
+ row:
+ Output only. Rows in the confusion matrix. The number of rows
+ is equal to the size of ``annotation_spec_id``.
+ ``row[i].value[j]`` is the number of examples that have ground
+ truth of the ``annotation_spec_id[i]`` and are predicted as
+ ``annotation_spec_id[j]`` by the model being evaluated.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfusionMatrix)
+ ),
+ ),
+ DESCRIPTOR=_CLASSIFICATIONEVALUATIONMETRICS,
+ __module__="google.cloud.automl_v1beta1.proto.classification_pb2",
+ __doc__="""Model evaluation metrics for classification problems. Note: For Video
+ Classification this metrics only describe quality of the Video
+ Classification predictions of "segment\_classification" type.
+
+
+ Attributes:
+ au_prc:
+ Output only. The Area Under Precision-Recall Curve metric.
+ Micro-averaged for the overall evaluation.
+ base_au_prc:
+ Output only. The Area Under Precision-Recall Curve metric
+ based on priors. Micro-averaged for the overall evaluation.
+ Deprecated.
+ au_roc:
+ Output only. The Area Under Receiver Operating Characteristic
+ curve metric. Micro-averaged for the overall evaluation.
+ log_loss:
+ Output only. The Log Loss metric.
+ confidence_metrics_entry:
+ Output only. Metrics for each confidence\_threshold in
+ 0.00,0.05,0.10,...,0.95,0.96,0.97,0.98,0.99 and
+ position\_threshold = INT32\_MAX\_VALUE. ROC and precision-
+ recall curves, and other aggregated metrics are derived from
+ them. The confidence metrics entries may also be supplied for
+ additional values of position\_threshold, but from these no
+ aggregated metrics are computed.
+ confusion_matrix:
+ Output only. Confusion matrix of the evaluation. Only set for
+ MULTICLASS classification problems where number of labels is
+ no more than 10. Only set for model level evaluation, not for
+ evaluation per label.
+ annotation_spec_id:
+ Output only. The annotation spec ids used for this evaluation.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ClassificationEvaluationMetrics)
+ ),
+)
+_sym_db.RegisterMessage(ClassificationEvaluationMetrics)
+_sym_db.RegisterMessage(ClassificationEvaluationMetrics.ConfidenceMetricsEntry)
+_sym_db.RegisterMessage(ClassificationEvaluationMetrics.ConfusionMatrix)
+_sym_db.RegisterMessage(ClassificationEvaluationMetrics.ConfusionMatrix.Row)
+
+
+DESCRIPTOR._options = None
+_CLASSIFICATIONEVALUATIONMETRICS.fields_by_name["base_au_prc"]._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/automl/google/cloud/automl_v1beta1/proto/classification_pb2_grpc.py b/automl/google/cloud/automl_v1beta1/proto/classification_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/classification_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/automl/google/cloud/automl_v1beta1/proto/column_spec.proto b/automl/google/cloud/automl_v1beta1/proto/column_spec.proto
new file mode 100644
index 000000000000..b8f437f793ee
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/column_spec.proto
@@ -0,0 +1,73 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.automl.v1beta1;
+
+import "google/cloud/automl/v1beta1/data_stats.proto";
+import "google/cloud/automl/v1beta1/data_types.proto";
+import "google/api/annotations.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl";
+option java_multiple_files = true;
+option java_package = "com.google.cloud.automl.v1beta1";
+option php_namespace = "Google\\Cloud\\AutoMl\\V1beta1";
+option ruby_package = "Google::Cloud::AutoML::V1beta1";
+
+// A representation of a column in a relational table. When listing them, column specs are returned in the same order in which they were
+// given on import .
+// Used by:
+// * Tables
+message ColumnSpec {
+ // Identifies the table's column, and its correlation with the column this
+ // ColumnSpec describes.
+ message CorrelatedColumn {
+ // The column_spec_id of the correlated column, which belongs to the same
+ // table as the in-context column.
+ string column_spec_id = 1;
+
+ // Correlation between this and the in-context column.
+ CorrelationStats correlation_stats = 2;
+ }
+
+ // Output only. The resource name of the column specs.
+ // Form:
+ //
+ // `projects/{project_id}/locations/{location_id}/datasets/{dataset_id}/tableSpecs/{table_spec_id}/columnSpecs/{column_spec_id}`
+ string name = 1;
+
+ // The data type of elements stored in the column.
+ DataType data_type = 2;
+
+ // Output only. The name of the column to show in the interface. The name can
+ // be up to 100 characters long and can consist only of ASCII Latin letters
+ // A-Z and a-z, ASCII digits 0-9, underscores(_), and forward slashes(/), and
+ // must start with a letter or a digit.
+ string display_name = 3;
+
+ // Output only. Stats of the series of values in the column.
+ // This field may be stale, see the ancestor's
+ // Dataset.tables_dataset_metadata.stats_update_time field
+ // for the timestamp at which these stats were last updated.
+ DataStats data_stats = 4;
+
+ // Deprecated.
+ repeated CorrelatedColumn top_correlated_columns = 5;
+
+ // Used to perform consistent read-modify-write updates. If not set, a blind
+ // "overwrite" update happens.
+ string etag = 6;
+}
diff --git a/automl/google/cloud/automl_v1beta1/proto/column_spec_pb2.py b/automl/google/cloud/automl_v1beta1/proto/column_spec_pb2.py
new file mode 100644
index 000000000000..1a6685c98346
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/column_spec_pb2.py
@@ -0,0 +1,314 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/automl_v1beta1/proto/column_spec.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.cloud.automl_v1beta1.proto import (
+ data_stats_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_data__stats__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ data_types_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_data__types__pb2,
+)
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/automl_v1beta1/proto/column_spec.proto",
+ package="google.cloud.automl.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\037com.google.cloud.automl.v1beta1P\001ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\312\002\033Google\\Cloud\\AutoMl\\V1beta1\352\002\036Google::Cloud::AutoML::V1beta1"
+ ),
+ serialized_pb=_b(
+ '\n3google/cloud/automl_v1beta1/proto/column_spec.proto\x12\x1bgoogle.cloud.automl.v1beta1\x1a\x32google/cloud/automl_v1beta1/proto/data_stats.proto\x1a\x32google/cloud/automl_v1beta1/proto/data_types.proto\x1a\x1cgoogle/api/annotations.proto"\x84\x03\n\nColumnSpec\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x38\n\tdata_type\x18\x02 \x01(\x0b\x32%.google.cloud.automl.v1beta1.DataType\x12\x14\n\x0c\x64isplay_name\x18\x03 \x01(\t\x12:\n\ndata_stats\x18\x04 \x01(\x0b\x32&.google.cloud.automl.v1beta1.DataStats\x12X\n\x16top_correlated_columns\x18\x05 \x03(\x0b\x32\x38.google.cloud.automl.v1beta1.ColumnSpec.CorrelatedColumn\x12\x0c\n\x04\x65tag\x18\x06 \x01(\t\x1at\n\x10\x43orrelatedColumn\x12\x16\n\x0e\x63olumn_spec_id\x18\x01 \x01(\t\x12H\n\x11\x63orrelation_stats\x18\x02 \x01(\x0b\x32-.google.cloud.automl.v1beta1.CorrelationStatsB\xa5\x01\n\x1f\x63om.google.cloud.automl.v1beta1P\x01ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\xca\x02\x1bGoogle\\Cloud\\AutoMl\\V1beta1\xea\x02\x1eGoogle::Cloud::AutoML::V1beta1b\x06proto3'
+ ),
+ dependencies=[
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_data__stats__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_data__types__pb2.DESCRIPTOR,
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ ],
+)
+
+
+_COLUMNSPEC_CORRELATEDCOLUMN = _descriptor.Descriptor(
+ name="CorrelatedColumn",
+ full_name="google.cloud.automl.v1beta1.ColumnSpec.CorrelatedColumn",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="column_spec_id",
+ full_name="google.cloud.automl.v1beta1.ColumnSpec.CorrelatedColumn.column_spec_id",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="correlation_stats",
+ full_name="google.cloud.automl.v1beta1.ColumnSpec.CorrelatedColumn.correlation_stats",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=491,
+ serialized_end=607,
+)
+
+_COLUMNSPEC = _descriptor.Descriptor(
+ name="ColumnSpec",
+ full_name="google.cloud.automl.v1beta1.ColumnSpec",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.automl.v1beta1.ColumnSpec.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="data_type",
+ full_name="google.cloud.automl.v1beta1.ColumnSpec.data_type",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="display_name",
+ full_name="google.cloud.automl.v1beta1.ColumnSpec.display_name",
+ index=2,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="data_stats",
+ full_name="google.cloud.automl.v1beta1.ColumnSpec.data_stats",
+ index=3,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="top_correlated_columns",
+ full_name="google.cloud.automl.v1beta1.ColumnSpec.top_correlated_columns",
+ index=4,
+ number=5,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="etag",
+ full_name="google.cloud.automl.v1beta1.ColumnSpec.etag",
+ index=5,
+ number=6,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[_COLUMNSPEC_CORRELATEDCOLUMN],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=219,
+ serialized_end=607,
+)
+
+_COLUMNSPEC_CORRELATEDCOLUMN.fields_by_name[
+ "correlation_stats"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_data__stats__pb2._CORRELATIONSTATS
+)
+_COLUMNSPEC_CORRELATEDCOLUMN.containing_type = _COLUMNSPEC
+_COLUMNSPEC.fields_by_name[
+ "data_type"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_data__types__pb2._DATATYPE
+)
+_COLUMNSPEC.fields_by_name[
+ "data_stats"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_data__stats__pb2._DATASTATS
+)
+_COLUMNSPEC.fields_by_name[
+ "top_correlated_columns"
+].message_type = _COLUMNSPEC_CORRELATEDCOLUMN
+DESCRIPTOR.message_types_by_name["ColumnSpec"] = _COLUMNSPEC
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+ColumnSpec = _reflection.GeneratedProtocolMessageType(
+ "ColumnSpec",
+ (_message.Message,),
+ dict(
+ CorrelatedColumn=_reflection.GeneratedProtocolMessageType(
+ "CorrelatedColumn",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_COLUMNSPEC_CORRELATEDCOLUMN,
+ __module__="google.cloud.automl_v1beta1.proto.column_spec_pb2",
+ __doc__="""Identifies the table's column, and its correlation with the column this
+ ColumnSpec describes.
+
+
+ Attributes:
+ column_spec_id:
+ The column\_spec\_id of the correlated column, which belongs
+ to the same table as the in-context column.
+ correlation_stats:
+ Correlation between this and the in-context column.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ColumnSpec.CorrelatedColumn)
+ ),
+ ),
+ DESCRIPTOR=_COLUMNSPEC,
+ __module__="google.cloud.automl_v1beta1.proto.column_spec_pb2",
+ __doc__="""A representation of a column in a relational table. When listing them,
+ column specs are returned in the same order in which they were given on
+ import . Used by: \* Tables
+
+
+ Attributes:
+ name:
+ Output only. The resource name of the column specs. Form: ``p
+ rojects/{project_id}/locations/{location_id}/datasets/{dataset
+ _id}/tableSpecs/{table_spec_id}/columnSpecs/{column_spec_id}``
+ data_type:
+ The data type of elements stored in the column.
+ display_name:
+ Output only. The name of the column to show in the interface.
+ The name can be up to 100 characters long and can consist only
+ of ASCII Latin letters A-Z and a-z, ASCII digits 0-9,
+ underscores(\_), and forward slashes(/), and must start with a
+ letter or a digit.
+ data_stats:
+ Output only. Stats of the series of values in the column. This
+ field may be stale, see the ancestor's
+ Dataset.tables\_dataset\_metadata.stats\_update\_time field
+ for the timestamp at which these stats were last updated.
+ top_correlated_columns:
+ Deprecated.
+ etag:
+ Used to perform consistent read-modify-write updates. If not
+ set, a blind "overwrite" update happens.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ColumnSpec)
+ ),
+)
+_sym_db.RegisterMessage(ColumnSpec)
+_sym_db.RegisterMessage(ColumnSpec.CorrelatedColumn)
+
+
+DESCRIPTOR._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/automl/google/cloud/automl_v1beta1/proto/column_spec_pb2_grpc.py b/automl/google/cloud/automl_v1beta1/proto/column_spec_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/column_spec_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/automl/google/cloud/automl_v1beta1/proto/data_items.proto b/automl/google/cloud/automl_v1beta1/proto/data_items.proto
new file mode 100644
index 000000000000..fdbea7042d84
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/data_items.proto
@@ -0,0 +1,229 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.automl.v1beta1;
+
+import "google/api/annotations.proto";
+import "google/cloud/automl/v1beta1/geometry.proto";
+import "google/cloud/automl/v1beta1/io.proto";
+import "google/cloud/automl/v1beta1/text_segment.proto";
+import "google/protobuf/any.proto";
+import "google/protobuf/duration.proto";
+import "google/protobuf/struct.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl";
+option java_multiple_files = true;
+option java_package = "com.google.cloud.automl.v1beta1";
+option php_namespace = "Google\\Cloud\\AutoMl\\V1beta1";
+option ruby_package = "Google::Cloud::AutoML::V1beta1";
+
+// A representation of an image.
+// Only images up to 30MB in size are supported.
+message Image {
+ // Input only. The data representing the image.
+ // For Predict calls [image_bytes][] must be set, as other options are not
+ // currently supported by prediction API. You can read the contents of an
+ // uploaded image by using the [content_uri][] field.
+ oneof data {
+ // Image content represented as a stream of bytes.
+ // Note: As with all `bytes` fields, protobuffers use a pure binary
+ // representation, whereas JSON representations use base64.
+ bytes image_bytes = 1;
+
+ // An input config specifying the content of the image.
+ InputConfig input_config = 6;
+ }
+
+ // Output only. HTTP URI to the thumbnail image.
+ string thumbnail_uri = 4;
+}
+
+// A representation of a text snippet.
+message TextSnippet {
+ // Required. The content of the text snippet as a string. Up to 250000
+ // characters long.
+ string content = 1;
+
+ // Optional. The format of
+ // [content][google.cloud.automl.v1beta1.TextSnippet.content]. Currently the
+ // only two allowed values are "text/html" and "text/plain". If left blank,
+ // the format is automatically determined from the type of the uploaded
+ // [content][google.cloud.automl.v1beta1.TextSnippet.content].
+ string mime_type = 2;
+
+ // Output only. HTTP URI where you can download the content.
+ string content_uri = 4;
+}
+
+// Message that describes dimension of a document.
+message DocumentDimensions {
+ // Unit of the document dimension.
+ enum DocumentDimensionUnit {
+ // Should not be used.
+ DOCUMENT_DIMENSION_UNIT_UNSPECIFIED = 0;
+
+ // Document dimension is measured in inches.
+ INCH = 1;
+
+ // Document dimension is measured in centimeters.
+ CENTIMETER = 2;
+
+ // Document dimension is measured in points. 72 points = 1 inch.
+ POINT = 3;
+ }
+
+ // Unit of the dimension.
+ DocumentDimensionUnit unit = 1;
+
+ // Width value of the document, works together with the unit.
+ float width = 2;
+
+ // Height value of the document, works together with the unit.
+ float height = 3;
+}
+
+// A structured text document e.g. a PDF.
+message Document {
+ // Describes the layout information of a
+ // [text_segment][google.cloud.automl.v1beta1.Document.Layout.text_segment] in
+ // the document.
+ message Layout {
+ // The type of TextSegment in the context of the original document.
+ enum TextSegmentType {
+ // Should not be used.
+ TEXT_SEGMENT_TYPE_UNSPECIFIED = 0;
+
+ // The text segment is a token. e.g. word.
+ TOKEN = 1;
+
+ // The text segment is a paragraph.
+ PARAGRAPH = 2;
+
+ // The text segment is a form field.
+ FORM_FIELD = 3;
+
+ // The text segment is the name part of a form field. It will be treated
+ // as child of another FORM_FIELD TextSegment if its span is subspan of
+ // another TextSegment with type FORM_FIELD.
+ FORM_FIELD_NAME = 4;
+
+ // The text segment is the text content part of a form field. It will be
+ // treated as child of another FORM_FIELD TextSegment if its span is
+ // subspan of another TextSegment with type FORM_FIELD.
+ FORM_FIELD_CONTENTS = 5;
+
+ // The text segment is a whole table, including headers, and all rows.
+ TABLE = 6;
+
+ // The text segment is a table's headers. It will be treated as child of
+ // another TABLE TextSegment if its span is subspan of another TextSegment
+ // with type TABLE.
+ TABLE_HEADER = 7;
+
+ // The text segment is a row in table. It will be treated as child of
+ // another TABLE TextSegment if its span is subspan of another TextSegment
+ // with type TABLE.
+ TABLE_ROW = 8;
+
+ // The text segment is a cell in table. It will be treated as child of
+ // another TABLE_ROW TextSegment if its span is subspan of another
+ // TextSegment with type TABLE_ROW.
+ TABLE_CELL = 9;
+ }
+
+ // Text Segment that represents a segment in
+ // [document_text][google.cloud.automl.v1beta1.Document.document_text].
+ TextSegment text_segment = 1;
+
+ // Page number of the
+ // [text_segment][google.cloud.automl.v1beta1.Document.Layout.text_segment]
+ // in the original document, starts from 1.
+ int32 page_number = 2;
+
+ // The position of the
+ // [text_segment][google.cloud.automl.v1beta1.Document.Layout.text_segment]
+ // in the page. Contains exactly 4
+ //
+ // [normalized_vertices][google.cloud.automl.v1beta1.BoundingPoly.normalized_vertices]
+ // and they are connected by edges in the order provided, which will
+ // represent a rectangle parallel to the frame. The
+ // [NormalizedVertex-s][google.cloud.automl.v1beta1.NormalizedVertex] are
+ // relative to the page.
+ // Coordinates are based on top-left as point (0,0).
+ BoundingPoly bounding_poly = 3;
+
+ // The type of the
+ // [text_segment][google.cloud.automl.v1beta1.Document.Layout.text_segment]
+ // in document.
+ TextSegmentType text_segment_type = 4;
+ }
+
+ // An input config specifying the content of the document.
+ DocumentInputConfig input_config = 1;
+
+ // The plain text version of this document.
+ TextSnippet document_text = 2;
+
+ // Describes the layout of the document.
+ // Sorted by [page_number][].
+ repeated Layout layout = 3;
+
+ // The dimensions of the page in the document.
+ DocumentDimensions document_dimensions = 4;
+
+ // Number of pages in the document.
+ int32 page_count = 5;
+}
+
+// A representation of a row in a relational table.
+message Row {
+ // The resource IDs of the column specs describing the columns of the row.
+ // If set must contain, but possibly in a different order, all input
+ // feature
+ //
+ // [column_spec_ids][google.cloud.automl.v1beta1.TablesModelMetadata.input_feature_column_specs]
+ // of the Model this row is being passed to.
+ // Note: The below `values` field must match order of this field, if this
+ // field is set.
+ repeated string column_spec_ids = 2;
+
+ // Required. The values of the row cells, given in the same order as the
+ // column_spec_ids, or, if not set, then in the same order as input
+ // feature
+ //
+ // [column_specs][google.cloud.automl.v1beta1.TablesModelMetadata.input_feature_column_specs]
+ // of the Model this row is being passed to.
+ repeated google.protobuf.Value values = 3;
+}
+
+// Example data used for training or prediction.
+message ExamplePayload {
+ // Required. Input only. The example data.
+ oneof payload {
+ // Example image.
+ Image image = 1;
+
+ // Example text.
+ TextSnippet text_snippet = 2;
+
+ // Example document.
+ Document document = 4;
+
+ // Example relational table row.
+ Row row = 3;
+ }
+}
diff --git a/automl/google/cloud/automl_v1beta1/proto/data_items_pb2.py b/automl/google/cloud/automl_v1beta1/proto/data_items_pb2.py
new file mode 100644
index 000000000000..ee388d632364
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/data_items_pb2.py
@@ -0,0 +1,1008 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/automl_v1beta1/proto/data_items.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+from google.cloud.automl_v1beta1.proto import (
+ geometry_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_geometry__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ io_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_io__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ text_segment_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_text__segment__pb2,
+)
+from google.protobuf import any_pb2 as google_dot_protobuf_dot_any__pb2
+from google.protobuf import duration_pb2 as google_dot_protobuf_dot_duration__pb2
+from google.protobuf import struct_pb2 as google_dot_protobuf_dot_struct__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/automl_v1beta1/proto/data_items.proto",
+ package="google.cloud.automl.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\037com.google.cloud.automl.v1beta1P\001ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\312\002\033Google\\Cloud\\AutoMl\\V1beta1\352\002\036Google::Cloud::AutoML::V1beta1"
+ ),
+ serialized_pb=_b(
+ '\n2google/cloud/automl_v1beta1/proto/data_items.proto\x12\x1bgoogle.cloud.automl.v1beta1\x1a\x1cgoogle/api/annotations.proto\x1a\x30google/cloud/automl_v1beta1/proto/geometry.proto\x1a*google/cloud/automl_v1beta1/proto/io.proto\x1a\x34google/cloud/automl_v1beta1/proto/text_segment.proto\x1a\x19google/protobuf/any.proto\x1a\x1egoogle/protobuf/duration.proto\x1a\x1cgoogle/protobuf/struct.proto"\x7f\n\x05Image\x12\x15\n\x0bimage_bytes\x18\x01 \x01(\x0cH\x00\x12@\n\x0cinput_config\x18\x06 \x01(\x0b\x32(.google.cloud.automl.v1beta1.InputConfigH\x00\x12\x15\n\rthumbnail_uri\x18\x04 \x01(\tB\x06\n\x04\x64\x61ta"F\n\x0bTextSnippet\x12\x0f\n\x07\x63ontent\x18\x01 \x01(\t\x12\x11\n\tmime_type\x18\x02 \x01(\t\x12\x13\n\x0b\x63ontent_uri\x18\x04 \x01(\t"\xef\x01\n\x12\x44ocumentDimensions\x12S\n\x04unit\x18\x01 \x01(\x0e\x32\x45.google.cloud.automl.v1beta1.DocumentDimensions.DocumentDimensionUnit\x12\r\n\x05width\x18\x02 \x01(\x02\x12\x0e\n\x06height\x18\x03 \x01(\x02"e\n\x15\x44ocumentDimensionUnit\x12\'\n#DOCUMENT_DIMENSION_UNIT_UNSPECIFIED\x10\x00\x12\x08\n\x04INCH\x10\x01\x12\x0e\n\nCENTIMETER\x10\x02\x12\t\n\x05POINT\x10\x03"\xf9\x05\n\x08\x44ocument\x12\x46\n\x0cinput_config\x18\x01 \x01(\x0b\x32\x30.google.cloud.automl.v1beta1.DocumentInputConfig\x12?\n\rdocument_text\x18\x02 \x01(\x0b\x32(.google.cloud.automl.v1beta1.TextSnippet\x12<\n\x06layout\x18\x03 \x03(\x0b\x32,.google.cloud.automl.v1beta1.Document.Layout\x12L\n\x13\x64ocument_dimensions\x18\x04 \x01(\x0b\x32/.google.cloud.automl.v1beta1.DocumentDimensions\x12\x12\n\npage_count\x18\x05 \x01(\x05\x1a\xc3\x03\n\x06Layout\x12>\n\x0ctext_segment\x18\x01 \x01(\x0b\x32(.google.cloud.automl.v1beta1.TextSegment\x12\x13\n\x0bpage_number\x18\x02 \x01(\x05\x12@\n\rbounding_poly\x18\x03 \x01(\x0b\x32).google.cloud.automl.v1beta1.BoundingPoly\x12W\n\x11text_segment_type\x18\x04 \x01(\x0e\x32<.google.cloud.automl.v1beta1.Document.Layout.TextSegmentType"\xc8\x01\n\x0fTextSegmentType\x12!\n\x1dTEXT_SEGMENT_TYPE_UNSPECIFIED\x10\x00\x12\t\n\x05TOKEN\x10\x01\x12\r\n\tPARAGRAPH\x10\x02\x12\x0e\n\nFORM_FIELD\x10\x03\x12\x13\n\x0f\x46ORM_FIELD_NAME\x10\x04\x12\x17\n\x13\x46ORM_FIELD_CONTENTS\x10\x05\x12\t\n\x05TABLE\x10\x06\x12\x10\n\x0cTABLE_HEADER\x10\x07\x12\r\n\tTABLE_ROW\x10\x08\x12\x0e\n\nTABLE_CELL\x10\t"F\n\x03Row\x12\x17\n\x0f\x63olumn_spec_ids\x18\x02 \x03(\t\x12&\n\x06values\x18\x03 \x03(\x0b\x32\x16.google.protobuf.Value"\xfe\x01\n\x0e\x45xamplePayload\x12\x33\n\x05image\x18\x01 \x01(\x0b\x32".google.cloud.automl.v1beta1.ImageH\x00\x12@\n\x0ctext_snippet\x18\x02 \x01(\x0b\x32(.google.cloud.automl.v1beta1.TextSnippetH\x00\x12\x39\n\x08\x64ocument\x18\x04 \x01(\x0b\x32%.google.cloud.automl.v1beta1.DocumentH\x00\x12/\n\x03row\x18\x03 \x01(\x0b\x32 .google.cloud.automl.v1beta1.RowH\x00\x42\t\n\x07payloadB\xa5\x01\n\x1f\x63om.google.cloud.automl.v1beta1P\x01ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\xca\x02\x1bGoogle\\Cloud\\AutoMl\\V1beta1\xea\x02\x1eGoogle::Cloud::AutoML::V1beta1b\x06proto3'
+ ),
+ dependencies=[
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_geometry__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_io__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_text__segment__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_any__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_duration__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_struct__pb2.DESCRIPTOR,
+ ],
+)
+
+
+_DOCUMENTDIMENSIONS_DOCUMENTDIMENSIONUNIT = _descriptor.EnumDescriptor(
+ name="DocumentDimensionUnit",
+ full_name="google.cloud.automl.v1beta1.DocumentDimensions.DocumentDimensionUnit",
+ filename=None,
+ file=DESCRIPTOR,
+ values=[
+ _descriptor.EnumValueDescriptor(
+ name="DOCUMENT_DIMENSION_UNIT_UNSPECIFIED",
+ index=0,
+ number=0,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="INCH", index=1, number=1, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="CENTIMETER", index=2, number=2, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="POINT", index=3, number=3, serialized_options=None, type=None
+ ),
+ ],
+ containing_type=None,
+ serialized_options=None,
+ serialized_start=690,
+ serialized_end=791,
+)
+_sym_db.RegisterEnumDescriptor(_DOCUMENTDIMENSIONS_DOCUMENTDIMENSIONUNIT)
+
+_DOCUMENT_LAYOUT_TEXTSEGMENTTYPE = _descriptor.EnumDescriptor(
+ name="TextSegmentType",
+ full_name="google.cloud.automl.v1beta1.Document.Layout.TextSegmentType",
+ filename=None,
+ file=DESCRIPTOR,
+ values=[
+ _descriptor.EnumValueDescriptor(
+ name="TEXT_SEGMENT_TYPE_UNSPECIFIED",
+ index=0,
+ number=0,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="TOKEN", index=1, number=1, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="PARAGRAPH", index=2, number=2, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="FORM_FIELD", index=3, number=3, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="FORM_FIELD_NAME",
+ index=4,
+ number=4,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="FORM_FIELD_CONTENTS",
+ index=5,
+ number=5,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="TABLE", index=6, number=6, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="TABLE_HEADER", index=7, number=7, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="TABLE_ROW", index=8, number=8, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="TABLE_CELL", index=9, number=9, serialized_options=None, type=None
+ ),
+ ],
+ containing_type=None,
+ serialized_options=None,
+ serialized_start=1355,
+ serialized_end=1555,
+)
+_sym_db.RegisterEnumDescriptor(_DOCUMENT_LAYOUT_TEXTSEGMENTTYPE)
+
+
+_IMAGE = _descriptor.Descriptor(
+ name="Image",
+ full_name="google.cloud.automl.v1beta1.Image",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="image_bytes",
+ full_name="google.cloud.automl.v1beta1.Image.image_bytes",
+ index=0,
+ number=1,
+ type=12,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b(""),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="input_config",
+ full_name="google.cloud.automl.v1beta1.Image.input_config",
+ index=1,
+ number=6,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="thumbnail_uri",
+ full_name="google.cloud.automl.v1beta1.Image.thumbnail_uri",
+ index=2,
+ number=4,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="data",
+ full_name="google.cloud.automl.v1beta1.Image.data",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=350,
+ serialized_end=477,
+)
+
+
+_TEXTSNIPPET = _descriptor.Descriptor(
+ name="TextSnippet",
+ full_name="google.cloud.automl.v1beta1.TextSnippet",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="content",
+ full_name="google.cloud.automl.v1beta1.TextSnippet.content",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="mime_type",
+ full_name="google.cloud.automl.v1beta1.TextSnippet.mime_type",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="content_uri",
+ full_name="google.cloud.automl.v1beta1.TextSnippet.content_uri",
+ index=2,
+ number=4,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=479,
+ serialized_end=549,
+)
+
+
+_DOCUMENTDIMENSIONS = _descriptor.Descriptor(
+ name="DocumentDimensions",
+ full_name="google.cloud.automl.v1beta1.DocumentDimensions",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="unit",
+ full_name="google.cloud.automl.v1beta1.DocumentDimensions.unit",
+ index=0,
+ number=1,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="width",
+ full_name="google.cloud.automl.v1beta1.DocumentDimensions.width",
+ index=1,
+ number=2,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="height",
+ full_name="google.cloud.automl.v1beta1.DocumentDimensions.height",
+ index=2,
+ number=3,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[_DOCUMENTDIMENSIONS_DOCUMENTDIMENSIONUNIT],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=552,
+ serialized_end=791,
+)
+
+
+_DOCUMENT_LAYOUT = _descriptor.Descriptor(
+ name="Layout",
+ full_name="google.cloud.automl.v1beta1.Document.Layout",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="text_segment",
+ full_name="google.cloud.automl.v1beta1.Document.Layout.text_segment",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="page_number",
+ full_name="google.cloud.automl.v1beta1.Document.Layout.page_number",
+ index=1,
+ number=2,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="bounding_poly",
+ full_name="google.cloud.automl.v1beta1.Document.Layout.bounding_poly",
+ index=2,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="text_segment_type",
+ full_name="google.cloud.automl.v1beta1.Document.Layout.text_segment_type",
+ index=3,
+ number=4,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[_DOCUMENT_LAYOUT_TEXTSEGMENTTYPE],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1104,
+ serialized_end=1555,
+)
+
+_DOCUMENT = _descriptor.Descriptor(
+ name="Document",
+ full_name="google.cloud.automl.v1beta1.Document",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="input_config",
+ full_name="google.cloud.automl.v1beta1.Document.input_config",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="document_text",
+ full_name="google.cloud.automl.v1beta1.Document.document_text",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="layout",
+ full_name="google.cloud.automl.v1beta1.Document.layout",
+ index=2,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="document_dimensions",
+ full_name="google.cloud.automl.v1beta1.Document.document_dimensions",
+ index=3,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="page_count",
+ full_name="google.cloud.automl.v1beta1.Document.page_count",
+ index=4,
+ number=5,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[_DOCUMENT_LAYOUT],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=794,
+ serialized_end=1555,
+)
+
+
+_ROW = _descriptor.Descriptor(
+ name="Row",
+ full_name="google.cloud.automl.v1beta1.Row",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="column_spec_ids",
+ full_name="google.cloud.automl.v1beta1.Row.column_spec_ids",
+ index=0,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="values",
+ full_name="google.cloud.automl.v1beta1.Row.values",
+ index=1,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1557,
+ serialized_end=1627,
+)
+
+
+_EXAMPLEPAYLOAD = _descriptor.Descriptor(
+ name="ExamplePayload",
+ full_name="google.cloud.automl.v1beta1.ExamplePayload",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="image",
+ full_name="google.cloud.automl.v1beta1.ExamplePayload.image",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="text_snippet",
+ full_name="google.cloud.automl.v1beta1.ExamplePayload.text_snippet",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="document",
+ full_name="google.cloud.automl.v1beta1.ExamplePayload.document",
+ index=2,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="row",
+ full_name="google.cloud.automl.v1beta1.ExamplePayload.row",
+ index=3,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="payload",
+ full_name="google.cloud.automl.v1beta1.ExamplePayload.payload",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=1630,
+ serialized_end=1884,
+)
+
+_IMAGE.fields_by_name[
+ "input_config"
+].message_type = google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_io__pb2._INPUTCONFIG
+_IMAGE.oneofs_by_name["data"].fields.append(_IMAGE.fields_by_name["image_bytes"])
+_IMAGE.fields_by_name["image_bytes"].containing_oneof = _IMAGE.oneofs_by_name["data"]
+_IMAGE.oneofs_by_name["data"].fields.append(_IMAGE.fields_by_name["input_config"])
+_IMAGE.fields_by_name["input_config"].containing_oneof = _IMAGE.oneofs_by_name["data"]
+_DOCUMENTDIMENSIONS.fields_by_name[
+ "unit"
+].enum_type = _DOCUMENTDIMENSIONS_DOCUMENTDIMENSIONUNIT
+_DOCUMENTDIMENSIONS_DOCUMENTDIMENSIONUNIT.containing_type = _DOCUMENTDIMENSIONS
+_DOCUMENT_LAYOUT.fields_by_name[
+ "text_segment"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_text__segment__pb2._TEXTSEGMENT
+)
+_DOCUMENT_LAYOUT.fields_by_name[
+ "bounding_poly"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_geometry__pb2._BOUNDINGPOLY
+)
+_DOCUMENT_LAYOUT.fields_by_name[
+ "text_segment_type"
+].enum_type = _DOCUMENT_LAYOUT_TEXTSEGMENTTYPE
+_DOCUMENT_LAYOUT.containing_type = _DOCUMENT
+_DOCUMENT_LAYOUT_TEXTSEGMENTTYPE.containing_type = _DOCUMENT_LAYOUT
+_DOCUMENT.fields_by_name[
+ "input_config"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_io__pb2._DOCUMENTINPUTCONFIG
+)
+_DOCUMENT.fields_by_name["document_text"].message_type = _TEXTSNIPPET
+_DOCUMENT.fields_by_name["layout"].message_type = _DOCUMENT_LAYOUT
+_DOCUMENT.fields_by_name["document_dimensions"].message_type = _DOCUMENTDIMENSIONS
+_ROW.fields_by_name["values"].message_type = google_dot_protobuf_dot_struct__pb2._VALUE
+_EXAMPLEPAYLOAD.fields_by_name["image"].message_type = _IMAGE
+_EXAMPLEPAYLOAD.fields_by_name["text_snippet"].message_type = _TEXTSNIPPET
+_EXAMPLEPAYLOAD.fields_by_name["document"].message_type = _DOCUMENT
+_EXAMPLEPAYLOAD.fields_by_name["row"].message_type = _ROW
+_EXAMPLEPAYLOAD.oneofs_by_name["payload"].fields.append(
+ _EXAMPLEPAYLOAD.fields_by_name["image"]
+)
+_EXAMPLEPAYLOAD.fields_by_name[
+ "image"
+].containing_oneof = _EXAMPLEPAYLOAD.oneofs_by_name["payload"]
+_EXAMPLEPAYLOAD.oneofs_by_name["payload"].fields.append(
+ _EXAMPLEPAYLOAD.fields_by_name["text_snippet"]
+)
+_EXAMPLEPAYLOAD.fields_by_name[
+ "text_snippet"
+].containing_oneof = _EXAMPLEPAYLOAD.oneofs_by_name["payload"]
+_EXAMPLEPAYLOAD.oneofs_by_name["payload"].fields.append(
+ _EXAMPLEPAYLOAD.fields_by_name["document"]
+)
+_EXAMPLEPAYLOAD.fields_by_name[
+ "document"
+].containing_oneof = _EXAMPLEPAYLOAD.oneofs_by_name["payload"]
+_EXAMPLEPAYLOAD.oneofs_by_name["payload"].fields.append(
+ _EXAMPLEPAYLOAD.fields_by_name["row"]
+)
+_EXAMPLEPAYLOAD.fields_by_name["row"].containing_oneof = _EXAMPLEPAYLOAD.oneofs_by_name[
+ "payload"
+]
+DESCRIPTOR.message_types_by_name["Image"] = _IMAGE
+DESCRIPTOR.message_types_by_name["TextSnippet"] = _TEXTSNIPPET
+DESCRIPTOR.message_types_by_name["DocumentDimensions"] = _DOCUMENTDIMENSIONS
+DESCRIPTOR.message_types_by_name["Document"] = _DOCUMENT
+DESCRIPTOR.message_types_by_name["Row"] = _ROW
+DESCRIPTOR.message_types_by_name["ExamplePayload"] = _EXAMPLEPAYLOAD
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+Image = _reflection.GeneratedProtocolMessageType(
+ "Image",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_IMAGE,
+ __module__="google.cloud.automl_v1beta1.proto.data_items_pb2",
+ __doc__="""A representation of an image. Only images up to 30MB in size are
+ supported.
+
+
+ Attributes:
+ data:
+ Input only. The data representing the image. For Predict calls
+ [image\_bytes][] must be set, as other options are not
+ currently supported by prediction API. You can read the
+ contents of an uploaded image by using the [content\_uri][]
+ field.
+ image_bytes:
+ Image content represented as a stream of bytes. Note: As with
+ all ``bytes`` fields, protobuffers use a pure binary
+ representation, whereas JSON representations use base64.
+ input_config:
+ An input config specifying the content of the image.
+ thumbnail_uri:
+ Output only. HTTP URI to the thumbnail image.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.Image)
+ ),
+)
+_sym_db.RegisterMessage(Image)
+
+TextSnippet = _reflection.GeneratedProtocolMessageType(
+ "TextSnippet",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TEXTSNIPPET,
+ __module__="google.cloud.automl_v1beta1.proto.data_items_pb2",
+ __doc__="""A representation of a text snippet.
+
+
+ Attributes:
+ content:
+ Required. The content of the text snippet as a string. Up to
+ 250000 characters long.
+ mime_type:
+ Optional. The format of
+ [content][google.cloud.automl.v1beta1.TextSnippet.content].
+ Currently the only two allowed values are "text/html" and
+ "text/plain". If left blank, the format is automatically
+ determined from the type of the uploaded
+ [content][google.cloud.automl.v1beta1.TextSnippet.content].
+ content_uri:
+ Output only. HTTP URI where you can download the content.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.TextSnippet)
+ ),
+)
+_sym_db.RegisterMessage(TextSnippet)
+
+DocumentDimensions = _reflection.GeneratedProtocolMessageType(
+ "DocumentDimensions",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_DOCUMENTDIMENSIONS,
+ __module__="google.cloud.automl_v1beta1.proto.data_items_pb2",
+ __doc__="""Message that describes dimension of a document.
+
+
+ Attributes:
+ unit:
+ Unit of the dimension.
+ width:
+ Width value of the document, works together with the unit.
+ height:
+ Height value of the document, works together with the unit.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.DocumentDimensions)
+ ),
+)
+_sym_db.RegisterMessage(DocumentDimensions)
+
+Document = _reflection.GeneratedProtocolMessageType(
+ "Document",
+ (_message.Message,),
+ dict(
+ Layout=_reflection.GeneratedProtocolMessageType(
+ "Layout",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_DOCUMENT_LAYOUT,
+ __module__="google.cloud.automl_v1beta1.proto.data_items_pb2",
+ __doc__="""Describes the layout information of a
+ [text\_segment][google.cloud.automl.v1beta1.Document.Layout.text\_segment]
+ in the document.
+
+
+ Attributes:
+ text_segment:
+ Text Segment that represents a segment in [document\_text][goo
+ gle.cloud.automl.v1beta1.Document.document\_text].
+ page_number:
+ Page number of the [text\_segment][google.cloud.automl.v1beta1
+ .Document.Layout.text\_segment] in the original document,
+ starts from 1.
+ bounding_poly:
+ The position of the [text\_segment][google.cloud.automl.v1beta
+ 1.Document.Layout.text\_segment] in the page. Contains exactly
+ 4 [normalized\_vertices][google.cloud.automl.v1beta1.Bounding
+ Poly.normalized\_vertices] and they are connected by edges in
+ the order provided, which will represent a rectangle parallel
+ to the frame. The [NormalizedVertex-s][google.cloud.automl.v1b
+ eta1.NormalizedVertex] are relative to the page. Coordinates
+ are based on top-left as point (0,0).
+ text_segment_type:
+ The type of the [text\_segment][google.cloud.automl.v1beta1.Do
+ cument.Layout.text\_segment] in document.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.Document.Layout)
+ ),
+ ),
+ DESCRIPTOR=_DOCUMENT,
+ __module__="google.cloud.automl_v1beta1.proto.data_items_pb2",
+ __doc__="""A structured text document e.g. a PDF.
+
+
+ Attributes:
+ input_config:
+ An input config specifying the content of the document.
+ document_text:
+ The plain text version of this document.
+ layout:
+ Describes the layout of the document. Sorted by
+ [page\_number][].
+ document_dimensions:
+ The dimensions of the page in the document.
+ page_count:
+ Number of pages in the document.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.Document)
+ ),
+)
+_sym_db.RegisterMessage(Document)
+_sym_db.RegisterMessage(Document.Layout)
+
+Row = _reflection.GeneratedProtocolMessageType(
+ "Row",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_ROW,
+ __module__="google.cloud.automl_v1beta1.proto.data_items_pb2",
+ __doc__="""A representation of a row in a relational table.
+
+
+ Attributes:
+ column_spec_ids:
+ The resource IDs of the column specs describing the columns of
+ the row. If set must contain, but possibly in a different
+ order, all input feature [column\_spec\_ids][google.cloud.aut
+ oml.v1beta1.TablesModelMetadata.input\_feature\_column\_specs]
+ of the Model this row is being passed to. Note: The below
+ ``values`` field must match order of this field, if this field
+ is set.
+ values:
+ Required. The values of the row cells, given in the same order
+ as the column\_spec\_ids, or, if not set, then in the same
+ order as input feature [column\_specs][google.cloud.automl.v1
+ beta1.TablesModelMetadata.input\_feature\_column\_specs] of
+ the Model this row is being passed to.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.Row)
+ ),
+)
+_sym_db.RegisterMessage(Row)
+
+ExamplePayload = _reflection.GeneratedProtocolMessageType(
+ "ExamplePayload",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_EXAMPLEPAYLOAD,
+ __module__="google.cloud.automl_v1beta1.proto.data_items_pb2",
+ __doc__="""Example data used for training or prediction.
+
+
+ Attributes:
+ payload:
+ Required. Input only. The example data.
+ image:
+ Example image.
+ text_snippet:
+ Example text.
+ document:
+ Example document.
+ row:
+ Example relational table row.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ExamplePayload)
+ ),
+)
+_sym_db.RegisterMessage(ExamplePayload)
+
+
+DESCRIPTOR._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/automl/google/cloud/automl_v1beta1/proto/data_items_pb2_grpc.py b/automl/google/cloud/automl_v1beta1/proto/data_items_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/data_items_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/automl/google/cloud/automl_v1beta1/proto/data_stats.proto b/automl/google/cloud/automl_v1beta1/proto/data_stats.proto
new file mode 100644
index 000000000000..5d941a5baa81
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/data_stats.proto
@@ -0,0 +1,167 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.automl.v1beta1;
+
+import "google/api/annotations.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl";
+option java_multiple_files = true;
+option java_package = "com.google.cloud.automl.v1beta1";
+option php_namespace = "Google\\Cloud\\AutoMl\\V1beta1";
+option ruby_package = "Google::Cloud::AutoML::V1beta1";
+
+// The data statistics of a series of values that share the same DataType.
+message DataStats {
+ // The data statistics specific to a DataType.
+ oneof stats {
+ // The statistics for FLOAT64 DataType.
+ Float64Stats float64_stats = 3;
+
+ // The statistics for STRING DataType.
+ StringStats string_stats = 4;
+
+ // The statistics for TIMESTAMP DataType.
+ TimestampStats timestamp_stats = 5;
+
+ // The statistics for ARRAY DataType.
+ ArrayStats array_stats = 6;
+
+ // The statistics for STRUCT DataType.
+ StructStats struct_stats = 7;
+
+ // The statistics for CATEGORY DataType.
+ CategoryStats category_stats = 8;
+ }
+
+ // The number of distinct values.
+ int64 distinct_value_count = 1;
+
+ // The number of values that are null.
+ int64 null_value_count = 2;
+
+ // The number of values that are valid.
+ int64 valid_value_count = 9;
+}
+
+// The data statistics of a series of FLOAT64 values.
+message Float64Stats {
+ // A bucket of a histogram.
+ message HistogramBucket {
+ // The minimum value of the bucket, inclusive.
+ double min = 1;
+
+ // The maximum value of the bucket, exclusive unless max = `"Infinity"`, in
+ // which case it's inclusive.
+ double max = 2;
+
+ // The number of data values that are in the bucket, i.e. are between
+ // min and max values.
+ int64 count = 3;
+ }
+
+ // The mean of the series.
+ double mean = 1;
+
+ // The standard deviation of the series.
+ double standard_deviation = 2;
+
+ // Ordered from 0 to k k-quantile values of the data series of n values.
+ // The value at index i is, approximately, the i*n/k-th smallest value in the
+ // series; for i = 0 and i = k these are, respectively, the min and max
+ // values.
+ repeated double quantiles = 3;
+
+ // Histogram buckets of the data series. Sorted by the min value of the
+ // bucket, ascendingly, and the number of the buckets is dynamically
+ // generated. The buckets are non-overlapping and completely cover whole
+ // FLOAT64 range with min of first bucket being `"-Infinity"`, and max of
+ // the last one being `"Infinity"`.
+ repeated HistogramBucket histogram_buckets = 4;
+}
+
+// The data statistics of a series of STRING values.
+message StringStats {
+ // The statistics of a unigram.
+ message UnigramStats {
+ // The unigram.
+ string value = 1;
+
+ // The number of occurrences of this unigram in the series.
+ int64 count = 2;
+ }
+
+ // The statistics of the top 20 unigrams, ordered by
+ // [count][google.cloud.automl.v1beta1.StringStats.UnigramStats.count].
+ repeated UnigramStats top_unigram_stats = 1;
+}
+
+// The data statistics of a series of TIMESTAMP values.
+message TimestampStats {
+ // Stats split by a defined in context granularity.
+ message GranularStats {
+ // A map from granularity key to example count for that key.
+ // E.g. for hour_of_day `13` means 1pm, or for month_of_year `5` means May).
+ map buckets = 1;
+ }
+
+ // The string key is the pre-defined granularity. Currently supported:
+ // hour_of_day, day_of_week, month_of_year.
+ // Granularities finer that the granularity of timestamp data are not
+ // populated (e.g. if timestamps are at day granularity, then hour_of_day
+ // is not populated).
+ map granular_stats = 1;
+}
+
+// The data statistics of a series of ARRAY values.
+message ArrayStats {
+ // Stats of all the values of all arrays, as if they were a single long
+ // series of data. The type depends on the element type of the array.
+ DataStats member_stats = 2;
+}
+
+// The data statistics of a series of STRUCT values.
+message StructStats {
+ // Map from a field name of the struct to data stats aggregated over series
+ // of all data in that field across all the structs.
+ map field_stats = 1;
+}
+
+// The data statistics of a series of CATEGORY values.
+message CategoryStats {
+ // The statistics of a single CATEGORY value.
+ message SingleCategoryStats {
+ // The CATEGORY value.
+ string value = 1;
+
+ // The number of occurrences of this value in the series.
+ int64 count = 2;
+ }
+
+ // The statistics of the top 20 CATEGORY values, ordered by
+ //
+ // [count][google.cloud.automl.v1beta1.CategoryStats.SingleCategoryStats.count].
+ repeated SingleCategoryStats top_category_stats = 1;
+}
+
+// A correlation statistics between two series of DataType values. The series
+// may have differing DataType-s, but within a single series the DataType must
+// be the same.
+message CorrelationStats {
+ // The correlation value using the Cramer's V measure.
+ double cramers_v = 1;
+}
diff --git a/automl/google/cloud/automl_v1beta1/proto/data_stats_pb2.py b/automl/google/cloud/automl_v1beta1/proto/data_stats_pb2.py
new file mode 100644
index 000000000000..f0415f19ae5d
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/data_stats_pb2.py
@@ -0,0 +1,1329 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/automl_v1beta1/proto/data_stats.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/automl_v1beta1/proto/data_stats.proto",
+ package="google.cloud.automl.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\037com.google.cloud.automl.v1beta1P\001ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\312\002\033Google\\Cloud\\AutoMl\\V1beta1\352\002\036Google::Cloud::AutoML::V1beta1"
+ ),
+ serialized_pb=_b(
+ '\n2google/cloud/automl_v1beta1/proto/data_stats.proto\x12\x1bgoogle.cloud.automl.v1beta1\x1a\x1cgoogle/api/annotations.proto"\xfd\x03\n\tDataStats\x12\x42\n\rfloat64_stats\x18\x03 \x01(\x0b\x32).google.cloud.automl.v1beta1.Float64StatsH\x00\x12@\n\x0cstring_stats\x18\x04 \x01(\x0b\x32(.google.cloud.automl.v1beta1.StringStatsH\x00\x12\x46\n\x0ftimestamp_stats\x18\x05 \x01(\x0b\x32+.google.cloud.automl.v1beta1.TimestampStatsH\x00\x12>\n\x0b\x61rray_stats\x18\x06 \x01(\x0b\x32\'.google.cloud.automl.v1beta1.ArrayStatsH\x00\x12@\n\x0cstruct_stats\x18\x07 \x01(\x0b\x32(.google.cloud.automl.v1beta1.StructStatsH\x00\x12\x44\n\x0e\x63\x61tegory_stats\x18\x08 \x01(\x0b\x32*.google.cloud.automl.v1beta1.CategoryStatsH\x00\x12\x1c\n\x14\x64istinct_value_count\x18\x01 \x01(\x03\x12\x18\n\x10null_value_count\x18\x02 \x01(\x03\x12\x19\n\x11valid_value_count\x18\t \x01(\x03\x42\x07\n\x05stats"\xdd\x01\n\x0c\x46loat64Stats\x12\x0c\n\x04mean\x18\x01 \x01(\x01\x12\x1a\n\x12standard_deviation\x18\x02 \x01(\x01\x12\x11\n\tquantiles\x18\x03 \x03(\x01\x12T\n\x11histogram_buckets\x18\x04 \x03(\x0b\x32\x39.google.cloud.automl.v1beta1.Float64Stats.HistogramBucket\x1a:\n\x0fHistogramBucket\x12\x0b\n\x03min\x18\x01 \x01(\x01\x12\x0b\n\x03max\x18\x02 \x01(\x01\x12\r\n\x05\x63ount\x18\x03 \x01(\x03"\x8d\x01\n\x0bStringStats\x12P\n\x11top_unigram_stats\x18\x01 \x03(\x0b\x32\x35.google.cloud.automl.v1beta1.StringStats.UnigramStats\x1a,\n\x0cUnigramStats\x12\r\n\x05value\x18\x01 \x01(\t\x12\r\n\x05\x63ount\x18\x02 \x01(\x03"\xf4\x02\n\x0eTimestampStats\x12V\n\x0egranular_stats\x18\x01 \x03(\x0b\x32>.google.cloud.automl.v1beta1.TimestampStats.GranularStatsEntry\x1a\x98\x01\n\rGranularStats\x12W\n\x07\x62uckets\x18\x01 \x03(\x0b\x32\x46.google.cloud.automl.v1beta1.TimestampStats.GranularStats.BucketsEntry\x1a.\n\x0c\x42ucketsEntry\x12\x0b\n\x03key\x18\x01 \x01(\x05\x12\r\n\x05value\x18\x02 \x01(\x03:\x02\x38\x01\x1ao\n\x12GranularStatsEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12H\n\x05value\x18\x02 \x01(\x0b\x32\x39.google.cloud.automl.v1beta1.TimestampStats.GranularStats:\x02\x38\x01"J\n\nArrayStats\x12<\n\x0cmember_stats\x18\x02 \x01(\x0b\x32&.google.cloud.automl.v1beta1.DataStats"\xb7\x01\n\x0bStructStats\x12M\n\x0b\x66ield_stats\x18\x01 \x03(\x0b\x32\x38.google.cloud.automl.v1beta1.StructStats.FieldStatsEntry\x1aY\n\x0f\x46ieldStatsEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\x35\n\x05value\x18\x02 \x01(\x0b\x32&.google.cloud.automl.v1beta1.DataStats:\x02\x38\x01"\xa0\x01\n\rCategoryStats\x12Z\n\x12top_category_stats\x18\x01 \x03(\x0b\x32>.google.cloud.automl.v1beta1.CategoryStats.SingleCategoryStats\x1a\x33\n\x13SingleCategoryStats\x12\r\n\x05value\x18\x01 \x01(\t\x12\r\n\x05\x63ount\x18\x02 \x01(\x03"%\n\x10\x43orrelationStats\x12\x11\n\tcramers_v\x18\x01 \x01(\x01\x42\xa5\x01\n\x1f\x63om.google.cloud.automl.v1beta1P\x01ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\xca\x02\x1bGoogle\\Cloud\\AutoMl\\V1beta1\xea\x02\x1eGoogle::Cloud::AutoML::V1beta1b\x06proto3'
+ ),
+ dependencies=[google_dot_api_dot_annotations__pb2.DESCRIPTOR],
+)
+
+
+_DATASTATS = _descriptor.Descriptor(
+ name="DataStats",
+ full_name="google.cloud.automl.v1beta1.DataStats",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="float64_stats",
+ full_name="google.cloud.automl.v1beta1.DataStats.float64_stats",
+ index=0,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="string_stats",
+ full_name="google.cloud.automl.v1beta1.DataStats.string_stats",
+ index=1,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="timestamp_stats",
+ full_name="google.cloud.automl.v1beta1.DataStats.timestamp_stats",
+ index=2,
+ number=5,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="array_stats",
+ full_name="google.cloud.automl.v1beta1.DataStats.array_stats",
+ index=3,
+ number=6,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="struct_stats",
+ full_name="google.cloud.automl.v1beta1.DataStats.struct_stats",
+ index=4,
+ number=7,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="category_stats",
+ full_name="google.cloud.automl.v1beta1.DataStats.category_stats",
+ index=5,
+ number=8,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="distinct_value_count",
+ full_name="google.cloud.automl.v1beta1.DataStats.distinct_value_count",
+ index=6,
+ number=1,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="null_value_count",
+ full_name="google.cloud.automl.v1beta1.DataStats.null_value_count",
+ index=7,
+ number=2,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="valid_value_count",
+ full_name="google.cloud.automl.v1beta1.DataStats.valid_value_count",
+ index=8,
+ number=9,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="stats",
+ full_name="google.cloud.automl.v1beta1.DataStats.stats",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=114,
+ serialized_end=623,
+)
+
+
+_FLOAT64STATS_HISTOGRAMBUCKET = _descriptor.Descriptor(
+ name="HistogramBucket",
+ full_name="google.cloud.automl.v1beta1.Float64Stats.HistogramBucket",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="min",
+ full_name="google.cloud.automl.v1beta1.Float64Stats.HistogramBucket.min",
+ index=0,
+ number=1,
+ type=1,
+ cpp_type=5,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="max",
+ full_name="google.cloud.automl.v1beta1.Float64Stats.HistogramBucket.max",
+ index=1,
+ number=2,
+ type=1,
+ cpp_type=5,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="count",
+ full_name="google.cloud.automl.v1beta1.Float64Stats.HistogramBucket.count",
+ index=2,
+ number=3,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=789,
+ serialized_end=847,
+)
+
+_FLOAT64STATS = _descriptor.Descriptor(
+ name="Float64Stats",
+ full_name="google.cloud.automl.v1beta1.Float64Stats",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="mean",
+ full_name="google.cloud.automl.v1beta1.Float64Stats.mean",
+ index=0,
+ number=1,
+ type=1,
+ cpp_type=5,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="standard_deviation",
+ full_name="google.cloud.automl.v1beta1.Float64Stats.standard_deviation",
+ index=1,
+ number=2,
+ type=1,
+ cpp_type=5,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="quantiles",
+ full_name="google.cloud.automl.v1beta1.Float64Stats.quantiles",
+ index=2,
+ number=3,
+ type=1,
+ cpp_type=5,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="histogram_buckets",
+ full_name="google.cloud.automl.v1beta1.Float64Stats.histogram_buckets",
+ index=3,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[_FLOAT64STATS_HISTOGRAMBUCKET],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=626,
+ serialized_end=847,
+)
+
+
+_STRINGSTATS_UNIGRAMSTATS = _descriptor.Descriptor(
+ name="UnigramStats",
+ full_name="google.cloud.automl.v1beta1.StringStats.UnigramStats",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="value",
+ full_name="google.cloud.automl.v1beta1.StringStats.UnigramStats.value",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="count",
+ full_name="google.cloud.automl.v1beta1.StringStats.UnigramStats.count",
+ index=1,
+ number=2,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=947,
+ serialized_end=991,
+)
+
+_STRINGSTATS = _descriptor.Descriptor(
+ name="StringStats",
+ full_name="google.cloud.automl.v1beta1.StringStats",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="top_unigram_stats",
+ full_name="google.cloud.automl.v1beta1.StringStats.top_unigram_stats",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[_STRINGSTATS_UNIGRAMSTATS],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=850,
+ serialized_end=991,
+)
+
+
+_TIMESTAMPSTATS_GRANULARSTATS_BUCKETSENTRY = _descriptor.Descriptor(
+ name="BucketsEntry",
+ full_name="google.cloud.automl.v1beta1.TimestampStats.GranularStats.BucketsEntry",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="key",
+ full_name="google.cloud.automl.v1beta1.TimestampStats.GranularStats.BucketsEntry.key",
+ index=0,
+ number=1,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="value",
+ full_name="google.cloud.automl.v1beta1.TimestampStats.GranularStats.BucketsEntry.value",
+ index=1,
+ number=2,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=_b("8\001"),
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1207,
+ serialized_end=1253,
+)
+
+_TIMESTAMPSTATS_GRANULARSTATS = _descriptor.Descriptor(
+ name="GranularStats",
+ full_name="google.cloud.automl.v1beta1.TimestampStats.GranularStats",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="buckets",
+ full_name="google.cloud.automl.v1beta1.TimestampStats.GranularStats.buckets",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[_TIMESTAMPSTATS_GRANULARSTATS_BUCKETSENTRY],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1101,
+ serialized_end=1253,
+)
+
+_TIMESTAMPSTATS_GRANULARSTATSENTRY = _descriptor.Descriptor(
+ name="GranularStatsEntry",
+ full_name="google.cloud.automl.v1beta1.TimestampStats.GranularStatsEntry",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="key",
+ full_name="google.cloud.automl.v1beta1.TimestampStats.GranularStatsEntry.key",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="value",
+ full_name="google.cloud.automl.v1beta1.TimestampStats.GranularStatsEntry.value",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=_b("8\001"),
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1255,
+ serialized_end=1366,
+)
+
+_TIMESTAMPSTATS = _descriptor.Descriptor(
+ name="TimestampStats",
+ full_name="google.cloud.automl.v1beta1.TimestampStats",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="granular_stats",
+ full_name="google.cloud.automl.v1beta1.TimestampStats.granular_stats",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[_TIMESTAMPSTATS_GRANULARSTATS, _TIMESTAMPSTATS_GRANULARSTATSENTRY],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=994,
+ serialized_end=1366,
+)
+
+
+_ARRAYSTATS = _descriptor.Descriptor(
+ name="ArrayStats",
+ full_name="google.cloud.automl.v1beta1.ArrayStats",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="member_stats",
+ full_name="google.cloud.automl.v1beta1.ArrayStats.member_stats",
+ index=0,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1368,
+ serialized_end=1442,
+)
+
+
+_STRUCTSTATS_FIELDSTATSENTRY = _descriptor.Descriptor(
+ name="FieldStatsEntry",
+ full_name="google.cloud.automl.v1beta1.StructStats.FieldStatsEntry",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="key",
+ full_name="google.cloud.automl.v1beta1.StructStats.FieldStatsEntry.key",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="value",
+ full_name="google.cloud.automl.v1beta1.StructStats.FieldStatsEntry.value",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=_b("8\001"),
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1539,
+ serialized_end=1628,
+)
+
+_STRUCTSTATS = _descriptor.Descriptor(
+ name="StructStats",
+ full_name="google.cloud.automl.v1beta1.StructStats",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="field_stats",
+ full_name="google.cloud.automl.v1beta1.StructStats.field_stats",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[_STRUCTSTATS_FIELDSTATSENTRY],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1445,
+ serialized_end=1628,
+)
+
+
+_CATEGORYSTATS_SINGLECATEGORYSTATS = _descriptor.Descriptor(
+ name="SingleCategoryStats",
+ full_name="google.cloud.automl.v1beta1.CategoryStats.SingleCategoryStats",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="value",
+ full_name="google.cloud.automl.v1beta1.CategoryStats.SingleCategoryStats.value",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="count",
+ full_name="google.cloud.automl.v1beta1.CategoryStats.SingleCategoryStats.count",
+ index=1,
+ number=2,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1740,
+ serialized_end=1791,
+)
+
+_CATEGORYSTATS = _descriptor.Descriptor(
+ name="CategoryStats",
+ full_name="google.cloud.automl.v1beta1.CategoryStats",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="top_category_stats",
+ full_name="google.cloud.automl.v1beta1.CategoryStats.top_category_stats",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[_CATEGORYSTATS_SINGLECATEGORYSTATS],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1631,
+ serialized_end=1791,
+)
+
+
+_CORRELATIONSTATS = _descriptor.Descriptor(
+ name="CorrelationStats",
+ full_name="google.cloud.automl.v1beta1.CorrelationStats",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="cramers_v",
+ full_name="google.cloud.automl.v1beta1.CorrelationStats.cramers_v",
+ index=0,
+ number=1,
+ type=1,
+ cpp_type=5,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1793,
+ serialized_end=1830,
+)
+
+_DATASTATS.fields_by_name["float64_stats"].message_type = _FLOAT64STATS
+_DATASTATS.fields_by_name["string_stats"].message_type = _STRINGSTATS
+_DATASTATS.fields_by_name["timestamp_stats"].message_type = _TIMESTAMPSTATS
+_DATASTATS.fields_by_name["array_stats"].message_type = _ARRAYSTATS
+_DATASTATS.fields_by_name["struct_stats"].message_type = _STRUCTSTATS
+_DATASTATS.fields_by_name["category_stats"].message_type = _CATEGORYSTATS
+_DATASTATS.oneofs_by_name["stats"].fields.append(
+ _DATASTATS.fields_by_name["float64_stats"]
+)
+_DATASTATS.fields_by_name["float64_stats"].containing_oneof = _DATASTATS.oneofs_by_name[
+ "stats"
+]
+_DATASTATS.oneofs_by_name["stats"].fields.append(
+ _DATASTATS.fields_by_name["string_stats"]
+)
+_DATASTATS.fields_by_name["string_stats"].containing_oneof = _DATASTATS.oneofs_by_name[
+ "stats"
+]
+_DATASTATS.oneofs_by_name["stats"].fields.append(
+ _DATASTATS.fields_by_name["timestamp_stats"]
+)
+_DATASTATS.fields_by_name[
+ "timestamp_stats"
+].containing_oneof = _DATASTATS.oneofs_by_name["stats"]
+_DATASTATS.oneofs_by_name["stats"].fields.append(
+ _DATASTATS.fields_by_name["array_stats"]
+)
+_DATASTATS.fields_by_name["array_stats"].containing_oneof = _DATASTATS.oneofs_by_name[
+ "stats"
+]
+_DATASTATS.oneofs_by_name["stats"].fields.append(
+ _DATASTATS.fields_by_name["struct_stats"]
+)
+_DATASTATS.fields_by_name["struct_stats"].containing_oneof = _DATASTATS.oneofs_by_name[
+ "stats"
+]
+_DATASTATS.oneofs_by_name["stats"].fields.append(
+ _DATASTATS.fields_by_name["category_stats"]
+)
+_DATASTATS.fields_by_name[
+ "category_stats"
+].containing_oneof = _DATASTATS.oneofs_by_name["stats"]
+_FLOAT64STATS_HISTOGRAMBUCKET.containing_type = _FLOAT64STATS
+_FLOAT64STATS.fields_by_name[
+ "histogram_buckets"
+].message_type = _FLOAT64STATS_HISTOGRAMBUCKET
+_STRINGSTATS_UNIGRAMSTATS.containing_type = _STRINGSTATS
+_STRINGSTATS.fields_by_name[
+ "top_unigram_stats"
+].message_type = _STRINGSTATS_UNIGRAMSTATS
+_TIMESTAMPSTATS_GRANULARSTATS_BUCKETSENTRY.containing_type = (
+ _TIMESTAMPSTATS_GRANULARSTATS
+)
+_TIMESTAMPSTATS_GRANULARSTATS.fields_by_name[
+ "buckets"
+].message_type = _TIMESTAMPSTATS_GRANULARSTATS_BUCKETSENTRY
+_TIMESTAMPSTATS_GRANULARSTATS.containing_type = _TIMESTAMPSTATS
+_TIMESTAMPSTATS_GRANULARSTATSENTRY.fields_by_name[
+ "value"
+].message_type = _TIMESTAMPSTATS_GRANULARSTATS
+_TIMESTAMPSTATS_GRANULARSTATSENTRY.containing_type = _TIMESTAMPSTATS
+_TIMESTAMPSTATS.fields_by_name[
+ "granular_stats"
+].message_type = _TIMESTAMPSTATS_GRANULARSTATSENTRY
+_ARRAYSTATS.fields_by_name["member_stats"].message_type = _DATASTATS
+_STRUCTSTATS_FIELDSTATSENTRY.fields_by_name["value"].message_type = _DATASTATS
+_STRUCTSTATS_FIELDSTATSENTRY.containing_type = _STRUCTSTATS
+_STRUCTSTATS.fields_by_name["field_stats"].message_type = _STRUCTSTATS_FIELDSTATSENTRY
+_CATEGORYSTATS_SINGLECATEGORYSTATS.containing_type = _CATEGORYSTATS
+_CATEGORYSTATS.fields_by_name[
+ "top_category_stats"
+].message_type = _CATEGORYSTATS_SINGLECATEGORYSTATS
+DESCRIPTOR.message_types_by_name["DataStats"] = _DATASTATS
+DESCRIPTOR.message_types_by_name["Float64Stats"] = _FLOAT64STATS
+DESCRIPTOR.message_types_by_name["StringStats"] = _STRINGSTATS
+DESCRIPTOR.message_types_by_name["TimestampStats"] = _TIMESTAMPSTATS
+DESCRIPTOR.message_types_by_name["ArrayStats"] = _ARRAYSTATS
+DESCRIPTOR.message_types_by_name["StructStats"] = _STRUCTSTATS
+DESCRIPTOR.message_types_by_name["CategoryStats"] = _CATEGORYSTATS
+DESCRIPTOR.message_types_by_name["CorrelationStats"] = _CORRELATIONSTATS
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+DataStats = _reflection.GeneratedProtocolMessageType(
+ "DataStats",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_DATASTATS,
+ __module__="google.cloud.automl_v1beta1.proto.data_stats_pb2",
+ __doc__="""The data statistics of a series of values that share the same DataType.
+
+
+ Attributes:
+ stats:
+ The data statistics specific to a DataType.
+ float64_stats:
+ The statistics for FLOAT64 DataType.
+ string_stats:
+ The statistics for STRING DataType.
+ timestamp_stats:
+ The statistics for TIMESTAMP DataType.
+ array_stats:
+ The statistics for ARRAY DataType.
+ struct_stats:
+ The statistics for STRUCT DataType.
+ category_stats:
+ The statistics for CATEGORY DataType.
+ distinct_value_count:
+ The number of distinct values.
+ null_value_count:
+ The number of values that are null.
+ valid_value_count:
+ The number of values that are valid.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.DataStats)
+ ),
+)
+_sym_db.RegisterMessage(DataStats)
+
+Float64Stats = _reflection.GeneratedProtocolMessageType(
+ "Float64Stats",
+ (_message.Message,),
+ dict(
+ HistogramBucket=_reflection.GeneratedProtocolMessageType(
+ "HistogramBucket",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_FLOAT64STATS_HISTOGRAMBUCKET,
+ __module__="google.cloud.automl_v1beta1.proto.data_stats_pb2",
+ __doc__="""A bucket of a histogram.
+
+
+ Attributes:
+ min:
+ The minimum value of the bucket, inclusive.
+ max:
+ The maximum value of the bucket, exclusive unless max =
+ ``"Infinity"``, in which case it's inclusive.
+ count:
+ The number of data values that are in the bucket, i.e. are
+ between min and max values.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.Float64Stats.HistogramBucket)
+ ),
+ ),
+ DESCRIPTOR=_FLOAT64STATS,
+ __module__="google.cloud.automl_v1beta1.proto.data_stats_pb2",
+ __doc__="""The data statistics of a series of FLOAT64 values.
+
+
+ Attributes:
+ mean:
+ The mean of the series.
+ standard_deviation:
+ The standard deviation of the series.
+ quantiles:
+ Ordered from 0 to k k-quantile values of the data series of n
+ values. The value at index i is, approximately, the i\*n/k-th
+ smallest value in the series; for i = 0 and i = k these are,
+ respectively, the min and max values.
+ histogram_buckets:
+ Histogram buckets of the data series. Sorted by the min value
+ of the bucket, ascendingly, and the number of the buckets is
+ dynamically generated. The buckets are non-overlapping and
+ completely cover whole FLOAT64 range with min of first bucket
+ being ``"-Infinity"``, and max of the last one being
+ ``"Infinity"``.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.Float64Stats)
+ ),
+)
+_sym_db.RegisterMessage(Float64Stats)
+_sym_db.RegisterMessage(Float64Stats.HistogramBucket)
+
+StringStats = _reflection.GeneratedProtocolMessageType(
+ "StringStats",
+ (_message.Message,),
+ dict(
+ UnigramStats=_reflection.GeneratedProtocolMessageType(
+ "UnigramStats",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_STRINGSTATS_UNIGRAMSTATS,
+ __module__="google.cloud.automl_v1beta1.proto.data_stats_pb2",
+ __doc__="""The statistics of a unigram.
+
+
+ Attributes:
+ value:
+ The unigram.
+ count:
+ The number of occurrences of this unigram in the series.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.StringStats.UnigramStats)
+ ),
+ ),
+ DESCRIPTOR=_STRINGSTATS,
+ __module__="google.cloud.automl_v1beta1.proto.data_stats_pb2",
+ __doc__="""The data statistics of a series of STRING values.
+
+
+ Attributes:
+ top_unigram_stats:
+ The statistics of the top 20 unigrams, ordered by [count][goog
+ le.cloud.automl.v1beta1.StringStats.UnigramStats.count].
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.StringStats)
+ ),
+)
+_sym_db.RegisterMessage(StringStats)
+_sym_db.RegisterMessage(StringStats.UnigramStats)
+
+TimestampStats = _reflection.GeneratedProtocolMessageType(
+ "TimestampStats",
+ (_message.Message,),
+ dict(
+ GranularStats=_reflection.GeneratedProtocolMessageType(
+ "GranularStats",
+ (_message.Message,),
+ dict(
+ BucketsEntry=_reflection.GeneratedProtocolMessageType(
+ "BucketsEntry",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TIMESTAMPSTATS_GRANULARSTATS_BUCKETSENTRY,
+ __module__="google.cloud.automl_v1beta1.proto.data_stats_pb2"
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.TimestampStats.GranularStats.BucketsEntry)
+ ),
+ ),
+ DESCRIPTOR=_TIMESTAMPSTATS_GRANULARSTATS,
+ __module__="google.cloud.automl_v1beta1.proto.data_stats_pb2",
+ __doc__="""Stats split by a defined in context granularity.
+
+
+ Attributes:
+ buckets:
+ A map from granularity key to example count for that key. E.g.
+ for hour\_of\_day ``13`` means 1pm, or for month\_of\_year
+ ``5`` means May).
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.TimestampStats.GranularStats)
+ ),
+ ),
+ GranularStatsEntry=_reflection.GeneratedProtocolMessageType(
+ "GranularStatsEntry",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TIMESTAMPSTATS_GRANULARSTATSENTRY,
+ __module__="google.cloud.automl_v1beta1.proto.data_stats_pb2"
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.TimestampStats.GranularStatsEntry)
+ ),
+ ),
+ DESCRIPTOR=_TIMESTAMPSTATS,
+ __module__="google.cloud.automl_v1beta1.proto.data_stats_pb2",
+ __doc__="""The data statistics of a series of TIMESTAMP values.
+
+
+ Attributes:
+ granular_stats:
+ The string key is the pre-defined granularity. Currently
+ supported: hour\_of\_day, day\_of\_week, month\_of\_year.
+ Granularities finer that the granularity of timestamp data are
+ not populated (e.g. if timestamps are at day granularity, then
+ hour\_of\_day is not populated).
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.TimestampStats)
+ ),
+)
+_sym_db.RegisterMessage(TimestampStats)
+_sym_db.RegisterMessage(TimestampStats.GranularStats)
+_sym_db.RegisterMessage(TimestampStats.GranularStats.BucketsEntry)
+_sym_db.RegisterMessage(TimestampStats.GranularStatsEntry)
+
+ArrayStats = _reflection.GeneratedProtocolMessageType(
+ "ArrayStats",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_ARRAYSTATS,
+ __module__="google.cloud.automl_v1beta1.proto.data_stats_pb2",
+ __doc__="""The data statistics of a series of ARRAY values.
+
+
+ Attributes:
+ member_stats:
+ Stats of all the values of all arrays, as if they were a
+ single long series of data. The type depends on the element
+ type of the array.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ArrayStats)
+ ),
+)
+_sym_db.RegisterMessage(ArrayStats)
+
+StructStats = _reflection.GeneratedProtocolMessageType(
+ "StructStats",
+ (_message.Message,),
+ dict(
+ FieldStatsEntry=_reflection.GeneratedProtocolMessageType(
+ "FieldStatsEntry",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_STRUCTSTATS_FIELDSTATSENTRY,
+ __module__="google.cloud.automl_v1beta1.proto.data_stats_pb2"
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.StructStats.FieldStatsEntry)
+ ),
+ ),
+ DESCRIPTOR=_STRUCTSTATS,
+ __module__="google.cloud.automl_v1beta1.proto.data_stats_pb2",
+ __doc__="""The data statistics of a series of STRUCT values.
+
+
+ Attributes:
+ field_stats:
+ Map from a field name of the struct to data stats aggregated
+ over series of all data in that field across all the structs.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.StructStats)
+ ),
+)
+_sym_db.RegisterMessage(StructStats)
+_sym_db.RegisterMessage(StructStats.FieldStatsEntry)
+
+CategoryStats = _reflection.GeneratedProtocolMessageType(
+ "CategoryStats",
+ (_message.Message,),
+ dict(
+ SingleCategoryStats=_reflection.GeneratedProtocolMessageType(
+ "SingleCategoryStats",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_CATEGORYSTATS_SINGLECATEGORYSTATS,
+ __module__="google.cloud.automl_v1beta1.proto.data_stats_pb2",
+ __doc__="""The statistics of a single CATEGORY value.
+
+
+ Attributes:
+ value:
+ The CATEGORY value.
+ count:
+ The number of occurrences of this value in the series.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.CategoryStats.SingleCategoryStats)
+ ),
+ ),
+ DESCRIPTOR=_CATEGORYSTATS,
+ __module__="google.cloud.automl_v1beta1.proto.data_stats_pb2",
+ __doc__="""The data statistics of a series of CATEGORY values.
+
+
+ Attributes:
+ top_category_stats:
+ The statistics of the top 20 CATEGORY values, ordered by [cou
+ nt][google.cloud.automl.v1beta1.CategoryStats.SingleCategorySt
+ ats.count].
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.CategoryStats)
+ ),
+)
+_sym_db.RegisterMessage(CategoryStats)
+_sym_db.RegisterMessage(CategoryStats.SingleCategoryStats)
+
+CorrelationStats = _reflection.GeneratedProtocolMessageType(
+ "CorrelationStats",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_CORRELATIONSTATS,
+ __module__="google.cloud.automl_v1beta1.proto.data_stats_pb2",
+ __doc__="""A correlation statistics between two series of DataType values. The
+ series may have differing DataType-s, but within a single series the
+ DataType must be the same.
+
+
+ Attributes:
+ cramers_v:
+ The correlation value using the Cramer's V measure.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.CorrelationStats)
+ ),
+)
+_sym_db.RegisterMessage(CorrelationStats)
+
+
+DESCRIPTOR._options = None
+_TIMESTAMPSTATS_GRANULARSTATS_BUCKETSENTRY._options = None
+_TIMESTAMPSTATS_GRANULARSTATSENTRY._options = None
+_STRUCTSTATS_FIELDSTATSENTRY._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/automl/google/cloud/automl_v1beta1/proto/data_stats_pb2_grpc.py b/automl/google/cloud/automl_v1beta1/proto/data_stats_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/data_stats_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/automl/google/cloud/automl_v1beta1/proto/data_types.proto b/automl/google/cloud/automl_v1beta1/proto/data_types.proto
new file mode 100644
index 000000000000..086e96e32629
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/data_types.proto
@@ -0,0 +1,106 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.automl.v1beta1;
+
+import "google/api/annotations.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl";
+option java_multiple_files = true;
+option java_package = "com.google.cloud.automl.v1beta1";
+option php_namespace = "Google\\Cloud\\AutoMl\\V1beta1";
+option ruby_package = "Google::Cloud::AutoML::V1beta1";
+
+// `TypeCode` is used as a part of
+// [DataType][google.cloud.automl.v1beta1.DataType].
+enum TypeCode {
+ // Not specified. Should not be used.
+ TYPE_CODE_UNSPECIFIED = 0;
+
+ // Encoded as `number`, or the strings `"NaN"`, `"Infinity"`, or
+ // `"-Infinity"`.
+ FLOAT64 = 3;
+
+ // Must be between 0AD and 9999AD. Encoded as `string` according to
+ // [time_format][google.cloud.automl.v1beta1.DataType.time_format], or, if
+ // that format is not set, then in RFC 3339 `date-time` format, where
+ // `time-offset` = `"Z"` (e.g. 1985-04-12T23:20:50.52Z).
+ TIMESTAMP = 4;
+
+ // Encoded as `string`.
+ STRING = 6;
+
+ // Encoded as `list`, where the list elements are represented according to
+ //
+ // [list_element_type][google.cloud.automl.v1beta1.DataType.list_element_type].
+ ARRAY = 8;
+
+ // Encoded as `struct`, where field values are represented according to
+ // [struct_type][google.cloud.automl.v1beta1.DataType.struct_type].
+ STRUCT = 9;
+
+ // Values of this type are not further understood by AutoML,
+ // e.g. AutoML is unable to tell the order of values (as it could with
+ // FLOAT64), or is unable to say if one value contains another (as it
+ // could with STRING).
+ // Encoded as `string` (bytes should be base64-encoded, as described in RFC
+ // 4648, section 4).
+ CATEGORY = 10;
+}
+
+// Indicated the type of data that can be stored in a structured data entity
+// (e.g. a table).
+message DataType {
+ // Details of DataType-s that need additional specification.
+ oneof details {
+ // If [type_code][google.cloud.automl.v1beta1.DataType.type_code] == [ARRAY][google.cloud.automl.v1beta1.TypeCode.ARRAY],
+ // then `list_element_type` is the type of the elements.
+ DataType list_element_type = 2;
+
+ // If [type_code][google.cloud.automl.v1beta1.DataType.type_code] == [STRUCT][google.cloud.automl.v1beta1.TypeCode.STRUCT], then `struct_type`
+ // provides type information for the struct's fields.
+ StructType struct_type = 3;
+
+ // If [type_code][google.cloud.automl.v1beta1.DataType.type_code] == [TIMESTAMP][google.cloud.automl.v1beta1.TypeCode.TIMESTAMP]
+ // then `time_format` provides the format in which that time field is
+ // expressed. The time_format must either be one of:
+ // * `UNIX_SECONDS`
+ // * `UNIX_MILLISECONDS`
+ // * `UNIX_MICROSECONDS`
+ // * `UNIX_NANOSECONDS`
+ // (for respectively number of seconds, milliseconds, microseconds and
+ // nanoseconds since start of the Unix epoch);
+ // or be written in `strftime` syntax. If time_format is not set, then the
+ // default format as described on the type_code is used.
+ string time_format = 5;
+ }
+
+ // Required. The [TypeCode][google.cloud.automl.v1beta1.TypeCode] for this type.
+ TypeCode type_code = 1;
+
+ // If true, this DataType can also be `NULL`. In .CSV files `NULL` value is
+ // expressed as an empty string.
+ bool nullable = 4;
+}
+
+// `StructType` defines the DataType-s of a [STRUCT][google.cloud.automl.v1beta1.TypeCode.STRUCT] type.
+message StructType {
+ // Unordered map of struct field names to their data types.
+ // Fields cannot be added or removed via Update. Their names and
+ // data types are still mutable.
+ map fields = 1;
+}
diff --git a/automl/google/cloud/automl_v1beta1/proto/data_types_pb2.py b/automl/google/cloud/automl_v1beta1/proto/data_types_pb2.py
new file mode 100644
index 000000000000..631850a58fc6
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/data_types_pb2.py
@@ -0,0 +1,408 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/automl_v1beta1/proto/data_types.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf.internal import enum_type_wrapper
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/automl_v1beta1/proto/data_types.proto",
+ package="google.cloud.automl.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\037com.google.cloud.automl.v1beta1P\001ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\312\002\033Google\\Cloud\\AutoMl\\V1beta1\352\002\036Google::Cloud::AutoML::V1beta1"
+ ),
+ serialized_pb=_b(
+ '\n2google/cloud/automl_v1beta1/proto/data_types.proto\x12\x1bgoogle.cloud.automl.v1beta1\x1a\x1cgoogle/api/annotations.proto"\xfc\x01\n\x08\x44\x61taType\x12\x42\n\x11list_element_type\x18\x02 \x01(\x0b\x32%.google.cloud.automl.v1beta1.DataTypeH\x00\x12>\n\x0bstruct_type\x18\x03 \x01(\x0b\x32\'.google.cloud.automl.v1beta1.StructTypeH\x00\x12\x15\n\x0btime_format\x18\x05 \x01(\tH\x00\x12\x38\n\ttype_code\x18\x01 \x01(\x0e\x32%.google.cloud.automl.v1beta1.TypeCode\x12\x10\n\x08nullable\x18\x04 \x01(\x08\x42\t\n\x07\x64\x65tails"\xa7\x01\n\nStructType\x12\x43\n\x06\x66ields\x18\x01 \x03(\x0b\x32\x33.google.cloud.automl.v1beta1.StructType.FieldsEntry\x1aT\n\x0b\x46ieldsEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\x34\n\x05value\x18\x02 \x01(\x0b\x32%.google.cloud.automl.v1beta1.DataType:\x02\x38\x01*r\n\x08TypeCode\x12\x19\n\x15TYPE_CODE_UNSPECIFIED\x10\x00\x12\x0b\n\x07\x46LOAT64\x10\x03\x12\r\n\tTIMESTAMP\x10\x04\x12\n\n\x06STRING\x10\x06\x12\t\n\x05\x41RRAY\x10\x08\x12\n\n\x06STRUCT\x10\t\x12\x0c\n\x08\x43\x41TEGORY\x10\nB\xa5\x01\n\x1f\x63om.google.cloud.automl.v1beta1P\x01ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\xca\x02\x1bGoogle\\Cloud\\AutoMl\\V1beta1\xea\x02\x1eGoogle::Cloud::AutoML::V1beta1b\x06proto3'
+ ),
+ dependencies=[google_dot_api_dot_annotations__pb2.DESCRIPTOR],
+)
+
+_TYPECODE = _descriptor.EnumDescriptor(
+ name="TypeCode",
+ full_name="google.cloud.automl.v1beta1.TypeCode",
+ filename=None,
+ file=DESCRIPTOR,
+ values=[
+ _descriptor.EnumValueDescriptor(
+ name="TYPE_CODE_UNSPECIFIED",
+ index=0,
+ number=0,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="FLOAT64", index=1, number=3, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="TIMESTAMP", index=2, number=4, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="STRING", index=3, number=6, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="ARRAY", index=4, number=8, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="STRUCT", index=5, number=9, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="CATEGORY", index=6, number=10, serialized_options=None, type=None
+ ),
+ ],
+ containing_type=None,
+ serialized_options=None,
+ serialized_start=538,
+ serialized_end=652,
+)
+_sym_db.RegisterEnumDescriptor(_TYPECODE)
+
+TypeCode = enum_type_wrapper.EnumTypeWrapper(_TYPECODE)
+TYPE_CODE_UNSPECIFIED = 0
+FLOAT64 = 3
+TIMESTAMP = 4
+STRING = 6
+ARRAY = 8
+STRUCT = 9
+CATEGORY = 10
+
+
+_DATATYPE = _descriptor.Descriptor(
+ name="DataType",
+ full_name="google.cloud.automl.v1beta1.DataType",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="list_element_type",
+ full_name="google.cloud.automl.v1beta1.DataType.list_element_type",
+ index=0,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="struct_type",
+ full_name="google.cloud.automl.v1beta1.DataType.struct_type",
+ index=1,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="time_format",
+ full_name="google.cloud.automl.v1beta1.DataType.time_format",
+ index=2,
+ number=5,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="type_code",
+ full_name="google.cloud.automl.v1beta1.DataType.type_code",
+ index=3,
+ number=1,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="nullable",
+ full_name="google.cloud.automl.v1beta1.DataType.nullable",
+ index=4,
+ number=4,
+ type=8,
+ cpp_type=7,
+ label=1,
+ has_default_value=False,
+ default_value=False,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="details",
+ full_name="google.cloud.automl.v1beta1.DataType.details",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=114,
+ serialized_end=366,
+)
+
+
+_STRUCTTYPE_FIELDSENTRY = _descriptor.Descriptor(
+ name="FieldsEntry",
+ full_name="google.cloud.automl.v1beta1.StructType.FieldsEntry",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="key",
+ full_name="google.cloud.automl.v1beta1.StructType.FieldsEntry.key",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="value",
+ full_name="google.cloud.automl.v1beta1.StructType.FieldsEntry.value",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=_b("8\001"),
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=452,
+ serialized_end=536,
+)
+
+_STRUCTTYPE = _descriptor.Descriptor(
+ name="StructType",
+ full_name="google.cloud.automl.v1beta1.StructType",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="fields",
+ full_name="google.cloud.automl.v1beta1.StructType.fields",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[_STRUCTTYPE_FIELDSENTRY],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=369,
+ serialized_end=536,
+)
+
+_DATATYPE.fields_by_name["list_element_type"].message_type = _DATATYPE
+_DATATYPE.fields_by_name["struct_type"].message_type = _STRUCTTYPE
+_DATATYPE.fields_by_name["type_code"].enum_type = _TYPECODE
+_DATATYPE.oneofs_by_name["details"].fields.append(
+ _DATATYPE.fields_by_name["list_element_type"]
+)
+_DATATYPE.fields_by_name[
+ "list_element_type"
+].containing_oneof = _DATATYPE.oneofs_by_name["details"]
+_DATATYPE.oneofs_by_name["details"].fields.append(
+ _DATATYPE.fields_by_name["struct_type"]
+)
+_DATATYPE.fields_by_name["struct_type"].containing_oneof = _DATATYPE.oneofs_by_name[
+ "details"
+]
+_DATATYPE.oneofs_by_name["details"].fields.append(
+ _DATATYPE.fields_by_name["time_format"]
+)
+_DATATYPE.fields_by_name["time_format"].containing_oneof = _DATATYPE.oneofs_by_name[
+ "details"
+]
+_STRUCTTYPE_FIELDSENTRY.fields_by_name["value"].message_type = _DATATYPE
+_STRUCTTYPE_FIELDSENTRY.containing_type = _STRUCTTYPE
+_STRUCTTYPE.fields_by_name["fields"].message_type = _STRUCTTYPE_FIELDSENTRY
+DESCRIPTOR.message_types_by_name["DataType"] = _DATATYPE
+DESCRIPTOR.message_types_by_name["StructType"] = _STRUCTTYPE
+DESCRIPTOR.enum_types_by_name["TypeCode"] = _TYPECODE
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+DataType = _reflection.GeneratedProtocolMessageType(
+ "DataType",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_DATATYPE,
+ __module__="google.cloud.automl_v1beta1.proto.data_types_pb2",
+ __doc__="""Indicated the type of data that can be stored in a structured data
+ entity (e.g. a table).
+
+
+ Attributes:
+ details:
+ Details of DataType-s that need additional specification.
+ list_element_type:
+ If
+ [type\_code][google.cloud.automl.v1beta1.DataType.type\_code]
+ == [ARRAY][google.cloud.automl.v1beta1.TypeCode.ARRAY], then
+ ``list_element_type`` is the type of the elements.
+ struct_type:
+ If
+ [type\_code][google.cloud.automl.v1beta1.DataType.type\_code]
+ == [STRUCT][google.cloud.automl.v1beta1.TypeCode.STRUCT], then
+ ``struct_type`` provides type information for the struct's
+ fields.
+ time_format:
+ If
+ [type\_code][google.cloud.automl.v1beta1.DataType.type\_code]
+ == [TIMESTAMP][google.cloud.automl.v1beta1.TypeCode.TIMESTAMP]
+ then ``time_format`` provides the format in which that time
+ field is expressed. The time\_format must either be one of: \*
+ ``UNIX_SECONDS`` \* ``UNIX_MILLISECONDS`` \*
+ ``UNIX_MICROSECONDS`` \* ``UNIX_NANOSECONDS`` (for
+ respectively number of seconds, milliseconds, microseconds and
+ nanoseconds since start of the Unix epoch); or be written in
+ ``strftime`` syntax. If time\_format is not set, then the
+ default format as described on the type\_code is used.
+ type_code:
+ Required. The [TypeCode][google.cloud.automl.v1beta1.TypeCode]
+ for this type.
+ nullable:
+ If true, this DataType can also be ``NULL``. In .CSV files
+ ``NULL`` value is expressed as an empty string.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.DataType)
+ ),
+)
+_sym_db.RegisterMessage(DataType)
+
+StructType = _reflection.GeneratedProtocolMessageType(
+ "StructType",
+ (_message.Message,),
+ dict(
+ FieldsEntry=_reflection.GeneratedProtocolMessageType(
+ "FieldsEntry",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_STRUCTTYPE_FIELDSENTRY,
+ __module__="google.cloud.automl_v1beta1.proto.data_types_pb2"
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.StructType.FieldsEntry)
+ ),
+ ),
+ DESCRIPTOR=_STRUCTTYPE,
+ __module__="google.cloud.automl_v1beta1.proto.data_types_pb2",
+ __doc__="""``StructType`` defines the DataType-s of a
+ [STRUCT][google.cloud.automl.v1beta1.TypeCode.STRUCT] type.
+
+
+ Attributes:
+ fields:
+ Unordered map of struct field names to their data types.
+ Fields cannot be added or removed via Update. Their names and
+ data types are still mutable.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.StructType)
+ ),
+)
+_sym_db.RegisterMessage(StructType)
+_sym_db.RegisterMessage(StructType.FieldsEntry)
+
+
+DESCRIPTOR._options = None
+_STRUCTTYPE_FIELDSENTRY._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/automl/google/cloud/automl_v1beta1/proto/data_types_pb2_grpc.py b/automl/google/cloud/automl_v1beta1/proto/data_types_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/data_types_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/automl/google/cloud/automl_v1beta1/proto/dataset.proto b/automl/google/cloud/automl_v1beta1/proto/dataset.proto
new file mode 100644
index 000000000000..e07b17841777
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/dataset.proto
@@ -0,0 +1,91 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.automl.v1beta1;
+
+import "google/cloud/automl/v1beta1/image.proto";
+import "google/cloud/automl/v1beta1/tables.proto";
+import "google/cloud/automl/v1beta1/text.proto";
+import "google/cloud/automl/v1beta1/translation.proto";
+import "google/cloud/automl/v1beta1/video.proto";
+import "google/protobuf/timestamp.proto";
+import "google/api/annotations.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl";
+option java_multiple_files = true;
+option java_package = "com.google.cloud.automl.v1beta1";
+option php_namespace = "Google\\Cloud\\AutoMl\\V1beta1";
+option ruby_package = "Google::Cloud::AutoML::V1beta1";
+
+// A workspace for solving a single, particular machine learning (ML) problem.
+// A workspace contains examples that may be annotated.
+message Dataset {
+ // Required.
+ // The dataset metadata that is specific to the problem type.
+ oneof dataset_metadata {
+ // Metadata for a dataset used for translation.
+ TranslationDatasetMetadata translation_dataset_metadata = 23;
+
+ // Metadata for a dataset used for image classification.
+ ImageClassificationDatasetMetadata image_classification_dataset_metadata = 24;
+
+ // Metadata for a dataset used for text classification.
+ TextClassificationDatasetMetadata text_classification_dataset_metadata = 25;
+
+ // Metadata for a dataset used for image object detection.
+ ImageObjectDetectionDatasetMetadata image_object_detection_dataset_metadata = 26;
+
+ // Metadata for a dataset used for video classification.
+ VideoClassificationDatasetMetadata video_classification_dataset_metadata = 31;
+
+ // Metadata for a dataset used for video object tracking.
+ VideoObjectTrackingDatasetMetadata video_object_tracking_dataset_metadata = 29;
+
+ // Metadata for a dataset used for text extraction.
+ TextExtractionDatasetMetadata text_extraction_dataset_metadata = 28;
+
+ // Metadata for a dataset used for text sentiment.
+ TextSentimentDatasetMetadata text_sentiment_dataset_metadata = 30;
+
+ // Metadata for a dataset used for Tables.
+ TablesDatasetMetadata tables_dataset_metadata = 33;
+ }
+
+ // Output only. The resource name of the dataset.
+ // Form: `projects/{project_id}/locations/{location_id}/datasets/{dataset_id}`
+ string name = 1;
+
+ // Required. The name of the dataset to show in the interface. The name can be
+ // up to 32 characters long and can consist only of ASCII Latin letters A-Z
+ // and a-z, underscores
+ // (_), and ASCII digits 0-9.
+ string display_name = 2;
+
+ // User-provided description of the dataset. The description can be up to
+ // 25000 characters long.
+ string description = 3;
+
+ // Output only. The number of examples in the dataset.
+ int32 example_count = 21;
+
+ // Output only. Timestamp when this dataset was created.
+ google.protobuf.Timestamp create_time = 14;
+
+ // Used to perform consistent read-modify-write updates. If not set, a blind
+ // "overwrite" update happens.
+ string etag = 17;
+}
diff --git a/automl/google/cloud/automl_v1beta1/proto/dataset_pb2.py b/automl/google/cloud/automl_v1beta1/proto/dataset_pb2.py
new file mode 100644
index 000000000000..b313da91b4dd
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/dataset_pb2.py
@@ -0,0 +1,520 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/automl_v1beta1/proto/dataset.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.cloud.automl_v1beta1.proto import (
+ image_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_image__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ tables_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_tables__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ text_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_text__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ translation_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_translation__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ video_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_video__pb2,
+)
+from google.protobuf import timestamp_pb2 as google_dot_protobuf_dot_timestamp__pb2
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/automl_v1beta1/proto/dataset.proto",
+ package="google.cloud.automl.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\037com.google.cloud.automl.v1beta1P\001ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\312\002\033Google\\Cloud\\AutoMl\\V1beta1\352\002\036Google::Cloud::AutoML::V1beta1"
+ ),
+ serialized_pb=_b(
+ "\n/google/cloud/automl_v1beta1/proto/dataset.proto\x12\x1bgoogle.cloud.automl.v1beta1\x1a-google/cloud/automl_v1beta1/proto/image.proto\x1a.google/cloud/automl_v1beta1/proto/tables.proto\x1a,google/cloud/automl_v1beta1/proto/text.proto\x1a\x33google/cloud/automl_v1beta1/proto/translation.proto\x1a-google/cloud/automl_v1beta1/proto/video.proto\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x1cgoogle/api/annotations.proto\"\xee\x08\n\x07\x44\x61taset\x12_\n\x1ctranslation_dataset_metadata\x18\x17 \x01(\x0b\x32\x37.google.cloud.automl.v1beta1.TranslationDatasetMetadataH\x00\x12p\n%image_classification_dataset_metadata\x18\x18 \x01(\x0b\x32?.google.cloud.automl.v1beta1.ImageClassificationDatasetMetadataH\x00\x12n\n$text_classification_dataset_metadata\x18\x19 \x01(\x0b\x32>.google.cloud.automl.v1beta1.TextClassificationDatasetMetadataH\x00\x12s\n'image_object_detection_dataset_metadata\x18\x1a \x01(\x0b\x32@.google.cloud.automl.v1beta1.ImageObjectDetectionDatasetMetadataH\x00\x12p\n%video_classification_dataset_metadata\x18\x1f \x01(\x0b\x32?.google.cloud.automl.v1beta1.VideoClassificationDatasetMetadataH\x00\x12q\n&video_object_tracking_dataset_metadata\x18\x1d \x01(\x0b\x32?.google.cloud.automl.v1beta1.VideoObjectTrackingDatasetMetadataH\x00\x12\x66\n text_extraction_dataset_metadata\x18\x1c \x01(\x0b\x32:.google.cloud.automl.v1beta1.TextExtractionDatasetMetadataH\x00\x12\x64\n\x1ftext_sentiment_dataset_metadata\x18\x1e \x01(\x0b\x32\x39.google.cloud.automl.v1beta1.TextSentimentDatasetMetadataH\x00\x12U\n\x17tables_dataset_metadata\x18! \x01(\x0b\x32\x32.google.cloud.automl.v1beta1.TablesDatasetMetadataH\x00\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x14\n\x0c\x64isplay_name\x18\x02 \x01(\t\x12\x13\n\x0b\x64\x65scription\x18\x03 \x01(\t\x12\x15\n\rexample_count\x18\x15 \x01(\x05\x12/\n\x0b\x63reate_time\x18\x0e \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12\x0c\n\x04\x65tag\x18\x11 \x01(\tB\x12\n\x10\x64\x61taset_metadataB\xa5\x01\n\x1f\x63om.google.cloud.automl.v1beta1P\x01ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\xca\x02\x1bGoogle\\Cloud\\AutoMl\\V1beta1\xea\x02\x1eGoogle::Cloud::AutoML::V1beta1b\x06proto3"
+ ),
+ dependencies=[
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_image__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_tables__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_text__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_translation__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_video__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_timestamp__pb2.DESCRIPTOR,
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ ],
+)
+
+
+_DATASET = _descriptor.Descriptor(
+ name="Dataset",
+ full_name="google.cloud.automl.v1beta1.Dataset",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="translation_dataset_metadata",
+ full_name="google.cloud.automl.v1beta1.Dataset.translation_dataset_metadata",
+ index=0,
+ number=23,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="image_classification_dataset_metadata",
+ full_name="google.cloud.automl.v1beta1.Dataset.image_classification_dataset_metadata",
+ index=1,
+ number=24,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="text_classification_dataset_metadata",
+ full_name="google.cloud.automl.v1beta1.Dataset.text_classification_dataset_metadata",
+ index=2,
+ number=25,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="image_object_detection_dataset_metadata",
+ full_name="google.cloud.automl.v1beta1.Dataset.image_object_detection_dataset_metadata",
+ index=3,
+ number=26,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="video_classification_dataset_metadata",
+ full_name="google.cloud.automl.v1beta1.Dataset.video_classification_dataset_metadata",
+ index=4,
+ number=31,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="video_object_tracking_dataset_metadata",
+ full_name="google.cloud.automl.v1beta1.Dataset.video_object_tracking_dataset_metadata",
+ index=5,
+ number=29,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="text_extraction_dataset_metadata",
+ full_name="google.cloud.automl.v1beta1.Dataset.text_extraction_dataset_metadata",
+ index=6,
+ number=28,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="text_sentiment_dataset_metadata",
+ full_name="google.cloud.automl.v1beta1.Dataset.text_sentiment_dataset_metadata",
+ index=7,
+ number=30,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="tables_dataset_metadata",
+ full_name="google.cloud.automl.v1beta1.Dataset.tables_dataset_metadata",
+ index=8,
+ number=33,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.automl.v1beta1.Dataset.name",
+ index=9,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="display_name",
+ full_name="google.cloud.automl.v1beta1.Dataset.display_name",
+ index=10,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="description",
+ full_name="google.cloud.automl.v1beta1.Dataset.description",
+ index=11,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="example_count",
+ full_name="google.cloud.automl.v1beta1.Dataset.example_count",
+ index=12,
+ number=21,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="create_time",
+ full_name="google.cloud.automl.v1beta1.Dataset.create_time",
+ index=13,
+ number=14,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="etag",
+ full_name="google.cloud.automl.v1beta1.Dataset.etag",
+ index=14,
+ number=17,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="dataset_metadata",
+ full_name="google.cloud.automl.v1beta1.Dataset.dataset_metadata",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=385,
+ serialized_end=1519,
+)
+
+_DATASET.fields_by_name[
+ "translation_dataset_metadata"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_translation__pb2._TRANSLATIONDATASETMETADATA
+)
+_DATASET.fields_by_name[
+ "image_classification_dataset_metadata"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_image__pb2._IMAGECLASSIFICATIONDATASETMETADATA
+)
+_DATASET.fields_by_name[
+ "text_classification_dataset_metadata"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_text__pb2._TEXTCLASSIFICATIONDATASETMETADATA
+)
+_DATASET.fields_by_name[
+ "image_object_detection_dataset_metadata"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_image__pb2._IMAGEOBJECTDETECTIONDATASETMETADATA
+)
+_DATASET.fields_by_name[
+ "video_classification_dataset_metadata"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_video__pb2._VIDEOCLASSIFICATIONDATASETMETADATA
+)
+_DATASET.fields_by_name[
+ "video_object_tracking_dataset_metadata"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_video__pb2._VIDEOOBJECTTRACKINGDATASETMETADATA
+)
+_DATASET.fields_by_name[
+ "text_extraction_dataset_metadata"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_text__pb2._TEXTEXTRACTIONDATASETMETADATA
+)
+_DATASET.fields_by_name[
+ "text_sentiment_dataset_metadata"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_text__pb2._TEXTSENTIMENTDATASETMETADATA
+)
+_DATASET.fields_by_name[
+ "tables_dataset_metadata"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_tables__pb2._TABLESDATASETMETADATA
+)
+_DATASET.fields_by_name[
+ "create_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_DATASET.oneofs_by_name["dataset_metadata"].fields.append(
+ _DATASET.fields_by_name["translation_dataset_metadata"]
+)
+_DATASET.fields_by_name[
+ "translation_dataset_metadata"
+].containing_oneof = _DATASET.oneofs_by_name["dataset_metadata"]
+_DATASET.oneofs_by_name["dataset_metadata"].fields.append(
+ _DATASET.fields_by_name["image_classification_dataset_metadata"]
+)
+_DATASET.fields_by_name[
+ "image_classification_dataset_metadata"
+].containing_oneof = _DATASET.oneofs_by_name["dataset_metadata"]
+_DATASET.oneofs_by_name["dataset_metadata"].fields.append(
+ _DATASET.fields_by_name["text_classification_dataset_metadata"]
+)
+_DATASET.fields_by_name[
+ "text_classification_dataset_metadata"
+].containing_oneof = _DATASET.oneofs_by_name["dataset_metadata"]
+_DATASET.oneofs_by_name["dataset_metadata"].fields.append(
+ _DATASET.fields_by_name["image_object_detection_dataset_metadata"]
+)
+_DATASET.fields_by_name[
+ "image_object_detection_dataset_metadata"
+].containing_oneof = _DATASET.oneofs_by_name["dataset_metadata"]
+_DATASET.oneofs_by_name["dataset_metadata"].fields.append(
+ _DATASET.fields_by_name["video_classification_dataset_metadata"]
+)
+_DATASET.fields_by_name[
+ "video_classification_dataset_metadata"
+].containing_oneof = _DATASET.oneofs_by_name["dataset_metadata"]
+_DATASET.oneofs_by_name["dataset_metadata"].fields.append(
+ _DATASET.fields_by_name["video_object_tracking_dataset_metadata"]
+)
+_DATASET.fields_by_name[
+ "video_object_tracking_dataset_metadata"
+].containing_oneof = _DATASET.oneofs_by_name["dataset_metadata"]
+_DATASET.oneofs_by_name["dataset_metadata"].fields.append(
+ _DATASET.fields_by_name["text_extraction_dataset_metadata"]
+)
+_DATASET.fields_by_name[
+ "text_extraction_dataset_metadata"
+].containing_oneof = _DATASET.oneofs_by_name["dataset_metadata"]
+_DATASET.oneofs_by_name["dataset_metadata"].fields.append(
+ _DATASET.fields_by_name["text_sentiment_dataset_metadata"]
+)
+_DATASET.fields_by_name[
+ "text_sentiment_dataset_metadata"
+].containing_oneof = _DATASET.oneofs_by_name["dataset_metadata"]
+_DATASET.oneofs_by_name["dataset_metadata"].fields.append(
+ _DATASET.fields_by_name["tables_dataset_metadata"]
+)
+_DATASET.fields_by_name[
+ "tables_dataset_metadata"
+].containing_oneof = _DATASET.oneofs_by_name["dataset_metadata"]
+DESCRIPTOR.message_types_by_name["Dataset"] = _DATASET
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+Dataset = _reflection.GeneratedProtocolMessageType(
+ "Dataset",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_DATASET,
+ __module__="google.cloud.automl_v1beta1.proto.dataset_pb2",
+ __doc__="""A workspace for solving a single, particular machine learning (ML)
+ problem. A workspace contains examples that may be annotated.
+
+
+ Attributes:
+ dataset_metadata:
+ Required. The dataset metadata that is specific to the problem
+ type.
+ translation_dataset_metadata:
+ Metadata for a dataset used for translation.
+ image_classification_dataset_metadata:
+ Metadata for a dataset used for image classification.
+ text_classification_dataset_metadata:
+ Metadata for a dataset used for text classification.
+ image_object_detection_dataset_metadata:
+ Metadata for a dataset used for image object detection.
+ video_classification_dataset_metadata:
+ Metadata for a dataset used for video classification.
+ video_object_tracking_dataset_metadata:
+ Metadata for a dataset used for video object tracking.
+ text_extraction_dataset_metadata:
+ Metadata for a dataset used for text extraction.
+ text_sentiment_dataset_metadata:
+ Metadata for a dataset used for text sentiment.
+ tables_dataset_metadata:
+ Metadata for a dataset used for Tables.
+ name:
+ Output only. The resource name of the dataset. Form: ``project
+ s/{project_id}/locations/{location_id}/datasets/{dataset_id}``
+ display_name:
+ Required. The name of the dataset to show in the interface.
+ The name can be up to 32 characters long and can consist only
+ of ASCII Latin letters A-Z and a-z, underscores (\_), and
+ ASCII digits 0-9.
+ description:
+ User-provided description of the dataset. The description can
+ be up to 25000 characters long.
+ example_count:
+ Output only. The number of examples in the dataset.
+ create_time:
+ Output only. Timestamp when this dataset was created.
+ etag:
+ Used to perform consistent read-modify-write updates. If not
+ set, a blind "overwrite" update happens.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.Dataset)
+ ),
+)
+_sym_db.RegisterMessage(Dataset)
+
+
+DESCRIPTOR._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/automl/google/cloud/automl_v1beta1/proto/dataset_pb2_grpc.py b/automl/google/cloud/automl_v1beta1/proto/dataset_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/dataset_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/automl/google/cloud/automl_v1beta1/proto/detection.proto b/automl/google/cloud/automl_v1beta1/proto/detection.proto
new file mode 100644
index 000000000000..99761fd52c57
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/detection.proto
@@ -0,0 +1,136 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.automl.v1beta1;
+
+import "google/cloud/automl/v1beta1/geometry.proto";
+import "google/protobuf/duration.proto";
+import "google/api/annotations.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl";
+option java_multiple_files = true;
+option java_package = "com.google.cloud.automl.v1beta1";
+option php_namespace = "Google\\Cloud\\AutoMl\\V1beta1";
+option ruby_package = "Google::Cloud::AutoML::V1beta1";
+
+// Annotation details for image object detection.
+message ImageObjectDetectionAnnotation {
+ // Output only. The rectangle representing the object location.
+ BoundingPoly bounding_box = 1;
+
+ // Output only. The confidence that this annotation is positive for the parent example,
+ // value in [0, 1], higher means higher positivity confidence.
+ float score = 2;
+}
+
+// Annotation details for video object tracking.
+message VideoObjectTrackingAnnotation {
+ // Optional. The instance of the object, expressed as a positive integer. Used to tell
+ // apart objects of the same type (i.e. AnnotationSpec) when multiple are
+ // present on a single example.
+ // NOTE: Instance ID prediction quality is not a part of model evaluation and
+ // is done as best effort. Especially in cases when an entity goes
+ // off-screen for a longer time (minutes), when it comes back it may be given
+ // a new instance ID.
+ string instance_id = 1;
+
+ // Required. A time (frame) of a video to which this annotation pertains.
+ // Represented as the duration since the video's start.
+ google.protobuf.Duration time_offset = 2;
+
+ // Required. The rectangle representing the object location on the frame (i.e.
+ // at the time_offset of the video).
+ BoundingPoly bounding_box = 3;
+
+ // Output only. The confidence that this annotation is positive for the video at
+ // the time_offset, value in [0, 1], higher means higher positivity
+ // confidence. For annotations created by the user the score is 1. When
+ // user approves an annotation, the original float score is kept (and not
+ // changed to 1).
+ float score = 4;
+}
+
+// Bounding box matching model metrics for a single intersection-over-union
+// threshold and multiple label match confidence thresholds.
+message BoundingBoxMetricsEntry {
+ // Metrics for a single confidence threshold.
+ message ConfidenceMetricsEntry {
+ // Output only. The confidence threshold value used to compute the metrics.
+ float confidence_threshold = 1;
+
+ // Output only. Recall under the given confidence threshold.
+ float recall = 2;
+
+ // Output only. Precision under the given confidence threshold.
+ float precision = 3;
+
+ // Output only. The harmonic mean of recall and precision.
+ float f1_score = 4;
+ }
+
+ // Output only. The intersection-over-union threshold value used to compute
+ // this metrics entry.
+ float iou_threshold = 1;
+
+ // Output only. The mean average precision, most often close to au_prc.
+ float mean_average_precision = 2;
+
+ // Output only. Metrics for each label-match confidence_threshold from
+ // 0.05,0.10,...,0.95,0.96,0.97,0.98,0.99. Precision-recall curve is
+ // derived from them.
+ repeated ConfidenceMetricsEntry confidence_metrics_entries = 3;
+}
+
+// Model evaluation metrics for image object detection problems.
+// Evaluates prediction quality of labeled bounding boxes.
+message ImageObjectDetectionEvaluationMetrics {
+ // Output only. The total number of bounding boxes (i.e. summed over all
+ // images) the ground truth used to create this evaluation had.
+ int32 evaluated_bounding_box_count = 1;
+
+ // Output only. The bounding boxes match metrics for each
+ // Intersection-over-union threshold 0.05,0.10,...,0.95,0.96,0.97,0.98,0.99
+ // and each label confidence threshold 0.05,0.10,...,0.95,0.96,0.97,0.98,0.99
+ // pair.
+ repeated BoundingBoxMetricsEntry bounding_box_metrics_entries = 2;
+
+ // Output only. The single metric for bounding boxes evaluation:
+ // the mean_average_precision averaged over all bounding_box_metrics_entries.
+ float bounding_box_mean_average_precision = 3;
+}
+
+// Model evaluation metrics for video object tracking problems.
+// Evaluates prediction quality of both labeled bounding boxes and labeled
+// tracks (i.e. series of bounding boxes sharing same label and instance ID).
+message VideoObjectTrackingEvaluationMetrics {
+ // Output only. The number of video frames used to create this evaluation.
+ int32 evaluated_frame_count = 1;
+
+ // Output only. The total number of bounding boxes (i.e. summed over all
+ // frames) the ground truth used to create this evaluation had.
+ int32 evaluated_bounding_box_count = 2;
+
+ // Output only. The bounding boxes match metrics for each
+ // Intersection-over-union threshold 0.05,0.10,...,0.95,0.96,0.97,0.98,0.99
+ // and each label confidence threshold 0.05,0.10,...,0.95,0.96,0.97,0.98,0.99
+ // pair.
+ repeated BoundingBoxMetricsEntry bounding_box_metrics_entries = 4;
+
+ // Output only. The single metric for bounding boxes evaluation:
+ // the mean_average_precision averaged over all bounding_box_metrics_entries.
+ float bounding_box_mean_average_precision = 6;
+}
diff --git a/automl/google/cloud/automl_v1beta1/proto/detection_pb2.py b/automl/google/cloud/automl_v1beta1/proto/detection_pb2.py
new file mode 100644
index 000000000000..c9c4296fedda
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/detection_pb2.py
@@ -0,0 +1,745 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/automl_v1beta1/proto/detection.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.cloud.automl_v1beta1.proto import (
+ geometry_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_geometry__pb2,
+)
+from google.protobuf import duration_pb2 as google_dot_protobuf_dot_duration__pb2
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/automl_v1beta1/proto/detection.proto",
+ package="google.cloud.automl.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\037com.google.cloud.automl.v1beta1P\001ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\312\002\033Google\\Cloud\\AutoMl\\V1beta1\352\002\036Google::Cloud::AutoML::V1beta1"
+ ),
+ serialized_pb=_b(
+ '\n1google/cloud/automl_v1beta1/proto/detection.proto\x12\x1bgoogle.cloud.automl.v1beta1\x1a\x30google/cloud/automl_v1beta1/proto/geometry.proto\x1a\x1egoogle/protobuf/duration.proto\x1a\x1cgoogle/api/annotations.proto"p\n\x1eImageObjectDetectionAnnotation\x12?\n\x0c\x62ounding_box\x18\x01 \x01(\x0b\x32).google.cloud.automl.v1beta1.BoundingPoly\x12\r\n\x05score\x18\x02 \x01(\x02"\xb4\x01\n\x1dVideoObjectTrackingAnnotation\x12\x13\n\x0binstance_id\x18\x01 \x01(\t\x12.\n\x0btime_offset\x18\x02 \x01(\x0b\x32\x19.google.protobuf.Duration\x12?\n\x0c\x62ounding_box\x18\x03 \x01(\x0b\x32).google.cloud.automl.v1beta1.BoundingPoly\x12\r\n\x05score\x18\x04 \x01(\x02"\xae\x02\n\x17\x42oundingBoxMetricsEntry\x12\x15\n\riou_threshold\x18\x01 \x01(\x02\x12\x1e\n\x16mean_average_precision\x18\x02 \x01(\x02\x12o\n\x1a\x63onfidence_metrics_entries\x18\x03 \x03(\x0b\x32K.google.cloud.automl.v1beta1.BoundingBoxMetricsEntry.ConfidenceMetricsEntry\x1ak\n\x16\x43onfidenceMetricsEntry\x12\x1c\n\x14\x63onfidence_threshold\x18\x01 \x01(\x02\x12\x0e\n\x06recall\x18\x02 \x01(\x02\x12\x11\n\tprecision\x18\x03 \x01(\x02\x12\x10\n\x08\x66\x31_score\x18\x04 \x01(\x02"\xd6\x01\n%ImageObjectDetectionEvaluationMetrics\x12$\n\x1c\x65valuated_bounding_box_count\x18\x01 \x01(\x05\x12Z\n\x1c\x62ounding_box_metrics_entries\x18\x02 \x03(\x0b\x32\x34.google.cloud.automl.v1beta1.BoundingBoxMetricsEntry\x12+\n#bounding_box_mean_average_precision\x18\x03 \x01(\x02"\xf4\x01\n$VideoObjectTrackingEvaluationMetrics\x12\x1d\n\x15\x65valuated_frame_count\x18\x01 \x01(\x05\x12$\n\x1c\x65valuated_bounding_box_count\x18\x02 \x01(\x05\x12Z\n\x1c\x62ounding_box_metrics_entries\x18\x04 \x03(\x0b\x32\x34.google.cloud.automl.v1beta1.BoundingBoxMetricsEntry\x12+\n#bounding_box_mean_average_precision\x18\x06 \x01(\x02\x42\xa5\x01\n\x1f\x63om.google.cloud.automl.v1beta1P\x01ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\xca\x02\x1bGoogle\\Cloud\\AutoMl\\V1beta1\xea\x02\x1eGoogle::Cloud::AutoML::V1beta1b\x06proto3'
+ ),
+ dependencies=[
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_geometry__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_duration__pb2.DESCRIPTOR,
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ ],
+)
+
+
+_IMAGEOBJECTDETECTIONANNOTATION = _descriptor.Descriptor(
+ name="ImageObjectDetectionAnnotation",
+ full_name="google.cloud.automl.v1beta1.ImageObjectDetectionAnnotation",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="bounding_box",
+ full_name="google.cloud.automl.v1beta1.ImageObjectDetectionAnnotation.bounding_box",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="score",
+ full_name="google.cloud.automl.v1beta1.ImageObjectDetectionAnnotation.score",
+ index=1,
+ number=2,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=194,
+ serialized_end=306,
+)
+
+
+_VIDEOOBJECTTRACKINGANNOTATION = _descriptor.Descriptor(
+ name="VideoObjectTrackingAnnotation",
+ full_name="google.cloud.automl.v1beta1.VideoObjectTrackingAnnotation",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="instance_id",
+ full_name="google.cloud.automl.v1beta1.VideoObjectTrackingAnnotation.instance_id",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="time_offset",
+ full_name="google.cloud.automl.v1beta1.VideoObjectTrackingAnnotation.time_offset",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="bounding_box",
+ full_name="google.cloud.automl.v1beta1.VideoObjectTrackingAnnotation.bounding_box",
+ index=2,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="score",
+ full_name="google.cloud.automl.v1beta1.VideoObjectTrackingAnnotation.score",
+ index=3,
+ number=4,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=309,
+ serialized_end=489,
+)
+
+
+_BOUNDINGBOXMETRICSENTRY_CONFIDENCEMETRICSENTRY = _descriptor.Descriptor(
+ name="ConfidenceMetricsEntry",
+ full_name="google.cloud.automl.v1beta1.BoundingBoxMetricsEntry.ConfidenceMetricsEntry",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="confidence_threshold",
+ full_name="google.cloud.automl.v1beta1.BoundingBoxMetricsEntry.ConfidenceMetricsEntry.confidence_threshold",
+ index=0,
+ number=1,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="recall",
+ full_name="google.cloud.automl.v1beta1.BoundingBoxMetricsEntry.ConfidenceMetricsEntry.recall",
+ index=1,
+ number=2,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="precision",
+ full_name="google.cloud.automl.v1beta1.BoundingBoxMetricsEntry.ConfidenceMetricsEntry.precision",
+ index=2,
+ number=3,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="f1_score",
+ full_name="google.cloud.automl.v1beta1.BoundingBoxMetricsEntry.ConfidenceMetricsEntry.f1_score",
+ index=3,
+ number=4,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=687,
+ serialized_end=794,
+)
+
+_BOUNDINGBOXMETRICSENTRY = _descriptor.Descriptor(
+ name="BoundingBoxMetricsEntry",
+ full_name="google.cloud.automl.v1beta1.BoundingBoxMetricsEntry",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="iou_threshold",
+ full_name="google.cloud.automl.v1beta1.BoundingBoxMetricsEntry.iou_threshold",
+ index=0,
+ number=1,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="mean_average_precision",
+ full_name="google.cloud.automl.v1beta1.BoundingBoxMetricsEntry.mean_average_precision",
+ index=1,
+ number=2,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="confidence_metrics_entries",
+ full_name="google.cloud.automl.v1beta1.BoundingBoxMetricsEntry.confidence_metrics_entries",
+ index=2,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[_BOUNDINGBOXMETRICSENTRY_CONFIDENCEMETRICSENTRY],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=492,
+ serialized_end=794,
+)
+
+
+_IMAGEOBJECTDETECTIONEVALUATIONMETRICS = _descriptor.Descriptor(
+ name="ImageObjectDetectionEvaluationMetrics",
+ full_name="google.cloud.automl.v1beta1.ImageObjectDetectionEvaluationMetrics",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="evaluated_bounding_box_count",
+ full_name="google.cloud.automl.v1beta1.ImageObjectDetectionEvaluationMetrics.evaluated_bounding_box_count",
+ index=0,
+ number=1,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="bounding_box_metrics_entries",
+ full_name="google.cloud.automl.v1beta1.ImageObjectDetectionEvaluationMetrics.bounding_box_metrics_entries",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="bounding_box_mean_average_precision",
+ full_name="google.cloud.automl.v1beta1.ImageObjectDetectionEvaluationMetrics.bounding_box_mean_average_precision",
+ index=2,
+ number=3,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=797,
+ serialized_end=1011,
+)
+
+
+_VIDEOOBJECTTRACKINGEVALUATIONMETRICS = _descriptor.Descriptor(
+ name="VideoObjectTrackingEvaluationMetrics",
+ full_name="google.cloud.automl.v1beta1.VideoObjectTrackingEvaluationMetrics",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="evaluated_frame_count",
+ full_name="google.cloud.automl.v1beta1.VideoObjectTrackingEvaluationMetrics.evaluated_frame_count",
+ index=0,
+ number=1,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="evaluated_bounding_box_count",
+ full_name="google.cloud.automl.v1beta1.VideoObjectTrackingEvaluationMetrics.evaluated_bounding_box_count",
+ index=1,
+ number=2,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="bounding_box_metrics_entries",
+ full_name="google.cloud.automl.v1beta1.VideoObjectTrackingEvaluationMetrics.bounding_box_metrics_entries",
+ index=2,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="bounding_box_mean_average_precision",
+ full_name="google.cloud.automl.v1beta1.VideoObjectTrackingEvaluationMetrics.bounding_box_mean_average_precision",
+ index=3,
+ number=6,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1014,
+ serialized_end=1258,
+)
+
+_IMAGEOBJECTDETECTIONANNOTATION.fields_by_name[
+ "bounding_box"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_geometry__pb2._BOUNDINGPOLY
+)
+_VIDEOOBJECTTRACKINGANNOTATION.fields_by_name[
+ "time_offset"
+].message_type = google_dot_protobuf_dot_duration__pb2._DURATION
+_VIDEOOBJECTTRACKINGANNOTATION.fields_by_name[
+ "bounding_box"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_geometry__pb2._BOUNDINGPOLY
+)
+_BOUNDINGBOXMETRICSENTRY_CONFIDENCEMETRICSENTRY.containing_type = (
+ _BOUNDINGBOXMETRICSENTRY
+)
+_BOUNDINGBOXMETRICSENTRY.fields_by_name[
+ "confidence_metrics_entries"
+].message_type = _BOUNDINGBOXMETRICSENTRY_CONFIDENCEMETRICSENTRY
+_IMAGEOBJECTDETECTIONEVALUATIONMETRICS.fields_by_name[
+ "bounding_box_metrics_entries"
+].message_type = _BOUNDINGBOXMETRICSENTRY
+_VIDEOOBJECTTRACKINGEVALUATIONMETRICS.fields_by_name[
+ "bounding_box_metrics_entries"
+].message_type = _BOUNDINGBOXMETRICSENTRY
+DESCRIPTOR.message_types_by_name[
+ "ImageObjectDetectionAnnotation"
+] = _IMAGEOBJECTDETECTIONANNOTATION
+DESCRIPTOR.message_types_by_name[
+ "VideoObjectTrackingAnnotation"
+] = _VIDEOOBJECTTRACKINGANNOTATION
+DESCRIPTOR.message_types_by_name["BoundingBoxMetricsEntry"] = _BOUNDINGBOXMETRICSENTRY
+DESCRIPTOR.message_types_by_name[
+ "ImageObjectDetectionEvaluationMetrics"
+] = _IMAGEOBJECTDETECTIONEVALUATIONMETRICS
+DESCRIPTOR.message_types_by_name[
+ "VideoObjectTrackingEvaluationMetrics"
+] = _VIDEOOBJECTTRACKINGEVALUATIONMETRICS
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+ImageObjectDetectionAnnotation = _reflection.GeneratedProtocolMessageType(
+ "ImageObjectDetectionAnnotation",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_IMAGEOBJECTDETECTIONANNOTATION,
+ __module__="google.cloud.automl_v1beta1.proto.detection_pb2",
+ __doc__="""Annotation details for image object detection.
+
+
+ Attributes:
+ bounding_box:
+ Output only. The rectangle representing the object location.
+ score:
+ Output only. The confidence that this annotation is positive
+ for the parent example, value in [0, 1], higher means higher
+ positivity confidence.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ImageObjectDetectionAnnotation)
+ ),
+)
+_sym_db.RegisterMessage(ImageObjectDetectionAnnotation)
+
+VideoObjectTrackingAnnotation = _reflection.GeneratedProtocolMessageType(
+ "VideoObjectTrackingAnnotation",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_VIDEOOBJECTTRACKINGANNOTATION,
+ __module__="google.cloud.automl_v1beta1.proto.detection_pb2",
+ __doc__="""Annotation details for video object tracking.
+
+
+ Attributes:
+ instance_id:
+ Optional. The instance of the object, expressed as a positive
+ integer. Used to tell apart objects of the same type (i.e.
+ AnnotationSpec) when multiple are present on a single example.
+ NOTE: Instance ID prediction quality is not a part of model
+ evaluation and is done as best effort. Especially in cases
+ when an entity goes off-screen for a longer time (minutes),
+ when it comes back it may be given a new instance ID.
+ time_offset:
+ Required. A time (frame) of a video to which this annotation
+ pertains. Represented as the duration since the video's start.
+ bounding_box:
+ Required. The rectangle representing the object location on
+ the frame (i.e. at the time\_offset of the video).
+ score:
+ Output only. The confidence that this annotation is positive
+ for the video at the time\_offset, value in [0, 1], higher
+ means higher positivity confidence. For annotations created by
+ the user the score is 1. When user approves an annotation, the
+ original float score is kept (and not changed to 1).
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.VideoObjectTrackingAnnotation)
+ ),
+)
+_sym_db.RegisterMessage(VideoObjectTrackingAnnotation)
+
+BoundingBoxMetricsEntry = _reflection.GeneratedProtocolMessageType(
+ "BoundingBoxMetricsEntry",
+ (_message.Message,),
+ dict(
+ ConfidenceMetricsEntry=_reflection.GeneratedProtocolMessageType(
+ "ConfidenceMetricsEntry",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_BOUNDINGBOXMETRICSENTRY_CONFIDENCEMETRICSENTRY,
+ __module__="google.cloud.automl_v1beta1.proto.detection_pb2",
+ __doc__="""Metrics for a single confidence threshold.
+
+
+ Attributes:
+ confidence_threshold:
+ Output only. The confidence threshold value used to compute
+ the metrics.
+ recall:
+ Output only. Recall under the given confidence threshold.
+ precision:
+ Output only. Precision under the given confidence threshold.
+ f1_score:
+ Output only. The harmonic mean of recall and precision.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.BoundingBoxMetricsEntry.ConfidenceMetricsEntry)
+ ),
+ ),
+ DESCRIPTOR=_BOUNDINGBOXMETRICSENTRY,
+ __module__="google.cloud.automl_v1beta1.proto.detection_pb2",
+ __doc__="""Bounding box matching model metrics for a single intersection-over-union
+ threshold and multiple label match confidence thresholds.
+
+
+ Attributes:
+ iou_threshold:
+ Output only. The intersection-over-union threshold value used
+ to compute this metrics entry.
+ mean_average_precision:
+ Output only. The mean average precision, most often close to
+ au\_prc.
+ confidence_metrics_entries:
+ Output only. Metrics for each label-match
+ confidence\_threshold from
+ 0.05,0.10,...,0.95,0.96,0.97,0.98,0.99. Precision-recall curve
+ is derived from them.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.BoundingBoxMetricsEntry)
+ ),
+)
+_sym_db.RegisterMessage(BoundingBoxMetricsEntry)
+_sym_db.RegisterMessage(BoundingBoxMetricsEntry.ConfidenceMetricsEntry)
+
+ImageObjectDetectionEvaluationMetrics = _reflection.GeneratedProtocolMessageType(
+ "ImageObjectDetectionEvaluationMetrics",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_IMAGEOBJECTDETECTIONEVALUATIONMETRICS,
+ __module__="google.cloud.automl_v1beta1.proto.detection_pb2",
+ __doc__="""Model evaluation metrics for image object detection problems. Evaluates
+ prediction quality of labeled bounding boxes.
+
+
+ Attributes:
+ evaluated_bounding_box_count:
+ Output only. The total number of bounding boxes (i.e. summed
+ over all images) the ground truth used to create this
+ evaluation had.
+ bounding_box_metrics_entries:
+ Output only. The bounding boxes match metrics for each
+ Intersection-over-union threshold
+ 0.05,0.10,...,0.95,0.96,0.97,0.98,0.99 and each label
+ confidence threshold 0.05,0.10,...,0.95,0.96,0.97,0.98,0.99
+ pair.
+ bounding_box_mean_average_precision:
+ Output only. The single metric for bounding boxes evaluation:
+ the mean\_average\_precision averaged over all
+ bounding\_box\_metrics\_entries.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ImageObjectDetectionEvaluationMetrics)
+ ),
+)
+_sym_db.RegisterMessage(ImageObjectDetectionEvaluationMetrics)
+
+VideoObjectTrackingEvaluationMetrics = _reflection.GeneratedProtocolMessageType(
+ "VideoObjectTrackingEvaluationMetrics",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_VIDEOOBJECTTRACKINGEVALUATIONMETRICS,
+ __module__="google.cloud.automl_v1beta1.proto.detection_pb2",
+ __doc__="""Model evaluation metrics for video object tracking problems. Evaluates
+ prediction quality of both labeled bounding boxes and labeled tracks
+ (i.e. series of bounding boxes sharing same label and instance ID).
+
+
+ Attributes:
+ evaluated_frame_count:
+ Output only. The number of video frames used to create this
+ evaluation.
+ evaluated_bounding_box_count:
+ Output only. The total number of bounding boxes (i.e. summed
+ over all frames) the ground truth used to create this
+ evaluation had.
+ bounding_box_metrics_entries:
+ Output only. The bounding boxes match metrics for each
+ Intersection-over-union threshold
+ 0.05,0.10,...,0.95,0.96,0.97,0.98,0.99 and each label
+ confidence threshold 0.05,0.10,...,0.95,0.96,0.97,0.98,0.99
+ pair.
+ bounding_box_mean_average_precision:
+ Output only. The single metric for bounding boxes evaluation:
+ the mean\_average\_precision averaged over all
+ bounding\_box\_metrics\_entries.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.VideoObjectTrackingEvaluationMetrics)
+ ),
+)
+_sym_db.RegisterMessage(VideoObjectTrackingEvaluationMetrics)
+
+
+DESCRIPTOR._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/automl/google/cloud/automl_v1beta1/proto/detection_pb2_grpc.py b/automl/google/cloud/automl_v1beta1/proto/detection_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/detection_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/automl/google/cloud/automl_v1beta1/proto/geometry.proto b/automl/google/cloud/automl_v1beta1/proto/geometry.proto
new file mode 100644
index 000000000000..e5379ab14df6
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/geometry.proto
@@ -0,0 +1,47 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.automl.v1beta1;
+
+import "google/api/annotations.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl";
+option java_multiple_files = true;
+option java_package = "com.google.cloud.automl.v1beta1";
+option php_namespace = "Google\\Cloud\\AutoMl\\V1beta1";
+option ruby_package = "Google::Cloud::AutoML::V1beta1";
+
+// A vertex represents a 2D point in the image.
+// The normalized vertex coordinates are between 0 to 1 fractions relative to
+// the original plane (image, video). E.g. if the plane (e.g. whole image) would
+// have size 10 x 20 then a point with normalized coordinates (0.1, 0.3) would
+// be at the position (1, 6) on that plane.
+message NormalizedVertex {
+ // Required. Horizontal coordinate.
+ float x = 1;
+
+ // Required. Vertical coordinate.
+ float y = 2;
+}
+
+// A bounding polygon of a detected object on a plane.
+// On output both vertices and normalized_vertices are provided.
+// The polygon is formed by connecting vertices in the order they are listed.
+message BoundingPoly {
+ // Output only . The bounding polygon normalized vertices.
+ repeated NormalizedVertex normalized_vertices = 2;
+}
diff --git a/automl/google/cloud/automl_v1beta1/proto/geometry_pb2.py b/automl/google/cloud/automl_v1beta1/proto/geometry_pb2.py
new file mode 100644
index 000000000000..935a2e4d7819
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/geometry_pb2.py
@@ -0,0 +1,175 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/automl_v1beta1/proto/geometry.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/automl_v1beta1/proto/geometry.proto",
+ package="google.cloud.automl.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\037com.google.cloud.automl.v1beta1P\001ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\312\002\033Google\\Cloud\\AutoMl\\V1beta1\352\002\036Google::Cloud::AutoML::V1beta1"
+ ),
+ serialized_pb=_b(
+ '\n0google/cloud/automl_v1beta1/proto/geometry.proto\x12\x1bgoogle.cloud.automl.v1beta1\x1a\x1cgoogle/api/annotations.proto"(\n\x10NormalizedVertex\x12\t\n\x01x\x18\x01 \x01(\x02\x12\t\n\x01y\x18\x02 \x01(\x02"Z\n\x0c\x42oundingPoly\x12J\n\x13normalized_vertices\x18\x02 \x03(\x0b\x32-.google.cloud.automl.v1beta1.NormalizedVertexB\xa5\x01\n\x1f\x63om.google.cloud.automl.v1beta1P\x01ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\xca\x02\x1bGoogle\\Cloud\\AutoMl\\V1beta1\xea\x02\x1eGoogle::Cloud::AutoML::V1beta1b\x06proto3'
+ ),
+ dependencies=[google_dot_api_dot_annotations__pb2.DESCRIPTOR],
+)
+
+
+_NORMALIZEDVERTEX = _descriptor.Descriptor(
+ name="NormalizedVertex",
+ full_name="google.cloud.automl.v1beta1.NormalizedVertex",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="x",
+ full_name="google.cloud.automl.v1beta1.NormalizedVertex.x",
+ index=0,
+ number=1,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="y",
+ full_name="google.cloud.automl.v1beta1.NormalizedVertex.y",
+ index=1,
+ number=2,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=111,
+ serialized_end=151,
+)
+
+
+_BOUNDINGPOLY = _descriptor.Descriptor(
+ name="BoundingPoly",
+ full_name="google.cloud.automl.v1beta1.BoundingPoly",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="normalized_vertices",
+ full_name="google.cloud.automl.v1beta1.BoundingPoly.normalized_vertices",
+ index=0,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=153,
+ serialized_end=243,
+)
+
+_BOUNDINGPOLY.fields_by_name["normalized_vertices"].message_type = _NORMALIZEDVERTEX
+DESCRIPTOR.message_types_by_name["NormalizedVertex"] = _NORMALIZEDVERTEX
+DESCRIPTOR.message_types_by_name["BoundingPoly"] = _BOUNDINGPOLY
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+NormalizedVertex = _reflection.GeneratedProtocolMessageType(
+ "NormalizedVertex",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_NORMALIZEDVERTEX,
+ __module__="google.cloud.automl_v1beta1.proto.geometry_pb2",
+ __doc__="""Required. Horizontal coordinate.
+
+
+ Attributes:
+ y:
+ Required. Vertical coordinate.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.NormalizedVertex)
+ ),
+)
+_sym_db.RegisterMessage(NormalizedVertex)
+
+BoundingPoly = _reflection.GeneratedProtocolMessageType(
+ "BoundingPoly",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_BOUNDINGPOLY,
+ __module__="google.cloud.automl_v1beta1.proto.geometry_pb2",
+ __doc__="""A bounding polygon of a detected object on a plane. On output both
+ vertices and normalized\_vertices are provided. The polygon is formed by
+ connecting vertices in the order they are listed.
+
+
+ Attributes:
+ normalized_vertices:
+ Output only . The bounding polygon normalized vertices.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.BoundingPoly)
+ ),
+)
+_sym_db.RegisterMessage(BoundingPoly)
+
+
+DESCRIPTOR._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/automl/google/cloud/automl_v1beta1/proto/geometry_pb2_grpc.py b/automl/google/cloud/automl_v1beta1/proto/geometry_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/geometry_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/automl/google/cloud/automl_v1beta1/proto/image.proto b/automl/google/cloud/automl_v1beta1/proto/image.proto
new file mode 100644
index 000000000000..241d11ebef04
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/image.proto
@@ -0,0 +1,158 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.automl.v1beta1;
+
+import "google/cloud/automl/v1beta1/annotation_spec.proto";
+import "google/cloud/automl/v1beta1/classification.proto";
+import "google/protobuf/timestamp.proto";
+import "google/api/annotations.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl";
+option java_multiple_files = true;
+option java_outer_classname = "ImageProto";
+option java_package = "com.google.cloud.automl.v1beta1";
+option php_namespace = "Google\\Cloud\\AutoMl\\V1beta1";
+option ruby_package = "Google::Cloud::AutoML::V1beta1";
+
+// Dataset metadata that is specific to image classification.
+message ImageClassificationDatasetMetadata {
+ // Required. Type of the classification problem.
+ ClassificationType classification_type = 1;
+}
+
+// Dataset metadata specific to image object detection.
+message ImageObjectDetectionDatasetMetadata {
+
+}
+
+// Model metadata for image classification.
+message ImageClassificationModelMetadata {
+ // Optional. The ID of the `base` model. If it is specified, the new model
+ // will be created based on the `base` model. Otherwise, the new model will be
+ // created from scratch. The `base` model must be in the same
+ // `project` and `location` as the new model to create, and have the same
+ // `model_type`.
+ string base_model_id = 1;
+
+ // Required. The train budget of creating this model, expressed in hours. The
+ // actual `train_cost` will be equal or less than this value.
+ int64 train_budget = 2;
+
+ // Output only. The actual train cost of creating this model, expressed in
+ // hours. If this model is created from a `base` model, the train cost used
+ // to create the `base` model are not included.
+ int64 train_cost = 3;
+
+ // Output only. The reason that this create model operation stopped,
+ // e.g. `BUDGET_REACHED`, `MODEL_CONVERGED`.
+ string stop_reason = 5;
+
+ // Optional. Type of the model. The available values are:
+ // * `cloud` - Model to be used via prediction calls to AutoML API.
+ // This is the default value.
+ // * `mobile-low-latency-1` - A model that, in addition to providing
+ // prediction via AutoML API, can also be exported (see
+ // [AutoMl.ExportModel][google.cloud.automl.v1beta1.AutoMl.ExportModel]) and used on a mobile or edge device
+ // with TensorFlow afterwards. Expected to have low latency, but
+ // may have lower prediction quality than other models.
+ // * `mobile-versatile-1` - A model that, in addition to providing
+ // prediction via AutoML API, can also be exported (see
+ // [AutoMl.ExportModel][google.cloud.automl.v1beta1.AutoMl.ExportModel]) and used on a mobile or edge device
+ // with TensorFlow afterwards.
+ // * `mobile-high-accuracy-1` - A model that, in addition to providing
+ // prediction via AutoML API, can also be exported (see
+ // [AutoMl.ExportModel][google.cloud.automl.v1beta1.AutoMl.ExportModel]) and used on a mobile or edge device
+ // with TensorFlow afterwards. Expected to have a higher
+ // latency, but should also have a higher prediction quality
+ // than other models.
+ // * `mobile-core-ml-low-latency-1` - A model that, in addition to providing
+ // prediction via AutoML API, can also be exported (see
+ // [AutoMl.ExportModel][google.cloud.automl.v1beta1.AutoMl.ExportModel]) and used on a mobile device with Core
+ // ML afterwards. Expected to have low latency, but may have
+ // lower prediction quality than other models.
+ // * `mobile-core-ml-versatile-1` - A model that, in addition to providing
+ // prediction via AutoML API, can also be exported (see
+ // [AutoMl.ExportModel][google.cloud.automl.v1beta1.AutoMl.ExportModel]) and used on a mobile device with Core
+ // ML afterwards.
+ // * `mobile-core-ml-high-accuracy-1` - A model that, in addition to
+ // providing prediction via AutoML API, can also be exported
+ // (see [AutoMl.ExportModel][google.cloud.automl.v1beta1.AutoMl.ExportModel]) and used on a mobile device with
+ // Core ML afterwards. Expected to have a higher latency, but
+ // should also have a higher prediction quality than other
+ // models.
+ string model_type = 7;
+}
+
+// Model metadata specific to image object detection.
+message ImageObjectDetectionModelMetadata {
+ // Optional. Type of the model. The available values are:
+ // * `cloud-high-accuracy-1` - (default) A model to be used via prediction
+ // calls to AutoML API. Expected to have a higher latency, but
+ // should also have a higher prediction quality than other
+ // models.
+ // * `cloud-low-latency-1` - A model to be used via prediction
+ // calls to AutoML API. Expected to have low latency, but may
+ // have lower prediction quality than other models.
+ string model_type = 1;
+
+ // Output only. The number of nodes this model is deployed on. A node is an
+ // abstraction of a machine resource, which can handle online prediction QPS
+ // as given in the qps_per_node field.
+ int64 node_count = 3;
+
+ // Output only. An approximate number of online prediction QPS that can
+ // be supported by this model per each node on which it is deployed.
+ double node_qps = 4;
+
+ // Output only. The reason that this create model operation stopped,
+ // e.g. `BUDGET_REACHED`, `MODEL_CONVERGED`.
+ string stop_reason = 5;
+
+ // The train budget of creating this model, expressed in milli node
+ // hours i.e. 1,000 value in this field means 1 node hour. The actual
+ // `train_cost` will be equal or less than this value. If further model
+ // training ceases to provide any improvements, it will stop without using
+ // full budget and the stop_reason will be `MODEL_CONVERGED`.
+ // Note, node_hour = actual_hour * number_of_nodes_invovled.
+ // For model type `cloud-high-accuracy-1`(default) and `cloud-low-latency-1`,
+ // the train budget must be between 20,000 and 2,000,000 milli node hours,
+ // inclusive. The default value is 216, 000 which represents one day in
+ // wall time.
+ // For model type `mobile-low-latency-1`, `mobile-versatile-1`,
+ // `mobile-high-accuracy-1`, `mobile-core-ml-low-latency-1`,
+ // `mobile-core-ml-versatile-1`, `mobile-core-ml-high-accuracy-1`, the train
+ // budget must be between 1,000 and 100,000 milli node hours, inclusive.
+ // The default value is 24, 000 which represents one day in wall time.
+ int64 train_budget_milli_node_hours = 6;
+
+ // Output only. The actual train cost of creating this model, expressed in
+ // milli node hours, i.e. 1,000 value in this field means 1 node hour.
+ // Guaranteed to not exceed the train budget.
+ int64 train_cost_milli_node_hours = 7;
+}
+
+// Model deployment metadata specific to Image Object Detection.
+message ImageObjectDetectionModelDeploymentMetadata {
+ // Input only. The number of nodes to deploy the model on. A node is an
+ // abstraction of a machine resource, which can handle online prediction QPS
+ // as given in the model's
+ //
+ // [qps_per_node][google.cloud.automl.v1beta1.ImageObjectDetectionModelMetadata.qps_per_node].
+ // Must be between 1 and 100, inclusive on both ends.
+ int64 node_count = 1;
+}
diff --git a/automl/google/cloud/automl_v1beta1/proto/image_pb2.py b/automl/google/cloud/automl_v1beta1/proto/image_pb2.py
new file mode 100644
index 000000000000..8efacf229e46
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/image_pb2.py
@@ -0,0 +1,590 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/automl_v1beta1/proto/image.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.cloud.automl_v1beta1.proto import (
+ annotation_spec_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_annotation__spec__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ classification_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_classification__pb2,
+)
+from google.protobuf import timestamp_pb2 as google_dot_protobuf_dot_timestamp__pb2
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/automl_v1beta1/proto/image.proto",
+ package="google.cloud.automl.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\037com.google.cloud.automl.v1beta1B\nImageProtoP\001ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\312\002\033Google\\Cloud\\AutoMl\\V1beta1\352\002\036Google::Cloud::AutoML::V1beta1"
+ ),
+ serialized_pb=_b(
+ '\n-google/cloud/automl_v1beta1/proto/image.proto\x12\x1bgoogle.cloud.automl.v1beta1\x1a\x37google/cloud/automl_v1beta1/proto/annotation_spec.proto\x1a\x36google/cloud/automl_v1beta1/proto/classification.proto\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x1cgoogle/api/annotations.proto"r\n"ImageClassificationDatasetMetadata\x12L\n\x13\x63lassification_type\x18\x01 \x01(\x0e\x32/.google.cloud.automl.v1beta1.ClassificationType"%\n#ImageObjectDetectionDatasetMetadata"\x8c\x01\n ImageClassificationModelMetadata\x12\x15\n\rbase_model_id\x18\x01 \x01(\t\x12\x14\n\x0ctrain_budget\x18\x02 \x01(\x03\x12\x12\n\ntrain_cost\x18\x03 \x01(\x03\x12\x13\n\x0bstop_reason\x18\x05 \x01(\t\x12\x12\n\nmodel_type\x18\x07 \x01(\t"\xbe\x01\n!ImageObjectDetectionModelMetadata\x12\x12\n\nmodel_type\x18\x01 \x01(\t\x12\x12\n\nnode_count\x18\x03 \x01(\x03\x12\x10\n\x08node_qps\x18\x04 \x01(\x01\x12\x13\n\x0bstop_reason\x18\x05 \x01(\t\x12%\n\x1dtrain_budget_milli_node_hours\x18\x06 \x01(\x03\x12#\n\x1btrain_cost_milli_node_hours\x18\x07 \x01(\x03"A\n+ImageObjectDetectionModelDeploymentMetadata\x12\x12\n\nnode_count\x18\x01 \x01(\x03\x42\xb1\x01\n\x1f\x63om.google.cloud.automl.v1beta1B\nImageProtoP\x01ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\xca\x02\x1bGoogle\\Cloud\\AutoMl\\V1beta1\xea\x02\x1eGoogle::Cloud::AutoML::V1beta1b\x06proto3'
+ ),
+ dependencies=[
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_annotation__spec__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_classification__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_timestamp__pb2.DESCRIPTOR,
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ ],
+)
+
+
+_IMAGECLASSIFICATIONDATASETMETADATA = _descriptor.Descriptor(
+ name="ImageClassificationDatasetMetadata",
+ full_name="google.cloud.automl.v1beta1.ImageClassificationDatasetMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="classification_type",
+ full_name="google.cloud.automl.v1beta1.ImageClassificationDatasetMetadata.classification_type",
+ index=0,
+ number=1,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=254,
+ serialized_end=368,
+)
+
+
+_IMAGEOBJECTDETECTIONDATASETMETADATA = _descriptor.Descriptor(
+ name="ImageObjectDetectionDatasetMetadata",
+ full_name="google.cloud.automl.v1beta1.ImageObjectDetectionDatasetMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=370,
+ serialized_end=407,
+)
+
+
+_IMAGECLASSIFICATIONMODELMETADATA = _descriptor.Descriptor(
+ name="ImageClassificationModelMetadata",
+ full_name="google.cloud.automl.v1beta1.ImageClassificationModelMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="base_model_id",
+ full_name="google.cloud.automl.v1beta1.ImageClassificationModelMetadata.base_model_id",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="train_budget",
+ full_name="google.cloud.automl.v1beta1.ImageClassificationModelMetadata.train_budget",
+ index=1,
+ number=2,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="train_cost",
+ full_name="google.cloud.automl.v1beta1.ImageClassificationModelMetadata.train_cost",
+ index=2,
+ number=3,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="stop_reason",
+ full_name="google.cloud.automl.v1beta1.ImageClassificationModelMetadata.stop_reason",
+ index=3,
+ number=5,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="model_type",
+ full_name="google.cloud.automl.v1beta1.ImageClassificationModelMetadata.model_type",
+ index=4,
+ number=7,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=410,
+ serialized_end=550,
+)
+
+
+_IMAGEOBJECTDETECTIONMODELMETADATA = _descriptor.Descriptor(
+ name="ImageObjectDetectionModelMetadata",
+ full_name="google.cloud.automl.v1beta1.ImageObjectDetectionModelMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="model_type",
+ full_name="google.cloud.automl.v1beta1.ImageObjectDetectionModelMetadata.model_type",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="node_count",
+ full_name="google.cloud.automl.v1beta1.ImageObjectDetectionModelMetadata.node_count",
+ index=1,
+ number=3,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="node_qps",
+ full_name="google.cloud.automl.v1beta1.ImageObjectDetectionModelMetadata.node_qps",
+ index=2,
+ number=4,
+ type=1,
+ cpp_type=5,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="stop_reason",
+ full_name="google.cloud.automl.v1beta1.ImageObjectDetectionModelMetadata.stop_reason",
+ index=3,
+ number=5,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="train_budget_milli_node_hours",
+ full_name="google.cloud.automl.v1beta1.ImageObjectDetectionModelMetadata.train_budget_milli_node_hours",
+ index=4,
+ number=6,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="train_cost_milli_node_hours",
+ full_name="google.cloud.automl.v1beta1.ImageObjectDetectionModelMetadata.train_cost_milli_node_hours",
+ index=5,
+ number=7,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=553,
+ serialized_end=743,
+)
+
+
+_IMAGEOBJECTDETECTIONMODELDEPLOYMENTMETADATA = _descriptor.Descriptor(
+ name="ImageObjectDetectionModelDeploymentMetadata",
+ full_name="google.cloud.automl.v1beta1.ImageObjectDetectionModelDeploymentMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="node_count",
+ full_name="google.cloud.automl.v1beta1.ImageObjectDetectionModelDeploymentMetadata.node_count",
+ index=0,
+ number=1,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=745,
+ serialized_end=810,
+)
+
+_IMAGECLASSIFICATIONDATASETMETADATA.fields_by_name[
+ "classification_type"
+].enum_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_classification__pb2._CLASSIFICATIONTYPE
+)
+DESCRIPTOR.message_types_by_name[
+ "ImageClassificationDatasetMetadata"
+] = _IMAGECLASSIFICATIONDATASETMETADATA
+DESCRIPTOR.message_types_by_name[
+ "ImageObjectDetectionDatasetMetadata"
+] = _IMAGEOBJECTDETECTIONDATASETMETADATA
+DESCRIPTOR.message_types_by_name[
+ "ImageClassificationModelMetadata"
+] = _IMAGECLASSIFICATIONMODELMETADATA
+DESCRIPTOR.message_types_by_name[
+ "ImageObjectDetectionModelMetadata"
+] = _IMAGEOBJECTDETECTIONMODELMETADATA
+DESCRIPTOR.message_types_by_name[
+ "ImageObjectDetectionModelDeploymentMetadata"
+] = _IMAGEOBJECTDETECTIONMODELDEPLOYMENTMETADATA
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+ImageClassificationDatasetMetadata = _reflection.GeneratedProtocolMessageType(
+ "ImageClassificationDatasetMetadata",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_IMAGECLASSIFICATIONDATASETMETADATA,
+ __module__="google.cloud.automl_v1beta1.proto.image_pb2",
+ __doc__="""Dataset metadata that is specific to image classification.
+
+
+ Attributes:
+ classification_type:
+ Required. Type of the classification problem.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ImageClassificationDatasetMetadata)
+ ),
+)
+_sym_db.RegisterMessage(ImageClassificationDatasetMetadata)
+
+ImageObjectDetectionDatasetMetadata = _reflection.GeneratedProtocolMessageType(
+ "ImageObjectDetectionDatasetMetadata",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_IMAGEOBJECTDETECTIONDATASETMETADATA,
+ __module__="google.cloud.automl_v1beta1.proto.image_pb2",
+ __doc__="""Dataset metadata specific to image object detection.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ImageObjectDetectionDatasetMetadata)
+ ),
+)
+_sym_db.RegisterMessage(ImageObjectDetectionDatasetMetadata)
+
+ImageClassificationModelMetadata = _reflection.GeneratedProtocolMessageType(
+ "ImageClassificationModelMetadata",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_IMAGECLASSIFICATIONMODELMETADATA,
+ __module__="google.cloud.automl_v1beta1.proto.image_pb2",
+ __doc__="""Model metadata for image classification.
+
+
+ Attributes:
+ base_model_id:
+ Optional. The ID of the ``base`` model. If it is specified,
+ the new model will be created based on the ``base`` model.
+ Otherwise, the new model will be created from scratch. The
+ ``base`` model must be in the same ``project`` and
+ ``location`` as the new model to create, and have the same
+ ``model_type``.
+ train_budget:
+ Required. The train budget of creating this model, expressed
+ in hours. The actual ``train_cost`` will be equal or less than
+ this value.
+ train_cost:
+ Output only. The actual train cost of creating this model,
+ expressed in hours. If this model is created from a ``base``
+ model, the train cost used to create the ``base`` model are
+ not included.
+ stop_reason:
+ Output only. The reason that this create model operation
+ stopped, e.g. ``BUDGET_REACHED``, ``MODEL_CONVERGED``.
+ model_type:
+ Optional. Type of the model. The available values are: \*
+ ``cloud`` - Model to be used via prediction calls to AutoML
+ API. This is the default value. \* ``mobile-low-latency-1`` -
+ A model that, in addition to providing prediction via AutoML
+ API, can also be exported (see [AutoMl.ExportModel][google.clo
+ ud.automl.v1beta1.AutoMl.ExportModel]) and used on a mobile or
+ edge device with TensorFlow afterwards. Expected to have low
+ latency, but may have lower prediction quality than other
+ models. \* ``mobile-versatile-1`` - A model that, in addition
+ to providing prediction via AutoML API, can also be exported
+ (see [AutoMl.ExportModel][google.cloud.automl.v1beta1.AutoMl.E
+ xportModel]) and used on a mobile or edge device with
+ TensorFlow afterwards. \* ``mobile-high-accuracy-1`` - A model
+ that, in addition to providing prediction via AutoML API, can
+ also be exported (see [AutoMl.ExportModel][google.cloud.automl
+ .v1beta1.AutoMl.ExportModel]) and used on a mobile or edge
+ device with TensorFlow afterwards. Expected to have a higher
+ latency, but should also have a higher prediction quality than
+ other models. \* ``mobile-core-ml-low-latency-1`` - A model
+ that, in addition to providing prediction via AutoML API, can
+ also be exported (see [AutoMl.ExportModel][google.cloud.automl
+ .v1beta1.AutoMl.ExportModel]) and used on a mobile device with
+ Core ML afterwards. Expected to have low latency, but may have
+ lower prediction quality than other models. \* ``mobile-core-
+ ml-versatile-1`` - A model that, in addition to providing
+ prediction via AutoML API, can also be exported (see [AutoMl.E
+ xportModel][google.cloud.automl.v1beta1.AutoMl.ExportModel])
+ and used on a mobile device with Core ML afterwards. \*
+ ``mobile-core-ml-high-accuracy-1`` - A model that, in addition
+ to providing prediction via AutoML API, can also be exported
+ (see [AutoMl.ExportModel][google.cloud.automl.v1beta1.AutoMl.E
+ xportModel]) and used on a mobile device with Core ML
+ afterwards. Expected to have a higher latency, but should also
+ have a higher prediction quality than other models.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ImageClassificationModelMetadata)
+ ),
+)
+_sym_db.RegisterMessage(ImageClassificationModelMetadata)
+
+ImageObjectDetectionModelMetadata = _reflection.GeneratedProtocolMessageType(
+ "ImageObjectDetectionModelMetadata",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_IMAGEOBJECTDETECTIONMODELMETADATA,
+ __module__="google.cloud.automl_v1beta1.proto.image_pb2",
+ __doc__="""Model metadata specific to image object detection.
+
+
+ Attributes:
+ model_type:
+ Optional. Type of the model. The available values are: \*
+ ``cloud-high-accuracy-1`` - (default) A model to be used via
+ prediction calls to AutoML API. Expected to have a higher
+ latency, but should also have a higher prediction quality than
+ other models. \* ``cloud-low-latency-1`` - A model to be used
+ via prediction calls to AutoML API. Expected to have low
+ latency, but may have lower prediction quality than other
+ models.
+ node_count:
+ Output only. The number of nodes this model is deployed on. A
+ node is an abstraction of a machine resource, which can handle
+ online prediction QPS as given in the qps\_per\_node field.
+ node_qps:
+ Output only. An approximate number of online prediction QPS
+ that can be supported by this model per each node on which it
+ is deployed.
+ stop_reason:
+ Output only. The reason that this create model operation
+ stopped, e.g. ``BUDGET_REACHED``, ``MODEL_CONVERGED``.
+ train_budget_milli_node_hours:
+ The train budget of creating this model, expressed in milli
+ node hours i.e. 1,000 value in this field means 1 node hour.
+ The actual ``train_cost`` will be equal or less than this
+ value. If further model training ceases to provide any
+ improvements, it will stop without using full budget and the
+ stop\_reason will be ``MODEL_CONVERGED``. Note, node\_hour =
+ actual\_hour \* number\_of\_nodes\_invovled. For model type
+ ``cloud-high-accuracy-1``\ (default) and ``cloud-low-
+ latency-1``, the train budget must be between 20,000 and
+ 2,000,000 milli node hours, inclusive. The default value is
+ 216, 000 which represents one day in wall time. For model type
+ ``mobile-low-latency-1``, ``mobile-versatile-1``, ``mobile-
+ high-accuracy-1``, ``mobile-core-ml-low-latency-1``, ``mobile-
+ core-ml-versatile-1``, ``mobile-core-ml-high-accuracy-1``, the
+ train budget must be between 1,000 and 100,000 milli node
+ hours, inclusive. The default value is 24, 000 which
+ represents one day in wall time.
+ train_cost_milli_node_hours:
+ Output only. The actual train cost of creating this model,
+ expressed in milli node hours, i.e. 1,000 value in this field
+ means 1 node hour. Guaranteed to not exceed the train budget.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ImageObjectDetectionModelMetadata)
+ ),
+)
+_sym_db.RegisterMessage(ImageObjectDetectionModelMetadata)
+
+ImageObjectDetectionModelDeploymentMetadata = _reflection.GeneratedProtocolMessageType(
+ "ImageObjectDetectionModelDeploymentMetadata",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_IMAGEOBJECTDETECTIONMODELDEPLOYMENTMETADATA,
+ __module__="google.cloud.automl_v1beta1.proto.image_pb2",
+ __doc__="""Model deployment metadata specific to Image Object Detection.
+
+
+ Attributes:
+ node_count:
+ Input only. The number of nodes to deploy the model on. A node
+ is an abstraction of a machine resource, which can handle
+ online prediction QPS as given in the model's [qps\_per\_node
+ ][google.cloud.automl.v1beta1.ImageObjectDetectionModelMetadat
+ a.qps\_per\_node]. Must be between 1 and 100, inclusive on
+ both ends.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ImageObjectDetectionModelDeploymentMetadata)
+ ),
+)
+_sym_db.RegisterMessage(ImageObjectDetectionModelDeploymentMetadata)
+
+
+DESCRIPTOR._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/automl/google/cloud/automl_v1beta1/proto/image_pb2_grpc.py b/automl/google/cloud/automl_v1beta1/proto/image_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/image_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/automl/google/cloud/automl_v1beta1/proto/io.proto b/automl/google/cloud/automl_v1beta1/proto/io.proto
new file mode 100644
index 000000000000..6f007f02a10d
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/io.proto
@@ -0,0 +1,1130 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.automl.v1beta1;
+
+import "google/api/annotations.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl";
+option java_multiple_files = true;
+option java_package = "com.google.cloud.automl.v1beta1";
+option php_namespace = "Google\\Cloud\\AutoMl\\V1beta1";
+option ruby_package = "Google::Cloud::AutoML::V1beta1";
+
+// Input configuration for ImportData Action.
+//
+// The format of input depends on dataset_metadata the Dataset into which
+// the import is happening has. As input source the
+// [gcs_source][google.cloud.automl.v1beta1.InputConfig.gcs_source]
+// is expected, unless specified otherwise. Additionally any input .CSV file
+// by itself must be 100MB or smaller, unless specified otherwise.
+// If an "example" file (that is, image, video etc.) with identical content
+// (even if it had different GCS_FILE_PATH) is mentioned multiple times, then
+// its label, bounding boxes etc. are appended. The same file should be always
+// provided with the same ML_USE and GCS_FILE_PATH, if it is not, then
+// these values are nondeterministically selected from the given ones.
+//
+// The formats are represented in EBNF with commas being literal and with
+// non-terminal symbols defined near the end of this comment. The formats are:
+//
+// * For Image Classification:
+// CSV file(s) with each line in format:
+// ML_USE,GCS_FILE_PATH,LABEL,LABEL,...
+// GCS_FILE_PATH leads to image of up to 30MB in size. Supported
+// extensions: .JPEG, .GIF, .PNG, .WEBP, .BMP, .TIFF, .ICO
+// For MULTICLASS classification type, at most one LABEL is allowed
+// per image. If an image has not yet been labeled, then it should be
+// mentioned just once with no LABEL.
+// Some sample rows:
+// TRAIN,gs://folder/image1.jpg,daisy
+// TEST,gs://folder/image2.jpg,dandelion,tulip,rose
+// UNASSIGNED,gs://folder/image3.jpg,daisy
+// UNASSIGNED,gs://folder/image4.jpg
+//
+// * For Image Object Detection:
+// CSV file(s) with each line in format:
+// ML_USE,GCS_FILE_PATH,(LABEL,BOUNDING_BOX | ,,,,,,,)
+// GCS_FILE_PATH leads to image of up to 30MB in size. Supported
+// extensions: .JPEG, .GIF, .PNG.
+// Each image is assumed to be exhaustively labeled. The minimum
+// allowed BOUNDING_BOX edge length is 0.01, and no more than 500
+// BOUNDING_BOX-es per image are allowed (one BOUNDING_BOX is defined
+// per line). If an image has not yet been labeled, then it should be
+// mentioned just once with no LABEL and the ",,,,,,," in place of the
+// BOUNDING_BOX. For images which are known to not contain any
+// bounding boxes, they should be labelled explictly as
+// "NEGATIVE_IMAGE", followed by ",,,,,,," in place of the
+// BOUNDING_BOX.
+// Sample rows:
+// TRAIN,gs://folder/image1.png,car,0.1,0.1,,,0.3,0.3,,
+// TRAIN,gs://folder/image1.png,bike,.7,.6,,,.8,.9,,
+// UNASSIGNED,gs://folder/im2.png,car,0.1,0.1,0.2,0.1,0.2,0.3,0.1,0.3
+// TEST,gs://folder/im3.png,,,,,,,,,
+// TRAIN,gs://folder/im4.png,NEGATIVE_IMAGE,,,,,,,,,
+//
+// * For Video Classification:
+// CSV file(s) with each line in format:
+// ML_USE,GCS_FILE_PATH
+// where ML_USE VALIDATE value should not be used. The GCS_FILE_PATH
+// should lead to another .csv file which describes examples that have
+// given ML_USE, using the following row format:
+// GCS_FILE_PATH,(LABEL,TIME_SEGMENT_START,TIME_SEGMENT_END | ,,)
+// Here GCS_FILE_PATH leads to a video of up to 50GB in size and up
+// to 3h duration. Supported extensions: .MOV, .MPEG4, .MP4, .AVI.
+// TIME_SEGMENT_START and TIME_SEGMENT_END must be within the
+// length of the video, and end has to be after the start. Any segment
+// of a video which has one or more labels on it, is considered a
+// hard negative for all other labels. Any segment with no labels on
+// it is considered to be unknown. If a whole video is unknown, then
+// it shuold be mentioned just once with ",," in place of LABEL,
+// TIME_SEGMENT_START,TIME_SEGMENT_END.
+// Sample top level CSV file:
+// TRAIN,gs://folder/train_videos.csv
+// TEST,gs://folder/test_videos.csv
+// UNASSIGNED,gs://folder/other_videos.csv
+// Sample rows of a CSV file for a particular ML_USE:
+// gs://folder/video1.avi,car,120,180.000021
+// gs://folder/video1.avi,bike,150,180.000021
+// gs://folder/vid2.avi,car,0,60.5
+// gs://folder/vid3.avi,,,
+//
+// * For Video Object Tracking:
+// CSV file(s) with each line in format:
+// ML_USE,GCS_FILE_PATH
+// where ML_USE VALIDATE value should not be used. The GCS_FILE_PATH
+// should lead to another .csv file which describes examples that have
+// given ML_USE, using one of the following row format:
+// GCS_FILE_PATH,LABEL,[INSTANCE_ID],TIMESTAMP,BOUNDING_BOX
+// or
+// GCS_FILE_PATH,,,,,,,,,,
+// Here GCS_FILE_PATH leads to a video of up to 50GB in size and up
+// to 3h duration. Supported extensions: .MOV, .MPEG4, .MP4, .AVI.
+// Providing INSTANCE_IDs can help to obtain a better model. When
+// a specific labeled entity leaves the video frame, and shows up
+// afterwards it is not required, albeit preferable, that the same
+// INSTANCE_ID is given to it.
+// TIMESTAMP must be within the length of the video, the
+// BOUNDING_BOX is assumed to be drawn on the closest video's frame
+// to the TIMESTAMP. Any mentioned by the TIMESTAMP frame is expected
+// to be exhaustively labeled and no more than 500 BOUNDING_BOX-es per
+// frame are allowed. If a whole video is unknown, then it should be
+// mentioned just once with ",,,,,,,,,," in place of LABEL,
+// [INSTANCE_ID],TIMESTAMP,BOUNDING_BOX.
+// Sample top level CSV file:
+// TRAIN,gs://folder/train_videos.csv
+// TEST,gs://folder/test_videos.csv
+// UNASSIGNED,gs://folder/other_videos.csv
+// Seven sample rows of a CSV file for a particular ML_USE:
+// gs://folder/video1.avi,car,1,12.10,0.8,0.8,0.9,0.8,0.9,0.9,0.8,0.9
+// gs://folder/video1.avi,car,1,12.90,0.4,0.8,0.5,0.8,0.5,0.9,0.4,0.9
+// gs://folder/video1.avi,car,2,12.10,.4,.2,.5,.2,.5,.3,.4,.3
+// gs://folder/video1.avi,car,2,12.90,.8,.2,,,.9,.3,,
+// gs://folder/video1.avi,bike,,12.50,.45,.45,,,.55,.55,,
+// gs://folder/video2.avi,car,1,0,.1,.9,,,.9,.1,,
+// gs://folder/video2.avi,,,,,,,,,,,
+// * For Text Extraction:
+// CSV file(s) with each line in format:
+// ML_USE,GCS_FILE_PATH
+// GCS_FILE_PATH leads to a .JSONL (that is, JSON Lines) file which
+// either imports text in-line or as documents. Any given
+// .JSONL file must be 100MB or smaller.
+// The in-line .JSONL file contains, per line, a proto that wraps a
+// TextSnippet proto (in json representation) followed by one or more
+// AnnotationPayload protos (called annotations), which have
+// display_name and text_extraction detail populated. The given text
+// is expected to be annotated exhaustively, for example, if you look
+// for animals and text contains "dolphin" that is not labeled, then
+// "dolphin" is assumed to not be an animal. Any given text snippet
+// content must be 10KB or smaller, and also be UTF-8 NFC encoded
+// (ASCII already is).
+// The document .JSONL file contains, per line, a proto that wraps a
+// Document proto. The Document proto must have either document_text
+// or input_config set. In document_text case, the Document proto may
+// also contain the spatial information of the document, including
+// layout, document dimension and page number. In input_config case,
+// only PDF documents are supported now, and each document may be up
+// to 2MB large. Currently, annotations on documents cannot be
+// specified at import.
+// Three sample CSV rows:
+// TRAIN,gs://folder/file1.jsonl
+// VALIDATE,gs://folder/file2.jsonl
+// TEST,gs://folder/file3.jsonl
+// Sample in-line JSON Lines file for entity extraction (presented here
+// with artificial line breaks, but the only actual line break is
+// denoted by \n).:
+// {
+// "document": {
+// "document_text": {"content": "dog cat"}
+// "layout": [
+// {
+// "text_segment": {
+// "start_offset": 0,
+// "end_offset": 3,
+// },
+// "page_number": 1,
+// "bounding_poly": {
+// "normalized_vertices": [
+// {"x": 0.1, "y": 0.1},
+// {"x": 0.1, "y": 0.3},
+// {"x": 0.3, "y": 0.3},
+// {"x": 0.3, "y": 0.1},
+// ],
+// },
+// "text_segment_type": TOKEN,
+// },
+// {
+// "text_segment": {
+// "start_offset": 4,
+// "end_offset": 7,
+// },
+// "page_number": 1,
+// "bounding_poly": {
+// "normalized_vertices": [
+// {"x": 0.4, "y": 0.1},
+// {"x": 0.4, "y": 0.3},
+// {"x": 0.8, "y": 0.3},
+// {"x": 0.8, "y": 0.1},
+// ],
+// },
+// "text_segment_type": TOKEN,
+// }
+//
+// ],
+// "document_dimensions": {
+// "width": 8.27,
+// "height": 11.69,
+// "unit": INCH,
+// }
+// "page_count": 1,
+// },
+// "annotations": [
+// {
+// "display_name": "animal",
+// "text_extraction": {"text_segment": {"start_offset": 0,
+// "end_offset": 3}}
+// },
+// {
+// "display_name": "animal",
+// "text_extraction": {"text_segment": {"start_offset": 4,
+// "end_offset": 7}}
+// }
+// ],
+// }\n
+// {
+// "text_snippet": {
+// "content": "This dog is good."
+// },
+// "annotations": [
+// {
+// "display_name": "animal",
+// "text_extraction": {
+// "text_segment": {"start_offset": 5, "end_offset": 8}
+// }
+// }
+// ]
+// }
+// Sample document JSON Lines file (presented here with artificial line
+// breaks, but the only actual line break is denoted by \n).:
+// {
+// "document": {
+// "input_config": {
+// "gcs_source": { "input_uris": [ "gs://folder/document1.pdf" ]
+// }
+// }
+// }
+// }\n
+// {
+// "document": {
+// "input_config": {
+// "gcs_source": { "input_uris": [ "gs://folder/document2.pdf" ]
+// }
+// }
+// }
+// }
+//
+// * For Text Classification:
+// CSV file(s) with each line in format:
+// ML_USE,(TEXT_SNIPPET | GCS_FILE_PATH),LABEL,LABEL,...
+// TEXT_SNIPPET and GCS_FILE_PATH are distinguished by a pattern. If
+// the column content is a valid gcs file path, i.e. prefixed by
+// "gs://", it will be treated as a GCS_FILE_PATH, else if the content
+// is enclosed within double quotes (""), it is
+// treated as a TEXT_SNIPPET. In the GCS_FILE_PATH case, the path
+// must lead to a .txt file with UTF-8 encoding, for example,
+// "gs://folder/content.txt", and the content in it is extracted
+// as a text snippet. In TEXT_SNIPPET case, the column content
+// excluding quotes is treated as to be imported text snippet. In
+// both cases, the text snippet/file size must be within 128kB.
+// Maximum 100 unique labels are allowed per CSV row.
+// Sample rows:
+// TRAIN,"They have bad food and very rude",RudeService,BadFood
+// TRAIN,gs://folder/content.txt,SlowService
+// TEST,"Typically always bad service there.",RudeService
+// VALIDATE,"Stomach ache to go.",BadFood
+//
+// * For Text Sentiment:
+// CSV file(s) with each line in format:
+// ML_USE,(TEXT_SNIPPET | GCS_FILE_PATH),SENTIMENT
+// TEXT_SNIPPET and GCS_FILE_PATH are distinguished by a pattern. If
+// the column content is a valid gcs file path, that is, prefixed by
+// "gs://", it is treated as a GCS_FILE_PATH, otherwise it is treated
+// as a TEXT_SNIPPET. In the GCS_FILE_PATH case, the path
+// must lead to a .txt file with UTF-8 encoding, for example,
+// "gs://folder/content.txt", and the content in it is extracted
+// as a text snippet. In TEXT_SNIPPET case, the column content itself
+// is treated as to be imported text snippet. In both cases, the
+// text snippet must be up to 500 characters long.
+// Sample rows:
+// TRAIN,"@freewrytin this is way too good for your product",2
+// TRAIN,"I need this product so bad",3
+// TEST,"Thank you for this product.",4
+// VALIDATE,gs://folder/content.txt,2
+//
+// * For Tables:
+// Either
+// [gcs_source][google.cloud.automl.v1beta1.InputConfig.gcs_source] or
+//
+// [bigquery_source][google.cloud.automl.v1beta1.InputConfig.bigquery_source]
+// can be used. All inputs is concatenated into a single
+//
+// [primary_table][google.cloud.automl.v1beta1.TablesDatasetMetadata.primary_table_name]
+// For gcs_source:
+// CSV file(s), where the first row of the first file is the header,
+// containing unique column names. If the first row of a subsequent
+// file is the same as the header, then it is also treated as a
+// header. All other rows contain values for the corresponding
+// columns.
+// Each .CSV file by itself must be 10GB or smaller, and their total
+// size must be 100GB or smaller.
+// First three sample rows of a CSV file:
+// "Id","First Name","Last Name","Dob","Addresses"
+//
+// "1","John","Doe","1968-01-22","[{"status":"current","address":"123_First_Avenue","city":"Seattle","state":"WA","zip":"11111","numberOfYears":"1"},{"status":"previous","address":"456_Main_Street","city":"Portland","state":"OR","zip":"22222","numberOfYears":"5"}]"
+//
+// "2","Jane","Doe","1980-10-16","[{"status":"current","address":"789_Any_Avenue","city":"Albany","state":"NY","zip":"33333","numberOfYears":"2"},{"status":"previous","address":"321_Main_Street","city":"Hoboken","state":"NJ","zip":"44444","numberOfYears":"3"}]}
+// For bigquery_source:
+// An URI of a BigQuery table. The user data size of the BigQuery
+// table must be 100GB or smaller.
+// An imported table must have between 2 and 1,000 columns, inclusive,
+// and between 1000 and 100,000,000 rows, inclusive. There are at most 5
+// import data running in parallel.
+// Definitions:
+// ML_USE = "TRAIN" | "VALIDATE" | "TEST" | "UNASSIGNED"
+// Describes how the given example (file) should be used for model
+// training. "UNASSIGNED" can be used when user has no preference.
+// GCS_FILE_PATH = A path to file on GCS, e.g. "gs://folder/image1.png".
+// LABEL = A display name of an object on an image, video etc., e.g. "dog".
+// Must be up to 32 characters long and can consist only of ASCII
+// Latin letters A-Z and a-z, underscores(_), and ASCII digits 0-9.
+// For each label an AnnotationSpec is created which display_name
+// becomes the label; AnnotationSpecs are given back in predictions.
+// INSTANCE_ID = A positive integer that identifies a specific instance of a
+// labeled entity on an example. Used e.g. to track two cars on
+// a video while being able to tell apart which one is which.
+// BOUNDING_BOX = VERTEX,VERTEX,VERTEX,VERTEX | VERTEX,,,VERTEX,,
+// A rectangle parallel to the frame of the example (image,
+// video). If 4 vertices are given they are connected by edges
+// in the order provided, if 2 are given they are recognized
+// as diagonally opposite vertices of the rectangle.
+// VERTEX = COORDINATE,COORDINATE
+// First coordinate is horizontal (x), the second is vertical (y).
+// COORDINATE = A float in 0 to 1 range, relative to total length of
+// image or video in given dimension. For fractions the
+// leading non-decimal 0 can be omitted (i.e. 0.3 = .3).
+// Point 0,0 is in top left.
+// TIME_SEGMENT_START = TIME_OFFSET
+// Expresses a beginning, inclusive, of a time segment
+// within an example that has a time dimension
+// (e.g. video).
+// TIME_SEGMENT_END = TIME_OFFSET
+// Expresses an end, exclusive, of a time segment within
+// an example that has a time dimension (e.g. video).
+// TIME_OFFSET = A number of seconds as measured from the start of an
+// example (e.g. video). Fractions are allowed, up to a
+// microsecond precision. "inf" is allowed, and it means the end
+// of the example.
+// TEXT_SNIPPET = A content of a text snippet, UTF-8 encoded, enclosed within
+// double quotes ("").
+// SENTIMENT = An integer between 0 and
+// Dataset.text_sentiment_dataset_metadata.sentiment_max
+// (inclusive). Describes the ordinal of the sentiment - higher
+// value means a more positive sentiment. All the values are
+// completely relative, i.e. neither 0 needs to mean a negative or
+// neutral sentiment nor sentiment_max needs to mean a positive one
+// - it is just required that 0 is the least positive sentiment
+// in the data, and sentiment_max is the most positive one.
+// The SENTIMENT shouldn't be confused with "score" or "magnitude"
+// from the previous Natural Language Sentiment Analysis API.
+// All SENTIMENT values between 0 and sentiment_max must be
+// represented in the imported data. On prediction the same 0 to
+// sentiment_max range will be used. The difference between
+// neighboring sentiment values needs not to be uniform, e.g. 1 and
+// 2 may be similar whereas the difference between 2 and 3 may be
+// huge.
+//
+// Errors:
+// If any of the provided CSV files can't be parsed or if more than certain
+// percent of CSV rows cannot be processed then the operation fails and
+// nothing is imported. Regardless of overall success or failure the per-row
+// failures, up to a certain count cap, is listed in
+// Operation.metadata.partial_failures.
+//
+message InputConfig {
+ // The source of the input.
+ oneof source {
+ // The Google Cloud Storage location for the input content.
+ // In ImportData, the gcs_source points to a csv with structure described in
+ // the comment.
+ GcsSource gcs_source = 1;
+
+ // The BigQuery location for the input content.
+ BigQuerySource bigquery_source = 3;
+ }
+
+ // Additional domain-specific parameters describing the semantic of the
+ // imported data, any string must be up to 25000
+ // characters long.
+ //
+ // * For Tables:
+ // `schema_inference_version` - (integer) Required. The version of the
+ // algorithm that should be used for the initial inference of the
+ // schema (columns' DataTypes) of the table the data is being imported
+ // into. Allowed values: "1".
+ map params = 2;
+}
+
+// Input configuration for BatchPredict Action.
+//
+// The format of input depends on the ML problem of the model used for
+// prediction. As input source the
+// [gcs_source][google.cloud.automl.v1beta1.InputConfig.gcs_source]
+// is expected, unless specified otherwise.
+//
+// The formats are represented in EBNF with commas being literal and with
+// non-terminal symbols defined near the end of this comment. The formats
+// are:
+//
+// * For Image Classification:
+// CSV file(s) with each line having just a single column:
+// GCS_FILE_PATH
+// which leads to image of up to 30MB in size. Supported
+// extensions: .JPEG, .GIF, .PNG. This path is treated as the ID in
+// the Batch predict output.
+// Three sample rows:
+// gs://folder/image1.jpeg
+// gs://folder/image2.gif
+// gs://folder/image3.png
+//
+// * For Image Object Detection:
+// CSV file(s) with each line having just a single column:
+// GCS_FILE_PATH
+// which leads to image of up to 30MB in size. Supported
+// extensions: .JPEG, .GIF, .PNG. This path is treated as the ID in
+// the Batch predict output.
+// Three sample rows:
+// gs://folder/image1.jpeg
+// gs://folder/image2.gif
+// gs://folder/image3.png
+// * For Video Classification:
+// CSV file(s) with each line in format:
+// GCS_FILE_PATH,TIME_SEGMENT_START,TIME_SEGMENT_END
+// GCS_FILE_PATH leads to video of up to 50GB in size and up to 3h
+// duration. Supported extensions: .MOV, .MPEG4, .MP4, .AVI.
+// TIME_SEGMENT_START and TIME_SEGMENT_END must be within the
+// length of the video, and end has to be after the start.
+// Three sample rows:
+// gs://folder/video1.mp4,10,40
+// gs://folder/video1.mp4,20,60
+// gs://folder/vid2.mov,0,inf
+//
+// * For Video Object Tracking:
+// CSV file(s) with each line in format:
+// GCS_FILE_PATH,TIME_SEGMENT_START,TIME_SEGMENT_END
+// GCS_FILE_PATH leads to video of up to 50GB in size and up to 3h
+// duration. Supported extensions: .MOV, .MPEG4, .MP4, .AVI.
+// TIME_SEGMENT_START and TIME_SEGMENT_END must be within the
+// length of the video, and end has to be after the start.
+// Three sample rows:
+// gs://folder/video1.mp4,10,240
+// gs://folder/video1.mp4,300,360
+// gs://folder/vid2.mov,0,inf
+// * For Text Classification:
+// CSV file(s) with each line having just a single column:
+// GCS_FILE_PATH | TEXT_SNIPPET
+// Any given text file can have size upto 128kB.
+// Any given text snippet content must have 60,000 characters or less.
+// Three sample rows:
+// gs://folder/text1.txt
+// "Some text content to predict"
+// gs://folder/text3.pdf
+// Supported file extensions: .txt, .pdf
+//
+// * For Text Sentiment:
+// CSV file(s) with each line having just a single column:
+// GCS_FILE_PATH | TEXT_SNIPPET
+// Any given text file can have size upto 128kB.
+// Any given text snippet content must have 500 characters or less.
+// Three sample rows:
+// gs://folder/text1.txt
+// "Some text content to predict"
+// gs://folder/text3.pdf
+// Supported file extensions: .txt, .pdf
+//
+// * For Text Extraction
+// .JSONL (i.e. JSON Lines) file(s) which either provide text in-line or
+// as documents (for a single BatchPredict call only one of the these
+// formats may be used).
+// The in-line .JSONL file(s) contain per line a proto that
+// wraps a temporary user-assigned TextSnippet ID (string up to 2000
+// characters long) called "id", a TextSnippet proto (in
+// json representation) and zero or more TextFeature protos. Any given
+// text snippet content must have 30,000 characters or less, and also
+// be UTF-8 NFC encoded (ASCII already is). The IDs provided should be
+// unique.
+// The document .JSONL file(s) contain, per line, a proto that wraps a
+// Document proto with input_config set. Only PDF documents are
+// supported now, and each document must be up to 2MB large.
+// Any given .JSONL file must be 100MB or smaller, and no more than 20
+// files may be given.
+// Sample in-line JSON Lines file (presented here with artificial line
+// breaks, but the only actual line break is denoted by \n):
+// {
+// "id": "my_first_id",
+// "text_snippet": { "content": "dog car cat"},
+// "text_features": [
+// {
+// "text_segment": {"start_offset": 4, "end_offset": 6},
+// "structural_type": PARAGRAPH,
+// "bounding_poly": {
+// "normalized_vertices": [
+// {"x": 0.1, "y": 0.1},
+// {"x": 0.1, "y": 0.3},
+// {"x": 0.3, "y": 0.3},
+// {"x": 0.3, "y": 0.1},
+// ]
+// },
+// }
+// ],
+// }\n
+// {
+// "id": "2",
+// "text_snippet": {
+// "content": "An elaborate content",
+// "mime_type": "text/plain"
+// }
+// }
+// Sample document JSON Lines file (presented here with artificial line
+// breaks, but the only actual line break is denoted by \n).:
+// {
+// "document": {
+// "input_config": {
+// "gcs_source": { "input_uris": [ "gs://folder/document1.pdf" ]
+// }
+// }
+// }
+// }\n
+// {
+// "document": {
+// "input_config": {
+// "gcs_source": { "input_uris": [ "gs://folder/document2.pdf" ]
+// }
+// }
+// }
+// }
+//
+// * For Tables:
+// Either
+// [gcs_source][google.cloud.automl.v1beta1.InputConfig.gcs_source] or
+//
+// [bigquery_source][google.cloud.automl.v1beta1.InputConfig.bigquery_source].
+// GCS case:
+// CSV file(s), each by itself 10GB or smaller and total size must be
+// 100GB or smaller, where first file must have a header containing
+// column names. If the first row of a subsequent file is the same as
+// the header, then it is also treated as a header. All other rows
+// contain values for the corresponding columns.
+// The column names must contain the model's
+//
+// [input_feature_column_specs'][google.cloud.automl.v1beta1.TablesModelMetadata.input_feature_column_specs]
+//
+// [display_name-s][google.cloud.automl.v1beta1.ColumnSpec.display_name]
+// (order doesn't matter). The columns corresponding to the model's
+// input feature column specs must contain values compatible with the
+// column spec's data types. Prediction on all the rows, i.e. the CSV
+// lines, will be attempted. For FORECASTING
+//
+// [prediction_type][google.cloud.automl.v1beta1.TablesModelMetadata.prediction_type]:
+// all columns having
+//
+// [TIME_SERIES_AVAILABLE_PAST_ONLY][google.cloud.automl.v1beta1.ColumnSpec.ForecastingMetadata.ColumnType]
+// type will be ignored.
+// First three sample rows of a CSV file:
+// "First Name","Last Name","Dob","Addresses"
+//
+// "John","Doe","1968-01-22","[{"status":"current","address":"123_First_Avenue","city":"Seattle","state":"WA","zip":"11111","numberOfYears":"1"},{"status":"previous","address":"456_Main_Street","city":"Portland","state":"OR","zip":"22222","numberOfYears":"5"}]"
+//
+// "Jane","Doe","1980-10-16","[{"status":"current","address":"789_Any_Avenue","city":"Albany","state":"NY","zip":"33333","numberOfYears":"2"},{"status":"previous","address":"321_Main_Street","city":"Hoboken","state":"NJ","zip":"44444","numberOfYears":"3"}]}
+// BigQuery case:
+// An URI of a BigQuery table. The user data size of the BigQuery
+// table must be 100GB or smaller.
+// The column names must contain the model's
+//
+// [input_feature_column_specs'][google.cloud.automl.v1beta1.TablesModelMetadata.input_feature_column_specs]
+//
+// [display_name-s][google.cloud.automl.v1beta1.ColumnSpec.display_name]
+// (order doesn't matter). The columns corresponding to the model's
+// input feature column specs must contain values compatible with the
+// column spec's data types. Prediction on all the rows of the table
+// will be attempted. For FORECASTING
+//
+// [prediction_type][google.cloud.automl.v1beta1.TablesModelMetadata.prediction_type]:
+// all columns having
+//
+// [TIME_SERIES_AVAILABLE_PAST_ONLY][google.cloud.automl.v1beta1.ColumnSpec.ForecastingMetadata.ColumnType]
+// type will be ignored.
+//
+// Definitions:
+// GCS_FILE_PATH = A path to file on GCS, e.g. "gs://folder/video.avi".
+// TEXT_SNIPPET = A content of a text snippet, UTF-8 encoded, enclosed within
+// double quotes ("")
+// TIME_SEGMENT_START = TIME_OFFSET
+// Expresses a beginning, inclusive, of a time segment
+// within an
+// example that has a time dimension (e.g. video).
+// TIME_SEGMENT_END = TIME_OFFSET
+// Expresses an end, exclusive, of a time segment within
+// an example that has a time dimension (e.g. video).
+// TIME_OFFSET = A number of seconds as measured from the start of an
+// example (e.g. video). Fractions are allowed, up to a
+// microsecond precision. "inf" is allowed and it means the end
+// of the example.
+//
+// Errors:
+// If any of the provided CSV files can't be parsed or if more than certain
+// percent of CSV rows cannot be processed then the operation fails and
+// prediction does not happen. Regardless of overall success or failure the
+// per-row failures, up to a certain count cap, will be listed in
+// Operation.metadata.partial_failures.
+message BatchPredictInputConfig {
+ // Required. The source of the input.
+ oneof source {
+ // The Google Cloud Storage location for the input content.
+ GcsSource gcs_source = 1;
+
+ // The BigQuery location for the input content.
+ BigQuerySource bigquery_source = 2;
+ }
+}
+
+// Input configuration of a [Document][google.cloud.automl.v1beta1.Document].
+message DocumentInputConfig {
+ // The Google Cloud Storage location of the document file. Only a single path
+ // should be given.
+ // Max supported size: 512MB.
+ // Supported extensions: .PDF.
+ GcsSource gcs_source = 1;
+}
+
+// * For Translation:
+// CSV file `translation.csv`, with each line in format:
+// ML_USE,GCS_FILE_PATH
+// GCS_FILE_PATH leads to a .TSV file which describes examples that have
+// given ML_USE, using the following row format per line:
+// TEXT_SNIPPET (in source language) \t TEXT_SNIPPET (in target
+// language)
+//
+// * For Tables:
+// Output depends on whether the dataset was imported from GCS or
+// BigQuery.
+// GCS case:
+//
+// [gcs_destination][google.cloud.automl.v1beta1.OutputConfig.gcs_destination]
+// must be set. Exported are CSV file(s) `tables_1.csv`,
+// `tables_2.csv`,...,`tables_N.csv` with each having as header line
+// the table's column names, and all other lines contain values for
+// the header columns.
+// BigQuery case:
+//
+// [bigquery_destination][google.cloud.automl.v1beta1.OutputConfig.bigquery_destination]
+// pointing to a BigQuery project must be set. In the given project a
+// new dataset will be created with name
+//
+// `export_data__`
+// where will be made
+// BigQuery-dataset-name compatible (e.g. most special characters will
+// become underscores), and timestamp will be in
+// YYYY_MM_DDThh_mm_ss_sssZ "based on ISO-8601" format. In that
+// dataset a new table called `primary_table` will be created, and
+// filled with precisely the same data as this obtained on import.
+message OutputConfig {
+ // Required. The destination of the output.
+ oneof destination {
+ // The Google Cloud Storage location where the output is to be written to.
+ // For Image Object Detection, Text Extraction, Video Classification and
+ // Tables, in the given directory a new directory will be created with name:
+ // export_data-- where
+ // timestamp is in YYYY-MM-DDThh:mm:ss.sssZ ISO-8601 format. All export
+ // output will be written into that directory.
+ GcsDestination gcs_destination = 1;
+
+ // The BigQuery location where the output is to be written to.
+ BigQueryDestination bigquery_destination = 2;
+ }
+}
+
+// Output configuration for BatchPredict Action.
+//
+// As destination the
+//
+// [gcs_destination][google.cloud.automl.v1beta1.BatchPredictOutputConfig.gcs_destination]
+// must be set unless specified otherwise for a domain. If gcs_destination is
+// set then in the given directory a new directory is created. Its name
+// will be
+// "prediction--",
+// where timestamp is in YYYY-MM-DDThh:mm:ss.sssZ ISO-8601 format. The contents
+// of it depends on the ML problem the predictions are made for.
+//
+// * For Image Classification:
+// In the created directory files `image_classification_1.jsonl`,
+// `image_classification_2.jsonl`,...,`image_classification_N.jsonl`
+// will be created, where N may be 1, and depends on the
+// total number of the successfully predicted images and annotations.
+// A single image will be listed only once with all its annotations,
+// and its annotations will never be split across files.
+// Each .JSONL file will contain, per line, a JSON representation of a
+// proto that wraps image's "ID" : "" followed by a list of
+// zero or more AnnotationPayload protos (called annotations), which
+// have classification detail populated.
+// If prediction for any image failed (partially or completely), then an
+// additional `errors_1.jsonl`, `errors_2.jsonl`,..., `errors_N.jsonl`
+// files will be created (N depends on total number of failed
+// predictions). These files will have a JSON representation of a proto
+// that wraps the same "ID" : "" but here followed by
+// exactly one
+//
+// [`google.rpc.Status`](https:
+// //github.com/googleapis/googleapis/blob/master/google/rpc/status.proto)
+// containing only `code` and `message`fields.
+//
+// * For Image Object Detection:
+// In the created directory files `image_object_detection_1.jsonl`,
+// `image_object_detection_2.jsonl`,...,`image_object_detection_N.jsonl`
+// will be created, where N may be 1, and depends on the
+// total number of the successfully predicted images and annotations.
+// Each .JSONL file will contain, per line, a JSON representation of a
+// proto that wraps image's "ID" : "" followed by a list of
+// zero or more AnnotationPayload protos (called annotations), which
+// have image_object_detection detail populated. A single image will
+// be listed only once with all its annotations, and its annotations
+// will never be split across files.
+// If prediction for any image failed (partially or completely), then
+// additional `errors_1.jsonl`, `errors_2.jsonl`,..., `errors_N.jsonl`
+// files will be created (N depends on total number of failed
+// predictions). These files will have a JSON representation of a proto
+// that wraps the same "ID" : "" but here followed by
+// exactly one
+//
+// [`google.rpc.Status`](https:
+// //github.com/googleapis/googleapis/blob/master/google/rpc/status.proto)
+// containing only `code` and `message`fields.
+// * For Video Classification:
+// In the created directory a video_classification.csv file, and a .JSON
+// file per each video classification requested in the input (i.e. each
+// line in given CSV(s)), will be created.
+//
+// The format of video_classification.csv is:
+//
+// GCS_FILE_PATH,TIME_SEGMENT_START,TIME_SEGMENT_END,JSON_FILE_NAME,STATUS
+// where:
+// GCS_FILE_PATH,TIME_SEGMENT_START,TIME_SEGMENT_END = matches 1 to 1
+// the prediction input lines (i.e. video_classification.csv has
+// precisely the same number of lines as the prediction input had.)
+// JSON_FILE_NAME = Name of .JSON file in the output directory, which
+// contains prediction responses for the video time segment.
+// STATUS = "OK" if prediction completed successfully, or an error code
+// with message otherwise. If STATUS is not "OK" then the .JSON file
+// for that line may not exist or be empty.
+//
+// Each .JSON file, assuming STATUS is "OK", will contain a list of
+// AnnotationPayload protos in JSON format, which are the predictions
+// for the video time segment the file is assigned to in the
+// video_classification.csv. All AnnotationPayload protos will have
+// video_classification field set, and will be sorted by
+// video_classification.type field (note that the returned types are
+// governed by `classifaction_types` parameter in
+// [PredictService.BatchPredictRequest.params][]).
+//
+// * For Video Object Tracking:
+// In the created directory a video_object_tracking.csv file will be
+// created, and multiple files video_object_trackinng_1.json,
+// video_object_trackinng_2.json,..., video_object_trackinng_N.json,
+// where N is the number of requests in the input (i.e. the number of
+// lines in given CSV(s)).
+//
+// The format of video_object_tracking.csv is:
+//
+// GCS_FILE_PATH,TIME_SEGMENT_START,TIME_SEGMENT_END,JSON_FILE_NAME,STATUS
+// where:
+// GCS_FILE_PATH,TIME_SEGMENT_START,TIME_SEGMENT_END = matches 1 to 1
+// the prediction input lines (i.e. video_object_tracking.csv has
+// precisely the same number of lines as the prediction input had.)
+// JSON_FILE_NAME = Name of .JSON file in the output directory, which
+// contains prediction responses for the video time segment.
+// STATUS = "OK" if prediction completed successfully, or an error
+// code with message otherwise. If STATUS is not "OK" then the .JSON
+// file for that line may not exist or be empty.
+//
+// Each .JSON file, assuming STATUS is "OK", will contain a list of
+// AnnotationPayload protos in JSON format, which are the predictions
+// for each frame of the video time segment the file is assigned to in
+// video_object_tracking.csv. All AnnotationPayload protos will have
+// video_object_tracking field set.
+// * For Text Classification:
+// In the created directory files `text_classification_1.jsonl`,
+// `text_classification_2.jsonl`,...,`text_classification_N.jsonl`
+// will be created, where N may be 1, and depends on the
+// total number of inputs and annotations found.
+//
+// Each .JSONL file will contain, per line, a JSON representation of a
+// proto that wraps input text snippet or input text file and a list of
+// zero or more AnnotationPayload protos (called annotations), which
+// have classification detail populated. A single text snippet or file
+// will be listed only once with all its annotations, and its
+// annotations will never be split across files.
+//
+// If prediction for any text snippet or file failed (partially or
+// completely), then additional `errors_1.jsonl`, `errors_2.jsonl`,...,
+// `errors_N.jsonl` files will be created (N depends on total number of
+// failed predictions). These files will have a JSON representation of a
+// proto that wraps input text snippet or input text file followed by
+// exactly one
+//
+// [`google.rpc.Status`](https:
+// //github.com/googleapis/googleapis/blob/master/google/rpc/status.proto)
+// containing only `code` and `message`.
+//
+// * For Text Sentiment:
+// In the created directory files `text_sentiment_1.jsonl`,
+// `text_sentiment_2.jsonl`,...,`text_sentiment_N.jsonl`
+// will be created, where N may be 1, and depends on the
+// total number of inputs and annotations found.
+//
+// Each .JSONL file will contain, per line, a JSON representation of a
+// proto that wraps input text snippet or input text file and a list of
+// zero or more AnnotationPayload protos (called annotations), which
+// have text_sentiment detail populated. A single text snippet or file
+// will be listed only once with all its annotations, and its
+// annotations will never be split across files.
+//
+// If prediction for any text snippet or file failed (partially or
+// completely), then additional `errors_1.jsonl`, `errors_2.jsonl`,...,
+// `errors_N.jsonl` files will be created (N depends on total number of
+// failed predictions). These files will have a JSON representation of a
+// proto that wraps input text snippet or input text file followed by
+// exactly one
+//
+// [`google.rpc.Status`](https:
+// //github.com/googleapis/googleapis/blob/master/google/rpc/status.proto)
+// containing only `code` and `message`.
+//
+// * For Text Extraction:
+// In the created directory files `text_extraction_1.jsonl`,
+// `text_extraction_2.jsonl`,...,`text_extraction_N.jsonl`
+// will be created, where N may be 1, and depends on the
+// total number of inputs and annotations found.
+// The contents of these .JSONL file(s) depend on whether the input
+// used inline text, or documents.
+// If input was inline, then each .JSONL file will contain, per line,
+// a JSON representation of a proto that wraps given in request text
+// snippet's "id" (if specified), followed by input text snippet,
+// and a list of zero or more
+// AnnotationPayload protos (called annotations), which have
+// text_extraction detail populated. A single text snippet will be
+// listed only once with all its annotations, and its annotations will
+// never be split across files.
+// If input used documents, then each .JSONL file will contain, per
+// line, a JSON representation of a proto that wraps given in request
+// document proto, followed by its OCR-ed representation in the form
+// of a text snippet, finally followed by a list of zero or more
+// AnnotationPayload protos (called annotations), which have
+// text_extraction detail populated and refer, via their indices, to
+// the OCR-ed text snippet. A single document (and its text snippet)
+// will be listed only once with all its annotations, and its
+// annotations will never be split across files.
+// If prediction for any text snippet failed (partially or completely),
+// then additional `errors_1.jsonl`, `errors_2.jsonl`,...,
+// `errors_N.jsonl` files will be created (N depends on total number of
+// failed predictions). These files will have a JSON representation of a
+// proto that wraps either the "id" : "" (in case of inline)
+// or the document proto (in case of document) but here followed by
+// exactly one
+//
+// [`google.rpc.Status`](https:
+// //github.com/googleapis/googleapis/blob/master/google/rpc/status.proto)
+// containing only `code` and `message`.
+//
+// * For Tables:
+// Output depends on whether
+//
+// [gcs_destination][google.cloud.automl.v1beta1.BatchPredictOutputConfig.gcs_destination]
+// or
+//
+// [bigquery_destination][google.cloud.automl.v1beta1.BatchPredictOutputConfig.bigquery_destination]
+// is set (either is allowed).
+// GCS case:
+// In the created directory files `tables_1.csv`, `tables_2.csv`,...,
+// `tables_N.csv` will be created, where N may be 1, and depends on
+// the total number of the successfully predicted rows.
+// For all CLASSIFICATION
+//
+// [prediction_type-s][google.cloud.automl.v1beta1.TablesModelMetadata.prediction_type]:
+// Each .csv file will contain a header, listing all columns'
+//
+// [display_name-s][google.cloud.automl.v1beta1.ColumnSpec.display_name]
+// given on input followed by M target column names in the format of
+//
+// "<[target_column_specs][google.cloud.automl.v1beta1.TablesModelMetadata.target_column_spec]
+//
+// [display_name][google.cloud.automl.v1beta1.ColumnSpec.display_name]>__score" where M is the number of distinct target values,
+// i.e. number of distinct values in the target column of the table
+// used to train the model. Subsequent lines will contain the
+// respective values of successfully predicted rows, with the last,
+// i.e. the target, columns having the corresponding prediction
+// [scores][google.cloud.automl.v1beta1.TablesAnnotation.score].
+// For REGRESSION and FORECASTING
+//
+// [prediction_type-s][google.cloud.automl.v1beta1.TablesModelMetadata.prediction_type]:
+// Each .csv file will contain a header, listing all columns'
+// [display_name-s][google.cloud.automl.v1beta1.display_name] given
+// on input followed by the predicted target column with name in the
+// format of
+//
+// "predicted_<[target_column_specs][google.cloud.automl.v1beta1.TablesModelMetadata.target_column_spec]
+//
+// [display_name][google.cloud.automl.v1beta1.ColumnSpec.display_name]>"
+// Subsequent lines will contain the respective values of
+// successfully predicted rows, with the last, i.e. the target,
+// column having the predicted target value.
+// If prediction for any rows failed, then an additional
+// `errors_1.csv`, `errors_2.csv`,..., `errors_N.csv` will be
+// created (N depends on total number of failed rows). These files
+// will have analogous format as `tables_*.csv`, but always with a
+// single target column having
+//
+// [`google.rpc.Status`](https:
+// //github.com/googleapis/googleapis/blob/master/google/rpc/status.proto)
+// represented as a JSON string, and containing only `code` and
+// `message`.
+// BigQuery case:
+//
+// [bigquery_destination][google.cloud.automl.v1beta1.OutputConfig.bigquery_destination]
+// pointing to a BigQuery project must be set. In the given project a
+// new dataset will be created with name
+// `prediction__`
+// where will be made
+// BigQuery-dataset-name compatible (e.g. most special characters will
+// become underscores), and timestamp will be in
+// YYYY_MM_DDThh_mm_ss_sssZ "based on ISO-8601" format. In the dataset
+// two tables will be created, `predictions`, and `errors`.
+// The `predictions` table's column names will be the input columns'
+//
+// [display_name-s][google.cloud.automl.v1beta1.ColumnSpec.display_name]
+// followed by the target column with name in the format of
+//
+// "predicted_<[target_column_specs][google.cloud.automl.v1beta1.TablesModelMetadata.target_column_spec]
+//
+// [display_name][google.cloud.automl.v1beta1.ColumnSpec.display_name]>"
+// The input feature columns will contain the respective values of
+// successfully predicted rows, with the target column having an
+// ARRAY of
+//
+// [AnnotationPayloads][google.cloud.automl.v1beta1.AnnotationPayload],
+// represented as STRUCT-s, containing
+// [TablesAnnotation][google.cloud.automl.v1beta1.TablesAnnotation].
+// The `errors` table contains rows for which the prediction has
+// failed, it has analogous input columns while the target column name
+// is in the format of
+//
+// "errors_<[target_column_specs][google.cloud.automl.v1beta1.TablesModelMetadata.target_column_spec]
+//
+// [display_name][google.cloud.automl.v1beta1.ColumnSpec.display_name]>",
+// and as a value has
+//
+// [`google.rpc.Status`](https:
+// //github.com/googleapis/googleapis/blob/master/google/rpc/status.proto)
+// represented as a STRUCT, and containing only `code` and `message`.
+message BatchPredictOutputConfig {
+ // Required. The destination of the output.
+ oneof destination {
+ // The Google Cloud Storage location of the directory where the output is to
+ // be written to.
+ GcsDestination gcs_destination = 1;
+
+ // The BigQuery location where the output is to be written to.
+ BigQueryDestination bigquery_destination = 2;
+ }
+}
+
+// Output configuration for ModelExport Action.
+message ModelExportOutputConfig {
+ // Required. The destination of the output.
+ oneof destination {
+ // The Google Cloud Storage location where the model is to be written to.
+ // This location may only be set for the following model formats:
+ // "tflite", "edgetpu_tflite", "core_ml", "docker".
+ //
+ // Under the directory given as the destination a new one with name
+ // "model-export--",
+ // where timestamp is in YYYY-MM-DDThh:mm:ss.sssZ ISO-8601 format,
+ // will be created. Inside the model and any of its supporting files
+ // will be written.
+ GcsDestination gcs_destination = 1;
+
+ // The GCR location where model image is to be pushed to. This location
+ // may only be set for the following model formats:
+ // "docker".
+ //
+ // The model image will be created under the given URI.
+ GcrDestination gcr_destination = 3;
+ }
+
+ // The format in which the model must be exported. The available, and default,
+ // formats depend on the problem and model type (if given problem and type
+ // combination doesn't have a format listed, it means its models are not
+ // exportable):
+ //
+ // * For Image Classification mobile-low-latency-1, mobile-versatile-1,
+ // mobile-high-accuracy-1:
+ // "tflite" (default), "edgetpu_tflite", "tf_saved_model", "docker".
+ //
+ // * For Image Classification mobile-core-ml-low-latency-1,
+ // mobile-core-ml-versatile-1, mobile-core-ml-high-accuracy-1:
+ // "core_ml" (default).
+ // Formats description:
+ //
+ // * tflite - Used for Android mobile devices.
+ // * edgetpu_tflite - Used for [Edge TPU](https://cloud.google.com/edge-tpu/)
+ // devices.
+ // * tf_saved_model - A tensorflow model in SavedModel format.
+ // * docker - Used for Docker containers. Use the params field to customize
+ // the container. The container is verified to work correctly on
+ // ubuntu 16.04 operating system. See more at
+ // [containers
+ //
+ // quickstart](https:
+ // //cloud.google.com/vision/automl/docs/containers-gcs-quickstart)
+ // * core_ml - Used for iOS mobile devices.
+ string model_format = 4;
+
+ // Additional model-type and format specific parameters describing the
+ // requirements for the to be exported model files, any string must be up to
+ // 25000 characters long.
+ //
+ // * For `docker` format:
+ // `cpu_architecture` - (string) "x86_64" (default).
+ // `gpu_architecture` - (string) "none" (default), "nvidia".
+ map params = 2;
+}
+
+// Output configuration for ExportEvaluatedExamples Action. Note that this call
+// is available only for 30 days since the moment the model was evaluated.
+// The output depends on the domain, as follows (note that only examples from
+// the TEST set are exported):
+//
+// * For Tables:
+//
+// [bigquery_destination][google.cloud.automl.v1beta1.OutputConfig.bigquery_destination]
+// pointing to a BigQuery project must be set. In the given project a
+// new dataset will be created with name
+//
+// `export_evaluated_examples__`
+// where will be made BigQuery-dataset-name
+// compatible (e.g. most special characters will become underscores),
+// and timestamp will be in YYYY_MM_DDThh_mm_ss_sssZ "based on ISO-8601"
+// format. In the dataset an `evaluated_examples` table will be
+// created. It will have all the same columns as the
+//
+// [primary_table][google.cloud.automl.v1beta1.TablesDatasetMetadata.primary_table_spec_id]
+// of the
+// [dataset][google.cloud.automl.v1beta1.Model.dataset_id] from which
+// the model was created, as they were at the moment of model's
+// evaluation (this includes the target column with its ground
+// truth), followed by a column called "predicted_". That
+// last column will contain the model's prediction result for each
+// respective row, given as ARRAY of
+// [AnnotationPayloads][google.cloud.automl.v1beta1.AnnotationPayload],
+// represented as STRUCT-s, containing
+// [TablesAnnotation][google.cloud.automl.v1beta1.TablesAnnotation].
+message ExportEvaluatedExamplesOutputConfig {
+ // Required. The destination of the output.
+ oneof destination {
+ // The BigQuery location where the output is to be written to.
+ BigQueryDestination bigquery_destination = 2;
+ }
+}
+
+// The Google Cloud Storage location for the input content.
+message GcsSource {
+ // Required. Google Cloud Storage URIs to input files, up to 2000 characters
+ // long. Accepted forms:
+ // * Full object path, e.g. gs://bucket/directory/object.csv
+ repeated string input_uris = 1;
+}
+
+// The BigQuery location for the input content.
+message BigQuerySource {
+ // Required. BigQuery URI to a table, up to 2000 characters long.
+ // Accepted forms:
+ // * BigQuery path e.g. bq://projectId.bqDatasetId.bqTableId
+ string input_uri = 1;
+}
+
+// The Google Cloud Storage location where the output is to be written to.
+message GcsDestination {
+ // Required. Google Cloud Storage URI to output directory, up to 2000
+ // characters long.
+ // Accepted forms:
+ // * Prefix path: gs://bucket/directory
+ // The requesting user must have write permission to the bucket.
+ // The directory is created if it doesn't exist.
+ string output_uri_prefix = 1;
+}
+
+// The BigQuery location for the output content.
+message BigQueryDestination {
+ // Required. BigQuery URI to a project, up to 2000 characters long.
+ // Accepted forms:
+ // * BigQuery path e.g. bq://projectId
+ string output_uri = 1;
+}
+
+// The GCR location where the image must be pushed to.
+message GcrDestination {
+ // Required. Google Contained Registry URI of the new image, up to 2000
+ // characters long. See
+ //
+ // https:
+ // //cloud.google.com/container-registry/do
+ // // cs/pushing-and-pulling#pushing_an_image_to_a_registry
+ // Accepted forms:
+ // * [HOSTNAME]/[PROJECT-ID]/[IMAGE]
+ // * [HOSTNAME]/[PROJECT-ID]/[IMAGE]:[TAG]
+ //
+ // The requesting user must have permission to push images the project.
+ string output_uri = 1;
+}
diff --git a/automl/google/cloud/automl_v1beta1/proto/io_pb2.py b/automl/google/cloud/automl_v1beta1/proto/io_pb2.py
new file mode 100644
index 000000000000..62cd25fdd121
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/io_pb2.py
@@ -0,0 +1,1918 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/automl_v1beta1/proto/io.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/automl_v1beta1/proto/io.proto",
+ package="google.cloud.automl.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\037com.google.cloud.automl.v1beta1P\001ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\312\002\033Google\\Cloud\\AutoMl\\V1beta1\352\002\036Google::Cloud::AutoML::V1beta1"
+ ),
+ serialized_pb=_b(
+ '\n*google/cloud/automl_v1beta1/proto/io.proto\x12\x1bgoogle.cloud.automl.v1beta1\x1a\x1cgoogle/api/annotations.proto"\x92\x02\n\x0bInputConfig\x12<\n\ngcs_source\x18\x01 \x01(\x0b\x32&.google.cloud.automl.v1beta1.GcsSourceH\x00\x12\x46\n\x0f\x62igquery_source\x18\x03 \x01(\x0b\x32+.google.cloud.automl.v1beta1.BigQuerySourceH\x00\x12\x44\n\x06params\x18\x02 \x03(\x0b\x32\x34.google.cloud.automl.v1beta1.InputConfig.ParamsEntry\x1a-\n\x0bParamsEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\x42\x08\n\x06source"\xa9\x01\n\x17\x42\x61tchPredictInputConfig\x12<\n\ngcs_source\x18\x01 \x01(\x0b\x32&.google.cloud.automl.v1beta1.GcsSourceH\x00\x12\x46\n\x0f\x62igquery_source\x18\x02 \x01(\x0b\x32+.google.cloud.automl.v1beta1.BigQuerySourceH\x00\x42\x08\n\x06source"Q\n\x13\x44ocumentInputConfig\x12:\n\ngcs_source\x18\x01 \x01(\x0b\x32&.google.cloud.automl.v1beta1.GcsSource"\xb7\x01\n\x0cOutputConfig\x12\x46\n\x0fgcs_destination\x18\x01 \x01(\x0b\x32+.google.cloud.automl.v1beta1.GcsDestinationH\x00\x12P\n\x14\x62igquery_destination\x18\x02 \x01(\x0b\x32\x30.google.cloud.automl.v1beta1.BigQueryDestinationH\x00\x42\r\n\x0b\x64\x65stination"\xc3\x01\n\x18\x42\x61tchPredictOutputConfig\x12\x46\n\x0fgcs_destination\x18\x01 \x01(\x0b\x32+.google.cloud.automl.v1beta1.GcsDestinationH\x00\x12P\n\x14\x62igquery_destination\x18\x02 \x01(\x0b\x32\x30.google.cloud.automl.v1beta1.BigQueryDestinationH\x00\x42\r\n\x0b\x64\x65stination"\xcf\x02\n\x17ModelExportOutputConfig\x12\x46\n\x0fgcs_destination\x18\x01 \x01(\x0b\x32+.google.cloud.automl.v1beta1.GcsDestinationH\x00\x12\x46\n\x0fgcr_destination\x18\x03 \x01(\x0b\x32+.google.cloud.automl.v1beta1.GcrDestinationH\x00\x12\x14\n\x0cmodel_format\x18\x04 \x01(\t\x12P\n\x06params\x18\x02 \x03(\x0b\x32@.google.cloud.automl.v1beta1.ModelExportOutputConfig.ParamsEntry\x1a-\n\x0bParamsEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\x42\r\n\x0b\x64\x65stination"\x86\x01\n#ExportEvaluatedExamplesOutputConfig\x12P\n\x14\x62igquery_destination\x18\x02 \x01(\x0b\x32\x30.google.cloud.automl.v1beta1.BigQueryDestinationH\x00\x42\r\n\x0b\x64\x65stination"\x1f\n\tGcsSource\x12\x12\n\ninput_uris\x18\x01 \x03(\t"#\n\x0e\x42igQuerySource\x12\x11\n\tinput_uri\x18\x01 \x01(\t"+\n\x0eGcsDestination\x12\x19\n\x11output_uri_prefix\x18\x01 \x01(\t")\n\x13\x42igQueryDestination\x12\x12\n\noutput_uri\x18\x01 \x01(\t"$\n\x0eGcrDestination\x12\x12\n\noutput_uri\x18\x01 \x01(\tB\xa5\x01\n\x1f\x63om.google.cloud.automl.v1beta1P\x01ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\xca\x02\x1bGoogle\\Cloud\\AutoMl\\V1beta1\xea\x02\x1eGoogle::Cloud::AutoML::V1beta1b\x06proto3'
+ ),
+ dependencies=[google_dot_api_dot_annotations__pb2.DESCRIPTOR],
+)
+
+
+_INPUTCONFIG_PARAMSENTRY = _descriptor.Descriptor(
+ name="ParamsEntry",
+ full_name="google.cloud.automl.v1beta1.InputConfig.ParamsEntry",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="key",
+ full_name="google.cloud.automl.v1beta1.InputConfig.ParamsEntry.key",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="value",
+ full_name="google.cloud.automl.v1beta1.InputConfig.ParamsEntry.value",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=_b("8\001"),
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=325,
+ serialized_end=370,
+)
+
+_INPUTCONFIG = _descriptor.Descriptor(
+ name="InputConfig",
+ full_name="google.cloud.automl.v1beta1.InputConfig",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="gcs_source",
+ full_name="google.cloud.automl.v1beta1.InputConfig.gcs_source",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="bigquery_source",
+ full_name="google.cloud.automl.v1beta1.InputConfig.bigquery_source",
+ index=1,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="params",
+ full_name="google.cloud.automl.v1beta1.InputConfig.params",
+ index=2,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[_INPUTCONFIG_PARAMSENTRY],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="source",
+ full_name="google.cloud.automl.v1beta1.InputConfig.source",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=106,
+ serialized_end=380,
+)
+
+
+_BATCHPREDICTINPUTCONFIG = _descriptor.Descriptor(
+ name="BatchPredictInputConfig",
+ full_name="google.cloud.automl.v1beta1.BatchPredictInputConfig",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="gcs_source",
+ full_name="google.cloud.automl.v1beta1.BatchPredictInputConfig.gcs_source",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="bigquery_source",
+ full_name="google.cloud.automl.v1beta1.BatchPredictInputConfig.bigquery_source",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="source",
+ full_name="google.cloud.automl.v1beta1.BatchPredictInputConfig.source",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=383,
+ serialized_end=552,
+)
+
+
+_DOCUMENTINPUTCONFIG = _descriptor.Descriptor(
+ name="DocumentInputConfig",
+ full_name="google.cloud.automl.v1beta1.DocumentInputConfig",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="gcs_source",
+ full_name="google.cloud.automl.v1beta1.DocumentInputConfig.gcs_source",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=554,
+ serialized_end=635,
+)
+
+
+_OUTPUTCONFIG = _descriptor.Descriptor(
+ name="OutputConfig",
+ full_name="google.cloud.automl.v1beta1.OutputConfig",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="gcs_destination",
+ full_name="google.cloud.automl.v1beta1.OutputConfig.gcs_destination",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="bigquery_destination",
+ full_name="google.cloud.automl.v1beta1.OutputConfig.bigquery_destination",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="destination",
+ full_name="google.cloud.automl.v1beta1.OutputConfig.destination",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=638,
+ serialized_end=821,
+)
+
+
+_BATCHPREDICTOUTPUTCONFIG = _descriptor.Descriptor(
+ name="BatchPredictOutputConfig",
+ full_name="google.cloud.automl.v1beta1.BatchPredictOutputConfig",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="gcs_destination",
+ full_name="google.cloud.automl.v1beta1.BatchPredictOutputConfig.gcs_destination",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="bigquery_destination",
+ full_name="google.cloud.automl.v1beta1.BatchPredictOutputConfig.bigquery_destination",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="destination",
+ full_name="google.cloud.automl.v1beta1.BatchPredictOutputConfig.destination",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=824,
+ serialized_end=1019,
+)
+
+
+_MODELEXPORTOUTPUTCONFIG_PARAMSENTRY = _descriptor.Descriptor(
+ name="ParamsEntry",
+ full_name="google.cloud.automl.v1beta1.ModelExportOutputConfig.ParamsEntry",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="key",
+ full_name="google.cloud.automl.v1beta1.ModelExportOutputConfig.ParamsEntry.key",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="value",
+ full_name="google.cloud.automl.v1beta1.ModelExportOutputConfig.ParamsEntry.value",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=_b("8\001"),
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=325,
+ serialized_end=370,
+)
+
+_MODELEXPORTOUTPUTCONFIG = _descriptor.Descriptor(
+ name="ModelExportOutputConfig",
+ full_name="google.cloud.automl.v1beta1.ModelExportOutputConfig",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="gcs_destination",
+ full_name="google.cloud.automl.v1beta1.ModelExportOutputConfig.gcs_destination",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="gcr_destination",
+ full_name="google.cloud.automl.v1beta1.ModelExportOutputConfig.gcr_destination",
+ index=1,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="model_format",
+ full_name="google.cloud.automl.v1beta1.ModelExportOutputConfig.model_format",
+ index=2,
+ number=4,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="params",
+ full_name="google.cloud.automl.v1beta1.ModelExportOutputConfig.params",
+ index=3,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[_MODELEXPORTOUTPUTCONFIG_PARAMSENTRY],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="destination",
+ full_name="google.cloud.automl.v1beta1.ModelExportOutputConfig.destination",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=1022,
+ serialized_end=1357,
+)
+
+
+_EXPORTEVALUATEDEXAMPLESOUTPUTCONFIG = _descriptor.Descriptor(
+ name="ExportEvaluatedExamplesOutputConfig",
+ full_name="google.cloud.automl.v1beta1.ExportEvaluatedExamplesOutputConfig",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="bigquery_destination",
+ full_name="google.cloud.automl.v1beta1.ExportEvaluatedExamplesOutputConfig.bigquery_destination",
+ index=0,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="destination",
+ full_name="google.cloud.automl.v1beta1.ExportEvaluatedExamplesOutputConfig.destination",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=1360,
+ serialized_end=1494,
+)
+
+
+_GCSSOURCE = _descriptor.Descriptor(
+ name="GcsSource",
+ full_name="google.cloud.automl.v1beta1.GcsSource",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="input_uris",
+ full_name="google.cloud.automl.v1beta1.GcsSource.input_uris",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1496,
+ serialized_end=1527,
+)
+
+
+_BIGQUERYSOURCE = _descriptor.Descriptor(
+ name="BigQuerySource",
+ full_name="google.cloud.automl.v1beta1.BigQuerySource",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="input_uri",
+ full_name="google.cloud.automl.v1beta1.BigQuerySource.input_uri",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1529,
+ serialized_end=1564,
+)
+
+
+_GCSDESTINATION = _descriptor.Descriptor(
+ name="GcsDestination",
+ full_name="google.cloud.automl.v1beta1.GcsDestination",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="output_uri_prefix",
+ full_name="google.cloud.automl.v1beta1.GcsDestination.output_uri_prefix",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1566,
+ serialized_end=1609,
+)
+
+
+_BIGQUERYDESTINATION = _descriptor.Descriptor(
+ name="BigQueryDestination",
+ full_name="google.cloud.automl.v1beta1.BigQueryDestination",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="output_uri",
+ full_name="google.cloud.automl.v1beta1.BigQueryDestination.output_uri",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1611,
+ serialized_end=1652,
+)
+
+
+_GCRDESTINATION = _descriptor.Descriptor(
+ name="GcrDestination",
+ full_name="google.cloud.automl.v1beta1.GcrDestination",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="output_uri",
+ full_name="google.cloud.automl.v1beta1.GcrDestination.output_uri",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1654,
+ serialized_end=1690,
+)
+
+_INPUTCONFIG_PARAMSENTRY.containing_type = _INPUTCONFIG
+_INPUTCONFIG.fields_by_name["gcs_source"].message_type = _GCSSOURCE
+_INPUTCONFIG.fields_by_name["bigquery_source"].message_type = _BIGQUERYSOURCE
+_INPUTCONFIG.fields_by_name["params"].message_type = _INPUTCONFIG_PARAMSENTRY
+_INPUTCONFIG.oneofs_by_name["source"].fields.append(
+ _INPUTCONFIG.fields_by_name["gcs_source"]
+)
+_INPUTCONFIG.fields_by_name[
+ "gcs_source"
+].containing_oneof = _INPUTCONFIG.oneofs_by_name["source"]
+_INPUTCONFIG.oneofs_by_name["source"].fields.append(
+ _INPUTCONFIG.fields_by_name["bigquery_source"]
+)
+_INPUTCONFIG.fields_by_name[
+ "bigquery_source"
+].containing_oneof = _INPUTCONFIG.oneofs_by_name["source"]
+_BATCHPREDICTINPUTCONFIG.fields_by_name["gcs_source"].message_type = _GCSSOURCE
+_BATCHPREDICTINPUTCONFIG.fields_by_name[
+ "bigquery_source"
+].message_type = _BIGQUERYSOURCE
+_BATCHPREDICTINPUTCONFIG.oneofs_by_name["source"].fields.append(
+ _BATCHPREDICTINPUTCONFIG.fields_by_name["gcs_source"]
+)
+_BATCHPREDICTINPUTCONFIG.fields_by_name[
+ "gcs_source"
+].containing_oneof = _BATCHPREDICTINPUTCONFIG.oneofs_by_name["source"]
+_BATCHPREDICTINPUTCONFIG.oneofs_by_name["source"].fields.append(
+ _BATCHPREDICTINPUTCONFIG.fields_by_name["bigquery_source"]
+)
+_BATCHPREDICTINPUTCONFIG.fields_by_name[
+ "bigquery_source"
+].containing_oneof = _BATCHPREDICTINPUTCONFIG.oneofs_by_name["source"]
+_DOCUMENTINPUTCONFIG.fields_by_name["gcs_source"].message_type = _GCSSOURCE
+_OUTPUTCONFIG.fields_by_name["gcs_destination"].message_type = _GCSDESTINATION
+_OUTPUTCONFIG.fields_by_name["bigquery_destination"].message_type = _BIGQUERYDESTINATION
+_OUTPUTCONFIG.oneofs_by_name["destination"].fields.append(
+ _OUTPUTCONFIG.fields_by_name["gcs_destination"]
+)
+_OUTPUTCONFIG.fields_by_name[
+ "gcs_destination"
+].containing_oneof = _OUTPUTCONFIG.oneofs_by_name["destination"]
+_OUTPUTCONFIG.oneofs_by_name["destination"].fields.append(
+ _OUTPUTCONFIG.fields_by_name["bigquery_destination"]
+)
+_OUTPUTCONFIG.fields_by_name[
+ "bigquery_destination"
+].containing_oneof = _OUTPUTCONFIG.oneofs_by_name["destination"]
+_BATCHPREDICTOUTPUTCONFIG.fields_by_name[
+ "gcs_destination"
+].message_type = _GCSDESTINATION
+_BATCHPREDICTOUTPUTCONFIG.fields_by_name[
+ "bigquery_destination"
+].message_type = _BIGQUERYDESTINATION
+_BATCHPREDICTOUTPUTCONFIG.oneofs_by_name["destination"].fields.append(
+ _BATCHPREDICTOUTPUTCONFIG.fields_by_name["gcs_destination"]
+)
+_BATCHPREDICTOUTPUTCONFIG.fields_by_name[
+ "gcs_destination"
+].containing_oneof = _BATCHPREDICTOUTPUTCONFIG.oneofs_by_name["destination"]
+_BATCHPREDICTOUTPUTCONFIG.oneofs_by_name["destination"].fields.append(
+ _BATCHPREDICTOUTPUTCONFIG.fields_by_name["bigquery_destination"]
+)
+_BATCHPREDICTOUTPUTCONFIG.fields_by_name[
+ "bigquery_destination"
+].containing_oneof = _BATCHPREDICTOUTPUTCONFIG.oneofs_by_name["destination"]
+_MODELEXPORTOUTPUTCONFIG_PARAMSENTRY.containing_type = _MODELEXPORTOUTPUTCONFIG
+_MODELEXPORTOUTPUTCONFIG.fields_by_name[
+ "gcs_destination"
+].message_type = _GCSDESTINATION
+_MODELEXPORTOUTPUTCONFIG.fields_by_name[
+ "gcr_destination"
+].message_type = _GCRDESTINATION
+_MODELEXPORTOUTPUTCONFIG.fields_by_name[
+ "params"
+].message_type = _MODELEXPORTOUTPUTCONFIG_PARAMSENTRY
+_MODELEXPORTOUTPUTCONFIG.oneofs_by_name["destination"].fields.append(
+ _MODELEXPORTOUTPUTCONFIG.fields_by_name["gcs_destination"]
+)
+_MODELEXPORTOUTPUTCONFIG.fields_by_name[
+ "gcs_destination"
+].containing_oneof = _MODELEXPORTOUTPUTCONFIG.oneofs_by_name["destination"]
+_MODELEXPORTOUTPUTCONFIG.oneofs_by_name["destination"].fields.append(
+ _MODELEXPORTOUTPUTCONFIG.fields_by_name["gcr_destination"]
+)
+_MODELEXPORTOUTPUTCONFIG.fields_by_name[
+ "gcr_destination"
+].containing_oneof = _MODELEXPORTOUTPUTCONFIG.oneofs_by_name["destination"]
+_EXPORTEVALUATEDEXAMPLESOUTPUTCONFIG.fields_by_name[
+ "bigquery_destination"
+].message_type = _BIGQUERYDESTINATION
+_EXPORTEVALUATEDEXAMPLESOUTPUTCONFIG.oneofs_by_name["destination"].fields.append(
+ _EXPORTEVALUATEDEXAMPLESOUTPUTCONFIG.fields_by_name["bigquery_destination"]
+)
+_EXPORTEVALUATEDEXAMPLESOUTPUTCONFIG.fields_by_name[
+ "bigquery_destination"
+].containing_oneof = _EXPORTEVALUATEDEXAMPLESOUTPUTCONFIG.oneofs_by_name["destination"]
+DESCRIPTOR.message_types_by_name["InputConfig"] = _INPUTCONFIG
+DESCRIPTOR.message_types_by_name["BatchPredictInputConfig"] = _BATCHPREDICTINPUTCONFIG
+DESCRIPTOR.message_types_by_name["DocumentInputConfig"] = _DOCUMENTINPUTCONFIG
+DESCRIPTOR.message_types_by_name["OutputConfig"] = _OUTPUTCONFIG
+DESCRIPTOR.message_types_by_name["BatchPredictOutputConfig"] = _BATCHPREDICTOUTPUTCONFIG
+DESCRIPTOR.message_types_by_name["ModelExportOutputConfig"] = _MODELEXPORTOUTPUTCONFIG
+DESCRIPTOR.message_types_by_name[
+ "ExportEvaluatedExamplesOutputConfig"
+] = _EXPORTEVALUATEDEXAMPLESOUTPUTCONFIG
+DESCRIPTOR.message_types_by_name["GcsSource"] = _GCSSOURCE
+DESCRIPTOR.message_types_by_name["BigQuerySource"] = _BIGQUERYSOURCE
+DESCRIPTOR.message_types_by_name["GcsDestination"] = _GCSDESTINATION
+DESCRIPTOR.message_types_by_name["BigQueryDestination"] = _BIGQUERYDESTINATION
+DESCRIPTOR.message_types_by_name["GcrDestination"] = _GCRDESTINATION
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+InputConfig = _reflection.GeneratedProtocolMessageType(
+ "InputConfig",
+ (_message.Message,),
+ dict(
+ ParamsEntry=_reflection.GeneratedProtocolMessageType(
+ "ParamsEntry",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_INPUTCONFIG_PARAMSENTRY,
+ __module__="google.cloud.automl_v1beta1.proto.io_pb2"
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.InputConfig.ParamsEntry)
+ ),
+ ),
+ DESCRIPTOR=_INPUTCONFIG,
+ __module__="google.cloud.automl_v1beta1.proto.io_pb2",
+ __doc__="""Input configuration for ImportData Action.
+
+ The format of input depends on dataset\_metadata the Dataset into which
+ the import is happening has. As input source the
+ [gcs\_source][google.cloud.automl.v1beta1.InputConfig.gcs\_source] is
+ expected, unless specified otherwise. Additionally any input .CSV file
+ by itself must be 100MB or smaller, unless specified otherwise. If an
+ "example" file (that is, image, video etc.) with identical content (even
+ if it had different GCS\_FILE\_PATH) is mentioned multiple times, then
+ its label, bounding boxes etc. are appended. The same file should be
+ always provided with the same ML\_USE and GCS\_FILE\_PATH, if it is not,
+ then these values are nondeterministically selected from the given ones.
+
+ The formats are represented in EBNF with commas being literal and with
+ non-terminal symbols defined near the end of this comment. The formats
+ are:
+
+ - For Image Classification: CSV file(s) with each line in format:
+ ML\_USE,GCS\_FILE\_PATH,LABEL,LABEL,... GCS\_FILE\_PATH leads to
+ image of up to 30MB in size. Supported extensions: .JPEG, .GIF, .PNG,
+ .WEBP, .BMP, .TIFF, .ICO For MULTICLASS classification type, at most
+ one LABEL is allowed per image. If an image has not yet been labeled,
+ then it should be mentioned just once with no LABEL. Some sample
+ rows: TRAIN,gs://folder/image1.jpg,daisy
+ TEST,gs://folder/image2.jpg,dandelion,tulip,rose
+ UNASSIGNED,gs://folder/image3.jpg,daisy
+ UNASSIGNED,gs://folder/image4.jpg
+
+ - For Image Object Detection: CSV file(s) with each line in format:
+ ML\_USE,GCS\_FILE\_PATH,(LABEL,BOUNDING\_BOX \| ,,,,,,,)
+ GCS\_FILE\_PATH leads to image of up to 30MB in size. Supported
+ extensions: .JPEG, .GIF, .PNG. Each image is assumed to be
+ exhaustively labeled. The minimum allowed BOUNDING\_BOX edge length
+ is 0.01, and no more than 500 BOUNDING\_BOX-es per image are allowed
+ (one BOUNDING\_BOX is defined per line). If an image has not yet been
+ labeled, then it should be mentioned just once with no LABEL and the
+ ",,,,,,," in place of the BOUNDING\_BOX. For images which are known
+ to not contain any bounding boxes, they should be labelled explictly
+ as "NEGATIVE\_IMAGE", followed by ",,,,,,," in place of the
+ BOUNDING\_BOX. Sample rows:
+ TRAIN,gs://folder/image1.png,car,0.1,0.1,,,0.3,0.3,,
+ TRAIN,gs://folder/image1.png,bike,.7,.6,,,.8,.9,,
+ UNASSIGNED,gs://folder/im2.png,car,0.1,0.1,0.2,0.1,0.2,0.3,0.1,0.3
+ TEST,gs://folder/im3.png,,,,,,,,,
+ TRAIN,gs://folder/im4.png,NEGATIVE\_IMAGE,,,,,,,,,
+
+ - For Video Classification: CSV file(s) with each line in format:
+ ML\_USE,GCS\_FILE\_PATH where ML\_USE VALIDATE value should not be
+ used. The GCS\_FILE\_PATH should lead to another .csv file which
+ describes examples that have given ML\_USE, using the following row
+ format:
+ GCS\_FILE\_PATH,(LABEL,TIME\_SEGMENT\_START,TIME\_SEGMENT\_END \| ,,)
+ Here GCS\_FILE\_PATH leads to a video of up to 50GB in size and up to
+ 3h duration. Supported extensions: .MOV, .MPEG4, .MP4, .AVI.
+ TIME\_SEGMENT\_START and TIME\_SEGMENT\_END must be within the length
+ of the video, and end has to be after the start. Any segment of a
+ video which has one or more labels on it, is considered a hard
+ negative for all other labels. Any segment with no labels on it is
+ considered to be unknown. If a whole video is unknown, then it shuold
+ be mentioned just once with ",," in place of LABEL,
+ TIME\_SEGMENT\_START,TIME\_SEGMENT\_END. Sample top level CSV file:
+ TRAIN,gs://folder/train\_videos.csv TEST,gs://folder/test\_videos.csv
+ UNASSIGNED,gs://folder/other\_videos.csv Sample rows of a CSV file
+ for a particular ML\_USE: gs://folder/video1.avi,car,120,180.000021
+ gs://folder/video1.avi,bike,150,180.000021
+ gs://folder/vid2.avi,car,0,60.5 gs://folder/vid3.avi,,,
+
+ - For Video Object Tracking: CSV file(s) with each line in format:
+ ML\_USE,GCS\_FILE\_PATH where ML\_USE VALIDATE value should not be
+ used. The GCS\_FILE\_PATH should lead to another .csv file which
+ describes examples that have given ML\_USE, using one of the
+ following row format:
+ GCS\_FILE\_PATH,LABEL,[INSTANCE\_ID],TIMESTAMP,BOUNDING\_BOX or
+ GCS\_FILE\_PATH,,,,,,,,,, Here GCS\_FILE\_PATH leads to a video of up
+ to 50GB in size and up to 3h duration. Supported extensions: .MOV,
+ .MPEG4, .MP4, .AVI. Providing INSTANCE\_IDs can help to obtain a
+ better model. When a specific labeled entity leaves the video frame,
+ and shows up afterwards it is not required, albeit preferable, that
+ the same INSTANCE\_ID is given to it. TIMESTAMP must be within the
+ length of the video, the BOUNDING\_BOX is assumed to be drawn on the
+ closest video's frame to the TIMESTAMP. Any mentioned by the
+ TIMESTAMP frame is expected to be exhaustively labeled and no more
+ than 500 BOUNDING\_BOX-es per frame are allowed. If a whole video is
+ unknown, then it should be mentioned just once with ",,,,,,,,,," in
+ place of LABEL, [INSTANCE\_ID],TIMESTAMP,BOUNDING\_BOX. Sample top
+ level CSV file: TRAIN,gs://folder/train\_videos.csv
+ TEST,gs://folder/test\_videos.csv
+ UNASSIGNED,gs://folder/other\_videos.csv Seven sample rows of a CSV
+ file for a particular ML\_USE:
+ gs://folder/video1.avi,car,1,12.10,0.8,0.8,0.9,0.8,0.9,0.9,0.8,0.9
+ gs://folder/video1.avi,car,1,12.90,0.4,0.8,0.5,0.8,0.5,0.9,0.4,0.9
+ gs://folder/video1.avi,car,2,12.10,.4,.2,.5,.2,.5,.3,.4,.3
+ gs://folder/video1.avi,car,2,12.90,.8,.2,,,.9,.3,,
+ gs://folder/video1.avi,bike,,12.50,.45,.45,,,.55,.55,,
+ gs://folder/video2.avi,car,1,0,.1,.9,,,.9,.1,,
+ gs://folder/video2.avi,,,,,,,,,,,
+ - For Text Extraction: CSV file(s) with each line in format:
+ ML\_USE,GCS\_FILE\_PATH GCS\_FILE\_PATH leads to a .JSONL (that is,
+ JSON Lines) file which either imports text in-line or as documents.
+ Any given .JSONL file must be 100MB or smaller. The in-line .JSONL
+ file contains, per line, a proto that wraps a TextSnippet proto (in
+ json representation) followed by one or more AnnotationPayload protos
+ (called annotations), which have display\_name and text\_extraction
+ detail populated. The given text is expected to be annotated
+ exhaustively, for example, if you look for animals and text contains
+ "dolphin" that is not labeled, then "dolphin" is assumed to not be an
+ animal. Any given text snippet content must be 10KB or smaller, and
+ also be UTF-8 NFC encoded (ASCII already is). The document .JSONL
+ file contains, per line, a proto that wraps a Document proto. The
+ Document proto must have either document\_text or input\_config set.
+ In document\_text case, the Document proto may also contain the
+ spatial information of the document, including layout, document
+ dimension and page number. In input\_config case, only PDF documents
+ are supported now, and each document may be up to 2MB large.
+ Currently, annotations on documents cannot be specified at import.
+ Three sample CSV rows: TRAIN,gs://folder/file1.jsonl
+ VALIDATE,gs://folder/file2.jsonl TEST,gs://folder/file3.jsonl
+
+ - For Text Classification: CSV file(s) with each line in format:
+ ML\_USE,(TEXT\_SNIPPET \| GCS\_FILE\_PATH),LABEL,LABEL,...
+ TEXT\_SNIPPET and GCS\_FILE\_PATH are distinguished by a pattern. If
+ the column content is a valid gcs file path, i.e. prefixed by
+ "gs://", it will be treated as a GCS\_FILE\_PATH, else if the content
+ is enclosed within double quotes (""), it is treated as a
+ TEXT\_SNIPPET. In the GCS\_FILE\_PATH case, the path must lead to a
+ .txt file with UTF-8 encoding, for example,
+ "gs://folder/content.txt", and the content in it is extracted as a
+ text snippet. In TEXT\_SNIPPET case, the column content excluding
+ quotes is treated as to be imported text snippet. In both cases, the
+ text snippet/file size must be within 128kB. Maximum 100 unique
+ labels are allowed per CSV row. Sample rows: TRAIN,"They have bad
+ food and very rude",RudeService,BadFood
+ TRAIN,gs://folder/content.txt,SlowService TEST,"Typically always bad
+ service there.",RudeService VALIDATE,"Stomach ache to go.",BadFood
+
+ - For Text Sentiment: CSV file(s) with each line in format:
+ ML\_USE,(TEXT\_SNIPPET \| GCS\_FILE\_PATH),SENTIMENT TEXT\_SNIPPET
+ and GCS\_FILE\_PATH are distinguished by a pattern. If the column
+ content is a valid gcs file path, that is, prefixed by "gs://", it is
+ treated as a GCS\_FILE\_PATH, otherwise it is treated as a
+ TEXT\_SNIPPET. In the GCS\_FILE\_PATH case, the path must lead to a
+ .txt file with UTF-8 encoding, for example,
+ "gs://folder/content.txt", and the content in it is extracted as a
+ text snippet. In TEXT\_SNIPPET case, the column content itself is
+ treated as to be imported text snippet. In both cases, the text
+ snippet must be up to 500 characters long. Sample rows:
+ TRAIN,"@freewrytin this is way too good for your product",2 TRAIN,"I
+ need this product so bad",3 TEST,"Thank you for this product.",4
+ VALIDATE,gs://folder/content.txt,2
+
+ - For Tables: Either
+ [gcs\_source][google.cloud.automl.v1beta1.InputConfig.gcs\_source] or
+
+ [bigquery\_source][google.cloud.automl.v1beta1.InputConfig.bigquery\_source]
+ can be used. All inputs is concatenated into a single
+
+ [primary\_table][google.cloud.automl.v1beta1.TablesDatasetMetadata.primary\_table\_name]
+ For gcs\_source: CSV file(s), where the first row of the first file is
+ the header, containing unique column names. If the first row of a
+ subsequent file is the same as the header, then it is also treated as a
+ header. All other rows contain values for the corresponding columns.
+ Each .CSV file by itself must be 10GB or smaller, and their total size
+ must be 100GB or smaller. First three sample rows of a CSV file:
+ "Id","First Name","Last Name","Dob","Addresses"
+
+ "1","John","Doe","1968-01-22","[{"status":"current","address":"123\_First\_Avenue","city":"Seattle","state":"WA","zip":"11111","numberOfYears":"1"},{"status":"previous","address":"456\_Main\_Street","city":"Portland","state":"OR","zip":"22222","numberOfYears":"5"}]"
+
+ "2","Jane","Doe","1980-10-16","[{"status":"current","address":"789\_Any\_Avenue","city":"Albany","state":"NY","zip":"33333","numberOfYears":"2"},{"status":"previous","address":"321\_Main\_Street","city":"Hoboken","state":"NJ","zip":"44444","numberOfYears":"3"}]}
+ For bigquery\_source: An URI of a BigQuery table. The user data size of
+ the BigQuery table must be 100GB or smaller. An imported table must have
+ between 2 and 1,000 columns, inclusive, and between 1000 and 100,000,000
+ rows, inclusive. There are at most 5 import data running in parallel.
+ Definitions: ML\_USE = "TRAIN" \| "VALIDATE" \| "TEST" \| "UNASSIGNED"
+ Describes how the given example (file) should be used for model
+ training. "UNASSIGNED" can be used when user has no preference.
+ GCS\_FILE\_PATH = A path to file on GCS, e.g. "gs://folder/image1.png".
+ LABEL = A display name of an object on an image, video etc., e.g. "dog".
+ Must be up to 32 characters long and can consist only of ASCII Latin
+ letters A-Z and a-z, underscores(\_), and ASCII digits 0-9. For each
+ label an AnnotationSpec is created which display\_name becomes the
+ label; AnnotationSpecs are given back in predictions. INSTANCE\_ID = A
+ positive integer that identifies a specific instance of a labeled entity
+ on an example. Used e.g. to track two cars on a video while being able
+ to tell apart which one is which. BOUNDING\_BOX =
+ VERTEX,VERTEX,VERTEX,VERTEX \| VERTEX,,,VERTEX,, A rectangle parallel to
+ the frame of the example (image, video). If 4 vertices are given they
+ are connected by edges in the order provided, if 2 are given they are
+ recognized as diagonally opposite vertices of the rectangle. VERTEX =
+ COORDINATE,COORDINATE First coordinate is horizontal (x), the second is
+ vertical (y). COORDINATE = A float in 0 to 1 range, relative to total
+ length of image or video in given dimension. For fractions the leading
+ non-decimal 0 can be omitted (i.e. 0.3 = .3). Point 0,0 is in top left.
+ TIME\_SEGMENT\_START = TIME\_OFFSET Expresses a beginning, inclusive, of
+ a time segment within an example that has a time dimension (e.g. video).
+ TIME\_SEGMENT\_END = TIME\_OFFSET Expresses an end, exclusive, of a time
+ segment within an example that has a time dimension (e.g. video).
+ TIME\_OFFSET = A number of seconds as measured from the start of an
+ example (e.g. video). Fractions are allowed, up to a microsecond
+ precision. "inf" is allowed, and it means the end of the example.
+ TEXT\_SNIPPET = A content of a text snippet, UTF-8 encoded, enclosed
+ within double quotes (""). SENTIMENT = An integer between 0 and
+ Dataset.text\_sentiment\_dataset\_metadata.sentiment\_max (inclusive).
+ Describes the ordinal of the sentiment - higher value means a more
+ positive sentiment. All the values are completely relative, i.e. neither
+ 0 needs to mean a negative or neutral sentiment nor sentiment\_max needs
+ to mean a positive one - it is just required that 0 is the least
+ positive sentiment in the data, and sentiment\_max is the most positive
+ one. The SENTIMENT shouldn't be confused with "score" or "magnitude"
+ from the previous Natural Language Sentiment Analysis API. All SENTIMENT
+ values between 0 and sentiment\_max must be represented in the imported
+ data. On prediction the same 0 to sentiment\_max range will be used. The
+ difference between neighboring sentiment values needs not to be uniform,
+ e.g. 1 and 2 may be similar whereas the difference between 2 and 3 may
+ be huge.
+
+ Errors: If any of the provided CSV files can't be parsed or if more than
+ certain percent of CSV rows cannot be processed then the operation fails
+ and nothing is imported. Regardless of overall success or failure the
+ per-row failures, up to a certain count cap, is listed in
+ Operation.metadata.partial\_failures.
+
+
+ Attributes:
+ source:
+ The source of the input.
+ gcs_source:
+ The Google Cloud Storage location for the input content. In
+ ImportData, the gcs\_source points to a csv with structure
+ described in the comment.
+ bigquery_source:
+ The BigQuery location for the input content.
+ params:
+ Additional domain-specific parameters describing the semantic
+ of the imported data, any string must be up to 25000
+ characters long. - For Tables: ``schema_inference_version``
+ - (integer) Required. The version of the algorithm that
+ should be used for the initial inference of the schema
+ (columns' DataTypes) of the table the data is being
+ imported into. Allowed values: "1".
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.InputConfig)
+ ),
+)
+_sym_db.RegisterMessage(InputConfig)
+_sym_db.RegisterMessage(InputConfig.ParamsEntry)
+
+BatchPredictInputConfig = _reflection.GeneratedProtocolMessageType(
+ "BatchPredictInputConfig",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_BATCHPREDICTINPUTCONFIG,
+ __module__="google.cloud.automl_v1beta1.proto.io_pb2",
+ __doc__="""Input configuration for BatchPredict Action.
+
+ The format of input depends on the ML problem of the model used for
+ prediction. As input source the
+ [gcs\_source][google.cloud.automl.v1beta1.InputConfig.gcs\_source] is
+ expected, unless specified otherwise.
+
+ The formats are represented in EBNF with commas being literal and with
+ non-terminal symbols defined near the end of this comment. The formats
+ are:
+
+ - For Image Classification: CSV file(s) with each line having just a
+ single column: GCS\_FILE\_PATH which leads to image of up to 30MB in
+ size. Supported extensions: .JPEG, .GIF, .PNG. This path is treated
+ as the ID in the Batch predict output. Three sample rows:
+ gs://folder/image1.jpeg gs://folder/image2.gif gs://folder/image3.png
+
+ - For Image Object Detection: CSV file(s) with each line having just a
+ single column: GCS\_FILE\_PATH which leads to image of up to 30MB in
+ size. Supported extensions: .JPEG, .GIF, .PNG. This path is treated
+ as the ID in the Batch predict output. Three sample rows:
+ gs://folder/image1.jpeg gs://folder/image2.gif gs://folder/image3.png
+ - For Video Classification: CSV file(s) with each line in format:
+ GCS\_FILE\_PATH,TIME\_SEGMENT\_START,TIME\_SEGMENT\_END
+ GCS\_FILE\_PATH leads to video of up to 50GB in size and up to 3h
+ duration. Supported extensions: .MOV, .MPEG4, .MP4, .AVI.
+ TIME\_SEGMENT\_START and TIME\_SEGMENT\_END must be within the length
+ of the video, and end has to be after the start. Three sample rows:
+ gs://folder/video1.mp4,10,40 gs://folder/video1.mp4,20,60
+ gs://folder/vid2.mov,0,inf
+
+ - For Video Object Tracking: CSV file(s) with each line in format:
+ GCS\_FILE\_PATH,TIME\_SEGMENT\_START,TIME\_SEGMENT\_END
+ GCS\_FILE\_PATH leads to video of up to 50GB in size and up to 3h
+ duration. Supported extensions: .MOV, .MPEG4, .MP4, .AVI.
+ TIME\_SEGMENT\_START and TIME\_SEGMENT\_END must be within the length
+ of the video, and end has to be after the start. Three sample rows:
+ gs://folder/video1.mp4,10,240 gs://folder/video1.mp4,300,360
+ gs://folder/vid2.mov,0,inf
+ - For Text Classification: CSV file(s) with each line having just a
+ single column: GCS\_FILE\_PATH \| TEXT\_SNIPPET Any given text file
+ can have size upto 128kB. Any given text snippet content must have
+ 60,000 characters or less. Three sample rows: gs://folder/text1.txt
+ "Some text content to predict" gs://folder/text3.pdf Supported file
+ extensions: .txt, .pdf
+
+ - For Text Sentiment: CSV file(s) with each line having just a single
+ column: GCS\_FILE\_PATH \| TEXT\_SNIPPET Any given text file can have
+ size upto 128kB. Any given text snippet content must have 500
+ characters or less. Three sample rows: gs://folder/text1.txt "Some
+ text content to predict" gs://folder/text3.pdf Supported file
+ extensions: .txt, .pdf
+
+ - For Text Extraction .JSONL (i.e. JSON Lines) file(s) which either
+ provide text in-line or as documents (for a single BatchPredict call
+ only one of the these formats may be used). The in-line .JSONL
+ file(s) contain per line a proto that wraps a temporary user-assigned
+ TextSnippet ID (string up to 2000 characters long) called "id", a
+ TextSnippet proto (in json representation) and zero or more
+ TextFeature protos. Any given text snippet content must have 30,000
+ characters or less, and also be UTF-8 NFC encoded (ASCII already is).
+ The IDs provided should be unique. The document .JSONL file(s)
+ contain, per line, a proto that wraps a Document proto with
+ input\_config set. Only PDF documents are supported now, and each
+ document must be up to 2MB large. Any given .JSONL file must be 100MB
+ or smaller, and no more than 20 files may be given.
+
+ - For Tables: Either
+ [gcs\_source][google.cloud.automl.v1beta1.InputConfig.gcs\_source] or
+
+ [bigquery\_source][google.cloud.automl.v1beta1.InputConfig.bigquery\_source].
+ GCS case: CSV file(s), each by itself 10GB or smaller and total size
+ must be 100GB or smaller, where first file must have a header containing
+ column names. If the first row of a subsequent file is the same as the
+ header, then it is also treated as a header. All other rows contain
+ values for the corresponding columns. The column names must contain the
+ model's
+
+ [input\_feature\_column\_specs'][google.cloud.automl.v1beta1.TablesModelMetadata.input\_feature\_column\_specs]
+
+ [display\_name-s][google.cloud.automl.v1beta1.ColumnSpec.display\_name]
+ (order doesn't matter). The columns corresponding to the model's input
+ feature column specs must contain values compatible with the column
+ spec's data types. Prediction on all the rows, i.e. the CSV lines, will
+ be attempted. For FORECASTING
+
+ [prediction\_type][google.cloud.automl.v1beta1.TablesModelMetadata.prediction\_type]:
+ all columns having
+
+ [TIME\_SERIES\_AVAILABLE\_PAST\_ONLY][google.cloud.automl.v1beta1.ColumnSpec.ForecastingMetadata.ColumnType]
+ type will be ignored. First three sample rows of a CSV file: "First
+ Name","Last Name","Dob","Addresses"
+
+ "John","Doe","1968-01-22","[{"status":"current","address":"123\_First\_Avenue","city":"Seattle","state":"WA","zip":"11111","numberOfYears":"1"},{"status":"previous","address":"456\_Main\_Street","city":"Portland","state":"OR","zip":"22222","numberOfYears":"5"}]"
+
+ "Jane","Doe","1980-10-16","[{"status":"current","address":"789\_Any\_Avenue","city":"Albany","state":"NY","zip":"33333","numberOfYears":"2"},{"status":"previous","address":"321\_Main\_Street","city":"Hoboken","state":"NJ","zip":"44444","numberOfYears":"3"}]}
+ BigQuery case: An URI of a BigQuery table. The user data size of the
+ BigQuery table must be 100GB or smaller. The column names must contain
+ the model's
+
+ [input\_feature\_column\_specs'][google.cloud.automl.v1beta1.TablesModelMetadata.input\_feature\_column\_specs]
+
+ [display\_name-s][google.cloud.automl.v1beta1.ColumnSpec.display\_name]
+ (order doesn't matter). The columns corresponding to the model's input
+ feature column specs must contain values compatible with the column
+ spec's data types. Prediction on all the rows of the table will be
+ attempted. For FORECASTING
+
+ [prediction\_type][google.cloud.automl.v1beta1.TablesModelMetadata.prediction\_type]:
+ all columns having
+
+ [TIME\_SERIES\_AVAILABLE\_PAST\_ONLY][google.cloud.automl.v1beta1.ColumnSpec.ForecastingMetadata.ColumnType]
+ type will be ignored.
+
+ Definitions: GCS\_FILE\_PATH = A path to file on GCS, e.g.
+ "gs://folder/video.avi". TEXT\_SNIPPET = A content of a text snippet,
+ UTF-8 encoded, enclosed within double quotes ("") TIME\_SEGMENT\_START =
+ TIME\_OFFSET Expresses a beginning, inclusive, of a time segment within
+ an example that has a time dimension (e.g. video). TIME\_SEGMENT\_END =
+ TIME\_OFFSET Expresses an end, exclusive, of a time segment within an
+ example that has a time dimension (e.g. video). TIME\_OFFSET = A number
+ of seconds as measured from the start of an example (e.g. video).
+ Fractions are allowed, up to a microsecond precision. "inf" is allowed
+ and it means the end of the example.
+
+ Errors: If any of the provided CSV files can't be parsed or if more than
+ certain percent of CSV rows cannot be processed then the operation fails
+ and prediction does not happen. Regardless of overall success or failure
+ the per-row failures, up to a certain count cap, will be listed in
+ Operation.metadata.partial\_failures.
+
+
+ Attributes:
+ source:
+ Required. The source of the input.
+ gcs_source:
+ The Google Cloud Storage location for the input content.
+ bigquery_source:
+ The BigQuery location for the input content.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.BatchPredictInputConfig)
+ ),
+)
+_sym_db.RegisterMessage(BatchPredictInputConfig)
+
+DocumentInputConfig = _reflection.GeneratedProtocolMessageType(
+ "DocumentInputConfig",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_DOCUMENTINPUTCONFIG,
+ __module__="google.cloud.automl_v1beta1.proto.io_pb2",
+ __doc__="""Input configuration of a
+ [Document][google.cloud.automl.v1beta1.Document].
+
+
+ Attributes:
+ gcs_source:
+ The Google Cloud Storage location of the document file. Only a
+ single path should be given. Max supported size: 512MB.
+ Supported extensions: .PDF.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.DocumentInputConfig)
+ ),
+)
+_sym_db.RegisterMessage(DocumentInputConfig)
+
+OutputConfig = _reflection.GeneratedProtocolMessageType(
+ "OutputConfig",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_OUTPUTCONFIG,
+ __module__="google.cloud.automl_v1beta1.proto.io_pb2",
+ __doc__="""
+- For Translation: CSV file ``translation.csv``, with each line in
+ format: ML\_USE,GCS\_FILE\_PATH GCS\_FILE\_PATH leads to a .TSV file
+ which describes examples that have given ML\_USE, using the following
+ row format per line: TEXT\_SNIPPET (in source language)
+ \\tTEXT\_SNIPPET (in target language)
+
+ - For Tables: Output depends on whether the dataset was imported from
+ GCS or BigQuery. GCS case:
+
+ [gcs\_destination][google.cloud.automl.v1beta1.OutputConfig.gcs\_destination]
+ must be set. Exported are CSV file(s) ``tables_1.csv``,
+ ``tables_2.csv``,...,\ ``tables_N.csv`` with each having as header line
+ the table's column names, and all other lines contain values for the
+ header columns. BigQuery case:
+
+ [bigquery\_destination][google.cloud.automl.v1beta1.OutputConfig.bigquery\_destination]
+ pointing to a BigQuery project must be set. In the given project a new
+ dataset will be created with name
+
+ ``export_data__``
+ where will be made BigQuery-dataset-name compatible (e.g. most special
+ characters will become underscores), and timestamp will be in
+ YYYY\_MM\_DDThh\_mm\_ss\_sssZ "based on ISO-8601" format. In that
+ dataset a new table called ``primary_table`` will be created, and filled
+ with precisely the same data as this obtained on import.
+
+
+ Attributes:
+ destination:
+ Required. The destination of the output.
+ gcs_destination:
+ The Google Cloud Storage location where the output is to be
+ written to. For Image Object Detection, Text Extraction, Video
+ Classification and Tables, in the given directory a new
+ directory will be created with name: export\_data-- where
+ timestamp is in YYYY-MM-DDThh:mm:ss.sssZ ISO-8601 format. All
+ export output will be written into that directory.
+ bigquery_destination:
+ The BigQuery location where the output is to be written to.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.OutputConfig)
+ ),
+)
+_sym_db.RegisterMessage(OutputConfig)
+
+BatchPredictOutputConfig = _reflection.GeneratedProtocolMessageType(
+ "BatchPredictOutputConfig",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_BATCHPREDICTOUTPUTCONFIG,
+ __module__="google.cloud.automl_v1beta1.proto.io_pb2",
+ __doc__="""Output configuration for BatchPredict Action.
+
+ As destination the
+
+ [gcs\_destination][google.cloud.automl.v1beta1.BatchPredictOutputConfig.gcs\_destination]
+ must be set unless specified otherwise for a domain. If gcs\_destination
+ is set then in the given directory a new directory is created. Its name
+ will be "prediction--", where timestamp is in YYYY-MM-DDThh:mm:ss.sssZ
+ ISO-8601 format. The contents of it depends on the ML problem the
+ predictions are made for.
+
+ - For Image Classification: In the created directory files
+ ``image_classification_1.jsonl``,
+ ``image_classification_2.jsonl``,...,\ ``image_classification_N.jsonl``
+ will be created, where N may be 1, and depends on the total number of
+ the successfully predicted images and annotations. A single image
+ will be listed only once with all its annotations, and its
+ annotations will never be split across files. Each .JSONL file will
+ contain, per line, a JSON representation of a proto that wraps
+ image's "ID" : "" followed by a list of zero or more
+ AnnotationPayload protos (called annotations), which have
+ classification detail populated. If prediction for any image failed
+ (partially or completely), then an additional ``errors_1.jsonl``,
+ ``errors_2.jsonl``,..., ``errors_N.jsonl`` files will be created (N
+ depends on total number of failed predictions). These files will have
+ a JSON representation of a proto that wraps the same "ID" : "" but
+ here followed by exactly one
+
+ ```google.rpc.Status`` `__
+ containing only ``code`` and ``message``\ fields.
+
+ - For Image Object Detection: In the created directory files
+ ``image_object_detection_1.jsonl``,
+ ``image_object_detection_2.jsonl``,...,\ ``image_object_detection_N.jsonl``
+ will be created, where N may be 1, and depends on the total number of
+ the successfully predicted images and annotations. Each .JSONL file
+ will contain, per line, a JSON representation of a proto that wraps
+ image's "ID" : "" followed by a list of zero or more
+ AnnotationPayload protos (called annotations), which have
+ image\_object\_detection detail populated. A single image will be
+ listed only once with all its annotations, and its annotations will
+ never be split across files. If prediction for any image failed
+ (partially or completely), then additional ``errors_1.jsonl``,
+ ``errors_2.jsonl``,..., ``errors_N.jsonl`` files will be created (N
+ depends on total number of failed predictions). These files will have
+ a JSON representation of a proto that wraps the same "ID" : "" but
+ here followed by exactly one
+
+ ```google.rpc.Status`` `__
+ containing only ``code`` and ``message``\ fields. \* For Video
+ Classification: In the created directory a video\_classification.csv
+ file, and a .JSON file per each video classification requested in the
+ input (i.e. each line in given CSV(s)), will be created.
+
+ ::
+
+ The format of video_classification.csv is:
+
+ GCS\_FILE\_PATH,TIME\_SEGMENT\_START,TIME\_SEGMENT\_END,JSON\_FILE\_NAME,STATUS
+ where: GCS\_FILE\_PATH,TIME\_SEGMENT\_START,TIME\_SEGMENT\_END = matches
+ 1 to 1 the prediction input lines (i.e. video\_classification.csv has
+ precisely the same number of lines as the prediction input had.)
+ JSON\_FILE\_NAME = Name of .JSON file in the output directory, which
+ contains prediction responses for the video time segment. STATUS = "OK"
+ if prediction completed successfully, or an error code with message
+ otherwise. If STATUS is not "OK" then the .JSON file for that line may
+ not exist or be empty.
+
+ ::
+
+ Each .JSON file, assuming STATUS is "OK", will contain a list of
+ AnnotationPayload protos in JSON format, which are the predictions
+ for the video time segment the file is assigned to in the
+ video_classification.csv. All AnnotationPayload protos will have
+ video_classification field set, and will be sorted by
+ video_classification.type field (note that the returned types are
+ governed by `classifaction_types` parameter in
+ [PredictService.BatchPredictRequest.params][]).
+
+ - For Video Object Tracking: In the created directory a
+ video\_object\_tracking.csv file will be created, and multiple files
+ video\_object\_trackinng\_1.json,
+ video\_object\_trackinng\_2.json,...,
+ video\_object\_trackinng\_N.json, where N is the number of requests
+ in the input (i.e. the number of lines in given CSV(s)).
+
+ ::
+
+ The format of video_object_tracking.csv is:
+
+ GCS\_FILE\_PATH,TIME\_SEGMENT\_START,TIME\_SEGMENT\_END,JSON\_FILE\_NAME,STATUS
+ where: GCS\_FILE\_PATH,TIME\_SEGMENT\_START,TIME\_SEGMENT\_END = matches
+ 1 to 1 the prediction input lines (i.e. video\_object\_tracking.csv has
+ precisely the same number of lines as the prediction input had.)
+ JSON\_FILE\_NAME = Name of .JSON file in the output directory, which
+ contains prediction responses for the video time segment. STATUS = "OK"
+ if prediction completed successfully, or an error code with message
+ otherwise. If STATUS is not "OK" then the .JSON file for that line may
+ not exist or be empty.
+
+ ::
+
+ Each .JSON file, assuming STATUS is "OK", will contain a list of
+ AnnotationPayload protos in JSON format, which are the predictions
+ for each frame of the video time segment the file is assigned to in
+ video_object_tracking.csv. All AnnotationPayload protos will have
+ video_object_tracking field set.
+
+ - For Text Classification: In the created directory files
+ ``text_classification_1.jsonl``,
+ ``text_classification_2.jsonl``,...,\ ``text_classification_N.jsonl``
+ will be created, where N may be 1, and depends on the total number of
+ inputs and annotations found.
+
+ ::
+
+ Each .JSONL file will contain, per line, a JSON representation of a
+ proto that wraps input text snippet or input text file and a list of
+ zero or more AnnotationPayload protos (called annotations), which
+ have classification detail populated. A single text snippet or file
+ will be listed only once with all its annotations, and its
+ annotations will never be split across files.
+
+ If prediction for any text snippet or file failed (partially or
+ completely), then additional `errors_1.jsonl`, `errors_2.jsonl`,...,
+ `errors_N.jsonl` files will be created (N depends on total number of
+ failed predictions). These files will have a JSON representation of a
+ proto that wraps input text snippet or input text file followed by
+ exactly one
+
+ ```google.rpc.Status`` `__
+ containing only ``code`` and ``message``.
+
+ - For Text Sentiment: In the created directory files
+ ``text_sentiment_1.jsonl``,
+ ``text_sentiment_2.jsonl``,...,\ ``text_sentiment_N.jsonl`` will be
+ created, where N may be 1, and depends on the total number of inputs
+ and annotations found.
+
+ ::
+
+ Each .JSONL file will contain, per line, a JSON representation of a
+ proto that wraps input text snippet or input text file and a list of
+ zero or more AnnotationPayload protos (called annotations), which
+ have text_sentiment detail populated. A single text snippet or file
+ will be listed only once with all its annotations, and its
+ annotations will never be split across files.
+
+ If prediction for any text snippet or file failed (partially or
+ completely), then additional `errors_1.jsonl`, `errors_2.jsonl`,...,
+ `errors_N.jsonl` files will be created (N depends on total number of
+ failed predictions). These files will have a JSON representation of a
+ proto that wraps input text snippet or input text file followed by
+ exactly one
+
+ ```google.rpc.Status`` `__
+ containing only ``code`` and ``message``.
+
+ - For Text Extraction: In the created directory files
+ ``text_extraction_1.jsonl``,
+ ``text_extraction_2.jsonl``,...,\ ``text_extraction_N.jsonl`` will be
+ created, where N may be 1, and depends on the total number of inputs
+ and annotations found. The contents of these .JSONL file(s) depend on
+ whether the input used inline text, or documents. If input was
+ inline, then each .JSONL file will contain, per line, a JSON
+ representation of a proto that wraps given in request text snippet's
+ "id" (if specified), followed by input text snippet, and a list of
+ zero or more AnnotationPayload protos (called annotations), which
+ have text\_extraction detail populated. A single text snippet will be
+ listed only once with all its annotations, and its annotations will
+ never be split across files. If input used documents, then each
+ .JSONL file will contain, per line, a JSON representation of a proto
+ that wraps given in request document proto, followed by its OCR-ed
+ representation in the form of a text snippet, finally followed by a
+ list of zero or more AnnotationPayload protos (called annotations),
+ which have text\_extraction detail populated and refer, via their
+ indices, to the OCR-ed text snippet. A single document (and its text
+ snippet) will be listed only once with all its annotations, and its
+ annotations will never be split across files. If prediction for any
+ text snippet failed (partially or completely), then additional
+ ``errors_1.jsonl``, ``errors_2.jsonl``,..., ``errors_N.jsonl`` files
+ will be created (N depends on total number of failed predictions).
+ These files will have a JSON representation of a proto that wraps
+ either the "id" : "" (in case of inline) or the document proto (in
+ case of document) but here followed by exactly one
+
+ ```google.rpc.Status`` `__
+ containing only ``code`` and ``message``.
+
+ - For Tables: Output depends on whether
+
+ [gcs\_destination][google.cloud.automl.v1beta1.BatchPredictOutputConfig.gcs\_destination]
+ or
+
+ [bigquery\_destination][google.cloud.automl.v1beta1.BatchPredictOutputConfig.bigquery\_destination]
+ is set (either is allowed). GCS case: In the created directory files
+ ``tables_1.csv``, ``tables_2.csv``,..., ``tables_N.csv`` will be
+ created, where N may be 1, and depends on the total number of the
+ successfully predicted rows. For all CLASSIFICATION
+
+ [prediction\_type-s][google.cloud.automl.v1beta1.TablesModelMetadata.prediction\_type]:
+ Each .csv file will contain a header, listing all columns'
+
+ [display\_name-s][google.cloud.automl.v1beta1.ColumnSpec.display\_name]
+ given on input followed by M target column names in the format of
+
+ "<[target\_column\_specs][google.cloud.automl.v1beta1.TablesModelMetadata.target\_column\_spec]
+
+ [display\_name][google.cloud.automl.v1beta1.ColumnSpec.display\_name]>\_\_score"
+ where M is the number of distinct target values, i.e. number of distinct
+ values in the target column of the table used to train the model.
+ Subsequent lines will contain the respective values of successfully
+ predicted rows, with the last, i.e. the target, columns having the
+ corresponding prediction
+ [scores][google.cloud.automl.v1beta1.TablesAnnotation.score]. For
+ REGRESSION and FORECASTING
+
+ [prediction\_type-s][google.cloud.automl.v1beta1.TablesModelMetadata.prediction\_type]:
+ Each .csv file will contain a header, listing all columns'
+ [display\_name-s][google.cloud.automl.v1beta1.display\_name] given on
+ input followed by the predicted target column with name in the format of
+
+ "predicted\_<[target\_column\_specs][google.cloud.automl.v1beta1.TablesModelMetadata.target\_column\_spec]
+
+ [display\_name][google.cloud.automl.v1beta1.ColumnSpec.display\_name]>"
+ Subsequent lines will contain the respective values of successfully
+ predicted rows, with the last, i.e. the target, column having the
+ predicted target value. If prediction for any rows failed, then an
+ additional ``errors_1.csv``, ``errors_2.csv``,..., ``errors_N.csv`` will
+ be created (N depends on total number of failed rows). These files will
+ have analogous format as ``tables_*.csv``, but always with a single
+ target column having
+
+ ```google.rpc.Status`` `__
+ represented as a JSON string, and containing only ``code`` and
+ ``message``. BigQuery case:
+
+ [bigquery\_destination][google.cloud.automl.v1beta1.OutputConfig.bigquery\_destination]
+ pointing to a BigQuery project must be set. In the given project a new
+ dataset will be created with name
+ ``prediction__`` where
+ will be made BigQuery-dataset-name compatible (e.g. most special
+ characters will become underscores), and timestamp will be in
+ YYYY\_MM\_DDThh\_mm\_ss\_sssZ "based on ISO-8601" format. In the dataset
+ two tables will be created, ``predictions``, and ``errors``. The
+ ``predictions`` table's column names will be the input columns'
+
+ [display\_name-s][google.cloud.automl.v1beta1.ColumnSpec.display\_name]
+ followed by the target column with name in the format of
+
+ "predicted\_<[target\_column\_specs][google.cloud.automl.v1beta1.TablesModelMetadata.target\_column\_spec]
+
+ [display\_name][google.cloud.automl.v1beta1.ColumnSpec.display\_name]>"
+ The input feature columns will contain the respective values of
+ successfully predicted rows, with the target column having an ARRAY of
+
+ [AnnotationPayloads][google.cloud.automl.v1beta1.AnnotationPayload],
+ represented as STRUCT-s, containing
+ [TablesAnnotation][google.cloud.automl.v1beta1.TablesAnnotation]. The
+ ``errors`` table contains rows for which the prediction has failed, it
+ has analogous input columns while the target column name is in the
+ format of
+
+ "errors\_<[target\_column\_specs][google.cloud.automl.v1beta1.TablesModelMetadata.target\_column\_spec]
+
+ [display\_name][google.cloud.automl.v1beta1.ColumnSpec.display\_name]>",
+ and as a value has
+
+ ```google.rpc.Status`` `__
+ represented as a STRUCT, and containing only ``code`` and ``message``.
+
+
+ Attributes:
+ destination:
+ Required. The destination of the output.
+ gcs_destination:
+ The Google Cloud Storage location of the directory where the
+ output is to be written to.
+ bigquery_destination:
+ The BigQuery location where the output is to be written to.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.BatchPredictOutputConfig)
+ ),
+)
+_sym_db.RegisterMessage(BatchPredictOutputConfig)
+
+ModelExportOutputConfig = _reflection.GeneratedProtocolMessageType(
+ "ModelExportOutputConfig",
+ (_message.Message,),
+ dict(
+ ParamsEntry=_reflection.GeneratedProtocolMessageType(
+ "ParamsEntry",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_MODELEXPORTOUTPUTCONFIG_PARAMSENTRY,
+ __module__="google.cloud.automl_v1beta1.proto.io_pb2"
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ModelExportOutputConfig.ParamsEntry)
+ ),
+ ),
+ DESCRIPTOR=_MODELEXPORTOUTPUTCONFIG,
+ __module__="google.cloud.automl_v1beta1.proto.io_pb2",
+ __doc__="""Output configuration for ModelExport Action.
+
+
+ Attributes:
+ destination:
+ Required. The destination of the output.
+ gcs_destination:
+ The Google Cloud Storage location where the model is to be
+ written to. This location may only be set for the following
+ model formats: "tflite", "edgetpu\_tflite", "core\_ml",
+ "docker". Under the directory given as the destination a new
+ one with name "model-export--", where timestamp is in YYYY-MM-
+ DDThh:mm:ss.sssZ ISO-8601 format, will be created. Inside the
+ model and any of its supporting files will be written.
+ gcr_destination:
+ The GCR location where model image is to be pushed to. This
+ location may only be set for the following model formats:
+ "docker". The model image will be created under the given
+ URI.
+ model_format:
+ The format in which the model must be exported. The available,
+ and default, formats depend on the problem and model type (if
+ given problem and type combination doesn't have a format
+ listed, it means its models are not exportable): - For Image
+ Classification mobile-low-latency-1, mobile-versatile-1,
+ mobile-high-accuracy-1: "tflite" (default), "edgetpu\_tflite",
+ "tf\_saved\_model", "docker". - For Image Classification
+ mobile-core-ml-low-latency-1, mobile-core-ml-versatile-1,
+ mobile-core-ml-high-accuracy-1: "core\_ml" (default).
+ Formats description: - tflite - Used for Android mobile
+ devices. - edgetpu\_tflite - Used for `Edge TPU
+ `__ devices. -
+ tf\_saved\_model - A tensorflow model in SavedModel format. -
+ docker - Used for Docker containers. Use the params field to
+ customize the container. The container is verified to work
+ correctly on ubuntu 16.04 operating system. See more at
+ [containers quickstart](https:
+ //cloud.google.com/vision/automl/docs/containers-gcs-
+ quickstart) \* core\_ml - Used for iOS mobile devices.
+ params:
+ Additional model-type and format specific parameters
+ describing the requirements for the to be exported model
+ files, any string must be up to 25000 characters long. - For
+ ``docker`` format: ``cpu_architecture`` - (string) "x86\_64"
+ (default). ``gpu_architecture`` - (string) "none" (default),
+ "nvidia".
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ModelExportOutputConfig)
+ ),
+)
+_sym_db.RegisterMessage(ModelExportOutputConfig)
+_sym_db.RegisterMessage(ModelExportOutputConfig.ParamsEntry)
+
+ExportEvaluatedExamplesOutputConfig = _reflection.GeneratedProtocolMessageType(
+ "ExportEvaluatedExamplesOutputConfig",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_EXPORTEVALUATEDEXAMPLESOUTPUTCONFIG,
+ __module__="google.cloud.automl_v1beta1.proto.io_pb2",
+ __doc__="""Output configuration for ExportEvaluatedExamples Action. Note that this
+ call is available only for 30 days since the moment the model was
+ evaluated. The output depends on the domain, as follows (note that only
+ examples from the TEST set are exported):
+
+ - For Tables:
+
+ [bigquery\_destination][google.cloud.automl.v1beta1.OutputConfig.bigquery\_destination]
+ pointing to a BigQuery project must be set. In the given project a new
+ dataset will be created with name
+
+ ``export_evaluated_examples__``
+ where will be made BigQuery-dataset-name compatible (e.g. most special
+ characters will become underscores), and timestamp will be in
+ YYYY\_MM\_DDThh\_mm\_ss\_sssZ "based on ISO-8601" format. In the dataset
+ an ``evaluated_examples`` table will be created. It will have all the
+ same columns as the
+
+ [primary\_table][google.cloud.automl.v1beta1.TablesDatasetMetadata.primary\_table\_spec\_id]
+ of the [dataset][google.cloud.automl.v1beta1.Model.dataset\_id] from
+ which the model was created, as they were at the moment of model's
+ evaluation (this includes the target column with its ground truth),
+ followed by a column called "predicted\_". That last column will contain
+ the model's prediction result for each respective row, given as ARRAY of
+ [AnnotationPayloads][google.cloud.automl.v1beta1.AnnotationPayload],
+ represented as STRUCT-s, containing
+ [TablesAnnotation][google.cloud.automl.v1beta1.TablesAnnotation].
+
+
+ Attributes:
+ destination:
+ Required. The destination of the output.
+ bigquery_destination:
+ The BigQuery location where the output is to be written to.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ExportEvaluatedExamplesOutputConfig)
+ ),
+)
+_sym_db.RegisterMessage(ExportEvaluatedExamplesOutputConfig)
+
+GcsSource = _reflection.GeneratedProtocolMessageType(
+ "GcsSource",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_GCSSOURCE,
+ __module__="google.cloud.automl_v1beta1.proto.io_pb2",
+ __doc__="""The Google Cloud Storage location for the input content.
+
+
+ Attributes:
+ input_uris:
+ Required. Google Cloud Storage URIs to input files, up to 2000
+ characters long. Accepted forms: \* Full object path, e.g.
+ gs://bucket/directory/object.csv
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.GcsSource)
+ ),
+)
+_sym_db.RegisterMessage(GcsSource)
+
+BigQuerySource = _reflection.GeneratedProtocolMessageType(
+ "BigQuerySource",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_BIGQUERYSOURCE,
+ __module__="google.cloud.automl_v1beta1.proto.io_pb2",
+ __doc__="""The BigQuery location for the input content.
+
+
+ Attributes:
+ input_uri:
+ Required. BigQuery URI to a table, up to 2000 characters long.
+ Accepted forms: \* BigQuery path e.g.
+ bq://projectId.bqDatasetId.bqTableId
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.BigQuerySource)
+ ),
+)
+_sym_db.RegisterMessage(BigQuerySource)
+
+GcsDestination = _reflection.GeneratedProtocolMessageType(
+ "GcsDestination",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_GCSDESTINATION,
+ __module__="google.cloud.automl_v1beta1.proto.io_pb2",
+ __doc__="""The Google Cloud Storage location where the output is to be written to.
+
+
+ Attributes:
+ output_uri_prefix:
+ Required. Google Cloud Storage URI to output directory, up to
+ 2000 characters long. Accepted forms: \* Prefix path:
+ gs://bucket/directory The requesting user must have write
+ permission to the bucket. The directory is created if it
+ doesn't exist.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.GcsDestination)
+ ),
+)
+_sym_db.RegisterMessage(GcsDestination)
+
+BigQueryDestination = _reflection.GeneratedProtocolMessageType(
+ "BigQueryDestination",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_BIGQUERYDESTINATION,
+ __module__="google.cloud.automl_v1beta1.proto.io_pb2",
+ __doc__="""The BigQuery location for the output content.
+
+
+ Attributes:
+ output_uri:
+ Required. BigQuery URI to a project, up to 2000 characters
+ long. Accepted forms: \* BigQuery path e.g. bq://projectId
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.BigQueryDestination)
+ ),
+)
+_sym_db.RegisterMessage(BigQueryDestination)
+
+GcrDestination = _reflection.GeneratedProtocolMessageType(
+ "GcrDestination",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_GCRDESTINATION,
+ __module__="google.cloud.automl_v1beta1.proto.io_pb2",
+ __doc__="""The GCR location where the image must be pushed to.
+
+
+ Attributes:
+ output_uri:
+ Required. Google Contained Registry URI of the new image, up
+ to 2000 characters long. See https:
+ //cloud.google.com/container-registry/do // cs/pushing-and-
+ pulling#pushing\_an\_image\_to\_a\_registry Accepted forms: \*
+ [HOSTNAME]/[PROJECT-ID]/[IMAGE] \* [HOSTNAME]/[PROJECT-
+ ID]/[IMAGE]:[TAG] The requesting user must have permission to
+ push images the project.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.GcrDestination)
+ ),
+)
+_sym_db.RegisterMessage(GcrDestination)
+
+
+DESCRIPTOR._options = None
+_INPUTCONFIG_PARAMSENTRY._options = None
+_MODELEXPORTOUTPUTCONFIG_PARAMSENTRY._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/automl/google/cloud/automl_v1beta1/proto/io_pb2_grpc.py b/automl/google/cloud/automl_v1beta1/proto/io_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/io_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/automl/google/cloud/automl_v1beta1/proto/model.proto b/automl/google/cloud/automl_v1beta1/proto/model.proto
new file mode 100644
index 000000000000..8c53d9b5832f
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/model.proto
@@ -0,0 +1,103 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.automl.v1beta1;
+
+import "google/cloud/automl/v1beta1/image.proto";
+import "google/cloud/automl/v1beta1/tables.proto";
+import "google/cloud/automl/v1beta1/text.proto";
+import "google/cloud/automl/v1beta1/translation.proto";
+import "google/cloud/automl/v1beta1/video.proto";
+import "google/protobuf/timestamp.proto";
+import "google/api/annotations.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl";
+option java_multiple_files = true;
+option java_package = "com.google.cloud.automl.v1beta1";
+option php_namespace = "Google\\Cloud\\AutoMl\\V1beta1";
+option ruby_package = "Google::Cloud::AutoML::V1beta1";
+
+// API proto representing a trained machine learning model.
+message Model {
+ // Deployment state of the model.
+ enum DeploymentState {
+ // Should not be used, an un-set enum has this value by default.
+ DEPLOYMENT_STATE_UNSPECIFIED = 0;
+
+ // Model is deployed.
+ DEPLOYED = 1;
+
+ // Model is not deployed.
+ UNDEPLOYED = 2;
+ }
+
+ // Required.
+ // The model metadata that is specific to the problem type.
+ // Must match the metadata type of the dataset used to train the model.
+ oneof model_metadata {
+ // Metadata for translation models.
+ TranslationModelMetadata translation_model_metadata = 15;
+
+ // Metadata for image classification models.
+ ImageClassificationModelMetadata image_classification_model_metadata = 13;
+
+ // Metadata for text classification models.
+ TextClassificationModelMetadata text_classification_model_metadata = 14;
+
+ // Metadata for image object detection models.
+ ImageObjectDetectionModelMetadata image_object_detection_model_metadata = 20;
+
+ // Metadata for video classification models.
+ VideoClassificationModelMetadata video_classification_model_metadata = 23;
+
+ // Metadata for video object tracking models.
+ VideoObjectTrackingModelMetadata video_object_tracking_model_metadata = 21;
+
+ // Metadata for text extraction models.
+ TextExtractionModelMetadata text_extraction_model_metadata = 19;
+
+ // Metadata for Tables models.
+ TablesModelMetadata tables_model_metadata = 24;
+
+ // Metadata for text sentiment models.
+ TextSentimentModelMetadata text_sentiment_model_metadata = 22;
+ }
+
+ // Output only. Resource name of the model.
+ // Format: `projects/{project_id}/locations/{location_id}/models/{model_id}`
+ string name = 1;
+
+ // Required. The name of the model to show in the interface. The name can be
+ // up to 32 characters long and can consist only of ASCII Latin letters A-Z
+ // and a-z, underscores
+ // (_), and ASCII digits 0-9. It must start with a letter.
+ string display_name = 2;
+
+ // Required. The resource ID of the dataset used to create the model. The dataset must
+ // come from the same ancestor project and location.
+ string dataset_id = 3;
+
+ // Output only. Timestamp when the model training finished and can be used for prediction.
+ google.protobuf.Timestamp create_time = 7;
+
+ // Output only. Timestamp when this model was last updated.
+ google.protobuf.Timestamp update_time = 11;
+
+ // Output only. Deployment state of the model. A model can only serve
+ // prediction requests after it gets deployed.
+ DeploymentState deployment_state = 8;
+}
diff --git a/automl/google/cloud/automl_v1beta1/proto/model_evaluation.proto b/automl/google/cloud/automl_v1beta1/proto/model_evaluation.proto
new file mode 100644
index 000000000000..ce2db614a182
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/model_evaluation.proto
@@ -0,0 +1,111 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.automl.v1beta1;
+
+import "google/cloud/automl/v1beta1/classification.proto";
+import "google/cloud/automl/v1beta1/detection.proto";
+import "google/cloud/automl/v1beta1/regression.proto";
+import "google/cloud/automl/v1beta1/tables.proto";
+import "google/cloud/automl/v1beta1/text_extraction.proto";
+import "google/cloud/automl/v1beta1/text_sentiment.proto";
+import "google/cloud/automl/v1beta1/translation.proto";
+import "google/protobuf/timestamp.proto";
+import "google/api/annotations.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl";
+option java_multiple_files = true;
+option java_package = "com.google.cloud.automl.v1beta1";
+option php_namespace = "Google\\Cloud\\AutoMl\\V1beta1";
+option ruby_package = "Google::Cloud::AutoML::V1beta1";
+
+// Evaluation results of a model.
+message ModelEvaluation {
+ // Output only. Problem type specific evaluation metrics.
+ oneof metrics {
+ // Model evaluation metrics for image, text, video and tables
+ // classification.
+ // Tables problem is considered a classification when the target column
+ // is CATEGORY DataType.
+ ClassificationEvaluationMetrics classification_evaluation_metrics = 8;
+
+ // Model evaluation metrics for Tables regression.
+ // Tables problem is considered a regression when the target column
+ // has FLOAT64 DataType.
+ RegressionEvaluationMetrics regression_evaluation_metrics = 24;
+
+ // Model evaluation metrics for translation.
+ TranslationEvaluationMetrics translation_evaluation_metrics = 9;
+
+ // Model evaluation metrics for image object detection.
+ ImageObjectDetectionEvaluationMetrics image_object_detection_evaluation_metrics = 12;
+
+ // Model evaluation metrics for video object tracking.
+ VideoObjectTrackingEvaluationMetrics video_object_tracking_evaluation_metrics = 14;
+
+ // Evaluation metrics for text sentiment models.
+ TextSentimentEvaluationMetrics text_sentiment_evaluation_metrics = 11;
+
+ // Evaluation metrics for text extraction models.
+ TextExtractionEvaluationMetrics text_extraction_evaluation_metrics = 13;
+ }
+
+ // Output only. Resource name of the model evaluation.
+ // Format:
+ //
+ // `projects/{project_id}/locations/{location_id}/models/{model_id}/modelEvaluations/{model_evaluation_id}`
+ string name = 1;
+
+ // Output only. The ID of the annotation spec that the model evaluation applies to. The
+ // The ID is empty for the overall model evaluation.
+ // For Tables annotation specs in the dataset do not exist and this ID is
+ // always not set, but for CLASSIFICATION
+ //
+ // [prediction_type-s][google.cloud.automl.v1beta1.TablesModelMetadata.prediction_type]
+ // the
+ // [display_name][google.cloud.automl.v1beta1.ModelEvaluation.display_name]
+ // field is used.
+ string annotation_spec_id = 2;
+
+ // Output only. The value of
+ // [display_name][google.cloud.automl.v1beta1.AnnotationSpec.display_name] at
+ // the moment when the model was trained. Because this field returns a value
+ // at model training time, for different models trained from the same dataset,
+ // the values may differ, since display names could had been changed between
+ // the two model's trainings.
+ // For Tables CLASSIFICATION
+ //
+ // [prediction_type-s][google.cloud.automl.v1beta1.TablesModelMetadata.prediction_type]
+ // distinct values of the target column at the moment of the model evaluation
+ // are populated here.
+ // The display_name is empty for the overall model evaluation.
+ string display_name = 15;
+
+ // Output only. Timestamp when this model evaluation was created.
+ google.protobuf.Timestamp create_time = 5;
+
+ // Output only. The number of examples used for model evaluation, i.e. for
+ // which ground truth from time of model creation is compared against the
+ // predicted annotations created by the model.
+ // For overall ModelEvaluation (i.e. with annotation_spec_id not set) this is
+ // the total number of all examples used for evaluation.
+ // Otherwise, this is the count of examples that according to the ground
+ // truth were annotated by the
+ //
+ // [annotation_spec_id][google.cloud.automl.v1beta1.ModelEvaluation.annotation_spec_id].
+ int32 evaluated_example_count = 6;
+}
diff --git a/automl/google/cloud/automl_v1beta1/proto/model_evaluation_pb2.py b/automl/google/cloud/automl_v1beta1/proto/model_evaluation_pb2.py
new file mode 100644
index 000000000000..681a2663a1d0
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/model_evaluation_pb2.py
@@ -0,0 +1,469 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/automl_v1beta1/proto/model_evaluation.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.cloud.automl_v1beta1.proto import (
+ classification_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_classification__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ detection_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_detection__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ regression_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_regression__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ tables_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_tables__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ text_extraction_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_text__extraction__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ text_sentiment_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_text__sentiment__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ translation_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_translation__pb2,
+)
+from google.protobuf import timestamp_pb2 as google_dot_protobuf_dot_timestamp__pb2
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/automl_v1beta1/proto/model_evaluation.proto",
+ package="google.cloud.automl.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\037com.google.cloud.automl.v1beta1P\001ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\312\002\033Google\\Cloud\\AutoMl\\V1beta1\352\002\036Google::Cloud::AutoML::V1beta1"
+ ),
+ serialized_pb=_b(
+ '\n8google/cloud/automl_v1beta1/proto/model_evaluation.proto\x12\x1bgoogle.cloud.automl.v1beta1\x1a\x36google/cloud/automl_v1beta1/proto/classification.proto\x1a\x31google/cloud/automl_v1beta1/proto/detection.proto\x1a\x32google/cloud/automl_v1beta1/proto/regression.proto\x1a.google/cloud/automl_v1beta1/proto/tables.proto\x1a\x37google/cloud/automl_v1beta1/proto/text_extraction.proto\x1a\x36google/cloud/automl_v1beta1/proto/text_sentiment.proto\x1a\x33google/cloud/automl_v1beta1/proto/translation.proto\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x1cgoogle/api/annotations.proto"\xa7\x07\n\x0fModelEvaluation\x12i\n!classification_evaluation_metrics\x18\x08 \x01(\x0b\x32<.google.cloud.automl.v1beta1.ClassificationEvaluationMetricsH\x00\x12\x61\n\x1dregression_evaluation_metrics\x18\x18 \x01(\x0b\x32\x38.google.cloud.automl.v1beta1.RegressionEvaluationMetricsH\x00\x12\x63\n\x1etranslation_evaluation_metrics\x18\t \x01(\x0b\x32\x39.google.cloud.automl.v1beta1.TranslationEvaluationMetricsH\x00\x12w\n)image_object_detection_evaluation_metrics\x18\x0c \x01(\x0b\x32\x42.google.cloud.automl.v1beta1.ImageObjectDetectionEvaluationMetricsH\x00\x12u\n(video_object_tracking_evaluation_metrics\x18\x0e \x01(\x0b\x32\x41.google.cloud.automl.v1beta1.VideoObjectTrackingEvaluationMetricsH\x00\x12h\n!text_sentiment_evaluation_metrics\x18\x0b \x01(\x0b\x32;.google.cloud.automl.v1beta1.TextSentimentEvaluationMetricsH\x00\x12j\n"text_extraction_evaluation_metrics\x18\r \x01(\x0b\x32<.google.cloud.automl.v1beta1.TextExtractionEvaluationMetricsH\x00\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x1a\n\x12\x61nnotation_spec_id\x18\x02 \x01(\t\x12\x14\n\x0c\x64isplay_name\x18\x0f \x01(\t\x12/\n\x0b\x63reate_time\x18\x05 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12\x1f\n\x17\x65valuated_example_count\x18\x06 \x01(\x05\x42\t\n\x07metricsB\xa5\x01\n\x1f\x63om.google.cloud.automl.v1beta1P\x01ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\xca\x02\x1bGoogle\\Cloud\\AutoMl\\V1beta1\xea\x02\x1eGoogle::Cloud::AutoML::V1beta1b\x06proto3'
+ ),
+ dependencies=[
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_classification__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_detection__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_regression__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_tables__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_text__extraction__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_text__sentiment__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_translation__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_timestamp__pb2.DESCRIPTOR,
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ ],
+)
+
+
+_MODELEVALUATION = _descriptor.Descriptor(
+ name="ModelEvaluation",
+ full_name="google.cloud.automl.v1beta1.ModelEvaluation",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="classification_evaluation_metrics",
+ full_name="google.cloud.automl.v1beta1.ModelEvaluation.classification_evaluation_metrics",
+ index=0,
+ number=8,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="regression_evaluation_metrics",
+ full_name="google.cloud.automl.v1beta1.ModelEvaluation.regression_evaluation_metrics",
+ index=1,
+ number=24,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="translation_evaluation_metrics",
+ full_name="google.cloud.automl.v1beta1.ModelEvaluation.translation_evaluation_metrics",
+ index=2,
+ number=9,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="image_object_detection_evaluation_metrics",
+ full_name="google.cloud.automl.v1beta1.ModelEvaluation.image_object_detection_evaluation_metrics",
+ index=3,
+ number=12,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="video_object_tracking_evaluation_metrics",
+ full_name="google.cloud.automl.v1beta1.ModelEvaluation.video_object_tracking_evaluation_metrics",
+ index=4,
+ number=14,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="text_sentiment_evaluation_metrics",
+ full_name="google.cloud.automl.v1beta1.ModelEvaluation.text_sentiment_evaluation_metrics",
+ index=5,
+ number=11,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="text_extraction_evaluation_metrics",
+ full_name="google.cloud.automl.v1beta1.ModelEvaluation.text_extraction_evaluation_metrics",
+ index=6,
+ number=13,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.automl.v1beta1.ModelEvaluation.name",
+ index=7,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="annotation_spec_id",
+ full_name="google.cloud.automl.v1beta1.ModelEvaluation.annotation_spec_id",
+ index=8,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="display_name",
+ full_name="google.cloud.automl.v1beta1.ModelEvaluation.display_name",
+ index=9,
+ number=15,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="create_time",
+ full_name="google.cloud.automl.v1beta1.ModelEvaluation.create_time",
+ index=10,
+ number=5,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="evaluated_example_count",
+ full_name="google.cloud.automl.v1beta1.ModelEvaluation.evaluated_example_count",
+ index=11,
+ number=6,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="metrics",
+ full_name="google.cloud.automl.v1beta1.ModelEvaluation.metrics",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=526,
+ serialized_end=1461,
+)
+
+_MODELEVALUATION.fields_by_name[
+ "classification_evaluation_metrics"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_classification__pb2._CLASSIFICATIONEVALUATIONMETRICS
+)
+_MODELEVALUATION.fields_by_name[
+ "regression_evaluation_metrics"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_regression__pb2._REGRESSIONEVALUATIONMETRICS
+)
+_MODELEVALUATION.fields_by_name[
+ "translation_evaluation_metrics"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_translation__pb2._TRANSLATIONEVALUATIONMETRICS
+)
+_MODELEVALUATION.fields_by_name[
+ "image_object_detection_evaluation_metrics"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_detection__pb2._IMAGEOBJECTDETECTIONEVALUATIONMETRICS
+)
+_MODELEVALUATION.fields_by_name[
+ "video_object_tracking_evaluation_metrics"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_detection__pb2._VIDEOOBJECTTRACKINGEVALUATIONMETRICS
+)
+_MODELEVALUATION.fields_by_name[
+ "text_sentiment_evaluation_metrics"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_text__sentiment__pb2._TEXTSENTIMENTEVALUATIONMETRICS
+)
+_MODELEVALUATION.fields_by_name[
+ "text_extraction_evaluation_metrics"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_text__extraction__pb2._TEXTEXTRACTIONEVALUATIONMETRICS
+)
+_MODELEVALUATION.fields_by_name[
+ "create_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_MODELEVALUATION.oneofs_by_name["metrics"].fields.append(
+ _MODELEVALUATION.fields_by_name["classification_evaluation_metrics"]
+)
+_MODELEVALUATION.fields_by_name[
+ "classification_evaluation_metrics"
+].containing_oneof = _MODELEVALUATION.oneofs_by_name["metrics"]
+_MODELEVALUATION.oneofs_by_name["metrics"].fields.append(
+ _MODELEVALUATION.fields_by_name["regression_evaluation_metrics"]
+)
+_MODELEVALUATION.fields_by_name[
+ "regression_evaluation_metrics"
+].containing_oneof = _MODELEVALUATION.oneofs_by_name["metrics"]
+_MODELEVALUATION.oneofs_by_name["metrics"].fields.append(
+ _MODELEVALUATION.fields_by_name["translation_evaluation_metrics"]
+)
+_MODELEVALUATION.fields_by_name[
+ "translation_evaluation_metrics"
+].containing_oneof = _MODELEVALUATION.oneofs_by_name["metrics"]
+_MODELEVALUATION.oneofs_by_name["metrics"].fields.append(
+ _MODELEVALUATION.fields_by_name["image_object_detection_evaluation_metrics"]
+)
+_MODELEVALUATION.fields_by_name[
+ "image_object_detection_evaluation_metrics"
+].containing_oneof = _MODELEVALUATION.oneofs_by_name["metrics"]
+_MODELEVALUATION.oneofs_by_name["metrics"].fields.append(
+ _MODELEVALUATION.fields_by_name["video_object_tracking_evaluation_metrics"]
+)
+_MODELEVALUATION.fields_by_name[
+ "video_object_tracking_evaluation_metrics"
+].containing_oneof = _MODELEVALUATION.oneofs_by_name["metrics"]
+_MODELEVALUATION.oneofs_by_name["metrics"].fields.append(
+ _MODELEVALUATION.fields_by_name["text_sentiment_evaluation_metrics"]
+)
+_MODELEVALUATION.fields_by_name[
+ "text_sentiment_evaluation_metrics"
+].containing_oneof = _MODELEVALUATION.oneofs_by_name["metrics"]
+_MODELEVALUATION.oneofs_by_name["metrics"].fields.append(
+ _MODELEVALUATION.fields_by_name["text_extraction_evaluation_metrics"]
+)
+_MODELEVALUATION.fields_by_name[
+ "text_extraction_evaluation_metrics"
+].containing_oneof = _MODELEVALUATION.oneofs_by_name["metrics"]
+DESCRIPTOR.message_types_by_name["ModelEvaluation"] = _MODELEVALUATION
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+ModelEvaluation = _reflection.GeneratedProtocolMessageType(
+ "ModelEvaluation",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_MODELEVALUATION,
+ __module__="google.cloud.automl_v1beta1.proto.model_evaluation_pb2",
+ __doc__="""Evaluation results of a model.
+
+
+ Attributes:
+ metrics:
+ Output only. Problem type specific evaluation metrics.
+ classification_evaluation_metrics:
+ Model evaluation metrics for image, text, video and tables
+ classification. Tables problem is considered a classification
+ when the target column is CATEGORY DataType.
+ regression_evaluation_metrics:
+ Model evaluation metrics for Tables regression. Tables problem
+ is considered a regression when the target column has FLOAT64
+ DataType.
+ translation_evaluation_metrics:
+ Model evaluation metrics for translation.
+ image_object_detection_evaluation_metrics:
+ Model evaluation metrics for image object detection.
+ video_object_tracking_evaluation_metrics:
+ Model evaluation metrics for video object tracking.
+ text_sentiment_evaluation_metrics:
+ Evaluation metrics for text sentiment models.
+ text_extraction_evaluation_metrics:
+ Evaluation metrics for text extraction models.
+ name:
+ Output only. Resource name of the model evaluation. Format: `
+ `projects/{project_id}/locations/{location_id}/models/{model_i
+ d}/modelEvaluations/{model_evaluation_id}``
+ annotation_spec_id:
+ Output only. The ID of the annotation spec that the model
+ evaluation applies to. The The ID is empty for the overall
+ model evaluation. For Tables annotation specs in the dataset
+ do not exist and this ID is always not set, but for
+ CLASSIFICATION [prediction\_type-s][google.cloud.automl.v1bet
+ a1.TablesModelMetadata.prediction\_type] the [display\_name][g
+ oogle.cloud.automl.v1beta1.ModelEvaluation.display\_name]
+ field is used.
+ display_name:
+ Output only. The value of [display\_name][google.cloud.automl.
+ v1beta1.AnnotationSpec.display\_name] at the moment when the
+ model was trained. Because this field returns a value at model
+ training time, for different models trained from the same
+ dataset, the values may differ, since display names could had
+ been changed between the two model's trainings. For Tables
+ CLASSIFICATION [prediction\_type-s][google.cloud.automl.v1bet
+ a1.TablesModelMetadata.prediction\_type] distinct values of
+ the target column at the moment of the model evaluation are
+ populated here. The display\_name is empty for the overall
+ model evaluation.
+ create_time:
+ Output only. Timestamp when this model evaluation was created.
+ evaluated_example_count:
+ Output only. The number of examples used for model evaluation,
+ i.e. for which ground truth from time of model creation is
+ compared against the predicted annotations created by the
+ model. For overall ModelEvaluation (i.e. with
+ annotation\_spec\_id not set) this is the total number of all
+ examples used for evaluation. Otherwise, this is the count of
+ examples that according to the ground truth were annotated by
+ the [annotation\_spec\_id][google.cloud.automl.v1beta1.ModelE
+ valuation.annotation\_spec\_id].
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ModelEvaluation)
+ ),
+)
+_sym_db.RegisterMessage(ModelEvaluation)
+
+
+DESCRIPTOR._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/automl/google/cloud/automl_v1beta1/proto/model_evaluation_pb2_grpc.py b/automl/google/cloud/automl_v1beta1/proto/model_evaluation_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/model_evaluation_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/automl/google/cloud/automl_v1beta1/proto/model_pb2.py b/automl/google/cloud/automl_v1beta1/proto/model_pb2.py
new file mode 100644
index 000000000000..f669af2061e7
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/model_pb2.py
@@ -0,0 +1,555 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/automl_v1beta1/proto/model.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.cloud.automl_v1beta1.proto import (
+ image_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_image__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ tables_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_tables__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ text_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_text__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ translation_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_translation__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ video_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_video__pb2,
+)
+from google.protobuf import timestamp_pb2 as google_dot_protobuf_dot_timestamp__pb2
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/automl_v1beta1/proto/model.proto",
+ package="google.cloud.automl.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\037com.google.cloud.automl.v1beta1P\001ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\312\002\033Google\\Cloud\\AutoMl\\V1beta1\352\002\036Google::Cloud::AutoML::V1beta1"
+ ),
+ serialized_pb=_b(
+ '\n-google/cloud/automl_v1beta1/proto/model.proto\x12\x1bgoogle.cloud.automl.v1beta1\x1a-google/cloud/automl_v1beta1/proto/image.proto\x1a.google/cloud/automl_v1beta1/proto/tables.proto\x1a,google/cloud/automl_v1beta1/proto/text.proto\x1a\x33google/cloud/automl_v1beta1/proto/translation.proto\x1a-google/cloud/automl_v1beta1/proto/video.proto\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x1cgoogle/api/annotations.proto"\xf2\t\n\x05Model\x12[\n\x1atranslation_model_metadata\x18\x0f \x01(\x0b\x32\x35.google.cloud.automl.v1beta1.TranslationModelMetadataH\x00\x12l\n#image_classification_model_metadata\x18\r \x01(\x0b\x32=.google.cloud.automl.v1beta1.ImageClassificationModelMetadataH\x00\x12j\n"text_classification_model_metadata\x18\x0e \x01(\x0b\x32<.google.cloud.automl.v1beta1.TextClassificationModelMetadataH\x00\x12o\n%image_object_detection_model_metadata\x18\x14 \x01(\x0b\x32>.google.cloud.automl.v1beta1.ImageObjectDetectionModelMetadataH\x00\x12l\n#video_classification_model_metadata\x18\x17 \x01(\x0b\x32=.google.cloud.automl.v1beta1.VideoClassificationModelMetadataH\x00\x12m\n$video_object_tracking_model_metadata\x18\x15 \x01(\x0b\x32=.google.cloud.automl.v1beta1.VideoObjectTrackingModelMetadataH\x00\x12\x62\n\x1etext_extraction_model_metadata\x18\x13 \x01(\x0b\x32\x38.google.cloud.automl.v1beta1.TextExtractionModelMetadataH\x00\x12Q\n\x15tables_model_metadata\x18\x18 \x01(\x0b\x32\x30.google.cloud.automl.v1beta1.TablesModelMetadataH\x00\x12`\n\x1dtext_sentiment_model_metadata\x18\x16 \x01(\x0b\x32\x37.google.cloud.automl.v1beta1.TextSentimentModelMetadataH\x00\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x14\n\x0c\x64isplay_name\x18\x02 \x01(\t\x12\x12\n\ndataset_id\x18\x03 \x01(\t\x12/\n\x0b\x63reate_time\x18\x07 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12/\n\x0bupdate_time\x18\x0b \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12L\n\x10\x64\x65ployment_state\x18\x08 \x01(\x0e\x32\x32.google.cloud.automl.v1beta1.Model.DeploymentState"Q\n\x0f\x44\x65ploymentState\x12 \n\x1c\x44\x45PLOYMENT_STATE_UNSPECIFIED\x10\x00\x12\x0c\n\x08\x44\x45PLOYED\x10\x01\x12\x0e\n\nUNDEPLOYED\x10\x02\x42\x10\n\x0emodel_metadataB\xa5\x01\n\x1f\x63om.google.cloud.automl.v1beta1P\x01ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\xca\x02\x1bGoogle\\Cloud\\AutoMl\\V1beta1\xea\x02\x1eGoogle::Cloud::AutoML::V1beta1b\x06proto3'
+ ),
+ dependencies=[
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_image__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_tables__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_text__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_translation__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_video__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_timestamp__pb2.DESCRIPTOR,
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ ],
+)
+
+
+_MODEL_DEPLOYMENTSTATE = _descriptor.EnumDescriptor(
+ name="DeploymentState",
+ full_name="google.cloud.automl.v1beta1.Model.DeploymentState",
+ filename=None,
+ file=DESCRIPTOR,
+ values=[
+ _descriptor.EnumValueDescriptor(
+ name="DEPLOYMENT_STATE_UNSPECIFIED",
+ index=0,
+ number=0,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="DEPLOYED", index=1, number=1, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="UNDEPLOYED", index=2, number=2, serialized_options=None, type=None
+ ),
+ ],
+ containing_type=None,
+ serialized_options=None,
+ serialized_start=1550,
+ serialized_end=1631,
+)
+_sym_db.RegisterEnumDescriptor(_MODEL_DEPLOYMENTSTATE)
+
+
+_MODEL = _descriptor.Descriptor(
+ name="Model",
+ full_name="google.cloud.automl.v1beta1.Model",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="translation_model_metadata",
+ full_name="google.cloud.automl.v1beta1.Model.translation_model_metadata",
+ index=0,
+ number=15,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="image_classification_model_metadata",
+ full_name="google.cloud.automl.v1beta1.Model.image_classification_model_metadata",
+ index=1,
+ number=13,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="text_classification_model_metadata",
+ full_name="google.cloud.automl.v1beta1.Model.text_classification_model_metadata",
+ index=2,
+ number=14,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="image_object_detection_model_metadata",
+ full_name="google.cloud.automl.v1beta1.Model.image_object_detection_model_metadata",
+ index=3,
+ number=20,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="video_classification_model_metadata",
+ full_name="google.cloud.automl.v1beta1.Model.video_classification_model_metadata",
+ index=4,
+ number=23,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="video_object_tracking_model_metadata",
+ full_name="google.cloud.automl.v1beta1.Model.video_object_tracking_model_metadata",
+ index=5,
+ number=21,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="text_extraction_model_metadata",
+ full_name="google.cloud.automl.v1beta1.Model.text_extraction_model_metadata",
+ index=6,
+ number=19,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="tables_model_metadata",
+ full_name="google.cloud.automl.v1beta1.Model.tables_model_metadata",
+ index=7,
+ number=24,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="text_sentiment_model_metadata",
+ full_name="google.cloud.automl.v1beta1.Model.text_sentiment_model_metadata",
+ index=8,
+ number=22,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.automl.v1beta1.Model.name",
+ index=9,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="display_name",
+ full_name="google.cloud.automl.v1beta1.Model.display_name",
+ index=10,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="dataset_id",
+ full_name="google.cloud.automl.v1beta1.Model.dataset_id",
+ index=11,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="create_time",
+ full_name="google.cloud.automl.v1beta1.Model.create_time",
+ index=12,
+ number=7,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="update_time",
+ full_name="google.cloud.automl.v1beta1.Model.update_time",
+ index=13,
+ number=11,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="deployment_state",
+ full_name="google.cloud.automl.v1beta1.Model.deployment_state",
+ index=14,
+ number=8,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[_MODEL_DEPLOYMENTSTATE],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="model_metadata",
+ full_name="google.cloud.automl.v1beta1.Model.model_metadata",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=383,
+ serialized_end=1649,
+)
+
+_MODEL.fields_by_name[
+ "translation_model_metadata"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_translation__pb2._TRANSLATIONMODELMETADATA
+)
+_MODEL.fields_by_name[
+ "image_classification_model_metadata"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_image__pb2._IMAGECLASSIFICATIONMODELMETADATA
+)
+_MODEL.fields_by_name[
+ "text_classification_model_metadata"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_text__pb2._TEXTCLASSIFICATIONMODELMETADATA
+)
+_MODEL.fields_by_name[
+ "image_object_detection_model_metadata"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_image__pb2._IMAGEOBJECTDETECTIONMODELMETADATA
+)
+_MODEL.fields_by_name[
+ "video_classification_model_metadata"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_video__pb2._VIDEOCLASSIFICATIONMODELMETADATA
+)
+_MODEL.fields_by_name[
+ "video_object_tracking_model_metadata"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_video__pb2._VIDEOOBJECTTRACKINGMODELMETADATA
+)
+_MODEL.fields_by_name[
+ "text_extraction_model_metadata"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_text__pb2._TEXTEXTRACTIONMODELMETADATA
+)
+_MODEL.fields_by_name[
+ "tables_model_metadata"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_tables__pb2._TABLESMODELMETADATA
+)
+_MODEL.fields_by_name[
+ "text_sentiment_model_metadata"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_text__pb2._TEXTSENTIMENTMODELMETADATA
+)
+_MODEL.fields_by_name[
+ "create_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_MODEL.fields_by_name[
+ "update_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_MODEL.fields_by_name["deployment_state"].enum_type = _MODEL_DEPLOYMENTSTATE
+_MODEL_DEPLOYMENTSTATE.containing_type = _MODEL
+_MODEL.oneofs_by_name["model_metadata"].fields.append(
+ _MODEL.fields_by_name["translation_model_metadata"]
+)
+_MODEL.fields_by_name[
+ "translation_model_metadata"
+].containing_oneof = _MODEL.oneofs_by_name["model_metadata"]
+_MODEL.oneofs_by_name["model_metadata"].fields.append(
+ _MODEL.fields_by_name["image_classification_model_metadata"]
+)
+_MODEL.fields_by_name[
+ "image_classification_model_metadata"
+].containing_oneof = _MODEL.oneofs_by_name["model_metadata"]
+_MODEL.oneofs_by_name["model_metadata"].fields.append(
+ _MODEL.fields_by_name["text_classification_model_metadata"]
+)
+_MODEL.fields_by_name[
+ "text_classification_model_metadata"
+].containing_oneof = _MODEL.oneofs_by_name["model_metadata"]
+_MODEL.oneofs_by_name["model_metadata"].fields.append(
+ _MODEL.fields_by_name["image_object_detection_model_metadata"]
+)
+_MODEL.fields_by_name[
+ "image_object_detection_model_metadata"
+].containing_oneof = _MODEL.oneofs_by_name["model_metadata"]
+_MODEL.oneofs_by_name["model_metadata"].fields.append(
+ _MODEL.fields_by_name["video_classification_model_metadata"]
+)
+_MODEL.fields_by_name[
+ "video_classification_model_metadata"
+].containing_oneof = _MODEL.oneofs_by_name["model_metadata"]
+_MODEL.oneofs_by_name["model_metadata"].fields.append(
+ _MODEL.fields_by_name["video_object_tracking_model_metadata"]
+)
+_MODEL.fields_by_name[
+ "video_object_tracking_model_metadata"
+].containing_oneof = _MODEL.oneofs_by_name["model_metadata"]
+_MODEL.oneofs_by_name["model_metadata"].fields.append(
+ _MODEL.fields_by_name["text_extraction_model_metadata"]
+)
+_MODEL.fields_by_name[
+ "text_extraction_model_metadata"
+].containing_oneof = _MODEL.oneofs_by_name["model_metadata"]
+_MODEL.oneofs_by_name["model_metadata"].fields.append(
+ _MODEL.fields_by_name["tables_model_metadata"]
+)
+_MODEL.fields_by_name["tables_model_metadata"].containing_oneof = _MODEL.oneofs_by_name[
+ "model_metadata"
+]
+_MODEL.oneofs_by_name["model_metadata"].fields.append(
+ _MODEL.fields_by_name["text_sentiment_model_metadata"]
+)
+_MODEL.fields_by_name[
+ "text_sentiment_model_metadata"
+].containing_oneof = _MODEL.oneofs_by_name["model_metadata"]
+DESCRIPTOR.message_types_by_name["Model"] = _MODEL
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+Model = _reflection.GeneratedProtocolMessageType(
+ "Model",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_MODEL,
+ __module__="google.cloud.automl_v1beta1.proto.model_pb2",
+ __doc__="""API proto representing a trained machine learning model.
+
+
+ Attributes:
+ model_metadata:
+ Required. The model metadata that is specific to the problem
+ type. Must match the metadata type of the dataset used to
+ train the model.
+ translation_model_metadata:
+ Metadata for translation models.
+ image_classification_model_metadata:
+ Metadata for image classification models.
+ text_classification_model_metadata:
+ Metadata for text classification models.
+ image_object_detection_model_metadata:
+ Metadata for image object detection models.
+ video_classification_model_metadata:
+ Metadata for video classification models.
+ video_object_tracking_model_metadata:
+ Metadata for video object tracking models.
+ text_extraction_model_metadata:
+ Metadata for text extraction models.
+ tables_model_metadata:
+ Metadata for Tables models.
+ text_sentiment_model_metadata:
+ Metadata for text sentiment models.
+ name:
+ Output only. Resource name of the model. Format: ``projects/{p
+ roject_id}/locations/{location_id}/models/{model_id}``
+ display_name:
+ Required. The name of the model to show in the interface. The
+ name can be up to 32 characters long and can consist only of
+ ASCII Latin letters A-Z and a-z, underscores (\_), and ASCII
+ digits 0-9. It must start with a letter.
+ dataset_id:
+ Required. The resource ID of the dataset used to create the
+ model. The dataset must come from the same ancestor project
+ and location.
+ create_time:
+ Output only. Timestamp when the model training finished and
+ can be used for prediction.
+ update_time:
+ Output only. Timestamp when this model was last updated.
+ deployment_state:
+ Output only. Deployment state of the model. A model can only
+ serve prediction requests after it gets deployed.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.Model)
+ ),
+)
+_sym_db.RegisterMessage(Model)
+
+
+DESCRIPTOR._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/automl/google/cloud/automl_v1beta1/proto/model_pb2_grpc.py b/automl/google/cloud/automl_v1beta1/proto/model_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/model_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/automl/google/cloud/automl_v1beta1/proto/operations.proto b/automl/google/cloud/automl_v1beta1/proto/operations.proto
new file mode 100644
index 000000000000..aba18cd6715c
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/operations.proto
@@ -0,0 +1,190 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.automl.v1beta1;
+
+import "google/cloud/automl/v1beta1/io.proto";
+import "google/cloud/automl/v1beta1/model.proto";
+import "google/cloud/automl/v1beta1/model_evaluation.proto";
+import "google/protobuf/empty.proto";
+import "google/protobuf/timestamp.proto";
+import "google/rpc/status.proto";
+import "google/api/annotations.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl";
+option java_multiple_files = true;
+option java_package = "com.google.cloud.automl.v1beta1";
+option php_namespace = "Google\\Cloud\\AutoMl\\V1beta1";
+option ruby_package = "Google::Cloud::AutoML::V1beta1";
+
+// Metadata used across all long running operations returned by AutoML API.
+message OperationMetadata {
+ // Ouptut only. Details of specific operation. Even if this field is empty,
+ // the presence allows to distinguish different types of operations.
+ oneof details {
+ // Details of a Delete operation.
+ DeleteOperationMetadata delete_details = 8;
+
+ // Details of a DeployModel operation.
+ DeployModelOperationMetadata deploy_model_details = 24;
+
+ // Details of an UndeployModel operation.
+ UndeployModelOperationMetadata undeploy_model_details = 25;
+
+ // Details of CreateModel operation.
+ CreateModelOperationMetadata create_model_details = 10;
+
+ // Details of ImportData operation.
+ ImportDataOperationMetadata import_data_details = 15;
+
+ // Details of BatchPredict operation.
+ BatchPredictOperationMetadata batch_predict_details = 16;
+
+ // Details of ExportData operation.
+ ExportDataOperationMetadata export_data_details = 21;
+
+ // Details of ExportModel operation.
+ ExportModelOperationMetadata export_model_details = 22;
+
+ // Details of ExportEvaluatedExamples operation.
+ ExportEvaluatedExamplesOperationMetadata export_evaluated_examples_details = 26;
+ }
+
+ // Output only. Progress of operation. Range: [0, 100].
+ // Not used currently.
+ int32 progress_percent = 13;
+
+ // Output only. Partial failures encountered.
+ // E.g. single files that couldn't be read.
+ // This field should never exceed 20 entries.
+ // Status details field will contain standard GCP error details.
+ repeated google.rpc.Status partial_failures = 2;
+
+ // Output only. Time when the operation was created.
+ google.protobuf.Timestamp create_time = 3;
+
+ // Output only. Time when the operation was updated for the last time.
+ google.protobuf.Timestamp update_time = 4;
+}
+
+// Details of operations that perform deletes of any entities.
+message DeleteOperationMetadata {
+
+}
+
+// Details of DeployModel operation.
+message DeployModelOperationMetadata {
+
+}
+
+// Details of UndeployModel operation.
+message UndeployModelOperationMetadata {
+
+}
+
+// Details of CreateModel operation.
+message CreateModelOperationMetadata {
+
+}
+
+// Details of ImportData operation.
+message ImportDataOperationMetadata {
+
+}
+
+// Details of ExportData operation.
+message ExportDataOperationMetadata {
+ // Further describes this export data's output.
+ // Supplements
+ // [OutputConfig][google.cloud.automl.v1beta1.OutputConfig].
+ message ExportDataOutputInfo {
+ // The output location to which the exported data is written.
+ oneof output_location {
+ // The full path of the Google Cloud Storage directory created, into which
+ // the exported data is written.
+ string gcs_output_directory = 1;
+
+ // The path of the BigQuery dataset created, in bq://projectId.bqDatasetId
+ // format, into which the exported data is written.
+ string bigquery_output_dataset = 2;
+ }
+ }
+
+ // Output only. Information further describing this export data's output.
+ ExportDataOutputInfo output_info = 1;
+}
+
+// Details of BatchPredict operation.
+message BatchPredictOperationMetadata {
+ // Further describes this batch predict's output.
+ // Supplements
+ //
+ // [BatchPredictOutputConfig][google.cloud.automl.v1beta1.BatchPredictOutputConfig].
+ message BatchPredictOutputInfo {
+ // The output location into which prediction output is written.
+ oneof output_location {
+ // The full path of the Google Cloud Storage directory created, into which
+ // the prediction output is written.
+ string gcs_output_directory = 1;
+
+ // The path of the BigQuery dataset created, in bq://projectId.bqDatasetId
+ // format, into which the prediction output is written.
+ string bigquery_output_dataset = 2;
+ }
+ }
+
+ // Output only. The input config that was given upon starting this
+ // batch predict operation.
+ BatchPredictInputConfig input_config = 1;
+
+ // Output only. Information further describing this batch predict's output.
+ BatchPredictOutputInfo output_info = 2;
+}
+
+// Details of ExportModel operation.
+message ExportModelOperationMetadata {
+ // Further describes the output of model export.
+ // Supplements
+ //
+ // [ModelExportOutputConfig][google.cloud.automl.v1beta1.ModelExportOutputConfig].
+ message ExportModelOutputInfo {
+ // The full path of the Google Cloud Storage directory created, into which
+ // the model will be exported.
+ string gcs_output_directory = 1;
+ }
+
+ // Output only. Information further describing the output of this model
+ // export.
+ ExportModelOutputInfo output_info = 2;
+}
+
+// Details of EvaluatedExamples operation.
+message ExportEvaluatedExamplesOperationMetadata {
+ // Further describes the output of the evaluated examples export.
+ // Supplements
+ //
+ // [ExportEvaluatedExamplesOutputConfig][google.cloud.automl.v1beta1.ExportEvaluatedExamplesOutputConfig].
+ message ExportEvaluatedExamplesOutputInfo {
+ // The path of the BigQuery dataset created, in bq://projectId.bqDatasetId
+ // format, into which the output of export evaluated examples is written.
+ string bigquery_output_dataset = 2;
+ }
+
+ // Output only. Information further describing the output of this evaluated
+ // examples export.
+ ExportEvaluatedExamplesOutputInfo output_info = 2;
+}
diff --git a/automl/google/cloud/automl_v1beta1/proto/operations_pb2.py b/automl/google/cloud/automl_v1beta1/proto/operations_pb2.py
new file mode 100644
index 000000000000..b6e3f132684d
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/operations_pb2.py
@@ -0,0 +1,1286 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/automl_v1beta1/proto/operations.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.cloud.automl_v1beta1.proto import (
+ io_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_io__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ model_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_model__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ model_evaluation_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_model__evaluation__pb2,
+)
+from google.protobuf import empty_pb2 as google_dot_protobuf_dot_empty__pb2
+from google.protobuf import timestamp_pb2 as google_dot_protobuf_dot_timestamp__pb2
+from google.rpc import status_pb2 as google_dot_rpc_dot_status__pb2
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/automl_v1beta1/proto/operations.proto",
+ package="google.cloud.automl.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\037com.google.cloud.automl.v1beta1P\001ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\312\002\033Google\\Cloud\\AutoMl\\V1beta1\352\002\036Google::Cloud::AutoML::V1beta1"
+ ),
+ serialized_pb=_b(
+ '\n2google/cloud/automl_v1beta1/proto/operations.proto\x12\x1bgoogle.cloud.automl.v1beta1\x1a*google/cloud/automl_v1beta1/proto/io.proto\x1a-google/cloud/automl_v1beta1/proto/model.proto\x1a\x38google/cloud/automl_v1beta1/proto/model_evaluation.proto\x1a\x1bgoogle/protobuf/empty.proto\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x17google/rpc/status.proto\x1a\x1cgoogle/api/annotations.proto"\x8b\x08\n\x11OperationMetadata\x12N\n\x0e\x64\x65lete_details\x18\x08 \x01(\x0b\x32\x34.google.cloud.automl.v1beta1.DeleteOperationMetadataH\x00\x12Y\n\x14\x64\x65ploy_model_details\x18\x18 \x01(\x0b\x32\x39.google.cloud.automl.v1beta1.DeployModelOperationMetadataH\x00\x12]\n\x16undeploy_model_details\x18\x19 \x01(\x0b\x32;.google.cloud.automl.v1beta1.UndeployModelOperationMetadataH\x00\x12Y\n\x14\x63reate_model_details\x18\n \x01(\x0b\x32\x39.google.cloud.automl.v1beta1.CreateModelOperationMetadataH\x00\x12W\n\x13import_data_details\x18\x0f \x01(\x0b\x32\x38.google.cloud.automl.v1beta1.ImportDataOperationMetadataH\x00\x12[\n\x15\x62\x61tch_predict_details\x18\x10 \x01(\x0b\x32:.google.cloud.automl.v1beta1.BatchPredictOperationMetadataH\x00\x12W\n\x13\x65xport_data_details\x18\x15 \x01(\x0b\x32\x38.google.cloud.automl.v1beta1.ExportDataOperationMetadataH\x00\x12Y\n\x14\x65xport_model_details\x18\x16 \x01(\x0b\x32\x39.google.cloud.automl.v1beta1.ExportModelOperationMetadataH\x00\x12r\n!export_evaluated_examples_details\x18\x1a \x01(\x0b\x32\x45.google.cloud.automl.v1beta1.ExportEvaluatedExamplesOperationMetadataH\x00\x12\x18\n\x10progress_percent\x18\r \x01(\x05\x12,\n\x10partial_failures\x18\x02 \x03(\x0b\x32\x12.google.rpc.Status\x12/\n\x0b\x63reate_time\x18\x03 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12/\n\x0bupdate_time\x18\x04 \x01(\x0b\x32\x1a.google.protobuf.TimestampB\t\n\x07\x64\x65tails"\x19\n\x17\x44\x65leteOperationMetadata"\x1e\n\x1c\x44\x65ployModelOperationMetadata" \n\x1eUndeployModelOperationMetadata"\x1e\n\x1c\x43reateModelOperationMetadata"\x1d\n\x1bImportDataOperationMetadata"\xef\x01\n\x1b\x45xportDataOperationMetadata\x12\x62\n\x0boutput_info\x18\x01 \x01(\x0b\x32M.google.cloud.automl.v1beta1.ExportDataOperationMetadata.ExportDataOutputInfo\x1al\n\x14\x45xportDataOutputInfo\x12\x1e\n\x14gcs_output_directory\x18\x01 \x01(\tH\x00\x12!\n\x17\x62igquery_output_dataset\x18\x02 \x01(\tH\x00\x42\x11\n\x0foutput_location"\xc3\x02\n\x1d\x42\x61tchPredictOperationMetadata\x12J\n\x0cinput_config\x18\x01 \x01(\x0b\x32\x34.google.cloud.automl.v1beta1.BatchPredictInputConfig\x12\x66\n\x0boutput_info\x18\x02 \x01(\x0b\x32Q.google.cloud.automl.v1beta1.BatchPredictOperationMetadata.BatchPredictOutputInfo\x1an\n\x16\x42\x61tchPredictOutputInfo\x12\x1e\n\x14gcs_output_directory\x18\x01 \x01(\tH\x00\x12!\n\x17\x62igquery_output_dataset\x18\x02 \x01(\tH\x00\x42\x11\n\x0foutput_location"\xbb\x01\n\x1c\x45xportModelOperationMetadata\x12\x64\n\x0boutput_info\x18\x02 \x01(\x0b\x32O.google.cloud.automl.v1beta1.ExportModelOperationMetadata.ExportModelOutputInfo\x1a\x35\n\x15\x45xportModelOutputInfo\x12\x1c\n\x14gcs_output_directory\x18\x01 \x01(\t"\xee\x01\n(ExportEvaluatedExamplesOperationMetadata\x12|\n\x0boutput_info\x18\x02 \x01(\x0b\x32g.google.cloud.automl.v1beta1.ExportEvaluatedExamplesOperationMetadata.ExportEvaluatedExamplesOutputInfo\x1a\x44\n!ExportEvaluatedExamplesOutputInfo\x12\x1f\n\x17\x62igquery_output_dataset\x18\x02 \x01(\tB\xa5\x01\n\x1f\x63om.google.cloud.automl.v1beta1P\x01ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\xca\x02\x1bGoogle\\Cloud\\AutoMl\\V1beta1\xea\x02\x1eGoogle::Cloud::AutoML::V1beta1b\x06proto3'
+ ),
+ dependencies=[
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_io__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_model__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_model__evaluation__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_empty__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_timestamp__pb2.DESCRIPTOR,
+ google_dot_rpc_dot_status__pb2.DESCRIPTOR,
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ ],
+)
+
+
+_OPERATIONMETADATA = _descriptor.Descriptor(
+ name="OperationMetadata",
+ full_name="google.cloud.automl.v1beta1.OperationMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="delete_details",
+ full_name="google.cloud.automl.v1beta1.OperationMetadata.delete_details",
+ index=0,
+ number=8,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="deploy_model_details",
+ full_name="google.cloud.automl.v1beta1.OperationMetadata.deploy_model_details",
+ index=1,
+ number=24,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="undeploy_model_details",
+ full_name="google.cloud.automl.v1beta1.OperationMetadata.undeploy_model_details",
+ index=2,
+ number=25,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="create_model_details",
+ full_name="google.cloud.automl.v1beta1.OperationMetadata.create_model_details",
+ index=3,
+ number=10,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="import_data_details",
+ full_name="google.cloud.automl.v1beta1.OperationMetadata.import_data_details",
+ index=4,
+ number=15,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="batch_predict_details",
+ full_name="google.cloud.automl.v1beta1.OperationMetadata.batch_predict_details",
+ index=5,
+ number=16,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="export_data_details",
+ full_name="google.cloud.automl.v1beta1.OperationMetadata.export_data_details",
+ index=6,
+ number=21,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="export_model_details",
+ full_name="google.cloud.automl.v1beta1.OperationMetadata.export_model_details",
+ index=7,
+ number=22,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="export_evaluated_examples_details",
+ full_name="google.cloud.automl.v1beta1.OperationMetadata.export_evaluated_examples_details",
+ index=8,
+ number=26,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="progress_percent",
+ full_name="google.cloud.automl.v1beta1.OperationMetadata.progress_percent",
+ index=9,
+ number=13,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="partial_failures",
+ full_name="google.cloud.automl.v1beta1.OperationMetadata.partial_failures",
+ index=10,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="create_time",
+ full_name="google.cloud.automl.v1beta1.OperationMetadata.create_time",
+ index=11,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="update_time",
+ full_name="google.cloud.automl.v1beta1.OperationMetadata.update_time",
+ index=12,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="details",
+ full_name="google.cloud.automl.v1beta1.OperationMetadata.details",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=350,
+ serialized_end=1385,
+)
+
+
+_DELETEOPERATIONMETADATA = _descriptor.Descriptor(
+ name="DeleteOperationMetadata",
+ full_name="google.cloud.automl.v1beta1.DeleteOperationMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1387,
+ serialized_end=1412,
+)
+
+
+_DEPLOYMODELOPERATIONMETADATA = _descriptor.Descriptor(
+ name="DeployModelOperationMetadata",
+ full_name="google.cloud.automl.v1beta1.DeployModelOperationMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1414,
+ serialized_end=1444,
+)
+
+
+_UNDEPLOYMODELOPERATIONMETADATA = _descriptor.Descriptor(
+ name="UndeployModelOperationMetadata",
+ full_name="google.cloud.automl.v1beta1.UndeployModelOperationMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1446,
+ serialized_end=1478,
+)
+
+
+_CREATEMODELOPERATIONMETADATA = _descriptor.Descriptor(
+ name="CreateModelOperationMetadata",
+ full_name="google.cloud.automl.v1beta1.CreateModelOperationMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1480,
+ serialized_end=1510,
+)
+
+
+_IMPORTDATAOPERATIONMETADATA = _descriptor.Descriptor(
+ name="ImportDataOperationMetadata",
+ full_name="google.cloud.automl.v1beta1.ImportDataOperationMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1512,
+ serialized_end=1541,
+)
+
+
+_EXPORTDATAOPERATIONMETADATA_EXPORTDATAOUTPUTINFO = _descriptor.Descriptor(
+ name="ExportDataOutputInfo",
+ full_name="google.cloud.automl.v1beta1.ExportDataOperationMetadata.ExportDataOutputInfo",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="gcs_output_directory",
+ full_name="google.cloud.automl.v1beta1.ExportDataOperationMetadata.ExportDataOutputInfo.gcs_output_directory",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="bigquery_output_dataset",
+ full_name="google.cloud.automl.v1beta1.ExportDataOperationMetadata.ExportDataOutputInfo.bigquery_output_dataset",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="output_location",
+ full_name="google.cloud.automl.v1beta1.ExportDataOperationMetadata.ExportDataOutputInfo.output_location",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=1675,
+ serialized_end=1783,
+)
+
+_EXPORTDATAOPERATIONMETADATA = _descriptor.Descriptor(
+ name="ExportDataOperationMetadata",
+ full_name="google.cloud.automl.v1beta1.ExportDataOperationMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="output_info",
+ full_name="google.cloud.automl.v1beta1.ExportDataOperationMetadata.output_info",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[_EXPORTDATAOPERATIONMETADATA_EXPORTDATAOUTPUTINFO],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1544,
+ serialized_end=1783,
+)
+
+
+_BATCHPREDICTOPERATIONMETADATA_BATCHPREDICTOUTPUTINFO = _descriptor.Descriptor(
+ name="BatchPredictOutputInfo",
+ full_name="google.cloud.automl.v1beta1.BatchPredictOperationMetadata.BatchPredictOutputInfo",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="gcs_output_directory",
+ full_name="google.cloud.automl.v1beta1.BatchPredictOperationMetadata.BatchPredictOutputInfo.gcs_output_directory",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="bigquery_output_dataset",
+ full_name="google.cloud.automl.v1beta1.BatchPredictOperationMetadata.BatchPredictOutputInfo.bigquery_output_dataset",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="output_location",
+ full_name="google.cloud.automl.v1beta1.BatchPredictOperationMetadata.BatchPredictOutputInfo.output_location",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=1999,
+ serialized_end=2109,
+)
+
+_BATCHPREDICTOPERATIONMETADATA = _descriptor.Descriptor(
+ name="BatchPredictOperationMetadata",
+ full_name="google.cloud.automl.v1beta1.BatchPredictOperationMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="input_config",
+ full_name="google.cloud.automl.v1beta1.BatchPredictOperationMetadata.input_config",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="output_info",
+ full_name="google.cloud.automl.v1beta1.BatchPredictOperationMetadata.output_info",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[_BATCHPREDICTOPERATIONMETADATA_BATCHPREDICTOUTPUTINFO],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1786,
+ serialized_end=2109,
+)
+
+
+_EXPORTMODELOPERATIONMETADATA_EXPORTMODELOUTPUTINFO = _descriptor.Descriptor(
+ name="ExportModelOutputInfo",
+ full_name="google.cloud.automl.v1beta1.ExportModelOperationMetadata.ExportModelOutputInfo",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="gcs_output_directory",
+ full_name="google.cloud.automl.v1beta1.ExportModelOperationMetadata.ExportModelOutputInfo.gcs_output_directory",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2246,
+ serialized_end=2299,
+)
+
+_EXPORTMODELOPERATIONMETADATA = _descriptor.Descriptor(
+ name="ExportModelOperationMetadata",
+ full_name="google.cloud.automl.v1beta1.ExportModelOperationMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="output_info",
+ full_name="google.cloud.automl.v1beta1.ExportModelOperationMetadata.output_info",
+ index=0,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[_EXPORTMODELOPERATIONMETADATA_EXPORTMODELOUTPUTINFO],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2112,
+ serialized_end=2299,
+)
+
+
+_EXPORTEVALUATEDEXAMPLESOPERATIONMETADATA_EXPORTEVALUATEDEXAMPLESOUTPUTINFO = _descriptor.Descriptor(
+ name="ExportEvaluatedExamplesOutputInfo",
+ full_name="google.cloud.automl.v1beta1.ExportEvaluatedExamplesOperationMetadata.ExportEvaluatedExamplesOutputInfo",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="bigquery_output_dataset",
+ full_name="google.cloud.automl.v1beta1.ExportEvaluatedExamplesOperationMetadata.ExportEvaluatedExamplesOutputInfo.bigquery_output_dataset",
+ index=0,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2472,
+ serialized_end=2540,
+)
+
+_EXPORTEVALUATEDEXAMPLESOPERATIONMETADATA = _descriptor.Descriptor(
+ name="ExportEvaluatedExamplesOperationMetadata",
+ full_name="google.cloud.automl.v1beta1.ExportEvaluatedExamplesOperationMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="output_info",
+ full_name="google.cloud.automl.v1beta1.ExportEvaluatedExamplesOperationMetadata.output_info",
+ index=0,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[
+ _EXPORTEVALUATEDEXAMPLESOPERATIONMETADATA_EXPORTEVALUATEDEXAMPLESOUTPUTINFO
+ ],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2302,
+ serialized_end=2540,
+)
+
+_OPERATIONMETADATA.fields_by_name[
+ "delete_details"
+].message_type = _DELETEOPERATIONMETADATA
+_OPERATIONMETADATA.fields_by_name[
+ "deploy_model_details"
+].message_type = _DEPLOYMODELOPERATIONMETADATA
+_OPERATIONMETADATA.fields_by_name[
+ "undeploy_model_details"
+].message_type = _UNDEPLOYMODELOPERATIONMETADATA
+_OPERATIONMETADATA.fields_by_name[
+ "create_model_details"
+].message_type = _CREATEMODELOPERATIONMETADATA
+_OPERATIONMETADATA.fields_by_name[
+ "import_data_details"
+].message_type = _IMPORTDATAOPERATIONMETADATA
+_OPERATIONMETADATA.fields_by_name[
+ "batch_predict_details"
+].message_type = _BATCHPREDICTOPERATIONMETADATA
+_OPERATIONMETADATA.fields_by_name[
+ "export_data_details"
+].message_type = _EXPORTDATAOPERATIONMETADATA
+_OPERATIONMETADATA.fields_by_name[
+ "export_model_details"
+].message_type = _EXPORTMODELOPERATIONMETADATA
+_OPERATIONMETADATA.fields_by_name[
+ "export_evaluated_examples_details"
+].message_type = _EXPORTEVALUATEDEXAMPLESOPERATIONMETADATA
+_OPERATIONMETADATA.fields_by_name[
+ "partial_failures"
+].message_type = google_dot_rpc_dot_status__pb2._STATUS
+_OPERATIONMETADATA.fields_by_name[
+ "create_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_OPERATIONMETADATA.fields_by_name[
+ "update_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_OPERATIONMETADATA.oneofs_by_name["details"].fields.append(
+ _OPERATIONMETADATA.fields_by_name["delete_details"]
+)
+_OPERATIONMETADATA.fields_by_name[
+ "delete_details"
+].containing_oneof = _OPERATIONMETADATA.oneofs_by_name["details"]
+_OPERATIONMETADATA.oneofs_by_name["details"].fields.append(
+ _OPERATIONMETADATA.fields_by_name["deploy_model_details"]
+)
+_OPERATIONMETADATA.fields_by_name[
+ "deploy_model_details"
+].containing_oneof = _OPERATIONMETADATA.oneofs_by_name["details"]
+_OPERATIONMETADATA.oneofs_by_name["details"].fields.append(
+ _OPERATIONMETADATA.fields_by_name["undeploy_model_details"]
+)
+_OPERATIONMETADATA.fields_by_name[
+ "undeploy_model_details"
+].containing_oneof = _OPERATIONMETADATA.oneofs_by_name["details"]
+_OPERATIONMETADATA.oneofs_by_name["details"].fields.append(
+ _OPERATIONMETADATA.fields_by_name["create_model_details"]
+)
+_OPERATIONMETADATA.fields_by_name[
+ "create_model_details"
+].containing_oneof = _OPERATIONMETADATA.oneofs_by_name["details"]
+_OPERATIONMETADATA.oneofs_by_name["details"].fields.append(
+ _OPERATIONMETADATA.fields_by_name["import_data_details"]
+)
+_OPERATIONMETADATA.fields_by_name[
+ "import_data_details"
+].containing_oneof = _OPERATIONMETADATA.oneofs_by_name["details"]
+_OPERATIONMETADATA.oneofs_by_name["details"].fields.append(
+ _OPERATIONMETADATA.fields_by_name["batch_predict_details"]
+)
+_OPERATIONMETADATA.fields_by_name[
+ "batch_predict_details"
+].containing_oneof = _OPERATIONMETADATA.oneofs_by_name["details"]
+_OPERATIONMETADATA.oneofs_by_name["details"].fields.append(
+ _OPERATIONMETADATA.fields_by_name["export_data_details"]
+)
+_OPERATIONMETADATA.fields_by_name[
+ "export_data_details"
+].containing_oneof = _OPERATIONMETADATA.oneofs_by_name["details"]
+_OPERATIONMETADATA.oneofs_by_name["details"].fields.append(
+ _OPERATIONMETADATA.fields_by_name["export_model_details"]
+)
+_OPERATIONMETADATA.fields_by_name[
+ "export_model_details"
+].containing_oneof = _OPERATIONMETADATA.oneofs_by_name["details"]
+_OPERATIONMETADATA.oneofs_by_name["details"].fields.append(
+ _OPERATIONMETADATA.fields_by_name["export_evaluated_examples_details"]
+)
+_OPERATIONMETADATA.fields_by_name[
+ "export_evaluated_examples_details"
+].containing_oneof = _OPERATIONMETADATA.oneofs_by_name["details"]
+_EXPORTDATAOPERATIONMETADATA_EXPORTDATAOUTPUTINFO.containing_type = (
+ _EXPORTDATAOPERATIONMETADATA
+)
+_EXPORTDATAOPERATIONMETADATA_EXPORTDATAOUTPUTINFO.oneofs_by_name[
+ "output_location"
+].fields.append(
+ _EXPORTDATAOPERATIONMETADATA_EXPORTDATAOUTPUTINFO.fields_by_name[
+ "gcs_output_directory"
+ ]
+)
+_EXPORTDATAOPERATIONMETADATA_EXPORTDATAOUTPUTINFO.fields_by_name[
+ "gcs_output_directory"
+].containing_oneof = _EXPORTDATAOPERATIONMETADATA_EXPORTDATAOUTPUTINFO.oneofs_by_name[
+ "output_location"
+]
+_EXPORTDATAOPERATIONMETADATA_EXPORTDATAOUTPUTINFO.oneofs_by_name[
+ "output_location"
+].fields.append(
+ _EXPORTDATAOPERATIONMETADATA_EXPORTDATAOUTPUTINFO.fields_by_name[
+ "bigquery_output_dataset"
+ ]
+)
+_EXPORTDATAOPERATIONMETADATA_EXPORTDATAOUTPUTINFO.fields_by_name[
+ "bigquery_output_dataset"
+].containing_oneof = _EXPORTDATAOPERATIONMETADATA_EXPORTDATAOUTPUTINFO.oneofs_by_name[
+ "output_location"
+]
+_EXPORTDATAOPERATIONMETADATA.fields_by_name[
+ "output_info"
+].message_type = _EXPORTDATAOPERATIONMETADATA_EXPORTDATAOUTPUTINFO
+_BATCHPREDICTOPERATIONMETADATA_BATCHPREDICTOUTPUTINFO.containing_type = (
+ _BATCHPREDICTOPERATIONMETADATA
+)
+_BATCHPREDICTOPERATIONMETADATA_BATCHPREDICTOUTPUTINFO.oneofs_by_name[
+ "output_location"
+].fields.append(
+ _BATCHPREDICTOPERATIONMETADATA_BATCHPREDICTOUTPUTINFO.fields_by_name[
+ "gcs_output_directory"
+ ]
+)
+_BATCHPREDICTOPERATIONMETADATA_BATCHPREDICTOUTPUTINFO.fields_by_name[
+ "gcs_output_directory"
+].containing_oneof = _BATCHPREDICTOPERATIONMETADATA_BATCHPREDICTOUTPUTINFO.oneofs_by_name[
+ "output_location"
+]
+_BATCHPREDICTOPERATIONMETADATA_BATCHPREDICTOUTPUTINFO.oneofs_by_name[
+ "output_location"
+].fields.append(
+ _BATCHPREDICTOPERATIONMETADATA_BATCHPREDICTOUTPUTINFO.fields_by_name[
+ "bigquery_output_dataset"
+ ]
+)
+_BATCHPREDICTOPERATIONMETADATA_BATCHPREDICTOUTPUTINFO.fields_by_name[
+ "bigquery_output_dataset"
+].containing_oneof = _BATCHPREDICTOPERATIONMETADATA_BATCHPREDICTOUTPUTINFO.oneofs_by_name[
+ "output_location"
+]
+_BATCHPREDICTOPERATIONMETADATA.fields_by_name[
+ "input_config"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_io__pb2._BATCHPREDICTINPUTCONFIG
+)
+_BATCHPREDICTOPERATIONMETADATA.fields_by_name[
+ "output_info"
+].message_type = _BATCHPREDICTOPERATIONMETADATA_BATCHPREDICTOUTPUTINFO
+_EXPORTMODELOPERATIONMETADATA_EXPORTMODELOUTPUTINFO.containing_type = (
+ _EXPORTMODELOPERATIONMETADATA
+)
+_EXPORTMODELOPERATIONMETADATA.fields_by_name[
+ "output_info"
+].message_type = _EXPORTMODELOPERATIONMETADATA_EXPORTMODELOUTPUTINFO
+_EXPORTEVALUATEDEXAMPLESOPERATIONMETADATA_EXPORTEVALUATEDEXAMPLESOUTPUTINFO.containing_type = (
+ _EXPORTEVALUATEDEXAMPLESOPERATIONMETADATA
+)
+_EXPORTEVALUATEDEXAMPLESOPERATIONMETADATA.fields_by_name[
+ "output_info"
+].message_type = (
+ _EXPORTEVALUATEDEXAMPLESOPERATIONMETADATA_EXPORTEVALUATEDEXAMPLESOUTPUTINFO
+)
+DESCRIPTOR.message_types_by_name["OperationMetadata"] = _OPERATIONMETADATA
+DESCRIPTOR.message_types_by_name["DeleteOperationMetadata"] = _DELETEOPERATIONMETADATA
+DESCRIPTOR.message_types_by_name[
+ "DeployModelOperationMetadata"
+] = _DEPLOYMODELOPERATIONMETADATA
+DESCRIPTOR.message_types_by_name[
+ "UndeployModelOperationMetadata"
+] = _UNDEPLOYMODELOPERATIONMETADATA
+DESCRIPTOR.message_types_by_name[
+ "CreateModelOperationMetadata"
+] = _CREATEMODELOPERATIONMETADATA
+DESCRIPTOR.message_types_by_name[
+ "ImportDataOperationMetadata"
+] = _IMPORTDATAOPERATIONMETADATA
+DESCRIPTOR.message_types_by_name[
+ "ExportDataOperationMetadata"
+] = _EXPORTDATAOPERATIONMETADATA
+DESCRIPTOR.message_types_by_name[
+ "BatchPredictOperationMetadata"
+] = _BATCHPREDICTOPERATIONMETADATA
+DESCRIPTOR.message_types_by_name[
+ "ExportModelOperationMetadata"
+] = _EXPORTMODELOPERATIONMETADATA
+DESCRIPTOR.message_types_by_name[
+ "ExportEvaluatedExamplesOperationMetadata"
+] = _EXPORTEVALUATEDEXAMPLESOPERATIONMETADATA
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+OperationMetadata = _reflection.GeneratedProtocolMessageType(
+ "OperationMetadata",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_OPERATIONMETADATA,
+ __module__="google.cloud.automl_v1beta1.proto.operations_pb2",
+ __doc__="""Metadata used across all long running operations returned by AutoML API.
+
+
+ Attributes:
+ details:
+ Ouptut only. Details of specific operation. Even if this field
+ is empty, the presence allows to distinguish different types
+ of operations.
+ delete_details:
+ Details of a Delete operation.
+ deploy_model_details:
+ Details of a DeployModel operation.
+ undeploy_model_details:
+ Details of an UndeployModel operation.
+ create_model_details:
+ Details of CreateModel operation.
+ import_data_details:
+ Details of ImportData operation.
+ batch_predict_details:
+ Details of BatchPredict operation.
+ export_data_details:
+ Details of ExportData operation.
+ export_model_details:
+ Details of ExportModel operation.
+ export_evaluated_examples_details:
+ Details of ExportEvaluatedExamples operation.
+ progress_percent:
+ Output only. Progress of operation. Range: [0, 100]. Not used
+ currently.
+ partial_failures:
+ Output only. Partial failures encountered. E.g. single files
+ that couldn't be read. This field should never exceed 20
+ entries. Status details field will contain standard GCP error
+ details.
+ create_time:
+ Output only. Time when the operation was created.
+ update_time:
+ Output only. Time when the operation was updated for the last
+ time.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.OperationMetadata)
+ ),
+)
+_sym_db.RegisterMessage(OperationMetadata)
+
+DeleteOperationMetadata = _reflection.GeneratedProtocolMessageType(
+ "DeleteOperationMetadata",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_DELETEOPERATIONMETADATA,
+ __module__="google.cloud.automl_v1beta1.proto.operations_pb2",
+ __doc__="""Details of operations that perform deletes of any entities.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.DeleteOperationMetadata)
+ ),
+)
+_sym_db.RegisterMessage(DeleteOperationMetadata)
+
+DeployModelOperationMetadata = _reflection.GeneratedProtocolMessageType(
+ "DeployModelOperationMetadata",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_DEPLOYMODELOPERATIONMETADATA,
+ __module__="google.cloud.automl_v1beta1.proto.operations_pb2",
+ __doc__="""Details of DeployModel operation.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.DeployModelOperationMetadata)
+ ),
+)
+_sym_db.RegisterMessage(DeployModelOperationMetadata)
+
+UndeployModelOperationMetadata = _reflection.GeneratedProtocolMessageType(
+ "UndeployModelOperationMetadata",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_UNDEPLOYMODELOPERATIONMETADATA,
+ __module__="google.cloud.automl_v1beta1.proto.operations_pb2",
+ __doc__="""Details of UndeployModel operation.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.UndeployModelOperationMetadata)
+ ),
+)
+_sym_db.RegisterMessage(UndeployModelOperationMetadata)
+
+CreateModelOperationMetadata = _reflection.GeneratedProtocolMessageType(
+ "CreateModelOperationMetadata",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_CREATEMODELOPERATIONMETADATA,
+ __module__="google.cloud.automl_v1beta1.proto.operations_pb2",
+ __doc__="""Details of CreateModel operation.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.CreateModelOperationMetadata)
+ ),
+)
+_sym_db.RegisterMessage(CreateModelOperationMetadata)
+
+ImportDataOperationMetadata = _reflection.GeneratedProtocolMessageType(
+ "ImportDataOperationMetadata",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_IMPORTDATAOPERATIONMETADATA,
+ __module__="google.cloud.automl_v1beta1.proto.operations_pb2",
+ __doc__="""Details of ImportData operation.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ImportDataOperationMetadata)
+ ),
+)
+_sym_db.RegisterMessage(ImportDataOperationMetadata)
+
+ExportDataOperationMetadata = _reflection.GeneratedProtocolMessageType(
+ "ExportDataOperationMetadata",
+ (_message.Message,),
+ dict(
+ ExportDataOutputInfo=_reflection.GeneratedProtocolMessageType(
+ "ExportDataOutputInfo",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_EXPORTDATAOPERATIONMETADATA_EXPORTDATAOUTPUTINFO,
+ __module__="google.cloud.automl_v1beta1.proto.operations_pb2",
+ __doc__="""Further describes this export data's output. Supplements
+ [OutputConfig][google.cloud.automl.v1beta1.OutputConfig].
+
+
+ Attributes:
+ output_location:
+ The output location to which the exported data is written.
+ gcs_output_directory:
+ The full path of the Google Cloud Storage directory created,
+ into which the exported data is written.
+ bigquery_output_dataset:
+ The path of the BigQuery dataset created, in
+ bq://projectId.bqDatasetId format, into which the exported
+ data is written.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ExportDataOperationMetadata.ExportDataOutputInfo)
+ ),
+ ),
+ DESCRIPTOR=_EXPORTDATAOPERATIONMETADATA,
+ __module__="google.cloud.automl_v1beta1.proto.operations_pb2",
+ __doc__="""Details of ExportData operation.
+
+
+ Attributes:
+ output_info:
+ Output only. Information further describing this export data's
+ output.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ExportDataOperationMetadata)
+ ),
+)
+_sym_db.RegisterMessage(ExportDataOperationMetadata)
+_sym_db.RegisterMessage(ExportDataOperationMetadata.ExportDataOutputInfo)
+
+BatchPredictOperationMetadata = _reflection.GeneratedProtocolMessageType(
+ "BatchPredictOperationMetadata",
+ (_message.Message,),
+ dict(
+ BatchPredictOutputInfo=_reflection.GeneratedProtocolMessageType(
+ "BatchPredictOutputInfo",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_BATCHPREDICTOPERATIONMETADATA_BATCHPREDICTOUTPUTINFO,
+ __module__="google.cloud.automl_v1beta1.proto.operations_pb2",
+ __doc__="""Further describes this batch predict's output. Supplements
+
+ [BatchPredictOutputConfig][google.cloud.automl.v1beta1.BatchPredictOutputConfig].
+
+
+ Attributes:
+ output_location:
+ The output location into which prediction output is written.
+ gcs_output_directory:
+ The full path of the Google Cloud Storage directory created,
+ into which the prediction output is written.
+ bigquery_output_dataset:
+ The path of the BigQuery dataset created, in
+ bq://projectId.bqDatasetId format, into which the prediction
+ output is written.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.BatchPredictOperationMetadata.BatchPredictOutputInfo)
+ ),
+ ),
+ DESCRIPTOR=_BATCHPREDICTOPERATIONMETADATA,
+ __module__="google.cloud.automl_v1beta1.proto.operations_pb2",
+ __doc__="""Details of BatchPredict operation.
+
+
+ Attributes:
+ input_config:
+ Output only. The input config that was given upon starting
+ this batch predict operation.
+ output_info:
+ Output only. Information further describing this batch
+ predict's output.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.BatchPredictOperationMetadata)
+ ),
+)
+_sym_db.RegisterMessage(BatchPredictOperationMetadata)
+_sym_db.RegisterMessage(BatchPredictOperationMetadata.BatchPredictOutputInfo)
+
+ExportModelOperationMetadata = _reflection.GeneratedProtocolMessageType(
+ "ExportModelOperationMetadata",
+ (_message.Message,),
+ dict(
+ ExportModelOutputInfo=_reflection.GeneratedProtocolMessageType(
+ "ExportModelOutputInfo",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_EXPORTMODELOPERATIONMETADATA_EXPORTMODELOUTPUTINFO,
+ __module__="google.cloud.automl_v1beta1.proto.operations_pb2",
+ __doc__="""Further describes the output of model export. Supplements
+
+ [ModelExportOutputConfig][google.cloud.automl.v1beta1.ModelExportOutputConfig].
+
+
+ Attributes:
+ gcs_output_directory:
+ The full path of the Google Cloud Storage directory created,
+ into which the model will be exported.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ExportModelOperationMetadata.ExportModelOutputInfo)
+ ),
+ ),
+ DESCRIPTOR=_EXPORTMODELOPERATIONMETADATA,
+ __module__="google.cloud.automl_v1beta1.proto.operations_pb2",
+ __doc__="""Details of ExportModel operation.
+
+
+ Attributes:
+ output_info:
+ Output only. Information further describing the output of this
+ model export.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ExportModelOperationMetadata)
+ ),
+)
+_sym_db.RegisterMessage(ExportModelOperationMetadata)
+_sym_db.RegisterMessage(ExportModelOperationMetadata.ExportModelOutputInfo)
+
+ExportEvaluatedExamplesOperationMetadata = _reflection.GeneratedProtocolMessageType(
+ "ExportEvaluatedExamplesOperationMetadata",
+ (_message.Message,),
+ dict(
+ ExportEvaluatedExamplesOutputInfo=_reflection.GeneratedProtocolMessageType(
+ "ExportEvaluatedExamplesOutputInfo",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_EXPORTEVALUATEDEXAMPLESOPERATIONMETADATA_EXPORTEVALUATEDEXAMPLESOUTPUTINFO,
+ __module__="google.cloud.automl_v1beta1.proto.operations_pb2",
+ __doc__="""Further describes the output of the evaluated examples export.
+ Supplements
+
+ [ExportEvaluatedExamplesOutputConfig][google.cloud.automl.v1beta1.ExportEvaluatedExamplesOutputConfig].
+
+
+ Attributes:
+ bigquery_output_dataset:
+ The path of the BigQuery dataset created, in
+ bq://projectId.bqDatasetId format, into which the output of
+ export evaluated examples is written.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ExportEvaluatedExamplesOperationMetadata.ExportEvaluatedExamplesOutputInfo)
+ ),
+ ),
+ DESCRIPTOR=_EXPORTEVALUATEDEXAMPLESOPERATIONMETADATA,
+ __module__="google.cloud.automl_v1beta1.proto.operations_pb2",
+ __doc__="""Details of EvaluatedExamples operation.
+
+
+ Attributes:
+ output_info:
+ Output only. Information further describing the output of this
+ evaluated examples export.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ExportEvaluatedExamplesOperationMetadata)
+ ),
+)
+_sym_db.RegisterMessage(ExportEvaluatedExamplesOperationMetadata)
+_sym_db.RegisterMessage(
+ ExportEvaluatedExamplesOperationMetadata.ExportEvaluatedExamplesOutputInfo
+)
+
+
+DESCRIPTOR._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/automl/google/cloud/automl_v1beta1/proto/operations_pb2_grpc.py b/automl/google/cloud/automl_v1beta1/proto/operations_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/operations_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/automl/google/cloud/automl_v1beta1/proto/prediction_service.proto b/automl/google/cloud/automl_v1beta1/proto/prediction_service.proto
new file mode 100644
index 000000000000..57f1b794e716
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/prediction_service.proto
@@ -0,0 +1,255 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.automl.v1beta1;
+
+import "google/api/annotations.proto";
+import "google/api/client.proto";
+import "google/cloud/automl/v1beta1/annotation_payload.proto";
+import "google/cloud/automl/v1beta1/data_items.proto";
+import "google/cloud/automl/v1beta1/io.proto";
+import "google/cloud/automl/v1beta1/operations.proto";
+import "google/longrunning/operations.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl";
+option java_multiple_files = true;
+option java_outer_classname = "PredictionServiceProto";
+option java_package = "com.google.cloud.automl.v1beta1";
+option php_namespace = "Google\\Cloud\\AutoMl\\V1beta1";
+option ruby_package = "Google::Cloud::AutoML::V1beta1";
+
+// AutoML Prediction API.
+//
+// On any input that is documented to expect a string parameter in
+// snake_case or kebab-case, either of those cases is accepted.
+service PredictionService {
+ option (google.api.default_host) = "automl.googleapis.com";
+ option (google.api.oauth_scopes) =
+ "https://www.googleapis.com/auth/cloud-platform";
+
+ // Perform an online prediction. The prediction result will be directly
+ // returned in the response.
+ // Available for following ML problems, and their expected request payloads:
+ // * Image Classification - Image in .JPEG, .GIF or .PNG format, image_bytes
+ // up to 30MB.
+ // * Image Object Detection - Image in .JPEG, .GIF or .PNG format, image_bytes
+ // up to 30MB.
+ // * Text Classification - TextSnippet, content up to 60,000 characters,
+ // UTF-8 encoded.
+ // * Text Extraction - TextSnippet, content up to 30,000 characters,
+ // UTF-8 NFC encoded.
+ // * Translation - TextSnippet, content up to 25,000 characters, UTF-8
+ // encoded.
+ // * Tables - Row, with column values matching the columns of the model,
+ // up to 5MB. Not available for FORECASTING
+ //
+ // [prediction_type][google.cloud.automl.v1beta1.TablesModelMetadata.prediction_type].
+ // * Text Sentiment - TextSnippet, content up 500 characters, UTF-8
+ // encoded.
+ rpc Predict(PredictRequest) returns (PredictResponse) {
+ option (google.api.http) = {
+ post: "/v1beta1/{name=projects/*/locations/*/models/*}:predict"
+ body: "*"
+ };
+ }
+
+ // Perform a batch prediction. Unlike the online
+ // [Predict][google.cloud.automl.v1beta1.PredictionService.Predict], batch
+ // prediction result won't be immediately available in the response. Instead,
+ // a long running operation object is returned. User can poll the operation
+ // result via [GetOperation][google.longrunning.Operations.GetOperation]
+ // method. Once the operation is done,
+ // [BatchPredictResult][google.cloud.automl.v1beta1.BatchPredictResult] is
+ // returned in the [response][google.longrunning.Operation.response] field.
+ // Available for following ML problems:
+ // * Image Classification
+ // * Image Object Detection
+ // * Video Classification
+ // * Video Object Tracking * Text Extraction
+ // * Tables
+ rpc BatchPredict(BatchPredictRequest) returns (google.longrunning.Operation) {
+ option (google.api.http) = {
+ post: "/v1beta1/{name=projects/*/locations/*/models/*}:batchPredict"
+ body: "*"
+ };
+ }
+}
+
+// Request message for
+// [PredictionService.Predict][google.cloud.automl.v1beta1.PredictionService.Predict].
+message PredictRequest {
+ // Name of the model requested to serve the prediction.
+ string name = 1;
+
+ // Required. Payload to perform a prediction on. The payload must match the
+ // problem type that the model was trained to solve.
+ ExamplePayload payload = 2;
+
+ // Additional domain-specific parameters, any string must be up to 25000
+ // characters long.
+ //
+ // * For Image Classification:
+ //
+ // `score_threshold` - (float) A value from 0.0 to 1.0. When the model
+ // makes predictions for an image, it will only produce results that have
+ // at least this confidence score. The default is 0.5.
+ //
+ // * For Image Object Detection:
+ // `score_threshold` - (float) When Model detects objects on the image,
+ // it will only produce bounding boxes which have at least this
+ // confidence score. Value in 0 to 1 range, default is 0.5.
+ // `max_bounding_box_count` - (int64) No more than this number of bounding
+ // boxes will be returned in the response. Default is 100, the
+ // requested value may be limited by server.
+ // * For Tables:
+ // `feature_importance` - (boolean) Whether
+ //
+ // [feature_importance][[google.cloud.automl.v1beta1.TablesModelColumnInfo.feature_importance]
+ // should be populated in the returned
+ //
+ // [TablesAnnotation(-s)][[google.cloud.automl.v1beta1.TablesAnnotation].
+ // The default is false.
+ map params = 3;
+}
+
+// Response message for
+// [PredictionService.Predict][google.cloud.automl.v1beta1.PredictionService.Predict].
+message PredictResponse {
+ // Prediction result.
+ // Translation and Text Sentiment will return precisely one payload.
+ repeated AnnotationPayload payload = 1;
+
+ // The preprocessed example that AutoML actually makes prediction on.
+ // Empty if AutoML does not preprocess the input example.
+ // * For Text Extraction:
+ // If the input is a .pdf file, the OCR'ed text will be provided in
+ // [document_text][google.cloud.automl.v1beta1.Document.document_text].
+ ExamplePayload preprocessed_input = 3;
+
+ // Additional domain-specific prediction response metadata.
+ //
+ // * For Image Object Detection:
+ // `max_bounding_box_count` - (int64) At most that many bounding boxes per
+ // image could have been returned.
+ //
+ // * For Text Sentiment:
+ // `sentiment_score` - (float, deprecated) A value between -1 and 1,
+ // -1 maps to least positive sentiment, while 1 maps to the most positive
+ // one and the higher the score, the more positive the sentiment in the
+ // document is. Yet these values are relative to the training data, so
+ // e.g. if all data was positive then -1 will be also positive (though
+ // the least).
+ // The sentiment_score shouldn't be confused with "score" or "magnitude"
+ // from the previous Natural Language Sentiment Analysis API.
+ map metadata = 2;
+}
+
+// Request message for
+// [PredictionService.BatchPredict][google.cloud.automl.v1beta1.PredictionService.BatchPredict].
+message BatchPredictRequest {
+ // Name of the model requested to serve the batch prediction.
+ string name = 1;
+
+ // Required. The input configuration for batch prediction.
+ BatchPredictInputConfig input_config = 3;
+
+ // Required. The Configuration specifying where output predictions should
+ // be written.
+ BatchPredictOutputConfig output_config = 4;
+
+ // Additional domain-specific parameters for the predictions, any string must
+ // be up to 25000 characters long.
+ //
+ // * For Text Classification:
+ //
+ // `score_threshold` - (float) A value from 0.0 to 1.0. When the model
+ // makes predictions for a text snippet, it will only produce results
+ // that have at least this confidence score. The default is 0.5.
+ //
+ // * For Image Classification:
+ //
+ // `score_threshold` - (float) A value from 0.0 to 1.0. When the model
+ // makes predictions for an image, it will only produce results that
+ // have at least this confidence score. The default is 0.5.
+ //
+ // * For Image Object Detection:
+ //
+ // `score_threshold` - (float) When Model detects objects on the image,
+ // it will only produce bounding boxes which have at least this
+ // confidence score. Value in 0 to 1 range, default is 0.5.
+ // `max_bounding_box_count` - (int64) No more than this number of bounding
+ // boxes will be produced per image. Default is 100, the
+ // requested value may be limited by server.
+ //
+ // * For Video Classification :
+ // `score_threshold` - (float) A value from 0.0 to 1.0. When the model
+ // makes predictions for a video, it will only produce results that
+ // have at least this confidence score. The default is 0.5.
+ // `segment_classification` - (boolean) Set to true to request
+ // segment-level classification. AutoML Video Intelligence returns
+ // labels and their confidence scores for the entire segment of the
+ // video that user specified in the request configuration.
+ // The default is "true".
+ // `shot_classification` - (boolean) Set to true to request shot-level
+ // classification. AutoML Video Intelligence determines the boundaries
+ // for each camera shot in the entire segment of the video that user
+ // specified in the request configuration. AutoML Video Intelligence
+ // then returns labels and their confidence scores for each detected
+ // shot, along with the start and end time of the shot.
+ // WARNING: Model evaluation is not done for this classification type,
+ // the quality of it depends on training data, but there are no metrics
+ // provided to describe that quality. The default is "false".
+ // `1s_interval_classification` - (boolean) Set to true to request
+ // classification for a video at one-second intervals. AutoML Video
+ // Intelligence returns labels and their confidence scores for each
+ // second of the entire segment of the video that user specified in the
+ // request configuration.
+ // WARNING: Model evaluation is not done for this classification
+ // type, the quality of it depends on training data, but there are no
+ // metrics provided to describe that quality. The default is
+ // "false".
+ //
+ // * For Video Object Tracking:
+ // `score_threshold` - (float) When Model detects objects on video frames,
+ // it will only produce bounding boxes which have at least this
+ // confidence score. Value in 0 to 1 range, default is 0.5.
+ // `max_bounding_box_count` - (int64) No more than this number of bounding
+ // boxes will be returned per frame. Default is 100, the requested
+ // value may be limited by server.
+ // `min_bounding_box_size` - (float) Only bounding boxes with shortest edge
+ // at least that long as a relative value of video frame size will be
+ // returned. Value in 0 to 1 range. Default is 0.
+ //
+ map params = 5;
+}
+
+// Result of the Batch Predict. This message is returned in
+// [response][google.longrunning.Operation.response] of the operation returned
+// by the
+// [PredictionService.BatchPredict][google.cloud.automl.v1beta1.PredictionService.BatchPredict].
+message BatchPredictResult {
+ // Additional domain-specific prediction response metadata.
+ //
+ // * For Image Object Detection:
+ // `max_bounding_box_count` - (int64) At most that many bounding boxes per
+ // image could have been returned.
+ //
+ // * For Video Object Tracking:
+ // `max_bounding_box_count` - (int64) At most that many bounding boxes per
+ // frame could have been returned.
+ map metadata = 1;
+}
diff --git a/automl/google/cloud/automl_v1beta1/proto/prediction_service_pb2.py b/automl/google/cloud/automl_v1beta1/proto/prediction_service_pb2.py
new file mode 100644
index 000000000000..751f16ef8f5b
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/prediction_service_pb2.py
@@ -0,0 +1,880 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/automl_v1beta1/proto/prediction_service.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+from google.api import client_pb2 as google_dot_api_dot_client__pb2
+from google.cloud.automl_v1beta1.proto import (
+ annotation_payload_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_annotation__payload__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ data_items_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_data__items__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ io_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_io__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ operations_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_operations__pb2,
+)
+from google.longrunning import (
+ operations_pb2 as google_dot_longrunning_dot_operations__pb2,
+)
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/automl_v1beta1/proto/prediction_service.proto",
+ package="google.cloud.automl.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\037com.google.cloud.automl.v1beta1B\026PredictionServiceProtoP\001ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\312\002\033Google\\Cloud\\AutoMl\\V1beta1\352\002\036Google::Cloud::AutoML::V1beta1"
+ ),
+ serialized_pb=_b(
+ '\n:google/cloud/automl_v1beta1/proto/prediction_service.proto\x12\x1bgoogle.cloud.automl.v1beta1\x1a\x1cgoogle/api/annotations.proto\x1a\x17google/api/client.proto\x1a:google/cloud/automl_v1beta1/proto/annotation_payload.proto\x1a\x32google/cloud/automl_v1beta1/proto/data_items.proto\x1a*google/cloud/automl_v1beta1/proto/io.proto\x1a\x32google/cloud/automl_v1beta1/proto/operations.proto\x1a#google/longrunning/operations.proto"\xd4\x01\n\x0ePredictRequest\x12\x0c\n\x04name\x18\x01 \x01(\t\x12<\n\x07payload\x18\x02 \x01(\x0b\x32+.google.cloud.automl.v1beta1.ExamplePayload\x12G\n\x06params\x18\x03 \x03(\x0b\x32\x37.google.cloud.automl.v1beta1.PredictRequest.ParamsEntry\x1a-\n\x0bParamsEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01"\x9a\x02\n\x0fPredictResponse\x12?\n\x07payload\x18\x01 \x03(\x0b\x32..google.cloud.automl.v1beta1.AnnotationPayload\x12G\n\x12preprocessed_input\x18\x03 \x01(\x0b\x32+.google.cloud.automl.v1beta1.ExamplePayload\x12L\n\x08metadata\x18\x02 \x03(\x0b\x32:.google.cloud.automl.v1beta1.PredictResponse.MetadataEntry\x1a/\n\rMetadataEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01"\xba\x02\n\x13\x42\x61tchPredictRequest\x12\x0c\n\x04name\x18\x01 \x01(\t\x12J\n\x0cinput_config\x18\x03 \x01(\x0b\x32\x34.google.cloud.automl.v1beta1.BatchPredictInputConfig\x12L\n\routput_config\x18\x04 \x01(\x0b\x32\x35.google.cloud.automl.v1beta1.BatchPredictOutputConfig\x12L\n\x06params\x18\x05 \x03(\x0b\x32<.google.cloud.automl.v1beta1.BatchPredictRequest.ParamsEntry\x1a-\n\x0bParamsEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01"\x96\x01\n\x12\x42\x61tchPredictResult\x12O\n\x08metadata\x18\x01 \x03(\x0b\x32=.google.cloud.automl.v1beta1.BatchPredictResult.MetadataEntry\x1a/\n\rMetadataEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\x32\xb4\x03\n\x11PredictionService\x12\xa8\x01\n\x07Predict\x12+.google.cloud.automl.v1beta1.PredictRequest\x1a,.google.cloud.automl.v1beta1.PredictResponse"B\x82\xd3\xe4\x93\x02<"7/v1beta1/{name=projects/*/locations/*/models/*}:predict:\x01*\x12\xa8\x01\n\x0c\x42\x61tchPredict\x12\x30.google.cloud.automl.v1beta1.BatchPredictRequest\x1a\x1d.google.longrunning.Operation"G\x82\xd3\xe4\x93\x02\x41" The dataset has
+ // translation_dataset_metadata.
+ string filter = 3;
+
+ // Requested page size. Server may return fewer results than requested.
+ // If unspecified, server will pick a default size.
+ int32 page_size = 4;
+
+ // A token identifying a page of results for the server to return
+ // Typically obtained via
+ // [ListDatasetsResponse.next_page_token][google.cloud.automl.v1beta1.ListDatasetsResponse.next_page_token] of the previous
+ // [AutoMl.ListDatasets][google.cloud.automl.v1beta1.AutoMl.ListDatasets] call.
+ string page_token = 6;
+}
+
+// Response message for [AutoMl.ListDatasets][google.cloud.automl.v1beta1.AutoMl.ListDatasets].
+message ListDatasetsResponse {
+ // The datasets read.
+ repeated Dataset datasets = 1;
+
+ // A token to retrieve next page of results.
+ // Pass to [ListDatasetsRequest.page_token][google.cloud.automl.v1beta1.ListDatasetsRequest.page_token] to obtain that page.
+ string next_page_token = 2;
+}
+
+// Request message for [AutoMl.UpdateDataset][google.cloud.automl.v1beta1.AutoMl.UpdateDataset]
+message UpdateDatasetRequest {
+ // The dataset which replaces the resource on the server.
+ Dataset dataset = 1;
+
+ // The update mask applies to the resource.
+ google.protobuf.FieldMask update_mask = 2;
+}
+
+// Request message for [AutoMl.DeleteDataset][google.cloud.automl.v1beta1.AutoMl.DeleteDataset].
+message DeleteDatasetRequest {
+ // The resource name of the dataset to delete.
+ string name = 1;
+}
+
+// Request message for [AutoMl.ImportData][google.cloud.automl.v1beta1.AutoMl.ImportData].
+message ImportDataRequest {
+ // Required. Dataset name. Dataset must already exist. All imported
+ // annotations and examples will be added.
+ string name = 1;
+
+ // Required. The desired input location and its domain specific semantics,
+ // if any.
+ InputConfig input_config = 3;
+}
+
+// Request message for [AutoMl.ExportData][google.cloud.automl.v1beta1.AutoMl.ExportData].
+message ExportDataRequest {
+ // Required. The resource name of the dataset.
+ string name = 1;
+
+ // Required. The desired output location.
+ OutputConfig output_config = 3;
+}
+
+// Request message for [AutoMl.GetAnnotationSpec][google.cloud.automl.v1beta1.AutoMl.GetAnnotationSpec].
+message GetAnnotationSpecRequest {
+ // The resource name of the annotation spec to retrieve.
+ string name = 1;
+}
+
+// Request message for [AutoMl.GetTableSpec][google.cloud.automl.v1beta1.AutoMl.GetTableSpec].
+message GetTableSpecRequest {
+ // The resource name of the table spec to retrieve.
+ string name = 1;
+
+ // Mask specifying which fields to read.
+ google.protobuf.FieldMask field_mask = 2;
+}
+
+// Request message for [AutoMl.ListTableSpecs][google.cloud.automl.v1beta1.AutoMl.ListTableSpecs].
+message ListTableSpecsRequest {
+ // The resource name of the dataset to list table specs from.
+ string parent = 1;
+
+ // Mask specifying which fields to read.
+ google.protobuf.FieldMask field_mask = 2;
+
+ // Filter expression, see go/filtering.
+ string filter = 3;
+
+ // Requested page size. The server can return fewer results than requested.
+ // If unspecified, the server will pick a default size.
+ int32 page_size = 4;
+
+ // A token identifying a page of results for the server to return.
+ // Typically obtained from the
+ // [ListTableSpecsResponse.next_page_token][google.cloud.automl.v1beta1.ListTableSpecsResponse.next_page_token] field of the previous
+ // [AutoMl.ListTableSpecs][google.cloud.automl.v1beta1.AutoMl.ListTableSpecs] call.
+ string page_token = 6;
+}
+
+// Response message for [AutoMl.ListTableSpecs][google.cloud.automl.v1beta1.AutoMl.ListTableSpecs].
+message ListTableSpecsResponse {
+ // The table specs read.
+ repeated TableSpec table_specs = 1;
+
+ // A token to retrieve next page of results.
+ // Pass to [ListTableSpecsRequest.page_token][google.cloud.automl.v1beta1.ListTableSpecsRequest.page_token] to obtain that page.
+ string next_page_token = 2;
+}
+
+// Request message for [AutoMl.UpdateTableSpec][google.cloud.automl.v1beta1.AutoMl.UpdateTableSpec]
+message UpdateTableSpecRequest {
+ // The table spec which replaces the resource on the server.
+ TableSpec table_spec = 1;
+
+ // The update mask applies to the resource.
+ google.protobuf.FieldMask update_mask = 2;
+}
+
+// Request message for [AutoMl.GetColumnSpec][google.cloud.automl.v1beta1.AutoMl.GetColumnSpec].
+message GetColumnSpecRequest {
+ // The resource name of the column spec to retrieve.
+ string name = 1;
+
+ // Mask specifying which fields to read.
+ google.protobuf.FieldMask field_mask = 2;
+}
+
+// Request message for [AutoMl.ListColumnSpecs][google.cloud.automl.v1beta1.AutoMl.ListColumnSpecs].
+message ListColumnSpecsRequest {
+ // The resource name of the table spec to list column specs from.
+ string parent = 1;
+
+ // Mask specifying which fields to read.
+ google.protobuf.FieldMask field_mask = 2;
+
+ // Filter expression, see go/filtering.
+ string filter = 3;
+
+ // Requested page size. The server can return fewer results than requested.
+ // If unspecified, the server will pick a default size.
+ int32 page_size = 4;
+
+ // A token identifying a page of results for the server to return.
+ // Typically obtained from the
+ // [ListColumnSpecsResponse.next_page_token][google.cloud.automl.v1beta1.ListColumnSpecsResponse.next_page_token] field of the previous
+ // [AutoMl.ListColumnSpecs][google.cloud.automl.v1beta1.AutoMl.ListColumnSpecs] call.
+ string page_token = 6;
+}
+
+// Response message for [AutoMl.ListColumnSpecs][google.cloud.automl.v1beta1.AutoMl.ListColumnSpecs].
+message ListColumnSpecsResponse {
+ // The column specs read.
+ repeated ColumnSpec column_specs = 1;
+
+ // A token to retrieve next page of results.
+ // Pass to [ListColumnSpecsRequest.page_token][google.cloud.automl.v1beta1.ListColumnSpecsRequest.page_token] to obtain that page.
+ string next_page_token = 2;
+}
+
+// Request message for [AutoMl.UpdateColumnSpec][google.cloud.automl.v1beta1.AutoMl.UpdateColumnSpec]
+message UpdateColumnSpecRequest {
+ // The column spec which replaces the resource on the server.
+ ColumnSpec column_spec = 1;
+
+ // The update mask applies to the resource.
+ google.protobuf.FieldMask update_mask = 2;
+}
+
+// Request message for [AutoMl.CreateModel][google.cloud.automl.v1beta1.AutoMl.CreateModel].
+message CreateModelRequest {
+ // Resource name of the parent project where the model is being created.
+ string parent = 1;
+
+ // The model to create.
+ Model model = 4;
+}
+
+// Request message for [AutoMl.GetModel][google.cloud.automl.v1beta1.AutoMl.GetModel].
+message GetModelRequest {
+ // Resource name of the model.
+ string name = 1;
+}
+
+// Request message for [AutoMl.ListModels][google.cloud.automl.v1beta1.AutoMl.ListModels].
+message ListModelsRequest {
+ // Resource name of the project, from which to list the models.
+ string parent = 1;
+
+ // An expression for filtering the results of the request.
+ //
+ // * `model_metadata` - for existence of the case (e.g.
+ // video_classification_model_metadata:*).
+ // * `dataset_id` - for = or !=. Some examples of using the filter are:
+ //
+ // * `image_classification_model_metadata:*` --> The model has
+ // image_classification_model_metadata.
+ // * `dataset_id=5` --> The model was created from a dataset with ID 5.
+ string filter = 3;
+
+ // Requested page size.
+ int32 page_size = 4;
+
+ // A token identifying a page of results for the server to return
+ // Typically obtained via
+ // [ListModelsResponse.next_page_token][google.cloud.automl.v1beta1.ListModelsResponse.next_page_token] of the previous
+ // [AutoMl.ListModels][google.cloud.automl.v1beta1.AutoMl.ListModels] call.
+ string page_token = 6;
+}
+
+// Response message for [AutoMl.ListModels][google.cloud.automl.v1beta1.AutoMl.ListModels].
+message ListModelsResponse {
+ // List of models in the requested page.
+ repeated Model model = 1;
+
+ // A token to retrieve next page of results.
+ // Pass to [ListModelsRequest.page_token][google.cloud.automl.v1beta1.ListModelsRequest.page_token] to obtain that page.
+ string next_page_token = 2;
+}
+
+// Request message for [AutoMl.DeleteModel][google.cloud.automl.v1beta1.AutoMl.DeleteModel].
+message DeleteModelRequest {
+ // Resource name of the model being deleted.
+ string name = 1;
+}
+
+// Request message for [AutoMl.DeployModel][google.cloud.automl.v1beta1.AutoMl.DeployModel].
+message DeployModelRequest {
+ // The per-domain specific deployment parameters.
+ oneof model_deployment_metadata {
+ // Model deployment metadata specific to Image Object Detection.
+ ImageObjectDetectionModelDeploymentMetadata image_object_detection_model_deployment_metadata = 2;
+ }
+
+ // Resource name of the model to deploy.
+ string name = 1;
+}
+
+// Request message for [AutoMl.UndeployModel][google.cloud.automl.v1beta1.AutoMl.UndeployModel].
+message UndeployModelRequest {
+ // Resource name of the model to undeploy.
+ string name = 1;
+}
+
+// Request message for [AutoMl.ExportModel][google.cloud.automl.v1beta1.AutoMl.ExportModel].
+// Models need to be enabled for exporting, otherwise an error code will be
+// returned.
+message ExportModelRequest {
+ // Required. The resource name of the model to export.
+ string name = 1;
+
+ // Required. The desired output location and configuration.
+ ModelExportOutputConfig output_config = 3;
+}
+
+// Request message for [AutoMl.ExportEvaluatedExamples][google.cloud.automl.v1beta1.AutoMl.ExportEvaluatedExamples].
+message ExportEvaluatedExamplesRequest {
+ // Required. The resource name of the model whose evaluated examples are to
+ // be exported.
+ string name = 1;
+
+ // Required. The desired output location and configuration.
+ ExportEvaluatedExamplesOutputConfig output_config = 3;
+}
+
+// Request message for [AutoMl.GetModelEvaluation][google.cloud.automl.v1beta1.AutoMl.GetModelEvaluation].
+message GetModelEvaluationRequest {
+ // Resource name for the model evaluation.
+ string name = 1;
+}
+
+// Request message for [AutoMl.ListModelEvaluations][google.cloud.automl.v1beta1.AutoMl.ListModelEvaluations].
+message ListModelEvaluationsRequest {
+ // Resource name of the model to list the model evaluations for.
+ // If modelId is set as "-", this will list model evaluations from across all
+ // models of the parent location.
+ string parent = 1;
+
+ // An expression for filtering the results of the request.
+ //
+ // * `annotation_spec_id` - for =, != or existence. See example below for
+ // the last.
+ //
+ // Some examples of using the filter are:
+ //
+ // * `annotation_spec_id!=4` --> The model evaluation was done for
+ // annotation spec with ID different than 4.
+ // * `NOT annotation_spec_id:*` --> The model evaluation was done for
+ // aggregate of all annotation specs.
+ string filter = 3;
+
+ // Requested page size.
+ int32 page_size = 4;
+
+ // A token identifying a page of results for the server to return.
+ // Typically obtained via
+ // [ListModelEvaluationsResponse.next_page_token][google.cloud.automl.v1beta1.ListModelEvaluationsResponse.next_page_token] of the previous
+ // [AutoMl.ListModelEvaluations][google.cloud.automl.v1beta1.AutoMl.ListModelEvaluations] call.
+ string page_token = 6;
+}
+
+// Response message for [AutoMl.ListModelEvaluations][google.cloud.automl.v1beta1.AutoMl.ListModelEvaluations].
+message ListModelEvaluationsResponse {
+ // List of model evaluations in the requested page.
+ repeated ModelEvaluation model_evaluation = 1;
+
+ // A token to retrieve next page of results.
+ // Pass to the [ListModelEvaluationsRequest.page_token][google.cloud.automl.v1beta1.ListModelEvaluationsRequest.page_token] field of a new
+ // [AutoMl.ListModelEvaluations][google.cloud.automl.v1beta1.AutoMl.ListModelEvaluations] request to obtain that page.
+ string next_page_token = 2;
+}
diff --git a/automl/google/cloud/automl_v1beta1/proto/service_pb2.py b/automl/google/cloud/automl_v1beta1/proto/service_pb2.py
new file mode 100644
index 000000000000..5533916b97b8
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/service_pb2.py
@@ -0,0 +1,2942 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/automl_v1beta1/proto/service.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+from google.cloud.automl_v1beta1.proto import (
+ annotation_payload_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_annotation__payload__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ annotation_spec_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_annotation__spec__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ column_spec_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_column__spec__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ dataset_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_dataset__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ image_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_image__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ io_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_io__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ model_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_model__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ model_evaluation_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_model__evaluation__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ operations_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_operations__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ table_spec_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_table__spec__pb2,
+)
+from google.longrunning import (
+ operations_pb2 as google_dot_longrunning_dot_operations__pb2,
+)
+from google.protobuf import field_mask_pb2 as google_dot_protobuf_dot_field__mask__pb2
+from google.api import client_pb2 as google_dot_api_dot_client__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/automl_v1beta1/proto/service.proto",
+ package="google.cloud.automl.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\037com.google.cloud.automl.v1beta1B\013AutoMlProtoP\001ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\312\002\033Google\\Cloud\\AutoMl\\V1beta1\352\002\036Google::Cloud::AutoML::V1beta1"
+ ),
+ serialized_pb=_b(
+ '\n/google/cloud/automl_v1beta1/proto/service.proto\x12\x1bgoogle.cloud.automl.v1beta1\x1a\x1cgoogle/api/annotations.proto\x1a:google/cloud/automl_v1beta1/proto/annotation_payload.proto\x1a\x37google/cloud/automl_v1beta1/proto/annotation_spec.proto\x1a\x33google/cloud/automl_v1beta1/proto/column_spec.proto\x1a/google/cloud/automl_v1beta1/proto/dataset.proto\x1a-google/cloud/automl_v1beta1/proto/image.proto\x1a*google/cloud/automl_v1beta1/proto/io.proto\x1a-google/cloud/automl_v1beta1/proto/model.proto\x1a\x38google/cloud/automl_v1beta1/proto/model_evaluation.proto\x1a\x32google/cloud/automl_v1beta1/proto/operations.proto\x1a\x32google/cloud/automl_v1beta1/proto/table_spec.proto\x1a#google/longrunning/operations.proto\x1a google/protobuf/field_mask.proto\x1a\x17google/api/client.proto"]\n\x14\x43reateDatasetRequest\x12\x0e\n\x06parent\x18\x01 \x01(\t\x12\x35\n\x07\x64\x61taset\x18\x02 \x01(\x0b\x32$.google.cloud.automl.v1beta1.Dataset"!\n\x11GetDatasetRequest\x12\x0c\n\x04name\x18\x01 \x01(\t"\\\n\x13ListDatasetsRequest\x12\x0e\n\x06parent\x18\x01 \x01(\t\x12\x0e\n\x06\x66ilter\x18\x03 \x01(\t\x12\x11\n\tpage_size\x18\x04 \x01(\x05\x12\x12\n\npage_token\x18\x06 \x01(\t"g\n\x14ListDatasetsResponse\x12\x36\n\x08\x64\x61tasets\x18\x01 \x03(\x0b\x32$.google.cloud.automl.v1beta1.Dataset\x12\x17\n\x0fnext_page_token\x18\x02 \x01(\t"~\n\x14UpdateDatasetRequest\x12\x35\n\x07\x64\x61taset\x18\x01 \x01(\x0b\x32$.google.cloud.automl.v1beta1.Dataset\x12/\n\x0bupdate_mask\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.FieldMask"$\n\x14\x44\x65leteDatasetRequest\x12\x0c\n\x04name\x18\x01 \x01(\t"a\n\x11ImportDataRequest\x12\x0c\n\x04name\x18\x01 \x01(\t\x12>\n\x0cinput_config\x18\x03 \x01(\x0b\x32(.google.cloud.automl.v1beta1.InputConfig"c\n\x11\x45xportDataRequest\x12\x0c\n\x04name\x18\x01 \x01(\t\x12@\n\routput_config\x18\x03 \x01(\x0b\x32).google.cloud.automl.v1beta1.OutputConfig"(\n\x18GetAnnotationSpecRequest\x12\x0c\n\x04name\x18\x01 \x01(\t"S\n\x13GetTableSpecRequest\x12\x0c\n\x04name\x18\x01 \x01(\t\x12.\n\nfield_mask\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.FieldMask"\x8e\x01\n\x15ListTableSpecsRequest\x12\x0e\n\x06parent\x18\x01 \x01(\t\x12.\n\nfield_mask\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.FieldMask\x12\x0e\n\x06\x66ilter\x18\x03 \x01(\t\x12\x11\n\tpage_size\x18\x04 \x01(\x05\x12\x12\n\npage_token\x18\x06 \x01(\t"n\n\x16ListTableSpecsResponse\x12;\n\x0btable_specs\x18\x01 \x03(\x0b\x32&.google.cloud.automl.v1beta1.TableSpec\x12\x17\n\x0fnext_page_token\x18\x02 \x01(\t"\x85\x01\n\x16UpdateTableSpecRequest\x12:\n\ntable_spec\x18\x01 \x01(\x0b\x32&.google.cloud.automl.v1beta1.TableSpec\x12/\n\x0bupdate_mask\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.FieldMask"T\n\x14GetColumnSpecRequest\x12\x0c\n\x04name\x18\x01 \x01(\t\x12.\n\nfield_mask\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.FieldMask"\x8f\x01\n\x16ListColumnSpecsRequest\x12\x0e\n\x06parent\x18\x01 \x01(\t\x12.\n\nfield_mask\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.FieldMask\x12\x0e\n\x06\x66ilter\x18\x03 \x01(\t\x12\x11\n\tpage_size\x18\x04 \x01(\x05\x12\x12\n\npage_token\x18\x06 \x01(\t"q\n\x17ListColumnSpecsResponse\x12=\n\x0c\x63olumn_specs\x18\x01 \x03(\x0b\x32\'.google.cloud.automl.v1beta1.ColumnSpec\x12\x17\n\x0fnext_page_token\x18\x02 \x01(\t"\x88\x01\n\x17UpdateColumnSpecRequest\x12<\n\x0b\x63olumn_spec\x18\x01 \x01(\x0b\x32\'.google.cloud.automl.v1beta1.ColumnSpec\x12/\n\x0bupdate_mask\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.FieldMask"W\n\x12\x43reateModelRequest\x12\x0e\n\x06parent\x18\x01 \x01(\t\x12\x31\n\x05model\x18\x04 \x01(\x0b\x32".google.cloud.automl.v1beta1.Model"\x1f\n\x0fGetModelRequest\x12\x0c\n\x04name\x18\x01 \x01(\t"Z\n\x11ListModelsRequest\x12\x0e\n\x06parent\x18\x01 \x01(\t\x12\x0e\n\x06\x66ilter\x18\x03 \x01(\t\x12\x11\n\tpage_size\x18\x04 \x01(\x05\x12\x12\n\npage_token\x18\x06 \x01(\t"`\n\x12ListModelsResponse\x12\x31\n\x05model\x18\x01 \x03(\x0b\x32".google.cloud.automl.v1beta1.Model\x12\x17\n\x0fnext_page_token\x18\x02 \x01(\t""\n\x12\x44\x65leteModelRequest\x12\x0c\n\x04name\x18\x01 \x01(\t"\xc6\x01\n\x12\x44\x65ployModelRequest\x12\x84\x01\n0image_object_detection_model_deployment_metadata\x18\x02 \x01(\x0b\x32H.google.cloud.automl.v1beta1.ImageObjectDetectionModelDeploymentMetadataH\x00\x12\x0c\n\x04name\x18\x01 \x01(\tB\x1b\n\x19model_deployment_metadata"$\n\x14UndeployModelRequest\x12\x0c\n\x04name\x18\x01 \x01(\t"o\n\x12\x45xportModelRequest\x12\x0c\n\x04name\x18\x01 \x01(\t\x12K\n\routput_config\x18\x03 \x01(\x0b\x32\x34.google.cloud.automl.v1beta1.ModelExportOutputConfig"\x87\x01\n\x1e\x45xportEvaluatedExamplesRequest\x12\x0c\n\x04name\x18\x01 \x01(\t\x12W\n\routput_config\x18\x03 \x01(\x0b\x32@.google.cloud.automl.v1beta1.ExportEvaluatedExamplesOutputConfig")\n\x19GetModelEvaluationRequest\x12\x0c\n\x04name\x18\x01 \x01(\t"d\n\x1bListModelEvaluationsRequest\x12\x0e\n\x06parent\x18\x01 \x01(\t\x12\x0e\n\x06\x66ilter\x18\x03 \x01(\t\x12\x11\n\tpage_size\x18\x04 \x01(\x05\x12\x12\n\npage_token\x18\x06 \x01(\t"\x7f\n\x1cListModelEvaluationsResponse\x12\x46\n\x10model_evaluation\x18\x01 \x03(\x0b\x32,.google.cloud.automl.v1beta1.ModelEvaluation\x12\x17\n\x0fnext_page_token\x18\x02 \x01(\t2\xd9"\n\x06\x41utoMl\x12\xac\x01\n\rCreateDataset\x12\x31.google.cloud.automl.v1beta1.CreateDatasetRequest\x1a$.google.cloud.automl.v1beta1.Dataset"B\x82\xd3\xe4\x93\x02<"1/v1beta1/{parent=projects/*/locations/*}/datasets:\x07\x64\x61taset\x12\x9d\x01\n\nGetDataset\x12..google.cloud.automl.v1beta1.GetDatasetRequest\x1a$.google.cloud.automl.v1beta1.Dataset"9\x82\xd3\xe4\x93\x02\x33\x12\x31/v1beta1/{name=projects/*/locations/*/datasets/*}\x12\xae\x01\n\x0cListDatasets\x12\x30.google.cloud.automl.v1beta1.ListDatasetsRequest\x1a\x31.google.cloud.automl.v1beta1.ListDatasetsResponse"9\x82\xd3\xe4\x93\x02\x33\x12\x31/v1beta1/{parent=projects/*/locations/*}/datasets\x12\xb4\x01\n\rUpdateDataset\x12\x31.google.cloud.automl.v1beta1.UpdateDatasetRequest\x1a$.google.cloud.automl.v1beta1.Dataset"J\x82\xd3\xe4\x93\x02\x44\x32\x39/v1beta1/{dataset.name=projects/*/locations/*/datasets/*}:\x07\x64\x61taset\x12\x9c\x01\n\rDeleteDataset\x12\x31.google.cloud.automl.v1beta1.DeleteDatasetRequest\x1a\x1d.google.longrunning.Operation"9\x82\xd3\xe4\x93\x02\x33*1/v1beta1/{name=projects/*/locations/*/datasets/*}\x12\xa4\x01\n\nImportData\x12..google.cloud.automl.v1beta1.ImportDataRequest\x1a\x1d.google.longrunning.Operation"G\x82\xd3\xe4\x93\x02\x41"/v1beta1/{name=projects/*/locations/*/datasets/*/tableSpecs/*}\x12\xc1\x01\n\x0eListTableSpecs\x12\x32.google.cloud.automl.v1beta1.ListTableSpecsRequest\x1a\x33.google.cloud.automl.v1beta1.ListTableSpecsResponse"F\x82\xd3\xe4\x93\x02@\x12>/v1beta1/{parent=projects/*/locations/*/datasets/*}/tableSpecs\x12\xcd\x01\n\x0fUpdateTableSpec\x12\x33.google.cloud.automl.v1beta1.UpdateTableSpecRequest\x1a&.google.cloud.automl.v1beta1.TableSpec"]\x82\xd3\xe4\x93\x02W2I/v1beta1/{table_spec.name=projects/*/locations/*/datasets/*/tableSpecs/*}:\ntable_spec\x12\xc1\x01\n\rGetColumnSpec\x12\x31.google.cloud.automl.v1beta1.GetColumnSpecRequest\x1a\'.google.cloud.automl.v1beta1.ColumnSpec"T\x82\xd3\xe4\x93\x02N\x12L/v1beta1/{name=projects/*/locations/*/datasets/*/tableSpecs/*/columnSpecs/*}\x12\xd2\x01\n\x0fListColumnSpecs\x12\x33.google.cloud.automl.v1beta1.ListColumnSpecsRequest\x1a\x34.google.cloud.automl.v1beta1.ListColumnSpecsResponse"T\x82\xd3\xe4\x93\x02N\x12L/v1beta1/{parent=projects/*/locations/*/datasets/*/tableSpecs/*}/columnSpecs\x12\xe0\x01\n\x10UpdateColumnSpec\x12\x34.google.cloud.automl.v1beta1.UpdateColumnSpecRequest\x1a\'.google.cloud.automl.v1beta1.ColumnSpec"m\x82\xd3\xe4\x93\x02g2X/v1beta1/{column_spec.name=projects/*/locations/*/datasets/*/tableSpecs/*/columnSpecs/*}:\x0b\x63olumn_spec\x12\x9d\x01\n\x0b\x43reateModel\x12/.google.cloud.automl.v1beta1.CreateModelRequest\x1a\x1d.google.longrunning.Operation">\x82\xd3\xe4\x93\x02\x38"//v1beta1/{parent=projects/*/locations/*}/models:\x05model\x12\x95\x01\n\x08GetModel\x12,.google.cloud.automl.v1beta1.GetModelRequest\x1a".google.cloud.automl.v1beta1.Model"7\x82\xd3\xe4\x93\x02\x31\x12//v1beta1/{name=projects/*/locations/*/models/*}\x12\xa6\x01\n\nListModels\x12..google.cloud.automl.v1beta1.ListModelsRequest\x1a/.google.cloud.automl.v1beta1.ListModelsResponse"7\x82\xd3\xe4\x93\x02\x31\x12//v1beta1/{parent=projects/*/locations/*}/models\x12\x96\x01\n\x0b\x44\x65leteModel\x12/.google.cloud.automl.v1beta1.DeleteModelRequest\x1a\x1d.google.longrunning.Operation"7\x82\xd3\xe4\x93\x02\x31*//v1beta1/{name=projects/*/locations/*/models/*}\x12\xa0\x01\n\x0b\x44\x65ployModel\x12/.google.cloud.automl.v1beta1.DeployModelRequest\x1a\x1d.google.longrunning.Operation"A\x82\xd3\xe4\x93\x02;"6/v1beta1/{name=projects/*/locations/*/models/*}:deploy:\x01*\x12\xa6\x01\n\rUndeployModel\x12\x31.google.cloud.automl.v1beta1.UndeployModelRequest\x1a\x1d.google.longrunning.Operation"C\x82\xd3\xe4\x93\x02="8/v1beta1/{name=projects/*/locations/*/models/*}:undeploy:\x01*\x12\xa0\x01\n\x0b\x45xportModel\x12/.google.cloud.automl.v1beta1.ExportModelRequest\x1a\x1d.google.longrunning.Operation"A\x82\xd3\xe4\x93\x02;"6/v1beta1/{name=projects/*/locations/*/models/*}:export:\x01*\x12\xc9\x01\n\x17\x45xportEvaluatedExamples\x12;.google.cloud.automl.v1beta1.ExportEvaluatedExamplesRequest\x1a\x1d.google.longrunning.Operation"R\x82\xd3\xe4\x93\x02L"G/v1beta1/{name=projects/*/locations/*/models/*}:exportEvaluatedExamples:\x01*\x12\xc6\x01\n\x12GetModelEvaluation\x12\x36.google.cloud.automl.v1beta1.GetModelEvaluationRequest\x1a,.google.cloud.automl.v1beta1.ModelEvaluation"J\x82\xd3\xe4\x93\x02\x44\x12\x42/v1beta1/{name=projects/*/locations/*/models/*/modelEvaluations/*}\x12\xd7\x01\n\x14ListModelEvaluations\x12\x38.google.cloud.automl.v1beta1.ListModelEvaluationsRequest\x1a\x39.google.cloud.automl.v1beta1.ListModelEvaluationsResponse"J\x82\xd3\xe4\x93\x02\x44\x12\x42/v1beta1/{parent=projects/*/locations/*/models/*}/modelEvaluations\x1aI\xca\x41\x15\x61utoml.googleapis.com\xd2\x41.https://www.googleapis.com/auth/cloud-platformB\xb2\x01\n\x1f\x63om.google.cloud.automl.v1beta1B\x0b\x41utoMlProtoP\x01ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\xca\x02\x1bGoogle\\Cloud\\AutoMl\\V1beta1\xea\x02\x1eGoogle::Cloud::AutoML::V1beta1b\x06proto3'
+ ),
+ dependencies=[
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_annotation__payload__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_annotation__spec__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_column__spec__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_dataset__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_image__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_io__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_model__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_model__evaluation__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_operations__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_table__spec__pb2.DESCRIPTOR,
+ google_dot_longrunning_dot_operations__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_field__mask__pb2.DESCRIPTOR,
+ google_dot_api_dot_client__pb2.DESCRIPTOR,
+ ],
+)
+
+
+_CREATEDATASETREQUEST = _descriptor.Descriptor(
+ name="CreateDatasetRequest",
+ full_name="google.cloud.automl.v1beta1.CreateDatasetRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="parent",
+ full_name="google.cloud.automl.v1beta1.CreateDatasetRequest.parent",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="dataset",
+ full_name="google.cloud.automl.v1beta1.CreateDatasetRequest.dataset",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=725,
+ serialized_end=818,
+)
+
+
+_GETDATASETREQUEST = _descriptor.Descriptor(
+ name="GetDatasetRequest",
+ full_name="google.cloud.automl.v1beta1.GetDatasetRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.automl.v1beta1.GetDatasetRequest.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=820,
+ serialized_end=853,
+)
+
+
+_LISTDATASETSREQUEST = _descriptor.Descriptor(
+ name="ListDatasetsRequest",
+ full_name="google.cloud.automl.v1beta1.ListDatasetsRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="parent",
+ full_name="google.cloud.automl.v1beta1.ListDatasetsRequest.parent",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="filter",
+ full_name="google.cloud.automl.v1beta1.ListDatasetsRequest.filter",
+ index=1,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="page_size",
+ full_name="google.cloud.automl.v1beta1.ListDatasetsRequest.page_size",
+ index=2,
+ number=4,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="page_token",
+ full_name="google.cloud.automl.v1beta1.ListDatasetsRequest.page_token",
+ index=3,
+ number=6,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=855,
+ serialized_end=947,
+)
+
+
+_LISTDATASETSRESPONSE = _descriptor.Descriptor(
+ name="ListDatasetsResponse",
+ full_name="google.cloud.automl.v1beta1.ListDatasetsResponse",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="datasets",
+ full_name="google.cloud.automl.v1beta1.ListDatasetsResponse.datasets",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="next_page_token",
+ full_name="google.cloud.automl.v1beta1.ListDatasetsResponse.next_page_token",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=949,
+ serialized_end=1052,
+)
+
+
+_UPDATEDATASETREQUEST = _descriptor.Descriptor(
+ name="UpdateDatasetRequest",
+ full_name="google.cloud.automl.v1beta1.UpdateDatasetRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="dataset",
+ full_name="google.cloud.automl.v1beta1.UpdateDatasetRequest.dataset",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="update_mask",
+ full_name="google.cloud.automl.v1beta1.UpdateDatasetRequest.update_mask",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1054,
+ serialized_end=1180,
+)
+
+
+_DELETEDATASETREQUEST = _descriptor.Descriptor(
+ name="DeleteDatasetRequest",
+ full_name="google.cloud.automl.v1beta1.DeleteDatasetRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.automl.v1beta1.DeleteDatasetRequest.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1182,
+ serialized_end=1218,
+)
+
+
+_IMPORTDATAREQUEST = _descriptor.Descriptor(
+ name="ImportDataRequest",
+ full_name="google.cloud.automl.v1beta1.ImportDataRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.automl.v1beta1.ImportDataRequest.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="input_config",
+ full_name="google.cloud.automl.v1beta1.ImportDataRequest.input_config",
+ index=1,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1220,
+ serialized_end=1317,
+)
+
+
+_EXPORTDATAREQUEST = _descriptor.Descriptor(
+ name="ExportDataRequest",
+ full_name="google.cloud.automl.v1beta1.ExportDataRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.automl.v1beta1.ExportDataRequest.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="output_config",
+ full_name="google.cloud.automl.v1beta1.ExportDataRequest.output_config",
+ index=1,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1319,
+ serialized_end=1418,
+)
+
+
+_GETANNOTATIONSPECREQUEST = _descriptor.Descriptor(
+ name="GetAnnotationSpecRequest",
+ full_name="google.cloud.automl.v1beta1.GetAnnotationSpecRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.automl.v1beta1.GetAnnotationSpecRequest.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1420,
+ serialized_end=1460,
+)
+
+
+_GETTABLESPECREQUEST = _descriptor.Descriptor(
+ name="GetTableSpecRequest",
+ full_name="google.cloud.automl.v1beta1.GetTableSpecRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.automl.v1beta1.GetTableSpecRequest.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="field_mask",
+ full_name="google.cloud.automl.v1beta1.GetTableSpecRequest.field_mask",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1462,
+ serialized_end=1545,
+)
+
+
+_LISTTABLESPECSREQUEST = _descriptor.Descriptor(
+ name="ListTableSpecsRequest",
+ full_name="google.cloud.automl.v1beta1.ListTableSpecsRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="parent",
+ full_name="google.cloud.automl.v1beta1.ListTableSpecsRequest.parent",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="field_mask",
+ full_name="google.cloud.automl.v1beta1.ListTableSpecsRequest.field_mask",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="filter",
+ full_name="google.cloud.automl.v1beta1.ListTableSpecsRequest.filter",
+ index=2,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="page_size",
+ full_name="google.cloud.automl.v1beta1.ListTableSpecsRequest.page_size",
+ index=3,
+ number=4,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="page_token",
+ full_name="google.cloud.automl.v1beta1.ListTableSpecsRequest.page_token",
+ index=4,
+ number=6,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1548,
+ serialized_end=1690,
+)
+
+
+_LISTTABLESPECSRESPONSE = _descriptor.Descriptor(
+ name="ListTableSpecsResponse",
+ full_name="google.cloud.automl.v1beta1.ListTableSpecsResponse",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="table_specs",
+ full_name="google.cloud.automl.v1beta1.ListTableSpecsResponse.table_specs",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="next_page_token",
+ full_name="google.cloud.automl.v1beta1.ListTableSpecsResponse.next_page_token",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1692,
+ serialized_end=1802,
+)
+
+
+_UPDATETABLESPECREQUEST = _descriptor.Descriptor(
+ name="UpdateTableSpecRequest",
+ full_name="google.cloud.automl.v1beta1.UpdateTableSpecRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="table_spec",
+ full_name="google.cloud.automl.v1beta1.UpdateTableSpecRequest.table_spec",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="update_mask",
+ full_name="google.cloud.automl.v1beta1.UpdateTableSpecRequest.update_mask",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1805,
+ serialized_end=1938,
+)
+
+
+_GETCOLUMNSPECREQUEST = _descriptor.Descriptor(
+ name="GetColumnSpecRequest",
+ full_name="google.cloud.automl.v1beta1.GetColumnSpecRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.automl.v1beta1.GetColumnSpecRequest.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="field_mask",
+ full_name="google.cloud.automl.v1beta1.GetColumnSpecRequest.field_mask",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1940,
+ serialized_end=2024,
+)
+
+
+_LISTCOLUMNSPECSREQUEST = _descriptor.Descriptor(
+ name="ListColumnSpecsRequest",
+ full_name="google.cloud.automl.v1beta1.ListColumnSpecsRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="parent",
+ full_name="google.cloud.automl.v1beta1.ListColumnSpecsRequest.parent",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="field_mask",
+ full_name="google.cloud.automl.v1beta1.ListColumnSpecsRequest.field_mask",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="filter",
+ full_name="google.cloud.automl.v1beta1.ListColumnSpecsRequest.filter",
+ index=2,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="page_size",
+ full_name="google.cloud.automl.v1beta1.ListColumnSpecsRequest.page_size",
+ index=3,
+ number=4,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="page_token",
+ full_name="google.cloud.automl.v1beta1.ListColumnSpecsRequest.page_token",
+ index=4,
+ number=6,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2027,
+ serialized_end=2170,
+)
+
+
+_LISTCOLUMNSPECSRESPONSE = _descriptor.Descriptor(
+ name="ListColumnSpecsResponse",
+ full_name="google.cloud.automl.v1beta1.ListColumnSpecsResponse",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="column_specs",
+ full_name="google.cloud.automl.v1beta1.ListColumnSpecsResponse.column_specs",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="next_page_token",
+ full_name="google.cloud.automl.v1beta1.ListColumnSpecsResponse.next_page_token",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2172,
+ serialized_end=2285,
+)
+
+
+_UPDATECOLUMNSPECREQUEST = _descriptor.Descriptor(
+ name="UpdateColumnSpecRequest",
+ full_name="google.cloud.automl.v1beta1.UpdateColumnSpecRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="column_spec",
+ full_name="google.cloud.automl.v1beta1.UpdateColumnSpecRequest.column_spec",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="update_mask",
+ full_name="google.cloud.automl.v1beta1.UpdateColumnSpecRequest.update_mask",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2288,
+ serialized_end=2424,
+)
+
+
+_CREATEMODELREQUEST = _descriptor.Descriptor(
+ name="CreateModelRequest",
+ full_name="google.cloud.automl.v1beta1.CreateModelRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="parent",
+ full_name="google.cloud.automl.v1beta1.CreateModelRequest.parent",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="model",
+ full_name="google.cloud.automl.v1beta1.CreateModelRequest.model",
+ index=1,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2426,
+ serialized_end=2513,
+)
+
+
+_GETMODELREQUEST = _descriptor.Descriptor(
+ name="GetModelRequest",
+ full_name="google.cloud.automl.v1beta1.GetModelRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.automl.v1beta1.GetModelRequest.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2515,
+ serialized_end=2546,
+)
+
+
+_LISTMODELSREQUEST = _descriptor.Descriptor(
+ name="ListModelsRequest",
+ full_name="google.cloud.automl.v1beta1.ListModelsRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="parent",
+ full_name="google.cloud.automl.v1beta1.ListModelsRequest.parent",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="filter",
+ full_name="google.cloud.automl.v1beta1.ListModelsRequest.filter",
+ index=1,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="page_size",
+ full_name="google.cloud.automl.v1beta1.ListModelsRequest.page_size",
+ index=2,
+ number=4,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="page_token",
+ full_name="google.cloud.automl.v1beta1.ListModelsRequest.page_token",
+ index=3,
+ number=6,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2548,
+ serialized_end=2638,
+)
+
+
+_LISTMODELSRESPONSE = _descriptor.Descriptor(
+ name="ListModelsResponse",
+ full_name="google.cloud.automl.v1beta1.ListModelsResponse",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="model",
+ full_name="google.cloud.automl.v1beta1.ListModelsResponse.model",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="next_page_token",
+ full_name="google.cloud.automl.v1beta1.ListModelsResponse.next_page_token",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2640,
+ serialized_end=2736,
+)
+
+
+_DELETEMODELREQUEST = _descriptor.Descriptor(
+ name="DeleteModelRequest",
+ full_name="google.cloud.automl.v1beta1.DeleteModelRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.automl.v1beta1.DeleteModelRequest.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2738,
+ serialized_end=2772,
+)
+
+
+_DEPLOYMODELREQUEST = _descriptor.Descriptor(
+ name="DeployModelRequest",
+ full_name="google.cloud.automl.v1beta1.DeployModelRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="image_object_detection_model_deployment_metadata",
+ full_name="google.cloud.automl.v1beta1.DeployModelRequest.image_object_detection_model_deployment_metadata",
+ index=0,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.automl.v1beta1.DeployModelRequest.name",
+ index=1,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="model_deployment_metadata",
+ full_name="google.cloud.automl.v1beta1.DeployModelRequest.model_deployment_metadata",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=2775,
+ serialized_end=2973,
+)
+
+
+_UNDEPLOYMODELREQUEST = _descriptor.Descriptor(
+ name="UndeployModelRequest",
+ full_name="google.cloud.automl.v1beta1.UndeployModelRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.automl.v1beta1.UndeployModelRequest.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2975,
+ serialized_end=3011,
+)
+
+
+_EXPORTMODELREQUEST = _descriptor.Descriptor(
+ name="ExportModelRequest",
+ full_name="google.cloud.automl.v1beta1.ExportModelRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.automl.v1beta1.ExportModelRequest.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="output_config",
+ full_name="google.cloud.automl.v1beta1.ExportModelRequest.output_config",
+ index=1,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=3013,
+ serialized_end=3124,
+)
+
+
+_EXPORTEVALUATEDEXAMPLESREQUEST = _descriptor.Descriptor(
+ name="ExportEvaluatedExamplesRequest",
+ full_name="google.cloud.automl.v1beta1.ExportEvaluatedExamplesRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.automl.v1beta1.ExportEvaluatedExamplesRequest.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="output_config",
+ full_name="google.cloud.automl.v1beta1.ExportEvaluatedExamplesRequest.output_config",
+ index=1,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=3127,
+ serialized_end=3262,
+)
+
+
+_GETMODELEVALUATIONREQUEST = _descriptor.Descriptor(
+ name="GetModelEvaluationRequest",
+ full_name="google.cloud.automl.v1beta1.GetModelEvaluationRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.automl.v1beta1.GetModelEvaluationRequest.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=3264,
+ serialized_end=3305,
+)
+
+
+_LISTMODELEVALUATIONSREQUEST = _descriptor.Descriptor(
+ name="ListModelEvaluationsRequest",
+ full_name="google.cloud.automl.v1beta1.ListModelEvaluationsRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="parent",
+ full_name="google.cloud.automl.v1beta1.ListModelEvaluationsRequest.parent",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="filter",
+ full_name="google.cloud.automl.v1beta1.ListModelEvaluationsRequest.filter",
+ index=1,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="page_size",
+ full_name="google.cloud.automl.v1beta1.ListModelEvaluationsRequest.page_size",
+ index=2,
+ number=4,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="page_token",
+ full_name="google.cloud.automl.v1beta1.ListModelEvaluationsRequest.page_token",
+ index=3,
+ number=6,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=3307,
+ serialized_end=3407,
+)
+
+
+_LISTMODELEVALUATIONSRESPONSE = _descriptor.Descriptor(
+ name="ListModelEvaluationsResponse",
+ full_name="google.cloud.automl.v1beta1.ListModelEvaluationsResponse",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="model_evaluation",
+ full_name="google.cloud.automl.v1beta1.ListModelEvaluationsResponse.model_evaluation",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="next_page_token",
+ full_name="google.cloud.automl.v1beta1.ListModelEvaluationsResponse.next_page_token",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=3409,
+ serialized_end=3536,
+)
+
+_CREATEDATASETREQUEST.fields_by_name[
+ "dataset"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_dataset__pb2._DATASET
+)
+_LISTDATASETSRESPONSE.fields_by_name[
+ "datasets"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_dataset__pb2._DATASET
+)
+_UPDATEDATASETREQUEST.fields_by_name[
+ "dataset"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_dataset__pb2._DATASET
+)
+_UPDATEDATASETREQUEST.fields_by_name[
+ "update_mask"
+].message_type = google_dot_protobuf_dot_field__mask__pb2._FIELDMASK
+_IMPORTDATAREQUEST.fields_by_name[
+ "input_config"
+].message_type = google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_io__pb2._INPUTCONFIG
+_EXPORTDATAREQUEST.fields_by_name[
+ "output_config"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_io__pb2._OUTPUTCONFIG
+)
+_GETTABLESPECREQUEST.fields_by_name[
+ "field_mask"
+].message_type = google_dot_protobuf_dot_field__mask__pb2._FIELDMASK
+_LISTTABLESPECSREQUEST.fields_by_name[
+ "field_mask"
+].message_type = google_dot_protobuf_dot_field__mask__pb2._FIELDMASK
+_LISTTABLESPECSRESPONSE.fields_by_name[
+ "table_specs"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_table__spec__pb2._TABLESPEC
+)
+_UPDATETABLESPECREQUEST.fields_by_name[
+ "table_spec"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_table__spec__pb2._TABLESPEC
+)
+_UPDATETABLESPECREQUEST.fields_by_name[
+ "update_mask"
+].message_type = google_dot_protobuf_dot_field__mask__pb2._FIELDMASK
+_GETCOLUMNSPECREQUEST.fields_by_name[
+ "field_mask"
+].message_type = google_dot_protobuf_dot_field__mask__pb2._FIELDMASK
+_LISTCOLUMNSPECSREQUEST.fields_by_name[
+ "field_mask"
+].message_type = google_dot_protobuf_dot_field__mask__pb2._FIELDMASK
+_LISTCOLUMNSPECSRESPONSE.fields_by_name[
+ "column_specs"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_column__spec__pb2._COLUMNSPEC
+)
+_UPDATECOLUMNSPECREQUEST.fields_by_name[
+ "column_spec"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_column__spec__pb2._COLUMNSPEC
+)
+_UPDATECOLUMNSPECREQUEST.fields_by_name[
+ "update_mask"
+].message_type = google_dot_protobuf_dot_field__mask__pb2._FIELDMASK
+_CREATEMODELREQUEST.fields_by_name[
+ "model"
+].message_type = google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_model__pb2._MODEL
+_LISTMODELSRESPONSE.fields_by_name[
+ "model"
+].message_type = google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_model__pb2._MODEL
+_DEPLOYMODELREQUEST.fields_by_name[
+ "image_object_detection_model_deployment_metadata"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_image__pb2._IMAGEOBJECTDETECTIONMODELDEPLOYMENTMETADATA
+)
+_DEPLOYMODELREQUEST.oneofs_by_name["model_deployment_metadata"].fields.append(
+ _DEPLOYMODELREQUEST.fields_by_name[
+ "image_object_detection_model_deployment_metadata"
+ ]
+)
+_DEPLOYMODELREQUEST.fields_by_name[
+ "image_object_detection_model_deployment_metadata"
+].containing_oneof = _DEPLOYMODELREQUEST.oneofs_by_name["model_deployment_metadata"]
+_EXPORTMODELREQUEST.fields_by_name[
+ "output_config"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_io__pb2._MODELEXPORTOUTPUTCONFIG
+)
+_EXPORTEVALUATEDEXAMPLESREQUEST.fields_by_name[
+ "output_config"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_io__pb2._EXPORTEVALUATEDEXAMPLESOUTPUTCONFIG
+)
+_LISTMODELEVALUATIONSRESPONSE.fields_by_name[
+ "model_evaluation"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_model__evaluation__pb2._MODELEVALUATION
+)
+DESCRIPTOR.message_types_by_name["CreateDatasetRequest"] = _CREATEDATASETREQUEST
+DESCRIPTOR.message_types_by_name["GetDatasetRequest"] = _GETDATASETREQUEST
+DESCRIPTOR.message_types_by_name["ListDatasetsRequest"] = _LISTDATASETSREQUEST
+DESCRIPTOR.message_types_by_name["ListDatasetsResponse"] = _LISTDATASETSRESPONSE
+DESCRIPTOR.message_types_by_name["UpdateDatasetRequest"] = _UPDATEDATASETREQUEST
+DESCRIPTOR.message_types_by_name["DeleteDatasetRequest"] = _DELETEDATASETREQUEST
+DESCRIPTOR.message_types_by_name["ImportDataRequest"] = _IMPORTDATAREQUEST
+DESCRIPTOR.message_types_by_name["ExportDataRequest"] = _EXPORTDATAREQUEST
+DESCRIPTOR.message_types_by_name["GetAnnotationSpecRequest"] = _GETANNOTATIONSPECREQUEST
+DESCRIPTOR.message_types_by_name["GetTableSpecRequest"] = _GETTABLESPECREQUEST
+DESCRIPTOR.message_types_by_name["ListTableSpecsRequest"] = _LISTTABLESPECSREQUEST
+DESCRIPTOR.message_types_by_name["ListTableSpecsResponse"] = _LISTTABLESPECSRESPONSE
+DESCRIPTOR.message_types_by_name["UpdateTableSpecRequest"] = _UPDATETABLESPECREQUEST
+DESCRIPTOR.message_types_by_name["GetColumnSpecRequest"] = _GETCOLUMNSPECREQUEST
+DESCRIPTOR.message_types_by_name["ListColumnSpecsRequest"] = _LISTCOLUMNSPECSREQUEST
+DESCRIPTOR.message_types_by_name["ListColumnSpecsResponse"] = _LISTCOLUMNSPECSRESPONSE
+DESCRIPTOR.message_types_by_name["UpdateColumnSpecRequest"] = _UPDATECOLUMNSPECREQUEST
+DESCRIPTOR.message_types_by_name["CreateModelRequest"] = _CREATEMODELREQUEST
+DESCRIPTOR.message_types_by_name["GetModelRequest"] = _GETMODELREQUEST
+DESCRIPTOR.message_types_by_name["ListModelsRequest"] = _LISTMODELSREQUEST
+DESCRIPTOR.message_types_by_name["ListModelsResponse"] = _LISTMODELSRESPONSE
+DESCRIPTOR.message_types_by_name["DeleteModelRequest"] = _DELETEMODELREQUEST
+DESCRIPTOR.message_types_by_name["DeployModelRequest"] = _DEPLOYMODELREQUEST
+DESCRIPTOR.message_types_by_name["UndeployModelRequest"] = _UNDEPLOYMODELREQUEST
+DESCRIPTOR.message_types_by_name["ExportModelRequest"] = _EXPORTMODELREQUEST
+DESCRIPTOR.message_types_by_name[
+ "ExportEvaluatedExamplesRequest"
+] = _EXPORTEVALUATEDEXAMPLESREQUEST
+DESCRIPTOR.message_types_by_name[
+ "GetModelEvaluationRequest"
+] = _GETMODELEVALUATIONREQUEST
+DESCRIPTOR.message_types_by_name[
+ "ListModelEvaluationsRequest"
+] = _LISTMODELEVALUATIONSREQUEST
+DESCRIPTOR.message_types_by_name[
+ "ListModelEvaluationsResponse"
+] = _LISTMODELEVALUATIONSRESPONSE
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+CreateDatasetRequest = _reflection.GeneratedProtocolMessageType(
+ "CreateDatasetRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_CREATEDATASETREQUEST,
+ __module__="google.cloud.automl_v1beta1.proto.service_pb2",
+ __doc__="""Request message for
+ [AutoMl.CreateDataset][google.cloud.automl.v1beta1.AutoMl.CreateDataset].
+
+
+ Attributes:
+ parent:
+ The resource name of the project to create the dataset for.
+ dataset:
+ The dataset to create.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.CreateDatasetRequest)
+ ),
+)
+_sym_db.RegisterMessage(CreateDatasetRequest)
+
+GetDatasetRequest = _reflection.GeneratedProtocolMessageType(
+ "GetDatasetRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_GETDATASETREQUEST,
+ __module__="google.cloud.automl_v1beta1.proto.service_pb2",
+ __doc__="""Request message for
+ [AutoMl.GetDataset][google.cloud.automl.v1beta1.AutoMl.GetDataset].
+
+
+ Attributes:
+ name:
+ The resource name of the dataset to retrieve.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.GetDatasetRequest)
+ ),
+)
+_sym_db.RegisterMessage(GetDatasetRequest)
+
+ListDatasetsRequest = _reflection.GeneratedProtocolMessageType(
+ "ListDatasetsRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_LISTDATASETSREQUEST,
+ __module__="google.cloud.automl_v1beta1.proto.service_pb2",
+ __doc__="""Request message for
+ [AutoMl.ListDatasets][google.cloud.automl.v1beta1.AutoMl.ListDatasets].
+
+
+ Attributes:
+ parent:
+ The resource name of the project from which to list datasets.
+ filter:
+ An expression for filtering the results of the request. -
+ ``dataset_metadata`` - for existence of the case (e.g.
+ image\_classification\_dataset\_metadata:\*). Some examples of
+ using the filter are: -
+ ``translation_dataset_metadata:*`` --> The dataset has
+ translation\_dataset\_metadata.
+ page_size:
+ Requested page size. Server may return fewer results than
+ requested. If unspecified, server will pick a default size.
+ page_token:
+ A token identifying a page of results for the server to return
+ Typically obtained via [ListDatasetsResponse.next\_page\_token
+ ][google.cloud.automl.v1beta1.ListDatasetsResponse.next\_page\
+ _token] of the previous [AutoMl.ListDatasets][google.cloud.aut
+ oml.v1beta1.AutoMl.ListDatasets] call.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ListDatasetsRequest)
+ ),
+)
+_sym_db.RegisterMessage(ListDatasetsRequest)
+
+ListDatasetsResponse = _reflection.GeneratedProtocolMessageType(
+ "ListDatasetsResponse",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_LISTDATASETSRESPONSE,
+ __module__="google.cloud.automl_v1beta1.proto.service_pb2",
+ __doc__="""Response message for
+ [AutoMl.ListDatasets][google.cloud.automl.v1beta1.AutoMl.ListDatasets].
+
+
+ Attributes:
+ datasets:
+ The datasets read.
+ next_page_token:
+ A token to retrieve next page of results. Pass to [ListDataset
+ sRequest.page\_token][google.cloud.automl.v1beta1.ListDatasets
+ Request.page\_token] to obtain that page.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ListDatasetsResponse)
+ ),
+)
+_sym_db.RegisterMessage(ListDatasetsResponse)
+
+UpdateDatasetRequest = _reflection.GeneratedProtocolMessageType(
+ "UpdateDatasetRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_UPDATEDATASETREQUEST,
+ __module__="google.cloud.automl_v1beta1.proto.service_pb2",
+ __doc__="""Request message for
+ [AutoMl.UpdateDataset][google.cloud.automl.v1beta1.AutoMl.UpdateDataset]
+
+
+ Attributes:
+ dataset:
+ The dataset which replaces the resource on the server.
+ update_mask:
+ The update mask applies to the resource.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.UpdateDatasetRequest)
+ ),
+)
+_sym_db.RegisterMessage(UpdateDatasetRequest)
+
+DeleteDatasetRequest = _reflection.GeneratedProtocolMessageType(
+ "DeleteDatasetRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_DELETEDATASETREQUEST,
+ __module__="google.cloud.automl_v1beta1.proto.service_pb2",
+ __doc__="""Request message for
+ [AutoMl.DeleteDataset][google.cloud.automl.v1beta1.AutoMl.DeleteDataset].
+
+
+ Attributes:
+ name:
+ The resource name of the dataset to delete.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.DeleteDatasetRequest)
+ ),
+)
+_sym_db.RegisterMessage(DeleteDatasetRequest)
+
+ImportDataRequest = _reflection.GeneratedProtocolMessageType(
+ "ImportDataRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_IMPORTDATAREQUEST,
+ __module__="google.cloud.automl_v1beta1.proto.service_pb2",
+ __doc__="""Request message for
+ [AutoMl.ImportData][google.cloud.automl.v1beta1.AutoMl.ImportData].
+
+
+ Attributes:
+ name:
+ Required. Dataset name. Dataset must already exist. All
+ imported annotations and examples will be added.
+ input_config:
+ Required. The desired input location and its domain specific
+ semantics, if any.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ImportDataRequest)
+ ),
+)
+_sym_db.RegisterMessage(ImportDataRequest)
+
+ExportDataRequest = _reflection.GeneratedProtocolMessageType(
+ "ExportDataRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_EXPORTDATAREQUEST,
+ __module__="google.cloud.automl_v1beta1.proto.service_pb2",
+ __doc__="""Request message for
+ [AutoMl.ExportData][google.cloud.automl.v1beta1.AutoMl.ExportData].
+
+
+ Attributes:
+ name:
+ Required. The resource name of the dataset.
+ output_config:
+ Required. The desired output location.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ExportDataRequest)
+ ),
+)
+_sym_db.RegisterMessage(ExportDataRequest)
+
+GetAnnotationSpecRequest = _reflection.GeneratedProtocolMessageType(
+ "GetAnnotationSpecRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_GETANNOTATIONSPECREQUEST,
+ __module__="google.cloud.automl_v1beta1.proto.service_pb2",
+ __doc__="""Request message for
+ [AutoMl.GetAnnotationSpec][google.cloud.automl.v1beta1.AutoMl.GetAnnotationSpec].
+
+
+ Attributes:
+ name:
+ The resource name of the annotation spec to retrieve.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.GetAnnotationSpecRequest)
+ ),
+)
+_sym_db.RegisterMessage(GetAnnotationSpecRequest)
+
+GetTableSpecRequest = _reflection.GeneratedProtocolMessageType(
+ "GetTableSpecRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_GETTABLESPECREQUEST,
+ __module__="google.cloud.automl_v1beta1.proto.service_pb2",
+ __doc__="""Request message for
+ [AutoMl.GetTableSpec][google.cloud.automl.v1beta1.AutoMl.GetTableSpec].
+
+
+ Attributes:
+ name:
+ The resource name of the table spec to retrieve.
+ field_mask:
+ Mask specifying which fields to read.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.GetTableSpecRequest)
+ ),
+)
+_sym_db.RegisterMessage(GetTableSpecRequest)
+
+ListTableSpecsRequest = _reflection.GeneratedProtocolMessageType(
+ "ListTableSpecsRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_LISTTABLESPECSREQUEST,
+ __module__="google.cloud.automl_v1beta1.proto.service_pb2",
+ __doc__="""Request message for
+ [AutoMl.ListTableSpecs][google.cloud.automl.v1beta1.AutoMl.ListTableSpecs].
+
+
+ Attributes:
+ parent:
+ The resource name of the dataset to list table specs from.
+ field_mask:
+ Mask specifying which fields to read.
+ filter:
+ Filter expression, see go/filtering.
+ page_size:
+ Requested page size. The server can return fewer results than
+ requested. If unspecified, the server will pick a default
+ size.
+ page_token:
+ A token identifying a page of results for the server to
+ return. Typically obtained from the [ListTableSpecsResponse.ne
+ xt\_page\_token][google.cloud.automl.v1beta1.ListTableSpecsRes
+ ponse.next\_page\_token] field of the previous [AutoMl.ListTab
+ leSpecs][google.cloud.automl.v1beta1.AutoMl.ListTableSpecs]
+ call.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ListTableSpecsRequest)
+ ),
+)
+_sym_db.RegisterMessage(ListTableSpecsRequest)
+
+ListTableSpecsResponse = _reflection.GeneratedProtocolMessageType(
+ "ListTableSpecsResponse",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_LISTTABLESPECSRESPONSE,
+ __module__="google.cloud.automl_v1beta1.proto.service_pb2",
+ __doc__="""Response message for
+ [AutoMl.ListTableSpecs][google.cloud.automl.v1beta1.AutoMl.ListTableSpecs].
+
+
+ Attributes:
+ table_specs:
+ The table specs read.
+ next_page_token:
+ A token to retrieve next page of results. Pass to [ListTableSp
+ ecsRequest.page\_token][google.cloud.automl.v1beta1.ListTableS
+ pecsRequest.page\_token] to obtain that page.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ListTableSpecsResponse)
+ ),
+)
+_sym_db.RegisterMessage(ListTableSpecsResponse)
+
+UpdateTableSpecRequest = _reflection.GeneratedProtocolMessageType(
+ "UpdateTableSpecRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_UPDATETABLESPECREQUEST,
+ __module__="google.cloud.automl_v1beta1.proto.service_pb2",
+ __doc__="""Request message for
+ [AutoMl.UpdateTableSpec][google.cloud.automl.v1beta1.AutoMl.UpdateTableSpec]
+
+
+ Attributes:
+ table_spec:
+ The table spec which replaces the resource on the server.
+ update_mask:
+ The update mask applies to the resource.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.UpdateTableSpecRequest)
+ ),
+)
+_sym_db.RegisterMessage(UpdateTableSpecRequest)
+
+GetColumnSpecRequest = _reflection.GeneratedProtocolMessageType(
+ "GetColumnSpecRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_GETCOLUMNSPECREQUEST,
+ __module__="google.cloud.automl_v1beta1.proto.service_pb2",
+ __doc__="""Request message for
+ [AutoMl.GetColumnSpec][google.cloud.automl.v1beta1.AutoMl.GetColumnSpec].
+
+
+ Attributes:
+ name:
+ The resource name of the column spec to retrieve.
+ field_mask:
+ Mask specifying which fields to read.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.GetColumnSpecRequest)
+ ),
+)
+_sym_db.RegisterMessage(GetColumnSpecRequest)
+
+ListColumnSpecsRequest = _reflection.GeneratedProtocolMessageType(
+ "ListColumnSpecsRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_LISTCOLUMNSPECSREQUEST,
+ __module__="google.cloud.automl_v1beta1.proto.service_pb2",
+ __doc__="""Request message for
+ [AutoMl.ListColumnSpecs][google.cloud.automl.v1beta1.AutoMl.ListColumnSpecs].
+
+
+ Attributes:
+ parent:
+ The resource name of the table spec to list column specs from.
+ field_mask:
+ Mask specifying which fields to read.
+ filter:
+ Filter expression, see go/filtering.
+ page_size:
+ Requested page size. The server can return fewer results than
+ requested. If unspecified, the server will pick a default
+ size.
+ page_token:
+ A token identifying a page of results for the server to
+ return. Typically obtained from the [ListColumnSpecsResponse.n
+ ext\_page\_token][google.cloud.automl.v1beta1.ListColumnSpecsR
+ esponse.next\_page\_token] field of the previous [AutoMl.ListC
+ olumnSpecs][google.cloud.automl.v1beta1.AutoMl.ListColumnSpecs
+ ] call.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ListColumnSpecsRequest)
+ ),
+)
+_sym_db.RegisterMessage(ListColumnSpecsRequest)
+
+ListColumnSpecsResponse = _reflection.GeneratedProtocolMessageType(
+ "ListColumnSpecsResponse",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_LISTCOLUMNSPECSRESPONSE,
+ __module__="google.cloud.automl_v1beta1.proto.service_pb2",
+ __doc__="""Response message for
+ [AutoMl.ListColumnSpecs][google.cloud.automl.v1beta1.AutoMl.ListColumnSpecs].
+
+
+ Attributes:
+ column_specs:
+ The column specs read.
+ next_page_token:
+ A token to retrieve next page of results. Pass to [ListColumnS
+ pecsRequest.page\_token][google.cloud.automl.v1beta1.ListColum
+ nSpecsRequest.page\_token] to obtain that page.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ListColumnSpecsResponse)
+ ),
+)
+_sym_db.RegisterMessage(ListColumnSpecsResponse)
+
+UpdateColumnSpecRequest = _reflection.GeneratedProtocolMessageType(
+ "UpdateColumnSpecRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_UPDATECOLUMNSPECREQUEST,
+ __module__="google.cloud.automl_v1beta1.proto.service_pb2",
+ __doc__="""Request message for
+ [AutoMl.UpdateColumnSpec][google.cloud.automl.v1beta1.AutoMl.UpdateColumnSpec]
+
+
+ Attributes:
+ column_spec:
+ The column spec which replaces the resource on the server.
+ update_mask:
+ The update mask applies to the resource.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.UpdateColumnSpecRequest)
+ ),
+)
+_sym_db.RegisterMessage(UpdateColumnSpecRequest)
+
+CreateModelRequest = _reflection.GeneratedProtocolMessageType(
+ "CreateModelRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_CREATEMODELREQUEST,
+ __module__="google.cloud.automl_v1beta1.proto.service_pb2",
+ __doc__="""Request message for
+ [AutoMl.CreateModel][google.cloud.automl.v1beta1.AutoMl.CreateModel].
+
+
+ Attributes:
+ parent:
+ Resource name of the parent project where the model is being
+ created.
+ model:
+ The model to create.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.CreateModelRequest)
+ ),
+)
+_sym_db.RegisterMessage(CreateModelRequest)
+
+GetModelRequest = _reflection.GeneratedProtocolMessageType(
+ "GetModelRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_GETMODELREQUEST,
+ __module__="google.cloud.automl_v1beta1.proto.service_pb2",
+ __doc__="""Request message for
+ [AutoMl.GetModel][google.cloud.automl.v1beta1.AutoMl.GetModel].
+
+
+ Attributes:
+ name:
+ Resource name of the model.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.GetModelRequest)
+ ),
+)
+_sym_db.RegisterMessage(GetModelRequest)
+
+ListModelsRequest = _reflection.GeneratedProtocolMessageType(
+ "ListModelsRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_LISTMODELSREQUEST,
+ __module__="google.cloud.automl_v1beta1.proto.service_pb2",
+ __doc__="""Request message for
+ [AutoMl.ListModels][google.cloud.automl.v1beta1.AutoMl.ListModels].
+
+
+ Attributes:
+ parent:
+ Resource name of the project, from which to list the models.
+ filter:
+ An expression for filtering the results of the request. -
+ ``model_metadata`` - for existence of the case (e.g.
+ video\_classification\_model\_metadata:\*). - ``dataset_id``
+ - for = or !=. Some examples of using the filter are: -
+ ``image_classification_model_metadata:*`` --> The model has
+ image\_classification\_model\_metadata. - ``dataset_id=5``
+ --> The model was created from a dataset with ID 5.
+ page_size:
+ Requested page size.
+ page_token:
+ A token identifying a page of results for the server to return
+ Typically obtained via [ListModelsResponse.next\_page\_token][
+ google.cloud.automl.v1beta1.ListModelsResponse.next\_page\_tok
+ en] of the previous [AutoMl.ListModels][google.cloud.automl.v1
+ beta1.AutoMl.ListModels] call.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ListModelsRequest)
+ ),
+)
+_sym_db.RegisterMessage(ListModelsRequest)
+
+ListModelsResponse = _reflection.GeneratedProtocolMessageType(
+ "ListModelsResponse",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_LISTMODELSRESPONSE,
+ __module__="google.cloud.automl_v1beta1.proto.service_pb2",
+ __doc__="""Response message for
+ [AutoMl.ListModels][google.cloud.automl.v1beta1.AutoMl.ListModels].
+
+
+ Attributes:
+ model:
+ List of models in the requested page.
+ next_page_token:
+ A token to retrieve next page of results. Pass to [ListModelsR
+ equest.page\_token][google.cloud.automl.v1beta1.ListModelsRequ
+ est.page\_token] to obtain that page.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ListModelsResponse)
+ ),
+)
+_sym_db.RegisterMessage(ListModelsResponse)
+
+DeleteModelRequest = _reflection.GeneratedProtocolMessageType(
+ "DeleteModelRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_DELETEMODELREQUEST,
+ __module__="google.cloud.automl_v1beta1.proto.service_pb2",
+ __doc__="""Request message for
+ [AutoMl.DeleteModel][google.cloud.automl.v1beta1.AutoMl.DeleteModel].
+
+
+ Attributes:
+ name:
+ Resource name of the model being deleted.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.DeleteModelRequest)
+ ),
+)
+_sym_db.RegisterMessage(DeleteModelRequest)
+
+DeployModelRequest = _reflection.GeneratedProtocolMessageType(
+ "DeployModelRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_DEPLOYMODELREQUEST,
+ __module__="google.cloud.automl_v1beta1.proto.service_pb2",
+ __doc__="""Request message for
+ [AutoMl.DeployModel][google.cloud.automl.v1beta1.AutoMl.DeployModel].
+
+
+ Attributes:
+ model_deployment_metadata:
+ The per-domain specific deployment parameters.
+ image_object_detection_model_deployment_metadata:
+ Model deployment metadata specific to Image Object Detection.
+ name:
+ Resource name of the model to deploy.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.DeployModelRequest)
+ ),
+)
+_sym_db.RegisterMessage(DeployModelRequest)
+
+UndeployModelRequest = _reflection.GeneratedProtocolMessageType(
+ "UndeployModelRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_UNDEPLOYMODELREQUEST,
+ __module__="google.cloud.automl_v1beta1.proto.service_pb2",
+ __doc__="""Request message for
+ [AutoMl.UndeployModel][google.cloud.automl.v1beta1.AutoMl.UndeployModel].
+
+
+ Attributes:
+ name:
+ Resource name of the model to undeploy.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.UndeployModelRequest)
+ ),
+)
+_sym_db.RegisterMessage(UndeployModelRequest)
+
+ExportModelRequest = _reflection.GeneratedProtocolMessageType(
+ "ExportModelRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_EXPORTMODELREQUEST,
+ __module__="google.cloud.automl_v1beta1.proto.service_pb2",
+ __doc__="""Request message for
+ [AutoMl.ExportModel][google.cloud.automl.v1beta1.AutoMl.ExportModel].
+ Models need to be enabled for exporting, otherwise an error code will be
+ returned.
+
+
+ Attributes:
+ name:
+ Required. The resource name of the model to export.
+ output_config:
+ Required. The desired output location and configuration.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ExportModelRequest)
+ ),
+)
+_sym_db.RegisterMessage(ExportModelRequest)
+
+ExportEvaluatedExamplesRequest = _reflection.GeneratedProtocolMessageType(
+ "ExportEvaluatedExamplesRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_EXPORTEVALUATEDEXAMPLESREQUEST,
+ __module__="google.cloud.automl_v1beta1.proto.service_pb2",
+ __doc__="""Request message for
+ [AutoMl.ExportEvaluatedExamples][google.cloud.automl.v1beta1.AutoMl.ExportEvaluatedExamples].
+
+
+ Attributes:
+ name:
+ Required. The resource name of the model whose evaluated
+ examples are to be exported.
+ output_config:
+ Required. The desired output location and configuration.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ExportEvaluatedExamplesRequest)
+ ),
+)
+_sym_db.RegisterMessage(ExportEvaluatedExamplesRequest)
+
+GetModelEvaluationRequest = _reflection.GeneratedProtocolMessageType(
+ "GetModelEvaluationRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_GETMODELEVALUATIONREQUEST,
+ __module__="google.cloud.automl_v1beta1.proto.service_pb2",
+ __doc__="""Request message for
+ [AutoMl.GetModelEvaluation][google.cloud.automl.v1beta1.AutoMl.GetModelEvaluation].
+
+
+ Attributes:
+ name:
+ Resource name for the model evaluation.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.GetModelEvaluationRequest)
+ ),
+)
+_sym_db.RegisterMessage(GetModelEvaluationRequest)
+
+ListModelEvaluationsRequest = _reflection.GeneratedProtocolMessageType(
+ "ListModelEvaluationsRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_LISTMODELEVALUATIONSREQUEST,
+ __module__="google.cloud.automl_v1beta1.proto.service_pb2",
+ __doc__="""Request message for
+ [AutoMl.ListModelEvaluations][google.cloud.automl.v1beta1.AutoMl.ListModelEvaluations].
+
+
+ Attributes:
+ parent:
+ Resource name of the model to list the model evaluations for.
+ If modelId is set as "-", this will list model evaluations
+ from across all models of the parent location.
+ filter:
+ An expression for filtering the results of the request. -
+ ``annotation_spec_id`` - for =, != or existence. See example
+ below for the last. Some examples of using the filter are:
+ - ``annotation_spec_id!=4`` --> The model evaluation was done
+ for annotation spec with ID different than 4. - ``NOT
+ annotation_spec_id:*`` --> The model evaluation was done for
+ aggregate of all annotation specs.
+ page_size:
+ Requested page size.
+ page_token:
+ A token identifying a page of results for the server to
+ return. Typically obtained via [ListModelEvaluationsResponse.n
+ ext\_page\_token][google.cloud.automl.v1beta1.ListModelEvaluat
+ ionsResponse.next\_page\_token] of the previous [AutoMl.ListMo
+ delEvaluations][google.cloud.automl.v1beta1.AutoMl.ListModelEv
+ aluations] call.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ListModelEvaluationsRequest)
+ ),
+)
+_sym_db.RegisterMessage(ListModelEvaluationsRequest)
+
+ListModelEvaluationsResponse = _reflection.GeneratedProtocolMessageType(
+ "ListModelEvaluationsResponse",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_LISTMODELEVALUATIONSRESPONSE,
+ __module__="google.cloud.automl_v1beta1.proto.service_pb2",
+ __doc__="""Response message for
+ [AutoMl.ListModelEvaluations][google.cloud.automl.v1beta1.AutoMl.ListModelEvaluations].
+
+
+ Attributes:
+ model_evaluation:
+ List of model evaluations in the requested page.
+ next_page_token:
+ A token to retrieve next page of results. Pass to the [ListMod
+ elEvaluationsRequest.page\_token][google.cloud.automl.v1beta1.
+ ListModelEvaluationsRequest.page\_token] field of a new [AutoM
+ l.ListModelEvaluations][google.cloud.automl.v1beta1.AutoMl.Lis
+ tModelEvaluations] request to obtain that page.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.ListModelEvaluationsResponse)
+ ),
+)
+_sym_db.RegisterMessage(ListModelEvaluationsResponse)
+
+
+DESCRIPTOR._options = None
+
+_AUTOML = _descriptor.ServiceDescriptor(
+ name="AutoMl",
+ full_name="google.cloud.automl.v1beta1.AutoMl",
+ file=DESCRIPTOR,
+ index=0,
+ serialized_options=_b(
+ "\312A\025automl.googleapis.com\322A.https://www.googleapis.com/auth/cloud-platform"
+ ),
+ serialized_start=3539,
+ serialized_end=7980,
+ methods=[
+ _descriptor.MethodDescriptor(
+ name="CreateDataset",
+ full_name="google.cloud.automl.v1beta1.AutoMl.CreateDataset",
+ index=0,
+ containing_service=None,
+ input_type=_CREATEDATASETREQUEST,
+ output_type=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_dataset__pb2._DATASET,
+ serialized_options=_b(
+ '\202\323\344\223\002<"1/v1beta1/{parent=projects/*/locations/*}/datasets:\007dataset'
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="GetDataset",
+ full_name="google.cloud.automl.v1beta1.AutoMl.GetDataset",
+ index=1,
+ containing_service=None,
+ input_type=_GETDATASETREQUEST,
+ output_type=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_dataset__pb2._DATASET,
+ serialized_options=_b(
+ "\202\323\344\223\0023\0221/v1beta1/{name=projects/*/locations/*/datasets/*}"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="ListDatasets",
+ full_name="google.cloud.automl.v1beta1.AutoMl.ListDatasets",
+ index=2,
+ containing_service=None,
+ input_type=_LISTDATASETSREQUEST,
+ output_type=_LISTDATASETSRESPONSE,
+ serialized_options=_b(
+ "\202\323\344\223\0023\0221/v1beta1/{parent=projects/*/locations/*}/datasets"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="UpdateDataset",
+ full_name="google.cloud.automl.v1beta1.AutoMl.UpdateDataset",
+ index=3,
+ containing_service=None,
+ input_type=_UPDATEDATASETREQUEST,
+ output_type=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_dataset__pb2._DATASET,
+ serialized_options=_b(
+ "\202\323\344\223\002D29/v1beta1/{dataset.name=projects/*/locations/*/datasets/*}:\007dataset"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="DeleteDataset",
+ full_name="google.cloud.automl.v1beta1.AutoMl.DeleteDataset",
+ index=4,
+ containing_service=None,
+ input_type=_DELETEDATASETREQUEST,
+ output_type=google_dot_longrunning_dot_operations__pb2._OPERATION,
+ serialized_options=_b(
+ "\202\323\344\223\0023*1/v1beta1/{name=projects/*/locations/*/datasets/*}"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="ImportData",
+ full_name="google.cloud.automl.v1beta1.AutoMl.ImportData",
+ index=5,
+ containing_service=None,
+ input_type=_IMPORTDATAREQUEST,
+ output_type=google_dot_longrunning_dot_operations__pb2._OPERATION,
+ serialized_options=_b(
+ '\202\323\344\223\002A"/v1beta1/{name=projects/*/locations/*/datasets/*/tableSpecs/*}"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="ListTableSpecs",
+ full_name="google.cloud.automl.v1beta1.AutoMl.ListTableSpecs",
+ index=9,
+ containing_service=None,
+ input_type=_LISTTABLESPECSREQUEST,
+ output_type=_LISTTABLESPECSRESPONSE,
+ serialized_options=_b(
+ "\202\323\344\223\002@\022>/v1beta1/{parent=projects/*/locations/*/datasets/*}/tableSpecs"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="UpdateTableSpec",
+ full_name="google.cloud.automl.v1beta1.AutoMl.UpdateTableSpec",
+ index=10,
+ containing_service=None,
+ input_type=_UPDATETABLESPECREQUEST,
+ output_type=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_table__spec__pb2._TABLESPEC,
+ serialized_options=_b(
+ "\202\323\344\223\002W2I/v1beta1/{table_spec.name=projects/*/locations/*/datasets/*/tableSpecs/*}:\ntable_spec"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="GetColumnSpec",
+ full_name="google.cloud.automl.v1beta1.AutoMl.GetColumnSpec",
+ index=11,
+ containing_service=None,
+ input_type=_GETCOLUMNSPECREQUEST,
+ output_type=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_column__spec__pb2._COLUMNSPEC,
+ serialized_options=_b(
+ "\202\323\344\223\002N\022L/v1beta1/{name=projects/*/locations/*/datasets/*/tableSpecs/*/columnSpecs/*}"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="ListColumnSpecs",
+ full_name="google.cloud.automl.v1beta1.AutoMl.ListColumnSpecs",
+ index=12,
+ containing_service=None,
+ input_type=_LISTCOLUMNSPECSREQUEST,
+ output_type=_LISTCOLUMNSPECSRESPONSE,
+ serialized_options=_b(
+ "\202\323\344\223\002N\022L/v1beta1/{parent=projects/*/locations/*/datasets/*/tableSpecs/*}/columnSpecs"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="UpdateColumnSpec",
+ full_name="google.cloud.automl.v1beta1.AutoMl.UpdateColumnSpec",
+ index=13,
+ containing_service=None,
+ input_type=_UPDATECOLUMNSPECREQUEST,
+ output_type=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_column__spec__pb2._COLUMNSPEC,
+ serialized_options=_b(
+ "\202\323\344\223\002g2X/v1beta1/{column_spec.name=projects/*/locations/*/datasets/*/tableSpecs/*/columnSpecs/*}:\013column_spec"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="CreateModel",
+ full_name="google.cloud.automl.v1beta1.AutoMl.CreateModel",
+ index=14,
+ containing_service=None,
+ input_type=_CREATEMODELREQUEST,
+ output_type=google_dot_longrunning_dot_operations__pb2._OPERATION,
+ serialized_options=_b(
+ '\202\323\344\223\0028"//v1beta1/{parent=projects/*/locations/*}/models:\005model'
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="GetModel",
+ full_name="google.cloud.automl.v1beta1.AutoMl.GetModel",
+ index=15,
+ containing_service=None,
+ input_type=_GETMODELREQUEST,
+ output_type=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_model__pb2._MODEL,
+ serialized_options=_b(
+ "\202\323\344\223\0021\022//v1beta1/{name=projects/*/locations/*/models/*}"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="ListModels",
+ full_name="google.cloud.automl.v1beta1.AutoMl.ListModels",
+ index=16,
+ containing_service=None,
+ input_type=_LISTMODELSREQUEST,
+ output_type=_LISTMODELSRESPONSE,
+ serialized_options=_b(
+ "\202\323\344\223\0021\022//v1beta1/{parent=projects/*/locations/*}/models"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="DeleteModel",
+ full_name="google.cloud.automl.v1beta1.AutoMl.DeleteModel",
+ index=17,
+ containing_service=None,
+ input_type=_DELETEMODELREQUEST,
+ output_type=google_dot_longrunning_dot_operations__pb2._OPERATION,
+ serialized_options=_b(
+ "\202\323\344\223\0021*//v1beta1/{name=projects/*/locations/*/models/*}"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="DeployModel",
+ full_name="google.cloud.automl.v1beta1.AutoMl.DeployModel",
+ index=18,
+ containing_service=None,
+ input_type=_DEPLOYMODELREQUEST,
+ output_type=google_dot_longrunning_dot_operations__pb2._OPERATION,
+ serialized_options=_b(
+ '\202\323\344\223\002;"6/v1beta1/{name=projects/*/locations/*/models/*}:deploy:\001*'
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="UndeployModel",
+ full_name="google.cloud.automl.v1beta1.AutoMl.UndeployModel",
+ index=19,
+ containing_service=None,
+ input_type=_UNDEPLOYMODELREQUEST,
+ output_type=google_dot_longrunning_dot_operations__pb2._OPERATION,
+ serialized_options=_b(
+ '\202\323\344\223\002="8/v1beta1/{name=projects/*/locations/*/models/*}:undeploy:\001*'
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="ExportModel",
+ full_name="google.cloud.automl.v1beta1.AutoMl.ExportModel",
+ index=20,
+ containing_service=None,
+ input_type=_EXPORTMODELREQUEST,
+ output_type=google_dot_longrunning_dot_operations__pb2._OPERATION,
+ serialized_options=_b(
+ '\202\323\344\223\002;"6/v1beta1/{name=projects/*/locations/*/models/*}:export:\001*'
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="ExportEvaluatedExamples",
+ full_name="google.cloud.automl.v1beta1.AutoMl.ExportEvaluatedExamples",
+ index=21,
+ containing_service=None,
+ input_type=_EXPORTEVALUATEDEXAMPLESREQUEST,
+ output_type=google_dot_longrunning_dot_operations__pb2._OPERATION,
+ serialized_options=_b(
+ '\202\323\344\223\002L"G/v1beta1/{name=projects/*/locations/*/models/*}:exportEvaluatedExamples:\001*'
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="GetModelEvaluation",
+ full_name="google.cloud.automl.v1beta1.AutoMl.GetModelEvaluation",
+ index=22,
+ containing_service=None,
+ input_type=_GETMODELEVALUATIONREQUEST,
+ output_type=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_model__evaluation__pb2._MODELEVALUATION,
+ serialized_options=_b(
+ "\202\323\344\223\002D\022B/v1beta1/{name=projects/*/locations/*/models/*/modelEvaluations/*}"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="ListModelEvaluations",
+ full_name="google.cloud.automl.v1beta1.AutoMl.ListModelEvaluations",
+ index=23,
+ containing_service=None,
+ input_type=_LISTMODELEVALUATIONSREQUEST,
+ output_type=_LISTMODELEVALUATIONSRESPONSE,
+ serialized_options=_b(
+ "\202\323\344\223\002D\022B/v1beta1/{parent=projects/*/locations/*/models/*}/modelEvaluations"
+ ),
+ ),
+ ],
+)
+_sym_db.RegisterServiceDescriptor(_AUTOML)
+
+DESCRIPTOR.services_by_name["AutoMl"] = _AUTOML
+
+# @@protoc_insertion_point(module_scope)
diff --git a/automl/google/cloud/automl_v1beta1/proto/service_pb2_grpc.py b/automl/google/cloud/automl_v1beta1/proto/service_pb2_grpc.py
new file mode 100644
index 000000000000..eb049c207ee7
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/service_pb2_grpc.py
@@ -0,0 +1,545 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
+
+from google.cloud.automl_v1beta1.proto import (
+ annotation_spec_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_annotation__spec__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ column_spec_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_column__spec__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ dataset_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_dataset__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ model_evaluation_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_model__evaluation__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ model_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_model__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ service_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ table_spec_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_table__spec__pb2,
+)
+from google.longrunning import (
+ operations_pb2 as google_dot_longrunning_dot_operations__pb2,
+)
+
+
+class AutoMlStub(object):
+ """AutoML Server API.
+
+ The resource names are assigned by the server.
+ The server never reuses names that it has created after the resources with
+ those names are deleted.
+
+ An ID of a resource is the last element of the item's resource name. For
+ `projects/{project_id}/locations/{location_id}/datasets/{dataset_id}`, then
+ the id for the item is `{dataset_id}`.
+
+ Currently the only supported `location_id` is "us-central1".
+
+ On any input that is documented to expect a string parameter in
+ snake_case or kebab-case, either of those cases is accepted.
+ """
+
+ def __init__(self, channel):
+ """Constructor.
+
+ Args:
+ channel: A grpc.Channel.
+ """
+ self.CreateDataset = channel.unary_unary(
+ "/google.cloud.automl.v1beta1.AutoMl/CreateDataset",
+ request_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.CreateDatasetRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_dataset__pb2.Dataset.FromString,
+ )
+ self.GetDataset = channel.unary_unary(
+ "/google.cloud.automl.v1beta1.AutoMl/GetDataset",
+ request_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.GetDatasetRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_dataset__pb2.Dataset.FromString,
+ )
+ self.ListDatasets = channel.unary_unary(
+ "/google.cloud.automl.v1beta1.AutoMl/ListDatasets",
+ request_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.ListDatasetsRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.ListDatasetsResponse.FromString,
+ )
+ self.UpdateDataset = channel.unary_unary(
+ "/google.cloud.automl.v1beta1.AutoMl/UpdateDataset",
+ request_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.UpdateDatasetRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_dataset__pb2.Dataset.FromString,
+ )
+ self.DeleteDataset = channel.unary_unary(
+ "/google.cloud.automl.v1beta1.AutoMl/DeleteDataset",
+ request_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.DeleteDatasetRequest.SerializeToString,
+ response_deserializer=google_dot_longrunning_dot_operations__pb2.Operation.FromString,
+ )
+ self.ImportData = channel.unary_unary(
+ "/google.cloud.automl.v1beta1.AutoMl/ImportData",
+ request_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.ImportDataRequest.SerializeToString,
+ response_deserializer=google_dot_longrunning_dot_operations__pb2.Operation.FromString,
+ )
+ self.ExportData = channel.unary_unary(
+ "/google.cloud.automl.v1beta1.AutoMl/ExportData",
+ request_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.ExportDataRequest.SerializeToString,
+ response_deserializer=google_dot_longrunning_dot_operations__pb2.Operation.FromString,
+ )
+ self.GetAnnotationSpec = channel.unary_unary(
+ "/google.cloud.automl.v1beta1.AutoMl/GetAnnotationSpec",
+ request_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.GetAnnotationSpecRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_annotation__spec__pb2.AnnotationSpec.FromString,
+ )
+ self.GetTableSpec = channel.unary_unary(
+ "/google.cloud.automl.v1beta1.AutoMl/GetTableSpec",
+ request_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.GetTableSpecRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_table__spec__pb2.TableSpec.FromString,
+ )
+ self.ListTableSpecs = channel.unary_unary(
+ "/google.cloud.automl.v1beta1.AutoMl/ListTableSpecs",
+ request_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.ListTableSpecsRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.ListTableSpecsResponse.FromString,
+ )
+ self.UpdateTableSpec = channel.unary_unary(
+ "/google.cloud.automl.v1beta1.AutoMl/UpdateTableSpec",
+ request_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.UpdateTableSpecRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_table__spec__pb2.TableSpec.FromString,
+ )
+ self.GetColumnSpec = channel.unary_unary(
+ "/google.cloud.automl.v1beta1.AutoMl/GetColumnSpec",
+ request_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.GetColumnSpecRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_column__spec__pb2.ColumnSpec.FromString,
+ )
+ self.ListColumnSpecs = channel.unary_unary(
+ "/google.cloud.automl.v1beta1.AutoMl/ListColumnSpecs",
+ request_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.ListColumnSpecsRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.ListColumnSpecsResponse.FromString,
+ )
+ self.UpdateColumnSpec = channel.unary_unary(
+ "/google.cloud.automl.v1beta1.AutoMl/UpdateColumnSpec",
+ request_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.UpdateColumnSpecRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_column__spec__pb2.ColumnSpec.FromString,
+ )
+ self.CreateModel = channel.unary_unary(
+ "/google.cloud.automl.v1beta1.AutoMl/CreateModel",
+ request_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.CreateModelRequest.SerializeToString,
+ response_deserializer=google_dot_longrunning_dot_operations__pb2.Operation.FromString,
+ )
+ self.GetModel = channel.unary_unary(
+ "/google.cloud.automl.v1beta1.AutoMl/GetModel",
+ request_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.GetModelRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_model__pb2.Model.FromString,
+ )
+ self.ListModels = channel.unary_unary(
+ "/google.cloud.automl.v1beta1.AutoMl/ListModels",
+ request_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.ListModelsRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.ListModelsResponse.FromString,
+ )
+ self.DeleteModel = channel.unary_unary(
+ "/google.cloud.automl.v1beta1.AutoMl/DeleteModel",
+ request_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.DeleteModelRequest.SerializeToString,
+ response_deserializer=google_dot_longrunning_dot_operations__pb2.Operation.FromString,
+ )
+ self.DeployModel = channel.unary_unary(
+ "/google.cloud.automl.v1beta1.AutoMl/DeployModel",
+ request_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.DeployModelRequest.SerializeToString,
+ response_deserializer=google_dot_longrunning_dot_operations__pb2.Operation.FromString,
+ )
+ self.UndeployModel = channel.unary_unary(
+ "/google.cloud.automl.v1beta1.AutoMl/UndeployModel",
+ request_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.UndeployModelRequest.SerializeToString,
+ response_deserializer=google_dot_longrunning_dot_operations__pb2.Operation.FromString,
+ )
+ self.ExportModel = channel.unary_unary(
+ "/google.cloud.automl.v1beta1.AutoMl/ExportModel",
+ request_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.ExportModelRequest.SerializeToString,
+ response_deserializer=google_dot_longrunning_dot_operations__pb2.Operation.FromString,
+ )
+ self.ExportEvaluatedExamples = channel.unary_unary(
+ "/google.cloud.automl.v1beta1.AutoMl/ExportEvaluatedExamples",
+ request_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.ExportEvaluatedExamplesRequest.SerializeToString,
+ response_deserializer=google_dot_longrunning_dot_operations__pb2.Operation.FromString,
+ )
+ self.GetModelEvaluation = channel.unary_unary(
+ "/google.cloud.automl.v1beta1.AutoMl/GetModelEvaluation",
+ request_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.GetModelEvaluationRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_model__evaluation__pb2.ModelEvaluation.FromString,
+ )
+ self.ListModelEvaluations = channel.unary_unary(
+ "/google.cloud.automl.v1beta1.AutoMl/ListModelEvaluations",
+ request_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.ListModelEvaluationsRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.ListModelEvaluationsResponse.FromString,
+ )
+
+
+class AutoMlServicer(object):
+ """AutoML Server API.
+
+ The resource names are assigned by the server.
+ The server never reuses names that it has created after the resources with
+ those names are deleted.
+
+ An ID of a resource is the last element of the item's resource name. For
+ `projects/{project_id}/locations/{location_id}/datasets/{dataset_id}`, then
+ the id for the item is `{dataset_id}`.
+
+ Currently the only supported `location_id` is "us-central1".
+
+ On any input that is documented to expect a string parameter in
+ snake_case or kebab-case, either of those cases is accepted.
+ """
+
+ def CreateDataset(self, request, context):
+ """Creates a dataset.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def GetDataset(self, request, context):
+ """Gets a dataset.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def ListDatasets(self, request, context):
+ """Lists datasets in a project.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def UpdateDataset(self, request, context):
+ """Updates a dataset.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def DeleteDataset(self, request, context):
+ """Deletes a dataset and all of its contents.
+ Returns empty response in the
+ [response][google.longrunning.Operation.response] field when it completes,
+ and `delete_details` in the
+ [metadata][google.longrunning.Operation.metadata] field.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def ImportData(self, request, context):
+ """Imports data into a dataset.
+ For Tables this method can only be called on an empty Dataset.
+
+ For Tables:
+ * A
+ [schema_inference_version][google.cloud.automl.v1beta1.InputConfig.params]
+ parameter must be explicitly set.
+ Returns an empty response in the
+ [response][google.longrunning.Operation.response] field when it completes.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def ExportData(self, request, context):
+ """Exports dataset's data to the provided output location.
+ Returns an empty response in the
+ [response][google.longrunning.Operation.response] field when it completes.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def GetAnnotationSpec(self, request, context):
+ """Gets an annotation spec.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def GetTableSpec(self, request, context):
+ """Gets a table spec.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def ListTableSpecs(self, request, context):
+ """Lists table specs in a dataset.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def UpdateTableSpec(self, request, context):
+ """Updates a table spec.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def GetColumnSpec(self, request, context):
+ """Gets a column spec.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def ListColumnSpecs(self, request, context):
+ """Lists column specs in a table spec.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def UpdateColumnSpec(self, request, context):
+ """Updates a column spec.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def CreateModel(self, request, context):
+ """Creates a model.
+ Returns a Model in the [response][google.longrunning.Operation.response]
+ field when it completes.
+ When you create a model, several model evaluations are created for it:
+ a global evaluation, and one evaluation for each annotation spec.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def GetModel(self, request, context):
+ """Gets a model.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def ListModels(self, request, context):
+ """Lists models.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def DeleteModel(self, request, context):
+ """Deletes a model.
+ Returns `google.protobuf.Empty` in the
+ [response][google.longrunning.Operation.response] field when it completes,
+ and `delete_details` in the
+ [metadata][google.longrunning.Operation.metadata] field.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def DeployModel(self, request, context):
+ """Deploys a model. If a model is already deployed, deploying it with the
+ same parameters has no effect. Deploying with different parametrs
+ (as e.g. changing
+
+ [node_number][google.cloud.automl.v1beta1.ImageObjectDetectionModelDeploymentMetadata.node_number])
+ will reset the deployment state without pausing the model's availability.
+
+ Only applicable for Text Classification, Image Object Detection and Tables; all other domains manage deployment automatically.
+
+ Returns an empty response in the
+ [response][google.longrunning.Operation.response] field when it completes.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def UndeployModel(self, request, context):
+ """Undeploys a model. If the model is not deployed this method has no effect.
+
+ Only applicable for Text Classification, Image Object Detection and Tables;
+ all other domains manage deployment automatically.
+
+ Returns an empty response in the
+ [response][google.longrunning.Operation.response] field when it completes.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def ExportModel(self, request, context):
+ """Exports a trained, "export-able", model to a user specified Google Cloud
+ Storage location. A model is considered export-able if and only if it has
+ an export format defined for it in
+
+ [ModelExportOutputConfig][google.cloud.automl.v1beta1.ModelExportOutputConfig].
+
+ Returns an empty response in the
+ [response][google.longrunning.Operation.response] field when it completes.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def ExportEvaluatedExamples(self, request, context):
+ """Exports examples on which the model was evaluated (i.e. which were in the
+ TEST set of the dataset the model was created from), together with their
+ ground truth annotations and the annotations created (predicted) by the
+ model.
+ The examples, ground truth and predictions are exported in the state
+ they were at the moment the model was evaluated.
+
+ This export is available only for 30 days since the model evaluation is
+ created.
+
+ Currently only available for Tables.
+
+ Returns an empty response in the
+ [response][google.longrunning.Operation.response] field when it completes.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def GetModelEvaluation(self, request, context):
+ """Gets a model evaluation.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def ListModelEvaluations(self, request, context):
+ """Lists model evaluations.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+
+def add_AutoMlServicer_to_server(servicer, server):
+ rpc_method_handlers = {
+ "CreateDataset": grpc.unary_unary_rpc_method_handler(
+ servicer.CreateDataset,
+ request_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.CreateDatasetRequest.FromString,
+ response_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_dataset__pb2.Dataset.SerializeToString,
+ ),
+ "GetDataset": grpc.unary_unary_rpc_method_handler(
+ servicer.GetDataset,
+ request_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.GetDatasetRequest.FromString,
+ response_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_dataset__pb2.Dataset.SerializeToString,
+ ),
+ "ListDatasets": grpc.unary_unary_rpc_method_handler(
+ servicer.ListDatasets,
+ request_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.ListDatasetsRequest.FromString,
+ response_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.ListDatasetsResponse.SerializeToString,
+ ),
+ "UpdateDataset": grpc.unary_unary_rpc_method_handler(
+ servicer.UpdateDataset,
+ request_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.UpdateDatasetRequest.FromString,
+ response_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_dataset__pb2.Dataset.SerializeToString,
+ ),
+ "DeleteDataset": grpc.unary_unary_rpc_method_handler(
+ servicer.DeleteDataset,
+ request_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.DeleteDatasetRequest.FromString,
+ response_serializer=google_dot_longrunning_dot_operations__pb2.Operation.SerializeToString,
+ ),
+ "ImportData": grpc.unary_unary_rpc_method_handler(
+ servicer.ImportData,
+ request_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.ImportDataRequest.FromString,
+ response_serializer=google_dot_longrunning_dot_operations__pb2.Operation.SerializeToString,
+ ),
+ "ExportData": grpc.unary_unary_rpc_method_handler(
+ servicer.ExportData,
+ request_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.ExportDataRequest.FromString,
+ response_serializer=google_dot_longrunning_dot_operations__pb2.Operation.SerializeToString,
+ ),
+ "GetAnnotationSpec": grpc.unary_unary_rpc_method_handler(
+ servicer.GetAnnotationSpec,
+ request_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.GetAnnotationSpecRequest.FromString,
+ response_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_annotation__spec__pb2.AnnotationSpec.SerializeToString,
+ ),
+ "GetTableSpec": grpc.unary_unary_rpc_method_handler(
+ servicer.GetTableSpec,
+ request_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.GetTableSpecRequest.FromString,
+ response_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_table__spec__pb2.TableSpec.SerializeToString,
+ ),
+ "ListTableSpecs": grpc.unary_unary_rpc_method_handler(
+ servicer.ListTableSpecs,
+ request_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.ListTableSpecsRequest.FromString,
+ response_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.ListTableSpecsResponse.SerializeToString,
+ ),
+ "UpdateTableSpec": grpc.unary_unary_rpc_method_handler(
+ servicer.UpdateTableSpec,
+ request_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.UpdateTableSpecRequest.FromString,
+ response_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_table__spec__pb2.TableSpec.SerializeToString,
+ ),
+ "GetColumnSpec": grpc.unary_unary_rpc_method_handler(
+ servicer.GetColumnSpec,
+ request_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.GetColumnSpecRequest.FromString,
+ response_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_column__spec__pb2.ColumnSpec.SerializeToString,
+ ),
+ "ListColumnSpecs": grpc.unary_unary_rpc_method_handler(
+ servicer.ListColumnSpecs,
+ request_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.ListColumnSpecsRequest.FromString,
+ response_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.ListColumnSpecsResponse.SerializeToString,
+ ),
+ "UpdateColumnSpec": grpc.unary_unary_rpc_method_handler(
+ servicer.UpdateColumnSpec,
+ request_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.UpdateColumnSpecRequest.FromString,
+ response_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_column__spec__pb2.ColumnSpec.SerializeToString,
+ ),
+ "CreateModel": grpc.unary_unary_rpc_method_handler(
+ servicer.CreateModel,
+ request_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.CreateModelRequest.FromString,
+ response_serializer=google_dot_longrunning_dot_operations__pb2.Operation.SerializeToString,
+ ),
+ "GetModel": grpc.unary_unary_rpc_method_handler(
+ servicer.GetModel,
+ request_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.GetModelRequest.FromString,
+ response_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_model__pb2.Model.SerializeToString,
+ ),
+ "ListModels": grpc.unary_unary_rpc_method_handler(
+ servicer.ListModels,
+ request_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.ListModelsRequest.FromString,
+ response_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.ListModelsResponse.SerializeToString,
+ ),
+ "DeleteModel": grpc.unary_unary_rpc_method_handler(
+ servicer.DeleteModel,
+ request_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.DeleteModelRequest.FromString,
+ response_serializer=google_dot_longrunning_dot_operations__pb2.Operation.SerializeToString,
+ ),
+ "DeployModel": grpc.unary_unary_rpc_method_handler(
+ servicer.DeployModel,
+ request_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.DeployModelRequest.FromString,
+ response_serializer=google_dot_longrunning_dot_operations__pb2.Operation.SerializeToString,
+ ),
+ "UndeployModel": grpc.unary_unary_rpc_method_handler(
+ servicer.UndeployModel,
+ request_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.UndeployModelRequest.FromString,
+ response_serializer=google_dot_longrunning_dot_operations__pb2.Operation.SerializeToString,
+ ),
+ "ExportModel": grpc.unary_unary_rpc_method_handler(
+ servicer.ExportModel,
+ request_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.ExportModelRequest.FromString,
+ response_serializer=google_dot_longrunning_dot_operations__pb2.Operation.SerializeToString,
+ ),
+ "ExportEvaluatedExamples": grpc.unary_unary_rpc_method_handler(
+ servicer.ExportEvaluatedExamples,
+ request_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.ExportEvaluatedExamplesRequest.FromString,
+ response_serializer=google_dot_longrunning_dot_operations__pb2.Operation.SerializeToString,
+ ),
+ "GetModelEvaluation": grpc.unary_unary_rpc_method_handler(
+ servicer.GetModelEvaluation,
+ request_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.GetModelEvaluationRequest.FromString,
+ response_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_model__evaluation__pb2.ModelEvaluation.SerializeToString,
+ ),
+ "ListModelEvaluations": grpc.unary_unary_rpc_method_handler(
+ servicer.ListModelEvaluations,
+ request_deserializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.ListModelEvaluationsRequest.FromString,
+ response_serializer=google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_service__pb2.ListModelEvaluationsResponse.SerializeToString,
+ ),
+ }
+ generic_handler = grpc.method_handlers_generic_handler(
+ "google.cloud.automl.v1beta1.AutoMl", rpc_method_handlers
+ )
+ server.add_generic_rpc_handlers((generic_handler,))
diff --git a/automl/google/cloud/automl_v1beta1/proto/table_spec.proto b/automl/google/cloud/automl_v1beta1/proto/table_spec.proto
new file mode 100644
index 000000000000..4475617ae275
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/table_spec.proto
@@ -0,0 +1,73 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.automl.v1beta1;
+
+import "google/cloud/automl/v1beta1/io.proto";
+import "google/api/annotations.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl";
+option java_multiple_files = true;
+option java_package = "com.google.cloud.automl.v1beta1";
+option php_namespace = "Google\\Cloud\\AutoMl\\V1beta1";
+option ruby_package = "Google::Cloud::AutoML::V1beta1";
+
+// A specification of a relational table.
+// The table's schema is represented via its child column specs. It is
+// pre-populated as part of ImportData by schema inference algorithm, the
+// version of which is a required parameter of ImportData InputConfig.
+// Note: While working with a table, at times the schema may be
+// inconsistent with the data in the table (e.g. string in a FLOAT64 column).
+// The consistency validation is done upon creation of a model.
+// Used by:
+// * Tables
+message TableSpec {
+ // Output only. The resource name of the table spec.
+ // Form:
+ //
+ // `projects/{project_id}/locations/{location_id}/datasets/{dataset_id}/tableSpecs/{table_spec_id}`
+ string name = 1;
+
+ // column_spec_id of the time column. Only used if the parent dataset's
+ // ml_use_column_spec_id is not set. Used to split rows into TRAIN, VALIDATE
+ // and TEST sets such that oldest rows go to TRAIN set, newest to TEST, and
+ // those in between to VALIDATE.
+ // Required type: TIMESTAMP.
+ // If both this column and ml_use_column are not set, then ML use of all rows
+ // will be assigned by AutoML. NOTE: Updates of this field will instantly
+ // affect any other users concurrently working with the dataset.
+ string time_column_spec_id = 2;
+
+ // Output only. The number of rows (i.e. examples) in the table.
+ int64 row_count = 3;
+
+ // Output only. The number of valid rows (i.e. without values that don't match
+ // DataType-s of their columns).
+ int64 valid_row_count = 4;
+
+ // Output only. The number of columns of the table. That is, the number of
+ // child ColumnSpec-s.
+ int64 column_count = 7;
+
+ // Output only. Input configs via which data currently residing in the table
+ // had been imported.
+ repeated InputConfig input_configs = 5;
+
+ // Used to perform consistent read-modify-write updates. If not set, a blind
+ // "overwrite" update happens.
+ string etag = 6;
+}
diff --git a/automl/google/cloud/automl_v1beta1/proto/table_spec_pb2.py b/automl/google/cloud/automl_v1beta1/proto/table_spec_pb2.py
new file mode 100644
index 000000000000..9f5f386f83a2
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/table_spec_pb2.py
@@ -0,0 +1,244 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/automl_v1beta1/proto/table_spec.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.cloud.automl_v1beta1.proto import (
+ io_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_io__pb2,
+)
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/automl_v1beta1/proto/table_spec.proto",
+ package="google.cloud.automl.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\037com.google.cloud.automl.v1beta1P\001ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\312\002\033Google\\Cloud\\AutoMl\\V1beta1\352\002\036Google::Cloud::AutoML::V1beta1"
+ ),
+ serialized_pb=_b(
+ '\n2google/cloud/automl_v1beta1/proto/table_spec.proto\x12\x1bgoogle.cloud.automl.v1beta1\x1a*google/cloud/automl_v1beta1/proto/io.proto\x1a\x1cgoogle/api/annotations.proto"\xc7\x01\n\tTableSpec\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x1b\n\x13time_column_spec_id\x18\x02 \x01(\t\x12\x11\n\trow_count\x18\x03 \x01(\x03\x12\x17\n\x0fvalid_row_count\x18\x04 \x01(\x03\x12\x14\n\x0c\x63olumn_count\x18\x07 \x01(\x03\x12?\n\rinput_configs\x18\x05 \x03(\x0b\x32(.google.cloud.automl.v1beta1.InputConfig\x12\x0c\n\x04\x65tag\x18\x06 \x01(\tB\xa5\x01\n\x1f\x63om.google.cloud.automl.v1beta1P\x01ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\xca\x02\x1bGoogle\\Cloud\\AutoMl\\V1beta1\xea\x02\x1eGoogle::Cloud::AutoML::V1beta1b\x06proto3'
+ ),
+ dependencies=[
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_io__pb2.DESCRIPTOR,
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ ],
+)
+
+
+_TABLESPEC = _descriptor.Descriptor(
+ name="TableSpec",
+ full_name="google.cloud.automl.v1beta1.TableSpec",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.automl.v1beta1.TableSpec.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="time_column_spec_id",
+ full_name="google.cloud.automl.v1beta1.TableSpec.time_column_spec_id",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="row_count",
+ full_name="google.cloud.automl.v1beta1.TableSpec.row_count",
+ index=2,
+ number=3,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="valid_row_count",
+ full_name="google.cloud.automl.v1beta1.TableSpec.valid_row_count",
+ index=3,
+ number=4,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="column_count",
+ full_name="google.cloud.automl.v1beta1.TableSpec.column_count",
+ index=4,
+ number=7,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="input_configs",
+ full_name="google.cloud.automl.v1beta1.TableSpec.input_configs",
+ index=5,
+ number=5,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="etag",
+ full_name="google.cloud.automl.v1beta1.TableSpec.etag",
+ index=6,
+ number=6,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=158,
+ serialized_end=357,
+)
+
+_TABLESPEC.fields_by_name[
+ "input_configs"
+].message_type = google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_io__pb2._INPUTCONFIG
+DESCRIPTOR.message_types_by_name["TableSpec"] = _TABLESPEC
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+TableSpec = _reflection.GeneratedProtocolMessageType(
+ "TableSpec",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TABLESPEC,
+ __module__="google.cloud.automl_v1beta1.proto.table_spec_pb2",
+ __doc__="""A specification of a relational table. The table's schema is represented
+ via its child column specs. It is pre-populated as part of ImportData by
+ schema inference algorithm, the version of which is a required parameter
+ of ImportData InputConfig. Note: While working with a table, at times
+ the schema may be inconsistent with the data in the table (e.g. string
+ in a FLOAT64 column). The consistency validation is done upon creation
+ of a model. Used by: \* Tables
+
+
+ Attributes:
+ name:
+ Output only. The resource name of the table spec. Form: ``pro
+ jects/{project_id}/locations/{location_id}/datasets/{dataset_i
+ d}/tableSpecs/{table_spec_id}``
+ time_column_spec_id:
+ column\_spec\_id of the time column. Only used if the parent
+ dataset's ml\_use\_column\_spec\_id is not set. Used to split
+ rows into TRAIN, VALIDATE and TEST sets such that oldest rows
+ go to TRAIN set, newest to TEST, and those in between to
+ VALIDATE. Required type: TIMESTAMP. If both this column and
+ ml\_use\_column are not set, then ML use of all rows will be
+ assigned by AutoML. NOTE: Updates of this field will instantly
+ affect any other users concurrently working with the dataset.
+ row_count:
+ Output only. The number of rows (i.e. examples) in the table.
+ valid_row_count:
+ Output only. The number of valid rows (i.e. without values
+ that don't match DataType-s of their columns).
+ column_count:
+ Output only. The number of columns of the table. That is, the
+ number of child ColumnSpec-s.
+ input_configs:
+ Output only. Input configs via which data currently residing
+ in the table had been imported.
+ etag:
+ Used to perform consistent read-modify-write updates. If not
+ set, a blind "overwrite" update happens.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.TableSpec)
+ ),
+)
+_sym_db.RegisterMessage(TableSpec)
+
+
+DESCRIPTOR._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/automl/google/cloud/automl_v1beta1/proto/table_spec_pb2_grpc.py b/automl/google/cloud/automl_v1beta1/proto/table_spec_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/table_spec_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/automl/google/cloud/automl_v1beta1/proto/tables.proto b/automl/google/cloud/automl_v1beta1/proto/tables.proto
new file mode 100644
index 000000000000..c8b7809b6cf1
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/tables.proto
@@ -0,0 +1,269 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.automl.v1beta1;
+
+import "google/cloud/automl/v1beta1/classification.proto";
+import "google/cloud/automl/v1beta1/column_spec.proto";
+import "google/cloud/automl/v1beta1/data_items.proto";
+import "google/cloud/automl/v1beta1/data_stats.proto";
+import "google/cloud/automl/v1beta1/ranges.proto";
+import "google/cloud/automl/v1beta1/temporal.proto";
+import "google/protobuf/struct.proto";
+import "google/protobuf/timestamp.proto";
+import "google/api/annotations.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl";
+option java_multiple_files = true;
+option java_package = "com.google.cloud.automl.v1beta1";
+option php_namespace = "Google\\Cloud\\AutoMl\\V1beta1";
+option ruby_package = "Google::Cloud::AutoML::V1beta1";
+
+// Metadata for a dataset used for AutoML Tables.
+message TablesDatasetMetadata {
+ // Output only. The table_spec_id of the primary table of this dataset.
+ string primary_table_spec_id = 1;
+
+ // column_spec_id of the primary table's column that should be used as the
+ // training & prediction target.
+ // This column must be non-nullable and have one of following data types
+ // (otherwise model creation will error):
+ //
+ // * CATEGORY
+ //
+ // * FLOAT64
+ //
+ // If the type is CATEGORY , only up to
+ // 100 unique values may exist in that column across all rows.
+ //
+ // NOTE: Updates of this field will instantly affect any other users
+ // concurrently working with the dataset.
+ string target_column_spec_id = 2;
+
+ // column_spec_id of the primary table's column that should be used as the
+ // weight column, i.e. the higher the value the more important the row will be
+ // during model training.
+ // Required type: FLOAT64.
+ // Allowed values: 0 to 10000, inclusive on both ends; 0 means the row is
+ // ignored for training.
+ // If not set all rows are assumed to have equal weight of 1.
+ // NOTE: Updates of this field will instantly affect any other users
+ // concurrently working with the dataset.
+ string weight_column_spec_id = 3;
+
+ // column_spec_id of the primary table column which specifies a possible ML
+ // use of the row, i.e. the column will be used to split the rows into TRAIN,
+ // VALIDATE and TEST sets.
+ // Required type: STRING.
+ // This column, if set, must either have all of `TRAIN`, `VALIDATE`, `TEST`
+ // among its values, or only have `TEST`, `UNASSIGNED` values. In the latter
+ // case the rows with `UNASSIGNED` value will be assigned by AutoML. Note
+ // that if a given ml use distribution makes it impossible to create a "good"
+ // model, that call will error describing the issue.
+ // If both this column_spec_id and primary table's time_column_spec_id are not
+ // set, then all rows are treated as `UNASSIGNED`.
+ // NOTE: Updates of this field will instantly affect any other users
+ // concurrently working with the dataset.
+ string ml_use_column_spec_id = 4;
+
+ // Output only. Correlations between
+ //
+ // [TablesDatasetMetadata.target_column_spec_id][google.cloud.automl.v1beta1.TablesDatasetMetadata.target_column_spec_id],
+ // and other columns of the
+ //
+ // [TablesDatasetMetadataprimary_table][google.cloud.automl.v1beta1.TablesDatasetMetadata.primary_table_spec_id].
+ // Only set if the target column is set. Mapping from other column spec id to
+ // its CorrelationStats with the target column.
+ // This field may be stale, see the stats_update_time field for
+ // for the timestamp at which these stats were last updated.
+ map target_column_correlations = 6;
+
+ // Output only. The most recent timestamp when target_column_correlations
+ // field and all descendant ColumnSpec.data_stats and
+ // ColumnSpec.top_correlated_columns fields were last (re-)generated. Any
+ // changes that happened to the dataset afterwards are not reflected in these
+ // fields values. The regeneration happens in the background on a best effort
+ // basis.
+ google.protobuf.Timestamp stats_update_time = 7;
+}
+
+// Model metadata specific to AutoML Tables.
+message TablesModelMetadata {
+ // Column spec of the dataset's primary table's column the model is
+ // predicting. Snapshotted when model creation started.
+ // Only 3 fields are used:
+ // name - May be set on CreateModel, if it's not then the ColumnSpec
+ // corresponding to the current target_column_spec_id of the dataset
+ // the model is trained from is used.
+ // If neither is set, CreateModel will error.
+ // display_name - Output only.
+ // data_type - Output only.
+ ColumnSpec target_column_spec = 2;
+
+ // Column specs of the dataset's primary table's columns, on which
+ // the model is trained and which are used as the input for predictions.
+ // The
+ //
+ // [target_column][google.cloud.automl.v1beta1.TablesModelMetadata.target_column_spec]
+ // as well as, according to dataset's state upon model creation,
+ //
+ // [weight_column][google.cloud.automl.v1beta1.TablesDatasetMetadata.weight_column_spec_id],
+ // and
+ //
+ // [ml_use_column][google.cloud.automl.v1beta1.TablesDatasetMetadata.ml_use_column_spec_id]
+ // must never be included here.
+ //
+ // Only 3 fields are used:
+ //
+ // * name - May be set on CreateModel, if set only the columns specified are
+ // used, otherwise all primary table's columns (except the ones listed
+ // above) are used for the training and prediction input.
+ //
+ // * display_name - Output only.
+ //
+ // * data_type - Output only.
+ repeated ColumnSpec input_feature_column_specs = 3;
+
+ // Objective function the model is optimizing towards. The training process
+ // creates a model that maximizes/minimizes the value of the objective
+ // function over the validation set.
+ //
+ // The supported optimization objectives depend on the prediction type.
+ // If the field is not set, a default objective function is used.
+ //
+ // CLASSIFICATION_BINARY:
+ // "MAXIMIZE_AU_ROC" (default) - Maximize the area under the receiver
+ // operating characteristic (ROC) curve.
+ // "MINIMIZE_LOG_LOSS" - Minimize log loss.
+ // "MAXIMIZE_AU_PRC" - Maximize the area under the precision-recall curve.
+ // "MAXIMIZE_PRECISION_AT_RECALL" - Maximize precision for a specified
+ // recall value.
+ // "MAXIMIZE_RECALL_AT_PRECISION" - Maximize recall for a specified
+ // precision value.
+ //
+ // CLASSIFICATION_MULTI_CLASS :
+ // "MINIMIZE_LOG_LOSS" (default) - Minimize log loss.
+ //
+ //
+ // REGRESSION:
+ // "MINIMIZE_RMSE" (default) - Minimize root-mean-squared error (RMSE).
+ // "MINIMIZE_MAE" - Minimize mean-absolute error (MAE).
+ // "MINIMIZE_RMSLE" - Minimize root-mean-squared log error (RMSLE).
+ string optimization_objective = 4;
+
+ // Output only. Auxiliary information for each of the
+ // input_feature_column_specs with respect to this particular model.
+ repeated TablesModelColumnInfo tables_model_column_info = 5;
+
+ // Required. The train budget of creating this model, expressed in milli node
+ // hours i.e. 1,000 value in this field means 1 node hour.
+ //
+ // The training cost of the model will not exceed this budget. The final cost
+ // will be attempted to be close to the budget, though may end up being (even)
+ // noticeably smaller - at the backend's discretion. This especially may
+ // happen when further model training ceases to provide any improvements.
+ //
+ // If the budget is set to a value known to be insufficient to train a
+ // model for the given dataset, the training won't be attempted and
+ // will error.
+ //
+ // The train budget must be between 1,000 and 72,000 milli node hours,
+ // inclusive.
+ int64 train_budget_milli_node_hours = 6;
+
+ // Output only. The actual training cost of the model, expressed in milli
+ // node hours, i.e. 1,000 value in this field means 1 node hour. Guaranteed
+ // to not exceed the train budget.
+ int64 train_cost_milli_node_hours = 7;
+
+ // Use the entire training budget. This disables the early stopping feature.
+ // By default, the early stopping feature is enabled, which means that AutoML
+ // Tables might stop training before the entire training budget has been used.
+ bool disable_early_stopping = 12;
+}
+
+// Contains annotation details specific to Tables.
+message TablesAnnotation {
+ // Output only. A confidence estimate between 0.0 and 1.0, inclusive. A higher
+ // value means greater confidence in the returned value.
+ // For
+ //
+ // [target_column_spec][google.cloud.automl.v1beta1.TablesModelMetadata.target_column_spec]
+ // of FLOAT64 data type the score is not populated.
+ float score = 1;
+
+ // Output only. Only populated when
+ //
+ // [target_column_spec][google.cloud.automl.v1beta1.TablesModelMetadata.target_column_spec]
+ // has FLOAT64 data type. An interval in which the exactly correct target
+ // value has 95% chance to be in.
+ DoubleRange prediction_interval = 4;
+
+ // The predicted value of the row's
+ //
+ // [target_column][google.cloud.automl.v1beta1.TablesModelMetadata.target_column_spec].
+ // The value depends on the column's DataType:
+ //
+ // * CATEGORY - the predicted (with the above confidence `score`) CATEGORY
+ // value.
+ //
+ // * FLOAT64 - the predicted (with above `prediction_interval`) FLOAT64 value.
+ google.protobuf.Value value = 2;
+
+ // Output only. Auxiliary information for each of the model's
+ //
+ // [input_feature_column_specs][google.cloud.automl.v1beta1.TablesModelMetadata.input_feature_column_specs]
+ // with respect to this particular prediction.
+ // If no other fields than
+ //
+ // [column_spec_name][google.cloud.automl.v1beta1.TablesModelColumnInfo.column_spec_name]
+ // and
+ //
+ // [column_display_name][google.cloud.automl.v1beta1.TablesModelColumnInfo.column_display_name]
+ // would be populated, then this whole field is not.
+ repeated TablesModelColumnInfo tables_model_column_info = 3;
+}
+
+// An information specific to given column and Tables Model, in context
+// of the Model and the predictions created by it.
+message TablesModelColumnInfo {
+ // Output only. The name of the ColumnSpec describing the column. Not
+ // populated when this proto is outputted to BigQuery.
+ string column_spec_name = 1;
+
+ // Output only. The display name of the column (same as the display_name of
+ // its ColumnSpec).
+ string column_display_name = 2;
+
+ // Output only. When given as part of a Model (always populated):
+ // Measurement of how much model predictions correctness on the TEST data
+ // depend on values in this column. A value between 0 and 1, higher means
+ // higher influence. These values are normalized - for all input feature
+ // columns of a given model they add to 1.
+ //
+ // When given back by Predict (populated iff
+ // [feature_importance
+ // param][google.cloud.automl.v1beta1.PredictRequest.params] is set) or Batch
+ // Predict (populated iff
+ // [feature_importance][google.cloud.automl.v1beta1.PredictRequest.params]
+ // param is set):
+ // Measurement of how impactful for the prediction returned for the given row
+ // the value in this column was. A value between 0 and 1, higher means larger
+ // impact. These values are normalized - for all input feature columns of a
+ // single predicted row they add to 1.
+ float feature_importance = 3;
+}
diff --git a/automl/google/cloud/automl_v1beta1/proto/tables_pb2.py b/automl/google/cloud/automl_v1beta1/proto/tables_pb2.py
new file mode 100644
index 000000000000..b55cfa1f247b
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/tables_pb2.py
@@ -0,0 +1,855 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/automl_v1beta1/proto/tables.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.cloud.automl_v1beta1.proto import (
+ classification_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_classification__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ column_spec_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_column__spec__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ data_items_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_data__items__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ data_stats_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_data__stats__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ ranges_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_ranges__pb2,
+)
+from google.cloud.automl_v1beta1.proto import (
+ temporal_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_temporal__pb2,
+)
+from google.protobuf import struct_pb2 as google_dot_protobuf_dot_struct__pb2
+from google.protobuf import timestamp_pb2 as google_dot_protobuf_dot_timestamp__pb2
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/automl_v1beta1/proto/tables.proto",
+ package="google.cloud.automl.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\037com.google.cloud.automl.v1beta1P\001ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\312\002\033Google\\Cloud\\AutoMl\\V1beta1\352\002\036Google::Cloud::AutoML::V1beta1"
+ ),
+ serialized_pb=_b(
+ '\n.google/cloud/automl_v1beta1/proto/tables.proto\x12\x1bgoogle.cloud.automl.v1beta1\x1a\x36google/cloud/automl_v1beta1/proto/classification.proto\x1a\x33google/cloud/automl_v1beta1/proto/column_spec.proto\x1a\x32google/cloud/automl_v1beta1/proto/data_items.proto\x1a\x32google/cloud/automl_v1beta1/proto/data_stats.proto\x1a.google/cloud/automl_v1beta1/proto/ranges.proto\x1a\x30google/cloud/automl_v1beta1/proto/temporal.proto\x1a\x1cgoogle/protobuf/struct.proto\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x1cgoogle/api/annotations.proto"\xb0\x03\n\x15TablesDatasetMetadata\x12\x1d\n\x15primary_table_spec_id\x18\x01 \x01(\t\x12\x1d\n\x15target_column_spec_id\x18\x02 \x01(\t\x12\x1d\n\x15weight_column_spec_id\x18\x03 \x01(\t\x12\x1d\n\x15ml_use_column_spec_id\x18\x04 \x01(\t\x12t\n\x1atarget_column_correlations\x18\x06 \x03(\x0b\x32P.google.cloud.automl.v1beta1.TablesDatasetMetadata.TargetColumnCorrelationsEntry\x12\x35\n\x11stats_update_time\x18\x07 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x1an\n\x1dTargetColumnCorrelationsEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12<\n\x05value\x18\x02 \x01(\x0b\x32-.google.cloud.automl.v1beta1.CorrelationStats:\x02\x38\x01"\x89\x03\n\x13TablesModelMetadata\x12\x43\n\x12target_column_spec\x18\x02 \x01(\x0b\x32\'.google.cloud.automl.v1beta1.ColumnSpec\x12K\n\x1ainput_feature_column_specs\x18\x03 \x03(\x0b\x32\'.google.cloud.automl.v1beta1.ColumnSpec\x12\x1e\n\x16optimization_objective\x18\x04 \x01(\t\x12T\n\x18tables_model_column_info\x18\x05 \x03(\x0b\x32\x32.google.cloud.automl.v1beta1.TablesModelColumnInfo\x12%\n\x1dtrain_budget_milli_node_hours\x18\x06 \x01(\x03\x12#\n\x1btrain_cost_milli_node_hours\x18\x07 \x01(\x03\x12\x1e\n\x16\x64isable_early_stopping\x18\x0c \x01(\x08"\xe5\x01\n\x10TablesAnnotation\x12\r\n\x05score\x18\x01 \x01(\x02\x12\x45\n\x13prediction_interval\x18\x04 \x01(\x0b\x32(.google.cloud.automl.v1beta1.DoubleRange\x12%\n\x05value\x18\x02 \x01(\x0b\x32\x16.google.protobuf.Value\x12T\n\x18tables_model_column_info\x18\x03 \x03(\x0b\x32\x32.google.cloud.automl.v1beta1.TablesModelColumnInfo"j\n\x15TablesModelColumnInfo\x12\x18\n\x10\x63olumn_spec_name\x18\x01 \x01(\t\x12\x1b\n\x13\x63olumn_display_name\x18\x02 \x01(\t\x12\x1a\n\x12\x66\x65\x61ture_importance\x18\x03 \x01(\x02\x42\xa5\x01\n\x1f\x63om.google.cloud.automl.v1beta1P\x01ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\xca\x02\x1bGoogle\\Cloud\\AutoMl\\V1beta1\xea\x02\x1eGoogle::Cloud::AutoML::V1beta1b\x06proto3'
+ ),
+ dependencies=[
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_classification__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_column__spec__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_data__items__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_data__stats__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_ranges__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_temporal__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_struct__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_timestamp__pb2.DESCRIPTOR,
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ ],
+)
+
+
+_TABLESDATASETMETADATA_TARGETCOLUMNCORRELATIONSENTRY = _descriptor.Descriptor(
+ name="TargetColumnCorrelationsEntry",
+ full_name="google.cloud.automl.v1beta1.TablesDatasetMetadata.TargetColumnCorrelationsEntry",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="key",
+ full_name="google.cloud.automl.v1beta1.TablesDatasetMetadata.TargetColumnCorrelationsEntry.key",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="value",
+ full_name="google.cloud.automl.v1beta1.TablesDatasetMetadata.TargetColumnCorrelationsEntry.value",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=_b("8\001"),
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=806,
+ serialized_end=916,
+)
+
+_TABLESDATASETMETADATA = _descriptor.Descriptor(
+ name="TablesDatasetMetadata",
+ full_name="google.cloud.automl.v1beta1.TablesDatasetMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="primary_table_spec_id",
+ full_name="google.cloud.automl.v1beta1.TablesDatasetMetadata.primary_table_spec_id",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="target_column_spec_id",
+ full_name="google.cloud.automl.v1beta1.TablesDatasetMetadata.target_column_spec_id",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="weight_column_spec_id",
+ full_name="google.cloud.automl.v1beta1.TablesDatasetMetadata.weight_column_spec_id",
+ index=2,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="ml_use_column_spec_id",
+ full_name="google.cloud.automl.v1beta1.TablesDatasetMetadata.ml_use_column_spec_id",
+ index=3,
+ number=4,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="target_column_correlations",
+ full_name="google.cloud.automl.v1beta1.TablesDatasetMetadata.target_column_correlations",
+ index=4,
+ number=6,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="stats_update_time",
+ full_name="google.cloud.automl.v1beta1.TablesDatasetMetadata.stats_update_time",
+ index=5,
+ number=7,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[_TABLESDATASETMETADATA_TARGETCOLUMNCORRELATIONSENTRY],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=484,
+ serialized_end=916,
+)
+
+
+_TABLESMODELMETADATA = _descriptor.Descriptor(
+ name="TablesModelMetadata",
+ full_name="google.cloud.automl.v1beta1.TablesModelMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="target_column_spec",
+ full_name="google.cloud.automl.v1beta1.TablesModelMetadata.target_column_spec",
+ index=0,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="input_feature_column_specs",
+ full_name="google.cloud.automl.v1beta1.TablesModelMetadata.input_feature_column_specs",
+ index=1,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="optimization_objective",
+ full_name="google.cloud.automl.v1beta1.TablesModelMetadata.optimization_objective",
+ index=2,
+ number=4,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="tables_model_column_info",
+ full_name="google.cloud.automl.v1beta1.TablesModelMetadata.tables_model_column_info",
+ index=3,
+ number=5,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="train_budget_milli_node_hours",
+ full_name="google.cloud.automl.v1beta1.TablesModelMetadata.train_budget_milli_node_hours",
+ index=4,
+ number=6,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="train_cost_milli_node_hours",
+ full_name="google.cloud.automl.v1beta1.TablesModelMetadata.train_cost_milli_node_hours",
+ index=5,
+ number=7,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="disable_early_stopping",
+ full_name="google.cloud.automl.v1beta1.TablesModelMetadata.disable_early_stopping",
+ index=6,
+ number=12,
+ type=8,
+ cpp_type=7,
+ label=1,
+ has_default_value=False,
+ default_value=False,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=919,
+ serialized_end=1312,
+)
+
+
+_TABLESANNOTATION = _descriptor.Descriptor(
+ name="TablesAnnotation",
+ full_name="google.cloud.automl.v1beta1.TablesAnnotation",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="score",
+ full_name="google.cloud.automl.v1beta1.TablesAnnotation.score",
+ index=0,
+ number=1,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="prediction_interval",
+ full_name="google.cloud.automl.v1beta1.TablesAnnotation.prediction_interval",
+ index=1,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="value",
+ full_name="google.cloud.automl.v1beta1.TablesAnnotation.value",
+ index=2,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="tables_model_column_info",
+ full_name="google.cloud.automl.v1beta1.TablesAnnotation.tables_model_column_info",
+ index=3,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1315,
+ serialized_end=1544,
+)
+
+
+_TABLESMODELCOLUMNINFO = _descriptor.Descriptor(
+ name="TablesModelColumnInfo",
+ full_name="google.cloud.automl.v1beta1.TablesModelColumnInfo",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="column_spec_name",
+ full_name="google.cloud.automl.v1beta1.TablesModelColumnInfo.column_spec_name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="column_display_name",
+ full_name="google.cloud.automl.v1beta1.TablesModelColumnInfo.column_display_name",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="feature_importance",
+ full_name="google.cloud.automl.v1beta1.TablesModelColumnInfo.feature_importance",
+ index=2,
+ number=3,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1546,
+ serialized_end=1652,
+)
+
+_TABLESDATASETMETADATA_TARGETCOLUMNCORRELATIONSENTRY.fields_by_name[
+ "value"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_data__stats__pb2._CORRELATIONSTATS
+)
+_TABLESDATASETMETADATA_TARGETCOLUMNCORRELATIONSENTRY.containing_type = (
+ _TABLESDATASETMETADATA
+)
+_TABLESDATASETMETADATA.fields_by_name[
+ "target_column_correlations"
+].message_type = _TABLESDATASETMETADATA_TARGETCOLUMNCORRELATIONSENTRY
+_TABLESDATASETMETADATA.fields_by_name[
+ "stats_update_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_TABLESMODELMETADATA.fields_by_name[
+ "target_column_spec"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_column__spec__pb2._COLUMNSPEC
+)
+_TABLESMODELMETADATA.fields_by_name[
+ "input_feature_column_specs"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_column__spec__pb2._COLUMNSPEC
+)
+_TABLESMODELMETADATA.fields_by_name[
+ "tables_model_column_info"
+].message_type = _TABLESMODELCOLUMNINFO
+_TABLESANNOTATION.fields_by_name[
+ "prediction_interval"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_ranges__pb2._DOUBLERANGE
+)
+_TABLESANNOTATION.fields_by_name[
+ "value"
+].message_type = google_dot_protobuf_dot_struct__pb2._VALUE
+_TABLESANNOTATION.fields_by_name[
+ "tables_model_column_info"
+].message_type = _TABLESMODELCOLUMNINFO
+DESCRIPTOR.message_types_by_name["TablesDatasetMetadata"] = _TABLESDATASETMETADATA
+DESCRIPTOR.message_types_by_name["TablesModelMetadata"] = _TABLESMODELMETADATA
+DESCRIPTOR.message_types_by_name["TablesAnnotation"] = _TABLESANNOTATION
+DESCRIPTOR.message_types_by_name["TablesModelColumnInfo"] = _TABLESMODELCOLUMNINFO
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+TablesDatasetMetadata = _reflection.GeneratedProtocolMessageType(
+ "TablesDatasetMetadata",
+ (_message.Message,),
+ dict(
+ TargetColumnCorrelationsEntry=_reflection.GeneratedProtocolMessageType(
+ "TargetColumnCorrelationsEntry",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TABLESDATASETMETADATA_TARGETCOLUMNCORRELATIONSENTRY,
+ __module__="google.cloud.automl_v1beta1.proto.tables_pb2"
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.TablesDatasetMetadata.TargetColumnCorrelationsEntry)
+ ),
+ ),
+ DESCRIPTOR=_TABLESDATASETMETADATA,
+ __module__="google.cloud.automl_v1beta1.proto.tables_pb2",
+ __doc__="""Metadata for a dataset used for AutoML Tables.
+
+
+ Attributes:
+ primary_table_spec_id:
+ Output only. The table\_spec\_id of the primary table of this
+ dataset.
+ target_column_spec_id:
+ column\_spec\_id of the primary table's column that should be
+ used as the training & prediction target. This column must be
+ non-nullable and have one of following data types (otherwise
+ model creation will error): - CATEGORY - FLOAT64 If the
+ type is CATEGORY , only up to 100 unique values may exist in
+ that column across all rows. NOTE: Updates of this field will
+ instantly affect any other users concurrently working with the
+ dataset.
+ weight_column_spec_id:
+ column\_spec\_id of the primary table's column that should be
+ used as the weight column, i.e. the higher the value the more
+ important the row will be during model training. Required
+ type: FLOAT64. Allowed values: 0 to 10000, inclusive on both
+ ends; 0 means the row is ignored for training. If not set all
+ rows are assumed to have equal weight of 1. NOTE: Updates of
+ this field will instantly affect any other users concurrently
+ working with the dataset.
+ ml_use_column_spec_id:
+ column\_spec\_id of the primary table column which specifies a
+ possible ML use of the row, i.e. the column will be used to
+ split the rows into TRAIN, VALIDATE and TEST sets. Required
+ type: STRING. This column, if set, must either have all of
+ ``TRAIN``, ``VALIDATE``, ``TEST`` among its values, or only
+ have ``TEST``, ``UNASSIGNED`` values. In the latter case the
+ rows with ``UNASSIGNED`` value will be assigned by AutoML.
+ Note that if a given ml use distribution makes it impossible
+ to create a "good" model, that call will error describing the
+ issue. If both this column\_spec\_id and primary table's
+ time\_column\_spec\_id are not set, then all rows are treated
+ as ``UNASSIGNED``. NOTE: Updates of this field will instantly
+ affect any other users concurrently working with the dataset.
+ target_column_correlations:
+ Output only. Correlations between [TablesDatasetMetadata.targ
+ et\_column\_spec\_id][google.cloud.automl.v1beta1.TablesDatase
+ tMetadata.target\_column\_spec\_id], and other columns of the
+ [TablesDatasetMetadataprimary\_table][google.cloud.automl.v1be
+ ta1.TablesDatasetMetadata.primary\_table\_spec\_id]. Only set
+ if the target column is set. Mapping from other column spec id
+ to its CorrelationStats with the target column. This field may
+ be stale, see the stats\_update\_time field for for the
+ timestamp at which these stats were last updated.
+ stats_update_time:
+ Output only. The most recent timestamp when
+ target\_column\_correlations field and all descendant
+ ColumnSpec.data\_stats and ColumnSpec.top\_correlated\_columns
+ fields were last (re-)generated. Any changes that happened to
+ the dataset afterwards are not reflected in these fields
+ values. The regeneration happens in the background on a best
+ effort basis.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.TablesDatasetMetadata)
+ ),
+)
+_sym_db.RegisterMessage(TablesDatasetMetadata)
+_sym_db.RegisterMessage(TablesDatasetMetadata.TargetColumnCorrelationsEntry)
+
+TablesModelMetadata = _reflection.GeneratedProtocolMessageType(
+ "TablesModelMetadata",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TABLESMODELMETADATA,
+ __module__="google.cloud.automl_v1beta1.proto.tables_pb2",
+ __doc__="""Model metadata specific to AutoML Tables.
+
+
+ Attributes:
+ target_column_spec:
+ Column spec of the dataset's primary table's column the model
+ is predicting. Snapshotted when model creation started. Only 3
+ fields are used: name - May be set on CreateModel, if it's not
+ then the ColumnSpec corresponding to the current
+ target\_column\_spec\_id of the dataset the model is trained
+ from is used. If neither is set, CreateModel will error.
+ display\_name - Output only. data\_type - Output only.
+ input_feature_column_specs:
+ Column specs of the dataset's primary table's columns, on
+ which the model is trained and which are used as the input for
+ predictions. The [target\_column][google.cloud.automl.v1beta1
+ .TablesModelMetadata.target\_column\_spec] as well as,
+ according to dataset's state upon model creation, [weight\_co
+ lumn][google.cloud.automl.v1beta1.TablesDatasetMetadata.weight
+ \_column\_spec\_id], and [ml\_use\_column][google.cloud.autom
+ l.v1beta1.TablesDatasetMetadata.ml\_use\_column\_spec\_id]
+ must never be included here. Only 3 fields are used: - name
+ - May be set on CreateModel, if set only the columns specified
+ are used, otherwise all primary table's columns (except the
+ ones listed above) are used for the training and prediction
+ input. - display\_name - Output only. - data\_type -
+ Output only.
+ optimization_objective:
+ Objective function the model is optimizing towards. The
+ training process creates a model that maximizes/minimizes the
+ value of the objective function over the validation set. The
+ supported optimization objectives depend on the prediction
+ type. If the field is not set, a default objective function is
+ used. CLASSIFICATION\_BINARY: "MAXIMIZE\_AU\_ROC" (default) -
+ Maximize the area under the receiver operating characteristic
+ (ROC) curve. "MINIMIZE\_LOG\_LOSS" - Minimize log loss.
+ "MAXIMIZE\_AU\_PRC" - Maximize the area under the precision-
+ recall curve. "MAXIMIZE\_PRECISION\_AT\_RECALL" - Maximize
+ precision for a specified recall value.
+ "MAXIMIZE\_RECALL\_AT\_PRECISION" - Maximize recall for a
+ specified precision value. CLASSIFICATION\_MULTI\_CLASS :
+ "MINIMIZE\_LOG\_LOSS" (default) - Minimize log loss.
+ REGRESSION: "MINIMIZE\_RMSE" (default) - Minimize root-mean-
+ squared error (RMSE). "MINIMIZE\_MAE" - Minimize mean-absolute
+ error (MAE). "MINIMIZE\_RMSLE" - Minimize root-mean-squared
+ log error (RMSLE).
+ tables_model_column_info:
+ Output only. Auxiliary information for each of the
+ input\_feature\_column\_specs with respect to this particular
+ model.
+ train_budget_milli_node_hours:
+ Required. The train budget of creating this model, expressed
+ in milli node hours i.e. 1,000 value in this field means 1
+ node hour. The training cost of the model will not exceed
+ this budget. The final cost will be attempted to be close to
+ the budget, though may end up being (even) noticeably smaller
+ - at the backend's discretion. This especially may happen when
+ further model training ceases to provide any improvements. If
+ the budget is set to a value known to be insufficient to train
+ a model for the given dataset, the training won't be attempted
+ and will error. The train budget must be between 1,000 and
+ 72,000 milli node hours, inclusive.
+ train_cost_milli_node_hours:
+ Output only. The actual training cost of the model, expressed
+ in milli node hours, i.e. 1,000 value in this field means 1
+ node hour. Guaranteed to not exceed the train budget.
+ disable_early_stopping:
+ Use the entire training budget. This disables the early
+ stopping feature. By default, the early stopping feature is
+ enabled, which means that AutoML Tables might stop training
+ before the entire training budget has been used.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.TablesModelMetadata)
+ ),
+)
+_sym_db.RegisterMessage(TablesModelMetadata)
+
+TablesAnnotation = _reflection.GeneratedProtocolMessageType(
+ "TablesAnnotation",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TABLESANNOTATION,
+ __module__="google.cloud.automl_v1beta1.proto.tables_pb2",
+ __doc__="""Contains annotation details specific to Tables.
+
+
+ Attributes:
+ score:
+ Output only. A confidence estimate between 0.0 and 1.0,
+ inclusive. A higher value means greater confidence in the
+ returned value. For [target\_column\_spec][google.cloud.autom
+ l.v1beta1.TablesModelMetadata.target\_column\_spec] of FLOAT64
+ data type the score is not populated.
+ prediction_interval:
+ Output only. Only populated when [target\_column\_spec][googl
+ e.cloud.automl.v1beta1.TablesModelMetadata.target\_column\_spe
+ c] has FLOAT64 data type. An interval in which the exactly
+ correct target value has 95% chance to be in.
+ value:
+ The predicted value of the row's [target\_column][google.clou
+ d.automl.v1beta1.TablesModelMetadata.target\_column\_spec].
+ The value depends on the column's DataType: - CATEGORY - the
+ predicted (with the above confidence ``score``) CATEGORY
+ value. - FLOAT64 - the predicted (with above
+ ``prediction_interval``) FLOAT64 value.
+ tables_model_column_info:
+ Output only. Auxiliary information for each of the model's [i
+ nput\_feature\_column\_specs][google.cloud.automl.v1beta1.Tabl
+ esModelMetadata.input\_feature\_column\_specs] with respect to
+ this particular prediction. If no other fields than [column\_
+ spec\_name][google.cloud.automl.v1beta1.TablesModelColumnInfo.
+ column\_spec\_name] and [column\_display\_name][google.cloud.
+ automl.v1beta1.TablesModelColumnInfo.column\_display\_name]
+ would be populated, then this whole field is not.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.TablesAnnotation)
+ ),
+)
+_sym_db.RegisterMessage(TablesAnnotation)
+
+TablesModelColumnInfo = _reflection.GeneratedProtocolMessageType(
+ "TablesModelColumnInfo",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TABLESMODELCOLUMNINFO,
+ __module__="google.cloud.automl_v1beta1.proto.tables_pb2",
+ __doc__="""An information specific to given column and Tables Model, in context of
+ the Model and the predictions created by it.
+
+
+ Attributes:
+ column_spec_name:
+ Output only. The name of the ColumnSpec describing the column.
+ Not populated when this proto is outputted to BigQuery.
+ column_display_name:
+ Output only. The display name of the column (same as the
+ display\_name of its ColumnSpec).
+ feature_importance:
+ Output only. When given as part of a Model (always populated):
+ Measurement of how much model predictions correctness on the
+ TEST data depend on values in this column. A value between 0
+ and 1, higher means higher influence. These values are
+ normalized - for all input feature columns of a given model
+ they add to 1. When given back by Predict (populated iff
+ [feature\_importance
+ param][google.cloud.automl.v1beta1.PredictRequest.params] is
+ set) or Batch Predict (populated iff [feature\_importance][goo
+ gle.cloud.automl.v1beta1.PredictRequest.params] param is set):
+ Measurement of how impactful for the prediction returned for
+ the given row the value in this column was. A value between 0
+ and 1, higher means larger impact. These values are normalized
+ - for all input feature columns of a single predicted row they
+ add to 1.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.TablesModelColumnInfo)
+ ),
+)
+_sym_db.RegisterMessage(TablesModelColumnInfo)
+
+
+DESCRIPTOR._options = None
+_TABLESDATASETMETADATA_TARGETCOLUMNCORRELATIONSENTRY._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/automl/google/cloud/automl_v1beta1/proto/tables_pb2_grpc.py b/automl/google/cloud/automl_v1beta1/proto/tables_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/tables_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/automl/google/cloud/automl_v1beta1/proto/temporal.proto b/automl/google/cloud/automl_v1beta1/proto/temporal.proto
new file mode 100644
index 000000000000..84874d99fa06
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/temporal.proto
@@ -0,0 +1,38 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.automl.v1beta1;
+
+import "google/protobuf/duration.proto";
+import "google/api/annotations.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl";
+option java_multiple_files = true;
+option java_package = "com.google.cloud.automl.v1beta1";
+option php_namespace = "Google\\Cloud\\AutoMl\\V1beta1";
+option ruby_package = "Google::Cloud::AutoML::V1beta1";
+
+// A time period inside of an example that has a time dimension (e.g. video).
+message TimeSegment {
+ // Start of the time segment (inclusive), represented as the duration since
+ // the example start.
+ google.protobuf.Duration start_time_offset = 1;
+
+ // End of the time segment (exclusive), represented as the duration since the
+ // example start.
+ google.protobuf.Duration end_time_offset = 2;
+}
diff --git a/automl/google/cloud/automl_v1beta1/proto/temporal_pb2.py b/automl/google/cloud/automl_v1beta1/proto/temporal_pb2.py
new file mode 100644
index 000000000000..99fa8d6635ae
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/temporal_pb2.py
@@ -0,0 +1,129 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/automl_v1beta1/proto/temporal.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.protobuf import duration_pb2 as google_dot_protobuf_dot_duration__pb2
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/automl_v1beta1/proto/temporal.proto",
+ package="google.cloud.automl.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\037com.google.cloud.automl.v1beta1P\001ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\312\002\033Google\\Cloud\\AutoMl\\V1beta1\352\002\036Google::Cloud::AutoML::V1beta1"
+ ),
+ serialized_pb=_b(
+ '\n0google/cloud/automl_v1beta1/proto/temporal.proto\x12\x1bgoogle.cloud.automl.v1beta1\x1a\x1egoogle/protobuf/duration.proto\x1a\x1cgoogle/api/annotations.proto"w\n\x0bTimeSegment\x12\x34\n\x11start_time_offset\x18\x01 \x01(\x0b\x32\x19.google.protobuf.Duration\x12\x32\n\x0f\x65nd_time_offset\x18\x02 \x01(\x0b\x32\x19.google.protobuf.DurationB\xa5\x01\n\x1f\x63om.google.cloud.automl.v1beta1P\x01ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\xca\x02\x1bGoogle\\Cloud\\AutoMl\\V1beta1\xea\x02\x1eGoogle::Cloud::AutoML::V1beta1b\x06proto3'
+ ),
+ dependencies=[
+ google_dot_protobuf_dot_duration__pb2.DESCRIPTOR,
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ ],
+)
+
+
+_TIMESEGMENT = _descriptor.Descriptor(
+ name="TimeSegment",
+ full_name="google.cloud.automl.v1beta1.TimeSegment",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="start_time_offset",
+ full_name="google.cloud.automl.v1beta1.TimeSegment.start_time_offset",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="end_time_offset",
+ full_name="google.cloud.automl.v1beta1.TimeSegment.end_time_offset",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=143,
+ serialized_end=262,
+)
+
+_TIMESEGMENT.fields_by_name[
+ "start_time_offset"
+].message_type = google_dot_protobuf_dot_duration__pb2._DURATION
+_TIMESEGMENT.fields_by_name[
+ "end_time_offset"
+].message_type = google_dot_protobuf_dot_duration__pb2._DURATION
+DESCRIPTOR.message_types_by_name["TimeSegment"] = _TIMESEGMENT
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+TimeSegment = _reflection.GeneratedProtocolMessageType(
+ "TimeSegment",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TIMESEGMENT,
+ __module__="google.cloud.automl_v1beta1.proto.temporal_pb2",
+ __doc__="""A time period inside of an example that has a time dimension (e.g.
+ video).
+
+
+ Attributes:
+ start_time_offset:
+ Start of the time segment (inclusive), represented as the
+ duration since the example start.
+ end_time_offset:
+ End of the time segment (exclusive), represented as the
+ duration since the example start.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.TimeSegment)
+ ),
+)
+_sym_db.RegisterMessage(TimeSegment)
+
+
+DESCRIPTOR._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/automl/google/cloud/automl_v1beta1/proto/temporal_pb2_grpc.py b/automl/google/cloud/automl_v1beta1/proto/temporal_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/temporal_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/automl/google/cloud/automl_v1beta1/proto/text.proto b/automl/google/cloud/automl_v1beta1/proto/text.proto
new file mode 100644
index 000000000000..6a01b7cff60f
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/text.proto
@@ -0,0 +1,65 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.automl.v1beta1;
+
+import "google/cloud/automl/v1beta1/classification.proto";
+import "google/api/annotations.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl";
+option java_multiple_files = true;
+option java_outer_classname = "TextProto";
+option java_package = "com.google.cloud.automl.v1beta1";
+option php_namespace = "Google\\Cloud\\AutoMl\\V1beta1";
+option ruby_package = "Google::Cloud::AutoML::V1beta1";
+
+// Dataset metadata for classification.
+message TextClassificationDatasetMetadata {
+ // Required. Type of the classification problem.
+ ClassificationType classification_type = 1;
+}
+
+// Model metadata that is specific to text classification.
+message TextClassificationModelMetadata {
+
+}
+
+// Dataset metadata that is specific to text extraction
+message TextExtractionDatasetMetadata {
+
+}
+
+// Model metadata that is specific to text extraction.
+message TextExtractionModelMetadata {
+
+}
+
+// Dataset metadata for text sentiment.
+message TextSentimentDatasetMetadata {
+ // Required. A sentiment is expressed as an integer ordinal, where higher value
+ // means a more positive sentiment. The range of sentiments that will be used
+ // is between 0 and sentiment_max (inclusive on both ends), and all the values
+ // in the range must be represented in the dataset before a model can be
+ // created.
+ // sentiment_max value must be between 1 and 10 (inclusive).
+ int32 sentiment_max = 1;
+}
+
+// Model metadata that is specific to text sentiment.
+message TextSentimentModelMetadata {
+
+}
diff --git a/automl/google/cloud/automl_v1beta1/proto/text_extraction.proto b/automl/google/cloud/automl_v1beta1/proto/text_extraction.proto
new file mode 100644
index 000000000000..07f0dda8169f
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/text_extraction.proto
@@ -0,0 +1,69 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.automl.v1beta1;
+
+import "google/cloud/automl/v1beta1/text_segment.proto";
+import "google/api/annotations.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl";
+option java_multiple_files = true;
+option java_package = "com.google.cloud.automl.v1beta1";
+option php_namespace = "Google\\Cloud\\AutoMl\\V1beta1";
+option ruby_package = "Google::Cloud::AutoML::V1beta1";
+
+// Annotation for identifying spans of text.
+message TextExtractionAnnotation {
+ // Required. Text extraction annotations can either be a text segment or a
+ // text relation.
+ oneof annotation {
+ // An entity annotation will set this, which is the part of the original
+ // text to which the annotation pertains.
+ TextSegment text_segment = 3;
+ }
+
+ // Output only. A confidence estimate between 0.0 and 1.0. A higher value
+ // means greater confidence in correctness of the annotation.
+ float score = 1;
+}
+
+// Model evaluation metrics for text extraction problems.
+message TextExtractionEvaluationMetrics {
+ // Metrics for a single confidence threshold.
+ message ConfidenceMetricsEntry {
+ // Output only. The confidence threshold value used to compute the metrics.
+ // Only annotations with score of at least this threshold are considered to
+ // be ones the model would return.
+ float confidence_threshold = 1;
+
+ // Output only. Recall under the given confidence threshold.
+ float recall = 3;
+
+ // Output only. Precision under the given confidence threshold.
+ float precision = 4;
+
+ // Output only. The harmonic mean of recall and precision.
+ float f1_score = 5;
+ }
+
+ // Output only. The Area under precision recall curve metric.
+ float au_prc = 1;
+
+ // Output only. Metrics that have confidence thresholds.
+ // Precision-recall curve can be derived from it.
+ repeated ConfidenceMetricsEntry confidence_metrics_entries = 2;
+}
diff --git a/automl/google/cloud/automl_v1beta1/proto/text_extraction_pb2.py b/automl/google/cloud/automl_v1beta1/proto/text_extraction_pb2.py
new file mode 100644
index 000000000000..04dc759c3f02
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/text_extraction_pb2.py
@@ -0,0 +1,351 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/automl_v1beta1/proto/text_extraction.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.cloud.automl_v1beta1.proto import (
+ text_segment_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_text__segment__pb2,
+)
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/automl_v1beta1/proto/text_extraction.proto",
+ package="google.cloud.automl.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\037com.google.cloud.automl.v1beta1P\001ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\312\002\033Google\\Cloud\\AutoMl\\V1beta1\352\002\036Google::Cloud::AutoML::V1beta1"
+ ),
+ serialized_pb=_b(
+ '\n7google/cloud/automl_v1beta1/proto/text_extraction.proto\x12\x1bgoogle.cloud.automl.v1beta1\x1a\x34google/cloud/automl_v1beta1/proto/text_segment.proto\x1a\x1cgoogle/api/annotations.proto"y\n\x18TextExtractionAnnotation\x12@\n\x0ctext_segment\x18\x03 \x01(\x0b\x32(.google.cloud.automl.v1beta1.TextSegmentH\x00\x12\r\n\x05score\x18\x01 \x01(\x02\x42\x0c\n\nannotation"\x97\x02\n\x1fTextExtractionEvaluationMetrics\x12\x0e\n\x06\x61u_prc\x18\x01 \x01(\x02\x12w\n\x1a\x63onfidence_metrics_entries\x18\x02 \x03(\x0b\x32S.google.cloud.automl.v1beta1.TextExtractionEvaluationMetrics.ConfidenceMetricsEntry\x1ak\n\x16\x43onfidenceMetricsEntry\x12\x1c\n\x14\x63onfidence_threshold\x18\x01 \x01(\x02\x12\x0e\n\x06recall\x18\x03 \x01(\x02\x12\x11\n\tprecision\x18\x04 \x01(\x02\x12\x10\n\x08\x66\x31_score\x18\x05 \x01(\x02\x42\xa5\x01\n\x1f\x63om.google.cloud.automl.v1beta1P\x01ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\xca\x02\x1bGoogle\\Cloud\\AutoMl\\V1beta1\xea\x02\x1eGoogle::Cloud::AutoML::V1beta1b\x06proto3'
+ ),
+ dependencies=[
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_text__segment__pb2.DESCRIPTOR,
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ ],
+)
+
+
+_TEXTEXTRACTIONANNOTATION = _descriptor.Descriptor(
+ name="TextExtractionAnnotation",
+ full_name="google.cloud.automl.v1beta1.TextExtractionAnnotation",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="text_segment",
+ full_name="google.cloud.automl.v1beta1.TextExtractionAnnotation.text_segment",
+ index=0,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="score",
+ full_name="google.cloud.automl.v1beta1.TextExtractionAnnotation.score",
+ index=1,
+ number=1,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="annotation",
+ full_name="google.cloud.automl.v1beta1.TextExtractionAnnotation.annotation",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=172,
+ serialized_end=293,
+)
+
+
+_TEXTEXTRACTIONEVALUATIONMETRICS_CONFIDENCEMETRICSENTRY = _descriptor.Descriptor(
+ name="ConfidenceMetricsEntry",
+ full_name="google.cloud.automl.v1beta1.TextExtractionEvaluationMetrics.ConfidenceMetricsEntry",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="confidence_threshold",
+ full_name="google.cloud.automl.v1beta1.TextExtractionEvaluationMetrics.ConfidenceMetricsEntry.confidence_threshold",
+ index=0,
+ number=1,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="recall",
+ full_name="google.cloud.automl.v1beta1.TextExtractionEvaluationMetrics.ConfidenceMetricsEntry.recall",
+ index=1,
+ number=3,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="precision",
+ full_name="google.cloud.automl.v1beta1.TextExtractionEvaluationMetrics.ConfidenceMetricsEntry.precision",
+ index=2,
+ number=4,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="f1_score",
+ full_name="google.cloud.automl.v1beta1.TextExtractionEvaluationMetrics.ConfidenceMetricsEntry.f1_score",
+ index=3,
+ number=5,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=468,
+ serialized_end=575,
+)
+
+_TEXTEXTRACTIONEVALUATIONMETRICS = _descriptor.Descriptor(
+ name="TextExtractionEvaluationMetrics",
+ full_name="google.cloud.automl.v1beta1.TextExtractionEvaluationMetrics",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="au_prc",
+ full_name="google.cloud.automl.v1beta1.TextExtractionEvaluationMetrics.au_prc",
+ index=0,
+ number=1,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="confidence_metrics_entries",
+ full_name="google.cloud.automl.v1beta1.TextExtractionEvaluationMetrics.confidence_metrics_entries",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[_TEXTEXTRACTIONEVALUATIONMETRICS_CONFIDENCEMETRICSENTRY],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=296,
+ serialized_end=575,
+)
+
+_TEXTEXTRACTIONANNOTATION.fields_by_name[
+ "text_segment"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_text__segment__pb2._TEXTSEGMENT
+)
+_TEXTEXTRACTIONANNOTATION.oneofs_by_name["annotation"].fields.append(
+ _TEXTEXTRACTIONANNOTATION.fields_by_name["text_segment"]
+)
+_TEXTEXTRACTIONANNOTATION.fields_by_name[
+ "text_segment"
+].containing_oneof = _TEXTEXTRACTIONANNOTATION.oneofs_by_name["annotation"]
+_TEXTEXTRACTIONEVALUATIONMETRICS_CONFIDENCEMETRICSENTRY.containing_type = (
+ _TEXTEXTRACTIONEVALUATIONMETRICS
+)
+_TEXTEXTRACTIONEVALUATIONMETRICS.fields_by_name[
+ "confidence_metrics_entries"
+].message_type = _TEXTEXTRACTIONEVALUATIONMETRICS_CONFIDENCEMETRICSENTRY
+DESCRIPTOR.message_types_by_name["TextExtractionAnnotation"] = _TEXTEXTRACTIONANNOTATION
+DESCRIPTOR.message_types_by_name[
+ "TextExtractionEvaluationMetrics"
+] = _TEXTEXTRACTIONEVALUATIONMETRICS
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+TextExtractionAnnotation = _reflection.GeneratedProtocolMessageType(
+ "TextExtractionAnnotation",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TEXTEXTRACTIONANNOTATION,
+ __module__="google.cloud.automl_v1beta1.proto.text_extraction_pb2",
+ __doc__="""Annotation for identifying spans of text.
+
+
+ Attributes:
+ annotation:
+ Required. Text extraction annotations can either be a text
+ segment or a text relation.
+ text_segment:
+ An entity annotation will set this, which is the part of the
+ original text to which the annotation pertains.
+ score:
+ Output only. A confidence estimate between 0.0 and 1.0. A
+ higher value means greater confidence in correctness of the
+ annotation.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.TextExtractionAnnotation)
+ ),
+)
+_sym_db.RegisterMessage(TextExtractionAnnotation)
+
+TextExtractionEvaluationMetrics = _reflection.GeneratedProtocolMessageType(
+ "TextExtractionEvaluationMetrics",
+ (_message.Message,),
+ dict(
+ ConfidenceMetricsEntry=_reflection.GeneratedProtocolMessageType(
+ "ConfidenceMetricsEntry",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TEXTEXTRACTIONEVALUATIONMETRICS_CONFIDENCEMETRICSENTRY,
+ __module__="google.cloud.automl_v1beta1.proto.text_extraction_pb2",
+ __doc__="""Metrics for a single confidence threshold.
+
+
+ Attributes:
+ confidence_threshold:
+ Output only. The confidence threshold value used to compute
+ the metrics. Only annotations with score of at least this
+ threshold are considered to be ones the model would return.
+ recall:
+ Output only. Recall under the given confidence threshold.
+ precision:
+ Output only. Precision under the given confidence threshold.
+ f1_score:
+ Output only. The harmonic mean of recall and precision.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.TextExtractionEvaluationMetrics.ConfidenceMetricsEntry)
+ ),
+ ),
+ DESCRIPTOR=_TEXTEXTRACTIONEVALUATIONMETRICS,
+ __module__="google.cloud.automl_v1beta1.proto.text_extraction_pb2",
+ __doc__="""Model evaluation metrics for text extraction problems.
+
+
+ Attributes:
+ au_prc:
+ Output only. The Area under precision recall curve metric.
+ confidence_metrics_entries:
+ Output only. Metrics that have confidence thresholds.
+ Precision-recall curve can be derived from it.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.TextExtractionEvaluationMetrics)
+ ),
+)
+_sym_db.RegisterMessage(TextExtractionEvaluationMetrics)
+_sym_db.RegisterMessage(TextExtractionEvaluationMetrics.ConfidenceMetricsEntry)
+
+
+DESCRIPTOR._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/automl/google/cloud/automl_v1beta1/proto/text_extraction_pb2_grpc.py b/automl/google/cloud/automl_v1beta1/proto/text_extraction_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/text_extraction_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/automl/google/cloud/automl_v1beta1/proto/text_pb2.py b/automl/google/cloud/automl_v1beta1/proto/text_pb2.py
new file mode 100644
index 000000000000..df8a45c5fa72
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/text_pb2.py
@@ -0,0 +1,319 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/automl_v1beta1/proto/text.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.cloud.automl_v1beta1.proto import (
+ classification_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_classification__pb2,
+)
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/automl_v1beta1/proto/text.proto",
+ package="google.cloud.automl.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\037com.google.cloud.automl.v1beta1B\tTextProtoP\001ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\312\002\033Google\\Cloud\\AutoMl\\V1beta1\352\002\036Google::Cloud::AutoML::V1beta1"
+ ),
+ serialized_pb=_b(
+ '\n,google/cloud/automl_v1beta1/proto/text.proto\x12\x1bgoogle.cloud.automl.v1beta1\x1a\x36google/cloud/automl_v1beta1/proto/classification.proto\x1a\x1cgoogle/api/annotations.proto"q\n!TextClassificationDatasetMetadata\x12L\n\x13\x63lassification_type\x18\x01 \x01(\x0e\x32/.google.cloud.automl.v1beta1.ClassificationType"!\n\x1fTextClassificationModelMetadata"\x1f\n\x1dTextExtractionDatasetMetadata"\x1d\n\x1bTextExtractionModelMetadata"5\n\x1cTextSentimentDatasetMetadata\x12\x15\n\rsentiment_max\x18\x01 \x01(\x05"\x1c\n\x1aTextSentimentModelMetadataB\xb0\x01\n\x1f\x63om.google.cloud.automl.v1beta1B\tTextProtoP\x01ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\xca\x02\x1bGoogle\\Cloud\\AutoMl\\V1beta1\xea\x02\x1eGoogle::Cloud::AutoML::V1beta1b\x06proto3'
+ ),
+ dependencies=[
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_classification__pb2.DESCRIPTOR,
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ ],
+)
+
+
+_TEXTCLASSIFICATIONDATASETMETADATA = _descriptor.Descriptor(
+ name="TextClassificationDatasetMetadata",
+ full_name="google.cloud.automl.v1beta1.TextClassificationDatasetMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="classification_type",
+ full_name="google.cloud.automl.v1beta1.TextClassificationDatasetMetadata.classification_type",
+ index=0,
+ number=1,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=163,
+ serialized_end=276,
+)
+
+
+_TEXTCLASSIFICATIONMODELMETADATA = _descriptor.Descriptor(
+ name="TextClassificationModelMetadata",
+ full_name="google.cloud.automl.v1beta1.TextClassificationModelMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=278,
+ serialized_end=311,
+)
+
+
+_TEXTEXTRACTIONDATASETMETADATA = _descriptor.Descriptor(
+ name="TextExtractionDatasetMetadata",
+ full_name="google.cloud.automl.v1beta1.TextExtractionDatasetMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=313,
+ serialized_end=344,
+)
+
+
+_TEXTEXTRACTIONMODELMETADATA = _descriptor.Descriptor(
+ name="TextExtractionModelMetadata",
+ full_name="google.cloud.automl.v1beta1.TextExtractionModelMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=346,
+ serialized_end=375,
+)
+
+
+_TEXTSENTIMENTDATASETMETADATA = _descriptor.Descriptor(
+ name="TextSentimentDatasetMetadata",
+ full_name="google.cloud.automl.v1beta1.TextSentimentDatasetMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="sentiment_max",
+ full_name="google.cloud.automl.v1beta1.TextSentimentDatasetMetadata.sentiment_max",
+ index=0,
+ number=1,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=377,
+ serialized_end=430,
+)
+
+
+_TEXTSENTIMENTMODELMETADATA = _descriptor.Descriptor(
+ name="TextSentimentModelMetadata",
+ full_name="google.cloud.automl.v1beta1.TextSentimentModelMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=432,
+ serialized_end=460,
+)
+
+_TEXTCLASSIFICATIONDATASETMETADATA.fields_by_name[
+ "classification_type"
+].enum_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_classification__pb2._CLASSIFICATIONTYPE
+)
+DESCRIPTOR.message_types_by_name[
+ "TextClassificationDatasetMetadata"
+] = _TEXTCLASSIFICATIONDATASETMETADATA
+DESCRIPTOR.message_types_by_name[
+ "TextClassificationModelMetadata"
+] = _TEXTCLASSIFICATIONMODELMETADATA
+DESCRIPTOR.message_types_by_name[
+ "TextExtractionDatasetMetadata"
+] = _TEXTEXTRACTIONDATASETMETADATA
+DESCRIPTOR.message_types_by_name[
+ "TextExtractionModelMetadata"
+] = _TEXTEXTRACTIONMODELMETADATA
+DESCRIPTOR.message_types_by_name[
+ "TextSentimentDatasetMetadata"
+] = _TEXTSENTIMENTDATASETMETADATA
+DESCRIPTOR.message_types_by_name[
+ "TextSentimentModelMetadata"
+] = _TEXTSENTIMENTMODELMETADATA
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+TextClassificationDatasetMetadata = _reflection.GeneratedProtocolMessageType(
+ "TextClassificationDatasetMetadata",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TEXTCLASSIFICATIONDATASETMETADATA,
+ __module__="google.cloud.automl_v1beta1.proto.text_pb2",
+ __doc__="""Dataset metadata for classification.
+
+
+ Attributes:
+ classification_type:
+ Required. Type of the classification problem.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.TextClassificationDatasetMetadata)
+ ),
+)
+_sym_db.RegisterMessage(TextClassificationDatasetMetadata)
+
+TextClassificationModelMetadata = _reflection.GeneratedProtocolMessageType(
+ "TextClassificationModelMetadata",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TEXTCLASSIFICATIONMODELMETADATA,
+ __module__="google.cloud.automl_v1beta1.proto.text_pb2",
+ __doc__="""Model metadata that is specific to text classification.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.TextClassificationModelMetadata)
+ ),
+)
+_sym_db.RegisterMessage(TextClassificationModelMetadata)
+
+TextExtractionDatasetMetadata = _reflection.GeneratedProtocolMessageType(
+ "TextExtractionDatasetMetadata",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TEXTEXTRACTIONDATASETMETADATA,
+ __module__="google.cloud.automl_v1beta1.proto.text_pb2",
+ __doc__="""Dataset metadata that is specific to text extraction
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.TextExtractionDatasetMetadata)
+ ),
+)
+_sym_db.RegisterMessage(TextExtractionDatasetMetadata)
+
+TextExtractionModelMetadata = _reflection.GeneratedProtocolMessageType(
+ "TextExtractionModelMetadata",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TEXTEXTRACTIONMODELMETADATA,
+ __module__="google.cloud.automl_v1beta1.proto.text_pb2",
+ __doc__="""Model metadata that is specific to text extraction.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.TextExtractionModelMetadata)
+ ),
+)
+_sym_db.RegisterMessage(TextExtractionModelMetadata)
+
+TextSentimentDatasetMetadata = _reflection.GeneratedProtocolMessageType(
+ "TextSentimentDatasetMetadata",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TEXTSENTIMENTDATASETMETADATA,
+ __module__="google.cloud.automl_v1beta1.proto.text_pb2",
+ __doc__="""Dataset metadata for text sentiment.
+
+
+ Attributes:
+ sentiment_max:
+ Required. A sentiment is expressed as an integer ordinal,
+ where higher value means a more positive sentiment. The range
+ of sentiments that will be used is between 0 and
+ sentiment\_max (inclusive on both ends), and all the values in
+ the range must be represented in the dataset before a model
+ can be created. sentiment\_max value must be between 1 and 10
+ (inclusive).
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.TextSentimentDatasetMetadata)
+ ),
+)
+_sym_db.RegisterMessage(TextSentimentDatasetMetadata)
+
+TextSentimentModelMetadata = _reflection.GeneratedProtocolMessageType(
+ "TextSentimentModelMetadata",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TEXTSENTIMENTMODELMETADATA,
+ __module__="google.cloud.automl_v1beta1.proto.text_pb2",
+ __doc__="""Model metadata that is specific to text sentiment.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.TextSentimentModelMetadata)
+ ),
+)
+_sym_db.RegisterMessage(TextSentimentModelMetadata)
+
+
+DESCRIPTOR._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/automl/google/cloud/automl_v1beta1/proto/text_pb2_grpc.py b/automl/google/cloud/automl_v1beta1/proto/text_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/text_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/automl/google/cloud/automl_v1beta1/proto/text_segment.proto b/automl/google/cloud/automl_v1beta1/proto/text_segment.proto
new file mode 100644
index 000000000000..41b8be1cca00
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/text_segment.proto
@@ -0,0 +1,42 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.automl.v1beta1;
+
+import "google/api/annotations.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl";
+option java_multiple_files = true;
+option java_outer_classname = "TextSegmentProto";
+option java_package = "com.google.cloud.automl.v1beta1";
+option php_namespace = "Google\\Cloud\\AutoMl\\V1beta1";
+option ruby_package = "Google::Cloud::AutoML::V1beta1";
+
+// A contiguous part of a text (string), assuming it has an UTF-8 NFC encoding.
+message TextSegment {
+ // Output only. The content of the TextSegment.
+ string content = 3;
+
+ // Required. Zero-based character index of the first character of the text
+ // segment (counting characters from the beginning of the text).
+ int64 start_offset = 1;
+
+ // Required. Zero-based character index of the first character past the end of
+ // the text segment (counting character from the beginning of the text).
+ // The character at the end_offset is NOT included in the text segment.
+ int64 end_offset = 2;
+}
diff --git a/automl/google/cloud/automl_v1beta1/proto/text_segment_pb2.py b/automl/google/cloud/automl_v1beta1/proto/text_segment_pb2.py
new file mode 100644
index 000000000000..79774587f8c1
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/text_segment_pb2.py
@@ -0,0 +1,142 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/automl_v1beta1/proto/text_segment.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/automl_v1beta1/proto/text_segment.proto",
+ package="google.cloud.automl.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\037com.google.cloud.automl.v1beta1B\020TextSegmentProtoP\001ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\312\002\033Google\\Cloud\\AutoMl\\V1beta1\352\002\036Google::Cloud::AutoML::V1beta1"
+ ),
+ serialized_pb=_b(
+ '\n4google/cloud/automl_v1beta1/proto/text_segment.proto\x12\x1bgoogle.cloud.automl.v1beta1\x1a\x1cgoogle/api/annotations.proto"H\n\x0bTextSegment\x12\x0f\n\x07\x63ontent\x18\x03 \x01(\t\x12\x14\n\x0cstart_offset\x18\x01 \x01(\x03\x12\x12\n\nend_offset\x18\x02 \x01(\x03\x42\xb7\x01\n\x1f\x63om.google.cloud.automl.v1beta1B\x10TextSegmentProtoP\x01ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\xca\x02\x1bGoogle\\Cloud\\AutoMl\\V1beta1\xea\x02\x1eGoogle::Cloud::AutoML::V1beta1b\x06proto3'
+ ),
+ dependencies=[google_dot_api_dot_annotations__pb2.DESCRIPTOR],
+)
+
+
+_TEXTSEGMENT = _descriptor.Descriptor(
+ name="TextSegment",
+ full_name="google.cloud.automl.v1beta1.TextSegment",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="content",
+ full_name="google.cloud.automl.v1beta1.TextSegment.content",
+ index=0,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="start_offset",
+ full_name="google.cloud.automl.v1beta1.TextSegment.start_offset",
+ index=1,
+ number=1,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="end_offset",
+ full_name="google.cloud.automl.v1beta1.TextSegment.end_offset",
+ index=2,
+ number=2,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=115,
+ serialized_end=187,
+)
+
+DESCRIPTOR.message_types_by_name["TextSegment"] = _TEXTSEGMENT
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+TextSegment = _reflection.GeneratedProtocolMessageType(
+ "TextSegment",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TEXTSEGMENT,
+ __module__="google.cloud.automl_v1beta1.proto.text_segment_pb2",
+ __doc__="""A contiguous part of a text (string), assuming it has an UTF-8 NFC
+ encoding.
+
+
+ Attributes:
+ content:
+ Output only. The content of the TextSegment.
+ start_offset:
+ Required. Zero-based character index of the first character of
+ the text segment (counting characters from the beginning of
+ the text).
+ end_offset:
+ Required. Zero-based character index of the first character
+ past the end of the text segment (counting character from the
+ beginning of the text). The character at the end\_offset is
+ NOT included in the text segment.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.TextSegment)
+ ),
+)
+_sym_db.RegisterMessage(TextSegment)
+
+
+DESCRIPTOR._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/automl/google/cloud/automl_v1beta1/proto/text_segment_pb2_grpc.py b/automl/google/cloud/automl_v1beta1/proto/text_segment_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/text_segment_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/automl/google/cloud/automl_v1beta1/proto/text_sentiment.proto b/automl/google/cloud/automl_v1beta1/proto/text_sentiment.proto
new file mode 100644
index 000000000000..978acb0f97c6
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/text_sentiment.proto
@@ -0,0 +1,81 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.automl.v1beta1;
+
+import "google/cloud/automl/v1beta1/classification.proto";
+import "google/api/annotations.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl";
+option java_outer_classname = "TextSentimentProto";
+option java_package = "com.google.cloud.automl.v1beta1";
+option php_namespace = "Google\\Cloud\\AutoMl\\V1beta1";
+option ruby_package = "Google::Cloud::AutoML::V1beta1";
+
+// Contains annotation details specific to text sentiment.
+message TextSentimentAnnotation {
+ // Output only. The sentiment with the semantic, as given to the
+ // [AutoMl.ImportData][google.cloud.automl.v1beta1.AutoMl.ImportData] when populating the dataset from which the model used
+ // for the prediction had been trained.
+ // The sentiment values are between 0 and
+ // Dataset.text_sentiment_dataset_metadata.sentiment_max (inclusive),
+ // with higher value meaning more positive sentiment. They are completely
+ // relative, i.e. 0 means least positive sentiment and sentiment_max means
+ // the most positive from the sentiments present in the train data. Therefore
+ // e.g. if train data had only negative sentiment, then sentiment_max, would
+ // be still negative (although least negative).
+ // The sentiment shouldn't be confused with "score" or "magnitude"
+ // from the previous Natural Language Sentiment Analysis API.
+ int32 sentiment = 1;
+}
+
+// Model evaluation metrics for text sentiment problems.
+message TextSentimentEvaluationMetrics {
+ // Output only. Precision.
+ float precision = 1;
+
+ // Output only. Recall.
+ float recall = 2;
+
+ // Output only. The harmonic mean of recall and precision.
+ float f1_score = 3;
+
+ // Output only. Mean absolute error. Only set for the overall model
+ // evaluation, not for evaluation of a single annotation spec.
+ float mean_absolute_error = 4;
+
+ // Output only. Mean squared error. Only set for the overall model
+ // evaluation, not for evaluation of a single annotation spec.
+ float mean_squared_error = 5;
+
+ // Output only. Linear weighted kappa. Only set for the overall model
+ // evaluation, not for evaluation of a single annotation spec.
+ float linear_kappa = 6;
+
+ // Output only. Quadratic weighted kappa. Only set for the overall model
+ // evaluation, not for evaluation of a single annotation spec.
+ float quadratic_kappa = 7;
+
+ // Output only. Confusion matrix of the evaluation.
+ // Only set for the overall model evaluation, not for evaluation of a single
+ // annotation spec.
+ ClassificationEvaluationMetrics.ConfusionMatrix confusion_matrix = 8;
+
+ // Output only. The annotation spec ids used for this evaluation.
+ // Deprecated .
+ repeated string annotation_spec_id = 9 [deprecated = true];
+}
diff --git a/automl/google/cloud/automl_v1beta1/proto/text_sentiment_pb2.py b/automl/google/cloud/automl_v1beta1/proto/text_sentiment_pb2.py
new file mode 100644
index 000000000000..c55c9979bb83
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/text_sentiment_pb2.py
@@ -0,0 +1,353 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/automl_v1beta1/proto/text_sentiment.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.cloud.automl_v1beta1.proto import (
+ classification_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_classification__pb2,
+)
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/automl_v1beta1/proto/text_sentiment.proto",
+ package="google.cloud.automl.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\037com.google.cloud.automl.v1beta1B\022TextSentimentProtoZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\312\002\033Google\\Cloud\\AutoMl\\V1beta1\352\002\036Google::Cloud::AutoML::V1beta1"
+ ),
+ serialized_pb=_b(
+ '\n6google/cloud/automl_v1beta1/proto/text_sentiment.proto\x12\x1bgoogle.cloud.automl.v1beta1\x1a\x36google/cloud/automl_v1beta1/proto/classification.proto\x1a\x1cgoogle/api/annotations.proto",\n\x17TextSentimentAnnotation\x12\x11\n\tsentiment\x18\x01 \x01(\x05"\xc5\x02\n\x1eTextSentimentEvaluationMetrics\x12\x11\n\tprecision\x18\x01 \x01(\x02\x12\x0e\n\x06recall\x18\x02 \x01(\x02\x12\x10\n\x08\x66\x31_score\x18\x03 \x01(\x02\x12\x1b\n\x13mean_absolute_error\x18\x04 \x01(\x02\x12\x1a\n\x12mean_squared_error\x18\x05 \x01(\x02\x12\x14\n\x0clinear_kappa\x18\x06 \x01(\x02\x12\x17\n\x0fquadratic_kappa\x18\x07 \x01(\x02\x12\x66\n\x10\x63onfusion_matrix\x18\x08 \x01(\x0b\x32L.google.cloud.automl.v1beta1.ClassificationEvaluationMetrics.ConfusionMatrix\x12\x1e\n\x12\x61nnotation_spec_id\x18\t \x03(\tB\x02\x18\x01\x42\xb7\x01\n\x1f\x63om.google.cloud.automl.v1beta1B\x12TextSentimentProtoZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\xca\x02\x1bGoogle\\Cloud\\AutoMl\\V1beta1\xea\x02\x1eGoogle::Cloud::AutoML::V1beta1b\x06proto3'
+ ),
+ dependencies=[
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_classification__pb2.DESCRIPTOR,
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ ],
+)
+
+
+_TEXTSENTIMENTANNOTATION = _descriptor.Descriptor(
+ name="TextSentimentAnnotation",
+ full_name="google.cloud.automl.v1beta1.TextSentimentAnnotation",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="sentiment",
+ full_name="google.cloud.automl.v1beta1.TextSentimentAnnotation.sentiment",
+ index=0,
+ number=1,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=173,
+ serialized_end=217,
+)
+
+
+_TEXTSENTIMENTEVALUATIONMETRICS = _descriptor.Descriptor(
+ name="TextSentimentEvaluationMetrics",
+ full_name="google.cloud.automl.v1beta1.TextSentimentEvaluationMetrics",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="precision",
+ full_name="google.cloud.automl.v1beta1.TextSentimentEvaluationMetrics.precision",
+ index=0,
+ number=1,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="recall",
+ full_name="google.cloud.automl.v1beta1.TextSentimentEvaluationMetrics.recall",
+ index=1,
+ number=2,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="f1_score",
+ full_name="google.cloud.automl.v1beta1.TextSentimentEvaluationMetrics.f1_score",
+ index=2,
+ number=3,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="mean_absolute_error",
+ full_name="google.cloud.automl.v1beta1.TextSentimentEvaluationMetrics.mean_absolute_error",
+ index=3,
+ number=4,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="mean_squared_error",
+ full_name="google.cloud.automl.v1beta1.TextSentimentEvaluationMetrics.mean_squared_error",
+ index=4,
+ number=5,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="linear_kappa",
+ full_name="google.cloud.automl.v1beta1.TextSentimentEvaluationMetrics.linear_kappa",
+ index=5,
+ number=6,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="quadratic_kappa",
+ full_name="google.cloud.automl.v1beta1.TextSentimentEvaluationMetrics.quadratic_kappa",
+ index=6,
+ number=7,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="confusion_matrix",
+ full_name="google.cloud.automl.v1beta1.TextSentimentEvaluationMetrics.confusion_matrix",
+ index=7,
+ number=8,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="annotation_spec_id",
+ full_name="google.cloud.automl.v1beta1.TextSentimentEvaluationMetrics.annotation_spec_id",
+ index=8,
+ number=9,
+ type=9,
+ cpp_type=9,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=_b("\030\001"),
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=220,
+ serialized_end=545,
+)
+
+_TEXTSENTIMENTEVALUATIONMETRICS.fields_by_name[
+ "confusion_matrix"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_classification__pb2._CLASSIFICATIONEVALUATIONMETRICS_CONFUSIONMATRIX
+)
+DESCRIPTOR.message_types_by_name["TextSentimentAnnotation"] = _TEXTSENTIMENTANNOTATION
+DESCRIPTOR.message_types_by_name[
+ "TextSentimentEvaluationMetrics"
+] = _TEXTSENTIMENTEVALUATIONMETRICS
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+TextSentimentAnnotation = _reflection.GeneratedProtocolMessageType(
+ "TextSentimentAnnotation",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TEXTSENTIMENTANNOTATION,
+ __module__="google.cloud.automl_v1beta1.proto.text_sentiment_pb2",
+ __doc__="""Contains annotation details specific to text sentiment.
+
+
+ Attributes:
+ sentiment:
+ Output only. The sentiment with the semantic, as given to the
+ [AutoMl.ImportData][google.cloud.automl.v1beta1.AutoMl.ImportD
+ ata] when populating the dataset from which the model used for
+ the prediction had been trained. The sentiment values are
+ between 0 and
+ Dataset.text\_sentiment\_dataset\_metadata.sentiment\_max
+ (inclusive), with higher value meaning more positive
+ sentiment. They are completely relative, i.e. 0 means least
+ positive sentiment and sentiment\_max means the most positive
+ from the sentiments present in the train data. Therefore e.g.
+ if train data had only negative sentiment, then
+ sentiment\_max, would be still negative (although least
+ negative). The sentiment shouldn't be confused with "score" or
+ "magnitude" from the previous Natural Language Sentiment
+ Analysis API.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.TextSentimentAnnotation)
+ ),
+)
+_sym_db.RegisterMessage(TextSentimentAnnotation)
+
+TextSentimentEvaluationMetrics = _reflection.GeneratedProtocolMessageType(
+ "TextSentimentEvaluationMetrics",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TEXTSENTIMENTEVALUATIONMETRICS,
+ __module__="google.cloud.automl_v1beta1.proto.text_sentiment_pb2",
+ __doc__="""Model evaluation metrics for text sentiment problems.
+
+
+ Attributes:
+ precision:
+ Output only. Precision.
+ recall:
+ Output only. Recall.
+ f1_score:
+ Output only. The harmonic mean of recall and precision.
+ mean_absolute_error:
+ Output only. Mean absolute error. Only set for the overall
+ model evaluation, not for evaluation of a single annotation
+ spec.
+ mean_squared_error:
+ Output only. Mean squared error. Only set for the overall
+ model evaluation, not for evaluation of a single annotation
+ spec.
+ linear_kappa:
+ Output only. Linear weighted kappa. Only set for the overall
+ model evaluation, not for evaluation of a single annotation
+ spec.
+ quadratic_kappa:
+ Output only. Quadratic weighted kappa. Only set for the
+ overall model evaluation, not for evaluation of a single
+ annotation spec.
+ confusion_matrix:
+ Output only. Confusion matrix of the evaluation. Only set for
+ the overall model evaluation, not for evaluation of a single
+ annotation spec.
+ annotation_spec_id:
+ Output only. The annotation spec ids used for this evaluation.
+ Deprecated .
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.TextSentimentEvaluationMetrics)
+ ),
+)
+_sym_db.RegisterMessage(TextSentimentEvaluationMetrics)
+
+
+DESCRIPTOR._options = None
+_TEXTSENTIMENTEVALUATIONMETRICS.fields_by_name["annotation_spec_id"]._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/automl/google/cloud/automl_v1beta1/proto/text_sentiment_pb2_grpc.py b/automl/google/cloud/automl_v1beta1/proto/text_sentiment_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/text_sentiment_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/automl/google/cloud/automl_v1beta1/proto/translation.proto b/automl/google/cloud/automl_v1beta1/proto/translation.proto
new file mode 100644
index 000000000000..ed02b2af5fd6
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/translation.proto
@@ -0,0 +1,69 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.automl.v1beta1;
+
+import "google/cloud/automl/v1beta1/data_items.proto";
+import "google/api/annotations.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl";
+option java_multiple_files = true;
+option java_outer_classname = "TranslationProto";
+option java_package = "com.google.cloud.automl.v1beta1";
+option php_namespace = "Google\\Cloud\\AutoMl\\V1beta1";
+option ruby_package = "Google::Cloud::AutoML::V1beta1";
+
+// Dataset metadata that is specific to translation.
+message TranslationDatasetMetadata {
+ // Required. The BCP-47 language code of the source language.
+ string source_language_code = 1;
+
+ // Required. The BCP-47 language code of the target language.
+ string target_language_code = 2;
+}
+
+// Evaluation metrics for the dataset.
+message TranslationEvaluationMetrics {
+ // Output only. BLEU score.
+ double bleu_score = 1;
+
+ // Output only. BLEU score for base model.
+ double base_bleu_score = 2;
+}
+
+// Model metadata that is specific to translation.
+message TranslationModelMetadata {
+ // The resource name of the model to use as a baseline to train the custom
+ // model. If unset, we use the default base model provided by Google
+ // Translate. Format:
+ // `projects/{project_id}/locations/{location_id}/models/{model_id}`
+ string base_model = 1;
+
+ // Output only. Inferred from the dataset.
+ // The source languge (The BCP-47 language code) that is used for training.
+ string source_language_code = 2;
+
+ // Output only. The target languge (The BCP-47 language code) that is used for
+ // training.
+ string target_language_code = 3;
+}
+
+// Annotation details specific to translation.
+message TranslationAnnotation {
+ // Output only . The translated content.
+ TextSnippet translated_content = 1;
+}
diff --git a/automl/google/cloud/automl_v1beta1/proto/translation_pb2.py b/automl/google/cloud/automl_v1beta1/proto/translation_pb2.py
new file mode 100644
index 000000000000..15a081761c0f
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/translation_pb2.py
@@ -0,0 +1,370 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/automl_v1beta1/proto/translation.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.cloud.automl_v1beta1.proto import (
+ data_items_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_data__items__pb2,
+)
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/automl_v1beta1/proto/translation.proto",
+ package="google.cloud.automl.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\037com.google.cloud.automl.v1beta1B\020TranslationProtoP\001ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\312\002\033Google\\Cloud\\AutoMl\\V1beta1\352\002\036Google::Cloud::AutoML::V1beta1"
+ ),
+ serialized_pb=_b(
+ '\n3google/cloud/automl_v1beta1/proto/translation.proto\x12\x1bgoogle.cloud.automl.v1beta1\x1a\x32google/cloud/automl_v1beta1/proto/data_items.proto\x1a\x1cgoogle/api/annotations.proto"X\n\x1aTranslationDatasetMetadata\x12\x1c\n\x14source_language_code\x18\x01 \x01(\t\x12\x1c\n\x14target_language_code\x18\x02 \x01(\t"K\n\x1cTranslationEvaluationMetrics\x12\x12\n\nbleu_score\x18\x01 \x01(\x01\x12\x17\n\x0f\x62\x61se_bleu_score\x18\x02 \x01(\x01"j\n\x18TranslationModelMetadata\x12\x12\n\nbase_model\x18\x01 \x01(\t\x12\x1c\n\x14source_language_code\x18\x02 \x01(\t\x12\x1c\n\x14target_language_code\x18\x03 \x01(\t"]\n\x15TranslationAnnotation\x12\x44\n\x12translated_content\x18\x01 \x01(\x0b\x32(.google.cloud.automl.v1beta1.TextSnippetB\xb7\x01\n\x1f\x63om.google.cloud.automl.v1beta1B\x10TranslationProtoP\x01ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\xca\x02\x1bGoogle\\Cloud\\AutoMl\\V1beta1\xea\x02\x1eGoogle::Cloud::AutoML::V1beta1b\x06proto3'
+ ),
+ dependencies=[
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_data__items__pb2.DESCRIPTOR,
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ ],
+)
+
+
+_TRANSLATIONDATASETMETADATA = _descriptor.Descriptor(
+ name="TranslationDatasetMetadata",
+ full_name="google.cloud.automl.v1beta1.TranslationDatasetMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="source_language_code",
+ full_name="google.cloud.automl.v1beta1.TranslationDatasetMetadata.source_language_code",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="target_language_code",
+ full_name="google.cloud.automl.v1beta1.TranslationDatasetMetadata.target_language_code",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=166,
+ serialized_end=254,
+)
+
+
+_TRANSLATIONEVALUATIONMETRICS = _descriptor.Descriptor(
+ name="TranslationEvaluationMetrics",
+ full_name="google.cloud.automl.v1beta1.TranslationEvaluationMetrics",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="bleu_score",
+ full_name="google.cloud.automl.v1beta1.TranslationEvaluationMetrics.bleu_score",
+ index=0,
+ number=1,
+ type=1,
+ cpp_type=5,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="base_bleu_score",
+ full_name="google.cloud.automl.v1beta1.TranslationEvaluationMetrics.base_bleu_score",
+ index=1,
+ number=2,
+ type=1,
+ cpp_type=5,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=256,
+ serialized_end=331,
+)
+
+
+_TRANSLATIONMODELMETADATA = _descriptor.Descriptor(
+ name="TranslationModelMetadata",
+ full_name="google.cloud.automl.v1beta1.TranslationModelMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="base_model",
+ full_name="google.cloud.automl.v1beta1.TranslationModelMetadata.base_model",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="source_language_code",
+ full_name="google.cloud.automl.v1beta1.TranslationModelMetadata.source_language_code",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="target_language_code",
+ full_name="google.cloud.automl.v1beta1.TranslationModelMetadata.target_language_code",
+ index=2,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=333,
+ serialized_end=439,
+)
+
+
+_TRANSLATIONANNOTATION = _descriptor.Descriptor(
+ name="TranslationAnnotation",
+ full_name="google.cloud.automl.v1beta1.TranslationAnnotation",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="translated_content",
+ full_name="google.cloud.automl.v1beta1.TranslationAnnotation.translated_content",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=441,
+ serialized_end=534,
+)
+
+_TRANSLATIONANNOTATION.fields_by_name[
+ "translated_content"
+].message_type = (
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_data__items__pb2._TEXTSNIPPET
+)
+DESCRIPTOR.message_types_by_name[
+ "TranslationDatasetMetadata"
+] = _TRANSLATIONDATASETMETADATA
+DESCRIPTOR.message_types_by_name[
+ "TranslationEvaluationMetrics"
+] = _TRANSLATIONEVALUATIONMETRICS
+DESCRIPTOR.message_types_by_name["TranslationModelMetadata"] = _TRANSLATIONMODELMETADATA
+DESCRIPTOR.message_types_by_name["TranslationAnnotation"] = _TRANSLATIONANNOTATION
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+TranslationDatasetMetadata = _reflection.GeneratedProtocolMessageType(
+ "TranslationDatasetMetadata",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TRANSLATIONDATASETMETADATA,
+ __module__="google.cloud.automl_v1beta1.proto.translation_pb2",
+ __doc__="""Dataset metadata that is specific to translation.
+
+
+ Attributes:
+ source_language_code:
+ Required. The BCP-47 language code of the source language.
+ target_language_code:
+ Required. The BCP-47 language code of the target language.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.TranslationDatasetMetadata)
+ ),
+)
+_sym_db.RegisterMessage(TranslationDatasetMetadata)
+
+TranslationEvaluationMetrics = _reflection.GeneratedProtocolMessageType(
+ "TranslationEvaluationMetrics",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TRANSLATIONEVALUATIONMETRICS,
+ __module__="google.cloud.automl_v1beta1.proto.translation_pb2",
+ __doc__="""Evaluation metrics for the dataset.
+
+
+ Attributes:
+ bleu_score:
+ Output only. BLEU score.
+ base_bleu_score:
+ Output only. BLEU score for base model.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.TranslationEvaluationMetrics)
+ ),
+)
+_sym_db.RegisterMessage(TranslationEvaluationMetrics)
+
+TranslationModelMetadata = _reflection.GeneratedProtocolMessageType(
+ "TranslationModelMetadata",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TRANSLATIONMODELMETADATA,
+ __module__="google.cloud.automl_v1beta1.proto.translation_pb2",
+ __doc__="""Model metadata that is specific to translation.
+
+
+ Attributes:
+ base_model:
+ The resource name of the model to use as a baseline to train
+ the custom model. If unset, we use the default base model
+ provided by Google Translate. Format: ``projects/{project_id}/
+ locations/{location_id}/models/{model_id}``
+ source_language_code:
+ Output only. Inferred from the dataset. The source languge
+ (The BCP-47 language code) that is used for training.
+ target_language_code:
+ Output only. The target languge (The BCP-47 language code)
+ that is used for training.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.TranslationModelMetadata)
+ ),
+)
+_sym_db.RegisterMessage(TranslationModelMetadata)
+
+TranslationAnnotation = _reflection.GeneratedProtocolMessageType(
+ "TranslationAnnotation",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TRANSLATIONANNOTATION,
+ __module__="google.cloud.automl_v1beta1.proto.translation_pb2",
+ __doc__="""Annotation details specific to translation.
+
+
+ Attributes:
+ translated_content:
+ Output only . The translated content.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.TranslationAnnotation)
+ ),
+)
+_sym_db.RegisterMessage(TranslationAnnotation)
+
+
+DESCRIPTOR._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/automl/google/cloud/automl_v1beta1/proto/translation_pb2_grpc.py b/automl/google/cloud/automl_v1beta1/proto/translation_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/translation_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/automl/google/cloud/automl_v1beta1/proto/video.proto b/automl/google/cloud/automl_v1beta1/proto/video.proto
new file mode 100644
index 000000000000..b7c7325bc2f1
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/video.proto
@@ -0,0 +1,49 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.automl.v1beta1;
+
+import "google/cloud/automl/v1beta1/classification.proto";
+import "google/api/annotations.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl";
+option java_multiple_files = true;
+option java_outer_classname = "VideoProto";
+option java_package = "com.google.cloud.automl.v1beta1";
+option php_namespace = "Google\\Cloud\\AutoMl\\V1beta1";
+option ruby_package = "Google::Cloud::AutoML::V1beta1";
+
+// Dataset metadata specific to video classification.
+// All Video Classification datasets are treated as multi label.
+message VideoClassificationDatasetMetadata {
+
+}
+
+// Dataset metadata specific to video object tracking.
+message VideoObjectTrackingDatasetMetadata {
+
+}
+
+// Model metadata specific to video classification.
+message VideoClassificationModelMetadata {
+
+}
+
+// Model metadata specific to video object tracking.
+message VideoObjectTrackingModelMetadata {
+
+}
diff --git a/automl/google/cloud/automl_v1beta1/proto/video_pb2.py b/automl/google/cloud/automl_v1beta1/proto/video_pb2.py
new file mode 100644
index 000000000000..156c58b3b364
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/video_pb2.py
@@ -0,0 +1,189 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/automl_v1beta1/proto/video.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.cloud.automl_v1beta1.proto import (
+ classification_pb2 as google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_classification__pb2,
+)
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/automl_v1beta1/proto/video.proto",
+ package="google.cloud.automl.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\037com.google.cloud.automl.v1beta1B\nVideoProtoP\001ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\312\002\033Google\\Cloud\\AutoMl\\V1beta1\352\002\036Google::Cloud::AutoML::V1beta1"
+ ),
+ serialized_pb=_b(
+ '\n-google/cloud/automl_v1beta1/proto/video.proto\x12\x1bgoogle.cloud.automl.v1beta1\x1a\x36google/cloud/automl_v1beta1/proto/classification.proto\x1a\x1cgoogle/api/annotations.proto"$\n"VideoClassificationDatasetMetadata"$\n"VideoObjectTrackingDatasetMetadata""\n VideoClassificationModelMetadata""\n VideoObjectTrackingModelMetadataB\xb1\x01\n\x1f\x63om.google.cloud.automl.v1beta1B\nVideoProtoP\x01ZAgoogle.golang.org/genproto/googleapis/cloud/automl/v1beta1;automl\xca\x02\x1bGoogle\\Cloud\\AutoMl\\V1beta1\xea\x02\x1eGoogle::Cloud::AutoML::V1beta1b\x06proto3'
+ ),
+ dependencies=[
+ google_dot_cloud_dot_automl__v1beta1_dot_proto_dot_classification__pb2.DESCRIPTOR,
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ ],
+)
+
+
+_VIDEOCLASSIFICATIONDATASETMETADATA = _descriptor.Descriptor(
+ name="VideoClassificationDatasetMetadata",
+ full_name="google.cloud.automl.v1beta1.VideoClassificationDatasetMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=164,
+ serialized_end=200,
+)
+
+
+_VIDEOOBJECTTRACKINGDATASETMETADATA = _descriptor.Descriptor(
+ name="VideoObjectTrackingDatasetMetadata",
+ full_name="google.cloud.automl.v1beta1.VideoObjectTrackingDatasetMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=202,
+ serialized_end=238,
+)
+
+
+_VIDEOCLASSIFICATIONMODELMETADATA = _descriptor.Descriptor(
+ name="VideoClassificationModelMetadata",
+ full_name="google.cloud.automl.v1beta1.VideoClassificationModelMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=240,
+ serialized_end=274,
+)
+
+
+_VIDEOOBJECTTRACKINGMODELMETADATA = _descriptor.Descriptor(
+ name="VideoObjectTrackingModelMetadata",
+ full_name="google.cloud.automl.v1beta1.VideoObjectTrackingModelMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=276,
+ serialized_end=310,
+)
+
+DESCRIPTOR.message_types_by_name[
+ "VideoClassificationDatasetMetadata"
+] = _VIDEOCLASSIFICATIONDATASETMETADATA
+DESCRIPTOR.message_types_by_name[
+ "VideoObjectTrackingDatasetMetadata"
+] = _VIDEOOBJECTTRACKINGDATASETMETADATA
+DESCRIPTOR.message_types_by_name[
+ "VideoClassificationModelMetadata"
+] = _VIDEOCLASSIFICATIONMODELMETADATA
+DESCRIPTOR.message_types_by_name[
+ "VideoObjectTrackingModelMetadata"
+] = _VIDEOOBJECTTRACKINGMODELMETADATA
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+VideoClassificationDatasetMetadata = _reflection.GeneratedProtocolMessageType(
+ "VideoClassificationDatasetMetadata",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_VIDEOCLASSIFICATIONDATASETMETADATA,
+ __module__="google.cloud.automl_v1beta1.proto.video_pb2",
+ __doc__="""Dataset metadata specific to video classification. All Video
+ Classification datasets are treated as multi label.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.VideoClassificationDatasetMetadata)
+ ),
+)
+_sym_db.RegisterMessage(VideoClassificationDatasetMetadata)
+
+VideoObjectTrackingDatasetMetadata = _reflection.GeneratedProtocolMessageType(
+ "VideoObjectTrackingDatasetMetadata",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_VIDEOOBJECTTRACKINGDATASETMETADATA,
+ __module__="google.cloud.automl_v1beta1.proto.video_pb2",
+ __doc__="""Dataset metadata specific to video object tracking.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.VideoObjectTrackingDatasetMetadata)
+ ),
+)
+_sym_db.RegisterMessage(VideoObjectTrackingDatasetMetadata)
+
+VideoClassificationModelMetadata = _reflection.GeneratedProtocolMessageType(
+ "VideoClassificationModelMetadata",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_VIDEOCLASSIFICATIONMODELMETADATA,
+ __module__="google.cloud.automl_v1beta1.proto.video_pb2",
+ __doc__="""Model metadata specific to video classification.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.VideoClassificationModelMetadata)
+ ),
+)
+_sym_db.RegisterMessage(VideoClassificationModelMetadata)
+
+VideoObjectTrackingModelMetadata = _reflection.GeneratedProtocolMessageType(
+ "VideoObjectTrackingModelMetadata",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_VIDEOOBJECTTRACKINGMODELMETADATA,
+ __module__="google.cloud.automl_v1beta1.proto.video_pb2",
+ __doc__="""Model metadata specific to video object tracking.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.automl.v1beta1.VideoObjectTrackingModelMetadata)
+ ),
+)
+_sym_db.RegisterMessage(VideoObjectTrackingModelMetadata)
+
+
+DESCRIPTOR._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/automl/google/cloud/automl_v1beta1/proto/video_pb2_grpc.py b/automl/google/cloud/automl_v1beta1/proto/video_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/proto/video_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/automl/google/cloud/automl_v1beta1/tables/__init__.py b/automl/google/cloud/automl_v1beta1/tables/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/automl/google/cloud/automl_v1beta1/tables/gcs_client.py b/automl/google/cloud/automl_v1beta1/tables/gcs_client.py
new file mode 100644
index 000000000000..980fd40b1f90
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/tables/gcs_client.py
@@ -0,0 +1,137 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Wraps the Google Cloud Storage client library for use in tables helper."""
+
+import logging
+import time
+
+from google.api_core import exceptions
+
+try:
+ import pandas
+except ImportError: # pragma: NO COVER
+ pandas = None
+
+try:
+ from google.cloud import storage
+except ImportError: # pragma: NO COVER
+ storage = None
+
+_LOGGER = logging.getLogger(__name__)
+_PANDAS_REQUIRED = "pandas is required to verify type DataFrame."
+_STORAGE_REQUIRED = (
+ "google-cloud-storage is required to create a Google Cloud Storage client."
+)
+
+
+class GcsClient(object):
+ """Uploads Pandas DataFrame to a bucket in Google Cloud Storage."""
+
+ def __init__(self, bucket_name=None, client=None, credentials=None):
+ """Constructor.
+
+ Args:
+ bucket_name (Optional[str]): The name of Google Cloud Storage
+ bucket for this client to send requests to.
+ client (Optional[storage.Client]): A Google Cloud Storage Client
+ instance.
+ credentials (Optional[google.auth.credentials.Credentials]): The
+ authorization credentials to attach to requests. These
+ credentials identify this application to the service. If none
+ are specified, the client will attempt to ascertain the
+ credentials from the environment.
+ """
+ if storage is None:
+ raise ImportError(_STORAGE_REQUIRED)
+
+ if client is not None:
+ self.client = client
+ elif credentials is not None:
+ self.client = storage.Client(credentials=credentials)
+ else:
+ self.client = storage.Client()
+
+ self.bucket_name = bucket_name
+
+ def ensure_bucket_exists(self, project, region):
+ """Checks if a bucket named '{project}-automl-tables-staging' exists.
+
+ If this bucket doesn't exist, creates one.
+ If this bucket already exists in `project`, do nothing.
+ If this bucket exists in a different project that we don't have
+ access to, creates a bucket named
+ '{project}-automl-tables-staging-{create_timestamp}' because bucket's
+ name must be globally unique.
+ Save the created bucket's name and reuse this for future requests.
+
+ Args:
+ project (str): The project that stores the bucket.
+ region (str): The region of the bucket.
+
+ Returns:
+ A string representing the created bucket name.
+ """
+ if self.bucket_name is None:
+ self.bucket_name = "{}-automl-tables-staging".format(project)
+
+ try:
+ self.client.get_bucket(self.bucket_name)
+ except (exceptions.Forbidden, exceptions.NotFound) as e:
+ if isinstance(e, exceptions.Forbidden):
+ used_bucket_name = self.bucket_name
+ self.bucket_name = used_bucket_name + "-{}".format(int(time.time()))
+ _LOGGER.warning(
+ "Created a bucket named {} because a bucket named {} already exists in a different project.".format(
+ self.bucket_name, used_bucket_name
+ )
+ )
+
+ bucket = self.client.bucket(self.bucket_name)
+ bucket.create(project=project, location=region)
+
+ return self.bucket_name
+
+ def upload_pandas_dataframe(self, dataframe, uploaded_csv_name=None):
+ """Uploads a Pandas DataFrame as CSV to the bucket.
+
+ Args:
+ dataframe (pandas.DataFrame): The Pandas Dataframe to be uploaded.
+ uploaded_csv_name (Optional[str]): The name for the uploaded CSV.
+
+ Returns:
+ A string representing the GCS URI of the uploaded CSV.
+ """
+ if pandas is None:
+ raise ImportError(_PANDAS_REQUIRED)
+
+ if not isinstance(dataframe, pandas.DataFrame):
+ raise ValueError("'dataframe' must be a pandas.DataFrame instance.")
+
+ if self.bucket_name is None:
+ raise ValueError("Must ensure a bucket exists before uploading data.")
+
+ if uploaded_csv_name is None:
+ uploaded_csv_name = "automl-tables-dataframe-{}.csv".format(
+ int(time.time())
+ )
+ csv_string = dataframe.to_csv()
+
+ bucket = self.client.get_bucket(self.bucket_name)
+ blob = bucket.blob(uploaded_csv_name)
+ blob.upload_from_string(csv_string)
+
+ return "gs://{}/{}".format(self.bucket_name, uploaded_csv_name)
diff --git a/automl/google/cloud/automl_v1beta1/tables/tables_client.py b/automl/google/cloud/automl_v1beta1/tables/tables_client.py
new file mode 100644
index 000000000000..80e301fbd6bd
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/tables/tables_client.py
@@ -0,0 +1,2780 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""A tables helper for the google.cloud.automl_v1beta1 AutoML API"""
+
+import pkg_resources
+import logging
+
+from google.api_core.gapic_v1 import client_info
+from google.api_core import exceptions
+from google.cloud.automl_v1beta1 import gapic
+from google.cloud.automl_v1beta1.proto import data_types_pb2
+from google.cloud.automl_v1beta1.tables import gcs_client
+
+_GAPIC_LIBRARY_VERSION = pkg_resources.get_distribution("google-cloud-automl").version
+_LOGGER = logging.getLogger(__name__)
+
+
+class TablesClient(object):
+ """
+ AutoML Tables API helper.
+
+ This is intended to simplify usage of the auto-generated python client,
+ in particular for the `AutoML Tables product
+ `_.
+ """
+
+ def __init__(
+ self,
+ project=None,
+ region="us-central1",
+ client=None,
+ prediction_client=None,
+ gcs_client=None,
+ **kwargs
+ ):
+ """Constructor.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> from google.oauth2 import service_account
+ >>>
+ >>> client = automl_v1beta1.TablesClient(
+ ... credentials=service_account.Credentials.from_service_account_file('~/.gcp/account.json')
+ ... project='my-project', region='us-central1')
+ ...
+
+ Args:
+ project (Optional[string]): The project all future calls will
+ default to. Most methods take `project` as an optional
+ parameter, and can override your choice of `project` supplied
+ here.
+ region (Optional[string]): The region all future calls will
+ default to. Most methods take `region` as an optional
+ parameter, and can override your choice of `region` supplied
+ here. Note, only `us-central1` is supported to-date.
+ transport (Union[~.AutoMlGrpcTransport, Callable[[~.Credentials, type], ~.AutoMlGrpcTransport]):
+ A transport instance, responsible for actually making the API
+ calls. The default transport uses the gRPC protocol. This
+ argument may also be a callable which returns a transport
+ instance. Callables will be sent the credentials as the first
+ argument and the default transport class as the second
+ argument.
+ channel (grpc.Channel): DEPRECATED. A ``Channel`` instance
+ through which to make calls. This argument is mutually exclusive
+ with ``credentials``; providing both will raise an exception.
+ credentials (google.auth.credentials.Credentials): The
+ authorization credentials to attach to requests. These
+ credentials identify this application to the service. If none
+ are specified, the client will attempt to ascertain the
+ credentials from the environment.
+ This argument is mutually exclusive with providing a
+ transport instance to ``transport``; doing so will raise
+ an exception.
+ client_config (dict): DEPRECATED. A dictionary of call options for
+ each method. If not specified, the default configuration is used.
+ client_options (Union[dict, google.api_core.client_options.ClientOptions]):
+ Client options used to set user options on the client. API Endpoint
+ should be set through client_options.
+ """
+ version = _GAPIC_LIBRARY_VERSION
+ user_agent = "automl-tables-wrapper/{}".format(version)
+
+ client_info_ = kwargs.get("client_info")
+ if client_info_ is None:
+ client_info_ = client_info.ClientInfo(
+ user_agent=user_agent, gapic_version=version
+ )
+ else:
+ client_info_.user_agent = user_agent
+ client_info_.gapic_version = version
+
+ if client is None:
+ self.auto_ml_client = gapic.auto_ml_client.AutoMlClient(
+ client_info=client_info_, **kwargs
+ )
+ else:
+ self.auto_ml_client = client
+
+ if prediction_client is None:
+ self.prediction_client = gapic.prediction_service_client.PredictionServiceClient(
+ client_info=client_info_, **kwargs
+ )
+ else:
+ self.prediction_client = prediction_client
+
+ self.project = project
+ self.region = region
+ self.gcs_client = gcs_client
+
+ def __lookup_by_display_name(self, object_type, items, display_name):
+ relevant_items = [i for i in items if i.display_name == display_name]
+ if len(relevant_items) == 0:
+ raise exceptions.NotFound(
+ "The {} with display_name='{}' was not found.".format(
+ object_type, display_name
+ )
+ )
+ elif len(relevant_items) == 1:
+ return relevant_items[0]
+ else:
+ raise ValueError(
+ (
+ "Multiple {}s match display_name='{}': {}\n\n"
+ "Please use the `.name` (unique identifier) field instead"
+ ).format(
+ object_type,
+ display_name,
+ ", ".join([str(i) for i in relevant_items]),
+ )
+ )
+
+ def __location_path(self, project=None, region=None):
+ if project is None:
+ if self.project is None:
+ raise ValueError(
+ "Either initialize your client with a value "
+ "for 'project', or provide 'project' as a "
+ "parameter for this method."
+ )
+ project = self.project
+
+ if region is None:
+ if self.region is None:
+ raise ValueError(
+ "Either initialize your client with a value "
+ "for 'region', or provide 'region' as a "
+ "parameter for this method."
+ )
+ region = self.region
+
+ return self.auto_ml_client.location_path(project, region)
+
+ # the returned metadata object doesn't allow for updating fields, so
+ # we need to manually copy user-updated fields over
+ def __update_metadata(self, metadata, k, v):
+ new_metadata = {}
+ new_metadata["ml_use_column_spec_id"] = metadata.ml_use_column_spec_id
+ new_metadata["weight_column_spec_id"] = metadata.weight_column_spec_id
+ new_metadata["target_column_spec_id"] = metadata.target_column_spec_id
+ new_metadata[k] = v
+
+ return new_metadata
+
+ def __dataset_from_args(
+ self,
+ dataset=None,
+ dataset_display_name=None,
+ dataset_name=None,
+ project=None,
+ region=None,
+ **kwargs
+ ):
+ if dataset is None and dataset_display_name is None and dataset_name is None:
+ raise ValueError(
+ "One of 'dataset', 'dataset_name' or "
+ "'dataset_display_name' must be set."
+ )
+ # we prefer to make a live call here in the case that the
+ # dataset object is out-of-date
+ if dataset is not None:
+ dataset_name = dataset.name
+
+ return self.get_dataset(
+ dataset_display_name=dataset_display_name,
+ dataset_name=dataset_name,
+ project=project,
+ region=region,
+ **kwargs
+ )
+
+ def __model_from_args(
+ self,
+ model=None,
+ model_display_name=None,
+ model_name=None,
+ project=None,
+ region=None,
+ **kwargs
+ ):
+ if model is None and model_display_name is None and model_name is None:
+ raise ValueError(
+ "One of 'model', 'model_name' or " "'model_display_name' must be set."
+ )
+ # we prefer to make a live call here in the case that the
+ # model object is out-of-date
+ if model is not None:
+ model_name = model.name
+
+ return self.get_model(
+ model_display_name=model_display_name,
+ model_name=model_name,
+ project=project,
+ region=region,
+ **kwargs
+ )
+
+ def __dataset_name_from_args(
+ self,
+ dataset=None,
+ dataset_display_name=None,
+ dataset_name=None,
+ project=None,
+ region=None,
+ **kwargs
+ ):
+ if dataset is None and dataset_display_name is None and dataset_name is None:
+ raise ValueError(
+ "One of 'dataset', 'dataset_name' or "
+ "'dataset_display_name' must be set."
+ )
+
+ if dataset_name is None:
+ if dataset is None:
+ dataset = self.get_dataset(
+ dataset_display_name=dataset_display_name,
+ project=project,
+ region=region,
+ **kwargs
+ )
+
+ dataset_name = dataset.name
+ else:
+ # we do this to force a NotFound error when needed
+ self.get_dataset(
+ dataset_name=dataset_name, project=project, region=region, **kwargs
+ )
+ return dataset_name
+
+ def __table_spec_name_from_args(
+ self,
+ table_spec_index=0,
+ dataset=None,
+ dataset_display_name=None,
+ dataset_name=None,
+ project=None,
+ region=None,
+ **kwargs
+ ):
+ dataset_name = self.__dataset_name_from_args(
+ dataset=dataset,
+ dataset_name=dataset_name,
+ dataset_display_name=dataset_display_name,
+ project=project,
+ region=region,
+ **kwargs
+ )
+
+ table_specs = [
+ t for t in self.list_table_specs(dataset_name=dataset_name, **kwargs)
+ ]
+
+ table_spec_full_id = table_specs[table_spec_index].name
+ return table_spec_full_id
+
+ def __model_name_from_args(
+ self,
+ model=None,
+ model_display_name=None,
+ model_name=None,
+ project=None,
+ region=None,
+ **kwargs
+ ):
+ if model is None and model_display_name is None and model_name is None:
+ raise ValueError(
+ "One of 'model', 'model_name' or " "'model_display_name' must be set."
+ )
+
+ if model_name is None:
+ if model is None:
+ model = self.get_model(
+ model_display_name=model_display_name,
+ project=project,
+ region=region,
+ **kwargs
+ )
+ model_name = model.name
+ else:
+ # we do this to force a NotFound error when needed
+ self.get_model(
+ model_name=model_name, project=project, region=region, **kwargs
+ )
+ return model_name
+
+ def __log_operation_info(self, message, op):
+ name = "UNKNOWN"
+ try:
+ if (
+ op is not None
+ and op.operation is not None
+ and op.operation.name is not None
+ ):
+ name = op.operation.name
+ except AttributeError:
+ pass
+ _LOGGER.info(
+ (
+ "Operation '{}' is running in the background. The returned "
+ "Operation '{}' can be used to query or block on the status "
+ "of this operation. Ending your python session will _not_ "
+ "cancel this operation. Read the documentation here:\n\n"
+ "\thttps://googleapis.dev/python/google-api-core/latest/operation.html\n\n"
+ "for more information on the Operation class."
+ ).format(message, name)
+ )
+ return op
+
+ def __column_spec_name_from_args(
+ self,
+ dataset=None,
+ dataset_display_name=None,
+ dataset_name=None,
+ table_spec_name=None,
+ table_spec_index=0,
+ column_spec_name=None,
+ column_spec_display_name=None,
+ project=None,
+ region=None,
+ **kwargs
+ ):
+ column_specs = self.list_column_specs(
+ dataset=dataset,
+ dataset_display_name=dataset_display_name,
+ dataset_name=dataset_name,
+ table_spec_name=table_spec_name,
+ table_spec_index=table_spec_index,
+ project=project,
+ region=region,
+ **kwargs
+ )
+ if column_spec_display_name is not None:
+ column_specs = {s.display_name: s for s in column_specs}
+ if column_specs.get(column_spec_display_name) is None:
+ raise exceptions.NotFound(
+ "No column with "
+ + "column_spec_display_name: '{}' found".format(
+ column_spec_display_name
+ )
+ )
+ column_spec_name = column_specs[column_spec_display_name].name
+ elif column_spec_name is not None:
+ column_specs = {s.name: s for s in column_specs}
+ if column_specs.get(column_spec_name) is None:
+ raise exceptions.NotFound(
+ "No column with "
+ + "column_spec_name: '{}' found".format(column_spec_name)
+ )
+ else:
+ raise ValueError(
+ "Either supply 'column_spec_name' or "
+ "'column_spec_display_name' for the column to update"
+ )
+
+ return column_spec_name
+
+ def __type_code_to_value_type(self, type_code, value):
+ if value is None:
+ return {"null_value": 0}
+ elif type_code == data_types_pb2.FLOAT64:
+ return {"number_value": value}
+ elif type_code == data_types_pb2.TIMESTAMP:
+ return {"string_value": value}
+ elif type_code == data_types_pb2.STRING:
+ return {"string_value": value}
+ elif type_code == data_types_pb2.ARRAY:
+ return {"list_value": value}
+ elif type_code == data_types_pb2.STRUCT:
+ return {"struct_value": value}
+ elif type_code == data_types_pb2.CATEGORY:
+ return {"string_value": value}
+ else:
+ raise ValueError("Unknown type_code: {}".format(type_code))
+
+ def __ensure_gcs_client_is_initialized(self, credentials=None):
+ """Checks if GCS client is initialized. Initializes it if not.
+
+ Args:
+ credentials (google.auth.credentials.Credentials): The
+ authorization credentials to attach to requests. These
+ credentials identify this application to the service. If none
+ are specified, the client will attempt to ascertain the
+ credentials from the environment.
+ """
+ if self.gcs_client is None:
+ self.gcs_client = gcs_client.GcsClient(credentials=credentials)
+
+ def list_datasets(self, project=None, region=None, **kwargs):
+ """List all datasets in a particular project and region.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> from google.oauth2 import service_account
+ >>>
+ >>> client = automl_v1beta1.TablesClient(
+ ... credentials=service_account.Credentials.from_service_account_file('~/.gcp/account.json')
+ ... project='my-project', region='us-central1')
+ ...
+ >>> ds = client.list_datasets()
+ >>>
+ >>> for d in ds:
+ ... # do something
+ ... pass
+ ...
+
+ Args:
+ project (Optional[string]):
+ If you have initialized the client with a value for `project`
+ it will be used if this parameter is not supplied. Keep in
+ mind, the service account this client was initialized with must
+ have access to this project.
+ region (Optional[string]):
+ If you have initialized the client with a value for `region` it
+ will be used if this parameter is not supplied.
+
+ Returns:
+ A :class:`~google.api_core.page_iterator.PageIterator` instance.
+ An iterable of :class:`~google.cloud.automl_v1beta1.types.Dataset`
+ instances. You can also iterate over the pages of the response
+ using its `pages` property.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If required parameters are missing.
+ """
+ return self.auto_ml_client.list_datasets(
+ self.__location_path(project=project, region=region), **kwargs
+ )
+
+ def get_dataset(
+ self,
+ project=None,
+ region=None,
+ dataset_name=None,
+ dataset_display_name=None,
+ **kwargs
+ ):
+ """Gets a single dataset in a particular project and region.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> from google.oauth2 import service_account
+ >>>
+ >>> client = automl_v1beta1.TablesClient(
+ ... credentials=service_account.Credentials.from_service_account_file('~/.gcp/account.json')
+ ... project='my-project', region='us-central1')
+ ...
+ >>> d = client.get_dataset(dataset_display_name='my_dataset')
+ >>>
+
+ Args:
+ project (Optional[string]):
+ If you have initialized the client with a value for `project`
+ it will be used if this parameter is not supplied. Keep in
+ mind, the service account this client was initialized with must
+ have access to this project.
+ region (Optional[string]):
+ If you have initialized the client with a value for `region` it
+ will be used if this parameter is not supplied.
+ dataset_name (Optional[string]):
+ This is the fully-qualified name generated by the AutoML API
+ for this dataset. This is not to be confused with the
+ human-assigned `dataset_display_name` that is provided when
+ creating a dataset. Either `dataset_name` or
+ `dataset_display_name` must be provided.
+ dataset_display_name (Optional[string]):
+ This is the name you provided for the dataset when first
+ creating it. Either `dataset_name` or `dataset_display_name`
+ must be provided.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types.Dataset` instance if
+ found, `None` otherwise.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If required parameters are missing.
+ """
+ if dataset_name is None and dataset_display_name is None:
+ raise ValueError(
+ "One of 'dataset_name' or " "'dataset_display_name' must be set."
+ )
+
+ if dataset_name is not None:
+ return self.auto_ml_client.get_dataset(dataset_name, **kwargs)
+
+ return self.__lookup_by_display_name(
+ "dataset",
+ self.list_datasets(project, region, **kwargs),
+ dataset_display_name,
+ )
+
+ def create_dataset(
+ self, dataset_display_name, metadata={}, project=None, region=None, **kwargs
+ ):
+ """Create a dataset. Keep in mind, importing data is a separate step.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> from google.oauth2 import service_account
+ >>>
+ >>> client = automl_v1beta1.TablesClient(
+ ... credentials=service_account.Credentials.from_service_account_file('~/.gcp/account.json')
+ ... project='my-project', region='us-central1')
+ ...
+ >>> d = client.create_dataset(dataset_display_name='my_dataset')
+ >>>
+
+ Args:
+ project (Optional[string]):
+ If you have initialized the client with a value for `project`
+ it will be used if this parameter is not supplied. Keep in
+ mind, the service account this client was initialized with must
+ have access to this project.
+ region (Optional[string]):
+ If you have initialized the client with a value for `region` it
+ will be used if this parameter is not supplied.
+ dataset_display_name (string):
+ A human-readable name to refer to this dataset by.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types.Dataset` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If required parameters are missing.
+ """
+ return self.auto_ml_client.create_dataset(
+ self.__location_path(project, region),
+ {"display_name": dataset_display_name, "tables_dataset_metadata": metadata},
+ **kwargs
+ )
+
+ def delete_dataset(
+ self,
+ dataset=None,
+ dataset_display_name=None,
+ dataset_name=None,
+ project=None,
+ region=None,
+ **kwargs
+ ):
+ """Deletes a dataset. This does not delete any models trained on
+ this dataset.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> from google.oauth2 import service_account
+ >>>
+ >>> client = automl_v1beta1.TablesClient(
+ ... credentials=service_account.Credentials.from_service_account_file('~/.gcp/account.json')
+ ... project='my-project', region='us-central1')
+ ...
+ >>> op = client.delete_dataset(dataset_display_name='my_dataset')
+ >>>
+ >>> op.result() # blocks on delete request
+ >>>
+
+ Args:
+ project (Optional[string]):
+ If you have initialized the client with a value for `project`
+ it will be used if this parameter is not supplied. Keep in
+ mind, the service account this client was initialized with must
+ have access to this project.
+ region (Optional[string]):
+ If you have initialized the client with a value for `region` it
+ will be used if this parameter is not supplied.
+ dataset_display_name (Optional[string]):
+ The human-readable name given to the dataset you want to
+ delete. This must be supplied if `dataset` or `dataset_name`
+ are not supplied.
+ dataset_name (Optional[string]):
+ The AutoML-assigned name given to the dataset you want to
+ delete. This must be supplied if `dataset_display_name` or
+ `dataset` are not supplied.
+ dataset (Optional[Dataset]):
+ The `Dataset` instance you want to delete. This must be
+ supplied if `dataset_display_name` or `dataset_name` are not
+ supplied.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types._OperationFuture`
+ instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If required parameters are missing.
+ """
+ try:
+ dataset_name = self.__dataset_name_from_args(
+ dataset=dataset,
+ dataset_name=dataset_name,
+ dataset_display_name=dataset_display_name,
+ project=project,
+ region=region,
+ **kwargs
+ )
+ # delete is idempotent
+ except exceptions.NotFound:
+ return None
+
+ op = self.auto_ml_client.delete_dataset(dataset_name, **kwargs)
+ self.__log_operation_info("Delete dataset", op)
+ return op
+
+ def import_data(
+ self,
+ dataset=None,
+ dataset_display_name=None,
+ dataset_name=None,
+ pandas_dataframe=None,
+ gcs_input_uris=None,
+ bigquery_input_uri=None,
+ project=None,
+ region=None,
+ credentials=None,
+ **kwargs
+ ):
+ """Imports data into a dataset.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> from google.oauth2 import service_account
+ >>>
+ >>> client = automl_v1beta1.TablesClient(
+ ... credentials=service_account.Credentials.from_service_account_file('~/.gcp/account.json')
+ ... project='my-project', region='us-central1')
+ ...
+ >>> d = client.create_dataset(dataset_display_name='my_dataset')
+ >>>
+ >>> client.import_data(dataset=d,
+ ... gcs_input_uris='gs://cloud-ml-tables-data/bank-marketing.csv')
+ ...
+ >>> def callback(operation_future):
+ ... result = operation_future.result()
+ ...
+ >>> response.add_done_callback(callback)
+ >>>
+
+ Args:
+ project (Optional[string]):
+ If you have initialized the client with a value for `project`
+ it will be used if this parameter is not supplied. Keep in
+ mind, the service account this client was initialized with must
+ have access to this project.
+ region (Optional[string]):
+ If you have initialized the client with a value for `region` it
+ will be used if this parameter is not supplied.
+ credentials (Optional[google.auth.credentials.Credentials]): The
+ authorization credentials to attach to requests. These
+ credentials identify this application to the service. If none
+ are specified, the client will attempt to ascertain the
+ credentials from the environment.
+ dataset_display_name (Optional[string]):
+ The human-readable name given to the dataset you want to import
+ data into. This must be supplied if `dataset` or `dataset_name`
+ are not supplied.
+ dataset_name (Optional[string]):
+ The AutoML-assigned name given to the dataset you want to
+ import data into. This must be supplied if
+ `dataset_display_name` or `dataset` are not supplied.
+ dataset (Optional[Dataset]):
+ The `Dataset` instance you want to import data into. This must
+ be supplied if `dataset_display_name` or `dataset_name` are not
+ supplied.
+ pandas_dataframe (Optional[pandas.DataFrame]):
+ A Pandas Dataframe object containing the data to import. The data
+ will be converted to CSV, and this CSV will be staged to GCS in
+ `gs://{project}-automl-tables-staging/{uploaded_csv_name}`
+ This parameter must be supplied if neither `gcs_input_uris` nor
+ `bigquery_input_uri` is supplied.
+ gcs_input_uris (Optional[Union[string, Sequence[string]]]):
+ Either a single `gs://..` prefixed URI, or a list of URIs
+ referring to GCS-hosted CSV files containing the data to
+ import. This must be supplied if neither `bigquery_input_uri`
+ nor `pandas_dataframe` is supplied.
+ bigquery_input_uri (Optional[string]):
+ A URI pointing to the BigQuery table containing the data to
+ import. This must be supplied if neither `gcs_input_uris` nor
+ `pandas_dataframe` is supplied.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types._OperationFuture`
+ instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If required parameters are missing.
+ """
+ dataset_name = self.__dataset_name_from_args(
+ dataset=dataset,
+ dataset_name=dataset_name,
+ dataset_display_name=dataset_display_name,
+ project=project,
+ region=region,
+ **kwargs
+ )
+
+ request = {}
+
+ if pandas_dataframe is not None:
+ self.__ensure_gcs_client_is_initialized(credentials)
+ self.gcs_client.ensure_bucket_exists(project, region)
+ gcs_input_uri = self.gcs_client.upload_pandas_dataframe(pandas_dataframe)
+ request = {"gcs_source": {"input_uris": [gcs_input_uri]}}
+ elif gcs_input_uris is not None:
+ if type(gcs_input_uris) != list:
+ gcs_input_uris = [gcs_input_uris]
+ request = {"gcs_source": {"input_uris": gcs_input_uris}}
+ elif bigquery_input_uri is not None:
+ request = {"bigquery_source": {"input_uri": bigquery_input_uri}}
+ else:
+ raise ValueError(
+ "One of 'gcs_input_uris', or 'bigquery_input_uri', or 'pandas_dataframe' must be set."
+ )
+
+ op = self.auto_ml_client.import_data(dataset_name, request, **kwargs)
+ self.__log_operation_info("Data import", op)
+ return op
+
+ def export_data(
+ self,
+ dataset=None,
+ dataset_display_name=None,
+ dataset_name=None,
+ gcs_output_uri_prefix=None,
+ bigquery_output_uri=None,
+ project=None,
+ region=None,
+ **kwargs
+ ):
+ """Exports data from a dataset.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> from google.oauth2 import service_account
+ >>>
+ >>> client = automl_v1beta1.TablesClient(
+ ... credentials=service_account.Credentials.from_service_account_file('~/.gcp/account.json')
+ ... project='my-project', region='us-central1')
+ ...
+ >>> d = client.create_dataset(dataset_display_name='my_dataset')
+ >>>
+ >>> client.export_data(dataset=d,
+ ... gcs_output_uri_prefix='gs://cloud-ml-tables-data/bank-marketing.csv')
+ ...
+ >>> def callback(operation_future):
+ ... result = operation_future.result()
+ ...
+ >>> response.add_done_callback(callback)
+ >>>
+
+ Args:
+ project (Optional[string]):
+ If you have initialized the client with a value for `project`
+ it will be used if this parameter is not supplied. Keep in
+ mind, the service account this client was initialized with must
+ have access to this project.
+ region (Optional[string]):
+ If you have initialized the client with a value for `region` it
+ will be used if this parameter is not supplied.
+ dataset_display_name (Optional[string]):
+ The human-readable name given to the dataset you want to export
+ data from. This must be supplied if `dataset` or `dataset_name`
+ are not supplied.
+ dataset_name (Optional[string]):
+ The AutoML-assigned name given to the dataset you want to
+ export data from. This must be supplied if
+ `dataset_display_name` or `dataset` are not supplied.
+ dataset (Optional[Dataset]):
+ The `Dataset` instance you want to export data from. This must
+ be supplied if `dataset_display_name` or `dataset_name` are not
+ supplied.
+ gcs_output_uri_prefix (Optional[Union[string, Sequence[string]]]):
+ A single `gs://..` prefixed URI to export to. This must be
+ supplied if `bigquery_output_uri` is not.
+ bigquery_output_uri (Optional[string]):
+ A URI pointing to the BigQuery table containing the data to
+ export. This must be supplied if `gcs_output_uri_prefix` is not.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types._OperationFuture`
+ instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If required parameters are missing.
+ """
+ dataset_name = self.__dataset_name_from_args(
+ dataset=dataset,
+ dataset_name=dataset_name,
+ dataset_display_name=dataset_display_name,
+ project=project,
+ region=region,
+ **kwargs
+ )
+
+ request = {}
+ if gcs_output_uri_prefix is not None:
+ request = {"gcs_destination": {"output_uri_prefix": gcs_output_uri_prefix}}
+ elif bigquery_output_uri is not None:
+ request = {"bigquery_destination": {"output_uri": bigquery_output_uri}}
+ else:
+ raise ValueError(
+ "One of 'gcs_output_uri_prefix', or 'bigquery_output_uri' must be set."
+ )
+
+ op = self.auto_ml_client.export_data(dataset_name, request, **kwargs)
+ self.__log_operation_info("Export data", op)
+ return op
+
+ def get_table_spec(self, table_spec_name, project=None, region=None, **kwargs):
+ """Gets a single table spec in a particular project and region.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> from google.oauth2 import service_account
+ >>>
+ >>> client = automl_v1beta1.TablesClient(
+ ... credentials=service_account.Credentials.from_service_account_file('~/.gcp/account.json')
+ ... project='my-project', region='us-central1')
+ ...
+ >>> d = client.get_table_spec('my_table_spec')
+ >>>
+
+ Args:
+ table_spec_name (string):
+ This is the fully-qualified name generated by the AutoML API
+ for this table spec.
+ project (Optional[string]):
+ If you have initialized the client with a value for `project`
+ it will be used if this parameter is not supplied. Keep in
+ mind, the service account this client was initialized with must
+ have access to this project.
+ region (Optional[string]):
+ If you have initialized the client with a value for `region` it
+ will be used if this parameter is not supplied.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types.TableSpec` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If required parameters are missing.
+ """
+ return self.auto_ml_client.get_table_spec(table_spec_name, **kwargs)
+
+ def list_table_specs(
+ self,
+ dataset=None,
+ dataset_display_name=None,
+ dataset_name=None,
+ project=None,
+ region=None,
+ **kwargs
+ ):
+ """Lists table specs.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> from google.oauth2 import service_account
+ >>>
+ >>> client = automl_v1beta1.TablesClient(
+ ... credentials=service_account.Credentials.from_service_account_file('~/.gcp/account.json')
+ ... project='my-project', region='us-central1')
+ ...
+ >>> for s in client.list_table_specs(dataset_display_name='my_dataset')
+ ... # process the spec
+ ... pass
+ ...
+
+ Args:
+ project (Optional[string]):
+ If you have initialized the client with a value for `project`
+ it will be used if this parameter is not supplied. Keep in
+ mind, the service account this client was initialized with must
+ have access to this project.
+ region (Optional[string]):
+ If you have initialized the client with a value for `region` it
+ will be used if this parameter is not supplied.
+ dataset_display_name (Optional[string]):
+ The human-readable name given to the dataset you want to read
+ specs from. This must be supplied if `dataset` or
+ `dataset_name` are not supplied.
+ dataset_name (Optional[string]):
+ The AutoML-assigned name given to the dataset you want to read
+ specs from. This must be supplied if `dataset_display_name` or
+ `dataset` are not supplied.
+ dataset (Optional[Dataset]):
+ The `Dataset` instance you want to read specs from. This must
+ be supplied if `dataset_display_name` or `dataset_name` are not
+ supplied.
+
+ Returns:
+ A :class:`~google.api_core.page_iterator.PageIterator` instance.
+ An iterable of
+ :class:`~google.cloud.automl_v1beta1.types.TableSpec` instances.
+ You can also iterate over the pages of the response using its
+ `pages` property.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If required parameters are missing.
+ """
+ dataset_name = self.__dataset_name_from_args(
+ dataset=dataset,
+ dataset_name=dataset_name,
+ dataset_display_name=dataset_display_name,
+ project=project,
+ region=region,
+ **kwargs
+ )
+
+ return self.auto_ml_client.list_table_specs(dataset_name, **kwargs)
+
+ def get_column_spec(self, column_spec_name, project=None, region=None, **kwargs):
+ """Gets a single column spec in a particular project and region.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> from google.oauth2 import service_account
+ >>>
+ >>> client = automl_v1beta1.TablesClient(
+ ... credentials=service_account.Credentials.from_service_account_file('~/.gcp/account.json')
+ ... project='my-project', region='us-central1')
+ ...
+ >>> d = client.get_column_spec('my_column_spec')
+ >>>
+
+ Args:
+ column_spec_name (string):
+ This is the fully-qualified name generated by the AutoML API
+ for this column spec.
+ project (Optional[string]):
+ If you have initialized the client with a value for `project`
+ it will be used if this parameter is not supplied. Keep in
+ mind, the service account this client was initialized with must
+ have access to this project.
+ region (Optional[string]):
+ If you have initialized the client with a value for `region` it
+ will be used if this parameter is not supplied.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types.ColumnSpec` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If required parameters are missing.
+ """
+ return self.auto_ml_client.get_column_spec(column_spec_name, **kwargs)
+
+ def list_column_specs(
+ self,
+ dataset=None,
+ dataset_display_name=None,
+ dataset_name=None,
+ table_spec_name=None,
+ table_spec_index=0,
+ project=None,
+ region=None,
+ **kwargs
+ ):
+ """Lists column specs.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> from google.oauth2 import service_account
+ >>>
+ >>> client = automl_v1beta1.TablesClient(
+ ... credentials=service_account.Credentials.from_service_account_file('~/.gcp/account.json')
+ ... project='my-project', region='us-central1')
+ ...
+ >>> for s in client.list_column_specs(dataset_display_name='my_dataset')
+ ... # process the spec
+ ... pass
+ ...
+
+ Args:
+ project (Optional[string]):
+ If you have initialized the client with a value for `project`
+ it will be used if this parameter is not supplied. Keep in
+ mind, the service account this client was initialized with must
+ have access to this project.
+ region (Optional[string]):
+ If you have initialized the client with a value for `region` it
+ will be used if this parameter is not supplied.
+ table_spec_name (Optional[string]):
+ The AutoML-assigned name for the table whose specs you want to
+ read. If not supplied, the client can determine this name from
+ a source `Dataset` object.
+ table_spec_index (Optional[int]):
+ If no `table_spec_name` was provided, we use this index to
+ determine which table to read column specs from.
+ dataset_display_name (Optional[string]):
+ The human-readable name given to the dataset you want to read
+ specs from. If no `table_spec_name` is supplied, this will be
+ used together with `table_spec_index` to infer the name of
+ table to read specs from. This must be supplied if
+ `table_spec_name`, `dataset` or `dataset_name` are not
+ supplied.
+ dataset_name (Optional[string]):
+ The AutoML-assigned name given to the dataset you want to read
+ specs from. If no `table_spec_name` is supplied, this will be
+ used together with `table_spec_index` to infer the name of
+ table to read specs from. This must be supplied if
+ `table_spec_name`, `dataset` or `dataset_display_name` are not
+ supplied.
+ dataset (Optional[Dataset]):
+ The `Dataset` instance you want to read specs from. If no
+ `table_spec_name` is supplied, this will be used together with
+ `table_spec_index` to infer the name of table to read specs
+ from. This must be supplied if `table_spec_name`,
+ `dataset_name` or `dataset_display_name` are not supplied.
+
+ Returns:
+ A :class:`~google.api_core.page_iterator.PageIterator` instance.
+ An iterable of
+ :class:`~google.cloud.automl_v1beta1.types.ColumnSpec` instances.
+ You can also iterate over the pages of the response using its
+ `pages` property.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If required parameters are missing.
+ """
+ if table_spec_name is None:
+ table_specs = [
+ t
+ for t in self.list_table_specs(
+ dataset=dataset,
+ dataset_display_name=dataset_display_name,
+ dataset_name=dataset_name,
+ project=project,
+ region=region,
+ **kwargs
+ )
+ ]
+
+ table_spec_name = table_specs[table_spec_index].name
+
+ return self.auto_ml_client.list_column_specs(table_spec_name, **kwargs)
+
+ def update_column_spec(
+ self,
+ dataset=None,
+ dataset_display_name=None,
+ dataset_name=None,
+ table_spec_name=None,
+ table_spec_index=0,
+ column_spec_name=None,
+ column_spec_display_name=None,
+ type_code=None,
+ nullable=None,
+ project=None,
+ region=None,
+ **kwargs
+ ):
+ """Updates a column's specs.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> from google.oauth2 import service_account
+ >>>
+ >>> client = automl_v1beta1.TablesClient(
+ ... credentials=service_account.Credentials.from_service_account_file('~/.gcp/account.json')
+ ... project='my-project', region='us-central1')
+ ...
+ >>> client.update_column_specs(dataset_display_name='my_dataset',
+ ... column_spec_display_name='Outcome', type_code='CATEGORY')
+ ...
+
+ Args:
+ project (Optional[string]):
+ If you have initialized the client with a value for `project`
+ it will be used if this parameter is not supplied. Keep in
+ mind, the service account this client was initialized with must
+ have access to this project.
+ region (Optional[string]):
+ If you have initialized the client with a value for `region` it
+ will be used if this parameter is not supplied.
+ column_spec_name (Optional[string]):
+ The name AutoML-assigned name for the column you want to
+ update.
+ column_spec_display_name (Optional[string]):
+ The human-readable name of the column you want to update. If
+ this is supplied in place of `column_spec_name`, you also need
+ to provide either a way to lookup the source dataset (using one
+ of the `dataset*` kwargs), or the `table_spec_name` of the
+ table this column belongs to.
+ table_spec_name (Optional[string]):
+ The AutoML-assigned name for the table whose specs you want to
+ update. If not supplied, the client can determine this name
+ from a source `Dataset` object.
+ table_spec_index (Optional[int]):
+ If no `table_spec_name` was provided, we use this index to
+ determine which table to update column specs on.
+ dataset_display_name (Optional[string]):
+ The human-readable name given to the dataset you want to update
+ specs on. If no `table_spec_name` is supplied, this will be
+ used together with `table_spec_index` to infer the name of
+ table to update specs on. This must be supplied if
+ `table_spec_name`, `dataset` or `dataset_name` are not
+ supplied.
+ dataset_name (Optional[string]):
+ The AutoML-assigned name given to the dataset you want to
+ update specs one. If no `table_spec_name` is supplied, this
+ will be used together with `table_spec_index` to infer the name
+ of table to update specs on. This must be supplied if
+ `table_spec_name`, `dataset` or `dataset_display_name` are not
+ supplied.
+ dataset (Optional[Dataset]):
+ The `Dataset` instance you want to update specs on. If no
+ `table_spec_name` is supplied, this will be used together with
+ `table_spec_index` to infer the name of table to update specs
+ on. This must be supplied if `table_spec_name`, `dataset_name`
+ or `dataset_display_name` are not supplied.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types.ColumnSpec` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If required parameters are missing.
+ """
+ column_spec_name = self.__column_spec_name_from_args(
+ dataset=dataset,
+ dataset_display_name=dataset_display_name,
+ dataset_name=dataset_name,
+ table_spec_name=table_spec_name,
+ table_spec_index=table_spec_index,
+ column_spec_name=column_spec_name,
+ column_spec_display_name=column_spec_display_name,
+ project=project,
+ region=region,
+ **kwargs
+ )
+
+ # type code must always be set
+ if type_code is None:
+ # this index is safe, we would have already thrown a NotFound
+ # had the column_spec_name not existed
+ type_code = {
+ s.name: s
+ for s in self.list_column_specs(
+ dataset=dataset,
+ dataset_display_name=dataset_display_name,
+ dataset_name=dataset_name,
+ table_spec_name=table_spec_name,
+ table_spec_index=table_spec_index,
+ project=project,
+ region=region,
+ **kwargs
+ )
+ }[column_spec_name].data_type.type_code
+
+ data_type = {}
+ if nullable is not None:
+ data_type["nullable"] = nullable
+
+ data_type["type_code"] = type_code
+
+ request = {"name": column_spec_name, "data_type": data_type}
+
+ return self.auto_ml_client.update_column_spec(request, **kwargs)
+
+ def set_target_column(
+ self,
+ dataset=None,
+ dataset_display_name=None,
+ dataset_name=None,
+ table_spec_name=None,
+ table_spec_index=0,
+ column_spec_name=None,
+ column_spec_display_name=None,
+ project=None,
+ region=None,
+ **kwargs
+ ):
+ """Sets the target column for a given table.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> from google.oauth2 import service_account
+ >>>
+ >>> client = automl_v1beta1.TablesClient(
+ ... credentials=service_account.Credentials.from_service_account_file('~/.gcp/account.json')
+ ... project='my-project', region='us-central1')
+ ...
+ >>> client.set_target_column(dataset_display_name='my_dataset',
+ ... column_spec_display_name='Income')
+ ...
+
+ Args:
+ project (Optional[string]):
+ If you have initialized the client with a value for `project`
+ it will be used if this parameter is not supplied. Keep in
+ mind, the service account this client was initialized with must
+ have access to this project.
+ region (Optional[string]):
+ If you have initialized the client with a value for `region` it
+ will be used if this parameter is not supplied.
+ column_spec_name (Optional[string]):
+ The name AutoML-assigned name for the column you want to set as
+ the target column.
+ column_spec_display_name (Optional[string]):
+ The human-readable name of the column you want to set as the
+ target column. If this is supplied in place of
+ `column_spec_name`, you also need to provide either a way to
+ lookup the source dataset (using one of the `dataset*` kwargs),
+ or the `table_spec_name` of the table this column belongs to.
+ table_spec_name (Optional[string]):
+ The AutoML-assigned name for the table whose target column you
+ want to set . If not supplied, the client can determine this
+ name from a source `Dataset` object.
+ table_spec_index (Optional[int]):
+ If no `table_spec_name` or `column_spec_name` was provided, we
+ use this index to determine which table to set the target
+ column on.
+ dataset_display_name (Optional[string]):
+ The human-readable name given to the dataset you want to update
+ the target column of. If no `table_spec_name` is supplied, this
+ will be used together with `table_spec_index` to infer the name
+ of table to update the target column of. This must be supplied
+ if `table_spec_name`, `dataset` or `dataset_name` are not
+ supplied.
+ dataset_name (Optional[string]):
+ The AutoML-assigned name given to the dataset you want to
+ update the target column of. If no `table_spec_name` is
+ supplied, this will be used together with `table_spec_index` to
+ infer the name of table to update the target column of. This
+ must be supplied if `table_spec_name`, `dataset` or
+ `dataset_display_name` are not supplied.
+ dataset (Optional[Dataset]):
+ The `Dataset` instance you want to update the target column of.
+ If no `table_spec_name` is supplied, this will be used together
+ with `table_spec_index` to infer the name of table to update
+ the target column of. This must be supplied if
+ `table_spec_name`, `dataset_name` or `dataset_display_name` are
+ not supplied.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types.Dataset` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If required parameters are missing.
+ """
+ column_spec_name = self.__column_spec_name_from_args(
+ dataset=dataset,
+ dataset_display_name=dataset_display_name,
+ dataset_name=dataset_name,
+ table_spec_name=table_spec_name,
+ table_spec_index=table_spec_index,
+ column_spec_name=column_spec_name,
+ column_spec_display_name=column_spec_display_name,
+ project=project,
+ region=region,
+ **kwargs
+ )
+ column_spec_id = column_spec_name.rsplit("/", 1)[-1]
+
+ dataset = self.__dataset_from_args(
+ dataset=dataset,
+ dataset_name=dataset_name,
+ dataset_display_name=dataset_display_name,
+ project=project,
+ region=region,
+ **kwargs
+ )
+ metadata = dataset.tables_dataset_metadata
+ metadata = self.__update_metadata(
+ metadata, "target_column_spec_id", column_spec_id
+ )
+
+ request = {"name": dataset.name, "tables_dataset_metadata": metadata}
+
+ return self.auto_ml_client.update_dataset(request, **kwargs)
+
+ def set_time_column(
+ self,
+ dataset=None,
+ dataset_display_name=None,
+ dataset_name=None,
+ table_spec_name=None,
+ table_spec_index=0,
+ column_spec_name=None,
+ column_spec_display_name=None,
+ project=None,
+ region=None,
+ **kwargs
+ ):
+ """Sets the time column which designates which data will be of type
+ timestamp and will be used for the timeseries data.
+ This column must be of type timestamp.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> client = automl_v1beta1.TablesClient(
+ ... credentials=service_account.Credentials.from_service_account_file('~/.gcp/account.json')
+ ... project='my-project', region='us-central1')
+ ...
+ >>> client.set_time_column(dataset_display_name='my_dataset',
+ ... column_spec_name='Unix Time')
+ ...
+
+ Args:
+ project (Optional[string]):
+ If you have initialized the client with a value for `project`
+ it will be used if this parameter is not supplied. Keep in
+ mind, the service account this client was initialized with must
+ have access to this project.
+ region (Optional[string]):
+ If you have initialized the client with a value for `region` it
+ will be used if this parameter is not supplied.
+ column_spec_name (Optional[string]):
+ The name AutoML-assigned name for the column you want to set as
+ the time column.
+ column_spec_display_name (Optional[string]):
+ The human-readable name of the column you want to set as the
+ time column. If this is supplied in place of
+ `column_spec_name`, you also need to provide either a way to
+ lookup the source dataset (using one of the `dataset*` kwargs),
+ or the `table_spec_name` of the table this column belongs to.
+ table_spec_name (Optional[string]):
+ The AutoML-assigned name for the table whose time column
+ you want to set . If not supplied, the client can determine
+ this name from a source `Dataset` object.
+ table_spec_index (Optional[int]):
+ If no `table_spec_name` or `column_spec_name` was provided, we
+ use this index to determine which table to set the time
+ column on.
+ dataset_display_name (Optional[string]):
+ The human-readable name given to the dataset you want to update
+ the time column of. If no `table_spec_name` is supplied,
+ this will be used together with `table_spec_index` to infer the
+ name of table to update the time column of. This must be
+ supplied if `table_spec_name`, `dataset` or `dataset_name` are
+ not supplied.
+ dataset_name (Optional[string]):
+ The AutoML-assigned name given to the dataset you want to
+ update the time column of. If no `table_spec_name` is
+ supplied, this will be used together with `table_spec_index` to
+ infer the name of table to update the time column of.
+ This must be supplied if `table_spec_name`, `dataset` or
+ `dataset_display_name` are not supplied.
+ dataset (Optional[Dataset]):
+ The `Dataset` instance you want to update the time column
+ of. If no `table_spec_name` is supplied, this will be used
+ together with `table_spec_index` to infer the name of table to
+ update the time column of. This must be supplied if
+ `table_spec_name`, `dataset_name` or `dataset_display_name` are
+ not supplied.
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types.TableSpec` instance.
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If required parameters are missing.
+ """
+ column_spec_name = self.__column_spec_name_from_args(
+ dataset=dataset,
+ dataset_display_name=dataset_display_name,
+ dataset_name=dataset_name,
+ table_spec_name=table_spec_name,
+ table_spec_index=table_spec_index,
+ column_spec_name=column_spec_name,
+ column_spec_display_name=column_spec_display_name,
+ project=project,
+ region=region,
+ **kwargs
+ )
+ column_spec_id = column_spec_name.rsplit("/", 1)[-1]
+
+ dataset_name = self.__dataset_name_from_args(
+ dataset=dataset,
+ dataset_name=dataset_name,
+ dataset_display_name=dataset_display_name,
+ project=project,
+ region=region,
+ **kwargs
+ )
+
+ table_spec_full_id = self.__table_spec_name_from_args(
+ dataset_name=dataset_name, **kwargs
+ )
+
+ my_table_spec = {
+ "name": table_spec_full_id,
+ "time_column_spec_id": column_spec_id,
+ }
+
+ return self.auto_ml_client.update_table_spec(my_table_spec, **kwargs)
+
+ def clear_time_column(
+ self,
+ dataset=None,
+ dataset_display_name=None,
+ dataset_name=None,
+ project=None,
+ region=None,
+ **kwargs
+ ):
+ """Clears the time column which designates which data will be of type
+ timestamp and will be used for the timeseries data.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> client = automl_v1beta1.TablesClient(
+ ... credentials=service_account.Credentials.from_service_account_file('~/.gcp/account.json')
+ ... project='my-project', region='us-central1')
+ ...
+ >>> client.set_time_column(dataset_display_name='my_dataset')
+ >>>
+
+ Args:
+ project (Optional[string]):
+ If you have initialized the client with a value for `project`
+ it will be used if this parameter is not supplied. Keep in
+ mind, the service account this client was initialized with must
+ have access to this project.
+ region (Optional[string]):
+ If you have initialized the client with a value for `region` it
+ will be used if this parameter is not supplied.
+ dataset_display_name (Optional[string]):
+ The human-readable name given to the dataset you want to update
+ the time column of. If no `table_spec_name` is supplied,
+ this will be used together with `table_spec_index` to infer the
+ name of table to update the time column of. This must be
+ supplied if `table_spec_name`, `dataset` or `dataset_name` are
+ not supplied.
+ dataset_name (Optional[string]):
+ The AutoML-assigned name given to the dataset you want to
+ update the time column of. If no `table_spec_name` is
+ supplied, this will be used together with `table_spec_index` to
+ infer the name of table to update the time column of.
+ This must be supplied if `table_spec_name`, `dataset` or
+ `dataset_display_name` are not supplied.
+ dataset (Optional[Dataset]):
+ The `Dataset` instance you want to update the time column
+ of. If no `table_spec_name` is supplied, this will be used
+ together with `table_spec_index` to infer the name of table to
+ update the time column of. This must be supplied if
+ `table_spec_name`, `dataset_name` or `dataset_display_name` are
+ not supplied.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types.TableSpec` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If required parameters are missing.
+ """
+ dataset_name = self.__dataset_name_from_args(
+ dataset=dataset,
+ dataset_name=dataset_name,
+ dataset_display_name=dataset_display_name,
+ project=project,
+ region=region,
+ **kwargs
+ )
+
+ table_spec_full_id = self.__table_spec_name_from_args(
+ dataset_name=dataset_name, **kwargs
+ )
+
+ my_table_spec = {"name": table_spec_full_id, "time_column_spec_id": None}
+
+ return self.auto_ml_client.update_table_spec(my_table_spec, **kwargs)
+
+ def set_weight_column(
+ self,
+ dataset=None,
+ dataset_display_name=None,
+ dataset_name=None,
+ table_spec_name=None,
+ table_spec_index=0,
+ column_spec_name=None,
+ column_spec_display_name=None,
+ project=None,
+ region=None,
+ **kwargs
+ ):
+ """Sets the weight column for a given table.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> from google.oauth2 import service_account
+ >>>
+ >>> client = automl_v1beta1.TablesClient(
+ ... credentials=service_account.Credentials.from_service_account_file('~/.gcp/account.json')
+ ... project='my-project', region='us-central1')
+ ...
+ >>> client.set_weight_column(dataset_display_name='my_dataset',
+ ... column_spec_display_name='Income')
+ ...
+
+ Args:
+ project (Optional[string]):
+ If you have initialized the client with a value for `project`
+ it will be used if this parameter is not supplied. Keep in
+ mind, the service account this client was initialized with must
+ have access to this project.
+ region (Optional[string]):
+ If you have initialized the client with a value for `region` it
+ will be used if this parameter is not supplied.
+ column_spec_name (Optional[string]):
+ The name AutoML-assigned name for the column you want to
+ set as the weight column.
+ column_spec_display_name (Optional[string]):
+ The human-readable name of the column you want to set as the
+ weight column. If this is supplied in place of
+ `column_spec_name`, you also need to provide either a way to
+ lookup the source dataset (using one of the `dataset*` kwargs),
+ or the `table_spec_name` of the table this column belongs to.
+ table_spec_name (Optional[string]):
+ The AutoML-assigned name for the table whose weight column you
+ want to set . If not supplied, the client can determine this
+ name from a source `Dataset` object.
+ table_spec_index (Optional[int]):
+ If no `table_spec_name` or `column_spec_name` was provided, we
+ use this index to determine which table to set the weight
+ column on.
+ dataset_display_name (Optional[string]):
+ The human-readable name given to the dataset you want to update
+ the weight column of. If no `table_spec_name` is supplied, this
+ will be used together with `table_spec_index` to infer the name
+ of table to update the weight column of. This must be supplied
+ if `table_spec_name`, `dataset` or `dataset_name` are not
+ supplied.
+ dataset_name (Optional[string]):
+ The AutoML-assigned name given to the dataset you want to
+ update the weight column of. If no `table_spec_name` is
+ supplied, this will be used together with `table_spec_index` to
+ infer the name of table to update the weight column of. This
+ must be supplied if `table_spec_name`, `dataset` or
+ `dataset_display_name` are not supplied.
+ dataset (Optional[Dataset]):
+ The `Dataset` instance you want to update the weight column of.
+ If no `table_spec_name` is supplied, this will be used together
+ with `table_spec_index` to infer the name of table to update
+ the weight column of. This must be supplied if
+ `table_spec_name`, `dataset_name` or `dataset_display_name` are
+ not supplied.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types.Dataset` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If required parameters are missing.
+ """
+ column_spec_name = self.__column_spec_name_from_args(
+ dataset=dataset,
+ dataset_display_name=dataset_display_name,
+ dataset_name=dataset_name,
+ table_spec_name=table_spec_name,
+ table_spec_index=table_spec_index,
+ column_spec_name=column_spec_name,
+ column_spec_display_name=column_spec_display_name,
+ project=project,
+ region=region,
+ **kwargs
+ )
+ column_spec_id = column_spec_name.rsplit("/", 1)[-1]
+
+ dataset = self.__dataset_from_args(
+ dataset=dataset,
+ dataset_name=dataset_name,
+ dataset_display_name=dataset_display_name,
+ project=project,
+ region=region,
+ **kwargs
+ )
+ metadata = dataset.tables_dataset_metadata
+ metadata = self.__update_metadata(
+ metadata, "weight_column_spec_id", column_spec_id
+ )
+
+ request = {"name": dataset.name, "tables_dataset_metadata": metadata}
+
+ return self.auto_ml_client.update_dataset(request, **kwargs)
+
+ def clear_weight_column(
+ self,
+ dataset=None,
+ dataset_display_name=None,
+ dataset_name=None,
+ project=None,
+ region=None,
+ **kwargs
+ ):
+ """Clears the weight column for a given dataset.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> from google.oauth2 import service_account
+ >>>
+ >>> client = automl_v1beta1.TablesClient(
+ ... credentials=service_account.Credentials.from_service_account_file('~/.gcp/account.json')
+ ... project='my-project', region='us-central1')
+ ...
+ >>> client.clear_weight_column(dataset_display_name='my_dataset')
+ >>>
+
+ Args:
+ project (Optional[string]):
+ If you have initialized the client with a value for `project`
+ it will be used if this parameter is not supplied. Keep in
+ mind, the service account this client was initialized with must
+ have access to this project.
+ region (Optional[string]):
+ If you have initialized the client with a value for `region` it
+ will be used if this parameter is not supplied.
+ dataset_display_name (Optional[string]):
+ The human-readable name given to the dataset you want to update
+ the weight column of. If no `table_spec_name` is supplied, this
+ will be used together with `table_spec_index` to infer the name
+ of table to update the weight column of. This must be supplied
+ if `table_spec_name`, `dataset` or `dataset_name` are not
+ supplied.
+ dataset_name (Optional[string]):
+ The AutoML-assigned name given to the dataset you want to
+ update the weight column of. If no `table_spec_name` is
+ supplied, this will be used together with `table_spec_index` to
+ infer the name of table to update the weight column of. This
+ must be supplied if `table_spec_name`, `dataset` or
+ `dataset_display_name` are not supplied.
+ dataset (Optional[Dataset]):
+ The `Dataset` instance you want to update the weight column of.
+ If no `table_spec_name` is supplied, this will be used together
+ with `table_spec_index` to infer the name of table to update
+ the weight column of. This must be supplied if
+ `table_spec_name`, `dataset_name` or `dataset_display_name` are
+ not supplied.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types.Dataset` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If required parameters are missing.
+ """
+ dataset = self.__dataset_from_args(
+ dataset=dataset,
+ dataset_name=dataset_name,
+ dataset_display_name=dataset_display_name,
+ project=project,
+ region=region,
+ **kwargs
+ )
+ metadata = dataset.tables_dataset_metadata
+ metadata = self.__update_metadata(metadata, "weight_column_spec_id", None)
+
+ request = {"name": dataset.name, "tables_dataset_metadata": metadata}
+
+ return self.auto_ml_client.update_dataset(request, **kwargs)
+
+ def set_test_train_column(
+ self,
+ dataset=None,
+ dataset_display_name=None,
+ dataset_name=None,
+ table_spec_name=None,
+ table_spec_index=0,
+ column_spec_name=None,
+ column_spec_display_name=None,
+ project=None,
+ region=None,
+ **kwargs
+ ):
+ """Sets the test/train (ml_use) column which designates which data
+ belongs to the test and train sets. This column must be categorical.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> from google.oauth2 import service_account
+ >>>
+ >>> client = automl_v1beta1.TablesClient(
+ ... credentials=service_account.Credentials.from_service_account_file('~/.gcp/account.json')
+ ... project='my-project', region='us-central1')
+ ...
+ >>> client.set_test_train_column(dataset_display_name='my_dataset',
+ ... column_spec_display_name='TestSplit')
+ ...
+
+ Args:
+ project (Optional[string]):
+ If you have initialized the client with a value for `project`
+ it will be used if this parameter is not supplied. Keep in
+ mind, the service account this client was initialized with must
+ have access to this project.
+ region (Optional[string]):
+ If you have initialized the client with a value for `region` it
+ will be used if this parameter is not supplied.
+ column_spec_name (Optional[string]):
+ The name AutoML-assigned name for the column you want to set as
+ the test/train column.
+ column_spec_display_name (Optional[string]):
+ The human-readable name of the column you want to set as the
+ test/train column. If this is supplied in place of
+ `column_spec_name`, you also need to provide either a way to
+ lookup the source dataset (using one of the `dataset*` kwargs),
+ or the `table_spec_name` of the table this column belongs to.
+ table_spec_name (Optional[string]):
+ The AutoML-assigned name for the table whose test/train column
+ you want to set . If not supplied, the client can determine
+ this name from a source `Dataset` object.
+ table_spec_index (Optional[int]):
+ If no `table_spec_name` or `column_spec_name` was provided, we
+ use this index to determine which table to set the test/train
+ column on.
+ dataset_display_name (Optional[string]):
+ The human-readable name given to the dataset you want to update
+ the test/train column of. If no `table_spec_name` is supplied,
+ this will be used together with `table_spec_index` to infer the
+ name of table to update the test/train column of. This must be
+ supplied if `table_spec_name`, `dataset` or `dataset_name` are
+ not supplied.
+ dataset_name (Optional[string]):
+ The AutoML-assigned name given to the dataset you want to
+ update the test/train column of. If no `table_spec_name` is
+ supplied, this will be used together with `table_spec_index` to
+ infer the name of table to update the test/train column of.
+ This must be supplied if `table_spec_name`, `dataset` or
+ `dataset_display_name` are not supplied.
+ dataset (Optional[Dataset]):
+ The `Dataset` instance you want to update the test/train column
+ of. If no `table_spec_name` is supplied, this will be used
+ together with `table_spec_index` to infer the name of table to
+ update the test/train column of. This must be supplied if
+ `table_spec_name`, `dataset_name` or `dataset_display_name` are
+ not supplied.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types.Dataset` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If required parameters are missing.
+ """
+ column_spec_name = self.__column_spec_name_from_args(
+ dataset=dataset,
+ dataset_display_name=dataset_display_name,
+ dataset_name=dataset_name,
+ table_spec_name=table_spec_name,
+ table_spec_index=table_spec_index,
+ column_spec_name=column_spec_name,
+ column_spec_display_name=column_spec_display_name,
+ project=project,
+ region=region,
+ **kwargs
+ )
+ column_spec_id = column_spec_name.rsplit("/", 1)[-1]
+
+ dataset = self.__dataset_from_args(
+ dataset=dataset,
+ dataset_name=dataset_name,
+ dataset_display_name=dataset_display_name,
+ project=project,
+ region=region,
+ **kwargs
+ )
+ metadata = dataset.tables_dataset_metadata
+ metadata = self.__update_metadata(
+ metadata, "ml_use_column_spec_id", column_spec_id
+ )
+
+ request = {"name": dataset.name, "tables_dataset_metadata": metadata}
+
+ return self.auto_ml_client.update_dataset(request, **kwargs)
+
+ def clear_test_train_column(
+ self,
+ dataset=None,
+ dataset_display_name=None,
+ dataset_name=None,
+ project=None,
+ region=None,
+ **kwargs
+ ):
+ """Clears the test/train (ml_use) column which designates which data
+ belongs to the test and train sets.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> from google.oauth2 import service_account
+ >>>
+ >>> client = automl_v1beta1.TablesClient(
+ ... credentials=service_account.Credentials.from_service_account_file('~/.gcp/account.json')
+ ... project='my-project', region='us-central1')
+ ...
+ >>> client.clear_test_train_column(dataset_display_name='my_dataset')
+ >>>
+
+ Args:
+ project (Optional[string]):
+ If you have initialized the client with a value for `project`
+ it will be used if this parameter is not supplied. Keep in
+ mind, the service account this client was initialized with must
+ have access to this project.
+ region (Optional[string]):
+ If you have initialized the client with a value for `region` it
+ will be used if this parameter is not supplied.
+ dataset_display_name (Optional[string]):
+ The human-readable name given to the dataset you want to update
+ the test/train column of. If no `table_spec_name` is supplied,
+ this will be used together with `table_spec_index` to infer the
+ name of table to update the test/train column of. This must be
+ supplied if `table_spec_name`, `dataset` or `dataset_name` are
+ not supplied.
+ dataset_name (Optional[string]):
+ The AutoML-assigned name given to the dataset you want to
+ update the test/train column of. If no `table_spec_name` is
+ supplied, this will be used together with `table_spec_index` to
+ infer the name of table to update the test/train column of.
+ This must be supplied if `table_spec_name`, `dataset` or
+ `dataset_display_name` are not supplied.
+ dataset (Optional[Dataset]):
+ The `Dataset` instance you want to update the test/train column
+ of. If no `table_spec_name` is supplied, this will be used
+ together with `table_spec_index` to infer the name of table to
+ update the test/train column of. This must be supplied if
+ `table_spec_name`, `dataset_name` or `dataset_display_name` are
+ not supplied.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types.Dataset` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If required parameters are missing.
+ """
+ dataset = self.__dataset_from_args(
+ dataset=dataset,
+ dataset_name=dataset_name,
+ dataset_display_name=dataset_display_name,
+ project=project,
+ region=region,
+ **kwargs
+ )
+ metadata = dataset.tables_dataset_metadata
+ metadata = self.__update_metadata(metadata, "ml_use_column_spec_id", None)
+
+ request = {"name": dataset.name, "tables_dataset_metadata": metadata}
+
+ return self.auto_ml_client.update_dataset(request, **kwargs)
+
+ def list_models(self, project=None, region=None, **kwargs):
+ """List all models in a particular project and region.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> from google.oauth2 import service_account
+ >>>
+ >>> client = automl_v1beta1.TablesClient(
+ ... credentials=service_account.Credentials.from_service_account_file('~/.gcp/account.json')
+ ... project='my-project', region='us-central1')
+ ...
+ >>> ms = client.list_models()
+ >>>
+ >>> for m in ms:
+ ... # do something
+ ... pass
+ ...
+
+ Args:
+ project (Optional[string]):
+ If you have initialized the client with a value for `project`
+ it will be used if this parameter is not supplied. Keep in
+ mind, the service account this client was initialized with must
+ have access to this project.
+ region (Optional[string]):
+ If you have initialized the client with a value for `region` it
+ will be used if this parameter is not supplied.
+
+ Returns:
+ A :class:`~google.api_core.page_iterator.PageIterator` instance.
+ An iterable of :class:`~google.cloud.automl_v1beta1.types.Model`
+ instances. You can also iterate over the pages of the response
+ using its `pages` property.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If required parameters are missing.
+ """
+ return self.auto_ml_client.list_models(
+ self.__location_path(project=project, region=region), **kwargs
+ )
+
+ def list_model_evaluations(
+ self,
+ project=None,
+ region=None,
+ model=None,
+ model_display_name=None,
+ model_name=None,
+ **kwargs
+ ):
+ """List all model evaluations for a given model.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> from google.oauth2 import service_account
+ >>>
+ >>> client = automl_v1beta1.TablesClient(
+ ... credentials=service_account.Credentials.from_service_account_file('~/.gcp/account.json')
+ ... project='my-project', region='us-central1')
+ ...
+ >>> ms = client.list_model_evaluations(model_display_name='my_model')
+ >>>
+ >>> for m in ms:
+ ... # do something
+ ... pass
+ ...
+
+ Args:
+ project (Optional[string]):
+ If you have initialized the client with a value for `project`
+ it will be used if this parameter is not supplied. Keep in
+ mind, the service account this client was initialized with must
+ have access to this project.
+ region (Optional[string]):
+ If you have initialized the client with a value for `region` it
+ will be used if this parameter is not supplied.
+ model_display_name (Optional[string]):
+ The human-readable name given to the model you want to list
+ evaluations for. This must be supplied if `model` or
+ `model_name` are not supplied.
+ model_name (Optional[string]):
+ The AutoML-assigned name given to the model you want to list
+ evaluations for. This must be supplied if `model_display_name`
+ or `model` are not supplied.
+ model (Optional[model]):
+ The `model` instance you want to list evaluations for. This
+ must be supplied if `model_display_name` or `model_name` are
+ not supplied.
+
+ Returns:
+ A :class:`~google.api_core.page_iterator.PageIterator` instance.
+ An iterable of
+ :class:`~google.cloud.automl_v1beta1.types.ModelEvaluation`
+ instances. You can also iterate over the pages of the response
+ using its `pages` property.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If required parameters are missing.
+ """
+ model_name = self.__model_name_from_args(
+ model=model,
+ model_name=model_name,
+ model_display_name=model_display_name,
+ project=project,
+ region=region,
+ **kwargs
+ )
+
+ return self.auto_ml_client.list_model_evaluations(model_name, **kwargs)
+
+ def create_model(
+ self,
+ model_display_name,
+ dataset=None,
+ dataset_display_name=None,
+ dataset_name=None,
+ train_budget_milli_node_hours=None,
+ optimization_objective=None,
+ project=None,
+ region=None,
+ model_metadata={},
+ include_column_spec_names=None,
+ exclude_column_spec_names=None,
+ **kwargs
+ ):
+ """Create a model. This will train your model on the given dataset.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> from google.oauth2 import service_account
+ >>>
+ >>> client = automl_v1beta1.TablesClient(
+ ... credentials=service_account.Credentials.from_service_account_file('~/.gcp/account.json')
+ ... project='my-project', region='us-central1')
+ ...
+ >>> m = client.create_model('my_model', dataset_display_name='my_dataset')
+ >>>
+ >>> m.result() # blocks on result
+ >>>
+
+ Args:
+ project (Optional[string]):
+ If you have initialized the client with a value for `project`
+ it will be used if this parameter is not supplied. Keep in
+ mind, the service account this client was initialized with must
+ have access to this project.
+ region (Optional[string]):
+ If you have initialized the client with a value for `region` it
+ will be used if this parameter is not supplied.
+ model_display_name (string):
+ A human-readable name to refer to this model by.
+ train_budget_milli_node_hours (int):
+ The amount of time (in thousandths of an hour) to spend
+ training. This value must be between 1,000 and 72,000 inclusive
+ (between 1 and 72 hours).
+ optimization_objective (string):
+ The metric AutoML tables should optimize for.
+ dataset_display_name (Optional[string]):
+ The human-readable name given to the dataset you want to train
+ your model on. This must be supplied if `dataset` or
+ `dataset_name` are not supplied.
+ dataset_name (Optional[string]):
+ The AutoML-assigned name given to the dataset you want to train
+ your model on. This must be supplied if `dataset_display_name`
+ or `dataset` are not supplied.
+ dataset (Optional[Dataset]):
+ The `Dataset` instance you want to train your model on. This
+ must be supplied if `dataset_display_name` or `dataset_name`
+ are not supplied.
+ model_metadata (Optional[Dict]):
+ Optional model metadata to supply to the client.
+ include_column_spec_names(Optional[string]):
+ The list of the names of the columns you want to include to train
+ your model on.
+ exclude_column_spec_names(Optional[string]):
+ The list of the names of the columns you want to exclude and
+ not train your model on.
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types._OperationFuture`
+ instance.
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If required parameters are missing.
+ """
+ if (
+ train_budget_milli_node_hours is None
+ or train_budget_milli_node_hours < 1000
+ or train_budget_milli_node_hours > 72000
+ ):
+ raise ValueError(
+ "'train_budget_milli_node_hours' must be a "
+ "value between 1,000 and 72,000 inclusive"
+ )
+
+ if exclude_column_spec_names not in [
+ None,
+ [],
+ ] and include_column_spec_names not in [None, []]:
+ raise ValueError(
+ "Cannot set both "
+ "'exclude_column_spec_names' and "
+ "'include_column_spec_names'"
+ )
+
+ dataset_name = self.__dataset_name_from_args(
+ dataset=dataset,
+ dataset_name=dataset_name,
+ dataset_display_name=dataset_display_name,
+ project=project,
+ region=region,
+ **kwargs
+ )
+
+ model_metadata["train_budget_milli_node_hours"] = train_budget_milli_node_hours
+ if optimization_objective is not None:
+ model_metadata["optimization_objective"] = optimization_objective
+
+ dataset_id = dataset_name.rsplit("/", 1)[-1]
+ columns = [
+ s
+ for s in self.list_column_specs(
+ dataset=dataset,
+ dataset_name=dataset_name,
+ dataset_display_name=dataset_display_name,
+ **kwargs
+ )
+ ]
+
+ final_columns = []
+ if include_column_spec_names:
+ for c in columns:
+ if c.display_name in include_column_spec_names:
+ final_columns.append(c)
+
+ model_metadata["input_feature_column_specs"] = final_columns
+ elif exclude_column_spec_names:
+ for a in columns:
+ if a.display_name not in exclude_column_spec_names:
+ final_columns.append(a)
+
+ model_metadata["input_feature_column_specs"] = final_columns
+
+ request = {
+ "display_name": model_display_name,
+ "dataset_id": dataset_id,
+ "tables_model_metadata": model_metadata,
+ }
+
+ op = self.auto_ml_client.create_model(
+ self.__location_path(project=project, region=region), request, **kwargs
+ )
+ self.__log_operation_info("Model creation", op)
+ return op
+
+ def delete_model(
+ self,
+ model=None,
+ model_display_name=None,
+ model_name=None,
+ project=None,
+ region=None,
+ **kwargs
+ ):
+ """Deletes a model. Note this will not delete any datasets associated
+ with this model.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> from google.oauth2 import service_account
+ >>>
+ >>> client = automl_v1beta1.TablesClient(
+ ... credentials=service_account.Credentials.from_service_account_file('~/.gcp/account.json')
+ ... project='my-project', region='us-central1')
+ ...
+ >>> op = client.delete_model(model_display_name='my_model')
+ >>>
+ >>> op.result() # blocks on delete request
+ >>>
+
+ Args:
+ project (Optional[string]):
+ If you have initialized the client with a value for `project`
+ it will be used if this parameter is not supplied. Keep in
+ mind, the service account this client was initialized with must
+ have access to this project.
+ region (Optional[string]):
+ If you have initialized the client with a value for `region` it
+ will be used if this parameter is not supplied.
+ model_display_name (Optional[string]):
+ The human-readable name given to the model you want to
+ delete. This must be supplied if `model` or `model_name`
+ are not supplied.
+ model_name (Optional[string]):
+ The AutoML-assigned name given to the model you want to
+ delete. This must be supplied if `model_display_name` or
+ `model` are not supplied.
+ model (Optional[model]):
+ The `model` instance you want to delete. This must be
+ supplied if `model_display_name` or `model_name` are not
+ supplied.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types._OperationFuture`
+ instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If required parameters are missing.
+ """
+ try:
+ model_name = self.__model_name_from_args(
+ model=model,
+ model_name=model_name,
+ model_display_name=model_display_name,
+ project=project,
+ region=region,
+ **kwargs
+ )
+ # delete is idempotent
+ except exceptions.NotFound:
+ return None
+
+ op = self.auto_ml_client.delete_model(model_name, **kwargs)
+ self.__log_operation_info("Delete model", op)
+ return op
+
+ def get_model_evaluation(
+ self, model_evaluation_name, project=None, region=None, **kwargs
+ ):
+ """Gets a single evaluation model in a particular project and region.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> from google.oauth2 import service_account
+ >>>
+ >>> client = automl_v1beta1.TablesClient(
+ ... credentials=service_account.Credentials.from_service_account_file('~/.gcp/account.json')
+ ... project='my-project', region='us-central1')
+ ...
+ >>> d = client.get_model_evaluation('my_model_evaluation')
+ >>>
+
+ Args:
+ model_evaluation_name (string):
+ This is the fully-qualified name generated by the AutoML API
+ for this model evaluation.
+ project (Optional[string]):
+ If you have initialized the client with a value for `project`
+ it will be used if this parameter is not supplied. Keep in
+ mind, the service account this client was initialized with must
+ have access to this project.
+ region (Optional[string]):
+ If you have initialized the client with a value for `region` it
+ will be used if this parameter is not supplied.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types.ModelEvaluation` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If required parameters are missing.
+ """
+ return self.auto_ml_client.get_model_evaluation(model_evaluation_name, **kwargs)
+
+ def get_model(
+ self,
+ project=None,
+ region=None,
+ model_name=None,
+ model_display_name=None,
+ **kwargs
+ ):
+ """Gets a single model in a particular project and region.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> from google.oauth2 import service_account
+ >>>
+ >>> client = automl_v1beta1.TablesClient(
+ ... credentials=service_account.Credentials.from_service_account_file('~/.gcp/account.json')
+ ... project='my-project', region='us-central1')
+ ...
+ >>> d = client.get_model(model_display_name='my_model')
+ >>>
+
+ Args:
+ project (Optional[string]):
+ If you have initialized the client with a value for `project`
+ it will be used if this parameter is not supplied. Keep in
+ mind, the service account this client was initialized with must
+ have access to this project.
+ region (Optional[string]):
+ If you have initialized the client with a value for `region` it
+ will be used if this parameter is not supplied.
+ model_name (Optional[string]):
+ This is the fully-qualified name generated by the AutoML API
+ for this model. This is not to be confused with the
+ human-assigned `model_display_name` that is provided when
+ creating a model. Either `model_name` or
+ `model_display_name` must be provided.
+ model_display_name (Optional[string]):
+ This is the name you provided for the model when first
+ creating it. Either `model_name` or `model_display_name`
+ must be provided.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types.Model` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If required parameters are missing.
+ """
+ if model_name is None and model_display_name is None:
+ raise ValueError(
+ "One of 'model_name' or " "'model_display_name' must be set."
+ )
+
+ if model_name is not None:
+ return self.auto_ml_client.get_model(model_name, **kwargs)
+
+ return self.__lookup_by_display_name(
+ "model", self.list_models(project, region, **kwargs), model_display_name
+ )
+
+ # TODO(jonathanskim): allow deployment from just model ID
+ def deploy_model(
+ self,
+ model=None,
+ model_name=None,
+ model_display_name=None,
+ project=None,
+ region=None,
+ **kwargs
+ ):
+ """Deploys a model. This allows you make online predictions using the
+ model you've deployed.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> from google.oauth2 import service_account
+ >>>
+ >>> client = automl_v1beta1.TablesClient(
+ ... credentials=service_account.Credentials.from_service_account_file('~/.gcp/account.json')
+ ... project='my-project', region='us-central1')
+ ...
+ >>> op = client.deploy_model(model_display_name='my_model')
+ >>>
+ >>> op.result() # blocks on deploy request
+ >>>
+
+ Args:
+ project (Optional[string]):
+ If you have initialized the client with a value for `project`
+ it will be used if this parameter is not supplied. Keep in
+ mind, the service account this client was initialized with must
+ have access to this project.
+ region (Optional[string]):
+ If you have initialized the client with a value for `region` it
+ will be used if this parameter is not supplied.
+ model_display_name (Optional[string]):
+ The human-readable name given to the model you want to
+ deploy. This must be supplied if `model` or `model_name`
+ are not supplied.
+ model_name (Optional[string]):
+ The AutoML-assigned name given to the model you want to
+ deploy. This must be supplied if `model_display_name` or
+ `model` are not supplied.
+ model (Optional[model]):
+ The `model` instance you want to deploy. This must be
+ supplied if `model_display_name` or `model_name` are not
+ supplied.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types._OperationFuture`
+ instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If required parameters are missing.
+ """
+ model_name = self.__model_name_from_args(
+ model=model,
+ model_name=model_name,
+ model_display_name=model_display_name,
+ project=project,
+ region=region,
+ **kwargs
+ )
+
+ op = self.auto_ml_client.deploy_model(model_name, **kwargs)
+ self.__log_operation_info("Deploy model", op)
+ return op
+
+ def undeploy_model(
+ self,
+ model=None,
+ model_name=None,
+ model_display_name=None,
+ project=None,
+ region=None,
+ **kwargs
+ ):
+ """Undeploys a model.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> from google.oauth2 import service_account
+ >>>
+ >>> client = automl_v1beta1.TablesClient(
+ ... credentials=service_account.Credentials.from_service_account_file('~/.gcp/account.json')
+ ... project='my-project', region='us-central1')
+ ...
+ >>> op = client.undeploy_model(model_display_name='my_model')
+ >>>
+ >>> op.result() # blocks on undeploy request
+ >>>
+
+ Args:
+ project (Optional[string]):
+ If you have initialized the client with a value for `project`
+ it will be used if this parameter is not supplied. Keep in
+ mind, the service account this client was initialized with must
+ have access to this project.
+ region (Optional[string]):
+ If you have initialized the client with a value for `region` it
+ will be used if this parameter is not supplied.
+ model_display_name (Optional[string]):
+ The human-readable name given to the model you want to
+ undeploy. This must be supplied if `model` or `model_name`
+ are not supplied.
+ model_name (Optional[string]):
+ The AutoML-assigned name given to the model you want to
+ undeploy. This must be supplied if `model_display_name` or
+ `model` are not supplied.
+ model (Optional[model]):
+ The `model` instance you want to undeploy. This must be
+ supplied if `model_display_name` or `model_name` are not
+ supplied.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types._OperationFuture`
+ instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If required parameters are missing.
+ """
+ model_name = self.__model_name_from_args(
+ model=model,
+ model_name=model_name,
+ model_display_name=model_display_name,
+ project=project,
+ region=region,
+ **kwargs
+ )
+
+ op = self.auto_ml_client.undeploy_model(model_name, **kwargs)
+ self.__log_operation_info("Undeploy model", op)
+ return op
+
+ ## TODO(lwander): support pandas DataFrame as input type
+ def predict(
+ self,
+ inputs,
+ model=None,
+ model_name=None,
+ model_display_name=None,
+ project=None,
+ region=None,
+ **kwargs
+ ):
+ """Makes a prediction on a deployed model. This will fail if the model
+ was not deployed.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> from google.oauth2 import service_account
+ >>>
+ >>> client = automl_v1beta1.TablesClient(
+ ... credentials=service_account.Credentials.from_service_account_file('~/.gcp/account.json')
+ ... project='my-project', region='us-central1')
+ ...
+ >>> client.predict(inputs={'Age': 30, 'Income': 12, 'Category': 'A'}
+ ... model_display_name='my_model')
+ ...
+ >>> client.predict([30, 12, 'A'], model_display_name='my_model')
+ >>>
+
+ Args:
+ project (Optional[string]):
+ If you have initialized the client with a value for `project`
+ it will be used if this parameter is not supplied. Keep in
+ mind, the service account this client was initialized with must
+ have access to this project.
+ region (Optional[string]):
+ If you have initialized the client with a value for `region` it
+ will be used if this parameter is not supplied.
+ inputs (Union[List[string], Dict[string, string]]):
+ Either the sorted list of column values to predict with, or a
+ key-value map of column display name to value to predict with.
+ model_display_name (Optional[string]):
+ The human-readable name given to the model you want to predict
+ with. This must be supplied if `model` or `model_name` are not
+ supplied.
+ model_name (Optional[string]):
+ The AutoML-assigned name given to the model you want to predict
+ with. This must be supplied if `model_display_name` or `model`
+ are not supplied.
+ model (Optional[model]):
+ The `model` instance you want to predict with . This must be
+ supplied if `model_display_name` or `model_name` are not
+ supplied.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types.PredictResponse`
+ instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If required parameters are missing.
+ """
+ model = self.__model_from_args(
+ model=model,
+ model_name=model_name,
+ model_display_name=model_display_name,
+ project=project,
+ region=region,
+ **kwargs
+ )
+
+ column_specs = model.tables_model_metadata.input_feature_column_specs
+ if type(inputs) == dict:
+ inputs = [inputs.get(c.display_name, None) for c in column_specs]
+
+ if len(inputs) != len(column_specs):
+ raise ValueError(
+ (
+ "Dimension mismatch, the number of provided "
+ "inputs ({}) does not match that of the model "
+ "({})"
+ ).format(len(inputs), len(column_specs))
+ )
+
+ values = []
+ for i, c in zip(inputs, column_specs):
+ value_type = self.__type_code_to_value_type(c.data_type.type_code, i)
+ values.append(value_type)
+
+ request = {"row": {"values": values}}
+
+ return self.prediction_client.predict(model.name, request, **kwargs)
+
+ def batch_predict(
+ self,
+ pandas_dataframe=None,
+ bigquery_input_uri=None,
+ bigquery_output_uri=None,
+ gcs_input_uris=None,
+ gcs_output_uri_prefix=None,
+ model=None,
+ model_name=None,
+ model_display_name=None,
+ project=None,
+ region=None,
+ credentials=None,
+ inputs=None,
+ **kwargs
+ ):
+ """Makes a batch prediction on a model. This does _not_ require the
+ model to be deployed.
+
+ Example:
+ >>> from google.cloud import automl_v1beta1
+ >>>
+ >>> from google.oauth2 import service_account
+ >>>
+ >>> client = automl_v1beta1.TablesClient(
+ ... credentials=service_account.Credentials.from_service_account_file('~/.gcp/account.json')
+ ... project='my-project', region='us-central1')
+ ...
+ >>> client.batch_predict(
+ ... gcs_input_uris='gs://inputs/input.csv',
+ ... gcs_output_uri_prefix='gs://outputs/',
+ ... model_display_name='my_model'
+ ... ).result()
+ ...
+
+ Args:
+ project (Optional[string]):
+ If you have initialized the client with a value for `project`
+ it will be used if this parameter is not supplied. Keep in
+ mind, the service account this client was initialized with must
+ have access to this project.
+ region (Optional[string]):
+ If you have initialized the client with a value for `region` it
+ will be used if this parameter is not supplied.
+ credentials (Optional[google.auth.credentials.Credentials]): The
+ authorization credentials to attach to requests. These
+ credentials identify this application to the service. If none
+ are specified, the client will attempt to ascertain the
+ credentials from the environment.
+ pandas_dataframe (Optional[pandas.DataFrame]):
+ A Pandas Dataframe object containing the data you want to predict
+ off of. The data will be converted to CSV, and this CSV will be
+ staged to GCS in `gs://{project}-automl-tables-staging/{uploaded_csv_name}`
+ This must be supplied if neither `gcs_input_uris` nor
+ `bigquery_input_uri` is supplied.
+ gcs_input_uris (Optional(Union[List[string], string]))
+ Either a list of or a single GCS URI containing the data you
+ want to predict off of. This must be supplied if neither
+ `pandas_dataframe` nor `bigquery_input_uri` is supplied.
+ gcs_output_uri_prefix (Optional[string])
+ The folder in GCS you want to write output to. This must be
+ supplied if `bigquery_output_uri` is not.
+ bigquery_input_uri (Optional[string])
+ The BigQuery table to input data from. This must be supplied if
+ neither `pandas_dataframe` nor `gcs_input_uris` is supplied.
+ bigquery_output_uri (Optional[string])
+ The BigQuery table to output data to. This must be supplied if
+ `gcs_output_uri_prefix` is not.
+ model_display_name (Optional[string]):
+ The human-readable name given to the model you want to predict
+ with. This must be supplied if `model` or `model_name` are not
+ supplied.
+ model_name (Optional[string]):
+ The AutoML-assigned name given to the model you want to predict
+ with. This must be supplied if `model_display_name` or `model`
+ are not supplied.
+ model (Optional[model]):
+ The `model` instance you want to predict with . This must be
+ supplied if `model_display_name` or `model_name` are not
+ supplied.
+
+ Returns:
+ A :class:`~google.cloud.automl_v1beta1.types._OperationFuture`
+ instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If required parameters are missing.
+ """
+ model_name = self.__model_name_from_args(
+ model=model,
+ model_name=model_name,
+ model_display_name=model_display_name,
+ project=project,
+ region=region,
+ **kwargs
+ )
+
+ input_request = None
+
+ if pandas_dataframe is not None:
+ self.__ensure_gcs_client_is_initialized(credentials)
+ self.gcs_client.ensure_bucket_exists(project, region)
+ gcs_input_uri = self.gcs_client.upload_pandas_dataframe(pandas_dataframe)
+ input_request = {"gcs_source": {"input_uris": [gcs_input_uri]}}
+ elif gcs_input_uris is not None:
+ if type(gcs_input_uris) != list:
+ gcs_input_uris = [gcs_input_uris]
+ input_request = {"gcs_source": {"input_uris": gcs_input_uris}}
+ elif bigquery_input_uri is not None:
+ input_request = {"bigquery_source": {"input_uri": bigquery_input_uri}}
+ else:
+ raise ValueError(
+ "One of 'gcs_input_uris'/'bigquery_input_uris' must" "be set"
+ )
+
+ output_request = None
+ if gcs_output_uri_prefix is not None:
+ output_request = {
+ "gcs_destination": {"output_uri_prefix": gcs_output_uri_prefix}
+ }
+ elif bigquery_output_uri is not None:
+ output_request = {
+ "bigquery_destination": {"output_uri": bigquery_output_uri}
+ }
+ else:
+ raise ValueError(
+ "One of 'gcs_output_uri_prefix'/'bigquery_output_uri' must be set"
+ )
+
+ op = self.prediction_client.batch_predict(
+ model_name, input_request, output_request, **kwargs
+ )
+ self.__log_operation_info("Batch predict", op)
+ return op
diff --git a/automl/google/cloud/automl_v1beta1/types.py b/automl/google/cloud/automl_v1beta1/types.py
new file mode 100644
index 000000000000..8025892487d6
--- /dev/null
+++ b/automl/google/cloud/automl_v1beta1/types.py
@@ -0,0 +1,114 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from __future__ import absolute_import
+import sys
+
+from google.api_core.protobuf_helpers import get_messages
+
+from google.cloud.automl_v1beta1.proto import annotation_payload_pb2
+from google.cloud.automl_v1beta1.proto import annotation_spec_pb2
+from google.cloud.automl_v1beta1.proto import classification_pb2
+from google.cloud.automl_v1beta1.proto import column_spec_pb2
+from google.cloud.automl_v1beta1.proto import data_items_pb2
+from google.cloud.automl_v1beta1.proto import data_stats_pb2
+from google.cloud.automl_v1beta1.proto import data_types_pb2
+from google.cloud.automl_v1beta1.proto import dataset_pb2
+from google.cloud.automl_v1beta1.proto import detection_pb2
+from google.cloud.automl_v1beta1.proto import geometry_pb2
+from google.cloud.automl_v1beta1.proto import image_pb2
+from google.cloud.automl_v1beta1.proto import io_pb2
+from google.cloud.automl_v1beta1.proto import model_evaluation_pb2
+from google.cloud.automl_v1beta1.proto import model_pb2
+from google.cloud.automl_v1beta1.proto import operations_pb2 as proto_operations_pb2
+from google.cloud.automl_v1beta1.proto import prediction_service_pb2
+from google.cloud.automl_v1beta1.proto import ranges_pb2
+from google.cloud.automl_v1beta1.proto import regression_pb2
+from google.cloud.automl_v1beta1.proto import service_pb2
+from google.cloud.automl_v1beta1.proto import table_spec_pb2
+from google.cloud.automl_v1beta1.proto import tables_pb2
+from google.cloud.automl_v1beta1.proto import temporal_pb2
+from google.cloud.automl_v1beta1.proto import text_extraction_pb2
+from google.cloud.automl_v1beta1.proto import text_pb2
+from google.cloud.automl_v1beta1.proto import text_segment_pb2
+from google.cloud.automl_v1beta1.proto import text_sentiment_pb2
+from google.cloud.automl_v1beta1.proto import translation_pb2
+from google.cloud.automl_v1beta1.proto import video_pb2
+from google.longrunning import operations_pb2 as longrunning_operations_pb2
+from google.protobuf import any_pb2
+from google.protobuf import duration_pb2
+from google.protobuf import field_mask_pb2
+from google.protobuf import struct_pb2
+from google.protobuf import timestamp_pb2
+from google.rpc import status_pb2
+
+
+_shared_modules = [
+ longrunning_operations_pb2,
+ any_pb2,
+ duration_pb2,
+ field_mask_pb2,
+ struct_pb2,
+ timestamp_pb2,
+ status_pb2,
+]
+
+_local_modules = [
+ annotation_payload_pb2,
+ annotation_spec_pb2,
+ classification_pb2,
+ column_spec_pb2,
+ data_items_pb2,
+ data_stats_pb2,
+ data_types_pb2,
+ dataset_pb2,
+ detection_pb2,
+ geometry_pb2,
+ image_pb2,
+ io_pb2,
+ model_evaluation_pb2,
+ model_pb2,
+ proto_operations_pb2,
+ prediction_service_pb2,
+ ranges_pb2,
+ regression_pb2,
+ service_pb2,
+ table_spec_pb2,
+ tables_pb2,
+ temporal_pb2,
+ text_extraction_pb2,
+ text_pb2,
+ text_segment_pb2,
+ text_sentiment_pb2,
+ translation_pb2,
+ video_pb2,
+]
+
+names = []
+
+for module in _shared_modules: # pragma: NO COVER
+ for name, message in get_messages(module).items():
+ setattr(sys.modules[__name__], name, message)
+ names.append(name)
+for module in _local_modules:
+ for name, message in get_messages(module).items():
+ message.__module__ = "google.cloud.automl_v1beta1.types"
+ setattr(sys.modules[__name__], name, message)
+ names.append(name)
+
+
+__all__ = tuple(sorted(names))
diff --git a/automl/noxfile.py b/automl/noxfile.py
new file mode 100644
index 000000000000..19fb3148ffd8
--- /dev/null
+++ b/automl/noxfile.py
@@ -0,0 +1,162 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Generated by synthtool. DO NOT EDIT!
+
+from __future__ import absolute_import
+import os
+import shutil
+
+import nox
+
+
+LOCAL_DEPS = (os.path.join("..", "api_core"), os.path.join("..", "core"))
+BLACK_VERSION = "black==19.3b0"
+BLACK_PATHS = ["docs", "google", "tests", "noxfile.py", "setup.py"]
+
+if os.path.exists("samples"):
+ BLACK_PATHS.append("samples")
+
+
+@nox.session(python="3.7")
+def lint(session):
+ """Run linters.
+
+ Returns a failure if the linters find linting errors or sufficiently
+ serious code quality issues.
+ """
+ session.install("flake8", BLACK_VERSION, *LOCAL_DEPS)
+ session.run("black", "--check", *BLACK_PATHS)
+ session.run("flake8", "google", "tests")
+
+
+@nox.session(python="3.6")
+def blacken(session):
+ """Run black.
+
+ Format code to uniform standard.
+
+ This currently uses Python 3.6 due to the automated Kokoro run of synthtool.
+ That run uses an image that doesn't have 3.6 installed. Before updating this
+ check the state of the `gcp_ubuntu_config` we use for that Kokoro run.
+ """
+ session.install(BLACK_VERSION)
+ session.run("black", *BLACK_PATHS)
+
+
+@nox.session(python="3.7")
+def lint_setup_py(session):
+ """Verify that setup.py is valid (including RST check)."""
+ session.install("docutils", "pygments")
+ session.run("python", "setup.py", "check", "--restructuredtext", "--strict")
+
+
+def default(session):
+ # Install all test dependencies, then install this package in-place.
+ session.install("mock", "pytest", "pytest-cov")
+ for local_dep in LOCAL_DEPS:
+ session.install("-e", local_dep)
+ session.install("-e", ".")
+ session.install("-e", ".[pandas,storage]")
+
+ # Run py.test against the unit tests.
+ session.run(
+ "py.test",
+ "--quiet",
+ "--cov=google.cloud",
+ "--cov=tests.unit",
+ "--cov-append",
+ "--cov-config=.coveragerc",
+ "--cov-report=",
+ "--cov-fail-under=0",
+ os.path.join("tests", "unit"),
+ *session.posargs,
+ )
+
+
+@nox.session(python=["2.7", "3.5", "3.6", "3.7"])
+def unit(session):
+ """Run the unit test suite."""
+ default(session)
+
+
+@nox.session(python=["2.7", "3.7"])
+def system(session):
+ """Run the system test suite."""
+ system_test_path = os.path.join("tests", "system.py")
+ system_test_folder_path = os.path.join("tests", "system")
+ # Sanity check: Only run tests if the environment variable is set.
+ if not os.environ.get("GOOGLE_APPLICATION_CREDENTIALS", ""):
+ session.skip("Credentials must be set via environment variable")
+
+ system_test_exists = os.path.exists(system_test_path)
+ system_test_folder_exists = os.path.exists(system_test_folder_path)
+ # Sanity check: only run tests if found.
+ if not system_test_exists and not system_test_folder_exists:
+ session.skip("System tests were not found")
+
+ # Use pre-release gRPC for system tests.
+ session.install("--pre", "grpcio")
+
+ # Install all test dependencies, then install this package into the
+ # virtualenv's dist-packages.
+ session.install("mock", "pytest")
+ for local_dep in LOCAL_DEPS:
+ session.install("-e", local_dep)
+ session.install("-e", "../test_utils/")
+ session.install("-e", ".")
+ session.install("-e", ".[pandas,storage]")
+
+ # Run py.test against the system tests.
+ if system_test_exists:
+ session.run("py.test", "--quiet", system_test_path, *session.posargs)
+ if system_test_folder_exists:
+ session.run("py.test", "--quiet", system_test_folder_path, *session.posargs)
+
+
+@nox.session(python="3.7")
+def cover(session):
+ """Run the final coverage report.
+
+ This outputs the coverage report aggregating coverage from the unit
+ test runs (not system test runs), and then erases coverage data.
+ """
+ session.install("coverage", "pytest-cov")
+ session.run("coverage", "report", "--show-missing", "--fail-under=83")
+
+ session.run("coverage", "erase")
+
+
+@nox.session(python="3.7")
+def docs(session):
+ """Build the docs for this library."""
+
+ session.install("-e", ".")
+ session.install("sphinx", "alabaster", "recommonmark")
+
+ shutil.rmtree(os.path.join("docs", "_build"), ignore_errors=True)
+ session.run(
+ "sphinx-build",
+ "-W", # warnings as errors
+ "-T", # show full traceback on exception
+ "-N", # no colors
+ "-b",
+ "html",
+ "-d",
+ os.path.join("docs", "_build", "doctrees", ""),
+ os.path.join("docs", ""),
+ os.path.join("docs", "_build", "html", ""),
+ )
diff --git a/automl/setup.cfg b/automl/setup.cfg
new file mode 100644
index 000000000000..3bd555500e37
--- /dev/null
+++ b/automl/setup.cfg
@@ -0,0 +1,3 @@
+# Generated by synthtool. DO NOT EDIT!
+[bdist_wheel]
+universal = 1
diff --git a/automl/setup.py b/automl/setup.py
new file mode 100644
index 000000000000..c76f945594e8
--- /dev/null
+++ b/automl/setup.py
@@ -0,0 +1,78 @@
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import io
+import os
+
+import setuptools
+
+name = "google-cloud-automl"
+description = "Cloud AutoML API client library"
+version = "0.5.0"
+release_status = "Development Status :: 3 - Alpha"
+dependencies = [
+ "google-api-core[grpc] >= 1.14.0, < 2.0.0dev",
+ 'enum34; python_version < "3.4"',
+]
+extras = {
+ "pandas": ["pandas>=0.24.0"],
+ "storage": ["google-cloud-storage >= 1.18.0, < 2.0.0dev"],
+}
+
+package_root = os.path.abspath(os.path.dirname(__file__))
+
+readme_filename = os.path.join(package_root, "README.rst")
+with io.open(readme_filename, encoding="utf-8") as readme_file:
+ readme = readme_file.read()
+
+packages = [
+ package for package in setuptools.find_packages() if package.startswith("google")
+]
+
+namespaces = ["google"]
+if "google.cloud" in packages:
+ namespaces.append("google.cloud")
+
+setuptools.setup(
+ name=name,
+ version=version,
+ description=description,
+ long_description=readme,
+ author="Google LLC",
+ author_email="googleapis-packages@oogle.com",
+ license="Apache 2.0",
+ url="https://github.com/GoogleCloudPlatform/google-cloud-python",
+ classifiers=[
+ release_status,
+ "Intended Audience :: Developers",
+ "License :: OSI Approved :: Apache Software License",
+ "Programming Language :: Python",
+ "Programming Language :: Python :: 2",
+ "Programming Language :: Python :: 2.7",
+ "Programming Language :: Python :: 3",
+ "Programming Language :: Python :: 3.5",
+ "Programming Language :: Python :: 3.6",
+ "Programming Language :: Python :: 3.7",
+ "Operating System :: OS Independent",
+ "Topic :: Internet",
+ ],
+ platforms="Posix; MacOS X; Windows",
+ packages=packages,
+ namespace_packages=namespaces,
+ install_requires=dependencies,
+ extras_require=extras,
+ python_requires=">=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*",
+ include_package_data=True,
+ zip_safe=False,
+)
diff --git a/automl/synth.metadata b/automl/synth.metadata
new file mode 100644
index 000000000000..7cf7214eecc8
--- /dev/null
+++ b/automl/synth.metadata
@@ -0,0 +1,39 @@
+{
+ "updateTime": "2019-08-26T23:45:30.964132Z",
+ "sources": [
+ {
+ "generator": {
+ "name": "artman",
+ "version": "0.35.0",
+ "dockerImage": "googleapis/artman@sha256:97ef134b6b1cc2c21868960d3b4352524023fb25b61fc137ca0783ce3c08c2cd"
+ }
+ },
+ {
+ "git": {
+ "name": "googleapis",
+ "remote": "https://github.com/googleapis/googleapis.git",
+ "sha": "650caad718bb063f189405c23972dc9818886358",
+ "internalRef": "265565344"
+ }
+ },
+ {
+ "template": {
+ "name": "python_library",
+ "origin": "synthtool.gcp",
+ "version": "2019.5.2"
+ }
+ }
+ ],
+ "destinations": [
+ {
+ "client": {
+ "source": "googleapis",
+ "apiName": "automl",
+ "apiVersion": "v1beta1",
+ "language": "python",
+ "generator": "gapic",
+ "config": "google/cloud/automl/artman_automl_v1beta1.yaml"
+ }
+ }
+ ]
+}
\ No newline at end of file
diff --git a/automl/synth.py b/automl/synth.py
new file mode 100644
index 000000000000..06407e9fa019
--- /dev/null
+++ b/automl/synth.py
@@ -0,0 +1,115 @@
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""This script is used to synthesize generated parts of this library."""
+
+import re
+
+import synthtool as s
+from synthtool import gcp
+
+gapic = gcp.GAPICGenerator()
+common = gcp.CommonTemplates()
+versions = ["v1beta1"]
+
+
+# ----------------------------------------------------------------------------
+# Generate automl GAPIC layer
+# ----------------------------------------------------------------------------
+for version in versions:
+ library = gapic.py_library("automl", version, include_protos=True)
+ s.move(library / f"google/cloud/automl_{version}")
+ s.move(library / f"tests/unit/gapic/{version}")
+ s.move(library / f"docs/gapic/{version}")
+
+s.move(library / f"docs/conf.py")
+
+# Use the highest version library to generate import alias.
+s.move(library / "google/cloud/automl.py")
+
+# Add tables client to v1beta1
+s.replace(
+ f"google/cloud/automl_v1beta1/__init__.py",
+ f"from google.cloud.automl_v1beta1.gapic import prediction_service_client",
+ f"from google.cloud.automl_v1beta1.gapic import prediction_service_client\n"
+ f"from google.cloud.automl_v1beta1.tables import tables_client"
+ f"\n\n"
+ f"class TablesClient(tables_client.TablesClient):"
+ f" __doc__ = tables_client.TablesClient.__doc__",
+)
+
+s.replace(
+ f"google/cloud/automl_v1beta1/__init__.py",
+ f"""__all__ = \(
+ 'enums',
+ 'types',
+ 'AutoMlClient',
+ 'PredictionServiceClient',
+\)""",
+ f'__all__ = ("enums", "types", "AutoMlClient", "PredictionServiceClient", "TablesClient")',
+)
+
+# Fixup issues in generated code
+s.replace(
+ "**/gapic/*_client.py",
+ r"metadata_type=operations_pb2.OperationMetadata",
+ r"metadata_type=proto_operations_pb2.OperationMetadata",
+)
+
+# Fix spacing/'::' issues in docstrings
+s.replace(
+ "google/cloud/automl_v1beta1/gapic/prediction_service_client.py", "^\s+::", ""
+)
+
+s.replace(
+ "google/cloud/automl_v1beta1/gapic/auto_ml_client.py",
+ "^(\s+)(::)\n\n\s+?([^\s])",
+ " \g<1>\g<2>\n \g<1>\g<3>",
+)
+
+# Remove 'raw-latex' sections with sample JSON Lines files
+s.replace(
+ "google/cloud/**/io_pb2.py",
+ r"""Sample in-line
+ JSON Lines file.*?\}`\n""",
+ "\n",
+ flags=re.DOTALL,
+)
+
+# Remove 'raw-latex' sections with sample JSON Lines files
+s.replace(
+ "google/cloud/**/io_pb2.py",
+ r"""Sample
+ in-line JSON Lines.*?(\n\s+-\s+For Text Classification.*\n)""",
+ "\g<1>",
+ flags=re.DOTALL,
+)
+
+# Replace docstring with no summary line
+s.replace(
+ "google/cloud/**/io_pb2.py",
+ r"""__doc__ = \"\"\"- For Translation: CSV file ``translation\.csv``, with each """,
+ r'''__doc__ = """
+- For Translation: CSV file ``translation.csv``, with each ''',
+ flags=re.DOTALL,
+)
+
+s.replace("google/cloud/**/io_pb2.py", r":raw-latex:`\\t `", r"\\\\t")
+# ----------------------------------------------------------------------------
+# Add templated files
+# ----------------------------------------------------------------------------
+templated_files = common.py_library(unit_cov_level=82, cov_level=83)
+s.move(templated_files)
+
+s.shell.run(["nox", "-s", "blacken"], hide_output=False)
diff --git a/automl/tests/system/gapic/v1beta1/test_system_tables_client_v1.py b/automl/tests/system/gapic/v1beta1/test_system_tables_client_v1.py
new file mode 100644
index 000000000000..01c49541dc70
--- /dev/null
+++ b/automl/tests/system/gapic/v1beta1/test_system_tables_client_v1.py
@@ -0,0 +1,294 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import pandas
+import pytest
+import random
+import string
+import time
+import unittest
+
+from google.cloud import automl_v1beta1
+from google.api_core import exceptions
+from google.cloud.automl_v1beta1.gapic import enums
+
+PROJECT = os.environ["PROJECT_ID"]
+REGION = "us-central1"
+MAX_WAIT_TIME_SECONDS = 30
+MAX_SLEEP_TIME_SECONDS = 5
+STATIC_DATASET = "test_dataset_do_not_delete"
+STATIC_MODEL = "test_model_do_not_delete"
+RUNNING_IN_VPCSC = os.getenv("GOOGLE_CLOUD_TESTS_IN_VPCSC", "").lower() == "true"
+
+ID = "{rand}_{time}".format(
+ rand="".join(
+ [random.choice(string.ascii_letters + string.digits) for n in range(4)]
+ ),
+ time=int(time.time()),
+)
+
+
+def _id(name):
+ return "{}_{}".format(name, ID)
+
+
+class TestSystemTablesClient(object):
+ def cancel_and_wait(self, op):
+ op.cancel()
+ start = time.time()
+ sleep_time = 1
+ while time.time() - start < MAX_WAIT_TIME_SECONDS:
+ if op.cancelled():
+ return
+ time.sleep(sleep_time)
+ sleep_time = min(sleep_time * 2, MAX_SLEEP_TIME_SECONDS)
+ assert op.cancelled()
+
+ @unittest.skipIf(RUNNING_IN_VPCSC, "Test is not VPCSC compatible.")
+ def test_list_datasets(self):
+ client = automl_v1beta1.TablesClient(project=PROJECT, region=REGION)
+ dataset = self.ensure_dataset_ready(client)
+ # will raise if not found
+ next(
+ iter(
+ [d for d in client.list_datasets(timeout=10) if d.name == dataset.name]
+ )
+ )
+
+ @unittest.skipIf(RUNNING_IN_VPCSC, "Test is not VPCSC compatible.")
+ def test_list_models(self):
+ client = automl_v1beta1.TablesClient(project=PROJECT, region=REGION)
+ model = self.ensure_model_ready(client)
+ # will raise if not found
+ next(iter([m for m in client.list_models(timeout=10) if m.name == model.name]))
+
+ def test_create_delete_dataset(self):
+ client = automl_v1beta1.TablesClient(project=PROJECT, region=REGION)
+ display_name = _id("t_cr_dl")
+ dataset = client.create_dataset(display_name)
+ assert dataset is not None
+ assert (
+ dataset.name == client.get_dataset(dataset_display_name=display_name).name
+ )
+ client.delete_dataset(dataset=dataset)
+
+ @unittest.skipIf(RUNNING_IN_VPCSC, "Test is not VPCSC compatible.")
+ def test_import_data(self):
+ client = automl_v1beta1.TablesClient(project=PROJECT, region=REGION)
+ display_name = _id("t_import")
+ dataset = client.create_dataset(display_name)
+ op = client.import_data(
+ dataset=dataset,
+ gcs_input_uris="gs://cloud-ml-tables-data/bank-marketing.csv",
+ )
+ self.cancel_and_wait(op)
+ client.delete_dataset(dataset=dataset)
+
+ @unittest.skipIf(RUNNING_IN_VPCSC, "Test is not VPCSC compatible.")
+ def test_import_pandas_dataframe(self):
+ client = automl_v1beta1.TablesClient(project=PROJECT, region=REGION)
+ display_name = _id("t_import_pandas")
+ dataset = client.create_dataset(display_name)
+ dataframe = pandas.DataFrame({"test-col1": [1, 2], "test-col2": [3, 4]})
+ op = client.import_data(
+ project=PROJECT, dataset=dataset, pandas_dataframe=dataframe
+ )
+ self.cancel_and_wait(op)
+ client.delete_dataset(dataset=dataset)
+
+ def ensure_dataset_ready(self, client):
+ dataset = None
+ try:
+ dataset = client.get_dataset(dataset_display_name=STATIC_DATASET)
+ except exceptions.NotFound:
+ dataset = client.create_dataset(STATIC_DATASET)
+
+ if dataset.example_count is None or dataset.example_count == 0:
+ op = client.import_data(
+ dataset=dataset,
+ gcs_input_uris="gs://cloud-ml-tables-data/bank-marketing.csv",
+ )
+ op.result()
+ dataset = client.get_dataset(dataset_name=dataset.name)
+
+ return dataset
+
+ @unittest.skipIf(RUNNING_IN_VPCSC, "Test is not VPCSC compatible.")
+ def test_list_column_specs(self):
+ client = automl_v1beta1.TablesClient(project=PROJECT, region=REGION)
+ dataset = self.ensure_dataset_ready(client)
+ # will raise if not found
+ next(
+ iter(
+ [
+ d
+ for d in client.list_column_specs(dataset=dataset)
+ if d.display_name == "Deposit"
+ ]
+ )
+ )
+
+ @unittest.skipIf(RUNNING_IN_VPCSC, "Test is not VPCSC compatible.")
+ def test_get_column_spec(self):
+ client = automl_v1beta1.TablesClient(project=PROJECT, region=REGION)
+ dataset = self.ensure_dataset_ready(client)
+ name = [d for d in client.list_column_specs(dataset=dataset)][0].name
+ assert client.get_column_spec(name).name == name
+
+ @unittest.skipIf(RUNNING_IN_VPCSC, "Test is not VPCSC compatible.")
+ def test_list_table_specs(self):
+ client = automl_v1beta1.TablesClient(project=PROJECT, region=REGION)
+ dataset = self.ensure_dataset_ready(client)
+ name = [d for d in client.list_table_specs(dataset=dataset)][0].name
+ assert client.get_table_spec(name).name == name
+
+ @unittest.skipIf(RUNNING_IN_VPCSC, "Test is not VPCSC compatible.")
+ def test_set_column_nullable(self):
+ client = automl_v1beta1.TablesClient(project=PROJECT, region=REGION)
+ dataset = self.ensure_dataset_ready(client)
+ client.update_column_spec(
+ dataset=dataset, column_spec_display_name="POutcome", nullable=True
+ )
+ columns = {c.display_name: c for c in client.list_column_specs(dataset=dataset)}
+ assert columns["POutcome"].data_type.nullable == True
+
+ @unittest.skipIf(RUNNING_IN_VPCSC, "Test is not VPCSC compatible.")
+ def test_set_target_column(self):
+ client = automl_v1beta1.TablesClient(project=PROJECT, region=REGION)
+ dataset = self.ensure_dataset_ready(client)
+ client.set_target_column(dataset=dataset, column_spec_display_name="Age")
+ columns = {c.display_name: c for c in client.list_column_specs(dataset=dataset)}
+ dataset = client.get_dataset(dataset_name=dataset.name)
+ metadata = dataset.tables_dataset_metadata
+ assert columns["Age"].name.endswith(
+ "/{}".format(metadata.target_column_spec_id)
+ )
+
+ @unittest.skipIf(RUNNING_IN_VPCSC, "Test is not VPCSC compatible.")
+ def test_set_weight_column(self):
+ client = automl_v1beta1.TablesClient(project=PROJECT, region=REGION)
+ dataset = self.ensure_dataset_ready(client)
+ client.set_weight_column(dataset=dataset, column_spec_display_name="Duration")
+ columns = {c.display_name: c for c in client.list_column_specs(dataset=dataset)}
+ dataset = client.get_dataset(dataset_name=dataset.name)
+ metadata = dataset.tables_dataset_metadata
+ assert columns["Duration"].name.endswith(
+ "/{}".format(metadata.weight_column_spec_id)
+ )
+
+ @unittest.skipIf(RUNNING_IN_VPCSC, "Test is not VPCSC compatible.")
+ def test_set_weight_and_target_column(self):
+ client = automl_v1beta1.TablesClient(project=PROJECT, region=REGION)
+ dataset = self.ensure_dataset_ready(client)
+ client.set_weight_column(dataset=dataset, column_spec_display_name="Day")
+ client.set_target_column(dataset=dataset, column_spec_display_name="Campaign")
+ columns = {c.display_name: c for c in client.list_column_specs(dataset=dataset)}
+ dataset = client.get_dataset(dataset_name=dataset.name)
+ metadata = dataset.tables_dataset_metadata
+ assert columns["Day"].name.endswith(
+ "/{}".format(metadata.weight_column_spec_id)
+ )
+ assert columns["Campaign"].name.endswith(
+ "/{}".format(metadata.target_column_spec_id)
+ )
+
+ @unittest.skipIf(RUNNING_IN_VPCSC, "Test is not VPCSC compatible.")
+ def test_create_delete_model(self):
+ client = automl_v1beta1.TablesClient(project=PROJECT, region=REGION)
+ dataset = self.ensure_dataset_ready(client)
+ client.set_target_column(dataset=dataset, column_spec_display_name="Deposit")
+ display_name = _id("t_cr_dl")
+ op = client.create_model(
+ display_name, dataset=dataset, train_budget_milli_node_hours=1000
+ )
+ self.cancel_and_wait(op)
+ client.delete_model(model_display_name=display_name)
+
+ @unittest.skipIf(RUNNING_IN_VPCSC, "Test is not VPCSC compatible.")
+ def test_list_model_evaluations(self):
+ client = automl_v1beta1.TablesClient(project=PROJECT, region=REGION)
+ model = self.ensure_model_online(client)
+ # will raise if not found
+ next(
+ iter(
+ [
+ m
+ for m in client.list_model_evaluations(model=model)
+ if m.display_name is not None
+ ]
+ )
+ )
+
+ @unittest.skipIf(RUNNING_IN_VPCSC, "Test is not VPCSC compatible.")
+ def test_get_model_evaluation(self):
+ client = automl_v1beta1.TablesClient(project=PROJECT, region=REGION)
+ model = self.ensure_model_online(client)
+ name = [m for m in client.list_model_evaluations(model=model)][0].name
+ assert client.get_model_evaluation(model_evaluation_name=name).name == name
+
+ @unittest.skipIf(RUNNING_IN_VPCSC, "Test is not VPCSC compatible.")
+ def test_online_predict(self):
+ client = automl_v1beta1.TablesClient(project=PROJECT, region=REGION)
+ model = self.ensure_model_online(client)
+ result = client.predict(
+ inputs={
+ "Age": 31,
+ "Balance": 200,
+ "Campaign": 2,
+ "Contact": "cellular",
+ "Day": 4,
+ "Default": "no",
+ "Duration": 12,
+ "Education": "primary",
+ "Housing": "yes",
+ "Job": "blue-collar",
+ "Loan": "no",
+ "MaritalStatus": "divorced",
+ "Month": "jul",
+ "PDays": 4,
+ "POutcome": None,
+ "Previous": 12,
+ },
+ model=model,
+ )
+ assert result is not None
+
+ def ensure_model_online(self, client):
+ model = self.ensure_model_ready(client)
+ if model.deployment_state != enums.Model.DeploymentState.DEPLOYED:
+ client.deploy_model(model=model).result()
+
+ return client.get_model(model_name=model.name)
+
+ def ensure_model_ready(self, client):
+ try:
+ return client.get_model(model_display_name=STATIC_MODEL)
+ except exceptions.NotFound:
+ pass
+
+ dataset = self.ensure_dataset_ready(client)
+ client.set_target_column(dataset=dataset, column_spec_display_name="Deposit")
+ client.clear_weight_column(dataset=dataset)
+ client.clear_test_train_column(dataset=dataset)
+ client.update_column_spec(
+ dataset=dataset, column_spec_display_name="POutcome", nullable=True
+ )
+ op = client.create_model(
+ STATIC_MODEL, dataset=dataset, train_budget_milli_node_hours=1000
+ )
+ op.result()
+ return client.get_model(model_display_name=STATIC_MODEL)
diff --git a/automl/tests/unit/gapic/v1beta1/test_auto_ml_client_v1beta1.py b/automl/tests/unit/gapic/v1beta1/test_auto_ml_client_v1beta1.py
new file mode 100644
index 000000000000..e3a5ce12f7cf
--- /dev/null
+++ b/automl/tests/unit/gapic/v1beta1/test_auto_ml_client_v1beta1.py
@@ -0,0 +1,1255 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Unit tests."""
+
+import mock
+import pytest
+
+from google.rpc import status_pb2
+
+from google.cloud import automl_v1beta1
+from google.cloud.automl_v1beta1.proto import annotation_spec_pb2
+from google.cloud.automl_v1beta1.proto import column_spec_pb2
+from google.cloud.automl_v1beta1.proto import dataset_pb2
+from google.cloud.automl_v1beta1.proto import io_pb2
+from google.cloud.automl_v1beta1.proto import model_evaluation_pb2
+from google.cloud.automl_v1beta1.proto import model_pb2
+from google.cloud.automl_v1beta1.proto import service_pb2
+from google.cloud.automl_v1beta1.proto import table_spec_pb2
+from google.longrunning import operations_pb2
+from google.protobuf import empty_pb2
+
+
+class MultiCallableStub(object):
+ """Stub for the grpc.UnaryUnaryMultiCallable interface."""
+
+ def __init__(self, method, channel_stub):
+ self.method = method
+ self.channel_stub = channel_stub
+
+ def __call__(self, request, timeout=None, metadata=None, credentials=None):
+ self.channel_stub.requests.append((self.method, request))
+
+ response = None
+ if self.channel_stub.responses:
+ response = self.channel_stub.responses.pop()
+
+ if isinstance(response, Exception):
+ raise response
+
+ if response:
+ return response
+
+
+class ChannelStub(object):
+ """Stub for the grpc.Channel interface."""
+
+ def __init__(self, responses=[]):
+ self.responses = responses
+ self.requests = []
+
+ def unary_unary(self, method, request_serializer=None, response_deserializer=None):
+ return MultiCallableStub(method, self)
+
+
+class CustomException(Exception):
+ pass
+
+
+class TestAutoMlClient(object):
+ def test_create_dataset(self):
+ # Setup Expected Response
+ name = "name3373707"
+ display_name = "displayName1615086568"
+ description = "description-1724546052"
+ example_count = 1517063674
+ etag = "etag3123477"
+ expected_response = {
+ "name": name,
+ "display_name": display_name,
+ "description": description,
+ "example_count": example_count,
+ "etag": etag,
+ }
+ expected_response = dataset_pb2.Dataset(**expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ parent = client.location_path("[PROJECT]", "[LOCATION]")
+ dataset = {}
+
+ response = client.create_dataset(parent, dataset)
+ assert expected_response == response
+
+ assert len(channel.requests) == 1
+ expected_request = service_pb2.CreateDatasetRequest(
+ parent=parent, dataset=dataset
+ )
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_create_dataset_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup request
+ parent = client.location_path("[PROJECT]", "[LOCATION]")
+ dataset = {}
+
+ with pytest.raises(CustomException):
+ client.create_dataset(parent, dataset)
+
+ def test_update_dataset(self):
+ # Setup Expected Response
+ name = "name3373707"
+ display_name = "displayName1615086568"
+ description = "description-1724546052"
+ example_count = 1517063674
+ etag = "etag3123477"
+ expected_response = {
+ "name": name,
+ "display_name": display_name,
+ "description": description,
+ "example_count": example_count,
+ "etag": etag,
+ }
+ expected_response = dataset_pb2.Dataset(**expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ dataset = {}
+
+ response = client.update_dataset(dataset)
+ assert expected_response == response
+
+ assert len(channel.requests) == 1
+ expected_request = service_pb2.UpdateDatasetRequest(dataset=dataset)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_update_dataset_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup request
+ dataset = {}
+
+ with pytest.raises(CustomException):
+ client.update_dataset(dataset)
+
+ def test_get_dataset(self):
+ # Setup Expected Response
+ name_2 = "name2-1052831874"
+ display_name = "displayName1615086568"
+ description = "description-1724546052"
+ example_count = 1517063674
+ etag = "etag3123477"
+ expected_response = {
+ "name": name_2,
+ "display_name": display_name,
+ "description": description,
+ "example_count": example_count,
+ "etag": etag,
+ }
+ expected_response = dataset_pb2.Dataset(**expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ name = client.dataset_path("[PROJECT]", "[LOCATION]", "[DATASET]")
+
+ response = client.get_dataset(name)
+ assert expected_response == response
+
+ assert len(channel.requests) == 1
+ expected_request = service_pb2.GetDatasetRequest(name=name)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_get_dataset_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup request
+ name = client.dataset_path("[PROJECT]", "[LOCATION]", "[DATASET]")
+
+ with pytest.raises(CustomException):
+ client.get_dataset(name)
+
+ def test_list_datasets(self):
+ # Setup Expected Response
+ next_page_token = ""
+ datasets_element = {}
+ datasets = [datasets_element]
+ expected_response = {"next_page_token": next_page_token, "datasets": datasets}
+ expected_response = service_pb2.ListDatasetsResponse(**expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ parent = client.location_path("[PROJECT]", "[LOCATION]")
+
+ paged_list_response = client.list_datasets(parent)
+ resources = list(paged_list_response)
+ assert len(resources) == 1
+
+ assert expected_response.datasets[0] == resources[0]
+
+ assert len(channel.requests) == 1
+ expected_request = service_pb2.ListDatasetsRequest(parent=parent)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_list_datasets_exception(self):
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup request
+ parent = client.location_path("[PROJECT]", "[LOCATION]")
+
+ paged_list_response = client.list_datasets(parent)
+ with pytest.raises(CustomException):
+ list(paged_list_response)
+
+ def test_delete_dataset(self):
+ # Setup Expected Response
+ expected_response = {}
+ expected_response = empty_pb2.Empty(**expected_response)
+ operation = operations_pb2.Operation(
+ name="operations/test_delete_dataset", done=True
+ )
+ operation.response.Pack(expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[operation])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ name = client.dataset_path("[PROJECT]", "[LOCATION]", "[DATASET]")
+
+ response = client.delete_dataset(name)
+ result = response.result()
+ assert expected_response == result
+
+ assert len(channel.requests) == 1
+ expected_request = service_pb2.DeleteDatasetRequest(name=name)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_delete_dataset_exception(self):
+ # Setup Response
+ error = status_pb2.Status()
+ operation = operations_pb2.Operation(
+ name="operations/test_delete_dataset_exception", done=True
+ )
+ operation.error.CopyFrom(error)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[operation])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ name = client.dataset_path("[PROJECT]", "[LOCATION]", "[DATASET]")
+
+ response = client.delete_dataset(name)
+ exception = response.exception()
+ assert exception.errors[0] == error
+
+ def test_import_data(self):
+ # Setup Expected Response
+ expected_response = {}
+ expected_response = empty_pb2.Empty(**expected_response)
+ operation = operations_pb2.Operation(
+ name="operations/test_import_data", done=True
+ )
+ operation.response.Pack(expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[operation])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ name = client.dataset_path("[PROJECT]", "[LOCATION]", "[DATASET]")
+ input_config = {}
+
+ response = client.import_data(name, input_config)
+ result = response.result()
+ assert expected_response == result
+
+ assert len(channel.requests) == 1
+ expected_request = service_pb2.ImportDataRequest(
+ name=name, input_config=input_config
+ )
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_import_data_exception(self):
+ # Setup Response
+ error = status_pb2.Status()
+ operation = operations_pb2.Operation(
+ name="operations/test_import_data_exception", done=True
+ )
+ operation.error.CopyFrom(error)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[operation])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ name = client.dataset_path("[PROJECT]", "[LOCATION]", "[DATASET]")
+ input_config = {}
+
+ response = client.import_data(name, input_config)
+ exception = response.exception()
+ assert exception.errors[0] == error
+
+ def test_export_data(self):
+ # Setup Expected Response
+ expected_response = {}
+ expected_response = empty_pb2.Empty(**expected_response)
+ operation = operations_pb2.Operation(
+ name="operations/test_export_data", done=True
+ )
+ operation.response.Pack(expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[operation])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ name = client.dataset_path("[PROJECT]", "[LOCATION]", "[DATASET]")
+ output_config = {}
+
+ response = client.export_data(name, output_config)
+ result = response.result()
+ assert expected_response == result
+
+ assert len(channel.requests) == 1
+ expected_request = service_pb2.ExportDataRequest(
+ name=name, output_config=output_config
+ )
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_export_data_exception(self):
+ # Setup Response
+ error = status_pb2.Status()
+ operation = operations_pb2.Operation(
+ name="operations/test_export_data_exception", done=True
+ )
+ operation.error.CopyFrom(error)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[operation])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ name = client.dataset_path("[PROJECT]", "[LOCATION]", "[DATASET]")
+ output_config = {}
+
+ response = client.export_data(name, output_config)
+ exception = response.exception()
+ assert exception.errors[0] == error
+
+ def test_create_model(self):
+ # Setup Expected Response
+ name = "name3373707"
+ display_name = "displayName1615086568"
+ dataset_id = "datasetId-2115646910"
+ expected_response = {
+ "name": name,
+ "display_name": display_name,
+ "dataset_id": dataset_id,
+ }
+ expected_response = model_pb2.Model(**expected_response)
+ operation = operations_pb2.Operation(
+ name="operations/test_create_model", done=True
+ )
+ operation.response.Pack(expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[operation])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ parent = client.location_path("[PROJECT]", "[LOCATION]")
+ model = {}
+
+ response = client.create_model(parent, model)
+ result = response.result()
+ assert expected_response == result
+
+ assert len(channel.requests) == 1
+ expected_request = service_pb2.CreateModelRequest(parent=parent, model=model)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_create_model_exception(self):
+ # Setup Response
+ error = status_pb2.Status()
+ operation = operations_pb2.Operation(
+ name="operations/test_create_model_exception", done=True
+ )
+ operation.error.CopyFrom(error)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[operation])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ parent = client.location_path("[PROJECT]", "[LOCATION]")
+ model = {}
+
+ response = client.create_model(parent, model)
+ exception = response.exception()
+ assert exception.errors[0] == error
+
+ def test_get_model(self):
+ # Setup Expected Response
+ name_2 = "name2-1052831874"
+ display_name = "displayName1615086568"
+ dataset_id = "datasetId-2115646910"
+ expected_response = {
+ "name": name_2,
+ "display_name": display_name,
+ "dataset_id": dataset_id,
+ }
+ expected_response = model_pb2.Model(**expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ name = client.model_path("[PROJECT]", "[LOCATION]", "[MODEL]")
+
+ response = client.get_model(name)
+ assert expected_response == response
+
+ assert len(channel.requests) == 1
+ expected_request = service_pb2.GetModelRequest(name=name)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_get_model_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup request
+ name = client.model_path("[PROJECT]", "[LOCATION]", "[MODEL]")
+
+ with pytest.raises(CustomException):
+ client.get_model(name)
+
+ def test_list_models(self):
+ # Setup Expected Response
+ next_page_token = ""
+ model_element = {}
+ model = [model_element]
+ expected_response = {"next_page_token": next_page_token, "model": model}
+ expected_response = service_pb2.ListModelsResponse(**expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ parent = client.location_path("[PROJECT]", "[LOCATION]")
+
+ paged_list_response = client.list_models(parent)
+ resources = list(paged_list_response)
+ assert len(resources) == 1
+
+ assert expected_response.model[0] == resources[0]
+
+ assert len(channel.requests) == 1
+ expected_request = service_pb2.ListModelsRequest(parent=parent)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_list_models_exception(self):
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup request
+ parent = client.location_path("[PROJECT]", "[LOCATION]")
+
+ paged_list_response = client.list_models(parent)
+ with pytest.raises(CustomException):
+ list(paged_list_response)
+
+ def test_delete_model(self):
+ # Setup Expected Response
+ expected_response = {}
+ expected_response = empty_pb2.Empty(**expected_response)
+ operation = operations_pb2.Operation(
+ name="operations/test_delete_model", done=True
+ )
+ operation.response.Pack(expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[operation])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ name = client.model_path("[PROJECT]", "[LOCATION]", "[MODEL]")
+
+ response = client.delete_model(name)
+ result = response.result()
+ assert expected_response == result
+
+ assert len(channel.requests) == 1
+ expected_request = service_pb2.DeleteModelRequest(name=name)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_delete_model_exception(self):
+ # Setup Response
+ error = status_pb2.Status()
+ operation = operations_pb2.Operation(
+ name="operations/test_delete_model_exception", done=True
+ )
+ operation.error.CopyFrom(error)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[operation])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ name = client.model_path("[PROJECT]", "[LOCATION]", "[MODEL]")
+
+ response = client.delete_model(name)
+ exception = response.exception()
+ assert exception.errors[0] == error
+
+ def test_deploy_model(self):
+ # Setup Expected Response
+ expected_response = {}
+ expected_response = empty_pb2.Empty(**expected_response)
+ operation = operations_pb2.Operation(
+ name="operations/test_deploy_model", done=True
+ )
+ operation.response.Pack(expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[operation])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ name = client.model_path("[PROJECT]", "[LOCATION]", "[MODEL]")
+
+ response = client.deploy_model(name)
+ result = response.result()
+ assert expected_response == result
+
+ assert len(channel.requests) == 1
+ expected_request = service_pb2.DeployModelRequest(name=name)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_deploy_model_exception(self):
+ # Setup Response
+ error = status_pb2.Status()
+ operation = operations_pb2.Operation(
+ name="operations/test_deploy_model_exception", done=True
+ )
+ operation.error.CopyFrom(error)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[operation])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ name = client.model_path("[PROJECT]", "[LOCATION]", "[MODEL]")
+
+ response = client.deploy_model(name)
+ exception = response.exception()
+ assert exception.errors[0] == error
+
+ def test_undeploy_model(self):
+ # Setup Expected Response
+ expected_response = {}
+ expected_response = empty_pb2.Empty(**expected_response)
+ operation = operations_pb2.Operation(
+ name="operations/test_undeploy_model", done=True
+ )
+ operation.response.Pack(expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[operation])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ name = client.model_path("[PROJECT]", "[LOCATION]", "[MODEL]")
+
+ response = client.undeploy_model(name)
+ result = response.result()
+ assert expected_response == result
+
+ assert len(channel.requests) == 1
+ expected_request = service_pb2.UndeployModelRequest(name=name)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_undeploy_model_exception(self):
+ # Setup Response
+ error = status_pb2.Status()
+ operation = operations_pb2.Operation(
+ name="operations/test_undeploy_model_exception", done=True
+ )
+ operation.error.CopyFrom(error)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[operation])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ name = client.model_path("[PROJECT]", "[LOCATION]", "[MODEL]")
+
+ response = client.undeploy_model(name)
+ exception = response.exception()
+ assert exception.errors[0] == error
+
+ def test_get_model_evaluation(self):
+ # Setup Expected Response
+ name_2 = "name2-1052831874"
+ annotation_spec_id = "annotationSpecId60690191"
+ display_name = "displayName1615086568"
+ evaluated_example_count = 277565350
+ expected_response = {
+ "name": name_2,
+ "annotation_spec_id": annotation_spec_id,
+ "display_name": display_name,
+ "evaluated_example_count": evaluated_example_count,
+ }
+ expected_response = model_evaluation_pb2.ModelEvaluation(**expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ name = client.model_evaluation_path(
+ "[PROJECT]", "[LOCATION]", "[MODEL]", "[MODEL_EVALUATION]"
+ )
+
+ response = client.get_model_evaluation(name)
+ assert expected_response == response
+
+ assert len(channel.requests) == 1
+ expected_request = service_pb2.GetModelEvaluationRequest(name=name)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_get_model_evaluation_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup request
+ name = client.model_evaluation_path(
+ "[PROJECT]", "[LOCATION]", "[MODEL]", "[MODEL_EVALUATION]"
+ )
+
+ with pytest.raises(CustomException):
+ client.get_model_evaluation(name)
+
+ def test_export_model(self):
+ # Setup Expected Response
+ expected_response = {}
+ expected_response = empty_pb2.Empty(**expected_response)
+ operation = operations_pb2.Operation(
+ name="operations/test_export_model", done=True
+ )
+ operation.response.Pack(expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[operation])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ name = client.model_path("[PROJECT]", "[LOCATION]", "[MODEL]")
+ output_config = {}
+
+ response = client.export_model(name, output_config)
+ result = response.result()
+ assert expected_response == result
+
+ assert len(channel.requests) == 1
+ expected_request = service_pb2.ExportModelRequest(
+ name=name, output_config=output_config
+ )
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_export_model_exception(self):
+ # Setup Response
+ error = status_pb2.Status()
+ operation = operations_pb2.Operation(
+ name="operations/test_export_model_exception", done=True
+ )
+ operation.error.CopyFrom(error)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[operation])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ name = client.model_path("[PROJECT]", "[LOCATION]", "[MODEL]")
+ output_config = {}
+
+ response = client.export_model(name, output_config)
+ exception = response.exception()
+ assert exception.errors[0] == error
+
+ def test_export_evaluated_examples(self):
+ # Setup Expected Response
+ expected_response = {}
+ expected_response = empty_pb2.Empty(**expected_response)
+ operation = operations_pb2.Operation(
+ name="operations/test_export_evaluated_examples", done=True
+ )
+ operation.response.Pack(expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[operation])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ name = client.model_path("[PROJECT]", "[LOCATION]", "[MODEL]")
+ output_config = {}
+
+ response = client.export_evaluated_examples(name, output_config)
+ result = response.result()
+ assert expected_response == result
+
+ assert len(channel.requests) == 1
+ expected_request = service_pb2.ExportEvaluatedExamplesRequest(
+ name=name, output_config=output_config
+ )
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_export_evaluated_examples_exception(self):
+ # Setup Response
+ error = status_pb2.Status()
+ operation = operations_pb2.Operation(
+ name="operations/test_export_evaluated_examples_exception", done=True
+ )
+ operation.error.CopyFrom(error)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[operation])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ name = client.model_path("[PROJECT]", "[LOCATION]", "[MODEL]")
+ output_config = {}
+
+ response = client.export_evaluated_examples(name, output_config)
+ exception = response.exception()
+ assert exception.errors[0] == error
+
+ def test_list_model_evaluations(self):
+ # Setup Expected Response
+ next_page_token = ""
+ model_evaluation_element = {}
+ model_evaluation = [model_evaluation_element]
+ expected_response = {
+ "next_page_token": next_page_token,
+ "model_evaluation": model_evaluation,
+ }
+ expected_response = service_pb2.ListModelEvaluationsResponse(
+ **expected_response
+ )
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ parent = client.model_path("[PROJECT]", "[LOCATION]", "[MODEL]")
+
+ paged_list_response = client.list_model_evaluations(parent)
+ resources = list(paged_list_response)
+ assert len(resources) == 1
+
+ assert expected_response.model_evaluation[0] == resources[0]
+
+ assert len(channel.requests) == 1
+ expected_request = service_pb2.ListModelEvaluationsRequest(parent=parent)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_list_model_evaluations_exception(self):
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup request
+ parent = client.model_path("[PROJECT]", "[LOCATION]", "[MODEL]")
+
+ paged_list_response = client.list_model_evaluations(parent)
+ with pytest.raises(CustomException):
+ list(paged_list_response)
+
+ def test_get_annotation_spec(self):
+ # Setup Expected Response
+ name_2 = "name2-1052831874"
+ display_name = "displayName1615086568"
+ example_count = 1517063674
+ expected_response = {
+ "name": name_2,
+ "display_name": display_name,
+ "example_count": example_count,
+ }
+ expected_response = annotation_spec_pb2.AnnotationSpec(**expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ name = client.annotation_spec_path(
+ "[PROJECT]", "[LOCATION]", "[DATASET]", "[ANNOTATION_SPEC]"
+ )
+
+ response = client.get_annotation_spec(name)
+ assert expected_response == response
+
+ assert len(channel.requests) == 1
+ expected_request = service_pb2.GetAnnotationSpecRequest(name=name)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_get_annotation_spec_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup request
+ name = client.annotation_spec_path(
+ "[PROJECT]", "[LOCATION]", "[DATASET]", "[ANNOTATION_SPEC]"
+ )
+
+ with pytest.raises(CustomException):
+ client.get_annotation_spec(name)
+
+ def test_get_table_spec(self):
+ # Setup Expected Response
+ name_2 = "name2-1052831874"
+ time_column_spec_id = "timeColumnSpecId1558734824"
+ row_count = 1340416618
+ valid_row_count = 406068761
+ column_count = 122671386
+ etag = "etag3123477"
+ expected_response = {
+ "name": name_2,
+ "time_column_spec_id": time_column_spec_id,
+ "row_count": row_count,
+ "valid_row_count": valid_row_count,
+ "column_count": column_count,
+ "etag": etag,
+ }
+ expected_response = table_spec_pb2.TableSpec(**expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ name = client.table_spec_path(
+ "[PROJECT]", "[LOCATION]", "[DATASET]", "[TABLE_SPEC]"
+ )
+
+ response = client.get_table_spec(name)
+ assert expected_response == response
+
+ assert len(channel.requests) == 1
+ expected_request = service_pb2.GetTableSpecRequest(name=name)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_get_table_spec_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup request
+ name = client.table_spec_path(
+ "[PROJECT]", "[LOCATION]", "[DATASET]", "[TABLE_SPEC]"
+ )
+
+ with pytest.raises(CustomException):
+ client.get_table_spec(name)
+
+ def test_list_table_specs(self):
+ # Setup Expected Response
+ next_page_token = ""
+ table_specs_element = {}
+ table_specs = [table_specs_element]
+ expected_response = {
+ "next_page_token": next_page_token,
+ "table_specs": table_specs,
+ }
+ expected_response = service_pb2.ListTableSpecsResponse(**expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ parent = client.dataset_path("[PROJECT]", "[LOCATION]", "[DATASET]")
+
+ paged_list_response = client.list_table_specs(parent)
+ resources = list(paged_list_response)
+ assert len(resources) == 1
+
+ assert expected_response.table_specs[0] == resources[0]
+
+ assert len(channel.requests) == 1
+ expected_request = service_pb2.ListTableSpecsRequest(parent=parent)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_list_table_specs_exception(self):
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup request
+ parent = client.dataset_path("[PROJECT]", "[LOCATION]", "[DATASET]")
+
+ paged_list_response = client.list_table_specs(parent)
+ with pytest.raises(CustomException):
+ list(paged_list_response)
+
+ def test_update_table_spec(self):
+ # Setup Expected Response
+ name = "name3373707"
+ time_column_spec_id = "timeColumnSpecId1558734824"
+ row_count = 1340416618
+ valid_row_count = 406068761
+ column_count = 122671386
+ etag = "etag3123477"
+ expected_response = {
+ "name": name,
+ "time_column_spec_id": time_column_spec_id,
+ "row_count": row_count,
+ "valid_row_count": valid_row_count,
+ "column_count": column_count,
+ "etag": etag,
+ }
+ expected_response = table_spec_pb2.TableSpec(**expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ table_spec = {}
+
+ response = client.update_table_spec(table_spec)
+ assert expected_response == response
+
+ assert len(channel.requests) == 1
+ expected_request = service_pb2.UpdateTableSpecRequest(table_spec=table_spec)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_update_table_spec_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup request
+ table_spec = {}
+
+ with pytest.raises(CustomException):
+ client.update_table_spec(table_spec)
+
+ def test_get_column_spec(self):
+ # Setup Expected Response
+ name_2 = "name2-1052831874"
+ display_name = "displayName1615086568"
+ etag = "etag3123477"
+ expected_response = {"name": name_2, "display_name": display_name, "etag": etag}
+ expected_response = column_spec_pb2.ColumnSpec(**expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ name = client.column_spec_path(
+ "[PROJECT]", "[LOCATION]", "[DATASET]", "[TABLE_SPEC]", "[COLUMN_SPEC]"
+ )
+
+ response = client.get_column_spec(name)
+ assert expected_response == response
+
+ assert len(channel.requests) == 1
+ expected_request = service_pb2.GetColumnSpecRequest(name=name)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_get_column_spec_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup request
+ name = client.column_spec_path(
+ "[PROJECT]", "[LOCATION]", "[DATASET]", "[TABLE_SPEC]", "[COLUMN_SPEC]"
+ )
+
+ with pytest.raises(CustomException):
+ client.get_column_spec(name)
+
+ def test_list_column_specs(self):
+ # Setup Expected Response
+ next_page_token = ""
+ column_specs_element = {}
+ column_specs = [column_specs_element]
+ expected_response = {
+ "next_page_token": next_page_token,
+ "column_specs": column_specs,
+ }
+ expected_response = service_pb2.ListColumnSpecsResponse(**expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ parent = client.table_spec_path(
+ "[PROJECT]", "[LOCATION]", "[DATASET]", "[TABLE_SPEC]"
+ )
+
+ paged_list_response = client.list_column_specs(parent)
+ resources = list(paged_list_response)
+ assert len(resources) == 1
+
+ assert expected_response.column_specs[0] == resources[0]
+
+ assert len(channel.requests) == 1
+ expected_request = service_pb2.ListColumnSpecsRequest(parent=parent)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_list_column_specs_exception(self):
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup request
+ parent = client.table_spec_path(
+ "[PROJECT]", "[LOCATION]", "[DATASET]", "[TABLE_SPEC]"
+ )
+
+ paged_list_response = client.list_column_specs(parent)
+ with pytest.raises(CustomException):
+ list(paged_list_response)
+
+ def test_update_column_spec(self):
+ # Setup Expected Response
+ name = "name3373707"
+ display_name = "displayName1615086568"
+ etag = "etag3123477"
+ expected_response = {"name": name, "display_name": display_name, "etag": etag}
+ expected_response = column_spec_pb2.ColumnSpec(**expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup Request
+ column_spec = {}
+
+ response = client.update_column_spec(column_spec)
+ assert expected_response == response
+
+ assert len(channel.requests) == 1
+ expected_request = service_pb2.UpdateColumnSpecRequest(column_spec=column_spec)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_update_column_spec_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.AutoMlClient()
+
+ # Setup request
+ column_spec = {}
+
+ with pytest.raises(CustomException):
+ client.update_column_spec(column_spec)
diff --git a/automl/tests/unit/gapic/v1beta1/test_gcs_client_v1beta1.py b/automl/tests/unit/gapic/v1beta1/test_gcs_client_v1beta1.py
new file mode 100644
index 000000000000..49d4a0f85423
--- /dev/null
+++ b/automl/tests/unit/gapic/v1beta1/test_gcs_client_v1beta1.py
@@ -0,0 +1,152 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Unit tests."""
+
+import mock
+import pandas
+import pytest
+import re
+
+from google.api_core import exceptions
+from google.cloud import automl_v1beta1
+
+
+class TestGcsClient(object):
+ def gcs_client(self, bucket_name=None, client_attrs={}):
+ client_mock = mock.Mock(**client_attrs)
+ return automl_v1beta1.tables.gcs_client.GcsClient(
+ bucket_name=bucket_name, client=client_mock
+ )
+
+ def test_ensure_bucket_exists(self):
+ mock_bucket = mock.Mock()
+ gcs_client = self.gcs_client(
+ client_attrs={
+ "get_bucket.side_effect": exceptions.NotFound("err"),
+ "bucket.return_value": mock_bucket,
+ }
+ )
+ returned_bucket_name = gcs_client.ensure_bucket_exists(
+ "my-project", "us-central1"
+ )
+ gcs_client.client.get_bucket.assert_called_with(
+ "my-project-automl-tables-staging"
+ )
+ gcs_client.client.bucket.assert_called_with("my-project-automl-tables-staging")
+ mock_bucket.create.assert_called_with(
+ project="my-project", location="us-central1"
+ )
+ assert returned_bucket_name == "my-project-automl-tables-staging"
+
+ def test_ensure_bucket_exists_bucket_already_exists_in_different_project(self):
+ mock_bucket = mock.Mock()
+ gcs_client = self.gcs_client(
+ client_attrs={
+ "get_bucket.side_effect": exceptions.Forbidden("err"),
+ "bucket.return_value": mock_bucket,
+ }
+ )
+ returned_bucket_name = gcs_client.ensure_bucket_exists(
+ "my-project", "us-central1"
+ )
+ gcs_client.client.get_bucket.assert_called_with(
+ "my-project-automl-tables-staging"
+ )
+ gcs_client.client.bucket.assert_called_with(returned_bucket_name)
+ mock_bucket.create.assert_called_with(
+ project="my-project", location="us-central1"
+ )
+
+ assert re.match(
+ "^my-project-automl-tables-staging-[0-9]*$", returned_bucket_name
+ )
+
+ def test_ensure_bucket_exists_bucket_already_exists_in_current_project(self):
+ gcs_client = self.gcs_client()
+ returned_bucket_name = gcs_client.ensure_bucket_exists(
+ "my-project", "us-central1"
+ )
+ gcs_client.client.get_bucket.assert_called_with(
+ "my-project-automl-tables-staging"
+ )
+ gcs_client.client.bucket.assert_not_called()
+ assert returned_bucket_name == "my-project-automl-tables-staging"
+
+ def test_ensure_bucket_exists_custom_bucket_name(self):
+ mock_bucket = mock.Mock()
+ gcs_client = self.gcs_client(
+ bucket_name="my-bucket",
+ client_attrs={
+ "get_bucket.side_effect": exceptions.NotFound("err"),
+ "bucket.return_value": mock_bucket,
+ },
+ )
+ returned_bucket_name = gcs_client.ensure_bucket_exists(
+ "my-project", "us-central1"
+ )
+ gcs_client.client.get_bucket.assert_called_with("my-bucket")
+ gcs_client.client.bucket.assert_called_with("my-bucket")
+ mock_bucket.create.assert_called_with(
+ project="my-project", location="us-central1"
+ )
+ assert returned_bucket_name == "my-bucket"
+
+ def test_upload_pandas_dataframe(self):
+ mock_blob = mock.Mock()
+ mock_bucket = mock.Mock(**{"blob.return_value": mock_blob})
+ gcs_client = self.gcs_client(
+ bucket_name="my-bucket",
+ client_attrs={"get_bucket.return_value": mock_bucket},
+ )
+ dataframe = pandas.DataFrame({"col1": [1, 2], "col2": [3, 4]})
+
+ gcs_uri = gcs_client.upload_pandas_dataframe(dataframe, "my-file.csv")
+
+ gcs_client.client.get_bucket.assert_called_with("my-bucket")
+ mock_bucket.blob.assert_called_with("my-file.csv")
+ mock_blob.upload_from_string.assert_called_with(",col1,col2\n0,1,3\n1,2,4\n")
+ assert gcs_uri == "gs://my-bucket/my-file.csv"
+
+ def test_upload_pandas_dataframe_no_csv_name(self):
+ mock_blob = mock.Mock()
+ mock_bucket = mock.Mock(**{"blob.return_value": mock_blob})
+ gcs_client = self.gcs_client(
+ bucket_name="my-bucket",
+ client_attrs={"get_bucket.return_value": mock_bucket},
+ )
+ dataframe = pandas.DataFrame({"col1": [1, 2], "col2": [3, 4]})
+
+ gcs_uri = gcs_client.upload_pandas_dataframe(dataframe)
+ generated_csv_name = gcs_uri.split("/")[-1]
+
+ gcs_client.client.get_bucket.assert_called_with("my-bucket")
+ mock_bucket.blob.assert_called_with(generated_csv_name)
+ mock_blob.upload_from_string.assert_called_with(",col1,col2\n0,1,3\n1,2,4\n")
+ assert re.match("^gs://my-bucket/automl-tables-dataframe-[0-9]*.csv$", gcs_uri)
+
+ def test_upload_pandas_dataframe_not_type_dataframe(self):
+ gcs_client = self.gcs_client()
+ with pytest.raises(ValueError):
+ gcs_client.upload_pandas_dataframe("my-dataframe")
+ gcs_client.client.upload_pandas_dataframe.assert_not_called()
+
+ def test_upload_pandas_dataframe_bucket_not_exist(self):
+ gcs_client = self.gcs_client()
+ dataframe = pandas.DataFrame({})
+ with pytest.raises(ValueError):
+ gcs_client.upload_pandas_dataframe(dataframe)
+ gcs_client.client.upload_pandas_dataframe.assert_not_called()
diff --git a/automl/tests/unit/gapic/v1beta1/test_prediction_service_client_v1beta1.py b/automl/tests/unit/gapic/v1beta1/test_prediction_service_client_v1beta1.py
new file mode 100644
index 000000000000..e3ab187f4644
--- /dev/null
+++ b/automl/tests/unit/gapic/v1beta1/test_prediction_service_client_v1beta1.py
@@ -0,0 +1,165 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Unit tests."""
+
+import mock
+import pytest
+
+from google.rpc import status_pb2
+
+from google.cloud import automl_v1beta1
+from google.cloud.automl_v1beta1.proto import data_items_pb2
+from google.cloud.automl_v1beta1.proto import io_pb2
+from google.cloud.automl_v1beta1.proto import prediction_service_pb2
+from google.longrunning import operations_pb2
+
+
+class MultiCallableStub(object):
+ """Stub for the grpc.UnaryUnaryMultiCallable interface."""
+
+ def __init__(self, method, channel_stub):
+ self.method = method
+ self.channel_stub = channel_stub
+
+ def __call__(self, request, timeout=None, metadata=None, credentials=None):
+ self.channel_stub.requests.append((self.method, request))
+
+ response = None
+ if self.channel_stub.responses:
+ response = self.channel_stub.responses.pop()
+
+ if isinstance(response, Exception):
+ raise response
+
+ if response:
+ return response
+
+
+class ChannelStub(object):
+ """Stub for the grpc.Channel interface."""
+
+ def __init__(self, responses=[]):
+ self.responses = responses
+ self.requests = []
+
+ def unary_unary(self, method, request_serializer=None, response_deserializer=None):
+ return MultiCallableStub(method, self)
+
+
+class CustomException(Exception):
+ pass
+
+
+class TestPredictionServiceClient(object):
+ def test_predict(self):
+ # Setup Expected Response
+ expected_response = {}
+ expected_response = prediction_service_pb2.PredictResponse(**expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.PredictionServiceClient()
+
+ # Setup Request
+ name = client.model_path("[PROJECT]", "[LOCATION]", "[MODEL]")
+ payload = {}
+
+ response = client.predict(name, payload)
+ assert expected_response == response
+
+ assert len(channel.requests) == 1
+ expected_request = prediction_service_pb2.PredictRequest(
+ name=name, payload=payload
+ )
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_predict_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.PredictionServiceClient()
+
+ # Setup request
+ name = client.model_path("[PROJECT]", "[LOCATION]", "[MODEL]")
+ payload = {}
+
+ with pytest.raises(CustomException):
+ client.predict(name, payload)
+
+ def test_batch_predict(self):
+ # Setup Expected Response
+ expected_response = {}
+ expected_response = prediction_service_pb2.BatchPredictResult(
+ **expected_response
+ )
+ operation = operations_pb2.Operation(
+ name="operations/test_batch_predict", done=True
+ )
+ operation.response.Pack(expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[operation])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.PredictionServiceClient()
+
+ # Setup Request
+ name = client.model_path("[PROJECT]", "[LOCATION]", "[MODEL]")
+ input_config = {}
+ output_config = {}
+
+ response = client.batch_predict(name, input_config, output_config)
+ result = response.result()
+ assert expected_response == result
+
+ assert len(channel.requests) == 1
+ expected_request = prediction_service_pb2.BatchPredictRequest(
+ name=name, input_config=input_config, output_config=output_config
+ )
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_batch_predict_exception(self):
+ # Setup Response
+ error = status_pb2.Status()
+ operation = operations_pb2.Operation(
+ name="operations/test_batch_predict_exception", done=True
+ )
+ operation.error.CopyFrom(error)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[operation])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = automl_v1beta1.PredictionServiceClient()
+
+ # Setup Request
+ name = client.model_path("[PROJECT]", "[LOCATION]", "[MODEL]")
+ input_config = {}
+ output_config = {}
+
+ response = client.batch_predict(name, input_config, output_config)
+ exception = response.exception()
+ assert exception.errors[0] == error
diff --git a/automl/tests/unit/gapic/v1beta1/test_tables_client_v1beta1.py b/automl/tests/unit/gapic/v1beta1/test_tables_client_v1beta1.py
new file mode 100644
index 000000000000..aa1babfa8752
--- /dev/null
+++ b/automl/tests/unit/gapic/v1beta1/test_tables_client_v1beta1.py
@@ -0,0 +1,1319 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Unit tests."""
+
+import mock
+import pandas
+import pytest
+
+from google.cloud import automl_v1beta1
+from google.api_core import exceptions
+from google.cloud.automl_v1beta1.proto import data_types_pb2
+
+PROJECT = "project"
+REGION = "region"
+LOCATION_PATH = "projects/{}/locations/{}".format(PROJECT, REGION)
+
+
+class TestTablesClient(object):
+ def tables_client(
+ self, client_attrs={}, prediction_client_attrs={}, gcs_client_attrs={}
+ ):
+ client_mock = mock.Mock(**client_attrs)
+ prediction_client_mock = mock.Mock(**prediction_client_attrs)
+ gcs_client_mock = mock.Mock(**gcs_client_attrs)
+ return automl_v1beta1.TablesClient(
+ client=client_mock,
+ prediction_client=prediction_client_mock,
+ gcs_client=gcs_client_mock,
+ project=PROJECT,
+ region=REGION,
+ )
+
+ def test_list_datasets_empty(self):
+ client = self.tables_client(
+ {
+ "list_datasets.return_value": [],
+ "location_path.return_value": LOCATION_PATH,
+ },
+ {},
+ )
+ ds = client.list_datasets()
+ client.auto_ml_client.location_path.assert_called_with(PROJECT, REGION)
+ client.auto_ml_client.list_datasets.assert_called_with(LOCATION_PATH)
+ assert ds == []
+
+ def test_list_datasets_not_empty(self):
+ datasets = ["some_dataset"]
+ client = self.tables_client(
+ {
+ "list_datasets.return_value": datasets,
+ "location_path.return_value": LOCATION_PATH,
+ },
+ {},
+ )
+ ds = client.list_datasets()
+ client.auto_ml_client.location_path.assert_called_with(PROJECT, REGION)
+ client.auto_ml_client.list_datasets.assert_called_with(LOCATION_PATH)
+ assert len(ds) == 1
+ assert ds[0] == "some_dataset"
+
+ def test_get_dataset_no_value(self):
+ dataset_actual = "dataset"
+ client = self.tables_client({}, {})
+ with pytest.raises(ValueError):
+ dataset = client.get_dataset()
+ client.auto_ml_client.get_dataset.assert_not_called()
+
+ def test_get_dataset_name(self):
+ dataset_actual = "dataset"
+ client = self.tables_client({"get_dataset.return_value": dataset_actual}, {})
+ dataset = client.get_dataset(dataset_name="my_dataset")
+ client.auto_ml_client.get_dataset.assert_called_with("my_dataset")
+ assert dataset == dataset_actual
+
+ def test_get_no_dataset(self):
+ client = self.tables_client(
+ {"get_dataset.side_effect": exceptions.NotFound("err")}, {}
+ )
+ with pytest.raises(exceptions.NotFound):
+ client.get_dataset(dataset_name="my_dataset")
+ client.auto_ml_client.get_dataset.assert_called_with("my_dataset")
+
+ def test_get_dataset_from_empty_list(self):
+ client = self.tables_client({"list_datasets.return_value": []}, {})
+ with pytest.raises(exceptions.NotFound):
+ client.get_dataset(dataset_display_name="my_dataset")
+
+ def test_get_dataset_from_list_not_found(self):
+ client = self.tables_client(
+ {"list_datasets.return_value": [mock.Mock(display_name="not_it")]}, {}
+ )
+ with pytest.raises(exceptions.NotFound):
+ client.get_dataset(dataset_display_name="my_dataset")
+
+ def test_get_dataset_from_list(self):
+ client = self.tables_client(
+ {
+ "list_datasets.return_value": [
+ mock.Mock(display_name="not_it"),
+ mock.Mock(display_name="my_dataset"),
+ ]
+ },
+ {},
+ )
+ dataset = client.get_dataset(dataset_display_name="my_dataset")
+ assert dataset.display_name == "my_dataset"
+
+ def test_get_dataset_from_list_ambiguous(self):
+ client = self.tables_client(
+ {
+ "list_datasets.return_value": [
+ mock.Mock(display_name="my_dataset"),
+ mock.Mock(display_name="not_my_dataset"),
+ mock.Mock(display_name="my_dataset"),
+ ]
+ },
+ {},
+ )
+ with pytest.raises(ValueError):
+ client.get_dataset(dataset_display_name="my_dataset")
+
+ def test_create_dataset(self):
+ client = self.tables_client(
+ {
+ "location_path.return_value": LOCATION_PATH,
+ "create_dataset.return_value": mock.Mock(display_name="name"),
+ },
+ {},
+ )
+ metadata = {"metadata": "values"}
+ dataset = client.create_dataset("name", metadata=metadata)
+ client.auto_ml_client.location_path.assert_called_with(PROJECT, REGION)
+ client.auto_ml_client.create_dataset.assert_called_with(
+ LOCATION_PATH, {"display_name": "name", "tables_dataset_metadata": metadata}
+ )
+ assert dataset.display_name == "name"
+
+ def test_delete_dataset(self):
+ dataset = mock.Mock()
+ dataset.configure_mock(name="name")
+ client = self.tables_client({"delete_dataset.return_value": None}, {})
+ client.delete_dataset(dataset=dataset)
+ client.auto_ml_client.delete_dataset.assert_called_with("name")
+
+ def test_delete_dataset_not_found(self):
+ client = self.tables_client({"list_datasets.return_value": []}, {})
+ client.delete_dataset(dataset_display_name="not_found")
+ client.auto_ml_client.delete_dataset.assert_not_called()
+
+ def test_delete_dataset_name(self):
+ client = self.tables_client({"delete_dataset.return_value": None}, {})
+ client.delete_dataset(dataset_name="name")
+ client.auto_ml_client.delete_dataset.assert_called_with("name")
+
+ def test_export_not_found(self):
+ client = self.tables_client({"list_datasets.return_value": []}, {})
+ with pytest.raises(exceptions.NotFound):
+ client.export_data(dataset_display_name="name", gcs_input_uris="uri")
+
+ client.auto_ml_client.export_data.assert_not_called()
+
+ def test_export_gcs_uri(self):
+ client = self.tables_client({"export_data.return_value": None}, {})
+ client.export_data(dataset_name="name", gcs_output_uri_prefix="uri")
+ client.auto_ml_client.export_data.assert_called_with(
+ "name", {"gcs_destination": {"output_uri_prefix": "uri"}}
+ )
+
+ def test_export_bq_uri(self):
+ client = self.tables_client({"export_data.return_value": None}, {})
+ client.export_data(dataset_name="name", bigquery_output_uri="uri")
+ client.auto_ml_client.export_data.assert_called_with(
+ "name", {"bigquery_destination": {"output_uri": "uri"}}
+ )
+
+ def test_import_not_found(self):
+ client = self.tables_client({"list_datasets.return_value": []}, {})
+ with pytest.raises(exceptions.NotFound):
+ client.import_data(dataset_display_name="name", gcs_input_uris="uri")
+
+ client.auto_ml_client.import_data.assert_not_called()
+
+ def test_import_pandas_dataframe(self):
+ client = self.tables_client(
+ gcs_client_attrs={
+ "bucket_name": "my_bucket",
+ "upload_pandas_dataframe.return_value": "uri",
+ }
+ )
+ dataframe = pandas.DataFrame({})
+ client.import_data(
+ project=PROJECT,
+ region=REGION,
+ dataset_name="name",
+ pandas_dataframe=dataframe,
+ )
+ client.gcs_client.ensure_bucket_exists.assert_called_with(PROJECT, REGION)
+ client.gcs_client.upload_pandas_dataframe.assert_called_with(dataframe)
+ client.auto_ml_client.import_data.assert_called_with(
+ "name", {"gcs_source": {"input_uris": ["uri"]}}
+ )
+
+ def test_import_gcs_uri(self):
+ client = self.tables_client({"import_data.return_value": None}, {})
+ client.import_data(dataset_name="name", gcs_input_uris="uri")
+ client.auto_ml_client.import_data.assert_called_with(
+ "name", {"gcs_source": {"input_uris": ["uri"]}}
+ )
+
+ def test_import_gcs_uris(self):
+ client = self.tables_client({"import_data.return_value": None}, {})
+ client.import_data(dataset_name="name", gcs_input_uris=["uri", "uri"])
+ client.auto_ml_client.import_data.assert_called_with(
+ "name", {"gcs_source": {"input_uris": ["uri", "uri"]}}
+ )
+
+ def test_import_bq_uri(self):
+ client = self.tables_client({"import_data.return_value": None}, {})
+ client.import_data(dataset_name="name", bigquery_input_uri="uri")
+ client.auto_ml_client.import_data.assert_called_with(
+ "name", {"bigquery_source": {"input_uri": "uri"}}
+ )
+
+ def test_list_table_specs(self):
+ client = self.tables_client({"list_table_specs.return_value": None}, {})
+ client.list_table_specs(dataset_name="name")
+ client.auto_ml_client.list_table_specs.assert_called_with("name")
+
+ def test_list_table_specs_not_found(self):
+ client = self.tables_client(
+ {"list_table_specs.side_effect": exceptions.NotFound("not found")}, {}
+ )
+ with pytest.raises(exceptions.NotFound):
+ client.list_table_specs(dataset_name="name")
+ client.auto_ml_client.list_table_specs.assert_called_with("name")
+
+ def test_get_table_spec(self):
+ client = self.tables_client({}, {})
+ client.get_table_spec("name")
+ client.auto_ml_client.get_table_spec.assert_called_with("name")
+
+ def test_get_column_spec(self):
+ client = self.tables_client({}, {})
+ client.get_column_spec("name")
+ client.auto_ml_client.get_column_spec.assert_called_with("name")
+
+ def test_list_column_specs(self):
+ table_spec_mock = mock.Mock()
+ # name is reserved in use of __init__, needs to be passed here
+ table_spec_mock.configure_mock(name="table")
+ client = self.tables_client(
+ {
+ "list_table_specs.return_value": [table_spec_mock],
+ "list_column_specs.return_value": [],
+ },
+ {},
+ )
+ client.list_column_specs(dataset_name="name")
+ client.auto_ml_client.list_table_specs.assert_called_with("name")
+ client.auto_ml_client.list_column_specs.assert_called_with("table")
+
+ def test_update_column_spec_not_found(self):
+ table_spec_mock = mock.Mock()
+ # name is reserved in use of __init__, needs to be passed here
+ table_spec_mock.configure_mock(name="table")
+ column_spec_mock = mock.Mock()
+ data_type_mock = mock.Mock(type_code="type_code")
+ column_spec_mock.configure_mock(
+ name="column", display_name="column", data_type=data_type_mock
+ )
+ client = self.tables_client(
+ {
+ "list_table_specs.return_value": [table_spec_mock],
+ "list_column_specs.return_value": [column_spec_mock],
+ },
+ {},
+ )
+ with pytest.raises(exceptions.NotFound):
+ client.update_column_spec(dataset_name="name", column_spec_name="column2")
+ client.auto_ml_client.list_table_specs.assert_called_with("name")
+ client.auto_ml_client.list_column_specs.assert_called_with("table")
+ client.auto_ml_client.update_column_spec.assert_not_called()
+
+ def test_update_column_spec_display_name_not_found(self):
+ table_spec_mock = mock.Mock()
+ # name is reserved in use of __init__, needs to be passed here
+ table_spec_mock.configure_mock(name="table")
+ column_spec_mock = mock.Mock()
+ data_type_mock = mock.Mock(type_code="type_code")
+ column_spec_mock.configure_mock(
+ name="column", display_name="column", data_type=data_type_mock
+ )
+ client = self.tables_client(
+ {
+ "list_table_specs.return_value": [table_spec_mock],
+ "list_column_specs.return_value": [column_spec_mock],
+ },
+ {},
+ )
+ with pytest.raises(exceptions.NotFound):
+ client.update_column_spec(
+ dataset_name="name", column_spec_display_name="column2"
+ )
+ client.auto_ml_client.list_table_specs.assert_called_with("name")
+ client.auto_ml_client.list_column_specs.assert_called_with("table")
+ client.auto_ml_client.update_column_spec.assert_not_called()
+
+ def test_update_column_spec_name_no_args(self):
+ table_spec_mock = mock.Mock()
+ # name is reserved in use of __init__, needs to be passed here
+ table_spec_mock.configure_mock(name="table")
+ column_spec_mock = mock.Mock()
+ data_type_mock = mock.Mock(type_code="type_code")
+ column_spec_mock.configure_mock(
+ name="column/2", display_name="column", data_type=data_type_mock
+ )
+ client = self.tables_client(
+ {
+ "list_table_specs.return_value": [table_spec_mock],
+ "list_column_specs.return_value": [column_spec_mock],
+ },
+ {},
+ )
+ client.update_column_spec(dataset_name="name", column_spec_name="column/2")
+ client.auto_ml_client.list_table_specs.assert_called_with("name")
+ client.auto_ml_client.list_column_specs.assert_called_with("table")
+ client.auto_ml_client.update_column_spec.assert_called_with(
+ {"name": "column/2", "data_type": {"type_code": "type_code"}}
+ )
+
+ def test_update_column_spec_no_args(self):
+ table_spec_mock = mock.Mock()
+ # name is reserved in use of __init__, needs to be passed here
+ table_spec_mock.configure_mock(name="table")
+ column_spec_mock = mock.Mock()
+ data_type_mock = mock.Mock(type_code="type_code")
+ column_spec_mock.configure_mock(
+ name="column", display_name="column", data_type=data_type_mock
+ )
+ client = self.tables_client(
+ {
+ "list_table_specs.return_value": [table_spec_mock],
+ "list_column_specs.return_value": [column_spec_mock],
+ },
+ {},
+ )
+ client.update_column_spec(
+ dataset_name="name", column_spec_display_name="column"
+ )
+ client.auto_ml_client.list_table_specs.assert_called_with("name")
+ client.auto_ml_client.list_column_specs.assert_called_with("table")
+ client.auto_ml_client.update_column_spec.assert_called_with(
+ {"name": "column", "data_type": {"type_code": "type_code"}}
+ )
+
+ def test_update_column_spec_nullable(self):
+ table_spec_mock = mock.Mock()
+ # name is reserved in use of __init__, needs to be passed here
+ table_spec_mock.configure_mock(name="table")
+ column_spec_mock = mock.Mock()
+ data_type_mock = mock.Mock(type_code="type_code")
+ column_spec_mock.configure_mock(
+ name="column", display_name="column", data_type=data_type_mock
+ )
+ client = self.tables_client(
+ {
+ "list_table_specs.return_value": [table_spec_mock],
+ "list_column_specs.return_value": [column_spec_mock],
+ },
+ {},
+ )
+ client.update_column_spec(
+ dataset_name="name", column_spec_display_name="column", nullable=True
+ )
+ client.auto_ml_client.list_table_specs.assert_called_with("name")
+ client.auto_ml_client.list_column_specs.assert_called_with("table")
+ client.auto_ml_client.update_column_spec.assert_called_with(
+ {
+ "name": "column",
+ "data_type": {"type_code": "type_code", "nullable": True},
+ }
+ )
+
+ def test_update_column_spec_type_code(self):
+ table_spec_mock = mock.Mock()
+ # name is reserved in use of __init__, needs to be passed here
+ table_spec_mock.configure_mock(name="table")
+ column_spec_mock = mock.Mock()
+ data_type_mock = mock.Mock(type_code="type_code")
+ column_spec_mock.configure_mock(
+ name="column", display_name="column", data_type=data_type_mock
+ )
+ client = self.tables_client(
+ {
+ "list_table_specs.return_value": [table_spec_mock],
+ "list_column_specs.return_value": [column_spec_mock],
+ },
+ {},
+ )
+ client.update_column_spec(
+ dataset_name="name",
+ column_spec_display_name="column",
+ type_code="type_code2",
+ )
+ client.auto_ml_client.list_table_specs.assert_called_with("name")
+ client.auto_ml_client.list_column_specs.assert_called_with("table")
+ client.auto_ml_client.update_column_spec.assert_called_with(
+ {"name": "column", "data_type": {"type_code": "type_code2"}}
+ )
+
+ def test_update_column_spec_type_code_nullable(self):
+ table_spec_mock = mock.Mock()
+ # name is reserved in use of __init__, needs to be passed here
+ table_spec_mock.configure_mock(name="table")
+ column_spec_mock = mock.Mock()
+ data_type_mock = mock.Mock(type_code="type_code")
+ column_spec_mock.configure_mock(
+ name="column", display_name="column", data_type=data_type_mock
+ )
+ client = self.tables_client(
+ {
+ "list_table_specs.return_value": [table_spec_mock],
+ "list_column_specs.return_value": [column_spec_mock],
+ },
+ {},
+ )
+ client.update_column_spec(
+ dataset_name="name",
+ nullable=True,
+ column_spec_display_name="column",
+ type_code="type_code2",
+ )
+ client.auto_ml_client.list_table_specs.assert_called_with("name")
+ client.auto_ml_client.list_column_specs.assert_called_with("table")
+ client.auto_ml_client.update_column_spec.assert_called_with(
+ {
+ "name": "column",
+ "data_type": {"type_code": "type_code2", "nullable": True},
+ }
+ )
+
+ def test_update_column_spec_type_code_nullable_false(self):
+ table_spec_mock = mock.Mock()
+ # name is reserved in use of __init__, needs to be passed here
+ table_spec_mock.configure_mock(name="table")
+ column_spec_mock = mock.Mock()
+ data_type_mock = mock.Mock(type_code="type_code")
+ column_spec_mock.configure_mock(
+ name="column", display_name="column", data_type=data_type_mock
+ )
+ client = self.tables_client(
+ {
+ "list_table_specs.return_value": [table_spec_mock],
+ "list_column_specs.return_value": [column_spec_mock],
+ },
+ {},
+ )
+ client.update_column_spec(
+ dataset_name="name",
+ nullable=False,
+ column_spec_display_name="column",
+ type_code="type_code2",
+ )
+ client.auto_ml_client.list_table_specs.assert_called_with("name")
+ client.auto_ml_client.list_column_specs.assert_called_with("table")
+ client.auto_ml_client.update_column_spec.assert_called_with(
+ {
+ "name": "column",
+ "data_type": {"type_code": "type_code2", "nullable": False},
+ }
+ )
+
+ def test_set_target_column_table_not_found(self):
+ client = self.tables_client(
+ {"list_table_specs.side_effect": exceptions.NotFound("err")}, {}
+ )
+ with pytest.raises(exceptions.NotFound):
+ client.set_target_column(
+ dataset_name="name", column_spec_display_name="column2"
+ )
+ client.auto_ml_client.list_table_specs.assert_called_with("name")
+ client.auto_ml_client.list_column_specs.assert_not_called()
+ client.auto_ml_client.update_dataset.assert_not_called()
+
+ def test_set_target_column_not_found(self):
+ table_spec_mock = mock.Mock()
+ # name is reserved in use of __init__, needs to be passed here
+ table_spec_mock.configure_mock(name="table")
+ column_spec_mock = mock.Mock()
+ column_spec_mock.configure_mock(name="column/1", display_name="column")
+ client = self.tables_client(
+ {
+ "list_table_specs.return_value": [table_spec_mock],
+ "list_column_specs.return_value": [column_spec_mock],
+ },
+ {},
+ )
+ with pytest.raises(exceptions.NotFound):
+ client.set_target_column(
+ dataset_name="name", column_spec_display_name="column2"
+ )
+ client.auto_ml_client.list_table_specs.assert_called_with("name")
+ client.auto_ml_client.list_column_specs.assert_called_with("table")
+ client.auto_ml_client.update_dataset.assert_not_called()
+
+ def test_set_target_column(self):
+ table_spec_mock = mock.Mock()
+ # name is reserved in use of __init__, needs to be passed here
+ table_spec_mock.configure_mock(name="table")
+ column_spec_mock = mock.Mock()
+ column_spec_mock.configure_mock(name="column/1", display_name="column")
+ dataset_mock = mock.Mock()
+ tables_dataset_metadata_mock = mock.Mock()
+ tables_dataset_metadata_mock.configure_mock(
+ target_column_spec_id="2",
+ weight_column_spec_id="2",
+ ml_use_column_spec_id="3",
+ )
+ dataset_mock.configure_mock(
+ name="dataset", tables_dataset_metadata=tables_dataset_metadata_mock
+ )
+ client = self.tables_client(
+ {
+ "get_dataset.return_value": dataset_mock,
+ "list_table_specs.return_value": [table_spec_mock],
+ "list_column_specs.return_value": [column_spec_mock],
+ },
+ {},
+ )
+ client.set_target_column(dataset_name="name", column_spec_display_name="column")
+ client.auto_ml_client.list_table_specs.assert_called_with("name")
+ client.auto_ml_client.list_column_specs.assert_called_with("table")
+ client.auto_ml_client.update_dataset.assert_called_with(
+ {
+ "name": "dataset",
+ "tables_dataset_metadata": {
+ "target_column_spec_id": "1",
+ "weight_column_spec_id": "2",
+ "ml_use_column_spec_id": "3",
+ },
+ }
+ )
+
+ def test_set_weight_column_table_not_found(self):
+ client = self.tables_client(
+ {"list_table_specs.side_effect": exceptions.NotFound("err")}, {}
+ )
+ try:
+ client.set_weight_column(
+ dataset_name="name", column_spec_display_name="column2"
+ )
+ except exceptions.NotFound:
+ pass
+ client.auto_ml_client.list_table_specs.assert_called_with("name")
+ client.auto_ml_client.list_column_specs.assert_not_called()
+ client.auto_ml_client.update_dataset.assert_not_called()
+
+ def test_set_weight_column_not_found(self):
+ table_spec_mock = mock.Mock()
+ # name is reserved in use of __init__, needs to be passed here
+ table_spec_mock.configure_mock(name="table")
+ column_spec_mock = mock.Mock()
+ column_spec_mock.configure_mock(name="column/1", display_name="column")
+ client = self.tables_client(
+ {
+ "list_table_specs.return_value": [table_spec_mock],
+ "list_column_specs.return_value": [column_spec_mock],
+ },
+ {},
+ )
+ with pytest.raises(exceptions.NotFound):
+ client.set_weight_column(
+ dataset_name="name", column_spec_display_name="column2"
+ )
+ client.auto_ml_client.list_table_specs.assert_called_with("name")
+ client.auto_ml_client.list_column_specs.assert_called_with("table")
+ client.auto_ml_client.update_dataset.assert_not_called()
+
+ def test_set_weight_column(self):
+ table_spec_mock = mock.Mock()
+ # name is reserved in use of __init__, needs to be passed here
+ table_spec_mock.configure_mock(name="table")
+ column_spec_mock = mock.Mock()
+ column_spec_mock.configure_mock(name="column/2", display_name="column")
+ dataset_mock = mock.Mock()
+ tables_dataset_metadata_mock = mock.Mock()
+ tables_dataset_metadata_mock.configure_mock(
+ target_column_spec_id="1",
+ weight_column_spec_id="1",
+ ml_use_column_spec_id="3",
+ )
+ dataset_mock.configure_mock(
+ name="dataset", tables_dataset_metadata=tables_dataset_metadata_mock
+ )
+ client = self.tables_client(
+ {
+ "get_dataset.return_value": dataset_mock,
+ "list_table_specs.return_value": [table_spec_mock],
+ "list_column_specs.return_value": [column_spec_mock],
+ },
+ {},
+ )
+ client.set_weight_column(dataset_name="name", column_spec_display_name="column")
+ client.auto_ml_client.list_table_specs.assert_called_with("name")
+ client.auto_ml_client.list_column_specs.assert_called_with("table")
+ client.auto_ml_client.update_dataset.assert_called_with(
+ {
+ "name": "dataset",
+ "tables_dataset_metadata": {
+ "target_column_spec_id": "1",
+ "weight_column_spec_id": "2",
+ "ml_use_column_spec_id": "3",
+ },
+ }
+ )
+
+ def test_clear_weight_column(self):
+ dataset_mock = mock.Mock()
+ tables_dataset_metadata_mock = mock.Mock()
+ tables_dataset_metadata_mock.configure_mock(
+ target_column_spec_id="1",
+ weight_column_spec_id="2",
+ ml_use_column_spec_id="3",
+ )
+ dataset_mock.configure_mock(
+ name="dataset", tables_dataset_metadata=tables_dataset_metadata_mock
+ )
+ client = self.tables_client({"get_dataset.return_value": dataset_mock}, {})
+ client.clear_weight_column(dataset_name="name")
+ client.auto_ml_client.update_dataset.assert_called_with(
+ {
+ "name": "dataset",
+ "tables_dataset_metadata": {
+ "target_column_spec_id": "1",
+ "weight_column_spec_id": None,
+ "ml_use_column_spec_id": "3",
+ },
+ }
+ )
+
+ def test_set_test_train_column_table_not_found(self):
+ client = self.tables_client(
+ {"list_table_specs.side_effect": exceptions.NotFound("err")}, {}
+ )
+ with pytest.raises(exceptions.NotFound):
+ client.set_test_train_column(
+ dataset_name="name", column_spec_display_name="column2"
+ )
+ client.auto_ml_client.list_table_specs.assert_called_with("name")
+ client.auto_ml_client.list_column_specs.assert_not_called()
+ client.auto_ml_client.update_dataset.assert_not_called()
+
+ def test_set_test_train_column_not_found(self):
+ table_spec_mock = mock.Mock()
+ # name is reserved in use of __init__, needs to be passed here
+ table_spec_mock.configure_mock(name="table")
+ column_spec_mock = mock.Mock()
+ column_spec_mock.configure_mock(name="column/1", display_name="column")
+ client = self.tables_client(
+ {
+ "list_table_specs.return_value": [table_spec_mock],
+ "list_column_specs.return_value": [column_spec_mock],
+ },
+ {},
+ )
+ with pytest.raises(exceptions.NotFound):
+ client.set_test_train_column(
+ dataset_name="name", column_spec_display_name="column2"
+ )
+ client.auto_ml_client.list_table_specs.assert_called_with("name")
+ client.auto_ml_client.list_column_specs.assert_called_with("table")
+ client.auto_ml_client.update_dataset.assert_not_called()
+
+ def test_set_test_train_column(self):
+ table_spec_mock = mock.Mock()
+ # name is reserved in use of __init__, needs to be passed here
+ table_spec_mock.configure_mock(name="table")
+ column_spec_mock = mock.Mock()
+ column_spec_mock.configure_mock(name="column/3", display_name="column")
+ dataset_mock = mock.Mock()
+ tables_dataset_metadata_mock = mock.Mock()
+ tables_dataset_metadata_mock.configure_mock(
+ target_column_spec_id="1",
+ weight_column_spec_id="2",
+ ml_use_column_spec_id="2",
+ )
+ dataset_mock.configure_mock(
+ name="dataset", tables_dataset_metadata=tables_dataset_metadata_mock
+ )
+ client = self.tables_client(
+ {
+ "get_dataset.return_value": dataset_mock,
+ "list_table_specs.return_value": [table_spec_mock],
+ "list_column_specs.return_value": [column_spec_mock],
+ },
+ {},
+ )
+ client.set_test_train_column(
+ dataset_name="name", column_spec_display_name="column"
+ )
+ client.auto_ml_client.list_table_specs.assert_called_with("name")
+ client.auto_ml_client.list_column_specs.assert_called_with("table")
+ client.auto_ml_client.update_dataset.assert_called_with(
+ {
+ "name": "dataset",
+ "tables_dataset_metadata": {
+ "target_column_spec_id": "1",
+ "weight_column_spec_id": "2",
+ "ml_use_column_spec_id": "3",
+ },
+ }
+ )
+
+ def test_clear_test_train_column(self):
+ dataset_mock = mock.Mock()
+ tables_dataset_metadata_mock = mock.Mock()
+ tables_dataset_metadata_mock.configure_mock(
+ target_column_spec_id="1",
+ weight_column_spec_id="2",
+ ml_use_column_spec_id="2",
+ )
+ dataset_mock.configure_mock(
+ name="dataset", tables_dataset_metadata=tables_dataset_metadata_mock
+ )
+ client = self.tables_client({"get_dataset.return_value": dataset_mock}, {})
+ client.clear_test_train_column(dataset_name="name")
+ client.auto_ml_client.update_dataset.assert_called_with(
+ {
+ "name": "dataset",
+ "tables_dataset_metadata": {
+ "target_column_spec_id": "1",
+ "weight_column_spec_id": "2",
+ "ml_use_column_spec_id": None,
+ },
+ }
+ )
+
+ def test_set_time_column(self):
+ table_spec_mock = mock.Mock()
+ # name is reserved in use of __init__, needs to be passed here
+ table_spec_mock.configure_mock(name="table")
+ column_spec_mock = mock.Mock()
+ column_spec_mock.configure_mock(name="column/3", display_name="column")
+ dataset_mock = mock.Mock()
+ dataset_mock.configure_mock(name="dataset")
+ client = self.tables_client(
+ {
+ "get_dataset.return_value": dataset_mock,
+ "list_table_specs.return_value": [table_spec_mock],
+ "list_column_specs.return_value": [column_spec_mock],
+ },
+ {},
+ )
+ client.set_time_column(dataset_name="name", column_spec_display_name="column")
+ client.auto_ml_client.list_table_specs.assert_called_with("name")
+ client.auto_ml_client.list_column_specs.assert_called_with("table")
+ client.auto_ml_client.update_table_spec.assert_called_with(
+ {"name": "table", "time_column_spec_id": "3"}
+ )
+
+ def test_clear_time_column(self):
+ table_spec_mock = mock.Mock()
+ # name is reserved in use of __init__, needs to be passed here
+ table_spec_mock.configure_mock(name="table")
+ dataset_mock = mock.Mock()
+ dataset_mock.configure_mock(name="dataset")
+ client = self.tables_client(
+ {
+ "get_dataset.return_value": dataset_mock,
+ "list_table_specs.return_value": [table_spec_mock],
+ },
+ {},
+ )
+ client.clear_time_column(dataset_name="name")
+ client.auto_ml_client.update_table_spec.assert_called_with(
+ {"name": "table", "time_column_spec_id": None}
+ )
+
+ def test_get_model_evaluation(self):
+ client = self.tables_client({}, {})
+ ds = client.get_model_evaluation(model_evaluation_name="x")
+ client.auto_ml_client.get_model_evaluation.assert_called_with("x")
+
+ def test_list_model_evaluations_empty(self):
+ client = self.tables_client({"list_model_evaluations.return_value": []}, {})
+ ds = client.list_model_evaluations(model_name="model")
+ client.auto_ml_client.list_model_evaluations.assert_called_with("model")
+ assert ds == []
+
+ def test_list_model_evaluations_not_empty(self):
+ evaluations = ["eval"]
+ client = self.tables_client(
+ {
+ "list_model_evaluations.return_value": evaluations,
+ "location_path.return_value": LOCATION_PATH,
+ },
+ {},
+ )
+ ds = client.list_model_evaluations(model_name="model")
+ client.auto_ml_client.list_model_evaluations.assert_called_with("model")
+ assert len(ds) == 1
+ assert ds[0] == "eval"
+
+ def test_list_models_empty(self):
+ client = self.tables_client(
+ {
+ "list_models.return_value": [],
+ "location_path.return_value": LOCATION_PATH,
+ },
+ {},
+ )
+ ds = client.list_models()
+ client.auto_ml_client.location_path.assert_called_with(PROJECT, REGION)
+ client.auto_ml_client.list_models.assert_called_with(LOCATION_PATH)
+ assert ds == []
+
+ def test_list_models_not_empty(self):
+ models = ["some_model"]
+ client = self.tables_client(
+ {
+ "list_models.return_value": models,
+ "location_path.return_value": LOCATION_PATH,
+ },
+ {},
+ )
+ ds = client.list_models()
+ client.auto_ml_client.location_path.assert_called_with(PROJECT, REGION)
+ client.auto_ml_client.list_models.assert_called_with(LOCATION_PATH)
+ assert len(ds) == 1
+ assert ds[0] == "some_model"
+
+ def test_get_model_name(self):
+ model_actual = "model"
+ client = self.tables_client({"get_model.return_value": model_actual}, {})
+ model = client.get_model(model_name="my_model")
+ client.auto_ml_client.get_model.assert_called_with("my_model")
+ assert model == model_actual
+
+ def test_get_no_model(self):
+ client = self.tables_client(
+ {"get_model.side_effect": exceptions.NotFound("err")}, {}
+ )
+ with pytest.raises(exceptions.NotFound):
+ client.get_model(model_name="my_model")
+ client.auto_ml_client.get_model.assert_called_with("my_model")
+
+ def test_get_model_from_empty_list(self):
+ client = self.tables_client({"list_models.return_value": []}, {})
+ with pytest.raises(exceptions.NotFound):
+ client.get_model(model_display_name="my_model")
+
+ def test_get_model_from_list_not_found(self):
+ client = self.tables_client(
+ {"list_models.return_value": [mock.Mock(display_name="not_it")]}, {}
+ )
+ with pytest.raises(exceptions.NotFound):
+ client.get_model(model_display_name="my_model")
+
+ def test_get_model_from_list(self):
+ client = self.tables_client(
+ {
+ "list_models.return_value": [
+ mock.Mock(display_name="not_it"),
+ mock.Mock(display_name="my_model"),
+ ]
+ },
+ {},
+ )
+ model = client.get_model(model_display_name="my_model")
+ assert model.display_name == "my_model"
+
+ def test_get_model_from_list_ambiguous(self):
+ client = self.tables_client(
+ {
+ "list_models.return_value": [
+ mock.Mock(display_name="my_model"),
+ mock.Mock(display_name="not_my_model"),
+ mock.Mock(display_name="my_model"),
+ ]
+ },
+ {},
+ )
+ with pytest.raises(ValueError):
+ client.get_model(model_display_name="my_model")
+
+ def test_delete_model(self):
+ model = mock.Mock()
+ model.configure_mock(name="name")
+ client = self.tables_client({"delete_model.return_value": None}, {})
+ client.delete_model(model=model)
+ client.auto_ml_client.delete_model.assert_called_with("name")
+
+ def test_delete_model_not_found(self):
+ client = self.tables_client({"list_models.return_value": []}, {})
+ client.delete_model(model_display_name="not_found")
+ client.auto_ml_client.delete_model.assert_not_called()
+
+ def test_delete_model_name(self):
+ client = self.tables_client({"delete_model.return_value": None}, {})
+ client.delete_model(model_name="name")
+ client.auto_ml_client.delete_model.assert_called_with("name")
+
+ def test_deploy_model_no_args(self):
+ client = self.tables_client({}, {})
+ with pytest.raises(ValueError):
+ client.deploy_model()
+ client.auto_ml_client.deploy_model.assert_not_called()
+
+ def test_deploy_model(self):
+ client = self.tables_client({}, {})
+ client.deploy_model(model_name="name")
+ client.auto_ml_client.deploy_model.assert_called_with("name")
+
+ def test_deploy_model_not_found(self):
+ client = self.tables_client({"list_models.return_value": []}, {})
+ with pytest.raises(exceptions.NotFound):
+ client.deploy_model(model_display_name="name")
+ client.auto_ml_client.deploy_model.assert_not_called()
+
+ def test_undeploy_model(self):
+ client = self.tables_client({}, {})
+ client.undeploy_model(model_name="name")
+ client.auto_ml_client.undeploy_model.assert_called_with("name")
+
+ def test_undeploy_model_not_found(self):
+ client = self.tables_client({"list_models.return_value": []}, {})
+ with pytest.raises(exceptions.NotFound):
+ client.undeploy_model(model_display_name="name")
+ client.auto_ml_client.undeploy_model.assert_not_called()
+
+ def test_create_model(self):
+ table_spec_mock = mock.Mock()
+ # name is reserved in use of __init__, needs to be passed here
+ table_spec_mock.configure_mock(name="table")
+ column_spec_mock = mock.Mock()
+ column_spec_mock.configure_mock(name="column/2", display_name="column")
+ client = self.tables_client(
+ {
+ "list_table_specs.return_value": [table_spec_mock],
+ "list_column_specs.return_value": [column_spec_mock],
+ "location_path.return_value": LOCATION_PATH,
+ },
+ {},
+ )
+ client.create_model(
+ "my_model", dataset_name="my_dataset", train_budget_milli_node_hours=1000
+ )
+ client.auto_ml_client.create_model.assert_called_with(
+ LOCATION_PATH,
+ {
+ "display_name": "my_model",
+ "dataset_id": "my_dataset",
+ "tables_model_metadata": {"train_budget_milli_node_hours": 1000},
+ },
+ )
+
+ def test_create_model_include_columns(self):
+ table_spec_mock = mock.Mock()
+ # name is reserved in use of __init__, needs to be passed here
+ table_spec_mock.configure_mock(name="table")
+ column_spec_mock1 = mock.Mock()
+ column_spec_mock1.configure_mock(name="column/1", display_name="column1")
+ column_spec_mock2 = mock.Mock()
+ column_spec_mock2.configure_mock(name="column/2", display_name="column2")
+ client = self.tables_client(
+ {
+ "list_table_specs.return_value": [table_spec_mock],
+ "list_column_specs.return_value": [
+ column_spec_mock1,
+ column_spec_mock2,
+ ],
+ "location_path.return_value": LOCATION_PATH,
+ },
+ {},
+ )
+ client.create_model(
+ "my_model",
+ dataset_name="my_dataset",
+ include_column_spec_names=["column1"],
+ train_budget_milli_node_hours=1000,
+ )
+ client.auto_ml_client.create_model.assert_called_with(
+ LOCATION_PATH,
+ {
+ "display_name": "my_model",
+ "dataset_id": "my_dataset",
+ "tables_model_metadata": {
+ "train_budget_milli_node_hours": 1000,
+ "input_feature_column_specs": [column_spec_mock1],
+ },
+ },
+ )
+
+ def test_create_model_exclude_columns(self):
+ table_spec_mock = mock.Mock()
+ # name is reserved in use of __init__, needs to be passed here
+ table_spec_mock.configure_mock(name="table")
+ column_spec_mock1 = mock.Mock()
+ column_spec_mock1.configure_mock(name="column/1", display_name="column1")
+ column_spec_mock2 = mock.Mock()
+ column_spec_mock2.configure_mock(name="column/2", display_name="column2")
+ client = self.tables_client(
+ {
+ "list_table_specs.return_value": [table_spec_mock],
+ "list_column_specs.return_value": [
+ column_spec_mock1,
+ column_spec_mock2,
+ ],
+ "location_path.return_value": LOCATION_PATH,
+ },
+ {},
+ )
+ client.create_model(
+ "my_model",
+ dataset_name="my_dataset",
+ exclude_column_spec_names=["column1"],
+ train_budget_milli_node_hours=1000,
+ )
+ client.auto_ml_client.create_model.assert_called_with(
+ LOCATION_PATH,
+ {
+ "display_name": "my_model",
+ "dataset_id": "my_dataset",
+ "tables_model_metadata": {
+ "train_budget_milli_node_hours": 1000,
+ "input_feature_column_specs": [column_spec_mock2],
+ },
+ },
+ )
+
+ def test_create_model_invalid_hours_small(self):
+ client = self.tables_client({}, {})
+ with pytest.raises(ValueError):
+ client.create_model(
+ "my_model", dataset_name="my_dataset", train_budget_milli_node_hours=1
+ )
+ client.auto_ml_client.create_model.assert_not_called()
+
+ def test_create_model_invalid_hours_large(self):
+ client = self.tables_client({}, {})
+ with pytest.raises(ValueError):
+ client.create_model(
+ "my_model",
+ dataset_name="my_dataset",
+ train_budget_milli_node_hours=1000000,
+ )
+ client.auto_ml_client.create_model.assert_not_called()
+
+ def test_create_model_invalid_no_dataset(self):
+ client = self.tables_client({}, {})
+ with pytest.raises(ValueError):
+ client.create_model("my_model", train_budget_milli_node_hours=1000)
+ client.auto_ml_client.get_dataset.assert_not_called()
+ client.auto_ml_client.create_model.assert_not_called()
+
+ def test_create_model_invalid_include_exclude(self):
+ client = self.tables_client({}, {})
+ with pytest.raises(ValueError):
+ client.create_model(
+ "my_model",
+ dataset_name="my_dataset",
+ include_column_spec_names=["a"],
+ exclude_column_spec_names=["b"],
+ train_budget_milli_node_hours=1000,
+ )
+ client.auto_ml_client.get_dataset.assert_not_called()
+ client.auto_ml_client.create_model.assert_not_called()
+
+ def test_predict_from_array(self):
+ data_type = mock.Mock(type_code=data_types_pb2.CATEGORY)
+ column_spec = mock.Mock(display_name="a", data_type=data_type)
+ model_metadata = mock.Mock(input_feature_column_specs=[column_spec])
+ model = mock.Mock()
+ model.configure_mock(tables_model_metadata=model_metadata, name="my_model")
+ client = self.tables_client({"get_model.return_value": model}, {})
+ client.predict(["1"], model_name="my_model")
+ client.prediction_client.predict.assert_called_with(
+ "my_model", {"row": {"values": [{"string_value": "1"}]}}
+ )
+
+ def test_predict_from_dict(self):
+ data_type = mock.Mock(type_code=data_types_pb2.CATEGORY)
+ column_spec_a = mock.Mock(display_name="a", data_type=data_type)
+ column_spec_b = mock.Mock(display_name="b", data_type=data_type)
+ model_metadata = mock.Mock(
+ input_feature_column_specs=[column_spec_a, column_spec_b]
+ )
+ model = mock.Mock()
+ model.configure_mock(tables_model_metadata=model_metadata, name="my_model")
+ client = self.tables_client({"get_model.return_value": model}, {})
+ client.predict({"a": "1", "b": "2"}, model_name="my_model")
+ client.prediction_client.predict.assert_called_with(
+ "my_model",
+ {"row": {"values": [{"string_value": "1"}, {"string_value": "2"}]}},
+ )
+
+ def test_predict_from_dict_missing(self):
+ data_type = mock.Mock(type_code=data_types_pb2.CATEGORY)
+ column_spec_a = mock.Mock(display_name="a", data_type=data_type)
+ column_spec_b = mock.Mock(display_name="b", data_type=data_type)
+ model_metadata = mock.Mock(
+ input_feature_column_specs=[column_spec_a, column_spec_b]
+ )
+ model = mock.Mock()
+ model.configure_mock(tables_model_metadata=model_metadata, name="my_model")
+ client = self.tables_client({"get_model.return_value": model}, {})
+ client.predict({"a": "1"}, model_name="my_model")
+ client.prediction_client.predict.assert_called_with(
+ "my_model", {"row": {"values": [{"string_value": "1"}, {"null_value": 0}]}}
+ )
+
+ def test_predict_all_types(self):
+ float_type = mock.Mock(type_code=data_types_pb2.FLOAT64)
+ timestamp_type = mock.Mock(type_code=data_types_pb2.TIMESTAMP)
+ string_type = mock.Mock(type_code=data_types_pb2.STRING)
+ array_type = mock.Mock(type_code=data_types_pb2.ARRAY)
+ struct_type = mock.Mock(type_code=data_types_pb2.STRUCT)
+ category_type = mock.Mock(type_code=data_types_pb2.CATEGORY)
+ column_spec_float = mock.Mock(display_name="float", data_type=float_type)
+ column_spec_timestamp = mock.Mock(
+ display_name="timestamp", data_type=timestamp_type
+ )
+ column_spec_string = mock.Mock(display_name="string", data_type=string_type)
+ column_spec_array = mock.Mock(display_name="array", data_type=array_type)
+ column_spec_struct = mock.Mock(display_name="struct", data_type=struct_type)
+ column_spec_category = mock.Mock(
+ display_name="category", data_type=category_type
+ )
+ column_spec_null = mock.Mock(display_name="null", data_type=category_type)
+ model_metadata = mock.Mock(
+ input_feature_column_specs=[
+ column_spec_float,
+ column_spec_timestamp,
+ column_spec_string,
+ column_spec_array,
+ column_spec_struct,
+ column_spec_category,
+ column_spec_null,
+ ]
+ )
+ model = mock.Mock()
+ model.configure_mock(tables_model_metadata=model_metadata, name="my_model")
+ client = self.tables_client({"get_model.return_value": model}, {})
+ client.predict(
+ {
+ "float": 1.0,
+ "timestamp": "EST",
+ "string": "text",
+ "array": [1],
+ "struct": {"a": "b"},
+ "category": "a",
+ "null": None,
+ },
+ model_name="my_model",
+ )
+ client.prediction_client.predict.assert_called_with(
+ "my_model",
+ {
+ "row": {
+ "values": [
+ {"number_value": 1.0},
+ {"string_value": "EST"},
+ {"string_value": "text"},
+ {"list_value": [1]},
+ {"struct_value": {"a": "b"}},
+ {"string_value": "a"},
+ {"null_value": 0},
+ ]
+ }
+ },
+ )
+
+ def test_predict_from_array_missing(self):
+ data_type = mock.Mock(type_code=data_types_pb2.CATEGORY)
+ column_spec = mock.Mock(display_name="a", data_type=data_type)
+ model_metadata = mock.Mock(input_feature_column_specs=[column_spec])
+ model = mock.Mock()
+ model.configure_mock(tables_model_metadata=model_metadata, name="my_model")
+ client = self.tables_client({"get_model.return_value": model}, {})
+ with pytest.raises(ValueError):
+ client.predict([], model_name="my_model")
+ client.prediction_client.predict.assert_not_called()
+
+ def test_batch_predict_pandas_dataframe(self):
+ client = self.tables_client(
+ gcs_client_attrs={
+ "bucket_name": "my_bucket",
+ "upload_pandas_dataframe.return_value": "gs://input",
+ }
+ )
+ dataframe = pandas.DataFrame({})
+ client.batch_predict(
+ project=PROJECT,
+ region=REGION,
+ model_name="my_model",
+ pandas_dataframe=dataframe,
+ gcs_output_uri_prefix="gs://output",
+ )
+
+ client.gcs_client.ensure_bucket_exists.assert_called_with(PROJECT, REGION)
+ client.gcs_client.upload_pandas_dataframe.assert_called_with(dataframe)
+
+ client.prediction_client.batch_predict.assert_called_with(
+ "my_model",
+ {"gcs_source": {"input_uris": ["gs://input"]}},
+ {"gcs_destination": {"output_uri_prefix": "gs://output"}},
+ )
+
+ def test_batch_predict_gcs(self):
+ client = self.tables_client({}, {})
+ client.batch_predict(
+ model_name="my_model",
+ gcs_input_uris="gs://input",
+ gcs_output_uri_prefix="gs://output",
+ )
+ client.prediction_client.batch_predict.assert_called_with(
+ "my_model",
+ {"gcs_source": {"input_uris": ["gs://input"]}},
+ {"gcs_destination": {"output_uri_prefix": "gs://output"}},
+ )
+
+ def test_batch_predict_bigquery(self):
+ client = self.tables_client({}, {})
+ client.batch_predict(
+ model_name="my_model",
+ bigquery_input_uri="bq://input",
+ bigquery_output_uri="bq://output",
+ )
+ client.prediction_client.batch_predict.assert_called_with(
+ "my_model",
+ {"bigquery_source": {"input_uri": "bq://input"}},
+ {"bigquery_destination": {"output_uri": "bq://output"}},
+ )
+
+ def test_batch_predict_mixed(self):
+ client = self.tables_client({}, {})
+ client.batch_predict(
+ model_name="my_model",
+ gcs_input_uris="gs://input",
+ bigquery_output_uri="bq://output",
+ )
+ client.prediction_client.batch_predict.assert_called_with(
+ "my_model",
+ {"gcs_source": {"input_uris": ["gs://input"]}},
+ {"bigquery_destination": {"output_uri": "bq://output"}},
+ )
+
+ def test_batch_predict_missing_input_gcs_uri(self):
+ client = self.tables_client({}, {})
+ with pytest.raises(ValueError):
+ client.batch_predict(
+ model_name="my_model",
+ gcs_input_uris=None,
+ gcs_output_uri_prefix="gs://output",
+ )
+ client.prediction_client.batch_predict.assert_not_called()
+
+ def test_batch_predict_missing_input_bigquery_uri(self):
+ client = self.tables_client({}, {})
+ with pytest.raises(ValueError):
+ client.batch_predict(
+ model_name="my_model",
+ bigquery_input_uri=None,
+ gcs_output_uri_prefix="gs://output",
+ )
+ client.prediction_client.batch_predict.assert_not_called()
+
+ def test_batch_predict_missing_output_gcs_uri(self):
+ client = self.tables_client({}, {})
+ with pytest.raises(ValueError):
+ client.batch_predict(
+ model_name="my_model",
+ gcs_input_uris="gs://input",
+ gcs_output_uri_prefix=None,
+ )
+ client.prediction_client.batch_predict.assert_not_called()
+
+ def test_batch_predict_missing_output_bigquery_uri(self):
+ client = self.tables_client({}, {})
+ with pytest.raises(ValueError):
+ client.batch_predict(
+ model_name="my_model",
+ gcs_input_uris="gs://input",
+ bigquery_output_uri=None,
+ )
+ client.prediction_client.batch_predict.assert_not_called()
+
+ def test_batch_predict_missing_model(self):
+ client = self.tables_client({"list_models.return_value": []}, {})
+ with pytest.raises(exceptions.NotFound):
+ client.batch_predict(
+ model_display_name="my_model",
+ gcs_input_uris="gs://input",
+ gcs_output_uri_prefix="gs://output",
+ )
+ client.prediction_client.batch_predict.assert_not_called()
+
+ def test_batch_predict_no_model(self):
+ client = self.tables_client({}, {})
+ with pytest.raises(ValueError):
+ client.batch_predict(
+ gcs_input_uris="gs://input", gcs_output_uri_prefix="gs://output"
+ )
+ client.auto_ml_client.list_models.assert_not_called()
+ client.prediction_client.batch_predict.assert_not_called()
diff --git a/bigquery/.coveragerc b/bigquery/.coveragerc
new file mode 100644
index 000000000000..098720f672e1
--- /dev/null
+++ b/bigquery/.coveragerc
@@ -0,0 +1,16 @@
+[run]
+branch = True
+
+[report]
+fail_under = 100
+show_missing = True
+exclude_lines =
+ # Re-enable the standard pragma
+ pragma: NO COVER
+ # Ignore debug-only repr
+ def __repr__
+ # Ignore abstract methods
+ raise NotImplementedError
+omit =
+ */gapic/*.py
+ */proto/*.py
diff --git a/bigquery/.flake8 b/bigquery/.flake8
new file mode 100644
index 000000000000..61766fa84d02
--- /dev/null
+++ b/bigquery/.flake8
@@ -0,0 +1,13 @@
+[flake8]
+ignore = E203, E266, E501, W503
+exclude =
+ # Exclude generated code.
+ **/proto/**
+ **/gapic/**
+ *_pb2.py
+
+ # Standard linting exemptions.
+ __pycache__,
+ .git,
+ *.pyc,
+ conf.py
diff --git a/bigquery/.gitignore b/bigquery/.gitignore
new file mode 100644
index 000000000000..9e3a5f25770c
--- /dev/null
+++ b/bigquery/.gitignore
@@ -0,0 +1 @@
+docs/_build
\ No newline at end of file
diff --git a/bigquery/.repo-metadata.json b/bigquery/.repo-metadata.json
new file mode 100644
index 000000000000..5b4734b8e389
--- /dev/null
+++ b/bigquery/.repo-metadata.json
@@ -0,0 +1,13 @@
+{
+ "name": "bigquery",
+ "name_pretty": "Google Cloud BigQuery",
+ "product_documentation": "https://cloud.google.com/bigquery",
+ "client_documentation": "https://googleapis.dev/python/bigquery/latest",
+ "issue_tracker": "https://issuetracker.google.com/savedsearches/559654",
+ "release_level": "ga",
+ "language": "python",
+ "repo": "googleapis/google-cloud-python",
+ "distribution_name": "google-cloud-bigquery",
+ "api_id": "bigquery.googleapis.com",
+ "requires_billing": false
+}
\ No newline at end of file
diff --git a/bigquery/CHANGELOG.md b/bigquery/CHANGELOG.md
new file mode 100644
index 000000000000..6d1c71ccbf79
--- /dev/null
+++ b/bigquery/CHANGELOG.md
@@ -0,0 +1,722 @@
+# Changelog
+
+[PyPI History][1]
+
+[1]: https://pypi.org/project/google-cloud-bigquery/#history
+
+## 1.18.0
+
+08-08-2019 12:28 PDT
+
+### New Features
+
+- Add `bqstorage_client` param to `QueryJob.to_arrow()` ([#8693](https://github.com/googleapis/google-cloud-python/pull/8693))
+- Include SQL query and job ID in exception messages. ([#8748](https://github.com/googleapis/google-cloud-python/pull/8748))
+- Allow using TableListItem to construct a Table object. ([#8738](https://github.com/googleapis/google-cloud-python/pull/8738))
+- Add StandardSqlDataTypes enum to BigQuery ([#8782](https://github.com/googleapis/google-cloud-python/pull/8782))
+- Add `to_standard_sql()` method to SchemaField ([#8880](https://github.com/googleapis/google-cloud-python/pull/8880))
+- Add debug logging statements to track when BQ Storage API is used. ([#8838](https://github.com/googleapis/google-cloud-python/pull/8838))
+- Hide error traceback in BigQuery cell magic ([#8808](https://github.com/googleapis/google-cloud-python/pull/8808))
+- Allow choice of compression when loading from dataframe ([#8938](https://github.com/googleapis/google-cloud-python/pull/8938))
+- Additional clustering metrics for BQML K-means models (via synth). ([#8945](https://github.com/googleapis/google-cloud-python/pull/8945))
+
+### Documentation
+
+- Add compatibility check badges to READMEs. ([#8288](https://github.com/googleapis/google-cloud-python/pull/8288))
+- Link to googleapis.dev documentation in READMEs. ([#8705](https://github.com/googleapis/google-cloud-python/pull/8705))
+- Remove redundant service account key code sample. ([#8891](https://github.com/googleapis/google-cloud-python/pull/8891))
+
+### Internal / Testing Changes
+
+- Fix several pytest "skip if" markers ([#8694](https://github.com/googleapis/google-cloud-python/pull/8694))
+- Update tests to support conversion of NaN as NULL in pyarrow `0.14.*`. ([#8785](https://github.com/googleapis/google-cloud-python/pull/8785))
+- Mock external calls in one of BigQuery unit tests ([#8727](https://github.com/googleapis/google-cloud-python/pull/8727))
+- Set IPython user agent when running queries with IPython cell magic ([#8713](https://github.com/googleapis/google-cloud-python/pull/8713))
+- Use configurable bucket name for GCS samples data in systems tests. ([#8783](https://github.com/googleapis/google-cloud-python/pull/8783))
+- Move `maybe_fail_import()` to top level test utils ([#8840](https://github.com/googleapis/google-cloud-python/pull/8840))
+- Set BQ Storage client user-agent when in Jupyter cell ([#8734](https://github.com/googleapis/google-cloud-python/pull/8734))
+
+## 1.17.0
+
+07-12-2019 07:56 PDT
+
+### New Features
+
+- Support faster Arrow data format in `to_dataframe` when using BigQuery Storage API. ([#8551](https://github.com/googleapis/google-cloud-python/pull/8551))
+- Add `to_arrow` to get a `pyarrow.Table` from query results. ([#8609](https://github.com/googleapis/google-cloud-python/pull/8609))
+
+### Dependencies
+
+- Exclude bad 0.14.0 `pyarrow` release. ([#8551](https://github.com/googleapis/google-cloud-python/pull/8551))
+
+## 1.16.0
+
+07-01-2019 10:22 PDT
+
+### New Features
+
+- Add Routines API. ([#8491](https://github.com/googleapis/google-cloud-python/pull/8491))
+- Add more stats to Models API, such as `optimization_strategy` (via synth). ([#8344](https://github.com/googleapis/google-cloud-python/pull/8344))
+
+### Documentation
+
+- Add docs job to publish to googleapis.dev. ([#8464](https://github.com/googleapis/google-cloud-python/pull/8464))
+- Add sample demonstrating how to create a job. ([#8422](https://github.com/googleapis/google-cloud-python/pull/8422))
+- Use autodetected location in code samples. ([#8340](https://github.com/googleapis/google-cloud-python/pull/8340), [#8341](https://github.com/googleapis/google-cloud-python/pull/8341))
+
+### Internal / Testing Changes
+
+- Refactor `to_dataframe` to deterministicly update progress bar. ([#8303](https://github.com/googleapis/google-cloud-python/pull/8303))
+
+## 1.15.0
+
+06-14-2019 10:10 PDT
+
+### Implementation Changes
+
+- Fix bug where `load_table_from_dataframe` could not append to REQUIRED fields. ([#8230](https://github.com/googleapis/google-cloud-python/pull/8230))
+
+### New Features
+
+- Add `page_size` parameter to `QueryJob.result`. ([#8206](https://github.com/googleapis/google-cloud-python/pull/8206))
+
+## 1.14.0
+
+06-04-2019 11:11 PDT
+
+
+### New Features
+- Add `maximum_bytes_billed` argument and `context.default_query_job_config` property to magics. ([#8179](https://github.com/googleapis/google-cloud-python/pull/8179))
+
+### Dependencies
+- Don't pin `google-api-core` in libs using `google-cloud-core`. ([#8213](https://github.com/googleapis/google-cloud-python/pull/8213))
+
+## 1.13.0
+
+05-31-2019 10:22 PDT
+
+### New Features
+
+- Use `job_config.schema` for data type conversion if specified in `load_table_from_dataframe`. ([#8105](https://github.com/googleapis/google-cloud-python/pull/8105))
+
+### Internal / Testing Changes
+
+- Adds private `_connection` object to magics context. ([#8192](https://github.com/googleapis/google-cloud-python/pull/8192))
+- Fix coverage in 'types.py' (via synth). ([#8146](https://github.com/googleapis/google-cloud-python/pull/8146))
+
+## 1.12.1
+
+05-21-2019 11:16 PDT
+
+### Implementation Changes
+
+- Don't raise error when encountering unknown fields in Models API. ([#8083](https://github.com/googleapis/google-cloud-python/pull/8083))
+
+### Documentation
+
+- Use alabaster theme everwhere. ([#8021](https://github.com/googleapis/google-cloud-python/pull/8021))
+
+### Internal / Testing Changes
+
+- Add empty lines (via synth). ([#8049](https://github.com/googleapis/google-cloud-python/pull/8049))
+
+## 1.12.0
+
+05-16-2019 11:25 PDT
+
+### Implementation Changes
+- Remove duplicates from index on pandas DataFrames returned by `to_dataframe()`. ([#7953](https://github.com/googleapis/google-cloud-python/pull/7953))
+- Prevent error when time partitioning is populated with empty dict ([#7904](https://github.com/googleapis/google-cloud-python/pull/7904))
+- Preserve order in `to_dataframe` with BQ Storage from queries containing `ORDER BY` ([#7793](https://github.com/googleapis/google-cloud-python/pull/7793))
+- Respect `progress_bar_type` in `to_dataframe` when used with BQ Storage API ([#7697](https://github.com/googleapis/google-cloud-python/pull/7697))
+- Refactor QueryJob.query to read from resource dictionary ([#7763](https://github.com/googleapis/google-cloud-python/pull/7763))
+- Close the `to_dataframe` progress bar when finished. ([#7757](https://github.com/googleapis/google-cloud-python/pull/7757))
+- Ensure that `KeyboardInterrupt` during `to_dataframe`no longer hangs. ([#7698](https://github.com/googleapis/google-cloud-python/pull/7698))
+- Raise ValueError when BQ Storage is required but missing ([#7726](https://github.com/googleapis/google-cloud-python/pull/7726))
+- Make `total_rows` available on RowIterator before iteration ([#7622](https://github.com/googleapis/google-cloud-python/pull/7622))
+- Avoid masking auth errors in `to_dataframe` with BQ Storage API ([#7674](https://github.com/googleapis/google-cloud-python/pull/7674))
+
+### New Features
+- Add support for passing `client_info`. ([#7849](https://github.com/googleapis/google-cloud-python/pull/7849) and ([#7806](https://github.com/googleapis/google-cloud-python/pull/7806))
+- Phase 1 for storing schemas for later use. ([#7761](https://github.com/googleapis/google-cloud-python/pull/7761))
+- Add `destination` and related properties to LoadJob. ([#7710](https://github.com/googleapis/google-cloud-python/pull/7710))
+- Add `clustering_fields` property to TableListItem ([#7692](https://github.com/googleapis/google-cloud-python/pull/7692))
+- Add `created` and `expires` properties to TableListItem ([#7684](https://github.com/googleapis/google-cloud-python/pull/7684))
+
+### Dependencies
+- Pin `google-cloud-core >= 1.0.0, < 2.0dev`. ([#7993](https://github.com/googleapis/google-cloud-python/pull/7993))
+- Add `[all]` extras to install all extra dependencies ([#7610](https://github.com/googleapis/google-cloud-python/pull/7610))
+
+### Documentation
+- Move table and dataset snippets to samples/ directory ([#7683](https://github.com/googleapis/google-cloud-python/pull/7683))
+
+### Internal / Testing Changes
+- Blacken unit tests. ([#7960](https://github.com/googleapis/google-cloud-python/pull/7960))
+- Cleanup client tests with method to create minimal table resource ([#7802](https://github.com/googleapis/google-cloud-python/pull/7802))
+
+## 1.11.2
+
+04-05-2019 08:16 PDT
+
+### Dependencies
+
+- Add dependency on protobuf. ([#7668](https://github.com/googleapis/google-cloud-python/pull/7668))
+
+## 1.11.1
+
+04-04-2019 09:19 PDT
+
+### Internal / Testing Changes
+
+- Increment version number in `setup.py`.
+
+## 1.11.0
+
+04-03-2019 19:33 PDT
+
+### Implementation Changes
+
+- Remove classifier for Python 3.4 for end-of-life. ([#7535](https://github.com/googleapis/google-cloud-python/pull/7535))
+
+### New Features
+
+- Enable fastparquet support by using temporary file in `load_table_from_dataframe` ([#7545](https://github.com/googleapis/google-cloud-python/pull/7545))
+- Allow string for copy sources, query destination, and default dataset ([#7560](https://github.com/googleapis/google-cloud-python/pull/7560))
+- Add `progress_bar_type` argument to `to_dataframe` to use `tqdm` to display a progress bar ([#7552](https://github.com/googleapis/google-cloud-python/pull/7552))
+- Call `get_table` in `list_rows` if the schema is not available ([#7621](https://github.com/googleapis/google-cloud-python/pull/7621))
+- Fallback to BQ API when there are problems reading from BQ Storage. ([#7633](https://github.com/googleapis/google-cloud-python/pull/7633))
+- Add methods for Models API ([#7562](https://github.com/googleapis/google-cloud-python/pull/7562))
+- Add option to use BigQuery Storage API from IPython magics ([#7640](https://github.com/googleapis/google-cloud-python/pull/7640))
+
+### Documentation
+
+- Remove typo in `Table.from_api_repr` docstring. ([#7509](https://github.com/googleapis/google-cloud-python/pull/7509))
+- Add docs session to nox configuration for BigQuery ([#7541](https://github.com/googleapis/google-cloud-python/pull/7541))
+
+### Internal / Testing Changes
+
+- Refactor `table()` methods into shared implementation. ([#7516](https://github.com/googleapis/google-cloud-python/pull/7516))
+- Blacken noxfile and setup file in nox session ([#7619](https://github.com/googleapis/google-cloud-python/pull/7619))
+- Actually use the `progress_bar_type` argument in `QueryJob.to_dataframe()`. ([#7616](https://github.com/googleapis/google-cloud-python/pull/7616))
+
+## 1.10.0
+
+03-06-2019 15:20 PST
+
+### Implementation Changes
+
+- Harden 'ArrayQueryParameter.from_api_repr' against missing 'parameterValue'. ([#7311](https://github.com/googleapis/google-cloud-python/pull/7311))
+- Allow nested records w/ null values. ([#7297](https://github.com/googleapis/google-cloud-python/pull/7297))
+
+### New Features
+
+- Add options to ignore errors when creating/deleting datasets/tables. ([#7491](https://github.com/googleapis/google-cloud-python/pull/7491))
+- Accept a string in Table and Dataset constructors. ([#7483](https://github.com/googleapis/google-cloud-python/pull/7483))
+
+### Documentation
+
+- Update docstring of RowIterator's to_dataframe ([#7306](https://github.com/googleapis/google-cloud-python/pull/7306))
+- Updated client library documentation URLs. ([#7307](https://github.com/googleapis/google-cloud-python/pull/7307))
+
+### Internal / Testing Changes
+
+- Fix lint. ([#7383](https://github.com/googleapis/google-cloud-python/pull/7383))
+
+## 1.9.0
+
+02-04-2019 13:28 PST
+
+### New Features
+
+- Add arguments to select `dtypes` and use BQ Storage API to `QueryJob.to_dataframe()`. ([#7241](https://github.com/googleapis/google-cloud-python/pull/7241))
+
+### Documentation
+
+- Add sample for fetching `total_rows` from query results. ([#7217](https://github.com/googleapis/google-cloud-python/pull/7217))
+
+## 1.8.1
+
+12-17-2018 17:53 PST
+
+
+### Documentation
+- Document Python 2 deprecation ([#6910](https://github.com/googleapis/google-cloud-python/pull/6910))
+- Normalize docs for 'page_size' / 'max_results' / 'page_token' ([#6842](https://github.com/googleapis/google-cloud-python/pull/6842))
+
+## 1.8.0
+
+12-10-2018 12:39 PST
+
+
+### Implementation Changes
+- Add option to use BQ Storage API with `to_dataframe` ([#6854](https://github.com/googleapis/google-cloud-python/pull/6854))
+- Fix exception type in comment ([#6847](https://github.com/googleapis/google-cloud-python/pull/6847))
+- Add `to_bqstorage` to convert from Table[Reference] google-cloud-bigquery-storage reference ([#6840](https://github.com/googleapis/google-cloud-python/pull/6840))
+- Import `iam.policy` from `google.api_core`. ([#6741](https://github.com/googleapis/google-cloud-python/pull/6741))
+- Add avro logical type control for load jobs. ([#6827](https://github.com/googleapis/google-cloud-python/pull/6827))
+- Allow setting partition expiration to 'None'. ([#6823](https://github.com/googleapis/google-cloud-python/pull/6823))
+- Add `retry` argument to `_AsyncJob.result`. ([#6302](https://github.com/googleapis/google-cloud-python/pull/6302))
+
+### Dependencies
+- Update dependency to google-cloud-core ([#6835](https://github.com/googleapis/google-cloud-python/pull/6835))
+
+### Documentation
+- Add avro load samples ([#6832](https://github.com/googleapis/google-cloud-python/pull/6832))
+
+### Internal / Testing Changes
+- Blacken libraries ([#6794](https://github.com/googleapis/google-cloud-python/pull/6794))
+- Fix copy/paste typos in noxfile comments ([#6831](https://github.com/googleapis/google-cloud-python/pull/6831))
+
+## 1.7.0
+
+11-05-2018 16:41 PST
+
+### Implementation Changes
+
+- Add destination table properties to `LoadJobConfig`. ([#6202](https://github.com/googleapis/google-cloud-python/pull/6202))
+- Allow strings or references in `create_dataset` and `create_table` ([#6199](https://github.com/googleapis/google-cloud-python/pull/6199))
+- Fix swallowed error message ([#6168](https://github.com/googleapis/google-cloud-python/pull/6168))
+
+### New Features
+
+- Add `--params option` to `%%bigquery` magic ([#6277](https://github.com/googleapis/google-cloud-python/pull/6277))
+- Expose `to_api_repr` method for jobs. ([#6176](https://github.com/googleapis/google-cloud-python/pull/6176))
+- Allow string in addition to DatasetReference / TableReference in Client methods. ([#6164](https://github.com/googleapis/google-cloud-python/pull/6164))
+- Add keyword arguments to job config constructors for setting properties ([#6397](https://github.com/googleapis/google-cloud-python/pull/6397))
+
+### Documentation
+
+- Update README service links in quickstart guides. ([#6322](https://github.com/googleapis/google-cloud-python/pull/6322))
+- Move usage guides to their own docs. ([#6238](https://github.com/googleapis/google-cloud-python/pull/6238))
+- Normalize use of support level badges ([#6159](https://github.com/googleapis/google-cloud-python/pull/6159))
+
+### Internal / Testing Changes
+
+- Deprecation cleanups ([#6304](https://github.com/googleapis/google-cloud-python/pull/6304))
+- Use `_get_sub_prop` helper so missing load stats don't raise. ([#6269](https://github.com/googleapis/google-cloud-python/pull/6269))
+- Use new Nox ([#6175](https://github.com/googleapis/google-cloud-python/pull/6175))
+- Harden snippets against transient GCS errors. ([#6184](https://github.com/googleapis/google-cloud-python/pull/6184))
+
+## 1.6.0
+
+### New Features
+- Add support for `GEOGRAPHY` type ([#6147](https://github.com/googleapis/google-cloud-python/pull/6147))
+- Add default QueryJobConfig to Client ([#6088](https://github.com/googleapis/google-cloud-python/pull/6088))
+
+### Documentation
+- Remove unused "append" samples ([#6100](https://github.com/googleapis/google-cloud-python/pull/6100))
+
+### Internal / Testing Changes
+- Address dataset leaks, conflicts in systests ([#6099](https://github.com/googleapis/google-cloud-python/pull/6099))
+- Harden bucket teardown against `429 Too Many Requests`. ([#6101](https://github.com/googleapis/google-cloud-python/pull/6101))
+
+## 1.5.1
+
+### Implementation Changes
+
+- Retry '502 Bad Gateway' errors by default. (#5930)
+- Avoid pulling entire result set into memory when constructing dataframe. (#5870)
+- Add support for retrying unstructured 429 / 500 / 502 responses. (#6011)
+- Populate the jobReference from the API response. (#6044)
+
+### Documentation
+
+- Prepare documentation for repo split (#5955)
+- Fix leakage of bigquery/spanner sections into sidebar menu. (#5986)
+
+### Internal / Testing Changes
+
+- Test pandas support under Python 3.7. (#5857)
+- Nox: use inplace installs (#5865)
+- Update system test to use test data in bigquery-public-data. (#5965)
+
+## 1.5.0
+
+### Implementation Changes
+
+- Make 'Table.location' read-only. (#5687)
+
+### New Features
+
+- Add 'clustering_fields' properties. (#5630)
+- Add support for job labels (#5654)
+- Add 'QueryJob.estimated_bytes_processed' property (#5655)
+- Add support/tests for loading tables from 'gzip.GzipFile'. (#5711)
+- Add 'ExternalSourceFormat' enum. (#5674)
+- Add default location to client (#5678)
+
+### Documentation
+
+- Fix typo in CopyJob sources docstring (#5690)
+
+### Internal / Testing Changes
+
+- Add/refactor snippets for managing BigQuery jobs (#5631)
+- Reenable systests for 'dataset.update'/'table.update'. (#5732)
+
+## 1.4.0
+
+### Implementation Changes
+
+- Add 'internalError' to retryable error reasons. (#5599)
+- Don't raise exception if viewing CREATE VIEW DDL results (#5602)
+
+### New Features
+
+- Add Orc source format support and samples (#5500)
+- Move 'DEFAULT_RETRY' (w/ its predicate) to a new public 'retry' module. (#5552)
+- Allow listing rows on an empty table. (#5584)
+
+### Documentation
+
+- Add load_table_from_dataframe() to usage docs and changelog and dedents snippets in usage page (#5501)
+- Add samples for query external data sources (GCS & Sheets) (#5491)
+- Add BigQuery authorized view samples (#5515)
+- Update docs to show pyarrow as the only dependency of load_table_from_dataframe() (#5582)
+
+### Internal / Testing Changes
+
+- Add missing explict coverage for '_helpers' (#5550)
+- Skip update_table and update_dataset tests until etag issue is resolved. (#5590)
+
+## 1.3.0
+
+### New Features
+
+- NUMERIC type support (#5331)
+- Add timeline and top-level slot-millis to query statistics. (#5312)
+- Add additional statistics to query plan stages. (#5307)
+- Add `client.load_table_from_dataframe()` (#5387)
+
+### Documentation
+
+- Use autosummary to split up API reference docs (#5340)
+- Fix typo in Client docstrings (#5342)
+
+### Internal / Testing Changes
+
+- Prune systests identified as reduntant to snippets. (#5365)
+- Modify system tests to use prerelease versions of grpcio (#5304)
+- Improve system test performance (#5319)
+
+## 1.2.0
+
+### Implementation Changes
+- Switch `list_partitions` helper to a direct metatable read (#5273)
+- Fix typo in `Encoding.ISO_8859_1` enum value (#5211)
+
+### New Features
+- Add UnknownJob type for redacted jobs. (#5281)
+- Add project parameter to `list_datasets` and `list_jobs` (#5217)
+- Add from_string factory methods to Dataset and Table (#5255)
+- Add column based time partitioning (#5267)
+
+### Documentation
+- Standardize docstrings for constants (#5289)
+- Fix docstring / impl of `ExtractJob.destination_uri_file_counts`. (#5245)
+
+### Internal / Testing Changes
+- Add testing support for Python 3.7; remove testing support for Python 3.4. (#5295)
+
+## 1.1.0
+
+### New Features
+- Add `client.get_service_account_email` (#5203)
+
+### Documentation
+- Update samples and standardize region tags (#5195)
+
+### Internal / Testing Changes
+- Fix trove classifier to be Production/Stable
+- Don't suppress 'dots' output on test (#5202)
+
+## 1.0.0
+
+### Implementation Changes
+- Remove deprecated Client methods (#5182)
+
+## 0.32.0
+
+### :warning: Interface changes
+
+- Use `job.configuration` resource for XXXJobConfig classes (#5036)
+
+### Interface additions
+
+- Add `page_size` parameter for `list_rows` and use in DB-API for `arraysize` (#4931)
+- Add IPython magics for running queries (#4983)
+
+### Documentation
+
+- Add job string constant parameters in init and snippets documentation (#4987)
+
+### Internal / Testing changes
+
+- Specify IPython version 5.5 when running Python 2.7 tests (#5145)
+- Move all Dataset property conversion logic into properties (#5130)
+- Remove unnecessary _Table class from test_job.py (#5126)
+- Use explicit bytes to initialize 'BytesIO'. (#5116)
+- Make SchemaField be able to include description via from_api_repr method (#5114)
+- Remove _ApiResourceProperty class (#5107)
+- Add dev version for 0.32.0 release (#5105)
+- StringIO to BytesIO (#5101)
+- Shorten snippets test name (#5091)
+- Don't use `selected_fields` for listing query result rows (#5072)
+- Add location property to job classes. (#5071)
+- Use autospec for Connection in tests. (#5066)
+- Add Parquet SourceFormat and samples (#5057)
+- Remove test_load_table_from_uri_w_autodetect_schema_then_get_job because of duplicate test in snippets (#5004)
+- Fix encoding variable and strings UTF-8 and ISO-8859-1 difference documentation (#4990)
+
+## 0.31.0
+
+### Interface additions
+
+- Add support for `EncryptionConfiguration` (#4845)
+
+### Implementation changes
+
+- Allow listing/getting jobs even when there is an "invalid" job. (#4786)
+
+### Dependencies
+
+- The minimum version for `google-api-core` has been updated to version 1.0.0. This may cause some incompatibility with older google-cloud libraries, you will need to update those libraries if you have a dependency conflict. (#4944, #4946)
+
+### Documentation
+
+- Update format in `Table.full_table_id` and `TableListItem.full_table_id` docstrings. (#4906)
+
+### Testing and internal changes
+
+- Install local dependencies when running lint (#4936)
+- Re-enable lint for tests, remove usage of pylint (#4921)
+- Normalize all setup.py files (#4909)
+- Remove unnecessary debug print from tests (#4907)
+- Use constant strings for job properties in tests (#4833)
+
+## 0.30.0
+
+This is the release candidate for v1.0.0.
+
+### Interface changes / additions
+
+- Add `delete_contents` to `delete_dataset`. (#4724)
+
+### Bugfixes
+
+- Add handling of missing properties in `SchemaField.from_api_repr()`. (#4754)
+- Fix missing return value in `LoadJobConfig.from_api_repr`. (#4727)
+
+### Documentation
+
+- Minor documentation and typo fixes. (#4782, #4718, #4784, #4835, #4836)
+
+## 0.29.0
+
+### Interface changes / additions
+
+- Add `to_dataframe()` method to row iterators. When Pandas is installed this
+ method returns a `DataFrame` containing the query's or table's rows.
+ ([#4354](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/4354))
+- Iterate over a `QueryJob` to wait for and get the query results.
+ ([#4350](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/4350))
+- Add `Table.reference` and `Dataset.reference` properties to get the
+ `TableReference` or `DatasetReference` corresponding to that `Table` or
+ `Dataset`, respectively.
+ ([#4405](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/4405))
+- Add `Row.keys()`, `Row.items()`, and `Row.get()`. This makes `Row` act
+ more like a built-in dictionary.
+ ([#4393](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/4393),
+ [#4413](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/4413))
+
+### Interface changes / breaking changes
+
+- Add `Client.insert_rows()` and `Client.insert_rows_json()`, deprecate
+ `Client.create_rows()` and `Client.create_rows_json()`.
+ ([#4657](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/4657))
+- Add `Client.list_tables`, deprecate `Client.list_dataset_tables`.
+ ([#4653](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/4653))
+- `Client.list_tables` returns an iterators of `TableListItem`. The API
+ only returns a subset of properties of a table when listing.
+ ([#4427](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/4427))
+- Remove `QueryJob.query_results()`. Use `QueryJob.result()` instead.
+ ([#4652](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/4652))
+- Remove `Client.query_rows()`. Use `Client.query()` instead.
+ ([#4429](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/4429))
+- `Client.list_datasets` returns an iterator of `DatasetListItem`. The API
+ only returns a subset of properties of a dataset when listing.
+ ([#4439](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/4439))
+
+## 0.28.0
+
+**0.28.0 significantly changes the interface for this package.** For examples
+of the differences between 0.28.0 and previous versions, see
+[Migrating to the BigQuery Python client library 0.28][2].
+These changes can be summarized as follows:
+
+- Query and view operations default to the standard SQL dialect. (#4192)
+- Client functions related to
+ [jobs](https://cloud.google.com/bigquery/docs/jobs-overview), like running
+ queries, immediately start the job.
+- Functions to create, get, update, delete datasets and tables moved to the
+ client class.
+
+[2]: https://cloud.google.com/bigquery/docs/python-client-migration
+
+### Fixes
+
+- Populate timeout parameter correctly for queries (#4209)
+- Automatically retry idempotent RPCs (#4148, #4178)
+- Parse timestamps in query parameters using canonical format (#3945)
+- Parse array parameters that contain a struct type. (#4040)
+- Support Sub Second Datetimes in row data (#3901, #3915, #3926), h/t @page1
+
+### Interface changes / additions
+
+- Support external table configuration (#4182) in query jobs (#4191) and
+ tables (#4193).
+- New `Row` class allows for access by integer index like a tuple, string
+ index like a dictionary, or attribute access like an object. (#4149)
+- Add option for job ID generation with user-supplied prefix (#4198)
+- Add support for update of dataset access entries (#4197)
+- Add support for atomic read-modify-write of a dataset using etag (#4052)
+- Add support for labels to `Dataset` (#4026)
+- Add support for labels to `Table` (#4207)
+- Add `Table.streaming_buffer` property (#4161)
+- Add `TableReference` class (#3942)
+- Add `DatasetReference` class (#3938, #3942, #3993)
+- Add `ExtractJob.destination_uri_file_counts` property. (#3803)
+- Add `client.create_rows_json()` to bypass conversions on streaming writes.
+ (#4189)
+- Add `client.get_job()` to get arbitrary jobs. (#3804, #4213)
+- Add filter to `client.list_datasets()` (#4205)
+- Add `QueryJob.undeclared_query_parameters` property. (#3802)
+- Add `QueryJob.referenced_tables` property. (#3801)
+- Add new scalar statistics properties to `QueryJob` (#3800)
+- Add `QueryJob.query_plan` property. (#3799)
+
+### Interface changes / breaking changes
+
+- Remove `client.run_async_query()`, use `client.query()` instead. (#4130)
+- Remove `client.run_sync_query()`, use `client.query_rows()` instead. (#4065, #4248)
+- Make `QueryResults` read-only. (#4094, #4144)
+- Make `get_query_results` private. Return rows for `QueryJob.result()` (#3883)
+- Move `*QueryParameter` and `UDFResource` classes to `query` module (also
+ exposed in `bigquery` module). (#4156)
+
+#### Changes to tables
+
+- Remove `client` from `Table` class (#4159)
+- Remove `table.exists()` (#4145)
+- Move `table.list_parations` to `client.list_partitions` (#4146)
+- Move `table.upload_from_file` to `client.load_table_from_file` (#4136)
+- Move `table.update()` and `table.patch()` to `client.update_table()` (#4076)
+- Move `table.insert_data()` to `client.create_rows()`. Automatically
+ generates row IDs if not supplied. (#4151, #4173)
+- Move `table.fetch_data()` to `client.list_rows()` (#4119, #4143)
+- Move `table.delete()` to `client.delete_table()` (#4066)
+- Move `table.create()` to `client.create_table()` (#4038, #4043)
+- Move `table.reload()` to `client.get_table()` (#4004)
+- Rename `Table.name` attribute to `Table.table_id` (#3959)
+- `Table` constructor takes a `TableReference` as parameter (#3997)
+
+#### Changes to datasets
+
+- Remove `client` from `Dataset` class (#4018)
+- Remove `dataset.exists()` (#3996)
+- Move `dataset.list_tables()` to `client.list_dataset_tables()` (#4013)
+- Move `dataset.delete()` to `client.delete_dataset()` (#4012)
+- Move `dataset.patch()` and `dataset.update()` to `client.update_dataset()` (#4003)
+- Move `dataset.create()` to `client.create_dataset()` (#3982)
+- Move `dataset.reload()` to `client.get_dataset()` (#3973)
+- Rename `Dataset.name` attribute to `Dataset.dataset_id` (#3955)
+- `client.dataset()` returns a `DatasetReference` instead of `Dataset`. (#3944)
+- Rename class: `dataset.AccessGrant -> dataset.AccessEntry`. (#3798)
+- `dataset.table()` returns a `TableReference` instead of a `Table` (#4014)
+- `Dataset` constructor takes a DatasetReference (#4036)
+
+#### Changes to jobs
+
+- Make `job.begin()` method private. (#4242)
+- Add `LoadJobConfig` class and modify `LoadJob` (#4103, #4137)
+- Add `CopyJobConfig` class and modify `CopyJob` (#4051, #4059)
+- Type of Job's and Query's `default_dataset` changed from `Dataset` to
+ `DatasetReference` (#4037)
+- Rename `client.load_table_from_storage()` to `client.load_table_from_uri()`
+ (#4235)
+- Rename `client.extract_table_to_storage` to `client.extract_table()`.
+ Method starts the extract job immediately. (#3991, #4177)
+- Rename `XJob.name` to `XJob.job_id`. (#3962)
+- Rename job classes. `LoadTableFromStorageJob -> LoadJob` and
+ `ExtractTableToStorageJob -> jobs.ExtractJob` (#3797)
+
+### Dependencies
+
+- Updating to `google-cloud-core ~= 0.28`, in particular, the
+ `google-api-core` package has been moved out of `google-cloud-core`. (#4221)
+
+PyPI: https://pypi.org/project/google-cloud-bigquery/0.28.0/
+
+
+## 0.27.0
+
+- Remove client-side enum validation. (#3735)
+- Add `Table.row_from_mapping` helper. (#3425)
+- Move `google.cloud.future` to `google.api.core` (#3764)
+- Fix `__eq__` and `__ne__`. (#3765)
+- Move `google.cloud.iterator` to `google.api.core.page_iterator` (#3770)
+- `nullMarker` support for BigQuery Load Jobs (#3777), h/t @leondealmeida
+- Allow `job_id` to be explicitly specified in DB-API. (#3779)
+- Add support for a custom null marker. (#3776)
+- Add `SchemaField` serialization and deserialization. (#3786)
+- Add `get_query_results` method to the client. (#3838)
+- Poll for query completion via `getQueryResults` method. (#3844)
+- Allow fetching more than the first page when `max_results` is set. (#3845)
+
+PyPI: https://pypi.org/project/google-cloud-bigquery/0.27.0/
+
+## 0.26.0
+
+### Notable implementation changes
+
+- Using the `requests` transport attached to a Client for for resumable media
+ (i.e. downloads and uploads) (#3705) (this relates to the `httplib2` to
+ `requests` switch)
+
+### Interface changes / additions
+
+- Adding `autodetect` property on `LoadTableFromStorageJob` to enable schema
+ autodetection. (#3648)
+- Implementing the Python Futures interface for Jobs. Call `job.result()` to
+ wait for jobs to complete instead of polling manually on the job status.
+ (#3626)
+- Adding `is_nullable` property on `SchemaField`. Can be used to check if a
+ column is nullable. (#3620)
+- `job_name` argument added to `Table.upload_from_file` for setting the job
+ ID. (#3605)
+- Adding `google.cloud.bigquery.dbapi` package, which implements PEP-249
+ DB-API specification. (#2921)
+- Adding `Table.view_use_legacy_sql` property. Can be used to create views
+ with legacy or standard SQL. (#3514)
+
+### Interface changes / breaking changes
+
+- Removing `results()` method from the `QueryJob` class. Use
+ `query_results()` instead. (#3661)
+- `SchemaField` is now immutable. It is also hashable so that it can be used
+ in sets. (#3601)
+
+### Dependencies
+
+- Updating to `google-cloud-core ~= 0.26`, in particular, the underlying HTTP
+ transport switched from `httplib2` to `requests` (#3654, #3674)
+- Adding dependency on `google-resumable-media` for loading BigQuery tables
+ from local files. (#3555)
+
+### Packaging
+
+- Fix inclusion of `tests` (vs. `unit_tests`) in `MANIFEST.in` (#3552)
+- Updating `author_email` in `setup.py` to `googleapis-publisher@google.com`.
+ (#3598)
+
+PyPI: https://pypi.org/project/google-cloud-bigquery/0.26.0/
diff --git a/bigquery/LICENSE b/bigquery/LICENSE
new file mode 100644
index 000000000000..a8ee855de2aa
--- /dev/null
+++ b/bigquery/LICENSE
@@ -0,0 +1,201 @@
+ Apache License
+ Version 2.0, January 2004
+ https://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [yyyy] [name of copyright owner]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
diff --git a/bigquery/MANIFEST.in b/bigquery/MANIFEST.in
new file mode 100644
index 000000000000..9cbf175afe6b
--- /dev/null
+++ b/bigquery/MANIFEST.in
@@ -0,0 +1,5 @@
+include README.rst LICENSE
+recursive-include google *.json *.proto
+recursive-include tests *
+global-exclude *.py[co]
+global-exclude __pycache__
diff --git a/bigquery/README.rst b/bigquery/README.rst
new file mode 100644
index 000000000000..8f73576d6f7d
--- /dev/null
+++ b/bigquery/README.rst
@@ -0,0 +1,104 @@
+Python Client for Google BigQuery
+=================================
+
+|GA| |pypi| |versions|
+
+Querying massive datasets can be time consuming and expensive without the
+right hardware and infrastructure. Google `BigQuery`_ solves this problem by
+enabling super-fast, SQL queries against append-mostly tables, using the
+processing power of Google's infrastructure.
+
+- `Client Library Documentation`_
+- `Product Documentation`_
+
+.. |GA| image:: https://img.shields.io/badge/support-GA-gold.svg
+ :target: https://github.com/googleapis/google-cloud-python/blob/master/README.rst#general-availability
+.. |pypi| image:: https://img.shields.io/pypi/v/google-cloud-bigquery.svg
+ :target: https://pypi.org/project/google-cloud-bigquery/
+.. |versions| image:: https://img.shields.io/pypi/pyversions/google-cloud-bigquery.svg
+ :target: https://pypi.org/project/google-cloud-bigquery/
+.. _BigQuery: https://cloud.google.com/bigquery/what-is-bigquery
+.. _Client Library Documentation: https://googleapis.dev/python/bigquery/latest
+.. _Product Documentation: https://cloud.google.com/bigquery/docs/reference/v2/
+
+Quick Start
+-----------
+
+In order to use this library, you first need to go through the following steps:
+
+1. `Select or create a Cloud Platform project.`_
+2. `Enable billing for your project.`_
+3. `Enable the Google Cloud BigQuery API.`_
+4. `Setup Authentication.`_
+
+.. _Select or create a Cloud Platform project.: https://console.cloud.google.com/project
+.. _Enable billing for your project.: https://cloud.google.com/billing/docs/how-to/modify-project#enable_billing_for_a_project
+.. _Enable the Google Cloud BigQuery API.: https://cloud.google.com/bigquery
+.. _Setup Authentication.: https://googleapis.dev/python/google-api-core/latest/auth.html
+
+Installation
+~~~~~~~~~~~~
+
+Install this library in a `virtualenv`_ using pip. `virtualenv`_ is a tool to
+create isolated Python environments. The basic problem it addresses is one of
+dependencies and versions, and indirectly permissions.
+
+With `virtualenv`_, it's possible to install this library without needing system
+install permissions, and without clashing with the installed system
+dependencies.
+
+.. _`virtualenv`: https://virtualenv.pypa.io/en/latest/
+
+
+Supported Python Versions
+^^^^^^^^^^^^^^^^^^^^^^^^^
+Python >= 3.5
+
+Deprecated Python Versions
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+Python == 2.7. Python 2.7 support will be removed on January 1, 2020.
+
+
+Mac/Linux
+^^^^^^^^^
+
+.. code-block:: console
+
+ pip install virtualenv
+ virtualenv
+ source /bin/activate
+ /bin/pip install google-cloud-bigquery
+
+
+Windows
+^^^^^^^
+
+.. code-block:: console
+
+ pip install virtualenv
+ virtualenv
+ \Scripts\activate
+ \Scripts\pip.exe install google-cloud-bigquery
+
+Example Usage
+-------------
+
+Perform a query
+~~~~~~~~~~~~~~~
+
+.. code:: python
+
+ from google.cloud import bigquery
+
+ client = bigquery.Client()
+
+ # Perform a query.
+ QUERY = (
+ 'SELECT name FROM `bigquery-public-data.usa_names.usa_1910_2013` '
+ 'WHERE state = "TX" '
+ 'LIMIT 100')
+ query_job = client.query(QUERY) # API request
+ rows = query_job.result() # Waits for query to finish
+
+ for row in rows:
+ print(row.name)
diff --git a/bigquery/benchmark/README.md b/bigquery/benchmark/README.md
new file mode 100644
index 000000000000..435926acb045
--- /dev/null
+++ b/bigquery/benchmark/README.md
@@ -0,0 +1,8 @@
+# BigQuery Benchmark
+This directory contains benchmarks for BigQuery client.
+
+## Usage
+`python benchmark.py queries.json`
+
+BigQuery service caches requests so the benchmark should be run
+at least twice, disregarding the first result.
diff --git a/bigquery/benchmark/benchmark.py b/bigquery/benchmark/benchmark.py
new file mode 100644
index 000000000000..2917f169aba1
--- /dev/null
+++ b/bigquery/benchmark/benchmark.py
@@ -0,0 +1,46 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from google.cloud import bigquery
+from datetime import datetime
+import json
+import sys
+
+if len(sys.argv) < 2:
+ raise Exception('need query file, usage: python {0} '.format(sys.argv[0]))
+
+with open(sys.argv[1], 'r') as f:
+ queries = json.loads(f.read())
+
+client = bigquery.Client()
+
+for query in queries:
+ start_time = datetime.now()
+ job = client.query(query)
+ rows = job.result()
+
+ num_rows = 0
+ num_cols = None
+ first_byte_time = None
+
+ for row in rows:
+ if num_rows == 0:
+ num_cols = len(row)
+ first_byte_time = datetime.now() - start_time
+ elif num_cols != len(row):
+ raise Exception('found {0} columsn, expected {1}'.format(len(row), num_cols))
+ num_rows += 1
+ total_time = datetime.now() - start_time
+ print("query {0}: {1} rows, {2} cols, first byte {3} sec, total {4} sec"
+ .format(query, num_rows, num_cols, first_byte_time.total_seconds(), total_time.total_seconds()))
diff --git a/bigquery/benchmark/queries.json b/bigquery/benchmark/queries.json
new file mode 100644
index 000000000000..13fed38b52b3
--- /dev/null
+++ b/bigquery/benchmark/queries.json
@@ -0,0 +1,10 @@
+[
+ "SELECT * FROM `nyc-tlc.yellow.trips` LIMIT 10000",
+ "SELECT * FROM `nyc-tlc.yellow.trips` LIMIT 100000",
+ "SELECT * FROM `nyc-tlc.yellow.trips` LIMIT 1000000",
+ "SELECT title FROM `bigquery-public-data.samples.wikipedia` ORDER BY title LIMIT 1000",
+ "SELECT title, id, timestamp, contributor_ip FROM `bigquery-public-data.samples.wikipedia` WHERE title like 'Blo%' ORDER BY id",
+ "SELECT * FROM `bigquery-public-data.baseball.games_post_wide` ORDER BY gameId",
+ "SELECT * FROM `bigquery-public-data.samples.github_nested` WHERE repository.has_downloads ORDER BY repository.created_at LIMIT 10000",
+ "SELECT repo_name, path FROM `bigquery-public-data.github_repos.files` WHERE path LIKE '%.java' ORDER BY id LIMIT 1000000"
+]
diff --git a/bigquery/docs/.gitignore b/bigquery/docs/.gitignore
new file mode 100644
index 000000000000..3fe20bec0f3a
--- /dev/null
+++ b/bigquery/docs/.gitignore
@@ -0,0 +1 @@
+generated/
\ No newline at end of file
diff --git a/bigquery/docs/README.rst b/bigquery/docs/README.rst
new file mode 120000
index 000000000000..89a0106941ff
--- /dev/null
+++ b/bigquery/docs/README.rst
@@ -0,0 +1 @@
+../README.rst
\ No newline at end of file
diff --git a/bigquery/docs/changelog.md b/bigquery/docs/changelog.md
new file mode 120000
index 000000000000..04c99a55caae
--- /dev/null
+++ b/bigquery/docs/changelog.md
@@ -0,0 +1 @@
+../CHANGELOG.md
\ No newline at end of file
diff --git a/bigquery/docs/conf.py b/bigquery/docs/conf.py
new file mode 100644
index 000000000000..b3cfa267dd7d
--- /dev/null
+++ b/bigquery/docs/conf.py
@@ -0,0 +1,363 @@
+# -*- coding: utf-8 -*-
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# google-cloud-bigquery documentation build configuration file
+#
+# This file is execfile()d with the current directory set to its
+# containing dir.
+#
+# Note that not all possible configuration values are present in this
+# autogenerated file.
+#
+# All configuration values have a default; values that are commented out
+# serve to show the default.
+
+import sys
+import os
+import shutil
+
+from sphinx.util import logging
+
+logger = logging.getLogger(__name__)
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+sys.path.insert(0, os.path.abspath(".."))
+
+__version__ = "0.1.0"
+
+# -- General configuration ------------------------------------------------
+
+# If your documentation needs a minimal Sphinx version, state it here.
+# needs_sphinx = '1.0'
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+ "sphinx.ext.autodoc",
+ "sphinx.ext.autosummary",
+ "sphinx.ext.intersphinx",
+ "sphinx.ext.coverage",
+ "sphinx.ext.napoleon",
+ "sphinx.ext.viewcode",
+]
+
+# autodoc/autosummary flags
+autoclass_content = "both"
+autodoc_default_flags = ["members"]
+autosummary_generate = True
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ["_templates", os.path.join("..", "..", "docs", "_templates")]
+
+# Allow markdown includes (so releases.md can include CHANGLEOG.md)
+# http://www.sphinx-doc.org/en/master/markdown.html
+source_parsers = {".md": "recommonmark.parser.CommonMarkParser"}
+
+# The suffix(es) of source filenames.
+source_suffix = [".rst", ".md"]
+
+# The encoding of source files.
+# source_encoding = 'utf-8-sig'
+
+# The master toctree document.
+master_doc = "index"
+
+# General information about the project.
+project = u"google-cloud-bigquery"
+copyright = u"2015, Google"
+author = u"Google APIs"
+
+# The version info for the project you're documenting, acts as replacement for
+# |version| and |release|, also used in various other places throughout the
+# built documents.
+#
+# The full version, including alpha/beta/rc tags.
+release = __version__
+# The short X.Y version.
+version = ".".join(release.split(".")[0:2])
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#
+# This is also used if you do content translation via gettext catalogs.
+# Usually you set "language" from the command line for these cases.
+language = None
+
+# There are two options for replacing |today|: either, you set today to some
+# non-false value, then it is used:
+# today = ''
+# Else, today_fmt is used as the format for a strftime call.
+# today_fmt = '%B %d, %Y'
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+exclude_patterns = ["_build"]
+
+# The reST default role (used for this markup: `text`) to use for all
+# documents.
+# default_role = None
+
+# If true, '()' will be appended to :func: etc. cross-reference text.
+# add_function_parentheses = True
+
+# If true, the current module name will be prepended to all description
+# unit titles (such as .. function::).
+# add_module_names = True
+
+# If true, sectionauthor and moduleauthor directives will be shown in the
+# output. They are ignored by default.
+# show_authors = False
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = "sphinx"
+
+# A list of ignored prefixes for module index sorting.
+# modindex_common_prefix = []
+
+# If true, keep warnings as "system message" paragraphs in the built documents.
+# keep_warnings = False
+
+# If true, `todo` and `todoList` produce output, else they produce nothing.
+todo_include_todos = True
+
+# -- Options for HTML output ----------------------------------------------
+
+# The theme to use for HTML and HTML Help pages. See the documentation for
+# a list of builtin themes.
+html_theme = "alabaster"
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further. For a list of options available for each theme, see the
+# documentation.
+# html_theme_options = {}
+
+# Add any paths that contain custom themes here, relative to this directory.
+# html_theme_path = []
+
+# The name for this set of Sphinx documents. If None, it defaults to
+# " v documentation".
+# html_title = None
+
+# A shorter title for the navigation bar. Default is the same as html_title.
+# html_short_title = None
+
+# The name of an image file (relative to this directory) to place at the top
+# of the sidebar.
+# html_logo = None
+
+# The name of an image file (within the static path) to use as favicon of the
+# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
+# pixels large.
+# html_favicon = None
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+# html_static_path = []
+
+# Add any extra paths that contain custom files (such as robots.txt or
+# .htaccess) here, relative to this directory. These files are copied
+# directly to the root of the documentation.
+# html_extra_path = []
+
+# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
+# using the given strftime format.
+# html_last_updated_fmt = '%b %d, %Y'
+
+# If true, SmartyPants will be used to convert quotes and dashes to
+# typographically correct entities.
+# html_use_smartypants = True
+
+# Custom sidebar templates, maps document names to template names.
+# html_sidebars = {}
+
+# Additional templates that should be rendered to pages, maps page names to
+# template names.
+# html_additional_pages = {}
+
+# If false, no module index is generated.
+# html_domain_indices = True
+
+# If false, no index is generated.
+# html_use_index = True
+
+# If true, the index is split into individual pages for each letter.
+# html_split_index = False
+
+# If true, links to the reST sources are added to the pages.
+# html_show_sourcelink = True
+
+# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
+# html_show_sphinx = True
+
+# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
+# html_show_copyright = True
+
+# If true, an OpenSearch description file will be output, and all pages will
+# contain a tag referring to it. The value of this option must be the
+# base URL from which the finished HTML is served.
+# html_use_opensearch = ''
+
+# This is the file name suffix for HTML files (e.g. ".xhtml").
+# html_file_suffix = None
+
+# Language to be used for generating the HTML full-text search index.
+# Sphinx supports the following languages:
+# 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja'
+# 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr'
+# html_search_language = 'en'
+
+# A dictionary with options for the search language support, empty by default.
+# Now only 'ja' uses this config value
+# html_search_options = {'type': 'default'}
+
+# The name of a javascript file (relative to the configuration directory) that
+# implements a search results scorer. If empty, the default will be used.
+# html_search_scorer = 'scorer.js'
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = "google-cloud-bigquery-doc"
+
+# -- Options for LaTeX output ---------------------------------------------
+
+latex_elements = {
+ # The paper size ('letterpaper' or 'a4paper').
+ #'papersize': 'letterpaper',
+ # The font size ('10pt', '11pt' or '12pt').
+ #'pointsize': '10pt',
+ # Additional stuff for the LaTeX preamble.
+ #'preamble': '',
+ # Latex figure (float) alignment
+ #'figure_align': 'htbp',
+}
+
+# Grouping the document tree into LaTeX files. List of tuples
+# (source start file, target name, title,
+# author, documentclass [howto, manual, or own class]).
+latex_documents = [
+ (
+ master_doc,
+ "google-cloud-bigquery.tex",
+ u"google-cloud-bigquery Documentation",
+ author,
+ "manual",
+ )
+]
+
+# The name of an image file (relative to this directory) to place at the top of
+# the title page.
+# latex_logo = None
+
+# For "manual" documents, if this is true, then toplevel headings are parts,
+# not chapters.
+# latex_use_parts = False
+
+# If true, show page references after internal links.
+# latex_show_pagerefs = False
+
+# If true, show URL addresses after external links.
+# latex_show_urls = False
+
+# Documents to append as an appendix to all manuals.
+# latex_appendices = []
+
+# If false, no module index is generated.
+# latex_domain_indices = True
+
+# -- Options for manual page output ---------------------------------------
+
+# One entry per manual page. List of tuples
+# (source start file, name, description, authors, manual section).
+man_pages = [
+ (
+ master_doc,
+ "google-cloud-bigquery",
+ u"google-cloud-bigquery Documentation",
+ [author],
+ 1,
+ )
+]
+
+# If true, show URL addresses after external links.
+# man_show_urls = False
+
+# -- Options for Texinfo output -------------------------------------------
+
+# Grouping the document tree into Texinfo files. List of tuples
+# (source start file, target name, title, author,
+# dir menu entry, description, category)
+texinfo_documents = [
+ (
+ master_doc,
+ "google-cloud-bigquery",
+ u"google-cloud-bigquery Documentation",
+ author,
+ "google-cloud-bigquery",
+ "APIs",
+ )
+]
+
+# Documents to append as an appendix to all manuals.
+# texinfo_appendices = []
+
+# If false, no module index is generated.
+# texinfo_domain_indices = True
+
+# How to display URL addresses: 'footnote', 'no', or 'inline'.
+# texinfo_show_urls = 'footnote'
+
+# If true, do not generate a @detailmenu in the "Top" node's menu.
+# texinfo_no_detailmenu = False
+
+# Example configuration for intersphinx: refer to the Python standard library.
+intersphinx_mapping = {
+ "gax": ("https://gax-python.readthedocs.org/en/latest/", None),
+ "pandas": ("https://pandas.pydata.org/pandas-docs/stable/", None),
+ "python": ("http://python.readthedocs.org/en/latest/", None),
+}
+
+# Napoleon settings
+napoleon_google_docstring = True
+napoleon_numpy_docstring = True
+napoleon_include_private_with_doc = False
+napoleon_include_special_with_doc = True
+napoleon_use_admonition_for_examples = False
+napoleon_use_admonition_for_notes = False
+napoleon_use_admonition_for_references = False
+napoleon_use_ivar = False
+napoleon_use_param = True
+napoleon_use_rtype = True
+
+# Static HTML pages, e.g. to support redirects
+# See: https://tech.signavio.com/2017/managing-sphinx-redirects
+# HTML pages to be copied from source to target
+static_html_pages = ["usage.html", "generated/google.cloud.bigquery.magics.html"]
+
+
+def copy_static_html_pages(app, exception):
+ if exception is None and app.builder.name == "html":
+ for static_html_page in static_html_pages:
+ target_path = app.outdir + "/" + static_html_page
+ src_path = app.srcdir + "/" + static_html_page
+ if os.path.isfile(src_path):
+ logger.info("Copying static html: %s -> %s", src_path, target_path)
+ shutil.copyfile(src_path, target_path)
+
+
+def setup(app):
+ app.connect("build-finished", copy_static_html_pages)
diff --git a/bigquery/docs/dbapi.rst b/bigquery/docs/dbapi.rst
new file mode 100644
index 000000000000..ca0256d3c8de
--- /dev/null
+++ b/bigquery/docs/dbapi.rst
@@ -0,0 +1,6 @@
+DB-API Reference
+~~~~~~~~~~~~~~~~
+
+.. automodule:: google.cloud.bigquery.dbapi
+ :members:
+ :show-inheritance:
diff --git a/bigquery/docs/gapic/v2/enums.rst b/bigquery/docs/gapic/v2/enums.rst
new file mode 100644
index 000000000000..0e0f05adaea5
--- /dev/null
+++ b/bigquery/docs/gapic/v2/enums.rst
@@ -0,0 +1,8 @@
+Enums for BigQuery API Client
+=============================
+
+.. autoclass:: google.cloud.bigquery_v2.gapic.enums.Model
+ :members:
+
+.. autoclass:: google.cloud.bigquery_v2.gapic.enums.StandardSqlDataType
+ :members:
diff --git a/bigquery/docs/gapic/v2/types.rst b/bigquery/docs/gapic/v2/types.rst
new file mode 100644
index 000000000000..97938768a690
--- /dev/null
+++ b/bigquery/docs/gapic/v2/types.rst
@@ -0,0 +1,5 @@
+Types for BigQuery API Client
+=============================
+
+.. automodule:: google.cloud.bigquery_v2.types
+ :members:
\ No newline at end of file
diff --git a/bigquery/docs/generated/google.cloud.bigquery.magics.html b/bigquery/docs/generated/google.cloud.bigquery.magics.html
new file mode 100644
index 000000000000..0d2a00fa14c9
--- /dev/null
+++ b/bigquery/docs/generated/google.cloud.bigquery.magics.html
@@ -0,0 +1,8 @@
+
+
+
+
+
+
diff --git a/bigquery/docs/index.rst b/bigquery/docs/index.rst
new file mode 100644
index 000000000000..ec360de69770
--- /dev/null
+++ b/bigquery/docs/index.rst
@@ -0,0 +1,29 @@
+.. include:: README.rst
+
+More Examples
+~~~~~~~~~~~~~
+
+.. toctree::
+ :maxdepth: 2
+
+ usage/index
+ Official Google BigQuery How-to Guides
+
+API Reference
+-------------
+
+.. toctree::
+ :maxdepth: 2
+
+ reference
+ dbapi
+
+Changelog
+---------
+
+For a list of all ``google-cloud-bigquery`` releases:
+
+.. toctree::
+ :maxdepth: 2
+
+ changelog
diff --git a/bigquery/docs/magics.rst b/bigquery/docs/magics.rst
new file mode 100644
index 000000000000..732c27af94dd
--- /dev/null
+++ b/bigquery/docs/magics.rst
@@ -0,0 +1,5 @@
+IPython Magics for BigQuery
+===========================
+
+.. automodule:: google.cloud.bigquery.magics
+ :members:
diff --git a/bigquery/docs/reference.rst b/bigquery/docs/reference.rst
new file mode 100644
index 000000000000..e01443808795
--- /dev/null
+++ b/bigquery/docs/reference.rst
@@ -0,0 +1,186 @@
+API Reference
+~~~~~~~~~~~~~
+
+.. currentmodule:: google.cloud.bigquery
+
+The main concepts with this API are:
+
+- :class:`~google.cloud.bigquery.client.Client` manages connections to the
+ BigQuery API. Use the client methods to run jobs (such as a
+ :class:`~google.cloud.bigquery.job.QueryJob` via
+ :meth:`~google.cloud.bigquery.client.Client.query`) and manage resources.
+
+- :class:`~google.cloud.bigquery.dataset.Dataset` represents a
+ collection of tables.
+
+- :class:`~google.cloud.bigquery.table.Table` represents a single "relation".
+
+Client
+======
+
+.. autosummary::
+ :toctree: generated
+
+ client.Client
+
+Job
+===
+
+Job Configuration
+-----------------
+
+.. autosummary::
+ :toctree: generated
+
+ job.QueryJobConfig
+ job.CopyJobConfig
+ job.LoadJobConfig
+ job.ExtractJobConfig
+
+Job Classes
+-----------
+
+.. autosummary::
+ :toctree: generated
+
+ job.QueryJob
+ job.CopyJob
+ job.LoadJob
+ job.ExtractJob
+ job.UnknownJob
+
+Job-Related Types
+-----------------
+
+.. autosummary::
+ :toctree: generated
+
+ job.Compression
+ job.CreateDisposition
+ job.DestinationFormat
+ job.Encoding
+ job.QueryPriority
+ job.SourceFormat
+ job.WriteDisposition
+ job.SchemaUpdateOption
+
+
+Dataset
+=======
+
+.. autosummary::
+ :toctree: generated
+
+ dataset.Dataset
+ dataset.DatasetListItem
+ dataset.DatasetReference
+ dataset.AccessEntry
+
+
+Table
+=====
+
+.. autosummary::
+ :toctree: generated
+
+ table.Table
+ table.TableListItem
+ table.TableReference
+ table.Row
+ table.RowIterator
+ table.EncryptionConfiguration
+ table.TimePartitioning
+ table.TimePartitioningType
+
+Model
+=====
+
+.. autosummary::
+ :toctree: generated
+
+ model.Model
+ model.ModelReference
+
+Routine
+=======
+
+.. autosummary::
+ :toctree: generated
+
+ routine.Routine
+ routine.RoutineArgument
+ routine.RoutineReference
+
+Schema
+======
+
+.. autosummary::
+ :toctree: generated
+
+ schema.SchemaField
+
+
+Query
+=====
+
+.. autosummary::
+ :toctree: generated
+
+ query.ArrayQueryParameter
+ query.ScalarQueryParameter
+ query.StructQueryParameter
+ query.UDFResource
+
+
+Retries
+=======
+
+.. autosummary::
+ :toctree: generated
+
+ retry.DEFAULT_RETRY
+
+
+External Configuration
+======================
+
+.. autosummary::
+ :toctree: generated
+
+ external_config.ExternalSourceFormat
+ external_config.ExternalConfig
+ external_config.BigtableOptions
+ external_config.BigtableColumnFamily
+ external_config.BigtableColumn
+ external_config.CSVOptions
+ external_config.GoogleSheetsOptions
+
+
+Magics
+======
+
+.. toctree::
+ :maxdepth: 2
+
+ magics
+
+
+Enums
+=====
+
+.. autosummary::
+ :toctree: generated
+
+ enums.StandardSqlDataTypes
+
+
+Additional Types
+================
+
+Protocol buffer classes for working with the Models API.
+
+.. toctree::
+ :maxdepth: 2
+
+ gapic/v2/enums
+ gapic/v2/types
diff --git a/bigquery/docs/samples b/bigquery/docs/samples
new file mode 120000
index 000000000000..18cd9a30aaff
--- /dev/null
+++ b/bigquery/docs/samples
@@ -0,0 +1 @@
+../samples/
\ No newline at end of file
diff --git a/bigquery/docs/snippets.py b/bigquery/docs/snippets.py
new file mode 100644
index 000000000000..4c39ff912230
--- /dev/null
+++ b/bigquery/docs/snippets.py
@@ -0,0 +1,2540 @@
+# Copyright 2016 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Testable usage examples for Google BigQuery API wrapper
+
+Each example function takes a ``client`` argument (which must be an instance
+of :class:`google.cloud.bigquery.client.Client`) and uses it to perform a task
+with the API.
+
+To facilitate running the examples as system tests, each example is also passed
+a ``to_delete`` list; the function adds to the list any objects created which
+need to be deleted during teardown.
+"""
+
+import os
+import time
+
+import pytest
+import six
+
+try:
+ import fastparquet
+except (ImportError, AttributeError):
+ fastparquet = None
+try:
+ import pandas
+except (ImportError, AttributeError):
+ pandas = None
+try:
+ import pyarrow
+except (ImportError, AttributeError):
+ pyarrow = None
+
+from google.api_core import datetime_helpers
+from google.api_core.exceptions import InternalServerError
+from google.api_core.exceptions import ServiceUnavailable
+from google.api_core.exceptions import TooManyRequests
+from google.cloud import bigquery
+from google.cloud import storage
+from test_utils.retry import RetryErrors
+
+ORIGINAL_FRIENDLY_NAME = "Original friendly name"
+ORIGINAL_DESCRIPTION = "Original description"
+LOCALLY_CHANGED_FRIENDLY_NAME = "Locally-changed friendly name"
+LOCALLY_CHANGED_DESCRIPTION = "Locally-changed description"
+UPDATED_FRIENDLY_NAME = "Updated friendly name"
+UPDATED_DESCRIPTION = "Updated description"
+
+SCHEMA = [
+ bigquery.SchemaField("full_name", "STRING", mode="REQUIRED"),
+ bigquery.SchemaField("age", "INTEGER", mode="REQUIRED"),
+]
+
+ROWS = [
+ ("Phred Phlyntstone", 32),
+ ("Bharney Rhubble", 33),
+ ("Wylma Phlyntstone", 29),
+ ("Bhettye Rhubble", 27),
+]
+
+QUERY = (
+ "SELECT name FROM `bigquery-public-data.usa_names.usa_1910_2013` "
+ 'WHERE state = "TX"'
+)
+
+
+retry_429 = RetryErrors(TooManyRequests)
+retry_storage_errors = RetryErrors(
+ (TooManyRequests, InternalServerError, ServiceUnavailable)
+)
+
+
+@pytest.fixture(scope="module")
+def client():
+ return bigquery.Client()
+
+
+@pytest.fixture
+def to_delete(client):
+ doomed = []
+ yield doomed
+ for item in doomed:
+ if isinstance(item, (bigquery.Dataset, bigquery.DatasetReference)):
+ retry_429(client.delete_dataset)(item, delete_contents=True)
+ elif isinstance(item, storage.Bucket):
+ retry_storage_errors(item.delete)()
+ else:
+ retry_429(item.delete)()
+
+
+def _millis():
+ return int(time.time() * 1000)
+
+
+class _CloseOnDelete(object):
+ def __init__(self, wrapped):
+ self._wrapped = wrapped
+
+ def delete(self):
+ self._wrapped.close()
+
+
+def test_create_client_default_credentials():
+ """Create a BigQuery client with Application Default Credentials"""
+
+ # [START bigquery_client_default_credentials]
+ from google.cloud import bigquery
+
+ # If you don't specify credentials when constructing the client, the
+ # client library will look for credentials in the environment.
+ client = bigquery.Client()
+ # [END bigquery_client_default_credentials]
+
+ assert client is not None
+
+
+def test_create_table_nested_repeated_schema(client, to_delete):
+ dataset_id = "create_table_nested_repeated_{}".format(_millis())
+ dataset_ref = client.dataset(dataset_id)
+ dataset = bigquery.Dataset(dataset_ref)
+ client.create_dataset(dataset)
+ to_delete.append(dataset)
+
+ # [START bigquery_nested_repeated_schema]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # dataset_ref = client.dataset('my_dataset')
+
+ schema = [
+ bigquery.SchemaField("id", "STRING", mode="NULLABLE"),
+ bigquery.SchemaField("first_name", "STRING", mode="NULLABLE"),
+ bigquery.SchemaField("last_name", "STRING", mode="NULLABLE"),
+ bigquery.SchemaField("dob", "DATE", mode="NULLABLE"),
+ bigquery.SchemaField(
+ "addresses",
+ "RECORD",
+ mode="REPEATED",
+ fields=[
+ bigquery.SchemaField("status", "STRING", mode="NULLABLE"),
+ bigquery.SchemaField("address", "STRING", mode="NULLABLE"),
+ bigquery.SchemaField("city", "STRING", mode="NULLABLE"),
+ bigquery.SchemaField("state", "STRING", mode="NULLABLE"),
+ bigquery.SchemaField("zip", "STRING", mode="NULLABLE"),
+ bigquery.SchemaField("numberOfYears", "STRING", mode="NULLABLE"),
+ ],
+ ),
+ ]
+ table_ref = dataset_ref.table("my_table")
+ table = bigquery.Table(table_ref, schema=schema)
+ table = client.create_table(table) # API request
+
+ print("Created table {}".format(table.full_table_id))
+ # [END bigquery_nested_repeated_schema]
+
+
+def test_create_table_cmek(client, to_delete):
+ dataset_id = "create_table_cmek_{}".format(_millis())
+ dataset = bigquery.Dataset(client.dataset(dataset_id))
+ client.create_dataset(dataset)
+ to_delete.append(dataset)
+
+ # [START bigquery_create_table_cmek]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # dataset_id = 'my_dataset'
+
+ table_ref = client.dataset(dataset_id).table("my_table")
+ table = bigquery.Table(table_ref)
+
+ # Set the encryption key to use for the table.
+ # TODO: Replace this key with a key you have created in Cloud KMS.
+ kms_key_name = "projects/{}/locations/{}/keyRings/{}/cryptoKeys/{}".format(
+ "cloud-samples-tests", "us-central1", "test", "test"
+ )
+ table.encryption_configuration = bigquery.EncryptionConfiguration(
+ kms_key_name=kms_key_name
+ )
+
+ table = client.create_table(table) # API request
+
+ assert table.encryption_configuration.kms_key_name == kms_key_name
+ # [END bigquery_create_table_cmek]
+
+
+def test_create_partitioned_table(client, to_delete):
+ dataset_id = "create_table_partitioned_{}".format(_millis())
+ dataset_ref = bigquery.Dataset(client.dataset(dataset_id))
+ dataset = client.create_dataset(dataset_ref)
+ to_delete.append(dataset)
+
+ # [START bigquery_create_table_partitioned]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # dataset_ref = client.dataset('my_dataset')
+
+ table_ref = dataset_ref.table("my_partitioned_table")
+ schema = [
+ bigquery.SchemaField("name", "STRING"),
+ bigquery.SchemaField("post_abbr", "STRING"),
+ bigquery.SchemaField("date", "DATE"),
+ ]
+ table = bigquery.Table(table_ref, schema=schema)
+ table.time_partitioning = bigquery.TimePartitioning(
+ type_=bigquery.TimePartitioningType.DAY,
+ field="date", # name of column to use for partitioning
+ expiration_ms=7776000000,
+ ) # 90 days
+
+ table = client.create_table(table)
+
+ print(
+ "Created table {}, partitioned on column {}".format(
+ table.table_id, table.time_partitioning.field
+ )
+ )
+ # [END bigquery_create_table_partitioned]
+
+ assert table.time_partitioning.type_ == "DAY"
+ assert table.time_partitioning.field == "date"
+ assert table.time_partitioning.expiration_ms == 7776000000
+
+
+def test_load_and_query_partitioned_table(client, to_delete):
+ dataset_id = "load_partitioned_table_dataset_{}".format(_millis())
+ dataset = bigquery.Dataset(client.dataset(dataset_id))
+ client.create_dataset(dataset)
+ to_delete.append(dataset)
+
+ # [START bigquery_load_table_partitioned]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # dataset_id = 'my_dataset'
+ table_id = "us_states_by_date"
+
+ dataset_ref = client.dataset(dataset_id)
+ job_config = bigquery.LoadJobConfig()
+ job_config.schema = [
+ bigquery.SchemaField("name", "STRING"),
+ bigquery.SchemaField("post_abbr", "STRING"),
+ bigquery.SchemaField("date", "DATE"),
+ ]
+ job_config.skip_leading_rows = 1
+ job_config.time_partitioning = bigquery.TimePartitioning(
+ type_=bigquery.TimePartitioningType.DAY,
+ field="date", # name of column to use for partitioning
+ expiration_ms=7776000000,
+ ) # 90 days
+ uri = "gs://cloud-samples-data/bigquery/us-states/us-states-by-date.csv"
+
+ load_job = client.load_table_from_uri(
+ uri, dataset_ref.table(table_id), job_config=job_config
+ ) # API request
+
+ assert load_job.job_type == "load"
+
+ load_job.result() # Waits for table load to complete.
+
+ table = client.get_table(dataset_ref.table(table_id))
+ print("Loaded {} rows to table {}".format(table.num_rows, table_id))
+ # [END bigquery_load_table_partitioned]
+ assert table.num_rows == 50
+
+ project_id = client.project
+
+ # [START bigquery_query_partitioned_table]
+ import datetime
+
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # project_id = 'my-project'
+ # dataset_id = 'my_dataset'
+ table_id = "us_states_by_date"
+
+ sql_template = """
+ SELECT *
+ FROM `{}.{}.{}`
+ WHERE date BETWEEN @start_date AND @end_date
+ """
+ sql = sql_template.format(project_id, dataset_id, table_id)
+ job_config = bigquery.QueryJobConfig()
+ job_config.query_parameters = [
+ bigquery.ScalarQueryParameter("start_date", "DATE", datetime.date(1800, 1, 1)),
+ bigquery.ScalarQueryParameter("end_date", "DATE", datetime.date(1899, 12, 31)),
+ ]
+
+ # API request
+ query_job = client.query(sql, job_config=job_config)
+
+ rows = list(query_job)
+ print("{} states were admitted to the US in the 1800s".format(len(rows)))
+ # [END bigquery_query_partitioned_table]
+ assert len(rows) == 29
+
+
+# [START bigquery_table_exists]
+def table_exists(client, table_reference):
+ """Return if a table exists.
+
+ Args:
+ client (google.cloud.bigquery.client.Client):
+ A client to connect to the BigQuery API.
+ table_reference (google.cloud.bigquery.table.TableReference):
+ A reference to the table to look for.
+
+ Returns:
+ bool: ``True`` if the table exists, ``False`` otherwise.
+ """
+ from google.cloud.exceptions import NotFound
+
+ try:
+ client.get_table(table_reference)
+ return True
+ except NotFound:
+ return False
+
+
+# [END bigquery_table_exists]
+
+
+def test_table_exists(client, to_delete):
+ """Determine if a table exists."""
+ DATASET_ID = "get_table_dataset_{}".format(_millis())
+ TABLE_ID = "get_table_table_{}".format(_millis())
+ dataset = bigquery.Dataset(client.dataset(DATASET_ID))
+ dataset = client.create_dataset(dataset)
+ to_delete.append(dataset)
+
+ table_ref = dataset.table(TABLE_ID)
+ table = bigquery.Table(table_ref, schema=SCHEMA)
+ table = client.create_table(table)
+
+ assert table_exists(client, table_ref)
+ assert not table_exists(client, dataset.table("i_dont_exist"))
+
+
+@pytest.mark.skip(
+ reason=(
+ "update_table() is flaky "
+ "https://github.com/GoogleCloudPlatform/google-cloud-python/issues/5589"
+ )
+)
+def test_manage_table_labels(client, to_delete):
+ dataset_id = "label_table_dataset_{}".format(_millis())
+ table_id = "label_table_{}".format(_millis())
+ dataset = bigquery.Dataset(client.dataset(dataset_id))
+ client.create_dataset(dataset)
+ to_delete.append(dataset)
+
+ table = bigquery.Table(dataset.table(table_id), schema=SCHEMA)
+ table = client.create_table(table)
+
+ # [START bigquery_label_table]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # table_ref = client.dataset('my_dataset').table('my_table')
+ # table = client.get_table(table_ref) # API request
+
+ assert table.labels == {}
+ labels = {"color": "green"}
+ table.labels = labels
+
+ table = client.update_table(table, ["labels"]) # API request
+
+ assert table.labels == labels
+ # [END bigquery_label_table]
+
+ # [START bigquery_get_table_labels]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # dataset_id = 'my_dataset'
+ # table_id = 'my_table'
+
+ dataset_ref = client.dataset(dataset_id)
+ table_ref = dataset_ref.table(table_id)
+ table = client.get_table(table_ref) # API Request
+
+ # View table labels
+ print("Table ID: {}".format(table_id))
+ print("Labels:")
+ if table.labels:
+ for label, value in table.labels.items():
+ print("\t{}: {}".format(label, value))
+ else:
+ print("\tTable has no labels defined.")
+ # [END bigquery_get_table_labels]
+ assert table.labels == labels
+
+ # [START bigquery_delete_label_table]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # table_ref = client.dataset('my_dataset').table('my_table')
+ # table = client.get_table(table_ref) # API request
+
+ # This example table starts with one label
+ assert table.labels == {"color": "green"}
+ # To delete a label from a table, set its value to None
+ table.labels["color"] = None
+
+ table = client.update_table(table, ["labels"]) # API request
+
+ assert table.labels == {}
+ # [END bigquery_delete_label_table]
+
+
+@pytest.mark.skip(
+ reason=(
+ "update_table() is flaky "
+ "https://github.com/GoogleCloudPlatform/google-cloud-python/issues/5589"
+ )
+)
+def test_update_table_description(client, to_delete):
+ """Update a table's description."""
+ dataset_id = "update_table_description_dataset_{}".format(_millis())
+ table_id = "update_table_description_table_{}".format(_millis())
+ dataset = bigquery.Dataset(client.dataset(dataset_id))
+ client.create_dataset(dataset)
+ to_delete.append(dataset)
+
+ table = bigquery.Table(dataset.table(table_id), schema=SCHEMA)
+ table.description = "Original description."
+ table = client.create_table(table)
+
+ # [START bigquery_update_table_description]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # table_ref = client.dataset('my_dataset').table('my_table')
+ # table = client.get_table(table_ref) # API request
+
+ assert table.description == "Original description."
+ table.description = "Updated description."
+
+ table = client.update_table(table, ["description"]) # API request
+
+ assert table.description == "Updated description."
+ # [END bigquery_update_table_description]
+
+
+@pytest.mark.skip(
+ reason=(
+ "update_table() is flaky "
+ "https://github.com/GoogleCloudPlatform/google-cloud-python/issues/5589"
+ )
+)
+def test_update_table_expiration(client, to_delete):
+ """Update a table's expiration time."""
+ dataset_id = "update_table_expiration_dataset_{}".format(_millis())
+ table_id = "update_table_expiration_table_{}".format(_millis())
+ dataset = bigquery.Dataset(client.dataset(dataset_id))
+ client.create_dataset(dataset)
+ to_delete.append(dataset)
+
+ table = bigquery.Table(dataset.table(table_id), schema=SCHEMA)
+ table = client.create_table(table)
+
+ # [START bigquery_update_table_expiration]
+ import datetime
+ import pytz
+
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # table_ref = client.dataset('my_dataset').table('my_table')
+ # table = client.get_table(table_ref) # API request
+
+ assert table.expires is None
+
+ # set table to expire 5 days from now
+ expiration = datetime.datetime.now(pytz.utc) + datetime.timedelta(days=5)
+ table.expires = expiration
+ table = client.update_table(table, ["expires"]) # API request
+
+ # expiration is stored in milliseconds
+ margin = datetime.timedelta(microseconds=1000)
+ assert expiration - margin <= table.expires <= expiration + margin
+ # [END bigquery_update_table_expiration]
+
+
+@pytest.mark.skip(
+ reason=(
+ "update_table() is flaky "
+ "https://github.com/GoogleCloudPlatform/google-cloud-python/issues/5589"
+ )
+)
+def test_relax_column(client, to_delete):
+ """Updates a schema field from required to nullable."""
+ dataset_id = "relax_column_dataset_{}".format(_millis())
+ table_id = "relax_column_table_{}".format(_millis())
+ dataset = bigquery.Dataset(client.dataset(dataset_id))
+ dataset = client.create_dataset(dataset)
+ to_delete.append(dataset)
+
+ # [START bigquery_relax_column]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # dataset_id = 'my_dataset'
+ # table_id = 'my_table'
+
+ original_schema = [
+ bigquery.SchemaField("full_name", "STRING", mode="REQUIRED"),
+ bigquery.SchemaField("age", "INTEGER", mode="REQUIRED"),
+ ]
+ table_ref = client.dataset(dataset_id).table(table_id)
+ table = bigquery.Table(table_ref, schema=original_schema)
+ table = client.create_table(table)
+ assert all(field.mode == "REQUIRED" for field in table.schema)
+
+ # SchemaField properties cannot be edited after initialization.
+ # To make changes, construct new SchemaField objects.
+ relaxed_schema = [
+ bigquery.SchemaField("full_name", "STRING", mode="NULLABLE"),
+ bigquery.SchemaField("age", "INTEGER", mode="NULLABLE"),
+ ]
+ table.schema = relaxed_schema
+ table = client.update_table(table, ["schema"])
+
+ assert all(field.mode == "NULLABLE" for field in table.schema)
+ # [END bigquery_relax_column]
+
+
+@pytest.mark.skip(
+ reason=(
+ "update_table() is flaky "
+ "https://github.com/GoogleCloudPlatform/google-cloud-python/issues/5589"
+ )
+)
+def test_update_table_cmek(client, to_delete):
+ """Patch a table's metadata."""
+ dataset_id = "update_table_cmek_{}".format(_millis())
+ table_id = "update_table_cmek_{}".format(_millis())
+ dataset = bigquery.Dataset(client.dataset(dataset_id))
+ client.create_dataset(dataset)
+ to_delete.append(dataset)
+
+ table = bigquery.Table(dataset.table(table_id))
+ original_kms_key_name = "projects/{}/locations/{}/keyRings/{}/cryptoKeys/{}".format(
+ "cloud-samples-tests", "us-central1", "test", "test"
+ )
+ table.encryption_configuration = bigquery.EncryptionConfiguration(
+ kms_key_name=original_kms_key_name
+ )
+ table = client.create_table(table)
+
+ # [START bigquery_update_table_cmek]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+
+ assert table.encryption_configuration.kms_key_name == original_kms_key_name
+
+ # Set a new encryption key to use for the destination.
+ # TODO: Replace this key with a key you have created in KMS.
+ updated_kms_key_name = (
+ "projects/cloud-samples-tests/locations/us-central1/"
+ "keyRings/test/cryptoKeys/otherkey"
+ )
+ table.encryption_configuration = bigquery.EncryptionConfiguration(
+ kms_key_name=updated_kms_key_name
+ )
+
+ table = client.update_table(table, ["encryption_configuration"]) # API request
+
+ assert table.encryption_configuration.kms_key_name == updated_kms_key_name
+ assert original_kms_key_name != updated_kms_key_name
+ # [END bigquery_update_table_cmek]
+
+
+@pytest.mark.skip(
+ reason=(
+ "update_table() is flaky "
+ "https://github.com/GoogleCloudPlatform/google-cloud-python/issues/5589"
+ )
+)
+def test_manage_views(client, to_delete):
+ project = client.project
+ source_dataset_id = "source_dataset_{}".format(_millis())
+ source_dataset_ref = client.dataset(source_dataset_id)
+ source_dataset = bigquery.Dataset(source_dataset_ref)
+ source_dataset = client.create_dataset(source_dataset)
+ to_delete.append(source_dataset)
+
+ job_config = bigquery.LoadJobConfig()
+ job_config.schema = [
+ bigquery.SchemaField("name", "STRING"),
+ bigquery.SchemaField("post_abbr", "STRING"),
+ ]
+ job_config.skip_leading_rows = 1
+ uri = "gs://cloud-samples-data/bigquery/us-states/us-states.csv"
+ source_table_id = "us_states"
+ load_job = client.load_table_from_uri(
+ uri, source_dataset.table(source_table_id), job_config=job_config
+ )
+ load_job.result()
+
+ shared_dataset_id = "shared_dataset_{}".format(_millis())
+ shared_dataset_ref = client.dataset(shared_dataset_id)
+ shared_dataset = bigquery.Dataset(shared_dataset_ref)
+ shared_dataset = client.create_dataset(shared_dataset)
+ to_delete.append(shared_dataset)
+
+ # [START bigquery_create_view]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # project = 'my-project'
+ # source_dataset_id = 'my_source_dataset'
+ # source_table_id = 'us_states'
+ # shared_dataset_ref = client.dataset('my_shared_dataset')
+
+ # This example shows how to create a shared view of a source table of
+ # US States. The source table contains all 50 states, while the view will
+ # contain only states with names starting with 'W'.
+ view_ref = shared_dataset_ref.table("my_shared_view")
+ view = bigquery.Table(view_ref)
+ sql_template = 'SELECT name, post_abbr FROM `{}.{}.{}` WHERE name LIKE "W%"'
+ view.view_query = sql_template.format(project, source_dataset_id, source_table_id)
+ view = client.create_table(view) # API request
+
+ print("Successfully created view at {}".format(view.full_table_id))
+ # [END bigquery_create_view]
+
+ # [START bigquery_update_view_query]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # project = 'my-project'
+ # source_dataset_id = 'my_source_dataset'
+ # source_table_id = 'us_states'
+ # shared_dataset_ref = client.dataset('my_shared_dataset')
+
+ # This example shows how to update a shared view of a source table of
+ # US States. The view's query will be updated to contain only states with
+ # names starting with 'M'.
+ view_ref = shared_dataset_ref.table("my_shared_view")
+ view = bigquery.Table(view_ref)
+ sql_template = 'SELECT name, post_abbr FROM `{}.{}.{}` WHERE name LIKE "M%"'
+ view.view_query = sql_template.format(project, source_dataset_id, source_table_id)
+ view = client.update_table(view, ["view_query"]) # API request
+ # [END bigquery_update_view_query]
+
+ # [START bigquery_get_view]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # shared_dataset_id = 'my_shared_dataset'
+
+ view_ref = client.dataset(shared_dataset_id).table("my_shared_view")
+ view = client.get_table(view_ref) # API Request
+
+ # Display view properties
+ print("View at {}".format(view.full_table_id))
+ print("View Query:\n{}".format(view.view_query))
+ # [END bigquery_get_view]
+ assert view.view_query is not None
+
+ analyst_group_email = "example-analyst-group@google.com"
+ # [START bigquery_grant_view_access]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+
+ # Assign access controls to the dataset containing the view
+ # shared_dataset_id = 'my_shared_dataset'
+ # analyst_group_email = 'data_analysts@example.com'
+ shared_dataset = client.get_dataset(
+ client.dataset(shared_dataset_id)
+ ) # API request
+ access_entries = shared_dataset.access_entries
+ access_entries.append(
+ bigquery.AccessEntry("READER", "groupByEmail", analyst_group_email)
+ )
+ shared_dataset.access_entries = access_entries
+ shared_dataset = client.update_dataset(
+ shared_dataset, ["access_entries"]
+ ) # API request
+
+ # Authorize the view to access the source dataset
+ # project = 'my-project'
+ # source_dataset_id = 'my_source_dataset'
+ source_dataset = client.get_dataset(
+ client.dataset(source_dataset_id)
+ ) # API request
+ view_reference = {
+ "projectId": project,
+ "datasetId": shared_dataset_id,
+ "tableId": "my_shared_view",
+ }
+ access_entries = source_dataset.access_entries
+ access_entries.append(bigquery.AccessEntry(None, "view", view_reference))
+ source_dataset.access_entries = access_entries
+ source_dataset = client.update_dataset(
+ source_dataset, ["access_entries"]
+ ) # API request
+ # [END bigquery_grant_view_access]
+
+
+def test_table_insert_rows(client, to_delete):
+ """Insert / fetch table data."""
+ dataset_id = "table_insert_rows_dataset_{}".format(_millis())
+ table_id = "table_insert_rows_table_{}".format(_millis())
+ dataset = bigquery.Dataset(client.dataset(dataset_id))
+ dataset = client.create_dataset(dataset)
+ dataset.location = "US"
+ to_delete.append(dataset)
+
+ table = bigquery.Table(dataset.table(table_id), schema=SCHEMA)
+ table = client.create_table(table)
+
+ # [START bigquery_table_insert_rows]
+ # TODO(developer): Uncomment the lines below and replace with your values.
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # dataset_id = 'my_dataset' # replace with your dataset ID
+ # For this sample, the table must already exist and have a defined schema
+ # table_id = 'my_table' # replace with your table ID
+ # table_ref = client.dataset(dataset_id).table(table_id)
+ # table = client.get_table(table_ref) # API request
+
+ rows_to_insert = [(u"Phred Phlyntstone", 32), (u"Wylma Phlyntstone", 29)]
+
+ errors = client.insert_rows(table, rows_to_insert) # API request
+
+ assert errors == []
+ # [END bigquery_table_insert_rows]
+
+
+def test_load_table_from_file(client, to_delete):
+ """Upload table data from a CSV file."""
+ dataset_id = "load_table_from_file_dataset_{}".format(_millis())
+ table_id = "load_table_from_file_table_{}".format(_millis())
+ dataset = bigquery.Dataset(client.dataset(dataset_id))
+ dataset.location = "US"
+ client.create_dataset(dataset)
+ to_delete.append(dataset)
+ snippets_dir = os.path.abspath(os.path.dirname(__file__))
+ filename = os.path.join(
+ snippets_dir, "..", "..", "bigquery", "tests", "data", "people.csv"
+ )
+
+ # [START bigquery_load_from_file]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # filename = '/path/to/file.csv'
+ # dataset_id = 'my_dataset'
+ # table_id = 'my_table'
+
+ dataset_ref = client.dataset(dataset_id)
+ table_ref = dataset_ref.table(table_id)
+ job_config = bigquery.LoadJobConfig()
+ job_config.source_format = bigquery.SourceFormat.CSV
+ job_config.skip_leading_rows = 1
+ job_config.autodetect = True
+
+ with open(filename, "rb") as source_file:
+ job = client.load_table_from_file(source_file, table_ref, job_config=job_config)
+
+ job.result() # Waits for table load to complete.
+
+ print("Loaded {} rows into {}:{}.".format(job.output_rows, dataset_id, table_id))
+ # [END bigquery_load_from_file]
+
+ table = client.get_table(table_ref)
+ rows = list(client.list_rows(table)) # API request
+
+ assert len(rows) == 2
+ # Order is not preserved, so compare individually
+ row1 = bigquery.Row(("Wylma Phlyntstone", 29), {"full_name": 0, "age": 1})
+ assert row1 in rows
+ row2 = bigquery.Row(("Phred Phlyntstone", 32), {"full_name": 0, "age": 1})
+ assert row2 in rows
+
+
+def test_load_table_from_uri_avro(client, to_delete, capsys):
+ dataset_id = "load_table_from_uri_avro_{}".format(_millis())
+ dataset = bigquery.Dataset(client.dataset(dataset_id))
+ client.create_dataset(dataset)
+ to_delete.append(dataset)
+
+ # [START bigquery_load_table_gcs_avro]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # dataset_id = 'my_dataset'
+
+ dataset_ref = client.dataset(dataset_id)
+ job_config = bigquery.LoadJobConfig()
+ job_config.source_format = bigquery.SourceFormat.AVRO
+ uri = "gs://cloud-samples-data/bigquery/us-states/us-states.avro"
+
+ load_job = client.load_table_from_uri(
+ uri, dataset_ref.table("us_states"), job_config=job_config
+ ) # API request
+ print("Starting job {}".format(load_job.job_id))
+
+ load_job.result() # Waits for table load to complete.
+ print("Job finished.")
+
+ destination_table = client.get_table(dataset_ref.table("us_states"))
+ print("Loaded {} rows.".format(destination_table.num_rows))
+ # [END bigquery_load_table_gcs_avro]
+
+ out, _ = capsys.readouterr()
+ assert "Loaded 50 rows." in out
+
+
+def test_load_table_from_uri_csv(client, to_delete, capsys):
+ dataset_id = "load_table_from_uri_csv_{}".format(_millis())
+ dataset = bigquery.Dataset(client.dataset(dataset_id))
+ client.create_dataset(dataset)
+ to_delete.append(dataset)
+
+ # [START bigquery_load_table_gcs_csv]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # dataset_id = 'my_dataset'
+
+ dataset_ref = client.dataset(dataset_id)
+ job_config = bigquery.LoadJobConfig()
+ job_config.schema = [
+ bigquery.SchemaField("name", "STRING"),
+ bigquery.SchemaField("post_abbr", "STRING"),
+ ]
+ job_config.skip_leading_rows = 1
+ # The source format defaults to CSV, so the line below is optional.
+ job_config.source_format = bigquery.SourceFormat.CSV
+ uri = "gs://cloud-samples-data/bigquery/us-states/us-states.csv"
+
+ load_job = client.load_table_from_uri(
+ uri, dataset_ref.table("us_states"), job_config=job_config
+ ) # API request
+ print("Starting job {}".format(load_job.job_id))
+
+ load_job.result() # Waits for table load to complete.
+ print("Job finished.")
+
+ destination_table = client.get_table(dataset_ref.table("us_states"))
+ print("Loaded {} rows.".format(destination_table.num_rows))
+ # [END bigquery_load_table_gcs_csv]
+
+ out, _ = capsys.readouterr()
+ assert "Loaded 50 rows." in out
+
+
+def test_load_table_from_uri_json(client, to_delete, capsys):
+ dataset_id = "load_table_from_uri_json_{}".format(_millis())
+ dataset = bigquery.Dataset(client.dataset(dataset_id))
+ dataset.location = "US"
+ client.create_dataset(dataset)
+ to_delete.append(dataset)
+
+ # [START bigquery_load_table_gcs_json]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # dataset_id = 'my_dataset'
+
+ dataset_ref = client.dataset(dataset_id)
+ job_config = bigquery.LoadJobConfig()
+ job_config.schema = [
+ bigquery.SchemaField("name", "STRING"),
+ bigquery.SchemaField("post_abbr", "STRING"),
+ ]
+ job_config.source_format = bigquery.SourceFormat.NEWLINE_DELIMITED_JSON
+ uri = "gs://cloud-samples-data/bigquery/us-states/us-states.json"
+
+ load_job = client.load_table_from_uri(
+ uri,
+ dataset_ref.table("us_states"),
+ location="US", # Location must match that of the destination dataset.
+ job_config=job_config,
+ ) # API request
+ print("Starting job {}".format(load_job.job_id))
+
+ load_job.result() # Waits for table load to complete.
+ print("Job finished.")
+
+ destination_table = client.get_table(dataset_ref.table("us_states"))
+ print("Loaded {} rows.".format(destination_table.num_rows))
+ # [END bigquery_load_table_gcs_json]
+
+ out, _ = capsys.readouterr()
+ assert "Loaded 50 rows." in out
+
+
+def test_load_table_from_uri_cmek(client, to_delete):
+ dataset_id = "load_table_from_uri_cmek_{}".format(_millis())
+ dataset = bigquery.Dataset(client.dataset(dataset_id))
+ dataset.location = "US"
+ client.create_dataset(dataset)
+ to_delete.append(dataset)
+
+ # [START bigquery_load_table_gcs_json_cmek]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # dataset_id = 'my_dataset'
+
+ dataset_ref = client.dataset(dataset_id)
+ job_config = bigquery.LoadJobConfig()
+ job_config.autodetect = True
+ job_config.source_format = bigquery.SourceFormat.NEWLINE_DELIMITED_JSON
+
+ # Set the encryption key to use for the destination.
+ # TODO: Replace this key with a key you have created in KMS.
+ kms_key_name = "projects/{}/locations/{}/keyRings/{}/cryptoKeys/{}".format(
+ "cloud-samples-tests", "us-central1", "test", "test"
+ )
+ encryption_config = bigquery.EncryptionConfiguration(kms_key_name=kms_key_name)
+ job_config.destination_encryption_configuration = encryption_config
+ uri = "gs://cloud-samples-data/bigquery/us-states/us-states.json"
+
+ load_job = client.load_table_from_uri(
+ uri,
+ dataset_ref.table("us_states"),
+ location="US", # Location must match that of the destination dataset.
+ job_config=job_config,
+ ) # API request
+
+ assert load_job.job_type == "load"
+
+ load_job.result() # Waits for table load to complete.
+
+ assert load_job.state == "DONE"
+ table = client.get_table(dataset_ref.table("us_states"))
+ assert table.encryption_configuration.kms_key_name == kms_key_name
+ # [END bigquery_load_table_gcs_json_cmek]
+
+
+def test_load_table_from_uri_parquet(client, to_delete, capsys):
+ dataset_id = "load_table_from_uri_parquet_{}".format(_millis())
+ dataset = bigquery.Dataset(client.dataset(dataset_id))
+ client.create_dataset(dataset)
+ to_delete.append(dataset)
+
+ # [START bigquery_load_table_gcs_parquet]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # dataset_id = 'my_dataset'
+
+ dataset_ref = client.dataset(dataset_id)
+ job_config = bigquery.LoadJobConfig()
+ job_config.source_format = bigquery.SourceFormat.PARQUET
+ uri = "gs://cloud-samples-data/bigquery/us-states/us-states.parquet"
+
+ load_job = client.load_table_from_uri(
+ uri, dataset_ref.table("us_states"), job_config=job_config
+ ) # API request
+ print("Starting job {}".format(load_job.job_id))
+
+ load_job.result() # Waits for table load to complete.
+ print("Job finished.")
+
+ destination_table = client.get_table(dataset_ref.table("us_states"))
+ print("Loaded {} rows.".format(destination_table.num_rows))
+ # [END bigquery_load_table_gcs_parquet]
+
+ out, _ = capsys.readouterr()
+ assert "Loaded 50 rows." in out
+
+
+def test_load_table_from_uri_orc(client, to_delete, capsys):
+ dataset_id = "load_table_from_uri_orc_{}".format(_millis())
+ dataset = bigquery.Dataset(client.dataset(dataset_id))
+ client.create_dataset(dataset)
+ to_delete.append(dataset)
+
+ # [START bigquery_load_table_gcs_orc]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # dataset_id = 'my_dataset'
+
+ dataset_ref = client.dataset(dataset_id)
+ job_config = bigquery.LoadJobConfig()
+ job_config.source_format = bigquery.SourceFormat.ORC
+ uri = "gs://cloud-samples-data/bigquery/us-states/us-states.orc"
+
+ load_job = client.load_table_from_uri(
+ uri, dataset_ref.table("us_states"), job_config=job_config
+ ) # API request
+ print("Starting job {}".format(load_job.job_id))
+
+ load_job.result() # Waits for table load to complete.
+ print("Job finished.")
+
+ destination_table = client.get_table(dataset_ref.table("us_states"))
+ print("Loaded {} rows.".format(destination_table.num_rows))
+ # [END bigquery_load_table_gcs_orc]
+
+ out, _ = capsys.readouterr()
+ assert "Loaded 50 rows." in out
+
+
+def test_load_table_from_uri_autodetect(client, to_delete, capsys):
+ """Load table from a GCS URI using various formats and auto-detected schema
+
+ Each file format has its own tested load from URI sample. Because most of
+ the code is common for autodetect, append, and truncate, this sample
+ includes snippets for all supported formats but only calls a single load
+ job.
+
+ This code snippet is made up of shared code, then format-specific code,
+ followed by more shared code. Note that only the last format in the
+ format-specific code section will be tested in this test.
+ """
+ dataset_id = "load_table_from_uri_auto_{}".format(_millis())
+ dataset = bigquery.Dataset(client.dataset(dataset_id))
+ client.create_dataset(dataset)
+ to_delete.append(dataset)
+
+ # Shared code
+ # [START bigquery_load_table_gcs_csv_autodetect]
+ # [START bigquery_load_table_gcs_json_autodetect]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # dataset_id = 'my_dataset'
+
+ dataset_ref = client.dataset(dataset_id)
+ job_config = bigquery.LoadJobConfig()
+ job_config.autodetect = True
+ # [END bigquery_load_table_gcs_csv_autodetect]
+ # [END bigquery_load_table_gcs_json_autodetect]
+
+ # Format-specific code
+ # [START bigquery_load_table_gcs_csv_autodetect]
+ job_config.skip_leading_rows = 1
+ # The source format defaults to CSV, so the line below is optional.
+ job_config.source_format = bigquery.SourceFormat.CSV
+ uri = "gs://cloud-samples-data/bigquery/us-states/us-states.csv"
+ # [END bigquery_load_table_gcs_csv_autodetect]
+ # unset csv-specific attribute
+ del job_config._properties["load"]["skipLeadingRows"]
+
+ # [START bigquery_load_table_gcs_json_autodetect]
+ job_config.source_format = bigquery.SourceFormat.NEWLINE_DELIMITED_JSON
+ uri = "gs://cloud-samples-data/bigquery/us-states/us-states.json"
+ # [END bigquery_load_table_gcs_json_autodetect]
+
+ # Shared code
+ # [START bigquery_load_table_gcs_csv_autodetect]
+ # [START bigquery_load_table_gcs_json_autodetect]
+ load_job = client.load_table_from_uri(
+ uri, dataset_ref.table("us_states"), job_config=job_config
+ ) # API request
+ print("Starting job {}".format(load_job.job_id))
+
+ load_job.result() # Waits for table load to complete.
+ print("Job finished.")
+
+ destination_table = client.get_table(dataset_ref.table("us_states"))
+ print("Loaded {} rows.".format(destination_table.num_rows))
+ # [END bigquery_load_table_gcs_csv_autodetect]
+ # [END bigquery_load_table_gcs_json_autodetect]
+
+ out, _ = capsys.readouterr()
+ assert "Loaded 50 rows." in out
+
+
+def test_load_table_from_uri_truncate(client, to_delete, capsys):
+ """Replaces table data with data from a GCS URI using various formats
+
+ Each file format has its own tested load from URI sample. Because most of
+ the code is common for autodetect, append, and truncate, this sample
+ includes snippets for all supported formats but only calls a single load
+ job.
+
+ This code snippet is made up of shared code, then format-specific code,
+ followed by more shared code. Note that only the last format in the
+ format-specific code section will be tested in this test.
+ """
+ dataset_id = "load_table_from_uri_trunc_{}".format(_millis())
+ dataset = bigquery.Dataset(client.dataset(dataset_id))
+ client.create_dataset(dataset)
+ to_delete.append(dataset)
+
+ job_config = bigquery.LoadJobConfig()
+ job_config.schema = [
+ bigquery.SchemaField("name", "STRING"),
+ bigquery.SchemaField("post_abbr", "STRING"),
+ ]
+ table_ref = dataset.table("us_states")
+ body = six.BytesIO(b"Washington,WA")
+ client.load_table_from_file(body, table_ref, job_config=job_config).result()
+ previous_rows = client.get_table(table_ref).num_rows
+ assert previous_rows > 0
+
+ # Shared code
+ # [START bigquery_load_table_gcs_avro_truncate]
+ # [START bigquery_load_table_gcs_csv_truncate]
+ # [START bigquery_load_table_gcs_json_truncate]
+ # [START bigquery_load_table_gcs_parquet_truncate]
+ # [START bigquery_load_table_gcs_orc_truncate]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # table_ref = client.dataset('my_dataset').table('existing_table')
+
+ job_config = bigquery.LoadJobConfig()
+ job_config.write_disposition = bigquery.WriteDisposition.WRITE_TRUNCATE
+ # [END bigquery_load_table_gcs_avro_truncate]
+ # [END bigquery_load_table_gcs_csv_truncate]
+ # [END bigquery_load_table_gcs_json_truncate]
+ # [END bigquery_load_table_gcs_parquet_truncate]
+ # [END bigquery_load_table_gcs_orc_truncate]
+
+ # Format-specific code
+ # [START bigquery_load_table_gcs_avro_truncate]
+ job_config.source_format = bigquery.SourceFormat.AVRO
+ uri = "gs://cloud-samples-data/bigquery/us-states/us-states.avro"
+ # [END bigquery_load_table_gcs_avro_truncate]
+
+ # [START bigquery_load_table_gcs_csv_truncate]
+ job_config.skip_leading_rows = 1
+ # The source format defaults to CSV, so the line below is optional.
+ job_config.source_format = bigquery.SourceFormat.CSV
+ uri = "gs://cloud-samples-data/bigquery/us-states/us-states.csv"
+ # [END bigquery_load_table_gcs_csv_truncate]
+ # unset csv-specific attribute
+ del job_config._properties["load"]["skipLeadingRows"]
+
+ # [START bigquery_load_table_gcs_json_truncate]
+ job_config.source_format = bigquery.SourceFormat.NEWLINE_DELIMITED_JSON
+ uri = "gs://cloud-samples-data/bigquery/us-states/us-states.json"
+ # [END bigquery_load_table_gcs_json_truncate]
+
+ # [START bigquery_load_table_gcs_parquet_truncate]
+ job_config.source_format = bigquery.SourceFormat.PARQUET
+ uri = "gs://cloud-samples-data/bigquery/us-states/us-states.parquet"
+ # [END bigquery_load_table_gcs_parquet_truncate]
+
+ # [START bigquery_load_table_gcs_orc_truncate]
+ job_config.source_format = bigquery.SourceFormat.ORC
+ uri = "gs://cloud-samples-data/bigquery/us-states/us-states.orc"
+ # [END bigquery_load_table_gcs_orc_truncate]
+
+ # Shared code
+ # [START bigquery_load_table_gcs_avro_truncate]
+ # [START bigquery_load_table_gcs_csv_truncate]
+ # [START bigquery_load_table_gcs_json_truncate]
+ # [START bigquery_load_table_gcs_parquet_truncate]
+ # [START bigquery_load_table_gcs_orc_truncate]
+ load_job = client.load_table_from_uri(
+ uri, table_ref, job_config=job_config
+ ) # API request
+ print("Starting job {}".format(load_job.job_id))
+
+ load_job.result() # Waits for table load to complete.
+ print("Job finished.")
+
+ destination_table = client.get_table(table_ref)
+ print("Loaded {} rows.".format(destination_table.num_rows))
+ # [END bigquery_load_table_gcs_avro_truncate]
+ # [END bigquery_load_table_gcs_csv_truncate]
+ # [END bigquery_load_table_gcs_json_truncate]
+ # [END bigquery_load_table_gcs_parquet_truncate]
+ # [END bigquery_load_table_gcs_orc_truncate]
+
+ out, _ = capsys.readouterr()
+ assert "Loaded 50 rows." in out
+
+
+def test_load_table_add_column(client, to_delete):
+ dataset_id = "load_table_add_column_{}".format(_millis())
+ dataset_ref = client.dataset(dataset_id)
+ dataset = bigquery.Dataset(dataset_ref)
+ dataset.location = "US"
+ dataset = client.create_dataset(dataset)
+ to_delete.append(dataset)
+
+ snippets_dir = os.path.abspath(os.path.dirname(__file__))
+ filepath = os.path.join(
+ snippets_dir, "..", "..", "bigquery", "tests", "data", "people.csv"
+ )
+ table_ref = dataset_ref.table("my_table")
+ old_schema = [bigquery.SchemaField("full_name", "STRING", mode="REQUIRED")]
+ table = client.create_table(bigquery.Table(table_ref, schema=old_schema))
+
+ # [START bigquery_add_column_load_append]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # dataset_ref = client.dataset('my_dataset')
+ # filepath = 'path/to/your_file.csv'
+
+ # Retrieves the destination table and checks the length of the schema
+ table_id = "my_table"
+ table_ref = dataset_ref.table(table_id)
+ table = client.get_table(table_ref)
+ print("Table {} contains {} columns.".format(table_id, len(table.schema)))
+
+ # Configures the load job to append the data to the destination table,
+ # allowing field addition
+ job_config = bigquery.LoadJobConfig()
+ job_config.write_disposition = bigquery.WriteDisposition.WRITE_APPEND
+ job_config.schema_update_options = [
+ bigquery.SchemaUpdateOption.ALLOW_FIELD_ADDITION
+ ]
+ # In this example, the existing table contains only the 'full_name' column.
+ # 'REQUIRED' fields cannot be added to an existing schema, so the
+ # additional column must be 'NULLABLE'.
+ job_config.schema = [
+ bigquery.SchemaField("full_name", "STRING", mode="REQUIRED"),
+ bigquery.SchemaField("age", "INTEGER", mode="NULLABLE"),
+ ]
+ job_config.source_format = bigquery.SourceFormat.CSV
+ job_config.skip_leading_rows = 1
+
+ with open(filepath, "rb") as source_file:
+ job = client.load_table_from_file(
+ source_file,
+ table_ref,
+ location="US", # Must match the destination dataset location.
+ job_config=job_config,
+ ) # API request
+
+ job.result() # Waits for table load to complete.
+ print(
+ "Loaded {} rows into {}:{}.".format(
+ job.output_rows, dataset_id, table_ref.table_id
+ )
+ )
+
+ # Checks the updated length of the schema
+ table = client.get_table(table)
+ print("Table {} now contains {} columns.".format(table_id, len(table.schema)))
+ # [END bigquery_add_column_load_append]
+ assert len(table.schema) == 2
+ assert table.num_rows > 0
+
+
+def test_load_table_relax_column(client, to_delete):
+ dataset_id = "load_table_relax_column_{}".format(_millis())
+ dataset_ref = client.dataset(dataset_id)
+ dataset = bigquery.Dataset(dataset_ref)
+ dataset.location = "US"
+ dataset = client.create_dataset(dataset)
+ to_delete.append(dataset)
+
+ snippets_dir = os.path.abspath(os.path.dirname(__file__))
+ filepath = os.path.join(
+ snippets_dir, "..", "..", "bigquery", "tests", "data", "people.csv"
+ )
+ table_ref = dataset_ref.table("my_table")
+ old_schema = [
+ bigquery.SchemaField("full_name", "STRING", mode="REQUIRED"),
+ bigquery.SchemaField("age", "INTEGER", mode="REQUIRED"),
+ bigquery.SchemaField("favorite_color", "STRING", mode="REQUIRED"),
+ ]
+ table = client.create_table(bigquery.Table(table_ref, schema=old_schema))
+
+ # [START bigquery_relax_column_load_append]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # dataset_ref = client.dataset('my_dataset')
+ # filepath = 'path/to/your_file.csv'
+
+ # Retrieves the destination table and checks the number of required fields
+ table_id = "my_table"
+ table_ref = dataset_ref.table(table_id)
+ table = client.get_table(table_ref)
+ original_required_fields = sum(field.mode == "REQUIRED" for field in table.schema)
+ # In this example, the existing table has 3 required fields.
+ print("{} fields in the schema are required.".format(original_required_fields))
+
+ # Configures the load job to append the data to a destination table,
+ # allowing field relaxation
+ job_config = bigquery.LoadJobConfig()
+ job_config.write_disposition = bigquery.WriteDisposition.WRITE_APPEND
+ job_config.schema_update_options = [
+ bigquery.SchemaUpdateOption.ALLOW_FIELD_RELAXATION
+ ]
+ # In this example, the existing table contains three required fields
+ # ('full_name', 'age', and 'favorite_color'), while the data to load
+ # contains only the first two fields.
+ job_config.schema = [
+ bigquery.SchemaField("full_name", "STRING", mode="REQUIRED"),
+ bigquery.SchemaField("age", "INTEGER", mode="REQUIRED"),
+ ]
+ job_config.source_format = bigquery.SourceFormat.CSV
+ job_config.skip_leading_rows = 1
+
+ with open(filepath, "rb") as source_file:
+ job = client.load_table_from_file(
+ source_file,
+ table_ref,
+ location="US", # Must match the destination dataset location.
+ job_config=job_config,
+ ) # API request
+
+ job.result() # Waits for table load to complete.
+ print(
+ "Loaded {} rows into {}:{}.".format(
+ job.output_rows, dataset_id, table_ref.table_id
+ )
+ )
+
+ # Checks the updated number of required fields
+ table = client.get_table(table)
+ current_required_fields = sum(field.mode == "REQUIRED" for field in table.schema)
+ print("{} fields in the schema are now required.".format(current_required_fields))
+ # [END bigquery_relax_column_load_append]
+ assert original_required_fields - current_required_fields == 1
+ assert len(table.schema) == 3
+ assert table.schema[2].mode == "NULLABLE"
+ assert table.num_rows > 0
+
+
+def test_copy_table(client, to_delete):
+ dataset_id = "copy_table_dataset_{}".format(_millis())
+ dest_dataset = bigquery.Dataset(client.dataset(dataset_id))
+ dest_dataset.location = "US"
+ dest_dataset = client.create_dataset(dest_dataset)
+ to_delete.append(dest_dataset)
+
+ # [START bigquery_copy_table]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+
+ source_dataset = client.dataset("samples", project="bigquery-public-data")
+ source_table_ref = source_dataset.table("shakespeare")
+
+ # dataset_id = 'my_dataset'
+ dest_table_ref = client.dataset(dataset_id).table("destination_table")
+
+ job = client.copy_table(
+ source_table_ref,
+ dest_table_ref,
+ # Location must match that of the source and destination tables.
+ location="US",
+ ) # API request
+
+ job.result() # Waits for job to complete.
+
+ assert job.state == "DONE"
+ dest_table = client.get_table(dest_table_ref) # API request
+ assert dest_table.num_rows > 0
+ # [END bigquery_copy_table]
+
+
+def test_copy_table_multiple_source(client, to_delete):
+ dest_dataset_id = "dest_dataset_{}".format(_millis())
+ dest_dataset = bigquery.Dataset(client.dataset(dest_dataset_id))
+ dest_dataset.location = "US"
+ dest_dataset = client.create_dataset(dest_dataset)
+ to_delete.append(dest_dataset)
+
+ source_dataset_id = "source_dataset_{}".format(_millis())
+ source_dataset = bigquery.Dataset(client.dataset(source_dataset_id))
+ source_dataset.location = "US"
+ source_dataset = client.create_dataset(source_dataset)
+ to_delete.append(source_dataset)
+
+ schema = [
+ bigquery.SchemaField("name", "STRING"),
+ bigquery.SchemaField("post_abbr", "STRING"),
+ ]
+
+ table_data = {"table1": b"Washington,WA", "table2": b"California,CA"}
+ for table_id, data in table_data.items():
+ table_ref = source_dataset.table(table_id)
+ job_config = bigquery.LoadJobConfig()
+ job_config.schema = schema
+ body = six.BytesIO(data)
+ client.load_table_from_file(
+ body,
+ table_ref,
+ # Location must match that of the destination dataset.
+ location="US",
+ job_config=job_config,
+ ).result()
+
+ # [START bigquery_copy_table_multiple_source]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # source_dataset_id = 'my_source_dataset'
+ # dest_dataset_id = 'my_destination_dataset'
+
+ table1_ref = client.dataset(source_dataset_id).table("table1")
+ table2_ref = client.dataset(source_dataset_id).table("table2")
+ dest_table_ref = client.dataset(dest_dataset_id).table("destination_table")
+
+ job = client.copy_table(
+ [table1_ref, table2_ref],
+ dest_table_ref,
+ # Location must match that of the source and destination tables.
+ location="US",
+ ) # API request
+ job.result() # Waits for job to complete.
+
+ assert job.state == "DONE"
+ dest_table = client.get_table(dest_table_ref) # API request
+ assert dest_table.num_rows > 0
+ # [END bigquery_copy_table_multiple_source]
+
+ assert dest_table.num_rows == 2
+
+
+def test_copy_table_cmek(client, to_delete):
+ dataset_id = "copy_table_cmek_{}".format(_millis())
+ dest_dataset = bigquery.Dataset(client.dataset(dataset_id))
+ dest_dataset.location = "US"
+ dest_dataset = client.create_dataset(dest_dataset)
+ to_delete.append(dest_dataset)
+
+ # [START bigquery_copy_table_cmek]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+
+ source_dataset = bigquery.DatasetReference("bigquery-public-data", "samples")
+ source_table_ref = source_dataset.table("shakespeare")
+
+ # dataset_id = 'my_dataset'
+ dest_dataset_ref = client.dataset(dataset_id)
+ dest_table_ref = dest_dataset_ref.table("destination_table")
+
+ # Set the encryption key to use for the destination.
+ # TODO: Replace this key with a key you have created in KMS.
+ kms_key_name = "projects/{}/locations/{}/keyRings/{}/cryptoKeys/{}".format(
+ "cloud-samples-tests", "us-central1", "test", "test"
+ )
+ encryption_config = bigquery.EncryptionConfiguration(kms_key_name=kms_key_name)
+ job_config = bigquery.CopyJobConfig()
+ job_config.destination_encryption_configuration = encryption_config
+
+ job = client.copy_table(
+ source_table_ref,
+ dest_table_ref,
+ # Location must match that of the source and destination tables.
+ location="US",
+ job_config=job_config,
+ ) # API request
+ job.result() # Waits for job to complete.
+
+ assert job.state == "DONE"
+ dest_table = client.get_table(dest_table_ref)
+ assert dest_table.encryption_configuration.kms_key_name == kms_key_name
+ # [END bigquery_copy_table_cmek]
+
+
+def test_extract_table(client, to_delete):
+ bucket_name = "extract_shakespeare_{}".format(_millis())
+ storage_client = storage.Client()
+ bucket = retry_storage_errors(storage_client.create_bucket)(bucket_name)
+ to_delete.append(bucket)
+
+ # [START bigquery_extract_table]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # bucket_name = 'my-bucket'
+ project = "bigquery-public-data"
+ dataset_id = "samples"
+ table_id = "shakespeare"
+
+ destination_uri = "gs://{}/{}".format(bucket_name, "shakespeare.csv")
+ dataset_ref = client.dataset(dataset_id, project=project)
+ table_ref = dataset_ref.table(table_id)
+
+ extract_job = client.extract_table(
+ table_ref,
+ destination_uri,
+ # Location must match that of the source table.
+ location="US",
+ ) # API request
+ extract_job.result() # Waits for job to complete.
+
+ print(
+ "Exported {}:{}.{} to {}".format(project, dataset_id, table_id, destination_uri)
+ )
+ # [END bigquery_extract_table]
+
+ blob = retry_storage_errors(bucket.get_blob)("shakespeare.csv")
+ assert blob.exists
+ assert blob.size > 0
+ to_delete.insert(0, blob)
+
+
+def test_extract_table_json(client, to_delete):
+ bucket_name = "extract_shakespeare_json_{}".format(_millis())
+ storage_client = storage.Client()
+ bucket = retry_storage_errors(storage_client.create_bucket)(bucket_name)
+ to_delete.append(bucket)
+
+ # [START bigquery_extract_table_json]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # bucket_name = 'my-bucket'
+
+ destination_uri = "gs://{}/{}".format(bucket_name, "shakespeare.json")
+ dataset_ref = client.dataset("samples", project="bigquery-public-data")
+ table_ref = dataset_ref.table("shakespeare")
+ job_config = bigquery.job.ExtractJobConfig()
+ job_config.destination_format = bigquery.DestinationFormat.NEWLINE_DELIMITED_JSON
+
+ extract_job = client.extract_table(
+ table_ref,
+ destination_uri,
+ job_config=job_config,
+ # Location must match that of the source table.
+ location="US",
+ ) # API request
+ extract_job.result() # Waits for job to complete.
+ # [END bigquery_extract_table_json]
+
+ blob = retry_storage_errors(bucket.get_blob)("shakespeare.json")
+ assert blob.exists
+ assert blob.size > 0
+ to_delete.insert(0, blob)
+
+
+def test_extract_table_compressed(client, to_delete):
+ bucket_name = "extract_shakespeare_compress_{}".format(_millis())
+ storage_client = storage.Client()
+ bucket = retry_storage_errors(storage_client.create_bucket)(bucket_name)
+ to_delete.append(bucket)
+
+ # [START bigquery_extract_table_compressed]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # bucket_name = 'my-bucket'
+
+ destination_uri = "gs://{}/{}".format(bucket_name, "shakespeare.csv.gz")
+ dataset_ref = client.dataset("samples", project="bigquery-public-data")
+ table_ref = dataset_ref.table("shakespeare")
+ job_config = bigquery.job.ExtractJobConfig()
+ job_config.compression = bigquery.Compression.GZIP
+
+ extract_job = client.extract_table(
+ table_ref,
+ destination_uri,
+ # Location must match that of the source table.
+ location="US",
+ job_config=job_config,
+ ) # API request
+ extract_job.result() # Waits for job to complete.
+ # [END bigquery_extract_table_compressed]
+
+ blob = retry_storage_errors(bucket.get_blob)("shakespeare.csv.gz")
+ assert blob.exists
+ assert blob.size > 0
+ to_delete.insert(0, blob)
+
+
+def test_undelete_table(client, to_delete):
+ dataset_id = "undelete_table_dataset_{}".format(_millis())
+ table_id = "undelete_table_table_{}".format(_millis())
+ dataset = bigquery.Dataset(client.dataset(dataset_id))
+ dataset.location = "US"
+ dataset = client.create_dataset(dataset)
+ to_delete.append(dataset)
+
+ table = bigquery.Table(dataset.table(table_id), schema=SCHEMA)
+ client.create_table(table)
+
+ # [START bigquery_undelete_table]
+ # TODO(developer): Uncomment the lines below and replace with your values.
+ # import time
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # dataset_id = 'my_dataset' # Replace with your dataset ID.
+ # table_id = 'my_table' # Replace with your table ID.
+
+ table_ref = client.dataset(dataset_id).table(table_id)
+
+ # TODO(developer): Choose an appropriate snapshot point as epoch
+ # milliseconds. For this example, we choose the current time as we're about
+ # to delete the table immediately afterwards.
+ snapshot_epoch = int(time.time() * 1000)
+ # [END bigquery_undelete_table]
+
+ # Due to very short lifecycle of the table, ensure we're not picking a time
+ # prior to the table creation due to time drift between backend and client.
+ table = client.get_table(table_ref)
+ created_epoch = datetime_helpers.to_microseconds(table.created)
+ if created_epoch > snapshot_epoch:
+ snapshot_epoch = created_epoch
+
+ # [START bigquery_undelete_table]
+
+ # "Accidentally" delete the table.
+ client.delete_table(table_ref) # API request
+
+ # Construct the restore-from table ID using a snapshot decorator.
+ snapshot_table_id = "{}@{}".format(table_id, snapshot_epoch)
+ source_table_ref = client.dataset(dataset_id).table(snapshot_table_id)
+
+ # Choose a new table ID for the recovered table data.
+ recovered_table_id = "{}_recovered".format(table_id)
+ dest_table_ref = client.dataset(dataset_id).table(recovered_table_id)
+
+ # Construct and run a copy job.
+ job = client.copy_table(
+ source_table_ref,
+ dest_table_ref,
+ # Location must match that of the source and destination tables.
+ location="US",
+ ) # API request
+
+ job.result() # Waits for job to complete.
+
+ print(
+ "Copied data from deleted table {} to {}".format(table_id, recovered_table_id)
+ )
+ # [END bigquery_undelete_table]
+
+
+def test_client_query(client):
+ """Run a simple query."""
+
+ # [START bigquery_query]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+
+ query = (
+ "SELECT name FROM `bigquery-public-data.usa_names.usa_1910_2013` "
+ 'WHERE state = "TX" '
+ "LIMIT 100"
+ )
+ query_job = client.query(
+ query,
+ # Location must match that of the dataset(s) referenced in the query.
+ location="US",
+ ) # API request - starts the query
+
+ for row in query_job: # API request - fetches results
+ # Row values can be accessed by field name or index
+ assert row[0] == row.name == row["name"]
+ print(row)
+ # [END bigquery_query]
+
+
+def test_client_query_legacy_sql(client):
+ """Run a query with Legacy SQL explicitly set"""
+ # [START bigquery_query_legacy]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+
+ query = (
+ "SELECT name FROM [bigquery-public-data:usa_names.usa_1910_2013] "
+ 'WHERE state = "TX" '
+ "LIMIT 100"
+ )
+
+ # Set use_legacy_sql to True to use legacy SQL syntax.
+ job_config = bigquery.QueryJobConfig()
+ job_config.use_legacy_sql = True
+
+ query_job = client.query(
+ query,
+ # Location must match that of the dataset(s) referenced in the query.
+ location="US",
+ job_config=job_config,
+ ) # API request - starts the query
+
+ # Print the results.
+ for row in query_job: # API request - fetches results
+ print(row)
+ # [END bigquery_query_legacy]
+
+
+def test_client_query_total_rows(client, capsys):
+ """Run a query and just check for how many rows."""
+ # [START bigquery_query_total_rows]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+
+ query = (
+ "SELECT name FROM `bigquery-public-data.usa_names.usa_1910_2013` "
+ 'WHERE state = "TX" '
+ "LIMIT 100"
+ )
+ query_job = client.query(
+ query,
+ # Location must match that of the dataset(s) referenced in the query.
+ location="US",
+ ) # API request - starts the query
+
+ results = query_job.result() # Wait for query to complete.
+ print("Got {} rows.".format(results.total_rows))
+ # [END bigquery_query_total_rows]
+
+ out, _ = capsys.readouterr()
+ assert "Got 100 rows." in out
+
+
+def test_manage_job(client):
+ sql = """
+ SELECT corpus
+ FROM `bigquery-public-data.samples.shakespeare`
+ GROUP BY corpus;
+ """
+ location = "us"
+ job = client.query(sql, location=location)
+ job_id = job.job_id
+
+ # [START bigquery_cancel_job]
+ # TODO(developer): Uncomment the lines below and replace with your values.
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # job_id = 'bq-job-123x456-123y123z123c' # replace with your job ID
+ # location = 'us' # replace with your location
+
+ job = client.cancel_job(job_id, location=location)
+ # [END bigquery_cancel_job]
+
+ # [START bigquery_get_job]
+ # TODO(developer): Uncomment the lines below and replace with your values.
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # job_id = 'bq-job-123x456-123y123z123c' # replace with your job ID
+ # location = 'us' # replace with your location
+
+ job = client.get_job(job_id, location=location) # API request
+
+ # Print selected job properties
+ print("Details for job {} running in {}:".format(job_id, location))
+ print(
+ "\tType: {}\n\tState: {}\n\tCreated: {}".format(
+ job.job_type, job.state, job.created
+ )
+ )
+ # [END bigquery_get_job]
+
+
+def test_client_query_destination_table(client, to_delete):
+ """Run a query"""
+ dataset_id = "query_destination_table_{}".format(_millis())
+ dataset_ref = client.dataset(dataset_id)
+ to_delete.append(dataset_ref)
+ dataset = bigquery.Dataset(dataset_ref)
+ dataset.location = "US"
+ client.create_dataset(dataset)
+
+ # [START bigquery_query_destination_table]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # dataset_id = 'your_dataset_id'
+
+ job_config = bigquery.QueryJobConfig()
+ # Set the destination table
+ table_ref = client.dataset(dataset_id).table("your_table_id")
+ job_config.destination = table_ref
+ sql = """
+ SELECT corpus
+ FROM `bigquery-public-data.samples.shakespeare`
+ GROUP BY corpus;
+ """
+
+ # Start the query, passing in the extra configuration.
+ query_job = client.query(
+ sql,
+ # Location must match that of the dataset(s) referenced in the query
+ # and of the destination table.
+ location="US",
+ job_config=job_config,
+ ) # API request - starts the query
+
+ query_job.result() # Waits for the query to finish
+ print("Query results loaded to table {}".format(table_ref.path))
+ # [END bigquery_query_destination_table]
+
+
+def test_client_query_destination_table_legacy(client, to_delete):
+ dataset_id = "query_destination_table_legacy_{}".format(_millis())
+ dataset_ref = client.dataset(dataset_id)
+ to_delete.append(dataset_ref)
+ dataset = bigquery.Dataset(dataset_ref)
+ dataset.location = "US"
+ client.create_dataset(dataset)
+
+ # [START bigquery_query_legacy_large_results]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # dataset_id = 'your_dataset_id'
+
+ job_config = bigquery.QueryJobConfig()
+ # Set use_legacy_sql to True to use legacy SQL syntax.
+ job_config.use_legacy_sql = True
+ # Set the destination table
+ table_ref = client.dataset(dataset_id).table("your_table_id")
+ job_config.destination = table_ref
+ job_config.allow_large_results = True
+ sql = """
+ SELECT corpus
+ FROM [bigquery-public-data:samples.shakespeare]
+ GROUP BY corpus;
+ """
+ # Start the query, passing in the extra configuration.
+ query_job = client.query(
+ sql,
+ # Location must match that of the dataset(s) referenced in the query
+ # and of the destination table.
+ location="US",
+ job_config=job_config,
+ ) # API request - starts the query
+
+ query_job.result() # Waits for the query to finish
+ print("Query results loaded to table {}".format(table_ref.path))
+ # [END bigquery_query_legacy_large_results]
+
+
+def test_client_query_destination_table_cmek(client, to_delete):
+ """Run a query"""
+ dataset_id = "query_destination_table_{}".format(_millis())
+ dataset_ref = client.dataset(dataset_id)
+ to_delete.append(dataset_ref)
+ dataset = bigquery.Dataset(dataset_ref)
+ dataset.location = "US"
+ client.create_dataset(dataset)
+
+ # [START bigquery_query_destination_table_cmek]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+
+ job_config = bigquery.QueryJobConfig()
+
+ # Set the destination table. Here, dataset_id is a string, such as:
+ # dataset_id = 'your_dataset_id'
+ table_ref = client.dataset(dataset_id).table("your_table_id")
+ job_config.destination = table_ref
+
+ # Set the encryption key to use for the destination.
+ # TODO: Replace this key with a key you have created in KMS.
+ kms_key_name = "projects/{}/locations/{}/keyRings/{}/cryptoKeys/{}".format(
+ "cloud-samples-tests", "us-central1", "test", "test"
+ )
+ encryption_config = bigquery.EncryptionConfiguration(kms_key_name=kms_key_name)
+ job_config.destination_encryption_configuration = encryption_config
+
+ # Start the query, passing in the extra configuration.
+ query_job = client.query(
+ "SELECT 17 AS my_col;",
+ # Location must match that of the dataset(s) referenced in the query
+ # and of the destination table.
+ location="US",
+ job_config=job_config,
+ ) # API request - starts the query
+ query_job.result()
+
+ # The destination table is written using the encryption configuration.
+ table = client.get_table(table_ref)
+ assert table.encryption_configuration.kms_key_name == kms_key_name
+ # [END bigquery_query_destination_table_cmek]
+
+
+def test_client_query_batch(client, to_delete):
+ # [START bigquery_query_batch]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+
+ job_config = bigquery.QueryJobConfig()
+ # Run at batch priority, which won't count toward concurrent rate limit.
+ job_config.priority = bigquery.QueryPriority.BATCH
+ sql = """
+ SELECT corpus
+ FROM `bigquery-public-data.samples.shakespeare`
+ GROUP BY corpus;
+ """
+ # Location must match that of the dataset(s) referenced in the query.
+ location = "US"
+
+ # API request - starts the query
+ query_job = client.query(sql, location=location, job_config=job_config)
+
+ # Check on the progress by getting the job's updated state. Once the state
+ # is `DONE`, the results are ready.
+ query_job = client.get_job(
+ query_job.job_id, location=location
+ ) # API request - fetches job
+ print("Job {} is currently in state {}".format(query_job.job_id, query_job.state))
+ # [END bigquery_query_batch]
+
+
+def test_client_query_relax_column(client, to_delete):
+ dataset_id = "query_relax_column_{}".format(_millis())
+ dataset_ref = client.dataset(dataset_id)
+ dataset = bigquery.Dataset(dataset_ref)
+ dataset.location = "US"
+ dataset = client.create_dataset(dataset)
+ to_delete.append(dataset)
+
+ table_ref = dataset_ref.table("my_table")
+ schema = [
+ bigquery.SchemaField("full_name", "STRING", mode="REQUIRED"),
+ bigquery.SchemaField("age", "INTEGER", mode="REQUIRED"),
+ ]
+ table = client.create_table(bigquery.Table(table_ref, schema=schema))
+
+ # [START bigquery_relax_column_query_append]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # dataset_ref = client.dataset('my_dataset')
+
+ # Retrieves the destination table and checks the number of required fields
+ table_id = "my_table"
+ table_ref = dataset_ref.table(table_id)
+ table = client.get_table(table_ref)
+ original_required_fields = sum(field.mode == "REQUIRED" for field in table.schema)
+ # In this example, the existing table has 2 required fields
+ print("{} fields in the schema are required.".format(original_required_fields))
+
+ # Configures the query to append the results to a destination table,
+ # allowing field relaxation
+ job_config = bigquery.QueryJobConfig()
+ job_config.schema_update_options = [
+ bigquery.SchemaUpdateOption.ALLOW_FIELD_RELAXATION
+ ]
+ job_config.destination = table_ref
+ job_config.write_disposition = bigquery.WriteDisposition.WRITE_APPEND
+
+ query_job = client.query(
+ # In this example, the existing table contains 'full_name' and 'age' as
+ # required columns, but the query results will omit the second column.
+ 'SELECT "Beyonce" as full_name;',
+ # Location must match that of the dataset(s) referenced in the query
+ # and of the destination table.
+ location="US",
+ job_config=job_config,
+ ) # API request - starts the query
+
+ query_job.result() # Waits for the query to finish
+ print("Query job {} complete.".format(query_job.job_id))
+
+ # Checks the updated number of required fields
+ table = client.get_table(table)
+ current_required_fields = sum(field.mode == "REQUIRED" for field in table.schema)
+ print("{} fields in the schema are now required.".format(current_required_fields))
+ # [END bigquery_relax_column_query_append]
+ assert original_required_fields - current_required_fields > 0
+ assert len(table.schema) == 2
+ assert table.schema[1].mode == "NULLABLE"
+ assert table.num_rows > 0
+
+
+def test_client_query_add_column(client, to_delete):
+ dataset_id = "query_add_column_{}".format(_millis())
+ dataset_ref = client.dataset(dataset_id)
+ dataset = bigquery.Dataset(dataset_ref)
+ dataset.location = "US"
+ dataset = client.create_dataset(dataset)
+ to_delete.append(dataset)
+
+ table_ref = dataset_ref.table("my_table")
+ schema = [
+ bigquery.SchemaField("full_name", "STRING", mode="REQUIRED"),
+ bigquery.SchemaField("age", "INTEGER", mode="REQUIRED"),
+ ]
+ table = client.create_table(bigquery.Table(table_ref, schema=schema))
+
+ # [START bigquery_add_column_query_append]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # dataset_ref = client.dataset('my_dataset')
+
+ # Retrieves the destination table and checks the length of the schema
+ table_id = "my_table"
+ table_ref = dataset_ref.table(table_id)
+ table = client.get_table(table_ref)
+ print("Table {} contains {} columns.".format(table_id, len(table.schema)))
+
+ # Configures the query to append the results to a destination table,
+ # allowing field addition
+ job_config = bigquery.QueryJobConfig()
+ job_config.schema_update_options = [
+ bigquery.SchemaUpdateOption.ALLOW_FIELD_ADDITION
+ ]
+ job_config.destination = table_ref
+ job_config.write_disposition = bigquery.WriteDisposition.WRITE_APPEND
+
+ query_job = client.query(
+ # In this example, the existing table contains only the 'full_name' and
+ # 'age' columns, while the results of this query will contain an
+ # additional 'favorite_color' column.
+ 'SELECT "Timmy" as full_name, 85 as age, "Blue" as favorite_color;',
+ # Location must match that of the dataset(s) referenced in the query
+ # and of the destination table.
+ location="US",
+ job_config=job_config,
+ ) # API request - starts the query
+
+ query_job.result() # Waits for the query to finish
+ print("Query job {} complete.".format(query_job.job_id))
+
+ # Checks the updated length of the schema
+ table = client.get_table(table)
+ print("Table {} now contains {} columns.".format(table_id, len(table.schema)))
+ # [END bigquery_add_column_query_append]
+ assert len(table.schema) == 3
+ assert table.num_rows > 0
+
+
+def test_client_query_w_named_params(client, capsys):
+ """Run a query using named query parameters"""
+
+ # [START bigquery_query_params_named]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+
+ query = """
+ SELECT word, word_count
+ FROM `bigquery-public-data.samples.shakespeare`
+ WHERE corpus = @corpus
+ AND word_count >= @min_word_count
+ ORDER BY word_count DESC;
+ """
+ query_params = [
+ bigquery.ScalarQueryParameter("corpus", "STRING", "romeoandjuliet"),
+ bigquery.ScalarQueryParameter("min_word_count", "INT64", 250),
+ ]
+ job_config = bigquery.QueryJobConfig()
+ job_config.query_parameters = query_params
+ query_job = client.query(
+ query,
+ # Location must match that of the dataset(s) referenced in the query.
+ location="US",
+ job_config=job_config,
+ ) # API request - starts the query
+
+ # Print the results
+ for row in query_job:
+ print("{}: \t{}".format(row.word, row.word_count))
+
+ assert query_job.state == "DONE"
+ # [END bigquery_query_params_named]
+
+ out, _ = capsys.readouterr()
+ assert "the" in out
+
+
+def test_client_query_w_positional_params(client, capsys):
+ """Run a query using query parameters"""
+
+ # [START bigquery_query_params_positional]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+
+ query = """
+ SELECT word, word_count
+ FROM `bigquery-public-data.samples.shakespeare`
+ WHERE corpus = ?
+ AND word_count >= ?
+ ORDER BY word_count DESC;
+ """
+ # Set the name to None to use positional parameters.
+ # Note that you cannot mix named and positional parameters.
+ query_params = [
+ bigquery.ScalarQueryParameter(None, "STRING", "romeoandjuliet"),
+ bigquery.ScalarQueryParameter(None, "INT64", 250),
+ ]
+ job_config = bigquery.QueryJobConfig()
+ job_config.query_parameters = query_params
+ query_job = client.query(
+ query,
+ # Location must match that of the dataset(s) referenced in the query.
+ location="US",
+ job_config=job_config,
+ ) # API request - starts the query
+
+ # Print the results
+ for row in query_job:
+ print("{}: \t{}".format(row.word, row.word_count))
+
+ assert query_job.state == "DONE"
+ # [END bigquery_query_params_positional]
+
+ out, _ = capsys.readouterr()
+ assert "the" in out
+
+
+def test_client_query_w_timestamp_params(client, capsys):
+ """Run a query using query parameters"""
+
+ # [START bigquery_query_params_timestamps]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+
+ import datetime
+ import pytz
+
+ query = "SELECT TIMESTAMP_ADD(@ts_value, INTERVAL 1 HOUR);"
+ query_params = [
+ bigquery.ScalarQueryParameter(
+ "ts_value",
+ "TIMESTAMP",
+ datetime.datetime(2016, 12, 7, 8, 0, tzinfo=pytz.UTC),
+ )
+ ]
+ job_config = bigquery.QueryJobConfig()
+ job_config.query_parameters = query_params
+ query_job = client.query(
+ query,
+ # Location must match that of the dataset(s) referenced in the query.
+ location="US",
+ job_config=job_config,
+ ) # API request - starts the query
+
+ # Print the results
+ for row in query_job:
+ print(row)
+
+ assert query_job.state == "DONE"
+ # [END bigquery_query_params_timestamps]
+
+ out, _ = capsys.readouterr()
+ assert "2016, 12, 7, 9, 0" in out
+
+
+def test_client_query_w_array_params(client, capsys):
+ """Run a query using array query parameters"""
+ # [START bigquery_query_params_arrays]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+
+ query = """
+ SELECT name, sum(number) as count
+ FROM `bigquery-public-data.usa_names.usa_1910_2013`
+ WHERE gender = @gender
+ AND state IN UNNEST(@states)
+ GROUP BY name
+ ORDER BY count DESC
+ LIMIT 10;
+ """
+ query_params = [
+ bigquery.ScalarQueryParameter("gender", "STRING", "M"),
+ bigquery.ArrayQueryParameter("states", "STRING", ["WA", "WI", "WV", "WY"]),
+ ]
+ job_config = bigquery.QueryJobConfig()
+ job_config.query_parameters = query_params
+ query_job = client.query(
+ query,
+ # Location must match that of the dataset(s) referenced in the query.
+ location="US",
+ job_config=job_config,
+ ) # API request - starts the query
+
+ # Print the results
+ for row in query_job:
+ print("{}: \t{}".format(row.name, row.count))
+
+ assert query_job.state == "DONE"
+ # [END bigquery_query_params_arrays]
+
+ out, _ = capsys.readouterr()
+ assert "James" in out
+
+
+def test_client_query_w_struct_params(client, capsys):
+ """Run a query using struct query parameters"""
+ # [START bigquery_query_params_structs]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+
+ query = "SELECT @struct_value AS s;"
+ query_params = [
+ bigquery.StructQueryParameter(
+ "struct_value",
+ bigquery.ScalarQueryParameter("x", "INT64", 1),
+ bigquery.ScalarQueryParameter("y", "STRING", "foo"),
+ )
+ ]
+ job_config = bigquery.QueryJobConfig()
+ job_config.query_parameters = query_params
+ query_job = client.query(
+ query,
+ # Location must match that of the dataset(s) referenced in the query.
+ location="US",
+ job_config=job_config,
+ ) # API request - starts the query
+
+ # Print the results
+ for row in query_job:
+ print(row.s)
+
+ assert query_job.state == "DONE"
+ # [END bigquery_query_params_structs]
+
+ out, _ = capsys.readouterr()
+ assert "1" in out
+ assert "foo" in out
+
+
+def test_client_query_dry_run(client):
+ """Run a dry run query"""
+
+ # [START bigquery_query_dry_run]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+
+ job_config = bigquery.QueryJobConfig()
+ job_config.dry_run = True
+ job_config.use_query_cache = False
+ query_job = client.query(
+ (
+ "SELECT name, COUNT(*) as name_count "
+ "FROM `bigquery-public-data.usa_names.usa_1910_2013` "
+ "WHERE state = 'WA' "
+ "GROUP BY name"
+ ),
+ # Location must match that of the dataset(s) referenced in the query.
+ location="US",
+ job_config=job_config,
+ ) # API request
+
+ # A dry run query completes immediately.
+ assert query_job.state == "DONE"
+ assert query_job.dry_run
+
+ print("This query will process {} bytes.".format(query_job.total_bytes_processed))
+ # [END bigquery_query_dry_run]
+
+ assert query_job.total_bytes_processed > 0
+
+
+def test_query_no_cache(client):
+ # [START bigquery_query_no_cache]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+
+ job_config = bigquery.QueryJobConfig()
+ job_config.use_query_cache = False
+ sql = """
+ SELECT corpus
+ FROM `bigquery-public-data.samples.shakespeare`
+ GROUP BY corpus;
+ """
+ query_job = client.query(
+ sql,
+ # Location must match that of the dataset(s) referenced in the query.
+ location="US",
+ job_config=job_config,
+ ) # API request
+
+ # Print the results.
+ for row in query_job: # API request - fetches results
+ print(row)
+ # [END bigquery_query_no_cache]
+
+
+def test_query_external_gcs_temporary_table(client):
+ # [START bigquery_query_external_gcs_temp]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+
+ # Configure the external data source and query job
+ external_config = bigquery.ExternalConfig("CSV")
+ external_config.source_uris = [
+ "gs://cloud-samples-data/bigquery/us-states/us-states.csv"
+ ]
+ external_config.schema = [
+ bigquery.SchemaField("name", "STRING"),
+ bigquery.SchemaField("post_abbr", "STRING"),
+ ]
+ external_config.options.skip_leading_rows = 1 # optionally skip header row
+ table_id = "us_states"
+ job_config = bigquery.QueryJobConfig()
+ job_config.table_definitions = {table_id: external_config}
+
+ # Example query to find states starting with 'W'
+ sql = 'SELECT * FROM `{}` WHERE name LIKE "W%"'.format(table_id)
+
+ query_job = client.query(sql, job_config=job_config) # API request
+
+ w_states = list(query_job) # Waits for query to finish
+ print("There are {} states with names starting with W.".format(len(w_states)))
+ # [END bigquery_query_external_gcs_temp]
+ assert len(w_states) == 4
+
+
+def test_query_external_gcs_permanent_table(client, to_delete):
+ dataset_id = "query_external_gcs_{}".format(_millis())
+ dataset = bigquery.Dataset(client.dataset(dataset_id))
+ client.create_dataset(dataset)
+ to_delete.append(dataset)
+
+ # [START bigquery_query_external_gcs_perm]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+ # dataset_id = 'my_dataset'
+
+ # Configure the external data source
+ dataset_ref = client.dataset(dataset_id)
+ table_id = "us_states"
+ schema = [
+ bigquery.SchemaField("name", "STRING"),
+ bigquery.SchemaField("post_abbr", "STRING"),
+ ]
+ table = bigquery.Table(dataset_ref.table(table_id), schema=schema)
+ external_config = bigquery.ExternalConfig("CSV")
+ external_config.source_uris = [
+ "gs://cloud-samples-data/bigquery/us-states/us-states.csv"
+ ]
+ external_config.options.skip_leading_rows = 1 # optionally skip header row
+ table.external_data_configuration = external_config
+
+ # Create a permanent table linked to the GCS file
+ table = client.create_table(table) # API request
+
+ # Example query to find states starting with 'W'
+ sql = 'SELECT * FROM `{}.{}` WHERE name LIKE "W%"'.format(dataset_id, table_id)
+
+ query_job = client.query(sql) # API request
+
+ w_states = list(query_job) # Waits for query to finish
+ print("There are {} states with names starting with W.".format(len(w_states)))
+ # [END bigquery_query_external_gcs_perm]
+ assert len(w_states) == 4
+
+
+def test_query_external_sheets_temporary_table(client):
+ # [START bigquery_query_external_sheets_temp]
+ # [START bigquery_auth_drive_scope]
+ import google.auth
+
+ # from google.cloud import bigquery
+
+ # Create credentials with Drive & BigQuery API scopes
+ # Both APIs must be enabled for your project before running this code
+ credentials, project = google.auth.default(
+ scopes=[
+ "https://www.googleapis.com/auth/drive",
+ "https://www.googleapis.com/auth/bigquery",
+ ]
+ )
+ client = bigquery.Client(credentials=credentials, project=project)
+ # [END bigquery_auth_drive_scope]
+
+ # Configure the external data source and query job
+ external_config = bigquery.ExternalConfig("GOOGLE_SHEETS")
+ # Use a shareable link or grant viewing access to the email address you
+ # used to authenticate with BigQuery (this example Sheet is public)
+ sheet_url = (
+ "https://docs.google.com/spreadsheets"
+ "/d/1i_QCL-7HcSyUZmIbP9E6lO_T5u3HnpLe7dnpHaijg_E/edit?usp=sharing"
+ )
+ external_config.source_uris = [sheet_url]
+ external_config.schema = [
+ bigquery.SchemaField("name", "STRING"),
+ bigquery.SchemaField("post_abbr", "STRING"),
+ ]
+ external_config.options.skip_leading_rows = 1 # optionally skip header row
+ table_id = "us_states"
+ job_config = bigquery.QueryJobConfig()
+ job_config.table_definitions = {table_id: external_config}
+
+ # Example query to find states starting with 'W'
+ sql = 'SELECT * FROM `{}` WHERE name LIKE "W%"'.format(table_id)
+
+ query_job = client.query(sql, job_config=job_config) # API request
+
+ w_states = list(query_job) # Waits for query to finish
+ print("There are {} states with names starting with W.".format(len(w_states)))
+ # [END bigquery_query_external_sheets_temp]
+ assert len(w_states) == 4
+
+
+def test_query_external_sheets_permanent_table(client, to_delete):
+ dataset_id = "query_external_sheets_{}".format(_millis())
+ dataset = bigquery.Dataset(client.dataset(dataset_id))
+ client.create_dataset(dataset)
+ to_delete.append(dataset)
+
+ # [START bigquery_query_external_sheets_perm]
+ import google.auth
+
+ # from google.cloud import bigquery
+ # dataset_id = 'my_dataset'
+
+ # Create credentials with Drive & BigQuery API scopes
+ # Both APIs must be enabled for your project before running this code
+ credentials, project = google.auth.default(
+ scopes=[
+ "https://www.googleapis.com/auth/drive",
+ "https://www.googleapis.com/auth/bigquery",
+ ]
+ )
+ client = bigquery.Client(credentials=credentials, project=project)
+
+ # Configure the external data source
+ dataset_ref = client.dataset(dataset_id)
+ table_id = "us_states"
+ schema = [
+ bigquery.SchemaField("name", "STRING"),
+ bigquery.SchemaField("post_abbr", "STRING"),
+ ]
+ table = bigquery.Table(dataset_ref.table(table_id), schema=schema)
+ external_config = bigquery.ExternalConfig("GOOGLE_SHEETS")
+ # Use a shareable link or grant viewing access to the email address you
+ # used to authenticate with BigQuery (this example Sheet is public)
+ sheet_url = (
+ "https://docs.google.com/spreadsheets"
+ "/d/1i_QCL-7HcSyUZmIbP9E6lO_T5u3HnpLe7dnpHaijg_E/edit?usp=sharing"
+ )
+ external_config.source_uris = [sheet_url]
+ external_config.options.skip_leading_rows = 1 # optionally skip header row
+ table.external_data_configuration = external_config
+
+ # Create a permanent table linked to the Sheets file
+ table = client.create_table(table) # API request
+
+ # Example query to find states starting with 'W'
+ sql = 'SELECT * FROM `{}.{}` WHERE name LIKE "W%"'.format(dataset_id, table_id)
+
+ query_job = client.query(sql) # API request
+
+ w_states = list(query_job) # Waits for query to finish
+ print("There are {} states with names starting with W.".format(len(w_states)))
+ # [END bigquery_query_external_sheets_perm]
+ assert len(w_states) == 4
+
+
+def test_ddl_create_view(client, to_delete, capsys):
+ """Create a view via a DDL query."""
+ project = client.project
+ dataset_id = "ddl_view_{}".format(_millis())
+ table_id = "new_view"
+ dataset = bigquery.Dataset(client.dataset(dataset_id))
+ client.create_dataset(dataset)
+ to_delete.append(dataset)
+
+ # [START bigquery_ddl_create_view]
+ # from google.cloud import bigquery
+ # project = 'my-project'
+ # dataset_id = 'my_dataset'
+ # table_id = 'new_view'
+ # client = bigquery.Client(project=project)
+
+ sql = """
+ CREATE VIEW `{}.{}.{}`
+ OPTIONS(
+ expiration_timestamp=TIMESTAMP_ADD(
+ CURRENT_TIMESTAMP(), INTERVAL 48 HOUR),
+ friendly_name="new_view",
+ description="a view that expires in 2 days",
+ labels=[("org_unit", "development")]
+ )
+ AS SELECT name, state, year, number
+ FROM `bigquery-public-data.usa_names.usa_1910_current`
+ WHERE state LIKE 'W%'
+ """.format(
+ project, dataset_id, table_id
+ )
+
+ job = client.query(sql) # API request.
+ job.result() # Waits for the query to finish.
+
+ print(
+ 'Created new view "{}.{}.{}".'.format(
+ job.destination.project,
+ job.destination.dataset_id,
+ job.destination.table_id,
+ )
+ )
+ # [END bigquery_ddl_create_view]
+
+ out, _ = capsys.readouterr()
+ assert 'Created new view "{}.{}.{}".'.format(project, dataset_id, table_id) in out
+
+ # Test that listing query result rows succeeds so that generic query
+ # processing tools work with DDL statements.
+ rows = list(job)
+ assert len(rows) == 0
+
+ if pandas is not None:
+ df = job.to_dataframe()
+ assert len(df) == 0
+
+
+def test_client_list_jobs(client):
+ """List jobs for a project."""
+
+ # [START bigquery_list_jobs]
+ # TODO(developer): Uncomment the lines below and replace with your values.
+ # from google.cloud import bigquery
+ # project = 'my_project' # replace with your project ID
+ # client = bigquery.Client(project=project)
+ import datetime
+
+ # List the 10 most recent jobs in reverse chronological order.
+ # Omit the max_results parameter to list jobs from the past 6 months.
+ print("Last 10 jobs:")
+ for job in client.list_jobs(max_results=10): # API request(s)
+ print(job.job_id)
+
+ # The following are examples of additional optional parameters:
+
+ # Use min_creation_time and/or max_creation_time to specify a time window.
+ print("Jobs from the last ten minutes:")
+ ten_mins_ago = datetime.datetime.utcnow() - datetime.timedelta(minutes=10)
+ for job in client.list_jobs(min_creation_time=ten_mins_ago):
+ print(job.job_id)
+
+ # Use all_users to include jobs run by all users in the project.
+ print("Last 10 jobs run by all users:")
+ for job in client.list_jobs(max_results=10, all_users=True):
+ print("{} run by user: {}".format(job.job_id, job.user_email))
+
+ # Use state_filter to filter by job state.
+ print("Jobs currently running:")
+ for job in client.list_jobs(state_filter="RUNNING"):
+ print(job.job_id)
+ # [END bigquery_list_jobs]
+
+
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+def test_query_results_as_dataframe(client):
+ # [START bigquery_query_results_dataframe]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+
+ sql = """
+ SELECT name, SUM(number) as count
+ FROM `bigquery-public-data.usa_names.usa_1910_current`
+ GROUP BY name
+ ORDER BY count DESC
+ LIMIT 10
+ """
+
+ df = client.query(sql).to_dataframe()
+ # [END bigquery_query_results_dataframe]
+ assert isinstance(df, pandas.DataFrame)
+ assert len(list(df)) == 2 # verify the number of columns
+ assert len(df) == 10 # verify the number of rows
+
+
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+def test_list_rows_as_dataframe(client):
+ # [START bigquery_list_rows_dataframe]
+ # from google.cloud import bigquery
+ # client = bigquery.Client()
+
+ dataset_ref = client.dataset("samples", project="bigquery-public-data")
+ table_ref = dataset_ref.table("shakespeare")
+ table = client.get_table(table_ref)
+
+ df = client.list_rows(table).to_dataframe()
+ # [END bigquery_list_rows_dataframe]
+ assert isinstance(df, pandas.DataFrame)
+ assert len(list(df)) == len(table.schema) # verify the number of columns
+ assert len(df) == table.num_rows # verify the number of rows
+
+
+if __name__ == "__main__":
+ pytest.main()
diff --git a/bigquery/docs/usage.html b/bigquery/docs/usage.html
new file mode 100644
index 000000000000..78dc14b9ca03
--- /dev/null
+++ b/bigquery/docs/usage.html
@@ -0,0 +1,8 @@
+
+
+
+
+
+
diff --git a/bigquery/docs/usage/client.rst b/bigquery/docs/usage/client.rst
new file mode 100644
index 000000000000..d631585ea2fe
--- /dev/null
+++ b/bigquery/docs/usage/client.rst
@@ -0,0 +1,25 @@
+Creating a Client
+~~~~~~~~~~~~~~~~~
+
+A project is the top-level container in the ``BigQuery`` API: it is tied
+closely to billing, and can provide default access control across all its
+datasets. If no ``project`` is passed to the client container, the library
+attempts to infer a project using the environment (including explicit
+environment variables, GAE, and GCE).
+
+To override the project inferred from the environment, pass an explicit
+``project`` to the :class:`~google.cloud.bigquery.client.Client` constructor,
+or to either of the alternative ``classmethod`` factories:
+
+.. code-block:: python
+
+ from google.cloud import bigquery
+ client = bigquery.Client(project='PROJECT_ID')
+
+
+Project ACLs
+^^^^^^^^^^^^
+
+Each project has an access control list granting reader / writer / owner
+permission to one or more entities. This list cannot be queried or set
+via the API; it must be managed using the Google Developer Console.
diff --git a/bigquery/docs/usage/datasets.rst b/bigquery/docs/usage/datasets.rst
new file mode 100644
index 000000000000..2daee77f36d2
--- /dev/null
+++ b/bigquery/docs/usage/datasets.rst
@@ -0,0 +1,131 @@
+Managing Datasets
+~~~~~~~~~~~~~~~~~
+
+A dataset represents a collection of tables, and applies several default
+policies to tables as they are created:
+
+- An access control list (ACL). When created, a dataset has an ACL
+ which maps to the ACL inherited from its project.
+
+- A default table expiration period. If set, tables created within the
+ dataset will have the value as their expiration period.
+
+See BigQuery documentation for more information on
+`Datasets `_.
+
+Listing Datasets
+^^^^^^^^^^^^^^^^
+
+List datasets for a project with the
+:func:`~google.cloud.bigquery.client.Client.list_datasets` method:
+
+.. literalinclude:: ../samples/list_datasets.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_list_datasets]
+ :end-before: [END bigquery_list_datasets]
+
+List datasets by label for a project with the
+:func:`~google.cloud.bigquery.client.Client.list_datasets` method:
+
+.. literalinclude:: ../samples/list_datasets_by_label.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_list_datasets_by_label]
+ :end-before: [END bigquery_list_datasets_by_label]
+
+Getting a Dataset
+^^^^^^^^^^^^^^^^^
+
+Get a dataset resource (to pick up changes made by another client) with the
+:func:`~google.cloud.bigquery.client.Client.get_dataset` method:
+
+.. literalinclude:: ../samples/get_dataset.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_get_dataset]
+ :end-before: [END bigquery_get_dataset]
+
+Determine if a dataset exists with the
+:func:`~google.cloud.bigquery.client.Client.get_dataset` method:
+
+.. literalinclude:: ../samples/dataset_exists.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_dataset_exists]
+ :end-before: [END bigquery_dataset_exists]
+
+Creating a Dataset
+^^^^^^^^^^^^^^^^^^
+
+Create a new dataset with the
+:func:`~google.cloud.bigquery.client.Client.create_dataset` method:
+
+.. literalinclude:: ../samples/create_dataset.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_create_dataset]
+ :end-before: [END bigquery_create_dataset]
+
+Updating a Dataset
+^^^^^^^^^^^^^^^^^^
+
+Update a property in a dataset's metadata with the
+:func:`~google.cloud.bigquery.client.Client.update_dataset` method:
+
+.. literalinclude:: ../samples/update_dataset_description.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_update_dataset_description]
+ :end-before: [END bigquery_update_dataset_description]
+
+Modify user permissions on a dataset with the
+:func:`~google.cloud.bigquery.client.Client.update_dataset` method:
+
+.. literalinclude:: ../samples/update_dataset_access.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_update_dataset_access]
+ :end-before: [END bigquery_update_dataset_access]
+
+Manage Dataset labels
+^^^^^^^^^^^^^^^^^^^^^
+
+Add labels to a dataset with the
+:func:`~google.cloud.bigquery.client.Client.update_dataset` method:
+
+.. literalinclude:: ../samples/label_dataset.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_label_dataset]
+ :end-before: [END bigquery_label_dataset]
+
+Get dataset's labels with the
+:func:`~google.cloud.bigquery.client.Client.get_dataset` method:
+
+.. literalinclude:: ../samples/get_dataset_labels.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_get_dataset_labels]
+ :end-before: [END bigquery_get_dataset_labels]
+
+Delete dataset's labels with the
+:func:`~google.cloud.bigquery.client.Client.update_dataset` method:
+
+.. literalinclude:: ../samples/delete_dataset_labels.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_delete_label_dataset]
+ :end-before: [END bigquery_delete_label_dataset]
+
+Deleting a Dataset
+^^^^^^^^^^^^^^^^^^
+
+Delete a dataset with the
+:func:`~google.cloud.bigquery.client.Client.delete_dataset` method:
+
+.. literalinclude:: ../samples/delete_dataset.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_delete_dataset]
+ :end-before: [END bigquery_delete_dataset]
diff --git a/bigquery/docs/usage/encryption.rst b/bigquery/docs/usage/encryption.rst
new file mode 100644
index 000000000000..88d23067995e
--- /dev/null
+++ b/bigquery/docs/usage/encryption.rst
@@ -0,0 +1,52 @@
+Using Customer Managed Encryption Keys
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Table data is always encrypted at rest, but BigQuery also provides a way for
+you to control what keys it uses to encrypt they data. See `Protecting data
+with Cloud KMS keys
+`_
+in the BigQuery documentation for more details.
+
+Create a new table, using a customer-managed encryption key from
+Cloud KMS to encrypt it.
+
+.. literalinclude:: ../snippets.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_create_table_cmek]
+ :end-before: [END bigquery_create_table_cmek]
+
+Change the key used to encrypt a table.
+
+.. literalinclude:: ../snippets.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_update_table_cmek]
+ :end-before: [END bigquery_update_table_cmek]
+
+Load a file from Cloud Storage, using a customer-managed encryption key from
+Cloud KMS for the destination table.
+
+.. literalinclude:: ../snippets.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_load_table_gcs_json_cmek]
+ :end-before: [END bigquery_load_table_gcs_json_cmek]
+
+Copy a table, using a customer-managed encryption key from Cloud KMS for the
+destination table.
+
+.. literalinclude:: ../snippets.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_copy_table_cmek]
+ :end-before: [END bigquery_copy_table_cmek]
+
+Write query results to a table, using a customer-managed encryption key from
+Cloud KMS for the destination table.
+
+.. literalinclude:: ../snippets.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_query_destination_table_cmek]
+ :end-before: [END bigquery_query_destination_table_cmek]
diff --git a/bigquery/docs/usage/index.rst b/bigquery/docs/usage/index.rst
new file mode 100644
index 000000000000..ff4c9d7f1a8f
--- /dev/null
+++ b/bigquery/docs/usage/index.rst
@@ -0,0 +1,35 @@
+Usage Guides
+~~~~~~~~~~~~
+
+BigQuery Basics
+^^^^^^^^^^^^^^^
+
+.. toctree::
+ :maxdepth: 1
+
+ client
+ queries
+
+Working with BigQuery Resources
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. toctree::
+ :maxdepth: 1
+
+ datasets
+ tables
+ encryption
+ jobs
+
+Integrations with Other Libraries
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. toctree::
+ :maxdepth: 1
+
+ pandas
+
+See also, the :mod:`google.cloud.bigquery.magics` module for integrations
+with Jupyter.
+
+
diff --git a/bigquery/docs/usage/jobs.rst b/bigquery/docs/usage/jobs.rst
new file mode 100644
index 000000000000..914d1d459ee7
--- /dev/null
+++ b/bigquery/docs/usage/jobs.rst
@@ -0,0 +1,18 @@
+Managing Jobs
+~~~~~~~~~~~~~
+
+List jobs for a project
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Jobs describe actions performed on data in BigQuery tables:
+
+- Load data into a table
+- Run a query against data in one or more tables
+- Extract data from a table
+- Copy a table
+
+.. literalinclude:: ../snippets.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_list_jobs]
+ :end-before: [END bigquery_list_jobs]
diff --git a/bigquery/docs/usage/pandas.rst b/bigquery/docs/usage/pandas.rst
new file mode 100644
index 000000000000..9db98dfbbccb
--- /dev/null
+++ b/bigquery/docs/usage/pandas.rst
@@ -0,0 +1,62 @@
+Using BigQuery with Pandas
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Retrieve BigQuery data as a Pandas DataFrame
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+As of version 0.29.0, you can use the
+:func:`~google.cloud.bigquery.table.RowIterator.to_dataframe` function to
+retrieve query results or table rows as a :class:`pandas.DataFrame`.
+
+First, ensure that the :mod:`pandas` library is installed by running:
+
+.. code-block:: bash
+
+ pip install --upgrade pandas
+
+Alternatively, you can install the BigQuery python client library with
+:mod:`pandas` by running:
+
+.. code-block:: bash
+
+ pip install --upgrade google-cloud-bigquery[pandas]
+
+To retrieve query results as a :class:`pandas.DataFrame`:
+
+.. literalinclude:: ../snippets.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_query_results_dataframe]
+ :end-before: [END bigquery_query_results_dataframe]
+
+To retrieve table rows as a :class:`pandas.DataFrame`:
+
+.. literalinclude:: ../snippets.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_list_rows_dataframe]
+ :end-before: [END bigquery_list_rows_dataframe]
+
+Load a Pandas DataFrame to a BigQuery Table
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+As of version 1.3.0, you can use the
+:func:`~google.cloud.bigquery.client.Client.load_table_from_dataframe` function
+to load data from a :class:`pandas.DataFrame` to a
+:class:`~google.cloud.bigquery.table.Table`. To use this function, in addition
+to :mod:`pandas`, you will need to install the :mod:`pyarrow` library. You can
+install the BigQuery python client library with :mod:`pandas` and
+:mod:`pyarrow` by running:
+
+.. code-block:: bash
+
+ pip install --upgrade google-cloud-bigquery[pandas,pyarrow]
+
+The following example demonstrates how to create a :class:`pandas.DataFrame`
+and load it into a new table:
+
+.. literalinclude:: ../samples/load_table_dataframe.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_load_table_dataframe]
+ :end-before: [END bigquery_load_table_dataframe]
diff --git a/bigquery/docs/usage/queries.rst b/bigquery/docs/usage/queries.rst
new file mode 100644
index 000000000000..fc77bb5b80cd
--- /dev/null
+++ b/bigquery/docs/usage/queries.rst
@@ -0,0 +1,49 @@
+Running Queries
+~~~~~~~~~~~~~~~
+
+Querying data
+^^^^^^^^^^^^^
+
+Run a query and wait for it to finish:
+
+.. literalinclude:: ../snippets.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_query]
+ :end-before: [END bigquery_query]
+
+
+Run a dry run query
+^^^^^^^^^^^^^^^^^^^
+
+.. literalinclude:: ../snippets.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_query_dry_run]
+ :end-before: [END bigquery_query_dry_run]
+
+
+Writing query results to a destination table
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+See BigQuery documentation for more information on
+`writing query results `_.
+
+.. literalinclude:: ../snippets.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_query_destination_table]
+ :end-before: [END bigquery_query_destination_table]
+
+
+Run a query using a named query parameter
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+See BigQuery documentation for more information on
+`parameterized queries `_.
+
+.. literalinclude:: ../snippets.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_query_params_named]
+ :end-before: [END bigquery_query_params_named]
diff --git a/bigquery/docs/usage/tables.rst b/bigquery/docs/usage/tables.rst
new file mode 100644
index 000000000000..458c5b0009ba
--- /dev/null
+++ b/bigquery/docs/usage/tables.rst
@@ -0,0 +1,156 @@
+Managing Tables
+~~~~~~~~~~~~~~~
+
+Tables exist within datasets. See BigQuery documentation for more information
+on `Tables `_.
+
+Listing Tables
+^^^^^^^^^^^^^^
+
+List the tables belonging to a dataset with the
+:func:`~google.cloud.bigquery.client.Client.list_tables` method:
+
+.. literalinclude:: ../samples/list_tables.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_list_tables]
+ :end-before: [END bigquery_list_tables]
+
+Getting a Table
+^^^^^^^^^^^^^^^
+
+Get a table resource with the
+:func:`~google.cloud.bigquery.client.Client.get_table` method:
+
+.. literalinclude:: ../samples/get_table.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_get_table]
+ :end-before: [END bigquery_get_table]
+
+Browse data rows in a table with the
+:func:`~google.cloud.bigquery.client.Client.list_rows` method:
+
+.. literalinclude:: ../samples/browse_table_data.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_browse_table]
+ :end-before: [END bigquery_browse_table]
+
+Creating a Table
+^^^^^^^^^^^^^^^^
+
+Create an empty table with the
+:func:`~google.cloud.bigquery.client.Client.create_table` method:
+
+.. literalinclude:: ../samples/create_table.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_create_table]
+ :end-before: [END bigquery_create_table]
+
+Load table data from a file with the
+:func:`~google.cloud.bigquery.client.Client.load_table_from_file` method:
+
+.. literalinclude:: ../snippets.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_load_from_file]
+ :end-before: [END bigquery_load_from_file]
+
+Load a CSV file from Cloud Storage with the
+:func:`~google.cloud.bigquery.client.Client.load_table_from_uri` method:
+
+.. literalinclude:: ../snippets.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_load_table_gcs_csv]
+ :end-before: [END bigquery_load_table_gcs_csv]
+
+See also: `Loading CSV data from Cloud Storage
+`_.
+
+Load a JSON file from Cloud Storage:
+
+.. literalinclude:: ../snippets.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_load_table_gcs_json]
+ :end-before: [END bigquery_load_table_gcs_json]
+
+See also: `Loading JSON data from Cloud Storage
+`_.
+
+Load a Parquet file from Cloud Storage:
+
+.. literalinclude:: ../snippets.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_load_table_gcs_parquet]
+ :end-before: [END bigquery_load_table_gcs_parquet]
+
+See also: `Loading Parquet data from Cloud Storage
+`_.
+
+Updating a Table
+^^^^^^^^^^^^^^^^
+
+Update a property in a table's metadata with the
+:func:`~google.cloud.bigquery.client.Client.update_table` method:
+
+.. literalinclude:: ../snippets.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_update_table_description]
+ :end-before: [END bigquery_update_table_description]
+
+Insert rows into a table's data with the
+:func:`~google.cloud.bigquery.client.Client.insert_rows` method:
+
+.. literalinclude:: ../snippets.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_table_insert_rows]
+ :end-before: [END bigquery_table_insert_rows]
+
+Add an empty column to the existing table with the
+:func:`~google.cloud.bigquery.update_table` method:
+
+.. literalinclude:: ../samples/add_empty_column.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_add_empty_column]
+ :end-before: [END bigquery_add_empty_column]
+
+Copying a Table
+^^^^^^^^^^^^^^^
+
+Copy a table with the
+:func:`~google.cloud.bigquery.client.Client.copy_table` method:
+
+.. literalinclude:: ../snippets.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_copy_table]
+ :end-before: [END bigquery_copy_table]
+
+Copy table data to Google Cloud Storage with the
+:func:`~google.cloud.bigquery.client.Client.extract_table` method:
+
+.. literalinclude:: ../snippets.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_extract_table]
+ :end-before: [END bigquery_extract_table]
+
+Deleting a Table
+^^^^^^^^^^^^^^^^
+
+Delete a table with the
+:func:`~google.cloud.bigquery.client.Client.delete_table` method:
+
+.. literalinclude:: ../samples/delete_table.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquery_delete_table]
+ :end-before: [END bigquery_delete_table]
diff --git a/bigquery/google/__init__.py b/bigquery/google/__init__.py
new file mode 100644
index 000000000000..8fcc60e2b9c6
--- /dev/null
+++ b/bigquery/google/__init__.py
@@ -0,0 +1,24 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+try:
+ import pkg_resources
+
+ pkg_resources.declare_namespace(__name__)
+except ImportError:
+ import pkgutil
+
+ __path__ = pkgutil.extend_path(__path__, __name__)
diff --git a/bigquery/google/cloud/__init__.py b/bigquery/google/cloud/__init__.py
new file mode 100644
index 000000000000..8fcc60e2b9c6
--- /dev/null
+++ b/bigquery/google/cloud/__init__.py
@@ -0,0 +1,24 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+try:
+ import pkg_resources
+
+ pkg_resources.declare_namespace(__name__)
+except ImportError:
+ import pkgutil
+
+ __path__ = pkgutil.extend_path(__path__, __name__)
diff --git a/bigquery/google/cloud/bigquery/__init__.py b/bigquery/google/cloud/bigquery/__init__.py
new file mode 100644
index 000000000000..bda8c5611435
--- /dev/null
+++ b/bigquery/google/cloud/bigquery/__init__.py
@@ -0,0 +1,148 @@
+# Copyright 2015 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Google BigQuery API wrapper.
+
+The main concepts with this API are:
+
+- :class:`~google.cloud.bigquery.client.Client` manages connections to the
+ BigQuery API. Use the client methods to run jobs (such as a
+ :class:`~google.cloud.bigquery.job.QueryJob` via
+ :meth:`~google.cloud.bigquery.client.Client.query`) and manage resources.
+
+- :class:`~google.cloud.bigquery.dataset.Dataset` represents a
+ collection of tables.
+
+- :class:`~google.cloud.bigquery.table.Table` represents a single "relation".
+"""
+
+
+from pkg_resources import get_distribution
+
+__version__ = get_distribution("google-cloud-bigquery").version
+
+from google.cloud.bigquery.client import Client
+from google.cloud.bigquery.dataset import AccessEntry
+from google.cloud.bigquery.dataset import Dataset
+from google.cloud.bigquery.dataset import DatasetReference
+from google.cloud.bigquery import enums
+from google.cloud.bigquery.enums import StandardSqlDataTypes
+from google.cloud.bigquery.external_config import ExternalConfig
+from google.cloud.bigquery.external_config import BigtableOptions
+from google.cloud.bigquery.external_config import BigtableColumnFamily
+from google.cloud.bigquery.external_config import BigtableColumn
+from google.cloud.bigquery.external_config import CSVOptions
+from google.cloud.bigquery.external_config import GoogleSheetsOptions
+from google.cloud.bigquery.external_config import ExternalSourceFormat
+from google.cloud.bigquery.job import Compression
+from google.cloud.bigquery.job import CopyJob
+from google.cloud.bigquery.job import CopyJobConfig
+from google.cloud.bigquery.job import CreateDisposition
+from google.cloud.bigquery.job import DestinationFormat
+from google.cloud.bigquery.job import Encoding
+from google.cloud.bigquery.job import ExtractJob
+from google.cloud.bigquery.job import ExtractJobConfig
+from google.cloud.bigquery.job import LoadJob
+from google.cloud.bigquery.job import LoadJobConfig
+from google.cloud.bigquery.job import QueryJob
+from google.cloud.bigquery.job import QueryJobConfig
+from google.cloud.bigquery.job import QueryPriority
+from google.cloud.bigquery.job import SchemaUpdateOption
+from google.cloud.bigquery.job import SourceFormat
+from google.cloud.bigquery.job import UnknownJob
+from google.cloud.bigquery.job import WriteDisposition
+from google.cloud.bigquery.model import Model
+from google.cloud.bigquery.model import ModelReference
+from google.cloud.bigquery.query import ArrayQueryParameter
+from google.cloud.bigquery.query import ScalarQueryParameter
+from google.cloud.bigquery.query import StructQueryParameter
+from google.cloud.bigquery.query import UDFResource
+from google.cloud.bigquery.retry import DEFAULT_RETRY
+from google.cloud.bigquery.routine import Routine
+from google.cloud.bigquery.routine import RoutineArgument
+from google.cloud.bigquery.routine import RoutineReference
+from google.cloud.bigquery.schema import SchemaField
+from google.cloud.bigquery.table import EncryptionConfiguration
+from google.cloud.bigquery.table import Table
+from google.cloud.bigquery.table import TableReference
+from google.cloud.bigquery.table import Row
+from google.cloud.bigquery.table import TimePartitioningType
+from google.cloud.bigquery.table import TimePartitioning
+
+__all__ = [
+ "__version__",
+ "Client",
+ # Queries
+ "QueryJob",
+ "QueryJobConfig",
+ "ArrayQueryParameter",
+ "ScalarQueryParameter",
+ "StructQueryParameter",
+ # Datasets
+ "Dataset",
+ "DatasetReference",
+ "AccessEntry",
+ # Tables
+ "EncryptionConfiguration",
+ "Table",
+ "TableReference",
+ "Row",
+ "CopyJob",
+ "CopyJobConfig",
+ "ExtractJob",
+ "ExtractJobConfig",
+ "LoadJob",
+ "LoadJobConfig",
+ "UnknownJob",
+ "TimePartitioningType",
+ "TimePartitioning",
+ # Models
+ "Model",
+ "ModelReference",
+ # Routines
+ "Routine",
+ "RoutineArgument",
+ "RoutineReference",
+ # Shared helpers
+ "SchemaField",
+ "UDFResource",
+ "ExternalConfig",
+ "BigtableOptions",
+ "BigtableColumnFamily",
+ "BigtableColumn",
+ "CSVOptions",
+ "GoogleSheetsOptions",
+ "DEFAULT_RETRY",
+ # Enum Constants
+ "enums",
+ "Compression",
+ "CreateDisposition",
+ "DestinationFormat",
+ "ExternalSourceFormat",
+ "Encoding",
+ "QueryPriority",
+ "SchemaUpdateOption",
+ "StandardSqlDataTypes",
+ "SourceFormat",
+ "WriteDisposition",
+]
+
+
+def load_ipython_extension(ipython):
+ """Called by IPython when this module is loaded as an IPython extension."""
+ from google.cloud.bigquery.magics import _cell_magic
+
+ ipython.register_magic_function(
+ _cell_magic, magic_kind="cell", magic_name="bigquery"
+ )
diff --git a/bigquery/google/cloud/bigquery/_helpers.py b/bigquery/google/cloud/bigquery/_helpers.py
new file mode 100644
index 000000000000..bb3998732a5a
--- /dev/null
+++ b/bigquery/google/cloud/bigquery/_helpers.py
@@ -0,0 +1,635 @@
+# Copyright 2015 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Shared helper functions for BigQuery API classes."""
+
+import base64
+import copy
+import datetime
+import decimal
+
+from google.cloud._helpers import UTC
+from google.cloud._helpers import _date_from_iso8601_date
+from google.cloud._helpers import _datetime_from_microseconds
+from google.cloud._helpers import _microseconds_from_datetime
+from google.cloud._helpers import _RFC3339_NO_FRACTION
+from google.cloud._helpers import _to_bytes
+
+_RFC3339_MICROS_NO_ZULU = "%Y-%m-%dT%H:%M:%S.%f"
+_TIMEONLY_WO_MICROS = "%H:%M:%S"
+_TIMEONLY_W_MICROS = "%H:%M:%S.%f"
+
+
+def _not_null(value, field):
+ """Check whether 'value' should be coerced to 'field' type."""
+ return value is not None or field.mode != "NULLABLE"
+
+
+def _int_from_json(value, field):
+ """Coerce 'value' to an int, if set or not nullable."""
+ if _not_null(value, field):
+ return int(value)
+
+
+def _float_from_json(value, field):
+ """Coerce 'value' to a float, if set or not nullable."""
+ if _not_null(value, field):
+ return float(value)
+
+
+def _decimal_from_json(value, field):
+ """Coerce 'value' to a Decimal, if set or not nullable."""
+ if _not_null(value, field):
+ return decimal.Decimal(value)
+
+
+def _bool_from_json(value, field):
+ """Coerce 'value' to a bool, if set or not nullable."""
+ if _not_null(value, field):
+ return value.lower() in ["t", "true", "1"]
+
+
+def _string_from_json(value, _):
+ """NOOP string -> string coercion"""
+ return value
+
+
+def _bytes_from_json(value, field):
+ """Base64-decode value"""
+ if _not_null(value, field):
+ return base64.standard_b64decode(_to_bytes(value))
+
+
+def _timestamp_from_json(value, field):
+ """Coerce 'value' to a datetime, if set or not nullable."""
+ if _not_null(value, field):
+ # value will be a float in seconds, to microsecond precision, in UTC.
+ return _datetime_from_microseconds(1e6 * float(value))
+
+
+def _timestamp_query_param_from_json(value, field):
+ """Coerce 'value' to a datetime, if set or not nullable.
+
+ Args:
+ value (str): The timestamp.
+ field (.SchemaField): The field corresponding to the value.
+
+ Returns:
+ Optional[datetime.datetime]: The parsed datetime object from
+ ``value`` if the ``field`` is not null (otherwise it is
+ :data:`None`).
+ """
+ if _not_null(value, field):
+ # Canonical formats for timestamps in BigQuery are flexible. See:
+ # g.co/cloud/bigquery/docs/reference/standard-sql/data-types#timestamp-type
+ # The separator between the date and time can be 'T' or ' '.
+ value = value.replace(" ", "T", 1)
+ # The UTC timezone may be formatted as Z or +00:00.
+ value = value.replace("Z", "")
+ value = value.replace("+00:00", "")
+
+ if "." in value:
+ # YYYY-MM-DDTHH:MM:SS.ffffff
+ return datetime.datetime.strptime(value, _RFC3339_MICROS_NO_ZULU).replace(
+ tzinfo=UTC
+ )
+ else:
+ # YYYY-MM-DDTHH:MM:SS
+ return datetime.datetime.strptime(value, _RFC3339_NO_FRACTION).replace(
+ tzinfo=UTC
+ )
+ else:
+ return None
+
+
+def _datetime_from_json(value, field):
+ """Coerce 'value' to a datetime, if set or not nullable.
+
+ Args:
+ value (str): The timestamp.
+ field (.SchemaField): The field corresponding to the value.
+
+ Returns:
+ Optional[datetime.datetime]: The parsed datetime object from
+ ``value`` if the ``field`` is not null (otherwise it is
+ :data:`None`).
+ """
+ if _not_null(value, field):
+ if "." in value:
+ # YYYY-MM-DDTHH:MM:SS.ffffff
+ return datetime.datetime.strptime(value, _RFC3339_MICROS_NO_ZULU)
+ else:
+ # YYYY-MM-DDTHH:MM:SS
+ return datetime.datetime.strptime(value, _RFC3339_NO_FRACTION)
+ else:
+ return None
+
+
+def _date_from_json(value, field):
+ """Coerce 'value' to a datetime date, if set or not nullable"""
+ if _not_null(value, field):
+ # value will be a string, in YYYY-MM-DD form.
+ return _date_from_iso8601_date(value)
+
+
+def _time_from_json(value, field):
+ """Coerce 'value' to a datetime date, if set or not nullable"""
+ if _not_null(value, field):
+ if len(value) == 8: # HH:MM:SS
+ fmt = _TIMEONLY_WO_MICROS
+ elif len(value) == 15: # HH:MM:SS.micros
+ fmt = _TIMEONLY_W_MICROS
+ else:
+ raise ValueError("Unknown time format: {}".format(value))
+ return datetime.datetime.strptime(value, fmt).time()
+
+
+def _record_from_json(value, field):
+ """Coerce 'value' to a mapping, if set or not nullable."""
+ if _not_null(value, field):
+ record = {}
+ record_iter = zip(field.fields, value["f"])
+ for subfield, cell in record_iter:
+ converter = _CELLDATA_FROM_JSON[subfield.field_type]
+ if subfield.mode == "REPEATED":
+ value = [converter(item["v"], subfield) for item in cell["v"]]
+ else:
+ value = converter(cell["v"], subfield)
+ record[subfield.name] = value
+ return record
+
+
+_CELLDATA_FROM_JSON = {
+ "INTEGER": _int_from_json,
+ "INT64": _int_from_json,
+ "FLOAT": _float_from_json,
+ "FLOAT64": _float_from_json,
+ "NUMERIC": _decimal_from_json,
+ "BOOLEAN": _bool_from_json,
+ "BOOL": _bool_from_json,
+ "STRING": _string_from_json,
+ "GEOGRAPHY": _string_from_json,
+ "BYTES": _bytes_from_json,
+ "TIMESTAMP": _timestamp_from_json,
+ "DATETIME": _datetime_from_json,
+ "DATE": _date_from_json,
+ "TIME": _time_from_json,
+ "RECORD": _record_from_json,
+}
+
+_QUERY_PARAMS_FROM_JSON = dict(_CELLDATA_FROM_JSON)
+_QUERY_PARAMS_FROM_JSON["TIMESTAMP"] = _timestamp_query_param_from_json
+
+
+def _field_to_index_mapping(schema):
+ """Create a mapping from schema field name to index of field."""
+ return {f.name: i for i, f in enumerate(schema)}
+
+
+def _field_from_json(resource, field):
+ converter = _CELLDATA_FROM_JSON.get(field.field_type, lambda value, _: value)
+ if field.mode == "REPEATED":
+ return [converter(item["v"], field) for item in resource]
+ else:
+ return converter(resource, field)
+
+
+def _row_tuple_from_json(row, schema):
+ """Convert JSON row data to row with appropriate types.
+
+ Note: ``row['f']`` and ``schema`` are presumed to be of the same length.
+
+ :type row: dict
+ :param row: A JSON response row to be converted.
+
+ :type schema: tuple
+ :param schema: A tuple of
+ :class:`~google.cloud.bigquery.schema.SchemaField`.
+
+ :rtype: tuple
+ :returns: A tuple of data converted to native types.
+ """
+ row_data = []
+ for field, cell in zip(schema, row["f"]):
+ row_data.append(_field_from_json(cell["v"], field))
+ return tuple(row_data)
+
+
+def _rows_from_json(values, schema):
+ """Convert JSON row data to rows with appropriate types."""
+ from google.cloud.bigquery import Row
+
+ field_to_index = _field_to_index_mapping(schema)
+ return [Row(_row_tuple_from_json(r, schema), field_to_index) for r in values]
+
+
+def _int_to_json(value):
+ """Coerce 'value' to an JSON-compatible representation."""
+ if isinstance(value, int):
+ value = str(value)
+ return value
+
+
+def _float_to_json(value):
+ """Coerce 'value' to an JSON-compatible representation."""
+ return value
+
+
+def _decimal_to_json(value):
+ """Coerce 'value' to a JSON-compatible representation."""
+ if isinstance(value, decimal.Decimal):
+ value = str(value)
+ return value
+
+
+def _bool_to_json(value):
+ """Coerce 'value' to an JSON-compatible representation."""
+ if isinstance(value, bool):
+ value = "true" if value else "false"
+ return value
+
+
+def _bytes_to_json(value):
+ """Coerce 'value' to an JSON-compatible representation."""
+ if isinstance(value, bytes):
+ value = base64.standard_b64encode(value).decode("ascii")
+ return value
+
+
+def _timestamp_to_json_parameter(value):
+ """Coerce 'value' to an JSON-compatible representation.
+
+ This version returns the string representation used in query parameters.
+ """
+ if isinstance(value, datetime.datetime):
+ if value.tzinfo not in (None, UTC):
+ # Convert to UTC and remove the time zone info.
+ value = value.replace(tzinfo=None) - value.utcoffset()
+ value = "%s %s+00:00" % (value.date().isoformat(), value.time().isoformat())
+ return value
+
+
+def _timestamp_to_json_row(value):
+ """Coerce 'value' to an JSON-compatible representation.
+
+ This version returns floating-point seconds value used in row data.
+ """
+ if isinstance(value, datetime.datetime):
+ value = _microseconds_from_datetime(value) * 1e-6
+ return value
+
+
+def _datetime_to_json(value):
+ """Coerce 'value' to an JSON-compatible representation."""
+ if isinstance(value, datetime.datetime):
+ value = value.strftime(_RFC3339_MICROS_NO_ZULU)
+ return value
+
+
+def _date_to_json(value):
+ """Coerce 'value' to an JSON-compatible representation."""
+ if isinstance(value, datetime.date):
+ value = value.isoformat()
+ return value
+
+
+def _time_to_json(value):
+ """Coerce 'value' to an JSON-compatible representation."""
+ if isinstance(value, datetime.time):
+ value = value.isoformat()
+ return value
+
+
+# Converters used for scalar values marshalled as row data.
+_SCALAR_VALUE_TO_JSON_ROW = {
+ "INTEGER": _int_to_json,
+ "INT64": _int_to_json,
+ "FLOAT": _float_to_json,
+ "FLOAT64": _float_to_json,
+ "NUMERIC": _decimal_to_json,
+ "BOOLEAN": _bool_to_json,
+ "BOOL": _bool_to_json,
+ "BYTES": _bytes_to_json,
+ "TIMESTAMP": _timestamp_to_json_row,
+ "DATETIME": _datetime_to_json,
+ "DATE": _date_to_json,
+ "TIME": _time_to_json,
+}
+
+
+# Converters used for scalar values marshalled as query parameters.
+_SCALAR_VALUE_TO_JSON_PARAM = _SCALAR_VALUE_TO_JSON_ROW.copy()
+_SCALAR_VALUE_TO_JSON_PARAM["TIMESTAMP"] = _timestamp_to_json_parameter
+
+
+def _scalar_field_to_json(field, row_value):
+ """Maps a field and value to a JSON-safe value.
+
+ Args:
+ field ( \
+ :class:`~google.cloud.bigquery.schema.SchemaField`, \
+ ):
+ The SchemaField to use for type conversion and field name.
+ row_value (any):
+ Value to be converted, based on the field's type.
+
+ Returns:
+ any:
+ A JSON-serializable object.
+ """
+ converter = _SCALAR_VALUE_TO_JSON_ROW.get(field.field_type)
+ if converter is None: # STRING doesn't need converting
+ return row_value
+ return converter(row_value)
+
+
+def _repeated_field_to_json(field, row_value):
+ """Convert a repeated/array field to its JSON representation.
+
+ Args:
+ field ( \
+ :class:`~google.cloud.bigquery.schema.SchemaField`, \
+ ):
+ The SchemaField to use for type conversion and field name. The
+ field mode must equal ``REPEATED``.
+ row_value (Sequence[any]):
+ A sequence of values to convert to JSON-serializable values.
+
+ Returns:
+ List[any]:
+ A list of JSON-serializable objects.
+ """
+ # Remove the REPEATED, but keep the other fields. This allows us to process
+ # each item as if it were a top-level field.
+ item_field = copy.deepcopy(field)
+ item_field._mode = "NULLABLE"
+ values = []
+ for item in row_value:
+ values.append(_field_to_json(item_field, item))
+ return values
+
+
+def _record_field_to_json(fields, row_value):
+ """Convert a record/struct field to its JSON representation.
+
+ Args:
+ fields ( \
+ Sequence[:class:`~google.cloud.bigquery.schema.SchemaField`], \
+ ):
+ The :class:`~google.cloud.bigquery.schema.SchemaField`s of the
+ record's subfields to use for type conversion and field names.
+ row_value (Union[Tuple[Any], Mapping[str, Any]):
+ A tuple or dictionary to convert to JSON-serializable values.
+
+ Returns:
+ Mapping[str, any]:
+ A JSON-serializable dictionary.
+ """
+ record = {}
+ isdict = isinstance(row_value, dict)
+
+ for subindex, subfield in enumerate(fields):
+ subname = subfield.name
+ if isdict:
+ subvalue = row_value.get(subname)
+ else:
+ subvalue = row_value[subindex]
+ record[subname] = _field_to_json(subfield, subvalue)
+ return record
+
+
+def _field_to_json(field, row_value):
+ """Convert a field into JSON-serializable values.
+
+ Args:
+ field ( \
+ :class:`~google.cloud.bigquery.schema.SchemaField`, \
+ ):
+ The SchemaField to use for type conversion and field name.
+
+ row_value (Union[ \
+ Sequence[list], \
+ any, \
+ ]):
+ Row data to be inserted. If the SchemaField's mode is
+ REPEATED, assume this is a list. If not, the type
+ is inferred from the SchemaField's field_type.
+
+ Returns:
+ any:
+ A JSON-serializable object.
+ """
+ if row_value is None:
+ return None
+
+ if field.mode == "REPEATED":
+ return _repeated_field_to_json(field, row_value)
+
+ if field.field_type == "RECORD":
+ return _record_field_to_json(field.fields, row_value)
+
+ return _scalar_field_to_json(field, row_value)
+
+
+def _snake_to_camel_case(value):
+ """Convert snake case string to camel case."""
+ words = value.split("_")
+ return words[0] + "".join(map(str.capitalize, words[1:]))
+
+
+def _get_sub_prop(container, keys, default=None):
+ """Get a nested value from a dictionary.
+
+ This method works like ``dict.get(key)``, but for nested values.
+
+ Arguments:
+ container (dict):
+ A dictionary which may contain other dictionaries as values.
+ keys (iterable):
+ A sequence of keys to attempt to get the value for. Each item in
+ the sequence represents a deeper nesting. The first key is for
+ the top level. If there is a dictionary there, the second key
+ attempts to get the value within that, and so on.
+ default (object):
+ (Optional) Value to returned if any of the keys are not found.
+ Defaults to ``None``.
+
+ Examples:
+ Get a top-level value (equivalent to ``container.get('key')``).
+
+ >>> _get_sub_prop({'key': 'value'}, ['key'])
+ 'value'
+
+ Get a top-level value, providing a default (equivalent to
+ ``container.get('key', default='default')``).
+
+ >>> _get_sub_prop({'nothere': 123}, ['key'], default='not found')
+ 'not found'
+
+ Get a nested value.
+
+ >>> _get_sub_prop({'key': {'subkey': 'value'}}, ['key', 'subkey'])
+ 'value'
+
+ Returns:
+ object: The value if present or the default.
+ """
+ sub_val = container
+ for key in keys:
+ if key not in sub_val:
+ return default
+ sub_val = sub_val[key]
+ return sub_val
+
+
+def _set_sub_prop(container, keys, value):
+ """Set a nested value in a dictionary.
+
+ Arguments:
+ container (dict):
+ A dictionary which may contain other dictionaries as values.
+ keys (iterable):
+ A sequence of keys to attempt to set the value for. Each item in
+ the sequence represents a deeper nesting. The first key is for
+ the top level. If there is a dictionary there, the second key
+ attempts to get the value within that, and so on.
+ value (object): Value to set within the container.
+
+ Examples:
+ Set a top-level value (equivalent to ``container['key'] = 'value'``).
+
+ >>> container = {}
+ >>> _set_sub_prop(container, ['key'], 'value')
+ >>> container
+ {'key': 'value'}
+
+ Set a nested value.
+
+ >>> container = {}
+ >>> _set_sub_prop(container, ['key', 'subkey'], 'value')
+ >>> container
+ {'key': {'subkey': 'value'}}
+
+ Replace a nested value.
+
+ >>> container = {'key': {'subkey': 'prev'}}
+ >>> _set_sub_prop(container, ['key', 'subkey'], 'new')
+ >>> container
+ {'key': {'subkey': 'new'}}
+ """
+ sub_val = container
+ for key in keys[:-1]:
+ if key not in sub_val:
+ sub_val[key] = {}
+ sub_val = sub_val[key]
+ sub_val[keys[-1]] = value
+
+
+def _del_sub_prop(container, keys):
+ """Remove a nested key fro a dictionary.
+
+ Arguments:
+ container (dict):
+ A dictionary which may contain other dictionaries as values.
+ keys (iterable):
+ A sequence of keys to attempt to clear the value for. Each item in
+ the sequence represents a deeper nesting. The first key is for
+ the top level. If there is a dictionary there, the second key
+ attempts to get the value within that, and so on.
+
+ Examples:
+ Remove a top-level value (equivalent to ``del container['key']``).
+
+ >>> container = {'key': 'value'}
+ >>> _del_sub_prop(container, ['key'])
+ >>> container
+ {}
+
+ Remove a nested value.
+
+ >>> container = {'key': {'subkey': 'value'}}
+ >>> _del_sub_prop(container, ['key', 'subkey'])
+ >>> container
+ {'key': {}}
+ """
+ sub_val = container
+ for key in keys[:-1]:
+ if key not in sub_val:
+ sub_val[key] = {}
+ sub_val = sub_val[key]
+ if keys[-1] in sub_val:
+ del sub_val[keys[-1]]
+
+
+def _int_or_none(value):
+ """Helper: deserialize int value from JSON string."""
+ if isinstance(value, int):
+ return value
+ if value is not None:
+ return int(value)
+
+
+def _str_or_none(value):
+ """Helper: serialize value to JSON string."""
+ if value is not None:
+ return str(value)
+
+
+def _parse_3_part_id(full_id, default_project=None, property_name="table_id"):
+ output_project_id = default_project
+ output_dataset_id = None
+ output_resource_id = None
+ parts = full_id.split(".")
+
+ if len(parts) != 2 and len(parts) != 3:
+ raise ValueError(
+ "{property_name} must be a fully-qualified ID in "
+ 'standard SQL format. e.g. "project.dataset.{property_name}", '
+ "got {}".format(full_id, property_name=property_name)
+ )
+
+ if len(parts) == 2 and not default_project:
+ raise ValueError(
+ "When default_project is not set, {property_name} must be a "
+ "fully-qualified ID in standard SQL format. "
+ 'e.g. "project.dataset_id.{property_name}", got {}'.format(
+ full_id, property_name=property_name
+ )
+ )
+
+ if len(parts) == 2:
+ output_dataset_id, output_resource_id = parts
+ else:
+ output_project_id, output_dataset_id, output_resource_id = parts
+
+ return output_project_id, output_dataset_id, output_resource_id
+
+
+def _build_resource_from_properties(obj, filter_fields):
+ """Build a resource based on a ``_properties`` dictionary, filtered by
+ ``filter_fields``, which follow the name of the Python object.
+ """
+ partial = {}
+ for filter_field in filter_fields:
+ api_field = obj._PROPERTY_TO_API_FIELD.get(filter_field)
+ if api_field is None and filter_field not in obj._properties:
+ raise ValueError("No property %s" % filter_field)
+ elif api_field is not None:
+ partial[api_field] = obj._properties.get(api_field)
+ else:
+ # allows properties that are not defined in the library
+ # and properties that have the same name as API resource key
+ partial[filter_field] = obj._properties[filter_field]
+
+ return partial
diff --git a/bigquery/google/cloud/bigquery/_http.py b/bigquery/google/cloud/bigquery/_http.py
new file mode 100644
index 000000000000..5dd660ea0b8d
--- /dev/null
+++ b/bigquery/google/cloud/bigquery/_http.py
@@ -0,0 +1,44 @@
+# Copyright 2015 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Create / interact with Google BigQuery connections."""
+
+from google.cloud import _http
+
+from google.cloud.bigquery import __version__
+
+
+class Connection(_http.JSONConnection):
+ """A connection to Google BigQuery via the JSON REST API.
+
+ :type client: :class:`~google.cloud.bigquery.client.Client`
+ :param client: The client that owns the current connection.
+
+ :type client_info: :class:`~google.api_core.client_info.ClientInfo`
+ :param client_info: (Optional) instance used to generate user agent.
+ """
+
+ DEFAULT_API_ENDPOINT = "https://www.googleapis.com"
+
+ def __init__(self, client, client_info=None, api_endpoint=DEFAULT_API_ENDPOINT):
+ super(Connection, self).__init__(client, client_info)
+ self.API_BASE_URL = api_endpoint
+ self._client_info.gapic_version = __version__
+ self._client_info.client_library_version = __version__
+
+ API_VERSION = "v2"
+ """The version of the API, used in building the API call's URL."""
+
+ API_URL_TEMPLATE = "{api_base_url}/bigquery/{api_version}{path}"
+ """A template for the URL of a particular API call."""
diff --git a/bigquery/google/cloud/bigquery/_pandas_helpers.py b/bigquery/google/cloud/bigquery/_pandas_helpers.py
new file mode 100644
index 000000000000..5e73c9f58e22
--- /dev/null
+++ b/bigquery/google/cloud/bigquery/_pandas_helpers.py
@@ -0,0 +1,594 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Shared helper functions for connecting BigQuery and pandas."""
+
+import concurrent.futures
+import functools
+import logging
+import warnings
+
+from six.moves import queue
+
+try:
+ from google.cloud import bigquery_storage_v1beta1
+except ImportError: # pragma: NO COVER
+ bigquery_storage_v1beta1 = None
+
+try:
+ import pandas
+except ImportError: # pragma: NO COVER
+ pandas = None
+
+try:
+ import pyarrow
+ import pyarrow.parquet
+except ImportError: # pragma: NO COVER
+ pyarrow = None
+
+from google.cloud.bigquery import schema
+
+
+_LOGGER = logging.getLogger(__name__)
+
+_NO_BQSTORAGE_ERROR = (
+ "The google-cloud-bigquery-storage library is not installed, "
+ "please install google-cloud-bigquery-storage to use bqstorage features."
+)
+
+_PROGRESS_INTERVAL = 0.2 # Maximum time between download status checks, in seconds.
+
+_PANDAS_DTYPE_TO_BQ = {
+ "bool": "BOOLEAN",
+ "datetime64[ns, UTC]": "TIMESTAMP",
+ "datetime64[ns]": "DATETIME",
+ "float32": "FLOAT",
+ "float64": "FLOAT",
+ "int8": "INTEGER",
+ "int16": "INTEGER",
+ "int32": "INTEGER",
+ "int64": "INTEGER",
+ "uint8": "INTEGER",
+ "uint16": "INTEGER",
+ "uint32": "INTEGER",
+}
+
+
+class _DownloadState(object):
+ """Flag to indicate that a thread should exit early."""
+
+ def __init__(self):
+ # No need for a lock because reading/replacing a variable is defined to
+ # be an atomic operation in the Python language definition (enforced by
+ # the global interpreter lock).
+ self.done = False
+
+
+def pyarrow_datetime():
+ return pyarrow.timestamp("us", tz=None)
+
+
+def pyarrow_numeric():
+ return pyarrow.decimal128(38, 9)
+
+
+def pyarrow_time():
+ return pyarrow.time64("us")
+
+
+def pyarrow_timestamp():
+ return pyarrow.timestamp("us", tz="UTC")
+
+
+if pyarrow:
+ # This dictionary is duplicated in bigquery_storage/test/unite/test_reader.py
+ # When modifying it be sure to update it there as well.
+ BQ_TO_ARROW_SCALARS = {
+ "BOOL": pyarrow.bool_,
+ "BOOLEAN": pyarrow.bool_,
+ "BYTES": pyarrow.binary,
+ "DATE": pyarrow.date32,
+ "DATETIME": pyarrow_datetime,
+ "FLOAT": pyarrow.float64,
+ "FLOAT64": pyarrow.float64,
+ "GEOGRAPHY": pyarrow.string,
+ "INT64": pyarrow.int64,
+ "INTEGER": pyarrow.int64,
+ "NUMERIC": pyarrow_numeric,
+ "STRING": pyarrow.string,
+ "TIME": pyarrow_time,
+ "TIMESTAMP": pyarrow_timestamp,
+ }
+else: # pragma: NO COVER
+ BQ_TO_ARROW_SCALARS = {} # pragma: NO COVER
+
+
+def bq_to_arrow_struct_data_type(field):
+ arrow_fields = []
+ for subfield in field.fields:
+ arrow_subfield = bq_to_arrow_field(subfield)
+ if arrow_subfield:
+ arrow_fields.append(arrow_subfield)
+ else:
+ # Could not determine a subfield type. Fallback to type
+ # inference.
+ return None
+ return pyarrow.struct(arrow_fields)
+
+
+def bq_to_arrow_data_type(field):
+ """Return the Arrow data type, corresponding to a given BigQuery column.
+
+ Returns None if default Arrow type inspection should be used.
+ """
+ if field.mode is not None and field.mode.upper() == "REPEATED":
+ inner_type = bq_to_arrow_data_type(
+ schema.SchemaField(field.name, field.field_type, fields=field.fields)
+ )
+ if inner_type:
+ return pyarrow.list_(inner_type)
+ return None
+
+ if field.field_type.upper() in schema._STRUCT_TYPES:
+ return bq_to_arrow_struct_data_type(field)
+
+ data_type_constructor = BQ_TO_ARROW_SCALARS.get(field.field_type.upper())
+ if data_type_constructor is None:
+ return None
+ return data_type_constructor()
+
+
+def bq_to_arrow_field(bq_field):
+ """Return the Arrow field, corresponding to a given BigQuery column.
+
+ Returns None if the Arrow type cannot be determined.
+ """
+ arrow_type = bq_to_arrow_data_type(bq_field)
+ if arrow_type:
+ is_nullable = bq_field.mode.upper() == "NULLABLE"
+ return pyarrow.field(bq_field.name, arrow_type, nullable=is_nullable)
+
+ warnings.warn("Unable to determine type for field '{}'.".format(bq_field.name))
+ return None
+
+
+def bq_to_arrow_schema(bq_schema):
+ """Return the Arrow schema, corresponding to a given BigQuery schema.
+
+ Returns None if any Arrow type cannot be determined.
+ """
+ arrow_fields = []
+ for bq_field in bq_schema:
+ arrow_field = bq_to_arrow_field(bq_field)
+ if arrow_field is None:
+ # Auto-detect the schema if there is an unknown field type.
+ return None
+ arrow_fields.append(arrow_field)
+ return pyarrow.schema(arrow_fields)
+
+
+def bq_to_arrow_array(series, bq_field):
+ arrow_type = bq_to_arrow_data_type(bq_field)
+ if bq_field.mode.upper() == "REPEATED":
+ return pyarrow.ListArray.from_pandas(series, type=arrow_type)
+ if bq_field.field_type.upper() in schema._STRUCT_TYPES:
+ return pyarrow.StructArray.from_pandas(series, type=arrow_type)
+ return pyarrow.array(series, type=arrow_type)
+
+
+def get_column_or_index(dataframe, name):
+ """Return a column or index as a pandas series."""
+ if name in dataframe.columns:
+ return dataframe[name].reset_index(drop=True)
+
+ if isinstance(dataframe.index, pandas.MultiIndex):
+ if name in dataframe.index.names:
+ return (
+ dataframe.index.get_level_values(name)
+ .to_series()
+ .reset_index(drop=True)
+ )
+ else:
+ if name == dataframe.index.name:
+ return dataframe.index.to_series().reset_index(drop=True)
+
+ raise ValueError("column or index '{}' not found.".format(name))
+
+
+def list_columns_and_indexes(dataframe):
+ """Return all index and column names with dtypes.
+
+ Returns:
+ Sequence[Tuple[dtype, str]]:
+ Returns a sorted list of indexes and column names with
+ corresponding dtypes. If an index is missing a name or has the
+ same name as a column, the index is omitted.
+ """
+ column_names = frozenset(dataframe.columns)
+ columns_and_indexes = []
+ if isinstance(dataframe.index, pandas.MultiIndex):
+ for name in dataframe.index.names:
+ if name and name not in column_names:
+ values = dataframe.index.get_level_values(name)
+ columns_and_indexes.append((name, values.dtype))
+ else:
+ if dataframe.index.name and dataframe.index.name not in column_names:
+ columns_and_indexes.append((dataframe.index.name, dataframe.index.dtype))
+
+ columns_and_indexes += zip(dataframe.columns, dataframe.dtypes)
+ return columns_and_indexes
+
+
+def dataframe_to_bq_schema(dataframe, bq_schema):
+ """Convert a pandas DataFrame schema to a BigQuery schema.
+
+ Args:
+ dataframe (pandas.DataFrame):
+ DataFrame for which the client determines the BigQuery schema.
+ bq_schema (Sequence[google.cloud.bigquery.schema.SchemaField]):
+ A BigQuery schema. Use this argument to override the autodetected
+ type for some or all of the DataFrame columns.
+
+ Returns:
+ Optional[Sequence[google.cloud.bigquery.schema.SchemaField]]:
+ The automatically determined schema. Returns None if the type of
+ any column cannot be determined.
+ """
+ if bq_schema:
+ for field in bq_schema:
+ if field.field_type in schema._STRUCT_TYPES:
+ raise ValueError(
+ "Uploading dataframes with struct (record) column types "
+ "is not supported. See: "
+ "https://github.com/googleapis/google-cloud-python/issues/8191"
+ )
+ bq_schema_index = {field.name: field for field in bq_schema}
+ bq_schema_unused = set(bq_schema_index.keys())
+ else:
+ bq_schema_index = {}
+ bq_schema_unused = set()
+
+ bq_schema_out = []
+ for column, dtype in list_columns_and_indexes(dataframe):
+ # Use provided type from schema, if present.
+ bq_field = bq_schema_index.get(column)
+ if bq_field:
+ bq_schema_out.append(bq_field)
+ bq_schema_unused.discard(bq_field.name)
+ continue
+
+ # Otherwise, try to automatically determine the type based on the
+ # pandas dtype.
+ bq_type = _PANDAS_DTYPE_TO_BQ.get(dtype.name)
+ if not bq_type:
+ warnings.warn(u"Unable to determine type of column '{}'.".format(column))
+ return None
+ bq_field = schema.SchemaField(column, bq_type)
+ bq_schema_out.append(bq_field)
+
+ # Catch any schema mismatch. The developer explicitly asked to serialize a
+ # column, but it was not found.
+ if bq_schema_unused:
+ raise ValueError(
+ u"bq_schema contains fields not present in dataframe: {}".format(
+ bq_schema_unused
+ )
+ )
+ return tuple(bq_schema_out)
+
+
+def dataframe_to_arrow(dataframe, bq_schema):
+ """Convert pandas dataframe to Arrow table, using BigQuery schema.
+
+ Args:
+ dataframe (pandas.DataFrame):
+ DataFrame to convert to Arrow table.
+ bq_schema (Sequence[google.cloud.bigquery.schema.SchemaField]):
+ Desired BigQuery schema. Number of columns must match number of
+ columns in the DataFrame.
+
+ Returns:
+ pyarrow.Table:
+ Table containing dataframe data, with schema derived from
+ BigQuery schema.
+ """
+ column_names = set(dataframe.columns)
+ column_and_index_names = set(
+ name for name, _ in list_columns_and_indexes(dataframe)
+ )
+ bq_field_names = set(field.name for field in bq_schema)
+
+ extra_fields = bq_field_names - column_and_index_names
+ if extra_fields:
+ raise ValueError(
+ u"bq_schema contains fields not present in dataframe: {}".format(
+ extra_fields
+ )
+ )
+
+ # It's okay for indexes to be missing from bq_schema, but it's not okay to
+ # be missing columns.
+ missing_fields = column_names - bq_field_names
+ if missing_fields:
+ raise ValueError(
+ u"bq_schema is missing fields from dataframe: {}".format(missing_fields)
+ )
+
+ arrow_arrays = []
+ arrow_names = []
+ arrow_fields = []
+ for bq_field in bq_schema:
+ arrow_fields.append(bq_to_arrow_field(bq_field))
+ arrow_names.append(bq_field.name)
+ arrow_arrays.append(
+ bq_to_arrow_array(get_column_or_index(dataframe, bq_field.name), bq_field)
+ )
+
+ if all((field is not None for field in arrow_fields)):
+ return pyarrow.Table.from_arrays(
+ arrow_arrays, schema=pyarrow.schema(arrow_fields)
+ )
+ return pyarrow.Table.from_arrays(arrow_arrays, names=arrow_names)
+
+
+def dataframe_to_parquet(dataframe, bq_schema, filepath, parquet_compression="SNAPPY"):
+ """Write dataframe as a Parquet file, according to the desired BQ schema.
+
+ This function requires the :mod:`pyarrow` package. Arrow is used as an
+ intermediate format.
+
+ Args:
+ dataframe (pandas.DataFrame):
+ DataFrame to convert to Parquet file.
+ bq_schema (Sequence[google.cloud.bigquery.schema.SchemaField]):
+ Desired BigQuery schema. Number of columns must match number of
+ columns in the DataFrame.
+ filepath (str):
+ Path to write Parquet file to.
+ parquet_compression (str):
+ (optional) The compression codec to use by the the
+ ``pyarrow.parquet.write_table`` serializing method. Defaults to
+ "SNAPPY".
+ https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html#pyarrow-parquet-write-table
+ """
+ if pyarrow is None:
+ raise ValueError("pyarrow is required for BigQuery schema conversion.")
+
+ arrow_table = dataframe_to_arrow(dataframe, bq_schema)
+ pyarrow.parquet.write_table(arrow_table, filepath, compression=parquet_compression)
+
+
+def _tabledata_list_page_to_arrow(page, column_names, arrow_types):
+ # Iterate over the page to force the API request to get the page data.
+ try:
+ next(iter(page))
+ except StopIteration:
+ pass
+
+ arrays = []
+ for column_index, arrow_type in enumerate(arrow_types):
+ arrays.append(pyarrow.array(page._columns[column_index], type=arrow_type))
+
+ return pyarrow.RecordBatch.from_arrays(arrays, column_names)
+
+
+def download_arrow_tabledata_list(pages, schema):
+ """Use tabledata.list to construct an iterable of RecordBatches."""
+ column_names = bq_to_arrow_schema(schema) or [field.name for field in schema]
+ arrow_types = [bq_to_arrow_data_type(field) for field in schema]
+
+ for page in pages:
+ yield _tabledata_list_page_to_arrow(page, column_names, arrow_types)
+
+
+def _tabledata_list_page_to_dataframe(page, column_names, dtypes):
+ # Iterate over the page to force the API request to get the page data.
+ try:
+ next(iter(page))
+ except StopIteration:
+ pass
+
+ columns = {}
+ for column_index, column_name in enumerate(column_names):
+ dtype = dtypes.get(column_name)
+ columns[column_name] = pandas.Series(page._columns[column_index], dtype=dtype)
+
+ return pandas.DataFrame(columns, columns=column_names)
+
+
+def download_dataframe_tabledata_list(pages, schema, dtypes):
+ """Use (slower, but free) tabledata.list to construct a DataFrame."""
+ column_names = [field.name for field in schema]
+ for page in pages:
+ yield _tabledata_list_page_to_dataframe(page, column_names, dtypes)
+
+
+def _bqstorage_page_to_arrow(page):
+ return page.to_arrow()
+
+
+def _bqstorage_page_to_dataframe(column_names, dtypes, page):
+ # page.to_dataframe() does not preserve column order in some versions
+ # of google-cloud-bigquery-storage. Access by column name to rearrange.
+ return page.to_dataframe(dtypes=dtypes)[column_names]
+
+
+def _download_table_bqstorage_stream(
+ download_state, bqstorage_client, session, stream, worker_queue, page_to_item
+):
+ position = bigquery_storage_v1beta1.types.StreamPosition(stream=stream)
+ rowstream = bqstorage_client.read_rows(position).rows(session)
+
+ for page in rowstream.pages:
+ if download_state.done:
+ return
+ item = page_to_item(page)
+ worker_queue.put(item)
+
+
+def _nowait(futures):
+ """Separate finished and unfinished threads, much like
+ :func:`concurrent.futures.wait`, but don't wait.
+ """
+ done = []
+ not_done = []
+ for future in futures:
+ if future.done():
+ done.append(future)
+ else:
+ not_done.append(future)
+ return done, not_done
+
+
+def _download_table_bqstorage(
+ project_id,
+ table,
+ bqstorage_client,
+ preserve_order=False,
+ selected_fields=None,
+ page_to_item=None,
+):
+ """Use (faster, but billable) BQ Storage API to construct DataFrame."""
+ if "$" in table.table_id:
+ raise ValueError(
+ "Reading from a specific partition is not currently supported."
+ )
+ if "@" in table.table_id:
+ raise ValueError("Reading from a specific snapshot is not currently supported.")
+
+ read_options = bigquery_storage_v1beta1.types.TableReadOptions()
+ if selected_fields is not None:
+ for field in selected_fields:
+ read_options.selected_fields.append(field.name)
+
+ requested_streams = 0
+ if preserve_order:
+ requested_streams = 1
+
+ session = bqstorage_client.create_read_session(
+ table.to_bqstorage(),
+ "projects/{}".format(project_id),
+ format_=bigquery_storage_v1beta1.enums.DataFormat.ARROW,
+ read_options=read_options,
+ requested_streams=requested_streams,
+ )
+ _LOGGER.debug(
+ "Started reading table '{}.{}.{}' with BQ Storage API session '{}'.".format(
+ table.project, table.dataset_id, table.table_id, session.name
+ )
+ )
+
+ # Avoid reading rows from an empty table.
+ if not session.streams:
+ return
+
+ total_streams = len(session.streams)
+
+ # Use _DownloadState to notify worker threads when to quit.
+ # See: https://stackoverflow.com/a/29237343/101923
+ download_state = _DownloadState()
+
+ # Create a queue to collect frames as they are created in each thread.
+ worker_queue = queue.Queue()
+
+ with concurrent.futures.ThreadPoolExecutor(max_workers=total_streams) as pool:
+ try:
+ # Manually submit jobs and wait for download to complete rather
+ # than using pool.map because pool.map continues running in the
+ # background even if there is an exception on the main thread.
+ # See: https://github.com/googleapis/google-cloud-python/pull/7698
+ not_done = [
+ pool.submit(
+ _download_table_bqstorage_stream,
+ download_state,
+ bqstorage_client,
+ session,
+ stream,
+ worker_queue,
+ page_to_item,
+ )
+ for stream in session.streams
+ ]
+
+ while not_done:
+ # Don't block on the worker threads. For performance reasons,
+ # we want to block on the queue's get method, instead. This
+ # prevents the queue from filling up, because the main thread
+ # has smaller gaps in time between calls to the queue's get
+ # method. For a detailed explaination, see:
+ # https://friendliness.dev/2019/06/18/python-nowait/
+ done, not_done = _nowait(not_done)
+ for future in done:
+ # Call result() on any finished threads to raise any
+ # exceptions encountered.
+ future.result()
+
+ try:
+ frame = worker_queue.get(timeout=_PROGRESS_INTERVAL)
+ yield frame
+ except queue.Empty: # pragma: NO COVER
+ continue
+
+ # Return any remaining values after the workers finished.
+ while not worker_queue.empty(): # pragma: NO COVER
+ try:
+ # Include a timeout because even though the queue is
+ # non-empty, it doesn't guarantee that a subsequent call to
+ # get() will not block.
+ frame = worker_queue.get(timeout=_PROGRESS_INTERVAL)
+ yield frame
+ except queue.Empty: # pragma: NO COVER
+ continue
+ finally:
+ # No need for a lock because reading/replacing a variable is
+ # defined to be an atomic operation in the Python language
+ # definition (enforced by the global interpreter lock).
+ download_state.done = True
+
+ # Shutdown all background threads, now that they should know to
+ # exit early.
+ pool.shutdown(wait=True)
+
+
+def download_arrow_bqstorage(
+ project_id, table, bqstorage_client, preserve_order=False, selected_fields=None
+):
+ return _download_table_bqstorage(
+ project_id,
+ table,
+ bqstorage_client,
+ preserve_order=preserve_order,
+ selected_fields=selected_fields,
+ page_to_item=_bqstorage_page_to_arrow,
+ )
+
+
+def download_dataframe_bqstorage(
+ project_id,
+ table,
+ bqstorage_client,
+ column_names,
+ dtypes,
+ preserve_order=False,
+ selected_fields=None,
+):
+ page_to_item = functools.partial(_bqstorage_page_to_dataframe, column_names, dtypes)
+ return _download_table_bqstorage(
+ project_id,
+ table,
+ bqstorage_client,
+ preserve_order=preserve_order,
+ selected_fields=selected_fields,
+ page_to_item=page_to_item,
+ )
diff --git a/bigquery/google/cloud/bigquery/client.py b/bigquery/google/cloud/bigquery/client.py
new file mode 100644
index 000000000000..047633642e1e
--- /dev/null
+++ b/bigquery/google/cloud/bigquery/client.py
@@ -0,0 +1,2500 @@
+# Copyright 2015 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Client for interacting with the Google BigQuery API."""
+
+from __future__ import absolute_import
+
+try:
+ from collections import abc as collections_abc
+except ImportError: # Python 2.7
+ import collections as collections_abc
+
+import copy
+import functools
+import gzip
+import io
+import json
+import os
+import tempfile
+import uuid
+import warnings
+
+try:
+ import pyarrow
+except ImportError: # pragma: NO COVER
+ pyarrow = None
+import six
+
+from google import resumable_media
+from google.resumable_media.requests import MultipartUpload
+from google.resumable_media.requests import ResumableUpload
+
+import google.api_core.client_options
+import google.api_core.exceptions
+from google.api_core import page_iterator
+import google.cloud._helpers
+from google.cloud import exceptions
+from google.cloud.client import ClientWithProject
+
+from google.cloud.bigquery._helpers import _record_field_to_json
+from google.cloud.bigquery._helpers import _str_or_none
+from google.cloud.bigquery._http import Connection
+from google.cloud.bigquery import _pandas_helpers
+from google.cloud.bigquery.dataset import Dataset
+from google.cloud.bigquery.dataset import DatasetListItem
+from google.cloud.bigquery.dataset import DatasetReference
+from google.cloud.bigquery import job
+from google.cloud.bigquery.model import Model
+from google.cloud.bigquery.model import ModelReference
+from google.cloud.bigquery.query import _QueryResults
+from google.cloud.bigquery.retry import DEFAULT_RETRY
+from google.cloud.bigquery.routine import Routine
+from google.cloud.bigquery.routine import RoutineReference
+from google.cloud.bigquery.schema import SchemaField
+from google.cloud.bigquery.table import _table_arg_to_table
+from google.cloud.bigquery.table import _table_arg_to_table_ref
+from google.cloud.bigquery.table import Table
+from google.cloud.bigquery.table import TableListItem
+from google.cloud.bigquery.table import TableReference
+from google.cloud.bigquery.table import RowIterator
+
+
+_DEFAULT_CHUNKSIZE = 1048576 # 1024 * 1024 B = 1 MB
+_MAX_MULTIPART_SIZE = 5 * 1024 * 1024
+_DEFAULT_NUM_RETRIES = 6
+_BASE_UPLOAD_TEMPLATE = (
+ u"https://www.googleapis.com/upload/bigquery/v2/projects/"
+ u"{project}/jobs?uploadType="
+)
+_MULTIPART_URL_TEMPLATE = _BASE_UPLOAD_TEMPLATE + u"multipart"
+_RESUMABLE_URL_TEMPLATE = _BASE_UPLOAD_TEMPLATE + u"resumable"
+_GENERIC_CONTENT_TYPE = u"*/*"
+_READ_LESS_THAN_SIZE = (
+ "Size {:d} was specified but the file-like object only had " "{:d} bytes remaining."
+)
+_NEED_TABLE_ARGUMENT = (
+ "The table argument should be a table ID string, Table, or TableReference"
+)
+
+
+class Project(object):
+ """Wrapper for resource describing a BigQuery project.
+
+ :type project_id: str
+ :param project_id: Opaque ID of the project
+
+ :type numeric_id: int
+ :param numeric_id: Numeric ID of the project
+
+ :type friendly_name: str
+ :param friendly_name: Display name of the project
+ """
+
+ def __init__(self, project_id, numeric_id, friendly_name):
+ self.project_id = project_id
+ self.numeric_id = numeric_id
+ self.friendly_name = friendly_name
+
+ @classmethod
+ def from_api_repr(cls, resource):
+ """Factory: construct an instance from a resource dict."""
+ return cls(resource["id"], resource["numericId"], resource["friendlyName"])
+
+
+class Client(ClientWithProject):
+ """Client to bundle configuration needed for API requests.
+
+ Args:
+ project (str):
+ Project ID for the project which the client acts on behalf of.
+ Will be passed when creating a dataset / job. If not passed,
+ falls back to the default inferred from the environment.
+ credentials (google.auth.credentials.Credentials):
+ (Optional) The OAuth2 Credentials to use for this client. If not
+ passed (and if no ``_http`` object is passed), falls back to the
+ default inferred from the environment.
+ _http (requests.Session):
+ (Optional) HTTP object to make requests. Can be any object that
+ defines ``request()`` with the same interface as
+ :meth:`requests.Session.request`. If not passed, an ``_http``
+ object is created that is bound to the ``credentials`` for the
+ current object.
+ This parameter should be considered private, and could change in
+ the future.
+ location (str):
+ (Optional) Default location for jobs / datasets / tables.
+ default_query_job_config (google.cloud.bigquery.job.QueryJobConfig):
+ (Optional) Default ``QueryJobConfig``.
+ Will be merged into job configs passed into the ``query`` method.
+ client_info (google.api_core.client_info.ClientInfo):
+ The client info used to send a user-agent string along with API
+ requests. If ``None``, then default info will be used. Generally,
+ you only need to set this if you're developing your own library
+ or partner tool.
+ client_options (Union[~google.api_core.client_options.ClientOptions, dict]):
+ (Optional) Client options used to set user options on the client.
+ API Endpoint should be set through client_options.
+
+ Raises:
+ google.auth.exceptions.DefaultCredentialsError:
+ Raised if ``credentials`` is not specified and the library fails
+ to acquire default credentials.
+ """
+
+ SCOPE = (
+ "https://www.googleapis.com/auth/bigquery",
+ "https://www.googleapis.com/auth/cloud-platform",
+ )
+ """The scopes required for authenticating as a BigQuery consumer."""
+
+ def __init__(
+ self,
+ project=None,
+ credentials=None,
+ _http=None,
+ location=None,
+ default_query_job_config=None,
+ client_info=None,
+ client_options=None,
+ ):
+ super(Client, self).__init__(
+ project=project, credentials=credentials, _http=_http
+ )
+
+ kw_args = {"client_info": client_info}
+ if client_options:
+ if type(client_options) == dict:
+ client_options = google.api_core.client_options.from_dict(
+ client_options
+ )
+ if client_options.api_endpoint:
+ api_endpoint = client_options.api_endpoint
+ kw_args["api_endpoint"] = api_endpoint
+
+ self._connection = Connection(self, **kw_args)
+ self._location = location
+ self._default_query_job_config = default_query_job_config
+
+ @property
+ def location(self):
+ """Default location for jobs / datasets / tables."""
+ return self._location
+
+ def get_service_account_email(self, project=None):
+ """Get the email address of the project's BigQuery service account
+
+ Note:
+ This is the service account that BigQuery uses to manage tables
+ encrypted by a key in KMS.
+
+ Args:
+ project (str, optional):
+ Project ID to use for retreiving service account email.
+ Defaults to the client's project.
+
+ Returns:
+ str: service account email address
+
+ Example:
+
+ >>> from google.cloud import bigquery
+ >>> client = bigquery.Client()
+ >>> client.get_service_account_email()
+ my_service_account@my-project.iam.gserviceaccount.com
+
+ """
+ if project is None:
+ project = self.project
+ path = "/projects/%s/serviceAccount" % (project,)
+ api_response = self._connection.api_request(method="GET", path=path)
+ return api_response["email"]
+
+ def list_projects(self, max_results=None, page_token=None, retry=DEFAULT_RETRY):
+ """List projects for the project associated with this client.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/projects/list
+
+ :type max_results: int
+ :param max_results: (Optional) maximum number of projects to return,
+ If not passed, defaults to a value set by the API.
+
+ :type page_token: str
+ :param page_token:
+ (Optional) Token representing a cursor into the projects. If
+ not passed, the API will return the first page of projects.
+ The token marks the beginning of the iterator to be returned
+ and the value of the ``page_token`` can be accessed at
+ ``next_page_token`` of the
+ :class:`~google.api_core.page_iterator.HTTPIterator`.
+
+ :type retry: :class:`google.api_core.retry.Retry`
+ :param retry: (Optional) How to retry the RPC.
+
+ :rtype: :class:`~google.api_core.page_iterator.Iterator`
+ :returns: Iterator of :class:`~google.cloud.bigquery.client.Project`
+ accessible to the current client.
+ """
+ return page_iterator.HTTPIterator(
+ client=self,
+ api_request=functools.partial(self._call_api, retry),
+ path="/projects",
+ item_to_value=_item_to_project,
+ items_key="projects",
+ page_token=page_token,
+ max_results=max_results,
+ )
+
+ def list_datasets(
+ self,
+ project=None,
+ include_all=False,
+ filter=None,
+ max_results=None,
+ page_token=None,
+ retry=DEFAULT_RETRY,
+ ):
+ """List datasets for the project associated with this client.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/list
+
+ Args:
+ project (str):
+ Optional. Project ID to use for retreiving datasets. Defaults
+ to the client's project.
+ include_all (bool):
+ Optional. True if results include hidden datasets. Defaults
+ to False.
+ filter (str):
+ Optional. An expression for filtering the results by label.
+ For syntax, see
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/list#filter.
+ max_results (int):
+ Optional. Maximum number of datasets to return.
+ page_token (str):
+ Optional. Token representing a cursor into the datasets. If
+ not passed, the API will return the first page of datasets.
+ The token marks the beginning of the iterator to be returned
+ and the value of the ``page_token`` can be accessed at
+ ``next_page_token`` of the
+ :class:`~google.api_core.page_iterator.HTTPIterator`.
+ retry (google.api_core.retry.Retry):
+ Optional. How to retry the RPC.
+
+ Returns:
+ google.api_core.page_iterator.Iterator:
+ Iterator of
+ :class:`~google.cloud.bigquery.dataset.DatasetListItem`.
+ associated with the project.
+ """
+ extra_params = {}
+ if project is None:
+ project = self.project
+ if include_all:
+ extra_params["all"] = True
+ if filter:
+ # TODO: consider supporting a dict of label -> value for filter,
+ # and converting it into a string here.
+ extra_params["filter"] = filter
+ path = "/projects/%s/datasets" % (project,)
+ return page_iterator.HTTPIterator(
+ client=self,
+ api_request=functools.partial(self._call_api, retry),
+ path=path,
+ item_to_value=_item_to_dataset,
+ items_key="datasets",
+ page_token=page_token,
+ max_results=max_results,
+ extra_params=extra_params,
+ )
+
+ def dataset(self, dataset_id, project=None):
+ """Construct a reference to a dataset.
+
+ :type dataset_id: str
+ :param dataset_id: ID of the dataset.
+
+ :type project: str
+ :param project: (Optional) project ID for the dataset (defaults to
+ the project of the client).
+
+ :rtype: :class:`google.cloud.bigquery.dataset.DatasetReference`
+ :returns: a new ``DatasetReference`` instance
+ """
+ if project is None:
+ project = self.project
+
+ return DatasetReference(project, dataset_id)
+
+ def create_dataset(self, dataset, exists_ok=False, retry=DEFAULT_RETRY):
+ """API call: create the dataset via a POST request.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables/insert
+
+ Args:
+ dataset (Union[ \
+ :class:`~google.cloud.bigquery.dataset.Dataset`, \
+ :class:`~google.cloud.bigquery.dataset.DatasetReference`, \
+ str, \
+ ]):
+ A :class:`~google.cloud.bigquery.dataset.Dataset` to create.
+ If ``dataset`` is a reference, an empty dataset is created
+ with the specified ID and client's default location.
+ exists_ok (bool):
+ Defaults to ``False``. If ``True``, ignore "already exists"
+ errors when creating the dataset.
+ retry (google.api_core.retry.Retry):
+ Optional. How to retry the RPC.
+
+ Returns:
+ google.cloud.bigquery.dataset.Dataset:
+ A new ``Dataset`` returned from the API.
+
+ Example:
+
+ >>> from google.cloud import bigquery
+ >>> client = bigquery.Client()
+ >>> dataset = bigquery.Dataset(client.dataset('my_dataset'))
+ >>> dataset = client.create_dataset(dataset)
+
+ """
+ if isinstance(dataset, str):
+ dataset = DatasetReference.from_string(
+ dataset, default_project=self.project
+ )
+ if isinstance(dataset, DatasetReference):
+ dataset = Dataset(dataset)
+
+ path = "/projects/%s/datasets" % (dataset.project,)
+
+ data = dataset.to_api_repr()
+ if data.get("location") is None and self.location is not None:
+ data["location"] = self.location
+
+ try:
+ api_response = self._call_api(retry, method="POST", path=path, data=data)
+ return Dataset.from_api_repr(api_response)
+ except google.api_core.exceptions.Conflict:
+ if not exists_ok:
+ raise
+ return self.get_dataset(dataset.reference, retry=retry)
+
+ def create_routine(self, routine, exists_ok=False, retry=DEFAULT_RETRY):
+ """[Beta] Create a routine via a POST request.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/routines/insert
+
+ Args:
+ routine (:class:`~google.cloud.bigquery.routine.Routine`):
+ A :class:`~google.cloud.bigquery.routine.Routine` to create.
+ The dataset that the routine belongs to must already exist.
+ exists_ok (bool):
+ Defaults to ``False``. If ``True``, ignore "already exists"
+ errors when creating the routine.
+ retry (google.api_core.retry.Retry):
+ Optional. How to retry the RPC.
+
+ Returns:
+ google.cloud.bigquery.routine.Routine:
+ A new ``Routine`` returned from the service.
+ """
+ reference = routine.reference
+ path = "/projects/{}/datasets/{}/routines".format(
+ reference.project, reference.dataset_id
+ )
+ resource = routine.to_api_repr()
+ try:
+ api_response = self._call_api(
+ retry, method="POST", path=path, data=resource
+ )
+ return Routine.from_api_repr(api_response)
+ except google.api_core.exceptions.Conflict:
+ if not exists_ok:
+ raise
+ return self.get_routine(routine.reference, retry=retry)
+
+ def create_table(self, table, exists_ok=False, retry=DEFAULT_RETRY):
+ """API call: create a table via a PUT request
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables/insert
+
+ Args:
+ table (Union[ \
+ :class:`~google.cloud.bigquery.table.Table`, \
+ :class:`~google.cloud.bigquery.table.TableReference`, \
+ str, \
+ ]):
+ A :class:`~google.cloud.bigquery.table.Table` to create.
+ If ``table`` is a reference, an empty table is created
+ with the specified ID. The dataset that the table belongs to
+ must already exist.
+ exists_ok (bool):
+ Defaults to ``False``. If ``True``, ignore "already exists"
+ errors when creating the table.
+ retry (google.api_core.retry.Retry):
+ Optional. How to retry the RPC.
+
+ Returns:
+ google.cloud.bigquery.table.Table:
+ A new ``Table`` returned from the service.
+ """
+ table = _table_arg_to_table(table, default_project=self.project)
+
+ path = "/projects/%s/datasets/%s/tables" % (table.project, table.dataset_id)
+ data = table.to_api_repr()
+ try:
+ api_response = self._call_api(retry, method="POST", path=path, data=data)
+ return Table.from_api_repr(api_response)
+ except google.api_core.exceptions.Conflict:
+ if not exists_ok:
+ raise
+ return self.get_table(table.reference, retry=retry)
+
+ def _call_api(self, retry, **kwargs):
+ call = functools.partial(self._connection.api_request, **kwargs)
+ if retry:
+ call = retry(call)
+ return call()
+
+ def get_dataset(self, dataset_ref, retry=DEFAULT_RETRY):
+ """Fetch the dataset referenced by ``dataset_ref``
+
+ Args:
+ dataset_ref (Union[ \
+ :class:`~google.cloud.bigquery.dataset.DatasetReference`, \
+ str, \
+ ]):
+ A reference to the dataset to fetch from the BigQuery API.
+ If a string is passed in, this method attempts to create a
+ dataset reference from a string using
+ :func:`~google.cloud.bigquery.dataset.DatasetReference.from_string`.
+ retry (:class:`google.api_core.retry.Retry`):
+ (Optional) How to retry the RPC.
+
+ Returns:
+ google.cloud.bigquery.dataset.Dataset:
+ A ``Dataset`` instance.
+ """
+ if isinstance(dataset_ref, str):
+ dataset_ref = DatasetReference.from_string(
+ dataset_ref, default_project=self.project
+ )
+
+ api_response = self._call_api(retry, method="GET", path=dataset_ref.path)
+ return Dataset.from_api_repr(api_response)
+
+ def get_model(self, model_ref, retry=DEFAULT_RETRY):
+ """[Beta] Fetch the model referenced by ``model_ref``.
+
+ Args:
+ model_ref (Union[ \
+ :class:`~google.cloud.bigquery.model.ModelReference`, \
+ str, \
+ ]):
+ A reference to the model to fetch from the BigQuery API.
+ If a string is passed in, this method attempts to create a
+ model reference from a string using
+ :func:`google.cloud.bigquery.model.ModelReference.from_string`.
+ retry (:class:`google.api_core.retry.Retry`):
+ (Optional) How to retry the RPC.
+
+ Returns:
+ google.cloud.bigquery.model.Model:
+ A ``Model`` instance.
+ """
+ if isinstance(model_ref, str):
+ model_ref = ModelReference.from_string(
+ model_ref, default_project=self.project
+ )
+
+ api_response = self._call_api(retry, method="GET", path=model_ref.path)
+ return Model.from_api_repr(api_response)
+
+ def get_routine(self, routine_ref, retry=DEFAULT_RETRY):
+ """[Beta] Get the routine referenced by ``routine_ref``.
+
+ Args:
+ routine_ref (Union[ \
+ :class:`~google.cloud.bigquery.routine.Routine`, \
+ :class:`~google.cloud.bigquery.routine.RoutineReference`, \
+ str, \
+ ]):
+ A reference to the routine to fetch from the BigQuery API. If
+ a string is passed in, this method attempts to create a
+ reference from a string using
+ :func:`google.cloud.bigquery.routine.RoutineReference.from_string`.
+ retry (:class:`google.api_core.retry.Retry`):
+ (Optional) How to retry the API call.
+
+ Returns:
+ google.cloud.bigquery.routine.Routine:
+ A ``Routine`` instance.
+ """
+ if isinstance(routine_ref, str):
+ routine_ref = RoutineReference.from_string(
+ routine_ref, default_project=self.project
+ )
+
+ api_response = self._call_api(retry, method="GET", path=routine_ref.path)
+ return Routine.from_api_repr(api_response)
+
+ def get_table(self, table, retry=DEFAULT_RETRY):
+ """Fetch the table referenced by ``table``.
+
+ Args:
+ table (Union[ \
+ :class:`~google.cloud.bigquery.table.Table`, \
+ :class:`~google.cloud.bigquery.table.TableReference`, \
+ str, \
+ ]):
+ A reference to the table to fetch from the BigQuery API.
+ If a string is passed in, this method attempts to create a
+ table reference from a string using
+ :func:`google.cloud.bigquery.table.TableReference.from_string`.
+ retry (:class:`google.api_core.retry.Retry`):
+ (Optional) How to retry the RPC.
+
+ Returns:
+ google.cloud.bigquery.table.Table:
+ A ``Table`` instance.
+ """
+ table_ref = _table_arg_to_table_ref(table, default_project=self.project)
+ api_response = self._call_api(retry, method="GET", path=table_ref.path)
+ return Table.from_api_repr(api_response)
+
+ def update_dataset(self, dataset, fields, retry=DEFAULT_RETRY):
+ """Change some fields of a dataset.
+
+ Use ``fields`` to specify which fields to update. At least one field
+ must be provided. If a field is listed in ``fields`` and is ``None`` in
+ ``dataset``, it will be deleted.
+
+ If ``dataset.etag`` is not ``None``, the update will only
+ succeed if the dataset on the server has the same ETag. Thus
+ reading a dataset with ``get_dataset``, changing its fields,
+ and then passing it to ``update_dataset`` will ensure that the changes
+ will only be saved if no modifications to the dataset occurred
+ since the read.
+
+ Args:
+ dataset (google.cloud.bigquery.dataset.Dataset):
+ The dataset to update.
+ fields (Sequence[str]):
+ The properties of ``dataset`` to change (e.g. "friendly_name").
+ retry (google.api_core.retry.Retry, optional):
+ How to retry the RPC.
+
+ Returns:
+ google.cloud.bigquery.dataset.Dataset:
+ The modified ``Dataset`` instance.
+ """
+ partial = dataset._build_resource(fields)
+ if dataset.etag is not None:
+ headers = {"If-Match": dataset.etag}
+ else:
+ headers = None
+ api_response = self._call_api(
+ retry, method="PATCH", path=dataset.path, data=partial, headers=headers
+ )
+ return Dataset.from_api_repr(api_response)
+
+ def update_model(self, model, fields, retry=DEFAULT_RETRY):
+ """[Beta] Change some fields of a model.
+
+ Use ``fields`` to specify which fields to update. At least one field
+ must be provided. If a field is listed in ``fields`` and is ``None``
+ in ``model``, the field value will be deleted.
+
+ If ``model.etag`` is not ``None``, the update will only succeed if
+ the model on the server has the same ETag. Thus reading a model with
+ ``get_model``, changing its fields, and then passing it to
+ ``update_model`` will ensure that the changes will only be saved if
+ no modifications to the model occurred since the read.
+
+ Args:
+ model (google.cloud.bigquery.model.Model): The model to update.
+ fields (Sequence[str]):
+ The fields of ``model`` to change, spelled as the Model
+ properties (e.g. "friendly_name").
+ retry (google.api_core.retry.Retry):
+ (Optional) A description of how to retry the API call.
+
+ Returns:
+ google.cloud.bigquery.model.Model:
+ The model resource returned from the API call.
+ """
+ partial = model._build_resource(fields)
+ if model.etag:
+ headers = {"If-Match": model.etag}
+ else:
+ headers = None
+ api_response = self._call_api(
+ retry, method="PATCH", path=model.path, data=partial, headers=headers
+ )
+ return Model.from_api_repr(api_response)
+
+ def update_routine(self, routine, fields, retry=DEFAULT_RETRY):
+ """[Beta] Change some fields of a routine.
+
+ Use ``fields`` to specify which fields to update. At least one field
+ must be provided. If a field is listed in ``fields`` and is ``None``
+ in ``routine``, the field value will be deleted.
+
+ .. warning::
+ During beta, partial updates are not supported. You must provide
+ all fields in the resource.
+
+ If :attr:`~google.cloud.bigquery.routine.Routine.etag` is not
+ ``None``, the update will only succeed if the resource on the server
+ has the same ETag. Thus reading a routine with
+ :func:`~google.cloud.bigquery.client.Client.get_routine`, changing
+ its fields, and then passing it to this method will ensure that the
+ changes will only be saved if no modifications to the resource
+ occurred since the read.
+
+ Args:
+ routine (google.cloud.bigquery.routine.Routine): The routine to update.
+ fields (Sequence[str]):
+ The fields of ``routine`` to change, spelled as the
+ :class:`~google.cloud.bigquery.routine.Routine` properties
+ (e.g. ``type_``).
+ retry (google.api_core.retry.Retry):
+ (Optional) A description of how to retry the API call.
+
+ Returns:
+ google.cloud.bigquery.routine.Routine:
+ The routine resource returned from the API call.
+ """
+ partial = routine._build_resource(fields)
+ if routine.etag:
+ headers = {"If-Match": routine.etag}
+ else:
+ headers = None
+
+ # TODO: remove when routines update supports partial requests.
+ partial["routineReference"] = routine.reference.to_api_repr()
+
+ api_response = self._call_api(
+ retry, method="PUT", path=routine.path, data=partial, headers=headers
+ )
+ return Routine.from_api_repr(api_response)
+
+ def update_table(self, table, fields, retry=DEFAULT_RETRY):
+ """Change some fields of a table.
+
+ Use ``fields`` to specify which fields to update. At least one field
+ must be provided. If a field is listed in ``fields`` and is ``None``
+ in ``table``, the field value will be deleted.
+
+ If ``table.etag`` is not ``None``, the update will only succeed if
+ the table on the server has the same ETag. Thus reading a table with
+ ``get_table``, changing its fields, and then passing it to
+ ``update_table`` will ensure that the changes will only be saved if
+ no modifications to the table occurred since the read.
+
+ Args:
+ table (google.cloud.bigquery.table.Table): The table to update.
+ fields (Sequence[str]):
+ The fields of ``table`` to change, spelled as the Table
+ properties (e.g. "friendly_name").
+ retry (google.api_core.retry.Retry):
+ (Optional) A description of how to retry the API call.
+
+ Returns:
+ google.cloud.bigquery.table.Table:
+ The table resource returned from the API call.
+ """
+ partial = table._build_resource(fields)
+ if table.etag is not None:
+ headers = {"If-Match": table.etag}
+ else:
+ headers = None
+ api_response = self._call_api(
+ retry, method="PATCH", path=table.path, data=partial, headers=headers
+ )
+ return Table.from_api_repr(api_response)
+
+ def list_models(
+ self, dataset, max_results=None, page_token=None, retry=DEFAULT_RETRY
+ ):
+ """[Beta] List models in the dataset.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/models/list
+
+ Args:
+ dataset (Union[ \
+ :class:`~google.cloud.bigquery.dataset.Dataset`, \
+ :class:`~google.cloud.bigquery.dataset.DatasetReference`, \
+ str, \
+ ]):
+ A reference to the dataset whose models to list from the
+ BigQuery API. If a string is passed in, this method attempts
+ to create a dataset reference from a string using
+ :func:`google.cloud.bigquery.dataset.DatasetReference.from_string`.
+ max_results (int):
+ (Optional) Maximum number of models to return. If not passed,
+ defaults to a value set by the API.
+ page_token (str):
+ (Optional) Token representing a cursor into the models. If
+ not passed, the API will return the first page of models. The
+ token marks the beginning of the iterator to be returned and
+ the value of the ``page_token`` can be accessed at
+ ``next_page_token`` of the
+ :class:`~google.api_core.page_iterator.HTTPIterator`.
+ retry (:class:`google.api_core.retry.Retry`):
+ (Optional) How to retry the RPC.
+
+ Returns:
+ google.api_core.page_iterator.Iterator:
+ Iterator of
+ :class:`~google.cloud.bigquery.model.Model` contained
+ within the requested dataset.
+ """
+ if isinstance(dataset, str):
+ dataset = DatasetReference.from_string(
+ dataset, default_project=self.project
+ )
+
+ if not isinstance(dataset, (Dataset, DatasetReference)):
+ raise TypeError("dataset must be a Dataset, DatasetReference, or string")
+
+ path = "%s/models" % dataset.path
+ result = page_iterator.HTTPIterator(
+ client=self,
+ api_request=functools.partial(self._call_api, retry),
+ path=path,
+ item_to_value=_item_to_model,
+ items_key="models",
+ page_token=page_token,
+ max_results=max_results,
+ )
+ result.dataset = dataset
+ return result
+
+ def list_routines(
+ self, dataset, max_results=None, page_token=None, retry=DEFAULT_RETRY
+ ):
+ """[Beta] List routines in the dataset.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/routines/list
+
+ Args:
+ dataset (Union[ \
+ :class:`~google.cloud.bigquery.dataset.Dataset`, \
+ :class:`~google.cloud.bigquery.dataset.DatasetReference`, \
+ str, \
+ ]):
+ A reference to the dataset whose routines to list from the
+ BigQuery API. If a string is passed in, this method attempts
+ to create a dataset reference from a string using
+ :func:`google.cloud.bigquery.dataset.DatasetReference.from_string`.
+ max_results (int):
+ (Optional) Maximum number of routines to return. If not passed,
+ defaults to a value set by the API.
+ page_token (str):
+ (Optional) Token representing a cursor into the routines. If
+ not passed, the API will return the first page of routines. The
+ token marks the beginning of the iterator to be returned and
+ the value of the ``page_token`` can be accessed at
+ ``next_page_token`` of the
+ :class:`~google.api_core.page_iterator.HTTPIterator`.
+ retry (:class:`google.api_core.retry.Retry`):
+ (Optional) How to retry the RPC.
+
+ Returns:
+ google.api_core.page_iterator.Iterator:
+ Iterator of all
+ :class:`~google.cloud.bigquery.routine.Routine`s contained
+ within the requested dataset, limited by ``max_results``.
+ """
+ if isinstance(dataset, str):
+ dataset = DatasetReference.from_string(
+ dataset, default_project=self.project
+ )
+
+ if not isinstance(dataset, (Dataset, DatasetReference)):
+ raise TypeError("dataset must be a Dataset, DatasetReference, or string")
+
+ path = "{}/routines".format(dataset.path)
+ result = page_iterator.HTTPIterator(
+ client=self,
+ api_request=functools.partial(self._call_api, retry),
+ path=path,
+ item_to_value=_item_to_routine,
+ items_key="routines",
+ page_token=page_token,
+ max_results=max_results,
+ )
+ result.dataset = dataset
+ return result
+
+ def list_tables(
+ self, dataset, max_results=None, page_token=None, retry=DEFAULT_RETRY
+ ):
+ """List tables in the dataset.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables/list
+
+ Args:
+ dataset (Union[ \
+ :class:`~google.cloud.bigquery.dataset.Dataset`, \
+ :class:`~google.cloud.bigquery.dataset.DatasetReference`, \
+ str, \
+ ]):
+ A reference to the dataset whose tables to list from the
+ BigQuery API. If a string is passed in, this method attempts
+ to create a dataset reference from a string using
+ :func:`google.cloud.bigquery.dataset.DatasetReference.from_string`.
+ max_results (int):
+ (Optional) Maximum number of tables to return. If not passed,
+ defaults to a value set by the API.
+ page_token (str):
+ (Optional) Token representing a cursor into the tables. If
+ not passed, the API will return the first page of tables. The
+ token marks the beginning of the iterator to be returned and
+ the value of the ``page_token`` can be accessed at
+ ``next_page_token`` of the
+ :class:`~google.api_core.page_iterator.HTTPIterator`.
+ retry (:class:`google.api_core.retry.Retry`):
+ (Optional) How to retry the RPC.
+
+ Returns:
+ google.api_core.page_iterator.Iterator:
+ Iterator of
+ :class:`~google.cloud.bigquery.table.TableListItem` contained
+ within the requested dataset.
+ """
+ if isinstance(dataset, str):
+ dataset = DatasetReference.from_string(
+ dataset, default_project=self.project
+ )
+
+ if not isinstance(dataset, (Dataset, DatasetReference)):
+ raise TypeError("dataset must be a Dataset, DatasetReference, or string")
+
+ path = "%s/tables" % dataset.path
+ result = page_iterator.HTTPIterator(
+ client=self,
+ api_request=functools.partial(self._call_api, retry),
+ path=path,
+ item_to_value=_item_to_table,
+ items_key="tables",
+ page_token=page_token,
+ max_results=max_results,
+ )
+ result.dataset = dataset
+ return result
+
+ def delete_dataset(
+ self, dataset, delete_contents=False, retry=DEFAULT_RETRY, not_found_ok=False
+ ):
+ """Delete a dataset.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/delete
+
+ Args
+ dataset (Union[ \
+ :class:`~google.cloud.bigquery.dataset.Dataset`, \
+ :class:`~google.cloud.bigquery.dataset.DatasetReference`, \
+ str, \
+ ]):
+ A reference to the dataset to delete. If a string is passed
+ in, this method attempts to create a dataset reference from a
+ string using
+ :func:`google.cloud.bigquery.dataset.DatasetReference.from_string`.
+ delete_contents (boolean):
+ (Optional) If True, delete all the tables in the dataset. If
+ False and the dataset contains tables, the request will fail.
+ Default is False.
+ retry (:class:`google.api_core.retry.Retry`):
+ (Optional) How to retry the RPC.
+ not_found_ok (bool):
+ Defaults to ``False``. If ``True``, ignore "not found" errors
+ when deleting the dataset.
+ """
+ if isinstance(dataset, str):
+ dataset = DatasetReference.from_string(
+ dataset, default_project=self.project
+ )
+
+ if not isinstance(dataset, (Dataset, DatasetReference)):
+ raise TypeError("dataset must be a Dataset or a DatasetReference")
+
+ params = {}
+ if delete_contents:
+ params["deleteContents"] = "true"
+
+ try:
+ self._call_api(
+ retry, method="DELETE", path=dataset.path, query_params=params
+ )
+ except google.api_core.exceptions.NotFound:
+ if not not_found_ok:
+ raise
+
+ def delete_model(self, model, retry=DEFAULT_RETRY, not_found_ok=False):
+ """[Beta] Delete a model
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/models/delete
+
+ Args:
+ model (Union[ \
+ :class:`~google.cloud.bigquery.model.Model`, \
+ :class:`~google.cloud.bigquery.model.ModelReference`, \
+ str, \
+ ]):
+ A reference to the model to delete. If a string is passed in,
+ this method attempts to create a model reference from a
+ string using
+ :func:`google.cloud.bigquery.model.ModelReference.from_string`.
+ retry (:class:`google.api_core.retry.Retry`):
+ (Optional) How to retry the RPC.
+ not_found_ok (bool):
+ Defaults to ``False``. If ``True``, ignore "not found" errors
+ when deleting the model.
+ """
+ if isinstance(model, str):
+ model = ModelReference.from_string(model, default_project=self.project)
+
+ if not isinstance(model, (Model, ModelReference)):
+ raise TypeError("model must be a Model or a ModelReference")
+
+ try:
+ self._call_api(retry, method="DELETE", path=model.path)
+ except google.api_core.exceptions.NotFound:
+ if not not_found_ok:
+ raise
+
+ def delete_routine(self, routine, retry=DEFAULT_RETRY, not_found_ok=False):
+ """[Beta] Delete a routine.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/routines/delete
+
+ Args:
+ model (Union[ \
+ :class:`~google.cloud.bigquery.routine.Routine`, \
+ :class:`~google.cloud.bigquery.routine.RoutineReference`, \
+ str, \
+ ]):
+ A reference to the routine to delete. If a string is passed
+ in, this method attempts to create a routine reference from a
+ string using
+ :func:`google.cloud.bigquery.routine.RoutineReference.from_string`.
+ retry (:class:`google.api_core.retry.Retry`):
+ (Optional) How to retry the RPC.
+ not_found_ok (bool):
+ Defaults to ``False``. If ``True``, ignore "not found" errors
+ when deleting the routine.
+ """
+ if isinstance(routine, str):
+ routine = RoutineReference.from_string(
+ routine, default_project=self.project
+ )
+
+ if not isinstance(routine, (Routine, RoutineReference)):
+ raise TypeError("routine must be a Routine or a RoutineReference")
+
+ try:
+ self._call_api(retry, method="DELETE", path=routine.path)
+ except google.api_core.exceptions.NotFound:
+ if not not_found_ok:
+ raise
+
+ def delete_table(self, table, retry=DEFAULT_RETRY, not_found_ok=False):
+ """Delete a table
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables/delete
+
+ Args:
+ table (Union[ \
+ :class:`~google.cloud.bigquery.table.Table`, \
+ :class:`~google.cloud.bigquery.table.TableReference`, \
+ str, \
+ ]):
+ A reference to the table to delete. If a string is passed in,
+ this method attempts to create a table reference from a
+ string using
+ :func:`google.cloud.bigquery.table.TableReference.from_string`.
+ retry (:class:`google.api_core.retry.Retry`):
+ (Optional) How to retry the RPC.
+ not_found_ok (bool):
+ Defaults to ``False``. If ``True``, ignore "not found" errors
+ when deleting the table.
+ """
+ table = _table_arg_to_table_ref(table, default_project=self.project)
+ if not isinstance(table, TableReference):
+ raise TypeError("Unable to get TableReference for table '{}'".format(table))
+
+ try:
+ self._call_api(retry, method="DELETE", path=table.path)
+ except google.api_core.exceptions.NotFound:
+ if not not_found_ok:
+ raise
+
+ def _get_query_results(
+ self, job_id, retry, project=None, timeout_ms=None, location=None
+ ):
+ """Get the query results object for a query job.
+
+ Arguments:
+ job_id (str): Name of the query job.
+ retry (google.api_core.retry.Retry):
+ (Optional) How to retry the RPC.
+ project (str):
+ (Optional) project ID for the query job (defaults to the
+ project of the client).
+ timeout_ms (int):
+ (Optional) number of milliseconds the the API call should
+ wait for the query to complete before the request times out.
+ location (str): Location of the query job.
+
+ Returns:
+ google.cloud.bigquery.query._QueryResults:
+ A new ``_QueryResults`` instance.
+ """
+
+ extra_params = {"maxResults": 0}
+
+ if project is None:
+ project = self.project
+
+ if timeout_ms is not None:
+ extra_params["timeoutMs"] = timeout_ms
+
+ if location is None:
+ location = self.location
+
+ if location is not None:
+ extra_params["location"] = location
+
+ path = "/projects/{}/queries/{}".format(project, job_id)
+
+ # This call is typically made in a polling loop that checks whether the
+ # job is complete (from QueryJob.done(), called ultimately from
+ # QueryJob.result()). So we don't need to poll here.
+ resource = self._call_api(
+ retry, method="GET", path=path, query_params=extra_params
+ )
+ return _QueryResults.from_api_repr(resource)
+
+ def job_from_resource(self, resource):
+ """Detect correct job type from resource and instantiate.
+
+ :type resource: dict
+ :param resource: one job resource from API response
+
+ :rtype: One of:
+ :class:`google.cloud.bigquery.job.LoadJob`,
+ :class:`google.cloud.bigquery.job.CopyJob`,
+ :class:`google.cloud.bigquery.job.ExtractJob`,
+ or :class:`google.cloud.bigquery.job.QueryJob`
+ :returns: the job instance, constructed via the resource
+ """
+ config = resource.get("configuration", {})
+ if "load" in config:
+ return job.LoadJob.from_api_repr(resource, self)
+ elif "copy" in config:
+ return job.CopyJob.from_api_repr(resource, self)
+ elif "extract" in config:
+ return job.ExtractJob.from_api_repr(resource, self)
+ elif "query" in config:
+ return job.QueryJob.from_api_repr(resource, self)
+ return job.UnknownJob.from_api_repr(resource, self)
+
+ def get_job(self, job_id, project=None, location=None, retry=DEFAULT_RETRY):
+ """Fetch a job for the project associated with this client.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/get
+
+ Arguments:
+ job_id (str): Unique job identifier.
+
+ Keyword Arguments:
+ project (str):
+ (Optional) ID of the project which ownsthe job (defaults to
+ the client's project).
+ location (str): Location where the job was run.
+ retry (google.api_core.retry.Retry):
+ (Optional) How to retry the RPC.
+
+ Returns:
+ Union[google.cloud.bigquery.job.LoadJob, \
+ google.cloud.bigquery.job.CopyJob, \
+ google.cloud.bigquery.job.ExtractJob, \
+ google.cloud.bigquery.job.QueryJob]:
+ Job instance, based on the resource returned by the API.
+ """
+ extra_params = {"projection": "full"}
+
+ if project is None:
+ project = self.project
+
+ if location is None:
+ location = self.location
+
+ if location is not None:
+ extra_params["location"] = location
+
+ path = "/projects/{}/jobs/{}".format(project, job_id)
+
+ resource = self._call_api(
+ retry, method="GET", path=path, query_params=extra_params
+ )
+
+ return self.job_from_resource(resource)
+
+ def cancel_job(self, job_id, project=None, location=None, retry=DEFAULT_RETRY):
+ """Attempt to cancel a job from a job ID.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/cancel
+
+ Arguments:
+ job_id (str): Unique job identifier.
+
+ Keyword Arguments:
+ project (str):
+ (Optional) ID of the project which owns the job (defaults to
+ the client's project).
+ location (str): Location where the job was run.
+ retry (google.api_core.retry.Retry):
+ (Optional) How to retry the RPC.
+
+ Returns:
+ Union[google.cloud.bigquery.job.LoadJob, \
+ google.cloud.bigquery.job.CopyJob, \
+ google.cloud.bigquery.job.ExtractJob, \
+ google.cloud.bigquery.job.QueryJob]:
+ Job instance, based on the resource returned by the API.
+ """
+ extra_params = {"projection": "full"}
+
+ if project is None:
+ project = self.project
+
+ if location is None:
+ location = self.location
+
+ if location is not None:
+ extra_params["location"] = location
+
+ path = "/projects/{}/jobs/{}/cancel".format(project, job_id)
+
+ resource = self._call_api(
+ retry, method="POST", path=path, query_params=extra_params
+ )
+
+ return self.job_from_resource(resource["job"])
+
+ def list_jobs(
+ self,
+ project=None,
+ max_results=None,
+ page_token=None,
+ all_users=None,
+ state_filter=None,
+ retry=DEFAULT_RETRY,
+ min_creation_time=None,
+ max_creation_time=None,
+ ):
+ """List jobs for the project associated with this client.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/list
+
+ Args:
+ project (str, optional):
+ Project ID to use for retreiving datasets. Defaults
+ to the client's project.
+ max_results (int, optional):
+ Maximum number of jobs to return.
+ page_token (str, optional):
+ Opaque marker for the next "page" of jobs. If not
+ passed, the API will return the first page of jobs. The token
+ marks the beginning of the iterator to be returned and the
+ value of the ``page_token`` can be accessed at
+ ``next_page_token`` of
+ :class:`~google.api_core.page_iterator.HTTPIterator`.
+ all_users (bool, optional):
+ If true, include jobs owned by all users in the project.
+ Defaults to :data:`False`.
+ state_filter (str, optional):
+ If set, include only jobs matching the given state. One of:
+ * ``"done"``
+ * ``"pending"``
+ * ``"running"``
+ retry (google.api_core.retry.Retry, optional):
+ How to retry the RPC.
+ min_creation_time (datetime.datetime, optional):
+ Min value for job creation time. If set, only jobs created
+ after or at this timestamp are returned. If the datetime has
+ no time zone assumes UTC time.
+ max_creation_time (datetime.datetime, optional):
+ Max value for job creation time. If set, only jobs created
+ before or at this timestamp are returned. If the datetime has
+ no time zone assumes UTC time.
+
+ Returns:
+ google.api_core.page_iterator.Iterator:
+ Iterable of job instances.
+ """
+ extra_params = {
+ "allUsers": all_users,
+ "stateFilter": state_filter,
+ "minCreationTime": _str_or_none(
+ google.cloud._helpers._millis_from_datetime(min_creation_time)
+ ),
+ "maxCreationTime": _str_or_none(
+ google.cloud._helpers._millis_from_datetime(max_creation_time)
+ ),
+ "projection": "full",
+ }
+
+ extra_params = {
+ param: value for param, value in extra_params.items() if value is not None
+ }
+
+ if project is None:
+ project = self.project
+
+ path = "/projects/%s/jobs" % (project,)
+ return page_iterator.HTTPIterator(
+ client=self,
+ api_request=functools.partial(self._call_api, retry),
+ path=path,
+ item_to_value=_item_to_job,
+ items_key="jobs",
+ page_token=page_token,
+ max_results=max_results,
+ extra_params=extra_params,
+ )
+
+ def load_table_from_uri(
+ self,
+ source_uris,
+ destination,
+ job_id=None,
+ job_id_prefix=None,
+ location=None,
+ project=None,
+ job_config=None,
+ retry=DEFAULT_RETRY,
+ ):
+ """Starts a job for loading data into a table from CloudStorage.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load
+
+ Arguments:
+ source_uris (Union[str, Sequence[str]]):
+ URIs of data files to be loaded; in format
+ ``gs:///``.
+ destination (Union[ \
+ :class:`~google.cloud.bigquery.table.Table`, \
+ :class:`~google.cloud.bigquery.table.TableReference`, \
+ str, \
+ ]):
+ Table into which data is to be loaded. If a string is passed
+ in, this method attempts to create a table reference from a
+ string using
+ :func:`google.cloud.bigquery.table.TableReference.from_string`.
+
+ Keyword Arguments:
+ job_id (str): (Optional) Name of the job.
+ job_id_prefix (str):
+ (Optional) the user-provided prefix for a randomly generated
+ job ID. This parameter will be ignored if a ``job_id`` is
+ also given.
+ location (str):
+ Location where to run the job. Must match the location of the
+ destination table.
+ project (str):
+ Project ID of the project of where to run the job. Defaults
+ to the client's project.
+ job_config (google.cloud.bigquery.job.LoadJobConfig):
+ (Optional) Extra configuration options for the job.
+ retry (google.api_core.retry.Retry):
+ (Optional) How to retry the RPC.
+
+ Returns:
+ google.cloud.bigquery.job.LoadJob: A new load job.
+ """
+ job_id = _make_job_id(job_id, job_id_prefix)
+
+ if project is None:
+ project = self.project
+
+ if location is None:
+ location = self.location
+
+ job_ref = job._JobReference(job_id, project=project, location=location)
+
+ if isinstance(source_uris, six.string_types):
+ source_uris = [source_uris]
+
+ destination = _table_arg_to_table_ref(destination, default_project=self.project)
+ load_job = job.LoadJob(job_ref, source_uris, destination, self, job_config)
+ load_job._begin(retry=retry)
+
+ return load_job
+
+ def load_table_from_file(
+ self,
+ file_obj,
+ destination,
+ rewind=False,
+ size=None,
+ num_retries=_DEFAULT_NUM_RETRIES,
+ job_id=None,
+ job_id_prefix=None,
+ location=None,
+ project=None,
+ job_config=None,
+ ):
+ """Upload the contents of this table from a file-like object.
+
+ Similar to :meth:`load_table_from_uri`, this method creates, starts and
+ returns a :class:`~google.cloud.bigquery.job.LoadJob`.
+
+ Arguments:
+ file_obj (file): A file handle opened in binary mode for reading.
+ destination (Union[ \
+ :class:`~google.cloud.bigquery.table.Table`, \
+ :class:`~google.cloud.bigquery.table.TableReference`, \
+ str, \
+ ]):
+ Table into which data is to be loaded. If a string is passed
+ in, this method attempts to create a table reference from a
+ string using
+ :func:`google.cloud.bigquery.table.TableReference.from_string`.
+
+ Keyword Arguments:
+ rewind (bool):
+ If True, seek to the beginning of the file handle before
+ reading the file.
+ size (int):
+ The number of bytes to read from the file handle. If size is
+ ``None`` or large, resumable upload will be used. Otherwise,
+ multipart upload will be used.
+ num_retries (int): Number of upload retries. Defaults to 6.
+ job_id (str): (Optional) Name of the job.
+ job_id_prefix (str):
+ (Optional) the user-provided prefix for a randomly generated
+ job ID. This parameter will be ignored if a ``job_id`` is
+ also given.
+ location (str):
+ Location where to run the job. Must match the location of the
+ destination table.
+ project (str):
+ Project ID of the project of where to run the job. Defaults
+ to the client's project.
+ job_config (google.cloud.bigquery.job.LoadJobConfig):
+ (Optional) Extra configuration options for the job.
+
+ Returns:
+ google.cloud.bigquery.job.LoadJob: A new load job.
+
+ Raises:
+ ValueError:
+ If ``size`` is not passed in and can not be determined, or if
+ the ``file_obj`` can be detected to be a file opened in text
+ mode.
+ """
+ job_id = _make_job_id(job_id, job_id_prefix)
+
+ if project is None:
+ project = self.project
+
+ if location is None:
+ location = self.location
+
+ destination = _table_arg_to_table_ref(destination, default_project=self.project)
+ job_ref = job._JobReference(job_id, project=project, location=location)
+ load_job = job.LoadJob(job_ref, None, destination, self, job_config)
+ job_resource = load_job.to_api_repr()
+
+ if rewind:
+ file_obj.seek(0, os.SEEK_SET)
+
+ _check_mode(file_obj)
+
+ try:
+ if size is None or size >= _MAX_MULTIPART_SIZE:
+ response = self._do_resumable_upload(
+ file_obj, job_resource, num_retries
+ )
+ else:
+ response = self._do_multipart_upload(
+ file_obj, job_resource, size, num_retries
+ )
+ except resumable_media.InvalidResponse as exc:
+ raise exceptions.from_http_response(exc.response)
+
+ return self.job_from_resource(response.json())
+
+ def load_table_from_dataframe(
+ self,
+ dataframe,
+ destination,
+ num_retries=_DEFAULT_NUM_RETRIES,
+ job_id=None,
+ job_id_prefix=None,
+ location=None,
+ project=None,
+ job_config=None,
+ parquet_compression="snappy",
+ ):
+ """Upload the contents of a table from a pandas DataFrame.
+
+ Similar to :meth:`load_table_from_uri`, this method creates, starts and
+ returns a :class:`~google.cloud.bigquery.job.LoadJob`.
+
+ Arguments:
+ dataframe (pandas.DataFrame):
+ A :class:`~pandas.DataFrame` containing the data to load.
+ destination (google.cloud.bigquery.table.TableReference):
+ The destination table to use for loading the data. If it is an
+ existing table, the schema of the :class:`~pandas.DataFrame`
+ must match the schema of the destination table. If the table
+ does not yet exist, the schema is inferred from the
+ :class:`~pandas.DataFrame`.
+
+ If a string is passed in, this method attempts to create a
+ table reference from a string using
+ :func:`google.cloud.bigquery.table.TableReference.from_string`.
+
+ Keyword Arguments:
+ num_retries (int, optional): Number of upload retries.
+ job_id (str, optional): Name of the job.
+ job_id_prefix (str, optional):
+ The user-provided prefix for a randomly generated
+ job ID. This parameter will be ignored if a ``job_id`` is
+ also given.
+ location (str):
+ Location where to run the job. Must match the location of the
+ destination table.
+ project (str, optional):
+ Project ID of the project of where to run the job. Defaults
+ to the client's project.
+ job_config (~google.cloud.bigquery.job.LoadJobConfig, optional):
+ Extra configuration options for the job.
+
+ To override the default pandas data type conversions, supply
+ a value for
+ :attr:`~google.cloud.bigquery.job.LoadJobConfig.schema` with
+ column names matching those of the dataframe. The BigQuery
+ schema is used to determine the correct data type conversion.
+ Indexes are not loaded. Requires the :mod:`pyarrow` library.
+ parquet_compression (str):
+ [Beta] The compression method to use if intermittently
+ serializing ``dataframe`` to a parquet file.
+
+ If ``pyarrow`` and job config schema are used, the argument
+ is directly passed as the ``compression`` argument to the
+ underlying ``pyarrow.parquet.write_table()`` method (the
+ default value "snappy" gets converted to uppercase).
+ https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html#pyarrow-parquet-write-table
+
+ If either ``pyarrow`` or job config schema are missing, the
+ argument is directly passed as the ``compression`` argument
+ to the underlying ``DataFrame.to_parquet()`` method.
+ https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_parquet.html#pandas.DataFrame.to_parquet
+
+ Returns:
+ google.cloud.bigquery.job.LoadJob: A new load job.
+
+ Raises:
+ ImportError:
+ If a usable parquet engine cannot be found. This method
+ requires :mod:`pyarrow` or :mod:`fastparquet` to be
+ installed.
+ """
+ job_id = _make_job_id(job_id, job_id_prefix)
+
+ if job_config is None:
+ job_config = job.LoadJobConfig()
+ else:
+ # Make a copy so that the job config isn't modified in-place.
+ job_config_properties = copy.deepcopy(job_config._properties)
+ job_config = job.LoadJobConfig()
+ job_config._properties = job_config_properties
+ job_config.source_format = job.SourceFormat.PARQUET
+
+ if location is None:
+ location = self.location
+
+ job_config.schema = _pandas_helpers.dataframe_to_bq_schema(
+ dataframe, job_config.schema
+ )
+
+ tmpfd, tmppath = tempfile.mkstemp(suffix="_job_{}.parquet".format(job_id[:8]))
+ os.close(tmpfd)
+
+ try:
+ if pyarrow and job_config.schema:
+ if parquet_compression == "snappy": # adjust the default value
+ parquet_compression = parquet_compression.upper()
+
+ _pandas_helpers.dataframe_to_parquet(
+ dataframe,
+ job_config.schema,
+ tmppath,
+ parquet_compression=parquet_compression,
+ )
+ else:
+ if job_config.schema:
+ warnings.warn(
+ "job_config.schema is set, but not used to assist in "
+ "identifying correct types for data serialization. "
+ "Please install the pyarrow package.",
+ PendingDeprecationWarning,
+ stacklevel=2,
+ )
+
+ dataframe.to_parquet(tmppath, compression=parquet_compression)
+
+ with open(tmppath, "rb") as parquet_file:
+ return self.load_table_from_file(
+ parquet_file,
+ destination,
+ num_retries=num_retries,
+ rewind=True,
+ job_id=job_id,
+ job_id_prefix=job_id_prefix,
+ location=location,
+ project=project,
+ job_config=job_config,
+ )
+
+ finally:
+ os.remove(tmppath)
+
+ def load_table_from_json(
+ self,
+ json_rows,
+ destination,
+ num_retries=_DEFAULT_NUM_RETRIES,
+ job_id=None,
+ job_id_prefix=None,
+ location=None,
+ project=None,
+ job_config=None,
+ ):
+ """Upload the contents of a table from a JSON string or dict.
+
+ Arguments:
+ json_rows (Iterable[Dict[str, Any]]):
+ Row data to be inserted. Keys must match the table schema fields
+ and values must be JSON-compatible representations.
+ destination (Union[ \
+ :class:`~google.cloud.bigquery.table.Table`, \
+ :class:`~google.cloud.bigquery.table.TableReference`, \
+ str, \
+ ]):
+ Table into which data is to be loaded. If a string is passed
+ in, this method attempts to create a table reference from a
+ string using
+ :func:`google.cloud.bigquery.table.TableReference.from_string`.
+
+ Keyword Arguments:
+ num_retries (int, optional): Number of upload retries.
+ job_id (str): (Optional) Name of the job.
+ job_id_prefix (str):
+ (Optional) the user-provided prefix for a randomly generated
+ job ID. This parameter will be ignored if a ``job_id`` is
+ also given.
+ location (str):
+ Location where to run the job. Must match the location of the
+ destination table.
+ project (str):
+ Project ID of the project of where to run the job. Defaults
+ to the client's project.
+ job_config (google.cloud.bigquery.job.LoadJobConfig):
+ (Optional) Extra configuration options for the job. The
+ ``source_format`` setting is always set to
+ :attr:`~google.cloud.bigquery.job.SourceFormat.NEWLINE_DELIMITED_JSON`.
+
+ Returns:
+ google.cloud.bigquery.job.LoadJob: A new load job.
+ """
+ job_id = _make_job_id(job_id, job_id_prefix)
+
+ if job_config is None:
+ job_config = job.LoadJobConfig()
+ else:
+ # Make a copy so that the job config isn't modified in-place.
+ job_config = copy.deepcopy(job_config)
+ job_config.source_format = job.SourceFormat.NEWLINE_DELIMITED_JSON
+
+ if job_config.schema is None:
+ job_config.autodetect = True
+
+ if project is None:
+ project = self.project
+
+ if location is None:
+ location = self.location
+
+ destination = _table_arg_to_table_ref(destination, default_project=self.project)
+
+ data_str = u"\n".join(json.dumps(item) for item in json_rows)
+ data_file = io.BytesIO(data_str.encode())
+
+ return self.load_table_from_file(
+ data_file,
+ destination,
+ num_retries=num_retries,
+ job_id=job_id,
+ job_id_prefix=job_id_prefix,
+ location=location,
+ project=project,
+ job_config=job_config,
+ )
+
+ def _do_resumable_upload(self, stream, metadata, num_retries):
+ """Perform a resumable upload.
+
+ :type stream: IO[bytes]
+ :param stream: A bytes IO object open for reading.
+
+ :type metadata: dict
+ :param metadata: The metadata associated with the upload.
+
+ :type num_retries: int
+ :param num_retries: Number of upload retries. (Deprecated: This
+ argument will be removed in a future release.)
+
+ :rtype: :class:`~requests.Response`
+ :returns: The "200 OK" response object returned after the final chunk
+ is uploaded.
+ """
+ upload, transport = self._initiate_resumable_upload(
+ stream, metadata, num_retries
+ )
+
+ while not upload.finished:
+ response = upload.transmit_next_chunk(transport)
+
+ return response
+
+ def _initiate_resumable_upload(self, stream, metadata, num_retries):
+ """Initiate a resumable upload.
+
+ :type stream: IO[bytes]
+ :param stream: A bytes IO object open for reading.
+
+ :type metadata: dict
+ :param metadata: The metadata associated with the upload.
+
+ :type num_retries: int
+ :param num_retries: Number of upload retries. (Deprecated: This
+ argument will be removed in a future release.)
+
+ :rtype: tuple
+ :returns:
+ Pair of
+
+ * The :class:`~google.resumable_media.requests.ResumableUpload`
+ that was created
+ * The ``transport`` used to initiate the upload.
+ """
+ chunk_size = _DEFAULT_CHUNKSIZE
+ transport = self._http
+ headers = _get_upload_headers(self._connection.user_agent)
+ upload_url = _RESUMABLE_URL_TEMPLATE.format(project=self.project)
+ # TODO: modify ResumableUpload to take a retry.Retry object
+ # that it can use for the initial RPC.
+ upload = ResumableUpload(upload_url, chunk_size, headers=headers)
+
+ if num_retries is not None:
+ upload._retry_strategy = resumable_media.RetryStrategy(
+ max_retries=num_retries
+ )
+
+ upload.initiate(
+ transport, stream, metadata, _GENERIC_CONTENT_TYPE, stream_final=False
+ )
+
+ return upload, transport
+
+ def _do_multipart_upload(self, stream, metadata, size, num_retries):
+ """Perform a multipart upload.
+
+ :type stream: IO[bytes]
+ :param stream: A bytes IO object open for reading.
+
+ :type metadata: dict
+ :param metadata: The metadata associated with the upload.
+
+ :type size: int
+ :param size: The number of bytes to be uploaded (which will be read
+ from ``stream``). If not provided, the upload will be
+ concluded once ``stream`` is exhausted (or :data:`None`).
+
+ :type num_retries: int
+ :param num_retries: Number of upload retries. (Deprecated: This
+ argument will be removed in a future release.)
+
+ :rtype: :class:`~requests.Response`
+ :returns: The "200 OK" response object returned after the multipart
+ upload request.
+ :raises: :exc:`ValueError` if the ``stream`` has fewer than ``size``
+ bytes remaining.
+ """
+ data = stream.read(size)
+ if len(data) < size:
+ msg = _READ_LESS_THAN_SIZE.format(size, len(data))
+ raise ValueError(msg)
+
+ headers = _get_upload_headers(self._connection.user_agent)
+
+ upload_url = _MULTIPART_URL_TEMPLATE.format(project=self.project)
+ upload = MultipartUpload(upload_url, headers=headers)
+
+ if num_retries is not None:
+ upload._retry_strategy = resumable_media.RetryStrategy(
+ max_retries=num_retries
+ )
+
+ response = upload.transmit(self._http, data, metadata, _GENERIC_CONTENT_TYPE)
+
+ return response
+
+ def copy_table(
+ self,
+ sources,
+ destination,
+ job_id=None,
+ job_id_prefix=None,
+ location=None,
+ project=None,
+ job_config=None,
+ retry=DEFAULT_RETRY,
+ ):
+ """Copy one or more tables to another table.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.copy
+
+ Arguments:
+ sources (Union[ \
+ :class:`~google.cloud.bigquery.table.Table`, \
+ :class:`~google.cloud.bigquery.table.TableReference`, \
+ str, \
+ Sequence[ \
+ Union[ \
+ :class:`~google.cloud.bigquery.table.Table`, \
+ :class:`~google.cloud.bigquery.table.TableReference`, \
+ str, \
+ ] \
+ ], \
+ ]):
+ Table or tables to be copied.
+ destination (Union[
+ :class:`~google.cloud.bigquery.table.Table`, \
+ :class:`~google.cloud.bigquery.table.TableReference`, \
+ str, \
+ ]):
+ Table into which data is to be copied.
+
+ Keyword Arguments:
+ job_id (str): (Optional) The ID of the job.
+ job_id_prefix (str)
+ (Optional) the user-provided prefix for a randomly generated
+ job ID. This parameter will be ignored if a ``job_id`` is
+ also given.
+ location (str):
+ Location where to run the job. Must match the location of any
+ source table as well as the destination table.
+ project (str):
+ Project ID of the project of where to run the job. Defaults
+ to the client's project.
+ job_config (google.cloud.bigquery.job.CopyJobConfig):
+ (Optional) Extra configuration options for the job.
+ retry (google.api_core.retry.Retry):
+ (Optional) How to retry the RPC.
+
+ Returns:
+ google.cloud.bigquery.job.CopyJob: A new copy job instance.
+ """
+ job_id = _make_job_id(job_id, job_id_prefix)
+
+ if project is None:
+ project = self.project
+
+ if location is None:
+ location = self.location
+
+ job_ref = job._JobReference(job_id, project=project, location=location)
+
+ # sources can be one of many different input types. (string, Table,
+ # TableReference, or a sequence of any of those.) Convert them all to a
+ # list of TableReferences.
+ #
+ # _table_arg_to_table_ref leaves lists unmodified.
+ sources = _table_arg_to_table_ref(sources, default_project=self.project)
+
+ if not isinstance(sources, collections_abc.Sequence):
+ sources = [sources]
+
+ sources = [
+ _table_arg_to_table_ref(source, default_project=self.project)
+ for source in sources
+ ]
+
+ destination = _table_arg_to_table_ref(destination, default_project=self.project)
+
+ copy_job = job.CopyJob(
+ job_ref, sources, destination, client=self, job_config=job_config
+ )
+ copy_job._begin(retry=retry)
+
+ return copy_job
+
+ def extract_table(
+ self,
+ source,
+ destination_uris,
+ job_id=None,
+ job_id_prefix=None,
+ location=None,
+ project=None,
+ job_config=None,
+ retry=DEFAULT_RETRY,
+ ):
+ """Start a job to extract a table into Cloud Storage files.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.extract
+
+ Arguments:
+ source (Union[ \
+ :class:`google.cloud.bigquery.table.Table`, \
+ :class:`google.cloud.bigquery.table.TableReference`, \
+ src, \
+ ]):
+ Table to be extracted.
+ destination_uris (Union[str, Sequence[str]]):
+ URIs of Cloud Storage file(s) into which table data is to be
+ extracted; in format
+ ``gs:///``.
+
+ Keyword Arguments:
+ job_id (str): (Optional) The ID of the job.
+ job_id_prefix (str)
+ (Optional) the user-provided prefix for a randomly generated
+ job ID. This parameter will be ignored if a ``job_id`` is
+ also given.
+ location (str):
+ Location where to run the job. Must match the location of the
+ source table.
+ project (str):
+ Project ID of the project of where to run the job. Defaults
+ to the client's project.
+ job_config (google.cloud.bigquery.job.ExtractJobConfig):
+ (Optional) Extra configuration options for the job.
+ retry (google.api_core.retry.Retry):
+ (Optional) How to retry the RPC.
+ :type source: :class:`google.cloud.bigquery.table.TableReference`
+ :param source: table to be extracted.
+
+
+ Returns:
+ google.cloud.bigquery.job.ExtractJob: A new extract job instance.
+ """
+ job_id = _make_job_id(job_id, job_id_prefix)
+
+ if project is None:
+ project = self.project
+
+ if location is None:
+ location = self.location
+
+ job_ref = job._JobReference(job_id, project=project, location=location)
+ source = _table_arg_to_table_ref(source, default_project=self.project)
+
+ if isinstance(destination_uris, six.string_types):
+ destination_uris = [destination_uris]
+
+ extract_job = job.ExtractJob(
+ job_ref, source, destination_uris, client=self, job_config=job_config
+ )
+ extract_job._begin(retry=retry)
+
+ return extract_job
+
+ def query(
+ self,
+ query,
+ job_config=None,
+ job_id=None,
+ job_id_prefix=None,
+ location=None,
+ project=None,
+ retry=DEFAULT_RETRY,
+ ):
+ """Run a SQL query.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query
+
+ Arguments:
+ query (str):
+ SQL query to be executed. Defaults to the standard SQL
+ dialect. Use the ``job_config`` parameter to change dialects.
+
+ Keyword Arguments:
+ job_config (google.cloud.bigquery.job.QueryJobConfig):
+ (Optional) Extra configuration options for the job.
+ To override any options that were previously set in
+ the ``default_query_job_config`` given to the
+ ``Client`` constructor, manually set those options to ``None``,
+ or whatever value is preferred.
+ job_id (str): (Optional) ID to use for the query job.
+ job_id_prefix (str):
+ (Optional) The prefix to use for a randomly generated job ID.
+ This parameter will be ignored if a ``job_id`` is also given.
+ location (str):
+ Location where to run the job. Must match the location of the
+ any table used in the query as well as the destination table.
+ project (str):
+ Project ID of the project of where to run the job. Defaults
+ to the client's project.
+ retry (google.api_core.retry.Retry):
+ (Optional) How to retry the RPC.
+
+ Returns:
+ google.cloud.bigquery.job.QueryJob: A new query job instance.
+ """
+ job_id = _make_job_id(job_id, job_id_prefix)
+
+ if project is None:
+ project = self.project
+
+ if location is None:
+ location = self.location
+
+ if self._default_query_job_config:
+ if job_config:
+ # anything that's not defined on the incoming
+ # that is in the default,
+ # should be filled in with the default
+ # the incoming therefore has precedence
+ job_config = job_config._fill_from_default(
+ self._default_query_job_config
+ )
+ else:
+ job_config = self._default_query_job_config
+
+ job_ref = job._JobReference(job_id, project=project, location=location)
+ query_job = job.QueryJob(job_ref, query, client=self, job_config=job_config)
+ query_job._begin(retry=retry)
+
+ return query_job
+
+ def insert_rows(self, table, rows, selected_fields=None, **kwargs):
+ """Insert rows into a table via the streaming API.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tabledata/insertAll
+
+ Args:
+ table (Union[ \
+ :class:`~google.cloud.bigquery.table.Table`, \
+ :class:`~google.cloud.bigquery.table.TableReference`, \
+ str, \
+ ]):
+ The destination table for the row data, or a reference to it.
+ rows (Union[ \
+ Sequence[Tuple], \
+ Sequence[dict], \
+ ]):
+ Row data to be inserted. If a list of tuples is given, each
+ tuple should contain data for each schema field on the
+ current table and in the same order as the schema fields. If
+ a list of dictionaries is given, the keys must include all
+ required fields in the schema. Keys which do not correspond
+ to a field in the schema are ignored.
+ selected_fields (Sequence[ \
+ :class:`~google.cloud.bigquery.schema.SchemaField`, \
+ ]):
+ The fields to return. Required if ``table`` is a
+ :class:`~google.cloud.bigquery.table.TableReference`.
+ kwargs (dict):
+ Keyword arguments to
+ :meth:`~google.cloud.bigquery.client.Client.insert_rows_json`.
+
+ Returns:
+ Sequence[Mappings]:
+ One mapping per row with insert errors: the "index" key
+ identifies the row, and the "errors" key contains a list of
+ the mappings describing one or more problems with the row.
+
+ Raises:
+ ValueError: if table's schema is not set
+ """
+ table = _table_arg_to_table(table, default_project=self.project)
+
+ if not isinstance(table, Table):
+ raise TypeError(_NEED_TABLE_ARGUMENT)
+
+ schema = table.schema
+
+ # selected_fields can override the table schema.
+ if selected_fields is not None:
+ schema = selected_fields
+
+ if len(schema) == 0:
+ raise ValueError(
+ (
+ "Could not determine schema for table '{}'. Call client.get_table() "
+ "or pass in a list of schema fields to the selected_fields argument."
+ ).format(table)
+ )
+
+ json_rows = [_record_field_to_json(schema, row) for row in rows]
+
+ return self.insert_rows_json(table, json_rows, **kwargs)
+
+ def insert_rows_json(
+ self,
+ table,
+ json_rows,
+ row_ids=None,
+ skip_invalid_rows=None,
+ ignore_unknown_values=None,
+ template_suffix=None,
+ retry=DEFAULT_RETRY,
+ ):
+ """Insert rows into a table without applying local type conversions.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tabledata/insertAll
+
+ table (Union[ \
+ :class:`~google.cloud.bigquery.table.Table` \
+ :class:`~google.cloud.bigquery.table.TableReference`, \
+ str, \
+ ]):
+ The destination table for the row data, or a reference to it.
+ json_rows (Sequence[dict]):
+ Row data to be inserted. Keys must match the table schema fields
+ and values must be JSON-compatible representations.
+ row_ids (Sequence[str]):
+ (Optional) Unique ids, one per row being inserted. If omitted,
+ unique IDs are created.
+ skip_invalid_rows (bool):
+ (Optional) Insert all valid rows of a request, even if invalid
+ rows exist. The default value is False, which causes the entire
+ request to fail if any invalid rows exist.
+ ignore_unknown_values (bool):
+ (Optional) Accept rows that contain values that do not match the
+ schema. The unknown values are ignored. Default is False, which
+ treats unknown values as errors.
+ template_suffix (str):
+ (Optional) treat ``name`` as a template table and provide a suffix.
+ BigQuery will create the table `` + `` based
+ on the schema of the template table. See
+ https://cloud.google.com/bigquery/streaming-data-into-bigquery#template-tables
+ retry (:class:`google.api_core.retry.Retry`):
+ (Optional) How to retry the RPC.
+
+ Returns:
+ Sequence[Mappings]:
+ One mapping per row with insert errors: the "index" key
+ identifies the row, and the "errors" key contains a list of
+ the mappings describing one or more problems with the row.
+ """
+ # Convert table to just a reference because unlike insert_rows,
+ # insert_rows_json doesn't need the table schema. It's not doing any
+ # type conversions.
+ table = _table_arg_to_table_ref(table, default_project=self.project)
+ rows_info = []
+ data = {"rows": rows_info}
+
+ for index, row in enumerate(json_rows):
+ info = {"json": row}
+ if row_ids is not None:
+ info["insertId"] = row_ids[index]
+ else:
+ info["insertId"] = str(uuid.uuid4())
+ rows_info.append(info)
+
+ if skip_invalid_rows is not None:
+ data["skipInvalidRows"] = skip_invalid_rows
+
+ if ignore_unknown_values is not None:
+ data["ignoreUnknownValues"] = ignore_unknown_values
+
+ if template_suffix is not None:
+ data["templateSuffix"] = template_suffix
+
+ # We can always retry, because every row has an insert ID.
+ response = self._call_api(
+ retry, method="POST", path="%s/insertAll" % table.path, data=data
+ )
+ errors = []
+
+ for error in response.get("insertErrors", ()):
+ errors.append({"index": int(error["index"]), "errors": error["errors"]})
+
+ return errors
+
+ def list_partitions(self, table, retry=DEFAULT_RETRY):
+ """List the partitions in a table.
+
+ Arguments:
+ table (Union[ \
+ :class:`~google.cloud.bigquery.table.Table`, \
+ :class:`~google.cloud.bigquery.table.TableReference`, \
+ str, \
+ ]):
+ The table or reference from which to get partition info
+ retry (google.api_core.retry.Retry):
+ (Optional) How to retry the RPC.
+
+ Returns:
+ List[str]:
+ A list of the partition ids present in the partitioned table
+ """
+ table = _table_arg_to_table_ref(table, default_project=self.project)
+ meta_table = self.get_table(
+ TableReference(
+ self.dataset(table.dataset_id, project=table.project),
+ "%s$__PARTITIONS_SUMMARY__" % table.table_id,
+ )
+ )
+
+ subset = [col for col in meta_table.schema if col.name == "partition_id"]
+ return [
+ row[0]
+ for row in self.list_rows(meta_table, selected_fields=subset, retry=retry)
+ ]
+
+ def list_rows(
+ self,
+ table,
+ selected_fields=None,
+ max_results=None,
+ page_token=None,
+ start_index=None,
+ page_size=None,
+ retry=DEFAULT_RETRY,
+ ):
+ """List the rows of the table.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tabledata/list
+
+ .. note::
+
+ This method assumes that the provided schema is up-to-date with the
+ schema as defined on the back-end: if the two schemas are not
+ identical, the values returned may be incomplete. To ensure that the
+ local copy of the schema is up-to-date, call ``client.get_table``.
+
+ Args:
+ table (Union[ \
+ :class:`~google.cloud.bigquery.table.Table`, \
+ :class:`~google.cloud.bigquery.table.TableListItem`, \
+ :class:`~google.cloud.bigquery.table.TableReference`, \
+ str, \
+ ]):
+ The table to list, or a reference to it. When the table
+ object does not contain a schema and ``selected_fields`` is
+ not supplied, this method calls ``get_table`` to fetch the
+ table schema.
+ selected_fields (Sequence[ \
+ :class:`~google.cloud.bigquery.schema.SchemaField` \
+ ]):
+ The fields to return. If not supplied, data for all columns
+ are downloaded.
+ max_results (int):
+ (Optional) maximum number of rows to return.
+ page_token (str):
+ (Optional) Token representing a cursor into the table's rows.
+ If not passed, the API will return the first page of the
+ rows. The token marks the beginning of the iterator to be
+ returned and the value of the ``page_token`` can be accessed
+ at ``next_page_token`` of the
+ :class:`~google.cloud.bigquery.table.RowIterator`.
+ start_index (int):
+ (Optional) The zero-based index of the starting row to read.
+ page_size (int):
+ Optional. The maximum number of rows in each page of results
+ from this request. Non-positive values are ignored. Defaults
+ to a sensible value set by the API.
+ retry (:class:`google.api_core.retry.Retry`):
+ (Optional) How to retry the RPC.
+
+ Returns:
+ google.cloud.bigquery.table.RowIterator:
+ Iterator of row data
+ :class:`~google.cloud.bigquery.table.Row`-s. During each
+ page, the iterator will have the ``total_rows`` attribute
+ set, which counts the total number of rows **in the table**
+ (this is distinct from the total number of rows in the
+ current page: ``iterator.page.num_items``).
+ """
+ table = _table_arg_to_table(table, default_project=self.project)
+
+ if not isinstance(table, Table):
+ raise TypeError(_NEED_TABLE_ARGUMENT)
+
+ schema = table.schema
+
+ # selected_fields can override the table schema.
+ if selected_fields is not None:
+ schema = selected_fields
+
+ # No schema, but no selected_fields. Assume the developer wants all
+ # columns, so get the table resource for them rather than failing.
+ elif len(schema) == 0:
+ table = self.get_table(table.reference, retry=retry)
+ schema = table.schema
+
+ params = {}
+ if selected_fields is not None:
+ params["selectedFields"] = ",".join(field.name for field in selected_fields)
+ if start_index is not None:
+ params["startIndex"] = start_index
+
+ row_iterator = RowIterator(
+ client=self,
+ api_request=functools.partial(self._call_api, retry),
+ path="%s/data" % (table.path,),
+ schema=schema,
+ page_token=page_token,
+ max_results=max_results,
+ page_size=page_size,
+ extra_params=params,
+ table=table,
+ # Pass in selected_fields separately from schema so that full
+ # tables can be fetched without a column filter.
+ selected_fields=selected_fields,
+ )
+ return row_iterator
+
+ def _schema_from_json_file_object(self, file_obj):
+ """Helper function for schema_from_json that takes a
+ file object that describes a table schema.
+
+ Returns:
+ List of schema field objects.
+ """
+ json_data = json.load(file_obj)
+ return [SchemaField.from_api_repr(field) for field in json_data]
+
+ def _schema_to_json_file_object(self, schema_list, file_obj):
+ """Helper function for schema_to_json that takes a schema list and file
+ object and writes the schema list to the file object with json.dump
+ """
+ json.dump(schema_list, file_obj, indent=2, sort_keys=True)
+
+ def schema_from_json(self, file_or_path):
+ """Takes a file object or file path that contains json that describes
+ a table schema.
+
+ Returns:
+ List of schema field objects.
+ """
+ if isinstance(file_or_path, io.IOBase):
+ return self._schema_from_json_file_object(file_or_path)
+
+ with open(file_or_path) as file_obj:
+ return self._schema_from_json_file_object(file_obj)
+
+ def schema_to_json(self, schema_list, destination):
+ """Takes a list of schema field objects.
+
+ Serializes the list of schema field objects as json to a file.
+
+ Destination is a file path or a file object.
+ """
+ json_schema_list = [f.to_api_repr() for f in schema_list]
+
+ if isinstance(destination, io.IOBase):
+ return self._schema_to_json_file_object(json_schema_list, destination)
+
+ with open(destination, mode="w") as file_obj:
+ return self._schema_to_json_file_object(json_schema_list, file_obj)
+
+
+# pylint: disable=unused-argument
+def _item_to_project(iterator, resource):
+ """Convert a JSON project to the native object.
+
+ :type iterator: :class:`~google.api_core.page_iterator.Iterator`
+ :param iterator: The iterator that is currently in use.
+
+ :type resource: dict
+ :param resource: An item to be converted to a project.
+
+ :rtype: :class:`.Project`
+ :returns: The next project in the page.
+ """
+ return Project.from_api_repr(resource)
+
+
+# pylint: enable=unused-argument
+
+
+def _item_to_dataset(iterator, resource):
+ """Convert a JSON dataset to the native object.
+
+ :type iterator: :class:`~google.api_core.page_iterator.Iterator`
+ :param iterator: The iterator that is currently in use.
+
+ :type resource: dict
+ :param resource: An item to be converted to a dataset.
+
+ :rtype: :class:`.DatasetListItem`
+ :returns: The next dataset in the page.
+ """
+ return DatasetListItem(resource)
+
+
+def _item_to_job(iterator, resource):
+ """Convert a JSON job to the native object.
+
+ :type iterator: :class:`~google.api_core.page_iterator.Iterator`
+ :param iterator: The iterator that is currently in use.
+
+ :type resource: dict
+ :param resource: An item to be converted to a job.
+
+ :rtype: job instance.
+ :returns: The next job in the page.
+ """
+ return iterator.client.job_from_resource(resource)
+
+
+def _item_to_model(iterator, resource):
+ """Convert a JSON model to the native object.
+
+ Args:
+ iterator (google.api_core.page_iterator.Iterator):
+ The iterator that is currently in use.
+ resource (dict):
+ An item to be converted to a model.
+
+ Returns:
+ google.cloud.bigquery.model.Model: The next model in the page.
+ """
+ return Model.from_api_repr(resource)
+
+
+def _item_to_routine(iterator, resource):
+ """Convert a JSON model to the native object.
+
+ Args:
+ iterator (google.api_core.page_iterator.Iterator):
+ The iterator that is currently in use.
+ resource (dict):
+ An item to be converted to a routine.
+
+ Returns:
+ google.cloud.bigquery.routine.Routine: The next routine in the page.
+ """
+ return Routine.from_api_repr(resource)
+
+
+def _item_to_table(iterator, resource):
+ """Convert a JSON table to the native object.
+
+ :type iterator: :class:`~google.api_core.page_iterator.Iterator`
+ :param iterator: The iterator that is currently in use.
+
+ :type resource: dict
+ :param resource: An item to be converted to a table.
+
+ :rtype: :class:`~google.cloud.bigquery.table.Table`
+ :returns: The next table in the page.
+ """
+ return TableListItem(resource)
+
+
+def _make_job_id(job_id, prefix=None):
+ """Construct an ID for a new job.
+
+ :type job_id: str or ``NoneType``
+ :param job_id: the user-provided job ID
+
+ :type prefix: str or ``NoneType``
+ :param prefix: (Optional) the user-provided prefix for a job ID
+
+ :rtype: str
+ :returns: A job ID
+ """
+ if job_id is not None:
+ return job_id
+ elif prefix is not None:
+ return str(prefix) + str(uuid.uuid4())
+ else:
+ return str(uuid.uuid4())
+
+
+def _check_mode(stream):
+ """Check that a stream was opened in read-binary mode.
+
+ :type stream: IO[bytes]
+ :param stream: A bytes IO object open for reading.
+
+ :raises: :exc:`ValueError` if the ``stream.mode`` is a valid attribute
+ and is not among ``rb``, ``r+b`` or ``rb+``.
+ """
+ mode = getattr(stream, "mode", None)
+
+ if isinstance(stream, gzip.GzipFile):
+ if mode != gzip.READ:
+ raise ValueError(
+ "Cannot upload gzip files opened in write mode: use "
+ "gzip.GzipFile(filename, mode='rb')"
+ )
+ else:
+ if mode is not None and mode not in ("rb", "r+b", "rb+"):
+ raise ValueError(
+ "Cannot upload files opened in text mode: use "
+ "open(filename, mode='rb') or open(filename, mode='r+b')"
+ )
+
+
+def _get_upload_headers(user_agent):
+ """Get the headers for an upload request.
+
+ :type user_agent: str
+ :param user_agent: The user-agent for requests.
+
+ :rtype: dict
+ :returns: The headers to be used for the request.
+ """
+ return {
+ "Accept": "application/json",
+ "Accept-Encoding": "gzip, deflate",
+ "User-Agent": user_agent,
+ "content-type": "application/json",
+ }
diff --git a/bigquery/google/cloud/bigquery/dataset.py b/bigquery/google/cloud/bigquery/dataset.py
new file mode 100644
index 000000000000..494c219d4f67
--- /dev/null
+++ b/bigquery/google/cloud/bigquery/dataset.py
@@ -0,0 +1,715 @@
+# Copyright 2015 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Define API Datasets."""
+
+from __future__ import absolute_import
+
+import six
+import copy
+import re
+
+import google.cloud._helpers
+from google.cloud.bigquery import _helpers
+from google.cloud.bigquery.model import ModelReference
+from google.cloud.bigquery.routine import RoutineReference
+from google.cloud.bigquery.table import TableReference
+
+
+_PROJECT_PREFIX_PATTERN = re.compile(
+ r"""
+ (?P\S+\:[^.]+)\.(?P[^.]+)$
+""",
+ re.VERBOSE,
+)
+
+
+def _get_table_reference(self, table_id):
+ """Constructs a TableReference.
+
+ Args:
+ table_id (str): The ID of the table.
+
+ Returns:
+ google.cloud.bigquery.table.TableReference:
+ A table reference for a table in this dataset.
+ """
+ return TableReference(self, table_id)
+
+
+def _get_model_reference(self, model_id):
+ """Constructs a ModelReference.
+
+ Args:
+ model_id (str): the ID of the model.
+
+ Returns:
+ google.cloud.bigquery.model.ModelReference:
+ A ModelReference for a model in this dataset.
+ """
+ return ModelReference.from_api_repr(
+ {"projectId": self.project, "datasetId": self.dataset_id, "modelId": model_id}
+ )
+
+
+def _get_routine_reference(self, routine_id):
+ """Constructs a RoutineReference.
+
+ Args:
+ routine_id (str): the ID of the routine.
+
+ Returns:
+ google.cloud.bigquery.routine.RoutineReference:
+ A RoutineReference for a routine in this dataset.
+ """
+ return RoutineReference.from_api_repr(
+ {
+ "projectId": self.project,
+ "datasetId": self.dataset_id,
+ "routineId": routine_id,
+ }
+ )
+
+
+class AccessEntry(object):
+ """Represents grant of an access role to an entity.
+
+ An entry must have exactly one of the allowed :attr:`ENTITY_TYPES`. If
+ anything but ``view`` is set, a ``role`` is also required. ``role`` is
+ omitted for a ``view``, because ``view`` s are always read-only.
+
+ See https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets.
+
+ Attributes:
+ role (str):
+ Role granted to the entity. The following string values are
+ supported: `'READER'`, `'WRITER'`, `'OWNER'`. It may also be
+ :data:`None` if the ``entity_type`` is ``view``.
+
+ entity_type (str):
+ Type of entity being granted the role. One of :attr:`ENTITY_TYPES`.
+
+ entity_id (Union[str, Dict[str, str]]):
+ If the ``entity_type`` is not 'view', the ``entity_id`` is the
+ ``str`` ID of the entity being granted the role. If the
+ ``entity_type`` is 'view', the ``entity_id`` is a ``dict``
+ representing the view from a different dataset to grant access to
+ in the following format::
+
+ {
+ 'projectId': string,
+ 'datasetId': string,
+ 'tableId': string
+ }
+
+ Raises:
+ ValueError:
+ If the ``entity_type`` is not among :attr:`ENTITY_TYPES`, or if a
+ ``view`` has ``role`` set, or a non ``view`` **does not** have a
+ ``role`` set.
+
+ Examples:
+ >>> entry = AccessEntry('OWNER', 'userByEmail', 'user@example.com')
+
+ >>> view = {
+ ... 'projectId': 'my-project',
+ ... 'datasetId': 'my_dataset',
+ ... 'tableId': 'my_table'
+ ... }
+ >>> entry = AccessEntry(None, 'view', view)
+ """
+
+ ENTITY_TYPES = frozenset(
+ ["userByEmail", "groupByEmail", "domain", "specialGroup", "view"]
+ )
+ """Allowed entity types."""
+
+ def __init__(self, role, entity_type, entity_id):
+ if entity_type not in self.ENTITY_TYPES:
+ message = "Entity type %r not among: %s" % (
+ entity_type,
+ ", ".join(self.ENTITY_TYPES),
+ )
+ raise ValueError(message)
+ if entity_type == "view":
+ if role is not None:
+ raise ValueError(
+ "Role must be None for a view. Received " "role: %r" % (role,)
+ )
+ else:
+ if role is None:
+ raise ValueError(
+ "Role must be set for entity " "type %r" % (entity_type,)
+ )
+
+ self.role = role
+ self.entity_type = entity_type
+ self.entity_id = entity_id
+
+ def __eq__(self, other):
+ if not isinstance(other, AccessEntry):
+ return NotImplemented
+ return (
+ self.role == other.role
+ and self.entity_type == other.entity_type
+ and self.entity_id == other.entity_id
+ )
+
+ def __ne__(self, other):
+ return not self == other
+
+ def __repr__(self):
+ return "" % (
+ self.role,
+ self.entity_type,
+ self.entity_id,
+ )
+
+ def to_api_repr(self):
+ """Construct the API resource representation of this access entry
+
+ Returns:
+ Dict[str, object]: Access entry represented as an API resource
+ """
+ resource = {self.entity_type: self.entity_id}
+ if self.role is not None:
+ resource["role"] = self.role
+ return resource
+
+ @classmethod
+ def from_api_repr(cls, resource):
+ """Factory: construct an access entry given its API representation
+
+ Args:
+ resource (Dict[str, object]):
+ Access entry resource representation returned from the API
+
+ Returns:
+ google.cloud.bigquery.dataset.AccessEntry:
+ Access entry parsed from ``resource``.
+
+ Raises:
+ ValueError:
+ If the resource has more keys than ``role`` and one additional
+ key.
+ """
+ entry = resource.copy()
+ role = entry.pop("role", None)
+ entity_type, entity_id = entry.popitem()
+ if len(entry) != 0:
+ raise ValueError("Entry has unexpected keys remaining.", entry)
+ return cls(role, entity_type, entity_id)
+
+
+class DatasetReference(object):
+ """DatasetReferences are pointers to datasets.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets
+
+ Args:
+ project (str): The ID of the project
+ dataset_id (str): The ID of the dataset
+
+ Raises:
+ ValueError: If either argument is not of type ``str``.
+ """
+
+ def __init__(self, project, dataset_id):
+ if not isinstance(project, six.string_types):
+ raise ValueError("Pass a string for project")
+ if not isinstance(dataset_id, six.string_types):
+ raise ValueError("Pass a string for dataset_id")
+ self._project = project
+ self._dataset_id = dataset_id
+
+ @property
+ def project(self):
+ """str: Project ID of the dataset."""
+ return self._project
+
+ @property
+ def dataset_id(self):
+ """str: Dataset ID."""
+ return self._dataset_id
+
+ @property
+ def path(self):
+ """str: URL path for the dataset based on project and dataset ID."""
+ return "/projects/%s/datasets/%s" % (self.project, self.dataset_id)
+
+ table = _get_table_reference
+
+ model = _get_model_reference
+
+ routine = _get_routine_reference
+
+ @classmethod
+ def from_api_repr(cls, resource):
+ """Factory: construct a dataset reference given its API representation
+
+ Args:
+ resource (Dict[str, str]):
+ Dataset reference resource representation returned from the API
+
+ Returns:
+ google.cloud.bigquery.dataset.DatasetReference:
+ Dataset reference parsed from ``resource``.
+ """
+ project = resource["projectId"]
+ dataset_id = resource["datasetId"]
+ return cls(project, dataset_id)
+
+ @classmethod
+ def from_string(cls, dataset_id, default_project=None):
+ """Construct a dataset reference from dataset ID string.
+
+ Args:
+ dataset_id (str):
+ A dataset ID in standard SQL format. If ``default_project``
+ is not specified, this must include both the project ID and
+ the dataset ID, separated by ``.``.
+ default_project (str):
+ Optional. The project ID to use when ``dataset_id`` does not
+ include a project ID.
+
+ Returns:
+ DatasetReference:
+ Dataset reference parsed from ``dataset_id``.
+
+ Examples:
+ >>> DatasetReference.from_string('my-project-id.some_dataset')
+ DatasetReference('my-project-id', 'some_dataset')
+
+ Raises:
+ ValueError:
+ If ``dataset_id`` is not a fully-qualified dataset ID in
+ standard SQL format.
+ """
+ output_dataset_id = dataset_id
+ output_project_id = default_project
+ with_prefix = _PROJECT_PREFIX_PATTERN.match(dataset_id)
+ if with_prefix is None:
+ parts = dataset_id.split(".")
+ else:
+ project_id = with_prefix.group("project_id")
+ dataset_id = with_prefix.group("dataset_id")
+ parts = [project_id, dataset_id]
+
+ if len(parts) == 1 and not default_project:
+ raise ValueError(
+ "When default_project is not set, dataset_id must be a "
+ "fully-qualified dataset ID in standard SQL format, "
+ 'e.g., "project.dataset_id" got {}'.format(dataset_id)
+ )
+ elif len(parts) == 2:
+ output_project_id, output_dataset_id = parts
+ elif len(parts) > 2:
+ raise ValueError(
+ "Too many parts in dataset_id. Expected a fully-qualified "
+ "dataset ID in standard SQL format. e.g. "
+ '"project.dataset_id", got {}'.format(dataset_id)
+ )
+
+ return cls(output_project_id, output_dataset_id)
+
+ def to_api_repr(self):
+ """Construct the API resource representation of this dataset reference
+
+ Returns:
+ Dict[str, str]: dataset reference represented as an API resource
+ """
+ return {"projectId": self._project, "datasetId": self._dataset_id}
+
+ def _key(self):
+ """A tuple key that uniquely describes this field.
+
+ Used to compute this instance's hashcode and evaluate equality.
+
+ Returns:
+ Tuple[str]: The contents of this :class:`.DatasetReference`.
+ """
+ return (self._project, self._dataset_id)
+
+ def __eq__(self, other):
+ if not isinstance(other, DatasetReference):
+ return NotImplemented
+ return self._key() == other._key()
+
+ def __ne__(self, other):
+ return not self == other
+
+ def __hash__(self):
+ return hash(self._key())
+
+ def __repr__(self):
+ return "DatasetReference{}".format(self._key())
+
+
+class Dataset(object):
+ """Datasets are containers for tables.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets
+
+ Args:
+ dataset_ref (Union[ \
+ :class:`~google.cloud.bigquery.dataset.DatasetReference`, \
+ str, \
+ ]):
+ A pointer to a dataset. If ``dataset_ref`` is a string, it must
+ include both the project ID and the dataset ID, separated by
+ ``.``.
+ """
+
+ _PROPERTY_TO_API_FIELD = {
+ "access_entries": "access",
+ "created": "creationTime",
+ "default_table_expiration_ms": "defaultTableExpirationMs",
+ "friendly_name": "friendlyName",
+ }
+
+ def __init__(self, dataset_ref):
+ if isinstance(dataset_ref, six.string_types):
+ dataset_ref = DatasetReference.from_string(dataset_ref)
+ self._properties = {"datasetReference": dataset_ref.to_api_repr(), "labels": {}}
+
+ @property
+ def project(self):
+ """str: Project ID of the project bound to the dataset."""
+ return self._properties["datasetReference"]["projectId"]
+
+ @property
+ def path(self):
+ """str: URL path for the dataset based on project and dataset ID."""
+ return "/projects/%s/datasets/%s" % (self.project, self.dataset_id)
+
+ @property
+ def access_entries(self):
+ """List[google.cloud.bigquery.dataset.AccessEntry]: Dataset's access
+ entries.
+
+ ``role`` augments the entity type and must be present **unless** the
+ entity type is ``view``.
+
+ Raises:
+ TypeError: If 'value' is not a sequence
+ ValueError:
+ If any item in the sequence is not an
+ :class:`~google.cloud.bigquery.dataset.AccessEntry`.
+ """
+ entries = self._properties.get("access", [])
+ return [AccessEntry.from_api_repr(entry) for entry in entries]
+
+ @access_entries.setter
+ def access_entries(self, value):
+ if not all(isinstance(field, AccessEntry) for field in value):
+ raise ValueError("Values must be AccessEntry instances")
+ entries = [entry.to_api_repr() for entry in value]
+ self._properties["access"] = entries
+
+ @property
+ def created(self):
+ """Union[datetime.datetime, None]: Datetime at which the dataset was
+ created (:data:`None` until set from the server).
+ """
+ creation_time = self._properties.get("creationTime")
+ if creation_time is not None:
+ # creation_time will be in milliseconds.
+ return google.cloud._helpers._datetime_from_microseconds(
+ 1000.0 * float(creation_time)
+ )
+
+ @property
+ def dataset_id(self):
+ """str: Dataset ID."""
+ return self._properties["datasetReference"]["datasetId"]
+
+ @property
+ def full_dataset_id(self):
+ """Union[str, None]: ID for the dataset resource (:data:`None` until
+ set from the server)
+
+ In the format ``project_id:dataset_id``.
+ """
+ return self._properties.get("id")
+
+ @property
+ def reference(self):
+ """google.cloud.bigquery.dataset.DatasetReference: A reference to this
+ dataset.
+ """
+ return DatasetReference(self.project, self.dataset_id)
+
+ @property
+ def etag(self):
+ """Union[str, None]: ETag for the dataset resource (:data:`None` until
+ set from the server).
+ """
+ return self._properties.get("etag")
+
+ @property
+ def modified(self):
+ """Union[datetime.datetime, None]: Datetime at which the dataset was
+ last modified (:data:`None` until set from the server).
+ """
+ modified_time = self._properties.get("lastModifiedTime")
+ if modified_time is not None:
+ # modified_time will be in milliseconds.
+ return google.cloud._helpers._datetime_from_microseconds(
+ 1000.0 * float(modified_time)
+ )
+
+ @property
+ def self_link(self):
+ """Union[str, None]: URL for the dataset resource (:data:`None` until
+ set from the server).
+ """
+ return self._properties.get("selfLink")
+
+ @property
+ def default_table_expiration_ms(self):
+ """Union[int, None]: Default expiration time for tables in the dataset
+ (defaults to :data:`None`).
+
+ Raises:
+ ValueError: For invalid value types.
+ """
+ return _helpers._int_or_none(self._properties.get("defaultTableExpirationMs"))
+
+ @default_table_expiration_ms.setter
+ def default_table_expiration_ms(self, value):
+ if not isinstance(value, six.integer_types) and value is not None:
+ raise ValueError("Pass an integer, or None")
+ self._properties["defaultTableExpirationMs"] = _helpers._str_or_none(value)
+
+ @property
+ def description(self):
+ """Union[str, None]: Description of the dataset as set by the user
+ (defaults to :data:`None`).
+
+ Raises:
+ ValueError: for invalid value types.
+ """
+ return self._properties.get("description")
+
+ @description.setter
+ def description(self, value):
+ if not isinstance(value, six.string_types) and value is not None:
+ raise ValueError("Pass a string, or None")
+ self._properties["description"] = value
+
+ @property
+ def friendly_name(self):
+ """Union[str, None]: Title of the dataset as set by the user
+ (defaults to :data:`None`).
+
+ Raises:
+ ValueError: for invalid value types.
+ """
+ return self._properties.get("friendlyName")
+
+ @friendly_name.setter
+ def friendly_name(self, value):
+ if not isinstance(value, six.string_types) and value is not None:
+ raise ValueError("Pass a string, or None")
+ self._properties["friendlyName"] = value
+
+ @property
+ def location(self):
+ """Union[str, None]: Location in which the dataset is hosted as set by
+ the user (defaults to :data:`None`).
+
+ Raises:
+ ValueError: for invalid value types.
+ """
+ return self._properties.get("location")
+
+ @location.setter
+ def location(self, value):
+ if not isinstance(value, six.string_types) and value is not None:
+ raise ValueError("Pass a string, or None")
+ self._properties["location"] = value
+
+ @property
+ def labels(self):
+ """Dict[str, str]: Labels for the dataset.
+
+ This method always returns a dict. To change a dataset's labels,
+ modify the dict, then call
+ :meth:`google.cloud.bigquery.client.Client.update_dataset`. To delete
+ a label, set its value to :data:`None` before updating.
+
+ Raises:
+ ValueError: for invalid value types.
+ """
+ return self._properties.setdefault("labels", {})
+
+ @labels.setter
+ def labels(self, value):
+ if not isinstance(value, dict):
+ raise ValueError("Pass a dict")
+ self._properties["labels"] = value
+
+ @classmethod
+ def from_string(cls, full_dataset_id):
+ """Construct a dataset from fully-qualified dataset ID.
+
+ Args:
+ full_dataset_id (str):
+ A fully-qualified dataset ID in standard SQL format. Must
+ include both the project ID and the dataset ID, separated by
+ ``.``.
+
+ Returns:
+ Dataset: Dataset parsed from ``full_dataset_id``.
+
+ Examples:
+ >>> Dataset.from_string('my-project-id.some_dataset')
+ Dataset(DatasetReference('my-project-id', 'some_dataset'))
+
+ Raises:
+ ValueError:
+ If ``full_dataset_id`` is not a fully-qualified dataset ID in
+ standard SQL format.
+ """
+ return cls(DatasetReference.from_string(full_dataset_id))
+
+ @classmethod
+ def from_api_repr(cls, resource):
+ """Factory: construct a dataset given its API representation
+
+ Args:
+ resource (Dict[str: object]):
+ Dataset resource representation returned from the API
+
+ Returns:
+ google.cloud.bigquery.dataset.Dataset:
+ Dataset parsed from ``resource``.
+ """
+ if (
+ "datasetReference" not in resource
+ or "datasetId" not in resource["datasetReference"]
+ ):
+ raise KeyError(
+ "Resource lacks required identity information:"
+ '["datasetReference"]["datasetId"]'
+ )
+ project_id = resource["datasetReference"]["projectId"]
+ dataset_id = resource["datasetReference"]["datasetId"]
+ dataset = cls(DatasetReference(project_id, dataset_id))
+ dataset._properties = copy.deepcopy(resource)
+ return dataset
+
+ def to_api_repr(self):
+ """Construct the API resource representation of this dataset
+
+ Returns:
+ Dict[str, object]: The dataset represented as an API resource
+ """
+ return copy.deepcopy(self._properties)
+
+ def _build_resource(self, filter_fields):
+ """Generate a resource for ``update``."""
+ return _helpers._build_resource_from_properties(self, filter_fields)
+
+ table = _get_table_reference
+
+ model = _get_model_reference
+
+ routine = _get_routine_reference
+
+ def __repr__(self):
+ return "Dataset({})".format(repr(self.reference))
+
+
+class DatasetListItem(object):
+ """A read-only dataset resource from a list operation.
+
+ For performance reasons, the BigQuery API only includes some of the
+ dataset properties when listing datasets. Notably,
+ :attr:`~google.cloud.bigquery.dataset.Dataset.access_entries` is missing.
+
+ For a full list of the properties that the BigQuery API returns, see the
+ `REST documentation for datasets.list
+ `_.
+
+
+ Args:
+ resource (Dict[str, str]):
+ A dataset-like resource object from a dataset list response. A
+ ``datasetReference`` property is required.
+
+ Raises:
+ ValueError:
+ If ``datasetReference`` or one of its required members is missing
+ from ``resource``.
+ """
+
+ def __init__(self, resource):
+ if "datasetReference" not in resource:
+ raise ValueError("resource must contain a datasetReference value")
+ if "projectId" not in resource["datasetReference"]:
+ raise ValueError(
+ "resource['datasetReference'] must contain a projectId value"
+ )
+ if "datasetId" not in resource["datasetReference"]:
+ raise ValueError(
+ "resource['datasetReference'] must contain a datasetId value"
+ )
+ self._properties = resource
+
+ @property
+ def project(self):
+ """str: Project bound to the dataset."""
+ return self._properties["datasetReference"]["projectId"]
+
+ @property
+ def dataset_id(self):
+ """str: Dataset ID."""
+ return self._properties["datasetReference"]["datasetId"]
+
+ @property
+ def full_dataset_id(self):
+ """Union[str, None]: ID for the dataset resource (:data:`None` until
+ set from the server)
+
+ In the format ``project_id:dataset_id``.
+ """
+ return self._properties.get("id")
+
+ @property
+ def friendly_name(self):
+ """Union[str, None]: Title of the dataset as set by the user
+ (defaults to :data:`None`).
+ """
+ return self._properties.get("friendlyName")
+
+ @property
+ def labels(self):
+ """Dict[str, str]: Labels for the dataset."""
+ return self._properties.setdefault("labels", {})
+
+ @property
+ def reference(self):
+ """google.cloud.bigquery.dataset.DatasetReference: A reference to this
+ dataset.
+ """
+ return DatasetReference(self.project, self.dataset_id)
+
+ table = _get_table_reference
+
+ model = _get_model_reference
+
+ routine = _get_routine_reference
diff --git a/bigquery/google/cloud/bigquery/dbapi/__init__.py b/bigquery/google/cloud/bigquery/dbapi/__init__.py
new file mode 100644
index 000000000000..d1a723949b10
--- /dev/null
+++ b/bigquery/google/cloud/bigquery/dbapi/__init__.py
@@ -0,0 +1,87 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Google BigQuery implementation of the Database API Specification v2.0.
+
+This module implements the `Python Database API Specification v2.0 (DB-API)`_
+for Google BigQuery.
+
+.. _Python Database API Specification v2.0 (DB-API):
+ https://www.python.org/dev/peps/pep-0249/
+"""
+
+from google.cloud.bigquery.dbapi.connection import connect
+from google.cloud.bigquery.dbapi.connection import Connection
+from google.cloud.bigquery.dbapi.cursor import Cursor
+from google.cloud.bigquery.dbapi.exceptions import Warning
+from google.cloud.bigquery.dbapi.exceptions import Error
+from google.cloud.bigquery.dbapi.exceptions import InterfaceError
+from google.cloud.bigquery.dbapi.exceptions import DatabaseError
+from google.cloud.bigquery.dbapi.exceptions import DataError
+from google.cloud.bigquery.dbapi.exceptions import OperationalError
+from google.cloud.bigquery.dbapi.exceptions import IntegrityError
+from google.cloud.bigquery.dbapi.exceptions import InternalError
+from google.cloud.bigquery.dbapi.exceptions import ProgrammingError
+from google.cloud.bigquery.dbapi.exceptions import NotSupportedError
+from google.cloud.bigquery.dbapi.types import Binary
+from google.cloud.bigquery.dbapi.types import Date
+from google.cloud.bigquery.dbapi.types import DateFromTicks
+from google.cloud.bigquery.dbapi.types import Time
+from google.cloud.bigquery.dbapi.types import TimeFromTicks
+from google.cloud.bigquery.dbapi.types import Timestamp
+from google.cloud.bigquery.dbapi.types import TimestampFromTicks
+from google.cloud.bigquery.dbapi.types import BINARY
+from google.cloud.bigquery.dbapi.types import DATETIME
+from google.cloud.bigquery.dbapi.types import NUMBER
+from google.cloud.bigquery.dbapi.types import ROWID
+from google.cloud.bigquery.dbapi.types import STRING
+
+
+apilevel = "2.0"
+
+# Threads may share the module and connections, but not cursors.
+threadsafety = 2
+
+paramstyle = "pyformat"
+
+__all__ = [
+ "apilevel",
+ "threadsafety",
+ "paramstyle",
+ "connect",
+ "Connection",
+ "Cursor",
+ "Warning",
+ "Error",
+ "InterfaceError",
+ "DatabaseError",
+ "DataError",
+ "OperationalError",
+ "IntegrityError",
+ "InternalError",
+ "ProgrammingError",
+ "NotSupportedError",
+ "Binary",
+ "Date",
+ "DateFromTicks",
+ "Time",
+ "TimeFromTicks",
+ "Timestamp",
+ "TimestampFromTicks",
+ "BINARY",
+ "DATETIME",
+ "NUMBER",
+ "ROWID",
+ "STRING",
+]
diff --git a/bigquery/google/cloud/bigquery/dbapi/_helpers.py b/bigquery/google/cloud/bigquery/dbapi/_helpers.py
new file mode 100644
index 000000000000..6e7f58bd4944
--- /dev/null
+++ b/bigquery/google/cloud/bigquery/dbapi/_helpers.py
@@ -0,0 +1,117 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+try:
+ from collections import abc as collections_abc
+except ImportError: # Python 2.7
+ import collections as collections_abc
+
+import datetime
+import decimal
+import numbers
+
+import six
+
+from google.cloud import bigquery
+from google.cloud.bigquery.dbapi import exceptions
+
+
+def scalar_to_query_parameter(value, name=None):
+ """Convert a scalar value into a query parameter.
+
+ :type value: any
+ :param value: A scalar value to convert into a query parameter.
+
+ :type name: str
+ :param name: (Optional) Name of the query parameter.
+
+ :rtype: :class:`~google.cloud.bigquery.ScalarQueryParameter`
+ :returns:
+ A query parameter corresponding with the type and value of the plain
+ Python object.
+ :raises: :class:`~google.cloud.bigquery.dbapi.exceptions.ProgrammingError`
+ if the type cannot be determined.
+ """
+ parameter_type = None
+
+ if isinstance(value, bool):
+ parameter_type = "BOOL"
+ elif isinstance(value, numbers.Integral):
+ parameter_type = "INT64"
+ elif isinstance(value, numbers.Real):
+ parameter_type = "FLOAT64"
+ elif isinstance(value, decimal.Decimal):
+ parameter_type = "NUMERIC"
+ elif isinstance(value, six.text_type):
+ parameter_type = "STRING"
+ elif isinstance(value, six.binary_type):
+ parameter_type = "BYTES"
+ elif isinstance(value, datetime.datetime):
+ parameter_type = "DATETIME" if value.tzinfo is None else "TIMESTAMP"
+ elif isinstance(value, datetime.date):
+ parameter_type = "DATE"
+ elif isinstance(value, datetime.time):
+ parameter_type = "TIME"
+ else:
+ raise exceptions.ProgrammingError(
+ "encountered parameter {} with value {} of unexpected type".format(
+ name, value
+ )
+ )
+ return bigquery.ScalarQueryParameter(name, parameter_type, value)
+
+
+def to_query_parameters_list(parameters):
+ """Converts a sequence of parameter values into query parameters.
+
+ :type parameters: Sequence[Any]
+ :param parameters: Sequence of query parameter values.
+
+ :rtype: List[google.cloud.bigquery.query._AbstractQueryParameter]
+ :returns: A list of query parameters.
+ """
+ return [scalar_to_query_parameter(value) for value in parameters]
+
+
+def to_query_parameters_dict(parameters):
+ """Converts a dictionary of parameter values into query parameters.
+
+ :type parameters: Mapping[str, Any]
+ :param parameters: Dictionary of query parameter values.
+
+ :rtype: List[google.cloud.bigquery.query._AbstractQueryParameter]
+ :returns: A list of named query parameters.
+ """
+ return [
+ scalar_to_query_parameter(value, name=name)
+ for name, value in six.iteritems(parameters)
+ ]
+
+
+def to_query_parameters(parameters):
+ """Converts DB-API parameter values into query parameters.
+
+ :type parameters: Mapping[str, Any] or Sequence[Any]
+ :param parameters: A dictionary or sequence of query parameter values.
+
+ :rtype: List[google.cloud.bigquery.query._AbstractQueryParameter]
+ :returns: A list of query parameters.
+ """
+ if parameters is None:
+ return []
+
+ if isinstance(parameters, collections_abc.Mapping):
+ return to_query_parameters_dict(parameters)
+
+ return to_query_parameters_list(parameters)
diff --git a/bigquery/google/cloud/bigquery/dbapi/connection.py b/bigquery/google/cloud/bigquery/dbapi/connection.py
new file mode 100644
index 000000000000..0dbc9143b255
--- /dev/null
+++ b/bigquery/google/cloud/bigquery/dbapi/connection.py
@@ -0,0 +1,59 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Connection for the Google BigQuery DB-API."""
+
+from google.cloud import bigquery
+from google.cloud.bigquery.dbapi import cursor
+
+
+class Connection(object):
+ """DB-API Connection to Google BigQuery.
+
+ :type client: :class:`~google.cloud.bigquery.Client`
+ :param client: A client used to connect to BigQuery.
+ """
+
+ def __init__(self, client):
+ self._client = client
+
+ def close(self):
+ """No-op."""
+
+ def commit(self):
+ """No-op."""
+
+ def cursor(self):
+ """Return a new cursor object.
+
+ :rtype: :class:`~google.cloud.bigquery.dbapi.Cursor`
+ :returns: A DB-API cursor that uses this connection.
+ """
+ return cursor.Cursor(self)
+
+
+def connect(client=None):
+ """Construct a DB-API connection to Google BigQuery.
+
+ :type client: :class:`~google.cloud.bigquery.Client`
+ :param client:
+ (Optional) A client used to connect to BigQuery. If not passed, a
+ client is created using default options inferred from the environment.
+
+ :rtype: :class:`~google.cloud.bigquery.dbapi.Connection`
+ :returns: A new DB-API connection to BigQuery.
+ """
+ if client is None:
+ client = bigquery.Client()
+ return Connection(client)
diff --git a/bigquery/google/cloud/bigquery/dbapi/cursor.py b/bigquery/google/cloud/bigquery/dbapi/cursor.py
new file mode 100644
index 000000000000..1fbd9fb10cc4
--- /dev/null
+++ b/bigquery/google/cloud/bigquery/dbapi/cursor.py
@@ -0,0 +1,358 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Cursor for the Google BigQuery DB-API."""
+
+import collections
+
+try:
+ from collections import abc as collections_abc
+except ImportError: # Python 2.7
+ import collections as collections_abc
+
+import six
+
+from google.cloud.bigquery import job
+from google.cloud.bigquery.dbapi import _helpers
+from google.cloud.bigquery.dbapi import exceptions
+import google.cloud.exceptions
+
+# Per PEP 249: A 7-item sequence containing information describing one result
+# column. The first two items (name and type_code) are mandatory, the other
+# five are optional and are set to None if no meaningful values can be
+# provided.
+Column = collections.namedtuple(
+ "Column",
+ [
+ "name",
+ "type_code",
+ "display_size",
+ "internal_size",
+ "precision",
+ "scale",
+ "null_ok",
+ ],
+)
+
+
+class Cursor(object):
+ """DB-API Cursor to Google BigQuery.
+
+ :type connection: :class:`~google.cloud.bigquery.dbapi.Connection`
+ :param connection: A DB-API connection to Google BigQuery.
+ """
+
+ def __init__(self, connection):
+ self.connection = connection
+ self.description = None
+ # Per PEP 249: The attribute is -1 in case no .execute*() has been
+ # performed on the cursor or the rowcount of the last operation
+ # cannot be determined by the interface.
+ self.rowcount = -1
+ # Per PEP 249: The arraysize attribute defaults to 1, meaning to fetch
+ # a single row at a time.
+ self.arraysize = 1
+ self._query_data = None
+ self._query_job = None
+
+ def close(self):
+ """No-op."""
+
+ def _set_description(self, schema):
+ """Set description from schema.
+
+ :type schema: Sequence[google.cloud.bigquery.schema.SchemaField]
+ :param schema: A description of fields in the schema.
+ """
+ if schema is None:
+ self.description = None
+ return
+
+ self.description = tuple(
+ [
+ Column(
+ name=field.name,
+ type_code=field.field_type,
+ display_size=None,
+ internal_size=None,
+ precision=None,
+ scale=None,
+ null_ok=field.is_nullable,
+ )
+ for field in schema
+ ]
+ )
+
+ def _set_rowcount(self, query_results):
+ """Set the rowcount from query results.
+
+ Normally, this sets rowcount to the number of rows returned by the
+ query, but if it was a DML statement, it sets rowcount to the number
+ of modified rows.
+
+ :type query_results:
+ :class:`~google.cloud.bigquery.query._QueryResults`
+ :param query_results: results of a query
+ """
+ total_rows = 0
+ num_dml_affected_rows = query_results.num_dml_affected_rows
+
+ if query_results.total_rows is not None and query_results.total_rows > 0:
+ total_rows = query_results.total_rows
+ if num_dml_affected_rows is not None and num_dml_affected_rows > 0:
+ total_rows = num_dml_affected_rows
+ self.rowcount = total_rows
+
+ def execute(self, operation, parameters=None, job_id=None):
+ """Prepare and execute a database operation.
+
+ .. note::
+ When setting query parameters, values which are "text"
+ (``unicode`` in Python2, ``str`` in Python3) will use
+ the 'STRING' BigQuery type. Values which are "bytes" (``str`` in
+ Python2, ``bytes`` in Python3), will use using the 'BYTES' type.
+
+ A `~datetime.datetime` parameter without timezone information uses
+ the 'DATETIME' BigQuery type (example: Global Pi Day Celebration
+ March 14, 2017 at 1:59pm). A `~datetime.datetime` parameter with
+ timezone information uses the 'TIMESTAMP' BigQuery type (example:
+ a wedding on April 29, 2011 at 11am, British Summer Time).
+
+ For more information about BigQuery data types, see:
+ https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types
+
+ ``STRUCT``/``RECORD`` and ``REPEATED`` query parameters are not
+ yet supported. See:
+ https://github.com/GoogleCloudPlatform/google-cloud-python/issues/3524
+
+ :type operation: str
+ :param operation: A Google BigQuery query string.
+
+ :type parameters: Mapping[str, Any] or Sequence[Any]
+ :param parameters:
+ (Optional) dictionary or sequence of parameter values.
+
+ :type job_id: str
+ :param job_id: (Optional) The job_id to use. If not set, a job ID
+ is generated at random.
+ """
+ self._query_data = None
+ self._query_job = None
+ client = self.connection._client
+
+ # The DB-API uses the pyformat formatting, since the way BigQuery does
+ # query parameters was not one of the standard options. Convert both
+ # the query and the parameters to the format expected by the client
+ # libraries.
+ formatted_operation = _format_operation(operation, parameters=parameters)
+ query_parameters = _helpers.to_query_parameters(parameters)
+
+ config = job.QueryJobConfig()
+ config.query_parameters = query_parameters
+ config.use_legacy_sql = False
+ self._query_job = client.query(
+ formatted_operation, job_config=config, job_id=job_id
+ )
+
+ # Wait for the query to finish.
+ try:
+ self._query_job.result()
+ except google.cloud.exceptions.GoogleCloudError as exc:
+ raise exceptions.DatabaseError(exc)
+
+ query_results = self._query_job._query_results
+ self._set_rowcount(query_results)
+ self._set_description(query_results.schema)
+
+ def executemany(self, operation, seq_of_parameters):
+ """Prepare and execute a database operation multiple times.
+
+ :type operation: str
+ :param operation: A Google BigQuery query string.
+
+ :type seq_of_parameters: Sequence[Mapping[str, Any] or Sequence[Any]]
+ :param parameters: Sequence of many sets of parameter values.
+ """
+ for parameters in seq_of_parameters:
+ self.execute(operation, parameters)
+
+ def _try_fetch(self, size=None):
+ """Try to start fetching data, if not yet started.
+
+ Mutates self to indicate that iteration has started.
+ """
+ if self._query_job is None:
+ raise exceptions.InterfaceError(
+ "No query results: execute() must be called before fetch."
+ )
+
+ is_dml = (
+ self._query_job.statement_type
+ and self._query_job.statement_type.upper() != "SELECT"
+ )
+ if is_dml:
+ self._query_data = iter([])
+ return
+
+ if self._query_data is None:
+ client = self.connection._client
+ rows_iter = client.list_rows(
+ self._query_job.destination,
+ selected_fields=self._query_job._query_results.schema,
+ page_size=self.arraysize,
+ )
+ self._query_data = iter(rows_iter)
+
+ def fetchone(self):
+ """Fetch a single row from the results of the last ``execute*()`` call.
+
+ :rtype: tuple
+ :returns:
+ A tuple representing a row or ``None`` if no more data is
+ available.
+ :raises: :class:`~google.cloud.bigquery.dbapi.InterfaceError`
+ if called before ``execute()``.
+ """
+ self._try_fetch()
+ try:
+ return six.next(self._query_data)
+ except StopIteration:
+ return None
+
+ def fetchmany(self, size=None):
+ """Fetch multiple results from the last ``execute*()`` call.
+
+ .. note::
+ The size parameter is not used for the request/response size.
+ Set the ``arraysize`` attribute before calling ``execute()`` to
+ set the batch size.
+
+ :type size: int
+ :param size:
+ (Optional) Maximum number of rows to return. Defaults to the
+ ``arraysize`` property value.
+
+ :rtype: List[tuple]
+ :returns: A list of rows.
+ :raises: :class:`~google.cloud.bigquery.dbapi.InterfaceError`
+ if called before ``execute()``.
+ """
+ if size is None:
+ size = self.arraysize
+
+ self._try_fetch(size=size)
+ rows = []
+
+ for row in self._query_data:
+ rows.append(row)
+ if len(rows) >= size:
+ break
+
+ return rows
+
+ def fetchall(self):
+ """Fetch all remaining results from the last ``execute*()`` call.
+
+ :rtype: List[tuple]
+ :returns: A list of all the rows in the results.
+ :raises: :class:`~google.cloud.bigquery.dbapi.InterfaceError`
+ if called before ``execute()``.
+ """
+ self._try_fetch()
+ return list(self._query_data)
+
+ def setinputsizes(self, sizes):
+ """No-op."""
+
+ def setoutputsize(self, size, column=None):
+ """No-op."""
+
+
+def _format_operation_list(operation, parameters):
+ """Formats parameters in operation in the way BigQuery expects.
+
+ The input operation will be a query like ``SELECT %s`` and the output
+ will be a query like ``SELECT ?``.
+
+ :type operation: str
+ :param operation: A Google BigQuery query string.
+
+ :type parameters: Sequence[Any]
+ :param parameters: Sequence of parameter values.
+
+ :rtype: str
+ :returns: A formatted query string.
+ :raises: :class:`~google.cloud.bigquery.dbapi.ProgrammingError`
+ if a parameter used in the operation is not found in the
+ ``parameters`` argument.
+ """
+ formatted_params = ["?" for _ in parameters]
+
+ try:
+ return operation % tuple(formatted_params)
+ except TypeError as exc:
+ raise exceptions.ProgrammingError(exc)
+
+
+def _format_operation_dict(operation, parameters):
+ """Formats parameters in operation in the way BigQuery expects.
+
+ The input operation will be a query like ``SELECT %(namedparam)s`` and
+ the output will be a query like ``SELECT @namedparam``.
+
+ :type operation: str
+ :param operation: A Google BigQuery query string.
+
+ :type parameters: Mapping[str, Any]
+ :param parameters: Dictionary of parameter values.
+
+ :rtype: str
+ :returns: A formatted query string.
+ :raises: :class:`~google.cloud.bigquery.dbapi.ProgrammingError`
+ if a parameter used in the operation is not found in the
+ ``parameters`` argument.
+ """
+ formatted_params = {}
+ for name in parameters:
+ escaped_name = name.replace("`", r"\`")
+ formatted_params[name] = "@`{}`".format(escaped_name)
+
+ try:
+ return operation % formatted_params
+ except KeyError as exc:
+ raise exceptions.ProgrammingError(exc)
+
+
+def _format_operation(operation, parameters=None):
+ """Formats parameters in operation in way BigQuery expects.
+
+ :type: str
+ :param operation: A Google BigQuery query string.
+
+ :type: Mapping[str, Any] or Sequence[Any]
+ :param parameters: Optional parameter values.
+
+ :rtype: str
+ :returns: A formatted query string.
+ :raises: :class:`~google.cloud.bigquery.dbapi.ProgrammingError`
+ if a parameter used in the operation is not found in the
+ ``parameters`` argument.
+ """
+ if parameters is None:
+ return operation
+
+ if isinstance(parameters, collections_abc.Mapping):
+ return _format_operation_dict(operation, parameters)
+
+ return _format_operation_list(operation, parameters)
diff --git a/bigquery/google/cloud/bigquery/dbapi/exceptions.py b/bigquery/google/cloud/bigquery/dbapi/exceptions.py
new file mode 100644
index 000000000000..37f7129d0618
--- /dev/null
+++ b/bigquery/google/cloud/bigquery/dbapi/exceptions.py
@@ -0,0 +1,58 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Exceptions used in the Google BigQuery DB-API."""
+
+
+class Warning(Exception):
+ """Exception raised for important DB-API warnings."""
+
+
+class Error(Exception):
+ """Exception representing all non-warning DB-API errors."""
+
+
+class InterfaceError(Error):
+ """DB-API error related to the database interface."""
+
+
+class DatabaseError(Error):
+ """DB-API error related to the database."""
+
+
+class DataError(DatabaseError):
+ """DB-API error due to problems with the processed data."""
+
+
+class OperationalError(DatabaseError):
+ """DB-API error related to the database operation.
+
+ These errors are not necessarily under the control of the programmer.
+ """
+
+
+class IntegrityError(DatabaseError):
+ """DB-API error when integrity of the database is affected."""
+
+
+class InternalError(DatabaseError):
+ """DB-API error when the database encounters an internal error."""
+
+
+class ProgrammingError(DatabaseError):
+ """DB-API exception raised for programming errors."""
+
+
+class NotSupportedError(DatabaseError):
+ """DB-API error for operations not supported by the database or API."""
diff --git a/bigquery/google/cloud/bigquery/dbapi/types.py b/bigquery/google/cloud/bigquery/dbapi/types.py
new file mode 100644
index 000000000000..3c8c454a011a
--- /dev/null
+++ b/bigquery/google/cloud/bigquery/dbapi/types.py
@@ -0,0 +1,85 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Types used in the Google BigQuery DB-API.
+
+See `PEP-249`_ for details.
+
+.. _PEP-249:
+ https://www.python.org/dev/peps/pep-0249/#type-objects-and-constructors
+"""
+
+import datetime
+
+
+Date = datetime.date
+Time = datetime.time
+Timestamp = datetime.datetime
+DateFromTicks = datetime.date.fromtimestamp
+TimestampFromTicks = datetime.datetime.fromtimestamp
+
+
+def Binary(string):
+ """Contruct a DB-API binary value.
+
+ :type string: str
+ :param string: A string to encode as a binary value.
+
+ :rtype: bytes
+ :returns: The UTF-8 encoded bytes representing the string.
+ """
+ return string.encode("utf-8")
+
+
+def TimeFromTicks(ticks, tz=None):
+ """Construct a DB-API time value from the given ticks value.
+
+ :type ticks: float
+ :param ticks:
+ a number of seconds since the epoch; see the documentation of the
+ standard Python time module for details.
+
+ :type tz: :class:`datetime.tzinfo`
+ :param tz: (Optional) time zone to use for conversion
+
+ :rtype: :class:`datetime.time`
+ :returns: time represented by ticks.
+ """
+ dt = datetime.datetime.fromtimestamp(ticks, tz=tz)
+ return dt.timetz()
+
+
+class _DBAPITypeObject(object):
+ """DB-API type object which compares equal to many different strings.
+
+ See `PEP-249`_ for details.
+
+ .. _PEP-249:
+ https://www.python.org/dev/peps/pep-0249/#implementation-hints-for-module-authors
+ """
+
+ def __init__(self, *values):
+ self.values = values
+
+ def __eq__(self, other):
+ return other in self.values
+
+
+STRING = "STRING"
+BINARY = _DBAPITypeObject("BYTES", "RECORD", "STRUCT")
+NUMBER = _DBAPITypeObject(
+ "INTEGER", "INT64", "FLOAT", "FLOAT64", "NUMERIC", "BOOLEAN", "BOOL"
+)
+DATETIME = _DBAPITypeObject("TIMESTAMP", "DATE", "TIME", "DATETIME")
+ROWID = "ROWID"
diff --git a/bigquery/google/cloud/bigquery/enums.py b/bigquery/google/cloud/bigquery/enums.py
new file mode 100644
index 000000000000..29fe543f6505
--- /dev/null
+++ b/bigquery/google/cloud/bigquery/enums.py
@@ -0,0 +1,92 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import re
+
+import enum
+import six
+
+from google.cloud.bigquery_v2.gapic import enums as gapic_enums
+
+
+_SQL_SCALAR_TYPES = frozenset(
+ (
+ "INT64",
+ "BOOL",
+ "FLOAT64",
+ "STRING",
+ "BYTES",
+ "TIMESTAMP",
+ "DATE",
+ "TIME",
+ "DATETIME",
+ "GEOGRAPHY",
+ "NUMERIC",
+ )
+)
+
+_SQL_NONSCALAR_TYPES = frozenset(("TYPE_KIND_UNSPECIFIED", "ARRAY", "STRUCT"))
+
+
+def _make_sql_scalars_enum():
+ """Create an enum based on a gapic enum containing only SQL scalar types."""
+
+ new_enum = enum.Enum(
+ "StandardSqlDataTypes",
+ (
+ (member.name, member.value)
+ for member in gapic_enums.StandardSqlDataType.TypeKind
+ if member.name in _SQL_SCALAR_TYPES
+ ),
+ )
+
+ # make sure the docstring for the new enum is also correct
+ orig_doc = gapic_enums.StandardSqlDataType.TypeKind.__doc__
+ skip_pattern = re.compile(
+ "|".join(_SQL_NONSCALAR_TYPES)
+ + "|because a JSON object" # the second description line of STRUCT member
+ )
+
+ new_doc = "\n".join(
+ six.moves.filterfalse(skip_pattern.search, orig_doc.splitlines())
+ )
+ new_enum.__doc__ = "An Enum of scalar SQL types.\n" + new_doc
+
+ return new_enum
+
+
+StandardSqlDataTypes = _make_sql_scalars_enum()
+
+
+# See also: https://cloud.google.com/bigquery/data-types#legacy_sql_data_types
+# and https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types
+class SqlTypeNames(str, enum.Enum):
+ """Enum of allowed SQL type names in schema.SchemaField."""
+
+ STRING = "STRING"
+ BYTES = "BYTES"
+ INTEGER = "INTEGER"
+ INT64 = "INTEGER"
+ FLOAT = "FLOAT"
+ FLOAT64 = "FLOAT"
+ NUMERIC = "NUMERIC"
+ BOOLEAN = "BOOLEAN"
+ BOOL = "BOOLEAN"
+ GEOGRAPHY = "GEOGRAPHY" # NOTE: not available in legacy types
+ RECORD = "RECORD"
+ STRUCT = "RECORD"
+ TIMESTAMP = "TIMESTAMP"
+ DATE = "DATE"
+ TIME = "TIME"
+ DATETIME = "DATETIME"
diff --git a/bigquery/google/cloud/bigquery/external_config.py b/bigquery/google/cloud/bigquery/external_config.py
new file mode 100644
index 000000000000..048c2178a654
--- /dev/null
+++ b/bigquery/google/cloud/bigquery/external_config.py
@@ -0,0 +1,718 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Define classes that describe external data sources.
+
+ These are used for both Table.externalDataConfiguration and
+ Job.configuration.query.tableDefinitions.
+"""
+
+from __future__ import absolute_import
+
+import base64
+import copy
+
+from google.cloud.bigquery._helpers import _to_bytes
+from google.cloud.bigquery._helpers import _bytes_to_json
+from google.cloud.bigquery._helpers import _int_or_none
+from google.cloud.bigquery.schema import SchemaField
+
+
+class ExternalSourceFormat(object):
+ """The format for external data files.
+
+ Note that the set of allowed values for external data sources is different
+ than the set used for loading data (see
+ :class:`~google.cloud.bigquery.job.SourceFormat`).
+ """
+
+ CSV = "CSV"
+ """Specifies CSV format."""
+
+ GOOGLE_SHEETS = "GOOGLE_SHEETS"
+ """Specifies Google Sheets format."""
+
+ NEWLINE_DELIMITED_JSON = "NEWLINE_DELIMITED_JSON"
+ """Specifies newline delimited JSON format."""
+
+ AVRO = "AVRO"
+ """Specifies Avro format."""
+
+ DATASTORE_BACKUP = "DATASTORE_BACKUP"
+ """Specifies datastore backup format"""
+
+ BIGTABLE = "BIGTABLE"
+ """Specifies Bigtable format."""
+
+
+class BigtableColumn(object):
+ """Options for a Bigtable column."""
+
+ def __init__(self):
+ self._properties = {}
+
+ @property
+ def encoding(self):
+ """str: The encoding of the values when the type is not `STRING`
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.%28key%29.bigtableOptions.columnFamilies.columns.encoding
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.bigtableOptions.columnFamilies.columns.encoding
+ """
+ return self._properties.get("encoding")
+
+ @encoding.setter
+ def encoding(self, value):
+ self._properties["encoding"] = value
+
+ @property
+ def field_name(self):
+ """str: An identifier to use if the qualifier is not a valid BigQuery
+ field identifier
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.%28key%29.bigtableOptions.columnFamilies.columns.fieldName
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.bigtableOptions.columnFamilies.columns.fieldName
+ """
+ return self._properties.get("fieldName")
+
+ @field_name.setter
+ def field_name(self, value):
+ self._properties["fieldName"] = value
+
+ @property
+ def only_read_latest(self):
+ """bool: If this is set, only the latest version of value in this
+ column are exposed.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.%28key%29.bigtableOptions.columnFamilies.columns.onlyReadLatest
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.bigtableOptions.columnFamilies.columns.onlyReadLatest
+ """
+ return self._properties.get("onlyReadLatest")
+
+ @only_read_latest.setter
+ def only_read_latest(self, value):
+ self._properties["onlyReadLatest"] = value
+
+ @property
+ def qualifier_encoded(self):
+ """Union[str, bytes]: The qualifier encoded in binary.
+
+ The type is ``str`` (Python 2.x) or ``bytes`` (Python 3.x). The module
+ will handle base64 encoding for you.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.%28key%29.bigtableOptions.columnFamilies.columns.qualifierEncoded
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.bigtableOptions.columnFamilies.columns.qualifierEncoded
+ """
+ prop = self._properties.get("qualifierEncoded")
+ if prop is None:
+ return None
+ return base64.standard_b64decode(_to_bytes(prop))
+
+ @qualifier_encoded.setter
+ def qualifier_encoded(self, value):
+ self._properties["qualifierEncoded"] = _bytes_to_json(value)
+
+ @property
+ def qualifier_string(self):
+ """str: A valid UTF-8 string qualifier
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.%28key%29.bigtableOptions.columnFamilies.columns.qualifierEncoded
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.bigtableOptions.columnFamilies.columns.qualifierEncoded
+ """
+ return self._properties.get("qualifierString")
+
+ @qualifier_string.setter
+ def qualifier_string(self, value):
+ self._properties["qualifierString"] = value
+
+ @property
+ def type_(self):
+ """str: The type to convert the value in cells of this column.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.%28key%29.bigtableOptions.columnFamilies.columns.type
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.bigtableOptions.columnFamilies.columns.type
+ """
+ return self._properties.get("type")
+
+ @type_.setter
+ def type_(self, value):
+ self._properties["type"] = value
+
+ def to_api_repr(self):
+ """Build an API representation of this object.
+
+ Returns:
+ Dict[str, Any]:
+ A dictionary in the format used by the BigQuery API.
+ """
+ return copy.deepcopy(self._properties)
+
+ @classmethod
+ def from_api_repr(cls, resource):
+ """Factory: construct a :class:`~.external_config.BigtableColumn`
+ instance given its API representation.
+
+ Args:
+ resource (Dict[str, Any]):
+ Definition of a :class:`~.external_config.BigtableColumn`
+ instance in the same representation as is returned from the
+ API.
+
+ Returns:
+ :class:`~.external_config.BigtableColumn`:
+ Configuration parsed from ``resource``.
+ """
+ config = cls()
+ config._properties = copy.deepcopy(resource)
+ return config
+
+
+class BigtableColumnFamily(object):
+ """Options for a Bigtable column family."""
+
+ def __init__(self):
+ self._properties = {}
+
+ @property
+ def encoding(self):
+ """str: The encoding of the values when the type is not `STRING`
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.(key).bigtableOptions.columnFamilies.encoding
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.bigtableOptions.columnFamilies.encoding
+ """
+ return self._properties.get("encoding")
+
+ @encoding.setter
+ def encoding(self, value):
+ self._properties["encoding"] = value
+
+ @property
+ def family_id(self):
+ """str: Identifier of the column family.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.(key).bigtableOptions.columnFamilies.familyId
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.bigtableOptions.columnFamilies.familyId
+ """
+ return self._properties.get("familyId")
+
+ @family_id.setter
+ def family_id(self, value):
+ self._properties["familyId"] = value
+
+ @property
+ def only_read_latest(self):
+ """bool: If this is set only the latest version of value are exposed
+ for all columns in this column family.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.(key).bigtableOptions.columnFamilies.onlyReadLatest
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.bigtableOptions.columnFamilies.onlyReadLatest
+ """
+ return self._properties.get("onlyReadLatest")
+
+ @only_read_latest.setter
+ def only_read_latest(self, value):
+ self._properties["onlyReadLatest"] = value
+
+ @property
+ def type_(self):
+ """str: The type to convert the value in cells of this column family.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.(key).bigtableOptions.columnFamilies.type
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.bigtableOptions.columnFamilies.type
+ """
+ return self._properties.get("type")
+
+ @type_.setter
+ def type_(self, value):
+ self._properties["type"] = value
+
+ @property
+ def columns(self):
+ """List[:class:`~.external_config.BigtableColumn`]: Lists of columns
+ that should be exposed as individual fields.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.(key).bigtableOptions.columnFamilies.columns
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.bigtableOptions.columnFamilies.columns
+ """
+ prop = self._properties.get("columns", [])
+ return [BigtableColumn.from_api_repr(col) for col in prop]
+
+ @columns.setter
+ def columns(self, value):
+ self._properties["columns"] = [col.to_api_repr() for col in value]
+
+ def to_api_repr(self):
+ """Build an API representation of this object.
+
+ Returns:
+ Dict[str, Any]:
+ A dictionary in the format used by the BigQuery API.
+ """
+ return copy.deepcopy(self._properties)
+
+ @classmethod
+ def from_api_repr(cls, resource):
+ """Factory: construct a :class:`~.external_config.BigtableColumnFamily`
+ instance given its API representation.
+
+ Args:
+ resource (Dict[str, Any]):
+ Definition of a :class:`~.external_config.BigtableColumnFamily`
+ instance in the same representation as is returned from the
+ API.
+
+ Returns:
+ :class:`~.external_config.BigtableColumnFamily`:
+ Configuration parsed from ``resource``.
+ """
+ config = cls()
+ config._properties = copy.deepcopy(resource)
+ return config
+
+
+class BigtableOptions(object):
+ """Options that describe how to treat Bigtable tables as BigQuery tables.
+ """
+
+ _SOURCE_FORMAT = "BIGTABLE"
+ _RESOURCE_NAME = "bigtableOptions"
+
+ def __init__(self):
+ self._properties = {}
+
+ @property
+ def ignore_unspecified_column_families(self):
+ """bool: If :data:`True`, ignore columns not specified in
+ :attr:`column_families` list. Defaults to :data:`False`.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.(key).bigtableOptions.ignoreUnspecifiedColumnFamilies
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.bigtableOptions.ignoreUnspecifiedColumnFamilies
+ """
+ return self._properties.get("ignoreUnspecifiedColumnFamilies")
+
+ @ignore_unspecified_column_families.setter
+ def ignore_unspecified_column_families(self, value):
+ self._properties["ignoreUnspecifiedColumnFamilies"] = value
+
+ @property
+ def read_rowkey_as_string(self):
+ """bool: If :data:`True`, rowkey column families will be read and
+ converted to string. Defaults to :data:`False`.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.(key).bigtableOptions.readRowkeyAsString
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.bigtableOptions.readRowkeyAsString
+ """
+ return self._properties.get("readRowkeyAsString")
+
+ @read_rowkey_as_string.setter
+ def read_rowkey_as_string(self, value):
+ self._properties["readRowkeyAsString"] = value
+
+ @property
+ def column_families(self):
+ """List[:class:`~.external_config.BigtableColumnFamily`]: List of
+ column families to expose in the table schema along with their types.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.(key).bigtableOptions.columnFamilies
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.bigtableOptions.columnFamilies
+ """
+ prop = self._properties.get("columnFamilies", [])
+ return [BigtableColumnFamily.from_api_repr(cf) for cf in prop]
+
+ @column_families.setter
+ def column_families(self, value):
+ self._properties["columnFamilies"] = [cf.to_api_repr() for cf in value]
+
+ def to_api_repr(self):
+ """Build an API representation of this object.
+
+ Returns:
+ Dict[str, Any]:
+ A dictionary in the format used by the BigQuery API.
+ """
+ return copy.deepcopy(self._properties)
+
+ @classmethod
+ def from_api_repr(cls, resource):
+ """Factory: construct a :class:`~.external_config.BigtableOptions`
+ instance given its API representation.
+
+ Args:
+ resource (Dict[str, Any]):
+ Definition of a :class:`~.external_config.BigtableOptions`
+ instance in the same representation as is returned from the
+ API.
+
+ Returns:
+ :class:`~.external_config.BigtableOptions`:
+ Configuration parsed from ``resource``.
+ """
+ config = cls()
+ config._properties = copy.deepcopy(resource)
+ return config
+
+
+class CSVOptions(object):
+ """Options that describe how to treat CSV files as BigQuery tables."""
+
+ _SOURCE_FORMAT = "CSV"
+ _RESOURCE_NAME = "csvOptions"
+
+ def __init__(self):
+ self._properties = {}
+
+ @property
+ def allow_jagged_rows(self):
+ """bool: If :data:`True`, BigQuery treats missing trailing columns as
+ null values. Defaults to :data:`False`.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.(key).csvOptions.allowJaggedRows
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.csvOptions.allowJaggedRows
+ """
+ return self._properties.get("allowJaggedRows")
+
+ @allow_jagged_rows.setter
+ def allow_jagged_rows(self, value):
+ self._properties["allowJaggedRows"] = value
+
+ @property
+ def allow_quoted_newlines(self):
+ """bool: If :data:`True`, quoted data sections that contain newline
+ characters in a CSV file are allowed. Defaults to :data:`False`.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.(key).csvOptions.allowQuotedNewlines
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.csvOptions.allowQuotedNewlines
+ """
+ return self._properties.get("allowQuotedNewlines")
+
+ @allow_quoted_newlines.setter
+ def allow_quoted_newlines(self, value):
+ self._properties["allowQuotedNewlines"] = value
+
+ @property
+ def encoding(self):
+ """str: The character encoding of the data.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.(key).csvOptions.encoding
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.csvOptions.encoding
+ """
+ return self._properties.get("encoding")
+
+ @encoding.setter
+ def encoding(self, value):
+ self._properties["encoding"] = value
+
+ @property
+ def field_delimiter(self):
+ """str: The separator for fields in a CSV file. Defaults to comma (',').
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.(key).csvOptions.fieldDelimiter
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.csvOptions.fieldDelimiter
+ """
+ return self._properties.get("fieldDelimiter")
+
+ @field_delimiter.setter
+ def field_delimiter(self, value):
+ self._properties["fieldDelimiter"] = value
+
+ @property
+ def quote_character(self):
+ """str: The value that is used to quote data sections in a CSV file.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.(key).csvOptions.quote
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.csvOptions.quote
+ """
+ return self._properties.get("quote")
+
+ @quote_character.setter
+ def quote_character(self, value):
+ self._properties["quote"] = value
+
+ @property
+ def skip_leading_rows(self):
+ """int: The number of rows at the top of a CSV file.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.(key).csvOptions.skipLeadingRows
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.csvOptions.skipLeadingRows
+ """
+ return _int_or_none(self._properties.get("skipLeadingRows"))
+
+ @skip_leading_rows.setter
+ def skip_leading_rows(self, value):
+ self._properties["skipLeadingRows"] = str(value)
+
+ def to_api_repr(self):
+ """Build an API representation of this object.
+
+ Returns:
+ Dict[str, Any]:
+ A dictionary in the format used by the BigQuery API.
+ """
+ return copy.deepcopy(self._properties)
+
+ @classmethod
+ def from_api_repr(cls, resource):
+ """Factory: construct a :class:`~.external_config.CSVOptions` instance
+ given its API representation.
+
+ Args:
+ resource (Dict[str, Any]):
+ Definition of a :class:`~.external_config.CSVOptions`
+ instance in the same representation as is returned from the
+ API.
+
+ Returns:
+ :class:`~.external_config.CSVOptions`:
+ Configuration parsed from ``resource``.
+ """
+ config = cls()
+ config._properties = copy.deepcopy(resource)
+ return config
+
+
+class GoogleSheetsOptions(object):
+ """Options that describe how to treat Google Sheets as BigQuery tables."""
+
+ _SOURCE_FORMAT = "GOOGLE_SHEETS"
+ _RESOURCE_NAME = "googleSheetsOptions"
+
+ def __init__(self):
+ self._properties = {}
+
+ @property
+ def skip_leading_rows(self):
+ """int: The number of rows at the top of a sheet that BigQuery will
+ skip when reading the data.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.(key).googleSheetsOptions.skipLeadingRows
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.googleSheetsOptions.skipLeadingRows
+ """
+ return _int_or_none(self._properties.get("skipLeadingRows"))
+
+ @skip_leading_rows.setter
+ def skip_leading_rows(self, value):
+ self._properties["skipLeadingRows"] = str(value)
+
+ def to_api_repr(self):
+ """Build an API representation of this object.
+
+ Returns:
+ Dict[str, Any]:
+ A dictionary in the format used by the BigQuery API.
+ """
+ return copy.deepcopy(self._properties)
+
+ @classmethod
+ def from_api_repr(cls, resource):
+ """Factory: construct a :class:`~.external_config.GoogleSheetsOptions`
+ instance given its API representation.
+
+ Args:
+ resource (Dict[str, Any]):
+ Definition of a :class:`~.external_config.GoogleSheetsOptions`
+ instance in the same representation as is returned from the
+ API.
+
+ Returns:
+ :class:`~.external_config.GoogleSheetsOptions`:
+ Configuration parsed from ``resource``.
+ """
+ config = cls()
+ config._properties = copy.deepcopy(resource)
+ return config
+
+
+_OPTION_CLASSES = (BigtableOptions, CSVOptions, GoogleSheetsOptions)
+
+
+class ExternalConfig(object):
+ """Description of an external data source.
+
+ Args:
+ source_format (:class:`~.external_config.ExternalSourceFormat`):
+ See :attr:`source_format`.
+ """
+
+ def __init__(self, source_format):
+ self._properties = {"sourceFormat": source_format}
+ self._options = None
+ for optcls in _OPTION_CLASSES:
+ if source_format == optcls._SOURCE_FORMAT:
+ self._options = optcls()
+ break
+
+ @property
+ def source_format(self):
+ """:class:`~.external_config.ExternalSourceFormat`:
+ Format of external source.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.sourceFormat
+ """
+ return self._properties["sourceFormat"]
+
+ @property
+ def options(self):
+ """Dict[str, Any]: Source-specific options."""
+ return self._options
+
+ @property
+ def autodetect(self):
+ """bool: If :data:`True`, try to detect schema and format options
+ automatically.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.(key).autodetect
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.autodetect
+ """
+ return self._properties.get("autodetect")
+
+ @autodetect.setter
+ def autodetect(self, value):
+ self._properties["autodetect"] = value
+
+ @property
+ def compression(self):
+ """str: The compression type of the data source.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.(key).compression
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.compression
+ """
+ return self._properties.get("compression")
+
+ @compression.setter
+ def compression(self, value):
+ self._properties["compression"] = value
+
+ @property
+ def ignore_unknown_values(self):
+ """bool: If :data:`True`, extra values that are not represented in the
+ table schema are ignored. Defaults to :data:`False`.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.(key).ignoreUnknownValues
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.ignoreUnknownValues
+ """
+ return self._properties.get("ignoreUnknownValues")
+
+ @ignore_unknown_values.setter
+ def ignore_unknown_values(self, value):
+ self._properties["ignoreUnknownValues"] = value
+
+ @property
+ def max_bad_records(self):
+ """int: The maximum number of bad records that BigQuery can ignore when
+ reading data.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.(key).maxBadRecords
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.maxBadRecords
+ """
+ return self._properties.get("maxBadRecords")
+
+ @max_bad_records.setter
+ def max_bad_records(self, value):
+ self._properties["maxBadRecords"] = value
+
+ @property
+ def source_uris(self):
+ """List[str]: URIs that point to your data in Google Cloud.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.(key).sourceUris
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.sourceUris
+ """
+ return self._properties.get("sourceUris", [])
+
+ @source_uris.setter
+ def source_uris(self, value):
+ self._properties["sourceUris"] = value
+
+ @property
+ def schema(self):
+ """List[:class:`~google.cloud.bigquery.schema.SchemaField`]: The schema
+ for the data.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.(key).schema
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.schema
+ """
+ prop = self._properties.get("schema", {})
+ return [SchemaField.from_api_repr(field) for field in prop.get("fields", [])]
+
+ @schema.setter
+ def schema(self, value):
+ prop = value
+ if value is not None:
+ prop = {"fields": [field.to_api_repr() for field in value]}
+ self._properties["schema"] = prop
+
+ def to_api_repr(self):
+ """Build an API representation of this object.
+
+ Returns:
+ Dict[str, Any]:
+ A dictionary in the format used by the BigQuery API.
+ """
+ config = copy.deepcopy(self._properties)
+ if self.options is not None:
+ r = self.options.to_api_repr()
+ if r != {}:
+ config[self.options._RESOURCE_NAME] = r
+ return config
+
+ @classmethod
+ def from_api_repr(cls, resource):
+ """Factory: construct an :class:`~.external_config.ExternalConfig`
+ instance given its API representation.
+
+ Args:
+ resource (Dict[str, Any]):
+ Definition of an :class:`~.external_config.ExternalConfig`
+ instance in the same representation as is returned from the
+ API.
+
+ Returns:
+ :class:`~.external_config.ExternalConfig`:
+ Configuration parsed from ``resource``.
+ """
+ config = cls(resource["sourceFormat"])
+ for optcls in _OPTION_CLASSES:
+ opts = resource.get(optcls._RESOURCE_NAME)
+ if opts is not None:
+ config._options = optcls.from_api_repr(opts)
+ break
+ config._properties = copy.deepcopy(resource)
+ return config
diff --git a/bigquery/google/cloud/bigquery/job.py b/bigquery/google/cloud/bigquery/job.py
new file mode 100644
index 000000000000..6e1eb81648f5
--- /dev/null
+++ b/bigquery/google/cloud/bigquery/job.py
@@ -0,0 +1,3427 @@
+# Copyright 2015 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Define API Jobs."""
+
+import copy
+import re
+import threading
+
+import six
+from six.moves import http_client
+
+import google.api_core.future.polling
+from google.cloud import exceptions
+from google.cloud.exceptions import NotFound
+from google.cloud.bigquery.dataset import Dataset
+from google.cloud.bigquery.dataset import DatasetListItem
+from google.cloud.bigquery.dataset import DatasetReference
+from google.cloud.bigquery.external_config import ExternalConfig
+from google.cloud.bigquery.query import _query_param_from_api_repr
+from google.cloud.bigquery.query import ArrayQueryParameter
+from google.cloud.bigquery.query import ScalarQueryParameter
+from google.cloud.bigquery.query import StructQueryParameter
+from google.cloud.bigquery.query import UDFResource
+from google.cloud.bigquery.retry import DEFAULT_RETRY
+from google.cloud.bigquery.routine import RoutineReference
+from google.cloud.bigquery.schema import SchemaField
+from google.cloud.bigquery.table import _EmptyRowIterator
+from google.cloud.bigquery.table import EncryptionConfiguration
+from google.cloud.bigquery.table import _table_arg_to_table_ref
+from google.cloud.bigquery.table import TableReference
+from google.cloud.bigquery.table import Table
+from google.cloud.bigquery.table import TimePartitioning
+from google.cloud.bigquery import _helpers
+
+_DONE_STATE = "DONE"
+_STOPPED_REASON = "stopped"
+_TIMEOUT_BUFFER_SECS = 0.1
+_CONTAINS_ORDER_BY = re.compile(r"ORDER\s+BY", re.IGNORECASE)
+
+_ERROR_REASON_TO_EXCEPTION = {
+ "accessDenied": http_client.FORBIDDEN,
+ "backendError": http_client.INTERNAL_SERVER_ERROR,
+ "billingNotEnabled": http_client.FORBIDDEN,
+ "billingTierLimitExceeded": http_client.BAD_REQUEST,
+ "blocked": http_client.FORBIDDEN,
+ "duplicate": http_client.CONFLICT,
+ "internalError": http_client.INTERNAL_SERVER_ERROR,
+ "invalid": http_client.BAD_REQUEST,
+ "invalidQuery": http_client.BAD_REQUEST,
+ "notFound": http_client.NOT_FOUND,
+ "notImplemented": http_client.NOT_IMPLEMENTED,
+ "quotaExceeded": http_client.FORBIDDEN,
+ "rateLimitExceeded": http_client.FORBIDDEN,
+ "resourceInUse": http_client.BAD_REQUEST,
+ "resourcesExceeded": http_client.BAD_REQUEST,
+ "responseTooLarge": http_client.FORBIDDEN,
+ "stopped": http_client.OK,
+ "tableUnavailable": http_client.BAD_REQUEST,
+}
+
+
+def _error_result_to_exception(error_result):
+ """Maps BigQuery error reasons to an exception.
+
+ The reasons and their matching HTTP status codes are documented on
+ the `troubleshooting errors`_ page.
+
+ .. _troubleshooting errors: https://cloud.google.com/bigquery\
+ /troubleshooting-errors
+
+ :type error_result: Mapping[str, str]
+ :param error_result: The error result from BigQuery.
+
+ :rtype google.cloud.exceptions.GoogleCloudError:
+ :returns: The mapped exception.
+ """
+ reason = error_result.get("reason")
+ status_code = _ERROR_REASON_TO_EXCEPTION.get(
+ reason, http_client.INTERNAL_SERVER_ERROR
+ )
+ return exceptions.from_http_status(
+ status_code, error_result.get("message", ""), errors=[error_result]
+ )
+
+
+def _contains_order_by(query):
+ """Do we need to preserve the order of the query results?
+
+ This function has known false positives, such as with ordered window
+ functions:
+
+ .. code-block:: sql
+
+ SELECT SUM(x) OVER (
+ window_name
+ PARTITION BY...
+ ORDER BY...
+ window_frame_clause)
+ FROM ...
+
+ This false positive failure case means the behavior will be correct, but
+ downloading results with the BigQuery Storage API may be slower than it
+ otherwise would. This is preferable to the false negative case, where
+ results are expected to be in order but are not (due to parallel reads).
+ """
+ return query and _CONTAINS_ORDER_BY.search(query)
+
+
+class Compression(object):
+ """The compression type to use for exported files. The default value is
+ :attr:`NONE`.
+
+ :attr:`DEFLATE` and :attr:`SNAPPY` are
+ only supported for Avro.
+ """
+
+ GZIP = "GZIP"
+ """Specifies GZIP format."""
+
+ DEFLATE = "DEFLATE"
+ """Specifies DEFLATE format."""
+
+ SNAPPY = "SNAPPY"
+ """Specifies SNAPPY format."""
+
+ NONE = "NONE"
+ """Specifies no compression."""
+
+
+class CreateDisposition(object):
+ """Specifies whether the job is allowed to create new tables. The default
+ value is :attr:`CREATE_IF_NEEDED`.
+
+ Creation, truncation and append actions occur as one atomic update
+ upon job completion.
+ """
+
+ CREATE_IF_NEEDED = "CREATE_IF_NEEDED"
+ """If the table does not exist, BigQuery creates the table."""
+
+ CREATE_NEVER = "CREATE_NEVER"
+ """The table must already exist. If it does not, a 'notFound' error is
+ returned in the job result."""
+
+
+class DestinationFormat(object):
+ """The exported file format. The default value is :attr:`CSV`.
+
+ Tables with nested or repeated fields cannot be exported as CSV.
+ """
+
+ CSV = "CSV"
+ """Specifies CSV format."""
+
+ NEWLINE_DELIMITED_JSON = "NEWLINE_DELIMITED_JSON"
+ """Specifies newline delimited JSON format."""
+
+ AVRO = "AVRO"
+ """Specifies Avro format."""
+
+
+class Encoding(object):
+ """The character encoding of the data. The default is :attr:`UTF_8`.
+
+ BigQuery decodes the data after the raw, binary data has been
+ split using the values of the quote and fieldDelimiter properties.
+ """
+
+ UTF_8 = "UTF-8"
+ """Specifies UTF-8 encoding."""
+
+ ISO_8859_1 = "ISO-8859-1"
+ """Specifies ISO-8859-1 encoding."""
+
+
+class QueryPriority(object):
+ """Specifies a priority for the query. The default value is
+ :attr:`INTERACTIVE`.
+ """
+
+ INTERACTIVE = "INTERACTIVE"
+ """Specifies interactive priority."""
+
+ BATCH = "BATCH"
+ """Specifies batch priority."""
+
+
+class SourceFormat(object):
+ """The format of the data files. The default value is :attr:`CSV`.
+
+ Note that the set of allowed values for loading data is different
+ than the set used for external data sources (see
+ :class:`~google.cloud.bigquery.external_config.ExternalSourceFormat`).
+ """
+
+ CSV = "CSV"
+ """Specifies CSV format."""
+
+ DATASTORE_BACKUP = "DATASTORE_BACKUP"
+ """Specifies datastore backup format"""
+
+ NEWLINE_DELIMITED_JSON = "NEWLINE_DELIMITED_JSON"
+ """Specifies newline delimited JSON format."""
+
+ AVRO = "AVRO"
+ """Specifies Avro format."""
+
+ PARQUET = "PARQUET"
+ """Specifies Parquet format."""
+
+ ORC = "ORC"
+ """Specifies Orc format."""
+
+
+class WriteDisposition(object):
+ """Specifies the action that occurs if destination table already exists.
+
+ The default value is :attr:`WRITE_APPEND`.
+
+ Each action is atomic and only occurs if BigQuery is able to complete
+ the job successfully. Creation, truncation and append actions occur as one
+ atomic update upon job completion.
+ """
+
+ WRITE_APPEND = "WRITE_APPEND"
+ """If the table already exists, BigQuery appends the data to the table."""
+
+ WRITE_TRUNCATE = "WRITE_TRUNCATE"
+ """If the table already exists, BigQuery overwrites the table data."""
+
+ WRITE_EMPTY = "WRITE_EMPTY"
+ """If the table already exists and contains data, a 'duplicate' error is
+ returned in the job result."""
+
+
+class SchemaUpdateOption(object):
+ """Specifies an update to the destination table schema as a side effect of
+ a load job.
+ """
+
+ ALLOW_FIELD_ADDITION = "ALLOW_FIELD_ADDITION"
+ """Allow adding a nullable field to the schema."""
+
+ ALLOW_FIELD_RELAXATION = "ALLOW_FIELD_RELAXATION"
+ """Allow relaxing a required field in the original schema to nullable."""
+
+
+class _JobReference(object):
+ """A reference to a job.
+
+ Arguments:
+ job_id (str): ID of the job to run.
+ project (str): ID of the project where the job runs.
+ location (str): Location of where the job runs.
+ """
+
+ def __init__(self, job_id, project, location):
+ self._properties = {"jobId": job_id, "projectId": project}
+ # The location field must not be populated if it is None.
+ if location:
+ self._properties["location"] = location
+
+ @property
+ def job_id(self):
+ """str: ID of the job."""
+ return self._properties.get("jobId")
+
+ @property
+ def project(self):
+ """str: ID of the project where the job runs."""
+ return self._properties.get("projectId")
+
+ @property
+ def location(self):
+ """str: Location where the job runs."""
+ return self._properties.get("location")
+
+ def _to_api_repr(self):
+ """Returns the API resource representation of the job reference."""
+ return copy.deepcopy(self._properties)
+
+ @classmethod
+ def _from_api_repr(cls, resource):
+ """Returns a job reference for an API resource representation."""
+ job_id = resource.get("jobId")
+ project = resource.get("projectId")
+ location = resource.get("location")
+ job_ref = cls(job_id, project, location)
+ return job_ref
+
+
+class _AsyncJob(google.api_core.future.polling.PollingFuture):
+ """Base class for asynchronous jobs.
+
+ Arguments:
+ job_id (Union[str, _JobReference]):
+ Job's ID in the project associated with the client or a
+ fully-qualified job reference.
+ client (google.cloud.bigquery.client.Client):
+ Client which holds credentials and project configuration.
+ """
+
+ def __init__(self, job_id, client):
+ super(_AsyncJob, self).__init__()
+
+ # The job reference can be either a plain job ID or the full resource.
+ # Populate the properties dictionary consistently depending on what has
+ # been passed in.
+ job_ref = job_id
+ if not isinstance(job_id, _JobReference):
+ job_ref = _JobReference(job_id, client.project, None)
+ self._properties = {"jobReference": job_ref._to_api_repr()}
+
+ self._client = client
+ self._result_set = False
+ self._completion_lock = threading.Lock()
+
+ @property
+ def job_id(self):
+ """str: ID of the job."""
+ return _helpers._get_sub_prop(self._properties, ["jobReference", "jobId"])
+
+ @property
+ def project(self):
+ """Project bound to the job.
+
+ :rtype: str
+ :returns: the project (derived from the client).
+ """
+ return _helpers._get_sub_prop(self._properties, ["jobReference", "projectId"])
+
+ @property
+ def location(self):
+ """str: Location where the job runs."""
+ return _helpers._get_sub_prop(self._properties, ["jobReference", "location"])
+
+ def _require_client(self, client):
+ """Check client or verify over-ride.
+
+ :type client: :class:`~google.cloud.bigquery.client.Client` or
+ ``NoneType``
+ :param client: the client to use. If not passed, falls back to the
+ ``client`` stored on the current dataset.
+
+ :rtype: :class:`google.cloud.bigquery.client.Client`
+ :returns: The client passed in or the currently bound client.
+ """
+ if client is None:
+ client = self._client
+ return client
+
+ @property
+ def job_type(self):
+ """Type of job
+
+ :rtype: str
+ :returns: one of 'load', 'copy', 'extract', 'query'
+ """
+ return self._JOB_TYPE
+
+ @property
+ def path(self):
+ """URL path for the job's APIs.
+
+ :rtype: str
+ :returns: the path based on project and job ID.
+ """
+ return "/projects/%s/jobs/%s" % (self.project, self.job_id)
+
+ @property
+ def labels(self):
+ """Dict[str, str]: Labels for the job."""
+ return self._properties.setdefault("labels", {})
+
+ @property
+ def etag(self):
+ """ETag for the job resource.
+
+ :rtype: str, or ``NoneType``
+ :returns: the ETag (None until set from the server).
+ """
+ return self._properties.get("etag")
+
+ @property
+ def self_link(self):
+ """URL for the job resource.
+
+ :rtype: str, or ``NoneType``
+ :returns: the URL (None until set from the server).
+ """
+ return self._properties.get("selfLink")
+
+ @property
+ def user_email(self):
+ """E-mail address of user who submitted the job.
+
+ :rtype: str, or ``NoneType``
+ :returns: the URL (None until set from the server).
+ """
+ return self._properties.get("user_email")
+
+ @property
+ def created(self):
+ """Datetime at which the job was created.
+
+ :rtype: ``datetime.datetime``, or ``NoneType``
+ :returns: the creation time (None until set from the server).
+ """
+ statistics = self._properties.get("statistics")
+ if statistics is not None:
+ millis = statistics.get("creationTime")
+ if millis is not None:
+ return _helpers._datetime_from_microseconds(millis * 1000.0)
+
+ @property
+ def started(self):
+ """Datetime at which the job was started.
+
+ :rtype: ``datetime.datetime``, or ``NoneType``
+ :returns: the start time (None until set from the server).
+ """
+ statistics = self._properties.get("statistics")
+ if statistics is not None:
+ millis = statistics.get("startTime")
+ if millis is not None:
+ return _helpers._datetime_from_microseconds(millis * 1000.0)
+
+ @property
+ def ended(self):
+ """Datetime at which the job finished.
+
+ :rtype: ``datetime.datetime``, or ``NoneType``
+ :returns: the end time (None until set from the server).
+ """
+ statistics = self._properties.get("statistics")
+ if statistics is not None:
+ millis = statistics.get("endTime")
+ if millis is not None:
+ return _helpers._datetime_from_microseconds(millis * 1000.0)
+
+ def _job_statistics(self):
+ """Helper for job-type specific statistics-based properties."""
+ statistics = self._properties.get("statistics", {})
+ return statistics.get(self._JOB_TYPE, {})
+
+ @property
+ def error_result(self):
+ """Error information about the job as a whole.
+
+ :rtype: mapping, or ``NoneType``
+ :returns: the error information (None until set from the server).
+ """
+ status = self._properties.get("status")
+ if status is not None:
+ return status.get("errorResult")
+
+ @property
+ def errors(self):
+ """Information about individual errors generated by the job.
+
+ :rtype: list of mappings, or ``NoneType``
+ :returns: the error information (None until set from the server).
+ """
+ status = self._properties.get("status")
+ if status is not None:
+ return status.get("errors")
+
+ @property
+ def state(self):
+ """Status of the job.
+
+ :rtype: str, or ``NoneType``
+ :returns: the state (None until set from the server).
+ """
+ status = self._properties.get("status")
+ if status is not None:
+ return status.get("state")
+
+ def _scrub_local_properties(self, cleaned):
+ """Helper: handle subclass properties in cleaned."""
+ pass
+
+ def _copy_configuration_properties(self, configuration):
+ """Helper: assign subclass configuration properties in cleaned."""
+ raise NotImplementedError("Abstract")
+
+ def _set_properties(self, api_response):
+ """Update properties from resource in body of ``api_response``
+
+ :type api_response: dict
+ :param api_response: response returned from an API call
+ """
+ cleaned = api_response.copy()
+ self._scrub_local_properties(cleaned)
+
+ statistics = cleaned.get("statistics", {})
+ if "creationTime" in statistics:
+ statistics["creationTime"] = float(statistics["creationTime"])
+ if "startTime" in statistics:
+ statistics["startTime"] = float(statistics["startTime"])
+ if "endTime" in statistics:
+ statistics["endTime"] = float(statistics["endTime"])
+
+ self._properties.clear()
+ self._properties.update(cleaned)
+ self._copy_configuration_properties(cleaned.get("configuration", {}))
+
+ # For Future interface
+ self._set_future_result()
+
+ @classmethod
+ def _get_resource_config(cls, resource):
+ """Helper for :meth:`from_api_repr`
+
+ :type resource: dict
+ :param resource: resource for the job
+
+ :rtype: dict
+ :returns: tuple (string, dict), where the first element is the
+ job ID and the second contains job-specific configuration.
+ :raises: :class:`KeyError` if the resource has no identifier, or
+ is missing the appropriate configuration.
+ """
+ if "jobReference" not in resource or "jobId" not in resource["jobReference"]:
+ raise KeyError(
+ "Resource lacks required identity information: "
+ '["jobReference"]["jobId"]'
+ )
+ job_id = resource["jobReference"]["jobId"]
+ if (
+ "configuration" not in resource
+ or cls._JOB_TYPE not in resource["configuration"]
+ ):
+ raise KeyError(
+ "Resource lacks required configuration: "
+ '["configuration"]["%s"]' % cls._JOB_TYPE
+ )
+ return job_id, resource["configuration"]
+
+ def to_api_repr(self):
+ """Generate a resource for the job."""
+ raise NotImplementedError("Abstract")
+
+ _build_resource = to_api_repr # backward-compatibility alias
+
+ def _begin(self, client=None, retry=DEFAULT_RETRY):
+ """API call: begin the job via a POST request
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/insert
+
+ Args:
+ client (Optional[google.cloud.bigquery.client.Client]):
+ The client to use. If not passed, falls back to the ``client``
+ associated with the job object or``NoneType``
+ retry (Optional[google.api_core.retry.Retry]):
+ How to retry the RPC.
+
+ Raises:
+ ValueError:
+ If the job has already begun.
+ """
+ if self.state is not None:
+ raise ValueError("Job already begun.")
+
+ client = self._require_client(client)
+ path = "/projects/%s/jobs" % (self.project,)
+
+ # jobs.insert is idempotent because we ensure that every new
+ # job has an ID.
+ api_response = client._call_api(
+ retry, method="POST", path=path, data=self.to_api_repr()
+ )
+ self._set_properties(api_response)
+
+ def exists(self, client=None, retry=DEFAULT_RETRY):
+ """API call: test for the existence of the job via a GET request
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/get
+
+ :type client: :class:`~google.cloud.bigquery.client.Client` or
+ ``NoneType``
+ :param client: the client to use. If not passed, falls back to the
+ ``client`` stored on the current dataset.
+
+ :type retry: :class:`google.api_core.retry.Retry`
+ :param retry: (Optional) How to retry the RPC.
+
+ :rtype: bool
+ :returns: Boolean indicating existence of the job.
+ """
+ client = self._require_client(client)
+
+ extra_params = {"fields": "id"}
+ if self.location:
+ extra_params["location"] = self.location
+
+ try:
+ client._call_api(
+ retry, method="GET", path=self.path, query_params=extra_params
+ )
+ except NotFound:
+ return False
+ else:
+ return True
+
+ def reload(self, client=None, retry=DEFAULT_RETRY):
+ """API call: refresh job properties via a GET request.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/get
+
+ :type client: :class:`~google.cloud.bigquery.client.Client` or
+ ``NoneType``
+ :param client: the client to use. If not passed, falls back to the
+ ``client`` stored on the current dataset.
+
+ :type retry: :class:`google.api_core.retry.Retry`
+ :param retry: (Optional) How to retry the RPC.
+ """
+ client = self._require_client(client)
+
+ extra_params = {}
+ if self.location:
+ extra_params["location"] = self.location
+
+ api_response = client._call_api(
+ retry, method="GET", path=self.path, query_params=extra_params
+ )
+ self._set_properties(api_response)
+
+ def cancel(self, client=None):
+ """API call: cancel job via a POST request
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/cancel
+
+ :type client: :class:`~google.cloud.bigquery.client.Client` or
+ ``NoneType``
+ :param client: the client to use. If not passed, falls back to the
+ ``client`` stored on the current dataset.
+
+ :rtype: bool
+ :returns: Boolean indicating that the cancel request was sent.
+ """
+ client = self._require_client(client)
+
+ extra_params = {}
+ if self.location:
+ extra_params["location"] = self.location
+
+ api_response = client._connection.api_request(
+ method="POST", path="%s/cancel" % (self.path,), query_params=extra_params
+ )
+ self._set_properties(api_response["job"])
+ # The Future interface requires that we return True if the *attempt*
+ # to cancel was successful.
+ return True
+
+ # The following methods implement the PollingFuture interface. Note that
+ # the methods above are from the pre-Future interface and are left for
+ # compatibility. The only "overloaded" method is :meth:`cancel`, which
+ # satisfies both interfaces.
+
+ def _set_future_result(self):
+ """Set the result or exception from the job if it is complete."""
+ # This must be done in a lock to prevent the polling thread
+ # and main thread from both executing the completion logic
+ # at the same time.
+ with self._completion_lock:
+ # If the operation isn't complete or if the result has already been
+ # set, do not call set_result/set_exception again.
+ # Note: self._result_set is set to True in set_result and
+ # set_exception, in case those methods are invoked directly.
+ if self.state != _DONE_STATE or self._result_set:
+ return
+
+ if self.error_result is not None:
+ exception = _error_result_to_exception(self.error_result)
+ self.set_exception(exception)
+ else:
+ self.set_result(self)
+
+ def done(self, retry=DEFAULT_RETRY):
+ """Refresh the job and checks if it is complete.
+
+ :type retry: :class:`google.api_core.retry.Retry`
+ :param retry: (Optional) How to retry the RPC.
+
+ :rtype: bool
+ :returns: True if the job is complete, False otherwise.
+ """
+ # Do not refresh is the state is already done, as the job will not
+ # change once complete.
+ if self.state != _DONE_STATE:
+ self.reload(retry=retry)
+ return self.state == _DONE_STATE
+
+ def result(self, timeout=None, retry=DEFAULT_RETRY):
+ """Start the job and wait for it to complete and get the result.
+
+ :type timeout: float
+ :param timeout:
+ How long (in seconds) to wait for job to complete before raising
+ a :class:`concurrent.futures.TimeoutError`.
+
+ :type retry: :class:`google.api_core.retry.Retry`
+ :param retry: (Optional) How to retry the RPC.
+
+ :rtype: _AsyncJob
+ :returns: This instance.
+
+ :raises:
+ :class:`~google.cloud.exceptions.GoogleCloudError` if the job
+ failed or :class:`concurrent.futures.TimeoutError` if the job did
+ not complete in the given timeout.
+ """
+ if self.state is None:
+ self._begin(retry=retry)
+ # TODO: modify PollingFuture so it can pass a retry argument to done().
+ return super(_AsyncJob, self).result(timeout=timeout)
+
+ def cancelled(self):
+ """Check if the job has been cancelled.
+
+ This always returns False. It's not possible to check if a job was
+ cancelled in the API. This method is here to satisfy the interface
+ for :class:`google.api_core.future.Future`.
+
+ :rtype: bool
+ :returns: False
+ """
+ return (
+ self.error_result is not None
+ and self.error_result.get("reason") == _STOPPED_REASON
+ )
+
+
+class _JobConfig(object):
+ """Abstract base class for job configuration objects.
+
+ Arguments:
+ job_type (str): The key to use for the job configuration.
+ """
+
+ def __init__(self, job_type, **kwargs):
+ self._job_type = job_type
+ self._properties = {job_type: {}}
+ for prop, val in kwargs.items():
+ setattr(self, prop, val)
+
+ @property
+ def labels(self):
+ """Dict[str, str]: Labels for the job.
+
+ This method always returns a dict. To change a job's labels,
+ modify the dict, then call ``Client.update_job``. To delete a
+ label, set its value to :data:`None` before updating.
+
+ Raises:
+ ValueError: If ``value`` type is invalid.
+ """
+ return self._properties.setdefault("labels", {})
+
+ @labels.setter
+ def labels(self, value):
+ if not isinstance(value, dict):
+ raise ValueError("Pass a dict")
+ self._properties["labels"] = value
+
+ def _get_sub_prop(self, key, default=None):
+ """Get a value in the ``self._properties[self._job_type]`` dictionary.
+
+ Most job properties are inside the dictionary related to the job type
+ (e.g. 'copy', 'extract', 'load', 'query'). Use this method to access
+ those properties::
+
+ self._get_sub_prop('destinationTable')
+
+ This is equivalent to using the ``_helpers._get_sub_prop`` function::
+
+ _helpers._get_sub_prop(
+ self._properties, ['query', 'destinationTable'])
+
+ Arguments:
+ key (str):
+ Key for the value to get in the
+ ``self._properties[self._job_type]`` dictionary.
+ default (object):
+ (Optional) Default value to return if the key is not found.
+ Defaults to :data:`None`.
+
+ Returns:
+ object: The value if present or the default.
+ """
+ return _helpers._get_sub_prop(
+ self._properties, [self._job_type, key], default=default
+ )
+
+ def _set_sub_prop(self, key, value):
+ """Set a value in the ``self._properties[self._job_type]`` dictionary.
+
+ Most job properties are inside the dictionary related to the job type
+ (e.g. 'copy', 'extract', 'load', 'query'). Use this method to set
+ those properties::
+
+ self._set_sub_prop('useLegacySql', False)
+
+ This is equivalent to using the ``_helper._set_sub_prop`` function::
+
+ _helper._set_sub_prop(
+ self._properties, ['query', 'useLegacySql'], False)
+
+ Arguments:
+ key (str):
+ Key to set in the ``self._properties[self._job_type]``
+ dictionary.
+ value (object): Value to set.
+ """
+ _helpers._set_sub_prop(self._properties, [self._job_type, key], value)
+
+ def _del_sub_prop(self, key):
+ """Remove ``key`` from the ``self._properties[self._job_type]`` dict.
+
+ Most job properties are inside the dictionary related to the job type
+ (e.g. 'copy', 'extract', 'load', 'query'). Use this method to clear
+ those properties::
+
+ self._del_sub_prop('useLegacySql')
+
+ This is equivalent to using the ``_helper._del_sub_prop`` function::
+
+ _helper._del_sub_prop(
+ self._properties, ['query', 'useLegacySql'])
+
+ Arguments:
+ key (str):
+ Key to remove in the ``self._properties[self._job_type]``
+ dictionary.
+ """
+ _helpers._del_sub_prop(self._properties, [self._job_type, key])
+
+ def to_api_repr(self):
+ """Build an API representation of the job config.
+
+ :rtype: dict
+ :returns: A dictionary in the format used by the BigQuery API.
+ """
+ return copy.deepcopy(self._properties)
+
+ def _fill_from_default(self, default_job_config):
+ """Merge this job config with a default job config.
+
+ The keys in this object take precedence over the keys in the default
+ config. The merge is done at the top-level as well as for keys one
+ level below the job type.
+
+ Arguments:
+ default_job_config (google.cloud.bigquery.job._JobConfig):
+ The default job config that will be used to fill in self.
+
+ Returns:
+ google.cloud.bigquery.job._JobConfig A new (merged) job config.
+ """
+ if self._job_type != default_job_config._job_type:
+ raise TypeError(
+ "attempted to merge two incompatible job types: "
+ + repr(self._job_type)
+ + ", "
+ + repr(default_job_config._job_type)
+ )
+
+ new_job_config = self.__class__()
+
+ default_job_properties = copy.deepcopy(default_job_config._properties)
+ for key in self._properties:
+ if key != self._job_type:
+ default_job_properties[key] = self._properties[key]
+
+ default_job_properties[self._job_type].update(self._properties[self._job_type])
+ new_job_config._properties = default_job_properties
+
+ return new_job_config
+
+ @classmethod
+ def from_api_repr(cls, resource):
+ """Factory: construct a job configuration given its API representation
+
+ :type resource: dict
+ :param resource:
+ An extract job configuration in the same representation as is
+ returned from the API.
+
+ :rtype: :class:`google.cloud.bigquery.job._JobConfig`
+ :returns: Configuration parsed from ``resource``.
+ """
+ config = cls()
+ config._properties = copy.deepcopy(resource)
+ return config
+
+
+class LoadJobConfig(_JobConfig):
+ """Configuration options for load jobs.
+
+ All properties in this class are optional. Values which are :data:`None` ->
+ server defaults. Set properties on the constructed configuration by using
+ the property name as the name of a keyword argument.
+ """
+
+ def __init__(self, **kwargs):
+ super(LoadJobConfig, self).__init__("load", **kwargs)
+
+ @property
+ def allow_jagged_rows(self):
+ """bool: Allow missing trailing optional columns (CSV only).
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.allowJaggedRows
+ """
+ return self._get_sub_prop("allowJaggedRows")
+
+ @allow_jagged_rows.setter
+ def allow_jagged_rows(self, value):
+ self._set_sub_prop("allowJaggedRows", value)
+
+ @property
+ def allow_quoted_newlines(self):
+ """bool: Allow quoted data containing newline characters (CSV only).
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.allowQuotedNewlines
+ """
+ return self._get_sub_prop("allowQuotedNewlines")
+
+ @allow_quoted_newlines.setter
+ def allow_quoted_newlines(self, value):
+ self._set_sub_prop("allowQuotedNewlines", value)
+
+ @property
+ def autodetect(self):
+ """bool: Automatically infer the schema from a sample of the data.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.autodetect
+ """
+ return self._get_sub_prop("autodetect")
+
+ @autodetect.setter
+ def autodetect(self, value):
+ self._set_sub_prop("autodetect", value)
+
+ @property
+ def clustering_fields(self):
+ """Union[List[str], None]: Fields defining clustering for the table
+
+ (Defaults to :data:`None`).
+
+ Clustering fields are immutable after table creation.
+
+ .. note::
+
+ As of 2018-06-29, clustering fields cannot be set on a table
+ which does not also have time partioning defined.
+ """
+ prop = self._get_sub_prop("clustering")
+ if prop is not None:
+ return list(prop.get("fields", ()))
+
+ @clustering_fields.setter
+ def clustering_fields(self, value):
+ """Union[List[str], None]: Fields defining clustering for the table
+
+ (Defaults to :data:`None`).
+ """
+ if value is not None:
+ self._set_sub_prop("clustering", {"fields": value})
+ else:
+ self._del_sub_prop("clustering")
+
+ @property
+ def create_disposition(self):
+ """google.cloud.bigquery.job.CreateDisposition: Specifies behavior
+ for creating tables.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.createDisposition
+ """
+ return self._get_sub_prop("createDisposition")
+
+ @create_disposition.setter
+ def create_disposition(self, value):
+ self._set_sub_prop("createDisposition", value)
+
+ @property
+ def destination_encryption_configuration(self):
+ """google.cloud.bigquery.table.EncryptionConfiguration: Custom
+ encryption configuration for the destination table.
+
+ Custom encryption configuration (e.g., Cloud KMS keys) or :data:`None`
+ if using default encryption.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.destinationEncryptionConfiguration
+ """
+ prop = self._get_sub_prop("destinationEncryptionConfiguration")
+ if prop is not None:
+ prop = EncryptionConfiguration.from_api_repr(prop)
+ return prop
+
+ @destination_encryption_configuration.setter
+ def destination_encryption_configuration(self, value):
+ api_repr = value
+ if value is not None:
+ api_repr = value.to_api_repr()
+ self._set_sub_prop("destinationEncryptionConfiguration", api_repr)
+ else:
+ self._del_sub_prop("destinationEncryptionConfiguration")
+
+ @property
+ def destination_table_description(self):
+ """Union[str, None] name given to destination table.
+
+ See:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.destinationTableProperties.description
+ """
+ prop = self._get_sub_prop("destinationTableProperties")
+ if prop is not None:
+ return prop["description"]
+
+ @destination_table_description.setter
+ def destination_table_description(self, value):
+ keys = [self._job_type, "destinationTableProperties", "description"]
+ if value is not None:
+ _helpers._set_sub_prop(self._properties, keys, value)
+ else:
+ _helpers._del_sub_prop(self._properties, keys)
+
+ @property
+ def destination_table_friendly_name(self):
+ """Union[str, None] name given to destination table.
+
+ See:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.destinationTableProperties.friendlyName
+ """
+ prop = self._get_sub_prop("destinationTableProperties")
+ if prop is not None:
+ return prop["friendlyName"]
+
+ @destination_table_friendly_name.setter
+ def destination_table_friendly_name(self, value):
+ keys = [self._job_type, "destinationTableProperties", "friendlyName"]
+ if value is not None:
+ _helpers._set_sub_prop(self._properties, keys, value)
+ else:
+ _helpers._del_sub_prop(self._properties, keys)
+
+ @property
+ def encoding(self):
+ """google.cloud.bigquery.job.Encoding: The character encoding of the
+ data.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.encoding
+ """
+ return self._get_sub_prop("encoding")
+
+ @encoding.setter
+ def encoding(self, value):
+ self._set_sub_prop("encoding", value)
+
+ @property
+ def field_delimiter(self):
+ """str: The separator for fields in a CSV file.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.fieldDelimiter
+ """
+ return self._get_sub_prop("fieldDelimiter")
+
+ @field_delimiter.setter
+ def field_delimiter(self, value):
+ self._set_sub_prop("fieldDelimiter", value)
+
+ @property
+ def ignore_unknown_values(self):
+ """bool: Ignore extra values not represented in the table schema.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.ignoreUnknownValues
+ """
+ return self._get_sub_prop("ignoreUnknownValues")
+
+ @ignore_unknown_values.setter
+ def ignore_unknown_values(self, value):
+ self._set_sub_prop("ignoreUnknownValues", value)
+
+ @property
+ def max_bad_records(self):
+ """int: Number of invalid rows to ignore.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.maxBadRecords
+ """
+ return _helpers._int_or_none(self._get_sub_prop("maxBadRecords"))
+
+ @max_bad_records.setter
+ def max_bad_records(self, value):
+ self._set_sub_prop("maxBadRecords", value)
+
+ @property
+ def null_marker(self):
+ """str: Represents a null value (CSV only).
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.nullMarker
+ """
+ return self._get_sub_prop("nullMarker")
+
+ @null_marker.setter
+ def null_marker(self, value):
+ self._set_sub_prop("nullMarker", value)
+
+ @property
+ def quote_character(self):
+ """str: Character used to quote data sections (CSV only).
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.quote
+ """
+ return self._get_sub_prop("quote")
+
+ @quote_character.setter
+ def quote_character(self, value):
+ self._set_sub_prop("quote", value)
+
+ @property
+ def schema(self):
+ """List[google.cloud.bigquery.schema.SchemaField]: Schema of the
+ destination table.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.schema
+ """
+ schema = _helpers._get_sub_prop(self._properties, ["load", "schema", "fields"])
+ if schema is None:
+ return
+ return [SchemaField.from_api_repr(field) for field in schema]
+
+ @schema.setter
+ def schema(self, value):
+ if value is None:
+ self._del_sub_prop("schema")
+ return
+
+ if not all(hasattr(field, "to_api_repr") for field in value):
+ raise ValueError("Schema items must be fields")
+ _helpers._set_sub_prop(
+ self._properties,
+ ["load", "schema", "fields"],
+ [field.to_api_repr() for field in value],
+ )
+
+ @property
+ def schema_update_options(self):
+ """List[google.cloud.bigquery.job.SchemaUpdateOption]: Specifies
+ updates to the destination table schema to allow as a side effect of
+ the load job.
+ """
+ return self._get_sub_prop("schemaUpdateOptions")
+
+ @schema_update_options.setter
+ def schema_update_options(self, values):
+ self._set_sub_prop("schemaUpdateOptions", values)
+
+ @property
+ def skip_leading_rows(self):
+ """int: Number of rows to skip when reading data (CSV only).
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.skipLeadingRows
+ """
+ return _helpers._int_or_none(self._get_sub_prop("skipLeadingRows"))
+
+ @skip_leading_rows.setter
+ def skip_leading_rows(self, value):
+ self._set_sub_prop("skipLeadingRows", str(value))
+
+ @property
+ def source_format(self):
+ """google.cloud.bigquery.job.SourceFormat: File format of the data.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.sourceFormat
+ """
+ return self._get_sub_prop("sourceFormat")
+
+ @source_format.setter
+ def source_format(self, value):
+ self._set_sub_prop("sourceFormat", value)
+
+ @property
+ def time_partitioning(self):
+ """google.cloud.bigquery.table.TimePartitioning: Specifies time-based
+ partitioning for the destination table.
+ """
+ prop = self._get_sub_prop("timePartitioning")
+ if prop is not None:
+ prop = TimePartitioning.from_api_repr(prop)
+ return prop
+
+ @time_partitioning.setter
+ def time_partitioning(self, value):
+ api_repr = value
+ if value is not None:
+ api_repr = value.to_api_repr()
+ self._set_sub_prop("timePartitioning", api_repr)
+ else:
+ self._del_sub_prop("timePartitioning")
+
+ @property
+ def use_avro_logical_types(self):
+ """bool: For loads of Avro data, governs whether Avro logical types are
+ converted to their corresponding BigQuery types(e.g. TIMESTAMP) rather than
+ raw types (e.g. INTEGER).
+ """
+ return self._get_sub_prop("useAvroLogicalTypes")
+
+ @use_avro_logical_types.setter
+ def use_avro_logical_types(self, value):
+ self._set_sub_prop("useAvroLogicalTypes", bool(value))
+
+ @property
+ def write_disposition(self):
+ """google.cloud.bigquery.job.WriteDisposition: Action that occurs if
+ the destination table already exists.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.writeDisposition
+ """
+ return self._get_sub_prop("writeDisposition")
+
+ @write_disposition.setter
+ def write_disposition(self, value):
+ self._set_sub_prop("writeDisposition", value)
+
+
+class LoadJob(_AsyncJob):
+ """Asynchronous job for loading data into a table.
+
+ Can load from Google Cloud Storage URIs or from a file.
+
+ :type job_id: str
+ :param job_id: the job's ID
+
+ :type source_uris: sequence of string or ``NoneType``
+ :param source_uris:
+ URIs of one or more data files to be loaded. See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.sourceUris
+ for supported URI formats. Pass None for jobs that load from a file.
+
+ :type destination: :class:`google.cloud.bigquery.table.TableReference`
+ :param destination: reference to table into which data is to be loaded.
+
+ :type client: :class:`google.cloud.bigquery.client.Client`
+ :param client: A client which holds credentials and project configuration
+ for the dataset (which requires a project).
+ """
+
+ _JOB_TYPE = "load"
+
+ def __init__(self, job_id, source_uris, destination, client, job_config=None):
+ super(LoadJob, self).__init__(job_id, client)
+
+ if job_config is None:
+ job_config = LoadJobConfig()
+
+ self.source_uris = source_uris
+ self._destination = destination
+ self._configuration = job_config
+
+ @property
+ def destination(self):
+ """google.cloud.bigquery.table.TableReference: table where loaded rows are written
+
+ See:
+ https://g.co/cloud/bigquery/docs/reference/rest/v2/jobs#configuration.load.destinationTable
+ """
+ return self._destination
+
+ @property
+ def allow_jagged_rows(self):
+ """See
+ :attr:`google.cloud.bigquery.job.LoadJobConfig.allow_jagged_rows`.
+ """
+ return self._configuration.allow_jagged_rows
+
+ @property
+ def allow_quoted_newlines(self):
+ """See
+ :attr:`google.cloud.bigquery.job.LoadJobConfig.allow_quoted_newlines`.
+ """
+ return self._configuration.allow_quoted_newlines
+
+ @property
+ def autodetect(self):
+ """See
+ :attr:`google.cloud.bigquery.job.LoadJobConfig.autodetect`.
+ """
+ return self._configuration.autodetect
+
+ @property
+ def create_disposition(self):
+ """See
+ :attr:`google.cloud.bigquery.job.LoadJobConfig.create_disposition`.
+ """
+ return self._configuration.create_disposition
+
+ @property
+ def encoding(self):
+ """See
+ :attr:`google.cloud.bigquery.job.LoadJobConfig.encoding`.
+ """
+ return self._configuration.encoding
+
+ @property
+ def field_delimiter(self):
+ """See
+ :attr:`google.cloud.bigquery.job.LoadJobConfig.field_delimiter`.
+ """
+ return self._configuration.field_delimiter
+
+ @property
+ def ignore_unknown_values(self):
+ """See
+ :attr:`google.cloud.bigquery.job.LoadJobConfig.ignore_unknown_values`.
+ """
+ return self._configuration.ignore_unknown_values
+
+ @property
+ def max_bad_records(self):
+ """See
+ :attr:`google.cloud.bigquery.job.LoadJobConfig.max_bad_records`.
+ """
+ return self._configuration.max_bad_records
+
+ @property
+ def null_marker(self):
+ """See
+ :attr:`google.cloud.bigquery.job.LoadJobConfig.null_marker`.
+ """
+ return self._configuration.null_marker
+
+ @property
+ def quote_character(self):
+ """See
+ :attr:`google.cloud.bigquery.job.LoadJobConfig.quote_character`.
+ """
+ return self._configuration.quote_character
+
+ @property
+ def skip_leading_rows(self):
+ """See
+ :attr:`google.cloud.bigquery.job.LoadJobConfig.skip_leading_rows`.
+ """
+ return self._configuration.skip_leading_rows
+
+ @property
+ def source_format(self):
+ """See
+ :attr:`google.cloud.bigquery.job.LoadJobConfig.source_format`.
+ """
+ return self._configuration.source_format
+
+ @property
+ def write_disposition(self):
+ """See
+ :attr:`google.cloud.bigquery.job.LoadJobConfig.write_disposition`.
+ """
+ return self._configuration.write_disposition
+
+ @property
+ def schema(self):
+ """See
+ :attr:`google.cloud.bigquery.job.LoadJobConfig.schema`.
+ """
+ return self._configuration.schema
+
+ @property
+ def destination_encryption_configuration(self):
+ """google.cloud.bigquery.table.EncryptionConfiguration: Custom
+ encryption configuration for the destination table.
+
+ Custom encryption configuration (e.g., Cloud KMS keys)
+ or :data:`None` if using default encryption.
+
+ See
+ :attr:`google.cloud.bigquery.job.LoadJobConfig.destination_encryption_configuration`.
+ """
+ return self._configuration.destination_encryption_configuration
+
+ @property
+ def destination_table_description(self):
+ """Union[str, None] name given to destination table.
+
+ See:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.destinationTableProperties.description
+ """
+ return self._configuration.destination_table_description
+
+ @property
+ def destination_table_friendly_name(self):
+ """Union[str, None] name given to destination table.
+
+ See:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.destinationTableProperties.friendlyName
+ """
+ return self._configuration.destination_table_friendly_name
+
+ @property
+ def time_partitioning(self):
+ """See
+ :attr:`google.cloud.bigquery.job.LoadJobConfig.time_partitioning`.
+ """
+ return self._configuration.time_partitioning
+
+ @property
+ def use_avro_logical_types(self):
+ """See
+ :attr:`google.cloud.bigquery.job.LoadJobConfig.use_avro_logical_types`.
+ """
+ return self._configuration.use_avro_logical_types
+
+ @property
+ def clustering_fields(self):
+ """See
+ :attr:`google.cloud.bigquery.job.LoadJobConfig.clustering_fields`.
+ """
+ return self._configuration.clustering_fields
+
+ @property
+ def schema_update_options(self):
+ """See
+ :attr:`google.cloud.bigquery.job.LoadJobConfig.schema_update_options`.
+ """
+ return self._configuration.schema_update_options
+
+ @property
+ def input_file_bytes(self):
+ """Count of bytes loaded from source files.
+
+ :rtype: int, or ``NoneType``
+ :returns: the count (None until set from the server).
+ :raises: ValueError for invalid value types.
+ """
+ return _helpers._int_or_none(
+ _helpers._get_sub_prop(
+ self._properties, ["statistics", "load", "inputFileBytes"]
+ )
+ )
+
+ @property
+ def input_files(self):
+ """Count of source files.
+
+ :rtype: int, or ``NoneType``
+ :returns: the count (None until set from the server).
+ """
+ return _helpers._int_or_none(
+ _helpers._get_sub_prop(
+ self._properties, ["statistics", "load", "inputFiles"]
+ )
+ )
+
+ @property
+ def output_bytes(self):
+ """Count of bytes saved to destination table.
+
+ :rtype: int, or ``NoneType``
+ :returns: the count (None until set from the server).
+ """
+ return _helpers._int_or_none(
+ _helpers._get_sub_prop(
+ self._properties, ["statistics", "load", "outputBytes"]
+ )
+ )
+
+ @property
+ def output_rows(self):
+ """Count of rows saved to destination table.
+
+ :rtype: int, or ``NoneType``
+ :returns: the count (None until set from the server).
+ """
+ return _helpers._int_or_none(
+ _helpers._get_sub_prop(
+ self._properties, ["statistics", "load", "outputRows"]
+ )
+ )
+
+ def to_api_repr(self):
+ """Generate a resource for :meth:`_begin`."""
+ configuration = self._configuration.to_api_repr()
+ if self.source_uris is not None:
+ _helpers._set_sub_prop(
+ configuration, ["load", "sourceUris"], self.source_uris
+ )
+ _helpers._set_sub_prop(
+ configuration, ["load", "destinationTable"], self.destination.to_api_repr()
+ )
+
+ return {
+ "jobReference": self._properties["jobReference"],
+ "configuration": configuration,
+ }
+
+ def _copy_configuration_properties(self, configuration):
+ """Helper: assign subclass configuration properties in cleaned."""
+ self._configuration._properties = copy.deepcopy(configuration)
+
+ @classmethod
+ def from_api_repr(cls, resource, client):
+ """Factory: construct a job given its API representation
+
+ .. note:
+
+ This method assumes that the project found in the resource matches
+ the client's project.
+
+ :type resource: dict
+ :param resource: dataset job representation returned from the API
+
+ :type client: :class:`google.cloud.bigquery.client.Client`
+ :param client: Client which holds credentials and project
+ configuration for the dataset.
+
+ :rtype: :class:`google.cloud.bigquery.job.LoadJob`
+ :returns: Job parsed from ``resource``.
+ """
+ config_resource = resource.get("configuration", {})
+ config = LoadJobConfig.from_api_repr(config_resource)
+ # A load job requires a destination table.
+ dest_config = config_resource["load"]["destinationTable"]
+ ds_ref = DatasetReference(dest_config["projectId"], dest_config["datasetId"])
+ destination = TableReference(ds_ref, dest_config["tableId"])
+ # sourceUris will be absent if this is a file upload.
+ source_uris = _helpers._get_sub_prop(config_resource, ["load", "sourceUris"])
+ job_ref = _JobReference._from_api_repr(resource["jobReference"])
+ job = cls(job_ref, source_uris, destination, client, config)
+ job._set_properties(resource)
+ return job
+
+
+class CopyJobConfig(_JobConfig):
+ """Configuration options for copy jobs.
+
+ All properties in this class are optional. Values which are :data:`None` ->
+ server defaults. Set properties on the constructed configuration by using
+ the property name as the name of a keyword argument.
+ """
+
+ def __init__(self, **kwargs):
+ super(CopyJobConfig, self).__init__("copy", **kwargs)
+
+ @property
+ def create_disposition(self):
+ """google.cloud.bigquery.job.CreateDisposition: Specifies behavior
+ for creating tables.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.copy.createDisposition
+ """
+ return self._get_sub_prop("createDisposition")
+
+ @create_disposition.setter
+ def create_disposition(self, value):
+ self._set_sub_prop("createDisposition", value)
+
+ @property
+ def write_disposition(self):
+ """google.cloud.bigquery.job.WriteDisposition: Action that occurs if
+ the destination table already exists.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.copy.writeDisposition
+ """
+ return self._get_sub_prop("writeDisposition")
+
+ @write_disposition.setter
+ def write_disposition(self, value):
+ self._set_sub_prop("writeDisposition", value)
+
+ @property
+ def destination_encryption_configuration(self):
+ """google.cloud.bigquery.table.EncryptionConfiguration: Custom
+ encryption configuration for the destination table.
+
+ Custom encryption configuration (e.g., Cloud KMS keys) or :data:`None`
+ if using default encryption.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.copy.destinationEncryptionConfiguration
+ """
+ prop = self._get_sub_prop("destinationEncryptionConfiguration")
+ if prop is not None:
+ prop = EncryptionConfiguration.from_api_repr(prop)
+ return prop
+
+ @destination_encryption_configuration.setter
+ def destination_encryption_configuration(self, value):
+ api_repr = value
+ if value is not None:
+ api_repr = value.to_api_repr()
+ self._set_sub_prop("destinationEncryptionConfiguration", api_repr)
+
+
+class CopyJob(_AsyncJob):
+ """Asynchronous job: copy data into a table from other tables.
+
+ :type job_id: str
+ :param job_id: the job's ID, within the project belonging to ``client``.
+
+ :type sources: list of :class:`google.cloud.bigquery.table.TableReference`
+ :param sources: Table from which data is to be loaded.
+
+ :type destination: :class:`google.cloud.bigquery.table.TableReference`
+ :param destination: Table into which data is to be loaded.
+
+ :type client: :class:`google.cloud.bigquery.client.Client`
+ :param client: A client which holds credentials and project configuration
+ for the dataset (which requires a project).
+
+ :type job_config: :class:`~google.cloud.bigquery.job.CopyJobConfig`
+ :param job_config:
+ (Optional) Extra configuration options for the copy job.
+ """
+
+ _JOB_TYPE = "copy"
+
+ def __init__(self, job_id, sources, destination, client, job_config=None):
+ super(CopyJob, self).__init__(job_id, client)
+
+ if job_config is None:
+ job_config = CopyJobConfig()
+
+ self.destination = destination
+ self.sources = sources
+ self._configuration = job_config
+
+ @property
+ def create_disposition(self):
+ """See
+ :attr:`google.cloud.bigquery.job.CopyJobConfig.create_disposition`.
+ """
+ return self._configuration.create_disposition
+
+ @property
+ def write_disposition(self):
+ """See
+ :attr:`google.cloud.bigquery.job.CopyJobConfig.write_disposition`.
+ """
+ return self._configuration.write_disposition
+
+ @property
+ def destination_encryption_configuration(self):
+ """google.cloud.bigquery.table.EncryptionConfiguration: Custom
+ encryption configuration for the destination table.
+
+ Custom encryption configuration (e.g., Cloud KMS keys) or :data:`None`
+ if using default encryption.
+
+ See
+ :attr:`google.cloud.bigquery.job.CopyJobConfig.destination_encryption_configuration`.
+ """
+ return self._configuration.destination_encryption_configuration
+
+ def to_api_repr(self):
+ """Generate a resource for :meth:`_begin`."""
+
+ source_refs = [
+ {
+ "projectId": table.project,
+ "datasetId": table.dataset_id,
+ "tableId": table.table_id,
+ }
+ for table in self.sources
+ ]
+
+ configuration = self._configuration.to_api_repr()
+ _helpers._set_sub_prop(configuration, ["copy", "sourceTables"], source_refs)
+ _helpers._set_sub_prop(
+ configuration,
+ ["copy", "destinationTable"],
+ {
+ "projectId": self.destination.project,
+ "datasetId": self.destination.dataset_id,
+ "tableId": self.destination.table_id,
+ },
+ )
+
+ return {
+ "jobReference": self._properties["jobReference"],
+ "configuration": configuration,
+ }
+
+ def _copy_configuration_properties(self, configuration):
+ """Helper: assign subclass configuration properties in cleaned."""
+ self._configuration._properties = copy.deepcopy(configuration)
+
+ @classmethod
+ def from_api_repr(cls, resource, client):
+ """Factory: construct a job given its API representation
+
+ .. note:
+
+ This method assumes that the project found in the resource matches
+ the client's project.
+
+ :type resource: dict
+ :param resource: dataset job representation returned from the API
+
+ :type client: :class:`google.cloud.bigquery.client.Client`
+ :param client: Client which holds credentials and project
+ configuration for the dataset.
+
+ :rtype: :class:`google.cloud.bigquery.job.CopyJob`
+ :returns: Job parsed from ``resource``.
+ """
+ job_id, config_resource = cls._get_resource_config(resource)
+ config = CopyJobConfig.from_api_repr(config_resource)
+ # Copy required fields to the job.
+ copy_resource = config_resource["copy"]
+ destination = TableReference.from_api_repr(copy_resource["destinationTable"])
+ sources = []
+ source_configs = copy_resource.get("sourceTables")
+ if source_configs is None:
+ single = copy_resource.get("sourceTable")
+ if single is None:
+ raise KeyError("Resource missing 'sourceTables' / 'sourceTable'")
+ source_configs = [single]
+ for source_config in source_configs:
+ table_ref = TableReference.from_api_repr(source_config)
+ sources.append(table_ref)
+ job = cls(job_id, sources, destination, client=client, job_config=config)
+ job._set_properties(resource)
+ return job
+
+
+class ExtractJobConfig(_JobConfig):
+ """Configuration options for extract jobs.
+
+ All properties in this class are optional. Values which are :data:`None` ->
+ server defaults. Set properties on the constructed configuration by using
+ the property name as the name of a keyword argument.
+ """
+
+ def __init__(self, **kwargs):
+ super(ExtractJobConfig, self).__init__("extract", **kwargs)
+
+ @property
+ def compression(self):
+ """google.cloud.bigquery.job.Compression: Compression type to use for
+ exported files.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.extract.compression
+ """
+ return self._get_sub_prop("compression")
+
+ @compression.setter
+ def compression(self, value):
+ self._set_sub_prop("compression", value)
+
+ @property
+ def destination_format(self):
+ """google.cloud.bigquery.job.DestinationFormat: Exported file format.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.extract.destinationFormat
+ """
+ return self._get_sub_prop("destinationFormat")
+
+ @destination_format.setter
+ def destination_format(self, value):
+ self._set_sub_prop("destinationFormat", value)
+
+ @property
+ def field_delimiter(self):
+ """str: Delimiter to use between fields in the exported data.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.extract.fieldDelimiter
+ """
+ return self._get_sub_prop("fieldDelimiter")
+
+ @field_delimiter.setter
+ def field_delimiter(self, value):
+ self._set_sub_prop("fieldDelimiter", value)
+
+ @property
+ def print_header(self):
+ """bool: Print a header row in the exported data.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.extract.printHeader
+ """
+ return self._get_sub_prop("printHeader")
+
+ @print_header.setter
+ def print_header(self, value):
+ self._set_sub_prop("printHeader", value)
+
+
+class ExtractJob(_AsyncJob):
+ """Asynchronous job: extract data from a table into Cloud Storage.
+
+ :type job_id: str
+ :param job_id: the job's ID
+
+ :type source: :class:`google.cloud.bigquery.table.TableReference`
+ :param source: Table into which data is to be loaded.
+
+ :type destination_uris: list of string
+ :param destination_uris:
+ URIs describing where the extracted data will be written in Cloud
+ Storage, using the format ``gs:///``.
+
+ :type client: :class:`google.cloud.bigquery.client.Client`
+ :param client:
+ A client which holds credentials and project configuration.
+
+ :type job_config: :class:`~google.cloud.bigquery.job.ExtractJobConfig`
+ :param job_config:
+ (Optional) Extra configuration options for the extract job.
+ """
+
+ _JOB_TYPE = "extract"
+
+ def __init__(self, job_id, source, destination_uris, client, job_config=None):
+ super(ExtractJob, self).__init__(job_id, client)
+
+ if job_config is None:
+ job_config = ExtractJobConfig()
+
+ self.source = source
+ self.destination_uris = destination_uris
+ self._configuration = job_config
+
+ @property
+ def compression(self):
+ """See
+ :attr:`google.cloud.bigquery.job.ExtractJobConfig.compression`.
+ """
+ return self._configuration.compression
+
+ @property
+ def destination_format(self):
+ """See
+ :attr:`google.cloud.bigquery.job.ExtractJobConfig.destination_format`.
+ """
+ return self._configuration.destination_format
+
+ @property
+ def field_delimiter(self):
+ """See
+ :attr:`google.cloud.bigquery.job.ExtractJobConfig.field_delimiter`.
+ """
+ return self._configuration.field_delimiter
+
+ @property
+ def print_header(self):
+ """See
+ :attr:`google.cloud.bigquery.job.ExtractJobConfig.print_header`.
+ """
+ return self._configuration.print_header
+
+ @property
+ def destination_uri_file_counts(self):
+ """Return file counts from job statistics, if present.
+
+ See:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#statistics.extract.destinationUriFileCounts
+
+ Returns:
+ a list of integer counts, each representing the number of files
+ per destination URI or URI pattern specified in the extract
+ configuration. These values will be in the same order as the URIs
+ specified in the 'destinationUris' field. Returns None if job is
+ not yet complete.
+ """
+ counts = self._job_statistics().get("destinationUriFileCounts")
+ if counts is not None:
+ return [int(count) for count in counts]
+ return None
+
+ def to_api_repr(self):
+ """Generate a resource for :meth:`_begin`."""
+
+ source_ref = {
+ "projectId": self.source.project,
+ "datasetId": self.source.dataset_id,
+ "tableId": self.source.table_id,
+ }
+
+ configuration = self._configuration.to_api_repr()
+ _helpers._set_sub_prop(configuration, ["extract", "sourceTable"], source_ref)
+ _helpers._set_sub_prop(
+ configuration, ["extract", "destinationUris"], self.destination_uris
+ )
+
+ return {
+ "jobReference": self._properties["jobReference"],
+ "configuration": configuration,
+ }
+
+ def _copy_configuration_properties(self, configuration):
+ """Helper: assign subclass configuration properties in cleaned."""
+ self._configuration._properties = copy.deepcopy(configuration)
+
+ @classmethod
+ def from_api_repr(cls, resource, client):
+ """Factory: construct a job given its API representation
+
+ .. note:
+
+ This method assumes that the project found in the resource matches
+ the client's project.
+
+ :type resource: dict
+ :param resource: dataset job representation returned from the API
+
+ :type client: :class:`google.cloud.bigquery.client.Client`
+ :param client: Client which holds credentials and project
+ configuration for the dataset.
+
+ :rtype: :class:`google.cloud.bigquery.job.ExtractJob`
+ :returns: Job parsed from ``resource``.
+ """
+ job_id, config_resource = cls._get_resource_config(resource)
+ config = ExtractJobConfig.from_api_repr(config_resource)
+ source_config = _helpers._get_sub_prop(
+ config_resource, ["extract", "sourceTable"]
+ )
+ dataset = DatasetReference(
+ source_config["projectId"], source_config["datasetId"]
+ )
+ source = dataset.table(source_config["tableId"])
+ destination_uris = _helpers._get_sub_prop(
+ config_resource, ["extract", "destinationUris"]
+ )
+
+ job = cls(job_id, source, destination_uris, client=client, job_config=config)
+ job._set_properties(resource)
+ return job
+
+
+def _from_api_repr_query_parameters(resource):
+ return [_query_param_from_api_repr(mapping) for mapping in resource]
+
+
+def _to_api_repr_query_parameters(value):
+ return [query_parameter.to_api_repr() for query_parameter in value]
+
+
+def _from_api_repr_udf_resources(resource):
+ udf_resources = []
+ for udf_mapping in resource:
+ for udf_type, udf_value in udf_mapping.items():
+ udf_resources.append(UDFResource(udf_type, udf_value))
+ return udf_resources
+
+
+def _to_api_repr_udf_resources(value):
+ return [{udf_resource.udf_type: udf_resource.value} for udf_resource in value]
+
+
+def _from_api_repr_table_defs(resource):
+ return {k: ExternalConfig.from_api_repr(v) for k, v in resource.items()}
+
+
+def _to_api_repr_table_defs(value):
+ return {k: ExternalConfig.to_api_repr(v) for k, v in value.items()}
+
+
+class QueryJobConfig(_JobConfig):
+ """Configuration options for query jobs.
+
+ All properties in this class are optional. Values which are :data:`None` ->
+ server defaults. Set properties on the constructed configuration by using
+ the property name as the name of a keyword argument.
+ """
+
+ def __init__(self, **kwargs):
+ super(QueryJobConfig, self).__init__("query", **kwargs)
+
+ @property
+ def destination_encryption_configuration(self):
+ """google.cloud.bigquery.table.EncryptionConfiguration: Custom
+ encryption configuration for the destination table.
+
+ Custom encryption configuration (e.g., Cloud KMS keys) or :data:`None`
+ if using default encryption.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.destinationEncryptionConfiguration
+ """
+ prop = self._get_sub_prop("destinationEncryptionConfiguration")
+ if prop is not None:
+ prop = EncryptionConfiguration.from_api_repr(prop)
+ return prop
+
+ @destination_encryption_configuration.setter
+ def destination_encryption_configuration(self, value):
+ api_repr = value
+ if value is not None:
+ api_repr = value.to_api_repr()
+ self._set_sub_prop("destinationEncryptionConfiguration", api_repr)
+
+ @property
+ def allow_large_results(self):
+ """bool: Allow large query results tables (legacy SQL, only)
+
+ See
+ https://g.co/cloud/bigquery/docs/reference/rest/v2/jobs#configuration.query.allowLargeResults
+ """
+ return self._get_sub_prop("allowLargeResults")
+
+ @allow_large_results.setter
+ def allow_large_results(self, value):
+ self._set_sub_prop("allowLargeResults", value)
+
+ @property
+ def create_disposition(self):
+ """google.cloud.bigquery.job.CreateDisposition: Specifies behavior
+ for creating tables.
+
+ See
+ https://g.co/cloud/bigquery/docs/reference/rest/v2/jobs#configuration.query.createDisposition
+ """
+ return self._get_sub_prop("createDisposition")
+
+ @create_disposition.setter
+ def create_disposition(self, value):
+ self._set_sub_prop("createDisposition", value)
+
+ @property
+ def default_dataset(self):
+ """google.cloud.bigquery.dataset.DatasetReference: the default dataset
+ to use for unqualified table names in the query or :data:`None` if not
+ set.
+
+ The ``default_dataset`` setter accepts:
+
+ - a :class:`~google.cloud.bigquery.dataset.Dataset`, or
+ - a :class:`~google.cloud.bigquery.dataset.DatasetReference`, or
+ - a :class:`str` of the fully-qualified dataset ID in standard SQL
+ format. The value must included a project ID and dataset ID
+ separated by ``.``. For example: ``your-project.your_dataset``.
+
+ See
+ https://g.co/cloud/bigquery/docs/reference/v2/jobs#configuration.query.defaultDataset
+ """
+ prop = self._get_sub_prop("defaultDataset")
+ if prop is not None:
+ prop = DatasetReference.from_api_repr(prop)
+ return prop
+
+ @default_dataset.setter
+ def default_dataset(self, value):
+ if value is None:
+ self._set_sub_prop("defaultDataset", None)
+ return
+
+ if isinstance(value, six.string_types):
+ value = DatasetReference.from_string(value)
+
+ if isinstance(value, (Dataset, DatasetListItem)):
+ value = value.reference
+
+ resource = value.to_api_repr()
+ self._set_sub_prop("defaultDataset", resource)
+
+ @property
+ def destination(self):
+ """google.cloud.bigquery.table.TableReference: table where results are
+ written or :data:`None` if not set.
+
+ The ``destination`` setter accepts:
+
+ - a :class:`~google.cloud.bigquery.table.Table`, or
+ - a :class:`~google.cloud.bigquery.table.TableReference`, or
+ - a :class:`str` of the fully-qualified table ID in standard SQL
+ format. The value must included a project ID, dataset ID, and table
+ ID, each separated by ``.``. For example:
+ ``your-project.your_dataset.your_table``.
+
+ See
+ https://g.co/cloud/bigquery/docs/reference/rest/v2/jobs#configuration.query.destinationTable
+ """
+ prop = self._get_sub_prop("destinationTable")
+ if prop is not None:
+ prop = TableReference.from_api_repr(prop)
+ return prop
+
+ @destination.setter
+ def destination(self, value):
+ if value is None:
+ self._set_sub_prop("destinationTable", None)
+ return
+
+ value = _table_arg_to_table_ref(value)
+ resource = value.to_api_repr()
+ self._set_sub_prop("destinationTable", resource)
+
+ @property
+ def dry_run(self):
+ """bool: :data:`True` if this query should be a dry run to estimate
+ costs.
+
+ See
+ https://g.co/cloud/bigquery/docs/reference/v2/jobs#configuration.dryRun
+ """
+ return self._properties.get("dryRun")
+
+ @dry_run.setter
+ def dry_run(self, value):
+ self._properties["dryRun"] = value
+
+ @property
+ def flatten_results(self):
+ """bool: Flatten nested/repeated fields in results. (Legacy SQL only)
+
+ See
+ https://g.co/cloud/bigquery/docs/reference/rest/v2/jobs#configuration.query.flattenResults
+ """
+ return self._get_sub_prop("flattenResults")
+
+ @flatten_results.setter
+ def flatten_results(self, value):
+ self._set_sub_prop("flattenResults", value)
+
+ @property
+ def maximum_billing_tier(self):
+ """int: Deprecated. Changes the billing tier to allow high-compute
+ queries.
+
+ See
+ https://g.co/cloud/bigquery/docs/reference/rest/v2/jobs#configuration.query.maximumBillingTier
+ """
+ return self._get_sub_prop("maximumBillingTier")
+
+ @maximum_billing_tier.setter
+ def maximum_billing_tier(self, value):
+ self._set_sub_prop("maximumBillingTier", value)
+
+ @property
+ def maximum_bytes_billed(self):
+ """int: Maximum bytes to be billed for this job or :data:`None` if not set.
+
+ See
+ https://g.co/cloud/bigquery/docs/reference/rest/v2/jobs#configuration.query.maximumBytesBilled
+ """
+ return _helpers._int_or_none(self._get_sub_prop("maximumBytesBilled"))
+
+ @maximum_bytes_billed.setter
+ def maximum_bytes_billed(self, value):
+ self._set_sub_prop("maximumBytesBilled", str(value))
+
+ @property
+ def priority(self):
+ """google.cloud.bigquery.job.QueryPriority: Priority of the query.
+
+ See
+ https://g.co/cloud/bigquery/docs/reference/rest/v2/jobs#configuration.query.priority
+ """
+ return self._get_sub_prop("priority")
+
+ @priority.setter
+ def priority(self, value):
+ self._set_sub_prop("priority", value)
+
+ @property
+ def query_parameters(self):
+ """List[Union[google.cloud.bigquery.query.ArrayQueryParameter, \
+ google.cloud.bigquery.query.ScalarQueryParameter, \
+ google.cloud.bigquery.query.StructQueryParameter]]: list of parameters
+ for parameterized query (empty by default)
+
+ See:
+ https://g.co/cloud/bigquery/docs/reference/rest/v2/jobs#configuration.query.queryParameters
+ """
+ prop = self._get_sub_prop("queryParameters", default=[])
+ return _from_api_repr_query_parameters(prop)
+
+ @query_parameters.setter
+ def query_parameters(self, values):
+ self._set_sub_prop("queryParameters", _to_api_repr_query_parameters(values))
+
+ @property
+ def udf_resources(self):
+ """List[google.cloud.bigquery.query.UDFResource]: user
+ defined function resources (empty by default)
+
+ See:
+ https://g.co/cloud/bigquery/docs/reference/rest/v2/jobs#configuration.query.userDefinedFunctionResources
+ """
+ prop = self._get_sub_prop("userDefinedFunctionResources", default=[])
+ return _from_api_repr_udf_resources(prop)
+
+ @udf_resources.setter
+ def udf_resources(self, values):
+ self._set_sub_prop(
+ "userDefinedFunctionResources", _to_api_repr_udf_resources(values)
+ )
+
+ @property
+ def use_legacy_sql(self):
+ """bool: Use legacy SQL syntax.
+
+ See
+ https://g.co/cloud/bigquery/docs/reference/v2/jobs#configuration.query.useLegacySql
+ """
+ return self._get_sub_prop("useLegacySql")
+
+ @use_legacy_sql.setter
+ def use_legacy_sql(self, value):
+ self._set_sub_prop("useLegacySql", value)
+
+ @property
+ def use_query_cache(self):
+ """bool: Look for the query result in the cache.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.useQueryCache
+ """
+ return self._get_sub_prop("useQueryCache")
+
+ @use_query_cache.setter
+ def use_query_cache(self, value):
+ self._set_sub_prop("useQueryCache", value)
+
+ @property
+ def write_disposition(self):
+ """google.cloud.bigquery.job.WriteDisposition: Action that occurs if
+ the destination table already exists.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.writeDisposition
+ """
+ return self._get_sub_prop("writeDisposition")
+
+ @write_disposition.setter
+ def write_disposition(self, value):
+ self._set_sub_prop("writeDisposition", value)
+
+ @property
+ def table_definitions(self):
+ """Dict[str, google.cloud.bigquery.external_config.ExternalConfig]:
+ Definitions for external tables or :data:`None` if not set.
+
+ See
+ https://g.co/cloud/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions
+ """
+ prop = self._get_sub_prop("tableDefinitions")
+ if prop is not None:
+ prop = _from_api_repr_table_defs(prop)
+ return prop
+
+ @table_definitions.setter
+ def table_definitions(self, values):
+ self._set_sub_prop("tableDefinitions", _to_api_repr_table_defs(values))
+
+ @property
+ def time_partitioning(self):
+ """google.cloud.bigquery.table.TimePartitioning: Specifies time-based
+ partitioning for the destination table.
+ """
+ prop = self._get_sub_prop("timePartitioning")
+ if prop is not None:
+ prop = TimePartitioning.from_api_repr(prop)
+ return prop
+
+ @time_partitioning.setter
+ def time_partitioning(self, value):
+ api_repr = value
+ if value is not None:
+ api_repr = value.to_api_repr()
+ self._set_sub_prop("timePartitioning", api_repr)
+
+ @property
+ def clustering_fields(self):
+ """Union[List[str], None]: Fields defining clustering for the table
+
+ (Defaults to :data:`None`).
+
+ Clustering fields are immutable after table creation.
+
+ .. note::
+
+ As of 2018-06-29, clustering fields cannot be set on a table
+ which does not also have time partioning defined.
+ """
+ prop = self._get_sub_prop("clustering")
+ if prop is not None:
+ return list(prop.get("fields", ()))
+
+ @clustering_fields.setter
+ def clustering_fields(self, value):
+ """Union[List[str], None]: Fields defining clustering for the table
+
+ (Defaults to :data:`None`).
+ """
+ if value is not None:
+ self._set_sub_prop("clustering", {"fields": value})
+ else:
+ self._del_sub_prop("clustering")
+
+ @property
+ def schema_update_options(self):
+ """List[google.cloud.bigquery.job.SchemaUpdateOption]: Specifies
+ updates to the destination table schema to allow as a side effect of
+ the query job.
+ """
+ return self._get_sub_prop("schemaUpdateOptions")
+
+ @schema_update_options.setter
+ def schema_update_options(self, values):
+ self._set_sub_prop("schemaUpdateOptions", values)
+
+ def to_api_repr(self):
+ """Build an API representation of the query job config.
+
+ Returns:
+ dict: A dictionary in the format used by the BigQuery API.
+ """
+ resource = copy.deepcopy(self._properties)
+
+ # Query parameters have an addition property associated with them
+ # to indicate if the query is using named or positional parameters.
+ query_parameters = resource["query"].get("queryParameters")
+ if query_parameters:
+ if query_parameters[0].get("name") is None:
+ resource["query"]["parameterMode"] = "POSITIONAL"
+ else:
+ resource["query"]["parameterMode"] = "NAMED"
+
+ return resource
+
+
+class QueryJob(_AsyncJob):
+ """Asynchronous job: query tables.
+
+ :type job_id: str
+ :param job_id: the job's ID, within the project belonging to ``client``.
+
+ :type query: str
+ :param query: SQL query string
+
+ :type client: :class:`google.cloud.bigquery.client.Client`
+ :param client: A client which holds credentials and project configuration
+ for the dataset (which requires a project).
+
+ :type job_config: :class:`~google.cloud.bigquery.job.QueryJobConfig`
+ :param job_config:
+ (Optional) Extra configuration options for the query job.
+ """
+
+ _JOB_TYPE = "query"
+ _UDF_KEY = "userDefinedFunctionResources"
+
+ def __init__(self, job_id, query, client, job_config=None):
+ super(QueryJob, self).__init__(job_id, client)
+
+ if job_config is None:
+ job_config = QueryJobConfig()
+ if job_config.use_legacy_sql is None:
+ job_config.use_legacy_sql = False
+
+ _helpers._set_sub_prop(
+ self._properties, ["configuration", "query", "query"], query
+ )
+
+ self._configuration = job_config
+ self._query_results = None
+ self._done_timeout = None
+
+ @property
+ def allow_large_results(self):
+ """See
+ :attr:`google.cloud.bigquery.job.QueryJobConfig.allow_large_results`.
+ """
+ return self._configuration.allow_large_results
+
+ @property
+ def create_disposition(self):
+ """See
+ :attr:`google.cloud.bigquery.job.QueryJobConfig.create_disposition`.
+ """
+ return self._configuration.create_disposition
+
+ @property
+ def default_dataset(self):
+ """See
+ :attr:`google.cloud.bigquery.job.QueryJobConfig.default_dataset`.
+ """
+ return self._configuration.default_dataset
+
+ @property
+ def destination(self):
+ """See
+ :attr:`google.cloud.bigquery.job.QueryJobConfig.destination`.
+ """
+ return self._configuration.destination
+
+ @property
+ def destination_encryption_configuration(self):
+ """google.cloud.bigquery.table.EncryptionConfiguration: Custom
+ encryption configuration for the destination table.
+
+ Custom encryption configuration (e.g., Cloud KMS keys) or :data:`None`
+ if using default encryption.
+
+ See
+ :attr:`google.cloud.bigquery.job.QueryJobConfig.destination_encryption_configuration`.
+ """
+ return self._configuration.destination_encryption_configuration
+
+ @property
+ def dry_run(self):
+ """See
+ :attr:`google.cloud.bigquery.job.QueryJobConfig.dry_run`.
+ """
+ return self._configuration.dry_run
+
+ @property
+ def flatten_results(self):
+ """See
+ :attr:`google.cloud.bigquery.job.QueryJobConfig.flatten_results`.
+ """
+ return self._configuration.flatten_results
+
+ @property
+ def priority(self):
+ """See
+ :attr:`google.cloud.bigquery.job.QueryJobConfig.priority`.
+ """
+ return self._configuration.priority
+
+ @property
+ def query(self):
+ """str: The query text used in this query job.
+
+ See:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.query
+ """
+ return _helpers._get_sub_prop(
+ self._properties, ["configuration", "query", "query"]
+ )
+
+ @property
+ def query_parameters(self):
+ """See
+ :attr:`google.cloud.bigquery.job.QueryJobConfig.query_parameters`.
+ """
+ return self._configuration.query_parameters
+
+ @property
+ def udf_resources(self):
+ """See
+ :attr:`google.cloud.bigquery.job.QueryJobConfig.udf_resources`.
+ """
+ return self._configuration.udf_resources
+
+ @property
+ def use_legacy_sql(self):
+ """See
+ :attr:`google.cloud.bigquery.job.QueryJobConfig.use_legacy_sql`.
+ """
+ return self._configuration.use_legacy_sql
+
+ @property
+ def use_query_cache(self):
+ """See
+ :attr:`google.cloud.bigquery.job.QueryJobConfig.use_query_cache`.
+ """
+ return self._configuration.use_query_cache
+
+ @property
+ def write_disposition(self):
+ """See
+ :attr:`google.cloud.bigquery.job.QueryJobConfig.write_disposition`.
+ """
+ return self._configuration.write_disposition
+
+ @property
+ def maximum_billing_tier(self):
+ """See
+ :attr:`google.cloud.bigquery.job.QueryJobConfig.maximum_billing_tier`.
+ """
+ return self._configuration.maximum_billing_tier
+
+ @property
+ def maximum_bytes_billed(self):
+ """See
+ :attr:`google.cloud.bigquery.job.QueryJobConfig.maximum_bytes_billed`.
+ """
+ return self._configuration.maximum_bytes_billed
+
+ @property
+ def table_definitions(self):
+ """See
+ :attr:`google.cloud.bigquery.job.QueryJobConfig.table_definitions`.
+ """
+ return self._configuration.table_definitions
+
+ @property
+ def time_partitioning(self):
+ """See
+ :attr:`google.cloud.bigquery.job.QueryJobConfig.time_partitioning`.
+ """
+ return self._configuration.time_partitioning
+
+ @property
+ def clustering_fields(self):
+ """See
+ :attr:`google.cloud.bigquery.job.QueryJobConfig.clustering_fields`.
+ """
+ return self._configuration.clustering_fields
+
+ @property
+ def schema_update_options(self):
+ """See
+ :attr:`google.cloud.bigquery.job.QueryJobConfig.schema_update_options`.
+ """
+ return self._configuration.schema_update_options
+
+ def to_api_repr(self):
+ """Generate a resource for :meth:`_begin`."""
+ configuration = self._configuration.to_api_repr()
+
+ resource = {
+ "jobReference": self._properties["jobReference"],
+ "configuration": configuration,
+ }
+ configuration["query"]["query"] = self.query
+
+ return resource
+
+ def _copy_configuration_properties(self, configuration):
+ """Helper: assign subclass configuration properties in cleaned."""
+ self._configuration._properties = copy.deepcopy(configuration)
+
+ @classmethod
+ def from_api_repr(cls, resource, client):
+ """Factory: construct a job given its API representation
+
+ :type resource: dict
+ :param resource: dataset job representation returned from the API
+
+ :type client: :class:`google.cloud.bigquery.client.Client`
+ :param client: Client which holds credentials and project
+ configuration for the dataset.
+
+ :rtype: :class:`google.cloud.bigquery.job.QueryJob`
+ :returns: Job parsed from ``resource``.
+ """
+ job_id, config = cls._get_resource_config(resource)
+ query = _helpers._get_sub_prop(config, ["query", "query"])
+ job = cls(job_id, query, client=client)
+ job._set_properties(resource)
+ return job
+
+ @property
+ def query_plan(self):
+ """Return query plan from job statistics, if present.
+
+ See:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#statistics.query.queryPlan
+
+ :rtype: list of :class:`QueryPlanEntry`
+ :returns: mappings describing the query plan, or an empty list
+ if the query has not yet completed.
+ """
+ plan_entries = self._job_statistics().get("queryPlan", ())
+ return [QueryPlanEntry.from_api_repr(entry) for entry in plan_entries]
+
+ @property
+ def timeline(self):
+ """List(TimelineEntry): Return the query execution timeline
+ from job statistics.
+ """
+ raw = self._job_statistics().get("timeline", ())
+ return [TimelineEntry.from_api_repr(entry) for entry in raw]
+
+ @property
+ def total_bytes_processed(self):
+ """Return total bytes processed from job statistics, if present.
+
+ See:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#statistics.query.totalBytesProcessed
+
+ :rtype: int or None
+ :returns: total bytes processed by the job, or None if job is not
+ yet complete.
+ """
+ result = self._job_statistics().get("totalBytesProcessed")
+ if result is not None:
+ result = int(result)
+ return result
+
+ @property
+ def total_bytes_billed(self):
+ """Return total bytes billed from job statistics, if present.
+
+ See:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#statistics.query.totalBytesBilled
+
+ :rtype: int or None
+ :returns: total bytes processed by the job, or None if job is not
+ yet complete.
+ """
+ result = self._job_statistics().get("totalBytesBilled")
+ if result is not None:
+ result = int(result)
+ return result
+
+ @property
+ def billing_tier(self):
+ """Return billing tier from job statistics, if present.
+
+ See:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#statistics.query.billingTier
+
+ :rtype: int or None
+ :returns: billing tier used by the job, or None if job is not
+ yet complete.
+ """
+ return self._job_statistics().get("billingTier")
+
+ @property
+ def cache_hit(self):
+ """Return whether or not query results were served from cache.
+
+ See:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#statistics.query.cacheHit
+
+ :rtype: bool or None
+ :returns: whether the query results were returned from cache, or None
+ if job is not yet complete.
+ """
+ return self._job_statistics().get("cacheHit")
+
+ @property
+ def ddl_operation_performed(self):
+ """Optional[str]: Return the DDL operation performed.
+
+ See:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#statistics.query.ddlOperationPerformed
+
+ """
+ return self._job_statistics().get("ddlOperationPerformed")
+
+ @property
+ def ddl_target_routine(self):
+ """Optional[google.cloud.bigquery.routine.RoutineReference]: Return the DDL target routine, present
+ for CREATE/DROP FUNCTION/PROCEDURE queries.
+
+ See:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/JobStatistics
+ """
+ prop = self._job_statistics().get("ddlTargetRoutine")
+ if prop is not None:
+ prop = RoutineReference.from_api_repr(prop)
+ return prop
+
+ @property
+ def ddl_target_table(self):
+ """Optional[google.cloud.bigquery.table.TableReference]: Return the DDL target table, present
+ for CREATE/DROP TABLE/VIEW queries.
+
+ See:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#statistics.query.ddlTargetTable
+ """
+ prop = self._job_statistics().get("ddlTargetTable")
+ if prop is not None:
+ prop = TableReference.from_api_repr(prop)
+ return prop
+
+ @property
+ def num_dml_affected_rows(self):
+ """Return the number of DML rows affected by the job.
+
+ See:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#statistics.query.numDmlAffectedRows
+
+ :rtype: int or None
+ :returns: number of DML rows affected by the job, or None if job is not
+ yet complete.
+ """
+ result = self._job_statistics().get("numDmlAffectedRows")
+ if result is not None:
+ result = int(result)
+ return result
+
+ @property
+ def slot_millis(self):
+ """Union[int, None]: Slot-milliseconds used by this query job."""
+ return _helpers._int_or_none(self._job_statistics().get("totalSlotMs"))
+
+ @property
+ def statement_type(self):
+ """Return statement type from job statistics, if present.
+
+ See:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#statistics.query.statementType
+
+ :rtype: str or None
+ :returns: type of statement used by the job, or None if job is not
+ yet complete.
+ """
+ return self._job_statistics().get("statementType")
+
+ @property
+ def referenced_tables(self):
+ """Return referenced tables from job statistics, if present.
+
+ See:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#statistics.query.referencedTables
+
+ :rtype: list of dict
+ :returns: mappings describing the query plan, or an empty list
+ if the query has not yet completed.
+ """
+ tables = []
+ datasets_by_project_name = {}
+
+ for table in self._job_statistics().get("referencedTables", ()):
+
+ t_project = table["projectId"]
+
+ ds_id = table["datasetId"]
+ t_dataset = datasets_by_project_name.get((t_project, ds_id))
+ if t_dataset is None:
+ t_dataset = DatasetReference(t_project, ds_id)
+ datasets_by_project_name[(t_project, ds_id)] = t_dataset
+
+ t_name = table["tableId"]
+ tables.append(t_dataset.table(t_name))
+
+ return tables
+
+ @property
+ def undeclared_query_parameters(self):
+ """Return undeclared query parameters from job statistics, if present.
+
+ See:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#statistics.query.undeclaredQueryParameters
+
+ :rtype:
+ list of
+ :class:`~google.cloud.bigquery.ArrayQueryParameter`,
+ :class:`~google.cloud.bigquery.ScalarQueryParameter`, or
+ :class:`~google.cloud.bigquery.StructQueryParameter`
+ :returns: undeclared parameters, or an empty list if the query has
+ not yet completed.
+ """
+ parameters = []
+ undeclared = self._job_statistics().get("undeclaredQueryParameters", ())
+
+ for parameter in undeclared:
+ p_type = parameter["parameterType"]
+
+ if "arrayType" in p_type:
+ klass = ArrayQueryParameter
+ elif "structTypes" in p_type:
+ klass = StructQueryParameter
+ else:
+ klass = ScalarQueryParameter
+
+ parameters.append(klass.from_api_repr(parameter))
+
+ return parameters
+
+ @property
+ def estimated_bytes_processed(self):
+ """Return the estimated number of bytes processed by the query.
+
+ See:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#statistics.query.estimatedBytesProcessed
+
+ :rtype: int or None
+ :returns: number of DML rows affected by the job, or None if job is not
+ yet complete.
+ """
+ result = self._job_statistics().get("estimatedBytesProcessed")
+ if result is not None:
+ result = int(result)
+ return result
+
+ def done(self, retry=DEFAULT_RETRY):
+ """Refresh the job and checks if it is complete.
+
+ :rtype: bool
+ :returns: True if the job is complete, False otherwise.
+ """
+ # Since the API to getQueryResults can hang up to the timeout value
+ # (default of 10 seconds), set the timeout parameter to ensure that
+ # the timeout from the futures API is respected. See:
+ # https://github.com/GoogleCloudPlatform/google-cloud-python/issues/4135
+ timeout_ms = None
+ if self._done_timeout is not None:
+ # Subtract a buffer for context switching, network latency, etc.
+ timeout = self._done_timeout - _TIMEOUT_BUFFER_SECS
+ timeout = max(min(timeout, 10), 0)
+ self._done_timeout -= timeout
+ self._done_timeout = max(0, self._done_timeout)
+ timeout_ms = int(timeout * 1000)
+
+ # Do not refresh is the state is already done, as the job will not
+ # change once complete.
+ if self.state != _DONE_STATE:
+ self._query_results = self._client._get_query_results(
+ self.job_id,
+ retry,
+ project=self.project,
+ timeout_ms=timeout_ms,
+ location=self.location,
+ )
+
+ # Only reload the job once we know the query is complete.
+ # This will ensure that fields such as the destination table are
+ # correctly populated.
+ if self._query_results.complete:
+ self.reload(retry=retry)
+
+ return self.state == _DONE_STATE
+
+ def _blocking_poll(self, timeout=None):
+ self._done_timeout = timeout
+ super(QueryJob, self)._blocking_poll(timeout=timeout)
+
+ @staticmethod
+ def _format_for_exception(query, job_id):
+ """Format a query for the output in exception message.
+
+ Args:
+ query (str): The SQL query to format.
+ job_id (str): The ID of the job that ran the query.
+
+ Returns: (str)
+ A formatted query text.
+ """
+ template = "\n\n(job ID: {job_id})\n\n{header}\n\n{ruler}\n{body}\n{ruler}"
+
+ lines = query.splitlines()
+ max_line_len = max(len(l) for l in lines)
+
+ header = "-----Query Job SQL Follows-----"
+ header = "{:^{total_width}}".format(header, total_width=max_line_len + 5)
+
+ # Print out a "ruler" above and below the SQL so we can judge columns.
+ # Left pad for the line numbers (4 digits plus ":").
+ ruler = " |" + " . |" * (max_line_len // 10)
+
+ # Put line numbers next to the SQL.
+ body = "\n".join(
+ "{:4}:{}".format(n, line) for n, line in enumerate(lines, start=1)
+ )
+
+ return template.format(job_id=job_id, header=header, ruler=ruler, body=body)
+
+ def _begin(self, client=None, retry=DEFAULT_RETRY):
+ """API call: begin the job via a POST request
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/insert
+
+ Args:
+ client (Optional[google.cloud.bigquery.client.Client]):
+ The client to use. If not passed, falls back to the ``client``
+ associated with the job object or``NoneType``.
+ retry (Optional[google.api_core.retry.Retry]):
+ How to retry the RPC.
+
+ Raises:
+ ValueError:
+ If the job has already begun.
+ """
+
+ try:
+ super(QueryJob, self)._begin(client=client, retry=retry)
+ except exceptions.GoogleCloudError as exc:
+ exc.message += self._format_for_exception(self.query, self.job_id)
+ raise
+
+ def result(self, timeout=None, page_size=None, retry=DEFAULT_RETRY):
+ """Start the job and wait for it to complete and get the result.
+
+ Args:
+ timeout (float):
+ How long (in seconds) to wait for job to complete before
+ raising a :class:`concurrent.futures.TimeoutError`.
+ page_size (int):
+ (Optional) The maximum number of rows in each page of results
+ from this request. Non-positive values are ignored.
+ retry (google.api_core.retry.Retry):
+ (Optional) How to retry the call that retrieves rows.
+
+ Returns:
+ google.cloud.bigquery.table.RowIterator:
+ Iterator of row data
+ :class:`~google.cloud.bigquery.table.Row`-s. During each
+ page, the iterator will have the ``total_rows`` attribute
+ set, which counts the total number of rows **in the result
+ set** (this is distinct from the total number of rows in the
+ current page: ``iterator.page.num_items``).
+
+ Raises:
+ google.cloud.exceptions.GoogleCloudError:
+ If the job failed.
+ concurrent.futures.TimeoutError:
+ If the job did not complete in the given timeout.
+ """
+ try:
+ super(QueryJob, self).result(timeout=timeout)
+
+ # Return an iterator instead of returning the job.
+ if not self._query_results:
+ self._query_results = self._client._get_query_results(
+ self.job_id, retry, project=self.project, location=self.location
+ )
+ except exceptions.GoogleCloudError as exc:
+ exc.message += self._format_for_exception(self.query, self.job_id)
+ raise
+
+ # If the query job is complete but there are no query results, this was
+ # special job, such as a DDL query. Return an empty result set to
+ # indicate success and avoid calling tabledata.list on a table which
+ # can't be read (such as a view table).
+ if self._query_results.total_rows is None:
+ return _EmptyRowIterator()
+
+ schema = self._query_results.schema
+ dest_table_ref = self.destination
+ dest_table = Table(dest_table_ref, schema=schema)
+ dest_table._properties["numRows"] = self._query_results.total_rows
+ rows = self._client.list_rows(dest_table, page_size=page_size, retry=retry)
+ rows._preserve_order = _contains_order_by(self.query)
+ return rows
+
+ # If changing the signature of this method, make sure to apply the same
+ # changes to table.RowIterator.to_arrow()
+ def to_arrow(self, progress_bar_type=None, bqstorage_client=None):
+ """[Beta] Create a class:`pyarrow.Table` by loading all pages of a
+ table or query.
+
+ Args:
+ progress_bar_type (Optional[str]):
+ If set, use the `tqdm `_ library to
+ display a progress bar while the data downloads. Install the
+ ``tqdm`` package to use this feature.
+
+ Possible values of ``progress_bar_type`` include:
+
+ ``None``
+ No progress bar.
+ ``'tqdm'``
+ Use the :func:`tqdm.tqdm` function to print a progress bar
+ to :data:`sys.stderr`.
+ ``'tqdm_notebook'``
+ Use the :func:`tqdm.tqdm_notebook` function to display a
+ progress bar as a Jupyter notebook widget.
+ ``'tqdm_gui'``
+ Use the :func:`tqdm.tqdm_gui` function to display a
+ progress bar as a graphical dialog box.
+ bqstorage_client ( \
+ google.cloud.bigquery_storage_v1beta1.BigQueryStorageClient \
+ ):
+ **Beta Feature** Optional. A BigQuery Storage API client. If
+ supplied, use the faster BigQuery Storage API to fetch rows
+ from BigQuery. This API is a billable API.
+
+ This method requires the ``pyarrow`` and
+ ``google-cloud-bigquery-storage`` libraries.
+
+ Reading from a specific partition or snapshot is not
+ currently supported by this method.
+
+ Returns:
+ pyarrow.Table
+ A :class:`pyarrow.Table` populated with row data and column
+ headers from the query results. The column headers are derived
+ from the destination table's schema.
+
+ Raises:
+ ValueError:
+ If the :mod:`pyarrow` library cannot be imported.
+
+ ..versionadded:: 1.17.0
+ """
+ return self.result().to_arrow(
+ progress_bar_type=progress_bar_type, bqstorage_client=bqstorage_client
+ )
+
+ # If changing the signature of this method, make sure to apply the same
+ # changes to table.RowIterator.to_dataframe()
+ def to_dataframe(self, bqstorage_client=None, dtypes=None, progress_bar_type=None):
+ """Return a pandas DataFrame from a QueryJob
+
+ Args:
+ bqstorage_client ( \
+ google.cloud.bigquery_storage_v1beta1.BigQueryStorageClient \
+ ):
+ **Alpha Feature** Optional. A BigQuery Storage API client. If
+ supplied, use the faster BigQuery Storage API to fetch rows
+ from BigQuery. This API is a billable API.
+
+ This method requires the ``fastavro`` and
+ ``google-cloud-bigquery-storage`` libraries.
+
+ Reading from a specific partition or snapshot is not
+ currently supported by this method.
+
+ **Caution**: There is a known issue reading small anonymous
+ query result tables with the BQ Storage API. Write your query
+ results to a destination table to work around this issue.
+ dtypes ( \
+ Map[str, Union[str, pandas.Series.dtype]] \
+ ):
+ Optional. A dictionary of column names pandas ``dtype``s. The
+ provided ``dtype`` is used when constructing the series for
+ the column specified. Otherwise, the default pandas behavior
+ is used.
+ progress_bar_type (Optional[str]):
+ If set, use the `tqdm `_ library to
+ display a progress bar while the data downloads. Install the
+ ``tqdm`` package to use this feature.
+
+ See
+ :func:`~google.cloud.bigquery.table.RowIterator.to_dataframe`
+ for details.
+
+ ..versionadded:: 1.11.0
+
+ Returns:
+ A :class:`~pandas.DataFrame` populated with row data and column
+ headers from the query results. The column headers are derived
+ from the destination table's schema.
+
+ Raises:
+ ValueError: If the `pandas` library cannot be imported.
+ """
+ return self.result().to_dataframe(
+ bqstorage_client=bqstorage_client,
+ dtypes=dtypes,
+ progress_bar_type=progress_bar_type,
+ )
+
+ def __iter__(self):
+ return iter(self.result())
+
+
+class QueryPlanEntryStep(object):
+ """Map a single step in a query plan entry.
+
+ :type kind: str
+ :param kind: step type
+
+ :type substeps:
+ :param substeps: names of substeps
+ """
+
+ def __init__(self, kind, substeps):
+ self.kind = kind
+ self.substeps = list(substeps)
+
+ @classmethod
+ def from_api_repr(cls, resource):
+ """Factory: construct instance from the JSON repr.
+
+ :type resource: dict
+ :param resource: JSON representation of the entry
+
+ :rtype: :class:`QueryPlanEntryStep`
+ :return: new instance built from the resource
+ """
+ return cls(kind=resource.get("kind"), substeps=resource.get("substeps", ()))
+
+ def __eq__(self, other):
+ if not isinstance(other, self.__class__):
+ return NotImplemented
+ return self.kind == other.kind and self.substeps == other.substeps
+
+
+class QueryPlanEntry(object):
+ """QueryPlanEntry represents a single stage of a query execution plan.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs
+ for the underlying API representation within query statistics.
+
+ """
+
+ def __init__(self):
+ self._properties = {}
+
+ @classmethod
+ def from_api_repr(cls, resource):
+ """Factory: construct instance from the JSON repr.
+
+ Args:
+ resource(Dict[str: object]):
+ ExplainQueryStage representation returned from API
+
+ Returns:
+ google.cloud.bigquery.QueryPlanEntry:
+ Query plan entry parsed from ``resource``
+ """
+ entry = cls()
+ entry._properties = resource
+ return entry
+
+ @property
+ def name(self):
+ """Union[str, None]: Human-readable name of the stage."""
+ return self._properties.get("name")
+
+ @property
+ def entry_id(self):
+ """Union[str, None]: Unique ID for the stage within the plan."""
+ return self._properties.get("id")
+
+ @property
+ def start(self):
+ """Union[Datetime, None]: Datetime when the stage started."""
+ if self._properties.get("startMs") is None:
+ return None
+ return _helpers._datetime_from_microseconds(
+ int(self._properties.get("startMs")) * 1000.0
+ )
+
+ @property
+ def end(self):
+ """Union[Datetime, None]: Datetime when the stage ended."""
+ if self._properties.get("endMs") is None:
+ return None
+ return _helpers._datetime_from_microseconds(
+ int(self._properties.get("endMs")) * 1000.0
+ )
+
+ @property
+ def input_stages(self):
+ """List(int): Entry IDs for stages that were inputs for this stage."""
+ if self._properties.get("inputStages") is None:
+ return []
+ return [
+ _helpers._int_or_none(entry)
+ for entry in self._properties.get("inputStages")
+ ]
+
+ @property
+ def parallel_inputs(self):
+ """Union[int, None]: Number of parallel input segments within
+ the stage.
+ """
+ return _helpers._int_or_none(self._properties.get("parallelInputs"))
+
+ @property
+ def completed_parallel_inputs(self):
+ """Union[int, None]: Number of parallel input segments completed."""
+ return _helpers._int_or_none(self._properties.get("completedParallelInputs"))
+
+ @property
+ def wait_ms_avg(self):
+ """Union[int, None]: Milliseconds the average worker spent waiting to
+ be scheduled.
+ """
+ return _helpers._int_or_none(self._properties.get("waitMsAvg"))
+
+ @property
+ def wait_ms_max(self):
+ """Union[int, None]: Milliseconds the slowest worker spent waiting to
+ be scheduled.
+ """
+ return _helpers._int_or_none(self._properties.get("waitMsMax"))
+
+ @property
+ def wait_ratio_avg(self):
+ """Union[float, None]: Ratio of time the average worker spent waiting
+ to be scheduled, relative to the longest time spent by any worker in
+ any stage of the overall plan.
+ """
+ return self._properties.get("waitRatioAvg")
+
+ @property
+ def wait_ratio_max(self):
+ """Union[float, None]: Ratio of time the slowest worker spent waiting
+ to be scheduled, relative to the longest time spent by any worker in
+ any stage of the overall plan.
+ """
+ return self._properties.get("waitRatioMax")
+
+ @property
+ def read_ms_avg(self):
+ """Union[int, None]: Milliseconds the average worker spent reading
+ input.
+ """
+ return _helpers._int_or_none(self._properties.get("readMsAvg"))
+
+ @property
+ def read_ms_max(self):
+ """Union[int, None]: Milliseconds the slowest worker spent reading
+ input.
+ """
+ return _helpers._int_or_none(self._properties.get("readMsMax"))
+
+ @property
+ def read_ratio_avg(self):
+ """Union[float, None]: Ratio of time the average worker spent reading
+ input, relative to the longest time spent by any worker in any stage
+ of the overall plan.
+ """
+ return self._properties.get("readRatioAvg")
+
+ @property
+ def read_ratio_max(self):
+ """Union[float, None]: Ratio of time the slowest worker spent reading
+ to be scheduled, relative to the longest time spent by any worker in
+ any stage of the overall plan.
+ """
+ return self._properties.get("readRatioMax")
+
+ @property
+ def compute_ms_avg(self):
+ """Union[int, None]: Milliseconds the average worker spent on CPU-bound
+ processing.
+ """
+ return _helpers._int_or_none(self._properties.get("computeMsAvg"))
+
+ @property
+ def compute_ms_max(self):
+ """Union[int, None]: Milliseconds the slowest worker spent on CPU-bound
+ processing.
+ """
+ return _helpers._int_or_none(self._properties.get("computeMsMax"))
+
+ @property
+ def compute_ratio_avg(self):
+ """Union[float, None]: Ratio of time the average worker spent on
+ CPU-bound processing, relative to the longest time spent by any
+ worker in any stage of the overall plan.
+ """
+ return self._properties.get("computeRatioAvg")
+
+ @property
+ def compute_ratio_max(self):
+ """Union[float, None]: Ratio of time the slowest worker spent on
+ CPU-bound processing, relative to the longest time spent by any
+ worker in any stage of the overall plan.
+ """
+ return self._properties.get("computeRatioMax")
+
+ @property
+ def write_ms_avg(self):
+ """Union[int, None]: Milliseconds the average worker spent writing
+ output data.
+ """
+ return _helpers._int_or_none(self._properties.get("writeMsAvg"))
+
+ @property
+ def write_ms_max(self):
+ """Union[int, None]: Milliseconds the slowest worker spent writing
+ output data.
+ """
+ return _helpers._int_or_none(self._properties.get("writeMsMax"))
+
+ @property
+ def write_ratio_avg(self):
+ """Union[float, None]: Ratio of time the average worker spent writing
+ output data, relative to the longest time spent by any worker in any
+ stage of the overall plan.
+ """
+ return self._properties.get("writeRatioAvg")
+
+ @property
+ def write_ratio_max(self):
+ """Union[float, None]: Ratio of time the slowest worker spent writing
+ output data, relative to the longest time spent by any worker in any
+ stage of the overall plan.
+ """
+ return self._properties.get("writeRatioMax")
+
+ @property
+ def records_read(self):
+ """Union[int, None]: Number of records read by this stage."""
+ return _helpers._int_or_none(self._properties.get("recordsRead"))
+
+ @property
+ def records_written(self):
+ """Union[int, None]: Number of records written by this stage."""
+ return _helpers._int_or_none(self._properties.get("recordsWritten"))
+
+ @property
+ def status(self):
+ """Union[str, None]: status of this stage."""
+ return self._properties.get("status")
+
+ @property
+ def shuffle_output_bytes(self):
+ """Union[int, None]: Number of bytes written by this stage to
+ intermediate shuffle.
+ """
+ return _helpers._int_or_none(self._properties.get("shuffleOutputBytes"))
+
+ @property
+ def shuffle_output_bytes_spilled(self):
+ """Union[int, None]: Number of bytes written by this stage to
+ intermediate shuffle and spilled to disk.
+ """
+ return _helpers._int_or_none(self._properties.get("shuffleOutputBytesSpilled"))
+
+ @property
+ def steps(self):
+ """List(QueryPlanEntryStep): List of step operations performed by
+ each worker in the stage.
+ """
+ return [
+ QueryPlanEntryStep.from_api_repr(step)
+ for step in self._properties.get("steps", [])
+ ]
+
+
+class TimelineEntry(object):
+ """TimelineEntry represents progress of a query job at a particular
+ point in time.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs
+ for the underlying API representation within query statistics.
+
+ """
+
+ def __init__(self):
+ self._properties = {}
+
+ @classmethod
+ def from_api_repr(cls, resource):
+ """Factory: construct instance from the JSON repr.
+
+ Args:
+ resource(Dict[str: object]):
+ QueryTimelineSample representation returned from API
+
+ Returns:
+ google.cloud.bigquery.TimelineEntry:
+ Timeline sample parsed from ``resource``
+ """
+ entry = cls()
+ entry._properties = resource
+ return entry
+
+ @property
+ def elapsed_ms(self):
+ """Union[int, None]: Milliseconds elapsed since start of query
+ execution."""
+ return _helpers._int_or_none(self._properties.get("elapsedMs"))
+
+ @property
+ def active_units(self):
+ """Union[int, None]: Current number of input units being processed
+ by workers, reported as largest value since the last sample."""
+ return _helpers._int_or_none(self._properties.get("activeUnits"))
+
+ @property
+ def pending_units(self):
+ """Union[int, None]: Current number of input units remaining for
+ query stages active at this sample time."""
+ return _helpers._int_or_none(self._properties.get("pendingUnits"))
+
+ @property
+ def completed_units(self):
+ """Union[int, None]: Current number of input units completed by
+ this query."""
+ return _helpers._int_or_none(self._properties.get("completedUnits"))
+
+ @property
+ def slot_millis(self):
+ """Union[int, None]: Cumulative slot-milliseconds consumed by
+ this query."""
+ return _helpers._int_or_none(self._properties.get("totalSlotMs"))
+
+
+class UnknownJob(_AsyncJob):
+ """A job whose type cannot be determined."""
+
+ @classmethod
+ def from_api_repr(cls, resource, client):
+ """Construct an UnknownJob from the JSON representation.
+
+ Args:
+ resource (dict): JSON representation of a job.
+ client (google.cloud.bigquery.client.Client):
+ Client connected to BigQuery API.
+
+ Returns:
+ UnknownJob: Job corresponding to the resource.
+ """
+ job_ref_properties = resource.get("jobReference", {"projectId": client.project})
+ job_ref = _JobReference._from_api_repr(job_ref_properties)
+ job = cls(job_ref, client)
+ # Populate the job reference with the project, even if it has been
+ # redacted, because we know it should equal that of the request.
+ resource["jobReference"] = job_ref_properties
+ job._properties = resource
+ return job
diff --git a/bigquery/google/cloud/bigquery/magics.py b/bigquery/google/cloud/bigquery/magics.py
new file mode 100644
index 000000000000..b2dae2511ec8
--- /dev/null
+++ b/bigquery/google/cloud/bigquery/magics.py
@@ -0,0 +1,494 @@
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""IPython Magics
+
+.. function:: %%bigquery
+
+ IPython cell magic to run a query and display the result as a DataFrame
+
+ .. code-block:: python
+
+ %%bigquery [] [--project ] [--use_legacy_sql]
+ [--verbose] [--params ]
+
+
+ Parameters:
+
+ * ```` (optional, line argument):
+ variable to store the query results. The results are not displayed if
+ this parameter is used.
+ * ``--project `` (optional, line argument):
+ Project to use for running the query. Defaults to the context
+ :attr:`~google.cloud.bigquery.magics.Context.project`.
+ * ``--use_bqstorage_api`` (optional, line argument):
+ Downloads the DataFrame using the BigQuery Storage API. To use this
+ option, install the ``google-cloud-bigquery-storage`` and ``fastavro``
+ packages, and `enable the BigQuery Storage API
+ `_.
+ * ``--use_legacy_sql`` (optional, line argument):
+ Runs the query using Legacy SQL syntax. Defaults to Standard SQL if
+ this argument not used.
+ * ``--verbose`` (optional, line argument):
+ If this flag is used, information including the query job ID and the
+ amount of time for the query to complete will not be cleared after the
+ query is finished. By default, this information will be displayed but
+ will be cleared after the query is finished.
+ * ``--params `` (optional, line argument):
+ If present, the argument following the ``--params`` flag must be
+ either:
+
+ * :class:`str` - A JSON string representation of a dictionary in the
+ format ``{"param_name": "param_value"}`` (ex. ``{"num": 17}``). Use
+ of the parameter in the query should be indicated with
+ ``@param_name``. See ``In[5]`` in the Examples section below.
+
+ * :class:`dict` reference - A reference to a ``dict`` in the format
+ ``{"param_name": "param_value"}``, where the value types must be JSON
+ serializable. The variable reference is indicated by a ``$`` before
+ the variable name (ex. ``$my_dict_var``). See ``In[6]`` and ``In[7]``
+ in the Examples section below.
+ * ```` (required, cell argument):
+ SQL query to run.
+
+ Returns:
+ A :class:`pandas.DataFrame` with the query results.
+
+ .. note::
+ All queries run using this magic will run using the context
+ :attr:`~google.cloud.bigquery.magics.Context.credentials`.
+
+ Examples:
+ The following examples can be run in an IPython notebook after loading
+ the bigquery IPython extension (see ``In[1]``) and setting up
+ Application Default Credentials.
+
+ .. code-block:: none
+
+ In [1]: %load_ext google.cloud.bigquery
+
+ In [2]: %%bigquery
+ ...: SELECT name, SUM(number) as count
+ ...: FROM `bigquery-public-data.usa_names.usa_1910_current`
+ ...: GROUP BY name
+ ...: ORDER BY count DESC
+ ...: LIMIT 3
+
+ Out[2]: name count
+ ...: -------------------
+ ...: 0 James 4987296
+ ...: 1 John 4866302
+ ...: 2 Robert 4738204
+
+ In [3]: %%bigquery df --project my-alternate-project --verbose
+ ...: SELECT name, SUM(number) as count
+ ...: FROM `bigquery-public-data.usa_names.usa_1910_current`
+ ...: WHERE gender = 'F'
+ ...: GROUP BY name
+ ...: ORDER BY count DESC
+ ...: LIMIT 3
+ Executing query with job ID: bf633912-af2c-4780-b568-5d868058632b
+ Query executing: 2.61s
+ Query complete after 2.92s
+
+ In [4]: df
+
+ Out[4]: name count
+ ...: ----------------------
+ ...: 0 Mary 3736239
+ ...: 1 Patricia 1568495
+ ...: 2 Elizabeth 1519946
+
+ In [5]: %%bigquery --params {"num": 17}
+ ...: SELECT @num AS num
+
+ Out[5]: num
+ ...: -------
+ ...: 0 17
+
+ In [6]: params = {"num": 17}
+
+ In [7]: %%bigquery --params $params
+ ...: SELECT @num AS num
+
+ Out[7]: num
+ ...: -------
+ ...: 0 17
+"""
+
+from __future__ import print_function
+
+import ast
+import sys
+import time
+from concurrent import futures
+
+try:
+ import IPython
+ from IPython import display
+ from IPython.core import magic_arguments
+except ImportError: # pragma: NO COVER
+ raise ImportError("This module can only be loaded in IPython.")
+
+from google.api_core import client_info
+import google.auth
+from google.cloud import bigquery
+from google.cloud.bigquery.dbapi import _helpers
+import six
+
+
+IPYTHON_USER_AGENT = "ipython-{}".format(IPython.__version__)
+
+
+class Context(object):
+ """Storage for objects to be used throughout an IPython notebook session.
+
+ A Context object is initialized when the ``magics`` module is imported,
+ and can be found at ``google.cloud.bigquery.magics.context``.
+ """
+
+ def __init__(self):
+ self._credentials = None
+ self._project = None
+ self._connection = None
+ self._use_bqstorage_api = None
+ self._default_query_job_config = bigquery.QueryJobConfig()
+
+ @property
+ def credentials(self):
+ """google.auth.credentials.Credentials: Credentials to use for queries
+ performed through IPython magics
+
+ Note:
+ These credentials do not need to be explicitly defined if you are
+ using Application Default Credentials. If you are not using
+ Application Default Credentials, manually construct a
+ :class:`google.auth.credentials.Credentials` object and set it as
+ the context credentials as demonstrated in the example below. See
+ `auth docs`_ for more information on obtaining credentials.
+
+ Example:
+ Manually setting the context credentials:
+
+ >>> from google.cloud.bigquery import magics
+ >>> from google.oauth2 import service_account
+ >>> credentials = (service_account
+ ... .Credentials.from_service_account_file(
+ ... '/path/to/key.json'))
+ >>> magics.context.credentials = credentials
+
+
+ .. _auth docs: http://google-auth.readthedocs.io
+ /en/latest/user-guide.html#obtaining-credentials
+ """
+ if self._credentials is None:
+ self._credentials, _ = google.auth.default()
+ return self._credentials
+
+ @credentials.setter
+ def credentials(self, value):
+ self._credentials = value
+
+ @property
+ def project(self):
+ """str: Default project to use for queries performed through IPython
+ magics
+
+ Note:
+ The project does not need to be explicitly defined if you have an
+ environment default project set. If you do not have a default
+ project set in your environment, manually assign the project as
+ demonstrated in the example below.
+
+ Example:
+ Manually setting the context project:
+
+ >>> from google.cloud.bigquery import magics
+ >>> magics.context.project = 'my-project'
+ """
+ if self._project is None:
+ _, self._project = google.auth.default()
+ return self._project
+
+ @project.setter
+ def project(self, value):
+ self._project = value
+
+ @property
+ def use_bqstorage_api(self):
+ """bool: [Beta] Set to True to use the BigQuery Storage API to
+ download query results
+
+ To use this option, install the ``google-cloud-bigquery-storage`` and
+ ``fastavro`` packages, and `enable the BigQuery Storage API
+ `_.
+ """
+ return self._use_bqstorage_api
+
+ @use_bqstorage_api.setter
+ def use_bqstorage_api(self, value):
+ self._use_bqstorage_api = value
+
+ @property
+ def default_query_job_config(self):
+ """google.cloud.bigquery.job.QueryJobConfig: Default job
+ configuration for queries.
+
+ The context's :class:`~google.cloud.bigquery.job.QueryJobConfig` is
+ used for queries. Some properties can be overridden with arguments to
+ the magics.
+
+ Example:
+ Manually setting the default value for ``maximum_bytes_billed``
+ to 100 MB:
+
+ >>> from google.cloud.bigquery import magics
+ >>> magics.context.default_query_job_config.maximum_bytes_billed = 100000000
+ """
+ return self._default_query_job_config
+
+ @default_query_job_config.setter
+ def default_query_job_config(self, value):
+ self._default_query_job_config = value
+
+
+context = Context()
+
+
+def _run_query(client, query, job_config=None):
+ """Runs a query while printing status updates
+
+ Args:
+ client (google.cloud.bigquery.client.Client):
+ Client to bundle configuration needed for API requests.
+ query (str):
+ SQL query to be executed. Defaults to the standard SQL dialect.
+ Use the ``job_config`` parameter to change dialects.
+ job_config (google.cloud.bigquery.job.QueryJobConfig, optional):
+ Extra configuration options for the job.
+
+ Returns:
+ google.cloud.bigquery.job.QueryJob: the query job created
+
+ Example:
+ >>> client = bigquery.Client()
+ >>> _run_query(client, "SELECT 17")
+ Executing query with job ID: bf633912-af2c-4780-b568-5d868058632b
+ Query executing: 1.66s
+ Query complete after 2.07s
+ 'bf633912-af2c-4780-b568-5d868058632b'
+ """
+ start_time = time.time()
+ query_job = client.query(query, job_config=job_config)
+
+ if job_config and job_config.dry_run:
+ return query_job
+
+ print("Executing query with job ID: {}".format(query_job.job_id))
+
+ while True:
+ print("\rQuery executing: {:0.2f}s".format(time.time() - start_time), end="")
+ try:
+ query_job.result(timeout=0.5)
+ break
+ except futures.TimeoutError:
+ continue
+ print("\nQuery complete after {:0.2f}s".format(time.time() - start_time))
+ return query_job
+
+
+@magic_arguments.magic_arguments()
+@magic_arguments.argument(
+ "destination_var",
+ nargs="?",
+ help=("If provided, save the output to this variable instead of displaying it."),
+)
+@magic_arguments.argument(
+ "--project",
+ type=str,
+ default=None,
+ help=("Project to use for executing this query. Defaults to the context project."),
+)
+@magic_arguments.argument(
+ "--maximum_bytes_billed",
+ default=None,
+ help=(
+ "maximum_bytes_billed to use for executing this query. Defaults to "
+ "the context default_query_job_config.maximum_bytes_billed."
+ ),
+)
+@magic_arguments.argument(
+ "--dry_run",
+ action="store_true",
+ default=False,
+ help=(
+ "Sets query to be a dry run to estimate costs. "
+ "Defaults to executing the query instead of dry run if this argument is not used."
+ ),
+)
+@magic_arguments.argument(
+ "--use_legacy_sql",
+ action="store_true",
+ default=False,
+ help=(
+ "Sets query to use Legacy SQL instead of Standard SQL. Defaults to "
+ "Standard SQL if this argument is not used."
+ ),
+)
+@magic_arguments.argument(
+ "--use_bqstorage_api",
+ action="store_true",
+ default=False,
+ help=(
+ "[Beta] Use the BigQuery Storage API to download large query results. "
+ "To use this option, install the google-cloud-bigquery-storage and "
+ "fastavro packages, and enable the BigQuery Storage API."
+ ),
+)
+@magic_arguments.argument(
+ "--verbose",
+ action="store_true",
+ default=False,
+ help=(
+ "If set, print verbose output, including the query job ID and the "
+ "amount of time for the query to finish. By default, this "
+ "information will be displayed as the query runs, but will be "
+ "cleared after the query is finished."
+ ),
+)
+@magic_arguments.argument(
+ "--params",
+ nargs="+",
+ default=None,
+ help=(
+ "Parameters to format the query string. If present, the --params "
+ "flag should be followed by a string representation of a dictionary "
+ "in the format {'param_name': 'param_value'} (ex. {\"num\": 17}), "
+ "or a reference to a dictionary in the same format. The dictionary "
+ "reference can be made by including a '$' before the variable "
+ "name (ex. $my_dict_var)."
+ ),
+)
+def _cell_magic(line, query):
+ """Underlying function for bigquery cell magic
+
+ Note:
+ This function contains the underlying logic for the 'bigquery' cell
+ magic. This function is not meant to be called directly.
+
+ Args:
+ line (str): "%%bigquery" followed by arguments as required
+ query (str): SQL query to run
+
+ Returns:
+ pandas.DataFrame: the query results.
+ """
+ args = magic_arguments.parse_argstring(_cell_magic, line)
+
+ params = []
+ if args.params is not None:
+ try:
+ params = _helpers.to_query_parameters(
+ ast.literal_eval("".join(args.params))
+ )
+ except Exception:
+ raise SyntaxError(
+ "--params is not a correctly formatted JSON string or a JSON "
+ "serializable dictionary"
+ )
+
+ project = args.project or context.project
+ client = bigquery.Client(
+ project=project,
+ credentials=context.credentials,
+ default_query_job_config=context.default_query_job_config,
+ client_info=client_info.ClientInfo(user_agent=IPYTHON_USER_AGENT),
+ )
+ if context._connection:
+ client._connection = context._connection
+ bqstorage_client = _make_bqstorage_client(
+ args.use_bqstorage_api or context.use_bqstorage_api, context.credentials
+ )
+ job_config = bigquery.job.QueryJobConfig()
+ job_config.query_parameters = params
+ job_config.use_legacy_sql = args.use_legacy_sql
+ job_config.dry_run = args.dry_run
+
+ if args.maximum_bytes_billed == "None":
+ job_config.maximum_bytes_billed = 0
+ elif args.maximum_bytes_billed is not None:
+ value = int(args.maximum_bytes_billed)
+ job_config.maximum_bytes_billed = value
+
+ error = None
+ try:
+ query_job = _run_query(client, query, job_config)
+ except Exception as ex:
+ error = str(ex)
+
+ if not args.verbose:
+ display.clear_output()
+
+ if error:
+ if args.destination_var:
+ print(
+ "Could not save output to variable '{}'.".format(args.destination_var),
+ file=sys.stderr,
+ )
+ print("\nERROR:\n", error, file=sys.stderr)
+ return
+
+ if args.dry_run and args.destination_var:
+ IPython.get_ipython().push({args.destination_var: query_job})
+ return
+ elif args.dry_run:
+ print(
+ "Query validated. This query will process {} bytes.".format(
+ query_job.total_bytes_processed
+ )
+ )
+ return query_job
+
+ result = query_job.to_dataframe(bqstorage_client=bqstorage_client)
+ if args.destination_var:
+ IPython.get_ipython().push({args.destination_var: result})
+ else:
+ return result
+
+
+def _make_bqstorage_client(use_bqstorage_api, credentials):
+ if not use_bqstorage_api:
+ return None
+
+ try:
+ from google.cloud import bigquery_storage_v1beta1
+ except ImportError as err:
+ customized_error = ImportError(
+ "Install the google-cloud-bigquery-storage and pyarrow packages "
+ "to use the BigQuery Storage API."
+ )
+ six.raise_from(customized_error, err)
+
+ try:
+ from google.api_core.gapic_v1 import client_info as gapic_client_info
+ except ImportError as err:
+ customized_error = ImportError(
+ "Install the grpcio package to use the BigQuery Storage API."
+ )
+ six.raise_from(customized_error, err)
+
+ return bigquery_storage_v1beta1.BigQueryStorageClient(
+ credentials=credentials,
+ client_info=gapic_client_info.ClientInfo(user_agent=IPYTHON_USER_AGENT),
+ )
diff --git a/bigquery/google/cloud/bigquery/model.py b/bigquery/google/cloud/bigquery/model.py
new file mode 100644
index 000000000000..4049a9232467
--- /dev/null
+++ b/bigquery/google/cloud/bigquery/model.py
@@ -0,0 +1,412 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Define resources for the BigQuery ML Models API."""
+
+import copy
+
+from google.protobuf import json_format
+import six
+
+import google.cloud._helpers
+from google.api_core import datetime_helpers
+from google.cloud.bigquery import _helpers
+from google.cloud.bigquery_v2 import types
+
+
+class Model(object):
+ """Model represents a machine learning model resource.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/models
+
+ Args:
+ model_ref (Union[ \
+ :class:`~google.cloud.bigquery.model.ModelReference`, \
+ str, \
+ ]):
+ A pointer to a model. If ``model_ref`` is a string, it must
+ included a project ID, dataset ID, and model ID, each separated
+ by ``.``.
+ """
+
+ _PROPERTY_TO_API_FIELD = {
+ "expires": "expirationTime",
+ "friendly_name": "friendlyName",
+ # Even though it's not necessary for field mapping to map when the
+ # property name equals the resource name, we add these here so that we
+ # have an exhaustive list of all mutable properties.
+ "labels": "labels",
+ "description": "description",
+ }
+
+ def __init__(self, model_ref):
+ # Use _proto on read-only properties to use it's built-in type
+ # conversion.
+ self._proto = types.Model()
+
+ # Use _properties on read-write properties to match the REST API
+ # semantics. The BigQuery API makes a distinction between an unset
+ # value, a null value, and a default value (0 or ""), but the protocol
+ # buffer classes do not.
+ self._properties = {}
+
+ if isinstance(model_ref, six.string_types):
+ model_ref = ModelReference.from_string(model_ref)
+
+ if model_ref:
+ self._proto.model_reference.CopyFrom(model_ref._proto)
+
+ @property
+ def reference(self):
+ """A :class:`~google.cloud.bigquery.model.ModelReference` pointing to
+ this model.
+
+ Read-only.
+
+ Returns:
+ google.cloud.bigquery.model.ModelReference: pointer to this model.
+ """
+ ref = ModelReference()
+ ref._proto = self._proto.model_reference
+ return ref
+
+ @property
+ def project(self):
+ """str: Project bound to the model"""
+ return self.reference.project
+
+ @property
+ def dataset_id(self):
+ """str: ID of dataset containing the model."""
+ return self.reference.dataset_id
+
+ @property
+ def model_id(self):
+ """str: The model ID."""
+ return self.reference.model_id
+
+ @property
+ def path(self):
+ """str: URL path for the model's APIs."""
+ return self.reference.path
+
+ @property
+ def location(self):
+ """str: The geographic location where the model resides. This value
+ is inherited from the dataset.
+
+ Read-only.
+ """
+ return self._proto.location
+
+ @property
+ def etag(self):
+ """str: ETag for the model resource (:data:`None` until
+ set from the server).
+
+ Read-only.
+ """
+ return self._proto.etag
+
+ @property
+ def created(self):
+ """Union[datetime.datetime, None]: Datetime at which the model was
+ created (:data:`None` until set from the server).
+
+ Read-only.
+ """
+ value = self._proto.creation_time
+ if value is not None and value != 0:
+ # value will be in milliseconds.
+ return google.cloud._helpers._datetime_from_microseconds(
+ 1000.0 * float(value)
+ )
+
+ @property
+ def modified(self):
+ """Union[datetime.datetime, None]: Datetime at which the model was last
+ modified (:data:`None` until set from the server).
+
+ Read-only.
+ """
+ value = self._proto.last_modified_time
+ if value is not None and value != 0:
+ # value will be in milliseconds.
+ return google.cloud._helpers._datetime_from_microseconds(
+ 1000.0 * float(value)
+ )
+
+ @property
+ def model_type(self):
+ """google.cloud.bigquery_v2.gapic.enums.Model.ModelType: Type of the
+ model resource.
+
+ Read-only.
+
+ The value is one of elements of the
+ :class:`~google.cloud.bigquery_v2.gapic.enums.Model.ModelType`
+ enumeration.
+ """
+ return self._proto.model_type
+
+ @property
+ def training_runs(self):
+ """Sequence[google.cloud.bigquery_v2.types.Model.TrainingRun]: Information
+ for all training runs in increasing order of start time.
+
+ Read-only.
+
+ An iterable of :class:`~google.cloud.bigquery_v2.types.Model.TrainingRun`.
+ """
+ return self._proto.training_runs
+
+ @property
+ def feature_columns(self):
+ """Sequence[google.cloud.bigquery_v2.types.StandardSqlField]: Input
+ feature columns that were used to train this model.
+
+ Read-only.
+
+ An iterable of :class:`~google.cloud.bigquery_v2.types.StandardSqlField`.
+ """
+ return self._proto.feature_columns
+
+ @property
+ def label_columns(self):
+ """Sequence[google.cloud.bigquery_v2.types.StandardSqlField]: Label
+ columns that were used to train this model. The output of the model
+ will have a ``predicted_`` prefix to these columns.
+
+ Read-only.
+
+ An iterable of :class:`~google.cloud.bigquery_v2.types.StandardSqlField`.
+ """
+ return self._proto.label_columns
+
+ @property
+ def expires(self):
+ """Union[datetime.datetime, None]: The datetime when this model
+ expires. If not present, the model will persist indefinitely. Expired
+ models will be deleted and their storage reclaimed.
+ """
+ value = self._properties.get("expirationTime")
+ if value is not None:
+ # value will be in milliseconds.
+ return google.cloud._helpers._datetime_from_microseconds(
+ 1000.0 * float(value)
+ )
+
+ @expires.setter
+ def expires(self, value):
+ if value is not None:
+ value = str(google.cloud._helpers._millis_from_datetime(value))
+ self._properties["expirationTime"] = value
+
+ @property
+ def description(self):
+ """Union[str, None]: Description of the model (defaults to
+ :data:`None`).
+ """
+ return self._properties.get("description")
+
+ @description.setter
+ def description(self, value):
+ self._properties["description"] = value
+
+ @property
+ def friendly_name(self):
+ """Union[str, None]: Title of the table (defaults to :data:`None`).
+
+ Raises:
+ ValueError: For invalid value types.
+ """
+ return self._properties.get("friendlyName")
+
+ @friendly_name.setter
+ def friendly_name(self, value):
+ self._properties["friendlyName"] = value
+
+ @property
+ def labels(self):
+ """Dict[str, str]: Labels for the table.
+
+ This method always returns a dict. To change a model's labels,
+ modify the dict, then call ``Client.update_model``. To delete a
+ label, set its value to :data:`None` before updating.
+ """
+ return self._properties.setdefault("labels", {})
+
+ @labels.setter
+ def labels(self, value):
+ if value is None:
+ value = {}
+ self._properties["labels"] = value
+
+ @classmethod
+ def from_api_repr(cls, resource):
+ """Factory: construct a model resource given its API representation
+
+ Args:
+ resource (Dict[str, object]):
+ Model resource representation from the API
+
+ Returns:
+ google.cloud.bigquery.model.Model: Model parsed from ``resource``.
+ """
+ this = cls(None)
+ # Keep a reference to the resource as a workaround to find unknown
+ # field values.
+ this._properties = resource
+
+ # Convert from millis-from-epoch to timestamp well-known type.
+ # TODO: Remove this hack once CL 238585470 hits prod.
+ resource = copy.deepcopy(resource)
+ for training_run in resource.get("trainingRuns", ()):
+ start_time = training_run.get("startTime")
+ if not start_time or "-" in start_time: # Already right format?
+ continue
+ start_time = datetime_helpers.from_microseconds(1e3 * float(start_time))
+ training_run["startTime"] = datetime_helpers.to_rfc3339(start_time)
+
+ this._proto = json_format.ParseDict(
+ resource, types.Model(), ignore_unknown_fields=True
+ )
+ return this
+
+ def _build_resource(self, filter_fields):
+ """Generate a resource for ``update``."""
+ return _helpers._build_resource_from_properties(self, filter_fields)
+
+ def __repr__(self):
+ return "Model(reference={})".format(repr(self.reference))
+
+
+class ModelReference(object):
+ """ModelReferences are pointers to models.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/models
+ """
+
+ def __init__(self):
+ self._proto = types.ModelReference()
+ self._properties = {}
+
+ @property
+ def project(self):
+ """str: Project bound to the model"""
+ return self._proto.project_id
+
+ @property
+ def dataset_id(self):
+ """str: ID of dataset containing the model."""
+ return self._proto.dataset_id
+
+ @property
+ def model_id(self):
+ """str: The model ID."""
+ return self._proto.model_id
+
+ @property
+ def path(self):
+ """str: URL path for the model's APIs."""
+ return "/projects/%s/datasets/%s/models/%s" % (
+ self._proto.project_id,
+ self._proto.dataset_id,
+ self._proto.model_id,
+ )
+
+ @classmethod
+ def from_api_repr(cls, resource):
+ """Factory: construct a model reference given its API representation
+
+ Args:
+ resource (Dict[str, object]):
+ Model reference representation returned from the API
+
+ Returns:
+ google.cloud.bigquery.model.ModelReference:
+ Model reference parsed from ``resource``.
+ """
+ ref = cls()
+ # Keep a reference to the resource as a workaround to find unknown
+ # field values.
+ ref._properties = resource
+ ref._proto = json_format.ParseDict(
+ resource, types.ModelReference(), ignore_unknown_fields=True
+ )
+ return ref
+
+ @classmethod
+ def from_string(cls, model_id, default_project=None):
+ """Construct a model reference from model ID string.
+
+ Args:
+ model_id (str):
+ A model ID in standard SQL format. If ``default_project``
+ is not specified, this must included a project ID, dataset
+ ID, and model ID, each separated by ``.``.
+ default_project (str):
+ Optional. The project ID to use when ``model_id`` does not
+ include a project ID.
+
+ Returns:
+ google.cloud.bigquery.model.ModelReference:
+ Model reference parsed from ``model_id``.
+
+ Raises:
+ ValueError:
+ If ``model_id`` is not a fully-qualified table ID in
+ standard SQL format.
+ """
+ proj, dset, model = _helpers._parse_3_part_id(
+ model_id, default_project=default_project, property_name="model_id"
+ )
+ return cls.from_api_repr(
+ {"projectId": proj, "datasetId": dset, "modelId": model}
+ )
+
+ def to_api_repr(self):
+ """Construct the API resource representation of this model reference.
+
+ Returns:
+ Dict[str, object]: Model reference represented as an API resource
+ """
+ return json_format.MessageToDict(self._proto)
+
+ def _key(self):
+ """Unique key for this model.
+
+ This is used for hashing a ModelReference.
+ """
+ return self.project, self.dataset_id, self.model_id
+
+ def __eq__(self, other):
+ if not isinstance(other, ModelReference):
+ return NotImplemented
+ return self._proto == other._proto
+
+ def __ne__(self, other):
+ return not self == other
+
+ def __hash__(self):
+ return hash(self._key())
+
+ def __repr__(self):
+ return "ModelReference(project='{}', dataset_id='{}', project_id='{}')".format(
+ self.project, self.dataset_id, self.model_id
+ )
diff --git a/bigquery/google/cloud/bigquery/query.py b/bigquery/google/cloud/bigquery/query.py
new file mode 100644
index 000000000000..4039be33db8c
--- /dev/null
+++ b/bigquery/google/cloud/bigquery/query.py
@@ -0,0 +1,632 @@
+# Copyright 2015 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""BigQuery query processing."""
+
+from collections import OrderedDict
+import copy
+
+from google.cloud.bigquery.table import _parse_schema_resource
+from google.cloud.bigquery._helpers import _rows_from_json
+from google.cloud.bigquery._helpers import _QUERY_PARAMS_FROM_JSON
+from google.cloud.bigquery._helpers import _SCALAR_VALUE_TO_JSON_PARAM
+
+
+class UDFResource(object):
+ """Describe a single user-defined function (UDF) resource.
+
+ :type udf_type: str
+ :param udf_type: the type of the resource ('inlineCode' or 'resourceUri')
+
+ :type value: str
+ :param value: the inline code or resource URI.
+
+ See
+ https://cloud.google.com/bigquery/user-defined-functions#api
+ """
+
+ def __init__(self, udf_type, value):
+ self.udf_type = udf_type
+ self.value = value
+
+ def __eq__(self, other):
+ if not isinstance(other, UDFResource):
+ return NotImplemented
+ return self.udf_type == other.udf_type and self.value == other.value
+
+ def __ne__(self, other):
+ return not self == other
+
+
+class _AbstractQueryParameter(object):
+ """Base class for named / positional query parameters.
+ """
+
+ @classmethod
+ def from_api_repr(cls, resource):
+ """Factory: construct parameter from JSON resource.
+
+ :type resource: dict
+ :param resource: JSON mapping of parameter
+
+ :rtype: :class:`~google.cloud.bigquery.query.ScalarQueryParameter`
+ """
+ raise NotImplementedError
+
+ def to_api_repr(self):
+ """Construct JSON API representation for the parameter.
+
+ :rtype: dict
+ """
+ raise NotImplementedError
+
+
+class ScalarQueryParameter(_AbstractQueryParameter):
+ """Named / positional query parameters for scalar values.
+
+ :type name: str or None
+ :param name: Parameter name, used via ``@foo`` syntax. If None, the
+ parameter can only be addressed via position (``?``).
+
+ :type type_: str
+ :param type_: name of parameter type. One of 'STRING', 'INT64',
+ 'FLOAT64', 'NUMERIC', 'BOOL', 'TIMESTAMP', 'DATETIME', or
+ 'DATE'.
+
+ :type value: str, int, float, :class:`decimal.Decimal`, bool,
+ :class:`datetime.datetime`, or :class:`datetime.date`.
+ :param value: the scalar parameter value.
+ """
+
+ def __init__(self, name, type_, value):
+ self.name = name
+ self.type_ = type_
+ self.value = value
+
+ @classmethod
+ def positional(cls, type_, value):
+ """Factory for positional paramater.
+
+ :type type_: str
+ :param type_:
+ name of parameter type. One of 'STRING', 'INT64',
+ 'FLOAT64', 'NUMERIC', 'BOOL', 'TIMESTAMP', 'DATETIME', or
+ 'DATE'.
+
+ :type value: str, int, float, :class:`decimal.Decimal`, bool,
+ :class:`datetime.datetime`, or
+ :class:`datetime.date`.
+ :param value: the scalar parameter value.
+
+ :rtype: :class:`~google.cloud.bigquery.query.ScalarQueryParameter`
+ :returns: instance without name
+ """
+ return cls(None, type_, value)
+
+ @classmethod
+ def from_api_repr(cls, resource):
+ """Factory: construct parameter from JSON resource.
+
+ :type resource: dict
+ :param resource: JSON mapping of parameter
+
+ :rtype: :class:`~google.cloud.bigquery.query.ScalarQueryParameter`
+ :returns: instance
+ """
+ name = resource.get("name")
+ type_ = resource["parameterType"]["type"]
+
+ # parameterValue might not be present if JSON resource originates
+ # from the back-end - the latter omits it for None values.
+ value = resource.get("parameterValue", {}).get("value")
+ if value is not None:
+ converted = _QUERY_PARAMS_FROM_JSON[type_](value, None)
+ else:
+ converted = None
+
+ return cls(name, type_, converted)
+
+ def to_api_repr(self):
+ """Construct JSON API representation for the parameter.
+
+ :rtype: dict
+ :returns: JSON mapping
+ """
+ value = self.value
+ converter = _SCALAR_VALUE_TO_JSON_PARAM.get(self.type_)
+ if converter is not None:
+ value = converter(value)
+ resource = {
+ "parameterType": {"type": self.type_},
+ "parameterValue": {"value": value},
+ }
+ if self.name is not None:
+ resource["name"] = self.name
+ return resource
+
+ def _key(self):
+ """A tuple key that uniquely describes this field.
+
+ Used to compute this instance's hashcode and evaluate equality.
+
+ Returns:
+ tuple: The contents of this
+ :class:`~google.cloud.bigquery.query.ScalarQueryParameter`.
+ """
+ return (self.name, self.type_.upper(), self.value)
+
+ def __eq__(self, other):
+ if not isinstance(other, ScalarQueryParameter):
+ return NotImplemented
+ return self._key() == other._key()
+
+ def __ne__(self, other):
+ return not self == other
+
+ def __repr__(self):
+ return "ScalarQueryParameter{}".format(self._key())
+
+
+class ArrayQueryParameter(_AbstractQueryParameter):
+ """Named / positional query parameters for array values.
+
+ :type name: str or None
+ :param name: Parameter name, used via ``@foo`` syntax. If None, the
+ parameter can only be addressed via position (``?``).
+
+ :type array_type: str
+ :param array_type:
+ name of type of array elements. One of `'STRING'`, `'INT64'`,
+ `'FLOAT64'`, `'NUMERIC'`, `'BOOL'`, `'TIMESTAMP'`, or `'DATE'`.
+
+ :type values: list of appropriate scalar type.
+ :param values: the parameter array values.
+ """
+
+ def __init__(self, name, array_type, values):
+ self.name = name
+ self.array_type = array_type
+ self.values = values
+
+ @classmethod
+ def positional(cls, array_type, values):
+ """Factory for positional parameters.
+
+ :type array_type: str
+ :param array_type:
+ name of type of array elements. One of `'STRING'`, `'INT64'`,
+ `'FLOAT64'`, `'NUMERIC'`, `'BOOL'`, `'TIMESTAMP'`, or `'DATE'`.
+
+ :type values: list of appropriate scalar type
+ :param values: the parameter array values.
+
+ :rtype: :class:`~google.cloud.bigquery.query.ArrayQueryParameter`
+ :returns: instance without name
+ """
+ return cls(None, array_type, values)
+
+ @classmethod
+ def _from_api_repr_struct(cls, resource):
+ name = resource.get("name")
+ converted = []
+ # We need to flatten the array to use the StructQueryParameter
+ # parse code.
+ resource_template = {
+ # The arrayType includes all the types of the fields of the STRUCT
+ "parameterType": resource["parameterType"]["arrayType"]
+ }
+ for array_value in resource["parameterValue"]["arrayValues"]:
+ struct_resource = copy.deepcopy(resource_template)
+ struct_resource["parameterValue"] = array_value
+ struct_value = StructQueryParameter.from_api_repr(struct_resource)
+ converted.append(struct_value)
+ return cls(name, "STRUCT", converted)
+
+ @classmethod
+ def _from_api_repr_scalar(cls, resource):
+ name = resource.get("name")
+ array_type = resource["parameterType"]["arrayType"]["type"]
+ parameter_value = resource.get("parameterValue", {})
+ array_values = parameter_value.get("arrayValues", ())
+ values = [value["value"] for value in array_values]
+ converted = [
+ _QUERY_PARAMS_FROM_JSON[array_type](value, None) for value in values
+ ]
+ return cls(name, array_type, converted)
+
+ @classmethod
+ def from_api_repr(cls, resource):
+ """Factory: construct parameter from JSON resource.
+
+ :type resource: dict
+ :param resource: JSON mapping of parameter
+
+ :rtype: :class:`~google.cloud.bigquery.query.ArrayQueryParameter`
+ :returns: instance
+ """
+ array_type = resource["parameterType"]["arrayType"]["type"]
+ if array_type == "STRUCT":
+ return cls._from_api_repr_struct(resource)
+ return cls._from_api_repr_scalar(resource)
+
+ def to_api_repr(self):
+ """Construct JSON API representation for the parameter.
+
+ :rtype: dict
+ :returns: JSON mapping
+ """
+ values = self.values
+ if self.array_type == "RECORD" or self.array_type == "STRUCT":
+ reprs = [value.to_api_repr() for value in values]
+ a_type = reprs[0]["parameterType"]
+ a_values = [repr_["parameterValue"] for repr_ in reprs]
+ else:
+ a_type = {"type": self.array_type}
+ converter = _SCALAR_VALUE_TO_JSON_PARAM.get(self.array_type)
+ if converter is not None:
+ values = [converter(value) for value in values]
+ a_values = [{"value": value} for value in values]
+ resource = {
+ "parameterType": {"type": "ARRAY", "arrayType": a_type},
+ "parameterValue": {"arrayValues": a_values},
+ }
+ if self.name is not None:
+ resource["name"] = self.name
+ return resource
+
+ def _key(self):
+ """A tuple key that uniquely describes this field.
+
+ Used to compute this instance's hashcode and evaluate equality.
+
+ Returns:
+ tuple: The contents of this
+ :class:`~google.cloud.bigquery.query.ArrayQueryParameter`.
+ """
+ return (self.name, self.array_type.upper(), self.values)
+
+ def __eq__(self, other):
+ if not isinstance(other, ArrayQueryParameter):
+ return NotImplemented
+ return self._key() == other._key()
+
+ def __ne__(self, other):
+ return not self == other
+
+ def __repr__(self):
+ return "ArrayQueryParameter{}".format(self._key())
+
+
+class StructQueryParameter(_AbstractQueryParameter):
+ """Named / positional query parameters for struct values.
+
+ :type name: str or None
+ :param name: Parameter name, used via ``@foo`` syntax. If None, the
+ parameter can only be addressed via position (``?``).
+
+ :type sub_params:
+ tuple of :class:`~google.cloud.bigquery.query.ScalarQueryParameter`,
+ :class:`~google.cloud.bigquery.query.ArrayQueryParameter`, or
+ :class:`~google.cloud.bigquery.query.StructQueryParameter`
+ :param sub_params: the sub-parameters for the struct
+ """
+
+ def __init__(self, name, *sub_params):
+ self.name = name
+ types = self.struct_types = OrderedDict()
+ values = self.struct_values = {}
+ for sub in sub_params:
+ if isinstance(sub, self.__class__):
+ types[sub.name] = "STRUCT"
+ values[sub.name] = sub
+ elif isinstance(sub, ArrayQueryParameter):
+ types[sub.name] = "ARRAY"
+ values[sub.name] = sub
+ else:
+ types[sub.name] = sub.type_
+ values[sub.name] = sub.value
+
+ @classmethod
+ def positional(cls, *sub_params):
+ """Factory for positional parameters.
+
+ :type sub_params:
+ tuple of
+ :class:`~google.cloud.bigquery.query.ScalarQueryParameter`,
+ :class:`~google.cloud.bigquery.query.ArrayQueryParameter`, or
+ :class:`~google.cloud.bigquery.query.StructQueryParameter`
+ :param sub_params: the sub-parameters for the struct
+
+ :rtype: :class:`~google.cloud.bigquery.query.StructQueryParameter`
+ :returns: instance without name
+ """
+ return cls(None, *sub_params)
+
+ @classmethod
+ def from_api_repr(cls, resource):
+ """Factory: construct parameter from JSON resource.
+
+ :type resource: dict
+ :param resource: JSON mapping of parameter
+
+ :rtype: :class:`~google.cloud.bigquery.query.StructQueryParameter`
+ :returns: instance
+ """
+ name = resource.get("name")
+ instance = cls(name)
+ type_resources = {}
+ types = instance.struct_types
+ for item in resource["parameterType"]["structTypes"]:
+ types[item["name"]] = item["type"]["type"]
+ type_resources[item["name"]] = item["type"]
+ struct_values = resource["parameterValue"]["structValues"]
+ for key, value in struct_values.items():
+ type_ = types[key]
+ converted = None
+ if type_ == "STRUCT":
+ struct_resource = {
+ "name": key,
+ "parameterType": type_resources[key],
+ "parameterValue": value,
+ }
+ converted = StructQueryParameter.from_api_repr(struct_resource)
+ elif type_ == "ARRAY":
+ struct_resource = {
+ "name": key,
+ "parameterType": type_resources[key],
+ "parameterValue": value,
+ }
+ converted = ArrayQueryParameter.from_api_repr(struct_resource)
+ else:
+ value = value["value"]
+ converted = _QUERY_PARAMS_FROM_JSON[type_](value, None)
+ instance.struct_values[key] = converted
+ return instance
+
+ def to_api_repr(self):
+ """Construct JSON API representation for the parameter.
+
+ :rtype: dict
+ :returns: JSON mapping
+ """
+ s_types = {}
+ values = {}
+ for name, value in self.struct_values.items():
+ type_ = self.struct_types[name]
+ if type_ in ("STRUCT", "ARRAY"):
+ repr_ = value.to_api_repr()
+ s_types[name] = {"name": name, "type": repr_["parameterType"]}
+ values[name] = repr_["parameterValue"]
+ else:
+ s_types[name] = {"name": name, "type": {"type": type_}}
+ converter = _SCALAR_VALUE_TO_JSON_PARAM.get(type_)
+ if converter is not None:
+ value = converter(value)
+ values[name] = {"value": value}
+
+ resource = {
+ "parameterType": {
+ "type": "STRUCT",
+ "structTypes": [s_types[key] for key in self.struct_types],
+ },
+ "parameterValue": {"structValues": values},
+ }
+ if self.name is not None:
+ resource["name"] = self.name
+ return resource
+
+ def _key(self):
+ """A tuple key that uniquely describes this field.
+
+ Used to compute this instance's hashcode and evaluate equality.
+
+ Returns:
+ tuple: The contents of this
+ :class:`~google.cloud.biquery.ArrayQueryParameter`.
+ """
+ return (self.name, self.struct_types, self.struct_values)
+
+ def __eq__(self, other):
+ if not isinstance(other, StructQueryParameter):
+ return NotImplemented
+ return self._key() == other._key()
+
+ def __ne__(self, other):
+ return not self == other
+
+ def __repr__(self):
+ return "StructQueryParameter{}".format(self._key())
+
+
+class _QueryResults(object):
+ """Results of a query.
+
+ See:
+ https://g.co/cloud/bigquery/docs/reference/rest/v2/jobs/getQueryResults
+ """
+
+ def __init__(self, properties):
+ self._properties = {}
+ self._set_properties(properties)
+
+ @classmethod
+ def from_api_repr(cls, api_response):
+ return cls(api_response)
+
+ @property
+ def project(self):
+ """Project bound to the query job.
+
+ :rtype: str
+ :returns: the project that the query job is associated with.
+ """
+ return self._properties.get("jobReference", {}).get("projectId")
+
+ @property
+ def cache_hit(self):
+ """Query results served from cache.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query#cacheHit
+
+ :rtype: bool or ``NoneType``
+ :returns: True if the query results were served from cache (None
+ until set by the server).
+ """
+ return self._properties.get("cacheHit")
+
+ @property
+ def complete(self):
+ """Server completed query.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query#jobComplete
+
+ :rtype: bool or ``NoneType``
+ :returns: True if the query completed on the server (None
+ until set by the server).
+ """
+ return self._properties.get("jobComplete")
+
+ @property
+ def errors(self):
+ """Errors generated by the query.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query#errors
+
+ :rtype: list of mapping, or ``NoneType``
+ :returns: Mappings describing errors generated on the server (None
+ until set by the server).
+ """
+ return self._properties.get("errors")
+
+ @property
+ def job_id(self):
+ """Job ID of the query job these results are from.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query#jobReference
+
+ :rtype: string
+ :returns: Job ID of the query job.
+ """
+ return self._properties.get("jobReference", {}).get("jobId")
+
+ @property
+ def page_token(self):
+ """Token for fetching next bach of results.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query#pageToken
+
+ :rtype: str, or ``NoneType``
+ :returns: Token generated on the server (None until set by the server).
+ """
+ return self._properties.get("pageToken")
+
+ @property
+ def total_rows(self):
+ """Total number of rows returned by the query.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query#totalRows
+
+ :rtype: int, or ``NoneType``
+ :returns: Count generated on the server (None until set by the server).
+ """
+ total_rows = self._properties.get("totalRows")
+ if total_rows is not None:
+ return int(total_rows)
+
+ @property
+ def total_bytes_processed(self):
+ """Total number of bytes processed by the query.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query#totalBytesProcessed
+
+ :rtype: int, or ``NoneType``
+ :returns: Count generated on the server (None until set by the server).
+ """
+ total_bytes_processed = self._properties.get("totalBytesProcessed")
+ if total_bytes_processed is not None:
+ return int(total_bytes_processed)
+
+ @property
+ def num_dml_affected_rows(self):
+ """Total number of rows affected by a DML query.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query#numDmlAffectedRows
+
+ :rtype: int, or ``NoneType``
+ :returns: Count generated on the server (None until set by the server).
+ """
+ num_dml_affected_rows = self._properties.get("numDmlAffectedRows")
+ if num_dml_affected_rows is not None:
+ return int(num_dml_affected_rows)
+
+ @property
+ def rows(self):
+ """Query results.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query#rows
+
+ :rtype: list of :class:`~google.cloud.bigquery.table.Row`
+ :returns: fields describing the schema (None until set by the server).
+ """
+ return _rows_from_json(self._properties.get("rows", ()), self.schema)
+
+ @property
+ def schema(self):
+ """Schema for query results.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query#schema
+
+ :rtype: list of :class:`SchemaField`, or ``NoneType``
+ :returns: fields describing the schema (None until set by the server).
+ """
+ return _parse_schema_resource(self._properties.get("schema", {}))
+
+ def _set_properties(self, api_response):
+ """Update properties from resource in body of ``api_response``
+
+ :type api_response: dict
+ :param api_response: response returned from an API call
+ """
+ job_id_present = (
+ "jobReference" in api_response
+ and "jobId" in api_response["jobReference"]
+ and "projectId" in api_response["jobReference"]
+ )
+ if not job_id_present:
+ raise ValueError("QueryResult requires a job reference")
+
+ self._properties.clear()
+ self._properties.update(copy.deepcopy(api_response))
+
+
+def _query_param_from_api_repr(resource):
+ """Helper: construct concrete query parameter from JSON resource."""
+ qp_type = resource["parameterType"]
+ if "arrayType" in qp_type:
+ klass = ArrayQueryParameter
+ elif "structTypes" in qp_type:
+ klass = StructQueryParameter
+ else:
+ klass = ScalarQueryParameter
+ return klass.from_api_repr(resource)
diff --git a/bigquery/google/cloud/bigquery/retry.py b/bigquery/google/cloud/bigquery/retry.py
new file mode 100644
index 000000000000..4bc4b757f45d
--- /dev/null
+++ b/bigquery/google/cloud/bigquery/retry.py
@@ -0,0 +1,55 @@
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from google.api_core import exceptions
+from google.api_core import retry
+
+
+_RETRYABLE_REASONS = frozenset(
+ ["rateLimitExceeded", "backendError", "internalError", "badGateway"]
+)
+
+_UNSTRUCTURED_RETRYABLE_TYPES = (
+ exceptions.TooManyRequests,
+ exceptions.InternalServerError,
+ exceptions.BadGateway,
+)
+
+
+def _should_retry(exc):
+ """Predicate for determining when to retry.
+
+ We retry if and only if the 'reason' is 'backendError'
+ or 'rateLimitExceeded'.
+ """
+ if not hasattr(exc, "errors"):
+ return False
+
+ if len(exc.errors) == 0:
+ # Check for unstructured error returns, e.g. from GFE
+ return isinstance(exc, _UNSTRUCTURED_RETRYABLE_TYPES)
+
+ reason = exc.errors[0]["reason"]
+ return reason in _RETRYABLE_REASONS
+
+
+DEFAULT_RETRY = retry.Retry(predicate=_should_retry)
+"""The default retry object.
+
+Any method with a ``retry`` parameter will be retried automatically,
+with reasonable defaults. To disable retry, pass ``retry=None``.
+To modify the default retry behavior, call a ``with_XXX`` method
+on ``DEFAULT_RETRY``. For example, to change the deadline to 30 seconds,
+pass ``retry=bigquery.DEFAULT_RETRY.with_deadline(30)``.
+"""
diff --git a/bigquery/google/cloud/bigquery/routine.py b/bigquery/google/cloud/bigquery/routine.py
new file mode 100644
index 000000000000..d5bb752dfddb
--- /dev/null
+++ b/bigquery/google/cloud/bigquery/routine.py
@@ -0,0 +1,513 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Define resources for the BigQuery Routines API."""
+
+from google.protobuf import json_format
+import six
+
+import google.cloud._helpers
+from google.cloud.bigquery import _helpers
+import google.cloud.bigquery_v2.types
+
+
+class Routine(object):
+ """Resource representing a user-defined routine.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/routines
+
+ Args:
+ routine_ref (Union[ \
+ str, \
+ google.cloud.bigquery.routine.RoutineReference, \
+ ]):
+ A pointer to a routine. If ``routine_ref`` is a string, it must
+ included a project ID, dataset ID, and routine ID, each separated
+ by ``.``.
+ ``**kwargs`` (Dict):
+ Initial property values.
+ """
+
+ _PROPERTY_TO_API_FIELD = {
+ "arguments": "arguments",
+ "body": "definitionBody",
+ "created": "creationTime",
+ "etag": "etag",
+ "imported_libraries": "importedLibraries",
+ "language": "language",
+ "modified": "lastModifiedTime",
+ "reference": "routineReference",
+ "return_type": "returnType",
+ "type_": "routineType",
+ }
+
+ def __init__(self, routine_ref, **kwargs):
+ if isinstance(routine_ref, six.string_types):
+ routine_ref = RoutineReference.from_string(routine_ref)
+
+ self._properties = {"routineReference": routine_ref.to_api_repr()}
+ for property_name in kwargs:
+ setattr(self, property_name, kwargs[property_name])
+
+ @property
+ def reference(self):
+ """google.cloud.bigquery.routine.RoutineReference: Reference
+ describing the ID of this routine.
+ """
+ return RoutineReference.from_api_repr(
+ self._properties[self._PROPERTY_TO_API_FIELD["reference"]]
+ )
+
+ @property
+ def path(self):
+ """str: URL path for the routine's APIs."""
+ return self.reference.path
+
+ @property
+ def project(self):
+ """str: ID of the project containing the routine."""
+ return self.reference.project
+
+ @property
+ def dataset_id(self):
+ """str: ID of dataset containing the routine."""
+ return self.reference.dataset_id
+
+ @property
+ def routine_id(self):
+ """str: The routine ID."""
+ return self.reference.routine_id
+
+ @property
+ def etag(self):
+ """str: ETag for the resource (:data:`None` until set from the
+ server).
+
+ Read-only.
+ """
+ return self._properties.get(self._PROPERTY_TO_API_FIELD["etag"])
+
+ @property
+ def type_(self):
+ """str: The fine-grained type of the routine.
+
+ See:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/routines#RoutineType
+ """
+ return self._properties.get(self._PROPERTY_TO_API_FIELD["type_"])
+
+ @type_.setter
+ def type_(self, value):
+ self._properties[self._PROPERTY_TO_API_FIELD["type_"]] = value
+
+ @property
+ def created(self):
+ """Optional[datetime.datetime]: Datetime at which the routine was
+ created (:data:`None` until set from the server).
+
+ Read-only.
+ """
+ value = self._properties.get(self._PROPERTY_TO_API_FIELD["created"])
+ if value is not None and value != 0:
+ # value will be in milliseconds.
+ return google.cloud._helpers._datetime_from_microseconds(
+ 1000.0 * float(value)
+ )
+
+ @property
+ def modified(self):
+ """Optional[datetime.datetime]: Datetime at which the routine was
+ last modified (:data:`None` until set from the server).
+
+ Read-only.
+ """
+ value = self._properties.get(self._PROPERTY_TO_API_FIELD["modified"])
+ if value is not None and value != 0:
+ # value will be in milliseconds.
+ return google.cloud._helpers._datetime_from_microseconds(
+ 1000.0 * float(value)
+ )
+
+ @property
+ def language(self):
+ """Optional[str]: The language of the routine.
+
+ Defaults to ``SQL``.
+ """
+ return self._properties.get(self._PROPERTY_TO_API_FIELD["language"])
+
+ @language.setter
+ def language(self, value):
+ self._properties[self._PROPERTY_TO_API_FIELD["language"]] = value
+
+ @property
+ def arguments(self):
+ """List[google.cloud.bigquery.routine.RoutineArgument]: Input/output
+ argument of a function or a stored procedure.
+
+ In-place modification is not supported. To set, replace the entire
+ property value with the modified list of
+ :class:`~google.cloud.bigquery.routine.RoutineArgument` objects.
+ """
+ resources = self._properties.get(self._PROPERTY_TO_API_FIELD["arguments"], [])
+ return [RoutineArgument.from_api_repr(resource) for resource in resources]
+
+ @arguments.setter
+ def arguments(self, value):
+ if not value:
+ resource = []
+ else:
+ resource = [argument.to_api_repr() for argument in value]
+ self._properties[self._PROPERTY_TO_API_FIELD["arguments"]] = resource
+
+ @property
+ def return_type(self):
+ """google.cloud.bigquery_v2.types.StandardSqlDataType: Return type of
+ the routine.
+
+ If absent, the return type is inferred from
+ :attr:`~google.cloud.bigquery.routine.Routine.body` at query time in
+ each query that references this routine. If present, then the
+ evaluated result will be cast to the specified returned type at query
+ time.
+
+ See:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/routines#resource-routine
+ """
+ resource = self._properties.get(self._PROPERTY_TO_API_FIELD["return_type"])
+ if not resource:
+ return resource
+ output = google.cloud.bigquery_v2.types.StandardSqlDataType()
+ output = json_format.ParseDict(resource, output, ignore_unknown_fields=True)
+ return output
+
+ @return_type.setter
+ def return_type(self, value):
+ if value:
+ resource = json_format.MessageToDict(value)
+ else:
+ resource = None
+ self._properties[self._PROPERTY_TO_API_FIELD["return_type"]] = resource
+
+ @property
+ def imported_libraries(self):
+ """List[str]: The path of the imported JavaScript libraries.
+
+ The :attr:`~google.cloud.bigquery.routine.Routine.language` must
+ equal ``JAVACRIPT``.
+
+ Examples:
+ Set the ``imported_libraries`` to a list of Google Cloud Storage
+ URIs.
+
+ .. code-block:: python
+
+ routine = bigquery.Routine("proj.dataset.routine_id")
+ routine.imported_libraries = [
+ "gs://cloud-samples-data/bigquery/udfs/max-value.js",
+ ]
+ """
+ return self._properties.get(
+ self._PROPERTY_TO_API_FIELD["imported_libraries"], []
+ )
+
+ @imported_libraries.setter
+ def imported_libraries(self, value):
+ if not value:
+ resource = []
+ else:
+ resource = value
+ self._properties[self._PROPERTY_TO_API_FIELD["imported_libraries"]] = resource
+
+ @property
+ def body(self):
+ """str: The body of the routine."""
+ return self._properties.get(self._PROPERTY_TO_API_FIELD["body"])
+
+ @body.setter
+ def body(self, value):
+ self._properties[self._PROPERTY_TO_API_FIELD["body"]] = value
+
+ @classmethod
+ def from_api_repr(cls, resource):
+ """Factory: construct a routine given its API representation.
+
+ Args:
+ resource (Dict[str, object]):
+ Resource, as returned from the API.
+
+ Returns:
+ google.cloud.bigquery.routine.Routine:
+ Python object, as parsed from ``resource``.
+ """
+ ref = cls(RoutineReference.from_api_repr(resource["routineReference"]))
+ ref._properties = resource
+ return ref
+
+ def to_api_repr(self):
+ """Construct the API resource representation of this routine.
+
+ Returns:
+ Dict[str, object]:
+ Routine represented as an API resource.
+ """
+ return self._properties
+
+ def _build_resource(self, filter_fields):
+ """Generate a resource for ``update``."""
+ return _helpers._build_resource_from_properties(self, filter_fields)
+
+ def __repr__(self):
+ return "Routine('{}.{}.{}')".format(
+ self.project, self.dataset_id, self.routine_id
+ )
+
+
+class RoutineArgument(object):
+ """Input/output argument of a function or a stored procedure.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/routines
+
+ Args:
+ ``**kwargs`` (Dict):
+ Initial property values.
+ """
+
+ _PROPERTY_TO_API_FIELD = {
+ "data_type": "dataType",
+ "kind": "argumentKind",
+ # Even though it's not necessary for field mapping to map when the
+ # property name equals the resource name, we add these here so that we
+ # have an exhaustive list of all properties.
+ "name": "name",
+ "mode": "mode",
+ }
+
+ def __init__(self, **kwargs):
+ self._properties = {}
+ for property_name in kwargs:
+ setattr(self, property_name, kwargs[property_name])
+
+ @property
+ def name(self):
+ """Optional[str]: Name of this argument.
+
+ Can be absent for function return argument.
+ """
+ return self._properties.get(self._PROPERTY_TO_API_FIELD["name"])
+
+ @name.setter
+ def name(self, value):
+ self._properties[self._PROPERTY_TO_API_FIELD["name"]] = value
+
+ @property
+ def kind(self):
+ """Optional[str]: The kind of argument, for example ``FIXED_TYPE`` or
+ ``ANY_TYPE``.
+
+ See:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/routines#ArgumentKind
+ """
+ return self._properties.get(self._PROPERTY_TO_API_FIELD["kind"])
+
+ @kind.setter
+ def kind(self, value):
+ self._properties[self._PROPERTY_TO_API_FIELD["kind"]] = value
+
+ @property
+ def mode(self):
+ """Optional[str]: The input/output mode of the argument."""
+ return self._properties.get(self._PROPERTY_TO_API_FIELD["mode"])
+
+ @mode.setter
+ def mode(self, value):
+ self._properties[self._PROPERTY_TO_API_FIELD["mode"]] = value
+
+ @property
+ def data_type(self):
+ """Optional[google.cloud.bigquery_v2.types.StandardSqlDataType]: Type
+ of a variable, e.g., a function argument.
+
+ See:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/StandardSqlDataType
+ """
+ resource = self._properties.get(self._PROPERTY_TO_API_FIELD["data_type"])
+ if not resource:
+ return resource
+ output = google.cloud.bigquery_v2.types.StandardSqlDataType()
+ output = json_format.ParseDict(resource, output, ignore_unknown_fields=True)
+ return output
+
+ @data_type.setter
+ def data_type(self, value):
+ if value:
+ resource = json_format.MessageToDict(value)
+ else:
+ resource = None
+ self._properties[self._PROPERTY_TO_API_FIELD["data_type"]] = resource
+
+ @classmethod
+ def from_api_repr(cls, resource):
+ """Factory: construct a routine argument given its API representation.
+
+ Args:
+ resource (Dict[str, object]):
+ Resource, as returned from the API.
+
+ Returns:
+ google.cloud.bigquery.routine.RoutineArgument:
+ Python object, as parsed from ``resource``.
+ """
+ ref = cls()
+ ref._properties = resource
+ return ref
+
+ def to_api_repr(self):
+ """Construct the API resource representation of this routine argument.
+
+ Returns:
+ Dict[str, object]:
+ Routine argument represented as an API resource.
+ """
+ return self._properties
+
+ def __eq__(self, other):
+ if not isinstance(other, RoutineArgument):
+ return NotImplemented
+ return self._properties == other._properties
+
+ def __ne__(self, other):
+ return not self == other
+
+ def __repr__(self):
+ all_properties = [
+ "{}={}".format(property_name, repr(getattr(self, property_name)))
+ for property_name in sorted(self._PROPERTY_TO_API_FIELD)
+ ]
+ return "RoutineArgument({})".format(", ".join(all_properties))
+
+
+class RoutineReference(object):
+ """A pointer to a routine.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/routines
+ """
+
+ def __init__(self):
+ self._properties = {}
+
+ @property
+ def project(self):
+ """str: ID of the project containing the routine."""
+ return self._properties["projectId"]
+
+ @property
+ def dataset_id(self):
+ """str: ID of dataset containing the routine."""
+ return self._properties["datasetId"]
+
+ @property
+ def routine_id(self):
+ """str: The routine ID."""
+ return self._properties["routineId"]
+
+ @property
+ def path(self):
+ """str: URL path for the routine's APIs."""
+ return "/projects/%s/datasets/%s/routines/%s" % (
+ self.project,
+ self.dataset_id,
+ self.routine_id,
+ )
+
+ @classmethod
+ def from_api_repr(cls, resource):
+ """Factory: construct a routine reference given its API representation.
+
+ Args:
+ resource (Dict[str, object]):
+ Routine reference representation returned from the API.
+
+ Returns:
+ google.cloud.bigquery.routine.RoutineReference:
+ Routine reference parsed from ``resource``.
+ """
+ ref = cls()
+ ref._properties = resource
+ return ref
+
+ @classmethod
+ def from_string(cls, routine_id, default_project=None):
+ """Factory: construct a routine reference from routine ID string.
+
+ Args:
+ routine_id (str):
+ A routine ID in standard SQL format. If ``default_project``
+ is not specified, this must included a project ID, dataset
+ ID, and routine ID, each separated by ``.``.
+ default_project (str):
+ Optional. The project ID to use when ``routine_id`` does not
+ include a project ID.
+
+ Returns:
+ google.cloud.bigquery.routine.RoutineReference:
+ Routine reference parsed from ``routine_id``.
+
+ Raises:
+ ValueError:
+ If ``routine_id`` is not a fully-qualified routine ID in
+ standard SQL format.
+ """
+ proj, dset, routine = _helpers._parse_3_part_id(
+ routine_id, default_project=default_project, property_name="routine_id"
+ )
+ return cls.from_api_repr(
+ {"projectId": proj, "datasetId": dset, "routineId": routine}
+ )
+
+ def to_api_repr(self):
+ """Construct the API resource representation of this routine reference.
+
+ Returns:
+ Dict[str, object]:
+ Routine reference represented as an API resource.
+ """
+ return self._properties
+
+ def __eq__(self, other):
+ """Two RoutineReferences are equal if they point to the same routine."""
+ if not isinstance(other, RoutineReference):
+ return NotImplemented
+ return str(self) == str(other)
+
+ def __hash__(self):
+ return hash(str(self))
+
+ def __ne__(self, other):
+ return not self == other
+
+ def __repr__(self):
+ return "RoutineReference.from_string('{}')".format(str(self))
+
+ def __str__(self):
+ """String representation of the reference.
+
+ This is a fully-qualified ID, including the project ID and dataset ID.
+ """
+ return "{}.{}.{}".format(self.project, self.dataset_id, self.routine_id)
diff --git a/bigquery/google/cloud/bigquery/schema.py b/bigquery/google/cloud/bigquery/schema.py
new file mode 100644
index 000000000000..e0673d85baf6
--- /dev/null
+++ b/bigquery/google/cloud/bigquery/schema.py
@@ -0,0 +1,262 @@
+# Copyright 2015 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Schemas for BigQuery tables / queries."""
+
+from google.cloud.bigquery_v2 import types
+
+
+_STRUCT_TYPES = ("RECORD", "STRUCT")
+
+# SQL types reference:
+# https://cloud.google.com/bigquery/data-types#legacy_sql_data_types
+# https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types
+LEGACY_TO_STANDARD_TYPES = {
+ "STRING": types.StandardSqlDataType.STRING,
+ "BYTES": types.StandardSqlDataType.BYTES,
+ "INTEGER": types.StandardSqlDataType.INT64,
+ "INT64": types.StandardSqlDataType.INT64,
+ "FLOAT": types.StandardSqlDataType.FLOAT64,
+ "FLOAT64": types.StandardSqlDataType.FLOAT64,
+ "NUMERIC": types.StandardSqlDataType.NUMERIC,
+ "BOOLEAN": types.StandardSqlDataType.BOOL,
+ "BOOL": types.StandardSqlDataType.BOOL,
+ "GEOGRAPHY": types.StandardSqlDataType.GEOGRAPHY,
+ "RECORD": types.StandardSqlDataType.STRUCT,
+ "STRUCT": types.StandardSqlDataType.STRUCT,
+ "TIMESTAMP": types.StandardSqlDataType.TIMESTAMP,
+ "DATE": types.StandardSqlDataType.DATE,
+ "TIME": types.StandardSqlDataType.TIME,
+ "DATETIME": types.StandardSqlDataType.DATETIME,
+ # no direct conversion from ARRAY, the latter is represented by mode="REPEATED"
+}
+"""String names of the legacy SQL types to integer codes of Standard SQL types."""
+
+
+class SchemaField(object):
+ """Describe a single field within a table schema.
+
+ Args:
+ name (str): the name of the field.
+
+ field_type (str): the type of the field. See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#schema.fields.type
+
+ mode (str): the mode of the field. See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#schema.fields.mode
+
+ description (Optional[str]):description for the field.
+
+ fields (Tuple[:class:`~google.cloud.bigquery.schema.SchemaField`]):
+ subfields (requires ``field_type`` of 'RECORD').
+ """
+
+ def __init__(self, name, field_type, mode="NULLABLE", description=None, fields=()):
+ self._name = name
+ self._field_type = field_type
+ self._mode = mode
+ self._description = description
+ self._fields = tuple(fields)
+
+ @classmethod
+ def from_api_repr(cls, api_repr):
+ """Return a ``SchemaField`` object deserialized from a dictionary.
+
+ Args:
+ api_repr (Mapping[str, str]): The serialized representation
+ of the SchemaField, such as what is output by
+ :meth:`to_api_repr`.
+
+ Returns:
+ google.cloud.biquery.schema.SchemaField:
+ The ``SchemaField`` object.
+ """
+ # Handle optional properties with default values
+ mode = api_repr.get("mode", "NULLABLE")
+ description = api_repr.get("description")
+ fields = api_repr.get("fields", ())
+ return cls(
+ field_type=api_repr["type"].upper(),
+ fields=[cls.from_api_repr(f) for f in fields],
+ mode=mode.upper(),
+ description=description,
+ name=api_repr["name"],
+ )
+
+ @property
+ def name(self):
+ """str: The name of the field."""
+ return self._name
+
+ @property
+ def field_type(self):
+ """str: The type of the field.
+
+ See:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#schema.fields.type
+ """
+ return self._field_type
+
+ @property
+ def mode(self):
+ """str: The mode of the field.
+
+ See:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#schema.fields.mode
+ """
+ return self._mode
+
+ @property
+ def is_nullable(self):
+ """bool: whether 'mode' is 'nullable'."""
+ return self._mode == "NULLABLE"
+
+ @property
+ def description(self):
+ """Optional[str]: description for the field."""
+ return self._description
+
+ @property
+ def fields(self):
+ """tuple: Subfields contained in this field.
+
+ Must be empty unset if ``field_type`` is not 'RECORD'.
+ """
+ return self._fields
+
+ def to_api_repr(self):
+ """Return a dictionary representing this schema field.
+
+ Returns:
+ dict: A dictionary representing the SchemaField in a serialized
+ form.
+ """
+ # Put together the basic representation. See http://bit.ly/2hOAT5u.
+ answer = {
+ "mode": self.mode.upper(),
+ "name": self.name,
+ "type": self.field_type.upper(),
+ "description": self.description,
+ }
+
+ # If this is a RECORD type, then sub-fields are also included,
+ # add this to the serialized representation.
+ if self.field_type.upper() in _STRUCT_TYPES:
+ answer["fields"] = [f.to_api_repr() for f in self.fields]
+
+ # Done; return the serialized dictionary.
+ return answer
+
+ def _key(self):
+ """A tuple key that uniquely describes this field.
+
+ Used to compute this instance's hashcode and evaluate equality.
+
+ Returns:
+ tuple: The contents of this
+ :class:`~google.cloud.bigquery.schema.SchemaField`.
+ """
+ return (
+ self._name,
+ self._field_type.upper(),
+ self._mode.upper(),
+ self._description,
+ self._fields,
+ )
+
+ def to_standard_sql(self):
+ """Return the field as the standard SQL field representation object.
+
+ Returns:
+ An instance of :class:`~google.cloud.bigquery_v2.types.StandardSqlField`.
+ """
+ sql_type = types.StandardSqlDataType()
+
+ if self.mode == "REPEATED":
+ sql_type.type_kind = types.StandardSqlDataType.ARRAY
+ else:
+ sql_type.type_kind = LEGACY_TO_STANDARD_TYPES.get(
+ self.field_type, types.StandardSqlDataType.TYPE_KIND_UNSPECIFIED
+ )
+
+ if sql_type.type_kind == types.StandardSqlDataType.ARRAY: # noqa: E721
+ array_element_type = LEGACY_TO_STANDARD_TYPES.get(
+ self.field_type, types.StandardSqlDataType.TYPE_KIND_UNSPECIFIED
+ )
+ sql_type.array_element_type.type_kind = array_element_type
+
+ # ARRAY cannot directly contain other arrays, only scalar types and STRUCTs
+ # https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#array-type
+ if array_element_type == types.StandardSqlDataType.STRUCT: # noqa: E721
+ sql_type.array_element_type.struct_type.fields.extend(
+ field.to_standard_sql() for field in self.fields
+ )
+
+ elif sql_type.type_kind == types.StandardSqlDataType.STRUCT: # noqa: E721
+ sql_type.struct_type.fields.extend(
+ field.to_standard_sql() for field in self.fields
+ )
+
+ return types.StandardSqlField(name=self.name, type=sql_type)
+
+ def __eq__(self, other):
+ if not isinstance(other, SchemaField):
+ return NotImplemented
+ return self._key() == other._key()
+
+ def __ne__(self, other):
+ return not self == other
+
+ def __hash__(self):
+ return hash(self._key())
+
+ def __repr__(self):
+ return "SchemaField{}".format(self._key())
+
+
+def _parse_schema_resource(info):
+ """Parse a resource fragment into a schema field.
+
+ Args:
+ info: (Mapping[str->dict]): should contain a "fields" key to be parsed
+
+ Returns:
+ (Union[Sequence[:class:`google.cloud.bigquery.schema.SchemaField`],None])
+ a list of parsed fields, or ``None`` if no "fields" key found.
+ """
+ if "fields" not in info:
+ return ()
+
+ schema = []
+ for r_field in info["fields"]:
+ name = r_field["name"]
+ field_type = r_field["type"]
+ mode = r_field.get("mode", "NULLABLE")
+ description = r_field.get("description")
+ sub_fields = _parse_schema_resource(r_field)
+ schema.append(SchemaField(name, field_type, mode, description, sub_fields))
+ return schema
+
+
+def _build_schema_resource(fields):
+ """Generate a resource fragment for a schema.
+
+ Args:
+ fields [Sequence[:class:`~google.cloud.bigquery.schema.SchemaField`]):
+ schema to be dumped
+
+ Returns: (Sequence[dict])
+ mappings describing the schema of the supplied fields.
+ """
+ return [field.to_api_repr() for field in fields]
diff --git a/bigquery/google/cloud/bigquery/table.py b/bigquery/google/cloud/bigquery/table.py
new file mode 100644
index 000000000000..62072cf88804
--- /dev/null
+++ b/bigquery/google/cloud/bigquery/table.py
@@ -0,0 +1,1953 @@
+# Copyright 2015 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Define API Tables."""
+
+from __future__ import absolute_import
+
+import copy
+import datetime
+import functools
+import logging
+import operator
+import warnings
+
+import six
+
+try:
+ from google.cloud import bigquery_storage_v1beta1
+except ImportError: # pragma: NO COVER
+ bigquery_storage_v1beta1 = None
+
+try:
+ import pandas
+except ImportError: # pragma: NO COVER
+ pandas = None
+
+try:
+ import pyarrow
+except ImportError: # pragma: NO COVER
+ pyarrow = None
+
+try:
+ import tqdm
+except ImportError: # pragma: NO COVER
+ tqdm = None
+
+import google.api_core.exceptions
+from google.api_core.page_iterator import HTTPIterator
+
+import google.cloud._helpers
+from google.cloud.bigquery import _helpers
+from google.cloud.bigquery import _pandas_helpers
+from google.cloud.bigquery.schema import SchemaField
+from google.cloud.bigquery.schema import _build_schema_resource
+from google.cloud.bigquery.schema import _parse_schema_resource
+from google.cloud.bigquery.external_config import ExternalConfig
+
+
+_LOGGER = logging.getLogger(__name__)
+
+_NO_BQSTORAGE_ERROR = (
+ "The google-cloud-bigquery-storage library is not installed, "
+ "please install google-cloud-bigquery-storage to use bqstorage features."
+)
+_NO_PANDAS_ERROR = (
+ "The pandas library is not installed, please install "
+ "pandas to use the to_dataframe() function."
+)
+_NO_PYARROW_ERROR = (
+ "The pyarrow library is not installed, please install "
+ "pandas to use the to_arrow() function."
+)
+_NO_TQDM_ERROR = (
+ "A progress bar was requested, but there was an error loading the tqdm "
+ "library. Please install tqdm to use the progress bar functionality."
+)
+_TABLE_HAS_NO_SCHEMA = 'Table has no schema: call "client.get_table()"'
+
+
+def _reference_getter(table):
+ """A :class:`~google.cloud.bigquery.table.TableReference` pointing to
+ this table.
+
+ Returns:
+ google.cloud.bigquery.table.TableReference: pointer to this table.
+ """
+ from google.cloud.bigquery import dataset
+
+ dataset_ref = dataset.DatasetReference(table.project, table.dataset_id)
+ return TableReference(dataset_ref, table.table_id)
+
+
+def _view_use_legacy_sql_getter(table):
+ """bool: Specifies whether to execute the view with Legacy or Standard SQL.
+
+ This boolean specifies whether to execute the view with Legacy SQL
+ (:data:`True`) or Standard SQL (:data:`False`). The client side default is
+ :data:`False`. The server-side default is :data:`True`. If this table is
+ not a view, :data:`None` is returned.
+
+ Raises:
+ ValueError: For invalid value types.
+ """
+ view = table._properties.get("view")
+ if view is not None:
+ # The server-side default for useLegacySql is True.
+ return view.get("useLegacySql", True)
+ # In some cases, such as in a table list no view object is present, but the
+ # resource still represents a view. Use the type as a fallback.
+ if table.table_type == "VIEW":
+ # The server-side default for useLegacySql is True.
+ return True
+
+
+class EncryptionConfiguration(object):
+ """Custom encryption configuration (e.g., Cloud KMS keys).
+
+ Args:
+ kms_key_name (str): resource ID of Cloud KMS key used for encryption
+ """
+
+ def __init__(self, kms_key_name=None):
+ self._properties = {}
+ if kms_key_name is not None:
+ self._properties["kmsKeyName"] = kms_key_name
+
+ @property
+ def kms_key_name(self):
+ """str: Resource ID of Cloud KMS key
+
+ Resource ID of Cloud KMS key or :data:`None` if using default
+ encryption.
+ """
+ return self._properties.get("kmsKeyName")
+
+ @kms_key_name.setter
+ def kms_key_name(self, value):
+ self._properties["kmsKeyName"] = value
+
+ @classmethod
+ def from_api_repr(cls, resource):
+ """Construct an encryption configuration from its API representation
+
+ Args:
+ resource (Dict[str, object]):
+ An encryption configuration representation as returned from
+ the API.
+
+ Returns:
+ google.cloud.bigquery.table.EncryptionConfiguration:
+ An encryption configuration parsed from ``resource``.
+ """
+ config = cls()
+ config._properties = copy.deepcopy(resource)
+ return config
+
+ def to_api_repr(self):
+ """Construct the API resource representation of this encryption
+ configuration.
+
+ Returns:
+ Dict[str, object]:
+ Encryption configuration as represented as an API resource
+ """
+ return copy.deepcopy(self._properties)
+
+ def __eq__(self, other):
+ if not isinstance(other, EncryptionConfiguration):
+ return NotImplemented
+ return self.kms_key_name == other.kms_key_name
+
+ def __ne__(self, other):
+ return not self == other
+
+ def __hash__(self):
+ return hash(self.kms_key_name)
+
+ def __repr__(self):
+ return "EncryptionConfiguration({})".format(self.kms_key_name)
+
+
+class TableReference(object):
+ """TableReferences are pointers to tables.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables
+
+ Args:
+ dataset_ref (google.cloud.bigquery.dataset.DatasetReference):
+ A pointer to the dataset
+ table_id (str): The ID of the table
+ """
+
+ def __init__(self, dataset_ref, table_id):
+ self._project = dataset_ref.project
+ self._dataset_id = dataset_ref.dataset_id
+ self._table_id = table_id
+
+ @property
+ def project(self):
+ """str: Project bound to the table"""
+ return self._project
+
+ @property
+ def dataset_id(self):
+ """str: ID of dataset containing the table."""
+ return self._dataset_id
+
+ @property
+ def table_id(self):
+ """str: The table ID."""
+ return self._table_id
+
+ @property
+ def path(self):
+ """str: URL path for the table's APIs."""
+ return "/projects/%s/datasets/%s/tables/%s" % (
+ self._project,
+ self._dataset_id,
+ self._table_id,
+ )
+
+ @classmethod
+ def from_string(cls, table_id, default_project=None):
+ """Construct a table reference from table ID string.
+
+ Args:
+ table_id (str):
+ A table ID in standard SQL format. If ``default_project``
+ is not specified, this must included a project ID, dataset
+ ID, and table ID, each separated by ``.``.
+ default_project (str):
+ Optional. The project ID to use when ``table_id`` does not
+ include a project ID.
+
+ Returns:
+ TableReference: Table reference parsed from ``table_id``.
+
+ Examples:
+ >>> TableReference.from_string('my-project.mydataset.mytable')
+ TableRef...(DatasetRef...('my-project', 'mydataset'), 'mytable')
+
+ Raises:
+ ValueError:
+ If ``table_id`` is not a fully-qualified table ID in
+ standard SQL format.
+ """
+ from google.cloud.bigquery.dataset import DatasetReference
+
+ (
+ output_project_id,
+ output_dataset_id,
+ output_table_id,
+ ) = _helpers._parse_3_part_id(
+ table_id, default_project=default_project, property_name="table_id"
+ )
+
+ return cls(
+ DatasetReference(output_project_id, output_dataset_id), output_table_id
+ )
+
+ @classmethod
+ def from_api_repr(cls, resource):
+ """Factory: construct a table reference given its API representation
+
+ Args:
+ resource (Dict[str, object]):
+ Table reference representation returned from the API
+
+ Returns:
+ google.cloud.bigquery.table.TableReference:
+ Table reference parsed from ``resource``.
+ """
+ from google.cloud.bigquery.dataset import DatasetReference
+
+ project = resource["projectId"]
+ dataset_id = resource["datasetId"]
+ table_id = resource["tableId"]
+ return cls(DatasetReference(project, dataset_id), table_id)
+
+ def to_api_repr(self):
+ """Construct the API resource representation of this table reference.
+
+ Returns:
+ Dict[str, object]: Table reference represented as an API resource
+ """
+ return {
+ "projectId": self._project,
+ "datasetId": self._dataset_id,
+ "tableId": self._table_id,
+ }
+
+ def to_bqstorage(self):
+ """Construct a BigQuery Storage API representation of this table.
+
+ Install the ``google-cloud-bigquery-storage`` package to use this
+ feature.
+
+ If the ``table_id`` contains a partition identifier (e.g.
+ ``my_table$201812``) or a snapshot identifier (e.g.
+ ``mytable@1234567890``), it is ignored. Use
+ :class:`google.cloud.bigquery_storage_v1beta1.types.TableReadOptions`
+ to filter rows by partition. Use
+ :class:`google.cloud.bigquery_storage_v1beta1.types.TableModifiers`
+ to select a specific snapshot to read from.
+
+ Returns:
+ google.cloud.bigquery_storage_v1beta1.types.TableReference:
+ A reference to this table in the BigQuery Storage API.
+
+ Raises:
+ ValueError:
+ If the :mod:`google.cloud.bigquery_storage_v1beta1` module
+ cannot be imported.
+ """
+ if bigquery_storage_v1beta1 is None:
+ raise ValueError(_NO_BQSTORAGE_ERROR)
+
+ table_ref = bigquery_storage_v1beta1.types.TableReference()
+ table_ref.project_id = self._project
+ table_ref.dataset_id = self._dataset_id
+ table_id = self._table_id
+
+ if "@" in table_id:
+ table_id = table_id.split("@")[0]
+
+ if "$" in table_id:
+ table_id = table_id.split("$")[0]
+
+ table_ref.table_id = table_id
+
+ return table_ref
+
+ def _key(self):
+ """A tuple key that uniquely describes this field.
+
+ Used to compute this instance's hashcode and evaluate equality.
+
+ Returns:
+ Tuple[str]: The contents of this :class:`DatasetReference`.
+ """
+ return (self._project, self._dataset_id, self._table_id)
+
+ def __eq__(self, other):
+ if not isinstance(other, TableReference):
+ return NotImplemented
+ return self._key() == other._key()
+
+ def __ne__(self, other):
+ return not self == other
+
+ def __hash__(self):
+ return hash(self._key())
+
+ def __repr__(self):
+ from google.cloud.bigquery.dataset import DatasetReference
+
+ dataset_ref = DatasetReference(self._project, self._dataset_id)
+ return "TableReference({}, '{}')".format(repr(dataset_ref), self._table_id)
+
+
+class Table(object):
+ """Tables represent a set of rows whose values correspond to a schema.
+
+ See
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables
+
+ Args:
+ table_ref (Union[ \
+ :class:`~google.cloud.bigquery.table.TableReference`, \
+ str, \
+ ]):
+ A pointer to a table. If ``table_ref`` is a string, it must
+ included a project ID, dataset ID, and table ID, each separated
+ by ``.``.
+ schema (List[google.cloud.bigquery.schema.SchemaField]):
+ The table's schema
+ """
+
+ _PROPERTY_TO_API_FIELD = {
+ "friendly_name": "friendlyName",
+ "expires": "expirationTime",
+ "time_partitioning": "timePartitioning",
+ "partitioning_type": "timePartitioning",
+ "partition_expiration": "timePartitioning",
+ "view_use_legacy_sql": "view",
+ "view_query": "view",
+ "external_data_configuration": "externalDataConfiguration",
+ "encryption_configuration": "encryptionConfiguration",
+ }
+
+ def __init__(self, table_ref, schema=None):
+ table_ref = _table_arg_to_table_ref(table_ref)
+ self._properties = {"tableReference": table_ref.to_api_repr(), "labels": {}}
+ # Let the @property do validation.
+ if schema is not None:
+ self.schema = schema
+
+ @property
+ def project(self):
+ """str: Project bound to the table."""
+ return self._properties["tableReference"]["projectId"]
+
+ @property
+ def dataset_id(self):
+ """str: ID of dataset containing the table."""
+ return self._properties["tableReference"]["datasetId"]
+
+ @property
+ def table_id(self):
+ """str: ID of the table."""
+ return self._properties["tableReference"]["tableId"]
+
+ reference = property(_reference_getter)
+
+ @property
+ def path(self):
+ """str: URL path for the table's APIs."""
+ return "/projects/%s/datasets/%s/tables/%s" % (
+ self.project,
+ self.dataset_id,
+ self.table_id,
+ )
+
+ @property
+ def schema(self):
+ """List[google.cloud.bigquery.schema.SchemaField]: Table's schema.
+
+ Raises:
+ TypeError: If 'value' is not a sequence
+ ValueError:
+ If any item in the sequence is not a
+ :class:`~google.cloud.bigquery.schema.SchemaField`
+ """
+ prop = self._properties.get("schema")
+ if not prop:
+ return []
+ else:
+ return _parse_schema_resource(prop)
+
+ @schema.setter
+ def schema(self, value):
+ if value is None:
+ self._properties["schema"] = None
+ elif not all(isinstance(field, SchemaField) for field in value):
+ raise ValueError("Schema items must be fields")
+ else:
+ self._properties["schema"] = {"fields": _build_schema_resource(value)}
+
+ @property
+ def labels(self):
+ """Dict[str, str]: Labels for the table.
+
+ This method always returns a dict. To change a table's labels,
+ modify the dict, then call ``Client.update_table``. To delete a
+ label, set its value to :data:`None` before updating.
+
+ Raises:
+ ValueError: If ``value`` type is invalid.
+ """
+ return self._properties.setdefault("labels", {})
+
+ @labels.setter
+ def labels(self, value):
+ if not isinstance(value, dict):
+ raise ValueError("Pass a dict")
+ self._properties["labels"] = value
+
+ @property
+ def encryption_configuration(self):
+ """google.cloud.bigquery.table.EncryptionConfiguration: Custom
+ encryption configuration for the table.
+
+ Custom encryption configuration (e.g., Cloud KMS keys) or :data:`None`
+ if using default encryption.
+
+ See `protecting data with Cloud KMS keys
+ `_
+ in the BigQuery documentation.
+ """
+ prop = self._properties.get("encryptionConfiguration")
+ if prop is not None:
+ prop = EncryptionConfiguration.from_api_repr(prop)
+ return prop
+
+ @encryption_configuration.setter
+ def encryption_configuration(self, value):
+ api_repr = value
+ if value is not None:
+ api_repr = value.to_api_repr()
+ self._properties["encryptionConfiguration"] = api_repr
+
+ @property
+ def created(self):
+ """Union[datetime.datetime, None]: Datetime at which the table was
+ created (:data:`None` until set from the server).
+ """
+ creation_time = self._properties.get("creationTime")
+ if creation_time is not None:
+ # creation_time will be in milliseconds.
+ return google.cloud._helpers._datetime_from_microseconds(
+ 1000.0 * float(creation_time)
+ )
+
+ @property
+ def etag(self):
+ """Union[str, None]: ETag for the table resource (:data:`None` until
+ set from the server).
+ """
+ return self._properties.get("etag")
+
+ @property
+ def modified(self):
+ """Union[datetime.datetime, None]: Datetime at which the table was last
+ modified (:data:`None` until set from the server).
+ """
+ modified_time = self._properties.get("lastModifiedTime")
+ if modified_time is not None:
+ # modified_time will be in milliseconds.
+ return google.cloud._helpers._datetime_from_microseconds(
+ 1000.0 * float(modified_time)
+ )
+
+ @property
+ def num_bytes(self):
+ """Union[int, None]: The size of the table in bytes (:data:`None` until
+ set from the server).
+ """
+ return _helpers._int_or_none(self._properties.get("numBytes"))
+
+ @property
+ def num_rows(self):
+ """Union[int, None]: The number of rows in the table (:data:`None`
+ until set from the server).
+ """
+ return _helpers._int_or_none(self._properties.get("numRows"))
+
+ @property
+ def self_link(self):
+ """Union[str, None]: URL for the table resource (:data:`None` until set
+ from the server).
+ """
+ return self._properties.get("selfLink")
+
+ @property
+ def full_table_id(self):
+ """Union[str, None]: ID for the table (:data:`None` until set from the
+ server).
+
+ In the format ``project_id:dataset_id.table_id``.
+ """
+ return self._properties.get("id")
+
+ @property
+ def table_type(self):
+ """Union[str, None]: The type of the table (:data:`None` until set from
+ the server).
+
+ Possible values are ``'TABLE'``, ``'VIEW'``, or ``'EXTERNAL'``.
+ """
+ return self._properties.get("type")
+
+ @property
+ def time_partitioning(self):
+ """google.cloud.bigquery.table.TimePartitioning: Configures time-based
+ partitioning for a table.
+
+ Raises:
+ ValueError:
+ If the value is not :class:`TimePartitioning` or :data:`None`.
+ """
+ prop = self._properties.get("timePartitioning")
+ if prop is not None:
+ return TimePartitioning.from_api_repr(prop)
+
+ @time_partitioning.setter
+ def time_partitioning(self, value):
+ api_repr = value
+ if isinstance(value, TimePartitioning):
+ api_repr = value.to_api_repr()
+ elif value is not None:
+ raise ValueError(
+ "value must be google.cloud.bigquery.table.TimePartitioning " "or None"
+ )
+ self._properties["timePartitioning"] = api_repr
+
+ @property
+ def partitioning_type(self):
+ """Union[str, None]: Time partitioning of the table if it is
+ partitioned (Defaults to :data:`None`).
+
+ The only partitioning type that is currently supported is
+ :attr:`~google.cloud.bigquery.table.TimePartitioningType.DAY`.
+ """
+ warnings.warn(
+ "This method will be deprecated in future versions. Please use "
+ "Table.time_partitioning.type_ instead.",
+ PendingDeprecationWarning,
+ stacklevel=2,
+ )
+ if self.time_partitioning is not None:
+ return self.time_partitioning.type_
+
+ @partitioning_type.setter
+ def partitioning_type(self, value):
+ warnings.warn(
+ "This method will be deprecated in future versions. Please use "
+ "Table.time_partitioning.type_ instead.",
+ PendingDeprecationWarning,
+ stacklevel=2,
+ )
+ if self.time_partitioning is None:
+ self._properties["timePartitioning"] = {}
+ self._properties["timePartitioning"]["type"] = value
+
+ @property
+ def partition_expiration(self):
+ """Union[int, None]: Expiration time in milliseconds for a partition.
+
+ If :attr:`partition_expiration` is set and :attr:`type_` is
+ not set, :attr:`type_` will default to
+ :attr:`~google.cloud.bigquery.table.TimePartitioningType.DAY`.
+ """
+ warnings.warn(
+ "This method will be deprecated in future versions. Please use "
+ "Table.time_partitioning.expiration_ms instead.",
+ PendingDeprecationWarning,
+ stacklevel=2,
+ )
+ if self.time_partitioning is not None:
+ return self.time_partitioning.expiration_ms
+
+ @partition_expiration.setter
+ def partition_expiration(self, value):
+ warnings.warn(
+ "This method will be deprecated in future versions. Please use "
+ "Table.time_partitioning.expiration_ms instead.",
+ PendingDeprecationWarning,
+ stacklevel=2,
+ )
+ if self.time_partitioning is None:
+ self._properties["timePartitioning"] = {"type": TimePartitioningType.DAY}
+ self._properties["timePartitioning"]["expirationMs"] = str(value)
+
+ @property
+ def clustering_fields(self):
+ """Union[List[str], None]: Fields defining clustering for the table
+
+ (Defaults to :data:`None`).
+
+ Clustering fields are immutable after table creation.
+
+ .. note::
+
+ As of 2018-06-29, clustering fields cannot be set on a table
+ which does not also have time partioning defined.
+ """
+ prop = self._properties.get("clustering")
+ if prop is not None:
+ return list(prop.get("fields", ()))
+
+ @clustering_fields.setter
+ def clustering_fields(self, value):
+ """Union[List[str], None]: Fields defining clustering for the table
+
+ (Defaults to :data:`None`).
+ """
+ if value is not None:
+ prop = self._properties.setdefault("clustering", {})
+ prop["fields"] = value
+ else:
+ if "clustering" in self._properties:
+ del self._properties["clustering"]
+
+ @property
+ def description(self):
+ """Union[str, None]: Description of the table (defaults to
+ :data:`None`).
+
+ Raises:
+ ValueError: For invalid value types.
+ """
+ return self._properties.get("description")
+
+ @description.setter
+ def description(self, value):
+ if not isinstance(value, six.string_types) and value is not None:
+ raise ValueError("Pass a string, or None")
+ self._properties["description"] = value
+
+ @property
+ def expires(self):
+ """Union[datetime.datetime, None]: Datetime at which the table will be
+ deleted.
+
+ Raises:
+ ValueError: For invalid value types.
+ """
+ expiration_time = self._properties.get("expirationTime")
+ if expiration_time is not None:
+ # expiration_time will be in milliseconds.
+ return google.cloud._helpers._datetime_from_microseconds(
+ 1000.0 * float(expiration_time)
+ )
+
+ @expires.setter
+ def expires(self, value):
+ if not isinstance(value, datetime.datetime) and value is not None:
+ raise ValueError("Pass a datetime, or None")
+ value_ms = google.cloud._helpers._millis_from_datetime(value)
+ self._properties["expirationTime"] = _helpers._str_or_none(value_ms)
+
+ @property
+ def friendly_name(self):
+ """Union[str, None]: Title of the table (defaults to :data:`None`).
+
+ Raises:
+ ValueError: For invalid value types.
+ """
+ return self._properties.get("friendlyName")
+
+ @friendly_name.setter
+ def friendly_name(self, value):
+ if not isinstance(value, six.string_types) and value is not None:
+ raise ValueError("Pass a string, or None")
+ self._properties["friendlyName"] = value
+
+ @property
+ def location(self):
+ """Union[str, None]: Location in which the table is hosted
+
+ Defaults to :data:`None`.
+ """
+ return self._properties.get("location")
+
+ @property
+ def view_query(self):
+ """Union[str, None]: SQL query defining the table as a view (defaults
+ to :data:`None`).
+
+ By default, the query is treated as Standard SQL. To use Legacy
+ SQL, set :attr:`view_use_legacy_sql` to :data:`True`.
+
+ Raises:
+ ValueError: For invalid value types.
+ """
+ view = self._properties.get("view")
+ if view is not None:
+ return view.get("query")
+
+ @view_query.setter
+ def view_query(self, value):
+ if not isinstance(value, six.string_types):
+ raise ValueError("Pass a string")
+ view = self._properties.get("view")
+ if view is None:
+ view = self._properties["view"] = {}
+ view["query"] = value
+ # The service defaults useLegacySql to True, but this
+ # client uses Standard SQL by default.
+ if view.get("useLegacySql") is None:
+ view["useLegacySql"] = False
+
+ @view_query.deleter
+ def view_query(self):
+ """Delete SQL query defining the table as a view."""
+ self._properties.pop("view", None)
+
+ view_use_legacy_sql = property(_view_use_legacy_sql_getter)
+
+ @view_use_legacy_sql.setter
+ def view_use_legacy_sql(self, value):
+ if not isinstance(value, bool):
+ raise ValueError("Pass a boolean")
+ if self._properties.get("view") is None:
+ self._properties["view"] = {}
+ self._properties["view"]["useLegacySql"] = value
+
+ @property
+ def streaming_buffer(self):
+ """google.cloud.bigquery.StreamingBuffer: Information about a table's
+ streaming buffer.
+ """
+ sb = self._properties.get("streamingBuffer")
+ if sb is not None:
+ return StreamingBuffer(sb)
+
+ @property
+ def external_data_configuration(self):
+ """Union[google.cloud.bigquery.ExternalConfig, None]: Configuration for
+ an external data source (defaults to :data:`None`).
+
+ Raises:
+ ValueError: For invalid value types.
+ """
+ prop = self._properties.get("externalDataConfiguration")
+ if prop is not None:
+ prop = ExternalConfig.from_api_repr(prop)
+ return prop
+
+ @external_data_configuration.setter
+ def external_data_configuration(self, value):
+ if not (value is None or isinstance(value, ExternalConfig)):
+ raise ValueError("Pass an ExternalConfig or None")
+ api_repr = value
+ if value is not None:
+ api_repr = value.to_api_repr()
+ self._properties["externalDataConfiguration"] = api_repr
+
+ @classmethod
+ def from_string(cls, full_table_id):
+ """Construct a table from fully-qualified table ID.
+
+ Args:
+ full_table_id (str):
+ A fully-qualified table ID in standard SQL format. Must
+ included a project ID, dataset ID, and table ID, each
+ separated by ``.``.
+
+ Returns:
+ Table: Table parsed from ``full_table_id``.
+
+ Examples:
+ >>> Table.from_string('my-project.mydataset.mytable')
+ Table(TableRef...(D...('my-project', 'mydataset'), 'mytable'))
+
+ Raises:
+ ValueError:
+ If ``full_table_id`` is not a fully-qualified table ID in
+ standard SQL format.
+ """
+ return cls(TableReference.from_string(full_table_id))
+
+ @classmethod
+ def from_api_repr(cls, resource):
+ """Factory: construct a table given its API representation
+
+ Args:
+ resource (Dict[str, object]):
+ Table resource representation from the API
+
+ Returns:
+ google.cloud.bigquery.table.Table: Table parsed from ``resource``.
+
+ Raises:
+ KeyError:
+ If the ``resource`` lacks the key ``'tableReference'``, or if
+ the ``dict`` stored within the key ``'tableReference'`` lacks
+ the keys ``'tableId'``, ``'projectId'``, or ``'datasetId'``.
+ """
+ from google.cloud.bigquery import dataset
+
+ if (
+ "tableReference" not in resource
+ or "tableId" not in resource["tableReference"]
+ ):
+ raise KeyError(
+ "Resource lacks required identity information:"
+ '["tableReference"]["tableId"]'
+ )
+ project_id = resource["tableReference"]["projectId"]
+ table_id = resource["tableReference"]["tableId"]
+ dataset_id = resource["tableReference"]["datasetId"]
+ dataset_ref = dataset.DatasetReference(project_id, dataset_id)
+
+ table = cls(dataset_ref.table(table_id))
+ table._properties = resource
+
+ return table
+
+ def to_api_repr(self):
+ """Constructs the API resource of this table
+
+ Returns:
+ Dict[str, object]: Table represented as an API resource
+ """
+ return copy.deepcopy(self._properties)
+
+ def to_bqstorage(self):
+ """Construct a BigQuery Storage API representation of this table.
+
+ Returns:
+ google.cloud.bigquery_storage_v1beta1.types.TableReference:
+ A reference to this table in the BigQuery Storage API.
+ """
+ return self.reference.to_bqstorage()
+
+ def _build_resource(self, filter_fields):
+ """Generate a resource for ``update``."""
+ return _helpers._build_resource_from_properties(self, filter_fields)
+
+ def __repr__(self):
+ return "Table({})".format(repr(self.reference))
+
+
+class TableListItem(object):
+ """A read-only table resource from a list operation.
+
+ For performance reasons, the BigQuery API only includes some of the table
+ properties when listing tables. Notably,
+ :attr:`~google.cloud.bigquery.table.Table.schema` and
+ :attr:`~google.cloud.bigquery.table.Table.num_rows` are missing.
+
+ For a full list of the properties that the BigQuery API returns, see the
+ `REST documentation for tables.list
+ `_.
+
+
+ Args:
+ resource (Dict[str, object]):
+ A table-like resource object from a table list response. A
+ ``tableReference`` property is required.
+
+ Raises:
+ ValueError:
+ If ``tableReference`` or one of its required members is missing
+ from ``resource``.
+ """
+
+ def __init__(self, resource):
+ if "tableReference" not in resource:
+ raise ValueError("resource must contain a tableReference value")
+ if "projectId" not in resource["tableReference"]:
+ raise ValueError(
+ "resource['tableReference'] must contain a projectId value"
+ )
+ if "datasetId" not in resource["tableReference"]:
+ raise ValueError(
+ "resource['tableReference'] must contain a datasetId value"
+ )
+ if "tableId" not in resource["tableReference"]:
+ raise ValueError("resource['tableReference'] must contain a tableId value")
+
+ self._properties = resource
+
+ @property
+ def created(self):
+ """Union[datetime.datetime, None]: Datetime at which the table was
+ created (:data:`None` until set from the server).
+ """
+ creation_time = self._properties.get("creationTime")
+ if creation_time is not None:
+ # creation_time will be in milliseconds.
+ return google.cloud._helpers._datetime_from_microseconds(
+ 1000.0 * float(creation_time)
+ )
+
+ @property
+ def expires(self):
+ """Union[datetime.datetime, None]: Datetime at which the table will be
+ deleted.
+ """
+ expiration_time = self._properties.get("expirationTime")
+ if expiration_time is not None:
+ # expiration_time will be in milliseconds.
+ return google.cloud._helpers._datetime_from_microseconds(
+ 1000.0 * float(expiration_time)
+ )
+
+ @property
+ def project(self):
+ """str: Project bound to the table."""
+ return self._properties["tableReference"]["projectId"]
+
+ @property
+ def dataset_id(self):
+ """str: ID of dataset containing the table."""
+ return self._properties["tableReference"]["datasetId"]
+
+ @property
+ def table_id(self):
+ """str: ID of the table."""
+ return self._properties["tableReference"]["tableId"]
+
+ reference = property(_reference_getter)
+
+ @property
+ def labels(self):
+ """Dict[str, str]: Labels for the table.
+
+ This method always returns a dict. To change a table's labels,
+ modify the dict, then call ``Client.update_table``. To delete a
+ label, set its value to :data:`None` before updating.
+ """
+ return self._properties.setdefault("labels", {})
+
+ @property
+ def full_table_id(self):
+ """Union[str, None]: ID for the table (:data:`None` until set from the
+ server).
+
+ In the format ``project_id:dataset_id.table_id``.
+ """
+ return self._properties.get("id")
+
+ @property
+ def table_type(self):
+ """Union[str, None]: The type of the table (:data:`None` until set from
+ the server).
+
+ Possible values are ``'TABLE'``, ``'VIEW'``, or ``'EXTERNAL'``.
+ """
+ return self._properties.get("type")
+
+ @property
+ def time_partitioning(self):
+ """google.cloud.bigquery.table.TimePartitioning: Configures time-based
+ partitioning for a table.
+ """
+ prop = self._properties.get("timePartitioning")
+ if prop is not None:
+ return TimePartitioning.from_api_repr(prop)
+
+ @property
+ def partitioning_type(self):
+ """Union[str, None]: Time partitioning of the table if it is
+ partitioned (Defaults to :data:`None`).
+ """
+ warnings.warn(
+ "This method will be deprecated in future versions. Please use "
+ "TableListItem.time_partitioning.type_ instead.",
+ PendingDeprecationWarning,
+ stacklevel=2,
+ )
+ if self.time_partitioning is not None:
+ return self.time_partitioning.type_
+
+ @property
+ def partition_expiration(self):
+ """Union[int, None]: Expiration time in milliseconds for a partition.
+
+ If this property is set and :attr:`type_` is not set, :attr:`type_`
+ will default to :attr:`TimePartitioningType.DAY`.
+ """
+ warnings.warn(
+ "This method will be deprecated in future versions. Please use "
+ "TableListItem.time_partitioning.expiration_ms instead.",
+ PendingDeprecationWarning,
+ stacklevel=2,
+ )
+ if self.time_partitioning is not None:
+ return self.time_partitioning.expiration_ms
+
+ @property
+ def friendly_name(self):
+ """Union[str, None]: Title of the table (defaults to :data:`None`)."""
+ return self._properties.get("friendlyName")
+
+ view_use_legacy_sql = property(_view_use_legacy_sql_getter)
+
+ @property
+ def clustering_fields(self):
+ """Union[List[str], None]: Fields defining clustering for the table
+
+ (Defaults to :data:`None`).
+
+ Clustering fields are immutable after table creation.
+
+ .. note::
+
+ As of 2018-06-29, clustering fields cannot be set on a table
+ which does not also have time partioning defined.
+ """
+ prop = self._properties.get("clustering")
+ if prop is not None:
+ return list(prop.get("fields", ()))
+
+ @classmethod
+ def from_string(cls, full_table_id):
+ """Construct a table from fully-qualified table ID.
+
+ Args:
+ full_table_id (str):
+ A fully-qualified table ID in standard SQL format. Must
+ included a project ID, dataset ID, and table ID, each
+ separated by ``.``.
+
+ Returns:
+ Table: Table parsed from ``full_table_id``.
+
+ Examples:
+ >>> Table.from_string('my-project.mydataset.mytable')
+ Table(TableRef...(D...('my-project', 'mydataset'), 'mytable'))
+
+ Raises:
+ ValueError:
+ If ``full_table_id`` is not a fully-qualified table ID in
+ standard SQL format.
+ """
+ return cls(
+ {"tableReference": TableReference.from_string(full_table_id).to_api_repr()}
+ )
+
+ def to_bqstorage(self):
+ """Construct a BigQuery Storage API representation of this table.
+
+ Returns:
+ google.cloud.bigquery_storage_v1beta1.types.TableReference:
+ A reference to this table in the BigQuery Storage API.
+ """
+ return self.reference.to_bqstorage()
+
+
+def _row_from_mapping(mapping, schema):
+ """Convert a mapping to a row tuple using the schema.
+
+ Args:
+ mapping (Dict[str, object])
+ Mapping of row data: must contain keys for all required fields in
+ the schema. Keys which do not correspond to a field in the schema
+ are ignored.
+ schema (List[google.cloud.bigquery.schema.SchemaField]):
+ The schema of the table destination for the rows
+
+ Returns:
+ Tuple[object]:
+ Tuple whose elements are ordered according to the schema.
+
+ Raises:
+ ValueError: If schema is empty.
+ """
+ if len(schema) == 0:
+ raise ValueError(_TABLE_HAS_NO_SCHEMA)
+
+ row = []
+ for field in schema:
+ if field.mode == "REQUIRED":
+ row.append(mapping[field.name])
+ elif field.mode == "REPEATED":
+ row.append(mapping.get(field.name, ()))
+ elif field.mode == "NULLABLE":
+ row.append(mapping.get(field.name))
+ else:
+ raise ValueError("Unknown field mode: {}".format(field.mode))
+ return tuple(row)
+
+
+class StreamingBuffer(object):
+ """Information about a table's streaming buffer.
+
+ See https://cloud.google.com/bigquery/streaming-data-into-bigquery.
+
+ Args:
+ resource (Dict[str, object]):
+ streaming buffer representation returned from the API
+ """
+
+ def __init__(self, resource):
+ self.estimated_bytes = int(resource["estimatedBytes"])
+ self.estimated_rows = int(resource["estimatedRows"])
+ # time is in milliseconds since the epoch.
+ self.oldest_entry_time = google.cloud._helpers._datetime_from_microseconds(
+ 1000.0 * int(resource["oldestEntryTime"])
+ )
+
+
+class Row(object):
+ """A BigQuery row.
+
+ Values can be accessed by position (index), by key like a dict,
+ or as properties.
+
+ Args:
+ values (Sequence[object]): The row values
+ field_to_index (Dict[str, int]):
+ A mapping from schema field names to indexes
+ """
+
+ # Choose unusual field names to try to avoid conflict with schema fields.
+ __slots__ = ("_xxx_values", "_xxx_field_to_index")
+
+ def __init__(self, values, field_to_index):
+ self._xxx_values = values
+ self._xxx_field_to_index = field_to_index
+
+ def values(self):
+ """Return the values included in this row.
+
+ Returns:
+ Sequence[object]: A sequence of length ``len(row)``.
+ """
+ return copy.deepcopy(self._xxx_values)
+
+ def keys(self):
+ """Return the keys for using a row as a dict.
+
+ Returns:
+ Iterable[str]: The keys corresponding to the columns of a row
+
+ Examples:
+
+ >>> list(Row(('a', 'b'), {'x': 0, 'y': 1}).keys())
+ ['x', 'y']
+ """
+ return six.iterkeys(self._xxx_field_to_index)
+
+ def items(self):
+ """Return items as ``(key, value)`` pairs.
+
+ Returns:
+ Iterable[Tuple[str, object]]:
+ The ``(key, value)`` pairs representing this row.
+
+ Examples:
+
+ >>> list(Row(('a', 'b'), {'x': 0, 'y': 1}).items())
+ [('x', 'a'), ('y', 'b')]
+ """
+ for key, index in six.iteritems(self._xxx_field_to_index):
+ yield (key, copy.deepcopy(self._xxx_values[index]))
+
+ def get(self, key, default=None):
+ """Return a value for key, with a default value if it does not exist.
+
+ Args:
+ key (str): The key of the column to access
+ default (object):
+ The default value to use if the key does not exist. (Defaults
+ to :data:`None`.)
+
+ Returns:
+ object:
+ The value associated with the provided key, or a default value.
+
+ Examples:
+ When the key exists, the value associated with it is returned.
+
+ >>> Row(('a', 'b'), {'x': 0, 'y': 1}).get('x')
+ 'a'
+
+ The default value is :data:`None` when the key does not exist.
+
+ >>> Row(('a', 'b'), {'x': 0, 'y': 1}).get('z')
+ None
+
+ The default value can be overrided with the ``default`` parameter.
+
+ >>> Row(('a', 'b'), {'x': 0, 'y': 1}).get('z', '')
+ ''
+
+ >>> Row(('a', 'b'), {'x': 0, 'y': 1}).get('z', default = '')
+ ''
+ """
+ index = self._xxx_field_to_index.get(key)
+ if index is None:
+ return default
+ return self._xxx_values[index]
+
+ def __getattr__(self, name):
+ value = self._xxx_field_to_index.get(name)
+ if value is None:
+ raise AttributeError("no row field {!r}".format(name))
+ return self._xxx_values[value]
+
+ def __len__(self):
+ return len(self._xxx_values)
+
+ def __getitem__(self, key):
+ if isinstance(key, six.string_types):
+ value = self._xxx_field_to_index.get(key)
+ if value is None:
+ raise KeyError("no row field {!r}".format(key))
+ key = value
+ return self._xxx_values[key]
+
+ def __eq__(self, other):
+ if not isinstance(other, Row):
+ return NotImplemented
+ return (
+ self._xxx_values == other._xxx_values
+ and self._xxx_field_to_index == other._xxx_field_to_index
+ )
+
+ def __ne__(self, other):
+ return not self == other
+
+ def __repr__(self):
+ # sort field dict by value, for determinism
+ items = sorted(self._xxx_field_to_index.items(), key=operator.itemgetter(1))
+ f2i = "{" + ", ".join("%r: %d" % item for item in items) + "}"
+ return "Row({}, {})".format(self._xxx_values, f2i)
+
+
+class _NoopProgressBarQueue(object):
+ """A fake Queue class that does nothing.
+
+ This is used when there is no progress bar to send updates to.
+ """
+
+ def put_nowait(self, item):
+ """Don't actually do anything with the item."""
+
+
+class RowIterator(HTTPIterator):
+ """A class for iterating through HTTP/JSON API row list responses.
+
+ Args:
+ client (google.cloud.bigquery.Client): The API client.
+ api_request (Callable[google.cloud._http.JSONConnection.api_request]):
+ The function to use to make API requests.
+ path (str): The method path to query for the list of items.
+ page_token (str): A token identifying a page in a result set to start
+ fetching results from.
+ max_results (int, optional): The maximum number of results to fetch.
+ page_size (int, optional): The maximum number of rows in each page
+ of results from this request. Non-positive values are ignored.
+ Defaults to a sensible value set by the API.
+ extra_params (Dict[str, object]):
+ Extra query string parameters for the API call.
+ table (Union[ \
+ :class:`~google.cloud.bigquery.table.Table`, \
+ :class:`~google.cloud.bigquery.table.TableReference`, \
+ ]):
+ Optional. The table which these rows belong to, or a reference to
+ it. Used to call the BigQuery Storage API to fetch rows.
+ selected_fields (Sequence[ \
+ google.cloud.bigquery.schema.SchemaField, \
+ ]):
+ Optional. A subset of columns to select from this table.
+
+ """
+
+ def __init__(
+ self,
+ client,
+ api_request,
+ path,
+ schema,
+ page_token=None,
+ max_results=None,
+ page_size=None,
+ extra_params=None,
+ table=None,
+ selected_fields=None,
+ ):
+ super(RowIterator, self).__init__(
+ client,
+ api_request,
+ path,
+ item_to_value=_item_to_row,
+ items_key="rows",
+ page_token=page_token,
+ max_results=max_results,
+ extra_params=extra_params,
+ page_start=_rows_page_start,
+ next_token="pageToken",
+ )
+ self._field_to_index = _helpers._field_to_index_mapping(schema)
+ self._page_size = page_size
+ self._preserve_order = False
+ self._project = client.project
+ self._schema = schema
+ self._selected_fields = selected_fields
+ self._table = table
+ self._total_rows = getattr(table, "num_rows", None)
+
+ def _get_next_page_response(self):
+ """Requests the next page from the path provided.
+
+ Returns:
+ Dict[str, object]:
+ The parsed JSON response of the next page's contents.
+ """
+ params = self._get_query_params()
+ if self._page_size is not None:
+ params["maxResults"] = self._page_size
+ return self.api_request(
+ method=self._HTTP_METHOD, path=self.path, query_params=params
+ )
+
+ @property
+ def schema(self):
+ """List[google.cloud.bigquery.schema.SchemaField]: The subset of
+ columns to be read from the table."""
+ return list(self._schema)
+
+ @property
+ def total_rows(self):
+ """int: The total number of rows in the table."""
+ return self._total_rows
+
+ def _get_progress_bar(self, progress_bar_type):
+ """Construct a tqdm progress bar object, if tqdm is installed."""
+ if tqdm is None:
+ if progress_bar_type is not None:
+ warnings.warn(_NO_TQDM_ERROR, UserWarning, stacklevel=3)
+ return None
+
+ description = "Downloading"
+ unit = "rows"
+
+ try:
+ if progress_bar_type == "tqdm":
+ return tqdm.tqdm(desc=description, total=self.total_rows, unit=unit)
+ elif progress_bar_type == "tqdm_notebook":
+ return tqdm.tqdm_notebook(
+ desc=description, total=self.total_rows, unit=unit
+ )
+ elif progress_bar_type == "tqdm_gui":
+ return tqdm.tqdm_gui(desc=description, total=self.total_rows, unit=unit)
+ except (KeyError, TypeError):
+ # Protect ourselves from any tqdm errors. In case of
+ # unexpected tqdm behavior, just fall back to showing
+ # no progress bar.
+ warnings.warn(_NO_TQDM_ERROR, UserWarning, stacklevel=3)
+ return None
+
+ def _to_page_iterable(
+ self, bqstorage_download, tabledata_list_download, bqstorage_client=None
+ ):
+ if bqstorage_client is not None:
+ try:
+ # Iterate over the stream so that read errors are raised (and
+ # the method can then fallback to tabledata.list).
+ for item in bqstorage_download():
+ yield item
+ return
+ except google.api_core.exceptions.Forbidden:
+ # Don't hide errors such as insufficient permissions to create
+ # a read session, or the API is not enabled. Both of those are
+ # clearly problems if the developer has explicitly asked for
+ # BigQuery Storage API support.
+ raise
+ except google.api_core.exceptions.GoogleAPICallError:
+ # There is a known issue with reading from small anonymous
+ # query results tables, so some errors are expected. Rather
+ # than throw those errors, try reading the DataFrame again, but
+ # with the tabledata.list API.
+ pass
+
+ _LOGGER.debug(
+ "Started reading table '{}.{}.{}' with tabledata.list.".format(
+ self._table.project, self._table.dataset_id, self._table.table_id
+ )
+ )
+ for item in tabledata_list_download():
+ yield item
+
+ def _to_arrow_iterable(self, bqstorage_client=None):
+ """Create an iterable of arrow RecordBatches, to process the table as a stream."""
+ bqstorage_download = functools.partial(
+ _pandas_helpers.download_arrow_bqstorage,
+ self._project,
+ self._table,
+ bqstorage_client,
+ preserve_order=self._preserve_order,
+ selected_fields=self._selected_fields,
+ )
+ tabledata_list_download = functools.partial(
+ _pandas_helpers.download_arrow_tabledata_list, iter(self.pages), self.schema
+ )
+ return self._to_page_iterable(
+ bqstorage_download,
+ tabledata_list_download,
+ bqstorage_client=bqstorage_client,
+ )
+
+ # If changing the signature of this method, make sure to apply the same
+ # changes to job.QueryJob.to_arrow()
+ def to_arrow(self, progress_bar_type=None, bqstorage_client=None):
+ """[Beta] Create a class:`pyarrow.Table` by loading all pages of a
+ table or query.
+
+ Args:
+ progress_bar_type (Optional[str]):
+ If set, use the `tqdm `_ library to
+ display a progress bar while the data downloads. Install the
+ ``tqdm`` package to use this feature.
+
+ Possible values of ``progress_bar_type`` include:
+
+ ``None``
+ No progress bar.
+ ``'tqdm'``
+ Use the :func:`tqdm.tqdm` function to print a progress bar
+ to :data:`sys.stderr`.
+ ``'tqdm_notebook'``
+ Use the :func:`tqdm.tqdm_notebook` function to display a
+ progress bar as a Jupyter notebook widget.
+ ``'tqdm_gui'``
+ Use the :func:`tqdm.tqdm_gui` function to display a
+ progress bar as a graphical dialog box.
+ bqstorage_client ( \
+ google.cloud.bigquery_storage_v1beta1.BigQueryStorageClient \
+ ):
+ **Beta Feature** Optional. A BigQuery Storage API client. If
+ supplied, use the faster BigQuery Storage API to fetch rows
+ from BigQuery. This API is a billable API.
+
+ This method requires the ``pyarrow`` and
+ ``google-cloud-bigquery-storage`` libraries.
+
+ Reading from a specific partition or snapshot is not
+ currently supported by this method.
+
+ Returns:
+ pyarrow.Table
+ A :class:`pyarrow.Table` populated with row data and column
+ headers from the query results. The column headers are derived
+ from the destination table's schema.
+
+ Raises:
+ ValueError:
+ If the :mod:`pyarrow` library cannot be imported.
+
+ ..versionadded:: 1.17.0
+ """
+ if pyarrow is None:
+ raise ValueError(_NO_PYARROW_ERROR)
+
+ progress_bar = self._get_progress_bar(progress_bar_type)
+
+ record_batches = []
+ for record_batch in self._to_arrow_iterable(bqstorage_client=bqstorage_client):
+ record_batches.append(record_batch)
+
+ if progress_bar is not None:
+ # In some cases, the number of total rows is not populated
+ # until the first page of rows is fetched. Update the
+ # progress bar's total to keep an accurate count.
+ progress_bar.total = progress_bar.total or self.total_rows
+ progress_bar.update(record_batch.num_rows)
+
+ if progress_bar is not None:
+ # Indicate that the download has finished.
+ progress_bar.close()
+
+ if record_batches:
+ return pyarrow.Table.from_batches(record_batches)
+ else:
+ # No records, use schema based on BigQuery schema.
+ arrow_schema = _pandas_helpers.bq_to_arrow_schema(self._schema)
+ return pyarrow.Table.from_batches(record_batches, schema=arrow_schema)
+
+ def _to_dataframe_iterable(self, bqstorage_client=None, dtypes=None):
+ """Create an iterable of pandas DataFrames, to process the table as a stream.
+
+ See ``to_dataframe`` for argument descriptions.
+ """
+ column_names = [field.name for field in self._schema]
+ bqstorage_download = functools.partial(
+ _pandas_helpers.download_dataframe_bqstorage,
+ self._project,
+ self._table,
+ bqstorage_client,
+ column_names,
+ dtypes,
+ preserve_order=self._preserve_order,
+ selected_fields=self._selected_fields,
+ )
+ tabledata_list_download = functools.partial(
+ _pandas_helpers.download_dataframe_tabledata_list,
+ iter(self.pages),
+ self.schema,
+ dtypes,
+ )
+ return self._to_page_iterable(
+ bqstorage_download,
+ tabledata_list_download,
+ bqstorage_client=bqstorage_client,
+ )
+
+ # If changing the signature of this method, make sure to apply the same
+ # changes to job.QueryJob.to_dataframe()
+ def to_dataframe(self, bqstorage_client=None, dtypes=None, progress_bar_type=None):
+ """Create a pandas DataFrame by loading all pages of a query.
+
+ Args:
+ bqstorage_client ( \
+ google.cloud.bigquery_storage_v1beta1.BigQueryStorageClient \
+ ):
+ **Beta Feature** Optional. A BigQuery Storage API client. If
+ supplied, use the faster BigQuery Storage API to fetch rows
+ from BigQuery. This API is a billable API.
+
+ This method requires the ``pyarrow`` and
+ ``google-cloud-bigquery-storage`` libraries.
+
+ Reading from a specific partition or snapshot is not
+ currently supported by this method.
+
+ **Caution**: There is a known issue reading small anonymous
+ query result tables with the BQ Storage API. When a problem
+ is encountered reading a table, the tabledata.list method
+ from the BigQuery API is used, instead.
+ dtypes ( \
+ Map[str, Union[str, pandas.Series.dtype]] \
+ ):
+ Optional. A dictionary of column names pandas ``dtype``s. The
+ provided ``dtype`` is used when constructing the series for
+ the column specified. Otherwise, the default pandas behavior
+ is used.
+ progress_bar_type (Optional[str]):
+ If set, use the `tqdm `_ library to
+ display a progress bar while the data downloads. Install the
+ ``tqdm`` package to use this feature.
+
+ Possible values of ``progress_bar_type`` include:
+
+ ``None``
+ No progress bar.
+ ``'tqdm'``
+ Use the :func:`tqdm.tqdm` function to print a progress bar
+ to :data:`sys.stderr`.
+ ``'tqdm_notebook'``
+ Use the :func:`tqdm.tqdm_notebook` function to display a
+ progress bar as a Jupyter notebook widget.
+ ``'tqdm_gui'``
+ Use the :func:`tqdm.tqdm_gui` function to display a
+ progress bar as a graphical dialog box.
+
+ ..versionadded:: 1.11.0
+
+ Returns:
+ pandas.DataFrame:
+ A :class:`~pandas.DataFrame` populated with row data and column
+ headers from the query results. The column headers are derived
+ from the destination table's schema.
+
+ Raises:
+ ValueError:
+ If the :mod:`pandas` library cannot be imported, or the
+ :mod:`google.cloud.bigquery_storage_v1beta1` module is
+ required but cannot be imported.
+
+ """
+ if pandas is None:
+ raise ValueError(_NO_PANDAS_ERROR)
+ if dtypes is None:
+ dtypes = {}
+
+ progress_bar = self._get_progress_bar(progress_bar_type)
+
+ frames = []
+ for frame in self._to_dataframe_iterable(
+ bqstorage_client=bqstorage_client, dtypes=dtypes
+ ):
+ frames.append(frame)
+
+ if progress_bar is not None:
+ # In some cases, the number of total rows is not populated
+ # until the first page of rows is fetched. Update the
+ # progress bar's total to keep an accurate count.
+ progress_bar.total = progress_bar.total or self.total_rows
+ progress_bar.update(len(frame))
+
+ if progress_bar is not None:
+ # Indicate that the download has finished.
+ progress_bar.close()
+
+ # Avoid concatting an empty list.
+ if not frames:
+ column_names = [field.name for field in self._schema]
+ return pandas.DataFrame(columns=column_names)
+ return pandas.concat(frames, ignore_index=True)
+
+
+class _EmptyRowIterator(object):
+ """An empty row iterator.
+
+ This class prevents API requests when there are no rows to fetch or rows
+ are impossible to fetch, such as with query results for DDL CREATE VIEW
+ statements.
+ """
+
+ schema = ()
+ pages = ()
+ total_rows = 0
+
+ def to_arrow(self, progress_bar_type=None):
+ """[Beta] Create an empty class:`pyarrow.Table`.
+
+ Args:
+ progress_bar_type (Optional[str]):
+ Ignored. Added for compatibility with RowIterator.
+
+ Returns:
+ pyarrow.Table:
+ An empty :class:`pyarrow.Table`.
+ """
+ if pyarrow is None:
+ raise ValueError(_NO_PYARROW_ERROR)
+ return pyarrow.Table.from_arrays(())
+
+ def to_dataframe(self, bqstorage_client=None, dtypes=None, progress_bar_type=None):
+ """Create an empty dataframe.
+
+ Args:
+ bqstorage_client (Any):
+ Ignored. Added for compatibility with RowIterator.
+ dtypes (Any):
+ Ignored. Added for compatibility with RowIterator.
+ progress_bar_type (Any):
+ Ignored. Added for compatibility with RowIterator.
+
+ Returns:
+ pandas.DataFrame:
+ An empty :class:`~pandas.DataFrame`.
+ """
+ if pandas is None:
+ raise ValueError(_NO_PANDAS_ERROR)
+ return pandas.DataFrame()
+
+ def __iter__(self):
+ return iter(())
+
+
+class TimePartitioningType(object):
+ """Specifies the type of time partitioning to perform."""
+
+ DAY = "DAY"
+ """str: Generates one partition per day."""
+
+
+class TimePartitioning(object):
+ """Configures time-based partitioning for a table.
+
+ Args:
+ type_ (google.cloud.bigquery.table.TimePartitioningType, optional):
+ Specifies the type of time partitioning to perform. Defaults to
+ :attr:`~google.cloud.bigquery.table.TimePartitioningType.DAY`,
+ which is the only currently supported type.
+ field (str, optional):
+ If set, the table is partitioned by this field. If not set, the
+ table is partitioned by pseudo column ``_PARTITIONTIME``. The field
+ must be a top-level ``TIMESTAMP`` or ``DATE`` field. Its mode must
+ be ``NULLABLE`` or ``REQUIRED``.
+ expiration_ms(int, optional):
+ Number of milliseconds for which to keep the storage for a
+ partition.
+ require_partition_filter (bool, optional):
+ If set to true, queries over the partitioned table require a
+ partition filter that can be used for partition elimination to be
+ specified.
+ """
+
+ def __init__(
+ self, type_=None, field=None, expiration_ms=None, require_partition_filter=None
+ ):
+ self._properties = {}
+ if type_ is None:
+ self.type_ = TimePartitioningType.DAY
+ else:
+ self.type_ = type_
+ if field is not None:
+ self.field = field
+ if expiration_ms is not None:
+ self.expiration_ms = expiration_ms
+ if require_partition_filter is not None:
+ self.require_partition_filter = require_partition_filter
+
+ @property
+ def type_(self):
+ """google.cloud.bigquery.table.TimePartitioningType: The type of time
+ partitioning to use.
+ """
+ return self._properties.get("type")
+
+ @type_.setter
+ def type_(self, value):
+ self._properties["type"] = value
+
+ @property
+ def field(self):
+ """str: Field in the table to use for partitioning"""
+ return self._properties.get("field")
+
+ @field.setter
+ def field(self, value):
+ self._properties["field"] = value
+
+ @property
+ def expiration_ms(self):
+ """int: Number of milliseconds to keep the storage for a partition."""
+ return _helpers._int_or_none(self._properties.get("expirationMs"))
+
+ @expiration_ms.setter
+ def expiration_ms(self, value):
+ if value is not None:
+ # Allow explicitly setting the expiration to None.
+ value = str(value)
+ self._properties["expirationMs"] = value
+
+ @property
+ def require_partition_filter(self):
+ """bool: Specifies whether partition filters are required for queries
+ """
+ return self._properties.get("requirePartitionFilter")
+
+ @require_partition_filter.setter
+ def require_partition_filter(self, value):
+ self._properties["requirePartitionFilter"] = value
+
+ @classmethod
+ def from_api_repr(cls, api_repr):
+ """Return a :class:`TimePartitioning` object deserialized from a dict.
+
+ This method creates a new ``TimePartitioning`` instance that points to
+ the ``api_repr`` parameter as its internal properties dict. This means
+ that when a ``TimePartitioning`` instance is stored as a property of
+ another object, any changes made at the higher level will also appear
+ here::
+
+ >>> time_partitioning = TimePartitioning()
+ >>> table.time_partitioning = time_partitioning
+ >>> table.time_partitioning.field = 'timecolumn'
+ >>> time_partitioning.field
+ 'timecolumn'
+
+ Args:
+ api_repr (Mapping[str, str]):
+ The serialized representation of the TimePartitioning, such as
+ what is output by :meth:`to_api_repr`.
+
+ Returns:
+ google.cloud.bigquery.table.TimePartitioning:
+ The ``TimePartitioning`` object.
+ """
+ instance = cls()
+ instance._properties = api_repr
+ return instance
+
+ def to_api_repr(self):
+ """Return a dictionary representing this object.
+
+ This method returns the properties dict of the ``TimePartitioning``
+ instance rather than making a copy. This means that when a
+ ``TimePartitioning`` instance is stored as a property of another
+ object, any changes made at the higher level will also appear here.
+
+ Returns:
+ dict:
+ A dictionary representing the TimePartitioning object in
+ serialized form.
+ """
+ return self._properties
+
+ def _key(self):
+ return tuple(sorted(self._properties.items()))
+
+ def __eq__(self, other):
+ if not isinstance(other, TimePartitioning):
+ return NotImplemented
+ return self._key() == other._key()
+
+ def __ne__(self, other):
+ return not self == other
+
+ def __hash__(self):
+ return hash(self._key())
+
+ def __repr__(self):
+ key_vals = ["{}={}".format(key, val) for key, val in self._key()]
+ return "TimePartitioning({})".format(",".join(key_vals))
+
+
+def _item_to_row(iterator, resource):
+ """Convert a JSON row to the native object.
+
+ .. note::
+
+ This assumes that the ``schema`` attribute has been
+ added to the iterator after being created, which
+ should be done by the caller.
+
+ :type iterator: :class:`~google.api_core.page_iterator.Iterator`
+ :param iterator: The iterator that is currently in use.
+
+ :type resource: dict
+ :param resource: An item to be converted to a row.
+
+ :rtype: :class:`~google.cloud.bigquery.table.Row`
+ :returns: The next row in the page.
+ """
+ return Row(
+ _helpers._row_tuple_from_json(resource, iterator.schema),
+ iterator._field_to_index,
+ )
+
+
+def _tabledata_list_page_columns(schema, response):
+ """Make a generator of all the columns in a page from tabledata.list.
+
+ This enables creating a :class:`pandas.DataFrame` and other
+ column-oriented data structures such as :class:`pyarrow.RecordBatch`
+ """
+ columns = []
+ rows = response.get("rows", [])
+
+ def get_column_data(field_index, field):
+ for row in rows:
+ yield _helpers._field_from_json(row["f"][field_index]["v"], field)
+
+ for field_index, field in enumerate(schema):
+ columns.append(get_column_data(field_index, field))
+
+ return columns
+
+
+# pylint: disable=unused-argument
+def _rows_page_start(iterator, page, response):
+ """Grab total rows when :class:`~google.cloud.iterator.Page` starts.
+
+ :type iterator: :class:`~google.api_core.page_iterator.Iterator`
+ :param iterator: The iterator that is currently in use.
+
+ :type page: :class:`~google.api_core.page_iterator.Page`
+ :param page: The page that was just created.
+
+ :type response: dict
+ :param response: The JSON API response for a page of rows in a table.
+ """
+ # Make a (lazy) copy of the page in column-oriented format for use in data
+ # science packages.
+ page._columns = _tabledata_list_page_columns(iterator._schema, response)
+
+ total_rows = response.get("totalRows")
+ if total_rows is not None:
+ total_rows = int(total_rows)
+ iterator._total_rows = total_rows
+
+
+# pylint: enable=unused-argument
+
+
+def _table_arg_to_table_ref(value, default_project=None):
+ """Helper to convert a string or Table to TableReference.
+
+ This function keeps TableReference and other kinds of objects unchanged.
+ """
+ if isinstance(value, six.string_types):
+ value = TableReference.from_string(value, default_project=default_project)
+ if isinstance(value, (Table, TableListItem)):
+ value = value.reference
+ return value
+
+
+def _table_arg_to_table(value, default_project=None):
+ """Helper to convert a string or TableReference to a Table.
+
+ This function keeps Table and other kinds of objects unchanged.
+ """
+ if isinstance(value, six.string_types):
+ value = TableReference.from_string(value, default_project=default_project)
+ if isinstance(value, TableReference):
+ value = Table(value)
+ if isinstance(value, TableListItem):
+ newvalue = Table(value.reference)
+ newvalue._properties = value._properties
+ value = newvalue
+
+ return value
diff --git a/bigquery/google/cloud/bigquery_v2/__init__.py b/bigquery/google/cloud/bigquery_v2/__init__.py
new file mode 100644
index 000000000000..e582214329f3
--- /dev/null
+++ b/bigquery/google/cloud/bigquery_v2/__init__.py
@@ -0,0 +1,33 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import absolute_import
+
+import pkg_resources
+
+__version__ = pkg_resources.get_distribution("google-cloud-bigquery").version # noqa
+
+from google.cloud.bigquery_v2 import types
+from google.cloud.bigquery_v2.gapic import enums
+
+
+__all__ = (
+ # google.cloud.bigquery_v2
+ "__version__",
+ "types",
+ # google.cloud.bigquery_v2
+ "enums",
+)
diff --git a/bigquery/google/cloud/bigquery_v2/gapic/__init__.py b/bigquery/google/cloud/bigquery_v2/gapic/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/bigquery/google/cloud/bigquery_v2/gapic/enums.py b/bigquery/google/cloud/bigquery_v2/gapic/enums.py
new file mode 100644
index 000000000000..5d95f2590785
--- /dev/null
+++ b/bigquery/google/cloud/bigquery_v2/gapic/enums.py
@@ -0,0 +1,171 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Wrappers for protocol buffer enum types."""
+
+import enum
+
+
+class Model(object):
+ class DataSplitMethod(enum.IntEnum):
+ """
+ Indicates the method to split input data into multiple tables.
+
+ Attributes:
+ DATA_SPLIT_METHOD_UNSPECIFIED (int)
+ RANDOM (int): Splits data randomly.
+ CUSTOM (int): Splits data with the user provided tags.
+ SEQUENTIAL (int): Splits data sequentially.
+ NO_SPLIT (int): Data split will be skipped.
+ AUTO_SPLIT (int): Splits data automatically: Uses NO\_SPLIT if the data size is small.
+ Otherwise uses RANDOM.
+ """
+
+ DATA_SPLIT_METHOD_UNSPECIFIED = 0
+ RANDOM = 1
+ CUSTOM = 2
+ SEQUENTIAL = 3
+ NO_SPLIT = 4
+ AUTO_SPLIT = 5
+
+ class DistanceType(enum.IntEnum):
+ """
+ Distance metric used to compute the distance between two points.
+
+ Attributes:
+ DISTANCE_TYPE_UNSPECIFIED (int)
+ EUCLIDEAN (int): Eculidean distance.
+ COSINE (int): Cosine distance.
+ """
+
+ DISTANCE_TYPE_UNSPECIFIED = 0
+ EUCLIDEAN = 1
+ COSINE = 2
+
+ class LearnRateStrategy(enum.IntEnum):
+ """
+ Indicates the learning rate optimization strategy to use.
+
+ Attributes:
+ LEARN_RATE_STRATEGY_UNSPECIFIED (int)
+ LINE_SEARCH (int): Use line search to determine learning rate.
+ CONSTANT (int): Use a constant learning rate.
+ """
+
+ LEARN_RATE_STRATEGY_UNSPECIFIED = 0
+ LINE_SEARCH = 1
+ CONSTANT = 2
+
+ class LossType(enum.IntEnum):
+ """
+ Loss metric to evaluate model training performance.
+
+ Attributes:
+ LOSS_TYPE_UNSPECIFIED (int)
+ MEAN_SQUARED_LOSS (int): Mean squared loss, used for linear regression.
+ MEAN_LOG_LOSS (int): Mean log loss, used for logistic regression.
+ """
+
+ LOSS_TYPE_UNSPECIFIED = 0
+ MEAN_SQUARED_LOSS = 1
+ MEAN_LOG_LOSS = 2
+
+ class ModelType(enum.IntEnum):
+ """
+ Indicates the type of the Model.
+
+ Attributes:
+ MODEL_TYPE_UNSPECIFIED (int)
+ LINEAR_REGRESSION (int): Linear regression model.
+ LOGISTIC_REGRESSION (int): Logistic regression based classification model.
+ KMEANS (int): K-means clustering model.
+ TENSORFLOW (int): [Beta] An imported TensorFlow model.
+ """
+
+ MODEL_TYPE_UNSPECIFIED = 0
+ LINEAR_REGRESSION = 1
+ LOGISTIC_REGRESSION = 2
+ KMEANS = 3
+ TENSORFLOW = 6
+
+ class OptimizationStrategy(enum.IntEnum):
+ """
+ Indicates the optimization strategy used for training.
+
+ Attributes:
+ OPTIMIZATION_STRATEGY_UNSPECIFIED (int)
+ BATCH_GRADIENT_DESCENT (int): Uses an iterative batch gradient descent algorithm.
+ NORMAL_EQUATION (int): Uses a normal equation to solve linear regression problem.
+ """
+
+ OPTIMIZATION_STRATEGY_UNSPECIFIED = 0
+ BATCH_GRADIENT_DESCENT = 1
+ NORMAL_EQUATION = 2
+
+ class KmeansEnums(object):
+ class KmeansInitializationMethod(enum.IntEnum):
+ """
+ Indicates the method used to initialize the centroids for KMeans
+ clustering algorithm.
+
+ Attributes:
+ KMEANS_INITIALIZATION_METHOD_UNSPECIFIED (int)
+ RANDOM (int): Initializes the centroids randomly.
+ CUSTOM (int): Initializes the centroids using data specified in
+ kmeans\_initialization\_column.
+ """
+
+ KMEANS_INITIALIZATION_METHOD_UNSPECIFIED = 0
+ RANDOM = 1
+ CUSTOM = 2
+
+
+class StandardSqlDataType(object):
+ class TypeKind(enum.IntEnum):
+ """
+ Attributes:
+ TYPE_KIND_UNSPECIFIED (int): Invalid type.
+ INT64 (int): Encoded as a string in decimal format.
+ BOOL (int): Encoded as a boolean "false" or "true".
+ FLOAT64 (int): Encoded as a number, or string "NaN", "Infinity" or "-Infinity".
+ STRING (int): Encoded as a string value.
+ BYTES (int): Encoded as a base64 string per RFC 4648, section 4.
+ TIMESTAMP (int): Encoded as an RFC 3339 timestamp with mandatory "Z" time zone string:
+ 1985-04-12T23:20:50.52Z
+ DATE (int): Encoded as RFC 3339 full-date format string: 1985-04-12
+ TIME (int): Encoded as RFC 3339 partial-time format string: 23:20:50.52
+ DATETIME (int): Encoded as RFC 3339 full-date "T" partial-time: 1985-04-12T23:20:50.52
+ GEOGRAPHY (int): Encoded as WKT
+ NUMERIC (int): Encoded as a decimal string.
+ ARRAY (int): Encoded as a list with types matching Type.array\_type.
+ STRUCT (int): Encoded as a list with fields of type Type.struct\_type[i]. List is used
+ because a JSON object cannot have duplicate field names.
+ """
+
+ TYPE_KIND_UNSPECIFIED = 0
+ INT64 = 2
+ BOOL = 5
+ FLOAT64 = 7
+ STRING = 8
+ BYTES = 9
+ TIMESTAMP = 19
+ DATE = 10
+ TIME = 20
+ DATETIME = 21
+ GEOGRAPHY = 22
+ NUMERIC = 23
+ ARRAY = 16
+ STRUCT = 17
diff --git a/bigquery/google/cloud/bigquery_v2/proto/__init__.py b/bigquery/google/cloud/bigquery_v2/proto/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/bigquery/google/cloud/bigquery_v2/proto/location_metadata.proto b/bigquery/google/cloud/bigquery_v2/proto/location_metadata.proto
new file mode 100644
index 000000000000..95a3133c5755
--- /dev/null
+++ b/bigquery/google/cloud/bigquery_v2/proto/location_metadata.proto
@@ -0,0 +1,34 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.bigquery.v2;
+
+import "google/api/annotations.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/bigquery/v2;bigquery";
+option java_outer_classname = "LocationMetadataProto";
+option java_package = "com.google.cloud.bigquery.v2";
+
+
+// BigQuery-specific metadata about a location. This will be set on
+// google.cloud.location.Location.metadata in Cloud Location API
+// responses.
+message LocationMetadata {
+ // The legacy BigQuery location ID, e.g. “EU” for the “europe” location.
+ // This is for any API consumers that need the legacy “US” and “EU” locations.
+ string legacy_location_id = 1;
+}
diff --git a/bigquery/google/cloud/bigquery_v2/proto/location_metadata_pb2.py b/bigquery/google/cloud/bigquery_v2/proto/location_metadata_pb2.py
new file mode 100644
index 000000000000..6dd9da52e4ed
--- /dev/null
+++ b/bigquery/google/cloud/bigquery_v2/proto/location_metadata_pb2.py
@@ -0,0 +1,98 @@
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/bigquery_v2/proto/location_metadata.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/bigquery_v2/proto/location_metadata.proto",
+ package="google.cloud.bigquery.v2",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\034com.google.cloud.bigquery.v2B\025LocationMetadataProtoZ@google.golang.org/genproto/googleapis/cloud/bigquery/v2;bigquery"
+ ),
+ serialized_pb=_b(
+ '\n6google/cloud/bigquery_v2/proto/location_metadata.proto\x12\x18google.cloud.bigquery.v2\x1a\x1cgoogle/api/annotations.proto".\n\x10LocationMetadata\x12\x1a\n\x12legacy_location_id\x18\x01 \x01(\tBw\n\x1c\x63om.google.cloud.bigquery.v2B\x15LocationMetadataProtoZ@google.golang.org/genproto/googleapis/cloud/bigquery/v2;bigqueryb\x06proto3'
+ ),
+ dependencies=[google_dot_api_dot_annotations__pb2.DESCRIPTOR],
+)
+
+
+_LOCATIONMETADATA = _descriptor.Descriptor(
+ name="LocationMetadata",
+ full_name="google.cloud.bigquery.v2.LocationMetadata",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="legacy_location_id",
+ full_name="google.cloud.bigquery.v2.LocationMetadata.legacy_location_id",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=114,
+ serialized_end=160,
+)
+
+DESCRIPTOR.message_types_by_name["LocationMetadata"] = _LOCATIONMETADATA
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+LocationMetadata = _reflection.GeneratedProtocolMessageType(
+ "LocationMetadata",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_LOCATIONMETADATA,
+ __module__="google.cloud.bigquery_v2.proto.location_metadata_pb2",
+ __doc__="""BigQuery-specific metadata about a location. This will be set on
+ google.cloud.location.Location.metadata in Cloud Location API responses.
+
+
+ Attributes:
+ legacy_location_id:
+ The legacy BigQuery location ID, e.g. ``EU`` for the ``europe``
+ location. This is for any API consumers that need the legacy
+ ``US`` and ``EU`` locations.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.LocationMetadata)
+ ),
+)
+_sym_db.RegisterMessage(LocationMetadata)
+
+
+DESCRIPTOR._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/bigquery/google/cloud/bigquery_v2/proto/location_metadata_pb2_grpc.py b/bigquery/google/cloud/bigquery_v2/proto/location_metadata_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/bigquery/google/cloud/bigquery_v2/proto/location_metadata_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/bigquery/google/cloud/bigquery_v2/proto/model.proto b/bigquery/google/cloud/bigquery_v2/proto/model.proto
new file mode 100644
index 000000000000..42246e8efff2
--- /dev/null
+++ b/bigquery/google/cloud/bigquery_v2/proto/model.proto
@@ -0,0 +1,632 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.bigquery.v2;
+
+import "google/cloud/bigquery/v2/model_reference.proto";
+import "google/cloud/bigquery/v2/standard_sql.proto";
+import "google/protobuf/empty.proto";
+import "google/protobuf/timestamp.proto";
+import "google/protobuf/wrappers.proto";
+import "google/api/annotations.proto";
+import "google/api/client.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/bigquery/v2;bigquery";
+option java_outer_classname = "ModelProto";
+option java_package = "com.google.cloud.bigquery.v2";
+
+service ModelService {
+ option (google.api.default_host) = "bigquery.googleapis.com";
+ option (google.api.oauth_scopes) =
+ "https://www.googleapis.com/auth/bigquery,"
+ "https://www.googleapis.com/auth/bigquery.readonly,"
+ "https://www.googleapis.com/auth/cloud-platform,"
+ "https://www.googleapis.com/auth/cloud-platform.read-only";
+
+ // Gets the specified model resource by model ID.
+ rpc GetModel(GetModelRequest) returns (Model) {
+ }
+
+ // Lists all models in the specified dataset. Requires the READER dataset
+ // role.
+ rpc ListModels(ListModelsRequest) returns (ListModelsResponse) {
+ }
+
+ // Patch specific fields in the specified model.
+ rpc PatchModel(PatchModelRequest) returns (Model) {
+ }
+
+ // Deletes the model specified by modelId from the dataset.
+ rpc DeleteModel(DeleteModelRequest) returns (google.protobuf.Empty) {
+ }
+}
+
+message Model {
+ message KmeansEnums {
+ // Indicates the method used to initialize the centroids for KMeans
+ // clustering algorithm.
+ enum KmeansInitializationMethod {
+ KMEANS_INITIALIZATION_METHOD_UNSPECIFIED = 0;
+
+ // Initializes the centroids randomly.
+ RANDOM = 1;
+
+ // Initializes the centroids using data specified in
+ // kmeans_initialization_column.
+ CUSTOM = 2;
+ }
+
+
+ }
+
+ // Evaluation metrics for regression and explicit feedback type matrix
+ // factorization models.
+ message RegressionMetrics {
+ // Mean absolute error.
+ google.protobuf.DoubleValue mean_absolute_error = 1;
+
+ // Mean squared error.
+ google.protobuf.DoubleValue mean_squared_error = 2;
+
+ // Mean squared log error.
+ google.protobuf.DoubleValue mean_squared_log_error = 3;
+
+ // Median absolute error.
+ google.protobuf.DoubleValue median_absolute_error = 4;
+
+ // R^2 score.
+ google.protobuf.DoubleValue r_squared = 5;
+ }
+
+ // Aggregate metrics for classification/classifier models. For multi-class
+ // models, the metrics are either macro-averaged or micro-averaged. When
+ // macro-averaged, the metrics are calculated for each label and then an
+ // unweighted average is taken of those values. When micro-averaged, the
+ // metric is calculated globally by counting the total number of correctly
+ // predicted rows.
+ message AggregateClassificationMetrics {
+ // Precision is the fraction of actual positive predictions that had
+ // positive actual labels. For multiclass this is a macro-averaged
+ // metric treating each class as a binary classifier.
+ google.protobuf.DoubleValue precision = 1;
+
+ // Recall is the fraction of actual positive labels that were given a
+ // positive prediction. For multiclass this is a macro-averaged metric.
+ google.protobuf.DoubleValue recall = 2;
+
+ // Accuracy is the fraction of predictions given the correct label. For
+ // multiclass this is a micro-averaged metric.
+ google.protobuf.DoubleValue accuracy = 3;
+
+ // Threshold at which the metrics are computed. For binary
+ // classification models this is the positive class threshold.
+ // For multi-class classfication models this is the confidence
+ // threshold.
+ google.protobuf.DoubleValue threshold = 4;
+
+ // The F1 score is an average of recall and precision. For multiclass
+ // this is a macro-averaged metric.
+ google.protobuf.DoubleValue f1_score = 5;
+
+ // Logarithmic Loss. For multiclass this is a macro-averaged metric.
+ google.protobuf.DoubleValue log_loss = 6;
+
+ // Area Under a ROC Curve. For multiclass this is a macro-averaged
+ // metric.
+ google.protobuf.DoubleValue roc_auc = 7;
+ }
+
+ // Evaluation metrics for binary classification/classifier models.
+ message BinaryClassificationMetrics {
+ // Confusion matrix for binary classification models.
+ message BinaryConfusionMatrix {
+ // Threshold value used when computing each of the following metric.
+ google.protobuf.DoubleValue positive_class_threshold = 1;
+
+ // Number of true samples predicted as true.
+ google.protobuf.Int64Value true_positives = 2;
+
+ // Number of false samples predicted as true.
+ google.protobuf.Int64Value false_positives = 3;
+
+ // Number of true samples predicted as false.
+ google.protobuf.Int64Value true_negatives = 4;
+
+ // Number of false samples predicted as false.
+ google.protobuf.Int64Value false_negatives = 5;
+
+ // The fraction of actual positive predictions that had positive actual
+ // labels.
+ google.protobuf.DoubleValue precision = 6;
+
+ // The fraction of actual positive labels that were given a positive
+ // prediction.
+ google.protobuf.DoubleValue recall = 7;
+
+ // The equally weighted average of recall and precision.
+ google.protobuf.DoubleValue f1_score = 8;
+
+ // The fraction of predictions given the correct label.
+ google.protobuf.DoubleValue accuracy = 9;
+ }
+
+ // Aggregate classification metrics.
+ AggregateClassificationMetrics aggregate_classification_metrics = 1;
+
+ // Binary confusion matrix at multiple thresholds.
+ repeated BinaryConfusionMatrix binary_confusion_matrix_list = 2;
+
+ // Label representing the positive class.
+ string positive_label = 3;
+
+ // Label representing the negative class.
+ string negative_label = 4;
+ }
+
+ // Evaluation metrics for multi-class classification/classifier models.
+ message MultiClassClassificationMetrics {
+ // Confusion matrix for multi-class classification models.
+ message ConfusionMatrix {
+ // A single entry in the confusion matrix.
+ message Entry {
+ // The predicted label. For confidence_threshold > 0, we will
+ // also add an entry indicating the number of items under the
+ // confidence threshold.
+ string predicted_label = 1;
+
+ // Number of items being predicted as this label.
+ google.protobuf.Int64Value item_count = 2;
+ }
+
+ // A single row in the confusion matrix.
+ message Row {
+ // The original label of this row.
+ string actual_label = 1;
+
+ // Info describing predicted label distribution.
+ repeated Entry entries = 2;
+ }
+
+ // Confidence threshold used when computing the entries of the
+ // confusion matrix.
+ google.protobuf.DoubleValue confidence_threshold = 1;
+
+ // One row per actual label.
+ repeated Row rows = 2;
+ }
+
+ // Aggregate classification metrics.
+ AggregateClassificationMetrics aggregate_classification_metrics = 1;
+
+ // Confusion matrix at different thresholds.
+ repeated ConfusionMatrix confusion_matrix_list = 2;
+ }
+
+ // Evaluation metrics for clustering models.
+ message ClusteringMetrics {
+ // Message containing the information about one cluster.
+ message Cluster {
+ // Representative value of a single feature within the cluster.
+ message FeatureValue {
+ // Representative value of a categorical feature.
+ message CategoricalValue {
+ // Represents the count of a single category within the cluster.
+ message CategoryCount {
+ // The name of category.
+ string category = 1;
+
+ // The count of training samples matching the category within the
+ // cluster.
+ google.protobuf.Int64Value count = 2;
+ }
+
+ // Counts of all categories for the categorical feature. If there are
+ // more than ten categories, we return top ten (by count) and return
+ // one more CategoryCount with category ‘_OTHER_’ and count as
+ // aggregate counts of remaining categories.
+ repeated CategoryCount category_counts = 1;
+ }
+
+ // The feature column name.
+ string feature_column = 1;
+
+ oneof value {
+ // The numerical feature value. This is the centroid value for this
+ // feature.
+ google.protobuf.DoubleValue numerical_value = 2;
+
+ // The categorical feature value.
+ CategoricalValue categorical_value = 3;
+ }
+ }
+
+ // Centroid id.
+ int64 centroid_id = 1;
+
+ // Values of highly variant features for this cluster.
+ repeated FeatureValue feature_values = 2;
+
+ // Count of training data rows that were assigned to this cluster.
+ google.protobuf.Int64Value count = 3;
+ }
+
+ // Davies-Bouldin index.
+ google.protobuf.DoubleValue davies_bouldin_index = 1;
+
+ // Mean of squared distances between each sample to its cluster centroid.
+ google.protobuf.DoubleValue mean_squared_distance = 2;
+
+ // [Beta] Information for all clusters.
+ repeated Cluster clusters = 3;
+ }
+
+ // Evaluation metrics of a model. These are either computed on all training
+ // data or just the eval data based on whether eval data was used during
+ // training. These are not present for imported models.
+ message EvaluationMetrics {
+ oneof metrics {
+ // Populated for regression models and explicit feedback type matrix
+ // factorization models.
+ RegressionMetrics regression_metrics = 1;
+
+ // Populated for binary classification/classifier models.
+ BinaryClassificationMetrics binary_classification_metrics = 2;
+
+ // Populated for multi-class classification/classifier models.
+ MultiClassClassificationMetrics multi_class_classification_metrics = 3;
+
+ // Populated for clustering models.
+ ClusteringMetrics clustering_metrics = 4;
+ }
+ }
+
+ // Information about a single training query run for the model.
+ message TrainingRun {
+ message TrainingOptions {
+ // The maximum number of iterations in training. Used only for iterative
+ // training algorithms.
+ int64 max_iterations = 1;
+
+ // Type of loss function used during training run.
+ LossType loss_type = 2;
+
+ // Learning rate in training. Used only for iterative training algorithms.
+ double learn_rate = 3;
+
+ // L1 regularization coefficient.
+ google.protobuf.DoubleValue l1_regularization = 4;
+
+ // L2 regularization coefficient.
+ google.protobuf.DoubleValue l2_regularization = 5;
+
+ // When early_stop is true, stops training when accuracy improvement is
+ // less than 'min_relative_progress'. Used only for iterative training
+ // algorithms.
+ google.protobuf.DoubleValue min_relative_progress = 6;
+
+ // Whether to train a model from the last checkpoint.
+ google.protobuf.BoolValue warm_start = 7;
+
+ // Whether to stop early when the loss doesn't improve significantly
+ // any more (compared to min_relative_progress). Used only for iterative
+ // training algorithms.
+ google.protobuf.BoolValue early_stop = 8;
+
+ // Name of input label columns in training data.
+ repeated string input_label_columns = 9;
+
+ // The data split type for training and evaluation, e.g. RANDOM.
+ DataSplitMethod data_split_method = 10;
+
+ // The fraction of evaluation data over the whole input data. The rest
+ // of data will be used as training data. The format should be double.
+ // Accurate to two decimal places.
+ // Default value is 0.2.
+ double data_split_eval_fraction = 11;
+
+ // The column to split data with. This column won't be used as a
+ // feature.
+ // 1. When data_split_method is CUSTOM, the corresponding column should
+ // be boolean. The rows with true value tag are eval data, and the false
+ // are training data.
+ // 2. When data_split_method is SEQ, the first DATA_SPLIT_EVAL_FRACTION
+ // rows (from smallest to largest) in the corresponding column are used
+ // as training data, and the rest are eval data. It respects the order
+ // in Orderable data types:
+ // https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#data-type-properties
+ string data_split_column = 12;
+
+ // The strategy to determine learn rate for the current iteration.
+ LearnRateStrategy learn_rate_strategy = 13;
+
+ // Specifies the initial learning rate for the line search learn rate
+ // strategy.
+ double initial_learn_rate = 16;
+
+ // Weights associated with each label class, for rebalancing the
+ // training data. Only applicable for classification models.
+ map label_class_weights = 17;
+
+ // Distance type for clustering models.
+ DistanceType distance_type = 20;
+
+ // Number of clusters for clustering models.
+ int64 num_clusters = 21;
+
+ // [Beta] Google Cloud Storage URI from which the model was imported. Only
+ // applicable for imported models.
+ string model_uri = 22;
+
+ // Optimization strategy for training linear regression models.
+ OptimizationStrategy optimization_strategy = 23;
+
+ // The method used to initialize the centroids for kmeans algorithm.
+ KmeansEnums.KmeansInitializationMethod kmeans_initialization_method = 33;
+
+ // The column used to provide the initial centroids for kmeans algorithm
+ // when kmeans_initialization_method is CUSTOM.
+ string kmeans_initialization_column = 34;
+ }
+
+ // Information about a single iteration of the training run.
+ message IterationResult {
+ // Information about a single cluster for clustering model.
+ message ClusterInfo {
+ // Centroid id.
+ int64 centroid_id = 1;
+
+ // Cluster radius, the average distance from centroid
+ // to each point assigned to the cluster.
+ google.protobuf.DoubleValue cluster_radius = 2;
+
+ // Cluster size, the total number of points assigned to the cluster.
+ google.protobuf.Int64Value cluster_size = 3;
+ }
+
+ // Index of the iteration, 0 based.
+ google.protobuf.Int32Value index = 1;
+
+ // Time taken to run the iteration in milliseconds.
+ google.protobuf.Int64Value duration_ms = 4;
+
+ // Loss computed on the training data at the end of iteration.
+ google.protobuf.DoubleValue training_loss = 5;
+
+ // Loss computed on the eval data at the end of iteration.
+ google.protobuf.DoubleValue eval_loss = 6;
+
+ // Learn rate used for this iteration.
+ double learn_rate = 7;
+
+ // Information about top clusters for clustering models.
+ repeated ClusterInfo cluster_infos = 8;
+ }
+
+ // Options that were used for this training run, includes
+ // user specified and default options that were used.
+ TrainingOptions training_options = 1;
+
+ // The start time of this training run.
+ google.protobuf.Timestamp start_time = 8;
+
+ // Output of each iteration run, results.size() <= max_iterations.
+ repeated IterationResult results = 6;
+
+ // The evaluation metrics over training/eval data that were computed at the
+ // end of training.
+ EvaluationMetrics evaluation_metrics = 7;
+ }
+
+ // Indicates the type of the Model.
+ enum ModelType {
+ MODEL_TYPE_UNSPECIFIED = 0;
+
+ // Linear regression model.
+ LINEAR_REGRESSION = 1;
+
+ // Logistic regression based classification model.
+ LOGISTIC_REGRESSION = 2;
+
+ // K-means clustering model.
+ KMEANS = 3;
+
+ // [Beta] An imported TensorFlow model.
+ TENSORFLOW = 6;
+ }
+
+ // Loss metric to evaluate model training performance.
+ enum LossType {
+ LOSS_TYPE_UNSPECIFIED = 0;
+
+ // Mean squared loss, used for linear regression.
+ MEAN_SQUARED_LOSS = 1;
+
+ // Mean log loss, used for logistic regression.
+ MEAN_LOG_LOSS = 2;
+ }
+
+ // Distance metric used to compute the distance between two points.
+ enum DistanceType {
+ DISTANCE_TYPE_UNSPECIFIED = 0;
+
+ // Eculidean distance.
+ EUCLIDEAN = 1;
+
+ // Cosine distance.
+ COSINE = 2;
+ }
+
+ // Indicates the method to split input data into multiple tables.
+ enum DataSplitMethod {
+ DATA_SPLIT_METHOD_UNSPECIFIED = 0;
+
+ // Splits data randomly.
+ RANDOM = 1;
+
+ // Splits data with the user provided tags.
+ CUSTOM = 2;
+
+ // Splits data sequentially.
+ SEQUENTIAL = 3;
+
+ // Data split will be skipped.
+ NO_SPLIT = 4;
+
+ // Splits data automatically: Uses NO_SPLIT if the data size is small.
+ // Otherwise uses RANDOM.
+ AUTO_SPLIT = 5;
+ }
+
+ // Indicates the learning rate optimization strategy to use.
+ enum LearnRateStrategy {
+ LEARN_RATE_STRATEGY_UNSPECIFIED = 0;
+
+ // Use line search to determine learning rate.
+ LINE_SEARCH = 1;
+
+ // Use a constant learning rate.
+ CONSTANT = 2;
+ }
+
+ // Indicates the optimization strategy used for training.
+ enum OptimizationStrategy {
+ OPTIMIZATION_STRATEGY_UNSPECIFIED = 0;
+
+ // Uses an iterative batch gradient descent algorithm.
+ BATCH_GRADIENT_DESCENT = 1;
+
+ // Uses a normal equation to solve linear regression problem.
+ NORMAL_EQUATION = 2;
+ }
+
+ // Output only. A hash of this resource.
+ string etag = 1;
+
+ // Required. Unique identifier for this model.
+ ModelReference model_reference = 2;
+
+ // Output only. The time when this model was created, in millisecs since the
+ // epoch.
+ int64 creation_time = 5;
+
+ // Output only. The time when this model was last modified, in millisecs
+ // since the epoch.
+ int64 last_modified_time = 6;
+
+ // [Optional] A user-friendly description of this model.
+ string description = 12;
+
+ // [Optional] A descriptive name for this model.
+ string friendly_name = 14;
+
+ // [Optional] The labels associated with this model. You can use these to
+ // organize and group your models. Label keys and values can be no longer
+ // than 63 characters, can only contain lowercase letters, numeric
+ // characters, underscores and dashes. International characters are allowed.
+ // Label values are optional. Label keys must start with a letter and each
+ // label in the list must have a different key.
+ map labels = 15;
+
+ // [Optional] The time when this model expires, in milliseconds since the
+ // epoch. If not present, the model will persist indefinitely. Expired models
+ // will be deleted and their storage reclaimed. The defaultTableExpirationMs
+ // property of the encapsulating dataset can be used to set a default
+ // expirationTime on newly created models.
+ int64 expiration_time = 16;
+
+ // Output only. The geographic location where the model resides. This value
+ // is inherited from the dataset.
+ string location = 13;
+
+ // Output only. Type of the model resource.
+ ModelType model_type = 7;
+
+ // Output only. Information for all training runs in increasing order of
+ // start_time.
+ repeated TrainingRun training_runs = 9;
+
+ // Output only. Input feature columns that were used to train this model.
+ repeated StandardSqlField feature_columns = 10;
+
+ // Output only. Label columns that were used to train this model.
+ // The output of the model will have a "predicted_" prefix to these columns.
+ repeated StandardSqlField label_columns = 11;
+}
+
+message GetModelRequest {
+ // Project ID of the requested model.
+ string project_id = 1;
+
+ // Dataset ID of the requested model.
+ string dataset_id = 2;
+
+ // Model ID of the requested model.
+ string model_id = 3;
+}
+
+message PatchModelRequest {
+ // Project ID of the model to patch.
+ string project_id = 1;
+
+ // Dataset ID of the model to patch.
+ string dataset_id = 2;
+
+ // Model ID of the model to patch.
+ string model_id = 3;
+
+ // Patched model.
+ // Follows RFC5789 patch semantics. Missing fields are not updated.
+ // To clear a field, explicitly set to default value.
+ Model model = 4;
+}
+
+message DeleteModelRequest {
+ // Project ID of the model to delete.
+ string project_id = 1;
+
+ // Dataset ID of the model to delete.
+ string dataset_id = 2;
+
+ // Model ID of the model to delete.
+ string model_id = 3;
+}
+
+message ListModelsRequest {
+ // Project ID of the models to list.
+ string project_id = 1;
+
+ // Dataset ID of the models to list.
+ string dataset_id = 2;
+
+ // The maximum number of results to return in a single response page.
+ // Leverage the page tokens to iterate through the entire collection.
+ google.protobuf.UInt32Value max_results = 3;
+
+ // Page token, returned by a previous call to request the next page of
+ // results
+ string page_token = 4;
+}
+
+message ListModelsResponse {
+ // Models in the requested dataset. Only the following fields are populated:
+ // model_reference, model_type, creation_time, last_modified_time and
+ // labels.
+ repeated Model models = 1;
+
+ // A token to request the next page of results.
+ string next_page_token = 2;
+}
diff --git a/bigquery/google/cloud/bigquery_v2/proto/model_pb2.py b/bigquery/google/cloud/bigquery_v2/proto/model_pb2.py
new file mode 100644
index 000000000000..45e6cefdf72c
--- /dev/null
+++ b/bigquery/google/cloud/bigquery_v2/proto/model_pb2.py
@@ -0,0 +1,4026 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/bigquery_v2/proto/model.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.cloud.bigquery_v2.proto import (
+ model_reference_pb2 as google_dot_cloud_dot_bigquery__v2_dot_proto_dot_model__reference__pb2,
+)
+from google.cloud.bigquery_v2.proto import (
+ standard_sql_pb2 as google_dot_cloud_dot_bigquery__v2_dot_proto_dot_standard__sql__pb2,
+)
+from google.protobuf import empty_pb2 as google_dot_protobuf_dot_empty__pb2
+from google.protobuf import timestamp_pb2 as google_dot_protobuf_dot_timestamp__pb2
+from google.protobuf import wrappers_pb2 as google_dot_protobuf_dot_wrappers__pb2
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+from google.api import client_pb2 as google_dot_api_dot_client__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/bigquery_v2/proto/model.proto",
+ package="google.cloud.bigquery.v2",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\034com.google.cloud.bigquery.v2B\nModelProtoZ@google.golang.org/genproto/googleapis/cloud/bigquery/v2;bigquery"
+ ),
+ serialized_pb=_b(
+ '\n*google/cloud/bigquery_v2/proto/model.proto\x12\x18google.cloud.bigquery.v2\x1a\x34google/cloud/bigquery_v2/proto/model_reference.proto\x1a\x31google/cloud/bigquery_v2/proto/standard_sql.proto\x1a\x1bgoogle/protobuf/empty.proto\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x1egoogle/protobuf/wrappers.proto\x1a\x1cgoogle/api/annotations.proto\x1a\x17google/api/client.proto"\x8a\x34\n\x05Model\x12\x0c\n\x04\x65tag\x18\x01 \x01(\t\x12\x41\n\x0fmodel_reference\x18\x02 \x01(\x0b\x32(.google.cloud.bigquery.v2.ModelReference\x12\x15\n\rcreation_time\x18\x05 \x01(\x03\x12\x1a\n\x12last_modified_time\x18\x06 \x01(\x03\x12\x13\n\x0b\x64\x65scription\x18\x0c \x01(\t\x12\x15\n\rfriendly_name\x18\x0e \x01(\t\x12;\n\x06labels\x18\x0f \x03(\x0b\x32+.google.cloud.bigquery.v2.Model.LabelsEntry\x12\x17\n\x0f\x65xpiration_time\x18\x10 \x01(\x03\x12\x10\n\x08location\x18\r \x01(\t\x12=\n\nmodel_type\x18\x07 \x01(\x0e\x32).google.cloud.bigquery.v2.Model.ModelType\x12\x42\n\rtraining_runs\x18\t \x03(\x0b\x32+.google.cloud.bigquery.v2.Model.TrainingRun\x12\x43\n\x0f\x66\x65\x61ture_columns\x18\n \x03(\x0b\x32*.google.cloud.bigquery.v2.StandardSqlField\x12\x41\n\rlabel_columns\x18\x0b \x03(\x0b\x32*.google.cloud.bigquery.v2.StandardSqlField\x1aq\n\x0bKmeansEnums"b\n\x1aKmeansInitializationMethod\x12,\n(KMEANS_INITIALIZATION_METHOD_UNSPECIFIED\x10\x00\x12\n\n\x06RANDOM\x10\x01\x12\n\n\x06\x43USTOM\x10\x02\x1a\xb4\x02\n\x11RegressionMetrics\x12\x39\n\x13mean_absolute_error\x18\x01 \x01(\x0b\x32\x1c.google.protobuf.DoubleValue\x12\x38\n\x12mean_squared_error\x18\x02 \x01(\x0b\x32\x1c.google.protobuf.DoubleValue\x12<\n\x16mean_squared_log_error\x18\x03 \x01(\x0b\x32\x1c.google.protobuf.DoubleValue\x12;\n\x15median_absolute_error\x18\x04 \x01(\x0b\x32\x1c.google.protobuf.DoubleValue\x12/\n\tr_squared\x18\x05 \x01(\x0b\x32\x1c.google.protobuf.DoubleValue\x1a\xef\x02\n\x1e\x41ggregateClassificationMetrics\x12/\n\tprecision\x18\x01 \x01(\x0b\x32\x1c.google.protobuf.DoubleValue\x12,\n\x06recall\x18\x02 \x01(\x0b\x32\x1c.google.protobuf.DoubleValue\x12.\n\x08\x61\x63\x63uracy\x18\x03 \x01(\x0b\x32\x1c.google.protobuf.DoubleValue\x12/\n\tthreshold\x18\x04 \x01(\x0b\x32\x1c.google.protobuf.DoubleValue\x12.\n\x08\x66\x31_score\x18\x05 \x01(\x0b\x32\x1c.google.protobuf.DoubleValue\x12.\n\x08log_loss\x18\x06 \x01(\x0b\x32\x1c.google.protobuf.DoubleValue\x12-\n\x07roc_auc\x18\x07 \x01(\x0b\x32\x1c.google.protobuf.DoubleValue\x1a\x9f\x06\n\x1b\x42inaryClassificationMetrics\x12h\n aggregate_classification_metrics\x18\x01 \x01(\x0b\x32>.google.cloud.bigquery.v2.Model.AggregateClassificationMetrics\x12w\n\x1c\x62inary_confusion_matrix_list\x18\x02 \x03(\x0b\x32Q.google.cloud.bigquery.v2.Model.BinaryClassificationMetrics.BinaryConfusionMatrix\x12\x16\n\x0epositive_label\x18\x03 \x01(\t\x12\x16\n\x0enegative_label\x18\x04 \x01(\t\x1a\xec\x03\n\x15\x42inaryConfusionMatrix\x12>\n\x18positive_class_threshold\x18\x01 \x01(\x0b\x32\x1c.google.protobuf.DoubleValue\x12\x33\n\x0etrue_positives\x18\x02 \x01(\x0b\x32\x1b.google.protobuf.Int64Value\x12\x34\n\x0f\x66\x61lse_positives\x18\x03 \x01(\x0b\x32\x1b.google.protobuf.Int64Value\x12\x33\n\x0etrue_negatives\x18\x04 \x01(\x0b\x32\x1b.google.protobuf.Int64Value\x12\x34\n\x0f\x66\x61lse_negatives\x18\x05 \x01(\x0b\x32\x1b.google.protobuf.Int64Value\x12/\n\tprecision\x18\x06 \x01(\x0b\x32\x1c.google.protobuf.DoubleValue\x12,\n\x06recall\x18\x07 \x01(\x0b\x32\x1c.google.protobuf.DoubleValue\x12.\n\x08\x66\x31_score\x18\x08 \x01(\x0b\x32\x1c.google.protobuf.DoubleValue\x12.\n\x08\x61\x63\x63uracy\x18\t \x01(\x0b\x32\x1c.google.protobuf.DoubleValue\x1a\x87\x05\n\x1fMultiClassClassificationMetrics\x12h\n aggregate_classification_metrics\x18\x01 \x01(\x0b\x32>.google.cloud.bigquery.v2.Model.AggregateClassificationMetrics\x12n\n\x15\x63onfusion_matrix_list\x18\x02 \x03(\x0b\x32O.google.cloud.bigquery.v2.Model.MultiClassClassificationMetrics.ConfusionMatrix\x1a\x89\x03\n\x0f\x43onfusionMatrix\x12:\n\x14\x63onfidence_threshold\x18\x01 \x01(\x0b\x32\x1c.google.protobuf.DoubleValue\x12\x61\n\x04rows\x18\x02 \x03(\x0b\x32S.google.cloud.bigquery.v2.Model.MultiClassClassificationMetrics.ConfusionMatrix.Row\x1aQ\n\x05\x45ntry\x12\x17\n\x0fpredicted_label\x18\x01 \x01(\t\x12/\n\nitem_count\x18\x02 \x01(\x0b\x32\x1b.google.protobuf.Int64Value\x1a\x83\x01\n\x03Row\x12\x14\n\x0c\x61\x63tual_label\x18\x01 \x01(\t\x12\x66\n\x07\x65ntries\x18\x02 \x03(\x0b\x32U.google.cloud.bigquery.v2.Model.MultiClassClassificationMetrics.ConfusionMatrix.Entry\x1a\xcb\x06\n\x11\x43lusteringMetrics\x12:\n\x14\x64\x61vies_bouldin_index\x18\x01 \x01(\x0b\x32\x1c.google.protobuf.DoubleValue\x12;\n\x15mean_squared_distance\x18\x02 \x01(\x0b\x32\x1c.google.protobuf.DoubleValue\x12K\n\x08\x63lusters\x18\x03 \x03(\x0b\x32\x39.google.cloud.bigquery.v2.Model.ClusteringMetrics.Cluster\x1a\xef\x04\n\x07\x43luster\x12\x13\n\x0b\x63\x65ntroid_id\x18\x01 \x01(\x03\x12^\n\x0e\x66\x65\x61ture_values\x18\x02 \x03(\x0b\x32\x46.google.cloud.bigquery.v2.Model.ClusteringMetrics.Cluster.FeatureValue\x12*\n\x05\x63ount\x18\x03 \x01(\x0b\x32\x1b.google.protobuf.Int64Value\x1a\xc2\x03\n\x0c\x46\x65\x61tureValue\x12\x16\n\x0e\x66\x65\x61ture_column\x18\x01 \x01(\t\x12\x37\n\x0fnumerical_value\x18\x02 \x01(\x0b\x32\x1c.google.protobuf.DoubleValueH\x00\x12t\n\x11\x63\x61tegorical_value\x18\x03 \x01(\x0b\x32W.google.cloud.bigquery.v2.Model.ClusteringMetrics.Cluster.FeatureValue.CategoricalValueH\x00\x1a\xe1\x01\n\x10\x43\x61tegoricalValue\x12~\n\x0f\x63\x61tegory_counts\x18\x01 \x03(\x0b\x32\x65.google.cloud.bigquery.v2.Model.ClusteringMetrics.Cluster.FeatureValue.CategoricalValue.CategoryCount\x1aM\n\rCategoryCount\x12\x10\n\x08\x63\x61tegory\x18\x01 \x01(\t\x12*\n\x05\x63ount\x18\x02 \x01(\x0b\x32\x1b.google.protobuf.Int64ValueB\x07\n\x05value\x1a\x95\x03\n\x11\x45valuationMetrics\x12O\n\x12regression_metrics\x18\x01 \x01(\x0b\x32\x31.google.cloud.bigquery.v2.Model.RegressionMetricsH\x00\x12\x64\n\x1d\x62inary_classification_metrics\x18\x02 \x01(\x0b\x32;.google.cloud.bigquery.v2.Model.BinaryClassificationMetricsH\x00\x12m\n"multi_class_classification_metrics\x18\x03 \x01(\x0b\x32?.google.cloud.bigquery.v2.Model.MultiClassClassificationMetricsH\x00\x12O\n\x12\x63lustering_metrics\x18\x04 \x01(\x0b\x32\x31.google.cloud.bigquery.v2.Model.ClusteringMetricsH\x00\x42\t\n\x07metrics\x1a\xab\x0f\n\x0bTrainingRun\x12U\n\x10training_options\x18\x01 \x01(\x0b\x32;.google.cloud.bigquery.v2.Model.TrainingRun.TrainingOptions\x12.\n\nstart_time\x18\x08 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12L\n\x07results\x18\x06 \x03(\x0b\x32;.google.cloud.bigquery.v2.Model.TrainingRun.IterationResult\x12M\n\x12\x65valuation_metrics\x18\x07 \x01(\x0b\x32\x31.google.cloud.bigquery.v2.Model.EvaluationMetrics\x1a\x9d\t\n\x0fTrainingOptions\x12\x16\n\x0emax_iterations\x18\x01 \x01(\x03\x12;\n\tloss_type\x18\x02 \x01(\x0e\x32(.google.cloud.bigquery.v2.Model.LossType\x12\x12\n\nlearn_rate\x18\x03 \x01(\x01\x12\x37\n\x11l1_regularization\x18\x04 \x01(\x0b\x32\x1c.google.protobuf.DoubleValue\x12\x37\n\x11l2_regularization\x18\x05 \x01(\x0b\x32\x1c.google.protobuf.DoubleValue\x12;\n\x15min_relative_progress\x18\x06 \x01(\x0b\x32\x1c.google.protobuf.DoubleValue\x12.\n\nwarm_start\x18\x07 \x01(\x0b\x32\x1a.google.protobuf.BoolValue\x12.\n\nearly_stop\x18\x08 \x01(\x0b\x32\x1a.google.protobuf.BoolValue\x12\x1b\n\x13input_label_columns\x18\t \x03(\t\x12J\n\x11\x64\x61ta_split_method\x18\n \x01(\x0e\x32/.google.cloud.bigquery.v2.Model.DataSplitMethod\x12 \n\x18\x64\x61ta_split_eval_fraction\x18\x0b \x01(\x01\x12\x19\n\x11\x64\x61ta_split_column\x18\x0c \x01(\t\x12N\n\x13learn_rate_strategy\x18\r \x01(\x0e\x32\x31.google.cloud.bigquery.v2.Model.LearnRateStrategy\x12\x1a\n\x12initial_learn_rate\x18\x10 \x01(\x01\x12o\n\x13label_class_weights\x18\x11 \x03(\x0b\x32R.google.cloud.bigquery.v2.Model.TrainingRun.TrainingOptions.LabelClassWeightsEntry\x12\x43\n\rdistance_type\x18\x14 \x01(\x0e\x32,.google.cloud.bigquery.v2.Model.DistanceType\x12\x14\n\x0cnum_clusters\x18\x15 \x01(\x03\x12\x11\n\tmodel_uri\x18\x16 \x01(\t\x12S\n\x15optimization_strategy\x18\x17 \x01(\x0e\x32\x34.google.cloud.bigquery.v2.Model.OptimizationStrategy\x12l\n\x1ckmeans_initialization_method\x18! \x01(\x0e\x32\x46.google.cloud.bigquery.v2.Model.KmeansEnums.KmeansInitializationMethod\x12$\n\x1ckmeans_initialization_column\x18" \x01(\t\x1a\x38\n\x16LabelClassWeightsEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\x01:\x02\x38\x01\x1a\xd7\x03\n\x0fIterationResult\x12*\n\x05index\x18\x01 \x01(\x0b\x32\x1b.google.protobuf.Int32Value\x12\x30\n\x0b\x64uration_ms\x18\x04 \x01(\x0b\x32\x1b.google.protobuf.Int64Value\x12\x33\n\rtraining_loss\x18\x05 \x01(\x0b\x32\x1c.google.protobuf.DoubleValue\x12/\n\teval_loss\x18\x06 \x01(\x0b\x32\x1c.google.protobuf.DoubleValue\x12\x12\n\nlearn_rate\x18\x07 \x01(\x01\x12^\n\rcluster_infos\x18\x08 \x03(\x0b\x32G.google.cloud.bigquery.v2.Model.TrainingRun.IterationResult.ClusterInfo\x1a\x8b\x01\n\x0b\x43lusterInfo\x12\x13\n\x0b\x63\x65ntroid_id\x18\x01 \x01(\x03\x12\x34\n\x0e\x63luster_radius\x18\x02 \x01(\x0b\x32\x1c.google.protobuf.DoubleValue\x12\x31\n\x0c\x63luster_size\x18\x03 \x01(\x0b\x32\x1b.google.protobuf.Int64Value\x1a-\n\x0bLabelsEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01"s\n\tModelType\x12\x1a\n\x16MODEL_TYPE_UNSPECIFIED\x10\x00\x12\x15\n\x11LINEAR_REGRESSION\x10\x01\x12\x17\n\x13LOGISTIC_REGRESSION\x10\x02\x12\n\n\x06KMEANS\x10\x03\x12\x0e\n\nTENSORFLOW\x10\x06"O\n\x08LossType\x12\x19\n\x15LOSS_TYPE_UNSPECIFIED\x10\x00\x12\x15\n\x11MEAN_SQUARED_LOSS\x10\x01\x12\x11\n\rMEAN_LOG_LOSS\x10\x02"H\n\x0c\x44istanceType\x12\x1d\n\x19\x44ISTANCE_TYPE_UNSPECIFIED\x10\x00\x12\r\n\tEUCLIDEAN\x10\x01\x12\n\n\x06\x43OSINE\x10\x02"z\n\x0f\x44\x61taSplitMethod\x12!\n\x1d\x44\x41TA_SPLIT_METHOD_UNSPECIFIED\x10\x00\x12\n\n\x06RANDOM\x10\x01\x12\n\n\x06\x43USTOM\x10\x02\x12\x0e\n\nSEQUENTIAL\x10\x03\x12\x0c\n\x08NO_SPLIT\x10\x04\x12\x0e\n\nAUTO_SPLIT\x10\x05"W\n\x11LearnRateStrategy\x12#\n\x1fLEARN_RATE_STRATEGY_UNSPECIFIED\x10\x00\x12\x0f\n\x0bLINE_SEARCH\x10\x01\x12\x0c\n\x08\x43ONSTANT\x10\x02"n\n\x14OptimizationStrategy\x12%\n!OPTIMIZATION_STRATEGY_UNSPECIFIED\x10\x00\x12\x1a\n\x16\x42\x41TCH_GRADIENT_DESCENT\x10\x01\x12\x13\n\x0fNORMAL_EQUATION\x10\x02"K\n\x0fGetModelRequest\x12\x12\n\nproject_id\x18\x01 \x01(\t\x12\x12\n\ndataset_id\x18\x02 \x01(\t\x12\x10\n\x08model_id\x18\x03 \x01(\t"}\n\x11PatchModelRequest\x12\x12\n\nproject_id\x18\x01 \x01(\t\x12\x12\n\ndataset_id\x18\x02 \x01(\t\x12\x10\n\x08model_id\x18\x03 \x01(\t\x12.\n\x05model\x18\x04 \x01(\x0b\x32\x1f.google.cloud.bigquery.v2.Model"N\n\x12\x44\x65leteModelRequest\x12\x12\n\nproject_id\x18\x01 \x01(\t\x12\x12\n\ndataset_id\x18\x02 \x01(\t\x12\x10\n\x08model_id\x18\x03 \x01(\t"\x82\x01\n\x11ListModelsRequest\x12\x12\n\nproject_id\x18\x01 \x01(\t\x12\x12\n\ndataset_id\x18\x02 \x01(\t\x12\x31\n\x0bmax_results\x18\x03 \x01(\x0b\x32\x1c.google.protobuf.UInt32Value\x12\x12\n\npage_token\x18\x04 \x01(\t"^\n\x12ListModelsResponse\x12/\n\x06models\x18\x01 \x03(\x0b\x32\x1f.google.cloud.bigquery.v2.Model\x12\x17\n\x0fnext_page_token\x18\x02 \x01(\t2\xeb\x04\n\x0cModelService\x12X\n\x08GetModel\x12).google.cloud.bigquery.v2.GetModelRequest\x1a\x1f.google.cloud.bigquery.v2.Model"\x00\x12i\n\nListModels\x12+.google.cloud.bigquery.v2.ListModelsRequest\x1a,.google.cloud.bigquery.v2.ListModelsResponse"\x00\x12\\\n\nPatchModel\x12+.google.cloud.bigquery.v2.PatchModelRequest\x1a\x1f.google.cloud.bigquery.v2.Model"\x00\x12U\n\x0b\x44\x65leteModel\x12,.google.cloud.bigquery.v2.DeleteModelRequest\x1a\x16.google.protobuf.Empty"\x00\x1a\xe0\x01\xca\x41\x17\x62igquery.googleapis.com\xd2\x41\xc2\x01https://www.googleapis.com/auth/bigquery,https://www.googleapis.com/auth/bigquery.readonly,https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/cloud-platform.read-onlyBl\n\x1c\x63om.google.cloud.bigquery.v2B\nModelProtoZ@google.golang.org/genproto/googleapis/cloud/bigquery/v2;bigqueryb\x06proto3'
+ ),
+ dependencies=[
+ google_dot_cloud_dot_bigquery__v2_dot_proto_dot_model__reference__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_bigquery__v2_dot_proto_dot_standard__sql__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_empty__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_timestamp__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_wrappers__pb2.DESCRIPTOR,
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ google_dot_api_dot_client__pb2.DESCRIPTOR,
+ ],
+)
+
+
+_MODEL_KMEANSENUMS_KMEANSINITIALIZATIONMETHOD = _descriptor.EnumDescriptor(
+ name="KmeansInitializationMethod",
+ full_name="google.cloud.bigquery.v2.Model.KmeansEnums.KmeansInitializationMethod",
+ filename=None,
+ file=DESCRIPTOR,
+ values=[
+ _descriptor.EnumValueDescriptor(
+ name="KMEANS_INITIALIZATION_METHOD_UNSPECIFIED",
+ index=0,
+ number=0,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="RANDOM", index=1, number=1, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="CUSTOM", index=2, number=2, serialized_options=None, type=None
+ ),
+ ],
+ containing_type=None,
+ serialized_options=None,
+ serialized_start=898,
+ serialized_end=996,
+)
+_sym_db.RegisterEnumDescriptor(_MODEL_KMEANSENUMS_KMEANSINITIALIZATIONMETHOD)
+
+_MODEL_MODELTYPE = _descriptor.EnumDescriptor(
+ name="ModelType",
+ full_name="google.cloud.bigquery.v2.Model.ModelType",
+ filename=None,
+ file=DESCRIPTOR,
+ values=[
+ _descriptor.EnumValueDescriptor(
+ name="MODEL_TYPE_UNSPECIFIED",
+ index=0,
+ number=0,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="LINEAR_REGRESSION",
+ index=1,
+ number=1,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="LOGISTIC_REGRESSION",
+ index=2,
+ number=2,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="KMEANS", index=3, number=3, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="TENSORFLOW", index=4, number=6, serialized_options=None, type=None
+ ),
+ ],
+ containing_type=None,
+ serialized_options=None,
+ serialized_start=6398,
+ serialized_end=6513,
+)
+_sym_db.RegisterEnumDescriptor(_MODEL_MODELTYPE)
+
+_MODEL_LOSSTYPE = _descriptor.EnumDescriptor(
+ name="LossType",
+ full_name="google.cloud.bigquery.v2.Model.LossType",
+ filename=None,
+ file=DESCRIPTOR,
+ values=[
+ _descriptor.EnumValueDescriptor(
+ name="LOSS_TYPE_UNSPECIFIED",
+ index=0,
+ number=0,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="MEAN_SQUARED_LOSS",
+ index=1,
+ number=1,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="MEAN_LOG_LOSS", index=2, number=2, serialized_options=None, type=None
+ ),
+ ],
+ containing_type=None,
+ serialized_options=None,
+ serialized_start=6515,
+ serialized_end=6594,
+)
+_sym_db.RegisterEnumDescriptor(_MODEL_LOSSTYPE)
+
+_MODEL_DISTANCETYPE = _descriptor.EnumDescriptor(
+ name="DistanceType",
+ full_name="google.cloud.bigquery.v2.Model.DistanceType",
+ filename=None,
+ file=DESCRIPTOR,
+ values=[
+ _descriptor.EnumValueDescriptor(
+ name="DISTANCE_TYPE_UNSPECIFIED",
+ index=0,
+ number=0,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="EUCLIDEAN", index=1, number=1, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="COSINE", index=2, number=2, serialized_options=None, type=None
+ ),
+ ],
+ containing_type=None,
+ serialized_options=None,
+ serialized_start=6596,
+ serialized_end=6668,
+)
+_sym_db.RegisterEnumDescriptor(_MODEL_DISTANCETYPE)
+
+_MODEL_DATASPLITMETHOD = _descriptor.EnumDescriptor(
+ name="DataSplitMethod",
+ full_name="google.cloud.bigquery.v2.Model.DataSplitMethod",
+ filename=None,
+ file=DESCRIPTOR,
+ values=[
+ _descriptor.EnumValueDescriptor(
+ name="DATA_SPLIT_METHOD_UNSPECIFIED",
+ index=0,
+ number=0,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="RANDOM", index=1, number=1, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="CUSTOM", index=2, number=2, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="SEQUENTIAL", index=3, number=3, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="NO_SPLIT", index=4, number=4, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="AUTO_SPLIT", index=5, number=5, serialized_options=None, type=None
+ ),
+ ],
+ containing_type=None,
+ serialized_options=None,
+ serialized_start=6670,
+ serialized_end=6792,
+)
+_sym_db.RegisterEnumDescriptor(_MODEL_DATASPLITMETHOD)
+
+_MODEL_LEARNRATESTRATEGY = _descriptor.EnumDescriptor(
+ name="LearnRateStrategy",
+ full_name="google.cloud.bigquery.v2.Model.LearnRateStrategy",
+ filename=None,
+ file=DESCRIPTOR,
+ values=[
+ _descriptor.EnumValueDescriptor(
+ name="LEARN_RATE_STRATEGY_UNSPECIFIED",
+ index=0,
+ number=0,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="LINE_SEARCH", index=1, number=1, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="CONSTANT", index=2, number=2, serialized_options=None, type=None
+ ),
+ ],
+ containing_type=None,
+ serialized_options=None,
+ serialized_start=6794,
+ serialized_end=6881,
+)
+_sym_db.RegisterEnumDescriptor(_MODEL_LEARNRATESTRATEGY)
+
+_MODEL_OPTIMIZATIONSTRATEGY = _descriptor.EnumDescriptor(
+ name="OptimizationStrategy",
+ full_name="google.cloud.bigquery.v2.Model.OptimizationStrategy",
+ filename=None,
+ file=DESCRIPTOR,
+ values=[
+ _descriptor.EnumValueDescriptor(
+ name="OPTIMIZATION_STRATEGY_UNSPECIFIED",
+ index=0,
+ number=0,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="BATCH_GRADIENT_DESCENT",
+ index=1,
+ number=1,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="NORMAL_EQUATION",
+ index=2,
+ number=2,
+ serialized_options=None,
+ type=None,
+ ),
+ ],
+ containing_type=None,
+ serialized_options=None,
+ serialized_start=6883,
+ serialized_end=6993,
+)
+_sym_db.RegisterEnumDescriptor(_MODEL_OPTIMIZATIONSTRATEGY)
+
+
+_MODEL_KMEANSENUMS = _descriptor.Descriptor(
+ name="KmeansEnums",
+ full_name="google.cloud.bigquery.v2.Model.KmeansEnums",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[],
+ extensions=[],
+ nested_types=[],
+ enum_types=[_MODEL_KMEANSENUMS_KMEANSINITIALIZATIONMETHOD],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=883,
+ serialized_end=996,
+)
+
+_MODEL_REGRESSIONMETRICS = _descriptor.Descriptor(
+ name="RegressionMetrics",
+ full_name="google.cloud.bigquery.v2.Model.RegressionMetrics",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="mean_absolute_error",
+ full_name="google.cloud.bigquery.v2.Model.RegressionMetrics.mean_absolute_error",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="mean_squared_error",
+ full_name="google.cloud.bigquery.v2.Model.RegressionMetrics.mean_squared_error",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="mean_squared_log_error",
+ full_name="google.cloud.bigquery.v2.Model.RegressionMetrics.mean_squared_log_error",
+ index=2,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="median_absolute_error",
+ full_name="google.cloud.bigquery.v2.Model.RegressionMetrics.median_absolute_error",
+ index=3,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="r_squared",
+ full_name="google.cloud.bigquery.v2.Model.RegressionMetrics.r_squared",
+ index=4,
+ number=5,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=999,
+ serialized_end=1307,
+)
+
+_MODEL_AGGREGATECLASSIFICATIONMETRICS = _descriptor.Descriptor(
+ name="AggregateClassificationMetrics",
+ full_name="google.cloud.bigquery.v2.Model.AggregateClassificationMetrics",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="precision",
+ full_name="google.cloud.bigquery.v2.Model.AggregateClassificationMetrics.precision",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="recall",
+ full_name="google.cloud.bigquery.v2.Model.AggregateClassificationMetrics.recall",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="accuracy",
+ full_name="google.cloud.bigquery.v2.Model.AggregateClassificationMetrics.accuracy",
+ index=2,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="threshold",
+ full_name="google.cloud.bigquery.v2.Model.AggregateClassificationMetrics.threshold",
+ index=3,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="f1_score",
+ full_name="google.cloud.bigquery.v2.Model.AggregateClassificationMetrics.f1_score",
+ index=4,
+ number=5,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="log_loss",
+ full_name="google.cloud.bigquery.v2.Model.AggregateClassificationMetrics.log_loss",
+ index=5,
+ number=6,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="roc_auc",
+ full_name="google.cloud.bigquery.v2.Model.AggregateClassificationMetrics.roc_auc",
+ index=6,
+ number=7,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1310,
+ serialized_end=1677,
+)
+
+_MODEL_BINARYCLASSIFICATIONMETRICS_BINARYCONFUSIONMATRIX = _descriptor.Descriptor(
+ name="BinaryConfusionMatrix",
+ full_name="google.cloud.bigquery.v2.Model.BinaryClassificationMetrics.BinaryConfusionMatrix",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="positive_class_threshold",
+ full_name="google.cloud.bigquery.v2.Model.BinaryClassificationMetrics.BinaryConfusionMatrix.positive_class_threshold",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="true_positives",
+ full_name="google.cloud.bigquery.v2.Model.BinaryClassificationMetrics.BinaryConfusionMatrix.true_positives",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="false_positives",
+ full_name="google.cloud.bigquery.v2.Model.BinaryClassificationMetrics.BinaryConfusionMatrix.false_positives",
+ index=2,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="true_negatives",
+ full_name="google.cloud.bigquery.v2.Model.BinaryClassificationMetrics.BinaryConfusionMatrix.true_negatives",
+ index=3,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="false_negatives",
+ full_name="google.cloud.bigquery.v2.Model.BinaryClassificationMetrics.BinaryConfusionMatrix.false_negatives",
+ index=4,
+ number=5,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="precision",
+ full_name="google.cloud.bigquery.v2.Model.BinaryClassificationMetrics.BinaryConfusionMatrix.precision",
+ index=5,
+ number=6,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="recall",
+ full_name="google.cloud.bigquery.v2.Model.BinaryClassificationMetrics.BinaryConfusionMatrix.recall",
+ index=6,
+ number=7,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="f1_score",
+ full_name="google.cloud.bigquery.v2.Model.BinaryClassificationMetrics.BinaryConfusionMatrix.f1_score",
+ index=7,
+ number=8,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="accuracy",
+ full_name="google.cloud.bigquery.v2.Model.BinaryClassificationMetrics.BinaryConfusionMatrix.accuracy",
+ index=8,
+ number=9,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1987,
+ serialized_end=2479,
+)
+
+_MODEL_BINARYCLASSIFICATIONMETRICS = _descriptor.Descriptor(
+ name="BinaryClassificationMetrics",
+ full_name="google.cloud.bigquery.v2.Model.BinaryClassificationMetrics",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="aggregate_classification_metrics",
+ full_name="google.cloud.bigquery.v2.Model.BinaryClassificationMetrics.aggregate_classification_metrics",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="binary_confusion_matrix_list",
+ full_name="google.cloud.bigquery.v2.Model.BinaryClassificationMetrics.binary_confusion_matrix_list",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="positive_label",
+ full_name="google.cloud.bigquery.v2.Model.BinaryClassificationMetrics.positive_label",
+ index=2,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="negative_label",
+ full_name="google.cloud.bigquery.v2.Model.BinaryClassificationMetrics.negative_label",
+ index=3,
+ number=4,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[_MODEL_BINARYCLASSIFICATIONMETRICS_BINARYCONFUSIONMATRIX],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1680,
+ serialized_end=2479,
+)
+
+_MODEL_MULTICLASSCLASSIFICATIONMETRICS_CONFUSIONMATRIX_ENTRY = _descriptor.Descriptor(
+ name="Entry",
+ full_name="google.cloud.bigquery.v2.Model.MultiClassClassificationMetrics.ConfusionMatrix.Entry",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="predicted_label",
+ full_name="google.cloud.bigquery.v2.Model.MultiClassClassificationMetrics.ConfusionMatrix.Entry.predicted_label",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="item_count",
+ full_name="google.cloud.bigquery.v2.Model.MultiClassClassificationMetrics.ConfusionMatrix.Entry.item_count",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2914,
+ serialized_end=2995,
+)
+
+_MODEL_MULTICLASSCLASSIFICATIONMETRICS_CONFUSIONMATRIX_ROW = _descriptor.Descriptor(
+ name="Row",
+ full_name="google.cloud.bigquery.v2.Model.MultiClassClassificationMetrics.ConfusionMatrix.Row",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="actual_label",
+ full_name="google.cloud.bigquery.v2.Model.MultiClassClassificationMetrics.ConfusionMatrix.Row.actual_label",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="entries",
+ full_name="google.cloud.bigquery.v2.Model.MultiClassClassificationMetrics.ConfusionMatrix.Row.entries",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2998,
+ serialized_end=3129,
+)
+
+_MODEL_MULTICLASSCLASSIFICATIONMETRICS_CONFUSIONMATRIX = _descriptor.Descriptor(
+ name="ConfusionMatrix",
+ full_name="google.cloud.bigquery.v2.Model.MultiClassClassificationMetrics.ConfusionMatrix",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="confidence_threshold",
+ full_name="google.cloud.bigquery.v2.Model.MultiClassClassificationMetrics.ConfusionMatrix.confidence_threshold",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="rows",
+ full_name="google.cloud.bigquery.v2.Model.MultiClassClassificationMetrics.ConfusionMatrix.rows",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[
+ _MODEL_MULTICLASSCLASSIFICATIONMETRICS_CONFUSIONMATRIX_ENTRY,
+ _MODEL_MULTICLASSCLASSIFICATIONMETRICS_CONFUSIONMATRIX_ROW,
+ ],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2736,
+ serialized_end=3129,
+)
+
+_MODEL_MULTICLASSCLASSIFICATIONMETRICS = _descriptor.Descriptor(
+ name="MultiClassClassificationMetrics",
+ full_name="google.cloud.bigquery.v2.Model.MultiClassClassificationMetrics",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="aggregate_classification_metrics",
+ full_name="google.cloud.bigquery.v2.Model.MultiClassClassificationMetrics.aggregate_classification_metrics",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="confusion_matrix_list",
+ full_name="google.cloud.bigquery.v2.Model.MultiClassClassificationMetrics.confusion_matrix_list",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[_MODEL_MULTICLASSCLASSIFICATIONMETRICS_CONFUSIONMATRIX],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2482,
+ serialized_end=3129,
+)
+
+_MODEL_CLUSTERINGMETRICS_CLUSTER_FEATUREVALUE_CATEGORICALVALUE_CATEGORYCOUNT = _descriptor.Descriptor(
+ name="CategoryCount",
+ full_name="google.cloud.bigquery.v2.Model.ClusteringMetrics.Cluster.FeatureValue.CategoricalValue.CategoryCount",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="category",
+ full_name="google.cloud.bigquery.v2.Model.ClusteringMetrics.Cluster.FeatureValue.CategoricalValue.CategoryCount.category",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="count",
+ full_name="google.cloud.bigquery.v2.Model.ClusteringMetrics.Cluster.FeatureValue.CategoricalValue.CategoryCount.count",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=3889,
+ serialized_end=3966,
+)
+
+_MODEL_CLUSTERINGMETRICS_CLUSTER_FEATUREVALUE_CATEGORICALVALUE = _descriptor.Descriptor(
+ name="CategoricalValue",
+ full_name="google.cloud.bigquery.v2.Model.ClusteringMetrics.Cluster.FeatureValue.CategoricalValue",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="category_counts",
+ full_name="google.cloud.bigquery.v2.Model.ClusteringMetrics.Cluster.FeatureValue.CategoricalValue.category_counts",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[
+ _MODEL_CLUSTERINGMETRICS_CLUSTER_FEATUREVALUE_CATEGORICALVALUE_CATEGORYCOUNT
+ ],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=3741,
+ serialized_end=3966,
+)
+
+_MODEL_CLUSTERINGMETRICS_CLUSTER_FEATUREVALUE = _descriptor.Descriptor(
+ name="FeatureValue",
+ full_name="google.cloud.bigquery.v2.Model.ClusteringMetrics.Cluster.FeatureValue",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="feature_column",
+ full_name="google.cloud.bigquery.v2.Model.ClusteringMetrics.Cluster.FeatureValue.feature_column",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="numerical_value",
+ full_name="google.cloud.bigquery.v2.Model.ClusteringMetrics.Cluster.FeatureValue.numerical_value",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="categorical_value",
+ full_name="google.cloud.bigquery.v2.Model.ClusteringMetrics.Cluster.FeatureValue.categorical_value",
+ index=2,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[_MODEL_CLUSTERINGMETRICS_CLUSTER_FEATUREVALUE_CATEGORICALVALUE],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="value",
+ full_name="google.cloud.bigquery.v2.Model.ClusteringMetrics.Cluster.FeatureValue.value",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=3525,
+ serialized_end=3975,
+)
+
+_MODEL_CLUSTERINGMETRICS_CLUSTER = _descriptor.Descriptor(
+ name="Cluster",
+ full_name="google.cloud.bigquery.v2.Model.ClusteringMetrics.Cluster",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="centroid_id",
+ full_name="google.cloud.bigquery.v2.Model.ClusteringMetrics.Cluster.centroid_id",
+ index=0,
+ number=1,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="feature_values",
+ full_name="google.cloud.bigquery.v2.Model.ClusteringMetrics.Cluster.feature_values",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="count",
+ full_name="google.cloud.bigquery.v2.Model.ClusteringMetrics.Cluster.count",
+ index=2,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[_MODEL_CLUSTERINGMETRICS_CLUSTER_FEATUREVALUE],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=3352,
+ serialized_end=3975,
+)
+
+_MODEL_CLUSTERINGMETRICS = _descriptor.Descriptor(
+ name="ClusteringMetrics",
+ full_name="google.cloud.bigquery.v2.Model.ClusteringMetrics",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="davies_bouldin_index",
+ full_name="google.cloud.bigquery.v2.Model.ClusteringMetrics.davies_bouldin_index",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="mean_squared_distance",
+ full_name="google.cloud.bigquery.v2.Model.ClusteringMetrics.mean_squared_distance",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="clusters",
+ full_name="google.cloud.bigquery.v2.Model.ClusteringMetrics.clusters",
+ index=2,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[_MODEL_CLUSTERINGMETRICS_CLUSTER],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=3132,
+ serialized_end=3975,
+)
+
+_MODEL_EVALUATIONMETRICS = _descriptor.Descriptor(
+ name="EvaluationMetrics",
+ full_name="google.cloud.bigquery.v2.Model.EvaluationMetrics",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="regression_metrics",
+ full_name="google.cloud.bigquery.v2.Model.EvaluationMetrics.regression_metrics",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="binary_classification_metrics",
+ full_name="google.cloud.bigquery.v2.Model.EvaluationMetrics.binary_classification_metrics",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="multi_class_classification_metrics",
+ full_name="google.cloud.bigquery.v2.Model.EvaluationMetrics.multi_class_classification_metrics",
+ index=2,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="clustering_metrics",
+ full_name="google.cloud.bigquery.v2.Model.EvaluationMetrics.clustering_metrics",
+ index=3,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="metrics",
+ full_name="google.cloud.bigquery.v2.Model.EvaluationMetrics.metrics",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=3978,
+ serialized_end=4383,
+)
+
+_MODEL_TRAININGRUN_TRAININGOPTIONS_LABELCLASSWEIGHTSENTRY = _descriptor.Descriptor(
+ name="LabelClassWeightsEntry",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.TrainingOptions.LabelClassWeightsEntry",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="key",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.TrainingOptions.LabelClassWeightsEntry.key",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="value",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.TrainingOptions.LabelClassWeightsEntry.value",
+ index=1,
+ number=2,
+ type=1,
+ cpp_type=5,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=_b("8\001"),
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=5819,
+ serialized_end=5875,
+)
+
+_MODEL_TRAININGRUN_TRAININGOPTIONS = _descriptor.Descriptor(
+ name="TrainingOptions",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.TrainingOptions",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="max_iterations",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.TrainingOptions.max_iterations",
+ index=0,
+ number=1,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="loss_type",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.TrainingOptions.loss_type",
+ index=1,
+ number=2,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="learn_rate",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.TrainingOptions.learn_rate",
+ index=2,
+ number=3,
+ type=1,
+ cpp_type=5,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="l1_regularization",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.TrainingOptions.l1_regularization",
+ index=3,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="l2_regularization",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.TrainingOptions.l2_regularization",
+ index=4,
+ number=5,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="min_relative_progress",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.TrainingOptions.min_relative_progress",
+ index=5,
+ number=6,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="warm_start",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.TrainingOptions.warm_start",
+ index=6,
+ number=7,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="early_stop",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.TrainingOptions.early_stop",
+ index=7,
+ number=8,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="input_label_columns",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.TrainingOptions.input_label_columns",
+ index=8,
+ number=9,
+ type=9,
+ cpp_type=9,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="data_split_method",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.TrainingOptions.data_split_method",
+ index=9,
+ number=10,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="data_split_eval_fraction",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.TrainingOptions.data_split_eval_fraction",
+ index=10,
+ number=11,
+ type=1,
+ cpp_type=5,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="data_split_column",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.TrainingOptions.data_split_column",
+ index=11,
+ number=12,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="learn_rate_strategy",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.TrainingOptions.learn_rate_strategy",
+ index=12,
+ number=13,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="initial_learn_rate",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.TrainingOptions.initial_learn_rate",
+ index=13,
+ number=16,
+ type=1,
+ cpp_type=5,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="label_class_weights",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.TrainingOptions.label_class_weights",
+ index=14,
+ number=17,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="distance_type",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.TrainingOptions.distance_type",
+ index=15,
+ number=20,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="num_clusters",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.TrainingOptions.num_clusters",
+ index=16,
+ number=21,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="model_uri",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.TrainingOptions.model_uri",
+ index=17,
+ number=22,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="optimization_strategy",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.TrainingOptions.optimization_strategy",
+ index=18,
+ number=23,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="kmeans_initialization_method",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.TrainingOptions.kmeans_initialization_method",
+ index=19,
+ number=33,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="kmeans_initialization_column",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.TrainingOptions.kmeans_initialization_column",
+ index=20,
+ number=34,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[_MODEL_TRAININGRUN_TRAININGOPTIONS_LABELCLASSWEIGHTSENTRY],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=4694,
+ serialized_end=5875,
+)
+
+_MODEL_TRAININGRUN_ITERATIONRESULT_CLUSTERINFO = _descriptor.Descriptor(
+ name="ClusterInfo",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.IterationResult.ClusterInfo",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="centroid_id",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.IterationResult.ClusterInfo.centroid_id",
+ index=0,
+ number=1,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="cluster_radius",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.IterationResult.ClusterInfo.cluster_radius",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="cluster_size",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.IterationResult.ClusterInfo.cluster_size",
+ index=2,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=6210,
+ serialized_end=6349,
+)
+
+_MODEL_TRAININGRUN_ITERATIONRESULT = _descriptor.Descriptor(
+ name="IterationResult",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.IterationResult",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="index",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.IterationResult.index",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="duration_ms",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.IterationResult.duration_ms",
+ index=1,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="training_loss",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.IterationResult.training_loss",
+ index=2,
+ number=5,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="eval_loss",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.IterationResult.eval_loss",
+ index=3,
+ number=6,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="learn_rate",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.IterationResult.learn_rate",
+ index=4,
+ number=7,
+ type=1,
+ cpp_type=5,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="cluster_infos",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.IterationResult.cluster_infos",
+ index=5,
+ number=8,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[_MODEL_TRAININGRUN_ITERATIONRESULT_CLUSTERINFO],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=5878,
+ serialized_end=6349,
+)
+
+_MODEL_TRAININGRUN = _descriptor.Descriptor(
+ name="TrainingRun",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="training_options",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.training_options",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="start_time",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.start_time",
+ index=1,
+ number=8,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="results",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.results",
+ index=2,
+ number=6,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="evaluation_metrics",
+ full_name="google.cloud.bigquery.v2.Model.TrainingRun.evaluation_metrics",
+ index=3,
+ number=7,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[
+ _MODEL_TRAININGRUN_TRAININGOPTIONS,
+ _MODEL_TRAININGRUN_ITERATIONRESULT,
+ ],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=4386,
+ serialized_end=6349,
+)
+
+_MODEL_LABELSENTRY = _descriptor.Descriptor(
+ name="LabelsEntry",
+ full_name="google.cloud.bigquery.v2.Model.LabelsEntry",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="key",
+ full_name="google.cloud.bigquery.v2.Model.LabelsEntry.key",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="value",
+ full_name="google.cloud.bigquery.v2.Model.LabelsEntry.value",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=_b("8\001"),
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=6351,
+ serialized_end=6396,
+)
+
+_MODEL = _descriptor.Descriptor(
+ name="Model",
+ full_name="google.cloud.bigquery.v2.Model",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="etag",
+ full_name="google.cloud.bigquery.v2.Model.etag",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="model_reference",
+ full_name="google.cloud.bigquery.v2.Model.model_reference",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="creation_time",
+ full_name="google.cloud.bigquery.v2.Model.creation_time",
+ index=2,
+ number=5,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="last_modified_time",
+ full_name="google.cloud.bigquery.v2.Model.last_modified_time",
+ index=3,
+ number=6,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="description",
+ full_name="google.cloud.bigquery.v2.Model.description",
+ index=4,
+ number=12,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="friendly_name",
+ full_name="google.cloud.bigquery.v2.Model.friendly_name",
+ index=5,
+ number=14,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="labels",
+ full_name="google.cloud.bigquery.v2.Model.labels",
+ index=6,
+ number=15,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="expiration_time",
+ full_name="google.cloud.bigquery.v2.Model.expiration_time",
+ index=7,
+ number=16,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="location",
+ full_name="google.cloud.bigquery.v2.Model.location",
+ index=8,
+ number=13,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="model_type",
+ full_name="google.cloud.bigquery.v2.Model.model_type",
+ index=9,
+ number=7,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="training_runs",
+ full_name="google.cloud.bigquery.v2.Model.training_runs",
+ index=10,
+ number=9,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="feature_columns",
+ full_name="google.cloud.bigquery.v2.Model.feature_columns",
+ index=11,
+ number=10,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="label_columns",
+ full_name="google.cloud.bigquery.v2.Model.label_columns",
+ index=12,
+ number=11,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[
+ _MODEL_KMEANSENUMS,
+ _MODEL_REGRESSIONMETRICS,
+ _MODEL_AGGREGATECLASSIFICATIONMETRICS,
+ _MODEL_BINARYCLASSIFICATIONMETRICS,
+ _MODEL_MULTICLASSCLASSIFICATIONMETRICS,
+ _MODEL_CLUSTERINGMETRICS,
+ _MODEL_EVALUATIONMETRICS,
+ _MODEL_TRAININGRUN,
+ _MODEL_LABELSENTRY,
+ ],
+ enum_types=[
+ _MODEL_MODELTYPE,
+ _MODEL_LOSSTYPE,
+ _MODEL_DISTANCETYPE,
+ _MODEL_DATASPLITMETHOD,
+ _MODEL_LEARNRATESTRATEGY,
+ _MODEL_OPTIMIZATIONSTRATEGY,
+ ],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=327,
+ serialized_end=6993,
+)
+
+
+_GETMODELREQUEST = _descriptor.Descriptor(
+ name="GetModelRequest",
+ full_name="google.cloud.bigquery.v2.GetModelRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="project_id",
+ full_name="google.cloud.bigquery.v2.GetModelRequest.project_id",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="dataset_id",
+ full_name="google.cloud.bigquery.v2.GetModelRequest.dataset_id",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="model_id",
+ full_name="google.cloud.bigquery.v2.GetModelRequest.model_id",
+ index=2,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=6995,
+ serialized_end=7070,
+)
+
+
+_PATCHMODELREQUEST = _descriptor.Descriptor(
+ name="PatchModelRequest",
+ full_name="google.cloud.bigquery.v2.PatchModelRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="project_id",
+ full_name="google.cloud.bigquery.v2.PatchModelRequest.project_id",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="dataset_id",
+ full_name="google.cloud.bigquery.v2.PatchModelRequest.dataset_id",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="model_id",
+ full_name="google.cloud.bigquery.v2.PatchModelRequest.model_id",
+ index=2,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="model",
+ full_name="google.cloud.bigquery.v2.PatchModelRequest.model",
+ index=3,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=7072,
+ serialized_end=7197,
+)
+
+
+_DELETEMODELREQUEST = _descriptor.Descriptor(
+ name="DeleteModelRequest",
+ full_name="google.cloud.bigquery.v2.DeleteModelRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="project_id",
+ full_name="google.cloud.bigquery.v2.DeleteModelRequest.project_id",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="dataset_id",
+ full_name="google.cloud.bigquery.v2.DeleteModelRequest.dataset_id",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="model_id",
+ full_name="google.cloud.bigquery.v2.DeleteModelRequest.model_id",
+ index=2,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=7199,
+ serialized_end=7277,
+)
+
+
+_LISTMODELSREQUEST = _descriptor.Descriptor(
+ name="ListModelsRequest",
+ full_name="google.cloud.bigquery.v2.ListModelsRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="project_id",
+ full_name="google.cloud.bigquery.v2.ListModelsRequest.project_id",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="dataset_id",
+ full_name="google.cloud.bigquery.v2.ListModelsRequest.dataset_id",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="max_results",
+ full_name="google.cloud.bigquery.v2.ListModelsRequest.max_results",
+ index=2,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="page_token",
+ full_name="google.cloud.bigquery.v2.ListModelsRequest.page_token",
+ index=3,
+ number=4,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=7280,
+ serialized_end=7410,
+)
+
+
+_LISTMODELSRESPONSE = _descriptor.Descriptor(
+ name="ListModelsResponse",
+ full_name="google.cloud.bigquery.v2.ListModelsResponse",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="models",
+ full_name="google.cloud.bigquery.v2.ListModelsResponse.models",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="next_page_token",
+ full_name="google.cloud.bigquery.v2.ListModelsResponse.next_page_token",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=7412,
+ serialized_end=7506,
+)
+
+_MODEL_KMEANSENUMS.containing_type = _MODEL
+_MODEL_KMEANSENUMS_KMEANSINITIALIZATIONMETHOD.containing_type = _MODEL_KMEANSENUMS
+_MODEL_REGRESSIONMETRICS.fields_by_name[
+ "mean_absolute_error"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._DOUBLEVALUE
+_MODEL_REGRESSIONMETRICS.fields_by_name[
+ "mean_squared_error"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._DOUBLEVALUE
+_MODEL_REGRESSIONMETRICS.fields_by_name[
+ "mean_squared_log_error"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._DOUBLEVALUE
+_MODEL_REGRESSIONMETRICS.fields_by_name[
+ "median_absolute_error"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._DOUBLEVALUE
+_MODEL_REGRESSIONMETRICS.fields_by_name[
+ "r_squared"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._DOUBLEVALUE
+_MODEL_REGRESSIONMETRICS.containing_type = _MODEL
+_MODEL_AGGREGATECLASSIFICATIONMETRICS.fields_by_name[
+ "precision"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._DOUBLEVALUE
+_MODEL_AGGREGATECLASSIFICATIONMETRICS.fields_by_name[
+ "recall"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._DOUBLEVALUE
+_MODEL_AGGREGATECLASSIFICATIONMETRICS.fields_by_name[
+ "accuracy"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._DOUBLEVALUE
+_MODEL_AGGREGATECLASSIFICATIONMETRICS.fields_by_name[
+ "threshold"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._DOUBLEVALUE
+_MODEL_AGGREGATECLASSIFICATIONMETRICS.fields_by_name[
+ "f1_score"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._DOUBLEVALUE
+_MODEL_AGGREGATECLASSIFICATIONMETRICS.fields_by_name[
+ "log_loss"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._DOUBLEVALUE
+_MODEL_AGGREGATECLASSIFICATIONMETRICS.fields_by_name[
+ "roc_auc"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._DOUBLEVALUE
+_MODEL_AGGREGATECLASSIFICATIONMETRICS.containing_type = _MODEL
+_MODEL_BINARYCLASSIFICATIONMETRICS_BINARYCONFUSIONMATRIX.fields_by_name[
+ "positive_class_threshold"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._DOUBLEVALUE
+_MODEL_BINARYCLASSIFICATIONMETRICS_BINARYCONFUSIONMATRIX.fields_by_name[
+ "true_positives"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._INT64VALUE
+_MODEL_BINARYCLASSIFICATIONMETRICS_BINARYCONFUSIONMATRIX.fields_by_name[
+ "false_positives"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._INT64VALUE
+_MODEL_BINARYCLASSIFICATIONMETRICS_BINARYCONFUSIONMATRIX.fields_by_name[
+ "true_negatives"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._INT64VALUE
+_MODEL_BINARYCLASSIFICATIONMETRICS_BINARYCONFUSIONMATRIX.fields_by_name[
+ "false_negatives"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._INT64VALUE
+_MODEL_BINARYCLASSIFICATIONMETRICS_BINARYCONFUSIONMATRIX.fields_by_name[
+ "precision"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._DOUBLEVALUE
+_MODEL_BINARYCLASSIFICATIONMETRICS_BINARYCONFUSIONMATRIX.fields_by_name[
+ "recall"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._DOUBLEVALUE
+_MODEL_BINARYCLASSIFICATIONMETRICS_BINARYCONFUSIONMATRIX.fields_by_name[
+ "f1_score"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._DOUBLEVALUE
+_MODEL_BINARYCLASSIFICATIONMETRICS_BINARYCONFUSIONMATRIX.fields_by_name[
+ "accuracy"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._DOUBLEVALUE
+_MODEL_BINARYCLASSIFICATIONMETRICS_BINARYCONFUSIONMATRIX.containing_type = (
+ _MODEL_BINARYCLASSIFICATIONMETRICS
+)
+_MODEL_BINARYCLASSIFICATIONMETRICS.fields_by_name[
+ "aggregate_classification_metrics"
+].message_type = _MODEL_AGGREGATECLASSIFICATIONMETRICS
+_MODEL_BINARYCLASSIFICATIONMETRICS.fields_by_name[
+ "binary_confusion_matrix_list"
+].message_type = _MODEL_BINARYCLASSIFICATIONMETRICS_BINARYCONFUSIONMATRIX
+_MODEL_BINARYCLASSIFICATIONMETRICS.containing_type = _MODEL
+_MODEL_MULTICLASSCLASSIFICATIONMETRICS_CONFUSIONMATRIX_ENTRY.fields_by_name[
+ "item_count"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._INT64VALUE
+_MODEL_MULTICLASSCLASSIFICATIONMETRICS_CONFUSIONMATRIX_ENTRY.containing_type = (
+ _MODEL_MULTICLASSCLASSIFICATIONMETRICS_CONFUSIONMATRIX
+)
+_MODEL_MULTICLASSCLASSIFICATIONMETRICS_CONFUSIONMATRIX_ROW.fields_by_name[
+ "entries"
+].message_type = _MODEL_MULTICLASSCLASSIFICATIONMETRICS_CONFUSIONMATRIX_ENTRY
+_MODEL_MULTICLASSCLASSIFICATIONMETRICS_CONFUSIONMATRIX_ROW.containing_type = (
+ _MODEL_MULTICLASSCLASSIFICATIONMETRICS_CONFUSIONMATRIX
+)
+_MODEL_MULTICLASSCLASSIFICATIONMETRICS_CONFUSIONMATRIX.fields_by_name[
+ "confidence_threshold"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._DOUBLEVALUE
+_MODEL_MULTICLASSCLASSIFICATIONMETRICS_CONFUSIONMATRIX.fields_by_name[
+ "rows"
+].message_type = _MODEL_MULTICLASSCLASSIFICATIONMETRICS_CONFUSIONMATRIX_ROW
+_MODEL_MULTICLASSCLASSIFICATIONMETRICS_CONFUSIONMATRIX.containing_type = (
+ _MODEL_MULTICLASSCLASSIFICATIONMETRICS
+)
+_MODEL_MULTICLASSCLASSIFICATIONMETRICS.fields_by_name[
+ "aggregate_classification_metrics"
+].message_type = _MODEL_AGGREGATECLASSIFICATIONMETRICS
+_MODEL_MULTICLASSCLASSIFICATIONMETRICS.fields_by_name[
+ "confusion_matrix_list"
+].message_type = _MODEL_MULTICLASSCLASSIFICATIONMETRICS_CONFUSIONMATRIX
+_MODEL_MULTICLASSCLASSIFICATIONMETRICS.containing_type = _MODEL
+_MODEL_CLUSTERINGMETRICS_CLUSTER_FEATUREVALUE_CATEGORICALVALUE_CATEGORYCOUNT.fields_by_name[
+ "count"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._INT64VALUE
+_MODEL_CLUSTERINGMETRICS_CLUSTER_FEATUREVALUE_CATEGORICALVALUE_CATEGORYCOUNT.containing_type = (
+ _MODEL_CLUSTERINGMETRICS_CLUSTER_FEATUREVALUE_CATEGORICALVALUE
+)
+_MODEL_CLUSTERINGMETRICS_CLUSTER_FEATUREVALUE_CATEGORICALVALUE.fields_by_name[
+ "category_counts"
+].message_type = (
+ _MODEL_CLUSTERINGMETRICS_CLUSTER_FEATUREVALUE_CATEGORICALVALUE_CATEGORYCOUNT
+)
+_MODEL_CLUSTERINGMETRICS_CLUSTER_FEATUREVALUE_CATEGORICALVALUE.containing_type = (
+ _MODEL_CLUSTERINGMETRICS_CLUSTER_FEATUREVALUE
+)
+_MODEL_CLUSTERINGMETRICS_CLUSTER_FEATUREVALUE.fields_by_name[
+ "numerical_value"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._DOUBLEVALUE
+_MODEL_CLUSTERINGMETRICS_CLUSTER_FEATUREVALUE.fields_by_name[
+ "categorical_value"
+].message_type = _MODEL_CLUSTERINGMETRICS_CLUSTER_FEATUREVALUE_CATEGORICALVALUE
+_MODEL_CLUSTERINGMETRICS_CLUSTER_FEATUREVALUE.containing_type = (
+ _MODEL_CLUSTERINGMETRICS_CLUSTER
+)
+_MODEL_CLUSTERINGMETRICS_CLUSTER_FEATUREVALUE.oneofs_by_name["value"].fields.append(
+ _MODEL_CLUSTERINGMETRICS_CLUSTER_FEATUREVALUE.fields_by_name["numerical_value"]
+)
+_MODEL_CLUSTERINGMETRICS_CLUSTER_FEATUREVALUE.fields_by_name[
+ "numerical_value"
+].containing_oneof = _MODEL_CLUSTERINGMETRICS_CLUSTER_FEATUREVALUE.oneofs_by_name[
+ "value"
+]
+_MODEL_CLUSTERINGMETRICS_CLUSTER_FEATUREVALUE.oneofs_by_name["value"].fields.append(
+ _MODEL_CLUSTERINGMETRICS_CLUSTER_FEATUREVALUE.fields_by_name["categorical_value"]
+)
+_MODEL_CLUSTERINGMETRICS_CLUSTER_FEATUREVALUE.fields_by_name[
+ "categorical_value"
+].containing_oneof = _MODEL_CLUSTERINGMETRICS_CLUSTER_FEATUREVALUE.oneofs_by_name[
+ "value"
+]
+_MODEL_CLUSTERINGMETRICS_CLUSTER.fields_by_name[
+ "feature_values"
+].message_type = _MODEL_CLUSTERINGMETRICS_CLUSTER_FEATUREVALUE
+_MODEL_CLUSTERINGMETRICS_CLUSTER.fields_by_name[
+ "count"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._INT64VALUE
+_MODEL_CLUSTERINGMETRICS_CLUSTER.containing_type = _MODEL_CLUSTERINGMETRICS
+_MODEL_CLUSTERINGMETRICS.fields_by_name[
+ "davies_bouldin_index"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._DOUBLEVALUE
+_MODEL_CLUSTERINGMETRICS.fields_by_name[
+ "mean_squared_distance"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._DOUBLEVALUE
+_MODEL_CLUSTERINGMETRICS.fields_by_name[
+ "clusters"
+].message_type = _MODEL_CLUSTERINGMETRICS_CLUSTER
+_MODEL_CLUSTERINGMETRICS.containing_type = _MODEL
+_MODEL_EVALUATIONMETRICS.fields_by_name[
+ "regression_metrics"
+].message_type = _MODEL_REGRESSIONMETRICS
+_MODEL_EVALUATIONMETRICS.fields_by_name[
+ "binary_classification_metrics"
+].message_type = _MODEL_BINARYCLASSIFICATIONMETRICS
+_MODEL_EVALUATIONMETRICS.fields_by_name[
+ "multi_class_classification_metrics"
+].message_type = _MODEL_MULTICLASSCLASSIFICATIONMETRICS
+_MODEL_EVALUATIONMETRICS.fields_by_name[
+ "clustering_metrics"
+].message_type = _MODEL_CLUSTERINGMETRICS
+_MODEL_EVALUATIONMETRICS.containing_type = _MODEL
+_MODEL_EVALUATIONMETRICS.oneofs_by_name["metrics"].fields.append(
+ _MODEL_EVALUATIONMETRICS.fields_by_name["regression_metrics"]
+)
+_MODEL_EVALUATIONMETRICS.fields_by_name[
+ "regression_metrics"
+].containing_oneof = _MODEL_EVALUATIONMETRICS.oneofs_by_name["metrics"]
+_MODEL_EVALUATIONMETRICS.oneofs_by_name["metrics"].fields.append(
+ _MODEL_EVALUATIONMETRICS.fields_by_name["binary_classification_metrics"]
+)
+_MODEL_EVALUATIONMETRICS.fields_by_name[
+ "binary_classification_metrics"
+].containing_oneof = _MODEL_EVALUATIONMETRICS.oneofs_by_name["metrics"]
+_MODEL_EVALUATIONMETRICS.oneofs_by_name["metrics"].fields.append(
+ _MODEL_EVALUATIONMETRICS.fields_by_name["multi_class_classification_metrics"]
+)
+_MODEL_EVALUATIONMETRICS.fields_by_name[
+ "multi_class_classification_metrics"
+].containing_oneof = _MODEL_EVALUATIONMETRICS.oneofs_by_name["metrics"]
+_MODEL_EVALUATIONMETRICS.oneofs_by_name["metrics"].fields.append(
+ _MODEL_EVALUATIONMETRICS.fields_by_name["clustering_metrics"]
+)
+_MODEL_EVALUATIONMETRICS.fields_by_name[
+ "clustering_metrics"
+].containing_oneof = _MODEL_EVALUATIONMETRICS.oneofs_by_name["metrics"]
+_MODEL_TRAININGRUN_TRAININGOPTIONS_LABELCLASSWEIGHTSENTRY.containing_type = (
+ _MODEL_TRAININGRUN_TRAININGOPTIONS
+)
+_MODEL_TRAININGRUN_TRAININGOPTIONS.fields_by_name[
+ "loss_type"
+].enum_type = _MODEL_LOSSTYPE
+_MODEL_TRAININGRUN_TRAININGOPTIONS.fields_by_name[
+ "l1_regularization"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._DOUBLEVALUE
+_MODEL_TRAININGRUN_TRAININGOPTIONS.fields_by_name[
+ "l2_regularization"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._DOUBLEVALUE
+_MODEL_TRAININGRUN_TRAININGOPTIONS.fields_by_name[
+ "min_relative_progress"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._DOUBLEVALUE
+_MODEL_TRAININGRUN_TRAININGOPTIONS.fields_by_name[
+ "warm_start"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._BOOLVALUE
+_MODEL_TRAININGRUN_TRAININGOPTIONS.fields_by_name[
+ "early_stop"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._BOOLVALUE
+_MODEL_TRAININGRUN_TRAININGOPTIONS.fields_by_name[
+ "data_split_method"
+].enum_type = _MODEL_DATASPLITMETHOD
+_MODEL_TRAININGRUN_TRAININGOPTIONS.fields_by_name[
+ "learn_rate_strategy"
+].enum_type = _MODEL_LEARNRATESTRATEGY
+_MODEL_TRAININGRUN_TRAININGOPTIONS.fields_by_name[
+ "label_class_weights"
+].message_type = _MODEL_TRAININGRUN_TRAININGOPTIONS_LABELCLASSWEIGHTSENTRY
+_MODEL_TRAININGRUN_TRAININGOPTIONS.fields_by_name[
+ "distance_type"
+].enum_type = _MODEL_DISTANCETYPE
+_MODEL_TRAININGRUN_TRAININGOPTIONS.fields_by_name[
+ "optimization_strategy"
+].enum_type = _MODEL_OPTIMIZATIONSTRATEGY
+_MODEL_TRAININGRUN_TRAININGOPTIONS.fields_by_name[
+ "kmeans_initialization_method"
+].enum_type = _MODEL_KMEANSENUMS_KMEANSINITIALIZATIONMETHOD
+_MODEL_TRAININGRUN_TRAININGOPTIONS.containing_type = _MODEL_TRAININGRUN
+_MODEL_TRAININGRUN_ITERATIONRESULT_CLUSTERINFO.fields_by_name[
+ "cluster_radius"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._DOUBLEVALUE
+_MODEL_TRAININGRUN_ITERATIONRESULT_CLUSTERINFO.fields_by_name[
+ "cluster_size"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._INT64VALUE
+_MODEL_TRAININGRUN_ITERATIONRESULT_CLUSTERINFO.containing_type = (
+ _MODEL_TRAININGRUN_ITERATIONRESULT
+)
+_MODEL_TRAININGRUN_ITERATIONRESULT.fields_by_name[
+ "index"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._INT32VALUE
+_MODEL_TRAININGRUN_ITERATIONRESULT.fields_by_name[
+ "duration_ms"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._INT64VALUE
+_MODEL_TRAININGRUN_ITERATIONRESULT.fields_by_name[
+ "training_loss"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._DOUBLEVALUE
+_MODEL_TRAININGRUN_ITERATIONRESULT.fields_by_name[
+ "eval_loss"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._DOUBLEVALUE
+_MODEL_TRAININGRUN_ITERATIONRESULT.fields_by_name[
+ "cluster_infos"
+].message_type = _MODEL_TRAININGRUN_ITERATIONRESULT_CLUSTERINFO
+_MODEL_TRAININGRUN_ITERATIONRESULT.containing_type = _MODEL_TRAININGRUN
+_MODEL_TRAININGRUN.fields_by_name[
+ "training_options"
+].message_type = _MODEL_TRAININGRUN_TRAININGOPTIONS
+_MODEL_TRAININGRUN.fields_by_name[
+ "start_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_MODEL_TRAININGRUN.fields_by_name[
+ "results"
+].message_type = _MODEL_TRAININGRUN_ITERATIONRESULT
+_MODEL_TRAININGRUN.fields_by_name[
+ "evaluation_metrics"
+].message_type = _MODEL_EVALUATIONMETRICS
+_MODEL_TRAININGRUN.containing_type = _MODEL
+_MODEL_LABELSENTRY.containing_type = _MODEL
+_MODEL.fields_by_name[
+ "model_reference"
+].message_type = (
+ google_dot_cloud_dot_bigquery__v2_dot_proto_dot_model__reference__pb2._MODELREFERENCE
+)
+_MODEL.fields_by_name["labels"].message_type = _MODEL_LABELSENTRY
+_MODEL.fields_by_name["model_type"].enum_type = _MODEL_MODELTYPE
+_MODEL.fields_by_name["training_runs"].message_type = _MODEL_TRAININGRUN
+_MODEL.fields_by_name[
+ "feature_columns"
+].message_type = (
+ google_dot_cloud_dot_bigquery__v2_dot_proto_dot_standard__sql__pb2._STANDARDSQLFIELD
+)
+_MODEL.fields_by_name[
+ "label_columns"
+].message_type = (
+ google_dot_cloud_dot_bigquery__v2_dot_proto_dot_standard__sql__pb2._STANDARDSQLFIELD
+)
+_MODEL_MODELTYPE.containing_type = _MODEL
+_MODEL_LOSSTYPE.containing_type = _MODEL
+_MODEL_DISTANCETYPE.containing_type = _MODEL
+_MODEL_DATASPLITMETHOD.containing_type = _MODEL
+_MODEL_LEARNRATESTRATEGY.containing_type = _MODEL
+_MODEL_OPTIMIZATIONSTRATEGY.containing_type = _MODEL
+_PATCHMODELREQUEST.fields_by_name["model"].message_type = _MODEL
+_LISTMODELSREQUEST.fields_by_name[
+ "max_results"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._UINT32VALUE
+_LISTMODELSRESPONSE.fields_by_name["models"].message_type = _MODEL
+DESCRIPTOR.message_types_by_name["Model"] = _MODEL
+DESCRIPTOR.message_types_by_name["GetModelRequest"] = _GETMODELREQUEST
+DESCRIPTOR.message_types_by_name["PatchModelRequest"] = _PATCHMODELREQUEST
+DESCRIPTOR.message_types_by_name["DeleteModelRequest"] = _DELETEMODELREQUEST
+DESCRIPTOR.message_types_by_name["ListModelsRequest"] = _LISTMODELSREQUEST
+DESCRIPTOR.message_types_by_name["ListModelsResponse"] = _LISTMODELSRESPONSE
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+Model = _reflection.GeneratedProtocolMessageType(
+ "Model",
+ (_message.Message,),
+ dict(
+ KmeansEnums=_reflection.GeneratedProtocolMessageType(
+ "KmeansEnums",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_MODEL_KMEANSENUMS,
+ __module__="google.cloud.bigquery_v2.proto.model_pb2"
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.Model.KmeansEnums)
+ ),
+ ),
+ RegressionMetrics=_reflection.GeneratedProtocolMessageType(
+ "RegressionMetrics",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_MODEL_REGRESSIONMETRICS,
+ __module__="google.cloud.bigquery_v2.proto.model_pb2",
+ __doc__="""Evaluation metrics for regression and explicit feedback type matrix
+ factorization models.
+
+
+ Attributes:
+ mean_absolute_error:
+ Mean absolute error.
+ mean_squared_error:
+ Mean squared error.
+ mean_squared_log_error:
+ Mean squared log error.
+ median_absolute_error:
+ Median absolute error.
+ r_squared:
+ R^2 score.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.Model.RegressionMetrics)
+ ),
+ ),
+ AggregateClassificationMetrics=_reflection.GeneratedProtocolMessageType(
+ "AggregateClassificationMetrics",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_MODEL_AGGREGATECLASSIFICATIONMETRICS,
+ __module__="google.cloud.bigquery_v2.proto.model_pb2",
+ __doc__="""Aggregate metrics for classification/classifier models. For multi-class
+ models, the metrics are either macro-averaged or micro-averaged. When
+ macro-averaged, the metrics are calculated for each label and then an
+ unweighted average is taken of those values. When micro-averaged, the
+ metric is calculated globally by counting the total number of correctly
+ predicted rows.
+
+
+ Attributes:
+ precision:
+ Precision is the fraction of actual positive predictions that
+ had positive actual labels. For multiclass this is a macro-
+ averaged metric treating each class as a binary classifier.
+ recall:
+ Recall is the fraction of actual positive labels that were
+ given a positive prediction. For multiclass this is a macro-
+ averaged metric.
+ accuracy:
+ Accuracy is the fraction of predictions given the correct
+ label. For multiclass this is a micro-averaged metric.
+ threshold:
+ Threshold at which the metrics are computed. For binary
+ classification models this is the positive class threshold.
+ For multi-class classfication models this is the confidence
+ threshold.
+ f1_score:
+ The F1 score is an average of recall and precision. For
+ multiclass this is a macro-averaged metric.
+ log_loss:
+ Logarithmic Loss. For multiclass this is a macro-averaged
+ metric.
+ roc_auc:
+ Area Under a ROC Curve. For multiclass this is a macro-
+ averaged metric.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.Model.AggregateClassificationMetrics)
+ ),
+ ),
+ BinaryClassificationMetrics=_reflection.GeneratedProtocolMessageType(
+ "BinaryClassificationMetrics",
+ (_message.Message,),
+ dict(
+ BinaryConfusionMatrix=_reflection.GeneratedProtocolMessageType(
+ "BinaryConfusionMatrix",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_MODEL_BINARYCLASSIFICATIONMETRICS_BINARYCONFUSIONMATRIX,
+ __module__="google.cloud.bigquery_v2.proto.model_pb2",
+ __doc__="""Confusion matrix for binary classification models.
+
+
+ Attributes:
+ positive_class_threshold:
+ Threshold value used when computing each of the following
+ metric.
+ true_positives:
+ Number of true samples predicted as true.
+ false_positives:
+ Number of false samples predicted as true.
+ true_negatives:
+ Number of true samples predicted as false.
+ false_negatives:
+ Number of false samples predicted as false.
+ precision:
+ The fraction of actual positive predictions that had positive
+ actual labels.
+ recall:
+ The fraction of actual positive labels that were given a
+ positive prediction.
+ f1_score:
+ The equally weighted average of recall and precision.
+ accuracy:
+ The fraction of predictions given the correct label.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.Model.BinaryClassificationMetrics.BinaryConfusionMatrix)
+ ),
+ ),
+ DESCRIPTOR=_MODEL_BINARYCLASSIFICATIONMETRICS,
+ __module__="google.cloud.bigquery_v2.proto.model_pb2",
+ __doc__="""Evaluation metrics for binary classification/classifier models.
+
+
+ Attributes:
+ aggregate_classification_metrics:
+ Aggregate classification metrics.
+ binary_confusion_matrix_list:
+ Binary confusion matrix at multiple thresholds.
+ positive_label:
+ Label representing the positive class.
+ negative_label:
+ Label representing the negative class.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.Model.BinaryClassificationMetrics)
+ ),
+ ),
+ MultiClassClassificationMetrics=_reflection.GeneratedProtocolMessageType(
+ "MultiClassClassificationMetrics",
+ (_message.Message,),
+ dict(
+ ConfusionMatrix=_reflection.GeneratedProtocolMessageType(
+ "ConfusionMatrix",
+ (_message.Message,),
+ dict(
+ Entry=_reflection.GeneratedProtocolMessageType(
+ "Entry",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_MODEL_MULTICLASSCLASSIFICATIONMETRICS_CONFUSIONMATRIX_ENTRY,
+ __module__="google.cloud.bigquery_v2.proto.model_pb2",
+ __doc__="""A single entry in the confusion matrix.
+
+
+ Attributes:
+ predicted_label:
+ The predicted label. For confidence\_threshold > 0, we will
+ also add an entry indicating the number of items under the
+ confidence threshold.
+ item_count:
+ Number of items being predicted as this label.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.Model.MultiClassClassificationMetrics.ConfusionMatrix.Entry)
+ ),
+ ),
+ Row=_reflection.GeneratedProtocolMessageType(
+ "Row",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_MODEL_MULTICLASSCLASSIFICATIONMETRICS_CONFUSIONMATRIX_ROW,
+ __module__="google.cloud.bigquery_v2.proto.model_pb2",
+ __doc__="""A single row in the confusion matrix.
+
+
+ Attributes:
+ actual_label:
+ The original label of this row.
+ entries:
+ Info describing predicted label distribution.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.Model.MultiClassClassificationMetrics.ConfusionMatrix.Row)
+ ),
+ ),
+ DESCRIPTOR=_MODEL_MULTICLASSCLASSIFICATIONMETRICS_CONFUSIONMATRIX,
+ __module__="google.cloud.bigquery_v2.proto.model_pb2",
+ __doc__="""Confusion matrix for multi-class classification models.
+
+
+ Attributes:
+ confidence_threshold:
+ Confidence threshold used when computing the entries of the
+ confusion matrix.
+ rows:
+ One row per actual label.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.Model.MultiClassClassificationMetrics.ConfusionMatrix)
+ ),
+ ),
+ DESCRIPTOR=_MODEL_MULTICLASSCLASSIFICATIONMETRICS,
+ __module__="google.cloud.bigquery_v2.proto.model_pb2",
+ __doc__="""Evaluation metrics for multi-class classification/classifier models.
+
+
+ Attributes:
+ aggregate_classification_metrics:
+ Aggregate classification metrics.
+ confusion_matrix_list:
+ Confusion matrix at different thresholds.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.Model.MultiClassClassificationMetrics)
+ ),
+ ),
+ ClusteringMetrics=_reflection.GeneratedProtocolMessageType(
+ "ClusteringMetrics",
+ (_message.Message,),
+ dict(
+ Cluster=_reflection.GeneratedProtocolMessageType(
+ "Cluster",
+ (_message.Message,),
+ dict(
+ FeatureValue=_reflection.GeneratedProtocolMessageType(
+ "FeatureValue",
+ (_message.Message,),
+ dict(
+ CategoricalValue=_reflection.GeneratedProtocolMessageType(
+ "CategoricalValue",
+ (_message.Message,),
+ dict(
+ CategoryCount=_reflection.GeneratedProtocolMessageType(
+ "CategoryCount",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_MODEL_CLUSTERINGMETRICS_CLUSTER_FEATUREVALUE_CATEGORICALVALUE_CATEGORYCOUNT,
+ __module__="google.cloud.bigquery_v2.proto.model_pb2",
+ __doc__="""Represents the count of a single category within the cluster.
+
+
+ Attributes:
+ category:
+ The name of category.
+ count:
+ The count of training samples matching the category within the
+ cluster.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.Model.ClusteringMetrics.Cluster.FeatureValue.CategoricalValue.CategoryCount)
+ ),
+ ),
+ DESCRIPTOR=_MODEL_CLUSTERINGMETRICS_CLUSTER_FEATUREVALUE_CATEGORICALVALUE,
+ __module__="google.cloud.bigquery_v2.proto.model_pb2",
+ __doc__="""Representative value of a categorical feature.
+
+
+ Attributes:
+ category_counts:
+ Counts of all categories for the categorical feature. If there
+ are more than ten categories, we return top ten (by count) and
+ return one more CategoryCount with category ‘\ *OTHER*\ ’ and
+ count as aggregate counts of remaining categories.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.Model.ClusteringMetrics.Cluster.FeatureValue.CategoricalValue)
+ ),
+ ),
+ DESCRIPTOR=_MODEL_CLUSTERINGMETRICS_CLUSTER_FEATUREVALUE,
+ __module__="google.cloud.bigquery_v2.proto.model_pb2",
+ __doc__="""Representative value of a single feature within the cluster.
+
+
+ Attributes:
+ feature_column:
+ The feature column name.
+ numerical_value:
+ The numerical feature value. This is the centroid value for
+ this feature.
+ categorical_value:
+ The categorical feature value.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.Model.ClusteringMetrics.Cluster.FeatureValue)
+ ),
+ ),
+ DESCRIPTOR=_MODEL_CLUSTERINGMETRICS_CLUSTER,
+ __module__="google.cloud.bigquery_v2.proto.model_pb2",
+ __doc__="""Message containing the information about one cluster.
+
+
+ Attributes:
+ centroid_id:
+ Centroid id.
+ feature_values:
+ Values of highly variant features for this cluster.
+ count:
+ Count of training data rows that were assigned to this
+ cluster.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.Model.ClusteringMetrics.Cluster)
+ ),
+ ),
+ DESCRIPTOR=_MODEL_CLUSTERINGMETRICS,
+ __module__="google.cloud.bigquery_v2.proto.model_pb2",
+ __doc__="""Evaluation metrics for clustering models.
+
+
+ Attributes:
+ davies_bouldin_index:
+ Davies-Bouldin index.
+ mean_squared_distance:
+ Mean of squared distances between each sample to its cluster
+ centroid.
+ clusters:
+ [Beta] Information for all clusters.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.Model.ClusteringMetrics)
+ ),
+ ),
+ EvaluationMetrics=_reflection.GeneratedProtocolMessageType(
+ "EvaluationMetrics",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_MODEL_EVALUATIONMETRICS,
+ __module__="google.cloud.bigquery_v2.proto.model_pb2",
+ __doc__="""Evaluation metrics of a model. These are either computed on all training
+ data or just the eval data based on whether eval data was used during
+ training. These are not present for imported models.
+
+
+ Attributes:
+ regression_metrics:
+ Populated for regression models and explicit feedback type
+ matrix factorization models.
+ binary_classification_metrics:
+ Populated for binary classification/classifier models.
+ multi_class_classification_metrics:
+ Populated for multi-class classification/classifier models.
+ clustering_metrics:
+ Populated for clustering models.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.Model.EvaluationMetrics)
+ ),
+ ),
+ TrainingRun=_reflection.GeneratedProtocolMessageType(
+ "TrainingRun",
+ (_message.Message,),
+ dict(
+ TrainingOptions=_reflection.GeneratedProtocolMessageType(
+ "TrainingOptions",
+ (_message.Message,),
+ dict(
+ LabelClassWeightsEntry=_reflection.GeneratedProtocolMessageType(
+ "LabelClassWeightsEntry",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_MODEL_TRAININGRUN_TRAININGOPTIONS_LABELCLASSWEIGHTSENTRY,
+ __module__="google.cloud.bigquery_v2.proto.model_pb2"
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.Model.TrainingRun.TrainingOptions.LabelClassWeightsEntry)
+ ),
+ ),
+ DESCRIPTOR=_MODEL_TRAININGRUN_TRAININGOPTIONS,
+ __module__="google.cloud.bigquery_v2.proto.model_pb2",
+ __doc__="""Protocol buffer.
+
+ Attributes:
+ max_iterations:
+ The maximum number of iterations in training. Used only for
+ iterative training algorithms.
+ loss_type:
+ Type of loss function used during training run.
+ learn_rate:
+ Learning rate in training. Used only for iterative training
+ algorithms.
+ l1_regularization:
+ L1 regularization coefficient.
+ l2_regularization:
+ L2 regularization coefficient.
+ min_relative_progress:
+ When early\_stop is true, stops training when accuracy
+ improvement is less than 'min\_relative\_progress'. Used only
+ for iterative training algorithms.
+ warm_start:
+ Whether to train a model from the last checkpoint.
+ early_stop:
+ Whether to stop early when the loss doesn't improve
+ significantly any more (compared to min\_relative\_progress).
+ Used only for iterative training algorithms.
+ input_label_columns:
+ Name of input label columns in training data.
+ data_split_method:
+ The data split type for training and evaluation, e.g. RANDOM.
+ data_split_eval_fraction:
+ The fraction of evaluation data over the whole input data. The
+ rest of data will be used as training data. The format should
+ be double. Accurate to two decimal places. Default value is
+ 0.2.
+ data_split_column:
+ The column to split data with. This column won't be used as a
+ feature. 1. When data\_split\_method is CUSTOM, the
+ corresponding column should be boolean. The rows with true
+ value tag are eval data, and the false are training data. 2.
+ When data\_split\_method is SEQ, the first
+ DATA\_SPLIT\_EVAL\_FRACTION rows (from smallest to largest) in
+ the corresponding column are used as training data, and the
+ rest are eval data. It respects the order in Orderable data
+ types:
+ https://cloud.google.com/bigquery/docs/reference/standard-
+ sql/data-types#data-type-properties
+ learn_rate_strategy:
+ The strategy to determine learn rate for the current
+ iteration.
+ initial_learn_rate:
+ Specifies the initial learning rate for the line search learn
+ rate strategy.
+ label_class_weights:
+ Weights associated with each label class, for rebalancing the
+ training data. Only applicable for classification models.
+ distance_type:
+ Distance type for clustering models.
+ num_clusters:
+ Number of clusters for clustering models.
+ model_uri:
+ [Beta] Google Cloud Storage URI from which the model was
+ imported. Only applicable for imported models.
+ optimization_strategy:
+ Optimization strategy for training linear regression models.
+ kmeans_initialization_method:
+ The method used to initialize the centroids for kmeans
+ algorithm.
+ kmeans_initialization_column:
+ The column used to provide the initial centroids for kmeans
+ algorithm when kmeans\_initialization\_method is CUSTOM.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.Model.TrainingRun.TrainingOptions)
+ ),
+ ),
+ IterationResult=_reflection.GeneratedProtocolMessageType(
+ "IterationResult",
+ (_message.Message,),
+ dict(
+ ClusterInfo=_reflection.GeneratedProtocolMessageType(
+ "ClusterInfo",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_MODEL_TRAININGRUN_ITERATIONRESULT_CLUSTERINFO,
+ __module__="google.cloud.bigquery_v2.proto.model_pb2",
+ __doc__="""Information about a single cluster for clustering model.
+
+
+ Attributes:
+ centroid_id:
+ Centroid id.
+ cluster_radius:
+ Cluster radius, the average distance from centroid to each
+ point assigned to the cluster.
+ cluster_size:
+ Cluster size, the total number of points assigned to the
+ cluster.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.Model.TrainingRun.IterationResult.ClusterInfo)
+ ),
+ ),
+ DESCRIPTOR=_MODEL_TRAININGRUN_ITERATIONRESULT,
+ __module__="google.cloud.bigquery_v2.proto.model_pb2",
+ __doc__="""Information about a single iteration of the training run.
+
+
+ Attributes:
+ index:
+ Index of the iteration, 0 based.
+ duration_ms:
+ Time taken to run the iteration in milliseconds.
+ training_loss:
+ Loss computed on the training data at the end of iteration.
+ eval_loss:
+ Loss computed on the eval data at the end of iteration.
+ learn_rate:
+ Learn rate used for this iteration.
+ cluster_infos:
+ Information about top clusters for clustering models.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.Model.TrainingRun.IterationResult)
+ ),
+ ),
+ DESCRIPTOR=_MODEL_TRAININGRUN,
+ __module__="google.cloud.bigquery_v2.proto.model_pb2",
+ __doc__="""Information about a single training query run for the model.
+
+
+ Attributes:
+ training_options:
+ Options that were used for this training run, includes user
+ specified and default options that were used.
+ start_time:
+ The start time of this training run.
+ results:
+ Output of each iteration run, results.size() <=
+ max\_iterations.
+ evaluation_metrics:
+ The evaluation metrics over training/eval data that were
+ computed at the end of training.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.Model.TrainingRun)
+ ),
+ ),
+ LabelsEntry=_reflection.GeneratedProtocolMessageType(
+ "LabelsEntry",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_MODEL_LABELSENTRY,
+ __module__="google.cloud.bigquery_v2.proto.model_pb2"
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.Model.LabelsEntry)
+ ),
+ ),
+ DESCRIPTOR=_MODEL,
+ __module__="google.cloud.bigquery_v2.proto.model_pb2",
+ __doc__="""Protocol buffer.
+
+ Attributes:
+ etag:
+ Output only. A hash of this resource.
+ model_reference:
+ Required. Unique identifier for this model.
+ creation_time:
+ Output only. The time when this model was created, in
+ millisecs since the epoch.
+ last_modified_time:
+ Output only. The time when this model was last modified, in
+ millisecs since the epoch.
+ description:
+ [Optional] A user-friendly description of this model.
+ friendly_name:
+ [Optional] A descriptive name for this model.
+ labels:
+ [Optional] The labels associated with this model. You can use
+ these to organize and group your models. Label keys and values
+ can be no longer than 63 characters, can only contain
+ lowercase letters, numeric characters, underscores and dashes.
+ International characters are allowed. Label values are
+ optional. Label keys must start with a letter and each label
+ in the list must have a different key.
+ expiration_time:
+ [Optional] The time when this model expires, in milliseconds
+ since the epoch. If not present, the model will persist
+ indefinitely. Expired models will be deleted and their storage
+ reclaimed. The defaultTableExpirationMs property of the
+ encapsulating dataset can be used to set a default
+ expirationTime on newly created models.
+ location:
+ Output only. The geographic location where the model resides.
+ This value is inherited from the dataset.
+ model_type:
+ Output only. Type of the model resource.
+ training_runs:
+ Output only. Information for all training runs in increasing
+ order of start\_time.
+ feature_columns:
+ Output only. Input feature columns that were used to train
+ this model.
+ label_columns:
+ Output only. Label columns that were used to train this model.
+ The output of the model will have a "predicted\_" prefix to
+ these columns.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.Model)
+ ),
+)
+_sym_db.RegisterMessage(Model)
+_sym_db.RegisterMessage(Model.KmeansEnums)
+_sym_db.RegisterMessage(Model.RegressionMetrics)
+_sym_db.RegisterMessage(Model.AggregateClassificationMetrics)
+_sym_db.RegisterMessage(Model.BinaryClassificationMetrics)
+_sym_db.RegisterMessage(Model.BinaryClassificationMetrics.BinaryConfusionMatrix)
+_sym_db.RegisterMessage(Model.MultiClassClassificationMetrics)
+_sym_db.RegisterMessage(Model.MultiClassClassificationMetrics.ConfusionMatrix)
+_sym_db.RegisterMessage(Model.MultiClassClassificationMetrics.ConfusionMatrix.Entry)
+_sym_db.RegisterMessage(Model.MultiClassClassificationMetrics.ConfusionMatrix.Row)
+_sym_db.RegisterMessage(Model.ClusteringMetrics)
+_sym_db.RegisterMessage(Model.ClusteringMetrics.Cluster)
+_sym_db.RegisterMessage(Model.ClusteringMetrics.Cluster.FeatureValue)
+_sym_db.RegisterMessage(Model.ClusteringMetrics.Cluster.FeatureValue.CategoricalValue)
+_sym_db.RegisterMessage(
+ Model.ClusteringMetrics.Cluster.FeatureValue.CategoricalValue.CategoryCount
+)
+_sym_db.RegisterMessage(Model.EvaluationMetrics)
+_sym_db.RegisterMessage(Model.TrainingRun)
+_sym_db.RegisterMessage(Model.TrainingRun.TrainingOptions)
+_sym_db.RegisterMessage(Model.TrainingRun.TrainingOptions.LabelClassWeightsEntry)
+_sym_db.RegisterMessage(Model.TrainingRun.IterationResult)
+_sym_db.RegisterMessage(Model.TrainingRun.IterationResult.ClusterInfo)
+_sym_db.RegisterMessage(Model.LabelsEntry)
+
+GetModelRequest = _reflection.GeneratedProtocolMessageType(
+ "GetModelRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_GETMODELREQUEST,
+ __module__="google.cloud.bigquery_v2.proto.model_pb2",
+ __doc__="""Protocol buffer.
+
+ Attributes:
+ project_id:
+ Project ID of the requested model.
+ dataset_id:
+ Dataset ID of the requested model.
+ model_id:
+ Model ID of the requested model.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.GetModelRequest)
+ ),
+)
+_sym_db.RegisterMessage(GetModelRequest)
+
+PatchModelRequest = _reflection.GeneratedProtocolMessageType(
+ "PatchModelRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_PATCHMODELREQUEST,
+ __module__="google.cloud.bigquery_v2.proto.model_pb2",
+ __doc__="""Protocol buffer.
+
+ Attributes:
+ project_id:
+ Project ID of the model to patch.
+ dataset_id:
+ Dataset ID of the model to patch.
+ model_id:
+ Model ID of the model to patch.
+ model:
+ Patched model. Follows RFC5789 patch semantics. Missing fields
+ are not updated. To clear a field, explicitly set to default
+ value.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.PatchModelRequest)
+ ),
+)
+_sym_db.RegisterMessage(PatchModelRequest)
+
+DeleteModelRequest = _reflection.GeneratedProtocolMessageType(
+ "DeleteModelRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_DELETEMODELREQUEST,
+ __module__="google.cloud.bigquery_v2.proto.model_pb2",
+ __doc__="""Protocol buffer.
+
+ Attributes:
+ project_id:
+ Project ID of the model to delete.
+ dataset_id:
+ Dataset ID of the model to delete.
+ model_id:
+ Model ID of the model to delete.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.DeleteModelRequest)
+ ),
+)
+_sym_db.RegisterMessage(DeleteModelRequest)
+
+ListModelsRequest = _reflection.GeneratedProtocolMessageType(
+ "ListModelsRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_LISTMODELSREQUEST,
+ __module__="google.cloud.bigquery_v2.proto.model_pb2",
+ __doc__="""Protocol buffer.
+
+ Attributes:
+ project_id:
+ Project ID of the models to list.
+ dataset_id:
+ Dataset ID of the models to list.
+ max_results:
+ The maximum number of results to return in a single response
+ page. Leverage the page tokens to iterate through the entire
+ collection.
+ page_token:
+ Page token, returned by a previous call to request the next
+ page of results
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.ListModelsRequest)
+ ),
+)
+_sym_db.RegisterMessage(ListModelsRequest)
+
+ListModelsResponse = _reflection.GeneratedProtocolMessageType(
+ "ListModelsResponse",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_LISTMODELSRESPONSE,
+ __module__="google.cloud.bigquery_v2.proto.model_pb2",
+ __doc__="""Protocol buffer.
+
+ Attributes:
+ models:
+ Models in the requested dataset. Only the following fields are
+ populated: model\_reference, model\_type, creation\_time,
+ last\_modified\_time and labels.
+ next_page_token:
+ A token to request the next page of results.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.ListModelsResponse)
+ ),
+)
+_sym_db.RegisterMessage(ListModelsResponse)
+
+
+DESCRIPTOR._options = None
+_MODEL_TRAININGRUN_TRAININGOPTIONS_LABELCLASSWEIGHTSENTRY._options = None
+_MODEL_LABELSENTRY._options = None
+
+_MODELSERVICE = _descriptor.ServiceDescriptor(
+ name="ModelService",
+ full_name="google.cloud.bigquery.v2.ModelService",
+ file=DESCRIPTOR,
+ index=0,
+ serialized_options=_b(
+ "\312A\027bigquery.googleapis.com\322A\302\001https://www.googleapis.com/auth/bigquery,https://www.googleapis.com/auth/bigquery.readonly,https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/cloud-platform.read-only"
+ ),
+ serialized_start=7509,
+ serialized_end=8128,
+ methods=[
+ _descriptor.MethodDescriptor(
+ name="GetModel",
+ full_name="google.cloud.bigquery.v2.ModelService.GetModel",
+ index=0,
+ containing_service=None,
+ input_type=_GETMODELREQUEST,
+ output_type=_MODEL,
+ serialized_options=None,
+ ),
+ _descriptor.MethodDescriptor(
+ name="ListModels",
+ full_name="google.cloud.bigquery.v2.ModelService.ListModels",
+ index=1,
+ containing_service=None,
+ input_type=_LISTMODELSREQUEST,
+ output_type=_LISTMODELSRESPONSE,
+ serialized_options=None,
+ ),
+ _descriptor.MethodDescriptor(
+ name="PatchModel",
+ full_name="google.cloud.bigquery.v2.ModelService.PatchModel",
+ index=2,
+ containing_service=None,
+ input_type=_PATCHMODELREQUEST,
+ output_type=_MODEL,
+ serialized_options=None,
+ ),
+ _descriptor.MethodDescriptor(
+ name="DeleteModel",
+ full_name="google.cloud.bigquery.v2.ModelService.DeleteModel",
+ index=3,
+ containing_service=None,
+ input_type=_DELETEMODELREQUEST,
+ output_type=google_dot_protobuf_dot_empty__pb2._EMPTY,
+ serialized_options=None,
+ ),
+ ],
+)
+_sym_db.RegisterServiceDescriptor(_MODELSERVICE)
+
+DESCRIPTOR.services_by_name["ModelService"] = _MODELSERVICE
+
+# @@protoc_insertion_point(module_scope)
diff --git a/bigquery/google/cloud/bigquery_v2/proto/model_pb2_grpc.py b/bigquery/google/cloud/bigquery_v2/proto/model_pb2_grpc.py
new file mode 100644
index 000000000000..5abcdf0f2606
--- /dev/null
+++ b/bigquery/google/cloud/bigquery_v2/proto/model_pb2_grpc.py
@@ -0,0 +1,102 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
+
+from google.cloud.bigquery_v2.proto import (
+ model_pb2 as google_dot_cloud_dot_bigquery__v2_dot_proto_dot_model__pb2,
+)
+from google.protobuf import empty_pb2 as google_dot_protobuf_dot_empty__pb2
+
+
+class ModelServiceStub(object):
+ # missing associated documentation comment in .proto file
+ pass
+
+ def __init__(self, channel):
+ """Constructor.
+
+ Args:
+ channel: A grpc.Channel.
+ """
+ self.GetModel = channel.unary_unary(
+ "/google.cloud.bigquery.v2.ModelService/GetModel",
+ request_serializer=google_dot_cloud_dot_bigquery__v2_dot_proto_dot_model__pb2.GetModelRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_bigquery__v2_dot_proto_dot_model__pb2.Model.FromString,
+ )
+ self.ListModels = channel.unary_unary(
+ "/google.cloud.bigquery.v2.ModelService/ListModels",
+ request_serializer=google_dot_cloud_dot_bigquery__v2_dot_proto_dot_model__pb2.ListModelsRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_bigquery__v2_dot_proto_dot_model__pb2.ListModelsResponse.FromString,
+ )
+ self.PatchModel = channel.unary_unary(
+ "/google.cloud.bigquery.v2.ModelService/PatchModel",
+ request_serializer=google_dot_cloud_dot_bigquery__v2_dot_proto_dot_model__pb2.PatchModelRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_bigquery__v2_dot_proto_dot_model__pb2.Model.FromString,
+ )
+ self.DeleteModel = channel.unary_unary(
+ "/google.cloud.bigquery.v2.ModelService/DeleteModel",
+ request_serializer=google_dot_cloud_dot_bigquery__v2_dot_proto_dot_model__pb2.DeleteModelRequest.SerializeToString,
+ response_deserializer=google_dot_protobuf_dot_empty__pb2.Empty.FromString,
+ )
+
+
+class ModelServiceServicer(object):
+ # missing associated documentation comment in .proto file
+ pass
+
+ def GetModel(self, request, context):
+ """Gets the specified model resource by model ID.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def ListModels(self, request, context):
+ """Lists all models in the specified dataset. Requires the READER dataset
+ role.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def PatchModel(self, request, context):
+ """Patch specific fields in the specified model.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def DeleteModel(self, request, context):
+ """Deletes the model specified by modelId from the dataset.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+
+def add_ModelServiceServicer_to_server(servicer, server):
+ rpc_method_handlers = {
+ "GetModel": grpc.unary_unary_rpc_method_handler(
+ servicer.GetModel,
+ request_deserializer=google_dot_cloud_dot_bigquery__v2_dot_proto_dot_model__pb2.GetModelRequest.FromString,
+ response_serializer=google_dot_cloud_dot_bigquery__v2_dot_proto_dot_model__pb2.Model.SerializeToString,
+ ),
+ "ListModels": grpc.unary_unary_rpc_method_handler(
+ servicer.ListModels,
+ request_deserializer=google_dot_cloud_dot_bigquery__v2_dot_proto_dot_model__pb2.ListModelsRequest.FromString,
+ response_serializer=google_dot_cloud_dot_bigquery__v2_dot_proto_dot_model__pb2.ListModelsResponse.SerializeToString,
+ ),
+ "PatchModel": grpc.unary_unary_rpc_method_handler(
+ servicer.PatchModel,
+ request_deserializer=google_dot_cloud_dot_bigquery__v2_dot_proto_dot_model__pb2.PatchModelRequest.FromString,
+ response_serializer=google_dot_cloud_dot_bigquery__v2_dot_proto_dot_model__pb2.Model.SerializeToString,
+ ),
+ "DeleteModel": grpc.unary_unary_rpc_method_handler(
+ servicer.DeleteModel,
+ request_deserializer=google_dot_cloud_dot_bigquery__v2_dot_proto_dot_model__pb2.DeleteModelRequest.FromString,
+ response_serializer=google_dot_protobuf_dot_empty__pb2.Empty.SerializeToString,
+ ),
+ }
+ generic_handler = grpc.method_handlers_generic_handler(
+ "google.cloud.bigquery.v2.ModelService", rpc_method_handlers
+ )
+ server.add_generic_rpc_handlers((generic_handler,))
diff --git a/bigquery/google/cloud/bigquery_v2/proto/model_reference.proto b/bigquery/google/cloud/bigquery_v2/proto/model_reference.proto
new file mode 100644
index 000000000000..f436659abd4f
--- /dev/null
+++ b/bigquery/google/cloud/bigquery_v2/proto/model_reference.proto
@@ -0,0 +1,38 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.bigquery.v2;
+
+import "google/api/annotations.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/bigquery/v2;bigquery";
+option java_outer_classname = "ModelReferenceProto";
+option java_package = "com.google.cloud.bigquery.v2";
+
+// Id path of a model.
+message ModelReference {
+ // [Required] The ID of the project containing this model.
+ string project_id = 1;
+
+ // [Required] The ID of the dataset containing this model.
+ string dataset_id = 2;
+
+ // [Required] The ID of the model. The ID must contain only
+ // letters (a-z, A-Z), numbers (0-9), or underscores (_). The maximum
+ // length is 1,024 characters.
+ string model_id = 3;
+}
diff --git a/bigquery/google/cloud/bigquery_v2/proto/model_reference_pb2.py b/bigquery/google/cloud/bigquery_v2/proto/model_reference_pb2.py
new file mode 100644
index 000000000000..94f6116b5a44
--- /dev/null
+++ b/bigquery/google/cloud/bigquery_v2/proto/model_reference_pb2.py
@@ -0,0 +1,138 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/bigquery_v2/proto/model_reference.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/bigquery_v2/proto/model_reference.proto",
+ package="google.cloud.bigquery.v2",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\034com.google.cloud.bigquery.v2B\023ModelReferenceProtoZ@google.golang.org/genproto/googleapis/cloud/bigquery/v2;bigquery"
+ ),
+ serialized_pb=_b(
+ '\n4google/cloud/bigquery_v2/proto/model_reference.proto\x12\x18google.cloud.bigquery.v2\x1a\x1cgoogle/api/annotations.proto"J\n\x0eModelReference\x12\x12\n\nproject_id\x18\x01 \x01(\t\x12\x12\n\ndataset_id\x18\x02 \x01(\t\x12\x10\n\x08model_id\x18\x03 \x01(\tBu\n\x1c\x63om.google.cloud.bigquery.v2B\x13ModelReferenceProtoZ@google.golang.org/genproto/googleapis/cloud/bigquery/v2;bigqueryb\x06proto3'
+ ),
+ dependencies=[google_dot_api_dot_annotations__pb2.DESCRIPTOR],
+)
+
+
+_MODELREFERENCE = _descriptor.Descriptor(
+ name="ModelReference",
+ full_name="google.cloud.bigquery.v2.ModelReference",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="project_id",
+ full_name="google.cloud.bigquery.v2.ModelReference.project_id",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="dataset_id",
+ full_name="google.cloud.bigquery.v2.ModelReference.dataset_id",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="model_id",
+ full_name="google.cloud.bigquery.v2.ModelReference.model_id",
+ index=2,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=112,
+ serialized_end=186,
+)
+
+DESCRIPTOR.message_types_by_name["ModelReference"] = _MODELREFERENCE
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+ModelReference = _reflection.GeneratedProtocolMessageType(
+ "ModelReference",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_MODELREFERENCE,
+ __module__="google.cloud.bigquery_v2.proto.model_reference_pb2",
+ __doc__="""Id path of a model.
+
+
+ Attributes:
+ project_id:
+ [Required] The ID of the project containing this model.
+ dataset_id:
+ [Required] The ID of the dataset containing this model.
+ model_id:
+ [Required] The ID of the model. The ID must contain only
+ letters (a-z, A-Z), numbers (0-9), or underscores (\_). The
+ maximum length is 1,024 characters.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.ModelReference)
+ ),
+)
+_sym_db.RegisterMessage(ModelReference)
+
+
+DESCRIPTOR._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/bigquery/google/cloud/bigquery_v2/proto/model_reference_pb2_grpc.py b/bigquery/google/cloud/bigquery_v2/proto/model_reference_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/bigquery/google/cloud/bigquery_v2/proto/model_reference_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/bigquery/google/cloud/bigquery_v2/proto/standard_sql.proto b/bigquery/google/cloud/bigquery_v2/proto/standard_sql.proto
new file mode 100644
index 000000000000..98173092ff71
--- /dev/null
+++ b/bigquery/google/cloud/bigquery_v2/proto/standard_sql.proto
@@ -0,0 +1,109 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.bigquery.v2;
+
+import "google/api/annotations.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/bigquery/v2;bigquery";
+option java_outer_classname = "StandardSqlProto";
+option java_package = "com.google.cloud.bigquery.v2";
+
+// The type of a variable, e.g., a function argument.
+// Examples:
+// INT64: {type_kind="INT64"}
+// ARRAY: {type_kind="ARRAY", array_element_type="STRING"}
+// STRUCT>:
+// {type_kind="STRUCT",
+// struct_type={fields=[
+// {name="x", type={type_kind="STRING"}},
+// {name="y", type={type_kind="ARRAY", array_element_type="DATE"}}
+// ]}}
+message StandardSqlDataType {
+ enum TypeKind {
+ // Invalid type.
+ TYPE_KIND_UNSPECIFIED = 0;
+
+ // Encoded as a string in decimal format.
+ INT64 = 2;
+
+ // Encoded as a boolean "false" or "true".
+ BOOL = 5;
+
+ // Encoded as a number, or string "NaN", "Infinity" or "-Infinity".
+ FLOAT64 = 7;
+
+ // Encoded as a string value.
+ STRING = 8;
+
+ // Encoded as a base64 string per RFC 4648, section 4.
+ BYTES = 9;
+
+ // Encoded as an RFC 3339 timestamp with mandatory "Z" time zone string:
+ // 1985-04-12T23:20:50.52Z
+ TIMESTAMP = 19;
+
+ // Encoded as RFC 3339 full-date format string: 1985-04-12
+ DATE = 10;
+
+ // Encoded as RFC 3339 partial-time format string: 23:20:50.52
+ TIME = 20;
+
+ // Encoded as RFC 3339 full-date "T" partial-time: 1985-04-12T23:20:50.52
+ DATETIME = 21;
+
+ // Encoded as WKT
+ GEOGRAPHY = 22;
+
+ // Encoded as a decimal string.
+ NUMERIC = 23;
+
+ // Encoded as a list with types matching Type.array_type.
+ ARRAY = 16;
+
+ // Encoded as a list with fields of type Type.struct_type[i]. List is used
+ // because a JSON object cannot have duplicate field names.
+ STRUCT = 17;
+ }
+
+ // Required. The top level type of this field.
+ // Can be any standard SQL data type (e.g., "INT64", "DATE", "ARRAY").
+ TypeKind type_kind = 1;
+
+ oneof sub_type {
+ // The type of the array's elements, if type_kind = "ARRAY".
+ StandardSqlDataType array_element_type = 2;
+
+ // The fields of this struct, in order, if type_kind = "STRUCT".
+ StandardSqlStructType struct_type = 3;
+ }
+}
+
+// A field or a column.
+message StandardSqlField {
+ // Optional. The name of this field. Can be absent for struct fields.
+ string name = 1;
+
+ // Optional. The type of this parameter. Absent if not explicitly
+ // specified (e.g., CREATE FUNCTION statement can omit the return type;
+ // in this case the output parameter does not have this "type" field).
+ StandardSqlDataType type = 2;
+}
+
+message StandardSqlStructType {
+ repeated StandardSqlField fields = 1;
+}
diff --git a/bigquery/google/cloud/bigquery_v2/proto/standard_sql_pb2.py b/bigquery/google/cloud/bigquery_v2/proto/standard_sql_pb2.py
new file mode 100644
index 000000000000..91f1554f7f80
--- /dev/null
+++ b/bigquery/google/cloud/bigquery_v2/proto/standard_sql_pb2.py
@@ -0,0 +1,366 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/bigquery_v2/proto/standard_sql.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/bigquery_v2/proto/standard_sql.proto",
+ package="google.cloud.bigquery.v2",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n\034com.google.cloud.bigquery.v2B\020StandardSqlProtoZ@google.golang.org/genproto/googleapis/cloud/bigquery/v2;bigquery"
+ ),
+ serialized_pb=_b(
+ '\n1google/cloud/bigquery_v2/proto/standard_sql.proto\x12\x18google.cloud.bigquery.v2\x1a\x1cgoogle/api/annotations.proto"\xc6\x03\n\x13StandardSqlDataType\x12I\n\ttype_kind\x18\x01 \x01(\x0e\x32\x36.google.cloud.bigquery.v2.StandardSqlDataType.TypeKind\x12K\n\x12\x61rray_element_type\x18\x02 \x01(\x0b\x32-.google.cloud.bigquery.v2.StandardSqlDataTypeH\x00\x12\x46\n\x0bstruct_type\x18\x03 \x01(\x0b\x32/.google.cloud.bigquery.v2.StandardSqlStructTypeH\x00"\xc2\x01\n\x08TypeKind\x12\x19\n\x15TYPE_KIND_UNSPECIFIED\x10\x00\x12\t\n\x05INT64\x10\x02\x12\x08\n\x04\x42OOL\x10\x05\x12\x0b\n\x07\x46LOAT64\x10\x07\x12\n\n\x06STRING\x10\x08\x12\t\n\x05\x42YTES\x10\t\x12\r\n\tTIMESTAMP\x10\x13\x12\x08\n\x04\x44\x41TE\x10\n\x12\x08\n\x04TIME\x10\x14\x12\x0c\n\x08\x44\x41TETIME\x10\x15\x12\r\n\tGEOGRAPHY\x10\x16\x12\x0b\n\x07NUMERIC\x10\x17\x12\t\n\x05\x41RRAY\x10\x10\x12\n\n\x06STRUCT\x10\x11\x42\n\n\x08sub_type"]\n\x10StandardSqlField\x12\x0c\n\x04name\x18\x01 \x01(\t\x12;\n\x04type\x18\x02 \x01(\x0b\x32-.google.cloud.bigquery.v2.StandardSqlDataType"S\n\x15StandardSqlStructType\x12:\n\x06\x66ields\x18\x01 \x03(\x0b\x32*.google.cloud.bigquery.v2.StandardSqlFieldBr\n\x1c\x63om.google.cloud.bigquery.v2B\x10StandardSqlProtoZ@google.golang.org/genproto/googleapis/cloud/bigquery/v2;bigqueryb\x06proto3'
+ ),
+ dependencies=[google_dot_api_dot_annotations__pb2.DESCRIPTOR],
+)
+
+
+_STANDARDSQLDATATYPE_TYPEKIND = _descriptor.EnumDescriptor(
+ name="TypeKind",
+ full_name="google.cloud.bigquery.v2.StandardSqlDataType.TypeKind",
+ filename=None,
+ file=DESCRIPTOR,
+ values=[
+ _descriptor.EnumValueDescriptor(
+ name="TYPE_KIND_UNSPECIFIED",
+ index=0,
+ number=0,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="INT64", index=1, number=2, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="BOOL", index=2, number=5, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="FLOAT64", index=3, number=7, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="STRING", index=4, number=8, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="BYTES", index=5, number=9, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="TIMESTAMP", index=6, number=19, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="DATE", index=7, number=10, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="TIME", index=8, number=20, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="DATETIME", index=9, number=21, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="GEOGRAPHY", index=10, number=22, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="NUMERIC", index=11, number=23, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="ARRAY", index=12, number=16, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="STRUCT", index=13, number=17, serialized_options=None, type=None
+ ),
+ ],
+ containing_type=None,
+ serialized_options=None,
+ serialized_start=358,
+ serialized_end=552,
+)
+_sym_db.RegisterEnumDescriptor(_STANDARDSQLDATATYPE_TYPEKIND)
+
+
+_STANDARDSQLDATATYPE = _descriptor.Descriptor(
+ name="StandardSqlDataType",
+ full_name="google.cloud.bigquery.v2.StandardSqlDataType",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="type_kind",
+ full_name="google.cloud.bigquery.v2.StandardSqlDataType.type_kind",
+ index=0,
+ number=1,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="array_element_type",
+ full_name="google.cloud.bigquery.v2.StandardSqlDataType.array_element_type",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="struct_type",
+ full_name="google.cloud.bigquery.v2.StandardSqlDataType.struct_type",
+ index=2,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[_STANDARDSQLDATATYPE_TYPEKIND],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="sub_type",
+ full_name="google.cloud.bigquery.v2.StandardSqlDataType.sub_type",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=110,
+ serialized_end=564,
+)
+
+
+_STANDARDSQLFIELD = _descriptor.Descriptor(
+ name="StandardSqlField",
+ full_name="google.cloud.bigquery.v2.StandardSqlField",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.bigquery.v2.StandardSqlField.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="type",
+ full_name="google.cloud.bigquery.v2.StandardSqlField.type",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=566,
+ serialized_end=659,
+)
+
+
+_STANDARDSQLSTRUCTTYPE = _descriptor.Descriptor(
+ name="StandardSqlStructType",
+ full_name="google.cloud.bigquery.v2.StandardSqlStructType",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="fields",
+ full_name="google.cloud.bigquery.v2.StandardSqlStructType.fields",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=661,
+ serialized_end=744,
+)
+
+_STANDARDSQLDATATYPE.fields_by_name[
+ "type_kind"
+].enum_type = _STANDARDSQLDATATYPE_TYPEKIND
+_STANDARDSQLDATATYPE.fields_by_name[
+ "array_element_type"
+].message_type = _STANDARDSQLDATATYPE
+_STANDARDSQLDATATYPE.fields_by_name["struct_type"].message_type = _STANDARDSQLSTRUCTTYPE
+_STANDARDSQLDATATYPE_TYPEKIND.containing_type = _STANDARDSQLDATATYPE
+_STANDARDSQLDATATYPE.oneofs_by_name["sub_type"].fields.append(
+ _STANDARDSQLDATATYPE.fields_by_name["array_element_type"]
+)
+_STANDARDSQLDATATYPE.fields_by_name[
+ "array_element_type"
+].containing_oneof = _STANDARDSQLDATATYPE.oneofs_by_name["sub_type"]
+_STANDARDSQLDATATYPE.oneofs_by_name["sub_type"].fields.append(
+ _STANDARDSQLDATATYPE.fields_by_name["struct_type"]
+)
+_STANDARDSQLDATATYPE.fields_by_name[
+ "struct_type"
+].containing_oneof = _STANDARDSQLDATATYPE.oneofs_by_name["sub_type"]
+_STANDARDSQLFIELD.fields_by_name["type"].message_type = _STANDARDSQLDATATYPE
+_STANDARDSQLSTRUCTTYPE.fields_by_name["fields"].message_type = _STANDARDSQLFIELD
+DESCRIPTOR.message_types_by_name["StandardSqlDataType"] = _STANDARDSQLDATATYPE
+DESCRIPTOR.message_types_by_name["StandardSqlField"] = _STANDARDSQLFIELD
+DESCRIPTOR.message_types_by_name["StandardSqlStructType"] = _STANDARDSQLSTRUCTTYPE
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+StandardSqlDataType = _reflection.GeneratedProtocolMessageType(
+ "StandardSqlDataType",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_STANDARDSQLDATATYPE,
+ __module__="google.cloud.bigquery_v2.proto.standard_sql_pb2",
+ __doc__="""The type of a variable, e.g., a function argument. Examples: INT64:
+ {type\_kind="INT64"} ARRAY: {type\_kind="ARRAY",
+ array\_element\_type="STRING"} STRUCT:
+ {type\_kind="STRUCT", struct\_type={fields=[ {name="x",
+ type={type\_kind="STRING"}}, {name="y", type={type\_kind="ARRAY",
+ array\_element\_type="DATE"}} ]}}
+
+
+ Attributes:
+ type_kind:
+ Required. The top level type of this field. Can be any
+ standard SQL data type (e.g., "INT64", "DATE", "ARRAY").
+ array_element_type:
+ The type of the array's elements, if type\_kind = "ARRAY".
+ struct_type:
+ The fields of this struct, in order, if type\_kind = "STRUCT".
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.StandardSqlDataType)
+ ),
+)
+_sym_db.RegisterMessage(StandardSqlDataType)
+
+StandardSqlField = _reflection.GeneratedProtocolMessageType(
+ "StandardSqlField",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_STANDARDSQLFIELD,
+ __module__="google.cloud.bigquery_v2.proto.standard_sql_pb2",
+ __doc__="""A field or a column.
+
+
+ Attributes:
+ name:
+ Optional. The name of this field. Can be absent for struct
+ fields.
+ type:
+ Optional. The type of this parameter. Absent if not explicitly
+ specified (e.g., CREATE FUNCTION statement can omit the return
+ type; in this case the output parameter does not have this
+ "type" field).
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.StandardSqlField)
+ ),
+)
+_sym_db.RegisterMessage(StandardSqlField)
+
+StandardSqlStructType = _reflection.GeneratedProtocolMessageType(
+ "StandardSqlStructType",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_STANDARDSQLSTRUCTTYPE,
+ __module__="google.cloud.bigquery_v2.proto.standard_sql_pb2"
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.v2.StandardSqlStructType)
+ ),
+)
+_sym_db.RegisterMessage(StandardSqlStructType)
+
+
+DESCRIPTOR._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/bigquery/google/cloud/bigquery_v2/proto/standard_sql_pb2_grpc.py b/bigquery/google/cloud/bigquery_v2/proto/standard_sql_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/bigquery/google/cloud/bigquery_v2/proto/standard_sql_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/bigquery/google/cloud/bigquery_v2/types.py b/bigquery/google/cloud/bigquery_v2/types.py
new file mode 100644
index 000000000000..36fc2d6bfbd2
--- /dev/null
+++ b/bigquery/google/cloud/bigquery_v2/types.py
@@ -0,0 +1,48 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from __future__ import absolute_import
+import sys
+
+from google.api_core.protobuf_helpers import get_messages
+
+from google.cloud.bigquery_v2.proto import model_pb2
+from google.cloud.bigquery_v2.proto import model_reference_pb2
+from google.cloud.bigquery_v2.proto import standard_sql_pb2
+from google.protobuf import empty_pb2
+from google.protobuf import timestamp_pb2
+from google.protobuf import wrappers_pb2
+
+
+_shared_modules = [empty_pb2, timestamp_pb2, wrappers_pb2]
+
+_local_modules = [model_pb2, model_reference_pb2, standard_sql_pb2]
+
+names = []
+
+for module in _shared_modules: # pragma: NO COVER
+ for name, message in get_messages(module).items():
+ setattr(sys.modules[__name__], name, message)
+ names.append(name)
+for module in _local_modules:
+ for name, message in get_messages(module).items():
+ message.__module__ = "google.cloud.bigquery_v2.types"
+ setattr(sys.modules[__name__], name, message)
+ names.append(name)
+
+
+__all__ = tuple(sorted(names))
diff --git a/bigquery/noxfile.py b/bigquery/noxfile.py
new file mode 100644
index 000000000000..37611a5ce296
--- /dev/null
+++ b/bigquery/noxfile.py
@@ -0,0 +1,196 @@
+# Copyright 2016 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import absolute_import
+
+import os
+import shutil
+
+import nox
+
+
+LOCAL_DEPS = (
+ os.path.join("..", "api_core[grpc]"),
+ os.path.join("..", "core"),
+ os.path.join("..", "test_utils"),
+)
+
+BLACK_PATHS = ("docs", "google", "samples", "tests", "noxfile.py", "setup.py")
+
+
+def default(session):
+ """Default unit test session.
+
+ This is intended to be run **without** an interpreter set, so
+ that the current ``python`` (on the ``PATH``) or the version of
+ Python corresponding to the ``nox`` binary the ``PATH`` can
+ run the tests.
+ """
+ # Install all test dependencies, then install local packages in-place.
+ session.install("mock", "pytest", "pytest-cov")
+ for local_dep in LOCAL_DEPS:
+ session.install("-e", local_dep)
+
+ dev_install = ".[all]"
+ session.install("-e", dev_install)
+
+ # IPython does not support Python 2 after version 5.x
+ if session.python == "2.7":
+ session.install("ipython==5.5")
+ else:
+ session.install("ipython")
+
+ # Run py.test against the unit tests.
+ session.run(
+ "py.test",
+ "--quiet",
+ "--cov=google.cloud.bigquery",
+ "--cov=tests.unit",
+ "--cov-append",
+ "--cov-config=.coveragerc",
+ "--cov-report=",
+ "--cov-fail-under=97",
+ os.path.join("tests", "unit"),
+ *session.posargs
+ )
+
+
+@nox.session(python=["2.7", "3.5", "3.6", "3.7"])
+def unit(session):
+ """Run the unit test suite."""
+ default(session)
+
+
+@nox.session(python=["2.7", "3.6"])
+def system(session):
+ """Run the system test suite."""
+
+ # Sanity check: Only run system tests if the environment variable is set.
+ if not os.environ.get("GOOGLE_APPLICATION_CREDENTIALS", ""):
+ session.skip("Credentials must be set via environment variable.")
+
+ # Use pre-release gRPC for system tests.
+ session.install("--pre", "grpcio")
+
+ # Install all test dependencies, then install local packages in place.
+ session.install("mock", "pytest")
+ for local_dep in LOCAL_DEPS:
+ session.install("-e", local_dep)
+ session.install("-e", os.path.join("..", "storage"))
+ session.install("-e", os.path.join("..", "test_utils"))
+ session.install("-e", ".[all]")
+
+ # IPython does not support Python 2 after version 5.x
+ if session.python == "2.7":
+ session.install("ipython==5.5")
+ else:
+ session.install("ipython")
+
+ # Run py.test against the system tests.
+ session.run(
+ "py.test", "--quiet", os.path.join("tests", "system.py"), *session.posargs
+ )
+
+
+@nox.session(python=["2.7", "3.6"])
+def snippets(session):
+ """Run the snippets test suite."""
+
+ # Sanity check: Only run snippets tests if the environment variable is set.
+ if not os.environ.get("GOOGLE_APPLICATION_CREDENTIALS", ""):
+ session.skip("Credentials must be set via environment variable.")
+
+ # Install all test dependencies, then install local packages in place.
+ session.install("mock", "pytest")
+ for local_dep in LOCAL_DEPS:
+ session.install("-e", local_dep)
+ session.install("-e", os.path.join("..", "storage"))
+ session.install("-e", os.path.join("..", "test_utils"))
+ session.install("-e", ".[all]")
+
+ # Run py.test against the snippets tests.
+ session.run("py.test", os.path.join("docs", "snippets.py"), *session.posargs)
+ session.run("py.test", "samples", *session.posargs)
+
+
+@nox.session(python="3.6")
+def cover(session):
+ """Run the final coverage report.
+
+ This outputs the coverage report aggregating coverage from the unit
+ test runs (not system test runs), and then erases coverage data.
+ """
+ session.install("coverage", "pytest-cov")
+ session.run("coverage", "report", "--show-missing", "--fail-under=100")
+ session.run("coverage", "erase")
+
+
+@nox.session(python="3.6")
+def lint(session):
+ """Run linters.
+
+ Returns a failure if the linters find linting errors or sufficiently
+ serious code quality issues.
+ """
+
+ session.install("black", "flake8")
+ for local_dep in LOCAL_DEPS:
+ session.install("-e", local_dep)
+ session.install("-e", ".")
+ session.run("flake8", os.path.join("google", "cloud", "bigquery"))
+ session.run("flake8", "tests")
+ session.run("flake8", os.path.join("docs", "snippets.py"))
+ session.run("black", "--check", *BLACK_PATHS)
+
+
+@nox.session(python="3.6")
+def lint_setup_py(session):
+ """Verify that setup.py is valid (including RST check)."""
+
+ session.install("docutils", "Pygments")
+ session.run("python", "setup.py", "check", "--restructuredtext", "--strict")
+
+
+@nox.session(python="3.6")
+def blacken(session):
+ """Run black.
+ Format code to uniform standard.
+ """
+ session.install("black")
+ session.run("black", *BLACK_PATHS)
+
+
+@nox.session(python="3.6")
+def docs(session):
+ """Build the docs."""
+
+ session.install("ipython", "recommonmark", "sphinx", "sphinx_rtd_theme")
+ for local_dep in LOCAL_DEPS:
+ session.install("-e", local_dep)
+ session.install("-e", os.path.join("..", "storage"))
+ session.install("-e", ".[all]")
+
+ shutil.rmtree(os.path.join("docs", "_build"), ignore_errors=True)
+ session.run(
+ "sphinx-build",
+ "-W", # warnings as errors
+ "-T", # show full traceback on exception
+ "-N", # no colors
+ "-b",
+ "html",
+ "-d",
+ os.path.join("docs", "_build", "doctrees", ""),
+ os.path.join("docs", ""),
+ os.path.join("docs", "_build", "html", ""),
+ )
diff --git a/bigquery/pylint.config.py b/bigquery/pylint.config.py
new file mode 100644
index 000000000000..5d64b9d2f256
--- /dev/null
+++ b/bigquery/pylint.config.py
@@ -0,0 +1,25 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""This module is used to configure gcp-devrel-py-tools run-pylint."""
+
+# Library configuration
+
+# library_additions = {}
+# library_replacements = {}
+
+# Test configuration
+
+# test_additions = copy.deepcopy(library_additions)
+# test_replacements = copy.deepcopy(library_replacements)
diff --git a/bigquery/samples/__init__.py b/bigquery/samples/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/bigquery/samples/add_empty_column.py b/bigquery/samples/add_empty_column.py
new file mode 100644
index 000000000000..eb84037598d3
--- /dev/null
+++ b/bigquery/samples/add_empty_column.py
@@ -0,0 +1,40 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def add_empty_column(client, table_id):
+
+ # [START bigquery_add_empty_column]
+ from google.cloud import bigquery
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ # TODO(developer): Set table_id to the ID of the table to add an empty column.
+ # table_id = "your-project.your_dataset.your_table_name"
+
+ table = client.get_table(table_id)
+
+ original_schema = table.schema
+ new_schema = original_schema[:] # creates a copy of the schema
+ new_schema.append(bigquery.SchemaField("phone", "STRING"))
+
+ table.schema = new_schema
+ table = client.update_table(table, ["schema"]) # API request
+
+ if len(table.schema) == len(original_schema) + 1 == len(new_schema):
+ print("A new column has been added.")
+ else:
+ print("The column has not been added.")
+ # [END bigquery_add_empty_column]
diff --git a/bigquery/samples/browse_table_data.py b/bigquery/samples/browse_table_data.py
new file mode 100644
index 000000000000..dd6c572cab6d
--- /dev/null
+++ b/bigquery/samples/browse_table_data.py
@@ -0,0 +1,47 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def browse_table_data(client, table_id):
+
+ # [START bigquery_browse_table]
+ # TODO(developer): Import the client library.
+ # from google.cloud import bigquery
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ # TODO(developer): Set table_id to the ID of the table to browse data rows.
+ # table_id = "your-project.your_dataset.your_table_name"
+
+ # Download all rows from a table.
+ rows_iter = client.list_rows(table_id)
+
+ # Iterate over rows to make the API requests to fetch row data.
+ rows = list(rows_iter)
+ print("Downloaded {} rows from table {}".format(len(rows), table_id))
+
+ # Download at most 10 rows.
+ rows_iter = client.list_rows(table_id, max_results=10)
+ rows = list(rows_iter)
+ print("Downloaded {} rows from table {}".format(len(rows), table_id))
+
+ # Specify selected fields to limit the results to certain columns.
+ table = client.get_table(table_id)
+ fields = table.schema[:2] # first two columns
+ rows_iter = client.list_rows(table_id, selected_fields=fields, max_results=10)
+ rows = list(rows_iter)
+ print("Selected {} columns from table {}.".format(len(rows_iter.schema), table_id))
+ print("Downloaded {} rows from table {}".format(len(rows), table_id))
+ # [END bigquery_browse_table]
diff --git a/bigquery/samples/create_dataset.py b/bigquery/samples/create_dataset.py
new file mode 100644
index 000000000000..89ca9d38f5f3
--- /dev/null
+++ b/bigquery/samples/create_dataset.py
@@ -0,0 +1,38 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def create_dataset(client, dataset_id):
+
+ # [START bigquery_create_dataset]
+ from google.cloud import bigquery
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ # TODO(developer): Set dataset_id to the ID of the dataset to create.
+ # dataset_id = "{}.your_dataset".format(client.project)
+
+ # Construct a full Dataset object to send to the API.
+ dataset = bigquery.Dataset(dataset_id)
+
+ # TODO(developer): Specify the geographic location where the dataset should reside.
+ dataset.location = "US"
+
+ # Send the dataset to the API for creation.
+ # Raises google.api_core.exceptions.Conflict if the Dataset already
+ # exists within the project.
+ dataset = client.create_dataset(dataset) # API request
+ print("Created dataset {}.{}".format(client.project, dataset.dataset_id))
+ # [END bigquery_create_dataset]
diff --git a/bigquery/samples/create_job.py b/bigquery/samples/create_job.py
new file mode 100644
index 000000000000..24bb85510598
--- /dev/null
+++ b/bigquery/samples/create_job.py
@@ -0,0 +1,40 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def create_job(client):
+
+ # [START bigquery_create_job]
+ from google.cloud import bigquery
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ query_job = client.query(
+ "SELECT country_name from `bigquery-public-data.utility_us.country_code_iso`",
+ # Explicitly force job execution to be routed to a specific processing
+ # location.
+ location="US",
+ # Specify a job configuration to set optional job resource properties.
+ job_config=bigquery.QueryJobConfig(
+ labels={"example-label": "example-value"}, maximum_bytes_billed=1000000
+ ),
+ # The client libraries automatically generate a job ID. Override the
+ # generated ID with either the job_id_prefix or job_id parameters.
+ job_id_prefix="code_sample_",
+ ) # API request
+
+ print("Started job: {}".format(query_job.job_id))
+ # [END bigquery_create_job]
+ return query_job
diff --git a/bigquery/samples/create_routine.py b/bigquery/samples/create_routine.py
new file mode 100644
index 000000000000..c08ec4799a3e
--- /dev/null
+++ b/bigquery/samples/create_routine.py
@@ -0,0 +1,47 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def create_routine(client, routine_id):
+
+ # [START bigquery_create_routine]
+ from google.cloud import bigquery
+ from google.cloud import bigquery_v2
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ # TODO(developer): Choose a fully-qualified ID for the routine.
+ # routine_id = "my-project.my_dataset.my_routine"
+
+ routine = bigquery.Routine(
+ routine_id,
+ type_="SCALAR_FUNCTION",
+ language="SQL",
+ body="x * 3",
+ arguments=[
+ bigquery.RoutineArgument(
+ name="x",
+ data_type=bigquery_v2.types.StandardSqlDataType(
+ type_kind=bigquery_v2.enums.StandardSqlDataType.TypeKind.INT64
+ ),
+ )
+ ],
+ )
+
+ routine = client.create_routine(routine)
+
+ print("Created routine {}".format(routine.reference))
+ # [END bigquery_create_routine]
+ return routine
diff --git a/bigquery/samples/create_routine_ddl.py b/bigquery/samples/create_routine_ddl.py
new file mode 100644
index 000000000000..a4ae3318e7b4
--- /dev/null
+++ b/bigquery/samples/create_routine_ddl.py
@@ -0,0 +1,45 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def create_routine_ddl(client, routine_id):
+
+ # [START bigquery_create_routine_ddl]
+ # TODO(developer): Import the client library.
+ # from google.cloud import bigquery
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ # TODO(developer): Choose a fully-qualified ID for the routine.
+ # routine_id = "my-project.my_dataset.my_routine"
+
+ sql = """
+ CREATE FUNCTION `{}`(
+ arr ARRAY>
+ ) AS (
+ (SELECT SUM(IF(elem.name = "foo",elem.val,null)) FROM UNNEST(arr) AS elem)
+ )
+ """.format(
+ routine_id
+ )
+
+ # Initiate the query to create the routine.
+ query_job = client.query(sql)
+
+ # Wait for the query to complete.
+ query_job.result()
+
+ print("Created routine {}".format(query_job.ddl_target_routine))
+ # [END bigquery_create_routine_ddl]
diff --git a/bigquery/samples/create_table.py b/bigquery/samples/create_table.py
new file mode 100644
index 000000000000..2a6e98fc72f6
--- /dev/null
+++ b/bigquery/samples/create_table.py
@@ -0,0 +1,37 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def create_table(client, table_id):
+
+ # [START bigquery_create_table]
+ from google.cloud import bigquery
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ # TODO(developer): Set table_id to the ID of the table to create
+ # table_id = "your-project.your_dataset.your_table_name"
+
+ schema = [
+ bigquery.SchemaField("full_name", "STRING", mode="REQUIRED"),
+ bigquery.SchemaField("age", "INTEGER", mode="REQUIRED"),
+ ]
+
+ table = bigquery.Table(table_id, schema=schema)
+ table = client.create_table(table) # API request
+ print(
+ "Created table {}.{}.{}".format(table.project, table.dataset_id, table.table_id)
+ )
+ # [END bigquery_create_table]
diff --git a/bigquery/samples/dataset_exists.py b/bigquery/samples/dataset_exists.py
new file mode 100644
index 000000000000..46cf26a623bf
--- /dev/null
+++ b/bigquery/samples/dataset_exists.py
@@ -0,0 +1,29 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def dataset_exists(client, dataset_id):
+
+ # [START bigquery_dataset_exists]
+ from google.cloud.exceptions import NotFound
+
+ # TODO(developer): Set dataset_id to the ID of the dataset to determine existence.
+ # dataset_id = "your-project.your_dataset"
+
+ try:
+ client.get_dataset(dataset_id)
+ print("Dataset {} already exists".format(dataset_id))
+ except NotFound:
+ print("Dataset {} is not found".format(dataset_id))
+ # [END bigquery_dataset_exists]
diff --git a/bigquery/samples/delete_dataset.py b/bigquery/samples/delete_dataset.py
new file mode 100644
index 000000000000..6cde1b6b2d27
--- /dev/null
+++ b/bigquery/samples/delete_dataset.py
@@ -0,0 +1,33 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def delete_dataset(client, dataset_id):
+
+ # [START bigquery_delete_dataset]
+ # TODO(developer): Import the client library.
+ # from google.cloud import bigquery
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ # TODO(developer): Set model_id to the ID of the model to fetch.
+ # dataset_id = 'your-project.your_dataset'
+
+ # Use the delete_contents parameter to delete a dataset and its contents
+ # Use the not_found_ok parameter to not receive an error if the dataset has already been deleted.
+ client.delete_dataset(dataset_id, delete_contents=True, not_found_ok=True)
+
+ print("Deleted dataset '{}'.".format(dataset_id))
+ # [END bigquery_delete_dataset]
diff --git a/bigquery/samples/delete_dataset_labels.py b/bigquery/samples/delete_dataset_labels.py
new file mode 100644
index 000000000000..33ff5c0f2620
--- /dev/null
+++ b/bigquery/samples/delete_dataset_labels.py
@@ -0,0 +1,36 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def delete_dataset_labels(client, dataset_id):
+
+ # [START bigquery_delete_label_dataset]
+ # TODO(developer): Import the client library.
+ # from google.cloud import bigquery
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ # TODO(developer): Set dataset_id to the ID of the dataset to fetch.
+ # dataset_id = "your-project.your_dataset"
+
+ dataset = client.get_dataset(dataset_id)
+
+ # To delete a label from a dataset, set its value to None
+ dataset.labels["color"] = None
+
+ dataset = client.update_dataset(dataset, ["labels"])
+ print("Labels deleted from {}".format(dataset_id))
+ # [END bigquery_delete_label_dataset]
+ return dataset
diff --git a/bigquery/samples/delete_model.py b/bigquery/samples/delete_model.py
new file mode 100644
index 000000000000..5ac4305bc97e
--- /dev/null
+++ b/bigquery/samples/delete_model.py
@@ -0,0 +1,32 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def delete_model(client, model_id):
+ """Sample ID: go/samples-tracker/1534"""
+
+ # [START bigquery_delete_model]
+ # TODO(developer): Import the client library.
+ # from google.cloud import bigquery
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ # TODO(developer): Set model_id to the ID of the model to fetch.
+ # model_id = 'your-project.your_dataset.your_model'
+
+ client.delete_model(model_id)
+
+ print("Deleted model '{}'.".format(model_id))
+ # [END bigquery_delete_model]
diff --git a/bigquery/samples/delete_routine.py b/bigquery/samples/delete_routine.py
new file mode 100644
index 000000000000..c0164b415008
--- /dev/null
+++ b/bigquery/samples/delete_routine.py
@@ -0,0 +1,31 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def delete_routine(client, routine_id):
+
+ # [START bigquery_delete_routine]
+ # TODO(developer): Import the client library.
+ # from google.cloud import bigquery
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ # TODO(developer): Set the fully-qualified ID for the routine.
+ # routine_id = "my-project.my_dataset.my_routine"
+
+ client.delete_routine(routine_id)
+
+ print("Deleted routine {}.".format(routine_id))
+ # [END bigquery_delete_routine]
diff --git a/bigquery/samples/delete_table.py b/bigquery/samples/delete_table.py
new file mode 100644
index 000000000000..dcdd3d855b2e
--- /dev/null
+++ b/bigquery/samples/delete_table.py
@@ -0,0 +1,32 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def delete_table(client, table_id):
+
+ # [START bigquery_delete_table]
+ # TODO(developer): Import the client library.
+ # from google.cloud import bigquery
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ # TODO(developer): Set table_id to the ID of the table to fetch.
+ # table_id = 'your-project.your_dataset.your_table'
+
+ # If the table does not exist, delete_table raises
+ # google.api_core.exceptions.NotFound unless not_found_ok is True
+ client.delete_table(table_id, not_found_ok=True)
+ print("Deleted table '{}'.".format(table_id))
+ # [END bigquery_delete_table]
diff --git a/bigquery/samples/get_dataset.py b/bigquery/samples/get_dataset.py
new file mode 100644
index 000000000000..5586c2b95ebb
--- /dev/null
+++ b/bigquery/samples/get_dataset.py
@@ -0,0 +1,56 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def get_dataset(client, dataset_id):
+
+ # [START bigquery_get_dataset]
+ # TODO(developer): Import the client library.
+ # from google.cloud import bigquery
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ # TODO(developer): Set dataset_id to the ID of the dataset to fetch.
+ # dataset_id = 'your-project.your_dataset'
+
+ dataset = client.get_dataset(dataset_id)
+
+ full_dataset_id = "{}.{}".format(dataset.project, dataset.dataset_id)
+ friendly_name = dataset.friendly_name
+ print(
+ "Got dataset '{}' with friendly_name '{}'.".format(
+ full_dataset_id, friendly_name
+ )
+ )
+
+ # View dataset properties
+ print("Description: {}".format(dataset.description))
+ print("Labels:")
+ labels = dataset.labels
+ if labels:
+ for label, value in labels.items():
+ print("\t{}: {}".format(label, value))
+ else:
+ print("\tDataset has no labels defined.")
+
+ # View tables in dataset
+ print("Tables:")
+ tables = list(client.list_tables(dataset)) # API request(s)
+ if tables:
+ for table in tables:
+ print("\t{}".format(table.table_id))
+ else:
+ print("\tThis dataset does not contain any tables.")
+ # [END bigquery_get_dataset]
diff --git a/bigquery/samples/get_dataset_labels.py b/bigquery/samples/get_dataset_labels.py
new file mode 100644
index 000000000000..2f21723a550b
--- /dev/null
+++ b/bigquery/samples/get_dataset_labels.py
@@ -0,0 +1,38 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def get_dataset_labels(client, dataset_id):
+
+ # [START bigquery_get_dataset_labels]
+ # TODO(developer): Import the client library.
+ # from google.cloud import bigquery
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ # TODO(developer): Set dataset_id to the ID of the dataset to fetch.
+ # dataset_id = "your-project.your_dataset"
+
+ dataset = client.get_dataset(dataset_id)
+
+ # View dataset labels
+ print("Dataset ID: {}".format(dataset_id))
+ print("Labels:")
+ if dataset.labels:
+ for label, value in dataset.labels.items():
+ print("\t{}: {}".format(label, value))
+ else:
+ print("\tDataset has no labels defined.")
+ # [END bigquery_get_dataset_labels]
diff --git a/bigquery/samples/get_model.py b/bigquery/samples/get_model.py
new file mode 100644
index 000000000000..69986733c50b
--- /dev/null
+++ b/bigquery/samples/get_model.py
@@ -0,0 +1,36 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def get_model(client, model_id):
+ """Sample ID: go/samples-tracker/1510"""
+
+ # [START bigquery_get_model]
+ # TODO(developer): Import the client library.
+ # from google.cloud import bigquery
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ # TODO(developer): Set model_id to the ID of the model to fetch.
+ # model_id = 'your-project.your_dataset.your_model'
+
+ model = client.get_model(model_id)
+
+ full_model_id = "{}.{}.{}".format(model.project, model.dataset_id, model.model_id)
+ friendly_name = model.friendly_name
+ print(
+ "Got model '{}' with friendly_name '{}'.".format(full_model_id, friendly_name)
+ )
+ # [END bigquery_get_model]
diff --git a/bigquery/samples/get_routine.py b/bigquery/samples/get_routine.py
new file mode 100644
index 000000000000..d9035c282438
--- /dev/null
+++ b/bigquery/samples/get_routine.py
@@ -0,0 +1,39 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def get_routine(client, routine_id):
+
+ # [START bigquery_get_routine]
+ # TODO(developer): Import the client library.
+ # from google.cloud import bigquery
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ # TODO(developer): Set the fully-qualified ID for the routine.
+ # routine_id = "my-project.my_dataset.my_routine"
+
+ routine = client.get_routine(routine_id)
+
+ print("Routine `{}`:".format(routine.reference))
+ print(" Type: '{}'".format(routine.type_))
+ print(" Language: '{}'".format(routine.language))
+ print(" Arguments:")
+
+ for argument in routine.arguments:
+ print(" Name: '{}'".format(argument.name))
+ print(" Type: '{}'".format(argument.type_))
+ # [END bigquery_get_routine]
+ return routine
diff --git a/bigquery/samples/get_table.py b/bigquery/samples/get_table.py
new file mode 100644
index 000000000000..e49e032f6e23
--- /dev/null
+++ b/bigquery/samples/get_table.py
@@ -0,0 +1,38 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def get_table(client, table_id):
+
+ # [START bigquery_get_table]
+ # TODO(developer): Import the client library.
+ # from google.cloud import bigquery
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ # TODO(developer): Set table_id to the ID of the model to fetch.
+ # table_id = 'your-project.your_dataset.your_table'
+
+ table = client.get_table(table_id)
+
+ print(
+ "Got table '{}.{}.{}'.".format(table.project, table.dataset_id, table.table_id)
+ )
+
+ # View table properties
+ print("Table schema: {}".format(table.schema))
+ print("Table description: {}".format(table.description))
+ print("Table has {} rows".format(table.num_rows))
+ # [END bigquery_get_table]
diff --git a/bigquery/samples/label_dataset.py b/bigquery/samples/label_dataset.py
new file mode 100644
index 000000000000..7840ea25a63f
--- /dev/null
+++ b/bigquery/samples/label_dataset.py
@@ -0,0 +1,33 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def label_dataset(client, dataset_id):
+
+ # [START bigquery_label_dataset]
+ # TODO(developer): Import the client library.
+ # from google.cloud import bigquery
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ # TODO(developer): Set dataset_id to the ID of the dataset to fetch.
+ # dataset_id = "your-project.your_dataset"
+
+ dataset = client.get_dataset(dataset_id)
+ dataset.labels = {"color": "green"}
+ dataset = client.update_dataset(dataset, ["labels"])
+
+ print("Labels added to {}".format(dataset_id))
+ # [END bigquery_label_dataset]
diff --git a/bigquery/samples/list_datasets.py b/bigquery/samples/list_datasets.py
new file mode 100644
index 000000000000..b57aad1b5e7b
--- /dev/null
+++ b/bigquery/samples/list_datasets.py
@@ -0,0 +1,34 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def list_datasets(client):
+
+ # [START bigquery_list_datasets]
+ # TODO(developer): Import the client library.
+ # from google.cloud import bigquery
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ datasets = list(client.list_datasets())
+ project = client.project
+
+ if datasets:
+ print("Datasets in project {}:".format(project))
+ for dataset in datasets: # API request(s)
+ print("\t{}".format(dataset.dataset_id))
+ else:
+ print("{} project does not contain any datasets.".format(project))
+ # [END bigquery_list_datasets]
diff --git a/bigquery/samples/list_datasets_by_label.py b/bigquery/samples/list_datasets_by_label.py
new file mode 100644
index 000000000000..8b574b1110eb
--- /dev/null
+++ b/bigquery/samples/list_datasets_by_label.py
@@ -0,0 +1,34 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def list_datasets_by_label(client):
+
+ # [START bigquery_list_datasets_by_label]
+ # TODO(developer): Import the client library.
+ # from google.cloud import bigquery
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ label_filter = "labels.color:green"
+ datasets = list(client.list_datasets(filter=label_filter))
+
+ if datasets:
+ print("Datasets filtered by {}:".format(label_filter))
+ for dataset in datasets:
+ print("\t{}.{}".format(dataset.project, dataset.dataset_id))
+ else:
+ print("No datasets found with this filter.")
+ # [END bigquery_list_datasets_by_label]
diff --git a/bigquery/samples/list_models.py b/bigquery/samples/list_models.py
new file mode 100644
index 000000000000..5b4d21799b28
--- /dev/null
+++ b/bigquery/samples/list_models.py
@@ -0,0 +1,39 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def list_models(client, dataset_id):
+ """Sample ID: go/samples-tracker/1512"""
+
+ # [START bigquery_list_models]
+ # TODO(developer): Import the client library.
+ # from google.cloud import bigquery
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ # TODO(developer): Set dataset_id to the ID of the dataset that contains
+ # the models you are listing.
+ # dataset_id = 'your-project.your_dataset'
+
+ models = client.list_models(dataset_id)
+
+ print("Models contained in '{}':".format(dataset_id))
+ for model in models:
+ full_model_id = "{}.{}.{}".format(
+ model.project, model.dataset_id, model.model_id
+ )
+ friendly_name = model.friendly_name
+ print("{}: friendly_name='{}'".format(full_model_id, friendly_name))
+ # [END bigquery_list_models]
diff --git a/bigquery/samples/list_routines.py b/bigquery/samples/list_routines.py
new file mode 100644
index 000000000000..1ae4f441cde1
--- /dev/null
+++ b/bigquery/samples/list_routines.py
@@ -0,0 +1,34 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def list_routines(client, dataset_id):
+
+ # [START bigquery_list_routines]
+ # TODO(developer): Import the client library.
+ # from google.cloud import bigquery
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ # TODO(developer): Set dataset_id to the ID of the dataset that contains
+ # the routines you are listing.
+ # dataset_id = 'your-project.your_dataset'
+
+ routines = client.list_routines(dataset_id)
+
+ print("Routines contained in dataset {}:".format(dataset_id))
+ for routine in routines:
+ print(routine.reference)
+ # [END bigquery_list_routines]
diff --git a/bigquery/samples/list_tables.py b/bigquery/samples/list_tables.py
new file mode 100644
index 000000000000..2057f2d73891
--- /dev/null
+++ b/bigquery/samples/list_tables.py
@@ -0,0 +1,34 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def list_tables(client, dataset_id):
+
+ # [START bigquery_list_tables]
+ # TODO(developer): Import the client library.
+ # from google.cloud import bigquery
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ # TODO(developer): Set dataset_id to the ID of the dataset that contains
+ # the tables you are listing.
+ # dataset_id = 'your-project.your_dataset'
+
+ tables = client.list_tables(dataset_id)
+
+ print("Tables contained in '{}':".format(dataset_id))
+ for table in tables:
+ print("{}.{}.{}".format(table.project, table.dataset_id, table.table_id))
+ # [END bigquery_list_tables]
diff --git a/bigquery/samples/load_table_dataframe.py b/bigquery/samples/load_table_dataframe.py
new file mode 100644
index 000000000000..69eeb6ef89d0
--- /dev/null
+++ b/bigquery/samples/load_table_dataframe.py
@@ -0,0 +1,73 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def load_table_dataframe(client, table_id):
+ # [START bigquery_load_table_dataframe]
+ from google.cloud import bigquery
+ import pandas
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ # TODO(developer): Set table_id to the ID of the table to create.
+ # table_id = "your-project.your_dataset.your_table_name"
+
+ records = [
+ {"title": u"The Meaning of Life", "release_year": 1983},
+ {"title": u"Monty Python and the Holy Grail", "release_year": 1975},
+ {"title": u"Life of Brian", "release_year": 1979},
+ {"title": u"And Now for Something Completely Different", "release_year": 1971},
+ ]
+ dataframe = pandas.DataFrame(
+ records,
+ # In the loaded table, the column order reflects the order of the
+ # columns in the DataFrame.
+ columns=["title", "release_year"],
+ # Optionally, set a named index, which can also be written to the
+ # BigQuery table.
+ index=pandas.Index(
+ [u"Q24980", u"Q25043", u"Q24953", u"Q16403"], name="wikidata_id"
+ ),
+ )
+ job_config = bigquery.LoadJobConfig(
+ # Specify a (partial) schema. All columns are always written to the
+ # table. The schema is used to assist in data type definitions.
+ schema=[
+ # Specify the type of columns whose type cannot be auto-detected. For
+ # example the "title" column uses pandas dtype "object", so its
+ # data type is ambiguous.
+ bigquery.SchemaField("title", bigquery.enums.SqlTypeNames.STRING),
+ # Indexes are written if included in the schema by name.
+ bigquery.SchemaField("wikidata_id", bigquery.enums.SqlTypeNames.STRING),
+ ],
+ # Optionally, set the write disposition. BigQuery appends loaded rows
+ # to an existing table by default, but with WRITE_TRUNCATE write
+ # disposition it replaces the table with the loaded data.
+ write_disposition="WRITE_TRUNCATE",
+ )
+
+ job = client.load_table_from_dataframe(
+ dataframe, table_id, job_config=job_config, location="US"
+ )
+ job.result() # Waits for table load to complete.
+
+ table = client.get_table(table_id)
+ print(
+ "Loaded {} rows and {} columns to {}".format(
+ table.num_rows, len(table.schema), table_id
+ )
+ )
+ # [END bigquery_load_table_dataframe]
+ return table
diff --git a/bigquery/samples/query_to_arrow.py b/bigquery/samples/query_to_arrow.py
new file mode 100644
index 000000000000..b13dcf3e1413
--- /dev/null
+++ b/bigquery/samples/query_to_arrow.py
@@ -0,0 +1,53 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def query_to_arrow(client):
+
+ # [START bigquery_query_to_arrow]
+ # TODO(developer): Import the client library.
+ # from google.cloud import bigquery
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ sql = """
+ WITH races AS (
+ SELECT "800M" AS race,
+ [STRUCT("Rudisha" as name, [23.4, 26.3, 26.4, 26.1] as splits),
+ STRUCT("Makhloufi" as name, [24.5, 25.4, 26.6, 26.1] as splits),
+ STRUCT("Murphy" as name, [23.9, 26.0, 27.0, 26.0] as splits),
+ STRUCT("Bosse" as name, [23.6, 26.2, 26.5, 27.1] as splits),
+ STRUCT("Rotich" as name, [24.7, 25.6, 26.9, 26.4] as splits),
+ STRUCT("Lewandowski" as name, [25.0, 25.7, 26.3, 27.2] as splits),
+ STRUCT("Kipketer" as name, [23.2, 26.1, 27.3, 29.4] as splits),
+ STRUCT("Berian" as name, [23.7, 26.1, 27.0, 29.3] as splits)]
+ AS participants)
+ SELECT
+ race,
+ participant
+ FROM races r
+ CROSS JOIN UNNEST(r.participants) as participant;
+ """
+ query_job = client.query(sql)
+ arrow_table = query_job.to_arrow()
+
+ print(
+ "Downloaded {} rows, {} columns.".format(
+ arrow_table.num_rows, arrow_table.num_columns
+ )
+ )
+ print("\nSchema:\n{}".format(repr(arrow_table.schema)))
+ # [END bigquery_query_to_arrow]
+ return arrow_table
diff --git a/bigquery/samples/tests/__init__.py b/bigquery/samples/tests/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/bigquery/samples/tests/conftest.py b/bigquery/samples/tests/conftest.py
new file mode 100644
index 000000000000..f2bb93112a22
--- /dev/null
+++ b/bigquery/samples/tests/conftest.py
@@ -0,0 +1,134 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import datetime
+import uuid
+
+import pytest
+
+from google.cloud import bigquery
+from google.cloud import bigquery_v2
+
+
+@pytest.fixture(scope="module")
+def client():
+ return bigquery.Client()
+
+
+@pytest.fixture
+def random_table_id(client, dataset_id):
+ now = datetime.datetime.now()
+ random_table_id = "example_table_{}_{}".format(
+ now.strftime("%Y%m%d%H%M%S"), uuid.uuid4().hex[:8]
+ )
+ return "{}.{}".format(dataset_id, random_table_id)
+
+
+@pytest.fixture
+def random_dataset_id(client):
+ now = datetime.datetime.now()
+ random_dataset_id = "example_dataset_{}_{}".format(
+ now.strftime("%Y%m%d%H%M%S"), uuid.uuid4().hex[:8]
+ )
+ yield "{}.{}".format(client.project, random_dataset_id)
+ client.delete_dataset(random_dataset_id, delete_contents=True, not_found_ok=True)
+
+
+@pytest.fixture
+def random_routine_id(client, dataset_id):
+ now = datetime.datetime.now()
+ random_routine_id = "example_routine_{}_{}".format(
+ now.strftime("%Y%m%d%H%M%S"), uuid.uuid4().hex[:8]
+ )
+ return "{}.{}".format(dataset_id, random_routine_id)
+
+
+@pytest.fixture
+def dataset_id(client):
+ now = datetime.datetime.now()
+ dataset_id = "python_samples_{}_{}".format(
+ now.strftime("%Y%m%d%H%M%S"), uuid.uuid4().hex[:8]
+ )
+ dataset = client.create_dataset(dataset_id)
+ yield "{}.{}".format(dataset.project, dataset.dataset_id)
+ client.delete_dataset(dataset, delete_contents=True, not_found_ok=True)
+
+
+@pytest.fixture
+def table_id(client, dataset_id):
+ now = datetime.datetime.now()
+ table_id = "python_samples_{}_{}".format(
+ now.strftime("%Y%m%d%H%M%S"), uuid.uuid4().hex[:8]
+ )
+
+ table = bigquery.Table("{}.{}".format(dataset_id, table_id))
+ table = client.create_table(table)
+ yield "{}.{}.{}".format(table.project, table.dataset_id, table.table_id)
+ client.delete_table(table, not_found_ok=True)
+
+
+@pytest.fixture
+def table_with_data_id(client):
+ return "bigquery-public-data.samples.shakespeare"
+
+
+@pytest.fixture
+def routine_id(client, dataset_id):
+ now = datetime.datetime.now()
+ routine_id = "python_samples_{}_{}".format(
+ now.strftime("%Y%m%d%H%M%S"), uuid.uuid4().hex[:8]
+ )
+
+ routine = bigquery.Routine("{}.{}".format(dataset_id, routine_id))
+ routine.type_ = "SCALAR_FUNCTION"
+ routine.language = "SQL"
+ routine.body = "x * 3"
+ routine.arguments = [
+ bigquery.RoutineArgument(
+ name="x",
+ data_type=bigquery_v2.types.StandardSqlDataType(
+ type_kind=bigquery_v2.enums.StandardSqlDataType.TypeKind.INT64
+ ),
+ )
+ ]
+
+ routine = client.create_routine(routine)
+ yield "{}.{}.{}".format(routine.project, routine.dataset_id, routine.routine_id)
+ client.delete_routine(routine, not_found_ok=True)
+
+
+@pytest.fixture
+def model_id(client, dataset_id):
+ model_id = "{}.{}".format(dataset_id, uuid.uuid4().hex)
+
+ # The only way to create a model resource is via SQL.
+ # Use a very small dataset (2 points), to train a model quickly.
+ sql = """
+ CREATE MODEL `{}`
+ OPTIONS (
+ model_type='linear_reg',
+ max_iteration=1,
+ learn_rate=0.4,
+ learn_rate_strategy='constant'
+ ) AS (
+ SELECT 'a' AS f1, 2.0 AS label
+ UNION ALL
+ SELECT 'b' AS f1, 3.8 AS label
+ )
+ """.format(
+ model_id
+ )
+
+ client.query(sql).result()
+ return model_id
diff --git a/bigquery/samples/tests/test_add_empty_column.py b/bigquery/samples/tests/test_add_empty_column.py
new file mode 100644
index 000000000000..e6c56e6cbfbc
--- /dev/null
+++ b/bigquery/samples/tests/test_add_empty_column.py
@@ -0,0 +1,23 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from .. import add_empty_column
+
+
+def test_add_empty_column(capsys, client, table_id):
+
+ add_empty_column.add_empty_column(client, table_id)
+ out, err = capsys.readouterr()
+ assert "A new column has been added." in out
diff --git a/bigquery/samples/tests/test_browse_table_data.py b/bigquery/samples/tests/test_browse_table_data.py
new file mode 100644
index 000000000000..f777bf91ca00
--- /dev/null
+++ b/bigquery/samples/tests/test_browse_table_data.py
@@ -0,0 +1,26 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from .. import browse_table_data
+
+
+def test_browse_table_data(capsys, client, table_with_data_id):
+
+ browse_table_data.browse_table_data(client, table_with_data_id)
+ out, err = capsys.readouterr()
+ assert "Downloaded 164656 rows from table {}".format(table_with_data_id) in out
+ assert "Downloaded 10 rows from table {}".format(table_with_data_id) in out
+ assert "Selected 2 columns from table {}".format(table_with_data_id) in out
+ assert "Downloaded 10 rows from table {}".format(table_with_data_id) in out
diff --git a/bigquery/samples/tests/test_create_dataset.py b/bigquery/samples/tests/test_create_dataset.py
new file mode 100644
index 000000000000..dfadc67d8468
--- /dev/null
+++ b/bigquery/samples/tests/test_create_dataset.py
@@ -0,0 +1,22 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from .. import create_dataset
+
+
+def test_create_dataset(capsys, client, random_dataset_id):
+
+ create_dataset.create_dataset(client, random_dataset_id)
+ out, err = capsys.readouterr()
+ assert "Created dataset {}".format(random_dataset_id) in out
diff --git a/bigquery/samples/tests/test_create_job.py b/bigquery/samples/tests/test_create_job.py
new file mode 100644
index 000000000000..fce005ae8236
--- /dev/null
+++ b/bigquery/samples/tests/test_create_job.py
@@ -0,0 +1,23 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from .. import create_job
+
+
+def test_create_job(capsys, client):
+
+ query_job = create_job.create_job(client)
+ client.cancel_job(query_job.job_id, location="US")
+ out, err = capsys.readouterr()
+ assert "Started job: {}".format(query_job.job_id) in out
diff --git a/bigquery/samples/tests/test_create_table.py b/bigquery/samples/tests/test_create_table.py
new file mode 100644
index 000000000000..093ee6e94277
--- /dev/null
+++ b/bigquery/samples/tests/test_create_table.py
@@ -0,0 +1,21 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from .. import create_table
+
+
+def test_create_table(capsys, client, random_table_id):
+ create_table.create_table(client, random_table_id)
+ out, err = capsys.readouterr()
+ assert "Created table {}".format(random_table_id) in out
diff --git a/bigquery/samples/tests/test_dataset_exists.py b/bigquery/samples/tests/test_dataset_exists.py
new file mode 100644
index 000000000000..203c4b884dc4
--- /dev/null
+++ b/bigquery/samples/tests/test_dataset_exists.py
@@ -0,0 +1,30 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from google.cloud import bigquery
+
+from .. import dataset_exists
+
+
+def test_dataset_exists(capsys, client, random_dataset_id):
+
+ dataset_exists.dataset_exists(client, random_dataset_id)
+ out, err = capsys.readouterr()
+ assert "Dataset {} is not found".format(random_dataset_id) in out
+ dataset = bigquery.Dataset(random_dataset_id)
+ dataset = client.create_dataset(dataset)
+ dataset_exists.dataset_exists(client, random_dataset_id)
+ out, err = capsys.readouterr()
+ assert "Dataset {} already exists".format(random_dataset_id) in out
diff --git a/bigquery/samples/tests/test_dataset_label_samples.py b/bigquery/samples/tests/test_dataset_label_samples.py
new file mode 100644
index 000000000000..94a2092407b0
--- /dev/null
+++ b/bigquery/samples/tests/test_dataset_label_samples.py
@@ -0,0 +1,33 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from .. import delete_dataset_labels
+from .. import get_dataset_labels
+from .. import label_dataset
+
+
+def test_dataset_label_samples(capsys, client, dataset_id):
+
+ label_dataset.label_dataset(client, dataset_id)
+ out, err = capsys.readouterr()
+ assert "Labels added to {}".format(dataset_id) in out
+
+ get_dataset_labels.get_dataset_labels(client, dataset_id)
+ out, err = capsys.readouterr()
+ assert "color: green" in out
+
+ dataset = delete_dataset_labels.delete_dataset_labels(client, dataset_id)
+ out, err = capsys.readouterr()
+ assert "Labels deleted from {}".format(dataset_id) in out
+ assert dataset.labels.get("color") is None
diff --git a/bigquery/samples/tests/test_delete_dataset.py b/bigquery/samples/tests/test_delete_dataset.py
new file mode 100644
index 000000000000..2b1b6ad06195
--- /dev/null
+++ b/bigquery/samples/tests/test_delete_dataset.py
@@ -0,0 +1,22 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from .. import delete_dataset
+
+
+def test_delete_dataset(capsys, client, dataset_id):
+
+ delete_dataset.delete_dataset(client, dataset_id)
+ out, err = capsys.readouterr()
+ assert "Deleted dataset '{}'.".format(dataset_id) in out
diff --git a/bigquery/samples/tests/test_delete_table.py b/bigquery/samples/tests/test_delete_table.py
new file mode 100644
index 000000000000..8f4796623a83
--- /dev/null
+++ b/bigquery/samples/tests/test_delete_table.py
@@ -0,0 +1,22 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from .. import delete_table
+
+
+def test_delete_table(capsys, client, table_id):
+
+ delete_table.delete_table(client, table_id)
+ out, err = capsys.readouterr()
+ assert "Deleted table '{}'.".format(table_id) in out
diff --git a/bigquery/samples/tests/test_get_dataset.py b/bigquery/samples/tests/test_get_dataset.py
new file mode 100644
index 000000000000..374f8835211a
--- /dev/null
+++ b/bigquery/samples/tests/test_get_dataset.py
@@ -0,0 +1,22 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from .. import get_dataset
+
+
+def test_get_dataset(capsys, client, dataset_id):
+
+ get_dataset.get_dataset(client, dataset_id)
+ out, err = capsys.readouterr()
+ assert "{}".format(dataset_id) in out
diff --git a/bigquery/samples/tests/test_get_table.py b/bigquery/samples/tests/test_get_table.py
new file mode 100644
index 000000000000..b811ccecad1f
--- /dev/null
+++ b/bigquery/samples/tests/test_get_table.py
@@ -0,0 +1,36 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from google.cloud import bigquery
+from .. import get_table
+
+
+def test_get_table(capsys, client, random_table_id):
+
+ schema = [
+ bigquery.SchemaField("full_name", "STRING", mode="REQUIRED"),
+ bigquery.SchemaField("age", "INTEGER", mode="REQUIRED"),
+ ]
+
+ table = bigquery.Table(random_table_id, schema)
+ table.description = "Sample Table"
+ table = client.create_table(table)
+
+ get_table.get_table(client, random_table_id)
+ out, err = capsys.readouterr()
+ assert "Got table '{}'.".format(random_table_id) in out
+ assert "full_name" in out # test that schema is printed
+ assert "Table description: Sample Table" in out
+ assert "Table has 0 rows" in out
+ client.delete_table(table, not_found_ok=True)
diff --git a/bigquery/samples/tests/test_list_datasets.py b/bigquery/samples/tests/test_list_datasets.py
new file mode 100644
index 000000000000..4c66a24f9b1a
--- /dev/null
+++ b/bigquery/samples/tests/test_list_datasets.py
@@ -0,0 +1,22 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from .. import list_datasets
+
+
+def test_list_datasets(capsys, client, dataset_id):
+
+ list_datasets.list_datasets(client)
+ out, err = capsys.readouterr()
+ assert "Datasets in project {}:".format(client.project) in out
diff --git a/bigquery/samples/tests/test_list_datasets_by_label.py b/bigquery/samples/tests/test_list_datasets_by_label.py
new file mode 100644
index 000000000000..346cbf1a982d
--- /dev/null
+++ b/bigquery/samples/tests/test_list_datasets_by_label.py
@@ -0,0 +1,26 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from .. import list_datasets_by_label
+
+
+def test_list_datasets_by_label(capsys, client, dataset_id):
+
+ dataset = client.get_dataset(dataset_id)
+ dataset.labels = {"color": "green"}
+ dataset = client.update_dataset(dataset, ["labels"])
+ list_datasets_by_label.list_datasets_by_label(client)
+ out, err = capsys.readouterr()
+ assert "{}".format(dataset_id) in out
diff --git a/bigquery/samples/tests/test_list_tables.py b/bigquery/samples/tests/test_list_tables.py
new file mode 100644
index 000000000000..ec1621ac7579
--- /dev/null
+++ b/bigquery/samples/tests/test_list_tables.py
@@ -0,0 +1,23 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from .. import list_tables
+
+
+def test_list_tables(capsys, client, dataset_id, table_id):
+
+ list_tables.list_tables(client, dataset_id)
+ out, err = capsys.readouterr()
+ assert "Tables contained in '{}':".format(dataset_id) in out
+ assert table_id in out
diff --git a/bigquery/samples/tests/test_load_table_dataframe.py b/bigquery/samples/tests/test_load_table_dataframe.py
new file mode 100644
index 000000000000..d553d449a525
--- /dev/null
+++ b/bigquery/samples/tests/test_load_table_dataframe.py
@@ -0,0 +1,30 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import pytest
+
+from .. import load_table_dataframe
+
+
+pytest.importorskip("pandas")
+pytest.importorskip("pyarrow")
+
+
+def test_load_table_dataframe(capsys, client, random_table_id):
+ table = load_table_dataframe.load_table_dataframe(client, random_table_id)
+ out, _ = capsys.readouterr()
+ assert "Loaded 4 rows and 3 columns" in out
+
+ column_names = [field.name for field in table.schema]
+ assert column_names == ["wikidata_id", "title", "release_year"]
diff --git a/bigquery/samples/tests/test_model_samples.py b/bigquery/samples/tests/test_model_samples.py
new file mode 100644
index 000000000000..d7b06a92a3e1
--- /dev/null
+++ b/bigquery/samples/tests/test_model_samples.py
@@ -0,0 +1,39 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from .. import delete_model
+from .. import get_model
+from .. import list_models
+from .. import update_model
+
+
+def test_model_samples(capsys, client, dataset_id, model_id):
+ """Since creating a model is a long operation, test all model samples in
+ the same test, following a typical end-to-end flow.
+ """
+ get_model.get_model(client, model_id)
+ out, err = capsys.readouterr()
+ assert model_id in out
+
+ list_models.list_models(client, dataset_id)
+ out, err = capsys.readouterr()
+ assert "Models contained in '{}':".format(dataset_id) in out
+
+ update_model.update_model(client, model_id)
+ out, err = capsys.readouterr()
+ assert "This model was modified from a Python program." in out
+
+ delete_model.delete_model(client, model_id)
+ out, err = capsys.readouterr()
+ assert "Deleted model '{}'.".format(model_id) in out
diff --git a/bigquery/samples/tests/test_query_to_arrow.py b/bigquery/samples/tests/test_query_to_arrow.py
new file mode 100644
index 000000000000..f70bd49fe565
--- /dev/null
+++ b/bigquery/samples/tests/test_query_to_arrow.py
@@ -0,0 +1,29 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import pyarrow
+
+from .. import query_to_arrow
+
+
+def test_query_to_arrow(capsys, client):
+
+ arrow_table = query_to_arrow.query_to_arrow(client)
+ out, err = capsys.readouterr()
+ assert "Downloaded 8 rows, 2 columns." in out
+
+ arrow_schema = arrow_table.schema
+ assert arrow_schema.names == ["race", "participant"]
+ assert pyarrow.types.is_string(arrow_schema.types[0])
+ assert pyarrow.types.is_struct(arrow_schema.types[1])
diff --git a/bigquery/samples/tests/test_routine_samples.py b/bigquery/samples/tests/test_routine_samples.py
new file mode 100644
index 000000000000..5a1c69c7f60f
--- /dev/null
+++ b/bigquery/samples/tests/test_routine_samples.py
@@ -0,0 +1,89 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from google.cloud import bigquery
+from google.cloud import bigquery_v2
+
+
+def test_create_routine(capsys, client, random_routine_id):
+ from .. import create_routine
+
+ create_routine.create_routine(client, random_routine_id)
+ out, err = capsys.readouterr()
+ assert "Created routine {}".format(random_routine_id) in out
+
+
+def test_create_routine_ddl(capsys, client, random_routine_id):
+ from .. import create_routine_ddl
+
+ create_routine_ddl.create_routine_ddl(client, random_routine_id)
+ routine = client.get_routine(random_routine_id)
+ out, err = capsys.readouterr()
+
+ assert "Created routine {}".format(random_routine_id) in out
+ return routine
+ assert routine.type_ == "SCALAR_FUNCTION"
+ assert routine.language == "SQL"
+ expected_arguments = [
+ bigquery.RoutineArgument(
+ name="arr",
+ data_type=bigquery_v2.types.StandardSqlDataType(
+ type_kind=bigquery_v2.enums.StandardSqlDataType.TypeKind.ARRAY,
+ array_element_type=bigquery_v2.types.StandardSqlDataType(
+ type_kind=bigquery_v2.enums.StandardSqlDataType.TypeKind.STRUCT,
+ struct_type=bigquery_v2.types.StandardSqlStructType(
+ fields=[
+ bigquery_v2.types.StandardSqlField(
+ name="name",
+ type=bigquery_v2.types.StandardSqlDataType(
+ type_kind=bigquery_v2.enums.StandardSqlDataType.TypeKind.STRING
+ ),
+ ),
+ bigquery_v2.types.StandardSqlField(
+ name="val",
+ type=bigquery_v2.types.StandardSqlDataType(
+ type_kind=bigquery_v2.enums.StandardSqlDataType.TypeKind.INT64
+ ),
+ ),
+ ]
+ ),
+ ),
+ ),
+ )
+ ]
+ assert routine.arguments == expected_arguments
+
+
+def test_list_routines(capsys, client, dataset_id, routine_id):
+ from .. import list_routines
+
+ list_routines.list_routines(client, dataset_id)
+ out, err = capsys.readouterr()
+ assert "Routines contained in dataset {}:".format(dataset_id) in out
+ assert routine_id in out
+
+
+def test_delete_routine(capsys, client, routine_id):
+ from .. import delete_routine
+
+ delete_routine.delete_routine(client, routine_id)
+ out, err = capsys.readouterr()
+ assert "Deleted routine {}.".format(routine_id) in out
+
+
+def test_update_routine(client, routine_id):
+ from .. import update_routine
+
+ routine = update_routine.update_routine(client, routine_id)
+ assert routine.body == "x * 4"
diff --git a/bigquery/samples/tests/test_update_dataset_access.py b/bigquery/samples/tests/test_update_dataset_access.py
new file mode 100644
index 000000000000..ae33dbfe4a4c
--- /dev/null
+++ b/bigquery/samples/tests/test_update_dataset_access.py
@@ -0,0 +1,24 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from .. import update_dataset_access
+
+
+def test_update_dataset_access(capsys, client, dataset_id):
+
+ update_dataset_access.update_dataset_access(client, dataset_id)
+ out, err = capsys.readouterr()
+ assert (
+ "Updated dataset '{}' with modified user permissions.".format(dataset_id) in out
+ )
diff --git a/bigquery/samples/tests/test_update_dataset_default_table_expiration.py b/bigquery/samples/tests/test_update_dataset_default_table_expiration.py
new file mode 100644
index 000000000000..46e9654209ed
--- /dev/null
+++ b/bigquery/samples/tests/test_update_dataset_default_table_expiration.py
@@ -0,0 +1,29 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from .. import update_dataset_default_table_expiration
+
+
+def test_update_dataset_default_table_expiration(capsys, client, dataset_id):
+
+ one_day_ms = 24 * 60 * 60 * 1000 # in milliseconds
+
+ update_dataset_default_table_expiration.update_dataset_default_table_expiration(
+ client, dataset_id
+ )
+ out, err = capsys.readouterr()
+ assert (
+ "Updated dataset {} with new expiration {}".format(dataset_id, one_day_ms)
+ in out
+ )
diff --git a/bigquery/samples/tests/test_update_dataset_description.py b/bigquery/samples/tests/test_update_dataset_description.py
new file mode 100644
index 000000000000..c6f8889f50da
--- /dev/null
+++ b/bigquery/samples/tests/test_update_dataset_description.py
@@ -0,0 +1,22 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from .. import update_dataset_description
+
+
+def test_update_dataset_description(capsys, client, dataset_id):
+
+ update_dataset_description.update_dataset_description(client, dataset_id)
+ out, err = capsys.readouterr()
+ assert "Updated description." in out
diff --git a/bigquery/samples/update_dataset_access.py b/bigquery/samples/update_dataset_access.py
new file mode 100644
index 000000000000..aa316a38dff9
--- /dev/null
+++ b/bigquery/samples/update_dataset_access.py
@@ -0,0 +1,45 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def update_dataset_access(client, dataset_id):
+
+ # [START bigquery_update_dataset_access]
+ from google.cloud import bigquery
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ # TODO(developer): Set dataset_id to the ID of the dataset to fetch.
+ # dataset_id = 'your-project.your_dataset'
+
+ dataset = client.get_dataset(dataset_id)
+
+ entry = bigquery.AccessEntry(
+ role="READER",
+ entity_type="userByEmail",
+ entity_id="sample.bigquery.dev@gmail.com",
+ )
+
+ entries = list(dataset.access_entries)
+ entries.append(entry)
+ dataset.access_entries = entries
+
+ dataset = client.update_dataset(dataset, ["access_entries"]) # API request
+
+ full_dataset_id = "{}.{}".format(dataset.project, dataset.dataset_id)
+ print(
+ "Updated dataset '{}' with modified user permissions.".format(full_dataset_id)
+ )
+ # [END bigquery_update_dataset_access]
diff --git a/bigquery/samples/update_dataset_default_table_expiration.py b/bigquery/samples/update_dataset_default_table_expiration.py
new file mode 100644
index 000000000000..4534bb2011eb
--- /dev/null
+++ b/bigquery/samples/update_dataset_default_table_expiration.py
@@ -0,0 +1,41 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def update_dataset_default_table_expiration(client, dataset_id):
+
+ # [START bigquery_update_dataset_expiration]
+ # TODO(developer): Import the client library.
+ # from google.cloud import bigquery
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ # TODO(developer): Set dataset_id to the ID of the dataset to fetch.
+ # dataset_id = 'your-project.your_dataset'
+
+ dataset = client.get_dataset(dataset_id)
+ dataset.default_table_expiration_ms = 24 * 60 * 60 * 1000 # in milliseconds
+
+ dataset = client.update_dataset(
+ dataset, ["default_table_expiration_ms"]
+ ) # API request
+
+ full_dataset_id = "{}.{}".format(dataset.project, dataset.dataset_id)
+ print(
+ "Updated dataset {} with new expiration {}".format(
+ full_dataset_id, dataset.default_table_expiration_ms
+ )
+ )
+ # [END bigquery_update_dataset_expiration]
diff --git a/bigquery/samples/update_dataset_description.py b/bigquery/samples/update_dataset_description.py
new file mode 100644
index 000000000000..f3afb7fa68ce
--- /dev/null
+++ b/bigquery/samples/update_dataset_description.py
@@ -0,0 +1,38 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def update_dataset_description(client, dataset_id):
+
+ # [START bigquery_update_dataset_description]
+ # TODO(developer): Import the client library.
+ # from google.cloud import bigquery
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ # TODO(developer): Set dataset_id to the ID of the dataset to fetch.
+ # dataset_id = 'your-project.your_dataset'
+
+ dataset = client.get_dataset(dataset_id)
+ dataset.description = "Updated description."
+ dataset = client.update_dataset(dataset, ["description"])
+
+ full_dataset_id = "{}.{}".format(dataset.project, dataset.dataset_id)
+ print(
+ "Updated dataset '{}' with description '{}'.".format(
+ full_dataset_id, dataset.description
+ )
+ )
+ # [END bigquery_update_dataset_description]
diff --git a/bigquery/samples/update_model.py b/bigquery/samples/update_model.py
new file mode 100644
index 000000000000..5df4ada886ed
--- /dev/null
+++ b/bigquery/samples/update_model.py
@@ -0,0 +1,39 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def update_model(client, model_id):
+ """Sample ID: go/samples-tracker/1533"""
+
+ # [START bigquery_update_model_description]
+ # TODO(developer): Import the client library.
+ # from google.cloud import bigquery
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ # TODO(developer): Set model_id to the ID of the model to fetch.
+ # model_id = 'your-project.your_dataset.your_model'
+
+ model = client.get_model(model_id)
+ model.description = "This model was modified from a Python program."
+ model = client.update_model(model, ["description"])
+
+ full_model_id = "{}.{}.{}".format(model.project, model.dataset_id, model.model_id)
+ print(
+ "Updated model '{}' with description '{}'.".format(
+ full_model_id, model.description
+ )
+ )
+ # [END bigquery_update_model_description]
diff --git a/bigquery/samples/update_routine.py b/bigquery/samples/update_routine.py
new file mode 100644
index 000000000000..4d491d42e168
--- /dev/null
+++ b/bigquery/samples/update_routine.py
@@ -0,0 +1,45 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def update_routine(client, routine_id):
+
+ # [START bigquery_update_routine]
+ # TODO(developer): Import the client library.
+ # from google.cloud import bigquery
+
+ # TODO(developer): Construct a BigQuery client object.
+ # client = bigquery.Client()
+
+ # TODO(developer): Set the fully-qualified ID for the routine.
+ # routine_id = "my-project.my_dataset.my_routine"
+
+ routine = client.get_routine(routine_id)
+
+ routine.body = "x * 4"
+
+ routine = client.update_routine(
+ routine,
+ [
+ "body",
+ # Due to a limitation of the API, all fields are required, not just
+ # those that have been updated.
+ "arguments",
+ "language",
+ "type_",
+ "return_type",
+ ],
+ )
+ # [END bigquery_update_routine]
+ return routine
diff --git a/bigquery/setup.cfg b/bigquery/setup.cfg
new file mode 100644
index 000000000000..2a9acf13daa9
--- /dev/null
+++ b/bigquery/setup.cfg
@@ -0,0 +1,2 @@
+[bdist_wheel]
+universal = 1
diff --git a/bigquery/setup.py b/bigquery/setup.py
new file mode 100644
index 000000000000..b3eeb15b5d64
--- /dev/null
+++ b/bigquery/setup.py
@@ -0,0 +1,118 @@
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import io
+import os
+
+import setuptools
+
+
+# Package metadata.
+
+name = "google-cloud-bigquery"
+description = "Google BigQuery API client library"
+version = "1.18.0"
+# Should be one of:
+# 'Development Status :: 3 - Alpha'
+# 'Development Status :: 4 - Beta'
+# 'Development Status :: 5 - Production/Stable'
+release_status = "Development Status :: 5 - Production/Stable"
+dependencies = [
+ 'enum34; python_version < "3.4"',
+ "google-cloud-core >= 1.0.3, < 2.0dev",
+ "google-resumable-media >= 0.3.1",
+ "protobuf >= 3.6.0",
+]
+extras = {
+ "bqstorage": [
+ "google-cloud-bigquery-storage >= 0.6.0, <2.0.0dev",
+ # Bad Linux release for 0.14.0.
+ # https://issues.apache.org/jira/browse/ARROW-5868
+ "pyarrow>=0.13.0, != 0.14.0",
+ ],
+ "pandas": ["pandas>=0.17.1"],
+ # Exclude PyArrow dependency from Windows Python 2.7.
+ 'pyarrow: platform_system != "Windows" or python_version >= "3.4"': [
+ # Bad Linux release for 0.14.0.
+ # https://issues.apache.org/jira/browse/ARROW-5868
+ "pyarrow>=0.4.1, != 0.14.0"
+ ],
+ "tqdm": ["tqdm >= 4.0.0, <5.0.0dev"],
+ "fastparquet": ["fastparquet", "python-snappy"],
+}
+
+all_extras = []
+
+for extra in extras:
+ if extra == "fastparquet":
+ # Skip fastparquet from "all" because it is redundant with pyarrow and
+ # creates a dependency on pre-release versions of numpy. See:
+ # https://github.com/googleapis/google-cloud-python/issues/8549
+ continue
+ all_extras.extend(extras[extra])
+
+extras["all"] = all_extras
+
+# Setup boilerplate below this line.
+
+package_root = os.path.abspath(os.path.dirname(__file__))
+
+readme_filename = os.path.join(package_root, "README.rst")
+with io.open(readme_filename, encoding="utf-8") as readme_file:
+ readme = readme_file.read()
+
+# Only include packages under the 'google' namespace. Do not include tests,
+# benchmarks, etc.
+packages = [
+ package for package in setuptools.find_packages() if package.startswith("google")
+]
+
+# Determine which namespaces are needed.
+namespaces = ["google"]
+if "google.cloud" in packages:
+ namespaces.append("google.cloud")
+
+
+setuptools.setup(
+ name=name,
+ version=version,
+ description=description,
+ long_description=readme,
+ author="Google LLC",
+ author_email="googleapis-packages@google.com",
+ license="Apache 2.0",
+ url="https://github.com/GoogleCloudPlatform/google-cloud-python",
+ classifiers=[
+ release_status,
+ "Intended Audience :: Developers",
+ "License :: OSI Approved :: Apache Software License",
+ "Programming Language :: Python",
+ "Programming Language :: Python :: 2",
+ "Programming Language :: Python :: 2.7",
+ "Programming Language :: Python :: 3",
+ "Programming Language :: Python :: 3.5",
+ "Programming Language :: Python :: 3.6",
+ "Programming Language :: Python :: 3.7",
+ "Operating System :: OS Independent",
+ "Topic :: Internet",
+ ],
+ platforms="Posix; MacOS X; Windows",
+ packages=packages,
+ namespace_packages=namespaces,
+ install_requires=dependencies,
+ extras_require=extras,
+ python_requires=">=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*",
+ include_package_data=True,
+ zip_safe=False,
+)
diff --git a/bigquery/synth.metadata b/bigquery/synth.metadata
new file mode 100644
index 000000000000..b5cce0083f8e
--- /dev/null
+++ b/bigquery/synth.metadata
@@ -0,0 +1,32 @@
+{
+ "updateTime": "2019-08-06T12:13:25.893023Z",
+ "sources": [
+ {
+ "generator": {
+ "name": "artman",
+ "version": "0.32.1",
+ "dockerImage": "googleapis/artman@sha256:a684d40ba9a4e15946f5f2ca6b4bd9fe301192f522e9de4fff622118775f309b"
+ }
+ },
+ {
+ "git": {
+ "name": "googleapis",
+ "remote": "https://github.com/googleapis/googleapis.git",
+ "sha": "e699b0cba64ffddfae39633417180f1f65875896",
+ "internalRef": "261759677"
+ }
+ }
+ ],
+ "destinations": [
+ {
+ "client": {
+ "source": "googleapis",
+ "apiName": "bigquery",
+ "apiVersion": "v2",
+ "language": "python",
+ "generator": "gapic",
+ "config": "google/cloud/bigquery/artman_bigquery_v2.yaml"
+ }
+ }
+ ]
+}
\ No newline at end of file
diff --git a/bigquery/synth.py b/bigquery/synth.py
new file mode 100644
index 000000000000..a8370fd4e96a
--- /dev/null
+++ b/bigquery/synth.py
@@ -0,0 +1,54 @@
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""This script is used to synthesize generated parts of this library."""
+
+import synthtool as s
+from synthtool import gcp
+
+gapic = gcp.GAPICGenerator()
+
+version = 'v2'
+
+library = gapic.py_library(
+ 'bigquery',
+ version,
+ config_path='/google/cloud/bigquery/'
+ 'artman_bigquery_v2.yaml',
+ artman_output_name='bigquery-v2',
+ include_protos=True,
+)
+
+s.move(
+ [
+ library / "google/cloud/bigquery_v2/gapic/enums.py",
+ library / "google/cloud/bigquery_v2/types.py",
+ library / "google/cloud/bigquery_v2/proto/location*",
+ library / "google/cloud/bigquery_v2/proto/model*",
+ library / "google/cloud/bigquery_v2/proto/standard_sql*",
+ ],
+)
+
+# Fix up proto docs that are missing summary line.
+s.replace(
+ "google/cloud/bigquery_v2/proto/model_pb2.py",
+ '"""Attributes:',
+ '"""Protocol buffer.\n\n Attributes:',
+)
+
+# Remove non-ascii characters from docstrings for Python 2.7.
+# Format quoted strings as plain text.
+s.replace("google/cloud/bigquery_v2/proto/*.py", "[“”]", '``')
+
+s.shell.run(["nox", "-s", "blacken"], hide_output=False)
diff --git a/bigquery/tests/__init__.py b/bigquery/tests/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/bigquery/tests/data/characters.json b/bigquery/tests/data/characters.json
new file mode 100644
index 000000000000..d38636810196
--- /dev/null
+++ b/bigquery/tests/data/characters.json
@@ -0,0 +1,68 @@
+[
+ {
+ "Age" : "111",
+ "Spells" : [],
+ "Name" : "Bilbo",
+ "Weight" : 67.2,
+ "TeaTime" : "10:00:00",
+ "NextVacation" : "2017-09-22",
+ "FavoriteTime" : "2031-04-01T05:09:27",
+ "IsMagic" : false
+ },
+ {
+ "Age" : "1000",
+ "Name" : "Gandalf",
+ "Spells" : [
+ {
+ "Name" : "Skydragon",
+ "Properties" : [
+ {
+ "Power" : 1,
+ "Name" : "Flying"
+ },
+ {
+ "Name" : "Creature",
+ "Power" : 1
+ },
+ {
+ "Power" : 11,
+ "Name" : "Explodey"
+ }
+ ],
+ "LastUsed" : "2015-10-31 23:59:56 UTC",
+ "Icon" : "iVBORw0KGgoAAAANSUhEUgAAAB4AAAAgCAYAAAAFQMh/AAAAAXNSR0IArs4c6QAAA9lJREFUSA21lk9OVEEQxvsRDImoiMG9mLjjCG5mEg7gEfQGsIcF7p0EDsBBSJiNO7ZsFRZqosb/QkSj7fer7ur33sw8GDFUUq+7q6vqq6qu7pkQzqG4EeI521e7FePVgM9cGPYwhCi6UO8qFOK+YY+Br66ujsmmxb84Yzwp6zCsxjJfWVkxnMsEMGuWHZ9Wcz11cM48hkq0vLwc1tbW4mAwqDpcdIqnMmgF0JMv2CiGnZ2dcHR0FA4PD8Pe3t5U/tx6bCSlb+JT8XfxT3HsUek0Li0tRdjWl+z6iRF+FNA1hXPDQ/IMNyRg3s8bD/OaZS+VP+9cOLSa64cA34oXZWagDkRzAaJxXaE+ufc4rCN7LrazZ2+8+STtpAL8WYDvpTaHKlkB2iQARMvb2+H27m4YaL7zaDtUw1BZAASi6T8T2UZnPZV2pvnJfCH5p8bewcGB6TrIfz8wBZgHQ83kjpuj6RBYQpuo09Tvmpd7TPe+ktZN8cKwS92KWXGuaqWowlYEwthtMcWOZUNJc8at+zuF/Xkqo69baS7P+AvWjYwJ4jyHXXsEnd74ZO/Pq+uXUuv6WNlso6cvnDsZB1V/unJab3D1/KrJDw9NCM9wHf2FK2ejTKMejnBHfGtfH7LGGCdQDqaqJgfgzWjXK1nYV4jRbPGnxUT7cqUaZfJrVZeOm9QmB21L6xXgbu/ScsYusJFMoU0x2fsamRJOd6kOYDRLUxv94ENZe8+0gM+0dyz+KgU7X8rLHHCIOZyrna4y6ykIu0YCs02TBXmk3PZssmEgaTxTo83xjCIjoE21h0Yah3MrV4+9kR8MaabGze+9NEILGAFE5nMOiiA32KnAr/sb7tED3nzlzC4dB38WMC+EjaqHfqvUKHi2gJPdWQ6AbH8hgyQ7QY6jvjj3QZWvX6pUAtduTX5Dss96Q7NI9RQRJeeKvRFbt0v2gb1Gx/PooJsztn1c1DqpAU3Hde2dB2aEHBhjgOFjMeDvxLafjQ3YZQSgOcHJZX611H45sGLHWvYTz9hiURlpNoBZvxb/Ft9lAQ1DmBfUiR+j1hAPkMBTE9L9+zLva1QvGFHurRBaZ5xLVitoBviiRkD/sIMDztKA5FA0b9/0OclzO2/XAQymJ0TcghZwEo9/AX8gMeAJMOvIsWWt5bwCoiFhVSllrdH0t5Q1JHAFlKJNkvTVdn2GHb9KdmacMT+d/Os05imJUccRX2YuZ93Sxf0Ilc4DPDeAq5SAvFEAY94cQc6BA26dzb4HWAJI4DPmQE5KCVUyvb2FcDZem7JdT2ggKUP3xX6n9XNq1DpzSf4Cy4ZqSlmM8d8AAAAASUVORK5CYII=",
+ "DiscoveredBy" : "Firebreather"
+ }
+ ],
+ "NextVacation" : "2666-06-06",
+ "TeaTime" : "15:00:00",
+ "Weight" : 198.6,
+ "FavoriteTime" : "2001-12-19T23:59:59",
+ "FavoriteNumber" : "3.141592654",
+ "IsMagic" : true
+ },
+ {
+ "Weight" : 128.3,
+ "TeaTime" : "12:00:00",
+ "NextVacation" : "2017-03-14",
+ "IsMagic" : true,
+ "FavoriteTime" : "2000-10-31T23:27:46",
+ "Age" : "17",
+ "FavoriteNumber" : "13",
+ "Spells" : [
+ {
+ "LastUsed" : "2017-02-14 12:07:23 UTC",
+ "Properties" : [
+ {
+ "Name" : "Makes you look crazy",
+ "Power" : 1
+ }
+ ],
+ "Icon" : "iVBORw0KGgoAAAANSUhEUgAAAEAAAABACAYAAACqaXHeAAAAAXNSR0IArs4c6QAAABxpRE9UAAAAAgAAAAAAAAAgAAAAKAAAACAAAAAgAAABxj2CfowAAAGSSURBVHgB7Jc9TsNAEIX3JDkCPUV6KlpKFHEGlD4nyA04ACUXQKTgCEipUnKGNEbP0otentayicZ24SlWs7tjO/N9u/5J2b2+NUtuZcnwYE8BuQPyGZAPwXwLLPk5kG+BJa9+fgfkh1B+CeancL4F8i2Q/wWm/S/w+XFoTseftn0dvhu0OXfhpM+AGvzcEiYVAFisPqE9zrETJhHAlXfg2lglMK9z0f3RBfB+ZyRUV3x+erzsEIjjOBqc1xtNAIrvguybV3A9lkVHxlEE6GrrPb/ZvAySwlUnfCmlPQ+R8JCExvGtcRQBLFwj4FGkznX1VYDKPG/f2/MjwCksXACgdNUxJjwK9xwl4JihOwTFR0kIF+CABEPRnvsvPFctMoYKqAFSAFaMwB4pp3Y+bodIYL9WmIAaIOHxo7W8wiHvAjTvhUeNwwSgeAeAABbqOewC5hBdwFD4+9+7puzXV9fS6/b1wwT4tsaYAhwOOQdUQch5vgZCeAhAv3ZM31yYAAUgvApQQQ6n5w6FB/RVe1jdJOAPAAD//1eMQwoAAAGQSURBVO1UMU4DQQy8X9AgWopIUINEkS4VlJQo4gvwAV7AD3gEH4iSgidESpWSXyyZExP5lr0c7K5PsXBhec/2+jzjuWtent9CLdtu1mG5+gjz+WNr7IsY7eH+tvO+xfuqk4vz7CH91edFaF5v9nb6dBKm13edvrL+0Lk5lMzJkQDeJSkkgHF6mR8CHwMHCQR/NAQQGD0BAlwK4FCefQiefq+A2Vn29tG7igLAfmwcnJu/nJy3BMQkMN9HEPr8AL3bfBv7Bp+7/SoExMDjZwKEJwmyhnnmQIQEBIlz2x0iKoAvJkAC6TsTIH6MqRrEWUMSZF2zAwqT4Eu/e6pzFAIkmNSZ4OFT+VYBIIF//UqbJwnF/4DU0GwOn8r/JQYCpPGufEfJuZiA37ycQw/5uFeqPq4pfR6FADmkBCXjfWdZj3NfXW58dAJyB9W65wRoMWulryvAyqa05nQFaDFrpa8rwMqmtOZ0BWgxa6WvK8DKprTmdAVoMWulryvAyqa05nQFaDFrpa8rwMqmtOb89wr4AtQ4aPoL6yVpAAAAAElFTkSuQmCC",
+ "Name" : "Talking cats",
+ "DiscoveredBy" : "Salem"
+ }
+ ],
+ "Name" : "Sabrina"
+ }
+]
diff --git a/bigquery/tests/data/characters.jsonl b/bigquery/tests/data/characters.jsonl
new file mode 100644
index 000000000000..42b5bdc6a152
--- /dev/null
+++ b/bigquery/tests/data/characters.jsonl
@@ -0,0 +1,3 @@
+{"Name":"Bilbo","Age":"111","Weight":67.2,"IsMagic":false,"Spells":[],"TeaTime":"10:00:00","NextVacation":"2017-09-22","FavoriteTime":"2031-04-01T05:09:27","FavoriteNumber":"111"}
+{"Name":"Gandalf","Age":"1000","Weight":198.6,"IsMagic":true,"Spells":[{"Name": "Skydragon", "Icon":"iVBORw0KGgoAAAANSUhEUgAAAB4AAAAgCAYAAAAFQMh/AAAAAXNSR0IArs4c6QAAA9lJREFUSA21lk9OVEEQxvsRDImoiMG9mLjjCG5mEg7gEfQGsIcF7p0EDsBBSJiNO7ZsFRZqosb/QkSj7fer7ur33sw8GDFUUq+7q6vqq6qu7pkQzqG4EeI521e7FePVgM9cGPYwhCi6UO8qFOK+YY+Br66ujsmmxb84Yzwp6zCsxjJfWVkxnMsEMGuWHZ9Wcz11cM48hkq0vLwc1tbW4mAwqDpcdIqnMmgF0JMv2CiGnZ2dcHR0FA4PD8Pe3t5U/tx6bCSlb+JT8XfxT3HsUek0Li0tRdjWl+z6iRF+FNA1hXPDQ/IMNyRg3s8bD/OaZS+VP+9cOLSa64cA34oXZWagDkRzAaJxXaE+ufc4rCN7LrazZ2+8+STtpAL8WYDvpTaHKlkB2iQARMvb2+H27m4YaL7zaDtUw1BZAASi6T8T2UZnPZV2pvnJfCH5p8bewcGB6TrIfz8wBZgHQ83kjpuj6RBYQpuo09Tvmpd7TPe+ktZN8cKwS92KWXGuaqWowlYEwthtMcWOZUNJc8at+zuF/Xkqo69baS7P+AvWjYwJ4jyHXXsEnd74ZO/Pq+uXUuv6WNlso6cvnDsZB1V/unJab3D1/KrJDw9NCM9wHf2FK2ejTKMejnBHfGtfH7LGGCdQDqaqJgfgzWjXK1nYV4jRbPGnxUT7cqUaZfJrVZeOm9QmB21L6xXgbu/ScsYusJFMoU0x2fsamRJOd6kOYDRLUxv94ENZe8+0gM+0dyz+KgU7X8rLHHCIOZyrna4y6ykIu0YCs02TBXmk3PZssmEgaTxTo83xjCIjoE21h0Yah3MrV4+9kR8MaabGze+9NEILGAFE5nMOiiA32KnAr/sb7tED3nzlzC4dB38WMC+EjaqHfqvUKHi2gJPdWQ6AbH8hgyQ7QY6jvjj3QZWvX6pUAtduTX5Dss96Q7NI9RQRJeeKvRFbt0v2gb1Gx/PooJsztn1c1DqpAU3Hde2dB2aEHBhjgOFjMeDvxLafjQ3YZQSgOcHJZX611H45sGLHWvYTz9hiURlpNoBZvxb/Ft9lAQ1DmBfUiR+j1hAPkMBTE9L9+zLva1QvGFHurRBaZ5xLVitoBviiRkD/sIMDztKA5FA0b9/0OclzO2/XAQymJ0TcghZwEo9/AX8gMeAJMOvIsWWt5bwCoiFhVSllrdH0t5Q1JHAFlKJNkvTVdn2GHb9KdmacMT+d/Os05imJUccRX2YuZ93Sxf0Ilc4DPDeAq5SAvFEAY94cQc6BA26dzb4HWAJI4DPmQE5KCVUyvb2FcDZem7JdT2ggKUP3xX6n9XNq1DpzSf4Cy4ZqSlmM8d8AAAAASUVORK5CYII=","DiscoveredBy":"Firebreather","Properties":[{"Name":"Flying","Power":1},{"Name":"Creature","Power":1},{"Name":"Explodey","Power":11}],"LastUsed":"2015-10-31 23:59:56 UTC"}],"TeaTime":"15:00:00","NextVacation":"2666-06-06","FavoriteTime":"2001-12-19T23:59:59","FavoriteNumber":"1.618033989"}
+{"Name":"Sabrina","Age":"17","Weight":128.3,"IsMagic":true,"Spells":[{"Name": "Talking cats", "Icon":"iVBORw0KGgoAAAANSUhEUgAAAEAAAABACAYAAACqaXHeAAAAAXNSR0IArs4c6QAAABxpRE9UAAAAAgAAAAAAAAAgAAAAKAAAACAAAAAgAAABxj2CfowAAAGSSURBVHgB7Jc9TsNAEIX3JDkCPUV6KlpKFHEGlD4nyA04ACUXQKTgCEipUnKGNEbP0otentayicZ24SlWs7tjO/N9u/5J2b2+NUtuZcnwYE8BuQPyGZAPwXwLLPk5kG+BJa9+fgfkh1B+CeancL4F8i2Q/wWm/S/w+XFoTseftn0dvhu0OXfhpM+AGvzcEiYVAFisPqE9zrETJhHAlXfg2lglMK9z0f3RBfB+ZyRUV3x+erzsEIjjOBqc1xtNAIrvguybV3A9lkVHxlEE6GrrPb/ZvAySwlUnfCmlPQ+R8JCExvGtcRQBLFwj4FGkznX1VYDKPG/f2/MjwCksXACgdNUxJjwK9xwl4JihOwTFR0kIF+CABEPRnvsvPFctMoYKqAFSAFaMwB4pp3Y+bodIYL9WmIAaIOHxo7W8wiHvAjTvhUeNwwSgeAeAABbqOewC5hBdwFD4+9+7puzXV9fS6/b1wwT4tsaYAhwOOQdUQch5vgZCeAhAv3ZM31yYAAUgvApQQQ6n5w6FB/RVe1jdJOAPAAD//1eMQwoAAAGQSURBVO1UMU4DQQy8X9AgWopIUINEkS4VlJQo4gvwAV7AD3gEH4iSgidESpWSXyyZExP5lr0c7K5PsXBhec/2+jzjuWtent9CLdtu1mG5+gjz+WNr7IsY7eH+tvO+xfuqk4vz7CH91edFaF5v9nb6dBKm13edvrL+0Lk5lMzJkQDeJSkkgHF6mR8CHwMHCQR/NAQQGD0BAlwK4FCefQiefq+A2Vn29tG7igLAfmwcnJu/nJy3BMQkMN9HEPr8AL3bfBv7Bp+7/SoExMDjZwKEJwmyhnnmQIQEBIlz2x0iKoAvJkAC6TsTIH6MqRrEWUMSZF2zAwqT4Eu/e6pzFAIkmNSZ4OFT+VYBIIF//UqbJwnF/4DU0GwOn8r/JQYCpPGufEfJuZiA37ycQw/5uFeqPq4pfR6FADmkBCXjfWdZj3NfXW58dAJyB9W65wRoMWulryvAyqa05nQFaDFrpa8rwMqmtOZ0BWgxa6WvK8DKprTmdAVoMWulryvAyqa05nQFaDFrpa8rwMqmtOb89wr4AtQ4aPoL6yVpAAAAAElFTkSuQmCC","DiscoveredBy":"Salem","Properties":[{"Name":"Makes you look crazy","Power":1}],"LastUsed":"2017-02-14 12:07:23 UTC"}],"TeaTime":"12:00:00","NextVacation":"2017-03-14","FavoriteTime":"2000-10-31T23:27:46","FavoriteNumber":"13"}
diff --git a/bigquery/tests/data/colors.avro b/bigquery/tests/data/colors.avro
new file mode 100644
index 000000000000..e0133fd027f4
Binary files /dev/null and b/bigquery/tests/data/colors.avro differ
diff --git a/bigquery/tests/data/people.csv b/bigquery/tests/data/people.csv
new file mode 100644
index 000000000000..d3c7d063892a
--- /dev/null
+++ b/bigquery/tests/data/people.csv
@@ -0,0 +1,3 @@
+full_name,age
+Phred Phlyntstone,32
+Wylma Phlyntstone,29
\ No newline at end of file
diff --git a/bigquery/tests/data/schema.json b/bigquery/tests/data/schema.json
new file mode 100644
index 000000000000..6a36e55e579a
--- /dev/null
+++ b/bigquery/tests/data/schema.json
@@ -0,0 +1,88 @@
+{
+ "fields" : [
+ {
+ "type" : "STRING",
+ "name" : "Name",
+ "mode" : "NULLABLE"
+ },
+ {
+ "name" : "Age",
+ "mode" : "NULLABLE",
+ "type" : "INTEGER"
+ },
+ {
+ "type" : "FLOAT",
+ "name" : "Weight",
+ "mode" : "NULLABLE"
+ },
+ {
+ "mode" : "NULLABLE",
+ "name" : "IsMagic",
+ "type" : "BOOLEAN"
+ },
+ {
+ "name" : "Spells",
+ "fields" : [
+ {
+ "mode" : "NULLABLE",
+ "name" : "Name",
+ "type" : "STRING"
+ },
+ {
+ "mode" : "NULLABLE",
+ "name" : "LastUsed",
+ "type" : "TIMESTAMP"
+ },
+ {
+ "type" : "STRING",
+ "mode" : "NULLABLE",
+ "name" : "DiscoveredBy"
+ },
+ {
+ "name" : "Properties",
+ "fields" : [
+ {
+ "name" : "Name",
+ "mode" : "NULLABLE",
+ "type" : "STRING"
+ },
+ {
+ "type" : "FLOAT",
+ "name" : "Power",
+ "mode" : "NULLABLE"
+ }
+ ],
+ "mode" : "REPEATED",
+ "type" : "RECORD"
+ },
+ {
+ "mode" : "NULLABLE",
+ "name" : "Icon",
+ "type" : "BYTES"
+ }
+ ],
+ "mode" : "REPEATED",
+ "type" : "RECORD"
+ },
+ {
+ "type" : "TIME",
+ "mode" : "NULLABLE",
+ "name" : "TeaTime"
+ },
+ {
+ "type" : "DATE",
+ "name" : "NextVacation",
+ "mode" : "NULLABLE"
+ },
+ {
+ "mode" : "NULLABLE",
+ "name" : "FavoriteTime",
+ "type" : "DATETIME"
+ },
+ {
+ "mode" : "NULLABLE",
+ "name" : "FavoriteNumber",
+ "type" : "NUMERIC"
+ }
+ ]
+}
diff --git a/bigquery/tests/scrub_datasets.py b/bigquery/tests/scrub_datasets.py
new file mode 100644
index 000000000000..9a8ab3e7b7c5
--- /dev/null
+++ b/bigquery/tests/scrub_datasets.py
@@ -0,0 +1,25 @@
+import re
+import sys
+
+from google.api_core.exceptions import NotFound
+from google.cloud.bigquery import Client
+
+
+def main(prefixes):
+ client = Client()
+
+ pattern = re.compile("|".join("^{}.*$".format(prefix) for prefix in prefixes))
+
+ ds_items = list(client.list_datasets())
+ for dataset in ds_items:
+ ds_id = dataset.dataset_id
+ if pattern.match(ds_id):
+ print("Deleting dataset: {}".format(ds_id))
+ try:
+ client.delete_dataset(dataset.reference, delete_contents=True)
+ except NotFound:
+ print(" NOT FOUND")
+
+
+if __name__ == "__main__":
+ main(sys.argv[1:])
diff --git a/bigquery/tests/system.py b/bigquery/tests/system.py
new file mode 100644
index 000000000000..3593e1ecb609
--- /dev/null
+++ b/bigquery/tests/system.py
@@ -0,0 +1,2394 @@
+# Copyright 2015 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import base64
+import collections
+import concurrent.futures
+import csv
+import datetime
+import decimal
+import json
+import operator
+import os
+import time
+import unittest
+import uuid
+import re
+
+import six
+import pytest
+import pytz
+
+try:
+ from google.cloud import bigquery_storage_v1beta1
+except ImportError: # pragma: NO COVER
+ bigquery_storage_v1beta1 = None
+try:
+ import pandas
+except ImportError: # pragma: NO COVER
+ pandas = None
+try:
+ import pyarrow
+ import pyarrow.types
+except ImportError: # pragma: NO COVER
+ pyarrow = None
+try:
+ import IPython
+ from IPython.utils import io
+ from IPython.testing import tools
+ from IPython.terminal import interactiveshell
+except ImportError: # pragma: NO COVER
+ IPython = None
+
+from google.api_core.exceptions import PreconditionFailed
+from google.api_core.exceptions import BadRequest
+from google.api_core.exceptions import Conflict
+from google.api_core.exceptions import Forbidden
+from google.api_core.exceptions import GoogleAPICallError
+from google.api_core.exceptions import NotFound
+from google.api_core.exceptions import InternalServerError
+from google.api_core.exceptions import ServiceUnavailable
+from google.api_core.exceptions import TooManyRequests
+from google.cloud import bigquery
+from google.cloud import bigquery_v2
+from google.cloud.bigquery.dataset import Dataset
+from google.cloud.bigquery.dataset import DatasetReference
+from google.cloud.bigquery.table import Table
+from google.cloud._helpers import UTC
+from google.cloud.bigquery import dbapi
+from google.cloud import storage
+
+from test_utils.retry import RetryErrors
+from test_utils.retry import RetryInstanceState
+from test_utils.retry import RetryResult
+from test_utils.system import unique_resource_id
+
+
+JOB_TIMEOUT = 120 # 2 minutes
+WHERE = os.path.abspath(os.path.dirname(__file__))
+
+# Common table data used for many tests.
+ROWS = [
+ ("Phred Phlyntstone", 32),
+ ("Bharney Rhubble", 33),
+ ("Wylma Phlyntstone", 29),
+ ("Bhettye Rhubble", 27),
+]
+HEADER_ROW = ("Full Name", "Age")
+SCHEMA = [
+ bigquery.SchemaField("full_name", "STRING", mode="REQUIRED"),
+ bigquery.SchemaField("age", "INTEGER", mode="REQUIRED"),
+]
+TIME_PARTITIONING_CLUSTERING_FIELDS_SCHEMA = [
+ bigquery.SchemaField("transaction_time", "TIMESTAMP", mode="REQUIRED"),
+ bigquery.SchemaField("transaction_id", "INTEGER", mode="REQUIRED"),
+ bigquery.SchemaField("user_email", "STRING", mode="REQUIRED"),
+ bigquery.SchemaField("store_code", "STRING", mode="REQUIRED"),
+ bigquery.SchemaField(
+ "items",
+ "RECORD",
+ mode="REPEATED",
+ fields=[
+ bigquery.SchemaField("item_code", "STRING", mode="REQUIRED"),
+ bigquery.SchemaField("quantity", "INTEGER", mode="REQUIRED"),
+ bigquery.SchemaField("comments", "STRING", mode="NULLABLE"),
+ bigquery.SchemaField("expiration_date", "DATE", mode="REQUIRED"),
+ ],
+ ),
+]
+
+# The VPC-SC team maintains a mirror of the GCS bucket used for code
+# samples. The public bucket crosses the configured security boundary.
+# See: https://github.com/googleapis/google-cloud-python/issues/8550
+SAMPLES_BUCKET = os.environ.get("GCLOUD_TEST_SAMPLES_BUCKET", "cloud-samples-data")
+
+retry_storage_errors = RetryErrors(
+ (TooManyRequests, InternalServerError, ServiceUnavailable)
+)
+
+
+def _has_rows(result):
+ return len(result) > 0
+
+
+def _make_dataset_id(prefix):
+ return "%s%s" % (prefix, unique_resource_id())
+
+
+def _load_json_schema(filename="data/schema.json"):
+ from google.cloud.bigquery.table import _parse_schema_resource
+
+ json_filename = os.path.join(WHERE, filename)
+
+ with open(json_filename, "r") as schema_file:
+ return _parse_schema_resource(json.load(schema_file))
+
+
+def _rate_limit_exceeded(forbidden):
+ """Predicate: pass only exceptions with 'rateLimitExceeded' as reason."""
+ return any(error["reason"] == "rateLimitExceeded" for error in forbidden._errors)
+
+
+# We need to wait to stay within the rate limits.
+# The alternative outcome is a 403 Forbidden response from upstream, which
+# they return instead of the more appropriate 429.
+# See https://cloud.google.com/bigquery/quota-policy
+retry_403 = RetryErrors(Forbidden, error_predicate=_rate_limit_exceeded)
+
+
+class Config(object):
+ """Run-time configuration to be modified at set-up.
+
+ This is a mutable stand-in to allow test set-up to modify
+ global state.
+ """
+
+ CLIENT = None
+ CURSOR = None
+
+
+def setUpModule():
+ Config.CLIENT = bigquery.Client()
+ Config.CURSOR = dbapi.connect(Config.CLIENT).cursor()
+
+
+class TestBigQuery(unittest.TestCase):
+ def setUp(self):
+ self.to_delete = []
+
+ def tearDown(self):
+ def _still_in_use(bad_request):
+ return any(
+ error["reason"] == "resourceInUse" for error in bad_request._errors
+ )
+
+ retry_in_use = RetryErrors(BadRequest, error_predicate=_still_in_use)
+ retry_storage_errors_conflict = RetryErrors(
+ (Conflict, TooManyRequests, InternalServerError, ServiceUnavailable)
+ )
+ for doomed in self.to_delete:
+ if isinstance(doomed, storage.Bucket):
+ retry_storage_errors_conflict(doomed.delete)(force=True)
+ elif isinstance(doomed, (Dataset, bigquery.DatasetReference)):
+ retry_in_use(Config.CLIENT.delete_dataset)(doomed, delete_contents=True)
+ elif isinstance(doomed, (Table, bigquery.TableReference)):
+ retry_in_use(Config.CLIENT.delete_table)(doomed)
+ else:
+ doomed.delete()
+
+ def test_get_service_account_email(self):
+ client = Config.CLIENT
+
+ got = client.get_service_account_email()
+
+ self.assertIsInstance(got, six.text_type)
+ self.assertIn("@", got)
+
+ def _create_bucket(self, bucket_name, location=None):
+ storage_client = storage.Client()
+ bucket = storage_client.bucket(bucket_name)
+ retry_storage_errors(bucket.create)(location=location)
+ self.to_delete.append(bucket)
+
+ return bucket
+
+ def test_create_dataset(self):
+ DATASET_ID = _make_dataset_id("create_dataset")
+ dataset = self.temp_dataset(DATASET_ID)
+
+ self.assertTrue(_dataset_exists(dataset))
+ self.assertEqual(dataset.dataset_id, DATASET_ID)
+ self.assertEqual(dataset.project, Config.CLIENT.project)
+
+ def test_get_dataset(self):
+ dataset_id = _make_dataset_id("get_dataset")
+ client = Config.CLIENT
+ dataset_arg = Dataset(client.dataset(dataset_id))
+ dataset_arg.friendly_name = "Friendly"
+ dataset_arg.description = "Description"
+ dataset = retry_403(client.create_dataset)(dataset_arg)
+ self.to_delete.append(dataset)
+ dataset_ref = client.dataset(dataset_id)
+
+ # Get with a reference.
+ got = client.get_dataset(dataset_ref)
+ self.assertEqual(got.friendly_name, "Friendly")
+ self.assertEqual(got.description, "Description")
+
+ # Get with a string.
+ got = client.get_dataset(dataset_id)
+ self.assertEqual(got.friendly_name, "Friendly")
+ self.assertEqual(got.description, "Description")
+
+ # Get with a fully-qualified string.
+ got = client.get_dataset("{}.{}".format(client.project, dataset_id))
+ self.assertEqual(got.friendly_name, "Friendly")
+ self.assertEqual(got.description, "Description")
+
+ def test_update_dataset(self):
+ dataset = self.temp_dataset(_make_dataset_id("update_dataset"))
+ self.assertTrue(_dataset_exists(dataset))
+ self.assertIsNone(dataset.friendly_name)
+ self.assertIsNone(dataset.description)
+ self.assertEqual(dataset.labels, {})
+
+ dataset.friendly_name = "Friendly"
+ dataset.description = "Description"
+ dataset.labels = {"priority": "high", "color": "blue"}
+ ds2 = Config.CLIENT.update_dataset(
+ dataset, ("friendly_name", "description", "labels")
+ )
+ self.assertEqual(ds2.friendly_name, "Friendly")
+ self.assertEqual(ds2.description, "Description")
+ self.assertEqual(ds2.labels, {"priority": "high", "color": "blue"})
+
+ ds2.labels = {
+ "color": "green", # change
+ "shape": "circle", # add
+ "priority": None, # delete
+ }
+ ds3 = Config.CLIENT.update_dataset(ds2, ["labels"])
+ self.assertEqual(ds3.labels, {"color": "green", "shape": "circle"})
+
+ # If we try to update using d2 again, it will fail because the
+ # previous update changed the ETag.
+ ds2.description = "no good"
+ with self.assertRaises(PreconditionFailed):
+ Config.CLIENT.update_dataset(ds2, ["description"])
+
+ def test_list_datasets(self):
+ datasets_to_create = [
+ "new" + unique_resource_id(),
+ "newer" + unique_resource_id(),
+ "newest" + unique_resource_id(),
+ ]
+ for dataset_id in datasets_to_create:
+ self.temp_dataset(dataset_id)
+
+ # Retrieve the datasets.
+ iterator = Config.CLIENT.list_datasets()
+ all_datasets = list(iterator)
+ self.assertIsNone(iterator.next_page_token)
+ created = [
+ dataset
+ for dataset in all_datasets
+ if dataset.dataset_id in datasets_to_create
+ and dataset.project == Config.CLIENT.project
+ ]
+ self.assertEqual(len(created), len(datasets_to_create))
+
+ def test_list_datasets_w_project(self):
+ # Retrieve datasets from a different project.
+ iterator = Config.CLIENT.list_datasets(project="bigquery-public-data")
+ all_datasets = frozenset([dataset.dataset_id for dataset in iterator])
+ self.assertIn("usa_names", all_datasets)
+
+ def test_create_table(self):
+ dataset = self.temp_dataset(_make_dataset_id("create_table"))
+ table_id = "test_table"
+ table_arg = Table(dataset.table(table_id), schema=SCHEMA)
+ self.assertFalse(_table_exists(table_arg))
+
+ table = retry_403(Config.CLIENT.create_table)(table_arg)
+ self.to_delete.insert(0, table)
+
+ self.assertTrue(_table_exists(table))
+ self.assertEqual(table.table_id, table_id)
+
+ def test_create_table_w_time_partitioning_w_clustering_fields(self):
+ from google.cloud.bigquery.table import TimePartitioning
+ from google.cloud.bigquery.table import TimePartitioningType
+
+ dataset = self.temp_dataset(_make_dataset_id("create_table_tp_cf"))
+ table_id = "test_table"
+ table_arg = Table(
+ dataset.table(table_id), schema=TIME_PARTITIONING_CLUSTERING_FIELDS_SCHEMA
+ )
+ self.assertFalse(_table_exists(table_arg))
+
+ table_arg.time_partitioning = TimePartitioning(field="transaction_time")
+
+ table_arg.clustering_fields = ["user_email", "store_code"]
+ table = retry_403(Config.CLIENT.create_table)(table_arg)
+ self.to_delete.insert(0, table)
+
+ self.assertTrue(_table_exists(table))
+ self.assertEqual(table.table_id, table_id)
+ time_partitioning = table.time_partitioning
+ self.assertEqual(time_partitioning.type_, TimePartitioningType.DAY)
+ self.assertEqual(time_partitioning.field, "transaction_time")
+ self.assertEqual(table.clustering_fields, ["user_email", "store_code"])
+
+ def test_delete_dataset_with_string(self):
+ dataset_id = _make_dataset_id("delete_table_true")
+ dataset_ref = Config.CLIENT.dataset(dataset_id)
+ retry_403(Config.CLIENT.create_dataset)(Dataset(dataset_ref))
+ self.assertTrue(_dataset_exists(dataset_ref))
+ Config.CLIENT.delete_dataset(dataset_id)
+ self.assertFalse(_dataset_exists(dataset_ref))
+
+ def test_delete_dataset_delete_contents_true(self):
+ dataset_id = _make_dataset_id("delete_table_true")
+ dataset = retry_403(Config.CLIENT.create_dataset)(
+ Dataset(Config.CLIENT.dataset(dataset_id))
+ )
+
+ table_id = "test_table"
+ table_arg = Table(dataset.table(table_id), schema=SCHEMA)
+ table = retry_403(Config.CLIENT.create_table)(table_arg)
+ Config.CLIENT.delete_dataset(dataset, delete_contents=True)
+
+ self.assertFalse(_table_exists(table))
+
+ def test_delete_dataset_delete_contents_false(self):
+ from google.api_core import exceptions
+
+ dataset = self.temp_dataset(_make_dataset_id("delete_table_false"))
+ table_id = "test_table"
+ table_arg = Table(dataset.table(table_id), schema=SCHEMA)
+
+ retry_403(Config.CLIENT.create_table)(table_arg)
+ with self.assertRaises(exceptions.BadRequest):
+ Config.CLIENT.delete_dataset(dataset)
+
+ def test_get_table_w_public_dataset(self):
+ public = "bigquery-public-data"
+ dataset_id = "samples"
+ table_id = "shakespeare"
+ table_ref = DatasetReference(public, dataset_id).table(table_id)
+
+ # Get table with reference.
+ table = Config.CLIENT.get_table(table_ref)
+ self.assertEqual(table.table_id, table_id)
+ self.assertEqual(table.dataset_id, dataset_id)
+ self.assertEqual(table.project, public)
+ schema_names = [field.name for field in table.schema]
+ self.assertEqual(schema_names, ["word", "word_count", "corpus", "corpus_date"])
+
+ # Get table with string.
+ table = Config.CLIENT.get_table("{}.{}.{}".format(public, dataset_id, table_id))
+ self.assertEqual(table.table_id, table_id)
+ self.assertEqual(table.dataset_id, dataset_id)
+ self.assertEqual(table.project, public)
+
+ def test_list_partitions(self):
+ table_ref = DatasetReference(
+ "bigquery-public-data", "ethereum_blockchain"
+ ).table("blocks")
+ all_rows = Config.CLIENT.list_partitions(table_ref)
+ self.assertIn("20180801", all_rows)
+ self.assertGreater(len(all_rows), 1000)
+
+ def test_list_tables(self):
+ dataset_id = _make_dataset_id("list_tables")
+ dataset = self.temp_dataset(dataset_id)
+ # Retrieve tables before any are created for the dataset.
+ iterator = Config.CLIENT.list_tables(dataset)
+ all_tables = list(iterator)
+ self.assertEqual(all_tables, [])
+ self.assertIsNone(iterator.next_page_token)
+
+ # Insert some tables to be listed.
+ tables_to_create = [
+ "new" + unique_resource_id(),
+ "newer" + unique_resource_id(),
+ "newest" + unique_resource_id(),
+ ]
+ for table_name in tables_to_create:
+ table = Table(dataset.table(table_name), schema=SCHEMA)
+ created_table = retry_403(Config.CLIENT.create_table)(table)
+ self.to_delete.insert(0, created_table)
+
+ # Retrieve the tables.
+ iterator = Config.CLIENT.list_tables(dataset)
+ all_tables = list(iterator)
+ self.assertIsNone(iterator.next_page_token)
+ created = [
+ table
+ for table in all_tables
+ if (table.table_id in tables_to_create and table.dataset_id == dataset_id)
+ ]
+ self.assertEqual(len(created), len(tables_to_create))
+
+ # List tables with a string ID.
+ iterator = Config.CLIENT.list_tables(dataset_id)
+ self.assertGreater(len(list(iterator)), 0)
+
+ # List tables with a fully-qualified string ID.
+ iterator = Config.CLIENT.list_tables(
+ "{}.{}".format(Config.CLIENT.project, dataset_id)
+ )
+ self.assertGreater(len(list(iterator)), 0)
+
+ def test_update_table(self):
+ dataset = self.temp_dataset(_make_dataset_id("update_table"))
+
+ TABLE_NAME = "test_table"
+ table_arg = Table(dataset.table(TABLE_NAME), schema=SCHEMA)
+ self.assertFalse(_table_exists(table_arg))
+ table = retry_403(Config.CLIENT.create_table)(table_arg)
+ self.to_delete.insert(0, table)
+ self.assertTrue(_table_exists(table))
+ self.assertIsNone(table.friendly_name)
+ self.assertIsNone(table.description)
+ self.assertEqual(table.labels, {})
+ table.friendly_name = "Friendly"
+ table.description = "Description"
+ table.labels = {"priority": "high", "color": "blue"}
+
+ table2 = Config.CLIENT.update_table(
+ table, ["friendly_name", "description", "labels"]
+ )
+
+ self.assertEqual(table2.friendly_name, "Friendly")
+ self.assertEqual(table2.description, "Description")
+ self.assertEqual(table2.labels, {"priority": "high", "color": "blue"})
+
+ table2.description = None
+ table2.labels = {
+ "color": "green", # change
+ "shape": "circle", # add
+ "priority": None, # delete
+ }
+ table3 = Config.CLIENT.update_table(table2, ["description", "labels"])
+ self.assertIsNone(table3.description)
+ self.assertEqual(table3.labels, {"color": "green", "shape": "circle"})
+
+ # If we try to update using table2 again, it will fail because the
+ # previous update changed the ETag.
+ table2.description = "no good"
+ with self.assertRaises(PreconditionFailed):
+ Config.CLIENT.update_table(table2, ["description"])
+
+ def test_update_table_schema(self):
+ dataset = self.temp_dataset(_make_dataset_id("update_table"))
+
+ TABLE_NAME = "test_table"
+ table_arg = Table(dataset.table(TABLE_NAME), schema=SCHEMA)
+ self.assertFalse(_table_exists(table_arg))
+ table = retry_403(Config.CLIENT.create_table)(table_arg)
+ self.to_delete.insert(0, table)
+ self.assertTrue(_table_exists(table))
+ voter = bigquery.SchemaField("voter", "BOOLEAN", mode="NULLABLE")
+ schema = table.schema
+ schema.append(voter)
+ table.schema = schema
+
+ updated_table = Config.CLIENT.update_table(table, ["schema"])
+
+ self.assertEqual(len(updated_table.schema), len(schema))
+ for found, expected in zip(updated_table.schema, schema):
+ self.assertEqual(found.name, expected.name)
+ self.assertEqual(found.field_type, expected.field_type)
+ self.assertEqual(found.mode, expected.mode)
+
+ @staticmethod
+ def _fetch_single_page(table, selected_fields=None):
+ iterator = Config.CLIENT.list_rows(table, selected_fields=selected_fields)
+ page = six.next(iterator.pages)
+ return list(page)
+
+ def _create_table_many_columns(self, rowcount):
+ # Generate a table of maximum width via CREATE TABLE AS SELECT.
+ # first column is named 'rowval', and has a value from 1..rowcount
+ # Subsequent column is named col_ and contains the value N*rowval,
+ # where N is between 1 and 9999 inclusive.
+ dsname = _make_dataset_id("wide_schema")
+ dataset = self.temp_dataset(dsname)
+ table_id = "many_columns"
+ table_ref = dataset.table(table_id)
+ self.to_delete.insert(0, table_ref)
+ colprojections = ",".join(
+ ["r * {} as col_{}".format(n, n) for n in range(1, 10000)]
+ )
+ sql = """
+ CREATE TABLE {}.{}
+ AS
+ SELECT
+ r as rowval,
+ {}
+ FROM
+ UNNEST(GENERATE_ARRAY(1,{},1)) as r
+ """.format(
+ dsname, table_id, colprojections, rowcount
+ )
+ query_job = Config.CLIENT.query(sql)
+ query_job.result()
+ self.assertEqual(query_job.statement_type, "CREATE_TABLE_AS_SELECT")
+ self.assertEqual(query_job.ddl_operation_performed, "CREATE")
+ self.assertEqual(query_job.ddl_target_table, table_ref)
+
+ return table_ref
+
+ def test_query_many_columns(self):
+ # Test working with the widest schema BigQuery supports, 10k columns.
+ row_count = 2
+ table_ref = self._create_table_many_columns(row_count)
+ rows = list(
+ Config.CLIENT.query(
+ "SELECT * FROM `{}.{}`".format(table_ref.dataset_id, table_ref.table_id)
+ )
+ )
+
+ self.assertEqual(len(rows), row_count)
+
+ # check field representations adhere to expected values.
+ correctwidth = 0
+ badvals = 0
+ for r in rows:
+ vals = r._xxx_values
+ rowval = vals[0]
+ if len(vals) == 10000:
+ correctwidth = correctwidth + 1
+ for n in range(1, 10000):
+ if vals[n] != rowval * (n):
+ badvals = badvals + 1
+ self.assertEqual(correctwidth, row_count)
+ self.assertEqual(badvals, 0)
+
+ def test_insert_rows_then_dump_table(self):
+ NOW_SECONDS = 1448911495.484366
+ NOW = datetime.datetime.utcfromtimestamp(NOW_SECONDS).replace(tzinfo=UTC)
+ ROWS = [
+ ("Phred Phlyntstone", 32, NOW),
+ ("Bharney Rhubble", 33, NOW + datetime.timedelta(seconds=10)),
+ ("Wylma Phlyntstone", 29, NOW + datetime.timedelta(seconds=20)),
+ ("Bhettye Rhubble", 27, None),
+ ]
+ ROW_IDS = range(len(ROWS))
+
+ dataset = self.temp_dataset(_make_dataset_id("insert_rows_then_dump"))
+ TABLE_ID = "test_table"
+ schema = [
+ bigquery.SchemaField("full_name", "STRING", mode="REQUIRED"),
+ bigquery.SchemaField("age", "INTEGER", mode="REQUIRED"),
+ bigquery.SchemaField("now", "TIMESTAMP"),
+ ]
+ table_arg = Table(dataset.table(TABLE_ID), schema=schema)
+ self.assertFalse(_table_exists(table_arg))
+ table = retry_403(Config.CLIENT.create_table)(table_arg)
+ self.to_delete.insert(0, table)
+ self.assertTrue(_table_exists(table))
+
+ errors = Config.CLIENT.insert_rows(table, ROWS, row_ids=ROW_IDS)
+ self.assertEqual(len(errors), 0)
+
+ rows = ()
+
+ # Allow for "warm up" before rows visible. See
+ # https://cloud.google.com/bigquery/streaming-data-into-bigquery#dataavailability
+ # 8 tries -> 1 + 2 + 4 + 8 + 16 + 32 + 64 = 127 seconds
+ retry = RetryResult(_has_rows, max_tries=8)
+ rows = retry(self._fetch_single_page)(table)
+ row_tuples = [r.values() for r in rows]
+ by_age = operator.itemgetter(1)
+ self.assertEqual(sorted(row_tuples, key=by_age), sorted(ROWS, key=by_age))
+
+ def test_load_table_from_local_avro_file_then_dump_table(self):
+ from google.cloud.bigquery.job import SourceFormat
+ from google.cloud.bigquery.job import WriteDisposition
+
+ TABLE_NAME = "test_table_avro"
+ ROWS = [
+ ("violet", 400),
+ ("indigo", 445),
+ ("blue", 475),
+ ("green", 510),
+ ("yellow", 570),
+ ("orange", 590),
+ ("red", 650),
+ ]
+
+ dataset = self.temp_dataset(_make_dataset_id("load_local_then_dump"))
+ table_ref = dataset.table(TABLE_NAME)
+ table = Table(table_ref)
+ self.to_delete.insert(0, table)
+
+ with open(os.path.join(WHERE, "data", "colors.avro"), "rb") as avrof:
+ config = bigquery.LoadJobConfig()
+ config.source_format = SourceFormat.AVRO
+ config.write_disposition = WriteDisposition.WRITE_TRUNCATE
+ job = Config.CLIENT.load_table_from_file(
+ avrof, table_ref, job_config=config
+ )
+ # Retry until done.
+ job.result(timeout=JOB_TIMEOUT)
+
+ self.assertEqual(job.output_rows, len(ROWS))
+
+ table = Config.CLIENT.get_table(table)
+ rows = self._fetch_single_page(table)
+ row_tuples = [r.values() for r in rows]
+ by_wavelength = operator.itemgetter(1)
+ self.assertEqual(
+ sorted(row_tuples, key=by_wavelength), sorted(ROWS, key=by_wavelength)
+ )
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ def test_load_table_from_dataframe_w_automatic_schema(self):
+ """Test that a DataFrame with dtypes that map well to BigQuery types
+ can be uploaded without specifying a schema.
+
+ https://github.com/googleapis/google-cloud-python/issues/9044
+ """
+ df_data = collections.OrderedDict(
+ [
+ ("bool_col", pandas.Series([True, False, True], dtype="bool")),
+ (
+ "ts_col",
+ pandas.Series(
+ [
+ datetime.datetime(2010, 1, 2, 3, 44, 50),
+ datetime.datetime(2011, 2, 3, 14, 50, 59),
+ datetime.datetime(2012, 3, 14, 15, 16),
+ ],
+ dtype="datetime64[ns]",
+ ).dt.tz_localize(pytz.utc),
+ ),
+ (
+ "dt_col",
+ pandas.Series(
+ [
+ datetime.datetime(2010, 1, 2, 3, 44, 50),
+ datetime.datetime(2011, 2, 3, 14, 50, 59),
+ datetime.datetime(2012, 3, 14, 15, 16),
+ ],
+ dtype="datetime64[ns]",
+ ),
+ ),
+ ("float32_col", pandas.Series([1.0, 2.0, 3.0], dtype="float32")),
+ ("float64_col", pandas.Series([4.0, 5.0, 6.0], dtype="float64")),
+ ("int8_col", pandas.Series([-12, -11, -10], dtype="int8")),
+ ("int16_col", pandas.Series([-9, -8, -7], dtype="int16")),
+ ("int32_col", pandas.Series([-6, -5, -4], dtype="int32")),
+ ("int64_col", pandas.Series([-3, -2, -1], dtype="int64")),
+ ("uint8_col", pandas.Series([0, 1, 2], dtype="uint8")),
+ ("uint16_col", pandas.Series([3, 4, 5], dtype="uint16")),
+ ("uint32_col", pandas.Series([6, 7, 8], dtype="uint32")),
+ ]
+ )
+ dataframe = pandas.DataFrame(df_data, columns=df_data.keys())
+
+ dataset_id = _make_dataset_id("bq_load_test")
+ self.temp_dataset(dataset_id)
+ table_id = "{}.{}.load_table_from_dataframe_w_automatic_schema".format(
+ Config.CLIENT.project, dataset_id
+ )
+
+ load_job = Config.CLIENT.load_table_from_dataframe(dataframe, table_id)
+ load_job.result()
+
+ table = Config.CLIENT.get_table(table_id)
+ self.assertEqual(
+ tuple(table.schema),
+ (
+ bigquery.SchemaField("bool_col", "BOOLEAN"),
+ bigquery.SchemaField("ts_col", "TIMESTAMP"),
+ bigquery.SchemaField("dt_col", "DATETIME"),
+ bigquery.SchemaField("float32_col", "FLOAT"),
+ bigquery.SchemaField("float64_col", "FLOAT"),
+ bigquery.SchemaField("int8_col", "INTEGER"),
+ bigquery.SchemaField("int16_col", "INTEGER"),
+ bigquery.SchemaField("int32_col", "INTEGER"),
+ bigquery.SchemaField("int64_col", "INTEGER"),
+ bigquery.SchemaField("uint8_col", "INTEGER"),
+ bigquery.SchemaField("uint16_col", "INTEGER"),
+ bigquery.SchemaField("uint32_col", "INTEGER"),
+ ),
+ )
+ self.assertEqual(table.num_rows, 3)
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ def test_load_table_from_dataframe_w_nulls(self):
+ """Test that a DataFrame with null columns can be uploaded if a
+ BigQuery schema is specified.
+
+ See: https://github.com/googleapis/google-cloud-python/issues/7370
+ """
+ # Schema with all scalar types.
+ scalars_schema = (
+ bigquery.SchemaField("bool_col", "BOOLEAN"),
+ bigquery.SchemaField("bytes_col", "BYTES"),
+ bigquery.SchemaField("date_col", "DATE"),
+ bigquery.SchemaField("dt_col", "DATETIME"),
+ bigquery.SchemaField("float_col", "FLOAT"),
+ bigquery.SchemaField("geo_col", "GEOGRAPHY"),
+ bigquery.SchemaField("int_col", "INTEGER"),
+ bigquery.SchemaField("num_col", "NUMERIC"),
+ bigquery.SchemaField("str_col", "STRING"),
+ bigquery.SchemaField("time_col", "TIME"),
+ bigquery.SchemaField("ts_col", "TIMESTAMP"),
+ )
+ table_schema = scalars_schema + (
+ # TODO: Array columns can't be read due to NULLABLE versus REPEATED
+ # mode mismatch. See:
+ # https://issuetracker.google.com/133415569#comment3
+ # bigquery.SchemaField("array_col", "INTEGER", mode="REPEATED"),
+ # TODO: Support writing StructArrays to Parquet. See:
+ # https://jira.apache.org/jira/browse/ARROW-2587
+ # bigquery.SchemaField("struct_col", "RECORD", fields=scalars_schema),
+ )
+ num_rows = 100
+ nulls = [None] * num_rows
+ df_data = collections.OrderedDict(
+ [
+ ("bool_col", nulls),
+ ("bytes_col", nulls),
+ ("date_col", nulls),
+ ("dt_col", nulls),
+ ("float_col", nulls),
+ ("geo_col", nulls),
+ ("int_col", nulls),
+ ("num_col", nulls),
+ ("str_col", nulls),
+ ("time_col", nulls),
+ ("ts_col", nulls),
+ ]
+ )
+ dataframe = pandas.DataFrame(df_data, columns=df_data.keys())
+
+ dataset_id = _make_dataset_id("bq_load_test")
+ self.temp_dataset(dataset_id)
+ table_id = "{}.{}.load_table_from_dataframe_w_nulls".format(
+ Config.CLIENT.project, dataset_id
+ )
+
+ # Create the table before loading so that schema mismatch errors are
+ # identified.
+ table = retry_403(Config.CLIENT.create_table)(
+ Table(table_id, schema=table_schema)
+ )
+ self.to_delete.insert(0, table)
+
+ job_config = bigquery.LoadJobConfig(schema=table_schema)
+ load_job = Config.CLIENT.load_table_from_dataframe(
+ dataframe, table_id, job_config=job_config
+ )
+ load_job.result()
+
+ table = Config.CLIENT.get_table(table)
+ self.assertEqual(tuple(table.schema), table_schema)
+ self.assertEqual(table.num_rows, num_rows)
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ def test_load_table_from_dataframe_w_required(self):
+ """Test that a DataFrame with required columns can be uploaded if a
+ BigQuery schema is specified.
+
+ See: https://github.com/googleapis/google-cloud-python/issues/8093
+ """
+ table_schema = (
+ bigquery.SchemaField("name", "STRING", mode="REQUIRED"),
+ bigquery.SchemaField("age", "INTEGER", mode="REQUIRED"),
+ )
+
+ records = [{"name": "Chip", "age": 2}, {"name": "Dale", "age": 3}]
+ dataframe = pandas.DataFrame(records, columns=["name", "age"])
+ job_config = bigquery.LoadJobConfig(schema=table_schema)
+ dataset_id = _make_dataset_id("bq_load_test")
+ self.temp_dataset(dataset_id)
+ table_id = "{}.{}.load_table_from_dataframe_w_required".format(
+ Config.CLIENT.project, dataset_id
+ )
+
+ # Create the table before loading so that schema mismatch errors are
+ # identified.
+ table = retry_403(Config.CLIENT.create_table)(
+ Table(table_id, schema=table_schema)
+ )
+ self.to_delete.insert(0, table)
+
+ job_config = bigquery.LoadJobConfig(schema=table_schema)
+ load_job = Config.CLIENT.load_table_from_dataframe(
+ dataframe, table_id, job_config=job_config
+ )
+ load_job.result()
+
+ table = Config.CLIENT.get_table(table)
+ self.assertEqual(tuple(table.schema), table_schema)
+ self.assertEqual(table.num_rows, 2)
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ def test_load_table_from_dataframe_w_explicit_schema(self):
+ # Schema with all scalar types.
+ scalars_schema = (
+ bigquery.SchemaField("bool_col", "BOOLEAN"),
+ bigquery.SchemaField("bytes_col", "BYTES"),
+ bigquery.SchemaField("date_col", "DATE"),
+ bigquery.SchemaField("dt_col", "DATETIME"),
+ bigquery.SchemaField("float_col", "FLOAT"),
+ bigquery.SchemaField("geo_col", "GEOGRAPHY"),
+ bigquery.SchemaField("int_col", "INTEGER"),
+ bigquery.SchemaField("num_col", "NUMERIC"),
+ bigquery.SchemaField("str_col", "STRING"),
+ bigquery.SchemaField("time_col", "TIME"),
+ bigquery.SchemaField("ts_col", "TIMESTAMP"),
+ )
+ table_schema = scalars_schema + (
+ # TODO: Array columns can't be read due to NULLABLE versus REPEATED
+ # mode mismatch. See:
+ # https://issuetracker.google.com/133415569#comment3
+ # bigquery.SchemaField("array_col", "INTEGER", mode="REPEATED"),
+ # TODO: Support writing StructArrays to Parquet. See:
+ # https://jira.apache.org/jira/browse/ARROW-2587
+ # bigquery.SchemaField("struct_col", "RECORD", fields=scalars_schema),
+ )
+ df_data = collections.OrderedDict(
+ [
+ ("bool_col", [True, None, False]),
+ ("bytes_col", [b"abc", None, b"def"]),
+ (
+ "date_col",
+ [datetime.date(1, 1, 1), None, datetime.date(9999, 12, 31)],
+ ),
+ (
+ "dt_col",
+ [
+ datetime.datetime(1, 1, 1, 0, 0, 0),
+ None,
+ datetime.datetime(9999, 12, 31, 23, 59, 59, 999999),
+ ],
+ ),
+ ("float_col", [float("-inf"), float("nan"), float("inf")]),
+ (
+ "geo_col",
+ [
+ "POINT(30 10)",
+ None,
+ "POLYGON ((30 10, 40 40, 20 40, 10 20, 30 10))",
+ ],
+ ),
+ ("int_col", [-9223372036854775808, None, 9223372036854775807]),
+ (
+ "num_col",
+ [
+ decimal.Decimal("-99999999999999999999999999999.999999999"),
+ None,
+ decimal.Decimal("99999999999999999999999999999.999999999"),
+ ],
+ ),
+ ("str_col", [u"abc", None, u"def"]),
+ (
+ "time_col",
+ [datetime.time(0, 0, 0), None, datetime.time(23, 59, 59, 999999)],
+ ),
+ (
+ "ts_col",
+ [
+ datetime.datetime(1, 1, 1, 0, 0, 0, tzinfo=pytz.utc),
+ None,
+ datetime.datetime(
+ 9999, 12, 31, 23, 59, 59, 999999, tzinfo=pytz.utc
+ ),
+ ],
+ ),
+ ]
+ )
+ dataframe = pandas.DataFrame(df_data, dtype="object", columns=df_data.keys())
+
+ dataset_id = _make_dataset_id("bq_load_test")
+ self.temp_dataset(dataset_id)
+ table_id = "{}.{}.load_table_from_dataframe_w_explicit_schema".format(
+ Config.CLIENT.project, dataset_id
+ )
+
+ job_config = bigquery.LoadJobConfig(schema=table_schema)
+ load_job = Config.CLIENT.load_table_from_dataframe(
+ dataframe, table_id, job_config=job_config
+ )
+ load_job.result()
+
+ table = Config.CLIENT.get_table(table_id)
+ self.assertEqual(tuple(table.schema), table_schema)
+ self.assertEqual(table.num_rows, 3)
+
+ def test_load_table_from_json_basic_use(self):
+ table_schema = (
+ bigquery.SchemaField("name", "STRING", mode="REQUIRED"),
+ bigquery.SchemaField("age", "INTEGER", mode="REQUIRED"),
+ bigquery.SchemaField("birthday", "DATE", mode="REQUIRED"),
+ bigquery.SchemaField("is_awesome", "BOOLEAN", mode="REQUIRED"),
+ )
+
+ json_rows = [
+ {"name": "John", "age": 18, "birthday": "2001-10-15", "is_awesome": False},
+ {"name": "Chuck", "age": 79, "birthday": "1940-03-10", "is_awesome": True},
+ ]
+
+ dataset_id = _make_dataset_id("bq_system_test")
+ self.temp_dataset(dataset_id)
+ table_id = "{}.{}.load_table_from_json_basic_use".format(
+ Config.CLIENT.project, dataset_id
+ )
+
+ # Create the table before loading so that schema mismatch errors are
+ # identified.
+ table = retry_403(Config.CLIENT.create_table)(
+ Table(table_id, schema=table_schema)
+ )
+ self.to_delete.insert(0, table)
+
+ job_config = bigquery.LoadJobConfig(schema=table_schema)
+ load_job = Config.CLIENT.load_table_from_json(
+ json_rows, table_id, job_config=job_config
+ )
+ load_job.result()
+
+ table = Config.CLIENT.get_table(table)
+ self.assertEqual(tuple(table.schema), table_schema)
+ self.assertEqual(table.num_rows, 2)
+
+ def test_load_table_from_json_schema_autodetect(self):
+ json_rows = [
+ {"name": "John", "age": 18, "birthday": "2001-10-15", "is_awesome": False},
+ {"name": "Chuck", "age": 79, "birthday": "1940-03-10", "is_awesome": True},
+ ]
+
+ dataset_id = _make_dataset_id("bq_system_test")
+ self.temp_dataset(dataset_id)
+ table_id = "{}.{}.load_table_from_json_basic_use".format(
+ Config.CLIENT.project, dataset_id
+ )
+
+ # Use schema with NULLABLE fields, because schema autodetection
+ # defaults to field mode NULLABLE.
+ table_schema = (
+ bigquery.SchemaField("name", "STRING", mode="NULLABLE"),
+ bigquery.SchemaField("age", "INTEGER", mode="NULLABLE"),
+ bigquery.SchemaField("birthday", "DATE", mode="NULLABLE"),
+ bigquery.SchemaField("is_awesome", "BOOLEAN", mode="NULLABLE"),
+ )
+ # create the table before loading so that the column order is predictable
+ table = retry_403(Config.CLIENT.create_table)(
+ Table(table_id, schema=table_schema)
+ )
+ self.to_delete.insert(0, table)
+
+ # do not pass an explicit job config to trigger automatic schema detection
+ load_job = Config.CLIENT.load_table_from_json(json_rows, table_id)
+ load_job.result()
+
+ table = Config.CLIENT.get_table(table)
+ self.assertEqual(tuple(table.schema), table_schema)
+ self.assertEqual(table.num_rows, 2)
+
+ def test_load_avro_from_uri_then_dump_table(self):
+ from google.cloud.bigquery.job import CreateDisposition
+ from google.cloud.bigquery.job import SourceFormat
+ from google.cloud.bigquery.job import WriteDisposition
+
+ table_name = "test_table"
+ rows = [
+ ("violet", 400),
+ ("indigo", 445),
+ ("blue", 475),
+ ("green", 510),
+ ("yellow", 570),
+ ("orange", 590),
+ ("red", 650),
+ ]
+ with open(os.path.join(WHERE, "data", "colors.avro"), "rb") as f:
+ GS_URL = self._write_avro_to_storage(
+ "bq_load_test" + unique_resource_id(), "colors.avro", f
+ )
+
+ dataset = self.temp_dataset(_make_dataset_id("bq_load_test"))
+ table_arg = dataset.table(table_name)
+ table = retry_403(Config.CLIENT.create_table)(Table(table_arg))
+ self.to_delete.insert(0, table)
+
+ config = bigquery.LoadJobConfig()
+ config.create_disposition = CreateDisposition.CREATE_NEVER
+ config.source_format = SourceFormat.AVRO
+ config.write_disposition = WriteDisposition.WRITE_EMPTY
+ job = Config.CLIENT.load_table_from_uri(GS_URL, table_arg, job_config=config)
+ job.result(timeout=JOB_TIMEOUT)
+ self.assertEqual(job.output_rows, len(rows))
+
+ table = Config.CLIENT.get_table(table)
+ fetched = self._fetch_single_page(table)
+ row_tuples = [r.values() for r in fetched]
+ self.assertEqual(
+ sorted(row_tuples, key=lambda x: x[1]), sorted(rows, key=lambda x: x[1])
+ )
+
+ def test_load_table_from_uri_then_dump_table(self):
+ from google.cloud.bigquery.job import CreateDisposition
+ from google.cloud.bigquery.job import SourceFormat
+ from google.cloud.bigquery.job import WriteDisposition
+
+ TABLE_ID = "test_table"
+ GS_URL = self._write_csv_to_storage(
+ "bq_load_test" + unique_resource_id(), "person_ages.csv", HEADER_ROW, ROWS
+ )
+
+ dataset = self.temp_dataset(_make_dataset_id("load_gcs_then_dump"))
+
+ table_arg = Table(dataset.table(TABLE_ID), schema=SCHEMA)
+ table = retry_403(Config.CLIENT.create_table)(table_arg)
+ self.to_delete.insert(0, table)
+
+ config = bigquery.LoadJobConfig()
+ config.create_disposition = CreateDisposition.CREATE_NEVER
+ config.skip_leading_rows = 1
+ config.source_format = SourceFormat.CSV
+ config.write_disposition = WriteDisposition.WRITE_EMPTY
+ job = Config.CLIENT.load_table_from_uri(
+ GS_URL, dataset.table(TABLE_ID), job_config=config
+ )
+
+ # Allow for 90 seconds of "warm up" before rows visible. See
+ # https://cloud.google.com/bigquery/streaming-data-into-bigquery#dataavailability
+ # 8 tries -> 1 + 2 + 4 + 8 + 16 + 32 + 64 = 127 seconds
+ retry = RetryInstanceState(_job_done, max_tries=8)
+ retry(job.reload)()
+
+ rows = self._fetch_single_page(table)
+ row_tuples = [r.values() for r in rows]
+ by_age = operator.itemgetter(1)
+ self.assertEqual(sorted(row_tuples, key=by_age), sorted(ROWS, key=by_age))
+
+ def test_load_table_from_file_w_explicit_location(self):
+ # Create a temporary bucket for extract files.
+ bucket_name = "bq_load_table_eu_extract_test" + unique_resource_id()
+ self._create_bucket(bucket_name, location="eu")
+
+ # Create a temporary dataset & table in the EU.
+ table_bytes = six.BytesIO(b"a,3\nb,2\nc,1\n")
+ client = Config.CLIENT
+ dataset = self.temp_dataset(_make_dataset_id("eu_load_file"), location="EU")
+ table_ref = dataset.table("letters")
+ job_config = bigquery.LoadJobConfig()
+ job_config.skip_leading_rows = 0
+ job_config.schema = [
+ bigquery.SchemaField("letter", "STRING"),
+ bigquery.SchemaField("value", "INTEGER"),
+ ]
+
+ # Load the file to an EU dataset with an EU load job.
+ load_job = client.load_table_from_file(
+ table_bytes, table_ref, location="EU", job_config=job_config
+ )
+ load_job.result()
+ job_id = load_job.job_id
+
+ # Can get the job from the EU.
+ load_job = client.get_job(job_id, location="EU")
+ self.assertEqual(job_id, load_job.job_id)
+ self.assertEqual("EU", load_job.location)
+ self.assertTrue(load_job.exists())
+
+ # Cannot get the job from the US.
+ with self.assertRaises(NotFound):
+ client.get_job(job_id, location="US")
+
+ load_job_us = client.get_job(job_id)
+ load_job_us._properties["jobReference"]["location"] = "US"
+ self.assertFalse(load_job_us.exists())
+ with self.assertRaises(NotFound):
+ load_job_us.reload()
+
+ # Can cancel the job from the EU.
+ self.assertTrue(load_job.cancel())
+ load_job = client.cancel_job(job_id, location="EU")
+ self.assertEqual(job_id, load_job.job_id)
+ self.assertEqual("EU", load_job.location)
+
+ # Cannot cancel the job from the US.
+ with self.assertRaises(NotFound):
+ client.cancel_job(job_id, location="US")
+ with self.assertRaises(NotFound):
+ load_job_us.cancel()
+
+ # Can list the table rows.
+ table = client.get_table(table_ref)
+ self.assertEqual(table.num_rows, 3)
+ rows = [(row.letter, row.value) for row in client.list_rows(table)]
+ self.assertEqual(list(sorted(rows)), [("a", 3), ("b", 2), ("c", 1)])
+
+ # Verify location behavior with queries
+ query_config = bigquery.QueryJobConfig()
+ query_config.dry_run = True
+
+ query_string = "SELECT * FROM `{}.letters` LIMIT 1".format(dataset.dataset_id)
+
+ eu_query = client.query(query_string, location="EU", job_config=query_config)
+ self.assertTrue(eu_query.done)
+
+ # Cannot query from US.
+ with self.assertRaises(GoogleAPICallError):
+ list(client.query(query_string, location="US", job_config=query_config))
+
+ # Cannot copy from US.
+ with self.assertRaises(GoogleAPICallError):
+ client.copy_table(
+ table_ref, dataset.table("letters2_us"), location="US"
+ ).result()
+
+ # Cannot extract from US.
+ with self.assertRaises(GoogleAPICallError):
+ client.extract_table(
+ table_ref, "gs://{}/letters-us.csv".format(bucket_name), location="US"
+ ).result()
+
+ def _write_csv_to_storage(self, bucket_name, blob_name, header_row, data_rows):
+ from google.cloud._testing import _NamedTemporaryFile
+
+ bucket = self._create_bucket(bucket_name)
+ blob = bucket.blob(blob_name)
+
+ with _NamedTemporaryFile() as temp:
+ with open(temp.name, "w") as csv_write:
+ writer = csv.writer(csv_write)
+ writer.writerow(header_row)
+ writer.writerows(data_rows)
+
+ with open(temp.name, "rb") as csv_read:
+ retry_storage_errors(blob.upload_from_file)(
+ csv_read, content_type="text/csv"
+ )
+
+ self.to_delete.insert(0, blob)
+ return "gs://{}/{}".format(bucket_name, blob_name)
+
+ def _write_avro_to_storage(self, bucket_name, blob_name, avro_file):
+ bucket = self._create_bucket(bucket_name)
+ blob = bucket.blob(blob_name)
+ retry_storage_errors(blob.upload_from_file)(
+ avro_file, content_type="application/x-avro-binary"
+ )
+ self.to_delete.insert(0, blob)
+ return "gs://{}/{}".format(bucket_name, blob_name)
+
+ def _load_table_for_extract_table(self, bucket, blob_name, table, rows):
+ from google.cloud._testing import _NamedTemporaryFile
+
+ blob = bucket.blob(blob_name)
+ with _NamedTemporaryFile() as temp:
+ with open(temp.name, "w") as csv_write:
+ writer = csv.writer(csv_write)
+ writer.writerow(HEADER_ROW)
+ writer.writerows(rows)
+
+ with open(temp.name, "rb") as csv_read:
+ retry_storage_errors(blob.upload_from_file)(
+ csv_read, content_type="text/csv"
+ )
+
+ self.to_delete.insert(0, blob)
+
+ dataset = self.temp_dataset(table.dataset_id)
+ table_ref = dataset.table(table.table_id)
+ config = bigquery.LoadJobConfig()
+ config.autodetect = True
+ gs_url = "gs://{}/{}".format(bucket.name, blob_name)
+ job = Config.CLIENT.load_table_from_uri(gs_url, table_ref, job_config=config)
+ # TODO(jba): do we need this retry now that we have job.result()?
+ # Allow for 90 seconds of "warm up" before rows visible. See
+ # https://cloud.google.com/bigquery/streaming-data-into-bigquery#dataavailability
+ # 8 tries -> 1 + 2 + 4 + 8 + 16 + 32 + 64 = 127 seconds
+ retry = RetryInstanceState(_job_done, max_tries=8)
+ retry(job.reload)()
+
+ def test_extract_table(self):
+ local_id = unique_resource_id()
+ bucket_name = "bq_extract_test" + local_id
+ source_blob_name = "person_ages.csv"
+ dataset_id = _make_dataset_id("load_gcs_then_extract")
+ table_id = "test_table"
+ table_ref = Config.CLIENT.dataset(dataset_id).table(table_id)
+ table = Table(table_ref)
+ self.to_delete.insert(0, table)
+ bucket = self._create_bucket(bucket_name)
+ self._load_table_for_extract_table(bucket, source_blob_name, table_ref, ROWS)
+ destination_blob_name = "person_ages_out.csv"
+ destination = bucket.blob(destination_blob_name)
+ destination_uri = "gs://{}/person_ages_out.csv".format(bucket_name)
+
+ job = Config.CLIENT.extract_table(table_ref, destination_uri)
+ job.result(timeout=100)
+
+ self.to_delete.insert(0, destination)
+ got_bytes = retry_storage_errors(destination.download_as_string)()
+ got = got_bytes.decode("utf-8")
+ self.assertIn("Bharney Rhubble", got)
+
+ def test_copy_table(self):
+ # If we create a new table to copy from, the test won't work
+ # because the new rows will be stored in the streaming buffer,
+ # and copy jobs don't read the streaming buffer.
+ # We could wait for the streaming buffer to empty, but that could
+ # take minutes. Instead we copy a small public table.
+ source_dataset = DatasetReference("bigquery-public-data", "samples")
+ source_ref = source_dataset.table("shakespeare")
+ dest_dataset = self.temp_dataset(_make_dataset_id("copy_table"))
+ dest_ref = dest_dataset.table("destination_table")
+ job_config = bigquery.CopyJobConfig()
+ job = Config.CLIENT.copy_table(source_ref, dest_ref, job_config=job_config)
+ job.result()
+
+ dest_table = Config.CLIENT.get_table(dest_ref)
+ self.to_delete.insert(0, dest_table)
+ # Just check that we got some rows.
+ got_rows = self._fetch_single_page(dest_table)
+ self.assertTrue(len(got_rows) > 0)
+
+ def test_job_cancel(self):
+ DATASET_ID = _make_dataset_id("job_cancel")
+ JOB_ID_PREFIX = "fetch_" + DATASET_ID
+ TABLE_NAME = "test_table"
+ QUERY = "SELECT * FROM %s.%s" % (DATASET_ID, TABLE_NAME)
+
+ dataset = self.temp_dataset(DATASET_ID)
+
+ table_arg = Table(dataset.table(TABLE_NAME), schema=SCHEMA)
+ table = retry_403(Config.CLIENT.create_table)(table_arg)
+ self.to_delete.insert(0, table)
+
+ job = Config.CLIENT.query(QUERY, job_id_prefix=JOB_ID_PREFIX)
+ job.cancel()
+
+ retry = RetryInstanceState(_job_done, max_tries=8)
+ retry(job.reload)()
+
+ # The `cancel` API doesn't leave any reliable traces on
+ # the status of the job resource, so we can't really assert for
+ # them here. The best we can do is not that the API call didn't
+ # raise an error, and that the job completed (in the `retry()`
+ # above).
+
+ def test_get_failed_job(self):
+ # issue 4246
+ from google.api_core.exceptions import BadRequest
+
+ JOB_ID = "invalid_{}".format(str(uuid.uuid4()))
+ QUERY = "SELECT TIMESTAMP_ADD(@ts_value, INTERVAL 1 HOUR);"
+ PARAM = bigquery.ScalarQueryParameter("ts_value", "TIMESTAMP", 1.4810976e9)
+
+ job_config = bigquery.QueryJobConfig()
+ job_config.query_parameters = [PARAM]
+
+ with self.assertRaises(BadRequest):
+ Config.CLIENT.query(QUERY, job_id=JOB_ID, job_config=job_config).result()
+
+ job = Config.CLIENT.get_job(JOB_ID)
+
+ with self.assertRaises(ValueError):
+ job.query_parameters
+
+ def test_query_w_legacy_sql_types(self):
+ naive = datetime.datetime(2016, 12, 5, 12, 41, 9)
+ stamp = "%s %s" % (naive.date().isoformat(), naive.time().isoformat())
+ zoned = naive.replace(tzinfo=UTC)
+ examples = [
+ {"sql": "SELECT 1", "expected": 1},
+ {"sql": "SELECT 1.3", "expected": 1.3},
+ {"sql": "SELECT TRUE", "expected": True},
+ {"sql": 'SELECT "ABC"', "expected": "ABC"},
+ {"sql": 'SELECT CAST("foo" AS BYTES)', "expected": b"foo"},
+ {"sql": 'SELECT CAST("%s" AS TIMESTAMP)' % (stamp,), "expected": zoned},
+ ]
+ for example in examples:
+ job_config = bigquery.QueryJobConfig()
+ job_config.use_legacy_sql = True
+ rows = list(Config.CLIENT.query(example["sql"], job_config=job_config))
+ self.assertEqual(len(rows), 1)
+ self.assertEqual(len(rows[0]), 1)
+ self.assertEqual(rows[0][0], example["expected"])
+
+ def _generate_standard_sql_types_examples(self):
+ naive = datetime.datetime(2016, 12, 5, 12, 41, 9)
+ naive_microseconds = datetime.datetime(2016, 12, 5, 12, 41, 9, 250000)
+ stamp = "%s %s" % (naive.date().isoformat(), naive.time().isoformat())
+ stamp_microseconds = stamp + ".250000"
+ zoned = naive.replace(tzinfo=UTC)
+ zoned_microseconds = naive_microseconds.replace(tzinfo=UTC)
+ numeric = decimal.Decimal("123456789.123456789")
+ return [
+ {"sql": "SELECT 1", "expected": 1},
+ {"sql": "SELECT 1.3", "expected": 1.3},
+ {"sql": "SELECT TRUE", "expected": True},
+ {"sql": 'SELECT "ABC"', "expected": "ABC"},
+ {"sql": 'SELECT CAST("foo" AS BYTES)', "expected": b"foo"},
+ {"sql": 'SELECT TIMESTAMP "%s"' % (stamp,), "expected": zoned},
+ {
+ "sql": 'SELECT TIMESTAMP "%s"' % (stamp_microseconds,),
+ "expected": zoned_microseconds,
+ },
+ {"sql": 'SELECT DATETIME(TIMESTAMP "%s")' % (stamp,), "expected": naive},
+ {
+ "sql": 'SELECT DATETIME(TIMESTAMP "%s")' % (stamp_microseconds,),
+ "expected": naive_microseconds,
+ },
+ {"sql": 'SELECT DATE(TIMESTAMP "%s")' % (stamp,), "expected": naive.date()},
+ {"sql": 'SELECT TIME(TIMESTAMP "%s")' % (stamp,), "expected": naive.time()},
+ {"sql": 'SELECT NUMERIC "%s"' % (numeric,), "expected": numeric},
+ {"sql": "SELECT (1, 2)", "expected": {"_field_1": 1, "_field_2": 2}},
+ {
+ "sql": "SELECT ((1, 2), (3, 4), 5)",
+ "expected": {
+ "_field_1": {"_field_1": 1, "_field_2": 2},
+ "_field_2": {"_field_1": 3, "_field_2": 4},
+ "_field_3": 5,
+ },
+ },
+ {"sql": "SELECT [1, 2, 3]", "expected": [1, 2, 3]},
+ {
+ "sql": "SELECT ([1, 2], 3, [4, 5])",
+ "expected": {"_field_1": [1, 2], "_field_2": 3, "_field_3": [4, 5]},
+ },
+ {
+ "sql": "SELECT [(1, 2, 3), (4, 5, 6)]",
+ "expected": [
+ {"_field_1": 1, "_field_2": 2, "_field_3": 3},
+ {"_field_1": 4, "_field_2": 5, "_field_3": 6},
+ ],
+ },
+ {
+ "sql": "SELECT [([1, 2, 3], 4), ([5, 6], 7)]",
+ "expected": [
+ {u"_field_1": [1, 2, 3], u"_field_2": 4},
+ {u"_field_1": [5, 6], u"_field_2": 7},
+ ],
+ },
+ {
+ "sql": "SELECT ARRAY(SELECT STRUCT([1, 2]))",
+ "expected": [{u"_field_1": [1, 2]}],
+ },
+ {"sql": "SELECT ST_GeogPoint(1, 2)", "expected": "POINT(1 2)"},
+ ]
+
+ def test_query_w_standard_sql_types(self):
+ examples = self._generate_standard_sql_types_examples()
+ for example in examples:
+ rows = list(Config.CLIENT.query(example["sql"]))
+ self.assertEqual(len(rows), 1)
+ self.assertEqual(len(rows[0]), 1)
+ self.assertEqual(rows[0][0], example["expected"])
+
+ def test_query_w_failed_query(self):
+ from google.api_core.exceptions import BadRequest
+
+ with self.assertRaises(BadRequest):
+ Config.CLIENT.query("invalid syntax;").result()
+
+ def test_query_w_wrong_config(self):
+ from google.cloud.bigquery.job import LoadJobConfig
+
+ good_query = "SELECT 1;"
+ rows = list(Config.CLIENT.query("SELECT 1;").result())
+ assert rows[0][0] == 1
+
+ bad_config = LoadJobConfig()
+ bad_config.destination = Config.CLIENT.dataset("dset").table("tbl")
+ with self.assertRaises(Exception):
+ Config.CLIENT.query(good_query, job_config=bad_config).result()
+
+ def test_query_w_timeout(self):
+ query_job = Config.CLIENT.query(
+ "SELECT * FROM `bigquery-public-data.github_repos.commits`;",
+ job_id_prefix="test_query_w_timeout_",
+ )
+
+ with self.assertRaises(concurrent.futures.TimeoutError):
+ # 1 second is much too short for this query.
+ query_job.result(timeout=1)
+
+ def test_query_w_page_size(self):
+ page_size = 45
+ query_job = Config.CLIENT.query(
+ "SELECT word FROM `bigquery-public-data.samples.shakespeare`;",
+ job_id_prefix="test_query_w_page_size_",
+ )
+ iterator = query_job.result(page_size=page_size)
+ self.assertEqual(next(iterator.pages).num_items, page_size)
+
+ def test_query_statistics(self):
+ """
+ A system test to exercise some of the extended query statistics.
+
+ Note: We construct a query that should need at least three stages by
+ specifying a JOIN query. Exact plan and stats are effectively
+ non-deterministic, so we're largely interested in confirming values
+ are present.
+ """
+
+ job_config = bigquery.QueryJobConfig()
+ job_config.use_query_cache = False
+
+ query_job = Config.CLIENT.query(
+ """
+ SELECT
+ COUNT(1)
+ FROM
+ (
+ SELECT
+ year,
+ wban_number
+ FROM `bigquery-public-data.samples.gsod`
+ LIMIT 1000
+ ) lside
+ INNER JOIN
+ (
+ SELECT
+ year,
+ state
+ FROM `bigquery-public-data.samples.natality`
+ LIMIT 1000
+ ) rside
+ ON
+ lside.year = rside.year
+ """,
+ location="US",
+ job_config=job_config,
+ )
+
+ # run the job to completion
+ query_job.result()
+
+ # Assert top-level stats
+ self.assertFalse(query_job.cache_hit)
+ self.assertIsNotNone(query_job.destination)
+ self.assertTrue(query_job.done)
+ self.assertFalse(query_job.dry_run)
+ self.assertIsNone(query_job.num_dml_affected_rows)
+ self.assertEqual(query_job.priority, "INTERACTIVE")
+ self.assertGreater(query_job.total_bytes_billed, 1)
+ self.assertGreater(query_job.total_bytes_processed, 1)
+ self.assertEqual(query_job.statement_type, "SELECT")
+ self.assertGreater(query_job.slot_millis, 1)
+
+ # Make assertions on the shape of the query plan.
+ plan = query_job.query_plan
+ self.assertGreaterEqual(len(plan), 3)
+ first_stage = plan[0]
+ self.assertIsNotNone(first_stage.start)
+ self.assertIsNotNone(first_stage.end)
+ self.assertIsNotNone(first_stage.entry_id)
+ self.assertIsNotNone(first_stage.name)
+ self.assertGreater(first_stage.parallel_inputs, 0)
+ self.assertGreater(first_stage.completed_parallel_inputs, 0)
+ self.assertGreater(first_stage.shuffle_output_bytes, 0)
+ self.assertEqual(first_stage.status, "COMPLETE")
+
+ # Query plan is a digraph. Ensure it has inter-stage links,
+ # but not every stage has inputs.
+ stages_with_inputs = 0
+ for entry in plan:
+ if len(entry.input_stages) > 0:
+ stages_with_inputs = stages_with_inputs + 1
+ self.assertGreater(stages_with_inputs, 0)
+ self.assertGreater(len(plan), stages_with_inputs)
+
+ def test_dbapi_w_standard_sql_types(self):
+ examples = self._generate_standard_sql_types_examples()
+ for example in examples:
+ Config.CURSOR.execute(example["sql"])
+ self.assertEqual(Config.CURSOR.rowcount, 1)
+ row = Config.CURSOR.fetchone()
+ self.assertEqual(len(row), 1)
+ self.assertEqual(row[0], example["expected"])
+ row = Config.CURSOR.fetchone()
+ self.assertIsNone(row)
+
+ def test_dbapi_fetchall(self):
+ query = "SELECT * FROM UNNEST([(1, 2), (3, 4), (5, 6)])"
+
+ for arraysize in range(1, 5):
+ Config.CURSOR.execute(query)
+ self.assertEqual(Config.CURSOR.rowcount, 3, "expected 3 rows")
+ Config.CURSOR.arraysize = arraysize
+ rows = Config.CURSOR.fetchall()
+ row_tuples = [r.values() for r in rows]
+ self.assertEqual(row_tuples, [(1, 2), (3, 4), (5, 6)])
+
+ def _load_table_for_dml(self, rows, dataset_id, table_id):
+ from google.cloud._testing import _NamedTemporaryFile
+ from google.cloud.bigquery.job import CreateDisposition
+ from google.cloud.bigquery.job import SourceFormat
+ from google.cloud.bigquery.job import WriteDisposition
+
+ dataset = self.temp_dataset(dataset_id)
+ greeting = bigquery.SchemaField("greeting", "STRING", mode="NULLABLE")
+ table_ref = dataset.table(table_id)
+ table_arg = Table(table_ref, schema=[greeting])
+ table = retry_403(Config.CLIENT.create_table)(table_arg)
+ self.to_delete.insert(0, table)
+
+ with _NamedTemporaryFile() as temp:
+ with open(temp.name, "w") as csv_write:
+ writer = csv.writer(csv_write)
+ writer.writerow(("Greeting",))
+ writer.writerows(rows)
+
+ with open(temp.name, "rb") as csv_read:
+ config = bigquery.LoadJobConfig()
+ config.source_format = SourceFormat.CSV
+ config.skip_leading_rows = 1
+ config.create_disposition = CreateDisposition.CREATE_NEVER
+ config.write_disposition = WriteDisposition.WRITE_EMPTY
+ job = Config.CLIENT.load_table_from_file(
+ csv_read, table_ref, job_config=config
+ )
+
+ # Retry until done.
+ job.result(timeout=JOB_TIMEOUT)
+ self._fetch_single_page(table)
+
+ def test_query_w_dml(self):
+ dataset_name = _make_dataset_id("dml_query")
+ table_name = "test_table"
+ self._load_table_for_dml([("Hello World",)], dataset_name, table_name)
+ query_template = """UPDATE {}.{}
+ SET greeting = 'Guten Tag'
+ WHERE greeting = 'Hello World'
+ """
+
+ query_job = Config.CLIENT.query(
+ query_template.format(dataset_name, table_name),
+ job_id_prefix="test_query_w_dml_",
+ )
+ query_job.result()
+
+ self.assertEqual(query_job.num_dml_affected_rows, 1)
+
+ def test_dbapi_w_dml(self):
+ dataset_name = _make_dataset_id("dml_dbapi")
+ table_name = "test_table"
+ self._load_table_for_dml([("Hello World",)], dataset_name, table_name)
+ query_template = """UPDATE {}.{}
+ SET greeting = 'Guten Tag'
+ WHERE greeting = 'Hello World'
+ """
+
+ Config.CURSOR.execute(
+ query_template.format(dataset_name, table_name),
+ job_id="test_dbapi_w_dml_{}".format(str(uuid.uuid4())),
+ )
+ self.assertEqual(Config.CURSOR.rowcount, 1)
+ self.assertIsNone(Config.CURSOR.fetchone())
+
+ def test_query_w_query_params(self):
+ from google.cloud.bigquery.job import QueryJobConfig
+ from google.cloud.bigquery.query import ArrayQueryParameter
+ from google.cloud.bigquery.query import ScalarQueryParameter
+ from google.cloud.bigquery.query import StructQueryParameter
+
+ question = "What is the answer to life, the universe, and everything?"
+ question_param = ScalarQueryParameter(
+ name="question", type_="STRING", value=question
+ )
+ answer = 42
+ answer_param = ScalarQueryParameter(name="answer", type_="INT64", value=answer)
+ pi = 3.1415926
+ pi_param = ScalarQueryParameter(name="pi", type_="FLOAT64", value=pi)
+ pi_numeric = decimal.Decimal("3.141592654")
+ pi_numeric_param = ScalarQueryParameter(
+ name="pi_numeric_param", type_="NUMERIC", value=pi_numeric
+ )
+ truthy = True
+ truthy_param = ScalarQueryParameter(name="truthy", type_="BOOL", value=truthy)
+ beef = b"DEADBEEF"
+ beef_param = ScalarQueryParameter(name="beef", type_="BYTES", value=beef)
+ naive = datetime.datetime(2016, 12, 5, 12, 41, 9)
+ naive_param = ScalarQueryParameter(name="naive", type_="DATETIME", value=naive)
+ naive_date_param = ScalarQueryParameter(
+ name="naive_date", type_="DATE", value=naive.date()
+ )
+ naive_time_param = ScalarQueryParameter(
+ name="naive_time", type_="TIME", value=naive.time()
+ )
+ zoned = naive.replace(tzinfo=UTC)
+ zoned_param = ScalarQueryParameter(name="zoned", type_="TIMESTAMP", value=zoned)
+ array_param = ArrayQueryParameter(
+ name="array_param", array_type="INT64", values=[1, 2]
+ )
+ struct_param = StructQueryParameter("hitchhiker", question_param, answer_param)
+ phred_name = "Phred Phlyntstone"
+ phred_name_param = ScalarQueryParameter(
+ name="name", type_="STRING", value=phred_name
+ )
+ phred_age = 32
+ phred_age_param = ScalarQueryParameter(
+ name="age", type_="INT64", value=phred_age
+ )
+ phred_param = StructQueryParameter(None, phred_name_param, phred_age_param)
+ bharney_name = "Bharney Rhubbyl"
+ bharney_name_param = ScalarQueryParameter(
+ name="name", type_="STRING", value=bharney_name
+ )
+ bharney_age = 31
+ bharney_age_param = ScalarQueryParameter(
+ name="age", type_="INT64", value=bharney_age
+ )
+ bharney_param = StructQueryParameter(
+ None, bharney_name_param, bharney_age_param
+ )
+ characters_param = ArrayQueryParameter(
+ name=None, array_type="RECORD", values=[phred_param, bharney_param]
+ )
+ hero_param = StructQueryParameter("hero", phred_name_param, phred_age_param)
+ sidekick_param = StructQueryParameter(
+ "sidekick", bharney_name_param, bharney_age_param
+ )
+ roles_param = StructQueryParameter("roles", hero_param, sidekick_param)
+ friends_param = ArrayQueryParameter(
+ name="friends", array_type="STRING", values=[phred_name, bharney_name]
+ )
+ with_friends_param = StructQueryParameter(None, friends_param)
+ top_left_param = StructQueryParameter(
+ "top_left",
+ ScalarQueryParameter("x", "INT64", 12),
+ ScalarQueryParameter("y", "INT64", 102),
+ )
+ bottom_right_param = StructQueryParameter(
+ "bottom_right",
+ ScalarQueryParameter("x", "INT64", 22),
+ ScalarQueryParameter("y", "INT64", 92),
+ )
+ rectangle_param = StructQueryParameter(
+ "rectangle", top_left_param, bottom_right_param
+ )
+ examples = [
+ {
+ "sql": "SELECT @question",
+ "expected": question,
+ "query_parameters": [question_param],
+ },
+ {
+ "sql": "SELECT @answer",
+ "expected": answer,
+ "query_parameters": [answer_param],
+ },
+ {"sql": "SELECT @pi", "expected": pi, "query_parameters": [pi_param]},
+ {
+ "sql": "SELECT @pi_numeric_param",
+ "expected": pi_numeric,
+ "query_parameters": [pi_numeric_param],
+ },
+ {
+ "sql": "SELECT @truthy",
+ "expected": truthy,
+ "query_parameters": [truthy_param],
+ },
+ {"sql": "SELECT @beef", "expected": beef, "query_parameters": [beef_param]},
+ {
+ "sql": "SELECT @naive",
+ "expected": naive,
+ "query_parameters": [naive_param],
+ },
+ {
+ "sql": "SELECT @naive_date",
+ "expected": naive.date(),
+ "query_parameters": [naive_date_param],
+ },
+ {
+ "sql": "SELECT @naive_time",
+ "expected": naive.time(),
+ "query_parameters": [naive_time_param],
+ },
+ {
+ "sql": "SELECT @zoned",
+ "expected": zoned,
+ "query_parameters": [zoned_param],
+ },
+ {
+ "sql": "SELECT @array_param",
+ "expected": [1, 2],
+ "query_parameters": [array_param],
+ },
+ {
+ "sql": "SELECT (@hitchhiker.question, @hitchhiker.answer)",
+ "expected": ({"_field_1": question, "_field_2": answer}),
+ "query_parameters": [struct_param],
+ },
+ {
+ "sql": "SELECT "
+ "((@rectangle.bottom_right.x - @rectangle.top_left.x) "
+ "* (@rectangle.top_left.y - @rectangle.bottom_right.y))",
+ "expected": 100,
+ "query_parameters": [rectangle_param],
+ },
+ {
+ "sql": "SELECT ?",
+ "expected": [
+ {"name": phred_name, "age": phred_age},
+ {"name": bharney_name, "age": bharney_age},
+ ],
+ "query_parameters": [characters_param],
+ },
+ {
+ "sql": "SELECT @roles",
+ "expected": {
+ "hero": {"name": phred_name, "age": phred_age},
+ "sidekick": {"name": bharney_name, "age": bharney_age},
+ },
+ "query_parameters": [roles_param],
+ },
+ {
+ "sql": "SELECT ?",
+ "expected": {"friends": [phred_name, bharney_name]},
+ "query_parameters": [with_friends_param],
+ },
+ ]
+ for example in examples:
+ jconfig = QueryJobConfig()
+ jconfig.query_parameters = example["query_parameters"]
+ query_job = Config.CLIENT.query(
+ example["sql"],
+ job_config=jconfig,
+ job_id_prefix="test_query_w_query_params",
+ )
+ rows = list(query_job.result())
+ self.assertEqual(len(rows), 1)
+ self.assertEqual(len(rows[0]), 1)
+ self.assertEqual(rows[0][0], example["expected"])
+
+ def test_dbapi_w_query_parameters(self):
+ examples = [
+ {
+ "sql": "SELECT %(boolval)s",
+ "expected": True,
+ "query_parameters": {"boolval": True},
+ },
+ {
+ "sql": 'SELECT %(a "very" weird `name`)s',
+ "expected": True,
+ "query_parameters": {'a "very" weird `name`': True},
+ },
+ {
+ "sql": "SELECT %(select)s",
+ "expected": True,
+ "query_parameters": {"select": True}, # this name is a keyword
+ },
+ {"sql": "SELECT %s", "expected": False, "query_parameters": [False]},
+ {
+ "sql": "SELECT %(intval)s",
+ "expected": 123,
+ "query_parameters": {"intval": 123},
+ },
+ {
+ "sql": "SELECT %s",
+ "expected": -123456789,
+ "query_parameters": [-123456789],
+ },
+ {
+ "sql": "SELECT %(floatval)s",
+ "expected": 1.25,
+ "query_parameters": {"floatval": 1.25},
+ },
+ {
+ "sql": "SELECT LOWER(%(strval)s)",
+ "query_parameters": {"strval": "I Am A String"},
+ "expected": "i am a string",
+ },
+ {
+ "sql": "SELECT DATE_SUB(%(dateval)s, INTERVAL 1 DAY)",
+ "query_parameters": {"dateval": datetime.date(2017, 4, 2)},
+ "expected": datetime.date(2017, 4, 1),
+ },
+ {
+ "sql": "SELECT TIME_ADD(%(timeval)s, INTERVAL 4 SECOND)",
+ "query_parameters": {"timeval": datetime.time(12, 34, 56)},
+ "expected": datetime.time(12, 35, 0),
+ },
+ {
+ "sql": ("SELECT DATETIME_ADD(%(datetimeval)s, INTERVAL 53 SECOND)"),
+ "query_parameters": {
+ "datetimeval": datetime.datetime(2012, 3, 4, 5, 6, 7)
+ },
+ "expected": datetime.datetime(2012, 3, 4, 5, 7, 0),
+ },
+ {
+ "sql": "SELECT TIMESTAMP_TRUNC(%(zoned)s, MINUTE)",
+ "query_parameters": {
+ "zoned": datetime.datetime(2012, 3, 4, 5, 6, 7, tzinfo=UTC)
+ },
+ "expected": datetime.datetime(2012, 3, 4, 5, 6, 0, tzinfo=UTC),
+ },
+ {
+ "sql": "SELECT TIMESTAMP_TRUNC(%(zoned)s, MINUTE)",
+ "query_parameters": {
+ "zoned": datetime.datetime(2012, 3, 4, 5, 6, 7, 250000, tzinfo=UTC)
+ },
+ "expected": datetime.datetime(2012, 3, 4, 5, 6, 0, tzinfo=UTC),
+ },
+ ]
+ for example in examples:
+ msg = "sql: {} query_parameters: {}".format(
+ example["sql"], example["query_parameters"]
+ )
+
+ Config.CURSOR.execute(example["sql"], example["query_parameters"])
+
+ self.assertEqual(Config.CURSOR.rowcount, 1, msg=msg)
+ row = Config.CURSOR.fetchone()
+ self.assertEqual(len(row), 1, msg=msg)
+ self.assertEqual(row[0], example["expected"], msg=msg)
+ row = Config.CURSOR.fetchone()
+ self.assertIsNone(row, msg=msg)
+
+ def test_large_query_w_public_data(self):
+ PUBLIC = "bigquery-public-data"
+ DATASET_ID = "samples"
+ TABLE_NAME = "natality"
+ LIMIT = 1000
+ SQL = "SELECT * from `{}.{}.{}` LIMIT {}".format(
+ PUBLIC, DATASET_ID, TABLE_NAME, LIMIT
+ )
+
+ query_job = Config.CLIENT.query(SQL)
+
+ rows = list(query_job)
+ self.assertEqual(len(rows), LIMIT)
+
+ def test_query_future(self):
+ query_job = Config.CLIENT.query("SELECT 1")
+ iterator = query_job.result(timeout=JOB_TIMEOUT)
+ row_tuples = [r.values() for r in iterator]
+ self.assertEqual(row_tuples, [(1,)])
+
+ def test_query_iter(self):
+ import types
+
+ query_job = Config.CLIENT.query("SELECT 1")
+ self.assertIsInstance(iter(query_job), types.GeneratorType)
+ row_tuples = [r.values() for r in query_job]
+ self.assertEqual(row_tuples, [(1,)])
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ def test_query_results_to_dataframe(self):
+ QUERY = """
+ SELECT id, author, time_ts, dead
+ FROM `bigquery-public-data.hacker_news.comments`
+ LIMIT 10
+ """
+
+ df = Config.CLIENT.query(QUERY).result().to_dataframe()
+
+ self.assertIsInstance(df, pandas.DataFrame)
+ self.assertEqual(len(df), 10) # verify the number of rows
+ column_names = ["id", "author", "time_ts", "dead"]
+ self.assertEqual(list(df), column_names) # verify the column names
+ exp_datatypes = {
+ "id": int,
+ "author": six.text_type,
+ "time_ts": pandas.Timestamp,
+ "dead": bool,
+ }
+ for index, row in df.iterrows():
+ for col in column_names:
+ # all the schema fields are nullable, so None is acceptable
+ if not row[col] is None:
+ self.assertIsInstance(row[col], exp_datatypes[col])
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(
+ bigquery_storage_v1beta1 is None, "Requires `google-cloud-bigquery-storage`"
+ )
+ def test_query_results_to_dataframe_w_bqstorage(self):
+ dest_dataset = self.temp_dataset(_make_dataset_id("bqstorage_to_dataframe_"))
+ dest_ref = dest_dataset.table("query_results")
+
+ query = """
+ SELECT id, author, time_ts, dead
+ FROM `bigquery-public-data.hacker_news.comments`
+ LIMIT 10
+ """
+
+ bqstorage_client = bigquery_storage_v1beta1.BigQueryStorageClient(
+ credentials=Config.CLIENT._credentials
+ )
+
+ job_configs = (
+ # There is a known issue reading small anonymous query result
+ # tables with the BQ Storage API. Writing to a destination
+ # table works around this issue.
+ bigquery.QueryJobConfig(
+ destination=dest_ref, write_disposition="WRITE_TRUNCATE"
+ ),
+ # Check that the client is able to work around the issue with
+ # reading small anonymous query result tables by falling back to
+ # the tabledata.list API.
+ None,
+ )
+
+ for job_config in job_configs:
+ df = (
+ Config.CLIENT.query(query, job_config=job_config)
+ .result()
+ .to_dataframe(bqstorage_client)
+ )
+
+ self.assertIsInstance(df, pandas.DataFrame)
+ self.assertEqual(len(df), 10) # verify the number of rows
+ column_names = ["id", "author", "time_ts", "dead"]
+ self.assertEqual(list(df), column_names)
+ exp_datatypes = {
+ "id": int,
+ "author": six.text_type,
+ "time_ts": pandas.Timestamp,
+ "dead": bool,
+ }
+ for index, row in df.iterrows():
+ for col in column_names:
+ # all the schema fields are nullable, so None is acceptable
+ if not row[col] is None:
+ self.assertIsInstance(row[col], exp_datatypes[col])
+
+ def test_insert_rows_nested_nested(self):
+ # See #2951
+ SF = bigquery.SchemaField
+ schema = [
+ SF("string_col", "STRING", mode="NULLABLE"),
+ SF(
+ "record_col",
+ "RECORD",
+ mode="NULLABLE",
+ fields=[
+ SF("nested_string", "STRING", mode="NULLABLE"),
+ SF("nested_repeated", "INTEGER", mode="REPEATED"),
+ SF(
+ "nested_record",
+ "RECORD",
+ mode="NULLABLE",
+ fields=[SF("nested_nested_string", "STRING", mode="NULLABLE")],
+ ),
+ ],
+ ),
+ ]
+ record = {
+ "nested_string": "another string value",
+ "nested_repeated": [0, 1, 2],
+ "nested_record": {"nested_nested_string": "some deep insight"},
+ }
+ to_insert = [("Some value", record)]
+ table_id = "test_table"
+ dataset = self.temp_dataset(_make_dataset_id("issue_2951"))
+ table_arg = Table(dataset.table(table_id), schema=schema)
+ table = retry_403(Config.CLIENT.create_table)(table_arg)
+ self.to_delete.insert(0, table)
+
+ Config.CLIENT.insert_rows(table, to_insert)
+
+ retry = RetryResult(_has_rows, max_tries=8)
+ rows = retry(self._fetch_single_page)(table)
+ row_tuples = [r.values() for r in rows]
+ self.assertEqual(row_tuples, to_insert)
+
+ def test_insert_rows_nested_nested_dictionary(self):
+ # See #2951
+ SF = bigquery.SchemaField
+ schema = [
+ SF("string_col", "STRING", mode="NULLABLE"),
+ SF(
+ "record_col",
+ "RECORD",
+ mode="NULLABLE",
+ fields=[
+ SF("nested_string", "STRING", mode="NULLABLE"),
+ SF("nested_repeated", "INTEGER", mode="REPEATED"),
+ SF(
+ "nested_record",
+ "RECORD",
+ mode="NULLABLE",
+ fields=[SF("nested_nested_string", "STRING", mode="NULLABLE")],
+ ),
+ ],
+ ),
+ ]
+ record = {
+ "nested_string": "another string value",
+ "nested_repeated": [0, 1, 2],
+ "nested_record": {"nested_nested_string": "some deep insight"},
+ }
+ to_insert = [{"string_col": "Some value", "record_col": record}]
+ table_id = "test_table"
+ dataset = self.temp_dataset(_make_dataset_id("issue_2951"))
+ table_arg = Table(dataset.table(table_id), schema=schema)
+ table = retry_403(Config.CLIENT.create_table)(table_arg)
+ self.to_delete.insert(0, table)
+
+ Config.CLIENT.insert_rows(table, to_insert)
+
+ retry = RetryResult(_has_rows, max_tries=8)
+ rows = retry(self._fetch_single_page)(table)
+ row_tuples = [r.values() for r in rows]
+ expected_rows = [("Some value", record)]
+ self.assertEqual(row_tuples, expected_rows)
+
+ def test_create_routine(self):
+ routine_name = "test_routine"
+ dataset = self.temp_dataset(_make_dataset_id("create_routine"))
+ float64_type = bigquery_v2.types.StandardSqlDataType(
+ type_kind=bigquery_v2.enums.StandardSqlDataType.TypeKind.FLOAT64
+ )
+ routine = bigquery.Routine(
+ dataset.routine(routine_name),
+ language="JAVASCRIPT",
+ type_="SCALAR_FUNCTION",
+ return_type=float64_type,
+ imported_libraries=[
+ "gs://{}/bigquery/udfs/max-value.js".format(SAMPLES_BUCKET)
+ ],
+ )
+ routine.arguments = [
+ bigquery.RoutineArgument(
+ name="arr",
+ data_type=bigquery_v2.types.StandardSqlDataType(
+ type_kind=bigquery_v2.enums.StandardSqlDataType.TypeKind.ARRAY,
+ array_element_type=float64_type,
+ ),
+ )
+ ]
+ routine.body = "return maxValue(arr)"
+ query_string = "SELECT `{}`([-100.0, 3.14, 100.0, 42.0]) as max_value;".format(
+ str(routine.reference)
+ )
+
+ routine = retry_403(Config.CLIENT.create_routine)(routine)
+ query_job = retry_403(Config.CLIENT.query)(query_string)
+ rows = list(query_job.result())
+
+ assert len(rows) == 1
+ assert rows[0].max_value == 100.0
+
+ def test_create_table_rows_fetch_nested_schema(self):
+ table_name = "test_table"
+ dataset = self.temp_dataset(_make_dataset_id("create_table_nested_schema"))
+ schema = _load_json_schema()
+ table_arg = Table(dataset.table(table_name), schema=schema)
+ table = retry_403(Config.CLIENT.create_table)(table_arg)
+ self.to_delete.insert(0, table)
+ self.assertTrue(_table_exists(table))
+ self.assertEqual(table.table_id, table_name)
+
+ to_insert = []
+ # Data is in "JSON Lines" format, see http://jsonlines.org/
+ json_filename = os.path.join(WHERE, "data", "characters.jsonl")
+ with open(json_filename) as rows_file:
+ for line in rows_file:
+ to_insert.append(json.loads(line))
+
+ errors = Config.CLIENT.insert_rows_json(table, to_insert)
+ self.assertEqual(len(errors), 0)
+
+ retry = RetryResult(_has_rows, max_tries=8)
+ fetched = retry(self._fetch_single_page)(table)
+ fetched_tuples = [f.values() for f in fetched]
+
+ self.assertEqual(len(fetched), len(to_insert))
+
+ for found, expected in zip(sorted(fetched_tuples), to_insert):
+ self.assertEqual(found[0], expected["Name"])
+ self.assertEqual(found[1], int(expected["Age"]))
+ self.assertEqual(found[2], expected["Weight"])
+ self.assertEqual(found[3], expected["IsMagic"])
+
+ self.assertEqual(len(found[4]), len(expected["Spells"]))
+ for f_spell, e_spell in zip(found[4], expected["Spells"]):
+ self.assertEqual(f_spell["Name"], e_spell["Name"])
+ parts = time.strptime(e_spell["LastUsed"], "%Y-%m-%d %H:%M:%S UTC")
+ e_used = datetime.datetime(*parts[0:6], tzinfo=UTC)
+ self.assertEqual(f_spell["LastUsed"], e_used)
+ self.assertEqual(f_spell["DiscoveredBy"], e_spell["DiscoveredBy"])
+ self.assertEqual(f_spell["Properties"], e_spell["Properties"])
+
+ e_icon = base64.standard_b64decode(e_spell["Icon"].encode("ascii"))
+ self.assertEqual(f_spell["Icon"], e_icon)
+
+ parts = time.strptime(expected["TeaTime"], "%H:%M:%S")
+ e_teatime = datetime.time(*parts[3:6])
+ self.assertEqual(found[5], e_teatime)
+
+ parts = time.strptime(expected["NextVacation"], "%Y-%m-%d")
+ e_nextvac = datetime.date(*parts[0:3])
+ self.assertEqual(found[6], e_nextvac)
+
+ parts = time.strptime(expected["FavoriteTime"], "%Y-%m-%dT%H:%M:%S")
+ e_favtime = datetime.datetime(*parts[0:6])
+ self.assertEqual(found[7], e_favtime)
+ self.assertEqual(found[8], decimal.Decimal(expected["FavoriteNumber"]))
+
+ def _fetch_dataframe(self, query):
+ return Config.CLIENT.query(query).result().to_dataframe()
+
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ @unittest.skipIf(
+ bigquery_storage_v1beta1 is None, "Requires `google-cloud-bigquery-storage`"
+ )
+ def test_nested_table_to_arrow(self):
+ from google.cloud.bigquery.job import SourceFormat
+ from google.cloud.bigquery.job import WriteDisposition
+
+ SF = bigquery.SchemaField
+ schema = [
+ SF("string_col", "STRING", mode="NULLABLE"),
+ SF(
+ "record_col",
+ "RECORD",
+ mode="NULLABLE",
+ fields=[
+ SF("nested_string", "STRING", mode="NULLABLE"),
+ SF("nested_repeated", "INTEGER", mode="REPEATED"),
+ ],
+ ),
+ SF("float_col", "FLOAT", mode="NULLABLE"),
+ ]
+ record = {"nested_string": "another string value", "nested_repeated": [0, 1, 2]}
+ to_insert = [
+ {"string_col": "Some value", "record_col": record, "float_col": 3.14}
+ ]
+ rows = [json.dumps(row) for row in to_insert]
+ body = six.BytesIO("{}\n".format("\n".join(rows)).encode("ascii"))
+ table_id = "test_table"
+ dataset = self.temp_dataset(_make_dataset_id("nested_df"))
+ table = dataset.table(table_id)
+ self.to_delete.insert(0, table)
+ job_config = bigquery.LoadJobConfig()
+ job_config.write_disposition = WriteDisposition.WRITE_TRUNCATE
+ job_config.source_format = SourceFormat.NEWLINE_DELIMITED_JSON
+ job_config.schema = schema
+ # Load a table using a local JSON file from memory.
+ Config.CLIENT.load_table_from_file(body, table, job_config=job_config).result()
+ bqstorage_client = bigquery_storage_v1beta1.BigQueryStorageClient(
+ credentials=Config.CLIENT._credentials
+ )
+
+ tbl = Config.CLIENT.list_rows(table, selected_fields=schema).to_arrow(
+ bqstorage_client=bqstorage_client
+ )
+
+ self.assertIsInstance(tbl, pyarrow.Table)
+ self.assertEqual(tbl.num_rows, 1)
+ self.assertEqual(tbl.num_columns, 3)
+ # Columns may not appear in the requested order.
+ self.assertTrue(
+ pyarrow.types.is_float64(tbl.schema.field_by_name("float_col").type)
+ )
+ self.assertTrue(
+ pyarrow.types.is_string(tbl.schema.field_by_name("string_col").type)
+ )
+ record_col = tbl.schema.field_by_name("record_col").type
+ self.assertTrue(pyarrow.types.is_struct(record_col))
+ self.assertEqual(record_col.num_children, 2)
+ self.assertEqual(record_col[0].name, "nested_string")
+ self.assertTrue(pyarrow.types.is_string(record_col[0].type))
+ self.assertEqual(record_col[1].name, "nested_repeated")
+ self.assertTrue(pyarrow.types.is_list(record_col[1].type))
+ self.assertTrue(pyarrow.types.is_int64(record_col[1].type.value_type))
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ def test_nested_table_to_dataframe(self):
+ from google.cloud.bigquery.job import SourceFormat
+ from google.cloud.bigquery.job import WriteDisposition
+
+ SF = bigquery.SchemaField
+ schema = [
+ SF("string_col", "STRING", mode="NULLABLE"),
+ SF(
+ "record_col",
+ "RECORD",
+ mode="NULLABLE",
+ fields=[
+ SF("nested_string", "STRING", mode="NULLABLE"),
+ SF("nested_repeated", "INTEGER", mode="REPEATED"),
+ SF(
+ "nested_record",
+ "RECORD",
+ mode="NULLABLE",
+ fields=[SF("nested_nested_string", "STRING", mode="NULLABLE")],
+ ),
+ ],
+ ),
+ SF("bigfloat_col", "FLOAT", mode="NULLABLE"),
+ SF("smallfloat_col", "FLOAT", mode="NULLABLE"),
+ ]
+ record = {
+ "nested_string": "another string value",
+ "nested_repeated": [0, 1, 2],
+ "nested_record": {"nested_nested_string": "some deep insight"},
+ }
+ to_insert = [
+ {
+ "string_col": "Some value",
+ "record_col": record,
+ "bigfloat_col": 3.14,
+ "smallfloat_col": 2.72,
+ }
+ ]
+ rows = [json.dumps(row) for row in to_insert]
+ body = six.BytesIO("{}\n".format("\n".join(rows)).encode("ascii"))
+ table_id = "test_table"
+ dataset = self.temp_dataset(_make_dataset_id("nested_df"))
+ table = dataset.table(table_id)
+ self.to_delete.insert(0, table)
+ job_config = bigquery.LoadJobConfig()
+ job_config.write_disposition = WriteDisposition.WRITE_TRUNCATE
+ job_config.source_format = SourceFormat.NEWLINE_DELIMITED_JSON
+ job_config.schema = schema
+ # Load a table using a local JSON file from memory.
+ Config.CLIENT.load_table_from_file(body, table, job_config=job_config).result()
+
+ df = Config.CLIENT.list_rows(table, selected_fields=schema).to_dataframe(
+ dtypes={"smallfloat_col": "float16"}
+ )
+
+ self.assertIsInstance(df, pandas.DataFrame)
+ self.assertEqual(len(df), 1) # verify the number of rows
+ exp_columns = ["string_col", "record_col", "bigfloat_col", "smallfloat_col"]
+ self.assertEqual(list(df), exp_columns) # verify the column names
+ row = df.iloc[0]
+ # verify the row content
+ self.assertEqual(row["string_col"], "Some value")
+ self.assertEqual(row["record_col"], record)
+ # verify that nested data can be accessed with indices/keys
+ self.assertEqual(row["record_col"]["nested_repeated"][0], 0)
+ self.assertEqual(
+ row["record_col"]["nested_record"]["nested_nested_string"],
+ "some deep insight",
+ )
+ # verify dtypes
+ self.assertEqual(df.dtypes["bigfloat_col"].name, "float64")
+ self.assertEqual(df.dtypes["smallfloat_col"].name, "float16")
+
+ def test_list_rows_empty_table(self):
+ from google.cloud.bigquery.table import RowIterator
+
+ dataset_id = _make_dataset_id("empty_table")
+ dataset = self.temp_dataset(dataset_id)
+ table_ref = dataset.table("empty_table")
+ table = Config.CLIENT.create_table(bigquery.Table(table_ref))
+
+ # It's a bit silly to list rows for an empty table, but this does
+ # happen as the result of a DDL query from an IPython magic command.
+ rows = Config.CLIENT.list_rows(table)
+ self.assertIsInstance(rows, RowIterator)
+ self.assertEqual(tuple(rows), ())
+
+ def test_list_rows_page_size(self):
+ from google.cloud.bigquery.job import SourceFormat
+ from google.cloud.bigquery.job import WriteDisposition
+
+ num_items = 7
+ page_size = 3
+ num_pages, num_last_page = divmod(num_items, page_size)
+
+ SF = bigquery.SchemaField
+ schema = [SF("string_col", "STRING", mode="NULLABLE")]
+ to_insert = [{"string_col": "item%d" % i} for i in range(num_items)]
+ rows = [json.dumps(row) for row in to_insert]
+ body = six.BytesIO("{}\n".format("\n".join(rows)).encode("ascii"))
+
+ table_id = "test_table"
+ dataset = self.temp_dataset(_make_dataset_id("nested_df"))
+ table = dataset.table(table_id)
+ self.to_delete.insert(0, table)
+ job_config = bigquery.LoadJobConfig()
+ job_config.write_disposition = WriteDisposition.WRITE_TRUNCATE
+ job_config.source_format = SourceFormat.NEWLINE_DELIMITED_JSON
+ job_config.schema = schema
+ # Load a table using a local JSON file from memory.
+ Config.CLIENT.load_table_from_file(body, table, job_config=job_config).result()
+
+ df = Config.CLIENT.list_rows(table, selected_fields=schema, page_size=page_size)
+ pages = df.pages
+
+ for i in range(num_pages):
+ page = next(pages)
+ self.assertEqual(page.num_items, page_size)
+ page = next(pages)
+ self.assertEqual(page.num_items, num_last_page)
+
+ def temp_dataset(self, dataset_id, location=None):
+ dataset = Dataset(Config.CLIENT.dataset(dataset_id))
+ if location:
+ dataset.location = location
+ dataset = retry_403(Config.CLIENT.create_dataset)(dataset)
+ self.to_delete.append(dataset)
+ return dataset
+
+
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+@pytest.mark.skipif(IPython is None, reason="Requires `ipython`")
+@pytest.mark.usefixtures("ipython_interactive")
+def test_bigquery_magic():
+ ip = IPython.get_ipython()
+ ip.extension_manager.load_extension("google.cloud.bigquery")
+ sql = """
+ SELECT
+ CONCAT(
+ 'https://stackoverflow.com/questions/',
+ CAST(id as STRING)) as url,
+ view_count
+ FROM `bigquery-public-data.stackoverflow.posts_questions`
+ WHERE tags like '%google-bigquery%'
+ ORDER BY view_count DESC
+ LIMIT 10
+ """
+ with io.capture_output() as captured:
+ result = ip.run_cell_magic("bigquery", "", sql)
+
+ lines = re.split("\n|\r", captured.stdout)
+ # Removes blanks & terminal code (result of display clearing)
+ updates = list(filter(lambda x: bool(x) and x != "\x1b[2K", lines))
+ assert re.match("Executing query with job ID: .*", updates[0])
+ assert all(re.match("Query executing: .*s", line) for line in updates[1:-1])
+ assert re.match("Query complete after .*s", updates[-1])
+ assert isinstance(result, pandas.DataFrame)
+ assert len(result) == 10 # verify row count
+ assert list(result) == ["url", "view_count"] # verify column names
+
+
+def _job_done(instance):
+ return instance.state.lower() == "done"
+
+
+def _dataset_exists(ds):
+ try:
+ Config.CLIENT.get_dataset(DatasetReference(ds.project, ds.dataset_id))
+ return True
+ except NotFound:
+ return False
+
+
+def _table_exists(t):
+ try:
+ tr = DatasetReference(t.project, t.dataset_id).table(t.table_id)
+ Config.CLIENT.get_table(tr)
+ return True
+ except NotFound:
+ return False
+
+
+@pytest.fixture(scope="session")
+def ipython():
+ config = tools.default_config()
+ config.TerminalInteractiveShell.simple_prompt = True
+ shell = interactiveshell.TerminalInteractiveShell.instance(config=config)
+ return shell
+
+
+@pytest.fixture()
+def ipython_interactive(request, ipython):
+ """Activate IPython's builtin hooks
+
+ for the duration of the test scope.
+ """
+ with ipython.builtin_trap:
+ yield ipython
diff --git a/bigquery/tests/unit/__init__.py b/bigquery/tests/unit/__init__.py
new file mode 100644
index 000000000000..df379f1e9d88
--- /dev/null
+++ b/bigquery/tests/unit/__init__.py
@@ -0,0 +1,13 @@
+# Copyright 2016 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
diff --git a/bigquery/tests/unit/enums/__init__.py b/bigquery/tests/unit/enums/__init__.py
new file mode 100644
index 000000000000..c5cce043083c
--- /dev/null
+++ b/bigquery/tests/unit/enums/__init__.py
@@ -0,0 +1,13 @@
+# Copyright 2019, Google LLC All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
diff --git a/bigquery/tests/unit/enums/test_standard_sql_data_types.py b/bigquery/tests/unit/enums/test_standard_sql_data_types.py
new file mode 100644
index 000000000000..6fa4f057fb98
--- /dev/null
+++ b/bigquery/tests/unit/enums/test_standard_sql_data_types.py
@@ -0,0 +1,73 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import pytest
+
+
+@pytest.fixture
+def module_under_test():
+ from google.cloud.bigquery import enums
+
+ return enums
+
+
+@pytest.fixture
+def enum_under_test():
+ from google.cloud.bigquery.enums import StandardSqlDataTypes
+
+ return StandardSqlDataTypes
+
+
+@pytest.fixture
+def gapic_enum():
+ """The referential autogenerated enum the enum under test is based on."""
+ from google.cloud.bigquery_v2.gapic.enums import StandardSqlDataType
+
+ return StandardSqlDataType.TypeKind
+
+
+def test_all_gapic_enum_members_are_known(module_under_test, gapic_enum):
+ gapic_names = set(type_.name for type_ in gapic_enum)
+ anticipated_names = (
+ module_under_test._SQL_SCALAR_TYPES | module_under_test._SQL_NONSCALAR_TYPES
+ )
+ assert not (gapic_names - anticipated_names) # no unhandled names
+
+
+def test_standard_sql_types_enum_members(enum_under_test, gapic_enum):
+ # check the presence of a few typical SQL types
+ for name in ("INT64", "FLOAT64", "DATE", "BOOL", "GEOGRAPHY"):
+ assert name in enum_under_test.__members__
+
+ # the enum members must match those in the original gapic enum
+ for member in enum_under_test:
+ assert member.name in gapic_enum.__members__
+ assert member.value == gapic_enum[member.name].value
+
+ # check a few members that should *not* be copied over from the gapic enum
+ for name in ("STRUCT", "ARRAY"):
+ assert name in gapic_enum.__members__
+ assert name not in enum_under_test.__members__
+
+
+def test_standard_sql_types_enum_docstring(enum_under_test, gapic_enum):
+ assert "STRUCT (int):" not in enum_under_test.__doc__
+ assert "BOOL (int):" in enum_under_test.__doc__
+ assert "TIME (int):" in enum_under_test.__doc__
+
+ # All lines in the docstring should actually come from the original docstring,
+ # except for the header.
+ assert "An Enum of scalar SQL types." in enum_under_test.__doc__
+ doc_lines = enum_under_test.__doc__.splitlines()
+ assert set(doc_lines[1:]) <= set(gapic_enum.__doc__.splitlines())
diff --git a/bigquery/tests/unit/helpers.py b/bigquery/tests/unit/helpers.py
new file mode 100644
index 000000000000..5b731a763a99
--- /dev/null
+++ b/bigquery/tests/unit/helpers.py
@@ -0,0 +1,24 @@
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+def make_connection(*responses):
+ import google.cloud.bigquery._http
+ import mock
+ from google.cloud.exceptions import NotFound
+
+ mock_conn = mock.create_autospec(google.cloud.bigquery._http.Connection)
+ mock_conn.user_agent = "testing 1.2.3"
+ mock_conn.api_request.side_effect = list(responses) + [NotFound("miss")]
+ return mock_conn
diff --git a/bigquery/tests/unit/model/__init__.py b/bigquery/tests/unit/model/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/bigquery/tests/unit/model/test_model.py b/bigquery/tests/unit/model/test_model.py
new file mode 100644
index 000000000000..b6d9756e15fe
--- /dev/null
+++ b/bigquery/tests/unit/model/test_model.py
@@ -0,0 +1,292 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import datetime
+
+import pytest
+
+import google.cloud._helpers
+from google.cloud.bigquery_v2.gapic import enums
+
+
+@pytest.fixture
+def target_class():
+ from google.cloud.bigquery import Model
+
+ return Model
+
+
+@pytest.fixture
+def object_under_test(target_class):
+ return target_class("project-id.dataset_id.model_id")
+
+
+def test_ctor(target_class):
+ from google.cloud.bigquery import ModelReference
+
+ ref = ModelReference.from_string("my-proj.my_dset.my_model")
+ got = target_class(ref)
+ assert got.reference == ref
+
+
+def test_ctor_string(target_class):
+ from google.cloud.bigquery import ModelReference
+
+ model_id = "my-proj.my_dset.my_model"
+ ref = ModelReference.from_string(model_id)
+ got = target_class(model_id)
+ assert got.reference == ref
+
+
+def test_from_api_repr(target_class):
+ from google.cloud.bigquery import ModelReference
+
+ creation_time = datetime.datetime(
+ 2010, 5, 19, 16, 0, 0, tzinfo=google.cloud._helpers.UTC
+ )
+ modified_time = datetime.datetime(
+ 2011, 10, 1, 16, 0, 0, tzinfo=google.cloud._helpers.UTC
+ )
+ expiration_time = datetime.datetime(
+ 2012, 12, 21, 16, 0, 0, tzinfo=google.cloud._helpers.UTC
+ )
+ resource = {
+ "modelReference": {
+ "projectId": "my-project",
+ "datasetId": "my_dataset",
+ "modelId": "my_model",
+ },
+ "location": "US",
+ "etag": "abcdefg",
+ "creationTime": str(google.cloud._helpers._millis(creation_time)),
+ "lastModifiedTime": str(google.cloud._helpers._millis(modified_time)),
+ "expirationTime": str(google.cloud._helpers._millis(expiration_time)),
+ "description": "A friendly description.",
+ "friendlyName": "A friendly name.",
+ "modelType": "LOGISTIC_REGRESSION",
+ "labels": {"greeting": u"こんにちは"},
+ "trainingRuns": [
+ {
+ "trainingOptions": {"initialLearnRate": 1.0},
+ "startTime": str(
+ google.cloud._helpers._datetime_to_rfc3339(creation_time)
+ ),
+ },
+ {
+ "trainingOptions": {"initialLearnRate": 0.5},
+ "startTime": str(
+ google.cloud._helpers._datetime_to_rfc3339(modified_time)
+ ),
+ },
+ {
+ "trainingOptions": {"initialLearnRate": 0.25},
+ # Allow milliseconds since epoch format.
+ # TODO: Remove this hack once CL 238585470 hits prod.
+ "startTime": str(google.cloud._helpers._millis(expiration_time)),
+ },
+ ],
+ "featureColumns": [],
+ }
+ got = target_class.from_api_repr(resource)
+
+ assert got.project == "my-project"
+ assert got.dataset_id == "my_dataset"
+ assert got.model_id == "my_model"
+ assert got.reference == ModelReference.from_string("my-project.my_dataset.my_model")
+ assert got.path == "/projects/my-project/datasets/my_dataset/models/my_model"
+ assert got.location == "US"
+ assert got.etag == "abcdefg"
+ assert got.created == creation_time
+ assert got.modified == modified_time
+ assert got.expires == expiration_time
+ assert got.description == u"A friendly description."
+ assert got.friendly_name == u"A friendly name."
+ assert got.model_type == enums.Model.ModelType.LOGISTIC_REGRESSION
+ assert got.labels == {"greeting": u"こんにちは"}
+ assert got.training_runs[0].training_options.initial_learn_rate == 1.0
+ assert (
+ got.training_runs[0]
+ .start_time.ToDatetime()
+ .replace(tzinfo=google.cloud._helpers.UTC)
+ == creation_time
+ )
+ assert got.training_runs[1].training_options.initial_learn_rate == 0.5
+ assert (
+ got.training_runs[1]
+ .start_time.ToDatetime()
+ .replace(tzinfo=google.cloud._helpers.UTC)
+ == modified_time
+ )
+ assert got.training_runs[2].training_options.initial_learn_rate == 0.25
+ assert (
+ got.training_runs[2]
+ .start_time.ToDatetime()
+ .replace(tzinfo=google.cloud._helpers.UTC)
+ == expiration_time
+ )
+
+
+def test_from_api_repr_w_minimal_resource(target_class):
+ from google.cloud.bigquery import ModelReference
+
+ resource = {
+ "modelReference": {
+ "projectId": "my-project",
+ "datasetId": "my_dataset",
+ "modelId": "my_model",
+ }
+ }
+ got = target_class.from_api_repr(resource)
+ assert got.reference == ModelReference.from_string("my-project.my_dataset.my_model")
+ assert got.location == ""
+ assert got.etag == ""
+ assert got.created is None
+ assert got.modified is None
+ assert got.expires is None
+ assert got.description is None
+ assert got.friendly_name is None
+ assert got.model_type == enums.Model.ModelType.MODEL_TYPE_UNSPECIFIED
+ assert got.labels == {}
+ assert len(got.training_runs) == 0
+ assert len(got.feature_columns) == 0
+ assert len(got.label_columns) == 0
+
+
+def test_from_api_repr_w_unknown_fields(target_class):
+ from google.cloud.bigquery import ModelReference
+
+ resource = {
+ "modelReference": {
+ "projectId": "my-project",
+ "datasetId": "my_dataset",
+ "modelId": "my_model",
+ },
+ "thisFieldIsNotInTheProto": "just ignore me",
+ }
+ got = target_class.from_api_repr(resource)
+ assert got.reference == ModelReference.from_string("my-project.my_dataset.my_model")
+ assert got._properties is resource
+
+
+@pytest.mark.parametrize(
+ "resource,filter_fields,expected",
+ [
+ (
+ {
+ "friendlyName": "hello",
+ "description": "world",
+ "expirationTime": "12345",
+ "labels": {"a-label": "a-value"},
+ },
+ ["description"],
+ {"description": "world"},
+ ),
+ (
+ {"friendlyName": "hello", "description": "world"},
+ ["friendlyName"],
+ {"friendlyName": "hello"},
+ ),
+ (
+ {
+ "friendlyName": "hello",
+ "description": "world",
+ "expirationTime": "12345",
+ "labels": {"a-label": "a-value"},
+ },
+ ["expires"],
+ {"expirationTime": "12345"},
+ ),
+ (
+ {
+ "friendlyName": "hello",
+ "description": "world",
+ "expirationTime": None,
+ "labels": {"a-label": "a-value"},
+ },
+ ["expires"],
+ {"expirationTime": None},
+ ),
+ (
+ {
+ "friendlyName": "hello",
+ "description": "world",
+ "expirationTime": None,
+ "labels": {"a-label": "a-value"},
+ },
+ ["labels"],
+ {"labels": {"a-label": "a-value"}},
+ ),
+ ],
+)
+def test_build_resource(object_under_test, resource, filter_fields, expected):
+ object_under_test._properties = resource
+ got = object_under_test._build_resource(filter_fields)
+ assert got == expected
+
+
+def test_set_description(object_under_test):
+ assert not object_under_test.description
+ object_under_test.description = "A model description."
+ assert object_under_test.description == "A model description."
+ object_under_test.description = None
+ assert not object_under_test.description
+
+
+def test_set_expires(object_under_test):
+ assert not object_under_test.expires
+ expiration_time = datetime.datetime(
+ 2012, 12, 21, 16, 0, 0, tzinfo=google.cloud._helpers.UTC
+ )
+ object_under_test.expires = expiration_time
+ assert object_under_test.expires == expiration_time
+ object_under_test.expires = None
+ assert not object_under_test.expires
+
+
+def test_set_friendly_name(object_under_test):
+ assert not object_under_test.friendly_name
+ object_under_test.friendly_name = "A model name."
+ assert object_under_test.friendly_name == "A model name."
+ object_under_test.friendly_name = None
+ assert not object_under_test.friendly_name
+
+
+def test_set_labels(object_under_test):
+ assert object_under_test.labels == {}
+ object_under_test.labels["data_owner"] = "someteam"
+ assert object_under_test.labels == {"data_owner": "someteam"}
+ del object_under_test.labels["data_owner"]
+ assert object_under_test.labels == {}
+
+
+def test_replace_labels(object_under_test):
+ assert object_under_test.labels == {}
+ object_under_test.labels = {"data_owner": "someteam"}
+ assert object_under_test.labels == {"data_owner": "someteam"}
+ labels = {}
+ object_under_test.labels = labels
+ assert object_under_test.labels is labels
+ object_under_test.labels = None
+ assert object_under_test.labels == {}
+
+
+def test_repr(target_class):
+ model = target_class("my-proj.my_dset.my_model")
+ got = repr(model)
+ assert got == (
+ "Model(reference=ModelReference("
+ "project='my-proj', dataset_id='my_dset', project_id='my_model'))"
+ )
diff --git a/bigquery/tests/unit/model/test_model_reference.py b/bigquery/tests/unit/model/test_model_reference.py
new file mode 100644
index 000000000000..ff1d1df7d499
--- /dev/null
+++ b/bigquery/tests/unit/model/test_model_reference.py
@@ -0,0 +1,140 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import pytest
+
+
+@pytest.fixture
+def target_class():
+ from google.cloud.bigquery import ModelReference
+
+ return ModelReference
+
+
+def test_from_api_repr(target_class):
+ resource = {
+ "projectId": "my-project",
+ "datasetId": "my_dataset",
+ "modelId": "my_model",
+ }
+ got = target_class.from_api_repr(resource)
+ assert got.project == "my-project"
+ assert got.dataset_id == "my_dataset"
+ assert got.model_id == "my_model"
+ assert got.path == "/projects/my-project/datasets/my_dataset/models/my_model"
+
+
+def test_from_api_repr_w_unknown_fields(target_class):
+ resource = {
+ "projectId": "my-project",
+ "datasetId": "my_dataset",
+ "modelId": "my_model",
+ "thisFieldIsNotInTheProto": "just ignore me",
+ }
+ got = target_class.from_api_repr(resource)
+ assert got.project == "my-project"
+ assert got.dataset_id == "my_dataset"
+ assert got.model_id == "my_model"
+ assert got._properties is resource
+
+
+def test_to_api_repr(target_class):
+ ref = target_class.from_string("my-project.my_dataset.my_model")
+ got = ref.to_api_repr()
+ assert got == {
+ "projectId": "my-project",
+ "datasetId": "my_dataset",
+ "modelId": "my_model",
+ }
+
+
+def test_from_string(target_class):
+ got = target_class.from_string("string-project.string_dataset.string_model")
+ assert got.project == "string-project"
+ assert got.dataset_id == "string_dataset"
+ assert got.model_id == "string_model"
+ assert got.path == (
+ "/projects/string-project/datasets/string_dataset/models/string_model"
+ )
+
+
+def test_from_string_legacy_string(target_class):
+ with pytest.raises(ValueError):
+ target_class.from_string("string-project:string_dataset.string_model")
+
+
+def test_from_string_not_fully_qualified(target_class):
+ with pytest.raises(ValueError):
+ target_class.from_string("string_model")
+
+ with pytest.raises(ValueError):
+ target_class.from_string("string_dataset.string_model")
+
+ with pytest.raises(ValueError):
+ target_class.from_string("a.b.c.d")
+
+
+def test_from_string_with_default_project(target_class):
+ got = target_class.from_string(
+ "string_dataset.string_model", default_project="default-project"
+ )
+ assert got.project == "default-project"
+ assert got.dataset_id == "string_dataset"
+ assert got.model_id == "string_model"
+
+
+def test_from_string_ignores_default_project(target_class):
+ got = target_class.from_string(
+ "string-project.string_dataset.string_model", default_project="default-project"
+ )
+ assert got.project == "string-project"
+ assert got.dataset_id == "string_dataset"
+ assert got.model_id == "string_model"
+
+
+def test_eq(target_class):
+ model = target_class.from_string("my-proj.my_dset.my_model")
+ model_too = target_class.from_string("my-proj.my_dset.my_model")
+ assert model == model_too
+ assert not (model != model_too)
+
+ other_model = target_class.from_string("my-proj.my_dset.my_model2")
+ assert not (model == other_model)
+ assert model != other_model
+
+ notamodel = object()
+ assert not (model == notamodel)
+ assert model != notamodel
+
+
+def test_hash(target_class):
+ model = target_class.from_string("my-proj.my_dset.my_model")
+ model2 = target_class.from_string("my-proj.my_dset.model2")
+ got = {model: "hello", model2: "world"}
+ assert got[model] == "hello"
+ assert got[model2] == "world"
+
+ model_too = target_class.from_string("my-proj.my_dset.my_model")
+ assert got[model_too] == "hello"
+
+
+def test_repr(target_class):
+ model = target_class.from_string("my-proj.my_dset.my_model")
+ got = repr(model)
+ assert (
+ got
+ == "ModelReference(project='my-proj', dataset_id='my_dset', project_id='my_model')"
+ )
diff --git a/bigquery/tests/unit/routine/__init__.py b/bigquery/tests/unit/routine/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/bigquery/tests/unit/routine/test_routine.py b/bigquery/tests/unit/routine/test_routine.py
new file mode 100644
index 000000000000..02d4a2ee2883
--- /dev/null
+++ b/bigquery/tests/unit/routine/test_routine.py
@@ -0,0 +1,305 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import datetime
+
+import pytest
+
+import google.cloud._helpers
+from google.cloud import bigquery_v2
+
+
+@pytest.fixture
+def target_class():
+ from google.cloud.bigquery.routine import Routine
+
+ return Routine
+
+
+@pytest.fixture
+def object_under_test(target_class):
+ return target_class("project-id.dataset_id.routine_id")
+
+
+def test_ctor(target_class):
+ from google.cloud.bigquery.routine import RoutineReference
+
+ ref = RoutineReference.from_string("my-proj.my_dset.my_routine")
+ actual_routine = target_class(ref)
+ assert actual_routine.reference == ref
+ assert (
+ actual_routine.path == "/projects/my-proj/datasets/my_dset/routines/my_routine"
+ )
+
+
+def test_ctor_w_string(target_class):
+ from google.cloud.bigquery.routine import RoutineReference
+
+ routine_id = "my-proj.my_dset.my_routine"
+ ref = RoutineReference.from_string(routine_id)
+ actual_routine = target_class(routine_id)
+ assert actual_routine.reference == ref
+
+
+def test_ctor_w_properties(target_class):
+ from google.cloud.bigquery.routine import RoutineArgument
+ from google.cloud.bigquery.routine import RoutineReference
+
+ routine_id = "my-proj.my_dset.my_routine"
+ arguments = [
+ RoutineArgument(
+ name="x",
+ data_type=bigquery_v2.types.StandardSqlDataType(
+ type_kind=bigquery_v2.enums.StandardSqlDataType.TypeKind.INT64
+ ),
+ )
+ ]
+ body = "x * 3"
+ language = "SQL"
+ return_type = bigquery_v2.types.StandardSqlDataType(
+ type_kind=bigquery_v2.enums.StandardSqlDataType.TypeKind.INT64
+ )
+ type_ = "SCALAR_FUNCTION"
+
+ actual_routine = target_class(
+ routine_id,
+ arguments=arguments,
+ body=body,
+ language=language,
+ return_type=return_type,
+ type_=type_,
+ )
+
+ ref = RoutineReference.from_string(routine_id)
+ assert actual_routine.reference == ref
+ assert actual_routine.arguments == arguments
+ assert actual_routine.body == body
+ assert actual_routine.language == language
+ assert actual_routine.return_type == return_type
+ assert actual_routine.type_ == type_
+
+
+def test_from_api_repr(target_class):
+ from google.cloud.bigquery.routine import RoutineArgument
+ from google.cloud.bigquery.routine import RoutineReference
+
+ creation_time = datetime.datetime(
+ 2010, 5, 19, 16, 0, 0, tzinfo=google.cloud._helpers.UTC
+ )
+ modified_time = datetime.datetime(
+ 2011, 10, 1, 16, 0, 0, tzinfo=google.cloud._helpers.UTC
+ )
+ resource = {
+ "routineReference": {
+ "projectId": "my-project",
+ "datasetId": "my_dataset",
+ "routineId": "my_routine",
+ },
+ "etag": "abcdefg",
+ "creationTime": str(google.cloud._helpers._millis(creation_time)),
+ "lastModifiedTime": str(google.cloud._helpers._millis(modified_time)),
+ "definitionBody": "42",
+ "arguments": [{"name": "x", "dataType": {"typeKind": "INT64"}}],
+ "language": "SQL",
+ "returnType": {"typeKind": "INT64"},
+ "routineType": "SCALAR_FUNCTION",
+ "someNewField": "someValue",
+ }
+ actual_routine = target_class.from_api_repr(resource)
+
+ assert actual_routine.project == "my-project"
+ assert actual_routine.dataset_id == "my_dataset"
+ assert actual_routine.routine_id == "my_routine"
+ assert (
+ actual_routine.path
+ == "/projects/my-project/datasets/my_dataset/routines/my_routine"
+ )
+ assert actual_routine.reference == RoutineReference.from_string(
+ "my-project.my_dataset.my_routine"
+ )
+ assert actual_routine.etag == "abcdefg"
+ assert actual_routine.created == creation_time
+ assert actual_routine.modified == modified_time
+ assert actual_routine.arguments == [
+ RoutineArgument(
+ name="x",
+ data_type=bigquery_v2.types.StandardSqlDataType(
+ type_kind=bigquery_v2.enums.StandardSqlDataType.TypeKind.INT64
+ ),
+ )
+ ]
+ assert actual_routine.body == "42"
+ assert actual_routine.language == "SQL"
+ assert actual_routine.return_type == bigquery_v2.types.StandardSqlDataType(
+ type_kind=bigquery_v2.enums.StandardSqlDataType.TypeKind.INT64
+ )
+ assert actual_routine.type_ == "SCALAR_FUNCTION"
+ assert actual_routine._properties["someNewField"] == "someValue"
+
+
+def test_from_api_repr_w_minimal_resource(target_class):
+ from google.cloud.bigquery.routine import RoutineReference
+
+ resource = {
+ "routineReference": {
+ "projectId": "my-project",
+ "datasetId": "my_dataset",
+ "routineId": "my_routine",
+ }
+ }
+ actual_routine = target_class.from_api_repr(resource)
+ assert actual_routine.reference == RoutineReference.from_string(
+ "my-project.my_dataset.my_routine"
+ )
+ assert actual_routine.etag is None
+ assert actual_routine.created is None
+ assert actual_routine.modified is None
+ assert actual_routine.arguments == []
+ assert actual_routine.body is None
+ assert actual_routine.language is None
+ assert actual_routine.return_type is None
+ assert actual_routine.type_ is None
+
+
+def test_from_api_repr_w_unknown_fields(target_class):
+ from google.cloud.bigquery.routine import RoutineReference
+
+ resource = {
+ "routineReference": {
+ "projectId": "my-project",
+ "datasetId": "my_dataset",
+ "routineId": "my_routine",
+ },
+ "thisFieldIsNotInTheProto": "just ignore me",
+ }
+ actual_routine = target_class.from_api_repr(resource)
+ assert actual_routine.reference == RoutineReference.from_string(
+ "my-project.my_dataset.my_routine"
+ )
+ assert actual_routine._properties is resource
+
+
+@pytest.mark.parametrize(
+ "resource,filter_fields,expected",
+ [
+ (
+ {
+ "arguments": [{"name": "x", "dataType": {"typeKind": "INT64"}}],
+ "definitionBody": "x * 3",
+ "language": "SQL",
+ "returnType": {"typeKind": "INT64"},
+ "routineType": "SCALAR_FUNCTION",
+ },
+ ["arguments"],
+ {"arguments": [{"name": "x", "dataType": {"typeKind": "INT64"}}]},
+ ),
+ (
+ {
+ "arguments": [{"name": "x", "dataType": {"typeKind": "INT64"}}],
+ "definitionBody": "x * 3",
+ "language": "SQL",
+ "returnType": {"typeKind": "INT64"},
+ "routineType": "SCALAR_FUNCTION",
+ },
+ ["body"],
+ {"definitionBody": "x * 3"},
+ ),
+ (
+ {
+ "arguments": [{"name": "x", "dataType": {"typeKind": "INT64"}}],
+ "definitionBody": "x * 3",
+ "language": "SQL",
+ "returnType": {"typeKind": "INT64"},
+ "routineType": "SCALAR_FUNCTION",
+ },
+ ["language"],
+ {"language": "SQL"},
+ ),
+ (
+ {
+ "arguments": [{"name": "x", "dataType": {"typeKind": "INT64"}}],
+ "definitionBody": "x * 3",
+ "language": "SQL",
+ "returnType": {"typeKind": "INT64"},
+ "routineType": "SCALAR_FUNCTION",
+ },
+ ["return_type"],
+ {"returnType": {"typeKind": "INT64"}},
+ ),
+ (
+ {
+ "arguments": [{"name": "x", "dataType": {"typeKind": "INT64"}}],
+ "definitionBody": "x * 3",
+ "language": "SQL",
+ "returnType": {"typeKind": "INT64"},
+ "routineType": "SCALAR_FUNCTION",
+ },
+ ["type_"],
+ {"routineType": "SCALAR_FUNCTION"},
+ ),
+ (
+ {},
+ ["arguments", "language", "body", "type_", "return_type"],
+ {
+ "arguments": None,
+ "definitionBody": None,
+ "language": None,
+ "returnType": None,
+ "routineType": None,
+ },
+ ),
+ (
+ {"someNewField": "someValue"},
+ ["someNewField"],
+ {"someNewField": "someValue"},
+ ),
+ ],
+)
+def test_build_resource(object_under_test, resource, filter_fields, expected):
+ object_under_test._properties = resource
+ actual_routine = object_under_test._build_resource(filter_fields)
+ assert actual_routine == expected
+
+
+def test_set_arguments_w_none(object_under_test):
+ object_under_test.arguments = None
+ assert object_under_test.arguments == []
+ assert object_under_test._properties["arguments"] == []
+
+
+def test_set_imported_libraries(object_under_test):
+ imported_libraries = ["gs://cloud-samples-data/bigquery/udfs/max-value.js"]
+ object_under_test.imported_libraries = imported_libraries
+ assert object_under_test.imported_libraries == imported_libraries
+ assert object_under_test._properties["importedLibraries"] == imported_libraries
+
+
+def test_set_imported_libraries_w_none(object_under_test):
+ object_under_test.imported_libraries = None
+ assert object_under_test.imported_libraries == []
+ assert object_under_test._properties["importedLibraries"] == []
+
+
+def test_set_return_type_w_none(object_under_test):
+ object_under_test.return_type = None
+ assert object_under_test.return_type is None
+ assert object_under_test._properties["returnType"] is None
+
+
+def test_repr(target_class):
+ model = target_class("my-proj.my_dset.my_routine")
+ actual_routine = repr(model)
+ assert actual_routine == "Routine('my-proj.my_dset.my_routine')"
diff --git a/bigquery/tests/unit/routine/test_routine_argument.py b/bigquery/tests/unit/routine/test_routine_argument.py
new file mode 100644
index 000000000000..7d17b5fc703f
--- /dev/null
+++ b/bigquery/tests/unit/routine/test_routine_argument.py
@@ -0,0 +1,100 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import pytest
+
+from google.cloud import bigquery_v2
+
+
+@pytest.fixture
+def target_class():
+ from google.cloud.bigquery.routine import RoutineArgument
+
+ return RoutineArgument
+
+
+def test_ctor(target_class):
+ data_type = bigquery_v2.types.StandardSqlDataType(
+ type_kind=bigquery_v2.enums.StandardSqlDataType.TypeKind.INT64
+ )
+ actual_arg = target_class(
+ name="field_name", kind="FIXED_TYPE", mode="IN", data_type=data_type
+ )
+ assert actual_arg.name == "field_name"
+ assert actual_arg.kind == "FIXED_TYPE"
+ assert actual_arg.mode == "IN"
+ assert actual_arg.data_type == data_type
+
+
+def test_from_api_repr(target_class):
+ resource = {
+ "argumentKind": "FIXED_TYPE",
+ "dataType": {"typeKind": "INT64"},
+ "mode": "IN",
+ "name": "field_name",
+ }
+ actual_arg = target_class.from_api_repr(resource)
+ assert actual_arg.name == "field_name"
+ assert actual_arg.kind == "FIXED_TYPE"
+ assert actual_arg.mode == "IN"
+ assert actual_arg.data_type == bigquery_v2.types.StandardSqlDataType(
+ type_kind=bigquery_v2.enums.StandardSqlDataType.TypeKind.INT64
+ )
+
+
+def test_from_api_repr_w_minimal_resource(target_class):
+ resource = {}
+ actual_arg = target_class.from_api_repr(resource)
+ assert actual_arg.name is None
+ assert actual_arg.kind is None
+ assert actual_arg.mode is None
+ assert actual_arg.data_type is None
+
+
+def test_from_api_repr_w_unknown_fields(target_class):
+ resource = {"thisFieldIsNotInTheProto": "just ignore me"}
+ actual_arg = target_class.from_api_repr(resource)
+ assert actual_arg._properties is resource
+
+
+def test_eq(target_class):
+ data_type = bigquery_v2.types.StandardSqlDataType(
+ type_kind=bigquery_v2.enums.StandardSqlDataType.TypeKind.INT64
+ )
+ arg = target_class(
+ name="field_name", kind="FIXED_TYPE", mode="IN", data_type=data_type
+ )
+ arg_too = target_class(
+ name="field_name", kind="FIXED_TYPE", mode="IN", data_type=data_type
+ )
+ assert arg == arg_too
+ assert not (arg != arg_too)
+
+ other_arg = target_class()
+ assert not (arg == other_arg)
+ assert arg != other_arg
+
+ notanarg = object()
+ assert not (arg == notanarg)
+ assert arg != notanarg
+
+
+def test_repr(target_class):
+ arg = target_class(name="field_name", kind="FIXED_TYPE", mode="IN", data_type=None)
+ actual_repr = repr(arg)
+ assert actual_repr == (
+ "RoutineArgument(data_type=None, kind='FIXED_TYPE', mode='IN', name='field_name')"
+ )
diff --git a/bigquery/tests/unit/routine/test_routine_reference.py b/bigquery/tests/unit/routine/test_routine_reference.py
new file mode 100644
index 000000000000..9d3d551a6294
--- /dev/null
+++ b/bigquery/tests/unit/routine/test_routine_reference.py
@@ -0,0 +1,138 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import pytest
+
+
+@pytest.fixture
+def target_class():
+ from google.cloud.bigquery.routine import RoutineReference
+
+ return RoutineReference
+
+
+def test_from_api_repr(target_class):
+ resource = {
+ "projectId": "my-project",
+ "datasetId": "my_dataset",
+ "routineId": "my_routine",
+ }
+ got = target_class.from_api_repr(resource)
+ assert got.project == "my-project"
+ assert got.dataset_id == "my_dataset"
+ assert got.routine_id == "my_routine"
+ assert got.path == "/projects/my-project/datasets/my_dataset/routines/my_routine"
+
+
+def test_from_api_repr_w_unknown_fields(target_class):
+ resource = {
+ "projectId": "my-project",
+ "datasetId": "my_dataset",
+ "routineId": "my_routine",
+ "thisFieldIsNotInTheProto": "just ignore me",
+ }
+ got = target_class.from_api_repr(resource)
+ assert got.project == "my-project"
+ assert got.dataset_id == "my_dataset"
+ assert got.routine_id == "my_routine"
+ assert got._properties is resource
+
+
+def test_to_api_repr(target_class):
+ ref = target_class.from_string("my-project.my_dataset.my_routine")
+ got = ref.to_api_repr()
+ assert got == {
+ "projectId": "my-project",
+ "datasetId": "my_dataset",
+ "routineId": "my_routine",
+ }
+
+
+def test_from_string(target_class):
+ got = target_class.from_string("string-project.string_dataset.string_routine")
+ assert got.project == "string-project"
+ assert got.dataset_id == "string_dataset"
+ assert got.routine_id == "string_routine"
+ assert got.path == (
+ "/projects/string-project/datasets/string_dataset/routines/string_routine"
+ )
+
+
+def test_from_string_legacy_string(target_class):
+ with pytest.raises(ValueError):
+ target_class.from_string("string-project:string_dataset.string_routine")
+
+
+def test_from_string_not_fully_qualified(target_class):
+ with pytest.raises(ValueError):
+ target_class.from_string("string_routine")
+
+ with pytest.raises(ValueError):
+ target_class.from_string("string_dataset.string_routine")
+
+ with pytest.raises(ValueError):
+ target_class.from_string("a.b.c.d")
+
+
+def test_from_string_with_default_project(target_class):
+ got = target_class.from_string(
+ "string_dataset.string_routine", default_project="default-project"
+ )
+ assert got.project == "default-project"
+ assert got.dataset_id == "string_dataset"
+ assert got.routine_id == "string_routine"
+
+
+def test_from_string_ignores_default_project(target_class):
+ got = target_class.from_string(
+ "string-project.string_dataset.string_routine",
+ default_project="default-project",
+ )
+ assert got.project == "string-project"
+ assert got.dataset_id == "string_dataset"
+ assert got.routine_id == "string_routine"
+
+
+def test_eq(target_class):
+ routine = target_class.from_string("my-proj.my_dset.my_routine")
+ routine_too = target_class.from_string("my-proj.my_dset.my_routine")
+ assert routine == routine_too
+ assert not (routine != routine_too)
+
+ other_routine = target_class.from_string("my-proj.my_dset.my_routine2")
+ assert not (routine == other_routine)
+ assert routine != other_routine
+
+ notaroutine = object()
+ assert not (routine == notaroutine)
+ assert routine != notaroutine
+
+
+def test_hash(target_class):
+ routine = target_class.from_string("my-proj.my_dset.my_routine")
+ routine2 = target_class.from_string("my-proj.my_dset.routine2")
+ got = {routine: "hello", routine2: "world"}
+ assert got[routine] == "hello"
+ assert got[routine2] == "world"
+
+ routine_too = target_class.from_string("my-proj.my_dset.my_routine")
+ assert got[routine_too] == "hello"
+
+
+def test_repr(target_class):
+ routine = target_class.from_string("my-proj.my_dset.my_routine")
+ got = repr(routine)
+ assert got == "RoutineReference.from_string('my-proj.my_dset.my_routine')"
diff --git a/bigquery/tests/unit/test__helpers.py b/bigquery/tests/unit/test__helpers.py
new file mode 100644
index 000000000000..3884695d83af
--- /dev/null
+++ b/bigquery/tests/unit/test__helpers.py
@@ -0,0 +1,1025 @@
+# Copyright 2015 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import base64
+import datetime
+import decimal
+import unittest
+
+
+class Test_not_null(unittest.TestCase):
+ def _call_fut(self, value, field):
+ from google.cloud.bigquery._helpers import _not_null
+
+ return _not_null(value, field)
+
+ def test_w_none_nullable(self):
+ self.assertFalse(self._call_fut(None, _Field("NULLABLE")))
+
+ def test_w_none_required(self):
+ self.assertTrue(self._call_fut(None, _Field("REQUIRED")))
+
+ def test_w_value(self):
+ self.assertTrue(self._call_fut(object(), object()))
+
+
+class Test_int_from_json(unittest.TestCase):
+ def _call_fut(self, value, field):
+ from google.cloud.bigquery._helpers import _int_from_json
+
+ return _int_from_json(value, field)
+
+ def test_w_none_nullable(self):
+ self.assertIsNone(self._call_fut(None, _Field("NULLABLE")))
+
+ def test_w_none_required(self):
+ with self.assertRaises(TypeError):
+ self._call_fut(None, _Field("REQUIRED"))
+
+ def test_w_string_value(self):
+ coerced = self._call_fut("42", object())
+ self.assertEqual(coerced, 42)
+
+ def test_w_float_value(self):
+ coerced = self._call_fut(42, object())
+ self.assertEqual(coerced, 42)
+
+
+class Test_float_from_json(unittest.TestCase):
+ def _call_fut(self, value, field):
+ from google.cloud.bigquery._helpers import _float_from_json
+
+ return _float_from_json(value, field)
+
+ def test_w_none_nullable(self):
+ self.assertIsNone(self._call_fut(None, _Field("NULLABLE")))
+
+ def test_w_none_required(self):
+ with self.assertRaises(TypeError):
+ self._call_fut(None, _Field("REQUIRED"))
+
+ def test_w_string_value(self):
+ coerced = self._call_fut("3.1415", object())
+ self.assertEqual(coerced, 3.1415)
+
+ def test_w_float_value(self):
+ coerced = self._call_fut(3.1415, object())
+ self.assertEqual(coerced, 3.1415)
+
+
+class Test_decimal_from_json(unittest.TestCase):
+ def _call_fut(self, value, field):
+ from google.cloud.bigquery._helpers import _decimal_from_json
+
+ return _decimal_from_json(value, field)
+
+ def test_w_none_nullable(self):
+ self.assertIsNone(self._call_fut(None, _Field("NULLABLE")))
+
+ def test_w_none_required(self):
+ with self.assertRaises(TypeError):
+ self._call_fut(None, _Field("REQUIRED"))
+
+ def test_w_string_value(self):
+ coerced = self._call_fut("3.1415", object())
+ self.assertEqual(coerced, decimal.Decimal("3.1415"))
+
+ def test_w_float_value(self):
+ coerced = self._call_fut(3.1415, object())
+ # There is no exact float representation of 3.1415.
+ self.assertEqual(coerced, decimal.Decimal(3.1415))
+
+
+class Test_bool_from_json(unittest.TestCase):
+ def _call_fut(self, value, field):
+ from google.cloud.bigquery._helpers import _bool_from_json
+
+ return _bool_from_json(value, field)
+
+ def test_w_none_nullable(self):
+ self.assertIsNone(self._call_fut(None, _Field("NULLABLE")))
+
+ def test_w_none_required(self):
+ with self.assertRaises(AttributeError):
+ self._call_fut(None, _Field("REQUIRED"))
+
+ def test_w_value_t(self):
+ coerced = self._call_fut("T", object())
+ self.assertTrue(coerced)
+
+ def test_w_value_true(self):
+ coerced = self._call_fut("True", object())
+ self.assertTrue(coerced)
+
+ def test_w_value_1(self):
+ coerced = self._call_fut("1", object())
+ self.assertTrue(coerced)
+
+ def test_w_value_other(self):
+ coerced = self._call_fut("f", object())
+ self.assertFalse(coerced)
+
+
+class Test_string_from_json(unittest.TestCase):
+ def _call_fut(self, value, field):
+ from google.cloud.bigquery._helpers import _string_from_json
+
+ return _string_from_json(value, field)
+
+ def test_w_none_nullable(self):
+ self.assertIsNone(self._call_fut(None, _Field("NULLABLE")))
+
+ def test_w_none_required(self):
+ self.assertIsNone(self._call_fut(None, _Field("REQUIRED")))
+
+ def test_w_string_value(self):
+ coerced = self._call_fut("Wonderful!", object())
+ self.assertEqual(coerced, "Wonderful!")
+
+
+class Test_bytes_from_json(unittest.TestCase):
+ def _call_fut(self, value, field):
+ from google.cloud.bigquery._helpers import _bytes_from_json
+
+ return _bytes_from_json(value, field)
+
+ def test_w_none_nullable(self):
+ self.assertIsNone(self._call_fut(None, _Field("NULLABLE")))
+
+ def test_w_none_required(self):
+ with self.assertRaises(TypeError):
+ self._call_fut(None, _Field("REQUIRED"))
+
+ def test_w_base64_encoded_bytes(self):
+ expected = b"Wonderful!"
+ encoded = base64.standard_b64encode(expected)
+ coerced = self._call_fut(encoded, object())
+ self.assertEqual(coerced, expected)
+
+ def test_w_base64_encoded_text(self):
+ expected = b"Wonderful!"
+ encoded = base64.standard_b64encode(expected).decode("ascii")
+ coerced = self._call_fut(encoded, object())
+ self.assertEqual(coerced, expected)
+
+
+class Test_timestamp_from_json(unittest.TestCase):
+ def _call_fut(self, value, field):
+ from google.cloud.bigquery._helpers import _timestamp_from_json
+
+ return _timestamp_from_json(value, field)
+
+ def test_w_none_nullable(self):
+ self.assertIsNone(self._call_fut(None, _Field("NULLABLE")))
+
+ def test_w_none_required(self):
+ with self.assertRaises(TypeError):
+ self._call_fut(None, _Field("REQUIRED"))
+
+ def test_w_string_value(self):
+ from google.cloud._helpers import _EPOCH
+
+ coerced = self._call_fut("1.234567", object())
+ self.assertEqual(
+ coerced, _EPOCH + datetime.timedelta(seconds=1, microseconds=234567)
+ )
+
+ def test_w_float_value(self):
+ from google.cloud._helpers import _EPOCH
+
+ coerced = self._call_fut(1.234567, object())
+ self.assertEqual(
+ coerced, _EPOCH + datetime.timedelta(seconds=1, microseconds=234567)
+ )
+
+
+class Test_timestamp_query_param_from_json(unittest.TestCase):
+ def _call_fut(self, value, field):
+ from google.cloud.bigquery import _helpers
+
+ return _helpers._timestamp_query_param_from_json(value, field)
+
+ def test_w_none_nullable(self):
+ self.assertIsNone(self._call_fut(None, _Field("NULLABLE")))
+
+ def test_w_timestamp_valid(self):
+ from google.cloud._helpers import UTC
+
+ samples = [
+ (
+ "2016-12-20 15:58:27.339328+00:00",
+ datetime.datetime(2016, 12, 20, 15, 58, 27, 339328, tzinfo=UTC),
+ ),
+ (
+ "2016-12-20 15:58:27+00:00",
+ datetime.datetime(2016, 12, 20, 15, 58, 27, tzinfo=UTC),
+ ),
+ (
+ "2016-12-20T15:58:27.339328+00:00",
+ datetime.datetime(2016, 12, 20, 15, 58, 27, 339328, tzinfo=UTC),
+ ),
+ (
+ "2016-12-20T15:58:27+00:00",
+ datetime.datetime(2016, 12, 20, 15, 58, 27, tzinfo=UTC),
+ ),
+ (
+ "2016-12-20 15:58:27.339328Z",
+ datetime.datetime(2016, 12, 20, 15, 58, 27, 339328, tzinfo=UTC),
+ ),
+ (
+ "2016-12-20 15:58:27Z",
+ datetime.datetime(2016, 12, 20, 15, 58, 27, tzinfo=UTC),
+ ),
+ (
+ "2016-12-20T15:58:27.339328Z",
+ datetime.datetime(2016, 12, 20, 15, 58, 27, 339328, tzinfo=UTC),
+ ),
+ (
+ "2016-12-20T15:58:27Z",
+ datetime.datetime(2016, 12, 20, 15, 58, 27, tzinfo=UTC),
+ ),
+ ]
+ for timestamp_str, expected_result in samples:
+ self.assertEqual(
+ self._call_fut(timestamp_str, _Field("NULLABLE")), expected_result
+ )
+
+ def test_w_timestamp_invalid(self):
+ with self.assertRaises(ValueError):
+ self._call_fut("definitely-not-a-timestamp", _Field("NULLABLE"))
+
+
+class Test_datetime_from_json(unittest.TestCase):
+ def _call_fut(self, value, field):
+ from google.cloud.bigquery._helpers import _datetime_from_json
+
+ return _datetime_from_json(value, field)
+
+ def test_w_none_nullable(self):
+ self.assertIsNone(self._call_fut(None, _Field("NULLABLE")))
+
+ def test_w_none_required(self):
+ with self.assertRaises(TypeError):
+ self._call_fut(None, _Field("REQUIRED"))
+
+ def test_w_string_value(self):
+ coerced = self._call_fut("2016-12-02T18:51:33", object())
+ self.assertEqual(coerced, datetime.datetime(2016, 12, 2, 18, 51, 33))
+
+ def test_w_microseconds(self):
+ coerced = self._call_fut("2015-05-22T10:11:12.987654", object())
+ self.assertEqual(coerced, datetime.datetime(2015, 5, 22, 10, 11, 12, 987654))
+
+
+class Test_date_from_json(unittest.TestCase):
+ def _call_fut(self, value, field):
+ from google.cloud.bigquery._helpers import _date_from_json
+
+ return _date_from_json(value, field)
+
+ def test_w_none_nullable(self):
+ self.assertIsNone(self._call_fut(None, _Field("NULLABLE")))
+
+ def test_w_none_required(self):
+ with self.assertRaises(TypeError):
+ self._call_fut(None, _Field("REQUIRED"))
+
+ def test_w_string_value(self):
+ coerced = self._call_fut("1987-09-22", object())
+ self.assertEqual(coerced, datetime.date(1987, 9, 22))
+
+
+class Test_time_from_json(unittest.TestCase):
+ def _call_fut(self, value, field):
+ from google.cloud.bigquery._helpers import _time_from_json
+
+ return _time_from_json(value, field)
+
+ def test_w_none_nullable(self):
+ self.assertIsNone(self._call_fut(None, _Field("NULLABLE")))
+
+ def test_w_none_required(self):
+ with self.assertRaises(TypeError):
+ self._call_fut(None, _Field("REQUIRED"))
+
+ def test_w_string_value(self):
+ coerced = self._call_fut("12:12:27", object())
+ self.assertEqual(coerced, datetime.time(12, 12, 27))
+
+ def test_w_subsecond_string_value(self):
+ coerced = self._call_fut("12:12:27.123456", object())
+ self.assertEqual(coerced, datetime.time(12, 12, 27, 123456))
+
+ def test_w_bogus_string_value(self):
+ with self.assertRaises(ValueError):
+ self._call_fut("12:12:27.123", object())
+
+
+class Test_record_from_json(unittest.TestCase):
+ def _call_fut(self, value, field):
+ from google.cloud.bigquery._helpers import _record_from_json
+
+ return _record_from_json(value, field)
+
+ def test_w_none_nullable(self):
+ self.assertIsNone(self._call_fut(None, _Field("NULLABLE")))
+
+ def test_w_none_required(self):
+ with self.assertRaises(TypeError):
+ self._call_fut(None, _Field("REQUIRED"))
+
+ def test_w_nullable_subfield_none(self):
+ subfield = _Field("NULLABLE", "age", "INTEGER")
+ field = _Field("REQUIRED", fields=[subfield])
+ value = {"f": [{"v": None}]}
+ coerced = self._call_fut(value, field)
+ self.assertEqual(coerced, {"age": None})
+
+ def test_w_scalar_subfield(self):
+ subfield = _Field("REQUIRED", "age", "INTEGER")
+ field = _Field("REQUIRED", fields=[subfield])
+ value = {"f": [{"v": 42}]}
+ coerced = self._call_fut(value, field)
+ self.assertEqual(coerced, {"age": 42})
+
+ def test_w_scalar_subfield_geography(self):
+ subfield = _Field("REQUIRED", "geo", "GEOGRAPHY")
+ field = _Field("REQUIRED", fields=[subfield])
+ value = {"f": [{"v": "POINT(1, 2)"}]}
+ coerced = self._call_fut(value, field)
+ self.assertEqual(coerced, {"geo": "POINT(1, 2)"})
+
+ def test_w_repeated_subfield(self):
+ subfield = _Field("REPEATED", "color", "STRING")
+ field = _Field("REQUIRED", fields=[subfield])
+ value = {"f": [{"v": [{"v": "red"}, {"v": "yellow"}, {"v": "blue"}]}]}
+ coerced = self._call_fut(value, field)
+ self.assertEqual(coerced, {"color": ["red", "yellow", "blue"]})
+
+ def test_w_record_subfield(self):
+ full_name = _Field("REQUIRED", "full_name", "STRING")
+ area_code = _Field("REQUIRED", "area_code", "STRING")
+ local_number = _Field("REQUIRED", "local_number", "STRING")
+ rank = _Field("REQUIRED", "rank", "INTEGER")
+ phone = _Field(
+ "NULLABLE", "phone", "RECORD", fields=[area_code, local_number, rank]
+ )
+ person = _Field("REQUIRED", "person", "RECORD", fields=[full_name, phone])
+ value = {
+ "f": [
+ {"v": "Phred Phlyntstone"},
+ {"v": {"f": [{"v": "800"}, {"v": "555-1212"}, {"v": 1}]}},
+ ]
+ }
+ expected = {
+ "full_name": "Phred Phlyntstone",
+ "phone": {"area_code": "800", "local_number": "555-1212", "rank": 1},
+ }
+ coerced = self._call_fut(value, person)
+ self.assertEqual(coerced, expected)
+
+
+class Test_field_to_index_mapping(unittest.TestCase):
+ def _call_fut(self, schema):
+ from google.cloud.bigquery._helpers import _field_to_index_mapping
+
+ return _field_to_index_mapping(schema)
+
+ def test_w_empty_schema(self):
+ self.assertEqual(self._call_fut([]), {})
+
+ def test_w_non_empty_schema(self):
+ schema = [
+ _Field("REPEATED", "first", "INTEGER"),
+ _Field("REQUIRED", "second", "INTEGER"),
+ _Field("REPEATED", "third", "INTEGER"),
+ ]
+ self.assertEqual(self._call_fut(schema), {"first": 0, "second": 1, "third": 2})
+
+
+class Test_row_tuple_from_json(unittest.TestCase):
+ def _call_fut(self, row, schema):
+ from google.cloud.bigquery._helpers import _row_tuple_from_json
+
+ return _row_tuple_from_json(row, schema)
+
+ def test_w_single_scalar_column(self):
+ # SELECT 1 AS col
+ col = _Field("REQUIRED", "col", "INTEGER")
+ row = {u"f": [{u"v": u"1"}]}
+ self.assertEqual(self._call_fut(row, schema=[col]), (1,))
+
+ def test_w_single_scalar_geography_column(self):
+ # SELECT 1 AS col
+ col = _Field("REQUIRED", "geo", "GEOGRAPHY")
+ row = {u"f": [{u"v": u"POINT(1, 2)"}]}
+ self.assertEqual(self._call_fut(row, schema=[col]), ("POINT(1, 2)",))
+
+ def test_w_single_struct_column(self):
+ # SELECT (1, 2) AS col
+ sub_1 = _Field("REQUIRED", "sub_1", "INTEGER")
+ sub_2 = _Field("REQUIRED", "sub_2", "INTEGER")
+ col = _Field("REQUIRED", "col", "RECORD", fields=[sub_1, sub_2])
+ row = {u"f": [{u"v": {u"f": [{u"v": u"1"}, {u"v": u"2"}]}}]}
+ self.assertEqual(self._call_fut(row, schema=[col]), ({"sub_1": 1, "sub_2": 2},))
+
+ def test_w_single_array_column(self):
+ # SELECT [1, 2, 3] as col
+ col = _Field("REPEATED", "col", "INTEGER")
+ row = {u"f": [{u"v": [{u"v": u"1"}, {u"v": u"2"}, {u"v": u"3"}]}]}
+ self.assertEqual(self._call_fut(row, schema=[col]), ([1, 2, 3],))
+
+ def test_w_struct_w_nested_array_column(self):
+ # SELECT ([1, 2], 3, [4, 5]) as col
+ first = _Field("REPEATED", "first", "INTEGER")
+ second = _Field("REQUIRED", "second", "INTEGER")
+ third = _Field("REPEATED", "third", "INTEGER")
+ col = _Field("REQUIRED", "col", "RECORD", fields=[first, second, third])
+ row = {
+ u"f": [
+ {
+ u"v": {
+ u"f": [
+ {u"v": [{u"v": u"1"}, {u"v": u"2"}]},
+ {u"v": u"3"},
+ {u"v": [{u"v": u"4"}, {u"v": u"5"}]},
+ ]
+ }
+ }
+ ]
+ }
+ self.assertEqual(
+ self._call_fut(row, schema=[col]),
+ ({u"first": [1, 2], u"second": 3, u"third": [4, 5]},),
+ )
+
+ def test_w_array_of_struct(self):
+ # SELECT [(1, 2, 3), (4, 5, 6)] as col
+ first = _Field("REQUIRED", "first", "INTEGER")
+ second = _Field("REQUIRED", "second", "INTEGER")
+ third = _Field("REQUIRED", "third", "INTEGER")
+ col = _Field("REPEATED", "col", "RECORD", fields=[first, second, third])
+ row = {
+ u"f": [
+ {
+ u"v": [
+ {u"v": {u"f": [{u"v": u"1"}, {u"v": u"2"}, {u"v": u"3"}]}},
+ {u"v": {u"f": [{u"v": u"4"}, {u"v": u"5"}, {u"v": u"6"}]}},
+ ]
+ }
+ ]
+ }
+ self.assertEqual(
+ self._call_fut(row, schema=[col]),
+ (
+ [
+ {u"first": 1, u"second": 2, u"third": 3},
+ {u"first": 4, u"second": 5, u"third": 6},
+ ],
+ ),
+ )
+
+ def test_w_array_of_struct_w_array(self):
+ # SELECT [([1, 2, 3], 4), ([5, 6], 7)]
+ first = _Field("REPEATED", "first", "INTEGER")
+ second = _Field("REQUIRED", "second", "INTEGER")
+ col = _Field("REPEATED", "col", "RECORD", fields=[first, second])
+ row = {
+ u"f": [
+ {
+ u"v": [
+ {
+ u"v": {
+ u"f": [
+ {u"v": [{u"v": u"1"}, {u"v": u"2"}, {u"v": u"3"}]},
+ {u"v": u"4"},
+ ]
+ }
+ },
+ {
+ u"v": {
+ u"f": [
+ {u"v": [{u"v": u"5"}, {u"v": u"6"}]},
+ {u"v": u"7"},
+ ]
+ }
+ },
+ ]
+ }
+ ]
+ }
+ self.assertEqual(
+ self._call_fut(row, schema=[col]),
+ ([{u"first": [1, 2, 3], u"second": 4}, {u"first": [5, 6], u"second": 7}],),
+ )
+
+
+class Test_rows_from_json(unittest.TestCase):
+ def _call_fut(self, rows, schema):
+ from google.cloud.bigquery._helpers import _rows_from_json
+
+ return _rows_from_json(rows, schema)
+
+ def test_w_record_subfield(self):
+ from google.cloud.bigquery.table import Row
+
+ full_name = _Field("REQUIRED", "full_name", "STRING")
+ area_code = _Field("REQUIRED", "area_code", "STRING")
+ local_number = _Field("REQUIRED", "local_number", "STRING")
+ rank = _Field("REQUIRED", "rank", "INTEGER")
+ phone = _Field(
+ "NULLABLE", "phone", "RECORD", fields=[area_code, local_number, rank]
+ )
+ color = _Field("REPEATED", "color", "STRING")
+ schema = [full_name, phone, color]
+ rows = [
+ {
+ "f": [
+ {"v": "Phred Phlyntstone"},
+ {"v": {"f": [{"v": "800"}, {"v": "555-1212"}, {"v": 1}]}},
+ {"v": [{"v": "orange"}, {"v": "black"}]},
+ ]
+ },
+ {
+ "f": [
+ {"v": "Bharney Rhubble"},
+ {"v": {"f": [{"v": "877"}, {"v": "768-5309"}, {"v": 2}]}},
+ {"v": [{"v": "brown"}]},
+ ]
+ },
+ {"f": [{"v": "Wylma Phlyntstone"}, {"v": None}, {"v": []}]},
+ ]
+ phred_phone = {"area_code": "800", "local_number": "555-1212", "rank": 1}
+ bharney_phone = {"area_code": "877", "local_number": "768-5309", "rank": 2}
+ f2i = {"full_name": 0, "phone": 1, "color": 2}
+ expected = [
+ Row(("Phred Phlyntstone", phred_phone, ["orange", "black"]), f2i),
+ Row(("Bharney Rhubble", bharney_phone, ["brown"]), f2i),
+ Row(("Wylma Phlyntstone", None, []), f2i),
+ ]
+ coerced = self._call_fut(rows, schema)
+ self.assertEqual(coerced, expected)
+
+ def test_w_int64_float64_bool(self):
+ from google.cloud.bigquery.table import Row
+
+ # "Standard" SQL dialect uses 'INT64', 'FLOAT64', 'BOOL'.
+ candidate = _Field("REQUIRED", "candidate", "STRING")
+ votes = _Field("REQUIRED", "votes", "INT64")
+ percentage = _Field("REQUIRED", "percentage", "FLOAT64")
+ incumbent = _Field("REQUIRED", "incumbent", "BOOL")
+ schema = [candidate, votes, percentage, incumbent]
+ rows = [
+ {"f": [{"v": "Phred Phlyntstone"}, {"v": 8}, {"v": 0.25}, {"v": "true"}]},
+ {"f": [{"v": "Bharney Rhubble"}, {"v": 4}, {"v": 0.125}, {"v": "false"}]},
+ {
+ "f": [
+ {"v": "Wylma Phlyntstone"},
+ {"v": 20},
+ {"v": 0.625},
+ {"v": "false"},
+ ]
+ },
+ ]
+ f2i = {"candidate": 0, "votes": 1, "percentage": 2, "incumbent": 3}
+ expected = [
+ Row(("Phred Phlyntstone", 8, 0.25, True), f2i),
+ Row(("Bharney Rhubble", 4, 0.125, False), f2i),
+ Row(("Wylma Phlyntstone", 20, 0.625, False), f2i),
+ ]
+ coerced = self._call_fut(rows, schema)
+ self.assertEqual(coerced, expected)
+
+
+class Test_int_to_json(unittest.TestCase):
+ def _call_fut(self, value):
+ from google.cloud.bigquery._helpers import _int_to_json
+
+ return _int_to_json(value)
+
+ def test_w_int(self):
+ self.assertEqual(self._call_fut(123), "123")
+
+ def test_w_string(self):
+ self.assertEqual(self._call_fut("123"), "123")
+
+
+class Test_float_to_json(unittest.TestCase):
+ def _call_fut(self, value):
+ from google.cloud.bigquery._helpers import _float_to_json
+
+ return _float_to_json(value)
+
+ def test_w_float(self):
+ self.assertEqual(self._call_fut(1.23), 1.23)
+
+
+class Test_decimal_to_json(unittest.TestCase):
+ def _call_fut(self, value):
+ from google.cloud.bigquery._helpers import _decimal_to_json
+
+ return _decimal_to_json(value)
+
+ def test_w_float(self):
+ self.assertEqual(self._call_fut(1.23), 1.23)
+
+ def test_w_string(self):
+ self.assertEqual(self._call_fut("1.23"), "1.23")
+
+ def test_w_decimal(self):
+ self.assertEqual(self._call_fut(decimal.Decimal("1.23")), "1.23")
+
+
+class Test_bool_to_json(unittest.TestCase):
+ def _call_fut(self, value):
+ from google.cloud.bigquery._helpers import _bool_to_json
+
+ return _bool_to_json(value)
+
+ def test_w_true(self):
+ self.assertEqual(self._call_fut(True), "true")
+
+ def test_w_false(self):
+ self.assertEqual(self._call_fut(False), "false")
+
+ def test_w_string(self):
+ self.assertEqual(self._call_fut("false"), "false")
+
+
+class Test_bytes_to_json(unittest.TestCase):
+ def _call_fut(self, value):
+ from google.cloud.bigquery._helpers import _bytes_to_json
+
+ return _bytes_to_json(value)
+
+ def test_w_non_bytes(self):
+ non_bytes = object()
+ self.assertIs(self._call_fut(non_bytes), non_bytes)
+
+ def test_w_bytes(self):
+ source = b"source"
+ expected = u"c291cmNl"
+ converted = self._call_fut(source)
+ self.assertEqual(converted, expected)
+
+
+class Test_timestamp_to_json_parameter(unittest.TestCase):
+ def _call_fut(self, value):
+ from google.cloud.bigquery._helpers import _timestamp_to_json_parameter
+
+ return _timestamp_to_json_parameter(value)
+
+ def test_w_float(self):
+ self.assertEqual(self._call_fut(1.234567), 1.234567)
+
+ def test_w_string(self):
+ ZULU = "2016-12-20 15:58:27.339328+00:00"
+ self.assertEqual(self._call_fut(ZULU), ZULU)
+
+ def test_w_datetime_wo_zone(self):
+ ZULU = "2016-12-20 15:58:27.339328+00:00"
+ when = datetime.datetime(2016, 12, 20, 15, 58, 27, 339328)
+ self.assertEqual(self._call_fut(when), ZULU)
+
+ def test_w_datetime_w_non_utc_zone(self):
+ class _Zone(datetime.tzinfo):
+ def utcoffset(self, _):
+ return datetime.timedelta(minutes=-240)
+
+ ZULU = "2016-12-20 19:58:27.339328+00:00"
+ when = datetime.datetime(2016, 12, 20, 15, 58, 27, 339328, tzinfo=_Zone())
+ self.assertEqual(self._call_fut(when), ZULU)
+
+ def test_w_datetime_w_utc_zone(self):
+ from google.cloud._helpers import UTC
+
+ ZULU = "2016-12-20 15:58:27.339328+00:00"
+ when = datetime.datetime(2016, 12, 20, 15, 58, 27, 339328, tzinfo=UTC)
+ self.assertEqual(self._call_fut(when), ZULU)
+
+
+class Test_timestamp_to_json_row(unittest.TestCase):
+ def _call_fut(self, value):
+ from google.cloud.bigquery._helpers import _timestamp_to_json_row
+
+ return _timestamp_to_json_row(value)
+
+ def test_w_float(self):
+ self.assertEqual(self._call_fut(1.234567), 1.234567)
+
+ def test_w_string(self):
+ ZULU = "2016-12-20 15:58:27.339328+00:00"
+ self.assertEqual(self._call_fut(ZULU), ZULU)
+
+ def test_w_datetime(self):
+ from google.cloud._helpers import _microseconds_from_datetime
+
+ when = datetime.datetime(2016, 12, 20, 15, 58, 27, 339328)
+ self.assertEqual(self._call_fut(when), _microseconds_from_datetime(when) / 1e6)
+
+
+class Test_datetime_to_json(unittest.TestCase):
+ def _call_fut(self, value):
+ from google.cloud.bigquery._helpers import _datetime_to_json
+
+ return _datetime_to_json(value)
+
+ def test_w_string(self):
+ RFC3339 = "2016-12-03T14:14:51Z"
+ self.assertEqual(self._call_fut(RFC3339), RFC3339)
+
+ def test_w_datetime(self):
+ from google.cloud._helpers import UTC
+
+ when = datetime.datetime(2016, 12, 3, 14, 11, 27, 123456, tzinfo=UTC)
+ self.assertEqual(self._call_fut(when), "2016-12-03T14:11:27.123456")
+
+
+class Test_date_to_json(unittest.TestCase):
+ def _call_fut(self, value):
+ from google.cloud.bigquery._helpers import _date_to_json
+
+ return _date_to_json(value)
+
+ def test_w_string(self):
+ RFC3339 = "2016-12-03"
+ self.assertEqual(self._call_fut(RFC3339), RFC3339)
+
+ def test_w_datetime(self):
+ when = datetime.date(2016, 12, 3)
+ self.assertEqual(self._call_fut(when), "2016-12-03")
+
+
+class Test_time_to_json(unittest.TestCase):
+ def _call_fut(self, value):
+ from google.cloud.bigquery._helpers import _time_to_json
+
+ return _time_to_json(value)
+
+ def test_w_string(self):
+ RFC3339 = "12:13:41"
+ self.assertEqual(self._call_fut(RFC3339), RFC3339)
+
+ def test_w_datetime(self):
+ when = datetime.time(12, 13, 41)
+ self.assertEqual(self._call_fut(when), "12:13:41")
+
+
+def _make_field(field_type, mode="NULLABLE", name="testing", fields=()):
+ from google.cloud.bigquery.schema import SchemaField
+
+ return SchemaField(name=name, field_type=field_type, mode=mode, fields=fields)
+
+
+class Test_scalar_field_to_json(unittest.TestCase):
+ def _call_fut(self, field, value):
+ from google.cloud.bigquery._helpers import _scalar_field_to_json
+
+ return _scalar_field_to_json(field, value)
+
+ def test_w_unknown_field_type(self):
+ field = _make_field("UNKNOWN")
+ original = object()
+ converted = self._call_fut(field, original)
+ self.assertIs(converted, original)
+
+ def test_w_known_field_type(self):
+ field = _make_field("INT64")
+ original = 42
+ converted = self._call_fut(field, original)
+ self.assertEqual(converted, str(original))
+
+
+class Test_repeated_field_to_json(unittest.TestCase):
+ def _call_fut(self, field, value):
+ from google.cloud.bigquery._helpers import _repeated_field_to_json
+
+ return _repeated_field_to_json(field, value)
+
+ def test_w_empty(self):
+ field = _make_field("INT64", mode="REPEATED")
+ original = []
+ converted = self._call_fut(field, original)
+ self.assertEqual(converted, original)
+ self.assertEqual(field.mode, "REPEATED")
+
+ def test_w_non_empty(self):
+ field = _make_field("INT64", mode="REPEATED")
+ original = [42]
+ converted = self._call_fut(field, original)
+ self.assertEqual(converted, [str(value) for value in original])
+ self.assertEqual(field.mode, "REPEATED")
+
+
+class Test_record_field_to_json(unittest.TestCase):
+ def _call_fut(self, field, value):
+ from google.cloud.bigquery._helpers import _record_field_to_json
+
+ return _record_field_to_json(field, value)
+
+ def test_w_empty(self):
+ fields = []
+ original = []
+ converted = self._call_fut(fields, original)
+ self.assertEqual(converted, {})
+
+ def test_w_non_empty_list(self):
+ fields = [
+ _make_field("INT64", name="one", mode="NULLABLE"),
+ _make_field("STRING", name="two", mode="NULLABLE"),
+ ]
+ original = [42, "two"]
+ converted = self._call_fut(fields, original)
+ self.assertEqual(converted, {"one": "42", "two": "two"})
+
+ def test_w_non_empty_dict(self):
+ fields = [
+ _make_field("INT64", name="one", mode="NULLABLE"),
+ _make_field("STRING", name="two", mode="NULLABLE"),
+ ]
+ original = {"one": 42, "two": "two"}
+ converted = self._call_fut(fields, original)
+ self.assertEqual(converted, {"one": "42", "two": "two"})
+
+ def test_w_missing_nullable(self):
+ fields = [
+ _make_field("INT64", name="one", mode="NULLABLE"),
+ _make_field("STRING", name="two", mode="NULLABLE"),
+ ]
+ original = {"one": 42}
+ converted = self._call_fut(fields, original)
+ self.assertEqual(converted, {"one": "42", "two": None})
+
+
+class Test_field_to_json(unittest.TestCase):
+ def _call_fut(self, field, value):
+ from google.cloud.bigquery._helpers import _field_to_json
+
+ return _field_to_json(field, value)
+
+ def test_w_none(self):
+ field = _make_field("INT64")
+ original = None
+ converted = self._call_fut(field, original)
+ self.assertIsNone(converted)
+
+ def test_w_repeated(self):
+ field = _make_field("INT64", mode="REPEATED")
+ original = [42, 17]
+ converted = self._call_fut(field, original)
+ self.assertEqual(converted, [str(value) for value in original])
+
+ def test_w_record(self):
+ subfields = [
+ _make_field("INT64", name="one"),
+ _make_field("STRING", name="two"),
+ ]
+ field = _make_field("RECORD", fields=subfields)
+ original = {"one": 42, "two": "two"}
+ converted = self._call_fut(field, original)
+ self.assertEqual(converted, {"one": "42", "two": "two"})
+
+ def test_w_scalar(self):
+ field = _make_field("INT64")
+ original = 42
+ converted = self._call_fut(field, original)
+ self.assertEqual(converted, str(original))
+
+
+class Test_snake_to_camel_case(unittest.TestCase):
+ def _call_fut(self, value):
+ from google.cloud.bigquery._helpers import _snake_to_camel_case
+
+ return _snake_to_camel_case(value)
+
+ def test_w_snake_case_string(self):
+ self.assertEqual(self._call_fut("friendly_name"), "friendlyName")
+
+ def test_w_camel_case_string(self):
+ self.assertEqual(self._call_fut("friendlyName"), "friendlyName")
+
+
+class Test__get_sub_prop(unittest.TestCase):
+ def _call_fut(self, container, keys, **kw):
+ from google.cloud.bigquery._helpers import _get_sub_prop
+
+ return _get_sub_prop(container, keys, **kw)
+
+ def test_w_empty_container_default_default(self):
+ self.assertIsNone(self._call_fut({}, ["key1"]))
+
+ def test_w_missing_key_explicit_default(self):
+ self.assertEqual(self._call_fut({"key2": 2}, ["key1"], default=1), 1)
+
+ def test_w_matching_single_key(self):
+ self.assertEqual(self._call_fut({"key1": 1}, ["key1"]), 1)
+
+ def test_w_matching_first_key_missing_second_key(self):
+ self.assertIsNone(self._call_fut({"key1": {"key3": 3}}, ["key1", "key2"]))
+
+ def test_w_matching_first_key_matching_second_key(self):
+ self.assertEqual(self._call_fut({"key1": {"key2": 2}}, ["key1", "key2"]), 2)
+
+
+class Test__set_sub_prop(unittest.TestCase):
+ def _call_fut(self, container, keys, value):
+ from google.cloud.bigquery._helpers import _set_sub_prop
+
+ return _set_sub_prop(container, keys, value)
+
+ def test_w_empty_container_single_key(self):
+ container = {}
+ self._call_fut(container, ["key1"], "value")
+ self.assertEqual(container, {"key1": "value"})
+
+ def test_w_empty_container_nested_keys(self):
+ container = {}
+ self._call_fut(container, ["key1", "key2", "key3"], "value")
+ self.assertEqual(container, {"key1": {"key2": {"key3": "value"}}})
+
+ def test_w_existing_value(self):
+ container = {"key1": "before"}
+ self._call_fut(container, ["key1"], "after")
+ self.assertEqual(container, {"key1": "after"})
+
+ def test_w_nested_keys_existing_value(self):
+ container = {"key1": {"key2": {"key3": "before"}}}
+ self._call_fut(container, ["key1", "key2", "key3"], "after")
+ self.assertEqual(container, {"key1": {"key2": {"key3": "after"}}})
+
+
+class Test__del_sub_prop(unittest.TestCase):
+ def _call_fut(self, container, keys):
+ from google.cloud.bigquery._helpers import _del_sub_prop
+
+ return _del_sub_prop(container, keys)
+
+ def test_w_single_key(self):
+ container = {"key1": "value"}
+ self._call_fut(container, ["key1"])
+ self.assertEqual(container, {})
+
+ def test_w_empty_container_nested_keys(self):
+ container = {}
+ self._call_fut(container, ["key1", "key2", "key3"])
+ self.assertEqual(container, {"key1": {"key2": {}}})
+
+ def test_w_existing_value_nested_keys(self):
+ container = {"key1": {"key2": {"key3": "value"}}}
+ self._call_fut(container, ["key1", "key2", "key3"])
+ self.assertEqual(container, {"key1": {"key2": {}}})
+
+
+class Test__int_or_none(unittest.TestCase):
+ def _call_fut(self, value):
+ from google.cloud.bigquery._helpers import _int_or_none
+
+ return _int_or_none(value)
+
+ def test_w_num_string(self):
+ self.assertEqual(self._call_fut("123"), 123)
+
+ def test_w_none(self):
+ self.assertIsNone(self._call_fut(None))
+
+ def test_w_int(self):
+ self.assertEqual(self._call_fut(123), 123)
+
+ def test_w_non_num_string(self):
+ with self.assertRaises(ValueError):
+ self._call_fut("ham")
+
+
+class Test__str_or_none(unittest.TestCase):
+ def _call_fut(self, value):
+ from google.cloud.bigquery._helpers import _str_or_none
+
+ return _str_or_none(value)
+
+ def test_w_int(self):
+ self.assertEqual(self._call_fut(123), "123")
+
+ def test_w_none(self):
+ self.assertIsNone(self._call_fut(None))
+
+ def test_w_str(self):
+ self.assertEqual(self._call_fut("ham"), "ham")
+
+
+class _Field(object):
+ def __init__(self, mode, name="unknown", field_type="UNKNOWN", fields=()):
+ self.mode = mode
+ self.name = name
+ self.field_type = field_type
+ self.fields = fields
diff --git a/bigquery/tests/unit/test__http.py b/bigquery/tests/unit/test__http.py
new file mode 100644
index 000000000000..629e8031a52f
--- /dev/null
+++ b/bigquery/tests/unit/test__http.py
@@ -0,0 +1,108 @@
+# Copyright 2015 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import unittest
+
+import mock
+import requests
+
+
+class TestConnection(unittest.TestCase):
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery._http import Connection
+
+ return Connection
+
+ def _make_one(self, *args, **kw):
+ return self._get_target_class()(*args, **kw)
+
+ def test_build_api_url_no_extra_query_params(self):
+ conn = self._make_one(object())
+ URI = "/".join([conn.DEFAULT_API_ENDPOINT, "bigquery", conn.API_VERSION, "foo"])
+ self.assertEqual(conn.build_api_url("/foo"), URI)
+
+ def test_build_api_url_w_custom_endpoint(self):
+ custom_endpoint = "https://www.foo-googleapis.com"
+ conn = self._make_one(object(), api_endpoint=custom_endpoint)
+ URI = "/".join([custom_endpoint, "bigquery", conn.API_VERSION, "foo"])
+ self.assertEqual(conn.build_api_url("/foo"), URI)
+
+ def test_build_api_url_w_extra_query_params(self):
+ from six.moves.urllib.parse import parse_qsl
+ from six.moves.urllib.parse import urlsplit
+
+ conn = self._make_one(object())
+ uri = conn.build_api_url("/foo", {"bar": "baz"})
+ scheme, netloc, path, qs, _ = urlsplit(uri)
+ self.assertEqual("%s://%s" % (scheme, netloc), conn.API_BASE_URL)
+ self.assertEqual(path, "/".join(["", "bigquery", conn.API_VERSION, "foo"]))
+ parms = dict(parse_qsl(qs))
+ self.assertEqual(parms["bar"], "baz")
+
+ def test_user_agent(self):
+ from google.cloud import _http as base_http
+
+ http = mock.create_autospec(requests.Session, instance=True)
+ response = requests.Response()
+ response.status_code = 200
+ data = b"brent-spiner"
+ response._content = data
+ http.request.return_value = response
+ client = mock.Mock(_http=http, spec=["_http"])
+
+ conn = self._make_one(client)
+ conn.user_agent = "my-application/1.2.3"
+ req_data = "req-data-boring"
+ result = conn.api_request("GET", "/rainbow", data=req_data, expect_json=False)
+ self.assertEqual(result, data)
+
+ expected_headers = {
+ "Accept-Encoding": "gzip",
+ base_http.CLIENT_INFO_HEADER: conn.user_agent,
+ "User-Agent": conn.user_agent,
+ }
+ expected_uri = conn.build_api_url("/rainbow")
+ http.request.assert_called_once_with(
+ data=req_data, headers=expected_headers, method="GET", url=expected_uri
+ )
+ self.assertIn("my-application/1.2.3", conn.user_agent)
+
+ def test_extra_headers_replace(self):
+ from google.cloud import _http as base_http
+
+ http = mock.create_autospec(requests.Session, instance=True)
+ response = requests.Response()
+ response.status_code = 200
+ data = b"brent-spiner"
+ response._content = data
+ http.request.return_value = response
+ client = mock.Mock(_http=http, spec=["_http"])
+
+ conn = self._make_one(client)
+ conn.extra_headers = {"x-test-header": "a test value"}
+ req_data = "req-data-boring"
+ result = conn.api_request("GET", "/rainbow", data=req_data, expect_json=False)
+ self.assertEqual(result, data)
+
+ expected_headers = {
+ "Accept-Encoding": "gzip",
+ base_http.CLIENT_INFO_HEADER: conn.user_agent,
+ "User-Agent": conn.user_agent,
+ "x-test-header": "a test value",
+ }
+ expected_uri = conn.build_api_url("/rainbow")
+ http.request.assert_called_once_with(
+ data=req_data, headers=expected_headers, method="GET", url=expected_uri
+ )
diff --git a/bigquery/tests/unit/test__pandas_helpers.py b/bigquery/tests/unit/test__pandas_helpers.py
new file mode 100644
index 000000000000..b539abe9a89a
--- /dev/null
+++ b/bigquery/tests/unit/test__pandas_helpers.py
@@ -0,0 +1,907 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import collections
+import datetime
+import decimal
+import functools
+import warnings
+
+import mock
+
+try:
+ import pandas
+ import pandas.api.types
+ import pandas.testing
+except ImportError: # pragma: NO COVER
+ pandas = None
+try:
+ import pyarrow
+ import pyarrow.types
+except ImportError: # pragma: NO COVER
+ pyarrow = None
+import pytest
+import pytz
+
+from google.cloud.bigquery import schema
+
+
+@pytest.fixture
+def module_under_test():
+ from google.cloud.bigquery import _pandas_helpers
+
+ return _pandas_helpers
+
+
+def is_none(value):
+ return value is None
+
+
+def is_datetime(type_):
+ # See: https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#datetime-type
+ return all_(
+ pyarrow.types.is_timestamp,
+ lambda type_: type_.unit == "us",
+ lambda type_: type_.tz is None,
+ )(type_)
+
+
+def is_numeric(type_):
+ # See: https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#numeric-type
+ return all_(
+ pyarrow.types.is_decimal,
+ lambda type_: type_.precision == 38,
+ lambda type_: type_.scale == 9,
+ )(type_)
+
+
+def is_timestamp(type_):
+ # See: https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#timestamp-type
+ return all_(
+ pyarrow.types.is_timestamp,
+ lambda type_: type_.unit == "us",
+ lambda type_: type_.tz == "UTC",
+ )(type_)
+
+
+def do_all(functions, value):
+ return all((func(value) for func in functions))
+
+
+def all_(*functions):
+ return functools.partial(do_all, functions)
+
+
+@pytest.mark.skipif(pyarrow is None, reason="Requires `pyarrow`")
+def test_is_datetime():
+ assert is_datetime(pyarrow.timestamp("us", tz=None))
+ assert not is_datetime(pyarrow.timestamp("ms", tz=None))
+ assert not is_datetime(pyarrow.timestamp("us", tz="UTC"))
+ assert not is_datetime(pyarrow.string())
+
+
+def test_do_all():
+ assert do_all((lambda _: True, lambda _: True), None)
+ assert not do_all((lambda _: True, lambda _: False), None)
+ assert not do_all((lambda _: False,), None)
+
+
+def test_all_():
+ assert all_(lambda _: True, lambda _: True)(None)
+ assert not all_(lambda _: True, lambda _: False)(None)
+
+
+@pytest.mark.parametrize(
+ "bq_type,bq_mode,is_correct_type",
+ [
+ ("STRING", "NULLABLE", pyarrow.types.is_string),
+ ("STRING", None, pyarrow.types.is_string),
+ ("string", "NULLABLE", pyarrow.types.is_string),
+ ("StRiNg", "NULLABLE", pyarrow.types.is_string),
+ ("BYTES", "NULLABLE", pyarrow.types.is_binary),
+ ("INTEGER", "NULLABLE", pyarrow.types.is_int64),
+ ("INT64", "NULLABLE", pyarrow.types.is_int64),
+ ("FLOAT", "NULLABLE", pyarrow.types.is_float64),
+ ("FLOAT64", "NULLABLE", pyarrow.types.is_float64),
+ ("NUMERIC", "NULLABLE", is_numeric),
+ ("BOOLEAN", "NULLABLE", pyarrow.types.is_boolean),
+ ("BOOL", "NULLABLE", pyarrow.types.is_boolean),
+ ("TIMESTAMP", "NULLABLE", is_timestamp),
+ ("DATE", "NULLABLE", pyarrow.types.is_date32),
+ ("TIME", "NULLABLE", pyarrow.types.is_time64),
+ ("DATETIME", "NULLABLE", is_datetime),
+ ("GEOGRAPHY", "NULLABLE", pyarrow.types.is_string),
+ ("UNKNOWN_TYPE", "NULLABLE", is_none),
+ # Use pyarrow.list_(item_type) for repeated (array) fields.
+ (
+ "STRING",
+ "REPEATED",
+ all_(
+ pyarrow.types.is_list,
+ lambda type_: pyarrow.types.is_string(type_.value_type),
+ ),
+ ),
+ (
+ "STRING",
+ "repeated",
+ all_(
+ pyarrow.types.is_list,
+ lambda type_: pyarrow.types.is_string(type_.value_type),
+ ),
+ ),
+ (
+ "STRING",
+ "RePeAtEd",
+ all_(
+ pyarrow.types.is_list,
+ lambda type_: pyarrow.types.is_string(type_.value_type),
+ ),
+ ),
+ (
+ "BYTES",
+ "REPEATED",
+ all_(
+ pyarrow.types.is_list,
+ lambda type_: pyarrow.types.is_binary(type_.value_type),
+ ),
+ ),
+ (
+ "INTEGER",
+ "REPEATED",
+ all_(
+ pyarrow.types.is_list,
+ lambda type_: pyarrow.types.is_int64(type_.value_type),
+ ),
+ ),
+ (
+ "INT64",
+ "REPEATED",
+ all_(
+ pyarrow.types.is_list,
+ lambda type_: pyarrow.types.is_int64(type_.value_type),
+ ),
+ ),
+ (
+ "FLOAT",
+ "REPEATED",
+ all_(
+ pyarrow.types.is_list,
+ lambda type_: pyarrow.types.is_float64(type_.value_type),
+ ),
+ ),
+ (
+ "FLOAT64",
+ "REPEATED",
+ all_(
+ pyarrow.types.is_list,
+ lambda type_: pyarrow.types.is_float64(type_.value_type),
+ ),
+ ),
+ (
+ "NUMERIC",
+ "REPEATED",
+ all_(pyarrow.types.is_list, lambda type_: is_numeric(type_.value_type)),
+ ),
+ (
+ "BOOLEAN",
+ "REPEATED",
+ all_(
+ pyarrow.types.is_list,
+ lambda type_: pyarrow.types.is_boolean(type_.value_type),
+ ),
+ ),
+ (
+ "BOOL",
+ "REPEATED",
+ all_(
+ pyarrow.types.is_list,
+ lambda type_: pyarrow.types.is_boolean(type_.value_type),
+ ),
+ ),
+ (
+ "TIMESTAMP",
+ "REPEATED",
+ all_(pyarrow.types.is_list, lambda type_: is_timestamp(type_.value_type)),
+ ),
+ (
+ "DATE",
+ "REPEATED",
+ all_(
+ pyarrow.types.is_list,
+ lambda type_: pyarrow.types.is_date32(type_.value_type),
+ ),
+ ),
+ (
+ "TIME",
+ "REPEATED",
+ all_(
+ pyarrow.types.is_list,
+ lambda type_: pyarrow.types.is_time64(type_.value_type),
+ ),
+ ),
+ (
+ "DATETIME",
+ "REPEATED",
+ all_(pyarrow.types.is_list, lambda type_: is_datetime(type_.value_type)),
+ ),
+ (
+ "GEOGRAPHY",
+ "REPEATED",
+ all_(
+ pyarrow.types.is_list,
+ lambda type_: pyarrow.types.is_string(type_.value_type),
+ ),
+ ),
+ ("RECORD", "REPEATED", is_none),
+ ("UNKNOWN_TYPE", "REPEATED", is_none),
+ ],
+)
+@pytest.mark.skipif(pyarrow is None, reason="Requires `pyarrow`")
+def test_bq_to_arrow_data_type(module_under_test, bq_type, bq_mode, is_correct_type):
+ field = schema.SchemaField("ignored_name", bq_type, mode=bq_mode)
+ actual = module_under_test.bq_to_arrow_data_type(field)
+ assert is_correct_type(actual)
+
+
+@pytest.mark.parametrize("bq_type", ["RECORD", "record", "STRUCT", "struct"])
+@pytest.mark.skipif(pyarrow is None, reason="Requires `pyarrow`")
+def test_bq_to_arrow_data_type_w_struct(module_under_test, bq_type):
+ fields = (
+ schema.SchemaField("field01", "STRING"),
+ schema.SchemaField("field02", "BYTES"),
+ schema.SchemaField("field03", "INTEGER"),
+ schema.SchemaField("field04", "INT64"),
+ schema.SchemaField("field05", "FLOAT"),
+ schema.SchemaField("field06", "FLOAT64"),
+ schema.SchemaField("field07", "NUMERIC"),
+ schema.SchemaField("field08", "BOOLEAN"),
+ schema.SchemaField("field09", "BOOL"),
+ schema.SchemaField("field10", "TIMESTAMP"),
+ schema.SchemaField("field11", "DATE"),
+ schema.SchemaField("field12", "TIME"),
+ schema.SchemaField("field13", "DATETIME"),
+ schema.SchemaField("field14", "GEOGRAPHY"),
+ )
+ field = schema.SchemaField("ignored_name", bq_type, mode="NULLABLE", fields=fields)
+ actual = module_under_test.bq_to_arrow_data_type(field)
+ expected = pyarrow.struct(
+ (
+ pyarrow.field("field01", pyarrow.string()),
+ pyarrow.field("field02", pyarrow.binary()),
+ pyarrow.field("field03", pyarrow.int64()),
+ pyarrow.field("field04", pyarrow.int64()),
+ pyarrow.field("field05", pyarrow.float64()),
+ pyarrow.field("field06", pyarrow.float64()),
+ pyarrow.field("field07", module_under_test.pyarrow_numeric()),
+ pyarrow.field("field08", pyarrow.bool_()),
+ pyarrow.field("field09", pyarrow.bool_()),
+ pyarrow.field("field10", module_under_test.pyarrow_timestamp()),
+ pyarrow.field("field11", pyarrow.date32()),
+ pyarrow.field("field12", module_under_test.pyarrow_time()),
+ pyarrow.field("field13", module_under_test.pyarrow_datetime()),
+ pyarrow.field("field14", pyarrow.string()),
+ )
+ )
+ assert pyarrow.types.is_struct(actual)
+ assert actual.num_children == len(fields)
+ assert actual.equals(expected)
+
+
+@pytest.mark.parametrize("bq_type", ["RECORD", "record", "STRUCT", "struct"])
+@pytest.mark.skipif(pyarrow is None, reason="Requires `pyarrow`")
+def test_bq_to_arrow_data_type_w_array_struct(module_under_test, bq_type):
+ fields = (
+ schema.SchemaField("field01", "STRING"),
+ schema.SchemaField("field02", "BYTES"),
+ schema.SchemaField("field03", "INTEGER"),
+ schema.SchemaField("field04", "INT64"),
+ schema.SchemaField("field05", "FLOAT"),
+ schema.SchemaField("field06", "FLOAT64"),
+ schema.SchemaField("field07", "NUMERIC"),
+ schema.SchemaField("field08", "BOOLEAN"),
+ schema.SchemaField("field09", "BOOL"),
+ schema.SchemaField("field10", "TIMESTAMP"),
+ schema.SchemaField("field11", "DATE"),
+ schema.SchemaField("field12", "TIME"),
+ schema.SchemaField("field13", "DATETIME"),
+ schema.SchemaField("field14", "GEOGRAPHY"),
+ )
+ field = schema.SchemaField("ignored_name", bq_type, mode="REPEATED", fields=fields)
+ actual = module_under_test.bq_to_arrow_data_type(field)
+ expected_value_type = pyarrow.struct(
+ (
+ pyarrow.field("field01", pyarrow.string()),
+ pyarrow.field("field02", pyarrow.binary()),
+ pyarrow.field("field03", pyarrow.int64()),
+ pyarrow.field("field04", pyarrow.int64()),
+ pyarrow.field("field05", pyarrow.float64()),
+ pyarrow.field("field06", pyarrow.float64()),
+ pyarrow.field("field07", module_under_test.pyarrow_numeric()),
+ pyarrow.field("field08", pyarrow.bool_()),
+ pyarrow.field("field09", pyarrow.bool_()),
+ pyarrow.field("field10", module_under_test.pyarrow_timestamp()),
+ pyarrow.field("field11", pyarrow.date32()),
+ pyarrow.field("field12", module_under_test.pyarrow_time()),
+ pyarrow.field("field13", module_under_test.pyarrow_datetime()),
+ pyarrow.field("field14", pyarrow.string()),
+ )
+ )
+ assert pyarrow.types.is_list(actual)
+ assert pyarrow.types.is_struct(actual.value_type)
+ assert actual.value_type.num_children == len(fields)
+ assert actual.value_type.equals(expected_value_type)
+
+
+@pytest.mark.skipif(pyarrow is None, reason="Requires `pyarrow`")
+def test_bq_to_arrow_data_type_w_struct_unknown_subfield(module_under_test):
+ fields = (
+ schema.SchemaField("field1", "STRING"),
+ schema.SchemaField("field2", "INTEGER"),
+ # Don't know what to convert UNKNOWN_TYPE to, let type inference work,
+ # instead.
+ schema.SchemaField("field3", "UNKNOWN_TYPE"),
+ )
+ field = schema.SchemaField("ignored_name", "RECORD", mode="NULLABLE", fields=fields)
+
+ with warnings.catch_warnings(record=True) as warned:
+ actual = module_under_test.bq_to_arrow_data_type(field)
+
+ assert actual is None
+ assert len(warned) == 1
+ warning = warned[0]
+ assert "field3" in str(warning)
+
+
+@pytest.mark.parametrize(
+ "bq_type,rows",
+ [
+ ("STRING", ["abc", None, "def", None]),
+ ("BYTES", [b"abc", None, b"def", None]),
+ ("INTEGER", [123, None, 456, None]),
+ ("INT64", [-9223372036854775808, None, 9223372036854775807, 123]),
+ ("FLOAT", [1.25, None, 3.5, None]),
+ (
+ "NUMERIC",
+ [
+ decimal.Decimal("-99999999999999999999999999999.999999999"),
+ None,
+ decimal.Decimal("99999999999999999999999999999.999999999"),
+ decimal.Decimal("999.123456789"),
+ ],
+ ),
+ ("BOOLEAN", [True, None, False, None]),
+ ("BOOL", [False, None, True, None]),
+ # TODO: Once https://issues.apache.org/jira/browse/ARROW-5450 is
+ # resolved, test with TIMESTAMP column. Conversion from pyarrow
+ # TimestampArray to list of Python objects fails with OverflowError:
+ # Python int too large to convert to C long.
+ #
+ # (
+ # "TIMESTAMP",
+ # [
+ # datetime.datetime(1, 1, 1, 0, 0, 0, tzinfo=pytz.utc),
+ # None,
+ # datetime.datetime(9999, 12, 31, 23, 59, 59, 999999, tzinfo=pytz.utc),
+ # datetime.datetime(1970, 1, 1, 0, 0, 0, tzinfo=pytz.utc),
+ # ],
+ # ),
+ (
+ "DATE",
+ [
+ datetime.date(1, 1, 1),
+ None,
+ datetime.date(9999, 12, 31),
+ datetime.date(1970, 1, 1),
+ ],
+ ),
+ (
+ "TIME",
+ [
+ datetime.time(0, 0, 0),
+ None,
+ datetime.time(23, 59, 59, 999999),
+ datetime.time(12, 0, 0),
+ ],
+ ),
+ # TODO: Once https://issues.apache.org/jira/browse/ARROW-5450 is
+ # resolved, test with DATETIME column. Conversion from pyarrow
+ # TimestampArray to list of Python objects fails with OverflowError:
+ # Python int too large to convert to C long.
+ #
+ # (
+ # "DATETIME",
+ # [
+ # datetime.datetime(1, 1, 1, 0, 0, 0),
+ # None,
+ # datetime.datetime(9999, 12, 31, 23, 59, 59, 999999),
+ # datetime.datetime(1970, 1, 1, 0, 0, 0),
+ # ],
+ # ),
+ (
+ "GEOGRAPHY",
+ [
+ "POINT(30 10)",
+ None,
+ "LINESTRING (30 10, 10 30, 40 40)",
+ "POLYGON ((30 10, 40 40, 20 40, 10 20, 30 10))",
+ ],
+ ),
+ ],
+)
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+@pytest.mark.skipif(pyarrow is None, reason="Requires `pyarrow`")
+def test_bq_to_arrow_array_w_nullable_scalars(module_under_test, bq_type, rows):
+ series = pandas.Series(rows, dtype="object")
+ bq_field = schema.SchemaField("field_name", bq_type)
+ arrow_array = module_under_test.bq_to_arrow_array(series, bq_field)
+ roundtrip = arrow_array.to_pylist()
+ assert rows == roundtrip
+
+
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+@pytest.mark.skipif(pyarrow is None, reason="Requires `pyarrow`")
+def test_bq_to_arrow_array_w_arrays(module_under_test):
+ rows = [[1, 2, 3], [], [4, 5, 6]]
+ series = pandas.Series(rows, dtype="object")
+ bq_field = schema.SchemaField("field_name", "INTEGER", mode="REPEATED")
+ arrow_array = module_under_test.bq_to_arrow_array(series, bq_field)
+ roundtrip = arrow_array.to_pylist()
+ assert rows == roundtrip
+
+
+@pytest.mark.parametrize("bq_type", ["RECORD", "record", "STRUCT", "struct"])
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+@pytest.mark.skipif(pyarrow is None, reason="Requires `pyarrow`")
+def test_bq_to_arrow_array_w_structs(module_under_test, bq_type):
+ rows = [
+ {"int_col": 123, "string_col": "abc"},
+ None,
+ {"int_col": 456, "string_col": "def"},
+ ]
+ series = pandas.Series(rows, dtype="object")
+ bq_field = schema.SchemaField(
+ "field_name",
+ bq_type,
+ fields=(
+ schema.SchemaField("int_col", "INTEGER"),
+ schema.SchemaField("string_col", "STRING"),
+ ),
+ )
+ arrow_array = module_under_test.bq_to_arrow_array(series, bq_field)
+ roundtrip = arrow_array.to_pylist()
+ assert rows == roundtrip
+
+
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+@pytest.mark.skipif(pyarrow is None, reason="Requires `pyarrow`")
+def test_bq_to_arrow_array_w_special_floats(module_under_test):
+ bq_field = schema.SchemaField("field_name", "FLOAT64")
+ rows = [float("-inf"), float("nan"), float("inf"), None]
+ series = pandas.Series(rows, dtype="object")
+ arrow_array = module_under_test.bq_to_arrow_array(series, bq_field)
+ roundtrip = arrow_array.to_pylist()
+ assert len(rows) == len(roundtrip)
+ assert roundtrip[0] == float("-inf")
+ # Since we are converting from pandas, NaN is treated as NULL in pyarrow
+ # due to pandas conventions.
+ # https://arrow.apache.org/docs/python/data.html#none-values-and-nan-handling
+ assert roundtrip[1] is None
+ assert roundtrip[2] == float("inf")
+ assert roundtrip[3] is None
+
+
+@pytest.mark.skipif(pyarrow is None, reason="Requires `pyarrow`")
+def test_bq_to_arrow_schema_w_unknown_type(module_under_test):
+ fields = (
+ schema.SchemaField("field1", "STRING"),
+ schema.SchemaField("field2", "INTEGER"),
+ # Don't know what to convert UNKNOWN_TYPE to, let type inference work,
+ # instead.
+ schema.SchemaField("field3", "UNKNOWN_TYPE"),
+ )
+ actual = module_under_test.bq_to_arrow_schema(fields)
+ assert actual is None
+
+
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+def test_get_column_or_index_not_found(module_under_test):
+ dataframe = pandas.DataFrame({"not_the_column_youre_looking_for": [1, 2, 3]})
+ with pytest.raises(ValueError, match="col_is_missing"):
+ module_under_test.get_column_or_index(dataframe, "col_is_missing")
+
+
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+def test_get_column_or_index_with_multiindex_not_found(module_under_test):
+ dataframe = pandas.DataFrame(
+ {"column_name": [1, 2, 3, 4, 5, 6]},
+ index=pandas.MultiIndex.from_tuples(
+ [("a", 0), ("a", 1), ("b", 0), ("b", 1), ("c", 0), ("c", 1)]
+ ),
+ )
+ with pytest.raises(ValueError, match="not_in_df"):
+ module_under_test.get_column_or_index(dataframe, "not_in_df")
+
+
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+def test_get_column_or_index_with_both_prefers_column(module_under_test):
+ dataframe = pandas.DataFrame(
+ {"some_name": [1, 2, 3]}, index=pandas.Index([0, 1, 2], name="some_name")
+ )
+ series = module_under_test.get_column_or_index(dataframe, "some_name")
+ expected = pandas.Series([1, 2, 3], name="some_name")
+ pandas.testing.assert_series_equal(series, expected)
+
+
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+def test_get_column_or_index_with_column(module_under_test):
+ dataframe = pandas.DataFrame({"column_name": [1, 2, 3], "other_column": [4, 5, 6]})
+ series = module_under_test.get_column_or_index(dataframe, "column_name")
+ expected = pandas.Series([1, 2, 3], name="column_name")
+ pandas.testing.assert_series_equal(series, expected)
+
+
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+def test_get_column_or_index_with_named_index(module_under_test):
+ dataframe = pandas.DataFrame(
+ {"column_name": [1, 2, 3]}, index=pandas.Index([4, 5, 6], name="index_name")
+ )
+ series = module_under_test.get_column_or_index(dataframe, "index_name")
+ expected = pandas.Series([4, 5, 6], name="index_name")
+ pandas.testing.assert_series_equal(series, expected)
+
+
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+def test_get_column_or_index_with_datetimeindex(module_under_test):
+ datetimes = [
+ datetime.datetime(2000, 1, 2, 3, 4, 5, 101),
+ datetime.datetime(2006, 7, 8, 9, 10, 11, 202),
+ datetime.datetime(2012, 1, 14, 15, 16, 17, 303),
+ ]
+ dataframe = pandas.DataFrame(
+ {"column_name": [1, 2, 3]},
+ index=pandas.DatetimeIndex(datetimes, name="index_name"),
+ )
+ series = module_under_test.get_column_or_index(dataframe, "index_name")
+ expected = pandas.Series(datetimes, name="index_name")
+ pandas.testing.assert_series_equal(series, expected)
+
+
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+def test_get_column_or_index_with_multiindex(module_under_test):
+ dataframe = pandas.DataFrame(
+ {"column_name": [1, 2, 3, 4, 5, 6]},
+ index=pandas.MultiIndex.from_tuples(
+ [("a", 0), ("a", 1), ("b", 0), ("b", 1), ("c", 0), ("c", 1)],
+ names=["letters", "numbers"],
+ ),
+ )
+
+ series = module_under_test.get_column_or_index(dataframe, "letters")
+ expected = pandas.Series(["a", "a", "b", "b", "c", "c"], name="letters")
+ pandas.testing.assert_series_equal(series, expected)
+
+ series = module_under_test.get_column_or_index(dataframe, "numbers")
+ expected = pandas.Series([0, 1, 0, 1, 0, 1], name="numbers")
+ pandas.testing.assert_series_equal(series, expected)
+
+
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+def test_list_columns_and_indexes_without_named_index(module_under_test):
+ df_data = collections.OrderedDict(
+ [
+ ("a_series", [1, 2, 3, 4]),
+ ("b_series", [0.1, 0.2, 0.3, 0.4]),
+ ("c_series", ["a", "b", "c", "d"]),
+ ]
+ )
+ dataframe = pandas.DataFrame(df_data)
+
+ columns_and_indexes = module_under_test.list_columns_and_indexes(dataframe)
+ expected = [
+ ("a_series", pandas.api.types.pandas_dtype("int64")),
+ ("b_series", pandas.api.types.pandas_dtype("float64")),
+ ("c_series", pandas.api.types.pandas_dtype("object")),
+ ]
+ assert columns_and_indexes == expected
+
+
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+def test_list_columns_and_indexes_with_named_index_same_as_column_name(
+ module_under_test
+):
+ df_data = collections.OrderedDict(
+ [
+ ("a_series", [1, 2, 3, 4]),
+ ("b_series", [0.1, 0.2, 0.3, 0.4]),
+ ("c_series", ["a", "b", "c", "d"]),
+ ]
+ )
+ dataframe = pandas.DataFrame(
+ df_data,
+ # Use same name as an integer column but a different datatype so that
+ # we can verify that the column is listed but the index isn't.
+ index=pandas.Index([0.1, 0.2, 0.3, 0.4], name="a_series"),
+ )
+
+ columns_and_indexes = module_under_test.list_columns_and_indexes(dataframe)
+ expected = [
+ ("a_series", pandas.api.types.pandas_dtype("int64")),
+ ("b_series", pandas.api.types.pandas_dtype("float64")),
+ ("c_series", pandas.api.types.pandas_dtype("object")),
+ ]
+ assert columns_and_indexes == expected
+
+
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+def test_list_columns_and_indexes_with_named_index(module_under_test):
+ df_data = collections.OrderedDict(
+ [
+ ("a_series", [1, 2, 3, 4]),
+ ("b_series", [0.1, 0.2, 0.3, 0.4]),
+ ("c_series", ["a", "b", "c", "d"]),
+ ]
+ )
+ dataframe = pandas.DataFrame(
+ df_data, index=pandas.Index([4, 5, 6, 7], name="a_index")
+ )
+
+ columns_and_indexes = module_under_test.list_columns_and_indexes(dataframe)
+ expected = [
+ ("a_index", pandas.api.types.pandas_dtype("int64")),
+ ("a_series", pandas.api.types.pandas_dtype("int64")),
+ ("b_series", pandas.api.types.pandas_dtype("float64")),
+ ("c_series", pandas.api.types.pandas_dtype("object")),
+ ]
+ assert columns_and_indexes == expected
+
+
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+def test_list_columns_and_indexes_with_multiindex(module_under_test):
+ df_data = collections.OrderedDict(
+ [
+ ("a_series", [1, 2, 3, 4]),
+ ("b_series", [0.1, 0.2, 0.3, 0.4]),
+ ("c_series", ["a", "b", "c", "d"]),
+ ]
+ )
+ dataframe = pandas.DataFrame(
+ df_data,
+ index=pandas.MultiIndex.from_tuples(
+ [(0, 0, 41), (0, 0, 42), (1, 0, 41), (1, 1, 41)],
+ names=[
+ "a_index",
+ # Use same name as column, but different dtype so we can verify
+ # the column type is included.
+ "b_series",
+ "c_index",
+ ],
+ ),
+ )
+
+ columns_and_indexes = module_under_test.list_columns_and_indexes(dataframe)
+ expected = [
+ ("a_index", pandas.api.types.pandas_dtype("int64")),
+ ("c_index", pandas.api.types.pandas_dtype("int64")),
+ ("a_series", pandas.api.types.pandas_dtype("int64")),
+ ("b_series", pandas.api.types.pandas_dtype("float64")),
+ ("c_series", pandas.api.types.pandas_dtype("object")),
+ ]
+ assert columns_and_indexes == expected
+
+
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+@pytest.mark.skipif(pyarrow is None, reason="Requires `pyarrow`")
+def test_dataframe_to_arrow_with_multiindex(module_under_test):
+ bq_schema = (
+ schema.SchemaField("str_index", "STRING"),
+ # int_index is intentionally omitted, to verify that it's okay to be
+ # missing indexes from the schema.
+ schema.SchemaField("dt_index", "DATETIME"),
+ schema.SchemaField("int_col", "INTEGER"),
+ schema.SchemaField("nullable_int_col", "INTEGER"),
+ schema.SchemaField("str_col", "STRING"),
+ )
+ df_data = collections.OrderedDict(
+ [
+ ("int_col", [1, 2, 3, 4, 5, 6]),
+ ("nullable_int_col", [6.0, float("nan"), 7.0, float("nan"), 8.0, 9.0]),
+ ("str_col", ["apple", "banana", "cherry", "durian", "etrog", "fig"]),
+ ]
+ )
+ df_index = pandas.MultiIndex.from_tuples(
+ [
+ ("a", 0, datetime.datetime(1999, 12, 31, 23, 59, 59, 999999)),
+ ("a", 0, datetime.datetime(2000, 1, 1, 0, 0, 0)),
+ ("a", 1, datetime.datetime(1999, 12, 31, 23, 59, 59, 999999)),
+ ("b", 1, datetime.datetime(2000, 1, 1, 0, 0, 0)),
+ ("b", 0, datetime.datetime(1999, 12, 31, 23, 59, 59, 999999)),
+ ("b", 0, datetime.datetime(2000, 1, 1, 0, 0, 0)),
+ ],
+ names=["str_index", "int_index", "dt_index"],
+ )
+ dataframe = pandas.DataFrame(df_data, index=df_index)
+
+ arrow_table = module_under_test.dataframe_to_arrow(dataframe, bq_schema)
+
+ assert arrow_table.schema.names == [
+ "str_index",
+ "dt_index",
+ "int_col",
+ "nullable_int_col",
+ "str_col",
+ ]
+ arrow_data = arrow_table.to_pydict()
+ assert arrow_data["str_index"] == ["a", "a", "a", "b", "b", "b"]
+ expected_dt_index = [
+ pandas.Timestamp(dt)
+ for dt in (
+ datetime.datetime(1999, 12, 31, 23, 59, 59, 999999),
+ datetime.datetime(2000, 1, 1, 0, 0, 0),
+ datetime.datetime(1999, 12, 31, 23, 59, 59, 999999),
+ datetime.datetime(2000, 1, 1, 0, 0, 0),
+ datetime.datetime(1999, 12, 31, 23, 59, 59, 999999),
+ datetime.datetime(2000, 1, 1, 0, 0, 0),
+ )
+ ]
+ assert arrow_data["dt_index"] == expected_dt_index
+ assert arrow_data["int_col"] == [1, 2, 3, 4, 5, 6]
+ assert arrow_data["nullable_int_col"] == [6, None, 7, None, 8, 9]
+ assert arrow_data["str_col"] == [
+ "apple",
+ "banana",
+ "cherry",
+ "durian",
+ "etrog",
+ "fig",
+ ]
+
+
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+@pytest.mark.skipif(pyarrow is None, reason="Requires `pyarrow`")
+def test_dataframe_to_arrow_with_required_fields(module_under_test):
+ bq_schema = (
+ schema.SchemaField("field01", "STRING", mode="REQUIRED"),
+ schema.SchemaField("field02", "BYTES", mode="REQUIRED"),
+ schema.SchemaField("field03", "INTEGER", mode="REQUIRED"),
+ schema.SchemaField("field04", "INT64", mode="REQUIRED"),
+ schema.SchemaField("field05", "FLOAT", mode="REQUIRED"),
+ schema.SchemaField("field06", "FLOAT64", mode="REQUIRED"),
+ schema.SchemaField("field07", "NUMERIC", mode="REQUIRED"),
+ schema.SchemaField("field08", "BOOLEAN", mode="REQUIRED"),
+ schema.SchemaField("field09", "BOOL", mode="REQUIRED"),
+ schema.SchemaField("field10", "TIMESTAMP", mode="REQUIRED"),
+ schema.SchemaField("field11", "DATE", mode="REQUIRED"),
+ schema.SchemaField("field12", "TIME", mode="REQUIRED"),
+ schema.SchemaField("field13", "DATETIME", mode="REQUIRED"),
+ schema.SchemaField("field14", "GEOGRAPHY", mode="REQUIRED"),
+ )
+ dataframe = pandas.DataFrame(
+ {
+ "field01": ["hello", "world"],
+ "field02": [b"abd", b"efg"],
+ "field03": [1, 2],
+ "field04": [3, 4],
+ "field05": [1.25, 9.75],
+ "field06": [-1.75, -3.5],
+ "field07": [decimal.Decimal("1.2345"), decimal.Decimal("6.7891")],
+ "field08": [True, False],
+ "field09": [False, True],
+ "field10": [
+ datetime.datetime(1970, 1, 1, 0, 0, 0, tzinfo=pytz.utc),
+ datetime.datetime(2012, 12, 21, 9, 7, 42, tzinfo=pytz.utc),
+ ],
+ "field11": [datetime.date(9999, 12, 31), datetime.date(1970, 1, 1)],
+ "field12": [datetime.time(23, 59, 59, 999999), datetime.time(12, 0, 0)],
+ "field13": [
+ datetime.datetime(1970, 1, 1, 0, 0, 0),
+ datetime.datetime(2012, 12, 21, 9, 7, 42),
+ ],
+ "field14": [
+ "POINT(30 10)",
+ "POLYGON ((30 10, 40 40, 20 40, 10 20, 30 10))",
+ ],
+ }
+ )
+
+ arrow_table = module_under_test.dataframe_to_arrow(dataframe, bq_schema)
+ arrow_schema = arrow_table.schema
+
+ assert len(arrow_schema) == len(bq_schema)
+ for arrow_field in arrow_schema:
+ assert not arrow_field.nullable
+
+
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+@pytest.mark.skipif(pyarrow is None, reason="Requires `pyarrow`")
+def test_dataframe_to_arrow_with_unknown_type(module_under_test):
+ bq_schema = (
+ schema.SchemaField("field00", "UNKNOWN_TYPE"),
+ schema.SchemaField("field01", "STRING"),
+ schema.SchemaField("field02", "BYTES"),
+ schema.SchemaField("field03", "INTEGER"),
+ )
+ dataframe = pandas.DataFrame(
+ {
+ "field00": ["whoami", "whatami"],
+ "field01": ["hello", "world"],
+ "field02": [b"abd", b"efg"],
+ "field03": [1, 2],
+ }
+ )
+
+ with warnings.catch_warnings(record=True) as warned:
+ arrow_table = module_under_test.dataframe_to_arrow(dataframe, bq_schema)
+ arrow_schema = arrow_table.schema
+
+ assert len(warned) == 1
+ warning = warned[0]
+ assert "field00" in str(warning)
+
+ assert len(arrow_schema) == len(bq_schema)
+ assert arrow_schema[0].name == "field00"
+ assert arrow_schema[1].name == "field01"
+ assert arrow_schema[2].name == "field02"
+ assert arrow_schema[3].name == "field03"
+
+
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+def test_dataframe_to_parquet_without_pyarrow(module_under_test, monkeypatch):
+ monkeypatch.setattr(module_under_test, "pyarrow", None)
+ with pytest.raises(ValueError) as exc_context:
+ module_under_test.dataframe_to_parquet(pandas.DataFrame(), (), None)
+ assert "pyarrow is required" in str(exc_context.value)
+
+
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+@pytest.mark.skipif(pyarrow is None, reason="Requires `pyarrow`")
+def test_dataframe_to_parquet_w_extra_fields(module_under_test, monkeypatch):
+ with pytest.raises(ValueError) as exc_context:
+ module_under_test.dataframe_to_parquet(
+ pandas.DataFrame(), (schema.SchemaField("not_in_df", "STRING"),), None
+ )
+ message = str(exc_context.value)
+ assert "bq_schema contains fields not present in dataframe" in message
+ assert "not_in_df" in message
+
+
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+@pytest.mark.skipif(pyarrow is None, reason="Requires `pyarrow`")
+def test_dataframe_to_parquet_w_missing_fields(module_under_test, monkeypatch):
+ with pytest.raises(ValueError) as exc_context:
+ module_under_test.dataframe_to_parquet(
+ pandas.DataFrame({"not_in_bq": [1, 2, 3]}), (), None
+ )
+ message = str(exc_context.value)
+ assert "bq_schema is missing fields from dataframe" in message
+ assert "not_in_bq" in message
+
+
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+@pytest.mark.skipif(pyarrow is None, reason="Requires `pyarrow`")
+def test_dataframe_to_parquet_compression_method(module_under_test):
+ bq_schema = (schema.SchemaField("field00", "STRING"),)
+ dataframe = pandas.DataFrame({"field00": ["foo", "bar"]})
+
+ write_table_patch = mock.patch.object(
+ module_under_test.pyarrow.parquet, "write_table", autospec=True
+ )
+
+ with write_table_patch as fake_write_table:
+ module_under_test.dataframe_to_parquet(
+ dataframe, bq_schema, None, parquet_compression="ZSTD"
+ )
+
+ call_args = fake_write_table.call_args
+ assert call_args is not None
+ assert call_args.kwargs.get("compression") == "ZSTD"
diff --git a/bigquery/tests/unit/test_client.py b/bigquery/tests/unit/test_client.py
new file mode 100644
index 000000000000..f31d8587322b
--- /dev/null
+++ b/bigquery/tests/unit/test_client.py
@@ -0,0 +1,6168 @@
+# Copyright 2015 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import copy
+import collections
+import datetime
+import decimal
+import email
+import gzip
+import io
+import json
+import unittest
+import warnings
+
+import mock
+import requests
+import six
+from six.moves import http_client
+import pytest
+import pytz
+
+try:
+ import pandas
+except (ImportError, AttributeError): # pragma: NO COVER
+ pandas = None
+try:
+ import pyarrow
+except (ImportError, AttributeError): # pragma: NO COVER
+ pyarrow = None
+
+import google.api_core.exceptions
+from google.api_core.gapic_v1 import client_info
+import google.cloud._helpers
+from google.cloud import bigquery_v2
+from google.cloud.bigquery.dataset import DatasetReference
+from tests.unit.helpers import make_connection
+
+
+def _make_credentials():
+ import google.auth.credentials
+
+ return mock.Mock(spec=google.auth.credentials.Credentials)
+
+
+def _make_list_partitons_meta_info(project, dataset_id, table_id, num_rows=0):
+ return {
+ "tableReference": {
+ "projectId": project,
+ "datasetId": dataset_id,
+ "tableId": "{}$__PARTITIONS_SUMMARY__".format(table_id),
+ },
+ "schema": {
+ "fields": [
+ {"name": "project_id", "type": "STRING", "mode": "NULLABLE"},
+ {"name": "dataset_id", "type": "STRING", "mode": "NULLABLE"},
+ {"name": "table_id", "type": "STRING", "mode": "NULLABLE"},
+ {"name": "partition_id", "type": "STRING", "mode": "NULLABLE"},
+ ]
+ },
+ "etag": "ETAG",
+ "numRows": num_rows,
+ }
+
+
+class TestClient(unittest.TestCase):
+
+ PROJECT = "PROJECT"
+ DS_ID = "DATASET_ID"
+ TABLE_ID = "TABLE_ID"
+ MODEL_ID = "MODEL_ID"
+ TABLE_REF = DatasetReference(PROJECT, DS_ID).table(TABLE_ID)
+ KMS_KEY_NAME = "projects/1/locations/global/keyRings/1/cryptoKeys/1"
+ LOCATION = "us-central"
+
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery.client import Client
+
+ return Client
+
+ def _make_one(self, *args, **kw):
+ return self._get_target_class()(*args, **kw)
+
+ def _make_table_resource(self):
+ return {
+ "id": "%s:%s:%s" % (self.PROJECT, self.DS_ID, self.TABLE_ID),
+ "tableReference": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.TABLE_ID,
+ },
+ }
+
+ def test_ctor_defaults(self):
+ from google.cloud.bigquery._http import Connection
+
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ self.assertIsInstance(client._connection, Connection)
+ self.assertIs(client._connection.credentials, creds)
+ self.assertIs(client._connection.http, http)
+ self.assertIsNone(client.location)
+ self.assertEqual(
+ client._connection.API_BASE_URL, Connection.DEFAULT_API_ENDPOINT
+ )
+
+ def test_ctor_w_empty_client_options(self):
+ from google.api_core.client_options import ClientOptions
+
+ creds = _make_credentials()
+ http = object()
+ client_options = ClientOptions()
+ client = self._make_one(
+ project=self.PROJECT,
+ credentials=creds,
+ _http=http,
+ client_options=client_options,
+ )
+ self.assertEqual(
+ client._connection.API_BASE_URL, client._connection.DEFAULT_API_ENDPOINT
+ )
+
+ def test_ctor_w_client_options_dict(self):
+
+ creds = _make_credentials()
+ http = object()
+ client_options = {"api_endpoint": "https://www.foo-googleapis.com"}
+ client = self._make_one(
+ project=self.PROJECT,
+ credentials=creds,
+ _http=http,
+ client_options=client_options,
+ )
+ self.assertEqual(
+ client._connection.API_BASE_URL, "https://www.foo-googleapis.com"
+ )
+
+ def test_ctor_w_client_options_object(self):
+ from google.api_core.client_options import ClientOptions
+
+ creds = _make_credentials()
+ http = object()
+ client_options = ClientOptions(api_endpoint="https://www.foo-googleapis.com")
+ client = self._make_one(
+ project=self.PROJECT,
+ credentials=creds,
+ _http=http,
+ client_options=client_options,
+ )
+ self.assertEqual(
+ client._connection.API_BASE_URL, "https://www.foo-googleapis.com"
+ )
+
+ def test_ctor_w_location(self):
+ from google.cloud.bigquery._http import Connection
+
+ creds = _make_credentials()
+ http = object()
+ location = "us-central"
+ client = self._make_one(
+ project=self.PROJECT, credentials=creds, _http=http, location=location
+ )
+ self.assertIsInstance(client._connection, Connection)
+ self.assertIs(client._connection.credentials, creds)
+ self.assertIs(client._connection.http, http)
+ self.assertEqual(client.location, location)
+
+ def test_ctor_w_query_job_config(self):
+ from google.cloud.bigquery._http import Connection
+ from google.cloud.bigquery import QueryJobConfig
+
+ creds = _make_credentials()
+ http = object()
+ location = "us-central"
+ job_config = QueryJobConfig()
+ job_config.dry_run = True
+
+ client = self._make_one(
+ project=self.PROJECT,
+ credentials=creds,
+ _http=http,
+ location=location,
+ default_query_job_config=job_config,
+ )
+ self.assertIsInstance(client._connection, Connection)
+ self.assertIs(client._connection.credentials, creds)
+ self.assertIs(client._connection.http, http)
+ self.assertEqual(client.location, location)
+
+ self.assertIsInstance(client._default_query_job_config, QueryJobConfig)
+ self.assertTrue(client._default_query_job_config.dry_run)
+
+ def test__get_query_results_miss_w_explicit_project_and_timeout(self):
+ from google.cloud.exceptions import NotFound
+
+ creds = _make_credentials()
+ client = self._make_one(self.PROJECT, creds)
+ conn = client._connection = make_connection()
+
+ with self.assertRaises(NotFound):
+ client._get_query_results(
+ "nothere",
+ None,
+ project="other-project",
+ location=self.LOCATION,
+ timeout_ms=500,
+ )
+
+ conn.api_request.assert_called_once_with(
+ method="GET",
+ path="/projects/other-project/queries/nothere",
+ query_params={"maxResults": 0, "timeoutMs": 500, "location": self.LOCATION},
+ )
+
+ def test__get_query_results_miss_w_client_location(self):
+ from google.cloud.exceptions import NotFound
+
+ creds = _make_credentials()
+ client = self._make_one(self.PROJECT, creds, location=self.LOCATION)
+ conn = client._connection = make_connection()
+
+ with self.assertRaises(NotFound):
+ client._get_query_results("nothere", None)
+
+ conn.api_request.assert_called_once_with(
+ method="GET",
+ path="/projects/PROJECT/queries/nothere",
+ query_params={"maxResults": 0, "location": self.LOCATION},
+ )
+
+ def test__get_query_results_hit(self):
+ job_id = "query_job"
+ data = {
+ "kind": "bigquery#getQueryResultsResponse",
+ "etag": "some-tag",
+ "schema": {
+ "fields": [
+ {"name": "title", "type": "STRING", "mode": "NULLABLE"},
+ {"name": "unique_words", "type": "INTEGER", "mode": "NULLABLE"},
+ ]
+ },
+ "jobReference": {"projectId": self.PROJECT, "jobId": job_id},
+ "totalRows": "10",
+ "totalBytesProcessed": "2464625",
+ "jobComplete": True,
+ "cacheHit": False,
+ }
+
+ creds = _make_credentials()
+ client = self._make_one(self.PROJECT, creds)
+ client._connection = make_connection(data)
+ query_results = client._get_query_results(job_id, None)
+
+ self.assertEqual(query_results.total_rows, 10)
+ self.assertTrue(query_results.complete)
+
+ def test_get_service_account_email(self):
+ path = "/projects/%s/serviceAccount" % (self.PROJECT,)
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ email = "bq-123@bigquery-encryption.iam.gserviceaccount.com"
+ resource = {"kind": "bigquery#getServiceAccountResponse", "email": email}
+ conn = client._connection = make_connection(resource)
+
+ service_account_email = client.get_service_account_email()
+
+ conn.api_request.assert_called_once_with(method="GET", path=path)
+ self.assertEqual(service_account_email, email)
+
+ def test_get_service_account_email_w_alternate_project(self):
+ project = "my-alternate-project"
+ path = "/projects/%s/serviceAccount" % (project,)
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ email = "bq-123@bigquery-encryption.iam.gserviceaccount.com"
+ resource = {"kind": "bigquery#getServiceAccountResponse", "email": email}
+ conn = client._connection = make_connection(resource)
+
+ service_account_email = client.get_service_account_email(project=project)
+
+ conn.api_request.assert_called_once_with(method="GET", path=path)
+ self.assertEqual(service_account_email, email)
+
+ def test_list_projects_defaults(self):
+ from google.cloud.bigquery.client import Project
+
+ PROJECT_1 = "PROJECT_ONE"
+ PROJECT_2 = "PROJECT_TWO"
+ TOKEN = "TOKEN"
+ DATA = {
+ "nextPageToken": TOKEN,
+ "projects": [
+ {
+ "kind": "bigquery#project",
+ "id": PROJECT_1,
+ "numericId": 1,
+ "projectReference": {"projectId": PROJECT_1},
+ "friendlyName": "One",
+ },
+ {
+ "kind": "bigquery#project",
+ "id": PROJECT_2,
+ "numericId": 2,
+ "projectReference": {"projectId": PROJECT_2},
+ "friendlyName": "Two",
+ },
+ ],
+ }
+ creds = _make_credentials()
+ client = self._make_one(PROJECT_1, creds)
+ conn = client._connection = make_connection(DATA)
+
+ iterator = client.list_projects()
+ page = six.next(iterator.pages)
+ projects = list(page)
+ token = iterator.next_page_token
+
+ self.assertEqual(len(projects), len(DATA["projects"]))
+ for found, expected in zip(projects, DATA["projects"]):
+ self.assertIsInstance(found, Project)
+ self.assertEqual(found.project_id, expected["id"])
+ self.assertEqual(found.numeric_id, expected["numericId"])
+ self.assertEqual(found.friendly_name, expected["friendlyName"])
+ self.assertEqual(token, TOKEN)
+
+ conn.api_request.assert_called_once_with(
+ method="GET", path="/projects", query_params={}
+ )
+
+ def test_list_projects_explicit_response_missing_projects_key(self):
+ TOKEN = "TOKEN"
+ DATA = {}
+ creds = _make_credentials()
+ client = self._make_one(self.PROJECT, creds)
+ conn = client._connection = make_connection(DATA)
+
+ iterator = client.list_projects(max_results=3, page_token=TOKEN)
+ page = six.next(iterator.pages)
+ projects = list(page)
+ token = iterator.next_page_token
+
+ self.assertEqual(len(projects), 0)
+ self.assertIsNone(token)
+
+ conn.api_request.assert_called_once_with(
+ method="GET",
+ path="/projects",
+ query_params={"maxResults": 3, "pageToken": TOKEN},
+ )
+
+ def test_list_datasets_defaults(self):
+ from google.cloud.bigquery.dataset import DatasetListItem
+
+ DATASET_1 = "dataset_one"
+ DATASET_2 = "dataset_two"
+ PATH = "projects/%s/datasets" % self.PROJECT
+ TOKEN = "TOKEN"
+ DATA = {
+ "nextPageToken": TOKEN,
+ "datasets": [
+ {
+ "kind": "bigquery#dataset",
+ "id": "%s:%s" % (self.PROJECT, DATASET_1),
+ "datasetReference": {
+ "datasetId": DATASET_1,
+ "projectId": self.PROJECT,
+ },
+ "friendlyName": None,
+ },
+ {
+ "kind": "bigquery#dataset",
+ "id": "%s:%s" % (self.PROJECT, DATASET_2),
+ "datasetReference": {
+ "datasetId": DATASET_2,
+ "projectId": self.PROJECT,
+ },
+ "friendlyName": "Two",
+ },
+ ],
+ }
+ creds = _make_credentials()
+ client = self._make_one(self.PROJECT, creds)
+ conn = client._connection = make_connection(DATA)
+
+ iterator = client.list_datasets()
+ page = six.next(iterator.pages)
+ datasets = list(page)
+ token = iterator.next_page_token
+
+ self.assertEqual(len(datasets), len(DATA["datasets"]))
+ for found, expected in zip(datasets, DATA["datasets"]):
+ self.assertIsInstance(found, DatasetListItem)
+ self.assertEqual(found.full_dataset_id, expected["id"])
+ self.assertEqual(found.friendly_name, expected["friendlyName"])
+ self.assertEqual(token, TOKEN)
+
+ conn.api_request.assert_called_once_with(
+ method="GET", path="/%s" % PATH, query_params={}
+ )
+
+ def test_list_datasets_w_project(self):
+ creds = _make_credentials()
+ client = self._make_one(self.PROJECT, creds)
+ conn = client._connection = make_connection({})
+
+ list(client.list_datasets(project="other-project"))
+
+ conn.api_request.assert_called_once_with(
+ method="GET", path="/projects/other-project/datasets", query_params={}
+ )
+
+ def test_list_datasets_explicit_response_missing_datasets_key(self):
+ PATH = "projects/%s/datasets" % self.PROJECT
+ TOKEN = "TOKEN"
+ FILTER = "FILTER"
+ DATA = {}
+ creds = _make_credentials()
+ client = self._make_one(self.PROJECT, creds)
+ conn = client._connection = make_connection(DATA)
+
+ iterator = client.list_datasets(
+ include_all=True, filter=FILTER, max_results=3, page_token=TOKEN
+ )
+ page = six.next(iterator.pages)
+ datasets = list(page)
+ token = iterator.next_page_token
+
+ self.assertEqual(len(datasets), 0)
+ self.assertIsNone(token)
+
+ conn.api_request.assert_called_once_with(
+ method="GET",
+ path="/%s" % PATH,
+ query_params={
+ "all": True,
+ "filter": FILTER,
+ "maxResults": 3,
+ "pageToken": TOKEN,
+ },
+ )
+
+ def test_dataset_with_specified_project(self):
+ from google.cloud.bigquery.dataset import DatasetReference
+
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ dataset = client.dataset(self.DS_ID, self.PROJECT)
+ self.assertIsInstance(dataset, DatasetReference)
+ self.assertEqual(dataset.dataset_id, self.DS_ID)
+ self.assertEqual(dataset.project, self.PROJECT)
+
+ def test_dataset_with_default_project(self):
+ from google.cloud.bigquery.dataset import DatasetReference
+
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ dataset = client.dataset(self.DS_ID)
+ self.assertIsInstance(dataset, DatasetReference)
+ self.assertEqual(dataset.dataset_id, self.DS_ID)
+ self.assertEqual(dataset.project, self.PROJECT)
+
+ def test_get_dataset(self):
+ from google.cloud.exceptions import ServerError
+
+ path = "projects/%s/datasets/%s" % (self.PROJECT, self.DS_ID)
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ resource = {
+ "id": "%s:%s" % (self.PROJECT, self.DS_ID),
+ "datasetReference": {"projectId": self.PROJECT, "datasetId": self.DS_ID},
+ }
+ conn = client._connection = make_connection(resource)
+ dataset_ref = client.dataset(self.DS_ID)
+
+ dataset = client.get_dataset(dataset_ref)
+
+ conn.api_request.assert_called_once_with(method="GET", path="/%s" % path)
+ self.assertEqual(dataset.dataset_id, self.DS_ID)
+
+ # Test retry.
+
+ # Not a cloud API exception (missing 'errors' field).
+ client._connection = make_connection(Exception(""), resource)
+ with self.assertRaises(Exception):
+ client.get_dataset(dataset_ref)
+
+ # Zero-length errors field.
+ client._connection = make_connection(ServerError(""), resource)
+ with self.assertRaises(ServerError):
+ client.get_dataset(dataset_ref)
+
+ # Non-retryable reason.
+ client._connection = make_connection(
+ ServerError("", errors=[{"reason": "serious"}]), resource
+ )
+ with self.assertRaises(ServerError):
+ client.get_dataset(dataset_ref)
+
+ # Retryable reason, but retry is disabled.
+ client._connection = make_connection(
+ ServerError("", errors=[{"reason": "backendError"}]), resource
+ )
+ with self.assertRaises(ServerError):
+ client.get_dataset(dataset_ref, retry=None)
+
+ # Retryable reason, default retry: success.
+ client._connection = make_connection(
+ ServerError("", errors=[{"reason": "backendError"}]), resource
+ )
+ dataset = client.get_dataset(
+ # Test with a string for dataset ID.
+ dataset_ref.dataset_id
+ )
+ self.assertEqual(dataset.dataset_id, self.DS_ID)
+
+ def test_create_dataset_minimal(self):
+ from google.cloud.bigquery.dataset import Dataset
+
+ PATH = "projects/%s/datasets" % self.PROJECT
+ RESOURCE = {
+ "datasetReference": {"projectId": self.PROJECT, "datasetId": self.DS_ID},
+ "etag": "etag",
+ "id": "%s:%s" % (self.PROJECT, self.DS_ID),
+ }
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ conn = client._connection = make_connection(RESOURCE)
+
+ ds_ref = client.dataset(self.DS_ID)
+ before = Dataset(ds_ref)
+
+ after = client.create_dataset(before)
+
+ self.assertEqual(after.dataset_id, self.DS_ID)
+ self.assertEqual(after.project, self.PROJECT)
+ self.assertEqual(after.etag, RESOURCE["etag"])
+ self.assertEqual(after.full_dataset_id, RESOURCE["id"])
+
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path="/%s" % PATH,
+ data={
+ "datasetReference": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ },
+ "labels": {},
+ },
+ )
+
+ def test_create_dataset_w_attrs(self):
+ from google.cloud.bigquery.dataset import Dataset, AccessEntry
+
+ PATH = "projects/%s/datasets" % self.PROJECT
+ DESCRIPTION = "DESC"
+ FRIENDLY_NAME = "FN"
+ LOCATION = "US"
+ USER_EMAIL = "phred@example.com"
+ LABELS = {"color": "red"}
+ VIEW = {
+ "projectId": "my-proj",
+ "datasetId": "starry-skies",
+ "tableId": "northern-hemisphere",
+ }
+ RESOURCE = {
+ "datasetReference": {"projectId": self.PROJECT, "datasetId": self.DS_ID},
+ "etag": "etag",
+ "id": "%s:%s" % (self.PROJECT, self.DS_ID),
+ "description": DESCRIPTION,
+ "friendlyName": FRIENDLY_NAME,
+ "location": LOCATION,
+ "defaultTableExpirationMs": "3600",
+ "labels": LABELS,
+ "access": [{"role": "OWNER", "userByEmail": USER_EMAIL}, {"view": VIEW}],
+ }
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ conn = client._connection = make_connection(RESOURCE)
+ entries = [
+ AccessEntry("OWNER", "userByEmail", USER_EMAIL),
+ AccessEntry(None, "view", VIEW),
+ ]
+
+ ds_ref = client.dataset(self.DS_ID)
+ before = Dataset(ds_ref)
+ before.access_entries = entries
+ before.description = DESCRIPTION
+ before.friendly_name = FRIENDLY_NAME
+ before.default_table_expiration_ms = 3600
+ before.location = LOCATION
+ before.labels = LABELS
+
+ after = client.create_dataset(before)
+
+ self.assertEqual(after.dataset_id, self.DS_ID)
+ self.assertEqual(after.project, self.PROJECT)
+ self.assertEqual(after.etag, RESOURCE["etag"])
+ self.assertEqual(after.full_dataset_id, RESOURCE["id"])
+ self.assertEqual(after.description, DESCRIPTION)
+ self.assertEqual(after.friendly_name, FRIENDLY_NAME)
+ self.assertEqual(after.location, LOCATION)
+ self.assertEqual(after.default_table_expiration_ms, 3600)
+ self.assertEqual(after.labels, LABELS)
+
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path="/%s" % PATH,
+ data={
+ "datasetReference": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ },
+ "description": DESCRIPTION,
+ "friendlyName": FRIENDLY_NAME,
+ "location": LOCATION,
+ "defaultTableExpirationMs": "3600",
+ "access": [
+ {"role": "OWNER", "userByEmail": USER_EMAIL},
+ {"view": VIEW},
+ ],
+ "labels": LABELS,
+ },
+ )
+
+ def test_create_dataset_w_custom_property(self):
+ # The library should handle sending properties to the API that are not
+ # yet part of the library
+ from google.cloud.bigquery.dataset import Dataset
+
+ path = "/projects/%s/datasets" % self.PROJECT
+ resource = {
+ "datasetReference": {"projectId": self.PROJECT, "datasetId": self.DS_ID},
+ "newAlphaProperty": "unreleased property",
+ }
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ conn = client._connection = make_connection(resource)
+
+ ds_ref = client.dataset(self.DS_ID)
+ before = Dataset(ds_ref)
+ before._properties["newAlphaProperty"] = "unreleased property"
+
+ after = client.create_dataset(before)
+
+ self.assertEqual(after.dataset_id, self.DS_ID)
+ self.assertEqual(after.project, self.PROJECT)
+ self.assertEqual(after._properties["newAlphaProperty"], "unreleased property")
+
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path=path,
+ data={
+ "datasetReference": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ },
+ "newAlphaProperty": "unreleased property",
+ "labels": {},
+ },
+ )
+
+ def test_create_dataset_w_client_location_wo_dataset_location(self):
+ from google.cloud.bigquery.dataset import Dataset
+
+ PATH = "projects/%s/datasets" % self.PROJECT
+ RESOURCE = {
+ "datasetReference": {"projectId": self.PROJECT, "datasetId": self.DS_ID},
+ "etag": "etag",
+ "id": "%s:%s" % (self.PROJECT, self.DS_ID),
+ "location": self.LOCATION,
+ }
+ creds = _make_credentials()
+ client = self._make_one(
+ project=self.PROJECT, credentials=creds, location=self.LOCATION
+ )
+ conn = client._connection = make_connection(RESOURCE)
+
+ ds_ref = client.dataset(self.DS_ID)
+ before = Dataset(ds_ref)
+
+ after = client.create_dataset(before)
+
+ self.assertEqual(after.dataset_id, self.DS_ID)
+ self.assertEqual(after.project, self.PROJECT)
+ self.assertEqual(after.etag, RESOURCE["etag"])
+ self.assertEqual(after.full_dataset_id, RESOURCE["id"])
+ self.assertEqual(after.location, self.LOCATION)
+
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path="/%s" % PATH,
+ data={
+ "datasetReference": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ },
+ "labels": {},
+ "location": self.LOCATION,
+ },
+ )
+
+ def test_create_dataset_w_client_location_w_dataset_location(self):
+ from google.cloud.bigquery.dataset import Dataset
+
+ PATH = "projects/%s/datasets" % self.PROJECT
+ OTHER_LOCATION = "EU"
+ RESOURCE = {
+ "datasetReference": {"projectId": self.PROJECT, "datasetId": self.DS_ID},
+ "etag": "etag",
+ "id": "%s:%s" % (self.PROJECT, self.DS_ID),
+ "location": OTHER_LOCATION,
+ }
+ creds = _make_credentials()
+ client = self._make_one(
+ project=self.PROJECT, credentials=creds, location=self.LOCATION
+ )
+ conn = client._connection = make_connection(RESOURCE)
+
+ ds_ref = client.dataset(self.DS_ID)
+ before = Dataset(ds_ref)
+ before.location = OTHER_LOCATION
+
+ after = client.create_dataset(before)
+
+ self.assertEqual(after.dataset_id, self.DS_ID)
+ self.assertEqual(after.project, self.PROJECT)
+ self.assertEqual(after.etag, RESOURCE["etag"])
+ self.assertEqual(after.full_dataset_id, RESOURCE["id"])
+ self.assertEqual(after.location, OTHER_LOCATION)
+
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path="/%s" % PATH,
+ data={
+ "datasetReference": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ },
+ "labels": {},
+ "location": OTHER_LOCATION,
+ },
+ )
+
+ def test_create_dataset_w_reference(self):
+ path = "/projects/%s/datasets" % self.PROJECT
+ resource = {
+ "datasetReference": {"projectId": self.PROJECT, "datasetId": self.DS_ID},
+ "etag": "etag",
+ "id": "%s:%s" % (self.PROJECT, self.DS_ID),
+ "location": self.LOCATION,
+ }
+ creds = _make_credentials()
+ client = self._make_one(
+ project=self.PROJECT, credentials=creds, location=self.LOCATION
+ )
+ conn = client._connection = make_connection(resource)
+
+ dataset = client.create_dataset(client.dataset(self.DS_ID))
+
+ self.assertEqual(dataset.dataset_id, self.DS_ID)
+ self.assertEqual(dataset.project, self.PROJECT)
+ self.assertEqual(dataset.etag, resource["etag"])
+ self.assertEqual(dataset.full_dataset_id, resource["id"])
+ self.assertEqual(dataset.location, self.LOCATION)
+
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path=path,
+ data={
+ "datasetReference": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ },
+ "labels": {},
+ "location": self.LOCATION,
+ },
+ )
+
+ def test_create_dataset_w_fully_qualified_string(self):
+ path = "/projects/%s/datasets" % self.PROJECT
+ resource = {
+ "datasetReference": {"projectId": self.PROJECT, "datasetId": self.DS_ID},
+ "etag": "etag",
+ "id": "%s:%s" % (self.PROJECT, self.DS_ID),
+ "location": self.LOCATION,
+ }
+ creds = _make_credentials()
+ client = self._make_one(
+ project=self.PROJECT, credentials=creds, location=self.LOCATION
+ )
+ conn = client._connection = make_connection(resource)
+
+ dataset = client.create_dataset("{}.{}".format(self.PROJECT, self.DS_ID))
+
+ self.assertEqual(dataset.dataset_id, self.DS_ID)
+ self.assertEqual(dataset.project, self.PROJECT)
+ self.assertEqual(dataset.etag, resource["etag"])
+ self.assertEqual(dataset.full_dataset_id, resource["id"])
+ self.assertEqual(dataset.location, self.LOCATION)
+
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path=path,
+ data={
+ "datasetReference": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ },
+ "labels": {},
+ "location": self.LOCATION,
+ },
+ )
+
+ def test_create_dataset_w_string(self):
+ path = "/projects/%s/datasets" % self.PROJECT
+ resource = {
+ "datasetReference": {"projectId": self.PROJECT, "datasetId": self.DS_ID},
+ "etag": "etag",
+ "id": "%s:%s" % (self.PROJECT, self.DS_ID),
+ "location": self.LOCATION,
+ }
+ creds = _make_credentials()
+ client = self._make_one(
+ project=self.PROJECT, credentials=creds, location=self.LOCATION
+ )
+ conn = client._connection = make_connection(resource)
+
+ dataset = client.create_dataset(self.DS_ID)
+
+ self.assertEqual(dataset.dataset_id, self.DS_ID)
+ self.assertEqual(dataset.project, self.PROJECT)
+ self.assertEqual(dataset.etag, resource["etag"])
+ self.assertEqual(dataset.full_dataset_id, resource["id"])
+ self.assertEqual(dataset.location, self.LOCATION)
+
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path=path,
+ data={
+ "datasetReference": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ },
+ "labels": {},
+ "location": self.LOCATION,
+ },
+ )
+
+ def test_create_dataset_alreadyexists_w_exists_ok_false(self):
+ creds = _make_credentials()
+ client = self._make_one(
+ project=self.PROJECT, credentials=creds, location=self.LOCATION
+ )
+ client._connection = make_connection(
+ google.api_core.exceptions.AlreadyExists("dataset already exists")
+ )
+
+ with pytest.raises(google.api_core.exceptions.AlreadyExists):
+ client.create_dataset(self.DS_ID)
+
+ def test_create_dataset_alreadyexists_w_exists_ok_true(self):
+ post_path = "/projects/{}/datasets".format(self.PROJECT)
+ get_path = "/projects/{}/datasets/{}".format(self.PROJECT, self.DS_ID)
+ resource = {
+ "datasetReference": {"projectId": self.PROJECT, "datasetId": self.DS_ID},
+ "etag": "etag",
+ "id": "{}:{}".format(self.PROJECT, self.DS_ID),
+ "location": self.LOCATION,
+ }
+ creds = _make_credentials()
+ client = self._make_one(
+ project=self.PROJECT, credentials=creds, location=self.LOCATION
+ )
+ conn = client._connection = make_connection(
+ google.api_core.exceptions.AlreadyExists("dataset already exists"), resource
+ )
+
+ dataset = client.create_dataset(self.DS_ID, exists_ok=True)
+
+ self.assertEqual(dataset.dataset_id, self.DS_ID)
+ self.assertEqual(dataset.project, self.PROJECT)
+ self.assertEqual(dataset.etag, resource["etag"])
+ self.assertEqual(dataset.full_dataset_id, resource["id"])
+ self.assertEqual(dataset.location, self.LOCATION)
+
+ conn.api_request.assert_has_calls(
+ [
+ mock.call(
+ method="POST",
+ path=post_path,
+ data={
+ "datasetReference": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ },
+ "labels": {},
+ "location": self.LOCATION,
+ },
+ ),
+ mock.call(method="GET", path=get_path),
+ ]
+ )
+
+ def test_create_routine_w_minimal_resource(self):
+ from google.cloud.bigquery.routine import Routine
+ from google.cloud.bigquery.routine import RoutineReference
+
+ creds = _make_credentials()
+ resource = {
+ "routineReference": {
+ "projectId": "test-routine-project",
+ "datasetId": "test_routines",
+ "routineId": "minimal_routine",
+ }
+ }
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ conn = client._connection = make_connection(resource)
+ full_routine_id = "test-routine-project.test_routines.minimal_routine"
+ routine = Routine(full_routine_id)
+
+ actual_routine = client.create_routine(routine)
+
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path="/projects/test-routine-project/datasets/test_routines/routines",
+ data=resource,
+ )
+ self.assertEqual(
+ actual_routine.reference, RoutineReference.from_string(full_routine_id)
+ )
+
+ def test_create_routine_w_conflict(self):
+ from google.cloud.bigquery.routine import Routine
+
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ conn = client._connection = make_connection(
+ google.api_core.exceptions.AlreadyExists("routine already exists")
+ )
+ full_routine_id = "test-routine-project.test_routines.minimal_routine"
+ routine = Routine(full_routine_id)
+
+ with pytest.raises(google.api_core.exceptions.AlreadyExists):
+ client.create_routine(routine)
+
+ resource = {
+ "routineReference": {
+ "projectId": "test-routine-project",
+ "datasetId": "test_routines",
+ "routineId": "minimal_routine",
+ }
+ }
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path="/projects/test-routine-project/datasets/test_routines/routines",
+ data=resource,
+ )
+
+ def test_create_routine_w_conflict_exists_ok(self):
+ from google.cloud.bigquery.routine import Routine
+
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ resource = {
+ "routineReference": {
+ "projectId": "test-routine-project",
+ "datasetId": "test_routines",
+ "routineId": "minimal_routine",
+ }
+ }
+ conn = client._connection = make_connection(
+ google.api_core.exceptions.AlreadyExists("routine already exists"), resource
+ )
+ full_routine_id = "test-routine-project.test_routines.minimal_routine"
+ routine = Routine(full_routine_id)
+
+ actual_routine = client.create_routine(routine, exists_ok=True)
+
+ self.assertEqual(actual_routine.project, "test-routine-project")
+ self.assertEqual(actual_routine.dataset_id, "test_routines")
+ self.assertEqual(actual_routine.routine_id, "minimal_routine")
+ conn.api_request.assert_has_calls(
+ [
+ mock.call(
+ method="POST",
+ path="/projects/test-routine-project/datasets/test_routines/routines",
+ data=resource,
+ ),
+ mock.call(
+ method="GET",
+ path="/projects/test-routine-project/datasets/test_routines/routines/minimal_routine",
+ ),
+ ]
+ )
+
+ def test_create_table_w_day_partition(self):
+ from google.cloud.bigquery.table import Table
+ from google.cloud.bigquery.table import TimePartitioning
+
+ path = "projects/%s/datasets/%s/tables" % (self.PROJECT, self.DS_ID)
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ resource = self._make_table_resource()
+ conn = client._connection = make_connection(resource)
+ table = Table(self.TABLE_REF)
+ table.time_partitioning = TimePartitioning()
+
+ got = client.create_table(table)
+
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path="/%s" % path,
+ data={
+ "tableReference": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.TABLE_ID,
+ },
+ "timePartitioning": {"type": "DAY"},
+ "labels": {},
+ },
+ )
+ self.assertEqual(table.time_partitioning.type_, "DAY")
+ self.assertEqual(got.table_id, self.TABLE_ID)
+
+ def test_create_table_w_custom_property(self):
+ # The library should handle sending properties to the API that are not
+ # yet part of the library
+ from google.cloud.bigquery.table import Table
+
+ path = "projects/%s/datasets/%s/tables" % (self.PROJECT, self.DS_ID)
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ resource = self._make_table_resource()
+ resource["newAlphaProperty"] = "unreleased property"
+ conn = client._connection = make_connection(resource)
+ table = Table(self.TABLE_REF)
+ table._properties["newAlphaProperty"] = "unreleased property"
+
+ got = client.create_table(table)
+
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path="/%s" % path,
+ data={
+ "tableReference": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.TABLE_ID,
+ },
+ "newAlphaProperty": "unreleased property",
+ "labels": {},
+ },
+ )
+ self.assertEqual(got._properties["newAlphaProperty"], "unreleased property")
+ self.assertEqual(got.table_id, self.TABLE_ID)
+
+ def test_create_table_w_encryption_configuration(self):
+ from google.cloud.bigquery.table import EncryptionConfiguration
+ from google.cloud.bigquery.table import Table
+
+ path = "projects/%s/datasets/%s/tables" % (self.PROJECT, self.DS_ID)
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ resource = self._make_table_resource()
+ conn = client._connection = make_connection(resource)
+ table = Table(self.TABLE_REF)
+ table.encryption_configuration = EncryptionConfiguration(
+ kms_key_name=self.KMS_KEY_NAME
+ )
+
+ got = client.create_table(table)
+
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path="/%s" % path,
+ data={
+ "tableReference": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.TABLE_ID,
+ },
+ "labels": {},
+ "encryptionConfiguration": {"kmsKeyName": self.KMS_KEY_NAME},
+ },
+ )
+ self.assertEqual(got.table_id, self.TABLE_ID)
+
+ def test_create_table_w_day_partition_and_expire(self):
+ from google.cloud.bigquery.table import Table
+ from google.cloud.bigquery.table import TimePartitioning
+
+ path = "projects/%s/datasets/%s/tables" % (self.PROJECT, self.DS_ID)
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ resource = self._make_table_resource()
+ conn = client._connection = make_connection(resource)
+ table = Table(self.TABLE_REF)
+ table.time_partitioning = TimePartitioning(expiration_ms=100)
+
+ got = client.create_table(table)
+
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path="/%s" % path,
+ data={
+ "tableReference": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.TABLE_ID,
+ },
+ "timePartitioning": {"type": "DAY", "expirationMs": "100"},
+ "labels": {},
+ },
+ )
+ self.assertEqual(table.time_partitioning.type_, "DAY")
+ self.assertEqual(table.time_partitioning.expiration_ms, 100)
+ self.assertEqual(got.table_id, self.TABLE_ID)
+
+ def test_create_table_w_schema_and_query(self):
+ from google.cloud.bigquery.table import Table, SchemaField
+
+ path = "projects/%s/datasets/%s/tables" % (self.PROJECT, self.DS_ID)
+ query = "SELECT * from %s:%s" % (self.DS_ID, self.TABLE_ID)
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ resource = self._make_table_resource()
+ resource.update(
+ {
+ "schema": {
+ "fields": [
+ {
+ "name": "full_name",
+ "type": "STRING",
+ "mode": "REQUIRED",
+ "description": None,
+ },
+ {
+ "name": "age",
+ "type": "INTEGER",
+ "mode": "REQUIRED",
+ "description": None,
+ },
+ ]
+ },
+ "view": {"query": query},
+ }
+ )
+ schema = [
+ SchemaField("full_name", "STRING", mode="REQUIRED"),
+ SchemaField("age", "INTEGER", mode="REQUIRED"),
+ ]
+ conn = client._connection = make_connection(resource)
+ table = Table(self.TABLE_REF, schema=schema)
+ table.view_query = query
+
+ got = client.create_table(table)
+
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path="/%s" % path,
+ data={
+ "tableReference": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.TABLE_ID,
+ },
+ "schema": {
+ "fields": [
+ {
+ "name": "full_name",
+ "type": "STRING",
+ "mode": "REQUIRED",
+ "description": None,
+ },
+ {
+ "name": "age",
+ "type": "INTEGER",
+ "mode": "REQUIRED",
+ "description": None,
+ },
+ ]
+ },
+ "view": {"query": query, "useLegacySql": False},
+ "labels": {},
+ },
+ )
+ self.assertEqual(got.table_id, self.TABLE_ID)
+ self.assertEqual(got.project, self.PROJECT)
+ self.assertEqual(got.dataset_id, self.DS_ID)
+ self.assertEqual(got.schema, schema)
+ self.assertEqual(got.view_query, query)
+
+ def test_create_table_w_external(self):
+ from google.cloud.bigquery.external_config import ExternalConfig
+ from google.cloud.bigquery.job import SourceFormat
+ from google.cloud.bigquery.table import Table
+
+ path = "projects/%s/datasets/%s/tables" % (self.PROJECT, self.DS_ID)
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ resource = self._make_table_resource()
+ resource.update(
+ {
+ "externalDataConfiguration": {
+ "sourceFormat": SourceFormat.CSV,
+ "autodetect": True,
+ }
+ }
+ )
+ conn = client._connection = make_connection(resource)
+ table = Table(self.TABLE_REF)
+ ec = ExternalConfig("CSV")
+ ec.autodetect = True
+ table.external_data_configuration = ec
+
+ got = client.create_table(table)
+
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path="/%s" % path,
+ data={
+ "tableReference": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.TABLE_ID,
+ },
+ "externalDataConfiguration": {
+ "sourceFormat": SourceFormat.CSV,
+ "autodetect": True,
+ },
+ "labels": {},
+ },
+ )
+ self.assertEqual(got.table_id, self.TABLE_ID)
+ self.assertEqual(got.project, self.PROJECT)
+ self.assertEqual(got.dataset_id, self.DS_ID)
+ self.assertEqual(
+ got.external_data_configuration.source_format, SourceFormat.CSV
+ )
+ self.assertEqual(got.external_data_configuration.autodetect, True)
+
+ def test_create_table_w_reference(self):
+ path = "projects/%s/datasets/%s/tables" % (self.PROJECT, self.DS_ID)
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ resource = self._make_table_resource()
+ conn = client._connection = make_connection(resource)
+
+ got = client.create_table(self.TABLE_REF)
+
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path="/%s" % path,
+ data={
+ "tableReference": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.TABLE_ID,
+ },
+ "labels": {},
+ },
+ )
+ self.assertEqual(got.table_id, self.TABLE_ID)
+
+ def test_create_table_w_fully_qualified_string(self):
+ path = "projects/%s/datasets/%s/tables" % (self.PROJECT, self.DS_ID)
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ resource = self._make_table_resource()
+ conn = client._connection = make_connection(resource)
+
+ got = client.create_table(
+ "{}.{}.{}".format(self.PROJECT, self.DS_ID, self.TABLE_ID)
+ )
+
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path="/%s" % path,
+ data={
+ "tableReference": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.TABLE_ID,
+ },
+ "labels": {},
+ },
+ )
+ self.assertEqual(got.table_id, self.TABLE_ID)
+
+ def test_create_table_w_string(self):
+ path = "projects/%s/datasets/%s/tables" % (self.PROJECT, self.DS_ID)
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ resource = self._make_table_resource()
+ conn = client._connection = make_connection(resource)
+
+ got = client.create_table("{}.{}".format(self.DS_ID, self.TABLE_ID))
+
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path="/%s" % path,
+ data={
+ "tableReference": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.TABLE_ID,
+ },
+ "labels": {},
+ },
+ )
+ self.assertEqual(got.table_id, self.TABLE_ID)
+
+ def test_create_table_alreadyexists_w_exists_ok_false(self):
+ post_path = "/projects/{}/datasets/{}/tables".format(self.PROJECT, self.DS_ID)
+ creds = _make_credentials()
+ client = self._make_one(
+ project=self.PROJECT, credentials=creds, location=self.LOCATION
+ )
+ conn = client._connection = make_connection(
+ google.api_core.exceptions.AlreadyExists("table already exists")
+ )
+
+ with pytest.raises(google.api_core.exceptions.AlreadyExists):
+ client.create_table("{}.{}".format(self.DS_ID, self.TABLE_ID))
+
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path=post_path,
+ data={
+ "tableReference": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.TABLE_ID,
+ },
+ "labels": {},
+ },
+ )
+
+ def test_create_table_alreadyexists_w_exists_ok_true(self):
+ post_path = "/projects/{}/datasets/{}/tables".format(self.PROJECT, self.DS_ID)
+ get_path = "/projects/{}/datasets/{}/tables/{}".format(
+ self.PROJECT, self.DS_ID, self.TABLE_ID
+ )
+ resource = self._make_table_resource()
+ creds = _make_credentials()
+ client = self._make_one(
+ project=self.PROJECT, credentials=creds, location=self.LOCATION
+ )
+ conn = client._connection = make_connection(
+ google.api_core.exceptions.AlreadyExists("table already exists"), resource
+ )
+
+ got = client.create_table(
+ "{}.{}".format(self.DS_ID, self.TABLE_ID), exists_ok=True
+ )
+
+ self.assertEqual(got.project, self.PROJECT)
+ self.assertEqual(got.dataset_id, self.DS_ID)
+ self.assertEqual(got.table_id, self.TABLE_ID)
+
+ conn.api_request.assert_has_calls(
+ [
+ mock.call(
+ method="POST",
+ path=post_path,
+ data={
+ "tableReference": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.TABLE_ID,
+ },
+ "labels": {},
+ },
+ ),
+ mock.call(method="GET", path=get_path),
+ ]
+ )
+
+ def test_get_model(self):
+ path = "projects/%s/datasets/%s/models/%s" % (
+ self.PROJECT,
+ self.DS_ID,
+ self.MODEL_ID,
+ )
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ resource = {
+ "modelReference": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "modelId": self.MODEL_ID,
+ }
+ }
+ conn = client._connection = make_connection(resource)
+
+ model_ref = client.dataset(self.DS_ID).model(self.MODEL_ID)
+ got = client.get_model(model_ref)
+
+ conn.api_request.assert_called_once_with(method="GET", path="/%s" % path)
+ self.assertEqual(got.model_id, self.MODEL_ID)
+
+ def test_get_model_w_string(self):
+ path = "projects/%s/datasets/%s/models/%s" % (
+ self.PROJECT,
+ self.DS_ID,
+ self.MODEL_ID,
+ )
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ resource = {
+ "modelReference": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "modelId": self.MODEL_ID,
+ }
+ }
+ conn = client._connection = make_connection(resource)
+
+ model_id = "{}.{}.{}".format(self.PROJECT, self.DS_ID, self.MODEL_ID)
+ got = client.get_model(model_id)
+
+ conn.api_request.assert_called_once_with(method="GET", path="/%s" % path)
+ self.assertEqual(got.model_id, self.MODEL_ID)
+
+ def test_get_routine(self):
+ from google.cloud.bigquery.routine import Routine
+ from google.cloud.bigquery.routine import RoutineReference
+
+ full_routine_id = "test-routine-project.test_routines.minimal_routine"
+ routines = [
+ full_routine_id,
+ Routine(full_routine_id),
+ RoutineReference.from_string(full_routine_id),
+ ]
+ for routine in routines:
+ creds = _make_credentials()
+ resource = {
+ "etag": "im-an-etag",
+ "routineReference": {
+ "projectId": "test-routine-project",
+ "datasetId": "test_routines",
+ "routineId": "minimal_routine",
+ },
+ "routineType": "SCALAR_FUNCTION",
+ }
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ conn = client._connection = make_connection(resource)
+
+ actual_routine = client.get_routine(routine)
+
+ conn.api_request.assert_called_once_with(
+ method="GET",
+ path="/projects/test-routine-project/datasets/test_routines/routines/minimal_routine",
+ )
+ self.assertEqual(
+ actual_routine.reference,
+ RoutineReference.from_string(full_routine_id),
+ msg="routine={}".format(repr(routine)),
+ )
+ self.assertEqual(
+ actual_routine.etag,
+ "im-an-etag",
+ msg="routine={}".format(repr(routine)),
+ )
+ self.assertEqual(
+ actual_routine.type_,
+ "SCALAR_FUNCTION",
+ msg="routine={}".format(repr(routine)),
+ )
+
+ def test_get_table(self):
+ path = "projects/%s/datasets/%s/tables/%s" % (
+ self.PROJECT,
+ self.DS_ID,
+ self.TABLE_ID,
+ )
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ resource = self._make_table_resource()
+ conn = client._connection = make_connection(resource)
+ table = client.get_table(self.TABLE_REF)
+
+ conn.api_request.assert_called_once_with(method="GET", path="/%s" % path)
+ self.assertEqual(table.table_id, self.TABLE_ID)
+
+ def test_get_table_sets_user_agent(self):
+ creds = _make_credentials()
+ http = mock.create_autospec(requests.Session)
+ mock_response = http.request(
+ url=mock.ANY, method=mock.ANY, headers=mock.ANY, data=mock.ANY
+ )
+ http.reset_mock()
+ mock_response.status_code = 200
+ mock_response.json.return_value = self._make_table_resource()
+ user_agent_override = client_info.ClientInfo(user_agent="my-application/1.2.3")
+ client = self._make_one(
+ project=self.PROJECT,
+ credentials=creds,
+ client_info=user_agent_override,
+ _http=http,
+ )
+
+ client.get_table(self.TABLE_REF)
+
+ expected_user_agent = user_agent_override.to_user_agent()
+ http.request.assert_called_once_with(
+ url=mock.ANY,
+ method="GET",
+ headers={
+ "X-Goog-API-Client": expected_user_agent,
+ "Accept-Encoding": "gzip",
+ "User-Agent": expected_user_agent,
+ },
+ data=mock.ANY,
+ )
+ self.assertIn("my-application/1.2.3", expected_user_agent)
+
+ def test_update_dataset_w_invalid_field(self):
+ from google.cloud.bigquery.dataset import Dataset
+
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ with self.assertRaises(ValueError):
+ client.update_dataset(Dataset(client.dataset(self.DS_ID)), ["foo"])
+
+ def test_update_dataset(self):
+ from google.cloud.bigquery.dataset import Dataset, AccessEntry
+
+ PATH = "projects/%s/datasets/%s" % (self.PROJECT, self.DS_ID)
+ DESCRIPTION = "DESCRIPTION"
+ FRIENDLY_NAME = "TITLE"
+ LOCATION = "loc"
+ LABELS = {"priority": "high"}
+ ACCESS = [{"role": "OWNER", "userByEmail": "phred@example.com"}]
+ EXP = 17
+ RESOURCE = {
+ "datasetReference": {"projectId": self.PROJECT, "datasetId": self.DS_ID},
+ "etag": "etag",
+ "description": DESCRIPTION,
+ "friendlyName": FRIENDLY_NAME,
+ "location": LOCATION,
+ "defaultTableExpirationMs": EXP,
+ "labels": LABELS,
+ "access": ACCESS,
+ }
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ conn = client._connection = make_connection(RESOURCE, RESOURCE)
+ ds = Dataset(client.dataset(self.DS_ID))
+ ds.description = DESCRIPTION
+ ds.friendly_name = FRIENDLY_NAME
+ ds.location = LOCATION
+ ds.default_table_expiration_ms = EXP
+ ds.labels = LABELS
+ ds.access_entries = [AccessEntry("OWNER", "userByEmail", "phred@example.com")]
+ ds2 = client.update_dataset(
+ ds, ["description", "friendly_name", "location", "labels", "access_entries"]
+ )
+ conn.api_request.assert_called_once_with(
+ method="PATCH",
+ data={
+ "description": DESCRIPTION,
+ "friendlyName": FRIENDLY_NAME,
+ "location": LOCATION,
+ "labels": LABELS,
+ "access": ACCESS,
+ },
+ path="/" + PATH,
+ headers=None,
+ )
+ self.assertEqual(ds2.description, ds.description)
+ self.assertEqual(ds2.friendly_name, ds.friendly_name)
+ self.assertEqual(ds2.location, ds.location)
+ self.assertEqual(ds2.labels, ds.labels)
+ self.assertEqual(ds2.access_entries, ds.access_entries)
+
+ # ETag becomes If-Match header.
+ ds._properties["etag"] = "etag"
+ client.update_dataset(ds, [])
+ req = conn.api_request.call_args
+ self.assertEqual(req[1]["headers"]["If-Match"], "etag")
+
+ def test_update_dataset_w_custom_property(self):
+ # The library should handle sending properties to the API that are not
+ # yet part of the library
+ from google.cloud.bigquery.dataset import Dataset
+
+ path = "/projects/%s/datasets/%s" % (self.PROJECT, self.DS_ID)
+ resource = {
+ "datasetReference": {"projectId": self.PROJECT, "datasetId": self.DS_ID},
+ "newAlphaProperty": "unreleased property",
+ }
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ conn = client._connection = make_connection(resource)
+ dataset = Dataset(client.dataset(self.DS_ID))
+ dataset._properties["newAlphaProperty"] = "unreleased property"
+
+ dataset = client.update_dataset(dataset, ["newAlphaProperty"])
+ conn.api_request.assert_called_once_with(
+ method="PATCH",
+ data={"newAlphaProperty": "unreleased property"},
+ path=path,
+ headers=None,
+ )
+
+ self.assertEqual(dataset.dataset_id, self.DS_ID)
+ self.assertEqual(dataset.project, self.PROJECT)
+ self.assertEqual(dataset._properties["newAlphaProperty"], "unreleased property")
+
+ def test_update_model(self):
+ from google.cloud.bigquery.model import Model
+
+ path = "projects/%s/datasets/%s/models/%s" % (
+ self.PROJECT,
+ self.DS_ID,
+ self.MODEL_ID,
+ )
+ description = "description"
+ title = "title"
+ expires = datetime.datetime(
+ 2012, 12, 21, 16, 0, 0, tzinfo=google.cloud._helpers.UTC
+ )
+ resource = {
+ "modelReference": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "modelId": self.MODEL_ID,
+ },
+ "description": description,
+ "etag": "etag",
+ "expirationTime": str(google.cloud._helpers._millis(expires)),
+ "friendlyName": title,
+ "labels": {"x": "y"},
+ }
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ conn = client._connection = make_connection(resource, resource)
+ model_id = "{}.{}.{}".format(self.PROJECT, self.DS_ID, self.MODEL_ID)
+ model = Model(model_id)
+ model.description = description
+ model.friendly_name = title
+ model.expires = expires
+ model.labels = {"x": "y"}
+
+ updated_model = client.update_model(
+ model, ["description", "friendly_name", "labels", "expires"]
+ )
+
+ sent = {
+ "description": description,
+ "expirationTime": str(google.cloud._helpers._millis(expires)),
+ "friendlyName": title,
+ "labels": {"x": "y"},
+ }
+ conn.api_request.assert_called_once_with(
+ method="PATCH", data=sent, path="/" + path, headers=None
+ )
+ self.assertEqual(updated_model.model_id, model.model_id)
+ self.assertEqual(updated_model.description, model.description)
+ self.assertEqual(updated_model.friendly_name, model.friendly_name)
+ self.assertEqual(updated_model.labels, model.labels)
+ self.assertEqual(updated_model.expires, model.expires)
+
+ # ETag becomes If-Match header.
+ model._proto.etag = "etag"
+ client.update_model(model, [])
+ req = conn.api_request.call_args
+ self.assertEqual(req[1]["headers"]["If-Match"], "etag")
+
+ def test_update_routine(self):
+ from google.cloud.bigquery.routine import Routine
+ from google.cloud.bigquery.routine import RoutineArgument
+
+ full_routine_id = "routines-project.test_routines.updated_routine"
+ resource = {
+ "routineReference": {
+ "projectId": "routines-project",
+ "datasetId": "test_routines",
+ "routineId": "updated_routine",
+ },
+ "routineType": "SCALAR_FUNCTION",
+ "language": "SQL",
+ "definitionBody": "x * 3",
+ "arguments": [{"name": "x", "dataType": {"typeKind": "INT64"}}],
+ "returnType": None,
+ "someNewField": "someValue",
+ }
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ conn = client._connection = make_connection(resource, resource)
+ routine = Routine(full_routine_id)
+ routine.arguments = [
+ RoutineArgument(
+ name="x",
+ data_type=bigquery_v2.types.StandardSqlDataType(
+ type_kind=bigquery_v2.enums.StandardSqlDataType.TypeKind.INT64
+ ),
+ )
+ ]
+ routine.body = "x * 3"
+ routine.language = "SQL"
+ routine.type_ = "SCALAR_FUNCTION"
+ routine._properties["someNewField"] = "someValue"
+
+ actual_routine = client.update_routine(
+ routine,
+ ["arguments", "language", "body", "type_", "return_type", "someNewField"],
+ )
+
+ # TODO: routineReference isn't needed when the Routines API supports
+ # partial updates.
+ sent = resource
+ conn.api_request.assert_called_once_with(
+ method="PUT",
+ data=sent,
+ path="/projects/routines-project/datasets/test_routines/routines/updated_routine",
+ headers=None,
+ )
+ self.assertEqual(actual_routine.arguments, routine.arguments)
+ self.assertEqual(actual_routine.body, routine.body)
+ self.assertEqual(actual_routine.language, routine.language)
+ self.assertEqual(actual_routine.type_, routine.type_)
+
+ # ETag becomes If-Match header.
+ routine._properties["etag"] = "im-an-etag"
+ client.update_routine(routine, [])
+ req = conn.api_request.call_args
+ self.assertEqual(req[1]["headers"]["If-Match"], "im-an-etag")
+
+ def test_update_table(self):
+ from google.cloud.bigquery.table import Table, SchemaField
+
+ path = "projects/%s/datasets/%s/tables/%s" % (
+ self.PROJECT,
+ self.DS_ID,
+ self.TABLE_ID,
+ )
+ description = "description"
+ title = "title"
+ resource = self._make_table_resource()
+ resource.update(
+ {
+ "schema": {
+ "fields": [
+ {
+ "name": "full_name",
+ "type": "STRING",
+ "mode": "REQUIRED",
+ "description": None,
+ },
+ {
+ "name": "age",
+ "type": "INTEGER",
+ "mode": "REQUIRED",
+ "description": None,
+ },
+ ]
+ },
+ "etag": "etag",
+ "description": description,
+ "friendlyName": title,
+ "labels": {"x": "y"},
+ }
+ )
+ schema = [
+ SchemaField("full_name", "STRING", mode="REQUIRED"),
+ SchemaField("age", "INTEGER", mode="REQUIRED"),
+ ]
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ conn = client._connection = make_connection(resource, resource)
+ table = Table(self.TABLE_REF, schema=schema)
+ table.description = description
+ table.friendly_name = title
+ table.labels = {"x": "y"}
+
+ updated_table = client.update_table(
+ table, ["schema", "description", "friendly_name", "labels"]
+ )
+
+ sent = {
+ "schema": {
+ "fields": [
+ {
+ "name": "full_name",
+ "type": "STRING",
+ "mode": "REQUIRED",
+ "description": None,
+ },
+ {
+ "name": "age",
+ "type": "INTEGER",
+ "mode": "REQUIRED",
+ "description": None,
+ },
+ ]
+ },
+ "description": description,
+ "friendlyName": title,
+ "labels": {"x": "y"},
+ }
+ conn.api_request.assert_called_once_with(
+ method="PATCH", data=sent, path="/" + path, headers=None
+ )
+ self.assertEqual(updated_table.description, table.description)
+ self.assertEqual(updated_table.friendly_name, table.friendly_name)
+ self.assertEqual(updated_table.schema, table.schema)
+ self.assertEqual(updated_table.labels, table.labels)
+
+ # ETag becomes If-Match header.
+ table._properties["etag"] = "etag"
+ client.update_table(table, [])
+ req = conn.api_request.call_args
+ self.assertEqual(req[1]["headers"]["If-Match"], "etag")
+
+ def test_update_table_w_custom_property(self):
+ from google.cloud.bigquery.table import Table
+
+ path = "projects/%s/datasets/%s/tables/%s" % (
+ self.PROJECT,
+ self.DS_ID,
+ self.TABLE_ID,
+ )
+ resource = self._make_table_resource()
+ resource["newAlphaProperty"] = "unreleased property"
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ conn = client._connection = make_connection(resource)
+ table = Table(self.TABLE_REF)
+ table._properties["newAlphaProperty"] = "unreleased property"
+
+ updated_table = client.update_table(table, ["newAlphaProperty"])
+
+ conn.api_request.assert_called_once_with(
+ method="PATCH",
+ path="/%s" % path,
+ data={"newAlphaProperty": "unreleased property"},
+ headers=None,
+ )
+ self.assertEqual(
+ updated_table._properties["newAlphaProperty"], "unreleased property"
+ )
+
+ def test_update_table_only_use_legacy_sql(self):
+ from google.cloud.bigquery.table import Table
+
+ path = "projects/%s/datasets/%s/tables/%s" % (
+ self.PROJECT,
+ self.DS_ID,
+ self.TABLE_ID,
+ )
+ resource = self._make_table_resource()
+ resource["view"] = {"useLegacySql": True}
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ conn = client._connection = make_connection(resource)
+ table = Table(self.TABLE_REF)
+ table.view_use_legacy_sql = True
+
+ updated_table = client.update_table(table, ["view_use_legacy_sql"])
+
+ conn.api_request.assert_called_once_with(
+ method="PATCH",
+ path="/%s" % path,
+ data={"view": {"useLegacySql": True}},
+ headers=None,
+ )
+ self.assertEqual(updated_table.view_use_legacy_sql, table.view_use_legacy_sql)
+
+ def test_update_table_w_query(self):
+ import datetime
+ from google.cloud._helpers import UTC
+ from google.cloud._helpers import _millis
+ from google.cloud.bigquery.table import Table, SchemaField
+
+ path = "projects/%s/datasets/%s/tables/%s" % (
+ self.PROJECT,
+ self.DS_ID,
+ self.TABLE_ID,
+ )
+ query = "select fullname, age from person_ages"
+ location = "EU"
+ exp_time = datetime.datetime(2015, 8, 1, 23, 59, 59, tzinfo=UTC)
+ schema_resource = {
+ "fields": [
+ {
+ "name": "full_name",
+ "type": "STRING",
+ "mode": "REQUIRED",
+ "description": None,
+ },
+ {
+ "name": "age",
+ "type": "INTEGER",
+ "mode": "REQUIRED",
+ "description": None,
+ },
+ ]
+ }
+ schema = [
+ SchemaField("full_name", "STRING", mode="REQUIRED"),
+ SchemaField("age", "INTEGER", mode="REQUIRED"),
+ ]
+ resource = self._make_table_resource()
+ resource.update(
+ {
+ "schema": schema_resource,
+ "view": {"query": query, "useLegacySql": True},
+ "location": location,
+ "expirationTime": _millis(exp_time),
+ }
+ )
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ conn = client._connection = make_connection(resource)
+ table = Table(self.TABLE_REF, schema=schema)
+ table.expires = exp_time
+ table.view_query = query
+ table.view_use_legacy_sql = True
+ updated_properties = ["schema", "view_query", "expires", "view_use_legacy_sql"]
+
+ updated_table = client.update_table(table, updated_properties)
+
+ self.assertEqual(updated_table.schema, table.schema)
+ self.assertEqual(updated_table.view_query, table.view_query)
+ self.assertEqual(updated_table.expires, table.expires)
+ self.assertEqual(updated_table.view_use_legacy_sql, table.view_use_legacy_sql)
+ self.assertEqual(updated_table.location, location)
+
+ conn.api_request.assert_called_once_with(
+ method="PATCH",
+ path="/%s" % path,
+ data={
+ "view": {"query": query, "useLegacySql": True},
+ "expirationTime": str(_millis(exp_time)),
+ "schema": schema_resource,
+ },
+ headers=None,
+ )
+
+ def test_update_table_w_schema_None(self):
+ # Simulate deleting schema: not sure if back-end will actually
+ # allow this operation, but the spec says it is optional.
+ path = "projects/%s/datasets/%s/tables/%s" % (
+ self.PROJECT,
+ self.DS_ID,
+ self.TABLE_ID,
+ )
+ resource1 = self._make_table_resource()
+ resource1.update(
+ {
+ "schema": {
+ "fields": [
+ {"name": "full_name", "type": "STRING", "mode": "REQUIRED"},
+ {"name": "age", "type": "INTEGER", "mode": "REQUIRED"},
+ ]
+ }
+ }
+ )
+ resource2 = self._make_table_resource()
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ conn = client._connection = make_connection(resource1, resource2)
+ table = client.get_table(
+ # Test with string for table ID
+ "{}.{}.{}".format(
+ self.TABLE_REF.project,
+ self.TABLE_REF.dataset_id,
+ self.TABLE_REF.table_id,
+ )
+ )
+ table.schema = None
+
+ updated_table = client.update_table(table, ["schema"])
+
+ self.assertEqual(len(conn.api_request.call_args_list), 2)
+ req = conn.api_request.call_args_list[1]
+ self.assertEqual(req[1]["method"], "PATCH")
+ sent = {"schema": None}
+ self.assertEqual(req[1]["data"], sent)
+ self.assertEqual(req[1]["path"], "/%s" % path)
+ self.assertEqual(len(updated_table.schema), 0)
+
+ def test_update_table_delete_property(self):
+ from google.cloud.bigquery.table import Table
+
+ description = "description"
+ title = "title"
+ path = "projects/%s/datasets/%s/tables/%s" % (
+ self.PROJECT,
+ self.DS_ID,
+ self.TABLE_ID,
+ )
+ resource1 = self._make_table_resource()
+ resource1.update({"description": description, "friendlyName": title})
+ resource2 = self._make_table_resource()
+ resource2["description"] = None
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ conn = client._connection = make_connection(resource1, resource2)
+ table = Table(self.TABLE_REF)
+ table.description = description
+ table.friendly_name = title
+ table2 = client.update_table(table, ["description", "friendly_name"])
+ self.assertEqual(table2.description, table.description)
+ table2.description = None
+
+ table3 = client.update_table(table2, ["description"])
+ self.assertEqual(len(conn.api_request.call_args_list), 2)
+ req = conn.api_request.call_args_list[1]
+ self.assertEqual(req[1]["method"], "PATCH")
+ self.assertEqual(req[1]["path"], "/%s" % path)
+ sent = {"description": None}
+ self.assertEqual(req[1]["data"], sent)
+ self.assertIsNone(table3.description)
+
+ def test_list_tables_empty(self):
+ path = "/projects/{}/datasets/{}/tables".format(self.PROJECT, self.DS_ID)
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ conn = client._connection = make_connection({})
+
+ dataset = client.dataset(self.DS_ID)
+ iterator = client.list_tables(dataset)
+ self.assertIs(iterator.dataset, dataset)
+ page = six.next(iterator.pages)
+ tables = list(page)
+ token = iterator.next_page_token
+
+ self.assertEqual(tables, [])
+ self.assertIsNone(token)
+ conn.api_request.assert_called_once_with(
+ method="GET", path=path, query_params={}
+ )
+
+ def test_list_models_empty(self):
+ path = "/projects/{}/datasets/{}/models".format(self.PROJECT, self.DS_ID)
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ conn = client._connection = make_connection({})
+
+ dataset_id = "{}.{}".format(self.PROJECT, self.DS_ID)
+ iterator = client.list_models(dataset_id)
+ page = six.next(iterator.pages)
+ models = list(page)
+ token = iterator.next_page_token
+
+ self.assertEqual(models, [])
+ self.assertIsNone(token)
+ conn.api_request.assert_called_once_with(
+ method="GET", path=path, query_params={}
+ )
+
+ def test_list_models_defaults(self):
+ from google.cloud.bigquery.model import Model
+
+ MODEL_1 = "model_one"
+ MODEL_2 = "model_two"
+ PATH = "projects/%s/datasets/%s/models" % (self.PROJECT, self.DS_ID)
+ TOKEN = "TOKEN"
+ DATA = {
+ "nextPageToken": TOKEN,
+ "models": [
+ {
+ "modelReference": {
+ "modelId": MODEL_1,
+ "datasetId": self.DS_ID,
+ "projectId": self.PROJECT,
+ }
+ },
+ {
+ "modelReference": {
+ "modelId": MODEL_2,
+ "datasetId": self.DS_ID,
+ "projectId": self.PROJECT,
+ }
+ },
+ ],
+ }
+
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ conn = client._connection = make_connection(DATA)
+ dataset = client.dataset(self.DS_ID)
+
+ iterator = client.list_models(dataset)
+ self.assertIs(iterator.dataset, dataset)
+ page = six.next(iterator.pages)
+ models = list(page)
+ token = iterator.next_page_token
+
+ self.assertEqual(len(models), len(DATA["models"]))
+ for found, expected in zip(models, DATA["models"]):
+ self.assertIsInstance(found, Model)
+ self.assertEqual(found.model_id, expected["modelReference"]["modelId"])
+ self.assertEqual(token, TOKEN)
+
+ conn.api_request.assert_called_once_with(
+ method="GET", path="/%s" % PATH, query_params={}
+ )
+
+ def test_list_models_wrong_type(self):
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ with self.assertRaises(TypeError):
+ client.list_models(client.dataset(self.DS_ID).model("foo"))
+
+ def test_list_routines_empty(self):
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ conn = client._connection = make_connection({})
+
+ iterator = client.list_routines("test-routines.test_routines")
+ page = six.next(iterator.pages)
+ routines = list(page)
+ token = iterator.next_page_token
+
+ self.assertEqual(routines, [])
+ self.assertIsNone(token)
+ conn.api_request.assert_called_once_with(
+ method="GET",
+ path="/projects/test-routines/datasets/test_routines/routines",
+ query_params={},
+ )
+
+ def test_list_routines_defaults(self):
+ from google.cloud.bigquery.routine import Routine
+
+ project_id = "test-routines"
+ dataset_id = "test_routines"
+ path = "/projects/test-routines/datasets/test_routines/routines"
+ routine_1 = "routine_one"
+ routine_2 = "routine_two"
+ token = "TOKEN"
+ resource = {
+ "nextPageToken": token,
+ "routines": [
+ {
+ "routineReference": {
+ "routineId": routine_1,
+ "datasetId": dataset_id,
+ "projectId": project_id,
+ }
+ },
+ {
+ "routineReference": {
+ "routineId": routine_2,
+ "datasetId": dataset_id,
+ "projectId": project_id,
+ }
+ },
+ ],
+ }
+
+ creds = _make_credentials()
+ client = self._make_one(project=project_id, credentials=creds)
+ conn = client._connection = make_connection(resource)
+ dataset = client.dataset(dataset_id)
+
+ iterator = client.list_routines(dataset)
+ self.assertIs(iterator.dataset, dataset)
+ page = six.next(iterator.pages)
+ routines = list(page)
+ actual_token = iterator.next_page_token
+
+ self.assertEqual(len(routines), len(resource["routines"]))
+ for found, expected in zip(routines, resource["routines"]):
+ self.assertIsInstance(found, Routine)
+ self.assertEqual(
+ found.routine_id, expected["routineReference"]["routineId"]
+ )
+ self.assertEqual(actual_token, token)
+
+ conn.api_request.assert_called_once_with(
+ method="GET", path=path, query_params={}
+ )
+
+ def test_list_routines_wrong_type(self):
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ with self.assertRaises(TypeError):
+ client.list_routines(client.dataset(self.DS_ID).table("foo"))
+
+ def test_list_tables_defaults(self):
+ from google.cloud.bigquery.table import TableListItem
+
+ TABLE_1 = "table_one"
+ TABLE_2 = "table_two"
+ PATH = "projects/%s/datasets/%s/tables" % (self.PROJECT, self.DS_ID)
+ TOKEN = "TOKEN"
+ DATA = {
+ "nextPageToken": TOKEN,
+ "tables": [
+ {
+ "kind": "bigquery#table",
+ "id": "%s:%s.%s" % (self.PROJECT, self.DS_ID, TABLE_1),
+ "tableReference": {
+ "tableId": TABLE_1,
+ "datasetId": self.DS_ID,
+ "projectId": self.PROJECT,
+ },
+ "type": "TABLE",
+ },
+ {
+ "kind": "bigquery#table",
+ "id": "%s:%s.%s" % (self.PROJECT, self.DS_ID, TABLE_2),
+ "tableReference": {
+ "tableId": TABLE_2,
+ "datasetId": self.DS_ID,
+ "projectId": self.PROJECT,
+ },
+ "type": "TABLE",
+ },
+ ],
+ }
+
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ conn = client._connection = make_connection(DATA)
+ dataset = client.dataset(self.DS_ID)
+
+ iterator = client.list_tables(dataset)
+ self.assertIs(iterator.dataset, dataset)
+ page = six.next(iterator.pages)
+ tables = list(page)
+ token = iterator.next_page_token
+
+ self.assertEqual(len(tables), len(DATA["tables"]))
+ for found, expected in zip(tables, DATA["tables"]):
+ self.assertIsInstance(found, TableListItem)
+ self.assertEqual(found.full_table_id, expected["id"])
+ self.assertEqual(found.table_type, expected["type"])
+ self.assertEqual(token, TOKEN)
+
+ conn.api_request.assert_called_once_with(
+ method="GET", path="/%s" % PATH, query_params={}
+ )
+
+ def test_list_tables_explicit(self):
+ from google.cloud.bigquery.table import TableListItem
+
+ TABLE_1 = "table_one"
+ TABLE_2 = "table_two"
+ PATH = "projects/%s/datasets/%s/tables" % (self.PROJECT, self.DS_ID)
+ TOKEN = "TOKEN"
+ DATA = {
+ "tables": [
+ {
+ "kind": "bigquery#dataset",
+ "id": "%s:%s.%s" % (self.PROJECT, self.DS_ID, TABLE_1),
+ "tableReference": {
+ "tableId": TABLE_1,
+ "datasetId": self.DS_ID,
+ "projectId": self.PROJECT,
+ },
+ "type": "TABLE",
+ },
+ {
+ "kind": "bigquery#dataset",
+ "id": "%s:%s.%s" % (self.PROJECT, self.DS_ID, TABLE_2),
+ "tableReference": {
+ "tableId": TABLE_2,
+ "datasetId": self.DS_ID,
+ "projectId": self.PROJECT,
+ },
+ "type": "TABLE",
+ },
+ ]
+ }
+
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ conn = client._connection = make_connection(DATA)
+ dataset = client.dataset(self.DS_ID)
+
+ iterator = client.list_tables(
+ # Test with string for dataset ID.
+ self.DS_ID,
+ max_results=3,
+ page_token=TOKEN,
+ )
+ self.assertEqual(iterator.dataset, dataset)
+ page = six.next(iterator.pages)
+ tables = list(page)
+ token = iterator.next_page_token
+
+ self.assertEqual(len(tables), len(DATA["tables"]))
+ for found, expected in zip(tables, DATA["tables"]):
+ self.assertIsInstance(found, TableListItem)
+ self.assertEqual(found.full_table_id, expected["id"])
+ self.assertEqual(found.table_type, expected["type"])
+ self.assertIsNone(token)
+
+ conn.api_request.assert_called_once_with(
+ method="GET",
+ path="/%s" % PATH,
+ query_params={"maxResults": 3, "pageToken": TOKEN},
+ )
+
+ def test_list_tables_wrong_type(self):
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ with self.assertRaises(TypeError):
+ client.list_tables(client.dataset(self.DS_ID).table("foo"))
+
+ def test_delete_dataset(self):
+ from google.cloud.bigquery.dataset import Dataset
+ from google.cloud.bigquery.dataset import DatasetReference
+
+ ds_ref = DatasetReference(self.PROJECT, self.DS_ID)
+ datasets = (ds_ref, Dataset(ds_ref), "{}.{}".format(self.PROJECT, self.DS_ID))
+ PATH = "projects/%s/datasets/%s" % (self.PROJECT, self.DS_ID)
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ conn = client._connection = make_connection(*([{}] * len(datasets)))
+ for arg in datasets:
+ client.delete_dataset(arg)
+ conn.api_request.assert_called_with(
+ method="DELETE", path="/%s" % PATH, query_params={}
+ )
+
+ def test_delete_dataset_delete_contents(self):
+ from google.cloud.bigquery.dataset import Dataset
+
+ PATH = "projects/%s/datasets/%s" % (self.PROJECT, self.DS_ID)
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ conn = client._connection = make_connection({}, {})
+ ds_ref = client.dataset(self.DS_ID)
+ for arg in (ds_ref, Dataset(ds_ref)):
+ client.delete_dataset(arg, delete_contents=True)
+ conn.api_request.assert_called_with(
+ method="DELETE",
+ path="/%s" % PATH,
+ query_params={"deleteContents": "true"},
+ )
+
+ def test_delete_dataset_wrong_type(self):
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ with self.assertRaises(TypeError):
+ client.delete_dataset(client.dataset(self.DS_ID).table("foo"))
+
+ def test_delete_dataset_w_not_found_ok_false(self):
+ path = "/projects/{}/datasets/{}".format(self.PROJECT, self.DS_ID)
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection(
+ google.api_core.exceptions.NotFound("dataset not found")
+ )
+
+ with self.assertRaises(google.api_core.exceptions.NotFound):
+ client.delete_dataset(self.DS_ID)
+
+ conn.api_request.assert_called_with(method="DELETE", path=path, query_params={})
+
+ def test_delete_dataset_w_not_found_ok_true(self):
+ path = "/projects/{}/datasets/{}".format(self.PROJECT, self.DS_ID)
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection(
+ google.api_core.exceptions.NotFound("dataset not found")
+ )
+
+ client.delete_dataset(self.DS_ID, not_found_ok=True)
+
+ conn.api_request.assert_called_with(method="DELETE", path=path, query_params={})
+
+ def test_delete_model(self):
+ from google.cloud.bigquery.model import Model
+
+ path = "projects/%s/datasets/%s/models/%s" % (
+ self.PROJECT,
+ self.DS_ID,
+ self.MODEL_ID,
+ )
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ model_id = "{}.{}.{}".format(self.PROJECT, self.DS_ID, self.MODEL_ID)
+ models = (
+ model_id,
+ client.dataset(self.DS_ID).model(self.MODEL_ID),
+ Model(model_id),
+ )
+ conn = client._connection = make_connection(*([{}] * len(models)))
+
+ for arg in models:
+ client.delete_model(arg)
+ conn.api_request.assert_called_with(method="DELETE", path="/%s" % path)
+
+ def test_delete_model_w_wrong_type(self):
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ with self.assertRaises(TypeError):
+ client.delete_model(client.dataset(self.DS_ID))
+
+ def test_delete_model_w_not_found_ok_false(self):
+ path = "/projects/{}/datasets/{}/models/{}".format(
+ self.PROJECT, self.DS_ID, self.MODEL_ID
+ )
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection(
+ google.api_core.exceptions.NotFound("model not found")
+ )
+
+ with self.assertRaises(google.api_core.exceptions.NotFound):
+ client.delete_model("{}.{}".format(self.DS_ID, self.MODEL_ID))
+
+ conn.api_request.assert_called_with(method="DELETE", path=path)
+
+ def test_delete_model_w_not_found_ok_true(self):
+ path = "/projects/{}/datasets/{}/models/{}".format(
+ self.PROJECT, self.DS_ID, self.MODEL_ID
+ )
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection(
+ google.api_core.exceptions.NotFound("model not found")
+ )
+
+ client.delete_model(
+ "{}.{}".format(self.DS_ID, self.MODEL_ID), not_found_ok=True
+ )
+
+ conn.api_request.assert_called_with(method="DELETE", path=path)
+
+ def test_delete_routine(self):
+ from google.cloud.bigquery.routine import Routine
+ from google.cloud.bigquery.routine import RoutineReference
+
+ full_routine_id = "test-routine-project.test_routines.minimal_routine"
+ routines = [
+ full_routine_id,
+ Routine(full_routine_id),
+ RoutineReference.from_string(full_routine_id),
+ ]
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection(*([{}] * len(routines)))
+
+ for routine in routines:
+ client.delete_routine(routine)
+ conn.api_request.assert_called_with(
+ method="DELETE",
+ path="/projects/test-routine-project/datasets/test_routines/routines/minimal_routine",
+ )
+
+ def test_delete_routine_w_wrong_type(self):
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ with self.assertRaises(TypeError):
+ client.delete_routine(client.dataset(self.DS_ID))
+
+ def test_delete_routine_w_not_found_ok_false(self):
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection(
+ google.api_core.exceptions.NotFound("routine not found")
+ )
+
+ with self.assertRaises(google.api_core.exceptions.NotFound):
+ client.delete_routine("routines-project.test_routines.test_routine")
+
+ conn.api_request.assert_called_with(
+ method="DELETE",
+ path="/projects/routines-project/datasets/test_routines/routines/test_routine",
+ )
+
+ def test_delete_routine_w_not_found_ok_true(self):
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection(
+ google.api_core.exceptions.NotFound("routine not found")
+ )
+
+ client.delete_routine(
+ "routines-project.test_routines.test_routine", not_found_ok=True
+ )
+
+ conn.api_request.assert_called_with(
+ method="DELETE",
+ path="/projects/routines-project/datasets/test_routines/routines/test_routine",
+ )
+
+ def test_delete_table(self):
+ from google.cloud.bigquery.table import Table
+
+ tables = (
+ self.TABLE_REF,
+ Table(self.TABLE_REF),
+ "{}.{}.{}".format(
+ self.TABLE_REF.project,
+ self.TABLE_REF.dataset_id,
+ self.TABLE_REF.table_id,
+ ),
+ )
+ path = "projects/%s/datasets/%s/tables/%s" % (
+ self.PROJECT,
+ self.DS_ID,
+ self.TABLE_ID,
+ )
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection(*([{}] * len(tables)))
+
+ for arg in tables:
+ client.delete_table(arg)
+ conn.api_request.assert_called_with(method="DELETE", path="/%s" % path)
+
+ def test_delete_table_w_wrong_type(self):
+ creds = _make_credentials()
+ client = self._make_one(project=self.PROJECT, credentials=creds)
+ with self.assertRaises(TypeError):
+ client.delete_table(client.dataset(self.DS_ID))
+
+ def test_delete_table_w_not_found_ok_false(self):
+ path = "/projects/{}/datasets/{}/tables/{}".format(
+ self.PROJECT, self.DS_ID, self.TABLE_ID
+ )
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection(
+ google.api_core.exceptions.NotFound("table not found")
+ )
+
+ with self.assertRaises(google.api_core.exceptions.NotFound):
+ client.delete_table("{}.{}".format(self.DS_ID, self.TABLE_ID))
+
+ conn.api_request.assert_called_with(method="DELETE", path=path)
+
+ def test_delete_table_w_not_found_ok_true(self):
+ path = "/projects/{}/datasets/{}/tables/{}".format(
+ self.PROJECT, self.DS_ID, self.TABLE_ID
+ )
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection(
+ google.api_core.exceptions.NotFound("table not found")
+ )
+
+ client.delete_table(
+ "{}.{}".format(self.DS_ID, self.TABLE_ID), not_found_ok=True
+ )
+
+ conn.api_request.assert_called_with(method="DELETE", path=path)
+
+ def test_job_from_resource_unknown_type(self):
+ from google.cloud.bigquery.job import UnknownJob
+
+ creds = _make_credentials()
+ client = self._make_one(self.PROJECT, creds)
+ got = client.job_from_resource({}) # Can parse redacted job.
+ self.assertIsInstance(got, UnknownJob)
+ self.assertEqual(got.project, self.PROJECT)
+
+ def test_get_job_miss_w_explict_project(self):
+ from google.cloud.exceptions import NotFound
+
+ OTHER_PROJECT = "OTHER_PROJECT"
+ JOB_ID = "NONESUCH"
+ creds = _make_credentials()
+ client = self._make_one(self.PROJECT, creds)
+ conn = client._connection = make_connection()
+
+ with self.assertRaises(NotFound):
+ client.get_job(JOB_ID, project=OTHER_PROJECT, location=self.LOCATION)
+
+ conn.api_request.assert_called_once_with(
+ method="GET",
+ path="/projects/OTHER_PROJECT/jobs/NONESUCH",
+ query_params={"projection": "full", "location": self.LOCATION},
+ )
+
+ def test_get_job_miss_w_client_location(self):
+ from google.cloud.exceptions import NotFound
+
+ OTHER_PROJECT = "OTHER_PROJECT"
+ JOB_ID = "NONESUCH"
+ creds = _make_credentials()
+ client = self._make_one(self.PROJECT, creds, location=self.LOCATION)
+ conn = client._connection = make_connection()
+
+ with self.assertRaises(NotFound):
+ client.get_job(JOB_ID, project=OTHER_PROJECT)
+
+ conn.api_request.assert_called_once_with(
+ method="GET",
+ path="/projects/OTHER_PROJECT/jobs/NONESUCH",
+ query_params={"projection": "full", "location": self.LOCATION},
+ )
+
+ def test_get_job_hit(self):
+ from google.cloud.bigquery.job import CreateDisposition
+ from google.cloud.bigquery.job import QueryJob
+ from google.cloud.bigquery.job import WriteDisposition
+
+ JOB_ID = "query_job"
+ QUERY_DESTINATION_TABLE = "query_destination_table"
+ QUERY = "SELECT * from test_dataset:test_table"
+ ASYNC_QUERY_DATA = {
+ "id": "{}:{}".format(self.PROJECT, JOB_ID),
+ "jobReference": {"projectId": self.PROJECT, "jobId": "query_job"},
+ "state": "DONE",
+ "configuration": {
+ "query": {
+ "query": QUERY,
+ "destinationTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": QUERY_DESTINATION_TABLE,
+ },
+ "createDisposition": CreateDisposition.CREATE_IF_NEEDED,
+ "writeDisposition": WriteDisposition.WRITE_TRUNCATE,
+ }
+ },
+ }
+ creds = _make_credentials()
+ client = self._make_one(self.PROJECT, creds)
+ conn = client._connection = make_connection(ASYNC_QUERY_DATA)
+
+ job = client.get_job(JOB_ID)
+
+ self.assertIsInstance(job, QueryJob)
+ self.assertEqual(job.job_id, JOB_ID)
+ self.assertEqual(job.create_disposition, CreateDisposition.CREATE_IF_NEEDED)
+ self.assertEqual(job.write_disposition, WriteDisposition.WRITE_TRUNCATE)
+
+ conn.api_request.assert_called_once_with(
+ method="GET",
+ path="/projects/PROJECT/jobs/query_job",
+ query_params={"projection": "full"},
+ )
+
+ def test_cancel_job_miss_w_explict_project(self):
+ from google.cloud.exceptions import NotFound
+
+ OTHER_PROJECT = "OTHER_PROJECT"
+ JOB_ID = "NONESUCH"
+ creds = _make_credentials()
+ client = self._make_one(self.PROJECT, creds)
+ conn = client._connection = make_connection()
+
+ with self.assertRaises(NotFound):
+ client.cancel_job(JOB_ID, project=OTHER_PROJECT, location=self.LOCATION)
+
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path="/projects/OTHER_PROJECT/jobs/NONESUCH/cancel",
+ query_params={"projection": "full", "location": self.LOCATION},
+ )
+
+ def test_cancel_job_miss_w_client_location(self):
+ from google.cloud.exceptions import NotFound
+
+ OTHER_PROJECT = "OTHER_PROJECT"
+ JOB_ID = "NONESUCH"
+ creds = _make_credentials()
+ client = self._make_one(self.PROJECT, creds, location=self.LOCATION)
+ conn = client._connection = make_connection()
+
+ with self.assertRaises(NotFound):
+ client.cancel_job(JOB_ID, project=OTHER_PROJECT)
+
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path="/projects/OTHER_PROJECT/jobs/NONESUCH/cancel",
+ query_params={"projection": "full", "location": self.LOCATION},
+ )
+
+ def test_cancel_job_hit(self):
+ from google.cloud.bigquery.job import QueryJob
+
+ JOB_ID = "query_job"
+ QUERY = "SELECT * from test_dataset:test_table"
+ QUERY_JOB_RESOURCE = {
+ "id": "{}:{}".format(self.PROJECT, JOB_ID),
+ "jobReference": {"projectId": self.PROJECT, "jobId": "query_job"},
+ "state": "RUNNING",
+ "configuration": {"query": {"query": QUERY}},
+ }
+ RESOURCE = {"job": QUERY_JOB_RESOURCE}
+ creds = _make_credentials()
+ client = self._make_one(self.PROJECT, creds)
+ conn = client._connection = make_connection(RESOURCE)
+
+ job = client.cancel_job(JOB_ID)
+
+ self.assertIsInstance(job, QueryJob)
+ self.assertEqual(job.job_id, JOB_ID)
+ self.assertEqual(job.query, QUERY)
+
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path="/projects/PROJECT/jobs/query_job/cancel",
+ query_params={"projection": "full"},
+ )
+
+ def test_list_jobs_defaults(self):
+ from google.cloud.bigquery.job import CopyJob
+ from google.cloud.bigquery.job import CreateDisposition
+ from google.cloud.bigquery.job import ExtractJob
+ from google.cloud.bigquery.job import LoadJob
+ from google.cloud.bigquery.job import QueryJob
+ from google.cloud.bigquery.job import WriteDisposition
+
+ SOURCE_TABLE = "source_table"
+ DESTINATION_TABLE = "destination_table"
+ QUERY_DESTINATION_TABLE = "query_destination_table"
+ SOURCE_URI = "gs://test_bucket/src_object*"
+ DESTINATION_URI = "gs://test_bucket/dst_object*"
+ JOB_TYPES = {
+ "load_job": LoadJob,
+ "copy_job": CopyJob,
+ "extract_job": ExtractJob,
+ "query_job": QueryJob,
+ }
+ PATH = "projects/%s/jobs" % self.PROJECT
+ TOKEN = "TOKEN"
+ QUERY = "SELECT * from test_dataset:test_table"
+ ASYNC_QUERY_DATA = {
+ "id": "%s:%s" % (self.PROJECT, "query_job"),
+ "jobReference": {"projectId": self.PROJECT, "jobId": "query_job"},
+ "state": "DONE",
+ "configuration": {
+ "query": {
+ "query": QUERY,
+ "destinationTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": QUERY_DESTINATION_TABLE,
+ },
+ "createDisposition": CreateDisposition.CREATE_IF_NEEDED,
+ "writeDisposition": WriteDisposition.WRITE_TRUNCATE,
+ }
+ },
+ }
+ EXTRACT_DATA = {
+ "id": "%s:%s" % (self.PROJECT, "extract_job"),
+ "jobReference": {"projectId": self.PROJECT, "jobId": "extract_job"},
+ "state": "DONE",
+ "configuration": {
+ "extract": {
+ "sourceTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": SOURCE_TABLE,
+ },
+ "destinationUris": [DESTINATION_URI],
+ }
+ },
+ }
+ COPY_DATA = {
+ "id": "%s:%s" % (self.PROJECT, "copy_job"),
+ "jobReference": {"projectId": self.PROJECT, "jobId": "copy_job"},
+ "state": "DONE",
+ "configuration": {
+ "copy": {
+ "sourceTables": [
+ {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": SOURCE_TABLE,
+ }
+ ],
+ "destinationTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": DESTINATION_TABLE,
+ },
+ }
+ },
+ }
+ LOAD_DATA = {
+ "id": "%s:%s" % (self.PROJECT, "load_job"),
+ "jobReference": {"projectId": self.PROJECT, "jobId": "load_job"},
+ "state": "DONE",
+ "configuration": {
+ "load": {
+ "destinationTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": SOURCE_TABLE,
+ },
+ "sourceUris": [SOURCE_URI],
+ }
+ },
+ }
+ DATA = {
+ "nextPageToken": TOKEN,
+ "jobs": [ASYNC_QUERY_DATA, EXTRACT_DATA, COPY_DATA, LOAD_DATA],
+ }
+ creds = _make_credentials()
+ client = self._make_one(self.PROJECT, creds)
+ conn = client._connection = make_connection(DATA)
+
+ iterator = client.list_jobs()
+ page = six.next(iterator.pages)
+ jobs = list(page)
+ token = iterator.next_page_token
+
+ self.assertEqual(len(jobs), len(DATA["jobs"]))
+ for found, expected in zip(jobs, DATA["jobs"]):
+ name = expected["jobReference"]["jobId"]
+ self.assertIsInstance(found, JOB_TYPES[name])
+ self.assertEqual(found.job_id, name)
+ self.assertEqual(token, TOKEN)
+
+ conn.api_request.assert_called_once_with(
+ method="GET", path="/%s" % PATH, query_params={"projection": "full"}
+ )
+
+ def test_list_jobs_load_job_wo_sourceUris(self):
+ from google.cloud.bigquery.job import LoadJob
+
+ SOURCE_TABLE = "source_table"
+ JOB_TYPES = {"load_job": LoadJob}
+ PATH = "projects/%s/jobs" % self.PROJECT
+ TOKEN = "TOKEN"
+ LOAD_DATA = {
+ "id": "%s:%s" % (self.PROJECT, "load_job"),
+ "jobReference": {"projectId": self.PROJECT, "jobId": "load_job"},
+ "state": "DONE",
+ "configuration": {
+ "load": {
+ "destinationTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": SOURCE_TABLE,
+ }
+ }
+ },
+ }
+ DATA = {"nextPageToken": TOKEN, "jobs": [LOAD_DATA]}
+ creds = _make_credentials()
+ client = self._make_one(self.PROJECT, creds)
+ conn = client._connection = make_connection(DATA)
+
+ iterator = client.list_jobs()
+ page = six.next(iterator.pages)
+ jobs = list(page)
+ token = iterator.next_page_token
+
+ self.assertEqual(len(jobs), len(DATA["jobs"]))
+ for found, expected in zip(jobs, DATA["jobs"]):
+ name = expected["jobReference"]["jobId"]
+ self.assertIsInstance(found, JOB_TYPES[name])
+ self.assertEqual(found.job_id, name)
+ self.assertEqual(token, TOKEN)
+
+ conn.api_request.assert_called_once_with(
+ method="GET", path="/%s" % PATH, query_params={"projection": "full"}
+ )
+
+ def test_list_jobs_explicit_missing(self):
+ PATH = "projects/%s/jobs" % self.PROJECT
+ DATA = {}
+ TOKEN = "TOKEN"
+ creds = _make_credentials()
+ client = self._make_one(self.PROJECT, creds)
+ conn = client._connection = make_connection(DATA)
+
+ iterator = client.list_jobs(
+ max_results=1000, page_token=TOKEN, all_users=True, state_filter="done"
+ )
+ page = six.next(iterator.pages)
+ jobs = list(page)
+ token = iterator.next_page_token
+
+ self.assertEqual(len(jobs), 0)
+ self.assertIsNone(token)
+
+ conn.api_request.assert_called_once_with(
+ method="GET",
+ path="/%s" % PATH,
+ query_params={
+ "projection": "full",
+ "maxResults": 1000,
+ "pageToken": TOKEN,
+ "allUsers": True,
+ "stateFilter": "done",
+ },
+ )
+
+ def test_list_jobs_w_project(self):
+ creds = _make_credentials()
+ client = self._make_one(self.PROJECT, creds)
+ conn = client._connection = make_connection({})
+
+ list(client.list_jobs(project="other-project"))
+
+ conn.api_request.assert_called_once_with(
+ method="GET",
+ path="/projects/other-project/jobs",
+ query_params={"projection": "full"},
+ )
+
+ def test_list_jobs_w_time_filter(self):
+ creds = _make_credentials()
+ client = self._make_one(self.PROJECT, creds)
+ conn = client._connection = make_connection({})
+
+ # One millisecond after the unix epoch.
+ start_time = datetime.datetime(1970, 1, 1, 0, 0, 0, 1000)
+ # One millisecond after the the 2038 31-bit signed int rollover
+ end_time = datetime.datetime(2038, 1, 19, 3, 14, 7, 1000)
+ end_time_millis = (((2 ** 31) - 1) * 1000) + 1
+
+ list(client.list_jobs(min_creation_time=start_time, max_creation_time=end_time))
+
+ conn.api_request.assert_called_once_with(
+ method="GET",
+ path="/projects/%s/jobs" % self.PROJECT,
+ query_params={
+ "projection": "full",
+ "minCreationTime": "1",
+ "maxCreationTime": str(end_time_millis),
+ },
+ )
+
+ def test_load_table_from_uri(self):
+ from google.cloud.bigquery.job import LoadJob
+
+ JOB = "job_name"
+ DESTINATION = "destination_table"
+ SOURCE_URI = "http://example.com/source.csv"
+ RESOURCE = {
+ "jobReference": {"projectId": self.PROJECT, "jobId": JOB},
+ "configuration": {
+ "load": {
+ "sourceUris": [SOURCE_URI],
+ "destinationTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": DESTINATION,
+ },
+ }
+ },
+ }
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection(RESOURCE)
+ destination = client.dataset(self.DS_ID).table(DESTINATION)
+
+ job = client.load_table_from_uri(SOURCE_URI, destination, job_id=JOB)
+
+ # Check that load_table_from_uri actually starts the job.
+ conn.api_request.assert_called_once_with(
+ method="POST", path="/projects/%s/jobs" % self.PROJECT, data=RESOURCE
+ )
+
+ self.assertIsInstance(job, LoadJob)
+ self.assertIs(job._client, client)
+ self.assertEqual(job.job_id, JOB)
+ self.assertEqual(list(job.source_uris), [SOURCE_URI])
+ self.assertIs(job.destination, destination)
+
+ conn = client._connection = make_connection(RESOURCE)
+
+ job = client.load_table_from_uri([SOURCE_URI], destination, job_id=JOB)
+ self.assertIsInstance(job, LoadJob)
+ self.assertIs(job._client, client)
+ self.assertEqual(job.job_id, JOB)
+ self.assertEqual(list(job.source_uris), [SOURCE_URI])
+ self.assertIs(job.destination, destination)
+
+ def test_load_table_from_uri_w_explicit_project(self):
+ job_id = "this-is-a-job-id"
+ destination_id = "destination_table"
+ source_uri = "gs://example/source.csv"
+ resource = {
+ "jobReference": {
+ "projectId": "other-project",
+ "location": self.LOCATION,
+ "jobId": job_id,
+ },
+ "configuration": {
+ "load": {
+ "sourceUris": [source_uri],
+ "destinationTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": destination_id,
+ },
+ }
+ },
+ }
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection(resource)
+ destination = client.dataset(self.DS_ID).table(destination_id)
+
+ client.load_table_from_uri(
+ source_uri,
+ destination,
+ job_id=job_id,
+ project="other-project",
+ location=self.LOCATION,
+ )
+
+ # Check that load_table_from_uri actually starts the job.
+ conn.api_request.assert_called_once_with(
+ method="POST", path="/projects/other-project/jobs", data=resource
+ )
+
+ def test_load_table_from_uri_w_client_location(self):
+ job_id = "this-is-a-job-id"
+ destination_id = "destination_table"
+ source_uri = "gs://example/source.csv"
+ resource = {
+ "jobReference": {
+ "projectId": "other-project",
+ "location": self.LOCATION,
+ "jobId": job_id,
+ },
+ "configuration": {
+ "load": {
+ "sourceUris": [source_uri],
+ "destinationTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": destination_id,
+ },
+ }
+ },
+ }
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(
+ project=self.PROJECT, credentials=creds, _http=http, location=self.LOCATION
+ )
+ conn = client._connection = make_connection(resource)
+
+ client.load_table_from_uri(
+ source_uri,
+ # Test with string for table ID.
+ "{}.{}".format(self.DS_ID, destination_id),
+ job_id=job_id,
+ project="other-project",
+ )
+
+ # Check that load_table_from_uri actually starts the job.
+ conn.api_request.assert_called_once_with(
+ method="POST", path="/projects/other-project/jobs", data=resource
+ )
+
+ @staticmethod
+ def _mock_requests_response(status_code, headers, content=b""):
+ return mock.Mock(
+ content=content,
+ headers=headers,
+ status_code=status_code,
+ spec=["content", "headers", "status_code"],
+ )
+
+ def _mock_transport(self, status_code, headers, content=b""):
+ fake_transport = mock.Mock(spec=["request"])
+ fake_response = self._mock_requests_response(
+ status_code, headers, content=content
+ )
+ fake_transport.request.return_value = fake_response
+ return fake_transport
+
+ def _initiate_resumable_upload_helper(self, num_retries=None):
+ from google.resumable_media.requests import ResumableUpload
+ from google.cloud.bigquery.client import _DEFAULT_CHUNKSIZE
+ from google.cloud.bigquery.client import _GENERIC_CONTENT_TYPE
+ from google.cloud.bigquery.client import _get_upload_headers
+ from google.cloud.bigquery.job import LoadJob
+ from google.cloud.bigquery.job import LoadJobConfig
+ from google.cloud.bigquery.job import SourceFormat
+
+ # Create mocks to be checked for doing transport.
+ resumable_url = "http://test.invalid?upload_id=hey-you"
+ response_headers = {"location": resumable_url}
+ fake_transport = self._mock_transport(http_client.OK, response_headers)
+ client = self._make_one(project=self.PROJECT, _http=fake_transport)
+ conn = client._connection = make_connection()
+
+ # Create some mock arguments and call the method under test.
+ data = b"goodbye gudbi gootbee"
+ stream = io.BytesIO(data)
+ config = LoadJobConfig()
+ config.source_format = SourceFormat.CSV
+ job = LoadJob(None, None, self.TABLE_REF, client, job_config=config)
+ metadata = job.to_api_repr()
+ upload, transport = client._initiate_resumable_upload(
+ stream, metadata, num_retries
+ )
+
+ # Check the returned values.
+ self.assertIsInstance(upload, ResumableUpload)
+ upload_url = (
+ "https://www.googleapis.com/upload/bigquery/v2/projects/"
+ + self.PROJECT
+ + "/jobs?uploadType=resumable"
+ )
+ self.assertEqual(upload.upload_url, upload_url)
+ expected_headers = _get_upload_headers(conn.user_agent)
+ self.assertEqual(upload._headers, expected_headers)
+ self.assertFalse(upload.finished)
+ self.assertEqual(upload._chunk_size, _DEFAULT_CHUNKSIZE)
+ self.assertIs(upload._stream, stream)
+ self.assertIsNone(upload._total_bytes)
+ self.assertEqual(upload._content_type, _GENERIC_CONTENT_TYPE)
+ self.assertEqual(upload.resumable_url, resumable_url)
+
+ retry_strategy = upload._retry_strategy
+ self.assertEqual(retry_strategy.max_sleep, 64.0)
+ if num_retries is None:
+ self.assertEqual(retry_strategy.max_cumulative_retry, 600.0)
+ self.assertIsNone(retry_strategy.max_retries)
+ else:
+ self.assertIsNone(retry_strategy.max_cumulative_retry)
+ self.assertEqual(retry_strategy.max_retries, num_retries)
+ self.assertIs(transport, fake_transport)
+ # Make sure we never read from the stream.
+ self.assertEqual(stream.tell(), 0)
+
+ # Check the mocks.
+ request_headers = expected_headers.copy()
+ request_headers["x-upload-content-type"] = _GENERIC_CONTENT_TYPE
+ fake_transport.request.assert_called_once_with(
+ "POST",
+ upload_url,
+ data=json.dumps(metadata).encode("utf-8"),
+ headers=request_headers,
+ timeout=mock.ANY,
+ )
+
+ def test__initiate_resumable_upload(self):
+ self._initiate_resumable_upload_helper()
+
+ def test__initiate_resumable_upload_with_retry(self):
+ self._initiate_resumable_upload_helper(num_retries=11)
+
+ def _do_multipart_upload_success_helper(self, get_boundary, num_retries=None):
+ from google.cloud.bigquery.client import _get_upload_headers
+ from google.cloud.bigquery.job import LoadJob
+ from google.cloud.bigquery.job import LoadJobConfig
+ from google.cloud.bigquery.job import SourceFormat
+
+ fake_transport = self._mock_transport(http_client.OK, {})
+ client = self._make_one(project=self.PROJECT, _http=fake_transport)
+ conn = client._connection = make_connection()
+
+ # Create some mock arguments.
+ data = b"Bzzzz-zap \x00\x01\xf4"
+ stream = io.BytesIO(data)
+ config = LoadJobConfig()
+ config.source_format = SourceFormat.CSV
+ job = LoadJob(None, None, self.TABLE_REF, client, job_config=config)
+ metadata = job.to_api_repr()
+ size = len(data)
+ response = client._do_multipart_upload(stream, metadata, size, num_retries)
+
+ # Check the mocks and the returned value.
+ self.assertIs(response, fake_transport.request.return_value)
+ self.assertEqual(stream.tell(), size)
+ get_boundary.assert_called_once_with()
+
+ upload_url = (
+ "https://www.googleapis.com/upload/bigquery/v2/projects/"
+ + self.PROJECT
+ + "/jobs?uploadType=multipart"
+ )
+ payload = (
+ b"--==0==\r\n"
+ + b"content-type: application/json; charset=UTF-8\r\n\r\n"
+ + json.dumps(metadata).encode("utf-8")
+ + b"\r\n"
+ + b"--==0==\r\n"
+ + b"content-type: */*\r\n\r\n"
+ + data
+ + b"\r\n"
+ + b"--==0==--"
+ )
+ headers = _get_upload_headers(conn.user_agent)
+ headers["content-type"] = b'multipart/related; boundary="==0=="'
+ fake_transport.request.assert_called_once_with(
+ "POST", upload_url, data=payload, headers=headers, timeout=mock.ANY
+ )
+
+ @mock.patch(u"google.resumable_media._upload.get_boundary", return_value=b"==0==")
+ def test__do_multipart_upload(self, get_boundary):
+ self._do_multipart_upload_success_helper(get_boundary)
+
+ @mock.patch(u"google.resumable_media._upload.get_boundary", return_value=b"==0==")
+ def test__do_multipart_upload_with_retry(self, get_boundary):
+ self._do_multipart_upload_success_helper(get_boundary, num_retries=8)
+
+ def test_copy_table(self):
+ from google.cloud.bigquery.job import CopyJob
+
+ JOB = "job_name"
+ SOURCE = "source_table"
+ DESTINATION = "destination_table"
+ RESOURCE = {
+ "jobReference": {"projectId": self.PROJECT, "jobId": JOB},
+ "configuration": {
+ "copy": {
+ "sourceTables": [
+ {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": SOURCE,
+ }
+ ],
+ "destinationTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": DESTINATION,
+ },
+ }
+ },
+ }
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection(RESOURCE)
+ dataset = client.dataset(self.DS_ID)
+ source = dataset.table(SOURCE)
+ destination = dataset.table(DESTINATION)
+
+ job = client.copy_table(source, destination, job_id=JOB)
+
+ # Check that copy_table actually starts the job.
+ conn.api_request.assert_called_once_with(
+ method="POST", path="/projects/%s/jobs" % self.PROJECT, data=RESOURCE
+ )
+
+ self.assertIsInstance(job, CopyJob)
+ self.assertIs(job._client, client)
+ self.assertEqual(job.job_id, JOB)
+ self.assertEqual(list(job.sources), [source])
+ self.assertIs(job.destination, destination)
+
+ conn = client._connection = make_connection(RESOURCE)
+ source2 = dataset.table(SOURCE + "2")
+ job = client.copy_table([source, source2], destination, job_id=JOB)
+ self.assertIsInstance(job, CopyJob)
+ self.assertIs(job._client, client)
+ self.assertEqual(job.job_id, JOB)
+ self.assertEqual(list(job.sources), [source, source2])
+ self.assertIs(job.destination, destination)
+
+ def test_copy_table_w_explicit_project(self):
+ job_id = "this-is-a-job-id"
+ source_id = "source_table"
+ destination_id = "destination_table"
+ resource = {
+ "jobReference": {
+ "projectId": "other-project",
+ "location": self.LOCATION,
+ "jobId": job_id,
+ },
+ "configuration": {
+ "copy": {
+ "sourceTables": [
+ {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": source_id,
+ }
+ ],
+ "destinationTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": destination_id,
+ },
+ }
+ },
+ }
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection(resource)
+ dataset = client.dataset(self.DS_ID)
+ source = dataset.table(source_id)
+ destination = dataset.table(destination_id)
+
+ client.copy_table(
+ source,
+ destination,
+ job_id=job_id,
+ project="other-project",
+ location=self.LOCATION,
+ )
+
+ # Check that copy_table actually starts the job.
+ conn.api_request.assert_called_once_with(
+ method="POST", path="/projects/other-project/jobs", data=resource
+ )
+
+ def test_copy_table_w_client_location(self):
+ job_id = "this-is-a-job-id"
+ source_id = "source_table"
+ destination_id = "destination_table"
+ resource = {
+ "jobReference": {
+ "projectId": "other-project",
+ "location": self.LOCATION,
+ "jobId": job_id,
+ },
+ "configuration": {
+ "copy": {
+ "sourceTables": [
+ {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": source_id,
+ }
+ ],
+ "destinationTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": destination_id,
+ },
+ }
+ },
+ }
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(
+ project=self.PROJECT, credentials=creds, _http=http, location=self.LOCATION
+ )
+ conn = client._connection = make_connection(resource)
+
+ client.copy_table(
+ # Test with string for table IDs.
+ "{}.{}".format(self.DS_ID, source_id),
+ "{}.{}".format(self.DS_ID, destination_id),
+ job_id=job_id,
+ project="other-project",
+ )
+
+ # Check that copy_table actually starts the job.
+ conn.api_request.assert_called_once_with(
+ method="POST", path="/projects/other-project/jobs", data=resource
+ )
+
+ def test_copy_table_w_source_strings(self):
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ client._connection = make_connection({})
+ sources = [
+ "dataset_wo_proj.some_table",
+ "other_project.other_dataset.other_table",
+ client.dataset("dataset_from_ref").table("table_from_ref"),
+ ]
+ destination = "some_project.some_dataset.destination_table"
+
+ job = client.copy_table(sources, destination)
+
+ expected_sources = [
+ client.dataset("dataset_wo_proj").table("some_table"),
+ client.dataset("other_dataset", project="other_project").table(
+ "other_table"
+ ),
+ client.dataset("dataset_from_ref").table("table_from_ref"),
+ ]
+ self.assertEqual(list(job.sources), expected_sources)
+ expected_destination = client.dataset(
+ "some_dataset", project="some_project"
+ ).table("destination_table")
+ self.assertEqual(job.destination, expected_destination)
+
+ def test_extract_table(self):
+ from google.cloud.bigquery.job import ExtractJob
+
+ JOB = "job_id"
+ SOURCE = "source_table"
+ DESTINATION = "gs://bucket_name/object_name"
+ RESOURCE = {
+ "jobReference": {"projectId": self.PROJECT, "jobId": JOB},
+ "configuration": {
+ "extract": {
+ "sourceTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": SOURCE,
+ },
+ "destinationUris": [DESTINATION],
+ }
+ },
+ }
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection(RESOURCE)
+ dataset = client.dataset(self.DS_ID)
+ source = dataset.table(SOURCE)
+
+ job = client.extract_table(source, DESTINATION, job_id=JOB)
+
+ # Check that extract_table actually starts the job.
+ conn.api_request.assert_called_once_with(
+ method="POST", path="/projects/PROJECT/jobs", data=RESOURCE
+ )
+
+ # Check the job resource.
+ self.assertIsInstance(job, ExtractJob)
+ self.assertIs(job._client, client)
+ self.assertEqual(job.job_id, JOB)
+ self.assertEqual(job.source, source)
+ self.assertEqual(list(job.destination_uris), [DESTINATION])
+
+ def test_extract_table_w_explicit_project(self):
+ job_id = "job_id"
+ source_id = "source_table"
+ destination = "gs://bucket_name/object_name"
+ resource = {
+ "jobReference": {
+ "projectId": "other-project",
+ "location": self.LOCATION,
+ "jobId": job_id,
+ },
+ "configuration": {
+ "extract": {
+ "sourceTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": source_id,
+ },
+ "destinationUris": [destination],
+ }
+ },
+ }
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection(resource)
+ dataset = client.dataset(self.DS_ID)
+ source = dataset.table(source_id)
+
+ client.extract_table(
+ source,
+ destination,
+ job_id=job_id,
+ project="other-project",
+ location=self.LOCATION,
+ )
+
+ # Check that extract_table actually starts the job.
+ conn.api_request.assert_called_once_with(
+ method="POST", path="/projects/other-project/jobs", data=resource
+ )
+
+ def test_extract_table_w_client_location(self):
+ job_id = "job_id"
+ source_id = "source_table"
+ destination = "gs://bucket_name/object_name"
+ resource = {
+ "jobReference": {
+ "projectId": "other-project",
+ "location": self.LOCATION,
+ "jobId": job_id,
+ },
+ "configuration": {
+ "extract": {
+ "sourceTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": source_id,
+ },
+ "destinationUris": [destination],
+ }
+ },
+ }
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(
+ project=self.PROJECT, credentials=creds, _http=http, location=self.LOCATION
+ )
+ conn = client._connection = make_connection(resource)
+
+ client.extract_table(
+ # Test with string for table ID.
+ "{}.{}".format(self.DS_ID, source_id),
+ destination,
+ job_id=job_id,
+ project="other-project",
+ )
+
+ # Check that extract_table actually starts the job.
+ conn.api_request.assert_called_once_with(
+ method="POST", path="/projects/other-project/jobs", data=resource
+ )
+
+ def test_extract_table_generated_job_id(self):
+ from google.cloud.bigquery.job import ExtractJob
+ from google.cloud.bigquery.job import ExtractJobConfig
+ from google.cloud.bigquery.job import DestinationFormat
+
+ JOB = "job_id"
+ SOURCE = "source_table"
+ DESTINATION = "gs://bucket_name/object_name"
+ RESOURCE = {
+ "jobReference": {"projectId": self.PROJECT, "jobId": JOB},
+ "configuration": {
+ "extract": {
+ "sourceTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": SOURCE,
+ },
+ "destinationUris": [DESTINATION],
+ "destinationFormat": "NEWLINE_DELIMITED_JSON",
+ }
+ },
+ }
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection(RESOURCE)
+ dataset = client.dataset(self.DS_ID)
+ source = dataset.table(SOURCE)
+ job_config = ExtractJobConfig()
+ job_config.destination_format = DestinationFormat.NEWLINE_DELIMITED_JSON
+
+ job = client.extract_table(source, DESTINATION, job_config=job_config)
+
+ # Check that extract_table actually starts the job.
+ conn.api_request.assert_called_once()
+ _, req = conn.api_request.call_args
+ self.assertEqual(req["method"], "POST")
+ self.assertEqual(req["path"], "/projects/PROJECT/jobs")
+ self.assertIsInstance(req["data"]["jobReference"]["jobId"], six.string_types)
+
+ # Check the job resource.
+ self.assertIsInstance(job, ExtractJob)
+ self.assertIs(job._client, client)
+ self.assertEqual(job.source, source)
+ self.assertEqual(list(job.destination_uris), [DESTINATION])
+
+ def test_extract_table_w_destination_uris(self):
+ from google.cloud.bigquery.job import ExtractJob
+
+ JOB = "job_id"
+ SOURCE = "source_table"
+ DESTINATION1 = "gs://bucket_name/object_one"
+ DESTINATION2 = "gs://bucket_name/object_two"
+ RESOURCE = {
+ "jobReference": {"projectId": self.PROJECT, "jobId": JOB},
+ "configuration": {
+ "extract": {
+ "sourceTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": SOURCE,
+ },
+ "destinationUris": [DESTINATION1, DESTINATION2],
+ }
+ },
+ }
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection(RESOURCE)
+ dataset = client.dataset(self.DS_ID)
+ source = dataset.table(SOURCE)
+
+ job = client.extract_table(source, [DESTINATION1, DESTINATION2], job_id=JOB)
+
+ # Check that extract_table actually starts the job.
+ conn.api_request.assert_called_once()
+ _, req = conn.api_request.call_args
+ self.assertEqual(req["method"], "POST")
+ self.assertEqual(req["path"], "/projects/PROJECT/jobs")
+
+ # Check the job resource.
+ self.assertIsInstance(job, ExtractJob)
+ self.assertIs(job._client, client)
+ self.assertEqual(job.job_id, JOB)
+ self.assertEqual(job.source, source)
+ self.assertEqual(list(job.destination_uris), [DESTINATION1, DESTINATION2])
+
+ def test_query_defaults(self):
+ from google.cloud.bigquery.job import QueryJob
+
+ QUERY = "select count(*) from persons"
+ RESOURCE = {
+ "jobReference": {"projectId": self.PROJECT, "jobId": "some-random-id"},
+ "configuration": {"query": {"query": QUERY, "useLegacySql": False}},
+ }
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection(RESOURCE)
+
+ job = client.query(QUERY)
+
+ self.assertIsInstance(job, QueryJob)
+ self.assertIsInstance(job.job_id, six.string_types)
+ self.assertIs(job._client, client)
+ self.assertEqual(job.query, QUERY)
+ self.assertEqual(job.udf_resources, [])
+ self.assertEqual(job.query_parameters, [])
+
+ # Check that query actually starts the job.
+ conn.api_request.assert_called_once()
+ _, req = conn.api_request.call_args
+ self.assertEqual(req["method"], "POST")
+ self.assertEqual(req["path"], "/projects/PROJECT/jobs")
+ sent = req["data"]
+ self.assertIsInstance(sent["jobReference"]["jobId"], six.string_types)
+ sent_config = sent["configuration"]["query"]
+ self.assertEqual(sent_config["query"], QUERY)
+ self.assertFalse(sent_config["useLegacySql"])
+
+ def test_query_w_explicit_project(self):
+ job_id = "some-job-id"
+ query = "select count(*) from persons"
+ resource = {
+ "jobReference": {
+ "projectId": "other-project",
+ "location": self.LOCATION,
+ "jobId": job_id,
+ },
+ "configuration": {"query": {"query": query, "useLegacySql": False}},
+ }
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection(resource)
+
+ client.query(
+ query, job_id=job_id, project="other-project", location=self.LOCATION
+ )
+
+ # Check that query actually starts the job.
+ conn.api_request.assert_called_once_with(
+ method="POST", path="/projects/other-project/jobs", data=resource
+ )
+
+ def test_query_w_explicit_job_config(self):
+ job_id = "some-job-id"
+ query = "select count(*) from persons"
+ resource = {
+ "jobReference": {
+ "jobId": job_id,
+ "projectId": self.PROJECT,
+ "location": self.LOCATION,
+ },
+ "configuration": {
+ "query": {
+ "query": query,
+ "defaultDataset": {
+ "projectId": self.PROJECT,
+ "datasetId": "some-dataset",
+ },
+ "useLegacySql": False,
+ "useQueryCache": True,
+ "maximumBytesBilled": "2000",
+ }
+ },
+ }
+
+ creds = _make_credentials()
+ http = object()
+
+ from google.cloud.bigquery import QueryJobConfig, DatasetReference
+
+ default_job_config = QueryJobConfig()
+ default_job_config.default_dataset = DatasetReference(
+ self.PROJECT, "some-dataset"
+ )
+ default_job_config.maximum_bytes_billed = 1000
+
+ client = self._make_one(
+ project=self.PROJECT,
+ credentials=creds,
+ _http=http,
+ default_query_job_config=default_job_config,
+ )
+ conn = client._connection = make_connection(resource)
+
+ job_config = QueryJobConfig()
+ job_config.use_query_cache = True
+ job_config.maximum_bytes_billed = 2000
+
+ client.query(
+ query, job_id=job_id, location=self.LOCATION, job_config=job_config
+ )
+
+ # Check that query actually starts the job.
+ conn.api_request.assert_called_once_with(
+ method="POST", path="/projects/PROJECT/jobs", data=resource
+ )
+
+ def test_query_w_explicit_job_config_override(self):
+ job_id = "some-job-id"
+ query = "select count(*) from persons"
+ resource = {
+ "jobReference": {
+ "jobId": job_id,
+ "projectId": self.PROJECT,
+ "location": self.LOCATION,
+ },
+ "configuration": {
+ "query": {
+ "query": query,
+ "defaultDataset": None,
+ "useLegacySql": False,
+ "useQueryCache": True,
+ "maximumBytesBilled": "2000",
+ }
+ },
+ }
+
+ creds = _make_credentials()
+ http = object()
+
+ from google.cloud.bigquery import QueryJobConfig, DatasetReference
+
+ default_job_config = QueryJobConfig()
+ default_job_config.default_dataset = DatasetReference(
+ self.PROJECT, "some-dataset"
+ )
+ default_job_config.maximum_bytes_billed = 1000
+
+ client = self._make_one(
+ project=self.PROJECT,
+ credentials=creds,
+ _http=http,
+ default_query_job_config=default_job_config,
+ )
+ conn = client._connection = make_connection(resource)
+
+ job_config = QueryJobConfig()
+ job_config.use_query_cache = True
+ job_config.maximum_bytes_billed = 2000
+ job_config.default_dataset = None
+
+ client.query(
+ query, job_id=job_id, location=self.LOCATION, job_config=job_config
+ )
+
+ # Check that query actually starts the job.
+ conn.api_request.assert_called_once_with(
+ method="POST", path="/projects/PROJECT/jobs", data=resource
+ )
+
+ def test_query_w_client_default_config_no_incoming(self):
+ job_id = "some-job-id"
+ query = "select count(*) from persons"
+ resource = {
+ "jobReference": {
+ "jobId": job_id,
+ "projectId": self.PROJECT,
+ "location": self.LOCATION,
+ },
+ "configuration": {
+ "query": {
+ "query": query,
+ "useLegacySql": False,
+ "maximumBytesBilled": "1000",
+ }
+ },
+ }
+
+ creds = _make_credentials()
+ http = object()
+
+ from google.cloud.bigquery import QueryJobConfig
+
+ default_job_config = QueryJobConfig()
+ default_job_config.maximum_bytes_billed = 1000
+
+ client = self._make_one(
+ project=self.PROJECT,
+ credentials=creds,
+ _http=http,
+ default_query_job_config=default_job_config,
+ )
+ conn = client._connection = make_connection(resource)
+
+ client.query(query, job_id=job_id, location=self.LOCATION)
+
+ # Check that query actually starts the job.
+ conn.api_request.assert_called_once_with(
+ method="POST", path="/projects/PROJECT/jobs", data=resource
+ )
+
+ def test_query_w_client_location(self):
+ job_id = "some-job-id"
+ query = "select count(*) from persons"
+ resource = {
+ "jobReference": {
+ "projectId": "other-project",
+ "location": self.LOCATION,
+ "jobId": job_id,
+ },
+ "configuration": {"query": {"query": query, "useLegacySql": False}},
+ }
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(
+ project=self.PROJECT, credentials=creds, _http=http, location=self.LOCATION
+ )
+ conn = client._connection = make_connection(resource)
+
+ client.query(query, job_id=job_id, project="other-project")
+
+ # Check that query actually starts the job.
+ conn.api_request.assert_called_once_with(
+ method="POST", path="/projects/other-project/jobs", data=resource
+ )
+
+ def test_query_detect_location(self):
+ query = "select count(*) from persons"
+ resource_location = "EU"
+ resource = {
+ "jobReference": {
+ "projectId": self.PROJECT,
+ # Location not set in request, but present in the response.
+ "location": resource_location,
+ "jobId": "some-random-id",
+ },
+ "configuration": {"query": {"query": query, "useLegacySql": False}},
+ }
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection(resource)
+
+ job = client.query(query)
+
+ self.assertEqual(job.location, resource_location)
+
+ # Check that request did not contain a location.
+ conn.api_request.assert_called_once()
+ _, req = conn.api_request.call_args
+ sent = req["data"]
+ self.assertIsNone(sent["jobReference"].get("location"))
+
+ def test_query_w_udf_resources(self):
+ from google.cloud.bigquery.job import QueryJob
+ from google.cloud.bigquery.job import QueryJobConfig
+ from google.cloud.bigquery.query import UDFResource
+
+ RESOURCE_URI = "gs://some-bucket/js/lib.js"
+ JOB = "job_name"
+ QUERY = "select count(*) from persons"
+ RESOURCE = {
+ "jobReference": {"projectId": self.PROJECT, "jobId": JOB},
+ "configuration": {
+ "query": {
+ "query": QUERY,
+ "useLegacySql": True,
+ "userDefinedFunctionResources": [{"resourceUri": RESOURCE_URI}],
+ }
+ },
+ }
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection(RESOURCE)
+ udf_resources = [UDFResource("resourceUri", RESOURCE_URI)]
+ config = QueryJobConfig()
+ config.udf_resources = udf_resources
+ config.use_legacy_sql = True
+
+ job = client.query(QUERY, job_config=config, job_id=JOB)
+
+ self.assertIsInstance(job, QueryJob)
+ self.assertIs(job._client, client)
+ self.assertEqual(job.job_id, JOB)
+ self.assertEqual(job.query, QUERY)
+ self.assertEqual(job.udf_resources, udf_resources)
+ self.assertEqual(job.query_parameters, [])
+
+ # Check that query actually starts the job.
+ conn.api_request.assert_called_once()
+ _, req = conn.api_request.call_args
+ self.assertEqual(req["method"], "POST")
+ self.assertEqual(req["path"], "/projects/PROJECT/jobs")
+ sent = req["data"]
+ self.assertIsInstance(sent["jobReference"]["jobId"], six.string_types)
+ sent_config = sent["configuration"]["query"]
+ self.assertEqual(sent_config["query"], QUERY)
+ self.assertTrue(sent_config["useLegacySql"])
+ self.assertEqual(
+ sent_config["userDefinedFunctionResources"][0],
+ {"resourceUri": RESOURCE_URI},
+ )
+
+ def test_query_w_query_parameters(self):
+ from google.cloud.bigquery.job import QueryJob
+ from google.cloud.bigquery.job import QueryJobConfig
+ from google.cloud.bigquery.query import ScalarQueryParameter
+
+ JOB = "job_name"
+ QUERY = "select count(*) from persons"
+ RESOURCE = {
+ "jobReference": {"projectId": self.PROJECT, "jobId": JOB},
+ "configuration": {
+ "query": {
+ "query": QUERY,
+ "useLegacySql": False,
+ "queryParameters": [
+ {
+ "name": "foo",
+ "parameterType": {"type": "INT64"},
+ "parameterValue": {"value": "123"},
+ }
+ ],
+ }
+ },
+ }
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection(RESOURCE)
+ query_parameters = [ScalarQueryParameter("foo", "INT64", 123)]
+ config = QueryJobConfig()
+ config.query_parameters = query_parameters
+
+ job = client.query(QUERY, job_config=config, job_id=JOB)
+
+ self.assertIsInstance(job, QueryJob)
+ self.assertIs(job._client, client)
+ self.assertEqual(job.job_id, JOB)
+ self.assertEqual(job.query, QUERY)
+ self.assertEqual(job.udf_resources, [])
+ self.assertEqual(job.query_parameters, query_parameters)
+
+ # Check that query actually starts the job.
+ conn.api_request.assert_called_once()
+ _, req = conn.api_request.call_args
+ self.assertEqual(req["method"], "POST")
+ self.assertEqual(req["path"], "/projects/PROJECT/jobs")
+ sent = req["data"]
+ self.assertEqual(sent["jobReference"]["jobId"], JOB)
+ sent_config = sent["configuration"]["query"]
+ self.assertEqual(sent_config["query"], QUERY)
+ self.assertFalse(sent_config["useLegacySql"])
+ self.assertEqual(
+ sent_config["queryParameters"][0],
+ {
+ "name": "foo",
+ "parameterType": {"type": "INT64"},
+ "parameterValue": {"value": "123"},
+ },
+ )
+
+ def test_insert_rows_wo_schema(self):
+ from google.cloud.bigquery.table import Table
+
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ table = Table(self.TABLE_REF)
+ ROWS = [
+ ("Phred Phlyntstone", 32),
+ ("Bharney Rhubble", 33),
+ ("Wylma Phlyntstone", 29),
+ ("Bhettye Rhubble", 27),
+ ]
+
+ with self.assertRaises(ValueError) as exc:
+ client.insert_rows(table, ROWS)
+
+ self.assertIn("Could not determine schema for table", exc.exception.args[0])
+
+ def test_insert_rows_w_schema(self):
+ import datetime
+ from google.cloud._helpers import UTC
+ from google.cloud._helpers import _datetime_to_rfc3339
+ from google.cloud._helpers import _microseconds_from_datetime
+ from google.cloud.bigquery.table import SchemaField
+
+ WHEN_TS = 1437767599.006
+ WHEN = datetime.datetime.utcfromtimestamp(WHEN_TS).replace(tzinfo=UTC)
+ PATH = "projects/%s/datasets/%s/tables/%s/insertAll" % (
+ self.PROJECT,
+ self.DS_ID,
+ self.TABLE_ID,
+ )
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection({})
+ schema = [
+ SchemaField("full_name", "STRING", mode="REQUIRED"),
+ SchemaField("age", "INTEGER", mode="REQUIRED"),
+ SchemaField("joined", "TIMESTAMP", mode="NULLABLE"),
+ ]
+ ROWS = [
+ ("Phred Phlyntstone", 32, _datetime_to_rfc3339(WHEN)),
+ ("Bharney Rhubble", 33, WHEN + datetime.timedelta(seconds=1)),
+ ("Wylma Phlyntstone", 29, WHEN + datetime.timedelta(seconds=2)),
+ ("Bhettye Rhubble", 27, None),
+ ]
+
+ def _row_data(row):
+ joined = row[2]
+ if isinstance(row[2], datetime.datetime):
+ joined = _microseconds_from_datetime(joined) * 1e-6
+ return {"full_name": row[0], "age": str(row[1]), "joined": joined}
+
+ SENT = {
+ "rows": [
+ {"json": _row_data(row), "insertId": str(i)}
+ for i, row in enumerate(ROWS)
+ ]
+ }
+
+ with mock.patch("uuid.uuid4", side_effect=map(str, range(len(ROWS)))):
+ # Test with using string IDs for the table.
+ errors = client.insert_rows(
+ "{}.{}".format(self.DS_ID, self.TABLE_ID), ROWS, selected_fields=schema
+ )
+
+ self.assertEqual(len(errors), 0)
+ conn.api_request.assert_called_once()
+ _, req = conn.api_request.call_args
+ self.assertEqual(req["method"], "POST")
+ self.assertEqual(req["path"], "/%s" % PATH)
+ self.assertEqual(req["data"], SENT)
+
+ def test_insert_rows_w_list_of_dictionaries(self):
+ import datetime
+ from google.cloud._helpers import UTC
+ from google.cloud._helpers import _datetime_to_rfc3339
+ from google.cloud._helpers import _microseconds_from_datetime
+ from google.cloud.bigquery.table import Table, SchemaField
+
+ WHEN_TS = 1437767599.006
+ WHEN = datetime.datetime.utcfromtimestamp(WHEN_TS).replace(tzinfo=UTC)
+ PATH = "projects/%s/datasets/%s/tables/%s/insertAll" % (
+ self.PROJECT,
+ self.DS_ID,
+ self.TABLE_ID,
+ )
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection({})
+ schema = [
+ SchemaField("full_name", "STRING", mode="REQUIRED"),
+ SchemaField("age", "INTEGER", mode="REQUIRED"),
+ SchemaField("joined", "TIMESTAMP", mode="NULLABLE"),
+ ]
+ table = Table(self.TABLE_REF, schema=schema)
+ ROWS = [
+ {
+ "full_name": "Phred Phlyntstone",
+ "age": 32,
+ "joined": _datetime_to_rfc3339(WHEN),
+ },
+ {
+ "full_name": "Bharney Rhubble",
+ "age": 33,
+ "joined": WHEN + datetime.timedelta(seconds=1),
+ },
+ {
+ "full_name": "Wylma Phlyntstone",
+ "age": 29,
+ "joined": WHEN + datetime.timedelta(seconds=2),
+ },
+ {"full_name": "Bhettye Rhubble", "age": 27, "joined": None},
+ ]
+
+ def _row_data(row):
+ joined = row["joined"]
+ if isinstance(joined, datetime.datetime):
+ row["joined"] = _microseconds_from_datetime(joined) * 1e-6
+ row["age"] = str(row["age"])
+ return row
+
+ SENT = {
+ "rows": [
+ {"json": _row_data(row), "insertId": str(i)}
+ for i, row in enumerate(ROWS)
+ ]
+ }
+
+ with mock.patch("uuid.uuid4", side_effect=map(str, range(len(ROWS)))):
+ errors = client.insert_rows(table, ROWS)
+
+ self.assertEqual(len(errors), 0)
+ conn.api_request.assert_called_once_with(
+ method="POST", path="/%s" % PATH, data=SENT
+ )
+
+ def test_insert_rows_w_list_of_Rows(self):
+ from google.cloud.bigquery.table import Table
+ from google.cloud.bigquery.table import SchemaField
+ from google.cloud.bigquery.table import Row
+
+ PATH = "projects/%s/datasets/%s/tables/%s/insertAll" % (
+ self.PROJECT,
+ self.DS_ID,
+ self.TABLE_ID,
+ )
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection({})
+ schema = [
+ SchemaField("full_name", "STRING", mode="REQUIRED"),
+ SchemaField("age", "INTEGER", mode="REQUIRED"),
+ ]
+ table = Table(self.TABLE_REF, schema=schema)
+ f2i = {"full_name": 0, "age": 1}
+ ROWS = [
+ Row(("Phred Phlyntstone", 32), f2i),
+ Row(("Bharney Rhubble", 33), f2i),
+ Row(("Wylma Phlyntstone", 29), f2i),
+ Row(("Bhettye Rhubble", 27), f2i),
+ ]
+
+ def _row_data(row):
+ return {"full_name": row[0], "age": str(row[1])}
+
+ SENT = {
+ "rows": [
+ {"json": _row_data(row), "insertId": str(i)}
+ for i, row in enumerate(ROWS)
+ ]
+ }
+
+ with mock.patch("uuid.uuid4", side_effect=map(str, range(len(ROWS)))):
+ errors = client.insert_rows(table, ROWS)
+
+ self.assertEqual(len(errors), 0)
+ conn.api_request.assert_called_once_with(
+ method="POST", path="/%s" % PATH, data=SENT
+ )
+
+ def test_insert_rows_w_skip_invalid_and_ignore_unknown(self):
+ from google.cloud.bigquery.table import Table, SchemaField
+
+ PATH = "projects/%s/datasets/%s/tables/%s/insertAll" % (
+ self.PROJECT,
+ self.DS_ID,
+ self.TABLE_ID,
+ )
+ RESPONSE = {
+ "insertErrors": [
+ {
+ "index": 1,
+ "errors": [
+ {
+ "reason": "REASON",
+ "location": "LOCATION",
+ "debugInfo": "INFO",
+ "message": "MESSAGE",
+ }
+ ],
+ }
+ ]
+ }
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection(RESPONSE)
+ schema = [
+ SchemaField("full_name", "STRING", mode="REQUIRED"),
+ SchemaField("age", "INTEGER", mode="REQUIRED"),
+ SchemaField("voter", "BOOLEAN", mode="NULLABLE"),
+ ]
+ table = Table(self.TABLE_REF, schema=schema)
+ ROWS = [
+ ("Phred Phlyntstone", 32, True),
+ ("Bharney Rhubble", 33, False),
+ ("Wylma Phlyntstone", 29, True),
+ ("Bhettye Rhubble", 27, True),
+ ]
+
+ def _row_data(row):
+ return {
+ "full_name": row[0],
+ "age": str(row[1]),
+ "voter": row[2] and "true" or "false",
+ }
+
+ SENT = {
+ "skipInvalidRows": True,
+ "ignoreUnknownValues": True,
+ "templateSuffix": "20160303",
+ "rows": [
+ {"insertId": index, "json": _row_data(row)}
+ for index, row in enumerate(ROWS)
+ ],
+ }
+
+ errors = client.insert_rows(
+ table,
+ ROWS,
+ row_ids=[index for index, _ in enumerate(ROWS)],
+ skip_invalid_rows=True,
+ ignore_unknown_values=True,
+ template_suffix="20160303",
+ )
+
+ self.assertEqual(len(errors), 1)
+ self.assertEqual(errors[0]["index"], 1)
+ self.assertEqual(len(errors[0]["errors"]), 1)
+ self.assertEqual(
+ errors[0]["errors"][0], RESPONSE["insertErrors"][0]["errors"][0]
+ )
+ conn.api_request.assert_called_once_with(
+ method="POST", path="/%s" % PATH, data=SENT
+ )
+
+ def test_insert_rows_w_repeated_fields(self):
+ from google.cloud.bigquery.table import Table, SchemaField
+
+ PATH = "projects/%s/datasets/%s/tables/%s/insertAll" % (
+ self.PROJECT,
+ self.DS_ID,
+ self.TABLE_ID,
+ )
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection({})
+ color = SchemaField("color", "STRING", mode="REPEATED")
+ items = SchemaField("items", "INTEGER", mode="REPEATED")
+ score = SchemaField("score", "INTEGER")
+ times = SchemaField("times", "TIMESTAMP", mode="REPEATED")
+ distances = SchemaField("distances", "FLOAT", mode="REPEATED")
+ structs = SchemaField(
+ "structs", "RECORD", mode="REPEATED", fields=[score, times, distances]
+ )
+ table = Table(self.TABLE_REF, schema=[color, items, structs])
+ ROWS = [
+ (
+ ["red", "green"],
+ [1, 2],
+ [
+ (
+ 12,
+ [
+ datetime.datetime(2018, 12, 1, 12, 0, 0, tzinfo=pytz.utc),
+ datetime.datetime(2018, 12, 1, 13, 0, 0, tzinfo=pytz.utc),
+ ],
+ [1.25, 2.5],
+ ),
+ {
+ "score": 13,
+ "times": [
+ datetime.datetime(2018, 12, 2, 12, 0, 0, tzinfo=pytz.utc),
+ datetime.datetime(2018, 12, 2, 13, 0, 0, tzinfo=pytz.utc),
+ ],
+ "distances": [-1.25, -2.5],
+ },
+ ],
+ ),
+ {"color": None, "items": [], "structs": [(None, [], [3.5])]},
+ ]
+
+ SENT = {
+ "rows": [
+ {
+ "json": {
+ "color": ["red", "green"],
+ "items": ["1", "2"],
+ "structs": [
+ {
+ "score": "12",
+ "times": [
+ 1543665600.0, # 2018-12-01 12:00 UTC
+ 1543669200.0, # 2018-12-01 13:00 UTC
+ ],
+ "distances": [1.25, 2.5],
+ },
+ {
+ "score": "13",
+ "times": [
+ 1543752000.0, # 2018-12-02 12:00 UTC
+ 1543755600.0, # 2018-12-02 13:00 UTC
+ ],
+ "distances": [-1.25, -2.5],
+ },
+ ],
+ },
+ "insertId": "0",
+ },
+ {
+ "json": {
+ "color": None,
+ "items": [],
+ "structs": [{"score": None, "times": [], "distances": [3.5]}],
+ },
+ "insertId": "1",
+ },
+ ]
+ }
+
+ with mock.patch("uuid.uuid4", side_effect=map(str, range(len(ROWS)))):
+ errors = client.insert_rows(table, ROWS)
+
+ self.assertEqual(len(errors), 0)
+ conn.api_request.assert_called_once_with(
+ method="POST", path="/%s" % PATH, data=SENT
+ )
+
+ def test_insert_rows_w_record_schema(self):
+ from google.cloud.bigquery.table import SchemaField
+
+ PATH = "projects/%s/datasets/%s/tables/%s/insertAll" % (
+ self.PROJECT,
+ self.DS_ID,
+ self.TABLE_ID,
+ )
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection({})
+ full_name = SchemaField("full_name", "STRING", mode="REQUIRED")
+ area_code = SchemaField("area_code", "STRING", "REQUIRED")
+ local_number = SchemaField("local_number", "STRING", "REQUIRED")
+ rank = SchemaField("rank", "INTEGER", "REQUIRED")
+ phone = SchemaField(
+ "phone", "RECORD", mode="NULLABLE", fields=[area_code, local_number, rank]
+ )
+ ROWS = [
+ (
+ "Phred Phlyntstone",
+ {"area_code": "800", "local_number": "555-1212", "rank": 1},
+ ),
+ ("Bharney Rhubble", ("877", "768-5309", 2)),
+ ("Wylma Phlyntstone", None),
+ ]
+
+ SENT = {
+ "rows": [
+ {
+ "json": {
+ "full_name": "Phred Phlyntstone",
+ "phone": {
+ "area_code": "800",
+ "local_number": "555-1212",
+ "rank": "1",
+ },
+ },
+ "insertId": "0",
+ },
+ {
+ "json": {
+ "full_name": "Bharney Rhubble",
+ "phone": {
+ "area_code": "877",
+ "local_number": "768-5309",
+ "rank": "2",
+ },
+ },
+ "insertId": "1",
+ },
+ {
+ "json": {"full_name": "Wylma Phlyntstone", "phone": None},
+ "insertId": "2",
+ },
+ ]
+ }
+
+ with mock.patch("uuid.uuid4", side_effect=map(str, range(len(ROWS)))):
+ errors = client.insert_rows(
+ self.TABLE_REF, ROWS, selected_fields=[full_name, phone]
+ )
+
+ self.assertEqual(len(errors), 0)
+ conn.api_request.assert_called_once_with(
+ method="POST", path="/%s" % PATH, data=SENT
+ )
+
+ def test_insert_rows_errors(self):
+ from google.cloud.bigquery.table import Table
+
+ ROWS = [
+ ("Phred Phlyntstone", 32, True),
+ ("Bharney Rhubble", 33, False),
+ ("Wylma Phlyntstone", 29, True),
+ ("Bhettye Rhubble", 27, True),
+ ]
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+
+ # table ref with no selected fields
+ with self.assertRaises(ValueError):
+ client.insert_rows(self.TABLE_REF, ROWS)
+
+ # table with no schema
+ with self.assertRaises(ValueError):
+ client.insert_rows(Table(self.TABLE_REF), ROWS)
+
+ # neither Table nor tableReference
+ with self.assertRaises(TypeError):
+ client.insert_rows(1, ROWS)
+
+ def test_insert_rows_w_numeric(self):
+ from google.cloud.bigquery import table
+
+ project = "PROJECT"
+ ds_id = "DS_ID"
+ table_id = "TABLE_ID"
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=project, credentials=creds, _http=http)
+ conn = client._connection = make_connection({})
+ table_ref = DatasetReference(project, ds_id).table(table_id)
+ schema = [
+ table.SchemaField("account", "STRING"),
+ table.SchemaField("balance", "NUMERIC"),
+ ]
+ insert_table = table.Table(table_ref, schema=schema)
+ rows = [
+ ("Savings", decimal.Decimal("23.47")),
+ ("Checking", decimal.Decimal("1.98")),
+ ("Mortgage", decimal.Decimal("-12345678909.87654321")),
+ ]
+
+ with mock.patch("uuid.uuid4", side_effect=map(str, range(len(rows)))):
+ errors = client.insert_rows(insert_table, rows)
+
+ self.assertEqual(len(errors), 0)
+ rows_json = [
+ {"account": "Savings", "balance": "23.47"},
+ {"account": "Checking", "balance": "1.98"},
+ {"account": "Mortgage", "balance": "-12345678909.87654321"},
+ ]
+ sent = {
+ "rows": [
+ {"json": row, "insertId": str(i)} for i, row in enumerate(rows_json)
+ ]
+ }
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path="/projects/{}/datasets/{}/tables/{}/insertAll".format(
+ project, ds_id, table_id
+ ),
+ data=sent,
+ )
+
+ def test_insert_rows_json(self):
+ from google.cloud.bigquery.table import Table, SchemaField
+ from google.cloud.bigquery.dataset import DatasetReference
+
+ PROJECT = "PROJECT"
+ DS_ID = "DS_ID"
+ TABLE_ID = "TABLE_ID"
+ PATH = "projects/%s/datasets/%s/tables/%s/insertAll" % (
+ PROJECT,
+ DS_ID,
+ TABLE_ID,
+ )
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection({})
+ table_ref = DatasetReference(PROJECT, DS_ID).table(TABLE_ID)
+ schema = [
+ SchemaField("full_name", "STRING", mode="REQUIRED"),
+ SchemaField("age", "INTEGER", mode="REQUIRED"),
+ SchemaField("joined", "TIMESTAMP", mode="NULLABLE"),
+ ]
+ table = Table(table_ref, schema=schema)
+ ROWS = [
+ {
+ "full_name": "Phred Phlyntstone",
+ "age": "32",
+ "joined": "2015-07-24T19:53:19.006000Z",
+ },
+ {"full_name": "Bharney Rhubble", "age": "33", "joined": 1437767600.006},
+ {"full_name": "Wylma Phlyntstone", "age": "29", "joined": 1437767601.006},
+ {"full_name": "Bhettye Rhubble", "age": "27", "joined": None},
+ ]
+
+ SENT = {
+ "rows": [{"json": row, "insertId": str(i)} for i, row in enumerate(ROWS)]
+ }
+
+ with mock.patch("uuid.uuid4", side_effect=map(str, range(len(ROWS)))):
+ errors = client.insert_rows_json(table, ROWS)
+
+ self.assertEqual(len(errors), 0)
+ conn.api_request.assert_called_once_with(
+ method="POST", path="/%s" % PATH, data=SENT
+ )
+
+ def test_insert_rows_json_with_string_id(self):
+ rows = [{"col1": "val1"}]
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(
+ project="default-project", credentials=creds, _http=http
+ )
+ conn = client._connection = make_connection({})
+
+ with mock.patch("uuid.uuid4", side_effect=map(str, range(len(rows)))):
+ errors = client.insert_rows_json("proj.dset.tbl", rows)
+
+ self.assertEqual(len(errors), 0)
+ expected = {
+ "rows": [{"json": row, "insertId": str(i)} for i, row in enumerate(rows)]
+ }
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path="/projects/proj/datasets/dset/tables/tbl/insertAll",
+ data=expected,
+ )
+
+ def test_list_partitions(self):
+ from google.cloud.bigquery.table import Table
+
+ rows = 3
+ meta_info = _make_list_partitons_meta_info(
+ self.PROJECT, self.DS_ID, self.TABLE_ID, rows
+ )
+
+ data = {
+ "totalRows": str(rows),
+ "rows": [
+ {"f": [{"v": "20180101"}]},
+ {"f": [{"v": "20180102"}]},
+ {"f": [{"v": "20180103"}]},
+ ],
+ }
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ client._connection = make_connection(meta_info, data)
+ table = Table(self.TABLE_REF)
+
+ partition_list = client.list_partitions(table)
+ self.assertEqual(len(partition_list), rows)
+ self.assertIn("20180102", partition_list)
+
+ def test_list_partitions_with_string_id(self):
+ meta_info = _make_list_partitons_meta_info(
+ self.PROJECT, self.DS_ID, self.TABLE_ID, 0
+ )
+
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ client._connection = make_connection(meta_info, {})
+
+ partition_list = client.list_partitions(
+ "{}.{}".format(self.DS_ID, self.TABLE_ID)
+ )
+
+ self.assertEqual(len(partition_list), 0)
+
+ def test_list_rows(self):
+ import datetime
+ from google.cloud._helpers import UTC
+ from google.cloud.bigquery.table import Table
+ from google.cloud.bigquery.table import SchemaField
+ from google.cloud.bigquery.table import Row
+
+ PATH = "projects/%s/datasets/%s/tables/%s/data" % (
+ self.PROJECT,
+ self.DS_ID,
+ self.TABLE_ID,
+ )
+ WHEN_TS = 1437767599.006
+ WHEN = datetime.datetime.utcfromtimestamp(WHEN_TS).replace(tzinfo=UTC)
+ WHEN_1 = WHEN + datetime.timedelta(seconds=1)
+ WHEN_2 = WHEN + datetime.timedelta(seconds=2)
+ ROWS = 1234
+ TOKEN = "TOKEN"
+
+ def _bigquery_timestamp_float_repr(ts_float):
+ # Preserve microsecond precision for E+09 timestamps
+ return "%0.15E" % (ts_float,)
+
+ DATA = {
+ "totalRows": str(ROWS),
+ "pageToken": TOKEN,
+ "rows": [
+ {
+ "f": [
+ {"v": "Phred Phlyntstone"},
+ {"v": "32"},
+ {"v": _bigquery_timestamp_float_repr(WHEN_TS)},
+ ]
+ },
+ {
+ "f": [
+ {"v": "Bharney Rhubble"},
+ {"v": "33"},
+ {"v": _bigquery_timestamp_float_repr(WHEN_TS + 1)},
+ ]
+ },
+ {
+ "f": [
+ {"v": "Wylma Phlyntstone"},
+ {"v": "29"},
+ {"v": _bigquery_timestamp_float_repr(WHEN_TS + 2)},
+ ]
+ },
+ {"f": [{"v": "Bhettye Rhubble"}, {"v": None}, {"v": None}]},
+ ],
+ }
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection(DATA, DATA)
+ full_name = SchemaField("full_name", "STRING", mode="REQUIRED")
+ age = SchemaField("age", "INTEGER", mode="NULLABLE")
+ joined = SchemaField("joined", "TIMESTAMP", mode="NULLABLE")
+ table = Table(self.TABLE_REF, schema=[full_name, age, joined])
+
+ iterator = client.list_rows(table)
+ page = six.next(iterator.pages)
+ rows = list(page)
+ total_rows = iterator.total_rows
+ page_token = iterator.next_page_token
+
+ f2i = {"full_name": 0, "age": 1, "joined": 2}
+ self.assertEqual(len(rows), 4)
+ self.assertEqual(rows[0], Row(("Phred Phlyntstone", 32, WHEN), f2i))
+ self.assertEqual(rows[1], Row(("Bharney Rhubble", 33, WHEN_1), f2i))
+ self.assertEqual(rows[2], Row(("Wylma Phlyntstone", 29, WHEN_2), f2i))
+ self.assertEqual(rows[3], Row(("Bhettye Rhubble", None, None), f2i))
+ self.assertEqual(total_rows, ROWS)
+ self.assertEqual(page_token, TOKEN)
+
+ conn.api_request.assert_called_once_with(
+ method="GET", path="/%s" % PATH, query_params={}
+ )
+
+ def test_list_rows_empty_table(self):
+ response = {"totalRows": "0", "rows": []}
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ client._connection = make_connection(response, response)
+
+ # Table that has no schema because it's an empty table.
+ rows = client.list_rows(
+ # Test with using a string for the table ID.
+ "{}.{}.{}".format(
+ self.TABLE_REF.project,
+ self.TABLE_REF.dataset_id,
+ self.TABLE_REF.table_id,
+ ),
+ selected_fields=[],
+ )
+
+ # When a table reference / string and selected_fields is provided,
+ # total_rows can't be populated until iteration starts.
+ self.assertIsNone(rows.total_rows)
+ self.assertEqual(tuple(rows), ())
+ self.assertEqual(rows.total_rows, 0)
+
+ def test_list_rows_query_params(self):
+ from google.cloud.bigquery.table import Table, SchemaField
+
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ table = Table(
+ self.TABLE_REF, schema=[SchemaField("age", "INTEGER", mode="NULLABLE")]
+ )
+ tests = [
+ ({}, {}),
+ ({"start_index": 1}, {"startIndex": 1}),
+ ({"max_results": 2}, {"maxResults": 2}),
+ ({"start_index": 1, "max_results": 2}, {"startIndex": 1, "maxResults": 2}),
+ ]
+ conn = client._connection = make_connection(*len(tests) * [{}])
+ for i, test in enumerate(tests):
+ iterator = client.list_rows(table, **test[0])
+ six.next(iterator.pages)
+ req = conn.api_request.call_args_list[i]
+ self.assertEqual(req[1]["query_params"], test[1], "for kwargs %s" % test[0])
+
+ def test_list_rows_repeated_fields(self):
+ from google.cloud.bigquery.table import SchemaField
+
+ PATH = "projects/%s/datasets/%s/tables/%s/data" % (
+ self.PROJECT,
+ self.DS_ID,
+ self.TABLE_ID,
+ )
+ ROWS = 1234
+ TOKEN = "TOKEN"
+ DATA = {
+ "totalRows": ROWS,
+ "pageToken": TOKEN,
+ "rows": [
+ {
+ "f": [
+ {"v": [{"v": "red"}, {"v": "green"}]},
+ {
+ "v": [
+ {
+ "v": {
+ "f": [
+ {"v": [{"v": "1"}, {"v": "2"}]},
+ {"v": [{"v": "3.1415"}, {"v": "1.414"}]},
+ ]
+ }
+ }
+ ]
+ },
+ ]
+ }
+ ],
+ }
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection(DATA)
+ color = SchemaField("color", "STRING", mode="REPEATED")
+ index = SchemaField("index", "INTEGER", "REPEATED")
+ score = SchemaField("score", "FLOAT", "REPEATED")
+ struct = SchemaField("struct", "RECORD", mode="REPEATED", fields=[index, score])
+
+ iterator = client.list_rows(self.TABLE_REF, selected_fields=[color, struct])
+ page = six.next(iterator.pages)
+ rows = list(page)
+ total_rows = iterator.total_rows
+ page_token = iterator.next_page_token
+
+ self.assertEqual(len(rows), 1)
+ self.assertEqual(rows[0][0], ["red", "green"])
+ self.assertEqual(rows[0][1], [{"index": [1, 2], "score": [3.1415, 1.414]}])
+ self.assertEqual(total_rows, ROWS)
+ self.assertEqual(page_token, TOKEN)
+
+ conn.api_request.assert_called_once_with(
+ method="GET",
+ path="/%s" % PATH,
+ query_params={"selectedFields": "color,struct"},
+ )
+
+ def test_list_rows_w_record_schema(self):
+ from google.cloud.bigquery.table import Table, SchemaField
+
+ PATH = "projects/%s/datasets/%s/tables/%s/data" % (
+ self.PROJECT,
+ self.DS_ID,
+ self.TABLE_ID,
+ )
+ ROWS = 1234
+ TOKEN = "TOKEN"
+ DATA = {
+ "totalRows": ROWS,
+ "pageToken": TOKEN,
+ "rows": [
+ {
+ "f": [
+ {"v": "Phred Phlyntstone"},
+ {"v": {"f": [{"v": "800"}, {"v": "555-1212"}, {"v": 1}]}},
+ ]
+ },
+ {
+ "f": [
+ {"v": "Bharney Rhubble"},
+ {"v": {"f": [{"v": "877"}, {"v": "768-5309"}, {"v": 2}]}},
+ ]
+ },
+ {"f": [{"v": "Wylma Phlyntstone"}, {"v": None}]},
+ ],
+ }
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection(DATA)
+ full_name = SchemaField("full_name", "STRING", mode="REQUIRED")
+ area_code = SchemaField("area_code", "STRING", "REQUIRED")
+ local_number = SchemaField("local_number", "STRING", "REQUIRED")
+ rank = SchemaField("rank", "INTEGER", "REQUIRED")
+ phone = SchemaField(
+ "phone", "RECORD", mode="NULLABLE", fields=[area_code, local_number, rank]
+ )
+ table = Table(self.TABLE_REF, schema=[full_name, phone])
+
+ iterator = client.list_rows(table)
+ page = six.next(iterator.pages)
+ rows = list(page)
+ total_rows = iterator.total_rows
+ page_token = iterator.next_page_token
+
+ self.assertEqual(len(rows), 3)
+ self.assertEqual(rows[0][0], "Phred Phlyntstone")
+ self.assertEqual(
+ rows[0][1], {"area_code": "800", "local_number": "555-1212", "rank": 1}
+ )
+ self.assertEqual(rows[1][0], "Bharney Rhubble")
+ self.assertEqual(
+ rows[1][1], {"area_code": "877", "local_number": "768-5309", "rank": 2}
+ )
+ self.assertEqual(rows[2][0], "Wylma Phlyntstone")
+ self.assertIsNone(rows[2][1])
+ self.assertEqual(total_rows, ROWS)
+ self.assertEqual(page_token, TOKEN)
+
+ conn.api_request.assert_called_once_with(
+ method="GET", path="/%s" % PATH, query_params={}
+ )
+
+ def test_list_rows_with_missing_schema(self):
+ from google.cloud.bigquery.table import Table, TableListItem
+
+ table_path = "/projects/{}/datasets/{}/tables/{}".format(
+ self.PROJECT, self.DS_ID, self.TABLE_ID
+ )
+ tabledata_path = "{}/data".format(table_path)
+
+ table_list_item_data = {
+ "id": "%s:%s:%s" % (self.PROJECT, self.DS_ID, self.TABLE_ID),
+ "tableReference": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.TABLE_ID,
+ },
+ }
+ table_data = copy.deepcopy(table_list_item_data)
+ # Intentionally make wrong, since total_rows can update during iteration.
+ table_data["numRows"] = 2
+ table_data["schema"] = {
+ "fields": [
+ {"name": "name", "type": "STRING"},
+ {"name": "age", "type": "INTEGER"},
+ ]
+ }
+ rows_data = {
+ "totalRows": 3,
+ "pageToken": None,
+ "rows": [
+ {"f": [{"v": "Phred Phlyntstone"}, {"v": "32"}]},
+ {"f": [{"v": "Bharney Rhubble"}, {"v": "31"}]},
+ {"f": [{"v": "Wylma Phlyntstone"}, {"v": None}]},
+ ],
+ }
+
+ creds = _make_credentials()
+ http = object()
+
+ schemaless_tables = (
+ "{}.{}".format(self.DS_ID, self.TABLE_ID),
+ self.TABLE_REF,
+ Table(self.TABLE_REF),
+ TableListItem(table_list_item_data),
+ )
+
+ for table in schemaless_tables:
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+ conn = client._connection = make_connection(table_data, rows_data)
+
+ row_iter = client.list_rows(table)
+
+ conn.api_request.assert_called_once_with(method="GET", path=table_path)
+ conn.api_request.reset_mock()
+ self.assertEqual(row_iter.total_rows, 2, msg=repr(table))
+
+ rows = list(row_iter)
+ conn.api_request.assert_called_once_with(
+ method="GET", path=tabledata_path, query_params={}
+ )
+ self.assertEqual(row_iter.total_rows, 3, msg=repr(table))
+ self.assertEqual(rows[0].name, "Phred Phlyntstone", msg=repr(table))
+ self.assertEqual(rows[1].age, 31, msg=repr(table))
+ self.assertIsNone(rows[2].age, msg=repr(table))
+
+ def test_list_rows_error(self):
+ creds = _make_credentials()
+ http = object()
+ client = self._make_one(project=self.PROJECT, credentials=creds, _http=http)
+
+ # neither Table nor tableReference
+ with self.assertRaises(TypeError):
+ client.list_rows(1)
+
+
+class Test_make_job_id(unittest.TestCase):
+ def _call_fut(self, job_id, prefix=None):
+ from google.cloud.bigquery.client import _make_job_id
+
+ return _make_job_id(job_id, prefix=prefix)
+
+ def test__make_job_id_wo_suffix(self):
+ job_id = self._call_fut("job_id")
+
+ self.assertEqual(job_id, "job_id")
+
+ def test__make_job_id_w_suffix(self):
+ with mock.patch("uuid.uuid4", side_effect=["212345"]):
+ job_id = self._call_fut(None, prefix="job_id")
+
+ self.assertEqual(job_id, "job_id212345")
+
+ def test__make_random_job_id(self):
+ with mock.patch("uuid.uuid4", side_effect=["212345"]):
+ job_id = self._call_fut(None)
+
+ self.assertEqual(job_id, "212345")
+
+ def test__make_job_id_w_job_id_overrides_prefix(self):
+ job_id = self._call_fut("job_id", prefix="unused_prefix")
+
+ self.assertEqual(job_id, "job_id")
+
+
+class TestClientUpload(object):
+ # NOTE: This is a "partner" to `TestClient` meant to test some of the
+ # "load_table_from_file" portions of `Client`. It also uses
+ # `pytest`-style tests rather than `unittest`-style.
+ from google.cloud.bigquery.job import SourceFormat
+
+ TABLE_REF = DatasetReference("project_id", "test_dataset").table("test_table")
+
+ LOCATION = "us-central"
+
+ @staticmethod
+ def _make_client(transport=None, location=None):
+ from google.cloud.bigquery import _http
+ from google.cloud.bigquery import client
+
+ cl = client.Client(
+ project="project_id",
+ credentials=_make_credentials(),
+ _http=transport,
+ location=location,
+ )
+ cl._connection = mock.create_autospec(_http.Connection, instance=True)
+ return cl
+
+ @staticmethod
+ def _make_response(status_code, content="", headers={}):
+ """Make a mock HTTP response."""
+ import requests
+
+ response = requests.Response()
+ response.request = requests.Request("POST", "http://example.com").prepare()
+ response._content = content.encode("utf-8")
+ response.headers.update(headers)
+ response.status_code = status_code
+ return response
+
+ @classmethod
+ def _make_do_upload_patch(cls, client, method, resource={}, side_effect=None):
+ """Patches the low-level upload helpers."""
+ if side_effect is None:
+ side_effect = [
+ cls._make_response(
+ http_client.OK,
+ json.dumps(resource),
+ {"Content-Type": "application/json"},
+ )
+ ]
+ return mock.patch.object(client, method, side_effect=side_effect, autospec=True)
+
+ EXPECTED_CONFIGURATION = {
+ "jobReference": {"projectId": "project_id", "jobId": "job_id"},
+ "configuration": {
+ "load": {
+ "sourceFormat": SourceFormat.CSV,
+ "destinationTable": {
+ "projectId": "project_id",
+ "datasetId": "test_dataset",
+ "tableId": "test_table",
+ },
+ }
+ },
+ }
+
+ @staticmethod
+ def _make_file_obj():
+ return io.BytesIO(b"hello, is it me you're looking for?")
+
+ def _make_gzip_file_obj(self, writable):
+ if writable:
+ return gzip.GzipFile(mode="w", fileobj=io.BytesIO())
+ else:
+ return gzip.GzipFile(mode="r", fileobj=self._make_file_obj())
+
+ @staticmethod
+ def _make_config():
+ from google.cloud.bigquery.job import LoadJobConfig
+ from google.cloud.bigquery.job import SourceFormat
+
+ config = LoadJobConfig()
+ config.source_format = SourceFormat.CSV
+ return config
+
+ # High-level tests
+
+ def test_load_table_from_file_resumable(self):
+ from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES
+
+ client = self._make_client()
+ file_obj = self._make_file_obj()
+
+ do_upload_patch = self._make_do_upload_patch(
+ client, "_do_resumable_upload", self.EXPECTED_CONFIGURATION
+ )
+ with do_upload_patch as do_upload:
+ client.load_table_from_file(
+ file_obj,
+ self.TABLE_REF,
+ job_id="job_id",
+ job_config=self._make_config(),
+ )
+
+ do_upload.assert_called_once_with(
+ file_obj, self.EXPECTED_CONFIGURATION, _DEFAULT_NUM_RETRIES
+ )
+
+ def test_load_table_from_file_w_explicit_project(self):
+ from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES
+
+ client = self._make_client()
+ file_obj = self._make_file_obj()
+
+ do_upload_patch = self._make_do_upload_patch(
+ client, "_do_resumable_upload", self.EXPECTED_CONFIGURATION
+ )
+ with do_upload_patch as do_upload:
+ client.load_table_from_file(
+ file_obj,
+ self.TABLE_REF,
+ job_id="job_id",
+ project="other-project",
+ location=self.LOCATION,
+ job_config=self._make_config(),
+ )
+
+ expected_resource = copy.deepcopy(self.EXPECTED_CONFIGURATION)
+ expected_resource["jobReference"]["location"] = self.LOCATION
+ expected_resource["jobReference"]["projectId"] = "other-project"
+ do_upload.assert_called_once_with(
+ file_obj, expected_resource, _DEFAULT_NUM_RETRIES
+ )
+
+ def test_load_table_from_file_w_client_location(self):
+ from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES
+
+ client = self._make_client(location=self.LOCATION)
+ file_obj = self._make_file_obj()
+
+ do_upload_patch = self._make_do_upload_patch(
+ client, "_do_resumable_upload", self.EXPECTED_CONFIGURATION
+ )
+ with do_upload_patch as do_upload:
+ client.load_table_from_file(
+ file_obj,
+ # Test with string for table ID.
+ "{}.{}.{}".format(
+ self.TABLE_REF.project,
+ self.TABLE_REF.dataset_id,
+ self.TABLE_REF.table_id,
+ ),
+ job_id="job_id",
+ project="other-project",
+ job_config=self._make_config(),
+ )
+
+ expected_resource = copy.deepcopy(self.EXPECTED_CONFIGURATION)
+ expected_resource["jobReference"]["location"] = self.LOCATION
+ expected_resource["jobReference"]["projectId"] = "other-project"
+ do_upload.assert_called_once_with(
+ file_obj, expected_resource, _DEFAULT_NUM_RETRIES
+ )
+
+ def test_load_table_from_file_resumable_metadata(self):
+ from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES
+ from google.cloud.bigquery.job import CreateDisposition
+ from google.cloud.bigquery.job import WriteDisposition
+
+ client = self._make_client()
+ file_obj = self._make_file_obj()
+
+ config = self._make_config()
+ config.allow_jagged_rows = False
+ config.allow_quoted_newlines = False
+ config.create_disposition = CreateDisposition.CREATE_IF_NEEDED
+ config.encoding = "utf8"
+ config.field_delimiter = ","
+ config.ignore_unknown_values = False
+ config.max_bad_records = 0
+ config.quote_character = '"'
+ config.skip_leading_rows = 1
+ config.write_disposition = WriteDisposition.WRITE_APPEND
+ config.null_marker = r"\N"
+
+ expected_config = {
+ "jobReference": {"projectId": "project_id", "jobId": "job_id"},
+ "configuration": {
+ "load": {
+ "destinationTable": {
+ "projectId": self.TABLE_REF.project,
+ "datasetId": self.TABLE_REF.dataset_id,
+ "tableId": self.TABLE_REF.table_id,
+ },
+ "sourceFormat": config.source_format,
+ "allowJaggedRows": config.allow_jagged_rows,
+ "allowQuotedNewlines": config.allow_quoted_newlines,
+ "createDisposition": config.create_disposition,
+ "encoding": config.encoding,
+ "fieldDelimiter": config.field_delimiter,
+ "ignoreUnknownValues": config.ignore_unknown_values,
+ "maxBadRecords": config.max_bad_records,
+ "quote": config.quote_character,
+ "skipLeadingRows": str(config.skip_leading_rows),
+ "writeDisposition": config.write_disposition,
+ "nullMarker": config.null_marker,
+ }
+ },
+ }
+
+ do_upload_patch = self._make_do_upload_patch(
+ client, "_do_resumable_upload", expected_config
+ )
+ with do_upload_patch as do_upload:
+ client.load_table_from_file(
+ file_obj, self.TABLE_REF, job_id="job_id", job_config=config
+ )
+
+ do_upload.assert_called_once_with(
+ file_obj, expected_config, _DEFAULT_NUM_RETRIES
+ )
+
+ def test_load_table_from_file_multipart(self):
+ from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES
+
+ client = self._make_client()
+ file_obj = self._make_file_obj()
+ file_obj_size = 10
+ config = self._make_config()
+
+ do_upload_patch = self._make_do_upload_patch(
+ client, "_do_multipart_upload", self.EXPECTED_CONFIGURATION
+ )
+ with do_upload_patch as do_upload:
+ client.load_table_from_file(
+ file_obj,
+ self.TABLE_REF,
+ job_id="job_id",
+ job_config=config,
+ size=file_obj_size,
+ )
+
+ do_upload.assert_called_once_with(
+ file_obj, self.EXPECTED_CONFIGURATION, file_obj_size, _DEFAULT_NUM_RETRIES
+ )
+
+ def test_load_table_from_file_with_retries(self):
+ client = self._make_client()
+ file_obj = self._make_file_obj()
+ num_retries = 20
+
+ do_upload_patch = self._make_do_upload_patch(
+ client, "_do_resumable_upload", self.EXPECTED_CONFIGURATION
+ )
+ with do_upload_patch as do_upload:
+ client.load_table_from_file(
+ file_obj,
+ self.TABLE_REF,
+ num_retries=num_retries,
+ job_id="job_id",
+ job_config=self._make_config(),
+ )
+
+ do_upload.assert_called_once_with(
+ file_obj, self.EXPECTED_CONFIGURATION, num_retries
+ )
+
+ def test_load_table_from_file_with_rewind(self):
+ client = self._make_client()
+ file_obj = self._make_file_obj()
+ file_obj.seek(2)
+
+ with self._make_do_upload_patch(
+ client, "_do_resumable_upload", self.EXPECTED_CONFIGURATION
+ ):
+ client.load_table_from_file(file_obj, self.TABLE_REF, rewind=True)
+
+ assert file_obj.tell() == 0
+
+ def test_load_table_from_file_with_readable_gzip(self):
+ from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES
+
+ client = self._make_client()
+ gzip_file = self._make_gzip_file_obj(writable=False)
+
+ do_upload_patch = self._make_do_upload_patch(
+ client, "_do_resumable_upload", self.EXPECTED_CONFIGURATION
+ )
+ with do_upload_patch as do_upload:
+ client.load_table_from_file(
+ gzip_file,
+ self.TABLE_REF,
+ job_id="job_id",
+ job_config=self._make_config(),
+ )
+
+ do_upload.assert_called_once_with(
+ gzip_file, self.EXPECTED_CONFIGURATION, _DEFAULT_NUM_RETRIES
+ )
+
+ def test_load_table_from_file_with_writable_gzip(self):
+ client = self._make_client()
+ gzip_file = self._make_gzip_file_obj(writable=True)
+
+ with pytest.raises(ValueError):
+ client.load_table_from_file(
+ gzip_file,
+ self.TABLE_REF,
+ job_id="job_id",
+ job_config=self._make_config(),
+ )
+
+ def test_load_table_from_file_failure(self):
+ from google.resumable_media import InvalidResponse
+ from google.cloud import exceptions
+
+ client = self._make_client()
+ file_obj = self._make_file_obj()
+
+ response = self._make_response(
+ content="Someone is already in this spot.", status_code=http_client.CONFLICT
+ )
+
+ do_upload_patch = self._make_do_upload_patch(
+ client, "_do_resumable_upload", side_effect=InvalidResponse(response)
+ )
+
+ with do_upload_patch, pytest.raises(exceptions.Conflict) as exc_info:
+ client.load_table_from_file(file_obj, self.TABLE_REF, rewind=True)
+
+ assert response.text in exc_info.value.message
+ assert exc_info.value.errors == []
+
+ def test_load_table_from_file_bad_mode(self):
+ client = self._make_client()
+ file_obj = mock.Mock(spec=["mode"])
+ file_obj.mode = "x"
+
+ with pytest.raises(ValueError):
+ client.load_table_from_file(file_obj, self.TABLE_REF)
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ def test_load_table_from_dataframe(self):
+ from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES
+ from google.cloud.bigquery import job
+
+ client = self._make_client()
+ records = [{"id": 1, "age": 100}, {"id": 2, "age": 60}]
+ dataframe = pandas.DataFrame(records)
+
+ load_patch = mock.patch(
+ "google.cloud.bigquery.client.Client.load_table_from_file", autospec=True
+ )
+ with load_patch as load_table_from_file:
+ client.load_table_from_dataframe(dataframe, self.TABLE_REF)
+
+ load_table_from_file.assert_called_once_with(
+ client,
+ mock.ANY,
+ self.TABLE_REF,
+ num_retries=_DEFAULT_NUM_RETRIES,
+ rewind=True,
+ job_id=mock.ANY,
+ job_id_prefix=None,
+ location=None,
+ project=None,
+ job_config=mock.ANY,
+ )
+
+ sent_file = load_table_from_file.mock_calls[0][1][1]
+ assert sent_file.closed
+
+ sent_config = load_table_from_file.mock_calls[0][2]["job_config"]
+ assert sent_config.source_format == job.SourceFormat.PARQUET
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ def test_load_table_from_dataframe_w_client_location(self):
+ from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES
+ from google.cloud.bigquery import job
+
+ client = self._make_client(location=self.LOCATION)
+ records = [{"id": 1, "age": 100}, {"id": 2, "age": 60}]
+ dataframe = pandas.DataFrame(records)
+
+ load_patch = mock.patch(
+ "google.cloud.bigquery.client.Client.load_table_from_file", autospec=True
+ )
+ with load_patch as load_table_from_file:
+ client.load_table_from_dataframe(dataframe, self.TABLE_REF)
+
+ load_table_from_file.assert_called_once_with(
+ client,
+ mock.ANY,
+ self.TABLE_REF,
+ num_retries=_DEFAULT_NUM_RETRIES,
+ rewind=True,
+ job_id=mock.ANY,
+ job_id_prefix=None,
+ location=self.LOCATION,
+ project=None,
+ job_config=mock.ANY,
+ )
+
+ sent_file = load_table_from_file.mock_calls[0][1][1]
+ assert sent_file.closed
+
+ sent_config = load_table_from_file.mock_calls[0][2]["job_config"]
+ assert sent_config.source_format == job.SourceFormat.PARQUET
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ def test_load_table_from_dataframe_w_custom_job_config(self):
+ from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES
+ from google.cloud.bigquery import job
+
+ client = self._make_client()
+ records = [{"id": 1, "age": 100}, {"id": 2, "age": 60}]
+ dataframe = pandas.DataFrame(records)
+ job_config = job.LoadJobConfig()
+
+ load_patch = mock.patch(
+ "google.cloud.bigquery.client.Client.load_table_from_file", autospec=True
+ )
+ with load_patch as load_table_from_file:
+ client.load_table_from_dataframe(
+ dataframe, self.TABLE_REF, job_config=job_config, location=self.LOCATION
+ )
+
+ load_table_from_file.assert_called_once_with(
+ client,
+ mock.ANY,
+ self.TABLE_REF,
+ num_retries=_DEFAULT_NUM_RETRIES,
+ rewind=True,
+ job_id=mock.ANY,
+ job_id_prefix=None,
+ location=self.LOCATION,
+ project=None,
+ job_config=mock.ANY,
+ )
+
+ sent_config = load_table_from_file.mock_calls[0][2]["job_config"]
+ assert sent_config.source_format == job.SourceFormat.PARQUET
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ def test_load_table_from_dataframe_w_automatic_schema(self):
+ from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES
+ from google.cloud.bigquery import job
+ from google.cloud.bigquery.schema import SchemaField
+
+ client = self._make_client()
+ df_data = collections.OrderedDict(
+ [
+ ("int_col", [1, 2, 3]),
+ ("float_col", [1.0, 2.0, 3.0]),
+ ("bool_col", [True, False, True]),
+ (
+ "dt_col",
+ pandas.Series(
+ [
+ datetime.datetime(2010, 1, 2, 3, 44, 50),
+ datetime.datetime(2011, 2, 3, 14, 50, 59),
+ datetime.datetime(2012, 3, 14, 15, 16),
+ ],
+ dtype="datetime64[ns]",
+ ),
+ ),
+ (
+ "ts_col",
+ pandas.Series(
+ [
+ datetime.datetime(2010, 1, 2, 3, 44, 50),
+ datetime.datetime(2011, 2, 3, 14, 50, 59),
+ datetime.datetime(2012, 3, 14, 15, 16),
+ ],
+ dtype="datetime64[ns]",
+ ).dt.tz_localize(pytz.utc),
+ ),
+ ]
+ )
+ dataframe = pandas.DataFrame(df_data, columns=df_data.keys())
+ load_patch = mock.patch(
+ "google.cloud.bigquery.client.Client.load_table_from_file", autospec=True
+ )
+
+ with load_patch as load_table_from_file:
+ client.load_table_from_dataframe(
+ dataframe, self.TABLE_REF, location=self.LOCATION
+ )
+
+ load_table_from_file.assert_called_once_with(
+ client,
+ mock.ANY,
+ self.TABLE_REF,
+ num_retries=_DEFAULT_NUM_RETRIES,
+ rewind=True,
+ job_id=mock.ANY,
+ job_id_prefix=None,
+ location=self.LOCATION,
+ project=None,
+ job_config=mock.ANY,
+ )
+
+ sent_config = load_table_from_file.mock_calls[0][2]["job_config"]
+ assert sent_config.source_format == job.SourceFormat.PARQUET
+ assert tuple(sent_config.schema) == (
+ SchemaField("int_col", "INTEGER"),
+ SchemaField("float_col", "FLOAT"),
+ SchemaField("bool_col", "BOOLEAN"),
+ SchemaField("dt_col", "DATETIME"),
+ SchemaField("ts_col", "TIMESTAMP"),
+ )
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ def test_load_table_from_dataframe_struct_fields_error(self):
+ from google.cloud.bigquery import job
+ from google.cloud.bigquery.schema import SchemaField
+
+ client = self._make_client()
+
+ records = [{"float_column": 3.14, "struct_column": [{"foo": 1}, {"bar": -1}]}]
+ dataframe = pandas.DataFrame(data=records)
+
+ schema = [
+ SchemaField("float_column", "FLOAT"),
+ SchemaField(
+ "agg_col",
+ "RECORD",
+ fields=[SchemaField("foo", "INTEGER"), SchemaField("bar", "INTEGER")],
+ ),
+ ]
+ job_config = job.LoadJobConfig(schema=schema)
+
+ load_patch = mock.patch(
+ "google.cloud.bigquery.client.Client.load_table_from_file", autospec=True
+ )
+
+ with pytest.raises(ValueError) as exc_info, load_patch:
+ client.load_table_from_dataframe(
+ dataframe, self.TABLE_REF, job_config=job_config, location=self.LOCATION
+ )
+
+ err_msg = str(exc_info.value)
+ assert "struct" in err_msg
+ assert "not support" in err_msg
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ def test_load_table_from_dataframe_w_partial_schema(self):
+ from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES
+ from google.cloud.bigquery import job
+ from google.cloud.bigquery.schema import SchemaField
+
+ client = self._make_client()
+ df_data = collections.OrderedDict(
+ [
+ ("int_col", [1, 2, 3]),
+ ("int_as_float_col", [1.0, float("nan"), 3.0]),
+ ("float_col", [1.0, 2.0, 3.0]),
+ ("bool_col", [True, False, True]),
+ (
+ "dt_col",
+ pandas.Series(
+ [
+ datetime.datetime(2010, 1, 2, 3, 44, 50),
+ datetime.datetime(2011, 2, 3, 14, 50, 59),
+ datetime.datetime(2012, 3, 14, 15, 16),
+ ],
+ dtype="datetime64[ns]",
+ ),
+ ),
+ (
+ "ts_col",
+ pandas.Series(
+ [
+ datetime.datetime(2010, 1, 2, 3, 44, 50),
+ datetime.datetime(2011, 2, 3, 14, 50, 59),
+ datetime.datetime(2012, 3, 14, 15, 16),
+ ],
+ dtype="datetime64[ns]",
+ ).dt.tz_localize(pytz.utc),
+ ),
+ ("string_col", [u"abc", None, u"def"]),
+ ("bytes_col", [b"abc", b"def", None]),
+ ]
+ )
+ dataframe = pandas.DataFrame(df_data, columns=df_data.keys())
+ load_patch = mock.patch(
+ "google.cloud.bigquery.client.Client.load_table_from_file", autospec=True
+ )
+
+ schema = (
+ SchemaField("int_as_float_col", "INTEGER"),
+ SchemaField("string_col", "STRING"),
+ SchemaField("bytes_col", "BYTES"),
+ )
+ job_config = job.LoadJobConfig(schema=schema)
+ with load_patch as load_table_from_file:
+ client.load_table_from_dataframe(
+ dataframe, self.TABLE_REF, job_config=job_config, location=self.LOCATION
+ )
+
+ load_table_from_file.assert_called_once_with(
+ client,
+ mock.ANY,
+ self.TABLE_REF,
+ num_retries=_DEFAULT_NUM_RETRIES,
+ rewind=True,
+ job_id=mock.ANY,
+ job_id_prefix=None,
+ location=self.LOCATION,
+ project=None,
+ job_config=mock.ANY,
+ )
+
+ sent_config = load_table_from_file.mock_calls[0][2]["job_config"]
+ assert sent_config.source_format == job.SourceFormat.PARQUET
+ assert tuple(sent_config.schema) == (
+ SchemaField("int_col", "INTEGER"),
+ SchemaField("int_as_float_col", "INTEGER"),
+ SchemaField("float_col", "FLOAT"),
+ SchemaField("bool_col", "BOOLEAN"),
+ SchemaField("dt_col", "DATETIME"),
+ SchemaField("ts_col", "TIMESTAMP"),
+ SchemaField("string_col", "STRING"),
+ SchemaField("bytes_col", "BYTES"),
+ )
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ def test_load_table_from_dataframe_w_partial_schema_extra_types(self):
+ from google.cloud.bigquery import job
+ from google.cloud.bigquery.schema import SchemaField
+
+ client = self._make_client()
+ df_data = collections.OrderedDict(
+ [
+ ("int_col", [1, 2, 3]),
+ ("int_as_float_col", [1.0, float("nan"), 3.0]),
+ ("string_col", [u"abc", None, u"def"]),
+ ]
+ )
+ dataframe = pandas.DataFrame(df_data, columns=df_data.keys())
+ load_patch = mock.patch(
+ "google.cloud.bigquery.client.Client.load_table_from_file", autospec=True
+ )
+
+ schema = (
+ SchemaField("int_as_float_col", "INTEGER"),
+ SchemaField("string_col", "STRING"),
+ SchemaField("unknown_col", "BYTES"),
+ )
+ job_config = job.LoadJobConfig(schema=schema)
+ with load_patch as load_table_from_file, pytest.raises(
+ ValueError
+ ) as exc_context:
+ client.load_table_from_dataframe(
+ dataframe, self.TABLE_REF, job_config=job_config, location=self.LOCATION
+ )
+
+ load_table_from_file.assert_not_called()
+ message = str(exc_context.value)
+ assert "bq_schema contains fields not present in dataframe" in message
+ assert "unknown_col" in message
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ def test_load_table_from_dataframe_w_partial_schema_missing_types(self):
+ from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES
+ from google.cloud.bigquery import job
+ from google.cloud.bigquery.schema import SchemaField
+
+ client = self._make_client()
+ df_data = collections.OrderedDict(
+ [
+ ("string_col", [u"abc", u"def", u"ghi"]),
+ ("unknown_col", [b"jkl", None, b"mno"]),
+ ]
+ )
+ dataframe = pandas.DataFrame(df_data, columns=df_data.keys())
+ load_patch = mock.patch(
+ "google.cloud.bigquery.client.Client.load_table_from_file", autospec=True
+ )
+
+ schema = (SchemaField("string_col", "STRING"),)
+ job_config = job.LoadJobConfig(schema=schema)
+ with load_patch as load_table_from_file, warnings.catch_warnings(
+ record=True
+ ) as warned:
+ client.load_table_from_dataframe(
+ dataframe, self.TABLE_REF, job_config=job_config, location=self.LOCATION
+ )
+
+ load_table_from_file.assert_called_once_with(
+ client,
+ mock.ANY,
+ self.TABLE_REF,
+ num_retries=_DEFAULT_NUM_RETRIES,
+ rewind=True,
+ job_id=mock.ANY,
+ job_id_prefix=None,
+ location=self.LOCATION,
+ project=None,
+ job_config=mock.ANY,
+ )
+
+ assert warned # there should be at least one warning
+ unknown_col_warnings = [
+ warning for warning in warned if "unknown_col" in str(warning)
+ ]
+ assert unknown_col_warnings
+ assert unknown_col_warnings[0].category == UserWarning
+
+ sent_config = load_table_from_file.mock_calls[0][2]["job_config"]
+ assert sent_config.source_format == job.SourceFormat.PARQUET
+ assert sent_config.schema is None
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ def test_load_table_from_dataframe_w_schema_wo_pyarrow(self):
+ from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES
+ from google.cloud.bigquery import job
+ from google.cloud.bigquery.schema import SchemaField
+
+ client = self._make_client()
+ records = [{"name": u"Monty", "age": 100}, {"name": u"Python", "age": 60}]
+ dataframe = pandas.DataFrame(records, columns=["name", "age"])
+ schema = (SchemaField("name", "STRING"), SchemaField("age", "INTEGER"))
+ job_config = job.LoadJobConfig(schema=schema)
+
+ load_patch = mock.patch(
+ "google.cloud.bigquery.client.Client.load_table_from_file", autospec=True
+ )
+ pyarrow_patch = mock.patch("google.cloud.bigquery.client.pyarrow", None)
+
+ with load_patch as load_table_from_file, pyarrow_patch, warnings.catch_warnings(
+ record=True
+ ) as warned:
+ client.load_table_from_dataframe(
+ dataframe, self.TABLE_REF, job_config=job_config, location=self.LOCATION
+ )
+
+ assert warned # there should be at least one warning
+ for warning in warned:
+ assert "pyarrow" in str(warning)
+ assert warning.category in (DeprecationWarning, PendingDeprecationWarning)
+
+ load_table_from_file.assert_called_once_with(
+ client,
+ mock.ANY,
+ self.TABLE_REF,
+ num_retries=_DEFAULT_NUM_RETRIES,
+ rewind=True,
+ job_id=mock.ANY,
+ job_id_prefix=None,
+ location=self.LOCATION,
+ project=None,
+ job_config=mock.ANY,
+ )
+
+ sent_config = load_table_from_file.mock_calls[0][2]["job_config"]
+ assert sent_config.source_format == job.SourceFormat.PARQUET
+ assert tuple(sent_config.schema) == schema
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ def test_load_table_from_dataframe_w_schema_arrow_custom_compression(self):
+ from google.cloud.bigquery import job
+ from google.cloud.bigquery.schema import SchemaField
+
+ client = self._make_client()
+ records = [{"name": u"Monty", "age": 100}, {"name": u"Python", "age": 60}]
+ dataframe = pandas.DataFrame(records)
+ schema = (SchemaField("name", "STRING"), SchemaField("age", "INTEGER"))
+ job_config = job.LoadJobConfig(schema=schema)
+
+ load_patch = mock.patch(
+ "google.cloud.bigquery.client.Client.load_table_from_file", autospec=True
+ )
+ to_parquet_patch = mock.patch(
+ "google.cloud.bigquery.client._pandas_helpers.dataframe_to_parquet",
+ autospec=True,
+ )
+
+ with load_patch, to_parquet_patch as fake_to_parquet:
+ client.load_table_from_dataframe(
+ dataframe,
+ self.TABLE_REF,
+ job_config=job_config,
+ location=self.LOCATION,
+ parquet_compression="LZ4",
+ )
+
+ call_args = fake_to_parquet.call_args
+ assert call_args is not None
+ assert call_args.kwargs.get("parquet_compression") == "LZ4"
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ def test_load_table_from_dataframe_wo_pyarrow_custom_compression(self):
+ client = self._make_client()
+ records = [{"id": 1, "age": 100}, {"id": 2, "age": 60}]
+ dataframe = pandas.DataFrame(records)
+
+ load_patch = mock.patch(
+ "google.cloud.bigquery.client.Client.load_table_from_file", autospec=True
+ )
+ pyarrow_patch = mock.patch("google.cloud.bigquery.client.pyarrow", None)
+ to_parquet_patch = mock.patch.object(
+ dataframe, "to_parquet", wraps=dataframe.to_parquet
+ )
+
+ with load_patch, pyarrow_patch, to_parquet_patch as to_parquet_spy:
+ client.load_table_from_dataframe(
+ dataframe,
+ self.TABLE_REF,
+ location=self.LOCATION,
+ parquet_compression="gzip",
+ )
+
+ call_args = to_parquet_spy.call_args
+ assert call_args is not None
+ assert call_args.kwargs.get("compression") == "gzip"
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ def test_load_table_from_dataframe_w_nulls(self):
+ """Test that a DataFrame with null columns can be uploaded if a
+ BigQuery schema is specified.
+
+ See: https://github.com/googleapis/google-cloud-python/issues/7370
+ """
+ from google.cloud.bigquery.schema import SchemaField
+ from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES
+ from google.cloud.bigquery import job
+
+ client = self._make_client()
+ records = [{"name": None, "age": None}, {"name": None, "age": None}]
+ dataframe = pandas.DataFrame(records, columns=["name", "age"])
+ schema = [SchemaField("name", "STRING"), SchemaField("age", "INTEGER")]
+ job_config = job.LoadJobConfig(schema=schema)
+
+ load_patch = mock.patch(
+ "google.cloud.bigquery.client.Client.load_table_from_file", autospec=True
+ )
+ with load_patch as load_table_from_file:
+ client.load_table_from_dataframe(
+ dataframe, self.TABLE_REF, job_config=job_config, location=self.LOCATION
+ )
+
+ load_table_from_file.assert_called_once_with(
+ client,
+ mock.ANY,
+ self.TABLE_REF,
+ num_retries=_DEFAULT_NUM_RETRIES,
+ rewind=True,
+ job_id=mock.ANY,
+ job_id_prefix=None,
+ location=self.LOCATION,
+ project=None,
+ job_config=mock.ANY,
+ )
+
+ sent_config = load_table_from_file.mock_calls[0][2]["job_config"]
+ assert sent_config.schema == schema
+ assert sent_config.source_format == job.SourceFormat.PARQUET
+
+ def test_load_table_from_json_basic_use(self):
+ from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES
+ from google.cloud.bigquery import job
+
+ client = self._make_client()
+
+ json_rows = [
+ {"name": "One", "age": 11, "birthday": "2008-09-10", "adult": False},
+ {"name": "Two", "age": 22, "birthday": "1997-08-09", "adult": True},
+ ]
+
+ load_patch = mock.patch(
+ "google.cloud.bigquery.client.Client.load_table_from_file", autospec=True
+ )
+
+ with load_patch as load_table_from_file:
+ client.load_table_from_json(json_rows, self.TABLE_REF)
+
+ load_table_from_file.assert_called_once_with(
+ client,
+ mock.ANY,
+ self.TABLE_REF,
+ num_retries=_DEFAULT_NUM_RETRIES,
+ job_id=mock.ANY,
+ job_id_prefix=None,
+ location=client.location,
+ project=client.project,
+ job_config=mock.ANY,
+ )
+
+ sent_config = load_table_from_file.mock_calls[0][2]["job_config"]
+ assert sent_config.source_format == job.SourceFormat.NEWLINE_DELIMITED_JSON
+ assert sent_config.schema is None
+ assert sent_config.autodetect
+
+ def test_load_table_from_json_non_default_args(self):
+ from google.cloud.bigquery import job
+ from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES
+ from google.cloud.bigquery.schema import SchemaField
+
+ client = self._make_client()
+
+ json_rows = [
+ {"name": "One", "age": 11, "birthday": "2008-09-10", "adult": False},
+ {"name": "Two", "age": 22, "birthday": "1997-08-09", "adult": True},
+ ]
+
+ schema = [
+ SchemaField("name", "STRING"),
+ SchemaField("age", "INTEGER"),
+ SchemaField("adult", "BOOLEAN"),
+ ]
+ job_config = job.LoadJobConfig(schema=schema)
+ job_config._properties["load"]["unknown_field"] = "foobar"
+
+ load_patch = mock.patch(
+ "google.cloud.bigquery.client.Client.load_table_from_file", autospec=True
+ )
+
+ with load_patch as load_table_from_file:
+ client.load_table_from_json(
+ json_rows,
+ self.TABLE_REF,
+ job_config=job_config,
+ project="project-x",
+ location="EU",
+ )
+
+ load_table_from_file.assert_called_once_with(
+ client,
+ mock.ANY,
+ self.TABLE_REF,
+ num_retries=_DEFAULT_NUM_RETRIES,
+ job_id=mock.ANY,
+ job_id_prefix=None,
+ location="EU",
+ project="project-x",
+ job_config=mock.ANY,
+ )
+
+ sent_config = load_table_from_file.mock_calls[0][2]["job_config"]
+ assert job_config.source_format is None # the original was not modified
+ assert sent_config.source_format == job.SourceFormat.NEWLINE_DELIMITED_JSON
+ assert sent_config.schema == schema
+ assert not sent_config.autodetect
+ # all properties should have been cloned and sent to the backend
+ assert sent_config._properties.get("load", {}).get("unknown_field") == "foobar"
+
+ # Low-level tests
+
+ @classmethod
+ def _make_resumable_upload_responses(cls, size):
+ """Make a series of responses for a successful resumable upload."""
+ from google import resumable_media
+
+ resumable_url = "http://test.invalid?upload_id=and-then-there-was-1"
+ initial_response = cls._make_response(
+ http_client.OK, "", {"location": resumable_url}
+ )
+ data_response = cls._make_response(
+ resumable_media.PERMANENT_REDIRECT,
+ "",
+ {"range": "bytes=0-{:d}".format(size - 1)},
+ )
+ final_response = cls._make_response(
+ http_client.OK,
+ json.dumps({"size": size}),
+ {"Content-Type": "application/json"},
+ )
+ return [initial_response, data_response, final_response]
+
+ @staticmethod
+ def _make_transport(responses=None):
+ import google.auth.transport.requests
+
+ transport = mock.create_autospec(
+ google.auth.transport.requests.AuthorizedSession, instance=True
+ )
+ transport.request.side_effect = responses
+ return transport
+
+ def test__do_resumable_upload(self):
+ file_obj = self._make_file_obj()
+ file_obj_len = len(file_obj.getvalue())
+ transport = self._make_transport(
+ self._make_resumable_upload_responses(file_obj_len)
+ )
+ client = self._make_client(transport)
+
+ result = client._do_resumable_upload(
+ file_obj, self.EXPECTED_CONFIGURATION, None
+ )
+
+ content = result.content.decode("utf-8")
+ assert json.loads(content) == {"size": file_obj_len}
+
+ # Verify that configuration data was passed in with the initial
+ # request.
+ transport.request.assert_any_call(
+ "POST",
+ mock.ANY,
+ data=json.dumps(self.EXPECTED_CONFIGURATION).encode("utf-8"),
+ headers=mock.ANY,
+ timeout=mock.ANY,
+ )
+
+ def test__do_multipart_upload(self):
+ transport = self._make_transport([self._make_response(http_client.OK)])
+ client = self._make_client(transport)
+ file_obj = self._make_file_obj()
+ file_obj_len = len(file_obj.getvalue())
+
+ client._do_multipart_upload(
+ file_obj, self.EXPECTED_CONFIGURATION, file_obj_len, None
+ )
+
+ # Verify that configuration data was passed in with the initial
+ # request.
+ request_args = transport.request.mock_calls[0][2]
+ request_data = request_args["data"].decode("utf-8")
+ request_headers = request_args["headers"]
+
+ request_content = email.message_from_string(
+ "Content-Type: {}\r\n{}".format(
+ request_headers["content-type"].decode("utf-8"), request_data
+ )
+ )
+
+ # There should be two payloads: the configuration and the binary daya.
+ configuration_data = request_content.get_payload(0).get_payload()
+ binary_data = request_content.get_payload(1).get_payload()
+
+ assert json.loads(configuration_data) == self.EXPECTED_CONFIGURATION
+ assert binary_data.encode("utf-8") == file_obj.getvalue()
+
+ def test__do_multipart_upload_wrong_size(self):
+ client = self._make_client()
+ file_obj = self._make_file_obj()
+ file_obj_len = len(file_obj.getvalue())
+
+ with pytest.raises(ValueError):
+ client._do_multipart_upload(file_obj, {}, file_obj_len + 1, None)
+
+ def test_schema_from_json_with_file_path(self):
+ from google.cloud.bigquery.schema import SchemaField
+
+ file_content = """[
+ {
+ "description": "quarter",
+ "mode": "REQUIRED",
+ "name": "qtr",
+ "type": "STRING"
+ },
+ {
+ "description": "sales representative",
+ "mode": "NULLABLE",
+ "name": "rep",
+ "type": "STRING"
+ },
+ {
+ "description": "total sales",
+ "mode": "NULLABLE",
+ "name": "sales",
+ "type": "FLOAT"
+ }
+ ]"""
+
+ expected = [
+ SchemaField("qtr", "STRING", "REQUIRED", "quarter"),
+ SchemaField("rep", "STRING", "NULLABLE", "sales representative"),
+ SchemaField("sales", "FLOAT", "NULLABLE", "total sales"),
+ ]
+
+ client = self._make_client()
+ mock_file_path = "/mocked/file.json"
+
+ if six.PY2:
+ open_patch = mock.patch(
+ "__builtin__.open", mock.mock_open(read_data=file_content)
+ )
+ else:
+ open_patch = mock.patch(
+ "builtins.open", new=mock.mock_open(read_data=file_content)
+ )
+
+ with open_patch as _mock_file:
+ actual = client.schema_from_json(mock_file_path)
+ _mock_file.assert_called_once_with(mock_file_path)
+ # This assert is to make sure __exit__ is called in the context
+ # manager that opens the file in the function
+ _mock_file().__exit__.assert_called_once()
+
+ assert expected == actual
+
+ def test_schema_from_json_with_file_object(self):
+ from google.cloud.bigquery.schema import SchemaField
+
+ file_content = """[
+ {
+ "description": "quarter",
+ "mode": "REQUIRED",
+ "name": "qtr",
+ "type": "STRING"
+ },
+ {
+ "description": "sales representative",
+ "mode": "NULLABLE",
+ "name": "rep",
+ "type": "STRING"
+ },
+ {
+ "description": "total sales",
+ "mode": "NULLABLE",
+ "name": "sales",
+ "type": "FLOAT"
+ }
+ ]"""
+
+ expected = [
+ SchemaField("qtr", "STRING", "REQUIRED", "quarter"),
+ SchemaField("rep", "STRING", "NULLABLE", "sales representative"),
+ SchemaField("sales", "FLOAT", "NULLABLE", "total sales"),
+ ]
+
+ client = self._make_client()
+
+ if six.PY2:
+ fake_file = io.BytesIO(file_content)
+ else:
+ fake_file = io.StringIO(file_content)
+
+ actual = client.schema_from_json(fake_file)
+
+ assert expected == actual
+
+ def test_schema_to_json_with_file_path(self):
+ from google.cloud.bigquery.schema import SchemaField
+
+ file_content = [
+ {
+ "description": "quarter",
+ "mode": "REQUIRED",
+ "name": "qtr",
+ "type": "STRING",
+ },
+ {
+ "description": "sales representative",
+ "mode": "NULLABLE",
+ "name": "rep",
+ "type": "STRING",
+ },
+ {
+ "description": "total sales",
+ "mode": "NULLABLE",
+ "name": "sales",
+ "type": "FLOAT",
+ },
+ ]
+
+ schema_list = [
+ SchemaField("qtr", "STRING", "REQUIRED", "quarter"),
+ SchemaField("rep", "STRING", "NULLABLE", "sales representative"),
+ SchemaField("sales", "FLOAT", "NULLABLE", "total sales"),
+ ]
+
+ client = self._make_client()
+ mock_file_path = "/mocked/file.json"
+
+ if six.PY2:
+ open_patch = mock.patch("__builtin__.open", mock.mock_open())
+ else:
+ open_patch = mock.patch("builtins.open", mock.mock_open())
+
+ with open_patch as mock_file, mock.patch("json.dump") as mock_dump:
+ client.schema_to_json(schema_list, mock_file_path)
+ mock_file.assert_called_once_with(mock_file_path, mode="w")
+ # This assert is to make sure __exit__ is called in the context
+ # manager that opens the file in the function
+ mock_file().__exit__.assert_called_once()
+ mock_dump.assert_called_with(
+ file_content, mock_file.return_value, indent=2, sort_keys=True
+ )
+
+ def test_schema_to_json_with_file_object(self):
+ from google.cloud.bigquery.schema import SchemaField
+
+ file_content = [
+ {
+ "description": "quarter",
+ "mode": "REQUIRED",
+ "name": "qtr",
+ "type": "STRING",
+ },
+ {
+ "description": "sales representative",
+ "mode": "NULLABLE",
+ "name": "rep",
+ "type": "STRING",
+ },
+ {
+ "description": "total sales",
+ "mode": "NULLABLE",
+ "name": "sales",
+ "type": "FLOAT",
+ },
+ ]
+
+ schema_list = [
+ SchemaField("qtr", "STRING", "REQUIRED", "quarter"),
+ SchemaField("rep", "STRING", "NULLABLE", "sales representative"),
+ SchemaField("sales", "FLOAT", "NULLABLE", "total sales"),
+ ]
+
+ if six.PY2:
+ fake_file = io.BytesIO()
+ else:
+ fake_file = io.StringIO()
+
+ client = self._make_client()
+
+ client.schema_to_json(schema_list, fake_file)
+ assert file_content == json.loads(fake_file.getvalue())
diff --git a/bigquery/tests/unit/test_dataset.py b/bigquery/tests/unit/test_dataset.py
new file mode 100644
index 000000000000..26b1729a240c
--- /dev/null
+++ b/bigquery/tests/unit/test_dataset.py
@@ -0,0 +1,671 @@
+# Copyright 2015 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import unittest
+
+import mock
+import pytest
+
+
+class TestAccessEntry(unittest.TestCase):
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery.dataset import AccessEntry
+
+ return AccessEntry
+
+ def _make_one(self, *args, **kw):
+ return self._get_target_class()(*args, **kw)
+
+ def test_ctor_defaults(self):
+ entry = self._make_one("OWNER", "userByEmail", "phred@example.com")
+ self.assertEqual(entry.role, "OWNER")
+ self.assertEqual(entry.entity_type, "userByEmail")
+ self.assertEqual(entry.entity_id, "phred@example.com")
+
+ def test_ctor_bad_entity_type(self):
+ with self.assertRaises(ValueError):
+ self._make_one(None, "unknown", None)
+
+ def test_ctor_view_with_role(self):
+ role = "READER"
+ entity_type = "view"
+ with self.assertRaises(ValueError):
+ self._make_one(role, entity_type, None)
+
+ def test_ctor_view_success(self):
+ role = None
+ entity_type = "view"
+ entity_id = object()
+ entry = self._make_one(role, entity_type, entity_id)
+ self.assertEqual(entry.role, role)
+ self.assertEqual(entry.entity_type, entity_type)
+ self.assertEqual(entry.entity_id, entity_id)
+
+ def test_ctor_nonview_without_role(self):
+ role = None
+ entity_type = "userByEmail"
+ with self.assertRaises(ValueError):
+ self._make_one(role, entity_type, None)
+
+ def test___eq___role_mismatch(self):
+ entry = self._make_one("OWNER", "userByEmail", "phred@example.com")
+ other = self._make_one("WRITER", "userByEmail", "phred@example.com")
+ self.assertNotEqual(entry, other)
+
+ def test___eq___entity_type_mismatch(self):
+ entry = self._make_one("OWNER", "userByEmail", "phred@example.com")
+ other = self._make_one("OWNER", "groupByEmail", "phred@example.com")
+ self.assertNotEqual(entry, other)
+
+ def test___eq___entity_id_mismatch(self):
+ entry = self._make_one("OWNER", "userByEmail", "phred@example.com")
+ other = self._make_one("OWNER", "userByEmail", "bharney@example.com")
+ self.assertNotEqual(entry, other)
+
+ def test___eq___hit(self):
+ entry = self._make_one("OWNER", "userByEmail", "phred@example.com")
+ other = self._make_one("OWNER", "userByEmail", "phred@example.com")
+ self.assertEqual(entry, other)
+
+ def test__eq___type_mismatch(self):
+ entry = self._make_one("OWNER", "userByEmail", "silly@example.com")
+ self.assertNotEqual(entry, object())
+ self.assertEqual(entry, mock.ANY)
+
+ def test_to_api_repr(self):
+ entry = self._make_one("OWNER", "userByEmail", "salmon@example.com")
+ resource = entry.to_api_repr()
+ exp_resource = {"role": "OWNER", "userByEmail": "salmon@example.com"}
+ self.assertEqual(resource, exp_resource)
+
+ def test_to_api_repr_view(self):
+ view = {
+ "projectId": "my-project",
+ "datasetId": "my_dataset",
+ "tableId": "my_table",
+ }
+ entry = self._make_one(None, "view", view)
+ resource = entry.to_api_repr()
+ exp_resource = {"view": view}
+ self.assertEqual(resource, exp_resource)
+
+ def test_from_api_repr(self):
+ resource = {"role": "OWNER", "userByEmail": "salmon@example.com"}
+ entry = self._get_target_class().from_api_repr(resource)
+ self.assertEqual(entry.role, "OWNER")
+ self.assertEqual(entry.entity_type, "userByEmail")
+ self.assertEqual(entry.entity_id, "salmon@example.com")
+
+ def test_from_api_repr_w_unknown_entity_type(self):
+ resource = {"role": "READER", "unknown": "UNKNOWN"}
+ with self.assertRaises(ValueError):
+ self._get_target_class().from_api_repr(resource)
+
+ def test_from_api_repr_entries_w_extra_keys(self):
+ resource = {
+ "role": "READER",
+ "specialGroup": "projectReaders",
+ "userByEmail": "salmon@example.com",
+ }
+ with self.assertRaises(ValueError):
+ self._get_target_class().from_api_repr(resource)
+
+
+class TestDatasetReference(unittest.TestCase):
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery.dataset import DatasetReference
+
+ return DatasetReference
+
+ def _make_one(self, *args, **kw):
+ return self._get_target_class()(*args, **kw)
+
+ def test_ctor_defaults(self):
+ dataset_ref = self._make_one("some-project-1", "dataset_1")
+ self.assertEqual(dataset_ref.project, "some-project-1")
+ self.assertEqual(dataset_ref.dataset_id, "dataset_1")
+
+ def test_ctor_bad_args(self):
+ with self.assertRaises(ValueError):
+ self._make_one(1, "d")
+ with self.assertRaises(ValueError):
+ self._make_one("p", 2)
+
+ def test_table(self):
+ dataset_ref = self._make_one("some-project-1", "dataset_1")
+ table_ref = dataset_ref.table("table_1")
+ self.assertEqual(table_ref.dataset_id, "dataset_1")
+ self.assertEqual(table_ref.project, "some-project-1")
+ self.assertEqual(table_ref.table_id, "table_1")
+
+ def test_model(self):
+ dataset_ref = self._make_one("some-project-1", "dataset_1")
+ model_ref = dataset_ref.model("model_1")
+ self.assertEqual(model_ref.project, "some-project-1")
+ self.assertEqual(model_ref.dataset_id, "dataset_1")
+ self.assertEqual(model_ref.model_id, "model_1")
+
+ def test_routine(self):
+ dataset_ref = self._make_one("some-project-1", "dataset_1")
+ routine_ref = dataset_ref.routine("routine_1")
+ self.assertEqual(routine_ref.project, "some-project-1")
+ self.assertEqual(routine_ref.dataset_id, "dataset_1")
+ self.assertEqual(routine_ref.routine_id, "routine_1")
+
+ def test_to_api_repr(self):
+ dataset = self._make_one("project_1", "dataset_1")
+
+ resource = dataset.to_api_repr()
+
+ self.assertEqual(resource, {"projectId": "project_1", "datasetId": "dataset_1"})
+
+ def test_from_api_repr(self):
+ cls = self._get_target_class()
+ expected = self._make_one("project_1", "dataset_1")
+
+ got = cls.from_api_repr({"projectId": "project_1", "datasetId": "dataset_1"})
+
+ self.assertEqual(expected, got)
+
+ def test_from_string(self):
+ cls = self._get_target_class()
+ got = cls.from_string("string-project.string_dataset")
+ self.assertEqual(got.project, "string-project")
+ self.assertEqual(got.dataset_id, "string_dataset")
+
+ def test_from_string_w_prefix(self):
+ cls = self._get_target_class()
+ got = cls.from_string("google.com:string-project.string_dataset")
+ self.assertEqual(got.project, "google.com:string-project")
+ self.assertEqual(got.dataset_id, "string_dataset")
+
+ def test_from_string_legacy_string(self):
+ cls = self._get_target_class()
+ with self.assertRaises(ValueError):
+ cls.from_string("string-project:string_dataset")
+
+ def test_from_string_w_incorrect_prefix(self):
+ cls = self._get_target_class()
+ with self.assertRaises(ValueError):
+ cls.from_string("google.com.string-project.dataset_id")
+
+ def test_from_string_w_prefix_and_too_many_parts(self):
+ cls = self._get_target_class()
+ with self.assertRaises(ValueError):
+ cls.from_string("google.com:string-project.dataset_id.table_id")
+
+ def test_from_string_not_fully_qualified(self):
+ cls = self._get_target_class()
+ with self.assertRaises(ValueError):
+ cls.from_string("string_dataset")
+ with self.assertRaises(ValueError):
+ cls.from_string("a.b.c")
+
+ def test_from_string_with_default_project(self):
+ cls = self._get_target_class()
+ got = cls.from_string("string_dataset", default_project="default-project")
+ self.assertEqual(got.project, "default-project")
+ self.assertEqual(got.dataset_id, "string_dataset")
+
+ def test_from_string_ignores_default_project(self):
+ cls = self._get_target_class()
+ got = cls.from_string(
+ "string-project.string_dataset", default_project="default-project"
+ )
+ self.assertEqual(got.project, "string-project")
+ self.assertEqual(got.dataset_id, "string_dataset")
+
+ def test___eq___wrong_type(self):
+ dataset = self._make_one("project_1", "dataset_1")
+ other = object()
+ self.assertNotEqual(dataset, other)
+ self.assertEqual(dataset, mock.ANY)
+
+ def test___eq___project_mismatch(self):
+ dataset = self._make_one("project_1", "dataset_1")
+ other = self._make_one("project_2", "dataset_1")
+ self.assertNotEqual(dataset, other)
+
+ def test___eq___dataset_mismatch(self):
+ dataset = self._make_one("project_1", "dataset_1")
+ other = self._make_one("project_1", "dataset_2")
+ self.assertNotEqual(dataset, other)
+
+ def test___eq___equality(self):
+ dataset = self._make_one("project_1", "dataset_1")
+ other = self._make_one("project_1", "dataset_1")
+ self.assertEqual(dataset, other)
+
+ def test___hash__set_equality(self):
+ dataset1 = self._make_one("project_1", "dataset_1")
+ dataset2 = self._make_one("project_1", "dataset_2")
+ set_one = {dataset1, dataset2}
+ set_two = {dataset1, dataset2}
+ self.assertEqual(set_one, set_two)
+
+ def test___hash__not_equals(self):
+ dataset1 = self._make_one("project_1", "dataset_1")
+ dataset2 = self._make_one("project_1", "dataset_2")
+ set_one = {dataset1}
+ set_two = {dataset2}
+ self.assertNotEqual(set_one, set_two)
+
+ def test___repr__(self):
+ dataset = self._make_one("project1", "dataset1")
+ expected = "DatasetReference('project1', 'dataset1')"
+ self.assertEqual(repr(dataset), expected)
+
+
+class TestDataset(unittest.TestCase):
+ from google.cloud.bigquery.dataset import DatasetReference
+
+ PROJECT = "project"
+ DS_ID = "dataset-id"
+ DS_REF = DatasetReference(PROJECT, DS_ID)
+
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery.dataset import Dataset
+
+ return Dataset
+
+ def _make_one(self, *args, **kw):
+ return self._get_target_class()(*args, **kw)
+
+ def _setUpConstants(self):
+ import datetime
+ from google.cloud._helpers import UTC
+
+ self.WHEN_TS = 1437767599.006
+ self.WHEN = datetime.datetime.utcfromtimestamp(self.WHEN_TS).replace(tzinfo=UTC)
+ self.ETAG = "ETAG"
+ self.DS_FULL_ID = "%s:%s" % (self.PROJECT, self.DS_ID)
+ self.RESOURCE_URL = "http://example.com/path/to/resource"
+
+ def _make_resource(self):
+ self._setUpConstants()
+ USER_EMAIL = "phred@example.com"
+ GROUP_EMAIL = "group-name@lists.example.com"
+ return {
+ "creationTime": self.WHEN_TS * 1000,
+ "datasetReference": {"projectId": self.PROJECT, "datasetId": self.DS_ID},
+ "etag": self.ETAG,
+ "id": self.DS_FULL_ID,
+ "lastModifiedTime": self.WHEN_TS * 1000,
+ "location": "US",
+ "selfLink": self.RESOURCE_URL,
+ "defaultTableExpirationMs": 3600,
+ "access": [
+ {"role": "OWNER", "userByEmail": USER_EMAIL},
+ {"role": "OWNER", "groupByEmail": GROUP_EMAIL},
+ {"role": "WRITER", "specialGroup": "projectWriters"},
+ {"role": "READER", "specialGroup": "projectReaders"},
+ ],
+ }
+
+ def _verify_access_entry(self, access_entries, resource):
+ r_entries = []
+ for r_entry in resource["access"]:
+ role = r_entry.pop("role")
+ for entity_type, entity_id in sorted(r_entry.items()):
+ r_entries.append(
+ {"role": role, "entity_type": entity_type, "entity_id": entity_id}
+ )
+
+ self.assertEqual(len(access_entries), len(r_entries))
+ for a_entry, r_entry in zip(access_entries, r_entries):
+ self.assertEqual(a_entry.role, r_entry["role"])
+ self.assertEqual(a_entry.entity_type, r_entry["entity_type"])
+ self.assertEqual(a_entry.entity_id, r_entry["entity_id"])
+
+ def _verify_readonly_resource_properties(self, dataset, resource):
+
+ self.assertEqual(dataset.project, self.PROJECT)
+ self.assertEqual(dataset.dataset_id, self.DS_ID)
+ self.assertEqual(dataset.reference.project, self.PROJECT)
+ self.assertEqual(dataset.reference.dataset_id, self.DS_ID)
+
+ if "creationTime" in resource:
+ self.assertEqual(dataset.created, self.WHEN)
+ else:
+ self.assertIsNone(dataset.created)
+ if "etag" in resource:
+ self.assertEqual(dataset.etag, self.ETAG)
+ else:
+ self.assertIsNone(dataset.etag)
+ if "lastModifiedTime" in resource:
+ self.assertEqual(dataset.modified, self.WHEN)
+ else:
+ self.assertIsNone(dataset.modified)
+ if "selfLink" in resource:
+ self.assertEqual(dataset.self_link, self.RESOURCE_URL)
+ else:
+ self.assertIsNone(dataset.self_link)
+
+ def _verify_resource_properties(self, dataset, resource):
+
+ self._verify_readonly_resource_properties(dataset, resource)
+
+ if "defaultTableExpirationMs" in resource:
+ self.assertEqual(
+ dataset.default_table_expiration_ms,
+ int(resource.get("defaultTableExpirationMs")),
+ )
+ else:
+ self.assertIsNone(dataset.default_table_expiration_ms)
+ self.assertEqual(dataset.description, resource.get("description"))
+ self.assertEqual(dataset.friendly_name, resource.get("friendlyName"))
+ self.assertEqual(dataset.location, resource.get("location"))
+
+ if "access" in resource:
+ self._verify_access_entry(dataset.access_entries, resource)
+ else:
+ self.assertEqual(dataset.access_entries, [])
+
+ def test_ctor_defaults(self):
+ dataset = self._make_one(self.DS_REF)
+ self.assertEqual(dataset.dataset_id, self.DS_ID)
+ self.assertEqual(dataset.project, self.PROJECT)
+ self.assertEqual(
+ dataset.path, "/projects/%s/datasets/%s" % (self.PROJECT, self.DS_ID)
+ )
+ self.assertEqual(dataset.access_entries, [])
+
+ self.assertIsNone(dataset.created)
+ self.assertIsNone(dataset.full_dataset_id)
+ self.assertIsNone(dataset.etag)
+ self.assertIsNone(dataset.modified)
+ self.assertIsNone(dataset.self_link)
+
+ self.assertIsNone(dataset.default_table_expiration_ms)
+ self.assertIsNone(dataset.description)
+ self.assertIsNone(dataset.friendly_name)
+ self.assertIsNone(dataset.location)
+
+ def test_ctor_string(self):
+ dataset = self._make_one("some-project.some_dset")
+ self.assertEqual(dataset.project, "some-project")
+ self.assertEqual(dataset.dataset_id, "some_dset")
+
+ def test_ctor_string_wo_project_id(self):
+ with pytest.raises(ValueError):
+ # Project ID is missing.
+ self._make_one("some_dset")
+
+ def test_ctor_explicit(self):
+ from google.cloud.bigquery.dataset import DatasetReference, AccessEntry
+
+ phred = AccessEntry("OWNER", "userByEmail", "phred@example.com")
+ bharney = AccessEntry("OWNER", "userByEmail", "bharney@example.com")
+ entries = [phred, bharney]
+ OTHER_PROJECT = "foo-bar-123"
+ dataset = self._make_one(DatasetReference(OTHER_PROJECT, self.DS_ID))
+ dataset.access_entries = entries
+ self.assertEqual(dataset.dataset_id, self.DS_ID)
+ self.assertEqual(dataset.project, OTHER_PROJECT)
+ self.assertEqual(
+ dataset.path, "/projects/%s/datasets/%s" % (OTHER_PROJECT, self.DS_ID)
+ )
+ self.assertEqual(dataset.access_entries, entries)
+
+ self.assertIsNone(dataset.created)
+ self.assertIsNone(dataset.full_dataset_id)
+ self.assertIsNone(dataset.etag)
+ self.assertIsNone(dataset.modified)
+ self.assertIsNone(dataset.self_link)
+
+ self.assertIsNone(dataset.default_table_expiration_ms)
+ self.assertIsNone(dataset.description)
+ self.assertIsNone(dataset.friendly_name)
+ self.assertIsNone(dataset.location)
+
+ def test_access_entries_setter_non_list(self):
+ dataset = self._make_one(self.DS_REF)
+ with self.assertRaises(TypeError):
+ dataset.access_entries = object()
+
+ def test_access_entries_setter_invalid_field(self):
+ from google.cloud.bigquery.dataset import AccessEntry
+
+ dataset = self._make_one(self.DS_REF)
+ phred = AccessEntry("OWNER", "userByEmail", "phred@example.com")
+ with self.assertRaises(ValueError):
+ dataset.access_entries = [phred, object()]
+
+ def test_access_entries_setter(self):
+ from google.cloud.bigquery.dataset import AccessEntry
+
+ dataset = self._make_one(self.DS_REF)
+ phred = AccessEntry("OWNER", "userByEmail", "phred@example.com")
+ bharney = AccessEntry("OWNER", "userByEmail", "bharney@example.com")
+ dataset.access_entries = [phred, bharney]
+ self.assertEqual(dataset.access_entries, [phred, bharney])
+
+ def test_default_table_expiration_ms_setter_bad_value(self):
+ dataset = self._make_one(self.DS_REF)
+ with self.assertRaises(ValueError):
+ dataset.default_table_expiration_ms = "bogus"
+
+ def test_default_table_expiration_ms_setter(self):
+ dataset = self._make_one(self.DS_REF)
+ dataset.default_table_expiration_ms = 12345
+ self.assertEqual(dataset.default_table_expiration_ms, 12345)
+
+ def test_description_setter_bad_value(self):
+ dataset = self._make_one(self.DS_REF)
+ with self.assertRaises(ValueError):
+ dataset.description = 12345
+
+ def test_description_setter(self):
+ dataset = self._make_one(self.DS_REF)
+ dataset.description = "DESCRIPTION"
+ self.assertEqual(dataset.description, "DESCRIPTION")
+
+ def test_friendly_name_setter_bad_value(self):
+ dataset = self._make_one(self.DS_REF)
+ with self.assertRaises(ValueError):
+ dataset.friendly_name = 12345
+
+ def test_friendly_name_setter(self):
+ dataset = self._make_one(self.DS_REF)
+ dataset.friendly_name = "FRIENDLY"
+ self.assertEqual(dataset.friendly_name, "FRIENDLY")
+
+ def test_location_setter_bad_value(self):
+ dataset = self._make_one(self.DS_REF)
+ with self.assertRaises(ValueError):
+ dataset.location = 12345
+
+ def test_location_setter(self):
+ dataset = self._make_one(self.DS_REF)
+ dataset.location = "LOCATION"
+ self.assertEqual(dataset.location, "LOCATION")
+
+ def test_labels_update_in_place(self):
+ dataset = self._make_one(self.DS_REF)
+ del dataset._properties["labels"] # don't start w/ existing dict
+ labels = dataset.labels
+ labels["foo"] = "bar" # update in place
+ self.assertEqual(dataset.labels, {"foo": "bar"})
+
+ def test_labels_setter(self):
+ dataset = self._make_one(self.DS_REF)
+ dataset.labels = {"color": "green"}
+ self.assertEqual(dataset.labels, {"color": "green"})
+
+ def test_labels_setter_bad_value(self):
+ dataset = self._make_one(self.DS_REF)
+ with self.assertRaises(ValueError):
+ dataset.labels = None
+
+ def test_labels_getter_missing_value(self):
+ dataset = self._make_one(self.DS_REF)
+ self.assertEqual(dataset.labels, {})
+
+ def test_from_api_repr_missing_identity(self):
+ self._setUpConstants()
+ RESOURCE = {}
+ klass = self._get_target_class()
+ with self.assertRaises(KeyError):
+ klass.from_api_repr(RESOURCE)
+
+ def test_from_api_repr_bare(self):
+ self._setUpConstants()
+ RESOURCE = {
+ "id": "%s:%s" % (self.PROJECT, self.DS_ID),
+ "datasetReference": {"projectId": self.PROJECT, "datasetId": self.DS_ID},
+ }
+ klass = self._get_target_class()
+ dataset = klass.from_api_repr(RESOURCE)
+ self._verify_resource_properties(dataset, RESOURCE)
+
+ def test_from_api_repr_w_properties(self):
+ RESOURCE = self._make_resource()
+ klass = self._get_target_class()
+ dataset = klass.from_api_repr(RESOURCE)
+ self._verify_resource_properties(dataset, RESOURCE)
+
+ def test_to_api_repr_w_custom_field(self):
+ dataset = self._make_one(self.DS_REF)
+ dataset._properties["newAlphaProperty"] = "unreleased property"
+ resource = dataset.to_api_repr()
+
+ exp_resource = {
+ "datasetReference": self.DS_REF.to_api_repr(),
+ "labels": {},
+ "newAlphaProperty": "unreleased property",
+ }
+ self.assertEqual(resource, exp_resource)
+
+ def test_from_string(self):
+ cls = self._get_target_class()
+ got = cls.from_string("string-project.string_dataset")
+ self.assertEqual(got.project, "string-project")
+ self.assertEqual(got.dataset_id, "string_dataset")
+
+ def test_from_string_legacy_string(self):
+ cls = self._get_target_class()
+ with self.assertRaises(ValueError):
+ cls.from_string("string-project:string_dataset")
+
+ def test__build_resource_w_custom_field(self):
+ dataset = self._make_one(self.DS_REF)
+ dataset._properties["newAlphaProperty"] = "unreleased property"
+ resource = dataset._build_resource(["newAlphaProperty"])
+
+ exp_resource = {"newAlphaProperty": "unreleased property"}
+ self.assertEqual(resource, exp_resource)
+
+ def test__build_resource_w_custom_field_not_in__properties(self):
+ dataset = self._make_one(self.DS_REF)
+ dataset.bad = "value"
+ with self.assertRaises(ValueError):
+ dataset._build_resource(["bad"])
+
+ def test_table(self):
+ from google.cloud.bigquery.table import TableReference
+
+ dataset = self._make_one(self.DS_REF)
+ table = dataset.table("table_id")
+ self.assertIsInstance(table, TableReference)
+ self.assertEqual(table.table_id, "table_id")
+ self.assertEqual(table.dataset_id, self.DS_ID)
+ self.assertEqual(table.project, self.PROJECT)
+
+ def test___repr__(self):
+ from google.cloud.bigquery.dataset import DatasetReference
+
+ dataset = self._make_one(DatasetReference("project1", "dataset1"))
+ expected = "Dataset(DatasetReference('project1', 'dataset1'))"
+ self.assertEqual(repr(dataset), expected)
+
+
+class TestDatasetListItem(unittest.TestCase):
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery.dataset import DatasetListItem
+
+ return DatasetListItem
+
+ def _make_one(self, *args, **kw):
+ return self._get_target_class()(*args, **kw)
+
+ def test_ctor(self):
+ project = "test-project"
+ dataset_id = "test_dataset"
+ resource = {
+ "kind": "bigquery#dataset",
+ "id": "{}:{}".format(project, dataset_id),
+ "datasetReference": {"projectId": project, "datasetId": dataset_id},
+ "friendlyName": "Data of the Test",
+ "labels": {"some-stuff": "this-is-a-label"},
+ }
+
+ dataset = self._make_one(resource)
+ self.assertEqual(dataset.project, project)
+ self.assertEqual(dataset.dataset_id, dataset_id)
+ self.assertEqual(dataset.full_dataset_id, "{}:{}".format(project, dataset_id))
+ self.assertEqual(dataset.reference.project, project)
+ self.assertEqual(dataset.reference.dataset_id, dataset_id)
+ self.assertEqual(dataset.friendly_name, "Data of the Test")
+ self.assertEqual(dataset.labels["some-stuff"], "this-is-a-label")
+
+ def test_ctor_missing_properties(self):
+ resource = {
+ "datasetReference": {"projectId": "testproject", "datasetId": "testdataset"}
+ }
+ dataset = self._make_one(resource)
+ self.assertEqual(dataset.project, "testproject")
+ self.assertEqual(dataset.dataset_id, "testdataset")
+ self.assertIsNone(dataset.full_dataset_id)
+ self.assertIsNone(dataset.friendly_name)
+ self.assertEqual(dataset.labels, {})
+
+ def test_ctor_wo_project(self):
+ resource = {"datasetReference": {"datasetId": "testdataset"}}
+ with self.assertRaises(ValueError):
+ self._make_one(resource)
+
+ def test_ctor_wo_dataset(self):
+ resource = {"datasetReference": {"projectId": "testproject"}}
+ with self.assertRaises(ValueError):
+ self._make_one(resource)
+
+ def test_ctor_wo_reference(self):
+ with self.assertRaises(ValueError):
+ self._make_one({})
+
+ def test_labels_update_in_place(self):
+ resource = {
+ "datasetReference": {"projectId": "testproject", "datasetId": "testdataset"}
+ }
+ dataset = self._make_one(resource)
+ labels = dataset.labels
+ labels["foo"] = "bar" # update in place
+ self.assertEqual(dataset.labels, {"foo": "bar"})
+
+ def test_table(self):
+ from google.cloud.bigquery.table import TableReference
+
+ project = "test-project"
+ dataset_id = "test_dataset"
+ resource = {"datasetReference": {"projectId": project, "datasetId": dataset_id}}
+ dataset = self._make_one(resource)
+ table = dataset.table("table_id")
+ self.assertIsInstance(table, TableReference)
+ self.assertEqual(table.table_id, "table_id")
+ self.assertEqual(table.dataset_id, dataset_id)
+ self.assertEqual(table.project, project)
diff --git a/bigquery/tests/unit/test_dbapi__helpers.py b/bigquery/tests/unit/test_dbapi__helpers.py
new file mode 100644
index 000000000000..bcc3e0879f87
--- /dev/null
+++ b/bigquery/tests/unit/test_dbapi__helpers.py
@@ -0,0 +1,94 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import datetime
+import decimal
+import math
+import unittest
+
+import google.cloud._helpers
+from google.cloud.bigquery.dbapi import _helpers
+from google.cloud.bigquery.dbapi import exceptions
+
+
+class TestQueryParameters(unittest.TestCase):
+ def test_scalar_to_query_parameter(self):
+ expected_types = [
+ (True, "BOOL"),
+ (False, "BOOL"),
+ (123, "INT64"),
+ (-123456789, "INT64"),
+ (1.25, "FLOAT64"),
+ (decimal.Decimal("1.25"), "NUMERIC"),
+ (b"I am some bytes", "BYTES"),
+ (u"I am a string", "STRING"),
+ (datetime.date(2017, 4, 1), "DATE"),
+ (datetime.time(12, 34, 56), "TIME"),
+ (datetime.datetime(2012, 3, 4, 5, 6, 7), "DATETIME"),
+ (
+ datetime.datetime(
+ 2012, 3, 4, 5, 6, 7, tzinfo=google.cloud._helpers.UTC
+ ),
+ "TIMESTAMP",
+ ),
+ ]
+ for value, expected_type in expected_types:
+ msg = "value: {} expected_type: {}".format(value, expected_type)
+ parameter = _helpers.scalar_to_query_parameter(value)
+ self.assertIsNone(parameter.name, msg=msg)
+ self.assertEqual(parameter.type_, expected_type, msg=msg)
+ self.assertEqual(parameter.value, value, msg=msg)
+ named_parameter = _helpers.scalar_to_query_parameter(value, name="myvar")
+ self.assertEqual(named_parameter.name, "myvar", msg=msg)
+ self.assertEqual(named_parameter.type_, expected_type, msg=msg)
+ self.assertEqual(named_parameter.value, value, msg=msg)
+
+ def test_scalar_to_query_parameter_w_unexpected_type(self):
+ with self.assertRaises(exceptions.ProgrammingError):
+ _helpers.scalar_to_query_parameter(value={"a": "dictionary"})
+
+ def test_scalar_to_query_parameter_w_special_floats(self):
+ nan_parameter = _helpers.scalar_to_query_parameter(float("nan"))
+ self.assertTrue(math.isnan(nan_parameter.value))
+ self.assertEqual(nan_parameter.type_, "FLOAT64")
+ inf_parameter = _helpers.scalar_to_query_parameter(float("inf"))
+ self.assertTrue(math.isinf(inf_parameter.value))
+ self.assertEqual(inf_parameter.type_, "FLOAT64")
+
+ def test_to_query_parameters_w_dict(self):
+ parameters = {"somebool": True, "somestring": u"a-string-value"}
+ query_parameters = _helpers.to_query_parameters(parameters)
+ query_parameter_tuples = []
+ for param in query_parameters:
+ query_parameter_tuples.append((param.name, param.type_, param.value))
+ self.assertSequenceEqual(
+ sorted(query_parameter_tuples),
+ sorted(
+ [
+ ("somebool", "BOOL", True),
+ ("somestring", "STRING", u"a-string-value"),
+ ]
+ ),
+ )
+
+ def test_to_query_parameters_w_list(self):
+ parameters = [True, u"a-string-value"]
+ query_parameters = _helpers.to_query_parameters(parameters)
+ query_parameter_tuples = []
+ for param in query_parameters:
+ query_parameter_tuples.append((param.name, param.type_, param.value))
+ self.assertSequenceEqual(
+ sorted(query_parameter_tuples),
+ sorted([(None, "BOOL", True), (None, "STRING", u"a-string-value")]),
+ )
diff --git a/bigquery/tests/unit/test_dbapi_connection.py b/bigquery/tests/unit/test_dbapi_connection.py
new file mode 100644
index 000000000000..19acec05bd34
--- /dev/null
+++ b/bigquery/tests/unit/test_dbapi_connection.py
@@ -0,0 +1,78 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import unittest
+
+import mock
+
+
+class TestConnection(unittest.TestCase):
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery.dbapi import Connection
+
+ return Connection
+
+ def _make_one(self, *args, **kw):
+ return self._get_target_class()(*args, **kw)
+
+ def _mock_client(self, rows=None, schema=None):
+ from google.cloud.bigquery import client
+
+ mock_client = mock.create_autospec(client.Client)
+ return mock_client
+
+ def test_ctor(self):
+ from google.cloud.bigquery.dbapi import Connection
+
+ mock_client = self._mock_client()
+ connection = self._make_one(client=mock_client)
+ self.assertIsInstance(connection, Connection)
+ self.assertIs(connection._client, mock_client)
+
+ @mock.patch("google.cloud.bigquery.Client", autospec=True)
+ def test_connect_wo_client(self, mock_client):
+ from google.cloud.bigquery.dbapi import connect
+ from google.cloud.bigquery.dbapi import Connection
+
+ connection = connect()
+ self.assertIsInstance(connection, Connection)
+ self.assertIsNotNone(connection._client)
+
+ def test_connect_w_client(self):
+ from google.cloud.bigquery.dbapi import connect
+ from google.cloud.bigquery.dbapi import Connection
+
+ mock_client = self._mock_client()
+ connection = connect(client=mock_client)
+ self.assertIsInstance(connection, Connection)
+ self.assertIs(connection._client, mock_client)
+
+ def test_close(self):
+ connection = self._make_one(client=self._mock_client())
+ # close() is a no-op, there is nothing to test.
+ connection.close()
+
+ def test_commit(self):
+ connection = self._make_one(client=self._mock_client())
+ # commit() is a no-op, there is nothing to test.
+ connection.commit()
+
+ def test_cursor(self):
+ from google.cloud.bigquery.dbapi import Cursor
+
+ connection = self._make_one(client=self._mock_client())
+ cursor = connection.cursor()
+ self.assertIsInstance(cursor, Cursor)
+ self.assertIs(cursor.connection, connection)
diff --git a/bigquery/tests/unit/test_dbapi_cursor.py b/bigquery/tests/unit/test_dbapi_cursor.py
new file mode 100644
index 000000000000..4a675c73958d
--- /dev/null
+++ b/bigquery/tests/unit/test_dbapi_cursor.py
@@ -0,0 +1,318 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import unittest
+
+import mock
+
+
+class TestCursor(unittest.TestCase):
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery.dbapi import Cursor
+
+ return Cursor
+
+ def _make_one(self, *args, **kw):
+ return self._get_target_class()(*args, **kw)
+
+ def _mock_client(self, rows=None, schema=None, num_dml_affected_rows=None):
+ from google.cloud.bigquery import client
+
+ if rows is None:
+ total_rows = 0
+ else:
+ total_rows = len(rows)
+
+ mock_client = mock.create_autospec(client.Client)
+ mock_client.query.return_value = self._mock_job(
+ total_rows=total_rows,
+ schema=schema,
+ num_dml_affected_rows=num_dml_affected_rows,
+ )
+ mock_client.list_rows.return_value = rows
+ return mock_client
+
+ def _mock_job(self, total_rows=0, schema=None, num_dml_affected_rows=None):
+ from google.cloud.bigquery import job
+
+ mock_job = mock.create_autospec(job.QueryJob)
+ mock_job.error_result = None
+ mock_job.state = "DONE"
+ mock_job.result.return_value = mock_job
+ mock_job._query_results = self._mock_results(
+ total_rows=total_rows,
+ schema=schema,
+ num_dml_affected_rows=num_dml_affected_rows,
+ )
+
+ if num_dml_affected_rows is None:
+ mock_job.statement_type = None # API sends back None for SELECT
+ else:
+ mock_job.statement_type = "UPDATE"
+
+ return mock_job
+
+ def _mock_results(self, total_rows=0, schema=None, num_dml_affected_rows=None):
+ from google.cloud.bigquery import query
+
+ mock_results = mock.create_autospec(query._QueryResults)
+ mock_results.schema = schema
+ mock_results.num_dml_affected_rows = num_dml_affected_rows
+ mock_results.total_rows = total_rows
+ return mock_results
+
+ def test_ctor(self):
+ from google.cloud.bigquery.dbapi import connect
+ from google.cloud.bigquery.dbapi import Cursor
+
+ connection = connect(self._mock_client())
+ cursor = self._make_one(connection)
+ self.assertIsInstance(cursor, Cursor)
+ self.assertIs(cursor.connection, connection)
+
+ def test_close(self):
+ from google.cloud.bigquery.dbapi import connect
+
+ connection = connect(self._mock_client())
+ cursor = connection.cursor()
+ # close() is a no-op, there is nothing to test.
+ cursor.close()
+
+ def test_fetchone_wo_execute_raises_error(self):
+ from google.cloud.bigquery import dbapi
+
+ connection = dbapi.connect(self._mock_client())
+ cursor = connection.cursor()
+ self.assertRaises(dbapi.Error, cursor.fetchone)
+
+ def test_fetchone_w_row(self):
+ from google.cloud.bigquery import dbapi
+
+ connection = dbapi.connect(self._mock_client(rows=[(1,)]))
+ cursor = connection.cursor()
+ cursor.execute("SELECT 1;")
+ row = cursor.fetchone()
+ self.assertEqual(row, (1,))
+ self.assertIsNone(cursor.fetchone())
+
+ def test_fetchmany_wo_execute_raises_error(self):
+ from google.cloud.bigquery import dbapi
+
+ connection = dbapi.connect(self._mock_client())
+ cursor = connection.cursor()
+ self.assertRaises(dbapi.Error, cursor.fetchmany)
+
+ def test_fetchmany_w_row(self):
+ from google.cloud.bigquery import dbapi
+
+ connection = dbapi.connect(self._mock_client(rows=[(1,)]))
+ cursor = connection.cursor()
+ cursor.execute("SELECT 1;")
+ rows = cursor.fetchmany()
+ self.assertEqual(len(rows), 1)
+ self.assertEqual(rows[0], (1,))
+
+ def test_fetchmany_w_size(self):
+ from google.cloud.bigquery import dbapi
+
+ connection = dbapi.connect(
+ self._mock_client(rows=[(1, 2, 3), (4, 5, 6), (7, 8, 9)])
+ )
+ cursor = connection.cursor()
+ cursor.execute("SELECT a, b, c;")
+ rows = cursor.fetchmany(size=2)
+ self.assertEqual(len(rows), 2)
+ self.assertEqual(rows[0], (1, 2, 3))
+ self.assertEqual(rows[1], (4, 5, 6))
+ second_page = cursor.fetchmany(size=2)
+ self.assertEqual(len(second_page), 1)
+ self.assertEqual(second_page[0], (7, 8, 9))
+ third_page = cursor.fetchmany(size=2)
+ self.assertEqual(third_page, [])
+
+ def test_fetchmany_w_arraysize(self):
+ from google.cloud.bigquery import dbapi
+
+ connection = dbapi.connect(
+ self._mock_client(rows=[(1, 2, 3), (4, 5, 6), (7, 8, 9)])
+ )
+ cursor = connection.cursor()
+ cursor.execute("SELECT a, b, c;")
+ cursor.arraysize = 2
+ rows = cursor.fetchmany()
+ self.assertEqual(len(rows), 2)
+ self.assertEqual(rows[0], (1, 2, 3))
+ self.assertEqual(rows[1], (4, 5, 6))
+ second_page = cursor.fetchmany()
+ self.assertEqual(len(second_page), 1)
+ self.assertEqual(second_page[0], (7, 8, 9))
+ third_page = cursor.fetchmany()
+ self.assertEqual(third_page, [])
+
+ def test_fetchall_wo_execute_raises_error(self):
+ from google.cloud.bigquery import dbapi
+
+ connection = dbapi.connect(self._mock_client())
+ cursor = connection.cursor()
+ self.assertRaises(dbapi.Error, cursor.fetchall)
+
+ def test_fetchall_w_row(self):
+ from google.cloud.bigquery import dbapi
+
+ connection = dbapi.connect(self._mock_client(rows=[(1,)]))
+ cursor = connection.cursor()
+ cursor.execute("SELECT 1;")
+ self.assertIsNone(cursor.description)
+ self.assertEqual(cursor.rowcount, 1)
+ rows = cursor.fetchall()
+ self.assertEqual(len(rows), 1)
+ self.assertEqual(rows[0], (1,))
+
+ def test_execute_custom_job_id(self):
+ from google.cloud.bigquery.dbapi import connect
+
+ client = self._mock_client(rows=[], num_dml_affected_rows=0)
+ connection = connect(client)
+ cursor = connection.cursor()
+ cursor.execute("SELECT 1;", job_id="foo")
+ args, kwargs = client.query.call_args
+ self.assertEqual(args[0], "SELECT 1;")
+ self.assertEqual(kwargs["job_id"], "foo")
+
+ def test_execute_w_dml(self):
+ from google.cloud.bigquery.dbapi import connect
+
+ connection = connect(self._mock_client(rows=[], num_dml_affected_rows=12))
+ cursor = connection.cursor()
+ cursor.execute("DELETE FROM UserSessions WHERE user_id = 'test';")
+ rows = cursor.fetchall()
+ self.assertIsNone(cursor.description)
+ self.assertEqual(cursor.rowcount, 12)
+ self.assertEqual(rows, [])
+
+ def test_execute_w_query(self):
+ from google.cloud.bigquery.schema import SchemaField
+ from google.cloud.bigquery import dbapi
+
+ connection = dbapi.connect(
+ self._mock_client(
+ rows=[("hello", "world", 1), ("howdy", "y'all", 2)],
+ schema=[
+ SchemaField("a", "STRING", mode="NULLABLE"),
+ SchemaField("b", "STRING", mode="REQUIRED"),
+ SchemaField("c", "INTEGER", mode="NULLABLE"),
+ ],
+ )
+ )
+ cursor = connection.cursor()
+ cursor.execute("SELECT a, b, c FROM hello_world WHERE d > 3;")
+
+ # Verify the description.
+ self.assertEqual(len(cursor.description), 3)
+ a_name, a_type, _, _, _, _, a_null_ok = cursor.description[0]
+ self.assertEqual(a_name, "a")
+ self.assertEqual(a_type, "STRING")
+ self.assertEqual(a_type, dbapi.STRING)
+ self.assertTrue(a_null_ok)
+ b_name, b_type, _, _, _, _, b_null_ok = cursor.description[1]
+ self.assertEqual(b_name, "b")
+ self.assertEqual(b_type, "STRING")
+ self.assertEqual(b_type, dbapi.STRING)
+ self.assertFalse(b_null_ok)
+ c_name, c_type, _, _, _, _, c_null_ok = cursor.description[2]
+ self.assertEqual(c_name, "c")
+ self.assertEqual(c_type, "INTEGER")
+ self.assertEqual(c_type, dbapi.NUMBER)
+ self.assertTrue(c_null_ok)
+
+ # Verify the results.
+ self.assertEqual(cursor.rowcount, 2)
+ row = cursor.fetchone()
+ self.assertEqual(row, ("hello", "world", 1))
+ row = cursor.fetchone()
+ self.assertEqual(row, ("howdy", "y'all", 2))
+ row = cursor.fetchone()
+ self.assertIsNone(row)
+
+ def test_execute_raises_if_result_raises(self):
+ import google.cloud.exceptions
+
+ from google.cloud.bigquery import client
+ from google.cloud.bigquery import job
+ from google.cloud.bigquery.dbapi import connect
+ from google.cloud.bigquery.dbapi import exceptions
+
+ job = mock.create_autospec(job.QueryJob)
+ job.result.side_effect = google.cloud.exceptions.GoogleCloudError("")
+ client = mock.create_autospec(client.Client)
+ client.query.return_value = job
+ connection = connect(client)
+ cursor = connection.cursor()
+
+ with self.assertRaises(exceptions.DatabaseError):
+ cursor.execute("SELECT 1")
+
+ def test_executemany_w_dml(self):
+ from google.cloud.bigquery.dbapi import connect
+
+ connection = connect(self._mock_client(rows=[], num_dml_affected_rows=12))
+ cursor = connection.cursor()
+ cursor.executemany(
+ "DELETE FROM UserSessions WHERE user_id = %s;",
+ (("test",), ("anothertest",)),
+ )
+ self.assertIsNone(cursor.description)
+ self.assertEqual(cursor.rowcount, 12)
+
+ def test__format_operation_w_dict(self):
+ from google.cloud.bigquery.dbapi import cursor
+
+ formatted_operation = cursor._format_operation(
+ "SELECT %(somevalue)s, %(a `weird` one)s;",
+ {"somevalue": "hi", "a `weird` one": "world"},
+ )
+ self.assertEqual(
+ formatted_operation, "SELECT @`somevalue`, @`a \\`weird\\` one`;"
+ )
+
+ def test__format_operation_w_wrong_dict(self):
+ from google.cloud.bigquery import dbapi
+ from google.cloud.bigquery.dbapi import cursor
+
+ self.assertRaises(
+ dbapi.ProgrammingError,
+ cursor._format_operation,
+ "SELECT %(somevalue)s, %(othervalue)s;",
+ {"somevalue-not-here": "hi", "othervalue": "world"},
+ )
+
+ def test__format_operation_w_sequence(self):
+ from google.cloud.bigquery.dbapi import cursor
+
+ formatted_operation = cursor._format_operation(
+ "SELECT %s, %s;", ("hello", "world")
+ )
+ self.assertEqual(formatted_operation, "SELECT ?, ?;")
+
+ def test__format_operation_w_too_short_sequence(self):
+ from google.cloud.bigquery import dbapi
+ from google.cloud.bigquery.dbapi import cursor
+
+ self.assertRaises(
+ dbapi.ProgrammingError,
+ cursor._format_operation,
+ "SELECT %s, %s;",
+ ("hello",),
+ )
diff --git a/bigquery/tests/unit/test_dbapi_types.py b/bigquery/tests/unit/test_dbapi_types.py
new file mode 100644
index 000000000000..e05660ffed14
--- /dev/null
+++ b/bigquery/tests/unit/test_dbapi_types.py
@@ -0,0 +1,42 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import datetime
+import unittest
+
+import google.cloud._helpers
+from google.cloud.bigquery.dbapi import types
+
+
+class TestTypes(unittest.TestCase):
+ def test_binary_type(self):
+ self.assertEqual("BYTES", types.BINARY)
+ self.assertEqual("RECORD", types.BINARY)
+ self.assertEqual("STRUCT", types.BINARY)
+ self.assertNotEqual("STRING", types.BINARY)
+
+ def test_binary_constructor(self):
+ self.assertEqual(types.Binary(u"hello"), b"hello")
+ self.assertEqual(types.Binary(u"\u1f60"), u"\u1f60".encode("utf-8"))
+
+ def test_timefromticks(self):
+ somedatetime = datetime.datetime(
+ 2017, 2, 18, 12, 47, 26, tzinfo=google.cloud._helpers.UTC
+ )
+ epoch = datetime.datetime(1970, 1, 1, tzinfo=google.cloud._helpers.UTC)
+ ticks = (somedatetime - epoch).total_seconds()
+ self.assertEqual(
+ types.TimeFromTicks(ticks, google.cloud._helpers.UTC),
+ datetime.time(12, 47, 26, tzinfo=google.cloud._helpers.UTC),
+ )
diff --git a/bigquery/tests/unit/test_external_config.py b/bigquery/tests/unit/test_external_config.py
new file mode 100644
index 000000000000..ddf95e317969
--- /dev/null
+++ b/bigquery/tests/unit/test_external_config.py
@@ -0,0 +1,368 @@
+# Copyright 2017 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import base64
+import copy
+import unittest
+
+from google.cloud.bigquery import external_config
+from google.cloud.bigquery import schema
+
+
+class TestExternalConfig(unittest.TestCase):
+
+ SOURCE_URIS = ["gs://foo", "gs://bar"]
+
+ BASE_RESOURCE = {
+ "sourceFormat": "",
+ "sourceUris": SOURCE_URIS,
+ "maxBadRecords": 17,
+ "autodetect": True,
+ "ignoreUnknownValues": False,
+ "compression": "compression",
+ }
+
+ def test_from_api_repr_base(self):
+ resource = copy.deepcopy(self.BASE_RESOURCE)
+ ec = external_config.ExternalConfig.from_api_repr(resource)
+ self._verify_base(ec)
+ self.assertEqual(ec.schema, [])
+ self.assertIsNone(ec.options)
+
+ got_resource = ec.to_api_repr()
+ self.assertEqual(got_resource, self.BASE_RESOURCE)
+
+ resource = _copy_and_update(
+ self.BASE_RESOURCE,
+ {
+ "schema": {
+ "fields": [
+ {
+ "name": "full_name",
+ "type": "STRING",
+ "mode": "REQUIRED",
+ "description": None,
+ }
+ ]
+ }
+ },
+ )
+ ec = external_config.ExternalConfig.from_api_repr(resource)
+ self._verify_base(ec)
+ exp_schema = [schema.SchemaField("full_name", "STRING", mode="REQUIRED")]
+ self.assertEqual(ec.schema, exp_schema)
+ self.assertIsNone(ec.options)
+
+ got_resource = ec.to_api_repr()
+ self.assertEqual(got_resource, resource)
+
+ def test_to_api_repr_base(self):
+ ec = external_config.ExternalConfig("")
+ ec.source_uris = self.SOURCE_URIS
+ ec.max_bad_records = 17
+ ec.autodetect = True
+ ec.ignore_unknown_values = False
+ ec.compression = "compression"
+ ec.schema = [schema.SchemaField("full_name", "STRING", mode="REQUIRED")]
+
+ exp_schema = {
+ "fields": [
+ {
+ "name": "full_name",
+ "type": "STRING",
+ "mode": "REQUIRED",
+ "description": None,
+ }
+ ]
+ }
+ got_resource = ec.to_api_repr()
+ exp_resource = {
+ "sourceFormat": "",
+ "sourceUris": self.SOURCE_URIS,
+ "maxBadRecords": 17,
+ "autodetect": True,
+ "ignoreUnknownValues": False,
+ "compression": "compression",
+ "schema": exp_schema,
+ }
+ self.assertEqual(got_resource, exp_resource)
+
+ def test_schema_None(self):
+ ec = external_config.ExternalConfig("")
+ ec.schema = None
+ got = ec.to_api_repr()
+ want = {"sourceFormat": "", "schema": None}
+ self.assertEqual(got, want)
+
+ def test_schema_empty(self):
+ ec = external_config.ExternalConfig("")
+ ec.schema = []
+ got = ec.to_api_repr()
+ want = {"sourceFormat": "", "schema": {"fields": []}}
+ self.assertEqual(got, want)
+
+ def _verify_base(self, ec):
+ self.assertEqual(ec.autodetect, True)
+ self.assertEqual(ec.compression, "compression")
+ self.assertEqual(ec.ignore_unknown_values, False)
+ self.assertEqual(ec.max_bad_records, 17)
+ self.assertEqual(ec.source_uris, self.SOURCE_URIS)
+
+ def test_to_api_repr_source_format(self):
+ ec = external_config.ExternalConfig("CSV")
+ got = ec.to_api_repr()
+ want = {"sourceFormat": "CSV"}
+ self.assertEqual(got, want)
+
+ def test_from_api_repr_sheets(self):
+ resource = _copy_and_update(
+ self.BASE_RESOURCE,
+ {
+ "sourceFormat": "GOOGLE_SHEETS",
+ "googleSheetsOptions": {"skipLeadingRows": "123"},
+ },
+ )
+
+ ec = external_config.ExternalConfig.from_api_repr(resource)
+
+ self._verify_base(ec)
+ self.assertEqual(ec.source_format, "GOOGLE_SHEETS")
+ self.assertIsInstance(ec.options, external_config.GoogleSheetsOptions)
+ self.assertEqual(ec.options.skip_leading_rows, 123)
+
+ got_resource = ec.to_api_repr()
+
+ self.assertEqual(got_resource, resource)
+
+ del resource["googleSheetsOptions"]["skipLeadingRows"]
+ ec = external_config.ExternalConfig.from_api_repr(resource)
+ self.assertIsNone(ec.options.skip_leading_rows)
+ got_resource = ec.to_api_repr()
+ self.assertEqual(got_resource, resource)
+
+ def test_to_api_repr_sheets(self):
+ ec = external_config.ExternalConfig("GOOGLE_SHEETS")
+ options = external_config.GoogleSheetsOptions()
+ options.skip_leading_rows = 123
+ ec._options = options
+
+ exp_resource = {
+ "sourceFormat": "GOOGLE_SHEETS",
+ "googleSheetsOptions": {"skipLeadingRows": "123"},
+ }
+
+ got_resource = ec.to_api_repr()
+
+ self.assertEqual(got_resource, exp_resource)
+
+ def test_from_api_repr_csv(self):
+ resource = _copy_and_update(
+ self.BASE_RESOURCE,
+ {
+ "sourceFormat": "CSV",
+ "csvOptions": {
+ "fieldDelimiter": "fieldDelimiter",
+ "skipLeadingRows": "123",
+ "quote": "quote",
+ "allowQuotedNewlines": True,
+ "allowJaggedRows": False,
+ "encoding": "encoding",
+ },
+ },
+ )
+
+ ec = external_config.ExternalConfig.from_api_repr(resource)
+
+ self._verify_base(ec)
+ self.assertEqual(ec.source_format, "CSV")
+ self.assertIsInstance(ec.options, external_config.CSVOptions)
+ self.assertEqual(ec.options.field_delimiter, "fieldDelimiter")
+ self.assertEqual(ec.options.skip_leading_rows, 123)
+ self.assertEqual(ec.options.quote_character, "quote")
+ self.assertEqual(ec.options.allow_quoted_newlines, True)
+ self.assertEqual(ec.options.allow_jagged_rows, False)
+ self.assertEqual(ec.options.encoding, "encoding")
+
+ got_resource = ec.to_api_repr()
+
+ self.assertEqual(got_resource, resource)
+
+ del resource["csvOptions"]["skipLeadingRows"]
+ ec = external_config.ExternalConfig.from_api_repr(resource)
+ self.assertIsNone(ec.options.skip_leading_rows)
+ got_resource = ec.to_api_repr()
+ self.assertEqual(got_resource, resource)
+
+ def test_to_api_repr_csv(self):
+ ec = external_config.ExternalConfig("CSV")
+ options = external_config.CSVOptions()
+ options.allow_quoted_newlines = True
+ options.encoding = "encoding"
+ options.field_delimiter = "fieldDelimiter"
+ options.quote_character = "quote"
+ options.skip_leading_rows = 123
+ options.allow_jagged_rows = False
+ ec._options = options
+
+ exp_resource = {
+ "sourceFormat": "CSV",
+ "csvOptions": {
+ "fieldDelimiter": "fieldDelimiter",
+ "skipLeadingRows": "123",
+ "quote": "quote",
+ "allowQuotedNewlines": True,
+ "allowJaggedRows": False,
+ "encoding": "encoding",
+ },
+ }
+
+ got_resource = ec.to_api_repr()
+
+ self.assertEqual(got_resource, exp_resource)
+
+ def test_from_api_repr_bigtable(self):
+ qualifier_encoded = base64.standard_b64encode(b"q").decode("ascii")
+ resource = _copy_and_update(
+ self.BASE_RESOURCE,
+ {
+ "sourceFormat": "BIGTABLE",
+ "bigtableOptions": {
+ "ignoreUnspecifiedColumnFamilies": True,
+ "readRowkeyAsString": False,
+ "columnFamilies": [
+ {
+ "familyId": "familyId",
+ "type": "type",
+ "encoding": "encoding",
+ "columns": [
+ {
+ "qualifierString": "q",
+ "fieldName": "fieldName1",
+ "type": "type1",
+ "encoding": "encoding1",
+ "onlyReadLatest": True,
+ },
+ {
+ "qualifierEncoded": qualifier_encoded,
+ "fieldName": "fieldName2",
+ "type": "type2",
+ "encoding": "encoding2",
+ },
+ ],
+ "onlyReadLatest": False,
+ }
+ ],
+ },
+ },
+ )
+
+ ec = external_config.ExternalConfig.from_api_repr(resource)
+
+ self._verify_base(ec)
+ self.assertEqual(ec.source_format, "BIGTABLE")
+ self.assertIsInstance(ec.options, external_config.BigtableOptions)
+ self.assertEqual(ec.options.ignore_unspecified_column_families, True)
+ self.assertEqual(ec.options.read_rowkey_as_string, False)
+ self.assertEqual(len(ec.options.column_families), 1)
+ fam1 = ec.options.column_families[0]
+ self.assertIsInstance(fam1, external_config.BigtableColumnFamily)
+ self.assertEqual(fam1.family_id, "familyId")
+ self.assertEqual(fam1.type_, "type")
+ self.assertEqual(fam1.encoding, "encoding")
+ self.assertEqual(len(fam1.columns), 2)
+ self.assertFalse(fam1.only_read_latest)
+ col1 = fam1.columns[0]
+ self.assertEqual(col1.qualifier_string, "q")
+ self.assertEqual(col1.field_name, "fieldName1")
+ self.assertEqual(col1.type_, "type1")
+ self.assertEqual(col1.encoding, "encoding1")
+ self.assertTrue(col1.only_read_latest)
+ self.assertIsNone(col1.qualifier_encoded)
+ col2 = ec.options.column_families[0].columns[1]
+ self.assertEqual(col2.qualifier_encoded, b"q")
+ self.assertEqual(col2.field_name, "fieldName2")
+ self.assertEqual(col2.type_, "type2")
+ self.assertEqual(col2.encoding, "encoding2")
+
+ got_resource = ec.to_api_repr()
+
+ self.assertEqual(got_resource, resource)
+
+ def test_to_api_repr_bigtable(self):
+ ec = external_config.ExternalConfig("BIGTABLE")
+ options = external_config.BigtableOptions()
+ options.ignore_unspecified_column_families = True
+ options.read_rowkey_as_string = False
+ ec._options = options
+
+ fam1 = external_config.BigtableColumnFamily()
+ fam1.family_id = "familyId"
+ fam1.type_ = "type"
+ fam1.encoding = "encoding"
+ fam1.only_read_latest = False
+ col1 = external_config.BigtableColumn()
+ col1.qualifier_string = "q"
+ col1.field_name = "fieldName1"
+ col1.type_ = "type1"
+ col1.encoding = "encoding1"
+ col1.only_read_latest = True
+ col2 = external_config.BigtableColumn()
+ col2.qualifier_encoded = b"q"
+ col2.field_name = "fieldName2"
+ col2.type_ = "type2"
+ col2.encoding = "encoding2"
+ fam1.columns = [col1, col2]
+ options.column_families = [fam1]
+
+ qualifier_encoded = base64.standard_b64encode(b"q").decode("ascii")
+ exp_resource = {
+ "sourceFormat": "BIGTABLE",
+ "bigtableOptions": {
+ "ignoreUnspecifiedColumnFamilies": True,
+ "readRowkeyAsString": False,
+ "columnFamilies": [
+ {
+ "familyId": "familyId",
+ "type": "type",
+ "encoding": "encoding",
+ "columns": [
+ {
+ "qualifierString": "q",
+ "fieldName": "fieldName1",
+ "type": "type1",
+ "encoding": "encoding1",
+ "onlyReadLatest": True,
+ },
+ {
+ "qualifierEncoded": qualifier_encoded,
+ "fieldName": "fieldName2",
+ "type": "type2",
+ "encoding": "encoding2",
+ },
+ ],
+ "onlyReadLatest": False,
+ }
+ ],
+ },
+ }
+
+ got_resource = ec.to_api_repr()
+
+ self.assertEqual(got_resource, exp_resource)
+
+
+def _copy_and_update(d, u):
+ d = copy.deepcopy(d)
+ d.update(u)
+ return d
diff --git a/bigquery/tests/unit/test_job.py b/bigquery/tests/unit/test_job.py
new file mode 100644
index 000000000000..19409d8d43c3
--- /dev/null
+++ b/bigquery/tests/unit/test_job.py
@@ -0,0 +1,5411 @@
+# Copyright 2015 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import copy
+import json
+import textwrap
+import unittest
+
+import mock
+import pytest
+from six.moves import http_client
+
+try:
+ import pandas
+except (ImportError, AttributeError): # pragma: NO COVER
+ pandas = None
+
+try:
+ import pyarrow
+except ImportError: # pragma: NO COVER
+ pyarrow = None
+try:
+ from google.cloud import bigquery_storage_v1beta1
+except (ImportError, AttributeError): # pragma: NO COVER
+ bigquery_storage_v1beta1 = None
+try:
+ from tqdm import tqdm
+except (ImportError, AttributeError): # pragma: NO COVER
+ tqdm = None
+
+
+def _make_credentials():
+ import google.auth.credentials
+
+ return mock.Mock(spec=google.auth.credentials.Credentials)
+
+
+def _make_client(project="test-project", connection=None):
+ from google.cloud.bigquery.client import Client
+
+ if connection is None:
+ connection = _make_connection()
+
+ client = Client(project=project, credentials=_make_credentials(), _http=object())
+ client._connection = connection
+ return client
+
+
+def _make_connection(*responses):
+ import google.cloud.bigquery._http
+ from google.cloud.exceptions import NotFound
+
+ mock_conn = mock.create_autospec(google.cloud.bigquery._http.Connection)
+ mock_conn.api_request.side_effect = list(responses) + [NotFound("miss")]
+ return mock_conn
+
+
+def _make_job_resource(
+ creation_time_ms=1437767599006,
+ started_time_ms=1437767600007,
+ ended_time_ms=1437767601008,
+ started=False,
+ ended=False,
+ etag="abc-def-hjk",
+ endpoint="https://www.googleapis.com",
+ job_type="load",
+ job_id="a-random-id",
+ project_id="some-project",
+ user_email="bq-user@example.com",
+):
+ resource = {
+ "configuration": {job_type: {}},
+ "statistics": {"creationTime": creation_time_ms, job_type: {}},
+ "etag": etag,
+ "id": "{}:{}".format(project_id, job_id),
+ "jobReference": {"projectId": project_id, "jobId": job_id},
+ "selfLink": "{}/bigquery/v2/projects/{}/jobs/{}".format(
+ endpoint, project_id, job_id
+ ),
+ "user_email": user_email,
+ }
+
+ if started or ended:
+ resource["statistics"]["startTime"] = started_time_ms
+
+ if ended:
+ resource["statistics"]["endTime"] = ended_time_ms
+
+ if job_type == "query":
+ resource["configuration"]["query"]["destinationTable"] = {
+ "projectId": project_id,
+ "datasetId": "_temp_dataset",
+ "tableId": "_temp_table",
+ }
+
+ return resource
+
+
+class Test__error_result_to_exception(unittest.TestCase):
+ def _call_fut(self, *args, **kwargs):
+ from google.cloud.bigquery import job
+
+ return job._error_result_to_exception(*args, **kwargs)
+
+ def test_simple(self):
+ error_result = {"reason": "invalid", "message": "bad request"}
+ exception = self._call_fut(error_result)
+ self.assertEqual(exception.code, http_client.BAD_REQUEST)
+ self.assertTrue(exception.message.startswith("bad request"))
+ self.assertIn(error_result, exception.errors)
+
+ def test_missing_reason(self):
+ error_result = {}
+ exception = self._call_fut(error_result)
+ self.assertEqual(exception.code, http_client.INTERNAL_SERVER_ERROR)
+
+
+class Test_JobReference(unittest.TestCase):
+ JOB_ID = "job-id"
+ PROJECT = "test-project-123"
+ LOCATION = "us-central"
+
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery import job
+
+ return job._JobReference
+
+ def _make_one(self, job_id, project, location):
+ return self._get_target_class()(job_id, project, location)
+
+ def test_ctor(self):
+ job_ref = self._make_one(self.JOB_ID, self.PROJECT, self.LOCATION)
+
+ self.assertEqual(job_ref.job_id, self.JOB_ID)
+ self.assertEqual(job_ref.project, self.PROJECT)
+ self.assertEqual(job_ref.location, self.LOCATION)
+
+ def test__to_api_repr(self):
+ job_ref = self._make_one(self.JOB_ID, self.PROJECT, self.LOCATION)
+
+ self.assertEqual(
+ job_ref._to_api_repr(),
+ {
+ "jobId": self.JOB_ID,
+ "projectId": self.PROJECT,
+ "location": self.LOCATION,
+ },
+ )
+
+ def test_from_api_repr(self):
+ api_repr = {
+ "jobId": self.JOB_ID,
+ "projectId": self.PROJECT,
+ "location": self.LOCATION,
+ }
+
+ job_ref = self._get_target_class()._from_api_repr(api_repr)
+
+ self.assertEqual(job_ref.job_id, self.JOB_ID)
+ self.assertEqual(job_ref.project, self.PROJECT)
+ self.assertEqual(job_ref.location, self.LOCATION)
+
+
+class Test_AsyncJob(unittest.TestCase):
+ JOB_ID = "job-id"
+ PROJECT = "test-project-123"
+ LOCATION = "us-central"
+
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery import job
+
+ return job._AsyncJob
+
+ def _make_one(self, job_id, client):
+ return self._get_target_class()(job_id, client)
+
+ def _make_derived_class(self):
+ class Derived(self._get_target_class()):
+ _JOB_TYPE = "derived"
+
+ return Derived
+
+ def _make_derived(self, job_id, client):
+ return self._make_derived_class()(job_id, client)
+
+ @staticmethod
+ def _job_reference(job_id, project, location):
+ from google.cloud.bigquery import job
+
+ return job._JobReference(job_id, project, location)
+
+ def test_ctor_w_bare_job_id(self):
+ import threading
+
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+
+ self.assertEqual(job.job_id, self.JOB_ID)
+ self.assertEqual(job.project, self.PROJECT)
+ self.assertIsNone(job.location)
+ self.assertIs(job._client, client)
+ self.assertEqual(
+ job._properties,
+ {"jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID}},
+ )
+ self.assertIsInstance(job._completion_lock, type(threading.Lock()))
+ self.assertEqual(
+ job.path, "/projects/{}/jobs/{}".format(self.PROJECT, self.JOB_ID)
+ )
+
+ def test_ctor_w_job_ref(self):
+ import threading
+
+ other_project = "other-project-234"
+ client = _make_client(project=other_project)
+ job_ref = self._job_reference(self.JOB_ID, self.PROJECT, self.LOCATION)
+ job = self._make_one(job_ref, client)
+
+ self.assertEqual(job.job_id, self.JOB_ID)
+ self.assertEqual(job.project, self.PROJECT)
+ self.assertEqual(job.location, self.LOCATION)
+ self.assertIs(job._client, client)
+ self.assertEqual(
+ job._properties,
+ {
+ "jobReference": {
+ "projectId": self.PROJECT,
+ "location": self.LOCATION,
+ "jobId": self.JOB_ID,
+ }
+ },
+ )
+ self.assertFalse(job._result_set)
+ self.assertIsInstance(job._completion_lock, type(threading.Lock()))
+ self.assertEqual(
+ job.path, "/projects/{}/jobs/{}".format(self.PROJECT, self.JOB_ID)
+ )
+
+ def test__require_client_w_none(self):
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+
+ self.assertIs(job._require_client(None), client)
+
+ def test__require_client_w_other(self):
+ client = _make_client(project=self.PROJECT)
+ other = object()
+ job = self._make_one(self.JOB_ID, client)
+
+ self.assertIs(job._require_client(other), other)
+
+ def test_job_type(self):
+ client = _make_client(project=self.PROJECT)
+ derived = self._make_derived(self.JOB_ID, client)
+
+ self.assertEqual(derived.job_type, "derived")
+
+ def test_labels_miss(self):
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+ self.assertEqual(job.labels, {})
+
+ def test_labels_update_in_place(self):
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+ labels = job.labels
+ labels["foo"] = "bar" # update in place
+ self.assertEqual(job.labels, {"foo": "bar"})
+
+ def test_labels_hit(self):
+ labels = {"foo": "bar"}
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+ job._properties["labels"] = labels
+ self.assertEqual(job.labels, labels)
+
+ def test_etag(self):
+ etag = "ETAG-123"
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+ self.assertIsNone(job.etag)
+ job._properties["etag"] = etag
+ self.assertEqual(job.etag, etag)
+
+ def test_self_link(self):
+ self_link = "https://api.example.com/123"
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+ self.assertIsNone(job.self_link)
+ job._properties["selfLink"] = self_link
+ self.assertEqual(job.self_link, self_link)
+
+ def test_user_email(self):
+ user_email = "user@example.com"
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+ self.assertIsNone(job.user_email)
+ job._properties["user_email"] = user_email
+ self.assertEqual(job.user_email, user_email)
+
+ @staticmethod
+ def _datetime_and_millis():
+ import datetime
+ import pytz
+ from google.cloud._helpers import _millis
+
+ now = datetime.datetime.utcnow().replace(
+ microsecond=123000, tzinfo=pytz.UTC # stats timestamps have ms precision
+ )
+ return now, _millis(now)
+
+ def test_created(self):
+ now, millis = self._datetime_and_millis()
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+ self.assertIsNone(job.created)
+ stats = job._properties["statistics"] = {}
+ self.assertIsNone(job.created)
+ stats["creationTime"] = millis
+ self.assertEqual(job.created, now)
+
+ def test_started(self):
+ now, millis = self._datetime_and_millis()
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+ self.assertIsNone(job.started)
+ stats = job._properties["statistics"] = {}
+ self.assertIsNone(job.started)
+ stats["startTime"] = millis
+ self.assertEqual(job.started, now)
+
+ def test_ended(self):
+ now, millis = self._datetime_and_millis()
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+ self.assertIsNone(job.ended)
+ stats = job._properties["statistics"] = {}
+ self.assertIsNone(job.ended)
+ stats["endTime"] = millis
+ self.assertEqual(job.ended, now)
+
+ def test__job_statistics(self):
+ statistics = {"foo": "bar"}
+ client = _make_client(project=self.PROJECT)
+ derived = self._make_derived(self.JOB_ID, client)
+ self.assertEqual(derived._job_statistics(), {})
+ stats = derived._properties["statistics"] = {}
+ self.assertEqual(derived._job_statistics(), {})
+ stats["derived"] = statistics
+ self.assertEqual(derived._job_statistics(), statistics)
+
+ def test_error_result(self):
+ error_result = {
+ "debugInfo": "DEBUG INFO",
+ "location": "LOCATION",
+ "message": "MESSAGE",
+ "reason": "REASON",
+ }
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+ self.assertIsNone(job.error_result)
+ status = job._properties["status"] = {}
+ self.assertIsNone(job.error_result)
+ status["errorResult"] = error_result
+ self.assertEqual(job.error_result, error_result)
+
+ def test_errors(self):
+ errors = [
+ {
+ "debugInfo": "DEBUG INFO",
+ "location": "LOCATION",
+ "message": "MESSAGE",
+ "reason": "REASON",
+ }
+ ]
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+ self.assertIsNone(job.errors)
+ status = job._properties["status"] = {}
+ self.assertIsNone(job.errors)
+ status["errors"] = errors
+ self.assertEqual(job.errors, errors)
+
+ def test_state(self):
+ state = "STATE"
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+ self.assertIsNone(job.state)
+ status = job._properties["status"] = {}
+ self.assertIsNone(job.state)
+ status["state"] = state
+ self.assertEqual(job.state, state)
+
+ def test__scrub_local_properties(self):
+ before = {"foo": "bar"}
+ resource = before.copy()
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+ job._scrub_local_properties(resource) # no raise
+ self.assertEqual(resource, before)
+
+ def test__copy_configuration_properties(self):
+ before = {"foo": "bar"}
+ resource = before.copy()
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+ with self.assertRaises(NotImplementedError):
+ job._copy_configuration_properties(resource)
+ self.assertEqual(resource, before)
+
+ def _set_properties_job(self):
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+ job._scrub_local_properties = mock.Mock()
+ job._copy_configuration_properties = mock.Mock()
+ job._set_future_result = mock.Mock()
+ job._properties = {
+ "jobReference": job._properties["jobReference"],
+ "foo": "bar",
+ }
+ return job
+
+ def test__set_properties_no_stats(self):
+ config = {"test": True}
+ resource = {"configuration": config}
+ job = self._set_properties_job()
+
+ job._set_properties(resource)
+
+ self.assertEqual(job._properties, resource)
+
+ job._scrub_local_properties.assert_called_once_with(resource)
+ job._copy_configuration_properties.assert_called_once_with(config)
+
+ def test__set_properties_w_creation_time(self):
+ now, millis = self._datetime_and_millis()
+ config = {"test": True}
+ stats = {"creationTime": str(millis)}
+ resource = {"configuration": config, "statistics": stats}
+ job = self._set_properties_job()
+
+ job._set_properties(resource)
+
+ cleaned = copy.deepcopy(resource)
+ cleaned["statistics"]["creationTime"] = float(millis)
+ self.assertEqual(job._properties, cleaned)
+
+ job._scrub_local_properties.assert_called_once_with(resource)
+ job._copy_configuration_properties.assert_called_once_with(config)
+
+ def test__set_properties_w_start_time(self):
+ now, millis = self._datetime_and_millis()
+ config = {"test": True}
+ stats = {"startTime": str(millis)}
+ resource = {"configuration": config, "statistics": stats}
+ job = self._set_properties_job()
+
+ job._set_properties(resource)
+
+ cleaned = copy.deepcopy(resource)
+ cleaned["statistics"]["startTime"] = float(millis)
+ self.assertEqual(job._properties, cleaned)
+
+ job._scrub_local_properties.assert_called_once_with(resource)
+ job._copy_configuration_properties.assert_called_once_with(config)
+
+ def test__set_properties_w_end_time(self):
+ now, millis = self._datetime_and_millis()
+ config = {"test": True}
+ stats = {"endTime": str(millis)}
+ resource = {"configuration": config, "statistics": stats}
+ job = self._set_properties_job()
+
+ job._set_properties(resource)
+
+ cleaned = copy.deepcopy(resource)
+ cleaned["statistics"]["endTime"] = float(millis)
+ self.assertEqual(job._properties, cleaned)
+
+ job._scrub_local_properties.assert_called_once_with(resource)
+ job._copy_configuration_properties.assert_called_once_with(config)
+
+ def test__get_resource_config_missing_job_ref(self):
+ resource = {}
+ klass = self._make_derived_class()
+
+ with self.assertRaises(KeyError):
+ klass._get_resource_config(resource)
+
+ def test__get_resource_config_missing_job_id(self):
+ resource = {"jobReference": {}}
+ klass = self._make_derived_class()
+
+ with self.assertRaises(KeyError):
+ klass._get_resource_config(resource)
+
+ def test__get_resource_config_missing_configuration(self):
+ resource = {"jobReference": {"jobId": self.JOB_ID}}
+ klass = self._make_derived_class()
+
+ with self.assertRaises(KeyError):
+ klass._get_resource_config(resource)
+
+ def test__get_resource_config_missing_config_type(self):
+ resource = {"jobReference": {"jobId": self.JOB_ID}, "configuration": {}}
+ klass = self._make_derived_class()
+
+ with self.assertRaises(KeyError):
+ klass._get_resource_config(resource)
+
+ def test__get_resource_config_ok(self):
+ derived_config = {"foo": "bar"}
+ resource = {
+ "jobReference": {"jobId": self.JOB_ID},
+ "configuration": {"derived": derived_config},
+ }
+ klass = self._make_derived_class()
+
+ job_id, config = klass._get_resource_config(resource)
+
+ self.assertEqual(job_id, self.JOB_ID)
+ self.assertEqual(config, {"derived": derived_config})
+
+ def test__build_resource(self):
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+ with self.assertRaises(NotImplementedError):
+ job._build_resource()
+
+ def test_to_api_repr(self):
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+ with self.assertRaises(NotImplementedError):
+ job.to_api_repr()
+
+ def test__begin_already(self):
+ job = self._set_properties_job()
+ job._properties["status"] = {"state": "WHATEVER"}
+
+ with self.assertRaises(ValueError):
+ job._begin()
+
+ def test__begin_defaults(self):
+ from google.cloud.bigquery.retry import DEFAULT_RETRY
+
+ resource = {
+ "jobReference": {
+ "jobId": self.JOB_ID,
+ "projectId": self.PROJECT,
+ "location": None,
+ },
+ "configuration": {"test": True},
+ }
+ job = self._set_properties_job()
+ builder = job.to_api_repr = mock.Mock()
+ builder.return_value = resource
+ call_api = job._client._call_api = mock.Mock()
+ call_api.return_value = resource
+
+ job._begin()
+
+ call_api.assert_called_once_with(
+ DEFAULT_RETRY,
+ method="POST",
+ path="/projects/{}/jobs".format(self.PROJECT),
+ data=resource,
+ )
+ self.assertEqual(job._properties, resource)
+
+ def test__begin_explicit(self):
+ from google.cloud.bigquery.retry import DEFAULT_RETRY
+
+ other_project = "other-project-234"
+ resource = {
+ "jobReference": {
+ "jobId": self.JOB_ID,
+ "projectId": self.PROJECT,
+ "location": None,
+ },
+ "configuration": {"test": True},
+ }
+ job = self._set_properties_job()
+ builder = job.to_api_repr = mock.Mock()
+ builder.return_value = resource
+ client = _make_client(project=other_project)
+ call_api = client._call_api = mock.Mock()
+ call_api.return_value = resource
+ retry = DEFAULT_RETRY.with_deadline(1)
+
+ job._begin(client=client, retry=retry)
+
+ call_api.assert_called_once_with(
+ retry,
+ method="POST",
+ path="/projects/{}/jobs".format(self.PROJECT),
+ data=resource,
+ )
+ self.assertEqual(job._properties, resource)
+
+ def test_exists_defaults_miss(self):
+ from google.cloud.exceptions import NotFound
+ from google.cloud.bigquery.retry import DEFAULT_RETRY
+
+ job = self._set_properties_job()
+ job._properties["jobReference"]["location"] = self.LOCATION
+ call_api = job._client._call_api = mock.Mock()
+ call_api.side_effect = NotFound("testing")
+
+ self.assertFalse(job.exists())
+
+ call_api.assert_called_once_with(
+ DEFAULT_RETRY,
+ method="GET",
+ path="/projects/{}/jobs/{}".format(self.PROJECT, self.JOB_ID),
+ query_params={"fields": "id", "location": self.LOCATION},
+ )
+
+ def test_exists_explicit_hit(self):
+ from google.cloud.bigquery.retry import DEFAULT_RETRY
+
+ other_project = "other-project-234"
+ resource = {
+ "jobReference": {
+ "jobId": self.JOB_ID,
+ "projectId": self.PROJECT,
+ "location": None,
+ },
+ "configuration": {"test": True},
+ }
+ job = self._set_properties_job()
+ client = _make_client(project=other_project)
+ call_api = client._call_api = mock.Mock()
+ call_api.return_value = resource
+ retry = DEFAULT_RETRY.with_deadline(1)
+
+ self.assertTrue(job.exists(client=client, retry=retry))
+
+ call_api.assert_called_once_with(
+ retry,
+ method="GET",
+ path="/projects/{}/jobs/{}".format(self.PROJECT, self.JOB_ID),
+ query_params={"fields": "id"},
+ )
+
+ def test_reload_defaults(self):
+ from google.cloud.bigquery.retry import DEFAULT_RETRY
+
+ resource = {
+ "jobReference": {
+ "jobId": self.JOB_ID,
+ "projectId": self.PROJECT,
+ "location": None,
+ },
+ "configuration": {"test": True},
+ }
+ job = self._set_properties_job()
+ job._properties["jobReference"]["location"] = self.LOCATION
+ call_api = job._client._call_api = mock.Mock()
+ call_api.return_value = resource
+
+ job.reload()
+
+ call_api.assert_called_once_with(
+ DEFAULT_RETRY,
+ method="GET",
+ path="/projects/{}/jobs/{}".format(self.PROJECT, self.JOB_ID),
+ query_params={"location": self.LOCATION},
+ )
+ self.assertEqual(job._properties, resource)
+
+ def test_reload_explicit(self):
+ from google.cloud.bigquery.retry import DEFAULT_RETRY
+
+ other_project = "other-project-234"
+ resource = {
+ "jobReference": {
+ "jobId": self.JOB_ID,
+ "projectId": self.PROJECT,
+ "location": None,
+ },
+ "configuration": {"test": True},
+ }
+ job = self._set_properties_job()
+ client = _make_client(project=other_project)
+ call_api = client._call_api = mock.Mock()
+ call_api.return_value = resource
+ retry = DEFAULT_RETRY.with_deadline(1)
+
+ job.reload(client=client, retry=retry)
+
+ call_api.assert_called_once_with(
+ retry,
+ method="GET",
+ path="/projects/{}/jobs/{}".format(self.PROJECT, self.JOB_ID),
+ query_params={},
+ )
+ self.assertEqual(job._properties, resource)
+
+ def test_cancel_defaults(self):
+ resource = {
+ "jobReference": {
+ "jobId": self.JOB_ID,
+ "projectId": self.PROJECT,
+ "location": None,
+ },
+ "configuration": {"test": True},
+ }
+ response = {"job": resource}
+ job = self._set_properties_job()
+ job._properties["jobReference"]["location"] = self.LOCATION
+ connection = job._client._connection = _make_connection(response)
+
+ self.assertTrue(job.cancel())
+
+ connection.api_request.assert_called_once_with(
+ method="POST",
+ path="/projects/{}/jobs/{}/cancel".format(self.PROJECT, self.JOB_ID),
+ query_params={"location": self.LOCATION},
+ )
+ self.assertEqual(job._properties, resource)
+
+ def test_cancel_explicit(self):
+ other_project = "other-project-234"
+ resource = {
+ "jobReference": {
+ "jobId": self.JOB_ID,
+ "projectId": self.PROJECT,
+ "location": None,
+ },
+ "configuration": {"test": True},
+ }
+ response = {"job": resource}
+ job = self._set_properties_job()
+ client = _make_client(project=other_project)
+ connection = client._connection = _make_connection(response)
+
+ self.assertTrue(job.cancel(client=client))
+
+ connection.api_request.assert_called_once_with(
+ method="POST",
+ path="/projects/{}/jobs/{}/cancel".format(self.PROJECT, self.JOB_ID),
+ query_params={},
+ )
+ self.assertEqual(job._properties, resource)
+
+ def test__set_future_result_wo_done(self):
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+ set_exception = job.set_exception = mock.Mock()
+ set_result = job.set_result = mock.Mock()
+
+ job._set_future_result()
+
+ set_exception.assert_not_called()
+ set_result.assert_not_called()
+
+ def test__set_future_result_w_result_set(self):
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+ job._properties["status"] = {"state": "DONE"}
+ job._result_set = True
+ set_exception = job.set_exception = mock.Mock()
+ set_result = job.set_result = mock.Mock()
+
+ job._set_future_result()
+
+ set_exception.assert_not_called()
+ set_result.assert_not_called()
+
+ def test__set_future_result_w_done_wo_result_set_w_error(self):
+ from google.cloud.exceptions import NotFound
+
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+ job._properties["status"] = {
+ "state": "DONE",
+ "errorResult": {"reason": "notFound", "message": "testing"},
+ }
+ set_exception = job.set_exception = mock.Mock()
+ set_result = job.set_result = mock.Mock()
+
+ job._set_future_result()
+
+ set_exception.assert_called_once()
+ args, kw = set_exception.call_args
+ exception, = args
+ self.assertIsInstance(exception, NotFound)
+ self.assertEqual(exception.message, "testing")
+ self.assertEqual(kw, {})
+ set_result.assert_not_called()
+
+ def test__set_future_result_w_done_wo_result_set_wo_error(self):
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+ job._properties["status"] = {"state": "DONE"}
+ set_exception = job.set_exception = mock.Mock()
+ set_result = job.set_result = mock.Mock()
+
+ job._set_future_result()
+
+ set_exception.assert_not_called()
+ set_result.assert_called_once_with(job)
+
+ def test_done_defaults_wo_state(self):
+ from google.cloud.bigquery.retry import DEFAULT_RETRY
+
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+ reload_ = job.reload = mock.Mock()
+
+ self.assertFalse(job.done())
+
+ reload_.assert_called_once_with(retry=DEFAULT_RETRY)
+
+ def test_done_explicit_wo_state(self):
+ from google.cloud.bigquery.retry import DEFAULT_RETRY
+
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+ reload_ = job.reload = mock.Mock()
+ retry = DEFAULT_RETRY.with_deadline(1)
+
+ self.assertFalse(job.done(retry=retry))
+
+ reload_.assert_called_once_with(retry=retry)
+
+ def test_done_already(self):
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+ job._properties["status"] = {"state": "DONE"}
+
+ self.assertTrue(job.done())
+
+ @mock.patch("google.api_core.future.polling.PollingFuture.result")
+ def test_result_default_wo_state(self, result):
+ from google.cloud.bigquery.retry import DEFAULT_RETRY
+
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+ begin = job._begin = mock.Mock()
+
+ self.assertIs(job.result(), result.return_value)
+
+ begin.assert_called_once_with(retry=DEFAULT_RETRY)
+ result.assert_called_once_with(timeout=None)
+
+ @mock.patch("google.api_core.future.polling.PollingFuture.result")
+ def test_result_w_retry_wo_state(self, result):
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+ begin = job._begin = mock.Mock()
+ retry = mock.Mock()
+
+ self.assertIs(job.result(retry=retry), result.return_value)
+
+ begin.assert_called_once_with(retry=retry)
+ result.assert_called_once_with(timeout=None)
+
+ @mock.patch("google.api_core.future.polling.PollingFuture.result")
+ def test_result_explicit_w_state(self, result):
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+ job._properties["status"] = {"state": "DONE"}
+ begin = job._begin = mock.Mock()
+ timeout = 1
+
+ self.assertIs(job.result(timeout=timeout), result.return_value)
+
+ begin.assert_not_called()
+ result.assert_called_once_with(timeout=timeout)
+
+ def test_cancelled_wo_error_result(self):
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+
+ self.assertFalse(job.cancelled())
+
+ def test_cancelled_w_error_result_not_stopped(self):
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+ job._properties["status"] = {"errorResult": {"reason": "other"}}
+
+ self.assertFalse(job.cancelled())
+
+ def test_cancelled_w_error_result_w_stopped(self):
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, client)
+ job._properties["status"] = {"errorResult": {"reason": "stopped"}}
+
+ self.assertTrue(job.cancelled())
+
+
+class Test_JobConfig(unittest.TestCase):
+ JOB_TYPE = "testing"
+
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery import job
+
+ return job._JobConfig
+
+ def _make_one(self, job_type=JOB_TYPE):
+ return self._get_target_class()(job_type)
+
+ def test_ctor(self):
+ job_config = self._make_one()
+ self.assertEqual(job_config._job_type, self.JOB_TYPE)
+ self.assertEqual(job_config._properties, {self.JOB_TYPE: {}})
+
+ def test_fill_from_default(self):
+ from google.cloud.bigquery import QueryJobConfig
+
+ job_config = QueryJobConfig()
+ job_config.dry_run = True
+ job_config.maximum_bytes_billed = 1000
+
+ default_job_config = QueryJobConfig()
+ default_job_config.use_query_cache = True
+ default_job_config.maximum_bytes_billed = 2000
+
+ final_job_config = job_config._fill_from_default(default_job_config)
+ self.assertTrue(final_job_config.dry_run)
+ self.assertTrue(final_job_config.use_query_cache)
+ self.assertEqual(final_job_config.maximum_bytes_billed, 1000)
+
+ def test_fill_from_default_conflict(self):
+ from google.cloud.bigquery import QueryJobConfig
+
+ basic_job_config = QueryJobConfig()
+ conflicting_job_config = self._make_one("conflicting_job_type")
+ self.assertNotEqual(
+ basic_job_config._job_type, conflicting_job_config._job_type
+ )
+
+ with self.assertRaises(TypeError):
+ basic_job_config._fill_from_default(conflicting_job_config)
+
+ @mock.patch("google.cloud.bigquery._helpers._get_sub_prop")
+ def test__get_sub_prop_wo_default(self, _get_sub_prop):
+ job_config = self._make_one()
+ key = "key"
+ self.assertIs(job_config._get_sub_prop(key), _get_sub_prop.return_value)
+ _get_sub_prop.assert_called_once_with(
+ job_config._properties, [self.JOB_TYPE, key], default=None
+ )
+
+ @mock.patch("google.cloud.bigquery._helpers._get_sub_prop")
+ def test__get_sub_prop_w_default(self, _get_sub_prop):
+ job_config = self._make_one()
+ key = "key"
+ default = "default"
+ self.assertIs(
+ job_config._get_sub_prop(key, default=default), _get_sub_prop.return_value
+ )
+ _get_sub_prop.assert_called_once_with(
+ job_config._properties, [self.JOB_TYPE, key], default=default
+ )
+
+ @mock.patch("google.cloud.bigquery._helpers._set_sub_prop")
+ def test__set_sub_prop(self, _set_sub_prop):
+ job_config = self._make_one()
+ key = "key"
+ value = "value"
+ job_config._set_sub_prop(key, value)
+ _set_sub_prop.assert_called_once_with(
+ job_config._properties, [self.JOB_TYPE, key], value
+ )
+
+ def test_to_api_repr(self):
+ job_config = self._make_one()
+ expected = job_config._properties = {self.JOB_TYPE: {"foo": "bar"}}
+ found = job_config.to_api_repr()
+ self.assertEqual(found, expected)
+ self.assertIsNot(found, expected) # copied
+
+ # 'from_api_repr' cannot be tested on '_JobConfig', because it presumes
+ # the ctor can be called w/o arguments
+
+ def test_labels_miss(self):
+ job_config = self._make_one()
+ self.assertEqual(job_config.labels, {})
+
+ def test_labels_update_in_place(self):
+ job_config = self._make_one()
+ labels = job_config.labels
+ labels["foo"] = "bar" # update in place
+ self.assertEqual(job_config.labels, {"foo": "bar"})
+
+ def test_labels_hit(self):
+ labels = {"foo": "bar"}
+ job_config = self._make_one()
+ job_config._properties["labels"] = labels
+ self.assertEqual(job_config.labels, labels)
+
+ def test_labels_setter_invalid(self):
+ labels = object()
+ job_config = self._make_one()
+ with self.assertRaises(ValueError):
+ job_config.labels = labels
+
+ def test_labels_setter(self):
+ labels = {"foo": "bar"}
+ job_config = self._make_one()
+ job_config.labels = labels
+ self.assertEqual(job_config._properties["labels"], labels)
+
+
+class _Base(object):
+ from google.cloud.bigquery.dataset import DatasetReference
+ from google.cloud.bigquery.table import TableReference
+
+ ENDPOINT = "https://www.googleapis.com"
+ PROJECT = "project"
+ SOURCE1 = "http://example.com/source1.csv"
+ DS_ID = "dataset_id"
+ DS_REF = DatasetReference(PROJECT, DS_ID)
+ TABLE_ID = "table_id"
+ TABLE_REF = TableReference(DS_REF, TABLE_ID)
+ JOB_ID = "JOB_ID"
+ KMS_KEY_NAME = "projects/1/locations/global/keyRings/1/cryptoKeys/1"
+
+ def _make_one(self, *args, **kw):
+ return self._get_target_class()(*args, **kw)
+
+ def _setUpConstants(self):
+ import datetime
+ from google.cloud._helpers import UTC
+
+ self.WHEN_TS = 1437767599.006
+ self.WHEN = datetime.datetime.utcfromtimestamp(self.WHEN_TS).replace(tzinfo=UTC)
+ self.ETAG = "ETAG"
+ self.FULL_JOB_ID = "%s:%s" % (self.PROJECT, self.JOB_ID)
+ self.RESOURCE_URL = "{}/bigquery/v2/projects/{}/jobs/{}".format(
+ self.ENDPOINT, self.PROJECT, self.JOB_ID
+ )
+ self.USER_EMAIL = "phred@example.com"
+
+ def _table_ref(self, table_id):
+ from google.cloud.bigquery.table import TableReference
+
+ return TableReference(self.DS_REF, table_id)
+
+ def _make_resource(self, started=False, ended=False):
+ self._setUpConstants()
+ return _make_job_resource(
+ creation_time_ms=int(self.WHEN_TS * 1000),
+ started_time_ms=int(self.WHEN_TS * 1000),
+ ended_time_ms=int(self.WHEN_TS * 1000) + 1000000,
+ started=started,
+ ended=ended,
+ etag=self.ETAG,
+ endpoint=self.ENDPOINT,
+ job_type=self.JOB_TYPE,
+ job_id=self.JOB_ID,
+ project_id=self.PROJECT,
+ user_email=self.USER_EMAIL,
+ )
+
+ def _verifyInitialReadonlyProperties(self, job):
+ # root elements of resource
+ self.assertIsNone(job.etag)
+ self.assertIsNone(job.self_link)
+ self.assertIsNone(job.user_email)
+
+ # derived from resource['statistics']
+ self.assertIsNone(job.created)
+ self.assertIsNone(job.started)
+ self.assertIsNone(job.ended)
+
+ # derived from resource['status']
+ self.assertIsNone(job.error_result)
+ self.assertIsNone(job.errors)
+ self.assertIsNone(job.state)
+
+ def _verifyReadonlyResourceProperties(self, job, resource):
+ from datetime import timedelta
+
+ statistics = resource.get("statistics", {})
+
+ if "creationTime" in statistics:
+ self.assertEqual(job.created, self.WHEN)
+ else:
+ self.assertIsNone(job.created)
+
+ if "startTime" in statistics:
+ self.assertEqual(job.started, self.WHEN)
+ else:
+ self.assertIsNone(job.started)
+
+ if "endTime" in statistics:
+ self.assertEqual(job.ended, self.WHEN + timedelta(seconds=1000))
+ else:
+ self.assertIsNone(job.ended)
+
+ if "etag" in resource:
+ self.assertEqual(job.etag, self.ETAG)
+ else:
+ self.assertIsNone(job.etag)
+
+ if "selfLink" in resource:
+ self.assertEqual(job.self_link, self.RESOURCE_URL)
+ else:
+ self.assertIsNone(job.self_link)
+
+ if "user_email" in resource:
+ self.assertEqual(job.user_email, self.USER_EMAIL)
+ else:
+ self.assertIsNone(job.user_email)
+
+
+class TestLoadJobConfig(unittest.TestCase, _Base):
+ JOB_TYPE = "load"
+
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery.job import LoadJobConfig
+
+ return LoadJobConfig
+
+ def test_ctor_w_properties(self):
+ config = self._get_target_class()(
+ allow_jagged_rows=True, allow_quoted_newlines=True
+ )
+
+ self.assertTrue(config.allow_jagged_rows)
+ self.assertTrue(config.allow_quoted_newlines)
+
+ def test_allow_jagged_rows_missing(self):
+ config = self._get_target_class()()
+ self.assertIsNone(config.allow_jagged_rows)
+
+ def test_allow_jagged_rows_hit(self):
+ config = self._get_target_class()()
+ config._properties["load"]["allowJaggedRows"] = True
+ self.assertTrue(config.allow_jagged_rows)
+
+ def test_allow_jagged_rows_setter(self):
+ config = self._get_target_class()()
+ config.allow_jagged_rows = True
+ self.assertTrue(config._properties["load"]["allowJaggedRows"])
+
+ def test_allow_quoted_newlines_missing(self):
+ config = self._get_target_class()()
+ self.assertIsNone(config.allow_quoted_newlines)
+
+ def test_allow_quoted_newlines_hit(self):
+ config = self._get_target_class()()
+ config._properties["load"]["allowQuotedNewlines"] = True
+ self.assertTrue(config.allow_quoted_newlines)
+
+ def test_allow_quoted_newlines_setter(self):
+ config = self._get_target_class()()
+ config.allow_quoted_newlines = True
+ self.assertTrue(config._properties["load"]["allowQuotedNewlines"])
+
+ def test_autodetect_missing(self):
+ config = self._get_target_class()()
+ self.assertIsNone(config.autodetect)
+
+ def test_autodetect_hit(self):
+ config = self._get_target_class()()
+ config._properties["load"]["autodetect"] = True
+ self.assertTrue(config.autodetect)
+
+ def test_autodetect_setter(self):
+ config = self._get_target_class()()
+ config.autodetect = True
+ self.assertTrue(config._properties["load"]["autodetect"])
+
+ def test_clustering_fields_miss(self):
+ config = self._get_target_class()()
+ self.assertIsNone(config.clustering_fields)
+
+ def test_clustering_fields_hit(self):
+ config = self._get_target_class()()
+ fields = ["email", "postal_code"]
+ config._properties["load"]["clustering"] = {"fields": fields}
+ self.assertEqual(config.clustering_fields, fields)
+
+ def test_clustering_fields_setter(self):
+ fields = ["email", "postal_code"]
+ config = self._get_target_class()()
+ config.clustering_fields = fields
+ self.assertEqual(config._properties["load"]["clustering"], {"fields": fields})
+
+ def test_clustering_fields_setter_w_none(self):
+ config = self._get_target_class()()
+ fields = ["email", "postal_code"]
+ config._properties["load"]["clustering"] = {"fields": fields}
+ config.clustering_fields = None
+ self.assertIsNone(config.clustering_fields)
+ self.assertNotIn("clustering", config._properties["load"])
+
+ def test_create_disposition_missing(self):
+ config = self._get_target_class()()
+ self.assertIsNone(config.create_disposition)
+
+ def test_create_disposition_hit(self):
+ from google.cloud.bigquery.job import CreateDisposition
+
+ disposition = CreateDisposition.CREATE_IF_NEEDED
+ config = self._get_target_class()()
+ config._properties["load"]["createDisposition"] = disposition
+ self.assertEqual(config.create_disposition, disposition)
+
+ def test_create_disposition_setter(self):
+ from google.cloud.bigquery.job import CreateDisposition
+
+ disposition = CreateDisposition.CREATE_IF_NEEDED
+ config = self._get_target_class()()
+ config.create_disposition = disposition
+ self.assertEqual(config._properties["load"]["createDisposition"], disposition)
+
+ def test_destination_encryption_configuration_missing(self):
+ config = self._get_target_class()()
+ self.assertIsNone(config.destination_encryption_configuration)
+
+ def test_destination_encryption_configuration_hit(self):
+ from google.cloud.bigquery.table import EncryptionConfiguration
+
+ kms_key_name = "kms-key-name"
+ encryption_configuration = EncryptionConfiguration(kms_key_name)
+ config = self._get_target_class()()
+ config._properties["load"]["destinationEncryptionConfiguration"] = {
+ "kmsKeyName": kms_key_name
+ }
+ self.assertEqual(
+ config.destination_encryption_configuration, encryption_configuration
+ )
+
+ def test_destination_encryption_configuration_setter(self):
+ from google.cloud.bigquery.table import EncryptionConfiguration
+
+ kms_key_name = "kms-key-name"
+ encryption_configuration = EncryptionConfiguration(kms_key_name)
+ config = self._get_target_class()()
+ config.destination_encryption_configuration = encryption_configuration
+ expected = {"kmsKeyName": kms_key_name}
+ self.assertEqual(
+ config._properties["load"]["destinationEncryptionConfiguration"], expected
+ )
+
+ def test_destination_encryption_configuration_setter_w_none(self):
+ kms_key_name = "kms-key-name"
+ config = self._get_target_class()()
+ config._properties["load"]["destinationEncryptionConfiguration"] = {
+ "kmsKeyName": kms_key_name
+ }
+ config.destination_encryption_configuration = None
+ self.assertIsNone(config.destination_encryption_configuration)
+ self.assertNotIn(
+ "destinationEncryptionConfiguration", config._properties["load"]
+ )
+
+ def test_destination_table_description_missing(self):
+ config = self._get_target_class()()
+ self.assertIsNone(config.destination_table_description)
+
+ def test_destination_table_description_hit(self):
+ description = "Description"
+ config = self._get_target_class()()
+ config._properties["load"]["destinationTableProperties"] = {
+ "description": description
+ }
+ self.assertEqual(config.destination_table_description, description)
+
+ def test_destination_table_description_setter(self):
+ description = "Description"
+ config = self._get_target_class()()
+ config.destination_table_description = description
+ expected = {"description": description}
+ self.assertEqual(
+ config._properties["load"]["destinationTableProperties"], expected
+ )
+
+ def test_destination_table_description_setter_w_fn_already(self):
+ description = "Description"
+ friendly_name = "Friendly Name"
+ config = self._get_target_class()()
+ config._properties["load"]["destinationTableProperties"] = {
+ "friendlyName": friendly_name
+ }
+ config.destination_table_description = description
+ expected = {"friendlyName": friendly_name, "description": description}
+ self.assertEqual(
+ config._properties["load"]["destinationTableProperties"], expected
+ )
+
+ def test_destination_table_description_w_none(self):
+ description = "Description"
+ friendly_name = "Friendly Name"
+ config = self._get_target_class()()
+ config._properties["load"]["destinationTableProperties"] = {
+ "description": description,
+ "friendlyName": friendly_name,
+ }
+ config.destination_table_description = None
+ expected = {"friendlyName": friendly_name}
+ self.assertEqual(
+ config._properties["load"]["destinationTableProperties"], expected
+ )
+
+ def test_destination_table_friendly_name_missing(self):
+ config = self._get_target_class()()
+ self.assertIsNone(config.destination_table_friendly_name)
+
+ def test_destination_table_friendly_name_hit(self):
+ friendly_name = "Friendly Name"
+ config = self._get_target_class()()
+ config._properties["load"]["destinationTableProperties"] = {
+ "friendlyName": friendly_name
+ }
+ self.assertEqual(config.destination_table_friendly_name, friendly_name)
+
+ def test_destination_table_friendly_name_setter(self):
+ friendly_name = "Friendly Name"
+ config = self._get_target_class()()
+ config.destination_table_friendly_name = friendly_name
+ expected = {"friendlyName": friendly_name}
+ self.assertEqual(
+ config._properties["load"]["destinationTableProperties"], expected
+ )
+
+ def test_destination_table_friendly_name_setter_w_descr_already(self):
+ friendly_name = "Friendly Name"
+ description = "Description"
+ config = self._get_target_class()()
+ config._properties["load"]["destinationTableProperties"] = {
+ "description": description
+ }
+ config.destination_table_friendly_name = friendly_name
+ expected = {"friendlyName": friendly_name, "description": description}
+ self.assertEqual(
+ config._properties["load"]["destinationTableProperties"], expected
+ )
+
+ def test_destination_table_friendly_name_w_none(self):
+ friendly_name = "Friendly Name"
+ description = "Description"
+ config = self._get_target_class()()
+ config._properties["load"]["destinationTableProperties"] = {
+ "description": description,
+ "friendlyName": friendly_name,
+ }
+ config.destination_table_friendly_name = None
+ expected = {"description": description}
+ self.assertEqual(
+ config._properties["load"]["destinationTableProperties"], expected
+ )
+
+ def test_encoding_missing(self):
+ config = self._get_target_class()()
+ self.assertIsNone(config.encoding)
+
+ def test_encoding_hit(self):
+ from google.cloud.bigquery.job import Encoding
+
+ encoding = Encoding.UTF_8
+ config = self._get_target_class()()
+ config._properties["load"]["encoding"] = encoding
+ self.assertEqual(config.encoding, encoding)
+
+ def test_encoding_setter(self):
+ from google.cloud.bigquery.job import Encoding
+
+ encoding = Encoding.UTF_8
+ config = self._get_target_class()()
+ config.encoding = encoding
+ self.assertEqual(config._properties["load"]["encoding"], encoding)
+
+ def test_field_delimiter_missing(self):
+ config = self._get_target_class()()
+ self.assertIsNone(config.field_delimiter)
+
+ def test_field_delimiter_hit(self):
+ field_delimiter = "|"
+ config = self._get_target_class()()
+ config._properties["load"]["fieldDelimiter"] = field_delimiter
+ self.assertEqual(config.field_delimiter, field_delimiter)
+
+ def test_field_delimiter_setter(self):
+ field_delimiter = "|"
+ config = self._get_target_class()()
+ config.field_delimiter = field_delimiter
+ self.assertEqual(config._properties["load"]["fieldDelimiter"], field_delimiter)
+
+ def test_ignore_unknown_values_missing(self):
+ config = self._get_target_class()()
+ self.assertIsNone(config.ignore_unknown_values)
+
+ def test_ignore_unknown_values_hit(self):
+ config = self._get_target_class()()
+ config._properties["load"]["ignoreUnknownValues"] = True
+ self.assertTrue(config.ignore_unknown_values)
+
+ def test_ignore_unknown_values_setter(self):
+ config = self._get_target_class()()
+ config.ignore_unknown_values = True
+ self.assertTrue(config._properties["load"]["ignoreUnknownValues"])
+
+ def test_max_bad_records_missing(self):
+ config = self._get_target_class()()
+ self.assertIsNone(config.max_bad_records)
+
+ def test_max_bad_records_hit(self):
+ max_bad_records = 13
+ config = self._get_target_class()()
+ config._properties["load"]["maxBadRecords"] = max_bad_records
+ self.assertEqual(config.max_bad_records, max_bad_records)
+
+ def test_max_bad_records_setter(self):
+ max_bad_records = 13
+ config = self._get_target_class()()
+ config.max_bad_records = max_bad_records
+ self.assertEqual(config._properties["load"]["maxBadRecords"], max_bad_records)
+
+ def test_null_marker_missing(self):
+ config = self._get_target_class()()
+ self.assertIsNone(config.null_marker)
+
+ def test_null_marker_hit(self):
+ null_marker = "XXX"
+ config = self._get_target_class()()
+ config._properties["load"]["nullMarker"] = null_marker
+ self.assertEqual(config.null_marker, null_marker)
+
+ def test_null_marker_setter(self):
+ null_marker = "XXX"
+ config = self._get_target_class()()
+ config.null_marker = null_marker
+ self.assertEqual(config._properties["load"]["nullMarker"], null_marker)
+
+ def test_quote_character_missing(self):
+ config = self._get_target_class()()
+ self.assertIsNone(config.quote_character)
+
+ def test_quote_character_hit(self):
+ quote_character = "'"
+ config = self._get_target_class()()
+ config._properties["load"]["quote"] = quote_character
+ self.assertEqual(config.quote_character, quote_character)
+
+ def test_quote_character_setter(self):
+ quote_character = "'"
+ config = self._get_target_class()()
+ config.quote_character = quote_character
+ self.assertEqual(config._properties["load"]["quote"], quote_character)
+
+ def test_schema_missing(self):
+ config = self._get_target_class()()
+ self.assertIsNone(config.schema)
+
+ def test_schema_hit(self):
+ from google.cloud.bigquery.schema import SchemaField
+
+ config = self._get_target_class()()
+ all_props_repr = {
+ "mode": "REQUIRED",
+ "name": "foo",
+ "type": "INTEGER",
+ "description": "Foo",
+ }
+ minimal_repr = {"name": "bar", "type": "STRING"}
+ config._properties["load"]["schema"] = {
+ "fields": [all_props_repr, minimal_repr]
+ }
+ all_props, minimal = config.schema
+ self.assertEqual(all_props, SchemaField.from_api_repr(all_props_repr))
+ self.assertEqual(minimal, SchemaField.from_api_repr(minimal_repr))
+
+ def test_schema_setter(self):
+ from google.cloud.bigquery.schema import SchemaField
+
+ config = self._get_target_class()()
+ full_name = SchemaField("full_name", "STRING", mode="REQUIRED")
+ age = SchemaField("age", "INTEGER", mode="REQUIRED")
+ config.schema = [full_name, age]
+ full_name_repr = {
+ "name": "full_name",
+ "type": "STRING",
+ "mode": "REQUIRED",
+ "description": None,
+ }
+ age_repr = {
+ "name": "age",
+ "type": "INTEGER",
+ "mode": "REQUIRED",
+ "description": None,
+ }
+ self.assertEqual(
+ config._properties["load"]["schema"], {"fields": [full_name_repr, age_repr]}
+ )
+
+ def test_schema_setter_unsetting_schema(self):
+ from google.cloud.bigquery.schema import SchemaField
+
+ config = self._get_target_class()()
+ config._properties["load"]["schema"] = [
+ SchemaField("full_name", "STRING", mode="REQUIRED"),
+ SchemaField("age", "INTEGER", mode="REQUIRED"),
+ ]
+
+ config.schema = None
+ self.assertNotIn("schema", config._properties["load"])
+ config.schema = None # no error, idempotent operation
+
+ def test_schema_update_options_missing(self):
+ config = self._get_target_class()()
+ self.assertIsNone(config.schema_update_options)
+
+ def test_schema_update_options_hit(self):
+ from google.cloud.bigquery.job import SchemaUpdateOption
+
+ options = [
+ SchemaUpdateOption.ALLOW_FIELD_ADDITION,
+ SchemaUpdateOption.ALLOW_FIELD_RELAXATION,
+ ]
+ config = self._get_target_class()()
+ config._properties["load"]["schemaUpdateOptions"] = options
+ self.assertEqual(config.schema_update_options, options)
+
+ def test_schema_update_options_setter(self):
+ from google.cloud.bigquery.job import SchemaUpdateOption
+
+ options = [
+ SchemaUpdateOption.ALLOW_FIELD_ADDITION,
+ SchemaUpdateOption.ALLOW_FIELD_RELAXATION,
+ ]
+ config = self._get_target_class()()
+ config.schema_update_options = options
+ self.assertEqual(config._properties["load"]["schemaUpdateOptions"], options)
+
+ def test_skip_leading_rows_missing(self):
+ config = self._get_target_class()()
+ self.assertIsNone(config.skip_leading_rows)
+
+ def test_skip_leading_rows_hit_w_str(self):
+ skip_leading_rows = 1
+ config = self._get_target_class()()
+ config._properties["load"]["skipLeadingRows"] = str(skip_leading_rows)
+ self.assertEqual(config.skip_leading_rows, skip_leading_rows)
+
+ def test_skip_leading_rows_hit_w_integer(self):
+ skip_leading_rows = 1
+ config = self._get_target_class()()
+ config._properties["load"]["skipLeadingRows"] = skip_leading_rows
+ self.assertEqual(config.skip_leading_rows, skip_leading_rows)
+
+ def test_skip_leading_rows_setter(self):
+ skip_leading_rows = 1
+ config = self._get_target_class()()
+ config.skip_leading_rows = skip_leading_rows
+ self.assertEqual(
+ config._properties["load"]["skipLeadingRows"], str(skip_leading_rows)
+ )
+
+ def test_source_format_missing(self):
+ config = self._get_target_class()()
+ self.assertIsNone(config.source_format)
+
+ def test_source_format_hit(self):
+ from google.cloud.bigquery.job import SourceFormat
+
+ source_format = SourceFormat.CSV
+ config = self._get_target_class()()
+ config._properties["load"]["sourceFormat"] = source_format
+ self.assertEqual(config.source_format, source_format)
+
+ def test_source_format_setter(self):
+ from google.cloud.bigquery.job import SourceFormat
+
+ source_format = SourceFormat.CSV
+ config = self._get_target_class()()
+ config.source_format = source_format
+ self.assertEqual(config._properties["load"]["sourceFormat"], source_format)
+
+ def test_time_partitioning_miss(self):
+ config = self._get_target_class()()
+ self.assertIsNone(config.time_partitioning)
+
+ def test_time_partitioning_hit(self):
+ from google.cloud.bigquery.table import TimePartitioning
+ from google.cloud.bigquery.table import TimePartitioningType
+
+ field = "creation_date"
+ year_ms = 86400 * 1000 * 365
+ config = self._get_target_class()()
+ config._properties["load"]["timePartitioning"] = {
+ "type": TimePartitioningType.DAY,
+ "field": field,
+ "expirationMs": str(year_ms),
+ "requirePartitionFilter": False,
+ }
+ expected = TimePartitioning(
+ type_=TimePartitioningType.DAY,
+ field=field,
+ expiration_ms=year_ms,
+ require_partition_filter=False,
+ )
+ self.assertEqual(config.time_partitioning, expected)
+
+ def test_time_partitioning_setter(self):
+ from google.cloud.bigquery.table import TimePartitioning
+ from google.cloud.bigquery.table import TimePartitioningType
+
+ field = "creation_date"
+ year_ms = 86400 * 1000 * 365
+ time_partitioning = TimePartitioning(
+ type_=TimePartitioningType.DAY,
+ field=field,
+ expiration_ms=year_ms,
+ require_partition_filter=False,
+ )
+ config = self._get_target_class()()
+ config.time_partitioning = time_partitioning
+ expected = {
+ "type": TimePartitioningType.DAY,
+ "field": field,
+ "expirationMs": str(year_ms),
+ "requirePartitionFilter": False,
+ }
+ self.assertEqual(config._properties["load"]["timePartitioning"], expected)
+
+ def test_time_partitioning_setter_w_none(self):
+ from google.cloud.bigquery.table import TimePartitioningType
+
+ field = "creation_date"
+ year_ms = 86400 * 1000 * 365
+ config = self._get_target_class()()
+ config._properties["load"]["timePartitioning"] = {
+ "type": TimePartitioningType.DAY,
+ "field": field,
+ "expirationMs": str(year_ms),
+ "requirePartitionFilter": False,
+ }
+ config.time_partitioning = None
+ self.assertIsNone(config.time_partitioning)
+ self.assertNotIn("timePartitioning", config._properties["load"])
+
+ def test_use_avro_logical_types(self):
+ config = self._get_target_class()()
+ self.assertIsNone(config.use_avro_logical_types)
+
+ def test_use_avro_logical_types_setter(self):
+ config = self._get_target_class()()
+ config.use_avro_logical_types = True
+ self.assertTrue(config._properties["load"]["useAvroLogicalTypes"])
+
+ def test_write_disposition_missing(self):
+ config = self._get_target_class()()
+ self.assertIsNone(config.write_disposition)
+
+ def test_write_disposition_hit(self):
+ from google.cloud.bigquery.job import WriteDisposition
+
+ write_disposition = WriteDisposition.WRITE_TRUNCATE
+ config = self._get_target_class()()
+ config._properties["load"]["writeDisposition"] = write_disposition
+ self.assertEqual(config.write_disposition, write_disposition)
+
+ def test_write_disposition_setter(self):
+ from google.cloud.bigquery.job import WriteDisposition
+
+ write_disposition = WriteDisposition.WRITE_TRUNCATE
+ config = self._get_target_class()()
+ config.write_disposition = write_disposition
+ self.assertEqual(
+ config._properties["load"]["writeDisposition"], write_disposition
+ )
+
+
+class TestLoadJob(unittest.TestCase, _Base):
+ JOB_TYPE = "load"
+
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery.job import LoadJob
+
+ return LoadJob
+
+ def _setUpConstants(self):
+ super(TestLoadJob, self)._setUpConstants()
+ self.INPUT_FILES = 2
+ self.INPUT_BYTES = 12345
+ self.OUTPUT_BYTES = 23456
+ self.OUTPUT_ROWS = 345
+
+ def _make_resource(self, started=False, ended=False):
+ resource = super(TestLoadJob, self)._make_resource(started, ended)
+ config = resource["configuration"]["load"]
+ config["sourceUris"] = [self.SOURCE1]
+ config["destinationTable"] = {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.TABLE_ID,
+ }
+
+ if ended:
+ resource["status"] = {"state": "DONE"}
+ resource["statistics"]["load"]["inputFiles"] = self.INPUT_FILES
+ resource["statistics"]["load"]["inputFileBytes"] = self.INPUT_BYTES
+ resource["statistics"]["load"]["outputBytes"] = self.OUTPUT_BYTES
+ resource["statistics"]["load"]["outputRows"] = self.OUTPUT_ROWS
+
+ return resource
+
+ def _verifyBooleanConfigProperties(self, job, config):
+ if "allowJaggedRows" in config:
+ self.assertEqual(job.allow_jagged_rows, config["allowJaggedRows"])
+ else:
+ self.assertIsNone(job.allow_jagged_rows)
+ if "allowQuotedNewlines" in config:
+ self.assertEqual(job.allow_quoted_newlines, config["allowQuotedNewlines"])
+ else:
+ self.assertIsNone(job.allow_quoted_newlines)
+ if "autodetect" in config:
+ self.assertEqual(job.autodetect, config["autodetect"])
+ else:
+ self.assertIsNone(job.autodetect)
+ if "ignoreUnknownValues" in config:
+ self.assertEqual(job.ignore_unknown_values, config["ignoreUnknownValues"])
+ else:
+ self.assertIsNone(job.ignore_unknown_values)
+ if "useAvroLogicalTypes" in config:
+ self.assertEqual(job.use_avro_logical_types, config["useAvroLogicalTypes"])
+ else:
+ self.assertIsNone(job.use_avro_logical_types)
+
+ def _verifyEnumConfigProperties(self, job, config):
+ if "createDisposition" in config:
+ self.assertEqual(job.create_disposition, config["createDisposition"])
+ else:
+ self.assertIsNone(job.create_disposition)
+ if "encoding" in config:
+ self.assertEqual(job.encoding, config["encoding"])
+ else:
+ self.assertIsNone(job.encoding)
+ if "sourceFormat" in config:
+ self.assertEqual(job.source_format, config["sourceFormat"])
+ else:
+ self.assertIsNone(job.source_format)
+ if "writeDisposition" in config:
+ self.assertEqual(job.write_disposition, config["writeDisposition"])
+ else:
+ self.assertIsNone(job.write_disposition)
+ if "schemaUpdateOptions" in config:
+ self.assertEqual(job.schema_update_options, config["schemaUpdateOptions"])
+ else:
+ self.assertIsNone(job.schema_update_options)
+
+ def _verifyResourceProperties(self, job, resource):
+ self._verifyReadonlyResourceProperties(job, resource)
+
+ config = resource.get("configuration", {}).get("load")
+
+ self._verifyBooleanConfigProperties(job, config)
+ self._verifyEnumConfigProperties(job, config)
+
+ self.assertEqual(job.source_uris, config["sourceUris"])
+
+ table_ref = config["destinationTable"]
+ self.assertEqual(job.destination.project, table_ref["projectId"])
+ self.assertEqual(job.destination.dataset_id, table_ref["datasetId"])
+ self.assertEqual(job.destination.table_id, table_ref["tableId"])
+
+ if "fieldDelimiter" in config:
+ self.assertEqual(job.field_delimiter, config["fieldDelimiter"])
+ else:
+ self.assertIsNone(job.field_delimiter)
+ if "maxBadRecords" in config:
+ self.assertEqual(job.max_bad_records, config["maxBadRecords"])
+ else:
+ self.assertIsNone(job.max_bad_records)
+ if "nullMarker" in config:
+ self.assertEqual(job.null_marker, config["nullMarker"])
+ else:
+ self.assertIsNone(job.null_marker)
+ if "quote" in config:
+ self.assertEqual(job.quote_character, config["quote"])
+ else:
+ self.assertIsNone(job.quote_character)
+ if "skipLeadingRows" in config:
+ self.assertEqual(str(job.skip_leading_rows), config["skipLeadingRows"])
+ else:
+ self.assertIsNone(job.skip_leading_rows)
+
+ if "destinationEncryptionConfiguration" in config:
+ self.assertIsNotNone(job.destination_encryption_configuration)
+ self.assertEqual(
+ job.destination_encryption_configuration.kms_key_name,
+ config["destinationEncryptionConfiguration"]["kmsKeyName"],
+ )
+ else:
+ self.assertIsNone(job.destination_encryption_configuration)
+
+ def test_ctor(self):
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, [self.SOURCE1], self.TABLE_REF, client)
+ self.assertIs(job.destination, self.TABLE_REF)
+ self.assertEqual(list(job.source_uris), [self.SOURCE1])
+ self.assertIs(job._client, client)
+ self.assertEqual(job.job_type, self.JOB_TYPE)
+ self.assertEqual(job.path, "/projects/%s/jobs/%s" % (self.PROJECT, self.JOB_ID))
+
+ self._verifyInitialReadonlyProperties(job)
+
+ # derived from resource['statistics']['load']
+ self.assertIsNone(job.input_file_bytes)
+ self.assertIsNone(job.input_files)
+ self.assertIsNone(job.output_bytes)
+ self.assertIsNone(job.output_rows)
+
+ # set/read from resource['configuration']['load']
+ self.assertIsNone(job.schema)
+ self.assertIsNone(job.allow_jagged_rows)
+ self.assertIsNone(job.allow_quoted_newlines)
+ self.assertIsNone(job.autodetect)
+ self.assertIsNone(job.create_disposition)
+ self.assertIsNone(job.encoding)
+ self.assertIsNone(job.field_delimiter)
+ self.assertIsNone(job.ignore_unknown_values)
+ self.assertIsNone(job.max_bad_records)
+ self.assertIsNone(job.null_marker)
+ self.assertIsNone(job.quote_character)
+ self.assertIsNone(job.skip_leading_rows)
+ self.assertIsNone(job.source_format)
+ self.assertIsNone(job.write_disposition)
+ self.assertIsNone(job.destination_encryption_configuration)
+ self.assertIsNone(job.destination_table_description)
+ self.assertIsNone(job.destination_table_friendly_name)
+ self.assertIsNone(job.time_partitioning)
+ self.assertIsNone(job.use_avro_logical_types)
+ self.assertIsNone(job.clustering_fields)
+ self.assertIsNone(job.schema_update_options)
+
+ def test_ctor_w_config(self):
+ from google.cloud.bigquery.schema import SchemaField
+ from google.cloud.bigquery.job import LoadJobConfig
+
+ client = _make_client(project=self.PROJECT)
+ full_name = SchemaField("full_name", "STRING", mode="REQUIRED")
+ age = SchemaField("age", "INTEGER", mode="REQUIRED")
+ config = LoadJobConfig()
+ config.schema = [full_name, age]
+ job = self._make_one(
+ self.JOB_ID, [self.SOURCE1], self.TABLE_REF, client, config
+ )
+ self.assertEqual(job.schema, [full_name, age])
+ config.destination_table_description = "Description"
+ expected = {"description": "Description"}
+ self.assertEqual(
+ config._properties["load"]["destinationTableProperties"], expected
+ )
+ friendly_name = "Friendly Name"
+ config._properties["load"]["destinationTableProperties"] = {
+ "friendlyName": friendly_name
+ }
+ self.assertEqual(config.destination_table_friendly_name, friendly_name)
+
+ def test_ctor_w_job_reference(self):
+ from google.cloud.bigquery import job
+
+ client = _make_client(project=self.PROJECT)
+ job_ref = job._JobReference(self.JOB_ID, "alternative-project", "US")
+ load_job = self._make_one(job_ref, [self.SOURCE1], self.TABLE_REF, client)
+ self.assertEqual(load_job.project, "alternative-project")
+ self.assertEqual(load_job.location, "US")
+
+ def test_done(self):
+ client = _make_client(project=self.PROJECT)
+ resource = self._make_resource(ended=True)
+ job = self._get_target_class().from_api_repr(resource, client)
+ self.assertTrue(job.done())
+
+ def test_result(self):
+ client = _make_client(project=self.PROJECT)
+ resource = self._make_resource(ended=True)
+ job = self._get_target_class().from_api_repr(resource, client)
+
+ result = job.result()
+
+ self.assertIs(result, job)
+
+ def test_result_invokes_begin(self):
+ begun_resource = self._make_resource()
+ done_resource = copy.deepcopy(begun_resource)
+ done_resource["status"] = {"state": "DONE"}
+ connection = _make_connection(begun_resource, done_resource)
+ client = _make_client(self.PROJECT)
+ client._connection = connection
+
+ job = self._make_one(self.JOB_ID, [self.SOURCE1], self.TABLE_REF, client)
+ job.result()
+
+ self.assertEqual(len(connection.api_request.call_args_list), 2)
+ begin_request, reload_request = connection.api_request.call_args_list
+ self.assertEqual(begin_request[1]["method"], "POST")
+ self.assertEqual(reload_request[1]["method"], "GET")
+
+ def test_schema_setter_non_list(self):
+ from google.cloud.bigquery.job import LoadJobConfig
+
+ config = LoadJobConfig()
+ with self.assertRaises(TypeError):
+ config.schema = object()
+
+ def test_schema_setter_invalid_field(self):
+ from google.cloud.bigquery.job import LoadJobConfig
+ from google.cloud.bigquery.schema import SchemaField
+
+ config = LoadJobConfig()
+ full_name = SchemaField("full_name", "STRING", mode="REQUIRED")
+ with self.assertRaises(ValueError):
+ config.schema = [full_name, object()]
+
+ def test_schema_setter(self):
+ from google.cloud.bigquery.job import LoadJobConfig
+ from google.cloud.bigquery.schema import SchemaField
+
+ config = LoadJobConfig()
+ full_name = SchemaField("full_name", "STRING", mode="REQUIRED")
+ age = SchemaField("age", "INTEGER", mode="REQUIRED")
+ config.schema = [full_name, age]
+ self.assertEqual(config.schema, [full_name, age])
+
+ def test_props_set_by_server(self):
+ import datetime
+ from google.cloud._helpers import UTC
+ from google.cloud._helpers import _millis
+
+ CREATED = datetime.datetime(2015, 8, 11, 12, 13, 22, tzinfo=UTC)
+ STARTED = datetime.datetime(2015, 8, 11, 13, 47, 15, tzinfo=UTC)
+ ENDED = datetime.datetime(2015, 8, 11, 14, 47, 15, tzinfo=UTC)
+ FULL_JOB_ID = "%s:%s" % (self.PROJECT, self.JOB_ID)
+ URL = "http://example.com/projects/%s/jobs/%s" % (self.PROJECT, self.JOB_ID)
+ EMAIL = "phred@example.com"
+ ERROR_RESULT = {
+ "debugInfo": "DEBUG",
+ "location": "LOCATION",
+ "message": "MESSAGE",
+ "reason": "REASON",
+ }
+
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, [self.SOURCE1], self.TABLE_REF, client)
+ job._properties["etag"] = "ETAG"
+ job._properties["id"] = FULL_JOB_ID
+ job._properties["selfLink"] = URL
+ job._properties["user_email"] = EMAIL
+
+ statistics = job._properties["statistics"] = {}
+ statistics["creationTime"] = _millis(CREATED)
+ statistics["startTime"] = _millis(STARTED)
+ statistics["endTime"] = _millis(ENDED)
+
+ self.assertEqual(job.etag, "ETAG")
+ self.assertEqual(job.self_link, URL)
+ self.assertEqual(job.user_email, EMAIL)
+
+ self.assertEqual(job.created, CREATED)
+ self.assertEqual(job.started, STARTED)
+ self.assertEqual(job.ended, ENDED)
+
+ # running jobs have no load stats not yet set.
+ self.assertIsNone(job.output_bytes)
+
+ load_stats = statistics["load"] = {}
+ load_stats["inputFileBytes"] = 12345
+ load_stats["inputFiles"] = 1
+ load_stats["outputBytes"] = 23456
+ load_stats["outputRows"] = 345
+
+ self.assertEqual(job.input_file_bytes, 12345)
+ self.assertEqual(job.input_files, 1)
+ self.assertEqual(job.output_bytes, 23456)
+ self.assertEqual(job.output_rows, 345)
+
+ status = job._properties["status"] = {}
+
+ self.assertIsNone(job.error_result)
+ self.assertIsNone(job.errors)
+ self.assertIsNone(job.state)
+
+ status["errorResult"] = ERROR_RESULT
+ status["errors"] = [ERROR_RESULT]
+ status["state"] = "STATE"
+
+ self.assertEqual(job.error_result, ERROR_RESULT)
+ self.assertEqual(job.errors, [ERROR_RESULT])
+ self.assertEqual(job.state, "STATE")
+
+ def test_from_api_repr_missing_identity(self):
+ self._setUpConstants()
+ client = _make_client(project=self.PROJECT)
+ RESOURCE = {}
+ klass = self._get_target_class()
+ with self.assertRaises(KeyError):
+ klass.from_api_repr(RESOURCE, client=client)
+
+ def test_from_api_repr_missing_config(self):
+ self._setUpConstants()
+ client = _make_client(project=self.PROJECT)
+ RESOURCE = {
+ "id": "%s:%s" % (self.PROJECT, self.JOB_ID),
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ }
+ klass = self._get_target_class()
+ with self.assertRaises(KeyError):
+ klass.from_api_repr(RESOURCE, client=client)
+
+ def test_from_api_repr_bare(self):
+ self._setUpConstants()
+ client = _make_client(project=self.PROJECT)
+ RESOURCE = {
+ "id": self.FULL_JOB_ID,
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "configuration": {
+ "load": {
+ "sourceUris": [self.SOURCE1],
+ "destinationTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.TABLE_ID,
+ },
+ }
+ },
+ }
+ klass = self._get_target_class()
+ job = klass.from_api_repr(RESOURCE, client=client)
+ self.assertIs(job._client, client)
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_from_api_with_encryption(self):
+ self._setUpConstants()
+ client = _make_client(project=self.PROJECT)
+ RESOURCE = {
+ "id": self.FULL_JOB_ID,
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "configuration": {
+ "load": {
+ "sourceUris": [self.SOURCE1],
+ "destinationTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.TABLE_ID,
+ },
+ "destinationEncryptionConfiguration": {
+ "kmsKeyName": self.KMS_KEY_NAME
+ },
+ }
+ },
+ }
+ klass = self._get_target_class()
+ job = klass.from_api_repr(RESOURCE, client=client)
+ self.assertIs(job._client, client)
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_from_api_repr_w_properties(self):
+ from google.cloud.bigquery.job import CreateDisposition
+
+ client = _make_client(project=self.PROJECT)
+ RESOURCE = self._make_resource()
+ load_config = RESOURCE["configuration"]["load"]
+ load_config["createDisposition"] = CreateDisposition.CREATE_IF_NEEDED
+ klass = self._get_target_class()
+ job = klass.from_api_repr(RESOURCE, client=client)
+ self.assertIs(job._client, client)
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_begin_w_already_running(self):
+ conn = _make_connection()
+ client = _make_client(project=self.PROJECT, connection=conn)
+ job = self._make_one(self.JOB_ID, [self.SOURCE1], self.TABLE_REF, client)
+ job._properties["status"] = {"state": "RUNNING"}
+
+ with self.assertRaises(ValueError):
+ job._begin()
+
+ def test_begin_w_bound_client(self):
+ RESOURCE = self._make_resource()
+ # Ensure None for missing server-set props
+ del RESOURCE["statistics"]["creationTime"]
+ del RESOURCE["etag"]
+ del RESOURCE["selfLink"]
+ del RESOURCE["user_email"]
+ conn = _make_connection(RESOURCE)
+ client = _make_client(project=self.PROJECT, connection=conn)
+ job = self._make_one(self.JOB_ID, [self.SOURCE1], self.TABLE_REF, client)
+
+ job._begin()
+
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path="/projects/{}/jobs".format(self.PROJECT),
+ data={
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "configuration": {
+ "load": {
+ "sourceUris": [self.SOURCE1],
+ "destinationTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.TABLE_ID,
+ },
+ }
+ },
+ },
+ )
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_begin_w_autodetect(self):
+ from google.cloud.bigquery.job import LoadJobConfig
+
+ path = "/projects/{}/jobs".format(self.PROJECT)
+ resource = self._make_resource()
+ resource["configuration"]["load"]["autodetect"] = True
+ # Ensure None for missing server-set props
+ del resource["statistics"]["creationTime"]
+ del resource["etag"]
+ del resource["selfLink"]
+ del resource["user_email"]
+ conn = _make_connection(resource)
+ client = _make_client(project=self.PROJECT, connection=conn)
+ config = LoadJobConfig()
+ config.autodetect = True
+ job = self._make_one(
+ self.JOB_ID, [self.SOURCE1], self.TABLE_REF, client, config
+ )
+ job._begin()
+
+ sent = {
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "configuration": {
+ "load": {
+ "sourceUris": [self.SOURCE1],
+ "destinationTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.TABLE_ID,
+ },
+ "autodetect": True,
+ }
+ },
+ }
+ conn.api_request.assert_called_once_with(method="POST", path=path, data=sent)
+ self._verifyResourceProperties(job, resource)
+
+ def test_begin_w_alternate_client(self):
+ from google.cloud.bigquery.job import CreateDisposition
+ from google.cloud.bigquery.job import LoadJobConfig
+ from google.cloud.bigquery.job import SchemaUpdateOption
+ from google.cloud.bigquery.job import WriteDisposition
+ from google.cloud.bigquery.schema import SchemaField
+
+ PATH = "/projects/%s/jobs" % (self.PROJECT,)
+ RESOURCE = self._make_resource(ended=True)
+ LOAD_CONFIGURATION = {
+ "sourceUris": [self.SOURCE1],
+ "destinationTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.TABLE_ID,
+ },
+ "allowJaggedRows": True,
+ "allowQuotedNewlines": True,
+ "createDisposition": CreateDisposition.CREATE_NEVER,
+ "encoding": "ISO-8559-1",
+ "fieldDelimiter": "|",
+ "ignoreUnknownValues": True,
+ "maxBadRecords": 100,
+ "nullMarker": r"\N",
+ "quote": "'",
+ "skipLeadingRows": "1",
+ "sourceFormat": "CSV",
+ "useAvroLogicalTypes": True,
+ "writeDisposition": WriteDisposition.WRITE_TRUNCATE,
+ "schema": {
+ "fields": [
+ {
+ "name": "full_name",
+ "type": "STRING",
+ "mode": "REQUIRED",
+ "description": None,
+ },
+ {
+ "name": "age",
+ "type": "INTEGER",
+ "mode": "REQUIRED",
+ "description": None,
+ },
+ ]
+ },
+ "schemaUpdateOptions": [SchemaUpdateOption.ALLOW_FIELD_ADDITION],
+ }
+ RESOURCE["configuration"]["load"] = LOAD_CONFIGURATION
+ conn1 = _make_connection()
+ client1 = _make_client(project=self.PROJECT, connection=conn1)
+ conn2 = _make_connection(RESOURCE)
+ client2 = _make_client(project=self.PROJECT, connection=conn2)
+ full_name = SchemaField("full_name", "STRING", mode="REQUIRED")
+ age = SchemaField("age", "INTEGER", mode="REQUIRED")
+ config = LoadJobConfig()
+ config.schema = [full_name, age]
+ job = self._make_one(
+ self.JOB_ID, [self.SOURCE1], self.TABLE_REF, client1, config
+ )
+ config.allow_jagged_rows = True
+ config.allow_quoted_newlines = True
+ config.create_disposition = CreateDisposition.CREATE_NEVER
+ config.encoding = "ISO-8559-1"
+ config.field_delimiter = "|"
+ config.ignore_unknown_values = True
+ config.max_bad_records = 100
+ config.null_marker = r"\N"
+ config.quote_character = "'"
+ config.skip_leading_rows = 1
+ config.source_format = "CSV"
+ config.use_avro_logical_types = True
+ config.write_disposition = WriteDisposition.WRITE_TRUNCATE
+ config.schema_update_options = [SchemaUpdateOption.ALLOW_FIELD_ADDITION]
+
+ job._begin(client=client2)
+
+ conn1.api_request.assert_not_called()
+ self.assertEqual(len(conn2.api_request.call_args_list), 1)
+ req = conn2.api_request.call_args_list[0]
+ self.assertEqual(req[1]["method"], "POST")
+ self.assertEqual(req[1]["path"], PATH)
+ SENT = {
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "configuration": {"load": LOAD_CONFIGURATION},
+ }
+ self.maxDiff = None
+ self.assertEqual(req[1]["data"], SENT)
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_begin_w_job_reference(self):
+ from google.cloud.bigquery import job
+
+ resource = self._make_resource()
+ resource["jobReference"]["projectId"] = "alternative-project"
+ resource["jobReference"]["location"] = "US"
+ job_ref = job._JobReference(self.JOB_ID, "alternative-project", "US")
+ conn = _make_connection(resource)
+ client = _make_client(project=self.PROJECT, connection=conn)
+ load_job = self._make_one(job_ref, [self.SOURCE1], self.TABLE_REF, client)
+
+ load_job._begin()
+
+ conn.api_request.assert_called_once()
+ _, request = conn.api_request.call_args
+ self.assertEqual(request["method"], "POST")
+ self.assertEqual(request["path"], "/projects/alternative-project/jobs")
+ self.assertEqual(
+ request["data"]["jobReference"]["projectId"], "alternative-project"
+ )
+ self.assertEqual(request["data"]["jobReference"]["location"], "US")
+ self.assertEqual(request["data"]["jobReference"]["jobId"], self.JOB_ID)
+
+ def test_exists_miss_w_bound_client(self):
+ PATH = "/projects/%s/jobs/%s" % (self.PROJECT, self.JOB_ID)
+ conn = _make_connection()
+ client = _make_client(project=self.PROJECT, connection=conn)
+ job = self._make_one(self.JOB_ID, [self.SOURCE1], self.TABLE_REF, client)
+
+ self.assertFalse(job.exists())
+
+ conn.api_request.assert_called_once_with(
+ method="GET", path=PATH, query_params={"fields": "id"}
+ )
+
+ def test_exists_hit_w_alternate_client(self):
+ PATH = "/projects/%s/jobs/%s" % (self.PROJECT, self.JOB_ID)
+ conn1 = _make_connection()
+ client1 = _make_client(project=self.PROJECT, connection=conn1)
+ conn2 = _make_connection({})
+ client2 = _make_client(project=self.PROJECT, connection=conn2)
+ job = self._make_one(self.JOB_ID, [self.SOURCE1], self.TABLE_REF, client1)
+
+ self.assertTrue(job.exists(client=client2))
+
+ conn1.api_request.assert_not_called()
+ conn2.api_request.assert_called_once_with(
+ method="GET", path=PATH, query_params={"fields": "id"}
+ )
+
+ def test_exists_miss_w_job_reference(self):
+ from google.cloud.bigquery import job
+
+ job_ref = job._JobReference("my-job-id", "other-project", "US")
+ conn = _make_connection()
+ client = _make_client(project=self.PROJECT, connection=conn)
+ load_job = self._make_one(job_ref, [self.SOURCE1], self.TABLE_REF, client)
+
+ self.assertFalse(load_job.exists())
+
+ conn.api_request.assert_called_once_with(
+ method="GET",
+ path="/projects/other-project/jobs/my-job-id",
+ query_params={"fields": "id", "location": "US"},
+ )
+
+ def test_reload_w_bound_client(self):
+ PATH = "/projects/%s/jobs/%s" % (self.PROJECT, self.JOB_ID)
+ RESOURCE = self._make_resource()
+ conn = _make_connection(RESOURCE)
+ client = _make_client(project=self.PROJECT, connection=conn)
+ job = self._make_one(self.JOB_ID, [self.SOURCE1], self.TABLE_REF, client)
+
+ job.reload()
+
+ conn.api_request.assert_called_once_with(
+ method="GET", path=PATH, query_params={}
+ )
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_reload_w_alternate_client(self):
+ PATH = "/projects/%s/jobs/%s" % (self.PROJECT, self.JOB_ID)
+ RESOURCE = self._make_resource()
+ conn1 = _make_connection()
+ client1 = _make_client(project=self.PROJECT, connection=conn1)
+ conn2 = _make_connection(RESOURCE)
+ client2 = _make_client(project=self.PROJECT, connection=conn2)
+ job = self._make_one(self.JOB_ID, [self.SOURCE1], self.TABLE_REF, client1)
+
+ job.reload(client=client2)
+
+ conn1.api_request.assert_not_called()
+ conn2.api_request.assert_called_once_with(
+ method="GET", path=PATH, query_params={}
+ )
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_reload_w_job_reference(self):
+ from google.cloud.bigquery import job
+
+ resource = self._make_resource(ended=True)
+ resource["jobReference"]["projectId"] = "alternative-project"
+ resource["jobReference"]["location"] = "US"
+ job_ref = job._JobReference(self.JOB_ID, "alternative-project", "US")
+ conn = _make_connection(resource)
+ client = _make_client(project=self.PROJECT, connection=conn)
+ load_job = self._make_one(job_ref, [self.SOURCE1], self.TABLE_REF, client)
+
+ load_job.reload()
+
+ conn.api_request.assert_called_once_with(
+ method="GET",
+ path="/projects/alternative-project/jobs/{}".format(self.JOB_ID),
+ query_params={"location": "US"},
+ )
+
+ def test_cancel_w_bound_client(self):
+ PATH = "/projects/%s/jobs/%s/cancel" % (self.PROJECT, self.JOB_ID)
+ RESOURCE = self._make_resource(ended=True)
+ RESPONSE = {"job": RESOURCE}
+ conn = _make_connection(RESPONSE)
+ client = _make_client(project=self.PROJECT, connection=conn)
+ job = self._make_one(self.JOB_ID, [self.SOURCE1], self.TABLE_REF, client)
+
+ job.cancel()
+
+ conn.api_request.assert_called_once_with(
+ method="POST", path=PATH, query_params={}
+ )
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_cancel_w_alternate_client(self):
+ PATH = "/projects/%s/jobs/%s/cancel" % (self.PROJECT, self.JOB_ID)
+ RESOURCE = self._make_resource(ended=True)
+ RESPONSE = {"job": RESOURCE}
+ conn1 = _make_connection()
+ client1 = _make_client(project=self.PROJECT, connection=conn1)
+ conn2 = _make_connection(RESPONSE)
+ client2 = _make_client(project=self.PROJECT, connection=conn2)
+ job = self._make_one(self.JOB_ID, [self.SOURCE1], self.TABLE_REF, client1)
+
+ job.cancel(client=client2)
+
+ conn1.api_request.assert_not_called()
+ conn2.api_request.assert_called_once_with(
+ method="POST", path=PATH, query_params={}
+ )
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_cancel_w_job_reference(self):
+ from google.cloud.bigquery import job
+
+ resource = self._make_resource(ended=True)
+ resource["jobReference"]["projectId"] = "alternative-project"
+ resource["jobReference"]["location"] = "US"
+ job_ref = job._JobReference(self.JOB_ID, "alternative-project", "US")
+ conn = _make_connection({"job": resource})
+ client = _make_client(project=self.PROJECT, connection=conn)
+ load_job = self._make_one(job_ref, [self.SOURCE1], self.TABLE_REF, client)
+
+ load_job.cancel()
+
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path="/projects/alternative-project/jobs/{}/cancel".format(self.JOB_ID),
+ query_params={"location": "US"},
+ )
+
+
+class TestCopyJobConfig(unittest.TestCase, _Base):
+ JOB_TYPE = "copy"
+
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery.job import CopyJobConfig
+
+ return CopyJobConfig
+
+ def test_ctor_w_properties(self):
+ from google.cloud.bigquery.job import CreateDisposition
+ from google.cloud.bigquery.job import WriteDisposition
+
+ create_disposition = CreateDisposition.CREATE_NEVER
+ write_disposition = WriteDisposition.WRITE_TRUNCATE
+ config = self._get_target_class()(
+ create_disposition=create_disposition, write_disposition=write_disposition
+ )
+
+ self.assertEqual(config.create_disposition, create_disposition)
+ self.assertEqual(config.write_disposition, write_disposition)
+
+ def test_to_api_repr_with_encryption(self):
+ from google.cloud.bigquery.table import EncryptionConfiguration
+
+ config = self._make_one()
+ config.destination_encryption_configuration = EncryptionConfiguration(
+ kms_key_name=self.KMS_KEY_NAME
+ )
+ resource = config.to_api_repr()
+ self.assertEqual(
+ resource,
+ {
+ "copy": {
+ "destinationEncryptionConfiguration": {
+ "kmsKeyName": self.KMS_KEY_NAME
+ }
+ }
+ },
+ )
+
+ def test_to_api_repr_with_encryption_none(self):
+ config = self._make_one()
+ config.destination_encryption_configuration = None
+ resource = config.to_api_repr()
+ self.assertEqual(
+ resource, {"copy": {"destinationEncryptionConfiguration": None}}
+ )
+
+
+class TestCopyJob(unittest.TestCase, _Base):
+ JOB_TYPE = "copy"
+ SOURCE_TABLE = "source_table"
+ DESTINATION_TABLE = "destination_table"
+
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery.job import CopyJob
+
+ return CopyJob
+
+ def _make_resource(self, started=False, ended=False):
+ resource = super(TestCopyJob, self)._make_resource(started, ended)
+ config = resource["configuration"]["copy"]
+ config["sourceTables"] = [
+ {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.SOURCE_TABLE,
+ }
+ ]
+ config["destinationTable"] = {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.DESTINATION_TABLE,
+ }
+
+ return resource
+
+ def _verifyResourceProperties(self, job, resource):
+ self._verifyReadonlyResourceProperties(job, resource)
+
+ config = resource.get("configuration", {}).get("copy")
+
+ table_ref = config["destinationTable"]
+ self.assertEqual(job.destination.project, table_ref["projectId"])
+ self.assertEqual(job.destination.dataset_id, table_ref["datasetId"])
+ self.assertEqual(job.destination.table_id, table_ref["tableId"])
+
+ sources = config.get("sourceTables")
+ if sources is None:
+ sources = [config["sourceTable"]]
+ self.assertEqual(len(sources), len(job.sources))
+ for table_ref, table in zip(sources, job.sources):
+ self.assertEqual(table.project, table_ref["projectId"])
+ self.assertEqual(table.dataset_id, table_ref["datasetId"])
+ self.assertEqual(table.table_id, table_ref["tableId"])
+
+ if "createDisposition" in config:
+ self.assertEqual(job.create_disposition, config["createDisposition"])
+ else:
+ self.assertIsNone(job.create_disposition)
+
+ if "writeDisposition" in config:
+ self.assertEqual(job.write_disposition, config["writeDisposition"])
+ else:
+ self.assertIsNone(job.write_disposition)
+
+ if "destinationEncryptionConfiguration" in config:
+ self.assertIsNotNone(job.destination_encryption_configuration)
+ self.assertEqual(
+ job.destination_encryption_configuration.kms_key_name,
+ config["destinationEncryptionConfiguration"]["kmsKeyName"],
+ )
+ else:
+ self.assertIsNone(job.destination_encryption_configuration)
+
+ def test_ctor(self):
+ client = _make_client(project=self.PROJECT)
+ source = self._table_ref(self.SOURCE_TABLE)
+ destination = self._table_ref(self.DESTINATION_TABLE)
+ job = self._make_one(self.JOB_ID, [source], destination, client)
+ self.assertIs(job.destination, destination)
+ self.assertEqual(job.sources, [source])
+ self.assertIs(job._client, client)
+ self.assertEqual(job.job_type, self.JOB_TYPE)
+ self.assertEqual(job.path, "/projects/%s/jobs/%s" % (self.PROJECT, self.JOB_ID))
+
+ self._verifyInitialReadonlyProperties(job)
+
+ # set/read from resource['configuration']['copy']
+ self.assertIsNone(job.create_disposition)
+ self.assertIsNone(job.write_disposition)
+ self.assertIsNone(job.destination_encryption_configuration)
+
+ def test_from_api_repr_missing_identity(self):
+ self._setUpConstants()
+ client = _make_client(project=self.PROJECT)
+ RESOURCE = {}
+ klass = self._get_target_class()
+ with self.assertRaises(KeyError):
+ klass.from_api_repr(RESOURCE, client=client)
+
+ def test_from_api_repr_missing_config(self):
+ self._setUpConstants()
+ client = _make_client(project=self.PROJECT)
+ RESOURCE = {
+ "id": "%s:%s" % (self.PROJECT, self.DS_ID),
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ }
+ klass = self._get_target_class()
+ with self.assertRaises(KeyError):
+ klass.from_api_repr(RESOURCE, client=client)
+
+ def test_from_api_repr_bare(self):
+ self._setUpConstants()
+ client = _make_client(project=self.PROJECT)
+ RESOURCE = {
+ "id": self.JOB_ID,
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "configuration": {
+ "copy": {
+ "sourceTables": [
+ {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.SOURCE_TABLE,
+ }
+ ],
+ "destinationTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.DESTINATION_TABLE,
+ },
+ }
+ },
+ }
+ klass = self._get_target_class()
+ job = klass.from_api_repr(RESOURCE, client=client)
+ self.assertIs(job._client, client)
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_from_api_with_encryption(self):
+ self._setUpConstants()
+ client = _make_client(project=self.PROJECT)
+ RESOURCE = {
+ "id": self.JOB_ID,
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "configuration": {
+ "copy": {
+ "sourceTables": [
+ {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.SOURCE_TABLE,
+ }
+ ],
+ "destinationTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.DESTINATION_TABLE,
+ },
+ "destinationEncryptionConfiguration": {
+ "kmsKeyName": self.KMS_KEY_NAME
+ },
+ }
+ },
+ }
+ klass = self._get_target_class()
+ job = klass.from_api_repr(RESOURCE, client=client)
+ self.assertIs(job._client, client)
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_from_api_repr_w_sourcetable(self):
+ self._setUpConstants()
+ client = _make_client(project=self.PROJECT)
+ RESOURCE = {
+ "id": self.JOB_ID,
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "configuration": {
+ "copy": {
+ "sourceTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.SOURCE_TABLE,
+ },
+ "destinationTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.DESTINATION_TABLE,
+ },
+ }
+ },
+ }
+ klass = self._get_target_class()
+ job = klass.from_api_repr(RESOURCE, client=client)
+ self.assertIs(job._client, client)
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_from_api_repr_wo_sources(self):
+ self._setUpConstants()
+ client = _make_client(project=self.PROJECT)
+ RESOURCE = {
+ "id": self.JOB_ID,
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "configuration": {
+ "copy": {
+ "destinationTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.DESTINATION_TABLE,
+ }
+ }
+ },
+ }
+ klass = self._get_target_class()
+ with self.assertRaises(KeyError):
+ klass.from_api_repr(RESOURCE, client=client)
+
+ def test_from_api_repr_w_properties(self):
+ from google.cloud.bigquery.job import CreateDisposition
+
+ client = _make_client(project=self.PROJECT)
+ RESOURCE = self._make_resource()
+ copy_config = RESOURCE["configuration"]["copy"]
+ copy_config["createDisposition"] = CreateDisposition.CREATE_IF_NEEDED
+ klass = self._get_target_class()
+ job = klass.from_api_repr(RESOURCE, client=client)
+ self.assertIs(job._client, client)
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_begin_w_bound_client(self):
+ PATH = "/projects/%s/jobs" % (self.PROJECT,)
+ RESOURCE = self._make_resource()
+ # Ensure None for missing server-set props
+ del RESOURCE["statistics"]["creationTime"]
+ del RESOURCE["etag"]
+ del RESOURCE["selfLink"]
+ del RESOURCE["user_email"]
+ conn = _make_connection(RESOURCE)
+ client = _make_client(project=self.PROJECT, connection=conn)
+ source = self._table_ref(self.SOURCE_TABLE)
+ destination = self._table_ref(self.DESTINATION_TABLE)
+ job = self._make_one(self.JOB_ID, [source], destination, client)
+
+ job._begin()
+
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path=PATH,
+ data={
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "configuration": {
+ "copy": {
+ "sourceTables": [
+ {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.SOURCE_TABLE,
+ }
+ ],
+ "destinationTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.DESTINATION_TABLE,
+ },
+ }
+ },
+ },
+ )
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_begin_w_alternate_client(self):
+ from google.cloud.bigquery.job import CopyJobConfig
+
+ from google.cloud.bigquery.job import CreateDisposition
+ from google.cloud.bigquery.job import WriteDisposition
+
+ PATH = "/projects/%s/jobs" % (self.PROJECT,)
+ RESOURCE = self._make_resource(ended=True)
+ COPY_CONFIGURATION = {
+ "sourceTables": [
+ {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.SOURCE_TABLE,
+ }
+ ],
+ "destinationTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.DESTINATION_TABLE,
+ },
+ "createDisposition": CreateDisposition.CREATE_NEVER,
+ "writeDisposition": WriteDisposition.WRITE_TRUNCATE,
+ }
+ RESOURCE["configuration"]["copy"] = COPY_CONFIGURATION
+ conn1 = _make_connection()
+ client1 = _make_client(project=self.PROJECT, connection=conn1)
+ conn2 = _make_connection(RESOURCE)
+ client2 = _make_client(project=self.PROJECT, connection=conn2)
+ source = self._table_ref(self.SOURCE_TABLE)
+ destination = self._table_ref(self.DESTINATION_TABLE)
+ config = CopyJobConfig()
+ config.create_disposition = CreateDisposition.CREATE_NEVER
+ config.write_disposition = WriteDisposition.WRITE_TRUNCATE
+ job = self._make_one(self.JOB_ID, [source], destination, client1, config)
+ job._begin(client=client2)
+
+ conn1.api_request.assert_not_called()
+ conn2.api_request.assert_called_once_with(
+ method="POST",
+ path=PATH,
+ data={
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "configuration": {"copy": COPY_CONFIGURATION},
+ },
+ )
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_exists_miss_w_bound_client(self):
+ PATH = "/projects/%s/jobs/%s" % (self.PROJECT, self.JOB_ID)
+ conn = _make_connection()
+ client = _make_client(project=self.PROJECT, connection=conn)
+
+ source = self._table_ref(self.SOURCE_TABLE)
+ destination = self._table_ref(self.DESTINATION_TABLE)
+ job = self._make_one(self.JOB_ID, [source], destination, client)
+
+ self.assertFalse(job.exists())
+
+ conn.api_request.assert_called_once_with(
+ method="GET", path=PATH, query_params={"fields": "id"}
+ )
+
+ def test_exists_hit_w_alternate_client(self):
+ PATH = "/projects/%s/jobs/%s" % (self.PROJECT, self.JOB_ID)
+ conn1 = _make_connection()
+ client1 = _make_client(project=self.PROJECT, connection=conn1)
+ conn2 = _make_connection({})
+ client2 = _make_client(project=self.PROJECT, connection=conn2)
+ source = self._table_ref(self.SOURCE_TABLE)
+ destination = self._table_ref(self.DESTINATION_TABLE)
+ job = self._make_one(self.JOB_ID, [source], destination, client1)
+
+ self.assertTrue(job.exists(client=client2))
+
+ conn1.api_request.assert_not_called()
+ conn2.api_request.assert_called_once_with(
+ method="GET", path=PATH, query_params={"fields": "id"}
+ )
+
+ def test_reload_w_bound_client(self):
+ PATH = "/projects/%s/jobs/%s" % (self.PROJECT, self.JOB_ID)
+ RESOURCE = self._make_resource()
+ conn = _make_connection(RESOURCE)
+ client = _make_client(project=self.PROJECT, connection=conn)
+ source = self._table_ref(self.SOURCE_TABLE)
+ destination = self._table_ref(self.DESTINATION_TABLE)
+ job = self._make_one(self.JOB_ID, [source], destination, client)
+
+ job.reload()
+
+ conn.api_request.assert_called_once_with(
+ method="GET", path=PATH, query_params={}
+ )
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_reload_w_alternate_client(self):
+ PATH = "/projects/%s/jobs/%s" % (self.PROJECT, self.JOB_ID)
+ RESOURCE = self._make_resource()
+ conn1 = _make_connection()
+ client1 = _make_client(project=self.PROJECT, connection=conn1)
+ conn2 = _make_connection(RESOURCE)
+ client2 = _make_client(project=self.PROJECT, connection=conn2)
+ source = self._table_ref(self.SOURCE_TABLE)
+ destination = self._table_ref(self.DESTINATION_TABLE)
+ job = self._make_one(self.JOB_ID, [source], destination, client1)
+
+ job.reload(client=client2)
+
+ conn1.api_request.assert_not_called()
+ conn2.api_request.assert_called_once_with(
+ method="GET", path=PATH, query_params={}
+ )
+ self._verifyResourceProperties(job, RESOURCE)
+
+
+class TestExtractJobConfig(unittest.TestCase, _Base):
+ JOB_TYPE = "extract"
+
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery.job import ExtractJobConfig
+
+ return ExtractJobConfig
+
+ def test_ctor_w_properties(self):
+ config = self._get_target_class()(field_delimiter="\t", print_header=True)
+
+ self.assertEqual(config.field_delimiter, "\t")
+ self.assertTrue(config.print_header)
+
+ def test_to_api_repr(self):
+ from google.cloud.bigquery import job
+
+ config = self._make_one()
+ config.compression = job.Compression.SNAPPY
+ config.destination_format = job.DestinationFormat.AVRO
+ config.field_delimiter = "ignored for avro"
+ config.print_header = False
+ config._properties["extract"]["someNewField"] = "some-value"
+ resource = config.to_api_repr()
+ self.assertEqual(
+ resource,
+ {
+ "extract": {
+ "compression": "SNAPPY",
+ "destinationFormat": "AVRO",
+ "fieldDelimiter": "ignored for avro",
+ "printHeader": False,
+ "someNewField": "some-value",
+ }
+ },
+ )
+
+ def test_from_api_repr(self):
+ cls = self._get_target_class()
+ config = cls.from_api_repr(
+ {
+ "extract": {
+ "compression": "NONE",
+ "destinationFormat": "CSV",
+ "fieldDelimiter": "\t",
+ "printHeader": True,
+ "someNewField": "some-value",
+ }
+ }
+ )
+ self.assertEqual(config.compression, "NONE")
+ self.assertEqual(config.destination_format, "CSV")
+ self.assertEqual(config.field_delimiter, "\t")
+ self.assertEqual(config.print_header, True)
+ self.assertEqual(config._properties["extract"]["someNewField"], "some-value")
+
+
+class TestExtractJob(unittest.TestCase, _Base):
+ JOB_TYPE = "extract"
+ SOURCE_TABLE = "source_table"
+ DESTINATION_URI = "gs://bucket_name/object_name"
+
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery.job import ExtractJob
+
+ return ExtractJob
+
+ def _make_resource(self, started=False, ended=False):
+ resource = super(TestExtractJob, self)._make_resource(started, ended)
+ config = resource["configuration"]["extract"]
+ config["sourceTable"] = {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.SOURCE_TABLE,
+ }
+ config["destinationUris"] = [self.DESTINATION_URI]
+ return resource
+
+ def _verifyResourceProperties(self, job, resource):
+ self._verifyReadonlyResourceProperties(job, resource)
+
+ config = resource.get("configuration", {}).get("extract")
+
+ self.assertEqual(job.destination_uris, config["destinationUris"])
+
+ table_ref = config["sourceTable"]
+ self.assertEqual(job.source.project, table_ref["projectId"])
+ self.assertEqual(job.source.dataset_id, table_ref["datasetId"])
+ self.assertEqual(job.source.table_id, table_ref["tableId"])
+
+ if "compression" in config:
+ self.assertEqual(job.compression, config["compression"])
+ else:
+ self.assertIsNone(job.compression)
+
+ if "destinationFormat" in config:
+ self.assertEqual(job.destination_format, config["destinationFormat"])
+ else:
+ self.assertIsNone(job.destination_format)
+
+ if "fieldDelimiter" in config:
+ self.assertEqual(job.field_delimiter, config["fieldDelimiter"])
+ else:
+ self.assertIsNone(job.field_delimiter)
+
+ if "printHeader" in config:
+ self.assertEqual(job.print_header, config["printHeader"])
+ else:
+ self.assertIsNone(job.print_header)
+
+ def test_ctor(self):
+ from google.cloud.bigquery.table import Table
+
+ client = _make_client(project=self.PROJECT)
+ source = Table(self.TABLE_REF)
+ job = self._make_one(self.JOB_ID, source, [self.DESTINATION_URI], client)
+ self.assertEqual(job.source.project, self.PROJECT)
+ self.assertEqual(job.source.dataset_id, self.DS_ID)
+ self.assertEqual(job.source.table_id, self.TABLE_ID)
+ self.assertEqual(job.destination_uris, [self.DESTINATION_URI])
+ self.assertIs(job._client, client)
+ self.assertEqual(job.job_type, self.JOB_TYPE)
+ self.assertEqual(job.path, "/projects/%s/jobs/%s" % (self.PROJECT, self.JOB_ID))
+
+ self._verifyInitialReadonlyProperties(job)
+
+ # set/read from resource['configuration']['extract']
+ self.assertIsNone(job.compression)
+ self.assertIsNone(job.destination_format)
+ self.assertIsNone(job.field_delimiter)
+ self.assertIsNone(job.print_header)
+
+ def test_destination_uri_file_counts(self):
+ file_counts = 23
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(
+ self.JOB_ID, self.TABLE_REF, [self.DESTINATION_URI], client
+ )
+ self.assertIsNone(job.destination_uri_file_counts)
+
+ statistics = job._properties["statistics"] = {}
+ self.assertIsNone(job.destination_uri_file_counts)
+
+ extract_stats = statistics["extract"] = {}
+ self.assertIsNone(job.destination_uri_file_counts)
+
+ extract_stats["destinationUriFileCounts"] = [str(file_counts)]
+ self.assertEqual(job.destination_uri_file_counts, [file_counts])
+
+ def test_from_api_repr_missing_identity(self):
+ self._setUpConstants()
+ client = _make_client(project=self.PROJECT)
+ RESOURCE = {}
+ klass = self._get_target_class()
+ with self.assertRaises(KeyError):
+ klass.from_api_repr(RESOURCE, client=client)
+
+ def test_from_api_repr_missing_config(self):
+ self._setUpConstants()
+ client = _make_client(project=self.PROJECT)
+ RESOURCE = {
+ "id": "%s:%s" % (self.PROJECT, self.DS_ID),
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ }
+ klass = self._get_target_class()
+ with self.assertRaises(KeyError):
+ klass.from_api_repr(RESOURCE, client=client)
+
+ def test_from_api_repr_bare(self):
+ self._setUpConstants()
+ client = _make_client(project=self.PROJECT)
+ RESOURCE = {
+ "id": self.JOB_ID,
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "configuration": {
+ "extract": {
+ "sourceTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.SOURCE_TABLE,
+ },
+ "destinationUris": [self.DESTINATION_URI],
+ }
+ },
+ }
+ klass = self._get_target_class()
+ job = klass.from_api_repr(RESOURCE, client=client)
+ self.assertIs(job._client, client)
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_from_api_repr_w_properties(self):
+ from google.cloud.bigquery.job import Compression
+
+ client = _make_client(project=self.PROJECT)
+ RESOURCE = self._make_resource()
+ extract_config = RESOURCE["configuration"]["extract"]
+ extract_config["compression"] = Compression.GZIP
+ klass = self._get_target_class()
+ job = klass.from_api_repr(RESOURCE, client=client)
+ self.assertIs(job._client, client)
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_begin_w_bound_client(self):
+ from google.cloud.bigquery.dataset import DatasetReference
+
+ PATH = "/projects/%s/jobs" % (self.PROJECT,)
+ RESOURCE = self._make_resource()
+ # Ensure None for missing server-set props
+ del RESOURCE["statistics"]["creationTime"]
+ del RESOURCE["etag"]
+ del RESOURCE["selfLink"]
+ del RESOURCE["user_email"]
+ conn = _make_connection(RESOURCE)
+ client = _make_client(project=self.PROJECT, connection=conn)
+ source_dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ source = source_dataset.table(self.SOURCE_TABLE)
+ job = self._make_one(self.JOB_ID, source, [self.DESTINATION_URI], client)
+
+ job._begin()
+
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path=PATH,
+ data={
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "configuration": {
+ "extract": {
+ "sourceTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.SOURCE_TABLE,
+ },
+ "destinationUris": [self.DESTINATION_URI],
+ }
+ },
+ },
+ )
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_begin_w_alternate_client(self):
+ from google.cloud.bigquery.dataset import DatasetReference
+ from google.cloud.bigquery.job import Compression
+ from google.cloud.bigquery.job import DestinationFormat
+ from google.cloud.bigquery.job import ExtractJobConfig
+
+ PATH = "/projects/%s/jobs" % (self.PROJECT,)
+ RESOURCE = self._make_resource(ended=True)
+ EXTRACT_CONFIGURATION = {
+ "sourceTable": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.SOURCE_TABLE,
+ },
+ "destinationUris": [self.DESTINATION_URI],
+ "compression": Compression.GZIP,
+ "destinationFormat": DestinationFormat.NEWLINE_DELIMITED_JSON,
+ "fieldDelimiter": "|",
+ "printHeader": False,
+ }
+ RESOURCE["configuration"]["extract"] = EXTRACT_CONFIGURATION
+ conn1 = _make_connection()
+ client1 = _make_client(project=self.PROJECT, connection=conn1)
+ conn2 = _make_connection(RESOURCE)
+ client2 = _make_client(project=self.PROJECT, connection=conn2)
+ source_dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ source = source_dataset.table(self.SOURCE_TABLE)
+ config = ExtractJobConfig()
+ config.compression = Compression.GZIP
+ config.destination_format = DestinationFormat.NEWLINE_DELIMITED_JSON
+ config.field_delimiter = "|"
+ config.print_header = False
+ job = self._make_one(
+ self.JOB_ID, source, [self.DESTINATION_URI], client1, config
+ )
+
+ job._begin(client=client2)
+
+ conn1.api_request.assert_not_called()
+ conn2.api_request.assert_called_once_with(
+ method="POST",
+ path=PATH,
+ data={
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "configuration": {"extract": EXTRACT_CONFIGURATION},
+ },
+ )
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_exists_miss_w_bound_client(self):
+ PATH = "/projects/%s/jobs/%s" % (self.PROJECT, self.JOB_ID)
+ conn = _make_connection()
+ client = _make_client(project=self.PROJECT, connection=conn)
+ job = self._make_one(
+ self.JOB_ID, self.TABLE_REF, [self.DESTINATION_URI], client
+ )
+
+ self.assertFalse(job.exists())
+
+ conn.api_request.assert_called_once_with(
+ method="GET", path=PATH, query_params={"fields": "id"}
+ )
+
+ def test_exists_hit_w_alternate_client(self):
+ PATH = "/projects/%s/jobs/%s" % (self.PROJECT, self.JOB_ID)
+ conn1 = _make_connection()
+ client1 = _make_client(project=self.PROJECT, connection=conn1)
+ conn2 = _make_connection({})
+ client2 = _make_client(project=self.PROJECT, connection=conn2)
+ job = self._make_one(
+ self.JOB_ID, self.TABLE_REF, [self.DESTINATION_URI], client1
+ )
+
+ self.assertTrue(job.exists(client=client2))
+
+ conn1.api_request.assert_not_called()
+ conn2.api_request.assert_called_once_with(
+ method="GET", path=PATH, query_params={"fields": "id"}
+ )
+
+ def test_reload_w_bound_client(self):
+ from google.cloud.bigquery.dataset import DatasetReference
+
+ PATH = "/projects/%s/jobs/%s" % (self.PROJECT, self.JOB_ID)
+ RESOURCE = self._make_resource()
+ conn = _make_connection(RESOURCE)
+ client = _make_client(project=self.PROJECT, connection=conn)
+ source_dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ source = source_dataset.table(self.SOURCE_TABLE)
+ job = self._make_one(self.JOB_ID, source, [self.DESTINATION_URI], client)
+
+ job.reload()
+
+ conn.api_request.assert_called_once_with(
+ method="GET", path=PATH, query_params={}
+ )
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_reload_w_alternate_client(self):
+ from google.cloud.bigquery.dataset import DatasetReference
+
+ PATH = "/projects/%s/jobs/%s" % (self.PROJECT, self.JOB_ID)
+ RESOURCE = self._make_resource()
+ conn1 = _make_connection()
+ client1 = _make_client(project=self.PROJECT, connection=conn1)
+ conn2 = _make_connection(RESOURCE)
+ client2 = _make_client(project=self.PROJECT, connection=conn2)
+ source_dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ source = source_dataset.table(self.SOURCE_TABLE)
+ job = self._make_one(self.JOB_ID, source, [self.DESTINATION_URI], client1)
+
+ job.reload(client=client2)
+
+ conn1.api_request.assert_not_called()
+ conn2.api_request.assert_called_once_with(
+ method="GET", path=PATH, query_params={}
+ )
+ self._verifyResourceProperties(job, RESOURCE)
+
+
+class TestQueryJobConfig(unittest.TestCase, _Base):
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery.job import QueryJobConfig
+
+ return QueryJobConfig
+
+ def _make_one(self, *args, **kw):
+ return self._get_target_class()(*args, **kw)
+
+ def test_ctor(self):
+ config = self._make_one()
+ self.assertEqual(config._properties, {"query": {}})
+
+ def test_ctor_w_none(self):
+ config = self._make_one()
+ config.default_dataset = None
+ config.destination = None
+ self.assertIsNone(config.default_dataset)
+ self.assertIsNone(config.destination)
+
+ def test_ctor_w_properties(self):
+ config = self._get_target_class()(use_query_cache=False, use_legacy_sql=True)
+
+ self.assertFalse(config.use_query_cache)
+ self.assertTrue(config.use_legacy_sql)
+
+ def test_ctor_w_string_default_dataset(self):
+ from google.cloud.bigquery import dataset
+
+ default_dataset = "default-proj.default_dset"
+ config = self._get_target_class()(default_dataset=default_dataset)
+ expected = dataset.DatasetReference.from_string(default_dataset)
+ self.assertEqual(config.default_dataset, expected)
+
+ def test_ctor_w_string_destinaton(self):
+ from google.cloud.bigquery import table
+
+ destination = "dest-proj.dest_dset.dest_tbl"
+ config = self._get_target_class()(destination=destination)
+ expected = table.TableReference.from_string(destination)
+ self.assertEqual(config.destination, expected)
+
+ def test_default_dataset_w_string(self):
+ from google.cloud.bigquery import dataset
+
+ default_dataset = "default-proj.default_dset"
+ config = self._make_one()
+ config.default_dataset = default_dataset
+ expected = dataset.DatasetReference.from_string(default_dataset)
+ self.assertEqual(config.default_dataset, expected)
+
+ def test_default_dataset_w_dataset(self):
+ from google.cloud.bigquery import dataset
+
+ default_dataset = "default-proj.default_dset"
+ expected = dataset.DatasetReference.from_string(default_dataset)
+ config = self._make_one()
+ config.default_dataset = dataset.Dataset(expected)
+ self.assertEqual(config.default_dataset, expected)
+
+ def test_destinaton_w_string(self):
+ from google.cloud.bigquery import table
+
+ destination = "dest-proj.dest_dset.dest_tbl"
+ config = self._make_one()
+ config.destination = destination
+ expected = table.TableReference.from_string(destination)
+ self.assertEqual(config.destination, expected)
+
+ def test_time_partitioning(self):
+ from google.cloud.bigquery import table
+
+ time_partitioning = table.TimePartitioning(
+ type_=table.TimePartitioningType.DAY, field="name"
+ )
+ config = self._make_one()
+ config.time_partitioning = time_partitioning
+ # TimePartitioning should be configurable after assigning
+ time_partitioning.expiration_ms = 10000
+
+ self.assertEqual(config.time_partitioning.type_, table.TimePartitioningType.DAY)
+ self.assertEqual(config.time_partitioning.field, "name")
+ self.assertEqual(config.time_partitioning.expiration_ms, 10000)
+
+ config.time_partitioning = None
+ self.assertIsNone(config.time_partitioning)
+
+ def test_clustering_fields(self):
+ fields = ["email", "postal_code"]
+ config = self._get_target_class()()
+ config.clustering_fields = fields
+ self.assertEqual(config.clustering_fields, fields)
+
+ config.clustering_fields = None
+ self.assertIsNone(config.clustering_fields)
+
+ def test_from_api_repr_empty(self):
+ klass = self._get_target_class()
+ config = klass.from_api_repr({})
+ self.assertIsNone(config.dry_run)
+ self.assertIsNone(config.use_legacy_sql)
+ self.assertIsNone(config.default_dataset)
+ self.assertIsNone(config.destination)
+ self.assertIsNone(config.destination_encryption_configuration)
+
+ def test_from_api_repr_normal(self):
+ from google.cloud.bigquery.dataset import DatasetReference
+
+ resource = {
+ "query": {
+ "useLegacySql": True,
+ "query": "no property for me",
+ "defaultDataset": {
+ "projectId": "someproject",
+ "datasetId": "somedataset",
+ },
+ "someNewProperty": "I should be saved, too.",
+ },
+ "dryRun": True,
+ }
+ klass = self._get_target_class()
+
+ config = klass.from_api_repr(resource)
+
+ self.assertTrue(config.use_legacy_sql)
+ self.assertEqual(
+ config.default_dataset, DatasetReference("someproject", "somedataset")
+ )
+ self.assertTrue(config.dry_run)
+ # Make sure unknown properties propagate.
+ self.assertEqual(config._properties["query"]["query"], "no property for me")
+ self.assertEqual(
+ config._properties["query"]["someNewProperty"], "I should be saved, too."
+ )
+
+ def test_to_api_repr_normal(self):
+ from google.cloud.bigquery.dataset import DatasetReference
+
+ config = self._make_one()
+ config.use_legacy_sql = True
+ config.default_dataset = DatasetReference("someproject", "somedataset")
+ config.dry_run = False
+ config._properties["someNewProperty"] = "Woohoo, alpha stuff."
+
+ resource = config.to_api_repr()
+
+ self.assertFalse(resource["dryRun"])
+ self.assertTrue(resource["query"]["useLegacySql"])
+ self.assertEqual(
+ resource["query"]["defaultDataset"]["projectId"], "someproject"
+ )
+ self.assertEqual(
+ resource["query"]["defaultDataset"]["datasetId"], "somedataset"
+ )
+ # Make sure unknown properties propagate.
+ self.assertEqual(resource["someNewProperty"], "Woohoo, alpha stuff.")
+
+ def test_to_api_repr_with_encryption(self):
+ from google.cloud.bigquery.table import EncryptionConfiguration
+
+ config = self._make_one()
+ config.destination_encryption_configuration = EncryptionConfiguration(
+ kms_key_name=self.KMS_KEY_NAME
+ )
+ resource = config.to_api_repr()
+ self.assertEqual(
+ resource,
+ {
+ "query": {
+ "destinationEncryptionConfiguration": {
+ "kmsKeyName": self.KMS_KEY_NAME
+ }
+ }
+ },
+ )
+
+ def test_to_api_repr_with_encryption_none(self):
+ config = self._make_one()
+ config.destination_encryption_configuration = None
+ resource = config.to_api_repr()
+ self.assertEqual(
+ resource, {"query": {"destinationEncryptionConfiguration": None}}
+ )
+
+ def test_from_api_repr_with_encryption(self):
+ resource = {
+ "query": {
+ "destinationEncryptionConfiguration": {"kmsKeyName": self.KMS_KEY_NAME}
+ }
+ }
+ klass = self._get_target_class()
+ config = klass.from_api_repr(resource)
+ self.assertEqual(
+ config.destination_encryption_configuration.kms_key_name, self.KMS_KEY_NAME
+ )
+
+
+class TestQueryJob(unittest.TestCase, _Base):
+ JOB_TYPE = "query"
+ QUERY = "select count(*) from persons"
+ DESTINATION_TABLE = "destination_table"
+
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery.job import QueryJob
+
+ return QueryJob
+
+ def _make_resource(self, started=False, ended=False):
+ resource = super(TestQueryJob, self)._make_resource(started, ended)
+ config = resource["configuration"]["query"]
+ config["query"] = self.QUERY
+
+ if ended:
+ resource["status"] = {"state": "DONE"}
+
+ return resource
+
+ def _verifyBooleanResourceProperties(self, job, config):
+
+ if "allowLargeResults" in config:
+ self.assertEqual(job.allow_large_results, config["allowLargeResults"])
+ else:
+ self.assertIsNone(job.allow_large_results)
+ if "flattenResults" in config:
+ self.assertEqual(job.flatten_results, config["flattenResults"])
+ else:
+ self.assertIsNone(job.flatten_results)
+ if "useQueryCache" in config:
+ self.assertEqual(job.use_query_cache, config["useQueryCache"])
+ else:
+ self.assertIsNone(job.use_query_cache)
+ if "useLegacySql" in config:
+ self.assertEqual(job.use_legacy_sql, config["useLegacySql"])
+ else:
+ self.assertIsNone(job.use_legacy_sql)
+
+ def _verifyIntegerResourceProperties(self, job, config):
+ if "maximumBillingTier" in config:
+ self.assertEqual(job.maximum_billing_tier, config["maximumBillingTier"])
+ else:
+ self.assertIsNone(job.maximum_billing_tier)
+ if "maximumBytesBilled" in config:
+ self.assertEqual(
+ str(job.maximum_bytes_billed), config["maximumBytesBilled"]
+ )
+ self.assertIsInstance(job.maximum_bytes_billed, int)
+ else:
+ self.assertIsNone(job.maximum_bytes_billed)
+
+ def _verify_udf_resources(self, job, config):
+ udf_resources = config.get("userDefinedFunctionResources", ())
+ self.assertEqual(len(job.udf_resources), len(udf_resources))
+ for found, expected in zip(job.udf_resources, udf_resources):
+ if "resourceUri" in expected:
+ self.assertEqual(found.udf_type, "resourceUri")
+ self.assertEqual(found.value, expected["resourceUri"])
+ else:
+ self.assertEqual(found.udf_type, "inlineCode")
+ self.assertEqual(found.value, expected["inlineCode"])
+
+ def _verifyQueryParameters(self, job, config):
+ query_parameters = config.get("queryParameters", ())
+ self.assertEqual(len(job.query_parameters), len(query_parameters))
+ for found, expected in zip(job.query_parameters, query_parameters):
+ self.assertEqual(found.to_api_repr(), expected)
+
+ def _verify_table_definitions(self, job, config):
+ table_defs = config.get("tableDefinitions")
+ if job.table_definitions is None:
+ self.assertIsNone(table_defs)
+ else:
+ self.assertEqual(len(job.table_definitions), len(table_defs))
+ for found_key, found_ec in job.table_definitions.items():
+ expected_ec = table_defs.get(found_key)
+ self.assertIsNotNone(expected_ec)
+ self.assertEqual(found_ec.to_api_repr(), expected_ec)
+
+ def _verify_configuration_properties(self, job, configuration):
+ if "dryRun" in configuration:
+ self.assertEqual(job.dry_run, configuration["dryRun"])
+ else:
+ self.assertIsNone(job.dry_run)
+
+ def _verifyResourceProperties(self, job, resource):
+ self._verifyReadonlyResourceProperties(job, resource)
+
+ configuration = resource.get("configuration", {})
+ self._verify_configuration_properties(job, configuration)
+
+ query_config = resource.get("configuration", {}).get("query")
+ self._verifyBooleanResourceProperties(job, query_config)
+ self._verifyIntegerResourceProperties(job, query_config)
+ self._verify_udf_resources(job, query_config)
+ self._verifyQueryParameters(job, query_config)
+ self._verify_table_definitions(job, query_config)
+
+ self.assertEqual(job.query, query_config["query"])
+ if "createDisposition" in query_config:
+ self.assertEqual(job.create_disposition, query_config["createDisposition"])
+ else:
+ self.assertIsNone(job.create_disposition)
+ if "defaultDataset" in query_config:
+ ds_ref = job.default_dataset
+ ds_ref = {"projectId": ds_ref.project, "datasetId": ds_ref.dataset_id}
+ self.assertEqual(ds_ref, query_config["defaultDataset"])
+ else:
+ self.assertIsNone(job.default_dataset)
+ if "destinationTable" in query_config:
+ table = job.destination
+ tb_ref = {
+ "projectId": table.project,
+ "datasetId": table.dataset_id,
+ "tableId": table.table_id,
+ }
+ self.assertEqual(tb_ref, query_config["destinationTable"])
+ else:
+ self.assertIsNone(job.destination)
+ if "priority" in query_config:
+ self.assertEqual(job.priority, query_config["priority"])
+ else:
+ self.assertIsNone(job.priority)
+ if "writeDisposition" in query_config:
+ self.assertEqual(job.write_disposition, query_config["writeDisposition"])
+ else:
+ self.assertIsNone(job.write_disposition)
+ if "destinationEncryptionConfiguration" in query_config:
+ self.assertIsNotNone(job.destination_encryption_configuration)
+ self.assertEqual(
+ job.destination_encryption_configuration.kms_key_name,
+ query_config["destinationEncryptionConfiguration"]["kmsKeyName"],
+ )
+ else:
+ self.assertIsNone(job.destination_encryption_configuration)
+ if "schemaUpdateOptions" in query_config:
+ self.assertEqual(
+ job.schema_update_options, query_config["schemaUpdateOptions"]
+ )
+ else:
+ self.assertIsNone(job.schema_update_options)
+
+ def test_ctor_defaults(self):
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, self.QUERY, client)
+ self.assertEqual(job.query, self.QUERY)
+ self.assertIs(job._client, client)
+ self.assertEqual(job.job_type, self.JOB_TYPE)
+ self.assertEqual(job.path, "/projects/%s/jobs/%s" % (self.PROJECT, self.JOB_ID))
+
+ self._verifyInitialReadonlyProperties(job)
+
+ self.assertFalse(job.use_legacy_sql)
+
+ # set/read from resource['configuration']['query']
+ self.assertIsNone(job.allow_large_results)
+ self.assertIsNone(job.create_disposition)
+ self.assertIsNone(job.default_dataset)
+ self.assertIsNone(job.destination)
+ self.assertIsNone(job.flatten_results)
+ self.assertIsNone(job.priority)
+ self.assertIsNone(job.use_query_cache)
+ self.assertIsNone(job.dry_run)
+ self.assertIsNone(job.write_disposition)
+ self.assertIsNone(job.maximum_billing_tier)
+ self.assertIsNone(job.maximum_bytes_billed)
+ self.assertIsNone(job.table_definitions)
+ self.assertIsNone(job.destination_encryption_configuration)
+ self.assertIsNone(job.time_partitioning)
+ self.assertIsNone(job.clustering_fields)
+ self.assertIsNone(job.schema_update_options)
+
+ def test_ctor_w_udf_resources(self):
+ from google.cloud.bigquery.job import QueryJobConfig
+ from google.cloud.bigquery.query import UDFResource
+
+ RESOURCE_URI = "gs://some-bucket/js/lib.js"
+ udf_resources = [UDFResource("resourceUri", RESOURCE_URI)]
+ client = _make_client(project=self.PROJECT)
+ config = QueryJobConfig()
+ config.udf_resources = udf_resources
+ job = self._make_one(self.JOB_ID, self.QUERY, client, job_config=config)
+ self.assertEqual(job.udf_resources, udf_resources)
+
+ def test_ctor_w_query_parameters(self):
+ from google.cloud.bigquery.job import QueryJobConfig
+ from google.cloud.bigquery.query import ScalarQueryParameter
+
+ query_parameters = [ScalarQueryParameter("foo", "INT64", 123)]
+ client = _make_client(project=self.PROJECT)
+ config = QueryJobConfig(query_parameters=query_parameters)
+ job = self._make_one(self.JOB_ID, self.QUERY, client, job_config=config)
+ self.assertEqual(job.query_parameters, query_parameters)
+
+ def test_from_api_repr_missing_identity(self):
+ self._setUpConstants()
+ client = _make_client(project=self.PROJECT)
+ RESOURCE = {}
+ klass = self._get_target_class()
+ with self.assertRaises(KeyError):
+ klass.from_api_repr(RESOURCE, client=client)
+
+ def test_from_api_repr_missing_config(self):
+ self._setUpConstants()
+ client = _make_client(project=self.PROJECT)
+ RESOURCE = {
+ "id": "%s:%s" % (self.PROJECT, self.DS_ID),
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ }
+ klass = self._get_target_class()
+ with self.assertRaises(KeyError):
+ klass.from_api_repr(RESOURCE, client=client)
+
+ def test_from_api_repr_bare(self):
+ self._setUpConstants()
+ client = _make_client(project=self.PROJECT)
+ RESOURCE = {
+ "id": self.JOB_ID,
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "configuration": {"query": {"query": self.QUERY}},
+ }
+ klass = self._get_target_class()
+ job = klass.from_api_repr(RESOURCE, client=client)
+ self.assertIs(job._client, client)
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_from_api_repr_with_encryption(self):
+ self._setUpConstants()
+ client = _make_client(project=self.PROJECT)
+ RESOURCE = {
+ "id": self.JOB_ID,
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "configuration": {
+ "query": {
+ "query": self.QUERY,
+ "destinationEncryptionConfiguration": {
+ "kmsKeyName": self.KMS_KEY_NAME
+ },
+ }
+ },
+ }
+ klass = self._get_target_class()
+ job = klass.from_api_repr(RESOURCE, client=client)
+ self.assertIs(job._client, client)
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_from_api_repr_w_properties(self):
+ from google.cloud.bigquery.job import CreateDisposition
+ from google.cloud.bigquery.job import SchemaUpdateOption
+ from google.cloud.bigquery.job import WriteDisposition
+
+ client = _make_client(project=self.PROJECT)
+ RESOURCE = self._make_resource()
+ query_config = RESOURCE["configuration"]["query"]
+ query_config["createDisposition"] = CreateDisposition.CREATE_IF_NEEDED
+ query_config["writeDisposition"] = WriteDisposition.WRITE_TRUNCATE
+ query_config["destinationTable"] = {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.DESTINATION_TABLE,
+ }
+ query_config["schemaUpdateOptions"] = [SchemaUpdateOption.ALLOW_FIELD_ADDITION]
+ klass = self._get_target_class()
+ job = klass.from_api_repr(RESOURCE, client=client)
+ self.assertIs(job._client, client)
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_cancelled(self):
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, self.QUERY, client)
+ job._properties["status"] = {
+ "state": "DONE",
+ "errorResult": {"reason": "stopped"},
+ }
+
+ self.assertTrue(job.cancelled())
+
+ def test_done(self):
+ client = _make_client(project=self.PROJECT)
+ resource = self._make_resource(ended=True)
+ job = self._get_target_class().from_api_repr(resource, client)
+ self.assertTrue(job.done())
+
+ def test_query_plan(self):
+ from google.cloud._helpers import _RFC3339_MICROS
+ from google.cloud.bigquery.job import QueryPlanEntry
+ from google.cloud.bigquery.job import QueryPlanEntryStep
+
+ plan_entries = [
+ {
+ "name": "NAME",
+ "id": "1234",
+ "inputStages": ["88", "101"],
+ "startMs": "1522540800000",
+ "endMs": "1522540804000",
+ "parallelInputs": "1000",
+ "completedParallelInputs": "5",
+ "waitMsAvg": "33",
+ "waitMsMax": "400",
+ "waitRatioAvg": 2.71828,
+ "waitRatioMax": 3.14159,
+ "readMsAvg": "45",
+ "readMsMax": "90",
+ "readRatioAvg": 1.41421,
+ "readRatioMax": 1.73205,
+ "computeMsAvg": "55",
+ "computeMsMax": "99",
+ "computeRatioAvg": 0.69315,
+ "computeRatioMax": 1.09861,
+ "writeMsAvg": "203",
+ "writeMsMax": "340",
+ "writeRatioAvg": 3.32193,
+ "writeRatioMax": 2.30258,
+ "recordsRead": "100",
+ "recordsWritten": "1",
+ "status": "STATUS",
+ "shuffleOutputBytes": "1024",
+ "shuffleOutputBytesSpilled": "1",
+ "steps": [{"kind": "KIND", "substeps": ["SUBSTEP1", "SUBSTEP2"]}],
+ }
+ ]
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, self.QUERY, client)
+ self.assertEqual(job.query_plan, [])
+
+ statistics = job._properties["statistics"] = {}
+ self.assertEqual(job.query_plan, [])
+
+ query_stats = statistics["query"] = {}
+ self.assertEqual(job.query_plan, [])
+
+ query_stats["queryPlan"] = plan_entries
+
+ self.assertEqual(len(job.query_plan), len(plan_entries))
+ for found, expected in zip(job.query_plan, plan_entries):
+ self.assertIsInstance(found, QueryPlanEntry)
+ self.assertEqual(found.name, expected["name"])
+ self.assertEqual(found.entry_id, expected["id"])
+ self.assertEqual(len(found.input_stages), len(expected["inputStages"]))
+ for f_id in found.input_stages:
+ self.assertIn(f_id, [int(e) for e in expected["inputStages"]])
+ self.assertEqual(
+ found.start.strftime(_RFC3339_MICROS), "2018-04-01T00:00:00.000000Z"
+ )
+ self.assertEqual(
+ found.end.strftime(_RFC3339_MICROS), "2018-04-01T00:00:04.000000Z"
+ )
+ self.assertEqual(found.parallel_inputs, int(expected["parallelInputs"]))
+ self.assertEqual(
+ found.completed_parallel_inputs,
+ int(expected["completedParallelInputs"]),
+ )
+ self.assertEqual(found.wait_ms_avg, int(expected["waitMsAvg"]))
+ self.assertEqual(found.wait_ms_max, int(expected["waitMsMax"]))
+ self.assertEqual(found.wait_ratio_avg, expected["waitRatioAvg"])
+ self.assertEqual(found.wait_ratio_max, expected["waitRatioMax"])
+ self.assertEqual(found.read_ms_avg, int(expected["readMsAvg"]))
+ self.assertEqual(found.read_ms_max, int(expected["readMsMax"]))
+ self.assertEqual(found.read_ratio_avg, expected["readRatioAvg"])
+ self.assertEqual(found.read_ratio_max, expected["readRatioMax"])
+ self.assertEqual(found.compute_ms_avg, int(expected["computeMsAvg"]))
+ self.assertEqual(found.compute_ms_max, int(expected["computeMsMax"]))
+ self.assertEqual(found.compute_ratio_avg, expected["computeRatioAvg"])
+ self.assertEqual(found.compute_ratio_max, expected["computeRatioMax"])
+ self.assertEqual(found.write_ms_avg, int(expected["writeMsAvg"]))
+ self.assertEqual(found.write_ms_max, int(expected["writeMsMax"]))
+ self.assertEqual(found.write_ratio_avg, expected["writeRatioAvg"])
+ self.assertEqual(found.write_ratio_max, expected["writeRatioMax"])
+ self.assertEqual(found.records_read, int(expected["recordsRead"]))
+ self.assertEqual(found.records_written, int(expected["recordsWritten"]))
+ self.assertEqual(found.status, expected["status"])
+ self.assertEqual(
+ found.shuffle_output_bytes, int(expected["shuffleOutputBytes"])
+ )
+ self.assertEqual(
+ found.shuffle_output_bytes_spilled,
+ int(expected["shuffleOutputBytesSpilled"]),
+ )
+
+ self.assertEqual(len(found.steps), len(expected["steps"]))
+ for f_step, e_step in zip(found.steps, expected["steps"]):
+ self.assertIsInstance(f_step, QueryPlanEntryStep)
+ self.assertEqual(f_step.kind, e_step["kind"])
+ self.assertEqual(f_step.substeps, e_step["substeps"])
+
+ def test_total_bytes_processed(self):
+ total_bytes = 1234
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, self.QUERY, client)
+ self.assertIsNone(job.total_bytes_processed)
+
+ statistics = job._properties["statistics"] = {}
+ self.assertIsNone(job.total_bytes_processed)
+
+ query_stats = statistics["query"] = {}
+ self.assertIsNone(job.total_bytes_processed)
+
+ query_stats["totalBytesProcessed"] = str(total_bytes)
+ self.assertEqual(job.total_bytes_processed, total_bytes)
+
+ def test_total_bytes_billed(self):
+ total_bytes = 1234
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, self.QUERY, client)
+ self.assertIsNone(job.total_bytes_billed)
+
+ statistics = job._properties["statistics"] = {}
+ self.assertIsNone(job.total_bytes_billed)
+
+ query_stats = statistics["query"] = {}
+ self.assertIsNone(job.total_bytes_billed)
+
+ query_stats["totalBytesBilled"] = str(total_bytes)
+ self.assertEqual(job.total_bytes_billed, total_bytes)
+
+ def test_billing_tier(self):
+ billing_tier = 1
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, self.QUERY, client)
+ self.assertIsNone(job.billing_tier)
+
+ statistics = job._properties["statistics"] = {}
+ self.assertIsNone(job.billing_tier)
+
+ query_stats = statistics["query"] = {}
+ self.assertIsNone(job.billing_tier)
+
+ query_stats["billingTier"] = billing_tier
+ self.assertEqual(job.billing_tier, billing_tier)
+
+ def test_cache_hit(self):
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, self.QUERY, client)
+ self.assertIsNone(job.cache_hit)
+
+ statistics = job._properties["statistics"] = {}
+ self.assertIsNone(job.cache_hit)
+
+ query_stats = statistics["query"] = {}
+ self.assertIsNone(job.cache_hit)
+
+ query_stats["cacheHit"] = True
+ self.assertTrue(job.cache_hit)
+
+ def test_ddl_operation_performed(self):
+ op = "SKIP"
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, self.QUERY, client)
+ self.assertIsNone(job.ddl_operation_performed)
+
+ statistics = job._properties["statistics"] = {}
+ self.assertIsNone(job.ddl_operation_performed)
+
+ query_stats = statistics["query"] = {}
+ self.assertIsNone(job.ddl_operation_performed)
+
+ query_stats["ddlOperationPerformed"] = op
+ self.assertEqual(job.ddl_operation_performed, op)
+
+ def test_ddl_target_routine(self):
+ from google.cloud.bigquery.routine import RoutineReference
+
+ ref_routine = {
+ "projectId": self.PROJECT,
+ "datasetId": "ddl_ds",
+ "routineId": "targetroutine",
+ }
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, self.QUERY, client)
+ self.assertIsNone(job.ddl_target_routine)
+
+ statistics = job._properties["statistics"] = {}
+ self.assertIsNone(job.ddl_target_routine)
+
+ query_stats = statistics["query"] = {}
+ self.assertIsNone(job.ddl_target_routine)
+
+ query_stats["ddlTargetRoutine"] = ref_routine
+ self.assertIsInstance(job.ddl_target_routine, RoutineReference)
+ self.assertEqual(job.ddl_target_routine.routine_id, "targetroutine")
+ self.assertEqual(job.ddl_target_routine.dataset_id, "ddl_ds")
+ self.assertEqual(job.ddl_target_routine.project, self.PROJECT)
+
+ def test_ddl_target_table(self):
+ from google.cloud.bigquery.table import TableReference
+
+ ref_table = {
+ "projectId": self.PROJECT,
+ "datasetId": "ddl_ds",
+ "tableId": "targettable",
+ }
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, self.QUERY, client)
+ self.assertIsNone(job.ddl_target_table)
+
+ statistics = job._properties["statistics"] = {}
+ self.assertIsNone(job.ddl_target_table)
+
+ query_stats = statistics["query"] = {}
+ self.assertIsNone(job.ddl_target_table)
+
+ query_stats["ddlTargetTable"] = ref_table
+ self.assertIsInstance(job.ddl_target_table, TableReference)
+ self.assertEqual(job.ddl_target_table.table_id, "targettable")
+ self.assertEqual(job.ddl_target_table.dataset_id, "ddl_ds")
+ self.assertEqual(job.ddl_target_table.project, self.PROJECT)
+
+ def test_num_dml_affected_rows(self):
+ num_rows = 1234
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, self.QUERY, client)
+ self.assertIsNone(job.num_dml_affected_rows)
+
+ statistics = job._properties["statistics"] = {}
+ self.assertIsNone(job.num_dml_affected_rows)
+
+ query_stats = statistics["query"] = {}
+ self.assertIsNone(job.num_dml_affected_rows)
+
+ query_stats["numDmlAffectedRows"] = str(num_rows)
+ self.assertEqual(job.num_dml_affected_rows, num_rows)
+
+ def test_slot_millis(self):
+ millis = 1234
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, self.QUERY, client)
+ self.assertIsNone(job.slot_millis)
+
+ statistics = job._properties["statistics"] = {}
+ self.assertIsNone(job.slot_millis)
+
+ query_stats = statistics["query"] = {}
+ self.assertIsNone(job.slot_millis)
+
+ query_stats["totalSlotMs"] = millis
+ self.assertEqual(job.slot_millis, millis)
+
+ def test_statement_type(self):
+ statement_type = "SELECT"
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, self.QUERY, client)
+ self.assertIsNone(job.statement_type)
+
+ statistics = job._properties["statistics"] = {}
+ self.assertIsNone(job.statement_type)
+
+ query_stats = statistics["query"] = {}
+ self.assertIsNone(job.statement_type)
+
+ query_stats["statementType"] = statement_type
+ self.assertEqual(job.statement_type, statement_type)
+
+ def test_referenced_tables(self):
+ from google.cloud.bigquery.table import TableReference
+
+ ref_tables_resource = [
+ {"projectId": self.PROJECT, "datasetId": "dataset", "tableId": "local1"},
+ {"projectId": self.PROJECT, "datasetId": "dataset", "tableId": "local2"},
+ {
+ "projectId": "other-project-123",
+ "datasetId": "other-dataset",
+ "tableId": "other-table",
+ },
+ ]
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, self.QUERY, client)
+ self.assertEqual(job.referenced_tables, [])
+
+ statistics = job._properties["statistics"] = {}
+ self.assertEqual(job.referenced_tables, [])
+
+ query_stats = statistics["query"] = {}
+ self.assertEqual(job.referenced_tables, [])
+
+ query_stats["referencedTables"] = ref_tables_resource
+
+ local1, local2, remote = job.referenced_tables
+
+ self.assertIsInstance(local1, TableReference)
+ self.assertEqual(local1.table_id, "local1")
+ self.assertEqual(local1.dataset_id, "dataset")
+ self.assertEqual(local1.project, self.PROJECT)
+
+ self.assertIsInstance(local2, TableReference)
+ self.assertEqual(local2.table_id, "local2")
+ self.assertEqual(local2.dataset_id, "dataset")
+ self.assertEqual(local2.project, self.PROJECT)
+
+ self.assertIsInstance(remote, TableReference)
+ self.assertEqual(remote.table_id, "other-table")
+ self.assertEqual(remote.dataset_id, "other-dataset")
+ self.assertEqual(remote.project, "other-project-123")
+
+ def test_timeline(self):
+ timeline_resource = [
+ {
+ "elapsedMs": 1,
+ "activeUnits": 22,
+ "pendingUnits": 33,
+ "completedUnits": 44,
+ "totalSlotMs": 101,
+ }
+ ]
+
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, self.QUERY, client)
+ self.assertEqual(job.timeline, [])
+
+ statistics = job._properties["statistics"] = {}
+ self.assertEqual(job.timeline, [])
+
+ query_stats = statistics["query"] = {}
+ self.assertEqual(job.timeline, [])
+
+ query_stats["timeline"] = timeline_resource
+
+ self.assertEqual(len(job.timeline), len(timeline_resource))
+ self.assertEqual(job.timeline[0].elapsed_ms, 1)
+ self.assertEqual(job.timeline[0].active_units, 22)
+ self.assertEqual(job.timeline[0].pending_units, 33)
+ self.assertEqual(job.timeline[0].completed_units, 44)
+ self.assertEqual(job.timeline[0].slot_millis, 101)
+
+ def test_undeclared_query_parameters(self):
+ from google.cloud.bigquery.query import ArrayQueryParameter
+ from google.cloud.bigquery.query import ScalarQueryParameter
+ from google.cloud.bigquery.query import StructQueryParameter
+
+ undeclared = [
+ {
+ "name": "my_scalar",
+ "parameterType": {"type": "STRING"},
+ "parameterValue": {"value": "value"},
+ },
+ {
+ "name": "my_array",
+ "parameterType": {"type": "ARRAY", "arrayType": {"type": "INT64"}},
+ "parameterValue": {
+ "arrayValues": [{"value": "1066"}, {"value": "1745"}]
+ },
+ },
+ {
+ "name": "my_struct",
+ "parameterType": {
+ "type": "STRUCT",
+ "structTypes": [{"name": "count", "type": {"type": "INT64"}}],
+ },
+ "parameterValue": {"structValues": {"count": {"value": "123"}}},
+ },
+ ]
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, self.QUERY, client)
+ self.assertEqual(job.undeclared_query_parameters, [])
+
+ statistics = job._properties["statistics"] = {}
+ self.assertEqual(job.undeclared_query_parameters, [])
+
+ query_stats = statistics["query"] = {}
+ self.assertEqual(job.undeclared_query_parameters, [])
+
+ query_stats["undeclaredQueryParameters"] = undeclared
+
+ scalar, array, struct = job.undeclared_query_parameters
+
+ self.assertIsInstance(scalar, ScalarQueryParameter)
+ self.assertEqual(scalar.name, "my_scalar")
+ self.assertEqual(scalar.type_, "STRING")
+ self.assertEqual(scalar.value, "value")
+
+ self.assertIsInstance(array, ArrayQueryParameter)
+ self.assertEqual(array.name, "my_array")
+ self.assertEqual(array.array_type, "INT64")
+ self.assertEqual(array.values, [1066, 1745])
+
+ self.assertIsInstance(struct, StructQueryParameter)
+ self.assertEqual(struct.name, "my_struct")
+ self.assertEqual(struct.struct_types, {"count": "INT64"})
+ self.assertEqual(struct.struct_values, {"count": 123})
+
+ def test_estimated_bytes_processed(self):
+ est_bytes = 123456
+
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, self.QUERY, client)
+ self.assertIsNone(job.estimated_bytes_processed)
+
+ statistics = job._properties["statistics"] = {}
+ self.assertIsNone(job.estimated_bytes_processed)
+
+ query_stats = statistics["query"] = {}
+ self.assertIsNone(job.estimated_bytes_processed)
+
+ query_stats["estimatedBytesProcessed"] = str(est_bytes)
+ self.assertEqual(job.estimated_bytes_processed, est_bytes)
+
+ def test_result(self):
+ from google.cloud.bigquery.table import RowIterator
+
+ query_resource = {
+ "jobComplete": True,
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "schema": {"fields": [{"name": "col1", "type": "STRING"}]},
+ "totalRows": "2",
+ }
+ tabledata_resource = {
+ # Explicitly set totalRows to be different from the query response.
+ # to test update during iteration.
+ "totalRows": "1",
+ "pageToken": None,
+ "rows": [{"f": [{"v": "abc"}]}],
+ }
+ connection = _make_connection(query_resource, tabledata_resource)
+ client = _make_client(self.PROJECT, connection=connection)
+ resource = self._make_resource(ended=True)
+ job = self._get_target_class().from_api_repr(resource, client)
+
+ result = job.result()
+
+ self.assertIsInstance(result, RowIterator)
+ self.assertEqual(result.total_rows, 2)
+
+ rows = list(result)
+ self.assertEqual(len(rows), 1)
+ self.assertEqual(rows[0].col1, "abc")
+ # Test that the total_rows property has changed during iteration, based
+ # on the response from tabledata.list.
+ self.assertEqual(result.total_rows, 1)
+
+ def test_result_w_empty_schema(self):
+ from google.cloud.bigquery.table import _EmptyRowIterator
+
+ # Destination table may have no schema for some DDL and DML queries.
+ query_resource = {
+ "jobComplete": True,
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "schema": {"fields": []},
+ }
+ connection = _make_connection(query_resource, query_resource)
+ client = _make_client(self.PROJECT, connection=connection)
+ resource = self._make_resource(ended=True)
+ job = self._get_target_class().from_api_repr(resource, client)
+
+ result = job.result()
+
+ self.assertIsInstance(result, _EmptyRowIterator)
+ self.assertEqual(list(result), [])
+
+ def test_result_invokes_begins(self):
+ begun_resource = self._make_resource()
+ incomplete_resource = {
+ "jobComplete": False,
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "schema": {"fields": [{"name": "col1", "type": "STRING"}]},
+ }
+ query_resource = copy.deepcopy(incomplete_resource)
+ query_resource["jobComplete"] = True
+ done_resource = copy.deepcopy(begun_resource)
+ done_resource["status"] = {"state": "DONE"}
+ connection = _make_connection(
+ begun_resource,
+ incomplete_resource,
+ query_resource,
+ done_resource,
+ query_resource,
+ )
+ client = _make_client(project=self.PROJECT, connection=connection)
+ job = self._make_one(self.JOB_ID, self.QUERY, client)
+
+ job.result()
+
+ self.assertEqual(len(connection.api_request.call_args_list), 4)
+ begin_request = connection.api_request.call_args_list[0]
+ query_request = connection.api_request.call_args_list[2]
+ reload_request = connection.api_request.call_args_list[3]
+ self.assertEqual(begin_request[1]["method"], "POST")
+ self.assertEqual(query_request[1]["method"], "GET")
+ self.assertEqual(reload_request[1]["method"], "GET")
+
+ def test_result_w_timeout(self):
+ begun_resource = self._make_resource()
+ query_resource = {
+ "jobComplete": True,
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "schema": {"fields": [{"name": "col1", "type": "STRING"}]},
+ }
+ done_resource = copy.deepcopy(begun_resource)
+ done_resource["status"] = {"state": "DONE"}
+ connection = _make_connection(begun_resource, query_resource, done_resource)
+ client = _make_client(project=self.PROJECT, connection=connection)
+ job = self._make_one(self.JOB_ID, self.QUERY, client)
+
+ job.result(timeout=1.0)
+
+ self.assertEqual(len(connection.api_request.call_args_list), 3)
+ begin_request = connection.api_request.call_args_list[0]
+ query_request = connection.api_request.call_args_list[1]
+ reload_request = connection.api_request.call_args_list[2]
+ self.assertEqual(begin_request[1]["method"], "POST")
+ self.assertEqual(query_request[1]["method"], "GET")
+ self.assertEqual(
+ query_request[1]["path"],
+ "/projects/{}/queries/{}".format(self.PROJECT, self.JOB_ID),
+ )
+ self.assertEqual(query_request[1]["query_params"]["timeoutMs"], 900)
+ self.assertEqual(reload_request[1]["method"], "GET")
+
+ def test_result_w_page_size(self):
+ # Arrange
+ query_results_resource = {
+ "jobComplete": True,
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "schema": {"fields": [{"name": "col1", "type": "STRING"}]},
+ "totalRows": "4",
+ }
+ job_resource = self._make_resource(started=True, ended=True)
+ q_config = job_resource["configuration"]["query"]
+ q_config["destinationTable"] = {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.TABLE_ID,
+ }
+ tabledata_resource = {
+ "totalRows": 4,
+ "pageToken": "some-page-token",
+ "rows": [
+ {"f": [{"v": "row1"}]},
+ {"f": [{"v": "row2"}]},
+ {"f": [{"v": "row3"}]},
+ ],
+ }
+ tabledata_resource_page_2 = {"totalRows": 4, "rows": [{"f": [{"v": "row4"}]}]}
+ conn = _make_connection(
+ query_results_resource, tabledata_resource, tabledata_resource_page_2
+ )
+ client = _make_client(self.PROJECT, connection=conn)
+ job = self._get_target_class().from_api_repr(job_resource, client)
+
+ # Act
+ result = job.result(page_size=3)
+
+ # Assert
+ actual_rows = list(result)
+ self.assertEqual(len(actual_rows), 4)
+
+ tabledata_path = "/projects/%s/datasets/%s/tables/%s/data" % (
+ self.PROJECT,
+ self.DS_ID,
+ self.TABLE_ID,
+ )
+ conn.api_request.assert_has_calls(
+ [
+ mock.call(
+ method="GET", path=tabledata_path, query_params={"maxResults": 3}
+ ),
+ mock.call(
+ method="GET",
+ path=tabledata_path,
+ query_params={"pageToken": "some-page-token", "maxResults": 3},
+ ),
+ ]
+ )
+
+ def test_result_error(self):
+ from google.cloud import exceptions
+
+ query = textwrap.dedent(
+ """
+ SELECT foo, bar
+ FROM table_baz
+ WHERE foo == bar"""
+ )
+
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, query, client)
+ error_result = {
+ "debugInfo": "DEBUG",
+ "location": "LOCATION",
+ "message": "MESSAGE",
+ "reason": "invalid",
+ }
+ job._properties["status"] = {
+ "errorResult": error_result,
+ "errors": [error_result],
+ "state": "DONE",
+ }
+ job._set_future_result()
+
+ with self.assertRaises(exceptions.GoogleCloudError) as exc_info:
+ job.result()
+
+ self.assertIsInstance(exc_info.exception, exceptions.GoogleCloudError)
+ self.assertEqual(exc_info.exception.code, http_client.BAD_REQUEST)
+
+ full_text = str(exc_info.exception)
+
+ assert job.job_id in full_text
+ assert "Query Job SQL Follows" in full_text
+
+ for i, line in enumerate(query.splitlines(), start=1):
+ expected_line = "{}:{}".format(i, line)
+ assert expected_line in full_text
+
+ def test__begin_error(self):
+ from google.cloud import exceptions
+
+ query = textwrap.dedent(
+ """
+ SELECT foo, bar
+ FROM table_baz
+ WHERE foo == bar"""
+ )
+
+ client = _make_client(project=self.PROJECT)
+ job = self._make_one(self.JOB_ID, query, client)
+ call_api_patch = mock.patch(
+ "google.cloud.bigquery.client.Client._call_api",
+ autospec=True,
+ side_effect=exceptions.BadRequest("Syntax error in SQL query"),
+ )
+
+ with call_api_patch, self.assertRaises(exceptions.GoogleCloudError) as exc_info:
+ job.result()
+
+ self.assertIsInstance(exc_info.exception, exceptions.GoogleCloudError)
+ self.assertEqual(exc_info.exception.code, http_client.BAD_REQUEST)
+
+ full_text = str(exc_info.exception)
+
+ assert job.job_id in full_text
+ assert "Query Job SQL Follows" in full_text
+
+ for i, line in enumerate(query.splitlines(), start=1):
+ expected_line = "{}:{}".format(i, line)
+ assert expected_line in full_text
+
+ def test_begin_w_bound_client(self):
+ from google.cloud.bigquery.dataset import DatasetReference
+ from google.cloud.bigquery.job import QueryJobConfig
+
+ PATH = "/projects/%s/jobs" % (self.PROJECT,)
+ DS_ID = "DATASET"
+ RESOURCE = self._make_resource()
+ # Ensure None for missing server-set props
+ del RESOURCE["statistics"]["creationTime"]
+ del RESOURCE["etag"]
+ del RESOURCE["selfLink"]
+ del RESOURCE["user_email"]
+ conn = _make_connection(RESOURCE)
+ client = _make_client(project=self.PROJECT, connection=conn)
+
+ config = QueryJobConfig()
+ config.default_dataset = DatasetReference(self.PROJECT, DS_ID)
+ job = self._make_one(self.JOB_ID, self.QUERY, client, job_config=config)
+
+ job._begin()
+
+ self.assertIsNone(job.default_dataset)
+ self.assertEqual(job.udf_resources, [])
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path=PATH,
+ data={
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "configuration": {
+ "query": {
+ "query": self.QUERY,
+ "useLegacySql": False,
+ "defaultDataset": {
+ "projectId": self.PROJECT,
+ "datasetId": DS_ID,
+ },
+ }
+ },
+ },
+ )
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_begin_w_alternate_client(self):
+ from google.cloud.bigquery.dataset import DatasetReference
+ from google.cloud.bigquery.job import CreateDisposition
+ from google.cloud.bigquery.job import QueryJobConfig
+ from google.cloud.bigquery.job import QueryPriority
+ from google.cloud.bigquery.job import SchemaUpdateOption
+ from google.cloud.bigquery.job import WriteDisposition
+
+ PATH = "/projects/%s/jobs" % (self.PROJECT,)
+ TABLE = "TABLE"
+ DS_ID = "DATASET"
+ RESOURCE = self._make_resource(ended=True)
+ QUERY_CONFIGURATION = {
+ "query": self.QUERY,
+ "allowLargeResults": True,
+ "createDisposition": CreateDisposition.CREATE_NEVER,
+ "defaultDataset": {"projectId": self.PROJECT, "datasetId": DS_ID},
+ "destinationTable": {
+ "projectId": self.PROJECT,
+ "datasetId": DS_ID,
+ "tableId": TABLE,
+ },
+ "flattenResults": True,
+ "priority": QueryPriority.INTERACTIVE,
+ "useQueryCache": True,
+ "useLegacySql": True,
+ "writeDisposition": WriteDisposition.WRITE_TRUNCATE,
+ "maximumBillingTier": 4,
+ "maximumBytesBilled": "123456",
+ "schemaUpdateOptions": [SchemaUpdateOption.ALLOW_FIELD_RELAXATION],
+ }
+ RESOURCE["configuration"]["query"] = QUERY_CONFIGURATION
+ RESOURCE["configuration"]["dryRun"] = True
+ conn1 = _make_connection()
+ client1 = _make_client(project=self.PROJECT, connection=conn1)
+ conn2 = _make_connection(RESOURCE)
+ client2 = _make_client(project=self.PROJECT, connection=conn2)
+ dataset_ref = DatasetReference(self.PROJECT, DS_ID)
+ table_ref = dataset_ref.table(TABLE)
+
+ config = QueryJobConfig()
+ config.allow_large_results = True
+ config.create_disposition = CreateDisposition.CREATE_NEVER
+ config.default_dataset = dataset_ref
+ config.destination = table_ref
+ config.dry_run = True
+ config.flatten_results = True
+ config.maximum_billing_tier = 4
+ config.priority = QueryPriority.INTERACTIVE
+ config.use_legacy_sql = True
+ config.use_query_cache = True
+ config.write_disposition = WriteDisposition.WRITE_TRUNCATE
+ config.maximum_bytes_billed = 123456
+ config.schema_update_options = [SchemaUpdateOption.ALLOW_FIELD_RELAXATION]
+ job = self._make_one(self.JOB_ID, self.QUERY, client1, job_config=config)
+
+ job._begin(client=client2)
+
+ conn1.api_request.assert_not_called()
+ conn2.api_request.assert_called_once_with(
+ method="POST",
+ path=PATH,
+ data={
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "configuration": {"dryRun": True, "query": QUERY_CONFIGURATION},
+ },
+ )
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_begin_w_udf(self):
+ from google.cloud.bigquery.job import QueryJobConfig
+ from google.cloud.bigquery.query import UDFResource
+
+ RESOURCE_URI = "gs://some-bucket/js/lib.js"
+ INLINE_UDF_CODE = 'var someCode = "here";'
+ PATH = "/projects/%s/jobs" % (self.PROJECT,)
+ RESOURCE = self._make_resource()
+ # Ensure None for missing server-set props
+ del RESOURCE["statistics"]["creationTime"]
+ del RESOURCE["etag"]
+ del RESOURCE["selfLink"]
+ del RESOURCE["user_email"]
+ RESOURCE["configuration"]["query"]["userDefinedFunctionResources"] = [
+ {"resourceUri": RESOURCE_URI},
+ {"inlineCode": INLINE_UDF_CODE},
+ ]
+ conn = _make_connection(RESOURCE)
+ client = _make_client(project=self.PROJECT, connection=conn)
+ udf_resources = [
+ UDFResource("resourceUri", RESOURCE_URI),
+ UDFResource("inlineCode", INLINE_UDF_CODE),
+ ]
+ config = QueryJobConfig()
+ config.udf_resources = udf_resources
+ config.use_legacy_sql = True
+ job = self._make_one(self.JOB_ID, self.QUERY, client, job_config=config)
+
+ job._begin()
+
+ self.assertEqual(job.udf_resources, udf_resources)
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path=PATH,
+ data={
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "configuration": {
+ "query": {
+ "query": self.QUERY,
+ "useLegacySql": True,
+ "userDefinedFunctionResources": [
+ {"resourceUri": RESOURCE_URI},
+ {"inlineCode": INLINE_UDF_CODE},
+ ],
+ }
+ },
+ },
+ )
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_begin_w_named_query_parameter(self):
+ from google.cloud.bigquery.job import QueryJobConfig
+ from google.cloud.bigquery.query import ScalarQueryParameter
+
+ query_parameters = [ScalarQueryParameter("foo", "INT64", 123)]
+ PATH = "/projects/%s/jobs" % (self.PROJECT,)
+ RESOURCE = self._make_resource()
+ # Ensure None for missing server-set props
+ del RESOURCE["statistics"]["creationTime"]
+ del RESOURCE["etag"]
+ del RESOURCE["selfLink"]
+ del RESOURCE["user_email"]
+ config = RESOURCE["configuration"]["query"]
+ config["parameterMode"] = "NAMED"
+ config["queryParameters"] = [
+ {
+ "name": "foo",
+ "parameterType": {"type": "INT64"},
+ "parameterValue": {"value": "123"},
+ }
+ ]
+ conn = _make_connection(RESOURCE)
+ client = _make_client(project=self.PROJECT, connection=conn)
+ jconfig = QueryJobConfig()
+ jconfig.query_parameters = query_parameters
+ job = self._make_one(self.JOB_ID, self.QUERY, client, job_config=jconfig)
+
+ job._begin()
+
+ self.assertEqual(job.query_parameters, query_parameters)
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path=PATH,
+ data={
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "configuration": {
+ "query": {
+ "query": self.QUERY,
+ "useLegacySql": False,
+ "parameterMode": "NAMED",
+ "queryParameters": config["queryParameters"],
+ }
+ },
+ },
+ )
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_begin_w_positional_query_parameter(self):
+ from google.cloud.bigquery.job import QueryJobConfig
+ from google.cloud.bigquery.query import ScalarQueryParameter
+
+ query_parameters = [ScalarQueryParameter.positional("INT64", 123)]
+ PATH = "/projects/%s/jobs" % (self.PROJECT,)
+ RESOURCE = self._make_resource()
+ # Ensure None for missing server-set props
+ del RESOURCE["statistics"]["creationTime"]
+ del RESOURCE["etag"]
+ del RESOURCE["selfLink"]
+ del RESOURCE["user_email"]
+ config = RESOURCE["configuration"]["query"]
+ config["parameterMode"] = "POSITIONAL"
+ config["queryParameters"] = [
+ {"parameterType": {"type": "INT64"}, "parameterValue": {"value": "123"}}
+ ]
+ conn = _make_connection(RESOURCE)
+ client = _make_client(project=self.PROJECT, connection=conn)
+ jconfig = QueryJobConfig()
+ jconfig.query_parameters = query_parameters
+ job = self._make_one(self.JOB_ID, self.QUERY, client, job_config=jconfig)
+
+ job._begin()
+
+ self.assertEqual(job.query_parameters, query_parameters)
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path=PATH,
+ data={
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "configuration": {
+ "query": {
+ "query": self.QUERY,
+ "useLegacySql": False,
+ "parameterMode": "POSITIONAL",
+ "queryParameters": config["queryParameters"],
+ }
+ },
+ },
+ )
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_begin_w_table_defs(self):
+ from google.cloud.bigquery.job import QueryJobConfig
+ from google.cloud.bigquery.external_config import ExternalConfig
+ from google.cloud.bigquery.external_config import BigtableColumn
+ from google.cloud.bigquery.external_config import BigtableColumnFamily
+
+ PATH = "/projects/%s/jobs" % (self.PROJECT,)
+ RESOURCE = self._make_resource()
+ # Ensure None for missing server-set props
+ del RESOURCE["statistics"]["creationTime"]
+ del RESOURCE["etag"]
+ del RESOURCE["selfLink"]
+ del RESOURCE["user_email"]
+
+ bt_config = ExternalConfig("BIGTABLE")
+ bt_config.ignore_unknown_values = True
+ bt_config.options.read_rowkey_as_string = True
+ cf = BigtableColumnFamily()
+ cf.family_id = "cf"
+ col = BigtableColumn()
+ col.field_name = "fn"
+ cf.columns = [col]
+ bt_config.options.column_families = [cf]
+ BT_CONFIG_RESOURCE = {
+ "sourceFormat": "BIGTABLE",
+ "ignoreUnknownValues": True,
+ "bigtableOptions": {
+ "readRowkeyAsString": True,
+ "columnFamilies": [
+ {"familyId": "cf", "columns": [{"fieldName": "fn"}]}
+ ],
+ },
+ }
+ CSV_CONFIG_RESOURCE = {
+ "sourceFormat": "CSV",
+ "maxBadRecords": 8,
+ "csvOptions": {"allowJaggedRows": True},
+ }
+ csv_config = ExternalConfig("CSV")
+ csv_config.max_bad_records = 8
+ csv_config.options.allow_jagged_rows = True
+ bt_table = "bigtable-table"
+ csv_table = "csv-table"
+ RESOURCE["configuration"]["query"]["tableDefinitions"] = {
+ bt_table: BT_CONFIG_RESOURCE,
+ csv_table: CSV_CONFIG_RESOURCE,
+ }
+ want_resource = copy.deepcopy(RESOURCE)
+ conn = _make_connection(RESOURCE)
+ client = _make_client(project=self.PROJECT, connection=conn)
+ config = QueryJobConfig()
+ config.table_definitions = {bt_table: bt_config, csv_table: csv_config}
+ config.use_legacy_sql = True
+ job = self._make_one(self.JOB_ID, self.QUERY, client, job_config=config)
+
+ job._begin()
+
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path=PATH,
+ data={
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "configuration": {
+ "query": {
+ "query": self.QUERY,
+ "useLegacySql": True,
+ "tableDefinitions": {
+ bt_table: BT_CONFIG_RESOURCE,
+ csv_table: CSV_CONFIG_RESOURCE,
+ },
+ }
+ },
+ },
+ )
+ self._verifyResourceProperties(job, want_resource)
+
+ def test_dry_run_query(self):
+ from google.cloud.bigquery.job import QueryJobConfig
+
+ PATH = "/projects/%s/jobs" % (self.PROJECT,)
+ RESOURCE = self._make_resource()
+ # Ensure None for missing server-set props
+ del RESOURCE["statistics"]["creationTime"]
+ del RESOURCE["etag"]
+ del RESOURCE["selfLink"]
+ del RESOURCE["user_email"]
+ RESOURCE["configuration"]["dryRun"] = True
+ conn = _make_connection(RESOURCE)
+ client = _make_client(project=self.PROJECT, connection=conn)
+ config = QueryJobConfig()
+ config.dry_run = True
+ job = self._make_one(self.JOB_ID, self.QUERY, client, job_config=config)
+
+ job._begin()
+ self.assertEqual(job.udf_resources, [])
+ conn.api_request.assert_called_once_with(
+ method="POST",
+ path=PATH,
+ data={
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "configuration": {
+ "query": {"query": self.QUERY, "useLegacySql": False},
+ "dryRun": True,
+ },
+ },
+ )
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_exists_miss_w_bound_client(self):
+ PATH = "/projects/%s/jobs/%s" % (self.PROJECT, self.JOB_ID)
+ conn = _make_connection()
+ client = _make_client(project=self.PROJECT, connection=conn)
+ job = self._make_one(self.JOB_ID, self.QUERY, client)
+
+ self.assertFalse(job.exists())
+
+ conn.api_request.assert_called_once_with(
+ method="GET", path=PATH, query_params={"fields": "id"}
+ )
+
+ def test_exists_hit_w_alternate_client(self):
+ PATH = "/projects/%s/jobs/%s" % (self.PROJECT, self.JOB_ID)
+ conn1 = _make_connection()
+ client1 = _make_client(project=self.PROJECT, connection=conn1)
+ conn2 = _make_connection({})
+ client2 = _make_client(project=self.PROJECT, connection=conn2)
+ job = self._make_one(self.JOB_ID, self.QUERY, client1)
+
+ self.assertTrue(job.exists(client=client2))
+
+ conn1.api_request.assert_not_called()
+ conn2.api_request.assert_called_once_with(
+ method="GET", path=PATH, query_params={"fields": "id"}
+ )
+
+ def test_reload_w_bound_client(self):
+ from google.cloud.bigquery.dataset import DatasetReference
+ from google.cloud.bigquery.job import QueryJobConfig
+
+ PATH = "/projects/%s/jobs/%s" % (self.PROJECT, self.JOB_ID)
+ DS_ID = "DATASET"
+ DEST_TABLE = "dest_table"
+ RESOURCE = self._make_resource()
+ conn = _make_connection(RESOURCE)
+ client = _make_client(project=self.PROJECT, connection=conn)
+ dataset_ref = DatasetReference(self.PROJECT, DS_ID)
+ table_ref = dataset_ref.table(DEST_TABLE)
+ config = QueryJobConfig()
+ config.destination = table_ref
+ job = self._make_one(self.JOB_ID, None, client, job_config=config)
+
+ job.reload()
+
+ self.assertNotEqual(job.destination, table_ref)
+
+ conn.api_request.assert_called_once_with(
+ method="GET", path=PATH, query_params={}
+ )
+ self._verifyResourceProperties(job, RESOURCE)
+
+ def test_reload_w_alternate_client(self):
+ PATH = "/projects/%s/jobs/%s" % (self.PROJECT, self.JOB_ID)
+ DS_ID = "DATASET"
+ DEST_TABLE = "dest_table"
+ RESOURCE = self._make_resource()
+ q_config = RESOURCE["configuration"]["query"]
+ q_config["destinationTable"] = {
+ "projectId": self.PROJECT,
+ "datasetId": DS_ID,
+ "tableId": DEST_TABLE,
+ }
+ conn1 = _make_connection()
+ client1 = _make_client(project=self.PROJECT, connection=conn1)
+ conn2 = _make_connection(RESOURCE)
+ client2 = _make_client(project=self.PROJECT, connection=conn2)
+ job = self._make_one(self.JOB_ID, self.QUERY, client1)
+
+ job.reload(client=client2)
+
+ conn1.api_request.assert_not_called()
+ conn2.api_request.assert_called_once_with(
+ method="GET", path=PATH, query_params={}
+ )
+ self._verifyResourceProperties(job, RESOURCE)
+
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ def test_to_arrow(self):
+ begun_resource = self._make_resource()
+ query_resource = {
+ "jobComplete": True,
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "totalRows": "4",
+ "schema": {
+ "fields": [
+ {
+ "name": "spouse_1",
+ "type": "RECORD",
+ "fields": [
+ {"name": "name", "type": "STRING", "mode": "NULLABLE"},
+ {"name": "age", "type": "INTEGER", "mode": "NULLABLE"},
+ ],
+ },
+ {
+ "name": "spouse_2",
+ "type": "RECORD",
+ "fields": [
+ {"name": "name", "type": "STRING", "mode": "NULLABLE"},
+ {"name": "age", "type": "INTEGER", "mode": "NULLABLE"},
+ ],
+ },
+ ]
+ },
+ }
+ tabledata_resource = {
+ "rows": [
+ {
+ "f": [
+ {"v": {"f": [{"v": "Phred Phlyntstone"}, {"v": "32"}]}},
+ {"v": {"f": [{"v": "Wylma Phlyntstone"}, {"v": "29"}]}},
+ ]
+ },
+ {
+ "f": [
+ {"v": {"f": [{"v": "Bhettye Rhubble"}, {"v": "27"}]}},
+ {"v": {"f": [{"v": "Bharney Rhubble"}, {"v": "33"}]}},
+ ]
+ },
+ ]
+ }
+ done_resource = copy.deepcopy(begun_resource)
+ done_resource["status"] = {"state": "DONE"}
+ connection = _make_connection(
+ begun_resource, query_resource, done_resource, tabledata_resource
+ )
+ client = _make_client(project=self.PROJECT, connection=connection)
+ job = self._make_one(self.JOB_ID, self.QUERY, client)
+
+ tbl = job.to_arrow()
+
+ self.assertIsInstance(tbl, pyarrow.Table)
+ self.assertEqual(tbl.num_rows, 2)
+
+ # Check the schema.
+ self.assertEqual(tbl.schema[0].name, "spouse_1")
+ self.assertEqual(tbl.schema[0].type[0].name, "name")
+ self.assertEqual(tbl.schema[0].type[1].name, "age")
+ self.assertTrue(pyarrow.types.is_struct(tbl.schema[0].type))
+ self.assertTrue(pyarrow.types.is_string(tbl.schema[0].type[0].type))
+ self.assertTrue(pyarrow.types.is_int64(tbl.schema[0].type[1].type))
+ self.assertEqual(tbl.schema[1].name, "spouse_2")
+ self.assertEqual(tbl.schema[1].type[0].name, "name")
+ self.assertEqual(tbl.schema[1].type[1].name, "age")
+ self.assertTrue(pyarrow.types.is_struct(tbl.schema[1].type))
+ self.assertTrue(pyarrow.types.is_string(tbl.schema[1].type[0].type))
+ self.assertTrue(pyarrow.types.is_int64(tbl.schema[1].type[1].type))
+
+ # Check the data.
+ tbl_data = tbl.to_pydict()
+ spouse_1 = tbl_data["spouse_1"]
+ self.assertEqual(
+ spouse_1,
+ [
+ {"name": "Phred Phlyntstone", "age": 32},
+ {"name": "Bhettye Rhubble", "age": 27},
+ ],
+ )
+ spouse_2 = tbl_data["spouse_2"]
+ self.assertEqual(
+ spouse_2,
+ [
+ {"name": "Wylma Phlyntstone", "age": 29},
+ {"name": "Bharney Rhubble", "age": 33},
+ ],
+ )
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ def test_to_dataframe(self):
+ begun_resource = self._make_resource()
+ query_resource = {
+ "jobComplete": True,
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "totalRows": "4",
+ "schema": {
+ "fields": [
+ {"name": "name", "type": "STRING", "mode": "NULLABLE"},
+ {"name": "age", "type": "INTEGER", "mode": "NULLABLE"},
+ ]
+ },
+ }
+ tabledata_resource = {
+ "rows": [
+ {"f": [{"v": "Phred Phlyntstone"}, {"v": "32"}]},
+ {"f": [{"v": "Bharney Rhubble"}, {"v": "33"}]},
+ {"f": [{"v": "Wylma Phlyntstone"}, {"v": "29"}]},
+ {"f": [{"v": "Bhettye Rhubble"}, {"v": "27"}]},
+ ]
+ }
+ done_resource = copy.deepcopy(begun_resource)
+ done_resource["status"] = {"state": "DONE"}
+ connection = _make_connection(
+ begun_resource, query_resource, done_resource, tabledata_resource
+ )
+ client = _make_client(project=self.PROJECT, connection=connection)
+ job = self._make_one(self.JOB_ID, self.QUERY, client)
+
+ df = job.to_dataframe()
+
+ self.assertIsInstance(df, pandas.DataFrame)
+ self.assertEqual(len(df), 4) # verify the number of rows
+ self.assertEqual(list(df), ["name", "age"]) # verify the column names
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ def test_to_dataframe_ddl_query(self):
+ # Destination table may have no schema for some DDL and DML queries.
+ query_resource = {
+ "jobComplete": True,
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "schema": {"fields": []},
+ }
+ connection = _make_connection(query_resource)
+ client = _make_client(self.PROJECT, connection=connection)
+ resource = self._make_resource(ended=True)
+ job = self._get_target_class().from_api_repr(resource, client)
+
+ df = job.to_dataframe()
+
+ self.assertEqual(len(df), 0)
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(
+ bigquery_storage_v1beta1 is None, "Requires `google-cloud-bigquery-storage`"
+ )
+ def test_to_dataframe_bqstorage(self):
+ query_resource = {
+ "jobComplete": True,
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "totalRows": "4",
+ "schema": {
+ "fields": [
+ {"name": "name", "type": "STRING", "mode": "NULLABLE"},
+ {"name": "age", "type": "INTEGER", "mode": "NULLABLE"},
+ ]
+ },
+ }
+ connection = _make_connection(query_resource)
+ client = _make_client(self.PROJECT, connection=connection)
+ resource = self._make_resource(ended=True)
+ job = self._get_target_class().from_api_repr(resource, client)
+ bqstorage_client = mock.create_autospec(
+ bigquery_storage_v1beta1.BigQueryStorageClient
+ )
+ session = bigquery_storage_v1beta1.types.ReadSession()
+ session.avro_schema.schema = json.dumps(
+ {
+ "type": "record",
+ "name": "__root__",
+ "fields": [
+ {"name": "name", "type": ["null", "string"]},
+ {"name": "age", "type": ["null", "long"]},
+ ],
+ }
+ )
+ bqstorage_client.create_read_session.return_value = session
+
+ job.to_dataframe(bqstorage_client=bqstorage_client)
+
+ bqstorage_client.create_read_session.assert_called_once_with(
+ mock.ANY,
+ "projects/{}".format(self.PROJECT),
+ format_=bigquery_storage_v1beta1.enums.DataFormat.ARROW,
+ read_options=mock.ANY,
+ # Use default number of streams for best performance.
+ requested_streams=0,
+ )
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ def test_to_dataframe_column_dtypes(self):
+ begun_resource = self._make_resource()
+ query_resource = {
+ "jobComplete": True,
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "totalRows": "4",
+ "schema": {
+ "fields": [
+ {"name": "start_timestamp", "type": "TIMESTAMP"},
+ {"name": "seconds", "type": "INT64"},
+ {"name": "miles", "type": "FLOAT64"},
+ {"name": "km", "type": "FLOAT64"},
+ {"name": "payment_type", "type": "STRING"},
+ {"name": "complete", "type": "BOOL"},
+ {"name": "date", "type": "DATE"},
+ ]
+ },
+ }
+ row_data = [
+ ["1.4338368E9", "420", "1.1", "1.77", "Cash", "true", "1999-12-01"],
+ ["1.3878117E9", "2580", "17.7", "28.5", "Cash", "false", "1953-06-14"],
+ ["1.3855653E9", "2280", "4.4", "7.1", "Credit", "true", "1981-11-04"],
+ ]
+ rows = [{"f": [{"v": field} for field in row]} for row in row_data]
+ query_resource["rows"] = rows
+ done_resource = copy.deepcopy(begun_resource)
+ done_resource["status"] = {"state": "DONE"}
+ connection = _make_connection(
+ begun_resource, query_resource, done_resource, query_resource
+ )
+ client = _make_client(project=self.PROJECT, connection=connection)
+ job = self._make_one(self.JOB_ID, self.QUERY, client)
+
+ df = job.to_dataframe(dtypes={"km": "float16"})
+
+ self.assertIsInstance(df, pandas.DataFrame)
+ self.assertEqual(len(df), 3) # verify the number of rows
+ exp_columns = [field["name"] for field in query_resource["schema"]["fields"]]
+ self.assertEqual(list(df), exp_columns) # verify the column names
+
+ self.assertEqual(df.start_timestamp.dtype.name, "datetime64[ns, UTC]")
+ self.assertEqual(df.seconds.dtype.name, "int64")
+ self.assertEqual(df.miles.dtype.name, "float64")
+ self.assertEqual(df.km.dtype.name, "float16")
+ self.assertEqual(df.payment_type.dtype.name, "object")
+ self.assertEqual(df.complete.dtype.name, "bool")
+ self.assertEqual(df.date.dtype.name, "object")
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(tqdm is None, "Requires `tqdm`")
+ @mock.patch("tqdm.tqdm")
+ def test_to_dataframe_with_progress_bar(self, tqdm_mock):
+ begun_resource = self._make_resource()
+ query_resource = {
+ "jobComplete": True,
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "totalRows": "4",
+ "schema": {
+ "fields": [{"name": "name", "type": "STRING", "mode": "NULLABLE"}]
+ },
+ }
+ done_resource = copy.deepcopy(begun_resource)
+ done_resource["status"] = {"state": "DONE"}
+ connection = _make_connection(
+ begun_resource,
+ query_resource,
+ done_resource,
+ query_resource,
+ query_resource,
+ )
+ client = _make_client(project=self.PROJECT, connection=connection)
+ job = self._make_one(self.JOB_ID, self.QUERY, client)
+
+ job.to_dataframe(progress_bar_type=None)
+ tqdm_mock.assert_not_called()
+
+ job.to_dataframe(progress_bar_type="tqdm")
+ tqdm_mock.assert_called()
+
+ def test_iter(self):
+ import types
+
+ begun_resource = self._make_resource()
+ query_resource = {
+ "jobComplete": True,
+ "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+ "totalRows": "0",
+ "schema": {"fields": [{"name": "col1", "type": "STRING"}]},
+ }
+ done_resource = copy.deepcopy(begun_resource)
+ done_resource["status"] = {"state": "DONE"}
+ connection = _make_connection(begun_resource, query_resource, done_resource)
+ client = _make_client(project=self.PROJECT, connection=connection)
+ job = self._make_one(self.JOB_ID, self.QUERY, client)
+
+ self.assertIsInstance(iter(job), types.GeneratorType)
+
+
+class TestQueryPlanEntryStep(unittest.TestCase, _Base):
+ KIND = "KIND"
+ SUBSTEPS = ("SUB1", "SUB2")
+
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery.job import QueryPlanEntryStep
+
+ return QueryPlanEntryStep
+
+ def _make_one(self, *args, **kw):
+ return self._get_target_class()(*args, **kw)
+
+ def test_ctor(self):
+ step = self._make_one(self.KIND, self.SUBSTEPS)
+ self.assertEqual(step.kind, self.KIND)
+ self.assertEqual(step.substeps, list(self.SUBSTEPS))
+
+ def test_from_api_repr_empty(self):
+ klass = self._get_target_class()
+ step = klass.from_api_repr({})
+ self.assertIsNone(step.kind)
+ self.assertEqual(step.substeps, [])
+
+ def test_from_api_repr_normal(self):
+ resource = {"kind": self.KIND, "substeps": self.SUBSTEPS}
+ klass = self._get_target_class()
+ step = klass.from_api_repr(resource)
+ self.assertEqual(step.kind, self.KIND)
+ self.assertEqual(step.substeps, list(self.SUBSTEPS))
+
+ def test___eq___mismatched_type(self):
+ step = self._make_one(self.KIND, self.SUBSTEPS)
+ self.assertNotEqual(step, object())
+
+ def test___eq___mismatch_kind(self):
+ step = self._make_one(self.KIND, self.SUBSTEPS)
+ other = self._make_one("OTHER", self.SUBSTEPS)
+ self.assertNotEqual(step, other)
+
+ def test___eq___mismatch_substeps(self):
+ step = self._make_one(self.KIND, self.SUBSTEPS)
+ other = self._make_one(self.KIND, ())
+ self.assertNotEqual(step, other)
+
+ def test___eq___hit(self):
+ step = self._make_one(self.KIND, self.SUBSTEPS)
+ other = self._make_one(self.KIND, self.SUBSTEPS)
+ self.assertEqual(step, other)
+
+ def test___eq___wrong_type(self):
+ step = self._make_one(self.KIND, self.SUBSTEPS)
+ self.assertFalse(step == "hello")
+
+
+class TestQueryPlanEntry(unittest.TestCase, _Base):
+ NAME = "NAME"
+ ENTRY_ID = 1234
+ START_MS = 1522540800000
+ END_MS = 1522540804000
+ INPUT_STAGES = (88, 101)
+ PARALLEL_INPUTS = 1000
+ COMPLETED_PARALLEL_INPUTS = 5
+ WAIT_MS_AVG = 33
+ WAIT_MS_MAX = 400
+ WAIT_RATIO_AVG = 2.71828
+ WAIT_RATIO_MAX = 3.14159
+ READ_MS_AVG = 45
+ READ_MS_MAX = 90
+ READ_RATIO_AVG = 1.41421
+ READ_RATIO_MAX = 1.73205
+ COMPUTE_MS_AVG = 55
+ COMPUTE_MS_MAX = 99
+ COMPUTE_RATIO_AVG = 0.69315
+ COMPUTE_RATIO_MAX = 1.09861
+ WRITE_MS_AVG = 203
+ WRITE_MS_MAX = 340
+ WRITE_RATIO_AVG = 3.32193
+ WRITE_RATIO_MAX = 2.30258
+ RECORDS_READ = 100
+ RECORDS_WRITTEN = 1
+ STATUS = "STATUS"
+ SHUFFLE_OUTPUT_BYTES = 1024
+ SHUFFLE_OUTPUT_BYTES_SPILLED = 1
+
+ START_RFC3339_MICROS = "2018-04-01T00:00:00.000000Z"
+ END_RFC3339_MICROS = "2018-04-01T00:00:04.000000Z"
+
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery.job import QueryPlanEntry
+
+ return QueryPlanEntry
+
+ def test_from_api_repr_empty(self):
+ klass = self._get_target_class()
+
+ entry = klass.from_api_repr({})
+
+ self.assertIsNone(entry.name)
+ self.assertIsNone(entry.entry_id)
+ self.assertEqual(entry.input_stages, [])
+ self.assertIsNone(entry.start)
+ self.assertIsNone(entry.end)
+ self.assertIsNone(entry.parallel_inputs)
+ self.assertIsNone(entry.completed_parallel_inputs)
+ self.assertIsNone(entry.wait_ms_avg)
+ self.assertIsNone(entry.wait_ms_max)
+ self.assertIsNone(entry.wait_ratio_avg)
+ self.assertIsNone(entry.wait_ratio_max)
+ self.assertIsNone(entry.read_ms_avg)
+ self.assertIsNone(entry.read_ms_max)
+ self.assertIsNone(entry.read_ratio_avg)
+ self.assertIsNone(entry.read_ratio_max)
+ self.assertIsNone(entry.compute_ms_avg)
+ self.assertIsNone(entry.compute_ms_max)
+ self.assertIsNone(entry.compute_ratio_avg)
+ self.assertIsNone(entry.compute_ratio_max)
+ self.assertIsNone(entry.write_ms_avg)
+ self.assertIsNone(entry.write_ms_max)
+ self.assertIsNone(entry.write_ratio_avg)
+ self.assertIsNone(entry.write_ratio_max)
+ self.assertIsNone(entry.records_read)
+ self.assertIsNone(entry.records_written)
+ self.assertIsNone(entry.status)
+ self.assertIsNone(entry.shuffle_output_bytes)
+ self.assertIsNone(entry.shuffle_output_bytes_spilled)
+ self.assertEqual(entry.steps, [])
+
+ def test_from_api_repr_normal(self):
+ from google.cloud.bigquery.job import QueryPlanEntryStep
+
+ steps = [
+ QueryPlanEntryStep(
+ kind=TestQueryPlanEntryStep.KIND,
+ substeps=TestQueryPlanEntryStep.SUBSTEPS,
+ )
+ ]
+ resource = {
+ "name": self.NAME,
+ "id": self.ENTRY_ID,
+ "inputStages": self.INPUT_STAGES,
+ "startMs": self.START_MS,
+ "endMs": self.END_MS,
+ "waitMsAvg": self.WAIT_MS_AVG,
+ "waitMsMax": self.WAIT_MS_MAX,
+ "waitRatioAvg": self.WAIT_RATIO_AVG,
+ "waitRatioMax": self.WAIT_RATIO_MAX,
+ "readMsAvg": self.READ_MS_AVG,
+ "readMsMax": self.READ_MS_MAX,
+ "readRatioAvg": self.READ_RATIO_AVG,
+ "readRatioMax": self.READ_RATIO_MAX,
+ "computeMsAvg": self.COMPUTE_MS_AVG,
+ "computeMsMax": self.COMPUTE_MS_MAX,
+ "computeRatioAvg": self.COMPUTE_RATIO_AVG,
+ "computeRatioMax": self.COMPUTE_RATIO_MAX,
+ "writeMsAvg": self.WRITE_MS_AVG,
+ "writeMsMax": self.WRITE_MS_MAX,
+ "writeRatioAvg": self.WRITE_RATIO_AVG,
+ "writeRatioMax": self.WRITE_RATIO_MAX,
+ "recordsRead": self.RECORDS_READ,
+ "recordsWritten": self.RECORDS_WRITTEN,
+ "status": self.STATUS,
+ "shuffleOutputBytes": self.SHUFFLE_OUTPUT_BYTES,
+ "shuffleOutputBytesSpilled": self.SHUFFLE_OUTPUT_BYTES_SPILLED,
+ "steps": [
+ {
+ "kind": TestQueryPlanEntryStep.KIND,
+ "substeps": TestQueryPlanEntryStep.SUBSTEPS,
+ }
+ ],
+ }
+ klass = self._get_target_class()
+
+ entry = klass.from_api_repr(resource)
+ self.assertEqual(entry.name, self.NAME)
+ self.assertEqual(entry.entry_id, self.ENTRY_ID)
+ self.assertEqual(entry.wait_ratio_avg, self.WAIT_RATIO_AVG)
+ self.assertEqual(entry.wait_ratio_max, self.WAIT_RATIO_MAX)
+ self.assertEqual(entry.read_ratio_avg, self.READ_RATIO_AVG)
+ self.assertEqual(entry.read_ratio_max, self.READ_RATIO_MAX)
+ self.assertEqual(entry.compute_ratio_avg, self.COMPUTE_RATIO_AVG)
+ self.assertEqual(entry.compute_ratio_max, self.COMPUTE_RATIO_MAX)
+ self.assertEqual(entry.write_ratio_avg, self.WRITE_RATIO_AVG)
+ self.assertEqual(entry.write_ratio_max, self.WRITE_RATIO_MAX)
+ self.assertEqual(entry.records_read, self.RECORDS_READ)
+ self.assertEqual(entry.records_written, self.RECORDS_WRITTEN)
+ self.assertEqual(entry.status, self.STATUS)
+ self.assertEqual(entry.steps, steps)
+
+ def test_start(self):
+ from google.cloud._helpers import _RFC3339_MICROS
+
+ klass = self._get_target_class()
+
+ entry = klass.from_api_repr({})
+ self.assertEqual(entry.start, None)
+
+ entry._properties["startMs"] = self.START_MS
+ self.assertEqual(
+ entry.start.strftime(_RFC3339_MICROS), self.START_RFC3339_MICROS
+ )
+
+ def test_end(self):
+ from google.cloud._helpers import _RFC3339_MICROS
+
+ klass = self._get_target_class()
+
+ entry = klass.from_api_repr({})
+ self.assertEqual(entry.end, None)
+
+ entry._properties["endMs"] = self.END_MS
+ self.assertEqual(entry.end.strftime(_RFC3339_MICROS), self.END_RFC3339_MICROS)
+
+
+class TestTimelineEntry(unittest.TestCase, _Base):
+ ELAPSED_MS = 101
+ ACTIVE_UNITS = 50
+ PENDING_UNITS = 98
+ COMPLETED_UNITS = 520
+ SLOT_MILLIS = 12029
+
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery.job import TimelineEntry
+
+ return TimelineEntry
+
+ def test_from_api_repr_empty(self):
+ klass = self._get_target_class()
+ entry = klass.from_api_repr({})
+ self.assertIsNone(entry.elapsed_ms)
+ self.assertIsNone(entry.active_units)
+ self.assertIsNone(entry.pending_units)
+ self.assertIsNone(entry.completed_units)
+ self.assertIsNone(entry.slot_millis)
+
+ def test_from_api_repr_normal(self):
+ resource = {
+ "elapsedMs": self.ELAPSED_MS,
+ "activeUnits": self.ACTIVE_UNITS,
+ "pendingUnits": self.PENDING_UNITS,
+ "completedUnits": self.COMPLETED_UNITS,
+ "totalSlotMs": self.SLOT_MILLIS,
+ }
+ klass = self._get_target_class()
+
+ entry = klass.from_api_repr(resource)
+ self.assertEqual(entry.elapsed_ms, self.ELAPSED_MS)
+ self.assertEqual(entry.active_units, self.ACTIVE_UNITS)
+ self.assertEqual(entry.pending_units, self.PENDING_UNITS)
+ self.assertEqual(entry.completed_units, self.COMPLETED_UNITS)
+ self.assertEqual(entry.slot_millis, self.SLOT_MILLIS)
+
+
+@pytest.mark.parametrize(
+ "query,expected",
+ (
+ (None, False),
+ ("", False),
+ ("select name, age from table", False),
+ ("select name, age from table LIMIT 10;", False),
+ ("select name, age from table order by other_column;", True),
+ ("Select name, age From table Order By other_column", True),
+ ("SELECT name, age FROM table ORDER BY other_column;", True),
+ ("select name, age from table order\nby other_column", True),
+ ("Select name, age From table Order\nBy other_column;", True),
+ ("SELECT name, age FROM table ORDER\nBY other_column", True),
+ ("SelecT name, age froM table OrdeR \n\t BY other_column;", True),
+ ),
+)
+def test__contains_order_by(query, expected):
+ from google.cloud.bigquery import job as mut
+
+ if expected:
+ assert mut._contains_order_by(query)
+ else:
+ assert not mut._contains_order_by(query)
+
+
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+@pytest.mark.skipif(
+ bigquery_storage_v1beta1 is None, reason="Requires `google-cloud-bigquery-storage`"
+)
+@pytest.mark.parametrize(
+ "query",
+ (
+ "select name, age from table order by other_column;",
+ "Select name, age From table Order By other_column;",
+ "SELECT name, age FROM table ORDER BY other_column;",
+ "select name, age from table order\nby other_column;",
+ "Select name, age From table Order\nBy other_column;",
+ "SELECT name, age FROM table ORDER\nBY other_column;",
+ "SelecT name, age froM table OrdeR \n\t BY other_column;",
+ ),
+)
+def test_to_dataframe_bqstorage_preserve_order(query):
+ from google.cloud.bigquery.job import QueryJob as target_class
+
+ job_resource = _make_job_resource(
+ project_id="test-project", job_type="query", ended=True
+ )
+ job_resource["configuration"]["query"]["query"] = query
+ job_resource["status"] = {"state": "DONE"}
+ get_query_results_resource = {
+ "jobComplete": True,
+ "jobReference": {"projectId": "test-project", "jobId": "test-job"},
+ "schema": {
+ "fields": [
+ {"name": "name", "type": "STRING", "mode": "NULLABLE"},
+ {"name": "age", "type": "INTEGER", "mode": "NULLABLE"},
+ ]
+ },
+ "totalRows": "4",
+ }
+ connection = _make_connection(get_query_results_resource, job_resource)
+ client = _make_client(connection=connection)
+ job = target_class.from_api_repr(job_resource, client)
+ bqstorage_client = mock.create_autospec(
+ bigquery_storage_v1beta1.BigQueryStorageClient
+ )
+ session = bigquery_storage_v1beta1.types.ReadSession()
+ session.avro_schema.schema = json.dumps(
+ {
+ "type": "record",
+ "name": "__root__",
+ "fields": [
+ {"name": "name", "type": ["null", "string"]},
+ {"name": "age", "type": ["null", "long"]},
+ ],
+ }
+ )
+ bqstorage_client.create_read_session.return_value = session
+
+ job.to_dataframe(bqstorage_client=bqstorage_client)
+
+ bqstorage_client.create_read_session.assert_called_once_with(
+ mock.ANY,
+ "projects/test-project",
+ format_=bigquery_storage_v1beta1.enums.DataFormat.ARROW,
+ read_options=mock.ANY,
+ # Use a single stream to preserve row order.
+ requested_streams=1,
+ )
diff --git a/bigquery/tests/unit/test_magics.py b/bigquery/tests/unit/test_magics.py
new file mode 100644
index 000000000000..fbea9bdd9050
--- /dev/null
+++ b/bigquery/tests/unit/test_magics.py
@@ -0,0 +1,1009 @@
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import copy
+import re
+from concurrent import futures
+
+import mock
+import pytest
+import six
+
+try:
+ import pandas
+except ImportError: # pragma: NO COVER
+ pandas = None
+try:
+ import IPython
+ from IPython.utils import io
+ from IPython.testing import tools
+ from IPython.terminal import interactiveshell
+except ImportError: # pragma: NO COVER
+ IPython = None
+
+from google.api_core import exceptions
+import google.auth.credentials
+
+try:
+ from google.cloud import bigquery_storage_v1beta1
+except ImportError: # pragma: NO COVER
+ bigquery_storage_v1beta1 = None
+from google.cloud.bigquery import job
+from google.cloud.bigquery import table
+from google.cloud.bigquery import magics
+from tests.unit.helpers import make_connection
+from test_utils.imports import maybe_fail_import
+
+
+pytestmark = pytest.mark.skipif(IPython is None, reason="Requires `ipython`")
+
+
+@pytest.fixture(scope="session")
+def ipython():
+ config = tools.default_config()
+ config.TerminalInteractiveShell.simple_prompt = True
+ shell = interactiveshell.TerminalInteractiveShell.instance(config=config)
+ return shell
+
+
+@pytest.fixture()
+def ipython_interactive(request, ipython):
+ """Activate IPython's builtin hooks
+
+ for the duration of the test scope.
+ """
+ with ipython.builtin_trap:
+ yield ipython
+
+
+@pytest.fixture(scope="session")
+def missing_bq_storage():
+ """Provide a patcher that can make the bigquery storage import to fail."""
+
+ def fail_if(name, globals, locals, fromlist, level):
+ # NOTE: *very* simplified, assuming a straightforward absolute import
+ return "bigquery_storage_v1beta1" in name or (
+ fromlist is not None and "bigquery_storage_v1beta1" in fromlist
+ )
+
+ return maybe_fail_import(predicate=fail_if)
+
+
+@pytest.fixture(scope="session")
+def missing_grpcio_lib():
+ """Provide a patcher that can make the gapic library import to fail."""
+
+ def fail_if(name, globals, locals, fromlist, level):
+ # NOTE: *very* simplified, assuming a straightforward absolute import
+ return "gapic_v1" in name or (fromlist is not None and "gapic_v1" in fromlist)
+
+ return maybe_fail_import(predicate=fail_if)
+
+
+JOB_REFERENCE_RESOURCE = {"projectId": "its-a-project-eh", "jobId": "some-random-id"}
+TABLE_REFERENCE_RESOURCE = {
+ "projectId": "its-a-project-eh",
+ "datasetId": "ds",
+ "tableId": "persons",
+}
+QUERY_RESOURCE = {
+ "jobReference": JOB_REFERENCE_RESOURCE,
+ "configuration": {
+ "query": {
+ "destinationTable": TABLE_REFERENCE_RESOURCE,
+ "query": "SELECT 42 FROM `life.the_universe.and_everything`;",
+ "queryParameters": [],
+ "useLegacySql": False,
+ }
+ },
+ "status": {"state": "DONE"},
+}
+
+
+def test_context_credentials_auto_set_w_application_default_credentials():
+ """When Application Default Credentials are set, the context credentials
+ will be created the first time it is called
+ """
+ assert magics.context._credentials is None
+ assert magics.context._project is None
+
+ project = "prahj-ekt"
+ credentials_mock = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+ default_patch = mock.patch(
+ "google.auth.default", return_value=(credentials_mock, project)
+ )
+ with default_patch as default_mock:
+ assert magics.context.credentials is credentials_mock
+ assert magics.context.project == project
+
+ assert default_mock.call_count == 2
+
+
+def test_context_credentials_and_project_can_be_set_explicitly():
+ project1 = "one-project-55564"
+ project2 = "other-project-52569"
+ credentials_mock = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+ default_patch = mock.patch(
+ "google.auth.default", return_value=(credentials_mock, project1)
+ )
+ with default_patch as default_mock:
+ magics.context.credentials = credentials_mock
+ magics.context.project = project2
+
+ assert magics.context.project == project2
+ assert magics.context.credentials is credentials_mock
+ # default should not be called if credentials & project are explicitly set
+ assert default_mock.call_count == 0
+
+
+@pytest.mark.usefixtures("ipython_interactive")
+def test_context_connection_can_be_overriden():
+ ip = IPython.get_ipython()
+ ip.extension_manager.load_extension("google.cloud.bigquery")
+ magics.context._project = None
+ magics.context._credentials = None
+
+ credentials_mock = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+ project = "project-123"
+ default_patch = mock.patch(
+ "google.auth.default", return_value=(credentials_mock, project)
+ )
+ job_reference = copy.deepcopy(JOB_REFERENCE_RESOURCE)
+ job_reference["projectId"] = project
+
+ query = "select * from persons"
+ resource = copy.deepcopy(QUERY_RESOURCE)
+ resource["jobReference"] = job_reference
+ resource["configuration"]["query"]["query"] = query
+ data = {"jobReference": job_reference, "totalRows": 0, "rows": []}
+
+ conn = magics.context._connection = make_connection(resource, data)
+ list_rows_patch = mock.patch(
+ "google.cloud.bigquery.client.Client.list_rows",
+ return_value=google.cloud.bigquery.table._EmptyRowIterator(),
+ )
+ with list_rows_patch as list_rows, default_patch:
+ ip.run_cell_magic("bigquery", "", query)
+
+ # Check that query actually starts the job.
+ list_rows.assert_called()
+ assert len(conn.api_request.call_args_list) == 2
+ _, req = conn.api_request.call_args_list[0]
+ assert req["method"] == "POST"
+ assert req["path"] == "/projects/{}/jobs".format(project)
+ sent = req["data"]
+ assert isinstance(sent["jobReference"]["jobId"], six.string_types)
+ sent_config = sent["configuration"]["query"]
+ assert sent_config["query"] == query
+
+
+@pytest.mark.usefixtures("ipython_interactive")
+def test_context_no_connection():
+ ip = IPython.get_ipython()
+ ip.extension_manager.load_extension("google.cloud.bigquery")
+ magics.context._project = None
+ magics.context._credentials = None
+ magics.context._connection = None
+
+ credentials_mock = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+ project = "project-123"
+ default_patch = mock.patch(
+ "google.auth.default", return_value=(credentials_mock, project)
+ )
+ job_reference = copy.deepcopy(JOB_REFERENCE_RESOURCE)
+ job_reference["projectId"] = project
+
+ query = "select * from persons"
+ resource = copy.deepcopy(QUERY_RESOURCE)
+ resource["jobReference"] = job_reference
+ resource["configuration"]["query"]["query"] = query
+ data = {"jobReference": job_reference, "totalRows": 0, "rows": []}
+
+ conn_mock = make_connection(resource, data, data, data)
+ conn_patch = mock.patch("google.cloud.bigquery.client.Connection", autospec=True)
+ list_rows_patch = mock.patch(
+ "google.cloud.bigquery.client.Client.list_rows",
+ return_value=google.cloud.bigquery.table._EmptyRowIterator(),
+ )
+ with conn_patch as conn, list_rows_patch as list_rows, default_patch:
+ conn.return_value = conn_mock
+ ip.run_cell_magic("bigquery", "", query)
+
+ # Check that query actually starts the job.
+ list_rows.assert_called()
+ assert len(conn_mock.api_request.call_args_list) == 2
+ _, req = conn_mock.api_request.call_args_list[0]
+ assert req["method"] == "POST"
+ assert req["path"] == "/projects/{}/jobs".format(project)
+ sent = req["data"]
+ assert isinstance(sent["jobReference"]["jobId"], six.string_types)
+ sent_config = sent["configuration"]["query"]
+ assert sent_config["query"] == query
+
+
+def test__run_query():
+ magics.context._credentials = None
+
+ job_id = "job_1234"
+ sql = "SELECT 17"
+ responses = [
+ futures.TimeoutError,
+ futures.TimeoutError,
+ [table.Row((17,), {"num": 0})],
+ ]
+
+ client_patch = mock.patch(
+ "google.cloud.bigquery.magics.bigquery.Client", autospec=True
+ )
+ with client_patch as client_mock, io.capture_output() as captured:
+ client_mock().query(sql).result.side_effect = responses
+ client_mock().query(sql).job_id = job_id
+
+ query_job = magics._run_query(client_mock(), sql)
+
+ lines = re.split("\n|\r", captured.stdout)
+ # Removes blanks & terminal code (result of display clearing)
+ updates = list(filter(lambda x: bool(x) and x != "\x1b[2K", lines))
+
+ assert query_job.job_id == job_id
+ expected_first_line = "Executing query with job ID: {}".format(job_id)
+ assert updates[0] == expected_first_line
+ execution_updates = updates[1:-1]
+ assert len(execution_updates) == 3 # one update per API response
+ for line in execution_updates:
+ assert re.match("Query executing: .*s", line)
+ assert re.match("Query complete after .*s", updates[-1])
+
+
+def test__run_query_dry_run_without_errors_is_silent():
+ magics.context._credentials = None
+
+ sql = "SELECT 17"
+
+ client_patch = mock.patch(
+ "google.cloud.bigquery.magics.bigquery.Client", autospec=True
+ )
+
+ job_config = job.QueryJobConfig()
+ job_config.dry_run = True
+ with client_patch as client_mock, io.capture_output() as captured:
+ client_mock().query(sql).job_id = None
+ magics._run_query(client_mock(), sql, job_config=job_config)
+
+ assert len(captured.stderr) == 0
+ assert len(captured.stdout) == 0
+
+
+def test__make_bqstorage_client_false():
+ credentials_mock = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+ got = magics._make_bqstorage_client(False, credentials_mock)
+ assert got is None
+
+
+@pytest.mark.skipif(
+ bigquery_storage_v1beta1 is None, reason="Requires `google-cloud-bigquery-storage`"
+)
+def test__make_bqstorage_client_true():
+ credentials_mock = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+ got = magics._make_bqstorage_client(True, credentials_mock)
+ assert isinstance(got, bigquery_storage_v1beta1.BigQueryStorageClient)
+
+
+def test__make_bqstorage_client_true_raises_import_error(missing_bq_storage):
+ credentials_mock = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+
+ with pytest.raises(ImportError) as exc_context, missing_bq_storage:
+ magics._make_bqstorage_client(True, credentials_mock)
+
+ error_msg = str(exc_context.value)
+ assert "google-cloud-bigquery-storage" in error_msg
+ assert "pyarrow" in error_msg
+
+
+def test__make_bqstorage_client_true_missing_gapic(missing_grpcio_lib):
+ credentials_mock = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+
+ with pytest.raises(ImportError) as exc_context, missing_grpcio_lib:
+ magics._make_bqstorage_client(True, credentials_mock)
+
+ assert "grpcio" in str(exc_context.value)
+
+
+@pytest.mark.usefixtures("ipython_interactive")
+def test_extension_load():
+ ip = IPython.get_ipython()
+ ip.extension_manager.load_extension("google.cloud.bigquery")
+
+ # verify that the magic is registered and has the correct source
+ magic = ip.magics_manager.magics["cell"].get("bigquery")
+ assert magic.__module__ == "google.cloud.bigquery.magics"
+
+
+@pytest.mark.usefixtures("ipython_interactive")
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+def test_bigquery_magic_without_optional_arguments(missing_bq_storage):
+ ip = IPython.get_ipython()
+ ip.extension_manager.load_extension("google.cloud.bigquery")
+ magics.context.credentials = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+
+ sql = "SELECT 17 AS num"
+ result = pandas.DataFrame([17], columns=["num"])
+ run_query_patch = mock.patch(
+ "google.cloud.bigquery.magics._run_query", autospec=True
+ )
+ query_job_mock = mock.create_autospec(
+ google.cloud.bigquery.job.QueryJob, instance=True
+ )
+ query_job_mock.to_dataframe.return_value = result
+
+ # Shouldn't fail when BigQuery Storage client isn't installed.
+ with run_query_patch as run_query_mock, missing_bq_storage:
+ run_query_mock.return_value = query_job_mock
+ return_value = ip.run_cell_magic("bigquery", "", sql)
+
+ assert isinstance(return_value, pandas.DataFrame)
+ assert len(return_value) == len(result) # verify row count
+ assert list(return_value) == list(result) # verify column names
+
+
+@pytest.mark.usefixtures("ipython_interactive")
+def test_bigquery_magic_default_connection_user_agent():
+ ip = IPython.get_ipython()
+ ip.extension_manager.load_extension("google.cloud.bigquery")
+ magics.context._connection = None
+
+ credentials_mock = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+ default_patch = mock.patch(
+ "google.auth.default", return_value=(credentials_mock, "general-project")
+ )
+ run_query_patch = mock.patch(
+ "google.cloud.bigquery.magics._run_query", autospec=True
+ )
+ conn_patch = mock.patch("google.cloud.bigquery.client.Connection", autospec=True)
+
+ with conn_patch as conn, run_query_patch, default_patch:
+ ip.run_cell_magic("bigquery", "", "SELECT 17 as num")
+
+ client_info_arg = conn.call_args.kwargs.get("client_info")
+ assert client_info_arg is not None
+ assert client_info_arg.user_agent == "ipython-" + IPython.__version__
+
+
+@pytest.mark.usefixtures("ipython_interactive")
+def test_bigquery_magic_with_legacy_sql():
+ ip = IPython.get_ipython()
+ ip.extension_manager.load_extension("google.cloud.bigquery")
+ magics.context.credentials = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+
+ run_query_patch = mock.patch(
+ "google.cloud.bigquery.magics._run_query", autospec=True
+ )
+ with run_query_patch as run_query_mock:
+ ip.run_cell_magic("bigquery", "--use_legacy_sql", "SELECT 17 AS num")
+
+ job_config_used = run_query_mock.call_args_list[0][0][-1]
+ assert job_config_used.use_legacy_sql is True
+
+
+@pytest.mark.usefixtures("ipython_interactive")
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+def test_bigquery_magic_with_result_saved_to_variable():
+ ip = IPython.get_ipython()
+ ip.extension_manager.load_extension("google.cloud.bigquery")
+ magics.context.credentials = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+
+ sql = "SELECT 17 AS num"
+ result = pandas.DataFrame([17], columns=["num"])
+ assert "df" not in ip.user_ns
+
+ run_query_patch = mock.patch(
+ "google.cloud.bigquery.magics._run_query", autospec=True
+ )
+ query_job_mock = mock.create_autospec(
+ google.cloud.bigquery.job.QueryJob, instance=True
+ )
+ query_job_mock.to_dataframe.return_value = result
+ with run_query_patch as run_query_mock:
+ run_query_mock.return_value = query_job_mock
+
+ return_value = ip.run_cell_magic("bigquery", "df", sql)
+
+ assert return_value is None
+ assert "df" in ip.user_ns # verify that variable exists
+ df = ip.user_ns["df"]
+ assert len(df) == len(result) # verify row count
+ assert list(df) == list(result) # verify column names
+
+
+@pytest.mark.usefixtures("ipython_interactive")
+def test_bigquery_magic_does_not_clear_display_in_verbose_mode():
+ ip = IPython.get_ipython()
+ ip.extension_manager.load_extension("google.cloud.bigquery")
+ magics.context.credentials = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+
+ clear_patch = mock.patch(
+ "google.cloud.bigquery.magics.display.clear_output", autospec=True
+ )
+ run_query_patch = mock.patch(
+ "google.cloud.bigquery.magics._run_query", autospec=True
+ )
+ with clear_patch as clear_mock, run_query_patch:
+ ip.run_cell_magic("bigquery", "--verbose", "SELECT 17 as num")
+
+ assert clear_mock.call_count == 0
+
+
+@pytest.mark.usefixtures("ipython_interactive")
+def test_bigquery_magic_clears_display_in_verbose_mode():
+ ip = IPython.get_ipython()
+ ip.extension_manager.load_extension("google.cloud.bigquery")
+ magics.context.credentials = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+
+ clear_patch = mock.patch(
+ "google.cloud.bigquery.magics.display.clear_output", autospec=True
+ )
+ run_query_patch = mock.patch(
+ "google.cloud.bigquery.magics._run_query", autospec=True
+ )
+ with clear_patch as clear_mock, run_query_patch:
+ ip.run_cell_magic("bigquery", "", "SELECT 17 as num")
+
+ assert clear_mock.call_count == 1
+
+
+@pytest.mark.usefixtures("ipython_interactive")
+@pytest.mark.skipif(
+ bigquery_storage_v1beta1 is None, reason="Requires `google-cloud-bigquery-storage`"
+)
+def test_bigquery_magic_with_bqstorage_from_argument(monkeypatch):
+ ip = IPython.get_ipython()
+ ip.extension_manager.load_extension("google.cloud.bigquery")
+ mock_credentials = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+
+ # Set up the context with monkeypatch so that it's reset for subsequent
+ # tests.
+ monkeypatch.setattr(magics.context, "credentials", mock_credentials)
+ monkeypatch.setattr(magics.context, "use_bqstorage_api", False)
+
+ # Mock out the BigQuery Storage API.
+ bqstorage_mock = mock.create_autospec(
+ bigquery_storage_v1beta1.BigQueryStorageClient
+ )
+ bqstorage_instance_mock = mock.create_autospec(
+ bigquery_storage_v1beta1.BigQueryStorageClient, instance=True
+ )
+ bqstorage_mock.return_value = bqstorage_instance_mock
+ bqstorage_client_patch = mock.patch(
+ "google.cloud.bigquery_storage_v1beta1.BigQueryStorageClient", bqstorage_mock
+ )
+
+ sql = "SELECT 17 AS num"
+ result = pandas.DataFrame([17], columns=["num"])
+ run_query_patch = mock.patch(
+ "google.cloud.bigquery.magics._run_query", autospec=True
+ )
+ query_job_mock = mock.create_autospec(
+ google.cloud.bigquery.job.QueryJob, instance=True
+ )
+ query_job_mock.to_dataframe.return_value = result
+ with run_query_patch as run_query_mock, bqstorage_client_patch:
+ run_query_mock.return_value = query_job_mock
+
+ return_value = ip.run_cell_magic("bigquery", "--use_bqstorage_api", sql)
+
+ assert len(bqstorage_mock.call_args_list) == 1
+ kwargs = bqstorage_mock.call_args_list[0].kwargs
+ assert kwargs.get("credentials") is mock_credentials
+ client_info = kwargs.get("client_info")
+ assert client_info is not None
+ assert client_info.user_agent == "ipython-" + IPython.__version__
+
+ query_job_mock.to_dataframe.assert_called_once_with(
+ bqstorage_client=bqstorage_instance_mock
+ )
+
+ assert isinstance(return_value, pandas.DataFrame)
+
+
+@pytest.mark.usefixtures("ipython_interactive")
+@pytest.mark.skipif(
+ bigquery_storage_v1beta1 is None, reason="Requires `google-cloud-bigquery-storage`"
+)
+def test_bigquery_magic_with_bqstorage_from_context(monkeypatch):
+ ip = IPython.get_ipython()
+ ip.extension_manager.load_extension("google.cloud.bigquery")
+ mock_credentials = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+
+ # Set up the context with monkeypatch so that it's reset for subsequent
+ # tests.
+ monkeypatch.setattr(magics.context, "credentials", mock_credentials)
+ monkeypatch.setattr(magics.context, "use_bqstorage_api", True)
+
+ # Mock out the BigQuery Storage API.
+ bqstorage_mock = mock.create_autospec(
+ bigquery_storage_v1beta1.BigQueryStorageClient
+ )
+ bqstorage_instance_mock = mock.create_autospec(
+ bigquery_storage_v1beta1.BigQueryStorageClient, instance=True
+ )
+ bqstorage_mock.return_value = bqstorage_instance_mock
+ bqstorage_client_patch = mock.patch(
+ "google.cloud.bigquery_storage_v1beta1.BigQueryStorageClient", bqstorage_mock
+ )
+
+ sql = "SELECT 17 AS num"
+ result = pandas.DataFrame([17], columns=["num"])
+ run_query_patch = mock.patch(
+ "google.cloud.bigquery.magics._run_query", autospec=True
+ )
+ query_job_mock = mock.create_autospec(
+ google.cloud.bigquery.job.QueryJob, instance=True
+ )
+ query_job_mock.to_dataframe.return_value = result
+ with run_query_patch as run_query_mock, bqstorage_client_patch:
+ run_query_mock.return_value = query_job_mock
+
+ return_value = ip.run_cell_magic("bigquery", "", sql)
+
+ assert len(bqstorage_mock.call_args_list) == 1
+ kwargs = bqstorage_mock.call_args_list[0].kwargs
+ assert kwargs.get("credentials") is mock_credentials
+ client_info = kwargs.get("client_info")
+ assert client_info is not None
+ assert client_info.user_agent == "ipython-" + IPython.__version__
+
+ query_job_mock.to_dataframe.assert_called_once_with(
+ bqstorage_client=bqstorage_instance_mock
+ )
+
+ assert isinstance(return_value, pandas.DataFrame)
+
+
+@pytest.mark.usefixtures("ipython_interactive")
+@pytest.mark.skipif(
+ bigquery_storage_v1beta1 is None, reason="Requires `google-cloud-bigquery-storage`"
+)
+def test_bigquery_magic_without_bqstorage(monkeypatch):
+ ip = IPython.get_ipython()
+ ip.extension_manager.load_extension("google.cloud.bigquery")
+ mock_credentials = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+
+ # Set up the context with monkeypatch so that it's reset for subsequent
+ # tests.
+ monkeypatch.setattr(magics.context, "credentials", mock_credentials)
+
+ # Mock out the BigQuery Storage API.
+ bqstorage_mock = mock.create_autospec(
+ bigquery_storage_v1beta1.BigQueryStorageClient
+ )
+ bqstorage_client_patch = mock.patch(
+ "google.cloud.bigquery_storage_v1beta1.BigQueryStorageClient", bqstorage_mock
+ )
+
+ sql = "SELECT 17 AS num"
+ result = pandas.DataFrame([17], columns=["num"])
+ run_query_patch = mock.patch(
+ "google.cloud.bigquery.magics._run_query", autospec=True
+ )
+ query_job_mock = mock.create_autospec(
+ google.cloud.bigquery.job.QueryJob, instance=True
+ )
+ query_job_mock.to_dataframe.return_value = result
+ with run_query_patch as run_query_mock, bqstorage_client_patch:
+ run_query_mock.return_value = query_job_mock
+
+ return_value = ip.run_cell_magic("bigquery", "", sql)
+
+ bqstorage_mock.assert_not_called()
+ query_job_mock.to_dataframe.assert_called_once_with(bqstorage_client=None)
+
+ assert isinstance(return_value, pandas.DataFrame)
+
+
+@pytest.mark.usefixtures("ipython_interactive")
+def test_bigquery_magic_dryrun_option_sets_job_config():
+ ip = IPython.get_ipython()
+ ip.extension_manager.load_extension("google.cloud.bigquery")
+ magics.context.credentials = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+
+ run_query_patch = mock.patch(
+ "google.cloud.bigquery.magics._run_query", autospec=True
+ )
+
+ sql = "SELECT 17 AS num"
+
+ with run_query_patch as run_query_mock:
+ ip.run_cell_magic("bigquery", "--dry_run", sql)
+
+ job_config_used = run_query_mock.call_args_list[0][0][-1]
+ assert job_config_used.dry_run is True
+
+
+@pytest.mark.usefixtures("ipython_interactive")
+def test_bigquery_magic_dryrun_option_returns_query_job():
+ ip = IPython.get_ipython()
+ ip.extension_manager.load_extension("google.cloud.bigquery")
+ magics.context.credentials = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+ query_job_mock = mock.create_autospec(
+ google.cloud.bigquery.job.QueryJob, instance=True
+ )
+ run_query_patch = mock.patch(
+ "google.cloud.bigquery.magics._run_query", autospec=True
+ )
+
+ sql = "SELECT 17 AS num"
+
+ with run_query_patch as run_query_mock, io.capture_output() as captured_io:
+ run_query_mock.return_value = query_job_mock
+ return_value = ip.run_cell_magic("bigquery", "--dry_run", sql)
+
+ assert "Query validated. This query will process" in captured_io.stdout
+ assert isinstance(return_value, job.QueryJob)
+
+
+@pytest.mark.usefixtures("ipython_interactive")
+def test_bigquery_magic_dryrun_option_variable_error_message():
+ ip = IPython.get_ipython()
+ ip.extension_manager.load_extension("google.cloud.bigquery")
+ magics.context.credentials = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+
+ run_query_patch = mock.patch(
+ "google.cloud.bigquery.magics._run_query",
+ autospec=True,
+ side_effect=exceptions.BadRequest("Syntax error in SQL query"),
+ )
+
+ sql = "SELECT SELECT 17 AS num"
+
+ assert "q_job" not in ip.user_ns
+
+ with run_query_patch, io.capture_output() as captured:
+ ip.run_cell_magic("bigquery", "q_job --dry_run", sql)
+
+ full_text = captured.stderr
+ assert "Could not save output to variable 'q_job'." in full_text
+
+
+@pytest.mark.usefixtures("ipython_interactive")
+def test_bigquery_magic_dryrun_option_saves_query_job_to_variable():
+ ip = IPython.get_ipython()
+ ip.extension_manager.load_extension("google.cloud.bigquery")
+ magics.context.credentials = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+ query_job_mock = mock.create_autospec(
+ google.cloud.bigquery.job.QueryJob, instance=True
+ )
+ run_query_patch = mock.patch(
+ "google.cloud.bigquery.magics._run_query", autospec=True
+ )
+
+ sql = "SELECT 17 AS num"
+
+ assert "q_job" not in ip.user_ns
+
+ with run_query_patch as run_query_mock:
+ run_query_mock.return_value = query_job_mock
+ return_value = ip.run_cell_magic("bigquery", "q_job --dry_run", sql)
+
+ assert return_value is None
+ assert "q_job" in ip.user_ns
+ q_job = ip.user_ns["q_job"]
+ assert isinstance(q_job, job.QueryJob)
+
+
+@pytest.mark.usefixtures("ipython_interactive")
+def test_bigquery_magic_w_maximum_bytes_billed_invalid():
+ ip = IPython.get_ipython()
+ ip.extension_manager.load_extension("google.cloud.bigquery")
+ magics.context._project = None
+
+ credentials_mock = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+ default_patch = mock.patch(
+ "google.auth.default", return_value=(credentials_mock, "general-project")
+ )
+ client_query_patch = mock.patch("google.cloud.bigquery.client.Client.query")
+
+ sql = "SELECT 17 AS num"
+
+ with pytest.raises(ValueError), default_patch, client_query_patch:
+ ip.run_cell_magic("bigquery", "--maximum_bytes_billed=abc", sql)
+
+
+@pytest.mark.parametrize(
+ "param_value,expected", [("987654321", "987654321"), ("None", "0")]
+)
+@pytest.mark.usefixtures("ipython_interactive")
+def test_bigquery_magic_w_maximum_bytes_billed_overrides_context(param_value, expected):
+ ip = IPython.get_ipython()
+ ip.extension_manager.load_extension("google.cloud.bigquery")
+ magics.context._project = None
+
+ # Set the default maximum bytes billed, so we know it's overridable by the param.
+ magics.context.default_query_job_config.maximum_bytes_billed = 1234567
+
+ project = "test-project"
+ job_reference = copy.deepcopy(JOB_REFERENCE_RESOURCE)
+ job_reference["projectId"] = project
+ query = "SELECT 17 AS num"
+ resource = copy.deepcopy(QUERY_RESOURCE)
+ resource["jobReference"] = job_reference
+ resource["configuration"]["query"]["query"] = query
+ data = {"jobReference": job_reference, "totalRows": 0, "rows": []}
+ credentials_mock = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+ default_patch = mock.patch(
+ "google.auth.default", return_value=(credentials_mock, "general-project")
+ )
+ conn = magics.context._connection = make_connection(resource, data)
+ list_rows_patch = mock.patch(
+ "google.cloud.bigquery.client.Client.list_rows",
+ return_value=google.cloud.bigquery.table._EmptyRowIterator(),
+ )
+ with list_rows_patch, default_patch:
+ ip.run_cell_magic(
+ "bigquery", "--maximum_bytes_billed={}".format(param_value), query
+ )
+
+ _, req = conn.api_request.call_args_list[0]
+ sent_config = req["data"]["configuration"]["query"]
+ assert sent_config["maximumBytesBilled"] == expected
+
+
+@pytest.mark.usefixtures("ipython_interactive")
+def test_bigquery_magic_w_maximum_bytes_billed_w_context_inplace():
+ ip = IPython.get_ipython()
+ ip.extension_manager.load_extension("google.cloud.bigquery")
+ magics.context._project = None
+
+ magics.context.default_query_job_config.maximum_bytes_billed = 1337
+
+ project = "test-project"
+ job_reference = copy.deepcopy(JOB_REFERENCE_RESOURCE)
+ job_reference["projectId"] = project
+ query = "SELECT 17 AS num"
+ resource = copy.deepcopy(QUERY_RESOURCE)
+ resource["jobReference"] = job_reference
+ resource["configuration"]["query"]["query"] = query
+ data = {"jobReference": job_reference, "totalRows": 0, "rows": []}
+ credentials_mock = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+ default_patch = mock.patch(
+ "google.auth.default", return_value=(credentials_mock, "general-project")
+ )
+ conn = magics.context._connection = make_connection(resource, data)
+ list_rows_patch = mock.patch(
+ "google.cloud.bigquery.client.Client.list_rows",
+ return_value=google.cloud.bigquery.table._EmptyRowIterator(),
+ )
+ with list_rows_patch, default_patch:
+ ip.run_cell_magic("bigquery", "", query)
+
+ _, req = conn.api_request.call_args_list[0]
+ sent_config = req["data"]["configuration"]["query"]
+ assert sent_config["maximumBytesBilled"] == "1337"
+
+
+@pytest.mark.usefixtures("ipython_interactive")
+def test_bigquery_magic_w_maximum_bytes_billed_w_context_setter():
+ ip = IPython.get_ipython()
+ ip.extension_manager.load_extension("google.cloud.bigquery")
+ magics.context._project = None
+
+ magics.context.default_query_job_config = job.QueryJobConfig(
+ maximum_bytes_billed=10203
+ )
+
+ project = "test-project"
+ job_reference = copy.deepcopy(JOB_REFERENCE_RESOURCE)
+ job_reference["projectId"] = project
+ query = "SELECT 17 AS num"
+ resource = copy.deepcopy(QUERY_RESOURCE)
+ resource["jobReference"] = job_reference
+ resource["configuration"]["query"]["query"] = query
+ data = {"jobReference": job_reference, "totalRows": 0, "rows": []}
+ credentials_mock = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+ default_patch = mock.patch(
+ "google.auth.default", return_value=(credentials_mock, "general-project")
+ )
+ conn = magics.context._connection = make_connection(resource, data)
+ list_rows_patch = mock.patch(
+ "google.cloud.bigquery.client.Client.list_rows",
+ return_value=google.cloud.bigquery.table._EmptyRowIterator(),
+ )
+ with list_rows_patch, default_patch:
+ ip.run_cell_magic("bigquery", "", query)
+
+ _, req = conn.api_request.call_args_list[0]
+ sent_config = req["data"]["configuration"]["query"]
+ assert sent_config["maximumBytesBilled"] == "10203"
+
+
+@pytest.mark.usefixtures("ipython_interactive")
+def test_bigquery_magic_with_project():
+ ip = IPython.get_ipython()
+ ip.extension_manager.load_extension("google.cloud.bigquery")
+ magics.context._project = None
+
+ credentials_mock = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+ default_patch = mock.patch(
+ "google.auth.default", return_value=(credentials_mock, "general-project")
+ )
+ run_query_patch = mock.patch(
+ "google.cloud.bigquery.magics._run_query", autospec=True
+ )
+ with run_query_patch as run_query_mock, default_patch:
+ ip.run_cell_magic("bigquery", "--project=specific-project", "SELECT 17 as num")
+
+ client_used = run_query_mock.call_args_list[0][0][0]
+ assert client_used.project == "specific-project"
+ # context project should not change
+ assert magics.context.project == "general-project"
+
+
+@pytest.mark.usefixtures("ipython_interactive")
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+def test_bigquery_magic_with_string_params():
+ ip = IPython.get_ipython()
+ ip.extension_manager.load_extension("google.cloud.bigquery")
+ magics.context.credentials = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+
+ sql = "SELECT @num AS num"
+ result = pandas.DataFrame([17], columns=["num"])
+ assert "params_string_df" not in ip.user_ns
+
+ run_query_patch = mock.patch(
+ "google.cloud.bigquery.magics._run_query", autospec=True
+ )
+ query_job_mock = mock.create_autospec(
+ google.cloud.bigquery.job.QueryJob, instance=True
+ )
+ query_job_mock.to_dataframe.return_value = result
+ with run_query_patch as run_query_mock:
+ run_query_mock.return_value = query_job_mock
+
+ ip.run_cell_magic("bigquery", 'params_string_df --params {"num":17}', sql)
+ run_query_mock.assert_called_once_with(mock.ANY, sql.format(num=17), mock.ANY)
+
+ assert "params_string_df" in ip.user_ns # verify that the variable exists
+ df = ip.user_ns["params_string_df"]
+ assert len(df) == len(result) # verify row count
+ assert list(df) == list(result) # verify column names
+
+
+@pytest.mark.usefixtures("ipython_interactive")
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+def test_bigquery_magic_with_dict_params():
+ ip = IPython.get_ipython()
+ ip.extension_manager.load_extension("google.cloud.bigquery")
+ magics.context.credentials = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+
+ sql = "SELECT @num AS num"
+ result = pandas.DataFrame([17], columns=["num"])
+ assert "params_dict_df" not in ip.user_ns
+
+ run_query_patch = mock.patch(
+ "google.cloud.bigquery.magics._run_query", autospec=True
+ )
+ query_job_mock = mock.create_autospec(
+ google.cloud.bigquery.job.QueryJob, instance=True
+ )
+ query_job_mock.to_dataframe.return_value = result
+ with run_query_patch as run_query_mock:
+ run_query_mock.return_value = query_job_mock
+
+ params = {"num": 17}
+ # Insert dictionary into user namespace so that it can be expanded
+ ip.user_ns["params"] = params
+ ip.run_cell_magic("bigquery", "params_dict_df --params $params", sql)
+ run_query_mock.assert_called_once_with(mock.ANY, sql.format(num=17), mock.ANY)
+
+ assert "params_dict_df" in ip.user_ns # verify that the variable exists
+ df = ip.user_ns["params_dict_df"]
+ assert len(df) == len(result) # verify row count
+ assert list(df) == list(result) # verify column names
+
+
+@pytest.mark.usefixtures("ipython_interactive")
+@pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
+def test_bigquery_magic_with_improperly_formatted_params():
+ ip = IPython.get_ipython()
+ ip.extension_manager.load_extension("google.cloud.bigquery")
+ magics.context.credentials = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+
+ sql = "SELECT @num AS num"
+
+ with pytest.raises(SyntaxError):
+ ip.run_cell_magic("bigquery", "--params {17}", sql)
+
+
+@pytest.mark.usefixtures("ipython_interactive")
+def test_bigquery_magic_omits_tracebacks_from_error_message():
+ ip = IPython.get_ipython()
+ ip.extension_manager.load_extension("google.cloud.bigquery")
+
+ credentials_mock = mock.create_autospec(
+ google.auth.credentials.Credentials, instance=True
+ )
+ default_patch = mock.patch(
+ "google.auth.default", return_value=(credentials_mock, "general-project")
+ )
+
+ run_query_patch = mock.patch(
+ "google.cloud.bigquery.magics._run_query",
+ autospec=True,
+ side_effect=exceptions.BadRequest("Syntax error in SQL query"),
+ )
+
+ with run_query_patch, default_patch, io.capture_output() as captured_io:
+ ip.run_cell_magic("bigquery", "", "SELECT foo FROM WHERE LIMIT bar")
+
+ output = captured_io.stderr
+ assert "400 Syntax error in SQL query" in output
+ assert "Traceback (most recent call last)" not in output
+ assert "Syntax error" not in captured_io.stdout
diff --git a/bigquery/tests/unit/test_query.py b/bigquery/tests/unit/test_query.py
new file mode 100644
index 000000000000..a7c639ed1e77
--- /dev/null
+++ b/bigquery/tests/unit/test_query.py
@@ -0,0 +1,1111 @@
+# Copyright 2015 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import datetime
+import unittest
+
+import mock
+
+
+class Test_UDFResource(unittest.TestCase):
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery.query import UDFResource
+
+ return UDFResource
+
+ def _make_one(self, *args, **kw):
+ return self._get_target_class()(*args, **kw)
+
+ def test_ctor(self):
+ udf = self._make_one("resourceUri", "gs://some_bucket/some_file")
+ self.assertEqual(udf.udf_type, "resourceUri")
+ self.assertEqual(udf.value, "gs://some_bucket/some_file")
+
+ def test___eq__(self):
+ udf = self._make_one("resourceUri", "gs://some_bucket/some_file")
+ self.assertEqual(udf, udf)
+ self.assertNotEqual(udf, object())
+ wrong_val = self._make_one("resourceUri", "gs://some_bucket/other_file")
+ self.assertNotEqual(udf, wrong_val)
+ wrong_type = self._make_one("inlineCode", udf.value)
+ self.assertNotEqual(udf, wrong_type)
+
+
+class Test__AbstractQueryParameter(unittest.TestCase):
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery.query import _AbstractQueryParameter
+
+ return _AbstractQueryParameter
+
+ def _make_one(self, *args, **kw):
+ return self._get_target_class()(*args, **kw)
+
+ def test_from_api_virtual(self):
+ klass = self._get_target_class()
+ with self.assertRaises(NotImplementedError):
+ klass.from_api_repr({})
+
+ def test_to_api_virtual(self):
+ param = self._make_one()
+ with self.assertRaises(NotImplementedError):
+ param.to_api_repr()
+
+
+class Test_ScalarQueryParameter(unittest.TestCase):
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery.query import ScalarQueryParameter
+
+ return ScalarQueryParameter
+
+ def _make_one(self, *args, **kw):
+ return self._get_target_class()(*args, **kw)
+
+ def test_ctor(self):
+ param = self._make_one(name="foo", type_="INT64", value=123)
+ self.assertEqual(param.name, "foo")
+ self.assertEqual(param.type_, "INT64")
+ self.assertEqual(param.value, 123)
+
+ def test___eq__(self):
+ param = self._make_one(name="foo", type_="INT64", value=123)
+ self.assertEqual(param, param)
+ self.assertNotEqual(param, object())
+ alias = self._make_one(name="bar", type_="INT64", value=123)
+ self.assertNotEqual(param, alias)
+ wrong_type = self._make_one(name="foo", type_="FLOAT64", value=123.0)
+ self.assertNotEqual(param, wrong_type)
+ wrong_val = self._make_one(name="foo", type_="INT64", value=234)
+ self.assertNotEqual(param, wrong_val)
+
+ def test_positional(self):
+ klass = self._get_target_class()
+ param = klass.positional(type_="INT64", value=123)
+ self.assertEqual(param.name, None)
+ self.assertEqual(param.type_, "INT64")
+ self.assertEqual(param.value, 123)
+
+ def test_from_api_repr_w_name(self):
+ RESOURCE = {
+ "name": "foo",
+ "parameterType": {"type": "INT64"},
+ "parameterValue": {"value": 123},
+ }
+ klass = self._get_target_class()
+ param = klass.from_api_repr(RESOURCE)
+ self.assertEqual(param.name, "foo")
+ self.assertEqual(param.type_, "INT64")
+ self.assertEqual(param.value, 123)
+
+ def test_from_api_repr_wo_name(self):
+ RESOURCE = {
+ "parameterType": {"type": "INT64"},
+ "parameterValue": {"value": "123"},
+ }
+ klass = self._get_target_class()
+ param = klass.from_api_repr(RESOURCE)
+ self.assertEqual(param.name, None)
+ self.assertEqual(param.type_, "INT64")
+ self.assertEqual(param.value, 123)
+
+ def test_from_api_repr_wo_value(self):
+ # Back-end may not send back values for None params. See #9027
+ RESOURCE = {"name": "foo", "parameterType": {"type": "INT64"}}
+ klass = self._get_target_class()
+ param = klass.from_api_repr(RESOURCE)
+ self.assertEqual(param.name, "foo")
+ self.assertEqual(param.type_, "INT64")
+ self.assertIs(param.value, None)
+
+ def test_to_api_repr_w_name(self):
+ EXPECTED = {
+ "name": "foo",
+ "parameterType": {"type": "INT64"},
+ "parameterValue": {"value": "123"},
+ }
+ param = self._make_one(name="foo", type_="INT64", value=123)
+ self.assertEqual(param.to_api_repr(), EXPECTED)
+
+ def test_to_api_repr_wo_name(self):
+ EXPECTED = {
+ "parameterType": {"type": "INT64"},
+ "parameterValue": {"value": "123"},
+ }
+ klass = self._get_target_class()
+ param = klass.positional(type_="INT64", value=123)
+ self.assertEqual(param.to_api_repr(), EXPECTED)
+
+ def test_to_api_repr_w_float(self):
+ EXPECTED = {
+ "parameterType": {"type": "FLOAT64"},
+ "parameterValue": {"value": 12.345},
+ }
+ klass = self._get_target_class()
+ param = klass.positional(type_="FLOAT64", value=12.345)
+ self.assertEqual(param.to_api_repr(), EXPECTED)
+
+ def test_to_api_repr_w_numeric(self):
+ EXPECTED = {
+ "parameterType": {"type": "NUMERIC"},
+ "parameterValue": {"value": "123456789.123456789"},
+ }
+ klass = self._get_target_class()
+ param = klass.positional(type_="NUMERIC", value="123456789.123456789")
+ self.assertEqual(param.to_api_repr(), EXPECTED)
+
+ def test_to_api_repr_w_bool(self):
+ EXPECTED = {
+ "parameterType": {"type": "BOOL"},
+ "parameterValue": {"value": "false"},
+ }
+ klass = self._get_target_class()
+ param = klass.positional(type_="BOOL", value=False)
+ self.assertEqual(param.to_api_repr(), EXPECTED)
+
+ def test_to_api_repr_w_timestamp_datetime(self):
+ from google.cloud._helpers import UTC
+
+ STAMP = "2016-12-20 15:58:27.339328+00:00"
+ when = datetime.datetime(2016, 12, 20, 15, 58, 27, 339328, tzinfo=UTC)
+ EXPECTED = {
+ "parameterType": {"type": "TIMESTAMP"},
+ "parameterValue": {"value": STAMP},
+ }
+ klass = self._get_target_class()
+ param = klass.positional(type_="TIMESTAMP", value=when)
+ self.assertEqual(param.to_api_repr(), EXPECTED)
+
+ def test_to_api_repr_w_timestamp_micros(self):
+ from google.cloud._helpers import _microseconds_from_datetime
+
+ now = datetime.datetime.utcnow()
+ seconds = _microseconds_from_datetime(now) / 1.0e6
+ EXPECTED = {
+ "parameterType": {"type": "TIMESTAMP"},
+ "parameterValue": {"value": seconds},
+ }
+ klass = self._get_target_class()
+ param = klass.positional(type_="TIMESTAMP", value=seconds)
+ self.assertEqual(param.to_api_repr(), EXPECTED)
+
+ def test_to_api_repr_w_datetime_datetime(self):
+ from google.cloud._helpers import _datetime_to_rfc3339
+
+ now = datetime.datetime.utcnow()
+ EXPECTED = {
+ "parameterType": {"type": "DATETIME"},
+ "parameterValue": {
+ "value": _datetime_to_rfc3339(now)[:-1] # strip trailing 'Z'
+ },
+ }
+ klass = self._get_target_class()
+ param = klass.positional(type_="DATETIME", value=now)
+ self.assertEqual(param.to_api_repr(), EXPECTED)
+
+ def test_to_api_repr_w_datetime_string(self):
+ from google.cloud._helpers import _datetime_to_rfc3339
+
+ now = datetime.datetime.utcnow()
+ now_str = _datetime_to_rfc3339(now)
+ EXPECTED = {
+ "parameterType": {"type": "DATETIME"},
+ "parameterValue": {"value": now_str},
+ }
+ klass = self._get_target_class()
+ param = klass.positional(type_="DATETIME", value=now_str)
+ self.assertEqual(param.to_api_repr(), EXPECTED)
+
+ def test_to_api_repr_w_date_date(self):
+ today = datetime.date.today()
+ EXPECTED = {
+ "parameterType": {"type": "DATE"},
+ "parameterValue": {"value": today.isoformat()},
+ }
+ klass = self._get_target_class()
+ param = klass.positional(type_="DATE", value=today)
+ self.assertEqual(param.to_api_repr(), EXPECTED)
+
+ def test_to_api_repr_w_date_string(self):
+ today = datetime.date.today()
+ today_str = (today.isoformat(),)
+ EXPECTED = {
+ "parameterType": {"type": "DATE"},
+ "parameterValue": {"value": today_str},
+ }
+ klass = self._get_target_class()
+ param = klass.positional(type_="DATE", value=today_str)
+ self.assertEqual(param.to_api_repr(), EXPECTED)
+
+ def test_to_api_repr_w_unknown_type(self):
+ EXPECTED = {
+ "parameterType": {"type": "UNKNOWN"},
+ "parameterValue": {"value": "unknown"},
+ }
+ klass = self._get_target_class()
+ param = klass.positional(type_="UNKNOWN", value="unknown")
+ self.assertEqual(param.to_api_repr(), EXPECTED)
+
+ def test___eq___wrong_type(self):
+ field = self._make_one("test", "STRING", "value")
+ other = object()
+ self.assertNotEqual(field, other)
+ self.assertEqual(field, mock.ANY)
+
+ def test___eq___name_mismatch(self):
+ field = self._make_one("test", "STRING", "value")
+ other = self._make_one("other", "STRING", "value")
+ self.assertNotEqual(field, other)
+
+ def test___eq___field_type_mismatch(self):
+ field = self._make_one("test", "STRING", None)
+ other = self._make_one("test", "INT64", None)
+ self.assertNotEqual(field, other)
+
+ def test___eq___value_mismatch(self):
+ field = self._make_one("test", "STRING", "hello")
+ other = self._make_one("test", "STRING", "world")
+ self.assertNotEqual(field, other)
+
+ def test___eq___hit(self):
+ field = self._make_one("test", "STRING", "gotcha")
+ other = self._make_one("test", "STRING", "gotcha")
+ self.assertEqual(field, other)
+
+ def test___ne___wrong_type(self):
+ field = self._make_one("toast", "INT64", 13)
+ other = object()
+ self.assertNotEqual(field, other)
+ self.assertEqual(field, mock.ANY)
+
+ def test___ne___same_value(self):
+ field1 = self._make_one("test", "INT64", 12)
+ field2 = self._make_one("test", "INT64", 12)
+ # unittest ``assertEqual`` uses ``==`` not ``!=``.
+ comparison_val = field1 != field2
+ self.assertFalse(comparison_val)
+
+ def test___ne___different_values(self):
+ field1 = self._make_one("test", "INT64", 11)
+ field2 = self._make_one("test", "INT64", 12)
+ self.assertNotEqual(field1, field2)
+
+ def test___repr__(self):
+ field1 = self._make_one("field1", "STRING", "value")
+ expected = "ScalarQueryParameter('field1', 'STRING', 'value')"
+ self.assertEqual(repr(field1), expected)
+
+
+def _make_subparam(name, type_, value):
+ from google.cloud.bigquery.query import ScalarQueryParameter
+
+ return ScalarQueryParameter(name, type_, value)
+
+
+class Test_ArrayQueryParameter(unittest.TestCase):
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery.query import ArrayQueryParameter
+
+ return ArrayQueryParameter
+
+ def _make_one(self, *args, **kw):
+ return self._get_target_class()(*args, **kw)
+
+ def test_ctor(self):
+ param = self._make_one(name="foo", array_type="INT64", values=[1, 2])
+ self.assertEqual(param.name, "foo")
+ self.assertEqual(param.array_type, "INT64")
+ self.assertEqual(param.values, [1, 2])
+
+ def test___eq__(self):
+ param = self._make_one(name="foo", array_type="INT64", values=[123])
+ self.assertEqual(param, param)
+ self.assertNotEqual(param, object())
+ alias = self._make_one(name="bar", array_type="INT64", values=[123])
+ self.assertNotEqual(param, alias)
+ wrong_type = self._make_one(name="foo", array_type="FLOAT64", values=[123.0])
+ self.assertNotEqual(param, wrong_type)
+ wrong_val = self._make_one(name="foo", array_type="INT64", values=[234])
+ self.assertNotEqual(param, wrong_val)
+
+ def test_positional(self):
+ klass = self._get_target_class()
+ param = klass.positional(array_type="INT64", values=[1, 2])
+ self.assertEqual(param.name, None)
+ self.assertEqual(param.array_type, "INT64")
+ self.assertEqual(param.values, [1, 2])
+
+ def test_from_api_repr_w_name(self):
+ RESOURCE = {
+ "name": "foo",
+ "parameterType": {"type": "ARRAY", "arrayType": {"type": "INT64"}},
+ "parameterValue": {"arrayValues": [{"value": "1"}, {"value": "2"}]},
+ }
+ klass = self._get_target_class()
+ param = klass.from_api_repr(RESOURCE)
+ self.assertEqual(param.name, "foo")
+ self.assertEqual(param.array_type, "INT64")
+ self.assertEqual(param.values, [1, 2])
+
+ def test_from_api_repr_wo_name(self):
+ RESOURCE = {
+ "parameterType": {"type": "ARRAY", "arrayType": {"type": "INT64"}},
+ "parameterValue": {"arrayValues": [{"value": "1"}, {"value": "2"}]},
+ }
+ klass = self._get_target_class()
+ param = klass.from_api_repr(RESOURCE)
+ self.assertEqual(param.name, None)
+ self.assertEqual(param.array_type, "INT64")
+ self.assertEqual(param.values, [1, 2])
+
+ def test_from_api_repr_wo_values(self):
+ # Back-end may not send back values for empty array params. See #7309
+ RESOURCE = {
+ "name": "foo",
+ "parameterType": {"type": "ARRAY", "arrayType": {"type": "INT64"}},
+ }
+ klass = self._get_target_class()
+ param = klass.from_api_repr(RESOURCE)
+ self.assertEqual(param.name, "foo")
+ self.assertEqual(param.array_type, "INT64")
+ self.assertEqual(param.values, [])
+
+ def test_from_api_repr_w_struct_type(self):
+ from google.cloud.bigquery.query import StructQueryParameter
+
+ RESOURCE = {
+ "parameterType": {
+ "type": "ARRAY",
+ "arrayType": {
+ "type": "STRUCT",
+ "structTypes": [
+ {"name": "name", "type": {"type": "STRING"}},
+ {"name": "age", "type": {"type": "INT64"}},
+ ],
+ },
+ },
+ "parameterValue": {
+ "arrayValues": [
+ {
+ "structValues": {
+ "name": {"value": "Phred Phlyntstone"},
+ "age": {"value": "32"},
+ }
+ },
+ {
+ "structValues": {
+ "name": {"value": "Bharney Rhubbyl"},
+ "age": {"value": "31"},
+ }
+ },
+ ]
+ },
+ }
+
+ klass = self._get_target_class()
+ param = klass.from_api_repr(RESOURCE)
+
+ phred = StructQueryParameter.positional(
+ _make_subparam("name", "STRING", "Phred Phlyntstone"),
+ _make_subparam("age", "INT64", 32),
+ )
+ bharney = StructQueryParameter.positional(
+ _make_subparam("name", "STRING", "Bharney Rhubbyl"),
+ _make_subparam("age", "INT64", 31),
+ )
+ self.assertEqual(param.array_type, "STRUCT")
+ self.assertEqual(param.values, [phred, bharney])
+
+ def test_to_api_repr_w_name(self):
+ EXPECTED = {
+ "name": "foo",
+ "parameterType": {"type": "ARRAY", "arrayType": {"type": "INT64"}},
+ "parameterValue": {"arrayValues": [{"value": "1"}, {"value": "2"}]},
+ }
+ param = self._make_one(name="foo", array_type="INT64", values=[1, 2])
+ self.assertEqual(param.to_api_repr(), EXPECTED)
+
+ def test_to_api_repr_wo_name(self):
+ EXPECTED = {
+ "parameterType": {"type": "ARRAY", "arrayType": {"type": "INT64"}},
+ "parameterValue": {"arrayValues": [{"value": "1"}, {"value": "2"}]},
+ }
+ klass = self._get_target_class()
+ param = klass.positional(array_type="INT64", values=[1, 2])
+ self.assertEqual(param.to_api_repr(), EXPECTED)
+
+ def test_to_api_repr_w_unknown_type(self):
+ EXPECTED = {
+ "parameterType": {"type": "ARRAY", "arrayType": {"type": "UNKNOWN"}},
+ "parameterValue": {"arrayValues": [{"value": "unknown"}]},
+ }
+ klass = self._get_target_class()
+ param = klass.positional(array_type="UNKNOWN", values=["unknown"])
+ self.assertEqual(param.to_api_repr(), EXPECTED)
+
+ def test_to_api_repr_w_record_type(self):
+ from google.cloud.bigquery.query import StructQueryParameter
+
+ EXPECTED = {
+ "parameterType": {
+ "type": "ARRAY",
+ "arrayType": {
+ "type": "STRUCT",
+ "structTypes": [
+ {"name": "foo", "type": {"type": "STRING"}},
+ {"name": "bar", "type": {"type": "INT64"}},
+ ],
+ },
+ },
+ "parameterValue": {
+ "arrayValues": [
+ {"structValues": {"foo": {"value": "Foo"}, "bar": {"value": "123"}}}
+ ]
+ },
+ }
+ one = _make_subparam("foo", "STRING", "Foo")
+ another = _make_subparam("bar", "INT64", 123)
+ struct = StructQueryParameter.positional(one, another)
+ klass = self._get_target_class()
+ param = klass.positional(array_type="RECORD", values=[struct])
+ self.assertEqual(param.to_api_repr(), EXPECTED)
+
+ def test___eq___wrong_type(self):
+ field = self._make_one("test", "STRING", ["value"])
+ other = object()
+ self.assertNotEqual(field, other)
+ self.assertEqual(field, mock.ANY)
+
+ def test___eq___name_mismatch(self):
+ field = self._make_one("field", "STRING", ["value"])
+ other = self._make_one("other", "STRING", ["value"])
+ self.assertNotEqual(field, other)
+
+ def test___eq___field_type_mismatch(self):
+ field = self._make_one("test", "STRING", [])
+ other = self._make_one("test", "INT64", [])
+ self.assertNotEqual(field, other)
+
+ def test___eq___value_mismatch(self):
+ field = self._make_one("test", "STRING", ["hello"])
+ other = self._make_one("test", "STRING", ["hello", "world"])
+ self.assertNotEqual(field, other)
+
+ def test___eq___hit(self):
+ field = self._make_one("test", "STRING", ["gotcha"])
+ other = self._make_one("test", "STRING", ["gotcha"])
+ self.assertEqual(field, other)
+
+ def test___ne___wrong_type(self):
+ field = self._make_one("toast", "INT64", [13])
+ other = object()
+ self.assertNotEqual(field, other)
+ self.assertEqual(field, mock.ANY)
+
+ def test___ne___same_value(self):
+ field1 = self._make_one("test", "INT64", [12])
+ field2 = self._make_one("test", "INT64", [12])
+ # unittest ``assertEqual`` uses ``==`` not ``!=``.
+ comparison_val = field1 != field2
+ self.assertFalse(comparison_val)
+
+ def test___ne___different_values(self):
+ field1 = self._make_one("test", "INT64", [11])
+ field2 = self._make_one("test", "INT64", [12])
+ self.assertNotEqual(field1, field2)
+
+ def test___repr__(self):
+ field1 = self._make_one("field1", "STRING", ["value"])
+ expected = "ArrayQueryParameter('field1', 'STRING', ['value'])"
+ self.assertEqual(repr(field1), expected)
+
+
+class Test_StructQueryParameter(unittest.TestCase):
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery.query import StructQueryParameter
+
+ return StructQueryParameter
+
+ def _make_one(self, *args, **kw):
+ return self._get_target_class()(*args, **kw)
+
+ def test_ctor(self):
+ sub_1 = _make_subparam("bar", "INT64", 123)
+ sub_2 = _make_subparam("baz", "STRING", "abc")
+ param = self._make_one("foo", sub_1, sub_2)
+ self.assertEqual(param.name, "foo")
+ self.assertEqual(param.struct_types, {"bar": "INT64", "baz": "STRING"})
+ self.assertEqual(param.struct_values, {"bar": 123, "baz": "abc"})
+
+ def test___eq__(self):
+ sub_1 = _make_subparam("bar", "INT64", 123)
+ sub_2 = _make_subparam("baz", "STRING", "abc")
+ sub_3 = _make_subparam("baz", "STRING", "def")
+ sub_1_float = _make_subparam("bar", "FLOAT64", 123.0)
+ param = self._make_one("foo", sub_1, sub_2)
+ self.assertEqual(param, param)
+ self.assertNotEqual(param, object())
+ alias = self._make_one("bar", sub_1, sub_2)
+ self.assertNotEqual(param, alias)
+ wrong_type = self._make_one("foo", sub_1_float, sub_2)
+ self.assertNotEqual(param, wrong_type)
+ wrong_val = self._make_one("foo", sub_2, sub_3)
+ self.assertNotEqual(param, wrong_val)
+
+ def test_positional(self):
+ sub_1 = _make_subparam("bar", "INT64", 123)
+ sub_2 = _make_subparam("baz", "STRING", "abc")
+ klass = self._get_target_class()
+ param = klass.positional(sub_1, sub_2)
+ self.assertEqual(param.name, None)
+ self.assertEqual(param.struct_types, {"bar": "INT64", "baz": "STRING"})
+ self.assertEqual(param.struct_values, {"bar": 123, "baz": "abc"})
+
+ def test_from_api_repr_w_name(self):
+ RESOURCE = {
+ "name": "foo",
+ "parameterType": {
+ "type": "STRUCT",
+ "structTypes": [
+ {"name": "bar", "type": {"type": "INT64"}},
+ {"name": "baz", "type": {"type": "STRING"}},
+ ],
+ },
+ "parameterValue": {
+ "structValues": {"bar": {"value": 123}, "baz": {"value": "abc"}}
+ },
+ }
+ klass = self._get_target_class()
+ param = klass.from_api_repr(RESOURCE)
+ self.assertEqual(param.name, "foo")
+ self.assertEqual(param.struct_types, {"bar": "INT64", "baz": "STRING"})
+ self.assertEqual(param.struct_values, {"bar": 123, "baz": "abc"})
+
+ def test_from_api_repr_wo_name(self):
+ RESOURCE = {
+ "parameterType": {
+ "type": "STRUCT",
+ "structTypes": [
+ {"name": "bar", "type": {"type": "INT64"}},
+ {"name": "baz", "type": {"type": "STRING"}},
+ ],
+ },
+ "parameterValue": {
+ "structValues": {"bar": {"value": 123}, "baz": {"value": "abc"}}
+ },
+ }
+ klass = self._get_target_class()
+ param = klass.from_api_repr(RESOURCE)
+ self.assertEqual(param.name, None)
+ self.assertEqual(param.struct_types, {"bar": "INT64", "baz": "STRING"})
+ self.assertEqual(param.struct_values, {"bar": 123, "baz": "abc"})
+
+ def test_from_api_repr_w_nested_array(self):
+ from google.cloud.bigquery.query import ArrayQueryParameter
+
+ RESOURCE = {
+ "name": "foo",
+ "parameterType": {
+ "type": "STRUCT",
+ "structTypes": [
+ {"name": "bar", "type": {"type": "STRING"}},
+ {
+ "name": "baz",
+ "type": {"type": "ARRAY", "arrayType": {"type": "INT64"}},
+ },
+ ],
+ },
+ "parameterValue": {
+ "structValues": {
+ "bar": {"value": "abc"},
+ "baz": {"arrayValues": [{"value": "123"}, {"value": "456"}]},
+ }
+ },
+ }
+ klass = self._get_target_class()
+ param = klass.from_api_repr(RESOURCE)
+ self.assertEqual(
+ param,
+ self._make_one(
+ "foo",
+ _make_subparam("bar", "STRING", "abc"),
+ ArrayQueryParameter("baz", "INT64", [123, 456]),
+ ),
+ )
+
+ def test_from_api_repr_w_nested_struct(self):
+ RESOURCE = {
+ "name": "foo",
+ "parameterType": {
+ "type": "STRUCT",
+ "structTypes": [
+ {"name": "bar", "type": {"type": "STRING"}},
+ {
+ "name": "baz",
+ "type": {
+ "type": "STRUCT",
+ "structTypes": [
+ {"name": "qux", "type": {"type": "INT64"}},
+ {"name": "spam", "type": {"type": "BOOL"}},
+ ],
+ },
+ },
+ ],
+ },
+ "parameterValue": {
+ "structValues": {
+ "bar": {"value": "abc"},
+ "baz": {
+ "structValues": {
+ "qux": {"value": "123"},
+ "spam": {"value": "true"},
+ }
+ },
+ }
+ },
+ }
+
+ klass = self._get_target_class()
+ param = klass.from_api_repr(RESOURCE)
+
+ expected = self._make_one(
+ "foo",
+ _make_subparam("bar", "STRING", "abc"),
+ self._make_one(
+ "baz",
+ _make_subparam("qux", "INT64", 123),
+ _make_subparam("spam", "BOOL", True),
+ ),
+ )
+ self.assertEqual(param.name, "foo")
+ self.assertEqual(param.struct_types, expected.struct_types)
+ self.assertEqual(param.struct_values, expected.struct_values)
+
+ def test_to_api_repr_w_name(self):
+ EXPECTED = {
+ "name": "foo",
+ "parameterType": {
+ "type": "STRUCT",
+ "structTypes": [
+ {"name": "bar", "type": {"type": "INT64"}},
+ {"name": "baz", "type": {"type": "STRING"}},
+ ],
+ },
+ "parameterValue": {
+ "structValues": {"bar": {"value": "123"}, "baz": {"value": "abc"}}
+ },
+ }
+ sub_1 = _make_subparam("bar", "INT64", 123)
+ sub_2 = _make_subparam("baz", "STRING", "abc")
+ param = self._make_one("foo", sub_1, sub_2)
+ self.assertEqual(param.to_api_repr(), EXPECTED)
+
+ def test_to_api_repr_wo_name(self):
+ EXPECTED = {
+ "parameterType": {
+ "type": "STRUCT",
+ "structTypes": [
+ {"name": "bar", "type": {"type": "INT64"}},
+ {"name": "baz", "type": {"type": "STRING"}},
+ ],
+ },
+ "parameterValue": {
+ "structValues": {"bar": {"value": "123"}, "baz": {"value": "abc"}}
+ },
+ }
+ sub_1 = _make_subparam("bar", "INT64", 123)
+ sub_2 = _make_subparam("baz", "STRING", "abc")
+ klass = self._get_target_class()
+ param = klass.positional(sub_1, sub_2)
+ self.assertEqual(param.to_api_repr(), EXPECTED)
+
+ def test_to_api_repr_w_nested_array(self):
+ from google.cloud.bigquery.query import ArrayQueryParameter
+
+ EXPECTED = {
+ "name": "foo",
+ "parameterType": {
+ "type": "STRUCT",
+ "structTypes": [
+ {"name": "bar", "type": {"type": "STRING"}},
+ {
+ "name": "baz",
+ "type": {"type": "ARRAY", "arrayType": {"type": "INT64"}},
+ },
+ ],
+ },
+ "parameterValue": {
+ "structValues": {
+ "bar": {"value": "abc"},
+ "baz": {"arrayValues": [{"value": "123"}, {"value": "456"}]},
+ }
+ },
+ }
+ scalar = _make_subparam("bar", "STRING", "abc")
+ array = ArrayQueryParameter("baz", "INT64", [123, 456])
+ param = self._make_one("foo", scalar, array)
+ self.assertEqual(param.to_api_repr(), EXPECTED)
+
+ def test_to_api_repr_w_nested_struct(self):
+ EXPECTED = {
+ "name": "foo",
+ "parameterType": {
+ "type": "STRUCT",
+ "structTypes": [
+ {"name": "bar", "type": {"type": "STRING"}},
+ {
+ "name": "baz",
+ "type": {
+ "type": "STRUCT",
+ "structTypes": [
+ {"name": "qux", "type": {"type": "INT64"}},
+ {"name": "spam", "type": {"type": "BOOL"}},
+ ],
+ },
+ },
+ ],
+ },
+ "parameterValue": {
+ "structValues": {
+ "bar": {"value": "abc"},
+ "baz": {
+ "structValues": {
+ "qux": {"value": "123"},
+ "spam": {"value": "true"},
+ }
+ },
+ }
+ },
+ }
+ scalar_1 = _make_subparam("bar", "STRING", "abc")
+ scalar_2 = _make_subparam("qux", "INT64", 123)
+ scalar_3 = _make_subparam("spam", "BOOL", True)
+ sub = self._make_one("baz", scalar_2, scalar_3)
+ param = self._make_one("foo", scalar_1, sub)
+ self.assertEqual(param.to_api_repr(), EXPECTED)
+
+ def test___eq___wrong_type(self):
+ field = self._make_one("test", _make_subparam("bar", "STRING", "abc"))
+ other = object()
+ self.assertNotEqual(field, other)
+ self.assertEqual(field, mock.ANY)
+
+ def test___eq___name_mismatch(self):
+ field = self._make_one("test", _make_subparam("bar", "STRING", "abc"))
+ other = self._make_one("other ", _make_subparam("bar", "STRING", "abc"))
+ self.assertNotEqual(field, other)
+
+ def test___eq___field_type_mismatch(self):
+ field = self._make_one("test", _make_subparam("bar", "STRING", None))
+ other = self._make_one("test", _make_subparam("bar", "INT64", None))
+ self.assertNotEqual(field, other)
+
+ def test___eq___value_mismatch(self):
+ field = self._make_one("test", _make_subparam("bar", "STRING", "hello"))
+ other = self._make_one("test", _make_subparam("bar", "STRING", "world"))
+ self.assertNotEqual(field, other)
+
+ def test___eq___hit(self):
+ field = self._make_one("test", _make_subparam("bar", "STRING", "gotcha"))
+ other = self._make_one("test", _make_subparam("bar", "STRING", "gotcha"))
+ self.assertEqual(field, other)
+
+ def test___ne___wrong_type(self):
+ field = self._make_one("test", _make_subparam("bar", "STRING", "hello"))
+ other = object()
+ self.assertNotEqual(field, other)
+ self.assertEqual(field, mock.ANY)
+
+ def test___ne___same_value(self):
+ field1 = self._make_one("test", _make_subparam("bar", "STRING", "hello"))
+ field2 = self._make_one("test", _make_subparam("bar", "STRING", "hello"))
+ # unittest ``assertEqual`` uses ``==`` not ``!=``.
+ comparison_val = field1 != field2
+ self.assertFalse(comparison_val)
+
+ def test___ne___different_values(self):
+ field1 = self._make_one("test", _make_subparam("bar", "STRING", "hello"))
+ field2 = self._make_one("test", _make_subparam("bar", "STRING", "world"))
+ self.assertNotEqual(field1, field2)
+
+ def test___repr__(self):
+ field1 = self._make_one("test", _make_subparam("field1", "STRING", "hello"))
+ got = repr(field1)
+ self.assertIn("StructQueryParameter", got)
+ self.assertIn("'field1', 'STRING'", got)
+ self.assertIn("'field1': 'hello'", got)
+
+
+class Test_QueryResults(unittest.TestCase):
+ PROJECT = "project"
+ JOB_ID = "test-synchronous-query"
+ TOKEN = "TOKEN"
+
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery.query import _QueryResults
+
+ return _QueryResults
+
+ def _make_one(self, *args, **kw):
+ return self._get_target_class()(*args, **kw)
+
+ def _make_resource(self):
+ return {"jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID}}
+
+ def _verifySchema(self, query, resource):
+ from google.cloud.bigquery.schema import SchemaField
+
+ if "schema" in resource:
+ fields = resource["schema"]["fields"]
+ self.assertEqual(len(query.schema), len(fields))
+ for found, expected in zip(query.schema, fields):
+ self.assertIsInstance(found, SchemaField)
+ self.assertEqual(found.name, expected["name"])
+ self.assertEqual(found.field_type, expected["type"])
+ self.assertEqual(found.mode, expected["mode"])
+ self.assertEqual(found.description, expected.get("description"))
+ self.assertEqual(found.fields, expected.get("fields", ()))
+ else:
+ self.assertEqual(query.schema, ())
+
+ def test_ctor_defaults(self):
+ query = self._make_one(self._make_resource())
+ self.assertIsNone(query.cache_hit)
+ self.assertIsNone(query.complete)
+ self.assertIsNone(query.errors)
+ self.assertIsNone(query.page_token)
+ self.assertEqual(query.project, self.PROJECT)
+ self.assertEqual(query.rows, [])
+ self.assertEqual(query.schema, ())
+ self.assertIsNone(query.total_rows)
+ self.assertIsNone(query.total_bytes_processed)
+
+ def test_cache_hit_missing(self):
+ query = self._make_one(self._make_resource())
+ self.assertIsNone(query.cache_hit)
+
+ def test_cache_hit_present(self):
+ resource = self._make_resource()
+ resource["cacheHit"] = True
+ query = self._make_one(resource)
+ self.assertTrue(query.cache_hit)
+
+ def test_complete_missing(self):
+ query = self._make_one(self._make_resource())
+ self.assertIsNone(query.complete)
+
+ def test_complete_present(self):
+ resource = self._make_resource()
+ resource["jobComplete"] = True
+ query = self._make_one(resource)
+ self.assertTrue(query.complete)
+
+ def test_errors_missing(self):
+ query = self._make_one(self._make_resource())
+ self.assertIsNone(query.errors)
+
+ def test_errors_present(self):
+ ERRORS = [{"reason": "testing"}]
+ resource = self._make_resource()
+ resource["errors"] = ERRORS
+ query = self._make_one(resource)
+ self.assertEqual(query.errors, ERRORS)
+
+ def test_job_id_missing(self):
+ with self.assertRaises(ValueError):
+ self._make_one({})
+
+ def test_job_id_broken_job_reference(self):
+ resource = {"jobReference": {"bogus": "BOGUS"}}
+ with self.assertRaises(ValueError):
+ self._make_one(resource)
+
+ def test_job_id_present(self):
+ resource = self._make_resource()
+ resource["jobReference"]["jobId"] = "custom-job"
+ query = self._make_one(resource)
+ self.assertEqual(query.job_id, "custom-job")
+
+ def test_page_token_missing(self):
+ query = self._make_one(self._make_resource())
+ self.assertIsNone(query.page_token)
+
+ def test_page_token_present(self):
+ resource = self._make_resource()
+ resource["pageToken"] = "TOKEN"
+ query = self._make_one(resource)
+ self.assertEqual(query.page_token, "TOKEN")
+
+ def test_total_rows_present_integer(self):
+ resource = self._make_resource()
+ resource["totalRows"] = 42
+ query = self._make_one(resource)
+ self.assertEqual(query.total_rows, 42)
+
+ def test_total_rows_present_string(self):
+ resource = self._make_resource()
+ resource["totalRows"] = "42"
+ query = self._make_one(resource)
+ self.assertEqual(query.total_rows, 42)
+
+ def test_total_bytes_processed_missing(self):
+ query = self._make_one(self._make_resource())
+ self.assertIsNone(query.total_bytes_processed)
+
+ def test_total_bytes_processed_present_integer(self):
+ resource = self._make_resource()
+ resource["totalBytesProcessed"] = 123456
+ query = self._make_one(resource)
+ self.assertEqual(query.total_bytes_processed, 123456)
+
+ def test_total_bytes_processed_present_string(self):
+ resource = self._make_resource()
+ resource["totalBytesProcessed"] = "123456"
+ query = self._make_one(resource)
+ self.assertEqual(query.total_bytes_processed, 123456)
+
+ def test_num_dml_affected_rows_missing(self):
+ query = self._make_one(self._make_resource())
+ self.assertIsNone(query.num_dml_affected_rows)
+
+ def test_num_dml_affected_rows_present_integer(self):
+ resource = self._make_resource()
+ resource["numDmlAffectedRows"] = 123456
+ query = self._make_one(resource)
+ self.assertEqual(query.num_dml_affected_rows, 123456)
+
+ def test_num_dml_affected_rows_present_string(self):
+ resource = self._make_resource()
+ resource["numDmlAffectedRows"] = "123456"
+ query = self._make_one(resource)
+ self.assertEqual(query.num_dml_affected_rows, 123456)
+
+ def test_schema(self):
+ query = self._make_one(self._make_resource())
+ self._verifySchema(query, self._make_resource())
+ resource = self._make_resource()
+ resource["schema"] = {
+ "fields": [
+ {"name": "full_name", "type": "STRING", "mode": "REQURED"},
+ {"name": "age", "type": "INTEGER", "mode": "REQURED"},
+ ]
+ }
+ query._set_properties(resource)
+ self._verifySchema(query, resource)
+
+
+class Test__query_param_from_api_repr(unittest.TestCase):
+ @staticmethod
+ def _call_fut(resource):
+ from google.cloud.bigquery.query import _query_param_from_api_repr
+
+ return _query_param_from_api_repr(resource)
+
+ def test_w_scalar(self):
+ from google.cloud.bigquery.query import ScalarQueryParameter
+
+ RESOURCE = {
+ "name": "foo",
+ "parameterType": {"type": "INT64"},
+ "parameterValue": {"value": "123"},
+ }
+
+ parameter = self._call_fut(RESOURCE)
+
+ self.assertIsInstance(parameter, ScalarQueryParameter)
+ self.assertEqual(parameter.name, "foo")
+ self.assertEqual(parameter.type_, "INT64")
+ self.assertEqual(parameter.value, 123)
+
+ def test_w_scalar_timestamp(self):
+ from google.cloud._helpers import UTC
+ from google.cloud.bigquery.query import ScalarQueryParameter
+
+ RESOURCE = {
+ "name": "zoned",
+ "parameterType": {"type": "TIMESTAMP"},
+ "parameterValue": {"value": "2012-03-04 05:06:07+00:00"},
+ }
+
+ parameter = self._call_fut(RESOURCE)
+
+ self.assertIsInstance(parameter, ScalarQueryParameter)
+ self.assertEqual(parameter.name, "zoned")
+ self.assertEqual(parameter.type_, "TIMESTAMP")
+ self.assertEqual(
+ parameter.value, datetime.datetime(2012, 3, 4, 5, 6, 7, tzinfo=UTC)
+ )
+
+ def test_w_scalar_timestamp_micros(self):
+ from google.cloud._helpers import UTC
+ from google.cloud.bigquery.query import ScalarQueryParameter
+
+ RESOURCE = {
+ "name": "zoned",
+ "parameterType": {"type": "TIMESTAMP"},
+ "parameterValue": {"value": "2012-03-04 05:06:07.250000+00:00"},
+ }
+
+ parameter = self._call_fut(RESOURCE)
+
+ self.assertIsInstance(parameter, ScalarQueryParameter)
+ self.assertEqual(parameter.name, "zoned")
+ self.assertEqual(parameter.type_, "TIMESTAMP")
+ self.assertEqual(
+ parameter.value, datetime.datetime(2012, 3, 4, 5, 6, 7, 250000, tzinfo=UTC)
+ )
+
+ def test_w_array(self):
+ from google.cloud.bigquery.query import ArrayQueryParameter
+
+ RESOURCE = {
+ "name": "foo",
+ "parameterType": {"type": "ARRAY", "arrayType": {"type": "INT64"}},
+ "parameterValue": {"arrayValues": [{"value": "123"}]},
+ }
+
+ parameter = self._call_fut(RESOURCE)
+
+ self.assertIsInstance(parameter, ArrayQueryParameter)
+ self.assertEqual(parameter.name, "foo")
+ self.assertEqual(parameter.array_type, "INT64")
+ self.assertEqual(parameter.values, [123])
+
+ def test_w_struct(self):
+ from google.cloud.bigquery.query import StructQueryParameter
+
+ RESOURCE = {
+ "name": "foo",
+ "parameterType": {
+ "type": "STRUCT",
+ "structTypes": [
+ {"name": "foo", "type": {"type": "STRING"}},
+ {"name": "bar", "type": {"type": "INT64"}},
+ ],
+ },
+ "parameterValue": {
+ "structValues": {"foo": {"value": "Foo"}, "bar": {"value": "123"}}
+ },
+ }
+
+ parameter = self._call_fut(RESOURCE)
+
+ self.assertIsInstance(parameter, StructQueryParameter)
+ self.assertEqual(parameter.name, "foo")
+ self.assertEqual(parameter.struct_types, {"foo": "STRING", "bar": "INT64"})
+ self.assertEqual(parameter.struct_values, {"foo": "Foo", "bar": 123})
diff --git a/bigquery/tests/unit/test_retry.py b/bigquery/tests/unit/test_retry.py
new file mode 100644
index 000000000000..d9f867cb30f7
--- /dev/null
+++ b/bigquery/tests/unit/test_retry.py
@@ -0,0 +1,69 @@
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import unittest
+
+import mock
+
+
+class Test_should_retry(unittest.TestCase):
+ def _call_fut(self, exc):
+ from google.cloud.bigquery.retry import _should_retry
+
+ return _should_retry(exc)
+
+ def test_wo_errors_attribute(self):
+ self.assertFalse(self._call_fut(object()))
+
+ def test_w_empty_errors(self):
+ exc = mock.Mock(errors=[], spec=["errors"])
+ self.assertFalse(self._call_fut(exc))
+
+ def test_w_non_matching_reason(self):
+ exc = mock.Mock(errors=[{"reason": "bogus"}], spec=["errors"])
+ self.assertFalse(self._call_fut(exc))
+
+ def test_w_backendError(self):
+ exc = mock.Mock(errors=[{"reason": "backendError"}], spec=["errors"])
+ self.assertTrue(self._call_fut(exc))
+
+ def test_w_rateLimitExceeded(self):
+ exc = mock.Mock(errors=[{"reason": "rateLimitExceeded"}], spec=["errors"])
+ self.assertTrue(self._call_fut(exc))
+
+ def test_w_unstructured_too_many_requests(self):
+ from google.api_core.exceptions import TooManyRequests
+
+ exc = TooManyRequests("testing")
+ self.assertTrue(self._call_fut(exc))
+
+ def test_w_internalError(self):
+ exc = mock.Mock(errors=[{"reason": "internalError"}], spec=["errors"])
+ self.assertTrue(self._call_fut(exc))
+
+ def test_w_unstructured_internal_server_error(self):
+ from google.api_core.exceptions import InternalServerError
+
+ exc = InternalServerError("testing")
+ self.assertTrue(self._call_fut(exc))
+
+ def test_w_badGateway(self):
+ exc = mock.Mock(errors=[{"reason": "badGateway"}], spec=["errors"])
+ self.assertTrue(self._call_fut(exc))
+
+ def test_w_unstructured_bad_gateway(self):
+ from google.api_core.exceptions import BadGateway
+
+ exc = BadGateway("testing")
+ self.assertTrue(self._call_fut(exc))
diff --git a/bigquery/tests/unit/test_schema.py b/bigquery/tests/unit/test_schema.py
new file mode 100644
index 000000000000..862d8a823e62
--- /dev/null
+++ b/bigquery/tests/unit/test_schema.py
@@ -0,0 +1,570 @@
+# Copyright 2015 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import unittest
+
+import mock
+
+
+class TestSchemaField(unittest.TestCase):
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery.schema import SchemaField
+
+ return SchemaField
+
+ @staticmethod
+ def _get_standard_sql_data_type_class():
+ from google.cloud.bigquery_v2 import types
+
+ return types.StandardSqlDataType
+
+ def _make_one(self, *args, **kw):
+ return self._get_target_class()(*args, **kw)
+
+ def test_constructor_defaults(self):
+ field = self._make_one("test", "STRING")
+ self.assertEqual(field._name, "test")
+ self.assertEqual(field._field_type, "STRING")
+ self.assertEqual(field._mode, "NULLABLE")
+ self.assertIsNone(field._description)
+ self.assertEqual(field._fields, ())
+
+ def test_constructor_explicit(self):
+ field = self._make_one("test", "STRING", mode="REQUIRED", description="Testing")
+ self.assertEqual(field._name, "test")
+ self.assertEqual(field._field_type, "STRING")
+ self.assertEqual(field._mode, "REQUIRED")
+ self.assertEqual(field._description, "Testing")
+ self.assertEqual(field._fields, ())
+
+ def test_constructor_subfields(self):
+ sub_field1 = self._make_one("area_code", "STRING")
+ sub_field2 = self._make_one("local_number", "STRING")
+ field = self._make_one(
+ "phone_number", "RECORD", fields=[sub_field1, sub_field2]
+ )
+ self.assertEqual(field._name, "phone_number")
+ self.assertEqual(field._field_type, "RECORD")
+ self.assertEqual(field._mode, "NULLABLE")
+ self.assertIsNone(field._description)
+ self.assertEqual(len(field._fields), 2)
+ self.assertIs(field._fields[0], sub_field1)
+ self.assertIs(field._fields[1], sub_field2)
+
+ def test_to_api_repr(self):
+ field = self._make_one("foo", "INTEGER", "NULLABLE")
+ self.assertEqual(
+ field.to_api_repr(),
+ {"mode": "NULLABLE", "name": "foo", "type": "INTEGER", "description": None},
+ )
+
+ def test_to_api_repr_with_subfield(self):
+ for record_type in ("RECORD", "STRUCT"):
+ subfield = self._make_one("bar", "INTEGER", "NULLABLE")
+ field = self._make_one("foo", record_type, "REQUIRED", fields=(subfield,))
+ self.assertEqual(
+ field.to_api_repr(),
+ {
+ "fields": [
+ {
+ "mode": "NULLABLE",
+ "name": "bar",
+ "type": "INTEGER",
+ "description": None,
+ }
+ ],
+ "mode": "REQUIRED",
+ "name": "foo",
+ "type": record_type,
+ "description": None,
+ },
+ )
+
+ def test_from_api_repr(self):
+ field = self._get_target_class().from_api_repr(
+ {
+ "fields": [{"mode": "nullable", "name": "bar", "type": "integer"}],
+ "mode": "required",
+ "description": "test_description",
+ "name": "foo",
+ "type": "record",
+ }
+ )
+ self.assertEqual(field.name, "foo")
+ self.assertEqual(field.field_type, "RECORD")
+ self.assertEqual(field.mode, "REQUIRED")
+ self.assertEqual(field.description, "test_description")
+ self.assertEqual(len(field.fields), 1)
+ self.assertEqual(field.fields[0].name, "bar")
+ self.assertEqual(field.fields[0].field_type, "INTEGER")
+ self.assertEqual(field.fields[0].mode, "NULLABLE")
+
+ def test_from_api_repr_defaults(self):
+ field = self._get_target_class().from_api_repr(
+ {"name": "foo", "type": "record"}
+ )
+ self.assertEqual(field.name, "foo")
+ self.assertEqual(field.field_type, "RECORD")
+ self.assertEqual(field.mode, "NULLABLE")
+ self.assertEqual(field.description, None)
+ self.assertEqual(len(field.fields), 0)
+
+ def test_name_property(self):
+ name = "lemon-ness"
+ schema_field = self._make_one(name, "INTEGER")
+ self.assertIs(schema_field.name, name)
+
+ def test_field_type_property(self):
+ field_type = "BOOLEAN"
+ schema_field = self._make_one("whether", field_type)
+ self.assertIs(schema_field.field_type, field_type)
+
+ def test_mode_property(self):
+ mode = "REPEATED"
+ schema_field = self._make_one("again", "FLOAT", mode=mode)
+ self.assertIs(schema_field.mode, mode)
+
+ def test_is_nullable(self):
+ mode = "NULLABLE"
+ schema_field = self._make_one("test", "FLOAT", mode=mode)
+ self.assertTrue(schema_field.is_nullable)
+
+ def test_is_not_nullable(self):
+ mode = "REPEATED"
+ schema_field = self._make_one("test", "FLOAT", mode=mode)
+ self.assertFalse(schema_field.is_nullable)
+
+ def test_description_property(self):
+ description = "It holds some data."
+ schema_field = self._make_one("do", "TIMESTAMP", description=description)
+ self.assertIs(schema_field.description, description)
+
+ def test_fields_property(self):
+ sub_field1 = self._make_one("one", "STRING")
+ sub_field2 = self._make_one("fish", "INTEGER")
+ fields = (sub_field1, sub_field2)
+ schema_field = self._make_one("boat", "RECORD", fields=fields)
+ self.assertIs(schema_field.fields, fields)
+
+ def test_to_standard_sql_simple_type(self):
+ sql_type = self._get_standard_sql_data_type_class()
+ examples = (
+ # a few legacy types
+ ("INTEGER", sql_type.INT64),
+ ("FLOAT", sql_type.FLOAT64),
+ ("BOOLEAN", sql_type.BOOL),
+ ("DATETIME", sql_type.DATETIME),
+ # a few standard types
+ ("INT64", sql_type.INT64),
+ ("FLOAT64", sql_type.FLOAT64),
+ ("BOOL", sql_type.BOOL),
+ ("GEOGRAPHY", sql_type.GEOGRAPHY),
+ )
+ for legacy_type, standard_type in examples:
+ field = self._make_one("some_field", legacy_type)
+ standard_field = field.to_standard_sql()
+ self.assertEqual(standard_field.name, "some_field")
+ self.assertEqual(standard_field.type.type_kind, standard_type)
+ self.assertFalse(standard_field.type.HasField("sub_type"))
+
+ def test_to_standard_sql_struct_type(self):
+ from google.cloud.bigquery_v2 import types
+
+ # Expected result object:
+ #
+ # name: "image_usage"
+ # type {
+ # type_kind: STRUCT
+ # struct_type {
+ # fields {
+ # name: "image_content"
+ # type {type_kind: BYTES}
+ # }
+ # fields {
+ # name: "last_used"
+ # type {
+ # type_kind: STRUCT
+ # struct_type {
+ # fields {
+ # name: "date_field"
+ # type {type_kind: DATE}
+ # }
+ # fields {
+ # name: "time_field"
+ # type {type_kind: TIME}
+ # }
+ # }
+ # }
+ # }
+ # }
+ # }
+
+ sql_type = self._get_standard_sql_data_type_class()
+
+ # level 2 fields
+ sub_sub_field_date = types.StandardSqlField(
+ name="date_field", type=sql_type(type_kind=sql_type.DATE)
+ )
+ sub_sub_field_time = types.StandardSqlField(
+ name="time_field", type=sql_type(type_kind=sql_type.TIME)
+ )
+
+ # level 1 fields
+ sub_field_struct = types.StandardSqlField(
+ name="last_used", type=sql_type(type_kind=sql_type.STRUCT)
+ )
+ sub_field_struct.type.struct_type.fields.extend(
+ [sub_sub_field_date, sub_sub_field_time]
+ )
+ sub_field_bytes = types.StandardSqlField(
+ name="image_content", type=sql_type(type_kind=sql_type.BYTES)
+ )
+
+ # level 0 (top level)
+ expected_result = types.StandardSqlField(
+ name="image_usage", type=sql_type(type_kind=sql_type.STRUCT)
+ )
+ expected_result.type.struct_type.fields.extend(
+ [sub_field_bytes, sub_field_struct]
+ )
+
+ # construct legacy SchemaField object
+ sub_sub_field1 = self._make_one("date_field", "DATE")
+ sub_sub_field2 = self._make_one("time_field", "TIME")
+ sub_field_record = self._make_one(
+ "last_used", "RECORD", fields=(sub_sub_field1, sub_sub_field2)
+ )
+ sub_field_bytes = self._make_one("image_content", "BYTES")
+
+ for type_name in ("RECORD", "STRUCT"):
+ schema_field = self._make_one(
+ "image_usage", type_name, fields=(sub_field_bytes, sub_field_record)
+ )
+ standard_field = schema_field.to_standard_sql()
+ self.assertEqual(standard_field, expected_result)
+
+ def test_to_standard_sql_array_type_simple(self):
+ from google.cloud.bigquery_v2 import types
+
+ sql_type = self._get_standard_sql_data_type_class()
+
+ # construct expected result object
+ expected_sql_type = sql_type(type_kind=sql_type.ARRAY)
+ expected_sql_type.array_element_type.type_kind = sql_type.INT64
+ expected_result = types.StandardSqlField(
+ name="valid_numbers", type=expected_sql_type
+ )
+
+ # construct "repeated" SchemaField object and convert to standard SQL
+ schema_field = self._make_one("valid_numbers", "INT64", mode="REPEATED")
+ standard_field = schema_field.to_standard_sql()
+
+ self.assertEqual(standard_field, expected_result)
+
+ def test_to_standard_sql_array_type_struct(self):
+ from google.cloud.bigquery_v2 import types
+
+ sql_type = self._get_standard_sql_data_type_class()
+
+ # define person STRUCT
+ name_field = types.StandardSqlField(
+ name="name", type=sql_type(type_kind=sql_type.STRING)
+ )
+ age_field = types.StandardSqlField(
+ name="age", type=sql_type(type_kind=sql_type.INT64)
+ )
+ person_struct = types.StandardSqlField(
+ name="person_info", type=sql_type(type_kind=sql_type.STRUCT)
+ )
+ person_struct.type.struct_type.fields.extend([name_field, age_field])
+
+ # define expected result - an ARRAY of person structs
+ expected_sql_type = sql_type(
+ type_kind=sql_type.ARRAY, array_element_type=person_struct.type
+ )
+ expected_result = types.StandardSqlField(
+ name="known_people", type=expected_sql_type
+ )
+
+ # construct legacy repeated SchemaField object
+ sub_field1 = self._make_one("name", "STRING")
+ sub_field2 = self._make_one("age", "INTEGER")
+ schema_field = self._make_one(
+ "known_people", "RECORD", fields=(sub_field1, sub_field2), mode="REPEATED"
+ )
+
+ standard_field = schema_field.to_standard_sql()
+ self.assertEqual(standard_field, expected_result)
+
+ def test_to_standard_sql_unknown_type(self):
+ sql_type = self._get_standard_sql_data_type_class()
+ field = self._make_one("weird_field", "TROOLEAN")
+
+ standard_field = field.to_standard_sql()
+
+ self.assertEqual(standard_field.name, "weird_field")
+ self.assertEqual(standard_field.type.type_kind, sql_type.TYPE_KIND_UNSPECIFIED)
+ self.assertFalse(standard_field.type.HasField("sub_type"))
+
+ def test___eq___wrong_type(self):
+ field = self._make_one("test", "STRING")
+ other = object()
+ self.assertNotEqual(field, other)
+ self.assertEqual(field, mock.ANY)
+
+ def test___eq___name_mismatch(self):
+ field = self._make_one("test", "STRING")
+ other = self._make_one("other", "STRING")
+ self.assertNotEqual(field, other)
+
+ def test___eq___field_type_mismatch(self):
+ field = self._make_one("test", "STRING")
+ other = self._make_one("test", "INTEGER")
+ self.assertNotEqual(field, other)
+
+ def test___eq___mode_mismatch(self):
+ field = self._make_one("test", "STRING", mode="REQUIRED")
+ other = self._make_one("test", "STRING", mode="NULLABLE")
+ self.assertNotEqual(field, other)
+
+ def test___eq___description_mismatch(self):
+ field = self._make_one("test", "STRING", description="Testing")
+ other = self._make_one("test", "STRING", description="Other")
+ self.assertNotEqual(field, other)
+
+ def test___eq___fields_mismatch(self):
+ sub1 = self._make_one("sub1", "STRING")
+ sub2 = self._make_one("sub2", "STRING")
+ field = self._make_one("test", "RECORD", fields=[sub1])
+ other = self._make_one("test", "RECORD", fields=[sub2])
+ self.assertNotEqual(field, other)
+
+ def test___eq___hit(self):
+ field = self._make_one("test", "STRING", mode="REQUIRED", description="Testing")
+ other = self._make_one("test", "STRING", mode="REQUIRED", description="Testing")
+ self.assertEqual(field, other)
+
+ def test___eq___hit_case_diff_on_type(self):
+ field = self._make_one("test", "STRING", mode="REQUIRED", description="Testing")
+ other = self._make_one("test", "string", mode="REQUIRED", description="Testing")
+ self.assertEqual(field, other)
+
+ def test___eq___hit_w_fields(self):
+ sub1 = self._make_one("sub1", "STRING")
+ sub2 = self._make_one("sub2", "STRING")
+ field = self._make_one("test", "RECORD", fields=[sub1, sub2])
+ other = self._make_one("test", "RECORD", fields=[sub1, sub2])
+ self.assertEqual(field, other)
+
+ def test___ne___wrong_type(self):
+ field = self._make_one("toast", "INTEGER")
+ other = object()
+ self.assertNotEqual(field, other)
+ self.assertEqual(field, mock.ANY)
+
+ def test___ne___same_value(self):
+ field1 = self._make_one("test", "TIMESTAMP", mode="REPEATED")
+ field2 = self._make_one("test", "TIMESTAMP", mode="REPEATED")
+ # unittest ``assertEqual`` uses ``==`` not ``!=``.
+ comparison_val = field1 != field2
+ self.assertFalse(comparison_val)
+
+ def test___ne___different_values(self):
+ field1 = self._make_one(
+ "test1", "FLOAT", mode="REPEATED", description="Not same"
+ )
+ field2 = self._make_one(
+ "test2", "FLOAT", mode="NULLABLE", description="Knot saym"
+ )
+ self.assertNotEqual(field1, field2)
+
+ def test___hash__set_equality(self):
+ sub1 = self._make_one("sub1", "STRING")
+ sub2 = self._make_one("sub2", "STRING")
+ field1 = self._make_one("test", "RECORD", fields=[sub1])
+ field2 = self._make_one("test", "RECORD", fields=[sub2])
+ set_one = {field1, field2}
+ set_two = {field1, field2}
+ self.assertEqual(set_one, set_two)
+
+ def test___hash__not_equals(self):
+ sub1 = self._make_one("sub1", "STRING")
+ sub2 = self._make_one("sub2", "STRING")
+ field1 = self._make_one("test", "RECORD", fields=[sub1])
+ field2 = self._make_one("test", "RECORD", fields=[sub2])
+ set_one = {field1}
+ set_two = {field2}
+ self.assertNotEqual(set_one, set_two)
+
+ def test___repr__(self):
+ field1 = self._make_one("field1", "STRING")
+ expected = "SchemaField('field1', 'STRING', 'NULLABLE', None, ())"
+ self.assertEqual(repr(field1), expected)
+
+
+# TODO: dedup with the same class in test_table.py.
+class _SchemaBase(object):
+ def _verify_field(self, field, r_field):
+ self.assertEqual(field.name, r_field["name"])
+ self.assertEqual(field.field_type, r_field["type"])
+ self.assertEqual(field.mode, r_field.get("mode", "NULLABLE"))
+
+ def _verifySchema(self, schema, resource):
+ r_fields = resource["schema"]["fields"]
+ self.assertEqual(len(schema), len(r_fields))
+
+ for field, r_field in zip(schema, r_fields):
+ self._verify_field(field, r_field)
+
+
+class Test_parse_schema_resource(unittest.TestCase, _SchemaBase):
+ def _call_fut(self, resource):
+ from google.cloud.bigquery.schema import _parse_schema_resource
+
+ return _parse_schema_resource(resource)
+
+ def _make_resource(self):
+ return {
+ "schema": {
+ "fields": [
+ {"name": "full_name", "type": "STRING", "mode": "REQUIRED"},
+ {"name": "age", "type": "INTEGER", "mode": "REQUIRED"},
+ ]
+ }
+ }
+
+ def test__parse_schema_resource_defaults(self):
+ RESOURCE = self._make_resource()
+ schema = self._call_fut(RESOURCE["schema"])
+ self._verifySchema(schema, RESOURCE)
+
+ def test__parse_schema_resource_subfields(self):
+ RESOURCE = self._make_resource()
+ RESOURCE["schema"]["fields"].append(
+ {
+ "name": "phone",
+ "type": "RECORD",
+ "mode": "REPEATED",
+ "fields": [
+ {"name": "type", "type": "STRING", "mode": "REQUIRED"},
+ {"name": "number", "type": "STRING", "mode": "REQUIRED"},
+ ],
+ }
+ )
+ schema = self._call_fut(RESOURCE["schema"])
+ self._verifySchema(schema, RESOURCE)
+
+ def test__parse_schema_resource_fields_without_mode(self):
+ RESOURCE = self._make_resource()
+ RESOURCE["schema"]["fields"].append({"name": "phone", "type": "STRING"})
+
+ schema = self._call_fut(RESOURCE["schema"])
+ self._verifySchema(schema, RESOURCE)
+
+
+class Test_build_schema_resource(unittest.TestCase, _SchemaBase):
+ def _call_fut(self, resource):
+ from google.cloud.bigquery.schema import _build_schema_resource
+
+ return _build_schema_resource(resource)
+
+ def test_defaults(self):
+ from google.cloud.bigquery.schema import SchemaField
+
+ full_name = SchemaField("full_name", "STRING", mode="REQUIRED")
+ age = SchemaField("age", "INTEGER", mode="REQUIRED")
+ resource = self._call_fut([full_name, age])
+ self.assertEqual(len(resource), 2)
+ self.assertEqual(
+ resource[0],
+ {
+ "name": "full_name",
+ "type": "STRING",
+ "mode": "REQUIRED",
+ "description": None,
+ },
+ )
+ self.assertEqual(
+ resource[1],
+ {"name": "age", "type": "INTEGER", "mode": "REQUIRED", "description": None},
+ )
+
+ def test_w_description(self):
+ from google.cloud.bigquery.schema import SchemaField
+
+ DESCRIPTION = "DESCRIPTION"
+ full_name = SchemaField(
+ "full_name", "STRING", mode="REQUIRED", description=DESCRIPTION
+ )
+ age = SchemaField("age", "INTEGER", mode="REQUIRED")
+ resource = self._call_fut([full_name, age])
+ self.assertEqual(len(resource), 2)
+ self.assertEqual(
+ resource[0],
+ {
+ "name": "full_name",
+ "type": "STRING",
+ "mode": "REQUIRED",
+ "description": DESCRIPTION,
+ },
+ )
+ self.assertEqual(
+ resource[1],
+ {"name": "age", "type": "INTEGER", "mode": "REQUIRED", "description": None},
+ )
+
+ def test_w_subfields(self):
+ from google.cloud.bigquery.schema import SchemaField
+
+ full_name = SchemaField("full_name", "STRING", mode="REQUIRED")
+ ph_type = SchemaField("type", "STRING", "REQUIRED")
+ ph_num = SchemaField("number", "STRING", "REQUIRED")
+ phone = SchemaField(
+ "phone", "RECORD", mode="REPEATED", fields=[ph_type, ph_num]
+ )
+ resource = self._call_fut([full_name, phone])
+ self.assertEqual(len(resource), 2)
+ self.assertEqual(
+ resource[0],
+ {
+ "name": "full_name",
+ "type": "STRING",
+ "mode": "REQUIRED",
+ "description": None,
+ },
+ )
+ self.assertEqual(
+ resource[1],
+ {
+ "name": "phone",
+ "type": "RECORD",
+ "mode": "REPEATED",
+ "description": None,
+ "fields": [
+ {
+ "name": "type",
+ "type": "STRING",
+ "mode": "REQUIRED",
+ "description": None,
+ },
+ {
+ "name": "number",
+ "type": "STRING",
+ "mode": "REQUIRED",
+ "description": None,
+ },
+ ],
+ },
+ )
diff --git a/bigquery/tests/unit/test_signature_compatibility.py b/bigquery/tests/unit/test_signature_compatibility.py
new file mode 100644
index 000000000000..6002ae3e87c9
--- /dev/null
+++ b/bigquery/tests/unit/test_signature_compatibility.py
@@ -0,0 +1,51 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import inspect
+
+import pytest
+
+
+@pytest.fixture
+def query_job_class():
+ from google.cloud.bigquery.job import QueryJob
+
+ return QueryJob
+
+
+@pytest.fixture
+def row_iterator_class():
+ from google.cloud.bigquery.table import RowIterator
+
+ return RowIterator
+
+
+@pytest.mark.skipif(
+ not hasattr(inspect, "signature"),
+ reason="inspect.signature() is not availalbe in older Python versions",
+)
+def test_to_arrow_method_signatures_match(query_job_class, row_iterator_class):
+ sig = inspect.signature(query_job_class.to_arrow)
+ sig2 = inspect.signature(row_iterator_class.to_arrow)
+ assert sig == sig2
+
+
+@pytest.mark.skipif(
+ not hasattr(inspect, "signature"),
+ reason="inspect.signature() is not availalbe in older Python versions",
+)
+def test_to_dataframe_method_signatures_match(query_job_class, row_iterator_class):
+ sig = inspect.signature(query_job_class.to_dataframe)
+ sig2 = inspect.signature(row_iterator_class.to_dataframe)
+ assert sig == sig2
diff --git a/bigquery/tests/unit/test_table.py b/bigquery/tests/unit/test_table.py
new file mode 100644
index 000000000000..8ba7fee892e5
--- /dev/null
+++ b/bigquery/tests/unit/test_table.py
@@ -0,0 +1,3020 @@
+# Copyright 2015 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import itertools
+import logging
+import time
+import unittest
+import warnings
+
+import mock
+import pytest
+import six
+
+import google.api_core.exceptions
+
+try:
+ from google.cloud import bigquery_storage_v1beta1
+except ImportError: # pragma: NO COVER
+ bigquery_storage_v1beta1 = None
+
+try:
+ import pandas
+except (ImportError, AttributeError): # pragma: NO COVER
+ pandas = None
+
+try:
+ import pyarrow
+ import pyarrow.types
+except ImportError: # pragma: NO COVER
+ pyarrow = None
+
+try:
+ from tqdm import tqdm
+except (ImportError, AttributeError): # pragma: NO COVER
+ tqdm = None
+
+from google.cloud.bigquery.dataset import DatasetReference
+
+
+def _mock_client():
+ from google.cloud.bigquery import client
+
+ mock_client = mock.create_autospec(client.Client)
+ mock_client.project = "my-project"
+ return mock_client
+
+
+class _SchemaBase(object):
+ def _verify_field(self, field, r_field):
+ self.assertEqual(field.name, r_field["name"])
+ self.assertEqual(field.field_type, r_field["type"])
+ self.assertEqual(field.mode, r_field.get("mode", "NULLABLE"))
+
+ def _verifySchema(self, schema, resource):
+ r_fields = resource["schema"]["fields"]
+ self.assertEqual(len(schema), len(r_fields))
+
+ for field, r_field in zip(schema, r_fields):
+ self._verify_field(field, r_field)
+
+
+class TestEncryptionConfiguration(unittest.TestCase):
+ KMS_KEY_NAME = "projects/1/locations/global/keyRings/1/cryptoKeys/1"
+
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery.table import EncryptionConfiguration
+
+ return EncryptionConfiguration
+
+ def _make_one(self, *args, **kw):
+ return self._get_target_class()(*args, **kw)
+
+ def test_ctor_defaults(self):
+ encryption_config = self._make_one()
+ self.assertIsNone(encryption_config.kms_key_name)
+
+ def test_ctor_with_key(self):
+ encryption_config = self._make_one(kms_key_name=self.KMS_KEY_NAME)
+ self.assertEqual(encryption_config.kms_key_name, self.KMS_KEY_NAME)
+
+ def test_kms_key_name_setter(self):
+ encryption_config = self._make_one()
+ self.assertIsNone(encryption_config.kms_key_name)
+ encryption_config.kms_key_name = self.KMS_KEY_NAME
+ self.assertEqual(encryption_config.kms_key_name, self.KMS_KEY_NAME)
+ encryption_config.kms_key_name = None
+ self.assertIsNone(encryption_config.kms_key_name)
+
+ def test_from_api_repr(self):
+ RESOURCE = {"kmsKeyName": self.KMS_KEY_NAME}
+ klass = self._get_target_class()
+ encryption_config = klass.from_api_repr(RESOURCE)
+ self.assertEqual(encryption_config.kms_key_name, self.KMS_KEY_NAME)
+
+ def test_to_api_repr(self):
+ encryption_config = self._make_one(kms_key_name=self.KMS_KEY_NAME)
+ resource = encryption_config.to_api_repr()
+ self.assertEqual(resource, {"kmsKeyName": self.KMS_KEY_NAME})
+
+ def test___eq___wrong_type(self):
+ encryption_config = self._make_one()
+ other = object()
+ self.assertNotEqual(encryption_config, other)
+ self.assertEqual(encryption_config, mock.ANY)
+
+ def test___eq___kms_key_name_mismatch(self):
+ encryption_config = self._make_one()
+ other = self._make_one(self.KMS_KEY_NAME)
+ self.assertNotEqual(encryption_config, other)
+
+ def test___eq___hit(self):
+ encryption_config = self._make_one(self.KMS_KEY_NAME)
+ other = self._make_one(self.KMS_KEY_NAME)
+ self.assertEqual(encryption_config, other)
+
+ def test___ne___wrong_type(self):
+ encryption_config = self._make_one()
+ other = object()
+ self.assertNotEqual(encryption_config, other)
+ self.assertEqual(encryption_config, mock.ANY)
+
+ def test___ne___same_value(self):
+ encryption_config1 = self._make_one(self.KMS_KEY_NAME)
+ encryption_config2 = self._make_one(self.KMS_KEY_NAME)
+ # unittest ``assertEqual`` uses ``==`` not ``!=``.
+ comparison_val = encryption_config1 != encryption_config2
+ self.assertFalse(comparison_val)
+
+ def test___ne___different_values(self):
+ encryption_config1 = self._make_one()
+ encryption_config2 = self._make_one(self.KMS_KEY_NAME)
+ self.assertNotEqual(encryption_config1, encryption_config2)
+
+ def test___hash__set_equality(self):
+ encryption_config1 = self._make_one(self.KMS_KEY_NAME)
+ encryption_config2 = self._make_one(self.KMS_KEY_NAME)
+ set_one = {encryption_config1, encryption_config2}
+ set_two = {encryption_config1, encryption_config2}
+ self.assertEqual(set_one, set_two)
+
+ def test___hash__not_equals(self):
+ encryption_config1 = self._make_one()
+ encryption_config2 = self._make_one(self.KMS_KEY_NAME)
+ set_one = {encryption_config1}
+ set_two = {encryption_config2}
+ self.assertNotEqual(set_one, set_two)
+
+ def test___repr__(self):
+ encryption_config = self._make_one(self.KMS_KEY_NAME)
+ expected = "EncryptionConfiguration({})".format(self.KMS_KEY_NAME)
+ self.assertEqual(repr(encryption_config), expected)
+
+
+class TestTableReference(unittest.TestCase):
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery.table import TableReference
+
+ return TableReference
+
+ def _make_one(self, *args, **kw):
+ return self._get_target_class()(*args, **kw)
+
+ def test_ctor_defaults(self):
+ from google.cloud.bigquery.dataset import DatasetReference
+
+ dataset_ref = DatasetReference("project_1", "dataset_1")
+
+ table_ref = self._make_one(dataset_ref, "table_1")
+ self.assertEqual(table_ref.dataset_id, dataset_ref.dataset_id)
+ self.assertEqual(table_ref.table_id, "table_1")
+
+ def test_to_api_repr(self):
+ from google.cloud.bigquery.dataset import DatasetReference
+
+ dataset_ref = DatasetReference("project_1", "dataset_1")
+ table_ref = self._make_one(dataset_ref, "table_1")
+
+ resource = table_ref.to_api_repr()
+
+ self.assertEqual(
+ resource,
+ {"projectId": "project_1", "datasetId": "dataset_1", "tableId": "table_1"},
+ )
+
+ def test_from_api_repr(self):
+ from google.cloud.bigquery.dataset import DatasetReference
+ from google.cloud.bigquery.table import TableReference
+
+ dataset_ref = DatasetReference("project_1", "dataset_1")
+ expected = self._make_one(dataset_ref, "table_1")
+
+ got = TableReference.from_api_repr(
+ {"projectId": "project_1", "datasetId": "dataset_1", "tableId": "table_1"}
+ )
+
+ self.assertEqual(expected, got)
+
+ def test_from_string(self):
+ cls = self._get_target_class()
+ got = cls.from_string("string-project.string_dataset.string_table")
+ self.assertEqual(got.project, "string-project")
+ self.assertEqual(got.dataset_id, "string_dataset")
+ self.assertEqual(got.table_id, "string_table")
+
+ def test_from_string_legacy_string(self):
+ cls = self._get_target_class()
+ with self.assertRaises(ValueError):
+ cls.from_string("string-project:string_dataset.string_table")
+
+ def test_from_string_not_fully_qualified(self):
+ cls = self._get_target_class()
+ with self.assertRaises(ValueError):
+ cls.from_string("string_table")
+
+ with self.assertRaises(ValueError):
+ cls.from_string("string_dataset.string_table")
+
+ with self.assertRaises(ValueError):
+ cls.from_string("a.b.c.d")
+
+ def test_from_string_with_default_project(self):
+ cls = self._get_target_class()
+ got = cls.from_string(
+ "string_dataset.string_table", default_project="default-project"
+ )
+ self.assertEqual(got.project, "default-project")
+ self.assertEqual(got.dataset_id, "string_dataset")
+ self.assertEqual(got.table_id, "string_table")
+
+ def test_from_string_ignores_default_project(self):
+ cls = self._get_target_class()
+ got = cls.from_string(
+ "string-project.string_dataset.string_table",
+ default_project="default-project",
+ )
+ self.assertEqual(got.project, "string-project")
+ self.assertEqual(got.dataset_id, "string_dataset")
+ self.assertEqual(got.table_id, "string_table")
+
+ def test___eq___wrong_type(self):
+ from google.cloud.bigquery.dataset import DatasetReference
+
+ dataset_ref = DatasetReference("project_1", "dataset_1")
+ table = self._make_one(dataset_ref, "table_1")
+ other = object()
+ self.assertNotEqual(table, other)
+ self.assertEqual(table, mock.ANY)
+
+ def test___eq___project_mismatch(self):
+ from google.cloud.bigquery.dataset import DatasetReference
+
+ dataset = DatasetReference("project_1", "dataset_1")
+ other_dataset = DatasetReference("project_2", "dataset_1")
+ table = self._make_one(dataset, "table_1")
+ other = self._make_one(other_dataset, "table_1")
+ self.assertNotEqual(table, other)
+
+ def test___eq___dataset_mismatch(self):
+ from google.cloud.bigquery.dataset import DatasetReference
+
+ dataset = DatasetReference("project_1", "dataset_1")
+ other_dataset = DatasetReference("project_1", "dataset_2")
+ table = self._make_one(dataset, "table_1")
+ other = self._make_one(other_dataset, "table_1")
+ self.assertNotEqual(table, other)
+
+ def test___eq___table_mismatch(self):
+ from google.cloud.bigquery.dataset import DatasetReference
+
+ dataset = DatasetReference("project_1", "dataset_1")
+ table = self._make_one(dataset, "table_1")
+ other = self._make_one(dataset, "table_2")
+ self.assertNotEqual(table, other)
+
+ def test___eq___equality(self):
+ from google.cloud.bigquery.dataset import DatasetReference
+
+ dataset = DatasetReference("project_1", "dataset_1")
+ table = self._make_one(dataset, "table_1")
+ other = self._make_one(dataset, "table_1")
+ self.assertEqual(table, other)
+
+ def test___hash__set_equality(self):
+ from google.cloud.bigquery.dataset import DatasetReference
+
+ dataset = DatasetReference("project_1", "dataset_1")
+ table1 = self._make_one(dataset, "table1")
+ table2 = self._make_one(dataset, "table2")
+ set_one = {table1, table2}
+ set_two = {table1, table2}
+ self.assertEqual(set_one, set_two)
+
+ def test___hash__not_equals(self):
+ from google.cloud.bigquery.dataset import DatasetReference
+
+ dataset = DatasetReference("project_1", "dataset_1")
+ table1 = self._make_one(dataset, "table1")
+ table2 = self._make_one(dataset, "table2")
+ set_one = {table1}
+ set_two = {table2}
+ self.assertNotEqual(set_one, set_two)
+
+ def test___repr__(self):
+ dataset = DatasetReference("project1", "dataset1")
+ table1 = self._make_one(dataset, "table1")
+ expected = (
+ "TableReference(DatasetReference('project1', 'dataset1'), " "'table1')"
+ )
+ self.assertEqual(repr(table1), expected)
+
+
+class TestTable(unittest.TestCase, _SchemaBase):
+
+ PROJECT = "prahj-ekt"
+ DS_ID = "dataset-name"
+ TABLE_NAME = "table-name"
+ KMS_KEY_NAME = "projects/1/locations/global/keyRings/1/cryptoKeys/1"
+
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery.table import Table
+
+ return Table
+
+ def _make_one(self, *args, **kw):
+ return self._get_target_class()(*args, **kw)
+
+ def _setUpConstants(self):
+ import datetime
+ from google.cloud._helpers import UTC
+
+ self.WHEN_TS = 1437767599.006
+ self.WHEN = datetime.datetime.utcfromtimestamp(self.WHEN_TS).replace(tzinfo=UTC)
+ self.ETAG = "ETAG"
+ self.TABLE_FULL_ID = "%s:%s.%s" % (self.PROJECT, self.DS_ID, self.TABLE_NAME)
+ self.RESOURCE_URL = "http://example.com/path/to/resource"
+ self.NUM_BYTES = 12345
+ self.NUM_ROWS = 67
+ self.NUM_EST_BYTES = 1234
+ self.NUM_EST_ROWS = 23
+
+ def _make_resource(self):
+ self._setUpConstants()
+ return {
+ "creationTime": self.WHEN_TS * 1000,
+ "tableReference": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.TABLE_NAME,
+ },
+ "schema": {
+ "fields": [
+ {"name": "full_name", "type": "STRING", "mode": "REQUIRED"},
+ {"name": "age", "type": "INTEGER", "mode": "REQUIRED"},
+ ]
+ },
+ "etag": "ETAG",
+ "id": self.TABLE_FULL_ID,
+ "lastModifiedTime": self.WHEN_TS * 1000,
+ "location": "US",
+ "selfLink": self.RESOURCE_URL,
+ "numRows": self.NUM_ROWS,
+ "numBytes": self.NUM_BYTES,
+ "type": "TABLE",
+ "streamingBuffer": {
+ "estimatedRows": str(self.NUM_EST_ROWS),
+ "estimatedBytes": str(self.NUM_EST_BYTES),
+ "oldestEntryTime": self.WHEN_TS * 1000,
+ },
+ "externalDataConfiguration": {
+ "sourceFormat": "CSV",
+ "csvOptions": {"allowJaggedRows": True, "encoding": "encoding"},
+ },
+ "labels": {"x": "y"},
+ }
+
+ def _verifyReadonlyResourceProperties(self, table, resource):
+ if "creationTime" in resource:
+ self.assertEqual(table.created, self.WHEN)
+ else:
+ self.assertIsNone(table.created)
+
+ if "etag" in resource:
+ self.assertEqual(table.etag, self.ETAG)
+ else:
+ self.assertIsNone(table.etag)
+
+ if "numRows" in resource:
+ self.assertEqual(table.num_rows, self.NUM_ROWS)
+ else:
+ self.assertIsNone(table.num_rows)
+
+ if "numBytes" in resource:
+ self.assertEqual(table.num_bytes, self.NUM_BYTES)
+ else:
+ self.assertIsNone(table.num_bytes)
+
+ if "selfLink" in resource:
+ self.assertEqual(table.self_link, self.RESOURCE_URL)
+ else:
+ self.assertIsNone(table.self_link)
+
+ if "streamingBuffer" in resource:
+ self.assertEqual(table.streaming_buffer.estimated_rows, self.NUM_EST_ROWS)
+ self.assertEqual(table.streaming_buffer.estimated_bytes, self.NUM_EST_BYTES)
+ self.assertEqual(table.streaming_buffer.oldest_entry_time, self.WHEN)
+ else:
+ self.assertIsNone(table.streaming_buffer)
+
+ self.assertEqual(table.full_table_id, self.TABLE_FULL_ID)
+ self.assertEqual(
+ table.table_type, "TABLE" if "view" not in resource else "VIEW"
+ )
+
+ def _verifyResourceProperties(self, table, resource):
+
+ self._verifyReadonlyResourceProperties(table, resource)
+
+ if "expirationTime" in resource:
+ self.assertEqual(table.expires, self.EXP_TIME)
+ else:
+ self.assertIsNone(table.expires)
+
+ self.assertEqual(table.description, resource.get("description"))
+ self.assertEqual(table.friendly_name, resource.get("friendlyName"))
+ self.assertEqual(table.location, resource.get("location"))
+
+ if "view" in resource:
+ self.assertEqual(table.view_query, resource["view"]["query"])
+ self.assertEqual(
+ table.view_use_legacy_sql, resource["view"].get("useLegacySql", True)
+ )
+ else:
+ self.assertIsNone(table.view_query)
+ self.assertIsNone(table.view_use_legacy_sql)
+
+ if "schema" in resource:
+ self._verifySchema(table.schema, resource)
+ else:
+ self.assertEqual(table.schema, [])
+
+ if "externalDataConfiguration" in resource:
+ edc = table.external_data_configuration
+ self.assertEqual(edc.source_format, "CSV")
+ self.assertEqual(edc.options.allow_jagged_rows, True)
+
+ if "labels" in resource:
+ self.assertEqual(table.labels, {"x": "y"})
+ else:
+ self.assertEqual(table.labels, {})
+
+ if "encryptionConfiguration" in resource:
+ self.assertIsNotNone(table.encryption_configuration)
+ self.assertEqual(
+ table.encryption_configuration.kms_key_name,
+ resource["encryptionConfiguration"]["kmsKeyName"],
+ )
+ else:
+ self.assertIsNone(table.encryption_configuration)
+
+ def test_ctor(self):
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+
+ self.assertEqual(table.table_id, self.TABLE_NAME)
+ self.assertEqual(table.project, self.PROJECT)
+ self.assertEqual(table.dataset_id, self.DS_ID)
+ self.assertEqual(table.reference.table_id, self.TABLE_NAME)
+ self.assertEqual(table.reference.project, self.PROJECT)
+ self.assertEqual(table.reference.dataset_id, self.DS_ID)
+ self.assertEqual(
+ table.path,
+ "/projects/%s/datasets/%s/tables/%s"
+ % (self.PROJECT, self.DS_ID, self.TABLE_NAME),
+ )
+ self.assertEqual(table.schema, [])
+
+ self.assertIsNone(table.created)
+ self.assertIsNone(table.etag)
+ self.assertIsNone(table.modified)
+ self.assertIsNone(table.num_bytes)
+ self.assertIsNone(table.num_rows)
+ self.assertIsNone(table.self_link)
+ self.assertIsNone(table.full_table_id)
+ self.assertIsNone(table.table_type)
+ self.assertIsNone(table.description)
+ self.assertIsNone(table.expires)
+ self.assertIsNone(table.friendly_name)
+ self.assertIsNone(table.location)
+ self.assertIsNone(table.view_query)
+ self.assertIsNone(table.view_use_legacy_sql)
+ self.assertIsNone(table.external_data_configuration)
+ self.assertEqual(table.labels, {})
+ self.assertIsNone(table.encryption_configuration)
+ self.assertIsNone(table.time_partitioning)
+ self.assertIsNone(table.clustering_fields)
+
+ def test_ctor_w_schema(self):
+ from google.cloud.bigquery.table import SchemaField
+
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ full_name = SchemaField("full_name", "STRING", mode="REQUIRED")
+ age = SchemaField("age", "INTEGER", mode="REQUIRED")
+ table = self._make_one(table_ref, schema=[full_name, age])
+
+ self.assertEqual(table.schema, [full_name, age])
+
+ def test_ctor_string(self):
+ table = self._make_one("some-project.some_dset.some_tbl")
+ self.assertEqual(table.project, "some-project")
+ self.assertEqual(table.dataset_id, "some_dset")
+ self.assertEqual(table.table_id, "some_tbl")
+
+ def test_ctor_tablelistitem(self):
+ from google.cloud.bigquery.table import Table, TableListItem
+
+ import datetime
+ from google.cloud._helpers import _millis, UTC
+
+ self.WHEN_TS = 1437767599.125
+ self.EXP_TIME = datetime.datetime(2015, 8, 1, 23, 59, 59, tzinfo=UTC)
+
+ project = "test-project"
+ dataset_id = "test_dataset"
+ table_id = "coffee_table"
+ resource = {
+ "creationTime": self.WHEN_TS * 1000,
+ "expirationTime": _millis(self.EXP_TIME),
+ "kind": "bigquery#table",
+ "id": "{}:{}.{}".format(project, dataset_id, table_id),
+ "tableReference": {
+ "projectId": project,
+ "datasetId": dataset_id,
+ "tableId": table_id,
+ },
+ "friendlyName": "Mahogany Coffee Table",
+ "type": "TABLE",
+ "timePartitioning": {
+ "type": "DAY",
+ "field": "mycolumn",
+ "expirationMs": "10000",
+ },
+ "labels": {"some-stuff": "this-is-a-label"},
+ "clustering": {"fields": ["string"]},
+ }
+
+ table_list_item = TableListItem(resource)
+ table = Table(table_list_item)
+
+ self.assertIsNone(table.created)
+ self.assertEqual(table.reference.project, project)
+ self.assertEqual(table.reference.dataset_id, dataset_id)
+ self.assertEqual(table.reference.table_id, table_id)
+
+ def test_ctor_string_wo_project_id(self):
+ with pytest.raises(ValueError):
+ # Project ID is missing.
+ self._make_one("some_dset.some_tbl")
+
+ def test_num_bytes_getter(self):
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+
+ # Check with no value set.
+ self.assertIsNone(table.num_bytes)
+
+ num_bytes = 1337
+ # Check with integer value set.
+ table._properties = {"numBytes": num_bytes}
+ self.assertEqual(table.num_bytes, num_bytes)
+
+ # Check with a string value set.
+ table._properties = {"numBytes": str(num_bytes)}
+ self.assertEqual(table.num_bytes, num_bytes)
+
+ # Check with invalid int value.
+ table._properties = {"numBytes": "x"}
+ with self.assertRaises(ValueError):
+ getattr(table, "num_bytes")
+
+ def test_num_rows_getter(self):
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+
+ # Check with no value set.
+ self.assertIsNone(table.num_rows)
+
+ num_rows = 42
+ # Check with integer value set.
+ table._properties = {"numRows": num_rows}
+ self.assertEqual(table.num_rows, num_rows)
+
+ # Check with a string value set.
+ table._properties = {"numRows": str(num_rows)}
+ self.assertEqual(table.num_rows, num_rows)
+
+ # Check with invalid int value.
+ table._properties = {"numRows": "x"}
+ with self.assertRaises(ValueError):
+ getattr(table, "num_rows")
+
+ def test_schema_setter_non_list(self):
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+ with self.assertRaises(TypeError):
+ table.schema = object()
+
+ def test_schema_setter_invalid_field(self):
+ from google.cloud.bigquery.table import SchemaField
+
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+ full_name = SchemaField("full_name", "STRING", mode="REQUIRED")
+ with self.assertRaises(ValueError):
+ table.schema = [full_name, object()]
+
+ def test_schema_setter(self):
+ from google.cloud.bigquery.table import SchemaField
+
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+ full_name = SchemaField("full_name", "STRING", mode="REQUIRED")
+ age = SchemaField("age", "INTEGER", mode="REQUIRED")
+ table.schema = [full_name, age]
+ self.assertEqual(table.schema, [full_name, age])
+
+ def test_props_set_by_server(self):
+ import datetime
+ from google.cloud._helpers import UTC
+ from google.cloud._helpers import _millis
+
+ CREATED = datetime.datetime(2015, 7, 29, 12, 13, 22, tzinfo=UTC)
+ MODIFIED = datetime.datetime(2015, 7, 29, 14, 47, 15, tzinfo=UTC)
+ TABLE_FULL_ID = "%s:%s.%s" % (self.PROJECT, self.DS_ID, self.TABLE_NAME)
+ URL = "http://example.com/projects/%s/datasets/%s/tables/%s" % (
+ self.PROJECT,
+ self.DS_ID,
+ self.TABLE_NAME,
+ )
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+ table._properties["creationTime"] = _millis(CREATED)
+ table._properties["etag"] = "ETAG"
+ table._properties["lastModifiedTime"] = _millis(MODIFIED)
+ table._properties["numBytes"] = 12345
+ table._properties["numRows"] = 66
+ table._properties["selfLink"] = URL
+ table._properties["id"] = TABLE_FULL_ID
+ table._properties["type"] = "TABLE"
+
+ self.assertEqual(table.created, CREATED)
+ self.assertEqual(table.etag, "ETAG")
+ self.assertEqual(table.modified, MODIFIED)
+ self.assertEqual(table.num_bytes, 12345)
+ self.assertEqual(table.num_rows, 66)
+ self.assertEqual(table.self_link, URL)
+ self.assertEqual(table.full_table_id, TABLE_FULL_ID)
+ self.assertEqual(table.table_type, "TABLE")
+
+ def test_description_setter_bad_value(self):
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+ with self.assertRaises(ValueError):
+ table.description = 12345
+
+ def test_description_setter(self):
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+ table.description = "DESCRIPTION"
+ self.assertEqual(table.description, "DESCRIPTION")
+
+ def test_expires_setter_bad_value(self):
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+ with self.assertRaises(ValueError):
+ table.expires = object()
+
+ def test_expires_setter(self):
+ import datetime
+ from google.cloud._helpers import UTC
+
+ WHEN = datetime.datetime(2015, 7, 28, 16, 39, tzinfo=UTC)
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+ table.expires = WHEN
+ self.assertEqual(table.expires, WHEN)
+
+ def test_friendly_name_setter_bad_value(self):
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+ with self.assertRaises(ValueError):
+ table.friendly_name = 12345
+
+ def test_friendly_name_setter(self):
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+ table.friendly_name = "FRIENDLY"
+ self.assertEqual(table.friendly_name, "FRIENDLY")
+
+ def test_view_query_setter_bad_value(self):
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+ with self.assertRaises(ValueError):
+ table.view_query = 12345
+
+ def test_view_query_setter(self):
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+ table.view_query = "select * from foo"
+ self.assertEqual(table.view_query, "select * from foo")
+ self.assertEqual(table.view_use_legacy_sql, False)
+
+ table.view_use_legacy_sql = True
+ self.assertEqual(table.view_use_legacy_sql, True)
+
+ def test_view_query_deleter(self):
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+ table.view_query = "select * from foo"
+ del table.view_query
+ self.assertIsNone(table.view_query)
+ self.assertIsNone(table.view_use_legacy_sql)
+
+ def test_view_use_legacy_sql_setter_bad_value(self):
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+ with self.assertRaises(ValueError):
+ table.view_use_legacy_sql = 12345
+
+ def test_view_use_legacy_sql_setter(self):
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+ table.view_use_legacy_sql = True
+ table.view_query = "select * from foo"
+ self.assertEqual(table.view_use_legacy_sql, True)
+ self.assertEqual(table.view_query, "select * from foo")
+
+ def test_external_data_configuration_setter(self):
+ from google.cloud.bigquery.external_config import ExternalConfig
+
+ external_config = ExternalConfig("CSV")
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+
+ table.external_data_configuration = external_config
+
+ self.assertEqual(
+ table.external_data_configuration.source_format,
+ external_config.source_format,
+ )
+
+ def test_external_data_configuration_setter_none(self):
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+
+ table.external_data_configuration = None
+
+ self.assertIsNone(table.external_data_configuration)
+
+ def test_external_data_configuration_setter_bad_value(self):
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+ with self.assertRaises(ValueError):
+ table.external_data_configuration = 12345
+
+ def test_labels_update_in_place(self):
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+ del table._properties["labels"] # don't start w/ existing dict
+ labels = table.labels
+ labels["foo"] = "bar" # update in place
+ self.assertEqual(table.labels, {"foo": "bar"})
+
+ def test_labels_setter_bad_value(self):
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+ with self.assertRaises(ValueError):
+ table.labels = 12345
+
+ def test_from_string(self):
+ cls = self._get_target_class()
+ got = cls.from_string("string-project.string_dataset.string_table")
+ self.assertEqual(got.project, "string-project")
+ self.assertEqual(got.dataset_id, "string_dataset")
+ self.assertEqual(got.table_id, "string_table")
+
+ def test_from_string_legacy_string(self):
+ cls = self._get_target_class()
+ with self.assertRaises(ValueError):
+ cls.from_string("string-project:string_dataset.string_table")
+
+ def test_from_string_not_fully_qualified(self):
+ cls = self._get_target_class()
+ with self.assertRaises(ValueError):
+ cls.from_string("string_dataset.string_table")
+
+ def test_from_api_repr_missing_identity(self):
+ self._setUpConstants()
+ RESOURCE = {}
+ klass = self._get_target_class()
+ with self.assertRaises(KeyError):
+ klass.from_api_repr(RESOURCE)
+
+ def test_from_api_repr_bare(self):
+ self._setUpConstants()
+ RESOURCE = {
+ "id": "%s:%s.%s" % (self.PROJECT, self.DS_ID, self.TABLE_NAME),
+ "tableReference": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.TABLE_NAME,
+ },
+ "type": "TABLE",
+ }
+ klass = self._get_target_class()
+ table = klass.from_api_repr(RESOURCE)
+ self.assertEqual(table.table_id, self.TABLE_NAME)
+ self._verifyResourceProperties(table, RESOURCE)
+
+ def test_from_api_repr_w_properties(self):
+ import datetime
+ from google.cloud._helpers import UTC
+ from google.cloud._helpers import _millis
+
+ RESOURCE = self._make_resource()
+ RESOURCE["view"] = {"query": "select fullname, age from person_ages"}
+ RESOURCE["type"] = "VIEW"
+ RESOURCE["location"] = "EU"
+ self.EXP_TIME = datetime.datetime(2015, 8, 1, 23, 59, 59, tzinfo=UTC)
+ RESOURCE["expirationTime"] = _millis(self.EXP_TIME)
+ klass = self._get_target_class()
+ table = klass.from_api_repr(RESOURCE)
+ self._verifyResourceProperties(table, RESOURCE)
+
+ def test_from_api_with_encryption(self):
+ self._setUpConstants()
+ RESOURCE = {
+ "id": "%s:%s.%s" % (self.PROJECT, self.DS_ID, self.TABLE_NAME),
+ "tableReference": {
+ "projectId": self.PROJECT,
+ "datasetId": self.DS_ID,
+ "tableId": self.TABLE_NAME,
+ },
+ "encryptionConfiguration": {"kmsKeyName": self.KMS_KEY_NAME},
+ "type": "TABLE",
+ }
+ klass = self._get_target_class()
+ table = klass.from_api_repr(RESOURCE)
+ self._verifyResourceProperties(table, RESOURCE)
+
+ def test_to_api_repr_w_custom_field(self):
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+ table._properties["newAlphaProperty"] = "unreleased property"
+ resource = table.to_api_repr()
+
+ exp_resource = {
+ "tableReference": table_ref.to_api_repr(),
+ "labels": {},
+ "newAlphaProperty": "unreleased property",
+ }
+ self.assertEqual(resource, exp_resource)
+
+ def test__build_resource_w_custom_field(self):
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+ table._properties["newAlphaProperty"] = "unreleased property"
+ resource = table._build_resource(["newAlphaProperty"])
+
+ exp_resource = {"newAlphaProperty": "unreleased property"}
+ self.assertEqual(resource, exp_resource)
+
+ def test__build_resource_w_custom_field_not_in__properties(self):
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table = self._make_one(dataset.table(self.TABLE_NAME))
+ table.bad = "value"
+ with self.assertRaises(ValueError):
+ table._build_resource(["bad"])
+
+ def test_time_partitioning_getter(self):
+ from google.cloud.bigquery.table import TimePartitioning
+ from google.cloud.bigquery.table import TimePartitioningType
+
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+
+ table._properties["timePartitioning"] = {
+ "type": "DAY",
+ "field": "col1",
+ "expirationMs": "123456",
+ "requirePartitionFilter": False,
+ }
+ self.assertIsInstance(table.time_partitioning, TimePartitioning)
+ self.assertEqual(table.time_partitioning.type_, TimePartitioningType.DAY)
+ self.assertEqual(table.time_partitioning.field, "col1")
+ self.assertEqual(table.time_partitioning.expiration_ms, 123456)
+ self.assertFalse(table.time_partitioning.require_partition_filter)
+
+ def test_time_partitioning_getter_w_none(self):
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+
+ table._properties["timePartitioning"] = None
+ self.assertIsNone(table.time_partitioning)
+
+ del table._properties["timePartitioning"]
+ self.assertIsNone(table.time_partitioning)
+
+ def test_time_partitioning_getter_w_empty(self):
+ from google.cloud.bigquery.table import TimePartitioning
+
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+
+ # Even though there are required properties according to the API
+ # specification, sometimes time partitioning is populated as an empty
+ # object. See internal bug 131167013.
+ table._properties["timePartitioning"] = {}
+ self.assertIsInstance(table.time_partitioning, TimePartitioning)
+ self.assertIsNone(table.time_partitioning.type_)
+ self.assertIsNone(table.time_partitioning.field)
+ self.assertIsNone(table.time_partitioning.expiration_ms)
+ self.assertIsNone(table.time_partitioning.require_partition_filter)
+
+ def test_time_partitioning_setter(self):
+ from google.cloud.bigquery.table import TimePartitioning
+ from google.cloud.bigquery.table import TimePartitioningType
+
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+ time_partitioning = TimePartitioning(type_=TimePartitioningType.DAY)
+
+ table.time_partitioning = time_partitioning
+
+ self.assertEqual(table.time_partitioning.type_, TimePartitioningType.DAY)
+ # Both objects point to the same properties dict
+ self.assertIs(
+ table._properties["timePartitioning"], time_partitioning._properties
+ )
+
+ time_partitioning.expiration_ms = 10000
+
+ # Changes to TimePartitioning object are reflected in Table properties
+ self.assertEqual(
+ table.time_partitioning.expiration_ms, time_partitioning.expiration_ms
+ )
+
+ def test_time_partitioning_setter_bad_type(self):
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+
+ with self.assertRaises(ValueError):
+ table.time_partitioning = {"timePartitioning": {"type": "DAY"}}
+
+ def test_time_partitioning_setter_none(self):
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+
+ table.time_partitioning = None
+
+ self.assertIsNone(table.time_partitioning)
+
+ def test_partitioning_type_setter(self):
+ from google.cloud.bigquery.table import TimePartitioningType
+
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+
+ with warnings.catch_warnings(record=True) as warned:
+ self.assertIsNone(table.partitioning_type)
+
+ table.partitioning_type = TimePartitioningType.DAY
+
+ self.assertEqual(table.partitioning_type, "DAY")
+
+ self.assertEqual(len(warned), 3)
+ for warning in warned:
+ self.assertIs(warning.category, PendingDeprecationWarning)
+
+ def test_partitioning_type_setter_w_time_partitioning_set(self):
+ from google.cloud.bigquery.table import TimePartitioning
+
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+ table.time_partitioning = TimePartitioning()
+
+ with warnings.catch_warnings(record=True) as warned:
+ table.partitioning_type = "NEW_FAKE_TYPE"
+
+ self.assertEqual(table.partitioning_type, "NEW_FAKE_TYPE")
+
+ self.assertEqual(len(warned), 2)
+ for warning in warned:
+ self.assertIs(warning.category, PendingDeprecationWarning)
+
+ def test_partitioning_expiration_setter_w_time_partitioning_set(self):
+ from google.cloud.bigquery.table import TimePartitioning
+
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+ table.time_partitioning = TimePartitioning()
+
+ with warnings.catch_warnings(record=True) as warned:
+ table.partition_expiration = 100000
+
+ self.assertEqual(table.partition_expiration, 100000)
+
+ self.assertEqual(len(warned), 2)
+ for warning in warned:
+ self.assertIs(warning.category, PendingDeprecationWarning)
+
+ def test_partition_expiration_setter(self):
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+
+ with warnings.catch_warnings(record=True) as warned:
+ self.assertIsNone(table.partition_expiration)
+
+ table.partition_expiration = 100
+
+ self.assertEqual(table.partition_expiration, 100)
+ # defaults to 'DAY' when expiration is set and type is not set
+ self.assertEqual(table.partitioning_type, "DAY")
+
+ self.assertEqual(len(warned), 4)
+ for warning in warned:
+ self.assertIs(warning.category, PendingDeprecationWarning)
+
+ def test_clustering_fields_setter_w_fields(self):
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+ fields = ["email", "phone"]
+
+ table.clustering_fields = fields
+ self.assertEqual(table.clustering_fields, fields)
+ self.assertEqual(table._properties["clustering"], {"fields": fields})
+
+ def test_clustering_fields_setter_w_none(self):
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+ fields = ["email", "phone"]
+
+ table._properties["clustering"] = {"fields": fields}
+ table.clustering_fields = None
+ self.assertEqual(table.clustering_fields, None)
+ self.assertFalse("clustering" in table._properties)
+
+ def test_clustering_fields_setter_w_none_noop(self):
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+
+ table.clustering_fields = None
+ self.assertEqual(table.clustering_fields, None)
+ self.assertFalse("clustering" in table._properties)
+
+ def test_encryption_configuration_setter(self):
+ from google.cloud.bigquery.table import EncryptionConfiguration
+
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = self._make_one(table_ref)
+ encryption_configuration = EncryptionConfiguration(
+ kms_key_name=self.KMS_KEY_NAME
+ )
+ table.encryption_configuration = encryption_configuration
+ self.assertEqual(table.encryption_configuration.kms_key_name, self.KMS_KEY_NAME)
+ table.encryption_configuration = None
+ self.assertIsNone(table.encryption_configuration)
+
+ def test___repr__(self):
+ from google.cloud.bigquery.table import TableReference
+
+ dataset = DatasetReference("project1", "dataset1")
+ table1 = self._make_one(TableReference(dataset, "table1"))
+ expected = (
+ "Table(TableReference("
+ "DatasetReference('project1', 'dataset1'), "
+ "'table1'))"
+ )
+ self.assertEqual(repr(table1), expected)
+
+
+class Test_row_from_mapping(unittest.TestCase, _SchemaBase):
+
+ PROJECT = "prahj-ekt"
+ DS_ID = "dataset-name"
+ TABLE_NAME = "table-name"
+
+ def _call_fut(self, mapping, schema):
+ from google.cloud.bigquery.table import _row_from_mapping
+
+ return _row_from_mapping(mapping, schema)
+
+ def test__row_from_mapping_wo_schema(self):
+ from google.cloud.bigquery.table import Table, _TABLE_HAS_NO_SCHEMA
+
+ MAPPING = {"full_name": "Phred Phlyntstone", "age": 32}
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ table = Table(table_ref)
+
+ with self.assertRaises(ValueError) as exc:
+ self._call_fut(MAPPING, table.schema)
+
+ self.assertEqual(exc.exception.args, (_TABLE_HAS_NO_SCHEMA,))
+
+ def test__row_from_mapping_w_invalid_schema(self):
+ from google.cloud.bigquery.table import Table, SchemaField
+
+ MAPPING = {
+ "full_name": "Phred Phlyntstone",
+ "age": 32,
+ "colors": ["red", "green"],
+ "bogus": "WHATEVER",
+ }
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ full_name = SchemaField("full_name", "STRING", mode="REQUIRED")
+ age = SchemaField("age", "INTEGER", mode="REQUIRED")
+ colors = SchemaField("colors", "DATETIME", mode="REPEATED")
+ bogus = SchemaField("joined", "STRING", mode="BOGUS")
+ table = Table(table_ref, schema=[full_name, age, colors, bogus])
+
+ with self.assertRaises(ValueError) as exc:
+ self._call_fut(MAPPING, table.schema)
+
+ self.assertIn("Unknown field mode: BOGUS", str(exc.exception))
+
+ def test__row_from_mapping_w_schema(self):
+ from google.cloud.bigquery.table import Table, SchemaField
+
+ MAPPING = {
+ "full_name": "Phred Phlyntstone",
+ "age": 32,
+ "colors": ["red", "green"],
+ "extra": "IGNORED",
+ }
+ dataset = DatasetReference(self.PROJECT, self.DS_ID)
+ table_ref = dataset.table(self.TABLE_NAME)
+ full_name = SchemaField("full_name", "STRING", mode="REQUIRED")
+ age = SchemaField("age", "INTEGER", mode="REQUIRED")
+ colors = SchemaField("colors", "DATETIME", mode="REPEATED")
+ joined = SchemaField("joined", "STRING", mode="NULLABLE")
+ table = Table(table_ref, schema=[full_name, age, colors, joined])
+
+ self.assertEqual(
+ self._call_fut(MAPPING, table.schema),
+ ("Phred Phlyntstone", 32, ["red", "green"], None),
+ )
+
+
+class TestTableListItem(unittest.TestCase):
+ @staticmethod
+ def _get_target_class():
+ from google.cloud.bigquery.table import TableListItem
+
+ return TableListItem
+
+ def _make_one(self, *args, **kw):
+ return self._get_target_class()(*args, **kw)
+
+ def _setUpConstants(self):
+ import datetime
+ from google.cloud._helpers import UTC
+
+ self.WHEN_TS = 1437767599.125
+ self.WHEN = datetime.datetime.utcfromtimestamp(self.WHEN_TS).replace(tzinfo=UTC)
+ self.EXP_TIME = datetime.datetime(2015, 8, 1, 23, 59, 59, tzinfo=UTC)
+
+ def test_ctor(self):
+ from google.cloud._helpers import _millis
+
+ self._setUpConstants()
+ project = "test-project"
+ dataset_id = "test_dataset"
+ table_id = "coffee_table"
+ resource = {
+ "creationTime": self.WHEN_TS * 1000,
+ "expirationTime": _millis(self.EXP_TIME),
+ "kind": "bigquery#table",
+ "id": "{}:{}.{}".format(project, dataset_id, table_id),
+ "tableReference": {
+ "projectId": project,
+ "datasetId": dataset_id,
+ "tableId": table_id,
+ },
+ "friendlyName": "Mahogany Coffee Table",
+ "type": "TABLE",
+ "timePartitioning": {
+ "type": "DAY",
+ "field": "mycolumn",
+ "expirationMs": "10000",
+ },
+ "labels": {"some-stuff": "this-is-a-label"},
+ "clustering": {"fields": ["string"]},
+ }
+
+ table = self._make_one(resource)
+
+ self.assertEqual(table.created, self.WHEN)
+ self.assertEqual(table.expires, self.EXP_TIME)
+ self.assertEqual(table.project, project)
+ self.assertEqual(table.dataset_id, dataset_id)
+ self.assertEqual(table.table_id, table_id)
+ self.assertEqual(
+ table.full_table_id, "{}:{}.{}".format(project, dataset_id, table_id)
+ )
+ self.assertEqual(table.reference.project, project)
+ self.assertEqual(table.reference.dataset_id, dataset_id)
+ self.assertEqual(table.reference.table_id, table_id)
+ self.assertEqual(table.friendly_name, "Mahogany Coffee Table")
+ self.assertEqual(table.table_type, "TABLE")
+ self.assertEqual(table.time_partitioning.type_, "DAY")
+ self.assertEqual(table.time_partitioning.expiration_ms, 10000)
+ self.assertEqual(table.time_partitioning.field, "mycolumn")
+ self.assertEqual(table.labels["some-stuff"], "this-is-a-label")
+ self.assertIsNone(table.view_use_legacy_sql)
+ self.assertEqual(table.clustering_fields, ["string"])
+
+ with warnings.catch_warnings(record=True) as warned:
+ self.assertEqual(table.partitioning_type, "DAY")
+ self.assertEqual(table.partition_expiration, 10000)
+
+ self.assertEqual(len(warned), 2)
+ for warning in warned:
+ self.assertIs(warning.category, PendingDeprecationWarning)
+
+ def test_ctor_view(self):
+ project = "test-project"
+ dataset_id = "test_dataset"
+ table_id = "just_looking"
+ resource = {
+ "kind": "bigquery#table",
+ "id": "{}:{}.{}".format(project, dataset_id, table_id),
+ "tableReference": {
+ "projectId": project,
+ "datasetId": dataset_id,
+ "tableId": table_id,
+ },
+ "type": "VIEW",
+ }
+
+ table = self._make_one(resource)
+ self.assertEqual(table.project, project)
+ self.assertEqual(table.dataset_id, dataset_id)
+ self.assertEqual(table.table_id, table_id)
+ self.assertEqual(
+ table.full_table_id, "{}:{}.{}".format(project, dataset_id, table_id)
+ )
+ self.assertEqual(table.reference.project, project)
+ self.assertEqual(table.reference.dataset_id, dataset_id)
+ self.assertEqual(table.reference.table_id, table_id)
+ self.assertEqual(table.table_type, "VIEW")
+ # Server default for useLegacySql is True.
+ self.assertTrue(table.view_use_legacy_sql)
+
+ def test_ctor_missing_properties(self):
+ resource = {
+ "tableReference": {
+ "projectId": "testproject",
+ "datasetId": "testdataset",
+ "tableId": "testtable",
+ }
+ }
+ table = self._make_one(resource)
+ self.assertEqual(table.project, "testproject")
+ self.assertEqual(table.dataset_id, "testdataset")
+ self.assertEqual(table.table_id, "testtable")
+ self.assertIsNone(table.created)
+ self.assertIsNone(table.expires)
+ self.assertIsNone(table.clustering_fields)
+ self.assertIsNone(table.full_table_id)
+ self.assertIsNone(table.friendly_name)
+ self.assertIsNone(table.table_type)
+ self.assertIsNone(table.time_partitioning)
+ self.assertEqual(table.labels, {})
+ self.assertIsNone(table.view_use_legacy_sql)
+
+ with warnings.catch_warnings(record=True) as warned:
+ self.assertIsNone(table.partitioning_type)
+ self.assertIsNone(table.partition_expiration)
+
+ self.assertEqual(len(warned), 2)
+ for warning in warned:
+ self.assertIs(warning.category, PendingDeprecationWarning)
+
+ def test_ctor_wo_project(self):
+ resource = {
+ "tableReference": {"datasetId": "testdataset", "tableId": "testtable"}
+ }
+ with self.assertRaises(ValueError):
+ self._make_one(resource)
+
+ def test_ctor_wo_dataset(self):
+ resource = {
+ "tableReference": {"projectId": "testproject", "tableId": "testtable"}
+ }
+ with self.assertRaises(ValueError):
+ self._make_one(resource)
+
+ def test_ctor_wo_table(self):
+ resource = {
+ "tableReference": {"projectId": "testproject", "datasetId": "testdataset"}
+ }
+ with self.assertRaises(ValueError):
+ self._make_one(resource)
+
+ def test_ctor_wo_reference(self):
+ with self.assertRaises(ValueError):
+ self._make_one({})
+
+ def test_labels_update_in_place(self):
+ resource = {
+ "tableReference": {
+ "projectId": "testproject",
+ "datasetId": "testdataset",
+ "tableId": "testtable",
+ }
+ }
+ table = self._make_one(resource)
+ labels = table.labels
+ labels["foo"] = "bar" # update in place
+ self.assertEqual(table.labels, {"foo": "bar"})
+
+
+class TestRow(unittest.TestCase):
+ def test_row(self):
+ from google.cloud.bigquery.table import Row
+
+ VALUES = (1, 2, 3)
+ row = Row(VALUES, {"a": 0, "b": 1, "c": 2})
+ self.assertEqual(row.a, 1)
+ self.assertEqual(row[1], 2)
+ self.assertEqual(row["c"], 3)
+ self.assertEqual(len(row), 3)
+ self.assertEqual(row.values(), VALUES)
+ self.assertEqual(set(row.keys()), set({"a": 1, "b": 2, "c": 3}.keys()))
+ self.assertEqual(set(row.items()), set({"a": 1, "b": 2, "c": 3}.items()))
+ self.assertEqual(row.get("a"), 1)
+ self.assertEqual(row.get("d"), None)
+ self.assertEqual(row.get("d", ""), "")
+ self.assertEqual(row.get("d", default=""), "")
+ self.assertEqual(repr(row), "Row((1, 2, 3), {'a': 0, 'b': 1, 'c': 2})")
+ self.assertFalse(row != row)
+ self.assertFalse(row == 3)
+ with self.assertRaises(AttributeError):
+ row.z
+ with self.assertRaises(KeyError):
+ row["z"]
+
+
+class Test_EmptyRowIterator(unittest.TestCase):
+ def _make_one(self):
+ from google.cloud.bigquery.table import _EmptyRowIterator
+
+ return _EmptyRowIterator()
+
+ def test_total_rows_eq_zero(self):
+ row_iterator = self._make_one()
+ self.assertEqual(row_iterator.total_rows, 0)
+
+ @mock.patch("google.cloud.bigquery.table.pyarrow", new=None)
+ def test_to_arrow_error_if_pyarrow_is_none(self):
+ row_iterator = self._make_one()
+ with self.assertRaises(ValueError):
+ row_iterator.to_arrow()
+
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ def test_to_arrow(self):
+ row_iterator = self._make_one()
+ tbl = row_iterator.to_arrow()
+ self.assertIsInstance(tbl, pyarrow.Table)
+ self.assertEqual(tbl.num_rows, 0)
+
+ @mock.patch("google.cloud.bigquery.table.pandas", new=None)
+ def test_to_dataframe_error_if_pandas_is_none(self):
+ row_iterator = self._make_one()
+ with self.assertRaises(ValueError):
+ row_iterator.to_dataframe()
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ def test_to_dataframe(self):
+ row_iterator = self._make_one()
+ df = row_iterator.to_dataframe()
+ self.assertIsInstance(df, pandas.DataFrame)
+ self.assertEqual(len(df), 0) # verify the number of rows
+
+
+class TestRowIterator(unittest.TestCase):
+ def _class_under_test(self):
+ from google.cloud.bigquery.table import RowIterator
+
+ return RowIterator
+
+ def _make_one(
+ self,
+ client=None,
+ api_request=None,
+ path=None,
+ schema=None,
+ table=None,
+ **kwargs
+ ):
+ from google.cloud.bigquery.table import TableReference
+
+ if client is None:
+ client = _mock_client()
+
+ if api_request is None:
+ api_request = mock.sentinel.api_request
+
+ if path is None:
+ path = "/foo"
+
+ if schema is None:
+ schema = []
+
+ if table is None:
+ table = TableReference.from_string("my-project.my_dataset.my_table")
+
+ return self._class_under_test()(
+ client, api_request, path, schema, table=table, **kwargs
+ )
+
+ def test_constructor(self):
+ from google.cloud.bigquery.table import _item_to_row
+ from google.cloud.bigquery.table import _rows_page_start
+
+ client = _mock_client()
+ path = "/some/path"
+ iterator = self._make_one(client=client, path=path)
+
+ # Objects are set without copying.
+ self.assertIs(iterator.client, client)
+ self.assertIs(iterator.item_to_value, _item_to_row)
+ self.assertIs(iterator._page_start, _rows_page_start)
+ # Properties have the expect value.
+ self.assertEqual(iterator.extra_params, {})
+ self.assertEqual(iterator._items_key, "rows")
+ self.assertIsNone(iterator.max_results)
+ self.assertEqual(iterator.path, path)
+ self.assertFalse(iterator._started)
+ self.assertIsNone(iterator.total_rows)
+ # Changing attributes.
+ self.assertEqual(iterator.page_number, 0)
+ self.assertIsNone(iterator.next_page_token)
+ self.assertEqual(iterator.num_results, 0)
+
+ def test_constructor_with_table(self):
+ from google.cloud.bigquery.table import Table
+
+ table = Table("proj.dset.tbl")
+ table._properties["numRows"] = 100
+
+ iterator = self._make_one(table=table)
+
+ self.assertIs(iterator._table, table)
+ self.assertEqual(iterator.total_rows, 100)
+
+ def test_iterate(self):
+ from google.cloud.bigquery.table import SchemaField
+
+ schema = [
+ SchemaField("name", "STRING", mode="REQUIRED"),
+ SchemaField("age", "INTEGER", mode="REQUIRED"),
+ ]
+ rows = [
+ {"f": [{"v": "Phred Phlyntstone"}, {"v": "32"}]},
+ {"f": [{"v": "Bharney Rhubble"}, {"v": "33"}]},
+ ]
+ path = "/foo"
+ api_request = mock.Mock(return_value={"rows": rows})
+ row_iterator = self._make_one(_mock_client(), api_request, path, schema)
+ self.assertEqual(row_iterator.num_results, 0)
+
+ rows_iter = iter(row_iterator)
+
+ val1 = six.next(rows_iter)
+ self.assertEqual(val1.name, "Phred Phlyntstone")
+ self.assertEqual(row_iterator.num_results, 1)
+
+ val2 = six.next(rows_iter)
+ self.assertEqual(val2.name, "Bharney Rhubble")
+ self.assertEqual(row_iterator.num_results, 2)
+
+ with self.assertRaises(StopIteration):
+ six.next(rows_iter)
+
+ api_request.assert_called_once_with(method="GET", path=path, query_params={})
+
+ def test_page_size(self):
+ from google.cloud.bigquery.table import SchemaField
+
+ schema = [
+ SchemaField("name", "STRING", mode="REQUIRED"),
+ SchemaField("age", "INTEGER", mode="REQUIRED"),
+ ]
+ rows = [
+ {"f": [{"v": "Phred Phlyntstone"}, {"v": "32"}]},
+ {"f": [{"v": "Bharney Rhubble"}, {"v": "33"}]},
+ ]
+ path = "/foo"
+ api_request = mock.Mock(return_value={"rows": rows})
+
+ row_iterator = self._make_one(
+ _mock_client(), api_request, path, schema, page_size=4
+ )
+ row_iterator._get_next_page_response()
+
+ api_request.assert_called_once_with(
+ method="GET",
+ path=path,
+ query_params={"maxResults": row_iterator._page_size},
+ )
+
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ def test_to_arrow(self):
+ from google.cloud.bigquery.table import SchemaField
+
+ schema = [
+ SchemaField("name", "STRING", mode="REQUIRED"),
+ SchemaField("age", "INTEGER", mode="REQUIRED"),
+ SchemaField(
+ "child",
+ "RECORD",
+ mode="REPEATED",
+ fields=[
+ SchemaField("name", "STRING", mode="REQUIRED"),
+ SchemaField("age", "INTEGER", mode="REQUIRED"),
+ ],
+ ),
+ ]
+ rows = [
+ {
+ "f": [
+ {"v": "Bharney Rhubble"},
+ {"v": "33"},
+ {
+ "v": [
+ {"v": {"f": [{"v": "Whamm-Whamm Rhubble"}, {"v": "3"}]}},
+ {"v": {"f": [{"v": "Hoppy"}, {"v": "1"}]}},
+ ]
+ },
+ ]
+ },
+ {
+ "f": [
+ {"v": "Wylma Phlyntstone"},
+ {"v": "29"},
+ {
+ "v": [
+ {"v": {"f": [{"v": "Bepples Phlyntstone"}, {"v": "0"}]}},
+ {"v": {"f": [{"v": "Dino"}, {"v": "4"}]}},
+ ]
+ },
+ ]
+ },
+ ]
+ path = "/foo"
+ api_request = mock.Mock(return_value={"rows": rows})
+ row_iterator = self._make_one(_mock_client(), api_request, path, schema)
+
+ tbl = row_iterator.to_arrow()
+
+ self.assertIsInstance(tbl, pyarrow.Table)
+ self.assertEqual(tbl.num_rows, 2)
+
+ # Check the schema.
+ self.assertEqual(tbl.schema[0].name, "name")
+ self.assertTrue(pyarrow.types.is_string(tbl.schema[0].type))
+ self.assertEqual(tbl.schema[1].name, "age")
+ self.assertTrue(pyarrow.types.is_int64(tbl.schema[1].type))
+ child_field = tbl.schema[2]
+ self.assertEqual(child_field.name, "child")
+ self.assertTrue(pyarrow.types.is_list(child_field.type))
+ self.assertTrue(pyarrow.types.is_struct(child_field.type.value_type))
+ self.assertEqual(child_field.type.value_type[0].name, "name")
+ self.assertEqual(child_field.type.value_type[1].name, "age")
+
+ # Check the data.
+ tbl_data = tbl.to_pydict()
+ names = tbl_data["name"]
+ ages = tbl_data["age"]
+ children = tbl_data["child"]
+ self.assertEqual(names, ["Bharney Rhubble", "Wylma Phlyntstone"])
+ self.assertEqual(ages, [33, 29])
+ self.assertEqual(
+ children,
+ [
+ [
+ {"name": "Whamm-Whamm Rhubble", "age": 3},
+ {"name": "Hoppy", "age": 1},
+ ],
+ [{"name": "Bepples Phlyntstone", "age": 0}, {"name": "Dino", "age": 4}],
+ ],
+ )
+
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ def test_to_arrow_w_nulls(self):
+ from google.cloud.bigquery.table import SchemaField
+
+ schema = [SchemaField("name", "STRING"), SchemaField("age", "INTEGER")]
+ rows = [
+ {"f": [{"v": "Donkey"}, {"v": 32}]},
+ {"f": [{"v": "Diddy"}, {"v": 29}]},
+ {"f": [{"v": "Dixie"}, {"v": None}]},
+ {"f": [{"v": None}, {"v": 111}]},
+ ]
+ path = "/foo"
+ api_request = mock.Mock(return_value={"rows": rows})
+ row_iterator = self._make_one(_mock_client(), api_request, path, schema)
+
+ tbl = row_iterator.to_arrow()
+
+ self.assertIsInstance(tbl, pyarrow.Table)
+ self.assertEqual(tbl.num_rows, 4)
+
+ # Check the schema.
+ self.assertEqual(tbl.schema[0].name, "name")
+ self.assertTrue(pyarrow.types.is_string(tbl.schema[0].type))
+ self.assertEqual(tbl.schema[1].name, "age")
+ self.assertTrue(pyarrow.types.is_int64(tbl.schema[1].type))
+
+ # Check the data.
+ tbl_data = tbl.to_pydict()
+ names = tbl_data["name"]
+ ages = tbl_data["age"]
+ self.assertEqual(names, ["Donkey", "Diddy", "Dixie", None])
+ self.assertEqual(ages, [32, 29, None, 111])
+
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ def test_to_arrow_w_unknown_type(self):
+ from google.cloud.bigquery.table import SchemaField
+
+ schema = [
+ SchemaField("name", "STRING", mode="REQUIRED"),
+ SchemaField("age", "INTEGER", mode="REQUIRED"),
+ SchemaField("sport", "UNKNOWN_TYPE", mode="REQUIRED"),
+ ]
+ rows = [
+ {"f": [{"v": "Bharney Rhubble"}, {"v": "33"}, {"v": "volleyball"}]},
+ {"f": [{"v": "Wylma Phlyntstone"}, {"v": "29"}, {"v": "basketball"}]},
+ ]
+ path = "/foo"
+ api_request = mock.Mock(return_value={"rows": rows})
+ row_iterator = self._make_one(_mock_client(), api_request, path, schema)
+
+ tbl = row_iterator.to_arrow()
+
+ self.assertIsInstance(tbl, pyarrow.Table)
+ self.assertEqual(tbl.num_rows, 2)
+
+ # Check the schema.
+ self.assertEqual(tbl.schema[0].name, "name")
+ self.assertTrue(pyarrow.types.is_string(tbl.schema[0].type))
+ self.assertEqual(tbl.schema[1].name, "age")
+ self.assertTrue(pyarrow.types.is_int64(tbl.schema[1].type))
+ self.assertEqual(tbl.schema[2].name, "sport")
+
+ # Check the data.
+ tbl_data = tbl.to_pydict()
+ names = tbl_data["name"]
+ ages = tbl_data["age"]
+ sports = tbl_data["sport"]
+ self.assertEqual(names, ["Bharney Rhubble", "Wylma Phlyntstone"])
+ self.assertEqual(ages, [33, 29])
+ self.assertEqual(sports, ["volleyball", "basketball"])
+
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ def test_to_arrow_w_empty_table(self):
+ from google.cloud.bigquery.table import SchemaField
+
+ schema = [
+ SchemaField("name", "STRING", mode="REQUIRED"),
+ SchemaField("age", "INTEGER", mode="REQUIRED"),
+ SchemaField(
+ "child",
+ "RECORD",
+ mode="REPEATED",
+ fields=[
+ SchemaField("name", "STRING", mode="REQUIRED"),
+ SchemaField("age", "INTEGER", mode="REQUIRED"),
+ ],
+ ),
+ ]
+ rows = []
+ path = "/foo"
+ api_request = mock.Mock(return_value={"rows": rows})
+ row_iterator = self._make_one(_mock_client(), api_request, path, schema)
+
+ tbl = row_iterator.to_arrow()
+
+ self.assertIsInstance(tbl, pyarrow.Table)
+ self.assertEqual(tbl.num_rows, 0)
+
+ # Check the schema.
+ self.assertEqual(tbl.schema[0].name, "name")
+ self.assertTrue(pyarrow.types.is_string(tbl.schema[0].type))
+ self.assertEqual(tbl.schema[1].name, "age")
+ self.assertTrue(pyarrow.types.is_int64(tbl.schema[1].type))
+ child_field = tbl.schema[2]
+ self.assertEqual(child_field.name, "child")
+ self.assertTrue(pyarrow.types.is_list(child_field.type))
+ self.assertTrue(pyarrow.types.is_struct(child_field.type.value_type))
+ self.assertEqual(child_field.type.value_type[0].name, "name")
+ self.assertEqual(child_field.type.value_type[1].name, "age")
+
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ @unittest.skipIf(
+ bigquery_storage_v1beta1 is None, "Requires `google-cloud-bigquery-storage`"
+ )
+ def test_to_arrow_w_bqstorage(self):
+ from google.cloud.bigquery import schema
+ from google.cloud.bigquery import table as mut
+ from google.cloud.bigquery_storage_v1beta1 import reader
+
+ bqstorage_client = mock.create_autospec(
+ bigquery_storage_v1beta1.BigQueryStorageClient
+ )
+ streams = [
+ # Use two streams we want to check frames are read from each stream.
+ {"name": "/projects/proj/dataset/dset/tables/tbl/streams/1234"},
+ {"name": "/projects/proj/dataset/dset/tables/tbl/streams/5678"},
+ ]
+ session = bigquery_storage_v1beta1.types.ReadSession(streams=streams)
+ arrow_schema = pyarrow.schema(
+ [
+ pyarrow.field("colA", pyarrow.int64()),
+ # Not alphabetical to test column order.
+ pyarrow.field("colC", pyarrow.float64()),
+ pyarrow.field("colB", pyarrow.string()),
+ ]
+ )
+ session.arrow_schema.serialized_schema = arrow_schema.serialize().to_pybytes()
+ bqstorage_client.create_read_session.return_value = session
+
+ mock_rowstream = mock.create_autospec(reader.ReadRowsStream)
+ bqstorage_client.read_rows.return_value = mock_rowstream
+
+ mock_rows = mock.create_autospec(reader.ReadRowsIterable)
+ mock_rowstream.rows.return_value = mock_rows
+ expected_num_rows = 2
+ expected_num_columns = 3
+ page_items = [
+ pyarrow.array([1, -1]),
+ pyarrow.array([2.0, 4.0]),
+ pyarrow.array(["abc", "def"]),
+ ]
+
+ mock_page = mock.create_autospec(reader.ReadRowsPage)
+ mock_page.to_arrow.return_value = pyarrow.RecordBatch.from_arrays(
+ page_items, arrow_schema
+ )
+ mock_pages = (mock_page, mock_page, mock_page)
+ type(mock_rows).pages = mock.PropertyMock(return_value=mock_pages)
+
+ schema = [
+ schema.SchemaField("colA", "INTEGER"),
+ schema.SchemaField("colC", "FLOAT"),
+ schema.SchemaField("colB", "STRING"),
+ ]
+
+ row_iterator = mut.RowIterator(
+ _mock_client(),
+ None, # api_request: ignored
+ None, # path: ignored
+ schema,
+ table=mut.TableReference.from_string("proj.dset.tbl"),
+ selected_fields=schema,
+ )
+
+ actual_tbl = row_iterator.to_arrow(bqstorage_client=bqstorage_client)
+
+ # Are the columns in the expected order?
+ self.assertEqual(actual_tbl.num_columns, expected_num_columns)
+ self.assertEqual(actual_tbl.schema[0].name, "colA")
+ self.assertEqual(actual_tbl.schema[1].name, "colC")
+ self.assertEqual(actual_tbl.schema[2].name, "colB")
+
+ # Have expected number of rows?
+ total_pages = len(streams) * len(mock_pages)
+ total_rows = expected_num_rows * total_pages
+ self.assertEqual(actual_tbl.num_rows, total_rows)
+
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ @unittest.skipIf(
+ bigquery_storage_v1beta1 is None, "Requires `google-cloud-bigquery-storage`"
+ )
+ def test_to_arrow_w_bqstorage_no_streams(self):
+ from google.cloud.bigquery import schema
+ from google.cloud.bigquery import table as mut
+
+ bqstorage_client = mock.create_autospec(
+ bigquery_storage_v1beta1.BigQueryStorageClient
+ )
+ session = bigquery_storage_v1beta1.types.ReadSession()
+ arrow_schema = pyarrow.schema(
+ [
+ pyarrow.field("colA", pyarrow.string()),
+ # Not alphabetical to test column order.
+ pyarrow.field("colC", pyarrow.string()),
+ pyarrow.field("colB", pyarrow.string()),
+ ]
+ )
+ session.arrow_schema.serialized_schema = arrow_schema.serialize().to_pybytes()
+ bqstorage_client.create_read_session.return_value = session
+
+ row_iterator = mut.RowIterator(
+ _mock_client(),
+ None, # api_request: ignored
+ None, # path: ignored
+ [
+ schema.SchemaField("colA", "STRING"),
+ schema.SchemaField("colC", "STRING"),
+ schema.SchemaField("colB", "STRING"),
+ ],
+ table=mut.TableReference.from_string("proj.dset.tbl"),
+ )
+
+ actual_table = row_iterator.to_arrow(bqstorage_client=bqstorage_client)
+ self.assertEqual(actual_table.num_columns, 3)
+ self.assertEqual(actual_table.num_rows, 0)
+ self.assertEqual(actual_table.schema[0].name, "colA")
+ self.assertEqual(actual_table.schema[1].name, "colC")
+ self.assertEqual(actual_table.schema[2].name, "colB")
+
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ @unittest.skipIf(tqdm is None, "Requires `tqdm`")
+ @mock.patch("tqdm.tqdm_gui")
+ @mock.patch("tqdm.tqdm_notebook")
+ @mock.patch("tqdm.tqdm")
+ def test_to_arrow_progress_bar(self, tqdm_mock, tqdm_notebook_mock, tqdm_gui_mock):
+ from google.cloud.bigquery.table import SchemaField
+
+ schema = [
+ SchemaField("name", "STRING", mode="REQUIRED"),
+ SchemaField("age", "INTEGER", mode="REQUIRED"),
+ ]
+ rows = [
+ {"f": [{"v": "Phred Phlyntstone"}, {"v": "32"}]},
+ {"f": [{"v": "Bharney Rhubble"}, {"v": "33"}]},
+ {"f": [{"v": "Wylma Phlyntstone"}, {"v": "29"}]},
+ {"f": [{"v": "Bhettye Rhubble"}, {"v": "27"}]},
+ ]
+ path = "/foo"
+ api_request = mock.Mock(return_value={"rows": rows})
+
+ progress_bars = (
+ ("tqdm", tqdm_mock),
+ ("tqdm_notebook", tqdm_notebook_mock),
+ ("tqdm_gui", tqdm_gui_mock),
+ )
+
+ for progress_bar_type, progress_bar_mock in progress_bars:
+ row_iterator = self._make_one(_mock_client(), api_request, path, schema)
+ tbl = row_iterator.to_arrow(progress_bar_type=progress_bar_type)
+
+ progress_bar_mock.assert_called()
+ progress_bar_mock().update.assert_called()
+ progress_bar_mock().close.assert_called_once()
+ self.assertEqual(tbl.num_rows, 4)
+
+ @mock.patch("google.cloud.bigquery.table.pyarrow", new=None)
+ def test_to_arrow_w_pyarrow_none(self):
+ schema = []
+ rows = []
+ path = "/foo"
+ api_request = mock.Mock(return_value={"rows": rows})
+ row_iterator = self._make_one(_mock_client(), api_request, path, schema)
+
+ with self.assertRaises(ValueError):
+ row_iterator.to_arrow()
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ def test_to_dataframe(self):
+ from google.cloud.bigquery.table import SchemaField
+
+ schema = [
+ SchemaField("name", "STRING", mode="REQUIRED"),
+ SchemaField("age", "INTEGER", mode="REQUIRED"),
+ ]
+ rows = [
+ {"f": [{"v": "Phred Phlyntstone"}, {"v": "32"}]},
+ {"f": [{"v": "Bharney Rhubble"}, {"v": "33"}]},
+ {"f": [{"v": "Wylma Phlyntstone"}, {"v": "29"}]},
+ {"f": [{"v": "Bhettye Rhubble"}, {"v": "27"}]},
+ ]
+ path = "/foo"
+ api_request = mock.Mock(return_value={"rows": rows})
+ row_iterator = self._make_one(_mock_client(), api_request, path, schema)
+
+ df = row_iterator.to_dataframe()
+
+ self.assertIsInstance(df, pandas.DataFrame)
+ self.assertEqual(len(df), 4) # verify the number of rows
+ self.assertEqual(list(df), ["name", "age"]) # verify the column names
+ self.assertEqual(df.name.dtype.name, "object")
+ self.assertEqual(df.age.dtype.name, "int64")
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(tqdm is None, "Requires `tqdm`")
+ @mock.patch("tqdm.tqdm_gui")
+ @mock.patch("tqdm.tqdm_notebook")
+ @mock.patch("tqdm.tqdm")
+ def test_to_dataframe_progress_bar(
+ self, tqdm_mock, tqdm_notebook_mock, tqdm_gui_mock
+ ):
+ from google.cloud.bigquery.table import SchemaField
+
+ schema = [
+ SchemaField("name", "STRING", mode="REQUIRED"),
+ SchemaField("age", "INTEGER", mode="REQUIRED"),
+ ]
+ rows = [
+ {"f": [{"v": "Phred Phlyntstone"}, {"v": "32"}]},
+ {"f": [{"v": "Bharney Rhubble"}, {"v": "33"}]},
+ {"f": [{"v": "Wylma Phlyntstone"}, {"v": "29"}]},
+ {"f": [{"v": "Bhettye Rhubble"}, {"v": "27"}]},
+ ]
+ path = "/foo"
+ api_request = mock.Mock(return_value={"rows": rows})
+
+ progress_bars = (
+ ("tqdm", tqdm_mock),
+ ("tqdm_notebook", tqdm_notebook_mock),
+ ("tqdm_gui", tqdm_gui_mock),
+ )
+
+ for progress_bar_type, progress_bar_mock in progress_bars:
+ row_iterator = self._make_one(_mock_client(), api_request, path, schema)
+ df = row_iterator.to_dataframe(progress_bar_type=progress_bar_type)
+
+ progress_bar_mock.assert_called()
+ progress_bar_mock().update.assert_called()
+ progress_bar_mock().close.assert_called_once()
+ self.assertEqual(len(df), 4)
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @mock.patch("google.cloud.bigquery.table.tqdm", new=None)
+ def test_to_dataframe_no_tqdm_no_progress_bar(self):
+ from google.cloud.bigquery.table import SchemaField
+
+ schema = [
+ SchemaField("name", "STRING", mode="REQUIRED"),
+ SchemaField("age", "INTEGER", mode="REQUIRED"),
+ ]
+ rows = [
+ {"f": [{"v": "Phred Phlyntstone"}, {"v": "32"}]},
+ {"f": [{"v": "Bharney Rhubble"}, {"v": "33"}]},
+ {"f": [{"v": "Wylma Phlyntstone"}, {"v": "29"}]},
+ {"f": [{"v": "Bhettye Rhubble"}, {"v": "27"}]},
+ ]
+ path = "/foo"
+ api_request = mock.Mock(return_value={"rows": rows})
+ row_iterator = self._make_one(_mock_client(), api_request, path, schema)
+
+ with warnings.catch_warnings(record=True) as warned:
+ df = row_iterator.to_dataframe()
+
+ self.assertEqual(len(warned), 0)
+ self.assertEqual(len(df), 4)
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @mock.patch("google.cloud.bigquery.table.tqdm", new=None)
+ def test_to_dataframe_no_tqdm(self):
+ from google.cloud.bigquery.table import SchemaField
+
+ schema = [
+ SchemaField("name", "STRING", mode="REQUIRED"),
+ SchemaField("age", "INTEGER", mode="REQUIRED"),
+ ]
+ rows = [
+ {"f": [{"v": "Phred Phlyntstone"}, {"v": "32"}]},
+ {"f": [{"v": "Bharney Rhubble"}, {"v": "33"}]},
+ {"f": [{"v": "Wylma Phlyntstone"}, {"v": "29"}]},
+ {"f": [{"v": "Bhettye Rhubble"}, {"v": "27"}]},
+ ]
+ path = "/foo"
+ api_request = mock.Mock(return_value={"rows": rows})
+ row_iterator = self._make_one(_mock_client(), api_request, path, schema)
+
+ with warnings.catch_warnings(record=True) as warned:
+ df = row_iterator.to_dataframe(progress_bar_type="tqdm")
+
+ self.assertEqual(len(warned), 1)
+ for warning in warned:
+ self.assertIs(warning.category, UserWarning)
+
+ # Even though the progress bar won't show, downloading the dataframe
+ # should still work.
+ self.assertEqual(len(df), 4)
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(tqdm is None, "Requires `tqdm`")
+ @mock.patch("tqdm.tqdm_gui", new=None) # will raise TypeError on call
+ @mock.patch("tqdm.tqdm_notebook", new=None) # will raise TypeError on call
+ @mock.patch("tqdm.tqdm", new=None) # will raise TypeError on call
+ def test_to_dataframe_tqdm_error(self):
+ from google.cloud.bigquery.table import SchemaField
+
+ schema = [
+ SchemaField("name", "STRING", mode="REQUIRED"),
+ SchemaField("age", "INTEGER", mode="REQUIRED"),
+ ]
+ rows = [
+ {"f": [{"v": "Phred Phlyntstone"}, {"v": "32"}]},
+ {"f": [{"v": "Bharney Rhubble"}, {"v": "33"}]},
+ {"f": [{"v": "Wylma Phlyntstone"}, {"v": "29"}]},
+ {"f": [{"v": "Bhettye Rhubble"}, {"v": "27"}]},
+ ]
+ path = "/foo"
+
+ for progress_bar_type in ("tqdm", "tqdm_notebook", "tqdm_gui"):
+ api_request = mock.Mock(return_value={"rows": rows})
+ row_iterator = self._make_one(_mock_client(), api_request, path, schema)
+
+ with warnings.catch_warnings(record=True) as warned:
+ df = row_iterator.to_dataframe(progress_bar_type=progress_bar_type)
+
+ self.assertEqual(len(df), 4) # all should be well
+
+ # Warn that a progress bar was requested, but creating the tqdm
+ # progress bar failed.
+ for warning in warned:
+ self.assertIs(warning.category, UserWarning)
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ def test_to_dataframe_w_empty_results(self):
+ from google.cloud.bigquery.table import SchemaField
+
+ schema = [
+ SchemaField("name", "STRING", mode="REQUIRED"),
+ SchemaField("age", "INTEGER", mode="REQUIRED"),
+ ]
+ api_request = mock.Mock(return_value={"rows": []})
+ row_iterator = self._make_one(_mock_client(), api_request, schema=schema)
+
+ df = row_iterator.to_dataframe()
+
+ self.assertIsInstance(df, pandas.DataFrame)
+ self.assertEqual(len(df), 0) # verify the number of rows
+ self.assertEqual(list(df), ["name", "age"]) # verify the column names
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ def test_to_dataframe_logs_tabledata_list(self):
+ from google.cloud.bigquery.table import Table
+
+ mock_logger = mock.create_autospec(logging.Logger)
+ api_request = mock.Mock(return_value={"rows": []})
+ row_iterator = self._make_one(
+ _mock_client(), api_request, table=Table("debug-proj.debug_dset.debug_tbl")
+ )
+
+ with mock.patch("google.cloud.bigquery.table._LOGGER", mock_logger):
+ row_iterator.to_dataframe()
+
+ mock_logger.debug.assert_any_call(
+ "Started reading table 'debug-proj.debug_dset.debug_tbl' with tabledata.list."
+ )
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ def test_to_dataframe_w_various_types_nullable(self):
+ import datetime
+ from google.cloud.bigquery.table import SchemaField
+
+ schema = [
+ SchemaField("start_timestamp", "TIMESTAMP"),
+ SchemaField("seconds", "INT64"),
+ SchemaField("miles", "FLOAT64"),
+ SchemaField("payment_type", "STRING"),
+ SchemaField("complete", "BOOL"),
+ SchemaField("date", "DATE"),
+ ]
+ row_data = [
+ [None, None, None, None, None, None],
+ ["1.4338368E9", "420", "1.1", "Cash", "true", "1999-12-01"],
+ ["1.3878117E9", "2580", "17.7", "Cash", "false", "1953-06-14"],
+ ["1.3855653E9", "2280", "4.4", "Credit", "true", "1981-11-04"],
+ ]
+ rows = [{"f": [{"v": field} for field in row]} for row in row_data]
+ path = "/foo"
+ api_request = mock.Mock(return_value={"rows": rows})
+ row_iterator = self._make_one(_mock_client(), api_request, path, schema)
+
+ df = row_iterator.to_dataframe()
+
+ self.assertIsInstance(df, pandas.DataFrame)
+ self.assertEqual(len(df), 4) # verify the number of rows
+ exp_columns = [field.name for field in schema]
+ self.assertEqual(list(df), exp_columns) # verify the column names
+
+ for index, row in df.iterrows():
+ if index == 0:
+ self.assertTrue(row.isnull().all())
+ else:
+ self.assertIsInstance(row.start_timestamp, pandas.Timestamp)
+ self.assertIsInstance(row.seconds, float)
+ self.assertIsInstance(row.payment_type, str)
+ self.assertIsInstance(row.complete, bool)
+ self.assertIsInstance(row.date, datetime.date)
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ def test_to_dataframe_column_dtypes(self):
+ from google.cloud.bigquery.table import SchemaField
+
+ schema = [
+ SchemaField("start_timestamp", "TIMESTAMP"),
+ SchemaField("seconds", "INT64"),
+ SchemaField("miles", "FLOAT64"),
+ SchemaField("km", "FLOAT64"),
+ SchemaField("payment_type", "STRING"),
+ SchemaField("complete", "BOOL"),
+ SchemaField("date", "DATE"),
+ ]
+ row_data = [
+ ["1.4338368E9", "420", "1.1", "1.77", "Cash", "true", "1999-12-01"],
+ ["1.3878117E9", "2580", "17.7", "28.5", "Cash", "false", "1953-06-14"],
+ ["1.3855653E9", "2280", "4.4", "7.1", "Credit", "true", "1981-11-04"],
+ ]
+ rows = [{"f": [{"v": field} for field in row]} for row in row_data]
+ path = "/foo"
+ api_request = mock.Mock(return_value={"rows": rows})
+ row_iterator = self._make_one(_mock_client(), api_request, path, schema)
+
+ df = row_iterator.to_dataframe(dtypes={"km": "float16"})
+
+ self.assertIsInstance(df, pandas.DataFrame)
+ self.assertEqual(len(df), 3) # verify the number of rows
+ exp_columns = [field.name for field in schema]
+ self.assertEqual(list(df), exp_columns) # verify the column names
+
+ self.assertEqual(df.start_timestamp.dtype.name, "datetime64[ns, UTC]")
+ self.assertEqual(df.seconds.dtype.name, "int64")
+ self.assertEqual(df.miles.dtype.name, "float64")
+ self.assertEqual(df.km.dtype.name, "float16")
+ self.assertEqual(df.payment_type.dtype.name, "object")
+ self.assertEqual(df.complete.dtype.name, "bool")
+ self.assertEqual(df.date.dtype.name, "object")
+
+ @mock.patch("google.cloud.bigquery.table.pandas", new=None)
+ def test_to_dataframe_error_if_pandas_is_none(self):
+ from google.cloud.bigquery.table import SchemaField
+
+ schema = [
+ SchemaField("name", "STRING", mode="REQUIRED"),
+ SchemaField("age", "INTEGER", mode="REQUIRED"),
+ ]
+ rows = [
+ {"f": [{"v": "Phred Phlyntstone"}, {"v": "32"}]},
+ {"f": [{"v": "Bharney Rhubble"}, {"v": "33"}]},
+ ]
+ path = "/foo"
+ api_request = mock.Mock(return_value={"rows": rows})
+ row_iterator = self._make_one(_mock_client(), api_request, path, schema)
+
+ with self.assertRaises(ValueError):
+ row_iterator.to_dataframe()
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(
+ bigquery_storage_v1beta1 is None, "Requires `google-cloud-bigquery-storage`"
+ )
+ def test_to_dataframe_w_bqstorage_no_streams(self):
+ from google.cloud.bigquery import schema
+ from google.cloud.bigquery import table as mut
+
+ bqstorage_client = mock.create_autospec(
+ bigquery_storage_v1beta1.BigQueryStorageClient
+ )
+ session = bigquery_storage_v1beta1.types.ReadSession()
+ bqstorage_client.create_read_session.return_value = session
+
+ row_iterator = mut.RowIterator(
+ _mock_client(),
+ api_request=None,
+ path=None,
+ schema=[
+ schema.SchemaField("colA", "IGNORED"),
+ schema.SchemaField("colC", "IGNORED"),
+ schema.SchemaField("colB", "IGNORED"),
+ ],
+ table=mut.TableReference.from_string("proj.dset.tbl"),
+ )
+
+ got = row_iterator.to_dataframe(bqstorage_client)
+ column_names = ["colA", "colC", "colB"]
+ self.assertEqual(list(got), column_names)
+ self.assertTrue(got.empty)
+
+ @unittest.skipIf(
+ bigquery_storage_v1beta1 is None, "Requires `google-cloud-bigquery-storage`"
+ )
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ def test_to_dataframe_w_bqstorage_logs_session(self):
+ from google.cloud.bigquery.table import Table
+
+ bqstorage_client = mock.create_autospec(
+ bigquery_storage_v1beta1.BigQueryStorageClient
+ )
+ session = bigquery_storage_v1beta1.types.ReadSession()
+ session.name = "projects/test-proj/locations/us/sessions/SOMESESSION"
+ bqstorage_client.create_read_session.return_value = session
+ mock_logger = mock.create_autospec(logging.Logger)
+ row_iterator = self._make_one(
+ _mock_client(), table=Table("debug-proj.debug_dset.debug_tbl")
+ )
+
+ with mock.patch("google.cloud.bigquery._pandas_helpers._LOGGER", mock_logger):
+ row_iterator.to_dataframe(bqstorage_client=bqstorage_client)
+
+ mock_logger.debug.assert_any_call(
+ "Started reading table 'debug-proj.debug_dset.debug_tbl' "
+ "with BQ Storage API session 'projects/test-proj/locations/us/sessions/SOMESESSION'."
+ )
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(
+ bigquery_storage_v1beta1 is None, "Requires `google-cloud-bigquery-storage`"
+ )
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ def test_to_dataframe_w_bqstorage_empty_streams(self):
+ from google.cloud.bigquery import schema
+ from google.cloud.bigquery import table as mut
+ from google.cloud.bigquery_storage_v1beta1 import reader
+
+ arrow_fields = [
+ pyarrow.field("colA", pyarrow.int64()),
+ # Not alphabetical to test column order.
+ pyarrow.field("colC", pyarrow.float64()),
+ pyarrow.field("colB", pyarrow.utf8()),
+ ]
+ arrow_schema = pyarrow.schema(arrow_fields)
+
+ bqstorage_client = mock.create_autospec(
+ bigquery_storage_v1beta1.BigQueryStorageClient
+ )
+ session = bigquery_storage_v1beta1.types.ReadSession(
+ streams=[{"name": "/projects/proj/dataset/dset/tables/tbl/streams/1234"}],
+ arrow_schema={"serialized_schema": arrow_schema.serialize().to_pybytes()},
+ )
+ bqstorage_client.create_read_session.return_value = session
+
+ mock_rowstream = mock.create_autospec(reader.ReadRowsStream)
+ bqstorage_client.read_rows.return_value = mock_rowstream
+
+ mock_rows = mock.create_autospec(reader.ReadRowsIterable)
+ mock_rowstream.rows.return_value = mock_rows
+ mock_pages = mock.PropertyMock(return_value=())
+ type(mock_rows).pages = mock_pages
+
+ schema = [
+ schema.SchemaField("colA", "IGNORED"),
+ schema.SchemaField("colC", "IGNORED"),
+ schema.SchemaField("colB", "IGNORED"),
+ ]
+
+ row_iterator = mut.RowIterator(
+ _mock_client(),
+ None, # api_request: ignored
+ None, # path: ignored
+ schema,
+ table=mut.TableReference.from_string("proj.dset.tbl"),
+ selected_fields=schema,
+ )
+
+ got = row_iterator.to_dataframe(bqstorage_client)
+
+ column_names = ["colA", "colC", "colB"]
+ self.assertEqual(list(got), column_names)
+ self.assertTrue(got.empty)
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(
+ bigquery_storage_v1beta1 is None, "Requires `google-cloud-bigquery-storage`"
+ )
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ def test_to_dataframe_w_bqstorage_nonempty(self):
+ from google.cloud.bigquery import schema
+ from google.cloud.bigquery import table as mut
+ from google.cloud.bigquery_storage_v1beta1 import reader
+
+ arrow_fields = [
+ pyarrow.field("colA", pyarrow.int64()),
+ # Not alphabetical to test column order.
+ pyarrow.field("colC", pyarrow.float64()),
+ pyarrow.field("colB", pyarrow.utf8()),
+ ]
+ arrow_schema = pyarrow.schema(arrow_fields)
+
+ bqstorage_client = mock.create_autospec(
+ bigquery_storage_v1beta1.BigQueryStorageClient
+ )
+ streams = [
+ # Use two streams we want to check frames are read from each stream.
+ {"name": "/projects/proj/dataset/dset/tables/tbl/streams/1234"},
+ {"name": "/projects/proj/dataset/dset/tables/tbl/streams/5678"},
+ ]
+ session = bigquery_storage_v1beta1.types.ReadSession(
+ streams=streams,
+ arrow_schema={"serialized_schema": arrow_schema.serialize().to_pybytes()},
+ )
+ bqstorage_client.create_read_session.return_value = session
+
+ mock_rowstream = mock.create_autospec(reader.ReadRowsStream)
+ bqstorage_client.read_rows.return_value = mock_rowstream
+
+ mock_rows = mock.create_autospec(reader.ReadRowsIterable)
+ mock_rowstream.rows.return_value = mock_rows
+ page_items = [
+ {"colA": 1, "colB": "abc", "colC": 2.0},
+ {"colA": -1, "colB": "def", "colC": 4.0},
+ ]
+
+ mock_page = mock.create_autospec(reader.ReadRowsPage)
+ mock_page.to_dataframe.return_value = pandas.DataFrame(
+ page_items, columns=["colA", "colB", "colC"]
+ )
+ mock_pages = (mock_page, mock_page, mock_page)
+ type(mock_rows).pages = mock.PropertyMock(return_value=mock_pages)
+
+ schema = [
+ schema.SchemaField("colA", "IGNORED"),
+ schema.SchemaField("colC", "IGNORED"),
+ schema.SchemaField("colB", "IGNORED"),
+ ]
+
+ row_iterator = mut.RowIterator(
+ _mock_client(),
+ None, # api_request: ignored
+ None, # path: ignored
+ schema,
+ table=mut.TableReference.from_string("proj.dset.tbl"),
+ selected_fields=schema,
+ )
+
+ got = row_iterator.to_dataframe(bqstorage_client=bqstorage_client)
+
+ # Are the columns in the expected order?
+ column_names = ["colA", "colC", "colB"]
+ self.assertEqual(list(got), column_names)
+
+ # Have expected number of rows?
+ total_pages = len(streams) * len(mock_pages)
+ total_rows = len(page_items) * total_pages
+ self.assertEqual(len(got.index), total_rows)
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(
+ bigquery_storage_v1beta1 is None, "Requires `google-cloud-bigquery-storage`"
+ )
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ def test_to_dataframe_w_bqstorage_multiple_streams_return_unique_index(self):
+ from google.cloud.bigquery import schema
+ from google.cloud.bigquery import table as mut
+ from google.cloud.bigquery_storage_v1beta1 import reader
+
+ arrow_fields = [pyarrow.field("colA", pyarrow.int64())]
+ arrow_schema = pyarrow.schema(arrow_fields)
+
+ streams = [
+ {"name": "/projects/proj/dataset/dset/tables/tbl/streams/1234"},
+ {"name": "/projects/proj/dataset/dset/tables/tbl/streams/5678"},
+ ]
+ session = bigquery_storage_v1beta1.types.ReadSession(
+ streams=streams,
+ arrow_schema={"serialized_schema": arrow_schema.serialize().to_pybytes()},
+ )
+
+ bqstorage_client = mock.create_autospec(
+ bigquery_storage_v1beta1.BigQueryStorageClient
+ )
+ bqstorage_client.create_read_session.return_value = session
+
+ mock_rowstream = mock.create_autospec(reader.ReadRowsStream)
+ bqstorage_client.read_rows.return_value = mock_rowstream
+
+ mock_rows = mock.create_autospec(reader.ReadRowsIterable)
+ mock_rowstream.rows.return_value = mock_rows
+
+ page_data_frame = pandas.DataFrame(
+ [{"colA": 1}, {"colA": -1}], columns=["colA"]
+ )
+ mock_page = mock.create_autospec(reader.ReadRowsPage)
+ mock_page.to_dataframe.return_value = page_data_frame
+ mock_pages = (mock_page, mock_page, mock_page)
+ type(mock_rows).pages = mock.PropertyMock(return_value=mock_pages)
+
+ row_iterator = self._make_one(
+ schema=[schema.SchemaField("colA", "IGNORED")],
+ table=mut.TableReference.from_string("proj.dset.tbl"),
+ )
+ got = row_iterator.to_dataframe(bqstorage_client=bqstorage_client)
+
+ self.assertEqual(list(got), ["colA"])
+ total_pages = len(streams) * len(mock_pages)
+ total_rows = len(page_data_frame) * total_pages
+ self.assertEqual(len(got.index), total_rows)
+ self.assertTrue(got.index.is_unique)
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(
+ bigquery_storage_v1beta1 is None, "Requires `google-cloud-bigquery-storage`"
+ )
+ @unittest.skipIf(tqdm is None, "Requires `tqdm`")
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ @mock.patch("tqdm.tqdm")
+ def test_to_dataframe_w_bqstorage_updates_progress_bar(self, tqdm_mock):
+ from google.cloud.bigquery import schema
+ from google.cloud.bigquery import table as mut
+ from google.cloud.bigquery_storage_v1beta1 import reader
+
+ # Speed up testing.
+ mut._PROGRESS_INTERVAL = 0.01
+
+ arrow_fields = [pyarrow.field("testcol", pyarrow.int64())]
+ arrow_schema = pyarrow.schema(arrow_fields)
+
+ bqstorage_client = mock.create_autospec(
+ bigquery_storage_v1beta1.BigQueryStorageClient
+ )
+ streams = [
+ # Use two streams we want to check that progress bar updates are
+ # sent from each stream.
+ {"name": "/projects/proj/dataset/dset/tables/tbl/streams/1234"},
+ {"name": "/projects/proj/dataset/dset/tables/tbl/streams/5678"},
+ ]
+ session = bigquery_storage_v1beta1.types.ReadSession(
+ streams=streams,
+ arrow_schema={"serialized_schema": arrow_schema.serialize().to_pybytes()},
+ )
+ bqstorage_client.create_read_session.return_value = session
+
+ mock_rowstream = mock.create_autospec(reader.ReadRowsStream)
+ bqstorage_client.read_rows.return_value = mock_rowstream
+
+ mock_rows = mock.create_autospec(reader.ReadRowsIterable)
+ mock_rowstream.rows.return_value = mock_rows
+ mock_page = mock.create_autospec(reader.ReadRowsPage)
+ page_items = [-1, 0, 1]
+ type(mock_page).num_items = mock.PropertyMock(return_value=len(page_items))
+
+ def blocking_to_dataframe(*args, **kwargs):
+ # Sleep for longer than the waiting interval. This ensures the
+ # progress_queue gets written to more than once because it gives
+ # the worker->progress updater time to sum intermediate updates.
+ time.sleep(2 * mut._PROGRESS_INTERVAL)
+ return pandas.DataFrame({"testcol": page_items})
+
+ mock_page.to_dataframe.side_effect = blocking_to_dataframe
+ mock_pages = (mock_page, mock_page, mock_page, mock_page, mock_page)
+ type(mock_rows).pages = mock.PropertyMock(return_value=mock_pages)
+
+ schema = [schema.SchemaField("testcol", "IGNORED")]
+
+ row_iterator = mut.RowIterator(
+ _mock_client(),
+ None, # api_request: ignored
+ None, # path: ignored
+ schema,
+ table=mut.TableReference.from_string("proj.dset.tbl"),
+ selected_fields=schema,
+ )
+
+ row_iterator.to_dataframe(
+ bqstorage_client=bqstorage_client, progress_bar_type="tqdm"
+ )
+
+ # Make sure that this test updated the progress bar once per page from
+ # each stream.
+ total_pages = len(streams) * len(mock_pages)
+ expected_total_rows = total_pages * len(page_items)
+ progress_updates = [
+ args[0] for args, kwargs in tqdm_mock().update.call_args_list
+ ]
+ # Should have sent >1 update due to delay in blocking_to_dataframe.
+ self.assertGreater(len(progress_updates), 1)
+ self.assertEqual(sum(progress_updates), expected_total_rows)
+ tqdm_mock().close.assert_called_once()
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(
+ bigquery_storage_v1beta1 is None, "Requires `google-cloud-bigquery-storage`"
+ )
+ @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+ def test_to_dataframe_w_bqstorage_exits_on_keyboardinterrupt(self):
+ from google.cloud.bigquery import schema
+ from google.cloud.bigquery import table as mut
+ from google.cloud.bigquery_storage_v1beta1 import reader
+
+ # Speed up testing.
+ mut._PROGRESS_INTERVAL = 0.01
+
+ arrow_fields = [
+ pyarrow.field("colA", pyarrow.int64()),
+ # Not alphabetical to test column order.
+ pyarrow.field("colC", pyarrow.float64()),
+ pyarrow.field("colB", pyarrow.utf8()),
+ ]
+ arrow_schema = pyarrow.schema(arrow_fields)
+
+ bqstorage_client = mock.create_autospec(
+ bigquery_storage_v1beta1.BigQueryStorageClient
+ )
+ session = bigquery_storage_v1beta1.types.ReadSession(
+ streams=[
+ # Use two streams because one will fail with a
+ # KeyboardInterrupt, and we want to check that the other stream
+ # ends early.
+ {"name": "/projects/proj/dataset/dset/tables/tbl/streams/1234"},
+ {"name": "/projects/proj/dataset/dset/tables/tbl/streams/5678"},
+ ],
+ arrow_schema={"serialized_schema": arrow_schema.serialize().to_pybytes()},
+ )
+ bqstorage_client.create_read_session.return_value = session
+
+ def blocking_to_dataframe(*args, **kwargs):
+ # Sleep for longer than the waiting interval so that we know we're
+ # only reading one page per loop at most.
+ time.sleep(2 * mut._PROGRESS_INTERVAL)
+ return pandas.DataFrame(
+ {"colA": [1, -1], "colB": ["abc", "def"], "colC": [2.0, 4.0]},
+ columns=["colA", "colB", "colC"],
+ )
+
+ mock_page = mock.create_autospec(reader.ReadRowsPage)
+ mock_page.to_dataframe.side_effect = blocking_to_dataframe
+ mock_rows = mock.create_autospec(reader.ReadRowsIterable)
+ mock_pages = mock.PropertyMock(return_value=(mock_page, mock_page, mock_page))
+ type(mock_rows).pages = mock_pages
+ mock_rowstream = mock.create_autospec(reader.ReadRowsStream)
+ mock_rowstream.rows.return_value = mock_rows
+
+ mock_cancelled_rows = mock.create_autospec(reader.ReadRowsIterable)
+ mock_cancelled_pages = mock.PropertyMock(side_effect=KeyboardInterrupt)
+ type(mock_cancelled_rows).pages = mock_cancelled_pages
+ mock_cancelled_rowstream = mock.create_autospec(reader.ReadRowsStream)
+ mock_cancelled_rowstream.rows.return_value = mock_cancelled_rows
+
+ bqstorage_client.read_rows.side_effect = (
+ mock_cancelled_rowstream,
+ mock_rowstream,
+ )
+
+ schema = [
+ schema.SchemaField("colA", "IGNORED"),
+ schema.SchemaField("colB", "IGNORED"),
+ schema.SchemaField("colC", "IGNORED"),
+ ]
+
+ row_iterator = mut.RowIterator(
+ _mock_client(),
+ None, # api_request: ignored
+ None, # path: ignored
+ schema,
+ table=mut.TableReference.from_string("proj.dset.tbl"),
+ selected_fields=schema,
+ )
+
+ with pytest.raises(KeyboardInterrupt):
+ row_iterator.to_dataframe(bqstorage_client=bqstorage_client)
+
+ # Should not have fetched the third page of results because exit_early
+ # should have been set.
+ self.assertLessEqual(mock_page.to_dataframe.call_count, 2)
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(
+ bigquery_storage_v1beta1 is None, "Requires `google-cloud-bigquery-storage`"
+ )
+ def test_to_dataframe_w_bqstorage_fallback_to_tabledata_list(self):
+ from google.cloud.bigquery import schema
+ from google.cloud.bigquery import table as mut
+
+ bqstorage_client = mock.create_autospec(
+ bigquery_storage_v1beta1.BigQueryStorageClient
+ )
+ bqstorage_client.create_read_session.side_effect = google.api_core.exceptions.InternalServerError(
+ "can't read with bqstorage_client"
+ )
+ iterator_schema = [
+ schema.SchemaField("name", "STRING", mode="REQUIRED"),
+ schema.SchemaField("age", "INTEGER", mode="REQUIRED"),
+ ]
+ rows = [
+ {"f": [{"v": "Phred Phlyntstone"}, {"v": "32"}]},
+ {"f": [{"v": "Bharney Rhubble"}, {"v": "33"}]},
+ {"f": [{"v": "Wylma Phlyntstone"}, {"v": "29"}]},
+ {"f": [{"v": "Bhettye Rhubble"}, {"v": "27"}]},
+ ]
+ path = "/foo"
+ api_request = mock.Mock(return_value={"rows": rows})
+ row_iterator = mut.RowIterator(
+ _mock_client(),
+ api_request,
+ path,
+ iterator_schema,
+ table=mut.Table("proj.dset.tbl"),
+ )
+
+ df = row_iterator.to_dataframe(bqstorage_client=bqstorage_client)
+
+ self.assertIsInstance(df, pandas.DataFrame)
+ self.assertEqual(len(df), 4) # verify the number of rows
+ self.assertEqual(list(df), ["name", "age"]) # verify the column names
+ self.assertEqual(df.name.dtype.name, "object")
+ self.assertEqual(df.age.dtype.name, "int64")
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ def test_to_dataframe_tabledata_list_w_multiple_pages_return_unique_index(self):
+ from google.cloud.bigquery import schema
+ from google.cloud.bigquery import table as mut
+
+ iterator_schema = [schema.SchemaField("name", "STRING", mode="REQUIRED")]
+ path = "/foo"
+ api_request = mock.Mock(
+ side_effect=[
+ {"rows": [{"f": [{"v": "Bengt"}]}], "pageToken": "NEXTPAGE"},
+ {"rows": [{"f": [{"v": "Sven"}]}]},
+ ]
+ )
+ row_iterator = mut.RowIterator(
+ _mock_client(),
+ api_request,
+ path,
+ iterator_schema,
+ table=mut.Table("proj.dset.tbl"),
+ )
+
+ df = row_iterator.to_dataframe(bqstorage_client=None)
+
+ self.assertIsInstance(df, pandas.DataFrame)
+ self.assertEqual(len(df), 2)
+ self.assertEqual(list(df), ["name"])
+ self.assertEqual(df.name.dtype.name, "object")
+ self.assertTrue(df.index.is_unique)
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(
+ bigquery_storage_v1beta1 is None, "Requires `google-cloud-bigquery-storage`"
+ )
+ def test_to_dataframe_w_bqstorage_raises_auth_error(self):
+ from google.cloud.bigquery import table as mut
+
+ bqstorage_client = mock.create_autospec(
+ bigquery_storage_v1beta1.BigQueryStorageClient
+ )
+ bqstorage_client.create_read_session.side_effect = google.api_core.exceptions.Forbidden(
+ "TEST BigQuery Storage API not enabled. TEST"
+ )
+ path = "/foo"
+ api_request = mock.Mock(return_value={"rows": []})
+ row_iterator = mut.RowIterator(
+ _mock_client(), api_request, path, [], table=mut.Table("proj.dset.tbl")
+ )
+
+ with pytest.raises(google.api_core.exceptions.Forbidden):
+ row_iterator.to_dataframe(bqstorage_client=bqstorage_client)
+
+ @unittest.skipIf(pandas is None, "Requires `pandas`")
+ @unittest.skipIf(
+ bigquery_storage_v1beta1 is None, "Requires `google-cloud-bigquery-storage`"
+ )
+ def test_to_dataframe_w_bqstorage_raises_import_error(self):
+ from google.cloud.bigquery import table as mut
+
+ bqstorage_client = mock.create_autospec(
+ bigquery_storage_v1beta1.BigQueryStorageClient
+ )
+ path = "/foo"
+ api_request = mock.Mock(return_value={"rows": []})
+ row_iterator = mut.RowIterator(
+ _mock_client(), api_request, path, [], table=mut.Table("proj.dset.tbl")
+ )
+
+ with mock.patch.object(mut, "bigquery_storage_v1beta1", None), pytest.raises(
+ ValueError
+ ) as exc_context:
+ row_iterator.to_dataframe(bqstorage_client=bqstorage_client)
+ assert mut._NO_BQSTORAGE_ERROR in str(exc_context.value)
+
+ @unittest.skipIf(
+ bigquery_storage_v1beta1 is None, "Requires `google-cloud-bigquery-storage`"
+ )
+ def test_to_dataframe_w_bqstorage_partition(self):
+ from google.cloud.bigquery import schema
+ from google.cloud.bigquery import table as mut
+
+ bqstorage_client = mock.create_autospec(
+ bigquery_storage_v1beta1.BigQueryStorageClient
+ )
+
+ row_iterator = mut.RowIterator(
+ _mock_client(),
+ None, # api_request: ignored
+ None, # path: ignored
+ [schema.SchemaField("colA", "IGNORED")],
+ table=mut.TableReference.from_string("proj.dset.tbl$20181225"),
+ )
+
+ with pytest.raises(ValueError):
+ row_iterator.to_dataframe(bqstorage_client)
+
+ @unittest.skipIf(
+ bigquery_storage_v1beta1 is None, "Requires `google-cloud-bigquery-storage`"
+ )
+ def test_to_dataframe_w_bqstorage_snapshot(self):
+ from google.cloud.bigquery import schema
+ from google.cloud.bigquery import table as mut
+
+ bqstorage_client = mock.create_autospec(
+ bigquery_storage_v1beta1.BigQueryStorageClient
+ )
+
+ row_iterator = mut.RowIterator(
+ _mock_client(),
+ None, # api_request: ignored
+ None, # path: ignored
+ [schema.SchemaField("colA", "IGNORED")],
+ table=mut.TableReference.from_string("proj.dset.tbl@1234567890000"),
+ )
+
+ with pytest.raises(ValueError):
+ row_iterator.to_dataframe(bqstorage_client)
+
+
+class TestTimePartitioning(unittest.TestCase):
+ def _get_target_class(self):
+ from google.cloud.bigquery.table import TimePartitioning
+
+ return TimePartitioning
+
+ def _make_one(self, *args, **kw):
+ return self._get_target_class()(*args, **kw)
+
+ def test_constructor_defaults(self):
+ time_partitioning = self._make_one()
+
+ self.assertEqual(time_partitioning.type_, "DAY")
+ self.assertIsNone(time_partitioning.field)
+ self.assertIsNone(time_partitioning.expiration_ms)
+ self.assertIsNone(time_partitioning.require_partition_filter)
+
+ def test_constructor_explicit(self):
+ from google.cloud.bigquery.table import TimePartitioningType
+
+ time_partitioning = self._make_one(
+ type_=TimePartitioningType.DAY,
+ field="name",
+ expiration_ms=10000,
+ require_partition_filter=True,
+ )
+
+ self.assertEqual(time_partitioning.type_, "DAY")
+ self.assertEqual(time_partitioning.field, "name")
+ self.assertEqual(time_partitioning.expiration_ms, 10000)
+ self.assertTrue(time_partitioning.require_partition_filter)
+
+ def test_from_api_repr_empty(self):
+ klass = self._get_target_class()
+
+ # Even though there are required properties according to the API
+ # specification, sometimes time partitioning is populated as an empty
+ # object. See internal bug 131167013.
+ api_repr = {}
+ time_partitioning = klass.from_api_repr(api_repr)
+
+ self.assertIsNone(time_partitioning.type_)
+ self.assertIsNone(time_partitioning.field)
+ self.assertIsNone(time_partitioning.expiration_ms)
+ self.assertIsNone(time_partitioning.require_partition_filter)
+
+ def test_from_api_repr_minimal(self):
+ from google.cloud.bigquery.table import TimePartitioningType
+
+ klass = self._get_target_class()
+ api_repr = {"type": "DAY"}
+ time_partitioning = klass.from_api_repr(api_repr)
+
+ self.assertEqual(time_partitioning.type_, TimePartitioningType.DAY)
+ self.assertIsNone(time_partitioning.field)
+ self.assertIsNone(time_partitioning.expiration_ms)
+ self.assertIsNone(time_partitioning.require_partition_filter)
+
+ def test_from_api_repr_doesnt_override_type(self):
+ klass = self._get_target_class()
+ api_repr = {"type": "HOUR"}
+ time_partitioning = klass.from_api_repr(api_repr)
+ self.assertEqual(time_partitioning.type_, "HOUR")
+
+ def test_from_api_repr_explicit(self):
+ from google.cloud.bigquery.table import TimePartitioningType
+
+ klass = self._get_target_class()
+ api_repr = {
+ "type": "DAY",
+ "field": "name",
+ "expirationMs": "10000",
+ "requirePartitionFilter": True,
+ }
+ time_partitioning = klass.from_api_repr(api_repr)
+
+ self.assertEqual(time_partitioning.type_, TimePartitioningType.DAY)
+ self.assertEqual(time_partitioning.field, "name")
+ self.assertEqual(time_partitioning.expiration_ms, 10000)
+ self.assertTrue(time_partitioning.require_partition_filter)
+
+ def test_to_api_repr_defaults(self):
+ time_partitioning = self._make_one()
+ expected = {"type": "DAY"}
+ self.assertEqual(time_partitioning.to_api_repr(), expected)
+
+ def test_to_api_repr_explicit(self):
+ from google.cloud.bigquery.table import TimePartitioningType
+
+ time_partitioning = self._make_one(
+ type_=TimePartitioningType.DAY,
+ field="name",
+ expiration_ms=10000,
+ require_partition_filter=True,
+ )
+
+ expected = {
+ "type": "DAY",
+ "field": "name",
+ "expirationMs": "10000",
+ "requirePartitionFilter": True,
+ }
+ self.assertEqual(time_partitioning.to_api_repr(), expected)
+
+ def test___eq___wrong_type(self):
+ time_partitioning = self._make_one()
+ other = object()
+ self.assertNotEqual(time_partitioning, other)
+ self.assertEqual(time_partitioning, mock.ANY)
+
+ def test___eq___type__mismatch(self):
+ time_partitioning = self._make_one()
+ other = self._make_one(type_="HOUR")
+ self.assertNotEqual(time_partitioning, other)
+
+ def test___eq___field_mismatch(self):
+ time_partitioning = self._make_one(field="foo")
+ other = self._make_one(field="bar")
+ self.assertNotEqual(time_partitioning, other)
+
+ def test___eq___expiration_ms_mismatch(self):
+ time_partitioning = self._make_one(field="foo", expiration_ms=100000)
+ other = self._make_one(field="foo", expiration_ms=200000)
+ self.assertNotEqual(time_partitioning, other)
+
+ def test___eq___require_partition_filter_mismatch(self):
+ time_partitioning = self._make_one(
+ field="foo", expiration_ms=100000, require_partition_filter=True
+ )
+ other = self._make_one(
+ field="foo", expiration_ms=100000, require_partition_filter=False
+ )
+ self.assertNotEqual(time_partitioning, other)
+
+ def test___eq___hit(self):
+ time_partitioning = self._make_one(
+ field="foo", expiration_ms=100000, require_partition_filter=True
+ )
+ other = self._make_one(
+ field="foo", expiration_ms=100000, require_partition_filter=True
+ )
+ self.assertEqual(time_partitioning, other)
+
+ def test___ne___wrong_type(self):
+ time_partitioning = self._make_one()
+ other = object()
+ self.assertNotEqual(time_partitioning, other)
+ self.assertEqual(time_partitioning, mock.ANY)
+
+ def test___ne___same_value(self):
+ time_partitioning1 = self._make_one()
+ time_partitioning2 = self._make_one()
+ # unittest ``assertEqual`` uses ``==`` not ``!=``.
+ comparison_val = time_partitioning1 != time_partitioning2
+ self.assertFalse(comparison_val)
+
+ def test___ne___different_values(self):
+ time_partitioning1 = self._make_one()
+ time_partitioning2 = self._make_one(type_="HOUR")
+ self.assertNotEqual(time_partitioning1, time_partitioning2)
+
+ def test___hash__set_equality(self):
+ time_partitioning1 = self._make_one(field="foo")
+ time_partitioning2 = self._make_one(field="foo")
+ set_one = {time_partitioning1, time_partitioning2}
+ set_two = {time_partitioning1, time_partitioning2}
+ self.assertEqual(set_one, set_two)
+
+ def test___hash__not_equals(self):
+ time_partitioning1 = self._make_one(field="foo")
+ time_partitioning2 = self._make_one(field="bar")
+ set_one = {time_partitioning1}
+ set_two = {time_partitioning2}
+ self.assertNotEqual(set_one, set_two)
+
+ def test___repr___minimal(self):
+ time_partitioning = self._make_one()
+ expected = "TimePartitioning(type=DAY)"
+ self.assertEqual(repr(time_partitioning), expected)
+
+ def test___repr___explicit(self):
+ from google.cloud.bigquery.table import TimePartitioningType
+
+ time_partitioning = self._make_one(
+ type_=TimePartitioningType.DAY,
+ field="name",
+ expiration_ms=10000,
+ require_partition_filter=True,
+ )
+ expected = (
+ "TimePartitioning("
+ "expirationMs=10000,"
+ "field=name,"
+ "requirePartitionFilter=True,"
+ "type=DAY)"
+ )
+ self.assertEqual(repr(time_partitioning), expected)
+
+ def test_set_expiration_w_none(self):
+ time_partitioning = self._make_one()
+ time_partitioning.expiration_ms = None
+ assert time_partitioning._properties["expirationMs"] is None
+
+
+@pytest.mark.skipif(
+ bigquery_storage_v1beta1 is None, reason="Requires `google-cloud-bigquery-storage`"
+)
+def test_table_reference_to_bqstorage():
+ from google.cloud.bigquery import table as mut
+
+ # Can't use parametrized pytest because bigquery_storage_v1beta1 may not be
+ # available.
+ expected = bigquery_storage_v1beta1.types.TableReference(
+ project_id="my-project", dataset_id="my_dataset", table_id="my_table"
+ )
+ cases = (
+ "my-project.my_dataset.my_table",
+ "my-project.my_dataset.my_table$20181225",
+ "my-project.my_dataset.my_table@1234567890",
+ "my-project.my_dataset.my_table$20181225@1234567890",
+ )
+
+ classes = (mut.TableReference, mut.Table, mut.TableListItem)
+
+ for case, cls in itertools.product(cases, classes):
+ got = cls.from_string(case).to_bqstorage()
+ assert got == expected
+
+
+@unittest.skipIf(
+ bigquery_storage_v1beta1 is None, "Requires `google-cloud-bigquery-storage`"
+)
+def test_table_reference_to_bqstorage_raises_import_error():
+ from google.cloud.bigquery import table as mut
+
+ classes = (mut.TableReference, mut.Table, mut.TableListItem)
+ for cls in classes:
+ with mock.patch.object(mut, "bigquery_storage_v1beta1", None), pytest.raises(
+ ValueError
+ ) as exc_context:
+ cls.from_string("my-project.my_dataset.my_table").to_bqstorage()
+ assert mut._NO_BQSTORAGE_ERROR in str(exc_context.value)
diff --git a/bigquery_datatransfer/.coveragerc b/bigquery_datatransfer/.coveragerc
new file mode 100644
index 000000000000..b178b094aa1d
--- /dev/null
+++ b/bigquery_datatransfer/.coveragerc
@@ -0,0 +1,19 @@
+# Generated by synthtool. DO NOT EDIT!
+[run]
+branch = True
+
+[report]
+fail_under = 100
+show_missing = True
+exclude_lines =
+ # Re-enable the standard pragma
+ pragma: NO COVER
+ # Ignore debug-only repr
+ def __repr__
+ # Ignore abstract methods
+ raise NotImplementedError
+omit =
+ */gapic/*.py
+ */proto/*.py
+ */core/*.py
+ */site-packages/*.py
\ No newline at end of file
diff --git a/bigquery_datatransfer/.flake8 b/bigquery_datatransfer/.flake8
new file mode 100644
index 000000000000..0268ecc9c55c
--- /dev/null
+++ b/bigquery_datatransfer/.flake8
@@ -0,0 +1,14 @@
+# Generated by synthtool. DO NOT EDIT!
+[flake8]
+ignore = E203, E266, E501, W503
+exclude =
+ # Exclude generated code.
+ **/proto/**
+ **/gapic/**
+ *_pb2.py
+
+ # Standard linting exemptions.
+ __pycache__,
+ .git,
+ *.pyc,
+ conf.py
diff --git a/bigquery_datatransfer/.repo-metadata.json b/bigquery_datatransfer/.repo-metadata.json
new file mode 100644
index 000000000000..c128b1bcfc27
--- /dev/null
+++ b/bigquery_datatransfer/.repo-metadata.json
@@ -0,0 +1,13 @@
+{
+ "name": "bigquerydatatransfer",
+ "name_pretty": "Google BigQuery Data Transfer Service",
+ "product_documentation": "https://cloud.google.com/bigquery/transfer/",
+ "client_documentation": "https://googleapis.dev/python/bigquerydatatransfer/latest",
+ "issue_tracker": "https://issuetracker.google.com/savedsearches/559654",
+ "release_level": "alpha",
+ "language": "python",
+ "repo": "googleapis/google-cloud-python",
+ "distribution_name": "google-cloud-bigquery-datatransfer",
+ "api_id": "bigquerydatatransfer.googleapis.com",
+ "requires_billing": true
+}
\ No newline at end of file
diff --git a/bigquery_datatransfer/CHANGELOG.md b/bigquery_datatransfer/CHANGELOG.md
new file mode 100644
index 000000000000..483716f73f63
--- /dev/null
+++ b/bigquery_datatransfer/CHANGELOG.md
@@ -0,0 +1,132 @@
+# Changelog
+
+[PyPI History][1]
+
+[1]: https://pypi.org/project/google-cloud-bigquery-datatransfer/#history
+
+## 0.4.1
+
+07-31-2019 17:50 PDT
+
+
+### Dependencies
+- Bump minimum version for google-api-core to 1.14.0. ([#8709](https://github.com/googleapis/google-cloud-python/pull/8709))
+
+### Documentation
+- Fix links to BigQuery Datatransfer documentation. ([#8859](https://github.com/googleapis/google-cloud-python/pull/8859))
+- Link to googleapis.dev documentation in READMEs. ([#8705](https://github.com/googleapis/google-cloud-python/pull/8705))
+
+### Internal / Testing Changes
+- Update intersphinx mapping for requests. ([#8805](https://github.com/googleapis/google-cloud-python/pull/8805))
+
+## 0.4.0
+
+07-16-2019 17:11 PDT
+
+### Implementation Changes
+
+- Retry DEADLINE_EXCEEDED (via synth). ([#7920](https://github.com/googleapis/google-cloud-python/pull/7920))
+- Remove classifier for Python 3.4 for end-of-life. ([#7535](https://github.com/googleapis/google-cloud-python/pull/7535))
+
+### New Features
+
+- Add `DatasourceServiceClient` (via synth). ([#8630](https://github.com/googleapis/google-cloud-python/pull/8630))
+- Add `start_manual_transfer_runs` method (via synth). ([#8630](https://github.com/googleapis/google-cloud-python/pull/8630))
+- Add `client_info`/`version_info` support (via synth). ([#8630](https://github.com/googleapis/google-cloud-python/pull/8630))
+- Allow passing kwargs to `create_channel` (via synth). ([#8630](https://github.com/googleapis/google-cloud-python/pull/8630))
+- Add path helpers (via synth). ([#8630](https://github.com/googleapis/google-cloud-python/pull/8630))
+- Add protos as an artifact to library ([#7205](https://github.com/googleapis/google-cloud-python/pull/7205))
+
+### Documentation
+
+- Add compatibility check badges to READMEs. ([#8288](https://github.com/googleapis/google-cloud-python/pull/8288))
+- Adjust indentation on scheduled query sample. ([#8493](https://github.com/googleapis/google-cloud-python/pull/8493))
+- Add docs job to publish to googleapis.dev. ([#8464](https://github.com/googleapis/google-cloud-python/pull/8464))
+- Add sample to schedule query with BQ DTS. ([#7703](https://github.com/googleapis/google-cloud-python/pull/7703))
+- Add nox session `docs` (via synth). ([#7765](https://github.com/googleapis/google-cloud-python/pull/7765))
+- Updated client library documentation URLs. ([#7307](https://github.com/googleapis/google-cloud-python/pull/7307))
+- Pick up stub docstring fix in GAPIC generator. ([#6965](https://github.com/googleapis/google-cloud-python/pull/6965))
+
+### Internal / Testing Changes
+
+- Blacken noxfile.py, setup.py (via synth). ([#8116](https://github.com/googleapis/google-cloud-python/pull/8116))
+- Add empty lines (via synth). ([#8050](https://github.com/googleapis/google-cloud-python/pull/8050))
+- Remove unused message exports (via synth). ([#7263](https://github.com/googleapis/google-cloud-python/pull/7263))
+- Protoc-generated serialization update. ([#7075](https://github.com/googleapis/google-cloud-python/pull/7075))
+
+## 0.3.0
+
+12-17-2018 17:59 PST
+
+
+### Implementation Changes
+- Pick up enum fixes in the GAPIC generator. ([#6608](https://github.com/googleapis/google-cloud-python/pull/6608))
+- Pick up fixes in GAPIC generator. ([#6491](https://github.com/googleapis/google-cloud-python/pull/6491))
+- Fix `client_info` bug, update docstrings. ([#6405](https://github.com/googleapis/google-cloud-python/pull/6405))
+- Re-generate library using bigquery_datatransfer/synth.py ([#5973](https://github.com/googleapis/google-cloud-python/pull/5973))
+- Fix stray, lint-breaking blank lines from autosynth. ([#5960](https://github.com/googleapis/google-cloud-python/pull/5960))
+- Re-generate library using `bigquery_datatransfer/synth.py`. ([#5947](https://github.com/googleapis/google-cloud-python/pull/5947))
+
+### Dependencies
+- Bump minimum api_core version for all GAPIC libs to 1.4.1. ([#6391](https://github.com/googleapis/google-cloud-python/pull/6391))
+
+### Documentation
+- Document Python 2 deprecation ([#6910](https://github.com/googleapis/google-cloud-python/pull/6910))
+- Fix GAX fossils ([#6264](https://github.com/googleapis/google-cloud-python/pull/6264))
+- Normalize use of support level badges ([#6159](https://github.com/googleapis/google-cloud-python/pull/6159))
+- Harmonize / DRY 'README.rst' / 'docs/index.rst'. ([#6013](https://github.com/googleapis/google-cloud-python/pull/6013))
+
+### Internal / Testing Changes
+- Update noxfile.
+- Blacken all gen'd libs ([#6792](https://github.com/googleapis/google-cloud-python/pull/6792))
+- Omit local deps ([#6701](https://github.com/googleapis/google-cloud-python/pull/6701))
+- Run black at end of synth.py ([#6698](https://github.com/googleapis/google-cloud-python/pull/6698))
+- Unblack bigquery gapic and protos.
+- Run Black on Generated libraries ([#6666](https://github.com/googleapis/google-cloud-python/pull/6666))
+- Add templates for flake8, coveragerc, noxfile, and black. ([#6642](https://github.com/googleapis/google-cloud-python/pull/6642))
+- Add synth metadata. ([#6562](https://github.com/googleapis/google-cloud-python/pull/6562))
+- Use new Nox ([#6175](https://github.com/googleapis/google-cloud-python/pull/6175))
+
+## 0.2.0
+
+### Implementation Changes
+- Regenerate bigquery-datatransfer (#5793)
+
+### Internal / Testing Changes
+- Avoid overwriting '__module__' of messages from shared modules. (#5364)
+- Modify system tests to use prerelease versions of grpcio (#5304)
+- Add Test runs for Python 3.7 and remove 3.4 (#5295)
+- Fix bad trove classifier
+- Rename releases to changelog and include from CHANGELOG.md (#5191)
+
+## 0.1.1
+
+### Dependencies
+
+- Update dependency range for api-core to include v1.0.0 releases (#4944)
+
+### Documentation
+
+- Fix package name in readme (#4670)
+- BigQueryDataTransfer: update 404 link for API documentation (#4672)
+- Replacing references to `stable/` docs with `latest/`. (#4638)
+
+### Testing and internal changes
+
+- Re-enable lint for tests, remove usage of pylint (#4921)
+- Normalize all setup.py files (#4909)
+- Update index.rst (#4816)
+- nox unittest updates (#4646)
+
+## 0.1.0
+
+[](https://cloud.google.com/terms/launch-stages)
+
+The BigQuery Data Transfer Service automates data movement from SaaS
+applications to Google BigQuery on a scheduled, managed basis. Your analytics
+team can lay the foundation for a data warehouse without writing a single line
+of code. BigQuery Data Transfer Service initially supports Google application
+sources like Adwords, DoubleClick Campaign Manager, DoubleClick for Publishers
+and YouTube.
+
+PyPI: https://pypi.org/project/google-cloud-bigquery-datatransfer/0.1.0/
diff --git a/bigquery_datatransfer/LICENSE b/bigquery_datatransfer/LICENSE
new file mode 100644
index 000000000000..a8ee855de2aa
--- /dev/null
+++ b/bigquery_datatransfer/LICENSE
@@ -0,0 +1,201 @@
+ Apache License
+ Version 2.0, January 2004
+ https://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [yyyy] [name of copyright owner]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
diff --git a/bigquery_datatransfer/MANIFEST.in b/bigquery_datatransfer/MANIFEST.in
new file mode 100644
index 000000000000..9cbf175afe6b
--- /dev/null
+++ b/bigquery_datatransfer/MANIFEST.in
@@ -0,0 +1,5 @@
+include README.rst LICENSE
+recursive-include google *.json *.proto
+recursive-include tests *
+global-exclude *.py[co]
+global-exclude __pycache__
diff --git a/bigquery_datatransfer/README.rst b/bigquery_datatransfer/README.rst
new file mode 100644
index 000000000000..bb4b5a10ef06
--- /dev/null
+++ b/bigquery_datatransfer/README.rst
@@ -0,0 +1,111 @@
+Python Client for BigQuery Data Transfer API
+============================================
+
+|alpha| |pypi| |versions|
+
+The `BigQuery Data Transfer API`_ allows users to transfer data from partner
+SaaS applications to Google BigQuery on a scheduled, managed basis.
+
+- `Client Library Documentation`_
+- `Product Documentation`_
+
+.. |alpha| image:: https://img.shields.io/badge/support-alpha-orange.svg
+ :target: https://github.com/googleapis/google-cloud-python/blob/master/README.rst#alpha-support
+.. |pypi| image:: https://img.shields.io/pypi/v/google-cloud-bigquery-datatransfer.svg
+ :target: https://pypi.org/project/google-cloud-bigquery-datatransfer/
+.. |versions| image:: https://img.shields.io/pypi/pyversions/google-cloud-bigquery-datatransfer.svg
+ :target: https://pypi.org/project/google-cloud-bigquery-datatransfer/
+.. _BigQuery Data Transfer API: https://cloud.google.com/bigquery/transfer
+.. _Client Library Documentation: https://googleapis.dev/python/bigquerydatatransfer/latest
+.. _Product Documentation: https://cloud.google.com/bigquery/docs/transfer-service-overview
+
+Quick Start
+-----------
+
+In order to use this library, you first need to go through the following steps:
+
+1. `Select or create a Cloud Platform project.`_
+2. `Enable the BigQuery Data Transfer API.`_
+3. `Setup Authentication.`_
+
+.. _Select or create a Cloud Platform project.: https://console.cloud.google.com/project
+.. _Enable the BigQuery Data Transfer API.: https://cloud.google.com/bigquery/docs/transfer-service-overview
+.. _Setup Authentication.: https://googleapis.dev/python/google-api-core/latest/auth.html
+
+Installation
+~~~~~~~~~~~~
+
+Install this library in a `virtualenv`_ using pip. `virtualenv`_ is a tool to
+create isolated Python environments. The basic problem it addresses is one of
+dependencies and versions, and indirectly permissions.
+
+With `virtualenv`_, it's possible to install this library without needing system
+install permissions, and without clashing with the installed system
+dependencies.
+
+.. _`virtualenv`: https://virtualenv.pypa.io/en/latest/
+
+
+Supported Python Versions
+^^^^^^^^^^^^^^^^^^^^^^^^^
+Python >= 3.5
+
+Deprecated Python Versions
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+Python == 2.7. Python 2.7 support will be removed on January 1, 2020.
+
+
+Mac/Linux
+^^^^^^^^^
+
+.. code-block:: console
+
+ pip install virtualenv
+ virtualenv
+ source /bin/activate
+ /bin/pip install google-cloud-bigquery-datatransfer
+
+
+Windows
+^^^^^^^
+
+.. code-block:: console
+
+ pip install virtualenv
+ virtualenv
+ \Scripts\activate
+ \Scripts\pip.exe install google-cloud-bigquery-datatransfer
+
+Example Usage
+~~~~~~~~~~~~~
+
+DataTransferServiceClient
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code:: py
+
+ from google.cloud.bigquery import datatransfer_v1
+
+ client = datatransfer_v1.DataTransferServiceClient()
+
+ parent = client.location_path('[PROJECT]', '[LOCATION]')
+
+
+ # Iterate over all results
+ for element in client.list_data_sources(parent):
+ # process element
+ pass
+
+ # Or iterate over results one page at a time
+ for page in client.list_data_sources(parent).pages:
+ for element in page:
+ # process element
+ pass
+
+Next Steps
+~~~~~~~~~~
+
+- Read the `Client Library Documentation`_ for BigQuery Data Transfer API
+ API to see other available methods on the client.
+- Read the `Product documentation`_ to learn
+ more about the product and see How-to Guides.
diff --git a/bigquery_datatransfer/docs/README.rst b/bigquery_datatransfer/docs/README.rst
new file mode 120000
index 000000000000..89a0106941ff
--- /dev/null
+++ b/bigquery_datatransfer/docs/README.rst
@@ -0,0 +1 @@
+../README.rst
\ No newline at end of file
diff --git a/bigquery_datatransfer/docs/changelog.md b/bigquery_datatransfer/docs/changelog.md
new file mode 120000
index 000000000000..05ed7b33f590
--- /dev/null
+++ b/bigquery_datatransfer/docs/changelog.md
@@ -0,0 +1 @@
+../../bigquery_datatransfer/CHANGELOG.md
\ No newline at end of file
diff --git a/bigquery_datatransfer/docs/conf.py b/bigquery_datatransfer/docs/conf.py
new file mode 100644
index 000000000000..ef6f120fefa6
--- /dev/null
+++ b/bigquery_datatransfer/docs/conf.py
@@ -0,0 +1,356 @@
+# -*- coding: utf-8 -*-
+#
+# google-cloud-bigquerydatatransfer documentation build configuration file
+#
+# This file is execfile()d with the current directory set to its
+# containing dir.
+#
+# Note that not all possible configuration values are present in this
+# autogenerated file.
+#
+# All configuration values have a default; values that are commented out
+# serve to show the default.
+
+import sys
+import os
+import shlex
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+sys.path.insert(0, os.path.abspath(".."))
+
+__version__ = "0.1.0"
+
+# -- General configuration ------------------------------------------------
+
+# If your documentation needs a minimal Sphinx version, state it here.
+needs_sphinx = "1.6.3"
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+ "sphinx.ext.autodoc",
+ "sphinx.ext.autosummary",
+ "sphinx.ext.intersphinx",
+ "sphinx.ext.coverage",
+ "sphinx.ext.napoleon",
+ "sphinx.ext.todo",
+ "sphinx.ext.viewcode",
+]
+
+# autodoc/autosummary flags
+autoclass_content = "both"
+autodoc_default_flags = ["members"]
+autosummary_generate = True
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ["_templates"]
+
+# Allow markdown includes (so releases.md can include CHANGLEOG.md)
+# http://www.sphinx-doc.org/en/master/markdown.html
+source_parsers = {".md": "recommonmark.parser.CommonMarkParser"}
+
+# The suffix(es) of source filenames.
+# You can specify multiple suffix as a list of string:
+# source_suffix = ['.rst', '.md']
+source_suffix = [".rst", ".md"]
+
+# The encoding of source files.
+# source_encoding = 'utf-8-sig'
+
+# The master toctree document.
+master_doc = "index"
+
+# General information about the project.
+project = u"google-cloud-bigquerydatatransfer"
+copyright = u"2017, Google"
+author = u"Google APIs"
+
+# The version info for the project you're documenting, acts as replacement for
+# |version| and |release|, also used in various other places throughout the
+# built documents.
+#
+# The full version, including alpha/beta/rc tags.
+release = __version__
+# The short X.Y version.
+version = ".".join(release.split(".")[0:2])
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#
+# This is also used if you do content translation via gettext catalogs.
+# Usually you set "language" from the command line for these cases.
+language = None
+
+# There are two options for replacing |today|: either, you set today to some
+# non-false value, then it is used:
+# today = ''
+# Else, today_fmt is used as the format for a strftime call.
+# today_fmt = '%B %d, %Y'
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+exclude_patterns = ["_build"]
+
+# The reST default role (used for this markup: `text`) to use for all
+# documents.
+# default_role = None
+
+# If true, '()' will be appended to :func: etc. cross-reference text.
+# add_function_parentheses = True
+
+# If true, the current module name will be prepended to all description
+# unit titles (such as .. function::).
+# add_module_names = True
+
+# If true, sectionauthor and moduleauthor directives will be shown in the
+# output. They are ignored by default.
+# show_authors = False
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = "sphinx"
+
+# A list of ignored prefixes for module index sorting.
+# modindex_common_prefix = []
+
+# If true, keep warnings as "system message" paragraphs in the built documents.
+# keep_warnings = False
+
+# If true, `todo` and `todoList` produce output, else they produce nothing.
+todo_include_todos = True
+
+# -- Options for HTML output ----------------------------------------------
+
+# The theme to use for HTML and HTML Help pages. See the documentation for
+# a list of builtin themes.
+html_theme = "alabaster"
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further. For a list of options available for each theme, see the
+# documentation.
+html_theme_options = {
+ "description": "Google Cloud Client Libraries for Python",
+ "github_user": "googleapis",
+ "github_repo": "google-cloud-python",
+ "github_banner": True,
+ "font_family": "'Roboto', Georgia, sans",
+ "head_font_family": "'Roboto', Georgia, serif",
+ "code_font_family": "'Roboto Mono', 'Consolas', monospace",
+}
+
+# Add any paths that contain custom themes here, relative to this directory.
+# html_theme_path = []
+
+# The name for this set of Sphinx documents. If None, it defaults to
+# " v documentation".
+# html_title = None
+
+# A shorter title for the navigation bar. Default is the same as html_title.
+# html_short_title = None
+
+# The name of an image file (relative to this directory) to place at the top
+# of the sidebar.
+# html_logo = None
+
+# The name of an image file (within the static path) to use as favicon of the
+# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
+# pixels large.
+# html_favicon = None
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+# html_static_path = []
+
+# Add any extra paths that contain custom files (such as robots.txt or
+# .htaccess) here, relative to this directory. These files are copied
+# directly to the root of the documentation.
+# html_extra_path = []
+
+# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
+# using the given strftime format.
+# html_last_updated_fmt = '%b %d, %Y'
+
+# If true, SmartyPants will be used to convert quotes and dashes to
+# typographically correct entities.
+# html_use_smartypants = True
+
+# Custom sidebar templates, maps document names to template names.
+# html_sidebars = {}
+
+# Additional templates that should be rendered to pages, maps page names to
+# template names.
+# html_additional_pages = {}
+
+# If false, no module index is generated.
+# html_domain_indices = True
+
+# If false, no index is generated.
+# html_use_index = True
+
+# If true, the index is split into individual pages for each letter.
+# html_split_index = False
+
+# If true, links to the reST sources are added to the pages.
+# html_show_sourcelink = True
+
+# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
+# html_show_sphinx = True
+
+# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
+# html_show_copyright = True
+
+# If true, an OpenSearch description file will be output, and all pages will
+# contain a tag referring to it. The value of this option must be the
+# base URL from which the finished HTML is served.
+# html_use_opensearch = ''
+
+# This is the file name suffix for HTML files (e.g. ".xhtml").
+# html_file_suffix = None
+
+# Language to be used for generating the HTML full-text search index.
+# Sphinx supports the following languages:
+# 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja'
+# 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr'
+# html_search_language = 'en'
+
+# A dictionary with options for the search language support, empty by default.
+# Now only 'ja' uses this config value
+# html_search_options = {'type': 'default'}
+
+# The name of a javascript file (relative to the configuration directory) that
+# implements a search results scorer. If empty, the default will be used.
+# html_search_scorer = 'scorer.js'
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = "google-cloud-bigquerydatatransfer-doc"
+
+# -- Options for warnings ------------------------------------------------------
+
+suppress_warnings = [
+ # Temporarily suppress this to avoid "more than one target found for
+ # cross-reference" warning, which are intractable for us to avoid while in
+ # a mono-repo.
+ # See https://github.com/sphinx-doc/sphinx/blob
+ # /2a65ffeef5c107c19084fabdd706cdff3f52d93c/sphinx/domains/python.py#L843
+ "ref.python"
+]
+
+# -- Options for LaTeX output ---------------------------------------------
+
+latex_elements = {
+ # The paper size ('letterpaper' or 'a4paper').
+ #'papersize': 'letterpaper',
+ # The font size ('10pt', '11pt' or '12pt').
+ #'pointsize': '10pt',
+ # Additional stuff for the LaTeX preamble.
+ #'preamble': '',
+ # Latex figure (float) alignment
+ #'figure_align': 'htbp',
+}
+
+# Grouping the document tree into LaTeX files. List of tuples
+# (source start file, target name, title,
+# author, documentclass [howto, manual, or own class]).
+latex_documents = [
+ (
+ master_doc,
+ "google-cloud-bigquerydatatransfer.tex",
+ u"google-cloud-bigquerydatatransfer Documentation",
+ author,
+ "manual",
+ )
+]
+
+# The name of an image file (relative to this directory) to place at the top of
+# the title page.
+# latex_logo = None
+
+# For "manual" documents, if this is true, then toplevel headings are parts,
+# not chapters.
+# latex_use_parts = False
+
+# If true, show page references after internal links.
+# latex_show_pagerefs = False
+
+# If true, show URL addresses after external links.
+# latex_show_urls = False
+
+# Documents to append as an appendix to all manuals.
+# latex_appendices = []
+
+# If false, no module index is generated.
+# latex_domain_indices = True
+
+# -- Options for manual page output ---------------------------------------
+
+# One entry per manual page. List of tuples
+# (source start file, name, description, authors, manual section).
+man_pages = [
+ (
+ master_doc,
+ "google-cloud-bigquerydatatransfer",
+ u"google-cloud-bigquerydatatransfer Documentation",
+ [author],
+ 1,
+ )
+]
+
+# If true, show URL addresses after external links.
+# man_show_urls = False
+
+# -- Options for Texinfo output -------------------------------------------
+
+# Grouping the document tree into Texinfo files. List of tuples
+# (source start file, target name, title, author,
+# dir menu entry, description, category)
+texinfo_documents = [
+ (
+ master_doc,
+ "google-cloud-bigquerydatatransfer",
+ u"google-cloud-bigquerydatatransfer Documentation",
+ author,
+ "google-cloud-bigquerydatatransfer",
+ "GAPIC library for the {metadata.shortName} v1 service",
+ "APIs",
+ )
+]
+
+# Documents to append as an appendix to all manuals.
+# texinfo_appendices = []
+
+# If false, no module index is generated.
+# texinfo_domain_indices = True
+
+# How to display URL addresses: 'footnote', 'no', or 'inline'.
+# texinfo_show_urls = 'footnote'
+
+# If true, do not generate a @detailmenu in the "Top" node's menu.
+# texinfo_no_detailmenu = False
+
+# Example configuration for intersphinx: refer to the Python standard library.
+intersphinx_mapping = {
+ "python": ("http://python.readthedocs.org/en/latest/", None),
+ "gax": ("https://gax-python.readthedocs.org/en/latest/", None),
+ "google-auth": ("https://google-auth.readthedocs.io/en/stable", None),
+ "google-gax": ("https://gax-python.readthedocs.io/en/latest/", None),
+ "google.api_core": ("https://googleapis.dev/python/google-api-core/latest", None),
+ "grpc": ("https://grpc.io/grpc/python/", None),
+ "requests": ("https://2.python-requests.org/en/master/", None),
+ "fastavro": ("https://fastavro.readthedocs.io/en/stable/", None),
+ "pandas": ("https://pandas.pydata.org/pandas-docs/stable/", None),
+}
+
+# Napoleon settings
+napoleon_google_docstring = True
+napoleon_numpy_docstring = True
+napoleon_include_private_with_doc = False
+napoleon_include_special_with_doc = True
+napoleon_use_admonition_for_examples = False
+napoleon_use_admonition_for_notes = False
+napoleon_use_admonition_for_references = False
+napoleon_use_ivar = False
+napoleon_use_param = True
+napoleon_use_rtype = True
diff --git a/bigquery_datatransfer/docs/gapic/v1/api.rst b/bigquery_datatransfer/docs/gapic/v1/api.rst
new file mode 100644
index 000000000000..a8b855bb4cc4
--- /dev/null
+++ b/bigquery_datatransfer/docs/gapic/v1/api.rst
@@ -0,0 +1,6 @@
+Client for BigQuery Data Transfer API
+=====================================
+
+.. automodule:: google.cloud.bigquery_datatransfer_v1
+ :members:
+ :inherited-members:
\ No newline at end of file
diff --git a/bigquery_datatransfer/docs/gapic/v1/types.rst b/bigquery_datatransfer/docs/gapic/v1/types.rst
new file mode 100644
index 000000000000..2a77ee73ecef
--- /dev/null
+++ b/bigquery_datatransfer/docs/gapic/v1/types.rst
@@ -0,0 +1,5 @@
+Types for BigQuery Data Transfer API Client
+===========================================
+
+.. automodule:: google.cloud.bigquery_datatransfer_v1.types
+ :members:
\ No newline at end of file
diff --git a/bigquery_datatransfer/docs/index.rst b/bigquery_datatransfer/docs/index.rst
new file mode 100644
index 000000000000..942e2634213c
--- /dev/null
+++ b/bigquery_datatransfer/docs/index.rst
@@ -0,0 +1,22 @@
+.. include:: README.rst
+
+
+API Reference
+-------------
+
+.. toctree::
+ :maxdepth: 2
+
+ gapic/v1/api
+ gapic/v1/types
+
+
+Changelog
+---------
+
+For a list of all ``google-cloud-bigquery-bigquery-datatransfer`` releases:
+
+.. toctree::
+ :maxdepth: 2
+
+ changelog
diff --git a/bigquery_datatransfer/google/__init__.py b/bigquery_datatransfer/google/__init__.py
new file mode 100644
index 000000000000..8fcc60e2b9c6
--- /dev/null
+++ b/bigquery_datatransfer/google/__init__.py
@@ -0,0 +1,24 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+try:
+ import pkg_resources
+
+ pkg_resources.declare_namespace(__name__)
+except ImportError:
+ import pkgutil
+
+ __path__ = pkgutil.extend_path(__path__, __name__)
diff --git a/bigquery_datatransfer/google/cloud/__init__.py b/bigquery_datatransfer/google/cloud/__init__.py
new file mode 100644
index 000000000000..8fcc60e2b9c6
--- /dev/null
+++ b/bigquery_datatransfer/google/cloud/__init__.py
@@ -0,0 +1,24 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+try:
+ import pkg_resources
+
+ pkg_resources.declare_namespace(__name__)
+except ImportError:
+ import pkgutil
+
+ __path__ = pkgutil.extend_path(__path__, __name__)
diff --git a/bigquery_datatransfer/google/cloud/bigquery_datatransfer.py b/bigquery_datatransfer/google/cloud/bigquery_datatransfer.py
new file mode 100644
index 000000000000..ddd78f62eea4
--- /dev/null
+++ b/bigquery_datatransfer/google/cloud/bigquery_datatransfer.py
@@ -0,0 +1,25 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from __future__ import absolute_import
+
+from google.cloud.bigquery_datatransfer_v1 import DataTransferServiceClient
+from google.cloud.bigquery_datatransfer_v1 import enums
+from google.cloud.bigquery_datatransfer_v1 import types
+
+
+__all__ = ("enums", "types", "DataTransferServiceClient")
diff --git a/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/__init__.py b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/__init__.py
new file mode 100644
index 000000000000..e4a3a5f9763c
--- /dev/null
+++ b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/__init__.py
@@ -0,0 +1,30 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from __future__ import absolute_import
+
+from google.cloud.bigquery_datatransfer_v1 import types
+from google.cloud.bigquery_datatransfer_v1.gapic import data_transfer_service_client
+from google.cloud.bigquery_datatransfer_v1.gapic import enums
+
+
+class DataTransferServiceClient(data_transfer_service_client.DataTransferServiceClient):
+ __doc__ = data_transfer_service_client.DataTransferServiceClient.__doc__
+ enums = enums
+
+
+__all__ = ("enums", "types", "DataTransferServiceClient")
diff --git a/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/gapic/__init__.py b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/gapic/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/gapic/data_transfer_service_client.py b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/gapic/data_transfer_service_client.py
new file mode 100644
index 000000000000..5c52eed8a43d
--- /dev/null
+++ b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/gapic/data_transfer_service_client.py
@@ -0,0 +1,1534 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Accesses the google.cloud.bigquery.datatransfer.v1 DataTransferService API."""
+
+import functools
+import pkg_resources
+import warnings
+
+from google.oauth2 import service_account
+import google.api_core.client_options
+import google.api_core.gapic_v1.client_info
+import google.api_core.gapic_v1.config
+import google.api_core.gapic_v1.method
+import google.api_core.path_template
+import google.api_core.gapic_v1.routing_header
+import google.api_core.grpc_helpers
+import google.api_core.page_iterator
+import google.api_core.path_template
+import google.api_core.protobuf_helpers
+import grpc
+
+from google.cloud.bigquery_datatransfer_v1.gapic import (
+ data_transfer_service_client_config,
+)
+from google.cloud.bigquery_datatransfer_v1.gapic import enums
+from google.cloud.bigquery_datatransfer_v1.gapic.transports import (
+ data_transfer_service_grpc_transport,
+)
+from google.cloud.bigquery_datatransfer_v1.proto import datatransfer_pb2
+from google.cloud.bigquery_datatransfer_v1.proto import datatransfer_pb2_grpc
+from google.cloud.bigquery_datatransfer_v1.proto import transfer_pb2
+from google.protobuf import empty_pb2
+from google.protobuf import field_mask_pb2
+from google.protobuf import timestamp_pb2
+
+
+_GAPIC_LIBRARY_VERSION = pkg_resources.get_distribution(
+ "google-cloud-bigquery-datatransfer"
+).version
+
+
+class DataTransferServiceClient(object):
+ """
+ The Google BigQuery Data Transfer Service API enables BigQuery users to
+ configure the transfer of their data from other Google Products into
+ BigQuery. This service contains methods that are end user exposed. It backs
+ up the frontend.
+ """
+
+ SERVICE_ADDRESS = "bigquerydatatransfer.googleapis.com:443"
+ """The default address of the service."""
+
+ # The name of the interface for this client. This is the key used to
+ # find the method configuration in the client_config dictionary.
+ _INTERFACE_NAME = "google.cloud.bigquery.datatransfer.v1.DataTransferService"
+
+ @classmethod
+ def from_service_account_file(cls, filename, *args, **kwargs):
+ """Creates an instance of this client using the provided credentials
+ file.
+
+ Args:
+ filename (str): The path to the service account private key json
+ file.
+ args: Additional arguments to pass to the constructor.
+ kwargs: Additional arguments to pass to the constructor.
+
+ Returns:
+ DataTransferServiceClient: The constructed client.
+ """
+ credentials = service_account.Credentials.from_service_account_file(filename)
+ kwargs["credentials"] = credentials
+ return cls(*args, **kwargs)
+
+ from_service_account_json = from_service_account_file
+
+ @classmethod
+ def location_path(cls, project, location):
+ """Return a fully-qualified location string."""
+ return google.api_core.path_template.expand(
+ "projects/{project}/locations/{location}",
+ project=project,
+ location=location,
+ )
+
+ @classmethod
+ def location_data_source_path(cls, project, location, data_source):
+ """Return a fully-qualified location_data_source string."""
+ return google.api_core.path_template.expand(
+ "projects/{project}/locations/{location}/dataSources/{data_source}",
+ project=project,
+ location=location,
+ data_source=data_source,
+ )
+
+ @classmethod
+ def location_run_path(cls, project, location, transfer_config, run):
+ """Return a fully-qualified location_run string."""
+ return google.api_core.path_template.expand(
+ "projects/{project}/locations/{location}/transferConfigs/{transfer_config}/runs/{run}",
+ project=project,
+ location=location,
+ transfer_config=transfer_config,
+ run=run,
+ )
+
+ @classmethod
+ def location_transfer_config_path(cls, project, location, transfer_config):
+ """Return a fully-qualified location_transfer_config string."""
+ return google.api_core.path_template.expand(
+ "projects/{project}/locations/{location}/transferConfigs/{transfer_config}",
+ project=project,
+ location=location,
+ transfer_config=transfer_config,
+ )
+
+ @classmethod
+ def project_path(cls, project):
+ """Return a fully-qualified project string."""
+ return google.api_core.path_template.expand(
+ "projects/{project}", project=project
+ )
+
+ @classmethod
+ def project_data_source_path(cls, project, data_source):
+ """Return a fully-qualified project_data_source string."""
+ return google.api_core.path_template.expand(
+ "projects/{project}/dataSources/{data_source}",
+ project=project,
+ data_source=data_source,
+ )
+
+ @classmethod
+ def project_run_path(cls, project, transfer_config, run):
+ """Return a fully-qualified project_run string."""
+ return google.api_core.path_template.expand(
+ "projects/{project}/transferConfigs/{transfer_config}/runs/{run}",
+ project=project,
+ transfer_config=transfer_config,
+ run=run,
+ )
+
+ @classmethod
+ def project_transfer_config_path(cls, project, transfer_config):
+ """Return a fully-qualified project_transfer_config string."""
+ return google.api_core.path_template.expand(
+ "projects/{project}/transferConfigs/{transfer_config}",
+ project=project,
+ transfer_config=transfer_config,
+ )
+
+ def __init__(
+ self,
+ transport=None,
+ channel=None,
+ credentials=None,
+ client_config=None,
+ client_info=None,
+ client_options=None,
+ ):
+ """Constructor.
+
+ Args:
+ transport (Union[~.DataTransferServiceGrpcTransport,
+ Callable[[~.Credentials, type], ~.DataTransferServiceGrpcTransport]): A transport
+ instance, responsible for actually making the API calls.
+ The default transport uses the gRPC protocol.
+ This argument may also be a callable which returns a
+ transport instance. Callables will be sent the credentials
+ as the first argument and the default transport class as
+ the second argument.
+ channel (grpc.Channel): DEPRECATED. A ``Channel`` instance
+ through which to make calls. This argument is mutually exclusive
+ with ``credentials``; providing both will raise an exception.
+ credentials (google.auth.credentials.Credentials): The
+ authorization credentials to attach to requests. These
+ credentials identify this application to the service. If none
+ are specified, the client will attempt to ascertain the
+ credentials from the environment.
+ This argument is mutually exclusive with providing a
+ transport instance to ``transport``; doing so will raise
+ an exception.
+ client_config (dict): DEPRECATED. A dictionary of call options for
+ each method. If not specified, the default configuration is used.
+ client_info (google.api_core.gapic_v1.client_info.ClientInfo):
+ The client info used to send a user-agent string along with
+ API requests. If ``None``, then default info will be used.
+ Generally, you only need to set this if you're developing
+ your own client library.
+ client_options (Union[dict, google.api_core.client_options.ClientOptions]):
+ Client options used to set user options on the client. API Endpoint
+ should be set through client_options.
+ """
+ # Raise deprecation warnings for things we want to go away.
+ if client_config is not None:
+ warnings.warn(
+ "The `client_config` argument is deprecated.",
+ PendingDeprecationWarning,
+ stacklevel=2,
+ )
+ else:
+ client_config = data_transfer_service_client_config.config
+
+ if channel:
+ warnings.warn(
+ "The `channel` argument is deprecated; use " "`transport` instead.",
+ PendingDeprecationWarning,
+ stacklevel=2,
+ )
+
+ api_endpoint = self.SERVICE_ADDRESS
+ if client_options:
+ if type(client_options) == dict:
+ client_options = google.api_core.client_options.from_dict(
+ client_options
+ )
+ if client_options.api_endpoint:
+ api_endpoint = client_options.api_endpoint
+
+ # Instantiate the transport.
+ # The transport is responsible for handling serialization and
+ # deserialization and actually sending data to the service.
+ if transport:
+ if callable(transport):
+ self.transport = transport(
+ credentials=credentials,
+ default_class=data_transfer_service_grpc_transport.DataTransferServiceGrpcTransport,
+ address=api_endpoint,
+ )
+ else:
+ if credentials:
+ raise ValueError(
+ "Received both a transport instance and "
+ "credentials; these are mutually exclusive."
+ )
+ self.transport = transport
+ else:
+ self.transport = data_transfer_service_grpc_transport.DataTransferServiceGrpcTransport(
+ address=api_endpoint, channel=channel, credentials=credentials
+ )
+
+ if client_info is None:
+ client_info = google.api_core.gapic_v1.client_info.ClientInfo(
+ gapic_version=_GAPIC_LIBRARY_VERSION
+ )
+ else:
+ client_info.gapic_version = _GAPIC_LIBRARY_VERSION
+ self._client_info = client_info
+
+ # Parse out the default settings for retry and timeout for each RPC
+ # from the client configuration.
+ # (Ordinarily, these are the defaults specified in the `*_config.py`
+ # file next to this one.)
+ self._method_configs = google.api_core.gapic_v1.config.parse_method_configs(
+ client_config["interfaces"][self._INTERFACE_NAME]
+ )
+
+ # Save a dictionary of cached API call functions.
+ # These are the actual callables which invoke the proper
+ # transport methods, wrapped with `wrap_method` to add retry,
+ # timeout, and the like.
+ self._inner_api_calls = {}
+
+ # Service calls
+ def get_data_source(
+ self,
+ name,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Retrieves a supported data source and returns its settings,
+ which can be used for UI rendering.
+
+ Example:
+ >>> from google.cloud import bigquery_datatransfer_v1
+ >>>
+ >>> client = bigquery_datatransfer_v1.DataTransferServiceClient()
+ >>>
+ >>> name = client.project_data_source_path('[PROJECT]', '[DATA_SOURCE]')
+ >>>
+ >>> response = client.get_data_source(name)
+
+ Args:
+ name (str): The field will contain name of the resource requested, for example:
+ ``projects/{project_id}/dataSources/{data_source_id}``
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.bigquery_datatransfer_v1.types.DataSource` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "get_data_source" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "get_data_source"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.get_data_source,
+ default_retry=self._method_configs["GetDataSource"].retry,
+ default_timeout=self._method_configs["GetDataSource"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = datatransfer_pb2.GetDataSourceRequest(name=name)
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("name", name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ return self._inner_api_calls["get_data_source"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+
+ def list_data_sources(
+ self,
+ parent,
+ page_size=None,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Lists supported data sources and returns their settings,
+ which can be used for UI rendering.
+
+ Example:
+ >>> from google.cloud import bigquery_datatransfer_v1
+ >>>
+ >>> client = bigquery_datatransfer_v1.DataTransferServiceClient()
+ >>>
+ >>> parent = client.project_path('[PROJECT]')
+ >>>
+ >>> # Iterate over all results
+ >>> for element in client.list_data_sources(parent):
+ ... # process element
+ ... pass
+ >>>
+ >>>
+ >>> # Alternatively:
+ >>>
+ >>> # Iterate over results one page at a time
+ >>> for page in client.list_data_sources(parent).pages:
+ ... for element in page:
+ ... # process element
+ ... pass
+
+ Args:
+ parent (str): The BigQuery project id for which data sources should be returned. Must
+ be in the form: ``projects/{project_id}``
+ page_size (int): The maximum number of resources contained in the
+ underlying API response. If page streaming is performed per-
+ resource, this parameter does not affect the return value. If page
+ streaming is performed per-page, this determines the maximum number
+ of resources in a page.
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.api_core.page_iterator.PageIterator` instance.
+ An iterable of :class:`~google.cloud.bigquery_datatransfer_v1.types.DataSource` instances.
+ You can also iterate over the pages of the response
+ using its `pages` property.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "list_data_sources" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "list_data_sources"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.list_data_sources,
+ default_retry=self._method_configs["ListDataSources"].retry,
+ default_timeout=self._method_configs["ListDataSources"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = datatransfer_pb2.ListDataSourcesRequest(
+ parent=parent, page_size=page_size
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("parent", parent)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ iterator = google.api_core.page_iterator.GRPCIterator(
+ client=None,
+ method=functools.partial(
+ self._inner_api_calls["list_data_sources"],
+ retry=retry,
+ timeout=timeout,
+ metadata=metadata,
+ ),
+ request=request,
+ items_field="data_sources",
+ request_token_field="page_token",
+ response_token_field="next_page_token",
+ )
+ return iterator
+
+ def create_transfer_config(
+ self,
+ parent,
+ transfer_config,
+ authorization_code=None,
+ version_info=None,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Creates a new data transfer configuration.
+
+ Example:
+ >>> from google.cloud import bigquery_datatransfer_v1
+ >>>
+ >>> client = bigquery_datatransfer_v1.DataTransferServiceClient()
+ >>>
+ >>> parent = client.project_path('[PROJECT]')
+ >>>
+ >>> # TODO: Initialize `transfer_config`:
+ >>> transfer_config = {}
+ >>>
+ >>> response = client.create_transfer_config(parent, transfer_config)
+
+ Args:
+ parent (str): The BigQuery project id where the transfer configuration should be
+ created. Must be in the format
+ projects/{project\_id}/locations/{location\_id} If specified location
+ and location of the destination bigquery dataset do not match - the
+ request will fail.
+ transfer_config (Union[dict, ~google.cloud.bigquery_datatransfer_v1.types.TransferConfig]): Data transfer configuration to create.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.bigquery_datatransfer_v1.types.TransferConfig`
+ authorization_code (str): Optional OAuth2 authorization code to use with this transfer
+ configuration. This is required if new credentials are needed, as
+ indicated by ``CheckValidCreds``. In order to obtain
+ authorization\_code, please make a request to
+ https://www.gstatic.com/bigquerydatatransfer/oauthz/auth?client\_id=&scope=&redirect\_uri=
+
+ - client\_id should be OAuth client\_id of BigQuery DTS API for the
+ given data source returned by ListDataSources method.
+ - data\_source\_scopes are the scopes returned by ListDataSources
+ method.
+ - redirect\_uri is an optional parameter. If not specified, then
+ authorization code is posted to the opener of authorization flow
+ window. Otherwise it will be sent to the redirect uri. A special
+ value of urn:ietf:wg:oauth:2.0:oob means that authorization code
+ should be returned in the title bar of the browser, with the page
+ text prompting the user to copy the code and paste it in the
+ application.
+ version_info (str): Optional version info. If users want to find a very recent access token,
+ that is, immediately after approving access, users have to set the
+ version\_info claim in the token request. To obtain the version\_info,
+ users must use the "none+gsession" response type. which be return a
+ version\_info back in the authorization response which be be put in a
+ JWT claim in the token request.
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.bigquery_datatransfer_v1.types.TransferConfig` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "create_transfer_config" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "create_transfer_config"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.create_transfer_config,
+ default_retry=self._method_configs["CreateTransferConfig"].retry,
+ default_timeout=self._method_configs["CreateTransferConfig"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = datatransfer_pb2.CreateTransferConfigRequest(
+ parent=parent,
+ transfer_config=transfer_config,
+ authorization_code=authorization_code,
+ version_info=version_info,
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("parent", parent)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ return self._inner_api_calls["create_transfer_config"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+
+ def update_transfer_config(
+ self,
+ transfer_config,
+ update_mask,
+ authorization_code=None,
+ version_info=None,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Updates a data transfer configuration.
+ All fields must be set, even if they are not updated.
+
+ Example:
+ >>> from google.cloud import bigquery_datatransfer_v1
+ >>>
+ >>> client = bigquery_datatransfer_v1.DataTransferServiceClient()
+ >>>
+ >>> # TODO: Initialize `transfer_config`:
+ >>> transfer_config = {}
+ >>>
+ >>> # TODO: Initialize `update_mask`:
+ >>> update_mask = {}
+ >>>
+ >>> response = client.update_transfer_config(transfer_config, update_mask)
+
+ Args:
+ transfer_config (Union[dict, ~google.cloud.bigquery_datatransfer_v1.types.TransferConfig]): Data transfer configuration to create.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.bigquery_datatransfer_v1.types.TransferConfig`
+ update_mask (Union[dict, ~google.cloud.bigquery_datatransfer_v1.types.FieldMask]): Required list of fields to be updated in this request.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.bigquery_datatransfer_v1.types.FieldMask`
+ authorization_code (str): Optional OAuth2 authorization code to use with this transfer
+ configuration. If it is provided, the transfer configuration will be
+ associated with the authorizing user. In order to obtain
+ authorization\_code, please make a request to
+ https://www.gstatic.com/bigquerydatatransfer/oauthz/auth?client\_id=&scope=&redirect\_uri=
+
+ - client\_id should be OAuth client\_id of BigQuery DTS API for the
+ given data source returned by ListDataSources method.
+ - data\_source\_scopes are the scopes returned by ListDataSources
+ method.
+ - redirect\_uri is an optional parameter. If not specified, then
+ authorization code is posted to the opener of authorization flow
+ window. Otherwise it will be sent to the redirect uri. A special
+ value of urn:ietf:wg:oauth:2.0:oob means that authorization code
+ should be returned in the title bar of the browser, with the page
+ text prompting the user to copy the code and paste it in the
+ application.
+ version_info (str): Optional version info. If users want to find a very recent access token,
+ that is, immediately after approving access, users have to set the
+ version\_info claim in the token request. To obtain the version\_info,
+ users must use the "none+gsession" response type. which be return a
+ version\_info back in the authorization response which be be put in a
+ JWT claim in the token request.
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.bigquery_datatransfer_v1.types.TransferConfig` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "update_transfer_config" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "update_transfer_config"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.update_transfer_config,
+ default_retry=self._method_configs["UpdateTransferConfig"].retry,
+ default_timeout=self._method_configs["UpdateTransferConfig"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = datatransfer_pb2.UpdateTransferConfigRequest(
+ transfer_config=transfer_config,
+ update_mask=update_mask,
+ authorization_code=authorization_code,
+ version_info=version_info,
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("transfer_config.name", transfer_config.name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ return self._inner_api_calls["update_transfer_config"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+
+ def delete_transfer_config(
+ self,
+ name,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Deletes a data transfer configuration,
+ including any associated transfer runs and logs.
+
+ Example:
+ >>> from google.cloud import bigquery_datatransfer_v1
+ >>>
+ >>> client = bigquery_datatransfer_v1.DataTransferServiceClient()
+ >>>
+ >>> name = client.project_transfer_config_path('[PROJECT]', '[TRANSFER_CONFIG]')
+ >>>
+ >>> client.delete_transfer_config(name)
+
+ Args:
+ name (str): The field will contain name of the resource requested, for example:
+ ``projects/{project_id}/transferConfigs/{config_id}``
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "delete_transfer_config" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "delete_transfer_config"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.delete_transfer_config,
+ default_retry=self._method_configs["DeleteTransferConfig"].retry,
+ default_timeout=self._method_configs["DeleteTransferConfig"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = datatransfer_pb2.DeleteTransferConfigRequest(name=name)
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("name", name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ self._inner_api_calls["delete_transfer_config"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+
+ def get_transfer_config(
+ self,
+ name,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Returns information about a data transfer config.
+
+ Example:
+ >>> from google.cloud import bigquery_datatransfer_v1
+ >>>
+ >>> client = bigquery_datatransfer_v1.DataTransferServiceClient()
+ >>>
+ >>> name = client.project_transfer_config_path('[PROJECT]', '[TRANSFER_CONFIG]')
+ >>>
+ >>> response = client.get_transfer_config(name)
+
+ Args:
+ name (str): The field will contain name of the resource requested, for example:
+ ``projects/{project_id}/transferConfigs/{config_id}``
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.bigquery_datatransfer_v1.types.TransferConfig` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "get_transfer_config" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "get_transfer_config"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.get_transfer_config,
+ default_retry=self._method_configs["GetTransferConfig"].retry,
+ default_timeout=self._method_configs["GetTransferConfig"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = datatransfer_pb2.GetTransferConfigRequest(name=name)
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("name", name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ return self._inner_api_calls["get_transfer_config"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+
+ def list_transfer_configs(
+ self,
+ parent,
+ data_source_ids=None,
+ page_size=None,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Returns information about all data transfers in the project.
+
+ Example:
+ >>> from google.cloud import bigquery_datatransfer_v1
+ >>>
+ >>> client = bigquery_datatransfer_v1.DataTransferServiceClient()
+ >>>
+ >>> parent = client.project_path('[PROJECT]')
+ >>>
+ >>> # Iterate over all results
+ >>> for element in client.list_transfer_configs(parent):
+ ... # process element
+ ... pass
+ >>>
+ >>>
+ >>> # Alternatively:
+ >>>
+ >>> # Iterate over results one page at a time
+ >>> for page in client.list_transfer_configs(parent).pages:
+ ... for element in page:
+ ... # process element
+ ... pass
+
+ Args:
+ parent (str): The BigQuery project id for which data sources should be returned:
+ ``projects/{project_id}``.
+ data_source_ids (list[str]): When specified, only configurations of requested data sources are returned.
+ page_size (int): The maximum number of resources contained in the
+ underlying API response. If page streaming is performed per-
+ resource, this parameter does not affect the return value. If page
+ streaming is performed per-page, this determines the maximum number
+ of resources in a page.
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.api_core.page_iterator.PageIterator` instance.
+ An iterable of :class:`~google.cloud.bigquery_datatransfer_v1.types.TransferConfig` instances.
+ You can also iterate over the pages of the response
+ using its `pages` property.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "list_transfer_configs" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "list_transfer_configs"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.list_transfer_configs,
+ default_retry=self._method_configs["ListTransferConfigs"].retry,
+ default_timeout=self._method_configs["ListTransferConfigs"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = datatransfer_pb2.ListTransferConfigsRequest(
+ parent=parent, data_source_ids=data_source_ids, page_size=page_size
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("parent", parent)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ iterator = google.api_core.page_iterator.GRPCIterator(
+ client=None,
+ method=functools.partial(
+ self._inner_api_calls["list_transfer_configs"],
+ retry=retry,
+ timeout=timeout,
+ metadata=metadata,
+ ),
+ request=request,
+ items_field="transfer_configs",
+ request_token_field="page_token",
+ response_token_field="next_page_token",
+ )
+ return iterator
+
+ def schedule_transfer_runs(
+ self,
+ parent,
+ start_time,
+ end_time,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Creates transfer runs for a time range [start\_time, end\_time]. For
+ each date - or whatever granularity the data source supports - in the
+ range, one transfer run is created. Note that runs are created per UTC
+ time in the time range. DEPRECATED: use StartManualTransferRuns instead.
+
+ Example:
+ >>> from google.cloud import bigquery_datatransfer_v1
+ >>>
+ >>> client = bigquery_datatransfer_v1.DataTransferServiceClient()
+ >>>
+ >>> parent = client.project_transfer_config_path('[PROJECT]', '[TRANSFER_CONFIG]')
+ >>>
+ >>> # TODO: Initialize `start_time`:
+ >>> start_time = {}
+ >>>
+ >>> # TODO: Initialize `end_time`:
+ >>> end_time = {}
+ >>>
+ >>> response = client.schedule_transfer_runs(parent, start_time, end_time)
+
+ Args:
+ parent (str): Transfer configuration name in the form:
+ ``projects/{project_id}/transferConfigs/{config_id}``.
+ start_time (Union[dict, ~google.cloud.bigquery_datatransfer_v1.types.Timestamp]): Start time of the range of transfer runs. For example,
+ ``"2017-05-25T00:00:00+00:00"``.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.bigquery_datatransfer_v1.types.Timestamp`
+ end_time (Union[dict, ~google.cloud.bigquery_datatransfer_v1.types.Timestamp]): End time of the range of transfer runs. For example,
+ ``"2017-05-30T00:00:00+00:00"``.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.bigquery_datatransfer_v1.types.Timestamp`
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.bigquery_datatransfer_v1.types.ScheduleTransferRunsResponse` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "schedule_transfer_runs" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "schedule_transfer_runs"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.schedule_transfer_runs,
+ default_retry=self._method_configs["ScheduleTransferRuns"].retry,
+ default_timeout=self._method_configs["ScheduleTransferRuns"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = datatransfer_pb2.ScheduleTransferRunsRequest(
+ parent=parent, start_time=start_time, end_time=end_time
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("parent", parent)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ return self._inner_api_calls["schedule_transfer_runs"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+
+ def get_transfer_run(
+ self,
+ name,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Returns information about the particular transfer run.
+
+ Example:
+ >>> from google.cloud import bigquery_datatransfer_v1
+ >>>
+ >>> client = bigquery_datatransfer_v1.DataTransferServiceClient()
+ >>>
+ >>> name = client.project_run_path('[PROJECT]', '[TRANSFER_CONFIG]', '[RUN]')
+ >>>
+ >>> response = client.get_transfer_run(name)
+
+ Args:
+ name (str): The field will contain name of the resource requested, for example:
+ ``projects/{project_id}/transferConfigs/{config_id}/runs/{run_id}``
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.bigquery_datatransfer_v1.types.TransferRun` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "get_transfer_run" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "get_transfer_run"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.get_transfer_run,
+ default_retry=self._method_configs["GetTransferRun"].retry,
+ default_timeout=self._method_configs["GetTransferRun"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = datatransfer_pb2.GetTransferRunRequest(name=name)
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("name", name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ return self._inner_api_calls["get_transfer_run"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+
+ def delete_transfer_run(
+ self,
+ name,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Deletes the specified transfer run.
+
+ Example:
+ >>> from google.cloud import bigquery_datatransfer_v1
+ >>>
+ >>> client = bigquery_datatransfer_v1.DataTransferServiceClient()
+ >>>
+ >>> name = client.project_run_path('[PROJECT]', '[TRANSFER_CONFIG]', '[RUN]')
+ >>>
+ >>> client.delete_transfer_run(name)
+
+ Args:
+ name (str): The field will contain name of the resource requested, for example:
+ ``projects/{project_id}/transferConfigs/{config_id}/runs/{run_id}``
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "delete_transfer_run" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "delete_transfer_run"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.delete_transfer_run,
+ default_retry=self._method_configs["DeleteTransferRun"].retry,
+ default_timeout=self._method_configs["DeleteTransferRun"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = datatransfer_pb2.DeleteTransferRunRequest(name=name)
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("name", name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ self._inner_api_calls["delete_transfer_run"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+
+ def list_transfer_runs(
+ self,
+ parent,
+ states=None,
+ page_size=None,
+ run_attempt=None,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Returns information about running and completed jobs.
+
+ Example:
+ >>> from google.cloud import bigquery_datatransfer_v1
+ >>>
+ >>> client = bigquery_datatransfer_v1.DataTransferServiceClient()
+ >>>
+ >>> parent = client.project_transfer_config_path('[PROJECT]', '[TRANSFER_CONFIG]')
+ >>>
+ >>> # Iterate over all results
+ >>> for element in client.list_transfer_runs(parent):
+ ... # process element
+ ... pass
+ >>>
+ >>>
+ >>> # Alternatively:
+ >>>
+ >>> # Iterate over results one page at a time
+ >>> for page in client.list_transfer_runs(parent).pages:
+ ... for element in page:
+ ... # process element
+ ... pass
+
+ Args:
+ parent (str): Name of transfer configuration for which transfer runs should be
+ retrieved. Format of transfer configuration resource name is:
+ ``projects/{project_id}/transferConfigs/{config_id}``.
+ states (list[~google.cloud.bigquery_datatransfer_v1.types.TransferState]): When specified, only transfer runs with requested states are returned.
+ page_size (int): The maximum number of resources contained in the
+ underlying API response. If page streaming is performed per-
+ resource, this parameter does not affect the return value. If page
+ streaming is performed per-page, this determines the maximum number
+ of resources in a page.
+ run_attempt (~google.cloud.bigquery_datatransfer_v1.types.RunAttempt): Indicates how run attempts are to be pulled.
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.api_core.page_iterator.PageIterator` instance.
+ An iterable of :class:`~google.cloud.bigquery_datatransfer_v1.types.TransferRun` instances.
+ You can also iterate over the pages of the response
+ using its `pages` property.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "list_transfer_runs" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "list_transfer_runs"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.list_transfer_runs,
+ default_retry=self._method_configs["ListTransferRuns"].retry,
+ default_timeout=self._method_configs["ListTransferRuns"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = datatransfer_pb2.ListTransferRunsRequest(
+ parent=parent, states=states, page_size=page_size, run_attempt=run_attempt
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("parent", parent)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ iterator = google.api_core.page_iterator.GRPCIterator(
+ client=None,
+ method=functools.partial(
+ self._inner_api_calls["list_transfer_runs"],
+ retry=retry,
+ timeout=timeout,
+ metadata=metadata,
+ ),
+ request=request,
+ items_field="transfer_runs",
+ request_token_field="page_token",
+ response_token_field="next_page_token",
+ )
+ return iterator
+
+ def list_transfer_logs(
+ self,
+ parent,
+ page_size=None,
+ message_types=None,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Returns user facing log messages for the data transfer run.
+
+ Example:
+ >>> from google.cloud import bigquery_datatransfer_v1
+ >>>
+ >>> client = bigquery_datatransfer_v1.DataTransferServiceClient()
+ >>>
+ >>> parent = client.project_run_path('[PROJECT]', '[TRANSFER_CONFIG]', '[RUN]')
+ >>>
+ >>> # Iterate over all results
+ >>> for element in client.list_transfer_logs(parent):
+ ... # process element
+ ... pass
+ >>>
+ >>>
+ >>> # Alternatively:
+ >>>
+ >>> # Iterate over results one page at a time
+ >>> for page in client.list_transfer_logs(parent).pages:
+ ... for element in page:
+ ... # process element
+ ... pass
+
+ Args:
+ parent (str): Transfer run name in the form:
+ ``projects/{project_id}/transferConfigs/{config_Id}/runs/{run_id}``.
+ page_size (int): The maximum number of resources contained in the
+ underlying API response. If page streaming is performed per-
+ resource, this parameter does not affect the return value. If page
+ streaming is performed per-page, this determines the maximum number
+ of resources in a page.
+ message_types (list[~google.cloud.bigquery_datatransfer_v1.types.MessageSeverity]): Message types to return. If not populated - INFO, WARNING and ERROR
+ messages are returned.
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.api_core.page_iterator.PageIterator` instance.
+ An iterable of :class:`~google.cloud.bigquery_datatransfer_v1.types.TransferMessage` instances.
+ You can also iterate over the pages of the response
+ using its `pages` property.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "list_transfer_logs" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "list_transfer_logs"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.list_transfer_logs,
+ default_retry=self._method_configs["ListTransferLogs"].retry,
+ default_timeout=self._method_configs["ListTransferLogs"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = datatransfer_pb2.ListTransferLogsRequest(
+ parent=parent, page_size=page_size, message_types=message_types
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("parent", parent)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ iterator = google.api_core.page_iterator.GRPCIterator(
+ client=None,
+ method=functools.partial(
+ self._inner_api_calls["list_transfer_logs"],
+ retry=retry,
+ timeout=timeout,
+ metadata=metadata,
+ ),
+ request=request,
+ items_field="transfer_messages",
+ request_token_field="page_token",
+ response_token_field="next_page_token",
+ )
+ return iterator
+
+ def check_valid_creds(
+ self,
+ name,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Returns true if valid credentials exist for the given data source and
+ requesting user.
+ Some data sources doesn't support service account, so we need to talk to
+ them on behalf of the end user. This API just checks whether we have OAuth
+ token for the particular user, which is a pre-requisite before user can
+ create a transfer config.
+
+ Example:
+ >>> from google.cloud import bigquery_datatransfer_v1
+ >>>
+ >>> client = bigquery_datatransfer_v1.DataTransferServiceClient()
+ >>>
+ >>> name = client.project_data_source_path('[PROJECT]', '[DATA_SOURCE]')
+ >>>
+ >>> response = client.check_valid_creds(name)
+
+ Args:
+ name (str): The data source in the form:
+ ``projects/{project_id}/dataSources/{data_source_id}``
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.bigquery_datatransfer_v1.types.CheckValidCredsResponse` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "check_valid_creds" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "check_valid_creds"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.check_valid_creds,
+ default_retry=self._method_configs["CheckValidCreds"].retry,
+ default_timeout=self._method_configs["CheckValidCreds"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = datatransfer_pb2.CheckValidCredsRequest(name=name)
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("name", name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ return self._inner_api_calls["check_valid_creds"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+
+ def start_manual_transfer_runs(
+ self,
+ parent=None,
+ requested_time_range=None,
+ requested_run_time=None,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Start manual transfer runs to be executed now with schedule\_time equal
+ to current time. The transfer runs can be created for a time range where
+ the run\_time is between start\_time (inclusive) and end\_time
+ (exclusive), or for a specific run\_time.
+
+ Example:
+ >>> from google.cloud import bigquery_datatransfer_v1
+ >>>
+ >>> client = bigquery_datatransfer_v1.DataTransferServiceClient()
+ >>>
+ >>> response = client.start_manual_transfer_runs()
+
+ Args:
+ parent (str): Transfer configuration name in the form:
+ ``projects/{project_id}/transferConfigs/{config_id}``.
+ requested_time_range (Union[dict, ~google.cloud.bigquery_datatransfer_v1.types.TimeRange]): Time range for the transfer runs that should be started.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.bigquery_datatransfer_v1.types.TimeRange`
+ requested_run_time (Union[dict, ~google.cloud.bigquery_datatransfer_v1.types.Timestamp]): Specific run\_time for a transfer run to be started. The
+ requested\_run\_time must not be in the future.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.bigquery_datatransfer_v1.types.Timestamp`
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.bigquery_datatransfer_v1.types.StartManualTransferRunsResponse` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "start_manual_transfer_runs" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "start_manual_transfer_runs"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.start_manual_transfer_runs,
+ default_retry=self._method_configs["StartManualTransferRuns"].retry,
+ default_timeout=self._method_configs["StartManualTransferRuns"].timeout,
+ client_info=self._client_info,
+ )
+
+ # Sanity check: We have some fields which are mutually exclusive;
+ # raise ValueError if more than one is sent.
+ google.api_core.protobuf_helpers.check_oneof(
+ requested_time_range=requested_time_range,
+ requested_run_time=requested_run_time,
+ )
+
+ request = datatransfer_pb2.StartManualTransferRunsRequest(
+ parent=parent,
+ requested_time_range=requested_time_range,
+ requested_run_time=requested_run_time,
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("parent", parent)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata)
+
+ return self._inner_api_calls["start_manual_transfer_runs"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
diff --git a/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/gapic/data_transfer_service_client_config.py b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/gapic/data_transfer_service_client_config.py
new file mode 100644
index 000000000000..28a9494f22ef
--- /dev/null
+++ b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/gapic/data_transfer_service_client_config.py
@@ -0,0 +1,93 @@
+config = {
+ "interfaces": {
+ "google.cloud.bigquery.datatransfer.v1.DataTransferService": {
+ "retry_codes": {
+ "idempotent": ["DEADLINE_EXCEEDED", "UNAVAILABLE"],
+ "non_idempotent": [],
+ },
+ "retry_params": {
+ "default": {
+ "initial_retry_delay_millis": 100,
+ "retry_delay_multiplier": 1.3,
+ "max_retry_delay_millis": 60000,
+ "initial_rpc_timeout_millis": 20000,
+ "rpc_timeout_multiplier": 1.0,
+ "max_rpc_timeout_millis": 20000,
+ "total_timeout_millis": 600000,
+ }
+ },
+ "methods": {
+ "GetDataSource": {
+ "timeout_millis": 20000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "ListDataSources": {
+ "timeout_millis": 20000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "CreateTransferConfig": {
+ "timeout_millis": 30000,
+ "retry_codes_name": "non_idempotent",
+ "retry_params_name": "default",
+ },
+ "UpdateTransferConfig": {
+ "timeout_millis": 30000,
+ "retry_codes_name": "non_idempotent",
+ "retry_params_name": "default",
+ },
+ "DeleteTransferConfig": {
+ "timeout_millis": 30000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "GetTransferConfig": {
+ "timeout_millis": 30000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "ListTransferConfigs": {
+ "timeout_millis": 30000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "ScheduleTransferRuns": {
+ "timeout_millis": 30000,
+ "retry_codes_name": "non_idempotent",
+ "retry_params_name": "default",
+ },
+ "GetTransferRun": {
+ "timeout_millis": 30000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "DeleteTransferRun": {
+ "timeout_millis": 30000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "ListTransferRuns": {
+ "timeout_millis": 30000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "ListTransferLogs": {
+ "timeout_millis": 30000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "CheckValidCreds": {
+ "timeout_millis": 30000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "StartManualTransferRuns": {
+ "timeout_millis": 60000,
+ "retry_codes_name": "non_idempotent",
+ "retry_params_name": "default",
+ },
+ },
+ }
+ }
+}
diff --git a/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/gapic/enums.py b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/gapic/enums.py
new file mode 100644
index 000000000000..2602eca70f2e
--- /dev/null
+++ b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/gapic/enums.py
@@ -0,0 +1,164 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Wrappers for protocol buffer enum types."""
+
+import enum
+
+
+class NullValue(enum.IntEnum):
+ """
+ ``NullValue`` is a singleton enumeration to represent the null value for
+ the ``Value`` type union.
+
+ The JSON representation for ``NullValue`` is JSON ``null``.
+
+ Attributes:
+ NULL_VALUE (int): Null value.
+ """
+
+ NULL_VALUE = 0
+
+
+class TransferState(enum.IntEnum):
+ """
+ Represents data transfer run state.
+
+ Attributes:
+ TRANSFER_STATE_UNSPECIFIED (int): State placeholder.
+ PENDING (int): Data transfer is scheduled and is waiting to be picked up by
+ data transfer backend.
+ RUNNING (int): Data transfer is in progress.
+ SUCCEEDED (int): Data transfer completed successfully.
+ FAILED (int): Data transfer failed.
+ CANCELLED (int): Data transfer is cancelled.
+ """
+
+ TRANSFER_STATE_UNSPECIFIED = 0
+ PENDING = 2
+ RUNNING = 3
+ SUCCEEDED = 4
+ FAILED = 5
+ CANCELLED = 6
+
+
+class TransferType(enum.IntEnum):
+ """
+ DEPRECATED. Represents data transfer type.
+
+ Attributes:
+ TRANSFER_TYPE_UNSPECIFIED (int): Invalid or Unknown transfer type placeholder.
+ BATCH (int): Batch data transfer.
+ STREAMING (int): Streaming data transfer. Streaming data source currently doesn't
+ support multiple transfer configs per project.
+ """
+
+ TRANSFER_TYPE_UNSPECIFIED = 0
+ BATCH = 1
+ STREAMING = 2
+
+
+class DataSource(object):
+ class AuthorizationType(enum.IntEnum):
+ """
+ The type of authorization needed for this data source.
+
+ Attributes:
+ AUTHORIZATION_TYPE_UNSPECIFIED (int): Type unspecified.
+ AUTHORIZATION_CODE (int): Use OAuth 2 authorization codes that can be exchanged
+ for a refresh token on the backend.
+ GOOGLE_PLUS_AUTHORIZATION_CODE (int): Return an authorization code for a given Google+ page that can then be
+ exchanged for a refresh token on the backend.
+ """
+
+ AUTHORIZATION_TYPE_UNSPECIFIED = 0
+ AUTHORIZATION_CODE = 1
+ GOOGLE_PLUS_AUTHORIZATION_CODE = 2
+
+ class DataRefreshType(enum.IntEnum):
+ """
+ Represents how the data source supports data auto refresh.
+
+ Attributes:
+ DATA_REFRESH_TYPE_UNSPECIFIED (int): The data source won't support data auto refresh, which is default value.
+ SLIDING_WINDOW (int): The data source supports data auto refresh, and runs will be scheduled
+ for the past few days. Does not allow custom values to be set for each
+ transfer config.
+ CUSTOM_SLIDING_WINDOW (int): The data source supports data auto refresh, and runs will be scheduled
+ for the past few days. Allows custom values to be set for each transfer
+ config.
+ """
+
+ DATA_REFRESH_TYPE_UNSPECIFIED = 0
+ SLIDING_WINDOW = 1
+ CUSTOM_SLIDING_WINDOW = 2
+
+
+class DataSourceParameter(object):
+ class Type(enum.IntEnum):
+ """
+ Parameter type.
+
+ Attributes:
+ TYPE_UNSPECIFIED (int): Type unspecified.
+ STRING (int): String parameter.
+ INTEGER (int): Integer parameter (64-bits).
+ Will be serialized to json as string.
+ DOUBLE (int): Double precision floating point parameter.
+ BOOLEAN (int): Boolean parameter.
+ RECORD (int): Deprecated. This field has no effect.
+ PLUS_PAGE (int): Page ID for a Google+ Page.
+ """
+
+ TYPE_UNSPECIFIED = 0
+ STRING = 1
+ INTEGER = 2
+ DOUBLE = 3
+ BOOLEAN = 4
+ RECORD = 5
+ PLUS_PAGE = 6
+
+
+class ListTransferRunsRequest(object):
+ class RunAttempt(enum.IntEnum):
+ """
+ Represents which runs should be pulled.
+
+ Attributes:
+ RUN_ATTEMPT_UNSPECIFIED (int): All runs should be returned.
+ LATEST (int): Only latest run per day should be returned.
+ """
+
+ RUN_ATTEMPT_UNSPECIFIED = 0
+ LATEST = 1
+
+
+class TransferMessage(object):
+ class MessageSeverity(enum.IntEnum):
+ """
+ Represents data transfer user facing message severity.
+
+ Attributes:
+ MESSAGE_SEVERITY_UNSPECIFIED (int): No severity specified.
+ INFO (int): Informational message.
+ WARNING (int): Warning message.
+ ERROR (int): Error message.
+ """
+
+ MESSAGE_SEVERITY_UNSPECIFIED = 0
+ INFO = 1
+ WARNING = 2
+ ERROR = 3
diff --git a/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/gapic/transports/__init__.py b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/gapic/transports/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/gapic/transports/data_transfer_service_grpc_transport.py b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/gapic/transports/data_transfer_service_grpc_transport.py
new file mode 100644
index 000000000000..fa84cac3ef30
--- /dev/null
+++ b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/gapic/transports/data_transfer_service_grpc_transport.py
@@ -0,0 +1,313 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import google.api_core.grpc_helpers
+
+from google.cloud.bigquery_datatransfer_v1.proto import datatransfer_pb2_grpc
+
+
+class DataTransferServiceGrpcTransport(object):
+ """gRPC transport class providing stubs for
+ google.cloud.bigquery.datatransfer.v1 DataTransferService API.
+
+ The transport provides access to the raw gRPC stubs,
+ which can be used to take advantage of advanced
+ features of gRPC.
+ """
+
+ # The scopes needed to make gRPC calls to all of the methods defined
+ # in this service.
+ _OAUTH_SCOPES = ("https://www.googleapis.com/auth/cloud-platform",)
+
+ def __init__(
+ self,
+ channel=None,
+ credentials=None,
+ address="bigquerydatatransfer.googleapis.com:443",
+ ):
+ """Instantiate the transport class.
+
+ Args:
+ channel (grpc.Channel): A ``Channel`` instance through
+ which to make calls. This argument is mutually exclusive
+ with ``credentials``; providing both will raise an exception.
+ credentials (google.auth.credentials.Credentials): The
+ authorization credentials to attach to requests. These
+ credentials identify this application to the service. If none
+ are specified, the client will attempt to ascertain the
+ credentials from the environment.
+ address (str): The address where the service is hosted.
+ """
+ # If both `channel` and `credentials` are specified, raise an
+ # exception (channels come with credentials baked in already).
+ if channel is not None and credentials is not None:
+ raise ValueError(
+ "The `channel` and `credentials` arguments are mutually " "exclusive."
+ )
+
+ # Create the channel.
+ if channel is None:
+ channel = self.create_channel(
+ address=address,
+ credentials=credentials,
+ options={
+ "grpc.max_send_message_length": -1,
+ "grpc.max_receive_message_length": -1,
+ }.items(),
+ )
+
+ self._channel = channel
+
+ # gRPC uses objects called "stubs" that are bound to the
+ # channel and provide a basic method for each RPC.
+ self._stubs = {
+ "data_transfer_service_stub": datatransfer_pb2_grpc.DataTransferServiceStub(
+ channel
+ )
+ }
+
+ @classmethod
+ def create_channel(
+ cls,
+ address="bigquerydatatransfer.googleapis.com:443",
+ credentials=None,
+ **kwargs
+ ):
+ """Create and return a gRPC channel object.
+
+ Args:
+ address (str): The host for the channel to use.
+ credentials (~.Credentials): The
+ authorization credentials to attach to requests. These
+ credentials identify this application to the service. If
+ none are specified, the client will attempt to ascertain
+ the credentials from the environment.
+ kwargs (dict): Keyword arguments, which are passed to the
+ channel creation.
+
+ Returns:
+ grpc.Channel: A gRPC channel object.
+ """
+ return google.api_core.grpc_helpers.create_channel(
+ address, credentials=credentials, scopes=cls._OAUTH_SCOPES, **kwargs
+ )
+
+ @property
+ def channel(self):
+ """The gRPC channel used by the transport.
+
+ Returns:
+ grpc.Channel: A gRPC channel object.
+ """
+ return self._channel
+
+ @property
+ def get_data_source(self):
+ """Return the gRPC stub for :meth:`DataTransferServiceClient.get_data_source`.
+
+ Retrieves a supported data source and returns its settings,
+ which can be used for UI rendering.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["data_transfer_service_stub"].GetDataSource
+
+ @property
+ def list_data_sources(self):
+ """Return the gRPC stub for :meth:`DataTransferServiceClient.list_data_sources`.
+
+ Lists supported data sources and returns their settings,
+ which can be used for UI rendering.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["data_transfer_service_stub"].ListDataSources
+
+ @property
+ def create_transfer_config(self):
+ """Return the gRPC stub for :meth:`DataTransferServiceClient.create_transfer_config`.
+
+ Creates a new data transfer configuration.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["data_transfer_service_stub"].CreateTransferConfig
+
+ @property
+ def update_transfer_config(self):
+ """Return the gRPC stub for :meth:`DataTransferServiceClient.update_transfer_config`.
+
+ Updates a data transfer configuration.
+ All fields must be set, even if they are not updated.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["data_transfer_service_stub"].UpdateTransferConfig
+
+ @property
+ def delete_transfer_config(self):
+ """Return the gRPC stub for :meth:`DataTransferServiceClient.delete_transfer_config`.
+
+ Deletes a data transfer configuration,
+ including any associated transfer runs and logs.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["data_transfer_service_stub"].DeleteTransferConfig
+
+ @property
+ def get_transfer_config(self):
+ """Return the gRPC stub for :meth:`DataTransferServiceClient.get_transfer_config`.
+
+ Returns information about a data transfer config.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["data_transfer_service_stub"].GetTransferConfig
+
+ @property
+ def list_transfer_configs(self):
+ """Return the gRPC stub for :meth:`DataTransferServiceClient.list_transfer_configs`.
+
+ Returns information about all data transfers in the project.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["data_transfer_service_stub"].ListTransferConfigs
+
+ @property
+ def schedule_transfer_runs(self):
+ """Return the gRPC stub for :meth:`DataTransferServiceClient.schedule_transfer_runs`.
+
+ Creates transfer runs for a time range [start\_time, end\_time]. For
+ each date - or whatever granularity the data source supports - in the
+ range, one transfer run is created. Note that runs are created per UTC
+ time in the time range. DEPRECATED: use StartManualTransferRuns instead.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["data_transfer_service_stub"].ScheduleTransferRuns
+
+ @property
+ def get_transfer_run(self):
+ """Return the gRPC stub for :meth:`DataTransferServiceClient.get_transfer_run`.
+
+ Returns information about the particular transfer run.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["data_transfer_service_stub"].GetTransferRun
+
+ @property
+ def delete_transfer_run(self):
+ """Return the gRPC stub for :meth:`DataTransferServiceClient.delete_transfer_run`.
+
+ Deletes the specified transfer run.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["data_transfer_service_stub"].DeleteTransferRun
+
+ @property
+ def list_transfer_runs(self):
+ """Return the gRPC stub for :meth:`DataTransferServiceClient.list_transfer_runs`.
+
+ Returns information about running and completed jobs.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["data_transfer_service_stub"].ListTransferRuns
+
+ @property
+ def list_transfer_logs(self):
+ """Return the gRPC stub for :meth:`DataTransferServiceClient.list_transfer_logs`.
+
+ Returns user facing log messages for the data transfer run.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["data_transfer_service_stub"].ListTransferLogs
+
+ @property
+ def check_valid_creds(self):
+ """Return the gRPC stub for :meth:`DataTransferServiceClient.check_valid_creds`.
+
+ Returns true if valid credentials exist for the given data source and
+ requesting user.
+ Some data sources doesn't support service account, so we need to talk to
+ them on behalf of the end user. This API just checks whether we have OAuth
+ token for the particular user, which is a pre-requisite before user can
+ create a transfer config.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["data_transfer_service_stub"].CheckValidCreds
+
+ @property
+ def start_manual_transfer_runs(self):
+ """Return the gRPC stub for :meth:`DataTransferServiceClient.start_manual_transfer_runs`.
+
+ Start manual transfer runs to be executed now with schedule\_time equal
+ to current time. The transfer runs can be created for a time range where
+ the run\_time is between start\_time (inclusive) and end\_time
+ (exclusive), or for a specific run\_time.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["data_transfer_service_stub"].StartManualTransferRuns
diff --git a/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/proto/__init__.py b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/proto/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/proto/datasource.proto b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/proto/datasource.proto
new file mode 100644
index 000000000000..d7400a55935d
--- /dev/null
+++ b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/proto/datasource.proto
@@ -0,0 +1,542 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.bigquery.datatransfer.v1;
+
+import "google/api/annotations.proto";
+import "google/cloud/bigquery/datatransfer/v1/datatransfer.proto";
+import "google/cloud/bigquery/datatransfer/v1/transfer.proto";
+import "google/protobuf/duration.proto";
+import "google/protobuf/empty.proto";
+import "google/protobuf/field_mask.proto";
+import "google/protobuf/timestamp.proto";
+import "google/protobuf/wrappers.proto";
+import "google/api/client.proto";
+
+option csharp_namespace = "Google.Cloud.BigQuery.DataTransfer.V1";
+option go_package = "google.golang.org/genproto/googleapis/cloud/bigquery/datatransfer/v1;datatransfer";
+option java_multiple_files = true;
+option java_outer_classname = "DataSourceProto";
+option java_package = "com.google.cloud.bigquery.datatransfer.v1";
+option php_namespace = "Google\\Cloud\\BigQuery\\DataTransfer\\V1";
+
+// The Google BigQuery Data Transfer API allows BigQuery users to
+// configure transfer of their data from other Google Products into BigQuery.
+// This service exposes methods that should be used by data source backend.
+service DataSourceService {
+ option (google.api.default_host) = "bigquerydatatransfer.googleapis.com";
+
+ // Update a transfer run. If successful, resets
+ // data_source.update_deadline_seconds timer.
+ rpc UpdateTransferRun(UpdateTransferRunRequest) returns (TransferRun) {
+ option (google.api.http) = {
+ patch: "/v1/{transfer_run.name=projects/*/locations/*/transferConfigs/*/runs/*}"
+ body: "transfer_run"
+ };
+ }
+
+ // Log messages for a transfer run. If successful (at least 1 message), resets
+ // data_source.update_deadline_seconds timer.
+ rpc LogTransferRunMessages(LogTransferRunMessagesRequest) returns (google.protobuf.Empty) {
+ option (google.api.http) = {
+ post: "/v1/{name=projects/*/locations/*/transferConfigs/*/runs/*}:logMessages"
+ body: "*"
+ };
+ }
+
+ // Notify the Data Transfer Service that data is ready for loading.
+ // The Data Transfer Service will start and monitor multiple BigQuery Load
+ // jobs for a transfer run. Monitored jobs will be automatically retried
+ // and produce log messages when starting and finishing a job.
+ // Can be called multiple times for the same transfer run.
+ rpc StartBigQueryJobs(StartBigQueryJobsRequest) returns (google.protobuf.Empty) {
+ option (google.api.http) = {
+ post: "/v1/{name=projects/*/locations/*/transferConfigs/*/runs/*}:startBigQueryJobs"
+ body: "*"
+ };
+ }
+
+ // Notify the Data Transfer Service that the data source is done processing
+ // the run. No more status updates or requests to start/monitor jobs will be
+ // accepted. The run will be finalized by the Data Transfer Service when all
+ // monitored jobs are completed.
+ // Does not need to be called if the run is set to FAILED.
+ rpc FinishRun(FinishRunRequest) returns (google.protobuf.Empty) {
+ option (google.api.http) = {
+ post: "/v1/{name=projects/*/locations/*/transferConfigs/*/runs/*}:finishRun"
+ body: "*"
+ };
+ }
+
+ // Creates a data source definition. Calling this method will automatically
+ // use your credentials to create the following Google Cloud resources in
+ // YOUR Google Cloud project.
+ // 1. OAuth client
+ // 2. Pub/Sub Topics and Subscriptions in each supported_location_ids. e.g.,
+ // projects/{project_id}/{topics|subscriptions}/bigquerydatatransfer.{data_source_id}.{location_id}.run
+ // The field data_source.client_id should be left empty in the input request,
+ // as the API will create a new OAuth client on behalf of the caller. On the
+ // other hand data_source.scopes usually need to be set when there are OAuth
+ // scopes that need to be granted by end users.
+ // 3. We need a longer deadline due to the 60 seconds SLO from Pub/Sub admin
+ // Operations. This also applies to update and delete data source definition.
+ rpc CreateDataSourceDefinition(CreateDataSourceDefinitionRequest) returns (DataSourceDefinition) {
+ option (google.api.http) = {
+ post: "/v1/{parent=projects/*/locations/*}/dataSourceDefinitions"
+ body: "data_source_definition"
+ };
+ }
+
+ // Updates an existing data source definition. If changing
+ // supported_location_ids, triggers same effects as mentioned in "Create a
+ // data source definition."
+ rpc UpdateDataSourceDefinition(UpdateDataSourceDefinitionRequest) returns (DataSourceDefinition) {
+ option (google.api.http) = {
+ patch: "/v1/{data_source_definition.name=projects/*/locations/*/dataSourceDefinitions/*}"
+ body: "data_source_definition"
+ };
+ }
+
+ // Deletes a data source definition, all of the transfer configs associated
+ // with this data source definition (if any) must be deleted first by the user
+ // in ALL regions, in order to delete the data source definition.
+ // This method is primarily meant for deleting data sources created during
+ // testing stage.
+ // If the data source is referenced by transfer configs in the region
+ // specified in the request URL, the method will fail immediately. If in the
+ // current region (e.g., US) it's not used by any transfer configs, but in
+ // another region (e.g., EU) it is, then although the method will succeed in
+ // region US, but it will fail when the deletion operation is replicated to
+ // region EU. And eventually, the system will replicate the data source
+ // definition back from EU to US, in order to bring all regions to
+ // consistency. The final effect is that the data source appears to be
+ // 'undeleted' in the US region.
+ rpc DeleteDataSourceDefinition(DeleteDataSourceDefinitionRequest) returns (google.protobuf.Empty) {
+ option (google.api.http) = {
+ delete: "/v1/{name=projects/*/locations/*/dataSourceDefinitions/*}"
+ };
+ }
+
+ // Retrieves an existing data source definition.
+ rpc GetDataSourceDefinition(GetDataSourceDefinitionRequest) returns (DataSourceDefinition) {
+ option (google.api.http) = {
+ get: "/v1/{name=projects/*/locations/*/dataSourceDefinitions/*}"
+ };
+ }
+
+ // Lists supported data source definitions.
+ rpc ListDataSourceDefinitions(ListDataSourceDefinitionsRequest) returns (ListDataSourceDefinitionsResponse) {
+ option (google.api.http) = {
+ get: "/v1/{parent=projects/*/locations/*}/dataSourceDefinitions"
+ };
+ }
+}
+
+// Describes data which should be imported.
+message ImportedDataInfo {
+ // Defines schema of a field in the imported data.
+ message FieldSchema {
+ // LINT.IfChange
+ // Field type.
+ enum Type {
+ // Illegal value.
+ TYPE_UNSPECIFIED = 0;
+
+ // 64K, UTF8.
+ STRING = 1;
+
+ // 64-bit signed.
+ INTEGER = 2;
+
+ // 64-bit IEEE floating point.
+ FLOAT = 3;
+
+ // Aggregate type.
+ RECORD = 4;
+
+ // 64K, Binary.
+ BYTES = 5;
+
+ // 2-valued.
+ BOOLEAN = 6;
+
+ // 64-bit signed usec since UTC epoch.
+ TIMESTAMP = 7;
+
+ // Civil date - Year, Month, Day.
+ DATE = 8;
+
+ // Civil time - Hour, Minute, Second, Microseconds.
+ TIME = 9;
+
+ // Combination of civil date and civil time.
+ DATETIME = 10;
+
+ // Numeric type with 38 decimal digits of precision and 9 decimal digits
+ // of scale.
+ NUMERIC = 11;
+
+ // Geography object (go/googlesql_geography).
+ GEOGRAPHY = 12;
+ }
+
+ // Field name. Matches: [A-Za-z_][A-Za-z_0-9]{0,127}
+ string field_name = 1;
+
+ // Field type
+ Type type = 2;
+
+ // Is field repeated.
+ bool is_repeated = 3;
+
+ // Description for this field.
+ string description = 4;
+
+ // Present iff type == RECORD.
+ RecordSchema schema = 5;
+ }
+
+ // Describes schema of the data to be ingested.
+ message RecordSchema {
+ // One field per column in the record.
+ repeated FieldSchema fields = 1;
+ }
+
+ // External table definition. These tables can be referenced with 'name'
+ // in the query and can be read just like any other table.
+ message TableDefinition {
+ // CSV specific options.
+ message CsvOptions {
+ // The delimiter. We currently restrict this to U+0001 to U+00FF and
+ // apply additional constraints during validation.
+ google.protobuf.StringValue field_delimiter = 1;
+
+ // Whether CSV files are allowed to have quoted newlines. If quoted
+ // newlines are allowed, we can't split CSV files.
+ google.protobuf.BoolValue allow_quoted_newlines = 2;
+
+ // The quote character. We currently restrict this to U+0000 to U+00FF
+ // and apply additional constraints during validation. Set to '\0' to
+ // indicate no quote is used.
+ google.protobuf.StringValue quote_char = 3;
+
+ // Number of leading rows to skip.
+ google.protobuf.Int64Value skip_leading_rows = 4;
+
+ // Accept rows that are missing trailing optional columns.
+ google.protobuf.BoolValue allow_jagged_rows = 5;
+ }
+
+ // BigQuery table_id (required). This will be used to reference this
+ // table in the query.
+ string table_id = 1;
+
+ // URIs for the data to be imported. All URIs must be from the same storage
+ // system.
+ repeated string source_uris = 2;
+
+ // Describes the format of the data in source_uri.
+ Format format = 3;
+
+ // Specify the maximum number of bad records that can be ignored.
+ // If bad records exceed this threshold the query is aborted.
+ int32 max_bad_records = 4;
+
+ // Character encoding of the input when applicable (CSV, JSON).
+ // Defaults to UTF8.
+ Encoding encoding = 5;
+
+ // CSV specific options.
+ CsvOptions csv_options = 6;
+
+ // Optional schema for the data. When not specified for JSON and CSV formats
+ // we will try to detect it automatically.
+ RecordSchema schema = 7;
+
+ // Indicates if extra values that are not represented in the table schema is
+ // allowed.
+ google.protobuf.BoolValue ignore_unknown_values = 10;
+ }
+
+ // Data format.
+ enum Format {
+ // Unspecified format. In this case, we have to infer the format from the
+ // data source.
+ FORMAT_UNSPECIFIED = 0;
+
+ // CSV format.
+ CSV = 1;
+
+ // Newline-delimited JSON.
+ JSON = 2;
+
+ // Avro format. See http://avro.apache.org .
+ AVRO = 3;
+
+ // RecordIO.
+ RECORDIO = 4;
+
+ // ColumnIO.
+ COLUMNIO = 5;
+
+ // Capacitor.
+ CAPACITOR = 6;
+
+ // Parquet format. See https://parquet.apache.org .
+ PARQUET = 7;
+
+ // ORC format. See https://orc.apache.org .
+ ORC = 8;
+ }
+
+ // Encoding of input data in CSV/JSON format.
+ enum Encoding {
+ // Default encoding (UTF8).
+ ENCODING_UNSPECIFIED = 0;
+
+ // ISO_8859_1 encoding.
+ ISO_8859_1 = 1;
+
+ // UTF8 encoding.
+ UTF8 = 2;
+ }
+
+ // SQL query to run. When empty, API checks that there is only one
+ // table_def specified and loads this table. Only Standard SQL queries
+ // are accepted. Legacy SQL is not allowed.
+ string sql = 1;
+
+ // Table where results should be written.
+ string destination_table_id = 2;
+
+ // The description of a destination table. This can be several sentences
+ // or paragraphs describing the table contents in detail.
+ string destination_table_description = 10;
+
+ // When used WITHOUT the "sql" parameter, describes the schema of the
+ // destination table.
+ // When used WITH the "sql" parameter, describes tables with data stored
+ // outside of BigQuery.
+ repeated TableDefinition table_defs = 3;
+
+ // Inline code for User-defined function resources.
+ // Ignored when "sql" parameter is empty.
+ repeated string user_defined_functions = 4;
+
+ // Specifies the action if the destination table already exists.
+ WriteDisposition write_disposition = 6;
+}
+
+// A request to update a transfer run.
+message UpdateTransferRunRequest {
+ // Run name must be set and correspond to an already existing run. Only
+ // state, error_status, and data_version fields will be updated. All other
+ // fields will be ignored.
+ TransferRun transfer_run = 1;
+
+ // Required list of fields to be updated in this request.
+ google.protobuf.FieldMask update_mask = 2;
+}
+
+// A request to add transfer status messages to the run.
+message LogTransferRunMessagesRequest {
+ // Name of the resource in the form:
+ // "projects/{project_id}/locations/{location_id}/transferConfigs/{config_id}/runs/{run_id}"
+ string name = 1;
+
+ // Messages to append.
+ repeated TransferMessage transfer_messages = 2;
+}
+
+// A request to start and monitor a BigQuery load job.
+message StartBigQueryJobsRequest {
+ // Name of the resource in the form:
+ // "projects/{project_id}/locations/{location_id}/transferConfigs/{config_id}/runs/{run_id}"
+ string name = 1;
+
+ // Import jobs which should be started and monitored.
+ repeated ImportedDataInfo imported_data = 2;
+
+ // User credentials which should be used to start/monitor
+ // BigQuery jobs. If not specified, then jobs
+ // are started using data source service account credentials.
+ // This may be OAuth token or JWT token.
+ bytes user_credentials = 3;
+
+ // The number of BQ Jobs that can run in parallel.
+ int32 max_parallelism = 8;
+}
+
+// A request to finish a run.
+message FinishRunRequest {
+ // Name of the resource in the form:
+ // "projects/{project_id}/locations/{location_id}/transferConfigs/{config_id}/runs/{run_id}"
+ string name = 1;
+}
+
+// Represents the request of the CreateDataSourceDefinition method.
+message CreateDataSourceDefinitionRequest {
+ // The BigQuery project id for which data source definition is associated.
+ // Must be in the form: `projects/{project_id}/locations/{location_id}`
+ string parent = 1;
+
+ // Data source definition.
+ DataSourceDefinition data_source_definition = 2;
+}
+
+// Represents the request of the UpdateDataSourceDefinition method.
+message UpdateDataSourceDefinitionRequest {
+ // Data source definition.
+ DataSourceDefinition data_source_definition = 1;
+
+ // Update field mask.
+ google.protobuf.FieldMask update_mask = 2;
+}
+
+// Represents the request of the DeleteDataSourceDefinition method. All transfer
+// configs associated with the data source must be deleted first, before the
+// data source can be deleted.
+message DeleteDataSourceDefinitionRequest {
+ // The field will contain name of the resource requested, for example:
+ // `projects/{project_id}/locations/{location_id}/dataSourceDefinitions/{data_source_id}`
+ string name = 1;
+}
+
+// Represents the request of the GetDataSourceDefinition method.
+message GetDataSourceDefinitionRequest {
+ // The field will contain name of the resource requested.
+ string name = 1;
+}
+
+// Options for writing to the table.
+// The WRITE_EMPTY option is intentionally excluded from the enum and is not
+// supported by the data transfer service.
+enum WriteDisposition {
+ // The default writeDispostion
+ WRITE_DISPOSITION_UNSPECIFIED = 0;
+
+ // overwrites the table data.
+ WRITE_TRUNCATE = 1;
+
+ // the data is appended to the table.
+ // Note duplication might happen if this mode is used.
+ WRITE_APPEND = 2;
+}
+
+// Represents the request of the ListDataSourceDefinitions method.
+message ListDataSourceDefinitionsRequest {
+ // The BigQuery project id for which data sources should be returned.
+ // Must be in the form: `projects/{project_id}/locations/{location_id}`
+ string parent = 1;
+
+ // Pagination token, which can be used to request a specific page
+ // of `ListDataSourceDefinitionsRequest` list results. For multiple-page
+ // results, `ListDataSourceDefinitionsResponse` outputs a `next_page` token,
+ // which can be used as the `page_token` value to request the next page of
+ // the list results.
+ string page_token = 2;
+
+ // Page size. The default page size is the maximum value of 1000 results.
+ int32 page_size = 3;
+}
+
+// Returns a list of supported data source definitions.
+message ListDataSourceDefinitionsResponse {
+ // List of supported data source definitions.
+ repeated DataSourceDefinition data_source_definitions = 1;
+
+ // Output only. The next-pagination token. For multiple-page list results,
+ // this token can be used as the
+ // `ListDataSourceDefinitionsRequest.page_token`
+ // to request the next page of the list results.
+ string next_page_token = 2;
+}
+
+// Represents the data source definition.
+message DataSourceDefinition {
+ // The resource name of the data source definition.
+ // Data source definition names have the form
+ // `projects/{project_id}/locations/{location}/dataSourceDefinitions/{data_source_id}`.
+ string name = 21;
+
+ // Data source metadata.
+ DataSource data_source = 1;
+
+ // The Pub/Sub topic to be used for broadcasting a message when a transfer run
+ // is created. Both this topic and transfer_config_pubsub_topic can be
+ // set to a custom topic. By default, both topics are auto-generated if none
+ // of them is provided when creating the definition. However, if one topic is
+ // manually set, the other topic has to be manually set as well. The only
+ // difference is that transfer_run_pubsub_topic must be a non-empty Pub/Sub
+ // topic, but transfer_config_pubsub_topic can be set to empty. The comments
+ // about "{location}" for transfer_config_pubsub_topic apply here too.
+ string transfer_run_pubsub_topic = 13;
+
+ // Duration which should be added to schedule_time to calculate
+ // run_time when job is scheduled. Only applicable for automatically
+ // scheduled transfer runs. Used to start a run early on a data source that
+ // supports continuous data refresh to compensate for unknown timezone
+ // offsets. Use a negative number to start a run late for data sources not
+ // supporting continuous data refresh.
+ google.protobuf.Duration run_time_offset = 16;
+
+ // Support e-mail address of the OAuth client's Brand, which contains the
+ // consent screen data.
+ string support_email = 22;
+
+ // When service account is specified, BigQuery will share created dataset
+ // with the given service account. Also, this service account will be
+ // eligible to perform status updates and message logging for data transfer
+ // runs for the corresponding data_source_id.
+ string service_account = 2;
+
+ // Is data source disabled? If true, data_source is not visible.
+ // API will also stop returning any data transfer configs and/or runs
+ // associated with the data source. This setting has higher priority
+ // than whitelisted_project_ids.
+ bool disabled = 5;
+
+ // The Pub/Sub topic to use for broadcasting a message for transfer config. If
+ // empty, a message will not be broadcasted. Both this topic and
+ // transfer_run_pubsub_topic are auto-generated if none of them is provided
+ // when creating the definition. It is recommended to provide
+ // transfer_config_pubsub_topic if a user-owned transfer_run_pubsub_topic is
+ // provided. Otherwise, it will be set to empty. If "{location}" is found in
+ // the value, then that means, data source wants to handle message separately
+ // for datasets in different regions. We will replace {location} with the
+ // actual dataset location, as the actual topic name. For example,
+ // projects/connector/topics/scheduler-{location} could become
+ // projects/connector/topics/scheduler-us. If "{location}" is not found, then
+ // we will use the input value as topic name.
+ string transfer_config_pubsub_topic = 12;
+
+ // Supported location_ids used for deciding in which locations Pub/Sub topics
+ // need to be created. If custom Pub/Sub topics are used and they contains
+ // '{location}', the location_ids will be used for validating the topics by
+ // replacing the '{location}' with the individual location in the list. The
+ // valid values are the "location_id" field of the response of `GET
+ // https://bigquerydatatransfer.googleapis.com/v1/{name=projects/*}/locations`
+ // In addition, if the data source needs to support all available regions,
+ // supported_location_ids can be set to "global" (a single string element).
+ // When "global" is specified:
+ // 1) the data source implementation is supposed to stage the data in proper
+ // region of the destination dataset;
+ // 2) Data source developer should be aware of the implications (e.g., network
+ // traffic latency, potential charge associated with cross-region traffic,
+ // etc.) of supporting the "global" region;
+ repeated string supported_location_ids = 23;
+}
diff --git a/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/proto/datasource_pb2.py b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/proto/datasource_pb2.py
new file mode 100644
index 000000000000..82c7e654e364
--- /dev/null
+++ b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/proto/datasource_pb2.py
@@ -0,0 +1,2221 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/bigquery/datatransfer_v1/proto/datasource.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf.internal import enum_type_wrapper
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+from google.cloud.bigquery.datatransfer_v1.proto import (
+ datatransfer_pb2 as google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2,
+)
+from google.cloud.bigquery.datatransfer_v1.proto import (
+ transfer_pb2 as google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2,
+)
+from google.protobuf import duration_pb2 as google_dot_protobuf_dot_duration__pb2
+from google.protobuf import empty_pb2 as google_dot_protobuf_dot_empty__pb2
+from google.protobuf import field_mask_pb2 as google_dot_protobuf_dot_field__mask__pb2
+from google.protobuf import timestamp_pb2 as google_dot_protobuf_dot_timestamp__pb2
+from google.protobuf import wrappers_pb2 as google_dot_protobuf_dot_wrappers__pb2
+from google.api import client_pb2 as google_dot_api_dot_client__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/bigquery/datatransfer_v1/proto/datasource.proto",
+ package="google.cloud.bigquery.datatransfer.v1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n)com.google.cloud.bigquery.datatransfer.v1B\017DataSourceProtoP\001ZQgoogle.golang.org/genproto/googleapis/cloud/bigquery/datatransfer/v1;datatransfer\252\002%Google.Cloud.BigQuery.DataTransfer.V1\312\002%Google\\Cloud\\BigQuery\\DataTransfer\\V1"
+ ),
+ serialized_pb=_b(
+ '\ngoogle/cloud/bigquery/datatransfer_v1/proto/datatransfer.proto\x1a:google/cloud/bigquery/datatransfer_v1/proto/transfer.proto\x1a\x1egoogle/protobuf/duration.proto\x1a\x1bgoogle/protobuf/empty.proto\x1a google/protobuf/field_mask.proto\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x1egoogle/protobuf/wrappers.proto\x1a\x17google/api/client.proto"\x9e\x0e\n\x10ImportedDataInfo\x12\x0b\n\x03sql\x18\x01 \x01(\t\x12\x1c\n\x14\x64\x65stination_table_id\x18\x02 \x01(\t\x12%\n\x1d\x64\x65stination_table_description\x18\n \x01(\t\x12[\n\ntable_defs\x18\x03 \x03(\x0b\x32G.google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.TableDefinition\x12\x1e\n\x16user_defined_functions\x18\x04 \x03(\t\x12R\n\x11write_disposition\x18\x06 \x01(\x0e\x32\x37.google.cloud.bigquery.datatransfer.v1.WriteDisposition\x1a\xad\x03\n\x0b\x46ieldSchema\x12\x12\n\nfield_name\x18\x01 \x01(\t\x12V\n\x04type\x18\x02 \x01(\x0e\x32H.google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.FieldSchema.Type\x12\x13\n\x0bis_repeated\x18\x03 \x01(\x08\x12\x13\n\x0b\x64\x65scription\x18\x04 \x01(\t\x12T\n\x06schema\x18\x05 \x01(\x0b\x32\x44.google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.RecordSchema"\xb1\x01\n\x04Type\x12\x14\n\x10TYPE_UNSPECIFIED\x10\x00\x12\n\n\x06STRING\x10\x01\x12\x0b\n\x07INTEGER\x10\x02\x12\t\n\x05\x46LOAT\x10\x03\x12\n\n\x06RECORD\x10\x04\x12\t\n\x05\x42YTES\x10\x05\x12\x0b\n\x07\x42OOLEAN\x10\x06\x12\r\n\tTIMESTAMP\x10\x07\x12\x08\n\x04\x44\x41TE\x10\x08\x12\x08\n\x04TIME\x10\t\x12\x0c\n\x08\x44\x41TETIME\x10\n\x12\x0b\n\x07NUMERIC\x10\x0b\x12\r\n\tGEOGRAPHY\x10\x0c\x1a\x63\n\x0cRecordSchema\x12S\n\x06\x66ields\x18\x01 \x03(\x0b\x32\x43.google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.FieldSchema\x1a\x91\x06\n\x0fTableDefinition\x12\x10\n\x08table_id\x18\x01 \x01(\t\x12\x13\n\x0bsource_uris\x18\x02 \x03(\t\x12N\n\x06\x66ormat\x18\x03 \x01(\x0e\x32>.google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.Format\x12\x17\n\x0fmax_bad_records\x18\x04 \x01(\x05\x12R\n\x08\x65ncoding\x18\x05 \x01(\x0e\x32@.google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.Encoding\x12g\n\x0b\x63sv_options\x18\x06 \x01(\x0b\x32R.google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.TableDefinition.CsvOptions\x12T\n\x06schema\x18\x07 \x01(\x0b\x32\x44.google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.RecordSchema\x12\x39\n\x15ignore_unknown_values\x18\n \x01(\x0b\x32\x1a.google.protobuf.BoolValue\x1a\x9f\x02\n\nCsvOptions\x12\x35\n\x0f\x66ield_delimiter\x18\x01 \x01(\x0b\x32\x1c.google.protobuf.StringValue\x12\x39\n\x15\x61llow_quoted_newlines\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.BoolValue\x12\x30\n\nquote_char\x18\x03 \x01(\x0b\x32\x1c.google.protobuf.StringValue\x12\x36\n\x11skip_leading_rows\x18\x04 \x01(\x0b\x32\x1b.google.protobuf.Int64Value\x12\x35\n\x11\x61llow_jagged_rows\x18\x05 \x01(\x0b\x32\x1a.google.protobuf.BoolValue"~\n\x06\x46ormat\x12\x16\n\x12\x46ORMAT_UNSPECIFIED\x10\x00\x12\x07\n\x03\x43SV\x10\x01\x12\x08\n\x04JSON\x10\x02\x12\x08\n\x04\x41VRO\x10\x03\x12\x0c\n\x08RECORDIO\x10\x04\x12\x0c\n\x08\x43OLUMNIO\x10\x05\x12\r\n\tCAPACITOR\x10\x06\x12\x0b\n\x07PARQUET\x10\x07\x12\x07\n\x03ORC\x10\x08">\n\x08\x45ncoding\x12\x18\n\x14\x45NCODING_UNSPECIFIED\x10\x00\x12\x0e\n\nISO_8859_1\x10\x01\x12\x08\n\x04UTF8\x10\x02"\x95\x01\n\x18UpdateTransferRunRequest\x12H\n\x0ctransfer_run\x18\x01 \x01(\x0b\x32\x32.google.cloud.bigquery.datatransfer.v1.TransferRun\x12/\n\x0bupdate_mask\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.FieldMask"\x80\x01\n\x1dLogTransferRunMessagesRequest\x12\x0c\n\x04name\x18\x01 \x01(\t\x12Q\n\x11transfer_messages\x18\x02 \x03(\x0b\x32\x36.google.cloud.bigquery.datatransfer.v1.TransferMessage"\xab\x01\n\x18StartBigQueryJobsRequest\x12\x0c\n\x04name\x18\x01 \x01(\t\x12N\n\rimported_data\x18\x02 \x03(\x0b\x32\x37.google.cloud.bigquery.datatransfer.v1.ImportedDataInfo\x12\x18\n\x10user_credentials\x18\x03 \x01(\x0c\x12\x17\n\x0fmax_parallelism\x18\x08 \x01(\x05" \n\x10\x46inishRunRequest\x12\x0c\n\x04name\x18\x01 \x01(\t"\x90\x01\n!CreateDataSourceDefinitionRequest\x12\x0e\n\x06parent\x18\x01 \x01(\t\x12[\n\x16\x64\x61ta_source_definition\x18\x02 \x01(\x0b\x32;.google.cloud.bigquery.datatransfer.v1.DataSourceDefinition"\xb1\x01\n!UpdateDataSourceDefinitionRequest\x12[\n\x16\x64\x61ta_source_definition\x18\x01 \x01(\x0b\x32;.google.cloud.bigquery.datatransfer.v1.DataSourceDefinition\x12/\n\x0bupdate_mask\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.FieldMask"1\n!DeleteDataSourceDefinitionRequest\x12\x0c\n\x04name\x18\x01 \x01(\t".\n\x1eGetDataSourceDefinitionRequest\x12\x0c\n\x04name\x18\x01 \x01(\t"Y\n ListDataSourceDefinitionsRequest\x12\x0e\n\x06parent\x18\x01 \x01(\t\x12\x12\n\npage_token\x18\x02 \x01(\t\x12\x11\n\tpage_size\x18\x03 \x01(\x05"\x9a\x01\n!ListDataSourceDefinitionsResponse\x12\\\n\x17\x64\x61ta_source_definitions\x18\x01 \x03(\x0b\x32;.google.cloud.bigquery.datatransfer.v1.DataSourceDefinition\x12\x17\n\x0fnext_page_token\x18\x02 \x01(\t"\xcb\x02\n\x14\x44\x61taSourceDefinition\x12\x0c\n\x04name\x18\x15 \x01(\t\x12\x46\n\x0b\x64\x61ta_source\x18\x01 \x01(\x0b\x32\x31.google.cloud.bigquery.datatransfer.v1.DataSource\x12!\n\x19transfer_run_pubsub_topic\x18\r \x01(\t\x12\x32\n\x0frun_time_offset\x18\x10 \x01(\x0b\x32\x19.google.protobuf.Duration\x12\x15\n\rsupport_email\x18\x16 \x01(\t\x12\x17\n\x0fservice_account\x18\x02 \x01(\t\x12\x10\n\x08\x64isabled\x18\x05 \x01(\x08\x12$\n\x1ctransfer_config_pubsub_topic\x18\x0c \x01(\t\x12\x1e\n\x16supported_location_ids\x18\x17 \x03(\t*[\n\x10WriteDisposition\x12!\n\x1dWRITE_DISPOSITION_UNSPECIFIED\x10\x00\x12\x12\n\x0eWRITE_TRUNCATE\x10\x01\x12\x10\n\x0cWRITE_APPEND\x10\x02\x32\x9d\x10\n\x11\x44\x61taSourceService\x12\xe7\x01\n\x11UpdateTransferRun\x12?.google.cloud.bigquery.datatransfer.v1.UpdateTransferRunRequest\x1a\x32.google.cloud.bigquery.datatransfer.v1.TransferRun"]\x82\xd3\xe4\x93\x02W2G/v1/{transfer_run.name=projects/*/locations/*/transferConfigs/*/runs/*}:\x0ctransfer_run\x12\xc9\x01\n\x16LogTransferRunMessages\x12\x44.google.cloud.bigquery.datatransfer.v1.LogTransferRunMessagesRequest\x1a\x16.google.protobuf.Empty"Q\x82\xd3\xe4\x93\x02K"F/v1/{name=projects/*/locations/*/transferConfigs/*/runs/*}:logMessages:\x01*\x12\xc5\x01\n\x11StartBigQueryJobs\x12?.google.cloud.bigquery.datatransfer.v1.StartBigQueryJobsRequest\x1a\x16.google.protobuf.Empty"W\x82\xd3\xe4\x93\x02Q"L/v1/{name=projects/*/locations/*/transferConfigs/*/runs/*}:startBigQueryJobs:\x01*\x12\xad\x01\n\tFinishRun\x12\x37.google.cloud.bigquery.datatransfer.v1.FinishRunRequest\x1a\x16.google.protobuf.Empty"O\x82\xd3\xe4\x93\x02I"D/v1/{name=projects/*/locations/*/transferConfigs/*/runs/*}:finishRun:\x01*\x12\xfe\x01\n\x1a\x43reateDataSourceDefinition\x12H.google.cloud.bigquery.datatransfer.v1.CreateDataSourceDefinitionRequest\x1a;.google.cloud.bigquery.datatransfer.v1.DataSourceDefinition"Y\x82\xd3\xe4\x93\x02S"9/v1/{parent=projects/*/locations/*}/dataSourceDefinitions:\x16\x64\x61ta_source_definition\x12\x95\x02\n\x1aUpdateDataSourceDefinition\x12H.google.cloud.bigquery.datatransfer.v1.UpdateDataSourceDefinitionRequest\x1a;.google.cloud.bigquery.datatransfer.v1.DataSourceDefinition"p\x82\xd3\xe4\x93\x02j2P/v1/{data_source_definition.name=projects/*/locations/*/dataSourceDefinitions/*}:\x16\x64\x61ta_source_definition\x12\xc1\x01\n\x1a\x44\x65leteDataSourceDefinition\x12H.google.cloud.bigquery.datatransfer.v1.DeleteDataSourceDefinitionRequest\x1a\x16.google.protobuf.Empty"A\x82\xd3\xe4\x93\x02;*9/v1/{name=projects/*/locations/*/dataSourceDefinitions/*}\x12\xe0\x01\n\x17GetDataSourceDefinition\x12\x45.google.cloud.bigquery.datatransfer.v1.GetDataSourceDefinitionRequest\x1a;.google.cloud.bigquery.datatransfer.v1.DataSourceDefinition"A\x82\xd3\xe4\x93\x02;\x12\x39/v1/{name=projects/*/locations/*/dataSourceDefinitions/*}\x12\xf1\x01\n\x19ListDataSourceDefinitions\x12G.google.cloud.bigquery.datatransfer.v1.ListDataSourceDefinitionsRequest\x1aH.google.cloud.bigquery.datatransfer.v1.ListDataSourceDefinitionsResponse"A\x82\xd3\xe4\x93\x02;\x12\x39/v1/{parent=projects/*/locations/*}/dataSourceDefinitions\x1a&\xca\x41#bigquerydatatransfer.googleapis.comB\xe1\x01\n)com.google.cloud.bigquery.datatransfer.v1B\x0f\x44\x61taSourceProtoP\x01ZQgoogle.golang.org/genproto/googleapis/cloud/bigquery/datatransfer/v1;datatransfer\xaa\x02%Google.Cloud.BigQuery.DataTransfer.V1\xca\x02%Google\\Cloud\\BigQuery\\DataTransfer\\V1b\x06proto3'
+ ),
+ dependencies=[
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_duration__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_empty__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_field__mask__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_timestamp__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_wrappers__pb2.DESCRIPTOR,
+ google_dot_api_dot_client__pb2.DESCRIPTOR,
+ ],
+)
+
+_WRITEDISPOSITION = _descriptor.EnumDescriptor(
+ name="WriteDisposition",
+ full_name="google.cloud.bigquery.datatransfer.v1.WriteDisposition",
+ filename=None,
+ file=DESCRIPTOR,
+ values=[
+ _descriptor.EnumValueDescriptor(
+ name="WRITE_DISPOSITION_UNSPECIFIED",
+ index=0,
+ number=0,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="WRITE_TRUNCATE", index=1, number=1, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="WRITE_APPEND", index=2, number=2, serialized_options=None, type=None
+ ),
+ ],
+ containing_type=None,
+ serialized_options=None,
+ serialized_start=3766,
+ serialized_end=3857,
+)
+_sym_db.RegisterEnumDescriptor(_WRITEDISPOSITION)
+
+WriteDisposition = enum_type_wrapper.EnumTypeWrapper(_WRITEDISPOSITION)
+WRITE_DISPOSITION_UNSPECIFIED = 0
+WRITE_TRUNCATE = 1
+WRITE_APPEND = 2
+
+
+_IMPORTEDDATAINFO_FIELDSCHEMA_TYPE = _descriptor.EnumDescriptor(
+ name="Type",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.FieldSchema.Type",
+ filename=None,
+ file=DESCRIPTOR,
+ values=[
+ _descriptor.EnumValueDescriptor(
+ name="TYPE_UNSPECIFIED",
+ index=0,
+ number=0,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="STRING", index=1, number=1, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="INTEGER", index=2, number=2, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="FLOAT", index=3, number=3, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="RECORD", index=4, number=4, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="BYTES", index=5, number=5, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="BOOLEAN", index=6, number=6, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="TIMESTAMP", index=7, number=7, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="DATE", index=8, number=8, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="TIME", index=9, number=9, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="DATETIME", index=10, number=10, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="NUMERIC", index=11, number=11, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="GEOGRAPHY", index=12, number=12, serialized_options=None, type=None
+ ),
+ ],
+ containing_type=None,
+ serialized_options=None,
+ serialized_start=1007,
+ serialized_end=1184,
+)
+_sym_db.RegisterEnumDescriptor(_IMPORTEDDATAINFO_FIELDSCHEMA_TYPE)
+
+_IMPORTEDDATAINFO_FORMAT = _descriptor.EnumDescriptor(
+ name="Format",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.Format",
+ filename=None,
+ file=DESCRIPTOR,
+ values=[
+ _descriptor.EnumValueDescriptor(
+ name="FORMAT_UNSPECIFIED",
+ index=0,
+ number=0,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="CSV", index=1, number=1, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="JSON", index=2, number=2, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="AVRO", index=3, number=3, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="RECORDIO", index=4, number=4, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="COLUMNIO", index=5, number=5, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="CAPACITOR", index=6, number=6, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="PARQUET", index=7, number=7, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="ORC", index=8, number=8, serialized_options=None, type=None
+ ),
+ ],
+ containing_type=None,
+ serialized_options=None,
+ serialized_start=2075,
+ serialized_end=2201,
+)
+_sym_db.RegisterEnumDescriptor(_IMPORTEDDATAINFO_FORMAT)
+
+_IMPORTEDDATAINFO_ENCODING = _descriptor.EnumDescriptor(
+ name="Encoding",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.Encoding",
+ filename=None,
+ file=DESCRIPTOR,
+ values=[
+ _descriptor.EnumValueDescriptor(
+ name="ENCODING_UNSPECIFIED",
+ index=0,
+ number=0,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="ISO_8859_1", index=1, number=1, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="UTF8", index=2, number=2, serialized_options=None, type=None
+ ),
+ ],
+ containing_type=None,
+ serialized_options=None,
+ serialized_start=2203,
+ serialized_end=2265,
+)
+_sym_db.RegisterEnumDescriptor(_IMPORTEDDATAINFO_ENCODING)
+
+
+_IMPORTEDDATAINFO_FIELDSCHEMA = _descriptor.Descriptor(
+ name="FieldSchema",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.FieldSchema",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="field_name",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.FieldSchema.field_name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="type",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.FieldSchema.type",
+ index=1,
+ number=2,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="is_repeated",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.FieldSchema.is_repeated",
+ index=2,
+ number=3,
+ type=8,
+ cpp_type=7,
+ label=1,
+ has_default_value=False,
+ default_value=False,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="description",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.FieldSchema.description",
+ index=3,
+ number=4,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="schema",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.FieldSchema.schema",
+ index=4,
+ number=5,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[_IMPORTEDDATAINFO_FIELDSCHEMA_TYPE],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=755,
+ serialized_end=1184,
+)
+
+_IMPORTEDDATAINFO_RECORDSCHEMA = _descriptor.Descriptor(
+ name="RecordSchema",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.RecordSchema",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="fields",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.RecordSchema.fields",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1186,
+ serialized_end=1285,
+)
+
+_IMPORTEDDATAINFO_TABLEDEFINITION_CSVOPTIONS = _descriptor.Descriptor(
+ name="CsvOptions",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.TableDefinition.CsvOptions",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="field_delimiter",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.TableDefinition.CsvOptions.field_delimiter",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="allow_quoted_newlines",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.TableDefinition.CsvOptions.allow_quoted_newlines",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="quote_char",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.TableDefinition.CsvOptions.quote_char",
+ index=2,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="skip_leading_rows",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.TableDefinition.CsvOptions.skip_leading_rows",
+ index=3,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="allow_jagged_rows",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.TableDefinition.CsvOptions.allow_jagged_rows",
+ index=4,
+ number=5,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1786,
+ serialized_end=2073,
+)
+
+_IMPORTEDDATAINFO_TABLEDEFINITION = _descriptor.Descriptor(
+ name="TableDefinition",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.TableDefinition",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="table_id",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.TableDefinition.table_id",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="source_uris",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.TableDefinition.source_uris",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="format",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.TableDefinition.format",
+ index=2,
+ number=3,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="max_bad_records",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.TableDefinition.max_bad_records",
+ index=3,
+ number=4,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="encoding",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.TableDefinition.encoding",
+ index=4,
+ number=5,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="csv_options",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.TableDefinition.csv_options",
+ index=5,
+ number=6,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="schema",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.TableDefinition.schema",
+ index=6,
+ number=7,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="ignore_unknown_values",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.TableDefinition.ignore_unknown_values",
+ index=7,
+ number=10,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[_IMPORTEDDATAINFO_TABLEDEFINITION_CSVOPTIONS],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1288,
+ serialized_end=2073,
+)
+
+_IMPORTEDDATAINFO = _descriptor.Descriptor(
+ name="ImportedDataInfo",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="sql",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.sql",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="destination_table_id",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.destination_table_id",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="destination_table_description",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.destination_table_description",
+ index=2,
+ number=10,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="table_defs",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.table_defs",
+ index=3,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="user_defined_functions",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.user_defined_functions",
+ index=4,
+ number=4,
+ type=9,
+ cpp_type=9,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="write_disposition",
+ full_name="google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.write_disposition",
+ index=5,
+ number=6,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[
+ _IMPORTEDDATAINFO_FIELDSCHEMA,
+ _IMPORTEDDATAINFO_RECORDSCHEMA,
+ _IMPORTEDDATAINFO_TABLEDEFINITION,
+ ],
+ enum_types=[_IMPORTEDDATAINFO_FORMAT, _IMPORTEDDATAINFO_ENCODING],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=443,
+ serialized_end=2265,
+)
+
+
+_UPDATETRANSFERRUNREQUEST = _descriptor.Descriptor(
+ name="UpdateTransferRunRequest",
+ full_name="google.cloud.bigquery.datatransfer.v1.UpdateTransferRunRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="transfer_run",
+ full_name="google.cloud.bigquery.datatransfer.v1.UpdateTransferRunRequest.transfer_run",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="update_mask",
+ full_name="google.cloud.bigquery.datatransfer.v1.UpdateTransferRunRequest.update_mask",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2268,
+ serialized_end=2417,
+)
+
+
+_LOGTRANSFERRUNMESSAGESREQUEST = _descriptor.Descriptor(
+ name="LogTransferRunMessagesRequest",
+ full_name="google.cloud.bigquery.datatransfer.v1.LogTransferRunMessagesRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.bigquery.datatransfer.v1.LogTransferRunMessagesRequest.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="transfer_messages",
+ full_name="google.cloud.bigquery.datatransfer.v1.LogTransferRunMessagesRequest.transfer_messages",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2420,
+ serialized_end=2548,
+)
+
+
+_STARTBIGQUERYJOBSREQUEST = _descriptor.Descriptor(
+ name="StartBigQueryJobsRequest",
+ full_name="google.cloud.bigquery.datatransfer.v1.StartBigQueryJobsRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.bigquery.datatransfer.v1.StartBigQueryJobsRequest.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="imported_data",
+ full_name="google.cloud.bigquery.datatransfer.v1.StartBigQueryJobsRequest.imported_data",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="user_credentials",
+ full_name="google.cloud.bigquery.datatransfer.v1.StartBigQueryJobsRequest.user_credentials",
+ index=2,
+ number=3,
+ type=12,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b(""),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="max_parallelism",
+ full_name="google.cloud.bigquery.datatransfer.v1.StartBigQueryJobsRequest.max_parallelism",
+ index=3,
+ number=8,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2551,
+ serialized_end=2722,
+)
+
+
+_FINISHRUNREQUEST = _descriptor.Descriptor(
+ name="FinishRunRequest",
+ full_name="google.cloud.bigquery.datatransfer.v1.FinishRunRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.bigquery.datatransfer.v1.FinishRunRequest.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2724,
+ serialized_end=2756,
+)
+
+
+_CREATEDATASOURCEDEFINITIONREQUEST = _descriptor.Descriptor(
+ name="CreateDataSourceDefinitionRequest",
+ full_name="google.cloud.bigquery.datatransfer.v1.CreateDataSourceDefinitionRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="parent",
+ full_name="google.cloud.bigquery.datatransfer.v1.CreateDataSourceDefinitionRequest.parent",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="data_source_definition",
+ full_name="google.cloud.bigquery.datatransfer.v1.CreateDataSourceDefinitionRequest.data_source_definition",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2759,
+ serialized_end=2903,
+)
+
+
+_UPDATEDATASOURCEDEFINITIONREQUEST = _descriptor.Descriptor(
+ name="UpdateDataSourceDefinitionRequest",
+ full_name="google.cloud.bigquery.datatransfer.v1.UpdateDataSourceDefinitionRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="data_source_definition",
+ full_name="google.cloud.bigquery.datatransfer.v1.UpdateDataSourceDefinitionRequest.data_source_definition",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="update_mask",
+ full_name="google.cloud.bigquery.datatransfer.v1.UpdateDataSourceDefinitionRequest.update_mask",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2906,
+ serialized_end=3083,
+)
+
+
+_DELETEDATASOURCEDEFINITIONREQUEST = _descriptor.Descriptor(
+ name="DeleteDataSourceDefinitionRequest",
+ full_name="google.cloud.bigquery.datatransfer.v1.DeleteDataSourceDefinitionRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.bigquery.datatransfer.v1.DeleteDataSourceDefinitionRequest.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=3085,
+ serialized_end=3134,
+)
+
+
+_GETDATASOURCEDEFINITIONREQUEST = _descriptor.Descriptor(
+ name="GetDataSourceDefinitionRequest",
+ full_name="google.cloud.bigquery.datatransfer.v1.GetDataSourceDefinitionRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.bigquery.datatransfer.v1.GetDataSourceDefinitionRequest.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=3136,
+ serialized_end=3182,
+)
+
+
+_LISTDATASOURCEDEFINITIONSREQUEST = _descriptor.Descriptor(
+ name="ListDataSourceDefinitionsRequest",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListDataSourceDefinitionsRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="parent",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListDataSourceDefinitionsRequest.parent",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="page_token",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListDataSourceDefinitionsRequest.page_token",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="page_size",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListDataSourceDefinitionsRequest.page_size",
+ index=2,
+ number=3,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=3184,
+ serialized_end=3273,
+)
+
+
+_LISTDATASOURCEDEFINITIONSRESPONSE = _descriptor.Descriptor(
+ name="ListDataSourceDefinitionsResponse",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListDataSourceDefinitionsResponse",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="data_source_definitions",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListDataSourceDefinitionsResponse.data_source_definitions",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="next_page_token",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListDataSourceDefinitionsResponse.next_page_token",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=3276,
+ serialized_end=3430,
+)
+
+
+_DATASOURCEDEFINITION = _descriptor.Descriptor(
+ name="DataSourceDefinition",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceDefinition",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceDefinition.name",
+ index=0,
+ number=21,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="data_source",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceDefinition.data_source",
+ index=1,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="transfer_run_pubsub_topic",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceDefinition.transfer_run_pubsub_topic",
+ index=2,
+ number=13,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="run_time_offset",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceDefinition.run_time_offset",
+ index=3,
+ number=16,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="support_email",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceDefinition.support_email",
+ index=4,
+ number=22,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="service_account",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceDefinition.service_account",
+ index=5,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="disabled",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceDefinition.disabled",
+ index=6,
+ number=5,
+ type=8,
+ cpp_type=7,
+ label=1,
+ has_default_value=False,
+ default_value=False,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="transfer_config_pubsub_topic",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceDefinition.transfer_config_pubsub_topic",
+ index=7,
+ number=12,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="supported_location_ids",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceDefinition.supported_location_ids",
+ index=8,
+ number=23,
+ type=9,
+ cpp_type=9,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=3433,
+ serialized_end=3764,
+)
+
+_IMPORTEDDATAINFO_FIELDSCHEMA.fields_by_name[
+ "type"
+].enum_type = _IMPORTEDDATAINFO_FIELDSCHEMA_TYPE
+_IMPORTEDDATAINFO_FIELDSCHEMA.fields_by_name[
+ "schema"
+].message_type = _IMPORTEDDATAINFO_RECORDSCHEMA
+_IMPORTEDDATAINFO_FIELDSCHEMA.containing_type = _IMPORTEDDATAINFO
+_IMPORTEDDATAINFO_FIELDSCHEMA_TYPE.containing_type = _IMPORTEDDATAINFO_FIELDSCHEMA
+_IMPORTEDDATAINFO_RECORDSCHEMA.fields_by_name[
+ "fields"
+].message_type = _IMPORTEDDATAINFO_FIELDSCHEMA
+_IMPORTEDDATAINFO_RECORDSCHEMA.containing_type = _IMPORTEDDATAINFO
+_IMPORTEDDATAINFO_TABLEDEFINITION_CSVOPTIONS.fields_by_name[
+ "field_delimiter"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._STRINGVALUE
+_IMPORTEDDATAINFO_TABLEDEFINITION_CSVOPTIONS.fields_by_name[
+ "allow_quoted_newlines"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._BOOLVALUE
+_IMPORTEDDATAINFO_TABLEDEFINITION_CSVOPTIONS.fields_by_name[
+ "quote_char"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._STRINGVALUE
+_IMPORTEDDATAINFO_TABLEDEFINITION_CSVOPTIONS.fields_by_name[
+ "skip_leading_rows"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._INT64VALUE
+_IMPORTEDDATAINFO_TABLEDEFINITION_CSVOPTIONS.fields_by_name[
+ "allow_jagged_rows"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._BOOLVALUE
+_IMPORTEDDATAINFO_TABLEDEFINITION_CSVOPTIONS.containing_type = (
+ _IMPORTEDDATAINFO_TABLEDEFINITION
+)
+_IMPORTEDDATAINFO_TABLEDEFINITION.fields_by_name[
+ "format"
+].enum_type = _IMPORTEDDATAINFO_FORMAT
+_IMPORTEDDATAINFO_TABLEDEFINITION.fields_by_name[
+ "encoding"
+].enum_type = _IMPORTEDDATAINFO_ENCODING
+_IMPORTEDDATAINFO_TABLEDEFINITION.fields_by_name[
+ "csv_options"
+].message_type = _IMPORTEDDATAINFO_TABLEDEFINITION_CSVOPTIONS
+_IMPORTEDDATAINFO_TABLEDEFINITION.fields_by_name[
+ "schema"
+].message_type = _IMPORTEDDATAINFO_RECORDSCHEMA
+_IMPORTEDDATAINFO_TABLEDEFINITION.fields_by_name[
+ "ignore_unknown_values"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._BOOLVALUE
+_IMPORTEDDATAINFO_TABLEDEFINITION.containing_type = _IMPORTEDDATAINFO
+_IMPORTEDDATAINFO.fields_by_name[
+ "table_defs"
+].message_type = _IMPORTEDDATAINFO_TABLEDEFINITION
+_IMPORTEDDATAINFO.fields_by_name["write_disposition"].enum_type = _WRITEDISPOSITION
+_IMPORTEDDATAINFO_FORMAT.containing_type = _IMPORTEDDATAINFO
+_IMPORTEDDATAINFO_ENCODING.containing_type = _IMPORTEDDATAINFO
+_UPDATETRANSFERRUNREQUEST.fields_by_name[
+ "transfer_run"
+].message_type = (
+ google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2._TRANSFERRUN
+)
+_UPDATETRANSFERRUNREQUEST.fields_by_name[
+ "update_mask"
+].message_type = google_dot_protobuf_dot_field__mask__pb2._FIELDMASK
+_LOGTRANSFERRUNMESSAGESREQUEST.fields_by_name[
+ "transfer_messages"
+].message_type = (
+ google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2._TRANSFERMESSAGE
+)
+_STARTBIGQUERYJOBSREQUEST.fields_by_name[
+ "imported_data"
+].message_type = _IMPORTEDDATAINFO
+_CREATEDATASOURCEDEFINITIONREQUEST.fields_by_name[
+ "data_source_definition"
+].message_type = _DATASOURCEDEFINITION
+_UPDATEDATASOURCEDEFINITIONREQUEST.fields_by_name[
+ "data_source_definition"
+].message_type = _DATASOURCEDEFINITION
+_UPDATEDATASOURCEDEFINITIONREQUEST.fields_by_name[
+ "update_mask"
+].message_type = google_dot_protobuf_dot_field__mask__pb2._FIELDMASK
+_LISTDATASOURCEDEFINITIONSRESPONSE.fields_by_name[
+ "data_source_definitions"
+].message_type = _DATASOURCEDEFINITION
+_DATASOURCEDEFINITION.fields_by_name[
+ "data_source"
+].message_type = (
+ google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2._DATASOURCE
+)
+_DATASOURCEDEFINITION.fields_by_name[
+ "run_time_offset"
+].message_type = google_dot_protobuf_dot_duration__pb2._DURATION
+DESCRIPTOR.message_types_by_name["ImportedDataInfo"] = _IMPORTEDDATAINFO
+DESCRIPTOR.message_types_by_name["UpdateTransferRunRequest"] = _UPDATETRANSFERRUNREQUEST
+DESCRIPTOR.message_types_by_name[
+ "LogTransferRunMessagesRequest"
+] = _LOGTRANSFERRUNMESSAGESREQUEST
+DESCRIPTOR.message_types_by_name["StartBigQueryJobsRequest"] = _STARTBIGQUERYJOBSREQUEST
+DESCRIPTOR.message_types_by_name["FinishRunRequest"] = _FINISHRUNREQUEST
+DESCRIPTOR.message_types_by_name[
+ "CreateDataSourceDefinitionRequest"
+] = _CREATEDATASOURCEDEFINITIONREQUEST
+DESCRIPTOR.message_types_by_name[
+ "UpdateDataSourceDefinitionRequest"
+] = _UPDATEDATASOURCEDEFINITIONREQUEST
+DESCRIPTOR.message_types_by_name[
+ "DeleteDataSourceDefinitionRequest"
+] = _DELETEDATASOURCEDEFINITIONREQUEST
+DESCRIPTOR.message_types_by_name[
+ "GetDataSourceDefinitionRequest"
+] = _GETDATASOURCEDEFINITIONREQUEST
+DESCRIPTOR.message_types_by_name[
+ "ListDataSourceDefinitionsRequest"
+] = _LISTDATASOURCEDEFINITIONSREQUEST
+DESCRIPTOR.message_types_by_name[
+ "ListDataSourceDefinitionsResponse"
+] = _LISTDATASOURCEDEFINITIONSRESPONSE
+DESCRIPTOR.message_types_by_name["DataSourceDefinition"] = _DATASOURCEDEFINITION
+DESCRIPTOR.enum_types_by_name["WriteDisposition"] = _WRITEDISPOSITION
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+ImportedDataInfo = _reflection.GeneratedProtocolMessageType(
+ "ImportedDataInfo",
+ (_message.Message,),
+ dict(
+ FieldSchema=_reflection.GeneratedProtocolMessageType(
+ "FieldSchema",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_IMPORTEDDATAINFO_FIELDSCHEMA,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datasource_pb2",
+ __doc__="""Defines schema of a field in the imported data.
+
+
+ Attributes:
+ field_name:
+ Field name. Matches: [A-Za-z\_][A-Za-z\_0-9]{0,127}
+ type:
+ Field type
+ is_repeated:
+ Is field repeated.
+ description:
+ Description for this field.
+ schema:
+ Present iff type == RECORD.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.FieldSchema)
+ ),
+ ),
+ RecordSchema=_reflection.GeneratedProtocolMessageType(
+ "RecordSchema",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_IMPORTEDDATAINFO_RECORDSCHEMA,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datasource_pb2",
+ __doc__="""Describes schema of the data to be ingested.
+
+
+ Attributes:
+ fields:
+ One field per column in the record.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.RecordSchema)
+ ),
+ ),
+ TableDefinition=_reflection.GeneratedProtocolMessageType(
+ "TableDefinition",
+ (_message.Message,),
+ dict(
+ CsvOptions=_reflection.GeneratedProtocolMessageType(
+ "CsvOptions",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_IMPORTEDDATAINFO_TABLEDEFINITION_CSVOPTIONS,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datasource_pb2",
+ __doc__="""CSV specific options.
+
+
+ Attributes:
+ field_delimiter:
+ The delimiter. We currently restrict this to U+0001 to U+00FF
+ and apply additional constraints during validation.
+ allow_quoted_newlines:
+ Whether CSV files are allowed to have quoted newlines. If
+ quoted newlines are allowed, we can't split CSV files.
+ quote_char:
+ The quote character. We currently restrict this to U+0000 to
+ U+00FF and apply additional constraints during validation. Set
+ to ':raw-latex:`\0`' to indicate no quote is used.
+ skip_leading_rows:
+ Number of leading rows to skip.
+ allow_jagged_rows:
+ Accept rows that are missing trailing optional columns.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.TableDefinition.CsvOptions)
+ ),
+ ),
+ DESCRIPTOR=_IMPORTEDDATAINFO_TABLEDEFINITION,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datasource_pb2",
+ __doc__="""External table definition. These tables can be referenced with 'name' in
+ the query and can be read just like any other table.
+
+
+ Attributes:
+ table_id:
+ BigQuery table\_id (required). This will be used to reference
+ this table in the query.
+ source_uris:
+ URIs for the data to be imported. All URIs must be from the
+ same storage system.
+ format:
+ Describes the format of the data in source\_uri.
+ max_bad_records:
+ Specify the maximum number of bad records that can be ignored.
+ If bad records exceed this threshold the query is aborted.
+ encoding:
+ Character encoding of the input when applicable (CSV, JSON).
+ Defaults to UTF8.
+ csv_options:
+ CSV specific options.
+ schema:
+ Optional schema for the data. When not specified for JSON and
+ CSV formats we will try to detect it automatically.
+ ignore_unknown_values:
+ Indicates if extra values that are not represented in the
+ table schema is allowed.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.ImportedDataInfo.TableDefinition)
+ ),
+ ),
+ DESCRIPTOR=_IMPORTEDDATAINFO,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datasource_pb2",
+ __doc__="""Describes data which should be imported.
+
+
+ Attributes:
+ sql:
+ SQL query to run. When empty, API checks that there is only
+ one table\_def specified and loads this table. Only Standard
+ SQL queries are accepted. Legacy SQL is not allowed.
+ destination_table_id:
+ Table where results should be written.
+ destination_table_description:
+ The description of a destination table. This can be several
+ sentences or paragraphs describing the table contents in
+ detail.
+ table_defs:
+ When used WITHOUT the "sql" parameter, describes the schema of
+ the destination table. When used WITH the "sql" parameter,
+ describes tables with data stored outside of BigQuery.
+ user_defined_functions:
+ Inline code for User-defined function resources. Ignored when
+ "sql" parameter is empty.
+ write_disposition:
+ Specifies the action if the destination table already exists.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.ImportedDataInfo)
+ ),
+)
+_sym_db.RegisterMessage(ImportedDataInfo)
+_sym_db.RegisterMessage(ImportedDataInfo.FieldSchema)
+_sym_db.RegisterMessage(ImportedDataInfo.RecordSchema)
+_sym_db.RegisterMessage(ImportedDataInfo.TableDefinition)
+_sym_db.RegisterMessage(ImportedDataInfo.TableDefinition.CsvOptions)
+
+UpdateTransferRunRequest = _reflection.GeneratedProtocolMessageType(
+ "UpdateTransferRunRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_UPDATETRANSFERRUNREQUEST,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datasource_pb2",
+ __doc__="""A request to update a transfer run.
+
+
+ Attributes:
+ transfer_run:
+ Run name must be set and correspond to an already existing
+ run. Only state, error\_status, and data\_version fields will
+ be updated. All other fields will be ignored.
+ update_mask:
+ Required list of fields to be updated in this request.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.UpdateTransferRunRequest)
+ ),
+)
+_sym_db.RegisterMessage(UpdateTransferRunRequest)
+
+LogTransferRunMessagesRequest = _reflection.GeneratedProtocolMessageType(
+ "LogTransferRunMessagesRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_LOGTRANSFERRUNMESSAGESREQUEST,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datasource_pb2",
+ __doc__="""A request to add transfer status messages to the run.
+
+
+ Attributes:
+ name:
+ Name of the resource in the form: "projects/{project\_id}/loca
+ tions/{location\_id}/transferConfigs/{config\_id}/runs/{run\_i
+ d}"
+ transfer_messages:
+ Messages to append.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.LogTransferRunMessagesRequest)
+ ),
+)
+_sym_db.RegisterMessage(LogTransferRunMessagesRequest)
+
+StartBigQueryJobsRequest = _reflection.GeneratedProtocolMessageType(
+ "StartBigQueryJobsRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_STARTBIGQUERYJOBSREQUEST,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datasource_pb2",
+ __doc__="""A request to start and monitor a BigQuery load job.
+
+
+ Attributes:
+ name:
+ Name of the resource in the form: "projects/{project\_id}/loca
+ tions/{location\_id}/transferConfigs/{config\_id}/runs/{run\_i
+ d}"
+ imported_data:
+ Import jobs which should be started and monitored.
+ user_credentials:
+ User credentials which should be used to start/monitor
+ BigQuery jobs. If not specified, then jobs are started using
+ data source service account credentials. This may be OAuth
+ token or JWT token.
+ max_parallelism:
+ The number of BQ Jobs that can run in parallel.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.StartBigQueryJobsRequest)
+ ),
+)
+_sym_db.RegisterMessage(StartBigQueryJobsRequest)
+
+FinishRunRequest = _reflection.GeneratedProtocolMessageType(
+ "FinishRunRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_FINISHRUNREQUEST,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datasource_pb2",
+ __doc__="""A request to finish a run.
+
+
+ Attributes:
+ name:
+ Name of the resource in the form: "projects/{project\_id}/loca
+ tions/{location\_id}/transferConfigs/{config\_id}/runs/{run\_i
+ d}"
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.FinishRunRequest)
+ ),
+)
+_sym_db.RegisterMessage(FinishRunRequest)
+
+CreateDataSourceDefinitionRequest = _reflection.GeneratedProtocolMessageType(
+ "CreateDataSourceDefinitionRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_CREATEDATASOURCEDEFINITIONREQUEST,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datasource_pb2",
+ __doc__="""Represents the request of the CreateDataSourceDefinition method.
+
+
+ Attributes:
+ parent:
+ The BigQuery project id for which data source definition is
+ associated. Must be in the form:
+ ``projects/{project_id}/locations/{location_id}``
+ data_source_definition:
+ Data source definition.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.CreateDataSourceDefinitionRequest)
+ ),
+)
+_sym_db.RegisterMessage(CreateDataSourceDefinitionRequest)
+
+UpdateDataSourceDefinitionRequest = _reflection.GeneratedProtocolMessageType(
+ "UpdateDataSourceDefinitionRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_UPDATEDATASOURCEDEFINITIONREQUEST,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datasource_pb2",
+ __doc__="""Represents the request of the UpdateDataSourceDefinition method.
+
+
+ Attributes:
+ data_source_definition:
+ Data source definition.
+ update_mask:
+ Update field mask.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.UpdateDataSourceDefinitionRequest)
+ ),
+)
+_sym_db.RegisterMessage(UpdateDataSourceDefinitionRequest)
+
+DeleteDataSourceDefinitionRequest = _reflection.GeneratedProtocolMessageType(
+ "DeleteDataSourceDefinitionRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_DELETEDATASOURCEDEFINITIONREQUEST,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datasource_pb2",
+ __doc__="""Represents the request of the DeleteDataSourceDefinition method. All
+ transfer configs associated with the data source must be deleted first,
+ before the data source can be deleted.
+
+
+ Attributes:
+ name:
+ The field will contain name of the resource requested, for
+ example: ``projects/{project_id}/locations/{location_id}/dataS
+ ourceDefinitions/{data_source_id}``
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.DeleteDataSourceDefinitionRequest)
+ ),
+)
+_sym_db.RegisterMessage(DeleteDataSourceDefinitionRequest)
+
+GetDataSourceDefinitionRequest = _reflection.GeneratedProtocolMessageType(
+ "GetDataSourceDefinitionRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_GETDATASOURCEDEFINITIONREQUEST,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datasource_pb2",
+ __doc__="""Represents the request of the GetDataSourceDefinition method.
+
+
+ Attributes:
+ name:
+ The field will contain name of the resource requested.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.GetDataSourceDefinitionRequest)
+ ),
+)
+_sym_db.RegisterMessage(GetDataSourceDefinitionRequest)
+
+ListDataSourceDefinitionsRequest = _reflection.GeneratedProtocolMessageType(
+ "ListDataSourceDefinitionsRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_LISTDATASOURCEDEFINITIONSREQUEST,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datasource_pb2",
+ __doc__="""Represents the request of the ListDataSourceDefinitions method.
+
+
+ Attributes:
+ parent:
+ The BigQuery project id for which data sources should be
+ returned. Must be in the form:
+ ``projects/{project_id}/locations/{location_id}``
+ page_token:
+ Pagination token, which can be used to request a specific page
+ of ``ListDataSourceDefinitionsRequest`` list results. For
+ multiple-page results, ``ListDataSourceDefinitionsResponse``
+ outputs a ``next_page`` token, which can be used as the
+ ``page_token`` value to request the next page of the list
+ results.
+ page_size:
+ Page size. The default page size is the maximum value of 1000
+ results.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.ListDataSourceDefinitionsRequest)
+ ),
+)
+_sym_db.RegisterMessage(ListDataSourceDefinitionsRequest)
+
+ListDataSourceDefinitionsResponse = _reflection.GeneratedProtocolMessageType(
+ "ListDataSourceDefinitionsResponse",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_LISTDATASOURCEDEFINITIONSRESPONSE,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datasource_pb2",
+ __doc__="""Returns a list of supported data source definitions.
+
+
+ Attributes:
+ data_source_definitions:
+ List of supported data source definitions.
+ next_page_token:
+ Output only. The next-pagination token. For multiple-page list
+ results, this token can be used as the
+ ``ListDataSourceDefinitionsRequest.page_token`` to request the
+ next page of the list results.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.ListDataSourceDefinitionsResponse)
+ ),
+)
+_sym_db.RegisterMessage(ListDataSourceDefinitionsResponse)
+
+DataSourceDefinition = _reflection.GeneratedProtocolMessageType(
+ "DataSourceDefinition",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_DATASOURCEDEFINITION,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datasource_pb2",
+ __doc__="""Represents the data source definition.
+
+
+ Attributes:
+ name:
+ The resource name of the data source definition. Data source
+ definition names have the form ``projects/{project_id}/locatio
+ ns/{location}/dataSourceDefinitions/{data_source_id}``.
+ data_source:
+ Data source metadata.
+ transfer_run_pubsub_topic:
+ The Pub/Sub topic to be used for broadcasting a message when a
+ transfer run is created. Both this topic and
+ transfer\_config\_pubsub\_topic can be set to a custom topic.
+ By default, both topics are auto-generated if none of them is
+ provided when creating the definition. However, if one topic
+ is manually set, the other topic has to be manually set as
+ well. The only difference is that transfer\_run\_pubsub\_topic
+ must be a non-empty Pub/Sub topic, but
+ transfer\_config\_pubsub\_topic can be set to empty. The
+ comments about "{location}" for
+ transfer\_config\_pubsub\_topic apply here too.
+ run_time_offset:
+ Duration which should be added to schedule\_time to calculate
+ run\_time when job is scheduled. Only applicable for
+ automatically scheduled transfer runs. Used to start a run
+ early on a data source that supports continuous data refresh
+ to compensate for unknown timezone offsets. Use a negative
+ number to start a run late for data sources not supporting
+ continuous data refresh.
+ support_email:
+ Support e-mail address of the OAuth client's Brand, which
+ contains the consent screen data.
+ service_account:
+ When service account is specified, BigQuery will share created
+ dataset with the given service account. Also, this service
+ account will be eligible to perform status updates and message
+ logging for data transfer runs for the corresponding
+ data\_source\_id.
+ disabled:
+ Is data source disabled? If true, data\_source is not visible.
+ API will also stop returning any data transfer configs and/or
+ runs associated with the data source. This setting has higher
+ priority than whitelisted\_project\_ids.
+ transfer_config_pubsub_topic:
+ The Pub/Sub topic to use for broadcasting a message for
+ transfer config. If empty, a message will not be broadcasted.
+ Both this topic and transfer\_run\_pubsub\_topic are auto-
+ generated if none of them is provided when creating the
+ definition. It is recommended to provide
+ transfer\_config\_pubsub\_topic if a user-owned
+ transfer\_run\_pubsub\_topic is provided. Otherwise, it will
+ be set to empty. If "{location}" is found in the value, then
+ that means, data source wants to handle message separately for
+ datasets in different regions. We will replace {location} with
+ the actual dataset location, as the actual topic name. For
+ example, projects/connector/topics/scheduler-{location} could
+ become projects/connector/topics/scheduler-us. If "{location}"
+ is not found, then we will use the input value as topic name.
+ supported_location_ids:
+ Supported location\_ids used for deciding in which locations
+ Pub/Sub topics need to be created. If custom Pub/Sub topics
+ are used and they contains '{location}', the location\_ids
+ will be used for validating the topics by replacing the
+ '{location}' with the individual location in the list. The
+ valid values are the "location\_id" field of the response of
+ ``GET https://bigquerydatatransfer.googleapis.com/v1/{name=pro
+ jects/*}/locations`` In addition, if the data source needs to
+ support all available regions, supported\_location\_ids can be
+ set to "global" (a single string element). When "global" is
+ specified: 1) the data source implementation is supposed to
+ stage the data in proper region of the destination dataset; 2)
+ Data source developer should be aware of the implications
+ (e.g., network traffic latency, potential charge associated
+ with cross-region traffic, etc.) of supporting the "global"
+ region;
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.DataSourceDefinition)
+ ),
+)
+_sym_db.RegisterMessage(DataSourceDefinition)
+
+
+DESCRIPTOR._options = None
+
+_DATASOURCESERVICE = _descriptor.ServiceDescriptor(
+ name="DataSourceService",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceService",
+ file=DESCRIPTOR,
+ index=0,
+ serialized_options=_b("\312A#bigquerydatatransfer.googleapis.com"),
+ serialized_start=3860,
+ serialized_end=5937,
+ methods=[
+ _descriptor.MethodDescriptor(
+ name="UpdateTransferRun",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceService.UpdateTransferRun",
+ index=0,
+ containing_service=None,
+ input_type=_UPDATETRANSFERRUNREQUEST,
+ output_type=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2._TRANSFERRUN,
+ serialized_options=_b(
+ "\202\323\344\223\002W2G/v1/{transfer_run.name=projects/*/locations/*/transferConfigs/*/runs/*}:\014transfer_run"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="LogTransferRunMessages",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceService.LogTransferRunMessages",
+ index=1,
+ containing_service=None,
+ input_type=_LOGTRANSFERRUNMESSAGESREQUEST,
+ output_type=google_dot_protobuf_dot_empty__pb2._EMPTY,
+ serialized_options=_b(
+ '\202\323\344\223\002K"F/v1/{name=projects/*/locations/*/transferConfigs/*/runs/*}:logMessages:\001*'
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="StartBigQueryJobs",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceService.StartBigQueryJobs",
+ index=2,
+ containing_service=None,
+ input_type=_STARTBIGQUERYJOBSREQUEST,
+ output_type=google_dot_protobuf_dot_empty__pb2._EMPTY,
+ serialized_options=_b(
+ '\202\323\344\223\002Q"L/v1/{name=projects/*/locations/*/transferConfigs/*/runs/*}:startBigQueryJobs:\001*'
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="FinishRun",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceService.FinishRun",
+ index=3,
+ containing_service=None,
+ input_type=_FINISHRUNREQUEST,
+ output_type=google_dot_protobuf_dot_empty__pb2._EMPTY,
+ serialized_options=_b(
+ '\202\323\344\223\002I"D/v1/{name=projects/*/locations/*/transferConfigs/*/runs/*}:finishRun:\001*'
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="CreateDataSourceDefinition",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceService.CreateDataSourceDefinition",
+ index=4,
+ containing_service=None,
+ input_type=_CREATEDATASOURCEDEFINITIONREQUEST,
+ output_type=_DATASOURCEDEFINITION,
+ serialized_options=_b(
+ '\202\323\344\223\002S"9/v1/{parent=projects/*/locations/*}/dataSourceDefinitions:\026data_source_definition'
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="UpdateDataSourceDefinition",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceService.UpdateDataSourceDefinition",
+ index=5,
+ containing_service=None,
+ input_type=_UPDATEDATASOURCEDEFINITIONREQUEST,
+ output_type=_DATASOURCEDEFINITION,
+ serialized_options=_b(
+ "\202\323\344\223\002j2P/v1/{data_source_definition.name=projects/*/locations/*/dataSourceDefinitions/*}:\026data_source_definition"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="DeleteDataSourceDefinition",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceService.DeleteDataSourceDefinition",
+ index=6,
+ containing_service=None,
+ input_type=_DELETEDATASOURCEDEFINITIONREQUEST,
+ output_type=google_dot_protobuf_dot_empty__pb2._EMPTY,
+ serialized_options=_b(
+ "\202\323\344\223\002;*9/v1/{name=projects/*/locations/*/dataSourceDefinitions/*}"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="GetDataSourceDefinition",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceService.GetDataSourceDefinition",
+ index=7,
+ containing_service=None,
+ input_type=_GETDATASOURCEDEFINITIONREQUEST,
+ output_type=_DATASOURCEDEFINITION,
+ serialized_options=_b(
+ "\202\323\344\223\002;\0229/v1/{name=projects/*/locations/*/dataSourceDefinitions/*}"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="ListDataSourceDefinitions",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceService.ListDataSourceDefinitions",
+ index=8,
+ containing_service=None,
+ input_type=_LISTDATASOURCEDEFINITIONSREQUEST,
+ output_type=_LISTDATASOURCEDEFINITIONSRESPONSE,
+ serialized_options=_b(
+ "\202\323\344\223\002;\0229/v1/{parent=projects/*/locations/*}/dataSourceDefinitions"
+ ),
+ ),
+ ],
+)
+_sym_db.RegisterServiceDescriptor(_DATASOURCESERVICE)
+
+DESCRIPTOR.services_by_name["DataSourceService"] = _DATASOURCESERVICE
+
+# @@protoc_insertion_point(module_scope)
diff --git a/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/proto/datasource_pb2_grpc.py b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/proto/datasource_pb2_grpc.py
new file mode 100644
index 000000000000..136c84c3623a
--- /dev/null
+++ b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/proto/datasource_pb2_grpc.py
@@ -0,0 +1,229 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
+
+from google.cloud.bigquery.datatransfer_v1.proto import (
+ datasource_pb2 as google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datasource__pb2,
+)
+from google.cloud.bigquery.datatransfer_v1.proto import (
+ transfer_pb2 as google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2,
+)
+from google.protobuf import empty_pb2 as google_dot_protobuf_dot_empty__pb2
+
+
+class DataSourceServiceStub(object):
+ """The Google BigQuery Data Transfer API allows BigQuery users to
+ configure transfer of their data from other Google Products into BigQuery.
+ This service exposes methods that should be used by data source backend.
+ """
+
+ def __init__(self, channel):
+ """Constructor.
+
+ Args:
+ channel: A grpc.Channel.
+ """
+ self.UpdateTransferRun = channel.unary_unary(
+ "/google.cloud.bigquery.datatransfer.v1.DataSourceService/UpdateTransferRun",
+ request_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datasource__pb2.UpdateTransferRunRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2.TransferRun.FromString,
+ )
+ self.LogTransferRunMessages = channel.unary_unary(
+ "/google.cloud.bigquery.datatransfer.v1.DataSourceService/LogTransferRunMessages",
+ request_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datasource__pb2.LogTransferRunMessagesRequest.SerializeToString,
+ response_deserializer=google_dot_protobuf_dot_empty__pb2.Empty.FromString,
+ )
+ self.StartBigQueryJobs = channel.unary_unary(
+ "/google.cloud.bigquery.datatransfer.v1.DataSourceService/StartBigQueryJobs",
+ request_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datasource__pb2.StartBigQueryJobsRequest.SerializeToString,
+ response_deserializer=google_dot_protobuf_dot_empty__pb2.Empty.FromString,
+ )
+ self.FinishRun = channel.unary_unary(
+ "/google.cloud.bigquery.datatransfer.v1.DataSourceService/FinishRun",
+ request_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datasource__pb2.FinishRunRequest.SerializeToString,
+ response_deserializer=google_dot_protobuf_dot_empty__pb2.Empty.FromString,
+ )
+ self.CreateDataSourceDefinition = channel.unary_unary(
+ "/google.cloud.bigquery.datatransfer.v1.DataSourceService/CreateDataSourceDefinition",
+ request_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datasource__pb2.CreateDataSourceDefinitionRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datasource__pb2.DataSourceDefinition.FromString,
+ )
+ self.UpdateDataSourceDefinition = channel.unary_unary(
+ "/google.cloud.bigquery.datatransfer.v1.DataSourceService/UpdateDataSourceDefinition",
+ request_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datasource__pb2.UpdateDataSourceDefinitionRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datasource__pb2.DataSourceDefinition.FromString,
+ )
+ self.DeleteDataSourceDefinition = channel.unary_unary(
+ "/google.cloud.bigquery.datatransfer.v1.DataSourceService/DeleteDataSourceDefinition",
+ request_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datasource__pb2.DeleteDataSourceDefinitionRequest.SerializeToString,
+ response_deserializer=google_dot_protobuf_dot_empty__pb2.Empty.FromString,
+ )
+ self.GetDataSourceDefinition = channel.unary_unary(
+ "/google.cloud.bigquery.datatransfer.v1.DataSourceService/GetDataSourceDefinition",
+ request_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datasource__pb2.GetDataSourceDefinitionRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datasource__pb2.DataSourceDefinition.FromString,
+ )
+ self.ListDataSourceDefinitions = channel.unary_unary(
+ "/google.cloud.bigquery.datatransfer.v1.DataSourceService/ListDataSourceDefinitions",
+ request_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datasource__pb2.ListDataSourceDefinitionsRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datasource__pb2.ListDataSourceDefinitionsResponse.FromString,
+ )
+
+
+class DataSourceServiceServicer(object):
+ """The Google BigQuery Data Transfer API allows BigQuery users to
+ configure transfer of their data from other Google Products into BigQuery.
+ This service exposes methods that should be used by data source backend.
+ """
+
+ def UpdateTransferRun(self, request, context):
+ """Update a transfer run. If successful, resets
+ data_source.update_deadline_seconds timer.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def LogTransferRunMessages(self, request, context):
+ """Log messages for a transfer run. If successful (at least 1 message), resets
+ data_source.update_deadline_seconds timer.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def StartBigQueryJobs(self, request, context):
+ """Notify the Data Transfer Service that data is ready for loading.
+ The Data Transfer Service will start and monitor multiple BigQuery Load
+ jobs for a transfer run. Monitored jobs will be automatically retried
+ and produce log messages when starting and finishing a job.
+ Can be called multiple times for the same transfer run.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def FinishRun(self, request, context):
+ """Notify the Data Transfer Service that the data source is done processing
+ the run. No more status updates or requests to start/monitor jobs will be
+ accepted. The run will be finalized by the Data Transfer Service when all
+ monitored jobs are completed.
+ Does not need to be called if the run is set to FAILED.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def CreateDataSourceDefinition(self, request, context):
+ """Creates a data source definition. Calling this method will automatically
+ use your credentials to create the following Google Cloud resources in
+ YOUR Google Cloud project.
+ 1. OAuth client
+ 2. Pub/Sub Topics and Subscriptions in each supported_location_ids. e.g.,
+ projects/{project_id}/{topics|subscriptions}/bigquerydatatransfer.{data_source_id}.{location_id}.run
+ The field data_source.client_id should be left empty in the input request,
+ as the API will create a new OAuth client on behalf of the caller. On the
+ other hand data_source.scopes usually need to be set when there are OAuth
+ scopes that need to be granted by end users.
+ 3. We need a longer deadline due to the 60 seconds SLO from Pub/Sub admin
+ Operations. This also applies to update and delete data source definition.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def UpdateDataSourceDefinition(self, request, context):
+ """Updates an existing data source definition. If changing
+ supported_location_ids, triggers same effects as mentioned in "Create a
+ data source definition."
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def DeleteDataSourceDefinition(self, request, context):
+ """Deletes a data source definition, all of the transfer configs associated
+ with this data source definition (if any) must be deleted first by the user
+ in ALL regions, in order to delete the data source definition.
+ This method is primarily meant for deleting data sources created during
+ testing stage.
+ If the data source is referenced by transfer configs in the region
+ specified in the request URL, the method will fail immediately. If in the
+ current region (e.g., US) it's not used by any transfer configs, but in
+ another region (e.g., EU) it is, then although the method will succeed in
+ region US, but it will fail when the deletion operation is replicated to
+ region EU. And eventually, the system will replicate the data source
+ definition back from EU to US, in order to bring all regions to
+ consistency. The final effect is that the data source appears to be
+ 'undeleted' in the US region.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def GetDataSourceDefinition(self, request, context):
+ """Retrieves an existing data source definition.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def ListDataSourceDefinitions(self, request, context):
+ """Lists supported data source definitions.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+
+def add_DataSourceServiceServicer_to_server(servicer, server):
+ rpc_method_handlers = {
+ "UpdateTransferRun": grpc.unary_unary_rpc_method_handler(
+ servicer.UpdateTransferRun,
+ request_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datasource__pb2.UpdateTransferRunRequest.FromString,
+ response_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2.TransferRun.SerializeToString,
+ ),
+ "LogTransferRunMessages": grpc.unary_unary_rpc_method_handler(
+ servicer.LogTransferRunMessages,
+ request_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datasource__pb2.LogTransferRunMessagesRequest.FromString,
+ response_serializer=google_dot_protobuf_dot_empty__pb2.Empty.SerializeToString,
+ ),
+ "StartBigQueryJobs": grpc.unary_unary_rpc_method_handler(
+ servicer.StartBigQueryJobs,
+ request_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datasource__pb2.StartBigQueryJobsRequest.FromString,
+ response_serializer=google_dot_protobuf_dot_empty__pb2.Empty.SerializeToString,
+ ),
+ "FinishRun": grpc.unary_unary_rpc_method_handler(
+ servicer.FinishRun,
+ request_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datasource__pb2.FinishRunRequest.FromString,
+ response_serializer=google_dot_protobuf_dot_empty__pb2.Empty.SerializeToString,
+ ),
+ "CreateDataSourceDefinition": grpc.unary_unary_rpc_method_handler(
+ servicer.CreateDataSourceDefinition,
+ request_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datasource__pb2.CreateDataSourceDefinitionRequest.FromString,
+ response_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datasource__pb2.DataSourceDefinition.SerializeToString,
+ ),
+ "UpdateDataSourceDefinition": grpc.unary_unary_rpc_method_handler(
+ servicer.UpdateDataSourceDefinition,
+ request_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datasource__pb2.UpdateDataSourceDefinitionRequest.FromString,
+ response_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datasource__pb2.DataSourceDefinition.SerializeToString,
+ ),
+ "DeleteDataSourceDefinition": grpc.unary_unary_rpc_method_handler(
+ servicer.DeleteDataSourceDefinition,
+ request_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datasource__pb2.DeleteDataSourceDefinitionRequest.FromString,
+ response_serializer=google_dot_protobuf_dot_empty__pb2.Empty.SerializeToString,
+ ),
+ "GetDataSourceDefinition": grpc.unary_unary_rpc_method_handler(
+ servicer.GetDataSourceDefinition,
+ request_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datasource__pb2.GetDataSourceDefinitionRequest.FromString,
+ response_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datasource__pb2.DataSourceDefinition.SerializeToString,
+ ),
+ "ListDataSourceDefinitions": grpc.unary_unary_rpc_method_handler(
+ servicer.ListDataSourceDefinitions,
+ request_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datasource__pb2.ListDataSourceDefinitionsRequest.FromString,
+ response_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datasource__pb2.ListDataSourceDefinitionsResponse.SerializeToString,
+ ),
+ }
+ generic_handler = grpc.method_handlers_generic_handler(
+ "google.cloud.bigquery.datatransfer.v1.DataSourceService", rpc_method_handlers
+ )
+ server.add_generic_rpc_handlers((generic_handler,))
diff --git a/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/proto/datatransfer.proto b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/proto/datatransfer.proto
new file mode 100644
index 000000000000..e9a39683494c
--- /dev/null
+++ b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/proto/datatransfer.proto
@@ -0,0 +1,730 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.bigquery.datatransfer.v1;
+
+import "google/api/annotations.proto";
+import "google/cloud/bigquery/datatransfer/v1/transfer.proto";
+import "google/protobuf/duration.proto";
+import "google/protobuf/empty.proto";
+import "google/protobuf/field_mask.proto";
+import "google/protobuf/timestamp.proto";
+import "google/protobuf/wrappers.proto";
+import "google/api/client.proto";
+
+option csharp_namespace = "Google.Cloud.BigQuery.DataTransfer.V1";
+option go_package = "google.golang.org/genproto/googleapis/cloud/bigquery/datatransfer/v1;datatransfer";
+option java_multiple_files = true;
+option java_outer_classname = "DataTransferProto";
+option java_package = "com.google.cloud.bigquery.datatransfer.v1";
+option php_namespace = "Google\\Cloud\\BigQuery\\DataTransfer\\V1";
+
+// The Google BigQuery Data Transfer Service API enables BigQuery users to
+// configure the transfer of their data from other Google Products into
+// BigQuery. This service contains methods that are end user exposed. It backs
+// up the frontend.
+service DataTransferService {
+ option (google.api.default_host) = "bigquerydatatransfer.googleapis.com";
+ option (google.api.oauth_scopes) = "https://www.googleapis.com/auth/cloud-platform";
+
+ // Retrieves a supported data source and returns its settings,
+ // which can be used for UI rendering.
+ rpc GetDataSource(GetDataSourceRequest) returns (DataSource) {
+ option (google.api.http) = {
+ get: "/v1/{name=projects/*/locations/*/dataSources/*}"
+ additional_bindings {
+ get: "/v1/{name=projects/*/dataSources/*}"
+ }
+ };
+ }
+
+ // Lists supported data sources and returns their settings,
+ // which can be used for UI rendering.
+ rpc ListDataSources(ListDataSourcesRequest) returns (ListDataSourcesResponse) {
+ option (google.api.http) = {
+ get: "/v1/{parent=projects/*/locations/*}/dataSources"
+ additional_bindings {
+ get: "/v1/{parent=projects/*}/dataSources"
+ }
+ };
+ }
+
+ // Creates a new data transfer configuration.
+ rpc CreateTransferConfig(CreateTransferConfigRequest) returns (TransferConfig) {
+ option (google.api.http) = {
+ post: "/v1/{parent=projects/*/locations/*}/transferConfigs"
+ body: "transfer_config"
+ additional_bindings {
+ post: "/v1/{parent=projects/*}/transferConfigs"
+ body: "transfer_config"
+ }
+ };
+ }
+
+ // Updates a data transfer configuration.
+ // All fields must be set, even if they are not updated.
+ rpc UpdateTransferConfig(UpdateTransferConfigRequest) returns (TransferConfig) {
+ option (google.api.http) = {
+ patch: "/v1/{transfer_config.name=projects/*/locations/*/transferConfigs/*}"
+ body: "transfer_config"
+ additional_bindings {
+ patch: "/v1/{transfer_config.name=projects/*/transferConfigs/*}"
+ body: "transfer_config"
+ }
+ };
+ }
+
+ // Deletes a data transfer configuration,
+ // including any associated transfer runs and logs.
+ rpc DeleteTransferConfig(DeleteTransferConfigRequest) returns (google.protobuf.Empty) {
+ option (google.api.http) = {
+ delete: "/v1/{name=projects/*/locations/*/transferConfigs/*}"
+ additional_bindings {
+ delete: "/v1/{name=projects/*/transferConfigs/*}"
+ }
+ };
+ }
+
+ // Returns information about a data transfer config.
+ rpc GetTransferConfig(GetTransferConfigRequest) returns (TransferConfig) {
+ option (google.api.http) = {
+ get: "/v1/{name=projects/*/locations/*/transferConfigs/*}"
+ additional_bindings {
+ get: "/v1/{name=projects/*/transferConfigs/*}"
+ }
+ };
+ }
+
+ // Returns information about all data transfers in the project.
+ rpc ListTransferConfigs(ListTransferConfigsRequest) returns (ListTransferConfigsResponse) {
+ option (google.api.http) = {
+ get: "/v1/{parent=projects/*/locations/*}/transferConfigs"
+ additional_bindings {
+ get: "/v1/{parent=projects/*}/transferConfigs"
+ }
+ };
+ }
+
+ // Creates transfer runs for a time range [start_time, end_time].
+ // For each date - or whatever granularity the data source supports - in the
+ // range, one transfer run is created.
+ // Note that runs are created per UTC time in the time range.
+ // DEPRECATED: use StartManualTransferRuns instead.
+ rpc ScheduleTransferRuns(ScheduleTransferRunsRequest) returns (ScheduleTransferRunsResponse) {
+ option deprecated = true;
+ option (google.api.http) = {
+ post: "/v1/{parent=projects/*/locations/*/transferConfigs/*}:scheduleRuns"
+ body: "*"
+ additional_bindings {
+ post: "/v1/{parent=projects/*/transferConfigs/*}:scheduleRuns"
+ body: "*"
+ }
+ };
+ }
+
+ // Start manual transfer runs to be executed now with schedule_time equal to
+ // current time. The transfer runs can be created for a time range where the
+ // run_time is between start_time (inclusive) and end_time (exclusive), or for
+ // a specific run_time.
+ rpc StartManualTransferRuns(StartManualTransferRunsRequest) returns (StartManualTransferRunsResponse) {
+ option (google.api.http) = {
+ post: "/v1/{parent=projects/*/locations/*/transferConfigs/*}:startManualRuns"
+ body: "*"
+ additional_bindings {
+ post: "/v1/{parent=projects/*/transferConfigs/*}:startManualRuns"
+ body: "*"
+ }
+ };
+ }
+
+ // Returns information about the particular transfer run.
+ rpc GetTransferRun(GetTransferRunRequest) returns (TransferRun) {
+ option (google.api.http) = {
+ get: "/v1/{name=projects/*/locations/*/transferConfigs/*/runs/*}"
+ additional_bindings {
+ get: "/v1/{name=projects/*/transferConfigs/*/runs/*}"
+ }
+ };
+ }
+
+ // Deletes the specified transfer run.
+ rpc DeleteTransferRun(DeleteTransferRunRequest) returns (google.protobuf.Empty) {
+ option (google.api.http) = {
+ delete: "/v1/{name=projects/*/locations/*/transferConfigs/*/runs/*}"
+ additional_bindings {
+ delete: "/v1/{name=projects/*/transferConfigs/*/runs/*}"
+ }
+ };
+ }
+
+ // Returns information about running and completed jobs.
+ rpc ListTransferRuns(ListTransferRunsRequest) returns (ListTransferRunsResponse) {
+ option (google.api.http) = {
+ get: "/v1/{parent=projects/*/locations/*/transferConfigs/*}/runs"
+ additional_bindings {
+ get: "/v1/{parent=projects/*/transferConfigs/*}/runs"
+ }
+ };
+ }
+
+ // Returns user facing log messages for the data transfer run.
+ rpc ListTransferLogs(ListTransferLogsRequest) returns (ListTransferLogsResponse) {
+ option (google.api.http) = {
+ get: "/v1/{parent=projects/*/locations/*/transferConfigs/*/runs/*}/transferLogs"
+ additional_bindings {
+ get: "/v1/{parent=projects/*/transferConfigs/*/runs/*}/transferLogs"
+ }
+ };
+ }
+
+ // Returns true if valid credentials exist for the given data source and
+ // requesting user.
+ // Some data sources doesn't support service account, so we need to talk to
+ // them on behalf of the end user. This API just checks whether we have OAuth
+ // token for the particular user, which is a pre-requisite before user can
+ // create a transfer config.
+ rpc CheckValidCreds(CheckValidCredsRequest) returns (CheckValidCredsResponse) {
+ option (google.api.http) = {
+ post: "/v1/{name=projects/*/locations/*/dataSources/*}:checkValidCreds"
+ body: "*"
+ additional_bindings {
+ post: "/v1/{name=projects/*/dataSources/*}:checkValidCreds"
+ body: "*"
+ }
+ };
+ }
+}
+
+// Represents a data source parameter with validation rules, so that
+// parameters can be rendered in the UI. These parameters are given to us by
+// supported data sources, and include all needed information for rendering
+// and validation.
+// Thus, whoever uses this api can decide to generate either generic ui,
+// or custom data source specific forms.
+message DataSourceParameter {
+ // Parameter type.
+ enum Type {
+ // Type unspecified.
+ TYPE_UNSPECIFIED = 0;
+
+ // String parameter.
+ STRING = 1;
+
+ // Integer parameter (64-bits).
+ // Will be serialized to json as string.
+ INTEGER = 2;
+
+ // Double precision floating point parameter.
+ DOUBLE = 3;
+
+ // Boolean parameter.
+ BOOLEAN = 4;
+
+ // Deprecated. This field has no effect.
+ RECORD = 5;
+
+ // Page ID for a Google+ Page.
+ PLUS_PAGE = 6;
+ }
+
+ // Parameter identifier.
+ string param_id = 1;
+
+ // Parameter display name in the user interface.
+ string display_name = 2;
+
+ // Parameter description.
+ string description = 3;
+
+ // Parameter type.
+ Type type = 4;
+
+ // Is parameter required.
+ bool required = 5;
+
+ // Deprecated. This field has no effect.
+ bool repeated = 6;
+
+ // Regular expression which can be used for parameter validation.
+ string validation_regex = 7;
+
+ // All possible values for the parameter.
+ repeated string allowed_values = 8;
+
+ // For integer and double values specifies minimum allowed value.
+ google.protobuf.DoubleValue min_value = 9;
+
+ // For integer and double values specifies maxminum allowed value.
+ google.protobuf.DoubleValue max_value = 10;
+
+ // Deprecated. This field has no effect.
+ repeated DataSourceParameter fields = 11;
+
+ // Description of the requirements for this field, in case the user input does
+ // not fulfill the regex pattern or min/max values.
+ string validation_description = 12;
+
+ // URL to a help document to further explain the naming requirements.
+ string validation_help_url = 13;
+
+ // Cannot be changed after initial creation.
+ bool immutable = 14;
+
+ // Deprecated. This field has no effect.
+ bool recurse = 15;
+
+ // If true, it should not be used in new transfers, and it should not be
+ // visible to users.
+ bool deprecated = 20;
+}
+
+// Represents data source metadata. Metadata is sufficient to
+// render UI and request proper OAuth tokens.
+message DataSource {
+ // The type of authorization needed for this data source.
+ enum AuthorizationType {
+ // Type unspecified.
+ AUTHORIZATION_TYPE_UNSPECIFIED = 0;
+
+ // Use OAuth 2 authorization codes that can be exchanged
+ // for a refresh token on the backend.
+ AUTHORIZATION_CODE = 1;
+
+ // Return an authorization code for a given Google+ page that can then be
+ // exchanged for a refresh token on the backend.
+ GOOGLE_PLUS_AUTHORIZATION_CODE = 2;
+ }
+
+ // Represents how the data source supports data auto refresh.
+ enum DataRefreshType {
+ // The data source won't support data auto refresh, which is default value.
+ DATA_REFRESH_TYPE_UNSPECIFIED = 0;
+
+ // The data source supports data auto refresh, and runs will be scheduled
+ // for the past few days. Does not allow custom values to be set for each
+ // transfer config.
+ SLIDING_WINDOW = 1;
+
+ // The data source supports data auto refresh, and runs will be scheduled
+ // for the past few days. Allows custom values to be set for each transfer
+ // config.
+ CUSTOM_SLIDING_WINDOW = 2;
+ }
+
+ // Output only. Data source resource name.
+ string name = 1;
+
+ // Data source id.
+ string data_source_id = 2;
+
+ // User friendly data source name.
+ string display_name = 3;
+
+ // User friendly data source description string.
+ string description = 4;
+
+ // Data source client id which should be used to receive refresh token.
+ string client_id = 5;
+
+ // Api auth scopes for which refresh token needs to be obtained. These are
+ // scopes needed by a data source to prepare data and ingest them into
+ // BigQuery, e.g., https://www.googleapis.com/auth/bigquery
+ repeated string scopes = 6;
+
+ // Deprecated. This field has no effect.
+ TransferType transfer_type = 7 [deprecated = true];
+
+ // Deprecated. This field has no effect.
+ bool supports_multiple_transfers = 8 [deprecated = true];
+
+ // The number of seconds to wait for an update from the data source
+ // before the Data Transfer Service marks the transfer as FAILED.
+ int32 update_deadline_seconds = 9;
+
+ // Default data transfer schedule.
+ // Examples of valid schedules include:
+ // `1st,3rd monday of month 15:30`,
+ // `every wed,fri of jan,jun 13:15`, and
+ // `first sunday of quarter 00:00`.
+ string default_schedule = 10;
+
+ // Specifies whether the data source supports a user defined schedule, or
+ // operates on the default schedule.
+ // When set to `true`, user can override default schedule.
+ bool supports_custom_schedule = 11;
+
+ // Data source parameters.
+ repeated DataSourceParameter parameters = 12;
+
+ // Url for the help document for this data source.
+ string help_url = 13;
+
+ // Indicates the type of authorization.
+ AuthorizationType authorization_type = 14;
+
+ // Specifies whether the data source supports automatic data refresh for the
+ // past few days, and how it's supported.
+ // For some data sources, data might not be complete until a few days later,
+ // so it's useful to refresh data automatically.
+ DataRefreshType data_refresh_type = 15;
+
+ // Default data refresh window on days.
+ // Only meaningful when `data_refresh_type` = `SLIDING_WINDOW`.
+ int32 default_data_refresh_window_days = 16;
+
+ // Disables backfilling and manual run scheduling
+ // for the data source.
+ bool manual_runs_disabled = 17;
+
+ // The minimum interval for scheduler to schedule runs.
+ google.protobuf.Duration minimum_schedule_interval = 18;
+}
+
+// A request to get data source info.
+message GetDataSourceRequest {
+ // The field will contain name of the resource requested, for example:
+ // `projects/{project_id}/dataSources/{data_source_id}`
+ string name = 1;
+}
+
+// Request to list supported data sources and their data transfer settings.
+message ListDataSourcesRequest {
+ // The BigQuery project id for which data sources should be returned.
+ // Must be in the form: `projects/{project_id}`
+ string parent = 1;
+
+ // Pagination token, which can be used to request a specific page
+ // of `ListDataSourcesRequest` list results. For multiple-page
+ // results, `ListDataSourcesResponse` outputs
+ // a `next_page` token, which can be used as the
+ // `page_token` value to request the next page of list results.
+ string page_token = 3;
+
+ // Page size. The default page size is the maximum value of 1000 results.
+ int32 page_size = 4;
+}
+
+// Returns list of supported data sources and their metadata.
+message ListDataSourcesResponse {
+ // List of supported data sources and their transfer settings.
+ repeated DataSource data_sources = 1;
+
+ // Output only. The next-pagination token. For multiple-page list results,
+ // this token can be used as the
+ // `ListDataSourcesRequest.page_token`
+ // to request the next page of list results.
+ string next_page_token = 2;
+}
+
+// A request to create a data transfer configuration. If new credentials are
+// needed for this transfer configuration, an authorization code must be
+// provided. If an authorization code is provided, the transfer configuration
+// will be associated with the user id corresponding to the
+// authorization code. Otherwise, the transfer configuration will be associated
+// with the calling user.
+message CreateTransferConfigRequest {
+ // The BigQuery project id where the transfer configuration should be created.
+ // Must be in the format projects/{project_id}/locations/{location_id}
+ // If specified location and location of the destination bigquery dataset
+ // do not match - the request will fail.
+ string parent = 1;
+
+ // Data transfer configuration to create.
+ TransferConfig transfer_config = 2;
+
+ // Optional OAuth2 authorization code to use with this transfer configuration.
+ // This is required if new credentials are needed, as indicated by
+ // `CheckValidCreds`.
+ // In order to obtain authorization_code, please make a
+ // request to
+ // https://www.gstatic.com/bigquerydatatransfer/oauthz/auth?client_id=&scope=&redirect_uri=
+ //
+ // * client_id should be OAuth client_id of BigQuery DTS API for the given
+ // data source returned by ListDataSources method.
+ // * data_source_scopes are the scopes returned by ListDataSources method.
+ // * redirect_uri is an optional parameter. If not specified, then
+ // authorization code is posted to the opener of authorization flow window.
+ // Otherwise it will be sent to the redirect uri. A special value of
+ // urn:ietf:wg:oauth:2.0:oob means that authorization code should be
+ // returned in the title bar of the browser, with the page text prompting
+ // the user to copy the code and paste it in the application.
+ string authorization_code = 3;
+
+ // Optional version info. If users want to find a very recent access token,
+ // that is, immediately after approving access, users have to set the
+ // version_info claim in the token request. To obtain the version_info, users
+ // must use the "none+gsession" response type. which be return a
+ // version_info back in the authorization response which be be put in a JWT
+ // claim in the token request.
+ string version_info = 5;
+}
+
+// A request to update a transfer configuration. To update the user id of the
+// transfer configuration, an authorization code needs to be provided.
+message UpdateTransferConfigRequest {
+ // Data transfer configuration to create.
+ TransferConfig transfer_config = 1;
+
+ // Optional OAuth2 authorization code to use with this transfer configuration.
+ // If it is provided, the transfer configuration will be associated with the
+ // authorizing user.
+ // In order to obtain authorization_code, please make a
+ // request to
+ // https://www.gstatic.com/bigquerydatatransfer/oauthz/auth?client_id=&scope=&redirect_uri=
+ //
+ // * client_id should be OAuth client_id of BigQuery DTS API for the given
+ // data source returned by ListDataSources method.
+ // * data_source_scopes are the scopes returned by ListDataSources method.
+ // * redirect_uri is an optional parameter. If not specified, then
+ // authorization code is posted to the opener of authorization flow window.
+ // Otherwise it will be sent to the redirect uri. A special value of
+ // urn:ietf:wg:oauth:2.0:oob means that authorization code should be
+ // returned in the title bar of the browser, with the page text prompting
+ // the user to copy the code and paste it in the application.
+ string authorization_code = 3;
+
+ // Required list of fields to be updated in this request.
+ google.protobuf.FieldMask update_mask = 4;
+
+ // Optional version info. If users want to find a very recent access token,
+ // that is, immediately after approving access, users have to set the
+ // version_info claim in the token request. To obtain the version_info, users
+ // must use the "none+gsession" response type. which be return a
+ // version_info back in the authorization response which be be put in a JWT
+ // claim in the token request.
+ string version_info = 5;
+}
+
+// A request to get data transfer information.
+message GetTransferConfigRequest {
+ // The field will contain name of the resource requested, for example:
+ // `projects/{project_id}/transferConfigs/{config_id}`
+ string name = 1;
+}
+
+// A request to delete data transfer information. All associated transfer runs
+// and log messages will be deleted as well.
+message DeleteTransferConfigRequest {
+ // The field will contain name of the resource requested, for example:
+ // `projects/{project_id}/transferConfigs/{config_id}`
+ string name = 1;
+}
+
+// A request to get data transfer run information.
+message GetTransferRunRequest {
+ // The field will contain name of the resource requested, for example:
+ // `projects/{project_id}/transferConfigs/{config_id}/runs/{run_id}`
+ string name = 1;
+}
+
+// A request to delete data transfer run information.
+message DeleteTransferRunRequest {
+ // The field will contain name of the resource requested, for example:
+ // `projects/{project_id}/transferConfigs/{config_id}/runs/{run_id}`
+ string name = 1;
+}
+
+// A request to list data transfers configured for a BigQuery project.
+message ListTransferConfigsRequest {
+ // The BigQuery project id for which data sources
+ // should be returned: `projects/{project_id}`.
+ string parent = 1;
+
+ // When specified, only configurations of requested data sources are returned.
+ repeated string data_source_ids = 2;
+
+ // Pagination token, which can be used to request a specific page
+ // of `ListTransfersRequest` list results. For multiple-page
+ // results, `ListTransfersResponse` outputs
+ // a `next_page` token, which can be used as the
+ // `page_token` value to request the next page of list results.
+ string page_token = 3;
+
+ // Page size. The default page size is the maximum value of 1000 results.
+ int32 page_size = 4;
+}
+
+// The returned list of pipelines in the project.
+message ListTransferConfigsResponse {
+ // Output only. The stored pipeline transfer configurations.
+ repeated TransferConfig transfer_configs = 1;
+
+ // Output only. The next-pagination token. For multiple-page list results,
+ // this token can be used as the
+ // `ListTransferConfigsRequest.page_token`
+ // to request the next page of list results.
+ string next_page_token = 2;
+}
+
+// A request to list data transfer runs. UI can use this method to show/filter
+// specific data transfer runs. The data source can use this method to request
+// all scheduled transfer runs.
+message ListTransferRunsRequest {
+ // Represents which runs should be pulled.
+ enum RunAttempt {
+ // All runs should be returned.
+ RUN_ATTEMPT_UNSPECIFIED = 0;
+
+ // Only latest run per day should be returned.
+ LATEST = 1;
+ }
+
+ // Name of transfer configuration for which transfer runs should be retrieved.
+ // Format of transfer configuration resource name is:
+ // `projects/{project_id}/transferConfigs/{config_id}`.
+ string parent = 1;
+
+ // When specified, only transfer runs with requested states are returned.
+ repeated TransferState states = 2;
+
+ // Pagination token, which can be used to request a specific page
+ // of `ListTransferRunsRequest` list results. For multiple-page
+ // results, `ListTransferRunsResponse` outputs
+ // a `next_page` token, which can be used as the
+ // `page_token` value to request the next page of list results.
+ string page_token = 3;
+
+ // Page size. The default page size is the maximum value of 1000 results.
+ int32 page_size = 4;
+
+ // Indicates how run attempts are to be pulled.
+ RunAttempt run_attempt = 5;
+}
+
+// The returned list of pipelines in the project.
+message ListTransferRunsResponse {
+ // Output only. The stored pipeline transfer runs.
+ repeated TransferRun transfer_runs = 1;
+
+ // Output only. The next-pagination token. For multiple-page list results,
+ // this token can be used as the
+ // `ListTransferRunsRequest.page_token`
+ // to request the next page of list results.
+ string next_page_token = 2;
+}
+
+// A request to get user facing log messages associated with data transfer run.
+message ListTransferLogsRequest {
+ // Transfer run name in the form:
+ // `projects/{project_id}/transferConfigs/{config_Id}/runs/{run_id}`.
+ string parent = 1;
+
+ // Pagination token, which can be used to request a specific page
+ // of `ListTransferLogsRequest` list results. For multiple-page
+ // results, `ListTransferLogsResponse` outputs
+ // a `next_page` token, which can be used as the
+ // `page_token` value to request the next page of list results.
+ string page_token = 4;
+
+ // Page size. The default page size is the maximum value of 1000 results.
+ int32 page_size = 5;
+
+ // Message types to return. If not populated - INFO, WARNING and ERROR
+ // messages are returned.
+ repeated TransferMessage.MessageSeverity message_types = 6;
+}
+
+// The returned list transfer run messages.
+message ListTransferLogsResponse {
+ // Output only. The stored pipeline transfer messages.
+ repeated TransferMessage transfer_messages = 1;
+
+ // Output only. The next-pagination token. For multiple-page list results,
+ // this token can be used as the
+ // `GetTransferRunLogRequest.page_token`
+ // to request the next page of list results.
+ string next_page_token = 2;
+}
+
+// A request to determine whether the user has valid credentials. This method
+// is used to limit the number of OAuth popups in the user interface. The
+// user id is inferred from the API call context.
+// If the data source has the Google+ authorization type, this method
+// returns false, as it cannot be determined whether the credentials are
+// already valid merely based on the user id.
+message CheckValidCredsRequest {
+ // The data source in the form:
+ // `projects/{project_id}/dataSources/{data_source_id}`
+ string name = 1;
+}
+
+// A response indicating whether the credentials exist and are valid.
+message CheckValidCredsResponse {
+ // If set to `true`, the credentials exist and are valid.
+ bool has_valid_creds = 1;
+}
+
+// A request to schedule transfer runs for a time range.
+message ScheduleTransferRunsRequest {
+ // Transfer configuration name in the form:
+ // `projects/{project_id}/transferConfigs/{config_id}`.
+ string parent = 1;
+
+ // Start time of the range of transfer runs. For example,
+ // `"2017-05-25T00:00:00+00:00"`.
+ google.protobuf.Timestamp start_time = 2;
+
+ // End time of the range of transfer runs. For example,
+ // `"2017-05-30T00:00:00+00:00"`.
+ google.protobuf.Timestamp end_time = 3;
+}
+
+// A response to schedule transfer runs for a time range.
+message ScheduleTransferRunsResponse {
+ // The transfer runs that were scheduled.
+ repeated TransferRun runs = 1;
+}
+
+// A request to start manual transfer runs.
+message StartManualTransferRunsRequest {
+ // A specification for a time range, this will request transfer runs with
+ // run_time between start_time (inclusive) and end_time (exclusive).
+ message TimeRange {
+ // Start time of the range of transfer runs. For example,
+ // `"2017-05-25T00:00:00+00:00"`. The start_time must be strictly less than
+ // the end_time. Creates transfer runs where run_time is in the range betwen
+ // start_time (inclusive) and end_time (exlusive).
+ google.protobuf.Timestamp start_time = 1;
+
+ // End time of the range of transfer runs. For example,
+ // `"2017-05-30T00:00:00+00:00"`. The end_time must not be in the future.
+ // Creates transfer runs where run_time is in the range betwen start_time
+ // (inclusive) and end_time (exlusive).
+ google.protobuf.Timestamp end_time = 2;
+ }
+
+ // Transfer configuration name in the form:
+ // `projects/{project_id}/transferConfigs/{config_id}`.
+ string parent = 1;
+
+ // The requested time specification - this can be a time range or a specific
+ // run_time.
+ oneof time {
+ // Time range for the transfer runs that should be started.
+ TimeRange requested_time_range = 3;
+
+ // Specific run_time for a transfer run to be started. The
+ // requested_run_time must not be in the future.
+ google.protobuf.Timestamp requested_run_time = 4;
+ }
+}
+
+// A response to start manual transfer runs.
+message StartManualTransferRunsResponse {
+ // The transfer runs that were created.
+ repeated TransferRun runs = 1;
+}
diff --git a/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/proto/datatransfer_pb2.py b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/proto/datatransfer_pb2.py
new file mode 100644
index 000000000000..55338100718e
--- /dev/null
+++ b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/proto/datatransfer_pb2.py
@@ -0,0 +1,3216 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/bigquery/datatransfer_v1/proto/datatransfer.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+from google.cloud.bigquery_datatransfer_v1.proto import (
+ transfer_pb2 as google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2,
+)
+from google.protobuf import duration_pb2 as google_dot_protobuf_dot_duration__pb2
+from google.protobuf import empty_pb2 as google_dot_protobuf_dot_empty__pb2
+from google.protobuf import field_mask_pb2 as google_dot_protobuf_dot_field__mask__pb2
+from google.protobuf import timestamp_pb2 as google_dot_protobuf_dot_timestamp__pb2
+from google.protobuf import wrappers_pb2 as google_dot_protobuf_dot_wrappers__pb2
+from google.api import client_pb2 as google_dot_api_dot_client__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/bigquery/datatransfer_v1/proto/datatransfer.proto",
+ package="google.cloud.bigquery.datatransfer.v1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n)com.google.cloud.bigquery.datatransfer.v1B\021DataTransferProtoP\001ZQgoogle.golang.org/genproto/googleapis/cloud/bigquery/datatransfer/v1;datatransfer\252\002%Google.Cloud.BigQuery.DataTransfer.V1\312\002%Google\\Cloud\\BigQuery\\DataTransfer\\V1"
+ ),
+ serialized_pb=_b(
+ '\n>google/cloud/bigquery/datatransfer_v1/proto/datatransfer.proto\x12%google.cloud.bigquery.datatransfer.v1\x1a\x1cgoogle/api/annotations.proto\x1a:google/cloud/bigquery/datatransfer_v1/proto/transfer.proto\x1a\x1egoogle/protobuf/duration.proto\x1a\x1bgoogle/protobuf/empty.proto\x1a google/protobuf/field_mask.proto\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x1egoogle/protobuf/wrappers.proto\x1a\x17google/api/client.proto"\x85\x05\n\x13\x44\x61taSourceParameter\x12\x10\n\x08param_id\x18\x01 \x01(\t\x12\x14\n\x0c\x64isplay_name\x18\x02 \x01(\t\x12\x13\n\x0b\x64\x65scription\x18\x03 \x01(\t\x12M\n\x04type\x18\x04 \x01(\x0e\x32?.google.cloud.bigquery.datatransfer.v1.DataSourceParameter.Type\x12\x10\n\x08required\x18\x05 \x01(\x08\x12\x10\n\x08repeated\x18\x06 \x01(\x08\x12\x18\n\x10validation_regex\x18\x07 \x01(\t\x12\x16\n\x0e\x61llowed_values\x18\x08 \x03(\t\x12/\n\tmin_value\x18\t \x01(\x0b\x32\x1c.google.protobuf.DoubleValue\x12/\n\tmax_value\x18\n \x01(\x0b\x32\x1c.google.protobuf.DoubleValue\x12J\n\x06\x66ields\x18\x0b \x03(\x0b\x32:.google.cloud.bigquery.datatransfer.v1.DataSourceParameter\x12\x1e\n\x16validation_description\x18\x0c \x01(\t\x12\x1b\n\x13validation_help_url\x18\r \x01(\t\x12\x11\n\timmutable\x18\x0e \x01(\x08\x12\x0f\n\x07recurse\x18\x0f \x01(\x08\x12\x12\n\ndeprecated\x18\x14 \x01(\x08"i\n\x04Type\x12\x14\n\x10TYPE_UNSPECIFIED\x10\x00\x12\n\n\x06STRING\x10\x01\x12\x0b\n\x07INTEGER\x10\x02\x12\n\n\x06\x44OUBLE\x10\x03\x12\x0b\n\x07\x42OOLEAN\x10\x04\x12\n\n\x06RECORD\x10\x05\x12\r\n\tPLUS_PAGE\x10\x06"\xd7\x07\n\nDataSource\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x16\n\x0e\x64\x61ta_source_id\x18\x02 \x01(\t\x12\x14\n\x0c\x64isplay_name\x18\x03 \x01(\t\x12\x13\n\x0b\x64\x65scription\x18\x04 \x01(\t\x12\x11\n\tclient_id\x18\x05 \x01(\t\x12\x0e\n\x06scopes\x18\x06 \x03(\t\x12N\n\rtransfer_type\x18\x07 \x01(\x0e\x32\x33.google.cloud.bigquery.datatransfer.v1.TransferTypeB\x02\x18\x01\x12\'\n\x1bsupports_multiple_transfers\x18\x08 \x01(\x08\x42\x02\x18\x01\x12\x1f\n\x17update_deadline_seconds\x18\t \x01(\x05\x12\x18\n\x10\x64\x65\x66\x61ult_schedule\x18\n \x01(\t\x12 \n\x18supports_custom_schedule\x18\x0b \x01(\x08\x12N\n\nparameters\x18\x0c \x03(\x0b\x32:.google.cloud.bigquery.datatransfer.v1.DataSourceParameter\x12\x10\n\x08help_url\x18\r \x01(\t\x12_\n\x12\x61uthorization_type\x18\x0e \x01(\x0e\x32\x43.google.cloud.bigquery.datatransfer.v1.DataSource.AuthorizationType\x12\\\n\x11\x64\x61ta_refresh_type\x18\x0f \x01(\x0e\x32\x41.google.cloud.bigquery.datatransfer.v1.DataSource.DataRefreshType\x12(\n default_data_refresh_window_days\x18\x10 \x01(\x05\x12\x1c\n\x14manual_runs_disabled\x18\x11 \x01(\x08\x12<\n\x19minimum_schedule_interval\x18\x12 \x01(\x0b\x32\x19.google.protobuf.Duration"s\n\x11\x41uthorizationType\x12"\n\x1e\x41UTHORIZATION_TYPE_UNSPECIFIED\x10\x00\x12\x16\n\x12\x41UTHORIZATION_CODE\x10\x01\x12"\n\x1eGOOGLE_PLUS_AUTHORIZATION_CODE\x10\x02"c\n\x0f\x44\x61taRefreshType\x12!\n\x1d\x44\x41TA_REFRESH_TYPE_UNSPECIFIED\x10\x00\x12\x12\n\x0eSLIDING_WINDOW\x10\x01\x12\x19\n\x15\x43USTOM_SLIDING_WINDOW\x10\x02"$\n\x14GetDataSourceRequest\x12\x0c\n\x04name\x18\x01 \x01(\t"O\n\x16ListDataSourcesRequest\x12\x0e\n\x06parent\x18\x01 \x01(\t\x12\x12\n\npage_token\x18\x03 \x01(\t\x12\x11\n\tpage_size\x18\x04 \x01(\x05"{\n\x17ListDataSourcesResponse\x12G\n\x0c\x64\x61ta_sources\x18\x01 \x03(\x0b\x32\x31.google.cloud.bigquery.datatransfer.v1.DataSource\x12\x17\n\x0fnext_page_token\x18\x02 \x01(\t"\xaf\x01\n\x1b\x43reateTransferConfigRequest\x12\x0e\n\x06parent\x18\x01 \x01(\t\x12N\n\x0ftransfer_config\x18\x02 \x01(\x0b\x32\x35.google.cloud.bigquery.datatransfer.v1.TransferConfig\x12\x1a\n\x12\x61uthorization_code\x18\x03 \x01(\t\x12\x14\n\x0cversion_info\x18\x05 \x01(\t"\xd0\x01\n\x1bUpdateTransferConfigRequest\x12N\n\x0ftransfer_config\x18\x01 \x01(\x0b\x32\x35.google.cloud.bigquery.datatransfer.v1.TransferConfig\x12\x1a\n\x12\x61uthorization_code\x18\x03 \x01(\t\x12/\n\x0bupdate_mask\x18\x04 \x01(\x0b\x32\x1a.google.protobuf.FieldMask\x12\x14\n\x0cversion_info\x18\x05 \x01(\t"(\n\x18GetTransferConfigRequest\x12\x0c\n\x04name\x18\x01 \x01(\t"+\n\x1b\x44\x65leteTransferConfigRequest\x12\x0c\n\x04name\x18\x01 \x01(\t"%\n\x15GetTransferRunRequest\x12\x0c\n\x04name\x18\x01 \x01(\t"(\n\x18\x44\x65leteTransferRunRequest\x12\x0c\n\x04name\x18\x01 \x01(\t"l\n\x1aListTransferConfigsRequest\x12\x0e\n\x06parent\x18\x01 \x01(\t\x12\x17\n\x0f\x64\x61ta_source_ids\x18\x02 \x03(\t\x12\x12\n\npage_token\x18\x03 \x01(\t\x12\x11\n\tpage_size\x18\x04 \x01(\x05"\x87\x01\n\x1bListTransferConfigsResponse\x12O\n\x10transfer_configs\x18\x01 \x03(\x0b\x32\x35.google.cloud.bigquery.datatransfer.v1.TransferConfig\x12\x17\n\x0fnext_page_token\x18\x02 \x01(\t"\xad\x02\n\x17ListTransferRunsRequest\x12\x0e\n\x06parent\x18\x01 \x01(\t\x12\x44\n\x06states\x18\x02 \x03(\x0e\x32\x34.google.cloud.bigquery.datatransfer.v1.TransferState\x12\x12\n\npage_token\x18\x03 \x01(\t\x12\x11\n\tpage_size\x18\x04 \x01(\x05\x12^\n\x0brun_attempt\x18\x05 \x01(\x0e\x32I.google.cloud.bigquery.datatransfer.v1.ListTransferRunsRequest.RunAttempt"5\n\nRunAttempt\x12\x1b\n\x17RUN_ATTEMPT_UNSPECIFIED\x10\x00\x12\n\n\x06LATEST\x10\x01"~\n\x18ListTransferRunsResponse\x12I\n\rtransfer_runs\x18\x01 \x03(\x0b\x32\x32.google.cloud.bigquery.datatransfer.v1.TransferRun\x12\x17\n\x0fnext_page_token\x18\x02 \x01(\t"\xaf\x01\n\x17ListTransferLogsRequest\x12\x0e\n\x06parent\x18\x01 \x01(\t\x12\x12\n\npage_token\x18\x04 \x01(\t\x12\x11\n\tpage_size\x18\x05 \x01(\x05\x12]\n\rmessage_types\x18\x06 \x03(\x0e\x32\x46.google.cloud.bigquery.datatransfer.v1.TransferMessage.MessageSeverity"\x86\x01\n\x18ListTransferLogsResponse\x12Q\n\x11transfer_messages\x18\x01 \x03(\x0b\x32\x36.google.cloud.bigquery.datatransfer.v1.TransferMessage\x12\x17\n\x0fnext_page_token\x18\x02 \x01(\t"&\n\x16\x43heckValidCredsRequest\x12\x0c\n\x04name\x18\x01 \x01(\t"2\n\x17\x43heckValidCredsResponse\x12\x17\n\x0fhas_valid_creds\x18\x01 \x01(\x08"\x8b\x01\n\x1bScheduleTransferRunsRequest\x12\x0e\n\x06parent\x18\x01 \x01(\t\x12.\n\nstart_time\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12,\n\x08\x65nd_time\x18\x03 \x01(\x0b\x32\x1a.google.protobuf.Timestamp"`\n\x1cScheduleTransferRunsResponse\x12@\n\x04runs\x18\x01 \x03(\x0b\x32\x32.google.cloud.bigquery.datatransfer.v1.TransferRun"\xce\x02\n\x1eStartManualTransferRunsRequest\x12\x0e\n\x06parent\x18\x01 \x01(\t\x12o\n\x14requested_time_range\x18\x03 \x01(\x0b\x32O.google.cloud.bigquery.datatransfer.v1.StartManualTransferRunsRequest.TimeRangeH\x00\x12\x38\n\x12requested_run_time\x18\x04 \x01(\x0b\x32\x1a.google.protobuf.TimestampH\x00\x1ai\n\tTimeRange\x12.\n\nstart_time\x18\x01 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12,\n\x08\x65nd_time\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.TimestampB\x06\n\x04time"c\n\x1fStartManualTransferRunsResponse\x12@\n\x04runs\x18\x01 \x03(\x0b\x32\x32.google.cloud.bigquery.datatransfer.v1.TransferRun2\xa1\x1e\n\x13\x44\x61taTransferService\x12\xdf\x01\n\rGetDataSource\x12;.google.cloud.bigquery.datatransfer.v1.GetDataSourceRequest\x1a\x31.google.cloud.bigquery.datatransfer.v1.DataSource"^\x82\xd3\xe4\x93\x02X\x12//v1/{name=projects/*/locations/*/dataSources/*}Z%\x12#/v1/{name=projects/*/dataSources/*}\x12\xf0\x01\n\x0fListDataSources\x12=.google.cloud.bigquery.datatransfer.v1.ListDataSourcesRequest\x1a>.google.cloud.bigquery.datatransfer.v1.ListDataSourcesResponse"^\x82\xd3\xe4\x93\x02X\x12//v1/{parent=projects/*/locations/*}/dataSourcesZ%\x12#/v1/{parent=projects/*}/dataSources\x12\x9d\x02\n\x14\x43reateTransferConfig\x12\x42.google.cloud.bigquery.datatransfer.v1.CreateTransferConfigRequest\x1a\x35.google.cloud.bigquery.datatransfer.v1.TransferConfig"\x89\x01\x82\xd3\xe4\x93\x02\x82\x01"3/v1/{parent=projects/*/locations/*}/transferConfigs:\x0ftransfer_configZ:"\'/v1/{parent=projects/*}/transferConfigs:\x0ftransfer_config\x12\xbd\x02\n\x14UpdateTransferConfig\x12\x42.google.cloud.bigquery.datatransfer.v1.UpdateTransferConfigRequest\x1a\x35.google.cloud.bigquery.datatransfer.v1.TransferConfig"\xa9\x01\x82\xd3\xe4\x93\x02\xa2\x01\x32\x43/v1/{transfer_config.name=projects/*/locations/*/transferConfigs/*}:\x0ftransfer_configZJ27/v1/{transfer_config.name=projects/*/transferConfigs/*}:\x0ftransfer_config\x12\xda\x01\n\x14\x44\x65leteTransferConfig\x12\x42.google.cloud.bigquery.datatransfer.v1.DeleteTransferConfigRequest\x1a\x16.google.protobuf.Empty"f\x82\xd3\xe4\x93\x02`*3/v1/{name=projects/*/locations/*/transferConfigs/*}Z)*\'/v1/{name=projects/*/transferConfigs/*}\x12\xf3\x01\n\x11GetTransferConfig\x12?.google.cloud.bigquery.datatransfer.v1.GetTransferConfigRequest\x1a\x35.google.cloud.bigquery.datatransfer.v1.TransferConfig"f\x82\xd3\xe4\x93\x02`\x12\x33/v1/{name=projects/*/locations/*/transferConfigs/*}Z)\x12\'/v1/{name=projects/*/transferConfigs/*}\x12\x84\x02\n\x13ListTransferConfigs\x12\x41.google.cloud.bigquery.datatransfer.v1.ListTransferConfigsRequest\x1a\x42.google.cloud.bigquery.datatransfer.v1.ListTransferConfigsResponse"f\x82\xd3\xe4\x93\x02`\x12\x33/v1/{parent=projects/*/locations/*}/transferConfigsZ)\x12\'/v1/{parent=projects/*}/transferConfigs\x12\xb0\x02\n\x14ScheduleTransferRuns\x12\x42.google.cloud.bigquery.datatransfer.v1.ScheduleTransferRunsRequest\x1a\x43.google.cloud.bigquery.datatransfer.v1.ScheduleTransferRunsResponse"\x8e\x01\x88\x02\x01\x82\xd3\xe4\x93\x02\x84\x01"B/v1/{parent=projects/*/locations/*/transferConfigs/*}:scheduleRuns:\x01*Z;"6/v1/{parent=projects/*/transferConfigs/*}:scheduleRuns:\x01*\x12\xbc\x02\n\x17StartManualTransferRuns\x12\x45.google.cloud.bigquery.datatransfer.v1.StartManualTransferRunsRequest\x1a\x46.google.cloud.bigquery.datatransfer.v1.StartManualTransferRunsResponse"\x91\x01\x82\xd3\xe4\x93\x02\x8a\x01"E/v1/{parent=projects/*/locations/*/transferConfigs/*}:startManualRuns:\x01*Z>"9/v1/{parent=projects/*/transferConfigs/*}:startManualRuns:\x01*\x12\xf8\x01\n\x0eGetTransferRun\x12<.google.cloud.bigquery.datatransfer.v1.GetTransferRunRequest\x1a\x32.google.cloud.bigquery.datatransfer.v1.TransferRun"t\x82\xd3\xe4\x93\x02n\x12:/v1/{name=projects/*/locations/*/transferConfigs/*/runs/*}Z0\x12./v1/{name=projects/*/transferConfigs/*/runs/*}\x12\xe2\x01\n\x11\x44\x65leteTransferRun\x12?.google.cloud.bigquery.datatransfer.v1.DeleteTransferRunRequest\x1a\x16.google.protobuf.Empty"t\x82\xd3\xe4\x93\x02n*:/v1/{name=projects/*/locations/*/transferConfigs/*/runs/*}Z0*./v1/{name=projects/*/transferConfigs/*/runs/*}\x12\x89\x02\n\x10ListTransferRuns\x12>.google.cloud.bigquery.datatransfer.v1.ListTransferRunsRequest\x1a?.google.cloud.bigquery.datatransfer.v1.ListTransferRunsResponse"t\x82\xd3\xe4\x93\x02n\x12:/v1/{parent=projects/*/locations/*/transferConfigs/*}/runsZ0\x12./v1/{parent=projects/*/transferConfigs/*}/runs\x12\xa9\x02\n\x10ListTransferLogs\x12>.google.cloud.bigquery.datatransfer.v1.ListTransferLogsRequest\x1a?.google.cloud.bigquery.datatransfer.v1.ListTransferLogsResponse"\x93\x01\x82\xd3\xe4\x93\x02\x8c\x01\x12I/v1/{parent=projects/*/locations/*/transferConfigs/*/runs/*}/transferLogsZ?\x12=/v1/{parent=projects/*/transferConfigs/*/runs/*}/transferLogs\x12\x97\x02\n\x0f\x43heckValidCreds\x12=.google.cloud.bigquery.datatransfer.v1.CheckValidCredsRequest\x1a>.google.cloud.bigquery.datatransfer.v1.CheckValidCredsResponse"\x84\x01\x82\xd3\xe4\x93\x02~"?/v1/{name=projects/*/locations/*/dataSources/*}:checkValidCreds:\x01*Z8"3/v1/{name=projects/*/dataSources/*}:checkValidCreds:\x01*\x1aW\xca\x41#bigquerydatatransfer.googleapis.com\xd2\x41.https://www.googleapis.com/auth/cloud-platformB\xe3\x01\n)com.google.cloud.bigquery.datatransfer.v1B\x11\x44\x61taTransferProtoP\x01ZQgoogle.golang.org/genproto/googleapis/cloud/bigquery/datatransfer/v1;datatransfer\xaa\x02%Google.Cloud.BigQuery.DataTransfer.V1\xca\x02%Google\\Cloud\\BigQuery\\DataTransfer\\V1b\x06proto3'
+ ),
+ dependencies=[
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_duration__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_empty__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_field__mask__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_timestamp__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_wrappers__pb2.DESCRIPTOR,
+ google_dot_api_dot_client__pb2.DESCRIPTOR,
+ ],
+)
+
+
+_DATASOURCEPARAMETER_TYPE = _descriptor.EnumDescriptor(
+ name="Type",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceParameter.Type",
+ filename=None,
+ file=DESCRIPTOR,
+ values=[
+ _descriptor.EnumValueDescriptor(
+ name="TYPE_UNSPECIFIED",
+ index=0,
+ number=0,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="STRING", index=1, number=1, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="INTEGER", index=2, number=2, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="DOUBLE", index=3, number=3, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="BOOLEAN", index=4, number=4, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="RECORD", index=5, number=5, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="PLUS_PAGE", index=6, number=6, serialized_options=None, type=None
+ ),
+ ],
+ containing_type=None,
+ serialized_options=None,
+ serialized_start=921,
+ serialized_end=1026,
+)
+_sym_db.RegisterEnumDescriptor(_DATASOURCEPARAMETER_TYPE)
+
+_DATASOURCE_AUTHORIZATIONTYPE = _descriptor.EnumDescriptor(
+ name="AuthorizationType",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSource.AuthorizationType",
+ filename=None,
+ file=DESCRIPTOR,
+ values=[
+ _descriptor.EnumValueDescriptor(
+ name="AUTHORIZATION_TYPE_UNSPECIFIED",
+ index=0,
+ number=0,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="AUTHORIZATION_CODE",
+ index=1,
+ number=1,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="GOOGLE_PLUS_AUTHORIZATION_CODE",
+ index=2,
+ number=2,
+ serialized_options=None,
+ type=None,
+ ),
+ ],
+ containing_type=None,
+ serialized_options=None,
+ serialized_start=1796,
+ serialized_end=1911,
+)
+_sym_db.RegisterEnumDescriptor(_DATASOURCE_AUTHORIZATIONTYPE)
+
+_DATASOURCE_DATAREFRESHTYPE = _descriptor.EnumDescriptor(
+ name="DataRefreshType",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSource.DataRefreshType",
+ filename=None,
+ file=DESCRIPTOR,
+ values=[
+ _descriptor.EnumValueDescriptor(
+ name="DATA_REFRESH_TYPE_UNSPECIFIED",
+ index=0,
+ number=0,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="SLIDING_WINDOW", index=1, number=1, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="CUSTOM_SLIDING_WINDOW",
+ index=2,
+ number=2,
+ serialized_options=None,
+ type=None,
+ ),
+ ],
+ containing_type=None,
+ serialized_options=None,
+ serialized_start=1913,
+ serialized_end=2012,
+)
+_sym_db.RegisterEnumDescriptor(_DATASOURCE_DATAREFRESHTYPE)
+
+_LISTTRANSFERRUNSREQUEST_RUNATTEMPT = _descriptor.EnumDescriptor(
+ name="RunAttempt",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListTransferRunsRequest.RunAttempt",
+ filename=None,
+ file=DESCRIPTOR,
+ values=[
+ _descriptor.EnumValueDescriptor(
+ name="RUN_ATTEMPT_UNSPECIFIED",
+ index=0,
+ number=0,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="LATEST", index=1, number=1, serialized_options=None, type=None
+ ),
+ ],
+ containing_type=None,
+ serialized_options=None,
+ serialized_start=3312,
+ serialized_end=3365,
+)
+_sym_db.RegisterEnumDescriptor(_LISTTRANSFERRUNSREQUEST_RUNATTEMPT)
+
+
+_DATASOURCEPARAMETER = _descriptor.Descriptor(
+ name="DataSourceParameter",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceParameter",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="param_id",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceParameter.param_id",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="display_name",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceParameter.display_name",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="description",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceParameter.description",
+ index=2,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="type",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceParameter.type",
+ index=3,
+ number=4,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="required",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceParameter.required",
+ index=4,
+ number=5,
+ type=8,
+ cpp_type=7,
+ label=1,
+ has_default_value=False,
+ default_value=False,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="repeated",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceParameter.repeated",
+ index=5,
+ number=6,
+ type=8,
+ cpp_type=7,
+ label=1,
+ has_default_value=False,
+ default_value=False,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="validation_regex",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceParameter.validation_regex",
+ index=6,
+ number=7,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="allowed_values",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceParameter.allowed_values",
+ index=7,
+ number=8,
+ type=9,
+ cpp_type=9,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="min_value",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceParameter.min_value",
+ index=8,
+ number=9,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="max_value",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceParameter.max_value",
+ index=9,
+ number=10,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="fields",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceParameter.fields",
+ index=10,
+ number=11,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="validation_description",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceParameter.validation_description",
+ index=11,
+ number=12,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="validation_help_url",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceParameter.validation_help_url",
+ index=12,
+ number=13,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="immutable",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceParameter.immutable",
+ index=13,
+ number=14,
+ type=8,
+ cpp_type=7,
+ label=1,
+ has_default_value=False,
+ default_value=False,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="recurse",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceParameter.recurse",
+ index=14,
+ number=15,
+ type=8,
+ cpp_type=7,
+ label=1,
+ has_default_value=False,
+ default_value=False,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="deprecated",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSourceParameter.deprecated",
+ index=15,
+ number=20,
+ type=8,
+ cpp_type=7,
+ label=1,
+ has_default_value=False,
+ default_value=False,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[_DATASOURCEPARAMETER_TYPE],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=381,
+ serialized_end=1026,
+)
+
+
+_DATASOURCE = _descriptor.Descriptor(
+ name="DataSource",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSource",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSource.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="data_source_id",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSource.data_source_id",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="display_name",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSource.display_name",
+ index=2,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="description",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSource.description",
+ index=3,
+ number=4,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="client_id",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSource.client_id",
+ index=4,
+ number=5,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="scopes",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSource.scopes",
+ index=5,
+ number=6,
+ type=9,
+ cpp_type=9,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="transfer_type",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSource.transfer_type",
+ index=6,
+ number=7,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=_b("\030\001"),
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="supports_multiple_transfers",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSource.supports_multiple_transfers",
+ index=7,
+ number=8,
+ type=8,
+ cpp_type=7,
+ label=1,
+ has_default_value=False,
+ default_value=False,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=_b("\030\001"),
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="update_deadline_seconds",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSource.update_deadline_seconds",
+ index=8,
+ number=9,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="default_schedule",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSource.default_schedule",
+ index=9,
+ number=10,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="supports_custom_schedule",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSource.supports_custom_schedule",
+ index=10,
+ number=11,
+ type=8,
+ cpp_type=7,
+ label=1,
+ has_default_value=False,
+ default_value=False,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="parameters",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSource.parameters",
+ index=11,
+ number=12,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="help_url",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSource.help_url",
+ index=12,
+ number=13,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="authorization_type",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSource.authorization_type",
+ index=13,
+ number=14,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="data_refresh_type",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSource.data_refresh_type",
+ index=14,
+ number=15,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="default_data_refresh_window_days",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSource.default_data_refresh_window_days",
+ index=15,
+ number=16,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="manual_runs_disabled",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSource.manual_runs_disabled",
+ index=16,
+ number=17,
+ type=8,
+ cpp_type=7,
+ label=1,
+ has_default_value=False,
+ default_value=False,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="minimum_schedule_interval",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataSource.minimum_schedule_interval",
+ index=17,
+ number=18,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[_DATASOURCE_AUTHORIZATIONTYPE, _DATASOURCE_DATAREFRESHTYPE],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1029,
+ serialized_end=2012,
+)
+
+
+_GETDATASOURCEREQUEST = _descriptor.Descriptor(
+ name="GetDataSourceRequest",
+ full_name="google.cloud.bigquery.datatransfer.v1.GetDataSourceRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.bigquery.datatransfer.v1.GetDataSourceRequest.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2014,
+ serialized_end=2050,
+)
+
+
+_LISTDATASOURCESREQUEST = _descriptor.Descriptor(
+ name="ListDataSourcesRequest",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListDataSourcesRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="parent",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListDataSourcesRequest.parent",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="page_token",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListDataSourcesRequest.page_token",
+ index=1,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="page_size",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListDataSourcesRequest.page_size",
+ index=2,
+ number=4,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2052,
+ serialized_end=2131,
+)
+
+
+_LISTDATASOURCESRESPONSE = _descriptor.Descriptor(
+ name="ListDataSourcesResponse",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListDataSourcesResponse",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="data_sources",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListDataSourcesResponse.data_sources",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="next_page_token",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListDataSourcesResponse.next_page_token",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2133,
+ serialized_end=2256,
+)
+
+
+_CREATETRANSFERCONFIGREQUEST = _descriptor.Descriptor(
+ name="CreateTransferConfigRequest",
+ full_name="google.cloud.bigquery.datatransfer.v1.CreateTransferConfigRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="parent",
+ full_name="google.cloud.bigquery.datatransfer.v1.CreateTransferConfigRequest.parent",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="transfer_config",
+ full_name="google.cloud.bigquery.datatransfer.v1.CreateTransferConfigRequest.transfer_config",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="authorization_code",
+ full_name="google.cloud.bigquery.datatransfer.v1.CreateTransferConfigRequest.authorization_code",
+ index=2,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="version_info",
+ full_name="google.cloud.bigquery.datatransfer.v1.CreateTransferConfigRequest.version_info",
+ index=3,
+ number=5,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2259,
+ serialized_end=2434,
+)
+
+
+_UPDATETRANSFERCONFIGREQUEST = _descriptor.Descriptor(
+ name="UpdateTransferConfigRequest",
+ full_name="google.cloud.bigquery.datatransfer.v1.UpdateTransferConfigRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="transfer_config",
+ full_name="google.cloud.bigquery.datatransfer.v1.UpdateTransferConfigRequest.transfer_config",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="authorization_code",
+ full_name="google.cloud.bigquery.datatransfer.v1.UpdateTransferConfigRequest.authorization_code",
+ index=1,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="update_mask",
+ full_name="google.cloud.bigquery.datatransfer.v1.UpdateTransferConfigRequest.update_mask",
+ index=2,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="version_info",
+ full_name="google.cloud.bigquery.datatransfer.v1.UpdateTransferConfigRequest.version_info",
+ index=3,
+ number=5,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2437,
+ serialized_end=2645,
+)
+
+
+_GETTRANSFERCONFIGREQUEST = _descriptor.Descriptor(
+ name="GetTransferConfigRequest",
+ full_name="google.cloud.bigquery.datatransfer.v1.GetTransferConfigRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.bigquery.datatransfer.v1.GetTransferConfigRequest.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2647,
+ serialized_end=2687,
+)
+
+
+_DELETETRANSFERCONFIGREQUEST = _descriptor.Descriptor(
+ name="DeleteTransferConfigRequest",
+ full_name="google.cloud.bigquery.datatransfer.v1.DeleteTransferConfigRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.bigquery.datatransfer.v1.DeleteTransferConfigRequest.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2689,
+ serialized_end=2732,
+)
+
+
+_GETTRANSFERRUNREQUEST = _descriptor.Descriptor(
+ name="GetTransferRunRequest",
+ full_name="google.cloud.bigquery.datatransfer.v1.GetTransferRunRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.bigquery.datatransfer.v1.GetTransferRunRequest.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2734,
+ serialized_end=2771,
+)
+
+
+_DELETETRANSFERRUNREQUEST = _descriptor.Descriptor(
+ name="DeleteTransferRunRequest",
+ full_name="google.cloud.bigquery.datatransfer.v1.DeleteTransferRunRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.bigquery.datatransfer.v1.DeleteTransferRunRequest.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2773,
+ serialized_end=2813,
+)
+
+
+_LISTTRANSFERCONFIGSREQUEST = _descriptor.Descriptor(
+ name="ListTransferConfigsRequest",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListTransferConfigsRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="parent",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListTransferConfigsRequest.parent",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="data_source_ids",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListTransferConfigsRequest.data_source_ids",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="page_token",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListTransferConfigsRequest.page_token",
+ index=2,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="page_size",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListTransferConfigsRequest.page_size",
+ index=3,
+ number=4,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2815,
+ serialized_end=2923,
+)
+
+
+_LISTTRANSFERCONFIGSRESPONSE = _descriptor.Descriptor(
+ name="ListTransferConfigsResponse",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListTransferConfigsResponse",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="transfer_configs",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListTransferConfigsResponse.transfer_configs",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="next_page_token",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListTransferConfigsResponse.next_page_token",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2926,
+ serialized_end=3061,
+)
+
+
+_LISTTRANSFERRUNSREQUEST = _descriptor.Descriptor(
+ name="ListTransferRunsRequest",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListTransferRunsRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="parent",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListTransferRunsRequest.parent",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="states",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListTransferRunsRequest.states",
+ index=1,
+ number=2,
+ type=14,
+ cpp_type=8,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="page_token",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListTransferRunsRequest.page_token",
+ index=2,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="page_size",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListTransferRunsRequest.page_size",
+ index=3,
+ number=4,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="run_attempt",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListTransferRunsRequest.run_attempt",
+ index=4,
+ number=5,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[_LISTTRANSFERRUNSREQUEST_RUNATTEMPT],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=3064,
+ serialized_end=3365,
+)
+
+
+_LISTTRANSFERRUNSRESPONSE = _descriptor.Descriptor(
+ name="ListTransferRunsResponse",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListTransferRunsResponse",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="transfer_runs",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListTransferRunsResponse.transfer_runs",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="next_page_token",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListTransferRunsResponse.next_page_token",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=3367,
+ serialized_end=3493,
+)
+
+
+_LISTTRANSFERLOGSREQUEST = _descriptor.Descriptor(
+ name="ListTransferLogsRequest",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListTransferLogsRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="parent",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListTransferLogsRequest.parent",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="page_token",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListTransferLogsRequest.page_token",
+ index=1,
+ number=4,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="page_size",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListTransferLogsRequest.page_size",
+ index=2,
+ number=5,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="message_types",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListTransferLogsRequest.message_types",
+ index=3,
+ number=6,
+ type=14,
+ cpp_type=8,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=3496,
+ serialized_end=3671,
+)
+
+
+_LISTTRANSFERLOGSRESPONSE = _descriptor.Descriptor(
+ name="ListTransferLogsResponse",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListTransferLogsResponse",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="transfer_messages",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListTransferLogsResponse.transfer_messages",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="next_page_token",
+ full_name="google.cloud.bigquery.datatransfer.v1.ListTransferLogsResponse.next_page_token",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=3674,
+ serialized_end=3808,
+)
+
+
+_CHECKVALIDCREDSREQUEST = _descriptor.Descriptor(
+ name="CheckValidCredsRequest",
+ full_name="google.cloud.bigquery.datatransfer.v1.CheckValidCredsRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.bigquery.datatransfer.v1.CheckValidCredsRequest.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=3810,
+ serialized_end=3848,
+)
+
+
+_CHECKVALIDCREDSRESPONSE = _descriptor.Descriptor(
+ name="CheckValidCredsResponse",
+ full_name="google.cloud.bigquery.datatransfer.v1.CheckValidCredsResponse",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="has_valid_creds",
+ full_name="google.cloud.bigquery.datatransfer.v1.CheckValidCredsResponse.has_valid_creds",
+ index=0,
+ number=1,
+ type=8,
+ cpp_type=7,
+ label=1,
+ has_default_value=False,
+ default_value=False,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=3850,
+ serialized_end=3900,
+)
+
+
+_SCHEDULETRANSFERRUNSREQUEST = _descriptor.Descriptor(
+ name="ScheduleTransferRunsRequest",
+ full_name="google.cloud.bigquery.datatransfer.v1.ScheduleTransferRunsRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="parent",
+ full_name="google.cloud.bigquery.datatransfer.v1.ScheduleTransferRunsRequest.parent",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="start_time",
+ full_name="google.cloud.bigquery.datatransfer.v1.ScheduleTransferRunsRequest.start_time",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="end_time",
+ full_name="google.cloud.bigquery.datatransfer.v1.ScheduleTransferRunsRequest.end_time",
+ index=2,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=3903,
+ serialized_end=4042,
+)
+
+
+_SCHEDULETRANSFERRUNSRESPONSE = _descriptor.Descriptor(
+ name="ScheduleTransferRunsResponse",
+ full_name="google.cloud.bigquery.datatransfer.v1.ScheduleTransferRunsResponse",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="runs",
+ full_name="google.cloud.bigquery.datatransfer.v1.ScheduleTransferRunsResponse.runs",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=4044,
+ serialized_end=4140,
+)
+
+
+_STARTMANUALTRANSFERRUNSREQUEST_TIMERANGE = _descriptor.Descriptor(
+ name="TimeRange",
+ full_name="google.cloud.bigquery.datatransfer.v1.StartManualTransferRunsRequest.TimeRange",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="start_time",
+ full_name="google.cloud.bigquery.datatransfer.v1.StartManualTransferRunsRequest.TimeRange.start_time",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="end_time",
+ full_name="google.cloud.bigquery.datatransfer.v1.StartManualTransferRunsRequest.TimeRange.end_time",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=4364,
+ serialized_end=4469,
+)
+
+_STARTMANUALTRANSFERRUNSREQUEST = _descriptor.Descriptor(
+ name="StartManualTransferRunsRequest",
+ full_name="google.cloud.bigquery.datatransfer.v1.StartManualTransferRunsRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="parent",
+ full_name="google.cloud.bigquery.datatransfer.v1.StartManualTransferRunsRequest.parent",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="requested_time_range",
+ full_name="google.cloud.bigquery.datatransfer.v1.StartManualTransferRunsRequest.requested_time_range",
+ index=1,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="requested_run_time",
+ full_name="google.cloud.bigquery.datatransfer.v1.StartManualTransferRunsRequest.requested_run_time",
+ index=2,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[_STARTMANUALTRANSFERRUNSREQUEST_TIMERANGE],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="time",
+ full_name="google.cloud.bigquery.datatransfer.v1.StartManualTransferRunsRequest.time",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=4143,
+ serialized_end=4477,
+)
+
+
+_STARTMANUALTRANSFERRUNSRESPONSE = _descriptor.Descriptor(
+ name="StartManualTransferRunsResponse",
+ full_name="google.cloud.bigquery.datatransfer.v1.StartManualTransferRunsResponse",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="runs",
+ full_name="google.cloud.bigquery.datatransfer.v1.StartManualTransferRunsResponse.runs",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=4479,
+ serialized_end=4578,
+)
+
+_DATASOURCEPARAMETER.fields_by_name["type"].enum_type = _DATASOURCEPARAMETER_TYPE
+_DATASOURCEPARAMETER.fields_by_name[
+ "min_value"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._DOUBLEVALUE
+_DATASOURCEPARAMETER.fields_by_name[
+ "max_value"
+].message_type = google_dot_protobuf_dot_wrappers__pb2._DOUBLEVALUE
+_DATASOURCEPARAMETER.fields_by_name["fields"].message_type = _DATASOURCEPARAMETER
+_DATASOURCEPARAMETER_TYPE.containing_type = _DATASOURCEPARAMETER
+_DATASOURCE.fields_by_name[
+ "transfer_type"
+].enum_type = (
+ google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2._TRANSFERTYPE
+)
+_DATASOURCE.fields_by_name["parameters"].message_type = _DATASOURCEPARAMETER
+_DATASOURCE.fields_by_name[
+ "authorization_type"
+].enum_type = _DATASOURCE_AUTHORIZATIONTYPE
+_DATASOURCE.fields_by_name["data_refresh_type"].enum_type = _DATASOURCE_DATAREFRESHTYPE
+_DATASOURCE.fields_by_name[
+ "minimum_schedule_interval"
+].message_type = google_dot_protobuf_dot_duration__pb2._DURATION
+_DATASOURCE_AUTHORIZATIONTYPE.containing_type = _DATASOURCE
+_DATASOURCE_DATAREFRESHTYPE.containing_type = _DATASOURCE
+_LISTDATASOURCESRESPONSE.fields_by_name["data_sources"].message_type = _DATASOURCE
+_CREATETRANSFERCONFIGREQUEST.fields_by_name[
+ "transfer_config"
+].message_type = (
+ google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2._TRANSFERCONFIG
+)
+_UPDATETRANSFERCONFIGREQUEST.fields_by_name[
+ "transfer_config"
+].message_type = (
+ google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2._TRANSFERCONFIG
+)
+_UPDATETRANSFERCONFIGREQUEST.fields_by_name[
+ "update_mask"
+].message_type = google_dot_protobuf_dot_field__mask__pb2._FIELDMASK
+_LISTTRANSFERCONFIGSRESPONSE.fields_by_name[
+ "transfer_configs"
+].message_type = (
+ google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2._TRANSFERCONFIG
+)
+_LISTTRANSFERRUNSREQUEST.fields_by_name[
+ "states"
+].enum_type = (
+ google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2._TRANSFERSTATE
+)
+_LISTTRANSFERRUNSREQUEST.fields_by_name[
+ "run_attempt"
+].enum_type = _LISTTRANSFERRUNSREQUEST_RUNATTEMPT
+_LISTTRANSFERRUNSREQUEST_RUNATTEMPT.containing_type = _LISTTRANSFERRUNSREQUEST
+_LISTTRANSFERRUNSRESPONSE.fields_by_name[
+ "transfer_runs"
+].message_type = (
+ google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2._TRANSFERRUN
+)
+_LISTTRANSFERLOGSREQUEST.fields_by_name[
+ "message_types"
+].enum_type = (
+ google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2._TRANSFERMESSAGE_MESSAGESEVERITY
+)
+_LISTTRANSFERLOGSRESPONSE.fields_by_name[
+ "transfer_messages"
+].message_type = (
+ google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2._TRANSFERMESSAGE
+)
+_SCHEDULETRANSFERRUNSREQUEST.fields_by_name[
+ "start_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_SCHEDULETRANSFERRUNSREQUEST.fields_by_name[
+ "end_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_SCHEDULETRANSFERRUNSRESPONSE.fields_by_name[
+ "runs"
+].message_type = (
+ google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2._TRANSFERRUN
+)
+_STARTMANUALTRANSFERRUNSREQUEST_TIMERANGE.fields_by_name[
+ "start_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_STARTMANUALTRANSFERRUNSREQUEST_TIMERANGE.fields_by_name[
+ "end_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_STARTMANUALTRANSFERRUNSREQUEST_TIMERANGE.containing_type = (
+ _STARTMANUALTRANSFERRUNSREQUEST
+)
+_STARTMANUALTRANSFERRUNSREQUEST.fields_by_name[
+ "requested_time_range"
+].message_type = _STARTMANUALTRANSFERRUNSREQUEST_TIMERANGE
+_STARTMANUALTRANSFERRUNSREQUEST.fields_by_name[
+ "requested_run_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_STARTMANUALTRANSFERRUNSREQUEST.oneofs_by_name["time"].fields.append(
+ _STARTMANUALTRANSFERRUNSREQUEST.fields_by_name["requested_time_range"]
+)
+_STARTMANUALTRANSFERRUNSREQUEST.fields_by_name[
+ "requested_time_range"
+].containing_oneof = _STARTMANUALTRANSFERRUNSREQUEST.oneofs_by_name["time"]
+_STARTMANUALTRANSFERRUNSREQUEST.oneofs_by_name["time"].fields.append(
+ _STARTMANUALTRANSFERRUNSREQUEST.fields_by_name["requested_run_time"]
+)
+_STARTMANUALTRANSFERRUNSREQUEST.fields_by_name[
+ "requested_run_time"
+].containing_oneof = _STARTMANUALTRANSFERRUNSREQUEST.oneofs_by_name["time"]
+_STARTMANUALTRANSFERRUNSRESPONSE.fields_by_name[
+ "runs"
+].message_type = (
+ google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2._TRANSFERRUN
+)
+DESCRIPTOR.message_types_by_name["DataSourceParameter"] = _DATASOURCEPARAMETER
+DESCRIPTOR.message_types_by_name["DataSource"] = _DATASOURCE
+DESCRIPTOR.message_types_by_name["GetDataSourceRequest"] = _GETDATASOURCEREQUEST
+DESCRIPTOR.message_types_by_name["ListDataSourcesRequest"] = _LISTDATASOURCESREQUEST
+DESCRIPTOR.message_types_by_name["ListDataSourcesResponse"] = _LISTDATASOURCESRESPONSE
+DESCRIPTOR.message_types_by_name[
+ "CreateTransferConfigRequest"
+] = _CREATETRANSFERCONFIGREQUEST
+DESCRIPTOR.message_types_by_name[
+ "UpdateTransferConfigRequest"
+] = _UPDATETRANSFERCONFIGREQUEST
+DESCRIPTOR.message_types_by_name["GetTransferConfigRequest"] = _GETTRANSFERCONFIGREQUEST
+DESCRIPTOR.message_types_by_name[
+ "DeleteTransferConfigRequest"
+] = _DELETETRANSFERCONFIGREQUEST
+DESCRIPTOR.message_types_by_name["GetTransferRunRequest"] = _GETTRANSFERRUNREQUEST
+DESCRIPTOR.message_types_by_name["DeleteTransferRunRequest"] = _DELETETRANSFERRUNREQUEST
+DESCRIPTOR.message_types_by_name[
+ "ListTransferConfigsRequest"
+] = _LISTTRANSFERCONFIGSREQUEST
+DESCRIPTOR.message_types_by_name[
+ "ListTransferConfigsResponse"
+] = _LISTTRANSFERCONFIGSRESPONSE
+DESCRIPTOR.message_types_by_name["ListTransferRunsRequest"] = _LISTTRANSFERRUNSREQUEST
+DESCRIPTOR.message_types_by_name["ListTransferRunsResponse"] = _LISTTRANSFERRUNSRESPONSE
+DESCRIPTOR.message_types_by_name["ListTransferLogsRequest"] = _LISTTRANSFERLOGSREQUEST
+DESCRIPTOR.message_types_by_name["ListTransferLogsResponse"] = _LISTTRANSFERLOGSRESPONSE
+DESCRIPTOR.message_types_by_name["CheckValidCredsRequest"] = _CHECKVALIDCREDSREQUEST
+DESCRIPTOR.message_types_by_name["CheckValidCredsResponse"] = _CHECKVALIDCREDSRESPONSE
+DESCRIPTOR.message_types_by_name[
+ "ScheduleTransferRunsRequest"
+] = _SCHEDULETRANSFERRUNSREQUEST
+DESCRIPTOR.message_types_by_name[
+ "ScheduleTransferRunsResponse"
+] = _SCHEDULETRANSFERRUNSRESPONSE
+DESCRIPTOR.message_types_by_name[
+ "StartManualTransferRunsRequest"
+] = _STARTMANUALTRANSFERRUNSREQUEST
+DESCRIPTOR.message_types_by_name[
+ "StartManualTransferRunsResponse"
+] = _STARTMANUALTRANSFERRUNSRESPONSE
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+DataSourceParameter = _reflection.GeneratedProtocolMessageType(
+ "DataSourceParameter",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_DATASOURCEPARAMETER,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datatransfer_pb2",
+ __doc__="""Represents a data source parameter with validation rules, so that
+ parameters can be rendered in the UI. These parameters are given to us
+ by supported data sources, and include all needed information for
+ rendering and validation. Thus, whoever uses this api can decide to
+ generate either generic ui, or custom data source specific forms.
+
+
+ Attributes:
+ param_id:
+ Parameter identifier.
+ display_name:
+ Parameter display name in the user interface.
+ description:
+ Parameter description.
+ type:
+ Parameter type.
+ required:
+ Is parameter required.
+ repeated:
+ Deprecated. This field has no effect.
+ validation_regex:
+ Regular expression which can be used for parameter validation.
+ allowed_values:
+ All possible values for the parameter.
+ min_value:
+ For integer and double values specifies minimum allowed value.
+ max_value:
+ For integer and double values specifies maxminum allowed
+ value.
+ fields:
+ Deprecated. This field has no effect.
+ validation_description:
+ Description of the requirements for this field, in case the
+ user input does not fulfill the regex pattern or min/max
+ values.
+ validation_help_url:
+ URL to a help document to further explain the naming
+ requirements.
+ immutable:
+ Cannot be changed after initial creation.
+ recurse:
+ Deprecated. This field has no effect.
+ deprecated:
+ If true, it should not be used in new transfers, and it should
+ not be visible to users.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.DataSourceParameter)
+ ),
+)
+_sym_db.RegisterMessage(DataSourceParameter)
+
+DataSource = _reflection.GeneratedProtocolMessageType(
+ "DataSource",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_DATASOURCE,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datatransfer_pb2",
+ __doc__="""Represents data source metadata. Metadata is sufficient to render UI and
+ request proper OAuth tokens.
+
+
+ Attributes:
+ name:
+ Output only. Data source resource name.
+ data_source_id:
+ Data source id.
+ display_name:
+ User friendly data source name.
+ description:
+ User friendly data source description string.
+ client_id:
+ Data source client id which should be used to receive refresh
+ token.
+ scopes:
+ Api auth scopes for which refresh token needs to be obtained.
+ These are scopes needed by a data source to prepare data and
+ ingest them into BigQuery, e.g.,
+ https://www.googleapis.com/auth/bigquery
+ transfer_type:
+ Deprecated. This field has no effect.
+ supports_multiple_transfers:
+ Deprecated. This field has no effect.
+ update_deadline_seconds:
+ The number of seconds to wait for an update from the data
+ source before the Data Transfer Service marks the transfer as
+ FAILED.
+ default_schedule:
+ Default data transfer schedule. Examples of valid schedules
+ include: ``1st,3rd monday of month 15:30``, ``every wed,fri of
+ jan,jun 13:15``, and ``first sunday of quarter 00:00``.
+ supports_custom_schedule:
+ Specifies whether the data source supports a user defined
+ schedule, or operates on the default schedule. When set to
+ ``true``, user can override default schedule.
+ parameters:
+ Data source parameters.
+ help_url:
+ Url for the help document for this data source.
+ authorization_type:
+ Indicates the type of authorization.
+ data_refresh_type:
+ Specifies whether the data source supports automatic data
+ refresh for the past few days, and how it's supported. For
+ some data sources, data might not be complete until a few days
+ later, so it's useful to refresh data automatically.
+ default_data_refresh_window_days:
+ Default data refresh window on days. Only meaningful when
+ ``data_refresh_type`` = ``SLIDING_WINDOW``.
+ manual_runs_disabled:
+ Disables backfilling and manual run scheduling for the data
+ source.
+ minimum_schedule_interval:
+ The minimum interval for scheduler to schedule runs.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.DataSource)
+ ),
+)
+_sym_db.RegisterMessage(DataSource)
+
+GetDataSourceRequest = _reflection.GeneratedProtocolMessageType(
+ "GetDataSourceRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_GETDATASOURCEREQUEST,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datatransfer_pb2",
+ __doc__="""A request to get data source info.
+
+
+ Attributes:
+ name:
+ The field will contain name of the resource requested, for
+ example:
+ ``projects/{project_id}/dataSources/{data_source_id}``
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.GetDataSourceRequest)
+ ),
+)
+_sym_db.RegisterMessage(GetDataSourceRequest)
+
+ListDataSourcesRequest = _reflection.GeneratedProtocolMessageType(
+ "ListDataSourcesRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_LISTDATASOURCESREQUEST,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datatransfer_pb2",
+ __doc__="""Request to list supported data sources and their data transfer settings.
+
+
+ Attributes:
+ parent:
+ The BigQuery project id for which data sources should be
+ returned. Must be in the form: ``projects/{project_id}``
+ page_token:
+ Pagination token, which can be used to request a specific page
+ of ``ListDataSourcesRequest`` list results. For multiple-page
+ results, ``ListDataSourcesResponse`` outputs a ``next_page``
+ token, which can be used as the ``page_token`` value to
+ request the next page of list results.
+ page_size:
+ Page size. The default page size is the maximum value of 1000
+ results.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.ListDataSourcesRequest)
+ ),
+)
+_sym_db.RegisterMessage(ListDataSourcesRequest)
+
+ListDataSourcesResponse = _reflection.GeneratedProtocolMessageType(
+ "ListDataSourcesResponse",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_LISTDATASOURCESRESPONSE,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datatransfer_pb2",
+ __doc__="""Returns list of supported data sources and their metadata.
+
+
+ Attributes:
+ data_sources:
+ List of supported data sources and their transfer settings.
+ next_page_token:
+ Output only. The next-pagination token. For multiple-page list
+ results, this token can be used as the
+ ``ListDataSourcesRequest.page_token`` to request the next page
+ of list results.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.ListDataSourcesResponse)
+ ),
+)
+_sym_db.RegisterMessage(ListDataSourcesResponse)
+
+CreateTransferConfigRequest = _reflection.GeneratedProtocolMessageType(
+ "CreateTransferConfigRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_CREATETRANSFERCONFIGREQUEST,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datatransfer_pb2",
+ __doc__="""A request to create a data transfer configuration. If new credentials
+ are needed for this transfer configuration, an authorization code must
+ be provided. If an authorization code is provided, the transfer
+ configuration will be associated with the user id corresponding to the
+ authorization code. Otherwise, the transfer configuration will be
+ associated with the calling user.
+
+
+ Attributes:
+ parent:
+ The BigQuery project id where the transfer configuration
+ should be created. Must be in the format
+ projects/{project\_id}/locations/{location\_id} If specified
+ location and location of the destination bigquery dataset do
+ not match - the request will fail.
+ transfer_config:
+ Data transfer configuration to create.
+ authorization_code:
+ Optional OAuth2 authorization code to use with this transfer
+ configuration. This is required if new credentials are needed,
+ as indicated by ``CheckValidCreds``. In order to obtain
+ authorization\_code, please make a request to https://www.gsta
+ tic.com/bigquerydatatransfer/oauthz/auth?client\_id=&scope=&re
+ direct\_uri= - client\_id should be OAuth client\_id of
+ BigQuery DTS API for the given data source returned by
+ ListDataSources method. - data\_source\_scopes are the scopes
+ returned by ListDataSources method. - redirect\_uri is an
+ optional parameter. If not specified, then authorization
+ code is posted to the opener of authorization flow window.
+ Otherwise it will be sent to the redirect uri. A special
+ value of urn:ietf:wg:oauth:2.0:oob means that authorization
+ code should be returned in the title bar of the browser,
+ with the page text prompting the user to copy the code and
+ paste it in the application.
+ version_info:
+ Optional version info. If users want to find a very recent
+ access token, that is, immediately after approving access,
+ users have to set the version\_info claim in the token
+ request. To obtain the version\_info, users must use the
+ "none+gsession" response type. which be return a version\_info
+ back in the authorization response which be be put in a JWT
+ claim in the token request.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.CreateTransferConfigRequest)
+ ),
+)
+_sym_db.RegisterMessage(CreateTransferConfigRequest)
+
+UpdateTransferConfigRequest = _reflection.GeneratedProtocolMessageType(
+ "UpdateTransferConfigRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_UPDATETRANSFERCONFIGREQUEST,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datatransfer_pb2",
+ __doc__="""A request to update a transfer configuration. To update the user id of
+ the transfer configuration, an authorization code needs to be provided.
+
+
+ Attributes:
+ transfer_config:
+ Data transfer configuration to create.
+ authorization_code:
+ Optional OAuth2 authorization code to use with this transfer
+ configuration. If it is provided, the transfer configuration
+ will be associated with the authorizing user. In order to
+ obtain authorization\_code, please make a request to https://w
+ ww.gstatic.com/bigquerydatatransfer/oauthz/auth?client\_id=&sc
+ ope=&redirect\_uri= - client\_id should be OAuth client\_id
+ of BigQuery DTS API for the given data source returned by
+ ListDataSources method. - data\_source\_scopes are the scopes
+ returned by ListDataSources method. - redirect\_uri is an
+ optional parameter. If not specified, then authorization
+ code is posted to the opener of authorization flow window.
+ Otherwise it will be sent to the redirect uri. A special
+ value of urn:ietf:wg:oauth:2.0:oob means that authorization
+ code should be returned in the title bar of the browser,
+ with the page text prompting the user to copy the code and
+ paste it in the application.
+ update_mask:
+ Required list of fields to be updated in this request.
+ version_info:
+ Optional version info. If users want to find a very recent
+ access token, that is, immediately after approving access,
+ users have to set the version\_info claim in the token
+ request. To obtain the version\_info, users must use the
+ "none+gsession" response type. which be return a version\_info
+ back in the authorization response which be be put in a JWT
+ claim in the token request.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.UpdateTransferConfigRequest)
+ ),
+)
+_sym_db.RegisterMessage(UpdateTransferConfigRequest)
+
+GetTransferConfigRequest = _reflection.GeneratedProtocolMessageType(
+ "GetTransferConfigRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_GETTRANSFERCONFIGREQUEST,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datatransfer_pb2",
+ __doc__="""A request to get data transfer information.
+
+
+ Attributes:
+ name:
+ The field will contain name of the resource requested, for
+ example: ``projects/{project_id}/transferConfigs/{config_id}``
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.GetTransferConfigRequest)
+ ),
+)
+_sym_db.RegisterMessage(GetTransferConfigRequest)
+
+DeleteTransferConfigRequest = _reflection.GeneratedProtocolMessageType(
+ "DeleteTransferConfigRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_DELETETRANSFERCONFIGREQUEST,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datatransfer_pb2",
+ __doc__="""A request to delete data transfer information. All associated transfer
+ runs and log messages will be deleted as well.
+
+
+ Attributes:
+ name:
+ The field will contain name of the resource requested, for
+ example: ``projects/{project_id}/transferConfigs/{config_id}``
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.DeleteTransferConfigRequest)
+ ),
+)
+_sym_db.RegisterMessage(DeleteTransferConfigRequest)
+
+GetTransferRunRequest = _reflection.GeneratedProtocolMessageType(
+ "GetTransferRunRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_GETTRANSFERRUNREQUEST,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datatransfer_pb2",
+ __doc__="""A request to get data transfer run information.
+
+
+ Attributes:
+ name:
+ The field will contain name of the resource requested, for
+ example: ``projects/{project_id}/transferConfigs/{config_id}/r
+ uns/{run_id}``
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.GetTransferRunRequest)
+ ),
+)
+_sym_db.RegisterMessage(GetTransferRunRequest)
+
+DeleteTransferRunRequest = _reflection.GeneratedProtocolMessageType(
+ "DeleteTransferRunRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_DELETETRANSFERRUNREQUEST,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datatransfer_pb2",
+ __doc__="""A request to delete data transfer run information.
+
+
+ Attributes:
+ name:
+ The field will contain name of the resource requested, for
+ example: ``projects/{project_id}/transferConfigs/{config_id}/r
+ uns/{run_id}``
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.DeleteTransferRunRequest)
+ ),
+)
+_sym_db.RegisterMessage(DeleteTransferRunRequest)
+
+ListTransferConfigsRequest = _reflection.GeneratedProtocolMessageType(
+ "ListTransferConfigsRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_LISTTRANSFERCONFIGSREQUEST,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datatransfer_pb2",
+ __doc__="""A request to list data transfers configured for a BigQuery project.
+
+
+ Attributes:
+ parent:
+ The BigQuery project id for which data sources should be
+ returned: ``projects/{project_id}``.
+ data_source_ids:
+ When specified, only configurations of requested data sources
+ are returned.
+ page_token:
+ Pagination token, which can be used to request a specific page
+ of ``ListTransfersRequest`` list results. For multiple-page
+ results, ``ListTransfersResponse`` outputs a ``next_page``
+ token, which can be used as the ``page_token`` value to
+ request the next page of list results.
+ page_size:
+ Page size. The default page size is the maximum value of 1000
+ results.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.ListTransferConfigsRequest)
+ ),
+)
+_sym_db.RegisterMessage(ListTransferConfigsRequest)
+
+ListTransferConfigsResponse = _reflection.GeneratedProtocolMessageType(
+ "ListTransferConfigsResponse",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_LISTTRANSFERCONFIGSRESPONSE,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datatransfer_pb2",
+ __doc__="""The returned list of pipelines in the project.
+
+
+ Attributes:
+ transfer_configs:
+ Output only. The stored pipeline transfer configurations.
+ next_page_token:
+ Output only. The next-pagination token. For multiple-page list
+ results, this token can be used as the
+ ``ListTransferConfigsRequest.page_token`` to request the next
+ page of list results.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.ListTransferConfigsResponse)
+ ),
+)
+_sym_db.RegisterMessage(ListTransferConfigsResponse)
+
+ListTransferRunsRequest = _reflection.GeneratedProtocolMessageType(
+ "ListTransferRunsRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_LISTTRANSFERRUNSREQUEST,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datatransfer_pb2",
+ __doc__="""A request to list data transfer runs. UI can use this method to
+ show/filter specific data transfer runs. The data source can use this
+ method to request all scheduled transfer runs.
+
+
+ Attributes:
+ parent:
+ Name of transfer configuration for which transfer runs should
+ be retrieved. Format of transfer configuration resource name
+ is: ``projects/{project_id}/transferConfigs/{config_id}``.
+ states:
+ When specified, only transfer runs with requested states are
+ returned.
+ page_token:
+ Pagination token, which can be used to request a specific page
+ of ``ListTransferRunsRequest`` list results. For multiple-page
+ results, ``ListTransferRunsResponse`` outputs a ``next_page``
+ token, which can be used as the ``page_token`` value to
+ request the next page of list results.
+ page_size:
+ Page size. The default page size is the maximum value of 1000
+ results.
+ run_attempt:
+ Indicates how run attempts are to be pulled.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.ListTransferRunsRequest)
+ ),
+)
+_sym_db.RegisterMessage(ListTransferRunsRequest)
+
+ListTransferRunsResponse = _reflection.GeneratedProtocolMessageType(
+ "ListTransferRunsResponse",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_LISTTRANSFERRUNSRESPONSE,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datatransfer_pb2",
+ __doc__="""The returned list of pipelines in the project.
+
+
+ Attributes:
+ transfer_runs:
+ Output only. The stored pipeline transfer runs.
+ next_page_token:
+ Output only. The next-pagination token. For multiple-page list
+ results, this token can be used as the
+ ``ListTransferRunsRequest.page_token`` to request the next
+ page of list results.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.ListTransferRunsResponse)
+ ),
+)
+_sym_db.RegisterMessage(ListTransferRunsResponse)
+
+ListTransferLogsRequest = _reflection.GeneratedProtocolMessageType(
+ "ListTransferLogsRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_LISTTRANSFERLOGSREQUEST,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datatransfer_pb2",
+ __doc__="""A request to get user facing log messages associated with data transfer
+ run.
+
+
+ Attributes:
+ parent:
+ Transfer run name in the form: ``projects/{project_id}/transfe
+ rConfigs/{config_Id}/runs/{run_id}``.
+ page_token:
+ Pagination token, which can be used to request a specific page
+ of ``ListTransferLogsRequest`` list results. For multiple-page
+ results, ``ListTransferLogsResponse`` outputs a ``next_page``
+ token, which can be used as the ``page_token`` value to
+ request the next page of list results.
+ page_size:
+ Page size. The default page size is the maximum value of 1000
+ results.
+ message_types:
+ Message types to return. If not populated - INFO, WARNING and
+ ERROR messages are returned.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.ListTransferLogsRequest)
+ ),
+)
+_sym_db.RegisterMessage(ListTransferLogsRequest)
+
+ListTransferLogsResponse = _reflection.GeneratedProtocolMessageType(
+ "ListTransferLogsResponse",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_LISTTRANSFERLOGSRESPONSE,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datatransfer_pb2",
+ __doc__="""The returned list transfer run messages.
+
+
+ Attributes:
+ transfer_messages:
+ Output only. The stored pipeline transfer messages.
+ next_page_token:
+ Output only. The next-pagination token. For multiple-page list
+ results, this token can be used as the
+ ``GetTransferRunLogRequest.page_token`` to request the next
+ page of list results.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.ListTransferLogsResponse)
+ ),
+)
+_sym_db.RegisterMessage(ListTransferLogsResponse)
+
+CheckValidCredsRequest = _reflection.GeneratedProtocolMessageType(
+ "CheckValidCredsRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_CHECKVALIDCREDSREQUEST,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datatransfer_pb2",
+ __doc__="""A request to determine whether the user has valid credentials. This
+ method is used to limit the number of OAuth popups in the user
+ interface. The user id is inferred from the API call context. If the
+ data source has the Google+ authorization type, this method returns
+ false, as it cannot be determined whether the credentials are already
+ valid merely based on the user id.
+
+
+ Attributes:
+ name:
+ The data source in the form:
+ ``projects/{project_id}/dataSources/{data_source_id}``
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.CheckValidCredsRequest)
+ ),
+)
+_sym_db.RegisterMessage(CheckValidCredsRequest)
+
+CheckValidCredsResponse = _reflection.GeneratedProtocolMessageType(
+ "CheckValidCredsResponse",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_CHECKVALIDCREDSRESPONSE,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datatransfer_pb2",
+ __doc__="""A response indicating whether the credentials exist and are valid.
+
+
+ Attributes:
+ has_valid_creds:
+ If set to ``true``, the credentials exist and are valid.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.CheckValidCredsResponse)
+ ),
+)
+_sym_db.RegisterMessage(CheckValidCredsResponse)
+
+ScheduleTransferRunsRequest = _reflection.GeneratedProtocolMessageType(
+ "ScheduleTransferRunsRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_SCHEDULETRANSFERRUNSREQUEST,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datatransfer_pb2",
+ __doc__="""A request to schedule transfer runs for a time range.
+
+
+ Attributes:
+ parent:
+ Transfer configuration name in the form:
+ ``projects/{project_id}/transferConfigs/{config_id}``.
+ start_time:
+ Start time of the range of transfer runs. For example,
+ ``"2017-05-25T00:00:00+00:00"``.
+ end_time:
+ End time of the range of transfer runs. For example,
+ ``"2017-05-30T00:00:00+00:00"``.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.ScheduleTransferRunsRequest)
+ ),
+)
+_sym_db.RegisterMessage(ScheduleTransferRunsRequest)
+
+ScheduleTransferRunsResponse = _reflection.GeneratedProtocolMessageType(
+ "ScheduleTransferRunsResponse",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_SCHEDULETRANSFERRUNSRESPONSE,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datatransfer_pb2",
+ __doc__="""A response to schedule transfer runs for a time range.
+
+
+ Attributes:
+ runs:
+ The transfer runs that were scheduled.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.ScheduleTransferRunsResponse)
+ ),
+)
+_sym_db.RegisterMessage(ScheduleTransferRunsResponse)
+
+StartManualTransferRunsRequest = _reflection.GeneratedProtocolMessageType(
+ "StartManualTransferRunsRequest",
+ (_message.Message,),
+ dict(
+ TimeRange=_reflection.GeneratedProtocolMessageType(
+ "TimeRange",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_STARTMANUALTRANSFERRUNSREQUEST_TIMERANGE,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datatransfer_pb2",
+ __doc__="""A specification for a time range, this will request transfer runs with
+ run\_time between start\_time (inclusive) and end\_time (exclusive).
+
+
+ Attributes:
+ start_time:
+ Start time of the range of transfer runs. For example,
+ ``"2017-05-25T00:00:00+00:00"``. The start\_time must be
+ strictly less than the end\_time. Creates transfer runs where
+ run\_time is in the range betwen start\_time (inclusive) and
+ end\_time (exlusive).
+ end_time:
+ End time of the range of transfer runs. For example,
+ ``"2017-05-30T00:00:00+00:00"``. The end\_time must not be in
+ the future. Creates transfer runs where run\_time is in the
+ range betwen start\_time (inclusive) and end\_time (exlusive).
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.StartManualTransferRunsRequest.TimeRange)
+ ),
+ ),
+ DESCRIPTOR=_STARTMANUALTRANSFERRUNSREQUEST,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datatransfer_pb2",
+ __doc__="""A request to start manual transfer runs.
+
+
+ Attributes:
+ parent:
+ Transfer configuration name in the form:
+ ``projects/{project_id}/transferConfigs/{config_id}``.
+ time:
+ The requested time specification - this can be a time range or
+ a specific run\_time.
+ requested_time_range:
+ Time range for the transfer runs that should be started.
+ requested_run_time:
+ Specific run\_time for a transfer run to be started. The
+ requested\_run\_time must not be in the future.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.StartManualTransferRunsRequest)
+ ),
+)
+_sym_db.RegisterMessage(StartManualTransferRunsRequest)
+_sym_db.RegisterMessage(StartManualTransferRunsRequest.TimeRange)
+
+StartManualTransferRunsResponse = _reflection.GeneratedProtocolMessageType(
+ "StartManualTransferRunsResponse",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_STARTMANUALTRANSFERRUNSRESPONSE,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.datatransfer_pb2",
+ __doc__="""A response to start manual transfer runs.
+
+
+ Attributes:
+ runs:
+ The transfer runs that were created.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.StartManualTransferRunsResponse)
+ ),
+)
+_sym_db.RegisterMessage(StartManualTransferRunsResponse)
+
+
+DESCRIPTOR._options = None
+_DATASOURCE.fields_by_name["transfer_type"]._options = None
+_DATASOURCE.fields_by_name["supports_multiple_transfers"]._options = None
+
+_DATATRANSFERSERVICE = _descriptor.ServiceDescriptor(
+ name="DataTransferService",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataTransferService",
+ file=DESCRIPTOR,
+ index=0,
+ serialized_options=_b(
+ "\312A#bigquerydatatransfer.googleapis.com\322A.https://www.googleapis.com/auth/cloud-platform"
+ ),
+ serialized_start=4581,
+ serialized_end=8454,
+ methods=[
+ _descriptor.MethodDescriptor(
+ name="GetDataSource",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataTransferService.GetDataSource",
+ index=0,
+ containing_service=None,
+ input_type=_GETDATASOURCEREQUEST,
+ output_type=_DATASOURCE,
+ serialized_options=_b(
+ "\202\323\344\223\002X\022//v1/{name=projects/*/locations/*/dataSources/*}Z%\022#/v1/{name=projects/*/dataSources/*}"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="ListDataSources",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataTransferService.ListDataSources",
+ index=1,
+ containing_service=None,
+ input_type=_LISTDATASOURCESREQUEST,
+ output_type=_LISTDATASOURCESRESPONSE,
+ serialized_options=_b(
+ "\202\323\344\223\002X\022//v1/{parent=projects/*/locations/*}/dataSourcesZ%\022#/v1/{parent=projects/*}/dataSources"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="CreateTransferConfig",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataTransferService.CreateTransferConfig",
+ index=2,
+ containing_service=None,
+ input_type=_CREATETRANSFERCONFIGREQUEST,
+ output_type=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2._TRANSFERCONFIG,
+ serialized_options=_b(
+ '\202\323\344\223\002\202\001"3/v1/{parent=projects/*/locations/*}/transferConfigs:\017transfer_configZ:"\'/v1/{parent=projects/*}/transferConfigs:\017transfer_config'
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="UpdateTransferConfig",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataTransferService.UpdateTransferConfig",
+ index=3,
+ containing_service=None,
+ input_type=_UPDATETRANSFERCONFIGREQUEST,
+ output_type=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2._TRANSFERCONFIG,
+ serialized_options=_b(
+ "\202\323\344\223\002\242\0012C/v1/{transfer_config.name=projects/*/locations/*/transferConfigs/*}:\017transfer_configZJ27/v1/{transfer_config.name=projects/*/transferConfigs/*}:\017transfer_config"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="DeleteTransferConfig",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataTransferService.DeleteTransferConfig",
+ index=4,
+ containing_service=None,
+ input_type=_DELETETRANSFERCONFIGREQUEST,
+ output_type=google_dot_protobuf_dot_empty__pb2._EMPTY,
+ serialized_options=_b(
+ "\202\323\344\223\002`*3/v1/{name=projects/*/locations/*/transferConfigs/*}Z)*'/v1/{name=projects/*/transferConfigs/*}"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="GetTransferConfig",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataTransferService.GetTransferConfig",
+ index=5,
+ containing_service=None,
+ input_type=_GETTRANSFERCONFIGREQUEST,
+ output_type=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2._TRANSFERCONFIG,
+ serialized_options=_b(
+ "\202\323\344\223\002`\0223/v1/{name=projects/*/locations/*/transferConfigs/*}Z)\022'/v1/{name=projects/*/transferConfigs/*}"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="ListTransferConfigs",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataTransferService.ListTransferConfigs",
+ index=6,
+ containing_service=None,
+ input_type=_LISTTRANSFERCONFIGSREQUEST,
+ output_type=_LISTTRANSFERCONFIGSRESPONSE,
+ serialized_options=_b(
+ "\202\323\344\223\002`\0223/v1/{parent=projects/*/locations/*}/transferConfigsZ)\022'/v1/{parent=projects/*}/transferConfigs"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="ScheduleTransferRuns",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataTransferService.ScheduleTransferRuns",
+ index=7,
+ containing_service=None,
+ input_type=_SCHEDULETRANSFERRUNSREQUEST,
+ output_type=_SCHEDULETRANSFERRUNSRESPONSE,
+ serialized_options=_b(
+ '\210\002\001\202\323\344\223\002\204\001"B/v1/{parent=projects/*/locations/*/transferConfigs/*}:scheduleRuns:\001*Z;"6/v1/{parent=projects/*/transferConfigs/*}:scheduleRuns:\001*'
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="StartManualTransferRuns",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataTransferService.StartManualTransferRuns",
+ index=8,
+ containing_service=None,
+ input_type=_STARTMANUALTRANSFERRUNSREQUEST,
+ output_type=_STARTMANUALTRANSFERRUNSRESPONSE,
+ serialized_options=_b(
+ '\202\323\344\223\002\212\001"E/v1/{parent=projects/*/locations/*/transferConfigs/*}:startManualRuns:\001*Z>"9/v1/{parent=projects/*/transferConfigs/*}:startManualRuns:\001*'
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="GetTransferRun",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataTransferService.GetTransferRun",
+ index=9,
+ containing_service=None,
+ input_type=_GETTRANSFERRUNREQUEST,
+ output_type=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2._TRANSFERRUN,
+ serialized_options=_b(
+ "\202\323\344\223\002n\022:/v1/{name=projects/*/locations/*/transferConfigs/*/runs/*}Z0\022./v1/{name=projects/*/transferConfigs/*/runs/*}"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="DeleteTransferRun",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataTransferService.DeleteTransferRun",
+ index=10,
+ containing_service=None,
+ input_type=_DELETETRANSFERRUNREQUEST,
+ output_type=google_dot_protobuf_dot_empty__pb2._EMPTY,
+ serialized_options=_b(
+ "\202\323\344\223\002n*:/v1/{name=projects/*/locations/*/transferConfigs/*/runs/*}Z0*./v1/{name=projects/*/transferConfigs/*/runs/*}"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="ListTransferRuns",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataTransferService.ListTransferRuns",
+ index=11,
+ containing_service=None,
+ input_type=_LISTTRANSFERRUNSREQUEST,
+ output_type=_LISTTRANSFERRUNSRESPONSE,
+ serialized_options=_b(
+ "\202\323\344\223\002n\022:/v1/{parent=projects/*/locations/*/transferConfigs/*}/runsZ0\022./v1/{parent=projects/*/transferConfigs/*}/runs"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="ListTransferLogs",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataTransferService.ListTransferLogs",
+ index=12,
+ containing_service=None,
+ input_type=_LISTTRANSFERLOGSREQUEST,
+ output_type=_LISTTRANSFERLOGSRESPONSE,
+ serialized_options=_b(
+ "\202\323\344\223\002\214\001\022I/v1/{parent=projects/*/locations/*/transferConfigs/*/runs/*}/transferLogsZ?\022=/v1/{parent=projects/*/transferConfigs/*/runs/*}/transferLogs"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="CheckValidCreds",
+ full_name="google.cloud.bigquery.datatransfer.v1.DataTransferService.CheckValidCreds",
+ index=13,
+ containing_service=None,
+ input_type=_CHECKVALIDCREDSREQUEST,
+ output_type=_CHECKVALIDCREDSRESPONSE,
+ serialized_options=_b(
+ '\202\323\344\223\002~"?/v1/{name=projects/*/locations/*/dataSources/*}:checkValidCreds:\001*Z8"3/v1/{name=projects/*/dataSources/*}:checkValidCreds:\001*'
+ ),
+ ),
+ ],
+)
+_sym_db.RegisterServiceDescriptor(_DATATRANSFERSERVICE)
+
+DESCRIPTOR.services_by_name["DataTransferService"] = _DATATRANSFERSERVICE
+
+# @@protoc_insertion_point(module_scope)
diff --git a/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/proto/datatransfer_pb2_grpc.py b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/proto/datatransfer_pb2_grpc.py
new file mode 100644
index 000000000000..ab525e22457c
--- /dev/null
+++ b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/proto/datatransfer_pb2_grpc.py
@@ -0,0 +1,296 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
+
+from google.cloud.bigquery_datatransfer_v1.proto import (
+ datatransfer_pb2 as google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2,
+)
+from google.cloud.bigquery_datatransfer_v1.proto import (
+ transfer_pb2 as google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2,
+)
+from google.protobuf import empty_pb2 as google_dot_protobuf_dot_empty__pb2
+
+
+class DataTransferServiceStub(object):
+ """The Google BigQuery Data Transfer Service API enables BigQuery users to
+ configure the transfer of their data from other Google Products into
+ BigQuery. This service contains methods that are end user exposed. It backs
+ up the frontend.
+ """
+
+ def __init__(self, channel):
+ """Constructor.
+
+ Args:
+ channel: A grpc.Channel.
+ """
+ self.GetDataSource = channel.unary_unary(
+ "/google.cloud.bigquery.datatransfer.v1.DataTransferService/GetDataSource",
+ request_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.GetDataSourceRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.DataSource.FromString,
+ )
+ self.ListDataSources = channel.unary_unary(
+ "/google.cloud.bigquery.datatransfer.v1.DataTransferService/ListDataSources",
+ request_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.ListDataSourcesRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.ListDataSourcesResponse.FromString,
+ )
+ self.CreateTransferConfig = channel.unary_unary(
+ "/google.cloud.bigquery.datatransfer.v1.DataTransferService/CreateTransferConfig",
+ request_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.CreateTransferConfigRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2.TransferConfig.FromString,
+ )
+ self.UpdateTransferConfig = channel.unary_unary(
+ "/google.cloud.bigquery.datatransfer.v1.DataTransferService/UpdateTransferConfig",
+ request_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.UpdateTransferConfigRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2.TransferConfig.FromString,
+ )
+ self.DeleteTransferConfig = channel.unary_unary(
+ "/google.cloud.bigquery.datatransfer.v1.DataTransferService/DeleteTransferConfig",
+ request_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.DeleteTransferConfigRequest.SerializeToString,
+ response_deserializer=google_dot_protobuf_dot_empty__pb2.Empty.FromString,
+ )
+ self.GetTransferConfig = channel.unary_unary(
+ "/google.cloud.bigquery.datatransfer.v1.DataTransferService/GetTransferConfig",
+ request_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.GetTransferConfigRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2.TransferConfig.FromString,
+ )
+ self.ListTransferConfigs = channel.unary_unary(
+ "/google.cloud.bigquery.datatransfer.v1.DataTransferService/ListTransferConfigs",
+ request_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.ListTransferConfigsRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.ListTransferConfigsResponse.FromString,
+ )
+ self.ScheduleTransferRuns = channel.unary_unary(
+ "/google.cloud.bigquery.datatransfer.v1.DataTransferService/ScheduleTransferRuns",
+ request_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.ScheduleTransferRunsRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.ScheduleTransferRunsResponse.FromString,
+ )
+ self.StartManualTransferRuns = channel.unary_unary(
+ "/google.cloud.bigquery.datatransfer.v1.DataTransferService/StartManualTransferRuns",
+ request_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.StartManualTransferRunsRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.StartManualTransferRunsResponse.FromString,
+ )
+ self.GetTransferRun = channel.unary_unary(
+ "/google.cloud.bigquery.datatransfer.v1.DataTransferService/GetTransferRun",
+ request_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.GetTransferRunRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2.TransferRun.FromString,
+ )
+ self.DeleteTransferRun = channel.unary_unary(
+ "/google.cloud.bigquery.datatransfer.v1.DataTransferService/DeleteTransferRun",
+ request_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.DeleteTransferRunRequest.SerializeToString,
+ response_deserializer=google_dot_protobuf_dot_empty__pb2.Empty.FromString,
+ )
+ self.ListTransferRuns = channel.unary_unary(
+ "/google.cloud.bigquery.datatransfer.v1.DataTransferService/ListTransferRuns",
+ request_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.ListTransferRunsRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.ListTransferRunsResponse.FromString,
+ )
+ self.ListTransferLogs = channel.unary_unary(
+ "/google.cloud.bigquery.datatransfer.v1.DataTransferService/ListTransferLogs",
+ request_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.ListTransferLogsRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.ListTransferLogsResponse.FromString,
+ )
+ self.CheckValidCreds = channel.unary_unary(
+ "/google.cloud.bigquery.datatransfer.v1.DataTransferService/CheckValidCreds",
+ request_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.CheckValidCredsRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.CheckValidCredsResponse.FromString,
+ )
+
+
+class DataTransferServiceServicer(object):
+ """The Google BigQuery Data Transfer Service API enables BigQuery users to
+ configure the transfer of their data from other Google Products into
+ BigQuery. This service contains methods that are end user exposed. It backs
+ up the frontend.
+ """
+
+ def GetDataSource(self, request, context):
+ """Retrieves a supported data source and returns its settings,
+ which can be used for UI rendering.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def ListDataSources(self, request, context):
+ """Lists supported data sources and returns their settings,
+ which can be used for UI rendering.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def CreateTransferConfig(self, request, context):
+ """Creates a new data transfer configuration.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def UpdateTransferConfig(self, request, context):
+ """Updates a data transfer configuration.
+ All fields must be set, even if they are not updated.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def DeleteTransferConfig(self, request, context):
+ """Deletes a data transfer configuration,
+ including any associated transfer runs and logs.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def GetTransferConfig(self, request, context):
+ """Returns information about a data transfer config.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def ListTransferConfigs(self, request, context):
+ """Returns information about all data transfers in the project.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def ScheduleTransferRuns(self, request, context):
+ """Creates transfer runs for a time range [start_time, end_time].
+ For each date - or whatever granularity the data source supports - in the
+ range, one transfer run is created.
+ Note that runs are created per UTC time in the time range.
+ DEPRECATED: use StartManualTransferRuns instead.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def StartManualTransferRuns(self, request, context):
+ """Start manual transfer runs to be executed now with schedule_time equal to
+ current time. The transfer runs can be created for a time range where the
+ run_time is between start_time (inclusive) and end_time (exclusive), or for
+ a specific run_time.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def GetTransferRun(self, request, context):
+ """Returns information about the particular transfer run.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def DeleteTransferRun(self, request, context):
+ """Deletes the specified transfer run.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def ListTransferRuns(self, request, context):
+ """Returns information about running and completed jobs.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def ListTransferLogs(self, request, context):
+ """Returns user facing log messages for the data transfer run.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def CheckValidCreds(self, request, context):
+ """Returns true if valid credentials exist for the given data source and
+ requesting user.
+ Some data sources doesn't support service account, so we need to talk to
+ them on behalf of the end user. This API just checks whether we have OAuth
+ token for the particular user, which is a pre-requisite before user can
+ create a transfer config.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+
+def add_DataTransferServiceServicer_to_server(servicer, server):
+ rpc_method_handlers = {
+ "GetDataSource": grpc.unary_unary_rpc_method_handler(
+ servicer.GetDataSource,
+ request_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.GetDataSourceRequest.FromString,
+ response_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.DataSource.SerializeToString,
+ ),
+ "ListDataSources": grpc.unary_unary_rpc_method_handler(
+ servicer.ListDataSources,
+ request_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.ListDataSourcesRequest.FromString,
+ response_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.ListDataSourcesResponse.SerializeToString,
+ ),
+ "CreateTransferConfig": grpc.unary_unary_rpc_method_handler(
+ servicer.CreateTransferConfig,
+ request_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.CreateTransferConfigRequest.FromString,
+ response_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2.TransferConfig.SerializeToString,
+ ),
+ "UpdateTransferConfig": grpc.unary_unary_rpc_method_handler(
+ servicer.UpdateTransferConfig,
+ request_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.UpdateTransferConfigRequest.FromString,
+ response_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2.TransferConfig.SerializeToString,
+ ),
+ "DeleteTransferConfig": grpc.unary_unary_rpc_method_handler(
+ servicer.DeleteTransferConfig,
+ request_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.DeleteTransferConfigRequest.FromString,
+ response_serializer=google_dot_protobuf_dot_empty__pb2.Empty.SerializeToString,
+ ),
+ "GetTransferConfig": grpc.unary_unary_rpc_method_handler(
+ servicer.GetTransferConfig,
+ request_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.GetTransferConfigRequest.FromString,
+ response_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2.TransferConfig.SerializeToString,
+ ),
+ "ListTransferConfigs": grpc.unary_unary_rpc_method_handler(
+ servicer.ListTransferConfigs,
+ request_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.ListTransferConfigsRequest.FromString,
+ response_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.ListTransferConfigsResponse.SerializeToString,
+ ),
+ "ScheduleTransferRuns": grpc.unary_unary_rpc_method_handler(
+ servicer.ScheduleTransferRuns,
+ request_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.ScheduleTransferRunsRequest.FromString,
+ response_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.ScheduleTransferRunsResponse.SerializeToString,
+ ),
+ "StartManualTransferRuns": grpc.unary_unary_rpc_method_handler(
+ servicer.StartManualTransferRuns,
+ request_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.StartManualTransferRunsRequest.FromString,
+ response_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.StartManualTransferRunsResponse.SerializeToString,
+ ),
+ "GetTransferRun": grpc.unary_unary_rpc_method_handler(
+ servicer.GetTransferRun,
+ request_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.GetTransferRunRequest.FromString,
+ response_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_transfer__pb2.TransferRun.SerializeToString,
+ ),
+ "DeleteTransferRun": grpc.unary_unary_rpc_method_handler(
+ servicer.DeleteTransferRun,
+ request_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.DeleteTransferRunRequest.FromString,
+ response_serializer=google_dot_protobuf_dot_empty__pb2.Empty.SerializeToString,
+ ),
+ "ListTransferRuns": grpc.unary_unary_rpc_method_handler(
+ servicer.ListTransferRuns,
+ request_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.ListTransferRunsRequest.FromString,
+ response_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.ListTransferRunsResponse.SerializeToString,
+ ),
+ "ListTransferLogs": grpc.unary_unary_rpc_method_handler(
+ servicer.ListTransferLogs,
+ request_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.ListTransferLogsRequest.FromString,
+ response_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.ListTransferLogsResponse.SerializeToString,
+ ),
+ "CheckValidCreds": grpc.unary_unary_rpc_method_handler(
+ servicer.CheckValidCreds,
+ request_deserializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.CheckValidCredsRequest.FromString,
+ response_serializer=google_dot_cloud_dot_bigquery_dot_datatransfer__v1_dot_proto_dot_datatransfer__pb2.CheckValidCredsResponse.SerializeToString,
+ ),
+ }
+ generic_handler = grpc.method_handlers_generic_handler(
+ "google.cloud.bigquery.datatransfer.v1.DataTransferService", rpc_method_handlers
+ )
+ server.add_generic_rpc_handlers((generic_handler,))
diff --git a/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/proto/transfer.proto b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/proto/transfer.proto
new file mode 100644
index 000000000000..9501ea3cc3b3
--- /dev/null
+++ b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/proto/transfer.proto
@@ -0,0 +1,242 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.bigquery.datatransfer.v1;
+
+import "google/api/annotations.proto";
+import "google/protobuf/struct.proto";
+import "google/protobuf/timestamp.proto";
+import "google/rpc/status.proto";
+
+option csharp_namespace = "Google.Cloud.BigQuery.DataTransfer.V1";
+option go_package = "google.golang.org/genproto/googleapis/cloud/bigquery/datatransfer/v1;datatransfer";
+option java_multiple_files = true;
+option java_outer_classname = "TransferProto";
+option java_package = "com.google.cloud.bigquery.datatransfer.v1";
+option objc_class_prefix = "GCBDT";
+option php_namespace = "Google\\Cloud\\BigQuery\\DataTransfer\\V1";
+
+// DEPRECATED. Represents data transfer type.
+enum TransferType {
+ option deprecated = true;
+
+ // Invalid or Unknown transfer type placeholder.
+ TRANSFER_TYPE_UNSPECIFIED = 0;
+
+ // Batch data transfer.
+ BATCH = 1;
+
+ // Streaming data transfer. Streaming data source currently doesn't
+ // support multiple transfer configs per project.
+ STREAMING = 2;
+}
+
+// Represents data transfer run state.
+enum TransferState {
+ // State placeholder.
+ TRANSFER_STATE_UNSPECIFIED = 0;
+
+ // Data transfer is scheduled and is waiting to be picked up by
+ // data transfer backend.
+ PENDING = 2;
+
+ // Data transfer is in progress.
+ RUNNING = 3;
+
+ // Data transfer completed successfully.
+ SUCCEEDED = 4;
+
+ // Data transfer failed.
+ FAILED = 5;
+
+ // Data transfer is cancelled.
+ CANCELLED = 6;
+}
+
+// Options customizing the data transfer schedule.
+message ScheduleOptions {
+ // If true, automatic scheduling of data transfer runs for this configuration
+ // will be disabled. The runs can be started on ad-hoc basis using
+ // StartManualTransferRuns API. When automatic scheduling is disabled, the
+ // TransferConfig.schedule field will be ignored.
+ bool disable_auto_scheduling = 3;
+
+ // Specifies time to start scheduling transfer runs. The first run will be
+ // scheduled at or after the start time according to a recurrence pattern
+ // defined in the schedule string. The start time can be changed at any
+ // moment. The time when a data transfer can be trigerred manually is not
+ // limited by this option.
+ google.protobuf.Timestamp start_time = 1;
+
+ // Defines time to stop scheduling transfer runs. A transfer run cannot be
+ // scheduled at or after the end time. The end time can be changed at any
+ // moment. The time when a data transfer can be trigerred manually is not
+ // limited by this option.
+ google.protobuf.Timestamp end_time = 2;
+}
+
+// Represents a data transfer configuration. A transfer configuration
+// contains all metadata needed to perform a data transfer. For example,
+// `destination_dataset_id` specifies where data should be stored.
+// When a new transfer configuration is created, the specified
+// `destination_dataset_id` is created when needed and shared with the
+// appropriate data source service account.
+message TransferConfig {
+ // The resource name of the transfer config.
+ // Transfer config names have the form of
+ // `projects/{project_id}/locations/{region}/transferConfigs/{config_id}`.
+ // The name is automatically generated based on the config_id specified in
+ // CreateTransferConfigRequest along with project_id and region. If config_id
+ // is not provided, usually a uuid, even though it is not guaranteed or
+ // required, will be generated for config_id.
+ string name = 1;
+
+ // The BigQuery target dataset id.
+ string destination_dataset_id = 2;
+
+ // User specified display name for the data transfer.
+ string display_name = 3;
+
+ // Data source id. Cannot be changed once data transfer is created.
+ string data_source_id = 5;
+
+ // Data transfer specific parameters.
+ google.protobuf.Struct params = 9;
+
+ // Data transfer schedule.
+ // If the data source does not support a custom schedule, this should be
+ // empty. If it is empty, the default value for the data source will be
+ // used.
+ // The specified times are in UTC.
+ // Examples of valid format:
+ // `1st,3rd monday of month 15:30`,
+ // `every wed,fri of jan,jun 13:15`, and
+ // `first sunday of quarter 00:00`.
+ // See more explanation about the format here:
+ // https://cloud.google.com/appengine/docs/flexible/python/scheduling-jobs-with-cron-yaml#the_schedule_format
+ // NOTE: the granularity should be at least 8 hours, or less frequent.
+ string schedule = 7;
+
+ // Options customizing the data transfer schedule.
+ ScheduleOptions schedule_options = 24;
+
+ // The number of days to look back to automatically refresh the data.
+ // For example, if `data_refresh_window_days = 10`, then every day
+ // BigQuery reingests data for [today-10, today-1], rather than ingesting data
+ // for just [today-1].
+ // Only valid if the data source supports the feature. Set the value to 0
+ // to use the default value.
+ int32 data_refresh_window_days = 12;
+
+ // Is this config disabled. When set to true, no runs are scheduled
+ // for a given transfer.
+ bool disabled = 13;
+
+ // Output only. Data transfer modification time. Ignored by server on input.
+ google.protobuf.Timestamp update_time = 4;
+
+ // Output only. Next time when data transfer will run.
+ google.protobuf.Timestamp next_run_time = 8;
+
+ // Output only. State of the most recently updated transfer run.
+ TransferState state = 10;
+
+ // Deprecated. Unique ID of the user on whose behalf transfer is done.
+ int64 user_id = 11;
+
+ // Output only. Region in which BigQuery dataset is located.
+ string dataset_region = 14;
+}
+
+// Represents a data transfer run.
+message TransferRun {
+ // The resource name of the transfer run.
+ // Transfer run names have the form
+ // `projects/{project_id}/locations/{location}/transferConfigs/{config_id}/runs/{run_id}`.
+ // The name is ignored when creating a transfer run.
+ string name = 1;
+
+ // Minimum time after which a transfer run can be started.
+ google.protobuf.Timestamp schedule_time = 3;
+
+ // For batch transfer runs, specifies the date and time of the data should be
+ // ingested.
+ google.protobuf.Timestamp run_time = 10;
+
+ // Status of the transfer run.
+ google.rpc.Status error_status = 21;
+
+ // Output only. Time when transfer run was started.
+ // Parameter ignored by server for input requests.
+ google.protobuf.Timestamp start_time = 4;
+
+ // Output only. Time when transfer run ended.
+ // Parameter ignored by server for input requests.
+ google.protobuf.Timestamp end_time = 5;
+
+ // Output only. Last time the data transfer run state was updated.
+ google.protobuf.Timestamp update_time = 6;
+
+ // Output only. Data transfer specific parameters.
+ google.protobuf.Struct params = 9;
+
+ // Output only. The BigQuery target dataset id.
+ string destination_dataset_id = 2;
+
+ // Output only. Data source id.
+ string data_source_id = 7;
+
+ // Data transfer run state. Ignored for input requests.
+ TransferState state = 8;
+
+ // Deprecated. Unique ID of the user on whose behalf transfer is done.
+ int64 user_id = 11;
+
+ // Output only. Describes the schedule of this transfer run if it was
+ // created as part of a regular schedule. For batch transfer runs that are
+ // scheduled manually, this is empty.
+ // NOTE: the system might choose to delay the schedule depending on the
+ // current load, so `schedule_time` doesn't always match this.
+ string schedule = 12;
+}
+
+// Represents a user facing message for a particular data transfer run.
+message TransferMessage {
+ // Represents data transfer user facing message severity.
+ enum MessageSeverity {
+ // No severity specified.
+ MESSAGE_SEVERITY_UNSPECIFIED = 0;
+
+ // Informational message.
+ INFO = 1;
+
+ // Warning message.
+ WARNING = 2;
+
+ // Error message.
+ ERROR = 3;
+ }
+
+ // Time when message was logged.
+ google.protobuf.Timestamp message_time = 1;
+
+ // Message severity.
+ MessageSeverity severity = 2;
+
+ // Message text.
+ string message_text = 3;
+}
diff --git a/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/proto/transfer_pb2.py b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/proto/transfer_pb2.py
new file mode 100644
index 000000000000..437774a5f201
--- /dev/null
+++ b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/proto/transfer_pb2.py
@@ -0,0 +1,1066 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/bigquery/datatransfer_v1/proto/transfer.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf.internal import enum_type_wrapper
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+from google.protobuf import struct_pb2 as google_dot_protobuf_dot_struct__pb2
+from google.protobuf import timestamp_pb2 as google_dot_protobuf_dot_timestamp__pb2
+from google.rpc import status_pb2 as google_dot_rpc_dot_status__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/bigquery/datatransfer_v1/proto/transfer.proto",
+ package="google.cloud.bigquery.datatransfer.v1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n)com.google.cloud.bigquery.datatransfer.v1B\rTransferProtoP\001ZQgoogle.golang.org/genproto/googleapis/cloud/bigquery/datatransfer/v1;datatransfer\242\002\005GCBDT\252\002%Google.Cloud.BigQuery.DataTransfer.V1\312\002%Google\\Cloud\\BigQuery\\DataTransfer\\V1"
+ ),
+ serialized_pb=_b(
+ '\n:google/cloud/bigquery/datatransfer_v1/proto/transfer.proto\x12%google.cloud.bigquery.datatransfer.v1\x1a\x1cgoogle/api/annotations.proto\x1a\x1cgoogle/protobuf/struct.proto\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x17google/rpc/status.proto"\x90\x01\n\x0fScheduleOptions\x12\x1f\n\x17\x64isable_auto_scheduling\x18\x03 \x01(\x08\x12.\n\nstart_time\x18\x01 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12,\n\x08\x65nd_time\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.Timestamp"\xff\x03\n\x0eTransferConfig\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x1e\n\x16\x64\x65stination_dataset_id\x18\x02 \x01(\t\x12\x14\n\x0c\x64isplay_name\x18\x03 \x01(\t\x12\x16\n\x0e\x64\x61ta_source_id\x18\x05 \x01(\t\x12\'\n\x06params\x18\t \x01(\x0b\x32\x17.google.protobuf.Struct\x12\x10\n\x08schedule\x18\x07 \x01(\t\x12P\n\x10schedule_options\x18\x18 \x01(\x0b\x32\x36.google.cloud.bigquery.datatransfer.v1.ScheduleOptions\x12 \n\x18\x64\x61ta_refresh_window_days\x18\x0c \x01(\x05\x12\x10\n\x08\x64isabled\x18\r \x01(\x08\x12/\n\x0bupdate_time\x18\x04 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12\x31\n\rnext_run_time\x18\x08 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12\x43\n\x05state\x18\n \x01(\x0e\x32\x34.google.cloud.bigquery.datatransfer.v1.TransferState\x12\x0f\n\x07user_id\x18\x0b \x01(\x03\x12\x16\n\x0e\x64\x61taset_region\x18\x0e \x01(\t"\xfe\x03\n\x0bTransferRun\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x31\n\rschedule_time\x18\x03 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12,\n\x08run_time\x18\n \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12(\n\x0c\x65rror_status\x18\x15 \x01(\x0b\x32\x12.google.rpc.Status\x12.\n\nstart_time\x18\x04 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12,\n\x08\x65nd_time\x18\x05 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12/\n\x0bupdate_time\x18\x06 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12\'\n\x06params\x18\t \x01(\x0b\x32\x17.google.protobuf.Struct\x12\x1e\n\x16\x64\x65stination_dataset_id\x18\x02 \x01(\t\x12\x16\n\x0e\x64\x61ta_source_id\x18\x07 \x01(\t\x12\x43\n\x05state\x18\x08 \x01(\x0e\x32\x34.google.cloud.bigquery.datatransfer.v1.TransferState\x12\x0f\n\x07user_id\x18\x0b \x01(\x03\x12\x10\n\x08schedule\x18\x0c \x01(\t"\x8a\x02\n\x0fTransferMessage\x12\x30\n\x0cmessage_time\x18\x01 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12X\n\x08severity\x18\x02 \x01(\x0e\x32\x46.google.cloud.bigquery.datatransfer.v1.TransferMessage.MessageSeverity\x12\x14\n\x0cmessage_text\x18\x03 \x01(\t"U\n\x0fMessageSeverity\x12 \n\x1cMESSAGE_SEVERITY_UNSPECIFIED\x10\x00\x12\x08\n\x04INFO\x10\x01\x12\x0b\n\x07WARNING\x10\x02\x12\t\n\x05\x45RROR\x10\x03*K\n\x0cTransferType\x12\x1d\n\x19TRANSFER_TYPE_UNSPECIFIED\x10\x00\x12\t\n\x05\x42\x41TCH\x10\x01\x12\r\n\tSTREAMING\x10\x02\x1a\x02\x18\x01*s\n\rTransferState\x12\x1e\n\x1aTRANSFER_STATE_UNSPECIFIED\x10\x00\x12\x0b\n\x07PENDING\x10\x02\x12\x0b\n\x07RUNNING\x10\x03\x12\r\n\tSUCCEEDED\x10\x04\x12\n\n\x06\x46\x41ILED\x10\x05\x12\r\n\tCANCELLED\x10\x06\x42\xe7\x01\n)com.google.cloud.bigquery.datatransfer.v1B\rTransferProtoP\x01ZQgoogle.golang.org/genproto/googleapis/cloud/bigquery/datatransfer/v1;datatransfer\xa2\x02\x05GCBDT\xaa\x02%Google.Cloud.BigQuery.DataTransfer.V1\xca\x02%Google\\Cloud\\BigQuery\\DataTransfer\\V1b\x06proto3'
+ ),
+ dependencies=[
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_struct__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_timestamp__pb2.DESCRIPTOR,
+ google_dot_rpc_dot_status__pb2.DESCRIPTOR,
+ ],
+)
+
+_TRANSFERTYPE = _descriptor.EnumDescriptor(
+ name="TransferType",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferType",
+ filename=None,
+ file=DESCRIPTOR,
+ values=[
+ _descriptor.EnumValueDescriptor(
+ name="TRANSFER_TYPE_UNSPECIFIED",
+ index=0,
+ number=0,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="BATCH", index=1, number=1, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="STREAMING", index=2, number=2, serialized_options=None, type=None
+ ),
+ ],
+ containing_type=None,
+ serialized_options=_b("\030\001"),
+ serialized_start=1662,
+ serialized_end=1737,
+)
+_sym_db.RegisterEnumDescriptor(_TRANSFERTYPE)
+
+TransferType = enum_type_wrapper.EnumTypeWrapper(_TRANSFERTYPE)
+_TRANSFERSTATE = _descriptor.EnumDescriptor(
+ name="TransferState",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferState",
+ filename=None,
+ file=DESCRIPTOR,
+ values=[
+ _descriptor.EnumValueDescriptor(
+ name="TRANSFER_STATE_UNSPECIFIED",
+ index=0,
+ number=0,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="PENDING", index=1, number=2, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="RUNNING", index=2, number=3, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="SUCCEEDED", index=3, number=4, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="FAILED", index=4, number=5, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="CANCELLED", index=5, number=6, serialized_options=None, type=None
+ ),
+ ],
+ containing_type=None,
+ serialized_options=None,
+ serialized_start=1739,
+ serialized_end=1854,
+)
+_sym_db.RegisterEnumDescriptor(_TRANSFERSTATE)
+
+TransferState = enum_type_wrapper.EnumTypeWrapper(_TRANSFERSTATE)
+TRANSFER_TYPE_UNSPECIFIED = 0
+BATCH = 1
+STREAMING = 2
+TRANSFER_STATE_UNSPECIFIED = 0
+PENDING = 2
+RUNNING = 3
+SUCCEEDED = 4
+FAILED = 5
+CANCELLED = 6
+
+
+_TRANSFERMESSAGE_MESSAGESEVERITY = _descriptor.EnumDescriptor(
+ name="MessageSeverity",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferMessage.MessageSeverity",
+ filename=None,
+ file=DESCRIPTOR,
+ values=[
+ _descriptor.EnumValueDescriptor(
+ name="MESSAGE_SEVERITY_UNSPECIFIED",
+ index=0,
+ number=0,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="INFO", index=1, number=1, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="WARNING", index=2, number=2, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="ERROR", index=3, number=3, serialized_options=None, type=None
+ ),
+ ],
+ containing_type=None,
+ serialized_options=None,
+ serialized_start=1575,
+ serialized_end=1660,
+)
+_sym_db.RegisterEnumDescriptor(_TRANSFERMESSAGE_MESSAGESEVERITY)
+
+
+_SCHEDULEOPTIONS = _descriptor.Descriptor(
+ name="ScheduleOptions",
+ full_name="google.cloud.bigquery.datatransfer.v1.ScheduleOptions",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="disable_auto_scheduling",
+ full_name="google.cloud.bigquery.datatransfer.v1.ScheduleOptions.disable_auto_scheduling",
+ index=0,
+ number=3,
+ type=8,
+ cpp_type=7,
+ label=1,
+ has_default_value=False,
+ default_value=False,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="start_time",
+ full_name="google.cloud.bigquery.datatransfer.v1.ScheduleOptions.start_time",
+ index=1,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="end_time",
+ full_name="google.cloud.bigquery.datatransfer.v1.ScheduleOptions.end_time",
+ index=2,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=220,
+ serialized_end=364,
+)
+
+
+_TRANSFERCONFIG = _descriptor.Descriptor(
+ name="TransferConfig",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferConfig",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferConfig.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="destination_dataset_id",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferConfig.destination_dataset_id",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="display_name",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferConfig.display_name",
+ index=2,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="data_source_id",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferConfig.data_source_id",
+ index=3,
+ number=5,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="params",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferConfig.params",
+ index=4,
+ number=9,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="schedule",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferConfig.schedule",
+ index=5,
+ number=7,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="schedule_options",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferConfig.schedule_options",
+ index=6,
+ number=24,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="data_refresh_window_days",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferConfig.data_refresh_window_days",
+ index=7,
+ number=12,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="disabled",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferConfig.disabled",
+ index=8,
+ number=13,
+ type=8,
+ cpp_type=7,
+ label=1,
+ has_default_value=False,
+ default_value=False,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="update_time",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferConfig.update_time",
+ index=9,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="next_run_time",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferConfig.next_run_time",
+ index=10,
+ number=8,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="state",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferConfig.state",
+ index=11,
+ number=10,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="user_id",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferConfig.user_id",
+ index=12,
+ number=11,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="dataset_region",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferConfig.dataset_region",
+ index=13,
+ number=14,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=367,
+ serialized_end=878,
+)
+
+
+_TRANSFERRUN = _descriptor.Descriptor(
+ name="TransferRun",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferRun",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferRun.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="schedule_time",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferRun.schedule_time",
+ index=1,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="run_time",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferRun.run_time",
+ index=2,
+ number=10,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="error_status",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferRun.error_status",
+ index=3,
+ number=21,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="start_time",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferRun.start_time",
+ index=4,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="end_time",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferRun.end_time",
+ index=5,
+ number=5,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="update_time",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferRun.update_time",
+ index=6,
+ number=6,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="params",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferRun.params",
+ index=7,
+ number=9,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="destination_dataset_id",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferRun.destination_dataset_id",
+ index=8,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="data_source_id",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferRun.data_source_id",
+ index=9,
+ number=7,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="state",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferRun.state",
+ index=10,
+ number=8,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="user_id",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferRun.user_id",
+ index=11,
+ number=11,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="schedule",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferRun.schedule",
+ index=12,
+ number=12,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=881,
+ serialized_end=1391,
+)
+
+
+_TRANSFERMESSAGE = _descriptor.Descriptor(
+ name="TransferMessage",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferMessage",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="message_time",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferMessage.message_time",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="severity",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferMessage.severity",
+ index=1,
+ number=2,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="message_text",
+ full_name="google.cloud.bigquery.datatransfer.v1.TransferMessage.message_text",
+ index=2,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[_TRANSFERMESSAGE_MESSAGESEVERITY],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1394,
+ serialized_end=1660,
+)
+
+_SCHEDULEOPTIONS.fields_by_name[
+ "start_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_SCHEDULEOPTIONS.fields_by_name[
+ "end_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_TRANSFERCONFIG.fields_by_name[
+ "params"
+].message_type = google_dot_protobuf_dot_struct__pb2._STRUCT
+_TRANSFERCONFIG.fields_by_name["schedule_options"].message_type = _SCHEDULEOPTIONS
+_TRANSFERCONFIG.fields_by_name[
+ "update_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_TRANSFERCONFIG.fields_by_name[
+ "next_run_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_TRANSFERCONFIG.fields_by_name["state"].enum_type = _TRANSFERSTATE
+_TRANSFERRUN.fields_by_name[
+ "schedule_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_TRANSFERRUN.fields_by_name[
+ "run_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_TRANSFERRUN.fields_by_name[
+ "error_status"
+].message_type = google_dot_rpc_dot_status__pb2._STATUS
+_TRANSFERRUN.fields_by_name[
+ "start_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_TRANSFERRUN.fields_by_name[
+ "end_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_TRANSFERRUN.fields_by_name[
+ "update_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_TRANSFERRUN.fields_by_name[
+ "params"
+].message_type = google_dot_protobuf_dot_struct__pb2._STRUCT
+_TRANSFERRUN.fields_by_name["state"].enum_type = _TRANSFERSTATE
+_TRANSFERMESSAGE.fields_by_name[
+ "message_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_TRANSFERMESSAGE.fields_by_name["severity"].enum_type = _TRANSFERMESSAGE_MESSAGESEVERITY
+_TRANSFERMESSAGE_MESSAGESEVERITY.containing_type = _TRANSFERMESSAGE
+DESCRIPTOR.message_types_by_name["ScheduleOptions"] = _SCHEDULEOPTIONS
+DESCRIPTOR.message_types_by_name["TransferConfig"] = _TRANSFERCONFIG
+DESCRIPTOR.message_types_by_name["TransferRun"] = _TRANSFERRUN
+DESCRIPTOR.message_types_by_name["TransferMessage"] = _TRANSFERMESSAGE
+DESCRIPTOR.enum_types_by_name["TransferType"] = _TRANSFERTYPE
+DESCRIPTOR.enum_types_by_name["TransferState"] = _TRANSFERSTATE
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+ScheduleOptions = _reflection.GeneratedProtocolMessageType(
+ "ScheduleOptions",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_SCHEDULEOPTIONS,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.transfer_pb2",
+ __doc__="""Options customizing the data transfer schedule.
+
+
+ Attributes:
+ disable_auto_scheduling:
+ If true, automatic scheduling of data transfer runs for this
+ configuration will be disabled. The runs can be started on ad-
+ hoc basis using StartManualTransferRuns API. When automatic
+ scheduling is disabled, the TransferConfig.schedule field will
+ be ignored.
+ start_time:
+ Specifies time to start scheduling transfer runs. The first
+ run will be scheduled at or after the start time according to
+ a recurrence pattern defined in the schedule string. The start
+ time can be changed at any moment. The time when a data
+ transfer can be trigerred manually is not limited by this
+ option.
+ end_time:
+ Defines time to stop scheduling transfer runs. A transfer run
+ cannot be scheduled at or after the end time. The end time can
+ be changed at any moment. The time when a data transfer can be
+ trigerred manually is not limited by this option.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.ScheduleOptions)
+ ),
+)
+_sym_db.RegisterMessage(ScheduleOptions)
+
+TransferConfig = _reflection.GeneratedProtocolMessageType(
+ "TransferConfig",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TRANSFERCONFIG,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.transfer_pb2",
+ __doc__="""Represents a data transfer configuration. A transfer configuration
+ contains all metadata needed to perform a data transfer. For example,
+ ``destination_dataset_id`` specifies where data should be stored. When a
+ new transfer configuration is created, the specified
+ ``destination_dataset_id`` is created when needed and shared with the
+ appropriate data source service account.
+
+
+ Attributes:
+ name:
+ The resource name of the transfer config. Transfer config
+ names have the form of ``projects/{project_id}/locations/{regi
+ on}/transferConfigs/{config_id}``. The name is automatically
+ generated based on the config\_id specified in
+ CreateTransferConfigRequest along with project\_id and region.
+ If config\_id is not provided, usually a uuid, even though it
+ is not guaranteed or required, will be generated for
+ config\_id.
+ destination_dataset_id:
+ The BigQuery target dataset id.
+ display_name:
+ User specified display name for the data transfer.
+ data_source_id:
+ Data source id. Cannot be changed once data transfer is
+ created.
+ params:
+ Data transfer specific parameters.
+ schedule:
+ Data transfer schedule. If the data source does not support a
+ custom schedule, this should be empty. If it is empty, the
+ default value for the data source will be used. The specified
+ times are in UTC. Examples of valid format: ``1st,3rd monday
+ of month 15:30``, ``every wed,fri of jan,jun 13:15``, and
+ ``first sunday of quarter 00:00``. See more explanation about
+ the format here: https://cloud.google.com/appengine/docs/flexi
+ ble/python/scheduling-jobs-with-cron-
+ yaml#the\_schedule\_format NOTE: the granularity should be at
+ least 8 hours, or less frequent.
+ schedule_options:
+ Options customizing the data transfer schedule.
+ data_refresh_window_days:
+ The number of days to look back to automatically refresh the
+ data. For example, if ``data_refresh_window_days = 10``, then
+ every day BigQuery reingests data for [today-10, today-1],
+ rather than ingesting data for just [today-1]. Only valid if
+ the data source supports the feature. Set the value to 0 to
+ use the default value.
+ disabled:
+ Is this config disabled. When set to true, no runs are
+ scheduled for a given transfer.
+ update_time:
+ Output only. Data transfer modification time. Ignored by
+ server on input.
+ next_run_time:
+ Output only. Next time when data transfer will run.
+ state:
+ Output only. State of the most recently updated transfer run.
+ user_id:
+ Deprecated. Unique ID of the user on whose behalf transfer is
+ done.
+ dataset_region:
+ Output only. Region in which BigQuery dataset is located.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.TransferConfig)
+ ),
+)
+_sym_db.RegisterMessage(TransferConfig)
+
+TransferRun = _reflection.GeneratedProtocolMessageType(
+ "TransferRun",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TRANSFERRUN,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.transfer_pb2",
+ __doc__="""Represents a data transfer run.
+
+
+ Attributes:
+ name:
+ The resource name of the transfer run. Transfer run names have
+ the form ``projects/{project_id}/locations/{location}/transfer
+ Configs/{config_id}/runs/{run_id}``. The name is ignored when
+ creating a transfer run.
+ schedule_time:
+ Minimum time after which a transfer run can be started.
+ run_time:
+ For batch transfer runs, specifies the date and time of the
+ data should be ingested.
+ error_status:
+ Status of the transfer run.
+ start_time:
+ Output only. Time when transfer run was started. Parameter
+ ignored by server for input requests.
+ end_time:
+ Output only. Time when transfer run ended. Parameter ignored
+ by server for input requests.
+ update_time:
+ Output only. Last time the data transfer run state was
+ updated.
+ params:
+ Output only. Data transfer specific parameters.
+ destination_dataset_id:
+ Output only. The BigQuery target dataset id.
+ data_source_id:
+ Output only. Data source id.
+ state:
+ Data transfer run state. Ignored for input requests.
+ user_id:
+ Deprecated. Unique ID of the user on whose behalf transfer is
+ done.
+ schedule:
+ Output only. Describes the schedule of this transfer run if it
+ was created as part of a regular schedule. For batch transfer
+ runs that are scheduled manually, this is empty. NOTE: the
+ system might choose to delay the schedule depending on the
+ current load, so ``schedule_time`` doesn't always match this.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.TransferRun)
+ ),
+)
+_sym_db.RegisterMessage(TransferRun)
+
+TransferMessage = _reflection.GeneratedProtocolMessageType(
+ "TransferMessage",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TRANSFERMESSAGE,
+ __module__="google.cloud.bigquery.datatransfer_v1.proto.transfer_pb2",
+ __doc__="""Represents a user facing message for a particular data transfer run.
+
+
+ Attributes:
+ message_time:
+ Time when message was logged.
+ severity:
+ Message severity.
+ message_text:
+ Message text.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.datatransfer.v1.TransferMessage)
+ ),
+)
+_sym_db.RegisterMessage(TransferMessage)
+
+
+DESCRIPTOR._options = None
+_TRANSFERTYPE._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/proto/transfer_pb2_grpc.py b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/proto/transfer_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/proto/transfer_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/types.py b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/types.py
new file mode 100644
index 000000000000..da6b3268fcc7
--- /dev/null
+++ b/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/types.py
@@ -0,0 +1,61 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from __future__ import absolute_import
+import sys
+
+from google.api_core.protobuf_helpers import get_messages
+
+from google.cloud.bigquery_datatransfer_v1.proto import datatransfer_pb2
+from google.cloud.bigquery_datatransfer_v1.proto import transfer_pb2
+from google.protobuf import any_pb2
+from google.protobuf import duration_pb2
+from google.protobuf import empty_pb2
+from google.protobuf import field_mask_pb2
+from google.protobuf import struct_pb2
+from google.protobuf import timestamp_pb2
+from google.protobuf import wrappers_pb2
+from google.rpc import status_pb2
+
+
+_shared_modules = [
+ any_pb2,
+ duration_pb2,
+ empty_pb2,
+ field_mask_pb2,
+ struct_pb2,
+ timestamp_pb2,
+ wrappers_pb2,
+ status_pb2,
+]
+
+_local_modules = [datatransfer_pb2, transfer_pb2]
+
+names = []
+
+for module in _shared_modules: # pragma: NO COVER
+ for name, message in get_messages(module).items():
+ setattr(sys.modules[__name__], name, message)
+ names.append(name)
+for module in _local_modules:
+ for name, message in get_messages(module).items():
+ message.__module__ = "google.cloud.bigquery_datatransfer_v1.types"
+ setattr(sys.modules[__name__], name, message)
+ names.append(name)
+
+
+__all__ = tuple(sorted(names))
diff --git a/bigquery_datatransfer/noxfile.py b/bigquery_datatransfer/noxfile.py
new file mode 100644
index 000000000000..a5d1ca866045
--- /dev/null
+++ b/bigquery_datatransfer/noxfile.py
@@ -0,0 +1,179 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Generated by synthtool. DO NOT EDIT!
+
+from __future__ import absolute_import
+import os
+import shutil
+
+import nox
+
+
+LOCAL_DEPS = (os.path.join("..", "api_core"), os.path.join("..", "core"))
+BLACK_VERSION = "black==19.3b0"
+BLACK_PATHS = ["docs", "google", "tests", "noxfile.py", "setup.py"]
+
+if os.path.exists("samples"):
+ BLACK_PATHS.append("samples")
+
+
+@nox.session(python="3.7")
+def lint(session):
+ """Run linters.
+
+ Returns a failure if the linters find linting errors or sufficiently
+ serious code quality issues.
+ """
+ session.install("flake8", BLACK_VERSION, *LOCAL_DEPS)
+ session.run("black", "--check", *BLACK_PATHS)
+ session.run("flake8", "google", "tests")
+
+
+@nox.session(python="3.6")
+def blacken(session):
+ """Run black.
+
+ Format code to uniform standard.
+
+ This currently uses Python 3.6 due to the automated Kokoro run of synthtool.
+ That run uses an image that doesn't have 3.6 installed. Before updating this
+ check the state of the `gcp_ubuntu_config` we use for that Kokoro run.
+ """
+ session.install(BLACK_VERSION)
+ session.run("black", *BLACK_PATHS)
+
+
+@nox.session(python="3.7")
+def lint_setup_py(session):
+ """Verify that setup.py is valid (including RST check)."""
+ session.install("docutils", "pygments")
+ session.run("python", "setup.py", "check", "--restructuredtext", "--strict")
+
+
+def default(session):
+ # Install all test dependencies, then install this package in-place.
+ session.install("mock", "pytest", "pytest-cov")
+ for local_dep in LOCAL_DEPS:
+ session.install("-e", local_dep)
+ session.install("-e", ".")
+
+ # Run py.test against the unit tests.
+ session.run(
+ "py.test",
+ "--quiet",
+ "--cov=google.cloud",
+ "--cov=tests.unit",
+ "--cov-append",
+ "--cov-config=.coveragerc",
+ "--cov-report=",
+ "--cov-fail-under=0",
+ os.path.join("tests", "unit"),
+ *session.posargs,
+ )
+
+
+@nox.session(python=["2.7", "3.5", "3.6", "3.7"])
+def unit(session):
+ """Run the unit test suite."""
+ default(session)
+
+
+@nox.session(python=["2.7", "3.7"])
+def system(session):
+ """Run the system test suite."""
+ system_test_path = os.path.join("tests", "system.py")
+ system_test_folder_path = os.path.join("tests", "system")
+ # Sanity check: Only run tests if the environment variable is set.
+ if not os.environ.get("GOOGLE_APPLICATION_CREDENTIALS", ""):
+ session.skip("Credentials must be set via environment variable")
+
+ system_test_exists = os.path.exists(system_test_path)
+ system_test_folder_exists = os.path.exists(system_test_folder_path)
+ # Sanity check: only run tests if found.
+ if not system_test_exists and not system_test_folder_exists:
+ session.skip("System tests were not found")
+
+ # Use pre-release gRPC for system tests.
+ session.install("--pre", "grpcio")
+
+ # Install all test dependencies, then install this package into the
+ # virtualenv's dist-packages.
+ session.install("mock", "pytest")
+ for local_dep in LOCAL_DEPS:
+ session.install("-e", local_dep)
+ session.install("-e", "../test_utils/")
+ session.install("-e", ".")
+
+ # Run py.test against the system tests.
+ if system_test_exists:
+ session.run("py.test", "--quiet", system_test_path, *session.posargs)
+ if system_test_folder_exists:
+ session.run("py.test", "--quiet", system_test_folder_path, *session.posargs)
+
+
+@nox.session(python=["2.7", "3.7"])
+def samples(session):
+ requirements_path = os.path.join("samples", "requirements.txt")
+ requirements_exists = os.path.exists(requirements_path)
+
+ # Sanity check: Only run tests if the environment variable is set.
+ if not os.environ.get("GOOGLE_APPLICATION_CREDENTIALS", ""):
+ session.skip("Credentials must be set via environment variable")
+
+ session.install("mock", "pytest")
+ for local_dep in LOCAL_DEPS:
+ session.install("-e", local_dep)
+ if requirements_exists:
+ session.install("-r", requirements_path)
+ session.install("-e", ".")
+
+ session.run("py.test", "--quiet", "samples", *session.posargs)
+
+
+@nox.session(python="3.7")
+def cover(session):
+ """Run the final coverage report.
+
+ This outputs the coverage report aggregating coverage from the unit
+ test runs (not system test runs), and then erases coverage data.
+ """
+ session.install("coverage", "pytest-cov")
+ session.run("coverage", "report", "--show-missing", "--fail-under=79")
+
+ session.run("coverage", "erase")
+
+
+@nox.session(python="3.7")
+def docs(session):
+ """Build the docs for this library."""
+
+ session.install("-e", ".")
+ session.install("sphinx", "alabaster", "recommonmark")
+
+ shutil.rmtree(os.path.join("docs", "_build"), ignore_errors=True)
+ session.run(
+ "sphinx-build",
+ "-W", # warnings as errors
+ "-T", # show full traceback on exception
+ "-N", # no colors
+ "-b",
+ "html",
+ "-d",
+ os.path.join("docs", "_build", "doctrees", ""),
+ os.path.join("docs", ""),
+ os.path.join("docs", "_build", "html", ""),
+ )
diff --git a/bigquery_datatransfer/samples/__init__.py b/bigquery_datatransfer/samples/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/bigquery_datatransfer/samples/create_scheduled_query.py b/bigquery_datatransfer/samples/create_scheduled_query.py
new file mode 100644
index 000000000000..b4e7437fb37d
--- /dev/null
+++ b/bigquery_datatransfer/samples/create_scheduled_query.py
@@ -0,0 +1,97 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# To install the latest published package dependency, execute the following:
+# pip install google-cloud-bigquery-datatransfer
+
+
+def sample_create_transfer_config(project_id, dataset_id, authorization_code=""):
+ # [START bigquerydatatransfer_create_scheduled_query]
+ from google.cloud import bigquery_datatransfer_v1
+ import google.protobuf.json_format
+
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+
+ # TODO(developer): Set the project_id to the project that contains the
+ # destination dataset.
+ # project_id = "your-project-id"
+
+ # TODO(developer): Set the destination dataset. The authorized user must
+ # have owner permissions on the dataset.
+ # dataset_id = "your_dataset_id"
+
+ # TODO(developer): The first time you run this sample, set the
+ # authorization code to a value from the URL:
+ # https://www.gstatic.com/bigquerydatatransfer/oauthz/auth?client_id=433065040935-hav5fqnc9p9cht3rqneus9115ias2kn1.apps.googleusercontent.com&scope=https://www.googleapis.com/auth/bigquery%20https://www.googleapis.com/auth/drive&redirect_uri=urn:ietf:wg:oauth:2.0:oob
+ #
+ # authorization_code = "_4/ABCD-EFGHIJKLMNOP-QRSTUVWXYZ"
+ #
+ # You can use an empty string for authorization_code in subsequent runs of
+ # this code sample with the same credentials.
+ #
+ # authorization_code = ""
+
+ # Use standard SQL syntax for the query.
+ query_string = """
+ SELECT
+ CURRENT_TIMESTAMP() as current_time,
+ @run_time as intended_run_time,
+ @run_date as intended_run_date,
+ 17 as some_integer
+ """
+
+ parent = client.project_path(project_id)
+
+ transfer_config = google.protobuf.json_format.ParseDict(
+ {
+ "destination_dataset_id": dataset_id,
+ "display_name": "Your Scheduled Query Name",
+ "data_source_id": "scheduled_query",
+ "params": {
+ "query": query_string,
+ "destination_table_name_template": "your_table_{run_date}",
+ "write_disposition": "WRITE_TRUNCATE",
+ "partitioning_field": "",
+ },
+ "schedule": "every 24 hours",
+ },
+ bigquery_datatransfer_v1.types.TransferConfig(),
+ )
+
+ response = client.create_transfer_config(
+ parent, transfer_config, authorization_code=authorization_code
+ )
+
+ print("Created scheduled query '{}'".format(response.name))
+ # [END bigquerydatatransfer_create_scheduled_query]
+ # Return the config name for testing purposes, so that it can be deleted.
+ return response.name
+
+
+def main():
+ import argparse
+
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--project_id", type=str, default="your-project-id")
+ parser.add_argument("--dataset_id", type=str, default="your_dataset_id")
+ parser.add_argument("--authorization_code", type=str, default="")
+ args = parser.parse_args()
+
+ sample_create_transfer_config(args.project_id, args.authorization_code)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/bigquery_datatransfer/samples/requirements.txt b/bigquery_datatransfer/samples/requirements.txt
new file mode 100644
index 000000000000..07315b55faa6
--- /dev/null
+++ b/bigquery_datatransfer/samples/requirements.txt
@@ -0,0 +1 @@
+-e ../bigquery
diff --git a/bigquery_datatransfer/samples/tests/__init__.py b/bigquery_datatransfer/samples/tests/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/bigquery_datatransfer/samples/tests/test_create_scheduled_query.py b/bigquery_datatransfer/samples/tests/test_create_scheduled_query.py
new file mode 100644
index 000000000000..a1a69c1ff0c3
--- /dev/null
+++ b/bigquery_datatransfer/samples/tests/test_create_scheduled_query.py
@@ -0,0 +1,82 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import time
+import os
+
+import google.api_core.exceptions
+import google.auth
+import google.cloud.bigquery
+import pytest
+
+from .. import create_scheduled_query
+
+
+@pytest.fixture
+def project_id():
+ return os.environ["PROJECT_ID"]
+
+
+@pytest.fixture(scope="module")
+def credentials():
+ # If using a service account, the BQ DTS robot associated with your project
+ # requires the roles/iam.serviceAccountShortTermTokenMinter permission to
+ # act on behalf of the account.
+ creds, _ = google.auth.default(["https://www.googleapis.com/auth/cloud-platform"])
+ return creds
+
+
+@pytest.fixture(scope="module")
+def bqdts_client(credentials):
+ from google.cloud import bigquery_datatransfer_v1
+
+ return bigquery_datatransfer_v1.DataTransferServiceClient(credentials=credentials)
+
+
+@pytest.fixture(scope="module")
+def bigquery_client(credentials):
+ return google.cloud.bigquery.Client(credentials=credentials)
+
+
+@pytest.fixture(scope="module")
+def dataset_id(bigquery_client):
+ # Ensure the test account has owner permissions on the dataset by creating
+ # one from scratch.
+ temp_ds_id = "bqdts_{}".format(int(time.clock() * 1000000))
+ bigquery_client.create_dataset(temp_ds_id)
+ yield temp_ds_id
+ bigquery_client.delete_dataset(temp_ds_id)
+
+
+@pytest.fixture
+def to_delete(bqdts_client):
+ doomed = []
+ yield doomed
+
+ for resource_name in doomed:
+ try:
+ bqdts_client.delete_transfer_config(resource_name)
+ except google.api_core.exceptions.NotFound:
+ pass
+
+
+def test_sample(project_id, dataset_id, capsys, to_delete):
+ config_name = create_scheduled_query.sample_create_transfer_config(
+ project_id, dataset_id
+ )
+ to_delete.append(config_name)
+ out, err = capsys.readouterr()
+ assert config_name in out
diff --git a/bigquery_datatransfer/setup.cfg b/bigquery_datatransfer/setup.cfg
new file mode 100644
index 000000000000..3bd555500e37
--- /dev/null
+++ b/bigquery_datatransfer/setup.cfg
@@ -0,0 +1,3 @@
+# Generated by synthtool. DO NOT EDIT!
+[bdist_wheel]
+universal = 1
diff --git a/bigquery_datatransfer/setup.py b/bigquery_datatransfer/setup.py
new file mode 100644
index 000000000000..49a6456283ce
--- /dev/null
+++ b/bigquery_datatransfer/setup.py
@@ -0,0 +1,86 @@
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import io
+import os
+
+import setuptools
+
+
+# Package metadata.
+
+name = "google-cloud-bigquery-datatransfer"
+description = "BigQuery Data Transfer API client library"
+version = "0.4.1"
+# Should be one of:
+# 'Development Status :: 3 - Alpha'
+# 'Development Status :: 4 - Beta'
+# 'Development Status :: 5 - Production/Stable'
+release_status = "Development Status :: 3 - Alpha"
+dependencies = ["google-api-core[grpc] >= 1.14.0, < 2.0.0dev"]
+extras = {}
+
+
+# Setup boilerplate below this line.
+
+package_root = os.path.abspath(os.path.dirname(__file__))
+
+readme_filename = os.path.join(package_root, "README.rst")
+with io.open(readme_filename, encoding="utf-8") as readme_file:
+ readme = readme_file.read()
+
+# Only include packages under the 'google' namespace. Do not include tests,
+# benchmarks, etc.
+packages = [
+ package for package in setuptools.find_packages() if package.startswith("google")
+]
+
+# Determine which namespaces are needed.
+namespaces = ["google"]
+if "google.cloud" in packages:
+ namespaces.append("google.cloud")
+
+
+setuptools.setup(
+ name=name,
+ version=version,
+ description=description,
+ long_description=readme,
+ author="Google LLC",
+ author_email="googleapis-packages@google.com",
+ license="Apache 2.0",
+ url="https://github.com/GoogleCloudPlatform/google-cloud-python",
+ classifiers=[
+ release_status,
+ "Intended Audience :: Developers",
+ "License :: OSI Approved :: Apache Software License",
+ "Programming Language :: Python",
+ "Programming Language :: Python :: 2",
+ "Programming Language :: Python :: 2.7",
+ "Programming Language :: Python :: 3",
+ "Programming Language :: Python :: 3.5",
+ "Programming Language :: Python :: 3.6",
+ "Programming Language :: Python :: 3.7",
+ "Operating System :: OS Independent",
+ "Topic :: Internet",
+ ],
+ platforms="Posix; MacOS X; Windows",
+ packages=packages,
+ namespace_packages=namespaces,
+ install_requires=dependencies,
+ extras_require=extras,
+ python_requires=">=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*",
+ include_package_data=True,
+ zip_safe=False,
+)
diff --git a/bigquery_datatransfer/synth.metadata b/bigquery_datatransfer/synth.metadata
new file mode 100644
index 000000000000..1643741ae8a9
--- /dev/null
+++ b/bigquery_datatransfer/synth.metadata
@@ -0,0 +1,39 @@
+{
+ "updateTime": "2019-08-03T12:11:16.935211Z",
+ "sources": [
+ {
+ "generator": {
+ "name": "artman",
+ "version": "0.32.1",
+ "dockerImage": "googleapis/artman@sha256:a684d40ba9a4e15946f5f2ca6b4bd9fe301192f522e9de4fff622118775f309b"
+ }
+ },
+ {
+ "git": {
+ "name": "googleapis",
+ "remote": "https://github.com/googleapis/googleapis.git",
+ "sha": "7b212a8d2319cd81a7b6942c25dbf4550480a06c",
+ "internalRef": "261339454"
+ }
+ },
+ {
+ "template": {
+ "name": "python_library",
+ "origin": "synthtool.gcp",
+ "version": "2019.5.2"
+ }
+ }
+ ],
+ "destinations": [
+ {
+ "client": {
+ "source": "googleapis",
+ "apiName": "bigquery_datatransfer",
+ "apiVersion": "v1",
+ "language": "python",
+ "generator": "gapic",
+ "config": "google/cloud/bigquery/datatransfer/artman_bigquerydatatransfer.yaml"
+ }
+ }
+ ]
+}
\ No newline at end of file
diff --git a/bigquery_datatransfer/synth.py b/bigquery_datatransfer/synth.py
new file mode 100644
index 000000000000..9a8acff4369c
--- /dev/null
+++ b/bigquery_datatransfer/synth.py
@@ -0,0 +1,68 @@
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""This script is used to synthesize generated parts of this library."""
+
+import synthtool as s
+from synthtool import gcp
+
+gapic = gcp.GAPICGenerator()
+common = gcp.CommonTemplates()
+version = "v1"
+
+# ----------------------------------------------------------------------------
+# Generate bigquery_datatransfer GAPIC layer
+# ----------------------------------------------------------------------------
+library = gapic.py_library(
+ "bigquery_datatransfer",
+ version,
+ config_path="/google/cloud/bigquery/datatransfer/"
+ "artman_bigquerydatatransfer.yaml",
+ artman_output_name="bigquerydatatransfer-v1",
+ include_protos=True,
+)
+
+s.move(
+ library,
+ excludes=["docs/conf.py", "docs/index.rst", "README.rst", "nox.py", "setup.py"],
+)
+
+s.replace(
+ [
+ "google/cloud/bigquery_datatransfer_v1/proto/datatransfer_pb2.py",
+ "google/cloud/bigquery_datatransfer_v1/proto/datatransfer_pb2_grpc.py",
+ ],
+ "from google.cloud.bigquery.datatransfer_v1.proto",
+ "from google.cloud.bigquery_datatransfer_v1.proto",
+)
+
+s.replace(
+ "google/cloud/bigquery_datatransfer_v1/gapic/" "data_transfer_service_client.py",
+ "google-cloud-bigquerydatatransfer",
+ "google-cloud-bigquery-datatransfer",
+)
+
+s.replace(
+ "google/cloud/bigquery_datatransfer_v1/gapic/" "data_transfer_service_client.py",
+ "import google.api_core.gapic_v1.method\n",
+ "\g<0>import google.api_core.path_template\n",
+)
+
+# ----------------------------------------------------------------------------
+# Add templated files
+# ----------------------------------------------------------------------------
+templated_files = common.py_library(unit_cov_level=79, cov_level=79, samples_test=True)
+s.move(templated_files)
+
+s.shell.run(["nox", "-s", "blacken"], hide_output=False)
diff --git a/bigquery_datatransfer/tests/system/gapic/v1/test_system_data_transfer_service_v1.py b/bigquery_datatransfer/tests/system/gapic/v1/test_system_data_transfer_service_v1.py
new file mode 100644
index 000000000000..28f975d9f846
--- /dev/null
+++ b/bigquery_datatransfer/tests/system/gapic/v1/test_system_data_transfer_service_v1.py
@@ -0,0 +1,30 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import time
+
+from google.cloud import bigquery_datatransfer_v1
+from google.cloud.bigquery_datatransfer_v1.proto import datatransfer_pb2
+
+
+class TestSystemDataTransferService(object):
+ def test_list_data_sources(self):
+ project_id = os.environ["PROJECT_ID"]
+
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+ parent = client.location_path(project_id, "us-central1")
+ response = client.list_data_sources(parent)
diff --git a/bigquery_datatransfer/tests/unit/gapic/v1/test_data_transfer_service_client_v1.py b/bigquery_datatransfer/tests/unit/gapic/v1/test_data_transfer_service_client_v1.py
new file mode 100644
index 000000000000..7878245ee30f
--- /dev/null
+++ b/bigquery_datatransfer/tests/unit/gapic/v1/test_data_transfer_service_client_v1.py
@@ -0,0 +1,723 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Unit tests."""
+
+import mock
+import pytest
+
+from google.cloud import bigquery_datatransfer_v1
+from google.cloud.bigquery_datatransfer_v1.proto import datatransfer_pb2
+from google.cloud.bigquery_datatransfer_v1.proto import transfer_pb2
+from google.protobuf import empty_pb2
+from google.protobuf import field_mask_pb2
+from google.protobuf import timestamp_pb2
+
+
+class MultiCallableStub(object):
+ """Stub for the grpc.UnaryUnaryMultiCallable interface."""
+
+ def __init__(self, method, channel_stub):
+ self.method = method
+ self.channel_stub = channel_stub
+
+ def __call__(self, request, timeout=None, metadata=None, credentials=None):
+ self.channel_stub.requests.append((self.method, request))
+
+ response = None
+ if self.channel_stub.responses:
+ response = self.channel_stub.responses.pop()
+
+ if isinstance(response, Exception):
+ raise response
+
+ if response:
+ return response
+
+
+class ChannelStub(object):
+ """Stub for the grpc.Channel interface."""
+
+ def __init__(self, responses=[]):
+ self.responses = responses
+ self.requests = []
+
+ def unary_unary(self, method, request_serializer=None, response_deserializer=None):
+ return MultiCallableStub(method, self)
+
+
+class CustomException(Exception):
+ pass
+
+
+class TestDataTransferServiceClient(object):
+ def test_get_data_source(self):
+ # Setup Expected Response
+ name_2 = "name2-1052831874"
+ data_source_id = "dataSourceId-1015796374"
+ display_name = "displayName1615086568"
+ description = "description-1724546052"
+ client_id = "clientId-1904089585"
+ supports_multiple_transfers = True
+ update_deadline_seconds = 991471694
+ default_schedule = "defaultSchedule-800168235"
+ supports_custom_schedule = True
+ help_url = "helpUrl-789431439"
+ default_data_refresh_window_days = 1804935157
+ manual_runs_disabled = True
+ expected_response = {
+ "name": name_2,
+ "data_source_id": data_source_id,
+ "display_name": display_name,
+ "description": description,
+ "client_id": client_id,
+ "supports_multiple_transfers": supports_multiple_transfers,
+ "update_deadline_seconds": update_deadline_seconds,
+ "default_schedule": default_schedule,
+ "supports_custom_schedule": supports_custom_schedule,
+ "help_url": help_url,
+ "default_data_refresh_window_days": default_data_refresh_window_days,
+ "manual_runs_disabled": manual_runs_disabled,
+ }
+ expected_response = datatransfer_pb2.DataSource(**expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+
+ # Setup Request
+ name = client.project_data_source_path("[PROJECT]", "[DATA_SOURCE]")
+
+ response = client.get_data_source(name)
+ assert expected_response == response
+
+ assert len(channel.requests) == 1
+ expected_request = datatransfer_pb2.GetDataSourceRequest(name=name)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_get_data_source_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+
+ # Setup request
+ name = client.project_data_source_path("[PROJECT]", "[DATA_SOURCE]")
+
+ with pytest.raises(CustomException):
+ client.get_data_source(name)
+
+ def test_list_data_sources(self):
+ # Setup Expected Response
+ next_page_token = ""
+ data_sources_element = {}
+ data_sources = [data_sources_element]
+ expected_response = {
+ "next_page_token": next_page_token,
+ "data_sources": data_sources,
+ }
+ expected_response = datatransfer_pb2.ListDataSourcesResponse(
+ **expected_response
+ )
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+
+ # Setup Request
+ parent = client.project_path("[PROJECT]")
+
+ paged_list_response = client.list_data_sources(parent)
+ resources = list(paged_list_response)
+ assert len(resources) == 1
+
+ assert expected_response.data_sources[0] == resources[0]
+
+ assert len(channel.requests) == 1
+ expected_request = datatransfer_pb2.ListDataSourcesRequest(parent=parent)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_list_data_sources_exception(self):
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+
+ # Setup request
+ parent = client.project_path("[PROJECT]")
+
+ paged_list_response = client.list_data_sources(parent)
+ with pytest.raises(CustomException):
+ list(paged_list_response)
+
+ def test_create_transfer_config(self):
+ # Setup Expected Response
+ name = "name3373707"
+ destination_dataset_id = "destinationDatasetId1541564179"
+ display_name = "displayName1615086568"
+ data_source_id = "dataSourceId-1015796374"
+ schedule = "schedule-697920873"
+ data_refresh_window_days = 327632845
+ disabled = True
+ user_id = 147132913
+ dataset_region = "datasetRegion959248539"
+ expected_response = {
+ "name": name,
+ "destination_dataset_id": destination_dataset_id,
+ "display_name": display_name,
+ "data_source_id": data_source_id,
+ "schedule": schedule,
+ "data_refresh_window_days": data_refresh_window_days,
+ "disabled": disabled,
+ "user_id": user_id,
+ "dataset_region": dataset_region,
+ }
+ expected_response = transfer_pb2.TransferConfig(**expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+
+ # Setup Request
+ parent = client.project_path("[PROJECT]")
+ transfer_config = {}
+
+ response = client.create_transfer_config(parent, transfer_config)
+ assert expected_response == response
+
+ assert len(channel.requests) == 1
+ expected_request = datatransfer_pb2.CreateTransferConfigRequest(
+ parent=parent, transfer_config=transfer_config
+ )
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_create_transfer_config_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+
+ # Setup request
+ parent = client.project_path("[PROJECT]")
+ transfer_config = {}
+
+ with pytest.raises(CustomException):
+ client.create_transfer_config(parent, transfer_config)
+
+ def test_update_transfer_config(self):
+ # Setup Expected Response
+ name = "name3373707"
+ destination_dataset_id = "destinationDatasetId1541564179"
+ display_name = "displayName1615086568"
+ data_source_id = "dataSourceId-1015796374"
+ schedule = "schedule-697920873"
+ data_refresh_window_days = 327632845
+ disabled = True
+ user_id = 147132913
+ dataset_region = "datasetRegion959248539"
+ expected_response = {
+ "name": name,
+ "destination_dataset_id": destination_dataset_id,
+ "display_name": display_name,
+ "data_source_id": data_source_id,
+ "schedule": schedule,
+ "data_refresh_window_days": data_refresh_window_days,
+ "disabled": disabled,
+ "user_id": user_id,
+ "dataset_region": dataset_region,
+ }
+ expected_response = transfer_pb2.TransferConfig(**expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+
+ # Setup Request
+ transfer_config = {}
+ update_mask = {}
+
+ response = client.update_transfer_config(transfer_config, update_mask)
+ assert expected_response == response
+
+ assert len(channel.requests) == 1
+ expected_request = datatransfer_pb2.UpdateTransferConfigRequest(
+ transfer_config=transfer_config, update_mask=update_mask
+ )
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_update_transfer_config_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+
+ # Setup request
+ transfer_config = {}
+ update_mask = {}
+
+ with pytest.raises(CustomException):
+ client.update_transfer_config(transfer_config, update_mask)
+
+ def test_delete_transfer_config(self):
+ channel = ChannelStub()
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+
+ # Setup Request
+ name = client.project_transfer_config_path("[PROJECT]", "[TRANSFER_CONFIG]")
+
+ client.delete_transfer_config(name)
+
+ assert len(channel.requests) == 1
+ expected_request = datatransfer_pb2.DeleteTransferConfigRequest(name=name)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_delete_transfer_config_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+
+ # Setup request
+ name = client.project_transfer_config_path("[PROJECT]", "[TRANSFER_CONFIG]")
+
+ with pytest.raises(CustomException):
+ client.delete_transfer_config(name)
+
+ def test_get_transfer_config(self):
+ # Setup Expected Response
+ name_2 = "name2-1052831874"
+ destination_dataset_id = "destinationDatasetId1541564179"
+ display_name = "displayName1615086568"
+ data_source_id = "dataSourceId-1015796374"
+ schedule = "schedule-697920873"
+ data_refresh_window_days = 327632845
+ disabled = True
+ user_id = 147132913
+ dataset_region = "datasetRegion959248539"
+ expected_response = {
+ "name": name_2,
+ "destination_dataset_id": destination_dataset_id,
+ "display_name": display_name,
+ "data_source_id": data_source_id,
+ "schedule": schedule,
+ "data_refresh_window_days": data_refresh_window_days,
+ "disabled": disabled,
+ "user_id": user_id,
+ "dataset_region": dataset_region,
+ }
+ expected_response = transfer_pb2.TransferConfig(**expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+
+ # Setup Request
+ name = client.project_transfer_config_path("[PROJECT]", "[TRANSFER_CONFIG]")
+
+ response = client.get_transfer_config(name)
+ assert expected_response == response
+
+ assert len(channel.requests) == 1
+ expected_request = datatransfer_pb2.GetTransferConfigRequest(name=name)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_get_transfer_config_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+
+ # Setup request
+ name = client.project_transfer_config_path("[PROJECT]", "[TRANSFER_CONFIG]")
+
+ with pytest.raises(CustomException):
+ client.get_transfer_config(name)
+
+ def test_list_transfer_configs(self):
+ # Setup Expected Response
+ next_page_token = ""
+ transfer_configs_element = {}
+ transfer_configs = [transfer_configs_element]
+ expected_response = {
+ "next_page_token": next_page_token,
+ "transfer_configs": transfer_configs,
+ }
+ expected_response = datatransfer_pb2.ListTransferConfigsResponse(
+ **expected_response
+ )
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+
+ # Setup Request
+ parent = client.project_path("[PROJECT]")
+
+ paged_list_response = client.list_transfer_configs(parent)
+ resources = list(paged_list_response)
+ assert len(resources) == 1
+
+ assert expected_response.transfer_configs[0] == resources[0]
+
+ assert len(channel.requests) == 1
+ expected_request = datatransfer_pb2.ListTransferConfigsRequest(parent=parent)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_list_transfer_configs_exception(self):
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+
+ # Setup request
+ parent = client.project_path("[PROJECT]")
+
+ paged_list_response = client.list_transfer_configs(parent)
+ with pytest.raises(CustomException):
+ list(paged_list_response)
+
+ def test_schedule_transfer_runs(self):
+ # Setup Expected Response
+ expected_response = {}
+ expected_response = datatransfer_pb2.ScheduleTransferRunsResponse(
+ **expected_response
+ )
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+
+ # Setup Request
+ parent = client.project_transfer_config_path("[PROJECT]", "[TRANSFER_CONFIG]")
+ start_time = {}
+ end_time = {}
+
+ response = client.schedule_transfer_runs(parent, start_time, end_time)
+ assert expected_response == response
+
+ assert len(channel.requests) == 1
+ expected_request = datatransfer_pb2.ScheduleTransferRunsRequest(
+ parent=parent, start_time=start_time, end_time=end_time
+ )
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_schedule_transfer_runs_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+
+ # Setup request
+ parent = client.project_transfer_config_path("[PROJECT]", "[TRANSFER_CONFIG]")
+ start_time = {}
+ end_time = {}
+
+ with pytest.raises(CustomException):
+ client.schedule_transfer_runs(parent, start_time, end_time)
+
+ def test_get_transfer_run(self):
+ # Setup Expected Response
+ name_2 = "name2-1052831874"
+ destination_dataset_id = "destinationDatasetId1541564179"
+ data_source_id = "dataSourceId-1015796374"
+ user_id = 147132913
+ schedule = "schedule-697920873"
+ expected_response = {
+ "name": name_2,
+ "destination_dataset_id": destination_dataset_id,
+ "data_source_id": data_source_id,
+ "user_id": user_id,
+ "schedule": schedule,
+ }
+ expected_response = transfer_pb2.TransferRun(**expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+
+ # Setup Request
+ name = client.project_run_path("[PROJECT]", "[TRANSFER_CONFIG]", "[RUN]")
+
+ response = client.get_transfer_run(name)
+ assert expected_response == response
+
+ assert len(channel.requests) == 1
+ expected_request = datatransfer_pb2.GetTransferRunRequest(name=name)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_get_transfer_run_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+
+ # Setup request
+ name = client.project_run_path("[PROJECT]", "[TRANSFER_CONFIG]", "[RUN]")
+
+ with pytest.raises(CustomException):
+ client.get_transfer_run(name)
+
+ def test_delete_transfer_run(self):
+ channel = ChannelStub()
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+
+ # Setup Request
+ name = client.project_run_path("[PROJECT]", "[TRANSFER_CONFIG]", "[RUN]")
+
+ client.delete_transfer_run(name)
+
+ assert len(channel.requests) == 1
+ expected_request = datatransfer_pb2.DeleteTransferRunRequest(name=name)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_delete_transfer_run_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+
+ # Setup request
+ name = client.project_run_path("[PROJECT]", "[TRANSFER_CONFIG]", "[RUN]")
+
+ with pytest.raises(CustomException):
+ client.delete_transfer_run(name)
+
+ def test_list_transfer_runs(self):
+ # Setup Expected Response
+ next_page_token = ""
+ transfer_runs_element = {}
+ transfer_runs = [transfer_runs_element]
+ expected_response = {
+ "next_page_token": next_page_token,
+ "transfer_runs": transfer_runs,
+ }
+ expected_response = datatransfer_pb2.ListTransferRunsResponse(
+ **expected_response
+ )
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+
+ # Setup Request
+ parent = client.project_transfer_config_path("[PROJECT]", "[TRANSFER_CONFIG]")
+
+ paged_list_response = client.list_transfer_runs(parent)
+ resources = list(paged_list_response)
+ assert len(resources) == 1
+
+ assert expected_response.transfer_runs[0] == resources[0]
+
+ assert len(channel.requests) == 1
+ expected_request = datatransfer_pb2.ListTransferRunsRequest(parent=parent)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_list_transfer_runs_exception(self):
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+
+ # Setup request
+ parent = client.project_transfer_config_path("[PROJECT]", "[TRANSFER_CONFIG]")
+
+ paged_list_response = client.list_transfer_runs(parent)
+ with pytest.raises(CustomException):
+ list(paged_list_response)
+
+ def test_list_transfer_logs(self):
+ # Setup Expected Response
+ next_page_token = ""
+ transfer_messages_element = {}
+ transfer_messages = [transfer_messages_element]
+ expected_response = {
+ "next_page_token": next_page_token,
+ "transfer_messages": transfer_messages,
+ }
+ expected_response = datatransfer_pb2.ListTransferLogsResponse(
+ **expected_response
+ )
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+
+ # Setup Request
+ parent = client.project_run_path("[PROJECT]", "[TRANSFER_CONFIG]", "[RUN]")
+
+ paged_list_response = client.list_transfer_logs(parent)
+ resources = list(paged_list_response)
+ assert len(resources) == 1
+
+ assert expected_response.transfer_messages[0] == resources[0]
+
+ assert len(channel.requests) == 1
+ expected_request = datatransfer_pb2.ListTransferLogsRequest(parent=parent)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_list_transfer_logs_exception(self):
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+
+ # Setup request
+ parent = client.project_run_path("[PROJECT]", "[TRANSFER_CONFIG]", "[RUN]")
+
+ paged_list_response = client.list_transfer_logs(parent)
+ with pytest.raises(CustomException):
+ list(paged_list_response)
+
+ def test_check_valid_creds(self):
+ # Setup Expected Response
+ has_valid_creds = False
+ expected_response = {"has_valid_creds": has_valid_creds}
+ expected_response = datatransfer_pb2.CheckValidCredsResponse(
+ **expected_response
+ )
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+
+ # Setup Request
+ name = client.project_data_source_path("[PROJECT]", "[DATA_SOURCE]")
+
+ response = client.check_valid_creds(name)
+ assert expected_response == response
+
+ assert len(channel.requests) == 1
+ expected_request = datatransfer_pb2.CheckValidCredsRequest(name=name)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_check_valid_creds_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+
+ # Setup request
+ name = client.project_data_source_path("[PROJECT]", "[DATA_SOURCE]")
+
+ with pytest.raises(CustomException):
+ client.check_valid_creds(name)
+
+ def test_start_manual_transfer_runs(self):
+ # Setup Expected Response
+ expected_response = {}
+ expected_response = datatransfer_pb2.StartManualTransferRunsResponse(
+ **expected_response
+ )
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+
+ response = client.start_manual_transfer_runs()
+ assert expected_response == response
+
+ assert len(channel.requests) == 1
+ expected_request = datatransfer_pb2.StartManualTransferRunsRequest()
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_start_manual_transfer_runs_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = bigquery_datatransfer_v1.DataTransferServiceClient()
+
+ with pytest.raises(CustomException):
+ client.start_manual_transfer_runs()
diff --git a/bigquery_datatransfer/tests/unit/test_shim.py b/bigquery_datatransfer/tests/unit/test_shim.py
new file mode 100644
index 000000000000..c27963bce4ab
--- /dev/null
+++ b/bigquery_datatransfer/tests/unit/test_shim.py
@@ -0,0 +1,29 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Unit tests for versionless import."""
+
+
+def test_shim():
+ from google.cloud import bigquery_datatransfer
+ from google.cloud import bigquery_datatransfer_v1
+
+ assert bigquery_datatransfer.__all__ == bigquery_datatransfer_v1.__all__
+
+ for name in bigquery_datatransfer.__all__:
+ found = getattr(bigquery_datatransfer, name)
+ expected = getattr(bigquery_datatransfer_v1, name)
+ assert found is expected
diff --git a/bigquery_storage/.coveragerc b/bigquery_storage/.coveragerc
new file mode 100644
index 000000000000..b178b094aa1d
--- /dev/null
+++ b/bigquery_storage/.coveragerc
@@ -0,0 +1,19 @@
+# Generated by synthtool. DO NOT EDIT!
+[run]
+branch = True
+
+[report]
+fail_under = 100
+show_missing = True
+exclude_lines =
+ # Re-enable the standard pragma
+ pragma: NO COVER
+ # Ignore debug-only repr
+ def __repr__
+ # Ignore abstract methods
+ raise NotImplementedError
+omit =
+ */gapic/*.py
+ */proto/*.py
+ */core/*.py
+ */site-packages/*.py
\ No newline at end of file
diff --git a/bigquery_storage/.flake8 b/bigquery_storage/.flake8
new file mode 100644
index 000000000000..0268ecc9c55c
--- /dev/null
+++ b/bigquery_storage/.flake8
@@ -0,0 +1,14 @@
+# Generated by synthtool. DO NOT EDIT!
+[flake8]
+ignore = E203, E266, E501, W503
+exclude =
+ # Exclude generated code.
+ **/proto/**
+ **/gapic/**
+ *_pb2.py
+
+ # Standard linting exemptions.
+ __pycache__,
+ .git,
+ *.pyc,
+ conf.py
diff --git a/bigquery_storage/.gitignore b/bigquery_storage/.gitignore
new file mode 100644
index 000000000000..9e3a5f25770c
--- /dev/null
+++ b/bigquery_storage/.gitignore
@@ -0,0 +1 @@
+docs/_build
\ No newline at end of file
diff --git a/bigquery_storage/.repo-metadata.json b/bigquery_storage/.repo-metadata.json
new file mode 100644
index 000000000000..a7f9c207bb7c
--- /dev/null
+++ b/bigquery_storage/.repo-metadata.json
@@ -0,0 +1,13 @@
+{
+ "name": "bigquerystorage",
+ "name_pretty": "Google BigQuery Storage",
+ "product_documentation": "https://cloud.google.com/bigquery/docs/reference/storage/",
+ "client_documentation": "https://googleapis.dev/python/bigquerystorage/latest",
+ "issue_tracker": "https://issuetracker.google.com/savedsearches/559654",
+ "release_level": "beta",
+ "language": "python",
+ "repo": "googleapis/google-cloud-python",
+ "distribution_name": "google-cloud-bigquery-storage",
+ "api_id": "bigquerystorage.googleapis.com",
+ "requires_billing": true
+}
\ No newline at end of file
diff --git a/bigquery_storage/CHANGELOG.md b/bigquery_storage/CHANGELOG.md
new file mode 100644
index 000000000000..f2fb6237e2b6
--- /dev/null
+++ b/bigquery_storage/CHANGELOG.md
@@ -0,0 +1,140 @@
+# Changelog
+
+[PyPI History][1]
+
+[1]: https://pypi.org/project/google-cloud-bigquery-storage/#history
+
+## 0.7.0
+
+07-31-2019 17:48 PDT
+
+
+### New Features
+- Support faster Arrow data format in `to_dataframe` and `to_arrow` when using BigQuery Storage API. ([#8551](https://github.com/googleapis/google-cloud-python/pull/8551))
+
+### Dependencies
+- Bump minimum version for google-api-core to 1.14.0. ([#8709](https://github.com/googleapis/google-cloud-python/pull/8709))
+- Update pins of 'googleapis-common-protos. ([#8688](https://github.com/googleapis/google-cloud-python/pull/8688))
+
+### Documentation
+- Update quickstart sample with data format and sharding options. ([#8665](https://github.com/googleapis/google-cloud-python/pull/8665))
+- Fix links to bigquery storage documentation. ([#8859](https://github.com/googleapis/google-cloud-python/pull/8859))
+- Link to googleapis.dev documentation in READMEs. ([#8705](https://github.com/googleapis/google-cloud-python/pull/8705))
+- Add compatibility check badges to READMEs. ([#8288](https://github.com/googleapis/google-cloud-python/pull/8288))
+
+### Internal / Testing Changes
+- Pin black version. (via synth). ([#8672](https://github.com/googleapis/google-cloud-python/pull/8672))
+
+## 0.6.0
+
+07-11-2019 13:15 PDT
+
+### New Features
+
+- Add `to_arrow` with support for Arrow data format. ([#8644](https://github.com/googleapis/google-cloud-python/pull/8644))
+- Add 'client_options' support (via synth). ([#8536](https://github.com/googleapis/google-cloud-python/pull/8536))
+- Add sharding strategy, stream splitting, Arrow support (via synth). ([#8477](https://github.com/googleapis/google-cloud-python/pull/8477))
+
+### Documentation
+
+- Add docs job to publish to googleapis.dev. ([#8464](https://github.com/googleapis/google-cloud-python/pull/8464))
+
+### Internal / Testing Changes
+
+- Allow kwargs to be passed to create_channel (via synth). ([#8441](https://github.com/googleapis/google-cloud-python/pull/8441))
+- Add encoding declaration to protoc-generated files (via synth). ([#8345](https://github.com/googleapis/google-cloud-python/pull/8345))
+- Refactor `reader.ReadRowsPage` to use `_StreamParser`. ([#8262](https://github.com/googleapis/google-cloud-python/pull/8262))
+- Fix coverage in 'types.py' (via synth). ([#8148](https://github.com/googleapis/google-cloud-python/pull/8148))
+- Add empty lines, remove coverage exclusions (via synth). ([#8051](https://github.com/googleapis/google-cloud-python/pull/8051))
+
+## 0.5.0
+
+05-20-2019 09:23 PDT
+
+### Implementation Changes
+
+- Increase default deadline on ReadRows. ([#8030](https://github.com/googleapis/google-cloud-python/pull/8030))
+- Respect timeout on `client.read_rows`. Don't resume on `DEADLINE_EXCEEDED` errors. ([#8025](https://github.com/googleapis/google-cloud-python/pull/8025))
+
+### Documentation
+
+- Use alabaster theme everwhere. ([#8021](https://github.com/googleapis/google-cloud-python/pull/8021))
+
+## 0.4.0
+
+04-16-2019 13:46 PDT
+
+### Implementation Changes
+
+- Remove gRPC size limit in the transport options ([#7664](https://github.com/googleapis/google-cloud-python/pull/7664))
+- Add retry params for create_read_session (via synth). ([#7658](https://github.com/googleapis/google-cloud-python/pull/7658))
+
+### New Features
+
+- Add page iterator to ReadRowsStream ([#7680](https://github.com/googleapis/google-cloud-python/pull/7680))
+
+### Internal / Testing Changes
+
+- Remove system test for split rows ([#7673](https://github.com/googleapis/google-cloud-python/pull/7673))
+
+## 0.3.0
+
+04-02-2019 15:22 PDT
+
+### Dependencies
+
+- Add dependency for resource proto. ([#7585](https://github.com/googleapis/google-cloud-python/pull/7585))
+- Remove classifier for Python 3.4 for end-of-life. ([#7535](https://github.com/googleapis/google-cloud-python/pull/7535))
+
+### Documentation
+
+- Fix links to BigQuery Storage API docs ([#7647](https://github.com/googleapis/google-cloud-python/pull/7647))
+- Update proto / docstrings (via synth). ([#7461](https://github.com/googleapis/google-cloud-python/pull/7461))
+- googlecloudplatform --> googleapis in READMEs ([#7411](https://github.com/googleapis/google-cloud-python/pull/7411))
+- Updated client library documentation URLs. ([#7307](https://github.com/googleapis/google-cloud-python/pull/7307))
+- Blacken new quickstart snippet. ([#7242](https://github.com/googleapis/google-cloud-python/pull/7242))
+- Add quickstart demonstrating most BQ Storage API read features ([#7223](https://github.com/googleapis/google-cloud-python/pull/7223))
+- Add bigquery_storage to docs ([#7222](https://github.com/googleapis/google-cloud-python/pull/7222))
+
+### Internal / Testing Changes
+
+- Add protos as an artifact to library ([#7205](https://github.com/googleapis/google-cloud-python/pull/7205))
+- Copy lintified proto files (via synth). ([#7475](https://github.com/googleapis/google-cloud-python/pull/7475))
+- Add annotations to protocol buffers indicating request parameters (via synth). ([#7550](https://github.com/googleapis/google-cloud-python/pull/7550))
+
+## 0.2.0
+
+01-25-2019 13:54 PST
+
+### New Features
+
+- Add option to choose dtypes by column in to_dataframe. ([#7126](https://github.com/googleapis/google-cloud-python/pull/7126))
+
+### Internal / Testing Changes
+
+- Update copyright headers
+- Protoc-generated serialization update. ([#7076](https://github.com/googleapis/google-cloud-python/pull/7076))
+- BigQuery Storage: run 'blacken' during synth ([#7047](https://github.com/googleapis/google-cloud-python/pull/7047))
+
+## 0.1.1
+
+12-17-2018 18:03 PST
+
+
+### Implementation Changes
+- Import `iam.policy` from `google.api_core`. ([#6741](https://github.com/googleapis/google-cloud-python/pull/6741))
+- Pick up fixes in GAPIC generator. ([#6708](https://github.com/googleapis/google-cloud-python/pull/6708))
+
+### Documentation
+- Document Python 2 deprecation ([#6910](https://github.com/googleapis/google-cloud-python/pull/6910))
+
+### Internal / Testing Changes
+- Blacken all gen'd libs ([#6792](https://github.com/googleapis/google-cloud-python/pull/6792))
+- Correct release_status for bigquery_storage ([#6767](https://github.com/googleapis/google-cloud-python/pull/6767))
+
+## 0.1.0
+
+11-29-2018 13:45 PST
+
+- Initial release of BigQuery Storage API client.
+
diff --git a/bigquery_storage/LICENSE b/bigquery_storage/LICENSE
new file mode 100644
index 000000000000..a8ee855de2aa
--- /dev/null
+++ b/bigquery_storage/LICENSE
@@ -0,0 +1,201 @@
+ Apache License
+ Version 2.0, January 2004
+ https://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [yyyy] [name of copyright owner]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
diff --git a/bigquery_storage/MANIFEST.in b/bigquery_storage/MANIFEST.in
new file mode 100644
index 000000000000..9cbf175afe6b
--- /dev/null
+++ b/bigquery_storage/MANIFEST.in
@@ -0,0 +1,5 @@
+include README.rst LICENSE
+recursive-include google *.json *.proto
+recursive-include tests *
+global-exclude *.py[co]
+global-exclude __pycache__
diff --git a/bigquery_storage/README.rst b/bigquery_storage/README.rst
new file mode 100644
index 000000000000..6b600ca0bd67
--- /dev/null
+++ b/bigquery_storage/README.rst
@@ -0,0 +1,102 @@
+Python Client for BigQuery Storage API (`Beta`_)
+=================================================
+
+
+
+`BigQuery Storage API`_:
+
+- `Client Library Documentation`_
+- `Product Documentation`_
+
+.. _Beta: https://github.com/googleapis/google-cloud-python/blob/master/README.rst
+.. _BigQuery Storage API: https://cloud.google.com/bigquery/docs/reference/storage/
+.. _Client Library Documentation: https://googleapis.dev/python/bigquerystorage/latest
+.. _Product Documentation: https://cloud.google.com/bigquery/docs/reference/storage/
+
+Quick Start
+-----------
+
+In order to use this library, you first need to go through the following steps:
+
+1. `Select or create a Cloud Platform project.`_
+2. `Enable billing for your project.`_
+3. `Enable the BigQuery Storage API.`_
+4. `Setup Authentication.`_
+
+.. _Select or create a Cloud Platform project.: https://console.cloud.google.com/project
+.. _Enable billing for your project.: https://cloud.google.com/billing/docs/how-to/modify-project#enable_billing_for_a_project
+.. _Enable the BigQuery Storage API.: https://console.cloud.google.com/apis/library/bigquerystorage.googleapis.com
+.. _Setup Authentication.: https://googleapis.dev/python/google-api-core/latest/auth.html
+
+Installation
+~~~~~~~~~~~~
+
+Install this library in a `virtualenv`_ using pip. `virtualenv`_ is a tool to
+create isolated Python environments. The basic problem it addresses is one of
+dependencies and versions, and indirectly permissions.
+
+With `virtualenv`_, it's possible to install this library without needing system
+install permissions, and without clashing with the installed system
+dependencies.
+
+.. _`virtualenv`: https://virtualenv.pypa.io/en/latest/
+
+
+Supported Python Versions
+^^^^^^^^^^^^^^^^^^^^^^^^^
+Python >= 3.5
+
+Deprecated Python Versions
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+Python == 2.7. Python 2.7 support will be removed on January 1, 2020.
+
+
+Mac/Linux
+^^^^^^^^^
+
+.. code-block:: console
+
+ pip install virtualenv
+ virtualenv
+ source /bin/activate
+ /bin/pip install google-cloud-bigquery-storage
+
+
+Windows
+^^^^^^^
+
+.. code-block:: console
+
+ pip install virtualenv
+ virtualenv
+ \Scripts\activate
+ \Scripts\pip.exe install google-cloud-bigquery-storage
+
+Optional Dependencies
+^^^^^^^^^^^^^^^^^^^^^
+
+Several features of ``google-cloud-bigquery-storage`` require additional
+dependencies.
+
+* Parse Avro blocks in a ``read_rows()`` stream using `fastavro
+ `_.
+
+ ``pip install google-cloud-bigquery-storage[fastavro]``
+
+* Write rows to a `pandas `_
+ dataframe.
+
+ ``pip install google-cloud-bigquery-storage[pandas,fastavro]``
+
+Next Steps
+~~~~~~~~~~
+
+- Read the `Client Library Documentation`_ for BigQuery Storage API
+ API to see other available methods on the client.
+- Read the `BigQuery Storage API Product documentation`_ to learn
+ more about the product and see How-to Guides.
+- View this `repository’s main README`_ to see the full list of Cloud
+ APIs that we cover.
+
+.. _BigQuery Storage API Product documentation: https://cloud.google.com/bigquery/docs/reference/storage/
+.. _repository’s main README: https://github.com/googleapis/google-cloud-python/blob/master/README.rst
diff --git a/bigquery_storage/docs/README.rst b/bigquery_storage/docs/README.rst
new file mode 120000
index 000000000000..89a0106941ff
--- /dev/null
+++ b/bigquery_storage/docs/README.rst
@@ -0,0 +1 @@
+../README.rst
\ No newline at end of file
diff --git a/bigquery_storage/docs/conf.py b/bigquery_storage/docs/conf.py
new file mode 100644
index 000000000000..9b93ad89594f
--- /dev/null
+++ b/bigquery_storage/docs/conf.py
@@ -0,0 +1,327 @@
+# -*- coding: utf-8 -*-
+#
+# google-cloud-bigquerystorage documentation build configuration file
+#
+# This file is execfile()d with the current directory set to its
+# containing dir.
+#
+# Note that not all possible configuration values are present in this
+# autogenerated file.
+#
+# All configuration values have a default; values that are commented out
+# serve to show the default.
+
+import sys
+import os
+import shlex
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+sys.path.insert(0, os.path.abspath(".."))
+
+__version__ = "0.1.0"
+
+# -- General configuration ------------------------------------------------
+
+# If your documentation needs a minimal Sphinx version, state it here.
+# needs_sphinx = '1.0'
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+ "sphinx.ext.autodoc",
+ "sphinx.ext.autosummary",
+ "sphinx.ext.intersphinx",
+ "sphinx.ext.coverage",
+ "sphinx.ext.napoleon",
+ "sphinx.ext.viewcode",
+]
+
+# autodoc/autosummary flags
+autoclass_content = "both"
+autodoc_default_flags = ["members"]
+autosummary_generate = True
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ["_templates"]
+
+# The suffix(es) of source filenames.
+# You can specify multiple suffix as a list of string:
+# source_suffix = ['.rst', '.md']
+source_suffix = ".rst"
+
+# The encoding of source files.
+# source_encoding = 'utf-8-sig'
+
+# The master toctree document.
+master_doc = "index"
+
+# General information about the project.
+project = u"google-cloud-bigquerystorage"
+copyright = u"2017, Google"
+author = u"Google APIs"
+
+# The version info for the project you're documenting, acts as replacement for
+# |version| and |release|, also used in various other places throughout the
+# built documents.
+#
+# The full version, including alpha/beta/rc tags.
+release = __version__
+# The short X.Y version.
+version = ".".join(release.split(".")[0:2])
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#
+# This is also used if you do content translation via gettext catalogs.
+# Usually you set "language" from the command line for these cases.
+language = None
+
+# There are two options for replacing |today|: either, you set today to some
+# non-false value, then it is used:
+# today = ''
+# Else, today_fmt is used as the format for a strftime call.
+# today_fmt = '%B %d, %Y'
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+exclude_patterns = ["_build"]
+
+# The reST default role (used for this markup: `text`) to use for all
+# documents.
+# default_role = None
+
+# If true, '()' will be appended to :func: etc. cross-reference text.
+# add_function_parentheses = True
+
+# If true, the current module name will be prepended to all description
+# unit titles (such as .. function::).
+# add_module_names = True
+
+# If true, sectionauthor and moduleauthor directives will be shown in the
+# output. They are ignored by default.
+# show_authors = False
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = "sphinx"
+
+# A list of ignored prefixes for module index sorting.
+# modindex_common_prefix = []
+
+# If true, keep warnings as "system message" paragraphs in the built documents.
+# keep_warnings = False
+
+# If true, `todo` and `todoList` produce output, else they produce nothing.
+todo_include_todos = True
+
+# -- Options for HTML output ----------------------------------------------
+
+# The theme to use for HTML and HTML Help pages. See the documentation for
+# a list of builtin themes.
+html_theme = "alabaster"
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further. For a list of options available for each theme, see the
+# documentation.
+# html_theme_options = {}
+
+# Add any paths that contain custom themes here, relative to this directory.
+# html_theme_path = []
+
+# The name for this set of Sphinx documents. If None, it defaults to
+# " v documentation".
+# html_title = None
+
+# A shorter title for the navigation bar. Default is the same as html_title.
+# html_short_title = None
+
+# The name of an image file (relative to this directory) to place at the top
+# of the sidebar.
+# html_logo = None
+
+# The name of an image file (within the static path) to use as favicon of the
+# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
+# pixels large.
+# html_favicon = None
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+# html_static_path = []
+
+# Add any extra paths that contain custom files (such as robots.txt or
+# .htaccess) here, relative to this directory. These files are copied
+# directly to the root of the documentation.
+# html_extra_path = []
+
+# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
+# using the given strftime format.
+# html_last_updated_fmt = '%b %d, %Y'
+
+# If true, SmartyPants will be used to convert quotes and dashes to
+# typographically correct entities.
+# html_use_smartypants = True
+
+# Custom sidebar templates, maps document names to template names.
+# html_sidebars = {}
+
+# Additional templates that should be rendered to pages, maps page names to
+# template names.
+# html_additional_pages = {}
+
+# If false, no module index is generated.
+# html_domain_indices = True
+
+# If false, no index is generated.
+# html_use_index = True
+
+# If true, the index is split into individual pages for each letter.
+# html_split_index = False
+
+# If true, links to the reST sources are added to the pages.
+# html_show_sourcelink = True
+
+# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
+# html_show_sphinx = True
+
+# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
+# html_show_copyright = True
+
+# If true, an OpenSearch description file will be output, and all pages will
+# contain a tag referring to it. The value of this option must be the
+# base URL from which the finished HTML is served.
+# html_use_opensearch = ''
+
+# This is the file name suffix for HTML files (e.g. ".xhtml").
+# html_file_suffix = None
+
+# Language to be used for generating the HTML full-text search index.
+# Sphinx supports the following languages:
+# 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja'
+# 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr'
+# html_search_language = 'en'
+
+# A dictionary with options for the search language support, empty by default.
+# Now only 'ja' uses this config value
+# html_search_options = {'type': 'default'}
+
+# The name of a javascript file (relative to the configuration directory) that
+# implements a search results scorer. If empty, the default will be used.
+# html_search_scorer = 'scorer.js'
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = "google-cloud-bigquerystorage-doc"
+
+# -- Options for LaTeX output ---------------------------------------------
+
+latex_elements = {
+ # The paper size ('letterpaper' or 'a4paper').
+ #'papersize': 'letterpaper',
+ # The font size ('10pt', '11pt' or '12pt').
+ #'pointsize': '10pt',
+ # Additional stuff for the LaTeX preamble.
+ #'preamble': '',
+ # Latex figure (float) alignment
+ #'figure_align': 'htbp',
+}
+
+# Grouping the document tree into LaTeX files. List of tuples
+# (source start file, target name, title,
+# author, documentclass [howto, manual, or own class]).
+latex_documents = [
+ (
+ master_doc,
+ "google-cloud-bigquerystorage.tex",
+ u"google-cloud-bigquerystorage Documentation",
+ author,
+ "manual",
+ )
+]
+
+# The name of an image file (relative to this directory) to place at the top of
+# the title page.
+# latex_logo = None
+
+# For "manual" documents, if this is true, then toplevel headings are parts,
+# not chapters.
+# latex_use_parts = False
+
+# If true, show page references after internal links.
+# latex_show_pagerefs = False
+
+# If true, show URL addresses after external links.
+# latex_show_urls = False
+
+# Documents to append as an appendix to all manuals.
+# latex_appendices = []
+
+# If false, no module index is generated.
+# latex_domain_indices = True
+
+# -- Options for manual page output ---------------------------------------
+
+# One entry per manual page. List of tuples
+# (source start file, name, description, authors, manual section).
+man_pages = [
+ (
+ master_doc,
+ "google-cloud-bigquerystorage",
+ u"google-cloud-bigquerystorage Documentation",
+ [author],
+ 1,
+ )
+]
+
+# If true, show URL addresses after external links.
+# man_show_urls = False
+
+# -- Options for Texinfo output -------------------------------------------
+
+# Grouping the document tree into Texinfo files. List of tuples
+# (source start file, target name, title, author,
+# dir menu entry, description, category)
+texinfo_documents = [
+ (
+ master_doc,
+ "google-cloud-bigquerystorage",
+ u"google-cloud-bigquerystorage Documentation",
+ author,
+ "google-cloud-bigquerystorage",
+ "GAPIC library for the {metadata.shortName} v1beta1 service",
+ "APIs",
+ )
+]
+
+# Documents to append as an appendix to all manuals.
+# texinfo_appendices = []
+
+# If false, no module index is generated.
+# texinfo_domain_indices = True
+
+# How to display URL addresses: 'footnote', 'no', or 'inline'.
+# texinfo_show_urls = 'footnote'
+
+# If true, do not generate a @detailmenu in the "Top" node's menu.
+# texinfo_no_detailmenu = False
+
+# Example configuration for intersphinx: refer to the Python standard library.
+intersphinx_mapping = {
+ "python": ("http://python.readthedocs.org/en/latest/", None),
+ "gax": ("https://gax-python.readthedocs.org/en/latest/", None),
+ "fastavro": ("https://fastavro.readthedocs.io/en/stable/", None),
+ "pandas": ("https://pandas.pydata.org/pandas-docs/stable/", None),
+}
+
+# Napoleon settings
+napoleon_google_docstring = True
+napoleon_numpy_docstring = True
+napoleon_include_private_with_doc = False
+napoleon_include_special_with_doc = True
+napoleon_use_admonition_for_examples = False
+napoleon_use_admonition_for_notes = False
+napoleon_use_admonition_for_references = False
+napoleon_use_ivar = False
+napoleon_use_param = True
+napoleon_use_rtype = True
diff --git a/bigquery_storage/docs/gapic/v1beta1/api.rst b/bigquery_storage/docs/gapic/v1beta1/api.rst
new file mode 100644
index 000000000000..d4df98557e15
--- /dev/null
+++ b/bigquery_storage/docs/gapic/v1beta1/api.rst
@@ -0,0 +1,6 @@
+Client for BigQuery Storage API
+===============================
+
+.. automodule:: google.cloud.bigquery_storage_v1beta1
+ :members:
+ :inherited-members:
\ No newline at end of file
diff --git a/bigquery_storage/docs/gapic/v1beta1/reader.rst b/bigquery_storage/docs/gapic/v1beta1/reader.rst
new file mode 100644
index 000000000000..5b6af828f53e
--- /dev/null
+++ b/bigquery_storage/docs/gapic/v1beta1/reader.rst
@@ -0,0 +1,6 @@
+Reader for BigQuery Storage API
+===============================
+
+.. automodule:: google.cloud.bigquery_storage_v1beta1.reader
+ :members:
+ :inherited-members:
diff --git a/bigquery_storage/docs/gapic/v1beta1/types.rst b/bigquery_storage/docs/gapic/v1beta1/types.rst
new file mode 100644
index 000000000000..a36210a64e52
--- /dev/null
+++ b/bigquery_storage/docs/gapic/v1beta1/types.rst
@@ -0,0 +1,5 @@
+Types for BigQuery Storage API Client
+=====================================
+
+.. automodule:: google.cloud.bigquery_storage_v1beta1.types
+ :members:
\ No newline at end of file
diff --git a/bigquery_storage/docs/index.rst b/bigquery_storage/docs/index.rst
new file mode 100644
index 000000000000..f34b0cfb4e5a
--- /dev/null
+++ b/bigquery_storage/docs/index.rst
@@ -0,0 +1,19 @@
+.. include:: README.rst
+
+API Reference
+-------------
+.. toctree::
+ :maxdepth: 2
+
+ gapic/v1beta1/api
+ gapic/v1beta1/reader
+ gapic/v1beta1/types
+
+Example Usage
+-------------
+
+.. literalinclude:: samples/quickstart.py
+ :language: python
+ :dedent: 4
+ :start-after: [START bigquerystorage_quickstart]
+ :end-before: [END bigquerystorage_quickstart]
diff --git a/bigquery_storage/docs/samples b/bigquery_storage/docs/samples
new file mode 120000
index 000000000000..e804737ed3a9
--- /dev/null
+++ b/bigquery_storage/docs/samples
@@ -0,0 +1 @@
+../samples
\ No newline at end of file
diff --git a/bigquery_storage/google/__init__.py b/bigquery_storage/google/__init__.py
new file mode 100644
index 000000000000..8fcc60e2b9c6
--- /dev/null
+++ b/bigquery_storage/google/__init__.py
@@ -0,0 +1,24 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+try:
+ import pkg_resources
+
+ pkg_resources.declare_namespace(__name__)
+except ImportError:
+ import pkgutil
+
+ __path__ = pkgutil.extend_path(__path__, __name__)
diff --git a/bigquery_storage/google/cloud/__init__.py b/bigquery_storage/google/cloud/__init__.py
new file mode 100644
index 000000000000..8fcc60e2b9c6
--- /dev/null
+++ b/bigquery_storage/google/cloud/__init__.py
@@ -0,0 +1,24 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+try:
+ import pkg_resources
+
+ pkg_resources.declare_namespace(__name__)
+except ImportError:
+ import pkgutil
+
+ __path__ = pkgutil.extend_path(__path__, __name__)
diff --git a/bigquery_storage/google/cloud/bigquery_storage.py b/bigquery_storage/google/cloud/bigquery_storage.py
new file mode 100644
index 000000000000..8613643ee25d
--- /dev/null
+++ b/bigquery_storage/google/cloud/bigquery_storage.py
@@ -0,0 +1,25 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from __future__ import absolute_import
+
+from google.cloud.bigquery_storage_v1beta1 import BigQueryStorageClient
+from google.cloud.bigquery_storage_v1beta1 import enums
+from google.cloud.bigquery_storage_v1beta1 import types
+
+
+__all__ = ("enums", "types", "BigQueryStorageClient")
diff --git a/bigquery_storage/google/cloud/bigquery_storage_v1beta1/__init__.py b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/__init__.py
new file mode 100644
index 000000000000..e355a0f6b8a5
--- /dev/null
+++ b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/__init__.py
@@ -0,0 +1,43 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import absolute_import
+
+import pkg_resources
+
+__version__ = pkg_resources.get_distribution(
+ "google-cloud-bigquery-storage"
+).version # noqa
+
+from google.cloud.bigquery_storage_v1beta1 import types
+from google.cloud.bigquery_storage_v1beta1 import client
+from google.cloud.bigquery_storage_v1beta1.gapic import enums
+
+
+class BigQueryStorageClient(client.BigQueryStorageClient):
+ __doc__ = client.BigQueryStorageClient.__doc__
+ enums = enums
+
+
+__all__ = (
+ # google.cloud.bigquery_storage_v1beta1
+ "__version__",
+ "types",
+ # google.cloud.bigquery_storage_v1beta1.client
+ "BigQueryStorageClient",
+ # google.cloud.bigquery_storage_v1beta1.gapic
+ "enums",
+)
diff --git a/bigquery_storage/google/cloud/bigquery_storage_v1beta1/client.py b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/client.py
new file mode 100644
index 000000000000..4f53a30550a8
--- /dev/null
+++ b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/client.py
@@ -0,0 +1,126 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Parent client for calling the Cloud BigQuery Storage API.
+
+This is the base from which all interactions with the API occur.
+"""
+
+from __future__ import absolute_import
+
+import google.api_core.gapic_v1.method
+
+from google.cloud.bigquery_storage_v1beta1 import reader
+from google.cloud.bigquery_storage_v1beta1.gapic import big_query_storage_client # noqa
+
+
+_SCOPES = (
+ "https://www.googleapis.com/auth/bigquery",
+ "https://www.googleapis.com/auth/cloud-platform",
+)
+
+
+class BigQueryStorageClient(big_query_storage_client.BigQueryStorageClient):
+ """Client for interacting with BigQuery Storage API.
+
+ The BigQuery storage API can be used to read data stored in BigQuery.
+ """
+
+ def read_rows(
+ self,
+ read_position,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Reads rows from the table in the format prescribed by the read
+ session. Each response contains one or more table rows, up to a
+ maximum of 10 MiB per response; read requests which attempt to read
+ individual rows larger than this will fail.
+
+ Each request also returns a set of stream statistics reflecting the
+ estimated total number of rows in the read stream. This number is
+ computed based on the total table size and the number of active
+ streams in the read session, and may change as other streams continue
+ to read data.
+
+ Example:
+ >>> from google.cloud import bigquery_storage_v1beta1
+ >>>
+ >>> client = bigquery_storage_v1beta1.BigQueryStorageClient()
+ >>>
+ >>> # TODO: Initialize ``table_reference``:
+ >>> table_reference = {
+ ... 'project_id': 'your-data-project-id',
+ ... 'dataset_id': 'your_dataset_id',
+ ... 'table_id': 'your_table_id',
+ ... }
+ >>>
+ >>> # TODO: Initialize `parent`:
+ >>> parent = 'projects/your-billing-project-id'
+ >>>
+ >>> session = client.create_read_session(table_reference, parent)
+ >>> read_position = bigquery_storage_v1beta1.types.StreamPosition(
+ ... stream=session.streams[0], # TODO: Read the other streams.
+ ... )
+ >>>
+ >>> for element in client.read_rows(read_position):
+ ... # process element
+ ... pass
+
+ Args:
+ read_position (Union[ \
+ dict, \
+ ~google.cloud.bigquery_storage_v1beta1.types.StreamPosition \
+ ]):
+ Required. Identifier of the position in the stream to start
+ reading from. The offset requested must be less than the last
+ row read from ReadRows. Requesting a larger offset is
+ undefined. If a dict is provided, it must be of the same form
+ as the protobuf message
+ :class:`~google.cloud.bigquery_storage_v1beta1.types.StreamPosition`
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will not
+ be retried.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ ~google.cloud.bigquery_storage_v1beta1.reader.ReadRowsStream:
+ An iterable of
+ :class:`~google.cloud.bigquery_storage_v1beta1.types.ReadRowsResponse`.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ gapic_client = super(BigQueryStorageClient, self)
+ stream = gapic_client.read_rows(
+ read_position, retry=retry, timeout=timeout, metadata=metadata
+ )
+ return reader.ReadRowsStream(
+ stream,
+ gapic_client,
+ read_position,
+ {"retry": retry, "timeout": timeout, "metadata": metadata},
+ )
diff --git a/bigquery_storage/google/cloud/bigquery_storage_v1beta1/gapic/__init__.py b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/gapic/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/bigquery_storage/google/cloud/bigquery_storage_v1beta1/gapic/big_query_storage_client.py b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/gapic/big_query_storage_client.py
new file mode 100644
index 000000000000..e44e74830031
--- /dev/null
+++ b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/gapic/big_query_storage_client.py
@@ -0,0 +1,671 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Accesses the google.cloud.bigquery.storage.v1beta1 BigQueryStorage API."""
+
+import pkg_resources
+import warnings
+
+from google.oauth2 import service_account
+import google.api_core.client_options
+import google.api_core.gapic_v1.client_info
+import google.api_core.gapic_v1.config
+import google.api_core.gapic_v1.method
+import google.api_core.path_template
+import google.api_core.gapic_v1.routing_header
+import google.api_core.grpc_helpers
+import grpc
+
+from google.cloud.bigquery_storage_v1beta1.gapic import big_query_storage_client_config
+from google.cloud.bigquery_storage_v1beta1.gapic import enums
+from google.cloud.bigquery_storage_v1beta1.gapic.transports import (
+ big_query_storage_grpc_transport,
+)
+from google.cloud.bigquery_storage_v1beta1.proto import read_options_pb2
+from google.cloud.bigquery_storage_v1beta1.proto import storage_pb2
+from google.cloud.bigquery_storage_v1beta1.proto import storage_pb2_grpc
+from google.cloud.bigquery_storage_v1beta1.proto import table_reference_pb2
+from google.protobuf import empty_pb2
+
+
+_GAPIC_LIBRARY_VERSION = pkg_resources.get_distribution(
+ "google-cloud-bigquery-storage"
+).version
+
+
+class BigQueryStorageClient(object):
+ """
+ BigQuery storage API.
+
+ The BigQuery storage API can be used to read data stored in BigQuery.
+ """
+
+ SERVICE_ADDRESS = "bigquerystorage.googleapis.com:443"
+ """The default address of the service."""
+
+ # The name of the interface for this client. This is the key used to
+ # find the method configuration in the client_config dictionary.
+ _INTERFACE_NAME = "google.cloud.bigquery.storage.v1beta1.BigQueryStorage"
+
+ @classmethod
+ def from_service_account_file(cls, filename, *args, **kwargs):
+ """Creates an instance of this client using the provided credentials
+ file.
+
+ Args:
+ filename (str): The path to the service account private key json
+ file.
+ args: Additional arguments to pass to the constructor.
+ kwargs: Additional arguments to pass to the constructor.
+
+ Returns:
+ BigQueryStorageClient: The constructed client.
+ """
+ credentials = service_account.Credentials.from_service_account_file(filename)
+ kwargs["credentials"] = credentials
+ return cls(*args, **kwargs)
+
+ from_service_account_json = from_service_account_file
+
+ def __init__(
+ self,
+ transport=None,
+ channel=None,
+ credentials=None,
+ client_config=None,
+ client_info=None,
+ client_options=None,
+ ):
+ """Constructor.
+
+ Args:
+ transport (Union[~.BigQueryStorageGrpcTransport,
+ Callable[[~.Credentials, type], ~.BigQueryStorageGrpcTransport]): A transport
+ instance, responsible for actually making the API calls.
+ The default transport uses the gRPC protocol.
+ This argument may also be a callable which returns a
+ transport instance. Callables will be sent the credentials
+ as the first argument and the default transport class as
+ the second argument.
+ channel (grpc.Channel): DEPRECATED. A ``Channel`` instance
+ through which to make calls. This argument is mutually exclusive
+ with ``credentials``; providing both will raise an exception.
+ credentials (google.auth.credentials.Credentials): The
+ authorization credentials to attach to requests. These
+ credentials identify this application to the service. If none
+ are specified, the client will attempt to ascertain the
+ credentials from the environment.
+ This argument is mutually exclusive with providing a
+ transport instance to ``transport``; doing so will raise
+ an exception.
+ client_config (dict): DEPRECATED. A dictionary of call options for
+ each method. If not specified, the default configuration is used.
+ client_info (google.api_core.gapic_v1.client_info.ClientInfo):
+ The client info used to send a user-agent string along with
+ API requests. If ``None``, then default info will be used.
+ Generally, you only need to set this if you're developing
+ your own client library.
+ client_options (Union[dict, google.api_core.client_options.ClientOptions]):
+ Client options used to set user options on the client. API Endpoint
+ should be set through client_options.
+ """
+ # Raise deprecation warnings for things we want to go away.
+ if client_config is not None:
+ warnings.warn(
+ "The `client_config` argument is deprecated.",
+ PendingDeprecationWarning,
+ stacklevel=2,
+ )
+ else:
+ client_config = big_query_storage_client_config.config
+
+ if channel:
+ warnings.warn(
+ "The `channel` argument is deprecated; use " "`transport` instead.",
+ PendingDeprecationWarning,
+ stacklevel=2,
+ )
+
+ api_endpoint = self.SERVICE_ADDRESS
+ if client_options:
+ if type(client_options) == dict:
+ client_options = google.api_core.client_options.from_dict(
+ client_options
+ )
+ if client_options.api_endpoint:
+ api_endpoint = client_options.api_endpoint
+
+ # Instantiate the transport.
+ # The transport is responsible for handling serialization and
+ # deserialization and actually sending data to the service.
+ if transport: # pragma: no cover
+ if callable(transport):
+ self.transport = transport(
+ credentials=credentials,
+ default_class=big_query_storage_grpc_transport.BigQueryStorageGrpcTransport,
+ address=api_endpoint,
+ )
+ else:
+ if credentials:
+ raise ValueError(
+ "Received both a transport instance and "
+ "credentials; these are mutually exclusive."
+ )
+ self.transport = transport
+ else:
+ self.transport = big_query_storage_grpc_transport.BigQueryStorageGrpcTransport(
+ address=api_endpoint, channel=channel, credentials=credentials
+ )
+
+ if client_info is None:
+ client_info = google.api_core.gapic_v1.client_info.ClientInfo(
+ gapic_version=_GAPIC_LIBRARY_VERSION
+ )
+ else:
+ client_info.gapic_version = _GAPIC_LIBRARY_VERSION
+ self._client_info = client_info
+
+ # Parse out the default settings for retry and timeout for each RPC
+ # from the client configuration.
+ # (Ordinarily, these are the defaults specified in the `*_config.py`
+ # file next to this one.)
+ self._method_configs = google.api_core.gapic_v1.config.parse_method_configs(
+ client_config["interfaces"][self._INTERFACE_NAME]
+ )
+
+ # Save a dictionary of cached API call functions.
+ # These are the actual callables which invoke the proper
+ # transport methods, wrapped with `wrap_method` to add retry,
+ # timeout, and the like.
+ self._inner_api_calls = {}
+
+ # Service calls
+ def create_read_session(
+ self,
+ table_reference,
+ parent,
+ table_modifiers=None,
+ requested_streams=None,
+ read_options=None,
+ format_=None,
+ sharding_strategy=None,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Creates a new read session. A read session divides the contents of a
+ BigQuery table into one or more streams, which can then be used to read
+ data from the table. The read session also specifies properties of the
+ data to be read, such as a list of columns or a push-down filter describing
+ the rows to be returned.
+
+ A particular row can be read by at most one stream. When the caller has
+ reached the end of each stream in the session, then all the data in the
+ table has been read.
+
+ Read sessions automatically expire 24 hours after they are created and do
+ not require manual clean-up by the caller.
+
+ Example:
+ >>> from google.cloud import bigquery_storage_v1beta1
+ >>>
+ >>> client = bigquery_storage_v1beta1.BigQueryStorageClient()
+ >>>
+ >>> # TODO: Initialize `table_reference`:
+ >>> table_reference = {}
+ >>>
+ >>> # TODO: Initialize `parent`:
+ >>> parent = ''
+ >>>
+ >>> response = client.create_read_session(table_reference, parent)
+
+ Args:
+ table_reference (Union[dict, ~google.cloud.bigquery_storage_v1beta1.types.TableReference]): Required. Reference to the table to read.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.bigquery_storage_v1beta1.types.TableReference`
+ parent (str): Required. String of the form ``projects/{project_id}`` indicating the
+ project this ReadSession is associated with. This is the project that
+ will be billed for usage.
+ table_modifiers (Union[dict, ~google.cloud.bigquery_storage_v1beta1.types.TableModifiers]): Optional. Any modifiers to the Table (e.g. snapshot timestamp).
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.bigquery_storage_v1beta1.types.TableModifiers`
+ requested_streams (int): Optional. Initial number of streams. If unset or 0, we will
+ provide a value of streams so as to produce reasonable throughput. Must be
+ non-negative. The number of streams may be lower than the requested number,
+ depending on the amount parallelism that is reasonable for the table and
+ the maximum amount of parallelism allowed by the system.
+
+ Streams must be read starting from offset 0.
+ read_options (Union[dict, ~google.cloud.bigquery_storage_v1beta1.types.TableReadOptions]): Optional. Read options for this session (e.g. column selection, filters).
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.bigquery_storage_v1beta1.types.TableReadOptions`
+ format_ (~google.cloud.bigquery_storage_v1beta1.types.DataFormat): Data output format. Currently default to Avro.
+ sharding_strategy (~google.cloud.bigquery_storage_v1beta1.types.ShardingStrategy): The strategy to use for distributing data among multiple streams. Currently
+ defaults to liquid sharding.
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.bigquery_storage_v1beta1.types.ReadSession` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "create_read_session" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "create_read_session"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.create_read_session,
+ default_retry=self._method_configs["CreateReadSession"].retry,
+ default_timeout=self._method_configs["CreateReadSession"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = storage_pb2.CreateReadSessionRequest(
+ table_reference=table_reference,
+ parent=parent,
+ table_modifiers=table_modifiers,
+ requested_streams=requested_streams,
+ read_options=read_options,
+ format=format_,
+ sharding_strategy=sharding_strategy,
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [
+ ("table_reference.project_id", table_reference.project_id),
+ ("table_reference.dataset_id", table_reference.dataset_id),
+ ]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata) # pragma: no cover
+
+ return self._inner_api_calls["create_read_session"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+
+ def read_rows(
+ self,
+ read_position,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Reads rows from the table in the format prescribed by the read session.
+ Each response contains one or more table rows, up to a maximum of 10 MiB
+ per response; read requests which attempt to read individual rows larger
+ than this will fail.
+
+ Each request also returns a set of stream statistics reflecting the
+ estimated total number of rows in the read stream. This number is computed
+ based on the total table size and the number of active streams in the read
+ session, and may change as other streams continue to read data.
+
+ Example:
+ >>> from google.cloud import bigquery_storage_v1beta1
+ >>>
+ >>> client = bigquery_storage_v1beta1.BigQueryStorageClient()
+ >>>
+ >>> # TODO: Initialize `read_position`:
+ >>> read_position = {}
+ >>>
+ >>> for element in client.read_rows(read_position):
+ ... # process element
+ ... pass
+
+ Args:
+ read_position (Union[dict, ~google.cloud.bigquery_storage_v1beta1.types.StreamPosition]): Required. Identifier of the position in the stream to start reading from.
+ The offset requested must be less than the last row read from ReadRows.
+ Requesting a larger offset is undefined.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.bigquery_storage_v1beta1.types.StreamPosition`
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ Iterable[~google.cloud.bigquery_storage_v1beta1.types.ReadRowsResponse].
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "read_rows" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "read_rows"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.read_rows,
+ default_retry=self._method_configs["ReadRows"].retry,
+ default_timeout=self._method_configs["ReadRows"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = storage_pb2.ReadRowsRequest(read_position=read_position)
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("read_position.stream.name", read_position.stream.name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata) # pragma: no cover
+
+ return self._inner_api_calls["read_rows"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+
+ def batch_create_read_session_streams(
+ self,
+ session,
+ requested_streams,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Creates additional streams for a ReadSession. This API can be used to
+ dynamically adjust the parallelism of a batch processing task upwards by
+ adding additional workers.
+
+ Example:
+ >>> from google.cloud import bigquery_storage_v1beta1
+ >>>
+ >>> client = bigquery_storage_v1beta1.BigQueryStorageClient()
+ >>>
+ >>> # TODO: Initialize `session`:
+ >>> session = {}
+ >>>
+ >>> # TODO: Initialize `requested_streams`:
+ >>> requested_streams = 0
+ >>>
+ >>> response = client.batch_create_read_session_streams(session, requested_streams)
+
+ Args:
+ session (Union[dict, ~google.cloud.bigquery_storage_v1beta1.types.ReadSession]): Required. Must be a non-expired session obtained from a call to
+ CreateReadSession. Only the name field needs to be set.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.bigquery_storage_v1beta1.types.ReadSession`
+ requested_streams (int): Required. Number of new streams requested. Must be positive.
+ Number of added streams may be less than this, see CreateReadSessionRequest
+ for more information.
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.bigquery_storage_v1beta1.types.BatchCreateReadSessionStreamsResponse` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "batch_create_read_session_streams" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "batch_create_read_session_streams"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.batch_create_read_session_streams,
+ default_retry=self._method_configs[
+ "BatchCreateReadSessionStreams"
+ ].retry,
+ default_timeout=self._method_configs[
+ "BatchCreateReadSessionStreams"
+ ].timeout,
+ client_info=self._client_info,
+ )
+
+ request = storage_pb2.BatchCreateReadSessionStreamsRequest(
+ session=session, requested_streams=requested_streams
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("session.name", session.name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata) # pragma: no cover
+
+ return self._inner_api_calls["batch_create_read_session_streams"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+
+ def finalize_stream(
+ self,
+ stream,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Triggers the graceful termination of a single stream in a ReadSession. This
+ API can be used to dynamically adjust the parallelism of a batch processing
+ task downwards without losing data.
+
+ This API does not delete the stream -- it remains visible in the
+ ReadSession, and any data processed by the stream is not released to other
+ streams. However, no additional data will be assigned to the stream once
+ this call completes. Callers must continue reading data on the stream until
+ the end of the stream is reached so that data which has already been
+ assigned to the stream will be processed.
+
+ This method will return an error if there are no other live streams
+ in the Session, or if SplitReadStream() has been called on the given
+ Stream.
+
+ Example:
+ >>> from google.cloud import bigquery_storage_v1beta1
+ >>>
+ >>> client = bigquery_storage_v1beta1.BigQueryStorageClient()
+ >>>
+ >>> # TODO: Initialize `stream`:
+ >>> stream = {}
+ >>>
+ >>> client.finalize_stream(stream)
+
+ Args:
+ stream (Union[dict, ~google.cloud.bigquery_storage_v1beta1.types.Stream]): Stream to finalize.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.bigquery_storage_v1beta1.types.Stream`
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "finalize_stream" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "finalize_stream"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.finalize_stream,
+ default_retry=self._method_configs["FinalizeStream"].retry,
+ default_timeout=self._method_configs["FinalizeStream"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = storage_pb2.FinalizeStreamRequest(stream=stream)
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("stream.name", stream.name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata) # pragma: no cover
+
+ self._inner_api_calls["finalize_stream"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
+
+ def split_read_stream(
+ self,
+ original_stream,
+ fraction=None,
+ retry=google.api_core.gapic_v1.method.DEFAULT,
+ timeout=google.api_core.gapic_v1.method.DEFAULT,
+ metadata=None,
+ ):
+ """
+ Splits a given read stream into two Streams. These streams are referred
+ to as the primary and the residual of the split. The original stream can
+ still be read from in the same manner as before. Both of the returned
+ streams can also be read from, and the total rows return by both child
+ streams will be the same as the rows read from the original stream.
+
+ Moreover, the two child streams will be allocated back to back in the
+ original Stream. Concretely, it is guaranteed that for streams Original,
+ Primary, and Residual, that Original[0-j] = Primary[0-j] and
+ Original[j-n] = Residual[0-m] once the streams have been read to
+ completion.
+
+ This method is guaranteed to be idempotent.
+
+ Example:
+ >>> from google.cloud import bigquery_storage_v1beta1
+ >>>
+ >>> client = bigquery_storage_v1beta1.BigQueryStorageClient()
+ >>>
+ >>> # TODO: Initialize `original_stream`:
+ >>> original_stream = {}
+ >>>
+ >>> response = client.split_read_stream(original_stream)
+
+ Args:
+ original_stream (Union[dict, ~google.cloud.bigquery_storage_v1beta1.types.Stream]): Stream to split.
+
+ If a dict is provided, it must be of the same form as the protobuf
+ message :class:`~google.cloud.bigquery_storage_v1beta1.types.Stream`
+ fraction (float): A value in the range (0.0, 1.0) that specifies the fractional point at
+ which the original stream should be split. The actual split point is
+ evaluated on pre-filtered rows, so if a filter is provided, then there is
+ no guarantee that the division of the rows between the new child streams
+ will be proportional to this fractional value. Additionally, because the
+ server-side unit for assigning data is collections of rows, this fraction
+ will always map to to a data storage boundary on the server side.
+ retry (Optional[google.api_core.retry.Retry]): A retry object used
+ to retry requests. If ``None`` is specified, requests will
+ be retried using a default configuration.
+ timeout (Optional[float]): The amount of time, in seconds, to wait
+ for the request to complete. Note that if ``retry`` is
+ specified, the timeout applies to each individual attempt.
+ metadata (Optional[Sequence[Tuple[str, str]]]): Additional metadata
+ that is provided to the method.
+
+ Returns:
+ A :class:`~google.cloud.bigquery_storage_v1beta1.types.SplitReadStreamResponse` instance.
+
+ Raises:
+ google.api_core.exceptions.GoogleAPICallError: If the request
+ failed for any reason.
+ google.api_core.exceptions.RetryError: If the request failed due
+ to a retryable error and retry attempts failed.
+ ValueError: If the parameters are invalid.
+ """
+ # Wrap the transport method to add retry and timeout logic.
+ if "split_read_stream" not in self._inner_api_calls:
+ self._inner_api_calls[
+ "split_read_stream"
+ ] = google.api_core.gapic_v1.method.wrap_method(
+ self.transport.split_read_stream,
+ default_retry=self._method_configs["SplitReadStream"].retry,
+ default_timeout=self._method_configs["SplitReadStream"].timeout,
+ client_info=self._client_info,
+ )
+
+ request = storage_pb2.SplitReadStreamRequest(
+ original_stream=original_stream, fraction=fraction
+ )
+ if metadata is None:
+ metadata = []
+ metadata = list(metadata)
+ try:
+ routing_header = [("original_stream.name", original_stream.name)]
+ except AttributeError:
+ pass
+ else:
+ routing_metadata = google.api_core.gapic_v1.routing_header.to_grpc_metadata(
+ routing_header
+ )
+ metadata.append(routing_metadata) # pragma: no cover
+
+ return self._inner_api_calls["split_read_stream"](
+ request, retry=retry, timeout=timeout, metadata=metadata
+ )
diff --git a/bigquery_storage/google/cloud/bigquery_storage_v1beta1/gapic/big_query_storage_client_config.py b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/gapic/big_query_storage_client_config.py
new file mode 100644
index 000000000000..e33b166f703b
--- /dev/null
+++ b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/gapic/big_query_storage_client_config.py
@@ -0,0 +1,67 @@
+config = {
+ "interfaces": {
+ "google.cloud.bigquery.storage.v1beta1.BigQueryStorage": {
+ "retry_codes": {
+ "idempotent": ["DEADLINE_EXCEEDED", "UNAVAILABLE"],
+ "non_idempotent": [],
+ "unary_streaming": ["UNAVAILABLE"],
+ },
+ "retry_params": {
+ "default": {
+ "initial_retry_delay_millis": 100,
+ "retry_delay_multiplier": 1.3,
+ "max_retry_delay_millis": 60000,
+ "initial_rpc_timeout_millis": 20000,
+ "rpc_timeout_multiplier": 1.0,
+ "max_rpc_timeout_millis": 20000,
+ "total_timeout_millis": 600000,
+ },
+ "create_read_session": {
+ "initial_retry_delay_millis": 100,
+ "retry_delay_multiplier": 1.3,
+ "max_retry_delay_millis": 60000,
+ "initial_rpc_timeout_millis": 120000,
+ "rpc_timeout_multiplier": 1.0,
+ "max_rpc_timeout_millis": 120000,
+ "total_timeout_millis": 600000,
+ },
+ "read_rows": {
+ "initial_retry_delay_millis": 100,
+ "retry_delay_multiplier": 1.3,
+ "max_retry_delay_millis": 60000,
+ "initial_rpc_timeout_millis": 86400000,
+ "rpc_timeout_multiplier": 1.0,
+ "max_rpc_timeout_millis": 86400000,
+ "total_timeout_millis": 86400000,
+ },
+ },
+ "methods": {
+ "CreateReadSession": {
+ "timeout_millis": 60000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "create_read_session",
+ },
+ "ReadRows": {
+ "timeout_millis": 86400000,
+ "retry_codes_name": "unary_streaming",
+ "retry_params_name": "read_rows",
+ },
+ "BatchCreateReadSessionStreams": {
+ "timeout_millis": 60000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "FinalizeStream": {
+ "timeout_millis": 60000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ "SplitReadStream": {
+ "timeout_millis": 60000,
+ "retry_codes_name": "idempotent",
+ "retry_params_name": "default",
+ },
+ },
+ }
+ }
+}
diff --git a/bigquery_storage/google/cloud/bigquery_storage_v1beta1/gapic/enums.py b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/gapic/enums.py
new file mode 100644
index 000000000000..bb3b3b7601c9
--- /dev/null
+++ b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/gapic/enums.py
@@ -0,0 +1,58 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Wrappers for protocol buffer enum types."""
+
+import enum
+
+
+class DataFormat(enum.IntEnum):
+ """
+ Data format for input or output data.
+
+ Attributes:
+ DATA_FORMAT_UNSPECIFIED (int): Data format is unspecified.
+ AVRO (int): Avro is a standard open source row based file format.
+ See https://avro.apache.org/ for more details.
+ ARROW (int)
+ """
+
+ DATA_FORMAT_UNSPECIFIED = 0
+ AVRO = 1
+ ARROW = 3
+
+
+class ShardingStrategy(enum.IntEnum):
+ """
+ Strategy for distributing data among multiple streams in a read session.
+
+ Attributes:
+ SHARDING_STRATEGY_UNSPECIFIED (int): Same as LIQUID.
+ LIQUID (int): Assigns data to each stream based on the client's read rate. The faster the
+ client reads from a stream, the more data is assigned to the stream. In
+ this strategy, it's possible to read all data from a single stream even if
+ there are other streams present.
+ BALANCED (int): Assigns data to each stream such that roughly the same number of rows can
+ be read from each stream. Because the server-side unit for assigning data
+ is collections of rows, the API does not guarantee that each stream will
+ return the same number or rows. Additionally, the limits are enforced based
+ on the number of pre-filtering rows, so some filters can lead to lopsided
+ assignments.
+ """
+
+ SHARDING_STRATEGY_UNSPECIFIED = 0
+ LIQUID = 1
+ BALANCED = 2
diff --git a/bigquery_storage/google/cloud/bigquery_storage_v1beta1/gapic/transports/__init__.py b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/gapic/transports/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/bigquery_storage/google/cloud/bigquery_storage_v1beta1/gapic/transports/big_query_storage_grpc_transport.py b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/gapic/transports/big_query_storage_grpc_transport.py
new file mode 100644
index 000000000000..bf3b8b6f5b24
--- /dev/null
+++ b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/gapic/transports/big_query_storage_grpc_transport.py
@@ -0,0 +1,225 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import google.api_core.grpc_helpers
+
+from google.cloud.bigquery_storage_v1beta1.proto import storage_pb2_grpc
+
+
+class BigQueryStorageGrpcTransport(object):
+ """gRPC transport class providing stubs for
+ google.cloud.bigquery.storage.v1beta1 BigQueryStorage API.
+
+ The transport provides access to the raw gRPC stubs,
+ which can be used to take advantage of advanced
+ features of gRPC.
+ """
+
+ # The scopes needed to make gRPC calls to all of the methods defined
+ # in this service.
+ _OAUTH_SCOPES = (
+ "https://www.googleapis.com/auth/bigquery",
+ "https://www.googleapis.com/auth/cloud-platform",
+ )
+
+ def __init__(
+ self,
+ channel=None,
+ credentials=None,
+ address="bigquerystorage.googleapis.com:443",
+ ):
+ """Instantiate the transport class.
+
+ Args:
+ channel (grpc.Channel): A ``Channel`` instance through
+ which to make calls. This argument is mutually exclusive
+ with ``credentials``; providing both will raise an exception.
+ credentials (google.auth.credentials.Credentials): The
+ authorization credentials to attach to requests. These
+ credentials identify this application to the service. If none
+ are specified, the client will attempt to ascertain the
+ credentials from the environment.
+ address (str): The address where the service is hosted.
+ """
+ # If both `channel` and `credentials` are specified, raise an
+ # exception (channels come with credentials baked in already).
+ if channel is not None and credentials is not None: # pragma: no cover
+ raise ValueError(
+ "The `channel` and `credentials` arguments are mutually " "exclusive."
+ )
+
+ # Create the channel.
+ if channel is None: # pragma: no cover
+ channel = self.create_channel(
+ address=address,
+ credentials=credentials,
+ options={
+ "grpc.max_send_message_length": -1,
+ "grpc.max_receive_message_length": -1,
+ }.items(),
+ )
+
+ self._channel = channel
+
+ # gRPC uses objects called "stubs" that are bound to the
+ # channel and provide a basic method for each RPC.
+ self._stubs = {
+ "big_query_storage_stub": storage_pb2_grpc.BigQueryStorageStub(channel)
+ }
+
+ @classmethod
+ def create_channel(
+ cls, address="bigquerystorage.googleapis.com:443", credentials=None, **kwargs
+ ):
+ """Create and return a gRPC channel object.
+
+ Args:
+ address (str): The host for the channel to use.
+ credentials (~.Credentials): The
+ authorization credentials to attach to requests. These
+ credentials identify this application to the service. If
+ none are specified, the client will attempt to ascertain
+ the credentials from the environment.
+ kwargs (dict): Keyword arguments, which are passed to the
+ channel creation.
+
+ Returns:
+ grpc.Channel: A gRPC channel object.
+ """
+ return google.api_core.grpc_helpers.create_channel( # pragma: no cover
+ address, credentials=credentials, scopes=cls._OAUTH_SCOPES, **kwargs
+ )
+
+ @property
+ def channel(self):
+ """The gRPC channel used by the transport.
+
+ Returns:
+ grpc.Channel: A gRPC channel object.
+ """
+ return self._channel
+
+ @property
+ def create_read_session(self):
+ """Return the gRPC stub for :meth:`BigQueryStorageClient.create_read_session`.
+
+ Creates a new read session. A read session divides the contents of a
+ BigQuery table into one or more streams, which can then be used to read
+ data from the table. The read session also specifies properties of the
+ data to be read, such as a list of columns or a push-down filter describing
+ the rows to be returned.
+
+ A particular row can be read by at most one stream. When the caller has
+ reached the end of each stream in the session, then all the data in the
+ table has been read.
+
+ Read sessions automatically expire 24 hours after they are created and do
+ not require manual clean-up by the caller.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["big_query_storage_stub"].CreateReadSession
+
+ @property
+ def read_rows(self):
+ """Return the gRPC stub for :meth:`BigQueryStorageClient.read_rows`.
+
+ Reads rows from the table in the format prescribed by the read session.
+ Each response contains one or more table rows, up to a maximum of 10 MiB
+ per response; read requests which attempt to read individual rows larger
+ than this will fail.
+
+ Each request also returns a set of stream statistics reflecting the
+ estimated total number of rows in the read stream. This number is computed
+ based on the total table size and the number of active streams in the read
+ session, and may change as other streams continue to read data.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["big_query_storage_stub"].ReadRows
+
+ @property
+ def batch_create_read_session_streams(self):
+ """Return the gRPC stub for :meth:`BigQueryStorageClient.batch_create_read_session_streams`.
+
+ Creates additional streams for a ReadSession. This API can be used to
+ dynamically adjust the parallelism of a batch processing task upwards by
+ adding additional workers.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["big_query_storage_stub"].BatchCreateReadSessionStreams
+
+ @property
+ def finalize_stream(self):
+ """Return the gRPC stub for :meth:`BigQueryStorageClient.finalize_stream`.
+
+ Triggers the graceful termination of a single stream in a ReadSession. This
+ API can be used to dynamically adjust the parallelism of a batch processing
+ task downwards without losing data.
+
+ This API does not delete the stream -- it remains visible in the
+ ReadSession, and any data processed by the stream is not released to other
+ streams. However, no additional data will be assigned to the stream once
+ this call completes. Callers must continue reading data on the stream until
+ the end of the stream is reached so that data which has already been
+ assigned to the stream will be processed.
+
+ This method will return an error if there are no other live streams
+ in the Session, or if SplitReadStream() has been called on the given
+ Stream.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["big_query_storage_stub"].FinalizeStream
+
+ @property
+ def split_read_stream(self):
+ """Return the gRPC stub for :meth:`BigQueryStorageClient.split_read_stream`.
+
+ Splits a given read stream into two Streams. These streams are referred
+ to as the primary and the residual of the split. The original stream can
+ still be read from in the same manner as before. Both of the returned
+ streams can also be read from, and the total rows return by both child
+ streams will be the same as the rows read from the original stream.
+
+ Moreover, the two child streams will be allocated back to back in the
+ original Stream. Concretely, it is guaranteed that for streams Original,
+ Primary, and Residual, that Original[0-j] = Primary[0-j] and
+ Original[j-n] = Residual[0-m] once the streams have been read to
+ completion.
+
+ This method is guaranteed to be idempotent.
+
+ Returns:
+ Callable: A callable which accepts the appropriate
+ deserialized request object and returns a
+ deserialized response object.
+ """
+ return self._stubs["big_query_storage_stub"].SplitReadStream
diff --git a/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/__init__.py b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/arrow.proto b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/arrow.proto
new file mode 100644
index 000000000000..3003de444c2d
--- /dev/null
+++ b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/arrow.proto
@@ -0,0 +1,37 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.bigquery.storage.v1beta1;
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/bigquery/storage/v1beta1;storage";
+option java_outer_classname = "ArrowProto";
+option java_package = "com.google.cloud.bigquery.storage.v1beta1";
+
+// Arrow schema.
+message ArrowSchema {
+ // IPC serialized Arrow schema.
+ bytes serialized_schema = 1;
+}
+
+// Arrow RecordBatch.
+message ArrowRecordBatch {
+ // IPC serialized Arrow RecordBatch.
+ bytes serialized_record_batch = 1;
+
+ // The count of rows in the returning block.
+ int64 row_count = 2;
+}
diff --git a/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/arrow_pb2.py b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/arrow_pb2.py
new file mode 100644
index 000000000000..4d7d90afaf9e
--- /dev/null
+++ b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/arrow_pb2.py
@@ -0,0 +1,170 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/bigquery/storage_v1beta1/proto/arrow.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/bigquery/storage_v1beta1/proto/arrow.proto",
+ package="google.cloud.bigquery.storage.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n)com.google.cloud.bigquery.storage.v1beta1B\nArrowProtoZLgoogle.golang.org/genproto/googleapis/cloud/bigquery/storage/v1beta1;storage"
+ ),
+ serialized_pb=_b(
+ '\n7google/cloud/bigquery/storage_v1beta1/proto/arrow.proto\x12%google.cloud.bigquery.storage.v1beta1"(\n\x0b\x41rrowSchema\x12\x19\n\x11serialized_schema\x18\x01 \x01(\x0c"F\n\x10\x41rrowRecordBatch\x12\x1f\n\x17serialized_record_batch\x18\x01 \x01(\x0c\x12\x11\n\trow_count\x18\x02 \x01(\x03\x42\x85\x01\n)com.google.cloud.bigquery.storage.v1beta1B\nArrowProtoZLgoogle.golang.org/genproto/googleapis/cloud/bigquery/storage/v1beta1;storageb\x06proto3'
+ ),
+)
+
+
+_ARROWSCHEMA = _descriptor.Descriptor(
+ name="ArrowSchema",
+ full_name="google.cloud.bigquery.storage.v1beta1.ArrowSchema",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="serialized_schema",
+ full_name="google.cloud.bigquery.storage.v1beta1.ArrowSchema.serialized_schema",
+ index=0,
+ number=1,
+ type=12,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b(""),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=98,
+ serialized_end=138,
+)
+
+
+_ARROWRECORDBATCH = _descriptor.Descriptor(
+ name="ArrowRecordBatch",
+ full_name="google.cloud.bigquery.storage.v1beta1.ArrowRecordBatch",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="serialized_record_batch",
+ full_name="google.cloud.bigquery.storage.v1beta1.ArrowRecordBatch.serialized_record_batch",
+ index=0,
+ number=1,
+ type=12,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b(""),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="row_count",
+ full_name="google.cloud.bigquery.storage.v1beta1.ArrowRecordBatch.row_count",
+ index=1,
+ number=2,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=140,
+ serialized_end=210,
+)
+
+DESCRIPTOR.message_types_by_name["ArrowSchema"] = _ARROWSCHEMA
+DESCRIPTOR.message_types_by_name["ArrowRecordBatch"] = _ARROWRECORDBATCH
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+ArrowSchema = _reflection.GeneratedProtocolMessageType(
+ "ArrowSchema",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_ARROWSCHEMA,
+ __module__="google.cloud.bigquery.storage_v1beta1.proto.arrow_pb2",
+ __doc__="""Arrow schema.
+
+
+ Attributes:
+ serialized_schema:
+ IPC serialized Arrow schema.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.storage.v1beta1.ArrowSchema)
+ ),
+)
+_sym_db.RegisterMessage(ArrowSchema)
+
+ArrowRecordBatch = _reflection.GeneratedProtocolMessageType(
+ "ArrowRecordBatch",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_ARROWRECORDBATCH,
+ __module__="google.cloud.bigquery.storage_v1beta1.proto.arrow_pb2",
+ __doc__="""Arrow RecordBatch.
+
+
+ Attributes:
+ serialized_record_batch:
+ IPC serialized Arrow RecordBatch.
+ row_count:
+ The count of rows in the returning block.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.storage.v1beta1.ArrowRecordBatch)
+ ),
+)
+_sym_db.RegisterMessage(ArrowRecordBatch)
+
+
+DESCRIPTOR._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/arrow_pb2_grpc.py b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/arrow_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/arrow_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/avro.proto b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/avro.proto
new file mode 100644
index 000000000000..021d8e44f9f2
--- /dev/null
+++ b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/avro.proto
@@ -0,0 +1,38 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.bigquery.storage.v1beta1;
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/bigquery/storage/v1beta1;storage";
+option java_outer_classname = "AvroProto";
+option java_package = "com.google.cloud.bigquery.storage.v1beta1";
+
+// Avro schema.
+message AvroSchema {
+ // Json serialized schema, as described at
+ // https://avro.apache.org/docs/1.8.1/spec.html
+ string schema = 1;
+}
+
+// Avro rows.
+message AvroRows {
+ // Binary serialized rows in a block.
+ bytes serialized_binary_rows = 1;
+
+ // The count of rows in the returning block.
+ int64 row_count = 2;
+}
diff --git a/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/avro_pb2.py b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/avro_pb2.py
new file mode 100644
index 000000000000..1f5ee11d4022
--- /dev/null
+++ b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/avro_pb2.py
@@ -0,0 +1,171 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/bigquery/storage_v1beta1/proto/avro.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/bigquery/storage_v1beta1/proto/avro.proto",
+ package="google.cloud.bigquery.storage.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n)com.google.cloud.bigquery.storage.v1beta1B\tAvroProtoZLgoogle.golang.org/genproto/googleapis/cloud/bigquery/storage/v1beta1;storage"
+ ),
+ serialized_pb=_b(
+ '\n6google/cloud/bigquery/storage_v1beta1/proto/avro.proto\x12%google.cloud.bigquery.storage.v1beta1"\x1c\n\nAvroSchema\x12\x0e\n\x06schema\x18\x01 \x01(\t"=\n\x08\x41vroRows\x12\x1e\n\x16serialized_binary_rows\x18\x01 \x01(\x0c\x12\x11\n\trow_count\x18\x02 \x01(\x03\x42\x84\x01\n)com.google.cloud.bigquery.storage.v1beta1B\tAvroProtoZLgoogle.golang.org/genproto/googleapis/cloud/bigquery/storage/v1beta1;storageb\x06proto3'
+ ),
+)
+
+
+_AVROSCHEMA = _descriptor.Descriptor(
+ name="AvroSchema",
+ full_name="google.cloud.bigquery.storage.v1beta1.AvroSchema",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="schema",
+ full_name="google.cloud.bigquery.storage.v1beta1.AvroSchema.schema",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=97,
+ serialized_end=125,
+)
+
+
+_AVROROWS = _descriptor.Descriptor(
+ name="AvroRows",
+ full_name="google.cloud.bigquery.storage.v1beta1.AvroRows",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="serialized_binary_rows",
+ full_name="google.cloud.bigquery.storage.v1beta1.AvroRows.serialized_binary_rows",
+ index=0,
+ number=1,
+ type=12,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b(""),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="row_count",
+ full_name="google.cloud.bigquery.storage.v1beta1.AvroRows.row_count",
+ index=1,
+ number=2,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=127,
+ serialized_end=188,
+)
+
+DESCRIPTOR.message_types_by_name["AvroSchema"] = _AVROSCHEMA
+DESCRIPTOR.message_types_by_name["AvroRows"] = _AVROROWS
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+AvroSchema = _reflection.GeneratedProtocolMessageType(
+ "AvroSchema",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_AVROSCHEMA,
+ __module__="google.cloud.bigquery.storage_v1beta1.proto.avro_pb2",
+ __doc__="""Avro schema.
+
+
+ Attributes:
+ schema:
+ Json serialized schema, as described at
+ https://avro.apache.org/docs/1.8.1/spec.html
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.storage.v1beta1.AvroSchema)
+ ),
+)
+_sym_db.RegisterMessage(AvroSchema)
+
+AvroRows = _reflection.GeneratedProtocolMessageType(
+ "AvroRows",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_AVROROWS,
+ __module__="google.cloud.bigquery.storage_v1beta1.proto.avro_pb2",
+ __doc__="""Avro rows.
+
+
+ Attributes:
+ serialized_binary_rows:
+ Binary serialized rows in a block.
+ row_count:
+ The count of rows in the returning block.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.storage.v1beta1.AvroRows)
+ ),
+)
+_sym_db.RegisterMessage(AvroRows)
+
+
+DESCRIPTOR._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/avro_pb2_grpc.py b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/avro_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/avro_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/read_options.proto b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/read_options.proto
new file mode 100644
index 000000000000..9591deba7f47
--- /dev/null
+++ b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/read_options.proto
@@ -0,0 +1,41 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.bigquery.storage.v1beta1;
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/bigquery/storage/v1beta1;storage";
+option java_package = "com.google.cloud.bigquery.storage.v1beta1";
+
+// Options dictating how we read a table.
+message TableReadOptions {
+ // Optional. Names of the fields in the table that should be read. If empty,
+ // all fields will be read. If the specified field is a nested field, all the
+ // sub-fields in the field will be selected. The output field order is
+ // unrelated to the order of fields in selected_fields.
+ repeated string selected_fields = 1;
+
+ // Optional. SQL text filtering statement, similar to a WHERE clause in
+ // a query. Currently, only a single predicate that is a comparison between
+ // a column and a constant value is supported. Aggregates are not supported.
+ //
+ // Examples: "int_field > 5"
+ // "date_field = CAST('2014-9-27' as DATE)"
+ // "nullable_field is not NULL"
+ // "st_equals(geo_field, st_geofromtext("POINT(2, 2)"))"
+ // "numeric_field BETWEEN 1.0 AND 5.0"
+ string row_restriction = 2;
+}
diff --git a/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/read_options_pb2.py b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/read_options_pb2.py
new file mode 100644
index 000000000000..7291232de3c5
--- /dev/null
+++ b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/read_options_pb2.py
@@ -0,0 +1,123 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/bigquery/storage_v1beta1/proto/read_options.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/bigquery/storage_v1beta1/proto/read_options.proto",
+ package="google.cloud.bigquery.storage.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n)com.google.cloud.bigquery.storage.v1beta1ZLgoogle.golang.org/genproto/googleapis/cloud/bigquery/storage/v1beta1;storage"
+ ),
+ serialized_pb=_b(
+ '\n>google/cloud/bigquery/storage_v1beta1/proto/read_options.proto\x12%google.cloud.bigquery.storage.v1beta1"D\n\x10TableReadOptions\x12\x17\n\x0fselected_fields\x18\x01 \x03(\t\x12\x17\n\x0frow_restriction\x18\x02 \x01(\tBy\n)com.google.cloud.bigquery.storage.v1beta1ZLgoogle.golang.org/genproto/googleapis/cloud/bigquery/storage/v1beta1;storageb\x06proto3'
+ ),
+)
+
+
+_TABLEREADOPTIONS = _descriptor.Descriptor(
+ name="TableReadOptions",
+ full_name="google.cloud.bigquery.storage.v1beta1.TableReadOptions",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="selected_fields",
+ full_name="google.cloud.bigquery.storage.v1beta1.TableReadOptions.selected_fields",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="row_restriction",
+ full_name="google.cloud.bigquery.storage.v1beta1.TableReadOptions.row_restriction",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=105,
+ serialized_end=173,
+)
+
+DESCRIPTOR.message_types_by_name["TableReadOptions"] = _TABLEREADOPTIONS
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+TableReadOptions = _reflection.GeneratedProtocolMessageType(
+ "TableReadOptions",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TABLEREADOPTIONS,
+ __module__="google.cloud.bigquery.storage_v1beta1.proto.read_options_pb2",
+ __doc__="""Options dictating how we read a table.
+
+
+ Attributes:
+ selected_fields:
+ Optional. Names of the fields in the table that should be
+ read. If empty, all fields will be read. If the specified
+ field is a nested field, all the sub-fields in the field will
+ be selected. The output field order is unrelated to the order
+ of fields in selected\_fields.
+ row_restriction:
+ Optional. SQL text filtering statement, similar to a WHERE
+ clause in a query. Currently, only a single predicate that is
+ a comparison between a column and a constant value is
+ supported. Aggregates are not supported. Examples:
+ "int\_field > 5" "date\_field = CAST('2014-9-27' as DATE)"
+ "nullable\_field is not NULL" "st\_equals(geo\_field,
+ st\_geofromtext("POINT(2, 2)"))" "numeric\_field BETWEEN 1.0
+ AND 5.0"
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.storage.v1beta1.TableReadOptions)
+ ),
+)
+_sym_db.RegisterMessage(TableReadOptions)
+
+
+DESCRIPTOR._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/read_options_pb2_grpc.py b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/read_options_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/read_options_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/storage.proto b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/storage.proto
new file mode 100644
index 000000000000..fa2de616d4f7
--- /dev/null
+++ b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/storage.proto
@@ -0,0 +1,362 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.bigquery.storage.v1beta1;
+
+import "google/api/annotations.proto";
+import "google/cloud/bigquery/storage/v1beta1/arrow.proto";
+import "google/cloud/bigquery/storage/v1beta1/avro.proto";
+import "google/cloud/bigquery/storage/v1beta1/read_options.proto";
+import "google/cloud/bigquery/storage/v1beta1/table_reference.proto";
+import "google/protobuf/empty.proto";
+import "google/protobuf/timestamp.proto";
+import "google/api/client.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/bigquery/storage/v1beta1;storage";
+option java_package = "com.google.cloud.bigquery.storage.v1beta1";
+
+// BigQuery storage API.
+//
+// The BigQuery storage API can be used to read data stored in BigQuery.
+service BigQueryStorage {
+ option (google.api.default_host) = "bigquerystorage.googleapis.com";
+ option (google.api.oauth_scopes) =
+ "https://www.googleapis.com/auth/bigquery,"
+ "https://www.googleapis.com/auth/cloud-platform";
+
+ // Creates a new read session. A read session divides the contents of a
+ // BigQuery table into one or more streams, which can then be used to read
+ // data from the table. The read session also specifies properties of the
+ // data to be read, such as a list of columns or a push-down filter describing
+ // the rows to be returned.
+ //
+ // A particular row can be read by at most one stream. When the caller has
+ // reached the end of each stream in the session, then all the data in the
+ // table has been read.
+ //
+ // Read sessions automatically expire 24 hours after they are created and do
+ // not require manual clean-up by the caller.
+ rpc CreateReadSession(CreateReadSessionRequest) returns (ReadSession) {
+ option (google.api.http) = {
+ post: "/v1beta1/{table_reference.project_id=projects/*}"
+ body: "*"
+ additional_bindings {
+ post: "/v1beta1/{table_reference.dataset_id=projects/*/datasets/*}"
+ body: "*"
+ }
+ };
+ }
+
+ // Reads rows from the table in the format prescribed by the read session.
+ // Each response contains one or more table rows, up to a maximum of 10 MiB
+ // per response; read requests which attempt to read individual rows larger
+ // than this will fail.
+ //
+ // Each request also returns a set of stream statistics reflecting the
+ // estimated total number of rows in the read stream. This number is computed
+ // based on the total table size and the number of active streams in the read
+ // session, and may change as other streams continue to read data.
+ rpc ReadRows(ReadRowsRequest) returns (stream ReadRowsResponse) {
+ option (google.api.http) = {
+ get: "/v1beta1/{read_position.stream.name=projects/*/streams/*}"
+ };
+ }
+
+ // Creates additional streams for a ReadSession. This API can be used to
+ // dynamically adjust the parallelism of a batch processing task upwards by
+ // adding additional workers.
+ rpc BatchCreateReadSessionStreams(BatchCreateReadSessionStreamsRequest) returns (BatchCreateReadSessionStreamsResponse) {
+ option (google.api.http) = {
+ post: "/v1beta1/{session.name=projects/*/sessions/*}"
+ body: "*"
+ };
+ }
+
+ // Triggers the graceful termination of a single stream in a ReadSession. This
+ // API can be used to dynamically adjust the parallelism of a batch processing
+ // task downwards without losing data.
+ //
+ // This API does not delete the stream -- it remains visible in the
+ // ReadSession, and any data processed by the stream is not released to other
+ // streams. However, no additional data will be assigned to the stream once
+ // this call completes. Callers must continue reading data on the stream until
+ // the end of the stream is reached so that data which has already been
+ // assigned to the stream will be processed.
+ //
+ // This method will return an error if there are no other live streams
+ // in the Session, or if SplitReadStream() has been called on the given
+ // Stream.
+ rpc FinalizeStream(FinalizeStreamRequest) returns (google.protobuf.Empty) {
+ option (google.api.http) = {
+ post: "/v1beta1/{stream.name=projects/*/streams/*}"
+ body: "*"
+ };
+ }
+
+ // Splits a given read stream into two Streams. These streams are referred to
+ // as the primary and the residual of the split. The original stream can still
+ // be read from in the same manner as before. Both of the returned streams can
+ // also be read from, and the total rows return by both child streams will be
+ // the same as the rows read from the original stream.
+ //
+ // Moreover, the two child streams will be allocated back to back in the
+ // original Stream. Concretely, it is guaranteed that for streams Original,
+ // Primary, and Residual, that Original[0-j] = Primary[0-j] and
+ // Original[j-n] = Residual[0-m] once the streams have been read to
+ // completion.
+ //
+ // This method is guaranteed to be idempotent.
+ rpc SplitReadStream(SplitReadStreamRequest) returns (SplitReadStreamResponse) {
+ option (google.api.http) = {
+ get: "/v1beta1/{original_stream.name=projects/*/streams/*}"
+ };
+ }
+}
+
+// Information about a single data stream within a read session.
+message Stream {
+ // Name of the stream, in the form
+ // `projects/{project_id}/locations/{location}/streams/{stream_id}`.
+ string name = 1;
+}
+
+// Expresses a point within a given stream using an offset position.
+message StreamPosition {
+ // Identifier for a given Stream.
+ Stream stream = 1;
+
+ // Position in the stream.
+ int64 offset = 2;
+}
+
+// Information returned from a `CreateReadSession` request.
+message ReadSession {
+ // Unique identifier for the session, in the form
+ // `projects/{project_id}/locations/{location}/sessions/{session_id}`.
+ string name = 1;
+
+ // Time at which the session becomes invalid. After this time, subsequent
+ // requests to read this Session will return errors.
+ google.protobuf.Timestamp expire_time = 2;
+
+ // The schema for the read. If read_options.selected_fields is set, the
+ // schema may be different from the table schema as it will only contain
+ // the selected fields.
+ oneof schema {
+ // Avro schema.
+ AvroSchema avro_schema = 5;
+
+ // Arrow schema.
+ ArrowSchema arrow_schema = 6;
+ }
+
+ // Streams associated with this session.
+ repeated Stream streams = 4;
+
+ // Table that this ReadSession is reading from.
+ TableReference table_reference = 7;
+
+ // Any modifiers which are applied when reading from the specified table.
+ TableModifiers table_modifiers = 8;
+
+ // The strategy to use for distributing data among the streams.
+ ShardingStrategy sharding_strategy = 9;
+}
+
+// Creates a new read session, which may include additional options such as
+// requested parallelism, projection filters and constraints.
+message CreateReadSessionRequest {
+ // Required. Reference to the table to read.
+ TableReference table_reference = 1;
+
+ // Required. String of the form `projects/{project_id}` indicating the
+ // project this ReadSession is associated with. This is the project that will
+ // be billed for usage.
+ string parent = 6;
+
+ // Optional. Any modifiers to the Table (e.g. snapshot timestamp).
+ TableModifiers table_modifiers = 2;
+
+ // Optional. Initial number of streams. If unset or 0, we will
+ // provide a value of streams so as to produce reasonable throughput. Must be
+ // non-negative. The number of streams may be lower than the requested number,
+ // depending on the amount parallelism that is reasonable for the table and
+ // the maximum amount of parallelism allowed by the system.
+ //
+ // Streams must be read starting from offset 0.
+ int32 requested_streams = 3;
+
+ // Optional. Read options for this session (e.g. column selection, filters).
+ TableReadOptions read_options = 4;
+
+ // Data output format. Currently default to Avro.
+ DataFormat format = 5;
+
+ // The strategy to use for distributing data among multiple streams. Currently
+ // defaults to liquid sharding.
+ ShardingStrategy sharding_strategy = 7;
+}
+
+// Data format for input or output data.
+enum DataFormat {
+ // Data format is unspecified.
+ DATA_FORMAT_UNSPECIFIED = 0;
+
+ // Avro is a standard open source row based file format.
+ // See https://avro.apache.org/ for more details.
+ AVRO = 1;
+
+ ARROW = 3;
+}
+
+// Strategy for distributing data among multiple streams in a read session.
+enum ShardingStrategy {
+ // Same as LIQUID.
+ SHARDING_STRATEGY_UNSPECIFIED = 0;
+
+ // Assigns data to each stream based on the client's read rate. The faster the
+ // client reads from a stream, the more data is assigned to the stream. In
+ // this strategy, it's possible to read all data from a single stream even if
+ // there are other streams present.
+ LIQUID = 1;
+
+ // Assigns data to each stream such that roughly the same number of rows can
+ // be read from each stream. Because the server-side unit for assigning data
+ // is collections of rows, the API does not guarantee that each stream will
+ // return the same number or rows. Additionally, the limits are enforced based
+ // on the number of pre-filtering rows, so some filters can lead to lopsided
+ // assignments.
+ BALANCED = 2;
+}
+
+// Requesting row data via `ReadRows` must provide Stream position information.
+message ReadRowsRequest {
+ // Required. Identifier of the position in the stream to start reading from.
+ // The offset requested must be less than the last row read from ReadRows.
+ // Requesting a larger offset is undefined.
+ StreamPosition read_position = 1;
+}
+
+// Progress information for a given Stream.
+message StreamStatus {
+ // Number of estimated rows in the current stream. May change over time as
+ // different readers in the stream progress at rates which are relatively fast
+ // or slow.
+ int64 estimated_row_count = 1;
+
+ // A value in the range [0.0, 1.0] that represents the fraction of rows
+ // assigned to this stream that have been processed by the server. In the
+ // presence of read filters, the server may process more rows than it returns,
+ // so this value reflects progress through the pre-filtering rows.
+ //
+ // This value is only populated for sessions created through the BALANCED
+ // sharding strategy.
+ float fraction_consumed = 2;
+
+ // Whether this stream can be split. For sessions that use the LIQUID sharding
+ // strategy, this value is always false. For BALANCED sessions, this value is
+ // false when enough data have been read such that no more splits are possible
+ // at that point or beyond. For small tables or streams that are the result of
+ // a chain of splits, this value may never be true.
+ bool is_splittable = 3;
+}
+
+// Information on if the current connection is being throttled.
+message ThrottleStatus {
+ // How much this connection is being throttled.
+ // 0 is no throttling, 100 is completely throttled.
+ int32 throttle_percent = 1;
+}
+
+// Response from calling `ReadRows` may include row data, progress and
+// throttling information.
+message ReadRowsResponse {
+ // Row data is returned in format specified during session creation.
+ oneof rows {
+ // Serialized row data in AVRO format.
+ AvroRows avro_rows = 3;
+
+ // Serialized row data in Arrow RecordBatch format.
+ ArrowRecordBatch arrow_record_batch = 4;
+ }
+
+ // Number of serialized rows in the rows block. This value is recorded here,
+ // in addition to the row_count values in the output-specific messages in
+ // `rows`, so that code which needs to record progress through the stream can
+ // do so in an output format-independent way.
+ int64 row_count = 6;
+
+ // Estimated stream statistics.
+ StreamStatus status = 2;
+
+ // Throttling status. If unset, the latest response still describes
+ // the current throttling status.
+ ThrottleStatus throttle_status = 5;
+}
+
+// Information needed to request additional streams for an established read
+// session.
+message BatchCreateReadSessionStreamsRequest {
+ // Required. Must be a non-expired session obtained from a call to
+ // CreateReadSession. Only the name field needs to be set.
+ ReadSession session = 1;
+
+ // Required. Number of new streams requested. Must be positive.
+ // Number of added streams may be less than this, see CreateReadSessionRequest
+ // for more information.
+ int32 requested_streams = 2;
+}
+
+// The response from `BatchCreateReadSessionStreams` returns the stream
+// identifiers for the newly created streams.
+message BatchCreateReadSessionStreamsResponse {
+ // Newly added streams.
+ repeated Stream streams = 1;
+}
+
+// Request information for invoking `FinalizeStream`.
+message FinalizeStreamRequest {
+ // Stream to finalize.
+ Stream stream = 2;
+}
+
+// Request information for `SplitReadStream`.
+message SplitReadStreamRequest {
+ // Stream to split.
+ Stream original_stream = 1;
+
+ // A value in the range (0.0, 1.0) that specifies the fractional point at
+ // which the original stream should be split. The actual split point is
+ // evaluated on pre-filtered rows, so if a filter is provided, then there is
+ // no guarantee that the division of the rows between the new child streams
+ // will be proportional to this fractional value. Additionally, because the
+ // server-side unit for assigning data is collections of rows, this fraction
+ // will always map to to a data storage boundary on the server side.
+ float fraction = 2;
+}
+
+// Response from `SplitReadStream`.
+message SplitReadStreamResponse {
+ // Primary stream, which contains the beginning portion of
+ // |original_stream|. An empty value indicates that the original stream can no
+ // longer be split.
+ Stream primary_stream = 1;
+
+ // Remainder stream, which contains the tail of |original_stream|. An empty
+ // value indicates that the original stream can no longer be split.
+ Stream remainder_stream = 2;
+}
diff --git a/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/storage_pb2.py b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/storage_pb2.py
new file mode 100644
index 000000000000..ec546f3f9e81
--- /dev/null
+++ b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/storage_pb2.py
@@ -0,0 +1,1589 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/bigquery/storage_v1beta1/proto/storage.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf.internal import enum_type_wrapper
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+from google.cloud.bigquery_storage_v1beta1.proto import (
+ arrow_pb2 as google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_arrow__pb2,
+)
+from google.cloud.bigquery_storage_v1beta1.proto import (
+ avro_pb2 as google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_avro__pb2,
+)
+from google.cloud.bigquery_storage_v1beta1.proto import (
+ read_options_pb2 as google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_read__options__pb2,
+)
+from google.cloud.bigquery_storage_v1beta1.proto import (
+ table_reference_pb2 as google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_table__reference__pb2,
+)
+from google.protobuf import empty_pb2 as google_dot_protobuf_dot_empty__pb2
+from google.protobuf import timestamp_pb2 as google_dot_protobuf_dot_timestamp__pb2
+from google.api import client_pb2 as google_dot_api_dot_client__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/bigquery/storage_v1beta1/proto/storage.proto",
+ package="google.cloud.bigquery.storage.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n)com.google.cloud.bigquery.storage.v1beta1ZLgoogle.golang.org/genproto/googleapis/cloud/bigquery/storage/v1beta1;storage"
+ ),
+ serialized_pb=_b(
+ '\n9google/cloud/bigquery/storage_v1beta1/proto/storage.proto\x12%google.cloud.bigquery.storage.v1beta1\x1a\x1cgoogle/api/annotations.proto\x1a\x37google/cloud/bigquery/storage_v1beta1/proto/arrow.proto\x1a\x36google/cloud/bigquery/storage_v1beta1/proto/avro.proto\x1a>google/cloud/bigquery/storage_v1beta1/proto/read_options.proto\x1a\x41google/cloud/bigquery/storage_v1beta1/proto/table_reference.proto\x1a\x1bgoogle/protobuf/empty.proto\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x17google/api/client.proto"\x16\n\x06Stream\x12\x0c\n\x04name\x18\x01 \x01(\t"_\n\x0eStreamPosition\x12=\n\x06stream\x18\x01 \x01(\x0b\x32-.google.cloud.bigquery.storage.v1beta1.Stream\x12\x0e\n\x06offset\x18\x02 \x01(\x03"\xa0\x04\n\x0bReadSession\x12\x0c\n\x04name\x18\x01 \x01(\t\x12/\n\x0b\x65xpire_time\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12H\n\x0b\x61vro_schema\x18\x05 \x01(\x0b\x32\x31.google.cloud.bigquery.storage.v1beta1.AvroSchemaH\x00\x12J\n\x0c\x61rrow_schema\x18\x06 \x01(\x0b\x32\x32.google.cloud.bigquery.storage.v1beta1.ArrowSchemaH\x00\x12>\n\x07streams\x18\x04 \x03(\x0b\x32-.google.cloud.bigquery.storage.v1beta1.Stream\x12N\n\x0ftable_reference\x18\x07 \x01(\x0b\x32\x35.google.cloud.bigquery.storage.v1beta1.TableReference\x12N\n\x0ftable_modifiers\x18\x08 \x01(\x0b\x32\x35.google.cloud.bigquery.storage.v1beta1.TableModifiers\x12R\n\x11sharding_strategy\x18\t \x01(\x0e\x32\x37.google.cloud.bigquery.storage.v1beta1.ShardingStrategyB\x08\n\x06schema"\xcb\x03\n\x18\x43reateReadSessionRequest\x12N\n\x0ftable_reference\x18\x01 \x01(\x0b\x32\x35.google.cloud.bigquery.storage.v1beta1.TableReference\x12\x0e\n\x06parent\x18\x06 \x01(\t\x12N\n\x0ftable_modifiers\x18\x02 \x01(\x0b\x32\x35.google.cloud.bigquery.storage.v1beta1.TableModifiers\x12\x19\n\x11requested_streams\x18\x03 \x01(\x05\x12M\n\x0cread_options\x18\x04 \x01(\x0b\x32\x37.google.cloud.bigquery.storage.v1beta1.TableReadOptions\x12\x41\n\x06\x66ormat\x18\x05 \x01(\x0e\x32\x31.google.cloud.bigquery.storage.v1beta1.DataFormat\x12R\n\x11sharding_strategy\x18\x07 \x01(\x0e\x32\x37.google.cloud.bigquery.storage.v1beta1.ShardingStrategy"_\n\x0fReadRowsRequest\x12L\n\rread_position\x18\x01 \x01(\x0b\x32\x35.google.cloud.bigquery.storage.v1beta1.StreamPosition"]\n\x0cStreamStatus\x12\x1b\n\x13\x65stimated_row_count\x18\x01 \x01(\x03\x12\x19\n\x11\x66raction_consumed\x18\x02 \x01(\x02\x12\x15\n\ris_splittable\x18\x03 \x01(\x08"*\n\x0eThrottleStatus\x12\x18\n\x10throttle_percent\x18\x01 \x01(\x05"\xdf\x02\n\x10ReadRowsResponse\x12\x44\n\tavro_rows\x18\x03 \x01(\x0b\x32/.google.cloud.bigquery.storage.v1beta1.AvroRowsH\x00\x12U\n\x12\x61rrow_record_batch\x18\x04 \x01(\x0b\x32\x37.google.cloud.bigquery.storage.v1beta1.ArrowRecordBatchH\x00\x12\x11\n\trow_count\x18\x06 \x01(\x03\x12\x43\n\x06status\x18\x02 \x01(\x0b\x32\x33.google.cloud.bigquery.storage.v1beta1.StreamStatus\x12N\n\x0fthrottle_status\x18\x05 \x01(\x0b\x32\x35.google.cloud.bigquery.storage.v1beta1.ThrottleStatusB\x06\n\x04rows"\x86\x01\n$BatchCreateReadSessionStreamsRequest\x12\x43\n\x07session\x18\x01 \x01(\x0b\x32\x32.google.cloud.bigquery.storage.v1beta1.ReadSession\x12\x19\n\x11requested_streams\x18\x02 \x01(\x05"g\n%BatchCreateReadSessionStreamsResponse\x12>\n\x07streams\x18\x01 \x03(\x0b\x32-.google.cloud.bigquery.storage.v1beta1.Stream"V\n\x15\x46inalizeStreamRequest\x12=\n\x06stream\x18\x02 \x01(\x0b\x32-.google.cloud.bigquery.storage.v1beta1.Stream"r\n\x16SplitReadStreamRequest\x12\x46\n\x0foriginal_stream\x18\x01 \x01(\x0b\x32-.google.cloud.bigquery.storage.v1beta1.Stream\x12\x10\n\x08\x66raction\x18\x02 \x01(\x02"\xa9\x01\n\x17SplitReadStreamResponse\x12\x45\n\x0eprimary_stream\x18\x01 \x01(\x0b\x32-.google.cloud.bigquery.storage.v1beta1.Stream\x12G\n\x10remainder_stream\x18\x02 \x01(\x0b\x32-.google.cloud.bigquery.storage.v1beta1.Stream*>\n\nDataFormat\x12\x1b\n\x17\x44\x41TA_FORMAT_UNSPECIFIED\x10\x00\x12\x08\n\x04\x41VRO\x10\x01\x12\t\n\x05\x41RROW\x10\x03*O\n\x10ShardingStrategy\x12!\n\x1dSHARDING_STRATEGY_UNSPECIFIED\x10\x00\x12\n\n\x06LIQUID\x10\x01\x12\x0c\n\x08\x42\x41LANCED\x10\x02\x32\xc4\t\n\x0f\x42igQueryStorage\x12\x87\x02\n\x11\x43reateReadSession\x12?.google.cloud.bigquery.storage.v1beta1.CreateReadSessionRequest\x1a\x32.google.cloud.bigquery.storage.v1beta1.ReadSession"}\x82\xd3\xe4\x93\x02w"0/v1beta1/{table_reference.project_id=projects/*}:\x01*Z@";/v1beta1/{table_reference.dataset_id=projects/*/datasets/*}:\x01*\x12\xc0\x01\n\x08ReadRows\x12\x36.google.cloud.bigquery.storage.v1beta1.ReadRowsRequest\x1a\x37.google.cloud.bigquery.storage.v1beta1.ReadRowsResponse"A\x82\xd3\xe4\x93\x02;\x12\x39/v1beta1/{read_position.stream.name=projects/*/streams/*}0\x01\x12\xf4\x01\n\x1d\x42\x61tchCreateReadSessionStreams\x12K.google.cloud.bigquery.storage.v1beta1.BatchCreateReadSessionStreamsRequest\x1aL.google.cloud.bigquery.storage.v1beta1.BatchCreateReadSessionStreamsResponse"8\x82\xd3\xe4\x93\x02\x32"-/v1beta1/{session.name=projects/*/sessions/*}:\x01*\x12\x9e\x01\n\x0e\x46inalizeStream\x12<.google.cloud.bigquery.storage.v1beta1.FinalizeStreamRequest\x1a\x16.google.protobuf.Empty"6\x82\xd3\xe4\x93\x02\x30"+/v1beta1/{stream.name=projects/*/streams/*}:\x01*\x12\xce\x01\n\x0fSplitReadStream\x12=.google.cloud.bigquery.storage.v1beta1.SplitReadStreamRequest\x1a>.google.cloud.bigquery.storage.v1beta1.SplitReadStreamResponse"<\x82\xd3\xe4\x93\x02\x36\x12\x34/v1beta1/{original_stream.name=projects/*/streams/*}\x1a{\xca\x41\x1e\x62igquerystorage.googleapis.com\xd2\x41Whttps://www.googleapis.com/auth/bigquery,https://www.googleapis.com/auth/cloud-platformBy\n)com.google.cloud.bigquery.storage.v1beta1ZLgoogle.golang.org/genproto/googleapis/cloud/bigquery/storage/v1beta1;storageb\x06proto3'
+ ),
+ dependencies=[
+ google_dot_api_dot_annotations__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_arrow__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_avro__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_read__options__pb2.DESCRIPTOR,
+ google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_table__reference__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_empty__pb2.DESCRIPTOR,
+ google_dot_protobuf_dot_timestamp__pb2.DESCRIPTOR,
+ google_dot_api_dot_client__pb2.DESCRIPTOR,
+ ],
+)
+
+_DATAFORMAT = _descriptor.EnumDescriptor(
+ name="DataFormat",
+ full_name="google.cloud.bigquery.storage.v1beta1.DataFormat",
+ filename=None,
+ file=DESCRIPTOR,
+ values=[
+ _descriptor.EnumValueDescriptor(
+ name="DATA_FORMAT_UNSPECIFIED",
+ index=0,
+ number=0,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="AVRO", index=1, number=1, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="ARROW", index=2, number=3, serialized_options=None, type=None
+ ),
+ ],
+ containing_type=None,
+ serialized_options=None,
+ serialized_start=2799,
+ serialized_end=2861,
+)
+_sym_db.RegisterEnumDescriptor(_DATAFORMAT)
+
+DataFormat = enum_type_wrapper.EnumTypeWrapper(_DATAFORMAT)
+_SHARDINGSTRATEGY = _descriptor.EnumDescriptor(
+ name="ShardingStrategy",
+ full_name="google.cloud.bigquery.storage.v1beta1.ShardingStrategy",
+ filename=None,
+ file=DESCRIPTOR,
+ values=[
+ _descriptor.EnumValueDescriptor(
+ name="SHARDING_STRATEGY_UNSPECIFIED",
+ index=0,
+ number=0,
+ serialized_options=None,
+ type=None,
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="LIQUID", index=1, number=1, serialized_options=None, type=None
+ ),
+ _descriptor.EnumValueDescriptor(
+ name="BALANCED", index=2, number=2, serialized_options=None, type=None
+ ),
+ ],
+ containing_type=None,
+ serialized_options=None,
+ serialized_start=2863,
+ serialized_end=2942,
+)
+_sym_db.RegisterEnumDescriptor(_SHARDINGSTRATEGY)
+
+ShardingStrategy = enum_type_wrapper.EnumTypeWrapper(_SHARDINGSTRATEGY)
+DATA_FORMAT_UNSPECIFIED = 0
+AVRO = 1
+ARROW = 3
+SHARDING_STRATEGY_UNSPECIFIED = 0
+LIQUID = 1
+BALANCED = 2
+
+
+_STREAM = _descriptor.Descriptor(
+ name="Stream",
+ full_name="google.cloud.bigquery.storage.v1beta1.Stream",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.bigquery.storage.v1beta1.Stream.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=461,
+ serialized_end=483,
+)
+
+
+_STREAMPOSITION = _descriptor.Descriptor(
+ name="StreamPosition",
+ full_name="google.cloud.bigquery.storage.v1beta1.StreamPosition",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="stream",
+ full_name="google.cloud.bigquery.storage.v1beta1.StreamPosition.stream",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="offset",
+ full_name="google.cloud.bigquery.storage.v1beta1.StreamPosition.offset",
+ index=1,
+ number=2,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=485,
+ serialized_end=580,
+)
+
+
+_READSESSION = _descriptor.Descriptor(
+ name="ReadSession",
+ full_name="google.cloud.bigquery.storage.v1beta1.ReadSession",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="name",
+ full_name="google.cloud.bigquery.storage.v1beta1.ReadSession.name",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="expire_time",
+ full_name="google.cloud.bigquery.storage.v1beta1.ReadSession.expire_time",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="avro_schema",
+ full_name="google.cloud.bigquery.storage.v1beta1.ReadSession.avro_schema",
+ index=2,
+ number=5,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="arrow_schema",
+ full_name="google.cloud.bigquery.storage.v1beta1.ReadSession.arrow_schema",
+ index=3,
+ number=6,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="streams",
+ full_name="google.cloud.bigquery.storage.v1beta1.ReadSession.streams",
+ index=4,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="table_reference",
+ full_name="google.cloud.bigquery.storage.v1beta1.ReadSession.table_reference",
+ index=5,
+ number=7,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="table_modifiers",
+ full_name="google.cloud.bigquery.storage.v1beta1.ReadSession.table_modifiers",
+ index=6,
+ number=8,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="sharding_strategy",
+ full_name="google.cloud.bigquery.storage.v1beta1.ReadSession.sharding_strategy",
+ index=7,
+ number=9,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="schema",
+ full_name="google.cloud.bigquery.storage.v1beta1.ReadSession.schema",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=583,
+ serialized_end=1127,
+)
+
+
+_CREATEREADSESSIONREQUEST = _descriptor.Descriptor(
+ name="CreateReadSessionRequest",
+ full_name="google.cloud.bigquery.storage.v1beta1.CreateReadSessionRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="table_reference",
+ full_name="google.cloud.bigquery.storage.v1beta1.CreateReadSessionRequest.table_reference",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="parent",
+ full_name="google.cloud.bigquery.storage.v1beta1.CreateReadSessionRequest.parent",
+ index=1,
+ number=6,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="table_modifiers",
+ full_name="google.cloud.bigquery.storage.v1beta1.CreateReadSessionRequest.table_modifiers",
+ index=2,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="requested_streams",
+ full_name="google.cloud.bigquery.storage.v1beta1.CreateReadSessionRequest.requested_streams",
+ index=3,
+ number=3,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="read_options",
+ full_name="google.cloud.bigquery.storage.v1beta1.CreateReadSessionRequest.read_options",
+ index=4,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="format",
+ full_name="google.cloud.bigquery.storage.v1beta1.CreateReadSessionRequest.format",
+ index=5,
+ number=5,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="sharding_strategy",
+ full_name="google.cloud.bigquery.storage.v1beta1.CreateReadSessionRequest.sharding_strategy",
+ index=6,
+ number=7,
+ type=14,
+ cpp_type=8,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1130,
+ serialized_end=1589,
+)
+
+
+_READROWSREQUEST = _descriptor.Descriptor(
+ name="ReadRowsRequest",
+ full_name="google.cloud.bigquery.storage.v1beta1.ReadRowsRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="read_position",
+ full_name="google.cloud.bigquery.storage.v1beta1.ReadRowsRequest.read_position",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1591,
+ serialized_end=1686,
+)
+
+
+_STREAMSTATUS = _descriptor.Descriptor(
+ name="StreamStatus",
+ full_name="google.cloud.bigquery.storage.v1beta1.StreamStatus",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="estimated_row_count",
+ full_name="google.cloud.bigquery.storage.v1beta1.StreamStatus.estimated_row_count",
+ index=0,
+ number=1,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="fraction_consumed",
+ full_name="google.cloud.bigquery.storage.v1beta1.StreamStatus.fraction_consumed",
+ index=1,
+ number=2,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="is_splittable",
+ full_name="google.cloud.bigquery.storage.v1beta1.StreamStatus.is_splittable",
+ index=2,
+ number=3,
+ type=8,
+ cpp_type=7,
+ label=1,
+ has_default_value=False,
+ default_value=False,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1688,
+ serialized_end=1781,
+)
+
+
+_THROTTLESTATUS = _descriptor.Descriptor(
+ name="ThrottleStatus",
+ full_name="google.cloud.bigquery.storage.v1beta1.ThrottleStatus",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="throttle_percent",
+ full_name="google.cloud.bigquery.storage.v1beta1.ThrottleStatus.throttle_percent",
+ index=0,
+ number=1,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=1783,
+ serialized_end=1825,
+)
+
+
+_READROWSRESPONSE = _descriptor.Descriptor(
+ name="ReadRowsResponse",
+ full_name="google.cloud.bigquery.storage.v1beta1.ReadRowsResponse",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="avro_rows",
+ full_name="google.cloud.bigquery.storage.v1beta1.ReadRowsResponse.avro_rows",
+ index=0,
+ number=3,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="arrow_record_batch",
+ full_name="google.cloud.bigquery.storage.v1beta1.ReadRowsResponse.arrow_record_batch",
+ index=1,
+ number=4,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="row_count",
+ full_name="google.cloud.bigquery.storage.v1beta1.ReadRowsResponse.row_count",
+ index=2,
+ number=6,
+ type=3,
+ cpp_type=2,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="status",
+ full_name="google.cloud.bigquery.storage.v1beta1.ReadRowsResponse.status",
+ index=3,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="throttle_status",
+ full_name="google.cloud.bigquery.storage.v1beta1.ReadRowsResponse.throttle_status",
+ index=4,
+ number=5,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[
+ _descriptor.OneofDescriptor(
+ name="rows",
+ full_name="google.cloud.bigquery.storage.v1beta1.ReadRowsResponse.rows",
+ index=0,
+ containing_type=None,
+ fields=[],
+ )
+ ],
+ serialized_start=1828,
+ serialized_end=2179,
+)
+
+
+_BATCHCREATEREADSESSIONSTREAMSREQUEST = _descriptor.Descriptor(
+ name="BatchCreateReadSessionStreamsRequest",
+ full_name="google.cloud.bigquery.storage.v1beta1.BatchCreateReadSessionStreamsRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="session",
+ full_name="google.cloud.bigquery.storage.v1beta1.BatchCreateReadSessionStreamsRequest.session",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="requested_streams",
+ full_name="google.cloud.bigquery.storage.v1beta1.BatchCreateReadSessionStreamsRequest.requested_streams",
+ index=1,
+ number=2,
+ type=5,
+ cpp_type=1,
+ label=1,
+ has_default_value=False,
+ default_value=0,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2182,
+ serialized_end=2316,
+)
+
+
+_BATCHCREATEREADSESSIONSTREAMSRESPONSE = _descriptor.Descriptor(
+ name="BatchCreateReadSessionStreamsResponse",
+ full_name="google.cloud.bigquery.storage.v1beta1.BatchCreateReadSessionStreamsResponse",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="streams",
+ full_name="google.cloud.bigquery.storage.v1beta1.BatchCreateReadSessionStreamsResponse.streams",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=3,
+ has_default_value=False,
+ default_value=[],
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2318,
+ serialized_end=2421,
+)
+
+
+_FINALIZESTREAMREQUEST = _descriptor.Descriptor(
+ name="FinalizeStreamRequest",
+ full_name="google.cloud.bigquery.storage.v1beta1.FinalizeStreamRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="stream",
+ full_name="google.cloud.bigquery.storage.v1beta1.FinalizeStreamRequest.stream",
+ index=0,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2423,
+ serialized_end=2509,
+)
+
+
+_SPLITREADSTREAMREQUEST = _descriptor.Descriptor(
+ name="SplitReadStreamRequest",
+ full_name="google.cloud.bigquery.storage.v1beta1.SplitReadStreamRequest",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="original_stream",
+ full_name="google.cloud.bigquery.storage.v1beta1.SplitReadStreamRequest.original_stream",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="fraction",
+ full_name="google.cloud.bigquery.storage.v1beta1.SplitReadStreamRequest.fraction",
+ index=1,
+ number=2,
+ type=2,
+ cpp_type=6,
+ label=1,
+ has_default_value=False,
+ default_value=float(0),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2511,
+ serialized_end=2625,
+)
+
+
+_SPLITREADSTREAMRESPONSE = _descriptor.Descriptor(
+ name="SplitReadStreamResponse",
+ full_name="google.cloud.bigquery.storage.v1beta1.SplitReadStreamResponse",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="primary_stream",
+ full_name="google.cloud.bigquery.storage.v1beta1.SplitReadStreamResponse.primary_stream",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="remainder_stream",
+ full_name="google.cloud.bigquery.storage.v1beta1.SplitReadStreamResponse.remainder_stream",
+ index=1,
+ number=2,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=2628,
+ serialized_end=2797,
+)
+
+_STREAMPOSITION.fields_by_name["stream"].message_type = _STREAM
+_READSESSION.fields_by_name[
+ "expire_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+_READSESSION.fields_by_name[
+ "avro_schema"
+].message_type = (
+ google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_avro__pb2._AVROSCHEMA
+)
+_READSESSION.fields_by_name[
+ "arrow_schema"
+].message_type = (
+ google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_arrow__pb2._ARROWSCHEMA
+)
+_READSESSION.fields_by_name["streams"].message_type = _STREAM
+_READSESSION.fields_by_name[
+ "table_reference"
+].message_type = (
+ google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_table__reference__pb2._TABLEREFERENCE
+)
+_READSESSION.fields_by_name[
+ "table_modifiers"
+].message_type = (
+ google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_table__reference__pb2._TABLEMODIFIERS
+)
+_READSESSION.fields_by_name["sharding_strategy"].enum_type = _SHARDINGSTRATEGY
+_READSESSION.oneofs_by_name["schema"].fields.append(
+ _READSESSION.fields_by_name["avro_schema"]
+)
+_READSESSION.fields_by_name[
+ "avro_schema"
+].containing_oneof = _READSESSION.oneofs_by_name["schema"]
+_READSESSION.oneofs_by_name["schema"].fields.append(
+ _READSESSION.fields_by_name["arrow_schema"]
+)
+_READSESSION.fields_by_name[
+ "arrow_schema"
+].containing_oneof = _READSESSION.oneofs_by_name["schema"]
+_CREATEREADSESSIONREQUEST.fields_by_name[
+ "table_reference"
+].message_type = (
+ google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_table__reference__pb2._TABLEREFERENCE
+)
+_CREATEREADSESSIONREQUEST.fields_by_name[
+ "table_modifiers"
+].message_type = (
+ google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_table__reference__pb2._TABLEMODIFIERS
+)
+_CREATEREADSESSIONREQUEST.fields_by_name[
+ "read_options"
+].message_type = (
+ google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_read__options__pb2._TABLEREADOPTIONS
+)
+_CREATEREADSESSIONREQUEST.fields_by_name["format"].enum_type = _DATAFORMAT
+_CREATEREADSESSIONREQUEST.fields_by_name[
+ "sharding_strategy"
+].enum_type = _SHARDINGSTRATEGY
+_READROWSREQUEST.fields_by_name["read_position"].message_type = _STREAMPOSITION
+_READROWSRESPONSE.fields_by_name[
+ "avro_rows"
+].message_type = (
+ google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_avro__pb2._AVROROWS
+)
+_READROWSRESPONSE.fields_by_name[
+ "arrow_record_batch"
+].message_type = (
+ google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_arrow__pb2._ARROWRECORDBATCH
+)
+_READROWSRESPONSE.fields_by_name["status"].message_type = _STREAMSTATUS
+_READROWSRESPONSE.fields_by_name["throttle_status"].message_type = _THROTTLESTATUS
+_READROWSRESPONSE.oneofs_by_name["rows"].fields.append(
+ _READROWSRESPONSE.fields_by_name["avro_rows"]
+)
+_READROWSRESPONSE.fields_by_name[
+ "avro_rows"
+].containing_oneof = _READROWSRESPONSE.oneofs_by_name["rows"]
+_READROWSRESPONSE.oneofs_by_name["rows"].fields.append(
+ _READROWSRESPONSE.fields_by_name["arrow_record_batch"]
+)
+_READROWSRESPONSE.fields_by_name[
+ "arrow_record_batch"
+].containing_oneof = _READROWSRESPONSE.oneofs_by_name["rows"]
+_BATCHCREATEREADSESSIONSTREAMSREQUEST.fields_by_name[
+ "session"
+].message_type = _READSESSION
+_BATCHCREATEREADSESSIONSTREAMSRESPONSE.fields_by_name["streams"].message_type = _STREAM
+_FINALIZESTREAMREQUEST.fields_by_name["stream"].message_type = _STREAM
+_SPLITREADSTREAMREQUEST.fields_by_name["original_stream"].message_type = _STREAM
+_SPLITREADSTREAMRESPONSE.fields_by_name["primary_stream"].message_type = _STREAM
+_SPLITREADSTREAMRESPONSE.fields_by_name["remainder_stream"].message_type = _STREAM
+DESCRIPTOR.message_types_by_name["Stream"] = _STREAM
+DESCRIPTOR.message_types_by_name["StreamPosition"] = _STREAMPOSITION
+DESCRIPTOR.message_types_by_name["ReadSession"] = _READSESSION
+DESCRIPTOR.message_types_by_name["CreateReadSessionRequest"] = _CREATEREADSESSIONREQUEST
+DESCRIPTOR.message_types_by_name["ReadRowsRequest"] = _READROWSREQUEST
+DESCRIPTOR.message_types_by_name["StreamStatus"] = _STREAMSTATUS
+DESCRIPTOR.message_types_by_name["ThrottleStatus"] = _THROTTLESTATUS
+DESCRIPTOR.message_types_by_name["ReadRowsResponse"] = _READROWSRESPONSE
+DESCRIPTOR.message_types_by_name[
+ "BatchCreateReadSessionStreamsRequest"
+] = _BATCHCREATEREADSESSIONSTREAMSREQUEST
+DESCRIPTOR.message_types_by_name[
+ "BatchCreateReadSessionStreamsResponse"
+] = _BATCHCREATEREADSESSIONSTREAMSRESPONSE
+DESCRIPTOR.message_types_by_name["FinalizeStreamRequest"] = _FINALIZESTREAMREQUEST
+DESCRIPTOR.message_types_by_name["SplitReadStreamRequest"] = _SPLITREADSTREAMREQUEST
+DESCRIPTOR.message_types_by_name["SplitReadStreamResponse"] = _SPLITREADSTREAMRESPONSE
+DESCRIPTOR.enum_types_by_name["DataFormat"] = _DATAFORMAT
+DESCRIPTOR.enum_types_by_name["ShardingStrategy"] = _SHARDINGSTRATEGY
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+Stream = _reflection.GeneratedProtocolMessageType(
+ "Stream",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_STREAM,
+ __module__="google.cloud.bigquery.storage_v1beta1.proto.storage_pb2",
+ __doc__="""Information about a single data stream within a read session.
+
+
+ Attributes:
+ name:
+ Name of the stream, in the form ``projects/{project_id}/locati
+ ons/{location}/streams/{stream_id}``.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.storage.v1beta1.Stream)
+ ),
+)
+_sym_db.RegisterMessage(Stream)
+
+StreamPosition = _reflection.GeneratedProtocolMessageType(
+ "StreamPosition",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_STREAMPOSITION,
+ __module__="google.cloud.bigquery.storage_v1beta1.proto.storage_pb2",
+ __doc__="""Expresses a point within a given stream using an offset position.
+
+
+ Attributes:
+ stream:
+ Identifier for a given Stream.
+ offset:
+ Position in the stream.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.storage.v1beta1.StreamPosition)
+ ),
+)
+_sym_db.RegisterMessage(StreamPosition)
+
+ReadSession = _reflection.GeneratedProtocolMessageType(
+ "ReadSession",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_READSESSION,
+ __module__="google.cloud.bigquery.storage_v1beta1.proto.storage_pb2",
+ __doc__="""Information returned from a ``CreateReadSession`` request.
+
+
+ Attributes:
+ name:
+ Unique identifier for the session, in the form ``projects/{pro
+ ject_id}/locations/{location}/sessions/{session_id}``.
+ expire_time:
+ Time at which the session becomes invalid. After this time,
+ subsequent requests to read this Session will return errors.
+ schema:
+ The schema for the read. If read\_options.selected\_fields is
+ set, the schema may be different from the table schema as it
+ will only contain the selected fields.
+ avro_schema:
+ Avro schema.
+ arrow_schema:
+ Arrow schema.
+ streams:
+ Streams associated with this session.
+ table_reference:
+ Table that this ReadSession is reading from.
+ table_modifiers:
+ Any modifiers which are applied when reading from the
+ specified table.
+ sharding_strategy:
+ The strategy to use for distributing data among the streams.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.storage.v1beta1.ReadSession)
+ ),
+)
+_sym_db.RegisterMessage(ReadSession)
+
+CreateReadSessionRequest = _reflection.GeneratedProtocolMessageType(
+ "CreateReadSessionRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_CREATEREADSESSIONREQUEST,
+ __module__="google.cloud.bigquery.storage_v1beta1.proto.storage_pb2",
+ __doc__="""Creates a new read session, which may include additional options such as
+ requested parallelism, projection filters and constraints.
+
+
+ Attributes:
+ table_reference:
+ Required. Reference to the table to read.
+ parent:
+ Required. String of the form ``projects/{project_id}``
+ indicating the project this ReadSession is associated with.
+ This is the project that will be billed for usage.
+ table_modifiers:
+ Optional. Any modifiers to the Table (e.g. snapshot
+ timestamp).
+ requested_streams:
+ Optional. Initial number of streams. If unset or 0, we will
+ provide a value of streams so as to produce reasonable
+ throughput. Must be non-negative. The number of streams may be
+ lower than the requested number, depending on the amount
+ parallelism that is reasonable for the table and the maximum
+ amount of parallelism allowed by the system. Streams must be
+ read starting from offset 0.
+ read_options:
+ Optional. Read options for this session (e.g. column
+ selection, filters).
+ format:
+ Data output format. Currently default to Avro.
+ sharding_strategy:
+ The strategy to use for distributing data among multiple
+ streams. Currently defaults to liquid sharding.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.storage.v1beta1.CreateReadSessionRequest)
+ ),
+)
+_sym_db.RegisterMessage(CreateReadSessionRequest)
+
+ReadRowsRequest = _reflection.GeneratedProtocolMessageType(
+ "ReadRowsRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_READROWSREQUEST,
+ __module__="google.cloud.bigquery.storage_v1beta1.proto.storage_pb2",
+ __doc__="""Requesting row data via ``ReadRows`` must provide Stream position
+ information.
+
+
+ Attributes:
+ read_position:
+ Required. Identifier of the position in the stream to start
+ reading from. The offset requested must be less than the last
+ row read from ReadRows. Requesting a larger offset is
+ undefined.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.storage.v1beta1.ReadRowsRequest)
+ ),
+)
+_sym_db.RegisterMessage(ReadRowsRequest)
+
+StreamStatus = _reflection.GeneratedProtocolMessageType(
+ "StreamStatus",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_STREAMSTATUS,
+ __module__="google.cloud.bigquery.storage_v1beta1.proto.storage_pb2",
+ __doc__="""Progress information for a given Stream.
+
+
+ Attributes:
+ estimated_row_count:
+ Number of estimated rows in the current stream. May change
+ over time as different readers in the stream progress at rates
+ which are relatively fast or slow.
+ fraction_consumed:
+ A value in the range [0.0, 1.0] that represents the fraction
+ of rows assigned to this stream that have been processed by
+ the server. In the presence of read filters, the server may
+ process more rows than it returns, so this value reflects
+ progress through the pre-filtering rows. This value is only
+ populated for sessions created through the BALANCED sharding
+ strategy.
+ is_splittable:
+ Whether this stream can be split. For sessions that use the
+ LIQUID sharding strategy, this value is always false. For
+ BALANCED sessions, this value is false when enough data have
+ been read such that no more splits are possible at that point
+ or beyond. For small tables or streams that are the result of
+ a chain of splits, this value may never be true.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.storage.v1beta1.StreamStatus)
+ ),
+)
+_sym_db.RegisterMessage(StreamStatus)
+
+ThrottleStatus = _reflection.GeneratedProtocolMessageType(
+ "ThrottleStatus",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_THROTTLESTATUS,
+ __module__="google.cloud.bigquery.storage_v1beta1.proto.storage_pb2",
+ __doc__="""Information on if the current connection is being throttled.
+
+
+ Attributes:
+ throttle_percent:
+ How much this connection is being throttled. 0 is no
+ throttling, 100 is completely throttled.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.storage.v1beta1.ThrottleStatus)
+ ),
+)
+_sym_db.RegisterMessage(ThrottleStatus)
+
+ReadRowsResponse = _reflection.GeneratedProtocolMessageType(
+ "ReadRowsResponse",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_READROWSRESPONSE,
+ __module__="google.cloud.bigquery.storage_v1beta1.proto.storage_pb2",
+ __doc__="""Response from calling ``ReadRows`` may include row data, progress and
+ throttling information.
+
+
+ Attributes:
+ rows:
+ Row data is returned in format specified during session
+ creation.
+ avro_rows:
+ Serialized row data in AVRO format.
+ arrow_record_batch:
+ Serialized row data in Arrow RecordBatch format.
+ row_count:
+ Number of serialized rows in the rows block. This value is
+ recorded here, in addition to the row\_count values in the
+ output-specific messages in ``rows``, so that code which needs
+ to record progress through the stream can do so in an output
+ format-independent way.
+ status:
+ Estimated stream statistics.
+ throttle_status:
+ Throttling status. If unset, the latest response still
+ describes the current throttling status.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.storage.v1beta1.ReadRowsResponse)
+ ),
+)
+_sym_db.RegisterMessage(ReadRowsResponse)
+
+BatchCreateReadSessionStreamsRequest = _reflection.GeneratedProtocolMessageType(
+ "BatchCreateReadSessionStreamsRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_BATCHCREATEREADSESSIONSTREAMSREQUEST,
+ __module__="google.cloud.bigquery.storage_v1beta1.proto.storage_pb2",
+ __doc__="""Information needed to request additional streams for an established read
+ session.
+
+
+ Attributes:
+ session:
+ Required. Must be a non-expired session obtained from a call
+ to CreateReadSession. Only the name field needs to be set.
+ requested_streams:
+ Required. Number of new streams requested. Must be positive.
+ Number of added streams may be less than this, see
+ CreateReadSessionRequest for more information.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.storage.v1beta1.BatchCreateReadSessionStreamsRequest)
+ ),
+)
+_sym_db.RegisterMessage(BatchCreateReadSessionStreamsRequest)
+
+BatchCreateReadSessionStreamsResponse = _reflection.GeneratedProtocolMessageType(
+ "BatchCreateReadSessionStreamsResponse",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_BATCHCREATEREADSESSIONSTREAMSRESPONSE,
+ __module__="google.cloud.bigquery.storage_v1beta1.proto.storage_pb2",
+ __doc__="""The response from ``BatchCreateReadSessionStreams`` returns the stream
+ identifiers for the newly created streams.
+
+
+ Attributes:
+ streams:
+ Newly added streams.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.storage.v1beta1.BatchCreateReadSessionStreamsResponse)
+ ),
+)
+_sym_db.RegisterMessage(BatchCreateReadSessionStreamsResponse)
+
+FinalizeStreamRequest = _reflection.GeneratedProtocolMessageType(
+ "FinalizeStreamRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_FINALIZESTREAMREQUEST,
+ __module__="google.cloud.bigquery.storage_v1beta1.proto.storage_pb2",
+ __doc__="""Request information for invoking ``FinalizeStream``.
+
+
+ Attributes:
+ stream:
+ Stream to finalize.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.storage.v1beta1.FinalizeStreamRequest)
+ ),
+)
+_sym_db.RegisterMessage(FinalizeStreamRequest)
+
+SplitReadStreamRequest = _reflection.GeneratedProtocolMessageType(
+ "SplitReadStreamRequest",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_SPLITREADSTREAMREQUEST,
+ __module__="google.cloud.bigquery.storage_v1beta1.proto.storage_pb2",
+ __doc__="""Request information for ``SplitReadStream``.
+
+
+ Attributes:
+ original_stream:
+ Stream to split.
+ fraction:
+ A value in the range (0.0, 1.0) that specifies the fractional
+ point at which the original stream should be split. The actual
+ split point is evaluated on pre-filtered rows, so if a filter
+ is provided, then there is no guarantee that the division of
+ the rows between the new child streams will be proportional to
+ this fractional value. Additionally, because the server-side
+ unit for assigning data is collections of rows, this fraction
+ will always map to to a data storage boundary on the server
+ side.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.storage.v1beta1.SplitReadStreamRequest)
+ ),
+)
+_sym_db.RegisterMessage(SplitReadStreamRequest)
+
+SplitReadStreamResponse = _reflection.GeneratedProtocolMessageType(
+ "SplitReadStreamResponse",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_SPLITREADSTREAMRESPONSE,
+ __module__="google.cloud.bigquery.storage_v1beta1.proto.storage_pb2",
+ __doc__="""Response from ``SplitReadStream``.
+
+
+ Attributes:
+ primary_stream:
+ Primary stream, which contains the beginning portion of
+ \|original\_stream\|. An empty value indicates that the
+ original stream can no longer be split.
+ remainder_stream:
+ Remainder stream, which contains the tail of
+ \|original\_stream\|. An empty value indicates that the
+ original stream can no longer be split.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.storage.v1beta1.SplitReadStreamResponse)
+ ),
+)
+_sym_db.RegisterMessage(SplitReadStreamResponse)
+
+
+DESCRIPTOR._options = None
+
+_BIGQUERYSTORAGE = _descriptor.ServiceDescriptor(
+ name="BigQueryStorage",
+ full_name="google.cloud.bigquery.storage.v1beta1.BigQueryStorage",
+ file=DESCRIPTOR,
+ index=0,
+ serialized_options=_b(
+ "\312A\036bigquerystorage.googleapis.com\322AWhttps://www.googleapis.com/auth/bigquery,https://www.googleapis.com/auth/cloud-platform"
+ ),
+ serialized_start=2945,
+ serialized_end=4165,
+ methods=[
+ _descriptor.MethodDescriptor(
+ name="CreateReadSession",
+ full_name="google.cloud.bigquery.storage.v1beta1.BigQueryStorage.CreateReadSession",
+ index=0,
+ containing_service=None,
+ input_type=_CREATEREADSESSIONREQUEST,
+ output_type=_READSESSION,
+ serialized_options=_b(
+ '\202\323\344\223\002w"0/v1beta1/{table_reference.project_id=projects/*}:\001*Z@";/v1beta1/{table_reference.dataset_id=projects/*/datasets/*}:\001*'
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="ReadRows",
+ full_name="google.cloud.bigquery.storage.v1beta1.BigQueryStorage.ReadRows",
+ index=1,
+ containing_service=None,
+ input_type=_READROWSREQUEST,
+ output_type=_READROWSRESPONSE,
+ serialized_options=_b(
+ "\202\323\344\223\002;\0229/v1beta1/{read_position.stream.name=projects/*/streams/*}"
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="BatchCreateReadSessionStreams",
+ full_name="google.cloud.bigquery.storage.v1beta1.BigQueryStorage.BatchCreateReadSessionStreams",
+ index=2,
+ containing_service=None,
+ input_type=_BATCHCREATEREADSESSIONSTREAMSREQUEST,
+ output_type=_BATCHCREATEREADSESSIONSTREAMSRESPONSE,
+ serialized_options=_b(
+ '\202\323\344\223\0022"-/v1beta1/{session.name=projects/*/sessions/*}:\001*'
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="FinalizeStream",
+ full_name="google.cloud.bigquery.storage.v1beta1.BigQueryStorage.FinalizeStream",
+ index=3,
+ containing_service=None,
+ input_type=_FINALIZESTREAMREQUEST,
+ output_type=google_dot_protobuf_dot_empty__pb2._EMPTY,
+ serialized_options=_b(
+ '\202\323\344\223\0020"+/v1beta1/{stream.name=projects/*/streams/*}:\001*'
+ ),
+ ),
+ _descriptor.MethodDescriptor(
+ name="SplitReadStream",
+ full_name="google.cloud.bigquery.storage.v1beta1.BigQueryStorage.SplitReadStream",
+ index=4,
+ containing_service=None,
+ input_type=_SPLITREADSTREAMREQUEST,
+ output_type=_SPLITREADSTREAMRESPONSE,
+ serialized_options=_b(
+ "\202\323\344\223\0026\0224/v1beta1/{original_stream.name=projects/*/streams/*}"
+ ),
+ ),
+ ],
+)
+_sym_db.RegisterServiceDescriptor(_BIGQUERYSTORAGE)
+
+DESCRIPTOR.services_by_name["BigQueryStorage"] = _BIGQUERYSTORAGE
+
+# @@protoc_insertion_point(module_scope)
diff --git a/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/storage_pb2_grpc.py b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/storage_pb2_grpc.py
new file mode 100644
index 000000000000..85b890f0e0cc
--- /dev/null
+++ b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/storage_pb2_grpc.py
@@ -0,0 +1,168 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
+
+from google.cloud.bigquery_storage_v1beta1.proto import (
+ storage_pb2 as google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_storage__pb2,
+)
+from google.protobuf import empty_pb2 as google_dot_protobuf_dot_empty__pb2
+
+
+class BigQueryStorageStub(object):
+ """BigQuery storage API.
+
+ The BigQuery storage API can be used to read data stored in BigQuery.
+ """
+
+ def __init__(self, channel):
+ """Constructor.
+
+ Args:
+ channel: A grpc.Channel.
+ """
+ self.CreateReadSession = channel.unary_unary(
+ "/google.cloud.bigquery.storage.v1beta1.BigQueryStorage/CreateReadSession",
+ request_serializer=google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_storage__pb2.CreateReadSessionRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_storage__pb2.ReadSession.FromString,
+ )
+ self.ReadRows = channel.unary_stream(
+ "/google.cloud.bigquery.storage.v1beta1.BigQueryStorage/ReadRows",
+ request_serializer=google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_storage__pb2.ReadRowsRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_storage__pb2.ReadRowsResponse.FromString,
+ )
+ self.BatchCreateReadSessionStreams = channel.unary_unary(
+ "/google.cloud.bigquery.storage.v1beta1.BigQueryStorage/BatchCreateReadSessionStreams",
+ request_serializer=google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_storage__pb2.BatchCreateReadSessionStreamsRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_storage__pb2.BatchCreateReadSessionStreamsResponse.FromString,
+ )
+ self.FinalizeStream = channel.unary_unary(
+ "/google.cloud.bigquery.storage.v1beta1.BigQueryStorage/FinalizeStream",
+ request_serializer=google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_storage__pb2.FinalizeStreamRequest.SerializeToString,
+ response_deserializer=google_dot_protobuf_dot_empty__pb2.Empty.FromString,
+ )
+ self.SplitReadStream = channel.unary_unary(
+ "/google.cloud.bigquery.storage.v1beta1.BigQueryStorage/SplitReadStream",
+ request_serializer=google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_storage__pb2.SplitReadStreamRequest.SerializeToString,
+ response_deserializer=google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_storage__pb2.SplitReadStreamResponse.FromString,
+ )
+
+
+class BigQueryStorageServicer(object):
+ """BigQuery storage API.
+
+ The BigQuery storage API can be used to read data stored in BigQuery.
+ """
+
+ def CreateReadSession(self, request, context):
+ """Creates a new read session. A read session divides the contents of a
+ BigQuery table into one or more streams, which can then be used to read
+ data from the table. The read session also specifies properties of the
+ data to be read, such as a list of columns or a push-down filter describing
+ the rows to be returned.
+
+ A particular row can be read by at most one stream. When the caller has
+ reached the end of each stream in the session, then all the data in the
+ table has been read.
+
+ Read sessions automatically expire 24 hours after they are created and do
+ not require manual clean-up by the caller.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def ReadRows(self, request, context):
+ """Reads rows from the table in the format prescribed by the read session.
+ Each response contains one or more table rows, up to a maximum of 10 MiB
+ per response; read requests which attempt to read individual rows larger
+ than this will fail.
+
+ Each request also returns a set of stream statistics reflecting the
+ estimated total number of rows in the read stream. This number is computed
+ based on the total table size and the number of active streams in the read
+ session, and may change as other streams continue to read data.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def BatchCreateReadSessionStreams(self, request, context):
+ """Creates additional streams for a ReadSession. This API can be used to
+ dynamically adjust the parallelism of a batch processing task upwards by
+ adding additional workers.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def FinalizeStream(self, request, context):
+ """Triggers the graceful termination of a single stream in a ReadSession. This
+ API can be used to dynamically adjust the parallelism of a batch processing
+ task downwards without losing data.
+
+ This API does not delete the stream -- it remains visible in the
+ ReadSession, and any data processed by the stream is not released to other
+ streams. However, no additional data will be assigned to the stream once
+ this call completes. Callers must continue reading data on the stream until
+ the end of the stream is reached so that data which has already been
+ assigned to the stream will be processed.
+
+ This method will return an error if there are no other live streams
+ in the Session, or if SplitReadStream() has been called on the given
+ Stream.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+ def SplitReadStream(self, request, context):
+ """Splits a given read stream into two Streams. These streams are referred to
+ as the primary and the residual of the split. The original stream can still
+ be read from in the same manner as before. Both of the returned streams can
+ also be read from, and the total rows return by both child streams will be
+ the same as the rows read from the original stream.
+
+ Moreover, the two child streams will be allocated back to back in the
+ original Stream. Concretely, it is guaranteed that for streams Original,
+ Primary, and Residual, that Original[0-j] = Primary[0-j] and
+ Original[j-n] = Residual[0-m] once the streams have been read to
+ completion.
+
+ This method is guaranteed to be idempotent.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details("Method not implemented!")
+ raise NotImplementedError("Method not implemented!")
+
+
+def add_BigQueryStorageServicer_to_server(servicer, server):
+ rpc_method_handlers = {
+ "CreateReadSession": grpc.unary_unary_rpc_method_handler(
+ servicer.CreateReadSession,
+ request_deserializer=google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_storage__pb2.CreateReadSessionRequest.FromString,
+ response_serializer=google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_storage__pb2.ReadSession.SerializeToString,
+ ),
+ "ReadRows": grpc.unary_stream_rpc_method_handler(
+ servicer.ReadRows,
+ request_deserializer=google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_storage__pb2.ReadRowsRequest.FromString,
+ response_serializer=google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_storage__pb2.ReadRowsResponse.SerializeToString,
+ ),
+ "BatchCreateReadSessionStreams": grpc.unary_unary_rpc_method_handler(
+ servicer.BatchCreateReadSessionStreams,
+ request_deserializer=google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_storage__pb2.BatchCreateReadSessionStreamsRequest.FromString,
+ response_serializer=google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_storage__pb2.BatchCreateReadSessionStreamsResponse.SerializeToString,
+ ),
+ "FinalizeStream": grpc.unary_unary_rpc_method_handler(
+ servicer.FinalizeStream,
+ request_deserializer=google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_storage__pb2.FinalizeStreamRequest.FromString,
+ response_serializer=google_dot_protobuf_dot_empty__pb2.Empty.SerializeToString,
+ ),
+ "SplitReadStream": grpc.unary_unary_rpc_method_handler(
+ servicer.SplitReadStream,
+ request_deserializer=google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_storage__pb2.SplitReadStreamRequest.FromString,
+ response_serializer=google_dot_cloud_dot_bigquery_dot_storage__v1beta1_dot_proto_dot_storage__pb2.SplitReadStreamResponse.SerializeToString,
+ ),
+ }
+ generic_handler = grpc.method_handlers_generic_handler(
+ "google.cloud.bigquery.storage.v1beta1.BigQueryStorage", rpc_method_handlers
+ )
+ server.add_generic_rpc_handlers((generic_handler,))
diff --git a/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/table_reference.proto b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/table_reference.proto
new file mode 100644
index 000000000000..fb74bbf6c642
--- /dev/null
+++ b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/table_reference.proto
@@ -0,0 +1,42 @@
+// Copyright 2019 Google LLC.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+syntax = "proto3";
+
+package google.cloud.bigquery.storage.v1beta1;
+
+import "google/protobuf/timestamp.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/cloud/bigquery/storage/v1beta1;storage";
+option java_outer_classname = "TableReferenceProto";
+option java_package = "com.google.cloud.bigquery.storage.v1beta1";
+
+// Table reference that includes just the 3 strings needed to identify a table.
+message TableReference {
+ // The assigned project ID of the project.
+ string project_id = 1;
+
+ // The ID of the dataset in the above project.
+ string dataset_id = 2;
+
+ // The ID of the table in the above dataset.
+ string table_id = 3;
+}
+
+// All fields in this message optional.
+message TableModifiers {
+ // The snapshot time of the table. If not set, interpreted as now.
+ google.protobuf.Timestamp snapshot_time = 1;
+}
diff --git a/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/table_reference_pb2.py b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/table_reference_pb2.py
new file mode 100644
index 000000000000..30c85aa2f469
--- /dev/null
+++ b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/table_reference_pb2.py
@@ -0,0 +1,199 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: google/cloud/bigquery/storage_v1beta1/proto/table_reference.proto
+
+import sys
+
+_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1"))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.protobuf import timestamp_pb2 as google_dot_protobuf_dot_timestamp__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+ name="google/cloud/bigquery/storage_v1beta1/proto/table_reference.proto",
+ package="google.cloud.bigquery.storage.v1beta1",
+ syntax="proto3",
+ serialized_options=_b(
+ "\n)com.google.cloud.bigquery.storage.v1beta1B\023TableReferenceProtoZLgoogle.golang.org/genproto/googleapis/cloud/bigquery/storage/v1beta1;storage"
+ ),
+ serialized_pb=_b(
+ '\nAgoogle/cloud/bigquery/storage_v1beta1/proto/table_reference.proto\x12%google.cloud.bigquery.storage.v1beta1\x1a\x1fgoogle/protobuf/timestamp.proto"J\n\x0eTableReference\x12\x12\n\nproject_id\x18\x01 \x01(\t\x12\x12\n\ndataset_id\x18\x02 \x01(\t\x12\x10\n\x08table_id\x18\x03 \x01(\t"C\n\x0eTableModifiers\x12\x31\n\rsnapshot_time\x18\x01 \x01(\x0b\x32\x1a.google.protobuf.TimestampB\x8e\x01\n)com.google.cloud.bigquery.storage.v1beta1B\x13TableReferenceProtoZLgoogle.golang.org/genproto/googleapis/cloud/bigquery/storage/v1beta1;storageb\x06proto3'
+ ),
+ dependencies=[google_dot_protobuf_dot_timestamp__pb2.DESCRIPTOR],
+)
+
+
+_TABLEREFERENCE = _descriptor.Descriptor(
+ name="TableReference",
+ full_name="google.cloud.bigquery.storage.v1beta1.TableReference",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="project_id",
+ full_name="google.cloud.bigquery.storage.v1beta1.TableReference.project_id",
+ index=0,
+ number=1,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="dataset_id",
+ full_name="google.cloud.bigquery.storage.v1beta1.TableReference.dataset_id",
+ index=1,
+ number=2,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ _descriptor.FieldDescriptor(
+ name="table_id",
+ full_name="google.cloud.bigquery.storage.v1beta1.TableReference.table_id",
+ index=2,
+ number=3,
+ type=9,
+ cpp_type=9,
+ label=1,
+ has_default_value=False,
+ default_value=_b("").decode("utf-8"),
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ ),
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=141,
+ serialized_end=215,
+)
+
+
+_TABLEMODIFIERS = _descriptor.Descriptor(
+ name="TableModifiers",
+ full_name="google.cloud.bigquery.storage.v1beta1.TableModifiers",
+ filename=None,
+ file=DESCRIPTOR,
+ containing_type=None,
+ fields=[
+ _descriptor.FieldDescriptor(
+ name="snapshot_time",
+ full_name="google.cloud.bigquery.storage.v1beta1.TableModifiers.snapshot_time",
+ index=0,
+ number=1,
+ type=11,
+ cpp_type=10,
+ label=1,
+ has_default_value=False,
+ default_value=None,
+ message_type=None,
+ enum_type=None,
+ containing_type=None,
+ is_extension=False,
+ extension_scope=None,
+ serialized_options=None,
+ file=DESCRIPTOR,
+ )
+ ],
+ extensions=[],
+ nested_types=[],
+ enum_types=[],
+ serialized_options=None,
+ is_extendable=False,
+ syntax="proto3",
+ extension_ranges=[],
+ oneofs=[],
+ serialized_start=217,
+ serialized_end=284,
+)
+
+_TABLEMODIFIERS.fields_by_name[
+ "snapshot_time"
+].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
+DESCRIPTOR.message_types_by_name["TableReference"] = _TABLEREFERENCE
+DESCRIPTOR.message_types_by_name["TableModifiers"] = _TABLEMODIFIERS
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+TableReference = _reflection.GeneratedProtocolMessageType(
+ "TableReference",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TABLEREFERENCE,
+ __module__="google.cloud.bigquery.storage_v1beta1.proto.table_reference_pb2",
+ __doc__="""Table reference that includes just the 3 strings needed to identify a
+ table.
+
+
+ Attributes:
+ project_id:
+ The assigned project ID of the project.
+ dataset_id:
+ The ID of the dataset in the above project.
+ table_id:
+ The ID of the table in the above dataset.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.storage.v1beta1.TableReference)
+ ),
+)
+_sym_db.RegisterMessage(TableReference)
+
+TableModifiers = _reflection.GeneratedProtocolMessageType(
+ "TableModifiers",
+ (_message.Message,),
+ dict(
+ DESCRIPTOR=_TABLEMODIFIERS,
+ __module__="google.cloud.bigquery.storage_v1beta1.proto.table_reference_pb2",
+ __doc__="""All fields in this message optional.
+
+
+ Attributes:
+ snapshot_time:
+ The snapshot time of the table. If not set, interpreted as
+ now.
+ """,
+ # @@protoc_insertion_point(class_scope:google.cloud.bigquery.storage.v1beta1.TableModifiers)
+ ),
+)
+_sym_db.RegisterMessage(TableModifiers)
+
+
+DESCRIPTOR._options = None
+# @@protoc_insertion_point(module_scope)
diff --git a/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/table_reference_pb2_grpc.py b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/table_reference_pb2_grpc.py
new file mode 100644
index 000000000000..07cb78fe03a9
--- /dev/null
+++ b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/table_reference_pb2_grpc.py
@@ -0,0 +1,2 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
diff --git a/bigquery_storage/google/cloud/bigquery_storage_v1beta1/reader.py b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/reader.py
new file mode 100644
index 000000000000..138fae4110eb
--- /dev/null
+++ b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/reader.py
@@ -0,0 +1,611 @@
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import absolute_import
+
+import collections
+import json
+
+try:
+ import fastavro
+except ImportError: # pragma: NO COVER
+ fastavro = None
+import google.api_core.exceptions
+
+try:
+ import pandas
+except ImportError: # pragma: NO COVER
+ pandas = None
+try:
+ import pyarrow
+except ImportError: # pragma: NO COVER
+ pyarrow = None
+import six
+
+try:
+ import pyarrow
+except ImportError: # pragma: NO COVER
+ pyarrow = None
+
+from google.cloud.bigquery_storage_v1beta1 import types
+
+
+_STREAM_RESUMPTION_EXCEPTIONS = (google.api_core.exceptions.ServiceUnavailable,)
+
+_FASTAVRO_REQUIRED = (
+ "fastavro is required to parse ReadRowResponse messages with Avro bytes."
+)
+_PANDAS_REQUIRED = "pandas is required to create a DataFrame"
+_PYARROW_REQUIRED = (
+ "pyarrow is required to parse ReadRowResponse messages with Arrow bytes."
+)
+
+
+class ReadRowsStream(object):
+ """A stream of results from a read rows request.
+
+ This stream is an iterable of
+ :class:`~google.cloud.bigquery_storage_v1beta1.types.ReadRowsResponse`.
+ Iterate over it to fetch all row messages.
+
+ If the fastavro library is installed, use the
+ :func:`~google.cloud.bigquery_storage_v1beta1.reader.ReadRowsStream.rows()`
+ method to parse all messages into a stream of row dictionaries.
+
+ If the pandas and fastavro libraries are installed, use the
+ :func:`~google.cloud.bigquery_storage_v1beta1.reader.ReadRowsStream.to_dataframe()`
+ method to parse all messages into a :class:`pandas.DataFrame`.
+ """
+
+ def __init__(self, wrapped, client, read_position, read_rows_kwargs):
+ """Construct a ReadRowsStream.
+
+ Args:
+ wrapped (Iterable[ \
+ ~google.cloud.bigquery_storage_v1beta1.types.ReadRowsResponse \
+ ]):
+ The ReadRows stream to read.
+ client ( \
+ ~google.cloud.bigquery_storage_v1beta1.gapic. \
+ big_query_storage_client.BigQueryStorageClient \
+ ):
+ A GAPIC client used to reconnect to a ReadRows stream. This
+ must be the GAPIC client to avoid a circular dependency on
+ this class.
+ read_position (Union[ \
+ dict, \
+ ~google.cloud.bigquery_storage_v1beta1.types.StreamPosition \
+ ]):
+ Required. Identifier of the position in the stream to start
+ reading from. The offset requested must be less than the last
+ row read from ReadRows. Requesting a larger offset is
+ undefined. If a dict is provided, it must be of the same form
+ as the protobuf message
+ :class:`~google.cloud.bigquery_storage_v1beta1.types.StreamPosition`
+ read_rows_kwargs (dict):
+ Keyword arguments to use when reconnecting to a ReadRows
+ stream.
+
+ Returns:
+ Iterable[ \
+ ~google.cloud.bigquery_storage_v1beta1.types.ReadRowsResponse \
+ ]:
+ A sequence of row messages.
+ """
+
+ # Make a copy of the read position so that we can update it without
+ # mutating the original input.
+ self._position = _copy_stream_position(read_position)
+ self._client = client
+ self._wrapped = wrapped
+ self._read_rows_kwargs = read_rows_kwargs
+
+ def __iter__(self):
+ """An iterable of messages.
+
+ Returns:
+ Iterable[ \
+ ~google.cloud.bigquery_storage_v1beta1.types.ReadRowsResponse \
+ ]:
+ A sequence of row messages.
+ """
+
+ # Infinite loop to reconnect on reconnectable errors while processing
+ # the row stream.
+ while True:
+ try:
+ for message in self._wrapped:
+ rowcount = message.row_count
+ self._position.offset += rowcount
+ yield message
+
+ return # Made it through the whole stream.
+ except _STREAM_RESUMPTION_EXCEPTIONS:
+ # Transient error, so reconnect to the stream.
+ pass
+
+ self._reconnect()
+
+ def _reconnect(self):
+ """Reconnect to the ReadRows stream using the most recent offset."""
+ self._wrapped = self._client.read_rows(
+ _copy_stream_position(self._position), **self._read_rows_kwargs
+ )
+
+ def rows(self, read_session):
+ """Iterate over all rows in the stream.
+
+ This method requires the fastavro library in order to parse row
+ messages.
+
+ .. warning::
+ DATETIME columns are not supported. They are currently parsed as
+ strings in the fastavro library.
+
+ Args:
+ read_session ( \
+ ~google.cloud.bigquery_storage_v1beta1.types.ReadSession \
+ ):
+ The read session associated with this read rows stream. This
+ contains the schema, which is required to parse the data
+ messages.
+
+ Returns:
+ Iterable[Mapping]:
+ A sequence of rows, represented as dictionaries.
+ """
+ return ReadRowsIterable(self, read_session)
+
+ def to_arrow(self, read_session):
+ """Create a :class:`pyarrow.Table` of all rows in the stream.
+
+ This method requires the pyarrow library and a stream using the Arrow
+ format.
+
+ Args:
+ read_session ( \
+ ~google.cloud.bigquery_storage_v1beta1.types.ReadSession \
+ ):
+ The read session associated with this read rows stream. This
+ contains the schema, which is required to parse the data
+ messages.
+
+ Returns:
+ pyarrow.Table:
+ A table of all rows in the stream.
+ """
+ return self.rows(read_session).to_arrow()
+
+ def to_dataframe(self, read_session, dtypes=None):
+ """Create a :class:`pandas.DataFrame` of all rows in the stream.
+
+ This method requires the pandas libary to create a data frame and the
+ fastavro library to parse row messages.
+
+ .. warning::
+ DATETIME columns are not supported. They are currently parsed as
+ strings.
+
+ Args:
+ read_session ( \
+ ~google.cloud.bigquery_storage_v1beta1.types.ReadSession \
+ ):
+ The read session associated with this read rows stream. This
+ contains the schema, which is required to parse the data
+ messages.
+ dtypes ( \
+ Map[str, Union[str, pandas.Series.dtype]] \
+ ):
+ Optional. A dictionary of column names pandas ``dtype``s. The
+ provided ``dtype`` is used when constructing the series for
+ the column specified. Otherwise, the default pandas behavior
+ is used.
+
+ Returns:
+ pandas.DataFrame:
+ A data frame of all rows in the stream.
+ """
+ if pandas is None:
+ raise ImportError(_PANDAS_REQUIRED)
+
+ return self.rows(read_session).to_dataframe(dtypes=dtypes)
+
+
+class ReadRowsIterable(object):
+ """An iterable of rows from a read session.
+
+ Args:
+ reader (google.cloud.bigquery_storage_v1beta1.reader.ReadRowsStream):
+ A read rows stream.
+ read_session (google.cloud.bigquery_storage_v1beta1.types.ReadSession):
+ A read session. This is required because it contains the schema
+ used in the stream messages.
+ """
+
+ # This class is modelled after the google.cloud.bigquery.table.RowIterator
+ # and aims to be API compatible where possible.
+
+ def __init__(self, reader, read_session):
+ self._status = None
+ self._reader = reader
+ self._read_session = read_session
+ self._stream_parser = _StreamParser.from_read_session(self._read_session)
+
+ @property
+ def total_rows(self):
+ """int: Number of estimated rows in the current stream.
+
+ May change over time.
+ """
+ return getattr(self._status, "estimated_row_count", None)
+
+ @property
+ def pages(self):
+ """A generator of all pages in the stream.
+
+ Returns:
+ types.GeneratorType[google.cloud.bigquery_storage_v1beta1.ReadRowsPage]:
+ A generator of pages.
+ """
+ # Each page is an iterator of rows. But also has num_items, remaining,
+ # and to_dataframe.
+ for message in self._reader:
+ self._status = message.status
+ yield ReadRowsPage(self._stream_parser, message)
+
+ def __iter__(self):
+ """Iterator for each row in all pages."""
+ for page in self.pages:
+ for row in page:
+ yield row
+
+ def to_arrow(self):
+ """Create a :class:`pyarrow.Table` of all rows in the stream.
+
+ This method requires the pyarrow library and a stream using the Arrow
+ format.
+
+ Returns:
+ pyarrow.Table:
+ A table of all rows in the stream.
+ """
+ record_batches = []
+ for page in self.pages:
+ record_batches.append(page.to_arrow())
+ return pyarrow.Table.from_batches(record_batches)
+
+ def to_dataframe(self, dtypes=None):
+ """Create a :class:`pandas.DataFrame` of all rows in the stream.
+
+ This method requires the pandas libary to create a data frame and the
+ fastavro library to parse row messages.
+
+ .. warning::
+ DATETIME columns are not supported. They are currently parsed as
+ strings in the fastavro library.
+
+ Args:
+ dtypes ( \
+ Map[str, Union[str, pandas.Series.dtype]] \
+ ):
+ Optional. A dictionary of column names pandas ``dtype``s. The
+ provided ``dtype`` is used when constructing the series for
+ the column specified. Otherwise, the default pandas behavior
+ is used.
+
+ Returns:
+ pandas.DataFrame:
+ A data frame of all rows in the stream.
+ """
+ if pandas is None:
+ raise ImportError(_PANDAS_REQUIRED)
+
+ frames = []
+ for page in self.pages:
+ frames.append(page.to_dataframe(dtypes=dtypes))
+ return pandas.concat(frames)
+
+
+class ReadRowsPage(object):
+ """An iterator of rows from a read session message.
+
+ Args:
+ stream_parser (google.cloud.bigquery_storage_v1beta1.reader._StreamParser):
+ A helper for parsing messages into rows.
+ message (google.cloud.bigquery_storage_v1beta1.types.ReadRowsResponse):
+ A message of data from a read rows stream.
+ """
+
+ # This class is modeled after google.api_core.page_iterator.Page and aims
+ # to provide API compatibility where possible.
+
+ def __init__(self, stream_parser, message):
+ self._stream_parser = stream_parser
+ self._message = message
+ self._iter_rows = None
+ self._num_items = self._message.row_count
+ self._remaining = self._message.row_count
+
+ def _parse_rows(self):
+ """Parse rows from the message only once."""
+ if self._iter_rows is not None:
+ return
+
+ rows = self._stream_parser.to_rows(self._message)
+ self._iter_rows = iter(rows)
+
+ @property
+ def num_items(self):
+ """int: Total items in the page."""
+ return self._num_items
+
+ @property
+ def remaining(self):
+ """int: Remaining items in the page."""
+ return self._remaining
+
+ def __iter__(self):
+ """A ``ReadRowsPage`` is an iterator."""
+ return self
+
+ def next(self):
+ """Get the next row in the page."""
+ self._parse_rows()
+ if self._remaining > 0:
+ self._remaining -= 1
+ return six.next(self._iter_rows)
+
+ # Alias needed for Python 2/3 support.
+ __next__ = next
+
+ def to_arrow(self):
+ """Create an :class:`pyarrow.RecordBatch` of rows in the page.
+
+ Returns:
+ pyarrow.RecordBatch:
+ Rows from the message, as an Arrow record batch.
+ """
+ return self._stream_parser.to_arrow(self._message)
+
+ def to_dataframe(self, dtypes=None):
+ """Create a :class:`pandas.DataFrame` of rows in the page.
+
+ This method requires the pandas libary to create a data frame and the
+ fastavro library to parse row messages.
+
+ .. warning::
+ DATETIME columns are not supported. They are currently parsed as
+ strings in the fastavro library.
+
+ Args:
+ dtypes ( \
+ Map[str, Union[str, pandas.Series.dtype]] \
+ ):
+ Optional. A dictionary of column names pandas ``dtype``s. The
+ provided ``dtype`` is used when constructing the series for
+ the column specified. Otherwise, the default pandas behavior
+ is used.
+
+ Returns:
+ pandas.DataFrame:
+ A data frame of all rows in the stream.
+ """
+ if pandas is None:
+ raise ImportError(_PANDAS_REQUIRED)
+ return self._stream_parser.to_dataframe(self._message, dtypes=dtypes)
+
+
+class _StreamParser(object):
+ def to_arrow(self, message):
+ raise NotImplementedError("Not implemented.")
+
+ def to_dataframe(self, message, dtypes=None):
+ raise NotImplementedError("Not implemented.")
+
+ def to_rows(self, message):
+ raise NotImplementedError("Not implemented.")
+
+ @staticmethod
+ def from_read_session(read_session):
+ schema_type = read_session.WhichOneof("schema")
+ if schema_type == "avro_schema":
+ return _AvroStreamParser(read_session)
+ elif schema_type == "arrow_schema":
+ return _ArrowStreamParser(read_session)
+ else:
+ raise TypeError(
+ "Unsupported schema type in read_session: {0}".format(schema_type)
+ )
+
+
+class _AvroStreamParser(_StreamParser):
+ """Helper to parse Avro messages into useful representations."""
+
+ def __init__(self, read_session):
+ """Construct an _AvroStreamParser.
+
+ Args:
+ read_session (google.cloud.bigquery_storage_v1beta1.types.ReadSession):
+ A read session. This is required because it contains the schema
+ used in the stream messages.
+ """
+ if fastavro is None:
+ raise ImportError(_FASTAVRO_REQUIRED)
+
+ self._read_session = read_session
+ self._avro_schema_json = None
+ self._fastavro_schema = None
+ self._column_names = None
+
+ def to_arrow(self, message):
+ """Create an :class:`pyarrow.RecordBatch` of rows in the page.
+
+ Args:
+ message (google.cloud.bigquery_storage_v1beta1.types.ReadRowsResponse):
+ Protocol buffer from the read rows stream, to convert into an
+ Arrow record batch.
+
+ Returns:
+ pyarrow.RecordBatch:
+ Rows from the message, as an Arrow record batch.
+ """
+ raise NotImplementedError("to_arrow not implemented for Avro streams.")
+
+ def to_dataframe(self, message, dtypes=None):
+ """Create a :class:`pandas.DataFrame` of rows in the page.
+
+ This method requires the pandas libary to create a data frame and the
+ fastavro library to parse row messages.
+
+ .. warning::
+ DATETIME columns are not supported. They are currently parsed as
+ strings in the fastavro library.
+
+ Args:
+ dtypes ( \
+ Map[str, Union[str, pandas.Series.dtype]] \
+ ):
+ Optional. A dictionary of column names pandas ``dtype``s. The
+ provided ``dtype`` is used when constructing the series for
+ the column specified. Otherwise, the default pandas behavior
+ is used.
+
+ Returns:
+ pandas.DataFrame:
+ A data frame of all rows in the stream.
+ """
+ self._parse_avro_schema()
+
+ if dtypes is None:
+ dtypes = {}
+
+ columns = collections.defaultdict(list)
+ for row in self.to_rows(message):
+ for column in row:
+ columns[column].append(row[column])
+ for column in dtypes:
+ columns[column] = pandas.Series(columns[column], dtype=dtypes[column])
+ return pandas.DataFrame(columns, columns=self._column_names)
+
+ def _parse_avro_schema(self):
+ """Extract and parse Avro schema from a read session."""
+ if self._avro_schema_json:
+ return
+
+ self._avro_schema_json = json.loads(self._read_session.avro_schema.schema)
+ self._column_names = tuple(
+ (field["name"] for field in self._avro_schema_json["fields"])
+ )
+
+ def _parse_fastavro(self):
+ """Convert parsed Avro schema to fastavro format."""
+ self._parse_avro_schema()
+ self._fastavro_schema = fastavro.parse_schema(self._avro_schema_json)
+
+ def to_rows(self, message):
+ """Parse all rows in a stream message.
+
+ Args:
+ message ( \
+ ~google.cloud.bigquery_storage_v1beta1.types.ReadRowsResponse \
+ ):
+ A message containing Avro bytes to parse into rows.
+
+ Returns:
+ Iterable[Mapping]:
+ A sequence of rows, represented as dictionaries.
+ """
+ self._parse_fastavro()
+ messageio = six.BytesIO(message.avro_rows.serialized_binary_rows)
+ while True:
+ # Loop in a while loop because schemaless_reader can only read
+ # a single record.
+ try:
+ # TODO: Parse DATETIME into datetime.datetime (no timezone),
+ # instead of as a string.
+ yield fastavro.schemaless_reader(messageio, self._fastavro_schema)
+ except StopIteration:
+ break # Finished with message
+
+
+class _ArrowStreamParser(_StreamParser):
+ def __init__(self, read_session):
+ if pyarrow is None:
+ raise ImportError(_PYARROW_REQUIRED)
+
+ self._read_session = read_session
+ self._schema = None
+
+ def to_arrow(self, message):
+ return self._parse_arrow_message(message)
+
+ def to_rows(self, message):
+ record_batch = self._parse_arrow_message(message)
+
+ # Iterate through each column simultaneously, and make a dict from the
+ # row values
+ for row in zip(*record_batch.columns):
+ yield dict(zip(self._column_names, row))
+
+ def to_dataframe(self, message, dtypes=None):
+ record_batch = self._parse_arrow_message(message)
+
+ if dtypes is None:
+ dtypes = {}
+
+ df = record_batch.to_pandas()
+
+ for column in dtypes:
+ df[column] = pandas.Series(df[column], dtype=dtypes[column])
+
+ return df
+
+ def _parse_arrow_message(self, message):
+ self._parse_arrow_schema()
+
+ return pyarrow.read_record_batch(
+ pyarrow.py_buffer(message.arrow_record_batch.serialized_record_batch),
+ self._schema,
+ )
+
+ def _parse_arrow_schema(self):
+ if self._schema:
+ return
+
+ self._schema = pyarrow.read_schema(
+ pyarrow.py_buffer(self._read_session.arrow_schema.serialized_schema)
+ )
+ self._column_names = [field.name for field in self._schema]
+
+
+def _copy_stream_position(position):
+ """Copy a StreamPosition.
+
+ Args:
+ position (Union[ \
+ dict, \
+ ~google.cloud.bigquery_storage_v1beta1.types.StreamPosition \
+ ]):
+ StreamPostion (or dictionary in StreamPosition format) to copy.
+
+ Returns:
+ ~google.cloud.bigquery_storage_v1beta1.types.StreamPosition:
+ A copy of the input StreamPostion.
+ """
+ if isinstance(position, types.StreamPosition):
+ output = types.StreamPosition()
+ output.CopyFrom(position)
+ return output
+
+ return types.StreamPosition(**position)
diff --git a/bigquery_storage/google/cloud/bigquery_storage_v1beta1/types.py b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/types.py
new file mode 100644
index 000000000000..fd5a3e35c830
--- /dev/null
+++ b/bigquery_storage/google/cloud/bigquery_storage_v1beta1/types.py
@@ -0,0 +1,55 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from __future__ import absolute_import
+import sys
+
+from google.api_core.protobuf_helpers import get_messages
+
+from google.cloud.bigquery_storage_v1beta1.proto import arrow_pb2
+from google.cloud.bigquery_storage_v1beta1.proto import avro_pb2
+from google.cloud.bigquery_storage_v1beta1.proto import read_options_pb2
+from google.cloud.bigquery_storage_v1beta1.proto import storage_pb2
+from google.cloud.bigquery_storage_v1beta1.proto import table_reference_pb2
+from google.protobuf import empty_pb2
+from google.protobuf import timestamp_pb2
+
+
+_shared_modules = [empty_pb2, timestamp_pb2]
+
+_local_modules = [
+ arrow_pb2,
+ avro_pb2,
+ read_options_pb2,
+ storage_pb2,
+ table_reference_pb2,
+]
+
+names = []
+
+for module in _shared_modules: # pragma: NO COVER
+ for name, message in get_messages(module).items():
+ setattr(sys.modules[__name__], name, message)
+ names.append(name)
+for module in _local_modules:
+ for name, message in get_messages(module).items():
+ message.__module__ = "google.cloud.bigquery_storage_v1beta1.types"
+ setattr(sys.modules[__name__], name, message)
+ names.append(name)
+
+
+__all__ = tuple(sorted(names))
diff --git a/bigquery_storage/noxfile.py b/bigquery_storage/noxfile.py
new file mode 100644
index 000000000000..79b9d3689512
--- /dev/null
+++ b/bigquery_storage/noxfile.py
@@ -0,0 +1,182 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Generated by synthtool. DO NOT EDIT!
+
+from __future__ import absolute_import
+import os
+import shutil
+
+import nox
+
+
+LOCAL_DEPS = (os.path.join("..", "api_core"), os.path.join("..", "core"))
+BLACK_VERSION = "black==19.3b0"
+BLACK_PATHS = ["docs", "google", "tests", "noxfile.py", "setup.py"]
+
+if os.path.exists("samples"):
+ BLACK_PATHS.append("samples")
+
+
+@nox.session(python="3.7")
+def lint(session):
+ """Run linters.
+
+ Returns a failure if the linters find linting errors or sufficiently
+ serious code quality issues.
+ """
+ session.install("flake8", BLACK_VERSION, *LOCAL_DEPS)
+ session.run("black", "--check", *BLACK_PATHS)
+ session.run("flake8", "google", "tests")
+
+
+@nox.session(python="3.6")
+def blacken(session):
+ """Run black.
+
+ Format code to uniform standard.
+
+ This currently uses Python 3.6 due to the automated Kokoro run of synthtool.
+ That run uses an image that doesn't have 3.6 installed. Before updating this
+ check the state of the `gcp_ubuntu_config` we use for that Kokoro run.
+ """
+ session.install(BLACK_VERSION)
+ session.run("black", *BLACK_PATHS)
+
+
+@nox.session(python="3.7")
+def lint_setup_py(session):
+ """Verify that setup.py is valid (including RST check)."""
+ session.install("docutils", "pygments")
+ session.run("python", "setup.py", "check", "--restructuredtext", "--strict")
+
+
+def default(session):
+ # Install all test dependencies, then install this package in-place.
+ session.install("mock", "pytest", "pytest-cov")
+ for local_dep in LOCAL_DEPS:
+ session.install("-e", local_dep)
+ session.install("-e", ".")
+ session.install("-e", ".[fastavro,pandas,pyarrow]")
+
+ # Run py.test against the unit tests.
+ session.run(
+ "py.test",
+ "--quiet",
+ "--cov=google.cloud",
+ "--cov=tests.unit",
+ "--cov-append",
+ "--cov-config=.coveragerc",
+ "--cov-report=",
+ "--cov-fail-under=0",
+ os.path.join("tests", "unit"),
+ *session.posargs,
+ )
+
+
+@nox.session(python=["2.7", "3.5", "3.6", "3.7"])
+def unit(session):
+ """Run the unit test suite."""
+ default(session)
+
+
+@nox.session(python=["2.7", "3.7"])
+def system(session):
+ """Run the system test suite."""
+ system_test_path = os.path.join("tests", "system.py")
+ system_test_folder_path = os.path.join("tests", "system")
+ # Sanity check: Only run tests if the environment variable is set.
+ if not os.environ.get("GOOGLE_APPLICATION_CREDENTIALS", ""):
+ session.skip("Credentials must be set via environment variable")
+
+ system_test_exists = os.path.exists(system_test_path)
+ system_test_folder_exists = os.path.exists(system_test_folder_path)
+ # Sanity check: only run tests if found.
+ if not system_test_exists and not system_test_folder_exists:
+ session.skip("System tests were not found")
+
+ # Use pre-release gRPC for system tests.
+ session.install("--pre", "grpcio")
+
+ # Install all test dependencies, then install this package into the
+ # virtualenv's dist-packages.
+ session.install("mock", "pytest")
+ for local_dep in LOCAL_DEPS:
+ session.install("-e", local_dep)
+ session.install("-e", "../test_utils/")
+ session.install("-e", ".[fastavro,pandas,pyarrow]")
+ session.install("-e", "../bigquery/")
+ session.install("-e", ".")
+
+ # Run py.test against the system tests.
+ if system_test_exists:
+ session.run("py.test", "--quiet", system_test_path, *session.posargs)
+ if system_test_folder_exists:
+ session.run("py.test", "--quiet", system_test_folder_path, *session.posargs)
+
+
+@nox.session(python=["2.7", "3.7"])
+def samples(session):
+ requirements_path = os.path.join("samples", "requirements.txt")
+ requirements_exists = os.path.exists(requirements_path)
+
+ # Sanity check: Only run tests if the environment variable is set.
+ if not os.environ.get("GOOGLE_APPLICATION_CREDENTIALS", ""):
+ session.skip("Credentials must be set via environment variable")
+
+ session.install("mock", "pytest")
+ for local_dep in LOCAL_DEPS:
+ session.install("-e", local_dep)
+ if requirements_exists:
+ session.install("-r", requirements_path)
+ session.install("-e", ".")
+
+ session.run("py.test", "--quiet", "samples", *session.posargs)
+
+
+@nox.session(python="3.7")
+def cover(session):
+ """Run the final coverage report.
+
+ This outputs the coverage report aggregating coverage from the unit
+ test runs (not system test runs), and then erases coverage data.
+ """
+ session.install("coverage", "pytest-cov")
+ session.run("coverage", "report", "--show-missing", "--fail-under=79")
+
+ session.run("coverage", "erase")
+
+
+@nox.session(python="3.7")
+def docs(session):
+ """Build the docs for this library."""
+
+ session.install("-e", ".")
+ session.install("sphinx", "alabaster", "recommonmark")
+
+ shutil.rmtree(os.path.join("docs", "_build"), ignore_errors=True)
+ session.run(
+ "sphinx-build",
+ "-W", # warnings as errors
+ "-T", # show full traceback on exception
+ "-N", # no colors
+ "-b",
+ "html",
+ "-d",
+ os.path.join("docs", "_build", "doctrees", ""),
+ os.path.join("docs", ""),
+ os.path.join("docs", "_build", "html", ""),
+ )
diff --git a/bigquery_storage/samples/__init__.py b/bigquery_storage/samples/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/bigquery_storage/samples/quickstart.py b/bigquery_storage/samples/quickstart.py
new file mode 100644
index 000000000000..8a3b8617b387
--- /dev/null
+++ b/bigquery_storage/samples/quickstart.py
@@ -0,0 +1,100 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+
+
+def main(project_id="your-project-id", snapshot_millis=0):
+ # [START bigquerystorage_quickstart]
+ from google.cloud import bigquery_storage_v1beta1
+
+ # TODO(developer): Set the project_id variable.
+ # project_id = 'your-project-id'
+ #
+ # The read session is created in this project. This project can be
+ # different from that which contains the table.
+
+ client = bigquery_storage_v1beta1.BigQueryStorageClient()
+
+ # This example reads baby name data from the public datasets.
+ table_ref = bigquery_storage_v1beta1.types.TableReference()
+ table_ref.project_id = "bigquery-public-data"
+ table_ref.dataset_id = "usa_names"
+ table_ref.table_id = "usa_1910_current"
+
+ # We limit the output columns to a subset of those allowed in the table,
+ # and set a simple filter to only report names from the state of
+ # Washington (WA).
+ read_options = bigquery_storage_v1beta1.types.TableReadOptions()
+ read_options.selected_fields.append("name")
+ read_options.selected_fields.append("number")
+ read_options.selected_fields.append("state")
+ read_options.row_restriction = 'state = "WA"'
+
+ # Set a snapshot time if it's been specified.
+ modifiers = None
+ if snapshot_millis > 0:
+ modifiers = bigquery_storage_v1beta1.types.TableModifiers()
+ modifiers.snapshot_time.FromMilliseconds(snapshot_millis)
+
+ parent = "projects/{}".format(project_id)
+ session = client.create_read_session(
+ table_ref,
+ parent,
+ table_modifiers=modifiers,
+ read_options=read_options,
+ # This API can also deliver data serialized in Apache Arrow format.
+ # This example leverages Apache Avro.
+ format_=bigquery_storage_v1beta1.enums.DataFormat.AVRO,
+ # We use a LIQUID strategy in this example because we only read from a
+ # single stream. Consider BALANCED if you're consuming multiple streams
+ # concurrently and want more consistent stream sizes.
+ sharding_strategy=(bigquery_storage_v1beta1.enums.ShardingStrategy.LIQUID),
+ ) # API request.
+
+ # We'll use only a single stream for reading data from the table. Because
+ # of dynamic sharding, this will yield all the rows in the table. However,
+ # if you wanted to fan out multiple readers you could do so by having a
+ # reader process each individual stream.
+ reader = client.read_rows(
+ bigquery_storage_v1beta1.types.StreamPosition(stream=session.streams[0])
+ )
+
+ # The read stream contains blocks of Avro-encoded bytes. The rows() method
+ # uses the fastavro library to parse these blocks as an interable of Python
+ # dictionaries. Install fastavro with the following command:
+ #
+ # pip install google-cloud-bigquery-storage[fastavro]
+ rows = reader.rows(session)
+
+ # Do any local processing by iterating over the rows. The
+ # google-cloud-bigquery-storage client reconnects to the API after any
+ # transient network errors or timeouts.
+ names = set()
+ states = set()
+
+ for row in rows:
+ names.add(row["name"])
+ states.add(row["state"])
+
+ print("Got {} unique names in states: {}".format(len(names), states))
+ # [END bigquerystorage_quickstart]
+
+
+if __name__ == "__main__":
+ parser = argparse.ArgumentParser()
+ parser.add_argument("project_id")
+ parser.add_argument("--snapshot_millis", default=0, type=int)
+ args = parser.parse_args()
+ main(project_id=args.project_id)
diff --git a/bigquery_storage/samples/requirements.txt b/bigquery_storage/samples/requirements.txt
new file mode 100644
index 000000000000..acd0800e713e
--- /dev/null
+++ b/bigquery_storage/samples/requirements.txt
@@ -0,0 +1 @@
+fastavro
\ No newline at end of file
diff --git a/bigquery_storage/samples/tests/__init__.py b/bigquery_storage/samples/tests/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/bigquery_storage/samples/tests/quickstart_test.py b/bigquery_storage/samples/tests/quickstart_test.py
new file mode 100644
index 000000000000..fde039f4620b
--- /dev/null
+++ b/bigquery_storage/samples/tests/quickstart_test.py
@@ -0,0 +1,44 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import datetime
+import os
+
+import pytest
+
+from .. import quickstart
+
+
+def now_millis():
+ return int(
+ (datetime.datetime.utcnow() - datetime.datetime(1970, 1, 1)).total_seconds()
+ * 1000
+ )
+
+
+@pytest.fixture()
+def project_id():
+ return os.environ["PROJECT_ID"]
+
+
+def test_quickstart_wo_snapshot(capsys, project_id):
+ quickstart.main(project_id)
+ out, _ = capsys.readouterr()
+ assert "WA" in out
+
+
+def test_quickstart_with_snapshot(capsys, project_id):
+ quickstart.main(project_id, now_millis() - 5000)
+ out, _ = capsys.readouterr()
+ assert "WA" in out
diff --git a/bigquery_storage/setup.cfg b/bigquery_storage/setup.cfg
new file mode 100644
index 000000000000..3bd555500e37
--- /dev/null
+++ b/bigquery_storage/setup.cfg
@@ -0,0 +1,3 @@
+# Generated by synthtool. DO NOT EDIT!
+[bdist_wheel]
+universal = 1
diff --git a/bigquery_storage/setup.py b/bigquery_storage/setup.py
new file mode 100644
index 000000000000..29b968a59501
--- /dev/null
+++ b/bigquery_storage/setup.py
@@ -0,0 +1,81 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import io
+import os
+
+import setuptools
+
+name = "google-cloud-bigquery-storage"
+description = "BigQuery Storage API API client library"
+version = "0.7.0"
+release_status = "Development Status :: 4 - Beta"
+dependencies = [
+ "google-api-core[grpc] >= 1.14.0, < 2.0.0dev",
+ 'enum34; python_version < "3.4"',
+]
+extras = {
+ "pandas": "pandas>=0.17.1",
+ "fastavro": "fastavro>=0.21.2",
+ "pyarrow": "pyarrow>=0.13.0, != 0.14.0",
+}
+
+package_root = os.path.abspath(os.path.dirname(__file__))
+
+readme_filename = os.path.join(package_root, "README.rst")
+with io.open(readme_filename, encoding="utf-8") as readme_file:
+ readme = readme_file.read()
+
+packages = [
+ package for package in setuptools.find_packages() if package.startswith("google")
+]
+
+namespaces = ["google"]
+if "google.cloud" in packages:
+ namespaces.append("google.cloud")
+
+setuptools.setup(
+ name=name,
+ version=version,
+ description=description,
+ long_description=readme,
+ author="Google LLC",
+ author_email="googleapis-packages@google.com",
+ license="Apache 2.0",
+ url="https://github.com/GoogleCloudPlatform/google-cloud-python",
+ classifiers=[
+ release_status,
+ "Intended Audience :: Developers",
+ "License :: OSI Approved :: Apache Software License",
+ "Programming Language :: Python",
+ "Programming Language :: Python :: 2",
+ "Programming Language :: Python :: 2.7",
+ "Programming Language :: Python :: 3",
+ "Programming Language :: Python :: 3.5",
+ "Programming Language :: Python :: 3.6",
+ "Programming Language :: Python :: 3.7",
+ "Operating System :: OS Independent",
+ "Topic :: Internet",
+ ],
+ platforms="Posix; MacOS X; Windows",
+ packages=packages,
+ namespace_packages=namespaces,
+ install_requires=dependencies,
+ extras_require=extras,
+ python_requires=">=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*",
+ include_package_data=True,
+ zip_safe=False,
+)
diff --git a/bigquery_storage/synth.metadata b/bigquery_storage/synth.metadata
new file mode 100644
index 000000000000..5695e76c7e81
--- /dev/null
+++ b/bigquery_storage/synth.metadata
@@ -0,0 +1,39 @@
+{
+ "updateTime": "2019-08-21T23:20:45.275738Z",
+ "sources": [
+ {
+ "generator": {
+ "name": "artman",
+ "version": "0.34.0",
+ "dockerImage": "googleapis/artman@sha256:38a27ba6245f96c3e86df7acb2ebcc33b4f186d9e475efe2d64303aec3d4e0ea"
+ }
+ },
+ {
+ "git": {
+ "name": "googleapis",
+ "remote": "git@github.com:googleapis/googleapis.git",
+ "sha": "92bebf78345af8b2d3585220527115bda8bdedf8",
+ "internalRef": "264715111"
+ }
+ },
+ {
+ "template": {
+ "name": "python_library",
+ "origin": "synthtool.gcp",
+ "version": "2019.5.2"
+ }
+ }
+ ],
+ "destinations": [
+ {
+ "client": {
+ "source": "googleapis",
+ "apiName": "bigquery_storage",
+ "apiVersion": "v1beta1",
+ "language": "python",
+ "generator": "gapic",
+ "config": "google/cloud/bigquery/storage/artman_bigquerystorage_v1beta1.yaml"
+ }
+ }
+ ]
+}
\ No newline at end of file
diff --git a/bigquery_storage/synth.py b/bigquery_storage/synth.py
new file mode 100644
index 000000000000..cbbeffd02acf
--- /dev/null
+++ b/bigquery_storage/synth.py
@@ -0,0 +1,136 @@
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""This script is used to synthesize generated parts of this library."""
+
+import synthtool as s
+from synthtool import gcp
+
+gapic = gcp.GAPICGenerator()
+common = gcp.CommonTemplates()
+version = "v1beta1"
+
+library = gapic.py_library(
+ "bigquery_storage",
+ version,
+ config_path="/google/cloud/bigquery/storage/" "artman_bigquerystorage_v1beta1.yaml",
+ artman_output_name="bigquerystorage-v1beta1",
+ include_protos=True,
+)
+
+s.move(
+ library,
+ excludes=[
+ "docs/conf.py",
+ "docs/index.rst",
+ "google/cloud/bigquery_storage_v1beta1/__init__.py",
+ "README.rst",
+ "nox*.py",
+ "setup.py",
+ "setup.cfg",
+ ],
+)
+
+s.replace(
+ [
+ "google/cloud/bigquery_storage_v1beta1/proto/storage_pb2.py",
+ "google/cloud/bigquery_storage_v1beta1/proto/storage_pb2_grpc.py",
+ ],
+ "from google.cloud.bigquery.storage_v1beta1.proto",
+ "from google.cloud.bigquery_storage_v1beta1.proto",
+)
+
+s.replace(
+ "google/cloud/bigquery_storage_v1beta1/gapic/" "big_query_storage_client.py",
+ "google-cloud-bigquerystorage",
+ "google-cloud-bigquery-storage",
+)
+
+s.replace(
+ "google/cloud/bigquery_storage_v1beta1/gapic/" "big_query_storage_client.py",
+ "import google.api_core.gapic_v1.method\n",
+ "\g<0>import google.api_core.path_template\n",
+)
+
+s.replace(
+ ["tests/unit/gapic/v1beta1/test_big_query_storage_client_v1beta1.py"],
+ "from google.cloud import bigquery_storage_v1beta1",
+ "from google.cloud.bigquery_storage_v1beta1.gapic import big_query_storage_client # noqa",
+)
+
+s.replace(
+ ["tests/unit/gapic/v1beta1/test_big_query_storage_client_v1beta1.py"],
+ "bigquery_storage_v1beta1.BigQueryStorageClient",
+ "big_query_storage_client.BigQueryStorageClient",
+)
+
+# START: Ignore lint and coverage
+s.replace(
+ ["google/cloud/bigquery_storage_v1beta1/gapic/big_query_storage_client.py"],
+ "if transport:",
+ "if transport: # pragma: no cover",
+)
+
+s.replace(
+ ["google/cloud/bigquery_storage_v1beta1/gapic/big_query_storage_client.py"],
+ r"to_grpc_metadata\(\n",
+ "to_grpc_metadata( # pragma: no cover\n",
+)
+
+s.replace(
+ ["google/cloud/bigquery_storage_v1beta1/gapic/big_query_storage_client.py"],
+ r"metadata.append\(routing_metadata\)",
+ "metadata.append(routing_metadata) # pragma: no cover",
+)
+
+s.replace(
+ [
+ "google/cloud/bigquery_storage_v1beta1/gapic/transports/big_query_storage_grpc_transport.py"
+ ],
+ "if channel is not None and credentials is not None:",
+ "if channel is not None and credentials is not None: # pragma: no cover",
+)
+
+s.replace(
+ [
+ "google/cloud/bigquery_storage_v1beta1/gapic/transports/big_query_storage_grpc_transport.py"
+ ],
+ "if channel is None:",
+ "if channel is None: # pragma: no cover",
+)
+
+s.replace(
+ [
+ "google/cloud/bigquery_storage_v1beta1/gapic/transports/big_query_storage_grpc_transport.py"
+ ],
+ r"google.api_core.grpc_helpers.create_channel\(",
+ "google.api_core.grpc_helpers.create_channel( # pragma: no cover",
+)
+# END: Ignore lint and coverage
+
+# ----------------------------------------------------------------------------
+# Add templated files
+# ----------------------------------------------------------------------------
+optional_deps = [".[fastavro,pandas,pyarrow]"]
+system_test_deps = optional_deps + ["../bigquery/"]
+templated_files = common.py_library(
+ unit_cov_level=79,
+ cov_level=79,
+ samples_test=True,
+ system_test_dependencies=system_test_deps,
+ unit_test_dependencies=optional_deps,
+)
+s.move(templated_files)
+
+s.shell.run(["nox", "-s", "blacken"], hide_output=False)
diff --git a/bigquery_storage/tests/system/assets/people_data.csv b/bigquery_storage/tests/system/assets/people_data.csv
new file mode 100644
index 000000000000..819adfc4bdf5
--- /dev/null
+++ b/bigquery_storage/tests/system/assets/people_data.csv
@@ -0,0 +1,6 @@
+first_name,last_name,age
+John,Doe,42
+Jack,Black,53
+Nick,Sleek,24
+Kevin,Powell,50
+Johnny,Young,2
diff --git a/bigquery_storage/tests/system/conftest.py b/bigquery_storage/tests/system/conftest.py
new file mode 100644
index 000000000000..5ca85c1b42c8
--- /dev/null
+++ b/bigquery_storage/tests/system/conftest.py
@@ -0,0 +1,233 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""System tests for reading rows from tables."""
+
+import os
+import uuid
+
+import pytest
+
+from google.cloud import bigquery_storage_v1beta1
+
+
+_ASSETS_DIR = os.path.join(os.path.abspath(os.path.dirname(__file__)), "assets")
+
+
+@pytest.fixture(scope="session")
+def project_id():
+ return os.environ["PROJECT_ID"]
+
+
+@pytest.fixture(scope="session")
+def credentials():
+ from google.oauth2 import service_account
+
+ # NOTE: the test config in noxfile checks that the env variable is indeed set
+ filename = os.environ["GOOGLE_APPLICATION_CREDENTIALS"]
+ return service_account.Credentials.from_service_account_file(filename)
+
+
+@pytest.fixture(scope="session")
+def bq_client(credentials):
+ from google.cloud import bigquery
+
+ return bigquery.Client(credentials=credentials)
+
+
+@pytest.fixture(scope="session")
+def dataset(project_id, bq_client):
+ from google.cloud import bigquery
+
+ unique_suffix = str(uuid.uuid4()).replace("-", "_")
+ dataset_name = "bq_storage_system_tests_" + unique_suffix
+
+ dataset_id = "{}.{}".format(project_id, dataset_name)
+ dataset = bigquery.Dataset(dataset_id)
+ dataset.location = "US"
+ created_dataset = bq_client.create_dataset(dataset)
+
+ yield created_dataset
+
+ bq_client.delete_dataset(dataset, delete_contents=True)
+
+
+@pytest.fixture(scope="session")
+def table(project_id, dataset, bq_client):
+ from google.cloud import bigquery
+
+ schema = [
+ bigquery.SchemaField("first_name", "STRING", mode="NULLABLE"),
+ bigquery.SchemaField("last_name", "STRING", mode="NULLABLE"),
+ bigquery.SchemaField("age", "INTEGER", mode="NULLABLE"),
+ ]
+
+ table_id = "{}.{}.{}".format(project_id, dataset.dataset_id, "users")
+ bq_table = bigquery.Table(table_id, schema=schema)
+ created_table = bq_client.create_table(bq_table)
+
+ yield created_table
+
+ bq_client.delete_table(created_table)
+
+
+@pytest.fixture
+def table_with_data_ref(dataset, table, bq_client):
+ from google.cloud import bigquery
+
+ job_config = bigquery.LoadJobConfig()
+ job_config.source_format = bigquery.SourceFormat.CSV
+ job_config.skip_leading_rows = 1
+ job_config.schema = table.schema
+
+ filename = os.path.join(_ASSETS_DIR, "people_data.csv")
+
+ with open(filename, "rb") as source_file:
+ job = bq_client.load_table_from_file(source_file, table, job_config=job_config)
+
+ job.result() # wait for the load to complete
+
+ table_ref = bigquery_storage_v1beta1.types.TableReference()
+ table_ref.project_id = table.project
+ table_ref.dataset_id = table.dataset_id
+ table_ref.table_id = table.table_id
+ yield table_ref
+
+ # truncate table data
+ query = "DELETE FROM {}.{} WHERE 1 = 1".format(dataset.dataset_id, table.table_id)
+ query_job = bq_client.query(query, location="US")
+ query_job.result()
+
+
+@pytest.fixture
+def col_partition_table_ref(project_id, dataset, bq_client):
+ from google.cloud import bigquery
+
+ schema = [
+ bigquery.SchemaField("occurred", "DATE", mode="NULLABLE"),
+ bigquery.SchemaField("description", "STRING", mode="NULLABLE"),
+ ]
+ time_partitioning = bigquery.table.TimePartitioning(
+ type_=bigquery.table.TimePartitioningType.DAY, field="occurred"
+ )
+ bq_table = bigquery.table.Table(
+ table_ref="{}.{}.notable_events".format(project_id, dataset.dataset_id),
+ schema=schema,
+ )
+ bq_table.time_partitioning = time_partitioning
+
+ created_table = bq_client.create_table(bq_table)
+
+ table_ref = bigquery_storage_v1beta1.types.TableReference()
+ table_ref.project_id = created_table.project
+ table_ref.dataset_id = created_table.dataset_id
+ table_ref.table_id = created_table.table_id
+ yield table_ref
+
+ bq_client.delete_table(created_table)
+
+
+@pytest.fixture
+def ingest_partition_table_ref(project_id, dataset, bq_client):
+ from google.cloud import bigquery
+
+ schema = [
+ bigquery.SchemaField("shape", "STRING", mode="NULLABLE"),
+ bigquery.SchemaField("altitude", "INT64", mode="NULLABLE"),
+ ]
+ time_partitioning = bigquery.table.TimePartitioning(
+ type_=bigquery.table.TimePartitioningType.DAY,
+ field=None, # use _PARTITIONTIME pseudo column
+ )
+ bq_table = bigquery.table.Table(
+ table_ref="{}.{}.ufo_sightings".format(project_id, dataset.dataset_id),
+ schema=schema,
+ )
+ bq_table.time_partitioning = time_partitioning
+
+ created_table = bq_client.create_table(bq_table)
+
+ table_ref = bigquery_storage_v1beta1.types.TableReference()
+ table_ref.project_id = created_table.project
+ table_ref.dataset_id = created_table.dataset_id
+ table_ref.table_id = created_table.table_id
+ yield table_ref
+
+ bq_client.delete_table(created_table)
+
+
+@pytest.fixture
+def all_types_table_ref(project_id, dataset, bq_client):
+ from google.cloud import bigquery
+
+ schema = [
+ bigquery.SchemaField("string_field", "STRING"),
+ bigquery.SchemaField("bytes_field", "BYTES"),
+ bigquery.SchemaField("int64_field", "INT64"),
+ bigquery.SchemaField("float64_field", "FLOAT64"),
+ bigquery.SchemaField("numeric_field", "NUMERIC"),
+ bigquery.SchemaField("bool_field", "BOOL"),
+ bigquery.SchemaField("geography_field", "GEOGRAPHY"),
+ bigquery.SchemaField(
+ "person_struct_field",
+ "STRUCT",
+ fields=(
+ bigquery.SchemaField("name", "STRING"),
+ bigquery.SchemaField("age", "INT64"),
+ ),
+ ),
+ bigquery.SchemaField("timestamp_field", "TIMESTAMP"),
+ bigquery.SchemaField("date_field", "DATE"),
+ bigquery.SchemaField("time_field", "TIME"),
+ bigquery.SchemaField("datetime_field", "DATETIME"),
+ bigquery.SchemaField("string_array_field", "STRING", mode="REPEATED"),
+ ]
+ bq_table = bigquery.table.Table(
+ table_ref="{}.{}.complex_records".format(project_id, dataset.dataset_id),
+ schema=schema,
+ )
+
+ created_table = bq_client.create_table(bq_table)
+
+ table_ref = bigquery_storage_v1beta1.types.TableReference()
+ table_ref.project_id = created_table.project
+ table_ref.dataset_id = created_table.dataset_id
+ table_ref.table_id = created_table.table_id
+ yield table_ref
+
+ bq_client.delete_table(created_table)
+
+
+@pytest.fixture(scope="session")
+def client(credentials):
+ return bigquery_storage_v1beta1.BigQueryStorageClient(credentials=credentials)
+
+
+@pytest.fixture()
+def table_reference():
+ table_ref = bigquery_storage_v1beta1.types.TableReference()
+ table_ref.project_id = "bigquery-public-data"
+ table_ref.dataset_id = "usa_names"
+ table_ref.table_id = "usa_1910_2013"
+ return table_ref
+
+
+@pytest.fixture()
+def small_table_reference():
+ table_ref = bigquery_storage_v1beta1.types.TableReference()
+ table_ref.project_id = "bigquery-public-data"
+ table_ref.dataset_id = "utility_us"
+ table_ref.table_id = "country_code_iso"
+ return table_ref
diff --git a/bigquery_storage/tests/system/test_reader.py b/bigquery_storage/tests/system/test_reader.py
new file mode 100644
index 000000000000..2ba1f99fb0a0
--- /dev/null
+++ b/bigquery_storage/tests/system/test_reader.py
@@ -0,0 +1,456 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""System tests for reading rows from tables."""
+
+import copy
+import datetime as dt
+import decimal
+import re
+
+import pytest
+import pytz
+
+from google.cloud import bigquery
+from google.cloud import bigquery_storage_v1beta1
+from google.protobuf import timestamp_pb2
+
+
+def _to_bq_table_ref(proto_table_ref, partition_suffix=""):
+ """Converts protobuf table reference to bigquery table reference.
+
+ Args:
+ proto_table_ref (bigquery_storage_v1beta1.types.TableReference):
+ A protobuf reference to a table.
+ partition_suffix (str):
+ An optional suffix to append to the table_id, useful for selecting
+ partitions of ingestion-time partitioned tables.
+
+ Returns:
+ google.cloud.bigquery.table.TableReference
+ """
+ return bigquery.table.TableReference.from_api_repr(
+ {
+ "projectId": proto_table_ref.project_id,
+ "datasetId": proto_table_ref.dataset_id,
+ "tableId": proto_table_ref.table_id + partition_suffix,
+ }
+ )
+
+
+@pytest.mark.parametrize(
+ "data_format,expected_schema_type",
+ (
+ (None, "avro_schema"), # Default format (Avro).
+ (bigquery_storage_v1beta1.enums.DataFormat.AVRO, "avro_schema"),
+ (bigquery_storage_v1beta1.enums.DataFormat.ARROW, "arrow_schema"),
+ ),
+)
+def test_read_rows_as_blocks_full_table(
+ client, project_id, small_table_reference, data_format, expected_schema_type
+):
+ session = client.create_read_session(
+ small_table_reference,
+ "projects/{}".format(project_id),
+ format_=data_format,
+ requested_streams=1,
+ )
+ stream_pos = bigquery_storage_v1beta1.types.StreamPosition(
+ stream=session.streams[0]
+ )
+ schema_type = session.WhichOneof("schema")
+ assert schema_type == expected_schema_type
+
+ blocks = list(client.read_rows(stream_pos))
+
+ assert len(blocks) > 0
+ block = blocks[0]
+ assert block.status.estimated_row_count > 0
+
+
+@pytest.mark.parametrize(
+ "data_format,expected_schema_type",
+ (
+ (bigquery_storage_v1beta1.enums.DataFormat.AVRO, "avro_schema"),
+ (bigquery_storage_v1beta1.enums.DataFormat.ARROW, "arrow_schema"),
+ ),
+)
+def test_read_rows_as_rows_full_table(
+ client, project_id, small_table_reference, data_format, expected_schema_type
+):
+ session = client.create_read_session(
+ small_table_reference,
+ "projects/{}".format(project_id),
+ format_=data_format,
+ requested_streams=1,
+ )
+ stream_pos = bigquery_storage_v1beta1.types.StreamPosition(
+ stream=session.streams[0]
+ )
+
+ rows = list(client.read_rows(stream_pos).rows(session))
+
+ assert len(rows) > 0
+
+
+@pytest.mark.parametrize(
+ "data_format",
+ (
+ (bigquery_storage_v1beta1.enums.DataFormat.AVRO),
+ (bigquery_storage_v1beta1.enums.DataFormat.ARROW),
+ ),
+)
+def test_basic_nonfiltered_read(client, project_id, table_with_data_ref, data_format):
+ session = client.create_read_session(
+ table_with_data_ref,
+ "projects/{}".format(project_id),
+ format_=data_format,
+ requested_streams=1,
+ )
+ stream_pos = bigquery_storage_v1beta1.types.StreamPosition(
+ stream=session.streams[0]
+ )
+
+ rows = list(client.read_rows(stream_pos).rows(session))
+
+ assert len(rows) == 5 # all table rows
+
+
+def test_filtered_rows_read(client, project_id, table_with_data_ref):
+ read_options = bigquery_storage_v1beta1.types.TableReadOptions()
+ read_options.row_restriction = "age >= 50"
+
+ session = client.create_read_session(
+ table_with_data_ref,
+ "projects/{}".format(project_id),
+ format_=bigquery_storage_v1beta1.enums.DataFormat.AVRO,
+ requested_streams=1,
+ read_options=read_options,
+ )
+ stream_pos = bigquery_storage_v1beta1.types.StreamPosition(
+ stream=session.streams[0]
+ )
+
+ rows = list(client.read_rows(stream_pos).rows(session))
+
+ assert len(rows) == 2
+
+
+@pytest.mark.parametrize(
+ "data_format",
+ (
+ (bigquery_storage_v1beta1.enums.DataFormat.AVRO),
+ (bigquery_storage_v1beta1.enums.DataFormat.ARROW),
+ ),
+)
+def test_column_selection_read(client, project_id, table_with_data_ref, data_format):
+ read_options = bigquery_storage_v1beta1.types.TableReadOptions()
+ read_options.selected_fields.append("first_name")
+ read_options.selected_fields.append("age")
+
+ session = client.create_read_session(
+ table_with_data_ref,
+ "projects/{}".format(project_id),
+ format_=data_format,
+ requested_streams=1,
+ read_options=read_options,
+ )
+ stream_pos = bigquery_storage_v1beta1.types.StreamPosition(
+ stream=session.streams[0]
+ )
+
+ rows = list(client.read_rows(stream_pos).rows(session))
+
+ for row in rows:
+ assert sorted(row.keys()) == ["age", "first_name"]
+
+
+def test_snapshot(client, project_id, table_with_data_ref, bq_client):
+ before_new_data = timestamp_pb2.Timestamp()
+ before_new_data.GetCurrentTime()
+
+ # load additional data into the table
+ new_data = [
+ {u"first_name": u"NewGuyFoo", u"last_name": u"Smith", u"age": 46},
+ {u"first_name": u"NewGuyBar", u"last_name": u"Jones", u"age": 30},
+ ]
+
+ destination = _to_bq_table_ref(table_with_data_ref)
+ bq_client.load_table_from_json(new_data, destination).result()
+
+ # read data using the timestamp before the additional data load
+ session = client.create_read_session(
+ table_with_data_ref,
+ "projects/{}".format(project_id),
+ format_=bigquery_storage_v1beta1.enums.DataFormat.AVRO,
+ requested_streams=1,
+ table_modifiers={"snapshot_time": before_new_data},
+ )
+ stream_pos = bigquery_storage_v1beta1.types.StreamPosition(
+ stream=session.streams[0]
+ )
+
+ rows = list(client.read_rows(stream_pos).rows(session))
+
+ # verify that only the data before the timestamp was returned
+ assert len(rows) == 5 # all initial records
+
+ for row in rows:
+ assert "NewGuy" not in row["first_name"] # no new records
+
+
+def test_column_partitioned_table(
+ client, project_id, col_partition_table_ref, bq_client
+):
+ data = [
+ {"description": "Tracking established.", "occurred": "2017-02-15"},
+ {"description": "Look, a solar eclipse!", "occurred": "2018-02-15"},
+ {"description": "Fake solar eclipse reported.", "occurred": "2018-02-15"},
+ {"description": "1 day after false eclipse report.", "occurred": "2018-02-16"},
+ {"description": "1 year after false eclipse report.", "occurred": "2019-02-15"},
+ ]
+
+ destination = _to_bq_table_ref(col_partition_table_ref)
+ bq_client.load_table_from_json(data, destination).result()
+
+ # Read from the table with a partition filter specified, and verify that
+ # only the expected data is returned.
+ read_options = bigquery_storage_v1beta1.types.TableReadOptions()
+ read_options.row_restriction = "occurred = '2018-02-15'"
+
+ session = client.create_read_session(
+ col_partition_table_ref,
+ "projects/{}".format(project_id),
+ format_=bigquery_storage_v1beta1.enums.DataFormat.AVRO,
+ requested_streams=1,
+ read_options=read_options,
+ )
+
+ assert session.streams # there should be some data to fetch
+
+ stream_pos = bigquery_storage_v1beta1.types.StreamPosition(
+ stream=session.streams[0]
+ )
+ rows = list(client.read_rows(stream_pos).rows(session))
+
+ assert len(rows) == 2
+
+ expected_descriptions = ("Look, a solar eclipse!", "Fake solar eclipse reported.")
+ for row in rows:
+ assert row["occurred"] == dt.date(2018, 2, 15)
+ assert row["description"] in expected_descriptions
+
+
+@pytest.mark.parametrize(
+ "data_format",
+ (
+ (bigquery_storage_v1beta1.enums.DataFormat.AVRO),
+ (bigquery_storage_v1beta1.enums.DataFormat.ARROW),
+ ),
+)
+def test_ingestion_time_partitioned_table(
+ client, project_id, ingest_partition_table_ref, bq_client, data_format
+):
+ data = [{"shape": "cigar", "altitude": 1200}, {"shape": "disc", "altitude": 750}]
+ destination = _to_bq_table_ref(
+ ingest_partition_table_ref, partition_suffix="$20190809"
+ )
+ bq_client.load_table_from_json(data, destination).result()
+
+ data = [
+ {"shape": "sphere", "altitude": 3500},
+ {"shape": "doughnut", "altitude": 100},
+ ]
+ destination = _to_bq_table_ref(
+ ingest_partition_table_ref, partition_suffix="$20190810"
+ )
+ bq_client.load_table_from_json(data, destination).result()
+
+ data = [
+ {"shape": "elephant", "altitude": 1},
+ {"shape": "rocket", "altitude": 12700},
+ ]
+ destination = _to_bq_table_ref(
+ ingest_partition_table_ref, partition_suffix="$20190811"
+ )
+ bq_client.load_table_from_json(data, destination).result()
+
+ read_options = bigquery_storage_v1beta1.types.TableReadOptions()
+ read_options.row_restriction = "DATE(_PARTITIONTIME) = '2019-08-10'"
+
+ session = client.create_read_session(
+ ingest_partition_table_ref,
+ "projects/{}".format(project_id),
+ format_=data_format,
+ requested_streams=1,
+ read_options=read_options,
+ )
+
+ assert session.streams # there should be some data to fetch
+
+ stream_pos = bigquery_storage_v1beta1.types.StreamPosition(
+ stream=session.streams[0]
+ )
+ rows = list(client.read_rows(stream_pos).rows(session))
+ assert len(rows) == 2
+
+ actual_items = {(row["shape"], row["altitude"]) for row in rows}
+ expected_items = {("sphere", 3500), ("doughnut", 100)}
+ assert actual_items == expected_items
+
+
+@pytest.mark.parametrize(
+ "data_format",
+ (
+ (bigquery_storage_v1beta1.enums.DataFormat.AVRO),
+ (bigquery_storage_v1beta1.enums.DataFormat.ARROW),
+ ),
+)
+def test_decoding_data_types(
+ client, project_id, all_types_table_ref, bq_client, data_format
+):
+ data = [
+ {
+ u"string_field": u"Price: € 9.95.",
+ u"bytes_field": bigquery._helpers._bytes_to_json(b"byteees"),
+ u"int64_field": -1085,
+ u"float64_field": -42.195,
+ u"numeric_field": "1.4142",
+ u"bool_field": True,
+ u"geography_field": '{"type": "Point", "coordinates": [-49.3028, 69.0622]}',
+ u"person_struct_field": {u"name": u"John", u"age": 42},
+ u"timestamp_field": 1565357902.017896, # 2019-08-09T13:38:22.017896
+ u"date_field": u"1995-03-17",
+ u"time_field": u"16:24:51",
+ u"datetime_field": u"2005-10-26T19:49:41",
+ u"string_array_field": [u"foo", u"bar", u"baz"],
+ }
+ ]
+
+ # Explicit schema is needed to recognize bytes_field as BYTES, and not STRING.
+ # Since partial schemas are not supported in load_table_from_json(), a full
+ # schema needs to be specified.
+ schema = [
+ bigquery.SchemaField("string_field", "STRING"),
+ bigquery.SchemaField("bytes_field", "BYTES"),
+ bigquery.SchemaField("int64_field", "INT64"),
+ bigquery.SchemaField("float64_field", "FLOAT64"),
+ bigquery.SchemaField("numeric_field", "NUMERIC"),
+ bigquery.SchemaField("bool_field", "BOOL"),
+ bigquery.SchemaField("geography_field", "GEOGRAPHY"),
+ bigquery.SchemaField(
+ "person_struct_field",
+ "STRUCT",
+ fields=(
+ bigquery.SchemaField("name", "STRING"),
+ bigquery.SchemaField("age", "INT64"),
+ ),
+ ),
+ bigquery.SchemaField("timestamp_field", "TIMESTAMP"),
+ bigquery.SchemaField("date_field", "DATE"),
+ bigquery.SchemaField("time_field", "TIME"),
+ bigquery.SchemaField("datetime_field", "DATETIME"),
+ bigquery.SchemaField("string_array_field", "STRING", mode="REPEATED"),
+ ]
+
+ job_config = bigquery.LoadJobConfig(schema=schema)
+ destination = _to_bq_table_ref(all_types_table_ref)
+ bq_client.load_table_from_json(data, destination, job_config=job_config).result()
+
+ session = client.create_read_session(
+ all_types_table_ref,
+ "projects/{}".format(project_id),
+ format_=data_format,
+ requested_streams=1,
+ )
+
+ assert session.streams # there should be data available
+
+ stream_pos = bigquery_storage_v1beta1.types.StreamPosition(
+ stream=session.streams[0]
+ )
+
+ rows = list(client.read_rows(stream_pos).rows(session))
+
+ expected_result = {
+ u"string_field": u"Price: € 9.95.",
+ u"bytes_field": b"byteees",
+ u"int64_field": -1085,
+ u"float64_field": -42.195,
+ u"numeric_field": decimal.Decimal("1.4142"),
+ u"bool_field": True,
+ u"geography_field": "POINT(-49.3028 69.0622)",
+ u"person_struct_field": {u"name": u"John", u"age": 42},
+ u"timestamp_field": dt.datetime(2019, 8, 9, 13, 38, 22, 17896, tzinfo=pytz.UTC),
+ u"date_field": dt.date(1995, 3, 17),
+ u"time_field": dt.time(16, 24, 51),
+ u"string_array_field": [u"foo", u"bar", u"baz"],
+ }
+
+ result_copy = copy.copy(rows[0])
+ del result_copy["datetime_field"]
+ assert result_copy == expected_result
+
+ # Compare datetime separately, AVRO and PYARROW return different object types,
+ # although they should both represent the same value.
+ # TODO: when fixed, change assertion to assert a datetime instance!
+ expected_pattern = re.compile(r"2005-10-26( |T)19:49:41")
+ assert expected_pattern.match(str(rows[0]["datetime_field"]))
+
+
+@pytest.mark.parametrize(
+ "data_format",
+ (
+ (bigquery_storage_v1beta1.enums.DataFormat.AVRO),
+ (bigquery_storage_v1beta1.enums.DataFormat.ARROW),
+ ),
+)
+def test_resuming_read_from_offset(client, project_id, data_format):
+ shakespeare_ref = bigquery_storage_v1beta1.types.TableReference()
+ shakespeare_ref.project_id = project_id
+ shakespeare_ref.dataset_id = "public_samples_copy"
+ shakespeare_ref.table_id = "shakespeare"
+
+ read_session = client.create_read_session(
+ shakespeare_ref,
+ "projects/{}".format(project_id),
+ format_=data_format,
+ requested_streams=1,
+ )
+
+ assert read_session.streams # there should be data available
+
+ stream_pos = bigquery_storage_v1beta1.types.StreamPosition(
+ stream=read_session.streams[0], offset=0
+ )
+ read_rows_stream = client.read_rows(stream_pos)
+
+ # fetch the first two batches of rows
+ rows_iter = iter(read_rows_stream)
+ some_rows = next(rows_iter)
+ more_rows = next(rows_iter)
+
+ # fetch the rest of the rows using the stream offset
+ new_stream_pos = bigquery_storage_v1beta1.types.StreamPosition(
+ stream=read_session.streams[0], offset=some_rows.row_count + more_rows.row_count
+ )
+ remaining_rows_count = sum(
+ 1 for _ in client.read_rows(new_stream_pos).rows(read_session)
+ )
+
+ # verify that the counts match
+ expected_len = 164656 # total rows in shakespeare table
+ actual_len = remaining_rows_count + some_rows.row_count + more_rows.row_count
+ assert actual_len == expected_len
diff --git a/bigquery_storage/tests/system/test_reader_dataframe.py b/bigquery_storage/tests/system/test_reader_dataframe.py
new file mode 100644
index 000000000000..07dcab384c05
--- /dev/null
+++ b/bigquery_storage/tests/system/test_reader_dataframe.py
@@ -0,0 +1,93 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""System tests for reading rows with pandas connector."""
+
+import numpy
+import pyarrow.types
+import pytest
+
+from google.cloud import bigquery_storage_v1beta1
+
+
+def test_read_rows_to_arrow(client, project_id):
+ table_ref = bigquery_storage_v1beta1.types.TableReference()
+ table_ref.project_id = "bigquery-public-data"
+ table_ref.dataset_id = "new_york_citibike"
+ table_ref.table_id = "citibike_stations"
+
+ read_options = bigquery_storage_v1beta1.types.TableReadOptions()
+ read_options.selected_fields.append("station_id")
+ read_options.selected_fields.append("latitude")
+ read_options.selected_fields.append("longitude")
+ read_options.selected_fields.append("name")
+ session = client.create_read_session(
+ table_ref,
+ "projects/{}".format(project_id),
+ format_=bigquery_storage_v1beta1.enums.DataFormat.ARROW,
+ read_options=read_options,
+ requested_streams=1,
+ )
+ stream_pos = bigquery_storage_v1beta1.types.StreamPosition(
+ stream=session.streams[0]
+ )
+
+ tbl = client.read_rows(stream_pos).to_arrow(session)
+
+ assert tbl.num_columns == 4
+ schema = tbl.schema
+ # Use field_by_name because the order doesn't currently match that of
+ # selected_fields.
+ assert pyarrow.types.is_int64(schema.field_by_name("station_id").type)
+ assert pyarrow.types.is_float64(schema.field_by_name("latitude").type)
+ assert pyarrow.types.is_float64(schema.field_by_name("longitude").type)
+ assert pyarrow.types.is_string(schema.field_by_name("name").type)
+
+
+@pytest.mark.parametrize(
+ "data_format,expected_schema_type",
+ (
+ (bigquery_storage_v1beta1.enums.DataFormat.AVRO, "avro_schema"),
+ (bigquery_storage_v1beta1.enums.DataFormat.ARROW, "arrow_schema"),
+ ),
+)
+def test_read_rows_to_dataframe(client, project_id, data_format, expected_schema_type):
+ table_ref = bigquery_storage_v1beta1.types.TableReference()
+ table_ref.project_id = "bigquery-public-data"
+ table_ref.dataset_id = "new_york_citibike"
+ table_ref.table_id = "citibike_stations"
+ session = client.create_read_session(
+ table_ref,
+ "projects/{}".format(project_id),
+ format_=data_format,
+ requested_streams=1,
+ )
+ schema_type = session.WhichOneof("schema")
+ assert schema_type == expected_schema_type
+
+ stream_pos = bigquery_storage_v1beta1.types.StreamPosition(
+ stream=session.streams[0]
+ )
+
+ frame = client.read_rows(stream_pos).to_dataframe(
+ session, dtypes={"latitude": numpy.float16}
+ )
+
+ # Station ID is a required field (no nulls), so the datatype should always
+ # be integer.
+ assert frame.station_id.dtype.name == "int64"
+ assert frame.latitude.dtype.name == "float16"
+ assert frame.longitude.dtype.name == "float64"
+ assert frame["name"].str.startswith("Central Park").any()
diff --git a/bigquery_storage/tests/unit/gapic/v1beta1/test_big_query_storage_client_v1beta1.py b/bigquery_storage/tests/unit/gapic/v1beta1/test_big_query_storage_client_v1beta1.py
new file mode 100644
index 000000000000..e7870b1d25d6
--- /dev/null
+++ b/bigquery_storage/tests/unit/gapic/v1beta1/test_big_query_storage_client_v1beta1.py
@@ -0,0 +1,261 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Unit tests."""
+
+import mock
+import pytest
+
+from google.cloud.bigquery_storage_v1beta1.gapic import big_query_storage_client # noqa
+from google.cloud.bigquery_storage_v1beta1.proto import storage_pb2
+from google.cloud.bigquery_storage_v1beta1.proto import table_reference_pb2
+from google.protobuf import empty_pb2
+
+
+class MultiCallableStub(object):
+ """Stub for the grpc.UnaryUnaryMultiCallable interface."""
+
+ def __init__(self, method, channel_stub):
+ self.method = method
+ self.channel_stub = channel_stub
+
+ def __call__(self, request, timeout=None, metadata=None, credentials=None):
+ self.channel_stub.requests.append((self.method, request))
+
+ response = None
+ if self.channel_stub.responses:
+ response = self.channel_stub.responses.pop()
+
+ if isinstance(response, Exception):
+ raise response
+
+ if response:
+ return response
+
+
+class ChannelStub(object):
+ """Stub for the grpc.Channel interface."""
+
+ def __init__(self, responses=[]):
+ self.responses = responses
+ self.requests = []
+
+ def unary_unary(self, method, request_serializer=None, response_deserializer=None):
+ return MultiCallableStub(method, self)
+
+ def unary_stream(self, method, request_serializer=None, response_deserializer=None):
+ return MultiCallableStub(method, self)
+
+
+class CustomException(Exception):
+ pass
+
+
+class TestBigQueryStorageClient(object):
+ def test_create_read_session(self):
+ # Setup Expected Response
+ name = "name3373707"
+ expected_response = {"name": name}
+ expected_response = storage_pb2.ReadSession(**expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = big_query_storage_client.BigQueryStorageClient()
+
+ # Setup Request
+ table_reference = {}
+ parent = "parent-995424086"
+
+ response = client.create_read_session(table_reference, parent)
+ assert expected_response == response
+
+ assert len(channel.requests) == 1
+ expected_request = storage_pb2.CreateReadSessionRequest(
+ table_reference=table_reference, parent=parent
+ )
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_create_read_session_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = big_query_storage_client.BigQueryStorageClient()
+
+ # Setup request
+ table_reference = {}
+ parent = "parent-995424086"
+
+ with pytest.raises(CustomException):
+ client.create_read_session(table_reference, parent)
+
+ def test_read_rows(self):
+ # Setup Expected Response
+ row_count = 1340416618
+ expected_response = {"row_count": row_count}
+ expected_response = storage_pb2.ReadRowsResponse(**expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[iter([expected_response])])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = big_query_storage_client.BigQueryStorageClient()
+
+ # Setup Request
+ read_position = {}
+
+ response = client.read_rows(read_position)
+ resources = list(response)
+ assert len(resources) == 1
+ assert expected_response == resources[0]
+
+ assert len(channel.requests) == 1
+ expected_request = storage_pb2.ReadRowsRequest(read_position=read_position)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_read_rows_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = big_query_storage_client.BigQueryStorageClient()
+
+ # Setup request
+ read_position = {}
+
+ with pytest.raises(CustomException):
+ client.read_rows(read_position)
+
+ def test_batch_create_read_session_streams(self):
+ # Setup Expected Response
+ expected_response = {}
+ expected_response = storage_pb2.BatchCreateReadSessionStreamsResponse(
+ **expected_response
+ )
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = big_query_storage_client.BigQueryStorageClient()
+
+ # Setup Request
+ session = {}
+ requested_streams = 1017221410
+
+ response = client.batch_create_read_session_streams(session, requested_streams)
+ assert expected_response == response
+
+ assert len(channel.requests) == 1
+ expected_request = storage_pb2.BatchCreateReadSessionStreamsRequest(
+ session=session, requested_streams=requested_streams
+ )
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_batch_create_read_session_streams_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = big_query_storage_client.BigQueryStorageClient()
+
+ # Setup request
+ session = {}
+ requested_streams = 1017221410
+
+ with pytest.raises(CustomException):
+ client.batch_create_read_session_streams(session, requested_streams)
+
+ def test_finalize_stream(self):
+ channel = ChannelStub()
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = big_query_storage_client.BigQueryStorageClient()
+
+ # Setup Request
+ stream = {}
+
+ client.finalize_stream(stream)
+
+ assert len(channel.requests) == 1
+ expected_request = storage_pb2.FinalizeStreamRequest(stream=stream)
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_finalize_stream_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = big_query_storage_client.BigQueryStorageClient()
+
+ # Setup request
+ stream = {}
+
+ with pytest.raises(CustomException):
+ client.finalize_stream(stream)
+
+ def test_split_read_stream(self):
+ # Setup Expected Response
+ expected_response = {}
+ expected_response = storage_pb2.SplitReadStreamResponse(**expected_response)
+
+ # Mock the API response
+ channel = ChannelStub(responses=[expected_response])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = big_query_storage_client.BigQueryStorageClient()
+
+ # Setup Request
+ original_stream = {}
+
+ response = client.split_read_stream(original_stream)
+ assert expected_response == response
+
+ assert len(channel.requests) == 1
+ expected_request = storage_pb2.SplitReadStreamRequest(
+ original_stream=original_stream
+ )
+ actual_request = channel.requests[0][1]
+ assert expected_request == actual_request
+
+ def test_split_read_stream_exception(self):
+ # Mock the API response
+ channel = ChannelStub(responses=[CustomException()])
+ patch = mock.patch("google.api_core.grpc_helpers.create_channel")
+ with patch as create_channel:
+ create_channel.return_value = channel
+ client = big_query_storage_client.BigQueryStorageClient()
+
+ # Setup request
+ original_stream = {}
+
+ with pytest.raises(CustomException):
+ client.split_read_stream(original_stream)
diff --git a/bigquery_storage/tests/unit/test_client.py b/bigquery_storage/tests/unit/test_client.py
new file mode 100644
index 000000000000..fbce027d136e
--- /dev/null
+++ b/bigquery_storage/tests/unit/test_client.py
@@ -0,0 +1,90 @@
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from google.api_core.gapic_v1 import client_info
+import mock
+import pytest
+
+from google.cloud.bigquery_storage_v1beta1 import types
+
+
+PROJECT = "my-project"
+SERVICE_ACCOUNT_PROJECT = "project-from-credentials"
+
+
+@pytest.fixture()
+def mock_transport(monkeypatch):
+ from google.cloud.bigquery_storage_v1beta1.gapic.transports import (
+ big_query_storage_grpc_transport,
+ )
+
+ transport = mock.create_autospec(
+ big_query_storage_grpc_transport.BigQueryStorageGrpcTransport
+ )
+ return transport
+
+
+@pytest.fixture()
+def client_under_test(mock_transport):
+ from google.cloud.bigquery_storage_v1beta1 import client
+
+ # The mock is detected as a callable. By creating a real callable here, the
+ # mock can still be used to verify RPCs.
+ def transport_callable(credentials=None, default_class=None, address=None):
+ return mock_transport
+
+ return client.BigQueryStorageClient(transport=transport_callable)
+
+
+def test_constructor_w_client_info(mock_transport):
+ from google.cloud.bigquery_storage_v1beta1 import client
+
+ def transport_callable(credentials=None, default_class=None, address=None):
+ return mock_transport
+
+ client_under_test = client.BigQueryStorageClient(
+ transport=transport_callable,
+ client_info=client_info.ClientInfo(
+ client_library_version="test-client-version"
+ ),
+ )
+
+ user_agent = client_under_test._client_info.to_user_agent()
+ assert "test-client-version" in user_agent
+
+
+def test_create_read_session(mock_transport, client_under_test):
+ table_reference = types.TableReference(
+ project_id="data-project-id", dataset_id="dataset_id", table_id="table_id"
+ )
+
+ client_under_test.create_read_session(table_reference, "projects/other-project")
+
+ expected_request = types.CreateReadSessionRequest(
+ table_reference=table_reference, parent="projects/other-project"
+ )
+ mock_transport.create_read_session.assert_called_once_with(
+ expected_request, metadata=mock.ANY, timeout=mock.ANY
+ )
+
+
+def test_read_rows(mock_transport, client_under_test):
+ stream_position = types.StreamPosition()
+
+ client_under_test.read_rows(stream_position)
+
+ expected_request = types.ReadRowsRequest(read_position=stream_position)
+ mock_transport.create_read_session.read_rows(
+ expected_request, metadata=mock.ANY, timeout=mock.ANY
+ )
diff --git a/bigquery_storage/tests/unit/test_reader.py b/bigquery_storage/tests/unit/test_reader.py
new file mode 100644
index 000000000000..748a45608f3a
--- /dev/null
+++ b/bigquery_storage/tests/unit/test_reader.py
@@ -0,0 +1,738 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2018 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the 'License');
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an 'AS IS' BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import datetime
+import decimal
+import itertools
+import json
+
+import fastavro
+import pyarrow
+import mock
+import pandas
+import pandas.testing
+import pytest
+import pytz
+import six
+
+import google.api_core.exceptions
+from google.cloud import bigquery_storage_v1beta1
+
+
+PROJECT = "my-project"
+BQ_TO_AVRO_TYPES = {
+ "int64": "long",
+ "float64": "double",
+ "bool": "boolean",
+ "numeric": {"type": "bytes", "logicalType": "decimal", "precision": 38, "scale": 9},
+ "string": "string",
+ "bytes": "bytes",
+ "date": {"type": "int", "logicalType": "date"},
+ "datetime": {"type": "string", "sqlType": "DATETIME"},
+ "time": {"type": "long", "logicalType": "time-micros"},
+ "timestamp": {"type": "long", "logicalType": "timestamp-micros"},
+}
+# This dictionary is duplicated in bigquery/google/cloud/bigquery/_pandas_helpers.py
+# When modifying it be sure to update it there as well.
+BQ_TO_ARROW_TYPES = {
+ "int64": pyarrow.int64(),
+ "float64": pyarrow.float64(),
+ "bool": pyarrow.bool_(),
+ "numeric": pyarrow.decimal128(38, 9),
+ "string": pyarrow.utf8(),
+ "bytes": pyarrow.binary(),
+ "date": pyarrow.date32(), # int32 days since epoch
+ "datetime": pyarrow.timestamp("us"),
+ "time": pyarrow.time64("us"),
+ "timestamp": pyarrow.timestamp("us", tz="UTC"),
+}
+SCALAR_COLUMNS = [
+ {"name": "int_col", "type": "int64"},
+ {"name": "float_col", "type": "float64"},
+ {"name": "num_col", "type": "numeric"},
+ {"name": "bool_col", "type": "bool"},
+ {"name": "str_col", "type": "string"},
+ {"name": "bytes_col", "type": "bytes"},
+ {"name": "date_col", "type": "date"},
+ {"name": "time_col", "type": "time"},
+ {"name": "ts_col", "type": "timestamp"},
+]
+SCALAR_COLUMN_NAMES = [field["name"] for field in SCALAR_COLUMNS]
+SCALAR_BLOCKS = [
+ [
+ {
+ "int_col": 123,
+ "float_col": 3.14,
+ "num_col": decimal.Decimal("9.99"),
+ "bool_col": True,
+ "str_col": "hello world",
+ "bytes_col": b"ascii bytes",
+ "date_col": datetime.date(1998, 9, 4),
+ "time_col": datetime.time(12, 0),
+ "ts_col": datetime.datetime(2000, 1, 1, 5, 0, tzinfo=pytz.utc),
+ },
+ {
+ "int_col": 456,
+ "float_col": 2.72,
+ "num_col": decimal.Decimal("0.99"),
+ "bool_col": False,
+ "str_col": "hallo welt",
+ "bytes_col": b"\xbb\xee\xff",
+ "date_col": datetime.date(1995, 3, 2),
+ "time_col": datetime.time(13, 37),
+ "ts_col": datetime.datetime(1965, 4, 3, 2, 1, tzinfo=pytz.utc),
+ },
+ ],
+ [
+ {
+ "int_col": 789,
+ "float_col": 1.23,
+ "num_col": decimal.Decimal("5.67"),
+ "bool_col": True,
+ "str_col": u"こんにちは世界",
+ "bytes_col": b"\x54\x69\x6d",
+ "date_col": datetime.date(1970, 1, 1),
+ "time_col": datetime.time(16, 20),
+ "ts_col": datetime.datetime(1991, 8, 25, 20, 57, 8, tzinfo=pytz.utc),
+ }
+ ],
+]
+
+
+@pytest.fixture()
+def mut():
+ from google.cloud.bigquery_storage_v1beta1 import reader
+
+ return reader
+
+
+@pytest.fixture()
+def class_under_test(mut):
+ return mut.ReadRowsStream
+
+
+@pytest.fixture()
+def mock_client():
+ from google.cloud.bigquery_storage_v1beta1.gapic import big_query_storage_client
+
+ return mock.create_autospec(big_query_storage_client.BigQueryStorageClient)
+
+
+def _bq_to_avro_blocks(bq_blocks, avro_schema_json):
+ avro_schema = fastavro.parse_schema(avro_schema_json)
+ avro_blocks = []
+ for block in bq_blocks:
+ blockio = six.BytesIO()
+ for row in block:
+ fastavro.schemaless_writer(blockio, avro_schema, row)
+
+ response = bigquery_storage_v1beta1.types.ReadRowsResponse()
+ response.row_count = len(block)
+ response.avro_rows.serialized_binary_rows = blockio.getvalue()
+ avro_blocks.append(response)
+ return avro_blocks
+
+
+def _bq_to_arrow_batch_objects(bq_blocks, arrow_schema):
+ arrow_batches = []
+ for block in bq_blocks:
+ arrays = []
+ for name in arrow_schema.names:
+ arrays.append(
+ pyarrow.array(
+ (row[name] for row in block),
+ type=arrow_schema.field_by_name(name).type,
+ size=len(block),
+ )
+ )
+ arrow_batches.append(pyarrow.RecordBatch.from_arrays(arrays, arrow_schema))
+ return arrow_batches
+
+
+def _bq_to_arrow_batches(bq_blocks, arrow_schema):
+ arrow_batches = []
+ for record_batch in _bq_to_arrow_batch_objects(bq_blocks, arrow_schema):
+ response = bigquery_storage_v1beta1.types.ReadRowsResponse()
+ response.arrow_record_batch.serialized_record_batch = (
+ record_batch.serialize().to_pybytes()
+ )
+ arrow_batches.append(response)
+ return arrow_batches
+
+
+def _avro_blocks_w_unavailable(avro_blocks):
+ for block in avro_blocks:
+ yield block
+ raise google.api_core.exceptions.ServiceUnavailable("test: please reconnect")
+
+
+def _avro_blocks_w_deadline(avro_blocks):
+ for block in avro_blocks:
+ yield block
+ raise google.api_core.exceptions.DeadlineExceeded("test: timeout, don't reconnect")
+
+
+def _generate_avro_read_session(avro_schema_json):
+ schema = json.dumps(avro_schema_json)
+ return bigquery_storage_v1beta1.types.ReadSession(avro_schema={"schema": schema})
+
+
+def _generate_arrow_read_session(arrow_schema):
+ return bigquery_storage_v1beta1.types.ReadSession(
+ arrow_schema={"serialized_schema": arrow_schema.serialize().to_pybytes()}
+ )
+
+
+def _bq_to_avro_schema(bq_columns):
+ fields = []
+ avro_schema = {"type": "record", "name": "__root__", "fields": fields}
+
+ for column in bq_columns:
+ doc = column.get("description")
+ name = column["name"]
+ type_ = BQ_TO_AVRO_TYPES[column["type"]]
+ mode = column.get("mode", "nullable").lower()
+
+ if mode == "nullable":
+ type_ = ["null", type_]
+
+ fields.append({"name": name, "type": type_, "doc": doc})
+
+ return avro_schema
+
+
+def _bq_to_arrow_schema(bq_columns):
+ def bq_col_as_field(column):
+ doc = column.get("description")
+ name = column["name"]
+ type_ = BQ_TO_ARROW_TYPES[column["type"]]
+ mode = column.get("mode", "nullable").lower()
+
+ return pyarrow.field(name, type_, mode == "nullable", {"description": doc})
+
+ return pyarrow.schema(bq_col_as_field(c) for c in bq_columns)
+
+
+def _get_avro_bytes(rows, avro_schema):
+ avro_file = six.BytesIO()
+ for row in rows:
+ fastavro.schemaless_writer(avro_file, avro_schema, row)
+ return avro_file.getvalue()
+
+
+def test_avro_rows_raises_import_error(mut, class_under_test, mock_client, monkeypatch):
+ monkeypatch.setattr(mut, "fastavro", None)
+ reader = class_under_test(
+ [], mock_client, bigquery_storage_v1beta1.types.StreamPosition(), {}
+ )
+
+ bq_columns = [{"name": "int_col", "type": "int64"}]
+ avro_schema = _bq_to_avro_schema(bq_columns)
+ read_session = _generate_avro_read_session(avro_schema)
+
+ with pytest.raises(ImportError):
+ reader.rows(read_session)
+
+
+def test_pyarrow_rows_raises_import_error(
+ mut, class_under_test, mock_client, monkeypatch
+):
+ monkeypatch.setattr(mut, "pyarrow", None)
+ reader = class_under_test(
+ [], mock_client, bigquery_storage_v1beta1.types.StreamPosition(), {}
+ )
+
+ bq_columns = [{"name": "int_col", "type": "int64"}]
+ arrow_schema = _bq_to_arrow_schema(bq_columns)
+ read_session = _generate_arrow_read_session(arrow_schema)
+
+ with pytest.raises(ImportError):
+ reader.rows(read_session)
+
+
+def test_rows_no_schema_set_raises_type_error(
+ mut, class_under_test, mock_client, monkeypatch
+):
+ reader = class_under_test(
+ [], mock_client, bigquery_storage_v1beta1.types.StreamPosition(), {}
+ )
+ read_session = bigquery_storage_v1beta1.types.ReadSession()
+
+ with pytest.raises(TypeError):
+ reader.rows(read_session)
+
+
+def test_rows_w_empty_stream(class_under_test, mock_client):
+ bq_columns = [{"name": "int_col", "type": "int64"}]
+ avro_schema = _bq_to_avro_schema(bq_columns)
+ read_session = _generate_avro_read_session(avro_schema)
+ reader = class_under_test(
+ [], mock_client, bigquery_storage_v1beta1.types.StreamPosition(), {}
+ )
+
+ got = reader.rows(read_session)
+ assert got.total_rows is None
+ assert tuple(got) == ()
+
+
+def test_rows_w_empty_stream_arrow(class_under_test, mock_client):
+ bq_columns = [{"name": "int_col", "type": "int64"}]
+ arrow_schema = _bq_to_arrow_schema(bq_columns)
+ read_session = _generate_arrow_read_session(arrow_schema)
+ reader = class_under_test(
+ [], mock_client, bigquery_storage_v1beta1.types.StreamPosition(), {}
+ )
+
+ got = reader.rows(read_session)
+ assert got.total_rows is None
+ assert tuple(got) == ()
+
+
+def test_rows_w_scalars(class_under_test, mock_client):
+ avro_schema = _bq_to_avro_schema(SCALAR_COLUMNS)
+ read_session = _generate_avro_read_session(avro_schema)
+ avro_blocks = _bq_to_avro_blocks(SCALAR_BLOCKS, avro_schema)
+
+ reader = class_under_test(
+ avro_blocks, mock_client, bigquery_storage_v1beta1.types.StreamPosition(), {}
+ )
+ got = tuple(reader.rows(read_session))
+
+ expected = tuple(itertools.chain.from_iterable(SCALAR_BLOCKS))
+ assert got == expected
+
+
+def test_rows_w_scalars_arrow(class_under_test, mock_client):
+ arrow_schema = _bq_to_arrow_schema(SCALAR_COLUMNS)
+ read_session = _generate_arrow_read_session(arrow_schema)
+ arrow_batches = _bq_to_arrow_batches(SCALAR_BLOCKS, arrow_schema)
+
+ reader = class_under_test(
+ arrow_batches, mock_client, bigquery_storage_v1beta1.types.StreamPosition(), {}
+ )
+ got = tuple(reader.rows(read_session))
+
+ expected = tuple(itertools.chain.from_iterable(SCALAR_BLOCKS))
+ assert got == expected
+
+
+def test_rows_w_timeout(class_under_test, mock_client):
+ bq_columns = [{"name": "int_col", "type": "int64"}]
+ avro_schema = _bq_to_avro_schema(bq_columns)
+ read_session = _generate_avro_read_session(avro_schema)
+ bq_blocks_1 = [
+ [{"int_col": 123}, {"int_col": 234}],
+ [{"int_col": 345}, {"int_col": 456}],
+ ]
+ avro_blocks_1 = _avro_blocks_w_deadline(
+ _bq_to_avro_blocks(bq_blocks_1, avro_schema)
+ )
+ bq_blocks_2 = [[{"int_col": 567}, {"int_col": 789}], [{"int_col": 890}]]
+ avro_blocks_2 = _bq_to_avro_blocks(bq_blocks_2, avro_schema)
+
+ mock_client.read_rows.return_value = avro_blocks_2
+ stream_position = bigquery_storage_v1beta1.types.StreamPosition(
+ stream={"name": "test"}
+ )
+
+ reader = class_under_test(
+ avro_blocks_1,
+ mock_client,
+ stream_position,
+ {"metadata": {"test-key": "test-value"}},
+ )
+
+ with pytest.raises(google.api_core.exceptions.DeadlineExceeded):
+ list(reader.rows(read_session))
+
+ # Don't reconnect on DeadlineException. This allows user-specified timeouts
+ # to be respected.
+ mock_client.read_rows.assert_not_called()
+
+
+def test_rows_w_reconnect(class_under_test, mock_client):
+ bq_columns = [{"name": "int_col", "type": "int64"}]
+ avro_schema = _bq_to_avro_schema(bq_columns)
+ read_session = _generate_avro_read_session(avro_schema)
+ bq_blocks_1 = [
+ [{"int_col": 123}, {"int_col": 234}],
+ [{"int_col": 345}, {"int_col": 456}],
+ ]
+ avro_blocks_1 = _avro_blocks_w_unavailable(
+ _bq_to_avro_blocks(bq_blocks_1, avro_schema)
+ )
+ bq_blocks_2 = [[{"int_col": 567}, {"int_col": 789}], [{"int_col": 890}]]
+ avro_blocks_2 = _bq_to_avro_blocks(bq_blocks_2, avro_schema)
+
+ for block in avro_blocks_2:
+ block.status.estimated_row_count = 7
+
+ mock_client.read_rows.return_value = avro_blocks_2
+ stream_position = bigquery_storage_v1beta1.types.StreamPosition(
+ stream={"name": "test"}
+ )
+
+ reader = class_under_test(
+ avro_blocks_1,
+ mock_client,
+ stream_position,
+ {"metadata": {"test-key": "test-value"}},
+ )
+ got = reader.rows(read_session)
+
+ expected = tuple(
+ itertools.chain(
+ itertools.chain.from_iterable(bq_blocks_1),
+ itertools.chain.from_iterable(bq_blocks_2),
+ )
+ )
+
+ assert tuple(got) == expected
+ assert got.total_rows == 7
+ mock_client.read_rows.assert_called_once_with(
+ bigquery_storage_v1beta1.types.StreamPosition(
+ stream={"name": "test"}, offset=4
+ ),
+ metadata={"test-key": "test-value"},
+ )
+
+
+def test_rows_w_reconnect_by_page(class_under_test, mock_client):
+ bq_columns = [{"name": "int_col", "type": "int64"}]
+ avro_schema = _bq_to_avro_schema(bq_columns)
+ read_session = _generate_avro_read_session(avro_schema)
+ bq_blocks_1 = [
+ [{"int_col": 123}, {"int_col": 234}],
+ [{"int_col": 345}, {"int_col": 456}],
+ ]
+ avro_blocks_1 = _bq_to_avro_blocks(bq_blocks_1, avro_schema)
+ bq_blocks_2 = [[{"int_col": 567}, {"int_col": 789}], [{"int_col": 890}]]
+ avro_blocks_2 = _bq_to_avro_blocks(bq_blocks_2, avro_schema)
+
+ avro_blocks_1[0].status.estimated_row_count = 8
+ avro_blocks_1[1].status.estimated_row_count = 6
+ avro_blocks_2[0].status.estimated_row_count = 9
+ avro_blocks_2[1].status.estimated_row_count = 7
+
+ mock_client.read_rows.return_value = avro_blocks_2
+ stream_position = bigquery_storage_v1beta1.types.StreamPosition(
+ stream={"name": "test"}
+ )
+
+ reader = class_under_test(
+ _avro_blocks_w_unavailable(avro_blocks_1),
+ mock_client,
+ stream_position,
+ {"metadata": {"test-key": "test-value"}},
+ )
+ got = reader.rows(read_session)
+ pages = iter(got.pages)
+
+ assert got.total_rows is None
+
+ page_1 = next(pages)
+ assert got.total_rows == 8
+ assert page_1.num_items == 2
+ assert page_1.remaining == 2
+ assert tuple(page_1) == tuple(bq_blocks_1[0])
+ assert page_1.num_items == 2
+ assert page_1.remaining == 0
+
+ page_2 = next(pages)
+ assert got.total_rows == 6
+ assert next(page_2) == bq_blocks_1[1][0]
+ assert page_2.num_items == 2
+ assert page_2.remaining == 1
+ assert next(page_2) == bq_blocks_1[1][1]
+
+ page_3 = next(pages)
+ assert tuple(page_3) == tuple(bq_blocks_2[0])
+ assert page_3.num_items == 2
+ assert page_3.remaining == 0
+ assert got.total_rows == 9
+
+ page_4 = next(pages)
+ assert got.total_rows == 7
+ assert tuple(page_4) == tuple(bq_blocks_2[1])
+ assert page_4.num_items == 1
+ assert page_4.remaining == 0
+
+
+def test_to_arrow_no_pyarrow_raises_import_error(
+ mut, class_under_test, mock_client, monkeypatch
+):
+ monkeypatch.setattr(mut, "pyarrow", None)
+ arrow_schema = _bq_to_arrow_schema(SCALAR_COLUMNS)
+ read_session = _generate_arrow_read_session(arrow_schema)
+ arrow_batches = _bq_to_arrow_batches(SCALAR_BLOCKS, arrow_schema)
+ reader = class_under_test(
+ arrow_batches, mock_client, bigquery_storage_v1beta1.types.StreamPosition(), {}
+ )
+
+ with pytest.raises(ImportError):
+ reader.to_arrow(read_session)
+
+ with pytest.raises(ImportError):
+ reader.rows(read_session).to_arrow()
+
+ with pytest.raises(ImportError):
+ next(reader.rows(read_session).pages).to_arrow()
+
+
+def test_to_arrow_w_scalars_arrow(class_under_test):
+ arrow_schema = _bq_to_arrow_schema(SCALAR_COLUMNS)
+ read_session = _generate_arrow_read_session(arrow_schema)
+ arrow_batches = _bq_to_arrow_batches(SCALAR_BLOCKS, arrow_schema)
+ reader = class_under_test(
+ arrow_batches, mock_client, bigquery_storage_v1beta1.types.StreamPosition(), {}
+ )
+ actual_table = reader.to_arrow(read_session)
+ expected_table = pyarrow.Table.from_batches(
+ _bq_to_arrow_batch_objects(SCALAR_BLOCKS, arrow_schema)
+ )
+ assert actual_table == expected_table
+
+
+def test_to_dataframe_no_pandas_raises_import_error(
+ mut, class_under_test, mock_client, monkeypatch
+):
+ monkeypatch.setattr(mut, "pandas", None)
+ avro_schema = _bq_to_avro_schema(SCALAR_COLUMNS)
+ read_session = _generate_avro_read_session(avro_schema)
+ avro_blocks = _bq_to_avro_blocks(SCALAR_BLOCKS, avro_schema)
+
+ reader = class_under_test(
+ avro_blocks, mock_client, bigquery_storage_v1beta1.types.StreamPosition(), {}
+ )
+
+ with pytest.raises(ImportError):
+ reader.to_dataframe(read_session)
+
+ with pytest.raises(ImportError):
+ reader.rows(read_session).to_dataframe()
+
+ with pytest.raises(ImportError):
+ next(reader.rows(read_session).pages).to_dataframe()
+
+
+def test_to_dataframe_no_schema_set_raises_type_error(
+ mut, class_under_test, mock_client, monkeypatch
+):
+ reader = class_under_test(
+ [], mock_client, bigquery_storage_v1beta1.types.StreamPosition(), {}
+ )
+ read_session = bigquery_storage_v1beta1.types.ReadSession()
+
+ with pytest.raises(TypeError):
+ reader.to_dataframe(read_session)
+
+
+def test_to_dataframe_w_scalars(class_under_test):
+ avro_schema = _bq_to_avro_schema(SCALAR_COLUMNS)
+ read_session = _generate_avro_read_session(avro_schema)
+ avro_blocks = _bq_to_avro_blocks(SCALAR_BLOCKS, avro_schema)
+
+ reader = class_under_test(
+ avro_blocks, mock_client, bigquery_storage_v1beta1.types.StreamPosition(), {}
+ )
+ got = reader.to_dataframe(read_session)
+
+ expected = pandas.DataFrame(
+ list(itertools.chain.from_iterable(SCALAR_BLOCKS)), columns=SCALAR_COLUMN_NAMES
+ )
+ # fastavro provides its own UTC definition, so
+ # compare the timestamp columns separately.
+ got_ts = got["ts_col"]
+ got = got.drop(columns=["ts_col"])
+ expected_ts = expected["ts_col"]
+ expected = expected.drop(columns=["ts_col"])
+
+ pandas.testing.assert_frame_equal(
+ got.reset_index(drop=True), # reset_index to ignore row labels
+ expected.reset_index(drop=True),
+ )
+ pandas.testing.assert_series_equal(
+ got_ts.reset_index(drop=True),
+ expected_ts.reset_index(drop=True),
+ check_dtype=False, # fastavro's UTC means different dtype
+ check_datetimelike_compat=True,
+ )
+
+
+def test_to_dataframe_w_scalars_arrow(class_under_test):
+ arrow_schema = _bq_to_arrow_schema(SCALAR_COLUMNS)
+ read_session = _generate_arrow_read_session(arrow_schema)
+ arrow_batches = _bq_to_arrow_batches(SCALAR_BLOCKS, arrow_schema)
+
+ reader = class_under_test(
+ arrow_batches, mock_client, bigquery_storage_v1beta1.types.StreamPosition(), {}
+ )
+ got = reader.to_dataframe(read_session)
+
+ expected = pandas.DataFrame(
+ list(itertools.chain.from_iterable(SCALAR_BLOCKS)), columns=SCALAR_COLUMN_NAMES
+ )
+
+ pandas.testing.assert_frame_equal(
+ got.reset_index(drop=True), # reset_index to ignore row labels
+ expected.reset_index(drop=True),
+ )
+
+
+def test_to_dataframe_w_dtypes(class_under_test):
+ avro_schema = _bq_to_avro_schema(
+ [
+ {"name": "bigfloat", "type": "float64"},
+ {"name": "lilfloat", "type": "float64"},
+ ]
+ )
+ read_session = _generate_avro_read_session(avro_schema)
+ blocks = [
+ [{"bigfloat": 1.25, "lilfloat": 30.5}, {"bigfloat": 2.5, "lilfloat": 21.125}],
+ [{"bigfloat": 3.75, "lilfloat": 11.0}],
+ ]
+ avro_blocks = _bq_to_avro_blocks(blocks, avro_schema)
+
+ reader = class_under_test(
+ avro_blocks, mock_client, bigquery_storage_v1beta1.types.StreamPosition(), {}
+ )
+ got = reader.to_dataframe(read_session, dtypes={"lilfloat": "float16"})
+
+ expected = pandas.DataFrame(
+ {
+ "bigfloat": [1.25, 2.5, 3.75],
+ "lilfloat": pandas.Series([30.5, 21.125, 11.0], dtype="float16"),
+ },
+ columns=["bigfloat", "lilfloat"],
+ )
+ pandas.testing.assert_frame_equal(
+ got.reset_index(drop=True), # reset_index to ignore row labels
+ expected.reset_index(drop=True),
+ )
+
+
+def test_to_dataframe_w_dtypes_arrow(class_under_test):
+ arrow_schema = _bq_to_arrow_schema(
+ [
+ {"name": "bigfloat", "type": "float64"},
+ {"name": "lilfloat", "type": "float64"},
+ ]
+ )
+ read_session = _generate_arrow_read_session(arrow_schema)
+ blocks = [
+ [{"bigfloat": 1.25, "lilfloat": 30.5}, {"bigfloat": 2.5, "lilfloat": 21.125}],
+ [{"bigfloat": 3.75, "lilfloat": 11.0}],
+ ]
+ arrow_batches = _bq_to_arrow_batches(blocks, arrow_schema)
+
+ reader = class_under_test(
+ arrow_batches, mock_client, bigquery_storage_v1beta1.types.StreamPosition(), {}
+ )
+ got = reader.to_dataframe(read_session, dtypes={"lilfloat": "float16"})
+
+ expected = pandas.DataFrame(
+ {
+ "bigfloat": [1.25, 2.5, 3.75],
+ "lilfloat": pandas.Series([30.5, 21.125, 11.0], dtype="float16"),
+ },
+ columns=["bigfloat", "lilfloat"],
+ )
+ pandas.testing.assert_frame_equal(
+ got.reset_index(drop=True), # reset_index to ignore row labels
+ expected.reset_index(drop=True),
+ )
+
+
+def test_to_dataframe_by_page(class_under_test, mock_client):
+ bq_columns = [
+ {"name": "int_col", "type": "int64"},
+ {"name": "bool_col", "type": "bool"},
+ ]
+ avro_schema = _bq_to_avro_schema(bq_columns)
+ read_session = _generate_avro_read_session(avro_schema)
+ block_1 = [{"int_col": 123, "bool_col": True}, {"int_col": 234, "bool_col": False}]
+ block_2 = [{"int_col": 345, "bool_col": True}, {"int_col": 456, "bool_col": False}]
+ block_3 = [{"int_col": 567, "bool_col": True}, {"int_col": 789, "bool_col": False}]
+ block_4 = [{"int_col": 890, "bool_col": True}]
+ # Break blocks into two groups to test that iteration continues across
+ # reconnection.
+ bq_blocks_1 = [block_1, block_2]
+ bq_blocks_2 = [block_3, block_4]
+ avro_blocks_1 = _bq_to_avro_blocks(bq_blocks_1, avro_schema)
+ avro_blocks_2 = _bq_to_avro_blocks(bq_blocks_2, avro_schema)
+
+ mock_client.read_rows.return_value = avro_blocks_2
+ stream_position = bigquery_storage_v1beta1.types.StreamPosition(
+ stream={"name": "test"}
+ )
+
+ reader = class_under_test(
+ _avro_blocks_w_unavailable(avro_blocks_1),
+ mock_client,
+ stream_position,
+ {"metadata": {"test-key": "test-value"}},
+ )
+ got = reader.rows(read_session)
+ pages = iter(got.pages)
+
+ page_1 = next(pages)
+ pandas.testing.assert_frame_equal(
+ page_1.to_dataframe().reset_index(drop=True),
+ pandas.DataFrame(block_1, columns=["int_col", "bool_col"]).reset_index(
+ drop=True
+ ),
+ )
+
+ page_2 = next(pages)
+ pandas.testing.assert_frame_equal(
+ page_2.to_dataframe().reset_index(drop=True),
+ pandas.DataFrame(block_2, columns=["int_col", "bool_col"]).reset_index(
+ drop=True
+ ),
+ )
+
+ page_3 = next(pages)
+ pandas.testing.assert_frame_equal(
+ page_3.to_dataframe().reset_index(drop=True),
+ pandas.DataFrame(block_3, columns=["int_col", "bool_col"]).reset_index(
+ drop=True
+ ),
+ )
+
+ page_4 = next(pages)
+ pandas.testing.assert_frame_equal(
+ page_4.to_dataframe().reset_index(drop=True),
+ pandas.DataFrame(block_4, columns=["int_col", "bool_col"]).reset_index(
+ drop=True
+ ),
+ )
+
+
+def test_copy_stream_position(mut):
+ read_position = bigquery_storage_v1beta1.types.StreamPosition(
+ stream={"name": "test"}, offset=41
+ )
+ got = mut._copy_stream_position(read_position)
+ assert got == read_position
+ got.offset = 42
+ assert read_position.offset == 41
+
+
+def test_copy_stream_position_w_dict(mut):
+ read_position = {"stream": {"name": "test"}, "offset": 42}
+ got = mut._copy_stream_position(read_position)
+ assert got.stream.name == "test"
+ assert got.offset == 42
diff --git a/bigtable/.coveragerc b/bigtable/.coveragerc
new file mode 100644
index 000000000000..b178b094aa1d
--- /dev/null
+++ b/bigtable/.coveragerc
@@ -0,0 +1,19 @@
+# Generated by synthtool. DO NOT EDIT!
+[run]
+branch = True
+
+[report]
+fail_under = 100
+show_missing = True
+exclude_lines =
+ # Re-enable the standard pragma
+ pragma: NO COVER
+ # Ignore debug-only repr
+ def __repr__
+ # Ignore abstract methods
+ raise NotImplementedError
+omit =
+ */gapic/*.py
+ */proto/*.py
+ */core/*.py
+ */site-packages/*.py
\ No newline at end of file
diff --git a/bigtable/.flake8 b/bigtable/.flake8
new file mode 100644
index 000000000000..0268ecc9c55c
--- /dev/null
+++ b/bigtable/.flake8
@@ -0,0 +1,14 @@
+# Generated by synthtool. DO NOT EDIT!
+[flake8]
+ignore = E203, E266, E501, W503
+exclude =
+ # Exclude generated code.
+ **/proto/**
+ **/gapic/**
+ *_pb2.py
+
+ # Standard linting exemptions.
+ __pycache__,
+ .git,
+ *.pyc,
+ conf.py
diff --git a/bigtable/.repo-metadata.json b/bigtable/.repo-metadata.json
new file mode 100644
index 000000000000..956c74b53395
--- /dev/null
+++ b/bigtable/.repo-metadata.json
@@ -0,0 +1,13 @@
+{
+ "name": "bigtable",
+ "name_pretty": "Cloud Bigtable",
+ "product_documentation": "https://cloud.google.com/bigtable",
+ "client_documentation": "https://googleapis.dev/python/bigtable/latest",
+ "issue_tracker": "https://issuetracker.google.com/savedsearches/559777",
+ "release_level": "ga",
+ "language": "python",
+ "repo": "googleapis/google-cloud-python",
+ "distribution_name": "google-cloud-bigtable",
+ "api_id": "bigtable.googleapis.com",
+ "requires_billing": true
+}
\ No newline at end of file
diff --git a/bigtable/CHANGELOG.md b/bigtable/CHANGELOG.md
new file mode 100644
index 000000000000..f6e28c23cd4e
--- /dev/null
+++ b/bigtable/CHANGELOG.md
@@ -0,0 +1,313 @@
+# Changelog
+
+[PyPI History][1]
+
+[1]: https://pypi.org/project/google-cloud-bigtable/#history
+
+## 0.34.0
+
+07-30-2019 10:05 PDT
+
+
+### Implementation Changes
+- Pick up changes to GAPIC client configuration (via synth). ([#8724](https://github.com/googleapis/google-cloud-python/pull/8724))
+- Add `Cell.__repr__`. ([#8683](https://github.com/googleapis/google-cloud-python/pull/8683))
+- Increase timeout for app profile update operation. ([#8417](https://github.com/googleapis/google-cloud-python/pull/8417))
+
+### New Features
+- Add methods returning Separate row types to remove confusion around return types of `row.commit`. ([#8662](https://github.com/googleapis/google-cloud-python/pull/8662))
+- Add `options_` argument to clients' `get_iam_policy` (via synth). ([#8652](https://github.com/googleapis/google-cloud-python/pull/8652))
+- Add `client_options` support, update list method docstrings (via synth). ([#8500](https://github.com/googleapis/google-cloud-python/pull/8500))
+
+### Dependencies
+- Bump minimum version for google-api-core to 1.14.0. ([#8709](https://github.com/googleapis/google-cloud-python/pull/8709))
+- Update pin for `grpc-google-iam-v1` to 0.12.3+. ([#8647](https://github.com/googleapis/google-cloud-python/pull/8647))
+- Allow kwargs to be passed to `create_channel` (via synth). ([#8458](https://github.com/googleapis/google-cloud-python/pull/8458))
+- Add `PartialRowsData.cancel`. ([#8176](https://github.com/googleapis/google-cloud-python/pull/8176))
+
+### Documentation
+- Update intersphinx mapping for requests. ([#8805](https://github.com/googleapis/google-cloud-python/pull/8805))
+- Link to googleapis.dev documentation in READMEs. ([#8705](https://github.com/googleapis/google-cloud-python/pull/8705))
+- Add compatibility check badges to READMEs. ([#8288](https://github.com/googleapis/google-cloud-python/pull/8288))
+- Add snppets illustrating use of application profiles. ([#7033](https://github.com/googleapis/google-cloud-python/pull/7033))
+
+### Internal / Testing Changes
+- Add nox session `docs` to remaining manual clients. ([#8478](https://github.com/googleapis/google-cloud-python/pull/8478))
+- All: Add docs job to publish to googleapis.dev. ([#8464](https://github.com/googleapis/google-cloud-python/pull/8464))
+- Force timeout for table creation to 90 seconds (in systests). ([#8450](https://github.com/googleapis/google-cloud-python/pull/8450))
+- Plug systest / snippet instance leaks. ([#8416](https://github.com/googleapis/google-cloud-python/pull/8416))
+- Declare encoding as utf-8 in pb2 files (via synth). ([#8346](https://github.com/googleapis/google-cloud-python/pull/8346))
+- Add disclaimer to auto-generated template files (via synth). ([#8308](https://github.com/googleapis/google-cloud-python/pull/8308))
+- Fix coverage in `types.py` (via synth). ([#8149](https://github.com/googleapis/google-cloud-python/pull/8149))
+- Integrate docstring / formatting tweaks (via synth). ([#8138](https://github.com/googleapis/google-cloud-python/pull/8138))
+- Use alabaster theme everwhere. ([#8021](https://github.com/googleapis/google-cloud-python/pull/8021))
+
+## 0.33.0
+
+05-16-2019 11:51 PDT
+
+
+### Implementation Changes
+- Fix typos in deprecation warnings. ([#7858](https://github.com/googleapis/google-cloud-python/pull/7858))
+- Add deprecation warnings for to-be-removed features. ([#7532](https://github.com/googleapis/google-cloud-python/pull/7532))
+- Remove classifier for Python 3.4 for end-of-life. ([#7535](https://github.com/googleapis/google-cloud-python/pull/7535))
+- Improve `Policy` interchange w/ JSON, gRPC payloads. ([#7378](https://github.com/googleapis/google-cloud-python/pull/7378))
+
+### New Features
+- Add support for passing `client_info` to client. ([#7876](https://github.com/googleapis/google-cloud-python/pull/7876)) and ([#7898](https://github.com/googleapis/google-cloud-python/pull/7898))
+- Add `Table.mutation_timeout`, allowing override of config timeouts. ([#7424](https://github.com/googleapis/google-cloud-python/pull/7424))
+
+### Dependencies
+- Pin `google-cloud-core >= 1.0.0, < 2.0dev`. ([#7993](https://github.com/googleapis/google-cloud-python/pull/7993))
+
+### Documentation
+- Remove duplicate snippet tags for Delete cluster. ([#7860](https://github.com/googleapis/google-cloud-python/pull/7860))
+- Fix rendering of instance admin snippets. ([#7797](https://github.com/googleapis/google-cloud-python/pull/7797))
+- Avoid leaking instances from snippets. ([#7800](https://github.com/googleapis/google-cloud-python/pull/7800))
+- Fix enum reference in documentation. ([#7724](https://github.com/googleapis/google-cloud-python/pull/7724))
+- Remove duplicate snippets. ([#7528](https://github.com/googleapis/google-cloud-python/pull/7528))
+- Add snippeds for Batcher, RowData, Row Operations, AppendRow. ([#7019](https://github.com/googleapis/google-cloud-python/pull/7019))
+- Add column family snippets. ([#7014](https://github.com/googleapis/google-cloud-python/pull/7014))
+- Add Row Set snippets. ([#7016](https://github.com/googleapis/google-cloud-python/pull/7016))
+- Update client library documentation URLs. ([#7307](https://github.com/googleapis/google-cloud-python/pull/7307))
+- Fix typos in Table docstrings. ([#7261](https://github.com/googleapis/google-cloud-python/pull/7261))
+- Update copyright headers (via synth). ([#7139](https://github.com/googleapis/google-cloud-python/pull/7139))
+- Fix linked classes in generated docstrings (via synth). ([#7060](https://github.com/googleapis/google-cloud-python/pull/7060))
+
+### Internal / Testing Changes
+- Run `instance_admin` system tests on a separate instance from `table_admin` and `data` system tests. ([#6579](https://github.com/googleapis/google-cloud-python/pull/6579))
+- Re-blacken. ([#7462](https://github.com/googleapis/google-cloud-python/pull/7462))
+- Copy lintified proto files (via synth). ([#7445](https://github.com/googleapis/google-cloud-python/pull/7445))
+- Remove unused message exports (via synth). ([#7264](https://github.com/googleapis/google-cloud-python/pull/7264))
+- Compare 0 using '!=', rather than 'is not'. ([#7312](https://github.com/googleapis/google-cloud-python/pull/7312))
+- Add protos as an artifact to library ([#7205](https://github.com/googleapis/google-cloud-python/pull/7205))
+- Protoc-generated serialization update. ([#7077](https://github.com/googleapis/google-cloud-python/pull/7077))
+- Blacken snippets. ([#7048](https://github.com/googleapis/google-cloud-python/pull/7048))
+- Bigtable client snippets ([#7020](https://github.com/googleapis/google-cloud-python/pull/7020))
+- Pick up order-of-enum fix from GAPIC generator. ([#6879](https://github.com/googleapis/google-cloud-python/pull/6879))
+- Plug systest instance leaks ([#7004](https://github.com/googleapis/google-cloud-python/pull/7004))
+
+## 0.32.1
+
+12-17-2018 16:38 PST
+
+
+### Documentation
+- Document Python 2 deprecation ([#6910](https://github.com/googleapis/google-cloud-python/pull/6910))
+- Add snippets for table operations. ([#6484](https://github.com/googleapis/google-cloud-python/pull/6484))
+
+## 0.32.0
+
+12-10-2018 12:47 PST
+
+
+### Implementation Changes
+- Import `iam.policy` from `google.api_core`. ([#6741](https://github.com/googleapis/google-cloud-python/pull/6741))
+- Remove `deepcopy` from `PartialRowData.cells` property. ([#6648](https://github.com/googleapis/google-cloud-python/pull/6648))
+- Pick up fixes to GAPIC generator. ([#6630](https://github.com/googleapis/google-cloud-python/pull/6630))
+
+### Dependencies
+- Update dependency to google-cloud-core ([#6835](https://github.com/googleapis/google-cloud-python/pull/6835))
+
+### Internal / Testing Changes
+- Blacken all gen'd libs ([#6792](https://github.com/googleapis/google-cloud-python/pull/6792))
+- Omit local deps ([#6701](https://github.com/googleapis/google-cloud-python/pull/6701))
+- Run black at end of synth.py ([#6698](https://github.com/googleapis/google-cloud-python/pull/6698))
+- Blackening Continued... ([#6667](https://github.com/googleapis/google-cloud-python/pull/6667))
+- Add templates for flake8, coveragerc, noxfile, and black. ([#6642](https://github.com/googleapis/google-cloud-python/pull/6642))
+
+## 0.31.1
+
+11-02-2018 08:13 PDT
+
+### Implementation Changes
+- Fix anonymous usage under Bigtable emulator ([#6385](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/6385))
+- Support `DirectRow` without a `Table` ([#6336](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/6336))
+- Add retry parameter to `Table.read_rows()`. ([#6281](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/6281))
+- Fix `ConditionalRow` interaction with `check_and_mutate_row` ([#6296](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/6296))
+- Deprecate `channel` arg to `Client` ([#6279](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/6279))
+
+### Dependencies
+- Update dependency: `google-api-core >= 1.4.1` ([#6391](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/6391))
+- Update IAM version in dependencies ([#6362](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/6362))
+
+### Documentation
+- Add `docs/snippets.py` and test ([#6012](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/6012))
+- Normalize use of support level badges ([#6159](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/6159))
+
+### Internal / Testing Changes
+- Fix client_info bug, update docstrings and timeouts. ([#6406)](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/6406))
+- Remove now-spurious fixup from 'synth.py'. ([#6400](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/6400))
+- Fix flaky systests / snippets ([#6367](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/6367))
+- Add explicit coverage for `row_data._retry_read_rows_exception`. ([#6364](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/6364))
+- Fix instance IAM test methods ([#6343](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/6343))
+- Fix error from new flake8 version. ([#6309](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/6309))
+- Use new Nox ([#6175](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/6175))
+
+## 0.31.0
+
+### New Features
+- Upgrade support level from `alpha` to `beta`. ([#6129](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/6129))
+
+### Implementation Changes
+- Improve admin operation timeouts. ([#6010](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/6010))
+
+### Documentation
+- Prepare docs for repo split. ([#6014](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/6014))
+
+### Internal / Testing Changes
+- Refactor `read_row` to call `read_rows` ([#6137](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/6102))
+- Harden instance teardown against '429 Too Many Requests'. ([#6102](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/6102))
+- Add `{RowSet,RowRange}.{__eq__,.__ne__}` ([#6025](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/6025))
+- Regenerate low-level GAPIC code ([#6036](https://github.com/GoogleCloudPlatform/google-cloud-python/pull/6036))
+
+## 0.30.2
+
+### New Features
+- Add iam policy implementation for an instance. (#5838)
+
+### Implementation Changes
+- Fix smart retries for 'read_rows()' when reading the full table (#5966)
+
+### Documentation
+- Replace links to `/stable/` with `/latest/`. (#5901)
+
+### Internal / Testing Changes
+- Re-generate library using bigtable/synth.py (#5974)
+- Refactor `read_rows` infrastructure (#5963)
+
+## 0.30.1
+
+### Implementation changes
+
+- Fix non-admin access to table data. (#5875)
+- Synth bigtable and bigtable admin GAPIC clients. (#5867)
+
+### Testing and internal changes
+
+- Nox: use in-place installs for local packages. (#5865)
+
+## 0.30.0
+
+### New Features
+
+- Improve performance and capabilities of reads. `read_rows` now returns a generator; has automatic retries; and can read an arbitrary set of keys and ranges
+ - Consolidate read_rows and yield_rows (#5840)
+ - Implement row set for yield_rows (#5506)
+ - Improve read rows validation performance (#5390)
+ - Add retry for yield_rows (#4882)
+ - Require TimestampRanges to be milliseconds granularity (#5002)
+ - Provide better access to cell values (#4908)
+ - Add data app profile id (#5369)
+
+- Improve writes: Writes are usable in Beam
+ - Create MutationBatcher for bigtable (#5651)
+ - Allow DirectRow to be created without a table (#5567)
+ - Add data app profile id (#5369)
+
+- Improve table admin: Table creation now can also create families in a single RPC. Add an `exist()` method. Add `get_cluster_states` for information about replication
+ - Add 'Table.get_cluster_states' method (#5790)
+ - Optimize 'Table.exists' performance (#5749)
+ - Add column creation in 'Table.create()'. (#5576)
+ - Add 'Table.exists' method (#5545)
+ - Add split keys on create table - v2 (#5513)
+ - Avoid sharing table names across unrelated systests. (#5421)
+ - Add truncate table and drop by prefix on top of GAPIC integration (#5360)
+
+- Improve instance admin: Instance creation allows for the creation of multiple clusters. Instance label management is now enabled.
+ - Create app_profile_object (#5782)
+ - Add 'Instance.exists' method (#5802)
+ - Add 'InstanceAdminClient.list_clusters' method (#5715)
+ - Add 'Instance._state' property (#5736)
+ - Convert 'instance.labels' to return a dictionary (#5728)
+ - Reshape cluster.py, adding cluster() factory to instance.py (#5663)
+ - Convert 'Instance.update' to use 'instance.partial_instance_update' API (#5643)
+ - Refactor 'InstanceAdminClient.update_app_profile' to remove update_mask argument (#5684)
+ - Add the ability to create an instance with multiple clusters (#5622)
+ - Add 'instance_type', 'labels' to 'Instance' ctor (#5614)
+ - Add optional app profile to 'Instance.table' (#5605)
+ - Clean up Instance creation. (#5542)
+ - Make 'InstanceAdminClient.list_instances' return actual instance objects, not protos. (#5420)
+ - Add admin app profile methods on Instance (#5315)
+
+### Internal / Testing Changes
+- Rename releases to changelog and include from CHANGELOG.md (#5191)
+- Fix bad trove classifier
+- Integrate new generated low-level client (#5178)
+- Override gRPC max message lengths. (#5498)
+- Use client properties rather than private attrs (#5398)
+- Fix the broken Bigtable system test. (#5607)
+- Fix Py3 breakage in new system test. (#5474)
+- Modify system test for new GAPIC code (#5302)
+- Add Test runs for Python 3.7 and remove 3.4 (#5295)
+- Disable Bigtable system tests (#5381)
+- Modify system tests to use prerelease versions of grpcio (#5304)
+- Pass through 'session.posargs' when running Bigtable system tests. (#5418)
+- Harden 'test_list_instances' against simultaneous test runs. (#5476)
+- Shorten instance / cluster name to fix CI breakage. (#5641)
+- Fix failing systest: 'test_create_instance_w_two_clusters'. (#5836)
+- Add labels {'python-system': ISO-timestamp} to systest instances (#5729)
+- Shorten cluster ID in system test (#5719)
+- Harden 'test_list_instances' further. (#5696)
+- Improve testing of create instance (#5544)
+
+## 0.29.0
+
+### New features
+
+- Use `api_core.retry` for `mutate_row` (#4665, #4341)
+- Added a row generator on a table. (#4679)
+
+### Implementation changes
+
+- Remove gax usage from BigTable (#4873)
+- BigTable: Cell.from_pb() performance improvement (#4745)
+
+### Dependencies
+
+- Update dependency range for api-core to include v1.0.0 releases (#4944)
+
+### Documentation
+
+- Minor typo (#4758)
+- Row filter end points documentation error (#4667)
+- Removing "rename" from bigtable table.py comments (#4526)
+- Small docs/hygiene tweaks after #4256. (#4333)
+
+### Testing and internal changes
+
+- Install local dependencies when running lint (#4936)
+- Re-enable lint for tests, remove usage of pylint (#4921)
+- Normalize all setup.py files (#4909)
+- Timestamp system test fix (#4765)
+
+## 0.28.1
+
+### Implementation Changes
+
+- Bugfix: Distinguish between an unset column qualifier and an empty string
+ column qualifier while parsing a `ReadRows` response (#4252)
+
+### Features added
+
+- Add a ``retry`` strategy that will be used for retry-able errors
+ in ``Table.mutate_rows``. This will be used for gRPC errors of type
+ ``ABORTED``, ``DEADLINE_EXCEEDED`` and ``SERVICE_UNAVAILABLE``. (#4256)
+
+PyPI: https://pypi.org/project/google-cloud-bigtable/0.28.1/
+
+## 0.28.0
+
+### Documentation
+
+- Fixed referenced types in `Table.row` docstring (#3934, h/t to
+ @MichaelTamm)
+- Added link to "Python Development Environment Setup Guide" in
+ project README (#4187, h/t to @michaelawyu)
+
+### Dependencies
+
+- Upgrading to `google-cloud-core >= 0.28.0` and adding dependency
+ on `google-api-core` (#4221, #4280)
+
+PyPI: https://pypi.org/project/google-cloud-bigtable/0.28.0/
diff --git a/bigtable/LICENSE b/bigtable/LICENSE
new file mode 100644
index 000000000000..d64569567334
--- /dev/null
+++ b/bigtable/LICENSE
@@ -0,0 +1,202 @@
+
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [yyyy] [name of copyright owner]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
diff --git a/bigtable/MANIFEST.in b/bigtable/MANIFEST.in
new file mode 100644
index 000000000000..9cbf175afe6b
--- /dev/null
+++ b/bigtable/MANIFEST.in
@@ -0,0 +1,5 @@
+include README.rst LICENSE
+recursive-include google *.json *.proto
+recursive-include tests *
+global-exclude *.py[co]
+global-exclude __pycache__
diff --git a/bigtable/README.rst b/bigtable/README.rst
new file mode 100644
index 000000000000..5660093c0fb6
--- /dev/null
+++ b/bigtable/README.rst
@@ -0,0 +1,99 @@
+Python Client for Google Cloud Bigtable
+=======================================
+
+|beta| |pypi| |versions|
+
+`Google Cloud Bigtable`_ is Google's NoSQL Big Data database service. It's the
+same database that powers many core Google services, including Search,
+Analytics, Maps, and Gmail.
+
+- `Client Library Documentation`_
+- `Product Documentation`_
+
+.. |beta| image:: https://img.shields.io/badge/support-beta-silver.svg
+ :target: https://github.com/googleapis/google-cloud-python/blob/master/README.rst#beta-support
+.. |pypi| image:: https://img.shields.io/pypi/v/google-cloud-bigtable.svg
+ :target: https://pypi.org/project/google-cloud-bigtable/
+.. |versions| image:: https://img.shields.io/pypi/pyversions/google-cloud-bigtable.svg
+ :target: https://pypi.org/project/google-cloud-bigtable/
+.. _Google Cloud Bigtable: https://cloud.google.com/bigtable
+.. _Client Library Documentation: https://googleapis.dev/python/bigtable/latest
+.. _Product Documentation: https://cloud.google.com/bigtable/docs
+
+Quick Start
+-----------
+
+In order to use this library, you first need to go through the following steps:
+
+1. `Select or create a Cloud Platform project.`_
+2. `Enable billing for your project.`_
+3. `Enable the Cloud Bigtable API.`_
+4. `Setup Authentication.`_
+
+.. _Select or create a Cloud Platform project.: https://console.cloud.google.com/project
+.. _Enable billing for your project.: https://cloud.google.com/billing/docs/how-to/modify-project#enable_billing_for_a_project
+.. _Enable the Cloud Bigtable API.: https://cloud.google.com/bigtable
+.. _Setup Authentication.: https://googleapis.dev/python/google-api-core/latest/auth.html
+
+Installation
+~~~~~~~~~~~~
+
+Install this library in a `virtualenv`_ using pip. `virtualenv`_ is a tool to
+create isolated Python environments. The basic problem it addresses is one of
+dependencies and versions, and indirectly permissions.
+
+With `virtualenv`_, it's possible to install this library without needing system
+install permissions, and without clashing with the installed system
+dependencies.
+
+.. _`virtualenv`: https://virtualenv.pypa.io/en/latest/
+
+
+Supported Python Versions
+^^^^^^^^^^^^^^^^^^^^^^^^^
+Python >= 3.5
+
+Deprecated Python Versions
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+Python == 2.7. Python 2.7 support will be removed on January 1, 2020.
+
+
+Mac/Linux
+^^^^^^^^^
+
+.. code-block:: console
+
+ pip install virtualenv
+ virtualenv
+ source /bin/activate
+ /bin/pip install google-cloud-bigtable
+
+
+Windows
+^^^^^^^
+
+.. code-block:: console
+
+ pip install virtualenv
+ virtualenv
+ \Scripts\activate
+ \Scripts\pip.exe install google-cloud-bigtable
+
+Next Steps
+~~~~~~~~~~
+
+- Read the `Client Library Documentation`_ for Cloud Bigtable API
+ to see other available methods on the client.
+- Read the `Product documentation`_ to learn
+ more about the product and see How-to Guides.
+
+``google-cloud-happybase``
+--------------------------
+
+In addition to the core ``google-cloud-bigtable``, we provide a
+`google-cloud-happybase
+`__ library
+with the same interface as the popular `HappyBase
+`__ library. Unlike HappyBase,
+``google-cloud-happybase`` uses ``google-cloud-bigtable`` under the covers,
+rather than Apache HBase.
diff --git a/bigtable/docs/README.rst b/bigtable/docs/README.rst
new file mode 120000
index 000000000000..89a0106941ff
--- /dev/null
+++ b/bigtable/docs/README.rst
@@ -0,0 +1 @@
+../README.rst
\ No newline at end of file
diff --git a/bigtable/docs/changelog.md b/bigtable/docs/changelog.md
new file mode 120000
index 000000000000..04c99a55caae
--- /dev/null
+++ b/bigtable/docs/changelog.md
@@ -0,0 +1 @@
+../CHANGELOG.md
\ No newline at end of file
diff --git a/bigtable/docs/client-intro.rst b/bigtable/docs/client-intro.rst
new file mode 100644
index 000000000000..6a38437790e2
--- /dev/null
+++ b/bigtable/docs/client-intro.rst
@@ -0,0 +1,90 @@
+Base for Everything
+===================
+
+To use the API, the :class:`Client `
+class defines a high-level interface which handles authorization
+and creating other objects:
+
+.. code:: python
+
+ from google.cloud.bigtable.client import Client
+ client = Client()
+
+Long-lived Defaults
+-------------------
+
+When creating a :class:`Client `, the
+``user_agent`` argument has sensible a default
+(:data:`DEFAULT_USER_AGENT `).
+However, you may over-ride it and the value will be used throughout all API
+requests made with the ``client`` you create.
+
+Configuration
+-------------
+
+- For an overview of authentication in ``google-cloud-python``,
+ see `Authentication `_.
+
+- In addition to any authentication configuration, you can also set the
+ :envvar:`GOOGLE_CLOUD_PROJECT` environment variable for the Google Cloud Console
+ project you'd like to interact with. If your code is running in Google App
+ Engine or Google Compute Engine the project will be detected automatically.
+ (Setting this environment variable is not required, you may instead pass the
+ ``project`` explicitly when constructing a
+ :class:`Client `).
+
+- After configuring your environment, create a
+ :class:`Client `
+
+ .. code::
+
+ >>> from google.cloud import bigtable
+ >>> client = bigtable.Client()
+
+ or pass in ``credentials`` and ``project`` explicitly
+
+ .. code::
+
+ >>> from google.cloud import bigtable
+ >>> client = bigtable.Client(project='my-project', credentials=creds)
+
+.. tip::
+
+ Be sure to use the **Project ID**, not the **Project Number**.
+
+Admin API Access
+----------------
+
+If you'll be using your client to make `Instance Admin`_ and `Table Admin`_
+API requests, you'll need to pass the ``admin`` argument:
+
+.. code:: python
+
+ client = bigtable.Client(admin=True)
+
+Read-Only Mode
+--------------
+
+If, on the other hand, you only have (or want) read access to the data,
+you can pass the ``read_only`` argument:
+
+.. code:: python
+
+ client = bigtable.Client(read_only=True)
+
+This will ensure that the
+:data:`READ_ONLY_SCOPE ` is used
+for API requests (so any accidental requests that would modify data will
+fail).
+
+Next Step
+---------
+
+After a :class:`Client `, the next highest-level
+object is an :class:`Instance `. You'll need
+one before you can interact with tables or data.
+
+Head next to learn about the :doc:`instance-api`.
+
+.. _Instance Admin: https://github.com/GoogleCloudPlatform/cloud-bigtable-client/tree/master/bigtable-protos/src/main/proto/google/bigtable/admin/instance/v1
+.. _Table Admin: https://github.com/GoogleCloudPlatform/cloud-bigtable-client/tree/master/bigtable-protos/src/main/proto/google/bigtable/admin/table/v1
diff --git a/docs/bigtable-client.rst b/bigtable/docs/client.rst
similarity index 100%
rename from docs/bigtable-client.rst
rename to bigtable/docs/client.rst
diff --git a/docs/bigtable-cluster.rst b/bigtable/docs/cluster.rst
similarity index 100%
rename from docs/bigtable-cluster.rst
rename to bigtable/docs/cluster.rst
diff --git a/docs/bigtable-column-family.rst b/bigtable/docs/column-family.rst
similarity index 100%
rename from docs/bigtable-column-family.rst
rename to bigtable/docs/column-family.rst
diff --git a/bigtable/docs/conf.py b/bigtable/docs/conf.py
new file mode 100644
index 000000000000..ebf93a3dbe5c
--- /dev/null
+++ b/bigtable/docs/conf.py
@@ -0,0 +1,359 @@
+# -*- coding: utf-8 -*-
+#
+# google-cloud-bigtable documentation build configuration file
+#
+# This file is execfile()d with the current directory set to its
+# containing dir.
+#
+# Note that not all possible configuration values are present in this
+# autogenerated file.
+#
+# All configuration values have a default; values that are commented out
+# serve to show the default.
+
+import sys
+import os
+import shlex
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+sys.path.insert(0, os.path.abspath(".."))
+
+__version__ = "0.1.0"
+
+# -- General configuration ------------------------------------------------
+
+# If your documentation needs a minimal Sphinx version, state it here.
+needs_sphinx = "1.6.3"
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+ "sphinx.ext.autodoc",
+ "sphinx.ext.autosummary",
+ "sphinx.ext.intersphinx",
+ "sphinx.ext.coverage",
+ "sphinx.ext.napoleon",
+ "sphinx.ext.todo",
+ "sphinx.ext.viewcode",
+]
+
+# autodoc/autosummary flags
+autoclass_content = "both"
+autodoc_default_flags = ["members"]
+autosummary_generate = True
+
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ["_templates"]
+
+# Allow markdown includes (so releases.md can include CHANGLEOG.md)
+# http://www.sphinx-doc.org/en/master/markdown.html
+source_parsers = {".md": "recommonmark.parser.CommonMarkParser"}
+
+# The suffix(es) of source filenames.
+# You can specify multiple suffix as a list of string:
+# source_suffix = ['.rst', '.md']
+source_suffix = [".rst", ".md"]
+
+# The encoding of source files.
+# source_encoding = 'utf-8-sig'
+
+# The master toctree document.
+master_doc = "index"
+
+# General information about the project.
+project = u"google-cloud-bigtable"
+copyright = u"2017, Google"
+author = u"Google APIs"
+
+# The version info for the project you're documenting, acts as replacement for
+# |version| and |release|, also used in various other places throughout the
+# built documents.
+#
+# The full version, including alpha/beta/rc tags.
+release = __version__
+# The short X.Y version.
+version = ".".join(release.split(".")[0:2])
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#
+# This is also used if you do content translation via gettext catalogs.
+# Usually you set "language" from the command line for these cases.
+language = None
+
+# There are two options for replacing |today|: either, you set today to some
+# non-false value, then it is used:
+# today = ''
+# Else, today_fmt is used as the format for a strftime call.
+# today_fmt = '%B %d, %Y'
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+exclude_patterns = ["_build"]
+
+# The reST default role (used for this markup: `text`) to use for all
+# documents.
+# default_role = None
+
+# If true, '()' will be appended to :func: etc. cross-reference text.
+# add_function_parentheses = True
+
+# If true, the current module name will be prepended to all description
+# unit titles (such as .. function::).
+# add_module_names = True
+
+# If true, sectionauthor and moduleauthor directives will be shown in the
+# output. They are ignored by default.
+# show_authors = False
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = "sphinx"
+
+# A list of ignored prefixes for module index sorting.
+# modindex_common_prefix = []
+
+# If true, keep warnings as "system message" paragraphs in the built documents.
+# keep_warnings = False
+
+# If true, `todo` and `todoList` produce output, else they produce nothing.
+todo_include_todos = True
+
+
+# -- Options for HTML output ----------------------------------------------
+
+# The theme to use for HTML and HTML Help pages. See the documentation for
+# a list of builtin themes.
+html_theme = "alabaster"
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further. For a list of options available for each theme, see the
+# documentation.
+html_theme_options = {
+ "description": "Google Cloud Client Libraries for Python",
+ "github_user": "googleapis",
+ "github_repo": "google-cloud-python",
+ "github_banner": True,
+ "font_family": "'Roboto', Georgia, sans",
+ "head_font_family": "'Roboto', Georgia, serif",
+ "code_font_family": "'Roboto Mono', 'Consolas', monospace",
+}
+
+# Add any paths that contain custom themes here, relative to this directory.
+# html_theme_path = []
+
+# The name for this set of Sphinx documents. If None, it defaults to
+# " v documentation".
+# html_title = None
+
+# A shorter title for the navigation bar. Default is the same as html_title.
+# html_short_title = None
+
+# The name of an image file (relative to this directory) to place at the top
+# of the sidebar.
+# html_logo = None
+
+# The name of an image file (within the static path) to use as favicon of the
+# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
+# pixels large.
+# html_favicon = None
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+# html_static_path = []
+
+# Add any extra paths that contain custom files (such as robots.txt or
+# .htaccess) here, relative to this directory. These files are copied
+# directly to the root of the documentation.
+# html_extra_path = []
+
+# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
+# using the given strftime format.
+# html_last_updated_fmt = '%b %d, %Y'
+
+# If true, SmartyPants will be used to convert quotes and dashes to
+# typographically correct entities.
+# html_use_smartypants = True
+
+# Custom sidebar templates, maps document names to template names.
+# html_sidebars = {}
+
+# Additional templates that should be rendered to pages, maps page names to
+# template names.
+# html_additional_pages = {}
+
+# If false, no module index is generated.
+# html_domain_indices = True
+
+# If false, no index is generated.
+# html_use_index = True
+
+# If true, the index is split into individual pages for each letter.
+# html_split_index = False
+
+# If true, links to the reST sources are added to the pages.
+# html_show_sourcelink = True
+
+# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
+# html_show_sphinx = True
+
+# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
+# html_show_copyright = True
+
+# If true, an OpenSearch description file will be output, and all pages will
+# contain a tag referring to it. The value of this option must be the
+# base URL from which the finished HTML is served.
+# html_use_opensearch = ''
+
+# This is the file name suffix for HTML files (e.g. ".xhtml").
+# html_file_suffix = None
+
+# Language to be used for generating the HTML full-text search index.
+# Sphinx supports the following languages:
+# 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja'
+# 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr'
+# html_search_language = 'en'
+
+# A dictionary with options for the search language support, empty by default.
+# Now only 'ja' uses this config value
+# html_search_options = {'type': 'default'}
+
+# The name of a javascript file (relative to the configuration directory) that
+# implements a search results scorer. If empty, the default will be used.
+# html_search_scorer = 'scorer.js'
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = "google-cloud-bigtable-doc"
+
+# -- Options for warnings ------------------------------------------------------
+
+
+suppress_warnings = [
+ # Temporarily suppress this to avoid "more than one target found for
+ # cross-reference" warning, which are intractable for us to avoid while in
+ # a mono-repo.
+ # See https://github.com/sphinx-doc/sphinx/blob
+ # /2a65ffeef5c107c19084fabdd706cdff3f52d93c/sphinx/domains/python.py#L843
+ "ref.python"
+]
+
+# -- Options for LaTeX output ---------------------------------------------
+
+latex_elements = {
+ # The paper size ('letterpaper' or 'a4paper').
+ #'papersize': 'letterpaper',
+ # The font size ('10pt', '11pt' or '12pt').
+ #'pointsize': '10pt',
+ # Additional stuff for the LaTeX preamble.
+ #'preamble': '',
+ # Latex figure (float) alignment
+ #'figure_align': 'htbp',
+}
+
+# Grouping the document tree into LaTeX files. List of tuples
+# (source start file, target name, title,
+# author, documentclass [howto, manual, or own class]).
+latex_documents = [
+ (
+ master_doc,
+ "google-cloud-bigtable.tex",
+ u"google-cloud-bigtable Documentation",
+ author,
+ "manual",
+ )
+]
+
+# The name of an image file (relative to this directory) to place at the top of
+# the title page.
+# latex_logo = None
+
+# For "manual" documents, if this is true, then toplevel headings are parts,
+# not chapters.
+# latex_use_parts = False
+
+# If true, show page references after internal links.
+# latex_show_pagerefs = False
+
+# If true, show URL addresses after external links.
+# latex_show_urls = False
+
+# Documents to append as an appendix to all manuals.
+# latex_appendices = []
+
+# If false, no module index is generated.
+# latex_domain_indices = True
+
+
+# -- Options for manual page output ---------------------------------------
+
+# One entry per manual page. List of tuples
+# (source start file, name, description, authors, manual section).
+man_pages = [
+ (
+ master_doc,
+ "google-cloud-bigtable",
+ u"google-cloud-bigtable Documentation",
+ [author],
+ 1,
+ )
+]
+
+# If true, show URL addresses after external links.
+# man_show_urls = False
+
+
+# -- Options for Texinfo output -------------------------------------------
+
+# Grouping the document tree into Texinfo files. List of tuples
+# (source start file, target name, title, author,
+# dir menu entry, description, category)
+texinfo_documents = [
+ (
+ master_doc,
+ "google-cloud-bigtable",
+ u"google-cloud-bigtable Documentation",
+ author,
+ "google-cloud-bigtable",
+ "GAPIC library for Bigtable",
+ "APIs",
+ )
+]
+
+# Documents to append as an appendix to all manuals.
+# texinfo_appendices = []
+
+# If false, no module index is generated.
+# texinfo_domain_indices = True
+
+# How to display URL addresses: 'footnote', 'no', or 'inline'.
+# texinfo_show_urls = 'footnote'
+
+# If true, do not generate a @detailmenu in the "Top" node's menu.
+# texinfo_no_detailmenu = False
+
+
+# Example configuration for intersphinx: refer to the Python standard library.
+intersphinx_mapping = {
+ "python": ("http://python.readthedocs.org/en/latest/", None),
+ "google-auth": ("https://google-auth.readthedocs.io/en/stable", None),
+ "google.api_core": ("https://googleapis.dev/python/google-api-core/latest", None),
+ "grpc": ("https://grpc.io/grpc/python/", None),
+ "requests": ("https://2.python-requests.org/en/master/", None),
+}
+
+
+# Napoleon settings
+napoleon_google_docstring = True
+napoleon_numpy_docstring = True
+napoleon_include_private_with_doc = False
+napoleon_include_special_with_doc = True
+napoleon_use_admonition_for_examples = False
+napoleon_use_admonition_for_notes = False
+napoleon_use_admonition_for_references = False
+napoleon_use_ivar = False
+napoleon_use_param = True
+napoleon_use_rtype = True
diff --git a/bigtable/docs/data-api.rst b/bigtable/docs/data-api.rst
new file mode 100644
index 000000000000..d35b50079426
--- /dev/null
+++ b/bigtable/docs/data-api.rst
@@ -0,0 +1,344 @@
+Data API
+========
+
+After creating a :class:`Table ` and some
+column families, you are ready to store and retrieve data.
+
+Cells vs. Columns vs. Column Families
++++++++++++++++++++++++++++++++++++++
+
+* As explained in the :doc:`table overview `, tables can
+ have many column families.
+* As described below, a table can also have many rows which are
+ specified by row keys.
+* Within a row, data is stored in a cell. A cell simply has a value (as
+ bytes) and a timestamp. The number of cells in each row can be
+ different, depending on what was stored in each row.
+* Each cell lies in a column (**not** a column family). A column is really
+ just a more **specific** modifier within a column family. A column
+ can be present in every column family, in only one or anywhere in between.
+* Within a column family there can be many columns. For example, within
+ the column family ``foo`` we could have columns ``bar`` and ``baz``.
+ These would typically be represented as ``foo:bar`` and ``foo:baz``.
+
+Modifying Data
+++++++++++++++
+
+Since data is stored in cells, which are stored in rows, we
+use the metaphor of a **row** in classes that are used to modify
+(write, update, delete) data in a
+:class:`Table `.
+
+Direct vs. Conditional vs. Append
+---------------------------------
+
+There are three ways to modify data in a table, described by the
+`MutateRow`_, `CheckAndMutateRow`_ and `ReadModifyWriteRow`_ API
+methods.
+
+* The **direct** way is via `MutateRow`_ which involves simply
+ adding, overwriting or deleting cells. The
+ :class:`DirectRow ` class
+ handles direct mutations.
+* The **conditional** way is via `CheckAndMutateRow`_. This method
+ first checks if some filter is matched in a given row, then
+ applies one of two sets of mutations, depending on if a match
+ occurred or not. (These mutation sets are called the "true
+ mutations" and "false mutations".) The
+ :class:`ConditionalRow ` class
+ handles conditional mutations.
+* The **append** way is via `ReadModifyWriteRow`_. This simply
+ appends (as bytes) or increments (as an integer) data in a presumed
+ existing cell in a row. The
+ :class:`AppendRow ` class
+ handles append mutations.
+
+Row Factory
+-----------
+
+A single factory can be used to create any of the three row types.
+To create a :class:`DirectRow `:
+
+.. code:: python
+
+ row = table.row(row_key)
+
+Unlike the previous string values we've used before, the row key must
+be ``bytes``.
+
+To create a :class:`ConditionalRow `,
+first create a :class:`RowFilter ` and
+then
+
+.. code:: python
+
+ cond_row = table.row(row_key, filter_=filter_)
+
+To create an :class:`AppendRow `
+
+.. code:: python
+
+ append_row = table.row(row_key, append=True)
+
+Building Up Mutations
+---------------------
+
+In all three cases, a set of mutations (or two sets) are built up
+on a row before they are sent of in a batch via
+
+.. code:: python
+
+ row.commit()
+
+Direct Mutations
+----------------
+
+Direct mutations can be added via one of four methods
+
+* :meth:`set_cell() ` allows a
+ single value to be written to a column
+
+ .. code:: python
+
+ row.set_cell(column_family_id, column, value,
+ timestamp=timestamp)
+
+ If the ``timestamp`` is omitted, the current time on the Google Cloud
+ Bigtable server will be used when the cell is stored.
+
+ The value can either be bytes or an integer, which will be converted to
+ bytes as a signed 64-bit integer.
+
+* :meth:`delete_cell() ` deletes
+ all cells (i.e. for all timestamps) in a given column
+
+ .. code:: python
+
+ row.delete_cell(column_family_id, column)
+
+ Remember, this only happens in the ``row`` we are using.
+
+ If we only want to delete cells from a limited range of time, a
+ :class:`TimestampRange ` can
+ be used
+
+ .. code:: python
+
+ row.delete_cell(column_family_id, column,
+ time_range=time_range)
+
+* :meth:`delete_cells() ` does
+ the same thing as
+ :meth:`delete_cell() `,
+ but accepts a list of columns in a column family rather than a single one.
+
+ .. code:: python
+
+ row.delete_cells(column_family_id, [column1, column2],
+ time_range=time_range)
+
+ In addition, if we want to delete cells from every column in a column family,
+ the special :attr:`ALL_COLUMNS `
+ value can be used
+
+ .. code:: python
+
+ row.delete_cells(column_family_id, row.ALL_COLUMNS,
+ time_range=time_range)
+
+* :meth:`delete() ` will delete the
+ entire row
+
+ .. code:: python
+
+ row.delete()
+
+Conditional Mutations
+---------------------
+
+Making **conditional** modifications is essentially identical
+to **direct** modifications: it uses the exact same methods
+to accumulate mutations.
+
+However, each mutation added must specify a ``state``: will the mutation be
+applied if the filter matches or if it fails to match.
+
+For example:
+
+.. code:: python
+
+ cond_row.set_cell(column_family_id, column, value,
+ timestamp=timestamp, state=True)
+
+will add to the set of true mutations.
+
+Append Mutations
+----------------
+
+Append mutations can be added via one of two methods
+
+* :meth:`append_cell_value() `
+ appends a bytes value to an existing cell:
+
+ .. code:: python
+
+ append_row.append_cell_value(column_family_id, column, bytes_value)
+
+* :meth:`increment_cell_value() `
+ increments an integer value in an existing cell:
+
+ .. code:: python
+
+ append_row.increment_cell_value(column_family_id, column, int_value)
+
+ Since only bytes are stored in a cell, the cell value is decoded as
+ a signed 64-bit integer before being incremented. (This happens on
+ the Google Cloud Bigtable server, not in the library.)
+
+Notice that no timestamp was specified. This is because **append** mutations
+operate on the latest value of the specified column.
+
+If there are no cells in the specified column, then the empty string (bytes
+case) or zero (integer case) are the assumed values.
+
+Starting Fresh
+--------------
+
+If accumulated mutations need to be dropped, use
+
+.. code:: python
+
+ row.clear()
+
+Reading Data
+++++++++++++
+
+Read Single Row from a Table
+----------------------------
+
+To make a `ReadRows`_ API request for a single row key, use
+:meth:`Table.read_row() `:
+
+.. code:: python
+
+ >>> row_data = table.read_row(row_key)
+ >>> row_data.cells
+ {
+ u'fam1': {
+ b'col1': [
+ ,
+ ,
+ ],
+ b'col2': [
+ ,
+ ],
+ },
+ u'fam2': {
+ b'col3': [
+ ,
+ ,
+ ,
+ ],
+ },
+ }
+ >>> cell = row_data.cells[u'fam1'][b'col1'][0]
+ >>> cell
+