The magics API do not support setting client options

[boazsade]

Environment details

• OS type and version: jupyter notebook
• Python version: Python 3.7.3
• pip version: pip 20.1.1
• google-cloud-bigquery version: 1.25.0

Steps to reproduce

This is a missing option that do exists in google bigquery client - "https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.client.Client.html#google.cloud.bigquery.client.Client"
But is missing from "https://github.com/googleapis/python-bigquery/blob/master/google/cloud/bigquery/magics.py"

Code example

client = bigquery.Client(
 project=project,
 credentials=context.credentials,
 default_query_job_config=context.default_query_job_config,
 client_info=client_info.ClientInfo(user_agent=IPYTHON_USER_AGENT),
 )

Note the the client_option is missing in creating the client in the code above

The patch to apply to fix this flow:

diff --git a/google/cloud/bigquery/magics.py b/google/cloud/bigquery/magics.py
index 7128e32..00e26d0 100644
--- a/google/cloud/bigquery/magics.py
+++ b/google/cloud/bigquery/magics.py
@@ -140,6 +140,10 @@
 Out[7]: num
 ...: -------
 ...: 0 17
+ Executing query with job ID: bf633912-af2c-4780-b568-5d868058632b
+ Query executing: 2.61s
+ Query complete after 2.92s
+
 """
 
 from __future__ import print_function
@@ -183,6 +187,7 @@ class Context(object):
 self._project = None
 self._connection = None
 self._default_query_job_config = bigquery.QueryJobConfig()
+ self._client_options = google.api_core.client_options.ClientOptions()
 
 @property
 def credentials(self):
@@ -244,6 +249,30 @@ class Context(object):
 def project(self, value):
 self._project = value
 
+ @property
+ def client_options(self):
+ """ google.api_core.client_options.ClientOptions: client options to be used through IPython
+ Magics
+
+ Note::
+ The client options does not need to be explicitly defined if you have an
+ no special network conenctions are required. Normally you would
+ be using https://www.googleapis.com/ end point
+
+ Example:
+ Manually setting the endpoint:
+
+ >>> from google.cloud.bigquery import magics
+ >>> client_option = {}
+ >>> client_option['api_endpoint'] = "https://some.special.url"
+ >>> magics.context.client_options = client_option
+ """
+ return self._client_options
+
+ @client_options.setter
+ def client_options(self, value):
+ self._client_options = value
+
 @property
 def default_query_job_config(self):
 """google.cloud.bigquery.job.QueryJobConfig: Default job
@@ -459,6 +488,18 @@ def _create_dataset_if_necessary(client, dataset_id):
 "name (ex. $my_dict_var)."
 ),
 )
+@magic_arguments.argument(
+ "--options",
+ nargs="+",
+ default=None,
+ help=(
+ "Parameters to format the query string. If present, the --options "
+ "flag should be followed by a string representation of a dictionary "
+ "in the format (Union[google.api_core.client_options.ClientOptions, Dict])"
+ "or a reference to a dictionary in the same format. The dictionary "
+ "key/value matches does at google.api_core.client_options.ClientOptions"
+ ),
+)
 def _cell_magic(line, query):
 """Underlying function for bigquery cell magic
 
@@ -501,6 +542,7 @@ def _cell_magic(line, query):
 credentials=context.credentials,
 default_query_job_config=context.default_query_job_config,
 client_info=client_info.ClientInfo(user_agent=IPYTHON_USER_AGENT),
+ client_options=context.client_options
 )
 if context._connection:
 client._connection = context._connection

[plamut]

@boazsade Thanks for the suggestion and the patch!

I'll just check if Product has any objections (cc: @shollyman), and if not, we'll add this.

P.S.: If confirmed, would you mind opening an actual PR with added tests, or are you fine with us taking the patch as a base and create a PR ourselves?
(if significant portions of the patch are used, it would probably still require signing the CLA, though)

[boazsade]

You can take this pach, this is something that we really need to be part of the code base, thanks for the reply

[boazsade]

@plamut is there any progress with this issue? I was unable to create pull request, what about applying the patch? was that approved? thanks

[plamut]

@boazsade Not at this point, unfortunately, but I can re-check at this week's meeting. Lately one of the primary things in focus are preparations to dropping Python 2 support and moving them to the new code generator (for those who have some auto-generated parts).

[yoavcloud]

Hi @plamut. I wanted to check in and see if there's an update regarding including this patch in an upcoming release. One of our customers is in the process of migrating their data environment to BigQuery and this has become a priority. Thanks in advance.

[cguardia]

@yoavcloud Hi, I'm working on a PR to add your patch. Should be ready for review in a couple of days.