WIP: [AIRFLOW-1894] Google cloud bigquery by jmcarp · Pull Request #4607 · apache/airflow

jmcarp · 2019-01-28T04:08:01Z

Make sure you have checked all steps below.

Jira

My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-1894
- In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.

Description

Here are some details about my PR, including screenshots of any UI changes:

Tests

My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters (not including Jira issue reference)
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"

Documentation

In case of new functionality, my PR adds documentation that describes how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.
- All the public functions and the classes in the PR contain docstrings that explain what it does

Code Quality

Passes flake8

codecov-io · 2019-01-31T02:33:31Z

Codecov Report

Merging #4607 into master will increase coverage by 0.23%.
The diff coverage is 47.05%.

@@ Coverage Diff @@## master #4607 +/- ##
==========================================
+ Coverage 74.3% 74.53% +0.23% 
==========================================
Files 426 426 Lines 27867 27577 -290 ==========================================
- Hits 20706 20554 -152 + Misses 7161 7023 -138

Impacted Files	Coverage Δ
...rflow/contrib/operators/bigquery_check_operator.py	`0% <ø> (ø)`	⬆️
airflow/contrib/operators/gcs_to_bq.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_get_data.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_to_bigquery.py	`0% <0%> (ø)`	⬆️
...ontrib/operators/bigquery_table_delete_operator.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_to_gcs.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_operator.py	`95.93% <100%> (+2.36%)`	⬆️
airflow/contrib/hooks/bigquery_hook.py	`60.69% <47.29%> (+2.62%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f07f3a8...d903f24. Read the comment docs.

jmcarp · 2019-02-04T17:07:00Z

@kaxil @potiuk: tests are passing, so this is ready for review when you have time. A few differences to point out:

The current python bigquery client library implements the dbapi connection and cursor interfaces, so we can drop our custom implementations
The current client also allows for polling a job by calling its result method, so we can also drop our custom polling logic
Because the current client includes classes for most job options that include basic validation, we're also able to drop some custom validation
Extra methods on the bigquery cursor have been moved to the hook

jmcarp · 2019-02-08T04:52:43Z

Ping @Fokko @feng-tao @mik-laj (not sure who knows this code best). I'm hoping to add some features after this is ready, so would be great to get feedback when you all have time.

kaxil · 2019-02-12T23:30:42Z

 sql,
 bigquery_conn_id='bigquery_default',
- use_legacy_sql=True,
+ use_legacy_sql=False,


We shouldn't change this behaviour

kaxil · 2019-02-12T23:32:18Z

 allow_jagged_rows=self.allow_jagged_rows,
-src_fmt_configs=self.src_fmt_configs,
- labels=self.labels
+external_config_options=self.external_config_options,


This will have to be for 2.0 as it will break things. Also, it needs to be backward-compatible to make updation smooth from 1.X to 2.0

Haven't yet reviewed the entire PR but schemed through it. Will try to find some time to look at it more thoroughly

Fokko · 2019-02-27T11:55:32Z

I'm seeing some good things in this PR, and it looks like it will simplify the logic. Any plans of moving this forward @jmcarp ?

potiuk · 2019-02-05T13:02:51Z

+ :type project_id: str
 """
- service = self.get_service()
+ project_id = project_id if project_id is not None else self.project_id


I think you could use fallback_to_default_project_id decorator. It has additional logic to raise the exception if none of the project_id s is specified. and you could remove this if altogether then. It forces to use keyword parameters though.

potiuk · 2019-03-01T17:24:19Z

- project_id=project,
- use_legacy_sql=self.use_legacy_sql,
+ def get_client(self, project_id=None):
+ project_id = project_id if project_id is not None else self.project_id


Same as below - fallback_to_default_project_id decorator is nicer way I think

potiuk · 2019-03-01T17:29:39Z

-}
-})
+ if external_config_options is not None:
+if not isinstance(external_config_options, type(external_config.options)):


I think you should use dict explicitly here. What if external_config.options are None (seem to be default).

potiuk · 2019-03-01T17:32:05Z

 https://cloud.google.com/bigquery/docs/locations#specifying_your_location
 :type location: str
 """
+ project_id = project_id if project_id is not None else self.project_id


Again - I think using fallback decorator is nicer as it keeps the project_id logic in one place.

potiuk · 2019-03-01T17:33:34Z

 passed to BigQuery
 :type labels: dict
 """
+ project_id = project_id if project_id is not None else self.project_id


Same here with decorator.

potiuk · 2019-03-01T17:34:09Z

 time_partitioning. The order of columns given determines the sort order.
 :type cluster_fields: list of str
 """
+ project_id = project_id if project_id is not None else self.project_id


potiuk · 2019-03-01T17:35:49Z

 """
- # check to see if the table exists
- table_id = table_resource['tableReference']['tableId']
 project_id = project_id if project_id is not None else self.project_id


And here :)

potiuk · 2019-03-01T17:36:34Z

 even if any insertion errors occur.
 :type fail_on_error: bool
 """
+ project_id = project_id if project_id is not None else self.project_id


here too :)

bjoernpollex-sc · 2019-03-11T08:35:21Z

 private_key=private_key)

- def table_exists(self, project_id, dataset_id, table_id):
+ def table_exists(self, dataset_id, table_id, project_id=None):


Hi, thanks for putting this together, this is great stuff!
Regarding these convenience methods, I'd argue that it would be better if the functionality would be implemented on the connection object returned by get_conn, and the here we just delegate, like this:
deftable_exists(self, dataset_id, table_id, project_id=None): returnself.get_conn().table_exists(dataset_id, table_id, project_id)
The reason is that sometimes it is very useful to have explicit control over when and where connections get created, for example when using multi-threading for I/O optimization. We've also had code that spuriously crashed because it was creating too many new connections (implicitly in such convenience methods), and at some point the authentication requests hit a rate-limit.

potiuk · 2019-04-20T15:29:23Z

Hey @jmcarp -> are you still working on this? As I am now committer, I am happy to review that one as well as soon as it gets rebased (if it's still something you want to add).

kaxil · 2019-04-30T21:35:07Z

@jmcarp Are you still working on this?

mik-laj · 2019-07-27T13:34:47Z

@jmcarp Can I help you with this? This looks very interesting to me.

mik-laj · 2019-09-05T13:59:23Z

Hi.

I made a change in the base class - GoogleCloudBaseHook. Your PR may need to be changed. Could you do rebase?

Cheers

Refenence:
#5907

potiuk · 2019-09-06T13:39:00Z

@jmcarp - is it possible to rebase/complete the work on this one ? It blocks us from moving operators/hook from contrib to core - after we move the operators/hooks it will be much more difficult to merge.

potiuk · 2019-09-21T19:04:46Z

Hey @jmcarp - are you still doing it? Or should we close that one?

stale · 2019-11-20T16:52:27Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

jmcarp · 2019-01-28T04:08:01Z

Make sure you have checked all steps below.

Jira

My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-1894
- In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.

Description

Here are some details about my PR, including screenshots of any UI changes:

Tests

My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters (not including Jira issue reference)
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"

Documentation

In case of new functionality, my PR adds documentation that describes how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.
- All the public functions and the classes in the PR contain docstrings that explain what it does

Code Quality

Passes flake8

codecov-io · 2019-01-31T02:33:31Z

Codecov Report

Merging #4607 into master will increase coverage by 0.23%.
The diff coverage is 47.05%.

@@ Coverage Diff @@## master #4607 +/- ##
==========================================
+ Coverage 74.3% 74.53% +0.23% 
==========================================
Files 426 426 Lines 27867 27577 -290 ==========================================
- Hits 20706 20554 -152 + Misses 7161 7023 -138

Impacted Files	Coverage Δ
...rflow/contrib/operators/bigquery_check_operator.py	`0% <ø> (ø)`	⬆️
airflow/contrib/operators/gcs_to_bq.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_get_data.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_to_bigquery.py	`0% <0%> (ø)`	⬆️
...ontrib/operators/bigquery_table_delete_operator.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_to_gcs.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_operator.py	`95.93% <100%> (+2.36%)`	⬆️
airflow/contrib/hooks/bigquery_hook.py	`60.69% <47.29%> (+2.62%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f07f3a8...d903f24. Read the comment docs.

jmcarp · 2019-02-04T17:07:00Z

@kaxil @potiuk: tests are passing, so this is ready for review when you have time. A few differences to point out:

The current python bigquery client library implements the dbapi connection and cursor interfaces, so we can drop our custom implementations
The current client also allows for polling a job by calling its result method, so we can also drop our custom polling logic
Because the current client includes classes for most job options that include basic validation, we're also able to drop some custom validation
Extra methods on the bigquery cursor have been moved to the hook

jmcarp · 2019-02-08T04:52:43Z

Ping @Fokko @feng-tao @mik-laj (not sure who knows this code best). I'm hoping to add some features after this is ready, so would be great to get feedback when you all have time.

kaxil · 2019-02-12T23:30:42Z

 sql,
 bigquery_conn_id='bigquery_default',
- use_legacy_sql=True,
+ use_legacy_sql=False,


We shouldn't change this behaviour

kaxil · 2019-02-12T23:32:18Z

 allow_jagged_rows=self.allow_jagged_rows,
-src_fmt_configs=self.src_fmt_configs,
- labels=self.labels
+external_config_options=self.external_config_options,


This will have to be for 2.0 as it will break things. Also, it needs to be backward-compatible to make updation smooth from 1.X to 2.0

Haven't yet reviewed the entire PR but schemed through it. Will try to find some time to look at it more thoroughly

Fokko · 2019-02-27T11:55:32Z

I'm seeing some good things in this PR, and it looks like it will simplify the logic. Any plans of moving this forward @jmcarp ?

potiuk · 2019-02-05T13:02:51Z

+ :type project_id: str
 """
- service = self.get_service()
+ project_id = project_id if project_id is not None else self.project_id


I think you could use fallback_to_default_project_id decorator. It has additional logic to raise the exception if none of the project_id s is specified. and you could remove this if altogether then. It forces to use keyword parameters though.

potiuk · 2019-03-01T17:24:19Z

- project_id=project,
- use_legacy_sql=self.use_legacy_sql,
+ def get_client(self, project_id=None):
+ project_id = project_id if project_id is not None else self.project_id


Same as below - fallback_to_default_project_id decorator is nicer way I think

potiuk · 2019-03-01T17:29:39Z

-}
-})
+ if external_config_options is not None:
+if not isinstance(external_config_options, type(external_config.options)):


I think you should use dict explicitly here. What if external_config.options are None (seem to be default).

potiuk · 2019-03-01T17:32:05Z

 https://cloud.google.com/bigquery/docs/locations#specifying_your_location
 :type location: str
 """
+ project_id = project_id if project_id is not None else self.project_id


Again - I think using fallback decorator is nicer as it keeps the project_id logic in one place.

potiuk · 2019-03-01T17:33:34Z

 passed to BigQuery
 :type labels: dict
 """
+ project_id = project_id if project_id is not None else self.project_id


Same here with decorator.

potiuk · 2019-03-01T17:34:09Z

 time_partitioning. The order of columns given determines the sort order.
 :type cluster_fields: list of str
 """
+ project_id = project_id if project_id is not None else self.project_id


potiuk · 2019-03-01T17:35:49Z

 """
- # check to see if the table exists
- table_id = table_resource['tableReference']['tableId']
 project_id = project_id if project_id is not None else self.project_id


And here :)

potiuk · 2019-03-01T17:36:34Z

 even if any insertion errors occur.
 :type fail_on_error: bool
 """
+ project_id = project_id if project_id is not None else self.project_id


here too :)

bjoernpollex-sc · 2019-03-11T08:35:21Z

 private_key=private_key)

- def table_exists(self, project_id, dataset_id, table_id):
+ def table_exists(self, dataset_id, table_id, project_id=None):


Hi, thanks for putting this together, this is great stuff!
Regarding these convenience methods, I'd argue that it would be better if the functionality would be implemented on the connection object returned by get_conn, and the here we just delegate, like this:
deftable_exists(self, dataset_id, table_id, project_id=None): returnself.get_conn().table_exists(dataset_id, table_id, project_id)
The reason is that sometimes it is very useful to have explicit control over when and where connections get created, for example when using multi-threading for I/O optimization. We've also had code that spuriously crashed because it was creating too many new connections (implicitly in such convenience methods), and at some point the authentication requests hit a rate-limit.

potiuk · 2019-04-20T15:29:23Z

Hey @jmcarp -> are you still working on this? As I am now committer, I am happy to review that one as well as soon as it gets rebased (if it's still something you want to add).

kaxil · 2019-04-30T21:35:07Z

@jmcarp Are you still working on this?

mik-laj · 2019-07-27T13:34:47Z

@jmcarp Can I help you with this? This looks very interesting to me.

mik-laj · 2019-09-05T13:59:23Z

Hi.

I made a change in the base class - GoogleCloudBaseHook. Your PR may need to be changed. Could you do rebase?

Cheers

Refenence:
#5907

potiuk · 2019-09-06T13:39:00Z

@jmcarp - is it possible to rebase/complete the work on this one ? It blocks us from moving operators/hook from contrib to core - after we move the operators/hooks it will be much more difficult to merge.

potiuk · 2019-09-21T19:04:46Z

Hey @jmcarp - are you still doing it? Or should we close that one?

stale · 2019-11-20T16:52:27Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

jmcarp · 2019-01-28T04:08:01Z

Make sure you have checked all steps below.

Jira

My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-1894
- In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.

Description

Here are some details about my PR, including screenshots of any UI changes:

Tests

My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters (not including Jira issue reference)
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"

Documentation

In case of new functionality, my PR adds documentation that describes how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.
- All the public functions and the classes in the PR contain docstrings that explain what it does

Code Quality

Passes flake8

codecov-io · 2019-01-31T02:33:31Z

Codecov Report

Merging #4607 into master will increase coverage by 0.23%.
The diff coverage is 47.05%.

@@ Coverage Diff @@## master #4607 +/- ##
==========================================
+ Coverage 74.3% 74.53% +0.23% 
==========================================
Files 426 426 Lines 27867 27577 -290 ==========================================
- Hits 20706 20554 -152 + Misses 7161 7023 -138

Impacted Files	Coverage Δ
...rflow/contrib/operators/bigquery_check_operator.py	`0% <ø> (ø)`	⬆️
airflow/contrib/operators/gcs_to_bq.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_get_data.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_to_bigquery.py	`0% <0%> (ø)`	⬆️
...ontrib/operators/bigquery_table_delete_operator.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_to_gcs.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_operator.py	`95.93% <100%> (+2.36%)`	⬆️
airflow/contrib/hooks/bigquery_hook.py	`60.69% <47.29%> (+2.62%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f07f3a8...d903f24. Read the comment docs.

jmcarp · 2019-02-04T17:07:00Z

@kaxil @potiuk: tests are passing, so this is ready for review when you have time. A few differences to point out:

The current python bigquery client library implements the dbapi connection and cursor interfaces, so we can drop our custom implementations
The current client also allows for polling a job by calling its result method, so we can also drop our custom polling logic
Because the current client includes classes for most job options that include basic validation, we're also able to drop some custom validation
Extra methods on the bigquery cursor have been moved to the hook

jmcarp · 2019-02-08T04:52:43Z

Ping @Fokko @feng-tao @mik-laj (not sure who knows this code best). I'm hoping to add some features after this is ready, so would be great to get feedback when you all have time.

kaxil · 2019-02-12T23:30:42Z

 sql,
 bigquery_conn_id='bigquery_default',
- use_legacy_sql=True,
+ use_legacy_sql=False,


We shouldn't change this behaviour

kaxil · 2019-02-12T23:32:18Z

 allow_jagged_rows=self.allow_jagged_rows,
-src_fmt_configs=self.src_fmt_configs,
- labels=self.labels
+external_config_options=self.external_config_options,


This will have to be for 2.0 as it will break things. Also, it needs to be backward-compatible to make updation smooth from 1.X to 2.0

Haven't yet reviewed the entire PR but schemed through it. Will try to find some time to look at it more thoroughly

Fokko · 2019-02-27T11:55:32Z

I'm seeing some good things in this PR, and it looks like it will simplify the logic. Any plans of moving this forward @jmcarp ?

potiuk · 2019-02-05T13:02:51Z

+ :type project_id: str
 """
- service = self.get_service()
+ project_id = project_id if project_id is not None else self.project_id


I think you could use fallback_to_default_project_id decorator. It has additional logic to raise the exception if none of the project_id s is specified. and you could remove this if altogether then. It forces to use keyword parameters though.

potiuk · 2019-03-01T17:24:19Z

- project_id=project,
- use_legacy_sql=self.use_legacy_sql,
+ def get_client(self, project_id=None):
+ project_id = project_id if project_id is not None else self.project_id


Same as below - fallback_to_default_project_id decorator is nicer way I think

potiuk · 2019-03-01T17:29:39Z

-}
-})
+ if external_config_options is not None:
+if not isinstance(external_config_options, type(external_config.options)):


I think you should use dict explicitly here. What if external_config.options are None (seem to be default).

potiuk · 2019-03-01T17:32:05Z

 https://cloud.google.com/bigquery/docs/locations#specifying_your_location
 :type location: str
 """
+ project_id = project_id if project_id is not None else self.project_id


Again - I think using fallback decorator is nicer as it keeps the project_id logic in one place.

potiuk · 2019-03-01T17:33:34Z

 passed to BigQuery
 :type labels: dict
 """
+ project_id = project_id if project_id is not None else self.project_id


Same here with decorator.

potiuk · 2019-03-01T17:34:09Z

 time_partitioning. The order of columns given determines the sort order.
 :type cluster_fields: list of str
 """
+ project_id = project_id if project_id is not None else self.project_id


potiuk · 2019-03-01T17:35:49Z

 """
- # check to see if the table exists
- table_id = table_resource['tableReference']['tableId']
 project_id = project_id if project_id is not None else self.project_id


And here :)

potiuk · 2019-03-01T17:36:34Z

 even if any insertion errors occur.
 :type fail_on_error: bool
 """
+ project_id = project_id if project_id is not None else self.project_id


here too :)

bjoernpollex-sc · 2019-03-11T08:35:21Z

 private_key=private_key)

- def table_exists(self, project_id, dataset_id, table_id):
+ def table_exists(self, dataset_id, table_id, project_id=None):


Hi, thanks for putting this together, this is great stuff!
Regarding these convenience methods, I'd argue that it would be better if the functionality would be implemented on the connection object returned by get_conn, and the here we just delegate, like this:
deftable_exists(self, dataset_id, table_id, project_id=None): returnself.get_conn().table_exists(dataset_id, table_id, project_id)
The reason is that sometimes it is very useful to have explicit control over when and where connections get created, for example when using multi-threading for I/O optimization. We've also had code that spuriously crashed because it was creating too many new connections (implicitly in such convenience methods), and at some point the authentication requests hit a rate-limit.

potiuk · 2019-04-20T15:29:23Z

Hey @jmcarp -> are you still working on this? As I am now committer, I am happy to review that one as well as soon as it gets rebased (if it's still something you want to add).

kaxil · 2019-04-30T21:35:07Z

@jmcarp Are you still working on this?

mik-laj · 2019-07-27T13:34:47Z

@jmcarp Can I help you with this? This looks very interesting to me.

mik-laj · 2019-09-05T13:59:23Z

Hi.

I made a change in the base class - GoogleCloudBaseHook. Your PR may need to be changed. Could you do rebase?

Cheers

Refenence:
#5907

potiuk · 2019-09-06T13:39:00Z

@jmcarp - is it possible to rebase/complete the work on this one ? It blocks us from moving operators/hook from contrib to core - after we move the operators/hooks it will be much more difficult to merge.

potiuk · 2019-09-21T19:04:46Z

Hey @jmcarp - are you still doing it? Or should we close that one?

stale · 2019-11-20T16:52:27Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

jmcarp · 2019-01-28T04:08:01Z

Make sure you have checked all steps below.

Jira

My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-1894
- In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.

Description

Here are some details about my PR, including screenshots of any UI changes:

Tests

My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters (not including Jira issue reference)
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"

Documentation

In case of new functionality, my PR adds documentation that describes how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.
- All the public functions and the classes in the PR contain docstrings that explain what it does

Code Quality

Passes flake8

codecov-io · 2019-01-31T02:33:31Z

Codecov Report

Merging #4607 into master will increase coverage by 0.23%.
The diff coverage is 47.05%.

@@ Coverage Diff @@## master #4607 +/- ##
==========================================
+ Coverage 74.3% 74.53% +0.23% 
==========================================
Files 426 426 Lines 27867 27577 -290 ==========================================
- Hits 20706 20554 -152 + Misses 7161 7023 -138

Impacted Files	Coverage Δ
...rflow/contrib/operators/bigquery_check_operator.py	`0% <ø> (ø)`	⬆️
airflow/contrib/operators/gcs_to_bq.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_get_data.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_to_bigquery.py	`0% <0%> (ø)`	⬆️
...ontrib/operators/bigquery_table_delete_operator.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_to_gcs.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_operator.py	`95.93% <100%> (+2.36%)`	⬆️
airflow/contrib/hooks/bigquery_hook.py	`60.69% <47.29%> (+2.62%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f07f3a8...d903f24. Read the comment docs.

jmcarp · 2019-02-04T17:07:00Z

@kaxil @potiuk: tests are passing, so this is ready for review when you have time. A few differences to point out:

The current python bigquery client library implements the dbapi connection and cursor interfaces, so we can drop our custom implementations
The current client also allows for polling a job by calling its result method, so we can also drop our custom polling logic
Because the current client includes classes for most job options that include basic validation, we're also able to drop some custom validation
Extra methods on the bigquery cursor have been moved to the hook

jmcarp · 2019-02-08T04:52:43Z

Ping @Fokko @feng-tao @mik-laj (not sure who knows this code best). I'm hoping to add some features after this is ready, so would be great to get feedback when you all have time.

kaxil · 2019-02-12T23:30:42Z

 sql,
 bigquery_conn_id='bigquery_default',
- use_legacy_sql=True,
+ use_legacy_sql=False,


We shouldn't change this behaviour

kaxil · 2019-02-12T23:32:18Z

 allow_jagged_rows=self.allow_jagged_rows,
-src_fmt_configs=self.src_fmt_configs,
- labels=self.labels
+external_config_options=self.external_config_options,


This will have to be for 2.0 as it will break things. Also, it needs to be backward-compatible to make updation smooth from 1.X to 2.0

Haven't yet reviewed the entire PR but schemed through it. Will try to find some time to look at it more thoroughly

Fokko · 2019-02-27T11:55:32Z

I'm seeing some good things in this PR, and it looks like it will simplify the logic. Any plans of moving this forward @jmcarp ?

potiuk · 2019-02-05T13:02:51Z

+ :type project_id: str
 """
- service = self.get_service()
+ project_id = project_id if project_id is not None else self.project_id


I think you could use fallback_to_default_project_id decorator. It has additional logic to raise the exception if none of the project_id s is specified. and you could remove this if altogether then. It forces to use keyword parameters though.

potiuk · 2019-03-01T17:24:19Z

- project_id=project,
- use_legacy_sql=self.use_legacy_sql,
+ def get_client(self, project_id=None):
+ project_id = project_id if project_id is not None else self.project_id


Same as below - fallback_to_default_project_id decorator is nicer way I think

potiuk · 2019-03-01T17:29:39Z

-}
-})
+ if external_config_options is not None:
+if not isinstance(external_config_options, type(external_config.options)):


I think you should use dict explicitly here. What if external_config.options are None (seem to be default).

potiuk · 2019-03-01T17:32:05Z

 https://cloud.google.com/bigquery/docs/locations#specifying_your_location
 :type location: str
 """
+ project_id = project_id if project_id is not None else self.project_id


Again - I think using fallback decorator is nicer as it keeps the project_id logic in one place.

potiuk · 2019-03-01T17:33:34Z

 passed to BigQuery
 :type labels: dict
 """
+ project_id = project_id if project_id is not None else self.project_id


Same here with decorator.

potiuk · 2019-03-01T17:34:09Z

 time_partitioning. The order of columns given determines the sort order.
 :type cluster_fields: list of str
 """
+ project_id = project_id if project_id is not None else self.project_id


potiuk · 2019-03-01T17:35:49Z

 """
- # check to see if the table exists
- table_id = table_resource['tableReference']['tableId']
 project_id = project_id if project_id is not None else self.project_id


And here :)

potiuk · 2019-03-01T17:36:34Z

 even if any insertion errors occur.
 :type fail_on_error: bool
 """
+ project_id = project_id if project_id is not None else self.project_id


here too :)

bjoernpollex-sc · 2019-03-11T08:35:21Z

 private_key=private_key)

- def table_exists(self, project_id, dataset_id, table_id):
+ def table_exists(self, dataset_id, table_id, project_id=None):


Hi, thanks for putting this together, this is great stuff!
Regarding these convenience methods, I'd argue that it would be better if the functionality would be implemented on the connection object returned by get_conn, and the here we just delegate, like this:
deftable_exists(self, dataset_id, table_id, project_id=None): returnself.get_conn().table_exists(dataset_id, table_id, project_id)
The reason is that sometimes it is very useful to have explicit control over when and where connections get created, for example when using multi-threading for I/O optimization. We've also had code that spuriously crashed because it was creating too many new connections (implicitly in such convenience methods), and at some point the authentication requests hit a rate-limit.

potiuk · 2019-04-20T15:29:23Z

Hey @jmcarp -> are you still working on this? As I am now committer, I am happy to review that one as well as soon as it gets rebased (if it's still something you want to add).

kaxil · 2019-04-30T21:35:07Z

@jmcarp Are you still working on this?

mik-laj · 2019-07-27T13:34:47Z

@jmcarp Can I help you with this? This looks very interesting to me.

mik-laj · 2019-09-05T13:59:23Z

Hi.

I made a change in the base class - GoogleCloudBaseHook. Your PR may need to be changed. Could you do rebase?

Cheers

Refenence:
#5907

potiuk · 2019-09-06T13:39:00Z

@jmcarp - is it possible to rebase/complete the work on this one ? It blocks us from moving operators/hook from contrib to core - after we move the operators/hooks it will be much more difficult to merge.

potiuk · 2019-09-21T19:04:46Z

Hey @jmcarp - are you still doing it? Or should we close that one?

stale · 2019-11-20T16:52:27Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

jmcarp · 2019-01-28T04:08:01Z

Make sure you have checked all steps below.

Jira

My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-1894
- In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.

Description

Here are some details about my PR, including screenshots of any UI changes:

Tests

My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters (not including Jira issue reference)
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"

Documentation

In case of new functionality, my PR adds documentation that describes how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.
- All the public functions and the classes in the PR contain docstrings that explain what it does

Code Quality

Passes flake8

codecov-io · 2019-01-31T02:33:31Z

Codecov Report

Merging #4607 into master will increase coverage by 0.23%.
The diff coverage is 47.05%.

@@ Coverage Diff @@## master #4607 +/- ##
==========================================
+ Coverage 74.3% 74.53% +0.23% 
==========================================
Files 426 426 Lines 27867 27577 -290 ==========================================
- Hits 20706 20554 -152 + Misses 7161 7023 -138

Impacted Files	Coverage Δ
...rflow/contrib/operators/bigquery_check_operator.py	`0% <ø> (ø)`	⬆️
airflow/contrib/operators/gcs_to_bq.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_get_data.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_to_bigquery.py	`0% <0%> (ø)`	⬆️
...ontrib/operators/bigquery_table_delete_operator.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_to_gcs.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_operator.py	`95.93% <100%> (+2.36%)`	⬆️
airflow/contrib/hooks/bigquery_hook.py	`60.69% <47.29%> (+2.62%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f07f3a8...d903f24. Read the comment docs.

jmcarp · 2019-02-04T17:07:00Z

@kaxil @potiuk: tests are passing, so this is ready for review when you have time. A few differences to point out:

The current python bigquery client library implements the dbapi connection and cursor interfaces, so we can drop our custom implementations
The current client also allows for polling a job by calling its result method, so we can also drop our custom polling logic
Because the current client includes classes for most job options that include basic validation, we're also able to drop some custom validation
Extra methods on the bigquery cursor have been moved to the hook

jmcarp · 2019-02-08T04:52:43Z

Ping @Fokko @feng-tao @mik-laj (not sure who knows this code best). I'm hoping to add some features after this is ready, so would be great to get feedback when you all have time.

kaxil · 2019-02-12T23:30:42Z

 sql,
 bigquery_conn_id='bigquery_default',
- use_legacy_sql=True,
+ use_legacy_sql=False,


We shouldn't change this behaviour

kaxil · 2019-02-12T23:32:18Z

 allow_jagged_rows=self.allow_jagged_rows,
-src_fmt_configs=self.src_fmt_configs,
- labels=self.labels
+external_config_options=self.external_config_options,


This will have to be for 2.0 as it will break things. Also, it needs to be backward-compatible to make updation smooth from 1.X to 2.0

Haven't yet reviewed the entire PR but schemed through it. Will try to find some time to look at it more thoroughly

Fokko · 2019-02-27T11:55:32Z

I'm seeing some good things in this PR, and it looks like it will simplify the logic. Any plans of moving this forward @jmcarp ?

potiuk · 2019-02-05T13:02:51Z

+ :type project_id: str
 """
- service = self.get_service()
+ project_id = project_id if project_id is not None else self.project_id


I think you could use fallback_to_default_project_id decorator. It has additional logic to raise the exception if none of the project_id s is specified. and you could remove this if altogether then. It forces to use keyword parameters though.

potiuk · 2019-03-01T17:24:19Z

- project_id=project,
- use_legacy_sql=self.use_legacy_sql,
+ def get_client(self, project_id=None):
+ project_id = project_id if project_id is not None else self.project_id


Same as below - fallback_to_default_project_id decorator is nicer way I think

potiuk · 2019-03-01T17:29:39Z

-}
-})
+ if external_config_options is not None:
+if not isinstance(external_config_options, type(external_config.options)):


I think you should use dict explicitly here. What if external_config.options are None (seem to be default).

potiuk · 2019-03-01T17:32:05Z

 https://cloud.google.com/bigquery/docs/locations#specifying_your_location
 :type location: str
 """
+ project_id = project_id if project_id is not None else self.project_id


Again - I think using fallback decorator is nicer as it keeps the project_id logic in one place.

potiuk · 2019-03-01T17:33:34Z

 passed to BigQuery
 :type labels: dict
 """
+ project_id = project_id if project_id is not None else self.project_id


Same here with decorator.

potiuk · 2019-03-01T17:34:09Z

 time_partitioning. The order of columns given determines the sort order.
 :type cluster_fields: list of str
 """
+ project_id = project_id if project_id is not None else self.project_id


potiuk · 2019-03-01T17:35:49Z

 """
- # check to see if the table exists
- table_id = table_resource['tableReference']['tableId']
 project_id = project_id if project_id is not None else self.project_id


And here :)

potiuk · 2019-03-01T17:36:34Z

 even if any insertion errors occur.
 :type fail_on_error: bool
 """
+ project_id = project_id if project_id is not None else self.project_id


here too :)

bjoernpollex-sc · 2019-03-11T08:35:21Z

 private_key=private_key)

- def table_exists(self, project_id, dataset_id, table_id):
+ def table_exists(self, dataset_id, table_id, project_id=None):


Hi, thanks for putting this together, this is great stuff!
Regarding these convenience methods, I'd argue that it would be better if the functionality would be implemented on the connection object returned by get_conn, and the here we just delegate, like this:
deftable_exists(self, dataset_id, table_id, project_id=None): returnself.get_conn().table_exists(dataset_id, table_id, project_id)
The reason is that sometimes it is very useful to have explicit control over when and where connections get created, for example when using multi-threading for I/O optimization. We've also had code that spuriously crashed because it was creating too many new connections (implicitly in such convenience methods), and at some point the authentication requests hit a rate-limit.

potiuk · 2019-04-20T15:29:23Z

Hey @jmcarp -> are you still working on this? As I am now committer, I am happy to review that one as well as soon as it gets rebased (if it's still something you want to add).

kaxil · 2019-04-30T21:35:07Z

@jmcarp Are you still working on this?

mik-laj · 2019-07-27T13:34:47Z

@jmcarp Can I help you with this? This looks very interesting to me.

mik-laj · 2019-09-05T13:59:23Z

Hi.

I made a change in the base class - GoogleCloudBaseHook. Your PR may need to be changed. Could you do rebase?

Cheers

Refenence:
#5907

potiuk · 2019-09-06T13:39:00Z

@jmcarp - is it possible to rebase/complete the work on this one ? It blocks us from moving operators/hook from contrib to core - after we move the operators/hooks it will be much more difficult to merge.

potiuk · 2019-09-21T19:04:46Z

Hey @jmcarp - are you still doing it? Or should we close that one?

stale · 2019-11-20T16:52:27Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

jmcarp · 2019-01-28T04:08:01Z

Make sure you have checked all steps below.

Jira

My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-1894
- In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.

Description

Here are some details about my PR, including screenshots of any UI changes:

Tests

My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters (not including Jira issue reference)
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"

Documentation

In case of new functionality, my PR adds documentation that describes how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.
- All the public functions and the classes in the PR contain docstrings that explain what it does

Code Quality

Passes flake8

codecov-io · 2019-01-31T02:33:31Z

Codecov Report

Merging #4607 into master will increase coverage by 0.23%.
The diff coverage is 47.05%.

@@ Coverage Diff @@## master #4607 +/- ##
==========================================
+ Coverage 74.3% 74.53% +0.23% 
==========================================
Files 426 426 Lines 27867 27577 -290 ==========================================
- Hits 20706 20554 -152 + Misses 7161 7023 -138

Impacted Files	Coverage Δ
...rflow/contrib/operators/bigquery_check_operator.py	`0% <ø> (ø)`	⬆️
airflow/contrib/operators/gcs_to_bq.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_get_data.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_to_bigquery.py	`0% <0%> (ø)`	⬆️
...ontrib/operators/bigquery_table_delete_operator.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_to_gcs.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_operator.py	`95.93% <100%> (+2.36%)`	⬆️
airflow/contrib/hooks/bigquery_hook.py	`60.69% <47.29%> (+2.62%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f07f3a8...d903f24. Read the comment docs.

jmcarp · 2019-02-04T17:07:00Z

@kaxil @potiuk: tests are passing, so this is ready for review when you have time. A few differences to point out:

The current python bigquery client library implements the dbapi connection and cursor interfaces, so we can drop our custom implementations
The current client also allows for polling a job by calling its result method, so we can also drop our custom polling logic
Because the current client includes classes for most job options that include basic validation, we're also able to drop some custom validation
Extra methods on the bigquery cursor have been moved to the hook

jmcarp · 2019-02-08T04:52:43Z

Ping @Fokko @feng-tao @mik-laj (not sure who knows this code best). I'm hoping to add some features after this is ready, so would be great to get feedback when you all have time.

kaxil · 2019-02-12T23:30:42Z

 sql,
 bigquery_conn_id='bigquery_default',
- use_legacy_sql=True,
+ use_legacy_sql=False,


We shouldn't change this behaviour

kaxil · 2019-02-12T23:32:18Z

 allow_jagged_rows=self.allow_jagged_rows,
-src_fmt_configs=self.src_fmt_configs,
- labels=self.labels
+external_config_options=self.external_config_options,


This will have to be for 2.0 as it will break things. Also, it needs to be backward-compatible to make updation smooth from 1.X to 2.0

Haven't yet reviewed the entire PR but schemed through it. Will try to find some time to look at it more thoroughly

Fokko · 2019-02-27T11:55:32Z

I'm seeing some good things in this PR, and it looks like it will simplify the logic. Any plans of moving this forward @jmcarp ?

potiuk · 2019-02-05T13:02:51Z

+ :type project_id: str
 """
- service = self.get_service()
+ project_id = project_id if project_id is not None else self.project_id


I think you could use fallback_to_default_project_id decorator. It has additional logic to raise the exception if none of the project_id s is specified. and you could remove this if altogether then. It forces to use keyword parameters though.

potiuk · 2019-03-01T17:24:19Z

- project_id=project,
- use_legacy_sql=self.use_legacy_sql,
+ def get_client(self, project_id=None):
+ project_id = project_id if project_id is not None else self.project_id


Same as below - fallback_to_default_project_id decorator is nicer way I think

potiuk · 2019-03-01T17:29:39Z

-}
-})
+ if external_config_options is not None:
+if not isinstance(external_config_options, type(external_config.options)):


I think you should use dict explicitly here. What if external_config.options are None (seem to be default).

potiuk · 2019-03-01T17:32:05Z

 https://cloud.google.com/bigquery/docs/locations#specifying_your_location
 :type location: str
 """
+ project_id = project_id if project_id is not None else self.project_id


Again - I think using fallback decorator is nicer as it keeps the project_id logic in one place.

potiuk · 2019-03-01T17:33:34Z

 passed to BigQuery
 :type labels: dict
 """
+ project_id = project_id if project_id is not None else self.project_id


Same here with decorator.

potiuk · 2019-03-01T17:34:09Z

 time_partitioning. The order of columns given determines the sort order.
 :type cluster_fields: list of str
 """
+ project_id = project_id if project_id is not None else self.project_id


potiuk · 2019-03-01T17:35:49Z

 """
- # check to see if the table exists
- table_id = table_resource['tableReference']['tableId']
 project_id = project_id if project_id is not None else self.project_id


And here :)

potiuk · 2019-03-01T17:36:34Z

 even if any insertion errors occur.
 :type fail_on_error: bool
 """
+ project_id = project_id if project_id is not None else self.project_id


here too :)

bjoernpollex-sc · 2019-03-11T08:35:21Z

 private_key=private_key)

- def table_exists(self, project_id, dataset_id, table_id):
+ def table_exists(self, dataset_id, table_id, project_id=None):


Hi, thanks for putting this together, this is great stuff!
Regarding these convenience methods, I'd argue that it would be better if the functionality would be implemented on the connection object returned by get_conn, and the here we just delegate, like this:
deftable_exists(self, dataset_id, table_id, project_id=None): returnself.get_conn().table_exists(dataset_id, table_id, project_id)
The reason is that sometimes it is very useful to have explicit control over when and where connections get created, for example when using multi-threading for I/O optimization. We've also had code that spuriously crashed because it was creating too many new connections (implicitly in such convenience methods), and at some point the authentication requests hit a rate-limit.

potiuk · 2019-04-20T15:29:23Z

Hey @jmcarp -> are you still working on this? As I am now committer, I am happy to review that one as well as soon as it gets rebased (if it's still something you want to add).

kaxil · 2019-04-30T21:35:07Z

@jmcarp Are you still working on this?

mik-laj · 2019-07-27T13:34:47Z

@jmcarp Can I help you with this? This looks very interesting to me.

mik-laj · 2019-09-05T13:59:23Z

Hi.

I made a change in the base class - GoogleCloudBaseHook. Your PR may need to be changed. Could you do rebase?

Cheers

Refenence:
#5907

potiuk · 2019-09-06T13:39:00Z

@jmcarp - is it possible to rebase/complete the work on this one ? It blocks us from moving operators/hook from contrib to core - after we move the operators/hooks it will be much more difficult to merge.

potiuk · 2019-09-21T19:04:46Z

Hey @jmcarp - are you still doing it? Or should we close that one?

stale · 2019-11-20T16:52:27Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

jmcarp · 2019-01-28T04:08:01Z

Make sure you have checked all steps below.

Jira

My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-1894
- In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.

Description

Here are some details about my PR, including screenshots of any UI changes:

Tests

My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters (not including Jira issue reference)
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"

Documentation

In case of new functionality, my PR adds documentation that describes how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.
- All the public functions and the classes in the PR contain docstrings that explain what it does

Code Quality

Passes flake8

codecov-io · 2019-01-31T02:33:31Z

Codecov Report

Merging #4607 into master will increase coverage by 0.23%.
The diff coverage is 47.05%.

@@ Coverage Diff @@## master #4607 +/- ##
==========================================
+ Coverage 74.3% 74.53% +0.23% 
==========================================
Files 426 426 Lines 27867 27577 -290 ==========================================
- Hits 20706 20554 -152 + Misses 7161 7023 -138

Impacted Files	Coverage Δ
...rflow/contrib/operators/bigquery_check_operator.py	`0% <ø> (ø)`	⬆️
airflow/contrib/operators/gcs_to_bq.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_get_data.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_to_bigquery.py	`0% <0%> (ø)`	⬆️
...ontrib/operators/bigquery_table_delete_operator.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_to_gcs.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_operator.py	`95.93% <100%> (+2.36%)`	⬆️
airflow/contrib/hooks/bigquery_hook.py	`60.69% <47.29%> (+2.62%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f07f3a8...d903f24. Read the comment docs.

jmcarp · 2019-02-04T17:07:00Z

@kaxil @potiuk: tests are passing, so this is ready for review when you have time. A few differences to point out:

The current python bigquery client library implements the dbapi connection and cursor interfaces, so we can drop our custom implementations
The current client also allows for polling a job by calling its result method, so we can also drop our custom polling logic
Because the current client includes classes for most job options that include basic validation, we're also able to drop some custom validation
Extra methods on the bigquery cursor have been moved to the hook

jmcarp · 2019-02-08T04:52:43Z

Ping @Fokko @feng-tao @mik-laj (not sure who knows this code best). I'm hoping to add some features after this is ready, so would be great to get feedback when you all have time.

kaxil · 2019-02-12T23:30:42Z

 sql,
 bigquery_conn_id='bigquery_default',
- use_legacy_sql=True,
+ use_legacy_sql=False,


We shouldn't change this behaviour

kaxil · 2019-02-12T23:32:18Z

 allow_jagged_rows=self.allow_jagged_rows,
-src_fmt_configs=self.src_fmt_configs,
- labels=self.labels
+external_config_options=self.external_config_options,


This will have to be for 2.0 as it will break things. Also, it needs to be backward-compatible to make updation smooth from 1.X to 2.0

Haven't yet reviewed the entire PR but schemed through it. Will try to find some time to look at it more thoroughly

Fokko · 2019-02-27T11:55:32Z

I'm seeing some good things in this PR, and it looks like it will simplify the logic. Any plans of moving this forward @jmcarp ?

potiuk · 2019-02-05T13:02:51Z

+ :type project_id: str
 """
- service = self.get_service()
+ project_id = project_id if project_id is not None else self.project_id


I think you could use fallback_to_default_project_id decorator. It has additional logic to raise the exception if none of the project_id s is specified. and you could remove this if altogether then. It forces to use keyword parameters though.

potiuk · 2019-03-01T17:24:19Z

- project_id=project,
- use_legacy_sql=self.use_legacy_sql,
+ def get_client(self, project_id=None):
+ project_id = project_id if project_id is not None else self.project_id


Same as below - fallback_to_default_project_id decorator is nicer way I think

potiuk · 2019-03-01T17:29:39Z

-}
-})
+ if external_config_options is not None:
+if not isinstance(external_config_options, type(external_config.options)):


I think you should use dict explicitly here. What if external_config.options are None (seem to be default).

potiuk · 2019-03-01T17:32:05Z

 https://cloud.google.com/bigquery/docs/locations#specifying_your_location
 :type location: str
 """
+ project_id = project_id if project_id is not None else self.project_id


Again - I think using fallback decorator is nicer as it keeps the project_id logic in one place.

potiuk · 2019-03-01T17:33:34Z

 passed to BigQuery
 :type labels: dict
 """
+ project_id = project_id if project_id is not None else self.project_id


Same here with decorator.

potiuk · 2019-03-01T17:34:09Z

 time_partitioning. The order of columns given determines the sort order.
 :type cluster_fields: list of str
 """
+ project_id = project_id if project_id is not None else self.project_id


potiuk · 2019-03-01T17:35:49Z

 """
- # check to see if the table exists
- table_id = table_resource['tableReference']['tableId']
 project_id = project_id if project_id is not None else self.project_id


And here :)

potiuk · 2019-03-01T17:36:34Z

 even if any insertion errors occur.
 :type fail_on_error: bool
 """
+ project_id = project_id if project_id is not None else self.project_id


here too :)

bjoernpollex-sc · 2019-03-11T08:35:21Z

 private_key=private_key)

- def table_exists(self, project_id, dataset_id, table_id):
+ def table_exists(self, dataset_id, table_id, project_id=None):


Hi, thanks for putting this together, this is great stuff!
Regarding these convenience methods, I'd argue that it would be better if the functionality would be implemented on the connection object returned by get_conn, and the here we just delegate, like this:
deftable_exists(self, dataset_id, table_id, project_id=None): returnself.get_conn().table_exists(dataset_id, table_id, project_id)
The reason is that sometimes it is very useful to have explicit control over when and where connections get created, for example when using multi-threading for I/O optimization. We've also had code that spuriously crashed because it was creating too many new connections (implicitly in such convenience methods), and at some point the authentication requests hit a rate-limit.

potiuk · 2019-04-20T15:29:23Z

Hey @jmcarp -> are you still working on this? As I am now committer, I am happy to review that one as well as soon as it gets rebased (if it's still something you want to add).

kaxil · 2019-04-30T21:35:07Z

@jmcarp Are you still working on this?

mik-laj · 2019-07-27T13:34:47Z

@jmcarp Can I help you with this? This looks very interesting to me.

mik-laj · 2019-09-05T13:59:23Z

Hi.

I made a change in the base class - GoogleCloudBaseHook. Your PR may need to be changed. Could you do rebase?

Cheers

Refenence:
#5907

potiuk · 2019-09-06T13:39:00Z

@jmcarp - is it possible to rebase/complete the work on this one ? It blocks us from moving operators/hook from contrib to core - after we move the operators/hooks it will be much more difficult to merge.

potiuk · 2019-09-21T19:04:46Z

Hey @jmcarp - are you still doing it? Or should we close that one?

stale · 2019-11-20T16:52:27Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

jmcarp · 2019-01-28T04:08:01Z

Make sure you have checked all steps below.

Jira

My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-1894
- In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.

Description

Here are some details about my PR, including screenshots of any UI changes:

Tests

My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters (not including Jira issue reference)
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"

Documentation

In case of new functionality, my PR adds documentation that describes how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.
- All the public functions and the classes in the PR contain docstrings that explain what it does

Code Quality

Passes flake8

codecov-io · 2019-01-31T02:33:31Z

Codecov Report

Merging #4607 into master will increase coverage by 0.23%.
The diff coverage is 47.05%.

@@ Coverage Diff @@## master #4607 +/- ##
==========================================
+ Coverage 74.3% 74.53% +0.23% 
==========================================
Files 426 426 Lines 27867 27577 -290 ==========================================
- Hits 20706 20554 -152 + Misses 7161 7023 -138

Impacted Files	Coverage Δ
...rflow/contrib/operators/bigquery_check_operator.py	`0% <ø> (ø)`	⬆️
airflow/contrib/operators/gcs_to_bq.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_get_data.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_to_bigquery.py	`0% <0%> (ø)`	⬆️
...ontrib/operators/bigquery_table_delete_operator.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_to_gcs.py	`0% <0%> (ø)`	⬆️
airflow/contrib/operators/bigquery_operator.py	`95.93% <100%> (+2.36%)`	⬆️
airflow/contrib/hooks/bigquery_hook.py	`60.69% <47.29%> (+2.62%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f07f3a8...d903f24. Read the comment docs.

jmcarp · 2019-02-04T17:07:00Z

@kaxil @potiuk: tests are passing, so this is ready for review when you have time. A few differences to point out:

The current python bigquery client library implements the dbapi connection and cursor interfaces, so we can drop our custom implementations
The current client also allows for polling a job by calling its result method, so we can also drop our custom polling logic
Because the current client includes classes for most job options that include basic validation, we're also able to drop some custom validation
Extra methods on the bigquery cursor have been moved to the hook

jmcarp · 2019-02-08T04:52:43Z

Ping @Fokko @feng-tao @mik-laj (not sure who knows this code best). I'm hoping to add some features after this is ready, so would be great to get feedback when you all have time.

kaxil · 2019-02-12T23:30:42Z

 sql,
 bigquery_conn_id='bigquery_default',
- use_legacy_sql=True,
+ use_legacy_sql=False,


We shouldn't change this behaviour

kaxil · 2019-02-12T23:32:18Z

 allow_jagged_rows=self.allow_jagged_rows,
-src_fmt_configs=self.src_fmt_configs,
- labels=self.labels
+external_config_options=self.external_config_options,


This will have to be for 2.0 as it will break things. Also, it needs to be backward-compatible to make updation smooth from 1.X to 2.0

Haven't yet reviewed the entire PR but schemed through it. Will try to find some time to look at it more thoroughly

Fokko · 2019-02-27T11:55:32Z

I'm seeing some good things in this PR, and it looks like it will simplify the logic. Any plans of moving this forward @jmcarp ?

potiuk · 2019-02-05T13:02:51Z

+ :type project_id: str
 """
- service = self.get_service()
+ project_id = project_id if project_id is not None else self.project_id


I think you could use fallback_to_default_project_id decorator. It has additional logic to raise the exception if none of the project_id s is specified. and you could remove this if altogether then. It forces to use keyword parameters though.

potiuk · 2019-03-01T17:24:19Z

- project_id=project,
- use_legacy_sql=self.use_legacy_sql,
+ def get_client(self, project_id=None):
+ project_id = project_id if project_id is not None else self.project_id


Same as below - fallback_to_default_project_id decorator is nicer way I think

potiuk · 2019-03-01T17:29:39Z

-}
-})
+ if external_config_options is not None:
+if not isinstance(external_config_options, type(external_config.options)):


I think you should use dict explicitly here. What if external_config.options are None (seem to be default).

potiuk · 2019-03-01T17:32:05Z

 https://cloud.google.com/bigquery/docs/locations#specifying_your_location
 :type location: str
 """
+ project_id = project_id if project_id is not None else self.project_id


Again - I think using fallback decorator is nicer as it keeps the project_id logic in one place.

potiuk · 2019-03-01T17:33:34Z

 passed to BigQuery
 :type labels: dict
 """
+ project_id = project_id if project_id is not None else self.project_id


Same here with decorator.

potiuk · 2019-03-01T17:34:09Z

 time_partitioning. The order of columns given determines the sort order.
 :type cluster_fields: list of str
 """
+ project_id = project_id if project_id is not None else self.project_id


potiuk · 2019-03-01T17:35:49Z

 """
- # check to see if the table exists
- table_id = table_resource['tableReference']['tableId']
 project_id = project_id if project_id is not None else self.project_id


And here :)

potiuk · 2019-03-01T17:36:34Z

 even if any insertion errors occur.
 :type fail_on_error: bool
 """
+ project_id = project_id if project_id is not None else self.project_id


here too :)

bjoernpollex-sc · 2019-03-11T08:35:21Z

 private_key=private_key)

- def table_exists(self, project_id, dataset_id, table_id):
+ def table_exists(self, dataset_id, table_id, project_id=None):


Hi, thanks for putting this together, this is great stuff!
Regarding these convenience methods, I'd argue that it would be better if the functionality would be implemented on the connection object returned by get_conn, and the here we just delegate, like this:
deftable_exists(self, dataset_id, table_id, project_id=None): returnself.get_conn().table_exists(dataset_id, table_id, project_id)
The reason is that sometimes it is very useful to have explicit control over when and where connections get created, for example when using multi-threading for I/O optimization. We've also had code that spuriously crashed because it was creating too many new connections (implicitly in such convenience methods), and at some point the authentication requests hit a rate-limit.

potiuk · 2019-04-20T15:29:23Z

Hey @jmcarp -> are you still working on this? As I am now committer, I am happy to review that one as well as soon as it gets rebased (if it's still something you want to add).

kaxil · 2019-04-30T21:35:07Z

@jmcarp Are you still working on this?

mik-laj · 2019-07-27T13:34:47Z

@jmcarp Can I help you with this? This looks very interesting to me.

mik-laj · 2019-09-05T13:59:23Z

Hi.

I made a change in the base class - GoogleCloudBaseHook. Your PR may need to be changed. Could you do rebase?

Cheers

Refenence:
#5907

potiuk · 2019-09-06T13:39:00Z

@jmcarp - is it possible to rebase/complete the work on this one ? It blocks us from moving operators/hook from contrib to core - after we move the operators/hooks it will be much more difficult to merge.

potiuk · 2019-09-21T19:04:46Z

Hey @jmcarp - are you still doing it? Or should we close that one?

stale · 2019-11-20T16:52:27Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

jmcarp force-pushed the google-cloud-bigquery branch 4 times, most recently from 94a6905 to d131b00Compare January 30, 2019 04:25

jmcarp changed the title ~~[AIRFLOW-3776] Google cloud bigquery~~[AIRFLOW-1894] Google cloud bigqueryJan 30, 2019

jmcarp force-pushed the google-cloud-bigquery branch 2 times, most recently from 8bd9564 to dc502cbCompare January 30, 2019 05:03

kaxil changed the title ~~[AIRFLOW-1894] Google cloud bigquery~~WIP: [AIRFLOW-1894] Google cloud bigqueryJan 30, 2019

jmcarp force-pushed the google-cloud-bigquery branch 2 times, most recently from 89b048d to 292d467Compare January 31, 2019 01:43

jmcarp force-pushed the google-cloud-bigquery branch 5 times, most recently from 7562a62 to 8698ae7Compare February 2, 2019 02:13

[AIRFLOW-1894] Use current bigquery client library.
d903f24

jmcarp force-pushed the google-cloud-bigquery branch from 8698ae7 to d903f24Compare February 2, 2019 02:18

kaxil reviewed Feb 12, 2019
View reviewed changes

potiuk reviewed Mar 1, 2019
View reviewed changes

bjoernpollex-sc reviewed Mar 11, 2019
View reviewed changes

mik-laj added the provider:google Google (including GCP) related issues label Jul 22, 2019

kaxil force-pushed the master branch from 558f9f1 to d719e1fCompare October 2, 2019 15:31

staleBot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Nov 20, 2019

staleBot closed this Nov 27, 2019