[AIRFLOW-4935] Add method in the bigquery hook to list tables in a dataset - #5566

Merged
mik-laj merged 1 commit into
apache:masterfrom
benjamingrenier:AIRFLOW-4935
Dec 8, 2019
Merged

[AIRFLOW-4935] Add method in the bigquery hook to list tables in a dataset#5566
mik-laj merged 1 commit into
apache:masterfrom
benjamingrenier:AIRFLOW-4935

Conversation

@benjamingrenier

@benjamingrenierbenjamingrenier commented Jul 11, 2019

Copy link
Copy Markdown
Contributor

Method get_dataset_tables_list added in the hook, permits to use a table prefix too.

@ryanyuan

ryanyuan commented Jul 12, 2019

Copy link
Copy Markdown
Contributor

Could you write some unit tests for it?

ryanyuan
ryanyuan reviewed Jul 12, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py Outdated
Comment threadairflow/contrib/hooks/bigquery_hook.py Outdated
ryanyuan
ryanyuan reviewed Jul 12, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py Outdated
@mik-lajmik-laj added the provider:google Google (including GCP) related issues label Jul 22, 2019
@zzlbuaa

zzlbuaa commented Jul 24, 2019

Copy link
Copy Markdown

Hi, this hook method is a useful feature for our use case. I'm wondering are you still working on this PR?
cc @criccomini@whynick1

@benjamingrenier

benjamingrenier commented Jul 24, 2019 via email

Copy link
Copy Markdown
ContributorAuthor

mik-laj
mik-laj reviewed Jul 27, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py Outdated
@zzlbuaa

zzlbuaa commented Aug 5, 2019

Copy link
Copy Markdown

@benjamingrenier any updates on this?

@benjamingrenier

benjamingrenier commented Aug 5, 2019 via email

Copy link
Copy Markdown
ContributorAuthor

@mik-laj

mik-laj commented Aug 12, 2019

Copy link
Copy Markdown
Member

Are you planning to continue working on this change? This week, I would like to deal with PR reviews related to GCP. I would be happy if you would respond to all comments

@benjamingrenier

benjamingrenier commented Aug 12, 2019•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Yes I will work on it this week. I try to resolve my docker errors to run unit-tests locally.

@mik-laj

mik-laj commented Aug 12, 2019

Copy link
Copy Markdown
Member

Recently introduced a new development environment. have you tried them? we plan to create a new extension to improve GCP development.

mik-laj
mik-laj reviewed Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py Outdated
mik-laj
mik-laj reviewed Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py Outdated
mik-laj
mik-laj reviewed Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py Outdated
mik-laj
mik-laj reviewed Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py Outdated
mik-laj
mik-laj reviewed Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py Outdated
mik-laj
mik-laj reviewed Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py Outdated
mik-laj
mik-laj reviewed Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py Outdated
mik-laj
mik-laj reviewed Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py Outdated
mik-laj
mik-laj reviewed Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py Outdated
mik-laj
mik-laj reviewed Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py Outdated
mik-laj
mik-laj reviewed Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py Outdated
mik-laj
mik-laj reviewed Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py Outdated
mik-laj
mik-laj reviewed Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py Outdated
mik-laj
mik-laj reviewed Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py Outdated
mik-laj
mik-laj reviewed Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py Outdated
@mik-laj

mik-laj commented Aug 14, 2019

Copy link
Copy Markdown
Member

Can you do rebase, please?

@benjamingrenier

benjamingrenier commented Aug 20, 2019

Copy link
Copy Markdown
ContributorAuthor

Can you do rebase, please?

It's done. Ok for you ?

@mik-laj

mik-laj commented Aug 20, 2019

Copy link
Copy Markdown
Member

Something is wrong with this PR. Too many commits.

I'm on vacation now🇫🇷. I'll look at this on Friday.

@benjamingrenier

benjamingrenier commented Aug 20, 2019 via email

Copy link
Copy Markdown
ContributorAuthor

@ryanyuan

ryanyuan commented Aug 20, 2019

Copy link
Copy Markdown
Contributor

@benjamingrenier After rebasing, squash your commits and then do a force push.

@mik-laj

mik-laj commented Aug 20, 2019

Copy link
Copy Markdown
Member

I suspect that the references to the main branch are broken and the history will have to be rewritten (git reset --hard, git cherry-pick).

@benjamingrenier
benjamingrenierforce-pushed the AIRFLOW-4935 branch 6 times, most recently from acd2c9f to af4e3d4CompareAugust 20, 2019 13:04
@benjamingrenier

benjamingrenier commented Aug 20, 2019

Copy link
Copy Markdown
ContributorAuthor

I think it's all right.

mik-laj
mik-laj reviewed Aug 22, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py Outdated

@mik-lajmik-lajAug 22, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
self.log.info(dataset_tables_list)

I haven't seen it before, but it seems to me that there will be too much information in the logs.

@benjamingrenierbenjamingrenierSep 2, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will fix it soon. I prepare a commit with more unit tests too.

@potiukpotiukSep 6, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please remove the log before merge?

@mik-laj

mik-laj commented Sep 5, 2019

Copy link
Copy Markdown
Member

Hi.

I made a change in the base class - GoogleCloudBaseHook. Your PR may need to be changed. Could you do rebase?

Cheers

Reference:
#5907

potiuk
potiuk requested changes Sep 6, 2019
View reviewed changes

@potiukpotiuk left a comment •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@benjamingrenier We are in the middle of moving hooks and operators from contrib to core. Is it possible to rebase (add more unittestss?) quickly so that we can merge the change before we move it?

Comment threadairflow/contrib/hooks/bigquery_hook.py Outdated

@potiukpotiukSep 6, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please remove the log before merge?

@benjamingrenier

benjamingrenier commented Sep 6, 2019

Copy link
Copy Markdown
ContributorAuthor

@benjamingrenier We are in the middle of moving hooks and operators from contrib to core. Is it possible to rebase (add more unittestss?) quickly so that we can merge the change before we move it?

Rebase done and unittests added

@potiuk

potiuk commented Sep 21, 2019

Copy link
Copy Markdown
Member

Hey @benjamingrenier -> I must ask for another rebase: (.

@benjamingrenier

benjamingrenier commented Sep 21, 2019 via email

Copy link
Copy Markdown
ContributorAuthor

@codecov-io

codecov-io commented Sep 22, 2019•
edited
Loading

Copy link
Copy Markdown

Codecov Report

Merging #5566 into master will increase coverage by 0.13%.
The diff coverage is 94.44%.

Impacted file tree graph

@@ Coverage Diff @@## master #5566 +/- ##
==========================================
+ Coverage 79.93% 80.06% +0.13% 
==========================================
Files 608 608 Lines 35030 35048 +18 ==========================================
+ Hits 28000 28062 +62 + Misses 7030 6986 -44
Impacted FilesCoverage Δ
airflow/gcp/hooks/bigquery.py70.85% <94.44%> (+0.64%)⬆️
airflow/models/taskinstance.py93.72% <0%> (+0.5%)⬆️
airflow/jobs/scheduler_job.py74.28% <0%> (+0.9%)⬆️
airflow/utils/dag_processing.py58.98% <0%> (+2.54%)⬆️
airflow/executors/__init__.py67.34% <0%> (+4.08%)⬆️
airflow/utils/sqlalchemy.py93.22% <0%> (+6.77%)⬆️
airflow/utils/log/colored_log.py93.18% <0%> (+11.36%)⬆️
airflow/executors/sequential_executor.py100% <0%> (+52.38%)⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2d19a41...e1d8ecd. Read the comment docs.

@whynick1

whynick1 commented Nov 15, 2019

Copy link
Copy Markdown
Contributor

I am also interested in this PR, any update? 😄
@potiuk@mik-laj@ryanyuan

@benjamingrenier

benjamingrenier commented Nov 29, 2019

Copy link
Copy Markdown
ContributorAuthor

Someone can do the review ?

@mik-laj

mik-laj commented Nov 30, 2019

Copy link
Copy Markdown
Member

I added this PR to the queue next week. I will try to review it as soon as possible.

@mik-lajmik-laj mentioned this pull request Nov 30, 2019
Closed
@mik-laj
mik-laj merged commit 4561978 into apache:masterDec 8, 2019
galuszkak pushed a commit to FlyrInc/apache-airflow that referenced this pull request Mar 5, 2020
@benjamingrenier@galuszkak
[AIRFLOW-4935] Add function get_dataset_tables_list to BigQueryHook (a…
62878fb
…pache#5566)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mik-lajmik-lajmik-laj approved these changes
@potiukpotiukAwaiting requested review from potiuk
+1 more reviewer
@ryanyuanryanyuanryanyuan left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

provider:googleGoogle (including GCP) related issues

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@benjamingrenier@ryanyuan@zzlbuaa@mik-laj@potiuk@codecov-io@whynick1
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
[AIRFLOW-4935] Add method in the bigquery hook to list tables in a dataset - #5566
Merged
mik-laj merged 1 commit into
apache:masterfrom 
benjamingrenier:AIRFLOW-4935
Dec 8, 2019
Merged
[AIRFLOW-4935] Add method in the bigquery hook to list tables in a dataset#5566
mik-laj merged 1 commit into
apache:masterfrom 
benjamingrenier:AIRFLOW-4935
Conversation
@benjamingrenier
@benjamingrenierbenjamingrenier
commented
Jul 11, 2019

Copy link


Copy Markdown

Contributor
Method get_dataset_tables_list added in the hook, permits to use a table prefix too.
@ryanyuan
ryanyuan
commented
Jul 12, 2019

Copy link


Copy Markdown

Contributor
Could you write some unit tests for it?
ryanyuan
ryanyuan
reviewed
Jul 12, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

ryanyuan
ryanyuan
reviewed
Jul 12, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@mik-lajmik-laj
added
the 
provider:google
Google (including GCP) related issues
label
Jul 22, 2019
@zzlbuaa
zzlbuaa
commented
Jul 24, 2019

Copy link


Copy Markdown

Hi, this hook method is a useful feature for our use case. I'm wondering are you still working on this PR?

cc @criccomini@whynick1
@benjamingrenier
benjamingrenier
commented
Jul 24, 2019
via email 

Copy link


Copy Markdown

ContributorAuthor
mik-laj
mik-laj
reviewed
Jul 27, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@zzlbuaa
zzlbuaa
commented
Aug 5, 2019

Copy link


Copy Markdown

@benjamingrenier any updates on this?
@benjamingrenier
benjamingrenier
commented
Aug 5, 2019
via email 

Copy link


Copy Markdown

ContributorAuthor
@mik-laj
mik-laj
commented
Aug 12, 2019

Copy link


Copy Markdown

Member
Are you planning to continue working on this change? This week, I would like to deal with PR reviews related to GCP. I would be happy if you would respond to all comments
@benjamingrenier
benjamingrenier
commented
Aug 12, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Yes I will work on it this week. I try to resolve my docker errors to run unit-tests locally.
@mik-laj
mik-laj
commented
Aug 12, 2019

Copy link


Copy Markdown

Member
Recently introduced a new development environment. have you tried them? we plan to create a new extension to improve GCP development.
mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@mik-laj
mik-laj
commented
Aug 14, 2019

Copy link


Copy Markdown

Member
Can you do rebase, please?
@benjamingrenier
benjamingrenier
commented
Aug 20, 2019

Copy link


Copy Markdown

ContributorAuthor
Can you do rebase, please?
It's done. Ok for you ?
@mik-laj
mik-laj
commented
Aug 20, 2019

Copy link


Copy Markdown

Member
Something is wrong with this PR. Too many commits.
I'm on vacation now🇫🇷. I'll look at this on Friday.
@benjamingrenier
benjamingrenier
commented
Aug 20, 2019
via email 

Copy link


Copy Markdown

ContributorAuthor
@ryanyuan
ryanyuan
commented
Aug 20, 2019

Copy link


Copy Markdown

Contributor
@benjamingrenier After rebasing, squash your commits and then do a force push.
@mik-laj
mik-laj
commented
Aug 20, 2019

Copy link


Copy Markdown

Member
I suspect that the references to the main branch are broken and the history will have to be rewritten (git reset --hard, git cherry-pick).
@benjamingrenier
benjamingrenierforce-pushed
the
AIRFLOW-4935
branch
6 times, most recently
from
acd2c9f to
af4e3d4CompareAugust 20, 2019 13:04
@benjamingrenier
benjamingrenier
commented
Aug 20, 2019

Copy link


Copy Markdown

ContributorAuthor
I think it's all right.
mik-laj
mik-laj
reviewed
Aug 22, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@mik-lajmik-lajAug 22, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


Suggested change

 self.log.info(dataset_tables_list)
I haven't seen it before, but it seems to me that there will be too much information in the logs.
@benjamingrenierbenjamingrenierSep 2, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will fix it soon. I prepare a commit with more unit tests too.
@potiukpotiukSep 6, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please remove the log before merge?
@mik-laj
mik-laj
commented
Sep 5, 2019

Copy link


Copy Markdown

Member
Hi.
I made a change in the base class - GoogleCloudBaseHook. Your PR may need to be changed. Could you do rebase?
Cheers
Reference:
#5907
potiuk
potiuk
requested changes
Sep 6, 2019
View reviewed changes
@potiukpotiuk
left a comment
•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@benjamingrenier We are in the middle of moving hooks and operators from contrib to core. Is it possible to rebase (add more unittestss?) quickly so that we can merge the change before we move it?
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@potiukpotiukSep 6, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please remove the log before merge?
@benjamingrenier
benjamingrenier
commented
Sep 6, 2019

Copy link


Copy Markdown

ContributorAuthor
@benjamingrenier We are in the middle of moving hooks and operators from contrib to core. Is it possible to rebase (add more unittestss?) quickly so that we can merge the change before we move it?
Rebase done and unittests added
@potiuk
potiuk
commented
Sep 21, 2019

Copy link


Copy Markdown

Member
Hey @benjamingrenier -> I must ask for another rebase: (.
@benjamingrenier
benjamingrenier
commented
Sep 21, 2019
via email 

Copy link


Copy Markdown

ContributorAuthor
@codecov-io
codecov-io
commented
Sep 22, 2019•

edited

Loading

Copy link


Copy Markdown

Codecov Report
Merging #5566 into master will increase coverage by 0.13%.

The diff coverage is 94.44%.
Impacted file tree graph
@@ Coverage Diff @@## master #5566 +/- ##
==========================================
+ Coverage 79.93% 80.06% +0.13% 
==========================================
Files 608 608 Lines 35030 35048 +18 ==========================================
+ Hits 28000 28062 +62 + Misses 7030 6986 -44
Impacted FilesCoverage Δ
airflow/gcp/hooks/bigquery.py70.85% <94.44%> (+0.64%)⬆️
airflow/models/taskinstance.py93.72% <0%> (+0.5%)⬆️
airflow/jobs/scheduler_job.py74.28% <0%> (+0.9%)⬆️
airflow/utils/dag_processing.py58.98% <0%> (+2.54%)⬆️
airflow/executors/__init__.py67.34% <0%> (+4.08%)⬆️
airflow/utils/sqlalchemy.py93.22% <0%> (+6.77%)⬆️
airflow/utils/log/colored_log.py93.18% <0%> (+11.36%)⬆️
airflow/executors/sequential_executor.py100% <0%> (+52.38%)⬆️
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data

Powered by Codecov. Last update 2d19a41...e1d8ecd. Read the comment docs.
@whynick1
whynick1
commented
Nov 15, 2019

Copy link


Copy Markdown

Contributor
I am also interested in this PR, any update? 😄
@potiuk@mik-laj@ryanyuan
@benjamingrenier
benjamingrenier
commented
Nov 29, 2019

Copy link


Copy Markdown

ContributorAuthor
Someone can do the review ?
@mik-laj
mik-laj
commented
Nov 30, 2019

Copy link


Copy Markdown

Member
I added this PR to the queue next week. I will try to review it as soon as possible.
@mik-lajmik-laj
mentioned this pull request
Nov 30, 2019

Closed

@mik-laj
mik-laj
merged commit 4561978
into
apache:masterDec 8, 2019
galuszkak pushed a commit
to FlyrInc/apache-airflow
that referenced
this pull request
Mar 5, 2020
@benjamingrenier@galuszkak
[AIRFLOW-4935] Add function get_dataset_tables_list to BigQueryHook (a…
62878fb
…pache#5566)
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@mik-lajmik-lajmik-laj approved these changes
@potiukpotiukAwaiting requested review from potiuk

+1 more reviewer

@ryanyuanryanyuanryanyuan left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

provider:googleGoogle (including GCP) related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

7 participants

@benjamingrenier@ryanyuan@zzlbuaa@mik-laj@potiuk@codecov-io@whynick1

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
[AIRFLOW-4935] Add method in the bigquery hook to list tables in a dataset - #5566
Merged
mik-laj merged 1 commit into
apache:masterfrom 
benjamingrenier:AIRFLOW-4935
Dec 8, 2019
Merged
[AIRFLOW-4935] Add method in the bigquery hook to list tables in a dataset#5566
mik-laj merged 1 commit into
apache:masterfrom 
benjamingrenier:AIRFLOW-4935
Conversation
@benjamingrenier
@benjamingrenierbenjamingrenier
commented
Jul 11, 2019

Copy link


Copy Markdown

Contributor
Method get_dataset_tables_list added in the hook, permits to use a table prefix too.
@ryanyuan
ryanyuan
commented
Jul 12, 2019

Copy link


Copy Markdown

Contributor
Could you write some unit tests for it?
ryanyuan
ryanyuan
reviewed
Jul 12, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

ryanyuan
ryanyuan
reviewed
Jul 12, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@mik-lajmik-laj
added
the 
provider:google
Google (including GCP) related issues
label
Jul 22, 2019
@zzlbuaa
zzlbuaa
commented
Jul 24, 2019

Copy link


Copy Markdown

Hi, this hook method is a useful feature for our use case. I'm wondering are you still working on this PR?

cc @criccomini@whynick1
@benjamingrenier
benjamingrenier
commented
Jul 24, 2019
via email 

Copy link


Copy Markdown

ContributorAuthor
mik-laj
mik-laj
reviewed
Jul 27, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@zzlbuaa
zzlbuaa
commented
Aug 5, 2019

Copy link


Copy Markdown

@benjamingrenier any updates on this?
@benjamingrenier
benjamingrenier
commented
Aug 5, 2019
via email 

Copy link


Copy Markdown

ContributorAuthor
@mik-laj
mik-laj
commented
Aug 12, 2019

Copy link


Copy Markdown

Member
Are you planning to continue working on this change? This week, I would like to deal with PR reviews related to GCP. I would be happy if you would respond to all comments
@benjamingrenier
benjamingrenier
commented
Aug 12, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Yes I will work on it this week. I try to resolve my docker errors to run unit-tests locally.
@mik-laj
mik-laj
commented
Aug 12, 2019

Copy link


Copy Markdown

Member
Recently introduced a new development environment. have you tried them? we plan to create a new extension to improve GCP development.
mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@mik-laj
mik-laj
commented
Aug 14, 2019

Copy link


Copy Markdown

Member
Can you do rebase, please?
@benjamingrenier
benjamingrenier
commented
Aug 20, 2019

Copy link


Copy Markdown

ContributorAuthor
Can you do rebase, please?
It's done. Ok for you ?
@mik-laj
mik-laj
commented
Aug 20, 2019

Copy link


Copy Markdown

Member
Something is wrong with this PR. Too many commits.
I'm on vacation now🇫🇷. I'll look at this on Friday.
@benjamingrenier
benjamingrenier
commented
Aug 20, 2019
via email 

Copy link


Copy Markdown

ContributorAuthor
@ryanyuan
ryanyuan
commented
Aug 20, 2019

Copy link


Copy Markdown

Contributor
@benjamingrenier After rebasing, squash your commits and then do a force push.
@mik-laj
mik-laj
commented
Aug 20, 2019

Copy link


Copy Markdown

Member
I suspect that the references to the main branch are broken and the history will have to be rewritten (git reset --hard, git cherry-pick).
@benjamingrenier
benjamingrenierforce-pushed
the
AIRFLOW-4935
branch
6 times, most recently
from
acd2c9f to
af4e3d4CompareAugust 20, 2019 13:04
@benjamingrenier
benjamingrenier
commented
Aug 20, 2019

Copy link


Copy Markdown

ContributorAuthor
I think it's all right.
mik-laj
mik-laj
reviewed
Aug 22, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@mik-lajmik-lajAug 22, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


Suggested change

 self.log.info(dataset_tables_list)
I haven't seen it before, but it seems to me that there will be too much information in the logs.
@benjamingrenierbenjamingrenierSep 2, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will fix it soon. I prepare a commit with more unit tests too.
@potiukpotiukSep 6, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please remove the log before merge?
@mik-laj
mik-laj
commented
Sep 5, 2019

Copy link


Copy Markdown

Member
Hi.
I made a change in the base class - GoogleCloudBaseHook. Your PR may need to be changed. Could you do rebase?
Cheers
Reference:
#5907
potiuk
potiuk
requested changes
Sep 6, 2019
View reviewed changes
@potiukpotiuk
left a comment
•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@benjamingrenier We are in the middle of moving hooks and operators from contrib to core. Is it possible to rebase (add more unittestss?) quickly so that we can merge the change before we move it?
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@potiukpotiukSep 6, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please remove the log before merge?
@benjamingrenier
benjamingrenier
commented
Sep 6, 2019

Copy link


Copy Markdown

ContributorAuthor
@benjamingrenier We are in the middle of moving hooks and operators from contrib to core. Is it possible to rebase (add more unittestss?) quickly so that we can merge the change before we move it?
Rebase done and unittests added
@potiuk
potiuk
commented
Sep 21, 2019

Copy link


Copy Markdown

Member
Hey @benjamingrenier -> I must ask for another rebase: (.
@benjamingrenier
benjamingrenier
commented
Sep 21, 2019
via email 

Copy link


Copy Markdown

ContributorAuthor
@codecov-io
codecov-io
commented
Sep 22, 2019•

edited

Loading

Copy link


Copy Markdown

Codecov Report
Merging #5566 into master will increase coverage by 0.13%.

The diff coverage is 94.44%.
Impacted file tree graph
@@ Coverage Diff @@## master #5566 +/- ##
==========================================
+ Coverage 79.93% 80.06% +0.13% 
==========================================
Files 608 608 Lines 35030 35048 +18 ==========================================
+ Hits 28000 28062 +62 + Misses 7030 6986 -44
Impacted FilesCoverage Δ
airflow/gcp/hooks/bigquery.py70.85% <94.44%> (+0.64%)⬆️
airflow/models/taskinstance.py93.72% <0%> (+0.5%)⬆️
airflow/jobs/scheduler_job.py74.28% <0%> (+0.9%)⬆️
airflow/utils/dag_processing.py58.98% <0%> (+2.54%)⬆️
airflow/executors/__init__.py67.34% <0%> (+4.08%)⬆️
airflow/utils/sqlalchemy.py93.22% <0%> (+6.77%)⬆️
airflow/utils/log/colored_log.py93.18% <0%> (+11.36%)⬆️
airflow/executors/sequential_executor.py100% <0%> (+52.38%)⬆️
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data

Powered by Codecov. Last update 2d19a41...e1d8ecd. Read the comment docs.
@whynick1
whynick1
commented
Nov 15, 2019

Copy link


Copy Markdown

Contributor
I am also interested in this PR, any update? 😄
@potiuk@mik-laj@ryanyuan
@benjamingrenier
benjamingrenier
commented
Nov 29, 2019

Copy link


Copy Markdown

ContributorAuthor
Someone can do the review ?
@mik-laj
mik-laj
commented
Nov 30, 2019

Copy link


Copy Markdown

Member
I added this PR to the queue next week. I will try to review it as soon as possible.
@mik-lajmik-laj
mentioned this pull request
Nov 30, 2019

Closed

@mik-laj
mik-laj
merged commit 4561978
into
apache:masterDec 8, 2019
galuszkak pushed a commit
to FlyrInc/apache-airflow
that referenced
this pull request
Mar 5, 2020
@benjamingrenier@galuszkak
[AIRFLOW-4935] Add function get_dataset_tables_list to BigQueryHook (a…
62878fb
…pache#5566)
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@mik-lajmik-lajmik-laj approved these changes
@potiukpotiukAwaiting requested review from potiuk

+1 more reviewer

@ryanyuanryanyuanryanyuan left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

provider:googleGoogle (including GCP) related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

7 participants

@benjamingrenier@ryanyuan@zzlbuaa@mik-laj@potiuk@codecov-io@whynick1

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
[AIRFLOW-4935] Add method in the bigquery hook to list tables in a dataset - #5566
Merged
mik-laj merged 1 commit into
apache:masterfrom 
benjamingrenier:AIRFLOW-4935
Dec 8, 2019
Merged
[AIRFLOW-4935] Add method in the bigquery hook to list tables in a dataset#5566
mik-laj merged 1 commit into
apache:masterfrom 
benjamingrenier:AIRFLOW-4935
Conversation
@benjamingrenier
@benjamingrenierbenjamingrenier
commented
Jul 11, 2019

Copy link


Copy Markdown

Contributor
Method get_dataset_tables_list added in the hook, permits to use a table prefix too.
@ryanyuan
ryanyuan
commented
Jul 12, 2019

Copy link


Copy Markdown

Contributor
Could you write some unit tests for it?
ryanyuan
ryanyuan
reviewed
Jul 12, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

ryanyuan
ryanyuan
reviewed
Jul 12, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@mik-lajmik-laj
added
the 
provider:google
Google (including GCP) related issues
label
Jul 22, 2019
@zzlbuaa
zzlbuaa
commented
Jul 24, 2019

Copy link


Copy Markdown

Hi, this hook method is a useful feature for our use case. I'm wondering are you still working on this PR?

cc @criccomini@whynick1
@benjamingrenier
benjamingrenier
commented
Jul 24, 2019
via email 

Copy link


Copy Markdown

ContributorAuthor
mik-laj
mik-laj
reviewed
Jul 27, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@zzlbuaa
zzlbuaa
commented
Aug 5, 2019

Copy link


Copy Markdown

@benjamingrenier any updates on this?
@benjamingrenier
benjamingrenier
commented
Aug 5, 2019
via email 

Copy link


Copy Markdown

ContributorAuthor
@mik-laj
mik-laj
commented
Aug 12, 2019

Copy link


Copy Markdown

Member
Are you planning to continue working on this change? This week, I would like to deal with PR reviews related to GCP. I would be happy if you would respond to all comments
@benjamingrenier
benjamingrenier
commented
Aug 12, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Yes I will work on it this week. I try to resolve my docker errors to run unit-tests locally.
@mik-laj
mik-laj
commented
Aug 12, 2019

Copy link


Copy Markdown

Member
Recently introduced a new development environment. have you tried them? we plan to create a new extension to improve GCP development.
mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@mik-laj
mik-laj
commented
Aug 14, 2019

Copy link


Copy Markdown

Member
Can you do rebase, please?
@benjamingrenier
benjamingrenier
commented
Aug 20, 2019

Copy link


Copy Markdown

ContributorAuthor
Can you do rebase, please?
It's done. Ok for you ?
@mik-laj
mik-laj
commented
Aug 20, 2019

Copy link


Copy Markdown

Member
Something is wrong with this PR. Too many commits.
I'm on vacation now🇫🇷. I'll look at this on Friday.
@benjamingrenier
benjamingrenier
commented
Aug 20, 2019
via email 

Copy link


Copy Markdown

ContributorAuthor
@ryanyuan
ryanyuan
commented
Aug 20, 2019

Copy link


Copy Markdown

Contributor
@benjamingrenier After rebasing, squash your commits and then do a force push.
@mik-laj
mik-laj
commented
Aug 20, 2019

Copy link


Copy Markdown

Member
I suspect that the references to the main branch are broken and the history will have to be rewritten (git reset --hard, git cherry-pick).
@benjamingrenier
benjamingrenierforce-pushed
the
AIRFLOW-4935
branch
6 times, most recently
from
acd2c9f to
af4e3d4CompareAugust 20, 2019 13:04
@benjamingrenier
benjamingrenier
commented
Aug 20, 2019

Copy link


Copy Markdown

ContributorAuthor
I think it's all right.
mik-laj
mik-laj
reviewed
Aug 22, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@mik-lajmik-lajAug 22, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


Suggested change

 self.log.info(dataset_tables_list)
I haven't seen it before, but it seems to me that there will be too much information in the logs.
@benjamingrenierbenjamingrenierSep 2, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will fix it soon. I prepare a commit with more unit tests too.
@potiukpotiukSep 6, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please remove the log before merge?
@mik-laj
mik-laj
commented
Sep 5, 2019

Copy link


Copy Markdown

Member
Hi.
I made a change in the base class - GoogleCloudBaseHook. Your PR may need to be changed. Could you do rebase?
Cheers
Reference:
#5907
potiuk
potiuk
requested changes
Sep 6, 2019
View reviewed changes
@potiukpotiuk
left a comment
•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@benjamingrenier We are in the middle of moving hooks and operators from contrib to core. Is it possible to rebase (add more unittestss?) quickly so that we can merge the change before we move it?
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@potiukpotiukSep 6, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please remove the log before merge?
@benjamingrenier
benjamingrenier
commented
Sep 6, 2019

Copy link


Copy Markdown

ContributorAuthor
@benjamingrenier We are in the middle of moving hooks and operators from contrib to core. Is it possible to rebase (add more unittestss?) quickly so that we can merge the change before we move it?
Rebase done and unittests added
@potiuk
potiuk
commented
Sep 21, 2019

Copy link


Copy Markdown

Member
Hey @benjamingrenier -> I must ask for another rebase: (.
@benjamingrenier
benjamingrenier
commented
Sep 21, 2019
via email 

Copy link


Copy Markdown

ContributorAuthor
@codecov-io
codecov-io
commented
Sep 22, 2019•

edited

Loading

Copy link


Copy Markdown

Codecov Report
Merging #5566 into master will increase coverage by 0.13%.

The diff coverage is 94.44%.
Impacted file tree graph
@@ Coverage Diff @@## master #5566 +/- ##
==========================================
+ Coverage 79.93% 80.06% +0.13% 
==========================================
Files 608 608 Lines 35030 35048 +18 ==========================================
+ Hits 28000 28062 +62 + Misses 7030 6986 -44
Impacted FilesCoverage Δ
airflow/gcp/hooks/bigquery.py70.85% <94.44%> (+0.64%)⬆️
airflow/models/taskinstance.py93.72% <0%> (+0.5%)⬆️
airflow/jobs/scheduler_job.py74.28% <0%> (+0.9%)⬆️
airflow/utils/dag_processing.py58.98% <0%> (+2.54%)⬆️
airflow/executors/__init__.py67.34% <0%> (+4.08%)⬆️
airflow/utils/sqlalchemy.py93.22% <0%> (+6.77%)⬆️
airflow/utils/log/colored_log.py93.18% <0%> (+11.36%)⬆️
airflow/executors/sequential_executor.py100% <0%> (+52.38%)⬆️
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data

Powered by Codecov. Last update 2d19a41...e1d8ecd. Read the comment docs.
@whynick1
whynick1
commented
Nov 15, 2019

Copy link


Copy Markdown

Contributor
I am also interested in this PR, any update? 😄
@potiuk@mik-laj@ryanyuan
@benjamingrenier
benjamingrenier
commented
Nov 29, 2019

Copy link


Copy Markdown

ContributorAuthor
Someone can do the review ?
@mik-laj
mik-laj
commented
Nov 30, 2019

Copy link


Copy Markdown

Member
I added this PR to the queue next week. I will try to review it as soon as possible.
@mik-lajmik-laj
mentioned this pull request
Nov 30, 2019

Closed

@mik-laj
mik-laj
merged commit 4561978
into
apache:masterDec 8, 2019
galuszkak pushed a commit
to FlyrInc/apache-airflow
that referenced
this pull request
Mar 5, 2020
@benjamingrenier@galuszkak
[AIRFLOW-4935] Add function get_dataset_tables_list to BigQueryHook (a…
62878fb
…pache#5566)
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@mik-lajmik-lajmik-laj approved these changes
@potiukpotiukAwaiting requested review from potiuk

+1 more reviewer

@ryanyuanryanyuanryanyuan left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

provider:googleGoogle (including GCP) related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

7 participants

@benjamingrenier@ryanyuan@zzlbuaa@mik-laj@potiuk@codecov-io@whynick1

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
[AIRFLOW-4935] Add method in the bigquery hook to list tables in a dataset - #5566
Merged
mik-laj merged 1 commit into
apache:masterfrom 
benjamingrenier:AIRFLOW-4935
Dec 8, 2019
Merged
[AIRFLOW-4935] Add method in the bigquery hook to list tables in a dataset#5566
mik-laj merged 1 commit into
apache:masterfrom 
benjamingrenier:AIRFLOW-4935
Conversation
@benjamingrenier
@benjamingrenierbenjamingrenier
commented
Jul 11, 2019

Copy link


Copy Markdown

Contributor
Method get_dataset_tables_list added in the hook, permits to use a table prefix too.
@ryanyuan
ryanyuan
commented
Jul 12, 2019

Copy link


Copy Markdown

Contributor
Could you write some unit tests for it?
ryanyuan
ryanyuan
reviewed
Jul 12, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

ryanyuan
ryanyuan
reviewed
Jul 12, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@mik-lajmik-laj
added
the 
provider:google
Google (including GCP) related issues
label
Jul 22, 2019
@zzlbuaa
zzlbuaa
commented
Jul 24, 2019

Copy link


Copy Markdown

Hi, this hook method is a useful feature for our use case. I'm wondering are you still working on this PR?

cc @criccomini@whynick1
@benjamingrenier
benjamingrenier
commented
Jul 24, 2019
via email 

Copy link


Copy Markdown

ContributorAuthor
mik-laj
mik-laj
reviewed
Jul 27, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@zzlbuaa
zzlbuaa
commented
Aug 5, 2019

Copy link


Copy Markdown

@benjamingrenier any updates on this?
@benjamingrenier
benjamingrenier
commented
Aug 5, 2019
via email 

Copy link


Copy Markdown

ContributorAuthor
@mik-laj
mik-laj
commented
Aug 12, 2019

Copy link


Copy Markdown

Member
Are you planning to continue working on this change? This week, I would like to deal with PR reviews related to GCP. I would be happy if you would respond to all comments
@benjamingrenier
benjamingrenier
commented
Aug 12, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Yes I will work on it this week. I try to resolve my docker errors to run unit-tests locally.
@mik-laj
mik-laj
commented
Aug 12, 2019

Copy link


Copy Markdown

Member
Recently introduced a new development environment. have you tried them? we plan to create a new extension to improve GCP development.
mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@mik-laj
mik-laj
commented
Aug 14, 2019

Copy link


Copy Markdown

Member
Can you do rebase, please?
@benjamingrenier
benjamingrenier
commented
Aug 20, 2019

Copy link


Copy Markdown

ContributorAuthor
Can you do rebase, please?
It's done. Ok for you ?
@mik-laj
mik-laj
commented
Aug 20, 2019

Copy link


Copy Markdown

Member
Something is wrong with this PR. Too many commits.
I'm on vacation now🇫🇷. I'll look at this on Friday.
@benjamingrenier
benjamingrenier
commented
Aug 20, 2019
via email 

Copy link


Copy Markdown

ContributorAuthor
@ryanyuan
ryanyuan
commented
Aug 20, 2019

Copy link


Copy Markdown

Contributor
@benjamingrenier After rebasing, squash your commits and then do a force push.
@mik-laj
mik-laj
commented
Aug 20, 2019

Copy link


Copy Markdown

Member
I suspect that the references to the main branch are broken and the history will have to be rewritten (git reset --hard, git cherry-pick).
@benjamingrenier
benjamingrenierforce-pushed
the
AIRFLOW-4935
branch
6 times, most recently
from
acd2c9f to
af4e3d4CompareAugust 20, 2019 13:04
@benjamingrenier
benjamingrenier
commented
Aug 20, 2019

Copy link


Copy Markdown

ContributorAuthor
I think it's all right.
mik-laj
mik-laj
reviewed
Aug 22, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@mik-lajmik-lajAug 22, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


Suggested change

 self.log.info(dataset_tables_list)
I haven't seen it before, but it seems to me that there will be too much information in the logs.
@benjamingrenierbenjamingrenierSep 2, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will fix it soon. I prepare a commit with more unit tests too.
@potiukpotiukSep 6, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please remove the log before merge?
@mik-laj
mik-laj
commented
Sep 5, 2019

Copy link


Copy Markdown

Member
Hi.
I made a change in the base class - GoogleCloudBaseHook. Your PR may need to be changed. Could you do rebase?
Cheers
Reference:
#5907
potiuk
potiuk
requested changes
Sep 6, 2019
View reviewed changes
@potiukpotiuk
left a comment
•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@benjamingrenier We are in the middle of moving hooks and operators from contrib to core. Is it possible to rebase (add more unittestss?) quickly so that we can merge the change before we move it?
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@potiukpotiukSep 6, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please remove the log before merge?
@benjamingrenier
benjamingrenier
commented
Sep 6, 2019

Copy link


Copy Markdown

ContributorAuthor
@benjamingrenier We are in the middle of moving hooks and operators from contrib to core. Is it possible to rebase (add more unittestss?) quickly so that we can merge the change before we move it?
Rebase done and unittests added
@potiuk
potiuk
commented
Sep 21, 2019

Copy link


Copy Markdown

Member
Hey @benjamingrenier -> I must ask for another rebase: (.
@benjamingrenier
benjamingrenier
commented
Sep 21, 2019
via email 

Copy link


Copy Markdown

ContributorAuthor
@codecov-io
codecov-io
commented
Sep 22, 2019•

edited

Loading

Copy link


Copy Markdown

Codecov Report
Merging #5566 into master will increase coverage by 0.13%.

The diff coverage is 94.44%.
Impacted file tree graph
@@ Coverage Diff @@## master #5566 +/- ##
==========================================
+ Coverage 79.93% 80.06% +0.13% 
==========================================
Files 608 608 Lines 35030 35048 +18 ==========================================
+ Hits 28000 28062 +62 + Misses 7030 6986 -44
Impacted FilesCoverage Δ
airflow/gcp/hooks/bigquery.py70.85% <94.44%> (+0.64%)⬆️
airflow/models/taskinstance.py93.72% <0%> (+0.5%)⬆️
airflow/jobs/scheduler_job.py74.28% <0%> (+0.9%)⬆️
airflow/utils/dag_processing.py58.98% <0%> (+2.54%)⬆️
airflow/executors/__init__.py67.34% <0%> (+4.08%)⬆️
airflow/utils/sqlalchemy.py93.22% <0%> (+6.77%)⬆️
airflow/utils/log/colored_log.py93.18% <0%> (+11.36%)⬆️
airflow/executors/sequential_executor.py100% <0%> (+52.38%)⬆️
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data

Powered by Codecov. Last update 2d19a41...e1d8ecd. Read the comment docs.
@whynick1
whynick1
commented
Nov 15, 2019

Copy link


Copy Markdown

Contributor
I am also interested in this PR, any update? 😄
@potiuk@mik-laj@ryanyuan
@benjamingrenier
benjamingrenier
commented
Nov 29, 2019

Copy link


Copy Markdown

ContributorAuthor
Someone can do the review ?
@mik-laj
mik-laj
commented
Nov 30, 2019

Copy link


Copy Markdown

Member
I added this PR to the queue next week. I will try to review it as soon as possible.
@mik-lajmik-laj
mentioned this pull request
Nov 30, 2019

Closed

@mik-laj
mik-laj
merged commit 4561978
into
apache:masterDec 8, 2019
galuszkak pushed a commit
to FlyrInc/apache-airflow
that referenced
this pull request
Mar 5, 2020
@benjamingrenier@galuszkak
[AIRFLOW-4935] Add function get_dataset_tables_list to BigQueryHook (a…
62878fb
…pache#5566)
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@mik-lajmik-lajmik-laj approved these changes
@potiukpotiukAwaiting requested review from potiuk

+1 more reviewer

@ryanyuanryanyuanryanyuan left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

provider:googleGoogle (including GCP) related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

7 participants

@benjamingrenier@ryanyuan@zzlbuaa@mik-laj@potiuk@codecov-io@whynick1

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
[AIRFLOW-4935] Add method in the bigquery hook to list tables in a dataset - #5566
Merged
mik-laj merged 1 commit into
apache:masterfrom 
benjamingrenier:AIRFLOW-4935
Dec 8, 2019
Merged
[AIRFLOW-4935] Add method in the bigquery hook to list tables in a dataset#5566
mik-laj merged 1 commit into
apache:masterfrom 
benjamingrenier:AIRFLOW-4935
Conversation
@benjamingrenier
@benjamingrenierbenjamingrenier
commented
Jul 11, 2019

Copy link


Copy Markdown

Contributor
Method get_dataset_tables_list added in the hook, permits to use a table prefix too.
@ryanyuan
ryanyuan
commented
Jul 12, 2019

Copy link


Copy Markdown

Contributor
Could you write some unit tests for it?
ryanyuan
ryanyuan
reviewed
Jul 12, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

ryanyuan
ryanyuan
reviewed
Jul 12, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@mik-lajmik-laj
added
the 
provider:google
Google (including GCP) related issues
label
Jul 22, 2019
@zzlbuaa
zzlbuaa
commented
Jul 24, 2019

Copy link


Copy Markdown

Hi, this hook method is a useful feature for our use case. I'm wondering are you still working on this PR?

cc @criccomini@whynick1
@benjamingrenier
benjamingrenier
commented
Jul 24, 2019
via email 

Copy link


Copy Markdown

ContributorAuthor
mik-laj
mik-laj
reviewed
Jul 27, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@zzlbuaa
zzlbuaa
commented
Aug 5, 2019

Copy link


Copy Markdown

@benjamingrenier any updates on this?
@benjamingrenier
benjamingrenier
commented
Aug 5, 2019
via email 

Copy link


Copy Markdown

ContributorAuthor
@mik-laj
mik-laj
commented
Aug 12, 2019

Copy link


Copy Markdown

Member
Are you planning to continue working on this change? This week, I would like to deal with PR reviews related to GCP. I would be happy if you would respond to all comments
@benjamingrenier
benjamingrenier
commented
Aug 12, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Yes I will work on it this week. I try to resolve my docker errors to run unit-tests locally.
@mik-laj
mik-laj
commented
Aug 12, 2019

Copy link


Copy Markdown

Member
Recently introduced a new development environment. have you tried them? we plan to create a new extension to improve GCP development.
mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@mik-laj
mik-laj
commented
Aug 14, 2019

Copy link


Copy Markdown

Member
Can you do rebase, please?
@benjamingrenier
benjamingrenier
commented
Aug 20, 2019

Copy link


Copy Markdown

ContributorAuthor
Can you do rebase, please?
It's done. Ok for you ?
@mik-laj
mik-laj
commented
Aug 20, 2019

Copy link


Copy Markdown

Member
Something is wrong with this PR. Too many commits.
I'm on vacation now🇫🇷. I'll look at this on Friday.
@benjamingrenier
benjamingrenier
commented
Aug 20, 2019
via email 

Copy link


Copy Markdown

ContributorAuthor
@ryanyuan
ryanyuan
commented
Aug 20, 2019

Copy link


Copy Markdown

Contributor
@benjamingrenier After rebasing, squash your commits and then do a force push.
@mik-laj
mik-laj
commented
Aug 20, 2019

Copy link


Copy Markdown

Member
I suspect that the references to the main branch are broken and the history will have to be rewritten (git reset --hard, git cherry-pick).
@benjamingrenier
benjamingrenierforce-pushed
the
AIRFLOW-4935
branch
6 times, most recently
from
acd2c9f to
af4e3d4CompareAugust 20, 2019 13:04
@benjamingrenier
benjamingrenier
commented
Aug 20, 2019

Copy link


Copy Markdown

ContributorAuthor
I think it's all right.
mik-laj
mik-laj
reviewed
Aug 22, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@mik-lajmik-lajAug 22, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


Suggested change

 self.log.info(dataset_tables_list)
I haven't seen it before, but it seems to me that there will be too much information in the logs.
@benjamingrenierbenjamingrenierSep 2, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will fix it soon. I prepare a commit with more unit tests too.
@potiukpotiukSep 6, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please remove the log before merge?
@mik-laj
mik-laj
commented
Sep 5, 2019

Copy link


Copy Markdown

Member
Hi.
I made a change in the base class - GoogleCloudBaseHook. Your PR may need to be changed. Could you do rebase?
Cheers
Reference:
#5907
potiuk
potiuk
requested changes
Sep 6, 2019
View reviewed changes
@potiukpotiuk
left a comment
•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@benjamingrenier We are in the middle of moving hooks and operators from contrib to core. Is it possible to rebase (add more unittestss?) quickly so that we can merge the change before we move it?
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@potiukpotiukSep 6, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please remove the log before merge?
@benjamingrenier
benjamingrenier
commented
Sep 6, 2019

Copy link


Copy Markdown

ContributorAuthor
@benjamingrenier We are in the middle of moving hooks and operators from contrib to core. Is it possible to rebase (add more unittestss?) quickly so that we can merge the change before we move it?
Rebase done and unittests added
@potiuk
potiuk
commented
Sep 21, 2019

Copy link


Copy Markdown

Member
Hey @benjamingrenier -> I must ask for another rebase: (.
@benjamingrenier
benjamingrenier
commented
Sep 21, 2019
via email 

Copy link


Copy Markdown

ContributorAuthor
@codecov-io
codecov-io
commented
Sep 22, 2019•

edited

Loading

Copy link


Copy Markdown

Codecov Report
Merging #5566 into master will increase coverage by 0.13%.

The diff coverage is 94.44%.
Impacted file tree graph
@@ Coverage Diff @@## master #5566 +/- ##
==========================================
+ Coverage 79.93% 80.06% +0.13% 
==========================================
Files 608 608 Lines 35030 35048 +18 ==========================================
+ Hits 28000 28062 +62 + Misses 7030 6986 -44
Impacted FilesCoverage Δ
airflow/gcp/hooks/bigquery.py70.85% <94.44%> (+0.64%)⬆️
airflow/models/taskinstance.py93.72% <0%> (+0.5%)⬆️
airflow/jobs/scheduler_job.py74.28% <0%> (+0.9%)⬆️
airflow/utils/dag_processing.py58.98% <0%> (+2.54%)⬆️
airflow/executors/__init__.py67.34% <0%> (+4.08%)⬆️
airflow/utils/sqlalchemy.py93.22% <0%> (+6.77%)⬆️
airflow/utils/log/colored_log.py93.18% <0%> (+11.36%)⬆️
airflow/executors/sequential_executor.py100% <0%> (+52.38%)⬆️
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data

Powered by Codecov. Last update 2d19a41...e1d8ecd. Read the comment docs.
@whynick1
whynick1
commented
Nov 15, 2019

Copy link


Copy Markdown

Contributor
I am also interested in this PR, any update? 😄
@potiuk@mik-laj@ryanyuan
@benjamingrenier
benjamingrenier
commented
Nov 29, 2019

Copy link


Copy Markdown

ContributorAuthor
Someone can do the review ?
@mik-laj
mik-laj
commented
Nov 30, 2019

Copy link


Copy Markdown

Member
I added this PR to the queue next week. I will try to review it as soon as possible.
@mik-lajmik-laj
mentioned this pull request
Nov 30, 2019

Closed

@mik-laj
mik-laj
merged commit 4561978
into
apache:masterDec 8, 2019
galuszkak pushed a commit
to FlyrInc/apache-airflow
that referenced
this pull request
Mar 5, 2020
@benjamingrenier@galuszkak
[AIRFLOW-4935] Add function get_dataset_tables_list to BigQueryHook (a…
62878fb
…pache#5566)
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@mik-lajmik-lajmik-laj approved these changes
@potiukpotiukAwaiting requested review from potiuk

+1 more reviewer

@ryanyuanryanyuanryanyuan left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

provider:googleGoogle (including GCP) related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

7 participants

@benjamingrenier@ryanyuan@zzlbuaa@mik-laj@potiuk@codecov-io@whynick1

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
[AIRFLOW-4935] Add method in the bigquery hook to list tables in a dataset - #5566
Merged
mik-laj merged 1 commit into
apache:masterfrom 
benjamingrenier:AIRFLOW-4935
Dec 8, 2019
Merged
[AIRFLOW-4935] Add method in the bigquery hook to list tables in a dataset#5566
mik-laj merged 1 commit into
apache:masterfrom 
benjamingrenier:AIRFLOW-4935
Conversation
@benjamingrenier
@benjamingrenierbenjamingrenier
commented
Jul 11, 2019

Copy link


Copy Markdown

Contributor
Method get_dataset_tables_list added in the hook, permits to use a table prefix too.
@ryanyuan
ryanyuan
commented
Jul 12, 2019

Copy link


Copy Markdown

Contributor
Could you write some unit tests for it?
ryanyuan
ryanyuan
reviewed
Jul 12, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

ryanyuan
ryanyuan
reviewed
Jul 12, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@mik-lajmik-laj
added
the 
provider:google
Google (including GCP) related issues
label
Jul 22, 2019
@zzlbuaa
zzlbuaa
commented
Jul 24, 2019

Copy link


Copy Markdown

Hi, this hook method is a useful feature for our use case. I'm wondering are you still working on this PR?

cc @criccomini@whynick1
@benjamingrenier
benjamingrenier
commented
Jul 24, 2019
via email 

Copy link


Copy Markdown

ContributorAuthor
mik-laj
mik-laj
reviewed
Jul 27, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@zzlbuaa
zzlbuaa
commented
Aug 5, 2019

Copy link


Copy Markdown

@benjamingrenier any updates on this?
@benjamingrenier
benjamingrenier
commented
Aug 5, 2019
via email 

Copy link


Copy Markdown

ContributorAuthor
@mik-laj
mik-laj
commented
Aug 12, 2019

Copy link


Copy Markdown

Member
Are you planning to continue working on this change? This week, I would like to deal with PR reviews related to GCP. I would be happy if you would respond to all comments
@benjamingrenier
benjamingrenier
commented
Aug 12, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Yes I will work on it this week. I try to resolve my docker errors to run unit-tests locally.
@mik-laj
mik-laj
commented
Aug 12, 2019

Copy link


Copy Markdown

Member
Recently introduced a new development environment. have you tried them? we plan to create a new extension to improve GCP development.
mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@mik-laj
mik-laj
commented
Aug 14, 2019

Copy link


Copy Markdown

Member
Can you do rebase, please?
@benjamingrenier
benjamingrenier
commented
Aug 20, 2019

Copy link


Copy Markdown

ContributorAuthor
Can you do rebase, please?
It's done. Ok for you ?
@mik-laj
mik-laj
commented
Aug 20, 2019

Copy link


Copy Markdown

Member
Something is wrong with this PR. Too many commits.
I'm on vacation now🇫🇷. I'll look at this on Friday.
@benjamingrenier
benjamingrenier
commented
Aug 20, 2019
via email 

Copy link


Copy Markdown

ContributorAuthor
@ryanyuan
ryanyuan
commented
Aug 20, 2019

Copy link


Copy Markdown

Contributor
@benjamingrenier After rebasing, squash your commits and then do a force push.
@mik-laj
mik-laj
commented
Aug 20, 2019

Copy link


Copy Markdown

Member
I suspect that the references to the main branch are broken and the history will have to be rewritten (git reset --hard, git cherry-pick).
@benjamingrenier
benjamingrenierforce-pushed
the
AIRFLOW-4935
branch
6 times, most recently
from
acd2c9f to
af4e3d4CompareAugust 20, 2019 13:04
@benjamingrenier
benjamingrenier
commented
Aug 20, 2019

Copy link


Copy Markdown

ContributorAuthor
I think it's all right.
mik-laj
mik-laj
reviewed
Aug 22, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@mik-lajmik-lajAug 22, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


Suggested change

 self.log.info(dataset_tables_list)
I haven't seen it before, but it seems to me that there will be too much information in the logs.
@benjamingrenierbenjamingrenierSep 2, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will fix it soon. I prepare a commit with more unit tests too.
@potiukpotiukSep 6, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please remove the log before merge?
@mik-laj
mik-laj
commented
Sep 5, 2019

Copy link


Copy Markdown

Member
Hi.
I made a change in the base class - GoogleCloudBaseHook. Your PR may need to be changed. Could you do rebase?
Cheers
Reference:
#5907
potiuk
potiuk
requested changes
Sep 6, 2019
View reviewed changes
@potiukpotiuk
left a comment
•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@benjamingrenier We are in the middle of moving hooks and operators from contrib to core. Is it possible to rebase (add more unittestss?) quickly so that we can merge the change before we move it?
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@potiukpotiukSep 6, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please remove the log before merge?
@benjamingrenier
benjamingrenier
commented
Sep 6, 2019

Copy link


Copy Markdown

ContributorAuthor
@benjamingrenier We are in the middle of moving hooks and operators from contrib to core. Is it possible to rebase (add more unittestss?) quickly so that we can merge the change before we move it?
Rebase done and unittests added
@potiuk
potiuk
commented
Sep 21, 2019

Copy link


Copy Markdown

Member
Hey @benjamingrenier -> I must ask for another rebase: (.
@benjamingrenier
benjamingrenier
commented
Sep 21, 2019
via email 

Copy link


Copy Markdown

ContributorAuthor
@codecov-io
codecov-io
commented
Sep 22, 2019•

edited

Loading

Copy link


Copy Markdown

Codecov Report
Merging #5566 into master will increase coverage by 0.13%.

The diff coverage is 94.44%.
Impacted file tree graph
@@ Coverage Diff @@## master #5566 +/- ##
==========================================
+ Coverage 79.93% 80.06% +0.13% 
==========================================
Files 608 608 Lines 35030 35048 +18 ==========================================
+ Hits 28000 28062 +62 + Misses 7030 6986 -44
Impacted FilesCoverage Δ
airflow/gcp/hooks/bigquery.py70.85% <94.44%> (+0.64%)⬆️
airflow/models/taskinstance.py93.72% <0%> (+0.5%)⬆️
airflow/jobs/scheduler_job.py74.28% <0%> (+0.9%)⬆️
airflow/utils/dag_processing.py58.98% <0%> (+2.54%)⬆️
airflow/executors/__init__.py67.34% <0%> (+4.08%)⬆️
airflow/utils/sqlalchemy.py93.22% <0%> (+6.77%)⬆️
airflow/utils/log/colored_log.py93.18% <0%> (+11.36%)⬆️
airflow/executors/sequential_executor.py100% <0%> (+52.38%)⬆️
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data

Powered by Codecov. Last update 2d19a41...e1d8ecd. Read the comment docs.
@whynick1
whynick1
commented
Nov 15, 2019

Copy link


Copy Markdown

Contributor
I am also interested in this PR, any update? 😄
@potiuk@mik-laj@ryanyuan
@benjamingrenier
benjamingrenier
commented
Nov 29, 2019

Copy link


Copy Markdown

ContributorAuthor
Someone can do the review ?
@mik-laj
mik-laj
commented
Nov 30, 2019

Copy link


Copy Markdown

Member
I added this PR to the queue next week. I will try to review it as soon as possible.
@mik-lajmik-laj
mentioned this pull request
Nov 30, 2019

Closed

@mik-laj
mik-laj
merged commit 4561978
into
apache:masterDec 8, 2019
galuszkak pushed a commit
to FlyrInc/apache-airflow
that referenced
this pull request
Mar 5, 2020
@benjamingrenier@galuszkak
[AIRFLOW-4935] Add function get_dataset_tables_list to BigQueryHook (a…
62878fb
…pache#5566)
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@mik-lajmik-lajmik-laj approved these changes
@potiukpotiukAwaiting requested review from potiuk

+1 more reviewer

@ryanyuanryanyuanryanyuan left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

provider:googleGoogle (including GCP) related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

7 participants

@benjamingrenier@ryanyuan@zzlbuaa@mik-laj@potiuk@codecov-io@whynick1

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
[AIRFLOW-4935] Add method in the bigquery hook to list tables in a dataset - #5566
Merged
mik-laj merged 1 commit into
apache:masterfrom 
benjamingrenier:AIRFLOW-4935
Dec 8, 2019
Merged
[AIRFLOW-4935] Add method in the bigquery hook to list tables in a dataset#5566
mik-laj merged 1 commit into
apache:masterfrom 
benjamingrenier:AIRFLOW-4935
Conversation
@benjamingrenier
@benjamingrenierbenjamingrenier
commented
Jul 11, 2019

Copy link


Copy Markdown

Contributor
Method get_dataset_tables_list added in the hook, permits to use a table prefix too.
@ryanyuan
ryanyuan
commented
Jul 12, 2019

Copy link


Copy Markdown

Contributor
Could you write some unit tests for it?
ryanyuan
ryanyuan
reviewed
Jul 12, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

ryanyuan
ryanyuan
reviewed
Jul 12, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@mik-lajmik-laj
added
the 
provider:google
Google (including GCP) related issues
label
Jul 22, 2019
@zzlbuaa
zzlbuaa
commented
Jul 24, 2019

Copy link


Copy Markdown

Hi, this hook method is a useful feature for our use case. I'm wondering are you still working on this PR?

cc @criccomini@whynick1
@benjamingrenier
benjamingrenier
commented
Jul 24, 2019
via email 

Copy link


Copy Markdown

ContributorAuthor
mik-laj
mik-laj
reviewed
Jul 27, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@zzlbuaa
zzlbuaa
commented
Aug 5, 2019

Copy link


Copy Markdown

@benjamingrenier any updates on this?
@benjamingrenier
benjamingrenier
commented
Aug 5, 2019
via email 

Copy link


Copy Markdown

ContributorAuthor
@mik-laj
mik-laj
commented
Aug 12, 2019

Copy link


Copy Markdown

Member
Are you planning to continue working on this change? This week, I would like to deal with PR reviews related to GCP. I would be happy if you would respond to all comments
@benjamingrenier
benjamingrenier
commented
Aug 12, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Yes I will work on it this week. I try to resolve my docker errors to run unit-tests locally.
@mik-laj
mik-laj
commented
Aug 12, 2019

Copy link


Copy Markdown

Member
Recently introduced a new development environment. have you tried them? we plan to create a new extension to improve GCP development.
mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

mik-laj
mik-laj
reviewed
Aug 13, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@mik-laj
mik-laj
commented
Aug 14, 2019

Copy link


Copy Markdown

Member
Can you do rebase, please?
@benjamingrenier
benjamingrenier
commented
Aug 20, 2019

Copy link


Copy Markdown

ContributorAuthor
Can you do rebase, please?
It's done. Ok for you ?
@mik-laj
mik-laj
commented
Aug 20, 2019

Copy link


Copy Markdown

Member
Something is wrong with this PR. Too many commits.
I'm on vacation now🇫🇷. I'll look at this on Friday.
@benjamingrenier
benjamingrenier
commented
Aug 20, 2019
via email 

Copy link


Copy Markdown

ContributorAuthor
@ryanyuan
ryanyuan
commented
Aug 20, 2019

Copy link


Copy Markdown

Contributor
@benjamingrenier After rebasing, squash your commits and then do a force push.
@mik-laj
mik-laj
commented
Aug 20, 2019

Copy link


Copy Markdown

Member
I suspect that the references to the main branch are broken and the history will have to be rewritten (git reset --hard, git cherry-pick).
@benjamingrenier
benjamingrenierforce-pushed
the
AIRFLOW-4935
branch
6 times, most recently
from
acd2c9f to
af4e3d4CompareAugust 20, 2019 13:04
@benjamingrenier
benjamingrenier
commented
Aug 20, 2019

Copy link


Copy Markdown

ContributorAuthor
I think it's all right.
mik-laj
mik-laj
reviewed
Aug 22, 2019
View reviewed changes
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@mik-lajmik-lajAug 22, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


Suggested change

 self.log.info(dataset_tables_list)
I haven't seen it before, but it seems to me that there will be too much information in the logs.
@benjamingrenierbenjamingrenierSep 2, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will fix it soon. I prepare a commit with more unit tests too.
@potiukpotiukSep 6, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please remove the log before merge?
@mik-laj
mik-laj
commented
Sep 5, 2019

Copy link


Copy Markdown

Member
Hi.
I made a change in the base class - GoogleCloudBaseHook. Your PR may need to be changed. Could you do rebase?
Cheers
Reference:
#5907
potiuk
potiuk
requested changes
Sep 6, 2019
View reviewed changes
@potiukpotiuk
left a comment
•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@benjamingrenier We are in the middle of moving hooks and operators from contrib to core. Is it possible to rebase (add more unittestss?) quickly so that we can merge the change before we move it?
Comment threadairflow/contrib/hooks/bigquery_hook.py
Outdated

@potiukpotiukSep 6, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please remove the log before merge?
@benjamingrenier
benjamingrenier
commented
Sep 6, 2019

Copy link


Copy Markdown

ContributorAuthor
@benjamingrenier We are in the middle of moving hooks and operators from contrib to core. Is it possible to rebase (add more unittestss?) quickly so that we can merge the change before we move it?
Rebase done and unittests added
@potiuk
potiuk
commented
Sep 21, 2019

Copy link


Copy Markdown

Member
Hey @benjamingrenier -> I must ask for another rebase: (.
@benjamingrenier
benjamingrenier
commented
Sep 21, 2019
via email 

Copy link


Copy Markdown

ContributorAuthor
@codecov-io
codecov-io
commented
Sep 22, 2019•

edited

Loading

Copy link


Copy Markdown

Codecov Report
Merging #5566 into master will increase coverage by 0.13%.

The diff coverage is 94.44%.
Impacted file tree graph
@@ Coverage Diff @@## master #5566 +/- ##
==========================================
+ Coverage 79.93% 80.06% +0.13% 
==========================================
Files 608 608 Lines 35030 35048 +18 ==========================================
+ Hits 28000 28062 +62 + Misses 7030 6986 -44
Impacted FilesCoverage Δ
airflow/gcp/hooks/bigquery.py70.85% <94.44%> (+0.64%)⬆️
airflow/models/taskinstance.py93.72% <0%> (+0.5%)⬆️
airflow/jobs/scheduler_job.py74.28% <0%> (+0.9%)⬆️
airflow/utils/dag_processing.py58.98% <0%> (+2.54%)⬆️
airflow/executors/__init__.py67.34% <0%> (+4.08%)⬆️
airflow/utils/sqlalchemy.py93.22% <0%> (+6.77%)⬆️
airflow/utils/log/colored_log.py93.18% <0%> (+11.36%)⬆️
airflow/executors/sequential_executor.py100% <0%> (+52.38%)⬆️
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data

Powered by Codecov. Last update 2d19a41...e1d8ecd. Read the comment docs.
@whynick1
whynick1
commented
Nov 15, 2019

Copy link


Copy Markdown

Contributor
I am also interested in this PR, any update? 😄
@potiuk@mik-laj@ryanyuan
@benjamingrenier
benjamingrenier
commented
Nov 29, 2019

Copy link


Copy Markdown

ContributorAuthor
Someone can do the review ?
@mik-laj
mik-laj
commented
Nov 30, 2019

Copy link


Copy Markdown

Member
I added this PR to the queue next week. I will try to review it as soon as possible.
@mik-lajmik-laj
mentioned this pull request
Nov 30, 2019

Closed

@mik-laj
mik-laj
merged commit 4561978
into
apache:masterDec 8, 2019
galuszkak pushed a commit
to FlyrInc/apache-airflow
that referenced
this pull request
Mar 5, 2020
@benjamingrenier@galuszkak
[AIRFLOW-4935] Add function get_dataset_tables_list to BigQueryHook (a…
62878fb
…pache#5566)
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@mik-lajmik-lajmik-laj approved these changes
@potiukpotiukAwaiting requested review from potiuk

+1 more reviewer

@ryanyuanryanyuanryanyuan left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

provider:googleGoogle (including GCP) related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

7 participants

@benjamingrenier@ryanyuan@zzlbuaa@mik-laj@potiuk@codecov-io@whynick1