Add Tabular provider by Fokko · Pull Request #23704 · apache/airflow

Fokko · 2022-05-13T21:40:30Z

First version of the Tabular hook :)

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragement file, named {pr_number}.significant.rst, in newsfragments.

kbendick

Thanks @Fokko! Left some comments with questions about various things.

turbaszek · 2022-05-16T19:40:28Z

eladkal

There is an ongoing discussion in mailing list about if we should add more providers
https://lists.apache.org/thread/nvfc75kj2w1tywvvkw8ho5wkx1dcvgrn

though I would say this is a unique case as a PMC is the codeowner :)

Fokko · 2022-05-17T14:05:10Z

Thanks for pointing me to the thread @eladkal

I actually started on a 3rd party provider, but setting it up within Airflow itself makes it so much easier because of Breeze :) @kbendick and I will maintain this one.

kaxil

This will probably be in the same boat as #22659 where we are waiting for what comes out of https://lists.apache.org/thread/nvfc75kj2w1tywvvkw8ho5wkx1dcvgrn

Co-authored-by: Kyle Bendickson <kyle@tabular.io>

kaxil · 2022-05-17T20:21:53Z

This will probably be in the same boat as #22659 where we are waiting for what comes out of https://lists.apache.org/thread/nvfc75kj2w1tywvvkw8ho5wkx1dcvgrn

Welp, looks like @eladkal had already commented about it :)

Fokko · 2022-06-14T14:41:32Z

@potiuk thanks for sharing the command. I just ran it and cleanup up the excess in the command hash file.

Would be great to get this in at some point, looking forward to the summary

potiuk · 2022-06-14T20:12:12Z

Would be great to get this in at some point, looking forward to the summary

Coming :)

fd-provider-tabular

Fokko · 2022-06-16T16:29:28Z

@potiuk I saw your mail: https://lists.apache.org/thread/6ngq79df7op541gfwntspdtsvzlv1cr6

It makes sense and I see the current issues. That being said, would it be possible to get this in? Both @kbendick and I will make sure that the operator will be maintained. One thing on the operator, the Tabular one will be quite small. I can also see that we release Apache Iceberg operators in the future as well, but that will probably also be just a wrapper around the Python pyiceberg library.

potiuk · 2022-06-16T18:04:41Z

@potiuk I saw your mail: https://lists.apache.org/thread/6ngq79df7op541gfwntspdtsvzlv1cr6
It makes sense and I see the current issues. That being said, would it be possible to get this in? Both @kbendick and I will make sure that the operator will be maintained. One thing on the operator, the Tabular one will be quite small. I can also see that we release Apache Iceberg operators in the future as well, but that will probably also be just a wrapper around the Python pyiceberg library.

I believe so :). I wanted just to get some community feedback, but I think having just a commitment from someone who has an interest (be it direct stakeholder or say a company that implements a lot of integrations with the product) - even verbally and confirmed in an issue should be more than enough. Feel free to comment in the discussion I started too :).

potiuk · 2022-06-27T15:18:52Z

Hey @Fokko, It took a bit long but I think we got to a consensus regarding new providers, and we also propose some kind of mix-governance aproach, where (among the others) the stakeholders for the future providers (which we are going to technically split to separate repositories soon) should take a bit more responsibility for maintenance:

#24680

If that does not scare you away, and you still want to add the provider to Airflow community providers, feel free to rebase the PR. I will also ask you (hopefully it will be merged soon) to rebase it after we merge #24672 - we are going to change the way how we keep depdencies for providers in order to prepare them to separate to different repository.

Let us know what you want to do, either close the PR or rebase it and lead it to completion after #24672 is merged.

…ular

Fokko · 2022-06-29T21:13:27Z

Hey @potiuk, of course I'm still interested in getting this in! I'm working on rebasing the PR, but I'm also seeing some issues with the recent changes. Let me dig into it and get back to you

…ular

eladkal · 2022-06-30T12:21:09Z

@@ -0,0 +1,48 @@
+# Licensed to the Apache Software Foundation (ASF) under one


new example dags should be written according to AIP-47
the example dag should be in system/providers/tabular path
see #24641 for reference

All green now :)

…ular

josh-fell

Just some small things but LGTM.

josh-fell · 2022-07-06T13:40:04Z

+`Tabular <https://tabular.io/>`__
+
+
+Release: 0.0.1


Suggested change
Release: 0.0.1
Release: 1.0.0

Ah.. was to quick :).. @Fokko - mind for a small follow-up?

Sure thing!

josh-fell · 2022-07-06T13:40:44Z

+Installation
+------------
+
+You can install this package on top of an existing Airflow 2.1+ installation via


Suggested change
You can install this package on top of an existing Airflow 2.1+ installation via
You can install this package on top of an existing Airflow 2.2+ installation via
The minimum version for providers was bumped to 2.2.0 recently 🙂

Missed that one, I'll do a follow-up! 👍🏻

josh-fell · 2022-07-06T13:43:26Z

+from airflow.providers.tabular.hooks.tabular import TabularHook
+
+
+class TestTabularHook(unittest.TestCase):


For net-new unit tests we should prefer pytest over unittest.

I actually prefer pytest as well. Let me follow up on this one

josh-fell · 2022-07-06T13:45:24Z

@@ -0,0 +1,39 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one


I think we typically want to have a URI example for connection in the doc. Is that right @mik-laj?

potiuk

Looks cool!

kbendick · 2022-07-06T20:08:48Z

Thank you @Fokko!

Fokko requested review from kaxil, mik-laj and potiuk as code owners May 13, 2022 21:40

boring-cyborgBot added area:dev-tools area:providers kind:documentation labels May 13, 2022

kbendick reviewed May 16, 2022
View reviewed changes

Comment threadairflow/providers/tabular/hooks/tabular.py Outdated

kbendick reviewed May 16, 2022
View reviewed changes

Comment threadairflow/providers/tabular/hooks/tabular.py Outdated
Comment threadairflow/utils/db.py Outdated
Comment threaddocs/apache-airflow-providers-tabular/connections.rst Outdated

turbaszek reviewed May 16, 2022
View reviewed changes

Comment thread.github/CODEOWNERS Outdated
turbaszekMay 16, 2022
Copy link
Copy Markdown
Member
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🚀

turbaszek reviewed May 16, 2022
View reviewed changes

Comment threadairflow/providers/tabular/example_dags/example_tabular.py Outdated

turbaszek reviewed May 16, 2022
View reviewed changes

Comment threadairflow/providers/tabular/example_dags/example_tabular.py Outdated

turbaszek reviewed May 16, 2022
View reviewed changes

Comment threadtests/providers/tabular/hooks/test_tabular.py Outdated

Fokko force-pushed the fd-provider-tabular branch 2 times, most recently from eb6028a to ddfe525Compare May 16, 2022 19:58

Fokko requested review from XD-DENG and ashb as code owners May 17, 2022 13:28

eladkal reviewed May 17, 2022
View reviewed changes

Comment threadairflow/models/dag.py Outdated

Fokko force-pushed the fd-provider-tabular branch from a682089 to 7c03bf4Compare May 17, 2022 14:01

Fokko force-pushed the fd-provider-tabular branch 2 times, most recently from b75b5a3 to 13c3d7aCompare May 17, 2022 14:56

eladkal reviewed May 17, 2022
View reviewed changes

Comment threadairflow/providers/tabular/provider.yaml Outdated

kaxil reviewed May 17, 2022
View reviewed changes

Fokko force-pushed the fd-provider-tabular branch 3 times, most recently from fa180e7 to 38afb69Compare May 17, 2022 18:20

Add Tabular provider
29e6511
Co-authored-by: Kyle Bendickson <kyle@tabular.io>

Fokko force-pushed the fd-provider-tabular branch from 38afb69 to 29e6511Compare May 17, 2022 18:52

kbendick reviewed May 19, 2022
View reviewed changes

Comment threadairflow/providers/tabular/example_dags/example_tabular.py Outdated

Cleanup command hash file
55346f7

Fokko added 2 commits June 16, 2022 18:12

Merge branch 'main' of github.com:apache/airflow into
1b9fc18
fd-provider-tabular

Regen the hash file
c3f515c

Merge branch 'main' into fd-provider-tabular
8e783e1

Fokko added 2 commits June 29, 2022 22:19

Merge branch 'main' of github.com:apache/airflow into fd-provider-tab…
032d765
…ular

Update providers
2bc51e7

Fokko added 3 commits June 29, 2022 23:13

Move to the new setup
5111a5d

Merge branch 'main' of github.com:apache/airflow into fd-provider-tab…
bd196be
…ular

Merge branch 'main' of github.com:apache/airflow into fd-provider-tab…
b529a36
…ular

eladkal reviewed Jun 30, 2022
View reviewed changes

Fokko added 7 commits July 5, 2022 16:20

Merge branch 'main' of github.com:apache/airflow into fd-provider-tab…
f6e8f07
…ular

Move example dag
287bf8c

Update docs
af93366

Add link
b963d20

Merge branch 'main' of github.com:apache/airflow into fd-provider-tab…
5c1fa56
…ular

Fix the doc
7075b91

Merge branch 'main' of github.com:apache/airflow into fd-provider-tab…
28a7fa1
…ular

josh-fell approved these changes Jul 6, 2022
View reviewed changes

potiuk approved these changes Jul 6, 2022
View reviewed changes

potiuk merged commit 10273e3 into apache:mainJul 6, 2022

Fokko deleted the fd-provider-tabular branch July 6, 2022 14:03

ephraimbuddy added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label Aug 15, 2022

		@@ -0,0 +1,48 @@
		# Licensed to the Apache Software Foundation (ASF) under one

	You can install this package on top of an existing Airflow 2.1+ installation via
	You can install this package on top of an existing Airflow 2.2+ installation via

		from airflow.providers.tabular.hooks.tabular import TabularHook


		class TestTabularHook(unittest.TestCase):

		@@ -0,0 +1,39 @@
		.. Licensed to the Apache Software Foundation (ASF) under one

Uh oh!

Conversation

Fokko commented May 13, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

kbendick left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eladkal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Fokko commented May 17, 2022

Uh oh!

Uh oh!

kaxil left a comment

Choose a reason for hiding this comment

Uh oh!

kaxil commented May 17, 2022

Uh oh!

Uh oh!

Fokko commented Jun 14, 2022

Uh oh!

potiuk commented Jun 14, 2022

Uh oh!

Fokko commented Jun 16, 2022

Uh oh!

potiuk commented Jun 16, 2022

Uh oh!

potiuk commented Jun 27, 2022

Uh oh!

Fokko commented Jun 29, 2022

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

josh-fell left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

potiuk left a comment

Choose a reason for hiding this comment

Uh oh!

kbendick commented Jul 6, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Fokko commented May 13, 2022•
edited
Loading