Adding ArangoDB Provider by pateash · Pull Request #22548 · apache/airflow

pateash · 2022-03-27T10:44:54Z

Description

Adding ArangoDB provider based on Python SDK https://github.com/ArangoDB-Community/python-arango

Users can create their own custom operators leveraging the ArangoDBHook directly
or building their operator on AQLOperator by providing result_processor method,

operator = AQLOperator(
task_id='aql_operator',
sql="FOR doc IN students " \
"RETURN doc",
dag=dag,
result_processor=lambda cursor: print([document["name"] for document in cursor])
)

Sensor can be implemented by SQL

sensor = AQLSensor(
task_id="aql_sensor",
sql="FOR doc IN students " \
"FILTER doc.name == 'judy' " \
"RETURN doc",
timeout=60,
poke_interval=10,
dag=dag,
)

eladkal

Over all LGTM

nice job @pateash

eladkal · 2022-03-27T12:14:48Z

+ :param arangodb_conn_id: Reference to :ref:`ArangoDB connection id <howto/connection:arangodb>`.
+ """
+
+ template_fields: Sequence[str] = ('sql',)


Should we add also template_ext?

yeah, make sense
added

eladkal · 2022-03-27T12:15:19Z

+ :param arangodb_db: Target ArangoDB name.
+ """
+
+ template_fields: Sequence[str] = ('sql',)


Should we add also template_ext?

potiuk · 2022-03-27T20:35:25Z

Can you fix static checks please.

pateash · 2022-03-29T13:35:59Z

pateash · 2022-03-29T13:36:31Z

eladkal · 2022-03-30T12:43:58Z

@potiuk can you take a look at the test failure?
AssertionError: List of expected installed packages and image content mismatch. Check /home/runner/work/airflow/airflow/scripts/ci/installed_providers.txt file.

I don't recall that when adding a new provider we need to edit the CI script

potiuk · 2022-03-30T12:52:32Z

@potiuk can you take a look at the test failure? AssertionError: List of expected installed packages and image content mismatch. Check /home/runner/work/airflow/airflow/scripts/ci/installed_providers.txt file.
I don't recall that when adding a new provider we need to edit the CI script

Not everything in providers has to be me :) - this test was added by @mik-laj actually: 621d17b

It looks like for some reason prodcution image produced in this build contains many more providers than it should

potiuk · 2022-03-30T12:59:56Z

Yeah: seems that for some reason it contains all providers:

docker run -it ghcr.io/apache/airflow/main/prod/python3.7:23b7d64b40261dcdcf73187464c6f09b67afcc57 bash
Unable to find image 'ghcr.io/apache/airflow/main/prod/python3.7:23b7d64b40261dcdcf73187464c6f09b67afcc57' locally
23b7d64b40261dcdcf73187464c6f09b67afcc57: Pulling from apache/airflow/main/prod/python3.7
c229119241af: Pull complete 5a3ae98ea812: Pull complete d6bab1fc351b: Pull complete f9cea33fb9b5: Pull complete 23c22d6e5b5d: Pull complete b21b38d9bc75: Pull complete e52ad88eda59: Pull complete 5938673019d8: Pull complete 10aec20ab867: Pull complete bfa0b2f2703d: Pull complete abea59e2f689: Pull complete ffd9264d5a4a: Pull complete ea7c97498e3e: Pull complete 4aed0971f3f7: Pull complete 8f85ceb1d546: Pull complete b6132f0f6227: Pull complete 83d18601cc4f: Pull complete 88748a7a2d95: Pull complete 4f4fb700ef54: Pull complete Digest: sha256:bf5da3a686feab47684c036de99a492c3d024fabd4e7a3b69ea9d63ce941b8c8
Status: Downloaded newer image for ghcr.io/apache/airflow/main/prod/python3.7:23b7d64b40261dcdcf73187464c6f09b67afcc57
airflow@54c94bf4e3b9:/opt/airflow$ airflow providers list
package_name | description | version
==========================================+=================================================================================================+========
apache-airflow-providers-airbyte | Airbyte https://airbyte.io/ | 2.1.4 apache-airflow-providers-alibaba | Alibaba Cloud integration (including Alibaba Cloud https://www.alibabacloud.com//) | 1.1.1 apache-airflow-providers-amazon | Amazon integration (including Amazon Web Services (AWS) https://aws.amazon.com/) | 3.2.0 apache-airflow-providers-apache-beam | Apache Beam https://beam.apache.org/ | 3.3.0 apache-airflow-providers-apache-cassandra | Apache Cassandra http://cassandra.apache.org/ | 2.1.3 apache-airflow-providers-apache-drill | Apache Drill https://drill.apache.org/ | 1.0.4 apache-airflow-providers-apache-druid | Apache Druid https://druid.apache.org/ | 2.3.3 apache-airflow-providers-apache-hdfs | Hadoop Distributed File System (HDFS) https://hadoop.apache.org/docs/r1.2.1/hdfsdesign.html | 2.2.3 | and WebHDFS https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html | apache-airflow-providers-apache-hive | Apache Hive https://hive.apache.org/ | 2.3.2 apache-airflow-providers-apache-kylin | Apache Kylin https://kylin.apache.org/ | 2.0.4 apache-airflow-providers-apache-livy | Apache Livy https://livy.apache.org/ | 2.2.2 apache-airflow-providers-apache-pig | Apache Pig https://pig.apache.org/ | 2.0.4 apache-airflow-providers-apache-pinot | Apache Pinot https://pinot.apache.org/ | 2.0.4 apache-airflow-providers-apache-spark | Apache Spark https://spark.apache.org/ | 2.1.3 apache-airflow-providers-apache-sqoop | Apache Sqoop https://sqoop.apache.org/ | 2.1.3 apache-airflow-providers-arangodb | ArangoDB https://www.arangodb.com/ | 1.0.0 apache-airflow-providers-asana | Asana https://app.asana.com/ | 1.1.3 apache-airflow-providers-celery | Celery http://www.celeryproject.org/ | 2.1.3 apache-airflow-providers-cloudant | IBM Cloudant https://www.ibm.com/cloud/cloudant | 2.0.4 apache-airflow-providers-cncf-kubernetes | Kubernetes https://kubernetes.io/ | 3.1.2 apache-airflow-providers-databricks | Databricks https://databricks.com/ | 2.5.0 apache-airflow-providers-datadog | Datadog https://www.datadoghq.com/ | 2.0.4 apache-airflow-providers-dbt-cloud | dbt Cloud https://www.getdbt.com/product/what-is-dbt/) | 1.0.2 apache-airflow-providers-dingding | Dingding https://oapi.dingtalk.com/ | 2.0.4 apache-airflow-providers-discord | Discord https://discordapp.com/ | 2.0.4 apache-airflow-providers-docker | Docker https://docs.docker.com/install/ | 2.5.2 apache-airflow-providers-elasticsearch | Elasticsearch https://www.elastic.co/elasticsearch | 3.0.2 apache-airflow-providers-exasol | Exasol https://docs.exasol.com/home.htm | 2.1.3 apache-airflow-providers-facebook | Facebook Ads http://business.facebook.com/ | 2.2.3 apache-airflow-providers-ftp | File Transfer Protocol (FTP) https://tools.ietf.org/html/rfc114 | 2.1.2 apache-airflow-providers-github | Github https://www.github.com/ | 1.0.3 apache-airflow-providers-google | Google services including: | 6.7.0 | | | - Google Ads https://ads.google.com/ | | - Google Cloud (GCP) https://cloud.google.com/ | | - Google Firebase https://firebase.google.com/ | | - Google LevelDB https://github.com/google/leveldb/ | | - Google Marketing Platform https://marketingplatform.google.com/ | | - Google Workspace https://workspace.google.pl/ (formerly Google Suite) | apache-airflow-providers-grpc | gRPC https://grpc.io/ | 2.0.4 apache-airflow-providers-hashicorp | Hashicorp including Hashicorp Vault https://www.vaultproject.io/ | 2.1.4 apache-airflow-providers-http | Hypertext Transfer Protocol (HTTP) https://www.w3.org/Protocols/ | 2.1.2 apache-airflow-providers-imap | Internet Message Access Protocol (IMAP) https://tools.ietf.org/html/rfc3501 | 2.2.3 apache-airflow-providers-influxdb | InfluxDB https://www.influxdata.com/ | 1.1.3 apache-airflow-providers-jdbc | Java Database Connectivity (JDBC) https://docs.oracle.com/javase/8/docs/technotes/guides/jdbc/ | 2.1.3 apache-airflow-providers-jenkins | Jenkins https://jenkins.io/ | 2.0.7 apache-airflow-providers-jira | Atlassian Jira https://www.atlassian.com/ | 2.0.4 apache-airflow-providers-microsoft-azure | Microsoft Azure https://azure.microsoft.com/ | 3.7.2 apache-airflow-providers-microsoft-mssql | Microsoft SQL Server (MSSQL) https://www.microsoft.com/en-us/sql-server/sql-server-downloads | 2.1.3 apache-airflow-providers-microsoft-psrp | This package provides remote execution capabilities via the | 1.1.3 | PowerShell Remoting Protocol (PSRP) | | https://docs.microsoft.com/en-us/openspecs/windowsprotocols/ms-psrp/ | apache-airflow-providers-microsoft-winrm | Windows Remote Management (WinRM) https://docs.microsoft.com/en-us/windows/win32/winrm/portal | 2.0.5 apache-airflow-providers-mongo | MongoDB https://www.mongodb.com/what-is-mongodb | 2.3.3 apache-airflow-providers-mysql | MySQL https://www.mysql.com/products/ | 2.2.3 apache-airflow-providers-neo4j | Neo4j https://neo4j.com/ | 2.1.3 apache-airflow-providers-odbc | ODBC https://github.com/mkleehammer/pyodbc/wiki | 2.0.4 apache-airflow-providers-openfaas | OpenFaaS https://www.openfaas.com/ | 2.0.3 apache-airflow-providers-opsgenie | Opsgenie https://www.opsgenie.com/ | 3.0.3 apache-airflow-providers-oracle | Oracle https://www.oracle.com/en/database/ | 2.2.3 apache-airflow-providers-pagerduty | Pagerduty https://www.pagerduty.com/ | 2.1.3 apache-airflow-providers-papermill | Papermill https://github.com/nteract/papermill | 2.2.3 apache-airflow-providers-plexus | Plexus https://plexus.corescientific.com/ | 2.0.4 apache-airflow-providers-postgres | PostgreSQL https://www.postgresql.org/ | 4.1.0 apache-airflow-providers-presto | Presto https://prestodb.github.io/ | 2.1.2 apache-airflow-providers-qubole | Qubole https://www.qubole.com/ | 2.1.3 apache-airflow-providers-redis | Redis https://redis.io/ | 2.0.4 apache-airflow-providers-salesforce | Salesforce https://www.salesforce.com/ | 3.4.3 apache-airflow-providers-samba | Samba https://www.samba.org/ | 3.0.4 apache-airflow-providers-segment | Segment https://segment.com/ | 2.0.4 apache-airflow-providers-sendgrid | Sendgrid https://sendgrid.com/ | 2.0.4 apache-airflow-providers-sftp | SSH File Transfer Protocol (SFTP) https://tools.ietf.org/wg/secsh/draft-ietf-secsh-filexfer/ | 2.5.2 apache-airflow-providers-singularity | Singularity https://sylabs.io/guides/latest/user-guide/ | 2.0.4 apache-airflow-providers-slack | Slack https://slack.com/ | 4.2.3 apache-airflow-providers-snowflake | Snowflake https://www.snowflake.com/ | 2.6.0 apache-airflow-providers-sqlite | SQLite https://www.sqlite.org/ | 2.1.3 apache-airflow-providers-ssh | Secure Shell (SSH) https://tools.ietf.org/html/rfc4251 | 2.4.3 apache-airflow-providers-tableau | Tableau https://www.tableau.com/ | 2.1.7 apache-airflow-providers-telegram | Telegram https://telegram.org/ | 2.0.4 apache-airflow-providers-trino | Trino https://trino.io/ | 2.1.2 apache-airflow-providers-vertica | Vertica https://www.vertica.com/ | 2.1.3 apache-airflow-providers-yandex | Yandex including Yandex.Cloud https://cloud.yandex.com/ | 2.2.3 apache-airflow-providers-zendesk | Zendesk https://www.zendesk.com/ | 3.0.3

potiuk · 2022-03-30T13:07:07Z

This is VERY strange as it seems that when the image was built, it actually used only a small subset (as expected):

#64 1.486 Force re-installing airflow and providers from local files with eager upgrade
#64 1.486 #64 2.925 Looking in links: file:///docker-context-files
#64 2.937 Processing /docker-context-files/apache_airflow_providers_amazon-3.2.0.dev0-py3-none-any.whl
#64 2.952 Processing /docker-context-files/apache_airflow_providers_celery-2.1.3.dev0-py3-none-any.whl
#64 2.959 Processing /docker-context-files/apache_airflow_providers_cncf_kubernetes-3.1.2.dev0-py3-none-any.whl
#64 2.966 Processing /docker-context-files/apache_airflow_providers_docker-2.5.2.dev0-py3-none-any.whl
#64 2.973 Processing /docker-context-files/apache_airflow_providers_elasticsearch-3.0.2.dev0-py3-none-any.whl
#64 2.980 Processing /docker-context-files/apache_airflow_providers_ftp-2.1.2.dev0-py3-none-any.whl
#64 2.988 Processing /docker-context-files/apache_airflow_providers_google-6.7.0.dev0-py3-none-any.whl
#64 2.997 Processing /docker-context-files/apache_airflow_providers_grpc-2.0.4.dev0-py3-none-any.whl
#64 3.004 Processing /docker-context-files/apache_airflow_providers_hashicorp-2.1.4.dev0-py3-none-any.whl
#64 3.011 Processing /docker-context-files/apache_airflow_providers_http-2.1.2.dev0-py3-none-any.whl
#64 3.018 Processing /docker-context-files/apache_airflow_providers_imap-2.2.3.dev0-py3-none-any.whl
#64 3.026 Processing /docker-context-files/apache_airflow_providers_microsoft_azure-3.7.2.dev0-py3-none-any.whl
#64 3.033 Processing /docker-context-files/apache_airflow_providers_mysql-2.2.3.dev0-py3-none-any.whl
#64 3.040 Processing /docker-context-files/apache_airflow_providers_odbc-2.0.4.dev0-py3-none-any.whl
#64 3.047 Processing /docker-context-files/apache_airflow_providers_postgres-4.1.0.dev0-py3-none-any.whl
#64 3.054 Processing /docker-context-files/apache_airflow_providers_redis-2.0.4.dev0-py3-none-any.whl
#64 3.062 Processing /docker-context-files/apache_airflow_providers_sendgrid-2.0.4.dev0-py3-none-any.whl
#64 3.069 Processing /docker-context-files/apache_airflow_providers_sftp-2.5.2.dev0-py3-none-any.whl
#64 3.076 Processing /docker-context-files/apache_airflow_providers_slack-4.2.3.dev0-py3-none-any.whl
#64 3.083 Processing /docker-context-files/apache_airflow_providers_sqlite-2.1.3.dev0-py3-none-any.whl
#64 3.090 Processing /docker-context-files/apache_airflow_providers_ssh-2.4.3.dev0-py3-none-any.whl
#64 3.223 Processing /docker-context-files/apache_airflow-2.3.0.dev0-py3-none-any.whl

potiuk · 2022-03-30T13:07:38Z

Let me rebase and see it happening again :)

eladkal · 2022-03-30T13:36:50Z

I don't recall we had such issue when GitHub provider was added (and it was after 621d17b )

potiuk · 2022-03-30T13:47:43Z

I don't recall we had such issue when GitHub provider was added (and it was after 621d17b )

Me neither. It basicallly SHOUD NOT happen :D. Yet it seems it did again

potiuk · 2022-03-30T13:58:35Z

OK. I know what causes it but I do not know why it happens yet. When PROD build image is prepared we prepare "airflow" package so that it can be installed there from latest sources. But for SOME reason, it contains "all" providers as well. not only airflow. I do not know where it came from yet. But It proves the tests from @mik-laj are useful to catch it.

potiuk · 2022-03-30T13:59:39Z

I actually think it could come from the new setuptools release https://pypi.org/project/setuptools/61.2.0/

potiuk · 2022-03-30T14:52:04Z

Still puzzled :) but I am getting closer to solve it

pateash · 2022-03-30T15:01:35Z

thanks @potiuk.

potiuk · 2022-03-31T13:54:37Z

Rebased it @pateash -> I have high hopes for #22649 to either fix it or make it easier to understand where it came from

potiuk · 2022-04-01T08:38:57Z

Hi maintainer of python-arango here. I've removed the dependency. Please try again with release version 7.3.2. Thanks.

Cool. Thanks! @pateash -> can you add >=7.3.2 to our requirements please ?

pateash · 2022-04-01T09:59:40Z

voila 🥳,
It worked.
Thanks @joowani

potiuk

LGTM :) - @eladkal ?

github-actions · 2022-04-01T11:20:56Z

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

eladkal · 2022-04-01T11:32:37Z

I'll take a look later today

eladkal

LGTM

eladkal · 2022-04-03T07:35:17Z

 +---------------------+-----------------------------------------------------+-------------------------------------------+
 | trino | ``pip install 'apache-airflow[trino]'`` | All Trino related operators & hooks |
 +---------------------+-----------------------------------------------------+-------------------------------------------+
+| arangodb | ``pip install 'apache-airflow[arangodb]'`` | ArangoDB operators, sensors and hook |


I think this list is sorted alphabetically?

potiuk · 2022-04-03T13:25:03Z

🎉 🎉 🎉 🎉 🎉 🎉 🎉

pateash requested review from kaxil, mik-laj and potiuk as code owners March 27, 2022 10:44

boring-cyborgBot added area:dev-tools area:providers kind:documentation labels Mar 27, 2022

pateash changed the title ~~Add Arango hook~~ WIP: Add Arango hook Mar 27, 2022

eladkal reviewed Mar 27, 2022
View reviewed changes

pateash changed the title ~~WIP: Add Arango hook~~ Adding ArangoDB ProviderMar 29, 2022

pateash requested a review from eladkal March 29, 2022 17:30

pateash closed this Mar 29, 2022

pateash reopened this Mar 29, 2022

pateash force-pushed the airflow-17778 branch from df1019b to 8737f4bCompare March 29, 2022 22:54

potiuk force-pushed the airflow-17778 branch from 23b7d64 to 19966f3Compare March 30, 2022 13:07

potiuk force-pushed the airflow-17778 branch from 19966f3 to 7d40c88Compare March 31, 2022 13:53

pateash force-pushed the airflow-17778 branch from de58a24 to ee6e46eCompare April 1, 2022 08:52

potiuk approved these changes Apr 1, 2022
View reviewed changes

github-actionsBot added the full tests needed We need to run full set of tests for this PR to merge label Apr 1, 2022

pateash added 17 commits April 1, 2022 13:36

17778 - Adding initial files
6c3e261

17778 - hook and operator added
1bd35a8

17778 - sensor added
e37de29

airflow-17778 example file
c0478ef

airflow-17778 - fixing arguments
4b04bb0

airflow-17778 - documentation added
9b3610b

17778 - tests for hook added
6ebcc9c

17778 - tests added
52b6917

airflow-17778 - fixing black lints
6bc97a0

airflow-17778 - template file added
899b399

airflow-17778 - added utility methods
e766b70

21720 - pre-commit fixes
8c34e63

airflow-17778 - documents added
ec99b81

airflow-17778 - precommit fixes
975a088

airflow-17778 - precommit fixes
bec2ea4

airflow-17778 - tests implemented
f5862e6

airflow-17778 - updating python-arango dependency
22fcaef

pateash force-pushed the airflow-17778 branch from f8ecf67 to 22fcaefCompare April 1, 2022 12:36

eladkal approved these changes Apr 3, 2022
View reviewed changes

eladkal merged commit c758c76 into apache:mainApr 3, 2022

ephraimbuddy added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label Apr 11, 2022

pateash deleted the airflow-17778 branch May 19, 2022 14:30

Uh oh!

Conversation

pateash commented Mar 27, 2022

Description

Uh oh!

eladkal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

eladkalMar 27, 2022

Choose a reason for hiding this comment

Uh oh!

pateashMar 29, 2022

Choose a reason for hiding this comment

Uh oh!

eladkalMar 27, 2022

Choose a reason for hiding this comment

Uh oh!

pateashMar 29, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

potiuk commented Mar 27, 2022

Uh oh!

pateash commented Mar 29, 2022

Uh oh!

pateash commented Mar 29, 2022

Uh oh!

eladkal commented Mar 30, 2022

Uh oh!

potiuk commented Mar 30, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

potiuk commented Mar 30, 2022

Uh oh!

potiuk commented Mar 30, 2022

Uh oh!

potiuk commented Mar 30, 2022

Uh oh!

eladkal commented Mar 30, 2022

Uh oh!

potiuk commented Mar 30, 2022

Uh oh!

potiuk commented Mar 30, 2022

Uh oh!

potiuk commented Mar 30, 2022

Uh oh!

potiuk commented Mar 30, 2022

Uh oh!

pateash commented Mar 30, 2022

Uh oh!

potiuk commented Mar 31, 2022

Uh oh!

potiuk commented Apr 1, 2022

Uh oh!

pateash commented Apr 1, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

potiuk left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Apr 1, 2022

Uh oh!

eladkal commented Apr 1, 2022

Uh oh!

eladkal left a comment

Choose a reason for hiding this comment

Uh oh!

eladkalApr 3, 2022

Choose a reason for hiding this comment

Uh oh!

potiuk commented Apr 3, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

pateashMar 29, 2022•
edited
Loading

potiuk commented Mar 30, 2022•
edited
Loading

pateash commented Apr 1, 2022•
edited
Loading