Remove Openlineage from list of pre-installed providers

[kaxil]

Not everyone will want OL installed.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[potiuk]

I guess we should add newsfragment about it - it changes the behaviour of pip install apache-airflow

[kaxil]

I guess we should add newsfragment about it - it changes the behaviour of pip install apache-airflow

OL wasn't installed in 2.10.x though

[potiuk]

Newsfragment - when it comes to SQLite at least.

Also it might break lowest dependency tests for Airflow core - I believe one reason I added it was that Airflow core was at that time some tests or even all tests failed. Maybe things changed since, but I think not.

Let's see how it works, but:

1. in case airflow-core has some imports, they should be made optional if possible.
2. In case tests have some imports - we can move the OpenLineage provider to "dev" dependency group.

[potiuk]

Yeah - the first failure is aiosqlite not importable in airflow tests - this can be solved by adding apache-airflow-providers-sqlite to dev dependency group.

[potiuk]

(And let's see what fails next)

[eladkal]

Correct me if I am wrong but we removed SequentialExecutor and right now Sqlite can not work with LocalExecutor so the breaking change already happened isn't it? (We had a plab for Sqlite to work with LocalExecutor but I don't remember it was done)

[kaxil]

Correct me if I am wrong but we removed SequentialExecutor and right now Sqlite can not work with LocalExecutor so the breaking change already happened isn't it? (We had a plab for Sqlite to work with LocalExecutor but I don't remember it was done)

Yeah that's done. But even that does not need anything in the Sqlite provider to work. It just has:

airflow/providers/sqlite/pyproject.toml

Lines 59 to 63 in 89df9fc

	dependencies = [
	"apache-airflow>=2.9.0",
	"aiosqlite>=0.20.0",
	"apache-airflow-providers-common-sql>=1.20.0",
	]

[kaxil]

Separating Sqlite removal to separate PR

[ashb]

and right now Sqlite can not work with LocalExecutor

@eladkal Not true anymore, I fixed it a few months back.

[kaxil]

in #44839

[potiuk]

The sqlite provider was never technically needed to use SQLite as meta-data. Those two have nothing to do with each other - provider simply create a DBAPI Hook so that SQLite (but not the one that has airflow metadata) can be used via common.sql operators.

The SQlite database is part of stdlib and the only depedency extra it has is "aiosqlite" -> Which BTW I think this is somewhat wrong - aiosqlite should be added to the core. I commented it in #48950

[kaxil]

Not everyone will want OL installed.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[potiuk]

I guess we should add newsfragment about it - it changes the behaviour of pip install apache-airflow

[kaxil]

I guess we should add newsfragment about it - it changes the behaviour of pip install apache-airflow

OL wasn't installed in 2.10.x though

[potiuk]

Newsfragment - when it comes to SQLite at least.

Also it might break lowest dependency tests for Airflow core - I believe one reason I added it was that Airflow core was at that time some tests or even all tests failed. Maybe things changed since, but I think not.

Let's see how it works, but:

1. in case airflow-core has some imports, they should be made optional if possible.
2. In case tests have some imports - we can move the OpenLineage provider to "dev" dependency group.

[potiuk]

Yeah - the first failure is aiosqlite not importable in airflow tests - this can be solved by adding apache-airflow-providers-sqlite to dev dependency group.

[potiuk]

(And let's see what fails next)

[eladkal]

Correct me if I am wrong but we removed SequentialExecutor and right now Sqlite can not work with LocalExecutor so the breaking change already happened isn't it? (We had a plab for Sqlite to work with LocalExecutor but I don't remember it was done)

[kaxil]

Correct me if I am wrong but we removed SequentialExecutor and right now Sqlite can not work with LocalExecutor so the breaking change already happened isn't it? (We had a plab for Sqlite to work with LocalExecutor but I don't remember it was done)

Yeah that's done. But even that does not need anything in the Sqlite provider to work. It just has:

airflow/providers/sqlite/pyproject.toml

Lines 59 to 63 in 89df9fc

	dependencies = [
	"apache-airflow>=2.9.0",
	"aiosqlite>=0.20.0",
	"apache-airflow-providers-common-sql>=1.20.0",
	]

[kaxil]

Separating Sqlite removal to separate PR

[ashb]

and right now Sqlite can not work with LocalExecutor

@eladkal Not true anymore, I fixed it a few months back.

[kaxil]

in #44839

[potiuk]

The sqlite provider was never technically needed to use SQLite as meta-data. Those two have nothing to do with each other - provider simply create a DBAPI Hook so that SQLite (but not the one that has airflow metadata) can be used via common.sql operators.

The SQlite database is part of stdlib and the only depedency extra it has is "aiosqlite" -> Which BTW I think this is somewhat wrong - aiosqlite should be added to the core. I commented it in #48950

[kaxil]

Not everyone will want OL installed.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[potiuk]

I guess we should add newsfragment about it - it changes the behaviour of pip install apache-airflow

[kaxil]

I guess we should add newsfragment about it - it changes the behaviour of pip install apache-airflow

OL wasn't installed in 2.10.x though

[potiuk]

Newsfragment - when it comes to SQLite at least.

Also it might break lowest dependency tests for Airflow core - I believe one reason I added it was that Airflow core was at that time some tests or even all tests failed. Maybe things changed since, but I think not.

Let's see how it works, but:

1. in case airflow-core has some imports, they should be made optional if possible.
2. In case tests have some imports - we can move the OpenLineage provider to "dev" dependency group.

[potiuk]

Yeah - the first failure is aiosqlite not importable in airflow tests - this can be solved by adding apache-airflow-providers-sqlite to dev dependency group.

[potiuk]

(And let's see what fails next)

[eladkal]

Correct me if I am wrong but we removed SequentialExecutor and right now Sqlite can not work with LocalExecutor so the breaking change already happened isn't it? (We had a plab for Sqlite to work with LocalExecutor but I don't remember it was done)

[kaxil]

Correct me if I am wrong but we removed SequentialExecutor and right now Sqlite can not work with LocalExecutor so the breaking change already happened isn't it? (We had a plab for Sqlite to work with LocalExecutor but I don't remember it was done)

Yeah that's done. But even that does not need anything in the Sqlite provider to work. It just has:

airflow/providers/sqlite/pyproject.toml

Lines 59 to 63 in 89df9fc

	dependencies = [
	"apache-airflow>=2.9.0",
	"aiosqlite>=0.20.0",
	"apache-airflow-providers-common-sql>=1.20.0",
	]

[kaxil]

Separating Sqlite removal to separate PR

[ashb]

and right now Sqlite can not work with LocalExecutor

@eladkal Not true anymore, I fixed it a few months back.

[kaxil]

in #44839

[potiuk]

The sqlite provider was never technically needed to use SQLite as meta-data. Those two have nothing to do with each other - provider simply create a DBAPI Hook so that SQLite (but not the one that has airflow metadata) can be used via common.sql operators.

The SQlite database is part of stdlib and the only depedency extra it has is "aiosqlite" -> Which BTW I think this is somewhat wrong - aiosqlite should be added to the core. I commented it in #48950

[kaxil]

Not everyone will want OL installed.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[potiuk]

I guess we should add newsfragment about it - it changes the behaviour of pip install apache-airflow

[kaxil]

I guess we should add newsfragment about it - it changes the behaviour of pip install apache-airflow

OL wasn't installed in 2.10.x though

[potiuk]

Newsfragment - when it comes to SQLite at least.

Also it might break lowest dependency tests for Airflow core - I believe one reason I added it was that Airflow core was at that time some tests or even all tests failed. Maybe things changed since, but I think not.

Let's see how it works, but:

1. in case airflow-core has some imports, they should be made optional if possible.
2. In case tests have some imports - we can move the OpenLineage provider to "dev" dependency group.

[potiuk]

Yeah - the first failure is aiosqlite not importable in airflow tests - this can be solved by adding apache-airflow-providers-sqlite to dev dependency group.

[potiuk]

(And let's see what fails next)

[eladkal]

Correct me if I am wrong but we removed SequentialExecutor and right now Sqlite can not work with LocalExecutor so the breaking change already happened isn't it? (We had a plab for Sqlite to work with LocalExecutor but I don't remember it was done)

[kaxil]

Correct me if I am wrong but we removed SequentialExecutor and right now Sqlite can not work with LocalExecutor so the breaking change already happened isn't it? (We had a plab for Sqlite to work with LocalExecutor but I don't remember it was done)

Yeah that's done. But even that does not need anything in the Sqlite provider to work. It just has:

airflow/providers/sqlite/pyproject.toml

Lines 59 to 63 in 89df9fc

	dependencies = [
	"apache-airflow>=2.9.0",
	"aiosqlite>=0.20.0",
	"apache-airflow-providers-common-sql>=1.20.0",
	]

[kaxil]

Separating Sqlite removal to separate PR

[ashb]

and right now Sqlite can not work with LocalExecutor

@eladkal Not true anymore, I fixed it a few months back.

[kaxil]

in #44839

[potiuk]

The sqlite provider was never technically needed to use SQLite as meta-data. Those two have nothing to do with each other - provider simply create a DBAPI Hook so that SQLite (but not the one that has airflow metadata) can be used via common.sql operators.

The SQlite database is part of stdlib and the only depedency extra it has is "aiosqlite" -> Which BTW I think this is somewhat wrong - aiosqlite should be added to the core. I commented it in #48950

[kaxil]

Not everyone will want OL installed.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[potiuk]

I guess we should add newsfragment about it - it changes the behaviour of pip install apache-airflow

[kaxil]

I guess we should add newsfragment about it - it changes the behaviour of pip install apache-airflow

OL wasn't installed in 2.10.x though

[potiuk]

Newsfragment - when it comes to SQLite at least.

Also it might break lowest dependency tests for Airflow core - I believe one reason I added it was that Airflow core was at that time some tests or even all tests failed. Maybe things changed since, but I think not.

Let's see how it works, but:

1. in case airflow-core has some imports, they should be made optional if possible.
2. In case tests have some imports - we can move the OpenLineage provider to "dev" dependency group.

[potiuk]

Yeah - the first failure is aiosqlite not importable in airflow tests - this can be solved by adding apache-airflow-providers-sqlite to dev dependency group.

[potiuk]

(And let's see what fails next)

[eladkal]

Correct me if I am wrong but we removed SequentialExecutor and right now Sqlite can not work with LocalExecutor so the breaking change already happened isn't it? (We had a plab for Sqlite to work with LocalExecutor but I don't remember it was done)

[kaxil]

Correct me if I am wrong but we removed SequentialExecutor and right now Sqlite can not work with LocalExecutor so the breaking change already happened isn't it? (We had a plab for Sqlite to work with LocalExecutor but I don't remember it was done)

Yeah that's done. But even that does not need anything in the Sqlite provider to work. It just has:

airflow/providers/sqlite/pyproject.toml

Lines 59 to 63 in 89df9fc

	dependencies = [
	"apache-airflow>=2.9.0",
	"aiosqlite>=0.20.0",
	"apache-airflow-providers-common-sql>=1.20.0",
	]

[kaxil]

Separating Sqlite removal to separate PR

[ashb]

and right now Sqlite can not work with LocalExecutor

@eladkal Not true anymore, I fixed it a few months back.

[kaxil]

in #44839

[potiuk]

The sqlite provider was never technically needed to use SQLite as meta-data. Those two have nothing to do with each other - provider simply create a DBAPI Hook so that SQLite (but not the one that has airflow metadata) can be used via common.sql operators.

The SQlite database is part of stdlib and the only depedency extra it has is "aiosqlite" -> Which BTW I think this is somewhat wrong - aiosqlite should be added to the core. I commented it in #48950

[kaxil]

Not everyone will want OL installed.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[potiuk]

I guess we should add newsfragment about it - it changes the behaviour of pip install apache-airflow

[kaxil]

I guess we should add newsfragment about it - it changes the behaviour of pip install apache-airflow

OL wasn't installed in 2.10.x though

[potiuk]

Newsfragment - when it comes to SQLite at least.

Also it might break lowest dependency tests for Airflow core - I believe one reason I added it was that Airflow core was at that time some tests or even all tests failed. Maybe things changed since, but I think not.

Let's see how it works, but:

1. in case airflow-core has some imports, they should be made optional if possible.
2. In case tests have some imports - we can move the OpenLineage provider to "dev" dependency group.

[potiuk]

Yeah - the first failure is aiosqlite not importable in airflow tests - this can be solved by adding apache-airflow-providers-sqlite to dev dependency group.

[potiuk]

(And let's see what fails next)

[eladkal]

Correct me if I am wrong but we removed SequentialExecutor and right now Sqlite can not work with LocalExecutor so the breaking change already happened isn't it? (We had a plab for Sqlite to work with LocalExecutor but I don't remember it was done)

[kaxil]

Correct me if I am wrong but we removed SequentialExecutor and right now Sqlite can not work with LocalExecutor so the breaking change already happened isn't it? (We had a plab for Sqlite to work with LocalExecutor but I don't remember it was done)

Yeah that's done. But even that does not need anything in the Sqlite provider to work. It just has:

airflow/providers/sqlite/pyproject.toml

Lines 59 to 63 in 89df9fc

	dependencies = [
	"apache-airflow>=2.9.0",
	"aiosqlite>=0.20.0",
	"apache-airflow-providers-common-sql>=1.20.0",
	]

[kaxil]

Separating Sqlite removal to separate PR

[ashb]

and right now Sqlite can not work with LocalExecutor

@eladkal Not true anymore, I fixed it a few months back.

[kaxil]

in #44839

[potiuk]

The sqlite provider was never technically needed to use SQLite as meta-data. Those two have nothing to do with each other - provider simply create a DBAPI Hook so that SQLite (but not the one that has airflow metadata) can be used via common.sql operators.

The SQlite database is part of stdlib and the only depedency extra it has is "aiosqlite" -> Which BTW I think this is somewhat wrong - aiosqlite should be added to the core. I commented it in #48950

[kaxil]

Not everyone will want OL installed.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[potiuk]

I guess we should add newsfragment about it - it changes the behaviour of pip install apache-airflow

[kaxil]

I guess we should add newsfragment about it - it changes the behaviour of pip install apache-airflow

OL wasn't installed in 2.10.x though

[potiuk]

Newsfragment - when it comes to SQLite at least.

Also it might break lowest dependency tests for Airflow core - I believe one reason I added it was that Airflow core was at that time some tests or even all tests failed. Maybe things changed since, but I think not.

Let's see how it works, but:

1. in case airflow-core has some imports, they should be made optional if possible.
2. In case tests have some imports - we can move the OpenLineage provider to "dev" dependency group.

[potiuk]

Yeah - the first failure is aiosqlite not importable in airflow tests - this can be solved by adding apache-airflow-providers-sqlite to dev dependency group.

[potiuk]

(And let's see what fails next)

[eladkal]

Correct me if I am wrong but we removed SequentialExecutor and right now Sqlite can not work with LocalExecutor so the breaking change already happened isn't it? (We had a plab for Sqlite to work with LocalExecutor but I don't remember it was done)

[kaxil]

Correct me if I am wrong but we removed SequentialExecutor and right now Sqlite can not work with LocalExecutor so the breaking change already happened isn't it? (We had a plab for Sqlite to work with LocalExecutor but I don't remember it was done)

Yeah that's done. But even that does not need anything in the Sqlite provider to work. It just has:

airflow/providers/sqlite/pyproject.toml

Lines 59 to 63 in 89df9fc

	dependencies = [
	"apache-airflow>=2.9.0",
	"aiosqlite>=0.20.0",
	"apache-airflow-providers-common-sql>=1.20.0",
	]

[kaxil]

Separating Sqlite removal to separate PR

[ashb]

and right now Sqlite can not work with LocalExecutor

@eladkal Not true anymore, I fixed it a few months back.

[kaxil]

in #44839

[potiuk]

The sqlite provider was never technically needed to use SQLite as meta-data. Those two have nothing to do with each other - provider simply create a DBAPI Hook so that SQLite (but not the one that has airflow metadata) can be used via common.sql operators.

The SQlite database is part of stdlib and the only depedency extra it has is "aiosqlite" -> Which BTW I think this is somewhat wrong - aiosqlite should be added to the core. I commented it in #48950

[kaxil]

Not everyone will want OL installed.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[potiuk]

I guess we should add newsfragment about it - it changes the behaviour of pip install apache-airflow

[kaxil]

I guess we should add newsfragment about it - it changes the behaviour of pip install apache-airflow

OL wasn't installed in 2.10.x though

[potiuk]

Newsfragment - when it comes to SQLite at least.

Also it might break lowest dependency tests for Airflow core - I believe one reason I added it was that Airflow core was at that time some tests or even all tests failed. Maybe things changed since, but I think not.

Let's see how it works, but:

1. in case airflow-core has some imports, they should be made optional if possible.
2. In case tests have some imports - we can move the OpenLineage provider to "dev" dependency group.

[potiuk]

Yeah - the first failure is aiosqlite not importable in airflow tests - this can be solved by adding apache-airflow-providers-sqlite to dev dependency group.

[potiuk]

(And let's see what fails next)

[eladkal]

Correct me if I am wrong but we removed SequentialExecutor and right now Sqlite can not work with LocalExecutor so the breaking change already happened isn't it? (We had a plab for Sqlite to work with LocalExecutor but I don't remember it was done)

[kaxil]

Correct me if I am wrong but we removed SequentialExecutor and right now Sqlite can not work with LocalExecutor so the breaking change already happened isn't it? (We had a plab for Sqlite to work with LocalExecutor but I don't remember it was done)

Yeah that's done. But even that does not need anything in the Sqlite provider to work. It just has:

airflow/providers/sqlite/pyproject.toml

Lines 59 to 63 in 89df9fc

	dependencies = [
	"apache-airflow>=2.9.0",
	"aiosqlite>=0.20.0",
	"apache-airflow-providers-common-sql>=1.20.0",
	]

[kaxil]

Separating Sqlite removal to separate PR

[ashb]

and right now Sqlite can not work with LocalExecutor

@eladkal Not true anymore, I fixed it a few months back.

[kaxil]

in #44839

[potiuk]

The sqlite provider was never technically needed to use SQLite as meta-data. Those two have nothing to do with each other - provider simply create a DBAPI Hook so that SQLite (but not the one that has airflow metadata) can be used via common.sql operators.

The SQlite database is part of stdlib and the only depedency extra it has is "aiosqlite" -> Which BTW I think this is somewhat wrong - aiosqlite should be added to the core. I commented it in #48950