Add basic asyncio support

[dstandish]

This is meant to be sort of a hello world for asyncio support in airflow. It will be refined and extended in the future. E.g. probably we would add more config flexibility re the libraries, connect args etc. But it's good to start simple.

Anyway, ultimately, I think Airflow really needs to go in this direction: in the new REST API, in the new AIP-72 internal API server, in triggers, and ultimately, in the scheduler.

[tirkarthi]

Is this related to AIP-70?

https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-70+Migrating+to+asynchronous+programming

[dstandish]

Is this related to AIP-70?

https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-70+Migrating+to+asynchronous+programming

I think you prob know the answer. The AIP deals with asyncio, so yes strictly speaking related in that sense. But not "part of". That AIP has been in draft a long time not sure actively worked on. Anyway, I don't think what I'm doing here really requires an AIP.

[tirkarthi]

There is a similar draft PR open as POC for it though not active. So just wanted to confirm. Thanks for the details.

#36504

[pierrejeambrun]

Really exciting.

Do we want an @provide_async_session as a follow up ? (I think Ash wanted to kill that decorator at some point, even if I find it kind of useful actually)

[ashb]

@pierrejeambrun A FastAPI dep to give an async session would be good, but I think anything in the models etc should be passed an explicit session.

[potiuk]

NICE! Yeah. I really like to see it working with mysql compatibility and benchmarks 📦

(because of course MySQL is our beloved database)

[dstandish]

Really exciting.

Do we want an @provide_async_session as a follow up ? (I think Ash wanted to kill that decorator at some point, even if I find it kind of useful actually)

i would say let's wait and see how / where / when we need it

[omkar-foss]

Lovely! Thank you for adding this, will try to start using this soon :)

[omkar-foss]

@dstandish in lowest direct dep tests, we may need to limit minimum version for aiosqlite to 0.5.0 instead of 0.2.0 being used currently, as it doesn't have DatabaseError, hence leading to this error.

More info on this slack thread.