Uh oh!
There was an error while loading. Please reload this page.
Allow keeping PostgresHook SQLAlchemy engines on psycopg2 - #72000
Allow keeping PostgresHook SQLAlchemy engines on psycopg2#72000sehajsandhu123 wants to merge 4 commits into
Conversation
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
|
Dev-iL
left a comment
There was a problem hiding this comment.
The implementation looks correct and the tests are helpful.
I'm conflicted about whether we want to support this functionality. On the one hand, we should be careful about adding workarounds for the behavior of other libraries. On the other hand, because this an explicit, non-default escape hatch for users migrating to ppg3, it might be justified.
My main request is related to regression tests: please ensure CI reproduces the pandas DataFrame.to_sql() -> uuid failure with ppg3 (via an xfail test) but passes on ppg2.
Please include links to any upstream pandas/SQLAlchemy discussions or issues related to this topic so we can track whether this remains necessary long-term. Examples:
| There is no connection or configuration option to keep hooks on psycopg2; the | ||
| ``sql_alchemy_conn`` workarounds above cover the metadata database only. If your Dags rely on | ||
| psycopg2-specific behaviour, test before upgrading or pin the provider below 7.0.0. | ||
| Starting with provider 7.1.0, the SQLAlchemy engines built by the hook can be kept on |
There was a problem hiding this comment.
@eladkal Do we have a placeholder for "next feature/bugfix version" for providers? One cannot know reliably in which release a feature might be included.
There was a problem hiding this comment.
@Dev-iL yeah, I shouldn't have edited the changelog note.
I've removed the 7.1.0 line now. The extra param is documented in connections/postgres.rst. whenever this actually ships, that release can mention it.
sehajsandhu123
commented
Aug 26, 2026
@Dev-iL Thanks for the review, I've added the requested regression test and upstream discussion links. I understand the caution, but this is an opt-in param, off by default. And it isn't really a new one — the convention already exists in the codebase: OdbcHook and MsSqlHook both support sqlalchemy_scheme for per-connection driver choice (pymssql vs pyodbc), and MySqlHook does the same through its client extra. Also, the 7.0.0 notes already give the metadata DB an explicit psycopg2 opt-out via sql_alchemy_conn — hook connections were the only part of this migration without one. Since it's off by default and the strict xfail will tell us when upstream makes it unnecessary, it should be easy to retire too. |
Provider 7.0.0 switched hook-built SQLAlchemy engines to psycopg (v3) whenever SQLAlchemy 2.x is installed, with no opt-out. The psycopg dialect renders typed bind casts, so string parameters PostgreSQL previously coerced implicitly now fail server-side — most visibly pandas.DataFrame.to_sql into uuid columns (apache#71977). Honor the existing DbApiHook sqlalchemy_scheme connection extra (and hook parameter) so connections that rely on psycopg2 behaviour can keep it.
7ea1579 to
4f717deCompare| conn.login, conn.password, conn.port = self.get_iam_token(conn) | ||
| return URL.create( | ||
| drivername="postgresql+psycopg" if USE_PSYCOPG3 else "postgresql", | ||
| drivername=self.sqlalchemy_scheme, |
There was a problem hiding this comment.
I'm not sure if this should be an operator/hook parameter or something defined inside the Connection itself.
What was your thinking for putting it here?
There was a problem hiding this comment.
@ashb It's primarily a connection property, the hook param is just an override on top. The intended usage is setting it once on the connection (or not setting it at all is also fine, for backwards compatibility) and the hook param only kicks in if explicitly passed.
The reason I added it is if most dags on a connection are fine on psycopg3 and only one or two break, it pins just those tasks instead of rolling the whole connection back to psycopg2. Keeps the opt-out small and lets the connection default move forward.
Happy to drop the hook param if you just want the connection extra.
Since postgres provider 7.0.0,
PostgresHookbuilds its SQLAlchemy engines on psycopg (v3) whenever SQLAlchemy 2.x is installed, and the 7.0.0 changelog notes there is no connection or configuration option to keep hooks on psycopg2.This silently changes semantics for Dag code that uses
hook.get_uri()orhook.get_sqlalchemy_engine(). The psycopg (v3) dialect renders typed bind casts, so string parameters that PostgreSQL used to coerce implicitly now fail server-side. The most common casualty ispandas.DataFrame.to_sqlinto a table with auuidcolumn, which worked for years on psycopg2 and now fails with:This PR adds the missing opt-out by honoring the
sqlalchemy_schemeconnection extra (and an equivalent hook parameter), the same conventionOdbcHookandDbApiHook.dialect_namealready follow.PostgresHookeven listssqlalchemy_schemeinignored_extra_options, so it is already excluded from libpq connect args — it just wasn't used when building the URL. With this change, a connection that needs the old behaviour can set:{"sqlalchemy_scheme": "postgresql+psycopg2"}The value is validated to be
postgresqlorpostgresql+<driver>with no:or/, so a connection extra can't smuggle in a different URL. Nothing changes for connections that don't set it.Tested against PostgreSQL 17 with pandas 3.0.5 / SQLAlchemy 2.0.51:
df.to_sqlinto auuidcolumn reproduces the error above on the default psycopg3 engine and succeeds with the extra set. Unit tests cover the override, parameter precedence,get_uripropagation, and rejection of invalid schemes.related: #71977
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Fable 5) following the guidelines