Skip to content

Fix SQL split string to include ;-less statements - #25713

Merged
potiuk merged 1 commit into
apache:mainfrom
potiuk:fix-split-sql-string-in-common-sql
Aug 15, 2022
Merged

Fix SQL split string to include ;-less statements#25713
potiuk merged 1 commit into
apache:mainfrom
potiuk:fix-split-sql-string-in-common-sql

Conversation

@potiuk

Copy link
Copy Markdown
Member

There was a bug in an incoming change to common-sql provider
introduced in #23971 where ;-less statements were removed
when "split_statements" flag was used. Since this flag is used
by default in Databricks statement, it introduced backwards
incompatible change.


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@potiuk

Copy link
Copy Markdown
MemberAuthor

cc: @kazanzhy@alexott

@potiuk
potiukforce-pushed the fix-split-sql-string-in-common-sql branch from e77b396 to 2c9c098CompareAugust 14, 2022 16:24
@potiuk
potiuk requested a review from uranusjrAugust 14, 2022 16:25
@potiuk
potiukforce-pushed the fix-split-sql-string-in-common-sql branch from 2c9c098 to b73e910CompareAugust 14, 2022 16:28
@potiuk

potiuk commented Aug 14, 2022

Copy link
Copy Markdown
MemberAuthor

@kazanzhy@alexott - I chose a little different approach than proposed in #25640 (comment) - after adding unit tests. I generaly filter out all empty statements during the split - that will even allow comment-only statements for example

@alexottalexott left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@kazanzhy

kazanzhy commented Aug 14, 2022

Copy link
Copy Markdown
Contributor

Sorry for being offline. I tried to work on #25259 finally.

Initially, I took this code from the drill provider where it was like:

sql = sqlparse.split(sqlparse.format(self.sql, strip_comments=True))
no_term_sql = [s[:-1] for s in sql if s[-1] == ';']

which is the like current [s.rstrip(';') for s in splits if s.endswith(';')]

I tested it with different queries and even with comments and I see that I really missed the query like 'SELECT * FROM table;;;;;' and query without ;

Comment threadairflow/providers/common/sql/hooks/sql.py Outdated
There was a bug in an incoming change to common-sql provider
introduced in apache#23971 where `;-less` statements were removed
when "split_statements" flag was used. Since this flag is used
by default in Databricks statement, it introduced backwards
incompatible change.
@potiuk
potiukforce-pushed the fix-split-sql-string-in-common-sql branch from b73e910 to 1224477CompareAugust 14, 2022 19:44
@potiuk

Copy link
Copy Markdown
MemberAuthor

I think this one should be ready to review :).

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@potiuk@kazanzhy@alexott@uranusjr