Add #[recursive]

[blaginin]

Closes#984, related to apache/datafusion#9375 (comment)

Todo:

• Test the performance overhead on large queries. If it's too high, move it to a separate feature to allow opting out

[blaginin]

I was wondering if we should remove RecursionCounter with this PR. In my opinion, we shouldn't, because the ability to limit max recursion might be useful for some users still

[blaginin]

It seems this PR doesn't have any significant performance impact.

critcmp main recursive group main recursive
----- ---- ---------
sqlparser-rs parsing benchmark/format_large_statement 1.00 309.1±12.72µs ? ?/sec 1.01 312.4±7.45µs ? ?/sec
sqlparser-rs parsing benchmark/parse_large_statement 1.02 6.3±0.35ms ? ?/sec 1.00 6.2±0.40ms ? ?/sec
sqlparser-rs parsing benchmark/parse_sql_large_query 1.00 6.1±0.23ms ? ?/sec sqlparser-rs parsing benchmark/sqlparser::select 1.02 1893.8±82.19ns ? ?/sec 1.00 1855.1±25.08ns ? ?/sec
sqlparser-rs parsing benchmark/sqlparser::with_select 1.03 11.7±1.20µs ? ?/sec 1.00 11.4±0.28µs ? ?/sec

[blaginin]

FYI, I marked this ready for review. @peter-toth@Eason0729, if you guys want to take a look 🌻

[iffyio]

@blaginin thanks for looking to fix this!

Currently the preference is to avoid a third-party dependency for this issue, ideally fixing up the parser behavior instead to properly handle deeply nested input. See comment here for a bit more context on rationale
From a quick look at recursive it seems to be a procmacro on top of the stacker library so that the same considerations should apply I imagine.

[Eason0729]

Thanks @blaginin. LGTM except for some number in unit test.

I left some comments, and any number above 40000 seems to overflow stack without #[recursive].

[Eason0729]

For large_statement, making separated test would make differentiating potential(future) regression easier.
It's a suggestion(fine to leave it as it is).

[blaginin]

Not sure I understood your comment, sorry. I added tests for parsing large statements in tests/sqlparser_common.rs. Do you think we should test something else?

[Eason0729]

I thinks it would be better to add as separated test.

[peter-toth]

Currently the preference is to avoid a third-party dependency for this issue, ideally fixing up the parser behavior instead to properly handle deeply nested input. See comment here for a bit more context on rationale From a quick look at recursive it seems to be a procmacro on top of the stacker library so that the same considerations should apply I imagine.

@iffyio, we had done some research on stacker / recursive in apache/datafusion#13310 to verify concerns before we added it to datafusion: apache/datafusion#13310 (review) / apache/datafusion#13177 (comment)

[Eason0729]

Currently the preference is to avoid a third-party dependency for this issue, ideally fixing up the parser behavior instead to properly handle deeply nested input. See comment here for a bit more context on rationale From a quick look at recursive it seems to be a procmacro on top of the stacker library so that the same considerations should apply I imagine.

@iffyio, we had done some research on stacker / recursive in apache/datafusion#13310 to verify concerns before we added it to datafusion: apache/datafusion#13310 (review) / apache/datafusion#13177 (comment)

Sorry for missing that context when reviewing. So we would like to avoid using recursive for stablility, maybe try...

1. vender the code from recursive
2. use stacker directly, like fix: add stacker and maybe_grow on recursion guard #1468

[blaginin]

Thanks for the review!! 🙂

Sorry for missing that context when reviewing. So we would like to avoid using recursive for stablility, maybe try...

We ended up using #[recursive] in Datafusion, as Peter highlighted. Feels like it’s good to stay consistent across projects?

[iffyio]

Ah I see, thanks for the context @peter-toth!

cc @alamb for overall thoughts on adding this dependency to sqlparser?

[alamb]

Ah I see, thanks for the context @peter-toth!

cc @alamb for overall thoughts on adding this dependency to sqlparser?

While adding new dependencies in general ls 🤮 I don't think there is any viable alternative in this case

We have tried to avoid doing something like stacker for several years with sqlparser and datafusion, but I feel like we have only been able to band aid over the problem, not fix the problem once and for all.

I am hopeful that if we adopt this particular crate we won't have to worry about it again 🤞

[alamb]

I think this PR needs a bit more documentation and we shoudl figure out how to rationalize with the existing recursion_limit argument.

https://docs.rs/sqlparser/latest/sqlparser/parser/struct.Parser.html#method.with_recursion_limit

[blaginin]

@alamb, I feel like recursion_limit should stay because:

• recursive doesn't solve the issue when std isn’t enabled.
• Removing it would be a breaking change.
• It can be a useful feature even with recursive protection, for example, if lib users have additional constraints on the parsed query.

I added notes in the methods I touched; should be better now ✋

[Eason0729]

LGTM

[Eason0729]

It seems like we reached the decision to add recursive instead of using underlying dependency(stacker).

[alamb]

I think it looks good -- any other thoughts @iffyio

Thank you @blaginin and @Eason0729 for pushing this along

[blaginin]

Just for transparency, there's apache/datafusion#13513 raised in Datafusion but I believe it shouldn't be the reason not to merge this one (happy to be challenged)

[alamb]

Just for transparency, there's apache/datafusion#13513 raised in Datafusion but I believe it shouldn't be the reason not to merge this one (happy to be challenged)

Maybe we could make it an optional dependency 🤔

[blaginin]

nice idea actually! will do

[iffyio]

LGTM!

[blaginin]

resolved conflicts, should be good to merge now 🤗

[alamb]

Given the potential for unintended side effects with this change, I think we should merge it in asap after we have released

• Release sqlparser-rs version 0.53.0 / sqlparser_derive 0.3.0 #1517

[Eason0729]

Should we merge this? or something is missing in this PR.
I have no write access, so cc @alamb .

[blaginin]

hey, based on the previous comment I think we want to make a release first 🙂

[Eason0729]

hey, based on the previous comment I think we want to make a release first 🙂

Thanks.
I am sorry for missing that! 👀

[alamb]

Sorry -- the reason I haven't previously merged this is exactly to meger it after release to give it enough "bake time" . I am glad we did wait, actually, as using this macro has caused trouble downstream in datafusion

• see Making the recursive dependency an optional feature datafusion#13766

• Now that we have released Release sqlparser-rs version 0.53.0 / sqlparser_derive 0.3.0 #1517 (finally made it yesterday)

I think we are good to go

Thank you again @blaginin and @Eason0729 for your contributions and patience

[lovasoa]

Hi @alamb , @iffyio ! Just wanted to report that I just received a crash report that seems to come from here: sqlpage/SQLPage#814

[alamb]

Hi @alamb , @iffyio ! Just wanted to report that I just received a crash report that seems to come from here: sqlpage/SQLPage#814

Thaks @lovasoa

Looks like you have also reported a bug to stacker:

• Assertion error: when pthread returns an error, this library crashes the entire program  rust-lang/stacker#115

I am glad there is a way to disable the stacker dependency in sqlparser

Is there anything else you think we should do here?

Thanks again

[lovasoa]

Yes, I discovered error handling is completely absent from stacker, and it just crashes the entire program with a cryptic error message when the underlying pthread library returns any error. I submitted a PR, but in the meantime, I wouldn't recommend including it by default in sqlparser, especially if the goal was to avoid crashes in the first place.

[lovasoa]

SQLPage does not have a huge install base, and it took just a few days before the first crash report.

[alamb]

SQLPage does not have a huge install base, and it took just a few days before the first crash report.

DataFusion does have a pretty large user base and we haven't gotten crash reports yet that I know of. I was somewhat worried about using stacker in datafusion too until @peter-toth pointed out that it was used by rustc itself which allayed my concerns.

it does seem like the usecase

glibc targets where procfs isn't mounted at /proc is

Is somewhat uncommon.

I don't really have a strong opinion one way or the other.

[lovasoa]

Yes, I initially thought that the problem was specific to their very restricted environment, but looking at the code in stacker, they crash the entire program on ANY error returned by any of the pthread functions used. And pthread functions can return an error code in a number of cases.

[alamb]

Interesting -- I haven't looked at the code or what functions are used (and thus under what circumstances such errors happen or how likely they are to occur)

[peter-toth]

I didn't notice the missing error handling either. Your rust-lang/stacker#116 seems like a nice improvement. But if it doesn't get accepted for some reason, then probably we could handle errors in DF. Maybe adjust the recursive macro to run some tests before using stacker...

[alamb]

I didn't notice the missing error handling either. Your rust-lang/stacker#116 seems like a nice improvement. But if it doesn't get accepted for some reason, then probably we could handle errors in DF. Maybe adjust the recursive macro to run some tests before using stacker...

• Looks like improve error handling rust-lang/stacker#116 was accepted 🎉