Upgrade to sqlparser 0.61.0 by alamb · Pull Request #20177 · apache/datafusion

alamb · 2026-02-05T22:07:50Z

DRAFT until SQL parser is released

Which issue does this PR close?

part of Release sqlparser-rs version 0.61.0 around 2026-02-01 datafusion-sqlparser-rs#2117

Rationale for this change

Keep up to date with dependencies

I think @Samyak2 specifically would like access to the : field syntax

What changes are included in this PR?

Update to 0.61.0
Update APIs

Are these changes tested?

Yes by existing tests

Are there any user-facing changes?

New dependency

alamb · 2026-02-05T22:14:37Z

There are a few sqllogictest errors like this

External error: 35 errors in file /Users/andrewlamb/Software/datafusion2/datafusion/sqllogictest/test_files/map.slt
1. statement failed: DataFusion error: SQL error: ParserError("Expected: ), found: { at Line: 3, Column: 8")
[SQL] CREATE TABLE map_array_table_1
AS VALUES
(MAP {1: [1, NULL, 3], 2: [4, NULL, 6], 3: [7, 8, 9]}, 1, 1.0, '1'),
(MAP {4: [1, NULL, 3], 5: [4, NULL, 6], 6: [7, 8, 9]}, 5, 5.0, '5'),
(MAP {7: [1, NULL, 3], 8: [9, NULL, 6], 9: [7, 8, 9]}, 4, 4.0, '4')
;
at /Users/andrewlamb/Software/datafusion2/datafusion/sqllogictest/test_files/map.slt:18

alamb · 2026-02-06T13:11:41Z

I had Codex make up a DataFusion specific dialect to get the tests passing. I now need to clean it up, Occam's razor style

Samyak2 · 2026-02-06T14:30:40Z

Should we perhaps consider the errors as bugs in GenericDialect instead? There's some SQL that was previously parsed successfully by GenericDialect but fails now -- sounds like something to fix

Samyak2 · 2026-02-06T18:13:30Z

I have an issue and a PR for this upstream: apache/datafusion-sqlparser-rs#2204 and apache/datafusion-sqlparser-rs#2205

Seems to fix all the 35 errors here

alamb · 2026-02-06T20:52:52Z

Should we perhaps consider the errors as bugs in GenericDialect instead? There's some SQL that was previously parsed successfully by GenericDialect but fails now -- sounds like something to fix

Yes, likely -- this is great news -- thank you @Samyak2

alamb · 2026-02-06T21:29:22Z

 }) => {
 if temporary {
- return not_impl_err!("Temporary tables not supported")?;
+ return not_impl_err!("Temporary tables not supported");


This is a drive by cleanup -- no need to return and ? -- just return is fine

alamb · 2026-02-06T21:48:15Z

Should we perhaps consider the errors as bugs in GenericDialect instead? There's some SQL that was previously parsed successfully by GenericDialect but fails now -- sounds like something to fix
Yes, likely -- this is great news -- thank you @Samyak2

Thanks again @Samyak2 -- I updated this branch to include the fix (and the 0.61.0 Release candidate) and all the tests pass now.

Thanks 🙏 !

…update

alamb · 2026-02-12T14:21:12Z

@jonahgao I wonder if you have a few minutes to review this PR?

jonahgao

Looks good to me, thanks @alamb

jonahgao · 2026-02-14T12:41:40Z

 // Process distinct clause
 let plan = match select.distinct {
 None => Ok(plan),
+ Some(Distinct::All) => Ok(plan),


Should we add a new slt test for this, e.g., select all * from test? It’s equivalent to select * from test.

This is a good idea. I added this in 0131226
I also double checked and this syntax is also checked in the extended sqllogictests: https://github.com/search?q=repo%3Aapache%2Fdatafusion-testing%20%22select%20all%20*%22&type=code
And I ran that locally as well and verified it passes
INCLUDE_SQLITE=true cargo test --profile release-nonlto --test sqllogictests

…update

Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>

…sion into alamb/test_sqlparser_update

alamb · 2026-02-23T18:49:03Z

Thanks again @Jefffrey and @jonahgao

@Samyak2

DRAFT until SQL parser is released ## Which issue does this PR close? - part of apache/datafusion-sqlparser-rs#2117 ## Rationale for this change Keep up to date with dependencies I think @Samyak2 specifically would like access to the `:` field syntax ## What changes are included in this PR? 1. Update to 0.61.0 2. Update APIs ## Are these changes tested? Yes by existing tests ## Are there any user-facing changes? New dependency --------- Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>

github-actionsBot added the sql SQL Planner label Feb 5, 2026

alamb mentioned this pull request Feb 5, 2026
Release sqlparser-rs version 0.61.0 around 2026-02-01 apache/datafusion-sqlparser-rs#2117
Closed
6 tasks

alamb force-pushed the alamb/test_sqlparser_update branch from c95affc to 93fa56eCompare February 6, 2026 13:11

github-actionsBot added optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) common Related to common crate labels Feb 6, 2026

Samyak2 mentioned this pull request Feb 6, 2026
Regression in MAP literals for GenericDialectapache/datafusion-sqlparser-rs#2204
Closed

alamb force-pushed the alamb/test_sqlparser_update branch from 93fa56e to 8bda485Compare February 6, 2026 21:00

github-actionsBot removed optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) common Related to common crate labels Feb 6, 2026

alamb force-pushed the alamb/test_sqlparser_update branch from 8bda485 to 315e7c0Compare February 6, 2026 21:13

alamb added 2 commits February 6, 2026 16:13

Update sqlparser to 0.61.0
e3e0e71

Update for new APIs
0461c5a

alamb force-pushed the alamb/test_sqlparser_update branch from 315e7c0 to 0461c5aCompare February 6, 2026 21:13

cleanups
d0373e8

github-actionsBot added the sqllogictest SQL Logic Tests (.slt) label Feb 6, 2026

alamb commented Feb 6, 2026
View reviewed changes

alamb mentioned this pull request Feb 6, 2026
Prepare to release sqlparser 0.61.0 and sqlparser_derive 0.5.0apache/datafusion-sqlparser-rs#2195
Merged

Samyak2 approved these changes Feb 7, 2026
View reviewed changes

alamb added 2 commits February 10, 2026 11:00

Merge remote-tracking branch 'apache/main' into alamb/test_sqlparser_…
5573fe4
…update

use released sqlparser
3764c31

alamb marked this pull request as ready for review February 10, 2026 17:01

alamb marked this pull request as draft February 10, 2026 17:01

Fix merge issue
1a4964f

alamb marked this pull request as ready for review February 10, 2026 18:10

Merge branch 'main' into alamb/test_sqlparser_update
79eaa41

alamb requested a review from adriangb February 12, 2026 20:32

jonahgao approved these changes Feb 14, 2026
View reviewed changes

Jefffrey approved these changes Feb 20, 2026
View reviewed changes

Comment threaddatafusion/sql/src/statement.rs Outdated

alamband others added 4 commits February 23, 2026 08:35

Merge remote-tracking branch 'apache/main' into alamb/test_sqlparser_…
22f9ea2
…update

Apply suggestion from@Jefffrey
d9f929a
Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>

Merge branch 'alamb/test_sqlparser_update' of github.com:alamb/datafu…
d3313f9
…sion into alamb/test_sqlparser_update

Add explicit test for SELECT ALL *
0131226

alamb mentioned this pull request Feb 23, 2026
Improve performance of query execution by not clone'ing SessionState #20495
Open

alamb added this pull request to the merge queue Feb 23, 2026

Merged via the queue into apache:main with commit b9328b9Feb 23, 2026
31 checks passed

alamb deleted the alamb/test_sqlparser_update branch February 23, 2026 19:46

Samyak2 mentioned this pull request Feb 28, 2026
colon : operator - initial implementation datafusion-contrib/datafusion-variant#26
Open
4 tasks

Navigation Menu

Uh oh!

Upgrade to sqlparser 0.61.0 - #20177

Upgrade to sqlparser 0.61.0#20177
alamb merged 11 commits into
apache:mainfrom
alamb:alamb/test_sqlparser_update

alamb commented Feb 5, 2026

Uh oh!

alamb commented Feb 5, 2026

Uh oh!

alamb commented Feb 6, 2026

Uh oh!

Samyak2 commented Feb 6, 2026

Uh oh!

Samyak2 commented Feb 6, 2026

Uh oh!

alamb commented Feb 6, 2026

Uh oh!

alambFeb 6, 2026

Uh oh!

alamb commented Feb 6, 2026

Uh oh!

alamb commented Feb 12, 2026

Uh oh!

jonahgao left a comment

Uh oh!

jonahgaoFeb 14, 2026

Uh oh!

alambFeb 23, 2026

Uh oh!

Uh oh!

alamb commented Feb 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

alamb commented Feb 5, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb commented Feb 5, 2026

Uh oh!

alamb commented Feb 6, 2026

Uh oh!

Samyak2 commented Feb 6, 2026

Uh oh!

Samyak2 commented Feb 6, 2026

Uh oh!

alamb commented Feb 6, 2026

Uh oh!

alambFeb 6, 2026

Choose a reason for hiding this comment

Uh oh!

alamb commented Feb 6, 2026

Uh oh!

alamb commented Feb 12, 2026

Uh oh!

jonahgao left a comment

Choose a reason for hiding this comment

Uh oh!

jonahgaoFeb 14, 2026

Choose a reason for hiding this comment

Uh oh!

alambFeb 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alamb commented Feb 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants