Skip to content

Add quote and escape attributes to create csv external table - #8351

Merged
alamb merged 27 commits into
apache:mainfrom
Asura7969:extend_csv_sql
Nov 29, 2023
Merged

Add quote and escape attributes to create csv external table#8351
alamb merged 27 commits into
apache:mainfrom
Asura7969:extend_csv_sql

Conversation

@Asura7969

@Asura7969Asura7969 commented Nov 29, 2023•
edited
Loading

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes#8310.

Rationale for this change

Create csv table in sql mode, support quote & escape

What changes are included in this PR?

Add quote and escape attributes to create csv external table

CREATE EXTERNAL TABLE custom(
c1 VARCHAR DEFAULT NULL,
c2 VARCHAR DEFAULT NULL
)
STORED AS CSV
DELIMITER ','
WITH HEADER ROW
OPTIONS ('escape''\', 'quote''~')LOCATION 'custom.csv';

Are these changes tested?

yes, csv_files.slt

Are there any user-facing changes?

Add quote & escape attributes to OPTIONS.

@Asura7969
Minor: Improve the document format of JoinHashMap
7afeb8b
@Asura7969
Merge remote-tracking branch 'origin/main'
6332bec
# Conflicts:
#	datafusion/physical-plan/src/joins/hash_join_utils.rs
@Asura7969
Merge remote-tracking branch 'origin/main'
cc5e0c7
# Conflicts:
#	datafusion/physical-plan/src/joins/hash_join_utils.rs
@Asura7969
Merge remote-tracking branch 'origin/main'
a114310
@Asura7969
Merge remote-tracking branch 'origin/main'
928c811
@Asura7969
Merge remote-tracking branch 'origin/main'
839093e
@Asura7969
Merge remote-tracking branch 'origin/main'
a836cde
@Asura7969
Merge branch 'apache:main' into main
5648dc7
@Asura7969
Merge branch 'apache:main' into main
a670409
@Asura7969
Merge branch 'apache:main' into main
22894a3
@Asura7969
Merge branch 'apache:main' into main
73a59d2
@Asura7969
Merge branch 'apache:main' into main
46409c2
@Asura7969
Merge branch 'apache:main' into main
8a86a4c
@Asura7969
Merge branch 'apache:main' into main
cf5c584
@Asura7969
Merge branch 'apache:main' into main
62ae9b9
@Asura7969
Merge branch 'apache:main' into main
da02fa2
@Asura7969
Merge branch 'apache:main' into main
d98eb2e
@Asura7969
Merge branch 'apache:main' into main
79e7216
@Asura7969
Merge branch 'apache:main' into main
ba51abd
@Asura7969
Merge branch 'apache:main' into main
2468f52
@Asura7969
Merge branch 'apache:main' into main
180c303
@Asura7969
Merge branch 'apache:main' into main
68980ba
@Asura7969
Merge branch 'apache:main' into main
9411940
@Asura7969
Merge branch 'apache:main' into main
ba28346
@Asura7969
Merge branch 'apache:main' into main
df0942f
@Asura7969
sql csv_with_quote_escape
dbba317
@github-actionsgithub-actionsBot added core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) labels Nov 29, 2023
Asura7969
Asura7969 commented Nov 29, 2023
View reviewed changes
Comment threaddatafusion/sqllogictest/test_files/csv_files.slt
WITH HEADER ROW
DELIMITER ','
OPTIONS ('escape' '\"')
LOCATION '../core/tests/data/escape.csv';

@Asura7969Asura7969Nov 29, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To CREATE EXTERNAL TABLE syntax coverage, I add this file

@Asura7969Asura7969 changed the title Extend csv sqlAdd quote & escape for create external tableNov 29, 2023
@Asura7969Asura7969 changed the title Add quote & escape for create external tableAdd quote and escape attributes to create csv external tableNov 29, 2023
@alambalamb mentioned this pull request Nov 29, 2023
Closed
alamb
alamb reviewed Nov 29, 2023
View reviewed changes
Comment threaddatafusion/common/src/file_options/csv_writer.rs
alamb
alamb approved these changes Nov 29, 2023
View reviewed changes

@alambalamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @Asura7969 -- this looks great. I had a suggestion related to settings and it would be nice to add a reference to the comments, but I also think we could merge this PR as is and do them as a follow on.

Let me know what you would like to do.

Thanks again!

Comment threaddatafusion/core/src/datasource/listing_table_factory.rs Outdated
@alambalamb mentioned this pull request Nov 29, 2023
Closed
8 tasks
@Asura7969

Asura7969 commented Nov 29, 2023

Copy link
Copy Markdown
ContributorAuthor

Thank you @Asura7969 -- this looks great. I had a suggestion related to settings and it would be nice to add a reference to the comments, but I also think we could merge this PR as is and do them as a follow on.

Let me know what you would like to do.

Thanks again!

I will follow your suggestions

@alamb

alamb commented Nov 29, 2023

Copy link
Copy Markdown
Contributor

Thank you @Asura7969 !

@alamb
alamb merged commit aeb012e into apache:mainNov 29, 2023
@alambalamb mentioned this pull request Nov 29, 2023
Closed
@Asura7969
Asura7969 deleted the extend_csv_sql branch November 30, 2023 00:58
@matthewgappmatthewgapp mentioned this pull request Jan 11, 2024
Closed
@simicdsimicd mentioned this pull request Jan 16, 2024
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alambalambalamb approved these changes

Assignees

No one assigned

Labels

coreCore DataFusion cratesqllogictestSQL Logic Tests (.slt)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add quote & escape for create external table

2 participants

@Asura7969@alamb