Add quote style and trimming to csv writier by xanderbailey · Pull Request #20813 · apache/datafusion

xanderbailey · 2026-03-08T21:11:31Z

Which issue does this PR close?

ClosesAdd quote-style parameter for CSV options #10669

Related arrow-rs PRs apache/arrow-rs#8960 and apache/arrow-rs#9004

Rationale for this change

The CSV writer was missing support for quote_style, ignore_leading_whitespace, and ignore_trailing_whitespace options that are available on the underlying arrow WriterBuilder. This meant users couldn't control quoting behaviour or whitespace trimming when writing CSV files.

What changes are included in this PR?

Adds three new CSV writer options wired through the full stack:

quote_style — controls when fields are quoted (Always, Necessary, NonNumeric, Never). Modelled as a protobuf enum (CsvQuoteStyle).
ignore_leading_whitespace — trims leading whitespace from string values on write.
ignore_trailing_whitespace — trims trailing whitespace from string values on write.

Are these changes tested?

Yes — sqllogictest coverage added in csv_files.slt

Are there any user-facing changes?

Three new format.* options available in COPY TO and CREATE EXTERNAL TABLE for CSV:

format.quote_style (string: Always, Necessary, NonNumeric, Never)
format.ignore_leading_whitespace (boolean)
format.ignore_trailing_whitespace (boolean)

xanderbailey · 2026-03-08T22:45:42Z

+ pub quote_style: i32,
+ /// Whether to ignore leading whitespace in string values
+ #[prost(bytes = "vec", tag = "21")]
+ pub ignore_leading_whitespace: ::prost::alloc::vec::Vec<u8>,


Following the pattern here for other bools being Vec

xanderbailey · 2026-03-08T22:45:48Z

+ pub quote_style: i32,
+ /// Whether to ignore leading whitespace in string values
+ #[prost(bytes = "vec", tag = "21")]
+ pub ignore_leading_whitespace: ::prost::alloc::vec::Vec<u8>,


Following the pattern here for other bools being Vec

xanderbailey · 2026-03-08T22:48:33Z

+ Ok(CsvQuoteStyleProto::Always) => CsvQuoteStyle::Always,
+ Ok(CsvQuoteStyleProto::NonNumeric) => CsvQuoteStyle::NonNumeric,
+ Ok(CsvQuoteStyleProto::Never) => CsvQuoteStyle::Never,
+ _ => CsvQuoteStyle::Necessary,


We don't error on:
compression: match proto.compression { 0 => CompressionTypeVariant::GZIP, 1 => CompressionTypeVariant::BZIP2, 2 => CompressionTypeVariant::XZ, 3 => CompressionTypeVariant::ZSTD, _ => CompressionTypeVariant::UNCOMPRESSED, },
So made the same true here

xanderbailey · 2026-03-24T10:05:59Z

@Jefffrey are you maybe able to take a look here?

rtyler

I have done some integration testing of this PR since it solves a problem one of my customers was having as well. I think it's good to merge, with a squashing of course 😈

I ended up backporting it to 52.4.0 which applied cleanly as well. I notice that arrow 57.2 actually introduced QuoteStyle which means this change requires arrow-csv 57.2.0 or later

Digging deeper into load failures I discovered that while arrow-csv and other Rust CSV parsers can handle some characters properly, some RDMS and C-based CSV parsers (older) cannot, without quoting. This relies on a backport of apache/datafusion#20813 to the 52.4.0 release in order to support this. It also requires arrow-csv 57.2.0 or later since that release is the first that `QuoteStyle` exists.

xanderbailey · 2026-04-14T07:27:05Z

@alamb could I bother you for a review here please?

Jefffrey · 2026-04-16T04:27:18Z

+ pub ignore_leading_whitespace: Option<bool>, default = None
+ /// Whether to ignore trailing whitespace in string values when writing CSV.
+ pub ignore_trailing_whitespace: Option<bool>, default = None
 /// Specifies whether newlines in (quoted) values are supported.


I'm wondering why we have this trend of doing Option<bool> instead of just bool 🤔
What is the default in case of None, and how is that different from just true/false (whichever it is)?

I was also curious about this but didn't want to break the pattern...

Is it some ser-deser quirk? I'll see if I can figure this out.

No I can't find a good reason for this... Would you like me to change or keep the pattern?

Perhaps its to do with hierarchy of configs and/or sql parsing 🤔
@alamb do you happen to know a reason for having Option<bool> instead of plain bool for cases where it'll end up being true or false anyway? (i.e. None doesn't represent a third state, but eventually maps to either true or false)

I fired a question off into the Discord to see if anyone has some more insight; ideally I'd like to get this config right the first time to avoid needing a breaking change later (or forever be stuck with this awkward interface if it turns out to be unnecessary)

Should be fine to leave them as Option<bool>; but would be good to have a docstring explaining what the default is in this case

Jefffrey

Changes look good to me, just want to get a second opinion on Option<bool> and which way is preferable

Jefffrey · 2026-04-21T15:20:49Z

+ pub ignore_leading_whitespace: Option<bool>, default = None
+ /// Whether to ignore trailing whitespace in string values when writing CSV.
+ pub ignore_trailing_whitespace: Option<bool>, default = None
 /// Specifies whether newlines in (quoted) values are supported.


I fired a question off into the Discord to see if anyone has some more insight; ideally I'd like to get this config right the first time to avoid needing a breaking change later (or forever be stuck with this awkward interface if it turns out to be unnecessary)

Jefffrey · 2026-04-23T10:23:47Z

+ pub ignore_leading_whitespace: Option<bool>, default = None
+ /// Whether to ignore trailing whitespace in string values when writing CSV.
+ pub ignore_trailing_whitespace: Option<bool>, default = None
 /// Specifies whether newlines in (quoted) values are supported.


Should be fine to leave them as Option<bool>; but would be good to have a docstring explaining what the default is in this case

This change still involves a cherry-pick of the changes in apache/datafusion#20813 to ensure the cdf-to-csv lambda behaves appropriately Signed-off-by: R. Tyler Croy <rtyler@buoyantdata.com>

Jefffrey · 2026-04-24T08:38:32Z

Thanks @xanderbailey & @rtyler

Add quote style and trimming to csv writier
fa967e9

github-actionsBot added sqllogictest SQL Logic Tests (.slt) common Related to common crate proto Related to proto crate labels Mar 8, 2026

fmt
c45b27f

xanderbailey commented Mar 8, 2026
View reviewed changes

Comment threaddatafusion/sqllogictest/test_files/csv_files.slt Outdated

xanderbailey added 3 commits March 8, 2026 21:38

fixup
da2f75e

fmt
c971adf

clean things up
07616a9

xanderbailey commented Mar 8, 2026
View reviewed changes

xanderbailey added 2 commits March 8, 2026 22:58

clippy
f59eb8a

code gen
eea1681

rtyler reviewed Apr 1, 2026
View reviewed changes

Comment threaddatafusion/common/src/parsers.rs

Jefffrey reviewed Apr 16, 2026
View reviewed changes

xanderbailey added 3 commits April 16, 2026 07:43

comments
2316319

update tests
c94a9a3

tests
b946b6d

xanderbailey requested a review from Jefffrey April 16, 2026 10:00

Tests read a single column
308b045

Jefffrey reviewed Apr 21, 2026
View reviewed changes

Jefffrey approved these changes Apr 23, 2026
View reviewed changes

docs
2a01fe7

Jefffrey added this pull request to the merge queue Apr 24, 2026

Merged via the queue into apache:main with commit 85e75e2Apr 24, 2026
31 checks passed

Uh oh!

Conversation

xanderbailey commented Mar 8, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xanderbailey commented Mar 24, 2026

Uh oh!

rtyler left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

xanderbailey commented Apr 14, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jefffrey left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Jefffrey commented Apr 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

xanderbailey commented Mar 8, 2026•
edited
Loading