Uh oh!
There was an error while loading. Please reload this page.
GH-47897: [C++][Python] Allow default column type for CSV columns - #47898
GH-47897: [C++][Python] Allow default column type for CSV columns#47898cottrell wants to merge 2 commits into
Conversation
AlenkaF
commented
Oct 24, 2025
cottrell
commented
Oct 24, 2025
Yes! I did search for this but didn't find it. Definitely go with that one if it's already in progress or better. It is partly agentic yes. Interesting that someone else has also hitting this very minor but useful addition. I will close this for now. |
Rationale for this change
Users who want to coerce every CSV column to a single type currently have to pre-compute a schema or enumerate column names. Adding a default type on
ConvertOptionsremoves that friction (e.g. “read everything as string”).What changes are included in this PR?
ConvertOptions::column_typein C++ and honor it in the CSV reader when no per-column mapping existscsv.ConvertOptions(column_type=…)in PyArrow, with documentation updatesinclude_missing_columnsAre these changes tested?
make test-csvpytest python/pyarrow/tests/test_cpp_internals.pypytest python/pyarrowattempt times out in the dataset backpressure test on this environment (documented limitation)Are there any user-facing changes?
column_typeparameter onpyarrow.csv.ConvertOptionsComponent(s)
C++, Python