GH-22232: [C++][Python] Introduce optional default_column_type parameter - #47663

Merged

AlenkaF merged 16 commits into

apache:mainfrom

vladborovtsov:default-column-type

May 13, 2026

vladborovtsov commented Sep 27, 2025•
edited by github-actions Bot
Loading

Contributor

Rationale for this change

Add an optional default_column_type parameter to the CSV reading API (C++ and Python) to provide a fallback type when per-column types aren’t specified, improving schema consistency and complementing the existing column_types logic.

What changes are included in this PR?

c++: new convert option "default_column_type" to augment logic around column_types parameter
3 reader tests: DefaultColumnTypePartialDefault, DefaultColumnTypeAllStringsWithHeader, DefaultColumnTypeAllStringsNoHeader). The last two tests are inspired by BUG: correctly honor specified dtype=str in read_csv pyarrow engine pandas-dev/pandas#62242 and BUG: pyarrow read_csv engine stripping leading zeros with dtype=str pandas-dev/pandas#57666
python: corresponding changes to make cpp change consumable from python
python: extended test_convert_options test - include. default_column_type
python: added new test "test_default_column_type" which tests how the field impacts schema; also test implicitly verifies leading zero preservation
relevant documentation update for python component;

Are these changes tested?

Yes. Existing and new tests are passing.

C++:

> [==========] Running 3 tests from 1 test suite.
> [----------] Global test environment set-up.
> [----------] 3 tests from ReaderTests
> [ RUN ] ReaderTests.DefaultColumnTypePartialDefault
> [ OK ] ReaderTests.DefaultColumnTypePartialDefault (3 ms)
> [ RUN ] ReaderTests.DefaultColumnTypeAllStringsWithHeader
> [ OK ] ReaderTests.DefaultColumnTypeAllStringsWithHeader (0 ms)
> [ RUN ] ReaderTests.DefaultColumnTypeAllStringsNoHeader
> [ OK ] ReaderTests.DefaultColumnTypeAllStringsNoHeader (0 ms)
> [----------] 3 tests from ReaderTests (4 ms total)
> > [----------] Global test environment tear-down
> [==========] 3 tests from 1 test suite ran. (4 ms total)
> [ PASSED ] 3 tests.
All:
> [==========] 264 tests from 46 test suites ran. (452 ms total)
> [ PASSED ] 264 tests.

pyarrow:
New tests are passing.

Are there any user-facing changes?

I believe this change is backward compatible. Parameter is optional and its default value doesn't change the existing behavior; All the existing rests are passing.

Maybe relevant: #22232

Relates to #47502

GitHub Issue: [C++] CSV reader: add a default column type (or sentinel mapping) to avoid per-column enumeration #47502
GitHub Issue: [C++] CSV reader: Ability to not infer column types. #22232

github-actionsBot commented Sep 27, 2025

⚠️ GitHub issue #47502has been automatically assigned in GitHub to PR creator.

github-actionsBot added Component: C++ Component: Python awaiting review labels

github-actionsBot commented Sep 27, 2025

⚠️ GitHub issue #47502has been automatically assigned in GitHub to PR creator.

4 similar comments

github-actionsBot commented Sep 27, 2025

⚠️ GitHub issue #47502has been automatically assigned in GitHub to PR creator.

github-actionsBot commented Sep 27, 2025

⚠️ GitHub issue #47502has been automatically assigned in GitHub to PR creator.

github-actionsBot commented Sep 27, 2025

⚠️ GitHub issue #47502has been automatically assigned in GitHub to PR creator.

github-actionsBot commented Sep 27, 2025

⚠️ GitHub issue #47502has been automatically assigned in GitHub to PR creator.

vladborovtsov commented Sep 27, 2025

ContributorAuthor

@github-actions crossbow submit preview-docs

github-actionsBot commented Sep 27, 2025

Only contributors can submit requests to this bot. Please ask someone from the community for help with getting the first commit in.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/18062577036

github-actionsBot added the Component: Documentation label

github-actionsBot commented Sep 27, 2025

⚠️ GitHub issue #47502has been automatically assigned in GitHub to PR creator.

vladborovtsov marked this pull request as ready for review

September 27, 2025 19:26

vladborovtsov requested review from AlenkaF, raulcd and rok as code owners

September 27, 2025 19:26

dxdc mentioned this pull request

BUG: correctly honor specified dtype=str in read_csv pyarrow engine pandas-dev/pandas#62242

Merged

4 tasks

kou changed the title ~~GH-47502: Introduce optional default_column_type parameter~~GH-47502: [C++] Introduce optional default_column_type parameter

github-actionsBot commented Oct 13, 2025

⚠️ GitHub issue #47502has been automatically assigned in GitHub to PR creator.

This was referenced Oct 24, 2025

[C++] CSV reader: add a default column type (or sentinel mapping) to avoid per-column enumeration #47502

Closed

GH-47897: [C++][Python] Allow default column type for CSV columns #47898

Closed

AlenkaF commented Oct 24, 2025

Member

Thank you @vladborovtsov for the contribution.
I will add info about the proposed solution in the original issue (#22232) so I can see opinions from C++ devs on the proposed solution.

AlenkaF changed the title ~~GH-47502: [C++] Introduce optional default_column_type parameter~~GH-22232: [C++][Python] Introduce optional default_column_type parameter

github-actionsBot commented Oct 24, 2025

⚠️ GitHub issue #22232has been automatically assigned in GitHub to PR creator.

vladborovtsov commented Oct 24, 2025

ContributorAuthor

Hi @AlenkaF
I'm happy to continue the labour and discussion to get that merged.
As for AI, it wasn't used much, although I tried :) With such huge codebase the generation quality is quite low.

AlenkaF commented Oct 24, 2025

Member

Happy to see a response!
All good, it is totally OK to use gen AI wisely ;)

I will wait for an opinion from a C++ dev and in the meantime try to look at the Python part.

vladborovtsov commented Dec 10, 2025

ContributorAuthor

Hi @AlenkaF
Any feedback?

AlenkaF reviewed

View reviewed changes

AlenkaF left a comment

Member

The Python part looks good to me. I only added a comment for the docstring examples.

@rok @raulcd do you have any opinions on the C++ approach to handle this use case?

python/pyarrow/_csv.pyx Outdated

github-actionsBot removed the awaiting review label

vladborovtsovand others added 13 commits

May 7, 2026 12:37

extend test_convert_options test - include. default_column_type

fedbac7

test how default_column_type affects schema

30bc209

docs: add examples for default_column_type; add missing reference

62e2d21

docs: remove redundant example for default_column_type usage

f1b44df

column builder: guard test to ensure default column type won't overri…

aff70b0

…de inference.

simplify reader tests to focus on default_column_type behavior

1f7da3d

fix codestyle

c42799f

resolve python lint check failure

3c209e2

Apply suggestion from@pitrou- better inline comment for default_col…

508b3f5

…umn_type
Co-authored-by: Antoine Pitrou <pitrou@free.fr>

Remove default_column_type initialization

f16f4c0

ensure column remains float64 in all cases as its explicitly typed

db247f1

enhance default_column_type tests

a4c7c34

add default_column_type to ConvertOptions cdef

4b2dd10

vladborovtsov force-pushed the default-column-type branch from 14606b5 to 4b2dd10Compare

May 7, 2026 10:38

put default_column_type field into proper place

c1ad5f6

vladborovtsov commented May 7, 2026

ContributorAuthor

Hi @AlenkaF
I believe now it should be fine?
I'm running tests locally in docker using pytest -v --pyargs pyarrow.tests.test_csv.
Am I missing something?

AlenkaF approved these changes

View reviewed changes

AlenkaF left a comment

Member

Yes, everything looking good. I will just run the extended builds before merging. Thanks!

AlenkaF commented May 12, 2026

Member

@github-actions crossbow submit -g python

github-actionsBot commented May 12, 2026

Revision: c1ad5f6

Submitted crossbow builds: ursacomputing/crossbow @ actions-9de857b990

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.10
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-1.3.4-numpy-1.21.2
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.12-pandas-latest-numpy-1.26
test-conda-python-3.12-pandas-latest-numpy-latest
test-conda-python-3.13
test-conda-python-3.13-pandas-nightly-numpy-nightly
test-conda-python-3.13-pandas-upstream_devel-numpy-nightly
test-conda-python-3.14
test-conda-python-emscripten
test-debian-13-python-3-amd64
test-debian-13-python-3-i386
test-fedora-42-python-3
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-python-3

AlenkaF commented May 13, 2026

Member

The failing emscripten build is not related. Will merge.
Thanks!

AlenkaF merged commit 3d6e138 into apache:main

58 checks passed

AlenkaF removed the awaiting change review label

AlenkaF mentioned this pull request

[C++] CSV reader: Ability to not infer column types. #22232

Closed

vladborovtsov deleted the default-column-type branch

May 13, 2026 15:01

conbench-apache-arrowBot commented May 14, 2026

After merging your PR, Conbench analyzed the 1 benchmarking run that has been run so far on merge-commit 3d6e138.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

conbench-apache-arrowBot commented May 14, 2026

After merging your PR, Conbench analyzed the 1 benchmarking run that has been run so far on merge-commit 3d6e138.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

conbench-apache-arrowBot commented May 14, 2026

After merging your PR, Conbench analyzed the 1 benchmarking run that has been run so far on merge-commit 3d6e138.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

conbench-apache-arrowBot commented May 15, 2026

After merging your PR, Conbench analyzed the 2 benchmarking runs that have been run so far on merge-commit 3d6e138.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

Mottl pushed a commit to Mottl/arrow that referenced this pull request

apacheGH-22232: [C++][Python] Introduce optional default_column_type …

9b7fa6f

…parameter (apache#47663)
### Rationale for this change
Add an optional default_column_type parameter to the CSV reading API (C++ and Python) to provide a fallback type when per-column types aren’t specified, improving schema consistency and complementing the existing column_types logic.
### What changes are included in this PR?
- c++: new convert option "default_column_type" to augment logic around column_types parameter
- 3 reader tests: DefaultColumnTypePartialDefault, DefaultColumnTypeAllStringsWithHeader, DefaultColumnTypeAllStringsNoHeader). The last two tests are inspired by pandas-dev/pandas#62242 and pandas-dev/pandas#57666
- python: corresponding changes to make cpp change consumable from python
- python: extended test_convert_options test - include. default_column_type
- python: added new test "test_default_column_type" which tests how the field impacts schema; also test implicitly verifies leading zero preservation - relevant documentation update for python component;
### Are these changes tested?
Yes. Existing and new tests are passing.
C++: > [==========] Running 3 tests from 1 test suite.
> [----------] Global test environment set-up.
> [----------] 3 tests from ReaderTests
> [ RUN ] ReaderTests.DefaultColumnTypePartialDefault
> [ OK ] ReaderTests.DefaultColumnTypePartialDefault (3 ms)
> [ RUN ] ReaderTests.DefaultColumnTypeAllStringsWithHeader
> [ OK ] ReaderTests.DefaultColumnTypeAllStringsWithHeader (0 ms)
> [ RUN ] ReaderTests.DefaultColumnTypeAllStringsNoHeader
> [ OK ] ReaderTests.DefaultColumnTypeAllStringsNoHeader (0 ms)
> [----------] 3 tests from ReaderTests (4 ms total)
> > [----------] Global test environment tear-down
> [==========] 3 tests from 1 test suite ran. (4 ms total)
> [ PASSED ] 3 tests.
All:
> [==========] 264 tests from 46 test suites ran. (452 ms total)
> [ PASSED ] 264 tests.
pyarrow:
New tests are passing. ### Are there any user-facing changes?
I believe this change is backward compatible. Parameter is optional and its default value doesn't change the existing behavior; All the existing rests are passing. Maybe relevant: apache#22232
Relates to apache#47502
* GitHub Issue: apache#47502
* GitHub Issue: apache#22232
Lead-authored-by: Vlad Borovtsov <borovcov.vlad@gmail.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>

rudolfbyker mentioned this pull request

Disable or configure date and/or time inference when reading CSV files #45814

Closed

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment