GH-47502: Introduce optional default_column_type parameter #47663

vladborovtsov · 2025-09-27T12:34:52Z

Rationale for this change

Add an optional default_column_type parameter to the CSV reading API (C++ and Python) to provide a fallback type when per-column types aren’t specified, improving schema consistency and complementing the existing column_types logic.

What changes are included in this PR?

c++: new convert option "default_column_type" to augment logic around column_types parameter
3 reader tests: DefaultColumnTypePartialDefault, DefaultColumnTypeAllStringsWithHeader, DefaultColumnTypeAllStringsNoHeader). The last two tests are inspired by PARTIAL FIX: Improve leading zeros preservation with dtype=str for dict-based dtypes pandas-dev/pandas#62242 and BUG: pyarrow stripping leading zeros with dtype=str pandas-dev/pandas#57666
python: corresponding changes to make cpp change consumable from python
python: extended test_convert_options test - include. default_column_type
python: added new test "test_default_column_type" which tests how the field impacts schema; also test implicitly verifies leading zero preservation
relevant documentation update for python component;

Are these changes tested?

Yes. Existing and new tests are passing.

C++:

> [==========] Running 3 tests from 1 test suite.
> [----------] Global test environment set-up.
> [----------] 3 tests from ReaderTests
> [ RUN      ] ReaderTests.DefaultColumnTypePartialDefault
> [       OK ] ReaderTests.DefaultColumnTypePartialDefault (3 ms)
> [ RUN      ] ReaderTests.DefaultColumnTypeAllStringsWithHeader
> [       OK ] ReaderTests.DefaultColumnTypeAllStringsWithHeader (0 ms)
> [ RUN      ] ReaderTests.DefaultColumnTypeAllStringsNoHeader
> [       OK ] ReaderTests.DefaultColumnTypeAllStringsNoHeader (0 ms)
> [----------] 3 tests from ReaderTests (4 ms total)
> 
> [----------] Global test environment tear-down
> [==========] 3 tests from 1 test suite ran. (4 ms total)
> [  PASSED  ] 3 tests.

All:

> [==========] 264 tests from 46 test suites ran. (452 ms total)
> [  PASSED  ] 264 tests.

pyarrow:
New tests are passing.

Are there any user-facing changes?

I believe this change is backward compatible. Parameter is optional and its default value doesn't change the existing behavior; All the existing rests are passing.

Maybe relevant: #22232

Relates to #47502

GitHub Issue: [C++] CSV reader: add a default column type (or sentinel mapping) to avoid per-column enumeration #47502

…_types logic;

github-actions · 2025-09-27T12:35:16Z