remove debug table validation checks by bharatnc · Pull Request #70 · ClickHouse/arrow

bharatnc · 2025-04-17T06:24:35Z

Follow up for previous PR: #69.

Remove debug validation checks on table. The validation checks mainly fail for debug builds and what they do is to actually validate the metadata for a table. While dealing with indices for dictionaries, it looks like for compatibility, we accept signed or unsigned 32 or 64 bit integers. However, arrow dictionary builder AppendIndices only accepts signed integers. So, we end up appending signed integers while dictionary schema could still be unsigned integers. So, when the validation happens, it checks if the dictionary schema and the actual column data are the same types. Since it's possible to have uint64 schema with int64 columns, this leads to errors like (and similar error for uint32):

Column data for field 0 with type dictionary<values=string, indices=int64,
ordered=0> is inconsistent with schema dictionary<values=string,
indices=uint64, ordered=0>

Unfortunately, I couldn't find a sane way to preserve this behavior since the current behavior exists for compatibility with pandas format it looks like. The settings to support both singed and unsigned dictionary indices was added in: ClickHouse/ClickHouse#58519.

This is where the current assert fails:

arrow/cpp/src/arrow/table.cc

Line 181 in 214fd3c

RETURN_NOT_OK(ValidateMeta());

which is ultimately checked here:

arrow/cpp/src/arrow/table.cc

Line 220 in 214fd3c

returnStatus::Invalid("Column data for field ", i, " with type ",

Remove validation checks on table. The validation checks mainly fail for debug builds and what they do is to actually validate the metadata for a table. While dealing with indices for dictionaries, it looks like for compatibility, we accept signed or unsigned uint32 or uint64 integers. However, arrow dictionary builder AppendIndices only accepts signed integers. So, we end up appending signed integers while dictionary schema could still be unsigned integers. So, when the validation happens, it checks if the dictionary schema and the actual column data are the same types. Since it's possible to have uint64 schema with int64 columns, this leads to errors like (and similar error for uint32): ``` Column data for field 0 with type dictionary<values=string, indices=int64, ordered=0> is inconsistent with schema dictionary<values=string, indices=uint64, ordered=0> ```

github-actions · 2025-04-17T06:25:03Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

remove debug table validation checks (cherry picked from commit 210f686)

See #70

bharatnc · 2025-04-17T06:24:35Z

Follow up for previous PR: #69.

Remove debug validation checks on table. The validation checks mainly fail for debug builds and what they do is to actually validate the metadata for a table. While dealing with indices for dictionaries, it looks like for compatibility, we accept signed or unsigned 32 or 64 bit integers. However, arrow dictionary builder AppendIndices only accepts signed integers. So, we end up appending signed integers while dictionary schema could still be unsigned integers. So, when the validation happens, it checks if the dictionary schema and the actual column data are the same types. Since it's possible to have uint64 schema with int64 columns, this leads to errors like (and similar error for uint32):

Column data for field 0 with type dictionary<values=string, indices=int64,
ordered=0> is inconsistent with schema dictionary<values=string,
indices=uint64, ordered=0>

Unfortunately, I couldn't find a sane way to preserve this behavior since the current behavior exists for compatibility with pandas format it looks like. The settings to support both singed and unsigned dictionary indices was added in: ClickHouse/ClickHouse#58519.

This is where the current assert fails:

arrow/cpp/src/arrow/table.cc

Line 181 in 214fd3c

RETURN_NOT_OK(ValidateMeta());

which is ultimately checked here:

arrow/cpp/src/arrow/table.cc

Line 220 in 214fd3c

returnStatus::Invalid("Column data for field ", i, " with type ",

Remove validation checks on table. The validation checks mainly fail for debug builds and what they do is to actually validate the metadata for a table. While dealing with indices for dictionaries, it looks like for compatibility, we accept signed or unsigned uint32 or uint64 integers. However, arrow dictionary builder AppendIndices only accepts signed integers. So, we end up appending signed integers while dictionary schema could still be unsigned integers. So, when the validation happens, it checks if the dictionary schema and the actual column data are the same types. Since it's possible to have uint64 schema with int64 columns, this leads to errors like (and similar error for uint32): ``` Column data for field 0 with type dictionary<values=string, indices=int64, ordered=0> is inconsistent with schema dictionary<values=string, indices=uint64, ordered=0> ```

github-actions · 2025-04-17T06:25:03Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

remove debug table validation checks (cherry picked from commit 210f686)

See #70

bharatnc · 2025-04-17T06:24:35Z

Follow up for previous PR: #69.

Remove debug validation checks on table. The validation checks mainly fail for debug builds and what they do is to actually validate the metadata for a table. While dealing with indices for dictionaries, it looks like for compatibility, we accept signed or unsigned 32 or 64 bit integers. However, arrow dictionary builder AppendIndices only accepts signed integers. So, we end up appending signed integers while dictionary schema could still be unsigned integers. So, when the validation happens, it checks if the dictionary schema and the actual column data are the same types. Since it's possible to have uint64 schema with int64 columns, this leads to errors like (and similar error for uint32):

Column data for field 0 with type dictionary<values=string, indices=int64,
ordered=0> is inconsistent with schema dictionary<values=string,
indices=uint64, ordered=0>

Unfortunately, I couldn't find a sane way to preserve this behavior since the current behavior exists for compatibility with pandas format it looks like. The settings to support both singed and unsigned dictionary indices was added in: ClickHouse/ClickHouse#58519.

This is where the current assert fails:

arrow/cpp/src/arrow/table.cc

Line 181 in 214fd3c

RETURN_NOT_OK(ValidateMeta());

which is ultimately checked here:

arrow/cpp/src/arrow/table.cc

Line 220 in 214fd3c

returnStatus::Invalid("Column data for field ", i, " with type ",

Remove validation checks on table. The validation checks mainly fail for debug builds and what they do is to actually validate the metadata for a table. While dealing with indices for dictionaries, it looks like for compatibility, we accept signed or unsigned uint32 or uint64 integers. However, arrow dictionary builder AppendIndices only accepts signed integers. So, we end up appending signed integers while dictionary schema could still be unsigned integers. So, when the validation happens, it checks if the dictionary schema and the actual column data are the same types. Since it's possible to have uint64 schema with int64 columns, this leads to errors like (and similar error for uint32): ``` Column data for field 0 with type dictionary<values=string, indices=int64, ordered=0> is inconsistent with schema dictionary<values=string, indices=uint64, ordered=0> ```

github-actions · 2025-04-17T06:25:03Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

remove debug table validation checks (cherry picked from commit 210f686)

See #70

bharatnc · 2025-04-17T06:24:35Z

Follow up for previous PR: #69.

Remove debug validation checks on table. The validation checks mainly fail for debug builds and what they do is to actually validate the metadata for a table. While dealing with indices for dictionaries, it looks like for compatibility, we accept signed or unsigned 32 or 64 bit integers. However, arrow dictionary builder AppendIndices only accepts signed integers. So, we end up appending signed integers while dictionary schema could still be unsigned integers. So, when the validation happens, it checks if the dictionary schema and the actual column data are the same types. Since it's possible to have uint64 schema with int64 columns, this leads to errors like (and similar error for uint32):

Column data for field 0 with type dictionary<values=string, indices=int64,
ordered=0> is inconsistent with schema dictionary<values=string,
indices=uint64, ordered=0>

Unfortunately, I couldn't find a sane way to preserve this behavior since the current behavior exists for compatibility with pandas format it looks like. The settings to support both singed and unsigned dictionary indices was added in: ClickHouse/ClickHouse#58519.

This is where the current assert fails:

arrow/cpp/src/arrow/table.cc

Line 181 in 214fd3c

RETURN_NOT_OK(ValidateMeta());

which is ultimately checked here:

arrow/cpp/src/arrow/table.cc

Line 220 in 214fd3c

returnStatus::Invalid("Column data for field ", i, " with type ",

Remove validation checks on table. The validation checks mainly fail for debug builds and what they do is to actually validate the metadata for a table. While dealing with indices for dictionaries, it looks like for compatibility, we accept signed or unsigned uint32 or uint64 integers. However, arrow dictionary builder AppendIndices only accepts signed integers. So, we end up appending signed integers while dictionary schema could still be unsigned integers. So, when the validation happens, it checks if the dictionary schema and the actual column data are the same types. Since it's possible to have uint64 schema with int64 columns, this leads to errors like (and similar error for uint32): ``` Column data for field 0 with type dictionary<values=string, indices=int64, ordered=0> is inconsistent with schema dictionary<values=string, indices=uint64, ordered=0> ```

github-actions · 2025-04-17T06:25:03Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

remove debug table validation checks (cherry picked from commit 210f686)

See #70

bharatnc · 2025-04-17T06:24:35Z

Follow up for previous PR: #69.

Remove debug validation checks on table. The validation checks mainly fail for debug builds and what they do is to actually validate the metadata for a table. While dealing with indices for dictionaries, it looks like for compatibility, we accept signed or unsigned 32 or 64 bit integers. However, arrow dictionary builder AppendIndices only accepts signed integers. So, we end up appending signed integers while dictionary schema could still be unsigned integers. So, when the validation happens, it checks if the dictionary schema and the actual column data are the same types. Since it's possible to have uint64 schema with int64 columns, this leads to errors like (and similar error for uint32):

Column data for field 0 with type dictionary<values=string, indices=int64,
ordered=0> is inconsistent with schema dictionary<values=string,
indices=uint64, ordered=0>

Unfortunately, I couldn't find a sane way to preserve this behavior since the current behavior exists for compatibility with pandas format it looks like. The settings to support both singed and unsigned dictionary indices was added in: ClickHouse/ClickHouse#58519.

This is where the current assert fails:

arrow/cpp/src/arrow/table.cc

Line 181 in 214fd3c

RETURN_NOT_OK(ValidateMeta());

which is ultimately checked here:

arrow/cpp/src/arrow/table.cc

Line 220 in 214fd3c

returnStatus::Invalid("Column data for field ", i, " with type ",

Remove validation checks on table. The validation checks mainly fail for debug builds and what they do is to actually validate the metadata for a table. While dealing with indices for dictionaries, it looks like for compatibility, we accept signed or unsigned uint32 or uint64 integers. However, arrow dictionary builder AppendIndices only accepts signed integers. So, we end up appending signed integers while dictionary schema could still be unsigned integers. So, when the validation happens, it checks if the dictionary schema and the actual column data are the same types. Since it's possible to have uint64 schema with int64 columns, this leads to errors like (and similar error for uint32): ``` Column data for field 0 with type dictionary<values=string, indices=int64, ordered=0> is inconsistent with schema dictionary<values=string, indices=uint64, ordered=0> ```

github-actions · 2025-04-17T06:25:03Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

remove debug table validation checks (cherry picked from commit 210f686)

See #70

bharatnc · 2025-04-17T06:24:35Z

Follow up for previous PR: #69.

Remove debug validation checks on table. The validation checks mainly fail for debug builds and what they do is to actually validate the metadata for a table. While dealing with indices for dictionaries, it looks like for compatibility, we accept signed or unsigned 32 or 64 bit integers. However, arrow dictionary builder AppendIndices only accepts signed integers. So, we end up appending signed integers while dictionary schema could still be unsigned integers. So, when the validation happens, it checks if the dictionary schema and the actual column data are the same types. Since it's possible to have uint64 schema with int64 columns, this leads to errors like (and similar error for uint32):

Column data for field 0 with type dictionary<values=string, indices=int64,
ordered=0> is inconsistent with schema dictionary<values=string,
indices=uint64, ordered=0>

Unfortunately, I couldn't find a sane way to preserve this behavior since the current behavior exists for compatibility with pandas format it looks like. The settings to support both singed and unsigned dictionary indices was added in: ClickHouse/ClickHouse#58519.

This is where the current assert fails:

arrow/cpp/src/arrow/table.cc

Line 181 in 214fd3c

RETURN_NOT_OK(ValidateMeta());

which is ultimately checked here:

arrow/cpp/src/arrow/table.cc

Line 220 in 214fd3c

returnStatus::Invalid("Column data for field ", i, " with type ",

Remove validation checks on table. The validation checks mainly fail for debug builds and what they do is to actually validate the metadata for a table. While dealing with indices for dictionaries, it looks like for compatibility, we accept signed or unsigned uint32 or uint64 integers. However, arrow dictionary builder AppendIndices only accepts signed integers. So, we end up appending signed integers while dictionary schema could still be unsigned integers. So, when the validation happens, it checks if the dictionary schema and the actual column data are the same types. Since it's possible to have uint64 schema with int64 columns, this leads to errors like (and similar error for uint32): ``` Column data for field 0 with type dictionary<values=string, indices=int64, ordered=0> is inconsistent with schema dictionary<values=string, indices=uint64, ordered=0> ```

github-actions · 2025-04-17T06:25:03Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

remove debug table validation checks (cherry picked from commit 210f686)

See #70

bharatnc · 2025-04-17T06:24:35Z

Follow up for previous PR: #69.

Remove debug validation checks on table. The validation checks mainly fail for debug builds and what they do is to actually validate the metadata for a table. While dealing with indices for dictionaries, it looks like for compatibility, we accept signed or unsigned 32 or 64 bit integers. However, arrow dictionary builder AppendIndices only accepts signed integers. So, we end up appending signed integers while dictionary schema could still be unsigned integers. So, when the validation happens, it checks if the dictionary schema and the actual column data are the same types. Since it's possible to have uint64 schema with int64 columns, this leads to errors like (and similar error for uint32):

Column data for field 0 with type dictionary<values=string, indices=int64,
ordered=0> is inconsistent with schema dictionary<values=string,
indices=uint64, ordered=0>

Unfortunately, I couldn't find a sane way to preserve this behavior since the current behavior exists for compatibility with pandas format it looks like. The settings to support both singed and unsigned dictionary indices was added in: ClickHouse/ClickHouse#58519.

This is where the current assert fails:

arrow/cpp/src/arrow/table.cc

Line 181 in 214fd3c

RETURN_NOT_OK(ValidateMeta());

which is ultimately checked here:

arrow/cpp/src/arrow/table.cc

Line 220 in 214fd3c

returnStatus::Invalid("Column data for field ", i, " with type ",

Remove validation checks on table. The validation checks mainly fail for debug builds and what they do is to actually validate the metadata for a table. While dealing with indices for dictionaries, it looks like for compatibility, we accept signed or unsigned uint32 or uint64 integers. However, arrow dictionary builder AppendIndices only accepts signed integers. So, we end up appending signed integers while dictionary schema could still be unsigned integers. So, when the validation happens, it checks if the dictionary schema and the actual column data are the same types. Since it's possible to have uint64 schema with int64 columns, this leads to errors like (and similar error for uint32): ``` Column data for field 0 with type dictionary<values=string, indices=int64, ordered=0> is inconsistent with schema dictionary<values=string, indices=uint64, ordered=0> ```

github-actions · 2025-04-17T06:25:03Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

remove debug table validation checks (cherry picked from commit 210f686)

See #70

bharatnc · 2025-04-17T06:24:35Z

Follow up for previous PR: #69.

Remove debug validation checks on table. The validation checks mainly fail for debug builds and what they do is to actually validate the metadata for a table. While dealing with indices for dictionaries, it looks like for compatibility, we accept signed or unsigned 32 or 64 bit integers. However, arrow dictionary builder AppendIndices only accepts signed integers. So, we end up appending signed integers while dictionary schema could still be unsigned integers. So, when the validation happens, it checks if the dictionary schema and the actual column data are the same types. Since it's possible to have uint64 schema with int64 columns, this leads to errors like (and similar error for uint32):

Column data for field 0 with type dictionary<values=string, indices=int64,
ordered=0> is inconsistent with schema dictionary<values=string,
indices=uint64, ordered=0>

Unfortunately, I couldn't find a sane way to preserve this behavior since the current behavior exists for compatibility with pandas format it looks like. The settings to support both singed and unsigned dictionary indices was added in: ClickHouse/ClickHouse#58519.

This is where the current assert fails:

arrow/cpp/src/arrow/table.cc

Line 181 in 214fd3c

RETURN_NOT_OK(ValidateMeta());

which is ultimately checked here:

arrow/cpp/src/arrow/table.cc

Line 220 in 214fd3c

returnStatus::Invalid("Column data for field ", i, " with type ",

Remove validation checks on table. The validation checks mainly fail for debug builds and what they do is to actually validate the metadata for a table. While dealing with indices for dictionaries, it looks like for compatibility, we accept signed or unsigned uint32 or uint64 integers. However, arrow dictionary builder AppendIndices only accepts signed integers. So, we end up appending signed integers while dictionary schema could still be unsigned integers. So, when the validation happens, it checks if the dictionary schema and the actual column data are the same types. Since it's possible to have uint64 schema with int64 columns, this leads to errors like (and similar error for uint32): ``` Column data for field 0 with type dictionary<values=string, indices=int64, ordered=0> is inconsistent with schema dictionary<values=string, indices=uint64, ordered=0> ```

github-actions · 2025-04-17T06:25:03Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

remove debug table validation checks (cherry picked from commit 210f686)

See #70

bharatnc mentioned this pull request Apr 17, 2025
update arrow submodule for table reader fixes ClickHouse/ClickHouse#78920
Merged
1 task

bharatnc removed Component: C++ Component: Parquet Component: Go awaiting review Component: JavaScript Component: GLib Component: C# Component: MATLAB Component: Python Component: R Component: Ruby Component: Swift labels Apr 17, 2025

bharatnc removed Component: FlightRPC Component: Gandiva Component: Documentation labels Apr 17, 2025

hanfei1991 approved these changes Apr 17, 2025
View reviewed changes

bharatnc merged commit 210f686 into ClickHouse/release/16.1.0Apr 17, 2025

bharatnc deleted the ncb/remove-asserts branch April 17, 2025 20:12

vitlibar pushed a commit that referenced this pull request Feb 9, 2026
Merge pull request#70from ClickHouse/ncb/remove-asserts
b7fa91b
remove debug table validation checks (cherry picked from commit 210f686)

chethan-64 pushed a commit to chethan-64/arrow that referenced this pull request Jun 15, 2026
Merge pull requestClickHouse#70from ClickHouse/ncb/remove-asserts
bc406cb
remove debug table validation checks (cherry picked from commit 210f686)

chethan-64 mentioned this pull request Jun 15, 2026
Click house/apache arrow 23.0.1 #74
Closed

rschu1ze pushed a commit that referenced this pull request Jun 16, 2026
Remove debug table validation checks
15f9be4
See #70