GH-33592: [C++] support casting nullable fields to non-nullable if there are no null values - #43782

Merged
pitrou merged 3 commits into
apache:mainfrom
NickCrews:field-cast-nullable-to-nonnullable
Feb 13, 2025
Merged

GH-33592: [C++] support casting nullable fields to non-nullable if there are no null values#43782
pitrou merged 3 commits into
apache:mainfrom
NickCrews:field-cast-nullable-to-nonnullable

Conversation

@NickCrews

@NickCrewsNickCrews commented Aug 21, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

Notes for myself/fixer:

  • tests that need to get updated (almost definitely not a complete list)
  • [update: actually we should handle the go implementation in the go repository.] hmm, looks like go wrapper does its own nullability checks. I assume this is just an optimization to not have to go into the C++ and hit the error down there. So if we just delete this check then I think the C++ logic will handle all of it???
  • how the plain column handles this cast, some logic like this probably needs to get ported over to the struct implementation
  • (running from /cpp/build) cmake .. --preset ninja-debug-basic, then cmake --build . && PYTHON=python ctest -R 'arrow-compute-scalar-cast-test' --output-on-failure to run the specific test
  • to format: uvx pre-commit run --all-files clang-format

@github-actions

github-actionsBot commented Aug 21, 2024

Copy link
Copy Markdown

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

@NickCrewsNickCrews mentioned this pull request Aug 21, 2024
Closed
@NickCrewsNickCrews changed the title [C++][Go]: support casting nullable fields to non-nullable if there are no null valuesGH-33592 [C++][Go]: support casting nullable fields to non-nullable if there are no null valuesAug 21, 2024
@github-actions

github-actionsBot commented Aug 21, 2024

Copy link
Copy Markdown

⚠️ GitHub issue #33592has been automatically assigned in GitHub to PR creator.

@NickCrews
NickCrewsforce-pushed the field-cast-nullable-to-nonnullable branch 3 times, most recently from 52100dd to 49d5a99CompareAugust 22, 2024 00:40
@rustyconover

rustyconover commented Sep 15, 2024

Copy link
Copy Markdown

This PR looks pretty good to me, what additional help would you like getting it merged?

@NickCrews

NickCrews commented Sep 15, 2024

Copy link
Copy Markdown
ContributorAuthor

Thanks for the help! I'm a random contributor, so I'm not sure about the exact workflow, but I think both getting the CI run approved and having a C++ owner approve it are needed.

Any more detailed thoughts on if the tests are adequate, if there's anywhere else in the code base that you think needs to change, my handling of the go implementation by just deleting the shortcircuit, or any other more detailed thoughts? Basically anything that would reduce the mental load on the code owner I think would increase the odds they approve it :)

@mapleFU

mapleFU commented Sep 15, 2024

Copy link
Copy Markdown
Member

You can mark as ready for review? Or this is still wip?

@NickCrews
NickCrews marked this pull request as ready for review September 15, 2024 20:11
@NickCrews

NickCrews commented Sep 15, 2024

Copy link
Copy Markdown
ContributorAuthor

Oops, didn't mean for this to still be marked WIP. Looks like a few failures that need to get fixed, but still a high-level review of the general approach would still be appreciated in the meantime.

@zeroshade

zeroshade commented Sep 15, 2024

Copy link
Copy Markdown
Member

You probably need the same corresponding check on the Go side to verify that it's only allowed if there are no nulls.

Also, the Go implementation has been moved to the apache/arrow-go repository. So please file the PR there for the Go side. Sorry for the confusion, we just haven't removed the Go code from here yet.

@NickCrews

NickCrews commented Sep 16, 2024

Copy link
Copy Markdown
ContributorAuthor

Thanks @zeroshade , will do. Do we need both PRs to land at about the same time, or can I do that independently?

I will undo my changes to the go code here, leaving it untouched.

@zeroshade

zeroshade commented Sep 16, 2024

Copy link
Copy Markdown
Member

They can land independently, no issues there. Thanks!

@NickCrews
NickCrewsforce-pushed the field-cast-nullable-to-nonnullable branch from 49d5a99 to eb9e7b6CompareSeptember 16, 2024 16:27
@NickCrews

NickCrews commented Sep 16, 2024

Copy link
Copy Markdown
ContributorAuthor

Just pushed a new version:

  • dropped the Go changes
  • rebased on top of 3600db8, which is one commit behind main, because main is failing CI, but the commit I chose passed CI. I did this because I think the failing CI checks in this PR are not related to this PR
  • I discovered archery and formatted the files with archery lint --clang-format --fix

We will see if this passes CI now...

@NickCrews
NickCrewsforce-pushed the field-cast-nullable-to-nonnullable branch from eb9e7b6 to e6d54d3CompareSeptember 18, 2024 07:49
@NickCrews

NickCrews commented Sep 18, 2024

Copy link
Copy Markdown
ContributorAuthor

Pushed a new version, hopefully that fixed the broken tests:

  • To go from nullable to non-nullable type, need to use the unsafe cast option
  • fix typo of int8 to int64 so the precision matches
  • removed the [Go] tag from the commit message

@NickCrews

NickCrews commented Sep 18, 2024

Copy link
Copy Markdown
ContributorAuthor

@zeroshade I think this is ready to review/merge, the failing CI runs look like flakes when trying to setup the environment?

@NickCrewsNickCrews mentioned this pull request Sep 19, 2024
Open
@kou

kou commented Sep 19, 2024

Copy link
Copy Markdown
Member

Can we move this to apache/arrow-go?

@koukou changed the title GH-33592 [C++][Go]: support casting nullable fields to non-nullable if there are no null valuesGH-33592: [C++] support casting nullable fields to non-nullable if there are no null valuesSep 19, 2024
@kou

kou commented Sep 19, 2024

Copy link
Copy Markdown
Member

Ah, the Go part was removed from this PR.
I've removed "[Go]" from the PR title.

@NickCrews
NickCrewsforce-pushed the field-cast-nullable-to-nonnullable branch from e6d54d3 to 106e627CompareSeptember 24, 2024 18:53
@NickCrews
NickCrewsforce-pushed the field-cast-nullable-to-nonnullable branch from b124c64 to ebe9a5aCompareFebruary 7, 2025 08:01
@NickCrews

NickCrews commented Feb 7, 2025

Copy link
Copy Markdown
ContributorAuthor

@mapleFU as I added the slicing tests, I also refactored the tests to make them much more consistent and concise. Take another look and make sure that your "LGTM" still holds after those changes.

pitrou
pitrou reviewed Feb 7, 2025
View reviewed changes

@pitroupitrou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a nit!

Comment threadcpp/src/arrow/compute/kernels/scalar_cast_nested.cc Outdated
@NickCrews
NickCrewsforce-pushed the field-cast-nullable-to-nonnullable branch 2 times, most recently from ce00550 to ea241a5CompareFebruary 10, 2025 16:56
NickCrews
NickCrews commented Feb 10, 2025
View reviewed changes
Comment threadcpp/src/arrow/compute/kernels/scalar_cast_nested.cc Outdated
@NickCrews

NickCrews commented Feb 10, 2025•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

I'm getting these failing UBSAN errors in CI. See below for the relevant logs. Is it because of the new std::vector<std::shared_ptr<Array>> arrays_dest_ac = {arrays_dest[0], arrays_dest[2]} code that I added? It looks like the line numbers referenced are the ones AFTER the macros/preprocessor occurs, so I'm not sure which line in the source file is actually the problem :( Is there a good way to silence/ignore this error? Any hint as to how to run this locally so I can reproduce/test?

Details
[ RUN ] Cast.StructToStructSubset
/usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1064:34: runtime error: addition of unsigned offset to 0x60700006f300 overflowed to 0x60700006f2e0
#0 0x7f811aca4bc8 in std::vector<std::shared_ptr<arrow::Field>, std::allocator<std::shared_ptr<arrow::Field> > >::operator[](unsigned long) const /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1064:34
#1 0x7f811ac9b27a in arrow::DataType::field(int) const /arrow/cpp/src/arrow/type.h:153:61
#2 0x7f8105e4f8f1 in arrow::compute::internal::(anonymous namespace)::CastStruct::Exec(arrow::compute::KernelContext*, arrow::compute::ExecSpan const&, arrow::compute::ExecResult*) /arrow/cpp/src/arrow/compute/kernels/scalar_cast_nested.cc:403:38
#3 0x7f8105ab206e in arrow::compute::detail::(anonymous namespace)::ScalarExecutor::ExecuteNonSpans(arrow::compute::detail::ExecListener*) /arrow/cpp/src/arrow/compute/exec.cc:920:7
#4 0x7f8105aa99fa in arrow::compute::detail::(anonymous namespace)::ScalarExecutor::Execute(arrow::compute::ExecBatch const&, arrow::compute::detail::ExecListener*) /arrow/cpp/src/arrow/compute/exec.cc:810:14
#5 0x7f8105c471a6 in arrow::compute::detail::FunctionExecutorImpl::Execute(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, long) /arrow/cpp/src/arrow/compute/function.cc:278:5
#6 0x7f8105c17bfe in arrow::compute::(anonymous namespace)::ExecuteInternal(arrow::compute::Function const&, std::vector<arrow::Datum, std::allocator<arrow::Datum> >, long, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) /arrow/cpp/src/arrow/compute/function.cc:343:21
#7 0x7f8105c16d84 in arrow::compute::Function::Execute(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) const /arrow/cpp/src/arrow/compute/function.cc:350:10
#8 0x7f8105a5d13c in arrow::compute::internal::(anonymous namespace)::CastMetaFunction::ExecuteImpl(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) const /arrow/cpp/src/arrow/compute/cast.cc:124:23
#9 0x7f8105c21b5c in arrow::compute::MetaFunction::Execute(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) const /arrow/cpp/src/arrow/compute/function.cc:483:10
#10 0x7f8105a8cfc8 in arrow::compute::CallFunction(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) /arrow/cpp/src/arrow/compute/exec.cc:1369:16
#11 0x55f68eeb23c9 in arrow::compute::CheckScalarNonRecursive(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::Datum const&, arrow::compute::FunctionOptions const*) /arrow/cpp/src/arrow/compute/kernels/test_util_internal.cc:80:3
#12 0x55f68eeb8480 in arrow::compute::CheckScalar(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::Datum, arrow::compute::FunctionOptions const*) /arrow/cpp/src/arrow/compute/kernels/test_util_internal.cc:109:3
#13 0x55f68eecc36c in arrow::compute::CheckScalarUnary(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, arrow::Datum, arrow::Datum, arrow::compute::FunctionOptions const*) /arrow/cpp/src/arrow/compute/kernels/test_util_internal.cc:255:3
#14 0x55f68ef4d8f7 in arrow::compute::CheckCast(std::shared_ptr<arrow::Array>, std::shared_ptr<arrow::Array>, arrow::compute::CastOptions) /arrow/cpp/src/arrow/compute/kernels/scalar_cast_test.cc:111:3
#15 0x55f68f079ab8 in arrow::compute::CheckStructToStructSubset(std::vector<std::shared_ptr<arrow::DataType>, std::allocator<std::shared_ptr<arrow::DataType> > > const&) /arrow/cpp/src/arrow/compute/kernels/scalar_cast_test.cc:3829:7
#16 0x55f68f06d09f in arrow::compute::Cast_StructToStructSubset_Test::TestBody() /arrow/cpp/src/arrow/compute/kernels/scalar_cast_test.cc:4003:36
#17 0x7f811af8260e in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/usr/local/lib/libarrow_testing.so.2000+0x108c60e) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#18 0x7f811af770c5 in testing::Test::Run() (/usr/local/lib/libarrow_testing.so.2000+0x10810c5) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#19 0x7f811af77244 in testing::TestInfo::Run() (/usr/local/lib/libarrow_testing.so.2000+0x1081244) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#20 0x7f811af777f8 in testing::TestSuite::Run() (/usr/local/lib/libarrow_testing.so.2000+0x10817f8) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#21 0x7f811af77efe in testing::internal::UnitTestImpl::RunAllTests() (/usr/local/lib/libarrow_testing.so.2000+0x1081efe) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#22 0x7f811af82bd6 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/usr/local/lib/libarrow_testing.so.2000+0x108cbd6) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#23 0x7f811af7730b in testing::UnitTest::Run() (/usr/local/lib/libarrow_testing.so.2000+0x108130b) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#24 0x55f68edefb33 in main (/build/cpp/debug/arrow-compute-scalar-cast-test+0x2adb33) (BuildId: c176b331f0537175685c7888a83fc8e112006f6a)
#0 0x7f80f7d06d8f in
#26 0x7f80f7d06e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: cd410b710f0f094c6832edd95931006d883af48e)
#27 0x55f68edefb94 in _start (/build/cpp/debug/arrow-compute-scalar-cast-test+0x2adb94) (BuildId: c176b331f0537175685c7888a83fc8e112006f6a)
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1064:34 in
/build/cpp/src/arrow/compute/kernels
Start 22: arrow-compute-scalar-cast-test
Test #22: arrow-compute-scalar-cast-test ...............***Failed 12.44 sec

@pitrou

pitrou commented Feb 11, 2025

Copy link
Copy Markdown
Member

Well, the Windows 2019 CI test shows a similar error, so I think it needs diagnosing and solving:

[ RUN ] Cast.StructToStructSubset
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\include\vector(1563) : Assertion failed: vector subscript out of range

pitrou
pitrou reviewed Feb 11, 2025
View reviewed changes
Comment threadcpp/src/arrow/compute/kernels/scalar_cast_nested.cc Outdated
@NickCrews
NickCrewsforce-pushed the field-cast-nullable-to-nonnullable branch 2 times, most recently from 45f1ac2 to 0959343CompareFebruary 11, 2025 17:07
@NickCrews

NickCrews commented Feb 11, 2025

Copy link
Copy Markdown
ContributorAuthor

I rebased on top of #45500 so that hopefully CI will pass

@pitrou
pitrouforce-pushed the field-cast-nullable-to-nonnullable branch from 0959343 to 4311938CompareFebruary 13, 2025 13:46
pitrou
pitrou approved these changes Feb 13, 2025
View reviewed changes

@pitroupitrou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. Thanks for this @NickCrews !

@pitrou

pitrou commented Feb 13, 2025

Copy link
Copy Markdown
Member

@github-actions crossbow submit -g cpp

@github-actions

github-actionsBot commented Feb 13, 2025

Copy link
Copy Markdown

Revision: 4311938

Submitted crossbow builds: ursacomputing/crossbow @ actions-f1ea7df869

TaskStatus
example-cpp-minimal-build-staticGitHub Actions
example-cpp-minimal-build-static-system-dependencyGitHub Actions
example-cpp-tutorialGitHub Actions
test-alpine-linux-cppGitHub Actions
test-build-cpp-fuzzGitHub Actions
test-conda-cppGitHub Actions
test-conda-cpp-valgrindGitHub Actions
test-cuda-cpp-ubuntu-20.04-cuda-11.2.2GitHub Actions
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1GitHub Actions
test-debian-12-cpp-amd64GitHub Actions
test-debian-12-cpp-i386GitHub Actions
test-fedora-39-cppGitHub Actions
test-ubuntu-20.04-cppGitHub Actions
test-ubuntu-20.04-cpp-bundledGitHub Actions
test-ubuntu-22.04-cppGitHub Actions
test-ubuntu-22.04-cpp-20GitHub Actions
test-ubuntu-22.04-cpp-emscriptenGitHub Actions
test-ubuntu-22.04-cpp-no-threadingGitHub Actions
test-ubuntu-24.04-cppGitHub Actions
test-ubuntu-24.04-cpp-bundled-offlineGitHub Actions
test-ubuntu-24.04-cpp-gcc-13-bundledGitHub Actions
test-ubuntu-24.04-cpp-gcc-14GitHub Actions
test-ubuntu-24.04-cpp-minimal-with-formatsGitHub Actions
test-ubuntu-24.04-cpp-thread-sanitizerGitHub Actions

@pitrou

pitrou commented Feb 13, 2025

Copy link
Copy Markdown
Member

The CI failures are unrelated (see #45524).

@pitrou
pitrou merged commit 6a47e4d into apache:mainFeb 13, 2025
@pitroupitrou removed the awaiting committer review Awaiting committer review label Feb 13, 2025
@NickCrews

NickCrews commented Feb 13, 2025

Copy link
Copy Markdown
ContributorAuthor

Thank you everyone! That was a lot of effort everyone did, I wouldn't have been able to do that without all the help. I'm excited for this to be released which will allow me to remove some hacky workarounds I have in my app code.

@conbench-apache-arrow

conbench-apache-arrowBot commented Feb 13, 2025

Copy link
Copy Markdown

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 6a47e4d.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.

@Tom-NewtonTom-Newton mentioned this pull request Apr 23, 2025
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zanmato1984zanmato1984zanmato1984 left review comments
@pitroupitroupitrou approved these changes
@mapleFUmapleFUmapleFU approved these changes
@zeroshadezeroshadeAwaiting requested review from zeroshade
@assignUserassignUserAwaiting requested review from assignUser
@jonkeanejonkeaneAwaiting requested review from jonkeane
@koukouAwaiting requested review from kou
@raulcdraulcdAwaiting requested review from raulcd

Assignees

No one assigned

Labels

Component: C++

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@NickCrews@rustyconover@mapleFU@zeroshade@kou@pitrou@zanmato1984
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
GH-33592: [C++] support casting nullable fields to non-nullable if there are no null values - #43782
Merged
pitrou merged 3 commits into
apache:mainfrom 
NickCrews:field-cast-nullable-to-nonnullable
Feb 13, 2025
Merged
GH-33592: [C++] support casting nullable fields to non-nullable if there are no null values#43782
pitrou merged 3 commits into
apache:mainfrom 
NickCrews:field-cast-nullable-to-nonnullable
Conversation
@NickCrews
@NickCrewsNickCrews
commented
Aug 21, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Notes for myself/fixer:
  • tests that need to get updated (almost definitely not a complete list)
  • [update: actually we should handle the go implementation in the go repository.] hmm, looks like go wrapper does its own nullability checks. I assume this is just an optimization to not have to go into the C++ and hit the error down there. So if we just delete this check then I think the C++ logic will handle all of it???
  • how the plain column handles this cast, some logic like this probably needs to get ported over to the struct implementation
  • (running from /cpp/build) cmake .. --preset ninja-debug-basic, then cmake --build . && PYTHON=python ctest -R 'arrow-compute-scalar-cast-test' --output-on-failure to run the specific test
  • to format: uvx pre-commit run --all-files clang-format
@github-actions
github-actionsBot
commented
Aug 21, 2024

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:
@NickCrewsNickCrews
mentioned this pull request
Aug 21, 2024

Closed

@NickCrewsNickCrews
changed the title
[C++][Go]: support casting nullable fields to non-nullable if there are no null valuesGH-33592 [C++][Go]: support casting nullable fields to non-nullable if there are no null valuesAug 21, 2024
@github-actions
github-actionsBot
commented
Aug 21, 2024

Copy link


Copy Markdown

⚠️ GitHub issue #33592has been automatically assigned in GitHub to PR creator.
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
3 times, most recently
from
52100dd to
49d5a99CompareAugust 22, 2024 00:40
@rustyconover
rustyconover
commented
Sep 15, 2024

Copy link


Copy Markdown

This PR looks pretty good to me, what additional help would you like getting it merged?
@NickCrews
NickCrews
commented
Sep 15, 2024

Copy link


Copy Markdown

ContributorAuthor
Thanks for the help! I'm a random contributor, so I'm not sure about the exact workflow, but I think both getting the CI run approved and having a C++ owner approve it are needed.
Any more detailed thoughts on if the tests are adequate, if there's anywhere else in the code base that you think needs to change, my handling of the go implementation by just deleting the shortcircuit, or any other more detailed thoughts? Basically anything that would reduce the mental load on the code owner I think would increase the odds they approve it :)
@mapleFU
mapleFU
commented
Sep 15, 2024

Copy link


Copy Markdown

Member
You can mark as ready for review? Or this is still wip?
@NickCrews
NickCrews
marked this pull request as ready for review
September 15, 2024 20:11
@NickCrews
NickCrews
commented
Sep 15, 2024

Copy link


Copy Markdown

ContributorAuthor
Oops, didn't mean for this to still be marked WIP. Looks like a few failures that need to get fixed, but still a high-level review of the general approach would still be appreciated in the meantime.
@zeroshade
zeroshade
commented
Sep 15, 2024

Copy link


Copy Markdown

Member
You probably need the same corresponding check on the Go side to verify that it's only allowed if there are no nulls.
Also, the Go implementation has been moved to the apache/arrow-go repository. So please file the PR there for the Go side. Sorry for the confusion, we just haven't removed the Go code from here yet.
@NickCrews
NickCrews
commented
Sep 16, 2024

Copy link


Copy Markdown

ContributorAuthor
Thanks @zeroshade , will do. Do we need both PRs to land at about the same time, or can I do that independently?
I will undo my changes to the go code here, leaving it untouched.
@zeroshade
zeroshade
commented
Sep 16, 2024

Copy link


Copy Markdown

Member
They can land independently, no issues there. Thanks!
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
49d5a99 to
eb9e7b6CompareSeptember 16, 2024 16:27
@NickCrews
NickCrews
commented
Sep 16, 2024

Copy link


Copy Markdown

ContributorAuthor
Just pushed a new version:
  • dropped the Go changes
  • rebased on top of 3600db8, which is one commit behind main, because main is failing CI, but the commit I chose passed CI. I did this because I think the failing CI checks in this PR are not related to this PR
  • I discovered archery and formatted the files with archery lint --clang-format --fix
We will see if this passes CI now...
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
eb9e7b6 to
e6d54d3CompareSeptember 18, 2024 07:49
@NickCrews
NickCrews
commented
Sep 18, 2024

Copy link


Copy Markdown

ContributorAuthor
Pushed a new version, hopefully that fixed the broken tests:
  • To go from nullable to non-nullable type, need to use the unsafe cast option
  • fix typo of int8 to int64 so the precision matches
  • removed the [Go] tag from the commit message
@NickCrews
NickCrews
commented
Sep 18, 2024

Copy link


Copy Markdown

ContributorAuthor
@zeroshade I think this is ready to review/merge, the failing CI runs look like flakes when trying to setup the environment?
@NickCrewsNickCrews
mentioned this pull request
Sep 19, 2024

Open

@kou
kou
commented
Sep 19, 2024

Copy link


Copy Markdown

Member
Can we move this to apache/arrow-go?
@koukou
changed the title
GH-33592 [C++][Go]: support casting nullable fields to non-nullable if there are no null valuesGH-33592: [C++] support casting nullable fields to non-nullable if there are no null valuesSep 19, 2024
@kou
kou
commented
Sep 19, 2024

Copy link


Copy Markdown

Member
Ah, the Go part was removed from this PR.

I've removed "[Go]" from the PR title.
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
e6d54d3 to
106e627CompareSeptember 24, 2024 18:53
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
b124c64 to
ebe9a5aCompareFebruary 7, 2025 08:01
@NickCrews
NickCrews
commented
Feb 7, 2025

Copy link


Copy Markdown

ContributorAuthor
@mapleFU as I added the slicing tests, I also refactored the tests to make them much more consistent and concise. Take another look and make sure that your "LGTM" still holds after those changes.
pitrou
pitrou
reviewed
Feb 7, 2025
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a nit!
Comment threadcpp/src/arrow/compute/kernels/scalar_cast_nested.cc
Outdated

@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
2 times, most recently
from
ce00550 to
ea241a5CompareFebruary 10, 2025 16:56
NickCrews
NickCrews
commented
Feb 10, 2025
View reviewed changes
Comment threadcpp/src/arrow/compute/kernels/scalar_cast_nested.cc
Outdated

@NickCrews
NickCrews
commented
Feb 10, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
I'm getting these failing UBSAN errors in CI. See below for the relevant logs. Is it because of the new std::vector<std::shared_ptr<Array>> arrays_dest_ac = {arrays_dest[0], arrays_dest[2]} code that I added? It looks like the line numbers referenced are the ones AFTER the macros/preprocessor occurs, so I'm not sure which line in the source file is actually the problem :( Is there a good way to silence/ignore this error? Any hint as to how to run this locally so I can reproduce/test?
Details
[ RUN ] Cast.StructToStructSubset
/usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1064:34: runtime error: addition of unsigned offset to 0x60700006f300 overflowed to 0x60700006f2e0
#0 0x7f811aca4bc8 in std::vector<std::shared_ptr<arrow::Field>, std::allocator<std::shared_ptr<arrow::Field> > >::operator[](unsigned long) const /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1064:34
#1 0x7f811ac9b27a in arrow::DataType::field(int) const /arrow/cpp/src/arrow/type.h:153:61
#2 0x7f8105e4f8f1 in arrow::compute::internal::(anonymous namespace)::CastStruct::Exec(arrow::compute::KernelContext*, arrow::compute::ExecSpan const&, arrow::compute::ExecResult*) /arrow/cpp/src/arrow/compute/kernels/scalar_cast_nested.cc:403:38
#3 0x7f8105ab206e in arrow::compute::detail::(anonymous namespace)::ScalarExecutor::ExecuteNonSpans(arrow::compute::detail::ExecListener*) /arrow/cpp/src/arrow/compute/exec.cc:920:7
#4 0x7f8105aa99fa in arrow::compute::detail::(anonymous namespace)::ScalarExecutor::Execute(arrow::compute::ExecBatch const&, arrow::compute::detail::ExecListener*) /arrow/cpp/src/arrow/compute/exec.cc:810:14
#5 0x7f8105c471a6 in arrow::compute::detail::FunctionExecutorImpl::Execute(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, long) /arrow/cpp/src/arrow/compute/function.cc:278:5
#6 0x7f8105c17bfe in arrow::compute::(anonymous namespace)::ExecuteInternal(arrow::compute::Function const&, std::vector<arrow::Datum, std::allocator<arrow::Datum> >, long, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) /arrow/cpp/src/arrow/compute/function.cc:343:21
#7 0x7f8105c16d84 in arrow::compute::Function::Execute(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) const /arrow/cpp/src/arrow/compute/function.cc:350:10
#8 0x7f8105a5d13c in arrow::compute::internal::(anonymous namespace)::CastMetaFunction::ExecuteImpl(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) const /arrow/cpp/src/arrow/compute/cast.cc:124:23
#9 0x7f8105c21b5c in arrow::compute::MetaFunction::Execute(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) const /arrow/cpp/src/arrow/compute/function.cc:483:10
#10 0x7f8105a8cfc8 in arrow::compute::CallFunction(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) /arrow/cpp/src/arrow/compute/exec.cc:1369:16
#11 0x55f68eeb23c9 in arrow::compute::CheckScalarNonRecursive(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::Datum const&, arrow::compute::FunctionOptions const*) /arrow/cpp/src/arrow/compute/kernels/test_util_internal.cc:80:3
#12 0x55f68eeb8480 in arrow::compute::CheckScalar(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::Datum, arrow::compute::FunctionOptions const*) /arrow/cpp/src/arrow/compute/kernels/test_util_internal.cc:109:3
#13 0x55f68eecc36c in arrow::compute::CheckScalarUnary(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, arrow::Datum, arrow::Datum, arrow::compute::FunctionOptions const*) /arrow/cpp/src/arrow/compute/kernels/test_util_internal.cc:255:3
#14 0x55f68ef4d8f7 in arrow::compute::CheckCast(std::shared_ptr<arrow::Array>, std::shared_ptr<arrow::Array>, arrow::compute::CastOptions) /arrow/cpp/src/arrow/compute/kernels/scalar_cast_test.cc:111:3
#15 0x55f68f079ab8 in arrow::compute::CheckStructToStructSubset(std::vector<std::shared_ptr<arrow::DataType>, std::allocator<std::shared_ptr<arrow::DataType> > > const&) /arrow/cpp/src/arrow/compute/kernels/scalar_cast_test.cc:3829:7
#16 0x55f68f06d09f in arrow::compute::Cast_StructToStructSubset_Test::TestBody() /arrow/cpp/src/arrow/compute/kernels/scalar_cast_test.cc:4003:36
#17 0x7f811af8260e in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/usr/local/lib/libarrow_testing.so.2000+0x108c60e) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#18 0x7f811af770c5 in testing::Test::Run() (/usr/local/lib/libarrow_testing.so.2000+0x10810c5) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#19 0x7f811af77244 in testing::TestInfo::Run() (/usr/local/lib/libarrow_testing.so.2000+0x1081244) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#20 0x7f811af777f8 in testing::TestSuite::Run() (/usr/local/lib/libarrow_testing.so.2000+0x10817f8) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#21 0x7f811af77efe in testing::internal::UnitTestImpl::RunAllTests() (/usr/local/lib/libarrow_testing.so.2000+0x1081efe) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#22 0x7f811af82bd6 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/usr/local/lib/libarrow_testing.so.2000+0x108cbd6) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#23 0x7f811af7730b in testing::UnitTest::Run() (/usr/local/lib/libarrow_testing.so.2000+0x108130b) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#24 0x55f68edefb33 in main (/build/cpp/debug/arrow-compute-scalar-cast-test+0x2adb33) (BuildId: c176b331f0537175685c7888a83fc8e112006f6a)
#0 0x7f80f7d06d8f in
#26 0x7f80f7d06e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: cd410b710f0f094c6832edd95931006d883af48e)
#27 0x55f68edefb94 in _start (/build/cpp/debug/arrow-compute-scalar-cast-test+0x2adb94) (BuildId: c176b331f0537175685c7888a83fc8e112006f6a)
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1064:34 in
/build/cpp/src/arrow/compute/kernels
Start 22: arrow-compute-scalar-cast-test
Test #22: arrow-compute-scalar-cast-test ...............***Failed 12.44 sec

@pitrou
pitrou
commented
Feb 11, 2025

Copy link


Copy Markdown

Member
Well, the Windows 2019 CI test shows a similar error, so I think it needs diagnosing and solving:
[ RUN ] Cast.StructToStructSubset
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\include\vector(1563) : Assertion failed: vector subscript out of range

pitrou
pitrou
reviewed
Feb 11, 2025
View reviewed changes
Comment threadcpp/src/arrow/compute/kernels/scalar_cast_nested.cc
Outdated

@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
2 times, most recently
from
45f1ac2 to
0959343CompareFebruary 11, 2025 17:07
@NickCrews
NickCrews
commented
Feb 11, 2025

Copy link


Copy Markdown

ContributorAuthor
I rebased on top of #45500 so that hopefully CI will pass
@pitrou
pitrouforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
0959343 to
4311938CompareFebruary 13, 2025 13:46
pitrou
pitrou
approved these changes
Feb 13, 2025
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. Thanks for this @NickCrews !
@pitrou
pitrou
commented
Feb 13, 2025

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g cpp
@github-actions
github-actionsBot
commented
Feb 13, 2025

Copy link


Copy Markdown

Revision: 4311938
Submitted crossbow builds: ursacomputing/crossbow @ actions-f1ea7df869
TaskStatus
example-cpp-minimal-build-staticGitHub Actions
example-cpp-minimal-build-static-system-dependencyGitHub Actions
example-cpp-tutorialGitHub Actions
test-alpine-linux-cppGitHub Actions
test-build-cpp-fuzzGitHub Actions
test-conda-cppGitHub Actions
test-conda-cpp-valgrindGitHub Actions
test-cuda-cpp-ubuntu-20.04-cuda-11.2.2GitHub Actions
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1GitHub Actions
test-debian-12-cpp-amd64GitHub Actions
test-debian-12-cpp-i386GitHub Actions
test-fedora-39-cppGitHub Actions
test-ubuntu-20.04-cppGitHub Actions
test-ubuntu-20.04-cpp-bundledGitHub Actions
test-ubuntu-22.04-cppGitHub Actions
test-ubuntu-22.04-cpp-20GitHub Actions
test-ubuntu-22.04-cpp-emscriptenGitHub Actions
test-ubuntu-22.04-cpp-no-threadingGitHub Actions
test-ubuntu-24.04-cppGitHub Actions
test-ubuntu-24.04-cpp-bundled-offlineGitHub Actions
test-ubuntu-24.04-cpp-gcc-13-bundledGitHub Actions
test-ubuntu-24.04-cpp-gcc-14GitHub Actions
test-ubuntu-24.04-cpp-minimal-with-formatsGitHub Actions
test-ubuntu-24.04-cpp-thread-sanitizerGitHub Actions
@pitrou
pitrou
commented
Feb 13, 2025

Copy link


Copy Markdown

Member
The CI failures are unrelated (see #45524).
@pitrou
pitrou
merged commit 6a47e4d
into
apache:mainFeb 13, 2025
@pitroupitrou
removed
the 
awaiting committer review
Awaiting committer review
label
Feb 13, 2025
@NickCrews
NickCrews
commented
Feb 13, 2025

Copy link


Copy Markdown

ContributorAuthor
Thank you everyone! That was a lot of effort everyone did, I wouldn't have been able to do that without all the help. I'm excited for this to be released which will allow me to remove some hacky workarounds I have in my app code.
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Feb 13, 2025

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 6a47e4d.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.
@Tom-NewtonTom-Newton
mentioned this pull request
Apr 23, 2025
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@zanmato1984zanmato1984zanmato1984 left review comments
@pitroupitroupitrou approved these changes
@mapleFUmapleFUmapleFU approved these changes
@zeroshadezeroshadeAwaiting requested review from zeroshade
@assignUserassignUserAwaiting requested review from assignUser
@jonkeanejonkeaneAwaiting requested review from jonkeane
@koukouAwaiting requested review from kou
@raulcdraulcdAwaiting requested review from raulcd

Assignees


No one assigned


Labels

Component: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

7 participants

@NickCrews@rustyconover@mapleFU@zeroshade@kou@pitrou@zanmato1984

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-33592: [C++] support casting nullable fields to non-nullable if there are no null values - #43782
Merged
pitrou merged 3 commits into
apache:mainfrom 
NickCrews:field-cast-nullable-to-nonnullable
Feb 13, 2025
Merged
GH-33592: [C++] support casting nullable fields to non-nullable if there are no null values#43782
pitrou merged 3 commits into
apache:mainfrom 
NickCrews:field-cast-nullable-to-nonnullable
Conversation
@NickCrews
@NickCrewsNickCrews
commented
Aug 21, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Notes for myself/fixer:
  • tests that need to get updated (almost definitely not a complete list)
  • [update: actually we should handle the go implementation in the go repository.] hmm, looks like go wrapper does its own nullability checks. I assume this is just an optimization to not have to go into the C++ and hit the error down there. So if we just delete this check then I think the C++ logic will handle all of it???
  • how the plain column handles this cast, some logic like this probably needs to get ported over to the struct implementation
  • (running from /cpp/build) cmake .. --preset ninja-debug-basic, then cmake --build . && PYTHON=python ctest -R 'arrow-compute-scalar-cast-test' --output-on-failure to run the specific test
  • to format: uvx pre-commit run --all-files clang-format
@github-actions
github-actionsBot
commented
Aug 21, 2024

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:
@NickCrewsNickCrews
mentioned this pull request
Aug 21, 2024

Closed

@NickCrewsNickCrews
changed the title
[C++][Go]: support casting nullable fields to non-nullable if there are no null valuesGH-33592 [C++][Go]: support casting nullable fields to non-nullable if there are no null valuesAug 21, 2024
@github-actions
github-actionsBot
commented
Aug 21, 2024

Copy link


Copy Markdown

⚠️ GitHub issue #33592has been automatically assigned in GitHub to PR creator.
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
3 times, most recently
from
52100dd to
49d5a99CompareAugust 22, 2024 00:40
@rustyconover
rustyconover
commented
Sep 15, 2024

Copy link


Copy Markdown

This PR looks pretty good to me, what additional help would you like getting it merged?
@NickCrews
NickCrews
commented
Sep 15, 2024

Copy link


Copy Markdown

ContributorAuthor
Thanks for the help! I'm a random contributor, so I'm not sure about the exact workflow, but I think both getting the CI run approved and having a C++ owner approve it are needed.
Any more detailed thoughts on if the tests are adequate, if there's anywhere else in the code base that you think needs to change, my handling of the go implementation by just deleting the shortcircuit, or any other more detailed thoughts? Basically anything that would reduce the mental load on the code owner I think would increase the odds they approve it :)
@mapleFU
mapleFU
commented
Sep 15, 2024

Copy link


Copy Markdown

Member
You can mark as ready for review? Or this is still wip?
@NickCrews
NickCrews
marked this pull request as ready for review
September 15, 2024 20:11
@NickCrews
NickCrews
commented
Sep 15, 2024

Copy link


Copy Markdown

ContributorAuthor
Oops, didn't mean for this to still be marked WIP. Looks like a few failures that need to get fixed, but still a high-level review of the general approach would still be appreciated in the meantime.
@zeroshade
zeroshade
commented
Sep 15, 2024

Copy link


Copy Markdown

Member
You probably need the same corresponding check on the Go side to verify that it's only allowed if there are no nulls.
Also, the Go implementation has been moved to the apache/arrow-go repository. So please file the PR there for the Go side. Sorry for the confusion, we just haven't removed the Go code from here yet.
@NickCrews
NickCrews
commented
Sep 16, 2024

Copy link


Copy Markdown

ContributorAuthor
Thanks @zeroshade , will do. Do we need both PRs to land at about the same time, or can I do that independently?
I will undo my changes to the go code here, leaving it untouched.
@zeroshade
zeroshade
commented
Sep 16, 2024

Copy link


Copy Markdown

Member
They can land independently, no issues there. Thanks!
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
49d5a99 to
eb9e7b6CompareSeptember 16, 2024 16:27
@NickCrews
NickCrews
commented
Sep 16, 2024

Copy link


Copy Markdown

ContributorAuthor
Just pushed a new version:
  • dropped the Go changes
  • rebased on top of 3600db8, which is one commit behind main, because main is failing CI, but the commit I chose passed CI. I did this because I think the failing CI checks in this PR are not related to this PR
  • I discovered archery and formatted the files with archery lint --clang-format --fix
We will see if this passes CI now...
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
eb9e7b6 to
e6d54d3CompareSeptember 18, 2024 07:49
@NickCrews
NickCrews
commented
Sep 18, 2024

Copy link


Copy Markdown

ContributorAuthor
Pushed a new version, hopefully that fixed the broken tests:
  • To go from nullable to non-nullable type, need to use the unsafe cast option
  • fix typo of int8 to int64 so the precision matches
  • removed the [Go] tag from the commit message
@NickCrews
NickCrews
commented
Sep 18, 2024

Copy link


Copy Markdown

ContributorAuthor
@zeroshade I think this is ready to review/merge, the failing CI runs look like flakes when trying to setup the environment?
@NickCrewsNickCrews
mentioned this pull request
Sep 19, 2024

Open

@kou
kou
commented
Sep 19, 2024

Copy link


Copy Markdown

Member
Can we move this to apache/arrow-go?
@koukou
changed the title
GH-33592 [C++][Go]: support casting nullable fields to non-nullable if there are no null valuesGH-33592: [C++] support casting nullable fields to non-nullable if there are no null valuesSep 19, 2024
@kou
kou
commented
Sep 19, 2024

Copy link


Copy Markdown

Member
Ah, the Go part was removed from this PR.

I've removed "[Go]" from the PR title.
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
e6d54d3 to
106e627CompareSeptember 24, 2024 18:53
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
b124c64 to
ebe9a5aCompareFebruary 7, 2025 08:01
@NickCrews
NickCrews
commented
Feb 7, 2025

Copy link


Copy Markdown

ContributorAuthor
@mapleFU as I added the slicing tests, I also refactored the tests to make them much more consistent and concise. Take another look and make sure that your "LGTM" still holds after those changes.
pitrou
pitrou
reviewed
Feb 7, 2025
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a nit!
Comment threadcpp/src/arrow/compute/kernels/scalar_cast_nested.cc
Outdated

@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
2 times, most recently
from
ce00550 to
ea241a5CompareFebruary 10, 2025 16:56
NickCrews
NickCrews
commented
Feb 10, 2025
View reviewed changes
Comment threadcpp/src/arrow/compute/kernels/scalar_cast_nested.cc
Outdated

@NickCrews
NickCrews
commented
Feb 10, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
I'm getting these failing UBSAN errors in CI. See below for the relevant logs. Is it because of the new std::vector<std::shared_ptr<Array>> arrays_dest_ac = {arrays_dest[0], arrays_dest[2]} code that I added? It looks like the line numbers referenced are the ones AFTER the macros/preprocessor occurs, so I'm not sure which line in the source file is actually the problem :( Is there a good way to silence/ignore this error? Any hint as to how to run this locally so I can reproduce/test?
Details
[ RUN ] Cast.StructToStructSubset
/usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1064:34: runtime error: addition of unsigned offset to 0x60700006f300 overflowed to 0x60700006f2e0
#0 0x7f811aca4bc8 in std::vector<std::shared_ptr<arrow::Field>, std::allocator<std::shared_ptr<arrow::Field> > >::operator[](unsigned long) const /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1064:34
#1 0x7f811ac9b27a in arrow::DataType::field(int) const /arrow/cpp/src/arrow/type.h:153:61
#2 0x7f8105e4f8f1 in arrow::compute::internal::(anonymous namespace)::CastStruct::Exec(arrow::compute::KernelContext*, arrow::compute::ExecSpan const&, arrow::compute::ExecResult*) /arrow/cpp/src/arrow/compute/kernels/scalar_cast_nested.cc:403:38
#3 0x7f8105ab206e in arrow::compute::detail::(anonymous namespace)::ScalarExecutor::ExecuteNonSpans(arrow::compute::detail::ExecListener*) /arrow/cpp/src/arrow/compute/exec.cc:920:7
#4 0x7f8105aa99fa in arrow::compute::detail::(anonymous namespace)::ScalarExecutor::Execute(arrow::compute::ExecBatch const&, arrow::compute::detail::ExecListener*) /arrow/cpp/src/arrow/compute/exec.cc:810:14
#5 0x7f8105c471a6 in arrow::compute::detail::FunctionExecutorImpl::Execute(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, long) /arrow/cpp/src/arrow/compute/function.cc:278:5
#6 0x7f8105c17bfe in arrow::compute::(anonymous namespace)::ExecuteInternal(arrow::compute::Function const&, std::vector<arrow::Datum, std::allocator<arrow::Datum> >, long, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) /arrow/cpp/src/arrow/compute/function.cc:343:21
#7 0x7f8105c16d84 in arrow::compute::Function::Execute(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) const /arrow/cpp/src/arrow/compute/function.cc:350:10
#8 0x7f8105a5d13c in arrow::compute::internal::(anonymous namespace)::CastMetaFunction::ExecuteImpl(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) const /arrow/cpp/src/arrow/compute/cast.cc:124:23
#9 0x7f8105c21b5c in arrow::compute::MetaFunction::Execute(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) const /arrow/cpp/src/arrow/compute/function.cc:483:10
#10 0x7f8105a8cfc8 in arrow::compute::CallFunction(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) /arrow/cpp/src/arrow/compute/exec.cc:1369:16
#11 0x55f68eeb23c9 in arrow::compute::CheckScalarNonRecursive(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::Datum const&, arrow::compute::FunctionOptions const*) /arrow/cpp/src/arrow/compute/kernels/test_util_internal.cc:80:3
#12 0x55f68eeb8480 in arrow::compute::CheckScalar(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::Datum, arrow::compute::FunctionOptions const*) /arrow/cpp/src/arrow/compute/kernels/test_util_internal.cc:109:3
#13 0x55f68eecc36c in arrow::compute::CheckScalarUnary(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, arrow::Datum, arrow::Datum, arrow::compute::FunctionOptions const*) /arrow/cpp/src/arrow/compute/kernels/test_util_internal.cc:255:3
#14 0x55f68ef4d8f7 in arrow::compute::CheckCast(std::shared_ptr<arrow::Array>, std::shared_ptr<arrow::Array>, arrow::compute::CastOptions) /arrow/cpp/src/arrow/compute/kernels/scalar_cast_test.cc:111:3
#15 0x55f68f079ab8 in arrow::compute::CheckStructToStructSubset(std::vector<std::shared_ptr<arrow::DataType>, std::allocator<std::shared_ptr<arrow::DataType> > > const&) /arrow/cpp/src/arrow/compute/kernels/scalar_cast_test.cc:3829:7
#16 0x55f68f06d09f in arrow::compute::Cast_StructToStructSubset_Test::TestBody() /arrow/cpp/src/arrow/compute/kernels/scalar_cast_test.cc:4003:36
#17 0x7f811af8260e in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/usr/local/lib/libarrow_testing.so.2000+0x108c60e) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#18 0x7f811af770c5 in testing::Test::Run() (/usr/local/lib/libarrow_testing.so.2000+0x10810c5) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#19 0x7f811af77244 in testing::TestInfo::Run() (/usr/local/lib/libarrow_testing.so.2000+0x1081244) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#20 0x7f811af777f8 in testing::TestSuite::Run() (/usr/local/lib/libarrow_testing.so.2000+0x10817f8) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#21 0x7f811af77efe in testing::internal::UnitTestImpl::RunAllTests() (/usr/local/lib/libarrow_testing.so.2000+0x1081efe) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#22 0x7f811af82bd6 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/usr/local/lib/libarrow_testing.so.2000+0x108cbd6) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#23 0x7f811af7730b in testing::UnitTest::Run() (/usr/local/lib/libarrow_testing.so.2000+0x108130b) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#24 0x55f68edefb33 in main (/build/cpp/debug/arrow-compute-scalar-cast-test+0x2adb33) (BuildId: c176b331f0537175685c7888a83fc8e112006f6a)
#0 0x7f80f7d06d8f in
#26 0x7f80f7d06e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: cd410b710f0f094c6832edd95931006d883af48e)
#27 0x55f68edefb94 in _start (/build/cpp/debug/arrow-compute-scalar-cast-test+0x2adb94) (BuildId: c176b331f0537175685c7888a83fc8e112006f6a)
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1064:34 in
/build/cpp/src/arrow/compute/kernels
Start 22: arrow-compute-scalar-cast-test
Test #22: arrow-compute-scalar-cast-test ...............***Failed 12.44 sec

@pitrou
pitrou
commented
Feb 11, 2025

Copy link


Copy Markdown

Member
Well, the Windows 2019 CI test shows a similar error, so I think it needs diagnosing and solving:
[ RUN ] Cast.StructToStructSubset
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\include\vector(1563) : Assertion failed: vector subscript out of range

pitrou
pitrou
reviewed
Feb 11, 2025
View reviewed changes
Comment threadcpp/src/arrow/compute/kernels/scalar_cast_nested.cc
Outdated

@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
2 times, most recently
from
45f1ac2 to
0959343CompareFebruary 11, 2025 17:07
@NickCrews
NickCrews
commented
Feb 11, 2025

Copy link


Copy Markdown

ContributorAuthor
I rebased on top of #45500 so that hopefully CI will pass
@pitrou
pitrouforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
0959343 to
4311938CompareFebruary 13, 2025 13:46
pitrou
pitrou
approved these changes
Feb 13, 2025
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. Thanks for this @NickCrews !
@pitrou
pitrou
commented
Feb 13, 2025

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g cpp
@github-actions
github-actionsBot
commented
Feb 13, 2025

Copy link


Copy Markdown

Revision: 4311938
Submitted crossbow builds: ursacomputing/crossbow @ actions-f1ea7df869
TaskStatus
example-cpp-minimal-build-staticGitHub Actions
example-cpp-minimal-build-static-system-dependencyGitHub Actions
example-cpp-tutorialGitHub Actions
test-alpine-linux-cppGitHub Actions
test-build-cpp-fuzzGitHub Actions
test-conda-cppGitHub Actions
test-conda-cpp-valgrindGitHub Actions
test-cuda-cpp-ubuntu-20.04-cuda-11.2.2GitHub Actions
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1GitHub Actions
test-debian-12-cpp-amd64GitHub Actions
test-debian-12-cpp-i386GitHub Actions
test-fedora-39-cppGitHub Actions
test-ubuntu-20.04-cppGitHub Actions
test-ubuntu-20.04-cpp-bundledGitHub Actions
test-ubuntu-22.04-cppGitHub Actions
test-ubuntu-22.04-cpp-20GitHub Actions
test-ubuntu-22.04-cpp-emscriptenGitHub Actions
test-ubuntu-22.04-cpp-no-threadingGitHub Actions
test-ubuntu-24.04-cppGitHub Actions
test-ubuntu-24.04-cpp-bundled-offlineGitHub Actions
test-ubuntu-24.04-cpp-gcc-13-bundledGitHub Actions
test-ubuntu-24.04-cpp-gcc-14GitHub Actions
test-ubuntu-24.04-cpp-minimal-with-formatsGitHub Actions
test-ubuntu-24.04-cpp-thread-sanitizerGitHub Actions
@pitrou
pitrou
commented
Feb 13, 2025

Copy link


Copy Markdown

Member
The CI failures are unrelated (see #45524).
@pitrou
pitrou
merged commit 6a47e4d
into
apache:mainFeb 13, 2025
@pitroupitrou
removed
the 
awaiting committer review
Awaiting committer review
label
Feb 13, 2025
@NickCrews
NickCrews
commented
Feb 13, 2025

Copy link


Copy Markdown

ContributorAuthor
Thank you everyone! That was a lot of effort everyone did, I wouldn't have been able to do that without all the help. I'm excited for this to be released which will allow me to remove some hacky workarounds I have in my app code.
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Feb 13, 2025

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 6a47e4d.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.
@Tom-NewtonTom-Newton
mentioned this pull request
Apr 23, 2025
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@zanmato1984zanmato1984zanmato1984 left review comments
@pitroupitroupitrou approved these changes
@mapleFUmapleFUmapleFU approved these changes
@zeroshadezeroshadeAwaiting requested review from zeroshade
@assignUserassignUserAwaiting requested review from assignUser
@jonkeanejonkeaneAwaiting requested review from jonkeane
@koukouAwaiting requested review from kou
@raulcdraulcdAwaiting requested review from raulcd

Assignees


No one assigned


Labels

Component: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

7 participants

@NickCrews@rustyconover@mapleFU@zeroshade@kou@pitrou@zanmato1984

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-33592: [C++] support casting nullable fields to non-nullable if there are no null values - #43782
Merged
pitrou merged 3 commits into
apache:mainfrom 
NickCrews:field-cast-nullable-to-nonnullable
Feb 13, 2025
Merged
GH-33592: [C++] support casting nullable fields to non-nullable if there are no null values#43782
pitrou merged 3 commits into
apache:mainfrom 
NickCrews:field-cast-nullable-to-nonnullable
Conversation
@NickCrews
@NickCrewsNickCrews
commented
Aug 21, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Notes for myself/fixer:
  • tests that need to get updated (almost definitely not a complete list)
  • [update: actually we should handle the go implementation in the go repository.] hmm, looks like go wrapper does its own nullability checks. I assume this is just an optimization to not have to go into the C++ and hit the error down there. So if we just delete this check then I think the C++ logic will handle all of it???
  • how the plain column handles this cast, some logic like this probably needs to get ported over to the struct implementation
  • (running from /cpp/build) cmake .. --preset ninja-debug-basic, then cmake --build . && PYTHON=python ctest -R 'arrow-compute-scalar-cast-test' --output-on-failure to run the specific test
  • to format: uvx pre-commit run --all-files clang-format
@github-actions
github-actionsBot
commented
Aug 21, 2024

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:
@NickCrewsNickCrews
mentioned this pull request
Aug 21, 2024

Closed

@NickCrewsNickCrews
changed the title
[C++][Go]: support casting nullable fields to non-nullable if there are no null valuesGH-33592 [C++][Go]: support casting nullable fields to non-nullable if there are no null valuesAug 21, 2024
@github-actions
github-actionsBot
commented
Aug 21, 2024

Copy link


Copy Markdown

⚠️ GitHub issue #33592has been automatically assigned in GitHub to PR creator.
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
3 times, most recently
from
52100dd to
49d5a99CompareAugust 22, 2024 00:40
@rustyconover
rustyconover
commented
Sep 15, 2024

Copy link


Copy Markdown

This PR looks pretty good to me, what additional help would you like getting it merged?
@NickCrews
NickCrews
commented
Sep 15, 2024

Copy link


Copy Markdown

ContributorAuthor
Thanks for the help! I'm a random contributor, so I'm not sure about the exact workflow, but I think both getting the CI run approved and having a C++ owner approve it are needed.
Any more detailed thoughts on if the tests are adequate, if there's anywhere else in the code base that you think needs to change, my handling of the go implementation by just deleting the shortcircuit, or any other more detailed thoughts? Basically anything that would reduce the mental load on the code owner I think would increase the odds they approve it :)
@mapleFU
mapleFU
commented
Sep 15, 2024

Copy link


Copy Markdown

Member
You can mark as ready for review? Or this is still wip?
@NickCrews
NickCrews
marked this pull request as ready for review
September 15, 2024 20:11
@NickCrews
NickCrews
commented
Sep 15, 2024

Copy link


Copy Markdown

ContributorAuthor
Oops, didn't mean for this to still be marked WIP. Looks like a few failures that need to get fixed, but still a high-level review of the general approach would still be appreciated in the meantime.
@zeroshade
zeroshade
commented
Sep 15, 2024

Copy link


Copy Markdown

Member
You probably need the same corresponding check on the Go side to verify that it's only allowed if there are no nulls.
Also, the Go implementation has been moved to the apache/arrow-go repository. So please file the PR there for the Go side. Sorry for the confusion, we just haven't removed the Go code from here yet.
@NickCrews
NickCrews
commented
Sep 16, 2024

Copy link


Copy Markdown

ContributorAuthor
Thanks @zeroshade , will do. Do we need both PRs to land at about the same time, or can I do that independently?
I will undo my changes to the go code here, leaving it untouched.
@zeroshade
zeroshade
commented
Sep 16, 2024

Copy link


Copy Markdown

Member
They can land independently, no issues there. Thanks!
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
49d5a99 to
eb9e7b6CompareSeptember 16, 2024 16:27
@NickCrews
NickCrews
commented
Sep 16, 2024

Copy link


Copy Markdown

ContributorAuthor
Just pushed a new version:
  • dropped the Go changes
  • rebased on top of 3600db8, which is one commit behind main, because main is failing CI, but the commit I chose passed CI. I did this because I think the failing CI checks in this PR are not related to this PR
  • I discovered archery and formatted the files with archery lint --clang-format --fix
We will see if this passes CI now...
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
eb9e7b6 to
e6d54d3CompareSeptember 18, 2024 07:49
@NickCrews
NickCrews
commented
Sep 18, 2024

Copy link


Copy Markdown

ContributorAuthor
Pushed a new version, hopefully that fixed the broken tests:
  • To go from nullable to non-nullable type, need to use the unsafe cast option
  • fix typo of int8 to int64 so the precision matches
  • removed the [Go] tag from the commit message
@NickCrews
NickCrews
commented
Sep 18, 2024

Copy link


Copy Markdown

ContributorAuthor
@zeroshade I think this is ready to review/merge, the failing CI runs look like flakes when trying to setup the environment?
@NickCrewsNickCrews
mentioned this pull request
Sep 19, 2024

Open

@kou
kou
commented
Sep 19, 2024

Copy link


Copy Markdown

Member
Can we move this to apache/arrow-go?
@koukou
changed the title
GH-33592 [C++][Go]: support casting nullable fields to non-nullable if there are no null valuesGH-33592: [C++] support casting nullable fields to non-nullable if there are no null valuesSep 19, 2024
@kou
kou
commented
Sep 19, 2024

Copy link


Copy Markdown

Member
Ah, the Go part was removed from this PR.

I've removed "[Go]" from the PR title.
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
e6d54d3 to
106e627CompareSeptember 24, 2024 18:53
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
b124c64 to
ebe9a5aCompareFebruary 7, 2025 08:01
@NickCrews
NickCrews
commented
Feb 7, 2025

Copy link


Copy Markdown

ContributorAuthor
@mapleFU as I added the slicing tests, I also refactored the tests to make them much more consistent and concise. Take another look and make sure that your "LGTM" still holds after those changes.
pitrou
pitrou
reviewed
Feb 7, 2025
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a nit!
Comment threadcpp/src/arrow/compute/kernels/scalar_cast_nested.cc
Outdated

@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
2 times, most recently
from
ce00550 to
ea241a5CompareFebruary 10, 2025 16:56
NickCrews
NickCrews
commented
Feb 10, 2025
View reviewed changes
Comment threadcpp/src/arrow/compute/kernels/scalar_cast_nested.cc
Outdated

@NickCrews
NickCrews
commented
Feb 10, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
I'm getting these failing UBSAN errors in CI. See below for the relevant logs. Is it because of the new std::vector<std::shared_ptr<Array>> arrays_dest_ac = {arrays_dest[0], arrays_dest[2]} code that I added? It looks like the line numbers referenced are the ones AFTER the macros/preprocessor occurs, so I'm not sure which line in the source file is actually the problem :( Is there a good way to silence/ignore this error? Any hint as to how to run this locally so I can reproduce/test?
Details
[ RUN ] Cast.StructToStructSubset
/usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1064:34: runtime error: addition of unsigned offset to 0x60700006f300 overflowed to 0x60700006f2e0
#0 0x7f811aca4bc8 in std::vector<std::shared_ptr<arrow::Field>, std::allocator<std::shared_ptr<arrow::Field> > >::operator[](unsigned long) const /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1064:34
#1 0x7f811ac9b27a in arrow::DataType::field(int) const /arrow/cpp/src/arrow/type.h:153:61
#2 0x7f8105e4f8f1 in arrow::compute::internal::(anonymous namespace)::CastStruct::Exec(arrow::compute::KernelContext*, arrow::compute::ExecSpan const&, arrow::compute::ExecResult*) /arrow/cpp/src/arrow/compute/kernels/scalar_cast_nested.cc:403:38
#3 0x7f8105ab206e in arrow::compute::detail::(anonymous namespace)::ScalarExecutor::ExecuteNonSpans(arrow::compute::detail::ExecListener*) /arrow/cpp/src/arrow/compute/exec.cc:920:7
#4 0x7f8105aa99fa in arrow::compute::detail::(anonymous namespace)::ScalarExecutor::Execute(arrow::compute::ExecBatch const&, arrow::compute::detail::ExecListener*) /arrow/cpp/src/arrow/compute/exec.cc:810:14
#5 0x7f8105c471a6 in arrow::compute::detail::FunctionExecutorImpl::Execute(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, long) /arrow/cpp/src/arrow/compute/function.cc:278:5
#6 0x7f8105c17bfe in arrow::compute::(anonymous namespace)::ExecuteInternal(arrow::compute::Function const&, std::vector<arrow::Datum, std::allocator<arrow::Datum> >, long, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) /arrow/cpp/src/arrow/compute/function.cc:343:21
#7 0x7f8105c16d84 in arrow::compute::Function::Execute(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) const /arrow/cpp/src/arrow/compute/function.cc:350:10
#8 0x7f8105a5d13c in arrow::compute::internal::(anonymous namespace)::CastMetaFunction::ExecuteImpl(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) const /arrow/cpp/src/arrow/compute/cast.cc:124:23
#9 0x7f8105c21b5c in arrow::compute::MetaFunction::Execute(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) const /arrow/cpp/src/arrow/compute/function.cc:483:10
#10 0x7f8105a8cfc8 in arrow::compute::CallFunction(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) /arrow/cpp/src/arrow/compute/exec.cc:1369:16
#11 0x55f68eeb23c9 in arrow::compute::CheckScalarNonRecursive(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::Datum const&, arrow::compute::FunctionOptions const*) /arrow/cpp/src/arrow/compute/kernels/test_util_internal.cc:80:3
#12 0x55f68eeb8480 in arrow::compute::CheckScalar(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::Datum, arrow::compute::FunctionOptions const*) /arrow/cpp/src/arrow/compute/kernels/test_util_internal.cc:109:3
#13 0x55f68eecc36c in arrow::compute::CheckScalarUnary(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, arrow::Datum, arrow::Datum, arrow::compute::FunctionOptions const*) /arrow/cpp/src/arrow/compute/kernels/test_util_internal.cc:255:3
#14 0x55f68ef4d8f7 in arrow::compute::CheckCast(std::shared_ptr<arrow::Array>, std::shared_ptr<arrow::Array>, arrow::compute::CastOptions) /arrow/cpp/src/arrow/compute/kernels/scalar_cast_test.cc:111:3
#15 0x55f68f079ab8 in arrow::compute::CheckStructToStructSubset(std::vector<std::shared_ptr<arrow::DataType>, std::allocator<std::shared_ptr<arrow::DataType> > > const&) /arrow/cpp/src/arrow/compute/kernels/scalar_cast_test.cc:3829:7
#16 0x55f68f06d09f in arrow::compute::Cast_StructToStructSubset_Test::TestBody() /arrow/cpp/src/arrow/compute/kernels/scalar_cast_test.cc:4003:36
#17 0x7f811af8260e in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/usr/local/lib/libarrow_testing.so.2000+0x108c60e) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#18 0x7f811af770c5 in testing::Test::Run() (/usr/local/lib/libarrow_testing.so.2000+0x10810c5) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#19 0x7f811af77244 in testing::TestInfo::Run() (/usr/local/lib/libarrow_testing.so.2000+0x1081244) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#20 0x7f811af777f8 in testing::TestSuite::Run() (/usr/local/lib/libarrow_testing.so.2000+0x10817f8) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#21 0x7f811af77efe in testing::internal::UnitTestImpl::RunAllTests() (/usr/local/lib/libarrow_testing.so.2000+0x1081efe) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#22 0x7f811af82bd6 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/usr/local/lib/libarrow_testing.so.2000+0x108cbd6) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#23 0x7f811af7730b in testing::UnitTest::Run() (/usr/local/lib/libarrow_testing.so.2000+0x108130b) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#24 0x55f68edefb33 in main (/build/cpp/debug/arrow-compute-scalar-cast-test+0x2adb33) (BuildId: c176b331f0537175685c7888a83fc8e112006f6a)
#0 0x7f80f7d06d8f in
#26 0x7f80f7d06e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: cd410b710f0f094c6832edd95931006d883af48e)
#27 0x55f68edefb94 in _start (/build/cpp/debug/arrow-compute-scalar-cast-test+0x2adb94) (BuildId: c176b331f0537175685c7888a83fc8e112006f6a)
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1064:34 in
/build/cpp/src/arrow/compute/kernels
Start 22: arrow-compute-scalar-cast-test
Test #22: arrow-compute-scalar-cast-test ...............***Failed 12.44 sec

@pitrou
pitrou
commented
Feb 11, 2025

Copy link


Copy Markdown

Member
Well, the Windows 2019 CI test shows a similar error, so I think it needs diagnosing and solving:
[ RUN ] Cast.StructToStructSubset
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\include\vector(1563) : Assertion failed: vector subscript out of range

pitrou
pitrou
reviewed
Feb 11, 2025
View reviewed changes
Comment threadcpp/src/arrow/compute/kernels/scalar_cast_nested.cc
Outdated

@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
2 times, most recently
from
45f1ac2 to
0959343CompareFebruary 11, 2025 17:07
@NickCrews
NickCrews
commented
Feb 11, 2025

Copy link


Copy Markdown

ContributorAuthor
I rebased on top of #45500 so that hopefully CI will pass
@pitrou
pitrouforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
0959343 to
4311938CompareFebruary 13, 2025 13:46
pitrou
pitrou
approved these changes
Feb 13, 2025
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. Thanks for this @NickCrews !
@pitrou
pitrou
commented
Feb 13, 2025

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g cpp
@github-actions
github-actionsBot
commented
Feb 13, 2025

Copy link


Copy Markdown

Revision: 4311938
Submitted crossbow builds: ursacomputing/crossbow @ actions-f1ea7df869
TaskStatus
example-cpp-minimal-build-staticGitHub Actions
example-cpp-minimal-build-static-system-dependencyGitHub Actions
example-cpp-tutorialGitHub Actions
test-alpine-linux-cppGitHub Actions
test-build-cpp-fuzzGitHub Actions
test-conda-cppGitHub Actions
test-conda-cpp-valgrindGitHub Actions
test-cuda-cpp-ubuntu-20.04-cuda-11.2.2GitHub Actions
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1GitHub Actions
test-debian-12-cpp-amd64GitHub Actions
test-debian-12-cpp-i386GitHub Actions
test-fedora-39-cppGitHub Actions
test-ubuntu-20.04-cppGitHub Actions
test-ubuntu-20.04-cpp-bundledGitHub Actions
test-ubuntu-22.04-cppGitHub Actions
test-ubuntu-22.04-cpp-20GitHub Actions
test-ubuntu-22.04-cpp-emscriptenGitHub Actions
test-ubuntu-22.04-cpp-no-threadingGitHub Actions
test-ubuntu-24.04-cppGitHub Actions
test-ubuntu-24.04-cpp-bundled-offlineGitHub Actions
test-ubuntu-24.04-cpp-gcc-13-bundledGitHub Actions
test-ubuntu-24.04-cpp-gcc-14GitHub Actions
test-ubuntu-24.04-cpp-minimal-with-formatsGitHub Actions
test-ubuntu-24.04-cpp-thread-sanitizerGitHub Actions
@pitrou
pitrou
commented
Feb 13, 2025

Copy link


Copy Markdown

Member
The CI failures are unrelated (see #45524).
@pitrou
pitrou
merged commit 6a47e4d
into
apache:mainFeb 13, 2025
@pitroupitrou
removed
the 
awaiting committer review
Awaiting committer review
label
Feb 13, 2025
@NickCrews
NickCrews
commented
Feb 13, 2025

Copy link


Copy Markdown

ContributorAuthor
Thank you everyone! That was a lot of effort everyone did, I wouldn't have been able to do that without all the help. I'm excited for this to be released which will allow me to remove some hacky workarounds I have in my app code.
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Feb 13, 2025

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 6a47e4d.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.
@Tom-NewtonTom-Newton
mentioned this pull request
Apr 23, 2025
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@zanmato1984zanmato1984zanmato1984 left review comments
@pitroupitroupitrou approved these changes
@mapleFUmapleFUmapleFU approved these changes
@zeroshadezeroshadeAwaiting requested review from zeroshade
@assignUserassignUserAwaiting requested review from assignUser
@jonkeanejonkeaneAwaiting requested review from jonkeane
@koukouAwaiting requested review from kou
@raulcdraulcdAwaiting requested review from raulcd

Assignees


No one assigned


Labels

Component: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

7 participants

@NickCrews@rustyconover@mapleFU@zeroshade@kou@pitrou@zanmato1984

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
GH-33592: [C++] support casting nullable fields to non-nullable if there are no null values - #43782
Merged
pitrou merged 3 commits into
apache:mainfrom 
NickCrews:field-cast-nullable-to-nonnullable
Feb 13, 2025
Merged
GH-33592: [C++] support casting nullable fields to non-nullable if there are no null values#43782
pitrou merged 3 commits into
apache:mainfrom 
NickCrews:field-cast-nullable-to-nonnullable
Conversation
@NickCrews
@NickCrewsNickCrews
commented
Aug 21, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Notes for myself/fixer:
  • tests that need to get updated (almost definitely not a complete list)
  • [update: actually we should handle the go implementation in the go repository.] hmm, looks like go wrapper does its own nullability checks. I assume this is just an optimization to not have to go into the C++ and hit the error down there. So if we just delete this check then I think the C++ logic will handle all of it???
  • how the plain column handles this cast, some logic like this probably needs to get ported over to the struct implementation
  • (running from /cpp/build) cmake .. --preset ninja-debug-basic, then cmake --build . && PYTHON=python ctest -R 'arrow-compute-scalar-cast-test' --output-on-failure to run the specific test
  • to format: uvx pre-commit run --all-files clang-format
@github-actions
github-actionsBot
commented
Aug 21, 2024

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:
@NickCrewsNickCrews
mentioned this pull request
Aug 21, 2024

Closed

@NickCrewsNickCrews
changed the title
[C++][Go]: support casting nullable fields to non-nullable if there are no null valuesGH-33592 [C++][Go]: support casting nullable fields to non-nullable if there are no null valuesAug 21, 2024
@github-actions
github-actionsBot
commented
Aug 21, 2024

Copy link


Copy Markdown

⚠️ GitHub issue #33592has been automatically assigned in GitHub to PR creator.
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
3 times, most recently
from
52100dd to
49d5a99CompareAugust 22, 2024 00:40
@rustyconover
rustyconover
commented
Sep 15, 2024

Copy link


Copy Markdown

This PR looks pretty good to me, what additional help would you like getting it merged?
@NickCrews
NickCrews
commented
Sep 15, 2024

Copy link


Copy Markdown

ContributorAuthor
Thanks for the help! I'm a random contributor, so I'm not sure about the exact workflow, but I think both getting the CI run approved and having a C++ owner approve it are needed.
Any more detailed thoughts on if the tests are adequate, if there's anywhere else in the code base that you think needs to change, my handling of the go implementation by just deleting the shortcircuit, or any other more detailed thoughts? Basically anything that would reduce the mental load on the code owner I think would increase the odds they approve it :)
@mapleFU
mapleFU
commented
Sep 15, 2024

Copy link


Copy Markdown

Member
You can mark as ready for review? Or this is still wip?
@NickCrews
NickCrews
marked this pull request as ready for review
September 15, 2024 20:11
@NickCrews
NickCrews
commented
Sep 15, 2024

Copy link


Copy Markdown

ContributorAuthor
Oops, didn't mean for this to still be marked WIP. Looks like a few failures that need to get fixed, but still a high-level review of the general approach would still be appreciated in the meantime.
@zeroshade
zeroshade
commented
Sep 15, 2024

Copy link


Copy Markdown

Member
You probably need the same corresponding check on the Go side to verify that it's only allowed if there are no nulls.
Also, the Go implementation has been moved to the apache/arrow-go repository. So please file the PR there for the Go side. Sorry for the confusion, we just haven't removed the Go code from here yet.
@NickCrews
NickCrews
commented
Sep 16, 2024

Copy link


Copy Markdown

ContributorAuthor
Thanks @zeroshade , will do. Do we need both PRs to land at about the same time, or can I do that independently?
I will undo my changes to the go code here, leaving it untouched.
@zeroshade
zeroshade
commented
Sep 16, 2024

Copy link


Copy Markdown

Member
They can land independently, no issues there. Thanks!
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
49d5a99 to
eb9e7b6CompareSeptember 16, 2024 16:27
@NickCrews
NickCrews
commented
Sep 16, 2024

Copy link


Copy Markdown

ContributorAuthor
Just pushed a new version:
  • dropped the Go changes
  • rebased on top of 3600db8, which is one commit behind main, because main is failing CI, but the commit I chose passed CI. I did this because I think the failing CI checks in this PR are not related to this PR
  • I discovered archery and formatted the files with archery lint --clang-format --fix
We will see if this passes CI now...
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
eb9e7b6 to
e6d54d3CompareSeptember 18, 2024 07:49
@NickCrews
NickCrews
commented
Sep 18, 2024

Copy link


Copy Markdown

ContributorAuthor
Pushed a new version, hopefully that fixed the broken tests:
  • To go from nullable to non-nullable type, need to use the unsafe cast option
  • fix typo of int8 to int64 so the precision matches
  • removed the [Go] tag from the commit message
@NickCrews
NickCrews
commented
Sep 18, 2024

Copy link


Copy Markdown

ContributorAuthor
@zeroshade I think this is ready to review/merge, the failing CI runs look like flakes when trying to setup the environment?
@NickCrewsNickCrews
mentioned this pull request
Sep 19, 2024

Open

@kou
kou
commented
Sep 19, 2024

Copy link


Copy Markdown

Member
Can we move this to apache/arrow-go?
@koukou
changed the title
GH-33592 [C++][Go]: support casting nullable fields to non-nullable if there are no null valuesGH-33592: [C++] support casting nullable fields to non-nullable if there are no null valuesSep 19, 2024
@kou
kou
commented
Sep 19, 2024

Copy link


Copy Markdown

Member
Ah, the Go part was removed from this PR.

I've removed "[Go]" from the PR title.
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
e6d54d3 to
106e627CompareSeptember 24, 2024 18:53
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
b124c64 to
ebe9a5aCompareFebruary 7, 2025 08:01
@NickCrews
NickCrews
commented
Feb 7, 2025

Copy link


Copy Markdown

ContributorAuthor
@mapleFU as I added the slicing tests, I also refactored the tests to make them much more consistent and concise. Take another look and make sure that your "LGTM" still holds after those changes.
pitrou
pitrou
reviewed
Feb 7, 2025
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a nit!
Comment threadcpp/src/arrow/compute/kernels/scalar_cast_nested.cc
Outdated

@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
2 times, most recently
from
ce00550 to
ea241a5CompareFebruary 10, 2025 16:56
NickCrews
NickCrews
commented
Feb 10, 2025
View reviewed changes
Comment threadcpp/src/arrow/compute/kernels/scalar_cast_nested.cc
Outdated

@NickCrews
NickCrews
commented
Feb 10, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
I'm getting these failing UBSAN errors in CI. See below for the relevant logs. Is it because of the new std::vector<std::shared_ptr<Array>> arrays_dest_ac = {arrays_dest[0], arrays_dest[2]} code that I added? It looks like the line numbers referenced are the ones AFTER the macros/preprocessor occurs, so I'm not sure which line in the source file is actually the problem :( Is there a good way to silence/ignore this error? Any hint as to how to run this locally so I can reproduce/test?
Details
[ RUN ] Cast.StructToStructSubset
/usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1064:34: runtime error: addition of unsigned offset to 0x60700006f300 overflowed to 0x60700006f2e0
#0 0x7f811aca4bc8 in std::vector<std::shared_ptr<arrow::Field>, std::allocator<std::shared_ptr<arrow::Field> > >::operator[](unsigned long) const /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1064:34
#1 0x7f811ac9b27a in arrow::DataType::field(int) const /arrow/cpp/src/arrow/type.h:153:61
#2 0x7f8105e4f8f1 in arrow::compute::internal::(anonymous namespace)::CastStruct::Exec(arrow::compute::KernelContext*, arrow::compute::ExecSpan const&, arrow::compute::ExecResult*) /arrow/cpp/src/arrow/compute/kernels/scalar_cast_nested.cc:403:38
#3 0x7f8105ab206e in arrow::compute::detail::(anonymous namespace)::ScalarExecutor::ExecuteNonSpans(arrow::compute::detail::ExecListener*) /arrow/cpp/src/arrow/compute/exec.cc:920:7
#4 0x7f8105aa99fa in arrow::compute::detail::(anonymous namespace)::ScalarExecutor::Execute(arrow::compute::ExecBatch const&, arrow::compute::detail::ExecListener*) /arrow/cpp/src/arrow/compute/exec.cc:810:14
#5 0x7f8105c471a6 in arrow::compute::detail::FunctionExecutorImpl::Execute(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, long) /arrow/cpp/src/arrow/compute/function.cc:278:5
#6 0x7f8105c17bfe in arrow::compute::(anonymous namespace)::ExecuteInternal(arrow::compute::Function const&, std::vector<arrow::Datum, std::allocator<arrow::Datum> >, long, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) /arrow/cpp/src/arrow/compute/function.cc:343:21
#7 0x7f8105c16d84 in arrow::compute::Function::Execute(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) const /arrow/cpp/src/arrow/compute/function.cc:350:10
#8 0x7f8105a5d13c in arrow::compute::internal::(anonymous namespace)::CastMetaFunction::ExecuteImpl(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) const /arrow/cpp/src/arrow/compute/cast.cc:124:23
#9 0x7f8105c21b5c in arrow::compute::MetaFunction::Execute(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) const /arrow/cpp/src/arrow/compute/function.cc:483:10
#10 0x7f8105a8cfc8 in arrow::compute::CallFunction(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) /arrow/cpp/src/arrow/compute/exec.cc:1369:16
#11 0x55f68eeb23c9 in arrow::compute::CheckScalarNonRecursive(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::Datum const&, arrow::compute::FunctionOptions const*) /arrow/cpp/src/arrow/compute/kernels/test_util_internal.cc:80:3
#12 0x55f68eeb8480 in arrow::compute::CheckScalar(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::Datum, arrow::compute::FunctionOptions const*) /arrow/cpp/src/arrow/compute/kernels/test_util_internal.cc:109:3
#13 0x55f68eecc36c in arrow::compute::CheckScalarUnary(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, arrow::Datum, arrow::Datum, arrow::compute::FunctionOptions const*) /arrow/cpp/src/arrow/compute/kernels/test_util_internal.cc:255:3
#14 0x55f68ef4d8f7 in arrow::compute::CheckCast(std::shared_ptr<arrow::Array>, std::shared_ptr<arrow::Array>, arrow::compute::CastOptions) /arrow/cpp/src/arrow/compute/kernels/scalar_cast_test.cc:111:3
#15 0x55f68f079ab8 in arrow::compute::CheckStructToStructSubset(std::vector<std::shared_ptr<arrow::DataType>, std::allocator<std::shared_ptr<arrow::DataType> > > const&) /arrow/cpp/src/arrow/compute/kernels/scalar_cast_test.cc:3829:7
#16 0x55f68f06d09f in arrow::compute::Cast_StructToStructSubset_Test::TestBody() /arrow/cpp/src/arrow/compute/kernels/scalar_cast_test.cc:4003:36
#17 0x7f811af8260e in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/usr/local/lib/libarrow_testing.so.2000+0x108c60e) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#18 0x7f811af770c5 in testing::Test::Run() (/usr/local/lib/libarrow_testing.so.2000+0x10810c5) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#19 0x7f811af77244 in testing::TestInfo::Run() (/usr/local/lib/libarrow_testing.so.2000+0x1081244) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#20 0x7f811af777f8 in testing::TestSuite::Run() (/usr/local/lib/libarrow_testing.so.2000+0x10817f8) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#21 0x7f811af77efe in testing::internal::UnitTestImpl::RunAllTests() (/usr/local/lib/libarrow_testing.so.2000+0x1081efe) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#22 0x7f811af82bd6 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/usr/local/lib/libarrow_testing.so.2000+0x108cbd6) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#23 0x7f811af7730b in testing::UnitTest::Run() (/usr/local/lib/libarrow_testing.so.2000+0x108130b) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#24 0x55f68edefb33 in main (/build/cpp/debug/arrow-compute-scalar-cast-test+0x2adb33) (BuildId: c176b331f0537175685c7888a83fc8e112006f6a)
#0 0x7f80f7d06d8f in
#26 0x7f80f7d06e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: cd410b710f0f094c6832edd95931006d883af48e)
#27 0x55f68edefb94 in _start (/build/cpp/debug/arrow-compute-scalar-cast-test+0x2adb94) (BuildId: c176b331f0537175685c7888a83fc8e112006f6a)
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1064:34 in
/build/cpp/src/arrow/compute/kernels
Start 22: arrow-compute-scalar-cast-test
Test #22: arrow-compute-scalar-cast-test ...............***Failed 12.44 sec

@pitrou
pitrou
commented
Feb 11, 2025

Copy link


Copy Markdown

Member
Well, the Windows 2019 CI test shows a similar error, so I think it needs diagnosing and solving:
[ RUN ] Cast.StructToStructSubset
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\include\vector(1563) : Assertion failed: vector subscript out of range

pitrou
pitrou
reviewed
Feb 11, 2025
View reviewed changes
Comment threadcpp/src/arrow/compute/kernels/scalar_cast_nested.cc
Outdated

@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
2 times, most recently
from
45f1ac2 to
0959343CompareFebruary 11, 2025 17:07
@NickCrews
NickCrews
commented
Feb 11, 2025

Copy link


Copy Markdown

ContributorAuthor
I rebased on top of #45500 so that hopefully CI will pass
@pitrou
pitrouforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
0959343 to
4311938CompareFebruary 13, 2025 13:46
pitrou
pitrou
approved these changes
Feb 13, 2025
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. Thanks for this @NickCrews !
@pitrou
pitrou
commented
Feb 13, 2025

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g cpp
@github-actions
github-actionsBot
commented
Feb 13, 2025

Copy link


Copy Markdown

Revision: 4311938
Submitted crossbow builds: ursacomputing/crossbow @ actions-f1ea7df869
TaskStatus
example-cpp-minimal-build-staticGitHub Actions
example-cpp-minimal-build-static-system-dependencyGitHub Actions
example-cpp-tutorialGitHub Actions
test-alpine-linux-cppGitHub Actions
test-build-cpp-fuzzGitHub Actions
test-conda-cppGitHub Actions
test-conda-cpp-valgrindGitHub Actions
test-cuda-cpp-ubuntu-20.04-cuda-11.2.2GitHub Actions
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1GitHub Actions
test-debian-12-cpp-amd64GitHub Actions
test-debian-12-cpp-i386GitHub Actions
test-fedora-39-cppGitHub Actions
test-ubuntu-20.04-cppGitHub Actions
test-ubuntu-20.04-cpp-bundledGitHub Actions
test-ubuntu-22.04-cppGitHub Actions
test-ubuntu-22.04-cpp-20GitHub Actions
test-ubuntu-22.04-cpp-emscriptenGitHub Actions
test-ubuntu-22.04-cpp-no-threadingGitHub Actions
test-ubuntu-24.04-cppGitHub Actions
test-ubuntu-24.04-cpp-bundled-offlineGitHub Actions
test-ubuntu-24.04-cpp-gcc-13-bundledGitHub Actions
test-ubuntu-24.04-cpp-gcc-14GitHub Actions
test-ubuntu-24.04-cpp-minimal-with-formatsGitHub Actions
test-ubuntu-24.04-cpp-thread-sanitizerGitHub Actions
@pitrou
pitrou
commented
Feb 13, 2025

Copy link


Copy Markdown

Member
The CI failures are unrelated (see #45524).
@pitrou
pitrou
merged commit 6a47e4d
into
apache:mainFeb 13, 2025
@pitroupitrou
removed
the 
awaiting committer review
Awaiting committer review
label
Feb 13, 2025
@NickCrews
NickCrews
commented
Feb 13, 2025

Copy link


Copy Markdown

ContributorAuthor
Thank you everyone! That was a lot of effort everyone did, I wouldn't have been able to do that without all the help. I'm excited for this to be released which will allow me to remove some hacky workarounds I have in my app code.
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Feb 13, 2025

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 6a47e4d.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.
@Tom-NewtonTom-Newton
mentioned this pull request
Apr 23, 2025
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@zanmato1984zanmato1984zanmato1984 left review comments
@pitroupitroupitrou approved these changes
@mapleFUmapleFUmapleFU approved these changes
@zeroshadezeroshadeAwaiting requested review from zeroshade
@assignUserassignUserAwaiting requested review from assignUser
@jonkeanejonkeaneAwaiting requested review from jonkeane
@koukouAwaiting requested review from kou
@raulcdraulcdAwaiting requested review from raulcd

Assignees


No one assigned


Labels

Component: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

7 participants

@NickCrews@rustyconover@mapleFU@zeroshade@kou@pitrou@zanmato1984

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-33592: [C++] support casting nullable fields to non-nullable if there are no null values - #43782
Merged
pitrou merged 3 commits into
apache:mainfrom 
NickCrews:field-cast-nullable-to-nonnullable
Feb 13, 2025
Merged
GH-33592: [C++] support casting nullable fields to non-nullable if there are no null values#43782
pitrou merged 3 commits into
apache:mainfrom 
NickCrews:field-cast-nullable-to-nonnullable
Conversation
@NickCrews
@NickCrewsNickCrews
commented
Aug 21, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Notes for myself/fixer:
  • tests that need to get updated (almost definitely not a complete list)
  • [update: actually we should handle the go implementation in the go repository.] hmm, looks like go wrapper does its own nullability checks. I assume this is just an optimization to not have to go into the C++ and hit the error down there. So if we just delete this check then I think the C++ logic will handle all of it???
  • how the plain column handles this cast, some logic like this probably needs to get ported over to the struct implementation
  • (running from /cpp/build) cmake .. --preset ninja-debug-basic, then cmake --build . && PYTHON=python ctest -R 'arrow-compute-scalar-cast-test' --output-on-failure to run the specific test
  • to format: uvx pre-commit run --all-files clang-format
@github-actions
github-actionsBot
commented
Aug 21, 2024

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:
@NickCrewsNickCrews
mentioned this pull request
Aug 21, 2024

Closed

@NickCrewsNickCrews
changed the title
[C++][Go]: support casting nullable fields to non-nullable if there are no null valuesGH-33592 [C++][Go]: support casting nullable fields to non-nullable if there are no null valuesAug 21, 2024
@github-actions
github-actionsBot
commented
Aug 21, 2024

Copy link


Copy Markdown

⚠️ GitHub issue #33592has been automatically assigned in GitHub to PR creator.
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
3 times, most recently
from
52100dd to
49d5a99CompareAugust 22, 2024 00:40
@rustyconover
rustyconover
commented
Sep 15, 2024

Copy link


Copy Markdown

This PR looks pretty good to me, what additional help would you like getting it merged?
@NickCrews
NickCrews
commented
Sep 15, 2024

Copy link


Copy Markdown

ContributorAuthor
Thanks for the help! I'm a random contributor, so I'm not sure about the exact workflow, but I think both getting the CI run approved and having a C++ owner approve it are needed.
Any more detailed thoughts on if the tests are adequate, if there's anywhere else in the code base that you think needs to change, my handling of the go implementation by just deleting the shortcircuit, or any other more detailed thoughts? Basically anything that would reduce the mental load on the code owner I think would increase the odds they approve it :)
@mapleFU
mapleFU
commented
Sep 15, 2024

Copy link


Copy Markdown

Member
You can mark as ready for review? Or this is still wip?
@NickCrews
NickCrews
marked this pull request as ready for review
September 15, 2024 20:11
@NickCrews
NickCrews
commented
Sep 15, 2024

Copy link


Copy Markdown

ContributorAuthor
Oops, didn't mean for this to still be marked WIP. Looks like a few failures that need to get fixed, but still a high-level review of the general approach would still be appreciated in the meantime.
@zeroshade
zeroshade
commented
Sep 15, 2024

Copy link


Copy Markdown

Member
You probably need the same corresponding check on the Go side to verify that it's only allowed if there are no nulls.
Also, the Go implementation has been moved to the apache/arrow-go repository. So please file the PR there for the Go side. Sorry for the confusion, we just haven't removed the Go code from here yet.
@NickCrews
NickCrews
commented
Sep 16, 2024

Copy link


Copy Markdown

ContributorAuthor
Thanks @zeroshade , will do. Do we need both PRs to land at about the same time, or can I do that independently?
I will undo my changes to the go code here, leaving it untouched.
@zeroshade
zeroshade
commented
Sep 16, 2024

Copy link


Copy Markdown

Member
They can land independently, no issues there. Thanks!
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
49d5a99 to
eb9e7b6CompareSeptember 16, 2024 16:27
@NickCrews
NickCrews
commented
Sep 16, 2024

Copy link


Copy Markdown

ContributorAuthor
Just pushed a new version:
  • dropped the Go changes
  • rebased on top of 3600db8, which is one commit behind main, because main is failing CI, but the commit I chose passed CI. I did this because I think the failing CI checks in this PR are not related to this PR
  • I discovered archery and formatted the files with archery lint --clang-format --fix
We will see if this passes CI now...
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
eb9e7b6 to
e6d54d3CompareSeptember 18, 2024 07:49
@NickCrews
NickCrews
commented
Sep 18, 2024

Copy link


Copy Markdown

ContributorAuthor
Pushed a new version, hopefully that fixed the broken tests:
  • To go from nullable to non-nullable type, need to use the unsafe cast option
  • fix typo of int8 to int64 so the precision matches
  • removed the [Go] tag from the commit message
@NickCrews
NickCrews
commented
Sep 18, 2024

Copy link


Copy Markdown

ContributorAuthor
@zeroshade I think this is ready to review/merge, the failing CI runs look like flakes when trying to setup the environment?
@NickCrewsNickCrews
mentioned this pull request
Sep 19, 2024

Open

@kou
kou
commented
Sep 19, 2024

Copy link


Copy Markdown

Member
Can we move this to apache/arrow-go?
@koukou
changed the title
GH-33592 [C++][Go]: support casting nullable fields to non-nullable if there are no null valuesGH-33592: [C++] support casting nullable fields to non-nullable if there are no null valuesSep 19, 2024
@kou
kou
commented
Sep 19, 2024

Copy link


Copy Markdown

Member
Ah, the Go part was removed from this PR.

I've removed "[Go]" from the PR title.
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
e6d54d3 to
106e627CompareSeptember 24, 2024 18:53
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
b124c64 to
ebe9a5aCompareFebruary 7, 2025 08:01
@NickCrews
NickCrews
commented
Feb 7, 2025

Copy link


Copy Markdown

ContributorAuthor
@mapleFU as I added the slicing tests, I also refactored the tests to make them much more consistent and concise. Take another look and make sure that your "LGTM" still holds after those changes.
pitrou
pitrou
reviewed
Feb 7, 2025
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a nit!
Comment threadcpp/src/arrow/compute/kernels/scalar_cast_nested.cc
Outdated

@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
2 times, most recently
from
ce00550 to
ea241a5CompareFebruary 10, 2025 16:56
NickCrews
NickCrews
commented
Feb 10, 2025
View reviewed changes
Comment threadcpp/src/arrow/compute/kernels/scalar_cast_nested.cc
Outdated

@NickCrews
NickCrews
commented
Feb 10, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
I'm getting these failing UBSAN errors in CI. See below for the relevant logs. Is it because of the new std::vector<std::shared_ptr<Array>> arrays_dest_ac = {arrays_dest[0], arrays_dest[2]} code that I added? It looks like the line numbers referenced are the ones AFTER the macros/preprocessor occurs, so I'm not sure which line in the source file is actually the problem :( Is there a good way to silence/ignore this error? Any hint as to how to run this locally so I can reproduce/test?
Details
[ RUN ] Cast.StructToStructSubset
/usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1064:34: runtime error: addition of unsigned offset to 0x60700006f300 overflowed to 0x60700006f2e0
#0 0x7f811aca4bc8 in std::vector<std::shared_ptr<arrow::Field>, std::allocator<std::shared_ptr<arrow::Field> > >::operator[](unsigned long) const /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1064:34
#1 0x7f811ac9b27a in arrow::DataType::field(int) const /arrow/cpp/src/arrow/type.h:153:61
#2 0x7f8105e4f8f1 in arrow::compute::internal::(anonymous namespace)::CastStruct::Exec(arrow::compute::KernelContext*, arrow::compute::ExecSpan const&, arrow::compute::ExecResult*) /arrow/cpp/src/arrow/compute/kernels/scalar_cast_nested.cc:403:38
#3 0x7f8105ab206e in arrow::compute::detail::(anonymous namespace)::ScalarExecutor::ExecuteNonSpans(arrow::compute::detail::ExecListener*) /arrow/cpp/src/arrow/compute/exec.cc:920:7
#4 0x7f8105aa99fa in arrow::compute::detail::(anonymous namespace)::ScalarExecutor::Execute(arrow::compute::ExecBatch const&, arrow::compute::detail::ExecListener*) /arrow/cpp/src/arrow/compute/exec.cc:810:14
#5 0x7f8105c471a6 in arrow::compute::detail::FunctionExecutorImpl::Execute(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, long) /arrow/cpp/src/arrow/compute/function.cc:278:5
#6 0x7f8105c17bfe in arrow::compute::(anonymous namespace)::ExecuteInternal(arrow::compute::Function const&, std::vector<arrow::Datum, std::allocator<arrow::Datum> >, long, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) /arrow/cpp/src/arrow/compute/function.cc:343:21
#7 0x7f8105c16d84 in arrow::compute::Function::Execute(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) const /arrow/cpp/src/arrow/compute/function.cc:350:10
#8 0x7f8105a5d13c in arrow::compute::internal::(anonymous namespace)::CastMetaFunction::ExecuteImpl(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) const /arrow/cpp/src/arrow/compute/cast.cc:124:23
#9 0x7f8105c21b5c in arrow::compute::MetaFunction::Execute(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) const /arrow/cpp/src/arrow/compute/function.cc:483:10
#10 0x7f8105a8cfc8 in arrow::compute::CallFunction(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) /arrow/cpp/src/arrow/compute/exec.cc:1369:16
#11 0x55f68eeb23c9 in arrow::compute::CheckScalarNonRecursive(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::Datum const&, arrow::compute::FunctionOptions const*) /arrow/cpp/src/arrow/compute/kernels/test_util_internal.cc:80:3
#12 0x55f68eeb8480 in arrow::compute::CheckScalar(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::Datum, arrow::compute::FunctionOptions const*) /arrow/cpp/src/arrow/compute/kernels/test_util_internal.cc:109:3
#13 0x55f68eecc36c in arrow::compute::CheckScalarUnary(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, arrow::Datum, arrow::Datum, arrow::compute::FunctionOptions const*) /arrow/cpp/src/arrow/compute/kernels/test_util_internal.cc:255:3
#14 0x55f68ef4d8f7 in arrow::compute::CheckCast(std::shared_ptr<arrow::Array>, std::shared_ptr<arrow::Array>, arrow::compute::CastOptions) /arrow/cpp/src/arrow/compute/kernels/scalar_cast_test.cc:111:3
#15 0x55f68f079ab8 in arrow::compute::CheckStructToStructSubset(std::vector<std::shared_ptr<arrow::DataType>, std::allocator<std::shared_ptr<arrow::DataType> > > const&) /arrow/cpp/src/arrow/compute/kernels/scalar_cast_test.cc:3829:7
#16 0x55f68f06d09f in arrow::compute::Cast_StructToStructSubset_Test::TestBody() /arrow/cpp/src/arrow/compute/kernels/scalar_cast_test.cc:4003:36
#17 0x7f811af8260e in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/usr/local/lib/libarrow_testing.so.2000+0x108c60e) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#18 0x7f811af770c5 in testing::Test::Run() (/usr/local/lib/libarrow_testing.so.2000+0x10810c5) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#19 0x7f811af77244 in testing::TestInfo::Run() (/usr/local/lib/libarrow_testing.so.2000+0x1081244) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#20 0x7f811af777f8 in testing::TestSuite::Run() (/usr/local/lib/libarrow_testing.so.2000+0x10817f8) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#21 0x7f811af77efe in testing::internal::UnitTestImpl::RunAllTests() (/usr/local/lib/libarrow_testing.so.2000+0x1081efe) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#22 0x7f811af82bd6 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/usr/local/lib/libarrow_testing.so.2000+0x108cbd6) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#23 0x7f811af7730b in testing::UnitTest::Run() (/usr/local/lib/libarrow_testing.so.2000+0x108130b) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#24 0x55f68edefb33 in main (/build/cpp/debug/arrow-compute-scalar-cast-test+0x2adb33) (BuildId: c176b331f0537175685c7888a83fc8e112006f6a)
#0 0x7f80f7d06d8f in
#26 0x7f80f7d06e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: cd410b710f0f094c6832edd95931006d883af48e)
#27 0x55f68edefb94 in _start (/build/cpp/debug/arrow-compute-scalar-cast-test+0x2adb94) (BuildId: c176b331f0537175685c7888a83fc8e112006f6a)
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1064:34 in
/build/cpp/src/arrow/compute/kernels
Start 22: arrow-compute-scalar-cast-test
Test #22: arrow-compute-scalar-cast-test ...............***Failed 12.44 sec

@pitrou
pitrou
commented
Feb 11, 2025

Copy link


Copy Markdown

Member
Well, the Windows 2019 CI test shows a similar error, so I think it needs diagnosing and solving:
[ RUN ] Cast.StructToStructSubset
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\include\vector(1563) : Assertion failed: vector subscript out of range

pitrou
pitrou
reviewed
Feb 11, 2025
View reviewed changes
Comment threadcpp/src/arrow/compute/kernels/scalar_cast_nested.cc
Outdated

@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
2 times, most recently
from
45f1ac2 to
0959343CompareFebruary 11, 2025 17:07
@NickCrews
NickCrews
commented
Feb 11, 2025

Copy link


Copy Markdown

ContributorAuthor
I rebased on top of #45500 so that hopefully CI will pass
@pitrou
pitrouforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
0959343 to
4311938CompareFebruary 13, 2025 13:46
pitrou
pitrou
approved these changes
Feb 13, 2025
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. Thanks for this @NickCrews !
@pitrou
pitrou
commented
Feb 13, 2025

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g cpp
@github-actions
github-actionsBot
commented
Feb 13, 2025

Copy link


Copy Markdown

Revision: 4311938
Submitted crossbow builds: ursacomputing/crossbow @ actions-f1ea7df869
TaskStatus
example-cpp-minimal-build-staticGitHub Actions
example-cpp-minimal-build-static-system-dependencyGitHub Actions
example-cpp-tutorialGitHub Actions
test-alpine-linux-cppGitHub Actions
test-build-cpp-fuzzGitHub Actions
test-conda-cppGitHub Actions
test-conda-cpp-valgrindGitHub Actions
test-cuda-cpp-ubuntu-20.04-cuda-11.2.2GitHub Actions
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1GitHub Actions
test-debian-12-cpp-amd64GitHub Actions
test-debian-12-cpp-i386GitHub Actions
test-fedora-39-cppGitHub Actions
test-ubuntu-20.04-cppGitHub Actions
test-ubuntu-20.04-cpp-bundledGitHub Actions
test-ubuntu-22.04-cppGitHub Actions
test-ubuntu-22.04-cpp-20GitHub Actions
test-ubuntu-22.04-cpp-emscriptenGitHub Actions
test-ubuntu-22.04-cpp-no-threadingGitHub Actions
test-ubuntu-24.04-cppGitHub Actions
test-ubuntu-24.04-cpp-bundled-offlineGitHub Actions
test-ubuntu-24.04-cpp-gcc-13-bundledGitHub Actions
test-ubuntu-24.04-cpp-gcc-14GitHub Actions
test-ubuntu-24.04-cpp-minimal-with-formatsGitHub Actions
test-ubuntu-24.04-cpp-thread-sanitizerGitHub Actions
@pitrou
pitrou
commented
Feb 13, 2025

Copy link


Copy Markdown

Member
The CI failures are unrelated (see #45524).
@pitrou
pitrou
merged commit 6a47e4d
into
apache:mainFeb 13, 2025
@pitroupitrou
removed
the 
awaiting committer review
Awaiting committer review
label
Feb 13, 2025
@NickCrews
NickCrews
commented
Feb 13, 2025

Copy link


Copy Markdown

ContributorAuthor
Thank you everyone! That was a lot of effort everyone did, I wouldn't have been able to do that without all the help. I'm excited for this to be released which will allow me to remove some hacky workarounds I have in my app code.
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Feb 13, 2025

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 6a47e4d.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.
@Tom-NewtonTom-Newton
mentioned this pull request
Apr 23, 2025
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@zanmato1984zanmato1984zanmato1984 left review comments
@pitroupitroupitrou approved these changes
@mapleFUmapleFUmapleFU approved these changes
@zeroshadezeroshadeAwaiting requested review from zeroshade
@assignUserassignUserAwaiting requested review from assignUser
@jonkeanejonkeaneAwaiting requested review from jonkeane
@koukouAwaiting requested review from kou
@raulcdraulcdAwaiting requested review from raulcd

Assignees


No one assigned


Labels

Component: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

7 participants

@NickCrews@rustyconover@mapleFU@zeroshade@kou@pitrou@zanmato1984

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-33592: [C++] support casting nullable fields to non-nullable if there are no null values - #43782
Merged
pitrou merged 3 commits into
apache:mainfrom 
NickCrews:field-cast-nullable-to-nonnullable
Feb 13, 2025
Merged
GH-33592: [C++] support casting nullable fields to non-nullable if there are no null values#43782
pitrou merged 3 commits into
apache:mainfrom 
NickCrews:field-cast-nullable-to-nonnullable
Conversation
@NickCrews
@NickCrewsNickCrews
commented
Aug 21, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Notes for myself/fixer:
  • tests that need to get updated (almost definitely not a complete list)
  • [update: actually we should handle the go implementation in the go repository.] hmm, looks like go wrapper does its own nullability checks. I assume this is just an optimization to not have to go into the C++ and hit the error down there. So if we just delete this check then I think the C++ logic will handle all of it???
  • how the plain column handles this cast, some logic like this probably needs to get ported over to the struct implementation
  • (running from /cpp/build) cmake .. --preset ninja-debug-basic, then cmake --build . && PYTHON=python ctest -R 'arrow-compute-scalar-cast-test' --output-on-failure to run the specific test
  • to format: uvx pre-commit run --all-files clang-format
@github-actions
github-actionsBot
commented
Aug 21, 2024

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:
@NickCrewsNickCrews
mentioned this pull request
Aug 21, 2024

Closed

@NickCrewsNickCrews
changed the title
[C++][Go]: support casting nullable fields to non-nullable if there are no null valuesGH-33592 [C++][Go]: support casting nullable fields to non-nullable if there are no null valuesAug 21, 2024
@github-actions
github-actionsBot
commented
Aug 21, 2024

Copy link


Copy Markdown

⚠️ GitHub issue #33592has been automatically assigned in GitHub to PR creator.
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
3 times, most recently
from
52100dd to
49d5a99CompareAugust 22, 2024 00:40
@rustyconover
rustyconover
commented
Sep 15, 2024

Copy link


Copy Markdown

This PR looks pretty good to me, what additional help would you like getting it merged?
@NickCrews
NickCrews
commented
Sep 15, 2024

Copy link


Copy Markdown

ContributorAuthor
Thanks for the help! I'm a random contributor, so I'm not sure about the exact workflow, but I think both getting the CI run approved and having a C++ owner approve it are needed.
Any more detailed thoughts on if the tests are adequate, if there's anywhere else in the code base that you think needs to change, my handling of the go implementation by just deleting the shortcircuit, or any other more detailed thoughts? Basically anything that would reduce the mental load on the code owner I think would increase the odds they approve it :)
@mapleFU
mapleFU
commented
Sep 15, 2024

Copy link


Copy Markdown

Member
You can mark as ready for review? Or this is still wip?
@NickCrews
NickCrews
marked this pull request as ready for review
September 15, 2024 20:11
@NickCrews
NickCrews
commented
Sep 15, 2024

Copy link


Copy Markdown

ContributorAuthor
Oops, didn't mean for this to still be marked WIP. Looks like a few failures that need to get fixed, but still a high-level review of the general approach would still be appreciated in the meantime.
@zeroshade
zeroshade
commented
Sep 15, 2024

Copy link


Copy Markdown

Member
You probably need the same corresponding check on the Go side to verify that it's only allowed if there are no nulls.
Also, the Go implementation has been moved to the apache/arrow-go repository. So please file the PR there for the Go side. Sorry for the confusion, we just haven't removed the Go code from here yet.
@NickCrews
NickCrews
commented
Sep 16, 2024

Copy link


Copy Markdown

ContributorAuthor
Thanks @zeroshade , will do. Do we need both PRs to land at about the same time, or can I do that independently?
I will undo my changes to the go code here, leaving it untouched.
@zeroshade
zeroshade
commented
Sep 16, 2024

Copy link


Copy Markdown

Member
They can land independently, no issues there. Thanks!
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
49d5a99 to
eb9e7b6CompareSeptember 16, 2024 16:27
@NickCrews
NickCrews
commented
Sep 16, 2024

Copy link


Copy Markdown

ContributorAuthor
Just pushed a new version:
  • dropped the Go changes
  • rebased on top of 3600db8, which is one commit behind main, because main is failing CI, but the commit I chose passed CI. I did this because I think the failing CI checks in this PR are not related to this PR
  • I discovered archery and formatted the files with archery lint --clang-format --fix
We will see if this passes CI now...
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
eb9e7b6 to
e6d54d3CompareSeptember 18, 2024 07:49
@NickCrews
NickCrews
commented
Sep 18, 2024

Copy link


Copy Markdown

ContributorAuthor
Pushed a new version, hopefully that fixed the broken tests:
  • To go from nullable to non-nullable type, need to use the unsafe cast option
  • fix typo of int8 to int64 so the precision matches
  • removed the [Go] tag from the commit message
@NickCrews
NickCrews
commented
Sep 18, 2024

Copy link


Copy Markdown

ContributorAuthor
@zeroshade I think this is ready to review/merge, the failing CI runs look like flakes when trying to setup the environment?
@NickCrewsNickCrews
mentioned this pull request
Sep 19, 2024

Open

@kou
kou
commented
Sep 19, 2024

Copy link


Copy Markdown

Member
Can we move this to apache/arrow-go?
@koukou
changed the title
GH-33592 [C++][Go]: support casting nullable fields to non-nullable if there are no null valuesGH-33592: [C++] support casting nullable fields to non-nullable if there are no null valuesSep 19, 2024
@kou
kou
commented
Sep 19, 2024

Copy link


Copy Markdown

Member
Ah, the Go part was removed from this PR.

I've removed "[Go]" from the PR title.
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
e6d54d3 to
106e627CompareSeptember 24, 2024 18:53
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
b124c64 to
ebe9a5aCompareFebruary 7, 2025 08:01
@NickCrews
NickCrews
commented
Feb 7, 2025

Copy link


Copy Markdown

ContributorAuthor
@mapleFU as I added the slicing tests, I also refactored the tests to make them much more consistent and concise. Take another look and make sure that your "LGTM" still holds after those changes.
pitrou
pitrou
reviewed
Feb 7, 2025
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a nit!
Comment threadcpp/src/arrow/compute/kernels/scalar_cast_nested.cc
Outdated

@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
2 times, most recently
from
ce00550 to
ea241a5CompareFebruary 10, 2025 16:56
NickCrews
NickCrews
commented
Feb 10, 2025
View reviewed changes
Comment threadcpp/src/arrow/compute/kernels/scalar_cast_nested.cc
Outdated

@NickCrews
NickCrews
commented
Feb 10, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
I'm getting these failing UBSAN errors in CI. See below for the relevant logs. Is it because of the new std::vector<std::shared_ptr<Array>> arrays_dest_ac = {arrays_dest[0], arrays_dest[2]} code that I added? It looks like the line numbers referenced are the ones AFTER the macros/preprocessor occurs, so I'm not sure which line in the source file is actually the problem :( Is there a good way to silence/ignore this error? Any hint as to how to run this locally so I can reproduce/test?
Details
[ RUN ] Cast.StructToStructSubset
/usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1064:34: runtime error: addition of unsigned offset to 0x60700006f300 overflowed to 0x60700006f2e0
#0 0x7f811aca4bc8 in std::vector<std::shared_ptr<arrow::Field>, std::allocator<std::shared_ptr<arrow::Field> > >::operator[](unsigned long) const /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1064:34
#1 0x7f811ac9b27a in arrow::DataType::field(int) const /arrow/cpp/src/arrow/type.h:153:61
#2 0x7f8105e4f8f1 in arrow::compute::internal::(anonymous namespace)::CastStruct::Exec(arrow::compute::KernelContext*, arrow::compute::ExecSpan const&, arrow::compute::ExecResult*) /arrow/cpp/src/arrow/compute/kernels/scalar_cast_nested.cc:403:38
#3 0x7f8105ab206e in arrow::compute::detail::(anonymous namespace)::ScalarExecutor::ExecuteNonSpans(arrow::compute::detail::ExecListener*) /arrow/cpp/src/arrow/compute/exec.cc:920:7
#4 0x7f8105aa99fa in arrow::compute::detail::(anonymous namespace)::ScalarExecutor::Execute(arrow::compute::ExecBatch const&, arrow::compute::detail::ExecListener*) /arrow/cpp/src/arrow/compute/exec.cc:810:14
#5 0x7f8105c471a6 in arrow::compute::detail::FunctionExecutorImpl::Execute(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, long) /arrow/cpp/src/arrow/compute/function.cc:278:5
#6 0x7f8105c17bfe in arrow::compute::(anonymous namespace)::ExecuteInternal(arrow::compute::Function const&, std::vector<arrow::Datum, std::allocator<arrow::Datum> >, long, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) /arrow/cpp/src/arrow/compute/function.cc:343:21
#7 0x7f8105c16d84 in arrow::compute::Function::Execute(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) const /arrow/cpp/src/arrow/compute/function.cc:350:10
#8 0x7f8105a5d13c in arrow::compute::internal::(anonymous namespace)::CastMetaFunction::ExecuteImpl(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) const /arrow/cpp/src/arrow/compute/cast.cc:124:23
#9 0x7f8105c21b5c in arrow::compute::MetaFunction::Execute(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) const /arrow/cpp/src/arrow/compute/function.cc:483:10
#10 0x7f8105a8cfc8 in arrow::compute::CallFunction(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) /arrow/cpp/src/arrow/compute/exec.cc:1369:16
#11 0x55f68eeb23c9 in arrow::compute::CheckScalarNonRecursive(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::Datum const&, arrow::compute::FunctionOptions const*) /arrow/cpp/src/arrow/compute/kernels/test_util_internal.cc:80:3
#12 0x55f68eeb8480 in arrow::compute::CheckScalar(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::Datum, arrow::compute::FunctionOptions const*) /arrow/cpp/src/arrow/compute/kernels/test_util_internal.cc:109:3
#13 0x55f68eecc36c in arrow::compute::CheckScalarUnary(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, arrow::Datum, arrow::Datum, arrow::compute::FunctionOptions const*) /arrow/cpp/src/arrow/compute/kernels/test_util_internal.cc:255:3
#14 0x55f68ef4d8f7 in arrow::compute::CheckCast(std::shared_ptr<arrow::Array>, std::shared_ptr<arrow::Array>, arrow::compute::CastOptions) /arrow/cpp/src/arrow/compute/kernels/scalar_cast_test.cc:111:3
#15 0x55f68f079ab8 in arrow::compute::CheckStructToStructSubset(std::vector<std::shared_ptr<arrow::DataType>, std::allocator<std::shared_ptr<arrow::DataType> > > const&) /arrow/cpp/src/arrow/compute/kernels/scalar_cast_test.cc:3829:7
#16 0x55f68f06d09f in arrow::compute::Cast_StructToStructSubset_Test::TestBody() /arrow/cpp/src/arrow/compute/kernels/scalar_cast_test.cc:4003:36
#17 0x7f811af8260e in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/usr/local/lib/libarrow_testing.so.2000+0x108c60e) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#18 0x7f811af770c5 in testing::Test::Run() (/usr/local/lib/libarrow_testing.so.2000+0x10810c5) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#19 0x7f811af77244 in testing::TestInfo::Run() (/usr/local/lib/libarrow_testing.so.2000+0x1081244) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#20 0x7f811af777f8 in testing::TestSuite::Run() (/usr/local/lib/libarrow_testing.so.2000+0x10817f8) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#21 0x7f811af77efe in testing::internal::UnitTestImpl::RunAllTests() (/usr/local/lib/libarrow_testing.so.2000+0x1081efe) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#22 0x7f811af82bd6 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/usr/local/lib/libarrow_testing.so.2000+0x108cbd6) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#23 0x7f811af7730b in testing::UnitTest::Run() (/usr/local/lib/libarrow_testing.so.2000+0x108130b) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#24 0x55f68edefb33 in main (/build/cpp/debug/arrow-compute-scalar-cast-test+0x2adb33) (BuildId: c176b331f0537175685c7888a83fc8e112006f6a)
#0 0x7f80f7d06d8f in
#26 0x7f80f7d06e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: cd410b710f0f094c6832edd95931006d883af48e)
#27 0x55f68edefb94 in _start (/build/cpp/debug/arrow-compute-scalar-cast-test+0x2adb94) (BuildId: c176b331f0537175685c7888a83fc8e112006f6a)
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1064:34 in
/build/cpp/src/arrow/compute/kernels
Start 22: arrow-compute-scalar-cast-test
Test #22: arrow-compute-scalar-cast-test ...............***Failed 12.44 sec

@pitrou
pitrou
commented
Feb 11, 2025

Copy link


Copy Markdown

Member
Well, the Windows 2019 CI test shows a similar error, so I think it needs diagnosing and solving:
[ RUN ] Cast.StructToStructSubset
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\include\vector(1563) : Assertion failed: vector subscript out of range

pitrou
pitrou
reviewed
Feb 11, 2025
View reviewed changes
Comment threadcpp/src/arrow/compute/kernels/scalar_cast_nested.cc
Outdated

@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
2 times, most recently
from
45f1ac2 to
0959343CompareFebruary 11, 2025 17:07
@NickCrews
NickCrews
commented
Feb 11, 2025

Copy link


Copy Markdown

ContributorAuthor
I rebased on top of #45500 so that hopefully CI will pass
@pitrou
pitrouforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
0959343 to
4311938CompareFebruary 13, 2025 13:46
pitrou
pitrou
approved these changes
Feb 13, 2025
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. Thanks for this @NickCrews !
@pitrou
pitrou
commented
Feb 13, 2025

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g cpp
@github-actions
github-actionsBot
commented
Feb 13, 2025

Copy link


Copy Markdown

Revision: 4311938
Submitted crossbow builds: ursacomputing/crossbow @ actions-f1ea7df869
TaskStatus
example-cpp-minimal-build-staticGitHub Actions
example-cpp-minimal-build-static-system-dependencyGitHub Actions
example-cpp-tutorialGitHub Actions
test-alpine-linux-cppGitHub Actions
test-build-cpp-fuzzGitHub Actions
test-conda-cppGitHub Actions
test-conda-cpp-valgrindGitHub Actions
test-cuda-cpp-ubuntu-20.04-cuda-11.2.2GitHub Actions
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1GitHub Actions
test-debian-12-cpp-amd64GitHub Actions
test-debian-12-cpp-i386GitHub Actions
test-fedora-39-cppGitHub Actions
test-ubuntu-20.04-cppGitHub Actions
test-ubuntu-20.04-cpp-bundledGitHub Actions
test-ubuntu-22.04-cppGitHub Actions
test-ubuntu-22.04-cpp-20GitHub Actions
test-ubuntu-22.04-cpp-emscriptenGitHub Actions
test-ubuntu-22.04-cpp-no-threadingGitHub Actions
test-ubuntu-24.04-cppGitHub Actions
test-ubuntu-24.04-cpp-bundled-offlineGitHub Actions
test-ubuntu-24.04-cpp-gcc-13-bundledGitHub Actions
test-ubuntu-24.04-cpp-gcc-14GitHub Actions
test-ubuntu-24.04-cpp-minimal-with-formatsGitHub Actions
test-ubuntu-24.04-cpp-thread-sanitizerGitHub Actions
@pitrou
pitrou
commented
Feb 13, 2025

Copy link


Copy Markdown

Member
The CI failures are unrelated (see #45524).
@pitrou
pitrou
merged commit 6a47e4d
into
apache:mainFeb 13, 2025
@pitroupitrou
removed
the 
awaiting committer review
Awaiting committer review
label
Feb 13, 2025
@NickCrews
NickCrews
commented
Feb 13, 2025

Copy link


Copy Markdown

ContributorAuthor
Thank you everyone! That was a lot of effort everyone did, I wouldn't have been able to do that without all the help. I'm excited for this to be released which will allow me to remove some hacky workarounds I have in my app code.
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Feb 13, 2025

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 6a47e4d.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.
@Tom-NewtonTom-Newton
mentioned this pull request
Apr 23, 2025
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@zanmato1984zanmato1984zanmato1984 left review comments
@pitroupitroupitrou approved these changes
@mapleFUmapleFUmapleFU approved these changes
@zeroshadezeroshadeAwaiting requested review from zeroshade
@assignUserassignUserAwaiting requested review from assignUser
@jonkeanejonkeaneAwaiting requested review from jonkeane
@koukouAwaiting requested review from kou
@raulcdraulcdAwaiting requested review from raulcd

Assignees


No one assigned


Labels

Component: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

7 participants

@NickCrews@rustyconover@mapleFU@zeroshade@kou@pitrou@zanmato1984

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
GH-33592: [C++] support casting nullable fields to non-nullable if there are no null values - #43782
Merged
pitrou merged 3 commits into
apache:mainfrom 
NickCrews:field-cast-nullable-to-nonnullable
Feb 13, 2025
Merged
GH-33592: [C++] support casting nullable fields to non-nullable if there are no null values#43782
pitrou merged 3 commits into
apache:mainfrom 
NickCrews:field-cast-nullable-to-nonnullable
Conversation
@NickCrews
@NickCrewsNickCrews
commented
Aug 21, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Notes for myself/fixer:
  • tests that need to get updated (almost definitely not a complete list)
  • [update: actually we should handle the go implementation in the go repository.] hmm, looks like go wrapper does its own nullability checks. I assume this is just an optimization to not have to go into the C++ and hit the error down there. So if we just delete this check then I think the C++ logic will handle all of it???
  • how the plain column handles this cast, some logic like this probably needs to get ported over to the struct implementation
  • (running from /cpp/build) cmake .. --preset ninja-debug-basic, then cmake --build . && PYTHON=python ctest -R 'arrow-compute-scalar-cast-test' --output-on-failure to run the specific test
  • to format: uvx pre-commit run --all-files clang-format
@github-actions
github-actionsBot
commented
Aug 21, 2024

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:
@NickCrewsNickCrews
mentioned this pull request
Aug 21, 2024

Closed

@NickCrewsNickCrews
changed the title
[C++][Go]: support casting nullable fields to non-nullable if there are no null valuesGH-33592 [C++][Go]: support casting nullable fields to non-nullable if there are no null valuesAug 21, 2024
@github-actions
github-actionsBot
commented
Aug 21, 2024

Copy link


Copy Markdown

⚠️ GitHub issue #33592has been automatically assigned in GitHub to PR creator.
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
3 times, most recently
from
52100dd to
49d5a99CompareAugust 22, 2024 00:40
@rustyconover
rustyconover
commented
Sep 15, 2024

Copy link


Copy Markdown

This PR looks pretty good to me, what additional help would you like getting it merged?
@NickCrews
NickCrews
commented
Sep 15, 2024

Copy link


Copy Markdown

ContributorAuthor
Thanks for the help! I'm a random contributor, so I'm not sure about the exact workflow, but I think both getting the CI run approved and having a C++ owner approve it are needed.
Any more detailed thoughts on if the tests are adequate, if there's anywhere else in the code base that you think needs to change, my handling of the go implementation by just deleting the shortcircuit, or any other more detailed thoughts? Basically anything that would reduce the mental load on the code owner I think would increase the odds they approve it :)
@mapleFU
mapleFU
commented
Sep 15, 2024

Copy link


Copy Markdown

Member
You can mark as ready for review? Or this is still wip?
@NickCrews
NickCrews
marked this pull request as ready for review
September 15, 2024 20:11
@NickCrews
NickCrews
commented
Sep 15, 2024

Copy link


Copy Markdown

ContributorAuthor
Oops, didn't mean for this to still be marked WIP. Looks like a few failures that need to get fixed, but still a high-level review of the general approach would still be appreciated in the meantime.
@zeroshade
zeroshade
commented
Sep 15, 2024

Copy link


Copy Markdown

Member
You probably need the same corresponding check on the Go side to verify that it's only allowed if there are no nulls.
Also, the Go implementation has been moved to the apache/arrow-go repository. So please file the PR there for the Go side. Sorry for the confusion, we just haven't removed the Go code from here yet.
@NickCrews
NickCrews
commented
Sep 16, 2024

Copy link


Copy Markdown

ContributorAuthor
Thanks @zeroshade , will do. Do we need both PRs to land at about the same time, or can I do that independently?
I will undo my changes to the go code here, leaving it untouched.
@zeroshade
zeroshade
commented
Sep 16, 2024

Copy link


Copy Markdown

Member
They can land independently, no issues there. Thanks!
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
49d5a99 to
eb9e7b6CompareSeptember 16, 2024 16:27
@NickCrews
NickCrews
commented
Sep 16, 2024

Copy link


Copy Markdown

ContributorAuthor
Just pushed a new version:
  • dropped the Go changes
  • rebased on top of 3600db8, which is one commit behind main, because main is failing CI, but the commit I chose passed CI. I did this because I think the failing CI checks in this PR are not related to this PR
  • I discovered archery and formatted the files with archery lint --clang-format --fix
We will see if this passes CI now...
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
eb9e7b6 to
e6d54d3CompareSeptember 18, 2024 07:49
@NickCrews
NickCrews
commented
Sep 18, 2024

Copy link


Copy Markdown

ContributorAuthor
Pushed a new version, hopefully that fixed the broken tests:
  • To go from nullable to non-nullable type, need to use the unsafe cast option
  • fix typo of int8 to int64 so the precision matches
  • removed the [Go] tag from the commit message
@NickCrews
NickCrews
commented
Sep 18, 2024

Copy link


Copy Markdown

ContributorAuthor
@zeroshade I think this is ready to review/merge, the failing CI runs look like flakes when trying to setup the environment?
@NickCrewsNickCrews
mentioned this pull request
Sep 19, 2024

Open

@kou
kou
commented
Sep 19, 2024

Copy link


Copy Markdown

Member
Can we move this to apache/arrow-go?
@koukou
changed the title
GH-33592 [C++][Go]: support casting nullable fields to non-nullable if there are no null valuesGH-33592: [C++] support casting nullable fields to non-nullable if there are no null valuesSep 19, 2024
@kou
kou
commented
Sep 19, 2024

Copy link


Copy Markdown

Member
Ah, the Go part was removed from this PR.

I've removed "[Go]" from the PR title.
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
e6d54d3 to
106e627CompareSeptember 24, 2024 18:53
@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
b124c64 to
ebe9a5aCompareFebruary 7, 2025 08:01
@NickCrews
NickCrews
commented
Feb 7, 2025

Copy link


Copy Markdown

ContributorAuthor
@mapleFU as I added the slicing tests, I also refactored the tests to make them much more consistent and concise. Take another look and make sure that your "LGTM" still holds after those changes.
pitrou
pitrou
reviewed
Feb 7, 2025
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a nit!
Comment threadcpp/src/arrow/compute/kernels/scalar_cast_nested.cc
Outdated

@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
2 times, most recently
from
ce00550 to
ea241a5CompareFebruary 10, 2025 16:56
NickCrews
NickCrews
commented
Feb 10, 2025
View reviewed changes
Comment threadcpp/src/arrow/compute/kernels/scalar_cast_nested.cc
Outdated

@NickCrews
NickCrews
commented
Feb 10, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
I'm getting these failing UBSAN errors in CI. See below for the relevant logs. Is it because of the new std::vector<std::shared_ptr<Array>> arrays_dest_ac = {arrays_dest[0], arrays_dest[2]} code that I added? It looks like the line numbers referenced are the ones AFTER the macros/preprocessor occurs, so I'm not sure which line in the source file is actually the problem :( Is there a good way to silence/ignore this error? Any hint as to how to run this locally so I can reproduce/test?
Details
[ RUN ] Cast.StructToStructSubset
/usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1064:34: runtime error: addition of unsigned offset to 0x60700006f300 overflowed to 0x60700006f2e0
#0 0x7f811aca4bc8 in std::vector<std::shared_ptr<arrow::Field>, std::allocator<std::shared_ptr<arrow::Field> > >::operator[](unsigned long) const /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1064:34
#1 0x7f811ac9b27a in arrow::DataType::field(int) const /arrow/cpp/src/arrow/type.h:153:61
#2 0x7f8105e4f8f1 in arrow::compute::internal::(anonymous namespace)::CastStruct::Exec(arrow::compute::KernelContext*, arrow::compute::ExecSpan const&, arrow::compute::ExecResult*) /arrow/cpp/src/arrow/compute/kernels/scalar_cast_nested.cc:403:38
#3 0x7f8105ab206e in arrow::compute::detail::(anonymous namespace)::ScalarExecutor::ExecuteNonSpans(arrow::compute::detail::ExecListener*) /arrow/cpp/src/arrow/compute/exec.cc:920:7
#4 0x7f8105aa99fa in arrow::compute::detail::(anonymous namespace)::ScalarExecutor::Execute(arrow::compute::ExecBatch const&, arrow::compute::detail::ExecListener*) /arrow/cpp/src/arrow/compute/exec.cc:810:14
#5 0x7f8105c471a6 in arrow::compute::detail::FunctionExecutorImpl::Execute(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, long) /arrow/cpp/src/arrow/compute/function.cc:278:5
#6 0x7f8105c17bfe in arrow::compute::(anonymous namespace)::ExecuteInternal(arrow::compute::Function const&, std::vector<arrow::Datum, std::allocator<arrow::Datum> >, long, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) /arrow/cpp/src/arrow/compute/function.cc:343:21
#7 0x7f8105c16d84 in arrow::compute::Function::Execute(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) const /arrow/cpp/src/arrow/compute/function.cc:350:10
#8 0x7f8105a5d13c in arrow::compute::internal::(anonymous namespace)::CastMetaFunction::ExecuteImpl(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) const /arrow/cpp/src/arrow/compute/cast.cc:124:23
#9 0x7f8105c21b5c in arrow::compute::MetaFunction::Execute(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) const /arrow/cpp/src/arrow/compute/function.cc:483:10
#10 0x7f8105a8cfc8 in arrow::compute::CallFunction(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) /arrow/cpp/src/arrow/compute/exec.cc:1369:16
#11 0x55f68eeb23c9 in arrow::compute::CheckScalarNonRecursive(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::Datum const&, arrow::compute::FunctionOptions const*) /arrow/cpp/src/arrow/compute/kernels/test_util_internal.cc:80:3
#12 0x55f68eeb8480 in arrow::compute::CheckScalar(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::Datum, arrow::compute::FunctionOptions const*) /arrow/cpp/src/arrow/compute/kernels/test_util_internal.cc:109:3
#13 0x55f68eecc36c in arrow::compute::CheckScalarUnary(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, arrow::Datum, arrow::Datum, arrow::compute::FunctionOptions const*) /arrow/cpp/src/arrow/compute/kernels/test_util_internal.cc:255:3
#14 0x55f68ef4d8f7 in arrow::compute::CheckCast(std::shared_ptr<arrow::Array>, std::shared_ptr<arrow::Array>, arrow::compute::CastOptions) /arrow/cpp/src/arrow/compute/kernels/scalar_cast_test.cc:111:3
#15 0x55f68f079ab8 in arrow::compute::CheckStructToStructSubset(std::vector<std::shared_ptr<arrow::DataType>, std::allocator<std::shared_ptr<arrow::DataType> > > const&) /arrow/cpp/src/arrow/compute/kernels/scalar_cast_test.cc:3829:7
#16 0x55f68f06d09f in arrow::compute::Cast_StructToStructSubset_Test::TestBody() /arrow/cpp/src/arrow/compute/kernels/scalar_cast_test.cc:4003:36
#17 0x7f811af8260e in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/usr/local/lib/libarrow_testing.so.2000+0x108c60e) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#18 0x7f811af770c5 in testing::Test::Run() (/usr/local/lib/libarrow_testing.so.2000+0x10810c5) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#19 0x7f811af77244 in testing::TestInfo::Run() (/usr/local/lib/libarrow_testing.so.2000+0x1081244) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#20 0x7f811af777f8 in testing::TestSuite::Run() (/usr/local/lib/libarrow_testing.so.2000+0x10817f8) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#21 0x7f811af77efe in testing::internal::UnitTestImpl::RunAllTests() (/usr/local/lib/libarrow_testing.so.2000+0x1081efe) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#22 0x7f811af82bd6 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/usr/local/lib/libarrow_testing.so.2000+0x108cbd6) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#23 0x7f811af7730b in testing::UnitTest::Run() (/usr/local/lib/libarrow_testing.so.2000+0x108130b) (BuildId: 408b230591f17f240ad9d2b7d868274f1179009d)
#24 0x55f68edefb33 in main (/build/cpp/debug/arrow-compute-scalar-cast-test+0x2adb33) (BuildId: c176b331f0537175685c7888a83fc8e112006f6a)
#0 0x7f80f7d06d8f in
#26 0x7f80f7d06e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: cd410b710f0f094c6832edd95931006d883af48e)
#27 0x55f68edefb94 in _start (/build/cpp/debug/arrow-compute-scalar-cast-test+0x2adb94) (BuildId: c176b331f0537175685c7888a83fc8e112006f6a)
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1064:34 in
/build/cpp/src/arrow/compute/kernels
Start 22: arrow-compute-scalar-cast-test
Test #22: arrow-compute-scalar-cast-test ...............***Failed 12.44 sec

@pitrou
pitrou
commented
Feb 11, 2025

Copy link


Copy Markdown

Member
Well, the Windows 2019 CI test shows a similar error, so I think it needs diagnosing and solving:
[ RUN ] Cast.StructToStructSubset
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\include\vector(1563) : Assertion failed: vector subscript out of range

pitrou
pitrou
reviewed
Feb 11, 2025
View reviewed changes
Comment threadcpp/src/arrow/compute/kernels/scalar_cast_nested.cc
Outdated

@NickCrews
NickCrewsforce-pushed
the
field-cast-nullable-to-nonnullable
branch
2 times, most recently
from
45f1ac2 to
0959343CompareFebruary 11, 2025 17:07
@NickCrews
NickCrews
commented
Feb 11, 2025

Copy link


Copy Markdown

ContributorAuthor
I rebased on top of #45500 so that hopefully CI will pass
@pitrou
pitrouforce-pushed
the
field-cast-nullable-to-nonnullable
branch
from
0959343 to
4311938CompareFebruary 13, 2025 13:46
pitrou
pitrou
approved these changes
Feb 13, 2025
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. Thanks for this @NickCrews !
@pitrou
pitrou
commented
Feb 13, 2025

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g cpp
@github-actions
github-actionsBot
commented
Feb 13, 2025

Copy link


Copy Markdown

Revision: 4311938
Submitted crossbow builds: ursacomputing/crossbow @ actions-f1ea7df869
TaskStatus
example-cpp-minimal-build-staticGitHub Actions
example-cpp-minimal-build-static-system-dependencyGitHub Actions
example-cpp-tutorialGitHub Actions
test-alpine-linux-cppGitHub Actions
test-build-cpp-fuzzGitHub Actions
test-conda-cppGitHub Actions
test-conda-cpp-valgrindGitHub Actions
test-cuda-cpp-ubuntu-20.04-cuda-11.2.2GitHub Actions
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1GitHub Actions
test-debian-12-cpp-amd64GitHub Actions
test-debian-12-cpp-i386GitHub Actions
test-fedora-39-cppGitHub Actions
test-ubuntu-20.04-cppGitHub Actions
test-ubuntu-20.04-cpp-bundledGitHub Actions
test-ubuntu-22.04-cppGitHub Actions
test-ubuntu-22.04-cpp-20GitHub Actions
test-ubuntu-22.04-cpp-emscriptenGitHub Actions
test-ubuntu-22.04-cpp-no-threadingGitHub Actions
test-ubuntu-24.04-cppGitHub Actions
test-ubuntu-24.04-cpp-bundled-offlineGitHub Actions
test-ubuntu-24.04-cpp-gcc-13-bundledGitHub Actions
test-ubuntu-24.04-cpp-gcc-14GitHub Actions
test-ubuntu-24.04-cpp-minimal-with-formatsGitHub Actions
test-ubuntu-24.04-cpp-thread-sanitizerGitHub Actions
@pitrou
pitrou
commented
Feb 13, 2025

Copy link


Copy Markdown

Member
The CI failures are unrelated (see #45524).
@pitrou
pitrou
merged commit 6a47e4d
into
apache:mainFeb 13, 2025
@pitroupitrou
removed
the 
awaiting committer review
Awaiting committer review
label
Feb 13, 2025
@NickCrews
NickCrews
commented
Feb 13, 2025

Copy link


Copy Markdown

ContributorAuthor
Thank you everyone! That was a lot of effort everyone did, I wouldn't have been able to do that without all the help. I'm excited for this to be released which will allow me to remove some hacky workarounds I have in my app code.
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Feb 13, 2025

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 6a47e4d.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.
@Tom-NewtonTom-Newton
mentioned this pull request
Apr 23, 2025
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@zanmato1984zanmato1984zanmato1984 left review comments
@pitroupitroupitrou approved these changes
@mapleFUmapleFUmapleFU approved these changes
@zeroshadezeroshadeAwaiting requested review from zeroshade
@assignUserassignUserAwaiting requested review from assignUser
@jonkeanejonkeaneAwaiting requested review from jonkeane
@koukouAwaiting requested review from kou
@raulcdraulcdAwaiting requested review from raulcd

Assignees


No one assigned


Labels

Component: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

7 participants

@NickCrews@rustyconover@mapleFU@zeroshade@kou@pitrou@zanmato1984