ARROW-12040: [C++] Fix potential deadlock in recursive S3 walks

[westonpace]

From a deadlocked run...

#0 0x00007f8a5d48dccd in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f8a5d486f05 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f8a566e7e89 in arrow::internal::FnOnce<void ()>::FnImpl<arrow::Future<Aws::Utils::Outcome<Aws::S3::Model::ListObjectsV2Result, Aws::S3::S3Error> >::Callback<arrow::fs::(anonymous namespace)::TreeWalker::ListObjectsV2Handler> >::invoke() () from /arrow/r/check/arrow.Rcheck/arrow/libs/arrow.so
#3 0x00007f8a5650efa0 in arrow::FutureImpl::AddCallback(arrow::internal::FnOnce<void ()>) () from /arrow/r/check/arrow.Rcheck/arrow/libs/arrow.so
#4 0x00007f8a566e67a9 in arrow::fs::(anonymous namespace)::TreeWalker::ListObjectsV2Handler::SpawnListObjectsV2() () from /arrow/r/check/arrow.Rcheck/arrow/libs/arrow.so
#5 0x00007f8a566e723f in arrow::fs::(anonymous namespace)::TreeWalker::WalkChild(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int) () from /arrow/r/check/arrow.Rcheck/arrow/libs/arrow.so
#6 0x00007f8a566e827d in arrow::internal::FnOnce<void ()>::FnImpl<arrow::Future<Aws::Utils::Outcome<Aws::S3::Model::ListObjectsV2Result, Aws::S3::S3Error> >::Callback<arrow::fs::(anonymous namespace)::TreeWalker::ListObjectsV2Handler> >::invoke() () from /arrow/r/check/arrow.Rcheck/arrow/libs/arrow.so
#7 0x00007f8a5650efa0 in arrow::FutureImpl::AddCallback(arrow::internal::FnOnce<void ()>) () from /arrow/r/check/arrow.Rcheck/arrow/libs/arrow.so
#8 0x00007f8a566e67a9 in arrow::fs::(anonymous namespace)::TreeWalker::ListObjectsV2Handler::SpawnListObjectsV2() () from /arrow/r/check/arrow.Rcheck/arrow/libs/arrow.so
#9 0x00007f8a566e723f in arrow::fs::(anonymous namespace)::TreeWalker::WalkChild(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int) () from /arrow/r/check/arrow.Rcheck/arrow/libs/arrow.so
#10 0x00007f8a566e74b1 in arrow::fs::(anonymous namespace)::TreeWalker::DoWalk() () from /arrow/r/check/arrow.Rcheck/arrow/libs/arrow.so

The callback ListObjectsV2Handler is being called recursively and the mutex is non-reentrant thus deadlock.

To fix it I got rid of the mutex on TreeWalker by using arrow::util::internal::TaskGroup instead of manually tracking the #/status of in-flight requests.

[github-actions]

https://issues.apache.org/jira/browse/ARROW-12040

[pitrou]

Hmm, I'd rather not use TaskGroup here, because it will have to be undone when implementing streaming GetFileInfo (ARROW-11924).

[westonpace]

I think it would end up using something very similar if not exactly TaskGroup in order to know when to write the end token to the generator. Something like

task_group_.FinishAsync().Then([producer] {
producer.Close();
});

Did you have an alternative approach in mind?

[pitrou]

I was thinking that this would make producing a stream of futures more difficult, but perhaps that's not the case actually.

[jonkeane]

It looks like the test that is hanging now is https://github.com/apache/arrow/blob/master/r/tests/testthat/test-dataset.R#L1345-L1367 which interestingly is skipped on windows reported in ARROW-9651. They might be different problems / solutions, but in case it provides clues about what might be going on.

[westonpace]

@jonkeane That second failure I have tracked now under ARROW-12161

@pitrou Future::All, Future::AllCompleted, and Future::Any (not sure if we've made this yet) are all related but none of them are really usable here since it is a recursive operation. I think some sort of future-centric solution could be devised but I don't see at the moment how it would be any nicer that TaskGroup.

[westonpace]

I rebased and got rid of the R build changes (which have since been addressed by ARROW-12143)

[westonpace]

@pitrou R3.5 failure is known and unrelated. JNI failure & Travis failure appear to be spurious failures. Consider review/merge.

[pitrou]

If you're capturing *this by value then you don't need to capture req and walker as well.

[westonpace]

Removed the extraneous captures.

[pitrou]

Is there a reason not to use std::mutex here?

[westonpace]

Nope. For some reason I thought the old implementation was using it. I've switched it back to std::mutex.

[westonpace]

I believe I've addressed the PR comments, CI is green. This is ready for re-review/merge.

[pitrou]

+1

[pitrou]

@github-actions crossbow submit -g r

[github-actions]

Revision: df50ffe

Submitted crossbow builds: ursacomputing/crossbow @ actions-274

Task	Status
conda-linux-gcc-py36-cpu-r36
conda-linux-gcc-py37-cpu-r40
conda-osx-clang-py36-r36
conda-osx-clang-py37-r40
conda-win-vs2017-py36-r36
conda-win-vs2017-py37-r40
homebrew-r-autobrew
test-r-install-local
test-r-linux-as-cran
test-r-minimal-build
test-r-rhub-ubuntu-gcc-release
test-r-rocker-r-base-latest
test-r-rstudio-r-base-3.6-bionic
test-r-rstudio-r-base-3.6-centos7-devtoolset-8
test-r-rstudio-r-base-3.6-centos8
test-r-rstudio-r-base-3.6-opensuse15
test-r-rstudio-r-base-3.6-opensuse42
test-r-version-compatibility
test-r-versions
test-ubuntu-18.04-r-sanitizer

[pitrou]

Thanks a lot @westonpace , I'm merging now.