[C++] Sporadic test errors in parallel mode

[pitrou]

Describe the bug, including details regarding any error messages, version, and platform.

I run C++ tests in parallel (26 threads) and I have started to get random sporadic errors like this:

76/97 Test #13: arrow-gtest-util-test ........................***Failed 0.15 sec
Running arrow-gtest-util-test, redirecting output into /build/build-test/build/test-logs/arrow-gtest-util-test.txt (attempt 1/1)
Running main() from gmock_main.cc
[==========] Running 8 tests from 3 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from TestAssertContainsNaN
[ RUN ] TestAssertContainsNaN.BatchesEqual
[ OK ] TestAssertContainsNaN.BatchesEqual (1 ms)
[ RUN ] TestAssertContainsNaN.TableEqual
[ OK ] TestAssertContainsNaN.TableEqual (0 ms)
[ RUN ] TestAssertContainsNaN.ArrayEqual
[ OK ] TestAssertContainsNaN.ArrayEqual (0 ms)
[ RUN ] TestAssertContainsNaN.ChunkedEqual
[ OK ] TestAssertContainsNaN.ChunkedEqual (0 ms)
[ RUN ] TestAssertContainsNaN.DatumEqual
[ OK ] TestAssertContainsNaN.DatumEqual (0 ms)
[----------] 5 tests from TestAssertContainsNaN (2 ms total)

[----------] 2 tests from TestTensorFromJSON
[ RUN ] TestTensorFromJSON.FromJSONAndArray
[ OK ] TestTensorFromJSON.FromJSONAndArray (0 ms)
[ RUN ] TestTensorFromJSON.FromJSON
[ OK ] TestTensorFromJSON.FromJSON (0 ms)
[----------] 2 tests from TestTensorFromJSON (0 ms total)

[----------] 1 test from AssertTestWithinUlp
[ RUN ] AssertTestWithinUlp.Basics
[ OK ] AssertTestWithinUlp.Basics (0 ms)
[----------] 1 test from AssertTestWithinUlp (0 ms total)

[----------] Global test environment tear-down
[==========] 8 tests from 3 test suites ran. (3 ms total)
[ PASSED ] 8 tests.
find: ‘/tmp/s3fs-test-7cm44otu’: Aucun fichier ou dossier de ce nom

Component(s)

C++

[pitrou]

I think this is due to the use of GNU find in #50935 . @kou @hiroyuki-sato

[pitrou]

It seems to happen in print_coredumps. I've tried passing -ignore_readdir_race but it doesn't seem to change everything.

Edit: the patch I've tried:

diff --git a/cpp/build-support/run-test.sh b/cpp/build-support/run-test.sh
index 2dc18860a1..9f2828a9d9 100755
--- a/cpp/build-support/run-test.sh
+++ b/cpp/build-support/run-test.sh
@@ -141,7 +141,7 @@ function print_coredumps() {
 FILENAME=$(basename "${TEST_EXECUTABLE}")
 FILENAME=$(echo "${FILENAME}" | cut -c-15)
 
- COREFILES=$(find /tmp -maxdepth 1 -type f -name "core.${FILENAME}*" -exec basename {} \;)
+ COREFILES=$(find /tmp -ignore_readdir_race -maxdepth 1 -type f -name "core.${FILENAME}*" -exec basename {} \;)
 if [ -n "$COREFILES" ]; then
 for COREFILE in $COREFILES; do
 COREPATH="/tmp/${COREFILE}"
@@ -199,12 +199,12 @@ for ATTEMPT_NUMBER in $(seq 1 "$TEST_EXECUTION_ATTEMPTS") ; do
 # contents before and after the test runs, and deleting anything new.
 #
 # The comm program requires that its two inputs be sorted.
- TEST_TMPDIR_BEFORE=$(find "$TEST_TMPDIR" -maxdepth 1 -type d | sort)
+ TEST_TMPDIR_BEFORE=$(find "$TEST_TMPDIR" -ignore_readdir_race -maxdepth 1 -type d | sort)
 fi
 
 if [ "$ATTEMPT_NUMBER" -lt "$TEST_EXECUTION_ATTEMPTS" ]; then
 # Now delete any new test output.
- TEST_TMPDIR_AFTER=$(find "$TEST_TMPDIR" -maxdepth 1 -type d | sort)
+ TEST_TMPDIR_AFTER=$(find "$TEST_TMPDIR" -ignore_readdir_race -maxdepth 1 -type d | sort)
 DIFF=$(comm -13 <(echo "$TEST_TMPDIR_BEFORE") \
 <(echo "$TEST_TMPDIR_AFTER"))
 for DIR in $DIFF; do

[pitrou]

What we could do perhaps is only print coredumps on error. This would not eliminate a potential race condition in find, but would make it much less likely:

diff --git a/cpp/build-support/run-test.sh b/cpp/build-support/run-test.sh
index 2dc18860a1..c618e90c98 100755
--- a/cpp/build-support/run-test.sh
+++ b/cpp/build-support/run-test.sh
@@ -229,18 +229,16 @@ for ATTEMPT_NUMBER in $(seq 1 "$TEST_EXECUTION_ATTEMPTS") ; do
 fi
 if [ "$STATUS" -eq "0" ]; then
 break
- elif [ "$ATTEMPT_NUMBER" -lt "$TEST_EXECUTION_ATTEMPTS" ]; then
- echo Test failed attempt number "$ATTEMPT_NUMBER"
- echo Will retry...
 fi
+ print_coredumps
+ echo Test failed attempt number "$ATTEMPT_NUMBER"
+ echo Will retry...
 done
 
 if [ "$RUN_TYPE" = "test" ]; then
 post_process_tests
 fi
 
-print_coredumps
-
 popd
 rm -Rf "$TEST_WORKDIR"
 

[kou]

Can we close this because we merged #51110 ?

[pitrou]

Can we close this because we merged #51110 ?

That PR doesn't fix the issue, I still see error occasionally.

[----------] Global test environment tear-down
[==========] 21 tests from 7 test suites ran. (127 ms total)
[ PASSED ] 21 tests.
find: ‘/tmp/buffered-test-0gzee19x’: Aucun fichier ou dossier de ce nom

The problem is in print_coredumps, which is still there.

[pitrou]

This patch avoids the problem if all tests pass successfully:

diff --git a/cpp/build-support/run-test.sh b/cpp/build-support/run-test.sh
index ae39bef671..e8e1629104 100755
--- a/cpp/build-support/run-test.sh
+++ b/cpp/build-support/run-test.sh
@@ -200,7 +200,9 @@ if [ "$RUN_TYPE" = "test" ]; then
 post_process_tests
 fi
 
-print_coredumps
+if [ "$STATUS" -ne "0" ]; then
+ print_coredumps
+fi
 
 popd
 rm -Rf "$TEST_WORKDIR"

Would you be ok with this @kou ?

[hiroyuki-sato]

@pitrou

In #50934, find was used instead of (ls /tmp | grep $PATTERN) to find core files in a ShellCheck-safe way, but the fix was incomplete. Could you try #51121?

[hiroyuki-sato]

@pitrou
By the way, is this error from a script before #51110 was merged?
After #51110 was merged, the only place in run-test.sh that uses find is the part that looks for core files.
#51095 (comment)

[pitrou]

@hiroyuki-sato Have you read #51095 (comment) ?

[hiroyuki-sato]

@pitrou Thank you for your comment.

In the latest run-test.sh at commit a769c29, this is the only place where find is used:

grep find cpp/build-support/run-test.sh
 COREFILES=$(find /tmp -maxdepth 1 -type f -name "core.${FILENAME}*" -exec basename {} \;)

Since the error below appears to come from the find command, I asked because I still haven't been able to identify exactly where the error is coming from.

find: ‘/tmp/buffered-test-0gzee19x’: Aucun fichier ou dossier de ce nom

For example, this is the kind of error you get when running a command like this:

find /tmp/buffered-test-0gzee19x -type d -print

find: /tmp/buffered-test-0gzee19x: No such file or directory

[pitrou]

Since the error below appears to come from the find command, I asked because I still haven't been able to identify exactly where the error is coming from.

As I said, it comes from the find in print_coredumps. This command iterates over directory entries in /tmp and then filters them using -type f -name "core.${FILENAME}*". Perhaps this involves a stat call that fails because the directory entry has vanished in the meantime.