Skip to content

Minor edits to C data interface doc - #6

Merged
pitrou merged 1 commit into
pitrou:doc-c-protocolfrom
nealrichardson:doc-c-protocol
Oct 17, 2019
Merged

Minor edits to C data interface doc#6
pitrou merged 1 commit into
pitrou:doc-c-protocolfrom
nealrichardson:doc-c-protocol

Conversation

@nealrichardson

@nealrichardsonnealrichardson commented Oct 17, 2019

Copy link
Copy Markdown

No description provided.

@pitrou

pitrou commented Oct 17, 2019

Copy link
Copy Markdown
Owner

Thank you :-)

@pitrou
pitrou merged this pull request into pitrou:doc-c-protocolOct 17, 2019
nealrichardson
nealrichardson commented Oct 17, 2019
View reviewed changes
Comment threaddocs/source/format/CDataInterface.rst
Comment threaddocs/source/format/CDataInterface.rst
* Enable integration without an explicit dependency (either at compile-time
or runtime) on the Arrow software project.

Ideally, the Arrow C data interface can become a low-level *lingua franca*

@nealrichardsonnealrichardsonOct 17, 2019

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will we change this wording to be more declarative and less hypothetical when this proposal is accepted?

@pitroupitrouOct 17, 2019•
edited
Loading

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can. But a spec alone can't determine that it's sufficiently used to actually become a lingua franca ;-)

Comment threaddocs/source/format/CDataInterface.rst
* Being a streamable format, the IPC format has room for composing more features (such as
integrity checks, compression...).

Data type description -- format strings

@nealrichardsonnealrichardsonOct 17, 2019

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other questions (I can't comment that far outside the diff:

  • L322 "If omitted, MUST be 0." That doesn't sound right.
  • Should we note below that you can use the ArrowArray struct shape to wrap up RecordBatches and other things?
  • "Examples" should probably point to the C++/Python/R stuff we did, right? (If not in this proposal, then later)

@pitroupitrouOct 17, 2019

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"If omitted, MUST be 0": I mean that the information is meant to be omitted, then the field should be set to 0. Do you want to propose another wording?

Exporting RecordBatches: yes, I could add a paragraph somewhere later.

I also could add pointers to in-progress implementations somewhere towards the end.

@nealrichardsonnealrichardsonOct 17, 2019

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess what was confusing to me was that setting a field to 0 doesn't sound like omitting it. If I were omitting it, I wouldn't set anything. It sounds like you need to set it always, but if you don't want to enable any of the options, it should be 0--is that right?

@pitroupitrouOct 17, 2019

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you don't set anything, then it could be any junk value, though (C doesn't zero-initialize stuff automatically).

@pitroupitrouOct 17, 2019

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, yes, it needs to be set to the value meaning "omitted" (in R it would be NA? :-)).

pitrou pushed a commit that referenced this pull request May 12, 2020
@kiszk@kou
ARROW-8754: [C++][CI] Enable additional tests on s390x
392cb9e
This PR enables tests for `ARROW_COMPUTE`, `ARROW_DATASET`, `ARROW_FILESYSTEM`, `ARROW_HDFS`, `ARROW_ORC`, and `ARROW_IPC` (default on). apache#7131 enabled a minimal set of tests as a starting point.
I confirmed that these tests pass locally with the current master. In the current TravisCI environment, we cannot see this result due to a lot of error messages in `arrow-utility-test`.
```
$ git log | head -1
commit ed5f534
% ctest
...
Start 1: arrow-array-test
1/51 Test #1: arrow-array-test ..................... Passed 4.62 sec
Start 2: arrow-buffer-test
2/51 Test #2: arrow-buffer-test .................... Passed 0.14 sec
Start 3: arrow-extension-type-test
3/51 Test #3: arrow-extension-type-test ............ Passed 0.12 sec
Start 4: arrow-misc-test
4/51 Test #4: arrow-misc-test ...................... Passed 0.14 sec
Start 5: arrow-public-api-test
5/51 Test #5: arrow-public-api-test ................ Passed 0.12 sec
Start 6: arrow-scalar-test
6/51 Test #6: arrow-scalar-test .................... Passed 0.13 sec
Start 7: arrow-type-test
7/51 Test #7: arrow-type-test ...................... Passed 0.14 sec
Start 8: arrow-table-test
8/51 Test #8: arrow-table-test ..................... Passed 0.13 sec
Start 9: arrow-tensor-test
9/51 Test #9: arrow-tensor-test .................... Passed 0.13 sec
Start 10: arrow-sparse-tensor-test
10/51 Test #10: arrow-sparse-tensor-test ............. Passed 0.16 sec
Start 11: arrow-stl-test
11/51 Test #11: arrow-stl-test ....................... Passed 0.12 sec
Start 12: arrow-concatenate-test
12/51 Test #12: arrow-concatenate-test ............... Passed 0.53 sec
Start 13: arrow-diff-test
13/51 Test #13: arrow-diff-test ...................... Passed 1.45 sec
Start 14: arrow-c-bridge-test
14/51 Test apache#14: arrow-c-bridge-test .................. Passed 0.18 sec
Start 15: arrow-io-buffered-test
15/51 Test apache#15: arrow-io-buffered-test ............... Passed 0.20 sec
Start 16: arrow-io-compressed-test
16/51 Test apache#16: arrow-io-compressed-test ............. Passed 3.48 sec
Start 17: arrow-io-file-test
17/51 Test apache#17: arrow-io-file-test ................... Passed 0.74 sec
Start 18: arrow-io-hdfs-test
18/51 Test apache#18: arrow-io-hdfs-test ................... Passed 0.12 sec
Start 19: arrow-io-memory-test
19/51 Test apache#19: arrow-io-memory-test ................. Passed 2.77 sec
Start 20: arrow-utility-test
20/51 Test apache#20: arrow-utility-test ...................***Failed 5.65 sec
Start 21: arrow-threading-utility-test
21/51 Test apache#21: arrow-threading-utility-test ......... Passed 1.34 sec
Start 22: arrow-compute-compute-test
22/51 Test apache#22: arrow-compute-compute-test ........... Passed 0.13 sec
Start 23: arrow-compute-boolean-test
23/51 Test apache#23: arrow-compute-boolean-test ........... Passed 0.15 sec
Start 24: arrow-compute-cast-test
24/51 Test apache#24: arrow-compute-cast-test .............. Passed 0.22 sec
Start 25: arrow-compute-hash-test
25/51 Test apache#25: arrow-compute-hash-test .............. Passed 2.61 sec
Start 26: arrow-compute-isin-test
26/51 Test apache#26: arrow-compute-isin-test .............. Passed 0.81 sec
Start 27: arrow-compute-match-test
27/51 Test apache#27: arrow-compute-match-test ............. Passed 0.40 sec
Start 28: arrow-compute-sort-to-indices-test
28/51 Test apache#28: arrow-compute-sort-to-indices-test ... Passed 3.33 sec
Start 29: arrow-compute-nth-to-indices-test
29/51 Test apache#29: arrow-compute-nth-to-indices-test .... Passed 1.51 sec
Start 30: arrow-compute-util-internal-test
30/51 Test apache#30: arrow-compute-util-internal-test ..... Passed 0.13 sec
Start 31: arrow-compute-add-test
31/51 Test apache#31: arrow-compute-add-test ............... Passed 0.12 sec
Start 32: arrow-compute-aggregate-test
32/51 Test apache#32: arrow-compute-aggregate-test ......... Passed 14.70 sec
Start 33: arrow-compute-compare-test
33/51 Test apache#33: arrow-compute-compare-test ........... Passed 7.96 sec
Start 34: arrow-compute-take-test
34/51 Test apache#34: arrow-compute-take-test .............. Passed 4.80 sec
Start 35: arrow-compute-filter-test
35/51 Test apache#35: arrow-compute-filter-test ............ Passed 8.23 sec
Start 36: arrow-dataset-dataset-test
36/51 Test apache#36: arrow-dataset-dataset-test ........... Passed 0.25 sec
Start 37: arrow-dataset-discovery-test
37/51 Test apache#37: arrow-dataset-discovery-test ......... Passed 0.13 sec
Start 38: arrow-dataset-file-ipc-test
38/51 Test apache#38: arrow-dataset-file-ipc-test .......... Passed 0.21 sec
Start 39: arrow-dataset-file-test
39/51 Test apache#39: arrow-dataset-file-test .............. Passed 0.12 sec
Start 40: arrow-dataset-filter-test
40/51 Test apache#40: arrow-dataset-filter-test ............ Passed 0.16 sec
Start 41: arrow-dataset-partition-test
41/51 Test apache#41: arrow-dataset-partition-test ......... Passed 0.13 sec
Start 42: arrow-dataset-scanner-test
42/51 Test apache#42: arrow-dataset-scanner-test ........... Passed 0.20 sec
Start 43: arrow-filesystem-test
43/51 Test apache#43: arrow-filesystem-test ................ Passed 1.62 sec
Start 44: arrow-hdfs-test
44/51 Test apache#44: arrow-hdfs-test ...................... Passed 0.13 sec
Start 45: arrow-feather-test
45/51 Test apache#45: arrow-feather-test ................... Passed 0.91 sec
Start 46: arrow-ipc-read-write-test
46/51 Test apache#46: arrow-ipc-read-write-test ............ Passed 5.77 sec
Start 47: arrow-ipc-json-simple-test
47/51 Test apache#47: arrow-ipc-json-simple-test ........... Passed 0.16 sec
Start 48: arrow-ipc-json-test
48/51 Test apache#48: arrow-ipc-json-test .................. Passed 0.27 sec
Start 49: arrow-json-integration-test
49/51 Test apache#49: arrow-json-integration-test .......... Passed 0.13 sec
Start 50: arrow-json-test
50/51 Test apache#50: arrow-json-test ...................... Passed 0.26 sec
Start 51: arrow-orc-adapter-test
51/51 Test apache#51: arrow-orc-adapter-test ............... Passed 1.92 sec
98% tests passed, 1 tests failed out of 51
Label Time Summary:
arrow-tests = 27.38 sec (27 tests)
arrow_compute = 45.11 sec (14 tests)
arrow_dataset = 1.21 sec (7 tests)
arrow_ipc = 6.20 sec (3 tests)
unittest = 79.91 sec (51 tests)
Total Test time (real) = 79.99 sec
The following tests FAILED:
20 - arrow-utility-test (Failed)
Errors while running CTest
```
Closesapache#7142 from kiszk/ARROW-8754
Authored-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
pitrou added a commit that referenced this pull request Apr 7, 2021
@westonpace@pitrou
ARROW-12040: [C++] Fix potential deadlock in recursive S3 walks
be8892e
From a deadlocked run...
```
#0 0x00007f8a5d48dccd in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f8a5d486f05 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f8a566e7e89 in arrow::internal::FnOnce<void ()>::FnImpl<arrow::Future<Aws::Utils::Outcome<Aws::S3::Model::ListObjectsV2Result, Aws::S3::S3Error> >::Callback<arrow::fs::(anonymous namespace)::TreeWalker::ListObjectsV2Handler> >::invoke() () from /arrow/r/check/arrow.Rcheck/arrow/libs/arrow.so
#3 0x00007f8a5650efa0 in arrow::FutureImpl::AddCallback(arrow::internal::FnOnce<void ()>) () from /arrow/r/check/arrow.Rcheck/arrow/libs/arrow.so
#4 0x00007f8a566e67a9 in arrow::fs::(anonymous namespace)::TreeWalker::ListObjectsV2Handler::SpawnListObjectsV2() () from /arrow/r/check/arrow.Rcheck/arrow/libs/arrow.so
#5 0x00007f8a566e723f in arrow::fs::(anonymous namespace)::TreeWalker::WalkChild(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int) () from /arrow/r/check/arrow.Rcheck/arrow/libs/arrow.so
#6 0x00007f8a566e827d in arrow::internal::FnOnce<void ()>::FnImpl<arrow::Future<Aws::Utils::Outcome<Aws::S3::Model::ListObjectsV2Result, Aws::S3::S3Error> >::Callback<arrow::fs::(anonymous namespace)::TreeWalker::ListObjectsV2Handler> >::invoke() () from /arrow/r/check/arrow.Rcheck/arrow/libs/arrow.so
#7 0x00007f8a5650efa0 in arrow::FutureImpl::AddCallback(arrow::internal::FnOnce<void ()>) () from /arrow/r/check/arrow.Rcheck/arrow/libs/arrow.so
#8 0x00007f8a566e67a9 in arrow::fs::(anonymous namespace)::TreeWalker::ListObjectsV2Handler::SpawnListObjectsV2() () from /arrow/r/check/arrow.Rcheck/arrow/libs/arrow.so
#9 0x00007f8a566e723f in arrow::fs::(anonymous namespace)::TreeWalker::WalkChild(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int) () from /arrow/r/check/arrow.Rcheck/arrow/libs/arrow.so
#10 0x00007f8a566e74b1 in arrow::fs::(anonymous namespace)::TreeWalker::DoWalk() () from /arrow/r/check/arrow.Rcheck/arrow/libs/arrow.so
```
The callback `ListObjectsV2Handler` is being called recursively and the mutex is non-reentrant thus deadlock.
To fix it I got rid of the mutex on `TreeWalker` by using `arrow::util::internal::TaskGroup` instead of manually tracking the #/status of in-flight requests.
Closesapache#9842 from westonpace/bugfix/arrow-12040
Lead-authored-by: Weston Pace <weston.pace@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pitroupitroupitrou left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@nealrichardson@pitrou