ARROW-30: [Python] Routines for converting between arrow::Array/Table and pandas.DataFrame by wesm · Pull Request #46 · apache/arrow

wesm · 2016-03-28T07:24:43Z

There is a lot to do here for maximum compatibility, but this gets things started.

…y C API globally

xhochy · 2016-03-28T07:54:05Z

 pyarrow_init()
+
+import numpy as np
+pyarrow_set_numpy_nan(np.nan)


Why do we need explicitly numpy's nan here? Couldn't we use libc.math.NAN or libc.math.isnan instead? Then PyObject_is_null could save a memory access to the heap.

Ok, got it, we compare memory addresses, not content in PyObject_is_null.

Yep, this is a pandas thing -- there are object arrays with numpy.nan floating around in them -- otherwise you have to PyFloat_Check and then unbox to double, and only then verify that it is NaN.

xhochy · 2016-03-28T09:10:34Z

+1, looks good.

xhochy · 2016-03-28T09:23:06Z

Currently we have zero-copy pandas-to-arrow if possible but we always create a copy in the arrow-to-pandas direction?

wesm · 2016-03-28T15:41:34Z

At some point we could implement zero-copy arrow-to-pandas, but it would only be possible if there are no nulls. There would be a little bit of finagling involved (setting a PyArrow array as the base of the resulting ndarray, so that memory does not get freed by Arrow while it's being used in pandas/NumPy land)

xhochy · 2016-03-28T16:01:22Z

Seems like we need a custom numpy.ndarray subclass to handle this. Created https://issues.apache.org/jira/browse/ARROW-86 for this. This is one of the main things I want to see in Arrow, so I'll have a look soon.

wesm · 2016-03-28T16:35:52Z

I'll merge this for now -- this will be the target of ongoing cleaning and testing, so we can keep working on it in follow up JIRAs

…ode paths I also did a bit of tidying / reorganization and giving interfaces more descriptive names. Author: Wes McKinney <wes@cloudera.com> Closesapache#46 from wesm/PARQUET-501 and squashes the following commits: 491aa89 [Wes McKinney] * Add a basic OutputStream abstract interface and an InMemoryOutputStream implementation for testing. * Refactor to use OutputStream on data encoding paths, reduce some code duplication in column-reader-test. * Collect all input/output classes into util/input.* and util/output.*. * Use int64_t in InputStream::Peek/Read. Change-Id: I0f29001dc8f23e3176e109ae9909a6539f285364

This PR enables tests for `ARROW_COMPUTE`, `ARROW_DATASET`, `ARROW_FILESYSTEM`, `ARROW_HDFS`, `ARROW_ORC`, and `ARROW_IPC` (default on). #7131 enabled a minimal set of tests as a starting point. I confirmed that these tests pass locally with the current master. In the current TravisCI environment, we cannot see this result due to a lot of error messages in `arrow-utility-test`. ``` $ git log | head -1 commit ed5f534 % ctest ... Start 1: arrow-array-test 1/51 Test #1: arrow-array-test ..................... Passed 4.62 sec Start 2: arrow-buffer-test 2/51 Test #2: arrow-buffer-test .................... Passed 0.14 sec Start 3: arrow-extension-type-test 3/51 Test #3: arrow-extension-type-test ............ Passed 0.12 sec Start 4: arrow-misc-test 4/51 Test #4: arrow-misc-test ...................... Passed 0.14 sec Start 5: arrow-public-api-test 5/51 Test #5: arrow-public-api-test ................ Passed 0.12 sec Start 6: arrow-scalar-test 6/51 Test #6: arrow-scalar-test .................... Passed 0.13 sec Start 7: arrow-type-test 7/51 Test #7: arrow-type-test ...................... Passed 0.14 sec Start 8: arrow-table-test 8/51 Test #8: arrow-table-test ..................... Passed 0.13 sec Start 9: arrow-tensor-test 9/51 Test #9: arrow-tensor-test .................... Passed 0.13 sec Start 10: arrow-sparse-tensor-test 10/51 Test #10: arrow-sparse-tensor-test ............. Passed 0.16 sec Start 11: arrow-stl-test 11/51 Test #11: arrow-stl-test ....................... Passed 0.12 sec Start 12: arrow-concatenate-test 12/51 Test #12: arrow-concatenate-test ............... Passed 0.53 sec Start 13: arrow-diff-test 13/51 Test #13: arrow-diff-test ...................... Passed 1.45 sec Start 14: arrow-c-bridge-test 14/51 Test #14: arrow-c-bridge-test .................. Passed 0.18 sec Start 15: arrow-io-buffered-test 15/51 Test #15: arrow-io-buffered-test ............... Passed 0.20 sec Start 16: arrow-io-compressed-test 16/51 Test #16: arrow-io-compressed-test ............. Passed 3.48 sec Start 17: arrow-io-file-test 17/51 Test #17: arrow-io-file-test ................... Passed 0.74 sec Start 18: arrow-io-hdfs-test 18/51 Test #18: arrow-io-hdfs-test ................... Passed 0.12 sec Start 19: arrow-io-memory-test 19/51 Test #19: arrow-io-memory-test ................. Passed 2.77 sec Start 20: arrow-utility-test 20/51 Test #20: arrow-utility-test ...................***Failed 5.65 sec Start 21: arrow-threading-utility-test 21/51 Test #21: arrow-threading-utility-test ......... Passed 1.34 sec Start 22: arrow-compute-compute-test 22/51 Test #22: arrow-compute-compute-test ........... Passed 0.13 sec Start 23: arrow-compute-boolean-test 23/51 Test #23: arrow-compute-boolean-test ........... Passed 0.15 sec Start 24: arrow-compute-cast-test 24/51 Test #24: arrow-compute-cast-test .............. Passed 0.22 sec Start 25: arrow-compute-hash-test 25/51 Test #25: arrow-compute-hash-test .............. Passed 2.61 sec Start 26: arrow-compute-isin-test 26/51 Test #26: arrow-compute-isin-test .............. Passed 0.81 sec Start 27: arrow-compute-match-test 27/51 Test #27: arrow-compute-match-test ............. Passed 0.40 sec Start 28: arrow-compute-sort-to-indices-test 28/51 Test #28: arrow-compute-sort-to-indices-test ... Passed 3.33 sec Start 29: arrow-compute-nth-to-indices-test 29/51 Test #29: arrow-compute-nth-to-indices-test .... Passed 1.51 sec Start 30: arrow-compute-util-internal-test 30/51 Test #30: arrow-compute-util-internal-test ..... Passed 0.13 sec Start 31: arrow-compute-add-test 31/51 Test #31: arrow-compute-add-test ............... Passed 0.12 sec Start 32: arrow-compute-aggregate-test 32/51 Test #32: arrow-compute-aggregate-test ......... Passed 14.70 sec Start 33: arrow-compute-compare-test 33/51 Test #33: arrow-compute-compare-test ........... Passed 7.96 sec Start 34: arrow-compute-take-test 34/51 Test #34: arrow-compute-take-test .............. Passed 4.80 sec Start 35: arrow-compute-filter-test 35/51 Test #35: arrow-compute-filter-test ............ Passed 8.23 sec Start 36: arrow-dataset-dataset-test 36/51 Test #36: arrow-dataset-dataset-test ........... Passed 0.25 sec Start 37: arrow-dataset-discovery-test 37/51 Test #37: arrow-dataset-discovery-test ......... Passed 0.13 sec Start 38: arrow-dataset-file-ipc-test 38/51 Test #38: arrow-dataset-file-ipc-test .......... Passed 0.21 sec Start 39: arrow-dataset-file-test 39/51 Test #39: arrow-dataset-file-test .............. Passed 0.12 sec Start 40: arrow-dataset-filter-test 40/51 Test #40: arrow-dataset-filter-test ............ Passed 0.16 sec Start 41: arrow-dataset-partition-test 41/51 Test #41: arrow-dataset-partition-test ......... Passed 0.13 sec Start 42: arrow-dataset-scanner-test 42/51 Test #42: arrow-dataset-scanner-test ........... Passed 0.20 sec Start 43: arrow-filesystem-test 43/51 Test #43: arrow-filesystem-test ................ Passed 1.62 sec Start 44: arrow-hdfs-test 44/51 Test #44: arrow-hdfs-test ...................... Passed 0.13 sec Start 45: arrow-feather-test 45/51 Test #45: arrow-feather-test ................... Passed 0.91 sec Start 46: arrow-ipc-read-write-test 46/51 Test #46: arrow-ipc-read-write-test ............ Passed 5.77 sec Start 47: arrow-ipc-json-simple-test 47/51 Test #47: arrow-ipc-json-simple-test ........... Passed 0.16 sec Start 48: arrow-ipc-json-test 48/51 Test #48: arrow-ipc-json-test .................. Passed 0.27 sec Start 49: arrow-json-integration-test 49/51 Test #49: arrow-json-integration-test .......... Passed 0.13 sec Start 50: arrow-json-test 50/51 Test #50: arrow-json-test ...................... Passed 0.26 sec Start 51: arrow-orc-adapter-test 51/51 Test #51: arrow-orc-adapter-test ............... Passed 1.92 sec 98% tests passed, 1 tests failed out of 51 Label Time Summary: arrow-tests = 27.38 sec (27 tests) arrow_compute = 45.11 sec (14 tests) arrow_dataset = 1.21 sec (7 tests) arrow_ipc = 6.20 sec (3 tests) unittest = 79.91 sec (51 tests) Total Test time (real) = 79.99 sec The following tests FAILED: 20 - arrow-utility-test (Failed) Errors while running CTest ``` Closes#7142 from kiszk/ARROW-8754 Authored-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

[C++][Gandiva] Adding abs function

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

Fix build for s390

wesm added 6 commits March 27, 2016 12:28

Compiling, but untested draft of pandas <-> arrow converters
5e09bfe

Give libpyarrow a reference to numpy.nan
f0cc451

cpplint
d1f05c5

More pandas conversion scaffolding, enable libpyarrow to use the NumP…
8475a0e
…y C API globally

More scaffolding. Table wrapper. Initial unit tests passing
4c9f766

Invoke py.test with python -m pytest
0a9e747

xhochy reviewed Mar 28, 2016
View reviewed changes

asfgit closed this in 1fd0668Mar 28, 2016

wesm deleted the ARROW-30 branch March 28, 2016 16:36

jikunshang pushed a commit to jikunshang/arrow that referenced this pull request May 29, 2020
Merge pull requestapache#46from zhouyuan/wip_abs
7a60602
[C++][Gandiva] Adding abs function

zhouyuan added a commit to zhouyuan/arrow that referenced this pull request Nov 24, 2021
fix fastpfor (apache#46)
cded559
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

zhztheplayer pushed a commit to zhztheplayer/arrow-1 that referenced this pull request Feb 8, 2022
fix fastpfor (apache#46)
b9af487
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

zhztheplayer pushed a commit to zhztheplayer/arrow-1 that referenced this pull request Mar 3, 2022
fix fastpfor (apache#46)
952dbd6
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

rui-mo pushed a commit to rui-mo/arrow-1 that referenced this pull request Mar 23, 2022
fix fastpfor (apache#46)
d0204b7
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

paleolimbot mentioned this pull request Jan 28, 2023
[R] Crash on MacOS (x86) when running tests with homebrew apache-arrow also installed #33903
Closed

asfimport mentioned this pull request Feb 20, 2023
[Python] Illegal hardware instruction on pyarrow import #20848
Closed

binmahone pushed a commit to binmahone/arrow that referenced this pull request Nov 29, 2023
Merge pull requestapache#46from ClickHouse/fix-build-for-s390
8cdbf43
Fix build for s390

prniii mentioned this pull request Jan 12, 2024
[Python] macOS Segfault on Import, Both arm64 and x86_64 #37010
Open

jonkeane mentioned this pull request May 11, 2025
[C++][R]: gcc-UBSAN errors on CRAN #46394
Closed

github-actionsBot mentioned this pull request Apr 13, 2025
GH-45412: [JS] Drop support for Node.js 18 #45415
Closed

cbb330 added a commit to cbb330/arrow that referenced this pull request Feb 20, 2026
Mark Task#19.5 as complete (apache#46)
c83c87d

cbb330 added a commit to cbb330/arrow that referenced this pull request Feb 24, 2026
Mark Task#19.5 as complete (apache#46)
89b080c

Navigation Menu

Uh oh!

ARROW-30: [Python] Routines for converting between arrow::Array/Table and pandas.DataFrame - #46

ARROW-30: [Python] Routines for converting between arrow::Array/Table and pandas.DataFrame #46
wesm wants to merge 6 commits into
apache:masterfrom
wesm:ARROW-30

wesm commented Mar 28, 2016

Uh oh!

xhochyMar 28, 2016

Uh oh!

xhochyMar 28, 2016

Uh oh!

wesmMar 28, 2016

Uh oh!

xhochy commented Mar 28, 2016

Uh oh!

xhochy commented Mar 28, 2016

Uh oh!

wesm commented Mar 28, 2016

Uh oh!

xhochy commented Mar 28, 2016

Uh oh!

wesm commented Mar 28, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

wesm commented Mar 28, 2016

Uh oh!

xhochyMar 28, 2016

Choose a reason for hiding this comment

Uh oh!

xhochyMar 28, 2016

Choose a reason for hiding this comment

Uh oh!

wesmMar 28, 2016

Choose a reason for hiding this comment

Uh oh!

xhochy commented Mar 28, 2016

Uh oh!

xhochy commented Mar 28, 2016

Uh oh!

wesm commented Mar 28, 2016

Uh oh!

xhochy commented Mar 28, 2016

Uh oh!

wesm commented Mar 28, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants