ARROW-18340: [Python] PyArrow C++ header files no longer always included in installed pyarrow - #14656

Merged
kou merged 7 commits into
apache:masterfrom
AlenkaF:ARROW-18340
Nov 22, 2022
Merged

ARROW-18340: [Python] PyArrow C++ header files no longer always included in installed pyarrow#14656
kou merged 7 commits into
apache:masterfrom
AlenkaF:ARROW-18340

Conversation

@AlenkaF

@AlenkaFAlenkaF commented Nov 16, 2022

Copy link
Copy Markdown
Member

This PR adds code to _run_cmake to always copy PyArrow C++ header files to pyarrow/iclude.

@github-actions

github-actionsBot commented Nov 16, 2022

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-18340

@github-actions

github-actionsBot commented Nov 16, 2022

Copy link
Copy Markdown

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

jorisvandenbossche
jorisvandenbossche reviewed Nov 16, 2022
View reviewed changes
Comment threadpython/setup.py Outdated
@jorisvandenbossche

jorisvandenbossche commented Nov 16, 2022

Copy link
Copy Markdown
Member

@github-actions crossbow submit -g wheel

@github-actions

github-actionsBot commented Nov 16, 2022

Copy link
Copy Markdown

Revision: d5f953f

Submitted crossbow builds: ursacomputing/crossbow @ actions-1c3aa4b08e

TaskStatus
wheel-macos-big-sur-cp310-arm64Github Actions
wheel-macos-big-sur-cp311-arm64Github Actions
wheel-macos-big-sur-cp38-arm64Github Actions
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-mojave-cp310-amd64Github Actions
wheel-macos-mojave-cp311-amd64Github Actions
wheel-macos-mojave-cp37-amd64Github Actions
wheel-macos-mojave-cp38-amd64Github Actions
wheel-macos-mojave-cp39-amd64Github Actions
wheel-manylinux2014-cp310-amd64Github Actions
wheel-manylinux2014-cp310-arm64Travis CI
wheel-manylinux2014-cp311-amd64Github Actions
wheel-manylinux2014-cp311-arm64Travis CI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64Travis CI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64Travis CI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64Travis CI
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions

kou
kou reviewed Nov 16, 2022
View reviewed changes
Comment threadpython/setup.py Outdated
@kou

kou commented Nov 16, 2022

Copy link
Copy Markdown
Member

@github-actions crossbow submit -g python -g conda

@github-actions

github-actionsBot commented Nov 16, 2022

Copy link
Copy Markdown

Revision: d5f953f

Submitted crossbow builds: ursacomputing/crossbow @ actions-9eb5a3c4ca

TaskStatus
conda-cleanAzure
conda-linux-gcc-py310-arm64Azure
conda-linux-gcc-py310-cpuAzure
conda-linux-gcc-py310-cudaAzure
conda-linux-gcc-py310-ppc64leAzure
conda-linux-gcc-py37-arm64Azure
conda-linux-gcc-py37-cpu-r40Azure
conda-linux-gcc-py37-cpu-r41Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py37-ppc64leAzure
conda-linux-gcc-py38-arm64Azure
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py38-ppc64leAzure
conda-linux-gcc-py39-arm64Azure
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-linux-gcc-py39-ppc64leAzure
conda-osx-arm64-clang-py310Azure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py310Azure
conda-osx-clang-py37-r40Azure
conda-osx-clang-py37-r41Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2019-py310Azure
conda-win-vs2019-py37-r40Azure
conda-win-vs2019-py37-r41Azure
conda-win-vs2019-py38Azure
conda-win-vs2019-py39Azure
test-conda-python-3.10Github Actions
test-conda-python-3.11Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-hdfs-2.9.2Github Actions
test-conda-python-3.7-hdfs-3.2.1Github Actions
test-conda-python-3.7-pandas-0.24Github Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-spark-v3.1.2Github Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-v3.2.0Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-dask-latestGithub Actions
test-conda-python-3.9-dask-upstream_develGithub Actions
test-conda-python-3.9-pandas-upstream_develGithub Actions
test-conda-python-3.9-spark-masterGithub Actions
test-cuda-pythonGithub Actions
test-debian-11-python-3Azure
test-fedora-35-python-3Azure
test-ubuntu-20.04-python-3Azure

@kou

kou commented Nov 16, 2022

Copy link
Copy Markdown
Member

Can we add a test to detect this problem?

@AlenkaF
AlenkaF requested a review from kouNovember 22, 2022 07:39
kou
kou approved these changes Nov 22, 2022
View reviewed changes

@koukou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Comment threadpython/pyarrow/tests/test_cpp_internals.py Outdated
# Arrow C++ and PyArrow C++ header files together

source = os.path.dirname(os.path.abspath(__file__))
pyarrow_dir = os.path.abspath(pjoin(source, '..'))

@koukouNov 22, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

os.path.dirname(source)?

@AlenkaFAlenkaFNov 22, 2022

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to go from test folder to the parent pyarrow folder to test if the include folder is present there.

@koukouNov 22, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry... I'm not sure why os.path.abspath() is needed for it.
Could you show the followings in your environment?

  • The source value
  • ls -lah ${source} (${source} is the source value)
  • The pyarrow_dir value
  • ls -lah ${pyarrow_dir} (${pyarrow_dir} is the pyarrow_dir value)

@AlenkaFAlenkaFNov 22, 2022

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure! Here it is:

(Pdb) print(source)
/Users/alenkafrim/repos/arrow/python/pyarrow/tests
(Pdb) pyarrow_dir
'/Users/alenkafrim/repos/arrow/python/pyarrow'
ls -lah ouput /test
(pyarrow-dev-9) (base) alenkafrim@Alenkas-MacBook-Pro python % ls -lah /Users/alenkafrim/repos/arrow/python/pyarrow/tests
total 3240
drwxr-xr-x 62 alenkafrim staff 1.9K 22 Nov 10:02 .
drwxr-xr-x 109 alenkafrim staff 3.4K 22 Nov 10:02 ..
-rw-r--r-- 1 alenkafrim staff 0B 5 May 2022 __init__.py
drwxr-xr-x 5 alenkafrim staff 160B 22 Nov 10:03 __pycache__
-rw-r--r-- 1 alenkafrim staff 1.3K 2 Jun 13:33 arrow_16597.py
-rw-r--r-- 1 alenkafrim staff 1.1K 3 Nov 2021 arrow_7980.py
-rw-r--r-- 1 alenkafrim staff 2.0K 11 Apr 2022 bound_function_visit_strings.pyx
-rw-r--r-- 1 alenkafrim staff 6.1K 29 Jun 09:38 conftest.py
drwxr-xr-x 5 alenkafrim staff 160B 27 Oct 12:17 data
-rw-r--r-- 1 alenkafrim staff 961B 3 Nov 2021 deserialize_buffer.py
drwxr-xr-x 3 alenkafrim staff 96B 21 Nov 14:54 interchange
-rw-r--r-- 1 alenkafrim staff 5.0K 3 Nov 2021 pandas_examples.py
-rw-r--r-- 1 alenkafrim staff 1.4K 3 Nov 2021 pandas_threaded_import.py
drwxr-xr-x 16 alenkafrim staff 512B 22 Nov 06:02 parquet
-rw-r--r-- 1 alenkafrim staff 1.9K 11 Apr 2022 pyarrow_cython_example.pyx
-rw-r--r-- 1 alenkafrim staff 953B 19 Apr 2022 read_record_batch.py
-rw-r--r-- 1 alenkafrim staff 13K 22 Apr 2022 strategies.py
-rw-r--r-- 1 alenkafrim staff 1.4K 3 Nov 2021 test_adhoc_memory_leak.py
-rw-r--r-- 1 alenkafrim staff 106K 22 Nov 06:02 test_array.py
-rw-r--r-- 1 alenkafrim staff 2.1K 3 Nov 2021 test_builder.py
-rw-r--r-- 1 alenkafrim staff 13K 19 Apr 2022 test_cffi.py
-rw-r--r-- 1 alenkafrim staff 108K 22 Nov 10:02 test_compute.py
-rw-r--r-- 1 alenkafrim staff 70K 27 Oct 14:04 test_convert_builtin.py
-rw-r--r-- 1 alenkafrim staff 1.8K 22 Nov 10:03 test_cpp_internals.py
-rw-r--r-- 1 alenkafrim staff 71K 22 Nov 06:02 test_csv.py
-rw-r--r-- 1 alenkafrim staff 27K 3 Nov 2021 test_cuda.py
-rw-r--r-- 1 alenkafrim staff 8.5K 3 Nov 2021 test_cuda_numba_interop.py
-rw-r--r-- 1 alenkafrim staff 6.5K 22 Nov 06:02 test_cython.py
-rw-r--r-- 1 alenkafrim staff 175K 22 Nov 06:02 test_dataset.py
-rw-r--r-- 1 alenkafrim staff 891B 3 Nov 2021 test_deprecations.py
-rw-r--r-- 1 alenkafrim staff 9.2K 27 Oct 14:04 test_exec_plan.py
-rw-r--r-- 1 alenkafrim staff 33K 22 Nov 06:02 test_extension_type.py
-rw-r--r-- 1 alenkafrim staff 24K 22 Nov 06:02 test_feather.py
-rw-r--r-- 1 alenkafrim staff 2.3K 19 Apr 2022 test_filesystem.py
-rw-r--r-- 1 alenkafrim staff 79K 22 Nov 06:02 test_flight.py
-rw-r--r-- 1 alenkafrim staff 55K 27 Oct 14:04 test_fs.py
-rw-r--r-- 1 alenkafrim staff 15K 27 Oct 14:04 test_gandiva.py
-rw-r--r-- 1 alenkafrim staff 42K 22 Nov 06:02 test_gdb.py
-rw-r--r-- 1 alenkafrim staff 13K 11 Apr 2022 test_hdfs.py
-rw-r--r-- 1 alenkafrim staff 56K 27 Oct 14:04 test_io.py
-rw-r--r-- 1 alenkafrim staff 36K 22 Nov 06:02 test_ipc.py
-rw-r--r-- 1 alenkafrim staff 12K 22 Nov 06:02 test_json.py
-rw-r--r-- 1 alenkafrim staff 15K 3 Nov 2021 test_jvm.py
-rw-r--r-- 1 alenkafrim staff 7.6K 11 Jul 08:19 test_memory.py
-rw-r--r-- 1 alenkafrim staff 5.8K 19 Apr 2022 test_misc.py
-rw-r--r-- 1 alenkafrim staff 19K 22 Nov 10:02 test_orc.py
-rw-r--r-- 1 alenkafrim staff 159K 22 Nov 06:02 test_pandas.py
-rw-r--r-- 1 alenkafrim staff 44K 22 Nov 06:02 test_plasma.py
-rw-r--r-- 1 alenkafrim staff 3.4K 3 Nov 2021 test_plasma_tf_op.py
-rw-r--r-- 1 alenkafrim staff 20K 22 Nov 10:02 test_scalars.py
-rw-r--r-- 1 alenkafrim staff 21K 22 Nov 06:02 test_schema.py
-rw-r--r-- 1 alenkafrim staff 41K 11 Apr 2022 test_serialization.py
-rw-r--r-- 1 alenkafrim staff 1.6K 11 Apr 2022 test_serialization_deprecated.py
-rw-r--r-- 1 alenkafrim staff 17K 3 Nov 2021 test_sparse_tensor.py
-rw-r--r-- 1 alenkafrim staff 1.7K 3 Nov 2021 test_strategies.py
-rw-r--r-- 1 alenkafrim staff 8.4K 22 Nov 06:02 test_substrait.py
-rw-r--r-- 1 alenkafrim staff 65K 27 Oct 14:04 test_table.py
-rw-r--r-- 1 alenkafrim staff 6.1K 11 Apr 2022 test_tensor.py
-rw-r--r-- 1 alenkafrim staff 33K 22 Nov 10:02 test_types.py
-rw-r--r-- 1 alenkafrim staff 17K 29 Jun 09:38 test_udf.py
-rw-r--r-- 1 alenkafrim staff 1.7K 11 Apr 2022 test_util.py
-rw-r--r-- 1 alenkafrim staff 13K 6 May 2022 util.py
ls -lah ouput pyarrow/
(pyarrow-dev-9) (base) alenkafrim@Alenkas-MacBook-Pro python % ls -lah /Users/alenkafrim/repos/arrow/python/pyarrow total 185744
drwxr-xr-x 109 alenkafrim staff 3.4K 22 Nov 10:02 .
drwxr-xr-x 36 alenkafrim staff 1.1K 22 Nov 10:02 ..
-rw-r--r--@ 1 alenkafrim staff 6.0K 17 Nov 12:41 .DS_Store
-rw-r--r-- 1 alenkafrim staff 2.1K 3 Nov 2021 __init__.pxd
-rw-r--r-- 1 alenkafrim staff 19K 22 Nov 08:17 __init__.py
drwxr-xr-x 20 alenkafrim staff 640B 22 Nov 10:03 __pycache__
-rwxr-xr-x 1 alenkafrim staff 3.8M 21 Nov 12:35 _compute.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 1.7K 6 May 2022 _compute.pxd
-rw-r--r-- 1 alenkafrim staff 83K 22 Nov 10:02 _compute.pyx
-rw-r--r-- 1 alenkafrim staff 1.7K 30 May 12:10 _compute_docstrings.py
-rwxr-xr-x 1 alenkafrim staff 970K 21 Nov 12:35 _csv.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 1.6K 11 Apr 2022 _csv.pxd
-rw-r--r-- 1 alenkafrim staff 51K 22 Nov 06:02 _csv.pyx
-rw-r--r-- 1 alenkafrim staff 1.9K 3 Nov 2021 _cuda.pxd
-rw-r--r-- 1 alenkafrim staff 34K 11 Apr 2022 _cuda.pyx
-rwxr-xr-x 1 alenkafrim staff 3.0M 21 Nov 12:35 _dataset.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 4.3K 27 Oct 14:04 _dataset.pxd
-rw-r--r-- 1 alenkafrim staff 100K 22 Nov 06:02 _dataset.pyx
-rw-r--r-- 1 alenkafrim staff 1.3K 11 Apr 2022 _dataset_orc.pyx
-rwxr-xr-x 1 alenkafrim staff 934K 21 Nov 12:35 _dataset_parquet.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 30K 29 Jun 09:38 _dataset_parquet.pyx
-rwxr-xr-x 1 alenkafrim staff 1.0M 21 Nov 12:35 _exec_plan.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 16K 22 Nov 06:02 _exec_plan.pyx
-rwxr-xr-x 1 alenkafrim staff 265K 21 Nov 12:35 _feather.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 3.7K 11 Apr 2022 _feather.pyx
-rw-r--r-- 1 alenkafrim staff 105K 22 Nov 06:02 _flight.pyx
-rwxr-xr-x 1 alenkafrim staff 1.9M 21 Nov 12:35 _fs.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 2.4K 3 Nov 2021 _fs.pxd
-rw-r--r-- 1 alenkafrim staff 51K 22 Nov 06:02 _fs.pyx
-rw-r--r-- 1 alenkafrim staff 8.0K 29 Jun 09:38 _gcsfs.pyx
-rw-r--r-- 1 alenkafrim staff 186B 21 Nov 12:35 _generated_version.py
-rw-r--r-- 1 alenkafrim staff 5.6K 29 Jun 09:38 _hdfs.pyx
-rwxr-xr-x 1 alenkafrim staff 483K 21 Nov 12:35 _hdfsio.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 13K 11 Apr 2022 _hdfsio.pyx
-rwxr-xr-x 1 alenkafrim staff 199K 21 Nov 12:35 _json.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 8.5K 19 Apr 2022 _json.pyx
-rw-r--r-- 1 alenkafrim staff 5.6K 11 Apr 2022 _orc.pxd
-rw-r--r-- 1 alenkafrim staff 15K 11 Apr 2022 _orc.pyx
-rwxr-xr-x 1 alenkafrim staff 1.6M 21 Nov 12:35 _parquet.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 25K 29 Jun 09:38 _parquet.pxd
-rw-r--r-- 1 alenkafrim staff 57K 27 Oct 14:04 _parquet.pyx
-rw-r--r-- 1 alenkafrim staff 5.9K 19 Apr 2022 _parquet_encryption.pxd
-rw-r--r-- 1 alenkafrim staff 17K 19 Apr 2022 _parquet_encryption.pyx
-rw-r--r-- 1 alenkafrim staff 30K 22 Nov 06:02 _plasma.pyx
-rwxr-xr-x 1 alenkafrim staff 207K 21 Nov 12:35 _pyarrow_cpp_tests.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 1.2K 22 Nov 06:02 _pyarrow_cpp_tests.pxd
-rw-r--r-- 1 alenkafrim staff 1.7K 22 Nov 06:02 _pyarrow_cpp_tests.pyx
-rwxr-xr-x 1 alenkafrim staff 388K 21 Nov 12:35 _s3fs.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 16K 22 Nov 06:02 _s3fs.pyx
-rw-r--r-- 1 alenkafrim staff 6.3K 22 Nov 06:02 _substrait.pyx
-rw-r--r-- 1 alenkafrim staff 91K 22 Nov 06:02 array.pxi
-rw-r--r-- 1 alenkafrim staff 869B 3 Nov 2021 benchmark.pxi
-rw-r--r-- 1 alenkafrim staff 856B 3 Nov 2021 benchmark.py
-rw-r--r-- 1 alenkafrim staff 2.6K 3 Nov 2021 builder.pxi
-rw-r--r-- 1 alenkafrim staff 2.1K 3 Nov 2021 cffi.py
-rw-r--r-- 1 alenkafrim staff 2.0K 11 Apr 2022 compat.pxi
-rw-r--r-- 1 alenkafrim staff 21K 22 Nov 10:02 compute.py
-rw-r--r-- 1 alenkafrim staff 2.7K 11 Apr 2022 config.pxi
-rw-r--r-- 1 alenkafrim staff 6.2K 27 Oct 14:04 conftest.py
-rw-r--r-- 1 alenkafrim staff 974B 11 Apr 2022 csv.py
-rw-r--r-- 1 alenkafrim staff 1.1K 3 Nov 2021 cuda.py
-rw-r--r-- 1 alenkafrim staff 38K 27 Oct 14:04 dataset.py
-rw-r--r-- 1 alenkafrim staff 8.3K 22 Nov 06:02 error.pxi
-rw-r--r-- 1 alenkafrim staff 9.9K 22 Nov 06:02 feather.py
-rw-r--r-- 1 alenkafrim staff 14K 10 Jun 10:00 filesystem.py
-rw-r--r-- 1 alenkafrim staff 1.8K 22 Nov 06:02 flight.py
-rw-r--r-- 1 alenkafrim staff 14K 27 Oct 14:04 fs.py
-rw-r--r-- 1 alenkafrim staff 20K 27 Oct 14:04 gandiva.pyx
-rw-r--r-- 1 alenkafrim staff 7.4K 11 Apr 2022 hdfs.py
drwxr-xr-x 3 alenkafrim staff 96B 17 Nov 11:37 include
drwxr-xr-x 15 alenkafrim staff 480B 22 Nov 10:02 includes
drwxr-xr-x 3 alenkafrim staff 96B 21 Nov 14:54 interchange
-rw-r--r-- 1 alenkafrim staff 68K 27 Oct 14:04 io.pxi
-rw-r--r-- 1 alenkafrim staff 33K 22 Nov 06:02 ipc.pxi
-rw-r--r-- 1 alenkafrim staff 9.7K 22 Nov 06:02 ipc.py
-rw-r--r-- 1 alenkafrim staff 858B 3 Nov 2021 json.py
-rw-r--r-- 1 alenkafrim staff 9.4K 3 Nov 2021 jvm.py
-rwxr-xr-x 1 alenkafrim staff 7.0M 21 Nov 12:35 lib.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 14K 22 Nov 10:02 lib.pxd
-rw-r--r-- 1 alenkafrim staff 4.6K 21 Apr 2022 lib.pyx
-rw-r--r-- 1 alenkafrim staff 19K 21 Nov 12:35 lib_api.h
-rwxr-xr-x 1 alenkafrim staff 12M 21 Nov 12:35 libarrow_python.1100.0.0.dylib
-rwxr-xr-x 1 alenkafrim staff 12M 21 Nov 12:35 libarrow_python.1100.dylib
-rw-r--r-- 1 alenkafrim staff 30M 21 Nov 12:35 libarrow_python.a
-rwxr-xr-x 1 alenkafrim staff 12M 21 Nov 12:35 libarrow_python.dylib
-rwxr-xr-x 1 alenkafrim staff 458K 21 Nov 12:35 libarrow_python_flight.1100.0.0.dylib
-rwxr-xr-x 1 alenkafrim staff 458K 21 Nov 12:35 libarrow_python_flight.1100.dylib
-rw-r--r-- 1 alenkafrim staff 1.2M 21 Nov 12:35 libarrow_python_flight.a
-rwxr-xr-x 1 alenkafrim staff 458K 21 Nov 12:35 libarrow_python_flight.dylib
-rw-r--r-- 1 alenkafrim staff 8.0K 27 Oct 14:04 memory.pxi
-rw-r--r-- 1 alenkafrim staff 12K 22 Nov 06:02 orc.py
-rw-r--r-- 1 alenkafrim staff 7.8K 17 Nov 10:51 pandas-shim.pxi
-rw-r--r-- 1 alenkafrim staff 43K 22 Nov 06:02 pandas_compat.py
drwxr-xr-x 6 alenkafrim staff 192B 22 Nov 06:02 parquet
-rw-r--r-- 1 alenkafrim staff 6.3K 22 Nov 06:02 plasma.py
-rw-r--r-- 1 alenkafrim staff 13K 22 Nov 10:02 public-api.pxi
-rw-r--r-- 1 alenkafrim staff 29K 22 Nov 10:02 scalar.pxi
-rw-r--r-- 1 alenkafrim staff 19K 29 Jun 09:38 serialization.pxi
-rw-r--r-- 1 alenkafrim staff 18K 3 Nov 2021 serialization.py
drwxr-xr-x 8 alenkafrim staff 256B 22 Nov 10:02 src
-rw-r--r-- 1 alenkafrim staff 873B 27 Oct 14:04 substrait.py
-rw-r--r-- 1 alenkafrim staff 158K 22 Nov 06:02 table.pxi
-rw-r--r-- 1 alenkafrim staff 38K 19 Apr 2022 tensor.pxi
drwxr-xr-x 3 alenkafrim staff 96B 27 Oct 14:04 tensorflow
drwxr-xr-x 62 alenkafrim staff 1.9K 22 Nov 10:02 tests
-rw-r--r-- 1 alenkafrim staff 90K 22 Nov 06:02 types.pxi
-rw-r--r-- 1 alenkafrim staff 10K 11 Apr 2022 types.py
-rw-r--r-- 1 alenkafrim staff 4.6K 22 Nov 06:02 util.py
drwxr-xr-x 6 alenkafrim staff 192B 17 Nov 11:07 vendored

If I remember correctly, if I didn't use os.path.abspath() the .. didn't have an effect.

@jorisvandenbosschejorisvandenbosscheNov 22, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For me it seems to work also with a ".." in it:

In [9]: os.path.exists("/home/joris/scipy/repos/arrow/python/pyarrow/tests/../include/arrow/python/")
Out[9]: True

@jorisvandenbosschejorisvandenbosscheNov 22, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But so about @kou's original comment, I think doing os.path.dirname(source) also works because if source is already a directory, dirname gives the parent directory.
Now personally I find .. more explicit to read that it goes one level up.

@AlenkaFAlenkaFNov 22, 2022

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, sorry for my confusion.
Can change to pyarrow_dir = os.path.dirname(source), @kou do you agree?

@AlenkaFAlenkaFNov 22, 2022

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, now I see Joris you already made a suggestion. Will commit that 👍

@kou

kou commented Nov 22, 2022

Copy link
Copy Markdown
Member

@github-actions crossbow submit -g python -g conda

@github-actions

github-actionsBot commented Nov 22, 2022

Copy link
Copy Markdown

Revision: acf8754

Submitted crossbow builds: ursacomputing/crossbow @ actions-d4595c1625

TaskStatus
conda-cleanAzure
conda-linux-gcc-py310-arm64Azure
conda-linux-gcc-py310-cpuAzure
conda-linux-gcc-py310-cudaAzure
conda-linux-gcc-py310-ppc64leAzure
conda-linux-gcc-py37-arm64Azure
conda-linux-gcc-py37-cpu-r40Azure
conda-linux-gcc-py37-cpu-r41Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py37-ppc64leAzure
conda-linux-gcc-py38-arm64Azure
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py38-ppc64leAzure
conda-linux-gcc-py39-arm64Azure
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-linux-gcc-py39-ppc64leAzure
conda-osx-arm64-clang-py310Azure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py310Azure
conda-osx-clang-py37-r40Azure
conda-osx-clang-py37-r41Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2019-py310Azure
conda-win-vs2019-py37-r40Azure
conda-win-vs2019-py37-r41Azure
conda-win-vs2019-py38Azure
conda-win-vs2019-py39Azure
test-conda-python-3.10Github Actions
test-conda-python-3.11Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-hdfs-2.9.2Github Actions
test-conda-python-3.7-hdfs-3.2.1Github Actions
test-conda-python-3.7-pandas-0.24Github Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-spark-v3.1.2Github Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-v3.2.0Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-dask-latestGithub Actions
test-conda-python-3.9-dask-upstream_develGithub Actions
test-conda-python-3.9-pandas-upstream_develGithub Actions
test-conda-python-3.9-spark-masterGithub Actions
test-cuda-pythonGithub Actions
test-debian-11-python-3Azure
test-fedora-35-python-3Azure
test-ubuntu-20.04-python-3Azure

jorisvandenbossche
jorisvandenbossche reviewed Nov 22, 2022
View reviewed changes
Comment threadpython/pyarrow/tests/test_cpp_internals.py Outdated
@AlenkaF@jorisvandenbossche
Update python/pyarrow/tests/test_cpp_internals.py
4669e94
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
@jorisvandenbosschejorisvandenbossche mentioned this pull request Nov 22, 2022
Merged
3 tasks
@kou
kou merged commit 1e9eb61 into apache:masterNov 22, 2022
@AlenkaF
AlenkaF deleted the ARROW-18340 branch November 23, 2022 08:41
@ursabot

ursabot commented Nov 24, 2022

Copy link
Copy Markdown

Benchmark runs are scheduled for baseline = 59f99d2 and contender = 1e9eb61. 1e9eb61 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.34% ⬆️0.54%] test-mac-arm
[Finished ⬇️0.27% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.14% ⬆️0.35%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 1e9eb615 ec2-t3-xlarge-us-east-2
[Finished] 1e9eb615 test-mac-arm
[Finished] 1e9eb615 ursa-i9-9960x
[Finished] 1e9eb615 ursa-thinkcentre-m75q
[Finished] 59f99d2c ec2-t3-xlarge-us-east-2
[Finished] 59f99d2c test-mac-arm
[Finished] 59f99d2c ursa-i9-9960x
[Finished] 59f99d2c ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

@asfimportasfimport mentioned this pull request Nov 24, 2022
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@koukoukou approved these changes
@jorisvandenbosschejorisvandenbosschejorisvandenbossche left review comments

Assignees

No one assigned

Labels

Component: Python

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@AlenkaF@jorisvandenbossche@kou@ursabot
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
ARROW-18340: [Python] PyArrow C++ header files no longer always included in installed pyarrow - #14656
Merged
kou merged 7 commits into
apache:masterfrom 
AlenkaF:ARROW-18340
Nov 22, 2022
Merged
ARROW-18340: [Python] PyArrow C++ header files no longer always included in installed pyarrow#14656
kou merged 7 commits into
apache:masterfrom 
AlenkaF:ARROW-18340
Conversation
@AlenkaF
@AlenkaFAlenkaF
commented
Nov 16, 2022

Copy link


Copy Markdown

Member
This PR adds code to _run_cmake to always copy PyArrow C++ header files to pyarrow/iclude.
@github-actions
github-actionsBot
commented
Nov 16, 2022

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-18340
@github-actions
github-actionsBot
commented
Nov 16, 2022

Copy link


Copy Markdown

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.
jorisvandenbossche
jorisvandenbossche
reviewed
Nov 16, 2022
View reviewed changes
Comment threadpython/setup.py
Outdated

@jorisvandenbossche
jorisvandenbossche
commented
Nov 16, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g wheel
@github-actions
github-actionsBot
commented
Nov 16, 2022

Copy link


Copy Markdown

Revision: d5f953f
Submitted crossbow builds: ursacomputing/crossbow @ actions-1c3aa4b08e
TaskStatus
wheel-macos-big-sur-cp310-arm64Github Actions
wheel-macos-big-sur-cp311-arm64Github Actions
wheel-macos-big-sur-cp38-arm64Github Actions
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-mojave-cp310-amd64Github Actions
wheel-macos-mojave-cp311-amd64Github Actions
wheel-macos-mojave-cp37-amd64Github Actions
wheel-macos-mojave-cp38-amd64Github Actions
wheel-macos-mojave-cp39-amd64Github Actions
wheel-manylinux2014-cp310-amd64Github Actions
wheel-manylinux2014-cp310-arm64Travis CI
wheel-manylinux2014-cp311-amd64Github Actions
wheel-manylinux2014-cp311-arm64Travis CI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64Travis CI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64Travis CI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64Travis CI
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
kou
kou
reviewed
Nov 16, 2022
View reviewed changes
Comment threadpython/setup.py
Outdated

@kou
kou
commented
Nov 16, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g python -g conda
@github-actions
github-actionsBot
commented
Nov 16, 2022

Copy link


Copy Markdown

Revision: d5f953f
Submitted crossbow builds: ursacomputing/crossbow @ actions-9eb5a3c4ca
TaskStatus
conda-cleanAzure
conda-linux-gcc-py310-arm64Azure
conda-linux-gcc-py310-cpuAzure
conda-linux-gcc-py310-cudaAzure
conda-linux-gcc-py310-ppc64leAzure
conda-linux-gcc-py37-arm64Azure
conda-linux-gcc-py37-cpu-r40Azure
conda-linux-gcc-py37-cpu-r41Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py37-ppc64leAzure
conda-linux-gcc-py38-arm64Azure
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py38-ppc64leAzure
conda-linux-gcc-py39-arm64Azure
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-linux-gcc-py39-ppc64leAzure
conda-osx-arm64-clang-py310Azure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py310Azure
conda-osx-clang-py37-r40Azure
conda-osx-clang-py37-r41Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2019-py310Azure
conda-win-vs2019-py37-r40Azure
conda-win-vs2019-py37-r41Azure
conda-win-vs2019-py38Azure
conda-win-vs2019-py39Azure
test-conda-python-3.10Github Actions
test-conda-python-3.11Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-hdfs-2.9.2Github Actions
test-conda-python-3.7-hdfs-3.2.1Github Actions
test-conda-python-3.7-pandas-0.24Github Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-spark-v3.1.2Github Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-v3.2.0Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-dask-latestGithub Actions
test-conda-python-3.9-dask-upstream_develGithub Actions
test-conda-python-3.9-pandas-upstream_develGithub Actions
test-conda-python-3.9-spark-masterGithub Actions
test-cuda-pythonGithub Actions
test-debian-11-python-3Azure
test-fedora-35-python-3Azure
test-ubuntu-20.04-python-3Azure
@kou
kou
commented
Nov 16, 2022

Copy link


Copy Markdown

Member
Can we add a test to detect this problem?
@AlenkaF
AlenkaF
requested a review
from kouNovember 22, 2022 07:39
kou
kou
approved these changes
Nov 22, 2022
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Comment threadpython/pyarrow/tests/test_cpp_internals.py
Outdated

 # Arrow C++ and PyArrow C++ header files together

 source = os.path.dirname(os.path.abspath(__file__))
 pyarrow_dir = os.path.abspath(pjoin(source, '..'))
@koukouNov 22, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

os.path.dirname(source)?
@AlenkaFAlenkaFNov 22, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to go from test folder to the parent pyarrow folder to test if the include folder is present there.
@koukouNov 22, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry... I'm not sure why os.path.abspath() is needed for it.

Could you show the followings in your environment?
  • The source value
  • ls -lah ${source} (${source} is the source value)
  • The pyarrow_dir value
  • ls -lah ${pyarrow_dir} (${pyarrow_dir} is the pyarrow_dir value)
@AlenkaFAlenkaFNov 22, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure! Here it is:
(Pdb) print(source)
/Users/alenkafrim/repos/arrow/python/pyarrow/tests
(Pdb) pyarrow_dir
'/Users/alenkafrim/repos/arrow/python/pyarrow'

ls -lah ouput /test
(pyarrow-dev-9) (base) alenkafrim@Alenkas-MacBook-Pro python % ls -lah /Users/alenkafrim/repos/arrow/python/pyarrow/tests
total 3240
drwxr-xr-x 62 alenkafrim staff 1.9K 22 Nov 10:02 .
drwxr-xr-x 109 alenkafrim staff 3.4K 22 Nov 10:02 ..
-rw-r--r-- 1 alenkafrim staff 0B 5 May 2022 __init__.py
drwxr-xr-x 5 alenkafrim staff 160B 22 Nov 10:03 __pycache__
-rw-r--r-- 1 alenkafrim staff 1.3K 2 Jun 13:33 arrow_16597.py
-rw-r--r-- 1 alenkafrim staff 1.1K 3 Nov 2021 arrow_7980.py
-rw-r--r-- 1 alenkafrim staff 2.0K 11 Apr 2022 bound_function_visit_strings.pyx
-rw-r--r-- 1 alenkafrim staff 6.1K 29 Jun 09:38 conftest.py
drwxr-xr-x 5 alenkafrim staff 160B 27 Oct 12:17 data
-rw-r--r-- 1 alenkafrim staff 961B 3 Nov 2021 deserialize_buffer.py
drwxr-xr-x 3 alenkafrim staff 96B 21 Nov 14:54 interchange
-rw-r--r-- 1 alenkafrim staff 5.0K 3 Nov 2021 pandas_examples.py
-rw-r--r-- 1 alenkafrim staff 1.4K 3 Nov 2021 pandas_threaded_import.py
drwxr-xr-x 16 alenkafrim staff 512B 22 Nov 06:02 parquet
-rw-r--r-- 1 alenkafrim staff 1.9K 11 Apr 2022 pyarrow_cython_example.pyx
-rw-r--r-- 1 alenkafrim staff 953B 19 Apr 2022 read_record_batch.py
-rw-r--r-- 1 alenkafrim staff 13K 22 Apr 2022 strategies.py
-rw-r--r-- 1 alenkafrim staff 1.4K 3 Nov 2021 test_adhoc_memory_leak.py
-rw-r--r-- 1 alenkafrim staff 106K 22 Nov 06:02 test_array.py
-rw-r--r-- 1 alenkafrim staff 2.1K 3 Nov 2021 test_builder.py
-rw-r--r-- 1 alenkafrim staff 13K 19 Apr 2022 test_cffi.py
-rw-r--r-- 1 alenkafrim staff 108K 22 Nov 10:02 test_compute.py
-rw-r--r-- 1 alenkafrim staff 70K 27 Oct 14:04 test_convert_builtin.py
-rw-r--r-- 1 alenkafrim staff 1.8K 22 Nov 10:03 test_cpp_internals.py
-rw-r--r-- 1 alenkafrim staff 71K 22 Nov 06:02 test_csv.py
-rw-r--r-- 1 alenkafrim staff 27K 3 Nov 2021 test_cuda.py
-rw-r--r-- 1 alenkafrim staff 8.5K 3 Nov 2021 test_cuda_numba_interop.py
-rw-r--r-- 1 alenkafrim staff 6.5K 22 Nov 06:02 test_cython.py
-rw-r--r-- 1 alenkafrim staff 175K 22 Nov 06:02 test_dataset.py
-rw-r--r-- 1 alenkafrim staff 891B 3 Nov 2021 test_deprecations.py
-rw-r--r-- 1 alenkafrim staff 9.2K 27 Oct 14:04 test_exec_plan.py
-rw-r--r-- 1 alenkafrim staff 33K 22 Nov 06:02 test_extension_type.py
-rw-r--r-- 1 alenkafrim staff 24K 22 Nov 06:02 test_feather.py
-rw-r--r-- 1 alenkafrim staff 2.3K 19 Apr 2022 test_filesystem.py
-rw-r--r-- 1 alenkafrim staff 79K 22 Nov 06:02 test_flight.py
-rw-r--r-- 1 alenkafrim staff 55K 27 Oct 14:04 test_fs.py
-rw-r--r-- 1 alenkafrim staff 15K 27 Oct 14:04 test_gandiva.py
-rw-r--r-- 1 alenkafrim staff 42K 22 Nov 06:02 test_gdb.py
-rw-r--r-- 1 alenkafrim staff 13K 11 Apr 2022 test_hdfs.py
-rw-r--r-- 1 alenkafrim staff 56K 27 Oct 14:04 test_io.py
-rw-r--r-- 1 alenkafrim staff 36K 22 Nov 06:02 test_ipc.py
-rw-r--r-- 1 alenkafrim staff 12K 22 Nov 06:02 test_json.py
-rw-r--r-- 1 alenkafrim staff 15K 3 Nov 2021 test_jvm.py
-rw-r--r-- 1 alenkafrim staff 7.6K 11 Jul 08:19 test_memory.py
-rw-r--r-- 1 alenkafrim staff 5.8K 19 Apr 2022 test_misc.py
-rw-r--r-- 1 alenkafrim staff 19K 22 Nov 10:02 test_orc.py
-rw-r--r-- 1 alenkafrim staff 159K 22 Nov 06:02 test_pandas.py
-rw-r--r-- 1 alenkafrim staff 44K 22 Nov 06:02 test_plasma.py
-rw-r--r-- 1 alenkafrim staff 3.4K 3 Nov 2021 test_plasma_tf_op.py
-rw-r--r-- 1 alenkafrim staff 20K 22 Nov 10:02 test_scalars.py
-rw-r--r-- 1 alenkafrim staff 21K 22 Nov 06:02 test_schema.py
-rw-r--r-- 1 alenkafrim staff 41K 11 Apr 2022 test_serialization.py
-rw-r--r-- 1 alenkafrim staff 1.6K 11 Apr 2022 test_serialization_deprecated.py
-rw-r--r-- 1 alenkafrim staff 17K 3 Nov 2021 test_sparse_tensor.py
-rw-r--r-- 1 alenkafrim staff 1.7K 3 Nov 2021 test_strategies.py
-rw-r--r-- 1 alenkafrim staff 8.4K 22 Nov 06:02 test_substrait.py
-rw-r--r-- 1 alenkafrim staff 65K 27 Oct 14:04 test_table.py
-rw-r--r-- 1 alenkafrim staff 6.1K 11 Apr 2022 test_tensor.py
-rw-r--r-- 1 alenkafrim staff 33K 22 Nov 10:02 test_types.py
-rw-r--r-- 1 alenkafrim staff 17K 29 Jun 09:38 test_udf.py
-rw-r--r-- 1 alenkafrim staff 1.7K 11 Apr 2022 test_util.py
-rw-r--r-- 1 alenkafrim staff 13K 6 May 2022 util.py

ls -lah ouput pyarrow/
(pyarrow-dev-9) (base) alenkafrim@Alenkas-MacBook-Pro python % ls -lah /Users/alenkafrim/repos/arrow/python/pyarrow total 185744
drwxr-xr-x 109 alenkafrim staff 3.4K 22 Nov 10:02 .
drwxr-xr-x 36 alenkafrim staff 1.1K 22 Nov 10:02 ..
-rw-r--r--@ 1 alenkafrim staff 6.0K 17 Nov 12:41 .DS_Store
-rw-r--r-- 1 alenkafrim staff 2.1K 3 Nov 2021 __init__.pxd
-rw-r--r-- 1 alenkafrim staff 19K 22 Nov 08:17 __init__.py
drwxr-xr-x 20 alenkafrim staff 640B 22 Nov 10:03 __pycache__
-rwxr-xr-x 1 alenkafrim staff 3.8M 21 Nov 12:35 _compute.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 1.7K 6 May 2022 _compute.pxd
-rw-r--r-- 1 alenkafrim staff 83K 22 Nov 10:02 _compute.pyx
-rw-r--r-- 1 alenkafrim staff 1.7K 30 May 12:10 _compute_docstrings.py
-rwxr-xr-x 1 alenkafrim staff 970K 21 Nov 12:35 _csv.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 1.6K 11 Apr 2022 _csv.pxd
-rw-r--r-- 1 alenkafrim staff 51K 22 Nov 06:02 _csv.pyx
-rw-r--r-- 1 alenkafrim staff 1.9K 3 Nov 2021 _cuda.pxd
-rw-r--r-- 1 alenkafrim staff 34K 11 Apr 2022 _cuda.pyx
-rwxr-xr-x 1 alenkafrim staff 3.0M 21 Nov 12:35 _dataset.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 4.3K 27 Oct 14:04 _dataset.pxd
-rw-r--r-- 1 alenkafrim staff 100K 22 Nov 06:02 _dataset.pyx
-rw-r--r-- 1 alenkafrim staff 1.3K 11 Apr 2022 _dataset_orc.pyx
-rwxr-xr-x 1 alenkafrim staff 934K 21 Nov 12:35 _dataset_parquet.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 30K 29 Jun 09:38 _dataset_parquet.pyx
-rwxr-xr-x 1 alenkafrim staff 1.0M 21 Nov 12:35 _exec_plan.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 16K 22 Nov 06:02 _exec_plan.pyx
-rwxr-xr-x 1 alenkafrim staff 265K 21 Nov 12:35 _feather.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 3.7K 11 Apr 2022 _feather.pyx
-rw-r--r-- 1 alenkafrim staff 105K 22 Nov 06:02 _flight.pyx
-rwxr-xr-x 1 alenkafrim staff 1.9M 21 Nov 12:35 _fs.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 2.4K 3 Nov 2021 _fs.pxd
-rw-r--r-- 1 alenkafrim staff 51K 22 Nov 06:02 _fs.pyx
-rw-r--r-- 1 alenkafrim staff 8.0K 29 Jun 09:38 _gcsfs.pyx
-rw-r--r-- 1 alenkafrim staff 186B 21 Nov 12:35 _generated_version.py
-rw-r--r-- 1 alenkafrim staff 5.6K 29 Jun 09:38 _hdfs.pyx
-rwxr-xr-x 1 alenkafrim staff 483K 21 Nov 12:35 _hdfsio.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 13K 11 Apr 2022 _hdfsio.pyx
-rwxr-xr-x 1 alenkafrim staff 199K 21 Nov 12:35 _json.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 8.5K 19 Apr 2022 _json.pyx
-rw-r--r-- 1 alenkafrim staff 5.6K 11 Apr 2022 _orc.pxd
-rw-r--r-- 1 alenkafrim staff 15K 11 Apr 2022 _orc.pyx
-rwxr-xr-x 1 alenkafrim staff 1.6M 21 Nov 12:35 _parquet.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 25K 29 Jun 09:38 _parquet.pxd
-rw-r--r-- 1 alenkafrim staff 57K 27 Oct 14:04 _parquet.pyx
-rw-r--r-- 1 alenkafrim staff 5.9K 19 Apr 2022 _parquet_encryption.pxd
-rw-r--r-- 1 alenkafrim staff 17K 19 Apr 2022 _parquet_encryption.pyx
-rw-r--r-- 1 alenkafrim staff 30K 22 Nov 06:02 _plasma.pyx
-rwxr-xr-x 1 alenkafrim staff 207K 21 Nov 12:35 _pyarrow_cpp_tests.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 1.2K 22 Nov 06:02 _pyarrow_cpp_tests.pxd
-rw-r--r-- 1 alenkafrim staff 1.7K 22 Nov 06:02 _pyarrow_cpp_tests.pyx
-rwxr-xr-x 1 alenkafrim staff 388K 21 Nov 12:35 _s3fs.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 16K 22 Nov 06:02 _s3fs.pyx
-rw-r--r-- 1 alenkafrim staff 6.3K 22 Nov 06:02 _substrait.pyx
-rw-r--r-- 1 alenkafrim staff 91K 22 Nov 06:02 array.pxi
-rw-r--r-- 1 alenkafrim staff 869B 3 Nov 2021 benchmark.pxi
-rw-r--r-- 1 alenkafrim staff 856B 3 Nov 2021 benchmark.py
-rw-r--r-- 1 alenkafrim staff 2.6K 3 Nov 2021 builder.pxi
-rw-r--r-- 1 alenkafrim staff 2.1K 3 Nov 2021 cffi.py
-rw-r--r-- 1 alenkafrim staff 2.0K 11 Apr 2022 compat.pxi
-rw-r--r-- 1 alenkafrim staff 21K 22 Nov 10:02 compute.py
-rw-r--r-- 1 alenkafrim staff 2.7K 11 Apr 2022 config.pxi
-rw-r--r-- 1 alenkafrim staff 6.2K 27 Oct 14:04 conftest.py
-rw-r--r-- 1 alenkafrim staff 974B 11 Apr 2022 csv.py
-rw-r--r-- 1 alenkafrim staff 1.1K 3 Nov 2021 cuda.py
-rw-r--r-- 1 alenkafrim staff 38K 27 Oct 14:04 dataset.py
-rw-r--r-- 1 alenkafrim staff 8.3K 22 Nov 06:02 error.pxi
-rw-r--r-- 1 alenkafrim staff 9.9K 22 Nov 06:02 feather.py
-rw-r--r-- 1 alenkafrim staff 14K 10 Jun 10:00 filesystem.py
-rw-r--r-- 1 alenkafrim staff 1.8K 22 Nov 06:02 flight.py
-rw-r--r-- 1 alenkafrim staff 14K 27 Oct 14:04 fs.py
-rw-r--r-- 1 alenkafrim staff 20K 27 Oct 14:04 gandiva.pyx
-rw-r--r-- 1 alenkafrim staff 7.4K 11 Apr 2022 hdfs.py
drwxr-xr-x 3 alenkafrim staff 96B 17 Nov 11:37 include
drwxr-xr-x 15 alenkafrim staff 480B 22 Nov 10:02 includes
drwxr-xr-x 3 alenkafrim staff 96B 21 Nov 14:54 interchange
-rw-r--r-- 1 alenkafrim staff 68K 27 Oct 14:04 io.pxi
-rw-r--r-- 1 alenkafrim staff 33K 22 Nov 06:02 ipc.pxi
-rw-r--r-- 1 alenkafrim staff 9.7K 22 Nov 06:02 ipc.py
-rw-r--r-- 1 alenkafrim staff 858B 3 Nov 2021 json.py
-rw-r--r-- 1 alenkafrim staff 9.4K 3 Nov 2021 jvm.py
-rwxr-xr-x 1 alenkafrim staff 7.0M 21 Nov 12:35 lib.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 14K 22 Nov 10:02 lib.pxd
-rw-r--r-- 1 alenkafrim staff 4.6K 21 Apr 2022 lib.pyx
-rw-r--r-- 1 alenkafrim staff 19K 21 Nov 12:35 lib_api.h
-rwxr-xr-x 1 alenkafrim staff 12M 21 Nov 12:35 libarrow_python.1100.0.0.dylib
-rwxr-xr-x 1 alenkafrim staff 12M 21 Nov 12:35 libarrow_python.1100.dylib
-rw-r--r-- 1 alenkafrim staff 30M 21 Nov 12:35 libarrow_python.a
-rwxr-xr-x 1 alenkafrim staff 12M 21 Nov 12:35 libarrow_python.dylib
-rwxr-xr-x 1 alenkafrim staff 458K 21 Nov 12:35 libarrow_python_flight.1100.0.0.dylib
-rwxr-xr-x 1 alenkafrim staff 458K 21 Nov 12:35 libarrow_python_flight.1100.dylib
-rw-r--r-- 1 alenkafrim staff 1.2M 21 Nov 12:35 libarrow_python_flight.a
-rwxr-xr-x 1 alenkafrim staff 458K 21 Nov 12:35 libarrow_python_flight.dylib
-rw-r--r-- 1 alenkafrim staff 8.0K 27 Oct 14:04 memory.pxi
-rw-r--r-- 1 alenkafrim staff 12K 22 Nov 06:02 orc.py
-rw-r--r-- 1 alenkafrim staff 7.8K 17 Nov 10:51 pandas-shim.pxi
-rw-r--r-- 1 alenkafrim staff 43K 22 Nov 06:02 pandas_compat.py
drwxr-xr-x 6 alenkafrim staff 192B 22 Nov 06:02 parquet
-rw-r--r-- 1 alenkafrim staff 6.3K 22 Nov 06:02 plasma.py
-rw-r--r-- 1 alenkafrim staff 13K 22 Nov 10:02 public-api.pxi
-rw-r--r-- 1 alenkafrim staff 29K 22 Nov 10:02 scalar.pxi
-rw-r--r-- 1 alenkafrim staff 19K 29 Jun 09:38 serialization.pxi
-rw-r--r-- 1 alenkafrim staff 18K 3 Nov 2021 serialization.py
drwxr-xr-x 8 alenkafrim staff 256B 22 Nov 10:02 src
-rw-r--r-- 1 alenkafrim staff 873B 27 Oct 14:04 substrait.py
-rw-r--r-- 1 alenkafrim staff 158K 22 Nov 06:02 table.pxi
-rw-r--r-- 1 alenkafrim staff 38K 19 Apr 2022 tensor.pxi
drwxr-xr-x 3 alenkafrim staff 96B 27 Oct 14:04 tensorflow
drwxr-xr-x 62 alenkafrim staff 1.9K 22 Nov 10:02 tests
-rw-r--r-- 1 alenkafrim staff 90K 22 Nov 06:02 types.pxi
-rw-r--r-- 1 alenkafrim staff 10K 11 Apr 2022 types.py
-rw-r--r-- 1 alenkafrim staff 4.6K 22 Nov 06:02 util.py
drwxr-xr-x 6 alenkafrim staff 192B 17 Nov 11:07 vendored

If I remember correctly, if I didn't use os.path.abspath() the .. didn't have an effect.
@jorisvandenbosschejorisvandenbosscheNov 22, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For me it seems to work also with a ".." in it:
In [9]: os.path.exists("/home/joris/scipy/repos/arrow/python/pyarrow/tests/../include/arrow/python/")
Out[9]: True

@jorisvandenbosschejorisvandenbosscheNov 22, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But so about @kou's original comment, I think doing os.path.dirname(source) also works because if source is already a directory, dirname gives the parent directory.

Now personally I find .. more explicit to read that it goes one level up.
@AlenkaFAlenkaFNov 22, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, sorry for my confusion.

Can change to pyarrow_dir = os.path.dirname(source), @kou do you agree?
@AlenkaFAlenkaFNov 22, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, now I see Joris you already made a suggestion. Will commit that 👍
@kou
kou
commented
Nov 22, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g python -g conda
@github-actions
github-actionsBot
commented
Nov 22, 2022

Copy link


Copy Markdown

Revision: acf8754
Submitted crossbow builds: ursacomputing/crossbow @ actions-d4595c1625
TaskStatus
conda-cleanAzure
conda-linux-gcc-py310-arm64Azure
conda-linux-gcc-py310-cpuAzure
conda-linux-gcc-py310-cudaAzure
conda-linux-gcc-py310-ppc64leAzure
conda-linux-gcc-py37-arm64Azure
conda-linux-gcc-py37-cpu-r40Azure
conda-linux-gcc-py37-cpu-r41Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py37-ppc64leAzure
conda-linux-gcc-py38-arm64Azure
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py38-ppc64leAzure
conda-linux-gcc-py39-arm64Azure
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-linux-gcc-py39-ppc64leAzure
conda-osx-arm64-clang-py310Azure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py310Azure
conda-osx-clang-py37-r40Azure
conda-osx-clang-py37-r41Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2019-py310Azure
conda-win-vs2019-py37-r40Azure
conda-win-vs2019-py37-r41Azure
conda-win-vs2019-py38Azure
conda-win-vs2019-py39Azure
test-conda-python-3.10Github Actions
test-conda-python-3.11Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-hdfs-2.9.2Github Actions
test-conda-python-3.7-hdfs-3.2.1Github Actions
test-conda-python-3.7-pandas-0.24Github Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-spark-v3.1.2Github Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-v3.2.0Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-dask-latestGithub Actions
test-conda-python-3.9-dask-upstream_develGithub Actions
test-conda-python-3.9-pandas-upstream_develGithub Actions
test-conda-python-3.9-spark-masterGithub Actions
test-cuda-pythonGithub Actions
test-debian-11-python-3Azure
test-fedora-35-python-3Azure
test-ubuntu-20.04-python-3Azure
jorisvandenbossche
jorisvandenbossche
reviewed
Nov 22, 2022
View reviewed changes
Comment threadpython/pyarrow/tests/test_cpp_internals.py
Outdated

@AlenkaF@jorisvandenbossche
Update python/pyarrow/tests/test_cpp_internals.py
4669e94
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
@jorisvandenbosschejorisvandenbossche
mentioned this pull request
Nov 22, 2022
 Merged

3 tasks
@kou
kou
merged commit 1e9eb61
into
apache:masterNov 22, 2022
@AlenkaF
AlenkaF
deleted the

ARROW-18340

branch
November 23, 2022 08:41
@ursabot
ursabot
commented
Nov 24, 2022

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = 59f99d2 and contender = 1e9eb61. 1e9eb61 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2

[Finished ⬇️0.34% ⬆️0.54%] test-mac-arm

[Finished ⬇️0.27% ⬆️0.0%] ursa-i9-9960x

[Finished ⬇️0.14% ⬆️0.35%] ursa-thinkcentre-m75q

Buildkite builds:

[Finished] 1e9eb615 ec2-t3-xlarge-us-east-2

[Finished] 1e9eb615 test-mac-arm

[Finished] 1e9eb615 ursa-i9-9960x

[Finished] 1e9eb615 ursa-thinkcentre-m75q

[Finished] 59f99d2c ec2-t3-xlarge-us-east-2

[Finished] 59f99d2c test-mac-arm

[Finished] 59f99d2c ursa-i9-9960x

[Finished] 59f99d2c ursa-thinkcentre-m75q

Supported benchmarks:

ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True

test-mac-arm: Supported benchmark langs: C++, Python, R

ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript

ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
@asfimportasfimport
mentioned this pull request
Nov 24, 2022

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@jorisvandenbosschejorisvandenbosschejorisvandenbossche left review comments

Assignees


No one assigned


Labels

Component: Python

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@AlenkaF@jorisvandenbossche@kou@ursabot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-18340: [Python] PyArrow C++ header files no longer always included in installed pyarrow - #14656
Merged
kou merged 7 commits into
apache:masterfrom 
AlenkaF:ARROW-18340
Nov 22, 2022
Merged
ARROW-18340: [Python] PyArrow C++ header files no longer always included in installed pyarrow#14656
kou merged 7 commits into
apache:masterfrom 
AlenkaF:ARROW-18340
Conversation
@AlenkaF
@AlenkaFAlenkaF
commented
Nov 16, 2022

Copy link


Copy Markdown

Member
This PR adds code to _run_cmake to always copy PyArrow C++ header files to pyarrow/iclude.
@github-actions
github-actionsBot
commented
Nov 16, 2022

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-18340
@github-actions
github-actionsBot
commented
Nov 16, 2022

Copy link


Copy Markdown

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.
jorisvandenbossche
jorisvandenbossche
reviewed
Nov 16, 2022
View reviewed changes
Comment threadpython/setup.py
Outdated

@jorisvandenbossche
jorisvandenbossche
commented
Nov 16, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g wheel
@github-actions
github-actionsBot
commented
Nov 16, 2022

Copy link


Copy Markdown

Revision: d5f953f
Submitted crossbow builds: ursacomputing/crossbow @ actions-1c3aa4b08e
TaskStatus
wheel-macos-big-sur-cp310-arm64Github Actions
wheel-macos-big-sur-cp311-arm64Github Actions
wheel-macos-big-sur-cp38-arm64Github Actions
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-mojave-cp310-amd64Github Actions
wheel-macos-mojave-cp311-amd64Github Actions
wheel-macos-mojave-cp37-amd64Github Actions
wheel-macos-mojave-cp38-amd64Github Actions
wheel-macos-mojave-cp39-amd64Github Actions
wheel-manylinux2014-cp310-amd64Github Actions
wheel-manylinux2014-cp310-arm64Travis CI
wheel-manylinux2014-cp311-amd64Github Actions
wheel-manylinux2014-cp311-arm64Travis CI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64Travis CI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64Travis CI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64Travis CI
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
kou
kou
reviewed
Nov 16, 2022
View reviewed changes
Comment threadpython/setup.py
Outdated

@kou
kou
commented
Nov 16, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g python -g conda
@github-actions
github-actionsBot
commented
Nov 16, 2022

Copy link


Copy Markdown

Revision: d5f953f
Submitted crossbow builds: ursacomputing/crossbow @ actions-9eb5a3c4ca
TaskStatus
conda-cleanAzure
conda-linux-gcc-py310-arm64Azure
conda-linux-gcc-py310-cpuAzure
conda-linux-gcc-py310-cudaAzure
conda-linux-gcc-py310-ppc64leAzure
conda-linux-gcc-py37-arm64Azure
conda-linux-gcc-py37-cpu-r40Azure
conda-linux-gcc-py37-cpu-r41Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py37-ppc64leAzure
conda-linux-gcc-py38-arm64Azure
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py38-ppc64leAzure
conda-linux-gcc-py39-arm64Azure
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-linux-gcc-py39-ppc64leAzure
conda-osx-arm64-clang-py310Azure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py310Azure
conda-osx-clang-py37-r40Azure
conda-osx-clang-py37-r41Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2019-py310Azure
conda-win-vs2019-py37-r40Azure
conda-win-vs2019-py37-r41Azure
conda-win-vs2019-py38Azure
conda-win-vs2019-py39Azure
test-conda-python-3.10Github Actions
test-conda-python-3.11Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-hdfs-2.9.2Github Actions
test-conda-python-3.7-hdfs-3.2.1Github Actions
test-conda-python-3.7-pandas-0.24Github Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-spark-v3.1.2Github Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-v3.2.0Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-dask-latestGithub Actions
test-conda-python-3.9-dask-upstream_develGithub Actions
test-conda-python-3.9-pandas-upstream_develGithub Actions
test-conda-python-3.9-spark-masterGithub Actions
test-cuda-pythonGithub Actions
test-debian-11-python-3Azure
test-fedora-35-python-3Azure
test-ubuntu-20.04-python-3Azure
@kou
kou
commented
Nov 16, 2022

Copy link


Copy Markdown

Member
Can we add a test to detect this problem?
@AlenkaF
AlenkaF
requested a review
from kouNovember 22, 2022 07:39
kou
kou
approved these changes
Nov 22, 2022
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Comment threadpython/pyarrow/tests/test_cpp_internals.py
Outdated

 # Arrow C++ and PyArrow C++ header files together

 source = os.path.dirname(os.path.abspath(__file__))
 pyarrow_dir = os.path.abspath(pjoin(source, '..'))
@koukouNov 22, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

os.path.dirname(source)?
@AlenkaFAlenkaFNov 22, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to go from test folder to the parent pyarrow folder to test if the include folder is present there.
@koukouNov 22, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry... I'm not sure why os.path.abspath() is needed for it.

Could you show the followings in your environment?
  • The source value
  • ls -lah ${source} (${source} is the source value)
  • The pyarrow_dir value
  • ls -lah ${pyarrow_dir} (${pyarrow_dir} is the pyarrow_dir value)
@AlenkaFAlenkaFNov 22, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure! Here it is:
(Pdb) print(source)
/Users/alenkafrim/repos/arrow/python/pyarrow/tests
(Pdb) pyarrow_dir
'/Users/alenkafrim/repos/arrow/python/pyarrow'

ls -lah ouput /test
(pyarrow-dev-9) (base) alenkafrim@Alenkas-MacBook-Pro python % ls -lah /Users/alenkafrim/repos/arrow/python/pyarrow/tests
total 3240
drwxr-xr-x 62 alenkafrim staff 1.9K 22 Nov 10:02 .
drwxr-xr-x 109 alenkafrim staff 3.4K 22 Nov 10:02 ..
-rw-r--r-- 1 alenkafrim staff 0B 5 May 2022 __init__.py
drwxr-xr-x 5 alenkafrim staff 160B 22 Nov 10:03 __pycache__
-rw-r--r-- 1 alenkafrim staff 1.3K 2 Jun 13:33 arrow_16597.py
-rw-r--r-- 1 alenkafrim staff 1.1K 3 Nov 2021 arrow_7980.py
-rw-r--r-- 1 alenkafrim staff 2.0K 11 Apr 2022 bound_function_visit_strings.pyx
-rw-r--r-- 1 alenkafrim staff 6.1K 29 Jun 09:38 conftest.py
drwxr-xr-x 5 alenkafrim staff 160B 27 Oct 12:17 data
-rw-r--r-- 1 alenkafrim staff 961B 3 Nov 2021 deserialize_buffer.py
drwxr-xr-x 3 alenkafrim staff 96B 21 Nov 14:54 interchange
-rw-r--r-- 1 alenkafrim staff 5.0K 3 Nov 2021 pandas_examples.py
-rw-r--r-- 1 alenkafrim staff 1.4K 3 Nov 2021 pandas_threaded_import.py
drwxr-xr-x 16 alenkafrim staff 512B 22 Nov 06:02 parquet
-rw-r--r-- 1 alenkafrim staff 1.9K 11 Apr 2022 pyarrow_cython_example.pyx
-rw-r--r-- 1 alenkafrim staff 953B 19 Apr 2022 read_record_batch.py
-rw-r--r-- 1 alenkafrim staff 13K 22 Apr 2022 strategies.py
-rw-r--r-- 1 alenkafrim staff 1.4K 3 Nov 2021 test_adhoc_memory_leak.py
-rw-r--r-- 1 alenkafrim staff 106K 22 Nov 06:02 test_array.py
-rw-r--r-- 1 alenkafrim staff 2.1K 3 Nov 2021 test_builder.py
-rw-r--r-- 1 alenkafrim staff 13K 19 Apr 2022 test_cffi.py
-rw-r--r-- 1 alenkafrim staff 108K 22 Nov 10:02 test_compute.py
-rw-r--r-- 1 alenkafrim staff 70K 27 Oct 14:04 test_convert_builtin.py
-rw-r--r-- 1 alenkafrim staff 1.8K 22 Nov 10:03 test_cpp_internals.py
-rw-r--r-- 1 alenkafrim staff 71K 22 Nov 06:02 test_csv.py
-rw-r--r-- 1 alenkafrim staff 27K 3 Nov 2021 test_cuda.py
-rw-r--r-- 1 alenkafrim staff 8.5K 3 Nov 2021 test_cuda_numba_interop.py
-rw-r--r-- 1 alenkafrim staff 6.5K 22 Nov 06:02 test_cython.py
-rw-r--r-- 1 alenkafrim staff 175K 22 Nov 06:02 test_dataset.py
-rw-r--r-- 1 alenkafrim staff 891B 3 Nov 2021 test_deprecations.py
-rw-r--r-- 1 alenkafrim staff 9.2K 27 Oct 14:04 test_exec_plan.py
-rw-r--r-- 1 alenkafrim staff 33K 22 Nov 06:02 test_extension_type.py
-rw-r--r-- 1 alenkafrim staff 24K 22 Nov 06:02 test_feather.py
-rw-r--r-- 1 alenkafrim staff 2.3K 19 Apr 2022 test_filesystem.py
-rw-r--r-- 1 alenkafrim staff 79K 22 Nov 06:02 test_flight.py
-rw-r--r-- 1 alenkafrim staff 55K 27 Oct 14:04 test_fs.py
-rw-r--r-- 1 alenkafrim staff 15K 27 Oct 14:04 test_gandiva.py
-rw-r--r-- 1 alenkafrim staff 42K 22 Nov 06:02 test_gdb.py
-rw-r--r-- 1 alenkafrim staff 13K 11 Apr 2022 test_hdfs.py
-rw-r--r-- 1 alenkafrim staff 56K 27 Oct 14:04 test_io.py
-rw-r--r-- 1 alenkafrim staff 36K 22 Nov 06:02 test_ipc.py
-rw-r--r-- 1 alenkafrim staff 12K 22 Nov 06:02 test_json.py
-rw-r--r-- 1 alenkafrim staff 15K 3 Nov 2021 test_jvm.py
-rw-r--r-- 1 alenkafrim staff 7.6K 11 Jul 08:19 test_memory.py
-rw-r--r-- 1 alenkafrim staff 5.8K 19 Apr 2022 test_misc.py
-rw-r--r-- 1 alenkafrim staff 19K 22 Nov 10:02 test_orc.py
-rw-r--r-- 1 alenkafrim staff 159K 22 Nov 06:02 test_pandas.py
-rw-r--r-- 1 alenkafrim staff 44K 22 Nov 06:02 test_plasma.py
-rw-r--r-- 1 alenkafrim staff 3.4K 3 Nov 2021 test_plasma_tf_op.py
-rw-r--r-- 1 alenkafrim staff 20K 22 Nov 10:02 test_scalars.py
-rw-r--r-- 1 alenkafrim staff 21K 22 Nov 06:02 test_schema.py
-rw-r--r-- 1 alenkafrim staff 41K 11 Apr 2022 test_serialization.py
-rw-r--r-- 1 alenkafrim staff 1.6K 11 Apr 2022 test_serialization_deprecated.py
-rw-r--r-- 1 alenkafrim staff 17K 3 Nov 2021 test_sparse_tensor.py
-rw-r--r-- 1 alenkafrim staff 1.7K 3 Nov 2021 test_strategies.py
-rw-r--r-- 1 alenkafrim staff 8.4K 22 Nov 06:02 test_substrait.py
-rw-r--r-- 1 alenkafrim staff 65K 27 Oct 14:04 test_table.py
-rw-r--r-- 1 alenkafrim staff 6.1K 11 Apr 2022 test_tensor.py
-rw-r--r-- 1 alenkafrim staff 33K 22 Nov 10:02 test_types.py
-rw-r--r-- 1 alenkafrim staff 17K 29 Jun 09:38 test_udf.py
-rw-r--r-- 1 alenkafrim staff 1.7K 11 Apr 2022 test_util.py
-rw-r--r-- 1 alenkafrim staff 13K 6 May 2022 util.py

ls -lah ouput pyarrow/
(pyarrow-dev-9) (base) alenkafrim@Alenkas-MacBook-Pro python % ls -lah /Users/alenkafrim/repos/arrow/python/pyarrow total 185744
drwxr-xr-x 109 alenkafrim staff 3.4K 22 Nov 10:02 .
drwxr-xr-x 36 alenkafrim staff 1.1K 22 Nov 10:02 ..
-rw-r--r--@ 1 alenkafrim staff 6.0K 17 Nov 12:41 .DS_Store
-rw-r--r-- 1 alenkafrim staff 2.1K 3 Nov 2021 __init__.pxd
-rw-r--r-- 1 alenkafrim staff 19K 22 Nov 08:17 __init__.py
drwxr-xr-x 20 alenkafrim staff 640B 22 Nov 10:03 __pycache__
-rwxr-xr-x 1 alenkafrim staff 3.8M 21 Nov 12:35 _compute.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 1.7K 6 May 2022 _compute.pxd
-rw-r--r-- 1 alenkafrim staff 83K 22 Nov 10:02 _compute.pyx
-rw-r--r-- 1 alenkafrim staff 1.7K 30 May 12:10 _compute_docstrings.py
-rwxr-xr-x 1 alenkafrim staff 970K 21 Nov 12:35 _csv.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 1.6K 11 Apr 2022 _csv.pxd
-rw-r--r-- 1 alenkafrim staff 51K 22 Nov 06:02 _csv.pyx
-rw-r--r-- 1 alenkafrim staff 1.9K 3 Nov 2021 _cuda.pxd
-rw-r--r-- 1 alenkafrim staff 34K 11 Apr 2022 _cuda.pyx
-rwxr-xr-x 1 alenkafrim staff 3.0M 21 Nov 12:35 _dataset.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 4.3K 27 Oct 14:04 _dataset.pxd
-rw-r--r-- 1 alenkafrim staff 100K 22 Nov 06:02 _dataset.pyx
-rw-r--r-- 1 alenkafrim staff 1.3K 11 Apr 2022 _dataset_orc.pyx
-rwxr-xr-x 1 alenkafrim staff 934K 21 Nov 12:35 _dataset_parquet.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 30K 29 Jun 09:38 _dataset_parquet.pyx
-rwxr-xr-x 1 alenkafrim staff 1.0M 21 Nov 12:35 _exec_plan.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 16K 22 Nov 06:02 _exec_plan.pyx
-rwxr-xr-x 1 alenkafrim staff 265K 21 Nov 12:35 _feather.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 3.7K 11 Apr 2022 _feather.pyx
-rw-r--r-- 1 alenkafrim staff 105K 22 Nov 06:02 _flight.pyx
-rwxr-xr-x 1 alenkafrim staff 1.9M 21 Nov 12:35 _fs.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 2.4K 3 Nov 2021 _fs.pxd
-rw-r--r-- 1 alenkafrim staff 51K 22 Nov 06:02 _fs.pyx
-rw-r--r-- 1 alenkafrim staff 8.0K 29 Jun 09:38 _gcsfs.pyx
-rw-r--r-- 1 alenkafrim staff 186B 21 Nov 12:35 _generated_version.py
-rw-r--r-- 1 alenkafrim staff 5.6K 29 Jun 09:38 _hdfs.pyx
-rwxr-xr-x 1 alenkafrim staff 483K 21 Nov 12:35 _hdfsio.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 13K 11 Apr 2022 _hdfsio.pyx
-rwxr-xr-x 1 alenkafrim staff 199K 21 Nov 12:35 _json.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 8.5K 19 Apr 2022 _json.pyx
-rw-r--r-- 1 alenkafrim staff 5.6K 11 Apr 2022 _orc.pxd
-rw-r--r-- 1 alenkafrim staff 15K 11 Apr 2022 _orc.pyx
-rwxr-xr-x 1 alenkafrim staff 1.6M 21 Nov 12:35 _parquet.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 25K 29 Jun 09:38 _parquet.pxd
-rw-r--r-- 1 alenkafrim staff 57K 27 Oct 14:04 _parquet.pyx
-rw-r--r-- 1 alenkafrim staff 5.9K 19 Apr 2022 _parquet_encryption.pxd
-rw-r--r-- 1 alenkafrim staff 17K 19 Apr 2022 _parquet_encryption.pyx
-rw-r--r-- 1 alenkafrim staff 30K 22 Nov 06:02 _plasma.pyx
-rwxr-xr-x 1 alenkafrim staff 207K 21 Nov 12:35 _pyarrow_cpp_tests.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 1.2K 22 Nov 06:02 _pyarrow_cpp_tests.pxd
-rw-r--r-- 1 alenkafrim staff 1.7K 22 Nov 06:02 _pyarrow_cpp_tests.pyx
-rwxr-xr-x 1 alenkafrim staff 388K 21 Nov 12:35 _s3fs.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 16K 22 Nov 06:02 _s3fs.pyx
-rw-r--r-- 1 alenkafrim staff 6.3K 22 Nov 06:02 _substrait.pyx
-rw-r--r-- 1 alenkafrim staff 91K 22 Nov 06:02 array.pxi
-rw-r--r-- 1 alenkafrim staff 869B 3 Nov 2021 benchmark.pxi
-rw-r--r-- 1 alenkafrim staff 856B 3 Nov 2021 benchmark.py
-rw-r--r-- 1 alenkafrim staff 2.6K 3 Nov 2021 builder.pxi
-rw-r--r-- 1 alenkafrim staff 2.1K 3 Nov 2021 cffi.py
-rw-r--r-- 1 alenkafrim staff 2.0K 11 Apr 2022 compat.pxi
-rw-r--r-- 1 alenkafrim staff 21K 22 Nov 10:02 compute.py
-rw-r--r-- 1 alenkafrim staff 2.7K 11 Apr 2022 config.pxi
-rw-r--r-- 1 alenkafrim staff 6.2K 27 Oct 14:04 conftest.py
-rw-r--r-- 1 alenkafrim staff 974B 11 Apr 2022 csv.py
-rw-r--r-- 1 alenkafrim staff 1.1K 3 Nov 2021 cuda.py
-rw-r--r-- 1 alenkafrim staff 38K 27 Oct 14:04 dataset.py
-rw-r--r-- 1 alenkafrim staff 8.3K 22 Nov 06:02 error.pxi
-rw-r--r-- 1 alenkafrim staff 9.9K 22 Nov 06:02 feather.py
-rw-r--r-- 1 alenkafrim staff 14K 10 Jun 10:00 filesystem.py
-rw-r--r-- 1 alenkafrim staff 1.8K 22 Nov 06:02 flight.py
-rw-r--r-- 1 alenkafrim staff 14K 27 Oct 14:04 fs.py
-rw-r--r-- 1 alenkafrim staff 20K 27 Oct 14:04 gandiva.pyx
-rw-r--r-- 1 alenkafrim staff 7.4K 11 Apr 2022 hdfs.py
drwxr-xr-x 3 alenkafrim staff 96B 17 Nov 11:37 include
drwxr-xr-x 15 alenkafrim staff 480B 22 Nov 10:02 includes
drwxr-xr-x 3 alenkafrim staff 96B 21 Nov 14:54 interchange
-rw-r--r-- 1 alenkafrim staff 68K 27 Oct 14:04 io.pxi
-rw-r--r-- 1 alenkafrim staff 33K 22 Nov 06:02 ipc.pxi
-rw-r--r-- 1 alenkafrim staff 9.7K 22 Nov 06:02 ipc.py
-rw-r--r-- 1 alenkafrim staff 858B 3 Nov 2021 json.py
-rw-r--r-- 1 alenkafrim staff 9.4K 3 Nov 2021 jvm.py
-rwxr-xr-x 1 alenkafrim staff 7.0M 21 Nov 12:35 lib.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 14K 22 Nov 10:02 lib.pxd
-rw-r--r-- 1 alenkafrim staff 4.6K 21 Apr 2022 lib.pyx
-rw-r--r-- 1 alenkafrim staff 19K 21 Nov 12:35 lib_api.h
-rwxr-xr-x 1 alenkafrim staff 12M 21 Nov 12:35 libarrow_python.1100.0.0.dylib
-rwxr-xr-x 1 alenkafrim staff 12M 21 Nov 12:35 libarrow_python.1100.dylib
-rw-r--r-- 1 alenkafrim staff 30M 21 Nov 12:35 libarrow_python.a
-rwxr-xr-x 1 alenkafrim staff 12M 21 Nov 12:35 libarrow_python.dylib
-rwxr-xr-x 1 alenkafrim staff 458K 21 Nov 12:35 libarrow_python_flight.1100.0.0.dylib
-rwxr-xr-x 1 alenkafrim staff 458K 21 Nov 12:35 libarrow_python_flight.1100.dylib
-rw-r--r-- 1 alenkafrim staff 1.2M 21 Nov 12:35 libarrow_python_flight.a
-rwxr-xr-x 1 alenkafrim staff 458K 21 Nov 12:35 libarrow_python_flight.dylib
-rw-r--r-- 1 alenkafrim staff 8.0K 27 Oct 14:04 memory.pxi
-rw-r--r-- 1 alenkafrim staff 12K 22 Nov 06:02 orc.py
-rw-r--r-- 1 alenkafrim staff 7.8K 17 Nov 10:51 pandas-shim.pxi
-rw-r--r-- 1 alenkafrim staff 43K 22 Nov 06:02 pandas_compat.py
drwxr-xr-x 6 alenkafrim staff 192B 22 Nov 06:02 parquet
-rw-r--r-- 1 alenkafrim staff 6.3K 22 Nov 06:02 plasma.py
-rw-r--r-- 1 alenkafrim staff 13K 22 Nov 10:02 public-api.pxi
-rw-r--r-- 1 alenkafrim staff 29K 22 Nov 10:02 scalar.pxi
-rw-r--r-- 1 alenkafrim staff 19K 29 Jun 09:38 serialization.pxi
-rw-r--r-- 1 alenkafrim staff 18K 3 Nov 2021 serialization.py
drwxr-xr-x 8 alenkafrim staff 256B 22 Nov 10:02 src
-rw-r--r-- 1 alenkafrim staff 873B 27 Oct 14:04 substrait.py
-rw-r--r-- 1 alenkafrim staff 158K 22 Nov 06:02 table.pxi
-rw-r--r-- 1 alenkafrim staff 38K 19 Apr 2022 tensor.pxi
drwxr-xr-x 3 alenkafrim staff 96B 27 Oct 14:04 tensorflow
drwxr-xr-x 62 alenkafrim staff 1.9K 22 Nov 10:02 tests
-rw-r--r-- 1 alenkafrim staff 90K 22 Nov 06:02 types.pxi
-rw-r--r-- 1 alenkafrim staff 10K 11 Apr 2022 types.py
-rw-r--r-- 1 alenkafrim staff 4.6K 22 Nov 06:02 util.py
drwxr-xr-x 6 alenkafrim staff 192B 17 Nov 11:07 vendored

If I remember correctly, if I didn't use os.path.abspath() the .. didn't have an effect.
@jorisvandenbosschejorisvandenbosscheNov 22, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For me it seems to work also with a ".." in it:
In [9]: os.path.exists("/home/joris/scipy/repos/arrow/python/pyarrow/tests/../include/arrow/python/")
Out[9]: True

@jorisvandenbosschejorisvandenbosscheNov 22, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But so about @kou's original comment, I think doing os.path.dirname(source) also works because if source is already a directory, dirname gives the parent directory.

Now personally I find .. more explicit to read that it goes one level up.
@AlenkaFAlenkaFNov 22, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, sorry for my confusion.

Can change to pyarrow_dir = os.path.dirname(source), @kou do you agree?
@AlenkaFAlenkaFNov 22, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, now I see Joris you already made a suggestion. Will commit that 👍
@kou
kou
commented
Nov 22, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g python -g conda
@github-actions
github-actionsBot
commented
Nov 22, 2022

Copy link


Copy Markdown

Revision: acf8754
Submitted crossbow builds: ursacomputing/crossbow @ actions-d4595c1625
TaskStatus
conda-cleanAzure
conda-linux-gcc-py310-arm64Azure
conda-linux-gcc-py310-cpuAzure
conda-linux-gcc-py310-cudaAzure
conda-linux-gcc-py310-ppc64leAzure
conda-linux-gcc-py37-arm64Azure
conda-linux-gcc-py37-cpu-r40Azure
conda-linux-gcc-py37-cpu-r41Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py37-ppc64leAzure
conda-linux-gcc-py38-arm64Azure
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py38-ppc64leAzure
conda-linux-gcc-py39-arm64Azure
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-linux-gcc-py39-ppc64leAzure
conda-osx-arm64-clang-py310Azure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py310Azure
conda-osx-clang-py37-r40Azure
conda-osx-clang-py37-r41Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2019-py310Azure
conda-win-vs2019-py37-r40Azure
conda-win-vs2019-py37-r41Azure
conda-win-vs2019-py38Azure
conda-win-vs2019-py39Azure
test-conda-python-3.10Github Actions
test-conda-python-3.11Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-hdfs-2.9.2Github Actions
test-conda-python-3.7-hdfs-3.2.1Github Actions
test-conda-python-3.7-pandas-0.24Github Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-spark-v3.1.2Github Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-v3.2.0Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-dask-latestGithub Actions
test-conda-python-3.9-dask-upstream_develGithub Actions
test-conda-python-3.9-pandas-upstream_develGithub Actions
test-conda-python-3.9-spark-masterGithub Actions
test-cuda-pythonGithub Actions
test-debian-11-python-3Azure
test-fedora-35-python-3Azure
test-ubuntu-20.04-python-3Azure
jorisvandenbossche
jorisvandenbossche
reviewed
Nov 22, 2022
View reviewed changes
Comment threadpython/pyarrow/tests/test_cpp_internals.py
Outdated

@AlenkaF@jorisvandenbossche
Update python/pyarrow/tests/test_cpp_internals.py
4669e94
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
@jorisvandenbosschejorisvandenbossche
mentioned this pull request
Nov 22, 2022
 Merged

3 tasks
@kou
kou
merged commit 1e9eb61
into
apache:masterNov 22, 2022
@AlenkaF
AlenkaF
deleted the

ARROW-18340

branch
November 23, 2022 08:41
@ursabot
ursabot
commented
Nov 24, 2022

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = 59f99d2 and contender = 1e9eb61. 1e9eb61 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2

[Finished ⬇️0.34% ⬆️0.54%] test-mac-arm

[Finished ⬇️0.27% ⬆️0.0%] ursa-i9-9960x

[Finished ⬇️0.14% ⬆️0.35%] ursa-thinkcentre-m75q

Buildkite builds:

[Finished] 1e9eb615 ec2-t3-xlarge-us-east-2

[Finished] 1e9eb615 test-mac-arm

[Finished] 1e9eb615 ursa-i9-9960x

[Finished] 1e9eb615 ursa-thinkcentre-m75q

[Finished] 59f99d2c ec2-t3-xlarge-us-east-2

[Finished] 59f99d2c test-mac-arm

[Finished] 59f99d2c ursa-i9-9960x

[Finished] 59f99d2c ursa-thinkcentre-m75q

Supported benchmarks:

ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True

test-mac-arm: Supported benchmark langs: C++, Python, R

ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript

ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
@asfimportasfimport
mentioned this pull request
Nov 24, 2022

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@jorisvandenbosschejorisvandenbosschejorisvandenbossche left review comments

Assignees


No one assigned


Labels

Component: Python

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@AlenkaF@jorisvandenbossche@kou@ursabot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-18340: [Python] PyArrow C++ header files no longer always included in installed pyarrow - #14656
Merged
kou merged 7 commits into
apache:masterfrom 
AlenkaF:ARROW-18340
Nov 22, 2022
Merged
ARROW-18340: [Python] PyArrow C++ header files no longer always included in installed pyarrow#14656
kou merged 7 commits into
apache:masterfrom 
AlenkaF:ARROW-18340
Conversation
@AlenkaF
@AlenkaFAlenkaF
commented
Nov 16, 2022

Copy link


Copy Markdown

Member
This PR adds code to _run_cmake to always copy PyArrow C++ header files to pyarrow/iclude.
@github-actions
github-actionsBot
commented
Nov 16, 2022

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-18340
@github-actions
github-actionsBot
commented
Nov 16, 2022

Copy link


Copy Markdown

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.
jorisvandenbossche
jorisvandenbossche
reviewed
Nov 16, 2022
View reviewed changes
Comment threadpython/setup.py
Outdated

@jorisvandenbossche
jorisvandenbossche
commented
Nov 16, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g wheel
@github-actions
github-actionsBot
commented
Nov 16, 2022

Copy link


Copy Markdown

Revision: d5f953f
Submitted crossbow builds: ursacomputing/crossbow @ actions-1c3aa4b08e
TaskStatus
wheel-macos-big-sur-cp310-arm64Github Actions
wheel-macos-big-sur-cp311-arm64Github Actions
wheel-macos-big-sur-cp38-arm64Github Actions
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-mojave-cp310-amd64Github Actions
wheel-macos-mojave-cp311-amd64Github Actions
wheel-macos-mojave-cp37-amd64Github Actions
wheel-macos-mojave-cp38-amd64Github Actions
wheel-macos-mojave-cp39-amd64Github Actions
wheel-manylinux2014-cp310-amd64Github Actions
wheel-manylinux2014-cp310-arm64Travis CI
wheel-manylinux2014-cp311-amd64Github Actions
wheel-manylinux2014-cp311-arm64Travis CI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64Travis CI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64Travis CI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64Travis CI
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
kou
kou
reviewed
Nov 16, 2022
View reviewed changes
Comment threadpython/setup.py
Outdated

@kou
kou
commented
Nov 16, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g python -g conda
@github-actions
github-actionsBot
commented
Nov 16, 2022

Copy link


Copy Markdown

Revision: d5f953f
Submitted crossbow builds: ursacomputing/crossbow @ actions-9eb5a3c4ca
TaskStatus
conda-cleanAzure
conda-linux-gcc-py310-arm64Azure
conda-linux-gcc-py310-cpuAzure
conda-linux-gcc-py310-cudaAzure
conda-linux-gcc-py310-ppc64leAzure
conda-linux-gcc-py37-arm64Azure
conda-linux-gcc-py37-cpu-r40Azure
conda-linux-gcc-py37-cpu-r41Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py37-ppc64leAzure
conda-linux-gcc-py38-arm64Azure
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py38-ppc64leAzure
conda-linux-gcc-py39-arm64Azure
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-linux-gcc-py39-ppc64leAzure
conda-osx-arm64-clang-py310Azure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py310Azure
conda-osx-clang-py37-r40Azure
conda-osx-clang-py37-r41Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2019-py310Azure
conda-win-vs2019-py37-r40Azure
conda-win-vs2019-py37-r41Azure
conda-win-vs2019-py38Azure
conda-win-vs2019-py39Azure
test-conda-python-3.10Github Actions
test-conda-python-3.11Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-hdfs-2.9.2Github Actions
test-conda-python-3.7-hdfs-3.2.1Github Actions
test-conda-python-3.7-pandas-0.24Github Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-spark-v3.1.2Github Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-v3.2.0Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-dask-latestGithub Actions
test-conda-python-3.9-dask-upstream_develGithub Actions
test-conda-python-3.9-pandas-upstream_develGithub Actions
test-conda-python-3.9-spark-masterGithub Actions
test-cuda-pythonGithub Actions
test-debian-11-python-3Azure
test-fedora-35-python-3Azure
test-ubuntu-20.04-python-3Azure
@kou
kou
commented
Nov 16, 2022

Copy link


Copy Markdown

Member
Can we add a test to detect this problem?
@AlenkaF
AlenkaF
requested a review
from kouNovember 22, 2022 07:39
kou
kou
approved these changes
Nov 22, 2022
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Comment threadpython/pyarrow/tests/test_cpp_internals.py
Outdated

 # Arrow C++ and PyArrow C++ header files together

 source = os.path.dirname(os.path.abspath(__file__))
 pyarrow_dir = os.path.abspath(pjoin(source, '..'))
@koukouNov 22, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

os.path.dirname(source)?
@AlenkaFAlenkaFNov 22, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to go from test folder to the parent pyarrow folder to test if the include folder is present there.
@koukouNov 22, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry... I'm not sure why os.path.abspath() is needed for it.

Could you show the followings in your environment?
  • The source value
  • ls -lah ${source} (${source} is the source value)
  • The pyarrow_dir value
  • ls -lah ${pyarrow_dir} (${pyarrow_dir} is the pyarrow_dir value)
@AlenkaFAlenkaFNov 22, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure! Here it is:
(Pdb) print(source)
/Users/alenkafrim/repos/arrow/python/pyarrow/tests
(Pdb) pyarrow_dir
'/Users/alenkafrim/repos/arrow/python/pyarrow'

ls -lah ouput /test
(pyarrow-dev-9) (base) alenkafrim@Alenkas-MacBook-Pro python % ls -lah /Users/alenkafrim/repos/arrow/python/pyarrow/tests
total 3240
drwxr-xr-x 62 alenkafrim staff 1.9K 22 Nov 10:02 .
drwxr-xr-x 109 alenkafrim staff 3.4K 22 Nov 10:02 ..
-rw-r--r-- 1 alenkafrim staff 0B 5 May 2022 __init__.py
drwxr-xr-x 5 alenkafrim staff 160B 22 Nov 10:03 __pycache__
-rw-r--r-- 1 alenkafrim staff 1.3K 2 Jun 13:33 arrow_16597.py
-rw-r--r-- 1 alenkafrim staff 1.1K 3 Nov 2021 arrow_7980.py
-rw-r--r-- 1 alenkafrim staff 2.0K 11 Apr 2022 bound_function_visit_strings.pyx
-rw-r--r-- 1 alenkafrim staff 6.1K 29 Jun 09:38 conftest.py
drwxr-xr-x 5 alenkafrim staff 160B 27 Oct 12:17 data
-rw-r--r-- 1 alenkafrim staff 961B 3 Nov 2021 deserialize_buffer.py
drwxr-xr-x 3 alenkafrim staff 96B 21 Nov 14:54 interchange
-rw-r--r-- 1 alenkafrim staff 5.0K 3 Nov 2021 pandas_examples.py
-rw-r--r-- 1 alenkafrim staff 1.4K 3 Nov 2021 pandas_threaded_import.py
drwxr-xr-x 16 alenkafrim staff 512B 22 Nov 06:02 parquet
-rw-r--r-- 1 alenkafrim staff 1.9K 11 Apr 2022 pyarrow_cython_example.pyx
-rw-r--r-- 1 alenkafrim staff 953B 19 Apr 2022 read_record_batch.py
-rw-r--r-- 1 alenkafrim staff 13K 22 Apr 2022 strategies.py
-rw-r--r-- 1 alenkafrim staff 1.4K 3 Nov 2021 test_adhoc_memory_leak.py
-rw-r--r-- 1 alenkafrim staff 106K 22 Nov 06:02 test_array.py
-rw-r--r-- 1 alenkafrim staff 2.1K 3 Nov 2021 test_builder.py
-rw-r--r-- 1 alenkafrim staff 13K 19 Apr 2022 test_cffi.py
-rw-r--r-- 1 alenkafrim staff 108K 22 Nov 10:02 test_compute.py
-rw-r--r-- 1 alenkafrim staff 70K 27 Oct 14:04 test_convert_builtin.py
-rw-r--r-- 1 alenkafrim staff 1.8K 22 Nov 10:03 test_cpp_internals.py
-rw-r--r-- 1 alenkafrim staff 71K 22 Nov 06:02 test_csv.py
-rw-r--r-- 1 alenkafrim staff 27K 3 Nov 2021 test_cuda.py
-rw-r--r-- 1 alenkafrim staff 8.5K 3 Nov 2021 test_cuda_numba_interop.py
-rw-r--r-- 1 alenkafrim staff 6.5K 22 Nov 06:02 test_cython.py
-rw-r--r-- 1 alenkafrim staff 175K 22 Nov 06:02 test_dataset.py
-rw-r--r-- 1 alenkafrim staff 891B 3 Nov 2021 test_deprecations.py
-rw-r--r-- 1 alenkafrim staff 9.2K 27 Oct 14:04 test_exec_plan.py
-rw-r--r-- 1 alenkafrim staff 33K 22 Nov 06:02 test_extension_type.py
-rw-r--r-- 1 alenkafrim staff 24K 22 Nov 06:02 test_feather.py
-rw-r--r-- 1 alenkafrim staff 2.3K 19 Apr 2022 test_filesystem.py
-rw-r--r-- 1 alenkafrim staff 79K 22 Nov 06:02 test_flight.py
-rw-r--r-- 1 alenkafrim staff 55K 27 Oct 14:04 test_fs.py
-rw-r--r-- 1 alenkafrim staff 15K 27 Oct 14:04 test_gandiva.py
-rw-r--r-- 1 alenkafrim staff 42K 22 Nov 06:02 test_gdb.py
-rw-r--r-- 1 alenkafrim staff 13K 11 Apr 2022 test_hdfs.py
-rw-r--r-- 1 alenkafrim staff 56K 27 Oct 14:04 test_io.py
-rw-r--r-- 1 alenkafrim staff 36K 22 Nov 06:02 test_ipc.py
-rw-r--r-- 1 alenkafrim staff 12K 22 Nov 06:02 test_json.py
-rw-r--r-- 1 alenkafrim staff 15K 3 Nov 2021 test_jvm.py
-rw-r--r-- 1 alenkafrim staff 7.6K 11 Jul 08:19 test_memory.py
-rw-r--r-- 1 alenkafrim staff 5.8K 19 Apr 2022 test_misc.py
-rw-r--r-- 1 alenkafrim staff 19K 22 Nov 10:02 test_orc.py
-rw-r--r-- 1 alenkafrim staff 159K 22 Nov 06:02 test_pandas.py
-rw-r--r-- 1 alenkafrim staff 44K 22 Nov 06:02 test_plasma.py
-rw-r--r-- 1 alenkafrim staff 3.4K 3 Nov 2021 test_plasma_tf_op.py
-rw-r--r-- 1 alenkafrim staff 20K 22 Nov 10:02 test_scalars.py
-rw-r--r-- 1 alenkafrim staff 21K 22 Nov 06:02 test_schema.py
-rw-r--r-- 1 alenkafrim staff 41K 11 Apr 2022 test_serialization.py
-rw-r--r-- 1 alenkafrim staff 1.6K 11 Apr 2022 test_serialization_deprecated.py
-rw-r--r-- 1 alenkafrim staff 17K 3 Nov 2021 test_sparse_tensor.py
-rw-r--r-- 1 alenkafrim staff 1.7K 3 Nov 2021 test_strategies.py
-rw-r--r-- 1 alenkafrim staff 8.4K 22 Nov 06:02 test_substrait.py
-rw-r--r-- 1 alenkafrim staff 65K 27 Oct 14:04 test_table.py
-rw-r--r-- 1 alenkafrim staff 6.1K 11 Apr 2022 test_tensor.py
-rw-r--r-- 1 alenkafrim staff 33K 22 Nov 10:02 test_types.py
-rw-r--r-- 1 alenkafrim staff 17K 29 Jun 09:38 test_udf.py
-rw-r--r-- 1 alenkafrim staff 1.7K 11 Apr 2022 test_util.py
-rw-r--r-- 1 alenkafrim staff 13K 6 May 2022 util.py

ls -lah ouput pyarrow/
(pyarrow-dev-9) (base) alenkafrim@Alenkas-MacBook-Pro python % ls -lah /Users/alenkafrim/repos/arrow/python/pyarrow total 185744
drwxr-xr-x 109 alenkafrim staff 3.4K 22 Nov 10:02 .
drwxr-xr-x 36 alenkafrim staff 1.1K 22 Nov 10:02 ..
-rw-r--r--@ 1 alenkafrim staff 6.0K 17 Nov 12:41 .DS_Store
-rw-r--r-- 1 alenkafrim staff 2.1K 3 Nov 2021 __init__.pxd
-rw-r--r-- 1 alenkafrim staff 19K 22 Nov 08:17 __init__.py
drwxr-xr-x 20 alenkafrim staff 640B 22 Nov 10:03 __pycache__
-rwxr-xr-x 1 alenkafrim staff 3.8M 21 Nov 12:35 _compute.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 1.7K 6 May 2022 _compute.pxd
-rw-r--r-- 1 alenkafrim staff 83K 22 Nov 10:02 _compute.pyx
-rw-r--r-- 1 alenkafrim staff 1.7K 30 May 12:10 _compute_docstrings.py
-rwxr-xr-x 1 alenkafrim staff 970K 21 Nov 12:35 _csv.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 1.6K 11 Apr 2022 _csv.pxd
-rw-r--r-- 1 alenkafrim staff 51K 22 Nov 06:02 _csv.pyx
-rw-r--r-- 1 alenkafrim staff 1.9K 3 Nov 2021 _cuda.pxd
-rw-r--r-- 1 alenkafrim staff 34K 11 Apr 2022 _cuda.pyx
-rwxr-xr-x 1 alenkafrim staff 3.0M 21 Nov 12:35 _dataset.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 4.3K 27 Oct 14:04 _dataset.pxd
-rw-r--r-- 1 alenkafrim staff 100K 22 Nov 06:02 _dataset.pyx
-rw-r--r-- 1 alenkafrim staff 1.3K 11 Apr 2022 _dataset_orc.pyx
-rwxr-xr-x 1 alenkafrim staff 934K 21 Nov 12:35 _dataset_parquet.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 30K 29 Jun 09:38 _dataset_parquet.pyx
-rwxr-xr-x 1 alenkafrim staff 1.0M 21 Nov 12:35 _exec_plan.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 16K 22 Nov 06:02 _exec_plan.pyx
-rwxr-xr-x 1 alenkafrim staff 265K 21 Nov 12:35 _feather.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 3.7K 11 Apr 2022 _feather.pyx
-rw-r--r-- 1 alenkafrim staff 105K 22 Nov 06:02 _flight.pyx
-rwxr-xr-x 1 alenkafrim staff 1.9M 21 Nov 12:35 _fs.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 2.4K 3 Nov 2021 _fs.pxd
-rw-r--r-- 1 alenkafrim staff 51K 22 Nov 06:02 _fs.pyx
-rw-r--r-- 1 alenkafrim staff 8.0K 29 Jun 09:38 _gcsfs.pyx
-rw-r--r-- 1 alenkafrim staff 186B 21 Nov 12:35 _generated_version.py
-rw-r--r-- 1 alenkafrim staff 5.6K 29 Jun 09:38 _hdfs.pyx
-rwxr-xr-x 1 alenkafrim staff 483K 21 Nov 12:35 _hdfsio.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 13K 11 Apr 2022 _hdfsio.pyx
-rwxr-xr-x 1 alenkafrim staff 199K 21 Nov 12:35 _json.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 8.5K 19 Apr 2022 _json.pyx
-rw-r--r-- 1 alenkafrim staff 5.6K 11 Apr 2022 _orc.pxd
-rw-r--r-- 1 alenkafrim staff 15K 11 Apr 2022 _orc.pyx
-rwxr-xr-x 1 alenkafrim staff 1.6M 21 Nov 12:35 _parquet.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 25K 29 Jun 09:38 _parquet.pxd
-rw-r--r-- 1 alenkafrim staff 57K 27 Oct 14:04 _parquet.pyx
-rw-r--r-- 1 alenkafrim staff 5.9K 19 Apr 2022 _parquet_encryption.pxd
-rw-r--r-- 1 alenkafrim staff 17K 19 Apr 2022 _parquet_encryption.pyx
-rw-r--r-- 1 alenkafrim staff 30K 22 Nov 06:02 _plasma.pyx
-rwxr-xr-x 1 alenkafrim staff 207K 21 Nov 12:35 _pyarrow_cpp_tests.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 1.2K 22 Nov 06:02 _pyarrow_cpp_tests.pxd
-rw-r--r-- 1 alenkafrim staff 1.7K 22 Nov 06:02 _pyarrow_cpp_tests.pyx
-rwxr-xr-x 1 alenkafrim staff 388K 21 Nov 12:35 _s3fs.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 16K 22 Nov 06:02 _s3fs.pyx
-rw-r--r-- 1 alenkafrim staff 6.3K 22 Nov 06:02 _substrait.pyx
-rw-r--r-- 1 alenkafrim staff 91K 22 Nov 06:02 array.pxi
-rw-r--r-- 1 alenkafrim staff 869B 3 Nov 2021 benchmark.pxi
-rw-r--r-- 1 alenkafrim staff 856B 3 Nov 2021 benchmark.py
-rw-r--r-- 1 alenkafrim staff 2.6K 3 Nov 2021 builder.pxi
-rw-r--r-- 1 alenkafrim staff 2.1K 3 Nov 2021 cffi.py
-rw-r--r-- 1 alenkafrim staff 2.0K 11 Apr 2022 compat.pxi
-rw-r--r-- 1 alenkafrim staff 21K 22 Nov 10:02 compute.py
-rw-r--r-- 1 alenkafrim staff 2.7K 11 Apr 2022 config.pxi
-rw-r--r-- 1 alenkafrim staff 6.2K 27 Oct 14:04 conftest.py
-rw-r--r-- 1 alenkafrim staff 974B 11 Apr 2022 csv.py
-rw-r--r-- 1 alenkafrim staff 1.1K 3 Nov 2021 cuda.py
-rw-r--r-- 1 alenkafrim staff 38K 27 Oct 14:04 dataset.py
-rw-r--r-- 1 alenkafrim staff 8.3K 22 Nov 06:02 error.pxi
-rw-r--r-- 1 alenkafrim staff 9.9K 22 Nov 06:02 feather.py
-rw-r--r-- 1 alenkafrim staff 14K 10 Jun 10:00 filesystem.py
-rw-r--r-- 1 alenkafrim staff 1.8K 22 Nov 06:02 flight.py
-rw-r--r-- 1 alenkafrim staff 14K 27 Oct 14:04 fs.py
-rw-r--r-- 1 alenkafrim staff 20K 27 Oct 14:04 gandiva.pyx
-rw-r--r-- 1 alenkafrim staff 7.4K 11 Apr 2022 hdfs.py
drwxr-xr-x 3 alenkafrim staff 96B 17 Nov 11:37 include
drwxr-xr-x 15 alenkafrim staff 480B 22 Nov 10:02 includes
drwxr-xr-x 3 alenkafrim staff 96B 21 Nov 14:54 interchange
-rw-r--r-- 1 alenkafrim staff 68K 27 Oct 14:04 io.pxi
-rw-r--r-- 1 alenkafrim staff 33K 22 Nov 06:02 ipc.pxi
-rw-r--r-- 1 alenkafrim staff 9.7K 22 Nov 06:02 ipc.py
-rw-r--r-- 1 alenkafrim staff 858B 3 Nov 2021 json.py
-rw-r--r-- 1 alenkafrim staff 9.4K 3 Nov 2021 jvm.py
-rwxr-xr-x 1 alenkafrim staff 7.0M 21 Nov 12:35 lib.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 14K 22 Nov 10:02 lib.pxd
-rw-r--r-- 1 alenkafrim staff 4.6K 21 Apr 2022 lib.pyx
-rw-r--r-- 1 alenkafrim staff 19K 21 Nov 12:35 lib_api.h
-rwxr-xr-x 1 alenkafrim staff 12M 21 Nov 12:35 libarrow_python.1100.0.0.dylib
-rwxr-xr-x 1 alenkafrim staff 12M 21 Nov 12:35 libarrow_python.1100.dylib
-rw-r--r-- 1 alenkafrim staff 30M 21 Nov 12:35 libarrow_python.a
-rwxr-xr-x 1 alenkafrim staff 12M 21 Nov 12:35 libarrow_python.dylib
-rwxr-xr-x 1 alenkafrim staff 458K 21 Nov 12:35 libarrow_python_flight.1100.0.0.dylib
-rwxr-xr-x 1 alenkafrim staff 458K 21 Nov 12:35 libarrow_python_flight.1100.dylib
-rw-r--r-- 1 alenkafrim staff 1.2M 21 Nov 12:35 libarrow_python_flight.a
-rwxr-xr-x 1 alenkafrim staff 458K 21 Nov 12:35 libarrow_python_flight.dylib
-rw-r--r-- 1 alenkafrim staff 8.0K 27 Oct 14:04 memory.pxi
-rw-r--r-- 1 alenkafrim staff 12K 22 Nov 06:02 orc.py
-rw-r--r-- 1 alenkafrim staff 7.8K 17 Nov 10:51 pandas-shim.pxi
-rw-r--r-- 1 alenkafrim staff 43K 22 Nov 06:02 pandas_compat.py
drwxr-xr-x 6 alenkafrim staff 192B 22 Nov 06:02 parquet
-rw-r--r-- 1 alenkafrim staff 6.3K 22 Nov 06:02 plasma.py
-rw-r--r-- 1 alenkafrim staff 13K 22 Nov 10:02 public-api.pxi
-rw-r--r-- 1 alenkafrim staff 29K 22 Nov 10:02 scalar.pxi
-rw-r--r-- 1 alenkafrim staff 19K 29 Jun 09:38 serialization.pxi
-rw-r--r-- 1 alenkafrim staff 18K 3 Nov 2021 serialization.py
drwxr-xr-x 8 alenkafrim staff 256B 22 Nov 10:02 src
-rw-r--r-- 1 alenkafrim staff 873B 27 Oct 14:04 substrait.py
-rw-r--r-- 1 alenkafrim staff 158K 22 Nov 06:02 table.pxi
-rw-r--r-- 1 alenkafrim staff 38K 19 Apr 2022 tensor.pxi
drwxr-xr-x 3 alenkafrim staff 96B 27 Oct 14:04 tensorflow
drwxr-xr-x 62 alenkafrim staff 1.9K 22 Nov 10:02 tests
-rw-r--r-- 1 alenkafrim staff 90K 22 Nov 06:02 types.pxi
-rw-r--r-- 1 alenkafrim staff 10K 11 Apr 2022 types.py
-rw-r--r-- 1 alenkafrim staff 4.6K 22 Nov 06:02 util.py
drwxr-xr-x 6 alenkafrim staff 192B 17 Nov 11:07 vendored

If I remember correctly, if I didn't use os.path.abspath() the .. didn't have an effect.
@jorisvandenbosschejorisvandenbosscheNov 22, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For me it seems to work also with a ".." in it:
In [9]: os.path.exists("/home/joris/scipy/repos/arrow/python/pyarrow/tests/../include/arrow/python/")
Out[9]: True

@jorisvandenbosschejorisvandenbosscheNov 22, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But so about @kou's original comment, I think doing os.path.dirname(source) also works because if source is already a directory, dirname gives the parent directory.

Now personally I find .. more explicit to read that it goes one level up.
@AlenkaFAlenkaFNov 22, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, sorry for my confusion.

Can change to pyarrow_dir = os.path.dirname(source), @kou do you agree?
@AlenkaFAlenkaFNov 22, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, now I see Joris you already made a suggestion. Will commit that 👍
@kou
kou
commented
Nov 22, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g python -g conda
@github-actions
github-actionsBot
commented
Nov 22, 2022

Copy link


Copy Markdown

Revision: acf8754
Submitted crossbow builds: ursacomputing/crossbow @ actions-d4595c1625
TaskStatus
conda-cleanAzure
conda-linux-gcc-py310-arm64Azure
conda-linux-gcc-py310-cpuAzure
conda-linux-gcc-py310-cudaAzure
conda-linux-gcc-py310-ppc64leAzure
conda-linux-gcc-py37-arm64Azure
conda-linux-gcc-py37-cpu-r40Azure
conda-linux-gcc-py37-cpu-r41Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py37-ppc64leAzure
conda-linux-gcc-py38-arm64Azure
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py38-ppc64leAzure
conda-linux-gcc-py39-arm64Azure
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-linux-gcc-py39-ppc64leAzure
conda-osx-arm64-clang-py310Azure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py310Azure
conda-osx-clang-py37-r40Azure
conda-osx-clang-py37-r41Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2019-py310Azure
conda-win-vs2019-py37-r40Azure
conda-win-vs2019-py37-r41Azure
conda-win-vs2019-py38Azure
conda-win-vs2019-py39Azure
test-conda-python-3.10Github Actions
test-conda-python-3.11Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-hdfs-2.9.2Github Actions
test-conda-python-3.7-hdfs-3.2.1Github Actions
test-conda-python-3.7-pandas-0.24Github Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-spark-v3.1.2Github Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-v3.2.0Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-dask-latestGithub Actions
test-conda-python-3.9-dask-upstream_develGithub Actions
test-conda-python-3.9-pandas-upstream_develGithub Actions
test-conda-python-3.9-spark-masterGithub Actions
test-cuda-pythonGithub Actions
test-debian-11-python-3Azure
test-fedora-35-python-3Azure
test-ubuntu-20.04-python-3Azure
jorisvandenbossche
jorisvandenbossche
reviewed
Nov 22, 2022
View reviewed changes
Comment threadpython/pyarrow/tests/test_cpp_internals.py
Outdated

@AlenkaF@jorisvandenbossche
Update python/pyarrow/tests/test_cpp_internals.py
4669e94
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
@jorisvandenbosschejorisvandenbossche
mentioned this pull request
Nov 22, 2022
 Merged

3 tasks
@kou
kou
merged commit 1e9eb61
into
apache:masterNov 22, 2022
@AlenkaF
AlenkaF
deleted the

ARROW-18340

branch
November 23, 2022 08:41
@ursabot
ursabot
commented
Nov 24, 2022

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = 59f99d2 and contender = 1e9eb61. 1e9eb61 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2

[Finished ⬇️0.34% ⬆️0.54%] test-mac-arm

[Finished ⬇️0.27% ⬆️0.0%] ursa-i9-9960x

[Finished ⬇️0.14% ⬆️0.35%] ursa-thinkcentre-m75q

Buildkite builds:

[Finished] 1e9eb615 ec2-t3-xlarge-us-east-2

[Finished] 1e9eb615 test-mac-arm

[Finished] 1e9eb615 ursa-i9-9960x

[Finished] 1e9eb615 ursa-thinkcentre-m75q

[Finished] 59f99d2c ec2-t3-xlarge-us-east-2

[Finished] 59f99d2c test-mac-arm

[Finished] 59f99d2c ursa-i9-9960x

[Finished] 59f99d2c ursa-thinkcentre-m75q

Supported benchmarks:

ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True

test-mac-arm: Supported benchmark langs: C++, Python, R

ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript

ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
@asfimportasfimport
mentioned this pull request
Nov 24, 2022

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@jorisvandenbosschejorisvandenbosschejorisvandenbossche left review comments

Assignees


No one assigned


Labels

Component: Python

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@AlenkaF@jorisvandenbossche@kou@ursabot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
ARROW-18340: [Python] PyArrow C++ header files no longer always included in installed pyarrow - #14656
Merged
kou merged 7 commits into
apache:masterfrom 
AlenkaF:ARROW-18340
Nov 22, 2022
Merged
ARROW-18340: [Python] PyArrow C++ header files no longer always included in installed pyarrow#14656
kou merged 7 commits into
apache:masterfrom 
AlenkaF:ARROW-18340
Conversation
@AlenkaF
@AlenkaFAlenkaF
commented
Nov 16, 2022

Copy link


Copy Markdown

Member
This PR adds code to _run_cmake to always copy PyArrow C++ header files to pyarrow/iclude.
@github-actions
github-actionsBot
commented
Nov 16, 2022

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-18340
@github-actions
github-actionsBot
commented
Nov 16, 2022

Copy link


Copy Markdown

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.
jorisvandenbossche
jorisvandenbossche
reviewed
Nov 16, 2022
View reviewed changes
Comment threadpython/setup.py
Outdated

@jorisvandenbossche
jorisvandenbossche
commented
Nov 16, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g wheel
@github-actions
github-actionsBot
commented
Nov 16, 2022

Copy link


Copy Markdown

Revision: d5f953f
Submitted crossbow builds: ursacomputing/crossbow @ actions-1c3aa4b08e
TaskStatus
wheel-macos-big-sur-cp310-arm64Github Actions
wheel-macos-big-sur-cp311-arm64Github Actions
wheel-macos-big-sur-cp38-arm64Github Actions
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-mojave-cp310-amd64Github Actions
wheel-macos-mojave-cp311-amd64Github Actions
wheel-macos-mojave-cp37-amd64Github Actions
wheel-macos-mojave-cp38-amd64Github Actions
wheel-macos-mojave-cp39-amd64Github Actions
wheel-manylinux2014-cp310-amd64Github Actions
wheel-manylinux2014-cp310-arm64Travis CI
wheel-manylinux2014-cp311-amd64Github Actions
wheel-manylinux2014-cp311-arm64Travis CI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64Travis CI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64Travis CI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64Travis CI
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
kou
kou
reviewed
Nov 16, 2022
View reviewed changes
Comment threadpython/setup.py
Outdated

@kou
kou
commented
Nov 16, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g python -g conda
@github-actions
github-actionsBot
commented
Nov 16, 2022

Copy link


Copy Markdown

Revision: d5f953f
Submitted crossbow builds: ursacomputing/crossbow @ actions-9eb5a3c4ca
TaskStatus
conda-cleanAzure
conda-linux-gcc-py310-arm64Azure
conda-linux-gcc-py310-cpuAzure
conda-linux-gcc-py310-cudaAzure
conda-linux-gcc-py310-ppc64leAzure
conda-linux-gcc-py37-arm64Azure
conda-linux-gcc-py37-cpu-r40Azure
conda-linux-gcc-py37-cpu-r41Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py37-ppc64leAzure
conda-linux-gcc-py38-arm64Azure
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py38-ppc64leAzure
conda-linux-gcc-py39-arm64Azure
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-linux-gcc-py39-ppc64leAzure
conda-osx-arm64-clang-py310Azure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py310Azure
conda-osx-clang-py37-r40Azure
conda-osx-clang-py37-r41Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2019-py310Azure
conda-win-vs2019-py37-r40Azure
conda-win-vs2019-py37-r41Azure
conda-win-vs2019-py38Azure
conda-win-vs2019-py39Azure
test-conda-python-3.10Github Actions
test-conda-python-3.11Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-hdfs-2.9.2Github Actions
test-conda-python-3.7-hdfs-3.2.1Github Actions
test-conda-python-3.7-pandas-0.24Github Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-spark-v3.1.2Github Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-v3.2.0Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-dask-latestGithub Actions
test-conda-python-3.9-dask-upstream_develGithub Actions
test-conda-python-3.9-pandas-upstream_develGithub Actions
test-conda-python-3.9-spark-masterGithub Actions
test-cuda-pythonGithub Actions
test-debian-11-python-3Azure
test-fedora-35-python-3Azure
test-ubuntu-20.04-python-3Azure
@kou
kou
commented
Nov 16, 2022

Copy link


Copy Markdown

Member
Can we add a test to detect this problem?
@AlenkaF
AlenkaF
requested a review
from kouNovember 22, 2022 07:39
kou
kou
approved these changes
Nov 22, 2022
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Comment threadpython/pyarrow/tests/test_cpp_internals.py
Outdated

 # Arrow C++ and PyArrow C++ header files together

 source = os.path.dirname(os.path.abspath(__file__))
 pyarrow_dir = os.path.abspath(pjoin(source, '..'))
@koukouNov 22, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

os.path.dirname(source)?
@AlenkaFAlenkaFNov 22, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to go from test folder to the parent pyarrow folder to test if the include folder is present there.
@koukouNov 22, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry... I'm not sure why os.path.abspath() is needed for it.

Could you show the followings in your environment?
  • The source value
  • ls -lah ${source} (${source} is the source value)
  • The pyarrow_dir value
  • ls -lah ${pyarrow_dir} (${pyarrow_dir} is the pyarrow_dir value)
@AlenkaFAlenkaFNov 22, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure! Here it is:
(Pdb) print(source)
/Users/alenkafrim/repos/arrow/python/pyarrow/tests
(Pdb) pyarrow_dir
'/Users/alenkafrim/repos/arrow/python/pyarrow'

ls -lah ouput /test
(pyarrow-dev-9) (base) alenkafrim@Alenkas-MacBook-Pro python % ls -lah /Users/alenkafrim/repos/arrow/python/pyarrow/tests
total 3240
drwxr-xr-x 62 alenkafrim staff 1.9K 22 Nov 10:02 .
drwxr-xr-x 109 alenkafrim staff 3.4K 22 Nov 10:02 ..
-rw-r--r-- 1 alenkafrim staff 0B 5 May 2022 __init__.py
drwxr-xr-x 5 alenkafrim staff 160B 22 Nov 10:03 __pycache__
-rw-r--r-- 1 alenkafrim staff 1.3K 2 Jun 13:33 arrow_16597.py
-rw-r--r-- 1 alenkafrim staff 1.1K 3 Nov 2021 arrow_7980.py
-rw-r--r-- 1 alenkafrim staff 2.0K 11 Apr 2022 bound_function_visit_strings.pyx
-rw-r--r-- 1 alenkafrim staff 6.1K 29 Jun 09:38 conftest.py
drwxr-xr-x 5 alenkafrim staff 160B 27 Oct 12:17 data
-rw-r--r-- 1 alenkafrim staff 961B 3 Nov 2021 deserialize_buffer.py
drwxr-xr-x 3 alenkafrim staff 96B 21 Nov 14:54 interchange
-rw-r--r-- 1 alenkafrim staff 5.0K 3 Nov 2021 pandas_examples.py
-rw-r--r-- 1 alenkafrim staff 1.4K 3 Nov 2021 pandas_threaded_import.py
drwxr-xr-x 16 alenkafrim staff 512B 22 Nov 06:02 parquet
-rw-r--r-- 1 alenkafrim staff 1.9K 11 Apr 2022 pyarrow_cython_example.pyx
-rw-r--r-- 1 alenkafrim staff 953B 19 Apr 2022 read_record_batch.py
-rw-r--r-- 1 alenkafrim staff 13K 22 Apr 2022 strategies.py
-rw-r--r-- 1 alenkafrim staff 1.4K 3 Nov 2021 test_adhoc_memory_leak.py
-rw-r--r-- 1 alenkafrim staff 106K 22 Nov 06:02 test_array.py
-rw-r--r-- 1 alenkafrim staff 2.1K 3 Nov 2021 test_builder.py
-rw-r--r-- 1 alenkafrim staff 13K 19 Apr 2022 test_cffi.py
-rw-r--r-- 1 alenkafrim staff 108K 22 Nov 10:02 test_compute.py
-rw-r--r-- 1 alenkafrim staff 70K 27 Oct 14:04 test_convert_builtin.py
-rw-r--r-- 1 alenkafrim staff 1.8K 22 Nov 10:03 test_cpp_internals.py
-rw-r--r-- 1 alenkafrim staff 71K 22 Nov 06:02 test_csv.py
-rw-r--r-- 1 alenkafrim staff 27K 3 Nov 2021 test_cuda.py
-rw-r--r-- 1 alenkafrim staff 8.5K 3 Nov 2021 test_cuda_numba_interop.py
-rw-r--r-- 1 alenkafrim staff 6.5K 22 Nov 06:02 test_cython.py
-rw-r--r-- 1 alenkafrim staff 175K 22 Nov 06:02 test_dataset.py
-rw-r--r-- 1 alenkafrim staff 891B 3 Nov 2021 test_deprecations.py
-rw-r--r-- 1 alenkafrim staff 9.2K 27 Oct 14:04 test_exec_plan.py
-rw-r--r-- 1 alenkafrim staff 33K 22 Nov 06:02 test_extension_type.py
-rw-r--r-- 1 alenkafrim staff 24K 22 Nov 06:02 test_feather.py
-rw-r--r-- 1 alenkafrim staff 2.3K 19 Apr 2022 test_filesystem.py
-rw-r--r-- 1 alenkafrim staff 79K 22 Nov 06:02 test_flight.py
-rw-r--r-- 1 alenkafrim staff 55K 27 Oct 14:04 test_fs.py
-rw-r--r-- 1 alenkafrim staff 15K 27 Oct 14:04 test_gandiva.py
-rw-r--r-- 1 alenkafrim staff 42K 22 Nov 06:02 test_gdb.py
-rw-r--r-- 1 alenkafrim staff 13K 11 Apr 2022 test_hdfs.py
-rw-r--r-- 1 alenkafrim staff 56K 27 Oct 14:04 test_io.py
-rw-r--r-- 1 alenkafrim staff 36K 22 Nov 06:02 test_ipc.py
-rw-r--r-- 1 alenkafrim staff 12K 22 Nov 06:02 test_json.py
-rw-r--r-- 1 alenkafrim staff 15K 3 Nov 2021 test_jvm.py
-rw-r--r-- 1 alenkafrim staff 7.6K 11 Jul 08:19 test_memory.py
-rw-r--r-- 1 alenkafrim staff 5.8K 19 Apr 2022 test_misc.py
-rw-r--r-- 1 alenkafrim staff 19K 22 Nov 10:02 test_orc.py
-rw-r--r-- 1 alenkafrim staff 159K 22 Nov 06:02 test_pandas.py
-rw-r--r-- 1 alenkafrim staff 44K 22 Nov 06:02 test_plasma.py
-rw-r--r-- 1 alenkafrim staff 3.4K 3 Nov 2021 test_plasma_tf_op.py
-rw-r--r-- 1 alenkafrim staff 20K 22 Nov 10:02 test_scalars.py
-rw-r--r-- 1 alenkafrim staff 21K 22 Nov 06:02 test_schema.py
-rw-r--r-- 1 alenkafrim staff 41K 11 Apr 2022 test_serialization.py
-rw-r--r-- 1 alenkafrim staff 1.6K 11 Apr 2022 test_serialization_deprecated.py
-rw-r--r-- 1 alenkafrim staff 17K 3 Nov 2021 test_sparse_tensor.py
-rw-r--r-- 1 alenkafrim staff 1.7K 3 Nov 2021 test_strategies.py
-rw-r--r-- 1 alenkafrim staff 8.4K 22 Nov 06:02 test_substrait.py
-rw-r--r-- 1 alenkafrim staff 65K 27 Oct 14:04 test_table.py
-rw-r--r-- 1 alenkafrim staff 6.1K 11 Apr 2022 test_tensor.py
-rw-r--r-- 1 alenkafrim staff 33K 22 Nov 10:02 test_types.py
-rw-r--r-- 1 alenkafrim staff 17K 29 Jun 09:38 test_udf.py
-rw-r--r-- 1 alenkafrim staff 1.7K 11 Apr 2022 test_util.py
-rw-r--r-- 1 alenkafrim staff 13K 6 May 2022 util.py

ls -lah ouput pyarrow/
(pyarrow-dev-9) (base) alenkafrim@Alenkas-MacBook-Pro python % ls -lah /Users/alenkafrim/repos/arrow/python/pyarrow total 185744
drwxr-xr-x 109 alenkafrim staff 3.4K 22 Nov 10:02 .
drwxr-xr-x 36 alenkafrim staff 1.1K 22 Nov 10:02 ..
-rw-r--r--@ 1 alenkafrim staff 6.0K 17 Nov 12:41 .DS_Store
-rw-r--r-- 1 alenkafrim staff 2.1K 3 Nov 2021 __init__.pxd
-rw-r--r-- 1 alenkafrim staff 19K 22 Nov 08:17 __init__.py
drwxr-xr-x 20 alenkafrim staff 640B 22 Nov 10:03 __pycache__
-rwxr-xr-x 1 alenkafrim staff 3.8M 21 Nov 12:35 _compute.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 1.7K 6 May 2022 _compute.pxd
-rw-r--r-- 1 alenkafrim staff 83K 22 Nov 10:02 _compute.pyx
-rw-r--r-- 1 alenkafrim staff 1.7K 30 May 12:10 _compute_docstrings.py
-rwxr-xr-x 1 alenkafrim staff 970K 21 Nov 12:35 _csv.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 1.6K 11 Apr 2022 _csv.pxd
-rw-r--r-- 1 alenkafrim staff 51K 22 Nov 06:02 _csv.pyx
-rw-r--r-- 1 alenkafrim staff 1.9K 3 Nov 2021 _cuda.pxd
-rw-r--r-- 1 alenkafrim staff 34K 11 Apr 2022 _cuda.pyx
-rwxr-xr-x 1 alenkafrim staff 3.0M 21 Nov 12:35 _dataset.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 4.3K 27 Oct 14:04 _dataset.pxd
-rw-r--r-- 1 alenkafrim staff 100K 22 Nov 06:02 _dataset.pyx
-rw-r--r-- 1 alenkafrim staff 1.3K 11 Apr 2022 _dataset_orc.pyx
-rwxr-xr-x 1 alenkafrim staff 934K 21 Nov 12:35 _dataset_parquet.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 30K 29 Jun 09:38 _dataset_parquet.pyx
-rwxr-xr-x 1 alenkafrim staff 1.0M 21 Nov 12:35 _exec_plan.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 16K 22 Nov 06:02 _exec_plan.pyx
-rwxr-xr-x 1 alenkafrim staff 265K 21 Nov 12:35 _feather.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 3.7K 11 Apr 2022 _feather.pyx
-rw-r--r-- 1 alenkafrim staff 105K 22 Nov 06:02 _flight.pyx
-rwxr-xr-x 1 alenkafrim staff 1.9M 21 Nov 12:35 _fs.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 2.4K 3 Nov 2021 _fs.pxd
-rw-r--r-- 1 alenkafrim staff 51K 22 Nov 06:02 _fs.pyx
-rw-r--r-- 1 alenkafrim staff 8.0K 29 Jun 09:38 _gcsfs.pyx
-rw-r--r-- 1 alenkafrim staff 186B 21 Nov 12:35 _generated_version.py
-rw-r--r-- 1 alenkafrim staff 5.6K 29 Jun 09:38 _hdfs.pyx
-rwxr-xr-x 1 alenkafrim staff 483K 21 Nov 12:35 _hdfsio.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 13K 11 Apr 2022 _hdfsio.pyx
-rwxr-xr-x 1 alenkafrim staff 199K 21 Nov 12:35 _json.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 8.5K 19 Apr 2022 _json.pyx
-rw-r--r-- 1 alenkafrim staff 5.6K 11 Apr 2022 _orc.pxd
-rw-r--r-- 1 alenkafrim staff 15K 11 Apr 2022 _orc.pyx
-rwxr-xr-x 1 alenkafrim staff 1.6M 21 Nov 12:35 _parquet.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 25K 29 Jun 09:38 _parquet.pxd
-rw-r--r-- 1 alenkafrim staff 57K 27 Oct 14:04 _parquet.pyx
-rw-r--r-- 1 alenkafrim staff 5.9K 19 Apr 2022 _parquet_encryption.pxd
-rw-r--r-- 1 alenkafrim staff 17K 19 Apr 2022 _parquet_encryption.pyx
-rw-r--r-- 1 alenkafrim staff 30K 22 Nov 06:02 _plasma.pyx
-rwxr-xr-x 1 alenkafrim staff 207K 21 Nov 12:35 _pyarrow_cpp_tests.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 1.2K 22 Nov 06:02 _pyarrow_cpp_tests.pxd
-rw-r--r-- 1 alenkafrim staff 1.7K 22 Nov 06:02 _pyarrow_cpp_tests.pyx
-rwxr-xr-x 1 alenkafrim staff 388K 21 Nov 12:35 _s3fs.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 16K 22 Nov 06:02 _s3fs.pyx
-rw-r--r-- 1 alenkafrim staff 6.3K 22 Nov 06:02 _substrait.pyx
-rw-r--r-- 1 alenkafrim staff 91K 22 Nov 06:02 array.pxi
-rw-r--r-- 1 alenkafrim staff 869B 3 Nov 2021 benchmark.pxi
-rw-r--r-- 1 alenkafrim staff 856B 3 Nov 2021 benchmark.py
-rw-r--r-- 1 alenkafrim staff 2.6K 3 Nov 2021 builder.pxi
-rw-r--r-- 1 alenkafrim staff 2.1K 3 Nov 2021 cffi.py
-rw-r--r-- 1 alenkafrim staff 2.0K 11 Apr 2022 compat.pxi
-rw-r--r-- 1 alenkafrim staff 21K 22 Nov 10:02 compute.py
-rw-r--r-- 1 alenkafrim staff 2.7K 11 Apr 2022 config.pxi
-rw-r--r-- 1 alenkafrim staff 6.2K 27 Oct 14:04 conftest.py
-rw-r--r-- 1 alenkafrim staff 974B 11 Apr 2022 csv.py
-rw-r--r-- 1 alenkafrim staff 1.1K 3 Nov 2021 cuda.py
-rw-r--r-- 1 alenkafrim staff 38K 27 Oct 14:04 dataset.py
-rw-r--r-- 1 alenkafrim staff 8.3K 22 Nov 06:02 error.pxi
-rw-r--r-- 1 alenkafrim staff 9.9K 22 Nov 06:02 feather.py
-rw-r--r-- 1 alenkafrim staff 14K 10 Jun 10:00 filesystem.py
-rw-r--r-- 1 alenkafrim staff 1.8K 22 Nov 06:02 flight.py
-rw-r--r-- 1 alenkafrim staff 14K 27 Oct 14:04 fs.py
-rw-r--r-- 1 alenkafrim staff 20K 27 Oct 14:04 gandiva.pyx
-rw-r--r-- 1 alenkafrim staff 7.4K 11 Apr 2022 hdfs.py
drwxr-xr-x 3 alenkafrim staff 96B 17 Nov 11:37 include
drwxr-xr-x 15 alenkafrim staff 480B 22 Nov 10:02 includes
drwxr-xr-x 3 alenkafrim staff 96B 21 Nov 14:54 interchange
-rw-r--r-- 1 alenkafrim staff 68K 27 Oct 14:04 io.pxi
-rw-r--r-- 1 alenkafrim staff 33K 22 Nov 06:02 ipc.pxi
-rw-r--r-- 1 alenkafrim staff 9.7K 22 Nov 06:02 ipc.py
-rw-r--r-- 1 alenkafrim staff 858B 3 Nov 2021 json.py
-rw-r--r-- 1 alenkafrim staff 9.4K 3 Nov 2021 jvm.py
-rwxr-xr-x 1 alenkafrim staff 7.0M 21 Nov 12:35 lib.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 14K 22 Nov 10:02 lib.pxd
-rw-r--r-- 1 alenkafrim staff 4.6K 21 Apr 2022 lib.pyx
-rw-r--r-- 1 alenkafrim staff 19K 21 Nov 12:35 lib_api.h
-rwxr-xr-x 1 alenkafrim staff 12M 21 Nov 12:35 libarrow_python.1100.0.0.dylib
-rwxr-xr-x 1 alenkafrim staff 12M 21 Nov 12:35 libarrow_python.1100.dylib
-rw-r--r-- 1 alenkafrim staff 30M 21 Nov 12:35 libarrow_python.a
-rwxr-xr-x 1 alenkafrim staff 12M 21 Nov 12:35 libarrow_python.dylib
-rwxr-xr-x 1 alenkafrim staff 458K 21 Nov 12:35 libarrow_python_flight.1100.0.0.dylib
-rwxr-xr-x 1 alenkafrim staff 458K 21 Nov 12:35 libarrow_python_flight.1100.dylib
-rw-r--r-- 1 alenkafrim staff 1.2M 21 Nov 12:35 libarrow_python_flight.a
-rwxr-xr-x 1 alenkafrim staff 458K 21 Nov 12:35 libarrow_python_flight.dylib
-rw-r--r-- 1 alenkafrim staff 8.0K 27 Oct 14:04 memory.pxi
-rw-r--r-- 1 alenkafrim staff 12K 22 Nov 06:02 orc.py
-rw-r--r-- 1 alenkafrim staff 7.8K 17 Nov 10:51 pandas-shim.pxi
-rw-r--r-- 1 alenkafrim staff 43K 22 Nov 06:02 pandas_compat.py
drwxr-xr-x 6 alenkafrim staff 192B 22 Nov 06:02 parquet
-rw-r--r-- 1 alenkafrim staff 6.3K 22 Nov 06:02 plasma.py
-rw-r--r-- 1 alenkafrim staff 13K 22 Nov 10:02 public-api.pxi
-rw-r--r-- 1 alenkafrim staff 29K 22 Nov 10:02 scalar.pxi
-rw-r--r-- 1 alenkafrim staff 19K 29 Jun 09:38 serialization.pxi
-rw-r--r-- 1 alenkafrim staff 18K 3 Nov 2021 serialization.py
drwxr-xr-x 8 alenkafrim staff 256B 22 Nov 10:02 src
-rw-r--r-- 1 alenkafrim staff 873B 27 Oct 14:04 substrait.py
-rw-r--r-- 1 alenkafrim staff 158K 22 Nov 06:02 table.pxi
-rw-r--r-- 1 alenkafrim staff 38K 19 Apr 2022 tensor.pxi
drwxr-xr-x 3 alenkafrim staff 96B 27 Oct 14:04 tensorflow
drwxr-xr-x 62 alenkafrim staff 1.9K 22 Nov 10:02 tests
-rw-r--r-- 1 alenkafrim staff 90K 22 Nov 06:02 types.pxi
-rw-r--r-- 1 alenkafrim staff 10K 11 Apr 2022 types.py
-rw-r--r-- 1 alenkafrim staff 4.6K 22 Nov 06:02 util.py
drwxr-xr-x 6 alenkafrim staff 192B 17 Nov 11:07 vendored

If I remember correctly, if I didn't use os.path.abspath() the .. didn't have an effect.
@jorisvandenbosschejorisvandenbosscheNov 22, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For me it seems to work also with a ".." in it:
In [9]: os.path.exists("/home/joris/scipy/repos/arrow/python/pyarrow/tests/../include/arrow/python/")
Out[9]: True

@jorisvandenbosschejorisvandenbosscheNov 22, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But so about @kou's original comment, I think doing os.path.dirname(source) also works because if source is already a directory, dirname gives the parent directory.

Now personally I find .. more explicit to read that it goes one level up.
@AlenkaFAlenkaFNov 22, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, sorry for my confusion.

Can change to pyarrow_dir = os.path.dirname(source), @kou do you agree?
@AlenkaFAlenkaFNov 22, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, now I see Joris you already made a suggestion. Will commit that 👍
@kou
kou
commented
Nov 22, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g python -g conda
@github-actions
github-actionsBot
commented
Nov 22, 2022

Copy link


Copy Markdown

Revision: acf8754
Submitted crossbow builds: ursacomputing/crossbow @ actions-d4595c1625
TaskStatus
conda-cleanAzure
conda-linux-gcc-py310-arm64Azure
conda-linux-gcc-py310-cpuAzure
conda-linux-gcc-py310-cudaAzure
conda-linux-gcc-py310-ppc64leAzure
conda-linux-gcc-py37-arm64Azure
conda-linux-gcc-py37-cpu-r40Azure
conda-linux-gcc-py37-cpu-r41Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py37-ppc64leAzure
conda-linux-gcc-py38-arm64Azure
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py38-ppc64leAzure
conda-linux-gcc-py39-arm64Azure
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-linux-gcc-py39-ppc64leAzure
conda-osx-arm64-clang-py310Azure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py310Azure
conda-osx-clang-py37-r40Azure
conda-osx-clang-py37-r41Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2019-py310Azure
conda-win-vs2019-py37-r40Azure
conda-win-vs2019-py37-r41Azure
conda-win-vs2019-py38Azure
conda-win-vs2019-py39Azure
test-conda-python-3.10Github Actions
test-conda-python-3.11Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-hdfs-2.9.2Github Actions
test-conda-python-3.7-hdfs-3.2.1Github Actions
test-conda-python-3.7-pandas-0.24Github Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-spark-v3.1.2Github Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-v3.2.0Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-dask-latestGithub Actions
test-conda-python-3.9-dask-upstream_develGithub Actions
test-conda-python-3.9-pandas-upstream_develGithub Actions
test-conda-python-3.9-spark-masterGithub Actions
test-cuda-pythonGithub Actions
test-debian-11-python-3Azure
test-fedora-35-python-3Azure
test-ubuntu-20.04-python-3Azure
jorisvandenbossche
jorisvandenbossche
reviewed
Nov 22, 2022
View reviewed changes
Comment threadpython/pyarrow/tests/test_cpp_internals.py
Outdated

@AlenkaF@jorisvandenbossche
Update python/pyarrow/tests/test_cpp_internals.py
4669e94
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
@jorisvandenbosschejorisvandenbossche
mentioned this pull request
Nov 22, 2022
 Merged

3 tasks
@kou
kou
merged commit 1e9eb61
into
apache:masterNov 22, 2022
@AlenkaF
AlenkaF
deleted the

ARROW-18340

branch
November 23, 2022 08:41
@ursabot
ursabot
commented
Nov 24, 2022

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = 59f99d2 and contender = 1e9eb61. 1e9eb61 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2

[Finished ⬇️0.34% ⬆️0.54%] test-mac-arm

[Finished ⬇️0.27% ⬆️0.0%] ursa-i9-9960x

[Finished ⬇️0.14% ⬆️0.35%] ursa-thinkcentre-m75q

Buildkite builds:

[Finished] 1e9eb615 ec2-t3-xlarge-us-east-2

[Finished] 1e9eb615 test-mac-arm

[Finished] 1e9eb615 ursa-i9-9960x

[Finished] 1e9eb615 ursa-thinkcentre-m75q

[Finished] 59f99d2c ec2-t3-xlarge-us-east-2

[Finished] 59f99d2c test-mac-arm

[Finished] 59f99d2c ursa-i9-9960x

[Finished] 59f99d2c ursa-thinkcentre-m75q

Supported benchmarks:

ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True

test-mac-arm: Supported benchmark langs: C++, Python, R

ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript

ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
@asfimportasfimport
mentioned this pull request
Nov 24, 2022

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@jorisvandenbosschejorisvandenbosschejorisvandenbossche left review comments

Assignees


No one assigned


Labels

Component: Python

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@AlenkaF@jorisvandenbossche@kou@ursabot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-18340: [Python] PyArrow C++ header files no longer always included in installed pyarrow - #14656
Merged
kou merged 7 commits into
apache:masterfrom 
AlenkaF:ARROW-18340
Nov 22, 2022
Merged
ARROW-18340: [Python] PyArrow C++ header files no longer always included in installed pyarrow#14656
kou merged 7 commits into
apache:masterfrom 
AlenkaF:ARROW-18340
Conversation
@AlenkaF
@AlenkaFAlenkaF
commented
Nov 16, 2022

Copy link


Copy Markdown

Member
This PR adds code to _run_cmake to always copy PyArrow C++ header files to pyarrow/iclude.
@github-actions
github-actionsBot
commented
Nov 16, 2022

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-18340
@github-actions
github-actionsBot
commented
Nov 16, 2022

Copy link


Copy Markdown

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.
jorisvandenbossche
jorisvandenbossche
reviewed
Nov 16, 2022
View reviewed changes
Comment threadpython/setup.py
Outdated

@jorisvandenbossche
jorisvandenbossche
commented
Nov 16, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g wheel
@github-actions
github-actionsBot
commented
Nov 16, 2022

Copy link


Copy Markdown

Revision: d5f953f
Submitted crossbow builds: ursacomputing/crossbow @ actions-1c3aa4b08e
TaskStatus
wheel-macos-big-sur-cp310-arm64Github Actions
wheel-macos-big-sur-cp311-arm64Github Actions
wheel-macos-big-sur-cp38-arm64Github Actions
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-mojave-cp310-amd64Github Actions
wheel-macos-mojave-cp311-amd64Github Actions
wheel-macos-mojave-cp37-amd64Github Actions
wheel-macos-mojave-cp38-amd64Github Actions
wheel-macos-mojave-cp39-amd64Github Actions
wheel-manylinux2014-cp310-amd64Github Actions
wheel-manylinux2014-cp310-arm64Travis CI
wheel-manylinux2014-cp311-amd64Github Actions
wheel-manylinux2014-cp311-arm64Travis CI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64Travis CI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64Travis CI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64Travis CI
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
kou
kou
reviewed
Nov 16, 2022
View reviewed changes
Comment threadpython/setup.py
Outdated

@kou
kou
commented
Nov 16, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g python -g conda
@github-actions
github-actionsBot
commented
Nov 16, 2022

Copy link


Copy Markdown

Revision: d5f953f
Submitted crossbow builds: ursacomputing/crossbow @ actions-9eb5a3c4ca
TaskStatus
conda-cleanAzure
conda-linux-gcc-py310-arm64Azure
conda-linux-gcc-py310-cpuAzure
conda-linux-gcc-py310-cudaAzure
conda-linux-gcc-py310-ppc64leAzure
conda-linux-gcc-py37-arm64Azure
conda-linux-gcc-py37-cpu-r40Azure
conda-linux-gcc-py37-cpu-r41Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py37-ppc64leAzure
conda-linux-gcc-py38-arm64Azure
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py38-ppc64leAzure
conda-linux-gcc-py39-arm64Azure
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-linux-gcc-py39-ppc64leAzure
conda-osx-arm64-clang-py310Azure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py310Azure
conda-osx-clang-py37-r40Azure
conda-osx-clang-py37-r41Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2019-py310Azure
conda-win-vs2019-py37-r40Azure
conda-win-vs2019-py37-r41Azure
conda-win-vs2019-py38Azure
conda-win-vs2019-py39Azure
test-conda-python-3.10Github Actions
test-conda-python-3.11Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-hdfs-2.9.2Github Actions
test-conda-python-3.7-hdfs-3.2.1Github Actions
test-conda-python-3.7-pandas-0.24Github Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-spark-v3.1.2Github Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-v3.2.0Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-dask-latestGithub Actions
test-conda-python-3.9-dask-upstream_develGithub Actions
test-conda-python-3.9-pandas-upstream_develGithub Actions
test-conda-python-3.9-spark-masterGithub Actions
test-cuda-pythonGithub Actions
test-debian-11-python-3Azure
test-fedora-35-python-3Azure
test-ubuntu-20.04-python-3Azure
@kou
kou
commented
Nov 16, 2022

Copy link


Copy Markdown

Member
Can we add a test to detect this problem?
@AlenkaF
AlenkaF
requested a review
from kouNovember 22, 2022 07:39
kou
kou
approved these changes
Nov 22, 2022
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Comment threadpython/pyarrow/tests/test_cpp_internals.py
Outdated

 # Arrow C++ and PyArrow C++ header files together

 source = os.path.dirname(os.path.abspath(__file__))
 pyarrow_dir = os.path.abspath(pjoin(source, '..'))
@koukouNov 22, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

os.path.dirname(source)?
@AlenkaFAlenkaFNov 22, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to go from test folder to the parent pyarrow folder to test if the include folder is present there.
@koukouNov 22, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry... I'm not sure why os.path.abspath() is needed for it.

Could you show the followings in your environment?
  • The source value
  • ls -lah ${source} (${source} is the source value)
  • The pyarrow_dir value
  • ls -lah ${pyarrow_dir} (${pyarrow_dir} is the pyarrow_dir value)
@AlenkaFAlenkaFNov 22, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure! Here it is:
(Pdb) print(source)
/Users/alenkafrim/repos/arrow/python/pyarrow/tests
(Pdb) pyarrow_dir
'/Users/alenkafrim/repos/arrow/python/pyarrow'

ls -lah ouput /test
(pyarrow-dev-9) (base) alenkafrim@Alenkas-MacBook-Pro python % ls -lah /Users/alenkafrim/repos/arrow/python/pyarrow/tests
total 3240
drwxr-xr-x 62 alenkafrim staff 1.9K 22 Nov 10:02 .
drwxr-xr-x 109 alenkafrim staff 3.4K 22 Nov 10:02 ..
-rw-r--r-- 1 alenkafrim staff 0B 5 May 2022 __init__.py
drwxr-xr-x 5 alenkafrim staff 160B 22 Nov 10:03 __pycache__
-rw-r--r-- 1 alenkafrim staff 1.3K 2 Jun 13:33 arrow_16597.py
-rw-r--r-- 1 alenkafrim staff 1.1K 3 Nov 2021 arrow_7980.py
-rw-r--r-- 1 alenkafrim staff 2.0K 11 Apr 2022 bound_function_visit_strings.pyx
-rw-r--r-- 1 alenkafrim staff 6.1K 29 Jun 09:38 conftest.py
drwxr-xr-x 5 alenkafrim staff 160B 27 Oct 12:17 data
-rw-r--r-- 1 alenkafrim staff 961B 3 Nov 2021 deserialize_buffer.py
drwxr-xr-x 3 alenkafrim staff 96B 21 Nov 14:54 interchange
-rw-r--r-- 1 alenkafrim staff 5.0K 3 Nov 2021 pandas_examples.py
-rw-r--r-- 1 alenkafrim staff 1.4K 3 Nov 2021 pandas_threaded_import.py
drwxr-xr-x 16 alenkafrim staff 512B 22 Nov 06:02 parquet
-rw-r--r-- 1 alenkafrim staff 1.9K 11 Apr 2022 pyarrow_cython_example.pyx
-rw-r--r-- 1 alenkafrim staff 953B 19 Apr 2022 read_record_batch.py
-rw-r--r-- 1 alenkafrim staff 13K 22 Apr 2022 strategies.py
-rw-r--r-- 1 alenkafrim staff 1.4K 3 Nov 2021 test_adhoc_memory_leak.py
-rw-r--r-- 1 alenkafrim staff 106K 22 Nov 06:02 test_array.py
-rw-r--r-- 1 alenkafrim staff 2.1K 3 Nov 2021 test_builder.py
-rw-r--r-- 1 alenkafrim staff 13K 19 Apr 2022 test_cffi.py
-rw-r--r-- 1 alenkafrim staff 108K 22 Nov 10:02 test_compute.py
-rw-r--r-- 1 alenkafrim staff 70K 27 Oct 14:04 test_convert_builtin.py
-rw-r--r-- 1 alenkafrim staff 1.8K 22 Nov 10:03 test_cpp_internals.py
-rw-r--r-- 1 alenkafrim staff 71K 22 Nov 06:02 test_csv.py
-rw-r--r-- 1 alenkafrim staff 27K 3 Nov 2021 test_cuda.py
-rw-r--r-- 1 alenkafrim staff 8.5K 3 Nov 2021 test_cuda_numba_interop.py
-rw-r--r-- 1 alenkafrim staff 6.5K 22 Nov 06:02 test_cython.py
-rw-r--r-- 1 alenkafrim staff 175K 22 Nov 06:02 test_dataset.py
-rw-r--r-- 1 alenkafrim staff 891B 3 Nov 2021 test_deprecations.py
-rw-r--r-- 1 alenkafrim staff 9.2K 27 Oct 14:04 test_exec_plan.py
-rw-r--r-- 1 alenkafrim staff 33K 22 Nov 06:02 test_extension_type.py
-rw-r--r-- 1 alenkafrim staff 24K 22 Nov 06:02 test_feather.py
-rw-r--r-- 1 alenkafrim staff 2.3K 19 Apr 2022 test_filesystem.py
-rw-r--r-- 1 alenkafrim staff 79K 22 Nov 06:02 test_flight.py
-rw-r--r-- 1 alenkafrim staff 55K 27 Oct 14:04 test_fs.py
-rw-r--r-- 1 alenkafrim staff 15K 27 Oct 14:04 test_gandiva.py
-rw-r--r-- 1 alenkafrim staff 42K 22 Nov 06:02 test_gdb.py
-rw-r--r-- 1 alenkafrim staff 13K 11 Apr 2022 test_hdfs.py
-rw-r--r-- 1 alenkafrim staff 56K 27 Oct 14:04 test_io.py
-rw-r--r-- 1 alenkafrim staff 36K 22 Nov 06:02 test_ipc.py
-rw-r--r-- 1 alenkafrim staff 12K 22 Nov 06:02 test_json.py
-rw-r--r-- 1 alenkafrim staff 15K 3 Nov 2021 test_jvm.py
-rw-r--r-- 1 alenkafrim staff 7.6K 11 Jul 08:19 test_memory.py
-rw-r--r-- 1 alenkafrim staff 5.8K 19 Apr 2022 test_misc.py
-rw-r--r-- 1 alenkafrim staff 19K 22 Nov 10:02 test_orc.py
-rw-r--r-- 1 alenkafrim staff 159K 22 Nov 06:02 test_pandas.py
-rw-r--r-- 1 alenkafrim staff 44K 22 Nov 06:02 test_plasma.py
-rw-r--r-- 1 alenkafrim staff 3.4K 3 Nov 2021 test_plasma_tf_op.py
-rw-r--r-- 1 alenkafrim staff 20K 22 Nov 10:02 test_scalars.py
-rw-r--r-- 1 alenkafrim staff 21K 22 Nov 06:02 test_schema.py
-rw-r--r-- 1 alenkafrim staff 41K 11 Apr 2022 test_serialization.py
-rw-r--r-- 1 alenkafrim staff 1.6K 11 Apr 2022 test_serialization_deprecated.py
-rw-r--r-- 1 alenkafrim staff 17K 3 Nov 2021 test_sparse_tensor.py
-rw-r--r-- 1 alenkafrim staff 1.7K 3 Nov 2021 test_strategies.py
-rw-r--r-- 1 alenkafrim staff 8.4K 22 Nov 06:02 test_substrait.py
-rw-r--r-- 1 alenkafrim staff 65K 27 Oct 14:04 test_table.py
-rw-r--r-- 1 alenkafrim staff 6.1K 11 Apr 2022 test_tensor.py
-rw-r--r-- 1 alenkafrim staff 33K 22 Nov 10:02 test_types.py
-rw-r--r-- 1 alenkafrim staff 17K 29 Jun 09:38 test_udf.py
-rw-r--r-- 1 alenkafrim staff 1.7K 11 Apr 2022 test_util.py
-rw-r--r-- 1 alenkafrim staff 13K 6 May 2022 util.py

ls -lah ouput pyarrow/
(pyarrow-dev-9) (base) alenkafrim@Alenkas-MacBook-Pro python % ls -lah /Users/alenkafrim/repos/arrow/python/pyarrow total 185744
drwxr-xr-x 109 alenkafrim staff 3.4K 22 Nov 10:02 .
drwxr-xr-x 36 alenkafrim staff 1.1K 22 Nov 10:02 ..
-rw-r--r--@ 1 alenkafrim staff 6.0K 17 Nov 12:41 .DS_Store
-rw-r--r-- 1 alenkafrim staff 2.1K 3 Nov 2021 __init__.pxd
-rw-r--r-- 1 alenkafrim staff 19K 22 Nov 08:17 __init__.py
drwxr-xr-x 20 alenkafrim staff 640B 22 Nov 10:03 __pycache__
-rwxr-xr-x 1 alenkafrim staff 3.8M 21 Nov 12:35 _compute.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 1.7K 6 May 2022 _compute.pxd
-rw-r--r-- 1 alenkafrim staff 83K 22 Nov 10:02 _compute.pyx
-rw-r--r-- 1 alenkafrim staff 1.7K 30 May 12:10 _compute_docstrings.py
-rwxr-xr-x 1 alenkafrim staff 970K 21 Nov 12:35 _csv.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 1.6K 11 Apr 2022 _csv.pxd
-rw-r--r-- 1 alenkafrim staff 51K 22 Nov 06:02 _csv.pyx
-rw-r--r-- 1 alenkafrim staff 1.9K 3 Nov 2021 _cuda.pxd
-rw-r--r-- 1 alenkafrim staff 34K 11 Apr 2022 _cuda.pyx
-rwxr-xr-x 1 alenkafrim staff 3.0M 21 Nov 12:35 _dataset.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 4.3K 27 Oct 14:04 _dataset.pxd
-rw-r--r-- 1 alenkafrim staff 100K 22 Nov 06:02 _dataset.pyx
-rw-r--r-- 1 alenkafrim staff 1.3K 11 Apr 2022 _dataset_orc.pyx
-rwxr-xr-x 1 alenkafrim staff 934K 21 Nov 12:35 _dataset_parquet.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 30K 29 Jun 09:38 _dataset_parquet.pyx
-rwxr-xr-x 1 alenkafrim staff 1.0M 21 Nov 12:35 _exec_plan.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 16K 22 Nov 06:02 _exec_plan.pyx
-rwxr-xr-x 1 alenkafrim staff 265K 21 Nov 12:35 _feather.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 3.7K 11 Apr 2022 _feather.pyx
-rw-r--r-- 1 alenkafrim staff 105K 22 Nov 06:02 _flight.pyx
-rwxr-xr-x 1 alenkafrim staff 1.9M 21 Nov 12:35 _fs.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 2.4K 3 Nov 2021 _fs.pxd
-rw-r--r-- 1 alenkafrim staff 51K 22 Nov 06:02 _fs.pyx
-rw-r--r-- 1 alenkafrim staff 8.0K 29 Jun 09:38 _gcsfs.pyx
-rw-r--r-- 1 alenkafrim staff 186B 21 Nov 12:35 _generated_version.py
-rw-r--r-- 1 alenkafrim staff 5.6K 29 Jun 09:38 _hdfs.pyx
-rwxr-xr-x 1 alenkafrim staff 483K 21 Nov 12:35 _hdfsio.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 13K 11 Apr 2022 _hdfsio.pyx
-rwxr-xr-x 1 alenkafrim staff 199K 21 Nov 12:35 _json.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 8.5K 19 Apr 2022 _json.pyx
-rw-r--r-- 1 alenkafrim staff 5.6K 11 Apr 2022 _orc.pxd
-rw-r--r-- 1 alenkafrim staff 15K 11 Apr 2022 _orc.pyx
-rwxr-xr-x 1 alenkafrim staff 1.6M 21 Nov 12:35 _parquet.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 25K 29 Jun 09:38 _parquet.pxd
-rw-r--r-- 1 alenkafrim staff 57K 27 Oct 14:04 _parquet.pyx
-rw-r--r-- 1 alenkafrim staff 5.9K 19 Apr 2022 _parquet_encryption.pxd
-rw-r--r-- 1 alenkafrim staff 17K 19 Apr 2022 _parquet_encryption.pyx
-rw-r--r-- 1 alenkafrim staff 30K 22 Nov 06:02 _plasma.pyx
-rwxr-xr-x 1 alenkafrim staff 207K 21 Nov 12:35 _pyarrow_cpp_tests.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 1.2K 22 Nov 06:02 _pyarrow_cpp_tests.pxd
-rw-r--r-- 1 alenkafrim staff 1.7K 22 Nov 06:02 _pyarrow_cpp_tests.pyx
-rwxr-xr-x 1 alenkafrim staff 388K 21 Nov 12:35 _s3fs.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 16K 22 Nov 06:02 _s3fs.pyx
-rw-r--r-- 1 alenkafrim staff 6.3K 22 Nov 06:02 _substrait.pyx
-rw-r--r-- 1 alenkafrim staff 91K 22 Nov 06:02 array.pxi
-rw-r--r-- 1 alenkafrim staff 869B 3 Nov 2021 benchmark.pxi
-rw-r--r-- 1 alenkafrim staff 856B 3 Nov 2021 benchmark.py
-rw-r--r-- 1 alenkafrim staff 2.6K 3 Nov 2021 builder.pxi
-rw-r--r-- 1 alenkafrim staff 2.1K 3 Nov 2021 cffi.py
-rw-r--r-- 1 alenkafrim staff 2.0K 11 Apr 2022 compat.pxi
-rw-r--r-- 1 alenkafrim staff 21K 22 Nov 10:02 compute.py
-rw-r--r-- 1 alenkafrim staff 2.7K 11 Apr 2022 config.pxi
-rw-r--r-- 1 alenkafrim staff 6.2K 27 Oct 14:04 conftest.py
-rw-r--r-- 1 alenkafrim staff 974B 11 Apr 2022 csv.py
-rw-r--r-- 1 alenkafrim staff 1.1K 3 Nov 2021 cuda.py
-rw-r--r-- 1 alenkafrim staff 38K 27 Oct 14:04 dataset.py
-rw-r--r-- 1 alenkafrim staff 8.3K 22 Nov 06:02 error.pxi
-rw-r--r-- 1 alenkafrim staff 9.9K 22 Nov 06:02 feather.py
-rw-r--r-- 1 alenkafrim staff 14K 10 Jun 10:00 filesystem.py
-rw-r--r-- 1 alenkafrim staff 1.8K 22 Nov 06:02 flight.py
-rw-r--r-- 1 alenkafrim staff 14K 27 Oct 14:04 fs.py
-rw-r--r-- 1 alenkafrim staff 20K 27 Oct 14:04 gandiva.pyx
-rw-r--r-- 1 alenkafrim staff 7.4K 11 Apr 2022 hdfs.py
drwxr-xr-x 3 alenkafrim staff 96B 17 Nov 11:37 include
drwxr-xr-x 15 alenkafrim staff 480B 22 Nov 10:02 includes
drwxr-xr-x 3 alenkafrim staff 96B 21 Nov 14:54 interchange
-rw-r--r-- 1 alenkafrim staff 68K 27 Oct 14:04 io.pxi
-rw-r--r-- 1 alenkafrim staff 33K 22 Nov 06:02 ipc.pxi
-rw-r--r-- 1 alenkafrim staff 9.7K 22 Nov 06:02 ipc.py
-rw-r--r-- 1 alenkafrim staff 858B 3 Nov 2021 json.py
-rw-r--r-- 1 alenkafrim staff 9.4K 3 Nov 2021 jvm.py
-rwxr-xr-x 1 alenkafrim staff 7.0M 21 Nov 12:35 lib.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 14K 22 Nov 10:02 lib.pxd
-rw-r--r-- 1 alenkafrim staff 4.6K 21 Apr 2022 lib.pyx
-rw-r--r-- 1 alenkafrim staff 19K 21 Nov 12:35 lib_api.h
-rwxr-xr-x 1 alenkafrim staff 12M 21 Nov 12:35 libarrow_python.1100.0.0.dylib
-rwxr-xr-x 1 alenkafrim staff 12M 21 Nov 12:35 libarrow_python.1100.dylib
-rw-r--r-- 1 alenkafrim staff 30M 21 Nov 12:35 libarrow_python.a
-rwxr-xr-x 1 alenkafrim staff 12M 21 Nov 12:35 libarrow_python.dylib
-rwxr-xr-x 1 alenkafrim staff 458K 21 Nov 12:35 libarrow_python_flight.1100.0.0.dylib
-rwxr-xr-x 1 alenkafrim staff 458K 21 Nov 12:35 libarrow_python_flight.1100.dylib
-rw-r--r-- 1 alenkafrim staff 1.2M 21 Nov 12:35 libarrow_python_flight.a
-rwxr-xr-x 1 alenkafrim staff 458K 21 Nov 12:35 libarrow_python_flight.dylib
-rw-r--r-- 1 alenkafrim staff 8.0K 27 Oct 14:04 memory.pxi
-rw-r--r-- 1 alenkafrim staff 12K 22 Nov 06:02 orc.py
-rw-r--r-- 1 alenkafrim staff 7.8K 17 Nov 10:51 pandas-shim.pxi
-rw-r--r-- 1 alenkafrim staff 43K 22 Nov 06:02 pandas_compat.py
drwxr-xr-x 6 alenkafrim staff 192B 22 Nov 06:02 parquet
-rw-r--r-- 1 alenkafrim staff 6.3K 22 Nov 06:02 plasma.py
-rw-r--r-- 1 alenkafrim staff 13K 22 Nov 10:02 public-api.pxi
-rw-r--r-- 1 alenkafrim staff 29K 22 Nov 10:02 scalar.pxi
-rw-r--r-- 1 alenkafrim staff 19K 29 Jun 09:38 serialization.pxi
-rw-r--r-- 1 alenkafrim staff 18K 3 Nov 2021 serialization.py
drwxr-xr-x 8 alenkafrim staff 256B 22 Nov 10:02 src
-rw-r--r-- 1 alenkafrim staff 873B 27 Oct 14:04 substrait.py
-rw-r--r-- 1 alenkafrim staff 158K 22 Nov 06:02 table.pxi
-rw-r--r-- 1 alenkafrim staff 38K 19 Apr 2022 tensor.pxi
drwxr-xr-x 3 alenkafrim staff 96B 27 Oct 14:04 tensorflow
drwxr-xr-x 62 alenkafrim staff 1.9K 22 Nov 10:02 tests
-rw-r--r-- 1 alenkafrim staff 90K 22 Nov 06:02 types.pxi
-rw-r--r-- 1 alenkafrim staff 10K 11 Apr 2022 types.py
-rw-r--r-- 1 alenkafrim staff 4.6K 22 Nov 06:02 util.py
drwxr-xr-x 6 alenkafrim staff 192B 17 Nov 11:07 vendored

If I remember correctly, if I didn't use os.path.abspath() the .. didn't have an effect.
@jorisvandenbosschejorisvandenbosscheNov 22, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For me it seems to work also with a ".." in it:
In [9]: os.path.exists("/home/joris/scipy/repos/arrow/python/pyarrow/tests/../include/arrow/python/")
Out[9]: True

@jorisvandenbosschejorisvandenbosscheNov 22, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But so about @kou's original comment, I think doing os.path.dirname(source) also works because if source is already a directory, dirname gives the parent directory.

Now personally I find .. more explicit to read that it goes one level up.
@AlenkaFAlenkaFNov 22, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, sorry for my confusion.

Can change to pyarrow_dir = os.path.dirname(source), @kou do you agree?
@AlenkaFAlenkaFNov 22, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, now I see Joris you already made a suggestion. Will commit that 👍
@kou
kou
commented
Nov 22, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g python -g conda
@github-actions
github-actionsBot
commented
Nov 22, 2022

Copy link


Copy Markdown

Revision: acf8754
Submitted crossbow builds: ursacomputing/crossbow @ actions-d4595c1625
TaskStatus
conda-cleanAzure
conda-linux-gcc-py310-arm64Azure
conda-linux-gcc-py310-cpuAzure
conda-linux-gcc-py310-cudaAzure
conda-linux-gcc-py310-ppc64leAzure
conda-linux-gcc-py37-arm64Azure
conda-linux-gcc-py37-cpu-r40Azure
conda-linux-gcc-py37-cpu-r41Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py37-ppc64leAzure
conda-linux-gcc-py38-arm64Azure
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py38-ppc64leAzure
conda-linux-gcc-py39-arm64Azure
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-linux-gcc-py39-ppc64leAzure
conda-osx-arm64-clang-py310Azure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py310Azure
conda-osx-clang-py37-r40Azure
conda-osx-clang-py37-r41Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2019-py310Azure
conda-win-vs2019-py37-r40Azure
conda-win-vs2019-py37-r41Azure
conda-win-vs2019-py38Azure
conda-win-vs2019-py39Azure
test-conda-python-3.10Github Actions
test-conda-python-3.11Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-hdfs-2.9.2Github Actions
test-conda-python-3.7-hdfs-3.2.1Github Actions
test-conda-python-3.7-pandas-0.24Github Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-spark-v3.1.2Github Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-v3.2.0Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-dask-latestGithub Actions
test-conda-python-3.9-dask-upstream_develGithub Actions
test-conda-python-3.9-pandas-upstream_develGithub Actions
test-conda-python-3.9-spark-masterGithub Actions
test-cuda-pythonGithub Actions
test-debian-11-python-3Azure
test-fedora-35-python-3Azure
test-ubuntu-20.04-python-3Azure
jorisvandenbossche
jorisvandenbossche
reviewed
Nov 22, 2022
View reviewed changes
Comment threadpython/pyarrow/tests/test_cpp_internals.py
Outdated

@AlenkaF@jorisvandenbossche
Update python/pyarrow/tests/test_cpp_internals.py
4669e94
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
@jorisvandenbosschejorisvandenbossche
mentioned this pull request
Nov 22, 2022
 Merged

3 tasks
@kou
kou
merged commit 1e9eb61
into
apache:masterNov 22, 2022
@AlenkaF
AlenkaF
deleted the

ARROW-18340

branch
November 23, 2022 08:41
@ursabot
ursabot
commented
Nov 24, 2022

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = 59f99d2 and contender = 1e9eb61. 1e9eb61 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2

[Finished ⬇️0.34% ⬆️0.54%] test-mac-arm

[Finished ⬇️0.27% ⬆️0.0%] ursa-i9-9960x

[Finished ⬇️0.14% ⬆️0.35%] ursa-thinkcentre-m75q

Buildkite builds:

[Finished] 1e9eb615 ec2-t3-xlarge-us-east-2

[Finished] 1e9eb615 test-mac-arm

[Finished] 1e9eb615 ursa-i9-9960x

[Finished] 1e9eb615 ursa-thinkcentre-m75q

[Finished] 59f99d2c ec2-t3-xlarge-us-east-2

[Finished] 59f99d2c test-mac-arm

[Finished] 59f99d2c ursa-i9-9960x

[Finished] 59f99d2c ursa-thinkcentre-m75q

Supported benchmarks:

ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True

test-mac-arm: Supported benchmark langs: C++, Python, R

ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript

ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
@asfimportasfimport
mentioned this pull request
Nov 24, 2022

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@jorisvandenbosschejorisvandenbosschejorisvandenbossche left review comments

Assignees


No one assigned


Labels

Component: Python

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@AlenkaF@jorisvandenbossche@kou@ursabot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-18340: [Python] PyArrow C++ header files no longer always included in installed pyarrow - #14656
Merged
kou merged 7 commits into
apache:masterfrom 
AlenkaF:ARROW-18340
Nov 22, 2022
Merged
ARROW-18340: [Python] PyArrow C++ header files no longer always included in installed pyarrow#14656
kou merged 7 commits into
apache:masterfrom 
AlenkaF:ARROW-18340
Conversation
@AlenkaF
@AlenkaFAlenkaF
commented
Nov 16, 2022

Copy link


Copy Markdown

Member
This PR adds code to _run_cmake to always copy PyArrow C++ header files to pyarrow/iclude.
@github-actions
github-actionsBot
commented
Nov 16, 2022

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-18340
@github-actions
github-actionsBot
commented
Nov 16, 2022

Copy link


Copy Markdown

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.
jorisvandenbossche
jorisvandenbossche
reviewed
Nov 16, 2022
View reviewed changes
Comment threadpython/setup.py
Outdated

@jorisvandenbossche
jorisvandenbossche
commented
Nov 16, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g wheel
@github-actions
github-actionsBot
commented
Nov 16, 2022

Copy link


Copy Markdown

Revision: d5f953f
Submitted crossbow builds: ursacomputing/crossbow @ actions-1c3aa4b08e
TaskStatus
wheel-macos-big-sur-cp310-arm64Github Actions
wheel-macos-big-sur-cp311-arm64Github Actions
wheel-macos-big-sur-cp38-arm64Github Actions
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-mojave-cp310-amd64Github Actions
wheel-macos-mojave-cp311-amd64Github Actions
wheel-macos-mojave-cp37-amd64Github Actions
wheel-macos-mojave-cp38-amd64Github Actions
wheel-macos-mojave-cp39-amd64Github Actions
wheel-manylinux2014-cp310-amd64Github Actions
wheel-manylinux2014-cp310-arm64Travis CI
wheel-manylinux2014-cp311-amd64Github Actions
wheel-manylinux2014-cp311-arm64Travis CI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64Travis CI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64Travis CI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64Travis CI
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
kou
kou
reviewed
Nov 16, 2022
View reviewed changes
Comment threadpython/setup.py
Outdated

@kou
kou
commented
Nov 16, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g python -g conda
@github-actions
github-actionsBot
commented
Nov 16, 2022

Copy link


Copy Markdown

Revision: d5f953f
Submitted crossbow builds: ursacomputing/crossbow @ actions-9eb5a3c4ca
TaskStatus
conda-cleanAzure
conda-linux-gcc-py310-arm64Azure
conda-linux-gcc-py310-cpuAzure
conda-linux-gcc-py310-cudaAzure
conda-linux-gcc-py310-ppc64leAzure
conda-linux-gcc-py37-arm64Azure
conda-linux-gcc-py37-cpu-r40Azure
conda-linux-gcc-py37-cpu-r41Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py37-ppc64leAzure
conda-linux-gcc-py38-arm64Azure
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py38-ppc64leAzure
conda-linux-gcc-py39-arm64Azure
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-linux-gcc-py39-ppc64leAzure
conda-osx-arm64-clang-py310Azure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py310Azure
conda-osx-clang-py37-r40Azure
conda-osx-clang-py37-r41Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2019-py310Azure
conda-win-vs2019-py37-r40Azure
conda-win-vs2019-py37-r41Azure
conda-win-vs2019-py38Azure
conda-win-vs2019-py39Azure
test-conda-python-3.10Github Actions
test-conda-python-3.11Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-hdfs-2.9.2Github Actions
test-conda-python-3.7-hdfs-3.2.1Github Actions
test-conda-python-3.7-pandas-0.24Github Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-spark-v3.1.2Github Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-v3.2.0Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-dask-latestGithub Actions
test-conda-python-3.9-dask-upstream_develGithub Actions
test-conda-python-3.9-pandas-upstream_develGithub Actions
test-conda-python-3.9-spark-masterGithub Actions
test-cuda-pythonGithub Actions
test-debian-11-python-3Azure
test-fedora-35-python-3Azure
test-ubuntu-20.04-python-3Azure
@kou
kou
commented
Nov 16, 2022

Copy link


Copy Markdown

Member
Can we add a test to detect this problem?
@AlenkaF
AlenkaF
requested a review
from kouNovember 22, 2022 07:39
kou
kou
approved these changes
Nov 22, 2022
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Comment threadpython/pyarrow/tests/test_cpp_internals.py
Outdated

 # Arrow C++ and PyArrow C++ header files together

 source = os.path.dirname(os.path.abspath(__file__))
 pyarrow_dir = os.path.abspath(pjoin(source, '..'))
@koukouNov 22, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

os.path.dirname(source)?
@AlenkaFAlenkaFNov 22, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to go from test folder to the parent pyarrow folder to test if the include folder is present there.
@koukouNov 22, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry... I'm not sure why os.path.abspath() is needed for it.

Could you show the followings in your environment?
  • The source value
  • ls -lah ${source} (${source} is the source value)
  • The pyarrow_dir value
  • ls -lah ${pyarrow_dir} (${pyarrow_dir} is the pyarrow_dir value)
@AlenkaFAlenkaFNov 22, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure! Here it is:
(Pdb) print(source)
/Users/alenkafrim/repos/arrow/python/pyarrow/tests
(Pdb) pyarrow_dir
'/Users/alenkafrim/repos/arrow/python/pyarrow'

ls -lah ouput /test
(pyarrow-dev-9) (base) alenkafrim@Alenkas-MacBook-Pro python % ls -lah /Users/alenkafrim/repos/arrow/python/pyarrow/tests
total 3240
drwxr-xr-x 62 alenkafrim staff 1.9K 22 Nov 10:02 .
drwxr-xr-x 109 alenkafrim staff 3.4K 22 Nov 10:02 ..
-rw-r--r-- 1 alenkafrim staff 0B 5 May 2022 __init__.py
drwxr-xr-x 5 alenkafrim staff 160B 22 Nov 10:03 __pycache__
-rw-r--r-- 1 alenkafrim staff 1.3K 2 Jun 13:33 arrow_16597.py
-rw-r--r-- 1 alenkafrim staff 1.1K 3 Nov 2021 arrow_7980.py
-rw-r--r-- 1 alenkafrim staff 2.0K 11 Apr 2022 bound_function_visit_strings.pyx
-rw-r--r-- 1 alenkafrim staff 6.1K 29 Jun 09:38 conftest.py
drwxr-xr-x 5 alenkafrim staff 160B 27 Oct 12:17 data
-rw-r--r-- 1 alenkafrim staff 961B 3 Nov 2021 deserialize_buffer.py
drwxr-xr-x 3 alenkafrim staff 96B 21 Nov 14:54 interchange
-rw-r--r-- 1 alenkafrim staff 5.0K 3 Nov 2021 pandas_examples.py
-rw-r--r-- 1 alenkafrim staff 1.4K 3 Nov 2021 pandas_threaded_import.py
drwxr-xr-x 16 alenkafrim staff 512B 22 Nov 06:02 parquet
-rw-r--r-- 1 alenkafrim staff 1.9K 11 Apr 2022 pyarrow_cython_example.pyx
-rw-r--r-- 1 alenkafrim staff 953B 19 Apr 2022 read_record_batch.py
-rw-r--r-- 1 alenkafrim staff 13K 22 Apr 2022 strategies.py
-rw-r--r-- 1 alenkafrim staff 1.4K 3 Nov 2021 test_adhoc_memory_leak.py
-rw-r--r-- 1 alenkafrim staff 106K 22 Nov 06:02 test_array.py
-rw-r--r-- 1 alenkafrim staff 2.1K 3 Nov 2021 test_builder.py
-rw-r--r-- 1 alenkafrim staff 13K 19 Apr 2022 test_cffi.py
-rw-r--r-- 1 alenkafrim staff 108K 22 Nov 10:02 test_compute.py
-rw-r--r-- 1 alenkafrim staff 70K 27 Oct 14:04 test_convert_builtin.py
-rw-r--r-- 1 alenkafrim staff 1.8K 22 Nov 10:03 test_cpp_internals.py
-rw-r--r-- 1 alenkafrim staff 71K 22 Nov 06:02 test_csv.py
-rw-r--r-- 1 alenkafrim staff 27K 3 Nov 2021 test_cuda.py
-rw-r--r-- 1 alenkafrim staff 8.5K 3 Nov 2021 test_cuda_numba_interop.py
-rw-r--r-- 1 alenkafrim staff 6.5K 22 Nov 06:02 test_cython.py
-rw-r--r-- 1 alenkafrim staff 175K 22 Nov 06:02 test_dataset.py
-rw-r--r-- 1 alenkafrim staff 891B 3 Nov 2021 test_deprecations.py
-rw-r--r-- 1 alenkafrim staff 9.2K 27 Oct 14:04 test_exec_plan.py
-rw-r--r-- 1 alenkafrim staff 33K 22 Nov 06:02 test_extension_type.py
-rw-r--r-- 1 alenkafrim staff 24K 22 Nov 06:02 test_feather.py
-rw-r--r-- 1 alenkafrim staff 2.3K 19 Apr 2022 test_filesystem.py
-rw-r--r-- 1 alenkafrim staff 79K 22 Nov 06:02 test_flight.py
-rw-r--r-- 1 alenkafrim staff 55K 27 Oct 14:04 test_fs.py
-rw-r--r-- 1 alenkafrim staff 15K 27 Oct 14:04 test_gandiva.py
-rw-r--r-- 1 alenkafrim staff 42K 22 Nov 06:02 test_gdb.py
-rw-r--r-- 1 alenkafrim staff 13K 11 Apr 2022 test_hdfs.py
-rw-r--r-- 1 alenkafrim staff 56K 27 Oct 14:04 test_io.py
-rw-r--r-- 1 alenkafrim staff 36K 22 Nov 06:02 test_ipc.py
-rw-r--r-- 1 alenkafrim staff 12K 22 Nov 06:02 test_json.py
-rw-r--r-- 1 alenkafrim staff 15K 3 Nov 2021 test_jvm.py
-rw-r--r-- 1 alenkafrim staff 7.6K 11 Jul 08:19 test_memory.py
-rw-r--r-- 1 alenkafrim staff 5.8K 19 Apr 2022 test_misc.py
-rw-r--r-- 1 alenkafrim staff 19K 22 Nov 10:02 test_orc.py
-rw-r--r-- 1 alenkafrim staff 159K 22 Nov 06:02 test_pandas.py
-rw-r--r-- 1 alenkafrim staff 44K 22 Nov 06:02 test_plasma.py
-rw-r--r-- 1 alenkafrim staff 3.4K 3 Nov 2021 test_plasma_tf_op.py
-rw-r--r-- 1 alenkafrim staff 20K 22 Nov 10:02 test_scalars.py
-rw-r--r-- 1 alenkafrim staff 21K 22 Nov 06:02 test_schema.py
-rw-r--r-- 1 alenkafrim staff 41K 11 Apr 2022 test_serialization.py
-rw-r--r-- 1 alenkafrim staff 1.6K 11 Apr 2022 test_serialization_deprecated.py
-rw-r--r-- 1 alenkafrim staff 17K 3 Nov 2021 test_sparse_tensor.py
-rw-r--r-- 1 alenkafrim staff 1.7K 3 Nov 2021 test_strategies.py
-rw-r--r-- 1 alenkafrim staff 8.4K 22 Nov 06:02 test_substrait.py
-rw-r--r-- 1 alenkafrim staff 65K 27 Oct 14:04 test_table.py
-rw-r--r-- 1 alenkafrim staff 6.1K 11 Apr 2022 test_tensor.py
-rw-r--r-- 1 alenkafrim staff 33K 22 Nov 10:02 test_types.py
-rw-r--r-- 1 alenkafrim staff 17K 29 Jun 09:38 test_udf.py
-rw-r--r-- 1 alenkafrim staff 1.7K 11 Apr 2022 test_util.py
-rw-r--r-- 1 alenkafrim staff 13K 6 May 2022 util.py

ls -lah ouput pyarrow/
(pyarrow-dev-9) (base) alenkafrim@Alenkas-MacBook-Pro python % ls -lah /Users/alenkafrim/repos/arrow/python/pyarrow total 185744
drwxr-xr-x 109 alenkafrim staff 3.4K 22 Nov 10:02 .
drwxr-xr-x 36 alenkafrim staff 1.1K 22 Nov 10:02 ..
-rw-r--r--@ 1 alenkafrim staff 6.0K 17 Nov 12:41 .DS_Store
-rw-r--r-- 1 alenkafrim staff 2.1K 3 Nov 2021 __init__.pxd
-rw-r--r-- 1 alenkafrim staff 19K 22 Nov 08:17 __init__.py
drwxr-xr-x 20 alenkafrim staff 640B 22 Nov 10:03 __pycache__
-rwxr-xr-x 1 alenkafrim staff 3.8M 21 Nov 12:35 _compute.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 1.7K 6 May 2022 _compute.pxd
-rw-r--r-- 1 alenkafrim staff 83K 22 Nov 10:02 _compute.pyx
-rw-r--r-- 1 alenkafrim staff 1.7K 30 May 12:10 _compute_docstrings.py
-rwxr-xr-x 1 alenkafrim staff 970K 21 Nov 12:35 _csv.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 1.6K 11 Apr 2022 _csv.pxd
-rw-r--r-- 1 alenkafrim staff 51K 22 Nov 06:02 _csv.pyx
-rw-r--r-- 1 alenkafrim staff 1.9K 3 Nov 2021 _cuda.pxd
-rw-r--r-- 1 alenkafrim staff 34K 11 Apr 2022 _cuda.pyx
-rwxr-xr-x 1 alenkafrim staff 3.0M 21 Nov 12:35 _dataset.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 4.3K 27 Oct 14:04 _dataset.pxd
-rw-r--r-- 1 alenkafrim staff 100K 22 Nov 06:02 _dataset.pyx
-rw-r--r-- 1 alenkafrim staff 1.3K 11 Apr 2022 _dataset_orc.pyx
-rwxr-xr-x 1 alenkafrim staff 934K 21 Nov 12:35 _dataset_parquet.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 30K 29 Jun 09:38 _dataset_parquet.pyx
-rwxr-xr-x 1 alenkafrim staff 1.0M 21 Nov 12:35 _exec_plan.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 16K 22 Nov 06:02 _exec_plan.pyx
-rwxr-xr-x 1 alenkafrim staff 265K 21 Nov 12:35 _feather.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 3.7K 11 Apr 2022 _feather.pyx
-rw-r--r-- 1 alenkafrim staff 105K 22 Nov 06:02 _flight.pyx
-rwxr-xr-x 1 alenkafrim staff 1.9M 21 Nov 12:35 _fs.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 2.4K 3 Nov 2021 _fs.pxd
-rw-r--r-- 1 alenkafrim staff 51K 22 Nov 06:02 _fs.pyx
-rw-r--r-- 1 alenkafrim staff 8.0K 29 Jun 09:38 _gcsfs.pyx
-rw-r--r-- 1 alenkafrim staff 186B 21 Nov 12:35 _generated_version.py
-rw-r--r-- 1 alenkafrim staff 5.6K 29 Jun 09:38 _hdfs.pyx
-rwxr-xr-x 1 alenkafrim staff 483K 21 Nov 12:35 _hdfsio.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 13K 11 Apr 2022 _hdfsio.pyx
-rwxr-xr-x 1 alenkafrim staff 199K 21 Nov 12:35 _json.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 8.5K 19 Apr 2022 _json.pyx
-rw-r--r-- 1 alenkafrim staff 5.6K 11 Apr 2022 _orc.pxd
-rw-r--r-- 1 alenkafrim staff 15K 11 Apr 2022 _orc.pyx
-rwxr-xr-x 1 alenkafrim staff 1.6M 21 Nov 12:35 _parquet.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 25K 29 Jun 09:38 _parquet.pxd
-rw-r--r-- 1 alenkafrim staff 57K 27 Oct 14:04 _parquet.pyx
-rw-r--r-- 1 alenkafrim staff 5.9K 19 Apr 2022 _parquet_encryption.pxd
-rw-r--r-- 1 alenkafrim staff 17K 19 Apr 2022 _parquet_encryption.pyx
-rw-r--r-- 1 alenkafrim staff 30K 22 Nov 06:02 _plasma.pyx
-rwxr-xr-x 1 alenkafrim staff 207K 21 Nov 12:35 _pyarrow_cpp_tests.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 1.2K 22 Nov 06:02 _pyarrow_cpp_tests.pxd
-rw-r--r-- 1 alenkafrim staff 1.7K 22 Nov 06:02 _pyarrow_cpp_tests.pyx
-rwxr-xr-x 1 alenkafrim staff 388K 21 Nov 12:35 _s3fs.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 16K 22 Nov 06:02 _s3fs.pyx
-rw-r--r-- 1 alenkafrim staff 6.3K 22 Nov 06:02 _substrait.pyx
-rw-r--r-- 1 alenkafrim staff 91K 22 Nov 06:02 array.pxi
-rw-r--r-- 1 alenkafrim staff 869B 3 Nov 2021 benchmark.pxi
-rw-r--r-- 1 alenkafrim staff 856B 3 Nov 2021 benchmark.py
-rw-r--r-- 1 alenkafrim staff 2.6K 3 Nov 2021 builder.pxi
-rw-r--r-- 1 alenkafrim staff 2.1K 3 Nov 2021 cffi.py
-rw-r--r-- 1 alenkafrim staff 2.0K 11 Apr 2022 compat.pxi
-rw-r--r-- 1 alenkafrim staff 21K 22 Nov 10:02 compute.py
-rw-r--r-- 1 alenkafrim staff 2.7K 11 Apr 2022 config.pxi
-rw-r--r-- 1 alenkafrim staff 6.2K 27 Oct 14:04 conftest.py
-rw-r--r-- 1 alenkafrim staff 974B 11 Apr 2022 csv.py
-rw-r--r-- 1 alenkafrim staff 1.1K 3 Nov 2021 cuda.py
-rw-r--r-- 1 alenkafrim staff 38K 27 Oct 14:04 dataset.py
-rw-r--r-- 1 alenkafrim staff 8.3K 22 Nov 06:02 error.pxi
-rw-r--r-- 1 alenkafrim staff 9.9K 22 Nov 06:02 feather.py
-rw-r--r-- 1 alenkafrim staff 14K 10 Jun 10:00 filesystem.py
-rw-r--r-- 1 alenkafrim staff 1.8K 22 Nov 06:02 flight.py
-rw-r--r-- 1 alenkafrim staff 14K 27 Oct 14:04 fs.py
-rw-r--r-- 1 alenkafrim staff 20K 27 Oct 14:04 gandiva.pyx
-rw-r--r-- 1 alenkafrim staff 7.4K 11 Apr 2022 hdfs.py
drwxr-xr-x 3 alenkafrim staff 96B 17 Nov 11:37 include
drwxr-xr-x 15 alenkafrim staff 480B 22 Nov 10:02 includes
drwxr-xr-x 3 alenkafrim staff 96B 21 Nov 14:54 interchange
-rw-r--r-- 1 alenkafrim staff 68K 27 Oct 14:04 io.pxi
-rw-r--r-- 1 alenkafrim staff 33K 22 Nov 06:02 ipc.pxi
-rw-r--r-- 1 alenkafrim staff 9.7K 22 Nov 06:02 ipc.py
-rw-r--r-- 1 alenkafrim staff 858B 3 Nov 2021 json.py
-rw-r--r-- 1 alenkafrim staff 9.4K 3 Nov 2021 jvm.py
-rwxr-xr-x 1 alenkafrim staff 7.0M 21 Nov 12:35 lib.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 14K 22 Nov 10:02 lib.pxd
-rw-r--r-- 1 alenkafrim staff 4.6K 21 Apr 2022 lib.pyx
-rw-r--r-- 1 alenkafrim staff 19K 21 Nov 12:35 lib_api.h
-rwxr-xr-x 1 alenkafrim staff 12M 21 Nov 12:35 libarrow_python.1100.0.0.dylib
-rwxr-xr-x 1 alenkafrim staff 12M 21 Nov 12:35 libarrow_python.1100.dylib
-rw-r--r-- 1 alenkafrim staff 30M 21 Nov 12:35 libarrow_python.a
-rwxr-xr-x 1 alenkafrim staff 12M 21 Nov 12:35 libarrow_python.dylib
-rwxr-xr-x 1 alenkafrim staff 458K 21 Nov 12:35 libarrow_python_flight.1100.0.0.dylib
-rwxr-xr-x 1 alenkafrim staff 458K 21 Nov 12:35 libarrow_python_flight.1100.dylib
-rw-r--r-- 1 alenkafrim staff 1.2M 21 Nov 12:35 libarrow_python_flight.a
-rwxr-xr-x 1 alenkafrim staff 458K 21 Nov 12:35 libarrow_python_flight.dylib
-rw-r--r-- 1 alenkafrim staff 8.0K 27 Oct 14:04 memory.pxi
-rw-r--r-- 1 alenkafrim staff 12K 22 Nov 06:02 orc.py
-rw-r--r-- 1 alenkafrim staff 7.8K 17 Nov 10:51 pandas-shim.pxi
-rw-r--r-- 1 alenkafrim staff 43K 22 Nov 06:02 pandas_compat.py
drwxr-xr-x 6 alenkafrim staff 192B 22 Nov 06:02 parquet
-rw-r--r-- 1 alenkafrim staff 6.3K 22 Nov 06:02 plasma.py
-rw-r--r-- 1 alenkafrim staff 13K 22 Nov 10:02 public-api.pxi
-rw-r--r-- 1 alenkafrim staff 29K 22 Nov 10:02 scalar.pxi
-rw-r--r-- 1 alenkafrim staff 19K 29 Jun 09:38 serialization.pxi
-rw-r--r-- 1 alenkafrim staff 18K 3 Nov 2021 serialization.py
drwxr-xr-x 8 alenkafrim staff 256B 22 Nov 10:02 src
-rw-r--r-- 1 alenkafrim staff 873B 27 Oct 14:04 substrait.py
-rw-r--r-- 1 alenkafrim staff 158K 22 Nov 06:02 table.pxi
-rw-r--r-- 1 alenkafrim staff 38K 19 Apr 2022 tensor.pxi
drwxr-xr-x 3 alenkafrim staff 96B 27 Oct 14:04 tensorflow
drwxr-xr-x 62 alenkafrim staff 1.9K 22 Nov 10:02 tests
-rw-r--r-- 1 alenkafrim staff 90K 22 Nov 06:02 types.pxi
-rw-r--r-- 1 alenkafrim staff 10K 11 Apr 2022 types.py
-rw-r--r-- 1 alenkafrim staff 4.6K 22 Nov 06:02 util.py
drwxr-xr-x 6 alenkafrim staff 192B 17 Nov 11:07 vendored

If I remember correctly, if I didn't use os.path.abspath() the .. didn't have an effect.
@jorisvandenbosschejorisvandenbosscheNov 22, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For me it seems to work also with a ".." in it:
In [9]: os.path.exists("/home/joris/scipy/repos/arrow/python/pyarrow/tests/../include/arrow/python/")
Out[9]: True

@jorisvandenbosschejorisvandenbosscheNov 22, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But so about @kou's original comment, I think doing os.path.dirname(source) also works because if source is already a directory, dirname gives the parent directory.

Now personally I find .. more explicit to read that it goes one level up.
@AlenkaFAlenkaFNov 22, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, sorry for my confusion.

Can change to pyarrow_dir = os.path.dirname(source), @kou do you agree?
@AlenkaFAlenkaFNov 22, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, now I see Joris you already made a suggestion. Will commit that 👍
@kou
kou
commented
Nov 22, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g python -g conda
@github-actions
github-actionsBot
commented
Nov 22, 2022

Copy link


Copy Markdown

Revision: acf8754
Submitted crossbow builds: ursacomputing/crossbow @ actions-d4595c1625
TaskStatus
conda-cleanAzure
conda-linux-gcc-py310-arm64Azure
conda-linux-gcc-py310-cpuAzure
conda-linux-gcc-py310-cudaAzure
conda-linux-gcc-py310-ppc64leAzure
conda-linux-gcc-py37-arm64Azure
conda-linux-gcc-py37-cpu-r40Azure
conda-linux-gcc-py37-cpu-r41Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py37-ppc64leAzure
conda-linux-gcc-py38-arm64Azure
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py38-ppc64leAzure
conda-linux-gcc-py39-arm64Azure
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-linux-gcc-py39-ppc64leAzure
conda-osx-arm64-clang-py310Azure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py310Azure
conda-osx-clang-py37-r40Azure
conda-osx-clang-py37-r41Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2019-py310Azure
conda-win-vs2019-py37-r40Azure
conda-win-vs2019-py37-r41Azure
conda-win-vs2019-py38Azure
conda-win-vs2019-py39Azure
test-conda-python-3.10Github Actions
test-conda-python-3.11Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-hdfs-2.9.2Github Actions
test-conda-python-3.7-hdfs-3.2.1Github Actions
test-conda-python-3.7-pandas-0.24Github Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-spark-v3.1.2Github Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-v3.2.0Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-dask-latestGithub Actions
test-conda-python-3.9-dask-upstream_develGithub Actions
test-conda-python-3.9-pandas-upstream_develGithub Actions
test-conda-python-3.9-spark-masterGithub Actions
test-cuda-pythonGithub Actions
test-debian-11-python-3Azure
test-fedora-35-python-3Azure
test-ubuntu-20.04-python-3Azure
jorisvandenbossche
jorisvandenbossche
reviewed
Nov 22, 2022
View reviewed changes
Comment threadpython/pyarrow/tests/test_cpp_internals.py
Outdated

@AlenkaF@jorisvandenbossche
Update python/pyarrow/tests/test_cpp_internals.py
4669e94
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
@jorisvandenbosschejorisvandenbossche
mentioned this pull request
Nov 22, 2022
 Merged

3 tasks
@kou
kou
merged commit 1e9eb61
into
apache:masterNov 22, 2022
@AlenkaF
AlenkaF
deleted the

ARROW-18340

branch
November 23, 2022 08:41
@ursabot
ursabot
commented
Nov 24, 2022

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = 59f99d2 and contender = 1e9eb61. 1e9eb61 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2

[Finished ⬇️0.34% ⬆️0.54%] test-mac-arm

[Finished ⬇️0.27% ⬆️0.0%] ursa-i9-9960x

[Finished ⬇️0.14% ⬆️0.35%] ursa-thinkcentre-m75q

Buildkite builds:

[Finished] 1e9eb615 ec2-t3-xlarge-us-east-2

[Finished] 1e9eb615 test-mac-arm

[Finished] 1e9eb615 ursa-i9-9960x

[Finished] 1e9eb615 ursa-thinkcentre-m75q

[Finished] 59f99d2c ec2-t3-xlarge-us-east-2

[Finished] 59f99d2c test-mac-arm

[Finished] 59f99d2c ursa-i9-9960x

[Finished] 59f99d2c ursa-thinkcentre-m75q

Supported benchmarks:

ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True

test-mac-arm: Supported benchmark langs: C++, Python, R

ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript

ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
@asfimportasfimport
mentioned this pull request
Nov 24, 2022

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@jorisvandenbosschejorisvandenbosschejorisvandenbossche left review comments

Assignees


No one assigned


Labels

Component: Python

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@AlenkaF@jorisvandenbossche@kou@ursabot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
ARROW-18340: [Python] PyArrow C++ header files no longer always included in installed pyarrow - #14656
Merged
kou merged 7 commits into
apache:masterfrom 
AlenkaF:ARROW-18340
Nov 22, 2022
Merged
ARROW-18340: [Python] PyArrow C++ header files no longer always included in installed pyarrow#14656
kou merged 7 commits into
apache:masterfrom 
AlenkaF:ARROW-18340
Conversation
@AlenkaF
@AlenkaFAlenkaF
commented
Nov 16, 2022

Copy link


Copy Markdown

Member
This PR adds code to _run_cmake to always copy PyArrow C++ header files to pyarrow/iclude.
@github-actions
github-actionsBot
commented
Nov 16, 2022

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-18340
@github-actions
github-actionsBot
commented
Nov 16, 2022

Copy link


Copy Markdown

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.
jorisvandenbossche
jorisvandenbossche
reviewed
Nov 16, 2022
View reviewed changes
Comment threadpython/setup.py
Outdated

@jorisvandenbossche
jorisvandenbossche
commented
Nov 16, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g wheel
@github-actions
github-actionsBot
commented
Nov 16, 2022

Copy link


Copy Markdown

Revision: d5f953f
Submitted crossbow builds: ursacomputing/crossbow @ actions-1c3aa4b08e
TaskStatus
wheel-macos-big-sur-cp310-arm64Github Actions
wheel-macos-big-sur-cp311-arm64Github Actions
wheel-macos-big-sur-cp38-arm64Github Actions
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-mojave-cp310-amd64Github Actions
wheel-macos-mojave-cp311-amd64Github Actions
wheel-macos-mojave-cp37-amd64Github Actions
wheel-macos-mojave-cp38-amd64Github Actions
wheel-macos-mojave-cp39-amd64Github Actions
wheel-manylinux2014-cp310-amd64Github Actions
wheel-manylinux2014-cp310-arm64Travis CI
wheel-manylinux2014-cp311-amd64Github Actions
wheel-manylinux2014-cp311-arm64Travis CI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64Travis CI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64Travis CI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64Travis CI
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
kou
kou
reviewed
Nov 16, 2022
View reviewed changes
Comment threadpython/setup.py
Outdated

@kou
kou
commented
Nov 16, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g python -g conda
@github-actions
github-actionsBot
commented
Nov 16, 2022

Copy link


Copy Markdown

Revision: d5f953f
Submitted crossbow builds: ursacomputing/crossbow @ actions-9eb5a3c4ca
TaskStatus
conda-cleanAzure
conda-linux-gcc-py310-arm64Azure
conda-linux-gcc-py310-cpuAzure
conda-linux-gcc-py310-cudaAzure
conda-linux-gcc-py310-ppc64leAzure
conda-linux-gcc-py37-arm64Azure
conda-linux-gcc-py37-cpu-r40Azure
conda-linux-gcc-py37-cpu-r41Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py37-ppc64leAzure
conda-linux-gcc-py38-arm64Azure
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py38-ppc64leAzure
conda-linux-gcc-py39-arm64Azure
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-linux-gcc-py39-ppc64leAzure
conda-osx-arm64-clang-py310Azure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py310Azure
conda-osx-clang-py37-r40Azure
conda-osx-clang-py37-r41Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2019-py310Azure
conda-win-vs2019-py37-r40Azure
conda-win-vs2019-py37-r41Azure
conda-win-vs2019-py38Azure
conda-win-vs2019-py39Azure
test-conda-python-3.10Github Actions
test-conda-python-3.11Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-hdfs-2.9.2Github Actions
test-conda-python-3.7-hdfs-3.2.1Github Actions
test-conda-python-3.7-pandas-0.24Github Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-spark-v3.1.2Github Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-v3.2.0Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-dask-latestGithub Actions
test-conda-python-3.9-dask-upstream_develGithub Actions
test-conda-python-3.9-pandas-upstream_develGithub Actions
test-conda-python-3.9-spark-masterGithub Actions
test-cuda-pythonGithub Actions
test-debian-11-python-3Azure
test-fedora-35-python-3Azure
test-ubuntu-20.04-python-3Azure
@kou
kou
commented
Nov 16, 2022

Copy link


Copy Markdown

Member
Can we add a test to detect this problem?
@AlenkaF
AlenkaF
requested a review
from kouNovember 22, 2022 07:39
kou
kou
approved these changes
Nov 22, 2022
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Comment threadpython/pyarrow/tests/test_cpp_internals.py
Outdated

 # Arrow C++ and PyArrow C++ header files together

 source = os.path.dirname(os.path.abspath(__file__))
 pyarrow_dir = os.path.abspath(pjoin(source, '..'))
@koukouNov 22, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

os.path.dirname(source)?
@AlenkaFAlenkaFNov 22, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to go from test folder to the parent pyarrow folder to test if the include folder is present there.
@koukouNov 22, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry... I'm not sure why os.path.abspath() is needed for it.

Could you show the followings in your environment?
  • The source value
  • ls -lah ${source} (${source} is the source value)
  • The pyarrow_dir value
  • ls -lah ${pyarrow_dir} (${pyarrow_dir} is the pyarrow_dir value)
@AlenkaFAlenkaFNov 22, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure! Here it is:
(Pdb) print(source)
/Users/alenkafrim/repos/arrow/python/pyarrow/tests
(Pdb) pyarrow_dir
'/Users/alenkafrim/repos/arrow/python/pyarrow'

ls -lah ouput /test
(pyarrow-dev-9) (base) alenkafrim@Alenkas-MacBook-Pro python % ls -lah /Users/alenkafrim/repos/arrow/python/pyarrow/tests
total 3240
drwxr-xr-x 62 alenkafrim staff 1.9K 22 Nov 10:02 .
drwxr-xr-x 109 alenkafrim staff 3.4K 22 Nov 10:02 ..
-rw-r--r-- 1 alenkafrim staff 0B 5 May 2022 __init__.py
drwxr-xr-x 5 alenkafrim staff 160B 22 Nov 10:03 __pycache__
-rw-r--r-- 1 alenkafrim staff 1.3K 2 Jun 13:33 arrow_16597.py
-rw-r--r-- 1 alenkafrim staff 1.1K 3 Nov 2021 arrow_7980.py
-rw-r--r-- 1 alenkafrim staff 2.0K 11 Apr 2022 bound_function_visit_strings.pyx
-rw-r--r-- 1 alenkafrim staff 6.1K 29 Jun 09:38 conftest.py
drwxr-xr-x 5 alenkafrim staff 160B 27 Oct 12:17 data
-rw-r--r-- 1 alenkafrim staff 961B 3 Nov 2021 deserialize_buffer.py
drwxr-xr-x 3 alenkafrim staff 96B 21 Nov 14:54 interchange
-rw-r--r-- 1 alenkafrim staff 5.0K 3 Nov 2021 pandas_examples.py
-rw-r--r-- 1 alenkafrim staff 1.4K 3 Nov 2021 pandas_threaded_import.py
drwxr-xr-x 16 alenkafrim staff 512B 22 Nov 06:02 parquet
-rw-r--r-- 1 alenkafrim staff 1.9K 11 Apr 2022 pyarrow_cython_example.pyx
-rw-r--r-- 1 alenkafrim staff 953B 19 Apr 2022 read_record_batch.py
-rw-r--r-- 1 alenkafrim staff 13K 22 Apr 2022 strategies.py
-rw-r--r-- 1 alenkafrim staff 1.4K 3 Nov 2021 test_adhoc_memory_leak.py
-rw-r--r-- 1 alenkafrim staff 106K 22 Nov 06:02 test_array.py
-rw-r--r-- 1 alenkafrim staff 2.1K 3 Nov 2021 test_builder.py
-rw-r--r-- 1 alenkafrim staff 13K 19 Apr 2022 test_cffi.py
-rw-r--r-- 1 alenkafrim staff 108K 22 Nov 10:02 test_compute.py
-rw-r--r-- 1 alenkafrim staff 70K 27 Oct 14:04 test_convert_builtin.py
-rw-r--r-- 1 alenkafrim staff 1.8K 22 Nov 10:03 test_cpp_internals.py
-rw-r--r-- 1 alenkafrim staff 71K 22 Nov 06:02 test_csv.py
-rw-r--r-- 1 alenkafrim staff 27K 3 Nov 2021 test_cuda.py
-rw-r--r-- 1 alenkafrim staff 8.5K 3 Nov 2021 test_cuda_numba_interop.py
-rw-r--r-- 1 alenkafrim staff 6.5K 22 Nov 06:02 test_cython.py
-rw-r--r-- 1 alenkafrim staff 175K 22 Nov 06:02 test_dataset.py
-rw-r--r-- 1 alenkafrim staff 891B 3 Nov 2021 test_deprecations.py
-rw-r--r-- 1 alenkafrim staff 9.2K 27 Oct 14:04 test_exec_plan.py
-rw-r--r-- 1 alenkafrim staff 33K 22 Nov 06:02 test_extension_type.py
-rw-r--r-- 1 alenkafrim staff 24K 22 Nov 06:02 test_feather.py
-rw-r--r-- 1 alenkafrim staff 2.3K 19 Apr 2022 test_filesystem.py
-rw-r--r-- 1 alenkafrim staff 79K 22 Nov 06:02 test_flight.py
-rw-r--r-- 1 alenkafrim staff 55K 27 Oct 14:04 test_fs.py
-rw-r--r-- 1 alenkafrim staff 15K 27 Oct 14:04 test_gandiva.py
-rw-r--r-- 1 alenkafrim staff 42K 22 Nov 06:02 test_gdb.py
-rw-r--r-- 1 alenkafrim staff 13K 11 Apr 2022 test_hdfs.py
-rw-r--r-- 1 alenkafrim staff 56K 27 Oct 14:04 test_io.py
-rw-r--r-- 1 alenkafrim staff 36K 22 Nov 06:02 test_ipc.py
-rw-r--r-- 1 alenkafrim staff 12K 22 Nov 06:02 test_json.py
-rw-r--r-- 1 alenkafrim staff 15K 3 Nov 2021 test_jvm.py
-rw-r--r-- 1 alenkafrim staff 7.6K 11 Jul 08:19 test_memory.py
-rw-r--r-- 1 alenkafrim staff 5.8K 19 Apr 2022 test_misc.py
-rw-r--r-- 1 alenkafrim staff 19K 22 Nov 10:02 test_orc.py
-rw-r--r-- 1 alenkafrim staff 159K 22 Nov 06:02 test_pandas.py
-rw-r--r-- 1 alenkafrim staff 44K 22 Nov 06:02 test_plasma.py
-rw-r--r-- 1 alenkafrim staff 3.4K 3 Nov 2021 test_plasma_tf_op.py
-rw-r--r-- 1 alenkafrim staff 20K 22 Nov 10:02 test_scalars.py
-rw-r--r-- 1 alenkafrim staff 21K 22 Nov 06:02 test_schema.py
-rw-r--r-- 1 alenkafrim staff 41K 11 Apr 2022 test_serialization.py
-rw-r--r-- 1 alenkafrim staff 1.6K 11 Apr 2022 test_serialization_deprecated.py
-rw-r--r-- 1 alenkafrim staff 17K 3 Nov 2021 test_sparse_tensor.py
-rw-r--r-- 1 alenkafrim staff 1.7K 3 Nov 2021 test_strategies.py
-rw-r--r-- 1 alenkafrim staff 8.4K 22 Nov 06:02 test_substrait.py
-rw-r--r-- 1 alenkafrim staff 65K 27 Oct 14:04 test_table.py
-rw-r--r-- 1 alenkafrim staff 6.1K 11 Apr 2022 test_tensor.py
-rw-r--r-- 1 alenkafrim staff 33K 22 Nov 10:02 test_types.py
-rw-r--r-- 1 alenkafrim staff 17K 29 Jun 09:38 test_udf.py
-rw-r--r-- 1 alenkafrim staff 1.7K 11 Apr 2022 test_util.py
-rw-r--r-- 1 alenkafrim staff 13K 6 May 2022 util.py

ls -lah ouput pyarrow/
(pyarrow-dev-9) (base) alenkafrim@Alenkas-MacBook-Pro python % ls -lah /Users/alenkafrim/repos/arrow/python/pyarrow total 185744
drwxr-xr-x 109 alenkafrim staff 3.4K 22 Nov 10:02 .
drwxr-xr-x 36 alenkafrim staff 1.1K 22 Nov 10:02 ..
-rw-r--r--@ 1 alenkafrim staff 6.0K 17 Nov 12:41 .DS_Store
-rw-r--r-- 1 alenkafrim staff 2.1K 3 Nov 2021 __init__.pxd
-rw-r--r-- 1 alenkafrim staff 19K 22 Nov 08:17 __init__.py
drwxr-xr-x 20 alenkafrim staff 640B 22 Nov 10:03 __pycache__
-rwxr-xr-x 1 alenkafrim staff 3.8M 21 Nov 12:35 _compute.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 1.7K 6 May 2022 _compute.pxd
-rw-r--r-- 1 alenkafrim staff 83K 22 Nov 10:02 _compute.pyx
-rw-r--r-- 1 alenkafrim staff 1.7K 30 May 12:10 _compute_docstrings.py
-rwxr-xr-x 1 alenkafrim staff 970K 21 Nov 12:35 _csv.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 1.6K 11 Apr 2022 _csv.pxd
-rw-r--r-- 1 alenkafrim staff 51K 22 Nov 06:02 _csv.pyx
-rw-r--r-- 1 alenkafrim staff 1.9K 3 Nov 2021 _cuda.pxd
-rw-r--r-- 1 alenkafrim staff 34K 11 Apr 2022 _cuda.pyx
-rwxr-xr-x 1 alenkafrim staff 3.0M 21 Nov 12:35 _dataset.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 4.3K 27 Oct 14:04 _dataset.pxd
-rw-r--r-- 1 alenkafrim staff 100K 22 Nov 06:02 _dataset.pyx
-rw-r--r-- 1 alenkafrim staff 1.3K 11 Apr 2022 _dataset_orc.pyx
-rwxr-xr-x 1 alenkafrim staff 934K 21 Nov 12:35 _dataset_parquet.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 30K 29 Jun 09:38 _dataset_parquet.pyx
-rwxr-xr-x 1 alenkafrim staff 1.0M 21 Nov 12:35 _exec_plan.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 16K 22 Nov 06:02 _exec_plan.pyx
-rwxr-xr-x 1 alenkafrim staff 265K 21 Nov 12:35 _feather.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 3.7K 11 Apr 2022 _feather.pyx
-rw-r--r-- 1 alenkafrim staff 105K 22 Nov 06:02 _flight.pyx
-rwxr-xr-x 1 alenkafrim staff 1.9M 21 Nov 12:35 _fs.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 2.4K 3 Nov 2021 _fs.pxd
-rw-r--r-- 1 alenkafrim staff 51K 22 Nov 06:02 _fs.pyx
-rw-r--r-- 1 alenkafrim staff 8.0K 29 Jun 09:38 _gcsfs.pyx
-rw-r--r-- 1 alenkafrim staff 186B 21 Nov 12:35 _generated_version.py
-rw-r--r-- 1 alenkafrim staff 5.6K 29 Jun 09:38 _hdfs.pyx
-rwxr-xr-x 1 alenkafrim staff 483K 21 Nov 12:35 _hdfsio.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 13K 11 Apr 2022 _hdfsio.pyx
-rwxr-xr-x 1 alenkafrim staff 199K 21 Nov 12:35 _json.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 8.5K 19 Apr 2022 _json.pyx
-rw-r--r-- 1 alenkafrim staff 5.6K 11 Apr 2022 _orc.pxd
-rw-r--r-- 1 alenkafrim staff 15K 11 Apr 2022 _orc.pyx
-rwxr-xr-x 1 alenkafrim staff 1.6M 21 Nov 12:35 _parquet.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 25K 29 Jun 09:38 _parquet.pxd
-rw-r--r-- 1 alenkafrim staff 57K 27 Oct 14:04 _parquet.pyx
-rw-r--r-- 1 alenkafrim staff 5.9K 19 Apr 2022 _parquet_encryption.pxd
-rw-r--r-- 1 alenkafrim staff 17K 19 Apr 2022 _parquet_encryption.pyx
-rw-r--r-- 1 alenkafrim staff 30K 22 Nov 06:02 _plasma.pyx
-rwxr-xr-x 1 alenkafrim staff 207K 21 Nov 12:35 _pyarrow_cpp_tests.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 1.2K 22 Nov 06:02 _pyarrow_cpp_tests.pxd
-rw-r--r-- 1 alenkafrim staff 1.7K 22 Nov 06:02 _pyarrow_cpp_tests.pyx
-rwxr-xr-x 1 alenkafrim staff 388K 21 Nov 12:35 _s3fs.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 16K 22 Nov 06:02 _s3fs.pyx
-rw-r--r-- 1 alenkafrim staff 6.3K 22 Nov 06:02 _substrait.pyx
-rw-r--r-- 1 alenkafrim staff 91K 22 Nov 06:02 array.pxi
-rw-r--r-- 1 alenkafrim staff 869B 3 Nov 2021 benchmark.pxi
-rw-r--r-- 1 alenkafrim staff 856B 3 Nov 2021 benchmark.py
-rw-r--r-- 1 alenkafrim staff 2.6K 3 Nov 2021 builder.pxi
-rw-r--r-- 1 alenkafrim staff 2.1K 3 Nov 2021 cffi.py
-rw-r--r-- 1 alenkafrim staff 2.0K 11 Apr 2022 compat.pxi
-rw-r--r-- 1 alenkafrim staff 21K 22 Nov 10:02 compute.py
-rw-r--r-- 1 alenkafrim staff 2.7K 11 Apr 2022 config.pxi
-rw-r--r-- 1 alenkafrim staff 6.2K 27 Oct 14:04 conftest.py
-rw-r--r-- 1 alenkafrim staff 974B 11 Apr 2022 csv.py
-rw-r--r-- 1 alenkafrim staff 1.1K 3 Nov 2021 cuda.py
-rw-r--r-- 1 alenkafrim staff 38K 27 Oct 14:04 dataset.py
-rw-r--r-- 1 alenkafrim staff 8.3K 22 Nov 06:02 error.pxi
-rw-r--r-- 1 alenkafrim staff 9.9K 22 Nov 06:02 feather.py
-rw-r--r-- 1 alenkafrim staff 14K 10 Jun 10:00 filesystem.py
-rw-r--r-- 1 alenkafrim staff 1.8K 22 Nov 06:02 flight.py
-rw-r--r-- 1 alenkafrim staff 14K 27 Oct 14:04 fs.py
-rw-r--r-- 1 alenkafrim staff 20K 27 Oct 14:04 gandiva.pyx
-rw-r--r-- 1 alenkafrim staff 7.4K 11 Apr 2022 hdfs.py
drwxr-xr-x 3 alenkafrim staff 96B 17 Nov 11:37 include
drwxr-xr-x 15 alenkafrim staff 480B 22 Nov 10:02 includes
drwxr-xr-x 3 alenkafrim staff 96B 21 Nov 14:54 interchange
-rw-r--r-- 1 alenkafrim staff 68K 27 Oct 14:04 io.pxi
-rw-r--r-- 1 alenkafrim staff 33K 22 Nov 06:02 ipc.pxi
-rw-r--r-- 1 alenkafrim staff 9.7K 22 Nov 06:02 ipc.py
-rw-r--r-- 1 alenkafrim staff 858B 3 Nov 2021 json.py
-rw-r--r-- 1 alenkafrim staff 9.4K 3 Nov 2021 jvm.py
-rwxr-xr-x 1 alenkafrim staff 7.0M 21 Nov 12:35 lib.cpython-39-darwin.so
-rw-r--r-- 1 alenkafrim staff 14K 22 Nov 10:02 lib.pxd
-rw-r--r-- 1 alenkafrim staff 4.6K 21 Apr 2022 lib.pyx
-rw-r--r-- 1 alenkafrim staff 19K 21 Nov 12:35 lib_api.h
-rwxr-xr-x 1 alenkafrim staff 12M 21 Nov 12:35 libarrow_python.1100.0.0.dylib
-rwxr-xr-x 1 alenkafrim staff 12M 21 Nov 12:35 libarrow_python.1100.dylib
-rw-r--r-- 1 alenkafrim staff 30M 21 Nov 12:35 libarrow_python.a
-rwxr-xr-x 1 alenkafrim staff 12M 21 Nov 12:35 libarrow_python.dylib
-rwxr-xr-x 1 alenkafrim staff 458K 21 Nov 12:35 libarrow_python_flight.1100.0.0.dylib
-rwxr-xr-x 1 alenkafrim staff 458K 21 Nov 12:35 libarrow_python_flight.1100.dylib
-rw-r--r-- 1 alenkafrim staff 1.2M 21 Nov 12:35 libarrow_python_flight.a
-rwxr-xr-x 1 alenkafrim staff 458K 21 Nov 12:35 libarrow_python_flight.dylib
-rw-r--r-- 1 alenkafrim staff 8.0K 27 Oct 14:04 memory.pxi
-rw-r--r-- 1 alenkafrim staff 12K 22 Nov 06:02 orc.py
-rw-r--r-- 1 alenkafrim staff 7.8K 17 Nov 10:51 pandas-shim.pxi
-rw-r--r-- 1 alenkafrim staff 43K 22 Nov 06:02 pandas_compat.py
drwxr-xr-x 6 alenkafrim staff 192B 22 Nov 06:02 parquet
-rw-r--r-- 1 alenkafrim staff 6.3K 22 Nov 06:02 plasma.py
-rw-r--r-- 1 alenkafrim staff 13K 22 Nov 10:02 public-api.pxi
-rw-r--r-- 1 alenkafrim staff 29K 22 Nov 10:02 scalar.pxi
-rw-r--r-- 1 alenkafrim staff 19K 29 Jun 09:38 serialization.pxi
-rw-r--r-- 1 alenkafrim staff 18K 3 Nov 2021 serialization.py
drwxr-xr-x 8 alenkafrim staff 256B 22 Nov 10:02 src
-rw-r--r-- 1 alenkafrim staff 873B 27 Oct 14:04 substrait.py
-rw-r--r-- 1 alenkafrim staff 158K 22 Nov 06:02 table.pxi
-rw-r--r-- 1 alenkafrim staff 38K 19 Apr 2022 tensor.pxi
drwxr-xr-x 3 alenkafrim staff 96B 27 Oct 14:04 tensorflow
drwxr-xr-x 62 alenkafrim staff 1.9K 22 Nov 10:02 tests
-rw-r--r-- 1 alenkafrim staff 90K 22 Nov 06:02 types.pxi
-rw-r--r-- 1 alenkafrim staff 10K 11 Apr 2022 types.py
-rw-r--r-- 1 alenkafrim staff 4.6K 22 Nov 06:02 util.py
drwxr-xr-x 6 alenkafrim staff 192B 17 Nov 11:07 vendored

If I remember correctly, if I didn't use os.path.abspath() the .. didn't have an effect.
@jorisvandenbosschejorisvandenbosscheNov 22, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For me it seems to work also with a ".." in it:
In [9]: os.path.exists("/home/joris/scipy/repos/arrow/python/pyarrow/tests/../include/arrow/python/")
Out[9]: True

@jorisvandenbosschejorisvandenbosscheNov 22, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But so about @kou's original comment, I think doing os.path.dirname(source) also works because if source is already a directory, dirname gives the parent directory.

Now personally I find .. more explicit to read that it goes one level up.
@AlenkaFAlenkaFNov 22, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, sorry for my confusion.

Can change to pyarrow_dir = os.path.dirname(source), @kou do you agree?
@AlenkaFAlenkaFNov 22, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, now I see Joris you already made a suggestion. Will commit that 👍
@kou
kou
commented
Nov 22, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g python -g conda
@github-actions
github-actionsBot
commented
Nov 22, 2022

Copy link


Copy Markdown

Revision: acf8754
Submitted crossbow builds: ursacomputing/crossbow @ actions-d4595c1625
TaskStatus
conda-cleanAzure
conda-linux-gcc-py310-arm64Azure
conda-linux-gcc-py310-cpuAzure
conda-linux-gcc-py310-cudaAzure
conda-linux-gcc-py310-ppc64leAzure
conda-linux-gcc-py37-arm64Azure
conda-linux-gcc-py37-cpu-r40Azure
conda-linux-gcc-py37-cpu-r41Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py37-ppc64leAzure
conda-linux-gcc-py38-arm64Azure
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py38-ppc64leAzure
conda-linux-gcc-py39-arm64Azure
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-linux-gcc-py39-ppc64leAzure
conda-osx-arm64-clang-py310Azure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py310Azure
conda-osx-clang-py37-r40Azure
conda-osx-clang-py37-r41Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2019-py310Azure
conda-win-vs2019-py37-r40Azure
conda-win-vs2019-py37-r41Azure
conda-win-vs2019-py38Azure
conda-win-vs2019-py39Azure
test-conda-python-3.10Github Actions
test-conda-python-3.11Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-hdfs-2.9.2Github Actions
test-conda-python-3.7-hdfs-3.2.1Github Actions
test-conda-python-3.7-pandas-0.24Github Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-spark-v3.1.2Github Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-v3.2.0Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-dask-latestGithub Actions
test-conda-python-3.9-dask-upstream_develGithub Actions
test-conda-python-3.9-pandas-upstream_develGithub Actions
test-conda-python-3.9-spark-masterGithub Actions
test-cuda-pythonGithub Actions
test-debian-11-python-3Azure
test-fedora-35-python-3Azure
test-ubuntu-20.04-python-3Azure
jorisvandenbossche
jorisvandenbossche
reviewed
Nov 22, 2022
View reviewed changes
Comment threadpython/pyarrow/tests/test_cpp_internals.py
Outdated

@AlenkaF@jorisvandenbossche
Update python/pyarrow/tests/test_cpp_internals.py
4669e94
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
@jorisvandenbosschejorisvandenbossche
mentioned this pull request
Nov 22, 2022
 Merged

3 tasks
@kou
kou
merged commit 1e9eb61
into
apache:masterNov 22, 2022
@AlenkaF
AlenkaF
deleted the

ARROW-18340

branch
November 23, 2022 08:41
@ursabot
ursabot
commented
Nov 24, 2022

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = 59f99d2 and contender = 1e9eb61. 1e9eb61 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2

[Finished ⬇️0.34% ⬆️0.54%] test-mac-arm

[Finished ⬇️0.27% ⬆️0.0%] ursa-i9-9960x

[Finished ⬇️0.14% ⬆️0.35%] ursa-thinkcentre-m75q

Buildkite builds:

[Finished] 1e9eb615 ec2-t3-xlarge-us-east-2

[Finished] 1e9eb615 test-mac-arm

[Finished] 1e9eb615 ursa-i9-9960x

[Finished] 1e9eb615 ursa-thinkcentre-m75q

[Finished] 59f99d2c ec2-t3-xlarge-us-east-2

[Finished] 59f99d2c test-mac-arm

[Finished] 59f99d2c ursa-i9-9960x

[Finished] 59f99d2c ursa-thinkcentre-m75q

Supported benchmarks:

ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True

test-mac-arm: Supported benchmark langs: C++, Python, R

ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript

ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
@asfimportasfimport
mentioned this pull request
Nov 24, 2022

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@jorisvandenbosschejorisvandenbosschejorisvandenbossche left review comments

Assignees


No one assigned


Labels

Component: Python

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@AlenkaF@jorisvandenbossche@kou@ursabot