GH-32619: [Python][Docs] Include options for PyArrow build explicitly#34463

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jorisvandenbossche merged 11 commits into from
Mar 10, 2023
Merged

GH-32619: [Python][Docs] Include options for PyArrow build explicitly #34463

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
73779bb
Add two lists of options for building PyArrow
AlenkaF Mar 6, 2023
d33bc73
Apply suggestions from code review - kou
AlenkaF Mar 7, 2023
105ff7d
Remove plasma from the lists
AlenkaF Mar 7, 2023
cc9e784
Add header row to the first table
AlenkaF Mar 7, 2023
2b7f6cf
Add extra column (default value)
AlenkaF Mar 7, 2023
b014ee9
Remove PYARROW_BUNDLE_ARROW_CPP_HEADERS
AlenkaF Mar 8, 2023
ad512e7
Remove PYARROW_BUNDLE_BOOST
AlenkaF Mar 8, 2023
04ec103
Remove PYARROW_WITH_STATIC_PARQUET and PYARROW_PARQUET_USE_SHARED
AlenkaF Mar 8, 2023
a4dd385
Minor typo corrections
AlenkaF Mar 8, 2023
b1ffac9
Change formatting of the variables
AlenkaF Mar 8, 2023
a5f2ce8
Remove PYARROW_BOOST_NAMESPACE, PYARROW_WITH_STATIC_BOOST and PYARROW…
AlenkaF Mar 9, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion ci/appveyor-cpp-build.bat
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -106,7 +106,6 @@ popd

pushd python

set PYARROW_BUNDLE_BOOST=OFF
set PYARROW_CMAKE_GENERATOR=%GENERATOR%
set PYARROW_CXXFLAGS=%ARROW_CXXFLAGS%
set PYARROW_PARALLEL=2
Expand Down
1 change: 0 additions & 1 deletion ci/scripts/python_wheel_windows_build.bat
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -103,7 +103,6 @@ popd
echo "=== (%PYTHON_VERSION%) Building wheel ==="
set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE%
set PYARROW_BUNDLE_ARROW_CPP=ON
set PYARROW_BUNDLE_BOOST=OFF
set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR%
set PYARROW_INSTALL_TESTS=ON
set PYARROW_WITH_DATASET=%ARROW_DATASET%
Expand Down
1 change: 0 additions & 1 deletion dev/tasks/conda-recipes/arrow-cpp/build-pyarrow.sh
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -8,7 +8,6 @@ export ARROW_HOME=$PREFIX
export PARQUET_HOME=$PREFIX
export SETUPTOOLS_SCM_PRETEND_VERSION=$PKG_VERSION
export PYARROW_BUILD_TYPE=release
export PYARROW_BUNDLE_ARROW_CPP_HEADERS=0
export PYARROW_WITH_DATASET=1
export PYARROW_WITH_FLIGHT=1
export PYARROW_WITH_GANDIVA=1
Expand Down
77 changes: 77 additions & 0 deletions docs/source/developers/python.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -586,6 +586,83 @@ Caveats

The Plasma component is not supported on Windows.

Relevant components and environment variables
=============================================

List of relevant Arrow CMake flags and corresponding environment variables
to be used when building PyArrow are:

.. list-table::
:widths: 30 30
:header-rows: 1
Comment thread
AlenkaF marked this conversation as resolved.

* - Arrow flags/options
- Corresponding environment variables for PyArrow
* - ``CMAKE_BUILD_TYPE``
- ``PYARROW_BUILD_TYPE`` (release, debug or relwithdebinfo)
Comment thread
assignUser marked this conversation as resolved.
* - ``ARROW_GCS``
- ``PYARROW_WITH_GCS``
* - ``ARROW_S3``
- ``PYARROW_WITH_S3``
* - ``ARROW_HDFS``
- ``PYARROW_WITH_HDFS``
* - ``ARROW_CUDA``
- ``PYARROW_WITH_CUDA``
* - ``ARROW_SUBSTRAIT``
- ``PYARROW_WITH_SUBSTRAIT``
* - ``ARROW_FLIGHT``
- ``PYARROW_WITH_FLIGHT``
* - ``ARROW_DATASET``
- ``PYARROW_WITH_DATASET``
* - ``ARROW_PARQUET``
- ``PYARROW_WITH_PARQUET``
* - ``PARQUET_REQUIRE_ENCRYPTION``
- ``PYARROW_WITH_PARQUET_ENCRYPTION``
* - ``ARROW_TENSORFLOW``
- ``PYARROW_WITH_TENSORFLOW``
* - ``ARROW_ORC``
- ``PYARROW_WITH_ORC``
* - ``ARROW_GANDIVA``
- ``PYARROW_WITH_GANDIVA``

List of relevant environment variables that can also be used to build
PyArrow are:

.. list-table::
:widths: 20 20 20
:header-rows: 1

* - PyArrow environment variable
- Description
Comment thread
AlenkaF marked this conversation as resolved.
- Default value
* - ``PYARROW_CMAKE_GENERATOR``
- Example: ``'Visual Studio 15 2017 Win64'``
- ``''``
* - ``PYARROW_CMAKE_OPTIONS``
- Extra CMake and Arrow options (ex. ``"-DARROW_SIMD_LEVEL=NONE -DCMAKE_OSX_ARCHITECTURES=x86_64;arm64"``)
- ``''``
* - ``PYARROW_CXXFLAGS``
- Extra C++ compiler flags
- ``''``
* - ``PYARROW_GENERATE_COVERAGE``
- Setting ``Xlinetrace`` flag to true for the Cython compiler
- ``false``
* - ``PYARROW_BUNDLE_ARROW_CPP``
- Bundle the Arrow C++ libraries
- ``0`` (``OFF``)
* - ``PYARROW_BUNDLE_CYTHON_CPP``
- Bundle the C++ files generated by Cython
- ``0`` (``OFF``)
* - ``PYARROW_INSTALL_TESTS``
- Add the test to the python package
- ``1`` (``ON``)
* - ``PYARROW_BUILD_VERBOSE``
- Enable verbose output from Makefile builds
- ``0`` (``OFF``)
* - ``PYARROW_PARALLEL``
- Number of processes used to compile PyArrow’s C++/Cython components
- ``''``

Deleting stale build artifacts
==============================

Expand Down
6 changes: 1 addition & 5 deletions python/CMakeLists.txt
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -116,16 +116,12 @@ if("${CMAKE_SOURCE_DIR}" STREQUAL "${CMAKE_CURRENT_SOURCE_DIR}")
option(PYARROW_BUILD_DATASET "Build the PyArrow Dataset integration" OFF)
option(PYARROW_BUILD_GANDIVA "Build the PyArrow Gandiva integration" OFF)
option(PYARROW_BUILD_PARQUET "Build the PyArrow Parquet integration" OFF)
option(PYARROW_PARQUET_USE_SHARED "Rely on parquet shared libraries where relevant" ON)
option(PYARROW_BUILD_PARQUET_ENCRYPTION
"Build the PyArrow Parquet encryption integration" OFF)
option(PYARROW_BOOST_USE_SHARED
"Rely on Boost shared libraries on linking static parquet" ON)
option(PYARROW_BUILD_PLASMA "Build the PyArrow Plasma integration" OFF)
option(PYARROW_USE_TENSORFLOW "Build PyArrow with TensorFlow support" OFF)
option(PYARROW_BUILD_ORC "Build the PyArrow ORC integration" OFF)
option(PYARROW_BUNDLE_ARROW_CPP "Bundle the Arrow C++ libraries" OFF)
option(PYARROW_BUNDLE_BOOST "Bundle the Boost libraries when we bundle Arrow C++" OFF)
option(PYARROW_BUNDLE_CYTHON_CPP "Bundle the C++ files generated by Cython" OFF)
option(PYARROW_BUNDLE_PLASMA_EXECUTABLE "Bundle the plasma-store-server" ON)
option(PYARROW_GENERATE_COVERAGE "Build with Cython code coverage enabled" OFF)
Expand DownExpand Up@@ -585,7 +581,7 @@ if(PYARROW_BUILD_PARQUET)
DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})
endif()

if(PYARROW_PARQUET_USE_SHARED)
if(ARROW_BUILD_SHARED)
if(PYARROW_BUNDLE_ARROW_CPP)
bundle_arrow_lib(${PARQUET_SHARED_LIB} SO_VERSION ${PARQUET_SO_VERSION})
if(MSVC)
Expand Down
20 changes: 0 additions & 20 deletions python/setup.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -144,8 +144,6 @@ def initialize_options(self):
self.extra_cmake_args = os.environ.get('PYARROW_CMAKE_OPTIONS', '')
self.build_type = os.environ.get('PYARROW_BUILD_TYPE',
'release').lower()
self.boost_namespace = os.environ.get('PYARROW_BOOST_NAMESPACE',
'boost')

self.cmake_cxxflags = os.environ.get('PYARROW_CXXFLAGS', '')

Expand All@@ -171,12 +169,8 @@ def initialize_options(self):
os.environ.get('PYARROW_WITH_DATASET', '0'))
self.with_parquet = strtobool(
os.environ.get('PYARROW_WITH_PARQUET', '0'))
self.with_static_parquet = strtobool(
os.environ.get('PYARROW_WITH_STATIC_PARQUET', '0'))
self.with_parquet_encryption = strtobool(
os.environ.get('PYARROW_WITH_PARQUET_ENCRYPTION', '0'))
self.with_static_boost = strtobool(
os.environ.get('PYARROW_WITH_STATIC_BOOST', '0'))
self.with_plasma = strtobool(
os.environ.get('PYARROW_WITH_PLASMA', '0'))
self.with_tensorflow = strtobool(
Expand All@@ -191,10 +185,6 @@ def initialize_options(self):
os.environ.get('PYARROW_BUNDLE_ARROW_CPP', '0'))
self.bundle_cython_cpp = strtobool(
os.environ.get('PYARROW_BUNDLE_CYTHON_CPP', '0'))
self.bundle_boost = strtobool(
os.environ.get('PYARROW_BUNDLE_BOOST', '0'))
self.bundle_arrow_cpp_headers = strtobool(
os.environ.get('PYARROW_BUNDLE_ARROW_CPP_HEADERS', '1'))
self.bundle_plasma_executable = strtobool(
os.environ.get('PYARROW_BUNDLE_PLASMA_EXECUTABLE', '1'))

Expand DownExpand Up@@ -297,26 +287,16 @@ def append_cmake_bool(value, varname):
append_cmake_bool(self.with_tensorflow, 'PYARROW_USE_TENSORFLOW')
append_cmake_bool(self.bundle_arrow_cpp,
'PYARROW_BUNDLE_ARROW_CPP')
append_cmake_bool(self.bundle_boost,
'PYARROW_BUNDLE_BOOST')
append_cmake_bool(self.bundle_cython_cpp,
'PYARROW_BUNDLE_CYTHON_CPP')
append_cmake_bool(self.bundle_plasma_executable,
'PYARROW_BUNDLE_PLASMA_EXECUTABLE')
append_cmake_bool(self.generate_coverage,
'PYARROW_GENERATE_COVERAGE')
append_cmake_bool(not self.with_static_boost,
'PYARROW_BOOST_USE_SHARED')
append_cmake_bool(not self.with_static_parquet,
'PYARROW_PARQUET_USE_SHARED')

cmake_options.append(
f'-DCMAKE_BUILD_TYPE={self.build_type.lower()}')

if self.boost_namespace != 'boost':
cmake_options.append(
f'-DBoost_NAMESPACE={self.boost_namespace}')

extra_cmake_args = shlex.split(self.extra_cmake_args)

build_tool_args = []
Expand Down
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
GH-32619: [Python][Docs] Include options for PyArrow build explicitly#34463

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

jorisvandenbossche
merged 11 commits into

from

Mar 10, 2023
 Merged


GH-32619: [Python][Docs] Include options for PyArrow build explicitly
#34463
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


11 commits


Select commit
Hold shift + click to select a range
73779bb

Add two lists of options for building PyArrow


AlenkaF Mar 6, 2023
d33bc73

Apply suggestions from code review - kou


AlenkaF Mar 7, 2023
105ff7d

Remove plasma from the lists


AlenkaF Mar 7, 2023
cc9e784

Add header row to the first table


AlenkaF Mar 7, 2023
2b7f6cf

Add extra column (default value)


AlenkaF Mar 7, 2023
b014ee9

Remove PYARROW_BUNDLE_ARROW_CPP_HEADERS


AlenkaF Mar 8, 2023
ad512e7

Remove PYARROW_BUNDLE_BOOST


AlenkaF Mar 8, 2023
04ec103

Remove PYARROW_WITH_STATIC_PARQUET and PYARROW_PARQUET_USE_SHARED


AlenkaF Mar 8, 2023
a4dd385

Minor typo corrections


AlenkaF Mar 8, 2023
b1ffac9

Change formatting of the variables


AlenkaF Mar 8, 2023
a5f2ce8

Remove PYARROW_BOOST_NAMESPACE, PYARROW_WITH_STATIC_BOOST and PYARROW…


AlenkaF Mar 9, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


1 change: 0 additions & 1 deletion
ci/appveyor-cpp-build.bat

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -106,7 +106,6 @@ popd

pushd python

set PYARROW_BUNDLE_BOOST=OFF
set PYARROW_CMAKE_GENERATOR=%GENERATOR%
set PYARROW_CXXFLAGS=%ARROW_CXXFLAGS%
set PYARROW_PARALLEL=2
Expand Down

1 change: 0 additions & 1 deletion
ci/scripts/python_wheel_windows_build.bat

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -103,7 +103,6 @@ popd
echo "=== (%PYTHON_VERSION%) Building wheel ==="
set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE%
set PYARROW_BUNDLE_ARROW_CPP=ON
set PYARROW_BUNDLE_BOOST=OFF
set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR%
set PYARROW_INSTALL_TESTS=ON
set PYARROW_WITH_DATASET=%ARROW_DATASET%
Expand Down

1 change: 0 additions & 1 deletion
dev/tasks/conda-recipes/arrow-cpp/build-pyarrow.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -8,7 +8,6 @@ export ARROW_HOME=$PREFIX
export PARQUET_HOME=$PREFIX
export SETUPTOOLS_SCM_PRETEND_VERSION=$PKG_VERSION
export PYARROW_BUILD_TYPE=release
export PYARROW_BUNDLE_ARROW_CPP_HEADERS=0
export PYARROW_WITH_DATASET=1
export PYARROW_WITH_FLIGHT=1
export PYARROW_WITH_GANDIVA=1
Expand Down

77 changes: 77 additions & 0 deletions
docs/source/developers/python.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -586,6 +586,83 @@ Caveats

The Plasma component is not supported on Windows.

Relevant components and environment variables
=============================================

List of relevant Arrow CMake flags and corresponding environment variables
to be used when building PyArrow are:

.. list-table::
 :widths: 30 30
 :header-rows: 1
Comment thread
AlenkaF marked this conversation as resolved.


 * - Arrow flags/options
 - Corresponding environment variables for PyArrow
 * - ``CMAKE_BUILD_TYPE``
 - ``PYARROW_BUILD_TYPE`` (release, debug or relwithdebinfo)
Comment thread
assignUser marked this conversation as resolved.

 * - ``ARROW_GCS``
 - ``PYARROW_WITH_GCS``
 * - ``ARROW_S3``
 - ``PYARROW_WITH_S3``
 * - ``ARROW_HDFS``
 - ``PYARROW_WITH_HDFS``
 * - ``ARROW_CUDA``
 - ``PYARROW_WITH_CUDA``
 * - ``ARROW_SUBSTRAIT``
 - ``PYARROW_WITH_SUBSTRAIT``
 * - ``ARROW_FLIGHT``
 - ``PYARROW_WITH_FLIGHT``
 * - ``ARROW_DATASET``
 - ``PYARROW_WITH_DATASET``
 * - ``ARROW_PARQUET``
 - ``PYARROW_WITH_PARQUET``
 * - ``PARQUET_REQUIRE_ENCRYPTION``
 - ``PYARROW_WITH_PARQUET_ENCRYPTION``
 * - ``ARROW_TENSORFLOW``
 - ``PYARROW_WITH_TENSORFLOW``
 * - ``ARROW_ORC``
 - ``PYARROW_WITH_ORC``
 * - ``ARROW_GANDIVA``
 - ``PYARROW_WITH_GANDIVA``

List of relevant environment variables that can also be used to build
PyArrow are:

.. list-table::
 :widths: 20 20 20
 :header-rows: 1

 * - PyArrow environment variable
 - Description
Comment thread
AlenkaF marked this conversation as resolved.

 - Default value
 * - ``PYARROW_CMAKE_GENERATOR``
 - Example: ``'Visual Studio 15 2017 Win64'``
 - ``''``
 * - ``PYARROW_CMAKE_OPTIONS``
 - Extra CMake and Arrow options (ex. ``"-DARROW_SIMD_LEVEL=NONE -DCMAKE_OSX_ARCHITECTURES=x86_64;arm64"``)
 - ``''``
 * - ``PYARROW_CXXFLAGS``
 - Extra C++ compiler flags
 - ``''``
 * - ``PYARROW_GENERATE_COVERAGE``
 - Setting ``Xlinetrace`` flag to true for the Cython compiler
 - ``false``
 * - ``PYARROW_BUNDLE_ARROW_CPP``
 - Bundle the Arrow C++ libraries
 - ``0`` (``OFF``)
 * - ``PYARROW_BUNDLE_CYTHON_CPP``
 - Bundle the C++ files generated by Cython
 - ``0`` (``OFF``)
 * - ``PYARROW_INSTALL_TESTS``
 - Add the test to the python package
 - ``1`` (``ON``)
 * - ``PYARROW_BUILD_VERBOSE``
 - Enable verbose output from Makefile builds
 - ``0`` (``OFF``)
 * - ``PYARROW_PARALLEL``
 - Number of processes used to compile PyArrow’s C++/Cython components
 - ``''``

Deleting stale build artifacts
==============================

Expand Down

6 changes: 1 addition & 5 deletions
python/CMakeLists.txt

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -116,16 +116,12 @@ if("${CMAKE_SOURCE_DIR}" STREQUAL "${CMAKE_CURRENT_SOURCE_DIR}")
 option(PYARROW_BUILD_DATASET "Build the PyArrow Dataset integration" OFF)
 option(PYARROW_BUILD_GANDIVA "Build the PyArrow Gandiva integration" OFF)
 option(PYARROW_BUILD_PARQUET "Build the PyArrow Parquet integration" OFF)
 option(PYARROW_PARQUET_USE_SHARED "Rely on parquet shared libraries where relevant" ON)
 option(PYARROW_BUILD_PARQUET_ENCRYPTION
 "Build the PyArrow Parquet encryption integration" OFF)
 option(PYARROW_BOOST_USE_SHARED
 "Rely on Boost shared libraries on linking static parquet" ON)
 option(PYARROW_BUILD_PLASMA "Build the PyArrow Plasma integration" OFF)
 option(PYARROW_USE_TENSORFLOW "Build PyArrow with TensorFlow support" OFF)
 option(PYARROW_BUILD_ORC "Build the PyArrow ORC integration" OFF)
 option(PYARROW_BUNDLE_ARROW_CPP "Bundle the Arrow C++ libraries" OFF)
 option(PYARROW_BUNDLE_BOOST "Bundle the Boost libraries when we bundle Arrow C++" OFF)
 option(PYARROW_BUNDLE_CYTHON_CPP "Bundle the C++ files generated by Cython" OFF)
 option(PYARROW_BUNDLE_PLASMA_EXECUTABLE "Bundle the plasma-store-server" ON)
 option(PYARROW_GENERATE_COVERAGE "Build with Cython code coverage enabled" OFF)
Expand DownExpand Up@@ -585,7 +581,7 @@ if(PYARROW_BUILD_PARQUET)
 DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})
 endif()

 if(PYARROW_PARQUET_USE_SHARED)
 if(ARROW_BUILD_SHARED)
 if(PYARROW_BUNDLE_ARROW_CPP)
 bundle_arrow_lib(${PARQUET_SHARED_LIB} SO_VERSION ${PARQUET_SO_VERSION})
 if(MSVC)
Expand Down

20 changes: 0 additions & 20 deletions
python/setup.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -144,8 +144,6 @@ def initialize_options(self):
 self.extra_cmake_args = os.environ.get('PYARROW_CMAKE_OPTIONS', '')
 self.build_type = os.environ.get('PYARROW_BUILD_TYPE',
 'release').lower()
 self.boost_namespace = os.environ.get('PYARROW_BOOST_NAMESPACE',
 'boost')

 self.cmake_cxxflags = os.environ.get('PYARROW_CXXFLAGS', '')

Expand All@@ -171,12 +169,8 @@ def initialize_options(self):
 os.environ.get('PYARROW_WITH_DATASET', '0'))
 self.with_parquet = strtobool(
 os.environ.get('PYARROW_WITH_PARQUET', '0'))
 self.with_static_parquet = strtobool(
 os.environ.get('PYARROW_WITH_STATIC_PARQUET', '0'))
 self.with_parquet_encryption = strtobool(
 os.environ.get('PYARROW_WITH_PARQUET_ENCRYPTION', '0'))
 self.with_static_boost = strtobool(
 os.environ.get('PYARROW_WITH_STATIC_BOOST', '0'))
 self.with_plasma = strtobool(
 os.environ.get('PYARROW_WITH_PLASMA', '0'))
 self.with_tensorflow = strtobool(
Expand All@@ -191,10 +185,6 @@ def initialize_options(self):
 os.environ.get('PYARROW_BUNDLE_ARROW_CPP', '0'))
 self.bundle_cython_cpp = strtobool(
 os.environ.get('PYARROW_BUNDLE_CYTHON_CPP', '0'))
 self.bundle_boost = strtobool(
 os.environ.get('PYARROW_BUNDLE_BOOST', '0'))
 self.bundle_arrow_cpp_headers = strtobool(
 os.environ.get('PYARROW_BUNDLE_ARROW_CPP_HEADERS', '1'))
 self.bundle_plasma_executable = strtobool(
 os.environ.get('PYARROW_BUNDLE_PLASMA_EXECUTABLE', '1'))

Expand DownExpand Up@@ -297,26 +287,16 @@ def append_cmake_bool(value, varname):
 append_cmake_bool(self.with_tensorflow, 'PYARROW_USE_TENSORFLOW')
 append_cmake_bool(self.bundle_arrow_cpp,
 'PYARROW_BUNDLE_ARROW_CPP')
 append_cmake_bool(self.bundle_boost,
 'PYARROW_BUNDLE_BOOST')
 append_cmake_bool(self.bundle_cython_cpp,
 'PYARROW_BUNDLE_CYTHON_CPP')
 append_cmake_bool(self.bundle_plasma_executable,
 'PYARROW_BUNDLE_PLASMA_EXECUTABLE')
 append_cmake_bool(self.generate_coverage,
 'PYARROW_GENERATE_COVERAGE')
 append_cmake_bool(not self.with_static_boost,
 'PYARROW_BOOST_USE_SHARED')
 append_cmake_bool(not self.with_static_parquet,
 'PYARROW_PARQUET_USE_SHARED')

 cmake_options.append(
 f'-DCMAKE_BUILD_TYPE={self.build_type.lower()}')

 if self.boost_namespace != 'boost':
 cmake_options.append(
 f'-DBoost_NAMESPACE={self.boost_namespace}')

 extra_cmake_args = shlex.split(self.extra_cmake_args)

 build_tool_args = []
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-32619: [Python][Docs] Include options for PyArrow build explicitly#34463

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

jorisvandenbossche
merged 11 commits into

from

Mar 10, 2023
 Merged


GH-32619: [Python][Docs] Include options for PyArrow build explicitly
#34463
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


11 commits


Select commit
Hold shift + click to select a range
73779bb

Add two lists of options for building PyArrow


AlenkaF Mar 6, 2023
d33bc73

Apply suggestions from code review - kou


AlenkaF Mar 7, 2023
105ff7d

Remove plasma from the lists


AlenkaF Mar 7, 2023
cc9e784

Add header row to the first table


AlenkaF Mar 7, 2023
2b7f6cf

Add extra column (default value)


AlenkaF Mar 7, 2023
b014ee9

Remove PYARROW_BUNDLE_ARROW_CPP_HEADERS


AlenkaF Mar 8, 2023
ad512e7

Remove PYARROW_BUNDLE_BOOST


AlenkaF Mar 8, 2023
04ec103

Remove PYARROW_WITH_STATIC_PARQUET and PYARROW_PARQUET_USE_SHARED


AlenkaF Mar 8, 2023
a4dd385

Minor typo corrections


AlenkaF Mar 8, 2023
b1ffac9

Change formatting of the variables


AlenkaF Mar 8, 2023
a5f2ce8

Remove PYARROW_BOOST_NAMESPACE, PYARROW_WITH_STATIC_BOOST and PYARROW…


AlenkaF Mar 9, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


1 change: 0 additions & 1 deletion
ci/appveyor-cpp-build.bat

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -106,7 +106,6 @@ popd

pushd python

set PYARROW_BUNDLE_BOOST=OFF
set PYARROW_CMAKE_GENERATOR=%GENERATOR%
set PYARROW_CXXFLAGS=%ARROW_CXXFLAGS%
set PYARROW_PARALLEL=2
Expand Down

1 change: 0 additions & 1 deletion
ci/scripts/python_wheel_windows_build.bat

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -103,7 +103,6 @@ popd
echo "=== (%PYTHON_VERSION%) Building wheel ==="
set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE%
set PYARROW_BUNDLE_ARROW_CPP=ON
set PYARROW_BUNDLE_BOOST=OFF
set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR%
set PYARROW_INSTALL_TESTS=ON
set PYARROW_WITH_DATASET=%ARROW_DATASET%
Expand Down

1 change: 0 additions & 1 deletion
dev/tasks/conda-recipes/arrow-cpp/build-pyarrow.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -8,7 +8,6 @@ export ARROW_HOME=$PREFIX
export PARQUET_HOME=$PREFIX
export SETUPTOOLS_SCM_PRETEND_VERSION=$PKG_VERSION
export PYARROW_BUILD_TYPE=release
export PYARROW_BUNDLE_ARROW_CPP_HEADERS=0
export PYARROW_WITH_DATASET=1
export PYARROW_WITH_FLIGHT=1
export PYARROW_WITH_GANDIVA=1
Expand Down

77 changes: 77 additions & 0 deletions
docs/source/developers/python.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -586,6 +586,83 @@ Caveats

The Plasma component is not supported on Windows.

Relevant components and environment variables
=============================================

List of relevant Arrow CMake flags and corresponding environment variables
to be used when building PyArrow are:

.. list-table::
 :widths: 30 30
 :header-rows: 1
Comment thread
AlenkaF marked this conversation as resolved.


 * - Arrow flags/options
 - Corresponding environment variables for PyArrow
 * - ``CMAKE_BUILD_TYPE``
 - ``PYARROW_BUILD_TYPE`` (release, debug or relwithdebinfo)
Comment thread
assignUser marked this conversation as resolved.

 * - ``ARROW_GCS``
 - ``PYARROW_WITH_GCS``
 * - ``ARROW_S3``
 - ``PYARROW_WITH_S3``
 * - ``ARROW_HDFS``
 - ``PYARROW_WITH_HDFS``
 * - ``ARROW_CUDA``
 - ``PYARROW_WITH_CUDA``
 * - ``ARROW_SUBSTRAIT``
 - ``PYARROW_WITH_SUBSTRAIT``
 * - ``ARROW_FLIGHT``
 - ``PYARROW_WITH_FLIGHT``
 * - ``ARROW_DATASET``
 - ``PYARROW_WITH_DATASET``
 * - ``ARROW_PARQUET``
 - ``PYARROW_WITH_PARQUET``
 * - ``PARQUET_REQUIRE_ENCRYPTION``
 - ``PYARROW_WITH_PARQUET_ENCRYPTION``
 * - ``ARROW_TENSORFLOW``
 - ``PYARROW_WITH_TENSORFLOW``
 * - ``ARROW_ORC``
 - ``PYARROW_WITH_ORC``
 * - ``ARROW_GANDIVA``
 - ``PYARROW_WITH_GANDIVA``

List of relevant environment variables that can also be used to build
PyArrow are:

.. list-table::
 :widths: 20 20 20
 :header-rows: 1

 * - PyArrow environment variable
 - Description
Comment thread
AlenkaF marked this conversation as resolved.

 - Default value
 * - ``PYARROW_CMAKE_GENERATOR``
 - Example: ``'Visual Studio 15 2017 Win64'``
 - ``''``
 * - ``PYARROW_CMAKE_OPTIONS``
 - Extra CMake and Arrow options (ex. ``"-DARROW_SIMD_LEVEL=NONE -DCMAKE_OSX_ARCHITECTURES=x86_64;arm64"``)
 - ``''``
 * - ``PYARROW_CXXFLAGS``
 - Extra C++ compiler flags
 - ``''``
 * - ``PYARROW_GENERATE_COVERAGE``
 - Setting ``Xlinetrace`` flag to true for the Cython compiler
 - ``false``
 * - ``PYARROW_BUNDLE_ARROW_CPP``
 - Bundle the Arrow C++ libraries
 - ``0`` (``OFF``)
 * - ``PYARROW_BUNDLE_CYTHON_CPP``
 - Bundle the C++ files generated by Cython
 - ``0`` (``OFF``)
 * - ``PYARROW_INSTALL_TESTS``
 - Add the test to the python package
 - ``1`` (``ON``)
 * - ``PYARROW_BUILD_VERBOSE``
 - Enable verbose output from Makefile builds
 - ``0`` (``OFF``)
 * - ``PYARROW_PARALLEL``
 - Number of processes used to compile PyArrow’s C++/Cython components
 - ``''``

Deleting stale build artifacts
==============================

Expand Down

6 changes: 1 addition & 5 deletions
python/CMakeLists.txt

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -116,16 +116,12 @@ if("${CMAKE_SOURCE_DIR}" STREQUAL "${CMAKE_CURRENT_SOURCE_DIR}")
 option(PYARROW_BUILD_DATASET "Build the PyArrow Dataset integration" OFF)
 option(PYARROW_BUILD_GANDIVA "Build the PyArrow Gandiva integration" OFF)
 option(PYARROW_BUILD_PARQUET "Build the PyArrow Parquet integration" OFF)
 option(PYARROW_PARQUET_USE_SHARED "Rely on parquet shared libraries where relevant" ON)
 option(PYARROW_BUILD_PARQUET_ENCRYPTION
 "Build the PyArrow Parquet encryption integration" OFF)
 option(PYARROW_BOOST_USE_SHARED
 "Rely on Boost shared libraries on linking static parquet" ON)
 option(PYARROW_BUILD_PLASMA "Build the PyArrow Plasma integration" OFF)
 option(PYARROW_USE_TENSORFLOW "Build PyArrow with TensorFlow support" OFF)
 option(PYARROW_BUILD_ORC "Build the PyArrow ORC integration" OFF)
 option(PYARROW_BUNDLE_ARROW_CPP "Bundle the Arrow C++ libraries" OFF)
 option(PYARROW_BUNDLE_BOOST "Bundle the Boost libraries when we bundle Arrow C++" OFF)
 option(PYARROW_BUNDLE_CYTHON_CPP "Bundle the C++ files generated by Cython" OFF)
 option(PYARROW_BUNDLE_PLASMA_EXECUTABLE "Bundle the plasma-store-server" ON)
 option(PYARROW_GENERATE_COVERAGE "Build with Cython code coverage enabled" OFF)
Expand DownExpand Up@@ -585,7 +581,7 @@ if(PYARROW_BUILD_PARQUET)
 DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})
 endif()

 if(PYARROW_PARQUET_USE_SHARED)
 if(ARROW_BUILD_SHARED)
 if(PYARROW_BUNDLE_ARROW_CPP)
 bundle_arrow_lib(${PARQUET_SHARED_LIB} SO_VERSION ${PARQUET_SO_VERSION})
 if(MSVC)
Expand Down

20 changes: 0 additions & 20 deletions
python/setup.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -144,8 +144,6 @@ def initialize_options(self):
 self.extra_cmake_args = os.environ.get('PYARROW_CMAKE_OPTIONS', '')
 self.build_type = os.environ.get('PYARROW_BUILD_TYPE',
 'release').lower()
 self.boost_namespace = os.environ.get('PYARROW_BOOST_NAMESPACE',
 'boost')

 self.cmake_cxxflags = os.environ.get('PYARROW_CXXFLAGS', '')

Expand All@@ -171,12 +169,8 @@ def initialize_options(self):
 os.environ.get('PYARROW_WITH_DATASET', '0'))
 self.with_parquet = strtobool(
 os.environ.get('PYARROW_WITH_PARQUET', '0'))
 self.with_static_parquet = strtobool(
 os.environ.get('PYARROW_WITH_STATIC_PARQUET', '0'))
 self.with_parquet_encryption = strtobool(
 os.environ.get('PYARROW_WITH_PARQUET_ENCRYPTION', '0'))
 self.with_static_boost = strtobool(
 os.environ.get('PYARROW_WITH_STATIC_BOOST', '0'))
 self.with_plasma = strtobool(
 os.environ.get('PYARROW_WITH_PLASMA', '0'))
 self.with_tensorflow = strtobool(
Expand All@@ -191,10 +185,6 @@ def initialize_options(self):
 os.environ.get('PYARROW_BUNDLE_ARROW_CPP', '0'))
 self.bundle_cython_cpp = strtobool(
 os.environ.get('PYARROW_BUNDLE_CYTHON_CPP', '0'))
 self.bundle_boost = strtobool(
 os.environ.get('PYARROW_BUNDLE_BOOST', '0'))
 self.bundle_arrow_cpp_headers = strtobool(
 os.environ.get('PYARROW_BUNDLE_ARROW_CPP_HEADERS', '1'))
 self.bundle_plasma_executable = strtobool(
 os.environ.get('PYARROW_BUNDLE_PLASMA_EXECUTABLE', '1'))

Expand DownExpand Up@@ -297,26 +287,16 @@ def append_cmake_bool(value, varname):
 append_cmake_bool(self.with_tensorflow, 'PYARROW_USE_TENSORFLOW')
 append_cmake_bool(self.bundle_arrow_cpp,
 'PYARROW_BUNDLE_ARROW_CPP')
 append_cmake_bool(self.bundle_boost,
 'PYARROW_BUNDLE_BOOST')
 append_cmake_bool(self.bundle_cython_cpp,
 'PYARROW_BUNDLE_CYTHON_CPP')
 append_cmake_bool(self.bundle_plasma_executable,
 'PYARROW_BUNDLE_PLASMA_EXECUTABLE')
 append_cmake_bool(self.generate_coverage,
 'PYARROW_GENERATE_COVERAGE')
 append_cmake_bool(not self.with_static_boost,
 'PYARROW_BOOST_USE_SHARED')
 append_cmake_bool(not self.with_static_parquet,
 'PYARROW_PARQUET_USE_SHARED')

 cmake_options.append(
 f'-DCMAKE_BUILD_TYPE={self.build_type.lower()}')

 if self.boost_namespace != 'boost':
 cmake_options.append(
 f'-DBoost_NAMESPACE={self.boost_namespace}')

 extra_cmake_args = shlex.split(self.extra_cmake_args)

 build_tool_args = []
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-32619: [Python][Docs] Include options for PyArrow build explicitly#34463

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

jorisvandenbossche
merged 11 commits into

from

Mar 10, 2023
 Merged


GH-32619: [Python][Docs] Include options for PyArrow build explicitly
#34463
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


11 commits


Select commit
Hold shift + click to select a range
73779bb

Add two lists of options for building PyArrow


AlenkaF Mar 6, 2023
d33bc73

Apply suggestions from code review - kou


AlenkaF Mar 7, 2023
105ff7d

Remove plasma from the lists


AlenkaF Mar 7, 2023
cc9e784

Add header row to the first table


AlenkaF Mar 7, 2023
2b7f6cf

Add extra column (default value)


AlenkaF Mar 7, 2023
b014ee9

Remove PYARROW_BUNDLE_ARROW_CPP_HEADERS


AlenkaF Mar 8, 2023
ad512e7

Remove PYARROW_BUNDLE_BOOST


AlenkaF Mar 8, 2023
04ec103

Remove PYARROW_WITH_STATIC_PARQUET and PYARROW_PARQUET_USE_SHARED


AlenkaF Mar 8, 2023
a4dd385

Minor typo corrections


AlenkaF Mar 8, 2023
b1ffac9

Change formatting of the variables


AlenkaF Mar 8, 2023
a5f2ce8

Remove PYARROW_BOOST_NAMESPACE, PYARROW_WITH_STATIC_BOOST and PYARROW…


AlenkaF Mar 9, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


1 change: 0 additions & 1 deletion
ci/appveyor-cpp-build.bat

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -106,7 +106,6 @@ popd

pushd python

set PYARROW_BUNDLE_BOOST=OFF
set PYARROW_CMAKE_GENERATOR=%GENERATOR%
set PYARROW_CXXFLAGS=%ARROW_CXXFLAGS%
set PYARROW_PARALLEL=2
Expand Down

1 change: 0 additions & 1 deletion
ci/scripts/python_wheel_windows_build.bat

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -103,7 +103,6 @@ popd
echo "=== (%PYTHON_VERSION%) Building wheel ==="
set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE%
set PYARROW_BUNDLE_ARROW_CPP=ON
set PYARROW_BUNDLE_BOOST=OFF
set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR%
set PYARROW_INSTALL_TESTS=ON
set PYARROW_WITH_DATASET=%ARROW_DATASET%
Expand Down

1 change: 0 additions & 1 deletion
dev/tasks/conda-recipes/arrow-cpp/build-pyarrow.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -8,7 +8,6 @@ export ARROW_HOME=$PREFIX
export PARQUET_HOME=$PREFIX
export SETUPTOOLS_SCM_PRETEND_VERSION=$PKG_VERSION
export PYARROW_BUILD_TYPE=release
export PYARROW_BUNDLE_ARROW_CPP_HEADERS=0
export PYARROW_WITH_DATASET=1
export PYARROW_WITH_FLIGHT=1
export PYARROW_WITH_GANDIVA=1
Expand Down

77 changes: 77 additions & 0 deletions
docs/source/developers/python.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -586,6 +586,83 @@ Caveats

The Plasma component is not supported on Windows.

Relevant components and environment variables
=============================================

List of relevant Arrow CMake flags and corresponding environment variables
to be used when building PyArrow are:

.. list-table::
 :widths: 30 30
 :header-rows: 1
Comment thread
AlenkaF marked this conversation as resolved.


 * - Arrow flags/options
 - Corresponding environment variables for PyArrow
 * - ``CMAKE_BUILD_TYPE``
 - ``PYARROW_BUILD_TYPE`` (release, debug or relwithdebinfo)
Comment thread
assignUser marked this conversation as resolved.

 * - ``ARROW_GCS``
 - ``PYARROW_WITH_GCS``
 * - ``ARROW_S3``
 - ``PYARROW_WITH_S3``
 * - ``ARROW_HDFS``
 - ``PYARROW_WITH_HDFS``
 * - ``ARROW_CUDA``
 - ``PYARROW_WITH_CUDA``
 * - ``ARROW_SUBSTRAIT``
 - ``PYARROW_WITH_SUBSTRAIT``
 * - ``ARROW_FLIGHT``
 - ``PYARROW_WITH_FLIGHT``
 * - ``ARROW_DATASET``
 - ``PYARROW_WITH_DATASET``
 * - ``ARROW_PARQUET``
 - ``PYARROW_WITH_PARQUET``
 * - ``PARQUET_REQUIRE_ENCRYPTION``
 - ``PYARROW_WITH_PARQUET_ENCRYPTION``
 * - ``ARROW_TENSORFLOW``
 - ``PYARROW_WITH_TENSORFLOW``
 * - ``ARROW_ORC``
 - ``PYARROW_WITH_ORC``
 * - ``ARROW_GANDIVA``
 - ``PYARROW_WITH_GANDIVA``

List of relevant environment variables that can also be used to build
PyArrow are:

.. list-table::
 :widths: 20 20 20
 :header-rows: 1

 * - PyArrow environment variable
 - Description
Comment thread
AlenkaF marked this conversation as resolved.

 - Default value
 * - ``PYARROW_CMAKE_GENERATOR``
 - Example: ``'Visual Studio 15 2017 Win64'``
 - ``''``
 * - ``PYARROW_CMAKE_OPTIONS``
 - Extra CMake and Arrow options (ex. ``"-DARROW_SIMD_LEVEL=NONE -DCMAKE_OSX_ARCHITECTURES=x86_64;arm64"``)
 - ``''``
 * - ``PYARROW_CXXFLAGS``
 - Extra C++ compiler flags
 - ``''``
 * - ``PYARROW_GENERATE_COVERAGE``
 - Setting ``Xlinetrace`` flag to true for the Cython compiler
 - ``false``
 * - ``PYARROW_BUNDLE_ARROW_CPP``
 - Bundle the Arrow C++ libraries
 - ``0`` (``OFF``)
 * - ``PYARROW_BUNDLE_CYTHON_CPP``
 - Bundle the C++ files generated by Cython
 - ``0`` (``OFF``)
 * - ``PYARROW_INSTALL_TESTS``
 - Add the test to the python package
 - ``1`` (``ON``)
 * - ``PYARROW_BUILD_VERBOSE``
 - Enable verbose output from Makefile builds
 - ``0`` (``OFF``)
 * - ``PYARROW_PARALLEL``
 - Number of processes used to compile PyArrow’s C++/Cython components
 - ``''``

Deleting stale build artifacts
==============================

Expand Down

6 changes: 1 addition & 5 deletions
python/CMakeLists.txt

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -116,16 +116,12 @@ if("${CMAKE_SOURCE_DIR}" STREQUAL "${CMAKE_CURRENT_SOURCE_DIR}")
 option(PYARROW_BUILD_DATASET "Build the PyArrow Dataset integration" OFF)
 option(PYARROW_BUILD_GANDIVA "Build the PyArrow Gandiva integration" OFF)
 option(PYARROW_BUILD_PARQUET "Build the PyArrow Parquet integration" OFF)
 option(PYARROW_PARQUET_USE_SHARED "Rely on parquet shared libraries where relevant" ON)
 option(PYARROW_BUILD_PARQUET_ENCRYPTION
 "Build the PyArrow Parquet encryption integration" OFF)
 option(PYARROW_BOOST_USE_SHARED
 "Rely on Boost shared libraries on linking static parquet" ON)
 option(PYARROW_BUILD_PLASMA "Build the PyArrow Plasma integration" OFF)
 option(PYARROW_USE_TENSORFLOW "Build PyArrow with TensorFlow support" OFF)
 option(PYARROW_BUILD_ORC "Build the PyArrow ORC integration" OFF)
 option(PYARROW_BUNDLE_ARROW_CPP "Bundle the Arrow C++ libraries" OFF)
 option(PYARROW_BUNDLE_BOOST "Bundle the Boost libraries when we bundle Arrow C++" OFF)
 option(PYARROW_BUNDLE_CYTHON_CPP "Bundle the C++ files generated by Cython" OFF)
 option(PYARROW_BUNDLE_PLASMA_EXECUTABLE "Bundle the plasma-store-server" ON)
 option(PYARROW_GENERATE_COVERAGE "Build with Cython code coverage enabled" OFF)
Expand DownExpand Up@@ -585,7 +581,7 @@ if(PYARROW_BUILD_PARQUET)
 DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})
 endif()

 if(PYARROW_PARQUET_USE_SHARED)
 if(ARROW_BUILD_SHARED)
 if(PYARROW_BUNDLE_ARROW_CPP)
 bundle_arrow_lib(${PARQUET_SHARED_LIB} SO_VERSION ${PARQUET_SO_VERSION})
 if(MSVC)
Expand Down

20 changes: 0 additions & 20 deletions
python/setup.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -144,8 +144,6 @@ def initialize_options(self):
 self.extra_cmake_args = os.environ.get('PYARROW_CMAKE_OPTIONS', '')
 self.build_type = os.environ.get('PYARROW_BUILD_TYPE',
 'release').lower()
 self.boost_namespace = os.environ.get('PYARROW_BOOST_NAMESPACE',
 'boost')

 self.cmake_cxxflags = os.environ.get('PYARROW_CXXFLAGS', '')

Expand All@@ -171,12 +169,8 @@ def initialize_options(self):
 os.environ.get('PYARROW_WITH_DATASET', '0'))
 self.with_parquet = strtobool(
 os.environ.get('PYARROW_WITH_PARQUET', '0'))
 self.with_static_parquet = strtobool(
 os.environ.get('PYARROW_WITH_STATIC_PARQUET', '0'))
 self.with_parquet_encryption = strtobool(
 os.environ.get('PYARROW_WITH_PARQUET_ENCRYPTION', '0'))
 self.with_static_boost = strtobool(
 os.environ.get('PYARROW_WITH_STATIC_BOOST', '0'))
 self.with_plasma = strtobool(
 os.environ.get('PYARROW_WITH_PLASMA', '0'))
 self.with_tensorflow = strtobool(
Expand All@@ -191,10 +185,6 @@ def initialize_options(self):
 os.environ.get('PYARROW_BUNDLE_ARROW_CPP', '0'))
 self.bundle_cython_cpp = strtobool(
 os.environ.get('PYARROW_BUNDLE_CYTHON_CPP', '0'))
 self.bundle_boost = strtobool(
 os.environ.get('PYARROW_BUNDLE_BOOST', '0'))
 self.bundle_arrow_cpp_headers = strtobool(
 os.environ.get('PYARROW_BUNDLE_ARROW_CPP_HEADERS', '1'))
 self.bundle_plasma_executable = strtobool(
 os.environ.get('PYARROW_BUNDLE_PLASMA_EXECUTABLE', '1'))

Expand DownExpand Up@@ -297,26 +287,16 @@ def append_cmake_bool(value, varname):
 append_cmake_bool(self.with_tensorflow, 'PYARROW_USE_TENSORFLOW')
 append_cmake_bool(self.bundle_arrow_cpp,
 'PYARROW_BUNDLE_ARROW_CPP')
 append_cmake_bool(self.bundle_boost,
 'PYARROW_BUNDLE_BOOST')
 append_cmake_bool(self.bundle_cython_cpp,
 'PYARROW_BUNDLE_CYTHON_CPP')
 append_cmake_bool(self.bundle_plasma_executable,
 'PYARROW_BUNDLE_PLASMA_EXECUTABLE')
 append_cmake_bool(self.generate_coverage,
 'PYARROW_GENERATE_COVERAGE')
 append_cmake_bool(not self.with_static_boost,
 'PYARROW_BOOST_USE_SHARED')
 append_cmake_bool(not self.with_static_parquet,
 'PYARROW_PARQUET_USE_SHARED')

 cmake_options.append(
 f'-DCMAKE_BUILD_TYPE={self.build_type.lower()}')

 if self.boost_namespace != 'boost':
 cmake_options.append(
 f'-DBoost_NAMESPACE={self.boost_namespace}')

 extra_cmake_args = shlex.split(self.extra_cmake_args)

 build_tool_args = []
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
GH-32619: [Python][Docs] Include options for PyArrow build explicitly#34463

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

jorisvandenbossche
merged 11 commits into

from

Mar 10, 2023
 Merged


GH-32619: [Python][Docs] Include options for PyArrow build explicitly
#34463
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


11 commits


Select commit
Hold shift + click to select a range
73779bb

Add two lists of options for building PyArrow


AlenkaF Mar 6, 2023
d33bc73

Apply suggestions from code review - kou


AlenkaF Mar 7, 2023
105ff7d

Remove plasma from the lists


AlenkaF Mar 7, 2023
cc9e784

Add header row to the first table


AlenkaF Mar 7, 2023
2b7f6cf

Add extra column (default value)


AlenkaF Mar 7, 2023
b014ee9

Remove PYARROW_BUNDLE_ARROW_CPP_HEADERS


AlenkaF Mar 8, 2023
ad512e7

Remove PYARROW_BUNDLE_BOOST


AlenkaF Mar 8, 2023
04ec103

Remove PYARROW_WITH_STATIC_PARQUET and PYARROW_PARQUET_USE_SHARED


AlenkaF Mar 8, 2023
a4dd385

Minor typo corrections


AlenkaF Mar 8, 2023
b1ffac9

Change formatting of the variables


AlenkaF Mar 8, 2023
a5f2ce8

Remove PYARROW_BOOST_NAMESPACE, PYARROW_WITH_STATIC_BOOST and PYARROW…


AlenkaF Mar 9, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


1 change: 0 additions & 1 deletion
ci/appveyor-cpp-build.bat

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -106,7 +106,6 @@ popd

pushd python

set PYARROW_BUNDLE_BOOST=OFF
set PYARROW_CMAKE_GENERATOR=%GENERATOR%
set PYARROW_CXXFLAGS=%ARROW_CXXFLAGS%
set PYARROW_PARALLEL=2
Expand Down

1 change: 0 additions & 1 deletion
ci/scripts/python_wheel_windows_build.bat

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -103,7 +103,6 @@ popd
echo "=== (%PYTHON_VERSION%) Building wheel ==="
set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE%
set PYARROW_BUNDLE_ARROW_CPP=ON
set PYARROW_BUNDLE_BOOST=OFF
set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR%
set PYARROW_INSTALL_TESTS=ON
set PYARROW_WITH_DATASET=%ARROW_DATASET%
Expand Down

1 change: 0 additions & 1 deletion
dev/tasks/conda-recipes/arrow-cpp/build-pyarrow.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -8,7 +8,6 @@ export ARROW_HOME=$PREFIX
export PARQUET_HOME=$PREFIX
export SETUPTOOLS_SCM_PRETEND_VERSION=$PKG_VERSION
export PYARROW_BUILD_TYPE=release
export PYARROW_BUNDLE_ARROW_CPP_HEADERS=0
export PYARROW_WITH_DATASET=1
export PYARROW_WITH_FLIGHT=1
export PYARROW_WITH_GANDIVA=1
Expand Down

77 changes: 77 additions & 0 deletions
docs/source/developers/python.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -586,6 +586,83 @@ Caveats

The Plasma component is not supported on Windows.

Relevant components and environment variables
=============================================

List of relevant Arrow CMake flags and corresponding environment variables
to be used when building PyArrow are:

.. list-table::
 :widths: 30 30
 :header-rows: 1
Comment thread
AlenkaF marked this conversation as resolved.


 * - Arrow flags/options
 - Corresponding environment variables for PyArrow
 * - ``CMAKE_BUILD_TYPE``
 - ``PYARROW_BUILD_TYPE`` (release, debug or relwithdebinfo)
Comment thread
assignUser marked this conversation as resolved.

 * - ``ARROW_GCS``
 - ``PYARROW_WITH_GCS``
 * - ``ARROW_S3``
 - ``PYARROW_WITH_S3``
 * - ``ARROW_HDFS``
 - ``PYARROW_WITH_HDFS``
 * - ``ARROW_CUDA``
 - ``PYARROW_WITH_CUDA``
 * - ``ARROW_SUBSTRAIT``
 - ``PYARROW_WITH_SUBSTRAIT``
 * - ``ARROW_FLIGHT``
 - ``PYARROW_WITH_FLIGHT``
 * - ``ARROW_DATASET``
 - ``PYARROW_WITH_DATASET``
 * - ``ARROW_PARQUET``
 - ``PYARROW_WITH_PARQUET``
 * - ``PARQUET_REQUIRE_ENCRYPTION``
 - ``PYARROW_WITH_PARQUET_ENCRYPTION``
 * - ``ARROW_TENSORFLOW``
 - ``PYARROW_WITH_TENSORFLOW``
 * - ``ARROW_ORC``
 - ``PYARROW_WITH_ORC``
 * - ``ARROW_GANDIVA``
 - ``PYARROW_WITH_GANDIVA``

List of relevant environment variables that can also be used to build
PyArrow are:

.. list-table::
 :widths: 20 20 20
 :header-rows: 1

 * - PyArrow environment variable
 - Description
Comment thread
AlenkaF marked this conversation as resolved.

 - Default value
 * - ``PYARROW_CMAKE_GENERATOR``
 - Example: ``'Visual Studio 15 2017 Win64'``
 - ``''``
 * - ``PYARROW_CMAKE_OPTIONS``
 - Extra CMake and Arrow options (ex. ``"-DARROW_SIMD_LEVEL=NONE -DCMAKE_OSX_ARCHITECTURES=x86_64;arm64"``)
 - ``''``
 * - ``PYARROW_CXXFLAGS``
 - Extra C++ compiler flags
 - ``''``
 * - ``PYARROW_GENERATE_COVERAGE``
 - Setting ``Xlinetrace`` flag to true for the Cython compiler
 - ``false``
 * - ``PYARROW_BUNDLE_ARROW_CPP``
 - Bundle the Arrow C++ libraries
 - ``0`` (``OFF``)
 * - ``PYARROW_BUNDLE_CYTHON_CPP``
 - Bundle the C++ files generated by Cython
 - ``0`` (``OFF``)
 * - ``PYARROW_INSTALL_TESTS``
 - Add the test to the python package
 - ``1`` (``ON``)
 * - ``PYARROW_BUILD_VERBOSE``
 - Enable verbose output from Makefile builds
 - ``0`` (``OFF``)
 * - ``PYARROW_PARALLEL``
 - Number of processes used to compile PyArrow’s C++/Cython components
 - ``''``

Deleting stale build artifacts
==============================

Expand Down

6 changes: 1 addition & 5 deletions
python/CMakeLists.txt

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -116,16 +116,12 @@ if("${CMAKE_SOURCE_DIR}" STREQUAL "${CMAKE_CURRENT_SOURCE_DIR}")
 option(PYARROW_BUILD_DATASET "Build the PyArrow Dataset integration" OFF)
 option(PYARROW_BUILD_GANDIVA "Build the PyArrow Gandiva integration" OFF)
 option(PYARROW_BUILD_PARQUET "Build the PyArrow Parquet integration" OFF)
 option(PYARROW_PARQUET_USE_SHARED "Rely on parquet shared libraries where relevant" ON)
 option(PYARROW_BUILD_PARQUET_ENCRYPTION
 "Build the PyArrow Parquet encryption integration" OFF)
 option(PYARROW_BOOST_USE_SHARED
 "Rely on Boost shared libraries on linking static parquet" ON)
 option(PYARROW_BUILD_PLASMA "Build the PyArrow Plasma integration" OFF)
 option(PYARROW_USE_TENSORFLOW "Build PyArrow with TensorFlow support" OFF)
 option(PYARROW_BUILD_ORC "Build the PyArrow ORC integration" OFF)
 option(PYARROW_BUNDLE_ARROW_CPP "Bundle the Arrow C++ libraries" OFF)
 option(PYARROW_BUNDLE_BOOST "Bundle the Boost libraries when we bundle Arrow C++" OFF)
 option(PYARROW_BUNDLE_CYTHON_CPP "Bundle the C++ files generated by Cython" OFF)
 option(PYARROW_BUNDLE_PLASMA_EXECUTABLE "Bundle the plasma-store-server" ON)
 option(PYARROW_GENERATE_COVERAGE "Build with Cython code coverage enabled" OFF)
Expand DownExpand Up@@ -585,7 +581,7 @@ if(PYARROW_BUILD_PARQUET)
 DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})
 endif()

 if(PYARROW_PARQUET_USE_SHARED)
 if(ARROW_BUILD_SHARED)
 if(PYARROW_BUNDLE_ARROW_CPP)
 bundle_arrow_lib(${PARQUET_SHARED_LIB} SO_VERSION ${PARQUET_SO_VERSION})
 if(MSVC)
Expand Down

20 changes: 0 additions & 20 deletions
python/setup.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -144,8 +144,6 @@ def initialize_options(self):
 self.extra_cmake_args = os.environ.get('PYARROW_CMAKE_OPTIONS', '')
 self.build_type = os.environ.get('PYARROW_BUILD_TYPE',
 'release').lower()
 self.boost_namespace = os.environ.get('PYARROW_BOOST_NAMESPACE',
 'boost')

 self.cmake_cxxflags = os.environ.get('PYARROW_CXXFLAGS', '')

Expand All@@ -171,12 +169,8 @@ def initialize_options(self):
 os.environ.get('PYARROW_WITH_DATASET', '0'))
 self.with_parquet = strtobool(
 os.environ.get('PYARROW_WITH_PARQUET', '0'))
 self.with_static_parquet = strtobool(
 os.environ.get('PYARROW_WITH_STATIC_PARQUET', '0'))
 self.with_parquet_encryption = strtobool(
 os.environ.get('PYARROW_WITH_PARQUET_ENCRYPTION', '0'))
 self.with_static_boost = strtobool(
 os.environ.get('PYARROW_WITH_STATIC_BOOST', '0'))
 self.with_plasma = strtobool(
 os.environ.get('PYARROW_WITH_PLASMA', '0'))
 self.with_tensorflow = strtobool(
Expand All@@ -191,10 +185,6 @@ def initialize_options(self):
 os.environ.get('PYARROW_BUNDLE_ARROW_CPP', '0'))
 self.bundle_cython_cpp = strtobool(
 os.environ.get('PYARROW_BUNDLE_CYTHON_CPP', '0'))
 self.bundle_boost = strtobool(
 os.environ.get('PYARROW_BUNDLE_BOOST', '0'))
 self.bundle_arrow_cpp_headers = strtobool(
 os.environ.get('PYARROW_BUNDLE_ARROW_CPP_HEADERS', '1'))
 self.bundle_plasma_executable = strtobool(
 os.environ.get('PYARROW_BUNDLE_PLASMA_EXECUTABLE', '1'))

Expand DownExpand Up@@ -297,26 +287,16 @@ def append_cmake_bool(value, varname):
 append_cmake_bool(self.with_tensorflow, 'PYARROW_USE_TENSORFLOW')
 append_cmake_bool(self.bundle_arrow_cpp,
 'PYARROW_BUNDLE_ARROW_CPP')
 append_cmake_bool(self.bundle_boost,
 'PYARROW_BUNDLE_BOOST')
 append_cmake_bool(self.bundle_cython_cpp,
 'PYARROW_BUNDLE_CYTHON_CPP')
 append_cmake_bool(self.bundle_plasma_executable,
 'PYARROW_BUNDLE_PLASMA_EXECUTABLE')
 append_cmake_bool(self.generate_coverage,
 'PYARROW_GENERATE_COVERAGE')
 append_cmake_bool(not self.with_static_boost,
 'PYARROW_BOOST_USE_SHARED')
 append_cmake_bool(not self.with_static_parquet,
 'PYARROW_PARQUET_USE_SHARED')

 cmake_options.append(
 f'-DCMAKE_BUILD_TYPE={self.build_type.lower()}')

 if self.boost_namespace != 'boost':
 cmake_options.append(
 f'-DBoost_NAMESPACE={self.boost_namespace}')

 extra_cmake_args = shlex.split(self.extra_cmake_args)

 build_tool_args = []
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-32619: [Python][Docs] Include options for PyArrow build explicitly#34463

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

jorisvandenbossche
merged 11 commits into

from

Mar 10, 2023
 Merged


GH-32619: [Python][Docs] Include options for PyArrow build explicitly
#34463
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


11 commits


Select commit
Hold shift + click to select a range
73779bb

Add two lists of options for building PyArrow


AlenkaF Mar 6, 2023
d33bc73

Apply suggestions from code review - kou


AlenkaF Mar 7, 2023
105ff7d

Remove plasma from the lists


AlenkaF Mar 7, 2023
cc9e784

Add header row to the first table


AlenkaF Mar 7, 2023
2b7f6cf

Add extra column (default value)


AlenkaF Mar 7, 2023
b014ee9

Remove PYARROW_BUNDLE_ARROW_CPP_HEADERS


AlenkaF Mar 8, 2023
ad512e7

Remove PYARROW_BUNDLE_BOOST


AlenkaF Mar 8, 2023
04ec103

Remove PYARROW_WITH_STATIC_PARQUET and PYARROW_PARQUET_USE_SHARED


AlenkaF Mar 8, 2023
a4dd385

Minor typo corrections


AlenkaF Mar 8, 2023
b1ffac9

Change formatting of the variables


AlenkaF Mar 8, 2023
a5f2ce8

Remove PYARROW_BOOST_NAMESPACE, PYARROW_WITH_STATIC_BOOST and PYARROW…


AlenkaF Mar 9, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


1 change: 0 additions & 1 deletion
ci/appveyor-cpp-build.bat

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -106,7 +106,6 @@ popd

pushd python

set PYARROW_BUNDLE_BOOST=OFF
set PYARROW_CMAKE_GENERATOR=%GENERATOR%
set PYARROW_CXXFLAGS=%ARROW_CXXFLAGS%
set PYARROW_PARALLEL=2
Expand Down

1 change: 0 additions & 1 deletion
ci/scripts/python_wheel_windows_build.bat

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -103,7 +103,6 @@ popd
echo "=== (%PYTHON_VERSION%) Building wheel ==="
set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE%
set PYARROW_BUNDLE_ARROW_CPP=ON
set PYARROW_BUNDLE_BOOST=OFF
set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR%
set PYARROW_INSTALL_TESTS=ON
set PYARROW_WITH_DATASET=%ARROW_DATASET%
Expand Down

1 change: 0 additions & 1 deletion
dev/tasks/conda-recipes/arrow-cpp/build-pyarrow.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -8,7 +8,6 @@ export ARROW_HOME=$PREFIX
export PARQUET_HOME=$PREFIX
export SETUPTOOLS_SCM_PRETEND_VERSION=$PKG_VERSION
export PYARROW_BUILD_TYPE=release
export PYARROW_BUNDLE_ARROW_CPP_HEADERS=0
export PYARROW_WITH_DATASET=1
export PYARROW_WITH_FLIGHT=1
export PYARROW_WITH_GANDIVA=1
Expand Down

77 changes: 77 additions & 0 deletions
docs/source/developers/python.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -586,6 +586,83 @@ Caveats

The Plasma component is not supported on Windows.

Relevant components and environment variables
=============================================

List of relevant Arrow CMake flags and corresponding environment variables
to be used when building PyArrow are:

.. list-table::
 :widths: 30 30
 :header-rows: 1
Comment thread
AlenkaF marked this conversation as resolved.


 * - Arrow flags/options
 - Corresponding environment variables for PyArrow
 * - ``CMAKE_BUILD_TYPE``
 - ``PYARROW_BUILD_TYPE`` (release, debug or relwithdebinfo)
Comment thread
assignUser marked this conversation as resolved.

 * - ``ARROW_GCS``
 - ``PYARROW_WITH_GCS``
 * - ``ARROW_S3``
 - ``PYARROW_WITH_S3``
 * - ``ARROW_HDFS``
 - ``PYARROW_WITH_HDFS``
 * - ``ARROW_CUDA``
 - ``PYARROW_WITH_CUDA``
 * - ``ARROW_SUBSTRAIT``
 - ``PYARROW_WITH_SUBSTRAIT``
 * - ``ARROW_FLIGHT``
 - ``PYARROW_WITH_FLIGHT``
 * - ``ARROW_DATASET``
 - ``PYARROW_WITH_DATASET``
 * - ``ARROW_PARQUET``
 - ``PYARROW_WITH_PARQUET``
 * - ``PARQUET_REQUIRE_ENCRYPTION``
 - ``PYARROW_WITH_PARQUET_ENCRYPTION``
 * - ``ARROW_TENSORFLOW``
 - ``PYARROW_WITH_TENSORFLOW``
 * - ``ARROW_ORC``
 - ``PYARROW_WITH_ORC``
 * - ``ARROW_GANDIVA``
 - ``PYARROW_WITH_GANDIVA``

List of relevant environment variables that can also be used to build
PyArrow are:

.. list-table::
 :widths: 20 20 20
 :header-rows: 1

 * - PyArrow environment variable
 - Description
Comment thread
AlenkaF marked this conversation as resolved.

 - Default value
 * - ``PYARROW_CMAKE_GENERATOR``
 - Example: ``'Visual Studio 15 2017 Win64'``
 - ``''``
 * - ``PYARROW_CMAKE_OPTIONS``
 - Extra CMake and Arrow options (ex. ``"-DARROW_SIMD_LEVEL=NONE -DCMAKE_OSX_ARCHITECTURES=x86_64;arm64"``)
 - ``''``
 * - ``PYARROW_CXXFLAGS``
 - Extra C++ compiler flags
 - ``''``
 * - ``PYARROW_GENERATE_COVERAGE``
 - Setting ``Xlinetrace`` flag to true for the Cython compiler
 - ``false``
 * - ``PYARROW_BUNDLE_ARROW_CPP``
 - Bundle the Arrow C++ libraries
 - ``0`` (``OFF``)
 * - ``PYARROW_BUNDLE_CYTHON_CPP``
 - Bundle the C++ files generated by Cython
 - ``0`` (``OFF``)
 * - ``PYARROW_INSTALL_TESTS``
 - Add the test to the python package
 - ``1`` (``ON``)
 * - ``PYARROW_BUILD_VERBOSE``
 - Enable verbose output from Makefile builds
 - ``0`` (``OFF``)
 * - ``PYARROW_PARALLEL``
 - Number of processes used to compile PyArrow’s C++/Cython components
 - ``''``

Deleting stale build artifacts
==============================

Expand Down

6 changes: 1 addition & 5 deletions
python/CMakeLists.txt

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -116,16 +116,12 @@ if("${CMAKE_SOURCE_DIR}" STREQUAL "${CMAKE_CURRENT_SOURCE_DIR}")
 option(PYARROW_BUILD_DATASET "Build the PyArrow Dataset integration" OFF)
 option(PYARROW_BUILD_GANDIVA "Build the PyArrow Gandiva integration" OFF)
 option(PYARROW_BUILD_PARQUET "Build the PyArrow Parquet integration" OFF)
 option(PYARROW_PARQUET_USE_SHARED "Rely on parquet shared libraries where relevant" ON)
 option(PYARROW_BUILD_PARQUET_ENCRYPTION
 "Build the PyArrow Parquet encryption integration" OFF)
 option(PYARROW_BOOST_USE_SHARED
 "Rely on Boost shared libraries on linking static parquet" ON)
 option(PYARROW_BUILD_PLASMA "Build the PyArrow Plasma integration" OFF)
 option(PYARROW_USE_TENSORFLOW "Build PyArrow with TensorFlow support" OFF)
 option(PYARROW_BUILD_ORC "Build the PyArrow ORC integration" OFF)
 option(PYARROW_BUNDLE_ARROW_CPP "Bundle the Arrow C++ libraries" OFF)
 option(PYARROW_BUNDLE_BOOST "Bundle the Boost libraries when we bundle Arrow C++" OFF)
 option(PYARROW_BUNDLE_CYTHON_CPP "Bundle the C++ files generated by Cython" OFF)
 option(PYARROW_BUNDLE_PLASMA_EXECUTABLE "Bundle the plasma-store-server" ON)
 option(PYARROW_GENERATE_COVERAGE "Build with Cython code coverage enabled" OFF)
Expand DownExpand Up@@ -585,7 +581,7 @@ if(PYARROW_BUILD_PARQUET)
 DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})
 endif()

 if(PYARROW_PARQUET_USE_SHARED)
 if(ARROW_BUILD_SHARED)
 if(PYARROW_BUNDLE_ARROW_CPP)
 bundle_arrow_lib(${PARQUET_SHARED_LIB} SO_VERSION ${PARQUET_SO_VERSION})
 if(MSVC)
Expand Down

20 changes: 0 additions & 20 deletions
python/setup.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -144,8 +144,6 @@ def initialize_options(self):
 self.extra_cmake_args = os.environ.get('PYARROW_CMAKE_OPTIONS', '')
 self.build_type = os.environ.get('PYARROW_BUILD_TYPE',
 'release').lower()
 self.boost_namespace = os.environ.get('PYARROW_BOOST_NAMESPACE',
 'boost')

 self.cmake_cxxflags = os.environ.get('PYARROW_CXXFLAGS', '')

Expand All@@ -171,12 +169,8 @@ def initialize_options(self):
 os.environ.get('PYARROW_WITH_DATASET', '0'))
 self.with_parquet = strtobool(
 os.environ.get('PYARROW_WITH_PARQUET', '0'))
 self.with_static_parquet = strtobool(
 os.environ.get('PYARROW_WITH_STATIC_PARQUET', '0'))
 self.with_parquet_encryption = strtobool(
 os.environ.get('PYARROW_WITH_PARQUET_ENCRYPTION', '0'))
 self.with_static_boost = strtobool(
 os.environ.get('PYARROW_WITH_STATIC_BOOST', '0'))
 self.with_plasma = strtobool(
 os.environ.get('PYARROW_WITH_PLASMA', '0'))
 self.with_tensorflow = strtobool(
Expand All@@ -191,10 +185,6 @@ def initialize_options(self):
 os.environ.get('PYARROW_BUNDLE_ARROW_CPP', '0'))
 self.bundle_cython_cpp = strtobool(
 os.environ.get('PYARROW_BUNDLE_CYTHON_CPP', '0'))
 self.bundle_boost = strtobool(
 os.environ.get('PYARROW_BUNDLE_BOOST', '0'))
 self.bundle_arrow_cpp_headers = strtobool(
 os.environ.get('PYARROW_BUNDLE_ARROW_CPP_HEADERS', '1'))
 self.bundle_plasma_executable = strtobool(
 os.environ.get('PYARROW_BUNDLE_PLASMA_EXECUTABLE', '1'))

Expand DownExpand Up@@ -297,26 +287,16 @@ def append_cmake_bool(value, varname):
 append_cmake_bool(self.with_tensorflow, 'PYARROW_USE_TENSORFLOW')
 append_cmake_bool(self.bundle_arrow_cpp,
 'PYARROW_BUNDLE_ARROW_CPP')
 append_cmake_bool(self.bundle_boost,
 'PYARROW_BUNDLE_BOOST')
 append_cmake_bool(self.bundle_cython_cpp,
 'PYARROW_BUNDLE_CYTHON_CPP')
 append_cmake_bool(self.bundle_plasma_executable,
 'PYARROW_BUNDLE_PLASMA_EXECUTABLE')
 append_cmake_bool(self.generate_coverage,
 'PYARROW_GENERATE_COVERAGE')
 append_cmake_bool(not self.with_static_boost,
 'PYARROW_BOOST_USE_SHARED')
 append_cmake_bool(not self.with_static_parquet,
 'PYARROW_PARQUET_USE_SHARED')

 cmake_options.append(
 f'-DCMAKE_BUILD_TYPE={self.build_type.lower()}')

 if self.boost_namespace != 'boost':
 cmake_options.append(
 f'-DBoost_NAMESPACE={self.boost_namespace}')

 extra_cmake_args = shlex.split(self.extra_cmake_args)

 build_tool_args = []
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-32619: [Python][Docs] Include options for PyArrow build explicitly#34463

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

jorisvandenbossche
merged 11 commits into

from

Mar 10, 2023
 Merged


GH-32619: [Python][Docs] Include options for PyArrow build explicitly
#34463
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


11 commits


Select commit
Hold shift + click to select a range
73779bb

Add two lists of options for building PyArrow


AlenkaF Mar 6, 2023
d33bc73

Apply suggestions from code review - kou


AlenkaF Mar 7, 2023
105ff7d

Remove plasma from the lists


AlenkaF Mar 7, 2023
cc9e784

Add header row to the first table


AlenkaF Mar 7, 2023
2b7f6cf

Add extra column (default value)


AlenkaF Mar 7, 2023
b014ee9

Remove PYARROW_BUNDLE_ARROW_CPP_HEADERS


AlenkaF Mar 8, 2023
ad512e7

Remove PYARROW_BUNDLE_BOOST


AlenkaF Mar 8, 2023
04ec103

Remove PYARROW_WITH_STATIC_PARQUET and PYARROW_PARQUET_USE_SHARED


AlenkaF Mar 8, 2023
a4dd385

Minor typo corrections


AlenkaF Mar 8, 2023
b1ffac9

Change formatting of the variables


AlenkaF Mar 8, 2023
a5f2ce8

Remove PYARROW_BOOST_NAMESPACE, PYARROW_WITH_STATIC_BOOST and PYARROW…


AlenkaF Mar 9, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


1 change: 0 additions & 1 deletion
ci/appveyor-cpp-build.bat

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -106,7 +106,6 @@ popd

pushd python

set PYARROW_BUNDLE_BOOST=OFF
set PYARROW_CMAKE_GENERATOR=%GENERATOR%
set PYARROW_CXXFLAGS=%ARROW_CXXFLAGS%
set PYARROW_PARALLEL=2
Expand Down

1 change: 0 additions & 1 deletion
ci/scripts/python_wheel_windows_build.bat

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -103,7 +103,6 @@ popd
echo "=== (%PYTHON_VERSION%) Building wheel ==="
set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE%
set PYARROW_BUNDLE_ARROW_CPP=ON
set PYARROW_BUNDLE_BOOST=OFF
set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR%
set PYARROW_INSTALL_TESTS=ON
set PYARROW_WITH_DATASET=%ARROW_DATASET%
Expand Down

1 change: 0 additions & 1 deletion
dev/tasks/conda-recipes/arrow-cpp/build-pyarrow.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -8,7 +8,6 @@ export ARROW_HOME=$PREFIX
export PARQUET_HOME=$PREFIX
export SETUPTOOLS_SCM_PRETEND_VERSION=$PKG_VERSION
export PYARROW_BUILD_TYPE=release
export PYARROW_BUNDLE_ARROW_CPP_HEADERS=0
export PYARROW_WITH_DATASET=1
export PYARROW_WITH_FLIGHT=1
export PYARROW_WITH_GANDIVA=1
Expand Down

77 changes: 77 additions & 0 deletions
docs/source/developers/python.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -586,6 +586,83 @@ Caveats

The Plasma component is not supported on Windows.

Relevant components and environment variables
=============================================

List of relevant Arrow CMake flags and corresponding environment variables
to be used when building PyArrow are:

.. list-table::
 :widths: 30 30
 :header-rows: 1
Comment thread
AlenkaF marked this conversation as resolved.


 * - Arrow flags/options
 - Corresponding environment variables for PyArrow
 * - ``CMAKE_BUILD_TYPE``
 - ``PYARROW_BUILD_TYPE`` (release, debug or relwithdebinfo)
Comment thread
assignUser marked this conversation as resolved.

 * - ``ARROW_GCS``
 - ``PYARROW_WITH_GCS``
 * - ``ARROW_S3``
 - ``PYARROW_WITH_S3``
 * - ``ARROW_HDFS``
 - ``PYARROW_WITH_HDFS``
 * - ``ARROW_CUDA``
 - ``PYARROW_WITH_CUDA``
 * - ``ARROW_SUBSTRAIT``
 - ``PYARROW_WITH_SUBSTRAIT``
 * - ``ARROW_FLIGHT``
 - ``PYARROW_WITH_FLIGHT``
 * - ``ARROW_DATASET``
 - ``PYARROW_WITH_DATASET``
 * - ``ARROW_PARQUET``
 - ``PYARROW_WITH_PARQUET``
 * - ``PARQUET_REQUIRE_ENCRYPTION``
 - ``PYARROW_WITH_PARQUET_ENCRYPTION``
 * - ``ARROW_TENSORFLOW``
 - ``PYARROW_WITH_TENSORFLOW``
 * - ``ARROW_ORC``
 - ``PYARROW_WITH_ORC``
 * - ``ARROW_GANDIVA``
 - ``PYARROW_WITH_GANDIVA``

List of relevant environment variables that can also be used to build
PyArrow are:

.. list-table::
 :widths: 20 20 20
 :header-rows: 1

 * - PyArrow environment variable
 - Description
Comment thread
AlenkaF marked this conversation as resolved.

 - Default value
 * - ``PYARROW_CMAKE_GENERATOR``
 - Example: ``'Visual Studio 15 2017 Win64'``
 - ``''``
 * - ``PYARROW_CMAKE_OPTIONS``
 - Extra CMake and Arrow options (ex. ``"-DARROW_SIMD_LEVEL=NONE -DCMAKE_OSX_ARCHITECTURES=x86_64;arm64"``)
 - ``''``
 * - ``PYARROW_CXXFLAGS``
 - Extra C++ compiler flags
 - ``''``
 * - ``PYARROW_GENERATE_COVERAGE``
 - Setting ``Xlinetrace`` flag to true for the Cython compiler
 - ``false``
 * - ``PYARROW_BUNDLE_ARROW_CPP``
 - Bundle the Arrow C++ libraries
 - ``0`` (``OFF``)
 * - ``PYARROW_BUNDLE_CYTHON_CPP``
 - Bundle the C++ files generated by Cython
 - ``0`` (``OFF``)
 * - ``PYARROW_INSTALL_TESTS``
 - Add the test to the python package
 - ``1`` (``ON``)
 * - ``PYARROW_BUILD_VERBOSE``
 - Enable verbose output from Makefile builds
 - ``0`` (``OFF``)
 * - ``PYARROW_PARALLEL``
 - Number of processes used to compile PyArrow’s C++/Cython components
 - ``''``

Deleting stale build artifacts
==============================

Expand Down

6 changes: 1 addition & 5 deletions
python/CMakeLists.txt

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -116,16 +116,12 @@ if("${CMAKE_SOURCE_DIR}" STREQUAL "${CMAKE_CURRENT_SOURCE_DIR}")
 option(PYARROW_BUILD_DATASET "Build the PyArrow Dataset integration" OFF)
 option(PYARROW_BUILD_GANDIVA "Build the PyArrow Gandiva integration" OFF)
 option(PYARROW_BUILD_PARQUET "Build the PyArrow Parquet integration" OFF)
 option(PYARROW_PARQUET_USE_SHARED "Rely on parquet shared libraries where relevant" ON)
 option(PYARROW_BUILD_PARQUET_ENCRYPTION
 "Build the PyArrow Parquet encryption integration" OFF)
 option(PYARROW_BOOST_USE_SHARED
 "Rely on Boost shared libraries on linking static parquet" ON)
 option(PYARROW_BUILD_PLASMA "Build the PyArrow Plasma integration" OFF)
 option(PYARROW_USE_TENSORFLOW "Build PyArrow with TensorFlow support" OFF)
 option(PYARROW_BUILD_ORC "Build the PyArrow ORC integration" OFF)
 option(PYARROW_BUNDLE_ARROW_CPP "Bundle the Arrow C++ libraries" OFF)
 option(PYARROW_BUNDLE_BOOST "Bundle the Boost libraries when we bundle Arrow C++" OFF)
 option(PYARROW_BUNDLE_CYTHON_CPP "Bundle the C++ files generated by Cython" OFF)
 option(PYARROW_BUNDLE_PLASMA_EXECUTABLE "Bundle the plasma-store-server" ON)
 option(PYARROW_GENERATE_COVERAGE "Build with Cython code coverage enabled" OFF)
Expand DownExpand Up@@ -585,7 +581,7 @@ if(PYARROW_BUILD_PARQUET)
 DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})
 endif()

 if(PYARROW_PARQUET_USE_SHARED)
 if(ARROW_BUILD_SHARED)
 if(PYARROW_BUNDLE_ARROW_CPP)
 bundle_arrow_lib(${PARQUET_SHARED_LIB} SO_VERSION ${PARQUET_SO_VERSION})
 if(MSVC)
Expand Down

20 changes: 0 additions & 20 deletions
python/setup.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -144,8 +144,6 @@ def initialize_options(self):
 self.extra_cmake_args = os.environ.get('PYARROW_CMAKE_OPTIONS', '')
 self.build_type = os.environ.get('PYARROW_BUILD_TYPE',
 'release').lower()
 self.boost_namespace = os.environ.get('PYARROW_BOOST_NAMESPACE',
 'boost')

 self.cmake_cxxflags = os.environ.get('PYARROW_CXXFLAGS', '')

Expand All@@ -171,12 +169,8 @@ def initialize_options(self):
 os.environ.get('PYARROW_WITH_DATASET', '0'))
 self.with_parquet = strtobool(
 os.environ.get('PYARROW_WITH_PARQUET', '0'))
 self.with_static_parquet = strtobool(
 os.environ.get('PYARROW_WITH_STATIC_PARQUET', '0'))
 self.with_parquet_encryption = strtobool(
 os.environ.get('PYARROW_WITH_PARQUET_ENCRYPTION', '0'))
 self.with_static_boost = strtobool(
 os.environ.get('PYARROW_WITH_STATIC_BOOST', '0'))
 self.with_plasma = strtobool(
 os.environ.get('PYARROW_WITH_PLASMA', '0'))
 self.with_tensorflow = strtobool(
Expand All@@ -191,10 +185,6 @@ def initialize_options(self):
 os.environ.get('PYARROW_BUNDLE_ARROW_CPP', '0'))
 self.bundle_cython_cpp = strtobool(
 os.environ.get('PYARROW_BUNDLE_CYTHON_CPP', '0'))
 self.bundle_boost = strtobool(
 os.environ.get('PYARROW_BUNDLE_BOOST', '0'))
 self.bundle_arrow_cpp_headers = strtobool(
 os.environ.get('PYARROW_BUNDLE_ARROW_CPP_HEADERS', '1'))
 self.bundle_plasma_executable = strtobool(
 os.environ.get('PYARROW_BUNDLE_PLASMA_EXECUTABLE', '1'))

Expand DownExpand Up@@ -297,26 +287,16 @@ def append_cmake_bool(value, varname):
 append_cmake_bool(self.with_tensorflow, 'PYARROW_USE_TENSORFLOW')
 append_cmake_bool(self.bundle_arrow_cpp,
 'PYARROW_BUNDLE_ARROW_CPP')
 append_cmake_bool(self.bundle_boost,
 'PYARROW_BUNDLE_BOOST')
 append_cmake_bool(self.bundle_cython_cpp,
 'PYARROW_BUNDLE_CYTHON_CPP')
 append_cmake_bool(self.bundle_plasma_executable,
 'PYARROW_BUNDLE_PLASMA_EXECUTABLE')
 append_cmake_bool(self.generate_coverage,
 'PYARROW_GENERATE_COVERAGE')
 append_cmake_bool(not self.with_static_boost,
 'PYARROW_BOOST_USE_SHARED')
 append_cmake_bool(not self.with_static_parquet,
 'PYARROW_PARQUET_USE_SHARED')

 cmake_options.append(
 f'-DCMAKE_BUILD_TYPE={self.build_type.lower()}')

 if self.boost_namespace != 'boost':
 cmake_options.append(
 f'-DBoost_NAMESPACE={self.boost_namespace}')

 extra_cmake_args = shlex.split(self.extra_cmake_args)

 build_tool_args = []
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
GH-32619: [Python][Docs] Include options for PyArrow build explicitly#34463

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

jorisvandenbossche
merged 11 commits into

from

Mar 10, 2023
 Merged


GH-32619: [Python][Docs] Include options for PyArrow build explicitly
#34463
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


11 commits


Select commit
Hold shift + click to select a range
73779bb

Add two lists of options for building PyArrow


AlenkaF Mar 6, 2023
d33bc73

Apply suggestions from code review - kou


AlenkaF Mar 7, 2023
105ff7d

Remove plasma from the lists


AlenkaF Mar 7, 2023
cc9e784

Add header row to the first table


AlenkaF Mar 7, 2023
2b7f6cf

Add extra column (default value)


AlenkaF Mar 7, 2023
b014ee9

Remove PYARROW_BUNDLE_ARROW_CPP_HEADERS


AlenkaF Mar 8, 2023
ad512e7

Remove PYARROW_BUNDLE_BOOST


AlenkaF Mar 8, 2023
04ec103

Remove PYARROW_WITH_STATIC_PARQUET and PYARROW_PARQUET_USE_SHARED


AlenkaF Mar 8, 2023
a4dd385

Minor typo corrections


AlenkaF Mar 8, 2023
b1ffac9

Change formatting of the variables


AlenkaF Mar 8, 2023
a5f2ce8

Remove PYARROW_BOOST_NAMESPACE, PYARROW_WITH_STATIC_BOOST and PYARROW…


AlenkaF Mar 9, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


1 change: 0 additions & 1 deletion
ci/appveyor-cpp-build.bat

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -106,7 +106,6 @@ popd

pushd python

set PYARROW_BUNDLE_BOOST=OFF
set PYARROW_CMAKE_GENERATOR=%GENERATOR%
set PYARROW_CXXFLAGS=%ARROW_CXXFLAGS%
set PYARROW_PARALLEL=2
Expand Down

1 change: 0 additions & 1 deletion
ci/scripts/python_wheel_windows_build.bat

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -103,7 +103,6 @@ popd
echo "=== (%PYTHON_VERSION%) Building wheel ==="
set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE%
set PYARROW_BUNDLE_ARROW_CPP=ON
set PYARROW_BUNDLE_BOOST=OFF
set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR%
set PYARROW_INSTALL_TESTS=ON
set PYARROW_WITH_DATASET=%ARROW_DATASET%
Expand Down

1 change: 0 additions & 1 deletion
dev/tasks/conda-recipes/arrow-cpp/build-pyarrow.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -8,7 +8,6 @@ export ARROW_HOME=$PREFIX
export PARQUET_HOME=$PREFIX
export SETUPTOOLS_SCM_PRETEND_VERSION=$PKG_VERSION
export PYARROW_BUILD_TYPE=release
export PYARROW_BUNDLE_ARROW_CPP_HEADERS=0
export PYARROW_WITH_DATASET=1
export PYARROW_WITH_FLIGHT=1
export PYARROW_WITH_GANDIVA=1
Expand Down

77 changes: 77 additions & 0 deletions
docs/source/developers/python.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -586,6 +586,83 @@ Caveats

The Plasma component is not supported on Windows.

Relevant components and environment variables
=============================================

List of relevant Arrow CMake flags and corresponding environment variables
to be used when building PyArrow are:

.. list-table::
 :widths: 30 30
 :header-rows: 1
Comment thread
AlenkaF marked this conversation as resolved.


 * - Arrow flags/options
 - Corresponding environment variables for PyArrow
 * - ``CMAKE_BUILD_TYPE``
 - ``PYARROW_BUILD_TYPE`` (release, debug or relwithdebinfo)
Comment thread
assignUser marked this conversation as resolved.

 * - ``ARROW_GCS``
 - ``PYARROW_WITH_GCS``
 * - ``ARROW_S3``
 - ``PYARROW_WITH_S3``
 * - ``ARROW_HDFS``
 - ``PYARROW_WITH_HDFS``
 * - ``ARROW_CUDA``
 - ``PYARROW_WITH_CUDA``
 * - ``ARROW_SUBSTRAIT``
 - ``PYARROW_WITH_SUBSTRAIT``
 * - ``ARROW_FLIGHT``
 - ``PYARROW_WITH_FLIGHT``
 * - ``ARROW_DATASET``
 - ``PYARROW_WITH_DATASET``
 * - ``ARROW_PARQUET``
 - ``PYARROW_WITH_PARQUET``
 * - ``PARQUET_REQUIRE_ENCRYPTION``
 - ``PYARROW_WITH_PARQUET_ENCRYPTION``
 * - ``ARROW_TENSORFLOW``
 - ``PYARROW_WITH_TENSORFLOW``
 * - ``ARROW_ORC``
 - ``PYARROW_WITH_ORC``
 * - ``ARROW_GANDIVA``
 - ``PYARROW_WITH_GANDIVA``

List of relevant environment variables that can also be used to build
PyArrow are:

.. list-table::
 :widths: 20 20 20
 :header-rows: 1

 * - PyArrow environment variable
 - Description
Comment thread
AlenkaF marked this conversation as resolved.

 - Default value
 * - ``PYARROW_CMAKE_GENERATOR``
 - Example: ``'Visual Studio 15 2017 Win64'``
 - ``''``
 * - ``PYARROW_CMAKE_OPTIONS``
 - Extra CMake and Arrow options (ex. ``"-DARROW_SIMD_LEVEL=NONE -DCMAKE_OSX_ARCHITECTURES=x86_64;arm64"``)
 - ``''``
 * - ``PYARROW_CXXFLAGS``
 - Extra C++ compiler flags
 - ``''``
 * - ``PYARROW_GENERATE_COVERAGE``
 - Setting ``Xlinetrace`` flag to true for the Cython compiler
 - ``false``
 * - ``PYARROW_BUNDLE_ARROW_CPP``
 - Bundle the Arrow C++ libraries
 - ``0`` (``OFF``)
 * - ``PYARROW_BUNDLE_CYTHON_CPP``
 - Bundle the C++ files generated by Cython
 - ``0`` (``OFF``)
 * - ``PYARROW_INSTALL_TESTS``
 - Add the test to the python package
 - ``1`` (``ON``)
 * - ``PYARROW_BUILD_VERBOSE``
 - Enable verbose output from Makefile builds
 - ``0`` (``OFF``)
 * - ``PYARROW_PARALLEL``
 - Number of processes used to compile PyArrow’s C++/Cython components
 - ``''``

Deleting stale build artifacts
==============================

Expand Down

6 changes: 1 addition & 5 deletions
python/CMakeLists.txt

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -116,16 +116,12 @@ if("${CMAKE_SOURCE_DIR}" STREQUAL "${CMAKE_CURRENT_SOURCE_DIR}")
 option(PYARROW_BUILD_DATASET "Build the PyArrow Dataset integration" OFF)
 option(PYARROW_BUILD_GANDIVA "Build the PyArrow Gandiva integration" OFF)
 option(PYARROW_BUILD_PARQUET "Build the PyArrow Parquet integration" OFF)
 option(PYARROW_PARQUET_USE_SHARED "Rely on parquet shared libraries where relevant" ON)
 option(PYARROW_BUILD_PARQUET_ENCRYPTION
 "Build the PyArrow Parquet encryption integration" OFF)
 option(PYARROW_BOOST_USE_SHARED
 "Rely on Boost shared libraries on linking static parquet" ON)
 option(PYARROW_BUILD_PLASMA "Build the PyArrow Plasma integration" OFF)
 option(PYARROW_USE_TENSORFLOW "Build PyArrow with TensorFlow support" OFF)
 option(PYARROW_BUILD_ORC "Build the PyArrow ORC integration" OFF)
 option(PYARROW_BUNDLE_ARROW_CPP "Bundle the Arrow C++ libraries" OFF)
 option(PYARROW_BUNDLE_BOOST "Bundle the Boost libraries when we bundle Arrow C++" OFF)
 option(PYARROW_BUNDLE_CYTHON_CPP "Bundle the C++ files generated by Cython" OFF)
 option(PYARROW_BUNDLE_PLASMA_EXECUTABLE "Bundle the plasma-store-server" ON)
 option(PYARROW_GENERATE_COVERAGE "Build with Cython code coverage enabled" OFF)
Expand DownExpand Up@@ -585,7 +581,7 @@ if(PYARROW_BUILD_PARQUET)
 DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})
 endif()

 if(PYARROW_PARQUET_USE_SHARED)
 if(ARROW_BUILD_SHARED)
 if(PYARROW_BUNDLE_ARROW_CPP)
 bundle_arrow_lib(${PARQUET_SHARED_LIB} SO_VERSION ${PARQUET_SO_VERSION})
 if(MSVC)
Expand Down

20 changes: 0 additions & 20 deletions
python/setup.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -144,8 +144,6 @@ def initialize_options(self):
 self.extra_cmake_args = os.environ.get('PYARROW_CMAKE_OPTIONS', '')
 self.build_type = os.environ.get('PYARROW_BUILD_TYPE',
 'release').lower()
 self.boost_namespace = os.environ.get('PYARROW_BOOST_NAMESPACE',
 'boost')

 self.cmake_cxxflags = os.environ.get('PYARROW_CXXFLAGS', '')

Expand All@@ -171,12 +169,8 @@ def initialize_options(self):
 os.environ.get('PYARROW_WITH_DATASET', '0'))
 self.with_parquet = strtobool(
 os.environ.get('PYARROW_WITH_PARQUET', '0'))
 self.with_static_parquet = strtobool(
 os.environ.get('PYARROW_WITH_STATIC_PARQUET', '0'))
 self.with_parquet_encryption = strtobool(
 os.environ.get('PYARROW_WITH_PARQUET_ENCRYPTION', '0'))
 self.with_static_boost = strtobool(
 os.environ.get('PYARROW_WITH_STATIC_BOOST', '0'))
 self.with_plasma = strtobool(
 os.environ.get('PYARROW_WITH_PLASMA', '0'))
 self.with_tensorflow = strtobool(
Expand All@@ -191,10 +185,6 @@ def initialize_options(self):
 os.environ.get('PYARROW_BUNDLE_ARROW_CPP', '0'))
 self.bundle_cython_cpp = strtobool(
 os.environ.get('PYARROW_BUNDLE_CYTHON_CPP', '0'))
 self.bundle_boost = strtobool(
 os.environ.get('PYARROW_BUNDLE_BOOST', '0'))
 self.bundle_arrow_cpp_headers = strtobool(
 os.environ.get('PYARROW_BUNDLE_ARROW_CPP_HEADERS', '1'))
 self.bundle_plasma_executable = strtobool(
 os.environ.get('PYARROW_BUNDLE_PLASMA_EXECUTABLE', '1'))

Expand DownExpand Up@@ -297,26 +287,16 @@ def append_cmake_bool(value, varname):
 append_cmake_bool(self.with_tensorflow, 'PYARROW_USE_TENSORFLOW')
 append_cmake_bool(self.bundle_arrow_cpp,
 'PYARROW_BUNDLE_ARROW_CPP')
 append_cmake_bool(self.bundle_boost,
 'PYARROW_BUNDLE_BOOST')
 append_cmake_bool(self.bundle_cython_cpp,
 'PYARROW_BUNDLE_CYTHON_CPP')
 append_cmake_bool(self.bundle_plasma_executable,
 'PYARROW_BUNDLE_PLASMA_EXECUTABLE')
 append_cmake_bool(self.generate_coverage,
 'PYARROW_GENERATE_COVERAGE')
 append_cmake_bool(not self.with_static_boost,
 'PYARROW_BOOST_USE_SHARED')
 append_cmake_bool(not self.with_static_parquet,
 'PYARROW_PARQUET_USE_SHARED')

 cmake_options.append(
 f'-DCMAKE_BUILD_TYPE={self.build_type.lower()}')

 if self.boost_namespace != 'boost':
 cmake_options.append(
 f'-DBoost_NAMESPACE={self.boost_namespace}')

 extra_cmake_args = shlex.split(self.extra_cmake_args)

 build_tool_args = []
Expand Down